[Nexus-developers] RE: NeXus-API XML in alpha

[Peterson, Peter F.]

Mark,

Thank you for the hard work. Please see my comments below.

P^2

-----Original Message-----
From: Mark Koennecke [mailto:Mark.Koennecke at psi.ch]
Sent: Tue 10/5/2004 9:06 AM
To: Peterson, Peter F.
Cc: nexus-developers at anl.gov
Subject: NeXus-API XML in alpha
 
  High,

  the last argument against NeXus: "It is not ASCII, I cannot edit my data"
  is crumbling.

  The NeXus-API for XML passes the standard NeXus-API self test which at
  least promotes it to alpha. Contrary to my previous e-mail, I decided
  to base the XML part of the API on mini-XML, a tree API for XML. The
  mxml homepage:
            http://www.easysw.com/~mike/mxml
  I worked with the author of mxml, Michael Sweet, to extend mxml in such
  a way that we can store NeXus datasets in the tree efficiently. I
  very much hope, that my changes will make it in the standard
  distribution of mxml; so that we do not have to maintain that
  library as well.  

At first I was curious why you didn't use something a more widespread, like libxml. Then I discovered that this provides a library for reading and writing. I did not find information in the documentation, does this only parse using DOM (no SAX support)? The reason I ask is utilities that use XML would be easiest if there is only one XML library used, and libxml is on redhat installations (v>9.0) for the system.

  For links I took the liberty to store them as elements of type NXlink
  with an attribute target which is the path to the linked item. NXopengroup,
  NXopendata, etc follow these links. The implementation of the
  NeXus-API is next to complete, but two issues remain:
  - I did not implement unlimited dimensions. This would be complicated,
    and I am not convinced that we need it.
  - Currently, the XML-API rejects non text array type attributes and
    converts single number attributes silently to text. Reading attributes
    returns only text attributes. If this is not acceptable, we need to
    store type information with the attribute. I would suggest a scheme 
    like some_attribute="NX_INT32[10]: 1 2 3 4 5 6 7 8 9 10". What does
    everybody else think?
  The XML NeXus API can be enabled through the define: NXXML.

Unlimited dimensions is not allowed by the API already. There is a limit to <25 dimensions. I do not know if there is an error check on it, but there will be a buffer overflow.

For attribute types, NXtranslate tries to convert them to ints, then floats, and finally character arrays. There is no concept of int arrays as attributes. Your solution of including the type and length in the value is much more elegant. I would add one special case: if there is no type (the first three characters are not "NX_") then the value is a character array.

  I also fixed the issue with NXIReportError which ocurred when the 
  different files of the API were compiled separatly. This was a
  prototyp problem which I solved by putting some more things into
  napi.h. All the NeXus-API bits can now be compiled separatly.

Didn't know about this. 

  Peter, long ago, reported an issue to me about performance problems
  when reading multi dimensional arrays from the Java-API. I looked
  into this problem and  I'am afraid this can only be fixed by
  introducing another performance issue, a memory problem now:
  Currently, the Java-API already needs two copies of the data for
  conversions from NeXus to Java number formats. If we fix the
  performance problem we need three:
  - the byte arrya coming from the C-API
  - An intermediate one dimensional array
  - The final multi dimensional array.
  I am not yet ready to do this.  I think the implementation in 
  HDFArray.java which converts rows of multi dimensional data sets is 
  the best we can do in the general case. Has anybody got a better idea?

There only needs to be two copies, one for C and a multi-dimensional version for Java. Java reads in slabs supplying subarrays from what you allocated to napi. A working example is found in NXvalid, the file gov.anl.util.NXutil, the function starting at line 159.  

  The autoconf stuff does not work again. Even on the upcoming PSI
  Linux distribution, Scientific Linux 3.2 (which is Redhat Enterprise
  3.2) it does not work. I think the problem is that the autoconf
  stuff was developed on a bleeding edge Linux distribution. For
  portability, may be, one should develop autoconf scripts on the oldest
  machine available. I still have a Redhat 6.2 machine. Perhaps we can
  resolve this issue during the NIAC meeting when all the experts are
  here. PSI also provides a sizable number of test platforms. I am
  quite sure that the problem is with the scripts. Normally configure
  scripts run on my machines. They usually complain because a library
  is too old, but this is a completely unrelated matter.

No comment.

  Back to NeXus-XML: If nobody objects, I will tag the current
  version in the CVS as 2.1 and upload the new stuff which will then
  become NeXus-3.0 after some more testing by you. 
  
Before we release a new version I would like to have the C++ API written. A thread on the wiki exists for what functions should be included at <http://www.neutron.anl.gov.:8080/NeXus/84>

                   See you soon,

		       Mark Koennecke