[Nexus-developers] NeXus file conversions, NOBUGS

[Ray Osborn]

On 10/7/02 1:38 AM, "Mark Koennecke" <Mark.Koennecke at psi.ch> wrote:

>  concerning Rays recent messages:
> 
>  - I hacked NCSA's standard h4toh5 utility which comes with the HDF-5
>    distribution to convert HDF-4 NeXus files to HDF-5 NeXus files
>    properly. Not much work really but some tedium about string handling.
>    This utility treats links properly. This was not only stated by NCSA
>    but I checked it also by converting a larger data file. Perhaps I
>    should create a patchkit for this sometime....
> 
>  - Hacking the h5toh4 utility proved to be to much work, as the file
>    structure is changed big time by this utility ( it creates a root
>    group from which everything starts).
> 
>  So there is a good path to convert to the new world of HDF-5. Is the
>  backwards path so important that we have to expend a lot of effort?

It may be that my unsophisticated utility, NXtoNX4, is all that we need so I
wouldn't recommend doing much work.  I think there is a need for such a
utility because there are still some systems that do not have HDF5 (such as
VMS).  I am happy for NXtoNX5 to be better implemented.

>  - One thing we should urgently sort out at NOBUGS is the XML issue.
>    People start to define XML file structures everywhere: Ray has
>    defined one, Steve King has done a similar thing, Emanuel Farhi as
>    well, there is the XML-file structure created by using the NCSA HDF-5
>    to XML utility. Perhaps this one could be check for the link handling
>    problem. I think we should agree on a standard NeXus-XML definition
>    as soon as possible. Nobody has yet a large investment in XML
>    software, this may not be the case by the time of the next NOBUGS
>    meeting. 

Well, there is an XML file structure already defined - it is defined by
<http://www.neutron.anl.gov/nexus/NeXus_metaformat.html> and is implemented
in NXtoXML.  Since there has been no discussion about it since I announced
it on the NeXus mailing list, it is the de facto NeXus standard.  I have a
submitted a poster about it to NOBUGS, and we can discuss it at the NeXus
break-out session.

Of course, it doesn't define what goes in the files for particular
instruments.  That is, I presume, the subject of Steve King's and several
other people's NOBUGS contributions (I don't know about Farhi).

I don't think that there are many ways to map NeXus files to XML so it
shouldn't be controversial.  Group classes have to become the keywords, with
the group names as attributes, and we need to define a data type attribute
that is not really a NeXus attribute.  Apart from that, the mapping is
obvious, apart from the linking issue.

> -  I believe now that it becomes unavoidable to extend the NeXus-API to
>    deal with XML-files as well. Forward to version 3.0! XML would also
>    address many peoples wish to have ASCII data files for easier
>    manipulation. 

If you run NXtoXML, you can realize people's dreams today.  Actually, I
found it helpful to see the NeXus files in this form, and I've started to
use it in the web documentation for illustrations, e.g. see
<http://www.neutron.anl.gov/nexus/NeXus_structure.html#Data>.

Mark, if you want to say something about instrument definitions in your
talk, please do so, but please base it on the XML specifications on the
NeXus web site.  This was announced to the mailing list several months ago
and has been used on the web site for several months, so we should use it as
the basis of future planning even if modifications are needed.  This will
also be the main subject of discussion at the break-out session.

This is the text of my NOBUGS abstract, since it isn't public yet.

NeXus Integration with XML

The NeXus data format utilizes a set of design principles to make it easy to
understand the contents of a NeXus file without any prior information (see
for more details). However, to assist those writing visualization or
analysis software for particular classes of neutron or x-ray instruments, we
are specifying their file definitions using XML. The NeXus data model may be
accurately represented in XML, as well as HDF, and conversion utilities
between the two now exist. The use of XML to formalize NeXus file
definitions brings with it a number of advantages, including proper version
control, format validation by standard XML utilities, use of the XML DOM for
parsing and displaying NeXus files, and simplified integration with
databases, e.g for use in web services or NeXus file catalogs.

Regards,
Ray
-- 
Dr Ray Osborn                Tel: +1 (630) 252-9011
Materials Science Division   Fax: +1 (630) 252-7777
Argonne National Laboratory  E-mail: ROsborn at anl.gov
Argonne, IL 60439-4845