Implications of HDF5

[Ray Osborn]

I know that some of you have been looking at HDF5 to see what impact it will
have on the NeXus format.  My own initial impressions are extremely positive
- in fact, I'm amused at how many decisions that we took for NeXus are also
adopted in HDF5.  For example, they have reduced the number of HDF objects
to two - groups and multidimensional data (i.e. Vgroups and SDSs).  They
have also used the same file directory model for navigating the files, and
have implemented Unix-style paths for accessing individual objects
(/scan1/data/yarray).  They don't seem to have group classes, although we
could easily implement them by using group attributes.  They also don't seem
to have any method for implementing dimension scales.  In some ways, this is
a relief, because limitations in the HDF4 implementation meant we couldn't
use them anyway.  However, I wonder how generic browsers will choose to plot
the data now. 

It looks to me as if the NeXus API can be transparently layered on HDF5,
which justifies our decision to write our own API layer.  The reason I am
writing is to ask whether we should consider starting the transition now,
even though HDF5 is only at beta level.  I know that Mark now has a lot of
HDF4 files, that may need translating (although the claim is that the HDF5
API will eventually read HDF4 files).  However, the rest of us are still at
the testing stage so now would be a good time to act.  It would also give us
an opportunity to contribute to the final definition of HDF5 by seeing if
there are limitations that we want them to address.  We have the potential
of being a large user group for HDF5, and so we should use the chances we
have to influence their final design decisions.  For example, we might be
able to persuade them to implement group classes (or even data classes) if
we feel it is essential.  HDF5 should have better performance than HDF4, but
we could at least see what the limitations are.

I would encourage such a move at the earliest opportunity.  The two barriers
are that HDF5 has not been tested on as many platforms as HDF4, so it will
be more difficult to implement, and secondly, we will depend on Mark (or
Przemek or Nick or Freddie) having the time to rewrite the code.  Any
volunteers?

Regards,
Ray
-- 
Dr Ray Osborn                Tel: +1 (630) 252-9011
Materials Science Division   Fax: +1 (630) 252-7777
Argonne National Laboratory  E-mail: ROsborn at anl.gov
Argonne, IL 60439-4845