area detector data?

[Ray Osborn]

on 99/11/16 3:37 PM, Brian Tieman at tieman at aps.anl.gov wrote:

> Origionally the design goal was to be 100% NeXus compatible--however it
> soon became apparent the the various pieces of metadata required would not
> fit into the current NeXus defined groups.  This coupled with the fact
> that the acquisition system is actually being used for numerous types of
> experiments (microscopy, tomography, holography, diffraction,
> etc...)--each with a differing set of metadata--led us to the development
> of a rather unique and highly useful (in our case anyway) library of code.
> 

I'm concerned about this statement for a number of reasons.  The original
NeXus team attempted to define the rules for creating NeXus files to be as
flexible as possible, without losing all the advantages of having a
standard.  In particular, NeXus is able to cope with an enormous amount of
metadata in addition to the detector counts themselves.  I find it difficult
to believe that we can't accommodate the extra information you require in
the existing skeletal structure.

If it's necessary to define other group classes, beyond what is currently on
the web page, we are certainly open to proposals.  In fact, the NeXus
glossary and instrument definitions are the least-developed aspects of the
format, and we have always intended to add to them as we got more input from
the community.  Perhaps we should have made that clearer on the web pages.

I would be interested to learn specifically what couldn't be stored in the
format.  I would prefer that we augment the standard to overcome these
limitations rather than fragment the format with local extensions.

> 
> We currently don't do any sort of file compression.  A typical image for
> us is 1024 pixel X 1024 pixels X 16 bits or ~ 2MB.  I haven't looked at
> NeXus 1.2.1 yet, but I don't see where NeXus supports compression.  I
> intend to do this, but I need to base everything off pure HDF first.  Once
> there, I will add file compression as an option.
> 

HDF 4.1r3 allows for internal data compression of datasets using a variety
of algorithms.  We have not yet implemented any of them because it had not
yet got to the top of the priority list.  If this is critical for any
particular user, we can see if it can be moved up.  I don't think it's that
difficult, unless the performance penalty is significant.

Ray Osborn

P.S. I've tried to configure the mailing list to direct replies back to the
list by default, but it hasn't been working.  Until I can fix it, please be
aware of the problem, and manually redirect the replies.   I think that some
of the correspondence following this posting never made it to the list, even
though the writers may have wanted it to.

-- 
Dr Ray Osborn                Tel: +1 (630) 252-9011
Materials Science Division   Fax: +1 (630) 252-7777
Argonne National Laboratory  E-mail: ROsborn at anl.gov
Argonne, IL 60439-4845