area detector data?

[Brian Tieman]

Ray,

Sorry for the late reply--I've been on vacation.

While reading this, please keep in mind that my view point was always to fit
what we could into the NeXus spec as it was already defined and then to lobby
for the rest of the needed info as needed.  This has, however, become such a
long drawn out battle in this group that I have thrown up my hands in the hopes
of getting some actual work done.  The library I have developed in no way
prevents users from writing NeXus standard files--but it also allows them to do
what they want without constantly pestering me for changes.

Ray Osborn wrote:

> on 99/11/16 3:37 PM, Brian Tieman at tieman at aps.anl.gov wrote:
>
> > Origionally the design goal was to be 100% NeXus compatible--however it
> > soon became apparent the the various pieces of metadata required would not
> > fit into the current NeXus defined groups.  This coupled with the fact
> > that the acquisition system is actually being used for numerous types of
> > experiments (microscopy, tomography, holography, diffraction,
> > etc...)--each with a differing set of metadata--led us to the development
> > of a rather unique and highly useful (in our case anyway) library of code.
> >
>
> I'm concerned about this statement for a number of reasons.  The original
> NeXus team attempted to define the rules for creating NeXus files to be as
> flexible as possible, without losing all the advantages of having a
> standard.  In particular, NeXus is able to cope with an enormous amount of
> metadata in addition to the detector counts themselves.  I find it difficult
> to believe that we can't accommodate the extra information you require in
> the existing skeletal structure.
>
> If it's necessary to define other group classes, beyond what is currently on
> the web page, we are certainly open to proposals.  In fact, the NeXus
> glossary and instrument definitions are the least-developed aspects of the
> format, and we have always intended to add to them as we got more input from
> the community.  Perhaps we should have made that clearer on the web pages.
>
> I would be interested to learn specifically what couldn't be stored in the
> format.  I would prefer that we augment the standard to overcome these
> limitations rather than fragment the format with local extensions.

Here is a sample of a couple of the groups the CMT guys have defined.  At one
point I asked them specifically to try and get the NeXus group to take a look at
what they wanted to do and see if things could reasonably be incorperated.  My
understanding is that someone from the NeXus group was contacted (I thought it
was you, even, although I'm really not sure.) about these additions, and that
they were turned down.  This is all heresay, however--maybe they never talked to
you guys at all.

Anyway, here's a couple of sample groups:

data_attribute--group
    black_field--the name of this group and what is in it differ based on the
type of data stored in this file
        name--field containing file name
        description--field containing type of data--matches group name in all
cases I know about
        data_file_index--index number for this file
        data_type--integer expressing data type--interestingly enough this
matched NX_INT16, etc...
        data_dimensions--number of dimensions
        n_i_pixels--x axis size
        n_j_pixels--y_axis size
        integration time--length of integration time
    NeXus_API_version--version of napi used
    experiment_file_name--name of HDF file containing groups common to all
images

data_array--group
    image_data--the actual 2D data

I don't have ready access to the definition of some of the other groups, but
suffice it to say that the document describing them is 100 pages long.  This
looks so much unlike what I think NeXus is about that I refuse to call the files
NeXus files.  They're really HDF files with a specific format for Computed
Micro-Tomography.

It took a long for this spec to be developed and I belive it would take a long
time to reconcile it with NeXus and the NeXus concept.  Therefor, I punted.  I
came up with a library which can be used to save these files as well as more
NeXus like files just be specifying an appropriate configuration file.  This
took a lot of development time/effort.  I'd much rather have used NXDict which I
think does a lot of what my library does, but since I don't control the CMT
spec, that option was unavailiable to me.

I have had more success in making other types of data more NeXus like--however,
most people I work for/with don't have a good grasp of what NeXus is/does (I'm
not even sure my grasp is as good as it should be).  So, I'm content to write
HDF files--for the moment at least...


> > We currently don't do any sort of file compression.  A typical image for
> > us is 1024 pixel X 1024 pixels X 16 bits or ~ 2MB.  I haven't looked at
> > NeXus 1.2.1 yet, but I don't see where NeXus supports compression.  I
> > intend to do this, but I need to base everything off pure HDF first.  Once
> > there, I will add file compression as an option.
> >
>
> HDF 4.1r3 allows for internal data compression of datasets using a variety
> of algorithms.  We have not yet implemented any of them because it had not
> yet got to the top of the priority list.  If this is critical for any
> particular user, we can see if it can be moved up.  I don't think it's that
> difficult, unless the performance penalty is significant.
>

I'd really like to see this done.  Most of our images are 1024x1024x2bytes or ~
2MB.  A complete data set may contain close to 1000 of these images.  Plus, the
hope is to go to 2048x2048x2bytes cameras within the next year or so.  That's an
awful lot of data.  As long as the file compression doesn't bottle neck the file
saving too much (we can currently save and image at ~1HZ) it's an option I'd
like to make use of.

>
> Ray Osborn
>
> P.S. I've tried to configure the mailing list to direct replies back to the
> list by default, but it hasn't been working.  Until I can fix it, please be
> aware of the problem, and manually redirect the replies.   I think that some
> of the correspondence following this posting never made it to the list, even
> though the writers may have wanted it to.
>
> --
> Dr Ray Osborn                Tel: +1 (630) 252-9011
> Materials Science Division   Fax: +1 (630) 252-7777
> Argonne National Laboratory  E-mail: ROsborn at anl.gov
> Argonne, IL 60439-4845
>
>