area detector data?

[Brian Tieman]

Dave,

I have done quite a bit of work for using NeXus to save area detector
data--specifically CCD camera images of various size.  I currently have
two different acquisition programs which use the same library to save the
images.

Origionally the design goal was to be 100% NeXus compatible--however it
soon became apparent the the various pieces of metadata required would not
fit into the current NeXus defined groups.  This coupled with the fact
that the acquisition system is actually being used for numerous types of
experiments (microscopy, tomography, holography, diffraction,
etc...)--each with a differing set of metadata--led us to the development
of a rather unique and highly useful (in our case anyway) library of code.

The library is currently built on top of napi.c--although this is historic
and the plan is to remove this dependence and build directly on top of
HDF.  The library takes in a template file (ascii) which is parsed to
determine the structure of the resulting files as well as the locations of
the various pieces of metadata.  This gives us the flexibility to decide
at run time the exact structure of the file (which can be NeXus compatible
if the user so chooses).  Locations of metadata can currently be one of
three places.  1) a constant value stored in the template file itself.  2)
a 1 to 1 correspondance with a program's internal variables, or 3) an
EPICS process variable in a reachable IOC.

It is the acquisition program's job to publish some variables to our HDF
Library (so the users can refer to them in the ASCII template files) and
to decide when PVs should be updated and files written--and that's about
it.  The details of writing the file are handled by our Library.  This
provides us an acquisition program with very little knowledge about a file
system which can save just about any piece of data availiable to us in
locations specified by average users--all without rebuilding!

We currently use this llibrary in two of our NT based acquisition programs
to write the data.  We also have linked it into IDL as an alternitive to
IDL's questionable support for HDF (we'll see if Fortner helps RSI in this
area!).  The linking into IDL was done on both NT and Solaris.  The code
is developed under NT, but the port to Solaris was pretty trivial so i
don't expect too many problems under other OS.

Now the bad news...

We currently don't do any sort of file compression.  A typical image for
us is 1024 pixel X 1024 pixels X 16 bits or ~ 2MB.  I haven't looked at
NeXus 1.2.1 yet, but I don't see where NeXus supports compression.  I
intend to do this, but I need to base everything off pure HDF first.  Once
there, I will add file compression as an option.

We currently only write single image files.  This eventually needs to be
addressed so we can save multiple images in a single file.  When this
happens, we'll also need to add some of the group linking options such as
NeXus supports.

While the tamplate file is in ASCII and human readable and fairly
straightforward to understand, it's also pretty simplistic and I'm unhappy
with it.  One critical feature I'd like to support is the inclusion of one
file within another.  This would mean that I can have a file which defines
a group called USER to save all the information about a typical user
(email address, home institution, etc...)  If I can include that into
another file things become a little easier to maintain and understand.

I suppose there are other limitations as well, but the bottom line is that
we've been using some implementation of this library for about 10 months
now with great success.

Thanks for indulging my long winded discription.  If you're interested in
more, feel free to contact me trhough the email address below.  I do spend
about 10% of my time continuing to work on this system so improvements are
beaing made.

Brian Tieman
tieman at aps.anl.gov

Dave Love wrote:

> Has anyone done any work on using NeXus for area detector data
> (`images' up to a few k square, possibly compressed)?  Otherwise any
> advice on proceeding with such work would be welcome.
>
> The interest is in an alternative to imgCIF/CBF, initially for protein
> crystallography, but with the intention of covering more areas of
> interest here at Daresbury.  The major issue overall is probably
> defining the metadata, but that's a separate job, partly done already.
>
> Thanks for any info.
>
> --
> Save Our Synchrotron  <URL: http://www.diamond.freewire.co.uk/>