NeXus and 2-D data

Thu Aug 13 18:42:12 BST 1998

re...

> >Most of our two-dimensional scans are actually sequences of
>one-dimensional scans,
> >and each one-dimensional scan has its own X axis (X is the independent
>variable).
> >...

> It's taken me a little while to understand the details of this problem.  Am
> I right in thinking that the scenario is for the instrument to set a
> particular Y-value, scan in X, and then set a new Y-value etc?

Yes.  Our multidimensional scan software consists of identical modules
each of which handles one dimension.  The module that implements the
innermost "loop" triggers data-acquisition hardware at each data point;
modules implementing outer loops trigger inner-loop scans, which behave
just like data acquisition hardware.  (Outer-loop scan modules can also
trigger their own data-acquisition hardware in addition to triggering
inner-loop scans).  This allows us easily to implement scans of any
dimension.  (Users have performed a few 4-D scans, but most do 1-D and
2-D.  Microtomography beamlines routinely do sequences of 2-D scans).
The modular arrangement also allows us to use an inner-loop scan module
to implement odd measurements, such as A-B-B-A sequences and cumulative
sweep scans.

The good thing about this system is that the same software can do all
kinds of scans.  The bad thing about doing all kinds of scans is that
the data can get really complex.  I was hoping the fact that NeXus is
based on a hierarchical format would bail us out, because our data is
naturally structured in this way.  However, it looks like the current
version of NeXus restricts users to a one-level hierarchy.

> If so, that
> would give just one 1-d array containing the Y-axis settings.  This would
> then be given the attribute axis=2 and would present no further problems.
> It seems to me that there are two possibilities for storing the X-axis
> settings.
>
> 1) Construct a default 1-d array containing the "nominal" X-values and give
> this the axis=1 attribute.  The actual X-values could be stored in a
> 2d-array as subsidiary data (given "signal=2" or higher ) and would be
> ignored by standard plotting packages.
>
> 2) Construct a set of 1-d arrays containing the actual X-values, each with
> axis=1, but with the "primary" attribute set to 1, 2, etc.  Standard
> plotting packages would choose the axis with "primary=1".

I think we're going to work from option 1.  I'm not sure what data
structure you're thinking about for the signals.  Option 1 seems pretty
understandable with either 1-D or 2-D signal data.  Option 2 would have
problems: with signals in 1-D arrays, standard tools would normally
plot signals against the wrong X axis, and a typical 2-D scan would
require around 3000 NXdata's, which would cause performance problems;
with signals in 2-D arrays, tools would have to match a particular 1-D
axis with a particular row of an array, and there aren't any free attributes
to direct this matching.

> I'm not sure how to deal with the case where there is a different number of
> X-values in each scan.  One of the rules of NXdata groups is that the data
> themselves are stored as a multi-dimensional data set with defined sizes for
> each dimension.  I guess that the writing program could work out the maximum
> number of X-values, make that the dimension size, and fill the missing
> values with the standard HDF fill value (which can be set to anything the
> user wants).  This requires that the NeXus file is written after all the
> data have been taken, which I know is not always possible.

We can't write directly to NeXus files anyway (can't keep up with the data, and
doesn't run under VxWorks), so this solution would work for us. The only
solution
I can think of that handles this problem cleanly is to regard all scans as
(sequences of) 1-D scans, but this solution has the problems described above for
large 2-D scans.

---

I didn't want to cloud the issues earlier, but we also have the problem
that signals are not necessarily scalar quantities.  For example, x-ray
microscope users want to acquire a multichannel-analyzer spectrum at
each pixel of a 2-D scan, in addition to the normal scalar signals.
(One guy was talking about acquiring a CCD image at each pixel, for
microcrystallography, but he stopped talking about it right after I shoved
a rock in his mouth, so I guess he didn't really want it all that badly.)

The difficulty with mixed-dimensional data is that axis 1 (2) really should
be associated with the added dimension(s).  For a 1-D scan involving scalars,
the positioner naturally is axis 1.  For MCA spectra, the natural axis 1 is
the channel number, and the positioner should be axis 2.  But since an axis
can't be axis 1 for some signals, and axis 2 for others, we have to reformat
the data so that standard tools will display it correctly.  This means you
can't write even 1-D scan data as they come in.

---

I want to make sure all my whining doesn't leave the impression that I'm
trying to back away from NeXus; I would do that silently.  NeXus is clearly
the right thing to do.  I've been waiting years for something this good, and
we are going to make it work for our data.  What we're talking about is how
best to make it work.  To me, root issues are whether/how to give NeXus users
some access to HDF's support for hierarchy, and whether/how signals should
participate in identifying axes.

Tim Mooney
Beamline Controls & Data Acquisition
Advanced Photon Source
Argonne National Laboratory