Storing repeating structures

[Mark.Koennecke]

On Tue, 2 Nov 1999, Ray Osborn wrote:

> on 99/11/2 1:27 AM, Mark.Koennecke at koenneck at psi.ch wrote:
> 
> > 
> > Options available where to have F77 style arrays for each single  value
> > to be stored or to have a vGroup per frame.
> > 
> > I choose the vGroup option for two reasons: Each frame represents a
> > single measurement. The structure of NeXus files required some data
> > to be stored in different vGroups: i.e PSD data in NXdata, NXinstrument,
> > NXdetector, sample data in NXsample etc. The vGroup option allowed to
> > preserve this structure. I also feel that the addition of an artifical
> > dimension (frame number) is a bit dodgy especially if we consider
> > automatic data analysis.
> > 
> 
> If I understand this correctly, you are storing a multi-dimensional data
> array in each frame, and then changing some variable, such as temperature or
> detector angle, from frame to frame.  If so, I would have thought that it
> would be better to add the frame number as an extra dimension, and included
> the temperature or angle as independent one-dimensional arrays in a single
> NXdata group.  You are, of course, allowed one unlimited dimension in each
> SDS, so you would not have to define the frame number in advance.  This
> would make the subsequent extraction of the data for plotting a lot easier.
> Each frame could be extracted as a slab, or you could take a PSD element and
> plot it versus temperature.  With your design, don't you have to reconstruct
> this information by a lot more i/o.
> 
  Ray,

  you are right about the additional I/O overhead. But an additional
  requirement was a facility to mark a frame as invalid to the analysis
  software. For instance if the kryostat turned out to be a heater. This
  means that I had to process each frame individually anyway. Then there 
  are two ways of viewing the data: as 2D frames or as a 3D volume. 

  Another argument is common usage: The expected operation of the
  instrument is like a protein crystallography rotation camera. And these
  people store each frame in a separate file! And the processing software
  works frame by frame as well. (In order to reduce memory requirements)
  So I thought a frame can at least have a entry group of its own.

  In the previous section I mentioned "expected operation": The instrument
  scientist wants to operate the instrument like this. But doubts have
  been raised by people who have to be taken serious that he has not
  enough intensity to do that. Then another option would be to center the 
  PSD on a reflection and just use it for integration. This would make
  each entry completely separate.

  And I still feel bad about additional unphysical dimensions which just
  represent frame numbers.....


                             Mark