[NeXus-committee] DECTRIS example summary

Mon Jan 14 21:23:19 GMT 2013

Hi Pete

On Mon, 2013-01-14 at 14:59 -0600, Pete Jemian wrote: 
> Armando's point is right (or on the right track, at least).
> 
> DECTRIS (and others) need to shovel large batches of data (image frames) 
> into HDF5 files as fast as possible.  For them, this means a very 
> shallow structure - no subgroups - just data in the root directory.  For 
> multiple access reasons (pointed out earlier today), each new batch goes 
> into a new file.

I do not see why subgroups shall have any influence on the performance.
In my approach you have 3 groups per batch-file where the detector group
holds the 3d-block with images. But it is just a suggestion. 

> 
> A master NeXus file needs to identify all the files and parts so they 
> can be reassembled into a sensible and complete set of data.  Since, in 
> the DECTRIS plan, they are representing slices of multi-dimensional data 
> in different HDF5 datasets, the resulting NeXus master file needs to be 
> able to describe the final structure and reference the locations of all 
> the parts.  The parts can be found using external links.  We don't have 
> a way to describe how the parts fit into the multi-dimensional data. 
> That's needed.

I can remember that Halil Pasic, a PHD student at KIT (Karslruher
Institut fuer Technologie), discussed this problem with me last year. He
has a framework for doing this. I guess that his draft still needs some
work but to my opinion he has the solution for this problem. 
I can contact him tomorrow and discuss this issue with him. 

> 
> I'm not sure we decided (maybe because I was not in the room at the 
> time) to move the signal attribute from the dataset to the parent NXdata 
> group.  Even if we did, for sure, the manual does not say that.  Yet. 
> I'm convinced NeXus should do this.  The DECTRIS data shows why we would 
> be so motivated.
> 
> Pete

Eugen

> 
> On 1/14/2013 2:44 PM, V. Armando Sole wrote:
> > Hi Eugene,
> >
> > That still does not make transparent the link mechanism (different
> > paths) and does not solve the problem that in fact we are segmenting
> > what in other case would be a single dataset as Ben pointed out.
> >
> > Following that approach, the main thing to be made to the supplied file is:
> >
> > - Make sure all the links are in the same NXdata group in the master file.
> > - Add a *new* attribute to that NXdata group in order to let us know the
> > "signal" inside is build up of a set of links.
> >
> > Anybody is free to come with other suggestion. This one has the
> > advantage that DECTRIS has almost nothing to do (make sure the links
> > -not the targets- are inside an NXdata group) and everything can be
> > solved at the master file level (adding attributes to the NXdata group
> > in line with the new approach of defining the plots).
> >
> > Armando
> >
> > On 14.01.2013 21:24, Wintersberger, Eugen wrote:
> >> Hi all
> >>   Ok - if I got the entire discussion right the problem is to access
> >> metadata for the different blocks as links are entirely transparent in
> >> HDF5. In order to increase confusion let me make another proposal
> >> (though I am not sure if someone has already made a suggestion like this
> >> - I somehow lost track)
> >>
> >> master file:
> >> -----------
> >> entry:NXentry/instrument:NXinstrument/detector_1
> >> ....
> >> entry:NXentry/instrument:NXinstrument/detector_n
> >> entry:NXentry/data:NXdata/data_1 (link to data in detector_1)
> >> ...
> >> entry:NXentry/data:NXdata/data_n (link to data in detector_n)
> >>
> >> detector_i is a link to the detector group in the file for the i-th
> >> image block
> >>
> >> image block file:
> >> ----------------
> >> entry:NXentry/instrument:NXinstrument/detector:NXdetector/data
> >> first_image_nr
> >> last_image_nr
> >> entry:NXentry/data:NXdata/data (link to data in detector group)
> >>
> >> To my opinion this approach has several advantages:
> >> 1.) because the detector group is linked to the master file all the
> >>     detector related metadata can be accessed easily
> >> 2.) as the entire detector group is available the different image
> >>     blocks can be of different pixels size, geometry, or what ever
> >> 3.) I have it when detector data is not stored in an NXdetector group
> >>     along with the metadata.
> >>
> >> best regards
> >>   Eugen
> >>
> >> On Mon, 2013-01-14 at 14:38 +0100, Mark Koennecke wrote:
> >>> Hi,
> >>>
> >>> from your feedback and my own study I find that there are the following
> >>> problems
> >>> with the example file as is:
> >>>
> >>> - Structure is wrong regarding linked data sets. Should be either:
> >>>
> >>>    * entry:NXentry/data_00001:NXdata/data (Preferred ....)
> >>>    * entry:NXData/data:NXdata/data_0001
> >>>
> >>> - On data, signal=1 attribute is missing
> >>> - On external link, NeXus attribute
> >>> NAPIMOUNT=nxfile://th02c_ps02_1_data_000001.h5#data
> >>>    missing
> >>> - In detector, flatfield, efficiency data is missing. This is not
> >>> critical,
> >>>    I just wonder.
> >>>
> >>> Am I right? Is this what I tell DECTRIS tomorrow? Or do I miss
> >>> something?
> >>>
> >>> Also, what is the opinion, do we need another Telco on this one?
> >>>
> >>> Regards,
> >>>
> >>>               Mark
> >>>
> >>>
> >>> _______________________________________________
> >>> NeXus-committee mailing list
> >>> NeXus-committee at nexusformat.org
> >>> http://lists.nexusformat.org/mailman/listinfo/nexus-committee
> >
> > _______________________________________________
> > NeXus-committee mailing list
> > NeXus-committee at nexusformat.org
> > http://lists.nexusformat.org/mailman/listinfo/nexus-committee
> _______________________________________________
> NeXus-committee mailing list
> NeXus-committee at nexusformat.org
> http://lists.nexusformat.org/mailman/listinfo/nexus-committee

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 230 bytes
Desc: This is a digitally signed message part
URL: <http://lists.nexusformat.org/pipermail/nexus-committee/attachments/20130114/fa90a768/attachment.sig>