[NeXus-committee] DECTRIS example summary

[POIRIER Stephane]

Hi Eugen,

We (SOLEIL) are facing this kind of problematic for a few months.
We store huge matrixes (sets of images into 2 or 3 dimensions
acquisitions resulting into up to 5-d data items) without compression
(compression is automatically done 'off-line' by our storage facility)
into a single Nexus dataset. If the whole matrix is expected to be too
huge to fit into one single file (because of the 2TB limit), the data is
splitted into several nexus files

Acquisitions may run for several hours, but accessing at the data
(stored into a unique nexus dataset) during the acquisition works well.
We just added a progress indicator that contains the position of the
last completed acquisition step.

regards,

Stephane

On lun., 2013-01-14 at 20:56 +0100, Wintersberger, Eugen wrote:
> Hi all
> sorry for entering the discussion so late - I had some troubles with
> email here. 
> 
> A quick topposting comment on this: there is an additional reason for
> using separate files to keep image blocks. You can access the images in
> the first block already while the second block is still written. This
> makes it possible to start with evaluation while the experiment is still
> running. Otherwise the family driver would have done the job too. 
> 
> regards 
> Eugen
> 
> On Mon, 2013-01-14 at 16:21 +0100, "V. Armando Solé" wrote: 
> > Hi Pete,
> > 
> > If I remember well the discussions at PSI, DECTRIS wants to be able to 
> > dump the data as fast as possible.
> > 
> > If you consider you are issuing a command that involves the collection 
> > of thousands of images, one can "assume" a single NXentry describes the 
> > result and that a 3-D dataset could easilly do the job. However, DECTRIS 
> > wanted to be able to use the external storage approach for performance 
> > reasons *including* compression (something not available at the time of 
> > discussion). I thought they would use the standard HDF5 family driver 
> > but it seems the fact HDF5 is not taking profit of multiple threads to 
> > speed up things (compression) made them go towards other solution.
> > 
> > Armando
> > 
> > 
> > _______________________________________________
> > NeXus-committee mailing list
> > NeXus-committee at nexusformat.org
> > http://lists.nexusformat.org/mailman/listinfo/nexus-committee
>