[NeXus-committee] DECTRIS example summary

[Watts Benjamin]

Looking at the data files more closely, there are included the
attributes "image_nr_low" and "image_nr_high". These were presumably
used to describe the position of the individual files in the overall
array of data.

Cheers,
Ben

-----Original Message-----
From: nexus-committee-bounces at nexusformat.org
[mailto:nexus-committee-bounces at nexusformat.org] On Behalf Of Watts
Benjamin
Sent: Montag, 14. Januar 2013 16:25
To: nexus-committee at nexusformat.org
Subject: Re: [NeXus-committee] DECTRIS example summary

Perhaps you are correct, Pete, and my view of the data being a single
array is something I've picked from various discussions that the rest of
the NeXus committee might be blind to. To share what I think is the
situation (Mark might correct me here): 

The detector takes a rapid stream of images, so the data is of the form
x_pixels * y_pixels * frames. The data then is broken up into 100 frame
chunks in order to stop individual files from being too big.

Therefore, my assumption is that the data is one big time-series of
images that has been broken up into pieces for storage, but will need to
be reassembled for analysis.

Cheers,
Ben

-----Original Message-----
From: nexus-committee-bounces at nexusformat.org
[mailto:nexus-committee-bounces at nexusformat.org] On Behalf Of Pete
Jemian
Sent: Montag, 14. Januar 2013 16:12
To: nexus-committee at nexusformat.org
Subject: Re: [NeXus-committee] DECTRIS example summary

Wow!  That is a new view and it confuses me incredibly.  There is no
metadata or other indication that one should recombine the data
presented this way.

Is there any NeXus mechanism to join together into one dataset the
content of several linked arrays?  Can someone sketch up an example? 
Who is prepared to receive such a dataset and reassemble it?

Your option 1 is the only way I can see now, yet I do not understand how
to concatenate the arrays.

Perhaps, could this be written as a multi-file chunked dataset?

Pete

On 1/14/2013 8:51 AM, Watts Benjamin wrote:
> I think that everyone is looking at the Dectris data the wrong way. It

> is a single data array and is only broken up into separate files 
> because of performance issues. Therefore, ideally, all the data shoud 
> be written into a single NXdata group. Since we are forced to receive 
> it in separate files, linking is an obvious solution. However, links 
> in NeXus have always been discrete - mapping one location to one array

> in another location. Our situation here is different, with the one 
> array being separated into a number of pieces in different locations.
> The master file needs some way of saying "my data is there, and then 
> continues there, and there and there...", which does not yet exist (to

> the best of my knowledge). Options for doing this include:
>
> 1. The NXdata group of the master file contains a list of external 
> links, with the order of the list describing the order in which the 
> linked arrays should be concatenated.
> 2. The master NXdata group links to the first part of the array and 
> then the file containing that first part has some record pointing to 
> where the data is continued.
> 3. Both could be done together.
>
> Option #1 seems best to me.
>
> Cheers,
> Ben

--
----------------------------------------------------------
Pete R. Jemian, Ph.D.                <jemian at anl.gov>
Beam line Controls and Data Acquisition, Group Leader
Advanced Photon Source,   Argonne National Laboratory
Argonne, IL  60439                   630 - 252 - 3189
-----------------------------------------------------------
    Education is the one thing for which people
       are willing to pay yet not receive.
-----------------------------------------------------------


_______________________________________________
NeXus-committee mailing list
NeXus-committee at nexusformat.org
http://lists.nexusformat.org/mailman/listinfo/nexus-committee

_______________________________________________
NeXus-committee mailing list
NeXus-committee at nexusformat.org
http://lists.nexusformat.org/mailman/listinfo/nexus-committee