[Nexus] Fwd: event data in NeXus data file

[Campbell, Stuart]

Hi,

Yes I am on this thread – I was just lurking!

So as I understand it, we have a different descriptor (i.e. sort of schema) for a stream of events (for Pete’s example file this is called ‘primary’).  If we were to have another data source recording a some other rate then we would just have another stream of events with it’s own descriptor.

The solution we currently have for interleaving separate streams of data (based on time) is called DataMuxer (https://nsls-ii.github.io/datamuxer/index.html)

Just for info, Dan Allan and I will be visiting Dan Flath at LCLS early next month to talk about Bluesky.

Cheers
Stu

From: NeXus <nexus-bounces at nexusformat.org> on behalf of Pete Jemian <prjemian at gmail.com>
Reply-To: Discussion forum for the NeXus data format <nexus at nexusformat.org>
Date: Friday, March 10, 2017 at 14:13
To: NeXus <nexus at nexusformat.org>
Subject: Re: [Nexus] Fwd: event data in NeXus data file

I'll pass this to the team at BNL.  Perhaps Stuart Campbell is on this thread already.

The BlueSky software *should* be handling the aspects of event alignment as you described.  This should be confirmed.

On Mar 10, 2017 12:59 PM, "Aaron Brewster" <asbrewster at lbl.gov<mailto:asbrewster at lbl.gov>> wrote:
Hi Pete, this is quite useful, thanks.  The thing that is missing is event alignment.  For example, you have K detectors and Z timestamps.  Each detector recorded a subset of timestamps <= Z that doesn't necessarily align between other detectors.  For example, there will likely be detectors and instruments recording at different speeds during an experiment, with the potential that each detector or instrument could drop events at random.

If the user wants to get events from multiple detectors in this usecase, they have to manually search each NXLog entry, which has a binary search time complexity, assuming the timestamps are ordered (which they might not be, in which case it has a linear search time complexity).  LCLS II will be writing something 25 gigabytes per second, so linear searches like this aren't feasible.  We need fast lookups between datasets.

David Schneider (who should be on this email list now) will be joining us on the next Telco.  He and I have been talking through solutions to the event alingment problem and we'll go over them then.  Briefly, we are considering an NXLog master table with indices between all the datasets, or K*(K-1) NKLog tables that serve as pairwise indices between tables.  The former suffers from sparseness if the readouts are at orders of magnitude different speeds, which they might be, and the latter suffers from too many tables.

Thanks,
-Aaron



On Fri, Mar 10, 2017 at 10:39 AM, Pete Jemian <jemian at anl.gov<mailto:jemian at anl.gov>> wrote:

including the NeXus discussion forum


-------- Forwarded Message --------
Subject: event data in NeXus data file
Date: Fri, 10 Mar 2017 12:31:48 -0600
From: Pete Jemian <jemian at anl.gov<mailto:jemian at anl.gov>>
To: Aaron Brewster <asbrewster at lbl.gov<mailto:asbrewster at lbl.gov>>


Aaron:

At the NeXus telco this week, you brought up the question of how to store event data.  As I recall, the suggestion was to use NXlog.

I'm working on adding a NeXus file writer (https://github.com/BCDA-APS/suitcase/blob/NeXus-dev/suitcase/nexus.py) to the NSLS-II BlueSky software suite (http://nsls-ii.github.io/).  That software records data as events, depositing each as a time-stamped record into a database.  Later, a tool can extract a stream of events into an output of their choosing.  The BlueSky software will take charge of binning those data in time and providing suitable arrays.

Attached is an example file of useless data from the unit test suite that shows the structure of such event data as a NeXus data file.  It plots in PyMCA and NeXpy and validates against NeXus definitions release 3.2 with punx.

There is no use of ragged arrays in this file from the unit test suite.

Is this file structured along the lines of the structure that you imagine?

Pete

--
----------------------------------------------------------
Pete R. Jemian, Ph.D.                <jemian at anl.gov<mailto:jemian at anl.gov>>
Beam line Controls and Data Acquisition, Group Leader
Advanced Photon Source,   Argonne National Laboratory
Argonne, IL  60439                   630 - 252 - 3189<tel:630%20-%20252%20-%203189>
-----------------------------------------------------------
    Education is the one thing for which people
       are willing to pay yet not receive.
-----------------------------------------------------------



_______________________________________________
NeXus mailing list
NeXus at nexusformat.org<mailto:NeXus at nexusformat.org>
http://lists.nexusformat.org/mailman/listinfo/nexus


_______________________________________________
NeXus mailing list
NeXus at nexusformat.org<mailto:NeXus at nexusformat.org>
http://lists.nexusformat.org/mailman/listinfo/nexus

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.nexusformat.org/pipermail/nexus/attachments/20170310/3cd1527e/attachment-0001.html>