[Nexus] Fwd: event data in NeXus data file

[Aaron Brewster]

Hi Pete, this is quite useful, thanks.  The thing that is missing is event
alignment.  For example, you have K detectors and Z timestamps.  Each
detector recorded a subset of timestamps <= Z that doesn't necessarily
align between other detectors.  For example, there will likely be detectors
and instruments recording at different speeds during an experiment, with
the potential that each detector or instrument could drop events at random.

If the user wants to get events from multiple detectors in this usecase,
they have to manually search each NXLog entry, which has a binary search
time complexity, assuming the timestamps are ordered (which they might not
be, in which case it has a linear search time complexity).  LCLS II will be
writing something 25 gigabytes per second, so linear searches like this
aren't feasible.  We need fast lookups between datasets.

David Schneider (who should be on this email list now) will be joining us
on the next Telco.  He and I have been talking through solutions to the
event alingment problem and we'll go over them then.  Briefly, we are
considering an NXLog master table with indices between all the datasets, or
K*(K-1) NKLog tables that serve as pairwise indices between tables.  The
former suffers from sparseness if the readouts are at orders of magnitude
different speeds, which they might be, and the latter suffers from too many
tables.

Thanks,
-Aaron



On Fri, Mar 10, 2017 at 10:39 AM, Pete Jemian <jemian at anl.gov> wrote:

>
> including the NeXus discussion forum
>
>
> -------- Forwarded Message --------
> Subject: event data in NeXus data file
> Date: Fri, 10 Mar 2017 12:31:48 -0600
> From: Pete Jemian <jemian at anl.gov>
> To: Aaron Brewster <asbrewster at lbl.gov>
>
>
> Aaron:
>
> At the NeXus telco this week, you brought up the question of how to store
> event data.  As I recall, the suggestion was to use NXlog.
>
> I'm working on adding a NeXus file writer (https://github.com/BCDA-APS/s
> uitcase/blob/NeXus-dev/suitcase/nexus.py) to the NSLS-II BlueSky software
> suite (http://nsls-ii.github.io/).  That software records data as events,
> depositing each as a time-stamped record into a database.  Later, a tool
> can extract a stream of events into an output of their choosing.  The
> BlueSky software will take charge of binning those data in time and
> providing suitable arrays.
>
> Attached is an example file of useless data from the unit test suite that
> shows the structure of such event data as a NeXus data file.  It plots in
> PyMCA and NeXpy and validates against NeXus definitions release 3.2 with
> punx.
>
> There is no use of ragged arrays in this file from the unit test suite.
>
> Is this file structured along the lines of the structure that you imagine?
>
> Pete
>
> --
> ----------------------------------------------------------
> Pete R. Jemian, Ph.D.                <jemian at anl.gov>
> Beam line Controls and Data Acquisition, Group Leader
> Advanced Photon Source,   Argonne National Laboratory
> Argonne, IL  60439                   630 - 252 - 3189
> -----------------------------------------------------------
>     Education is the one thing for which people
>        are willing to pay yet not receive.
> -----------------------------------------------------------
>
>
>
>
> _______________________________________________
> NeXus mailing list
> NeXus at nexusformat.org
> http://lists.nexusformat.org/mailman/listinfo/nexus
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.nexusformat.org/pipermail/nexus/attachments/20170310/2c1b14ea/attachment.html>