<div dir="ltr">Hi Pete, this is quite useful, thanks. The thing that is missing is event alignment. For example, you have K detectors and Z timestamps. Each detector recorded a subset of timestamps <= Z that doesn't necessarily align between other detectors. For example, there will likely be detectors and instruments recording at different speeds during an experiment, with the potential that each detector or instrument could drop events at random.<div><br></div><div>If the user wants to get events from multiple detectors in this usecase, they have to manually search each NXLog entry, which has a binary search time complexity, assuming the timestamps are ordered (which they might not be, in which case it has a linear search time complexity). LCLS II will be writing something 25 gigabytes per second, so linear searches like this aren't feasible. We need fast lookups between datasets.</div><div><br></div><div>David Schneider (who should be on this email list now) will be joining us on the next Telco. He and I have been talking through solutions to the event alingment problem and we'll go over them then. Briefly, we are considering an NXLog master table with indices between all the datasets, or K*(K-1) NKLog tables that serve as pairwise indices between tables. The former suffers from sparseness if the readouts are at orders of magnitude different speeds, which they might be, and the latter suffers from too many tables.</div><div><br></div><div>Thanks,</div><div>-Aaron<br><div><br></div><div><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Mar 10, 2017 at 10:39 AM, Pete Jemian <span dir="ltr"><<a href="mailto:jemian@anl.gov" target="_blank">jemian@anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
including the NeXus discussion forum<div class="HOEnZb"><div class="h5"><br>
<br>
-------- Forwarded Message --------<br>
Subject: event data in NeXus data file<br>
Date: Fri, 10 Mar 2017 12:31:48 -0600<br>
From: Pete Jemian <<a href="mailto:jemian@anl.gov" target="_blank">jemian@anl.gov</a>><br>
To: Aaron Brewster <<a href="mailto:asbrewster@lbl.gov" target="_blank">asbrewster@lbl.gov</a>><br>
<br>
<br>
Aaron:<br>
<br>
At the NeXus telco this week, you brought up the question of how to store event data. As I recall, the suggestion was to use NXlog.<br>
<br>
I'm working on adding a NeXus file writer (<a href="https://github.com/BCDA-APS/suitcase/blob/NeXus-dev/suitcase/nexus.py" rel="noreferrer" target="_blank">https://github.com/BCDA-APS/s<wbr>uitcase/blob/NeXus-dev/suitcas<wbr>e/nexus.py</a>) to the NSLS-II BlueSky software suite (<a href="http://nsls-ii.github.io/" rel="noreferrer" target="_blank">http://nsls-ii.github.io/</a>). That software records data as events, depositing each as a time-stamped record into a database. Later, a tool can extract a stream of events into an output of their choosing. The BlueSky software will take charge of binning those data in time and providing suitable arrays.<br>
<br>
Attached is an example file of useless data from the unit test suite that shows the structure of such event data as a NeXus data file. It plots in PyMCA and NeXpy and validates against NeXus definitions release 3.2 with punx.<br>
<br>
There is no use of ragged arrays in this file from the unit test suite.<br>
<br>
Is this file structured along the lines of the structure that you imagine?<br>
<br>
Pete<br>
<br>
-- <br>
------------------------------<wbr>----------------------------<br>
Pete R. Jemian, Ph.D. <<a href="mailto:jemian@anl.gov" target="_blank">jemian@anl.gov</a>><br>
Beam line Controls and Data Acquisition, Group Leader<br>
Advanced Photon Source, Argonne National Laboratory<br>
Argonne, IL 60439 <a href="tel:630%20-%20252%20-%203189" value="+16302523189" target="_blank">630 - 252 - 3189</a><br>
------------------------------<wbr>-----------------------------<br>
Education is the one thing for which people<br>
are willing to pay yet not receive.<br>
------------------------------<wbr>-----------------------------<br>
<br>
<br>
<br>
</div></div><br>______________________________<wbr>_________________<br>
NeXus mailing list<br>
<a href="mailto:NeXus@nexusformat.org">NeXus@nexusformat.org</a><br>
<a href="http://lists.nexusformat.org/mailman/listinfo/nexus" rel="noreferrer" target="_blank">http://lists.nexusformat.org/<wbr>mailman/listinfo/nexus</a><br>
<br></blockquote></div><br></div>