<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:Calibri;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body bgcolor="white" lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;mso-fareast-language:EN-US">Hi,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;mso-fareast-language:EN-US">Yes I am on this thread – I was just lurking!
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;mso-fareast-language:EN-US">So as I understand it, we have a different descriptor (i.e. sort of schema) for a stream of events (for Pete’s example file this is called ‘primary’). If we
were to have another data source recording a some other rate then we would just have another stream of events with it’s own descriptor.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;mso-fareast-language:EN-US">The solution we currently have for interleaving separate streams of data (based on time) is called DataMuxer (<a href="https://nsls-ii.github.io/datamuxer/index.html)">https://nsls-ii.github.io/datamuxer/index.html)</a>
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;mso-fareast-language:EN-US">Just for info, Dan Allan and I will be visiting Dan Flath at LCLS early next month to talk about Bluesky.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;mso-fareast-language:EN-US">Cheers<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;mso-fareast-language:EN-US">Stu<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Calibri;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-family:Calibri;color:black">From: </span>
</b><span style="font-family:Calibri;color:black">NeXus <nexus-bounces@nexusformat.org> on behalf of Pete Jemian <prjemian@gmail.com><br>
<b>Reply-To: </b>Discussion forum for the NeXus data format <nexus@nexusformat.org><br>
<b>Date: </b>Friday, March 10, 2017 at 14:13<br>
<b>To: </b>NeXus <nexus@nexusformat.org><br>
<b>Subject: </b>Re: [Nexus] Fwd: event data in NeXus data file<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I'll pass this to the team at BNL. Perhaps Stuart Campbell is on this thread already.
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The BlueSky software *should* be handling the aspects of event alignment as you described. This should be confirmed.<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Mar 10, 2017 12:59 PM, "Aaron Brewster" <<a href="mailto:asbrewster@lbl.gov">asbrewster@lbl.gov</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal">Hi Pete, this is quite useful, thanks. The thing that is missing is event alignment. For example, you have K detectors and Z timestamps. Each detector recorded a subset of timestamps <= Z that doesn't necessarily align between other
detectors. For example, there will likely be detectors and instruments recording at different speeds during an experiment, with the potential that each detector or instrument could drop events at random.
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">If the user wants to get events from multiple detectors in this usecase, they have to manually search each NXLog entry, which has a binary search time complexity, assuming the timestamps are ordered (which they might not be, in which case
it has a linear search time complexity). LCLS II will be writing something 25 gigabytes per second, so linear searches like this aren't feasible. We need fast lookups between datasets.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">David Schneider (who should be on this email list now) will be joining us on the next Telco. He and I have been talking through solutions to the event alingment problem and we'll go over them then. Briefly, we are considering an NXLog
master table with indices between all the datasets, or K*(K-1) NKLog tables that serve as pairwise indices between tables. The former suffers from sparseness if the readouts are at orders of magnitude different speeds, which they might be, and the latter
suffers from too many tables.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">-Aaron<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Fri, Mar 10, 2017 at 10:39 AM, Pete Jemian <<a href="mailto:jemian@anl.gov" target="_blank">jemian@anl.gov</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal"><br>
including the NeXus discussion forum <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
<br>
-------- Forwarded Message --------<br>
Subject: event data in NeXus data file<br>
Date: Fri, 10 Mar 2017 12:31:48 -0600<br>
From: Pete Jemian <<a href="mailto:jemian@anl.gov" target="_blank">jemian@anl.gov</a>><br>
To: Aaron Brewster <<a href="mailto:asbrewster@lbl.gov" target="_blank">asbrewster@lbl.gov</a>><br>
<br>
<br>
Aaron:<br>
<br>
At the NeXus telco this week, you brought up the question of how to store event data. As I recall, the suggestion was to use NXlog.<br>
<br>
I'm working on adding a NeXus file writer (<a href="https://github.com/BCDA-APS/suitcase/blob/NeXus-dev/suitcase/nexus.py" target="_blank">https://github.com/BCDA-APS/suitcase/blob/NeXus-dev/suitcase/nexus.py</a>) to the NSLS-II BlueSky software suite (<a href="http://nsls-ii.github.io/" target="_blank">http://nsls-ii.github.io/</a>).
That software records data as events, depositing each as a time-stamped record into a database. Later, a tool can extract a stream of events into an output of their choosing. The BlueSky software will take charge of binning those data in time and providing
suitable arrays.<br>
<br>
Attached is an example file of useless data from the unit test suite that shows the structure of such event data as a NeXus data file. It plots in PyMCA and NeXpy and validates against NeXus definitions release 3.2 with punx.<br>
<br>
There is no use of ragged arrays in this file from the unit test suite.<br>
<br>
Is this file structured along the lines of the structure that you imagine?<br>
<br>
Pete<br>
<br>
-- <br>
----------------------------------------------------------<br>
Pete R. Jemian, Ph.D. <<a href="mailto:jemian@anl.gov" target="_blank">jemian@anl.gov</a>><br>
Beam line Controls and Data Acquisition, Group Leader<br>
Advanced Photon Source, Argonne National Laboratory<br>
Argonne, IL 60439 <a href="tel:630%20-%20252%20-%203189" target="_blank">630 - 252 - 3189</a><br>
-----------------------------------------------------------<br>
Education is the one thing for which people<br>
are willing to pay yet not receive.<br>
-----------------------------------------------------------<br>
<br>
<br>
<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">_______________________________________________<br>
NeXus mailing list<br>
<a href="mailto:NeXus@nexusformat.org" target="_blank">NeXus@nexusformat.org</a><br>
<a href="http://lists.nexusformat.org/mailman/listinfo/nexus" target="_blank">http://lists.nexusformat.org/mailman/listinfo/nexus</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
NeXus mailing list<br>
<a href="mailto:NeXus@nexusformat.org">NeXus@nexusformat.org</a><br>
<a href="http://lists.nexusformat.org/mailman/listinfo/nexus" target="_blank">http://lists.nexusformat.org/mailman/listinfo/nexus</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>