[Nexus] Fwd: event data in NeXus data file

[Aaron Brewster]

Thanks, very interesting.  I presume under the hood DataMuxer is doing some
searching to align the events, which is certainly helpful for users.  If
the data at LCLS is chunked finely enough, searches may be fine.

Thanks,
-Aaron

On Fri, Mar 10, 2017 at 11:49 AM, Campbell, Stuart <scampbell at bnl.gov>
wrote:

> Hi,
>
>
>
> Yes I am on this thread – I was just lurking!
>
>
>
> So as I understand it, we have a different descriptor (i.e. sort of
> schema) for a stream of events (for Pete’s example file this is called
> ‘primary’).  If we were to have another data source recording a some other
> rate then we would just have another stream of events with it’s own
> descriptor.
>
>
>
> The solution we currently have for interleaving separate streams of data
> (based on time) is called DataMuxer (https://nsls-ii.github.io/
> datamuxer/index.html)
>
>
>
> Just for info, Dan Allan and I will be visiting Dan Flath at LCLS early
> next month to talk about Bluesky.
>
>
>
> Cheers
>
> Stu
>
>
>
> *From: *NeXus <nexus-bounces at nexusformat.org> on behalf of Pete Jemian <
> prjemian at gmail.com>
> *Reply-To: *Discussion forum for the NeXus data format <
> nexus at nexusformat.org>
> *Date: *Friday, March 10, 2017 at 14:13
> *To: *NeXus <nexus at nexusformat.org>
> *Subject: *Re: [Nexus] Fwd: event data in NeXus data file
>
>
>
> I'll pass this to the team at BNL.  Perhaps Stuart Campbell is on this
> thread already.
>
>
>
> The BlueSky software *should* be handling the aspects of event alignment
> as you described.  This should be confirmed.
>
>
>
> On Mar 10, 2017 12:59 PM, "Aaron Brewster" <asbrewster at lbl.gov> wrote:
>
> Hi Pete, this is quite useful, thanks.  The thing that is missing is event
> alignment.  For example, you have K detectors and Z timestamps.  Each
> detector recorded a subset of timestamps <= Z that doesn't necessarily
> align between other detectors.  For example, there will likely be detectors
> and instruments recording at different speeds during an experiment, with
> the potential that each detector or instrument could drop events at random.
>
>
>
> If the user wants to get events from multiple detectors in this usecase,
> they have to manually search each NXLog entry, which has a binary search
> time complexity, assuming the timestamps are ordered (which they might not
> be, in which case it has a linear search time complexity).  LCLS II will be
> writing something 25 gigabytes per second, so linear searches like this
> aren't feasible.  We need fast lookups between datasets.
>
>
>
> David Schneider (who should be on this email list now) will be joining us
> on the next Telco.  He and I have been talking through solutions to the
> event alingment problem and we'll go over them then.  Briefly, we are
> considering an NXLog master table with indices between all the datasets, or
> K*(K-1) NKLog tables that serve as pairwise indices between tables.  The
> former suffers from sparseness if the readouts are at orders of magnitude
> different speeds, which they might be, and the latter suffers from too many
> tables.
>
>
>
> Thanks,
>
> -Aaron
>
>
>
>
>
>
>
> On Fri, Mar 10, 2017 at 10:39 AM, Pete Jemian <jemian at anl.gov> wrote:
>
>
> including the NeXus discussion forum
>
>
>
> -------- Forwarded Message --------
> Subject: event data in NeXus data file
> Date: Fri, 10 Mar 2017 12:31:48 -0600
> From: Pete Jemian <jemian at anl.gov>
> To: Aaron Brewster <asbrewster at lbl.gov>
>
>
> Aaron:
>
> At the NeXus telco this week, you brought up the question of how to store
> event data.  As I recall, the suggestion was to use NXlog.
>
> I'm working on adding a NeXus file writer (https://github.com/BCDA-APS/
> suitcase/blob/NeXus-dev/suitcase/nexus.py) to the NSLS-II BlueSky
> software suite (http://nsls-ii.github.io/).  That software records data
> as events, depositing each as a time-stamped record into a database.
> Later, a tool can extract a stream of events into an output of their
> choosing.  The BlueSky software will take charge of binning those data in
> time and providing suitable arrays.
>
> Attached is an example file of useless data from the unit test suite that
> shows the structure of such event data as a NeXus data file.  It plots in
> PyMCA and NeXpy and validates against NeXus definitions release 3.2 with
> punx.
>
> There is no use of ragged arrays in this file from the unit test suite.
>
> Is this file structured along the lines of the structure that you imagine?
>
> Pete
>
> --
> ----------------------------------------------------------
> Pete R. Jemian, Ph.D.                <jemian at anl.gov>
> Beam line Controls and Data Acquisition, Group Leader
> Advanced Photon Source,   Argonne National Laboratory
> Argonne, IL  60439                   630 - 252 - 3189
> -----------------------------------------------------------
>     Education is the one thing for which people
>        are willing to pay yet not receive.
> -----------------------------------------------------------
>
>
>
>
> _______________________________________________
> NeXus mailing list
> NeXus at nexusformat.org
> http://lists.nexusformat.org/mailman/listinfo/nexus
>
>
>
>
> _______________________________________________
> NeXus mailing list
> NeXus at nexusformat.org
> http://lists.nexusformat.org/mailman/listinfo/nexus
>
>
>
> _______________________________________________
> NeXus mailing list
> NeXus at nexusformat.org
> http://lists.nexusformat.org/mailman/listinfo/nexus
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.nexusformat.org/pipermail/nexus/attachments/20170310/22dabba2/attachment.html>