[NeXus-committee] NXvalidate

[Herbert J. Bernstein]

Dear Mark,

  To help give a sense of the validation performance requirement -- for the
Eiger, the data rates are likely to see at NSLS-II for 16M images will be
based on either 133 frames per second for full 16M images or 750 frames per
second for 4M ROIs.  For many applications we are more likely to have to
work in streaming mode, rather than waiting for full data sets that will
have less demanding processing timing requirements, so we well may be
looking at between 1 and 50 frames per HDF5 file.  If we are lucky we will
be able to make the 50 frame/file version work, but even that would mean
needing to validate between 3 and 25 NeXus files per second.  If we are
unlucky, we might need to validate between 133 and 750 files per second.

  There are much worse cases likely to come up in the future, so, if the
validation is to be used as flexibly as possible in as many cases as
possible, I would suggest going for performance, and having the heart of
the code in C or C++.  We should of course follow the good examples for
dials, cctbx, and pymol of allowing the high performance C or C++ code to
be hidden from most users behind a python wrapper, but when it is necessary
to interface directly to the C or C++ code, that should be an available
option.

  We're going to be starting by trying to drink water from a firehose.  In
the end I think many of us are going to be trying to drink water from
Niagra Falls.

  Regards,
    Herbert



On Thu, Oct 8, 2015 at 10:09 AM, Koennecke Mark (PSI) <mark.koennecke at psi.ch
> wrote:

> Hi,
>
> I have written a little white paper about how I would reimplement
> NXvalidate. It details what needs to be
> done and a little how. There are a number of TODO items in the text.
> Please have a look. May be we
> can sort some of this out during one of our next Telcos. Then I might
> start hacking away at this.
> Otherwise this may Code Camp material. The txt is the markdown original if
> someone feels like
> editing the text.
>
>
> Best Regards,
>
>      Mark Koennecke
>
>
>
> _______________________________________________
> NeXus-committee mailing list
> NeXus-committee at nexusformat.org
> http://lists.nexusformat.org/mailman/listinfo/nexus-committee
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.nexusformat.org/pipermail/nexus-committee/attachments/20151008/a9e94a46/attachment.html>