[NeXus-committee] NXvalidate

[Pete Jemian]

It is not reasonable to imagine in the case of the Eiger detector that 
every file MUST be validated.  If real-time validation is desired, as in 
the factory floor, where product is being produced at a high rate, 
random sampling will (eventually) detect the presence of a non-compliant 
data set.

The heart of the validation code should be in C or C++ to allow its 
inclusion into libraries for use in just about any language and host 
architecture.

Pete

On 10/8/2015 9:49 AM, Herbert J. Bernstein wrote:
> Dear Mark,
>
>    To help give a sense of the validation performance requirement -- for
> the Eiger, the data rates are likely to see at NSLS-II for 16M images
> will be based on either 133 frames per second for full 16M images or 750
> frames per second for 4M ROIs.  For many applications we are more likely
> to have to work in streaming mode, rather than waiting for full data
> sets that will have less demanding processing timing requirements, so we
> well may be looking at between 1 and 50 frames per HDF5 file.  If we are
> lucky we will be able to make the 50 frame/file version work, but even
> that would mean needing to validate between 3 and 25 NeXus files per
> second.  If we are unlucky, we might need to validate between 133 and
> 750 files per second.
>
>    There are much worse cases likely to come up in the future, so, if
> the validation is to be used as flexibly as possible in as many cases as
> possible, I would suggest going for performance, and having the heart of
> the code in C or C++.  We should of course follow the good examples for
> dials, cctbx, and pymol of allowing the high performance C or C++ code
> to be hidden from most users behind a python wrapper, but when it is
> necessary to interface directly to the C or C++ code, that should be an
> available option.
>
>    We're going to be starting by trying to drink water from a firehose.
> In the end I think many of us are going to be trying to drink water from
> Niagra Falls.
>
>    Regards,
>      Herbert
>
>
>
> On Thu, Oct 8, 2015 at 10:09 AM, Koennecke Mark (PSI)
> <mark.koennecke at psi.ch <mailto:mark.koennecke at psi.ch>> wrote:
>
>     Hi,
>
>     I have written a little white paper about how I would reimplement
>     NXvalidate. It details what needs to be
>     done and a little how. There are a number of TODO items in the text.
>     Please have a look. May be we
>     can sort some of this out during one of our next Telcos. Then I
>     might start hacking away at this.
>     Otherwise this may Code Camp material. The txt is the markdown
>     original if someone feels like
>     editing the text.
>
>
>     Best Regards,
>
>           Mark Koennecke
>
>
>
>     _______________________________________________
>     NeXus-committee mailing list
>     NeXus-committee at nexusformat.org <mailto:NeXus-committee at nexusformat.org>
>     http://lists.nexusformat.org/mailman/listinfo/nexus-committee
>
>
>
>
> _______________________________________________
> NeXus-committee mailing list
> NeXus-committee at nexusformat.org
> http://lists.nexusformat.org/mailman/listinfo/nexus-committee
>