<div dir="ltr"><div><div><div>Dear Pete,<br><br></div> There is no case in which _every_ image must be validated, but if the performance of the validation is not somewhere close to the rate at which images are being collected, then we are likely to end up with _no_ images being validated. We are at the beginning of a major increase in speeds of collection, not the end. If we don't design for high performance now, it will be much, much harder to do later.<br><br></div> Regards,<br></div> Herbert<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Oct 8, 2015 at 10:54 AM, Pete Jemian <span dir="ltr"><<a href="mailto:prjemian@gmail.com" target="_blank">prjemian@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">It is not reasonable to imagine in the case of the Eiger detector that every file MUST be validated. If real-time validation is desired, as in the factory floor, where product is being produced at a high rate, random sampling will (eventually) detect the presence of a non-compliant data set.<br>
<br>
The heart of the validation code should be in C or C++ to allow its inclusion into libraries for use in just about any language and host architecture.<br>
<br>
Pete<div><div class="h5"><br>
<br>
On 10/8/2015 9:49 AM, Herbert J. Bernstein wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">
Dear Mark,<br>
<br>
To help give a sense of the validation performance requirement -- for<br>
the Eiger, the data rates are likely to see at NSLS-II for 16M images<br>
will be based on either 133 frames per second for full 16M images or 750<br>
frames per second for 4M ROIs. For many applications we are more likely<br>
to have to work in streaming mode, rather than waiting for full data<br>
sets that will have less demanding processing timing requirements, so we<br>
well may be looking at between 1 and 50 frames per HDF5 file. If we are<br>
lucky we will be able to make the 50 frame/file version work, but even<br>
that would mean needing to validate between 3 and 25 NeXus files per<br>
second. If we are unlucky, we might need to validate between 133 and<br>
750 files per second.<br>
<br>
There are much worse cases likely to come up in the future, so, if<br>
the validation is to be used as flexibly as possible in as many cases as<br>
possible, I would suggest going for performance, and having the heart of<br>
the code in C or C++. We should of course follow the good examples for<br>
dials, cctbx, and pymol of allowing the high performance C or C++ code<br>
to be hidden from most users behind a python wrapper, but when it is<br>
necessary to interface directly to the C or C++ code, that should be an<br>
available option.<br>
<br>
We're going to be starting by trying to drink water from a firehose.<br>
In the end I think many of us are going to be trying to drink water from<br>
Niagra Falls.<br>
<br>
Regards,<br>
Herbert<br>
<br>
<br>
<br>
On Thu, Oct 8, 2015 at 10:09 AM, Koennecke Mark (PSI)<br></div></div><span class="">
<<a href="mailto:mark.koennecke@psi.ch" target="_blank">mark.koennecke@psi.ch</a> <mailto:<a href="mailto:mark.koennecke@psi.ch" target="_blank">mark.koennecke@psi.ch</a>>> wrote:<br>
<br>
Hi,<br>
<br>
I have written a little white paper about how I would reimplement<br>
NXvalidate. It details what needs to be<br>
done and a little how. There are a number of TODO items in the text.<br>
Please have a look. May be we<br>
can sort some of this out during one of our next Telcos. Then I<br>
might start hacking away at this.<br>
Otherwise this may Code Camp material. The txt is the markdown<br>
original if someone feels like<br>
editing the text.<br>
<br>
<br>
Best Regards,<br>
<br>
Mark Koennecke<br>
<br>
<br>
<br>
_______________________________________________<br>
NeXus-committee mailing list<br></span>
<a href="mailto:NeXus-committee@nexusformat.org" target="_blank">NeXus-committee@nexusformat.org</a> <mailto:<a href="mailto:NeXus-committee@nexusformat.org" target="_blank">NeXus-committee@nexusformat.org</a>><br>
<a href="http://lists.nexusformat.org/mailman/listinfo/nexus-committee" rel="noreferrer" target="_blank">http://lists.nexusformat.org/mailman/listinfo/nexus-committee</a><span class=""><br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
NeXus-committee mailing list<br>
<a href="mailto:NeXus-committee@nexusformat.org" target="_blank">NeXus-committee@nexusformat.org</a><br>
<a href="http://lists.nexusformat.org/mailman/listinfo/nexus-committee" rel="noreferrer" target="_blank">http://lists.nexusformat.org/mailman/listinfo/nexus-committee</a><br>
<br>
</span></blockquote><div class="HOEnZb"><div class="h5">
_______________________________________________<br>
NeXus-committee mailing list<br>
<a href="mailto:NeXus-committee@nexusformat.org" target="_blank">NeXus-committee@nexusformat.org</a><br>
<a href="http://lists.nexusformat.org/mailman/listinfo/nexus-committee" rel="noreferrer" target="_blank">http://lists.nexusformat.org/mailman/listinfo/nexus-committee</a><br>
<br>
</div></div></blockquote></div><br></div>