[Nexus] NeXus - a solution to what is not the real problem ?

[Sebastian Busch]

Dear Joachim, dear list,

first off: Greetings, I'm new here; I've neither written a single byte
of NeXus file nor ever worked with one. My statements reflect my current
understanding which is not very advanced -- please correct me if I
assert wrong things!


Nevertheless, I would like to say in the following why I think that your
requirements can all be met by (a YAML version of) NeXus. Joachim, I'll
cite parts of your different e.mails without distinction.

For me, the most important feature is that NeXus requires a minimal set
of declarations. A user can be absolutely sure that all the measures,
numbers and units he needs for data reduction are in the file he takes
home. Nowadays, he finds only what the instrument scientist regarded
necessary.

Wuttke, Joachim wrote:
> When migrating an existing spectrometer towards NeXus,
> the instrument scientist needs ... to provide routines
> that achieve lossless conversion from the old
> into the new format.

"lossless" is not a problem. There is no information which you cannot
store in a NeXus file. It imposes a minimal set of parameters on you but
if you want to store more, you can do so.

> ... YAML \cite{qda5}
> allow to store and retrieve
> complex data, composed of scalars, hashes, arrays
> in arbitrary tree-like structures,
> at zero cost through a much simpler API.

using YAML together with its simple API is not contradicting using
NeXus. NeXus will only force you to give your YAML file specific entries
and a specific structure. using your example:
> puts d['Shortpar']['scan_since']
the user would have to guess that you call the start time "scan_since".
if you were using the NeXus names, the user knows that he/she has to
load "start_time". simpler for him/her, no difference for you.

the user could then choose to use some NeXus API which returns the data
as numpy array or whatever -- but of course he/she doesn't have to.

> Most fundamentally,
> I think that efforts to unify the raw data format
> are adressing the wrong interface:
> most users do not want to see raw data at all.
> What users want is a calibrated, normalized, reasonably binned
> scattering law $S(q,\omega)$.

It seems to me that this is not a point against NeXus (the "why the
xy-instrument would be worse off if it were using NeXus" kind of
against), rather a reason why many give it such a low priority.

> it will remain the instrument scientist's resposibility
> to plug in a low-level routine that reads in and calibrates the
> raw data from his instrument.
> Only he has the technical knowledge required to do it correctly,
> and hardly anybody else needs to care about the raw data and their format.

I think that this technical knowledge must be documented: the numbers
(distances, units, ...) in the data files; the procedure what to do with
these numbers in some kind of publication. Otherwise the user relies on
the instrument scientist to an unhealthy degree. I think that it is the
duty of every user=scientist that he/she can reproduce every step which
led to his/her results. This must under no circumstances involve a step
"then the instrument scientist does some voodoo with the data".

[now talking about an example YAML file from SPHERES]
> This data format allows me
> to see within milliseconds whether there are neutron counts, at which
> energy they start, whether all detectors work correctly.
> If I understand correctly, NeXus requires that all array elements be
> of the same type, and that the energy scale be stored in a separate
> array - hence it does not allow the above form of data storage.

Maybe it does not allow the form of data storage you mention, but you
should be able to reach your millisecond-goals nevertheless. The
detector would be a NXdetector
http://www.nexusformat.org/NXdetector
to which you can add monitors ad libitum -- without having real data,
the examples on the website do not look very different from your file
for me...



Perhaps I did not understand your wishes and/or the NeXus requirements
but I think that you could be perfectly happy with a YAML version of NeXus.


Best regards,
Sebastian.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: sbusch.vcf
Type: text/x-vcard
Size: 446 bytes
Desc: not available
URL: <http://lists.nexusformat.org/pipermail/nexus/attachments/20100309/bda34a70/attachment.vcf>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 260 bytes
Desc: OpenPGP digital signature
URL: <http://lists.nexusformat.org/pipermail/nexus/attachments/20100309/bda34a70/attachment.sig>