[NeXus-tech] RFC Summary

Wed Nov 13 11:20:34 GMT 2013

Hi all,

Not surprisingly I agree with Pete on the uncertainties. 

I really would like to avoid having an unlimited number of variant fields at the same level as the original (current) value. If NXnote or NXcharaterization could be used for the history, that's fine. Otherwise I like Ben's approach, with the tweak that in Mark's example the field names inside the wavelength_variant group would not need to be named "wavelength" (that's clear from the container), but could have the ISO timestamp as name.

Regards,

Tobias

________________________________________
From: nexus-tech-bounces at nexusformat.org [nexus-tech-bounces at nexusformat.org] on behalf of Wintersberger, Eugen [eugen.wintersberger at desy.de]
Sent: 13 November 2013 08:34
To: NeXus Technical sub-committee list
Subject: Re: [NeXus-tech] RFC Summary

Hi all

On Tue, 2013-11-12 at 17:17 -0600, Pete Jemian wrote:

> http://www.cansas.org/formats/canSAS2012/1.0/doc/examples.html#representing-uncertainty-components
>
> Basically, all NeXus "fields" (realistically, this would be only
> numerical value fields, not strings) would have an attribute
> "uncertainty" described: (optional for numerical arrays) Name of the
> data object (in this NeXus group) that provides the uncertainty to be
> used for data analysis.  The user-named uncertainty data object would be
> another field in the same group with the same geometry.

In general this is a good idea as it makes parsing much easier (just
read an attribute and get the name of the error field). However, I would
still suggest to have a standardized naming pattern for variant fields
in order to avoid confusion when looking only at the field names.

> If an
> uncertainty requires more explanation, this is possible by putting the
> explanation in a subgroup (NXcharacterization) and defining its name in
> the "components" attribute of the uncertainty field.  When you see the
> example, you will realize it is not hard to understand, not hard to
> parse, and not hard to define.

Sounds fine for me.

>
> Variants
> -------------
>
> Some of the same design and consideration described by the canSAS for
> reporting of uncertainties might be extended to variants.  Leave the
> entire process of naming to be more flexible, defining the variant by
> means of an attribute (suggest that attribute be named "variant").

Again a good idea but I would again use a naming scheme  for field
names. Maybe one can do something like with the depends_on attribute to
establish a chain of variants where one can walk easily from one to the
next.

regards
 Eugen

>
> On 11/11/2013 2:47 AM, Mark Koennecke wrote:
> >
> > =============================================
> > Summary on Comments for variants and errors
> > =============================================
> >
> > Some weeks ago we asked for comments on two controversial topics,
> > variants and errors, from the wider NIAC. This now  is a summary off
> > the additional options which came in by email, together with the original
> > text. This summary serves as material for the next Telco and to reboot
> > the discussion.
> >
> > Variants
> > ==========
> >
> > The problem: there are parameters in a NeXus file for which better
> > values may become available later on through a refinement, calibration
> > or whatever. The requirement is to store both the old and the new value
> > together with a time stamp. Variants are required for a full mapping
> > of CIF concepts to NeXus. The experience of the CIF people is that
> > variants will be used only occasionally and only for a few fields.
> > Thus a heavy weight scheme for handling this situation is not necessary.
> >
> > The suggestion: Apply the following process when a new value is available
> > for an existing field in an NeXus file:
> > 1) memorize the current value
> > 2) Update the field and assign a time_stamp attribute to it.
> > 3) Store the old value in a new field with a name derived by appending
> >     _variant to the original field name. Assign a time_stamp attribute
> > to this
> >    new field too.
> >
> > When more then one variant is required, build names by appending
> > _variant_somename
> > , _variant_someothername to the original field name.
> >
> > An example:
> >
> > wavelength
> >    @time_stamp=sometime
> > wavelength_variant
> >    @time_stamp=someothertime
> >
> > A more elaborate example:
> >
> > wavelength
> >    @time_stamp=sometime
> > wavelength_variant_asdialled
> >    @time_stamp=someothertime
> > wavelength_variant_fromSI
> >    @time_stamp=someothertime
> >
> >
> >
> > Discussion
> > ------------
> >
> > We considered using a variant attribute to the field. But this causes
> > problems
> > when a larger field, like for example an area detector efficiency
> > correction,
> > needs a variant.  It is also non obvious how to store the time stamp then.
> >
> > We are aware of the fact that the suggested scheme can lead to long names.
> > We rely on the fact that the CIF experience is that variants are rarely
> > used.
> > And CIF has been around for a while. Reading software only needs to concern
> > itself with the real field name which is supposed to hold the best
> > available
> > value.
> >
> > Comments
> > ------------------
> >
> > Ben did not like this and argues for putting those variants a level deeper
> > in the hierarchy like:
> >
> > wavelength
> >    @time_stamp=sometime
> > wavelength_variant/
> >    wavelength
> >    @time_stamp=someothertime
> >
> > A more elaborate example:
> >
> > wavelength
> >    @time_stamp=sometime
> > wavelength_variant/
> >    wavelength
> >    NXnote = asdialled
> >    @time_stamp=someothertime
> >    wavelength_variant/
> >      wavelength
> >      NXnote = fromSI
> >      @time_stamp=someothertime
> >
> > I assmume that the groups are of class NXvariant. There is a problem in
> > the second example as NeXus (and HDF5?) do not allow two groups of the
> > same name.
> >
> >  From the CIF community, represented by Graeme Winter and Aaron Schwartz
> > there
> > was aconcern to keep the CIF details field. Which conveys more
> > information on
> > the variant. They seemed happy when another field is added when necessary.
> >
> > I am not so happy to have an NXvariant group. NXvariant will be one of
> > those
> > groups which is ill defined as it can hold almost everything.
> >
> >
> > Standard Deviation
> > =====================
> >
> >
> > The problem: Real physical values have errors. NeXus accounted for this by
> > having fields with _error appended for a few fields. The reqirement now is
> > to have a standard scheme for marking up experimental errors for any field.
> >
> > The suggestion: store the errors of a field value in a new field. The name
> > of this field is derived from the original field name by appending _esd to
> > it. Example:
> >
> > wavelength
> > wavelength_esd
> >
> > Comments
> > ---------------
> >
> > Peter Peterson commented that we more or less aggreed to use the appendix
> > _error for errors. Like in:
> >
> > wavelength_error
> >
> > Pete Jemian, on behalf on canSAS, added that there may be more complicated
> > cases of errors which may require multiple different error fields. His
> > suggested
> > solution is to have an uncertainties attribute which names the fields in
> > the same
> > group which actually hold the errors. For example:
> >
> > wavelength
> >    @uncertainties=wavelength_errors, wavelength_stddev
> > wavelength_errors
> > wavelength_stddev
> >
> > The case made for this is to describe uncertainities resulting from
> > different sources
> > or determined in different ways. Which basically is a variant on the
> > error......
> >
> >
> >
> > Best Regards,
> >
> >           Mark Koennecke
> >
> > _______________________________________________
> > NeXus-tech mailing list
> > NeXus-tech at nexusformat.org
> > http://lists.nexusformat.org/mailman/listinfo/nexus-tech
> _______________________________________________
> NeXus-tech mailing list
> NeXus-tech at nexusformat.org
> http://lists.nexusformat.org/mailman/listinfo/nexus-tech

-- 
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. 
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom