[NeXus-tech] RFC Summary

[Pete Jemian]

Mark:

I agree with your summary.  Since I cannot make tomorrow's meeting, my 
comments and suggestions are here instead:

Uncertainties
-------------

Despite previous agreement to use the appendix "_error" for errors, 
please examine the canSAS work more deeply.  The canSAS work has already 
considered the naming of things and the inclusion of common and unusual 
sorts of uncertainty.

http://www.cansas.org/formats/canSAS2012/1.0/doc/examples.html#representing-uncertainty-components

Basically, all NeXus "fields" (realistically, this would be only 
numerical value fields, not strings) would have an attribute 
"uncertainty" described: (optional for numerical arrays) Name of the 
data object (in this NeXus group) that provides the uncertainty to be 
used for data analysis.  The user-named uncertainty data object would be 
another field in the same group with the same geometry.   If an 
uncertainty requires more explanation, this is possible by putting the 
explanation in a subgroup (NXcharacterization) and defining its name in 
the "components" attribute of the uncertainty field.  When you see the 
example, you will realize it is not hard to understand, not hard to 
parse, and not hard to define.

Is the use of the "_error" appendix so ingrained within the NeXus 
standard that we cannot reconsider?

When we say "standard deviations" or "errors", we really mean 
_estimates_ of these quantities.  There is no way we can know the error 
on any given datum.  We might have a standard deviation based on some 
statistical measure (repeat measurements, for example) but this is not 
always the case.  Rather, these are all gathered together under the more 
general name of "uncertainties".  It is improper to generalize these 
various uncertainties as "errors" or "standard deviations".

Variants
-------------

Some of the same design and consideration described by the canSAS for 
reporting of uncertainties might be extended to variants.  Leave the 
entire process of naming to be more flexible, defining the variant by 
means of an attribute (suggest that attribute be named "variant").  The 
canSAS scheme includes an easy way to avoid having two items (fields or 
groups) with the same name within the same HDF group.

When updating the variant, the old value could be pushed into the 
subgroup (NXlog seems the right base class for this, or NXnote) named in 
the "variant_history" attribute with suitable provenance metadata attached.

NXvariant group
-----------------

I agree there is no need to create a container group for variants.  We 
already have two base classes which could be used: NXcharacterization 
and NXnote.

Wish you a great meeting.

Pete



On 11/11/2013 2:47 AM, Mark Koennecke wrote:
>
> =============================================
> Summary on Comments for variants and errors
> =============================================
>
> Some weeks ago we asked for comments on two controversial topics,
> variants and errors, from the wider NIAC. This now  is a summary off
> the additional options which came in by email, together with the original
> text. This summary serves as material for the next Telco and to reboot
> the discussion.
>
> Variants
> ==========
>
> The problem: there are parameters in a NeXus file for which better
> values may become available later on through a refinement, calibration
> or whatever. The requirement is to store both the old and the new value
> together with a time stamp. Variants are required for a full mapping
> of CIF concepts to NeXus. The experience of the CIF people is that
> variants will be used only occasionally and only for a few fields.
> Thus a heavy weight scheme for handling this situation is not necessary.
>
> The suggestion: Apply the following process when a new value is available
> for an existing field in an NeXus file:
> 1) memorize the current value
> 2) Update the field and assign a time_stamp attribute to it.
> 3) Store the old value in a new field with a name derived by appending
>     _variant to the original field name. Assign a time_stamp attribute
> to this
>    new field too.
>
> When more then one variant is required, build names by appending
> _variant_somename
> , _variant_someothername to the original field name.
>
> An example:
>
> wavelength
>    @time_stamp=sometime
> wavelength_variant
>    @time_stamp=someothertime
>
> A more elaborate example:
>
> wavelength
>    @time_stamp=sometime
> wavelength_variant_asdialled
>    @time_stamp=someothertime
> wavelength_variant_fromSI
>    @time_stamp=someothertime
>
>
>
> Discussion
> ------------
>
> We considered using a variant attribute to the field. But this causes
> problems
> when a larger field, like for example an area detector efficiency
> correction,
> needs a variant.  It is also non obvious how to store the time stamp then.
>
> We are aware of the fact that the suggested scheme can lead to long names.
> We rely on the fact that the CIF experience is that variants are rarely
> used.
> And CIF has been around for a while. Reading software only needs to concern
> itself with the real field name which is supposed to hold the best
> available
> value.
>
> Comments
> ------------------
>
> Ben did not like this and argues for putting those variants a level deeper
> in the hierarchy like:
>
> wavelength
>    @time_stamp=sometime
> wavelength_variant/
>    wavelength
>    @time_stamp=someothertime
>
> A more elaborate example:
>
> wavelength
>    @time_stamp=sometime
> wavelength_variant/
>    wavelength
>    NXnote = asdialled
>    @time_stamp=someothertime
>    wavelength_variant/
>      wavelength
>      NXnote = fromSI
>      @time_stamp=someothertime
>
> I assmume that the groups are of class NXvariant. There is a problem in
> the second example as NeXus (and HDF5?) do not allow two groups of the
> same name.
>
>  From the CIF community, represented by Graeme Winter and Aaron Schwartz
> there
> was aconcern to keep the CIF details field. Which conveys more
> information on
> the variant. They seemed happy when another field is added when necessary.
>
> I am not so happy to have an NXvariant group. NXvariant will be one of
> those
> groups which is ill defined as it can hold almost everything.
>
>
> Standard Deviation
> =====================
>
>
> The problem: Real physical values have errors. NeXus accounted for this by
> having fields with _error appended for a few fields. The reqirement now is
> to have a standard scheme for marking up experimental errors for any field.
>
> The suggestion: store the errors of a field value in a new field. The name
> of this field is derived from the original field name by appending _esd to
> it. Example:
>
> wavelength
> wavelength_esd
>
> Comments
> ---------------
>
> Peter Peterson commented that we more or less aggreed to use the appendix
> _error for errors. Like in:
>
> wavelength_error
>
> Pete Jemian, on behalf on canSAS, added that there may be more complicated
> cases of errors which may require multiple different error fields. His
> suggested
> solution is to have an uncertainties attribute which names the fields in
> the same
> group which actually hold the errors. For example:
>
> wavelength
>    @uncertainties=wavelength_errors, wavelength_stddev
> wavelength_errors
> wavelength_stddev
>
> The case made for this is to describe uncertainities resulting from
> different sources
> or determined in different ways. Which basically is a variant on the
> error......
>
>
>
> Best Regards,
>
>           Mark Koennecke
>
> _______________________________________________
> NeXus-tech mailing list
> NeXus-tech at nexusformat.org
> http://lists.nexusformat.org/mailman/listinfo/nexus-tech