[Nexus] About application definitions

Tue Jan 26 09:19:04 GMT 2016

Hi Armando, Ray, and others,

Ray’s summary is representing well what we discussed in the teleconference.
The subentry allows a file to conform to a number of application
definitions in the same entry, despite the fact that application
definitions are nominally exclusive and not designed to co-exist.

Offering analysis application developers to define their own “application
definition” allows them to use NeXus in the way they like. The people in
charge of writing files will then “only” have to include subentries for
all the major analysis packages the consumers of the data might want to
use. And they need to do that at the time their write the raw data. At
that point we’ve given up being a standard.

I’m not saying every step in that direction is a step too far. But we
should proceed carefully.

Regards

Tobias

P.S. I’d be happy to advertise these “features” that Ray mentioned as a
potential solution, but that would open a different discussion and they
don’t really solve Armando’s problem as far as I understand it.

On 20/01/2016, 21:23, "NeXus on behalf of Osborn, Raymond"
<nexus-bounces at nexusformat.org on behalf of rosborn at anl.gov> wrote:

>Hi Armando,
>Thanks for raising this. As you know, we discussed this at the telco this
>morning so here is, I hope, an accurate summary of what we said.
>
>As you point out, the NXsubentry group is designed to let the same
>NXentry group contain data that might be used in different contexts,
>e.g., small-angle scattering and fluorescence spectroscopy. The data
>might have been collected at the same time, but different parts of the
>data are intended for different types of analysis. The purpose of the
>application definition, which is placed in each NXsubentry, is to define
>what type of analysis that sub-entry requires. Obviously, to conform to
>the standard, each sub-entry must conform to the definition, which is
>described by an NXDL file.
>
>The committee has agreed that, if the community is happy with a simple
>application definition that doesn’t contain the hierarchy normal to most
>NeXus files (the sub-entry could even be completely flat), then that is
>fine. Just create an application definition that defines what is to be in
>the sub-entry. I think your example is to create an application
>definition for what is required for PyMCA analysis - if a few parameters
>are sufficient, then you are not required to put them in
>NXinstrument/NXmonochromator, etc, groups. You can specify them at the
>top level. If that is sufficient for the PyMCA user community, that is
>fine by us. We do encourage you to publish your application definition,
>since a flat collection of metadata without any context will be
>meaningless to outsiders.
>
>If, however, you want to analyze small-angle scattering, and the
>application definition you are conforming to requires hierarchical
>groups, you obviously have to follow the definition. That is the purpose
>of having a definition. However, NeXus allows you to link groups or
>fields multiple times, even with different names, so you don’t have to
>duplicate data required in more than one sub-entry. Just add a link and
>the data is only stored once.
>
>There is a new concept, called ‘features,' in which a single NXentry can
>specify that it implements more than one type of analysis, but that is
>still under development.
>
>You are correct that the purpose of NeXus is to give communities the
>flexibility they need to store metadata in a meaningful way. Sometimes, a
>simple file is all that is needed, but other times, the complexity is
>driven by necessity. The goal is for the meaning of data to be clear to
>other people. If you follow the NeXus group hierarchy, then the data is
>usually self-describing without any external documentation. That is our
>preference. It would be a shame, for example, not to put plottable data
>within an NXdata group since that makes it easier to identify what kind
>of data you are storing. However, if you want to use a flat file format,
>then the application definition you write will allow people to understand
>what’s in it.
>
>I hope others will chime in if I got any of this wrong.
>
>With best regards,
>Ray
>
>> On Jan 16, 2016, at 10:29 AM, V. Armando Sole <sole at esrf.fr> wrote:
>> 
>> Happy New Year!
>> 
>> At the ESRF we do not intend to use the old fashion way of writing the
>>application definitions at the top level inside an NXentry but inside
>>one (or several) NXsubentry. That will allow us to treat the files in
>>the same way whether they are single or multiple technique files.
>> 
>> I am revising the documentation concerning NXsubentry but I do not know
>>if the latest documentation is the one at:
>> 
>> 
>>http://download.nexusformat.org/sphinx/classes/base_classes/NXsubentry.ht
>>ml?highlight=nxsubentry
>> 
>> So, I would like to get that point confirmed.
>> 
>> I have to say (repeat?) that the way definitions are presented to the
>>potential users look very tedious. We *do* intend to recreate the NeXus
>>structure as much as we can, but as definitions are currently presented
>>they almost imply to repeat the NXentry structure in the NXsubentry. For
>>you it was easy to say "we have foreseen a single application definition
>>at the NXentry level, we just give the option to have it at an
>>NXsubentry level and that's all". My point is that when having it at the
>>NXentry level, you needed some structure to keep things tidy, but when
>>NXsubentry appeared, the structure is overkill.
>> 
>> From a practical point of view, a simple definition for technique XXXX,
>>requiting datasets I0 and It and a set of energies, it would require:
>> 
>> entry at NXentry
>>    ...
>>    whatever_name at NXsubentry
>>        definition (NX_CHAR="XXXX", @version, @URL)
>>        I0 [NXFLOAT]
>>        It [NX_FLOAT]
>>        energy [NX_FLOAT]
>> 
>> So, it continues to look a mistake to me, to present the definitions as
>>forcing to have "energy" inside an NXmonochromator group, I0 inside an
>>NXmonitor or an NXdetector and It inside an NXdetector. Please note that
>>I am not saying it is a bad idea to have them inside those groups!. What
>>I am saying is that to validate that a file implements the above
>>definition, one only needs to validate the presence of I0, It and
>>energy. Whether they are datasets of links to actual datasets is
>>irrelevant.
>> 
>> You may say that since we intend to implement the NeXus structure, we
>>should not worry so much and you are correct. The point is, and I repeat
>>myself, that some communities and/or facilities are elaborating their
>>own "data exchange" formats based on HDF5 that are *nothing else* than
>>the equivalent of an application definition the way I have presented it
>>here.
>> 
>> I would aim to present application definitions in that way. The
>>entrance ticket for the potential users just interested on data analysis
>>would be "less expensive". All what you would be asking compared to
>>community driven definitions would be to include strings specifying the
>>technique they implement, the version (optional but great!) and the URL
>>describing it (optional but also great!).
>> 
>> You could go as far as presenting that approach as the "official NeXus
>>approach to incorporate community driven definitions". I think you have
>>no choice, so better show some flexibility.
>> 
>> Best wishes,
>> 
>> Armando
>> 
>> _______________________________________________
>> NeXus mailing list
>> NeXus at nexusformat.org
>> http://lists.nexusformat.org/mailman/listinfo/nexus
>
>-- 
>Ray Osborn, Senior Scientist
>Materials Science Division
>Argonne National Laboratory
>Argonne, IL 60439, USA
>Phone: +1 (630) 252-9011
>Email: ROsborn at anl.gov
>
>
>
>_______________________________________________
>NeXus mailing list
>NeXus at nexusformat.org
>http://lists.nexusformat.org/mailman/listinfo/nexus