[NeXus-committee] obsolescence and evolution: NeXus needs a historic dimension

[Osborn, Raymond]

To add some more historical context, since the earliest NeXus discussions, we talked about different levels of adherence to the standard. The lowest is to use HDF5 and conform to the design guidelines (put plottable data in NXdata groups, sample information in NXsample groups, etc). The next is to use the NeXus base classes as a glossary of the correct names to use for NeXus objects. The top level is to adhere to a strict application definition, so that programs can access all required variables reliably.

A program like NeXpy already shows the value of the first two levels. I can open, inspect, plot, and manipulate any (HDF5) NeXus file wherever it comes from, SNS, Diamond, ISIS, etc. That is a huge advance over what I’ve been used to over the last 20 years. I now have control over my own data. These two levels are also sufficient for me, as a user, to store the results of my own data analysis - depending on the particular experiment I’m doing, I may want to store different parameters, such as the results of least-squares fits, or the results of simulations. They won’t validate using standard tools, but they can again be inspected, plotted, etc. and other people can also inspect and plot the results, using a generic NeXus tool like NeXpy. Of course, if we expect a NeXus file to be read by a specific application, like a Rietveld analysis program, then it needs to conform to well-defined standard. We have a growing list of application definitions to make this possible and we should encourage those who write applications to contribute to this list so that data reduction software, like Mantid, can export validated files.

I’m sure that the NeXus classes can be improved, and, in spite of the heroic efforts of Pete Jemian, I expect we could make improvements to the documentation. The tools are in constant need of improvement. However, I’m not sure I understand why we need a whole new version of the language because we’ve found some files that don’t validate. I know several people who are put off NeXus because they perceive it as overly bureaucratic, so we need to guard against that perception.

With regards,
Ray 

On Jul 16, 2014, at 7:13 AM, Joachim Wuttke <j.wuttke at fz-juelich.de> wrote:

> Pete:
> 
>> Perhaps you have been misinformed.
>> 
>>> From recent correspondence I learned that to this date not a single
>>> institute is writing fully valid NeXus.
>> 
>> At one of the NIAC meetings in the last five years, we took an informal
>> poll to discover which facilities use the NAPI to read/write NeXus data
>> files.  "Less than half" was the result.  On testing data files for
>> validity, many tested as valid.  I know of several instruments that have
>> presented data files to me that have tested as valid against the current
>> NXDL.  The software NeXpy now writes files based _solely_ on the
>> structures specified in the NXDL.
> 
> Maybe I misunderstood or misremembered some communication, which
> possibly was concerned more specifically with event-mode files.
> 
> Anyway, formal validity is perhaps a necessary, but certainly not a
> sufficient condition for interoperability.
> 
>> This topic has been often discussed at NIAC meetings, generally during
>> the deliberations over the request for a new feature.  Compatibility is
>> one reason why major versions are not presented frequently.
>> 
>>> has time come to start thinking
>>> about a compatibility breaking next major version?
> 
> Maybe "next major version" is misleading wording for a leap like in
> the Fortran example. It's certainly not to be done frequently.
> I was rather thinking of a long time scale, and of a structured
> process that starts with evaluating what has been achieved in the
> past, what has been done right, what needs repair, and what is broken
> beyond repair. If it is determined that the latter category is empty,
> then the process can be stopped.
> 
> - Joachim
> 
> _______________________________________________
> NeXus-committee mailing list
> NeXus-committee at nexusformat.org
> http://lists.nexusformat.org/mailman/listinfo/nexus-committee

-- 
Ray Osborn, Senior Scientist
Materials Science Division
Argonne National Laboratory
Argonne, IL 60439, USA
Phone: +1 (630) 252-9011
Email: ROsborn at anl.gov