[NeXus-committee] obsolescence and evolution: NeXus needs a historic dimension

[Tobias.Richter at diamond.ac.uk]

I would agree with Joachim in at least that real world examples of a fully valid and useful NeXus files are not easy to come by. From a quick survey I could not find a single file in our examples that claims to conform to an application definition. And that seems to be the cases for many files to this day. So for the formal validation we're left with the absolute basics, which is of limited use to a consumer. You get something to plot (NXdata), so we're providing a poor version of, say, svg. 

A lot can be learned by manually inspecting those files, if they are well written (some are, some are not). But we are in general quite some way of from being able to automatically process/analyse files from different facilities. That would meet my "valid and useful" objective. We're relatively close for MX, where we've had a huge influx of experience. In other areas we may never get to that state, because of the diversity of the experiments.

In general I would think we're going in the right direction and being a moving target has proven to be a benefit, I find. Because some initial solutions were clearly not fit for purpose. We still need to move a bit, see my recent comments about "access by name" (which does not work as advertised) or recording the scan intent (which we do not do). But the current move to the CIF transformations for geometry makes NeXus a lot more machine readable. We do need to be more precise in our documentation, though. 

The NAPI/Container issue is an additional source for confusion. As Joachim highlights there are many things we do the way we do them for historic reasons. In the best case that requires a lot of explaining. But less patient or persistent people just end up doing things "wrong"/"their way" or being frustrated with the whole affair.

Tobias


> -----Original Message-----
> From: NeXus-committee [mailto:nexus-committee-bounces at nexusformat.org]
> On Behalf Of Joachim Wuttke
> Sent: 16 July 2014 13:14
> To: nexus-committee at nexusformat.org
> Subject: Re: [NeXus-committee] obsolescence and evolution: NeXus needs
> a historic dimension
> 
> Pete:
> 
> > Perhaps you have been misinformed.
> >
> >>  From recent correspondence I learned that to this date not a single
> >> institute is writing fully valid NeXus.
> >
> > At one of the NIAC meetings in the last five years, we took an
> > informal poll to discover which facilities use the NAPI to read/write
> > NeXus data files.  "Less than half" was the result.  On testing data
> > files for validity, many tested as valid.  I know of several
> > instruments that have presented data files to me that have tested as
> > valid against the current NXDL.  The software NeXpy now writes files
> > based _solely_ on the structures specified in the NXDL.
> 
> Maybe I misunderstood or misremembered some communication, which
> possibly was concerned more specifically with event-mode files.
> 
> Anyway, formal validity is perhaps a necessary, but certainly not a
> sufficient condition for interoperability.
> 
> > This topic has been often discussed at NIAC meetings, generally
> during
> > the deliberations over the request for a new feature.  Compatibility
> > is one reason why major versions are not presented frequently.
> >
> >> has time come to start thinking
> >> about a compatibility breaking next major version?
> 
> Maybe "next major version" is misleading wording for a leap like in the
> Fortran example. It's certainly not to be done frequently.
> I was rather thinking of a long time scale, and of a structured process
> that starts with evaluating what has been achieved in the past, what
> has been done right, what needs repair, and what is broken beyond
> repair. If it is determined that the latter category is empty, then the
> process can be stopped.
> 
> - Joachim


-- 
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. 
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom