[Nexus] NeXus - a solution to what is not the real problem ?

[Wellenreuther, Gerd]

Hi Joachim,

Wuttke, Joachim wrote:
> this little example illustrates perfectly my point: at SPHERES, a scan
> consists of several subscans, each stored in a separate file. Each
> subscan contains the two parameters scan_since and subscan_since.
> Therefore, start_time would be ambiguous.
> 
> This shows: you cannot unify a raw data format without unifying all
> experimental procedures. Therefore, you will always have different
> flavors of raw data formats at different instruments, just as described
> by Gerd (http://lists.nexusformat.org/pipermail/nexus/2010/000371.html).

Well, I disagree :). Nobody will stop you from saving individual times 
for all your subscans. You will not, repeat, will not break NeXus by 
saving additional data in your file. It will still be NeXus compliant. 
If pushed to the extremes every HDF5-file is NeXus-compliant, as long as 
it is not using anything defined by NeXus in a different way.

But of course the aim is to use common fields like start_time, 
wavelength, energy, etc.

What I meant with flavor is that e.g. at Soleil the software is assuming 
that you do not only follow NeXus, but also some further Soleil-specific 
definitions (e.g. about how to find images in a NeXus-file). Of course, 
nobody else was using this definitions, so nobody else is able to use 
that software. But this is a problem on the software-side. Armando Sole 
has demonstrated that you can always interpret any NeXus-file as HDF5, 
and let the user decide which data to interpret as a picture. That is a 
very safe fallback solution!

So, I can not see why there is a need to "unify all experimental 
procedures". Just do what you have ever done. Data for which a sensible 
definition in NeXus exists should be written in that field properly. 
Everything else can be stored in the NeXus-file, hopefully roughly at 
the right place, just under names which are not conflicting with NeXus 
(just using a pre- or postfix like _SPHERES should do).

> To be perfectly sincere: effort is not even the main argument. If you offered me
> to do the complete conversion, I fear I would still refuse for esthetic reasons.
> For my taste, the NeXus documentation looks hopelessly bloated. As one
> contributor said today: for my purpose, NeXus would be overkill.

Aesthetic is a hole different can of worms. To you, a solid ASCII-file 
appears beautiful (at least that is my impression). From my perspective, 
working at the synchrotron, I would have to face working with hundred 
thousands of these nice, human-readable files for each mapping. All 
containing a couple of counters, and a spectrum of lets say 4096 points. 
That is a scientists nightmare, for several reasons:

* you need high performance file systems to be able to do *anything* 
with these files
* you waste approximately a factor of 10 in hard drive space for storing 
the files
* you have to transform the numeric data into strings during acquisition 
time, open a single file, write that data, close the file. This is much 
more time consuming than writing HDF5/NeXus. Our experiments are 
currently limited by the overhead if we are writing single ASCII-files.
* you have to do the same transformation back again if you want to use 
the data - this goes well into the couple of minutes / hours for the 
above mentioned example. Using HDF5/NeXus we are talking about seconds.

I hope, you understand my perspective. And, as you might have guessed, 
there are a lot of facilities with a lot of instruments each going into 
operation in the next years which are facing the same problems!

I can understand your point, though. For you and your users comparing 
data from one instrument to others is not a high priority. You just want 
to have an easy access to the data, and keep things as simple as they 
are now. As long as you can live with the constraints of an isolated 
environment everything is fine.

But as you just mentioned, the workload put on you to implement NeXus 
itself does not seem to be the actual problem - so maybe you are talking 
to the wrong people right now. Who is actually trying to force you to 
use NeXus?

Cheers, Gerd

-- 
Dr. Gerd Wellenreuther
beamline scientist P06 "Hard X-Ray Micro/Nano-Probe"
Petra III project
HASYLAB at DESY
Notkestr. 85
22603 Hamburg

Tel.: + 49 40 8998 5701