[Nexus] NeXus - a solution to what is not the real problem ?

[Wuttke, Joachim]

Dear colleagues:

thank you for your immediate responses. Please let me correct my cover letter:
»attack on NeXus« was unnecessarily harsh. What I am attacking is not the
serious work you are doing for really complicated data set, but the idea, popular
at management level, that NeXus should ultimately be used at _all_ neutron and
X-ray instruments. All I want to say is: for certain types of instruments, migrating
to NeXus would be considerable effort, would rather increase than reduce the
diversity of data formats, would not improve the messy state of software seen by
the users.

Concerning readability by humans, let me show you an extract (omissions markes
by three dots ...) from a SPHERES raw data file in YAML format:

---
Meta:
  format: usr5.2 for yaml1
  type: compact subscan data
Shortpar:
  spectrometer: SPHERES
  expdir: S30/
  subscan: 2a39
  incremental: true
  scan_since: 2010-03-09 Tue 05:36:08 CET
...
  lambda_f: 6.27095
  ndet: 16
Longpar:
...
  T1:
    unit: K
    nobs: 1230
    avge: 279.769861788617
    min: 279.7
    max: 282.56
    stdv: 0.000309254861525375
  Tset:
    unit: K
    nobs: 1230
    avge: 280.0
    min: 280.0
    max: 280.0
    stdv: 0
Detectors:
- angle: 9.5
- angle: 12.5
...
Histogram:
...
- - -28.8
  - 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
  - 0
  - 0
  - 0
- - -28.72
  - 0 36 211 184 214 708 964 318 375 233 490 301 411 244 324 213
  - 10204646
  - 13573385
  - 11661447
- - -28.64
  - 0 19 140 115 139 454 579 203 263 197 262 209 254 166 203 140
  - 6573870
  - 8091510
  - 7512778
...

First, there is much less overhead than in XML, it is much easier to read.
So, you should think about adding YAML as another low-level format
within NeXus.

Second, the histogram section consists of an array. Each array element
contains results for one energy channel; it is an array itself; it contains
five elements: the energy of the channel, a string containing scattered
neutron counts, and three monitor counts. This data format allows me
to see within milliseconds whether there are neutron counts, at which
energy they start, whether all detectors work correctly.

If I understand correctly, NeXus requires that all array elements be of
the same type, and that the energy scale be stored in a separate array -
hence it does not allow the above form of data storage.

Best - Joachim

------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzende des Aufsichtsrats: MinDir'in Baerbel Brumme-Bothe
Geschaeftsfuehrung: Prof. Dr. Achim Bachem (Vorsitzender),
Dr. Ulrich Krafft (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------