[NeXus-tech] First draft of NeXus publication
Pete Jemian
prjemian at gmail.com
Mon Jul 14 21:44:54 BST 2014
Mark:
Nice job.
Enclosed are my revisions. Various spelling was changed and some other
changes. Nothing very big. Without the bibliography file, it was not
possible to build the complete PDF. However, these changes compile by
latex (revtex) with no build errors.
Another URL citation for canSAS was added since that work is ongoing and
near completion as well.
The table organization that you present is most valuable. This content
is now missing in the manual and not easy to deduce with such a
presentation. Once you are settled on the design, I plan to put it into
the Design section at a "good" location. (Suggestions are welcome.)
Regards,
Pete
On 7/14/2014 2:36 AM, Mark Könnecke wrote:
> Hi,
>
> here is a first draft of a new publication on NeXus intended for Reviews in Scientific Instrumentation (RSI).
> There are still two things wrong with this draft:
>
> - The proper attribution of NIAC. I have contacted the AIP helpline about how they want to have that done.
> - The lines in the tables are incorrect. I will fix this once we are sure that this is the representation we
> want.
>
> I think we can start discussing the content based on this draft.
>
> When we are done with this I will ask an american colleague to proofread this. The journal
> wants american english which I find a challenge.
>
> I included the text source files for people who wish to edit themselves. But the revtex package is
> required to compile the latex.
>
> Best Regards,
>
>
> Mark
>
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> NeXus-tech mailing list
> NeXus-tech at nexusformat.org
> http://lists.nexusformat.org/mailman/listinfo/nexus-tech
>
-------------- next part --------------
% ****** Start of file aipsamp.tex ******
%
% This file is part of the AIP files in the AIP distribution for REVTeX 4.
% Version 4.1 of REVTeX, October 2009
%
% Copyright (c) 2009 American Institute of Physics.
%
% See the AIP README file for restrictions and more information.
%
% TeX'ing this file requires that you have AMS-LaTeX 2.0 installed
% as well as the rest of the prerequisites for REVTeX 4.1
%
% It also requires runnin. The commands are as follows:
%
% 1) latex aipsamp
% 2) bibtex aipsamp
% 3) latex aipsamp
% 4) latex aipsamp
%
% Use this file as a source of example code for your aip document.
% Use the file aiptemplate.tex as a template for your document.
\documentclass[%
aip,
%jmp,%
%bmf,%
rsi,
amsmath,amssymb,
%preprint,%
reprint,%
%author-year,%
%author-numerical,%
]{revtex4-1}
\usepackage{graphicx}% Include figure files
\usepackage{dcolumn}% Align table columns on decimal point
\usepackage{bm}% bold math
\usepackage{url}
%\usepackage[mathlines]{lineno}% Enable numbering of text and display math
%\linenumbers\relax % Commence numbering lines
\begin{document}
\preprint{AIP/123-QED}
\title{The NeXus Data Format}
\author{Mark K\"onnecke\\on behalf of the\\NeXus International Advisory Committee}
\affiliation{
Laboratory for Development and Methods\\Paul Scherrer Institute\\5232 Villigen-PSI\\Switzerland
}
\date{\today}% It is always \today, today,
% but any date may be explicitly specified
\begin{abstract}
NeXus is an effort by an international group of scientists to define
a common data exchange format for neutron, muon and x-ray scattering.
NeXus is built on top of the scientific data format HDF-5. It adds domain
specific rules for organizing data in HDF-5 files and a dictionary of well
defined domain specific field names. The NeXus data format is dual purpose: NeXus
can be used as a container format to store as much data as possible about a beamline.
Such use is actively encouraged. The other use case of NeXus is to define verifiable
standards for the application of NeXus to a well defined experimental technique.
NeXus provided structures for raw experimental data as well as for processed data.
\end{abstract}
%%\pacs{Valid PACS appear here}% PACS, the Physics and Astronomy
% Classification Scheme.
\keywords{NeXus,data format, HDF5, x-ray, neutron,data analysis, data management}
\maketitle
\section{Introduction}
Increasingly, major neutron and x-ray facilities choose to store their data in the NeXus data format.
Since 2006, NeXus\cite{nxold} has undergone substantial refocussing,
refinement and enhancement as described in this paper.
Historically, neutron and x-ray facilities choose to store their data in a plethora of
home grown data formats. This scheme has a number of drawbacks addressed by NeXus:
\begin{itemize}
\item It makes the life of the travelling scientist more difficult then it needs to be as she has to deal with multiple files
in different formats, file converters and such in order to extract scientific information from the data.
\item An unnecessary burden is imposed on data analysis software producers as they have to accommodate so many different formats.
\item The whole idea of open access to data is sabotaged if the data is in a format which cannot be easily understood.
\item Modern high speed detectors produce data at such a rate that many older single image storage schemes become impractical and
an efficient container format is a necessity.
\end{itemize}
The first necessity in a data format is a physical file format: how is the data written to disk? Rather then inventing
yet another format, NeXus choose HDF-5\cite{hdf5} as the binary container format. HDF-5 is efficient, self describing,
platform independent, in the public domain and well supported by both commercial and free software tools.
NeXus adds to HDF-5:
\begin{itemize}
\item Rules for organizing domain specific data within a HDF-5 file
\item A dictionary of documented domain specific field names
\item Definitions of standards that can be validated
\end{itemize}
The development of NeXus is overseen by a committee, the NeXus International Advisory Committee (NIAC).
\section{NeXus Design Principles}
NeXus encourages its users to store as much data as possible about their experiment or data analysis
in a NeXus container file. The general idea is for users to have to deal with one file only. The user benefits
from a complete container file in other ways: if there are problems with the primary data, she might find the reason
amongst the other content stored within
the data file. The user may be able to analyse data in ways yet unforeseen because the necessary additional
fields have been stored. The burden of writing complete files is usually with the producing
facility or software, not the user. Thus this is no problem for the user.
Consistent with the container file objective, it is always possible to add data, even non-NeXus data,
to a NeXus data file without breaking NeXus.
Of course, NeXus strives for platform independence, a self describing efficient file format, public
domain specifications and tools. These aims were the reasons for choosing HDF-5 as the physical file format.
A NeXus container file may contain hundreds of fields. Often, all that is needed for data reduction or analysis are 20-30
fields or less. These necessary fields are dependent on the experimental technique. NeXus provides a way to
describe these necessary fields as a NeXus application definition. Files can be validated against such an application definition
by checking for the presence of the required fields. This is the way how NeXus expresses file standards.
\section{NeXus File Hierarchies}
For an overview of the NeXus data file structure for raw experimental data see table \ref{rawfile}.
\begin{table}
\label{rawfile}
\begin{tabular}{|p{1cm}|p{6.5cm}|}
\hline
\multicolumn{2}{|l|}{NXroot}\\
\hline
The root level of a NeXus file &
\begin{tabular}{p{1.5cm}|p{5.0cm}}
\multicolumn{2}{l}{NXentry}\\
\hline
All data belonging to one scan or run. A given NeXus file
can contain multiple related scans or runs &
\begin{tabular}{p{1.5cm}|p{3.3cm}}
\multicolumn{2}{l}{NXinstrument}\\
\hline
The data needed to describe an instrument. Contains groups for
each relevant instrument component&
\begin{tabular}{p{3.3cm}}
NXsource\\
\hline
NXmoderator\\
\hline
NXvelocity\_selector\\
\hline
NXcollimator\\
\hline
NXattenuator\\
\hline
NXdetector\\
\end{tabular}\\
\hline
\multicolumn{2}{l}{NXsample}\\
\multicolumn{2}{p{3.3cm}}{All the information about the sample}\\
\hline
\multicolumn{2}{l}{NXmonitor}\\
\multicolumn{2}{l}{Control monitor}\\
\hline
\multicolumn{2}{l}{NXuser}\\
\multicolumn{2}{l}{User information}\\
\hline
\multicolumn{2}{l}{NXdata}\\
\multicolumn{2}{p{3.3cm}}{Links to plottable data in the NXdetector group --
one instance for each detector bank. This provides support for generating a
typical plot automatically}\\
\end{tabular}\\
\end{tabular}\\
\hline
\end{tabular}
\caption{Overview of the structure of a NeXus raw data file}
\end{table}
NeXus data files are organized into a hierarchy of groups which, in turn, can contain further groups or fields.
This is very much like a file system of data within a file.
\subsection{NeXus Raw Data File Hierarchy}
The NeXus raw data file hierarchy is the consequence of some considerations: When looking at a beamline it is easy to
discern different components: beam optic components, sample position, detectors and such. It comes quite natural to store the
data for each component in a separate group. This approach explains the list of beamline components in the NXinstrument group.
As there can be multiple instances of the same kind of equipment, like slits or detectors, in a given beamline it becomes necesseary
to add a type information to the group name. This type information is the NeXus class name. By convention NeXus class names start
with the prefix NX. Each NeXus group describing a beamline
component contains further groups and fields describing the component. Fields can be single numbers or arrays, as appropriate to
the data to be described.
The requirement to store multiple related experiments in the same file or to capture
a complete workflow in a file causes the beamline component hierarchy to be pushed one level deeper into a NXentry
group in the hierarchy. The NXentry group thus represents one experiment or a processed data entry. The NXentry also holds
the experiment metadata.
The user requirement to be able to access sample information and counting information such as monitors quickly
pulled this information out of the beamline hierarchy into the NXentry level.
Another requirement was to provide easy access to a default visualization of the data. This idea was realised with the
NXdata type group at NXentry level. The NXdata group contains the data and axes data necessary to create a default plot of
the data in the entry. This is data which will typically live in the NXdetector or other groups in the
beamline component list. In order to avoid duplicating data, a HDF-5 feature called links is used. A link
is a pointer to data sitting at another place in the file hierarchy. Like a symbolic link in a unix
file system. The NXdata group is supposed to hold such links to the relevant pieces of information.
\subsubsection{Multiple Method Instruments}
Especially at X-ray sources, some beamlines implement multiple techniques in the same instrument.
For example small-angle scattering (SAS) and powder diffraction in the same beamline are sometimes measured
simultaneously (such as SAXS/WAXS). This poses the problem for an automatic data analysis tool to find its data
for processing. NeXus solves this problem through a scheme involving these steps:
\begin{enumerate}
\item All the data is stored in one NXentry hierarchy
\item At entry level, for each experimental method of the beamline a NXsubentry group is introduced. The
NXsubentry groups have the same hierarchy as the NXentry group. NXsubentry holds only the data relevant
for the experimental method it describes. In order to avoid data duplication, links into the main NXentry
hierarchy are used.
% This is still awkward without referring to NXDL (which adds jargon). NXDL is described later.
NXsubentry usually only holds the 20-30 data items required by its corresponding NeXus
application definition.
\end{enumerate}
\subsubsection{Scans}
Scans come in all shapes and sizes. Almost anything can be scanned against anything.
% Thus it is very difficult to capture scans in a standard.
An additional difficulty is that in practice, the number of scan points in the scan
cannot be known in advance since it is possible that a scan may be interrupted or terminated
before its planned number of observations.
% Because scans can and will be interrupted.
Thus, it is a challenge to standardize a scan.
NeXus solves these difficulties through
a couple of conventions and the use of a HDF-5 feature called unlimited dimensions. With the HDF-5
unlimited dimensions feature, one axis of the data is allowed to expand without limit and
the size of a data array does not need to be declared in advance. Data can be appended
to an array along the unlimited dimension as required.
Scans are stored in NeXus following these conventions:
\begin{itemize}
\item Each variable varied or collected in the scan is stored at its appropriate place in the NeXus beamline
hierarchy as an array. The array's first dimension is the number of scan points. This is the unlimited dimension in
the implementation and data is appended at each scan point to the array.
\item The NXdata group holds links to all the variables varied or collected during the scan. This serves two purposes:
It recreates something similar to the tabular representation people are used to for representing scans. The purpose
of the scan, what was scanned against what, can be discovered by inspecting the NXdata group alone.
\end{itemize}
\subsubsection{Coordinate System and Positioning of Components}
For analysing data it is often necessary to know the exact position and orientation of beamline components.
The first thing needed is a reference coordinate system. NeXus chose to use same coordinate system as the
neutron beamline simulation software McStas\cite{mcstas}.
For describing the placement and orientation of components, NeXus stores the same information as used for the
same purpose in the Crystallographic Interchange Format (CIF)\cite{ITCVG}. CIF (and NeXus) stores the details
of the translations and rotations necessary to move a given component from the zero point of the coordinate
system to its actual position. As coordinate transformations are not commutative, the order of transformations
must also be stored.
\begin{table}
\label{procfile}
\begin{tabular}{|p{1cm}|p{6.5cm}|}
\hline
\multicolumn{2}{|l|}{NXroot}\\
\hline
The root level of a NeXus file &
\begin{tabular}{p{1.5cm}|p{5.0cm}}
\multicolumn{2}{l}{NXentry}\\
\hline
All data belonging to this processed data entry&
\begin{tabular}{p{1.5cm}|p{3.3cm}}
\multicolumn{2}{l}{NXprocess}\\
\hline
The data needed to describe this processing step&
\begin{tabular}{p{3.3cm}}
input:NXparameter\\
\hline
output:NXparameter\\
\hline
\end{tabular}\\
\hline
\multicolumn{2}{l}{NXsample}\\
\multicolumn{2}{p{3.3cm}}{All the information about the sample}\\
\hline
\multicolumn{2}{l}{NXdata}\\
\multicolumn{2}{p{3.3cm}}{The result data from the data processing including its axes}\\
\end{tabular}\\
\end{tabular}\\
\hline
\end{tabular}
\caption{Overview of the structure of a NeXus processed data file}
\end{table}
\subsection{Processed Data}
On request of the user community, NeXus created a simplified structure for storing the result of data
processing: be it reduction or analysis. In many cases even the reduced data is big enough to ask for
an efficient binary representation. A good example being a tomography reconstruction. A tabular representation
of the NeXus processed data file structure is given in table \ref{procfile}.
The hierarchy is much reduced as it is not important to carry all experimental information in the data
reduction. In contrast to the raw data file structure, NXdata in the processed file structure is the place
to store the results of the processing, together with its associated axes if the result is a multi
dimensional array.
Information about the sample and instrument can be stored in NXsample and NXinstrument groups as required.
In addition, there is a new structure to store details about the processing such as the program used, its version,
the date of processing, and other metadata
in the NXprocess group. The NXprocess group can hold additional NXparameter groups which are containers
for storing the input and output parameters of the program used to perform the processing.
\subsection{Further Rules}
In addition there are further rules in NeXus which will not be detailed here for the sake of brevity.
All NeXus rules are given in the NeXus manual\cite{nxman}. These additional rules govern:
\begin{itemize}
\item How axes are associated with data
\item That units must be given with the data
\item How data is to be stored in NeXus fields
\item How to describe array data which is not in ANSI C storage order
\end{itemize}
and some more.
\section{The NeXus Dictionary}
As can be seen from the discussion of the NeXus file hierarchy, NeXus arranges data into groups which have a
type descriptor, the NeXus base class, associated with them. For each of these NeXus base classes, there
exists a description of all the fields and groups possible within such a NeXus base class. And a documentation
of the scientific meaning of the field. The collection of these NeXus base classes constitute the NeXus dictionary.
The term class is a little misleading: NeXus base classes are no classes in strict object oriented notation. Rather
they are dictionaries of allowed names. A common misconception is that NeXus users have to give values for all the
names in a NeXus base class. This is not the case, only those applicable to the instrument and situation at hand need
to be written to file.
The NeXus base classes are encoded in NeXus Description Language (NXDL).
NXDL is an XML file that specifies the content of the NeXus base class.
Procedures are in place to extend NeXus base classes quickly when necessary.
\section{NeXus Application Definitions}
The NeXus approach to standardization is unique for a standard. At first NeXus asks you to store as much
information as possible about your data. In a second step NeXus then defines a standard for a certain use case of NeXus,
an application definition, wherein domain specific fields for that use case
are described as well as their locations within the NeXus file. This
definition is the NeXus application definition. Use cases can be experimental techniques like SAS,
powder diffraction, processed data use cases such as tomography reconstructions or S(Q,Omega), SAS,
or even administrative use cases such as archiving. This scheme works because data in a NeXus file
is accessed by its path. Thus it is trivial to ignore data which is not of interest for a particular use case.
Another way to look at a NeXus application definition is as a contract (or use case) between file writers and file consumers
about the minimum content of the file necessary to fulfil that particular contract.
NeXus application definitions are expressed in NXDL. They may be parsed either by humans or by software and
they may be validated for syntax and content. The NXDL files are used to validate the structure of
NeXus data files. A tool exists to perform such validation.
A great number of candidate NeXus application definitions exist which were derived from our understanding
of the technique described. For each of these, the NeXus team seeks community approval. Unfortunately this is
quite difficult to achieve as the communities involved are not organized enough to approve or enforce
standards. One community which is organized enough is the crystallographic community. Currently scientists
from NeXus and the IUCr are nearly finished with a NeXus application definition for protein crystallography.
CBFlib\cite{cbflib} is being extended to work with NeXus-PX format. This work will be published in another paper.
Work on another NeXus application definition for reduced small-angle scattering data
is also in progress (http://www.cansas.org/formats/canSAS2012/1.0/doc/) by members of
canSAS, NeXus, and the IUCr Commission on Small-Angle Scattering.
\section{Uptake of NeXus}
NeXus is already in use as the main data format at facilities like Soleil, Diamond, SINQ, SNS, Lujan/LANL
and KEK. Other facilities like ISIS, DESY and the muSR community are in the process of moving towards
NeXus as data format. The adoption of NeXus took time. The reason is that NeXus is often chosen whenever
a facility starts operation or undergoes major refurbishments. For those facilities where there is an existing and working
pipeline from data acquisition to data analysis, the resources are usually lacking to move
towards NeXus.
\section{NeXus Governance}
The development of NeXus is overseen by the NeXus International Advisory Committee (NIAC).
In the NIAC, most relevant facilities are represented. It is easy to join the NIAC if
there is justified interest.
\section{Backwards Compatability}
In the past, NeXus supported data files in HDF-4, HDF-5, and XML file formats.
To support writing software to write and read these file formats, the NeXus
application/programmer interface (NeXus-API) was provided.
This API still exists and is maintained at a bug fix level.
On request of the community, we now concentrate our efforts only
on the HDF-5 files and tools.
\section{Summary}
NeXus has matured considerably over the last 10 years and is now in use in many facilities. NeXus
is flexible enough to accommodate a wide variety of instruments and scientific applications.
Yet it is efficient enough to
handle the data coming from modern high speed detectors. For more information,
do not hesitate to consult the NeXus WWW--site\cite{nxwww} or to contact
the members of the NIAC.
\nocite{*}
\bibliography{nexus14aip}% Produces the bibliography via BibTeX.
\end{document}
%
% ****** End of file aipsamp.tex ******
More information about the NeXus-tech
mailing list