[Nexus] Release of NeXpy v1.0.0

Fri Mar 3 00:01:39 GMT 2023

I am pleased to announce the official release of both NeXpy v1.0.0 and nexusformat v1.0.0. Complete documentation for both packages is available online (https://nexpy.github.io/nexpy/).

NeXpy
NeXpy provides a GUI interface to data stored as HDF5 files, particularly those conforming to the NeXus standard. It comprises a simple PyQt GUI that combines a tree view of loaded data, Matplotlib panes to visualize multidimensional arrays, and an IPython shell for manipulating the data and metadata from the command line. NeXpy is intended to be a simple, intuitive, interactive toolbox that allows users to :

   * visualize NeXus data rapidly
   * make arbitrary projections and slices
   * easily compare multiple datasets with synchronized plot windows
   * consolidate data from multiple files into single multidimensional arrays with minimal overhead
   * perform least-squares fitting of 1D data with a GUI interface to LMFIT
   * create new NeXus-conforming data files without expert knowledge of the file format, and
   * develop new modes of analysis using a built-in Python script editor.

In more detail, NeXpy has the following features :

* Individual HDF5 files or entire directories of files can be loaded into the tree view. Files are loaded ‘lazily’; only the top levels of the file hierarchy are loaded by default and only small datasets are read into memory. This allows many files to be loaded rapidly, even when they are extremely large. Slabs from multidimensional fields are loaded from the files dynamically when they are needed, e.g., in plotting a 2D slice, obviating the need for large system memory. Non-HDF5 data, e.g., TIFF or CBF images, SPEC files, or multicolumn ASCII files, can also be imported and added to the tree as NeXus groups.

* Data of arbitrary dimension can be plotted as 1D or 2D slices in one or more plot windows. Each window has a tabbed interface for manipulating plot limits, switching axes, using linear or log scaling, choosing color maps, etc. Other aspects of the plots, such as marker types, colors, and sizes, grid lines, legends, and axis labelling can be adjusted in each plot's Customize Panel. A number of specialized Matplotlib features are conveniently available within the GUI, such as skew axes, symmetric color maps with automatically symmetrized limits, and various smoothing algorithms for 1D and 2D plots.

* For higher-dimensional data (>2), an additional tab defines the values of the out-of-plane axes, which can be easily adjusted manually or scanned as a movie, with or without autoscaling in each plane.

* A Projection Panel allows 1D or 2D slices and projections of arbitrary extent and thickness to be plotted. This panel has a tabbed interface that allows projection limits from multiple plots to be synchronized and, in the case of 1D projections, over-plotted.

* A Limits Panel allows the plots from multiple data sets to be synchronized, so that changes to the plot limits or even plot axes are automatically applied to all the plot windows. This allows, for example, data taken at multiple temperatures to be compared easily.

* A Scan Panel allows compatible data sets from multiple files to be consolidated into a single array with an extra dimension corresponding to a scanned parameter. If data are taken at multiple temperatures, for example, the n-dimensional arrays in each file could be combined to form a (n+1)-dimensional array, with temperature as the additional axis. This allows the temperature axis to be included as a projection axis in the Projection Panel. The Scan Panel consolidates the arrays using HDF5 Virtual Datasets, so there is no increased cost in either data storage or memory.

* A built-in Script Editor with Python syntax coloring allows repeated operations to be coded or new data analysis methods to be developed and tested in the IPython shell. All data loaded in the NeXpy tree are available for use in these scripts. Additional data can be created or loaded directly from additional files using the nexusformat API.

* All 1D plots have a ‘Fit’ button that launches a GUI interface to the LMFIT package. This allows LMFIT models, e.g., Gaussians, Lorentzians, Voigt functions, etc, to be combined in arbitrary ways to produce a composite model. Additional models can be added to the list of pre-defined models. The default is to add such models together, but NeXpy provides a GUI dialog allowing other operators to be used in combining them. Parameters can be fixed or constrained either with limits or by algebraic functions of other parameters, again customizable through GUI dialogs. Fits can be saved for storage with the NeXus data itself and/or reloaded in order to further refine the models and parameters.

* The values of NeXus fields can be viewed in tabular form, along with the other HDF5 parameters that define them (data type, chunk size, compression, etc).

* A number of keyboard shortcuts accelerate interactions with the plot windows to turn log scaling and grid lines on and off, to set aspect ratios, etc.

* A plugin architecture allows additional menus to be added to NeXpy for specialized applications. The package modules contain a number of PyQt widgets that make it easier to add, e.g, parameter tables and pull-down menus, to specialist GUI dialogs.

PyPI: https://pypi.org/project/NeXpy
conda-forge: https://anaconda.org/conda-forge/nexpy

nexusformat
The underlying API used by NeXpy is available as a separate download from PyPI (https://pypi.org/project/nexusformat/) and conda-forge (https://anaconda.org/conda-forge/nexusformat). This provides an intuitive syntax for creating, reading, and writing HDF5 files that conform to the NeXus standard. As described above, the tree of groups and fields that comprise the hierarchical structure of NeXus files are ‘lazily’ loaded with a single function call, giving instant access to all the data stored in the file as nested dictionaries without loading it all into memory. Data objects anywhere in the file can also be accessed using Posix-style paths as the dictionary keys. The API makes it easy to link fields both within the same file and to external files. The API uses h5py to read and write the HDF5 files, but significantly reduces the code required to implement NeXus design rules, while ensuring compliance with the latest version of the standard.

Tools allow the data to be manipulated and augmented either interactively or using Python scripts. Large multidimensional fields can be created dynamically and filled incrementally as slabs either in core memory files or saved directly to a file on disk. This allows the API to handle very large data sets without requiring significant system memory.

A file locking mechanism is provided to prevent data corruption from simultaneous access by multiple processes. Backups can be created and automatically restored if necessary.

A tutorial on using the nexusformat API is available as a Jupyter Notebook. It can be accessed online in a Google colaboratory:

https://colab.research.google.com/github/nexpy/nexusformat/blob/master/src/nexusformat/notebooks/nexusformat.ipynb

PyPI: https://pypi.org/project/nexusformat
conda-forge: https://<https://anaconda.org/conda-forge/nexusformat>anaconda.org/conda-forge<https://anaconda.org/conda-forge/nexpy>/nexusformat

Installation

NeXpy is available on both the Python Package Index server and Anaconda, using

 $ pip install nexpy

or

 $ conda install -c conda-forge nexpy

Please report any bugs to the Github issues page. Other questions can be addressed to the NeXus Mailing List <nexus at nexusformat.org<mailto:nexus at nexusformat.org>>.

NeXpy Home Page: https://nexpy.github.io/nexpy/

NeXpy Github: https://github.com/nexpy/nexpy
NeXpy Release Notes: https://github.com/nexpy/nexpy/releases
nexusformat Github: https://github.com/nexpy/nexusformat
nexusformat  Release Notes: https://github.com/nexpy/nexpy/releases

Ray Osborn
--
Ray Osborn, Senior Scientist
Materials Science Division
Argonne National Laboratory
Argonne, IL 60439, USA
Phone: +1 (630) 252-9011
Email: ROsborn at anl.gov<mailto:ROsborn at anl.gov>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.nexusformat.org/pipermail/nexus/attachments/20230303/a8d9c222/attachment-0001.htm>