[Nexus] Announcing beta-version of the NXvalidate Python package

Wed Sep 18 13:55:35 BST 2024

Hi Jan,
I think some of the issues you identify are deficiencies in the package. If by checking the base class, you believe that a warning message is incorrect, please post this as an issue on Github (https://github.com/nexpy/nxvalidate/issues). If it’s a problem with the standard or you think the standard needs modifying, we should discuss it at the NIAC. Stimulating such discussions is one of the reasons I announced the package here.

I believe the NXroot NXDL file only allows NXentry groups, so I think the package is issuing the correct message. Whether we should modify the NXDL to allow non-NeXus groups as well may be something for the NIAC to consider. Of course, it’s only an error message; it can be ignored.

The axes validation requires a little work but I’m not sure I understand your statement that “if axes attribute contain a single string it is separated to single letters.” I don’t think that has ever been true. We have allowed single strings containing delimiters such as ‘:’ to separate the axes, although lists of strings are now preferred.

I should probably downgrade some of the other warning messages to be informational messages. In fact, I confess I hadn’t spotted the “ignoreExtraFields” and related attributes in the NXDL definition header. I had added a few groups, such as NXcollection and NXdata, manually to a list where additional fields were allowed, but I hadn’t included NXtransformations. I now see that the header attributes would allow me to do that automatically. The axes validation requires a little work, and I also don’t handle placeholder name patterns, such as FIELDNAME_errors, or symbols, which are used mostly in dimension statements.

So this is a work in progress and feedback such as this is appreciated.

With regards,
Ray
--
Ray Osborn, Senior Scientist
Materials Science Division
Argonne National Laboratory
Lemont, IL 60439, USA
Phone: +1 (630) 252-9011
Email: ROsborn at anl.gov

From: Kotanski, Jan <jan.kotanski at desy.de>
Date: Wednesday, September 18, 2024 at 12:42 AM
To: Discussion forum for the NeXus data format <nexus at nexusformat.org>
Cc: Osborn, Raymond <rosborn at anl.gov>
Subject: Re: [Nexus] Announcing beta-version of the NXvalidate Python package
Dear Ray, thank you for providing the tool. I've just run the nxinspect on one of my file and I get unexpected warning i. e. - This field is not defined in the base class NXtransformations (but below NXtransformations we can create a field with
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.

ZjQcmQRYFpfptBannerEnd
Dear Ray,

thank you for providing the tool.

I've just run the nxinspect on one of my file and I get unexpected warning i.e.

- This field is not defined in the base class NXtransformations
    (but below NXtransformations we can create a field with any name)

-  Axis ." not present in group "/scan/data"
    (I thought '.'   one of the possible values i.e.
      'The NXdata group attribute axes = . . declares that both dimensions of the data field do not
     have associated dimension scales (plotting routines should use integer scaling for each axis).' )

- Axis e" not present in group "/scan/data"
   (if axes attribute contain a single string it is separated to single letters)

- NXcollection is an invalid class in NXcollection
   (however in the documentation it is written that  'Anything (groups, fields, or attributes) placed in an NXcollection<https://urldefense.us/v3/__https:/manual.nexusformat.org/classes/base_classes/NXcollection.html*nxcollection__;Iw!!G_uCfscf7eWS!Yc3QrTHscDtsRbsZ8VdtJNChFi2w9o-rYY6PCL8kgeorvry5Yw0Hc9EeUC_bmMw-idwAaI2c2xMYdgtUOduCsQ$> group will not be validated.'
    and NXcollection is a group so nesting of NXcollection should be allowed)

 - NXcollection is an invalid class in NXroot
    Is the NXcollection group really forbidden below NXroot? I saw some hdf5 multi-format files where NXcollection was used on the root level
    to denote non-NeXus (meta)data. If NXcollection below in not allowed I would like to ask if it is possible to create HDF5 multiformat files
    and how they should be create to be valid NeXus file?

Best regards,
Jan

________________________________
From: "Discussion forum for the NeXus data format" <nexus at shadow.nd.rl.ac.uk>
To: "Discussion forum for the NeXus data format" <nexus at shadow.isis.cclrc.ac.uk>, "Discussion forum for the NeXus data format" <nexus at nexusformat.org>
Cc: "Osborn, Raymond" <rosborn at anl.gov>
Sent: Tuesday, 17 September, 2024 15:03:22
Subject: Re: [Nexus] Announcing beta-version of the NXvalidate Python package

Hi Jens,
Thanks for the suggestion. Code maintenance is always an issue, but I had planned to issue a new release whenever there was an official NeXus release, which is not that frequent. I’m reluctant to complicate the installation process, but I think we could achieve what you are requesting by allowing an option to point to another directory containing the definitions. That directory, for example, could be part of a cloned nexusformat git repository that the user can keep updated. I should be able to implement that in the next few days.

With regards,
Ray
--
Ray Osborn, Senior Scientist
Materials Science Division
Argonne National Laboratory
Lemont, IL 60439, USA
Phone: +1 (630) 252-9011
Email: ROsborn at anl.gov

From: NeXus <nexus-bounces at shadow.nd.rl.ac.uk> on behalf of Jens Krüger via NeXus <nexus at shadow.nd.rl.ac.uk>
Date: Tuesday, September 17, 2024 at 1:29 AM
To: nexus at nexusformat.org <nexus at nexusformat.org>
Cc: Jens Krüger <Jens.Krueger at frm2.tum.de>
Subject: Re: [Nexus] Announcing beta-version of the NXvalidate Python package
Hi Ray, thanks for the information about the nxinspect tool and the effort you put into this project. I have a question: Wouldn't it make sense to download all the definitions from the nexusformat. org during the installation and use them locally
ZjQcmQRYFpfptBannerStart
This Message Is From an External Sender
This message came from outside your organization.

ZjQcmQRYFpfptBannerEnd

Hi Ray,

thanks for the information about the nxinspect tool and the effort you

put into this project.

I have a question: Wouldn't it make sense to download all the

definitions from the nexusformat.org

during the installation and use them locally instead of put them into

the repo? I'm afraid,

that the sync between nexpy and nexusformat repos could be a big

challenge in the future.

Best regards

Jens

Am 16.09.24 um 18:04 schrieb Osborn, Raymond via NeXus:

> I am happy to announce that we have just issued the first beta-release of NXvalidate (https://urldefense.us/v3/__https://github.com/nexpy/nxvalidate__;!!G_uCfscf7eWS!dBKs6J3MTRu8kT6R8Vfk1ILUz-I_1i71_Vh7ySBK_KpCdajICcg6uNBz-EHhHtwoin6wbp59N6si9gtziFrBLA$<https://urldefense.us/v3/__https:/github.com/nexpy/nxvalidate__;!!G_uCfscf7eWS!dBKs6J3MTRu8kT6R8Vfk1ILUz-I_1i71_Vh7ySBK_KpCdajICcg6uNBz-EHhHtwoin6wbp59N6si9gtziFrBLA$>), a new Python package used to inspect and validate NeXus files. It is currently available for installation on the PyPI server and the source code can be downloaded from Github. When it is more mature, it will be merged into the nexusformat package, with a GUI interface in NeXpy (https://urldefense.us/v3/__https://nexpy.github.io/nexpy/__;!!G_uCfscf7eWS!dBKs6J3MTRu8kT6R8Vfk1ILUz-I_1i71_Vh7ySBK_KpCdajICcg6uNBz-EHhHtwoin6wbp59N6si9gv42bvToA$<https://urldefense.us/v3/__https:/nexpy.github.io/nexpy/__;!!G_uCfscf7eWS!dBKs6J3MTRu8kT6R8Vfk1ILUz-I_1i71_Vh7ySBK_KpCdajICcg6uNBz-EHhHtwoin6wbp59N6si9gv42bvToA$>). I hope to discuss this at the upcoming meeting of the NeXus International Advisory Committee and would welcome any feedback if you try it out before then. Please note that this is not a production release. Please post any issues to https://urldefense.us/v3/__https://github.com/nexpy/nxvalidate/issues__;!!G_uCfscf7eWS!dBKs6J3MTRu8kT6R8Vfk1ILUz-I_1i71_Vh7ySBK_KpCdajICcg6uNBz-EHhHtwoin6wbp59N6si9gsrqLSP2Q$<https://urldefense.us/v3/__https:/github.com/nexpy/nxvalidate/issues__;!!G_uCfscf7eWS!dBKs6J3MTRu8kT6R8Vfk1ILUz-I_1i71_Vh7ySBK_KpCdajICcg6uNBz-EHhHtwoin6wbp59N6si9gsrqLSP2Q$>.

>

> Installation

> Released versions of nxvalidate can be installed using “pip install nxvalidate” and the source code can be downloaded from the NeXpy Git repository:

>

>      % git clone https://urldefense.us/v3/__http://github.com/nexpy/nxvalidate.git__;!!G_uCfscf7eWS!dBKs6J3MTRu8kT6R8Vfk1ILUz-I_1i71_Vh7ySBK_KpCdajICcg6uNBz-EHhHtwoin6wbp59N6si9gtz4Uru8A$<https://urldefense.us/v3/__http:/github.com/nexpy/nxvalidate.git__;!!G_uCfscf7eWS!dBKs6J3MTRu8kT6R8Vfk1ILUz-I_1i71_Vh7ySBK_KpCdajICcg6uNBz-EHhHtwoin6wbp59N6si9gtz4Uru8A$>

>

> Usage

> The nxvalidate package provides a single command-line script.

>

> % nxinspect -h

> usage: nxinspect [-h] [-f FILENAME] [-p PATH] [-a [APPLICATION]]

>                   [-b BASECLASS] [-i] [-w] [-e] [-v]

>

> Inspects and validates NeXus files.

>

> options:

>    -h, --help            show this help message and exit

>    -f FILENAME, --filename FILENAME

>                          name of the NeXus file to be validated

>    -p PATH, --path PATH

>                          path to group to be validated in the NeXus file

>    -b BASECLASS, --baseclass BASECLASS

>                          name of the base class to be listed

>    -a [APPLICATION], --application [APPLICATION]

>                          validate the NeXus file against its application definition

>    -i, --info            output info messages in addition to warnings and errors

>    -w, --warning         output info messages in addition to warnings and errors

>    -e, --error           output info messages in addition to warnings and errors

>    -v, --version         show program's version number and exit

>

> N.B., the command is “nxinspect,” rather than “nxvalidate” to avoid confusion with the existing “cnxvalidate” application (https://urldefense.us/v3/__https://github.com/nexusformat/cnxvalidate__;!!G_uCfscf7eWS!dBKs6J3MTRu8kT6R8Vfk1ILUz-I_1i71_Vh7ySBK_KpCdajICcg6uNBz-EHhHtwoin6wbp59N6si9gtoRsqAFg$<https://urldefense.us/v3/__https:/github.com/nexusformat/cnxvalidate__;!!G_uCfscf7eWS!dBKs6J3MTRu8kT6R8Vfk1ILUz-I_1i71_Vh7ySBK_KpCdajICcg6uNBz-EHhHtwoin6wbp59N6si9gtoRsqAFg$>).

>

> Examples

>

>    1.  To compare the contents of a NeXus file with the base classes defined by the NeXus standard and print conflicting fields or groups, type:

>

> % nxinspect -f <filename.nxs> -e

>

> The --info, --warning and --error switches control how much information is output. The default is --warning.

>

>    2.  To check whether the contents of the NeXus file conform to the required contents of the application definition specified in the file, type:

>

> % nxinspect -f <filename.nxs> -a

>

>    3.  To check whether the contents of the NeXus file conform to the required contents of an application definition file, type:

>

> % nxinspect -f <filename.nxs> -a <application.nxdl.xml>

>

>    4.  To print the contents of a base class, type:

>

> % nxinspect -b <base-class-name>

>

> With regards,

> Ray Osborn

> --

> Ray Osborn, Senior Scientist

> Materials Science Division

> Argonne National Laboratory

> Lemont, IL 60439, USA

> Phone: +1 (630) 252-9011

> Email: ROsborn at anl.gov

>

>

>

>

>

> _______________________________________________

> NeXus mailing list

> NeXus at nexusformat.org

> https://urldefense.us/v3/__https://lists.nexusformat.org/mailman/listinfo/nexus__;!!G_uCfscf7eWS!dBKs6J3MTRu8kT6R8Vfk1ILUz-I_1i71_Vh7ySBK_KpCdajICcg6uNBz-EHhHtwoin6wbp59N6si9guwW37CUg$<https://urldefense.us/v3/__https:/lists.nexusformat.org/mailman/listinfo/nexus__;!!G_uCfscf7eWS!dBKs6J3MTRu8kT6R8Vfk1ILUz-I_1i71_Vh7ySBK_KpCdajICcg6uNBz-EHhHtwoin6wbp59N6si9guwW37CUg$>

--

Dipl.-Phys. Jens Krüger

Leiter Gruppe Instrumentsteuerung

Head of Instrument control group

Technische Universität München

Forschungsneutronenquelle

Heinz Maier-Leibnitz (FRM II)

Lichtenberg-Str. 1

D-85748 Garching

Tel: + 49 (0)89 289 14716

Fax: + 49 (0)89 289 14620

mailto: jens.krueger at frm2.tum.de

https://urldefense.us/v3/__https://www.frm2.tum.de__;!!G_uCfscf7eWS!dBKs6J3MTRu8kT6R8Vfk1ILUz-I_1i71_Vh7ySBK_KpCdajICcg6uNBz-EHhHtwoin6wbp59N6si9gu-04bayA$<https://urldefense.us/v3/__https:/www.frm2.tum.de__;!!G_uCfscf7eWS!dBKs6J3MTRu8kT6R8Vfk1ILUz-I_1i71_Vh7ySBK_KpCdajICcg6uNBz-EHhHtwoin6wbp59N6si9gu-04bayA$>

https://urldefense.us/v3/__https://www.mlz-garching.de__;!!G_uCfscf7eWS!dBKs6J3MTRu8kT6R8Vfk1ILUz-I_1i71_Vh7ySBK_KpCdajICcg6uNBz-EHhHtwoin6wbp59N6si9gtT85dm6Q$<https://urldefense.us/v3/__https:/www.mlz-garching.de__;!!G_uCfscf7eWS!dBKs6J3MTRu8kT6R8Vfk1ILUz-I_1i71_Vh7ySBK_KpCdajICcg6uNBz-EHhHtwoin6wbp59N6si9gtT85dm6Q$>

_______________________________________________

NeXus mailing list

NeXus at nexusformat.org

https://urldefense.us/v3/__https://lists.nexusformat.org/mailman/listinfo/nexus__;!!G_uCfscf7eWS!dBKs6J3MTRu8kT6R8Vfk1ILUz-I_1i71_Vh7ySBK_KpCdajICcg6uNBz-EHhHtwoin6wbp59N6si9guwW37CUg$<https://urldefense.us/v3/__https:/lists.nexusformat.org/mailman/listinfo/nexus__;!!G_uCfscf7eWS!dBKs6J3MTRu8kT6R8Vfk1ILUz-I_1i71_Vh7ySBK_KpCdajICcg6uNBz-EHhHtwoin6wbp59N6si9guwW37CUg$>

_______________________________________________
NeXus mailing list
NeXus at nexusformat.org
https://lists.nexusformat.org/mailman/listinfo/nexus<https://urldefense.us/v3/__https:/lists.nexusformat.org/mailman/listinfo/nexus__;!!G_uCfscf7eWS!Yc3QrTHscDtsRbsZ8VdtJNChFi2w9o-rYY6PCL8kgeorvry5Yw0Hc9EeUC_bmMw-idwAaI2c2xMYdguZTaVYdg$>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.nexusformat.org/pipermail/nexus/attachments/20240918/04f9636d/attachment-0001.htm>