[Nexus] Nexpy

Mon Jun 19 01:55:56 BST 2017

Hi Steve,
Sorry to take a while to reply, but enabling what you are asking for was one of my goals in NeXpy, but I found a few deficiencies in the way that the nexusformat API identifies plottable data, which prevented them working with your file.

To show how it should work, I am attaching an example of a SPEC scan, which was imported to a NeXus file using Pete Jemian’s 'spec2nexus package', which I highly recommend.  NeXpy uses it if it is installed. If you read it into NeXpy, the ‘scan_8’ NXentry contains an NXdata group that contains all the columns of a typical SPEC scan. If you double-click on the group itself, it plots the default signal against the default axis, but if you double-click on one of the other fields in the group, it will allow you to select a second field as the axis, so you could plot, e.g., ‘ic0’ against ‘Epoch’.

It’s almost as simple to do this from the command-line, but cumbersome to type, even with tab-completion.

>>> data = NXdata(spec_example.scan_8.data.ic0, spec_example.scan_8.data.Epoch)
>>> data.plot()

In other words, you just create a temporary NXdata group, containing a signal and axis, and use its plot method.

In the case of your example file, it failed partly because of a design flaw in nexusformat that affects links used in this context, and partly because your fields all have an ‘axis’ attribute set to ‘1’. This used to be a way of identifying plot axes, but it is now deprecated, and in this case, prevented the identification of the correct axis. If these attributes are not needed for other reasons, I would recommend deleting them.

You can get the plot you want in your example file from the command line if you access the parent data, not the links :

>>> f=nxload('633777.nxs')
>>> g=f.entry1.instrument
>>> NXdata(g.ic1monitor.ic1monitor, g.idgap.idgap).plot()

I’m attaching the resulting plot.

I have got a version that fixes this, and hope to release it soon.

With best regards,
Ray

On Jun 16, 2017, at 3:56 AM, steve.collins at diamond.ac.uk<mailto:steve.collins at diamond.ac.uk> wrote:

Sorry to be an idiot, but is there a more compact way to make a plot than this:

figure();plt.plot(array(n.entry1.default.idgap+0), array(n.entry1.default.ic1monitor+0))

I realize that we probably have some attributes missing that facilitate the Nexpy default plotting, but Python seems also to need a bit of help with type conversions.

Cheers,
Steve

________________________________
From: NeXus [nexus-bounces at nexusformat.org<mailto:nexus-bounces at nexusformat.org>] on behalf of Osborn, Raymond [rosborn at anl.gov<mailto:rosborn at anl.gov>]
Sent: 16 June 2017 03:25
To: Discussion forum for the NeXus data format
Subject: Re: [Nexus] Nexpy

Hi Steve,
I have updated nexusformat package to v0.4.8, which should be available on both PyPI and Anaconda (conda update -c nexpy nexusformat). I have checked that it reads your target attributes correctly.

Although Pete says that your file is technically valid (and I trust him to be right on such things), I think it would cause fewer problems if you could write the attributes as scalar strings. For example, if I open your file in h5py and read your ‘target’ attribute, I get:

target = h5_file['/entry1/default/Time'].attrs['target'][()]
target
array(['/entry1/instrument/Waittime/Time'],
     dtype='|S33’)

I can only use this as a string within my Python shell either by converting it to a scalar, which is what nexusformat now does, or accessing the first element of the array:

np.asscalar(target)
'/entry1/instrument/Waittime/Time'
target[0]
'/entry1/instrument/Waittime/Time’

I’m not sure how other languages would handle this, so perhaps this is just a Python issue.

By the way, NeXus (and the nexusformat package) do allow attributes to contain arrays. If you look at the chopper example, the ‘axes’ attribute of ‘chopper.entry.data' is an array of variable length strings (with a special h5py object dtype), which can be used in list comprehensions, etc:

chopper.entry.data.axes
array(['polar_angle', 'time_of_flight'], dtype=object)

However, size 1 arrays are now interpreted as scalars.

With best regards,
Ray

On Jun 15, 2017, at 11:50 AM, steve.collins at diamond.ac.uk<mailto:steve.collins at diamond.ac.uk><mailto:steve.collins at diamond.ac.uk> wrote:

Great – thanks People,
I’ve tried to follow the thread…
I understand that (a) my nxs file is valid and (b) the new version of nexusformat will solve the problem.
Cheers!
Steve

From: NeXus [mailto:nexus-bounces at nexusformat.org] On Behalf Of Osborn, Raymond
Sent: 15 June 2017 13:25
To: V. Armando Solé <sole at esrf.fr<mailto:sole at esrf.fr><mailto:sole at esrf.fr>>
Cc: Mita, Charles (DLSLtd,RAL,LSCI) <charles.mita at diamond.ac.uk<mailto:charles.mita at diamond.ac.uk><mailto:charles.mita at diamond.ac.uk>>; Discussion forum for the NeXus data format <nexus at nexusformat.org<mailto:nexus at nexusformat.org><mailto:nexus at nexusformat.org>>; Sharpe, Chris (DLSLtd,RAL,LSCI) <chris.sharpe at diamond.ac.uk<mailto:chris.sharpe at diamond.ac.uk><mailto:chris.sharpe at diamond.ac.uk>>
Subject: Re: [Nexus] Nexpy

Thanks, Armando,
That is exactly the aim of the nexusformat package - to give scientists a convenient and, hopefully, reliable tool for reading and writing NeXus files that just works. Speaking as a scientist, we don’t want to worry about whether they should be using variable-length strings or byte arrays - we just want to have variables we can manipulate in Python and store in our data files when required. We don’t want to worry about whether we have obeyed the standard, but would be delighted if the program sorted that out for us. That is what nexusformat tries to do, and NeXpy adds other conveniences, like showing visually what items are linked in a file tree.

Those who spend a lot of time developing software, on the other hand, are welcome to do their own thing. If you are happy writing your own read/write modules using, e.g., h5py, then that’s fine. But please don’t expect scientists to have to learn h5py, then study all the NeXus rules for writing NXdata groups (not forgetting to add an ’NX_class’ attribute, a signal attribute, axes, …), and then worry about whether they should be using byte arrays or strings.

This is a philosophical issue that I have brought up many times at NIAC meetings. The original goal of NeXus was to be a convenience tool for scientists, but it is now largely maintained by developers who are willing to read manuals. I was a little reluctant about deprecating the NAPI for this reason, but it started to become a barrier to using the more advanced features of HDF5 and few scientists write in C anyway. Python is different - scientists will use it if we give them the right modules - I have a student with little programming experience who writes standard-conforming NeXus files every day. My suggestion that more people use the nexusformat package, rather than just h5py, is that, as this thread shows, it is helpful to have a wide range of people helping to debug it so that it doesn’t trip scientists up when they do use it.

With best regards,
Ray

On Jun 15, 2017, at 3:21 AM, V. Armando Solé <sole at esrf.fr<mailto:sole at esrf.fr><mailto:sole at esrf.fr>> wrote:

Hello,

On 15/06/2017 09:17, Carlos Pascual wrote:

And, on the other hand, if the NIAC does bless nexusformat as the preferred
access method for python, I think it should be stated unambiguously in the
manual.

Well, at the ESRF Python programmers are using h5py.

However we recommend scientists to use nexusformat as a simpler tool
because they often work with interactive sessions.
As you see our point of view is that nexusformat is just a convenience
tool on top of h5py being the later the standard (Python) tool to access
HDF5 files.

What we do not recommend at all is to use the NeXus library from Python
(or from anywhere else).

The use of NAPI was unambiguously discouraged by the NIAC in past NIAC
meetings in favour of standard tools. However I never understood why
trainings were kept going on around an API that was considered obsolete.

Best,

Armando

--
Ray Osborn, Senior Scientist
Materials Science Division
Argonne National Laboratory
Argonne, IL 60439, USA
Phone: +1 (630) 252-9011
Email: ROsborn at anl.gov<mailto:ROsborn at anl.gov><mailto:ROsborn at anl.gov>

--

This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

_______________________________________________
NeXus mailing list
NeXus at nexusformat.org<mailto:NeXus at nexusformat.org><mailto:NeXus at nexusformat.org>
http://lists.nexusformat.org/mailman/listinfo/nexus

--
Ray Osborn, Senior Scientist
Materials Science Division
Argonne National Laboratory
Argonne, IL 60439, USA
Phone: +1 (630) 252-9011
Email: ROsborn at anl.gov<mailto:ROsborn at anl.gov><mailto:ROsborn at anl.gov>

_______________________________________________
NeXus mailing list
NeXus at nexusformat.org<mailto:NeXus at nexusformat.org>
http://lists.nexusformat.org/mailman/listinfo/nexus

--
Ray Osborn, Senior Scientist
Materials Science Division
Argonne National Laboratory
Argonne, IL 60439, USA
Phone: +1 (630) 252-9011
Email: ROsborn at anl.gov<mailto:ROsborn at anl.gov>

[cid:968e8a29-6b93-4181-8135-400c1c19b9c2 at localhost]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.nexusformat.org/pipermail/nexus/attachments/20170619/e2dec879/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: example.png
Type: image/png
Size: 20460 bytes
Desc: example.png
URL: <http://lists.nexusformat.org/pipermail/nexus/attachments/20170619/e2dec879/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: spec_example.nxs
Type: application/octet-stream
Size: 28696 bytes
Desc: spec_example.nxs
URL: <http://lists.nexusformat.org/pipermail/nexus/attachments/20170619/e2dec879/attachment-0001.obj>