[Nexus-developers] Memory use during putslab and getslab

[Ray Osborn]

We are trying to the use the putslab and getslab modules in Paul Kienzle's Python interface in order to avoid memory consumption but observe memory spikes during the operation itself. I'm afraid that I can't tell at this stage whether it's caused by the Python interface or the underlying HDF5 C-interface so I would welcome some comments. 

Basically, I am allocating an SDS with the right type and shape but no values and saving it as a NeXus file. This seems to work - the file is still small and there are no memory spikes.

>>> entry=SDS(name='data', dtype='float32', shape=[200000000])
>>> entry.nxsave('tmp.nxs')
>>> entry.data.nxget(1,2)
array([ 0.,  0.], dtype=float32)

If you are familiar with the Python interface, you will know that a.nxdata is always None, i.e., the data array is never allocated, but the nxget module, which just calls the C getslab routine returns an array filled with zeros, presumably the HDF5 default.

Then I try to write a slab.

>>> slab=np.array((1.0,2.0),dtype='float32')
>>> entry.data.nxput(slab,1)
>>> entry.data.nxget(1,2)
array([ 1.,  2.], dtype=float32)

This time, both the nxput and nxget calls produce a memory spike of over 800MB, the size of the allocated data, before returning to normal.

Does anybody know why the whole array gets allocated temporarily, and is there a way of avoiding this? Obviously, if we are talking about a 50GB array, then this is likely to produce a memory error (although I haven't tested that yet). 

Also, is there a way to make the undefined data get returned as NaNs? Presumably, HDF5 is returning a default value when it finds no stored data, so can that default value be changed, and could this be an option in the NeXus API?

Any comments welcome.

Thanks,
Ray
-- 
Dr Raymond Osborn
Group Leader, Neutron & X-ray Scattering
Materials Science Division
Argonne National Laboratory
Argonne, IL 60439, USA
Phone: +1 (630) 252-9011
Email: ROsborn at anl.gov