[Nexus-developers] Memory use during putslab and getslab

[Pedro Vicente]

Ray

I may be able to help you with that.
By coincidence I am doing some work regarding NeXus.putSlab for SNS.
It would be helpful if you have some small program that reproduces that 
behavior (ideally in C/C++)
You can send more details to my email

----------------------
Pedro Vicente
pedro.vicente at space-research.org
http://www.space-research.org/


----- Original Message ----- 
From: "Ray Osborn" <rosborn at anl.gov>
To: "NeXus Developers" <nexus-developers at nexusformat.org>
Sent: Wednesday, March 16, 2011 11:55 PM
Subject: [Nexus-developers] Memory use during putslab and getslab


> We are trying to the use the putslab and getslab modules in Paul Kienzle's 
> Python interface in order to avoid memory consumption but observe memory 
> spikes during the operation itself. I'm afraid that I can't tell at this 
> stage whether it's caused by the Python interface or the underlying HDF5 
> C-interface so I would welcome some comments.
>
> Basically, I am allocating an SDS with the right type and shape but no 
> values and saving it as a NeXus file. This seems to work - the file is 
> still small and there are no memory spikes.
>
>>>> entry=SDS(name='data', dtype='float32', shape=[200000000])
>>>> entry.nxsave('tmp.nxs')
>>>> entry.data.nxget(1,2)
> array([ 0.,  0.], dtype=float32)
>
> If you are familiar with the Python interface, you will know that a.nxdata 
> is always None, i.e., the data array is never allocated, but the nxget 
> module, which just calls the C getslab routine returns an array filled 
> with zeros, presumably the HDF5 default.
>
> Then I try to write a slab.
>
>>>> slab=np.array((1.0,2.0),dtype='float32')
>>>> entry.data.nxput(slab,1)
>>>> entry.data.nxget(1,2)
> array([ 1.,  2.], dtype=float32)
>
> This time, both the nxput and nxget calls produce a memory spike of over 
> 800MB, the size of the allocated data, before returning to normal.
>
> Does anybody know why the whole array gets allocated temporarily, and is 
> there a way of avoiding this? Obviously, if we are talking about a 50GB 
> array, then this is likely to produce a memory error (although I haven't 
> tested that yet).
>
> Also, is there a way to make the undefined data get returned as NaNs? 
> Presumably, HDF5 is returning a default value when it finds no stored 
> data, so can that default value be changed, and could this be an option in 
> the NeXus API?
>
> Any comments welcome.
>
> Thanks,
> Ray
> -- 
> Dr Raymond Osborn
> Group Leader, Neutron & X-ray Scattering
> Materials Science Division
> Argonne National Laboratory
> Argonne, IL 60439, USA
> Phone: +1 (630) 252-9011
> Email: ROsborn at anl.gov
>
>
>
>
>
> _______________________________________________
> NeXus-developers mailing list
> NeXus-developers at nexusformat.org
> http://lists.nexusformat.org/mailman/listinfo/nexus-developers
>
>