[Nexus-developers] Memory use during putslab and getslab

[Ray Osborn]

Mark managed to hit the nail on the head. I did the same tests as before, but on a [1000000,200]-size array so that the chunk size should have been set to 200. I was able to use both nxput and nxget (and the array indices, which also use nxget) without noticeably increasing the memory. I am still concerned about how we stop the user from making a costly mistake, which could freeze the python session (which I have succeeded in doing), but that's another issue.

As a follow-up, it's worth noting that this won't work in the current version of tree.py in the NeXus repository because it doesn't initialize the _file and _path variables required by nxput. I have a version which does that whenever you do save the data (even if there's nothing in it). We'll need to merge these changes before the next release.

With best regards,
Ray

On Mar 17, 2011, at 8:54 AM, Ray Osborn wrote:

> Hi Mark,
> 
> On Mar 17, 2011, at 3:31 AM, Mark Koennecke wrote:
> 
>> I am missing one important morsel of information: when creating the array, what do you specify as the chunksize?
>> The recommendation is to set the chunksize to approximately  the size of the slabs you are going to read and write.
>> If the whole dataset is just one chunk this would explain your observations.
> 
> Thanks for the tip. The chunk size is set automatically in tree.py to the last dimension, when it calls compmakedata. I was testing it with only one dimension, so that may be it. I will test it with multidimensional arrays, which is the intended use anyway.
> 
> I'll get back to you when I've done more tests.
> 
>>> Also, is there a way to make the undefined data get returned as NaNs? Presumably, HDF5 is returning a default value when it finds no stored data, so can that default value be changed, and could this be an option in the NeXus API?
>>> 
>>> 
>> We have to look into this...
> 
> I think it would be very useful because I presume otherwise there is no way of knowing what the meaning of the zero is. Does HDF5 automatically fill a chunk with zeros? 
> 
> Ray
> -- 
> Dr Raymond Osborn
> Group Leader, Neutron & X-ray Scattering
> Materials Science Division
> Argonne National Laboratory
> Argonne, IL 60439, USA
> Phone: +1 (630) 252-9011
> Email: ROsborn at anl.gov
> 
> 
> 
> 
> 
> _______________________________________________
> NeXus-developers mailing list
> NeXus-developers at nexusformat.org
> http://lists.nexusformat.org/mailman/listinfo/nexus-developers

-- 
Dr Raymond Osborn
Group Leader, Neutron & X-ray Scattering
Materials Science Division
Argonne National Laboratory
Argonne, IL 60439, USA
Phone: +1 (630) 252-9011
Email: ROsborn at anl.gov