[Nexus-developers] PyTree: NXfield should give morere control over compression

[Carlos Pascual]

Hello,

I see that the NXfield object "automagically" chooses compression if the set is 
large (checking if np.prod(shape) > 10000).

This has two problems for me:
a) It will never compress an "extendable" field, regardless of the dimnension. 
(e.g. consider a field with shape=[nxs.UNLIMITED, 1000,1000]). Note that 
nxs.UNLIMITED=-1

b) The automatic choice of the slab_dims ("chunk size" in the napi lingo) may 
not always be the optimal one. For example, if I am writing data from a 
800x600 camera during a 100 points scan, my field will have shape [100,800,600] 
and I want the slab_dims to be [1,800,600]since all I/O is naturally done one 
image at a time. The current implementation of NXfield would choose [1,1,600] 
instead, which is less performant.

To fix these two issues, I propose to add a slab_dims property to the NXfield 
object. If this property is not set (or set to None), everything stays as 
before. But if the user sets it, it is used when calling to the write() method 
of NXfield.

Please find attached my implementation. I did it as a derived class but I 
suggest to modify the code of NXfield itself, except maybe for the constructor 
part (which would make it easier to use, but it also would add another 
reserved keyword)

Cheers,

Carlos

-- 
+----------------------------------------------------+
 Carlos Pascual Izarra
 Scientific Software Coordinator
 Computing Division
 Cells / Alba Synchrotron  [http:/www.cells.es]
 Carretera BP 1413 de Cerdanyola-Sant Cugat, Km. 3.3
 E-08290 Cerdanyola del Valles (Barcelona), Spain
 E-mail: cpascual at cells.es
 Phone: +34 93 592 4428
+----------------------------------------------------+
-------------- next part --------------
A non-text attachment was scrubbed...
Name: nxfieldcomp.py
Type: text/x-python
Size: 2278 bytes
Desc: not available
URL: <http://lists.nexusformat.org/pipermail/nexus-developers/attachments/20120910/688c9ebe/attachment.py>