[Nexus] Slow writing in HDF4 and HDF5 with many NXdata blocks

[Mark Koennecke]

Hi Emmanuel,

On 04/13/2012 03:41 PM, Emmanuel FARHI wrote:
> Dear NeXus users,
>
> I'm creating HDF files with many NXdata blocks (about 850).

I wonder if this a good approach. I would rather append the data to one 
big array
with an extendible dimension.
> I thus open the file, create a group, and then write iteratively the 
> blocks with e.g.:
>
>     NXhandle pHandle;
>     NXopen(filename, NXACC_CREATE, &pHandle);
>
>     ...
>
>     float *data; /* n*m elements */
>     int length[2]={n,m};
>
>     ...
>
>     NXcompmakedata(pHandle, name, NX_FLOAT32, 2, length, NX_COMP_LZW,
>     length);
>     NXopendata(pHandle, name);
>     NXputdata(pHandle, data);
>     NXclosedata(pHandle);
>
> This opens an HDF4 file, which is written in 2 seconds. If I do the 
> same with an HDF5 file (NXACC_CREATE5), then it takes 37 seconds.
> Any clue why HDF5 is so slow with many blocks ? The first few hundred 
> blocks are as fast to write as with HDF4, but then the program lags... 
> The behaviour is the same if I do not write compressed data blocks 
> (NXmakedata). I do not know if this comes from the NeXus library, or 
> from the HDF libraries.
>
NeXus is only a thin layer on top of HDF-5. This presumably comes from 
the size of the HDF-5 buffer.
If this is to small, HDF-5 is forced to make many small writes to disk 
which slows it down. Try setting
the global variable nx_cacheSize to a larger value. The default is 
1024000, which is 1MB.

Generally, HDF is good at writing large chunks of data; generating file 
structure always takes time.

> As a conclusion, I actually tend to recommend to use HDF4 files if 
> your files are smaller than 2Gb...
>
Regards,

           Mark Koennecke

> Emmanuel.
>
> -- 
> Emmanuel FARHI,www.ill.eu/computing/people/emmanuel-farhi  \|/ ____ \|/
> CS-Group ILL4/221, Institut Laue-Langevin (ILL) Grenoble  ~@-/ oO \-@~
> 6 rue J. Horowitz, BP 156, 38042 Grenoble Cedex 9,France  /_( \__/ )_\
> Work :Tel (33/0) 4 76 20 71 35. Fax (33/0) 4 76 48 39 06     \__U_/
>
>
> _______________________________________________
> NeXus mailing list
> NeXus at nexusformat.org
> http://lists.nexusformat.org/mailman/listinfo/nexus