[Nexus] large overhead on small hdf5 files

[Osborn, Raymond]

There’s a StackOverflow issue at http://stackoverflow.com/questions/14332193/hdf5-storage-overhead that discusses this.

If you’re using NeXus and you have a large number of groups, perhaps this is a case where they should be consolidated - can you put all the metadata into a single NXparameters group? Or perhaps a gzipped XML file is the way to go. Or you just resign yourself to buying bigger hard drives. 

Ray

On Sep 23, 2014, at 6:05 PM, Paul Kienzle <Paul.Kienzle at nist.gov> wrote:

> Anyone have any hints for reducing HDF5 file sizes?
> 
> Using h5stat, I have 750k of metadata for 75k of raw data.  The data is all small, but with 1300 data items in the file, the overhead adds up.
> 
> I’ve been using the h5py python HDF5 wrapper to create the files, but even when copying them using napi they are still huge.
> 
> Thanks in advance,
> 
>   - Paul
> 
> Paul Kienzle
> paul.kienzle at nist.gov
> 
> 
> _______________________________________________
> NeXus mailing list
> NeXus at nexusformat.org
> http://lists.nexusformat.org/mailman/listinfo/nexus

-- 
Ray Osborn, Senior Scientist
Materials Science Division
Argonne National Laboratory
Argonne, IL 60439, USA
Phone: +1 (630) 252-9011
Email: ROsborn at anl.gov