[Nexus] large overhead on small hdf5 files

[Wintersberger, Eugen]

Hi Ray
On Wed, 2014-09-24 at 15:43 +0000, Osborn, Raymond wrote: 
> There’s a StackOverflow issue at http://stackoverflow.com/questions/14332193/hdf5-storage-overhead that discusses this.

This is an interesting issue - never thought about it. I will have a
look on one of our testing Nexus files from beamline P02. 

> 
> If you’re using NeXus and you have a large number of groups, perhaps this is a case where they should be consolidated - can you put all the metadata into a single NXparameters group? Or perhaps a gzipped XML file is the way to go. Or you just resign yourself to buying bigger hard drives. 
> 
> Ray
> 
> On Sep 23, 2014, at 6:05 PM, Paul Kienzle <Paul.Kienzle at nist.gov> wrote:
> 
> > Anyone have any hints for reducing HDF5 file sizes?
> > 
> > Using h5stat, I have 750k of metadata for 75k of raw data.  The data is all small, but with 1300 data items in the file, the overhead adds up.
> > 
> > I’ve been using the h5py python HDF5 wrapper to create the files, but even when copying them using napi they are still huge.
> > 
> > Thanks in advance,
> > 
> >   - Paul
> > 
> > Paul Kienzle
> > paul.kienzle at nist.gov
> > 
> > 
> > _______________________________________________
> > NeXus mailing list
> > NeXus at nexusformat.org
> > http://lists.nexusformat.org/mailman/listinfo/nexus
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 230 bytes
Desc: This is a digitally signed message part
URL: <http://lists.nexusformat.org/pipermail/nexus/attachments/20140926/ce090799/attachment.sig>