Data compression vrs Tiling

[Paul Langan]

Dear colleagues,

        while compression is still topical, I thought I'd share with you some
results of tests on compression vrs tiling which reminded me of something I'd
forgotten: the more tiles you have the less the gains in the first level of
compression (I assume everyone will use double compression to save on archive
space and remove the second level of compression only when processing NeXus
files). 

I generated NeXus files with different numbers of equally sized tiles (SDS's).
The size of the file after the second level of compression depended very little
on the first level of compression. I used a TOF neutron single crystal Ruby
data 80x80x150 unsigned short integers for the tests.

                File Size in MB after first level of compression

Number of Tiles         1       2       4       8       
LZW                     .257    1.12    1.51    1.70
SH                      .366    .934    1.05    1.47    
RLE                     1.03    .938    1.40    1.65    
no comp'                1.87    1.88    1.89    1.9


                File Size in MB after second level of compression

Number of Tiles 1       2       4       8       
whole file gzipped      .213    .361    .359    .357

Despite these results there are very large advantages in using tiling (as David
Love first suggested to me) when handling very large files (some of ours will
be ~0.8GB) on PC's with limited memory and with a desire for fast processing
(faster IO to disk). 

One other thing related to my ignorance of programming. I could not use the
1.3.1 API untill I had edited the header file to select the appropriate calling
convention for my PC (winNT C++)
#       define MANGLE(__arg)            CONCAT(__arg,_)
Is this because I'm not using the API correctly or just becuase I didn't read
the installation note properly?

And while I have your attention, my Fortner Browser crasheswhen I try to browse
files with many compressed SDS's. Has anyone else found this?



Paul Langan
B-Division
MS M888
Los Alamos National Laboratory
Los Alamos NM87545
USA

Tel: 505 665 8125
Fax: 505 665 3024
email: langan_paul at lanl.gov
http://lsdiv.lanl.gov/nsb/ 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.nexusformat.org/pipermail/nexus/attachments/20000225/a0ab5f7a/attachment.html