Release of NeXus v1.3.0 with data compression

[Mark Koennecke]

On Wed, 15 Dec 1999, Brian Tieman wrote:

> I took some time to test the various data compression schemes on some typical
> data we collect here.  The preliminary results were both very exciting and
> disappointing.  I haven't been able to test an active system yet so I don't yet
> know how dissappointing the results are.  Below is a description of what I found
> if anyone cares to read it.

Thanks for taking the time for such a thorough test!

> 
> Set1
> Compression Type     |    time for 100 files   |     % origional file size
> None                         |            4s                 |              1.00
> 
> LZW                         |            35s                |              0.18
> HUF                          |            17s               |              0.19
> RLE                          |             33s               |              0.50
> 
> Set2
> Compression Type     |    time for 100 files   |     % origional file size
> None                         |            6s                 |              1.00
> 
> LZW                         |            39s                |              0.51
> HUF                          |            42s               |              0.52
> RLE                          |             12s               |              1.01
> 
> It's interesting to note that in the seconds data set the RLE algorithm actuall
> blew up the data rather than compressing it.  This help illustrate why one needs
> to be careful which algorithm is used--it can help tremendously to know what the
> data will look like in advance!

  Well, what is disappointing about this. A compression ratio of about 5
  is very good for lossless compression. Apparently you also get the
  necessary speed. BTW: what kind of machine did you use for the test? 
  As compression is CPU bounded the next generation processor will
  probably give you 8hz. 

  Users don't need to know about compression. But I think the instrument
  scientist should be able to select a compression scheme based on some
  tests.

                  Seasonal Greetings

                                Mark