NeXus efficiency

[Ray Osborn]

Mark asked if we had any reason to believe that NeXus performance was
inadequate.  I had hoped to do some tests myself, but haven't found time
yet.  However, I thought I should pass on some comments from Tim Mooney,
who is co-chairing the next NOBUGS, and is responsible for instrument
control and data acquisition developments at APS.  About a year ago, when a
co-worker of his worked a lot on HDF, they found that, while HDF
performance was acceptable for writing a small number of large arrays, it
was very slow to write lots of small arrays e.g. 10 one-dimensional arrays
(around 50 long) and another 50 or so scalar values.  This represented a
typical APS file.  It could sometimes take minutes (!!!) to write such a
file.

I think that this was the problem Shiming Xu said would be helped by
enlarging the HDF header block size.  It might also be helped by using the
newer method of generating fake dimensions.  However, we have not tested it
yet, so I wondered if anyone else has similar experience.

A week ago, when I was at NIST, Przemek, Nick and I discussed the radical
strategy of abandoning SDS's altogether, and writing everything as Vdata.
The reasoning behind this is that SDSs are, I believe, constructs
consisting of Vdata, with extra Vdata for the dimension scales, all grouped
together into a Vgroup.  This is what we have done in NXdata groups - we
have created a type of SDS, which does not have the name limitation of the
HDF version.  Why then do we use SDSs at all?  The answer used to be that
Vdata could not have attributes, but that is no longer true.  There may be
reasons for using SDSs if we need to write slabs of data - this I don't
know.  However, the advantages of using Vdata are a potentially major gain
in storage efficiency when we are storing lots of small arrays.

I don't know if they have had more thoughts on the subject, but we didn't
have enough information at the time to come to a conclusion.  Does anyone
else have an opinion about whether NeXus might be better implemented as
Vdata?  This is the sort of decision that we must come to urgently before
we have problems of backward compatibility.

Regards,
Ray

------------------------------------------------------
Dr Ray Osborn                  Tel: +1 (630) 252-9011
Materials Science Division     Fax: +1 (630) 252-7777
Argonne National Laboratory    E-mail: ROsborn at anl.gov
Argonne, IL 60439-4845


--------------------------------------------------------------------------------
Return-Path: <ROsborn at anl.gov>
Received: from dns2.anl.gov by ipns.pns.anl.gov (MX V4.2 AXP) with SMTP; Mon,
          08 Sep 1997 14:27:35 CST
Received: from osborn.msd.anl.gov (osborn.msd.anl.gov [146.139.244.39]) by
          dns2.anl.gov (8.6.11/8.6.11) with ESMTP id OAA19756 for
          <ROsborn at anl.gov>; Mon, 8 Sep 1997 14:27:31 -0500
Received: from dns2.anl.gov (146.139.254.3) by osborn.msd.anl.gov with SMTP
          (Eudora Internet Mail Server 1.1.2); Mon, 8 Sep 1997 14:26:31 -0500
Received: from osborn.msd.anl.gov (osborn.msd.anl.gov [146.139.244.39]) by
          dns2.anl.gov (8.6.11/8.6.11) with ESMTP id OAA19732 for
          <NAPI at anl.gov>; Mon, 8 Sep 1997 14:26:26 -0500
Received: from [146.139.244.39] by osborn.msd.anl.gov with ESMTP (Eudora
          Internet Mail Server 1.1.2); Mon, 8 Sep 1997 14:25:48 -0500
Message-ID: <l0310280ab039fc6c301c@[146.139.244.39]>