[NeXus-tech] Fwd: RE: New areaDetector HDF5 file saving plugin

[Pete R. Jemian]

FYI about NAPI efficiency vs. raw HDF5

-------- Original Message --------
Subject: RE: New areaDetector HDF5 file saving plugin
Date: Fri, 13 May 2011 10:43:28 -0500
From: Mark Rivers <rivers at cars.uchicago.edu>
To: <ulrik.pedersen at diamond.ac.uk>
CC: BCDA <bcda at aps.anl.gov>

Hi Ulrik,

That's great!  I will definitely bundle it with the next release.  Jeff
Gebhardt in the BCDA group here has already been working on updating the
Nexus stuff on Windows.  I'll see if he can move all platforms to HDF
1.8, so we can build both plugins.

Have you or Tom had a chance to test the new SIS3820 driver yet?

Mark


-----Original Message-----
From: ulrik.pedersen at diamond.ac.uk [mailto:ulrik.pedersen at diamond.ac.uk]

Sent: Friday, May 13, 2011 10:23 AM
To: Mark Rivers
Subject: New areaDetector HDF5 file saving plugin

Hi Mark,

I have been tasked with writing support for areaDetector to store files
in the HDF5 file format. I realise that you have already got the nexus
file saving plugin which essentially uses the HDF5 format to store the
data. However, the nexus libraries are less efficient and have fewer
options for fine-tuning for performance.

So I have now got this new HDF5 file plugin working. It's pretty
complete and in a few terms support:
  * Multiple frames per file by adding one extra dimension.
  * All areaDetector datatypes.
  * At least 3 (perhaps any, I can't recall) number of dimensions from
NDArray.
  * Storing NDArray attributes for each stored frame.
  * Additional 2 'virtual' dimensions for instance for doing 2D
raster-scans with multiple frames acquired at each point (this sounds a
bit strange but was a particular request from a couple of our
beamlines).
  * Storing write performance timestamps for each stored frame. This
feature is to help us debug our storage server performance issues.
  * Built and tested on linux-x86 and win32-x86 (the platforms I have
available).

I've come across a dependency issue in this project: I've used the
latest release of the HDF5 libraries version 1.8. However, the
nexusSupport in areaDetector bundles and uses an older version of HDF5
version 1.6. The two versions of HDF5 cannot be linked together in one
application so I've had to disable building the nexus plugin while
building my HDF5 plugin.

The solution is probably to build a newer version of the Nexus library
against the same recent HDF5 library - I've only had a brief look at
this so far and it seems a bit complicated (in particular on my least
favourite platform windows). However, I'll give it a try to rebuild the
nexus libraries and hopefully get that working.


I've not built this as a separate support module. In fact I've tried to
follow your approach for supporting various file formats though
libraries. I wonder if you would be interested in bundling this plugin
with the rest of the plugins in your next areaDetector release?

Cheers,
Ulrik

  --------------------------------------------------------------------

   Ulrik Pedersen               phone: +44(0)1235-778580
   Software Engineer            email: Ulrik.Pedersen at diamond.ac.uk

       Diamond Light Source Ltd.
       Rutherford Appleton Laboratory,
       Chilton, Didcot
       Oxfordshire OX11 0DE

--------------------------------------------------------------------


-- 
This e-mail and any attachments may contain confidential, copyright and
or privileged material, and are for the use of the intended addressee
only. If you are not the intended addressee or an authorised recipient
of the addressee please notify us of receipt by returning the e-mail and
do not use, copy, retain, distribute or disclose the information in or
attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual
and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any
attachments are free from viruses and we cannot accept liability for any
damage which you may sustain as a result of software viruses which may
be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in
England and Wales with its registered office at Diamond House, Harwell
Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United
Kingdom