[Nexus] HDF5 External Links in NeXus files

[Jacob.Filik at Diamond.ac.uk]

Hi Zbigniew,

I think the only way you could do this would be to use the virtual dataset (VDS) feature of HDF5, only available in the 1.10 release. This should allow you to create a “virtual” stack from 4 separate images.

I don’t this is a backwardly compatible feature though, so the file may not be readable by applications using a pre-1.10 library version.

Kind regards,

Jacob

Dr Jacob Filik
Senior Software Scientist
Tel: +441235 77 8690

Diamond Light Source Ltd.
Diamond House
Harwell Science & Innovation Campus
Didcot
Oxfordshire
OX11 0DE



From: NeXus <nexus-bounces at shadow.nd.rl.ac.uk> On Behalf Of Zbigniew Reszela
Sent: 07 January 2019 14:20
To: nexus at nexusformat.org
Subject: [Nexus] HDF5 External Links in NeXus files

Hi NeXus experts,

I'm not sure if your list is for this kind of questions. If not, could you please advice where to look for help and forgive me this message.

We would like to write a HDF5 file, let's call it the master file, with multiple NXScan entries. The scan can be configured so the detector's data are directly stored in the master file or the detector's data are stored in a separate file(s), let's call it the slave file(s). In the later case, in the master file, we would like to use the HDF5 external links to the slave file(s). The idea is to provide as transparent as possible (between these two saving modalities) access to the data.

Let me illustrate it with an example. A scan involves just one detector and provides 4 frames each of 1024x1024 shape.
If the scan is configured to store the data in the master file, this can be accessed in the following way:

In [6]: f["entry1"]["measurement"]["twod01"]
Out[6]: <HDF5 dataset "twod01": shape (4, 1024, 1024), type "<f8">

If the scan is configured to store the detector's data in the slave file (all 4 frames in one dataset) and we use the following code to insert the external link:

f = h5py.File("master.h5")
f["entry1"]["measurement"]["twod01"] = h5py.ExternalLink("slave.h5", "/data")

then we can access the data from the master file:

In [24]: f["entry1"]["measurement"]["twod01"]
Out[24]: <HDF5 dataset "data": shape (4, 1024, 1024), type "<u4">

My question is how to create the external links (or use some alternative solution) if the detector's data are stored in multiple files, for example 1 frame per file (4 slave files in total), or 2 frame per file (2 slave files in total). Here we would like to maintain the way of accessing all the scan frames with the f["entry1"]["measurement"]["twod01"] syntax.

Thanks in advance for sharing your experience,

Zbigniew Reszela
(Alba Synchrotron)


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.nexusformat.org/pipermail/nexus/attachments/20190107/f727fd27/attachment-0002.html>