[Nexus] HDF5 External Links in NeXus files

[V. Armando Solé]

+1

Clear use case for virtual data sets and therefore HDF5 1.10.x

On 07/01/2019 15:25, Jacob.Filik at Diamond.ac.uk wrote:
>
> Hi Zbigniew,
>
>  
>
> I think the only way you could do this would be to use the virtual
> dataset (VDS) feature of HDF5, only available in the 1.10 release.
> This should allow you to create a “virtual” stack from 4 separate images.
>
>  
>
> I don’t this is a backwardly compatible feature though, so the file
> may not be readable by applications using a pre-1.10 library version.
>
>  
>
> Kind regards,
>
>  
>
> Jacob
>
>  
>
> Dr Jacob Filik
>
> Senior Software Scientist
>
> Tel: +441235 77 8690
>
>  
>
> Diamond Light Source Ltd.
>
> Diamond House
>
> Harwell Science & Innovation Campus
>
> Didcot
>
> Oxfordshire
>
> OX11 0DE
>
>  
>
>  
>
>  
>
> *From:*NeXus <nexus-bounces at shadow.nd.rl.ac.uk> *On Behalf Of
> *Zbigniew Reszela
> *Sent:* 07 January 2019 14:20
> *To:* nexus at nexusformat.org
> *Subject:* [Nexus] HDF5 External Links in NeXus files
>
>  
>
> Hi NeXus experts,
>
> I'm not sure if your list is for this kind of questions. If not, could
> you please advice where to look for help and forgive me this message.
>
> We would like to write a HDF5 file, let's call it the master file,
> with multiple NXScan entries. The scan can be configured so the
> detector's data are directly stored in the master file or the
> detector's data are stored in a separate file(s), let's call it the
> slave file(s). In the later case, in the master file, we would like to
> use the HDF5 external links to the slave file(s). The idea is to
> provide as transparent as possible (between these two saving
> modalities) access to the data.
>
> Let me illustrate it with an example. A scan involves just one
> detector and provides 4 frames each of 1024x1024 shape.
> If the scan is configured to store the data in the master file, this
> can be accessed in the following way:
>
> In [6]: f["entry1"]["measurement"]["twod01"]
> Out[6]: <HDF5 dataset "twod01": shape (4, 1024, 1024), type "<f8">
>
> If the scan is configured to store the detector's data in the slave
> file (all 4 frames in one dataset) and we use the following code to
> insert the external link:
>
> f = h5py.File("master.h5")
> f["entry1"]["measurement"]["twod01"] = h5py.ExternalLink("slave.h5",
> "/data")
>
> then we can access the data from the master file:
>
> In [24]: f["entry1"]["measurement"]["twod01"]
> Out[24]: <HDF5 dataset "data": shape (4, 1024, 1024), type "<u4">
>
> My question is how to create the external links (or use some
> alternative solution) if the detector's data are stored in multiple
> files, for example 1 frame per file (4 slave files in total), or 2
> frame per file (2 slave files in total). Here we would like to
> maintain the way of accessing all the scan frames with the
> f["entry1"]["measurement"]["twod01"] syntax.
>
> Thanks in advance for sharing your experience,
>
> Zbigniew Reszela
> (Alba Synchrotron)
>
>  
>
>  
>
>
> _______________________________________________
> NeXus mailing list
> NeXus at nexusformat.org
> http://lists.nexusformat.org/mailman/listinfo/nexus


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.nexusformat.org/pipermail/nexus/attachments/20190107/ac570f8f/attachment-0001.html>