[Nexus] HDF5 External Links in NeXus files

[Zbigniew Reszela]

Thanks Jacob and all for your input!

Definitely VDS looks like the way to do it.

Cheers,

Zibi

On 1/7/19 3:25 PM, Jacob.Filik at Diamond.ac.uk wrote:
>
> Hi Zbigniew,
>
> I think the only way you could do this would be to use the virtual 
> dataset (VDS) feature of HDF5, only available in the 1.10 release. 
> This should allow you to create a “virtual” stack from 4 separate images.
>
> I don’t this is a backwardly compatible feature though, so the file 
> may not be readable by applications using a pre-1.10 library version.
>
> Kind regards,
>
> Jacob
>
> Dr Jacob Filik
>
> Senior Software Scientist
>
> Tel: +441235 77 8690
>
> Diamond Light Source Ltd.
>
> Diamond House
>
> Harwell Science & Innovation Campus
>
> Didcot
>
> Oxfordshire
>
> OX11 0DE
>
> *From:*NeXus <nexus-bounces at shadow.nd.rl.ac.uk> *On Behalf Of 
> *Zbigniew Reszela
> *Sent:* 07 January 2019 14:20
> *To:* nexus at nexusformat.org
> *Subject:* [Nexus] HDF5 External Links in NeXus files
>
> Hi NeXus experts,
>
> I'm not sure if your list is for this kind of questions. If not, could 
> you please advice where to look for help and forgive me this message.
>
> We would like to write a HDF5 file, let's call it the master file, 
> with multiple NXScan entries. The scan can be configured so the 
> detector's data are directly stored in the master file or the 
> detector's data are stored in a separate file(s), let's call it the 
> slave file(s). In the later case, in the master file, we would like to 
> use the HDF5 external links to the slave file(s). The idea is to 
> provide as transparent as possible (between these two saving 
> modalities) access to the data.
>
> Let me illustrate it with an example. A scan involves just one 
> detector and provides 4 frames each of 1024x1024 shape.
> If the scan is configured to store the data in the master file, this 
> can be accessed in the following way:
>
> In [6]: f["entry1"]["measurement"]["twod01"]
> Out[6]: <HDF5 dataset "twod01": shape (4, 1024, 1024), type "<f8">
>
> If the scan is configured to store the detector's data in the slave 
> file (all 4 frames in one dataset) and we use the following code to 
> insert the external link:
>
> f = h5py.File("master.h5")
> f["entry1"]["measurement"]["twod01"] = h5py.ExternalLink("slave.h5", 
> "/data")
>
> then we can access the data from the master file:
>
> In [24]: f["entry1"]["measurement"]["twod01"]
> Out[24]: <HDF5 dataset "data": shape (4, 1024, 1024), type "<u4">
>
> My question is how to create the external links (or use some 
> alternative solution) if the detector's data are stored in multiple 
> files, for example 1 frame per file (4 slave files in total), or 2 
> frame per file (2 slave files in total). Here we would like to 
> maintain the way of accessing all the scan frames with the 
> f["entry1"]["measurement"]["twod01"] syntax.
>
> Thanks in advance for sharing your experience,
>
> Zbigniew Reszela
> (Alba Synchrotron)
>
> -- 
>
> This e-mail and any attachments may contain confidential, copyright 
> and or privileged material, and are for the use of the intended 
> addressee only. If you are not the intended addressee or an authorised 
> recipient of the addressee please notify us of receipt by returning 
> the e-mail and do not use, copy, retain, distribute or disclose the 
> information in or attached to the e-mail.
> Any opinions expressed within this e-mail are those of the individual 
> and not necessarily of Diamond Light Source Ltd.
> Diamond Light Source Ltd. cannot guarantee that this e-mail or any 
> attachments are free from viruses and we cannot accept liability for 
> any damage which you may sustain as a result of software viruses which 
> may be transmitted in or with the message.
> Diamond Light Source Limited (company no. 4375679). Registered in 
> England and Wales with its registered office at Diamond House, Harwell 
> Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United 
> Kingdom
>
>
> _______________________________________________
> NeXus mailing list
> NeXus at nexusformat.org
> http://lists.nexusformat.org/mailman/listinfo/nexus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.nexusformat.org/pipermail/nexus/attachments/20190108/f183829b/attachment-0001.html>