<div dir="ltr"><div><div><div>Dear Pete,<br><br></div> If you are using tools built on HDF5 1.8.11 or above, all you should need to do is to install the plugin libraries and set the HDF5_PLUGIN_PATH environment variable. For tools built on earlier HDF5 releases, your advice to users is indeed their best option for the moment. I would appreciate hearing about specific tools that are incompatible, so we can work on getting updated versions.<br><br></div> Regards,<br></div> Herbert<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Jun 26, 2016 at 4:54 PM, Pete Jemian <span dir="ltr"><<a href="mailto:jemian@anl.gov" target="_blank">jemian@anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Thanks, all, for the various detailed advice.<br>
<br>
My coworker and I have tried some of the suggestions and come to the conclusion that these compressed data files from the Eiger are not yet suitable for general distribution in places where other HDF5 files are handled routinely.<br>
<br>
Assembling the special tools to read the data in these Eiger files is going to take much more patience to make work than is available now. Until the toolset becomes part of common distribution (we are using HDF 1.8 libraries that are build into tools created by others), we will advise those who wish to analyze data with these compression formats to work with the instrument teams where the data was acquired.<br>
<br>
Regards,<br>
Pete<span class=""><br>
<br>
<br>
On 6/24/2016 9:23 AM, Herbert J. Bernstein wrote:<br>
</span><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Dear Pete,<br>
<br><span class="">
If you are on a system with good dlfnc support and are using a recent<br>
HDF5 library, then use of the plugin filters is transparent (just<br>
install the filters and define the HDF5_PLUGIN_PATH environment<br>
variable). The serious problem is on a system or applications with an<br>
old hdf5 library (e.g. 1.6). Then you need an application such as<br>
h5repack built against a recent hdf5 to repack the file with<br>
compressions that are known to the old application, but somewhere you<br>
need a new hdf5 installation to make this work.<br>
<br>
This will make a good test case to document in detail on the<br>
</span><a href="http://hdrmx.medsbio.org" rel="noreferrer" target="_blank">hdrmx.medsbio.org</a> <<a href="http://hdrmx.medsbio.org" rel="noreferrer" target="_blank">http://hdrmx.medsbio.org</a>> website as a tutorial both<span class=""><br>
for users and sysadmins.<br>
<br>
Regards,<br>
Herbert<br>
<br>
On Fri, Jun 24, 2016 at 8:45 AM, Jemian, Pete R. <<a href="mailto:jemian@anl.gov" target="_blank">jemian@anl.gov</a><br></span><span class="">
<mailto:<a href="mailto:jemian@anl.gov" target="_blank">jemian@anl.gov</a>>> wrote:<br>
<br>
Eugen<br>
<br>
First off, I (representing the user who asked me about this) want to<br>
read that file with the HDF5 plugin in IgorPro on a Mac.<br>
<br>
Adding the compression library to this toolset is not trivial. I<br>
must persuade the 3rd party developer of that plugin to make this<br>
addition.<br>
<br>
I also want to read that file with the h5py package in python on<br>
mac, Windows, and Linux.<br>
<br>
We are on the leading edge here but our users have already begun to<br>
struggle with reading these files. Until the compression filters<br>
are part of common installations, a converter tool is needed.<br>
<br>
Pete<br>
<br>
<br>
<br>
<br>
Sent from a device with a dreadfully small screen<br>
<br>
<br>
-------- Original message --------<br>
From: Eugen Wintersberger <<a href="mailto:eugen.wintersberger@desy.de" target="_blank">eugen.wintersberger@desy.de</a><br></span><span class="">
<mailto:<a href="mailto:eugen.wintersberger@desy.de" target="_blank">eugen.wintersberger@desy.de</a>>><br>
Date: 6/24/2016 3:05 AM (GMT-06:00)<br></span><span class="">
To: "Jemian, Pete R." <<a href="mailto:jemian@anl.gov" target="_blank">jemian@anl.gov</a> <mailto:<a href="mailto:jemian@anl.gov" target="_blank">jemian@anl.gov</a>>>,<br>
"Herbert J. Bernstein" <<a href="mailto:yayahjb@gmail.com" target="_blank">yayahjb@gmail.com</a><br></span>
<mailto:<a href="mailto:yayahjb@gmail.com" target="_blank">yayahjb@gmail.com</a>>>, Discussion forum for the NeXus data<br>
format <<a href="mailto:nexus@nexusformat.org" target="_blank">nexus@nexusformat.org</a> <mailto:<a href="mailto:nexus@nexusformat.org" target="_blank">nexus@nexusformat.org</a>>><br>
Cc: <a href="mailto:Tobias.Richter@diamond.ac.uk" target="_blank">Tobias.Richter@diamond.ac.uk</a><br>
<mailto:<a href="mailto:Tobias.Richter@diamond.ac.uk" target="_blank">Tobias.Richter@diamond.ac.uk</a>>, <a href="mailto:jjakoncic@bnl.gov" target="_blank">jjakoncic@bnl.gov</a><br>
<mailto:<a href="mailto:jjakoncic@bnl.gov" target="_blank">jjakoncic@bnl.gov</a>>, <a href="mailto:andreas.foerster@dectris.com" target="_blank">andreas.foerster@dectris.com</a><br>
<mailto:<a href="mailto:andreas.foerster@dectris.com" target="_blank">andreas.foerster@dectris.com</a>>, <a href="mailto:rsweet@bnl.gov" target="_blank">rsweet@bnl.gov</a><br>
<mailto:<a href="mailto:rsweet@bnl.gov" target="_blank">rsweet@bnl.gov</a>><span class=""><br>
Subject: Re: [Nexus] how to read files from Eiger?<br>
<br>
Hi Pete,<br>
<br>
On which Linux systems do you want to read the files. For Ubuntu<br>
14.04 and Debian 8 we have packages available.<br>
<br>
On 06/23/16 19:27, Pete Jemian wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Very helpful. Thanks for the quick turnaround.<br>
Now I go off in a corner with an internet connection for quiet<br>
time ...<br>
<br>
On 6/23/2016 12:25 PM, Herbert J. Bernstein wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Dear Pete,<br>
<br>
This is a solvable problem, but requires some preliminary<br>
software installations. First, be sure your HDF5 kit is<br>
reasonably recent and has filter plugin support. Then pickup and<br>
install both the LZ4 and BitShuffle filters. Now make sure<br>
HDF5_PLUGIN_PATH is defined to point to the directory containing<br>
the filter libraries. Then h5dump should be able to handle the<br>
Eiger files.<br>
</blockquote></blockquote>
For the current files BittShuffle is not required. Most probably<br>
required for latter releases of the detector?<br>
<br>
regards<br>
Eugen<br>
</span><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
<br>
Eugen is gathering the filters on github. I am gathering other<br>
software and links to test data and templates at<br></span>
<a href="http://HDRMX.medsbio.org" rel="noreferrer" target="_blank">HDRMX.medsbio.org</a> <<a href="http://HDRMX.medsbio.org" rel="noreferrer" target="_blank">http://HDRMX.medsbio.org</a>>. There is a link<span class=""><br>
for the hdrmx-bb list on the page. Please post comments,<br>
questions and suggestions there.<br>
<br>
Regards,<br>
Herbert<br>
<br>
Sent from my iPad<br>
<br>
</span><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
On Jun 23, 2016, at 12:48, Pete Jemian <<a href="mailto:prjemian@gmail.com" target="_blank">prjemian@gmail.com</a>><br></span><div><div class="h5">
<mailto:<a href="mailto:prjemian@gmail.com" target="_blank">prjemian@gmail.com</a>> wrote:<br>
<br>
<br>
We have received a linked fileset from an Eiger 1M Anderson<br>
want to read it.<br>
Cannot seem to progress past knowing that there is data.<br>
Here is an excerpt from h5dump (version 1.8.16):<br>
<br>
DATASET "data" {<br>
DATATYPE H5T_STD_U32LE<br>
DATASPACE SIMPLE { ( 1, 1065, 1030 ) / (<br>
H5S_UNLIMITED, 1065, 1030 ) }<br>
DATA {h5dump error: unable to print data<br>
<br>
}<br>
ATTRIBUTE "image_nr_high" {<br>
DATATYPE H5T_STD_I32LE<br>
DATASPACE SCALAR<br>
DATA {<br>
(0): 1<br>
}<br>
}<br>
ATTRIBUTE "image_nr_low" {<br>
DATATYPE H5T_STD_I32LE<br>
DATASPACE SCALAR<br>
DATA {<br>
(0): 1<br>
}<br>
}<br>
}<br>
<br>
It's the error: "h5dump error: unable to print data"<br>
<br>
Is this the case where we have the wrong version of the HDF5<br>
libraries?<br>
Is there a tool to render such files compatible with the 1.8<br>
libraries?<br>
If so, where is that tool?<br>
<br>
Pete<br>
<br>
_______________________________________________<br>
NeXus mailing list<br></div></div>
<a href="mailto:NeXus@nexusformat.org" target="_blank">NeXus@nexusformat.org</a> <mailto:<a href="mailto:NeXus@nexusformat.org" target="_blank">NeXus@nexusformat.org</a>><br>
<a href="http://lists.nexusformat.org/mailman/listinfo/nexus" rel="noreferrer" target="_blank">http://lists.nexusformat.org/mailman/listinfo/nexus</a><br>
<br>
</blockquote></blockquote>
<br>
</blockquote><span class="">
<br>
--<br>
Dr. Eugen Wintersberger<br>
<br>
FS-EC<br>
DESY<br>
Notkestr. 85<br>
D-22607 Hamburg<br>
Germany<br>
<br>
E-Mail: <a href="mailto:eugen.wintersberger@desy.de" target="_blank">eugen.wintersberger@desy.de</a><br></span>
<mailto:<a href="mailto:eugen.wintersberger@desy.de" target="_blank">eugen.wintersberger@desy.de</a>><br>
Telefon: <a href="tel:%2B49-40-8998-1917" value="+494089981917" target="_blank">+49-40-8998-1917</a> <tel:%2B49-40-8998-1917><br>
<br>
<br>
</blockquote><span class="im HOEnZb">
<br>
-- <br>
----------------------------------------------------------<br>
Pete R. Jemian, Ph.D. <<a href="mailto:jemian@anl.gov" target="_blank">jemian@anl.gov</a>><br>
Beam line Controls and Data Acquisition, Group Leader<br>
Advanced Photon Source, Argonne National Laboratory<br>
Argonne, IL 60439 <a href="tel:630%20-%20252%20-%203189" value="+16302523189" target="_blank">630 - 252 - 3189</a><br>
-----------------------------------------------------------<br>
Education is the one thing for which people<br>
are willing to pay yet not receive.<br>
-----------------------------------------------------------<br>
<br>
<br></span><div class="HOEnZb"><div class="h5">
_______________________________________________<br>
NeXus mailing list<br>
<a href="mailto:NeXus@nexusformat.org" target="_blank">NeXus@nexusformat.org</a><br>
<a href="http://lists.nexusformat.org/mailman/listinfo/nexus" rel="noreferrer" target="_blank">http://lists.nexusformat.org/mailman/listinfo/nexus</a><br>
<br>
</div></div></blockquote></div><br></div>