[Nexus] how to read files from Eiger?
Osborn, Raymond
rosborn at anl.gov
Fri Jun 24 14:41:20 BST 2016
You can install LZ4 and bitshuffle through Conda if you are using h5py. I’m on travel at the moment, but I could probably come up with a set of instructions when I get back.
On Jun 24, 2016, at 3:06 PM, Andreas Förster <andreas.foerster at dectris.com<mailto:andreas.foerster at dectris.com>> wrote:
Here are instructions for the Mac:
Homebrew
========
brew install hdf5 # install hdf5 library, might not be necessary
brew install lz4 # install lz4 library
git clone https://github.com/nexusformat/HDF5-External-Filter-Plugins/tree/master/LZ4 # download hdf lz4 filter source
cd LZ4 # change directory to lz4 source
./configure # prepare the compilation
make # compile
This should produce the file ./plugins/lib/libh5lz4.dylib. Copy this file into your hdf5 plugin folder e.g. cp libh5lz4.dylib /usr/local/hdf5/lib/plugin/.
Now, the decompression in hdf5 should work. Please keep in mind that the directories and files might differ for your system installation and must be modified accordingly. It might be necessary to supply the h5 and lz4 installation path to the configure command ( e.g. ../configure --with-hdf5=/temp/hdf5 --with-lz4lib=/temp/lz4).
Fink
===
I did that by emulating the source instructions below in fink.
For source junkies
==============
download the latest versions of gcc, mpc, mpfr, gmp and isl
http://gcc.gnu.org/mirrors.html
http://www.multiprecision.org/index.php?prog=mpc&page=download
http://www.mpfr.org/mpfr-current/#download
http://gmplib.org/
ftp://gcc.gnu.org/pub/gcc/infrastructure/
compile and install gmp, mofr, mpc and isl successively
to achieve that
cd gmp* (or whatever package)
mkdir build
cd build
../configure [with prefix, see list below]
make -j 4
sudo make install
configure flags are
gmp:
--prefix=/usr/local/gcc-5.3.0 --with-gmp=/usr/local/gcc-5.3.0
mpfr:
--prefix=/usr/local/gcc-5.3.0 --with-gmp=/usr/local/gcc-5.3.0
mpc:
--prefix=/usr/local/gcc-5.3.0 --with-gmp=/usr/local/gcc-5.3.0 --with-mpfr=/usr/local/gcc-5.3.0
isl:
--prefix=/usr/local/gcc-5.3.0 --with-gmp-prefix=/usr/local/gcc-5.3.0
then install gcc:
cd gcc*
mkdir build && cd build
../configure --prefix=/usr/local/gcc-5.3.0 --enable-checking=release --with-gmp=/usr/local/gcc-5.3.0 --with-mpfr=/usr/local/gcc-5.3.0 --with-mpc=/usr/local/gcc-5.3.0 --enable-languages=c,c++,fortran --with-isl=/usr/local/gcc-5.3.0 --program-suffix=-5.3.0
(takes 1 h)
make -j 4
sudo make install
get bitshuffle:
clone https://github.com/kiyo-masui/bitshuffle.git
cd bitshuffle
modify INCLUDE_DIRS and LIBRARY_DIRS in setup.py to point to gcc-5.3.0:
vi setup.py
modify:
if sys.platform == 'darwin':
pass
# putting here both macports and homebrew paths will generate
# "ld: warning: dir not found" at the linking phase
#INCLUDE_DIRS += ['/opt/local/include'] # macports
#LIBRARY_DIRS += ['/opt/local/lib'] # macports
#INCLUDE_DIRS += ['/usr/local/include'] # homebrew
#LIBRARY_DIRS += ['/usr/local/lib'] # homebrew
elif sys.platform.startswith('freebsd'):
pass
#INCLUDE_DIRS += ['/usr/local/include'] # homebrew
#LIBRARY_DIRS += ['/usr/local/lib'] # homebrew
INCLUDE_DIRS += ['/usr/local/gcc-5.3.0/include'] # gcc 5.3
LIBRARY_DIRS += ['/usr/local/gcc-5.3.0/lib'] # gcc 5.3
pip install .
easy as this :)
Best regards.
Andreas
On Fri, Jun 24, 2016 at 1:51 PM, Eugen Wintersberger <eugen.wintersberger at desy.de<mailto:eugen.wintersberger at desy.de>> wrote:
Hi Pete
On 06/24/16 14:45, Jemian, Pete R. wrote:
Eugen
First off, I (representing the user who asked me about this) want to read that file with the HDF5 plugin in IgorPro on a Mac.
Unfortunately I do not have a Mac test environment here ;)
Adding the compression library to this toolset is not trivial. I must persuade the 3rd party developer of that plugin to make this addition.
I also want to read that file with the h5py package in python on mac, Windows, and Linux.
We are currently testing an MSI package for Windows here. On Linux everything is easy. For Debian and Ubuntu we have packages which also set
the correct environment variables. We tried to use all kinds of software to access the data which works fine on Linux once the package is installed.
For Windows we have seen that with the current build (Windows 7) at least Matlab R2016 and Anaconda Python 3.5 work flawlessly with the plugin.
We are on the leading edge here but our users have already begun to struggle with reading these files. Until the compression filters are part of common installations, a converter tool is needed.
The point is simply that as long as you are on a Linux system everything is fine. At least on Windows (I have no experience with Mac)
things are a bit more complicated. My personal suggestions to users is to use Linux for analysis whenever possible.
I am on vacation next week but will add the LZ4+BitShuffle filter immediately when I am back ;) Promissed ;)
regards
Eugen
Pete
Sent from a device with a dreadfully small screen
-------- Original message --------
From: Eugen Wintersberger <eugen.wintersberger at desy.de><mailto:eugen.wintersberger at desy.de>
Date: 6/24/2016 3:05 AM (GMT-06:00)
To: "Jemian, Pete R." <jemian at anl.gov><mailto:jemian at anl.gov>, "Herbert J. Bernstein" <yayahjb at gmail.com><mailto:yayahjb at gmail.com>, Discussion forum for the NeXus data format <nexus at nexusformat.org><mailto:nexus at nexusformat.org>
Cc: Tobias.Richter at diamond.ac.uk<mailto:Tobias.Richter at diamond.ac.uk>, jjakoncic at bnl.gov<mailto:jjakoncic at bnl.gov>, andreas.foerster at dectris.com<mailto:andreas.foerster at dectris.com>, rsweet at bnl.gov<mailto:rsweet at bnl.gov>
Subject: Re: [Nexus] how to read files from Eiger?
Hi Pete,
On which Linux systems do you want to read the files. For Ubuntu 14.04 and Debian 8 we have packages available.
On 06/23/16 19:27, Pete Jemian wrote:
Very helpful. Thanks for the quick turnaround.
Now I go off in a corner with an internet connection for quiet time ...
On 6/23/2016 12:25 PM, Herbert J. Bernstein wrote:
Dear Pete,
This is a solvable problem, but requires some preliminary software installations. First, be sure your HDF5 kit is reasonably recent and has filter plugin support. Then pickup and install both the LZ4 and BitShuffle filters. Now make sure HDF5_PLUGIN_PATH is defined to point to the directory containing the filter libraries. Then h5dump should be able to handle the Eiger files.
For the current files BittShuffle is not required. Most probably required for latter releases of the detector?
regards
Eugen
Eugen is gathering the filters on github. I am gathering other software and links to test data and templates atHDRMX.medsbio.org<http://hdrmx.medsbio.org/>. There is a link for the hdrmx-bb list on the page. Please post comments, questions and suggestions there.
Regards,
Herbert
Sent from my iPad
On Jun 23, 2016, at 12:48, Pete Jemian <prjemian at gmail.com><mailto:prjemian at gmail.com> wrote:
We have received a linked fileset from an Eiger 1M Anderson want to read it.
Cannot seem to progress past knowing that there is data.
Here is an excerpt from h5dump (version 1.8.16):
DATASET "data" {
DATATYPE H5T_STD_U32LE
DATASPACE SIMPLE { ( 1, 1065, 1030 ) / ( H5S_UNLIMITED, 1065, 1030 ) }
DATA {h5dump error: unable to print data
}
ATTRIBUTE "image_nr_high" {
DATATYPE H5T_STD_I32LE
DATASPACE SCALAR
DATA {
(0): 1
}
}
ATTRIBUTE "image_nr_low" {
DATATYPE H5T_STD_I32LE
DATASPACE SCALAR
DATA {
(0): 1
}
}
}
It's the error: "h5dump error: unable to print data"
Is this the case where we have the wrong version of the HDF5 libraries?
Is there a tool to render such files compatible with the 1.8 libraries?
If so, where is that tool?
Pete
_______________________________________________
NeXus mailing list
NeXus at nexusformat.org<mailto:NeXus at nexusformat.org>
http://lists.nexusformat.org/mailman/listinfo/nexus
--
Dr. Eugen Wintersberger
FS-EC
DESY
Notkestr. 85
D-22607 Hamburg
Germany
E-Mail: eugen.wintersberger at desy.de<mailto:eugen.wintersberger at desy.de>
Telefon: +49-40-8998-1917<tel:%2B49-40-8998-1917>
--
Dr. Eugen Wintersberger
FS-EC
DESY
Notkestr. 85
D-22607 Hamburg
Germany
E-Mail: eugen.wintersberger at desy.de<mailto:eugen.wintersberger at desy.de>
Telefon: +49-40-8998-1917<tel:%2B49-40-8998-1917>
--
<https://www.dectris.com/>
Andreas Förster, Ph.D.
MX Application Scientist, Scientific Sales
Phone: +41 56 500 2100 | Direct: +41 56 500 2176 | Email: andreas.foerster at dectris.com<mailto:andreas.foerster at dectris.com>
DECTRIS Ltd. | Taefernweg 1 | 5405 Baden-Daettwil | Switzerland | www.dectris.com<https://www.dectris.com/>
[LinkedIn]<https://www.linkedin.com/company/5067919>
[facebook] <https://www.facebook.com/pages/Dectris-Ltd/623855944369304> [https://www.dectris.com/tl_files/root/signatur/twitter-ico1.png] <https://twitter.com/DECTRIS_News>
[https://www.dectris.com/tl_files/root/signatur/dectris_plus%20slogan.png]
Confidentiality Note: This message is intended only for the use of the named
recipient(s) and may contain confidential and/or privileged information. If you
are not the intended recipient, please contact the sender and delete the message.
Any unauthorized use of the information contained in this message is prohibited.
_______________________________________________
NeXus mailing list
NeXus at nexusformat.org<mailto:NeXus at nexusformat.org>
http://lists.nexusformat.org/mailman/listinfo/nexus
--
Ray Osborn, Senior Scientist
Materials Science Division
Argonne National Laboratory
Argonne, IL 60439, USA
Phone: +1 (630) 252-9011
Email: ROsborn at anl.gov<mailto:ROsborn at anl.gov>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.nexusformat.org/pipermail/nexus/attachments/20160624/492dde85/attachment-0001.html>
More information about the NeXus
mailing list