[Nexus] how to read files from Eiger?
Andreas Förster
andreas.foerster at dectris.com
Fri Jun 24 14:06:28 BST 2016
Here are instructions for the Mac:
Homebrew
========
brew install hdf5 # install hdf5 library, might not be necessary
brew install lz4 # install lz4 library
git clone
https://github.com/nexusformat/HDF5-External-Filter-Plugins/tree/master/LZ4 #
download hdf lz4 filter source
cd LZ4 # change directory to lz4 source
./configure # prepare the compilation
make # compile
This should produce the file ./plugins/lib/libh5lz4.dylib. Copy this file
into your hdf5 plugin folder e.g. cp libh5lz4.dylib
/usr/local/hdf5/lib/plugin/.
Now, the decompression in hdf5 should work. Please keep in mind that the
directories and files might differ for your system installation and must be
modified accordingly. It might be necessary to supply the h5 and lz4
installation path to the configure command ( e.g. ../configure
--with-hdf5=/temp/hdf5 --with-lz4lib=/temp/lz4).
Fink
===
I did that by emulating the source instructions below in fink.
For source junkies
==============
download the latest versions of gcc, mpc, mpfr, gmp and isl
http://gcc.gnu.org/mirrors.html
http://www.multiprecision.org/index.php?prog=mpc&page=download
http://www.mpfr.org/mpfr-current/#download
http://gmplib.org/
ftp://gcc.gnu.org/pub/gcc/infrastructure/
compile and install gmp, mofr, mpc and isl successively
to achieve that
cd gmp* (or whatever package)
mkdir build
cd build
../configure [with prefix, see list below]
make -j 4
sudo make install
configure flags are
gmp:
--prefix=/usr/local/gcc-5.3.0 --with-gmp=/usr/local/gcc-5.3.0
mpfr:
--prefix=/usr/local/gcc-5.3.0 --with-gmp=/usr/local/gcc-5.3.0
mpc:
--prefix=/usr/local/gcc-5.3.0 --with-gmp=/usr/local/gcc-5.3.0
--with-mpfr=/usr/local/gcc-5.3.0
isl:
--prefix=/usr/local/gcc-5.3.0 --with-gmp-prefix=/usr/local/gcc-5.3.0
then install gcc:
cd gcc*
mkdir build && cd build
../configure --prefix=/usr/local/gcc-5.3.0 --enable-checking=release
--with-gmp=/usr/local/gcc-5.3.0 --with-mpfr=/usr/local/gcc-5.3.0
--with-mpc=/usr/local/gcc-5.3.0 --enable-languages=c,c++,fortran
--with-isl=/usr/local/gcc-5.3.0 --program-suffix=-5.3.0
(takes 1 h)
make -j 4
sudo make install
get bitshuffle:
clone https://github.com/kiyo-masui/bitshuffle.git
cd bitshuffle
modify INCLUDE_DIRS and LIBRARY_DIRS in setup.py to point to gcc-5.3.0:
vi setup.py
modify:
if sys.platform == 'darwin':
pass
# putting here both macports and homebrew paths will generate
# "ld: warning: dir not found" at the linking phase
#INCLUDE_DIRS += ['/opt/local/include'] # macports
#LIBRARY_DIRS += ['/opt/local/lib'] # macports
#INCLUDE_DIRS += ['/usr/local/include'] # homebrew
#LIBRARY_DIRS += ['/usr/local/lib'] # homebrew
elif sys.platform.startswith('freebsd'):
pass
#INCLUDE_DIRS += ['/usr/local/include'] # homebrew
#LIBRARY_DIRS += ['/usr/local/lib'] # homebrew
INCLUDE_DIRS += ['/usr/local/gcc-5.3.0/include'] # gcc 5.3
LIBRARY_DIRS += ['/usr/local/gcc-5.3.0/lib'] # gcc 5.3
pip install .
easy as this :)
Best regards.
Andreas
On Fri, Jun 24, 2016 at 1:51 PM, Eugen Wintersberger <
eugen.wintersberger at desy.de> wrote:
> Hi Pete
>
> On 06/24/16 14:45, Jemian, Pete R. wrote:
>
> Eugen
>
> First off, I (representing the user who asked me about this) want to read
> that file with the HDF5 plugin in IgorPro on a Mac.
>
> Unfortunately I do not have a Mac test environment here ;)
>
>
> Adding the compression library to this toolset is not trivial. I must
> persuade the 3rd party developer of that plugin to make this addition.
>
> I also want to read that file with the h5py package in python on mac,
> Windows, and Linux.
>
> We are currently testing an MSI package for Windows here. On Linux
> everything is easy. For Debian and Ubuntu we have packages which also set
> the correct environment variables. We tried to use all kinds of software
> to access the data which works fine on Linux once the package is installed.
>
> For Windows we have seen that with the current build (Windows 7) at least
> Matlab R2016 and Anaconda Python 3.5 work flawlessly with the plugin.
>
>
> We are on the leading edge here but our users have already begun to
> struggle with reading these files. Until the compression filters are part
> of common installations, a converter tool is needed.
>
> The point is simply that as long as you are on a Linux system everything
> is fine. At least on Windows (I have no experience with Mac)
> things are a bit more complicated. My personal suggestions to users is to
> use Linux for analysis whenever possible.
>
> I am on vacation next week but will add the LZ4+BitShuffle filter
> immediately when I am back ;) Promissed ;)
>
> regards
> Eugen
>
>
> Pete
>
>
>
>
> Sent from a device with a dreadfully small screen
>
>
> -------- Original message --------
> From: Eugen Wintersberger <eugen.wintersberger at desy.de>
> <eugen.wintersberger at desy.de>
> Date: 6/24/2016 3:05 AM (GMT-06:00)
> To: "Jemian, Pete R." <jemian at anl.gov> <jemian at anl.gov>, "Herbert J.
> Bernstein" <yayahjb at gmail.com> <yayahjb at gmail.com>, Discussion forum for
> the NeXus data format <nexus at nexusformat.org> <nexus at nexusformat.org>
> Cc: Tobias.Richter at diamond.ac.uk, jjakoncic at bnl.gov,
> andreas.foerster at dectris.com, rsweet at bnl.gov
> Subject: Re: [Nexus] how to read files from Eiger?
>
> Hi Pete,
>
> On which Linux systems do you want to read the files. For Ubuntu 14.04 and
> Debian 8 we have packages available.
> On 06/23/16 19:27, Pete Jemian wrote:
>
> Very helpful. Thanks for the quick turnaround.
> Now I go off in a corner with an internet connection for quiet time ...
>
> On 6/23/2016 12:25 PM, Herbert J. Bernstein wrote:
>
> Dear Pete,
>
> This is a solvable problem, but requires some preliminary software
> installations. First, be sure your HDF5 kit is reasonably recent and has
> filter plugin support. Then pickup and install both the LZ4 and BitShuffle
> filters. Now make sure HDF5_PLUGIN_PATH is defined to point to the
> directory containing the filter libraries. Then h5dump should be able to
> handle the Eiger files.
>
> For the current files BittShuffle is not required. Most probably required
> for latter releases of the detector?
>
> regards
> Eugen
>
>
> Eugen is gathering the filters on github. I am gathering other software
> and links to test data and templates at HDRMX.medsbio.org. There is a
> link for the hdrmx-bb list on the page. Please post comments, questions
> and suggestions there.
>
> Regards,
> Herbert
>
> Sent from my iPad
>
> On Jun 23, 2016, at 12:48, Pete Jemian <prjemian at gmail.com>
> <prjemian at gmail.com> wrote:
>
>
> We have received a linked fileset from an Eiger 1M Anderson want to read
> it.
> Cannot seem to progress past knowing that there is data.
> Here is an excerpt from h5dump (version 1.8.16):
>
> DATASET "data" {
> DATATYPE H5T_STD_U32LE
> DATASPACE SIMPLE { ( 1, 1065, 1030 ) / ( H5S_UNLIMITED, 1065,
> 1030 ) }
> DATA {h5dump error: unable to print data
>
> }
> ATTRIBUTE "image_nr_high" {
> DATATYPE H5T_STD_I32LE
> DATASPACE SCALAR
> DATA {
> (0): 1
> }
> }
> ATTRIBUTE "image_nr_low" {
> DATATYPE H5T_STD_I32LE
> DATASPACE SCALAR
> DATA {
> (0): 1
> }
> }
> }
>
> It's the error: "h5dump error: unable to print data"
>
> Is this the case where we have the wrong version of the HDF5 libraries?
> Is there a tool to render such files compatible with the 1.8 libraries?
> If so, where is that tool?
>
> Pete
>
> _______________________________________________
> NeXus mailing list
> NeXus at nexusformat.org
> http://lists.nexusformat.org/mailman/listinfo/nexus
>
>
>
> --
> Dr. Eugen Wintersberger
>
> FS-EC
> DESY
> Notkestr. 85
> D-22607 Hamburg
> Germany
>
> E-Mail: eugen.wintersberger at desy.de
> Telefon: +49-40-8998-1917
>
>
> --
> Dr. Eugen Wintersberger
>
> FS-EC
> DESY
> Notkestr. 85
> D-22607 Hamburg
> Germany
>
> E-Mail: eugen.wintersberger at desy.de
> Telefon: +49-40-8998-1917
>
--
<https://www.dectris.com>
Andreas Förster, Ph.D.
MX Application Scientist, Scientific Sales
Phone: +41 56 500 2100 | Direct: +41 56 500 2176 | Email:
andreas.foerster at dectris.com
DECTRIS Ltd. | Taefernweg 1 | 5405 Baden-Daettwil | Switzerland |
www.dectris.com
[image: LinkedIn] <https://www.linkedin.com/company/5067919>
[image: facebook]
<https://www.facebook.com/pages/Dectris-Ltd/623855944369304>
<https://twitter.com/DECTRIS_News>
Confidentiality Note: This message is intended only for the use of the
named
recipient(s) and may contain confidential and/or privileged information. If
you
are not the intended recipient, please contact the sender and delete the
message.
Any unauthorized use of the information contained in this message is
prohibited.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.nexusformat.org/pipermail/nexus/attachments/20160624/bfcc032f/attachment-0001.html>
More information about the NeXus
mailing list