From sam.tygier at stfc.ac.uk Fri Feb 3 16:03:04 2023 From: sam.tygier at stfc.ac.uk (Sam Tygier - STFC UKRI) Date: Fri, 3 Feb 2023 16:03:04 +0000 Subject: [Nexus] Float imaging data in NXtomo and NXtomoproc Message-ID: Hello, I am working on Mantid Imaging which features reading and writing of neutron imaging data from NXtomo and NXtomoproc. While the data from the instrument usually arrives as uint16 we do our processing in, and reconstruct to float32. The NXtomo and NXtomoproc definitions both specify the data array as NX_INT. Ideally, we'd like to be able to store our datasets in float32, both for reconstructed output and intermediate processed projections. We do this with TIFFs and it avoids any loss of precision. Is it ok to store float32 in these arrays? I assume this is bad because the files would not be valid and other software may not be able to correctly open them. Could the definitions be updated to use NX_NUMBER? What's the best way to propose this change? The NXtomoproc data array has attributes 'transform', 'offset', 'scaling'. Are these to specify int-float conversion? Is there any documentation? Could these be added to NXtomo? Thanks, Sam ??Dr Sam Tygier?? Senior Scientific? Software Engineer Science and Technology Facilities Council sam.tygier at stfc.ac.uk?? From paul.millar at desy.de Mon Feb 6 11:26:49 2023 From: paul.millar at desy.de (Paul Millar) Date: Mon, 6 Feb 2023 12:26:49 +0100 Subject: [Nexus] Float imaging data in NXtomo and NXtomoproc In-Reply-To: References: Message-ID: <2711155d-41ce-fd5b-a6a9-67ed5ba3585f@desy.de> Hi Sam, On 03/02/2023 17:03, Sam Tygier - STFC UKRI via NeXus wrote: > I am working on Mantid Imaging which features reading and writing of > neutron imaging data from NXtomo and NXtomoproc. > > While the data from the instrument usually arrives as uint16 we do > our processing in, and reconstruct to float32. The NXtomo and > NXtomoproc definitions both specify the data array as NX_INT. Just for clarity, I believe NXtomo has the following "data" arrays: NXtomo/entry/instrument/detector/data NXtomo/entry/control/data NXtomo/entry/data Only NXtomo/entry/instrument/detector/data limits the values to NX_INT, the other two have NX_FLOAT and NX_NUMBER, respectively. In NXtomoproc, there is only NXtomoproc/entry/data/data, which is (indeed) limited to allow only NX_INT values. > Ideally, we'd like to be able to store our datasets in float32, both > for reconstructed output and intermediate processed projections. We > do this with TIFFs and it avoids any loss of precision. I have limited knowledge of tomography processing, but you desire here makes sense to me. > Is it ok to store float32 in these arrays? I assume this is bad > because the files would not be valid and other software may not be > able to correctly open them. Yes, I believe it would be non-conforming to store float16 data directly under NXtomo/entry/instrument/detector/data > Could the definitions be updated to use NX_NUMBER? What's the best > way to propose this change? There is an issue tracker that may be used to propose changes or ask general questions: https://github.com/nexusformat/definitions/issues I would suggest opening an issue, describing your use-case and asking for support. > The NXtomoproc data array has attributes 'transform', 'offset', > 'scaling'. Are these to specify int-float conversion? Is there any > documentation? NXtomoproc does seem to suffer from being badly documented. There's an obvious typo in NXtomoproc/entry/data/data "Dimensions". From reading the documentation, the meaning of @transform, @offset and @scaling is (to me) completely opaque. Since they target the "data" field, I *guess* that these attributes are indented to support quantised/binned data, and describing how to transform the recorded integer values back to a corresponding floating-point representation. However, that's really just a guess. > Could these be added to NXtomo? As I understand it, NXtomo already supports float numbers under NXtomo/entry/data Again, the exact semantics of the different fields within NXtomo are somewhat opaque to me, give NXtomo is described as containing raw data and has two entries where (at least, to my naive understanding) raw data might reasonably be stored: NXtomo/entry/instrument/detector/data NXtomo/entry/data At the least, I think there should be a task to tidy up the document for NXtomo and NXtomoproc, to make these things clearer. Cheers, Paul. From paul.millar at desy.de Mon Feb 6 11:26:49 2023 From: paul.millar at desy.de (Paul Millar) Date: Mon, 6 Feb 2023 12:26:49 +0100 Subject: [Nexus] Float imaging data in NXtomo and NXtomoproc In-Reply-To: References: Message-ID: <2711155d-41ce-fd5b-a6a9-67ed5ba3585f@desy.de> Hi Sam, On 03/02/2023 17:03, Sam Tygier - STFC UKRI via NeXus wrote: > I am working on Mantid Imaging which features reading and writing of > neutron imaging data from NXtomo and NXtomoproc. > > While the data from the instrument usually arrives as uint16 we do > our processing in, and reconstruct to float32. The NXtomo and > NXtomoproc definitions both specify the data array as NX_INT. Just for clarity, I believe NXtomo has the following "data" arrays: NXtomo/entry/instrument/detector/data NXtomo/entry/control/data NXtomo/entry/data Only NXtomo/entry/instrument/detector/data limits the values to NX_INT, the other two have NX_FLOAT and NX_NUMBER, respectively. In NXtomoproc, there is only NXtomoproc/entry/data/data, which is (indeed) limited to allow only NX_INT values. > Ideally, we'd like to be able to store our datasets in float32, both > for reconstructed output and intermediate processed projections. We > do this with TIFFs and it avoids any loss of precision. I have limited knowledge of tomography processing, but you desire here makes sense to me. > Is it ok to store float32 in these arrays? I assume this is bad > because the files would not be valid and other software may not be > able to correctly open them. Yes, I believe it would be non-conforming to store float16 data directly under NXtomo/entry/instrument/detector/data > Could the definitions be updated to use NX_NUMBER? What's the best > way to propose this change? There is an issue tracker that may be used to propose changes or ask general questions: https://github.com/nexusformat/definitions/issues I would suggest opening an issue, describing your use-case and asking for support. > The NXtomoproc data array has attributes 'transform', 'offset', > 'scaling'. Are these to specify int-float conversion? Is there any > documentation? NXtomoproc does seem to suffer from being badly documented. There's an obvious typo in NXtomoproc/entry/data/data "Dimensions". From reading the documentation, the meaning of @transform, @offset and @scaling is (to me) completely opaque. Since they target the "data" field, I *guess* that these attributes are indented to support quantised/binned data, and describing how to transform the recorded integer values back to a corresponding floating-point representation. However, that's really just a guess. > Could these be added to NXtomo? As I understand it, NXtomo already supports float numbers under NXtomo/entry/data Again, the exact semantics of the different fields within NXtomo are somewhat opaque to me, give NXtomo is described as containing raw data and has two entries where (at least, to my naive understanding) raw data might reasonably be stored: NXtomo/entry/instrument/detector/data NXtomo/entry/data At the least, I think there should be a task to tidy up the document for NXtomo and NXtomoproc, to make these things clearer. Cheers, Paul. From rosborn at anl.gov Fri Mar 3 00:01:39 2023 From: rosborn at anl.gov (Osborn, Raymond) Date: Fri, 3 Mar 2023 00:01:39 +0000 Subject: [Nexus] Release of NeXpy v1.0.0 Message-ID: I am pleased to announce the official release of both NeXpy v1.0.0 and nexusformat v1.0.0. Complete documentation for both packages is available online (https://nexpy.github.io/nexpy/). NeXpy NeXpy provides a GUI interface to data stored as HDF5 files, particularly those conforming to the NeXus standard. It comprises a simple PyQt GUI that combines a tree view of loaded data, Matplotlib panes to visualize multidimensional arrays, and an IPython shell for manipulating the data and metadata from the command line. NeXpy is intended to be a simple, intuitive, interactive toolbox that allows users to : * visualize NeXus data rapidly * make arbitrary projections and slices * easily compare multiple datasets with synchronized plot windows * consolidate data from multiple files into single multidimensional arrays with minimal overhead * perform least-squares fitting of 1D data with a GUI interface to LMFIT * create new NeXus-conforming data files without expert knowledge of the file format, and * develop new modes of analysis using a built-in Python script editor. In more detail, NeXpy has the following features : * Individual HDF5 files or entire directories of files can be loaded into the tree view. Files are loaded ?lazily?; only the top levels of the file hierarchy are loaded by default and only small datasets are read into memory. This allows many files to be loaded rapidly, even when they are extremely large. Slabs from multidimensional fields are loaded from the files dynamically when they are needed, e.g., in plotting a 2D slice, obviating the need for large system memory. Non-HDF5 data, e.g., TIFF or CBF images, SPEC files, or multicolumn ASCII files, can also be imported and added to the tree as NeXus groups. * Data of arbitrary dimension can be plotted as 1D or 2D slices in one or more plot windows. Each window has a tabbed interface for manipulating plot limits, switching axes, using linear or log scaling, choosing color maps, etc. Other aspects of the plots, such as marker types, colors, and sizes, grid lines, legends, and axis labelling can be adjusted in each plot's Customize Panel. A number of specialized Matplotlib features are conveniently available within the GUI, such as skew axes, symmetric color maps with automatically symmetrized limits, and various smoothing algorithms for 1D and 2D plots. * For higher-dimensional data (>2), an additional tab defines the values of the out-of-plane axes, which can be easily adjusted manually or scanned as a movie, with or without autoscaling in each plane. * A Projection Panel allows 1D or 2D slices and projections of arbitrary extent and thickness to be plotted. This panel has a tabbed interface that allows projection limits from multiple plots to be synchronized and, in the case of 1D projections, over-plotted. * A Limits Panel allows the plots from multiple data sets to be synchronized, so that changes to the plot limits or even plot axes are automatically applied to all the plot windows. This allows, for example, data taken at multiple temperatures to be compared easily. * A Scan Panel allows compatible data sets from multiple files to be consolidated into a single array with an extra dimension corresponding to a scanned parameter. If data are taken at multiple temperatures, for example, the n-dimensional arrays in each file could be combined to form a (n+1)-dimensional array, with temperature as the additional axis. This allows the temperature axis to be included as a projection axis in the Projection Panel. The Scan Panel consolidates the arrays using HDF5 Virtual Datasets, so there is no increased cost in either data storage or memory. * A built-in Script Editor with Python syntax coloring allows repeated operations to be coded or new data analysis methods to be developed and tested in the IPython shell. All data loaded in the NeXpy tree are available for use in these scripts. Additional data can be created or loaded directly from additional files using the nexusformat API. * All 1D plots have a ?Fit? button that launches a GUI interface to the LMFIT package. This allows LMFIT models, e.g., Gaussians, Lorentzians, Voigt functions, etc, to be combined in arbitrary ways to produce a composite model. Additional models can be added to the list of pre-defined models. The default is to add such models together, but NeXpy provides a GUI dialog allowing other operators to be used in combining them. Parameters can be fixed or constrained either with limits or by algebraic functions of other parameters, again customizable through GUI dialogs. Fits can be saved for storage with the NeXus data itself and/or reloaded in order to further refine the models and parameters. * The values of NeXus fields can be viewed in tabular form, along with the other HDF5 parameters that define them (data type, chunk size, compression, etc). * A number of keyboard shortcuts accelerate interactions with the plot windows to turn log scaling and grid lines on and off, to set aspect ratios, etc. * A plugin architecture allows additional menus to be added to NeXpy for specialized applications. The package modules contain a number of PyQt widgets that make it easier to add, e.g, parameter tables and pull-down menus, to specialist GUI dialogs. PyPI: https://pypi.org/project/NeXpy conda-forge: https://anaconda.org/conda-forge/nexpy nexusformat The underlying API used by NeXpy is available as a separate download from PyPI (https://pypi.org/project/nexusformat/) and conda-forge (https://anaconda.org/conda-forge/nexusformat). This provides an intuitive syntax for creating, reading, and writing HDF5 files that conform to the NeXus standard. As described above, the tree of groups and fields that comprise the hierarchical structure of NeXus files are ?lazily? loaded with a single function call, giving instant access to all the data stored in the file as nested dictionaries without loading it all into memory. Data objects anywhere in the file can also be accessed using Posix-style paths as the dictionary keys. The API makes it easy to link fields both within the same file and to external files. The API uses h5py to read and write the HDF5 files, but significantly reduces the code required to implement NeXus design rules, while ensuring compliance with the latest version of the standard. Tools allow the data to be manipulated and augmented either interactively or using Python scripts. Large multidimensional fields can be created dynamically and filled incrementally as slabs either in core memory files or saved directly to a file on disk. This allows the API to handle very large data sets without requiring significant system memory. A file locking mechanism is provided to prevent data corruption from simultaneous access by multiple processes. Backups can be created and automatically restored if necessary. A tutorial on using the nexusformat API is available as a Jupyter Notebook. It can be accessed online in a Google colaboratory: https://colab.research.google.com/github/nexpy/nexusformat/blob/master/src/nexusformat/notebooks/nexusformat.ipynb PyPI: https://pypi.org/project/nexusformat conda-forge: https://anaconda.org/conda-forge/nexusformat Installation NeXpy is available on both the Python Package Index server and Anaconda, using $ pip install nexpy or $ conda install -c conda-forge nexpy Please report any bugs to the Github issues page. Other questions can be addressed to the NeXus Mailing List >. NeXpy Home Page: https://nexpy.github.io/nexpy/ NeXpy Github: https://github.com/nexpy/nexpy NeXpy Release Notes: https://github.com/nexpy/nexpy/releases nexusformat Github: https://github.com/nexpy/nexusformat nexusformat Release Notes: https://github.com/nexpy/nexpy/releases Ray Osborn -- Ray Osborn, Senior Scientist Materials Science Division Argonne National Laboratory Argonne, IL 60439, USA Phone: +1 (630) 252-9011 Email: ROsborn at anl.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From fsaiz at cells.es Thu Mar 16 09:37:14 2023 From: fsaiz at cells.es (Fernan Saiz) Date: Thu, 16 Mar 2023 10:37:14 +0100 Subject: [Nexus] Using NXxas for XMCD experiments Message-ID: <183c3e12-3107-7ed9-daae-9df26570f6d9@cells.es> Dear Nexus community, I was wondering if any member here has been used NXxas for XMCD experiments given that XMCD can be assumed to be the difference of two XAS spectra obtained under plus and minus magnetic fields and divided their sum. The reason behind my question is that I see much simpler to add the information about the magnetic field by filling out e.g. *magnetic_field *in the NXsamplerather than coming up with a new application definition. The former option can be also complemented by providing information about the magnet (e.g. NXquadrupole_magnet). Any thoughts would be welcomed. Sincerely, -- Fernan Saiz, PhD Scientific Data Management Engineering - Computing Division ALBA SYNCHROTRON LIGHT SOURCE Carrer de la Llum 2-26 | 08290 | Cerdanyola del Vall?s| Barcelona | Spain (+34) 93 592 4075 | fsaiz at cells.es www.albasynchrotron.es | email at cells.es | legal notice / av?s legal / aviso legal **Please, do not print this e-mail unless it is absolutely necessary. **Si heu rebut aquest correu per error, us informo que pot contenir informaci? confidencial i privada i que est? prohibit el seu ?s. Us agrair?em que ho comuniqueu al remitent i l'elimineu. Gr?cies. Si ha recibido este correo por error, le informo de que puede contener informaci?n confidencial y privada y que est? prohibido su uso. Le agradecer? que lo comunique a su remitente y lo elimine. Gracias. If you have received this e-mail in error, please note that it may contain confidential and private information, therefore, the use of this information is strictly forbidden. Please inform the sender of the error and delete the information received. Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: logoALBA_positiu.png Type: image/png Size: 8076 bytes Desc: not available URL: From steve.collins at diamond.ac.uk Thu Mar 16 09:58:51 2023 From: steve.collins at diamond.ac.uk (Collins, Steve (DLSLtd,RAL,SCI)) Date: Thu, 16 Mar 2023 09:58:51 +0000 Subject: [Nexus] Using NXxas for XMCD experiments In-Reply-To: <183c3e12-3107-7ed9-daae-9df26570f6d9@cells.es> References: <183c3e12-3107-7ed9-daae-9df26570f6d9@cells.es> Message-ID: Dear Fernan, That sounds sensible in as far as XMCD (and XMLD) is XAS with different values for field and/or polarization, both of which are documented. We have an issue with magnetic_field in that the possible directions are currently very restricted. This probably needs to be changed. Regards, Steve From: NeXus On Behalf Of Fernan Saiz via NeXus Sent: 16 March 2023 09:37 To: nexus at nexusformat.org Cc: Fernan Saiz Subject: [Nexus] Using NXxas for XMCD experiments Dear Nexus community, I was wondering if any member here has been used NXxas for XMCD experiments given that XMCD can be assumed to be the difference of two XAS spectra obtained under plus and minus magnetic fields and divided their sum. The reason behind my question is that I see much simpler to add the information about the magnetic field by filling out e.g. magnetic_field in the NXsample rather than coming up with a new application definition. The former option can be also complemented by providing information about the magnet (e.g. NXquadrupole_magnet). Any thoughts would be welcomed. Sincerely, -- [cid:image001.png at 01D957ED.E8485360] Fernan Saiz, PhD Scientific Data Management Engineering - Computing Division ALBA SYNCHROTRON LIGHT SOURCE Carrer de la Llum 2-26 | 08290 | Cerdanyola del Vall?s| Barcelona | Spain (+34) 93 592 4075 | fsaiz at cells.es www.albasynchrotron.es | email at cells.es | legal notice / av?s legal / aviso legal Please, do not print this e-mail unless it is absolutely necessary. Si heu rebut aquest correu per error, us informo que pot contenir informaci? confidencial i privada i que est? prohibit el seu ?s. Us agrair?em que ho comuniqueu al remitent i l'elimineu. Gr?cies. Si ha recibido este correo por error, le informo de que puede contener informaci?n confidencial y privada y que est? prohibido su uso. Le agradecer? que lo comunique a su remitente y lo elimine. Gracias. If you have received this e-mail in error, please note that it may contain confidential and private information, therefore, the use of this information is strictly forbidden. Please inform the sender of the error and delete the information received. Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 8076 bytes Desc: image001.png URL: From fsaiz at cells.es Thu Mar 16 14:18:20 2023 From: fsaiz at cells.es (Fernan Saiz) Date: Thu, 16 Mar 2023 15:18:20 +0100 Subject: [Nexus] Using NXxas for XMCD experiments In-Reply-To: References: <183c3e12-3107-7ed9-daae-9df26570f6d9@cells.es> Message-ID: <0ac2f968-65a9-9716-6391-8c020515c2f9@cells.es> Dear Steve, Yes, I agree with you. It would be probably 'enough' to add the fields *flux *and *incident_polarization *of the magnetic field from NXbeam to NXsample. NXbeam is for the incident photons. Regards, ?- Fernan On 3/16/23 10:58, Collins, Steve (DLSLtd,RAL,SCI) via NeXus wrote: > > Dear Fernan, > > That sounds sensible in as far as XMCD (and XMLD) is XAS with > different values for field and/or polarization, both of which are > documented. > > We have an issue with magnetic_field in that the possible directions > are currently very restricted. This probably needs to be changed. > > Regards, > > Steve > > *From:*NeXus *On Behalf Of *Fernan > Saiz via NeXus > *Sent:* 16 March 2023 09:37 > *To:* nexus at nexusformat.org > *Cc:* Fernan Saiz > *Subject:* [Nexus] Using NXxas for XMCD experiments > > Dear Nexus community, > > I was wondering if any member here has been used NXxas > for > XMCD experiments given that XMCD can be assumed to be the difference > of two XAS spectra obtained under plus and minus magnetic fields and > divided their sum. The reason behind my question is that I see much > simpler to add the information about the magnetic field by filling out > e.g. *magnetic**_field *in the NXsample rather than coming up with a > new application definition. The former option can be also complemented > by providing information about the magnet (e.g. NXquadrupole_magnet). > > Any thoughts would be welcomed. > > Sincerely, > > -- > > > > > > *Fernan Saiz, PhD * > > Scientific Data Management Engineering - Computing Division > > *ALBA SYNCHROTRON LIGHT SOURCE* > > Carrer de la Llum 2-26 | 08290 | Cerdanyola del Vall?s| Barcelona | > Spain > (+34) 93 592 4075 | fsaiz at cells.es > > *www.albasynchrotron.es **| > **email at cells.es | legal notice > / av?s legal > / aviso > legal * > > *Please, do not print this e-mail unless it is absolutely necessary. > *Si heu rebut aquest correu per error, us informo que pot contenir > informaci? confidencial i privada i que est? prohibit el seu ?s. Us > agrair?em que ho comuniqueu al remitent i l'elimineu. Gr?cies. > Si ha recibido este correo por error, le informo de que puede contener > informaci?n confidencial y privada y que est? prohibido su uso. Le > agradecer? que lo comunique a su remitente y lo elimine. Gracias. > If you have received this e-mail in error, please note that it may > contain confidential and private information, therefore, the use of > this information is strictly forbidden. Please inform the sender of > the error and delete the information received. Thank you. > > > _______________________________________________ > NeXus mailing list > NeXus at nexusformat.org > https://lists.nexusformat.org/mailman/listinfo/nexus -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 8076 bytes Desc: not available URL: From fsaiz at cells.es Thu Mar 23 13:04:31 2023 From: fsaiz at cells.es (Fernan Saiz) Date: Thu, 23 Mar 2023 14:04:31 +0100 Subject: [Nexus] NXxas structure for multiple datasets Message-ID: <4c668adc-a1f5-f9c3-f522-dda572e05ae9@cells.es> Dear Nexus community, I would like to enquire to any of the XAS users here about the structure they employ to build hdf5 files when XAS is used to study chemical reactions. For this specific problem, here in ALBA the user needs to collect the XAS spectra every few minutes as well as the temperature, pressure, and the concentrations of the involved chemical species (reactants and products) every several seconds. This data collection takes place during long periods of time of up to a few days. Hence, I would to know if it would be more convenient to lump all these measurements in one NXentry, which would contain data and metadata for all these physico-chemical properties; or instead, to write different NXentries for each one of them (e.g. one NXentry for temperature, another one for pressure, and so on). Sincerely, -- Fernan Saiz, PhD Scientific Data Management Engineering - Computing Division ALBA SYNCHROTRON LIGHT SOURCE Carrer de la Llum 2-26 | 08290 | Cerdanyola del Vall?s| Barcelona | Spain (+34) 93 592 4075 | fsaiz at cells.es www.albasynchrotron.es | email at cells.es | legal notice / av?s legal / aviso legal **Please, do not print this e-mail unless it is absolutely necessary. **Si heu rebut aquest correu per error, us informo que pot contenir informaci? confidencial i privada i que est? prohibit el seu ?s. Us agrair?em que ho comuniqueu al remitent i l'elimineu. Gr?cies. Si ha recibido este correo por error, le informo de que puede contener informaci?n confidencial y privada y que est? prohibido su uso. Le agradecer? que lo comunique a su remitente y lo elimine. Gracias. If you have received this e-mail in error, please note that it may contain confidential and private information, therefore, the use of this information is strictly forbidden. Please inform the sender of the error and delete the information received. Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: logoALBA_positiu.png Type: image/png Size: 8076 bytes Desc: not available URL: From contact at hdfql.com Mon Sep 4 22:53:22 2023 From: contact at hdfql.com (HDFql) Date: Mon, 04 Sep 2023 23:53:22 +0200 Subject: [Nexus] HDFql version 2.5.0 Message-ID: We are happy to announce the release of HDFql [1] 2.5.0! This version includes: - Added support for sliding cursors (to enable reading a dataset that does not fit in (RAM) memory in a sliding fashion through a cursor, allowing a user to (seamlessly) load/process the dataset in an out-of-core manner) - Added support to create a dataset/attribute based on the characteristics (i.e. data type and dimensions) of the input redirecting (e.g. when executing "CREATE DATASET my_dataset VALUES FROM BINARY FILE my_file.bin", a dataset named "my_dataset" is created with the appropriate data type and dimensions to store all the data from a binary file named "my_file.bin", alleviating the user from specifying these) - Improved performance and memory footprint of a cursor populated with values from datasets/attributes (thanks to a zero-copy policy which reutilizes the buffer used to read these - e.g. given a dataset of data type INT of three dimensions (size 100x1024x1024), it is 10x faster and takes 15x less memory to populate a cursor with values from the dataset in comparison with the previous version of HDFql) - Added support to write a result set into a dataset/attribute (e.g. when executing "SHOW FILE INTO DATASET my_dataset", a dataset named "my_dataset" is created (if it does not exist) with the appropriate data type and dimensions to store all the names of files found in the directory currently in use) (Please check the release notes [2] for further details and some examples [3] that illustrate how HDFql works) Links: ------ [1] https://www.hdfql.com [2] https://www.hdfql.com/resources/RELEASE.txt [3] https://www.hdfql.com/examples -------------- next part -------------- An HTML attachment was scrubbed... URL: From petersonpf at ornl.gov Thu Sep 7 20:53:15 2023 From: petersonpf at ornl.gov (Peterson, Peter) Date: Thu, 7 Sep 2023 19:53:15 +0000 Subject: [Nexus] Migration from nexus API to hdf5 API Message-ID: I am looking to migrate a code from using the nexus API to directly using hdf5. Changing everything all at once is a ton of work. The ideas I've had so far for a more gradual migration: 1. Is there a way to check an NXhandle that it is to an hdf5 file and then use a member of the NXhandle directly with the hdf5 api? 2. Can one open the same file with hdf5 and nexus and carry around both handles at the same time? This would probably require using "reopen" and being careful about access to make it work. 3. Any other ideas? P^2 -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark.koennecke at psi.ch Fri Sep 8 08:13:20 2023 From: mark.koennecke at psi.ch (Koennecke Mark) Date: Fri, 8 Sep 2023 07:13:20 +0000 Subject: [Nexus] Migration from nexus API to hdf5 API In-Reply-To: References: Message-ID: Peter, Am 07.09.2023 um 21:53 schrieb Peterson, Peter via NeXus : I am looking to migrate a code from using the nexus API to directly using hdf5. Changing everything all at once is a ton of work. The ideas I?ve had so far for a more gradual migration: 1. Is there a way to check an NXhandle that it is to an hdf5 file and then use a member of the NXhandle directly with the hdf5 api? 2. Can one open the same file with hdf5 and nexus and carry around both handles at the same time? This would probably require using ?reopen? and being careful about access to make it work. 3. Any other ideas? This may indeed by possible. But I would be cautious to complete operations either in direct H5 or NAPI in sequence and not in any order. And make sure that you close all HDF5 handles you use in your direct H5 calls before continuing with NAPI. That is one thing NAPI does for you: keeping track of and closing all the handles required by H5. Now, the NXhandle you get from an NXopen() really points to a NexusFile structure as defined in napi_internal.h. The first most member of this structure is another handle pointing to a NexusFile5 structure as defined at the top of napi5.c. And this NexusFile5 structure contains the FID, the HDF5 file ID, you are looking for. You probably want to modify your NAPI somewhat in order to make yourself a helper function which retrieves the FID for you. Is that enough to get you going? Best Regards, Mark Koennecke P^2 _______________________________________________ NeXus mailing list NeXus at nexusformat.org https://lists.nexusformat.org/mailman/listinfo/nexus -------------- next part -------------- An HTML attachment was scrubbed... URL: From petersonpf at ornl.gov Fri Sep 8 20:34:50 2023 From: petersonpf at ornl.gov (Peterson, Peter) Date: Fri, 8 Sep 2023 19:34:50 +0000 Subject: [Nexus] Migration from nexus API to hdf5 API In-Reply-To: References: Message-ID: That was the sort of information I was hoping for. I'll probably have follow-up questions once I dig in properly. From: NeXus On Behalf Of Koennecke Mark via NeXus Sent: Friday, September 8, 2023 3:13 AM To: Discussion forum for the NeXus data format Cc: Koennecke Mark Subject: [EXTERNAL] Re: [Nexus] Migration from nexus API to hdf5 API Peter, Am 07.09.2023 um 21:53 schrieb Peterson, Peter via NeXus >: I am looking to migrate a code from using the nexus API to directly using hdf5. Changing everything all at once is a ton of work. The ideas I've had so far for a more gradual migration: 1. Is there a way to check an NXhandle that it is to an hdf5 file and then use a member of the NXhandle directly with the hdf5 api? 2. Can one open the same file with hdf5 and nexus and carry around both handles at the same time? This would probably require using "reopen" and being careful about access to make it work. 3. Any other ideas? This may indeed by possible. But I would be cautious to complete operations either in direct H5 or NAPI in sequence and not in any order. And make sure that you close all HDF5 handles you use in your direct H5 calls before continuing with NAPI. That is one thing NAPI does for you: keeping track of and closing all the handles required by H5. Now, the NXhandle you get from an NXopen() really points to a NexusFile structure as defined in napi_internal.h. The first most member of this structure is another handle pointing to a NexusFile5 structure as defined at the top of napi5.c. And this NexusFile5 structure contains the FID, the HDF5 file ID, you are looking for. You probably want to modify your NAPI somewhat in order to make yourself a helper function which retrieves the FID for you. Is that enough to get you going? Best Regards, Mark Koennecke P^2 _______________________________________________ NeXus mailing list NeXus at nexusformat.org https://lists.nexusformat.org/mailman/listinfo/nexus -------------- next part -------------- An HTML attachment was scrubbed... URL: