[Nexus] Advice on using NeXus for neuroscience data?

[Osborn, Raymond]

Jeff,
That’s an extremely interesting question that doesn’t really have an easy answer. NeXus represents a way of organizing HDF5 files that could inspire similar efforts in other fields. Essentially, we try to represent objects, in our case the sample being measured, components of a beamline, etc. by collecting their metadata into groups of a particular class (which is defined by an HDF5 group attribute). We also collect all the data representing the results of an experiment in separate groups of class NXdata so that they are automatically plottable. Both of those aspects of the format could be transferable to other techniques. Indeed, we already have one example, in that muon spin resonance is now part of the standard. There is also some interest in the electron microscopy community. The advantage is that there could be some generic tools that could work in multiple fields, e.g., the NXDL language for defining class structures and the Python API for manipulating HDF5 files.

On the other hand, the specific classes defined within the format and the glossary of field names that can go in those classes are clearly specific to scattering, and there is probably insufficient overlap in our scientific fields for it to make sense to merge. We would have to document the neurophysiology classes separately from the neutron and x-ray classes so as not to cause confusion. That could perhaps be achieved through having completely separate branches of the format.

In a way, NeXus is just a way of adding semantics to HDF5, making the files self-describing in a way that HDF5 files are not guaranteed to be. There may be value in multiple groups using the same design philosophy and sharing similar tools, and we would probably benefit if the community were enlarged. I’m sure we would be happy to give further advice if you decide to pursue this further.

With regards,
Ray

On Oct 1, 2014, at 4:56 PM, Jeff Teeters <jteeters at berkeley.edu<mailto:jteeters at berkeley.edu>> wrote:

Hi,
I'm helping to coordinate a project (called "Neurodata Without Borders", described below), which has the goal to develop a standard file format for neurophysiology data.   It seems likely that HDF5 will be used as the basis for the standard format, but the organization of data and metadata within the HDF5 file still must be determined.  I'm interested in any opinions or input about the following:
a. Is NeXus potentially useful for domains unrelated to particle physics experiments?
b. Are there any examples of NeXus being used in other domains?
c. What other methods of specifying the organization of data in an HDF5 file might be applicable for this project?
Thanks,
Jeff Teeters
Redwood Center for Theoretical Neuroscience
UC Berkeley


---------- Forwarded message ----------
From: Jeff Teeters <jteeters at berkeley.edu<mailto:jteeters at berkeley.edu>>
Date: Mon, Aug 11, 2014 at 11:47 PM
Subject: Seeking community input for Neurodata Without Borders
To: Jeff Teeters <jteeters at berkeley.edu<mailto:jteeters at berkeley.edu>>
Cc: Fritz Sommer <fsommer at berkeley.edu<mailto:fsommer at berkeley.edu>>


Dear Colleague,

The Neurodata Without Borders (NWB) project has just started. The project goal is to build a common data format for neurophysiology data from Allen Brain Institute, Janelia Farm and two labs from NYU and Caltech.  Although focusing on a limited set of use cases, the project also aims to develop products that will serve the broader community.

At this point we would like to solicit community input about ideas/approaches for designing a generalizable neurophysiology data format. If you are interested in contributing to this project, please review the project description at:
https://crcns.org/NWB
and fill out the questionnaire:
https://docs.google.com/forms/d/1CNTd4M-8kQ_WhEZc7n7WxpTa0LOupt_q3z21E1fRxjM/viewform

On the basis of the questionnaire responses and ensuing communication, we will organize the first hackathon meeting of the project, to be held November 20 – 22, 2014 (just after SfN) at Janelia Farm, in Ashburn, Virginia.  At this hackathon we will discuss in detail the requirements for a common format based on the project use cases and also discuss, compare and evaluate alternative techniques for implementing the common format.

More information about the project is available in a recent press release: http://www.kavlifoundation.org/kavli-news/prominent-us-research-institutions-announce-collaboration-toward-sharing-and

Please forward this email to anyone you know with relevant expertise who may be interested in contributing to this project.

Thank you,
Fritz Sommer and Jeff Teeters
Redwood Center for Theoretical Neuroscience
UC Berkeley
_______________________________________________
NeXus mailing list
NeXus at nexusformat.org<mailto:NeXus at nexusformat.org>
http://lists.nexusformat.org/mailman/listinfo/nexus

--
Ray Osborn, Senior Scientist
Materials Science Division
Argonne National Laboratory
Argonne, IL 60439, USA
Phone: +1 (630) 252-9011
Email: ROsborn at anl.gov<mailto:ROsborn at anl.gov>


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.nexusformat.org/pipermail/nexus/attachments/20141001/7f1f89a5/attachment.html>