[NeXus-tech] Common Data Model API (CDMAPI) [SEC=UNCLASSIFIED]

[HAUSER, Nick]

It seems that the understanding of the CDM has diverged from our current implementation. I'm putting together some documentation to try to explain the CDM, but it seems we might need to get together at NOBUGS/NIAC and do a code review, or for you to have a closer look at what ANSTO currently have up on SVN at codehaus.org to appreciate what we've done.

In production at ANSTO, we use the NAPI which is compiled into Mark's SICS instrument control software to write NeXus files, using our interpretation of the definitions at nexusformat.org. We read the same files into Gumtree, our Java control, data reduction and visualisation app, using the CDM. In our implementation it uses the Java HDF api to do the file IO, but you could use the NAPI, which is what Soleil are doing. 

We use only one CDM API for all our instruments, and pass the dictionary as a config. To retain the flexibility of NeXus, the dictionary can be whatever file structure you like. It saves writing a file reader for every instrument, you just define a dictionary. The idea is the same as nxdict. The CDM is our in-memory representation of a NeXus file. The CDM also has array manipulation, maths and curve fitting. If you like, it is a scientific data model. It allows us to do instrument correction, data reduction and finally visualisation. So the CDM extends the NAPI. The CDM doesn't replace or seek to be an alternative to the NAPI. 

This is only a brief summary. I hope it hasn't confused the debate even further.

Regards,

Nick.


-----Original Message-----
From: nexus-tech-bounces at nexusformat.org on behalf of Tobias Richter
Sent: Fri 16/07/2010 00:43
To: NeXus Technical sub-committee list
Subject: Re: [NeXus-tech] Common Data Model API (CDMAPI)
 
That's a funny way of doing it. Here are some of my thoughts in no
particular order:

The idea that using those "tags/attributes" would work is just an idea.
The example on slide 10 had the tags 'pitch' and 'roll'. What is that
then?

It pushes a lot of effort into the API, because writing software is easy
and quick. Sure, everyone knows that.

With some legacy data the mapping between "tags/attributes" and internal
representation isn't straight forward you may end up being close to
distributing an API per file. That's handy, because you can store any
information potentially missing in the legacy file in the API then.

When you receive a binary blob, you've got no way of knowing which
plugin to use or where to find it.

Even with the plugin I am not sure how to work out what has been done in
the experiment. (I am writing this not knowing the API or the tagging
scheme to be fair.)

You cannot archive data that way long term.

Regards,
Tobias

On Wed, 2010-07-14 at 17:51 +0100, freddie.akeroyd at stfc.ac.uk wrote:
> At a recent PANDATA meeting (http://www.pan-data.net/) Alain Buteau from Soleil gave a presentation on using the CDMAPI as an alternative way of sharing data between facilities. Mark and I were asked to start a discussion about this approach as one of the work packages of PanDATA is "Development of standards for scientific data" which aims at defining a common file format (e.g. NeXus + application definitions) for use across the facilities. We have gathered together some points which we plan to pass onto Alain before starting a discussion on the PANDATA list (my knowledge of CDMAPI is limited to the slides so the comparison below may not be the whole story). I enclose the presentation on CDMAPI and would be grateful if you could have a quick look through and let us know if we have missed any important advantages/disadvantages/comments about the two approaches to data sharing/interoperability. 
> 
> Thanks,
> 
> Freddie & Mark
> 
> ------------
> 
> The "common data API" and "nexus instrument definition" are two ways of solving the same problem, both with advantages and disadvantages. The aim of both is to make data analysis easier by allowing access to known data items and thus allow transfer of programs between facilities. In NeXus standardisation is at the file level whereas with the CDM API it is at the library level.
>  
> NeXus Approach:
> - File based on HDF5
> - Generic NeXus API can access any parts of file
> - An "application definition" specifies where to find items for a particular type of analysis. Using links and subentries, a file can support multiple application definitions ("views")
> - An NXdict API allows run-time alias for parts of a NeXus file to be specified, so data can be accessed using simple names/tags if required
> - The NXtranslate application can assemble a new nexus file from other nexus files and, via plugins, older non-nexus files too
> 
> Advantages: 
> - HDF5 tools as well as the NeXus API can be used to read or generate files; there may be no need to install any specific API on your computer and you can use the HDF5 language bindings available for many languages and platforms. There is less need for NeXus to maintain bindings itself. 
> - HDF5 is platform independent and an ISO standard (data providence)
> - As the NeXus/HDF5 format is both extensible and self describing, any additional information added to the file is easily accessible via either the NeXus or HDF5 API.
> - HDF5 supports compression and can be much more efficient than legacy file formats
> - The NXopenpath function plus external file support allows you to rapidly access data in a single or externally linked NeXus file via a simple path string (this string could be shortened using NXdict)
> 
> Disadvantages:
> - It is important to get the application definitions right and not change them too often: a file binding is a "static binding" and changing a definition may mean changing the file format
> - You need to convert your old files to NeXus format using e.g. NXtranslate
> 
> 
> 
> CDM API approach:
> - Build a library that supports access to known data items via a dictionary of tags. This library will require plugins for all supported files formats. 
> 
> Advantages:
> - Logic in API and programming code is easier to change than static files formats
> - You can do additional data manipulations in the API
> - No need to convert old files as reading is handled by special plugins for each format 
> 
> Disadvantages:
> - No incentive to move away from legacy file formats
> - If the "data format plugin" does a lot of conversion work, this work is duplicated each time the file is used and may become a performance penalty
> - May need to maintain many different language bindings 
> - May need to maintain data format plugins on many different platforms and into the future
> - Always need to install the CDM API locally 
> - Can only see parts of the file made visible by the "tags" - other bits are hidden. Hence a non data analysis application that needs access to parts of the file that are not currently exported cannot use CDMAPI
> 
> 
> 
> Notes:
> - NeXus requires agreement between interested parties as to the layout of the file (the "application definition"). CDM requires a similar agreement as to the names and meaning of the tags to be made available. These two sets of information are equivalent and would take the same amount of work to do.
> - The work required to write an NXtranslate plugin is likely comparable to that to write a CDMAPI plugin 
> 
> 
> 
> 
> _______________________________________________
> NeXus-tech mailing list
> NeXus-tech at nexusformat.org
> http://lists.nexusformat.org/mailman/listinfo/nexus-tech


-- 
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. 
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
 




_______________________________________________
NeXus-tech mailing list
NeXus-tech at nexusformat.org
http://lists.nexusformat.org/mailman/listinfo/nexus-tech

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.nexusformat.org/mailman/private/nexus-tech/attachments/20100716/00bb7ec1/attachment-0001.htm>