[NeXus-tech] Common Data Model API (CDMAPI)

[freddie.akeroyd at stfc.ac.uk]

At a recent PANDATA meeting (http://www.pan-data.net/) Alain Buteau from Soleil gave a presentation on using the CDMAPI as an alternative way of sharing data between facilities. Mark and I were asked to start a discussion about this approach as one of the work packages of PanDATA is "Development of standards for scientific data" which aims at defining a common file format (e.g. NeXus + application definitions) for use across the facilities. We have gathered together some points which we plan to pass onto Alain before starting a discussion on the PANDATA list (my knowledge of CDMAPI is limited to the slides so the comparison below may not be the whole story). I enclose the presentation on CDMAPI and would be grateful if you could have a quick look through and let us know if we have missed any important advantages/disadvantages/comments about the two approaches to data sharing/interoperability. 

Thanks,

Freddie & Mark

------------

The "common data API" and "nexus instrument definition" are two ways of solving the same problem, both with advantages and disadvantages. The aim of both is to make data analysis easier by allowing access to known data items and thus allow transfer of programs between facilities. In NeXus standardisation is at the file level whereas with the CDM API it is at the library level.
 
NeXus Approach:
- File based on HDF5
- Generic NeXus API can access any parts of file
- An "application definition" specifies where to find items for a particular type of analysis. Using links and subentries, a file can support multiple application definitions ("views")
- An NXdict API allows run-time alias for parts of a NeXus file to be specified, so data can be accessed using simple names/tags if required
- The NXtranslate application can assemble a new nexus file from other nexus files and, via plugins, older non-nexus files too

Advantages: 
- HDF5 tools as well as the NeXus API can be used to read or generate files; there may be no need to install any specific API on your computer and you can use the HDF5 language bindings available for many languages and platforms. There is less need for NeXus to maintain bindings itself. 
- HDF5 is platform independent and an ISO standard (data providence)
- As the NeXus/HDF5 format is both extensible and self describing, any additional information added to the file is easily accessible via either the NeXus or HDF5 API.
- HDF5 supports compression and can be much more efficient than legacy file formats
- The NXopenpath function plus external file support allows you to rapidly access data in a single or externally linked NeXus file via a simple path string (this string could be shortened using NXdict)

Disadvantages:
- It is important to get the application definitions right and not change them too often: a file binding is a "static binding" and changing a definition may mean changing the file format
- You need to convert your old files to NeXus format using e.g. NXtranslate



CDM API approach:
- Build a library that supports access to known data items via a dictionary of tags. This library will require plugins for all supported files formats. 

Advantages:
- Logic in API and programming code is easier to change than static files formats
- You can do additional data manipulations in the API
- No need to convert old files as reading is handled by special plugins for each format 

Disadvantages:
- No incentive to move away from legacy file formats
- If the "data format plugin" does a lot of conversion work, this work is duplicated each time the file is used and may become a performance penalty
- May need to maintain many different language bindings 
- May need to maintain data format plugins on many different platforms and into the future
- Always need to install the CDM API locally 
- Can only see parts of the file made visible by the "tags" - other bits are hidden. Hence a non data analysis application that needs access to parts of the file that are not currently exported cannot use CDMAPI



Notes:
- NeXus requires agreement between interested parties as to the layout of the file (the "application definition"). CDM requires a similar agreement as to the names and meaning of the tags to be made available. These two sets of information are equivalent and would take the same amount of work to do.
- The work required to write an NXtranslate plugin is likely comparable to that to write a CDMAPI plugin 




-- 
Scanned by iCritical.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: SOLEILCommonDAtaModelAPI.pdf
Type: application/pdf
Size: 830872 bytes
Desc: SOLEILCommonDAtaModelAPI.pdf
URL: <http://lists.nexusformat.org/mailman/private/nexus-tech/attachments/20100714/f1d17abe/attachment-0001.pdf>