[NeXus-tech] Common Data Model API (CDMAPI)

[Tobias Richter]

That's a funny way of doing it. Here are some of my thoughts in no
particular order:

The idea that using those "tags/attributes" would work is just an idea.
The example on slide 10 had the tags 'pitch' and 'roll'. What is that
then?

It pushes a lot of effort into the API, because writing software is easy
and quick. Sure, everyone knows that.

With some legacy data the mapping between "tags/attributes" and internal
representation isn't straight forward you may end up being close to
distributing an API per file. That's handy, because you can store any
information potentially missing in the legacy file in the API then.

When you receive a binary blob, you've got no way of knowing which
plugin to use or where to find it.

Even with the plugin I am not sure how to work out what has been done in
the experiment. (I am writing this not knowing the API or the tagging
scheme to be fair.)

You cannot archive data that way long term.

Regards,
Tobias

On Wed, 2010-07-14 at 17:51 +0100, freddie.akeroyd at stfc.ac.uk wrote:
> At a recent PANDATA meeting (http://www.pan-data.net/) Alain Buteau from Soleil gave a presentation on using the CDMAPI as an alternative way of sharing data between facilities. Mark and I were asked to start a discussion about this approach as one of the work packages of PanDATA is "Development of standards for scientific data" which aims at defining a common file format (e.g. NeXus + application definitions) for use across the facilities. We have gathered together some points which we plan to pass onto Alain before starting a discussion on the PANDATA list (my knowledge of CDMAPI is limited to the slides so the comparison below may not be the whole story). I enclose the presentation on CDMAPI and would be grateful if you could have a quick look through and let us know if we have missed any important advantages/disadvantages/comments about the two approaches to data sharing/interoperability. 
> 
> Thanks,
> 
> Freddie & Mark
> 
> ------------
> 
> The "common data API" and "nexus instrument definition" are two ways of solving the same problem, both with advantages and disadvantages. The aim of both is to make data analysis easier by allowing access to known data items and thus allow transfer of programs between facilities. In NeXus standardisation is at the file level whereas with the CDM API it is at the library level.
>  
> NeXus Approach:
> - File based on HDF5
> - Generic NeXus API can access any parts of file
> - An "application definition" specifies where to find items for a particular type of analysis. Using links and subentries, a file can support multiple application definitions ("views")
> - An NXdict API allows run-time alias for parts of a NeXus file to be specified, so data can be accessed using simple names/tags if required
> - The NXtranslate application can assemble a new nexus file from other nexus files and, via plugins, older non-nexus files too
> 
> Advantages: 
> - HDF5 tools as well as the NeXus API can be used to read or generate files; there may be no need to install any specific API on your computer and you can use the HDF5 language bindings available for many languages and platforms. There is less need for NeXus to maintain bindings itself. 
> - HDF5 is platform independent and an ISO standard (data providence)
> - As the NeXus/HDF5 format is both extensible and self describing, any additional information added to the file is easily accessible via either the NeXus or HDF5 API.
> - HDF5 supports compression and can be much more efficient than legacy file formats
> - The NXopenpath function plus external file support allows you to rapidly access data in a single or externally linked NeXus file via a simple path string (this string could be shortened using NXdict)
> 
> Disadvantages:
> - It is important to get the application definitions right and not change them too often: a file binding is a "static binding" and changing a definition may mean changing the file format
> - You need to convert your old files to NeXus format using e.g. NXtranslate
> 
> 
> 
> CDM API approach:
> - Build a library that supports access to known data items via a dictionary of tags. This library will require plugins for all supported files formats. 
> 
> Advantages:
> - Logic in API and programming code is easier to change than static files formats
> - You can do additional data manipulations in the API
> - No need to convert old files as reading is handled by special plugins for each format 
> 
> Disadvantages:
> - No incentive to move away from legacy file formats
> - If the "data format plugin" does a lot of conversion work, this work is duplicated each time the file is used and may become a performance penalty
> - May need to maintain many different language bindings 
> - May need to maintain data format plugins on many different platforms and into the future
> - Always need to install the CDM API locally 
> - Can only see parts of the file made visible by the "tags" - other bits are hidden. Hence a non data analysis application that needs access to parts of the file that are not currently exported cannot use CDMAPI
> 
> 
> 
> Notes:
> - NeXus requires agreement between interested parties as to the layout of the file (the "application definition"). CDM requires a similar agreement as to the names and meaning of the tags to be made available. These two sets of information are equivalent and would take the same amount of work to do.
> - The work required to write an NXtranslate plugin is likely comparable to that to write a CDMAPI plugin 
> 
> 
> 
> 
> _______________________________________________
> NeXus-tech mailing list
> NeXus-tech at nexusformat.org
> http://lists.nexusformat.org/mailman/listinfo/nexus-tech


-- 
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. 
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom