[Nexus] proposed additions to NXdata for non-linear scaling - to aid cbf interoperability

[jonathan.sloan at diamond.ac.uk]

Hi,

I have run into a limitation in the allowable scaling methods in NXdata when compared to those that CBF offers.

>From what I understand, nexus currently defines the fields 'scaling_factor' and 'offset' within NXdata. These appear to be defined so that the data which is stored must be scaled linearly, and its true value can be recovered by calculating:

value = offset + scaling_factor * data

In comparison, CBF allows 'sqrt' and 'logarithmic' scaling of the raw data which it stores. It isn't possible to directly convert data in either of these forms into the linearly scaled data which must be stored in nexus by simply calculating the appropriate scaling factors and offsets. The data must first be extracted and re-packed to avoid losing information and/or changing its meaning, but this is inefficient and could easily cause file size to be needlessly increased.

I have also been informed that similar non-linear scaling methods may be used in future detectors.

I think this limitation/inefficiency can easily be avoided by extending the allowable scaling methods in nexus, so I propose adding to the NXdata base class:

- An 'exponent' field.
This would be a one dimensional field storing one floating point value, without units, for each frame of data.
A field of size 1 would imply that the given value is applied to each frame - as an alternative to storing the same value many times.
If the field is not present it may be assumed to have a value of '2' for each frame - purely for ease of converting CBF files.

- A 'base' field.
This would be a one dimensional field storing one floating point value, without units, for each frame of data.
A field of size 1 would imply that the given value is applied to each frame - as an alternative to storing the same value many times.
If the field is not present it may be assumed to have a value of '10' for each frame - again, purely for ease of converting CBF files.

- A 'scaling_method' field:
This would be a one-dimensional field storing one of several pre-defined values, which would include 'linear', 'power' and 'exponential', per frame of data.
A field of size 1 would imply that the given value is applied to each frame - as an alternative to storing the same value many times.
If the field is not present it may be assumed to have a value of 'linear' for each frame, which should allow compatibility with older nexus files.

'linear' scaling would be defined in same way as the current scaling method, for compatibility with existing nexus files:
value = offset + scaling_factor * data.
'power' scaling would use the 'exponent' field to define the values in the associated frame as:
value = offset + scaling_factor * ( data ^ exponent ).
'exponential' scaling would use the 'base' field to define the values in the associated frame as:
value = offset + scaling_factor * ( base ^ data ).


This would allow a wider range of CBF files to be converted to nexus efficiently and without loss of information, as well as increasing the expressive power of nexus, without breaking compatibility with existing nexus files.

I believe that this should be an addition to the base class as it would alter the interpretation of the 'data' field in cases where a non-default scaling method is used.

Thanks.


-- 
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. 
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom