[Nexus] proposed additions to NXdata for non-linear scaling - to aid cbf interoperability

[Benjamin Watts]

Hi Jonathan,
    This kind of thing has been discussed at previous NIAC meetings. 
Your proposal makes for a simple change, but does more to push the 
problem further away from yourself than to actually solve it. What we 
really need is some "standard" way to present mathematical formulae in 
general that is accessible across different programming languages. I 
think maybe Eugen Wintersberger had a good suggestion, but I don't 
recall any follow-up on it.

Cheers,
Ben


On 18/02/14 16:42, jonathan.sloan at diamond.ac.uk wrote:
> Hi,
>
> I have run into a limitation in the allowable scaling methods in NXdata when compared to those that CBF offers.
>
>  From what I understand, nexus currently defines the fields 'scaling_factor' and 'offset' within NXdata. These appear to be defined so that the data which is stored must be scaled linearly, and its true value can be recovered by calculating:
>
> value = offset + scaling_factor * data
>
> In comparison, CBF allows 'sqrt' and 'logarithmic' scaling of the raw data which it stores. It isn't possible to directly convert data in either of these forms into the linearly scaled data which must be stored in nexus by simply calculating the appropriate scaling factors and offsets. The data must first be extracted and re-packed to avoid losing information and/or changing its meaning, but this is inefficient and could easily cause file size to be needlessly increased.
>
> I have also been informed that similar non-linear scaling methods may be used in future detectors.
>
> I think this limitation/inefficiency can easily be avoided by extending the allowable scaling methods in nexus, so I propose adding to the NXdata base class:
>
> - An 'exponent' field.
> This would be a one dimensional field storing one floating point value, without units, for each frame of data.
> A field of size 1 would imply that the given value is applied to each frame - as an alternative to storing the same value many times.
> If the field is not present it may be assumed to have a value of '2' for each frame - purely for ease of converting CBF files.
>
> - A 'base' field.
> This would be a one dimensional field storing one floating point value, without units, for each frame of data.
> A field of size 1 would imply that the given value is applied to each frame - as an alternative to storing the same value many times.
> If the field is not present it may be assumed to have a value of '10' for each frame - again, purely for ease of converting CBF files.
>
> - A 'scaling_method' field:
> This would be a one-dimensional field storing one of several pre-defined values, which would include 'linear', 'power' and 'exponential', per frame of data.
> A field of size 1 would imply that the given value is applied to each frame - as an alternative to storing the same value many times.
> If the field is not present it may be assumed to have a value of 'linear' for each frame, which should allow compatibility with older nexus files.
>
> 'linear' scaling would be defined in same way as the current scaling method, for compatibility with existing nexus files:
> value = offset + scaling_factor * data.
> 'power' scaling would use the 'exponent' field to define the values in the associated frame as:
> value = offset + scaling_factor * ( data ^ exponent ).
> 'exponential' scaling would use the 'base' field to define the values in the associated frame as:
> value = offset + scaling_factor * ( base ^ data ).
>
>
> This would allow a wider range of CBF files to be converted to nexus efficiently and without loss of information, as well as increasing the expressive power of nexus, without breaking compatibility with existing nexus files.
>
> I believe that this should be an addition to the base class as it would alter the interpretation of the 'data' field in cases where a non-default scaling method is used.
>
> Thanks.
>
>