[Nexus] proposed additions to NXdata for non-linear scaling - to aid cbf interoperability

[Mark Koennecke]

Hi all,

On 02/20/2014 05:30 PM, Benjamin Watts wrote:
> Hi Jonathan,
>    I think this is too big an issue (as in possibly creating a lot of 
> implementation and maintenance work) to jump into like that. I don't 
> think the NIAC is ready to committing itself to integral use of this 
> NXformula. We need to better understand how the details would work and 
> how much machinery would be required in total. Pete Jemian 
> specifically said that he was opposed to forcing all NeXus-reading 
> libraries to implement something so general as this sounds - and I 
> have to agree. If you want to volunteer to further develop and 
> demonstrate the idea so that we can know that NXformula won't extend 
> the scope of NeXus beyond what we can implement/support, then that's 
> great, but be mindful of creating something that may only be supported 
> by half the community (and thus breaking the NeXus standard).
>
I second that. There are various reasons  why NeXus has not got in with 
formulas:
- There is the strong preference to store data in a form which directly 
makes sense. The formula thing is for
    exceptional use cases where you need to stream the data so quickly 
that you cannot convert.
- Anything with formulas, especially complex ones, kick up the question 
how to apply them. You are then compelled
   to provide a reading tool which does the conversion. This looks 
trivial but is not: if you stream data data so fast
   that you cannot convert on the fly while writing, then we are talking 
big data. This applies that any formula
   application has to be efficient, possibly even parallel. Anyone 
volunteering to implement this?
- For the time being: why not use some attributes to implement what CBF 
is doing now. May be with muParser syntax.
   This allows the CBF-NeXus thing to go forward. For the next NIAC we 
table the NXformula thing again for discussion.
   It will take only a little python script to convert anything written 
in the old form to the then agreed NXformula way.

Personally, I am very reluctant to commit to some formula scheme without 
thorough discussion.

Best Regards,

        Mark
> Cheers,
> Ben
>
> On 20/02/14 17:05, jonathan.sloan at diamond.ac.uk wrote:
>> Hi,
>>
>> I've looked at muParser in a bit more detail now, it appears that 
>> while it isn't a completely general solution with full support for 
>> tensor operations (like MATLAB, for example) it does have a 'bulk' 
>> mode which would allow expressions to be defined per-frame and data 
>> to be extracted with per-frame scaling factors and even per-frame 
>> expressions. Limiting expressions to applying a uniform expression to 
>> an entire frame will still allow it to cover everything that the 
>> cbf-nexus conversions might use.
>>
>> This would mean that (in the case of rank-3 data) 'NXforumla' could 
>> contain a rank-1 tensor of expressions, which would be passed a set 
>> of rank-1 parameters. The actual values of the data could be 
>> extracted by calculating 'value[j,k] = f(data[i,j,k],var[i]...)' for 
>> each frame 'i' of dimensions 'j,k'. Higher dimensional data could be 
>> converted by extending this to 'value[k,l] = 
>> f(data[i,j,k,l],var[i,j]...)' for each frame 'i,j' of dimensions 
>> 'k,l', and so on for higher dimensions. I think it should largely be 
>> a matter of plumbing to implement this, rather than adding any 
>> functionality which doesn't currently exist.
>>
>> Since everything required should already be implemented in the 
>> muParser library, could we start using this when converting cbf data 
>> and refine the implementation later on once an 'NXforumla' class 
>> definition has been standardised?
>>
>> Thanks.
>>
>
> _______________________________________________
> NeXus mailing list
> NeXus at nexusformat.org
> http://lists.nexusformat.org/mailman/listinfo/nexus