[Nexus] Float imaging data in NXtomo and NXtomoproc

[Paul Millar]

Hi Sam,

On 03/02/2023 17:03, Sam Tygier - STFC UKRI via NeXus wrote:
> I am working on Mantid Imaging which features reading and writing of
> neutron imaging data from NXtomo and NXtomoproc. >
> While the data from the instrument usually arrives as uint16 we do
> our processing in, and reconstruct to float32. The NXtomo and
> NXtomoproc definitions both specify the data array as NX_INT.

Just for clarity, I believe NXtomo has the following "data" arrays:

     NXtomo/entry/instrument/detector/data
     NXtomo/entry/control/data
     NXtomo/entry/data

Only NXtomo/entry/instrument/detector/data limits the values to NX_INT, 
the other two have NX_FLOAT and NX_NUMBER, respectively.

In NXtomoproc, there is only NXtomoproc/entry/data/data, which is 
(indeed) limited to allow only NX_INT values.


> Ideally, we'd like to be able to store our datasets in float32, both
> for reconstructed output and intermediate processed projections. We
> do this with TIFFs and it avoids any loss of precision.

I have limited knowledge of tomography processing, but you desire here 
makes sense to me.


> Is it ok to store float32 in these arrays? I assume this is bad
> because the files would not be valid and other software may not be
> able to correctly open them.

Yes, I believe it would be non-conforming to store float16 data directly 
under

	NXtomo/entry/instrument/detector/data



> Could the definitions be updated to use NX_NUMBER? What's the best 
> way to propose this change?

There is an issue tracker that may be used to propose changes or ask 
general questions:

	https://github.com/nexusformat/definitions/issues

I would suggest opening an issue, describing your use-case and asking 
for support.

> The NXtomoproc data array has attributes 'transform', 'offset',
> 'scaling'. Are these to specify int-float conversion? Is there any
> documentation? 

NXtomoproc does seem to suffer from being badly documented.  There's an 
obvious typo in NXtomoproc/entry/data/data "Dimensions".

 From reading the documentation, the meaning of @transform, @offset and 
@scaling is (to me) completely opaque.

Since they target the "data" field, I *guess* that these attributes are 
indented to support quantised/binned data, and describing how to 
transform the recorded integer values back to a corresponding 
floating-point representation.

However, that's really just a guess.

> Could these be added to NXtomo?
As I understand it, NXtomo already supports float numbers under

     NXtomo/entry/data

Again, the exact semantics of the different fields within NXtomo are 
somewhat opaque to me, give NXtomo is described as containing raw data 
and has two entries where (at least, to my naive understanding) raw data 
might reasonably be stored:

     NXtomo/entry/instrument/detector/data
     NXtomo/entry/data

At the least, I think there should be a task to tidy up the document for 
NXtomo and NXtomoproc, to make these things clearer.

Cheers,
Paul.