[NeXus-committee] Concordance paper, ECM
Tobias.Richter at diamond.ac.uk
Tobias.Richter at diamond.ac.uk
Wed Apr 24 14:02:58 BST 2013
I may recall that incorrectly but as of December or so we (NIAC) have agreed everything that Dectris has asked for.
That includes gain_setting (["fast", "medium", "slow", "auto"]). The Pilatus3 only has "auto" and I assume the same is true for an Eiger. It may still be a useful field for older detectors or other vendors. But they now cannot record a gain_setting as a float between 0 and 1 any more. Things like that change and we should allow some freedom, especially for diagnostic areas when this information is mainly for human consumption.
Cheers,
Tobias
________________________________________
From: yayahjb [yayahjb at gmail.com]
Sent: 24 April 2013 13:08
To: Richter, Tobias (DLSLtd,RAL,DIA)
Cc: Mark.Koennecke at psi.ch; nexus-committee at nexusformat.org; marcus.mueller at dectris.com; Winter, Graeme (DLSLtd,RAL,DIA); Sloan, Jonathan (DLSLtd,RAL,DIA); sweet at bnl.gov; michael.rissi at dectris.com
Subject: Re: [NeXus-committee] Concordance paper, ECM
Sorry, I'll change "groups" to "NeXus classes". I suspect that almost
everything that Dectris
has done for the Eiger will need to be supported in general for
pixel-array detectors -- just
that there will be additional terms to deal with as other vendors get
into the NeXus act. What we
have so far is what Dectris would like. To me what they are asking for
looks reasonable and
general. I would suggest we ratify that. So, if everybody is
agreeable, I would suggest
we add the top level items directly into NXdetector and change the
"_dectris_eiger" suffix
on the lower lever NeXus classes to "_pad" keeping the NX prefix.
On the completeness of the CIF mapping -- please recall that this is
just what is needed
to be able to build a database for being able to combine multiple
experiments in an
overall data management system, both at facilities and for archiving.
Everybody loses
if we don't encourage a common ontology for those purposes. This will
become
particularly important if we end up with a distributed system of image
archiving
in support of journal refereeing of crystallographic papers -- which is
increasingly
likely. With the issue of fraud in crystallography to deal with, having
all the details
on, say, exact module positioning available, while not needed to get a
structure can
be useful in recognizing fraudulent images.
What I am saying is that giving "any vendor the freedom to progress in
technology
without having to consult us," should not be interpreted as encouraging
two vendors
to use the same terms with different meanings or two different terms for
exactly the
same concept.. By taking whatever terms each vendor settles on and, to
the greatest
extent possible, incorporating them in the standard as they get settled
on, we should
reduce the chances of such conflicts. This is _not_ a matter which of
CIF or NeXus
or HDF5 or ..... is best. It is a matter of having a common a
vocabulary as possible
where possible, and as clear faithful translations as possible for the
rest. We may not
be able to do things perfectly and we will never be able to do things
completely, but
where we can settle part of the problem-- as we can and should with the
Eiger --
we should try.
Regards,
Herbert
On 4/24/13 7:20 AM, Tobias.Richter at diamond.ac.uk wrote:
> There is some confusion about the terms here. What is described in this document are classes (groups in the HDF5 universe).
> Application definitions standardise the entire tree below NXentry (or NXsubentry) for a particular experimental technique (or application, hence the name).
>
> Currently there are no vendor specific classes (standardised) and my understanding was that it wasn't planned to introduce any. The NIAC has agreed to a good number of fields to be added to NXdetector for the Eiger. Anything that was deemed too detector/vendor specific (by Dectris, I do not recall the NIAC rejecting anything) can be obviously be added to the files, but not as part of the NeXus standard. So vendor specific groups would not have an NX prefix. That gives any vendor the freedom to progress in technology without having to consult us. After all they decided what they wanted to have ratified and what not.
>
> There is nothing wrong with mapping everything into CIF but for the non-standard NeXus parts, Dectris is not forced to comply with the examples for all times.
> Obviously if we feel crucial information required to analyse the data is in kept in these potentially volatile places, we need to address this and define a slot for it.
>
> Kind reagrds,
>
> Tobias
>
> ________________________________________
> From: yayahjb [yayahjb at gmail.com]
> Sent: 24 April 2013 03:49
> To: Richter, Tobias (DLSLtd,RAL,DIA)
> Cc: Mark.Koennecke at psi.ch; nexus-committee at nexusformat.org; Marcus Mueller; Winter, Graeme (DLSLtd,RAL,DIA); Sloan, Jonathan (DLSLtd,RAL,DIA); Sweet, Robert
> Subject: Re: [NeXus-committee] Concordance paper, ECM
>
> I kept going on the Dectris Eiger application definition and CIF mapping.
> I broke up the application definition into one per class and renamed some
> of the classes to begin with NX and avoid conflicts with existing
> definitions
> until we can see if we agree on extending exsiting ones. I have extracted
> just the portion of the 112 page document that deals with the Eiger.
>
> More follows later this week.
> -- Herbert
>
>
> On 4/22/13 7:52 AM, yayahjb wrote:
>
>> I'll go through all Mark's comments carefully later today, but his
>> last one is
>> very important -- I had some typos in the Eiger application definition
>> in addition to the one Mark found. Here is an update, hopefully with
>> fewer errors. Further proofreading would be much appreciated.
>>
>>
>> NXdectris_eiger (application definition, version 0.1)
>> (overlays NXentry)
>> entry:NXentry
>> data_000001:NXDATA
>> data:NXINT[np_000001,number of x pixels,number of y pixels]
>> @image_nr_low
>> @image_nr_high
>> data_000002:NXDATA
>> data:NXINT[np_000001,number of x pixels,number of y pixels]
>> @image_nr_low
>> @image_nr_high
>> ...
>> data_nnnnnn:NXDATA
>> data:NXINT[np_nnnnnn,number of x pixels,number of y pixels]
>> @image_nr_low
>> @image_nr_high
>> instrument:NXinstrument
>> detector:NXdetector
>> acquisition_mode:NX_CHAR
>> angular_calibration_applied:NX_BOOLEAN
>> beam_center_x:NX_FLOAT
>> @units
>> beam_center_y:NX_FLOAT
>> @units
>> bit_depth_readout:NX_UINT
>> count_time:NX_FLOAT[np]
>> @units
>> countrate_correction_applied:NX_BOOLEAN
>> description:NX_CHAR
>> detector_number:NX_CHAR
>> detectorExtended:DetectorExtended
>> detector_distance:NXFLOAT
>> @units
>> rotation_angle_step:NX_FLOAT[np]
>> @units
>> wavelength:NX_FLOAT
>> @units
>> detectorSpecific:DetectorSpecific
>> countrate_correction_bunch_mode:NX_CHAR
>> countrate_correction_count_cutoff:NX_UINT
>> countrate_correction_lookup_table:NX_FLOAT[1000000]
>> data_collection_date:NX_CHAR
>> detectorModule_000:DetectorModule
>> dac_names:NX_CHAR[6]
>> dac_values:NX_UINT[7]
>> data_origin:NX_UINT[2]
>> data_size:NX_UINT[2]
>> detectorChip_00:DetectorChip
>> chip_type:NX_CHAR
>> x_pixels_in_chip:NX_UINT
>> x_position:NX_UINT
>> y_pixels_in_chip:NX_UINT
>> y_position:NX_UINT
>> detectorChip_01:DetectorChip
>> chip_type:NX_CHAR
>> x_pixels_in_chip:NX_UINT
>> x_position:NX_UINT
>> y_pixels_in_chip:NX_UINT
>> y_position:NX_UINT
>> ...
>> detectorChip_15:DetectorChip
>> chip_type:NX_CHAR
>> x_pixels_in_chip:NX_UINT
>> x_position:NX_UINT
>> y_pixels_in_chip:NX_UINT
>> y_position:NX_UINT
>> fast_pixel_direction:NX_FLOAT64
>> @depends_on
>> @transformation
>> @units
>> @vector
>> firmware_version:NX_CHAR
>> module_offset:NX_FLOAT64
>> @depends_on
>> @transformation
>> @units
>> @vector
>> nbits:NX_UINT
>> nchips:NX_UINT
>> readout_frequency:NX_FLOAT
>> @units
>> region_of_interest:NX_UINT[4]
>> slow_pixel_direction:NX_FLOAT64
>> @depends_on
>> @transformation
>> @units
>> @vector
>> x_pixels_in_module:NX_UINT
>> y_pixels_in_module:NX_UINT
>> detectorModule_001:DetectorModule
>> dac_names:NX_CHAR[6]
>> dac_values:NX_UINT[7]
>> data_origin:NX_UINT[2]
>> data_size:NX_UINT[2]
>> detectorChip_00:DetectorChip
>> chip_type:NX_CHAR
>> x_pixels_in_chip:NX_UINT
>> x_position:NX_UINT
>> y_pixels_in_chip:NX_UINT
>> y_position:NX_UINT
>> detectorChip_01:DetectorChip
>> chip_type:NX_CHAR
>> x_pixels_in_chip:NX_UINT
>> x_position:NX_UINT
>> y_pixels_in_chip:NX_UINT
>> y_position:NX_UINT
>> ...
>> detectorChip_15:DetectorChip
>> chip_type:NX_CHAR
>> x_pixels_in_chip:NX_UINT
>> x_position:NX_UINT
>> y_pixels_in_chip:NX_UINT
>> y_position:NX_UINT
>> fast_pixel_direction:NX_FLOAT64
>> @depends_on
>> @transformation
>> @units
>> @vector
>> firmware_version:NX_CHAR
>> module_offset:NX_FLOAT64
>> @depends_on
>> @transformation
>> @units
>> @vector
>> nbits:NX_UINT
>> nchips:NX_UINT
>> readout_frequency:NX_FLOAT
>> @units
>> region_of_interest:NX_UINT[4]
>> slow_pixel_direction:NX_FLOAT64
>> @depends_on
>> @transformation
>> @units
>> @vector
>> x_pixels_in_module:NX_UINT
>> y_pixels_in_module:NX_UINT
>> ...
>> detectorModule_059:DetectorModule
>> dac_names:NX_CHAR[6]
>> dac_values:NX_UINT[7]
>> data_origin:NX_UINT[2]
>> data_size:NX_UINT[2]
>> detectorChip_00:DetectorChip
>> chip_type:NX_CHAR
>> x_pixels_in_chip:NX_UINT
>> x_position:NX_UINT
>> y_pixels_in_chip:NX_UINT
>> y_position:NX_UINT
>> detectorChip_01:DetectorChip
>> chip_type:NX_CHAR
>> x_pixels_in_chip:NX_UINT
>> x_position:NX_UINT
>> y_pixels_in_chip:NX_UINT
>> y_position:NX_UINT
>> ...
>> detectorChip_15:DetectorChip
>> chip_type:NX_CHAR
>> x_pixels_in_chip:NX_UINT
>> x_position:NX_UINT
>> y_pixels_in_chip:NX_UINT
>> y_position:NX_UINT
>> fast_pixel_direction:NX_FLOAT64
>> @depends_on
>> @transformation
>> @units
>> @vector
>> firmware_version:NX_CHAR
>> module_offset:NX_FLOAT64
>> @depends_on
>> @transformation
>> @units
>> @vector
>> nbits:NX_UINT
>> nchips:NX_UINT
>> readout_frequency:NX_FLOAT
>> @units
>> region_of_interest:NX_UINT[4]
>> slow_pixel_direction:NX_FLOAT64
>> @depends_on
>> @transformation
>> @units
>> @vector
>> x_pixels_in_module:NX_UINT
>> y_pixels_in_module:NX_UINT
>> detector_origin:NX_FLOAT64
>> @depends_on
>> @transformation
>> @units
>> @vector
>> detector_origin_rotated:NX_FLOAT64
>> @depends_on
>> @transformation
>> @units
>> @vector
>> flat field:NX_FLOAT[number of x pixels,number of y pixels]
>> imgType:NX_UINT
>> mode_register:NX_UINT
>> nimages:NX_UINT
>> number_of_excluded_pixels:NX_UINT
>> photon_energy:NX_FLOAT
>> pixel_mask:NX_FLOAT[number of x pixels,number of y pixels]
>> readout_mode:NX_CHAR
>> seriedId:NX_ULONG
>> software_version:NX_CHAR
>> sub_image_exposure_time:NX_FLOAT
>> summation_mode:NX_CHAR
>> summation_nimages:NX_UINT
>> trigger_mode:NX_CHAR
>> x_pixels_in_detector:NX_UINT
>> y_pixels_in_detector:NX_UINT
>> detector_number:NX_CHAR
>> detector_readout_time:NX_FLOAT[np]
>> @units
>> efficiency_correction_applied:NX_BOOL
>> flatfield_correction_applied:NX_BOOL
>> frame_time:NX_FLOAT[np]
>> @units
>> gain_setting:NX_CHAR
>> number_of_cycles:NX_UINT
>> pixel_mask_applied:NX_BOOL
>> sensor_material:NX_STRING
>> sensor_thickness:NX_FLOAT
>> @units
>> threshold_energy:NX_FLOAT
>> @units
>> virtual_pixel_correction_applied:NX_BOOL
>> x_pixel_size:NX_FLOAT
>> @units
>> y_pixel_size:NX_FLOAT
>> @units
>>
>>
>>
>> On 4/22/13 6:43 AM, Tobias.Richter at diamond.ac.uk wrote:
>>
>>> Hi all,
>>>
>>> Just to reiterate what Herbert has written in the document: This
>>> draft is a work in progress. In particular this version is one that I
>>> didn't have time to comment on.
>>>
>>> The idea behind this whole operation is to make a minimum of changes
>>> in either format. If we need to preserve information that currently
>>> cannot be expressed we add a place for it. Everything else should go
>>> into existing slots.
>>> In rare cases were the existing slots are not suitable, we may need
>>> to make amends. But I would say we clearly haven exhausted our
>>> options yet. That's why we ask for comments.
>>>
>>> In line with the NeXus philosophy there a number (potentially
>>> optional) fields and attributes may be added to aid a faithful
>>> roundtrip conversion between NeXus and CBF. There is a fair number
>>> of fields from that category and where we aren't mapping correctly
>>> yet, it would probably we helpful to identify which is which.
>>>
>>> Just one comment on a few points raised by Mark (leaving the major
>>> share for Herbert):
>>>
>>> Section 3.2, P 10, Admin Feature Request
>>> ---------------------------------------------------------
>>> We already have experiment_identifier and entry_identifier defined.
>>> With entry = scan (more or less) we would only need a
>>> diffraction_identifier (to keep the same naming convention), which in
>>> my understanding would be a per-frame array. Right?
>>>
>>> Section 3.16, P36, Variant feature
>>> ----------------------------------------------
>>> NXsubentry addresses a different problem. You might also want to
>>> record a variant of something that is in a subentry.
>>> My current idea is that of a NXvariant group that inherits from
>>> NXentry and also live at the root level. It would reference the
>>> NXentry or other NXvariant it operates on (in e.g. an attribute) and
>>> would only carry the item at are changed and the required structure
>>> to hold them.
>>> An alternative would be to link the unchanged entries. It keeps
>>> reading the variant simple in software but that hides the purpose of
>>> that feature and makes the changes less accessible to humans.
>>>
>>>
>>> I am not sure I see much scope for discussing this with the CIF
>>> committee. I am not even sure they have seen this document yet.
>>>
>>> Interest for doing anything in August was low in the Doodle poll. We
>>> have some discussions around the ECM meeting if people change their
>>> mind. For the moment I do have Jonathan devoted to the project. We
>>> have to keep making progress.
>>>
>>> Regards,
>>>
>>> Tobias
>>>
>>> ________________________________________
>>> From: nexus-committee-bounces at nexusformat.org
>>> [nexus-committee-bounces at nexusformat.org] on behalf of Mark Koennecke
>>> [Mark.Koennecke at psi.ch]
>>> Sent: 22 April 2013 10:14
>>> To: Discussions of the NeXus Advisory Committee
>>> Subject: [NeXus-committee] Concordance paper, ECM
>>>
>>> Hi everyone,
>>>
>>> There are two things which we need to be decide upon rather quickly
>>> in order to allow planning to get done.
>>>
>>> The first is that it occurred to me that merging NeXus and CIF might
>>> mean that we need to understand their dictionary technology. So may be
>>> a few of us should ask to join the CIF workshop, August 23-24. I myself
>>> would be interested. Anyone else?
>>>
>>> The second is that Herberts concordance paper raised a number of issues.
>>> Thus I wonder if we should not have a 1 day code camp/meeting before
>>> meeting with the CIF people to sort this out. Anyone opinions on this?
>>>
>>> I also took a little time to look through the concordance paper. Find my
>>> comments attached below.
>>>
>>>
>>> Best Regards,
>>>
>>> Mark Koennecke
>>>
>>> Notes on Herberts Concordance Summary: 4/2013
>>> ================================================
>>>
>>> Herberts paper raises a few issues and asks for some new features in
>>> NeXus.
>>> This is my humble attempt to summarize and comment the issues.
>>>
>>> Section 2, Page 2
>>> --------------------
>>>
>>> I was a little confused: As I see it this comes down to:
>>> CBF uses tables, NeXus hierarchical groups. The basic mapping is
>>> a CBF table to a NeXus group. CBF table columns map to NeXus fields.
>>> In many cases a CBF table has only one row. If there are multiple rows
>>> in CBF for example, NR,then in NeXus the fields will be
>>> arrays of length NR.
>>>
>>> Page 3: During the NeXus Telco this was resolved: different scans map to
>>> different NXentries/ files.
>>>
>>>
>>> Section 3.2, P 10, Admin Feature Request
>>> ------------------------------------------
>>>
>>> This is a feature request: especially in PX there is a deeper level
>>> hierarchy of experiments: there is the study, there are multiple
>>> crystals
>>> (of the same protein) and there are multiple scans on those crystals.
>>> In order
>>> not to loose this hierarchy it is needed to include this administrative
>>> information in each file. Herbert suggest 3 additional fields or
>>> attributes:
>>> scan_id, diffrn_id, entry_id at NXentry level. I personally prefer
>>> fields.
>>>
>>>
>>> Section 3.3, P10, Multiple detectors
>>> -----------------------------------------
>>>
>>> NeXus deals with multiple detectors or ROI's by storing data in
>>> separate NXdetector groups. Which makes a lot of sense. Q: Why does this
>>> fall over?
>>>
>>> Section 3.6, P12-13, NXdetector fields
>>> -------------------------------------
>>>
>>> This asks for some new fields in NXdetector.
>>>
>>>
>>> Section 3.7.1, P14, Array_Structure_List
>>> ------------------------------------------
>>>
>>> I compared this with the CBF description. This is about array order
>>> and dimensions etc.
>>> IMHO, NeXus covers this by the HDF-5 dimensions and the offset and
>>> stride attributes.
>>>
>>>
>>> Section 3.7.2, P15, array_structure_list_section, ROI feature missing
>>> -------------------------------------------------------------------
>>>
>>> Again looking at the CBF documentation this seems to declare a region
>>> of interest into
>>> a larger array. NeXus does not have this per se. Currently I fail to
>>> understand how this
>>> is used: will there be another data array? Or does this mean: I have
>>> this big fat array,
>>> but just use a smaller part of it?
>>>
>>>
>>> Section 3.7.3, P 15-16, array_structure_list_axis, CBF-NeXus difference
>>> -------------------------------------------------------------------------
>>>
>>>
>>> This is about associating scan axis with the data array. CBF does
>>> this by stating
>>> a lot of parameters: starts, increments etc. for each axis. Or
>>> appears to be doing
>>> this. I might have misread the CBF docs. In NeXus this is solved by
>>> storing actual
>>> positions along the scan in arrays and the association is done by the
>>> scan rules,
>>> most notably the population of NXdata in the scan case. This actually
>>> may require some
>>> discussion: IMHO, storing arrays is better as it allows for the
>>> possibility of
>>> mispositionings.
>>>
>>>
>>> Section 3.8, Axis, Coordinate System Features
>>> --------------------------------------------------
>>>
>>> Here is a major difference: CBF supports multiple coordinate systems.
>>> Correspondingly CBF has a way to document the coordinate system used
>>> by defining the
>>> new coordinate system in terms of the standard laboratory system and
>>> by defining the
>>> directions of gravity and the beam.
>>>
>>> NeXus does not do this yet. The question is if we want this? How
>>> frequently is this
>>> used? This would require a means to describe the coordinate system
>>> and another attribute
>>> on each axis which denotes in which coordinate system it acts.
>>>
>>> Otherwise the main difference is that NeXus does not collect all the
>>> axes in one place but
>>> keeps them at their place in the hierarchy. The attributes we added
>>> recently allow for a
>>> full mapping. Especially, NeXus, does not single out a goniometer but
>>> stores the relevant
>>> fields in the NXsample group. The argument being: this is stuff which
>>> positions the sample
>>> and thus belongs into NXsample. The question here is: does NeXus need
>>> to change and
>>> devise a NXgoniometer group?
>>>
>>>
>>> Section 3.9, Diffrn_data_frame, Clarification
>>> -------------------------------------------------
>>>
>>> I fail to graps the concept even from the CBF documentation. The
>>> appears as if this
>>> allows to group data from various detectors together. Is this right?
>>> What is this for?
>>>
>>>
>>> Section 3.10, Some more NXdetetcor fields
>>> --------------------------------------------
>>>
>>> Some more fields for NXdetector in here.
>>>
>>>
>>> Section 3.11, Diffrn_measurement
>>> ------------------------------------
>>>
>>> See 3.8, coordinate systems and comments on NXsample and goniometers.
>>>
>>>
>>> Section 3.12, P29, Diffrn_radiation, NXpolariser
>>> -----------------------------------------------------
>>>
>>> Some of the fields here do not belong into NXmonochromator but into
>>> NXpolariser. Which needs some fields added.
>>>
>>> What is SIEGBAHNTYPE, IUPACXRAYSMB?
>>>
>>>
>>> Section 3.14.1, 3.14.2, P32-33, scan axis mapping,
>>> ------------------------------------------------
>>>
>>> This is again about associating scan axis and data. See comment on
>>> 3.7.3
>>>
>>>
>>> Section 3.16, P36, Variant feature
>>> ------------------------------------
>>>
>>> This is actually a feature missing in NeXus: variants. This means
>>> that there are
>>> versions of a parameter: for example you have a theoretical
>>> beam_center you use in the
>>> first tages, then you refine it and have another beam center and so
>>> on. There are as
>>> of now 4 suggestions how to add this feature to NeXus:
>>> - Use NXsubentry. Has the disadvantage to reproduce the whole tree.
>>> Also not the purpose
>>> of NXsubentry
>>> - A sepcial NXvariant group. Can occur in any group and contains
>>> different values for fields
>>> of the parent group
>>> - Simply add _variant to the normal NeXus name. If there multiple
>>> variants add: _variant_role
>>> where role becomes an indcator for the type of the variant: may
>>> be refined, from XXXplot, etc
>>> - Ask HDFgroup if there is a HDF-5 mechanism to implement versions.
>>> Or planned.
>>>
>>> This needs discussion.
>>>
>>>
>>> Section 4, P38, Eiger
>>> ---------------------
>>>
>>> Probably a typo: angular_calibration_applied is a boolena, not an
>>> array. The matching
>>> array would be angular_calibration.
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> NeXus-committee mailing list
>>> NeXus-committee at nexusformat.org
>>> http://lists.nexusformat.org/mailman/listinfo/nexus-committee
>>>
>>>
>>
>
>
--
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom
More information about the NeXus-committee
mailing list