[NeXus-committee] NXroot is required - or should be made so

[Benjamin Watts]

Dear Herbert,
    OK, I understand that it is possible to have "turtles all the way 
down" with HDF5. But NeXus doesn't have to codify everything that is 
possible with HDF5 - NeXus defines a set of standardised structures. And 
like Mark says, we never accepted nested HDF5 files as valid NeXus 
structures. However, this discussion reminds me that the converse is 
allowed: references to external HDF5 files is allowed within NeXus 
(which Dectris are making use of with the Eiger). Therefore, there can 
be a master HDF5 file, containing an NXroot, and a set of secondary HDF5 
files without an NXroot and are linked to by the master file. So NXroot 
also serves to differentiate between master and secondary HDF5 files.

Cheers,
Ben


On 29/08/14 14:22, Herbert J. Bernstein wrote:
> Dear Mark,
>
>    The reality is that an "HDF5 file" is not simply a file.  Using,
> say, fuse, you can easily write what you think is a separate HDF5
> file, but which is actually just a deeper node in a bigger HDF5 file,
> which is, itself a deeper
> node in a still larger HDF5 file, etc., so those "file attributes" we
> need with NXroots are, in the context of
> the overall HDF5 file, just attributes of a group somewhere deep down.
>
>    I know most of us will work with an HDF5 file containing a single
> NeXus file and the root of the HDF5 file
> will be the root of the NeXus tree, but the facility that is
> supporting that HDF5 file could be doing so with
> a larger HDF5 file as the "real" container.
>
>    Regards,
>      Herbert
>
> On Fri, Aug 29, 2014 at 8:08 AM, Mark Könnecke <mark.koennecke at psi.ch> wrote:
>> Colleagues,
>>
>>
>> Am 29.08.2014 um 13:38 schrieb Herbert J. Bernstein <yayahjb at gmail.com>:
>>
>>> Dear Colleagues,
>>>
>>>   This is actually a trickier and more subtle issue than it might at
>>> first appear. And HDF5 "file" is actually more like a file system in
>>> which it is perfectly legitimate to nest multiple HDF5 trees into a
>>> larger one.    More than one completely independent NeXus file could
>>> easily exist within the same overall HDF5 file serving as the file
>>> system.  What I would suggest is:
>>>
>>> "The root of a NeXus file has the attribute NX_class="NXroot" as a
>>> node within an HDF5 file.  Most commonly, but not necessariiy, that
>>> node will also be the root of the containing HDF5 file.  If a NeXus
>>> file is started with an NXroot further down in the containing HDF5
>>> file, the nodes of the NeXus file will not refer to nodes outside
>>> NeXus file, except through the HDF5 external link mechanism, just as
>>> if they were is different files.“
>>>
>>
>> hold on, hold  on. As far as I remember there  are two reasons why NXroot actually exists:
>>
>> * When we coded NeXus using XML as a file format, we actually physically needed it.
>> * Now, at the root level of a NeXus file we like to have some file attributes. In order to  make that available for validation somehow, NXroot
>>    was defined.
>>
>> The use case of nesting multiple NeXus files in one hdf5 file was never agreed upon. This would require a strong argument why
>> we need this and a vote from the NIAC. And we already have NXentry for storing multiple data sets.
>>
>> I thus suggest to change the documentation on NXroot that it exists for purely technical reasons, in order to validate
>> file attributes.
>>
>> Regards,
>>
>>      Mark
>>
>>
>>
>>>   Regards,
>>>     Herbert
>>>
>>> On Thu, Aug 28, 2014 at 11:57 PM, Osborn, Raymond <rosborn at anl.gov> wrote:
>>>> Is this a ratified decision? I’m still not sure why this is necessary.
>>>>
>>>> Ray
>>>>
>>>> On Aug 28, 2014, at 11:23 AM, Pete Jemian <prjemian at gmail.com> wrote:
>>>>
>>>>
>>>> require that the attribute NX_class="NXroot" is assigned to the root of the
>>>> HDF5 file
>>>>
>>>> On 8/28/2014 11:20 AM, Osborn, Raymond wrote:
>>>>
>>>> I’m not sure I’m understanding this correspondence. NXroot is the root level
>>>> of the HDF5 file. When you read in the file, it might have some meaning as
>>>> the root of the file tree, but within the HDF5 file, it doesn’t exist as a
>>>> separate entity, so what does it mean to ‘require’ it?
>>>>
>>>> Ray
>>>>
>>>>
>>>> On Aug 28, 2014, at 10:51 AM, Pete Jemian <prjemian at gmail.com> wrote:
>>>>
>>>> yes
>>>>
>>>> On 8/28/2014 10:50 AM, Joachim Wuttke wrote:
>>>>
>>>> Shouldn't the manuscript describe best practice as of now,
>>>> regardless of compatibility issues?
>>>>
>>>> _______________________________________________
>>>> NeXus-committee mailing list
>>>> NeXus-committee at nexusformat.org
>>>> http://lists.nexusformat.org/mailman/listinfo/nexus-committee
>>>>
>>>>
>>>> _______________________________________________
>>>> NeXus-committee mailing list
>>>> NeXus-committee at nexusformat.org
>>>> http://lists.nexusformat.org/mailman/listinfo/nexus-committee
>>>>
>>>>
>>>> --
>>>> Ray Osborn, Senior Scientist
>>>> Materials Science Division
>>>> Argonne National Laboratory
>>>> Argonne, IL 60439, USA
>>>> Phone: +1 (630) 252-9011
>>>> Email: ROsborn at anl.gov
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> NeXus-committee mailing list
>>>> NeXus-committee at nexusformat.org
>>>> http://lists.nexusformat.org/mailman/listinfo/nexus-committee
>>>>
>>> _______________________________________________
>>> NeXus-committee mailing list
>>> NeXus-committee at nexusformat.org
>>> http://lists.nexusformat.org/mailman/listinfo/nexus-committee
>>
>> _______________________________________________
>> NeXus-committee mailing list
>> NeXus-committee at nexusformat.org
>> http://lists.nexusformat.org/mailman/listinfo/nexus-committee
>>
> _______________________________________________
> NeXus-committee mailing list
> NeXus-committee at nexusformat.org
> http://lists.nexusformat.org/mailman/listinfo/nexus-committee