<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
A couple of quick thoughts:</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<ul data-editing-info="{"applyListStyleFromLevel":false,"unorderedStyleType":2}">
<li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); list-style-type: "- ";">
<div class="elementToProof">Handling hdf5 external links could presumable be done with an extra level of indirection (use @target to look at an internal dataset and have that be the actual external link)</div>
</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); list-style-type: "- ";">
<div class="elementToProof">The internal links can be extended to handle external in a straight forward way by adding a URI scheme (something like
<code>nexus_external://</code> ) to the spec with the understanding that if the scheme is missing it means THIS FILE [well, "this nexus tree" to be pedantic) as in standard for URIs. All existing files are still valid and new files grow the ability to target
external files. </div>
</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); list-style-type: "- ";">
<div class="elementToProof">I don't fully grasp why we care where the data "really" is. The important thing is that if you start from the top of (one of) the defined trees you can walk down and find what you need with the names you expect. It should not matter
to a consumer if that is implemented as a combination of soft/hard/external links or the only copy of the data or via a database that fabricates the structure on demand. </div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0); list-style-type: "- ";">
<div class="elementToProof"><br>
</div>
</div>
</li></ul>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Tom</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="color: inherit;" class="elementToProof" id="Signature">
<p style="margin: 0in; font-family: Calibri, sans-serif; font-size: 11pt;"><span style="font-family: Tahoma, sans-serif; font-size: 10pt;">Thomas A Caswell (he,him,his)<br>
Data Acquisition and Detectors Deputy Group Leader</span></p>
<p style="margin: 0in; font-family: Calibri, sans-serif; font-size: 11pt;"><span style="font-family: Tahoma, sans-serif; font-size: 10pt;">Data Science and Systems Integration Program</span></p>
<p style="margin: 0in; font-family: Calibri, sans-serif; font-size: 11pt;"><span style="font-family: Tahoma, sans-serif; font-size: 10pt;">National Synchrotron Light Source II<br>
Brookhaven National Laboratory<br>
Office: 631.344.3146<br>
tcaswell@bnl.gov</span></p>
<p style="margin: 0in; font-family: Calibri, sans-serif; font-size: 11pt;"> </p>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> NeXus-committee <nexus-committee-bounces@shadow.nd.rl.ac.uk> on behalf of Raymond Osborn via NeXus-committee <nexus-committee@shadow.nd.rl.ac.uk><br>
<b>Sent:</b> Monday, February 3, 2025 12:45<br>
<b>To:</b> Brockhauser Sandor <sandor.brockhauser@physik.hu-berlin.de><br>
<b>Cc:</b> NeXus Committee <nexus-committee@nexusformat.org>; Lukas Pielsticker <lukas.pielsticker@physik.hu-berlin.de><br>
<b>Subject:</b> Re: [NeXus-committee] Example of links</font>
<div> </div>
</div>
<div style="line-break:after-white-space">Hi Sandor,
<div>Thanks for taking the time to summarize your thinking. It is good to know that your reasoning is similar to numerous discussions we had 20 or more years ago, which was then incorporated by Mark Koennecke into the NAPI (RIP) and then by Paul Kienzle and
I into the Python API. Now that NIAC doesn’t support an API, I think it does make the documentation of links much more important than it was when we provided the APIs, because it is very easy to make mistakes when writing directly using the HDF5 libraries.</div>
<div><br>
</div>
<div>Here are a few comments on your notes.</div>
<div>
<div><br>
<blockquote type="cite">
<div>On Jan 31, 2025, at 10:58 AM, Brockhauser Sandor <sandor.brockhauser@physik.hu-berlin.de> wrote:</div>
<br class="x_Apple-interchange-newline">
<div>
<div style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
In fact, it does not know anymore, if /g3/g32 was supposed to point to /g1/g12 (e.g. nice_instrument/nice_detector) and not to /g2/g22 (e.g. bad_instrument/bad_detector), because it does not point to a path(!), but to the physical object. </div>
<div style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
<br>
</div>
<div style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
This is a big difference between hard links and soft links in hdf5! In case of a soft link, the link is actually a path and it is resolved in runtime. Just like linux symbolic links, these can be broken and can point to different things if the targeted object
is changed or replaced. </div>
</div>
</blockquote>
<div><br>
</div>
<div>I agree that the HDF5 object pointed to by a soft link could in principle be replaced by </div>
<div><br>
</div>
<br>
<blockquote type="cite">
<div style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
Additionally, the so called external links can even point you to a path in a different file. Obviously, if you change the content of this file, such links can easily point to a different physical object.</div>
</blockquote>
<div><br>
</div>
<div>We have to have a completely separate discussion about external links. They are not the same as internal links. For example, you cannot add a ’target’ attribute to an external link because the physical object is in a different file with a completely different
tree. If it was added to the external file object, then the path would apply to the external file, not the local file, and would, indeed, be meaningless in the local file, which has no way of knowing the tree structure of the external file. </div>
<br>
<blockquote type="cite">
<div style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
The reason why we need a concept of a "target" attribute, so we can register for any group or dataset this attribute is attached to that this object was actually <b>derived from</b> here and there. Please note the difference, that we do not assume that the
data object here would be the same as the referenced one (e.g. the one here may contain only the relevant section what a monitor was measuring during the experiment, or the one here is converted to a different uint compared to the referenced one). This is
a big difference compared to a simple hdf5 link (or even a soft link). We argue, that in some cases the community using NeXus would like to know where the data was originated from.</div>
<div style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
Hence, additionally to the data (which is either a new dataset, a hard/soft/external link, or even a virtual dataset which one it is just an hdf5 implementation details when NeXus is used on top of hdf5) we would like to allow attaching an attribute telling
where it is coming from.</div>
</blockquote>
<div><br>
</div>
<div>Although there are reasons to criticize the documentation, this is precisely what is described in <a href="https://urldefense.com/v3/__https://manual.nexusformat.org/design.html*links__;Iw!!P4SdNyxKAPE!BAHVOCezB8o6HzbWvmbr8u55pCfK_fXv1JlcV24bmMQ7xSuldijdXEoIHKS-uyo3qZB83wCC1glKthByH8i-3ncypIVdUIHKzxEH$">https://manual.nexusformat.org/design.html#links</a>.
The diagram explicitly shows a link from the two-theta axis in the NXdata group to the same array in the detector group. However, I think your description is clearer, because unfortunately, the text above talks about avoiding replication of the data between
the two groups, which may have made people think the link was to save space. That has never been the main reason for needing links.</div>
<br>
<blockquote type="cite">
<div style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
- @napimount: doc says that it is a group attribute, but is not it a linkType attribute? Note that the provided link for further explanation (<a class="x_reference x_external" href="https://urldefense.com/v3/__http://manual.nexusformat.org/_static/NeXusIntern.pdf__;!!P4SdNyxKAPE!BAHVOCezB8o6HzbWvmbr8u55pCfK_fXv1JlcV24bmMQ7xSuldijdXEoIHKS-uyo3qZB83wCC1glKthByH8i-3ncypIVdUMFYOL8o$">http://manual.nexusformat.org/_static/NeXusIntern.pdf</a>)
is not valid.</div>
</blockquote>
<div><br>
</div>
<div>The ’napimount’ attribute was a programmatic mechanism for the NAPI to handle external links. It is not part of the standard, and is not used by the Python API at all. You cannot explicitly add a napimount attribute within the local file for the same reason
you can’t add a target attribute.</div>
<br>
<blockquote type="cite">
<div style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
- @target: doc says that it is added only because of hdf5, but we believe that its usefulness is independent of the backend if it is hdf5 or something else.</div>
</blockquote>
<div><br>
</div>
<div>The concept of links using the target attribute was introduced when the NeXus supported HDF4 and XML files. In fact, if we had only supported HDF5, we could have used soft links instead. So it was limitations particularly in XML that made the target attribute
necessary. The documentation must have been written after dropping support for XML and was probably explaining why the attribute was necessary when using hard links.</div>
<br>
<blockquote type="cite">
<div style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
- in the example @target is added to /entry/data/polar_angle which corresponds to the explanation, but it is also added to /entry/instrument/detector/polar_angle. It is not explained why is it needed there. It is because this is not derived from anywhere else?
Why is not it then simply a "." which convention is used throughout NeXus? </div>
</blockquote>
<div><br>
</div>
<div>They are the same object, so of course the target attribute appears in both groups. </div>
<div><br>
</div>
<div>In the Python API, when an object has a target attribute added, it checks if the target is the same as the object path. If it’s the same, the dataset is read in as a NXfield object. If it’s different, the dataset is read in as a NXlink object. The NXlink
object can still be used as if it were a field (e.g., you can check its dtype or shape), but it is in fact a sub-class of both NXlink and NXfield. Structurally, this works well. The use is alerted to the fact that it is a link, and can recover the link target
using the ’nxlink’ attribute, but it can be used in most contexts that a NXfield is used. However, changes directly to the NXlink are forbidden. If the user wants to change a link value, they need to explicitly change the parent’s value (even though they are
the same object). Of course, this is the same behavior as soft links. </div>
<div><br>
</div>
<blockquote type="cite">
<div style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
- If these two datasets would actually be the same physical objects (e.g. both<span class="x_Apple-converted-space"> </span>occurrences<span class="x_Apple-converted-space"> </span>would be hdf5 hard links to the same object), this would explain this example,
but as pointed out above, we foresee other usecases, too.</div>
</blockquote>
<div><br>
</div>
<div>I think this could be dangerous. If you make the NXlink object a physically different object to its target, then any assumption that the user might make about their equivalence could be invalid. Whether we use hard or soft links, I believe that the two
objects have to be identical. </div>
<br>
<blockquote type="cite">
<div style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
- according to the documentation @target must be an absolute path (although validTargetName suggests its future extension to relative paths, even including 'parent' relationship although we have seen that in case of hdf5 hard links in the tree, the interpretation
of 'parent' can become tricky)</div>
</blockquote>
<div><br>
</div>
<div>The interpretation of the parent may be tricky in HDF5, but it’s not in the NeXus standard, where it is explicitly defined by the target attribute.</div>
<div><br>
</div>
<blockquote type="cite">
<div style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
- note that the example at validTargetName is not at all an absolute path what is explained at linkType. Instead of an absolute (hdf5) path, here a class_path is used, like "NXentry/NXinstrument/analyser:NXcrystal/ef". This is not at all pointing to a unique
location in a NeXus file (e.g. if we have 2 entries with their respective instruments and analyser-s defined) resulting in ambiguity when link targets are tried to be resolved.</div>
</blockquote>
<div><br>
</div>
<div>As you know, the use of classes in validTargetName rather than names is because NeXus allows different names, so a validator would only be able to check that the actual target, which has to use names, contains the right chain of classes. </div>
<br>
<blockquote type="cite">
<div style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
- <a href="https://urldefense.com/v3/__https://manual.nexusformat.org/datarules.html*index-3__;Iw!!P4SdNyxKAPE!BAHVOCezB8o6HzbWvmbr8u55pCfK_fXv1JlcV24bmMQ7xSuldijdXEoIHKS-uyo3qZB83wCC1glKthByH8i-3ncypIVdUERzC4du$">https://manual.nexusformat.org/datarules.html#index-3</a> explains
the use of NXdata. In explaining signal, it says that it shall point to a Field (field or link) with such name. This either suggests </div>
<div style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
(1) that an NXdata group shall have the referenced dataset child inside either as a fieldType or as a linkType implemented. Note that ; or</div>
<div style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
(2) NXdata needs a dataset inside as a child which is either a field (aka hdf5 dataset) or a link (aka hdf5 link), but both are actually fieldType from NeXus point of view. </div>
</blockquote>
<div><br>
</div>
<div>
<div>Personally, I think it’s safer for the NXDL files not to specify whether a NXfield or NXgroup should be a link or not, i.e., NXDL should only refer to fields and groups, with the understanding that a user could choose to make some of them links at runtime.
Those links would have to conform to what is described in the NeXus Design web page. Is this equivalent to your option 2?</div>
<div><br>
</div>
</div>
<blockquote type="cite">
<div style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; text-decoration:none">
- how to implement and use a linkType object in hdf5 nexus file? It is actually stated (<a href="https://urldefense.com/v3/__https://manual.nexusformat.org/design.html*index-17__;Iw!!P4SdNyxKAPE!BAHVOCezB8o6HzbWvmbr8u55pCfK_fXv1JlcV24bmMQ7xSuldijdXEoIHKS-uyo3qZB83wCC1glKthByH8i-3ncypIVdUO9bm7hJ$">https://manual.nexusformat.org/design.html#index-17</a>)
that NeXus links are hdf5 hard links to objects having a @target attribute inside. This statement alone makes linkType unusable in practice since data collected in many facilities (e.g. EuXFEL) are in multiple (huge) hdf5 files, so one cannot just create hard
links between them.</div>
</blockquote>
<div><br>
</div>
<div>As I wrote above, external links cannot be handled the same way as internal links. Even if we think it’s a limitation, there is simply no practical way of treating them the same. I tried within the Python API and I couldn’t make it work. </div>
<div><br>
</div>
<div>I don’t actually think this is a problem because I am fairly sure that external links do not serve the same structural purpose as internal links in the NeXus standard. We too make extensive use of external links to point to raw data, which is really big
(at least to me - 36GB) but which we never want to touch. But it’s only the data - never the axes - and it’s the axes that contains metadata that we might want to associate with other metadata, e.g., associating 'time-of-flight' with ‘distance’. These are
small arrays so there is no reason not to keep them in the local file. If you are wanting to store all the metadata in an external file, then I think you will find that there is no way to make the ’target’ attribute work. Please let me know if I’m wrong.</div>
<div><br>
</div>
<div>With best regards,</div>
<div>Ray</div>
<div>
<div style="font-size:13px">-- </div>
<div style="font-size:13px">Ray Osborn, Senior Scientist</div>
<div style="font-size:13px">Materials Science Division</div>
<div style="font-size:13px">Argonne National Laboratory</div>
<div style="font-size:13px">Lemont, IL 60439, USA</div>
<div style="font-size:13px">Phone: +1 (630) 252-9011</div>
<div style="font-size:13px">Email: <a href="mailto:ROsborn@anl.gov">ROsborn@anl.gov</a></div>
<br class="x_Apple-interchange-newline">
</div>
<blockquote type="cite">
<blockquote type="cite" style="margin:0px 0px 0px 0.8ex; font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:auto; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:auto; word-spacing:0px; text-decoration:none; border-left-width:2px; border-left-style:solid; border-left-color:rgb(114,159,207); padding-left:1ex">
<pre style="margin:0px"></pre>
</blockquote>
</blockquote>
</div>
<br>
</div>
</div>
</body>
</html>