<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif; ">
<div>Agreed. Sounds like a good solution.</div>
<div><br>
</div>
<div>Re: netCDF opening vanilla HDF5 files…I wasn't aware that works but cool! Agreed you wouldn't want to loose that capability either.</div>
<div><br>
</div>
<div>Mark</div>
<div><br>
</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>"<a href="mailto:dmh@ucar.edu">dmh@ucar.edu</a>" <<a href="mailto:dmh@ucar.edu">dmh@ucar.edu</a>><br>
<span style="font-weight:bold">Date: </span>Thursday, April 21, 2016 3:04 PM<br>
<span style="font-weight:bold">To: </span>Ed Hartnett <<a href="mailto:edwardjameshartnett@gmail.com">edwardjameshartnett@gmail.com</a>><br>
<span style="font-weight:bold">Cc: </span>Pedro Vicente <<a href="mailto:pedro.vicente@space-research.org">pedro.vicente@space-research.org</a>>, "<a href="mailto:cf-metadata@cgd.ucar.edu">cf-metadata@cgd.ucar.edu</a>" <<a href="mailto:cf-metadata@cgd.ucar.edu">cf-metadata@cgd.ucar.edu</a>>,
Discussion forum for the NeXus data format <<a href="mailto:nexus@nexusformat.org">nexus@nexusformat.org</a>>, "<a href="mailto:netcdfgroup@unidata.ucar.edu">netcdfgroup@unidata.ucar.edu</a>" <<a href="mailto:netcdfgroup@unidata.ucar.edu">netcdfgroup@unidata.ucar.edu</a>>,
John Shalf <<a href="mailto:jshalf@lbl.gov">jshalf@lbl.gov</a>>, "<a href="mailto:Richard.E.Ullman@nasa.gov">Richard.E.Ullman@nasa.gov</a>" <<a href="mailto:Richard.E.Ullman@nasa.gov">Richard.E.Ullman@nasa.gov</a>>, "Marinelli, Daniel J. (GSFC-5810)" <<a href="mailto:daniel.j.marinelli@nasa.gov">daniel.j.marinelli@nasa.gov</a>>,
"Miller, Mark C." <<a href="mailto:miller86@llnl.gov">miller86@llnl.gov</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [netcdfgroup] [Hdf-forum] Detecting netCDF versus HDF5 -- PROPOSED SOLUTIONS --REQUEST FOR COMMENTS<br>
</div>
<div><br>
</div>
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div>
<div>
<div>> But netCDF will still open files without this attribute, right?</div>
<div>yes.</div>
<div><br>
</div>
<div>On 4/21/2016 1:56 PM, Ed Hartnett wrote:</div>
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;">
<div>Howdy Dennis!</div>
<div><br>
</div>
<div>It sounds like a good solution.</div>
<div><br>
</div>
<div>But netCDF will still open files without this attribute, right?</div>
<div><br>
</div>
<div>The ability of netCDF-4 to open HDF5 files which were written by HDF5 </div>
<div>is an important feature. It means that users can use tools that were </div>
<div>written for netCDF on their existing HDF5 files.</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Ed</div>
<div><br>
</div>
<div><br>
</div>
<div>On Thu, Apr 21, 2016 at 3:30 PM, <a href="mailto:dmh@ucar.edu">dmh@ucar.edu</a> <<a href="mailto:dmh@ucar.edu">mailto:dmh@ucar.edu</a>>
</div>
<div><<a href="mailto:dmh@ucar.edu">dmh@ucar.edu</a> <<a href="mailto:dmh@ucar.edu>">mailto:dmh@ucar.edu></a>> wrote:</div>
<div><br>
</div>
<div> I am in the process of adding a global attribute in the root group</div>
<div> that captures both the netcdf library version and the hdf5 library</div>
<div> version</div>
<div> whenever a netcdf file is created. The current form is</div>
<div> _NCProperties="version=...|netcdflibversion=...|hdflibversion=..."</div>
<div> Where version is the version of the _NCProperties attribute and</div>
<div> the others</div>
<div> are e.g. 1.8.18 or 4.4.1-rc1.</div>
<div> Issues:</div>
<div> 1. I am open to suggestions about changing the format or adding</div>
<div> info to it.</div>
<div> 2. Of course this attribute will not exist in files written using</div>
<div> older versions</div>
<div> of the netcdf library, but at least the process will have begun.</div>
<div> 3. This technically does not address the original issue because</div>
<div> there exist</div>
<div> hdf5 files not written by netcdf that are still compatible</div>
<div> with and can be</div>
<div> read by netcdf. Not sure this case is important or not.</div>
<div> =Dennis Heimbigner</div>
<div> Unidata</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div> On 4/21/2016 9:33 AM, Pedro Vicente wrote:</div>
<div><br>
</div>
<div> DETECTING HDF5 VERSUS NETCDF GENERATED FILES</div>
<div> REQUEST FOR COMMENTS</div>
<div> AUTHOR: Pedro Vicente</div>
<div><br>
</div>
<div> AUDIENCE:</div>
<div> 1) HDF, netcdf developers,</div>
<div> Ed Hartnett</div>
<div> Kent Yang</div>
<div> 2) HDF, netcdf users, that replied to this thread</div>
<div> Miller, Mark C.</div>
<div> John Shalf</div>
<div> 3 ) netcdf tools developers</div>
<div> Mary Haley , NCL</div>
<div> 4) HDF, netcdf managers and sponsors</div>
<div> David Pearah , CEO HDF Group</div>
<div> Ward Fisher, UCAR</div>
<div> Marinelli, Daniel J. , Richard Ullmman, Christopher Lynnes, NASA</div>
<div> 5)</div>
<div> [CF-metadata] list</div>
<div> After this thread started 2 months ago, there was an</div>
<div> annoucement on the [CF-metadata] mail list</div>
<div> about</div>
<div> "a meeting to discuss current and future netCDF-CF efforts and</div>
<div> directions.</div>
<div> The meeting will be held on 24-26 May 2016 in Boulder, CO, USA</div>
<div> at the UCAR Center Green facility."</div>
<div> This would be a good topic to put on the agenda, maybe?</div>
<div> THE PROBLEM:</div>
<div> Currently it is impossible to detect if an HDF5 file was</div>
<div> generated by the HDF5 API or by the netCDF API.</div>
<div> See previous email about the reasons why.</div>
<div> WHY THIS MATTERS:</div>
<div> Software applications that need to handle both netCDF and HDF5</div>
<div> files cannot decide which API to use.</div>
<div> This includes popular visualization tools like IDL, Matlab,</div>
<div> NCL, HDF Explorer.</div>
<div> SOLUTIONS PROPOSED: 2</div>
<div> SOLUTION 1: Add a flag to HDF5 source</div>
<div> The hdf5 format specification, listed here</div>
<div> <a href="https://secure-web.cisco.com/1wKy-j9FSupG_v_6CUbHG6_yTgp268G1MSKWYZTaPtp7Jwdrjq5PdvPDfpgcPG5Am4TtnS27SHMG8LPa8IRjQzVL2YV703Vx8s91PsfxJsaPxgnr5W8XMlIgK1_DIvUwoA4n3aRWIWfXNlmM_k52uvVgHmLYb_H3_qHO9m_GrD5yCwc3PtSwaRqJKKPYvClxh3Ec5kZUNduDeHdv134RN1wpd-ifxDTqypTB0UVRt5PPoEFYFMeICDULUtTHvZWC6OgFpOPjimJZpxHqYhcokitB--yd0RJRCKZpTml6XbpSjBF8952hFQyBxPDfSjHzCa9mO053ACOKH-CjNrGrWa2u4yDB6jQvU51mJbDHwm-Y/https%3A%2F%2Fwww.hdfgroup.org%2FHDF5%2Fdoc%2FH5.format.html">
https://secure-web.cisco.com/1wKy-j9FSupG_v_6CUbHG6_yTgp268G1MSKWYZTaPtp7Jwdrjq5PdvPDfpgcPG5Am4TtnS27SHMG8LPa8IRjQzVL2YV703Vx8s91PsfxJsaPxgnr5W8XMlIgK1_DIvUwoA4n3aRWIWfXNlmM_k52uvVgHmLYb_H3_qHO9m_GrD5yCwc3PtSwaRqJKKPYvClxh3Ec5kZUNduDeHdv134RN1wpd-ifxDTqypTB0UVRt5PPoEFYFMeICDULUtTHvZWC6OgFpOPjimJZpxHqYhcokitB--yd0RJRCKZpTml6XbpSjBF8952hFQyBxPDfSjHzCa9mO053ACOKH-CjNrGrWa2u4yDB6jQvU51mJbDHwm-Y/https%3A%2F%2Fwww.hdfgroup.org%2FHDF5%2Fdoc%2FH5.format.html</a></div>
<div> describes a sequence of bytes in the file layout that have</div>
<div> special meaning for the HDF5 API. It is common practice, when</div>
<div> designing a data format,</div>
<div> so leave some fields "reserved for future use".</div>
<div> This solution makes use of one of these empty "reserved for</div>
<div> future use" spaces to save a byte (for example) that describes</div>
<div> an enumerator</div>
<div> of "HDF5 compatible formats".</div>
<div> An "HDF5 compatible format" is a data format that uses the</div>
<div> HDF5 API at a lower level (usually hidden from the user of the</div>
<div> upper API),</div>
<div> and providing its own API.</div>
<div> This category can still be divide in 2 formats:</div>
<div> 1) A "pure HDF5 compatible format". Example, NeXus</div>
<div> <a href="http://www.nexusformat.org/">http://www.nexusformat.org/</a></div>
<div> NeXus just writes some metadata (attributes) on top of the</div>
<div> HDF5 API, that has some special meaning for the NeXus community</div>
<div> 2) A "non pure HDF5 compatible format". Example, netCDF</div>
<div> Here, the format adds some extra feature besides HDF5. In the</div>
<div> case of netCDF, these are shared dimensions between variables.</div>
<div> This sub-division between 1) and 2) is irrelevant for the</div>
<div> problem and solution in question</div>
<div> The solution consists of writing a different enumerator value</div>
<div> on the "reserved for future use" space. For example</div>
<div> Value decimal 0 (current value): This file was generated by</div>
<div> the HDF5 API (meaning the HDF5 only API)</div>
<div> Value decimal 1: This file was generated by the netCDF API</div>
<div> (using HDF5)</div>
<div> Value decimal 2: This file was generated by <put here another</div>
<div> HDF5 based format></div>
<div> and so on</div>
<div> The advantage of this solution is that this process involves 2</div>
<div> parties: the HDF Group and the other format's organization.</div>
<div> This allows the HDF Group to "keep track" of new HDF5 based</div>
<div> formats. It allows to make the other format "HDF5 certified" .</div>
<div> SOLUTION 2: Add some metadata to the other API on top of HDF5</div>
<div> This is what Nexus uses.</div>
<div> A Nexus file on creation writes several attributes on the root</div>
<div> group, like "NeXus_version" and other numeric data.</div>
<div> This is done using the public HDF5 API calls.</div>
<div> The solution for netCDF consists of the same approach, just</div>
<div> write some specific attributes, and a special netCDF API to</div>
<div> write/read them.</div>
<div> This solutions just requires the work of one party (the netCDF</div>
<div> group)</div>
<div> END OF RFC</div>
<div> In reply to people that commented in the thread</div>
<div> @John Shalf</div>
<div> >>Perhaps NetCDF (and other higher-level APIs that are built</div>
<div> on top of HDF5) should include an attribute attached</div>
<div> >>to the root group that identifies the name and version of</div>
<div> the API that created the file? (adopt this as a convention)</div>
<div> yes, that's one way to do it, Solution 2 above</div>
<div> @Mark Miller</div>
<div> >>>Hmmm. Is there any big reason NOT to try to read a netCDF</div>
<div> produced HDF5 file with the native HDF5 library if someone so</div>
<div> chooses?</div>
<div> It's possible to read a netCDF file using HDF5, yes.</div>
<div> There are 2 things that you will miss doing this:</div>
<div> 1) the ability to inquire about shared netCDF dimensions.</div>
<div> 2) the ability to read remotely with openDAP.</div>
<div> Reading with HDF5 also exposes metadata that is supposed to be</div>
<div> private to netCDF. See below</div>
<div> >>>> And, attempting to read an HDF5 file produced by Silo</div>
<div> using just the HDF5 library (e.g. w/o Silo) is a major pain.</div>
<div> This I don't understand. Why not read the Silo file with the</div>
<div> Silo API?</div>
<div> That's the all purpose of this issue, each higher level API on</div>
<div> top of HDF5 should be able to detect "itself".</div>
<div> I am not familiar with Silo, but if Silo cannot do this, then</div>
<div> you have the same design flaw that netCDF has.</div>
<div><br>
</div>
<div> >>> In a cursory look over the libsrc4 sources in netCDF</div>
<div> distro, I see a few things that might give a hint a file was</div>
<div> created with netCDF. . .</div>
<div> >>>> First, in NC_CLASSIC_MODEL, an attribute gets attached to</div>
<div> the root group named "_nc3_strict". So, the existence of an</div>
<div> attribute on the root group by that name would suggest the</div>
<div> HDF5 file was generated by netCDF.</div>
<div> I think this is done only by the "old" netCDF3 format.</div>
<div> >>>>> Also, I tested a simple case of nc_open, nc_def_dim,</div>
<div> etc. nc_close to see what it produced.</div>
<div> >>>> It appears to produce datasets for each 'dimension'</div>
<div> defined with two attributes named "CLASS" and "NAME".</div>
<div> This is because netCDF uses the HDF5 Dimension Scales API</div>
<div> internally to keep track of shared dimensions. These are</div>
<div> internal attributes</div>
<div> of Dimension Scales. This approach would not work because an</div>
<div> HDF5 only file with Dimension Scales would have the same</div>
<div> attributes.</div>
<div><br>
</div>
<div> >>>> I like John's suggestion here.</div>
<div> >>>>>But, any code you add to any applications now will work</div>
<div> *only* for files that were produced post-adoption of this</div>
<div> convention.</div>
<div> yes. there are 2 actions to take here.</div>
<div> 1) fix the issue for the future</div>
<div> 2) try to retroactively have some workaround that makes</div>
<div> possible now to differentiate a HDF5/netCDF files made before</div>
<div> the adopted convention</div>
<div> see below</div>
<div><br>
</div>
<div> >>>> In VisIt, we support >140 format readers. Over 20 of</div>
<div> those are different variants of HDF5 files (H5part, Xdmf,</div>
<div> Pixie, Silo, Samrai, netCDF, Flash, Enzo, Chombo, etc., etc.)</div>
<div> >>>>When opening a file, how does VisIt figure out which</div>
<div> plugin to use? In particular, how do we avoid one poorly</div>
<div> written reader plugin (which may be the wrong one for a given</div>
<div> file) from preventing the correct one from being found. Its</div>
<div> kinda a hard problem.</div>
<div><br>
</div>
<div> Yes, that's the problem we are trying to solve. I have to say,</div>
<div> that is quick a list of HDF5 based formats there.</div>
<div> >>>> Some of our discussion is captured here. . .</div>
<div> <a href="http://www.visitusers.org/index.php?title=Database_Format_Detection">
http://www.visitusers.org/index.php?title=Database_Format_Detection</a></div>
<div> I"ll check it out, thank you for the suggestions</div>
<div> @Ed Hartnett</div>
<div> >>>I must admit that when putting netCDF-4 together I never</div>
<div> considered that someone might want to tell the difference</div>
<div> between a "native" HDF5 file and a netCDF-4/HDF5 file.</div>
<div> >>>>>Well, you can't think of everything.</div>
<div> This is a major design flaw.</div>
<div> If you are in the business of designing data file formats, one</div>
<div> of the things you have to do is how to make it possible to</div>
<div> identify it from the other formats.</div>
<div><br>
</div>
<div> >>> I agree that it is not possible to canonically tell the</div>
<div> difference. The netCDF-4 API does use some special attributes</div>
<div> to track named dimensions,</div>
<div> >>>>and to tell whether classic mode should be enforced. But</div>
<div> it can easily produce files without any named dimensions, etc.</div>
<div> >>>So I don't think there is any easy way to tell.</div>
<div> I remember you wrote that code together with Kent Yang from</div>
<div> the HDF Group.</div>
<div> At the time I was with the HDF Group but unfortunately I did</div>
<div> follow closely what you were doing.</div>
<div> I don't remember any design document being circulated that</div>
<div> explains the internals of the "how to" make the netCDF</div>
<div> (classic) model of shared dimensions</div>
<div> use the hierarchical group model of HDF5.</div>
<div> I know this was done using the HDF5 Dimension Scales (that I</div>
<div> wrote), but is there any design document that explains it?</div>
<div> Maybe just some internal email exchange between you and Kent Yang?</div>
<div> Kent, how are you?</div>
<div> Do you remember having any design document that explains this?</div>
<div> Maybe something like a unique private attribute that is</div>
<div> written somewhere in the netCDF file?</div>
<div><br>
</div>
<div> @Mary Haley, NCL</div>
<div> NCL is a widely used tool that handles both netCDF and HDF5</div>
<div> Mary, how are you?</div>
<div> How does NCL deal with the case of reading both pure HDF5</div>
<div> files and netCDF files that use HDF5?</div>
<div> Would you be interested in joining a community based effort to</div>
<div> deal with this, in case this is an issue for you?</div>
<div><br>
</div>
<div> @David Pearah , CEO HDF Group</div>
<div> I volunteer to participate in the effort of this RFC together</div>
<div> with the HDF Group (and netCDF Group).</div>
<div> Maybe we could make a "task force" between HDF Group, netCDF</div>
<div> Group and any volunteer (such as tools developers that happen</div>
<div> to be in these mail lists)?</div>
<div> The "task force" would have 2 tasks:</div>
<div> 1) make a HDF5 based convention for the future and</div>
<div> 2) try to retroactively salvage the current design issue of netCDF</div>
<div> My phone is 217-898-9356 <tel:217-898-9356>, you are welcome</div>
<div> to call in anytime.</div>
<div> ----------------------</div>
<div> Pedro Vicente</div>
<div> <a href="mailto:pedro.vicente@space-research.org">pedro.vicente@space-research.org</a></div>
<div> <<a href="mailto:pedro.vicente@space-research.org">mailto:pedro.vicente@space-research.org</a>></div>
<div> <<a href="mailto:pedro.vicente@space-research.org">mailto:pedro.vicente@space-research.org</a></div>
<div> <<a href="mailto:pedro.vicente@space-research.org>">mailto:pedro.vicente@space-research.org></a>></div>
<div> <a href="https://twitter.com/_pedro__vicente">https://twitter.com/_pedro__vicente</a></div>
<div> <a href="http://www.space-research.org/">http://www.space-research.org/</a></div>
<div><br>
</div>
<div> ----- Original Message -----</div>
<div> *From:* Miller, Mark C. <<a href="mailto:miller86@llnl.gov">mailto:miller86@llnl.gov</a></div>
<div> <<a href="mailto:miller86@llnl.gov>">mailto:miller86@llnl.gov></a>></div>
<div> *To:* HDF Users Discussion List</div>
<div> <<a href="mailto:hdf-forum@lists.hdfgroup.org">mailto:hdf-forum@lists.hdfgroup.org</a></div>
<div> <<a href="mailto:hdf-forum@lists.hdfgroup.org>">mailto:hdf-forum@lists.hdfgroup.org></a>></div>
<div> *Cc:* <a href="mailto:netcdfgroup@unidata.ucar.edu">netcdfgroup@unidata.ucar.edu</a></div>
<div> <<a href="mailto:netcdfgroup@unidata.ucar.edu">mailto:netcdfgroup@unidata.ucar.edu</a>></div>
<div> <<a href="mailto:netcdfgroup@unidata.ucar.edu">mailto:netcdfgroup@unidata.ucar.edu</a></div>
<div> <<a href="mailto:netcdfgroup@unidata.ucar.edu>">mailto:netcdfgroup@unidata.ucar.edu></a>> ; Ward Fisher</div>
<div> <<a href="mailto:wfisher@ucar.edu">mailto:wfisher@ucar.edu</a> <<a href="mailto:wfisher@ucar.edu>">mailto:wfisher@ucar.edu></a>></div>
<div> *Sent:* Wednesday, March 02, 2016 7:07 PM</div>
<div> *Subject:* Re: [Hdf-forum] Detecting netCDF versus HDF5</div>
<div><br>
</div>
<div> I like John's suggestion here.</div>
<div><br>
</div>
<div> But, any code you add to any applications now will work</div>
<div> *only* for</div>
<div> files that were produced post-adoption of this convention.</div>
<div><br>
</div>
<div> There are probably a bazillion files out there at this</div>
<div> point that</div>
<div> don't follow that convention and you probably still want your</div>
<div> applications to be able to read them.</div>
<div><br>
</div>
<div> In VisIt, we support >140 format readers. Over 20 of those are</div>
<div> different variants of HDF5 files (H5part, Xdmf, Pixie, Silo,</div>
<div> Samrai, netCDF, Flash, Enzo, Chombo, etc., etc.) When</div>
<div> opening a</div>
<div> file, how does VisIt figure out which plugin to use? In</div>
<div> particular, how do we avoid one poorly written reader plugin</div>
<div> (which may be the wrong one for a given file) from</div>
<div> preventing the</div>
<div> correct one from being found. Its kinda a hard problem.</div>
<div><br>
</div>
<div> Some of our discussion is captured here. . .</div>
<div><br>
</div>
<div> <a href="http://www.visitusers.org/index.php?title=Database_Format_Detection">
http://www.visitusers.org/index.php?title=Database_Format_Detection</a></div>
<div><br>
</div>
<div> Mark</div>
<div><br>
</div>
<div><br>
</div>
<div> From: Hdf-forum <<a href="mailto:hdf-forum-bounces@lists.hdfgroup.org">hdf-forum-bounces@lists.hdfgroup.org</a></div>
<div> <<a href="mailto:hdf-forum-bounces@lists.hdfgroup.org">mailto:hdf-forum-bounces@lists.hdfgroup.org</a>></div>
<div> <<a href="mailto:hdf-forum-bounces@lists.hdfgroup.org">mailto:hdf-forum-bounces@lists.hdfgroup.org</a></div>
<div> <<a href="mailto:hdf-forum-bounces@lists.hdfgroup.org>>">mailto:hdf-forum-bounces@lists.hdfgroup.org>></a>> on behalf of John</div>
<div> Shalf <<a href="mailto:jshalf@lbl.gov">jshalf@lbl.gov</a> <<a href="mailto:jshalf@lbl.gov">mailto:jshalf@lbl.gov</a>></div>
<div> <<a href="mailto:jshalf@lbl.gov">mailto:jshalf@lbl.gov</a> <<a href="mailto:jshalf@lbl.gov>>">mailto:jshalf@lbl.gov>></a>></div>
<div> Reply-To: HDF Users Discussion List</div>
<div> <<a href="mailto:hdf-forum@lists.hdfgroup.org">hdf-forum@lists.hdfgroup.org</a></div>
<div> <<a href="mailto:hdf-forum@lists.hdfgroup.org">mailto:hdf-forum@lists.hdfgroup.org</a>></div>
<div> <<a href="mailto:hdf-forum@lists.hdfgroup.org">mailto:hdf-forum@lists.hdfgroup.org</a></div>
<div> <<a href="mailto:hdf-forum@lists.hdfgroup.org>>">mailto:hdf-forum@lists.hdfgroup.org>></a>></div>
<div> Date: Wednesday, March 2, 2016 1:02 PM</div>
<div> To: HDF Users Discussion List</div>
<div> <<a href="mailto:hdf-forum@lists.hdfgroup.org">hdf-forum@lists.hdfgroup.org</a></div>
<div> <<a href="mailto:hdf-forum@lists.hdfgroup.org">mailto:hdf-forum@lists.hdfgroup.org</a>></div>
<div> <<a href="mailto:hdf-forum@lists.hdfgroup.org">mailto:hdf-forum@lists.hdfgroup.org</a></div>
<div> <<a href="mailto:hdf-forum@lists.hdfgroup.org>>">mailto:hdf-forum@lists.hdfgroup.org>></a>></div>
<div> Cc: "<a href="mailto:netcdfgroup@unidata.ucar.edu">netcdfgroup@unidata.ucar.edu</a></div>
<div> <<a href="mailto:netcdfgroup@unidata.ucar.edu">mailto:netcdfgroup@unidata.ucar.edu</a>></div>
<div> <<a href="mailto:netcdfgroup@unidata.ucar.edu">mailto:netcdfgroup@unidata.ucar.edu</a></div>
<div> <<a href="mailto:netcdfgroup@unidata.ucar.edu>>">mailto:netcdfgroup@unidata.ucar.edu>></a>"</div>
<div> <<a href="mailto:netcdfgroup@unidata.ucar.edu">netcdfgroup@unidata.ucar.edu</a></div>
<div> <<a href="mailto:netcdfgroup@unidata.ucar.edu">mailto:netcdfgroup@unidata.ucar.edu</a>></div>
<div> <<a href="mailto:netcdfgroup@unidata.ucar.edu">mailto:netcdfgroup@unidata.ucar.edu</a></div>
<div> <<a href="mailto:netcdfgroup@unidata.ucar.edu>>">mailto:netcdfgroup@unidata.ucar.edu>></a>>, Ward Fisher</div>
<div> <<a href="mailto:wfisher@ucar.edu">wfisher@ucar.edu</a> <<a href="mailto:wfisher@ucar.edu">mailto:wfisher@ucar.edu</a>></div>
<div> <<a href="mailto:wfisher@ucar.edu">mailto:wfisher@ucar.edu</a> <<a href="mailto:wfisher@ucar.edu>>">mailto:wfisher@ucar.edu>></a>></div>
<div> Subject: Re: [Hdf-forum] Detecting netCDF versus HDF5</div>
<div><br>
</div>
<div> Perhaps NetCDF (and other higher-level APIs that are</div>
<div> built on</div>
<div> top of HDF5) should include an attribute attached to</div>
<div> the root</div>
<div> group that identifies the name and version of the API that</div>
<div> created the file? (adopt this as a convention)</div>
<div><br>
</div>
<div> -john</div>
<div><br>
</div>
<div> On Mar 2, 2016, at 12:55 PM, Pedro Vicente</div>
<div> <<a href="mailto:pedro.vicente@space-research.org">pedro.vicente@space-research.org</a></div>
<div> <<a href="mailto:pedro.vicente@space-research.org">mailto:pedro.vicente@space-research.org</a>></div>
<div> <<a href="mailto:pedro.vicente@space-research.org">mailto:pedro.vicente@space-research.org</a></div>
<div> <<a href="mailto:pedro.vicente@space-research.org>>">mailto:pedro.vicente@space-research.org>></a>> wrote:</div>
<div> Hi Ward</div>
<div> As you know, Data Explorer is going to be a general</div>
<div> purpose data reader for many formats, including</div>
<div> HDF5 and</div>
<div> netCDF.</div>
<div> Here</div>
<div> <a href="http://www.space-research.org/">http://www.space-research.org/</a></div>
<div> Regarding the handling of both HDF5 and netCDF, it</div>
<div> seems</div>
<div> there is a potential issue, which is, how to tell</div>
<div> if any</div>
<div> HDF5 file was saved by the HDF5 API or by the</div>
<div> netCDF API?</div>
<div> It seems to me that this is not possible. Is this</div>
<div> correct?</div>
<div> netCDF uses an internal function NC_check_file_type to</div>
<div> examine the first few bytes of a file, and for</div>
<div> example for</div>
<div> any HDF5 file the test is</div>
<div> /* Look at the magic number */</div>
<div> /* Ignore the first byte for HDF */</div>
<div> if(magic[1] == 'H' && magic[2] == 'D' &&</div>
<div> magic[3] == 'F') {</div>
<div> *filetype = FT_HDF;</div>
<div> *version = 5;</div>
<div> The problem is that this test works for any HDF5</div>
<div> file and</div>
<div> for any netCDF file, which makes it impossible to tell</div>
<div> which is which.</div>
<div> Which makes it impossible for any general purpose data</div>
<div> reader to decide to use the netCDF API or the HDF5</div>
<div> API.</div>
<div> I have a possible solution for this , but before</div>
<div> going any</div>
<div> further, I would just like to confirm that</div>
<div> 1) Is indeed not possible</div>
<div> 2) See if you have a solid workaround for this,</div>
<div> excluding the dumb ones, for example deciding on a</div>
<div> extension .nc or .h5, or traversing the HDF5 file</div>
<div> to see</div>
<div> if it's non netCDF conforming one. Yes, to further</div>
<div> complicate things, it is possible that the above</div>
<div> test says</div>
<div> OK for a HDF5 file, but then the read by the</div>
<div> netCDF API</div>
<div> fails because the file is a HDF5 non netCDF conformant</div>
<div> Thanks</div>
<div> ----------------------</div>
<div> Pedro Vicente</div>
<div> <a href="mailto:pedro.vicente@space-research.org">pedro.vicente@space-research.org</a></div>
<div> <<a href="mailto:pedro.vicente@space-research.org">mailto:pedro.vicente@space-research.org</a>></div>
<div> <<a href="mailto:pedro.vicente@space-research.org">mailto:pedro.vicente@space-research.org</a></div>
<div> <<a href="mailto:pedro.vicente@space-research.org>">mailto:pedro.vicente@space-research.org></a>></div>
<div> <a href="http://www.space-research.org/">http://www.space-research.org/</a></div>
<div> _______________________________________________</div>
<div> Hdf-forum is for HDF software users discussion.</div>
<div> <a href="mailto:Hdf-forum@lists.hdfgroup.org">Hdf-forum@lists.hdfgroup.org</a> <<a href="mailto:Hdf-forum@lists.hdfgroup.org">mailto:Hdf-forum@lists.hdfgroup.org</a>></div>
<div> <<a href="mailto:Hdf-forum@lists.hdfgroup.org">mailto:Hdf-forum@lists.hdfgroup.org</a></div>
<div> <<a href="mailto:Hdf-forum@lists.hdfgroup.org>">mailto:Hdf-forum@lists.hdfgroup.org></a>></div>
<div> <a href="http://secure-web.cisco.com/1r-EJFFfg6rWlpQsvXstBNTjaHQaKT_NkYRN0Jj_f-Z3EK0-hs6IbYc8XUBRyPsH3mU3CS0iiY7_qnchCA0QxNzQt270d_2HikCwpAWFmuHdacin62eaODutktDSOULIJmVbVYqFVSKWPzoX7kdP0yN9wIzSFxZfTwfhU8ebsN409xRg1PsW_8cvNiWzxDNm9wv9yBf9yK6nkEm-bOx2S0kBLbg9WfIChWzZrkpE3AHU9I-c2ZRH_IN-UF4g_g0_Dh4qE1VETs7tZTfKd1ox1MtBmeyKf7EKUCd3ezR9EbI5tK4hCU5qW4v5WWOxOrD17e8yCVmob27xz84Lr3bCK5wIQdH5VzFRTtyaAhudpt9E/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org">
http://secure-web.cisco.com/1r-EJFFfg6rWlpQsvXstBNTjaHQaKT_NkYRN0Jj_f-Z3EK0-hs6IbYc8XUBRyPsH3mU3CS0iiY7_qnchCA0QxNzQt270d_2HikCwpAWFmuHdacin62eaODutktDSOULIJmVbVYqFVSKWPzoX7kdP0yN9wIzSFxZfTwfhU8ebsN409xRg1PsW_8cvNiWzxDNm9wv9yBf9yK6nkEm-bOx2S0kBLbg9WfIChWzZrkpE3AHU9I-c2ZRH_IN-UF4g_g0_Dh4qE1VETs7tZTfKd1ox1MtBmeyKf7EKUCd3ezR9EbI5tK4hCU5qW4v5WWOxOrD17e8yCVmob27xz84Lr3bCK5wIQdH5VzFRTtyaAhudpt9E/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org</a></div>
<div> Twitter: <a href="https://twitter.com/hdf5">https://twitter.com/hdf5</a></div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div> _______________________________________________</div>
<div> Hdf-forum is for HDF software users discussion.</div>
<div> <a href="mailto:Hdf-forum@lists.hdfgroup.org">Hdf-forum@lists.hdfgroup.org</a></div>
<div> <<a href="mailto:Hdf-forum@lists.hdfgroup.org">mailto:Hdf-forum@lists.hdfgroup.org</a>></div>
<div> <<a href="mailto:Hdf-forum@lists.hdfgroup.org">mailto:Hdf-forum@lists.hdfgroup.org</a></div>
<div> <<a href="mailto:Hdf-forum@lists.hdfgroup.org>">mailto:Hdf-forum@lists.hdfgroup.org></a>></div>
<div> <a href="http://secure-web.cisco.com/1r-EJFFfg6rWlpQsvXstBNTjaHQaKT_NkYRN0Jj_f-Z3EK0-hs6IbYc8XUBRyPsH3mU3CS0iiY7_qnchCA0QxNzQt270d_2HikCwpAWFmuHdacin62eaODutktDSOULIJmVbVYqFVSKWPzoX7kdP0yN9wIzSFxZfTwfhU8ebsN409xRg1PsW_8cvNiWzxDNm9wv9yBf9yK6nkEm-bOx2S0kBLbg9WfIChWzZrkpE3AHU9I-c2ZRH_IN-UF4g_g0_Dh4qE1VETs7tZTfKd1ox1MtBmeyKf7EKUCd3ezR9EbI5tK4hCU5qW4v5WWOxOrD17e8yCVmob27xz84Lr3bCK5wIQdH5VzFRTtyaAhudpt9E/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org">
http://secure-web.cisco.com/1r-EJFFfg6rWlpQsvXstBNTjaHQaKT_NkYRN0Jj_f-Z3EK0-hs6IbYc8XUBRyPsH3mU3CS0iiY7_qnchCA0QxNzQt270d_2HikCwpAWFmuHdacin62eaODutktDSOULIJmVbVYqFVSKWPzoX7kdP0yN9wIzSFxZfTwfhU8ebsN409xRg1PsW_8cvNiWzxDNm9wv9yBf9yK6nkEm-bOx2S0kBLbg9WfIChWzZrkpE3AHU9I-c2ZRH_IN-UF4g_g0_Dh4qE1VETs7tZTfKd1ox1MtBmeyKf7EKUCd3ezR9EbI5tK4hCU5qW4v5WWOxOrD17e8yCVmob27xz84Lr3bCK5wIQdH5VzFRTtyaAhudpt9E/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org</a></div>
<div> Twitter: <a href="https://twitter.com/hdf5">https://twitter.com/hdf5</a></div>
<div><br>
</div>
<div><br>
</div>
<div> ------------------------------------------------------------------------</div>
<div> _______________________________________________</div>
<div> Hdf-forum is for HDF software users discussion.</div>
<div> <a href="mailto:Hdf-forum@lists.hdfgroup.org">Hdf-forum@lists.hdfgroup.org</a> <<a href="mailto:Hdf-forum@lists.hdfgroup.org">mailto:Hdf-forum@lists.hdfgroup.org</a>></div>
<div> <a href="http://secure-web.cisco.com/13uklvjmU6j7lQMFv4FEQ5naH9A_wipLX0OS6-gWz02tHyPOcpKoBIbeO6IAx4NhGeFC2NsOHgpNoyLOhXIOKRa7jXYD_4hmhW_A_u01OoLp8ThO66jhqsGnswO03uCRdsTNbfUu-iAMC4Y5jDBNJKbFvgmlZYu8tq6USyztV1oBng_Hp07B3j8dR2Aw9u-heuJpA2asO3Iudir9iyHQixAmWOrVxDQOy7jlWfVMOV4FtehEZJApCfa3LZ9zjcZT-cmDHvgMIntMn8kXA_ZYt3zeO-xjEeSRB_D49KJvjx77t4Z6UwPpToTw8FGAmHcUWuCGb6zHNfCBRS7Cm1b6OJdxkvP37MNfNjR6A3DVZhfo/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org">
http://secure-web.cisco.com/13uklvjmU6j7lQMFv4FEQ5naH9A_wipLX0OS6-gWz02tHyPOcpKoBIbeO6IAx4NhGeFC2NsOHgpNoyLOhXIOKRa7jXYD_4hmhW_A_u01OoLp8ThO66jhqsGnswO03uCRdsTNbfUu-iAMC4Y5jDBNJKbFvgmlZYu8tq6USyztV1oBng_Hp07B3j8dR2Aw9u-heuJpA2asO3Iudir9iyHQixAmWOrVxDQOy7jlWfVMOV4FtehEZJApCfa3LZ9zjcZT-cmDHvgMIntMn8kXA_ZYt3zeO-xjEeSRB_D49KJvjx77t4Z6UwPpToTw8FGAmHcUWuCGb6zHNfCBRS7Cm1b6OJdxkvP37MNfNjR6A3DVZhfo/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org</a></div>
<div> Twitter: <a href="https://twitter.com/hdf5">https://twitter.com/hdf5</a></div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div> _______________________________________________</div>
<div> netcdfgroup mailing list</div>
<div> <a href="mailto:netcdfgroup@unidata.ucar.edu">netcdfgroup@unidata.ucar.edu</a> <<a href="mailto:netcdfgroup@unidata.ucar.edu">mailto:netcdfgroup@unidata.ucar.edu</a>></div>
<div> For list information or to unsubscribe, visit:</div>
<div> <a href="http://www.unidata.ucar.edu/mailing_lists/">http://www.unidata.ucar.edu/mailing_lists/</a></div>
<div><br>
</div>
<div><br>
</div>
<div> _______________________________________________</div>
<div> netcdfgroup mailing list</div>
<div> <a href="mailto:netcdfgroup@unidata.ucar.edu">netcdfgroup@unidata.ucar.edu</a> <<a href="mailto:netcdfgroup@unidata.ucar.edu">mailto:netcdfgroup@unidata.ucar.edu</a>></div>
<div> For list information or to unsubscribe, visit:</div>
<div> <a href="http://www.unidata.ucar.edu/mailing_lists/">http://www.unidata.ucar.edu/mailing_lists/</a>
</div>
<div><br>
</div>
<div><br>
</div>
</blockquote>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</blockquote>
</span>
</body>
</html>