[Nexus-developers] [Nexus] XML format--A way to serialize binary?

Fri Dec 8 17:03:14 GMT 2006

Peter,

What I have may not be quite what people think of when they talk C++ 
wrapper to nexus...

Enclosed are the headers and a stupid little document I wrote about how 
the library works--don't let the date on the document scare you, it's 
still very relevant!

The short form synopsis:

Napi routines are mapped almost directly to C++ methods in a class 
called nexusapi (nexusapi.h)

An interface class called NexusBoxClass privately inherits nexusapi to 
handle all the direct file access.  NexusBoxClass then provides 
interface methods to the data (nexusbox.h)  Groups, Fields and 
Attributes in Napi are each instantiated as their own classes.  The 
NexusBoxClass acts as a container (hence "box") to hold all the Group, 
Field and Attribute classes.  The Group, Field and Attribute classes are 
arranged in a memory tree that mimics the directory structure of the 
file.  Routines exist to query paths to the data--I call them "index" in 
the header.  An index would be an ascii readable path to the data and 
serves as the "handle" to the data.

Some utility methods are thrown in for good measure.  It's possible to 
read the entire file into the memory tree, or, to save memory, large 
multidimensional data would be read in as "index only" until you 
actually need to access the data.  It's also possible to supply a 
"template file" (described better in NexusBox.doc) that configures how a 
file will be written and allows the library to automagically find some 
of the data it needs to write to the file.  This is akin to the XML data 
definitions and probably should be integrated with them somehow (I've 
been punting this one for a while now!) to provide file validation at 
write time.

Currently, there is no support for linking within a file as I haven't 
had a need for this feature yet.  I'm currently building for Nexus3.0 
and have it all working on Windows.  I need to build it for linux and 
solaris yet, but don't anticipate any problems there.  There is an 
accompanying C wrapper which I've used to call this library from IDL and 
Java plus it's likely python bindings are in the near future...

If this is of interest to you, or anyone else, let me know.  I'll need 
to clean up a few things and add the appropriate contract numbers and 
typical disclaimers, but there's no real barrier to others making use of 
this code--well, other than my general laziness anyway :)

Brian

Peterson, Peter F. wrote:
> Brian,
>
> I have been interested in producing a C++ wrapper around the napi for
> some time. Would you be willing to share this code with NeXus and
> possibly incorporate it into a future release?
>
> P^2
>
> -----Original Message-----
> From: nexus-bounces at nexusformat.org
> [mailto:nexus-bounces at nexusformat.org] On Behalf Of tieman
> Sent: Friday, December 08, 2006 10:39 AM
> To: Mark Koennecke; Nexus List
> Subject: Re: [Nexus] XML format--A way to serialize binary?
>
> Mark,
>
> Comments inline...
>
> Mark Koennecke wrote:
>   
>> Dear Brian Tiemann,
>>
>> tieman schrieb:
>>     
>>> Hello all!
>>>
>>> After much trial and tribulation (linking on Windows is a nightmare!)
>>>       
>
>   
>>> I have all my windows programs using Nexus 3.0...on to linux!
>>>
>>> Well, not quite yet...is there a way to serialize binary data into a 
>>> more compact format with NAPI?  I generally need to write image data 
>>> that can be quite large.  The "default" serialization method 
>>> generates an ascii string for each pixel.  While this is very 
>>> readable--it typically makes the files 4-5 times larger than encoding
>>>       
>
>   
>>> the data as binary.  It would be nice to be able to write the image 
>>> data as a raw binary dump to save space.
>>>
>>>   
>>>       
>> We implemented NeXus-XML in order to:
>> * be buzword compliant
>> * allow people to edit their data...
>> So the editable data is by design. If you want efficiency, use HDF-5. 
>> In principle you can
>> convert your data into an UINT8 array and print that as hex. The 
>> NeXus-API allows to set
>> the format strings for printing numbers. This will yield a compact 
>> representation but will not be readable
>> with the NeXus-API.
>>     
> Hmmm...seems like a simple hack in the Nexus-API should handle this.  If
>
> the appropriate flag is set, serialize binary data as a byte stream and 
> tag it such that we can unserialize it properly on read...I'm not sure 
> I'll ever need this feature, but I do currently have a web app that 
> reads Nexus nexus files and passes them over the net as serialized XML 
> in this manner so I did want to at least try writing XML formatted data.
>   
>>> I don't anticipate needing the use the XML format often, but XML 
>>> would make it easier to write a web app that can browse the data 
>>> online for instance...
>>>
>>>   
>>>       
>> Well, how do you write your web apps? If you do Java servlets, you may
>>     
>
>   
>> use the NeXus-Java interface.
>> If you do Tcl you may use the SWIG generated wrapper around the 
>> NeXus-API, if you use perl or php
>> you may help out with your knowledge to make the SWIG interface to 
>> those languages tick. This is
>> little work. If you use python, contact Peter Peterson, they have a 
>> python interface to NeXus.
>>
>>     
> I use Java.  I do not, and probably will not, use the NeXus-Java 
> interface.  The reason is that I have a C++ wrapper around Nexus that 
> I've been using for many years now.  The C++ wrapper maps the data file 
> to a data tree that can be accessed easily by paths and is an even 
> higher level interface than Napi.  This has been linked to Java and 
> other languages through a simple C wrapper that has also been in use for
>
> years now.  In short--now that I have a robust class structure that's 
> cross platform and well integrated into a number of apps--I don't want 
> to change it :)
>
> Thanks!
>
> Brian
> _______________________________________________
> NeXus mailing list
> NeXus at nexusformat.org
> http://lists.nexusformat.org/mailman/listinfo/nexus
>   

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: nexusbox.h
Url: http://lists.nexusformat.org/pipermail/nexus-developers/attachments/20061208/a1c00aad/attachment-0002.h 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: HDFBox.doc
Type: application/octet-stream
Size: 26624 bytes
Desc: not available
Url : http://lists.nexusformat.org/pipermail/nexus-developers/attachments/20061208/a1c00aad/attachment-0001.obj 
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: nexusapi.h
Url: http://lists.nexusformat.org/pipermail/nexus-developers/attachments/20061208/a1c00aad/attachment-0003.h 