[Nexus-developers] NXconvert-NXtranslate

[Mark Koennecke]

High,

On Wed, 19 Nov 2003, Ray Osborn wrote:

> On 11/19/03 8:46, "Mark Koennecke" <Mark.Koennecke at psi.ch> wrote:
> 
> > Well, yes, you have to learn a scripting language. But you have to learn
> > something to use NXtranslate.
> 
> For NXtranslate, you need either to know the path to any particular item in
> the source file or to have a document of the source library tag names.
> That's all.  I think that is the minimum that would be required by any
> scheme, so it's not a great overhead.
> 
> > I think here is a major point which I missed to make sufficiently
> > clear in my first posting. I actually think that scripting is more
> > versatile, at the expense of a little overhead for the SEPD case. I
> > fail to see how Peters scheme handles the case when we have to compute
> > values before we write to a file. This is not negligible, I reckon I
> > have to redo distances in +80000 files! Or if you wish to convert to
> > our coordinate system. The latter would probably involve calculations
> > involving several angles and values from the source file. How
> > would this be reasonably expressed in Peter's tag scheme:
> >   <distance tag="nexus:/entry1/FOCUS/bank1/distance * -1 "/
> >          mime-type="nxtranslate-script">
> > or something?? Note that I need some scripting language here as well to
> > parse the expression. Or an own expression parser.
> 
> You raise some important issues here and in the rest of your message.  Peter
> and I did discuss the need for adding some scripting capabilities for the
> few cases where the data needed manipulation.  The thing I like about
> Peter's scheme is that it allows for external data to be read in, while
> leaving the rest of the NeXus file the same.  Peter came up with this scheme
> after talking with data acquisition people, who wanted to have a regular
> NeXus file containing all the instrumental information, but with a means of
> accessing the live data stored in electronic memory.  Mark's scheme doesn't
> allow that because it requires every data item to contain a script command,
> not just the external data.
> 
  I still seem not to get it: The NXtranslate-XML file is a recipe for
  converting files. It produces a NeXus file. If this NeXus file is to
  be filled with live data later or a SDS is to be rpelaced by live data,
  this must be indicated somehow (This is still undefined). Please give a
  more detailed usage scenario (who calls NXtranslate, when and where is
  the live data taken from, etc.)   

> It seems to me that there is a possible compromise here, combining the best
> of both schemes, i.e., put the script into the tag attribute.  In fact,
> Peter's example had a tag that contained SQL commands when the mime_type was
> a database, so this is not a significant extension.  The mime_type will
> determine whether this is a data path, a keyword, or a script command.  By
> putting the script into the "tag" attribute, we leave the contents of any
> data tag to the data itself.
> 
  I agree, this could be a compromise. With the restriction, that we might
  be limited in the length of the script if we package it in
  attribute. But this might always be overcome by defining a function in a
  <script>
  </script> section. If this is still alive? 

> I haven't thought through all the consequences of such a scheme, but I am
> anxious not to sacrifice simplicity for the cases where greater flexibility
> are required.  I believe that it is the job of NeXus utilities to hide
> complexity from the average user (like me).  Even with this compromise
> scheme, I am worried about the user having to learn a new scripting
> interface, with its own set of commands, for every different type of data we
> access.  However, I agree that some scripting is useful in more complicated
> translations.
> 
  I think my scheme is still simple enough for simple cases. For simple
  NeXus-NeXus conversions you get away with:
     copyFromSource file path
  For added data with:
     putText sometext
     putFloat someFloat
     putInt someInt
  This is a function style interface virtually everybody understands.
  And there is full scripting power if you need it.

  Moreover, I'am afraid that such conversions will usually be done 
  by wizards anyway, just to get it right. Especially if it comes to
  converting a large database of files. 

  Peter's scheme also involves considerable implementation overhead.
  - we need a mime-type to evaluator mapping.
  - we might need a mechanism to load and register a new mime type
    at runtime. 
  - we need several parsers in order to parse different nx_tags for
    different mime types. Rather then using the one coming for free with
    the embeddable scripting language.
  - we need to administer a stack of mime types and to keep track of
    stacking depth in order to pop at the appropriate place.  
			
			Best Regards,

				Mark