[Nexus-developers] NXconvert-NXtranslate

[Akeroyd, FA (Freddie)]

NXconvert/NXtranslate both offer a powerful new tool, which is more than
their names suggest: they will allow a NeXus file to be more created
from external sources and a template file. Mark's NXconvert approach of
using a 
scripting language is completely flexible - from the script you can call 
arbitrary external programs and parse the results, but it does require some 
knowledge of scripting even once the basic data access functions have been
set up.
Peter's NXtranslate approach leads to simpler looking template files, but 
requires a more complicated extending of NXtranslate to handle each input 
source and tag. Data transformation may be a problem with NXtranslate - you 
may only have a predefined amount of information that is passed out to your
function 
at any one point. Though I like the simpler look of Peters NXtranslate files
better, and users would find them easier, it may not be flexible enough. 
By the way, I don't quite understand the "live data" discussion? True you
can just paste data directly into an NXtranslate file from anywhere and it
will
be interpreted correctly, but why not just have a "getlivedata" command
embedded in 
the nxconvert input file?

I did wonder if nxconvert input files would look better if all scripting
commands had 
<script></script> around them - that would allow you to embed real data and
not
need to use "writeText" commands etc. each time. It would also allow you the
option
to use multiple scripting languages or interpreters by passing the
interpreter reference as an attribute of <script>, or also to do things like


<script interpreter="/bin/sh">date</script> and
<script interpreter="/path/to/my/program">command_line_for_program</script>

to generate NeXus files. The NXconvert program could also automatically set 
certain information up in environment variables, and maybe an "export" 
attribute to <script> specify any other environment bits that might be
needed?

A combined nxtranslate/nxconvert scheme where you can reference built-in
capabilities from something like NXtranslate, but with the option to
accessing
external scripting, is probably best. Script could be put in the "data"
area, and the "mime type" would tell how it should be treated. Unless each
script was going to be executed by spawning a new process, Nxtranslate 
would need to know a bit about scripting so it could create
the initial interpreter and pre-set any required variables. Though spawning
an
external process is slower, it does remove any script tie ins - all
attributes
of the current <script> level (tag, input file etc) could be created as 
temporary environment variables before the external script is invoked. If
this 
proves too slow, it would always remain as an option but we could build in 
support for popular scripting languages (perl,tcl,python) that are run
natively.   
 
In both schemes the final file structure has to be specified fully in the
translation
file - you have commands to insert data at a given point, but not new groups
or to
loop over groups. If you were moving <data1> <data2> ... <datan> from one
file 
to another you would have to specify them each individually in the
translation file, and may need multiple translation files if n varied for
your
instruments. However to get round this may mean the translation file looks
too "script like" and would also require extra parsing and interpreting.

Freddie