[Nexus-developers] NXtranslate

[Mark Koennecke]

High everybody,

On Mon, 1 Dec 2003, Peterson, Peter F. wrote:

> Mark et al,
> 
> *majority* of file conversions in a simple way that allowed distributing
> work among developers and users. For cases that are not covered by
> NXtranslate, one can write a single purpose code using the napi
> directly. After talking with various people at various facilities, doing
> some *very* random reading and looking at code for other programs that I
> found useful (but not at all applicable) the following requirements were
> made:
>  - Simple: Do one thing and do it well. If there is a complicated case,
> a general purpose tool is not going to be sufficient anyhow. The person
> who wants to do more than NXtranslate can always has the napi.
>  - Flexible: it must be able to deal with writing files in a standard
> that has not been fully formalized.
>  - Extensible: people want to write NeXus files from a variety of
> formats, we should make it easy to do. This means that we give them a
> straightforward framework to plug into and their code will work without
> messing with our internals. 
>  - Straightforward: end users that want to convert from their old
> favorite format to their new favorite format (NeXus) can edit an
> existing XML translation file and run, without having to read any
> documentation.
>  - Small: This goes back to being simple, extensible, and
> straightforward; the less it does, and the fewer knobs, the easier it
> will be to document, maintain, and use.
>  - Starting format independent: It will not have a favorite language.
> While reading with napi is obvious, it should be no easier than reading
> from ISIS raw files or NCNR(ASCII) files.
>  - Portable: Must be able to run on the major platforms for facilities
> (Unix/Linux, Windows, and MacOS X). This restricts the implementation
> language, but not much.
> 
> My question for you (everybody on the list): Are you in a situation not
> covered by this design? This is not whether you can think of a case that
> isn't, but actually know of one.
> 
  Yes, I think so. My usage cases:
  - Example TRICS. This instrument is supposed to have three PSD. In
    practive any between 0-3 were available and stored at different times. 
    Currently, I believe, the count is two.
    I need a IF condition for: 
    IF group detector2 exists; 
	copy it
	create detector2, NXdata and make links
    ENDIF 
    A similar condition exists at FOCUS which started with one detector
    bank and grew another two over time. 
  - Again TRICS: They scan on their PSD. At each scanpoint I do a new
    entry. TRICS files can contain hundreds of entries. I need a FOR
    loop together with text replacement (or shall I call it a 
    variable?) in order to process all entries. 
  - Generally, my NeXus files grew over time. Instrument scientists kept
    coming and asking: I have this brand new magnet now, I want to put its
    value in the file, please do it. And it should be labelled magnet and
    not temperature. Well, NeXus is extendable, I fulfilled these wishes
    and now I will have to convert all theses files with variable content
    properly.  
  - My TOF files contain distances in the old format, i.e. distance to
    previous component. In order to generate new distances, I will have to
    add several of them, possibly invert sign etc. before storing them
    again. Some of the distances vary from file to file, so I do not get
    away with a simple constant. 

  In short if I do not have some form of programmatic control, I will have
  to write an own conversion utility. Sorry. So, I would add generality
  and programmatic control to the requirements list.


  Possible solutions to this dilemma include:
  - embedded code in the XML file alike Java Server Pages or ASP
  - Perhaps Jelly, an XML scripting language, can be a solution. See:
    http://jakarta.apache.org/commons/sandbox/jelly
    Unfortunately jelly is java only these days. 
  - Enhance the XML-DTD with programming constructs and use it as a
    template from which NXtranslate generates a suitable C program
    which can then be compiled and linked. This would also guarantee
    the most efficient translation programs. 

			Best Regards,

				Mark Koennecke