[NeXus-tech] flush in hdf4

[Paul Kienzle]

On Apr 8, 2009, at 2:54 AM, Mark Koennecke wrote:

> Paul,
>
> Paul Kienzle wrote:
>> Hi,
>>
>> After many hours I managed to build and test the new napi on OS X for
>> HDF4/5/xml.
>>
>> The storage size is now used by the python bindings when putting
>> character data rather than the length of the stored data.  In the
>> process of fixing this I cleaned up a dangling pointer reference.
>> Between these, I hope the python wrapper is now stable, and #162 can
>> be closed.
>>
>> A separate issue I fixed in the python bindings was to force a close
>> data for each open data. Otherwise, the path we are maintaining on
>> the python side gets confused.
>>
>> An unresolved issue is that HDF4 but not HDF5 or XML closes the
>> currently open data group on NXflush.  I noticed this because I was
>> interleaving flush class with putslab calls.  This wasn't caught
>> before because the opendata call was in the loop with putslab.  This
>> behaviour isn't documented.
>>
>>
> It is even worth. With HDF4 the file is closed and reopened. This is
> also why NXflush takes a pointer to the handle and not only the
> handle as an argument. There is a reason for this behaviour: At the
> time it was implemented and I think even now, the HDF-4 API did
> not provide for a proper flush. Thus to  get flush functionality  
> with HDF-4
> the only option is to close and reopen the file.

As a developer using the API I would rather not know what backend I'm  
using, because then I have to write code like:

   if file.format is not 'hdf4': file.flush()

For scanning instruments you want to read the next frame from the  
detector, write a slab to the file and flush the results so that even  
if the control computer goes down the partial measurement is still  
available.  In this case, the data rate is low enough that the caller  
can close and reopen the file.

For event mode data where frames are written frequently you want to  
flush the file structure before writing frames and periodically write  
frames so that a partial measurement is still available.  However,  
you don't want to incur extra overhead closing and reopening the data  
every frame.

Both of these can be handled if flush is supported only while data is  
closed, with the event mode program responsible for batching the  
writes so that flushes don't happen too frequently.  So long as it is  
documented that way there won't be any surprise, though better would  
be to raise an error when trying to flush while data is open even in  
HDF5.

   - Paul