ࡱ> '(&ܥhc e<K9JJJJJJJJJJJJJJJKCJJJJJKKKKKKKKKKUKXKCKJK!J"KKKKKJJJJKKKKJJJJK0`^JJJJJJKKKKTitle: The Nexus File System Revision Number: 2 Author: Brian Tieman Date: 3/10/1999 Technical Review: Date: Introduction: The HDF file system is a specification and associated API for saving date in a self describing, hierarchial, platform independent file. The HDFBox file system is a class library developed to allow the developer a simpler interface for writing HDF files while providing for a flexible way for the user to define the HDF file structure at run time rather than compile time. Traditionally, programs are developed to allow for the saving of data in very specific ways. Once a program is compiled, it understands how to write specific data to specific locations. If the user wants to change the data saved, or its location, the program often needs to be rebuilt to handle these changes. The HDF file system is designed to handle a large variety of data in a self describing manner. The HDFBox file system is a set of classes designed to allow the user more flexibility in what data gets saved as well as where it gets saved. By making an appropriate template file, the user can easily decide which pieces of data to save and where in the file they need to be. Currently, the data to be saved may reside in one of three locations: the template file itself, the program being run, or in an EPICS environment, in an EPICS process variable somewhere on a reachable IOC. A sample template file: Below is a sample template file. ;This is a template file to save a bare minimum of info. [name,CONST,"APS",NX_CHAR] [sector,CONST,"SRICAT",NX_CHAR] [beamline,CONST,"2TST",NX_CHAR] [station,CONST,"A",NX_CHAR] [data,VAR,"Image",NX_UINT16] {signal,CONST,"1",NX_INT32} [program_name,VAR,"ProgramName",NX_CHAR] {program_version,VAR,"ProgramVersion",NX_CHAR} [program_author,VAR,"ProgramAuthor",NX_CHAR] [start_time,VAR,"AcquisitionTime",NX_CHAR] ;End of file This file specifies a group named facility to be of class Facility. The Facility group will contain four fields--name, sector, beamline, and station. It also specifies a group named entry1 to be of class NXentry. Entry1 has three fields of its ownprogram_name, program_author, and start_time. The program_name field has an attribute named program_version. Entry1 also has a subgroup named data of type NXdata. The data group contains one fielddatawith one attributesignal. When loaded into the HDFBox class, the classes will parse the file and handle the information internally with minimal interaction from the acquisition process. When the acquisition process requests that a file be written, it will make that request of the HDFBox class. The HDFBox class will then take charge and make sure all the necessary information is retrieved and put into the desired location in the HDF file. The template file structure: The template file allows for three different containers for data. These containers are the Group, the Field, and the Attribute. The field is the most basic container. The field is used to hold actual pieces of data such as an image or users name. Fields are much like files in a directory tree. Fields may hold a variety of data types from strings to various sized integers to floating point values. Fields may also hold scalar data as well as multi dimensional data Fields may have attributes. Attributes are specific pieces of data to help define fields. A field named program_name used to hold the name of the acquisition program taking the data may have an attribute called program_version, which would contain information about the specific version of the program used to acquire the data. Attributes may only contain string data. Groups help organize fields and attributes into useful sets. In the above example, the facility group contains fields related to the facility where the acquisition program was run. The field name is known to be the name of the facility by virtue of being in the facility group. Somewhere in the file may exist another field called name, but it may be the name of the user or program. Hopefully the group in which it resides helps to eliminate any confusion. Groups may contain other groups as well as fields and attributes. These three containers when used together provide the user a flexible way to organize their data into files which can be read latter and understood easily by others. When making the template file, certain tags are used to determine whether you want a group, field, or attribute. Information about groups is always included in <>. Information about fields appears within [] and attribute information appears in {}. What appears within the <>, [], and {} needs to follow the details below. What appears outside of them is interpreted as whitespace by the parser and is thus ignored. In general it is a good idea to use tabs and spaces to organize the template file such that its easy to see the resulting structure and is easy to read. Comments may also be included by using the ; character. When a ; is encountered, everything until the end of the line is discarded by the parser. Currently, no fields or attributes may appear at the root level. All fields and attributes muct belong to a group. Also, the tag is a special tag to denote the end of this template file. Groups and fields defined after the tag will be ignored. Defining Groups: Groups are defined with the <> characters. Since fields and attributes can not belong to the root level, at least one group needs to be defined. Groups are defined with begin group, end group pairs with anything between the beginning and end belonging to the group. The start of a group is defined by a line similar to where group_class is the class of the group (for more about classes refer the HDF documentation) and group_name is the name of this particular group as it will appear in the HDF file. Both group_class and group_name are case sensitive. The group is ended with the following line where group_class is the group class to end and End is a literal string which must appear. In the sample above the lines [name,CONST,"APS",NX_CHAR] [sector,CONST,"SRICAT",NX_CHAR] [beamline,CONST,"2TST",NX_CHAR] [station,CONST,"A",NX_CHAR] define the group facility of class Facility. Defining Fields: Fields are defined by a line similar to the following [field_name location_type,location,data_type] Field_name is the name of the field as it should appear in the HDF file and is case sensitive. Location_type can be one of three valuesCONST, VAR, or PV. CONST simply means interpret whats in Location as a constant value which should always be written to this field as type data_type whenever files are generated. Location is read as a string value. If data_type is something other than NX_CHAR, the HDFBox class will attempt to make the conversion. VAR tells the parser to interpret location to be a variable association with one of the programs internal variables. In this case data_type is ignored in favor of the program internal data type for the associated variable. Associations to internal variables are defined at build time. Valid values for location, therefore, are limited to what was allowed for at build time. See the section on PV Associations below for more details. PV may also be a valid value for location_type provided the application was built to exist in an EPICS environment. If the application can speak EPICS, PV will tell the parser to interperat the value in location to be an EPICS PV. The HDFBox class will handle the PV connection and retrieve the information from the PV. An attempt will be made to retrieve the PV as the specified data_type. Location is the location of the data to be saved in the field. The location must be surrounded by and, where appropriate, may contain white space. If the location_type is CONST, for instance, the value within will be interpreted literally and saved to the field every time a file is written. If its PV, location will be expected to be a valid PV name. Data_type is the desired data type to be saved. Valid data_types are NX_CHAR, NX_INT8, NX_UINT8, NX_INT16, NX_UINT16, NX_INT32, NX_UINT32, NX_FLOAT32, and NX_FLOAT64. The meaning of these data types should be obvious. An attempt will be made by the parser to honor the requested data type, but in some cases, this may be overridden. Field_name, location_type, location and data_type must all be defined in a field definition. Their values must be separated by a , and are case sensitive. No whitespace can appear between the [] with the exception of what is in the for location. These are current limitations of the parser which should be removed in future versions. In the example above [name,CONST,"APS",NX_CHAR] defines a field called name to be of type NX_CHAR and have a constant value of APS. Every time a file is generated with this template, APS will be written into the field name. An example of using an EPICS PV might be [x_position,PV,"2idb:m1.VAL",NX_FLOAT32] Defining Attributes: Attributes are defined in a manner similar to fields. Attributes are defined by a line similar to the following {attribute_name location_type,location,data_type} Attribute_name is the name of the attribute as it should appear in the HDF file and is case sensitive. Location_type can be one of three valuesCONST, VAR, or PV. CONST simply means interpret whats in Location as a constant value which should always be written to this attribute as type data_type whenever files are generated. Location is read as a string value. If data_type is something other than NX_CHAR, the HDFBox class will attempt to make the conversion. VAR tells the parser to interpret location to be a variable association with one of the programs internal variables. In this case data_type is ignored in favor of the program internal data type for the associated variable. Associations to internal variables are defined at build time. Valid values for location, therefore, are limited to what was allowed for at build time. See the section on PV Associations below for more details. PV may also be a valid value for location_type provided the application was built to exist in an EPICS environment. If the application can speak EPICS, PV will tell the parser to interperat the value in location to be an EPICS PV. The HDFBox class will handle the PV connection and retrieve the information from the PV. An attempt will be made to retrieve the PV as the specified data_type. Location is the location of the data to be saved in the attribute. The location must be surrounded by and, where appropriate, may contain white space. If the location_type is CONST, for instance, the value within will be interpreted literally and saved to the attribute every time a file is written. If its PV, location will be expected to be a valid PV name. Data_type is the desired data type to be saved. Valid data_types are NX_CHAR, NX_INT8, NX_UINT8, NX_INT16, NX_UINT16, NX_INT32, NX_UINT32, NX_FLOAT32, and NX_FLOAT64. The meaning of these data types should be obvious. An attempt will be made by the parser to honor the requested data type, but in some cases, this may be overridden. It is most typical for attribute to appear under fields. Attribute_name, location_type, location and data_type must all be defined in a attribute definition. Their values must be separated by a , and are case sensitive. No whitespace can appear between the [] with the exception of what is in the for location. These are current limitations of the parser which should be removed in future versions. In the example above {program_version,VAR,"ProgramVersion",NX_CHAR} defines an attribute called program_version to be of type NX_CHAR and have a VAR association with an internal program variable. Every time a file is generated with this template, the current value of the variable associated with ProgramVersion will be written into the attribute name. PV Associations for use with the HDFBox file system Programs built with the HDFBox file system often acquire and track their own data. They may perform special calculations of the data or in some way modify the data. Users will want to save this data. To avoid having to constantly rebuild the program to accomidate saving new sets of data, and to lessen the dependance of the program itself on the HDFBox file system, the concept of associations was developed. Basically an association is a way for the HDFBox classes to associated a request for information from the user with the internal variable where the data is stored. For this to happen, the program must register all the variables it wishes to make available to the HDFBox file system with a name the HDFBox file system can associate with that information. Once this registration is done, the HDFBox classes need only make access to the association to retrieve the datanot the variable itself. For example, if a program acquires images, one association it might want to make is an association between a piece of memory where the image resides and the name Image. Once Image is registered, the template file can request a field be a VAR field with a location of Image. Communication between the HDFBox class and the program will then insure that the right piece of memory gets used. If the acquisition uses a ping-pon buffer to acquire the image into, it may redirect the association to point to the appropriate buffer automatically. The user needs to have no knowledge of this since the template file requests the association. It is up to the program to make sure the HDFBox is up to date. This in mind, here is a list of associations for various programs in use. CCD_Image_Server associations: This section lists sample associations specific to the CCD_Image_Server program. These associations are not inclusive and may not be valid for any other program. Association Name Internal Variable Used Meaning ProgramName PROGRAM_NAME Name of program writing the files ProgramVersion PROGRAM_VERSION Current version of the program ProgramAuthor PROGRAM_AUTHOR Author of the program AcquisitionTime time_buffer Time image was taken Image roi->GetBufferAddress () Acquired image /=0 &aa0 & Q"Q"0 &Q"Q"56{ "'-2<FLQ %-8AS[gz%(2?Hhrx& 0 x!!-"6"8"B"""s#}###%%)%%%L&a&&&&]VV]Ucc uDa]&&&&_''''(K(M(\((()#)q)z)))))G*Q***X+b++++ ,,,_-h-j-t-......000d1l1112222A277778 8:!:::<<u^]UV]VB5`{|}~@  3 W {  - . [ "& ' ( ) 1-"& ' ( ) 1-'        G' \  &'?@TU'Kkz{-{&' 8"#$L&a&b&&&4'^'_'''''((L(M(((%*+j-.00i01112,22435363j3k364 5m669::!:"::::6;w;;;<<K@Normala "@" Heading 1Uc"@" Heading 2V"A@"Default Paragraph Font9<&<  {2<!"#$59/<_<<@CTimes New Roman Symbol "Arial1Courier New"hZ;,RaA/dCThe HDFBox file system Briant Tieman Brian Tieman  !"#$%*2Root Entry F0`^)WordDocumentKCompObjjSummaryInformation(  FMicrosoft Word Document MSWordDocWord.Document.69qOh+'0 ( P \ ht|The HDFBox file systemBriant TiemanNormal Brian Tieman5Microsoft Word for Windows 955DocumentSummaryInformation8   FMicrosoft Word Document MSWordDocWord.Document.89qArgonne National Laboratoryd The HDFBox file system@@!h,@i=A/՜.+,0@Hlt | Argonne National Laboratoryd The HDFBox file system