[NeXus-code-tickets] [NeXusCode] #31: Memory Leak

[NeXus Data Format Library and Applications]

#31: Memory Leak
----------------------------+-----------------------------------------------
Reporter:  Freddie Akeroyd  |       Owner:     
    Type:  defect           |      Status:  new
Priority:  major            |   Milestone:     
Keywords:                   |  
----------------------------+-----------------------------------------------
 * Reported by POIRIER Stephane [stephane.poirier at synchrotron-soleil.fr] *

 Hi all,

 The process (at SOLEIL) in charge of recording NeXus files opens, writes
 data and closes each NeXus file several times. Furthermore this process is
 intended to be never stopped. During intensives tests I have noticed a
 systematic and continuous increase of the memory consumption. After
 investigation (using Valgrind 3.2.4) the problem seems to be somewhere
 between the NeXus and HDF5 libraries.

 Technical environment is :
 - NeXus API 3.0.0 (with memory leak fixed for time_buffer allocation in
 NX5Open function)
 - OS Linux RHEL4
 - gcc 3.4.4

 The problem can be reproduced by this simple piece of code:

 int main ()
 {
         const int nReOpen = 1000000000;
         int iReOpen;
         const char szFile[] = "files/leak_test_oc2.nxs";

         NXhandle fileid;
         if (NXopen(szFile, NXACC_CREATE5, &fileid) != NX_OK) return 1;
         if( NXclose(&fileid) != NX_OK) return 1;
         for( iReOpen = 0; iReOpen < nReOpen; iReOpen++ )
         {
                 if( 0 == iReOpen % 1000 )
                         printf("loop count %d\n", iReOpen);
                 if( NXopen(szFile, NXACC_RDWR, &fileid ) != NX_OK) return
 1;
                 if( NXclose(&fileid) != NX_OK ) return 1;
         }
         return 0;
 }

 On a PC running linux RHEL4 with 1Go of RAM this program 'eat' 0.1% more
 memory every 20-25 secs. So in one or two days the process eat all the
 memory (I made the test) and then crash.

 The attached files log_1000.txt and log_10000.txt are log files produced
 by Valgrind (with option --leak-check=yes) with 1000 and 10000 executions
 of the loop. One can see that the memory consumption is nearly linear
 between 1000 and 10000 loop counts.
 Surprisingly, setting nReOpen to 1000 (or 10000) and launch the program
 using Valgrind and wait for normal ending, the memory check tool will say
 that there is no leak, so I set nReopen constant to a huge number and
 break the execution (using ctrl+C) after roughly the expected count (using
 the printf) to get the leaks reports.


 Also, using the following C++ code, I can see a abnormal memory
 consumption when running for a long time:

 #include <napi.h>
 #include <sstream>

 #define PSZ(s) (s).c_str()

 const int nFiles = 1000000;
 const int nEntry = 10;
 const int nData = 10;
 int array_dims[2] = {5, 4};
 short int i2_array[4] = {1000, 2000, 3000, 4000}; int iFile, iReOpen,
 iEntry, iData, iNXdata, iSimpleArraySize = 4; const char szFile[] =
 "leak_test.nxs";

 int main ()
 {
         for( iFile = 0; iFile < nFiles; iFile++ )
         {
                 strFile.Printf("files/leak_test_%03d.nxs", iFile);
                 printf("file %s\n", PSZ(strFile));
                 NXhandle fileid;
                 if (NXopen(PSZ(strFile), NXACC_CREATE5, &fileid) != NX_OK)
 return 1;

                 for( iEntry = 0; iEntry < nEntry; iEntry++ )
                 {
                         ostringstream oss;
                 oss << "entry_" << iEntry;
                         if (NXmakegroup (fileid, PSZ(oss.str()),
 "NXentry") != NX_OK) return 1;
                         if (NXopengroup (fileid, PSZ(oss.str()),
 "NXentry") != NX_OK) return 1;
                         for( iNXdata = 0; iNXdata < nData; iNXdata++ )
                         {
                                 ostringstream oss;
                                 oss << "data_" << iNXdata;
                                 if (NXmakegroup (fileid, PSZ(oss.str()),
 "NXdata") != NX_OK) return 1;
                                 if (NXopengroup (fileid, PSZ(oss.str()),
 "NXdata") != NX_OK) return 1;
                                 for( iData = 0; iData < nData; iData++ )
                                 {
                                         std::ostringstream oss;
                                         oss << "i2_data_" << iData;
                                         if (NXmakedata (fileid,
 PSZ(oss.str()), NX_INT16, 1, &array_dims[1]) != NX_OK) return 1;
                                         if (NXopendata (fileid,
 PSZ(oss.str())) != NX_OK) return 1;
                                         if (NXputdata (fileid, i2_array)
 != NX_OK) return 1;
                                         if (NXclosedata (fileid) != NX_OK)
 return 1;
                                 }
                                 if (NXclosegroup (fileid) != NX_OK) return
 1;
                         }
                         if (NXclosegroup (fileid) != NX_OK) return 1;
                 }
                 if (NXclose (&fileid) != NX_OK) return 1;

                 // Delete file
                 unlink(szFile);
         }

 }

 In this case the increase in memory consumption is more slowly, +0.1% of
 memory consumption in about 16 minutes. Comparing with the real recording
 process it seems that the memory consumption is related to the amount of
 recorded data.

 Thanks for your attention.


 °°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°
 Stéphane POIRIER
 Groupe Gestion des données
 Tél. : 01 69 35 93 38
 SYNCHROTRON SOLEIL
 Bâtiment Central pièce A1.0.28
 L'Orme des Merisiers
 Saint-Aubin - BP 48
 91192 GIF-SUR-YVETTE CEDEX
 Site Web: <http://www.synchrotron-soleil.fr/>
 °°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°

-- 
Ticket URL: <http://trac.nexusformat.org/code/ticket/31>
NeXus Data Format Library and Applications <http://www.nexusformat.org/>
NeXus Data Format Library and Applications