[Nexus] Another memory leak

[POIRIER Stephane]

Hi all,

The process (at SOLEIL) in charge of recording NeXus files opens, writes data and closes each NeXus file several times. Furthermore this process is intended to be never stopped. During intensives tests I have noticed a systematic and continuous increase of the memory consumption. After investigation (using Valgrind 3.2.4) the problem seems to be somewhere between the NeXus and HDF5 libraries.

Technical environment is :
- NeXus API 3.0.0 (with memory leak fixed for time_buffer allocation in NX5Open function)
- OS Linux RHEL4
- gcc 3.4.4

The problem can be reproduced by this simple piece of code:

int main ()
{
        const int nReOpen = 1000000000;
        int iReOpen;
        const char szFile[] = "files/leak_test_oc2.nxs";

        NXhandle fileid;
        if (NXopen(szFile, NXACC_CREATE5, &fileid) != NX_OK) return 1;
        if( NXclose(&fileid) != NX_OK) return 1;
        for( iReOpen = 0; iReOpen < nReOpen; iReOpen++ )
        {
                if( 0 == iReOpen % 1000 )
                        printf("loop count %d\n", iReOpen);
                if( NXopen(szFile, NXACC_RDWR, &fileid ) != NX_OK) return 1;
                if( NXclose(&fileid) != NX_OK ) return 1;
        }      
        return 0;
}

On a PC running linux RHEL4 with 1Go of RAM this program 'eat' 0.1% more memory every 20-25 secs. So in one or two days the process eat all the memory (I made the test) and then crash.

The attached files log_1000.txt and log_10000.txt are log files produced by Valgrind (with option --leak-check=yes) with 1000 and 10000 executions of the loop. One can see that the memory consumption is nearly linear between 1000 and 10000 loop counts.
Surprisingly, setting nReOpen to 1000 (or 10000) and launch the program using Valgrind and wait for normal ending, the memory check tool will say that there is no leak, so I set nReopen constant to a huge number and break the execution (using ctrl+C) after roughly the expected count (using the printf) to get the leaks reports.


Also, using the following C++ code, I can see a abnormal memory consumption when running for a long time:

#include <napi.h>
#include <sstream>

#define PSZ(s) (s).c_str()

const int nFiles = 1000000;
const int nEntry = 10;
const int nData = 10;
int array_dims[2] = {5, 4};
short int i2_array[4] = {1000, 2000, 3000, 4000};
int iFile, iReOpen, iEntry, iData, iNXdata, iSimpleArraySize = 4;
const char szFile[] = "leak_test.nxs";

int main ()
{
        for( iFile = 0; iFile < nFiles; iFile++ )
        {
                strFile.Printf("files/leak_test_%03d.nxs", iFile);
                printf("file %s\n", PSZ(strFile));
                NXhandle fileid;
                if (NXopen(PSZ(strFile), NXACC_CREATE5, &fileid) != NX_OK) return 1;

                for( iEntry = 0; iEntry < nEntry; iEntry++ )
                {
                        ostringstream oss;
                oss << "entry_" << iEntry;
                        if (NXmakegroup (fileid, PSZ(oss.str()), "NXentry") != NX_OK) return 1;
                        if (NXopengroup (fileid, PSZ(oss.str()), "NXentry") != NX_OK) return 1;
                        for( iNXdata = 0; iNXdata < nData; iNXdata++ )
                        {
                                ostringstream oss;
                                oss << "data_" << iNXdata;
                                if (NXmakegroup (fileid, PSZ(oss.str()), "NXdata") != NX_OK) return 1;
                                if (NXopengroup (fileid, PSZ(oss.str()), "NXdata") != NX_OK) return 1;
                                for( iData = 0; iData < nData; iData++ )
                                {
                                        std::ostringstream oss;
                                        oss << "i2_data_" << iData;
                                        if (NXmakedata (fileid, PSZ(oss.str()), NX_INT16, 1, &array_dims[1]) != NX_OK) return 1;
                                        if (NXopendata (fileid, PSZ(oss.str())) != NX_OK) return 1;
                                        if (NXputdata (fileid, i2_array) != NX_OK) return 1;
                                        if (NXclosedata (fileid) != NX_OK) return 1;
                                }
                                if (NXclosegroup (fileid) != NX_OK) return 1;
                        }
                        if (NXclosegroup (fileid) != NX_OK) return 1;
                }
                if (NXclose (&fileid) != NX_OK) return 1;

                // Delete file
                unlink(szFile);
        }

}

In this case the increase in memory consumption is more slowly, +0.1% of memory consumption in about 16 minutes. Comparing with the real recording process it seems that the memory consumption is related to the amount of recorded data.

Thanks for your attention.


°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°
Stéphane POIRIER
Groupe Gestion des données
Tél. : 01 69 35 93 38
SYNCHROTRON SOLEIL
Bâtiment Central pièce A1.0.28
L'Orme des Merisiers
Saint-Aubin - BP 48
91192 GIF-SUR-YVETTE CEDEX
Site Web: <http://www.synchrotron-soleil.fr/>
°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°°
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: log_1000.txt
Url: http://lists.nexusformat.org/pipermail/nexus/attachments/20070312/de2fc26f/attachment.txt 
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: log_10000.txt
Url: http://lists.nexusformat.org/pipermail/nexus/attachments/20070312/de2fc26f/attachment-0001.txt