A recent query:
Date: Mon, 21 Nov 2011 08:27:16 +0000 From: Mark Morgan Lloyd Subject: [fpc-pascal] Debugging ancient heap allocation seems not to have received a reply. The problem of debugging memory allocation, however, is wider and not just involving ancient code!
I am using:
Compiler 2.4.2: IDE 1.0.12: Windows 7, Intel CORE i7 Application: image processing. Straightforward programming come out at 8 seconds per image: test data set is about 3000 images. Conversion of the program to use multiple threads for the time consuming raw image read and process part reduced this to 3 seconds per image, once I had purged the memory leaks. Initially, my program ran out of memory after 500 to 1000 images.
HeapTrc proved to be essentially useless. Depending apparently on random factors, HeapTrc reported usually no more than 4 blocks of unfreed memory, often was unable to reveal which routine had allocated some, most or all of the remaining blocks: on occasion the HeapTrc report sputtered to a halt with error messages. In particular, HeapTrc never reported unfreed blocks for any routine in the section of code subsequently found to be in error.
At this point, I added diagnostic logging: one file for the main program and one for each thread started:-
GetHeapStatus giving TotalAddrSpace, TotalAllocated, TotalFree. plus StackBottom, SPtr, StackTop, StackLength.
These were logged at the start, strategic places within, and at the end of the main program and each thread.
At no point did the heap status reveal the growing amount of space tied up: the only odd thing was that TotalAllocated sometimes came back negative in the threads as the program approached the point of running out of memory (at the 2GB addressing limit; the machine has 6GB).
The Stack numbers were more useful, showing intermittent increases in the addresses for the thread stacks. Oddly, the StackTop is returned as zero in the threads and StackLength always comes back as the value set in the IDE Options Memory Sizes and not as the value set in BeginThread. I think the correct value is actually used.
The problem, however, was finally solved only by going through the program line by line and:
Moving all workspace arrays to Global Common, including all the workspace for the threads (100 MB or so for each of 6).
Eliminating all dynamic arrays - this involved some preliminary runs to estimate (guess) the maximum size of arrays determined at run time. Eliminating all New/Dispose pairs and such like.
Even though many arrays are thus much larger than their typical size and all of them occupy space all the time, the space required was only around 1GB - half the 2GB the program previously ate up before running out of memory.
In the course of eliminating the workspace originally allocated as dynamic arrays, I discovered 2 unmatched SetLength procedures. As I mentioned above, these were never reported by HeapTrc.
You will gather I am not too impressed with the tools provided for tackling memory allocation problems!
Andrew Bennett