Does anyone else have any trouble reading heaptrc output? The only thing that I can see of use is that it tells me there /is/ a leak. However, trying to figure out where it is according to heaptrc output seems more difficult and time consuming that searching through the code.

I'm not criticizing per se, just frustrated that I've spend 40 mins looking at a heaptrc output file with still no clue as to where the leak may be.

Heap dump by heaptrc unit
4669 memory blocks allocated : 1093920/1110888
4621 memory blocks freed : 1092756/1109608
48 unfreed memory blocks : 1164
True heap size : 360448 (112 used in System startup) True free heap : 355808
Should be : 356368
Call trace for block $030FFC70 size 45
$0040734F
$005E9916 TTIOIDGENERATORGUID__ASSIGNNEXTOID, line 489 of M:/lazarus/components/tiOPF2/Core/tiOIDGUID.pas
$005CFA38 TTIOBJECT__CREATENEW, line 2036 of
M:/lazarus/components/tiOPF2/Core/tiObject.pas
$005CFAF5 TTIOBJECT__CREATENEW, line 2042 of
M:/lazarus/components/tiOPF2/Core/tiObject.pas
$004237AF TFLEXOBJECTMAPPING__REGISTERPROPERTY, line 259 of M:/lazarus/projects/flexserver/src/flex_persistence.pas $00423293 TMAPPINGREADER__REGISTERMAPPINGS, line 128 of M:/lazarus/projects/flexserver/src/flex_mapping_reader.pas $00422EBB TMAPPINGREADER__LOADMAPPINGS, line 66 of M:/lazarus/projects/flexserver/src/flex_mapping_reader.pas $00422C86 TFORM1__BUTTON1CLICK, line 40 of main_form.pas Call trace for block $00157D98 size 20
$0040A692
$005D5DAD TTIOID__CREATE, line 136 of
M:/lazarus/components/tiOPF2/Core/tiOID.pas
$005CF95E TTIOBJECT__GETOID, line 2022 of
M:/lazarus/components/tiOPF2/Core/tiObject.pas
$005CFA2B TTIOBJECT__CREATENEW, line 2036 of
M:/lazarus/components/tiOPF2/Core/tiObject.pas
$005CFAF5 TTIOBJECT__CREATENEW, line 2042 of
M:/lazarus/components/tiOPF2/Core/tiObject.pas
$004237AF TFLEXOBJECTMAPPING__REGISTERPROPERTY, line 259 of M:/lazarus/projects/flexserver/src/flex_persistence.pas $00423293 TMAPPINGREADER__REGISTERMAPPINGS, line 128 of M:/lazarus/projects/flexserver/src/flex_mapping_reader.pas $00422EBB TMAPPINGREADER__LOADMAPPINGS, line 66 of M:/lazarus/projects/flexserver/src/flex_mapping_reader.pas Call trace for block $0016D1C0 size 56
$0040A692
$005CFAB3 TTIOBJECT__CREATENEW, line 2041 of
M:/lazarus/components/tiOPF2/Core/tiObject.pas
$004237AF TFLEXOBJECTMAPPING__REGISTERPROPERTY, line 259 of M:/lazarus/projects/flexserver/src/flex_persistence.pas $00423293 TMAPPINGREADER__REGISTERMAPPINGS, line 128 of M:/lazarus/projects/flexserver/src/flex_mapping_reader.pas $00422EBB TMAPPINGREADER__LOADMAPPINGS, line 66 of M:/lazarus/projects/flexserver/src/flex_mapping_reader.pas $00422C86 TFORM1__BUTTON1CLICK, line 40 of main_form.pas $004C0BB4 TCONTROL__CLICK, line 2227 of ./include/control.inc $0050352F TBUTTONCONTROL__CLICK, line 72 of ./include/buttoncontrol.inc Call trace for block $00157D38 size 18
$00407ABF
$004087CA
$0042321C TMAPPINGREADER__REGISTERMAPPINGS, line 123 of M:/lazarus/projects/flexserver/src/flex_mapping_reader.pas $00422EBB TMAPPINGREADER__LOADMAPPINGS, line 66 of M:/lazarus/projects/flexserver/src/flex_mapping_reader.pas $00422C86 TFORM1__BUTTON1CLICK, line 40 of main_form.pas $004C0BB4 TCONTROL__CLICK, line 2227 of ./include/control.inc $0050352F TBUTTONCONTROL__CLICK, line 72 of ./include/buttoncontrol.inc $005039F5 TCUSTOMBUTTON__CLICK, line 164 of ./include/buttons.inc

...more and more of the same

What is not clear about this ?

In line 489 of M:/lazarus/components/tiOPF2/Core/tiOIDGUID.pas a memory block of 45 bytes is allocated and never freed.

Michael.

First, thanks for responding so quickly, Michael. Secondly, my apologies as I pasted the wrong output. The above was easy enough to find as I was not freeing an object list that held the offending objects.

The output that I meant to paste is this:

M:\lazarus\projects\flexserver\gui_test\flexserver.exe Heap dump by heaptrc unit
4669 memory blocks allocated : 1093938/1110912
4651 memory blocks freed : 1093626/1110576
18 unfreed memory blocks : 312
True heap size : 294912 (112 used in System startup) True free heap : 293264
Should be : 293456
Call trace for block $03579D08 size 16
$0040D45B
$00439AB7
$00422EBB TMAPPINGREADER__LOADMAPPINGS, line 66 of M:/lazarus/projects/flexserver/src/flex_mapping_reader.pas $00422C86 TFORM1__BUTTON1CLICK, line 40 of main_form.pas $004C0BB4 TCONTROL__CLICK, line 2227 of ./include/control.inc $0050352F TBUTTONCONTROL__CLICK, line 72 of ./include/buttoncontrol.inc $005039F5 TCUSTOMBUTTON__CLICK, line 164 of ./include/buttons.inc $00503F51 TBUTTON__CLICK, line 331 of ./include/buttons.inc Call trace for block $03579EE8 size 16
$0040A692
$0054783A
$00546A31
$00422EBB TMAPPINGREADER__LOADMAPPINGS, line 66 of M:/lazarus/projects/flexserver/src/flex_mapping_reader.pas $00422C86 TFORM1__BUTTON1CLICK, line 40 of main_form.pas $004C0BB4 TCONTROL__CLICK, line 2227 of ./include/control.inc $0050352F TBUTTONCONTROL__CLICK, line 72 of ./include/buttoncontrol.inc $005039F5 TCUSTOMBUTTON__CLICK, line 164 of ./include/buttons.inc Call trace for block $00157D08 size 20
$0040A692
$00546A31
$00422EBB TMAPPINGREADER__LOADMAPPINGS, line 66 of M:/lazarus/projects/flexserver/src/flex_mapping_reader.pas $00422C86 TFORM1__BUTTON1CLICK, line 40 of main_form.pas $004C0BB4 TCONTROL__CLICK, line 2227 of ./include/control.inc $0050352F TBUTTONCONTROL__CLICK, line 72 of ./include/buttoncontrol.inc $005039F5 TCUSTOMBUTTON__CLICK, line 164 of ./include/buttons.inc $00503F51 TBUTTON__CLICK, line 331 of ./include/buttons.inc Call trace for block $03579DA8 size 16
$0040D45B
$00439AB7
$00422EBB TMAPPINGREADER__LOADMAPPINGS, line 66 of M:/lazarus/projects/flexserver/src/flex_mapping_reader.pas $00422C86 TFORM1__BUTTON1CLICK, line 40 of main_form.pas $004C0BB4 TCONTROL__CLICK, line 2227 of ./include/control.inc $0050352F TBUTTONCONTROL__CLICK, line 72 of ./include/buttoncontrol.inc $005039F5 TCUSTOMBUTTON__CLICK, line 164 of ./include/buttons.inc $00503F51 TBUTTON__CLICK, line 331 of ./include/buttons.inc Call trace for block $03579D58 size 16
$0040A692
$0054783A
$00546A31
$00422EBB TMAPPINGREADER__LOADMAPPINGS, line 66 of M:/lazarus/projects/flexserver/src/flex_mapping_reader.pas $00422C86 TFORM1__BUTTON1CLICK, line 40 of main_form.pas $004C0BB4 TCONTROL__CLICK, line 2227 of ./include/control.inc $0050352F TBUTTONCONTROL__CLICK, line 72 of ./include/buttoncontrol.inc $005039F5 TCUSTOMBUTTON__CLICK, line 164 of ./include/buttons.inc //.. etc, etc.

As you can see, its the same thing, repeatedly and unfortunately, it doesn't make any sense to me where the rest of execution is going. I am using TXMLDocument in the "LoadMappings" method which is the only thing that I can think of (code below).

Maybe its related to this?
http://bugs.freepascal.org/view.php?id=13605

The only thing is that I have updated my lazarus with a latter snapshot but I guess it could be the same thing maybe.

My version is 9.27 rev. 21413 so it looks doubtful that is the problem but I cannot say for sure.

procedure TMappingReader.RegisterMappings;
var
   lNode: TDOMNode;
   lAttr: TDOMAttr;
   lRegs: TDomNode;
   lCounter: integer;
   lPropCounter: integer;
   lRegItem: TDOMNode;
   lObjReg: TFlexObjectMapping;
   // intermediary vars to hold xml object properties
   lClassName: string;
   lDataTable: string;
   lObjType: string;
   lPK: boolean;
   lPropName: string;
   lColName: string;
   lValType: string;
begin
   if FXML = nil then
     raise Exception.Create('XML DOM object is nil');
   lRegs := FXML.DocumentElement.FindNode('registrations');
   for lCounter := 0 to lRegs.ChildNodes.Length -1 do
     begin
       lRegItem := lRegs.ChildNodes[lCounter];
       lClassName :=
lRegItem.Attributes.GetNamedItem('classname').FirstChild.NodeValue;
       lDataTable :=
lRegItem.Attributes.GetNamedItem('tablename').FirstChild.NodeValue;
       lObjType :=
lRegitem.Attributes.GetNamedItem('objecttype').FirstChild.NodeValue;
       lObjReg := gMapppings.RegisterMapping(lClassName, lDataTable, lObjType);
       with lObjReg do
         begin
           for lPropCounter := 0 to lRegItem.ChildNodes.Length - 1 do
             begin
               lNode := lRegItem.ChildNodes.Item[lPropCounter];
               lpK :=
StrToBool(lNode.Attributes.GetNamedItem('pk').FirstChild.NodeValue);
               lPropName :=
lNode.Attributes.GetNamedItem('propname').FirstChild.NodeValue;
               lColName :=
lNode.Attributes.GetNamedItem('colname').FirstChild.NodeValue;
               lValType :=
lNode.Attributes.GetNamedItem('valuetype').FirstChild.NodeValue;
               //dtInteger, dtFloat, dtString, dtDateTime, dtBookean, dtBinary)
               lObjReg.RegisterProperty(lPropName, lColName,
                 TFlexDataType(Integer(gDataTypeStrToInt(lValType))), lPK);
             end;
         end;
     end;
end;

2009/10/5 Lee Jenkins :

hehehe .... Now I don't feel alone. The heap trace is from bottom-up. So when you read from top to bottom, the leak appear near the first source-code line you see.

But it's not always that easy or clear cut. There are many false positives! Sometimes you get 20 memory leaks. But that is simply because a crash occured and proper code clean-up was not triggered. Which is also bad by the way. Anyway, when you do find the actually memory leak, it will probably remove all the others too. But that proper code-cleanup is still an issue that needs to be fixed.

HINT HINT HINT HINT HINT HINT :
HINT HINT HINT HINT HINT HINT :
Develop you code from day one, line one with the heaptrc enabled. That way, as soon as you introduce a memory leak, it will be detected. Now at least you know what code you were working on before you compiled an ran you application. This way you find those buggers much easier! :-)

I have spent many hours fixing one or two hard to find memory leaks! I don't program now without heaptrc enabled. I only disable it on final compilation before the product is shipped.

Definitely related to TXMLDocument. Apparently, it is either a complete mess and fraught with memory leaks or am misunderstanding how to use it. Although, its not the first XML API I've used.

Once again, its back to Delphi I go :-(

Your trouble looks like issue 13605, which was fixed in the meantime. One way out is to update the FPC; if that's not possible, you might want to change your code to look like this:

   lRegItem := lRegs.FirstChild;
   while Assigned(lRegItem) do
     begin
       if lRegItem.nodeType = ELEMENT_NODE then
       begin
{1}     lClassName := TDOMElement(lRegItem).GetAttribute('classname');
         lDataTable := ...
         lObjType := ...
         lObjReg := gMapppings.RegisterMapping(lClassName, lDataTable,
lObjType);
         with lObjReg do
         begin
           lNode := lRegItem.FirstChild;
           while Assigned(lNode) do
             begin
               if lNode.nodeType = ELEMENT_NODE tnen
               begin
                 lpK := ...
                 lPropName := ...
                 lColName := ...
                 lValType := ...
               //dtInteger, dtFloat, dtString, dtDateTime, dtBookean,
dtBinary)
                 lObjReg.RegisterProperty(lPropName, lColName,
                   TFlexDataType(Integer(gDataTypeStrToInt(lValType))),
lPK);
               end;  // if lNode.nodeType
               lNode := lNode.NextSibling;
             end;  // while
         end;  // with lObjReg
      end;     // if lRegItem.nodeType
      lRegItem := lRegItem.nextSibling;
    end;       // while

Note that the change marked as {1} is not related to the problem, it just shows how to get the attribute value in a more short way.

Regards,
Sergei

I used heaptrc to detect memory leaks in a game with >20k lines of code. It took me just one day to remove all (out of lot of) memory leaks. I remember that I had to check every frame of stack to find the exact location of memory leak - it could be anywhere on the reported stack trace! I also had to set constant TraceSize to 16/32 in order to understand some memory leaks. Hope that helps :-)

Aleksa

I'll give it a try. I'm updating my snapshot installation of Lazarus on Windows, though I'm not sure if the fix will be included.

Also, Node.NextSibling doesn't seem to work either as it seems to return the exact same node and results in an endless loop.

I'll try again with the latest snapshot available. Pulling down and building from source is a bit of a pain on Windows.

I'm just a little surprised that there is/was such a problem with XML implementation in Lazarus as XML is fairly important these days. :-)

Just one thing in case you don't know already. There is a package "leakview" (it's included in lazarus, but not installed by default (afaik))

Once installed , it appears in the tools menu, and you can load the leak-trace output. it will display similar like the stack view, allowing you to click the lines and jump to the code.
That should save some time.

Martin

I have really a LOT of XML code in lazarus, and I've never encountered this.

Apart from private code, the whole of the FPC reference documentation is in XML, and is built almost daily. We'd definitely notice a problem of this magnitude.

Michael.

I will try this. Thanks Martin.

That shouldn't be possible. Please check if your code does not reassign the node within the loop (my sample was just to demonstrate the basic principle, I don't pretend it's completely correct). If not, please file a bug report with a sample which can be used to reproduce.

Speaking frankly, this one was probably the least important of all problems with that package :-)

Regards,
Sergei

Thanks Michael, that is very reassuring to know. I've updated my Lazarus install with latest snapshot and it seems to be ok as long as I only use .FirstSibling and .NextSibling to iterate over elements/nodes.

That seems to work fine.

Thanks again,

It should not be LoadMappings method?

Luiz

That was indeed a mistake on my part <sheepish/>

I have updated my lazarus to latest snapshot and it seems to work OK now.

Now, for my own personal education:

M:\lazarus\projects\flexserver\gui_test\flexserver.exe Heap dump by heaptrc unit
4669 memory blocks allocated : 1093938/1110912
4651 memory blocks freed : 1093626/1110576
18 unfreed memory blocks : 312
True heap size : 294912 (112 used in System startup) True free heap : 293264
Should be : 293456
Call trace for block $03579D08 size 16
$0040D45B
$00439AB7
$00422EBB TMAPPINGREADER__LOADMAPPINGS, line 66 of M:/lazarus/projects/flexserver/src/flex_mapping_reader.pas $00422C86 TFORM1__BUTTON1CLICK, line 40 of main_form.pas $004C0BB4 TCONTROL__CLICK, line 2227 of ./include/control.inc $0050352F TBUTTONCONTROL__CLICK, line 72 of ./include/buttoncontrol.inc $005039F5 TCUSTOMBUTTON__CLICK, line 164 of ./include/buttons.inc $00503F51 TBUTTON__CLICK, line 331 of ./include/buttons.inc

In the above trace, it looks like (and I have confirmed) the leak occurs inside of the LoadMappings method when I was using the TXMLDocument object before updating my lazarus snapshot, but I am curious as to why the addresses shown after "$00422EBB TMAPPINGREADER__LOADMAPPINGS" are blank and do not show entry into or use of TXMLDocument object. Unfortunately, I am not that experienced or skilled in the lower level stuff.

Thanks for any clarification.

Luiz,

As it turns out, it was my use of TXMLDocument inside of LoadMappings, but the heaptrc output did not include mention of TXMLDocument or related objects to infer it was the problem or related to the problem.

Thank you,

That's most probably because the fcl-xml package was compiled without debug info. Heaptrc uses debug info to resolve addresses into function names. Recompling the RTL and packages with `make clean all OPT=-gl; sudo make install' command typically solves such issues.

Lineinfo stuff is also not bug-free. Sometimes it helps (at least, helped for me) to run 'gdb your_program' at console and then enter 'disassemble <address>'. Gdb reads debuginfo in a different way, and therefore sometimes it prints function names where lineinfo fails.

Regards,
Sergei

Ah, got it. Thanks Sergei.