Hi again,
another question: is there any limit in FPC, which causes that only cca 1GB of memory can be used by FPC ?
I have program, which reads big XML file (cca 500MB) using TXMLDocument class and attemot to expand it in memory ends with "Out of memory" exception.
It is 32 bit program on 32 bit Windows 7 (with 4GB RAM). When exception is raised I still have more than 1 GB free memory and in Task mamager I see, that my program takes only 1 GB memory. Probably this is some other limitation , which causes that (max. stack size or so ?)
Can I workaround this somehow ?
Thanks
-Laco.

In our previous episode, LacaK said:

another question: is there any limit in FPC, which causes that only cca 1GB of memory can be used by FPC ?

No, but there is one that limits to 2GB.

With {$peflags $20} in the main file that is increasable to 3GB (reportedly 4GB for 32-bits apps under 64-bit windows, but I never tested that)

> another question: is there any limit in FPC, which causes that only cca > 1GB of memory can be used by FPC ? No, but there is one that limits to 2GB.

Yes

With {$peflags $20} in the main file that is increasable to 3GB (reportedly 4GB for 32-bits apps under 64-bit windows, but I never tested that)

Wow I never heared about such option.
This are my results:
(compiled application is Win 32 bit)

Before I have used this flag:
Win7 32 bit with 4GB RAM: Out of GB
Win7 64 bit with 4GB RAM: Out of GB

After I have used this flag:
Win7 32 bit with 4GB RAM: Out of memory at 1 GB (no difference) Win7 64 bit with 4GB RAM: Success! (application consumes approx 1.8 GB RAM, 1GB remains free)

Strange, does not ?
Why I still encounter 1 GB limit and not 2 GB without {$setpeflag $20} ? Why on Win32 I still get Out of memory also with {$setpeflag $20} , but not on Win64 ?
Take into account, that both OSes have 4GB RAM and application is same.

According to
https://msdn.microsoft.com/en-us/library/windows/desktop/aa366778%28v=vs.85%29.aspx it should be 2GB on 32bit and 4GB on 64bit Windows with *IMAGE_FILE_LARGE_ADDRESS_AWARE*

-Laco.

Is there a different behaviour on linux?

I guess there is no limit...(!?)

LacaK wrote

Wow I never heared about such option. This are my results: (compiled application is Win 32 bit)

How large are the memory blocks that you try to allocate.

it should be 2GB on 32bit and 4GB on 64bit Windows with *IMAGE_FILE_LARGE_ADDRESS_AWARE*

There is a difference between the total amount of memory you can allocate, and the maximum size of a single block of memory that you can allocate. The latter is much smaller than the former, because the address space gets divided in multiple parts due to where the program gets loaded, where dll's get loaded and where the kernel gets mapped.

Jonas

allocate, and the maximum size of a single block of memory that you can allocate. The latter is much smaller than the former, because the address space gets divided in multiple parts due to where the program gets loaded, where dll's get loaded and where the kernel gets mapped.

The kernel mapping *is* what gets influenced by that switch (among others).

Regards,
Sven

Hard to say.
This happens inside TXMLDocument class as content is reading from file in procedure ReadXMLFile.
May be that somewhere during processing occurs request for large block of data.
(when I look at Resource Monitor as amount of memory is allocated I see there jump in one moment . more than 500 MB at once)

I tried allocate at once 1GB and it succed.

I did simple program:

var
   m: array[1..15] of pointer;
   i: integer;
begin
   for i:=1 to 15 do begin
     m[i] := getmem(1024*1024*200);
     fillbyte(m[i]^, 1024*1024*200, 0);
     writeln(i);
     readln;
   end;
   for i:=1 to 15 do begin
     freemem(m[i]);
   end;
end.

Which "confirms" limit 2GB, because it stops with run time error just before 2GB boundary.
With {$setpeflags $20} under Win64 it can bypass 2GB.

So it leads me to conclusion, that somewhere inside ReadXMLFile must be big request for memory block (may be for only short time), which overcomes limit 2GB.
May be some kind of copy (assign) class instance to another ?

-Laco.

Btw is there any limit for number of memory block, which can be requested ? As far as my XML file has milions of nodes, may be that for each node is created instance of TDOMNode class (or TDOMElement I do not know), which in turn can cause that there is large amount of relative small memory blocks ...
L.

There is no limit, other than the fact that every block has some overhead, and hence you can allocate less memory in total if you allocate tons of small blocks instead of a few large ones. It's just 16 or 32 bytes per block though, so even with millions of blocks you won't lose more than a few tens of megabytes.

Jonas

I think that, I found root of my problem.
As far as I have million nodes under few nodes, there is maintained TDOMNodeList, which stores nodes in TFPList
As new nodes are appended TFPList.Expand is called And there is:

   if FCapacity > 127 then Inc(IncSize, FCapacity shr 2);

So if I have in list 1 000 000 items, then at once list is expanded by 250 000, which causes in one step out of memory
So my question is: can I somehow control increment count ? I think that ATM no.
So second question is can TFPList.Expand be modified, that for large FCapacity will be used smaller increment ? :-)

Thanks
-Laco.

In our previous episode, LacaK said:

TMemoryStream also has this behaviour, and indeed I override it in my own applications to let the growth taper off after 128MB. If you have a lot of midsized memory streams, it otherwise results in having 25%/2 = 12.5% unused memory lieing around.

But of course all datastructures based on a single array (and thus inviting very large block allocations) are fundamentally flawed because of it in the first place.

This is of course possible, but I think that you would be served better by either
- going to 64-bit. Today you have million nodes, tomorrow maybe 2, and then it will be a problem just as well.
- Using SAX and not store the whole DOM in memory, and at the same time use your own memory allocation routines for your structures.

Michael.

Which is why I recommended going the SAX way and store data manually.

There is no single good and simple catch-all mechanism.

Michael.

What about to simply adjust Expand procedure like:

   if FCapacity > 8*1024*1024 then IncSize := FCapacity shr 3
   else if FCapacity > 128 then IncSize := FCapacity shr 2
   else if FCapacity > 8 then IncSize := 16
   else IncSize := 4;
   SetCapacity(FCapacity + IncSize);

-Laco.

Good idea. I was not aware of such option.
(Unfortunatelly all work and testing on application was done, so rewrite is completelly is brrr ;-( )
Thanks for hint
-Laco.

It does not solve your problem at a fundamental level.

Maybe it does today, but in a year you will again bump into this problem, if your file grows. Then what ?

As Marco said: all datastructures based on a single array are flawed in this regard.

Michael.

In our previous episode, Michael Van Canneyt said:

Maybe in this case, but the question remains if tfplist is as good as it can be.

For TMemoryStream it was not such a big problem because realloc is virtual.

There is no single good and simple catch-all mechanism.

So that brings up the question of what is the advantage of still using a percentage with so very large block sizes? I assume less reallocation tries make the chance of reaching a larger blocksize more feasable?

Yes it is only kind of approximation and balancing "optimal step" between "less reallocation" and "more memory"
Which is application dependant, but may be that there are less applications, which will handle billions items in TFPList and which will need expand by 25% to save rellocations.
IMO it is reasonable for large number of items decrease "step" as saving memory will give higher gain for most applications.

Maybe it does today, but in a year you will again bump into this problem, if your file grows. Then what ?

In my case it will not happen ;-)), size is stable and will not change

-Laco.

LacaK wrote

So second question is can TFPList.Expand be modified, that for large FCapacity will be used smaller increment ?

I think a “x percentage of capacity” increment is the way to go. eg: 5 or 10%

Regards,
Graeme

Graeme Geldenhuys wrote

I think a “x percentage of capacity” increment is the way to go. eg: 5 or 10%

Scrap that idea. In hind sight, % increments is what is causing the problem.

Regards,
Graeme

And actually why not 'just' allocate exactly what is needed on demand ? May be slower but even less overhead.

-L.

Because Grow is called once for every item added. And if you add thousands of items, the cost of allocating 4 more bytes at a time and maybe moving the memory all the time is extremely high. This is why a balance needs to be found between the allocation frequency and the memory over usage cost, and also why you should always set Capacity to a plausible value before filling the list.

Exactly.

Michael.