Hi all,

I'm having a problem here in a sequential image processing application that seems to test a particularly bad operation mode for the RTL heap manager (on Windows, but I don't think this matters here).
The work load looks like this: load "normal sized" image, do some processing, calculate a few values, create thumbnail in memory, next. By "normal sized" I mean something like 35MB uncompressed (regular DSLR resolution) and smaller. It's threaded code, but I'm describing the single worker operation here.

This appears to trigger insane memory fragmentation: after execution on 40 test files, I'm left with 250MB working set, while GetHeapStatus only reports 600k used (which seems correct, my data structures add up to 500k). This is never released to the OS. In fact, I've had to set the LAA flag just so that the program can work on the real data at all, with some 2.6GB working set for 1.07MB of used memory.

Is there any tuning option that could use maybe fix that? Some growing or reallocating flag or something?

As a quick test, I've tried my partial port of FastMM4 which works just fine, no fragmentation and peaks at 40MB used memory, which fits with the largest image.

But this is such a reproducible test case, maybe there is something that can be done to improve the RTL MM?

can you break it down into a test program and post?

Regards,
Ryan Joseph

can you break it down into a test program and post?

I should probably put the project on Github anyway, it's rather small, and it shouldn't have any dependencies. Give me an hour to clean it up a bit.

You'll have to supply your own images though (just copy one 10 times or something), as I don't own the copyright to anything in my test dataset.

In the mean time, here's an overlay of Process Explorer for two test runs: http://puu.sh/DAHMA/d83bfe7215.png
Red/Left axis is FastMM, Green/Right axis is the RTL MM. So the high memory use occurs at the same time, only the RTLMM rarely returns memory so it builds up.

> can you break it down into a test program and post? I should probably put the project on Github anyway, it's rather small, and it shouldn't have any dependencies. Give me an hour to clean it up a bit.

This should be good enough. It's been only used by like four people so the UI is a bit of a mess, but you'll only need one button to see the problem.

So, here are the steps to reproduce:
- Get the code from
<https://github.com/martok/ImageHash/tree/bug-fpcmm-fragment>
- Compile (originally written with FPC trunk >= late 2017, best use current trunk; LCL version shouldn't matter much)
- The default mode puts the exe in bin_i386, right next to the DLLs
- Run
- Provide data (~10 3MB JPEGs should do, or just paste one 10 times), either:
- paste the full path to a directory containing a number of images in the top-center memo (no, there's no SelectDirectory dialog)
- put the images in a folder 'data', that gets auto-added in this version
- Click 'Read'
- Wait for it to find visually similar images in your input
- Compare memory as reported by task manager or ProcExp to HeapStatus that is printed in the bottom-left log memo.

Note that the number of threads used (defaults to HostCPUs-1) doesn't matter (can be changed down to 1), so it's not a concurrent alloc problem. You can also apply the attached patch to even run the comparator in serial, but that doesn't change anything either.

On Jun 2, 2019, at 2:49 PM, Martok wrote: This should be good enough. It's been only used by like four people so the UI is a bit of a mess, but you'll only need one button to see the problem.

Sorry this is just too much for me to look at without spending considerable time (and I’m on Mac also). Can’t you reduce it down to a single test program?

Regards,
Ryan Joseph

Op 6/2/2019 om 6:49 PM schreef Martok:

Note that it is fairly typical that also frustrated pre fastmm D7. Back then I wrote a simple pool-factory combo class for it, and with D2009 I upgraded it to generics,so that I can easily create pools for many such objects.

A possible issue might be that I use a simple tthreadlist for its storage, but since D2009 all those classes level primitives work with tmonitor spinlocks before them. That might be slower on FPC for high counts.

Marco van de Voort wrote

Note that it is fairly typical that also frustrated pre fastmm D7. Back then I wrote a simple pool-factory combo class for it, and with D2009 I upgraded it to generics,so that I can easily create pools for many such objects.

Well, there is a general-purpose pooled allocator already. It's called the heap manager ;-)

Jokes aside, I did a bunch of counting. The only thing that sticks around are 50kB TLazIntfImage instances. Everything else gets freed right after processing. Just from the order of alloc/frees, there shouldn't even be gaps, so I would expect the next iteration to just find those allocations to be empty and reuse them. But for some reason, that doesn't happen.

I've compiled with FastMM for now, but it turns out this is in fact a bit slower than the RTL MM, and I don't really trust my port yet...

Try compiling the heap manager with "-dBESTMATCH". This makes it a bit slower but greatly reduces fragmentation.

Burkhard Carstens wrote

Try compiling the heap manager with "-dBESTMATCH". This makes it a bit slower but greatly reduces fragmentation.

Thanks, I'll give that a try!

Just to be clear, that needs to be set when compiling the RTL, right?

I think so. I usually set it when I "make all OPT=-dBESTMATCH" the compiler.

>> Try compiling the heap manager with "-dBESTMATCH". This makes it a >> bit slower but greatly reduces fragmentation. I think so. I usually set it when I "make all OPT=-dBESTMATCH" the compiler.

Thanks.

BESTMATCH is only marginally better for the test case, but is much slower (i.e. fully bootstrapping takes 50% longer). I guess there is something fundamentally not very well suited to this type of workload in the RTL MM.

But it was worth a shot...

Best,

Sebastian