Hello,

I encountered a problem using AnsiStrings. It isn't a real bug, but can make problems, like it does for me.

I noticed it with Free Pascal version 1.1, snapshot 17/11/2000 under Linux (i386).

The problem is that when altering AnsiStrings the greatest available memory block is becoming smaller and smaller, but much smaller than it is really needed.

For example, I made the following example program that creates a string of 5000 chars by adding one char at a time. I gave the program 1 mb of heap memory but it runs out of memory.

Here is this program:

-----------------8<------------------8<-------------------------------

{$MODE Delphi}       // I prefer Delphi programming style
{$M 16387, 1000000}  // 16 kb stack, 1 mb heap
program StringTest;
var
  S        : AnsiString;  // test-string
  i        : Integer;     // count-variables
  StartMem : Integer;     // total free memory at startup
  StartMax : Integer;     // greatest free memory block atstartup
  c        : Char;

begin
{ This disables the automatic heap growing when more memory is needed. As with the $M switch defined, the program has 1 mb of heap available, that should be enough for a 5,000 chars string... }

  HeapError := NIL;

  writeln('ANSISTRINGS TEST');
  writeln;
  writeln('Please note heap griwing is disabled. ',
          'The program may only use 1 mb of heap.');
  StartMem := MemAvail;
  StartMax := MaxAvail;
  writeln('Memory at program start: ', StartMem, ' bytes free, ',
           StartMax, ' in largest block.');
  writeln('Hit RETURN to continue...');
  ReadLn;
  c:='.';
  S:='start';
  For i:=1 to 5000 do
  begin
    S := S + c; // simple instruction to add one single character to a string
    writeln('String length = ', Length(S),' chars;  memory: ',
            MemAvail,' free, ',MaxAvail,' largest block.');
  end;
  writeln('memory loss: ',StartMem-MemAvail, ' bytes, ',
          StartMax-MaxAvail, ' in largest block');
  writeln('done.');
  writeln('S="',S,'"');
  writeln('That is, ', Length(S),' chars.');
  writeln('Memory at program end: ', MemAvail, ' bytes free, ',
          MaxAvail, ' in largest block.');
end.

-----------------8<------------------8<-------------------------------

And on my machine it generates the following output:

ANSISTRINGS TEST

Please note heap griwing is disabled. The program may only use 1 mb of heap. Memory at program start: 999920 bytes free, 999920 in largest block. Hit RETURN to continue...
String length = 6 chars; memory: 999856 free, 999856 largest block. String length = 7 chars; memory: 999856 free, 999824 largest block. String length = 8 chars; memory: 999856 free, 999824 largest block. String length = 9 chars; memory: 999856 free, 999824 largest block. String length = 10 chars; memory: 999856 free, 999824 largest block. [...]
String length = 3958 chars; memory: 995904 free, 7904 largest block. String length = 3959 chars; memory: 995904 free, 3984 largest block. String length = 3960 chars; memory: 995904 free, 7904 largest block. String length = 3961 chars; memory: 995904 free, 3984 largest block. String length = 3962 chars; memory: 995904 free, 7904 largest block. String length = 3963 chars; memory: 995904 free, 3984 largest block. String length = 3964 chars; memory: 995904 free, 7904 largest block. String length = 3965 chars; memory: 995904 free, 3984 largest block. String length = 3966 chars; memory: 995904 free, 7904 largest block. String length = 3967 chars; memory: 995904 free, 3984 largest block. Runtime error 203 at 0x080503B0
0x080503B0
0x0804FF40
0x0804C1F6
0x0804C696
0x0804C379
0x080539BF
0x080480B0
0x00000000

As you can see, the used memory begins to grow exponencially (well, to be exact, the free memory sinks correctly).

Can you correct this behavior? I have a big problem with a web server application that has to process large strings and in some cases it runs out of memory, even with 15 mb heap.

I restricted the maximum heap size to prevent the program use too much memory. So, if there is a way to restrict the maximum heap size in another way, let me know. But I don't want to give 50 mb heap at program startup. Best would be a growing heap, but with a maximum value.

Best regards,
Udo

This is nearly impossible to fix because the problem is the memory fragmentation:
there are a lot of memory blocks allocated and deallocated but the new blocks never fit into the available free blocks because there is new data concatenated.

I suggest that you estimate first, how long you string will get and then
you do a setlength(s,<estimated length>);

If you can't estimate the length, but you know that you strings are quite large,
say several kB, you do a manual setlength in some kB steps (untested):

aktsize:=1024;
setlength(s,1024);
For i:=1 to 5000 do
   begin
     S := S + c; // simple instruction to add one single character to a string
     if length(s)=aktsize then
       begin
          inc(aktsize,1024);
          setlength(s,aktsize);
       end;
     writeln('String length = ', Length(S),' chars;  memory: ',
             MemAvail,' free, ',MaxAvail,' largest block.');
   end;

This will also improve the performance of your program a lot because much fewer memory (re)allocations are performed and because setlength can use reallocmem
while s:=s+c always performs a copy with move.

Hello Florian,

Sunday, November 19, 2000, 5:12:47 PM, you wrote:

FK> This is nearly impossible to fix because the problem is the memory FK> fragmentation:
FK> there are a lot of memory blocks allocated and deallocated but the FK> new blocks never fit into the available free blocks because there FK> is new data concatenated.

I already supposed something like that and I already implemented your solution in some parts of the program, I just hoped there was a better solution.
Anyway, I don't understand completely why this problem comes up. As far as I understood, when I add something to a string, the new string is placed to a new memory block (because it wouldn't fit in the original memory block). So far so good, but I think the memory loss shouldn't be so much.
Have a look at this example (works like my sample program I posted):

First the string is set to "start" then a point "." is added in each step. AAAA, BBBB etc. marking previously used blocks (can be allocated):

"start"
"AAAAAstart."
"AAAAABBBBBBstart.."

shouldn't now fit the next block in the previously used blocks?

"start...BBBCCCCCCC"
"DDDDDDDDstart....C" (new additional memory needed!) "DDDDDDDDEEEEEEEEEstart....."
"start......EEEEEEFFFFFFFFFF" (again, no more memory needed) "GGGGGGGGGGGstart.......FFFF" (same here)

You see, never more memory than 3 times the real string is used (15 kb in my example and not >1 mb). Of course I don't know enough about memory organisation in fpc, but this seems quite logical to me. After two steps there should be always enough space left to re-use old blocks.

What makes the problem? Do you have problems re-using previously allocated (and then freed) blocks?

It's the all famous tradeoff problem memory<->speed.

We've received a new heap implementation from an user, some parts of it will be implemented in the future as the compiler is more stable.

Peter

IIRC
Remerging old blocks is slow, and only done for the smallest blocks.

Since you are using Linux, you can try to use CMem as a workaround.

Marco van de Voort ( or )