Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / fpc-devel / 2006-12

Growing a memory stream

11 messages · 5 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

Vincent Snijders

Hi,

Currently the TMemoryStream grows in steps of 4096 bytes.

When writing 100000 chunks of say 80 bytes to it, this causes a lot of reallocations. I think it is better to grow at least a quarter for example.

The new code could look something like:
function TMemoryStream.Realloc(var NewCapacity: Longint): Pointer;
begin
   // round off to block size.
   If NewCapacity<0 Then
     NewCapacity:=0
   else begin
     if NewCapacity - Capacity < Capacity div 4 then
	NewCapacity := Capacity + Capacity div 4;
     NewCapacity := (NewCapacity + (TMSGrow-1)) and not (TMSGROW-1);
   end;
   // Only now check !
   If NewCapacity=FCapacity then
     Result:=FMemory
   else
     begin
       Result:=Reallocmem(FMemory,Newcapacity);
       If (Result=Nil) and (Newcapacity>0) then
         Raise EStreamError.Create(SMemoryStreamError);
     end;
end;

What do you think?

For background see:
http://www.freepascal.org/mantis/view.php?id=7270

Vincent

Reply by email

Michael Van Canneyt
to Vincent Snijders: Currently the TMemoryStream grows in steps of 4096 bytes. When writing 100000 chunks of…
Vincent Snijders wrote 31 lines
Hi, Currently the TMemoryStream grows in steps of 4096 bytes. When writing 100000 chunks of say 80 bytes to it, this causes a lot of reallocations. I think it is better to grow at least a quarter for example. The new code could look something like: function TMemoryStream.Realloc(var NewCapacity: Longint): Pointer; begin // round off to block size. If NewCapacity<0 Then NewCapacity:=0 else begin if NewCapacity - Capacity < Capacity div 4 then NewCapacity := Capacity + Capacity div 4; NewCapacity := (NewCapacity + (TMSGrow-1)) and not (TMSGROW-1); end; // Only now check ! If NewCapacity=FCapacity then Result:=FMemory else begin Result:=Reallocmem(FMemory,Newcapacity); If (Result=Nil) and (Newcapacity>0) then Raise EStreamError.Create(SMemoryStreamError); end; end; What do you think?

I don't see a problem with that. A similar mechanism is used to TStringList.

Michael.

Reply by email

Michael Van Canneyt
to Micha Nelissen: Very nice would be a linked list of 8KB blocks or so, and when you access .Memory, then…
Micha Nelissen wrote 7 lines
Vincent Snijders wrote: > Hi, > > Currently the TMemoryStream grows in steps of 4096 bytes. Very nice would be a linked list of 8KB blocks or so, and when you access .Memory, then it's copied into an array.

I think you can create a separate implementation for this, descended from TCustomMemoryStream ?

Michael.

Reply by email

Vincent Snijders
to Micha Nelissen: Very nice would be a linked list of 8KB blocks or so, and when you access .Memory, then…
Micha Nelissen wrote 8 lines
Vincent Snijders wrote: > Hi, > > Currently the TMemoryStream grows in steps of 4096 bytes. Very nice would be a linked list of 8KB blocks or so, and when you access .Memory, then it's copied into an array.

A (maybe shortsighted) drawback would be that it doubles the amount of needed memory, once for the list and once for the memory pointed to with Memory?

Vincent

Reply by email

Micha Nelissen
to Vincent Snijders
Vincent Snijders wrote
A (maybe shortsighted) drawback would be that it doubles the amount of needed memory, once for the list and once for the memory pointed to with Memory?

Well, the linked list would be freed (or the first item of it enlarged to the total size, depending on implementation details).

Micha

Reply by email

Vincent Snijders
to Micha Nelissen
Micha Nelissen wrote 7 lines
Vincent Snijders wrote: > A (maybe shortsighted) drawback would be that it doubles the amount of > needed memory, once for the list and once for the memory pointed to with > Memory? Well, the linked list would be freed (or the first item of it enlarged to the total size, depending on implementation details).

Still, the peak memory use will twice the size, won't it?

Vincent

Reply by email

peter green
to Vincent Snijders
> Well, the linked list would be freed (or the first item of it enlarged > to the total size, depending on implementation details). Still, the peak memory use will twice the size, won't it?

sure but unless you are very lucky a call to reallocmem also means alocate-copy-deallocate.

Reply by email

Vincent Snijders
to peter green
peter green wrote
>> Well, the linked list would be freed (or the first item of it enlarged >> to the total size, depending on implementation details). > Still, the peak memory use will twice the size, won't it? sure but unless you are very lucky a call to reallocmem also means alocate-copy-deallocate.

That is why I said "shortsighted" in my initial question :-)

Vincent

Reply by email