Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / fpc-pascal / 2005-03

Re: ReAllocMem problem (?)

14 messages · 8 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

Micha Nelissen
Show quoted text 68 lines
---------------------------- Original Message ---------------------------- Subject: Re: [fpc-pascal] ReAllocMem problem (?) From: "Marcel Martin" Date: Wed, January 19, 2005 0:57 To: "FPC-Pascal users discussions" -------------------------------------------------------------------------- Marcel Martin a écrit : > > Hello, > > The following function SysReAllocMem comes from the file > /rtl/inc/heap.inc (FPC 1.9.7 / Win32) > > function SysReAllocMem(var p: pointer; size: ptrint):pointer; > [...] > { Resize block } > if not SysTryResizeMem(p,size) then > begin > minsize := MemoryManager.MemSize(p); > if size < minsize then > minsize := size; > p2 := MemoryManager.AllocMem(size); > if p2nil then > Move(p^,p2^,minsize); > MemoryManager.FreeMem(p); > p := p2; > end; > [...] > > Maybe you have good reasons to write it like you did but is > the instruction "MemoryManager.FreeMem(p);" intentionally > always executed? > I thought that, in case it cannot increase the size of the > memory allocated to p^, the memory manager would set p to nil > but not that it would free it. > > Suppose OneClass has a field FP which points to an array of > pointers. In case of problem when attempting to increase the > size of FP^, FP is freed and OneClass can no more destroy the > pointed objects it created. > > Up to now, I believed the code was something like > > if p2nil then > begin > Move(p^,p2^,minsize); > MemoryManager.FreeMem(p); > end; > p := p2; <- p might be set to nil but it is not freed > > and, in OneClass, one could do > > Q := FP; > ReAllocMem(FP, NewSize (greater than current size)); > if FP = nil then > begin > FP := Q; > Signal the 'Out of Memory' problem > end; > > Was your programming intentional? And if so, why? > Thanks. Considering the numerous answers I received, I am going to ask the questions otherwise: Is it a bug or not? Should I wait for a fixed version or should I manage to use it as it is because it will stay as it is? :-) Thanks.

The problem with your approach is: how do you know whether the memory was extended, or left alone because there was no memory available to extend it ? In general, shortage of memory produces a runtime error, but this is configurable by a global var.

Show quoted text 5 lines
There is an other thing I didn't notice yesterday when I sent my post. SysReAllocMem makes use of MemoryManager.AllocMem instead of MemoryManager.GetMem, i.e., SysReAllocMem fills up with 0s the new allocated memory (whereas the ReAllocMem doc says "only the used memory is initialized, extra memory will not be zeroed out").

Not zeroing it would trigger a bug in the compiler, so I left it in. (Can not remember precisely, but the context was an array of pointers that is realloc'ed, needing the new entries to be zero/nil).

Micha

Reply by email

Marcel Martin
to Micha Nelissen
Micha Nelissen wrote 8 lines
The problem with your approach is: how do you know whether the memory was extended, or left alone because there was no memory available to extend it ? In general, shortage of memory produces a runtime error, but this is configurable by a global var.

No, that's easy. If the returned pointer is nil, it signals the problem. But returning a nil pointer doesn't mean losing the pointed memory area. We can save the pointer value before calling ReAllocMem (in such situation, Delphi 5 returns a nil pointer but the memory area is not lost).
The problem comes overall from the fact that ReAllocMem was poorly designed (presumably by Microsoft, but I don't know). But, now, it is as it is and we live with that. ReAllocMem shouldn't destroy anything just because it cannot increase something.
For instance, in my program Primo (Primality certification, written with Delphi 5), if such a problem occurs (OutOfMemory), Primo will close down but BEFORE, it will try to save data in order to not lose all the work already done [*]. Now, if the memory manager kills the addresses of the created objects, no more things can be saved.

[*] Well, Primo automatically saves the work in progress every 10 minutes but anyway (I will be died otherwise. Some Primo users reported certification running times greater than 4000 hours :-)

Show quoted text 14 lines
>There is an other thing I didn't notice yesterday when I sent my >post. SysReAllocMem makes use of MemoryManager.AllocMem instead of >MemoryManager.GetMem, i.e., SysReAllocMem fills up with 0s the >new allocated memory (whereas the ReAllocMem doc says "only the >used memory is initialized, extra memory will not be zeroed out"). > Not zeroing it would trigger a bug in the compiler, so I left it in. (Can not remember precisely, but the context was an array of pointers that is realloc'ed, needing the new entries to be zero/nil).

It would have been preferable to try to fix this particular bug otherwise. The fact that the blocks of memory are set to 0 is a big waste of time. In the big integer library [*] I am adapting to Free Pascal (and not only big integers, but also polynomials over GF(2), big real/complex floats), the routines make lots of memory 'reallocations'.

[*] BTW, congratulations for the 1.9.8 version. At the moment the running times I get are slightly better than those got with Delphi. Not much better but better anyway ;-)

Well, if you cannot change this, you cannot. Currently, I only use ReAllocMem when the memory size decreases, otherwise I put a BUGGY_REALLOCMEM directive in the code that enables the use of GetMem and Move instead of ReAllocMem. Using GetMem and Move is not really slower than the current behaviour of ReAllocMem when the memory size increases. And, this way, I am sure (well, almost) that the content of the memory area I want to increase won't be destroyed.

mm

Reply by email

mm
to Micha Nelissen: The problem with your approach is: how do you know whether the memory was extended, or…
Micha Nelissen wrote
The problem with your approach is: how do you know whether the memory was extended, or left alone because there was no memory available to extend it ? In general, shortage of memory produces a runtime error, but this is configurable by a global var.

No, that's easy. If the returned pointer is nil, it signals the problem. But returning a nil pointer doesn't mean losing the pointed memory area. We can save the pointer value before calling ReAllocMem (in such situation, Delphi 5 returns a nil pointer but the memory area is not lost).
The problem comes overall from the fact that ReAllocMem was badly designed (presumably by Microsoft, but I don't know). But, now, it is as it is and we live with that. ReAllocMem shouldn't destroy anything just because it cannot increase something.
For instance, in my program Primo (Primality certification, written with Delphi 5), if such a problem occurs (OutOfMemory), Primo will terminate but BEFORE, it will try to save data in order to not lose all the work already done [*]. Now, if the memory manager kills the addresses of the created objects, no more things can be saved.

[*] Well, Primo automatically saves the work in progress every 10 minutes but anyway. I would be dead since long otherwise, some Primo users reported certification running times greater than 4000 hours :-)

Show quoted text 9 lines
>There is an other thing I didn't notice yesterday when I sent my >post. SysReAllocMem makes use of MemoryManager.AllocMem instead of >MemoryManager.GetMem, i.e., SysReAllocMem fills up with 0s the >new allocated memory (whereas the ReAllocMem doc says "only the >used memory is initialized, extra memory will not be zeroed out"). > Not zeroing it would trigger a bug in the compiler, so I left it in. (Can not remember precisely, but the context was an array of pointers that is realloc'ed, needing the new entries to be zero/nil).

It would have been preferable to try to fix this particular bug otherwise. The fact that the blocks of memory are set to 0 is a big waste of time. In the big integer library [*] I am adapting to Free Pascal (and not only big integers, but also big polynomials over GF(2), big real/complex floats), the routines make lots of memory 'reallocations'.

[*] BTW, congratulations for the 1.9.8 version. At the moment the running times I get are slightly better than those got with Delphi. Not much better but better anyway ;-)

Well, if you cannot change this, you cannot. Currently, I only use ReAllocMem when the memory size decreases, otherwise I put a BUGGY_REALLOCMEM directive in the code that enables the use of GetMem and Move instead of ReAllocMem. Using GetMem and Move is not really slower than the current behaviour of ReAllocMem when the memory size increases. And, this way, I am sure (well, almost) that the content of the memory area I want to increase won't be destroyed.

mm

Reply by email

Micha Nelissen
to mm
Show quoted text 12 lines
Micha Nelissen wrote: >The problem with your approach is: how do you know whether the memory >was extended, or left alone because there was no memory available to >extend it ? In general, shortage of memory produces a runtime error, >but this is configurable by a global var. > No, that's easy. If the returned pointer is nil, it signals the problem. But returning a nil pointer doesn't mean losing the pointed memory area. We can save the pointer value before calling ReAllocMem (in such situation, Delphi 5 returns a nil pointer but the memory area is not lost).

Ah I see, you are right. The documentation does not explicitly say that returning a nil pointer means the memory area has been freed. So both choices are possible within documentation.

In this case I would advise not to free the pointer and explicitly document this choice in the freepascal documentation. (As the user can free the area himself anyways, but can not bring it back). What do the other fpc developers think?

Show quoted text 10 lines
>Not zeroing it would trigger a bug in the compiler, so I left it in. >(Can not remember precisely, but the context was an array of pointers >that is realloc'ed, needing the new entries to be zero/nil). > It would have been preferable to try to fix this particular bug otherwise. The fact that the blocks of memory are set to 0 is a big waste of time. In the big integer library [*] I am adapting to Free Pascal (and not only big integers, but also big polynomials over GF(2), big real/complex floats), the routines make lots of memory 'reallocations'.

Maybe it would help you more to allocate more space in advance? Or use a data structure not requiring reallocations ? Just some suggestions...

I think it's the same as with classes that are automatically zeroed when creating, but those aren't so big usually... Probably the bugs in the compiler can be found and fixed easily, but also this choice I leave to the core developers. Also seen the fact that delphi does not zero it also, can argue for us not doing it either...

Micha

Reply by email

mm
to Micha Nelissen
Micha Nelissen wrote
In this case I would advise not to free the pointer and explicitly document this choice in the freepascal documentation. (As the user can free the area himself anyways, but can not bring it back).

Yes, exactly.

Maybe it would help you more to allocate more space in advance? Or use a data structure not requiring reallocations ? Just some suggestions...

Your are right, it would be faster but this is not possible. For instance, suppose you are a user of the library and you want to compute !10000, i.e., Factorial 10000. Currently, all you have to do is to write "IFactorial(N,10000)". Now what if, before calling IFactorial, it is necessary to estimate the size of !10000 in order to oversize N? Yes, you lovely archive the library in the trush and you search for an other one :-)

mm

Reply by email

Jilani Khaldi
to Marcel Martin: No, that's easy. If the returned pointer is nil, it signals the problem. But returning a…

Hi All,

I have written a little library in C (compiled in as a DLL) to use it in Free Pascal and Delphi applications. While the code (below) works with Delphi (7), I get an Access Violation using FPC (1.98). Perhaps I am missing something.
Any hint? Thanks!

program fpc_dll;
{$H+}{$MODE OBJFPC}
uses
  sysutils;
type
 myarray = array of double;
// if I declare "myarray = array[0..9] of double;" the code runs as
expected.
function get_array(a, b: myarray; size: integer; value: double):
integer; cdecl; external 'mylib.dll';
// extern "C"  __cdecl __declspec(dllexport) int get_array(double a[],
double b[], int size, double val);
var
  x, y, z: double;
  t: integer;
  i, kk, qq: integer;
  mat1, mat2: pmyarray;
  st: string;
begin
  kk := 10;
  SetLength(mat1,kk);
  SetLength(mat2,kk);
  for i := 0 to kk-1 do
  begin
    mat1[i] := i+1;
    st := FloatToStr(mat1[i]);
    writeln(st);
  end;
  qq := get_array(mat1,mat2,kk,3.14);
  st := IntToStr(qq);
  writeln(st);
  for i := 0 to kk-1 do
  begin
    st := FloatToStr(mat1[i]);
    writeln(st);
  end;
  st := '------------------------------';
  writeln(st);
  for i := 0 to kk-1 do
  begin
    st := FloatToStr(mat2[i]);
    writeln(st);
  end;
  SetLength(mat1,0);
  SetLength(mat2,0);
end.

jk

// Jilani KHALDI *** K-Book Interactive Scientific & Technical Pubblications http://jkhaldi.oltrelinux.com

Reply by email

Peter Vreman
to Jilani Khaldi
you wrote 22 lines
Hi All, I have written a little library in C (compiled in as a DLL) to use it in Free Pascal and Delphi applications. While the code (below) works with Delphi (7), I get an Access Violation using FPC (1.98). Perhaps I am missing something. Any hint? Thanks! program fpc_dll; {$H+}{$MODE OBJFPC} uses sysutils; type myarray = array of double; // if I declare "myarray = array[0..9] of double;" the code runs as expected. function get_array(a, b: myarray; size: integer; value: double): integer; cdecl; external 'mylib.dll'; // extern "C" __cdecl __declspec(dllexport) int get_array(double a[], double b[], int size, double val);

Dynamic arrays are an internal type of pascal. C doesn't know dynamic arrays. That it works with delphi is pure luck.

Peter

Reply by email

Jonas Maebe
to Michael Van Canneyt: Totally wrong. They are a reference counted type. See e.g. page 5-19 of the Delphi 7…
Michael Van Canneyt wrote 6 lines
>> Dynamic arrays are an internal type of pascal. C doesn't know dynamic >> arrays. That it works with delphi is pure luck. > > Sorry, but pascal dynamic arrays are just pointers. Totally wrong. They are a reference counted type.

Yes, but aren't they "just" C-style arrays (i.e. pointers) in the same way that ansistrings are "just" pchars?

Jonas

Reply by email

Michael Van Canneyt
to Jilani Khaldi: Sorry, but pascal dynamic arrays are just pointers. jk // Jilani KHALDI *** K-Book…
Jilani Khaldi wrote 5 lines
> Dynamic arrays are an internal type of pascal. C doesn't know dynamic > arrays. That it works with delphi is pure luck. Sorry, but pascal dynamic arrays are just pointers.

Totally wrong. They are a reference counted type.

See e.g. page 5-19 of the Delphi 7 Language guide.

(Chapter 5 'Data types', section structured types - arrays - dynamic arrays)

Michael.

Reply by email

Peter Vreman
to Jonas Maebe: Yes, but aren't they "just" C-style arrays (i.e. pointers) in the same way that…
you wrote 11 lines
On 05 May 2005, at 13:06, Michael Van Canneyt wrote: >>>Dynamic arrays are an internal type of pascal. C doesn't know dynamic >>>arrays. That it works with delphi is pure luck. >> >>Sorry, but pascal dynamic arrays are just pointers. > >Totally wrong. They are a reference counted type. Yes, but aren't they "just" C-style arrays (i.e. pointers) in the same way that ansistrings are "just" pchars?

It is also calling convention related how C expects arrays to be pushed. From the example:

// extern "C"  __cdecl __declspec(dllexport) int get_array(double a[],
double b[], int size, double val);

Here the array is expected on the stack on some i386 targets. Using 'double *a' is less ambigious, because it then always needs to be a pointer.

Peter

Reply by email

Michalis Kamburelis
to Jilani Khaldi: I have written a little library in C (compiled in as a DLL) to use it in Free Pascal and …
Jilani Khaldi wrote 21 lines
Hi All, I have written a little library in C (compiled in as a DLL) to use it in Free Pascal and Delphi applications. While the code (below) works with Delphi (7), I get an Access Violation using FPC (1.98). Perhaps I am missing something. Any hint? Thanks! program fpc_dll; {$H+}{$MODE OBJFPC} uses sysutils; type myarray = array of double; // if I declare "myarray = array[0..9] of double;" the code runs as expected. function get_array(a, b: myarray; size: integer; value: double): integer; cdecl; external 'mylib.dll';

Change this to

type
   PDouble = ^Double;
function get_array(a, b: PDouble; size: integer; value: double):
integer; cdecl; external 'mylib.dll';

In C "double a[]" is the same as "double *a" which is the same as "A: PDouble" in Pascal. "array of Double" is not the same thing as "PDouble", as fpc developers already explained.

// extern "C" __cdecl __declspec(dllexport) int get_array(double a[], double b[], int size, double val);

...

qq := get_array(mat1,mat2,kk,3.14);

Change this to

qq := get_array(PDouble(mat1),PDouble(mat2),kk,3.14);

Michalis

Reply by email