Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / fpc-pascal / 2003-05

Issues regarding available memory

11 messages · 3 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

Alan Mead

I need to handle arbitrarily-large datafiles. So I need to detect the amount of memory want my code to run on
Linux and Windows.

Given that the heap grows, how can I determine the maximum available memory (in a platform-independent way)?

I thought I would GetMem() until it reports nil but that does not apparently work... Or I am confused about the cause of error 216. (Or I made a dumb mistake. :) When I compile my code (below) using FPC 1.0.4 and run it on my Linux machine, I see:

ian]$ ./dummy
Total heap available is 262128 bytes
Largest block available is 262128 bytes
size=44000000 (storage=352000000)
getmem() was successful
..[lots of dots]..OK
size=48400000 (storage=387200000)
Runtime error 216 at 0x0804EB61
0x0804EB61
0x0804E730
0x08052263
0x080480B0
0x00000000

Also, I have a performance question: there is a noticable difference in the time between printing these periods. Am I forcing the Linux machine to use virtual memory? If so, is there a way to know when I've done that (a way that will work under Windows and Linux, perferrably).

Thanks!

-Alan Mead

-------------------------------------------
ian]$ cat dummy.pas
program dummy;

type
  longarray = array[0..0] of real;
var p:pointer;
  l : ^longarray;
  size, storage : longint;
  i,j:longint;
  done:boolean;
begin
  writeln('Total heap available is ',MemAvail,' bytes');
  writeln('Largest block available is ',MaxAvail,' bytes');
  done := false;
  size := 40000000;
  repeat
    size := round(size * 1.1);
    storage := size * sizeof(real);
    writeln('size=',size,' (storage=',storage,')');
    getmem(l,storage);
    if (l=nil) then
      begin
        done := true;
      end
    else
      begin
        writeln('getmem() was successful');
        j := 0;
        for i := 1 to size do
          begin
            inc(j);
            if (j MOD 100000 = 0) then
              begin
                write('.');
                j := 0;
              end;
            l^[i] := 1.0;
          end;
        writeln('OK');
        freemem(l,storage);
      end;
  until (done);
end.

=====
A Congressman was once asked about his attitude toward whiskey. "If you mean the demon drink that poisons the mind, pollutes the body, desecrates family life, and inflames sinners, then I'm against it. "

"But if you mean the elixir of Christmas cheer, the shield against winter chill, the taxable potion that puts needed funds into public coffers to comfort little crippled children, then I'm for it."

"This is my position, and I will not compromise."

__________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com

Alan Mead
to Alan Mead
--- Alan Mead wrote 22 lines
Given that the heap grows, how can I determine the maximum available memory (in a platform-independent way)? I thought I would GetMem() until it reports nil but that does not apparently work... Or I am confused about the cause of error 216. (Or I made a dumb mistake. :) When I compile my code (below) using FPC 1.0.4 and run it on my Linux machine, I see: ian]$ ./dummy Total heap available is 262128 bytes Largest block available is 262128 bytes size=44000000 (storage=352000000) getmem() was successful ..[lots of dots]..OK size=48400000 (storage=387200000) Runtime error 216 at 0x0804EB61 0x0804EB61 0x0804E730 0x08052263 0x080480B0 0x00000000

I noticed that 1.0.4 is out of date so I tried this program on my Windows 2000 computer which has 1.0.6 and I get very similar results except the error is 203 rather than 216.

I usually assume that these things are my error but it seems like this might possibly be a problem with the compiler?

-Alan

=====
A Congressman was once asked about his attitude toward whiskey. "If you mean the demon drink that poisons the mind, pollutes the body, desecrates family life, and inflames sinners, then I'm against it. "

"But if you mean the elixir of Christmas cheer, the shield against winter chill, the taxable potion that puts needed funds into public coffers to comfort little crippled children, then I'm for it."

"This is my position, and I will not compromise."

__________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com

Jonas Maebe
to Alan Mead
Alan Mead wrote 6 lines
I noticed that 1.0.4 is out of date so I tried this program on my Windows 2000 computer which has 1.0.6 and I get very similar results except the error is 203 rather than 216. I usually assume that these things are my error but it seems like this might possibly be a problem with the compiler?

The heap manager is written to give a run time error 203 when you ask for memory and it can't be allocated. You can catch this using exceptions.

Jonas

Alan Mead
to Jonas Maebe
--- Jonas Maebe wrote
The heap manager is written to give a run time error 203 when you ask for memory and it can't be allocated. You can catch this using exceptions.

Ah. I was looking for a nil pointer from GetMem. But you're saying that I never get it because the heap manager never refuses GetMem, it just grows the heap or, if it cannot grow the heap, throws an error?

So I only have an abstract understanding of exceptions. I wrap the getmem in a try..except structure? Do I need to (can I) test for specific error conditions? I seem to get RTE 216 on Linux and 203 on Windows. Or, I don't see a memory exception class on page 102 of the refenece guide... Is there a more complete list elsewhere?

Thanks very much for your answer!

BTW, is getmem depracated?

-Alan

=====
A Congressman was once asked about his attitude toward whiskey. "If you mean the demon drink that poisons the mind, pollutes the body, desecrates family life, and inflames sinners, then I'm against it. "

"But if you mean the elixir of Christmas cheer, the shield against winter chill, the taxable potion that puts needed funds into public coffers to comfort little crippled children, then I'm for it."

"This is my position, and I will not compromise."

__________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com

Jonas Maebe
to Alan Mead
Alan Mead wrote
Ah. I was looking for a nil pointer from GetMem. But you're saying that I never get it because the heap manager never refuses GetMem, it just grows the heap or, if it cannot grow the heap, throws an error?

By default, yes. You can set "ReturnNilIfGrowHeapFails" to true (it's a typed constant defined in the interface of the system unit which is false by default) to get the behaviour you want though (and then you don't need all the exception stuff below, I had forgotten about this possibility).

So I only have an abstract understanding of exceptions. I wrap the getmem in a try..except structure?

Yes.

Do I need to (can I) test for specific error conditions?

Yes, you can look at the type of the exception.

I seem to get RTE 216 on Linux and 203 on Windows.

That's because of the version difference. RTE 216 = general protection fault/segmentation fault. The heap manager apparently crashed in 1.0.4 if it couldn't grow the heap anymore.

Or, I don't see a memory exception class on page 102 of the refenece guide... Is there a more complete list elsewhere?

I don't know, Michael?

BTW, is getmem depracated?

No.

Jonas

Peter Vreman
to Alan Mead: I need to handle arbitrarily-large datafiles. So I need to detect the amount of memory …
Given that the heap grows, how can I determine the maximum available memory (in a platform-independent way)?

There is no way. It is up to the operator system. FOr example under linux you can't determine it correctly as an other program can fill the memory also so there is less left for your program. The only thing you can retrieve is the limit set by getrlimit, but this syscall is not yet implemented

Peter

Peter Vreman
to Alan Mead: Ah. I was looking for a nil pointer from GetMem. But you're saying that I never get it…
you wrote 10 lines
--- Jonas Maebe wrote: > The heap manager is written to give a run time error 203 when you > ask > for memory and it can't be allocated. You can catch this using > exceptions. Ah. I was looking for a nil pointer from GetMem. But you're saying that I never get it because the heap manager never refuses GetMem, it just grows the heap or, if it cannot grow the heap, throws an error?
Set the global variable: ReturnNilIfGrowHeapFail:=true and it'll return nil
as you wanted

Peter

Alan Mead
to Jonas Maebe: By default, yes. You can set "ReturnNilIfGrowHeapFails" to true (it's a typed constant…
Show quoted text 17 lines
On vrijdag, mei 16, 2003, at 23:30 Europe/Brussels, Alan Mead wrote: > Ah. I was looking for a nil pointer from GetMem. But you're saying > that I never get it because the heap manager never refuses GetMem, it > just grows the heap or, if it cannot grow the heap, throws an error? By default, yes. You can set "ReturnNilIfGrowHeapFails" to true (it's a typed constant defined in the interface of the system unit which is false by default) to get the behaviour you want though (and then you

Thanks to the people who helped but this is not working, quite possibly to my lack of knowledge.

This global variable should gte be available (i.e., I don't have to explicitly use the system unit), right? I tried adding both spellings suggested and also upgrading my FPC but this won't compile (see below). Are these globals documented somewhere (I've printed all four PDF manuals and I don't recall seeing these) or do I need to unpack the system unit source to browse them?

-Alan

ian]$ ppc386 dummy.pas
Free Pascal Compiler version 1.0.4 [2000/12/18] for i386 Copyright (c) 1993-2000 by Florian Klaempfl
Target OS: Linux for i386
Compiling dummy.pas
dummy.pas(19,3) Error: Identifier not found RETURNNILIFGROWHEAPFAIL dummy.pas(52) Fatal: There were 1 errors compiling module, stopping ian]$ vi dummy.pas
ian]$ ppc386 dummy.pas
Free Pascal Compiler version 1.0.4 [2000/12/18] for i386 Copyright (c) 1993-2000 by Florian Klaempfl
Target OS: Linux for i386
Compiling dummy.pas
dummy.pas(19,3) Error: Identifier not found RETURNNILIFHEAPGROWFAIL dummy.pas(52) Fatal: There were 1 errors compiling module, stopping ian]$ ppc386 dummy.pas
Free Pascal Compiler version 1.0.6 [2002/05/23] for i386 Copyright (c) 1993-2002 by Florian Klaempfl
Target OS: Linux for i386
Compiling dummy.pas
dummy.pas(19,3) Error: Identifier not found RETURNNILIFHEAPGROWFAIL dummy.pas(52) Fatal: There were 1 errors compiling module, stopping ian]$ vi dummy.pas
ian]$ ppc386 dummy.pas
Free Pascal Compiler version 1.0.6 [2002/05/23] for i386 Copyright (c) 1993-2002 by Florian Klaempfl
Target OS: Linux for i386
Compiling dummy.pas
dummy.pas(19,3) Error: Identifier not found RETURNNILIFGROWHEAPFAIL dummy.pas(52) Fatal: There were 1 errors compiling module, stopping

-Alan

=====
A Congressman was once asked about his attitude toward whiskey. "If you mean the demon drink that poisons the mind, pollutes the body, desecrates family life, and inflames sinners, then I'm against it. "

"But if you mean the elixir of Christmas cheer, the shield against winter chill, the taxable potion that puts needed funds into public coffers to comfort little crippled children, then I'm for it."

"This is my position, and I will not compromise."

__________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com

Jonas Maebe
to Alan Mead
Alan Mead wrote
This global variable should gte be available (i.e., I don't have to explicitly use the system unit), right?

Indeed, the system unit is automatically included in every program and unit you compile.

Show quoted text 5 lines
I tried adding both spellings suggested and also upgrading my FPC but this won't compile (see below). Are these globals documented somewhere (I've printed all four PDF manuals and I don't recall seeing these) or do I need to unpack the system unit source to browse them?

Try upgrading to 1.0.6.

Jonas

Alan Mead
to Alan Mead: Thanks to the people who helped but this is not working, quite possibly to my lack of…
--- Alan Mead wrote
Are these globals documented somewhere (I've printed all four PDF manuals and I don't recall seeing these) or do I need to unpack the system unit source to browse them?

My mistake. I see that ReturnNilIfGrowHeapFails is described clearly on Page 77 of the Programmer's manual. And I see that I've left off the ending 's'. Now my program compiles.

However, it still doesn't have the behavior I expect (see below). BTW, note also that even when using the same 1.0.6 version of FPC, I get RTE 216 on Linux and 203 on Windows 2000.

-Alan

ian]$ ./dummy
Total heap available is 262128 bytes
Largest block available is 262128 bytes
size=44000000 (storage=352000000)
getmem() was successful
..[lots of periods] ..OK
size=48400000 (storage=387200000)
Runtime error 216 at 0x0804F2B9
0x0804F2B9
0x0804EE80
0x08052BDA
0x080480B0

 ian]$ cat dummy.pas
program dummy;
type
  matrix_element = array[1..1] of byte;
  big_matrix = array[1..1000000,1..610] of matrix_element;
  longarray = array[0..0] of real;

{var
a : big_matrix;}

var p:pointer;
  l : ^longarray;
  size, storage : longint;
  i,j:longint;
  done:boolean;
begin
  ReturnNilIfGrowHeapFails:=true;
  writeln('Total heap available is ',MemAvail,' bytes');
  writeln('Largest block available is ',MaxAvail,' bytes');
  done := false;
  size := 40000000;
  repeat
    size := round(size * 1.1);
    storage := size * sizeof(real);
    writeln('size=',size,' (storage=',storage,')');
    getmem(l,storage);
    if (l=nil) then
      begin
        done := true;
      end
    else
      begin
        writeln('getmem() was successful');
        j := 0;
        for i := 1 to size do
          begin
            inc(j);
            if (j MOD 100000 = 0) then
              begin
                write('.');
                j := 0;
              end;
            l^[i] := 1.0;
          end;
        writeln('OK');
        freemem(l,storage);
      end;
  until (done);
end.

=====
A Congressman was once asked about his attitude toward whiskey. "If you mean the demon drink that poisons the mind, pollutes the body, desecrates family life, and inflames sinners, then I'm against it. "

"But if you mean the elixir of Christmas cheer, the shield against winter chill, the taxable potion that puts needed funds into public coffers to comfort little crippled children, then I'm for it."

"This is my position, and I will not compromise."

__________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com

Alan Mead
to Jonas Maebe: Indeed, the system unit is automatically included in every program and unit you compile.…
--- Jonas Maebe wrote
[Alan's having trouble] Try upgrading to 1.0.6.

I just did and I still get the same behavior.

-Alan

=====
A Congressman was once asked about his attitude toward whiskey. "If you mean the demon drink that poisons the mind, pollutes the body, desecrates family life, and inflames sinners, then I'm against it. "

"But if you mean the elixir of Christmas cheer, the shield against winter chill, the taxable potion that puts needed funds into public coffers to comfort little crippled children, then I'm for it."

"This is my position, and I will not compromise."

__________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com