Hi everybody!

Why in fact were MaxAvail,MemAvail,HeapSize removed from the RTL? The explanation in install/doc/whatsnew.txt is puzzling me a bit:

- Removed MaxAvail, MemAvail, HeapSize due to their unreliability (bogus/misleading return values) in multitasking environment with swapping

Is the incompatibility of the source code preferred over a misleading result? Such a major change should have a bit better explanation. I just checked the FPC introduction that there is still the claim for "excellent compatibility with TP 7.0" which seems I.M.H.O. now to be broken.

I understand that this may lead to strange behaviour of a program but wouldn't it have been better to include a remark in the documentation than to remove the function(s)? So it would be programmer's choice to leave it or to change it to something appropriate for the environment of the program.

So, what are now the options? What are the recommended replacements for these functions? What's the best way to make working programs to compile again?

By the way, there are still references in the RTL, the examples and the docs to these now missing functions. Some seem to have been missed on cleaning ;-).

Have a nice day,

In this case, yes. Better fail totally than under rare circumstance and hard to reproduce because the crashes are caused by a lack of memory causing strange behaviour of memavail in combination with new/getmem.

Keep in mind that the code
if maxavail>=10000 then
  getmem(p,10000);
is completly useless in any multitasking environment.

Such a major change should have a bit better explanation. I just checked the FPC introduction that there is still the claim for "excellent compatibility with TP 7.0" which seems I.M.H.O. now to be broken.

How should work MaxAvail/MemAvail in a multitasking environment to be TP compatible?

In general speaking that's not true. What would be the replacement for this? When you skip the check and allocate always then the program will crash anyway when there is not enough memory. What would be the downside of the bogus answer? If there was not enough memory when doing the maxavail call but would have been on the getmem call then there would be one "memory low warning" (or whatever an "else" condition action above would be) more than really needed but how often would this "flaw" happen in comparison to the crash without the check?

The only code I would say is really bogus with this is something like getmem(maxavail) which would quite often fail without obvious reason but it is a bad code for a multitasking environment anyway.

It depends a bit on the assumptions you make about the environment. If you assume that you have always any amount of memory (as would it be if you skip the check) then it should return its max. possible value (defining a constant will do).

I don't know exactly what value MaxAvail returned before on a Linux system but returning in general the maximal allocatable memory by a single allocation call at the moment of the MaxAvail call would be best. The only problems to be expected would be on a system with a very low memory where most programs would crash/abort/fail anyway or when trying to allocate big chunks of memory. Big chunks for BP7 are 64KB. For todays computers if there are no 64KB more memory left then this IS a very low memory condition so this is not an issue for compatibility.

For allocating bigger chunks in a multitasking environment special precautions must be taken anyway. I'm not aware of any standard function which can guarantee this. I still assume that the failure of GetMem causes a Heap-Runtime-Error like in BP7 so checking if the resulting pointer is NIL is worthless ;-).

I don't know if the "best" operation described above is easily possible as it may differ from the previous operation. In this case leaving the original functions and putting a big fat warning in the docs about it would be better than removing them. Printing warnings (like gcc does for deprecated things) on compiling for the affected platforms is also a good option but I don't know how easy this would be.

Just my 5 cent.

How do you allocate memory in a fashion that is suitable for all environments?

Try to allocate and check if it worked or let the heap manager throw an exeception if it can't allocate the memory.

When you skip the check and allocate always then the program will crash anyway when there is not enough memory.

With the check the chances are only less that the program crashes but it can still do so. So your program is still buggy.

See above.

Konstantin Münning wrote

When you skip the check and allocate always then the program will crash anyway when there is not enough memory.

No. Either you catch exceptions resulting from a lack of memory and recover, and then you have the same checking as before, except that the check happens atomically (by the OS: you ask for more memory and if there is no more memory, you get an exception).

Another possibility to set the global system unit variable ReturnNilIfGrowHeapFails to true, and then you can check after each allocation whether the resulting pointer is nil (if so, not enough memory was available) or not.

Jonas

It seems you are missing a "small" point. Compatibility. With BP7 there are no exceptions - if you try to allocate more memory than availible you get a runtime error. This runtime error can't be switched off like IO-errors. OK, there are the exit procedures, but there is no recover.

The structure of a program is completely different when designing it to work with exceptions and recovering from these. At least for the affected parts. All the "old" code needs to be redesigned in a way it would work ONLY with the new compiler. OK, there are the IFDEF's, but for most cases it isn't worth the effort.

Last but not least - exception handlers can still be added to deal with the tiny chance that the problem occurs - no need to change the "original" code.

Another possibility to set the global system unit variable ReturnNilIfGrowHeapFails to true, and then you can check after each allocation whether the resulting pointer is nil (if so, not enough memory was available) or not.

That's easier than exceptions but unfortunately also an incompatible way.

I don't think MaxAvail is so bad that everyone must be enforced to stop using it and is not left to choose by himself. I think most programmers now would either omit the check expecting the program to fail on low memory anyway or they would write a replacement function to avoid work and by this keeping the flaw you are so afraid of. Some will avoid switching to the new compiler and only a few with impotrtant code which have not yet done it better would consider doing a major rewrite.

I'm sorry, but I still prefer keeping the compatibility and adding a warning (with hints) so anyone can do it better. Of course you are right that this might be a bug source under the specific circumstances but please don't try to enforce it the community. Otherwise great compiler :-).

Konstantin.

There was already no compatibility, because maxavail and memavail simply worked with the currently allocated memory to the program. So if you had bad luck (currently allocated memory was almost used up, even though the OS possibly had 2GB free) you would already get the message that no more memory was available.

The structure of a program is completely different when designing it to work with exceptions and recovering from these. At least for the affected parts.

It is simply impossible to write a program running under a multitasking system which has virtual memory 100% exactly the same as one running under a single tasking no virtual memory "OS". There are other things you have to change as well: assembler code, inline machine code statements, mem[] statement (at least if you're not working under Dos), ...

I don't think MaxAvail is so bad that everyone must be enforced to stop using it and is not left to choose by himself.

What would you suggest to return under an OS like Windows, Mac OS X or Linux? The current "free" memory of the OS? Free memory + buffer cache - minimal buffer cache size enforced by the OS? The previous + available swap space - "wired" (i.e., not swappable to disk) memory? Just a constant? Something different possibly?

And in all cases, the program will still crash occasionally because it runs out of memory in certain cases, because the check and the allocation do not happen atomically. You can't put something into the standard Run Time Library which works "most of the time, except under heavy load". Imagine reading the docs and encountering that comment, you'd immediately start wondering how many more procedures there are which work "most of the time".

Jonas

You are right. I assumed something like that when I looked at the values of maxavail but hadn't investigated further and that's how the possible flaw I mentioned in one of my first mails could look like.

The only assembler code I had to change was the one I had binary patched for 32-bit operation. This and mem[] statements must be adjusted anyway when trying to run on different architectures. So that's not the trouble. I am still very happy with my units which allow me to write my programs to run in 16 bit DOS real mode, DOS protected mode and Linux without changing the main code and this including the GUI (VGA - X11). Only the device drivers must be rewritten, of course ;-).

I have written in my previous mail(s) that I think the best return value would be the biggest value with which a getmem would be successful if memory conditions haven't changed inbetween. As this might be difficult to implement on all supported OSes something between this and the "old" one should do.

As I said before - adding an otherwise imperative error handling like exceptions is still possible in addition to the old code.

By the way, I have never had a noticeable bug in a program (and some of them are controlling machines running 24/7) because of the discussed possibility. OK, they are not doing extensive memory allocation/deallocation but also the possible danger is confined to non-critical areas - a shortened item list for example would hurt nobody :-).

Most programs work most of the time except under heavy load. That's not a good argument. Most small (C) programs don't do any memory availability at most of the command line tools on any OS ;-). Of course I would like to have any code behave perfect under any condition but it has always been a tradeoff between effort (=expense) and impact. But when there is no memory left and you need some you can't behave perfect. Please let it be the programmers choice what to do and how to handle this. A compiler should be providing the means, not taking them away. Anybody writing a program knows that there may be flaws. A warning (with an advice how to make it better) should still be the best for this.

By the way, I've found the following code in compiler/cclasses.pas when I searched for the whereabouts of maxavail/memavail and there were some more like this in the sources:

{$ifdef HASGETHEAPSTATUS}
        status:=GetFPCHeapStatus;
        startmem:=status.CurrHeapUsed;
{$else HASGETHEAPSTATUS}
        startmem:=memavail;
{$endif HASGETHEAPSTATUS}

I'm not sure if I understand it right as GetHeapStatus is not documented well (I've found no reference for the meaning of the fields of the returned record) but it seems that someone repaced here memavail with another code which does about the same and has therefore the same flaw.

I would hapily contribute the code for a better MemAvail/MaxAvail function if that's the problem. I would like to be able to use the latest compiler which now I definitely can't as I must have the BP7 compatibility which is very good in 1.0.10.

Pascal is not C ;)

It has been moved to heap status which contains more info which also reliable on multitasking systems.

I would hapily contribute the code for a better MemAvail/MaxAvail function if that's the problem. I would like to be able to use the latest compiler which now I definitely can't

See below.

as I must have the BP7 compatibility which is very good in 1.0.10.

What about Mark and Release when talking about heap management ;)?

A compromise would be a tpascal unit which is loaded only in -So mode.

But when there is no memory left and you need some you can't behave perfect. Please let it be the programmers choice what to do and how to handle this.

It is. Exceptions, ReturnNil, ...

A compiler should be providing the means, not taking them away.

Environment changes sometimes you know! Thus, so does the RTL. There are new means now.

Anybody writing a program knows that there may be flaws. A warning (with an advice how to make it better) should still be the best for this.

Warning: compilation failure; make it better: here on mailing list.

I would hapily contribute the code for a better MemAvail/MaxAvail function if that's the problem. I would like to be able to use the latest compiler which now I definitely can't as I must have the BP7 compatibility which is very good in 1.0.10.

Well, define these functions in your own compatibility unit then ?

Micha

Yes, I've already found out that there is the GetHeapStatus function but the use here is also not atomic so there is no difference regarding the discussed bug. If it is reliable then here's a good information source for the "new" MaxAvail :-).

By the way, what's the information provided with GetHeapStatus? I've found no description in the docs except the record member names. Or where to look for?

I never had a use for that and as nobody reported them missing...

A compromise would be a tpascal unit which is loaded only in -So mode.

That's I.M.H.O. a good solution. For the affected units I use -So anyway. Can I do this "myself" by some compiler option or must it be included in the compiler source? If you tell me how this unit should be called I would happily provide it.

Konstantin

The only way to know that is to perform getmem's with every increasing values until one fails.

Most programs work most of the time except under heavy load. That's not a good argument.

I beg you pardon? I think the argument that most programs do not work well under heavy load is an extremely bad argument to encourage even more buggy programming.

Please let it be the programmers choice what to do and how to handle this.

Only using tools that make sense, everything else would be snake oil (and in reality nothing would be handled).

A compiler should be providing the means, not taking them away.

It should not provide known defunct means.

Anybody writing a program knows that there may be flaws.

No, if you use RTL routines, you expect them to work.

A warning (with an advice how to make it better) should still be the best for this.

People don't read the manual.

The above code is not used to check whether there's still enough free memory left, it's only used to calculate how much memory is used by the compiler.

I would hapily contribute the code for a better MemAvail/MaxAvail function if that's the problem. I would like to be able to use the latest compiler which now I definitely can't as I must have the BP7 compatibility which is very good in 1.0.10.

You can make a unit which contains a dummy memavail and maxavail which always return a value of 1GB or so, then your program will also always work except under heavy load (if that is really what you want). There's no use in investing lots of time in making very complex routines to query all sorts of information from the OS and then have it fail in exactly the same circumstances as when you return a plain constant.

Jonas

Hi Micha!

Thanks for the amusing comments :-). Let me add some of mine.

How much can you limit the choice and still pretend that you can choose?

New means are good. I'm happy with this. But the removal of some "basics" which is not really enforced by the environment is what makes me unhappy.

Hint: a failure is not a warning.

The -So autoload of such a unit as discussed with Florian would do the job so you don't have to change the good'ol code ;-).

Konstantin

P.S.: Please remember - the PC you are using is still around not because it's good but because it's compatible - you can still run the first programs although most of the old games are unplayable on todays hardware ;-). Quite everybody seems to be happy with this "bug".

Konstantin Münning wrote

I would hapily contribute the code for a better MemAvail/MaxAvail function if that's the problem. I would like to be able to use the latest compiler which now I definitely can't as I must have the BP7 compatibility which is very good in 1.0.10.

IMO we have two issues here:

- compiler compatibility
- platform portability

The BP7 handling is very platform specific, and it doesn't make much sense to implement the legacy DOS/Go32 API for other (multitasking...) platforms. It would just *prevent* easy porting of such code, when the functions returned some inadequate value on other platforms, at runtime, making the program either fail immediately or crash later.

This means that such code *must* be changed (conditionalized...) before it can be compiled for other platforms. Such a change can be enforced when the legacy API only is implemented for DOS compatible target platforms, inavailable on other platforms. But this doesn't mean that the API is completely removed, it should exist and behave as before, on the platforms to which it applies.

With regards to compiler compatibility I see no need for special handling of this API, it is "standard" Pascal with no special syntactical/semantical aberrations?

DoDi

I was not implying that a getmem(maxavail) must work under any circumstances on a multitasking OS. But as the OS knows at a given point what the maximum availible memory for a single allocation call is, so could a program running on that OS.

Please don't understand me wrong - I don't encourage the use of these functions. I'm only speaking about compatibility to old code which was there and is now missing.

But back to the argument - as on a multitasking system ANY memory allocation could fail at ANY time, if this was so severe, none of these OSes should be accepted by the community as they are buggy. But as you see everybody seems to live happily with this as the chances these failures to happen are acceptable low. So allowing old programs initially designed for other environments to "live" with the same limits, that's OK I think.

And as to buggy programming - what can you really do when a getmem fails? Even doing fancy tricks with exceptions and such will not help. If you haven't added some "own" memory management (like garbage collection) to free some "unneeded" memory in such a case there are only two things you can do - wait until there is the requested memory avalable and allowing by this your program to hang indefinitely or fail in some more or less informative way. Show me one program which can run without the required memory and does not fail in any way. Are you still considering it buggy programming?

It's not the means but the way you are using them. Would you then remowe the Reset/Rewrite functions as well? In many (old) programs they are used to check for presence of file(s) to prevent overwriting by first trying to open them and if this fails then the file can be written... But as in a multitasking environment the file can appear just between the Reset and the Rewrite - well, bad bug. There are the means to do this right by the proper OS functions so maybe this should be enforced as well!?

(I hope I haven't brought you to another "imrpovement" idea ;-))

(see above)

Well, then they maybe will not know about the existence of the function :-). Still a compiler warning for deprecated functions would be nice.

I just picked one.

I mentioned the plain constant alernative in one of my first mails as it depends on the assumptions you make. Its about compilability of old code and this would also do but a better function would reduce the unavoidable failures to a minimum.

The -So loading of a compatibility unit is probably the best solution here.

Konstantin.

If it can't be implemented sanely, it is not a basic.

Any of these functions (including *status *avail) basically have only a few uses:
1) test and take decisions on memory use/availability
2) display user visible memory information
3) detect memory leaks.
4) internal use in e.g. unit tests to get a feel for the magnitude

(1) is seriously flawed, and should be avoided. Period. Not only the multitasking problem, but also trashing because the program tries to use more than the available physical memory. Either due to other programs or because the total virtual size is reported by the OS. (2) is dangerous since it confuses the user if available memory suddenly increases while the program only requested another block from the system.
(3) is a very primitive and somewhat incorrect (what if an automated type happens to allocate a block of the same size somewhere as you deallocate between memavails, it can really confuse. Better use heaptrc)
(4) Is the only somewhat valid reason, but requires expert devel interpretation to correct for problems mentioned in (1) and (2).

The fact is that the mem* routines are abundant in use for all these 4 types, and we got really a lot of questions and bugreports for these fatally flawed uses.

The Delphi eq Getheapstatus is not abused like that (since Delphi is windows oriented and protected mode), so it was left in.

>>I would hapily contribute the code for a better MemAvail/MaxAvail >>function if that's the problem.

This was all considered, and the end conclusion was that a satisfying solution that lifts the problems 1..4 above is impossible. This, and to avoid bugs creeping in while porting is one of the main reasons.

If this really bothers you a lot, add a memavail to your compability unit and reroute it to getheapstatus or getfpcheapstatus (which are unfortunately either wrongly, or not documented in 2.0.0)

> Well, define these functions in your own compatibility unit then ? The -So autoload of such a unit as discussed with Florian would do the job so you don't have to change the good'ol code ;-).

IMHO these functions should not be brought back to the main distribution. There were good reasons to remove them, and these still hold. In theory they could be kept for go32v2 only use, but I don't like that either.

The few people that have TP compatible code and don't want to fix them despite the warnings, should implement their own versions.

P.S.: Please remember - the PC you are using is still around not because it's good but because it's compatible

PCS are not that compatible. I can not run my old games, I can not connect my MFM disc, and a lot of old stuff doesn't work on new dos versions.

Only the usable subset has survived. FPC actually retains more than the usable subset. (e.g. it retains dos functions that are superseeded by better delphi functions). However the limit is at functions whose use is dangerous, has the possibility of introducing bugs, and

you can still run the first programs although most of the old games are unplayable on todays hardware ;-). Quite everybody seems to be happy with this "bug".

There is a lot that doesn't run. Actually, MSDOS 5.x fails to install on most of todays computers. Compability is a good thing, but the 0.1% of it that is relatively unused and even dangerous should be removed.

I can't lowlevel my IDE disc the same way from the BIOS as I did my MFM discs. And that is a good thing, since that would ruin it, or it would fail for no visible reason.

Yes, by trying to allocate that memory. There is no other way that I know of.

But back to the argument - as on a multitasking system ANY memory allocation could fail at ANY time, if this was so severe, none of these OSes should be accepted by the community as they are buggy.

No, because they do not crash if it fails. They simply return an error which you can check. And this error is consistent. There are no cases where you do not get an error, but that the allocation failed nevertheless (or vice versa).

But as you see everybody seems to live happily with this as the chances these failures to happen are acceptable low. So allowing old programs initially designed for other environments to "live" with the same limits, that's OK I think.

These old programs "think" they are checking for errors, but in practice they are not.

And as to buggy programming - what can you really do when a getmem fails?

That completely depends on the program. A game will probably terminate with an error, our compiler will terminate with a run time error and a server app will probable reject a new connection or terminate the current session.

Reset and rewrite also have other functionality. The compiler nor the RTL can enforce proper usage of file locking semantics, just like it can't force you not to have any dataraces.

The compiler never checks whether enough memory is left. It will simply terminate with a run time error if no memory is left anymore.

The HeapStatus stuff can also not be used to query how much "free OS memory" (whatever that may be, see my previous list of possibilities) is left. It can only tell you how much memory has been allocated to the program, how much of that memory is actually in use etc.

There is no "better" function afaik (other functions will simply require more work to implement, but the end result will be pretty much the same).

Jonas

{$ifdef fpc}const memavail=high(ptrint);maxavail=high(ptrint);{$endif}

{$ifdef fpc}const memavail=high(ptrint);maxavail=high(ptrint);{$endif}

That's a nice one :-)

Peter Vreman wrote

{$ifdef fpc}const memavail=high(ptrint);maxavail=high(ptrint);{$endif}

Then we could maybe also add an optimization that checks whether you are comparing against the upper or the lower bound of the current type and completely optimize away the checks :)

Jonas

Op Mon, 23 May 2005, schreef Jonas Maebe:

Define them as constants and that's exactly what'll happen.

Daniël

The idea that for "dos" it is the same is not true, since this is extender dependant. Some extenders might return virtual size allocated by the host OS (DPMI host, or include their own swap.

So this is not between dos and the rest, but between real and 32-bit protected mode (and I leave in the middle what 16-bits protected mode does, since that is irrelevant for FPC purposes)

And FPC dos programs _are_ protected mode, not real.

In a message dated 5/23/2005 5:00:24 A.M. Central Daylight Time, Peter Vreman writes:

{$ifdef  fpc}const  memavail=high(ptrint);maxavail=high(ptrint);{$endif}

Peter, you are, indeed, a brilliant man!!! There is true beauty in simplicity.

Hans-Peter Diettrich wrote

the API is completely removed, it should exist and behave as before, on the platforms to which it applies.

It doesn't apply unconditionally to any current fpc target. Even go32v2 might run on windows with changing memory availabilities.