On Mon 30/07/18 01:58 , Ben Grasset
sent:

I was even able to successfully actually

fix some of the leaks, but others

boiled down to a "node" being created as

a local variable in the middle of

some 1000+ line method, getting assigned

to the "left" or "right" property

of some other globally visible node, and

then never being freed.

To add to that, some of then actually

*could not* be freed even when I

tried to do so without raising an

untraceable access violation later in the

compilation process, also, which means

the compiler is basically knowingly

relying on undebuggable undefined

behaviour in multiple places that could

easily be affected or altered by

anything else anywhere else in the

codebase at any time. The way-too-short, highly undescriptive

naming of variables doesn't

exactly help, either! "hp" and "p" are

not good variable names. They really

aren't!  Overall, I'm not trying to put blame on

anyone in particular here, but as

someone who loves using FPC and would

like to see it continue to grow, it

would be nice if we could collectively

address some of these basic,

fundamental issues (which I'd be happy

to help with myself, of course.)

I too love Free Pascal and seek to improve
it. If you've found a fix, by all means
submit it as a patch in the bug tracker.

I've noticed that the compiler doesn't use
try...finally blocks to help with freeing
blocks. I'm not sure why this is the case,
but might be speed related.

More complex refactoring might require
deeper planning and approval from Florian.
For example, the node construction relies
a lot on global variables (especially
current_procinfo) that could possibly use
some improvement.

I think a big issue with refactoring is
the amount of testing required because of
the numerous target platforms.

Don't be disheartened though. With enough
eyes, all bugs are shallow!

Gareth aka. Kit

Yes. try/finally blocks cause *tremendous* slowdown.

Michael.

J. Gareth Moreton schrieb am Mo., 30. Juli
2018, 13:31:

Correct. Even implicit try-finally frame generation is disabled for the compiler source.

Regards,
Sven

On Mon, Jul 30, 2018 at 9:14 AM, Sven Barth via fpc-devel <

Is performance more important than being correct? :|

regards,
Marcos Douglas

Marcos Douglas B. Santos wrote

Is performance more important than being correct?  :|

In this project, the answer is always taken to be yes.

You've been around long enough to know that complaints about the speed of the compiler is something that pops up on a regular basis.

Also, and this is important: the compiler was developed for TP. It had no concept of exceptions. Everything was done using records and pointers.

Many people will probably be surprised, but the use of try/finally is not required for an application:

Gradually, some parts of the compiler were switched to classes, but even then, without exceptions: As long as you do not use the sysutils unit, (which was the case for many years for the compiler) exceptions will not be raised, and so there is simply no need for try/finally blocks.

Probably it would be better to remove the dependency on sysutils, to keep the compiler free of try/finally blocks.

But the compiler devs will be able to provide a more correct&detailed picture than this.

Michael.

From: fpc-devel On Behalf Of Marcos Douglas B. Santos Sent: Monday, 30 July 2018 22:25
To: FPC developers' list
Subject: Re: [fpc-devel] Why/how does the compiler have a non-trivial number ofmemory leaks after over two decades of development?

On Mon, Jul 30, 2018 at 9:14 AM, Sven Barth via fpc-devel < > wrote:

J. Gareth Moreton < > schrieb am Mo., 30. Juli 2018, 13:31:

I've noticed that the compiler doesn't use
try...finally blocks to help with freeing
blocks. I'm not sure why this is the case,
but might be speed related.

Correct. Even implicit try-finally frame generation is disabled for the compiler source.

Is performance more important than being correct? :|

Freeing memory is irrelevant if your process is going to exit anyway right after freeing it. The OS doesn’t care about any memory your process has allocated at that time. All pages go back into the available pool.

Depending on how the allocation/free behaviour of the compiler looks like, but it might even make sense to use a memory manager that doesn’t even know how to free memory.

(I’m using windows terminology here, but I’m sure equivalent can be found on other platforms)

Reserve a large block of address space.

Install a vectored exception handler that reacts to an access violation that falls inside the reserved block by committing the memory between start and error address + 1MB (or more or less).

Naïvely and without error checking, Alloc gets reduced to:

var
  CurrentPtr: PByte;
function Alloc(Size: Integer):Pointer;
begin
  Size := (Size + 3) and not 3; //adjust depending on what allocation alignment is desired;
  If Size > 0 then begin
    Result := CurrentPtr;
    CurrentPtr := CurrentPtr + Size;
  end else
    Result := nil;
end;
procedure Free(p: Pointer);
begin
  {do nothing}
end;

Many people will probably surprised, that try/finally/except works without sysutils and try/except does not need "Exception". It works with any TObject.

Probably it would be better to remove the dependency on sysutils, to keep the compiler free of try/finally blocks.

Sysutils has nothing to do with creating exception frames.

Mattias

It does: if you know the program will never raise exceptions, it makes no sense to install try/finally blocks.

Runtime errors are converted to exceptions by the sysutils unit. By not including it, you know you will just get runtime errors, and hence you don't need try..finally.

Obviously provided you don't install another mechanism that does this and don't raise exceptions manually, which - AFAIK - is the case in the compiler...

Michael.

Michael Van Canneyt schrieb am Mo., 30. Juli 2018, 15:30:

Obviously provided you don't install another mechanism that does this and don't raise exceptions manually, which - AFAIK - is the case in the compiler...

The compiler does use exceptions when the compilation needs to be aborted e.g. when too many errors occurred or an internal error is encountered.

Regards,
Sven

This is bad news, because it in effect means that the try/finally is necessary if people want to fix memleaks :(

This will considerably slow down the compiler.

Michael.

Martok wrote

Am 30.07.2018 um 14:24 schrieb Marcos Douglas B. Santos: > Is performance more important than being correct? :| In this project, the answer is always taken to be yes.

To hopefully offer some explanation for Mr. Santos - if I can compile and run the program 5 times over the course of 20 minutes, that is certainly better than running it and testing it once over the same amount of time, no?

It might be hard to imagine FPC taking that much longer than it does currently but ~30min for a large program is the standard with other compilers. I very much enjoy the speed of FPC.

30 *Minutes*, is this real ?

Hm. I just complained to a component provider because Delphi takes 25 seconds to compile my program with the way they set up their paths...

Michael.

This is real.[1] I've experienced it myself and you can see some good discussions on https://news.ycombinator.com with a search. FPC/Lazarus has made some waves there in the last few years.

FPC does have quite a bit of a lead it could shed in an attempt to be "more correct" but I would caution against this. If at every point in a software stack someone says "modern computers are just so fast, we have a little bit of wiggle room" you end up with something slower than molasses. Perfection requires effort at every single step.

A good case study on this is keyboard latency in modern systems.[2]

Cheers,
R0b0t1

[1]: The most extreme comparison was closer to being something like a couple of minutes vs. a couple of hours, but I can't source that right now. Most people think it's a lie; granted, some of it is likely due to the excellent incremental compilation FPC has by default. [2]: https://danluu.com/keyboard-latency/

I would very much like to see the compiler usable as a dynamic library. Compartmentalizing the bulk of the code in this manner might aid readability and comprehension even if it is not immediately wise to actually use the shared object in a third party application due to memory leaks.

I wouldn't mind running some static analysis on FPC in an attempt to get rid of memory leaks. It's fairly mechanical and probably a good introduction to the codebase. This page[1] is promising as it indicates valgrind should work well on FPC binaries (I didn't have much a reason to expect otherwise, but you never know).

Is there any reason this hasn't been done? What could raise statements be replaced with if they must be replaced?

Cheers,
R0b0t1

[1]: http://wiki.lazarus.freepascal.org/Profiling

All us like fast compiling...
However, IMHO, a compiler should use the same good practices that others programs must use.
If it cannot use them because it's very slow, what is the matter about good practices?

"Do what I'm telling you to do, but don't do what I do" ? :)

best regards,
Marcos Douglas

If a program isn't long running I see programmers tend to not care about memory management, nor do I make a habit of caring about it myself. I find it hard to see anything wrong with this.

As someone has mentioned this can fail on smaller computers, but how much RAM is too little? SBCs now typically come with 1-4GB of memory.

Cheers,
R0b0t1

yep, for sufficiently large projects... when we compile OSG (OpenSceneGraph) over here (using -j 8) it takes 20+ minutes for the first time or if we reconfigure from debug to release mode or the other way around...

now, after the first time we compile it, the next times takes mere 10's of seconds... then we move on to SceneGear and FlightGear when we're building that whole project which uses OSG... 30-45 minutes for a complete, from the ground up builds are not uncommon... it takes me back to the days of 30 years ago when you'd start the (borland/turbo pascal) compile process and go get some c0ffee ;)

note: the machines these builds are being done on have 16Gig RAM, 1Tb HD, and a 4Ghz 8-core AMD FX-8350 CPU...

I'm sorry, but we work with a language that expected programmers should manage their objects.
The language even provides a clean way (try-finall) to do that. I use try-finally even to write a snippet to show an example of code...

IMO, that must be a habit in such languages.

regards,
Marcos Douglas