Hello,

Problem N1. It is a dependency problems. I have some 50-60 units that are in various ways interconnected (although, without crosslinks). The main problem is that the main unit initializes after some other second-level units. This is a very unlikely thing and in fact I don't know how to avoid it. I suspect that this problem appeared in the latest FPC versions (I use 1.0.5 branch, 1-apr-2001).

Problem N2. win32 version (1.0.5, 9-mar-2001) asks for windows.ppu file when I compile program (it has uses Windows directive).. That's something strange, like I'm using go32 version.

Problem N3. 1.1 branch always reports "not enough memory" when I compile programs (1.0.5 compiles well). I have 16 MB and running win95osr2. Seems to be that compiler don't want to use virtual memory or uses 40-50 megs (I have a little swap space).

Best regards,
Aleksey

Aleksey V. Vaneev wrote

Problem N3. 1.1 branch always reports "not enough memory" when I compile programs (1.0.5 compiles well). I have 16 MB and running win95osr2. Seems to be that compiler don't want to use virtual memory or uses 40-50 megs (I have a little swap space).

Try using -vl, then the compiler will report when it allocates memory. It' not 100% accurate (during assembling/linking, not all allocations are reported), but should give a general ide. There may be some memory leaks in the compiler as well, though I thought Peter fixed em all recently.

Jonas

?? And this was not true in earlier versions ?
I don't think anything has changed in this area of the compiler ?

Michael.

Seems to be it was in earlier versions too... But anyways I cannot see what I'm doing wrong (this bug appeared some days ago when I've did several changes to programs I compile). Btw, what's the algo of finding the most main unit that must be initialized first? Maybe there are some logic problems?

At least compiler could give an error if it cannot decide which unit to initialize first... It gives such error if units declare usage of each other in the interface part. But it says nothing if units declare usage in the implementation part (this is a correct behaviour I suppose, but then why it doesn't works correctly?).

Best regards,
Aleksey

The order of unit initialization is simply the order in which they are declared in your program. If you use a unit X within another unit Y, that that unit X will be initialized before unit Y. All units will be initialized only once.

At least compiler could give an error if it cannot decide which unit to initialize first...

This never happens, since the order is simply read from the source. If you compile your program with -a, you can see the order in which the units are initialized by looking at the generated .s file for the main program and by then searching for "INITFINALTABLE".

Jonas

1. Used units in the interface are loaded
2. Interface parsed
3. Used units from implementation loaded
4. Implementation parsed

The order in which the initialization is written is the order of loading of the units. This is hard to guess in a generic way when the project contains interdependent units.

The only thing how you can control is by placing the unit that must be initialized before the current unit in the interface uses line.

Hello,

The only thing how you can control is by placing the unit that must be initialized before the current unit in the interface uses line.

The main thing is that not all units declare unit usage in the interface part. Sometimes it is not possible because of interdependency.

Maybe it is possible to build some "clever" dependency tree?

Best regards,
Aleksey

Hello, > The only thing how you can control is by placing the unit that must be initialized

before

> the current unit in the interface uses line. The main thing is that not all units declare unit usage in the interface part. Sometimes it is not possible because of interdependency.

That's the problem in your project not from the compiler. You need to rethink your module structure in your project to solve this or think about an different initialization using InitModule and DoneModule procedures and calling those yourselves.

Maybe it is possible to build some "clever" dependency tree?

Go ahead if you know how to do it, the main parts of the code are located in pmodules.pas and fmodule.pas. But don't forget the whole unit loading situation is very complex and dangers of recursion.

Hello,

That's the problem in your project not from the compiler. You need to rethink your module structure in your project to solve this or think about an different initialization using InitModule and DoneModule procedures and calling those yourselves.

That's impossible in all the cases. I will need then to join all modules or at least a huge amount of them into one module. Regarding "initmodule" and "donemodule".. Can't understand why initialization and finalization things be needed if I will be forced to do things this way. FPC will look like C after all.

BTW, my (ours, really) project is currently private, so I cannot spread huge parts of it (btw, HRC compressor is a part of it).

> Maybe it is possible to build some "clever" dependency tree? Go ahead if you know how to do it, the main parts of the code are located in pmodules.pas and fmodule.pas. But don't forget the whole unit loading situation is very complex and dangers of recursion.

It is generally difficult to dig some other's code. Maybe I can think of a general algo. Dunno. If something pops up I will write here.

BTW, can you briefly explain work of built in FPC memory manager (what kind of trees are used if any, etc)? Just wanted to know - maybe there's some place for the development...

Best regards,
Aleksey

It's just the way things are supposed to work: initialize everything in the order they are declared. How on earth is the compiler supposed to to know that unit x's initialization code depends on unit y being initialized already except from the order of declaration in the uses clauses??

I really don't see what kind of algorithm could solve that. It's as Peter said: you have to desing your project in such a way that the initialization routines are called int he correct order and if that isn't possible, use some initmodule/donemodule technique.

BTW, can you briefly explain work of built in FPC memory manager (what kind of trees are used if any, etc)? Just wanted to know - maybe there's some place for the development...

It's a hashtable based on the size of the memory blocks rounded up to a multiple of 16bytes. Each location in the hashtable contains a linked list of blocks of that size * 16. The maximum blocksize for which such a list is kept, is 512 bytes. All blocks > 512 bytes end up in a separate (common) linked list.

If you allocate a block and it's <= 512, we first look in the hashtable to see if there are any free blocks of that size. If so, it is returned and removed from its list. If not, we first search bigger blocks to split into smaller ones (upto 8 times the size of the requested block) and otherwise we simply create a new block at the end of the heap.

If you allocate a block > 512 bytes we search the freelist of large blocks (linear search) and if no large enough block is found, then we allocate a new one at the end of the heap. By default, we search the freelist of the large blocks to find the best match (enabled via a define in rtl/inc/heap.inc).

When releasing a block,

a) if it's <= 512 bytes, it's added to the appropriate freelist in the hashtable
b) otherwise we first check whether there aren't any other blocks coming after it and if so, we concatenate those (it's possible to disable this, again using a define)

But as Peter already said, the heap manager is very optimized. We got some patches a couple of months ago to make it slightly faster (10% or so), but I don't think they're already applied.

Jonas

BTW, can you briefly explain work of built in FPC memory manager (what kind of trees are used if any, etc)? Just wanted to know - maybe there's some place for the development...

all list are double-linkedlists. Every unused block is one of the freelists. For blocks < 1024 there is a list for each size. For other blocks there is a general list.

Out of one of the freelist are the blocks allocated. When no free block is found a new block from the OS is requested.

When blocks are freed they are merge together with their next neighboor.

To expiriment you can just copy the routines from heap.inc and create an own memorymanager (see heaptrc.pp) that can be plugged in. Instead of requesting blocks from the OS you can than request big blocks from the internal heapmanager.

Possible optimizations:

- Better handling of end of OS memoryblock. This is currently done by a flag that also needs inheritance handling when merging. Maybe allocating a small block at the end will make this more simple.

- Deffered merging

For more memory allocation info look at the malloc implementation from Doug Lea http://gee.cs.oswego.edu/dl/html/malloc.html

But as Peter already said, the heap manager is very optimized. We got some patches a couple of months ago to make it slightly faster (10% or so), but I don't think they're already applied.

Because the patches made the heapmanager exponantional slower. The compiler cycle took then about 5 minutes instead of 1 minute.

It's always a speed versus size tradeoff...

Aren't we talking about different patches? I meant the ones that did the merging (tryconcatfree) at allocation time instead of at freemem time. I believe the submitter said those made the compilation of windows.pp a bit faster.

Jonas

PV> From: Peter Vreman

> But as Peter already said, the heap manager is very optimized. We > got some patches a couple of months ago to make it slightly faster > (10% or so), but I don't think they're already applied.

PV> Because the patches made the heapmanager exponantional slower. The PV> compiler cycle took then about 5 minutes instead of 1 minute. PV> It's always a speed versus size tradeoff...
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Are You shure that small RAM usage always assume lower speed? What about running application which need 100 Mb RAM on PC with 16 Mb? And how about UNIX which start to kill applications on low memory condition?

About patches: it is slower because it is trying do not allocate new block from system if we can get memory by mergeing small blocks. But this can be improved: ideally we must use memory limit value for process (`man limits` for UNIX) to start such allocation politic. If process memory usage is lower than 2/3 of this limit then we can skip small blocks mergeing: it is what causing speed down.

In such case we would work fast when process memory usage is low and _fast_ when process memory usage is big: because we continue to use RAM and do not start swapping.

If You add to runtime a routine which return such limit value then I modify a patches...

Regards,
Sergey Korshunoff

PS: There is some strange memory usage in PFC: if to interrupt a FPC compilation by Ctrl-C and then start it again -- I got new FPC much faster (more than 10 times) than when I do not interrupt a FPC making. This is true for my case (16 Mbytes of RAM). I got a result faster because after interruppt (second start) FPC use _much less_ memory (less swap is used). But why FPC
keep all in memory? May be it is not to hard to implement such politics: -- we start a UNIT A compilation
-- we found a dependency on UNIT B (in `uses` or `implementation` part) X) -- we free all memory allocated by UNIT A compilation and go to compile UNIT B
-- repeat

As I think FPC currently do not perform a X step...

Ofcourse a new faster algoritm that uses less memory is the best solution. But in these periods where RAM is cheap and all new PCs get at least 64MB and most already 128MB and game-pcs already 256MB. You must decide why not using a little bit more memory to get more speed.

I already commented in a private mail on your heap patch. I didn't say that everything you did was wrong. The heapblock allocation at the end of a OS memoryblock is a good thing. And also the deferred merging of blocks approach is good. But there were some small things like the small block merging that was taking to much time.

That's true. FPC keeps the symtables of all loaded units. Unloading the units after each unit compilation and then loading the units (including the units were these units depend on) again is hard to implement (what to do with a unit were only the interface was already compiled?) and very terrible for the performance. In case of the compiler: for compiler.pas all 100 units are loaded and unloaded, then pp.pas is compiled and that requires than 101 units to be loaded and unloaded.