Hi,

Just wanted to say I like the new less verbose compilation of fpc trunk. Is this part of the result of moving to fpmake?

Regards,

Andrew Haines

Andrew Haines wrote

Just wanted to say I like the new less verbose compilation of fpc trunk. Is this part of the result of moving to fpmake?

Sort of, yes, although it would be trivial have regular "make" be less verbose also ( output of make when compiling the Linux kernel).

Jonas

Andrew Haines wrote

Hi, Just wanted to say I like the new less verbose compilation of fpc trunk.

+1 [ there was no "Like" button, so I had to resort to Google's +1 ;-) ]

I really like the compilation percentage being displayed. This gives me a good indication if I can go for that coffee break or not. :)

PS:
Adding any more (if possible) parallel compilation support would be awesome too. This already saves me over 1 minute in compile time on my quad core. Yes, I hate it if CPU's just sit idle.

In our previous episode, Graeme Geldenhuys said:

PS: Adding any more (if possible) parallel compilation support would be awesome too. This already saves me over 1 minute in compile time on my quad core. Yes, I hate it if CPU's just sit idle.

IIRC add FPMAKEOPT='-T 3' (3= number of paralel threads)

Marco van de Voort hat am 7. März 2012 um 11:27
geschrieben:

I tried this and saw only short periods more than one processor used. Only for a few seconds three were used. Is this normal?

Mattias

In our previous episode, Mattias Gaertner said:

IIRC I noticed a small improvement when Joost mentioned (like 10s on an overal 200 secs overall build). I have no other info on this than that it exists :-)

This makes a big difference for me:

Without FPCMAKEOPT: 1m48.056s
With FPCMAKEOPT="-T 9": 1m13.833s

Unfortunately, cycling the compiler is taking really long and -j 9 isn't helping much (or at all).

Henry

In our previous episode, Henry Vermaak said:

fpmake only handles packages/.

fpmake (or make -j the way FPC used it in the past) can't really speed up runs that are a single compiler invocation (like the compiler), unless it can run two such runs in paralel.

In packages this is possible, in the compiler cycle compiler depends on RTL, and the "next" rtl depends on the "previous" compiler, so nothing can be done there.

To exploit paralellism in single compiler invocations, threading should be brought into the compiler, but that is an huge and uncertain job.

With c, the compilation of .c -> .o is isolated, so many of them can be run in parallel, which means that compiling e.g. the linux kernel, you get great performance from make -j %d. Is it possible to have fpc do the same thing somehow? Bringing threading into the compiler sounds very scary, indeed.

Henry

Marco van de Voort wrote

IIRC add  FPMAKEOPT='-T 3'   (3= number of paralel threads)

I use the following when I recompile a new FPC. Recommendation (on the internet) was to always use a -j value one greater than your number of cores.

make all -j 5 FPC=$COMPILER

How does this differ to your suggestion? Or does it translate to something similar.

In C the dependencies between the .o files are resolved at link time while in Pascal they are already resolved at compile time (the unit concept). So the only feasible possibility for a speed up there is threading support in the compiler as only the compiler fully knows the dependencies between the units.

Regards,
Sven

With fpmake "-j x" isn't usable anymore, because AFAIK Joost found no way to "hook" into Make's threading mechanism to utilize that for fpmake. So you'll need to use the parameter above (which is handed over to fpmake). As threading is basically only useable when compiling the packages this shouldn't make a big difference to the "-j" approach.

Regards,
Sven

Henry Vermaak wrote

Unfortunately, cycling the compiler is taking really long and -j 9 isn't helping much (or at all).

It can be sped up by adding proper unit dependency information in your platform's RTL Makefile.fpc, and then removing the .NOTPARALLEL. Except for the Darwin RTL, no RTL currently has proper dependency information. It also won't work for the Windows RTLs, because they use build units (which is another way to speed up things, although you can't take advantage of any parallelism in that case without modifying the compiler).

Jonas

Thanks for the information. Do you have benchmarks of the Darwin RTL compilation before and after adding the dependency information?

Henry

Sven Barth hat am 7. März 2012 um 14:04
geschrieben:

my

isn't

up

>> runs that are a single compiler invocation (like the compiler), unless

it

be

the

Compiling C typically starts many times the compiler. Each time the OS has to load the compiler, resolve the symbols, load all needed header files, translate them, ... This is a waste of resources. Of course you can do this waste easily in parallel.
FPC is clever enough to do most things only once. That's hard to do in parallel.

Loading the PPU files takes about 10-50%. This might be doable in parallel.

Mattias

Sven Barth wrote

With fpmake "-j x" isn't usable anymore, because AFAIK Joost found no way to

Ah, thanks for that tip, I adjusted by Trunk build script.

Henry Vermaak wrote

Thanks for the information. Do you have benchmarks of the Darwin RTL compilation before and after adding the dependency information?

No, but you can simply compile without -j and with -j 2 to see the difference. Note that these numbers are on a PowerMac G5/1.8GHz (dual cpu). I don't know how representative these numbers are for a modern x86 machine.

Without -j

real 0m21.710s
user 0m14.912s
sys 0m6.541s

With -j 2

real 0m14.848s
user 0m15.187s
sys 0m6.891s

Jonas

If you want to optimize for speed on large parallel systems, starting from a fast sequential algorithm is usually the worst approach you can take. Optimizing for parallelism requires a completely different way of thinking, and indeed often includes approaches that would seem wasteful in a purely sequential environment.

Jonas

Apologies for reviving an old thread, but I've been reinstalling Debian on one of may larger systems and rebuilding FPC on it.

Sven Barth wrote

Am 07.03.2012 13:51, schrieb Graeme Geldenhuys: > On 7 March 2012 12:27, Marco van de Voort wrote: >> >> IIRC add FPMAKEOPT='-T 3' (3= number of paralel threads)

What version did this start working with- 2.6 or 2.7?

> I use the following when I recompile a new FPC. Recommendation (on the > internet) was to always use a -j value one greater than your number of > cores.

That recommendation goes back to "mainframe shops", and is based on the premise that CPUs are expensive and it's best to keep them busy. My experience (from kernel builds etc.) suggests that there's little if any advantage attempting to parallelise beyond the number of CPUs, and there's rarely any advantage going beyond 8 since that tends to saturate the cache/bus bandwidth.

Using a test that I've run for years on a wide range of systems, and noting that these are from Suns which might not scale quite the same as PCs,

Two-CPU system:

Job count: 1
real 1m40.216s
user 0m42.423s
sys 1m4.216s

Job count: 2
real 1m10.183s
user 0m43.903s
sys 0m59.856s

Job count: 3
real 1m9.460s
user 0m44.787s
sys 1m0.524s

Job count: 4
real 1m9.618s
user 0m45.883s
sys 0m59.880s

Twelve-CPU system:

Job count: 1
real 1m39.623s
user 1m8.568s
sys 0m55.355s

Job count: 2
real 0m52.394s
user 1m9.072s
sys 0m58.828s

Job count: 3
real 0m36.387s
user 1m10.608s
sys 0m58.600s

Job count: 4
real 0m28.359s
user 1m11.712s
sys 0m59.476s

Job count: 5
real 0m23.438s
user 1m12.893s
sys 0m58.936s

Job count: 6
real 0m20.338s
user 1m12.825s
sys 0m58.256s

Job count: 7
real 0m18.658s
user 1m13.021s
sys 0m57.944s

Job count: 8
real 0m17.824s
user 1m12.509s
sys 0m58.556s

Job count: 9
real 0m17.210s
user 1m12.593s
sys 0m59.168s

Job count: 10
real 0m16.808s
user 1m12.589s
sys 0m59.764s

Job count: 11
real 0m16.531s
user 1m12.385s
sys 1m0.324s

Job count: 12
real 0m16.599s
user 1m13.093s
sys 0m59.652s

Job count: 13
real 0m16.584s
user 1m13.557s
sys 0m59.536s

Job count: 14
real 0m16.713s
user 1m12.629s
sys 1m0.624s

The need for FPMAKEOPT was added with 2.7.1 where we transitioned the compilation of packages to fpmake. In 2.6.0 you should be able to just pass "-j N" (N = number of parallel threads) to "make".

Regards,
Sven

Thanks, noted.