The build faq (that I have) states:
"Due to some new developments with parallel compiling using the FPC makefiles, make 3.80 is strongly recommended as of FPC 2.1.1 (januari 2007 and later). Unix people with dual cores and up to date source trees might want to try ?make -j 2?."

With an eye on increasing fpcup [1] performance:
1. Does make -j work reliably on Windows, too?
2. I intend to detect the number of logical cores as per http://wiki.lazarus.freepascal.org/Example_of_multi-threaded_application:_array_of_threads#1._Detect_number_of_cores_available. and run that many jobs. Is that a good idea?
3. If somebody could extend the detection code wiki article for FreeBSD, I would appreciate it ;)

Thanks,
Reinier

[1]
https://bitbucket.org/reiniero/fpcup/overview

With make 3.82, yes.

2. I intend to detect the number of logical cores as per http://wiki.lazarus.freepascal.org/Example_of_multi-threaded_application:_array_of_threads#1._Detect_number_of_cores_available. and run that many jobs. Is that a good idea?

I simply use make -j and set "FPMAKEOPT=--threads=4" on a Quad core.

Reinier Olislagers wrote

2. I intend to detect the number of logical cores as per http://wiki.lazarus.freepascal.org/Example_of_multi-threaded_application:_array_of_threads#1._Detect_number_of_cores_available. and run that many jobs. Is that a good idea?

That's non-portable, even on Linux. There's a syscall which returns a bit vector of available CPUs, you need to count those bits since the layout might be sparse.

qqv taskset(1), sched_setaffinity(2), sched_getaffinity(2)

Thanks, guys - and also Mattias who I noticed has already updated the code on the wiki ;)

In our previous episode, Reinier Olislagers said:

No. Even not on Windows. You need the number of physical cores, not logical ones. This will return the duplicate number of cores on systems with hyperthreading. (I tested on my system and indeed, 8 instead of 4)

Do you see a performance dropoff or instability/crashes... or both when increasing the job number past the amount of physical cores?

In our previous episode, Reinier Olislagers said:

IIRC yes, 5 is still increasing, 6 is lower than 4 or 5. If I boot it up in windows, I'll retest.

The point was more that the autodetection is IMHO still too immature to enable by default.

But those are SSD based values. Increased disk seeking might make it worse on HDDs.

Ok, thanks.

The point was more that the autodetection is IMHO still too immature to enable by default.

Got it.
It's disabled for now on Windows anyway as fpcup uses make 3.80 by default to support the 2.6.x series (see Florian's post).

But those are SSD based values. Increased disk seeking might make it worse on HDDs.

Ok.

Quick test on Linux x64 VM seems to show increasing to 8 (matching my # of physical cores, no hyperthreading on my CPU) does increase performance... using the famous unit of time called "number of cups of coffee I thought I drank while waiting for the compile"

The limited testing of make etc. that I've done on systems with hyperthreading showed that optimal performance matched the HT count, not the core count. Obviously "your mileage may vary" depending on cache architecture and the amount of RAM.

Irrespective of whether we're talking about cores or HT, for a system that isn't grossly NUMA I'd expect time-to-completion to be roughly logarithmic against the number of CPUs in play. I'd suspect that by some metric, there's a "point of inflection" roughly corresponding to the number of CPUs or cores, but that's not to say that performance can't be improved by going slightly higher.

For a large-scale NUMA system, or for something like MOSIX or an explicit build farm, all bets are off.

Reinier Olislagers wrote

Quick test on Linux x64 VM seems to show increasing to 8 (matching my # of physical cores, no hyperthreading on my CPU) does increase performance... using the famous unit of time called "number of cups of coffee I thought I drank while waiting for the compile"

Historically, 8 has been a bit of a limit on many different architectures. I don't know where this figure has come from, ultimately it could be something like a speed ratio between main memory devices and cache. Some of the larger systems have attempted to get round this limit by having multiple busses, e.g. 4x on the Cray CS6400 which was a precursor of the Sun E10K.

My tests suggest that the shape of the rolloff is OS-specific. IBM mainframes, of course, claim to vastly exceed anybody else's throughput, but I believe that they've got things like four-level cache as well as fibre-connected NUMA.