On low-spec systems (for example Linux on ARM with 32Mb or Solaris on SPARC with 128Mb) linking a large project such as the Lazarus IDE can take several days.

Is it possible to get any sort of activity indication that confirms that the linker is actually doing something useful, rather than just sitting there locked up?

Mark Morgan Lloyd wrote

Is it possible to get any sort of activity indication that confirms that the linker is actually doing something useful, rather than just sitting there locked up?

No, there isn't. FPC currently does not show any output from the linker, but even if it did I don't think there's any parameter you can give to the linker to make it print a progress report.

Jonas

If you ask specifically about linking, I'm afraid that we depend on GNU linker features here. Even if we wanted to launch ld asynchronously and provide some sort of moving dash from the compiler, it doesn't tell anything about ld still doing anything useful...

Tomas

Thanks Jonas- it's taking about a week to compile/link in the extreme case and something predictable- even if it was just a temporary file with a timestamp that changed predictably- would be really useful.

Fortunately I've got a somewhat more powerful system that I can put Solaris on, but compared with Linux it really is a resource hog.

2010/12/28 Mark Morgan Lloyd :

I did some experimenting. During linking I saw the lazarus executable growing, so you could do a ls -l on it.
What also give some indication of progress is using OPT="-k-t" which outputs all input files.

On my computer, linking of lazarus is well under 1 min, so not comparable. Between the last output of -k-t and the creation of new lazarus executable is some time too, maybe that is a days for you, without any output.

Vincent

Output file is sitting here with zero length, so either the linker is sitting in a dark corner giggling to itself or it's slowly writing output but hasn't committed it. I'll try OPT="-k-t" on the next run, I was wondering about that one earlier.

FPC works well enough to compile itself on Solaris (as discussed elsewhere) so I've got no reason to think that things are totally broken. However I've not got a number of tools that I rely heavily on when running Linux- in particular since I've not got GNU Screen if the SSH session goes down I'll have to start over, and I've got neither free nor top so don't have an easy way of looking at what memory allocation is doing.

There are obviously a lot of facets of Solaris that I haven't explored yet, but the CPU and memory it requires for even basic operation appear to have increased exponentially between v8 and v11.

Mark Morgan Lloyd wrote

FPC works well enough to compile itself on Solaris (as discussed elsewhere) so I've got no reason to think that things are totally broken. However I've not got a number of tools that I rely heavily on when running Linux- in particular since I've not got GNU Screen if the SSH session goes down I'll have to start over, and I've got neither free nor top so don't have an easy way of looking at what memory allocation is doing.

Try prstat. http://bhami.com/rosetta.html is handy in such situations.

Jonas

Thanks, I'd forgotten that site. Tries to tell me that CPU is 0.05 but still takes 10 secs to login- I'm pretty sure the whole system's thrashing... [applies ear to case] yup, it's thrashing.

One thing that this obviously emphasises is that individual CPU speed and overall memory capacity can still be important: even if compilation could be reliably parallelised (gmake -j) I doubt that linkage could.

I'll get on with the other holiday tasks- vacuuming servers etc., I'll be back.

Mark Morgan Lloyd wrote

Thanks, I'd forgotten that site. Tries to tell me that CPU is 0.05 but still takes 10 secs to login- I'm pretty sure the whole system's thrashing... [applies ear to case] yup, it's thrashing.

The ld manual page lists some options you can use to reduce its memory usage:
* --no-keep-memory (makes it re-read object symbol tables from time to time instead of keeping them in memory all the time)
* --reduce-memory-overheads (small hash tables and slower but less memory-intensive link map generation algorithm)

Jonas

Thanks Boris, I'll investigate although I must say that I'm being fairly careful not to install anything non-standard since what I'm trying to do is check out FPC and hopefully Lazarus on a virgin system.

I've just noticed that one of the prstat runs I did earlier shows a copy of Java taking up 137Mb (on a 128Mb machine) and that this is consuming

0% CPU, so it's probably contributing to the overcommit. I haven't a

clue what it's supposed to be doing in there...

Jonas Maebe wrote

The ld manual page lists some options you can use to reduce its memory usage: * --no-keep-memory (makes it re-read object symbol tables from time to time instead of keeping them in memory all the time) * --reduce-memory-overheads (small hash tables and slower but less memory-intensive link map generation algorithm)

I scripted some builds over the afternoon with all combinations of those, running on a Linux SPARC system with 192Mb which I was hoping would be tight enough to be a good test. Unfortunately the timings on all four were very similar: --no-keep-memory (without --reduce-memory-overheads) gave about a minute's advantage out of about 18 minutes link time.

I was hoping that --stats output might show something useful, but I think it gets swallowed. Vincent's suggestion of -t shows some useful info during the first part of the link, but in the sort of extreme case that I've currently got with Solaris I don't know at what point that would peter out.

--stats does work. Running on a tighter system (128Mb) I get times for make all plus seconds in the linker:

time make OPT='-k-t -k--stats' all
real 138m30.494s
user 26m43.840s
sys 3m38.614s
/usr/bin/ld: total time in link: 103.262453 lazarus /usr/bin/ld: total time in link: 32.394024 lazbuild /usr/bin/ld: total time in link: 34.706169 startlazarus Time to link all three binaries: 170 secs

time make OPT='-k-t -k--stats -k--no-keep-memory' all real 130m47.770s
user 28m1.017s
sys 3m52.851s
/usr/bin/ld: total time in link: 150.777423 lazarus /usr/bin/ld: total time in link: 44.086755 lazbuild /usr/bin/ld: total time in link: 50.603162 startlazarus Time to link all three binaries: 246 secs

time make OPT='-k-t -k--stats -k--reduce-memory-overheads' all real 125m16.758s
user 26m57.849s
sys 3m28.657s
/usr/bin/ld: total time in link: 93.757858 lazarus /usr/bin/ld: total time in link: 31.613976 lazbuild /usr/bin/ld: total time in link: 33.146071 startlazarus Time to link all three binaries: 159 secs

time make OPT='-k-t -k--stats -k--reduce-memory-overheads -k--no-keep-memory' all
real 114m6.972s
user 27m51.780s
sys 3m34.465s
/usr/bin/ld: total time in link: 142.376898 lazarus /usr/bin/ld: total time in link: 43.362710 lazbuild /usr/bin/ld: total time in link: 49.955122 startlazarus Time to link all three binaries: 235 secs

That appears to show a correlation between "user time" and the time the linker thinks its linking the big binaries, suggesting that the latter is got from process accounting rather than simple wallclock.

The thing that really matters is the wallclock time, where performance on a tight system can be improved slightly by feeding options to ld. I've not tried this on a very tight system, i.e. 32Mb RAM since runtime for the four tests would be around a month.

In all cases ld reports that linking the lazarus binary requires >200Mb working store, while the others require 120-150Mb. I think that explains why differences in timing start to stand out once there's less than 128Mb physical RAM.

I don't know whether there's a significant downside to using these options on a large system, there might be if disc storage was comparatively slow.

The thing that would probably make the most difference would be running in single user mode without any background daemons, but that's probably not possible unless logged in via a console connection.