Hi,

I'm not sure how it works, but does FPC compiler optimizations for the amd64 (x86_64) CPU's get shared across various OSes?

eg: I've read various reviews that FreeBSD benchmarks often perform better than the same benchmarks run on Linux. It was even shown that FreeBSD runs Linux executable faster than Linux can. I can quote the websites if really need.

So while setting up my automated unit tests for tiOPF across my various systems I compared the test results and timings.

NOTE:
My host OS is FreeBSD 9.1 (64-bit) using a Intel i7-3770K @ 3.5Ghz with 16GB RAM. FreeBSD boot OS is on a 128GB OCZ Vertex 4 SSD. The rest of the system runs off a 3x 2TB ZFS in RAID-z1 setup (similar to RAID5). All other OS's run in VirtualBox VM sessions. I have a single Firebird 2.5 DB server running natively under FreeBSD. All file access tests run on each OS's native file system. The Linux VM session I have is Ubuntu 10.04.4 (64-bit).

Both OS's run FPC 2.6.0 and the exact same revision of tiOPF, FPTest and fpGUI. fpGUI is used for the GUI test runner of FPTest.

Here is the summary of the unit test results, and the times it took in minutes and seconds.

No of tests | Type of Tests | Linux | FreeBSD -------------+-----------------+------------+---------------- 151 | CSV persistence | 0:22 | 0:27 -------------+-----------------+------------+---------------- 151 | TAB persistence | 0:22 | 0:27 -------------+-----------------+------------+---------------- 151 | XMLLight | 0:23 | 0:26 -------------+-----------------+------------+---------------- 151 | SqlDB-Firebird | 3:14 | 3:38 -------------+-----------------+------------+---------------- 682 | Non-Persistent | 1:09 | 1:30 -------------+-----------------+------------+----------------

As you can see, consistently the FreeBSD tests take longer than the Linux ones. The test project on each platform was compiled with exactly the same compiler settings.

Also the Non-Persistent tests is 99% in-memory tests (no disk access). So this eliminates the idea that the file systems might cause the speed difference, though the SqlDB tests used the same DB server, and there was still a large difference in speed.

So this got me wondering. Does FPC compiler optimizations differ between FreeBSD and Linux, even for the same CPU type? If so, then the results is probably understandable, because there are more Linux developers and testers in the Free Pascal project, than for FreeBSD. Thus more working going into Linux optimization than for FreeBSD.

If my assumption about FPC optimization is incorrect, then I'm lost for ideas why my FreeBSD setup is so much slower than a Linux VM session (and contrary to other benchmarks on the net).

[sorry for the long winded email]

Regards,
- Graeme -

AFAIK the optimizations are CPU specific, not OS specific. And in that case your findings are indeed interesting especially since the Linux tests were performed in a VM. It could be though (I'm definitely guessing here) that the FreeBSD release was created with different optimizer settings than the Linux one (regarding the RTL and packages). Otherwise I'm as puzzled as you...

Regards,
Sven

What exactly are we looking at there: 151 iterations inside a single program, or 151 programs? Assuming the former, I think the first thing to do is to run with different numbers of iterations to determine whether FreeBSD is imposing a "one-time charge" or if it's an overhead per iteration. Also it would probably be worth breaking the result out in the standard format shown by the unix time command.

I would say the "No of tests" is the number of different test functions that have been invoked (all within one single test program).

Regards,
Sven

The unit testing project is a single executable running all the above tests. 151 is 151 different unit tests to test the various persistence layers. The test suite is based on a hierarchy of classes, that is why all persistence tests have 151. The exact same persistence tests are run for each persistence layer. The 682 is again different tests for anything non persisting - testing various classes, and pretty much all functionality of those classes.

I did not setup the testing framework to run multiple iterations, only one run was completed with a total of around 1200+ unit tests taking around 4:30-5:00 to complete, from start to finish.

So memory cache etc should really have an effect. Because each test case starts from scratch, does the test, then cleans up. Then the next test etc etc.

Regards,
- Graeme -

Sven Barth wrote

I would say the "No of tests" is the number of different test functions that have been invoked (all within one single test program).

Correct. The first four sets are the same persistence tests, but against different persistence backends (CSV, TAB, various client/server DBs). The last set is for anything in tiOPF that is not related to persistence (storage). The last set includes base classes, visitor, iterators, parsers, encryption, object/list management, mediators etc. - the core functionality of the tiOPF framework.

I only selected some of the persistence layers for this message thread. tiOPF supports many more like SqlDB-Postgress, SqlDB-MySQL, Zeos-Firebird, Zeos-MySQL, etc.. pushing the total unit test count over 2000+ But the ones I showed above gives a clear indication of what I am after.... Why is native FreeBSD slower than Linux (in a VM)?

Regards,
- Graeme -

Sven Barth wrote

AFAIK the optimizations are CPU specific, not OS specific.

That is what I expected too.

guessing here) that the FreeBSD release was created with different optimizer settings than the Linux one (regarding the RTL and packages).

Both FPC 2.6.0 installs were from the binary releases made by the FPC team. I never install FPC from FreeBSD ports of Linux repositories.

Otherwise I'm as puzzled as you...

:)

Regards,
- Graeme -

Could it be OS calls are slower on FreeBSD? I don't suppose there's an easy way to profile application versus OS call execution time, is there?

Jeff.

DaWorm wrote

Could it be OS calls are slower on FreeBSD? I don't suppose there's an easy way to profile application versus OS call execution time, is there?

There is:

=== command begin ===

$> time fpc -h > /dev/null
fpc -h > /dev/null 0,01s user 0,01s system 46% cpu 0,035 total

=== command end ===

Regards,
Sven

It could nevertheless be that both releases were created with different optimization options and this is what I was guessing about :)

Regards,
Sven

There's a vast number of factors that you're underreporting. For example, the underlying VM system could detect which OS is running as a guest, and behave differently. Or the OS could reconfigure the CPU and chipset (specifically, the cache architecture) to a different extent. Or Linux's habit of using all available memory as disc buffering could be accelerating startup and/or execution. Or BSD could be making heavy going of fixing up dynamic library references. Or the different guests could be running their (simulated) disc controllers differently.

The only way that you're going to get anywhere is by tweaking the programs to loop, so that you can factor startup time out. Then separate groups of tests that don't make heavy OS use from those that do. All basic engineering stuff.

That should normally point to the fact why VM's run slower than the host. Simulated hardware etc. But in my case, it is other way round. Linux is in the VM, yet runs faster than FreeBSD.

The only way that you're going to get anywhere is by tweaking the programs to loop, so that you can factor startup time out.

I seriously doubt it is startup time that accounts for the difference - especially in the case of non-persistent test, where a test might simply search a string for tokens, create a list object, add a few objects, then free the list object, making sure the children are freed too. Such tests all happen simply in memory and don't have any OS or other library API calls. Plus these exact tests run on both OS's, so if a test startup would be a factor, it would apply to both OS's.

Looping the test suite is no problem, the testing framework already supports that. But I doubt timing will be much different. I will run a 10 iteration anyway, just to see.

I'll try and investigate further, and see if I can create more benchmark tests.

For now I have the following FPC questions...

1) Does the binary releases of FPC for Linux and FreeBSD use the same compiler settings when a release is created?

2) Does FPC optimize code per CPU and OS, or just per CPU architecture? I assume the latter, but would appreciate a confirmation from somebody that knows FPC internals.

Regards,
- Graeme -