As per good unit test guidelines, tests should be independent of each other and not rely on results from earlier tests. The perfect candidate for parallelism. So what about the idea of running x amount of test suites in parallel? It should at least give our multi-core CPU's something to do and maybe complete 2000+ tests in a shorter period. ;-)

Thoughts?

Regards,
- Graeme -

In theory, I am not against.

But: Not all tests can be run in parallel. DB code is for instance not thread safe.

Then:
Given the nature of decorators, it will require extensive checking to make sure that tests that rely on the same decorator will be run in the same thread anyway.

Just some random thoughts.

Michael.

I haven't got as many tests as that yet, but surely a parallel execution will make a good use of my multicore processor.

Maybe a new class named TTestCaseAsync, ie, all test cases that inherit it will run in parallel.

Marcos Douglas

I think this is maybe the best approach.

Michael.

In any event the output would need to be presented in a reproducible sequence.

Indeed, and this would require a huge change to the current output formatters. (at least the text output formatter)

Michael.

Mark Morgan Lloyd wrote

In any event the output would need to be presented in a reproducible sequence.

A good point, but thinking about it I don't think it would be that hard. The text output might be a bit more tricky, but not impossible.

Regards,
- Graeme -

Marcos Douglas wrote

Maybe a new class named TTestCaseAsync, ie, all test cases that inherit it will run in parallel.

That could work.

Does FPC 2.6.4 and later already support the attributes language feature? If so, that could be a possible option too.

Regards,
- Graeme -

Michael Van Canneyt wrote

Just some random thoughts.

Thanks for raising some good points.

At this stage I'm just thinking of possibilities on improving FPCUnit. As a testing framework, it is seriously lacking behind others. But, I'm not just all talk, I've submitted my first patch set last night to Mantis. There are more in the works if those go through.

Regards,
- Graeme -

I have seen the patch, and will look at it monday or tuesday.

As always: improvements are welcome.

And therein lies the catch: it's not always clear what constitutes an improvement ;)

Michael.

Blue-sky thought if there's to be /any/ modification. It would be useful to have something comparable with a Merkle tree here, so that there's a single overall fingerprint indicating that the entire run is OK (irrespective of the order of the tests), subsignatures indicating that the first and last halves are OK, and so on.

Michael Van Canneyt wrote

And therein lies the catch: it's not always clear what constitutes an improvement ;)

Ah, that's an easy one. Whatever helps speeds up the developer's (end-user) work. :-)

After using DUnit2 for so long I have gotten very used to the useful test output it gives and the many features in the GUI Test Runner. eg: A test Error (the unexpected one) in FPCUnit gave no hint as to where the error actually occurred. So you have to run it via the IDE and debugger just to find the location (could be many units deep), then go from there to fix it. With DUnit2 and with the latest patches for FPCUnit, it now tells you the exact unit and line number for Errors and Failures.

The next step would be to improve the GUI Test Runner in LCL.
- scroll treeview to first error or failure [done]
- improved keyboard shorcuts [done]
- save state of selected tests
- option to uncheck all passed tests so the next testrun only runs the previous failed ones
- double click on failed tests and let the IDE open the related unit and jump to the exactl line number in question.
- improved overview of failed tests

and many more others...

Regards,
- Graeme -

Graeme Geldenhuys wrote on Thu, 19 Mar 2015:

The easiest, most maintainable, least invasive (and in case you don't need/want the threads to interact, often also the best performing) way to introduce parallelism is to do it as coarse-grained as possible. I.e., simply run multiple self-contained unit-test programs in parallel, rather than trying to parallelise the execution of unit tests within a single program. Since the tests by definition are supposed to be independent, introducing dependencies by adding threading is a step backwards in several ways (performance, maintainability, debugability, ..).

This, of course, requires the availability of as sufficient number of test programs to actually execute in parallel (which definitely can be a problem if you're used to putting all unit tests into a single program). But if you do have that amount of independent work available, you'll generally get much better performance because you won't need any locking anywhere (even the overhead of the OS starting a program can easily be lower than the one caused by work queue maintenance and contention over a bunch of locks in a program, especially if you have very fine-grained work units such as a single function call to evaluate and test). You'll also have much better productivity because you'll have to spend much less time on writing, optimising and debugging the actual testing framework (and on taking into account all possible threading hazards when extending it).

An extreme example of this is the FPC testsuite: on an IBM Power8 system with 10 cores that each can run 16 threads, the entire testsuite of 4000-5000 test programs finishes in about 30 seconds (including running fpmake install on the rtl and packages directories). The parallelism, in this case, is handled via make -j 160, but it could just as well be a Pascal program that fires up threads and launches programs from within those threads. Every parallel invocation of the test runner (dotest) executes 10 to 20 tests sequentially and the results are all written to separate files. At the very end, all of those text files are concatenated in the right order, so the output is deterministic.

Most OS kernels have a lot of smp optimisations, probably way better than any of us could every write (or even would want to bother to write and maintain for something like a testing harness). I very much doubt a multi-threaded implementation of the dotest program, apart from the work this would entail, could outperform the current approach without several weeks or even months of work.

Jonas

Absolutely

The above ones: clearly an improvement, though I fear some will need changes in the fpcunit core code.

And: 'double click and let IDE open...': yes please !! :)

Michael.

While all what you say is correct, writing a test program per test is simply not an option. For e.g. database tests the performance hit alone would be prohibitive.

Michael.

Hi,

Many thanks for your comments.

Jonas Maebe wrote

I.e., simply run multiple self-contained unit-test programs in parallel,

Based on the projects I work on that has unit testing, that doesn't seem like a feasible option. It would require a lot of work, not only splitting the test suites, but also compiling all of them etc. Plus the fact that Lazarus IDE still doesn't support Project Groups, it makes working with multiple related projects very tedious.

tests sequentially and the results are all written to separate files. At the very end, all of those text files are concatenated in the right order, so the output is deterministic.

This is actually what I had in mind for a single test program. Test suites that run in parallel will return a TTestResult object. The main program will then collect all those results at the end of the complete run and present it in some good manner. Sounds simple enough. ;-)

DUnit2 and FPTest (my Free Pascal fork of DUnit2) already support an interface for repeating tests. Creating parallel tests could be handled in a similar fashion. I'll have to review the FPCUnit code a bit more to see what would be needed.

Regards,
- Graeme -