Testing FPC: what changed
Revision 1 to 2. Removed lines have a minus sign; added lines have a plus sign.
| Old | New | Line | |
|---|---|---|---|
| 1 | 1 | Development of a compiler and a corresponding runtime library requires an extensive amount of tests to ensure that the addition of a new feature or the fixing of a bug doesn't break any existing code. In FPC this task is handled by the testsuite located in the *tests* subdirectory of the top level directory ([SVN WebView](http://svn.freepascal.org/cgi-bin/viewvc.cgi/trunk/tests/)). | |
| 2 | 2 | ||
| 3 | 3 | # Overview | |
| 4 | 4 | ||
| 5 | 5 | The testsuite consists of the makefiles and utilities that execute the tests as well as the tests themselves. For this article the makefiles and utilities will be used as is and thus they won't be looked at any further. | |
| 6 | 6 | ||
| 7 | 7 | Detailed information about the testsuite and its parameters can be found in the [ReadMe](http://svn.freepascal.org/cgi-bin/viewvc.cgi/trunk/tests/readme.txt?view=markup). | |
| 8 | 8 | ||
| 9 | 9 | ## Directory Structure | |
| 10 | 10 | ||
| 11 | 11 | The tests themselves are separated into multiple directories: | |
| 12 | 12 | ||
| 13 | 13 | - *test*: systematic tests, usually developed by test driven development | |
| 14 | 14 | - *tbs*: tests derived from non tracker reports or ideas while fixing something requiring successfull compilation and (optionally) run | |
| 15 | 15 | - *tbf*: tests derived from non tracker reports or ideas while fixing something requiring failing compilation | |
| 16 | 16 | - *webtbs*: tests derived from bug tracker bugs requiring successful compilation and (optionally) run | |
| 17 | 17 | - *webtbf*: tests derived from bug tracker bugs requiring failing of compilation | |
| 18 | 18 | ||
| 19 | 19 | Additionally selected *tests* subdirectories of directories in *packages* are searched. These are specified in *\$fpcdir/tests/Makefile.fpc* with the variable *TESTPACKAGESDIRECTDIRS* (the directories are simply separated by spaces). You need to regenerate the Makefile using *fpcmake -Tall* if you change this. | |
| 20 | 20 | ||
| 21 | 21 | Not all files in these directories are eligible for the testsuite however as they need to fullfill two formal criteria: | |
| 22 | 22 | ||
| 23 | 23 | - correct prefix (*t* for *test*, *tb* for *tbs* and *tbf*, *tw* for *webtbs* and *webtbf*) | |
| 24 | 24 | - correct file extension (only '.pp' is accepted) | |
| 25 | 25 | ||
| 26 | 26 | The remaining name of the test is freely chosen, however there are some general guidelines for this as well: | |
| 27 | 27 | ||
| 28 | 28 | - tests in *test* have some short name describing the feature the tests are testing (e.g. *rhlp* for record helper tests and *genfunc* for generic function tests) | |
| 29 | 29 | - tests in *tbf* and *tbs* have a unique, four digit index | |
| 30 | 30 | - tests in *webtbs* and *webtbf* have their corresponding bug ID from Mantis as name plus eventually a small letter as suffix if there are multiple tests for the same report | |
| 31 | 31 | - units that are used by some test have the prefix *u*, *ub* or *uw* depending on the test directory as well as the name of the first test they are used for (e.g. if both *thlp3.pp* and *thlp7.pp* use a unit the unit is named *uhlp3.pp*) | |
| 32 | 32 | ||
| 33 | 33 | ## Test Results | |
| 34 | 34 | ||
| 35 | 35 | When a test is executed its execution essentially consists of two *stages*, namely *compilation* and *run*. The *run* stage is optional (see Test Options) or not required at all (e.g. if the test should already fail the *compilation* stage). | |
| 36 | 36 | ||
| 37 | 37 | The *compilation* stage can have the following outcomes: | |
| 38 | 38 | ||
| 39 | 39 | - successful compilation | |
| 40 | 40 | - compilation failed with a compiler error | |
| 41 | 41 | - compilation failed with an internal error | |
| 42 | 42 | - compilation failed with another exception | |
| 43 | 43 | ||
| 44 | 44 | If the test should succeed then the first is treated as success, the other three as failures. If a test should fail then the second outcome is treated as a success, the other three are treated as failures. | |
| 45 | 45 | ||
| 46 | 46 | The *run* stage can have the following outcomes: | |
| 47 | 47 | ||
| 48 | 48 | - successful run | |
| 49 | 49 | - failed run | |
| 50 | 50 | ||
| 51 | 51 | The result of the run is determined with the exit code of the test's execution. | |
| 52 | 52 | ||
| 53 | 53 | ## Test Files | |
| 54 | 54 | ||
| 55 | 55 | A test file is either a program, library or unit file. A library and unit test is never run, only compiled while a program is always run unless told otherwise with the *{%NORUN}* option. | |
| 56 | 56 | ||
| 57 | 57 | ## Test Options | |
| 58 | 58 | ||
| 59 | 59 | Various options can be used to influence the behavior of the testsuite. This options are given at the top of the testfile using a Borland style comment that starts with a *%*. The first comment without that marker ends the option parsing (usually that's the mode switch). | |
| 60 | 60 | ||
| 61 | 61 | The following is a non-extensive list of options that are supported: | |
| 62 | 62 | ||
| 63 | 63 | - *NORUN*: don't execute the resulting test program (useful if merely the compilation should be tested) | |
| 64 | 64 | - *SKIP*: don't compile the test at all | |
| 65 | 65 | - *CONFIGFILE*: Specifies a config file in *\$fpcdir/tests/config* that is to be copied to test. This option can take one or two parameters: in the two parameter form the first name is the source filename and the second the destination filename, in the one parameter form source and destination filename are equal. | |
| 66 | 66 | ||
| 67 | 67 | For a complete overview of all possible options, see the [ReadMe](http://svn.freepascal.org/cgi-bin/viewvc.cgi/trunk/tests/readme.txt?view=markup), under the header *Test directives*. | |
| 68 | 68 | ||
| 69 | 69 | # Running the testsuite | |
| 70 | 70 | ||
| 71 | 71 | To run the test suite a complete compilation (compiler, rtl, packages) should have been done beforehand. Then after changing into the *tests* directory you execute the following command: | |
| 72 | 72 | ||
| 73 | - | ||
| 73 | + | ```text | |
| 74 | + | make full TEST_FPC=/path/to/ppc TEST_CPU=<cpu> TEST_OS=<os> TEST_OPT=<options> CHUNKSIZE=50 -j <count> | |
| 75 | + | ``` | |
| 74 | 76 | ||
| 75 | 77 | - *TEST_FPC*: absolute path to the compiler that should be tested (mandatory) | |
| 76 | 78 | - *TEST_CPU*: name of the CPU for which the tests are run (default is the given compiler's default CPU) | |
| 77 | 79 | - *TEST_OS*: name of the OS for which the test should be run (default is the given compiler's default OS) | |
| 78 | 80 | - *TEST_OPT*: options that should be passed to the compiler | |
| 79 | 81 | - *CHUNKSIZE* and *-j \<count\>*: the testsuite can use multiple cpu cores in parallel. It does so by launching *\<count\>* instances of the *dotest* program, each of which will execute *CHUNKSIZE* tests before terminating (after which a new instance of *dotest* will be started to check *CHUNKSIZE* more tests). We split up the test collection in chunks because not all tests take the same amount of time to compile and run. To get optimal throughput (avoiding some cores sitting idle near the end of the testsuite run), choose a smaller CHUNKSIZE when your number of cores increases. The default, *CHUNKSIZE=100*, performs fairly well for up to 4 cores. Don't make *CHUNKSIZE* too small, or the overhead from starting more instances of *dotest* will erase any gains you may get. | |
| 80 | 82 | ||
| 81 | 83 | At the end of the testsuite run there'll be an ordered list of failed tests available in *tests/output/\<cpu\>-\<os\>/faillist* that can be compared with other runs. | |
| 82 | 84 | ||
| 83 | 85 | # Recommended Workflow | |
| 84 | 86 | ||
| 85 | 87 | Before doing any changes: | |
| 86 | 88 | ||
| 87 | 89 | - complete build | |
| 88 | 90 | - testsuite run | |
| 89 | 91 | - save *faillist* file (e.g. to *tests/output/unmodified/faillist*) | |
| 90 | 92 | ||
| 91 | 93 | To test your changes: | |
| 92 | 94 | ||
| 93 | 95 | - complete build (fix any failures that might occur here) | |
| 94 | 96 | - testsuite run | |
| 95 | 97 | - compare new *faillist* with stored *faillist* (due to the tests being ordered a diff or similar is sufficient) | |
| 96 | 98 | - fix failures in compiler/RTL/packages | |
| 97 | 99 | - rinse and repeat | |
| 98 | 100 | ||
| 99 | 101 | --- | |
| 100 | 102 | *From the Free Pascal wiki, page "Testing FPC", as it stood on 2018-06-24 after a change by FTurtle, out of the wiki's dump of 2024-05-26. Copyright stays with the author who wrote it, according to the old wiki's [[Licensing]] page. If you have concerns about this material, [message the admin](/wiki-contact).* | |
| 101 | 103 | Category: FPC development | |
| 102 | 104 | Category: Testing |