Hi list,

I've been working on a listener for FPCUnit that writes test results to a database.

I intend to use it to run e.g. database tests on my Jenkins CI server whenever a new build is done. Therefore, the db is quite normalized in order to limit used space.
I've included a flattening view that denormalizes the output into a grid suitable for human consumption.

The unit:
- stores results in a database, specified by db
type/server/name/user/password in testdbwriter.ini
- the testing program/test runner can override these connection settings
- can create a Firebird embedded database and use it if Firebird driver+empty hostname is chosen, or as a fallback if no db info is specified at all
- stores CPU+OS model (in the format FPC detects them) by default into the test run table; can be overridden (e.g. to not store them)
- saves application name (can be overridden) and allows adding a comment field into the test run table
- optionally stores a revision id/application version field per test run. Useful for doing regression testing
- stores test suite names and test results, as well as any exception/error messages (e.g. reason for test failure).
- can run in principle against any sqldb RDBMS as it uses simple SQL

SQL included for PostgreSQL and Firebird server to create the db; should be easily adaptable to other environments.

Simple example included in examples directory.

I've also included a dbtestframework workalike (dbtests2db) with units that runs the FPC 2.6.1 r22717 database test framework; only plug in database.ini for the database test connection and testdbwriter.ini for the storage option (can be the same server, but different db) and go.

I have tested it on Firebird, Firebird embedded and PostgreSQL.

Test results, bug reports, improvement suggestions and patches are all welcome.

Next up for me is to integrate this into my Jenkins server and see how it works. Based on this heavier testing, some improvements may be apparent.

Also, more queries/views will need to be defined in order to get statistics and regression tests.

Finally, while the database is ready for multi-level/nested test suites, the code is not yet. It simply uses the lowest test suite name as the name. While that is sufficient for the database tests, it will need to be improved as well.

Download:
https://bitbucket.org/reiniero/testdbwriter
download tab

Thanks very much,
Reinier

Reinier Olislagers wrote

I've been working on a listener for FPCUnit that writes test results to a database.

<snip>

Just got a working Jenkins setup documented on Debian x86 Linux [1].

Compiles FPC 2.6.1 and runs database tests against PostgreSQL and Firebird using testdbwriter [2] mentioned above, writing test results - including svn revision, platform and database tested - to another Firebird database.
It is set up to track SVN every 20 minutes and compile if there are changes.

Next up: adding more dbs to be tested (at least mysql), looking at removing the use of fpcup in the Jenkins build, as well as adding queries to the testdbwriter to get e.g. failed tests, regressions (including since which build they happened).

I might also have a look to see whether the fpcunit XML (the "second version" that Graeme participated in) is accepted by Jenkins...

Currently fpc trunk fails to compile with stacktrace errors but that's probably being worked on (I saw on the dev list it was fixed for SPARC)

[1] http://wiki.lazarus.freepascal.org/Jenkins
[2] https://bitbucket.org/reiniero/testdbwriter

Reinier Olislagers wrote

[1] http://wiki.lazarus.freepascal.org/Jenkins

I've only heard about Jenkins since your recent post. I only read the overview on the wiki, but will investigate this further.

I thought I would also mention something we use in our company.

We use Git repositories, and have custom git hook scripts to create release builds for our software.

eg:
- we develop and push commits as normal
- one day we decide to create a new build. We then create a tag "build" and push it to our server
- the git hook script detects the "build" tag, then does a checkout of that revision and starts creating a new release of our software.
- The release can even be packaged as a zip archive or as a setup.sh or setup.exe These setups or zip archives are placed in a location for easy download. eg: a SMB share or ftp location.
- Once the release is done, the "build" tag is removed on the server.

We have found this very convenient and useful. Anybody can trigger a new release and get easy access to the final setup/archive.

Regards,
- Graeme -

Reinier Olislagers wrote

More related to my initial post: what do you do with test results (e.g. tiopf tests, your own software tests)? Do you store them in a database?

I have never had the need for storing unit test results for long term. In the tiOPF project, only the latest test result is viewable with a web browser (I could easily add a timestamp to the filename though).

http://opensoft.homeip.net/tiopf/fpcunit/index64.html http://opensoft.homeip.net/tiopf/fpcunit/index32.html

That is the processed results - XML results converted to HTML using a small XSL file I wrote. At the moment the above links are for 32-bit and 64-bit FPC Linux test runs. I plan to add more such links for Windows and FreeBSD test results as well.

The text results (summary information with errors or failures listed) are posted to the tiopf.dailybuilds newsgroup. Any posts in that newsgroup expire after 7 days. So there we can view a history of unit test results.

news://news.tiopf.com

or view the newsgroup via the web interface:

http://opensoft.homeip.net/webnews/

Being able to store fpcunit, dunit, dunit2, even junit and other test results in that db may be a handy thing...

Side note:
DUnit2 (as on SourceForge) is becoming a Delphi only project. Originally DUnit2 was developed inside the tiOPF repository, but since the passing of Peter McNabb, the project was forked. The Delphi version of DUnit2 is now on SourceForge, and the Free Pascal version (now called FPTest [Free Pascal Testing Framework] to prevent confusion) lives in GitHub.

http://wiki.freepascal.org/FPTest

FPTest removes things like Kylix, Delphi.NET etc support, plus lot of other code cleanup. New GUI Test Runners are being developed as well.

Regards,
- Graeme -

More related to my initial post: what do you do with test results (e.g. tiopf tests, your own software tests)?
Do you store them in a database?

Would the schema in
https://bitbucket.org/reiniero/testdbwriter/src
testdbwriter_firebird.sql
or
testdbwriter_postgresql.sql
fit?

The 2 schemas are basically the same, though I'm developing regression and statistics views in the Firebird version before porting them to PostgreSQL.

Any recommendations?

Being able to store fpcunit, dunit, dunit2, even junit and other test results in that db may be a handy thing...

Thanks,
Reinier

Ok, thanks.

I know ;)