Hello,

The Editor of Toolbox Magazine has allowed me to put an article about FPCUnit online.

FPCUnit is (similarly to JUnit, DUnit) an Object Pascal testing framework. It comes standard with your Free Pascal/Lazarus installation. It was written by Dean Zobec, and Vincent Snijders made the integration in Lazarus.

The article covers basic usage, just as well as usage and installation in Lazarus. It can be considered a tutorial (till a proper tutorial appears)

Here is the URL:
http://www.freepascal.org/docs-html/fpcunit.pdf

Enjoy!

Michael.

Thank you very much! I hesitated to start with FPCUnit because of lacking time for wading through source and docs, but now I'll start testing. :)

Marc

Since I don't exactly know what test frameworks are, even after reading about them for the past few years, I'm going to ask some risky questions. This is not a flame thrower attempt at the test framework advocates, I'm just trying to understand what exactly they do, from a "newbie to test frameworks" perspective.

Do test frameworks
-cause you to spend lots of time writing test frameworks instead of program code? -even apply to languages with compilers and strong typing? -cause you to write your programs to conform to the test framework, instead of the program framework?
-have any actual statistics of success?
-have popularity statistics that prove they are popular (in the Delphi community) as stated in the article?
-have any real world examples of how they helped personX fix problemY in significantly less time?

Hoping the answers are no, yes, no yes, yes, yes.

Too much to explain here imo. I think this article will put some light on what testing is good for and how it works:

http://junit.sourceforge.net/doc/testinfected/testing.htm

There is more starters documentation at the JUnit site if you like reading...

HTH,
Marc

i started to write a rather lengthy reply to this note, and realized i was starting to rant.

so i'm gonna shorten my reply up quite a bit.

i'm a java web app programmer by trade. i use Pascal for fun stuff.

up until a couple of years ago, i went through a series of companies where i couldn't escape the _nightmare_ of unit testing mantra, being forced down the development teams throat ... _usually_ with a whole helping of other ridicoulous development "paradigms" freshly read by the manager right out of "CIO" magazine.

i will offer this opinion ...

i think generally, and especially the way they've been implemented in the past where i've worked, they (unit tests) have been a huge waste of time and a soul sucking experience for the development team.

my last 2 years, i've worked with ONE other senior developer, for a smallish company, with a web app in production, having done ~6 releases in the last 1.5 years since we went 1.0 with the product.

in terms of scale, i would say it is just a smidgen smaller than the web application at my last job, where there were 12 developers, 3 managers and a "technical lead".

let me express that in a formula :

webapp1 + 2 developers == 12 developers, 3 managers, lead + webapp2

the current application i work on is light years ahead of the other one i worked on in terms of stability and quality

what is the difference ?

well, currently, me and my peer don't have several middle managers dictating our every move.

we don't use RAD "tools"

we use simple, best of breed open source technologies where we can

we write code to solve problems when we think the investment is a better choice for us in the long run, rather than buying some "kitchen sink" solution from some vendor whose fancy sales guy took the boss out to lunch one day.

we don't spend time writing unit tests.

my $0.02 ...

i'm a java web app programmer by trade. i use Pascal for fun stuff.

Well I hope you try some of the Pascal web development projects too, and not JUST java for all the web apps... ;)

I demand at least some fun pascal web apps out of you, minimum!

Your story is an example of how a 'good' technology can be ruined by applying it too much or in an inappropriate way. Alas, this happens too often when people who actually don't have a clue start dicating the rules.

But that doesn't mean this technology is bad. FPC uses a testsuite (not based on FPCunit) to test the compiler. And luckily so, because this has helped the compiler team to eliminate bugs on many an occasion.

As in all things: use the right tools for the right job, and with moderation.

I personally wouldn't recommend using FPCunit in all cases. The purpose of the article is to show that it is present in FPC for those who want to use such a technology.

Michael.

That depends entirely on you. You should not write test frameworks for visual aspects of your application, it will take a lot of time. But any nonvisual code can be tested with little code.

-even apply to languages with compilers and strong typing?

Yes, of course.

-cause you to write your programs to conform to the test framework, instead of the program framework?

No.

-have any actual statistics of success?

Yes.

-have popularity statistics that prove they are popular (in the Delphi community) as stated in the article?

I'm not sure what you mean by this question ?

-have any real world examples of how they helped personX fix problemY in significantly less time?

I personally don't, but as far as I know, for instance InstantObjects use unit testing and in particular FPCUnit extensively to test whether their objects code functions correctly.

Michael.

Without the regression tests of our testsuite, FPC would have much less quality than it has now. Also code reviewing is something which helps a lot for complex stuff.

L505 ha scritto:

I did not understand the full benefits of unit tests till I've tried them. And it took a lot to get accustomed to them.

Do test frameworks -cause you to spend lots of time writing test frameworks instead of program code?

The function of the xUnit testing framework is to let you write tests faster and with less effort. It's a matter of personal development habits and preferences of course, but the greatest benefit of unit testing I've experienced was the reduction of time spent with the debugger (I still don't know how to use it effectively with Free Pascal and in Lazarus I usually keep it disabled :) A failing test usually leads me directly to the problem. The safety net provided by the tests, written at the same time as I write the code gives me courage to do the refactoring that I find usefull when I review the code to improve the design, or the changes that I need to apply when I do some profiling. It assures me I've not introduced collateral damage.

-even apply to languages with compilers and strong typing?

I has nothing to do with strong typing IMHO, I love strong typing but it does not solve all my problems. I'm not a perfect programmer, I'm quite successful in making mistakes and I need to be sure that a piece of code does exactly what I thought it should do, without making assumptions and discover the problem later, when the code has been used in other higher level functions or frameworks

-cause you to write your programs to conform to the test framework, instead of the program framework?

Not all kind of code can benefit from unit testing, it's no use in my opinion to try to test the GUI code, and it's difficult to test some parts of web or db applications.
But in my case, when I'm doing unit testing of non visual code in more complex frameworks and I find a class that is difficult to test in isolation, I've noticed that in most occasions this is due to some flaw in my design, often the classes proved to be too tightly coupled or had a wrong interface. So, from my experience, writing tests has the side effect to improve the design of my code.

-have any actual statistics of success?

The Free Pascal Compiler is the best example of the success of testing, have you seen how many test have our compiler gods put in place in their testing suite? It's not fpcunit but the effect is the same. It was the first think I've noticed when I checked out the code from cvs the first time. Then I've noticed that each time a bug was found it was pinned down in a test to prevent it to emerge again, and I've begin to learn some other good tricks from them.

-have popularity statistics that prove they are popular (in the Delphi community) as stated in the article?

I'm quite sure that the folks at Borland don't do a lot of unit testing ;) after all DUnit and NUnit are open source products :), but yes, in the open source Delphi community they are quite popular, see the Jedi JCL project that uses DUnit for example:
http://cvs.sourceforge.net/viewcvs.py/jcl/qa/automated/dunit/units/ The developers that took over the InstantObjects framework development are now successfully using FPCUnit to begin to understand the complex IO framework, pinpointing the code to be able to do some changes safely and maybe to pave the way for the Lazarus port. (See
http://cvs.sourceforge.net/viewcvs.py/instantobjects/Source/Tests/)

The Mono project is successfully using NUnit too.

-have any real world examples of how they helped personX fix problemY in significantly less time?

It's not about fixing a problem in my experience. It's about maintaining the same level of productivity when the system grows. To be able to do fearless refactoring in the phase of reviewing the code when discovering some "bad smells", to be able to improve the design. I find it difficult to begin some large refactoring without a solid base of tests in place. When I publish some code I prefer to thoroughly test it before and when I submit a bug report I found practical to provide a failing test that isolates it. Unit Tests are not a panacea to all the problems, but they make my work easier. It helped me to drastically reduce the time I've spent debugging. It took some time and effort to learn to write good tests but for me it was a good investment. By doing the tests as I write the code I've learned to spend less time in developing good tests (it's like mowing a lawn, when the grass is small it's easier). It keeps my productivity stable even when the code is getting complex and it even helps me to isolate a reported bug in my code. Now I'm really having fun in coding.

Hoping the answers are no, yes, no yes, yes, yes.

The answer was quite more articulated and subjective :) there's no use in saying that unit testing is good for everyone. If the developer enjoys doing it, as I do, it's a good thing in my opinion. But I'm convinced that there's not one true way of writing good quality software, every one has to find the one best suited to his needs.

Dean

debugger (I still don't know how to use it effectively with Free Pascal and in Lazarus I usually keep it disabled :) A failing test usually

Same here, I may be speaking from small project experience though, rather than huge/large level project experience. I think I may use the debugger more often on large projects in the future.

But in my case, when I'm doing unit testing of non visual code in more complex frameworks and I find a class that is difficult to test in

Right, after reading some more about unit tests, I figure testing will be useful more for stdout/piping/cgi/compiler/HaveToBeCorrect style applications moreso than in GUI style apps. And I do build those type of apps.

in my design, often the classes proved to be too tightly coupled or had a wrong interface. So, from my experience, writing tests has the side effect to improve the design of my code.

I've written about stuff like this before too, such as "Problems with designing software around forms" which many delphi programmers do. So the ideas behind unit tests would agree with some of my thinking. I think many people tightly couple their applications around Form1 too much, mainly in larger applications. In smaller applications, making a form1 style application is quick, convenient, and more beneficial than going from scratch with your own thought out design.

The Free Pascal Compiler is the best example of the success of testing,

A compiler of course needs to be tested... :) which i think is bit biased bringing it up.. but I see the purpose of tests more now that you mention the compiler. So biased examples are okay, to bring about a point. I bring up biases all the time.

Compilers need to be tested vigorously for correctness - moreso than any other peice of software.. since it creates the software, and that software better well be correct when fed source code that is correct!

I personally wouldn't recommend using FPCunit in all cases. The purpose of the article is to show that it is present in FPC for those who want to use such a technology.

Exactly, and I think this discussion has helped me see where I can start experimenting it first, and what cases it would apply more greatly to.

I found the article and the discussion on this list very helpful. Thanks for writing it and making it available.

These are probably stupid questions, but all the examples I've read about use silly tests like 1+1=2 or checking that a list is empty. I'm having trouble seeing how I would write tests for my code.

I have an ugly little hack that reads some logged data and counts certain things. The code is fragile so I'd love to add unit testing... But the code is mostly procedures for reading and parsing the log data. Would the unit tests create fake input and compare it to known output? (How do I fake reading data from disk?) For that matter, how do I test the data-reading procedures? (Write fake data and then see if the routine reads it correctly?)

Or, I've been messing around with SQLite... to add unit tests for this code, would I need to create fake databases?

And what about the GUI code? How can that be tested?

Thanks again for the article and stimulating discussion.

-Alan

Without the regression tests of our testsuite, FPC would have much less quality than it has now. Also code reviewing is something which helps a lot for complex stuff.

Some weeks ago I've discussed unit testing with a friend (who is working in a larger team), as I had a light suspiciosly view on this topic. For _complex_ systems with _clear_ structure and interfaces, they absolutely make sense and are productive. But the entry level where they are worth the effort is quite high. I guess that even for average industrial automation applications (where I have used FPC in practice), the usage of unit tests is a little bit exaggerated and won't help much. As Michael stated: It depends on the usage case... And it's something which I would call in German: "Da wird mal wieder eine neue Sau durchs Dorf getrieben." (Probably not translateable to English, sorry). It's a buzzword, managers like it, but there are cases where they really make sense. But by far not in all cases.

- Sebastian

Those were some of my suspicions too, since on for example the C2 wiki, I've read a lot about these "tests" but I never see any or many actual code or real world examples on the C2 wiki. Even if these tests are extremely useful in several areas, the inventors of the "test cases" would be better off explaining the tests with real world examples, instead of all the tricky English and tricky wording.

So in other words, "tests" could be very useful, but the suspicious human needs to see the usefulness demonstrated first. The suspicious human also needs to clarify whether the word "test" really means anything, since a test is a vague word (kind of like the phrase DotNet). You just can't believe something because it was in a book written by some celebrity programmer. We have seen the compiler can use "tests" in order to test the validity of the compiler. Are these "unit tests" or just "tests". It all seems vague to me. A compiler is very specific and a special program which obviously needs to be "tested". As for whether the FPC compiler tests have anything to do with "extreme programming", "test cases" and "unit tests" I'm not sure. It may be just a vague term for "general tests to do on your code", or it may actually be something more than just that. It's their job to make it more clear.. or if not, maybe a "unit test consulting business" will call you in for an appointment on Tuesday.

Alan Mead ha scritto:

Take a look at some real world tests to see how they were written and then try to write yours to make practice. When writing test you should learn to identify the border conditions (strange cases that could give problems to your code), you know, bugs generally live in the edges :) like wrong input values, badly formatted data, values in excess of any reasonable expectation, empty of missing values (null, 0, ""), unordered values passed in input where the routine expects ordered items ... Ask yourself some questions as Does the value conform to my expected format? Is the value within a reasonable range of values? Is something happening in the right order in time? Are there exactly enough parameters? Some methods can be tested by applying their inverse: like the result of y=sqrt(x) could be tested applying y^2 to see if it equals to x. To see if an item was deleted from a list you perform a search after the deletion. For some examples of more complex unit tests with fpcunit you can look at the tests we wrote for the EOS framework: a sort of simple xml object persistence framework that automatically saves the published properties of objects and complex structures of objects to xml and automatically recreates the objects from the xml text (it was used to transport those objects over http from the server to the client and vice versa)
Here you can find the code:
http://cvs.sourceforge.net/viewcvs.py/camelos/EOS/fpcEOS/ and here are the related tests:
http://cvs.sourceforge.net/viewcvs.py/camelos/EOS/fpcEOS/tests/

Yes exactly, obviously you'll also write strange faked data to see how your procedures and functions respond. It is much more difficult to write tests when all the code is already written, so begin with writing tests for low level simple methods first and when they pass you can begin to test higher level methods. This is the natural way of proceeding when you write tests at the same time that you write the code . You'll see that the tests you'll find here
http://cvs.sourceforge.net/viewcvs.py/camelos/EOS/fpcEOS/tests/ are testing functionality very similar to your case hope you'll find them useful. Some where even written over legacy code like in your case. If you have specific questions you can even write me in private mail, I'll try to help you if I can (here we would be off-topic).

This are two of the cases where testing is most difficult. Some suggest writing mock objects that mock the real behaviour
http://www.mockobjects.com . But is generally difficult to implement in case of databases.
For the GUI I'm against testing it using unit testing, it's a waste of time.

If you should need a book to learn more about the testing techniques from a very practical point of view, with many examples, I suggest this:

Pragmatic Unit Testing
in Java with JUnit
by Andy Hunt and Dave Thomas

Dean

05-10-11 23.52, skrev L505 följande:

programmer. We have seen the compiler can use "tests" in order to test the validity of the compiler. Are these "unit tests" or just "tests". It all seems vague to me. A compiler is

A unit test should test a small module of a program, like a pascal unit. Since most of the tests in the FPC compiler test suite do test the compiler as a whole, they are not unit tests, rather some kind of system tests (I think). Which does not make them less useful.

Olle