Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / fpc-pascal / 2006-11

fpcUnit todo list

18 messages · 6 people ·

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

Graeme Geldenhuys

Dean,

As requested. Here goes...

Let me know what you think of Item 1? If you wanted to start with any of these, Item 1 would be good. :-)

===========================
Item 1
===========================
I have an idea on improving the console runner output. Running the app as show below, it always defaults to xml output, which gives you a strange undesired result when combine with other parameters. What if we change the default output (when no --format has been specified) to just print a summary when the tests are complete.

That way if specify the params '-a -p' we will get

Example 1:

./fpcunitTextM2 -a -p

....................
Number of run tests: 20
Number of errors: 0
Number of failures: 0

And if we specify the params '-a -p --file=results.xml' we will get

Example 2:

./fpcunitTextM2 -a -p --file=results.xml

....................
Number of run tests: 20
Number of errors: 0
Number of failures: 0

The default --format= was applied because the --file= triggered it. So xml output will appear in the results.xml file and the above output to the console window.

And if we specify the params '-a -p --format=plain' we will get

Example 3:

./fpcunitTextM2 -a -p --format=plain

Test: TTestSlideParser.IsTest
Test: TTestSlideParser.CreateSlide
Test: TTestSlideParser.ReferenceNumber

Number of run tests: 20
Number of errors: 0
Number of failures: 0

...because the --format was applied it overrides the -p parameter and just ouputs the --format specified to the console window. No '.' are printed between every 'Test:...' line.

And if we specify the params '-a -p --format=plain --file=results.xml' we will get

Example 4:

./fpcunitTextM2 -a -p --format=plain --file=results.xml

....................
Number of run tests: 20
Number of errors: 0
Number of failures: 0

...because the --file= was specified, the -p parameter is active again (unlike example 3) and outputs the progress and summary to the console and the test results to the results.xml file.

And the last one, if we specify the params '-a' we will get

Example 4:

./fpcunitTextM2 -a

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="results.xsl"?> <TestResults>
<!-- Generated using FPCUnit on 2006-10-26 11:11:18--> <TestListing>
....snip....

As normal. No '.' are printed, and the to default --format= output is applied and printed to the console window. Or should be instead only print the summary?

---------- Current undesired output -----------------------

./fpcunitTextM2 -a -p

....................<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="results.xsl"?> <TestResults>
<!-- Generated using FPCUnit on 2006-10-26 10:48:41--> <TestListing>

...snip....
----------- END ----------------------

===========================
Item 2
===========================

DUnit interface compatability needs updating... I created a few more, just haven't submitted a patch yet.

===========================
Item 3
===========================

In Windows using the GUI Test Runner. If you RMC (Right Mouse Click) in the treeview you get a error: List index (3) out of bounds. This does not occur in Linux.

===========================
Item 4
===========================

Hierarchy view for test listing in XML. The XSLT file needs to be modified to support this. The XSLT files for Summary (used for emails) and Full (used for html pages) reporting needs to be submitted to SubVersion.

I can't remember the others... the list is at home somewhere.

Regards,
- Graeme -

Vincent Snijders
to Graeme Geldenhuys
Graeme Geldenhuys wrote 24 lines
Dean, As requested. Here goes... Let me know what you think of Item 1? If you wanted to start with any of these, Item 1 would be good. :-) =========================== Item 1 =========================== I have an idea on improving the console runner output. Running the app as show below, it always defaults to xml output, which gives you a strange undesired result when combine with other parameters. What if we change the default output (when no --format has been specified) to just print a summary when the tests are complete. That way if specify the params '-a -p' we will get Example 1: > ./fpcunitTextM2 -a -p .................... Number of run tests: 20 Number of errors: 0 Number of failures: 0

IMHO the 'Number of ...' should be enabled with a new -s (and --summary) switch

Show quoted text 8 lines
=========================== Item 4 =========================== Hierarchy view for test listing in XML. The XSLT file needs to be modified to support this. The XSLT files for Summary (used for emails) and Full (used for html pages) reporting needs to be submitted to SubVersion.

First the XML needs to change and then the XSLT files can be adapted, correct?

Vincent

Graeme Geldenhuys
to Vincent Snijders
Vincent Snijders wrote 6 lines
> Number of run tests: 20 > Number of errors: 0 > Number of failures: 0 IMHO the 'Number of ...' should be enabled with a new -s (and --summary) switch

That makes sence, yes.

Show quoted text 7 lines
> Hierarchy view for test listing in XML. The XSLT file needs to be > modified to support this. The XSLT files for Summary (used for > emails) and Full (used for html pages) reporting needs to be submitted > to SubVersion. First the XML needs to change and then the XSLT files can be adapted, correct?

Yes, the XML will change, then the XSLT adapted.

Regards,
- Graeme -

Michael Van Canneyt
to Florian Klaempfl
Florian Klaempfl wrote
First, I would prefer a patch to the fpcunit which makes the unit itself indenpendend from other stuff (xml) so it can be used in the fpc test suite :)

The unit fpcunit is independent of XML ?

The package as a whole is dependent on XML, this is correct.

Michael.

Graeme Geldenhuys
to Florian Klaempfl: First, I would prefer a patch to the fpcunit which makes the unit itself indenpendend…
Florian Klaempfl wrote
First, I would prefer a patch to the fpcunit which makes the unit itself indenpendend from other stuff (xml) so it can be used in the fpc test suite :)

The xml used is the xmlwriter unit - part of FPC. Is this a problem? Test output is optional, so you can chose no output, plain text output, xml output, latex output (not sure if this works) and all the previous to a file. The fpcunit.pp doesn't contain any reference to the testreport.pp unit...

Maybe if you could explain 'independant' a bit better, I would get a better understanding of you requirement. We could change the output formats to register with fpcUnit, by creating a OutputFactory, and as soon as any output format unit is included (in uses clause) in a project (as in GUI Runner or Console Runner), it registers with the OutputFactory.

I use this a lot in my projects and in tiOPF. Examples of this is different compression algorithms used by a tiCompress class used in tiOPF.

Regards,
- Graeme -

Jonas Maebe
to Graeme Geldenhuys
Graeme Geldenhuys wrote 7 lines
On 09/11/06, Florian Klaempfl wrote: > First, I would prefer a patch to the fpcunit which makes the unit > itself > indenpendend from other stuff (xml) so it can be used in the fpc > test suite :) The xml used is the xmlwriter unit - part of FPC. Is this a problem?

The problem is that the xml stuff must be built before you can run the testsuite with fpcunit tests included (since the fpcunit package depends on the xml stuff). This means that before you can start running the testsuite, your compiler has to be stable and complete enough to compile packages and fcl. That's not reasonable.

Jonas

Graeme Geldenhuys
to Jonas Maebe
Jonas Maebe wrote 7 lines
> The xml used is the xmlwriter unit - part of FPC. Is this a problem? The problem is that the xml stuff must be built before you can run the testsuite with fpcunit tests included (since the fpcunit package depends on the xml stuff). This means that before you can start running the testsuite, your compiler has to be stable and complete enough to compile packages and fcl. That's not reasonable.

Ok, so if the OutputFactory idea is also not going to work. Why not try something completely different. What if every test suite compiled to a .dll or .so file - nothing more. The fpcUnit executable compiled previously with a stable compiler and includes all the output format are then run during the nightly builds.

The fpcUnit executable can parse and load the .dll or .so test suites and execute them (run their contained test cases). The nightly compiler run then only needs to compile the test suites to .dll or .so files.

Well something like that anyway... :-) Would something like this be possible?

Regards,
- Graeme -

Jonas Maebe
to Graeme Geldenhuys
Graeme Geldenhuys wrote 5 lines
Ok, so if the OutputFactory idea is also not going to work. Why not try something completely different. What if every test suite compiled to a .dll or .so file - nothing more. The fpcUnit executable compiled previously with a stable compiler and includes all the output format are then run during the nightly builds.

There may not be a previous stable compiler (there isn't when you just start porting). Also, the target may be an embedded platform without any dynamic library loading capability.

Jonas

Michael Van Canneyt
to Jonas Maebe: The problem is that the xml stuff must be built before you can run the testsuite with…
Jonas Maebe wrote 15 lines
On 9 nov 2006, at 11:47, Graeme Geldenhuys wrote: >On 09/11/06, Florian Klaempfl wrote: > >First, I would prefer a patch to the fpcunit which makes the unit itself > >indenpendend from other stuff (xml) so it can be used in the fpc test suite > >:) > >The xml used is the xmlwriter unit - part of FPC. Is this a problem? The problem is that the xml stuff must be built before you can run the testsuite with fpcunit tests included (since the fpcunit package depends on the xml stuff). This means that before you can start running the testsuite, your compiler has to be stable and complete enough to compile packages and fcl. That's not reasonable.

The easiest thing to do is to include fpcunit itself in fcl/inc.

All the units around it can be done in fcl/fpcunit, which is built at a later stage.

Michael.

Unknown author
to Jonas Maebe: The problem is that the xml stuff must be built before you can run the testsuite with…
Show quoted text 13 lines
> On 09/11/06, Florian Klaempfl wrote: >> First, I would prefer a patch to the fpcunit which makes the unit >> itself >> indenpendend from other stuff (xml) so it can be used in the fpc >> test suite :) > > The xml used is the xmlwriter unit - part of FPC. Is this a problem? The problem is that the xml stuff must be built before you can run the testsuite with fpcunit tests included (since the fpcunit package depends on the xml stuff). This means that before you can start running the testsuite, your compiler has to be stable and complete enough to compile packages and fcl. That's not reasonable.

Wait a minute.... But you need a stable enough compiler to compile the testsuite right? Hmm either you're inconsistent or you plan to test only the RTL and compiler itself, am I right?

Darius

Graeme Geldenhuys
to Unknown author
Wait a minute.... But you need a stable enough compiler to compile the testsuite right?

That's what I wast trying to get at... You need a compiler already. As to the order of how things are compiler and tested (RTL, FCL, etc) I have no idea how that currently works.

Graeme.

Michael Van Canneyt
to Unknown author: Wait a minute.... But you need a stable enough compiler to compile the testsuite right?…
Show quoted text 17 lines
>> On 09/11/06, Florian Klaempfl wrote: >>> First, I would prefer a patch to the fpcunit which makes the unit >>> itself >>> indenpendend from other stuff (xml) so it can be used in the fpc >>> test suite :) >> >> The xml used is the xmlwriter unit - part of FPC. Is this a problem? > > The problem is that the xml stuff must be built before you can run > the testsuite with fpcunit tests included (since the fpcunit package > depends on the xml stuff). This means that before you can start > running the testsuite, your compiler has to be stable and complete > enough to compile packages and fcl. That's not reasonable. Wait a minute.... But you need a stable enough compiler to compile the testsuite right? Hmm either you're inconsistent or you plan to test only the RTL and compiler itself, am I right?

That is what the current testsuite does: test the compiler and (parts of) RTL.

But if the compiler has problems compiling classes and/or handling exceptions, you can forget the whole fpcunit anyway.

Michael.

Jonas Maebe
to Unknown author: Wait a minute.... But you need a stable enough compiler to compile the testsuite right?…
Wait a minute.... But you need a stable enough compiler to compile the testsuite right?

You need a lot more compiler functionality to compile the fcl than to compile individual tests (not to mention that a problem which occurs when compiling a small test is much easier to debug than one which occurs when compiling a large fcl or packages unit).

Hmm either you're inconsistent or you plan to test only the RTL and compiler itself, am I right?

Initially (or when things are really messed up), that's the case yes.

Jonas

Jonas Maebe

Begin forwarded message:

Show quoted text 67 lines
From: Dean Zobec Date: 9 november 2006 19:45:56 GMT+01:00 To: Florian Klaempfl , Michael Van Canneyt Cc: Graeme Geldenhuys , Subject: Re: [fpc-pascal] fpcUnit todo list I'm unable to post in the mailing list this evening, I constantly get an error, so I'm sending you a private mail. Excuse me for this. > Florian Klaempfl wrote: > First, I would prefer a patch to the fpcunit which makes the unit > itself > indenpendend from other stuff (xml) so it can be used in the fpc > test suite > :) This is my opinion too. fpcunit.pp is independent from xml, you can build a simple runner for the tests that does not rely on other stuff. But fpcunit.pp depends on classes and sysutils, the whole framework is built on exceptions and classes, and these are not available in early ports to new platforms. So fpcunit cannot be used as a regression testsuite for the compiler and low level rtl stuff. Btw, the fpc testsuite is very advanced in it's simplicity and low dependencies and it's probably the main thing that made fpc great. The first thing that made me to begin using fpc (well before I started working on fpcunit) was the existence of the testsuite covering all aspects of the compiler. I trust fpc more than Delphi because of this :) Of course to test the fcl and maybe some high level rtl classes fpcunit is very handy, it simplifies even more the writing of powerful tests for the fcl classes, some are using it as a Test Driven Development tool, others may need it to refactor existing fcl classes, some would use it as simple regression tests, some even for functional (acceptance) tests (Vincent has a lot of fantasy for finding new uses for it, e.g. he uses it to assure that the Lazarus examples compile :) . So I agree with Florian and Michael, I would like to integrate as much as possible fpcunit in the existing fpc testsuite environment in addition to the existing fpc testsuite. In my opinion it's a pity that the fpcunit tests built by the programmers that use it to test some new fcl classes are sitting on the programmer's computer instead of being part of the fpc official testsuite and run each night against the problems that may arise with changes applied each day to the fcl. If output format of the results is a concern for some programmers my best bet would be to use simple latex as a reporting tool, so the results can be easily converted in pdf and html (much more powerful imo then xml), thus avoiding any dependency on xml in the test runner. > The easiest thing to do is to include fpcunit itself in fcl/inc. > All the units around it can be done in fcl/fpcunit, which is built at > a later stage. > > Michael. I agree with this. Dean
Florian Klaempfl
to Jonas Maebe
Show quoted text 7 lines
>> The easiest thing to do is to include fpcunit itself in fcl/inc. > >> All the units around it can be done in fcl/fpcunit, which is built at >> a later stage. >> >> Michael. > I agree with this.

This arises a fundamental question: should the tests for all package (non rtl/compiler!) go into fpc/tests/<package> or should they be in fpc/<package>/test and being executed by make tests in fpc/? At least it should be possible to run the package specific tests without running the whole testsuite.

Michael Van Canneyt
to Florian Klaempfl
Florian Klaempfl wrote 12 lines
Jonas Maebe schrieb: > > > The easiest thing to do is to include fpcunit itself in fcl/inc. > > > > > All the units around it can be done in fcl/fpcunit, which is built at > > > a later stage. > > > > > > Michael. > > I agree with this. This arises a fundamental question: should the tests for all package (non rtl/compiler!) go into fpc/tests/<package> or should they be in fpc/<package>/test and being executed by make tests in fpc/? At least it

It must be fpc/package/tests.

Peter and I decided to build support in the packager for this, so you can

- download
- compile
- test
- install

a package...

Michael.

Jonas Maebe

Begin forwarded message:

Earlier message 52 lines
From: Dean Zobec Date: Fri 10 Nov 2006 00:04:44 GMT+01:00 To: Florian Klaempfl , Michael Van Canneyt , Jonas Maebe Subject: Re: Fwd: [fpc-pascal] fpcUnit todo list Michael Van Canneyt ha scritto: > > On Thu, 9 Nov 2006, Florian Klaempfl wrote: > >> Jonas Maebe schrieb: >>>>> The easiest thing to do is to include fpcunit itself in fcl/inc. >>>>> All the units around it can be done in fcl/fpcunit, which is >>>>> built at >>>>> a later stage. >>>>> >>>>> Michael. >>>> I agree with this. >> This arises a fundamental question: should the tests for all >> package (non >> rtl/compiler!) go into fpc/tests/<package> or should they be in >> fpc/<package>/test and being executed by make tests in fpc/? At >> least it > > It must be fpc/package/tests. > > Peter and I decided to build support in the packager for this, so > you can > > - download > - compile > - test > - install > > a package... > > Michael. Good idea ! :) I agree, the first thought I had was to put them in fpc/<package>/tests directory, they should be somehow close to the production code. The support in the packager is absolutely great! But it would be also useful that the main build process would be able to build all the tests in a single runner program and run think that even with a very large number of tests the suite of all unit tests could be built and run in less than a minute considering that is a single program (of course a configuration file could be added to be able to skip some tests that take more time to run). Dean P.s Michael, did you receive my mail with the attached packages tests you've asked for?