Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / fpc-devel / 2025-06

Speed of TTestCase

9 messages · 4 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

Martin Frb

I have/had several tests that took very long to run.

I have no figured out that between 50% and 80% of the time, went into concatenating/formatting the message for "AssertEqual" (and related). Yes, some of my tests now run in less than 20% of the original time.

That is
- Any concatenation my code does before calling AssertEqual
- The call to ComparisonMsg in AssertEqual

So I started overloading AssertEqual

class procedure TFastTestCase.AssertEquals(const AMessage: string;
Expected, Actual: integer);
begin
   if Expected = Actual then exit;
   inherited AssertEquals(AMessage, Expected, Actual);
end;
class procedure TFastTestCase.AssertEquals(const AMessage: string;
   const AFormatArgs: array of const; Expected, Actual: integer);
begin
   if Expected = Actual then exit;
   inherited AssertEquals(Format(AMessage, AFormatArgs), Expected, Actual);
end;

* Yes, I am aware that skips counting the amount of calls. (needs "Inc(AssertCount)" in case of early exit)
* The 2nd version eliminates any call to format in my own code.

I wonder if it is worth adding such changes to the official testcase.

------------------

Or maybe it is just the way I write tests?

E.g. I have a test, that computes lists, it then does

   for i := 0 to count do
     AssertEquals(format('...', [...]), AExp[i], ATestRes[i]);

Additionally those lists are tested in a loop, iterating through big amounts of control parameters for each test-list to be generated.

So I do end up with millions and millions of calls to AssertEqual. And the same amount of calls to Format and ComparisonMsg, doing string processing.

Some of the time savings happen (and are significant) even when the test runs WITHOUT heaptrc.
But, using heaptrc in tests, is something that I commonly do, and string processing often involves mem re-allocs.

Reply by email

Michael Van Canneyt
to Martin Frb
Martin Frb wrote 32 lines
I have/had several tests that took very long to run. I have no figured out that between 50% and 80% of the time, went into concatenating/formatting the message for "AssertEqual" (and related). Yes, some of my tests now run in less than 20% of the original time. That is - Any concatenation my code does before calling AssertEqual - The call to ComparisonMsg in AssertEqual So I started overloading AssertEqual class procedure TFastTestCase.AssertEquals(const AMessage: string; Expected, Actual: integer); begin   if Expected = Actual then exit;   inherited AssertEquals(AMessage, Expected, Actual); end; class procedure TFastTestCase.AssertEquals(const AMessage: string;   const AFormatArgs: array of const; Expected, Actual: integer); begin   if Expected = Actual then exit;   inherited AssertEquals(Format(AMessage, AFormatArgs), Expected, Actual); end; * Yes, I am aware that skips counting the amount of calls. (needs "Inc(AssertCount)" in case of early exit) * The 2nd version eliminates any call to format in my own code. I wonder if it is worth adding such changes to the official testcase.

I have heaps and heaps of testcases, but I've never felt the need to investigate the speed of the Assert* calls.

Database access and HTTP mocking is much more time-intensive in the majority of tests I do.

Show quoted text 14 lines
------------------ Or maybe it is just the way I write tests? E.g. I have a test, that computes lists, it then does   for i := 0 to count do     AssertEquals(format('...', [...]), AExp[i], ATestRes[i]); Additionally those lists are tested in a loop, iterating through big amounts of control parameters for each test-list to be generated. So I do end up with millions and millions of calls to AssertEqual.

Millions of AssertEquals calls ?
I'm surprised that you are surprised it takes so long? :-)

If you present a patch to avoid constructing the message in case the test is OK, that's certainly fine for me.

But then please do it so the comparison is done only once, i.e. something like:

if aExpected=aValue then
   Inc(AssertCount)
else
   Fail(ComparisonMsg(aMessage, aExpected,aValue),CallerAddress);

Comparing long strings also takes some time...

Michael.

Reply by email

Martin Frb
to Michael Van Canneyt
Michael Van Canneyt wrote
I have heaps and heaps of testcases, but I've never felt the need to investigate the speed of the Assert* calls.

I wasn't actually looking at it... I found it by random chance.

I was trying to kcachegrind some of my code, and the results said that the majority of time went into string handling that wasn't part of the tested code.

Millions of AssertEquals calls ? I'm surprised that you are surprised it takes so long? :-)

I haven't counted them.

But some of my tests are simply fuzzing (ish) tests (not necessarily random, but just throwing large amount of data at the code).

If you present a patch to avoid constructing the message in case the test is OK, that's certainly fine for me.

I may... Need to find some time.

Also, what do you think about overloading AssertEquals (and the like) taking AssertEqual(
  const AnFormatString: string;
  const AFormatData: array of const;
  AnExpect,
  AnActual: TFoo
)
?

That allows to defer the call to format too.

Or to replace
  AssertEqual('Testing element ' + inttostr(n), ...) with
  AssertEqual('Testing element %d', [n], ...)

Show quoted text 9 lines
But then please do it so the comparison is done only once, i.e. something like: if aExpected=aValue then   Inc(AssertCount) else   Fail(ComparisonMsg(aMessage, aExpected,aValue),CallerAddress);

Either that, or (protected or public)

procedure CheckEqual(AExp, AnActual: integer): Boolean; overload;
begin
   result :=  aExpected=aValue;
   Inc(AssertCount) ;
end

procedure AssertEqual
if not checkequal(AExp, AnActual) then
  Fail(....)

Either one can be marked inline. But doesn't have to, even non inline they are fast enough.

After all, its testcases, its not about a few seconds extra or less. But the changes for me, brought some tests from 5 minutes to just 1 minute. And that helps, if I am making changes to the tested code....

Reply by email

Hairy Pixels
to Martin Frb

On Jun 14, 2025 at 10:50:46 PM, Martin Frb via fpc-devel <

Show quoted text 14 lines
wrote: I may... Need to find some time. Also, what do you think about overloading AssertEquals (and the like) taking AssertEqual( const AnFormatString: string; const AFormatData: array of const; AnExpect, AnActual: TFoo ) ?

Curious, if the function was actually inline wouldn’t the condition be tripped and exit before calling the format call which was passed in? FPC often fails to inline for various reason so if it can’t I agree your solution makes sense.

I had this same problem before when trying to test some metrics. I think I decided to use PChar even to ensure I only passed string constants and avoided some copies.

Regards,
Ryan Joseph

Reply by email

Michael Van Canneyt
to Martin Frb: I wasn't actually looking at it... I found it by random chance. I was trying to…
Martin Frb wrote 36 lines
On 14/06/2025 15:44, Michael Van Canneyt via fpc-devel wrote: > > I have heaps and heaps of testcases, but I've never felt the need to > investigate > the speed of the Assert* calls. I wasn't actually looking at it... I found it by random chance. I was trying to kcachegrind some of my code, and the results said that the majority of time went into string handling that wasn't part of the tested code. > > Millions of AssertEquals calls ? I'm surprised that you are surprised > it takes so long? :-) I haven't counted them. But some of my tests are simply fuzzing (ish) tests (not necessarily random, but just throwing large amount of data at the code). > > If you present a patch to avoid constructing the message in case the > test is > OK, that's certainly fine for me. I may... Need to find some time. Also, what do you think about overloading AssertEquals (and the like) taking AssertEqual(   const AnFormatString: string;   const AFormatData: array of const;   AnExpect,   AnActual: TFoo ) ?

OK with me.

I would not do the checkequal since that will lead to confusion with the Delphi DUnit compatibility layer.

Michael.

Reply by email

Sven Barth
to Hairy Pixels: On Jun 14, 2025 at 10:50:46 PM, Martin Frb via fpc-devel < Curious, if the function was…

Hairy Pixels via fpc-devel schrieb am Sa.,
14. Juni 2025, 18:20:

Show quoted text 21 lines
On Jun 14, 2025 at 10:50:46 PM, Martin Frb via fpc-devel < > wrote: > I may... Need to find some time. > > Also, what do you think about overloading AssertEquals (and the like) > taking > AssertEqual( > const AnFormatString: string; > const AFormatData: array of const; > AnExpect, > AnActual: TFoo > ) > ? > Curious, if the function was actually inline wouldn’t the condition be tripped and exit before calling the format call which was passed in? FPC often fails to inline for various reason so if it can’t I agree your solution makes sense.

If the compiler would indeed inline in this case, then yes, the Format call would be skipped. However open arrays are currently not supported for inlining.

But even without inlining this is simply more convenient for a user (I often have such formatting related overloads in my own code as well).

Regards,
Sven

Reply by email