Hello together!

I have taken the time to implement a first version of Delphi's class helpers. Thanks to Jonas' code for Objective C categories it was a rather easy to get that working.

I have yet to test the compatibility of this implementation with Delphi and this is where the FPC community comes in. I'd like you (@all list readers) - if possible - to test this extension with tasks you'd use them in and also to test what behaves differently from Delphi if you have a current Delphi version available (I believe you need Delphi 2007 or newer for this).

Especially at the core devs, but maybe also for other interested persons I have the following questions regarding the implementation of this feature:
- should class helpers have access to protected fields of the extended class? I propose not or this would beat the complete idea of "sealed" (I also believe that Delphi doesn't allow it as well)
- should class helpers be able to be instantiated or even referenced in any way? (including forward declarations)
- should class helpers be allowed to override/reintroduce methods?
- should class helpers be able to extend other class helpers (as class helpers are implemented as child classes of the extended class this might currently be possible)?
- should class helpers be able to implement interfaces?
- can/should "message" methods be forbidden?
- should abstract methods be forbidden?
- should a class helper for class X hide a method which was introduced in a subclass of X?
- can class helpers extend generic classes?
- can class helpers BE generic?

I have attached the patch which implements the class helper syntax in the trunk compiler. Please note that this might not be the final version, so please do not rely on the syntax and especially the behavior.

Class helpers adhere to the following syntax (I'm still thinking whether I should slap Borland/Codegear for that...)

type
   [ClassName] = class helper for [ExtendedClass]
     class procedure SomeClassMethod;
     procedure SomeMethod;
     (...)
   end;

The methods are implemented like in a normal class, but "Self" refers to an instance of the extended class in normal methods and to the class itself if used inside a class method (of course other methods implemented by the class helper are available as well... basically a class helper is a child class of the extended class).

The methods implemented by class helpers are automatically available if the class helper is available in the current scope. The methods are called as if they belong to the extended class:

var
   c: [ExtendedClass];
begin
   // class methods
   [ExtendedClass].SomeClassMethod;
   c := [ExtendedClass].Create;
   c.SomeClassMethod;
   c.SomeMethod;
end;

For the case you are still looking for a usecase: On the Lazarus mailing list was posted such one by ugaciaka in this mail: http://lists.lazarus.freepascal.org/pipermail/lazarus/2010-December/058368.html A possible answer for his first question would be the following (untested and maybe also properties would work...):

type
   TCheckListBoxHelper = class helper for TCheckListBox
     function GetCheckedCount: Integer;
   end;
function TCheckListBoxHelper.GetCheckedCount: Integer;
var
   i: Integer;
begin
   Result := 0;
// Count and Checked belong to the extended TCheckListBox
   for i := 0 to Count - 1 do
     if Checked[i] then
       Inc(Result);
end;
(...)
// somewhere in a piece of code that has access to a TCheckListBox instance
c := SomeCheckListBox.GetCheckedCount;

Please feel free to comment, criticise, test and discuss this feature and more especially its implementation.

Regards,
Sven

Sven Barth wrote

I have taken the time to implement a first version of Delphi's class helpers. Thanks to Jonas' code for Objective C categories it was a rather easy to get that working.

Note that Delphi-style class helpers have several limitations that still have to be added to your patch: a) from http://docwiki.embarcadero.com/RADStudio/2010/en/Class_and_Record_Helpers : "You can define and associate multiple helpers with a single type. However, only zero or one helper applies in any specific location in source code. The helper defined in the nearest scope will apply. Class or record helper scope is determined in the normal Delphi fashion (for example, right to left in the unit's uses clause)." -- In Objective-C, there is no limit on the number of categories for a class that applies at the same time. b) afaik, Delphi class helpers cannot contain virtual methods, and definitely cannot override methods defined in a parent class (the used VMT is that of the helped class, so no additions/changes to the class that require VMT changes are allowed or even possible)

- should class helpers be able to be instantiated or even referenced in any way? (including forward declarations)

No. They are not first class entities, they just add a bunch of method declarations to the scope of the extended class.

- should class helpers be allowed to override/reintroduce methods?

Definitely no override. I don't know about reintroduce.

- should class helpers be able to extend other class helpers (as class helpers are implemented as child classes of the extended class this might currently be possible)?

No, that does not make any semantic sense. How they are implemented in the compiler is irrelevant.

- should abstract methods be forbidden?

Yes, since
a) they are virtual
b) even if it were possible to have virtual methods, there would be no way to override them with implementations that do something (since you cannot inherit from a class helper)

Jonas

Actually, it seems that inheritance is in fact allowed for Delphi-style class helpers: http://qc.embarcadero.com/wc/qcmain.aspx?d=76577 (it's not for Objective-C categories). But I'm still quite sure you cannot do anything that requires changes to the VMT of the helped class.

There are also at some extra checks that must be added for constructors/destructors introduced via class helpers (Objective-C/Pascal doesn't have constructors or destructors, so it doesn't matter there):
* http://docwiki.embarcadero.com/RADStudio/en/E2295_A_class_helper_cannot_introduce_a_destructor_(Delphi)
* http://docwiki.embarcadero.com/RADStudio/en/E2296_A_constructor_introduced_in_a_class_helper_must_call_the_parameterless_constructor_of_the_helped_class_as_the_first_statement_(Delphi)
* http://docwiki.embarcadero.com/RADStudio/en/E2358_Class_constructors_not_allowed_in_class_helpers_(Delphi)

Jonas

I will do some tests.

I have attached the patch which implements the class helper syntax in the trunk compiler. Please note that this might not be the final version, so please do not rely on the syntax and especially the behavior.

Maybe you should create a branch so testing will go easily? I can't give you a branch myself but Florian usually do this is there is a need.

Best regards,
Paul Ishenin

Protected, strict protected and private but not strict private.

- should class helpers be able to be instantiated or even referenced in any way? (including forward declarations)

no

- should class helpers be allowed to override/reintroduce methods?

reintroduce - yes, but since class helper is not a child class using reitroduce does not do anything.

- should class helpers be able to extend other class helpers (as class helpers are implemented as child classes of the extended class this might currently be possible)?

no

- should class helpers be able to implement interfaces?

This needs to be tested but I don't think they can. You can't define a helper that implements interface - so all interfaces should be listed in class. Another thing that you can reintroduce QueryInterface, GetInterface in the helper but I don't think they will be used by 'is' and 'as' operators.

- can/should "message" methods be forbidden?

In delphi they are not forbidden but also they can't be reached by Dispatch call.

- should abstract methods be forbidden?

In delphi they are not forbidden and require an implementation. It looks tht delphi skips 'abstract'. Also delphi skips 'virtual' and 'dynamic'. I think in FPC this should be forbidden.

- should a class helper for class X hide a method which was introduced in a subclass of X?

no

- can class helpers extend generic classes?

no one example I tried was able to compile

I have attached few tests which answers to most of your questions.

Best regards,
Paul Ishenin

After reading of Jonas mail and following to QC I found that next example indeed works:

   TFooHelper1 = class helper(TFooHelper) for TFoo
   end;

Best regards,
Paul Ishenin

Paul Ishenin wrote

After reading of Jonas mail and following to QC I found that next example indeed works: TFooHelper1 = class helper(TFooHelper) for TFoo end;

I think it's probably related to the fact that at most one class helper can be used at a time. If you could not inherit from other class helpers, you would to completely reimplement them if you wanted to add one extra method later in another unit.

Jonas

I don't know whether we should really follow this limitation (and when, then only in mode Delphi)

I'll take your answers into account.

Regards,
Sven

Ok... I should be able to implement that rather easy.

Should be easy to check.

* http://docwiki.embarcadero.com/RADStudio/en/E2296_A_constructor_introduced_in_a_class_helper_must_call_the_parameterless_constructor_of_the_helped_class_as_the_first_statement_(Delphi)

Ok... O.o that will be an interesting one.

* http://docwiki.embarcadero.com/RADStudio/en/E2358_Class_constructors_not_allowed_in_class_helpers_(Delphi)

As we don't have them yet...

Regards,
Sven

Sven Barth wrote

> http://docwiki.embarcadero.com/RADStudio/en/E2358_Class_constructors_not_allowed_in_class_helpers_(Delphi) > As we don't have them yet...

we have.

Best regards,
Paul Ishenin

I would suggest asking on the Embarcadero forums what the reason is for this limitation. It would be annoying to have to add to wiki/ User_Changes_2.6.2 "Disabled multiple class helpers for a single class because it breaks/conflicts with/... X", like we have had to do already with several other cases where we removed what we considered to be useless limitations of Delphi features. The fact that it works in Objective-C/Pascal does not mean that it also fits to Object Pascal, because both languages are quite different (and because Objective-C/Pascal has a complex runtime system that does a lot of things behind your back).

Regarding some of the other limitations mentioned earlier: it seems that in Delphi 2010, you can add virtual methods to a class helper: http://windwings.wordpress.com/2009/10/07/turbocharging-delphi-2010-adding-dynamic-functionality-to-3rd-party-frameworks-read-vcl/ . You cannot override any methods of a helped class, but you can override virtual methods defined in one class helper in a class helper that inherits from it. Apparently support for class constructors/ destructors has also been added to class helpers.

This feature (virtual class helper methods) might be part of the reason why only one class helper can be used, depending on how they handle the class helper VMTs. You start getting some kind of multiple inheritance with multiple VMTs in terms of figuring out which class helper VMT to pass when calling one class helper method from another class helper.

Take this example:

*** unit1:

type
   tclasshelperbase = class helper for tobject
     procedure test; virtual;
   end;
   ttestcaller = class helper for tobject
     procedure calltest;
   end;
procedure tclasshelperbase.test;
   begin
     writeln('in base');
   end;
procedure ttestcaller.calltest;
   begin
     test;
   end;

*** unit 2:

type
   tclasshelperderived = class helper(tclasshelperbase) for tobject
     procedure test; override;
   end;
procedure tclasshelperderived.test;
   begin
     writeln('in derived');
   end;
*** main program:
uses
   unit1, unit2;
var
   o: tobject;
begin
   o:=tobject.create;
   o.calltest;
end.

Now which "test" should be called from ttestcaller.calltest? Since both unit1 and unit2 are in scope, I'd assume
tclasshelperderived.test. But how does ttestcaller.calltest get the VMT of tclasshelperderived? Do you pass a linked list or array of VMTs of all class helper hierarchies currently in scope to all class helper routines, which then walk this list/array every time in search for the appropriate VMT to use?

Jonas

And here I thought that I'm closely following FPC development... that must have slipt my attention. How embarrassing. -.-

Paul Ishenin wrote

I have attached few tests which answers to most of your questions.

Thank you for the tests. They indeed solved some questions, but some are still left.

Would you please test the following:
- identifier visibility if class helper and extended class reside in different units (because I can't believe that a helper can access private members while a derived class can't)
- can overloaded methods be defined? (e.g. class contains "Test(a: String);" and helper contains "Test(a: Integer);")
- what happens if you define a method in the helper that has the same name as a virtual method in the extended class (without "override" of course ^^)
- is a class helper method used in a derived class if the derived class does not contain a method with the same name?
- class helper and class in two different units, both containing a method with the same name. which method get's called if the unit with the class is used before the unit with the class helper in the main program?
- can class helpers have fields and if so is their value saved somehow?
- can class helpers extend interfaces?
- can a class helper reintroduce a virtual method from the extended class?

Jonas Maebe wrote

This feature (virtual class helper methods) might be part of the reason why only one class helper can be used, depending on how they handle the class helper VMTs. You start getting some kind of multiple inheritance with multiple VMTs in terms of figuring out which class helper VMT to pass when calling one class helper method from another class helper. Take this example:

[illustrative example]

This could indeed be the reason why only one class helper is used. So either we follow Delphi compatibility and allow virtual methods in class helpers and forbidding the use of multiple class helpers or forbidding virtual methods and allowing the use of multiple class helpers (although I must admit that I don't yet know which other problems might come around)...

So let me summarize what I know now:
- class helper syntax is the followingː

ClassHelperName = class helper[(BaseClassHelper)] for ExtendedClass (...)
end;

BaseClassHelper must be a class helper for the same class.

- in "virtual; abstract;" and "override;" after methods of the extended class generate an error, while "dynamic;" and "virtual;" are allowed

- "message" is allowed but in the end not accessable - should that be forbidden in mode "objfpc"?

- only the first found class helper for a given class is used

- class helpers can not be used in forward declarations

- class helpers can not extend generic classes

- class helpers may not contain destructors

- class helpers may contain constructors, but these need to call the inherited parameterless constructor without any argument as first statement (regarding class constructors I need to read the article linked by Jonas)

- class helpers hide methods with the same name in the ExtendedClass, but not in derived classes - is this a good behavior? (in the current test implementation a class helper method is only searched if no other identifier is found)

Are those assumptions correct so far? Did I miss something we already know?

Maybe I should write a wiki page to have the "specification" saved somewhere :)

Regards,
Sven

Sven Barth wrote

> we have. And here I thought that I'm closely following FPC development... that must have slipt my attention. How embarrassing. -.-

They still don't work the same way as in delphi. They are initialization sections but not "clever" :( I was not able to find a solution how to tell linker not to reference that sections by init/final tables.

Maybe I should write a wiki page to have the "specification" saved somewhere :)

This is a good idea.

I will try to answer to your other questions with a new test.

Best regards,
Paul Ishenin

Sven Barth wrote

Would you please test the following: - identifier visibility if class helper and extended class reside in different units (because I can't believe that a helper can access private members while a derived class can't)

Class helper can access the same as derived class

- can overloaded methods be defined? (e.g. class contains "Test(a: String);" and helper contains "Test(a: Integer);")

If overload is used then yes, if not then no. Look at test for more info.

- what happens if you define a method in the helper that has the same name as a virtual method in the extended class (without "override" of course ^^)

Look at test.

- is a class helper method used in a derived class if the derived class does not contain a method with the same name?

Look at test.

- class helper and class in two different units, both containing a method with the same name. which method get's called if the unit with the class is used before the unit with the class helper in the main program?

Will test later.

- can class helpers have fields and if so is their value saved somehow?

They can't.

- can class helpers extend interfaces?

How you see this?

- can a class helper reintroduce a virtual method from the extended class?

Yes.

Best regards,
Paul Ishenin

----- "Jonas Maebe" wrote

On 21 Dec 2010, at 00:16, Jonas Maebe wrote:

Sven Barth wrote

- should abstract methods be forbidden?

Actually, it seems that inheritance is in fact allowed for Delphi-style class helpers: http://qc.embarcadero.com/wc/qcmain.aspx?d=76577 (it's not for Objective-C categories). But I'm still quite sure you cannot do anything that requires changes to the VMT of the helped class.

There are also at some extra checks that must be added for constructors/destructors introduced via class helpers (Objective-C/Pascal doesn't have constructors or destructors, so it doesn't matter there):
* http://docwiki.embarcadero.com/RADStudio/en/E2295_A_class_helper_cannot_introduce_a_destructor_(Delphi)
* http://docwiki.embarcadero.com/RADStudio/en/E2296_A_constructor_introduced_in_a_class_helper_must_call_the_parameterless_constructor_of_the_helped_class_as_the_first_statement_(Delphi)
* http://docwiki.embarcadero.com/RADStudio/en/E2358_Class_constructors_not_allowed_in_class_helpers_(Delphi)

hi,

it seems that the virtual methods can be overridden (AOP style) using the TVirtualMethodInterceptor that comes with Delphi XE. It can also be on a per instance base (as opposed to on a per type) if I've read up on it correctly.

I'd say that would be a better candidate for that purpose/intent/use case.

kind regards,
Dimitri Smits

I don't know whether I understand you right, but we aren't talking about how we should allow the override of virtual methods, but about what Delphi allows regarding class helpers and what not, so we can have a as compatible as possible implementation in FPC.

Regards,
Sven

Hello again!

In the last few days I've written quite a few tests regarding class helpers and have come across some interesting cases. I've used Delphi 2007 to compile these tests.

Case 1:

The following code compiles in Delphi

=== source begin ===

program tchlp20;
{$ifdef fpc}
   {$mode objfpc}
{$endif}
type
   TObjectHelper = class helper for TObject
   end;
   TSomeRec = record
     helper: TObjectHelper;
   end;
begin
end.

=== source end ===

This is strange, because every other reference to a class helper ( variable declaration, parameter, parent class) results either in a error or even in an internal error (haha ^^). Can someone test with a newer version of Delphi (e.g. XE)?
If a newer Delphi also successfully generates a binary for this, should I consider this a bug or does someone knew more about this?

Case 2:

The following code compiles as well, which scares me a bit, cause I have to implement that as well :P

=== source begin ===

program tchlp36;
type
   TObjectHelper = class helper for TObject
     procedure Test;
   end;
   TFoo = class
   end;
   TFooHelper = class helper(TObjectHelper) for TFoo
   end;
procedure TObjectHelper.Test;
begin
end;
var
   f: TFoo;
begin
   f.Test;
end.

=== source end ===

I yet need to test what happens if the extended classes don't inherit from eachother...

Case 3:

I have attached two sources (tchlp35.pp and uchlp35.pp) for a test regarding virtual methods in class helpers. This code does not compile in Delphi 2007, but it might in Delphi XE. So could someone who has access to a Delphi XE compile those tests and send me the binary, please (you need to compile the unit first, cause Delphi does not find *.pp sources)?

At last I have a question regarding compiler development: Should I use new error messages like "reference to class helper not allowed here" and so on or should I rely on the messages that are already there? If the first: what needs to be changed to add a new message?

Regards,
Sven

In our previous episode, Sven Barth said:

In the last few days I've written quite a few tests regarding class helpers and have come across some interesting cases. I've used Delphi 2007 to compile these tests.

C:\testing>dcc32 tchlp20.dpr
Embarcadero Delphi for Win32 compiler version 22.0 Copyright (c) 1983,2010 Embarcadero Technologies, Inc. tchlp20.dpr(14) Error: E2527 Helper type 'TObjectHelper' cannot be used in declarations
tchlp20.dpr(19)

Embarcadero Delphi for Win32 compiler version 22.0 Copyright (c) 1983,2010 Embarcadero Technologies, Inc. tchlp36.dpr(23) Warning: W1036 Variable 'f' might not have been initialized tchlp36.dpr(25)
26 lines, 0.03 seconds, 13512 bytes code, 13104 bytes data.

Embarcadero Delphi for Win32 compiler version 22.0 Copyright (c) 1983,2010 Embarcadero Technologies, Inc. tchlp35.dpr(6) Error: E2003 Undeclared identifier: 'TObjectHelperA' tchlp35.dpr(6) Error: E2022 Class helper type required tchlp35.dpr(7) Error: E2137 Method 'VirtualTest' not found in base class tchlp35.dpr(20) Error: E2003 Undeclared identifier: 'Test' tchlp35.dpr(25)

uchlp36 compiles without warnings

While it compiles I think it is bug I believe. This should not be implemented so in FPC.

Anyway, this better to report to Embarcadero forum and/or QC.

Best regards,
Paul Ishenin

Sven Barth wrote

Should I use new error messages like "reference to class helper not allowed here" and so on or should I rely on the messages that are already there? If the first: what needs to be changed to add a new message?

When you can reuse (or *slightly* modify/generalise) an existing message, do so. Otherwise add a new one. To do so, edit compiler/msg/ errore.msg and add a new entry at the end of the appropriate category, along with a description. If you're not sure, search for a related message and add it to the end of its category. See the text at the top for an explanation of the format of the messages.

When you "make" the compiler, the message files will automatically be regenerated from errore.msg

Jonas

Sometimes I need to edit the message. In this case I also modify other error[lang].msg files. Is this correct or I should use some automated process?

Best regards,
Paul Ishenin

There's a helper utility (utils/msgdif.pp), but I don't think it handles changed messages (except for a change in severity). So it's ok to manually modify the other language msg files.

Jonas

Ok, then it's a bug indeed. Thanks.

Oops. I forgot to add a "uses uchlp35" to tchlp35. Can you retest with that change, please?

Regards,
Sven

Sven Barth wrote

Oops. I forgot to add a "uses uchlp35" to tchlp35. Can you retest with that change, please?

It compiles fine.

Best regards,
Paul Ishenin

It seems to be intentional. If I change the code to the following:

=== source begin ===

program tchlp36;
type
   TBar = class
	end;
   TObjectHelper = class helper for TBar
     procedure Test;
   end;
   TFoo = class
   end;
   TFooHelper = class helper(TObjectHelper) for TFoo
   end;
procedure TObjectHelper.Test;
begin
end;
var
   f: TFoo;
begin
   f.Test;
end.

=== source end ===

I get the following output now:

=== output begin ===

P:\test>"c:\Programme\CodeGear\RAD Studio\5.0\bin\DCC32.EXE" tchlp36.pp CodeGear Delphi für Win32 Compiler-Version 18.5
Copyright (c) 1983,2007 CodeGear
tchlp36.pp(19) Fehler: E2294 Eine von 'TObjectHelper' abgeleitete unterstützende
Klasse steht nur für von 'TBar' abgeleitete Klassen zur Verfügung tchlp36.pp(30)

=== output end ===

Rough translation of the error:
A class helper that inherits from 'TObjectHelper' is only available for classes that inherit from 'TBar'.

Regards,
Sven

Ah.. and res = 1. So the test fails.

Best regards,
Paul Ishenin

Ok, thanks.

Regards,
Sven

Can you send me the binary, please, cause I'm interested in the runtime behavior of that test.

Regards,
Sven

Ok, thanks (no need to send the binary then ^^)

Regards,
Sven

In our previous episode, Sven Barth said:

Then it compiles, no warnings.

In our previous episode, Paul Ishenin said:

Same with XE.

Thank you, too.

Now I know that overriding virtual methods in a class helper is of no use (which was expected a bit after the example Jonas gave the last time).

So... should "virtual", "dynamic" and "override" be forbidden in mode objfpc in class helpers?

Regards,
Sven

I think yes.

Anyway, please also search for examples on web. Like on this page: http://www.davinciunltd.com/code/delphi-class-helpers/ you can download class helper examples
(http://www.davinciunltd.com/download/JimMcKeeth-ClassHelpers-CodeRageII.zip) and references to pages where this feature was discussed by delphi users and developers.

Best regards,
Paul Ishenin

Will do.

Especially the entry about "class helpers and enumerators" is interesting.

Regards,
Sven