Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / fpc-devel / 2011-08

Generics in FPC have many problems

12 messages · 4 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

Anthony Walter

Jump to "So here are some of the problems I've found with fpc generics" if you don't want to read the backstory.

I haven't been active in the FPC for a while, but last week I was asking some questions about the upcoming Delphi FireMonkey project which is supposed to allow broader development with Pascal/Delphi. My first question to a newsgroup was, what about FPC, is the FireMonkey project going to work with it? I was redirected to this article:

http://wiki.freepascal.org/FPC_New_Features_Trunk

Great, these were the new features I was looking for, bridging Delphi and FPC {$mode delphi} features. So then last week I switched my laptop windows SSD for my linux SSD and got the latest sources of FPC and Lazarus sources. I wrote a new XML wrapper around both libxml and msxml, and then dove into converting my Delphi generics library. My eventual goal is to create the best cross platform game engine ever based on Delphi/Pascal.

I filled out a few bugs on the issue tracker for Lazarus where the new generic syntax for {$mode delphi} broke CodeTools. Then sent a few patches to fix those issues myself (yeah me).

But then on Saturday I started out porting my generics library, and quickly began to fall into depression. It seems FPC have a lot of problems with generics. I had to take a break from all my language testing due to all the problems I kept running into which made my head spin after each session of tryng to figure out what the problems are.

On Sunday I sat down again and started hashing out how to get by with FPC generics. In the evening I finally was able to focus in on the core problems with the way generics work in FPC.

I'd like to contribute to fixing these problems, but honestly I have no idea how to modify the compiler. I managed to send some patches off for CodeTools, but believe the compiler is a whole other level that needs gross amounts of fixings when it comes to generics. Something I'm not capable of.

Jump: So here are some of the problems I've found with fpc generics.

This does not include all the bugs that cause the compiler to act irrational one moment and a different irrational the next moment. This is simply a list of the things FPC generics can't do. I mention these rather than the generic compiler bugs, because IMO if the way generics work aren't changed, it's not even worth fixing the compiler bugs the existing syntax brings.

You can't write a generic method in a non generic class. The syntax 'TMyForm.ShowContents<T>' is invalid is invalid in both objfpc and delphi modes. You are forced to define a generic class if you want to use a generic method.

type
  TStringConverter<T> = function(Item: T): string; // okay delphi mode, in
objfpc this line should begin with generic
  TMyForm = class(TForm)
    procedure ShowContents<T>(List: TList<T>; Converter:
TStringConverter<T>);

...

// Having a generic method is not okay
procedure TMyForm.ShowContents<T>(List: TList<T>; Converter:
TStringConverter<T>);
var
  Item: T;
begin
  ListBox.Items.BeginUpdate;
  try
    ListBox.Items.Clear;
    for Item in List do
      ListBox.Items.Add(Converter(Item));
  finally
    ListBox.Items.EndUpdate;
  end;
end;

You can't use generics without specialization.

// This is not okay. Instead you have to specialize TList<T> and
TStringConverter<T> inside TMyTestObject<T>
procedure TMyTestObject<T>.DumpContents(List: TList<T>; Converter:
TStringConverter<T>);
var
  Item: T;
begin
  FStrings.BeginUpdate;
  try
    FStrings.Clear;
    for Item in List do
      FStrings.Add(Converter(Item));
  finally
    FStrings.EndUpdate;
  end;
end;

Specialization leads to problems, as specialized types from one class cannot be share with another class. Also you cannot use specialization inside an interface.

  IMyInterface<T> = interface
    function GetCurrent: T;
    // because of we specialization can't use other generics in an interface
    function CurrentString(Converter: TStringConverter<T>): string;
    procedure TakeFirst(List: TList<T>);
    property Current: T read GetCurrent;
  end;

The solution:

Get rid of the whole specialization concept in fpc. Generics should be able to be declared at any point using the type or a generic parameter. You should be able to simply say things like:

type TMyClass = class List: TList<Integer>; end;

or

type TMyClass = class procedure CountDistinct<T>(List: List<T>); end;

or

type TMyClass = class function Combine(List: List<string>): string; end;

or

type TMyInterface<T> = function MyMethod(List: List<T>;
Converter: TStringConverter<T>): Integer; end;

And if someone ever does intent to fix generics in FPC, there are a whole slew of errors that they already bring into the compiler. I'll make a list of examples of those errors at a later time, but they inner workings of generics should be redone to remove the specialization requirement before those errors are addressed.

Reply by email

Paul Ishenin
to Anthony Walter
Anthony Walter wrote 8 lines
Jump to "So here are some of the problems I've found with fpc generics" if you don't want to read the backstory. ... And if someone ever does intent to fix generics in FPC, there are a whole slew of errors that they already bring into the compiler. I'll make a list of examples of those errors at a later time, but they inner workings of generics should be redone to remove the specialization requirement before those errors are addressed.

Yes. These are known problems and you would found even more a year ago. Sven Barth now trying to fix exising problems and implement missing things.

If you wish to try fix them yourself you need to study how parser part of the compiler works.

Best regards,
Paul Ishenin.

Reply by email

Anthony Walter
to Paul Ishenin
Paul Ishenin wrote 10 lines
Yes. These are known problems and you would found even more a year ago. Sven Barth now trying to fix exising problems and implement missing things. If you wish to try fix them yourself you need to study how parser part of the compiler works. Best regards, Paul Ishenin.

Paul, thanks for the reply. Did you mean to say that the generics are actually being improved ... that is that someday you might be able to say:

procedure TMyNonGenericClass.MyGenericMethod<T>(List: TList<T>);

Or that the random seeming bugs with the existing way generics works is being fixed?

I think the way generics work needs to fixed first, to allow code like above to work. Then the strange bugs surrounding the new way they work can be addressed.

One thing that basically needs to go away is the need to "specialize" a generic. This causes problems even in {$mode delphi}.

Reply by email

Jonas Maebe
to Anthony Walter: Paul, thanks for the reply. Did you mean to say that the generics are actually being…
Anthony Walter wrote 9 lines
Paul, thanks for the reply. Did you mean to say that the generics are actually being improved ... that is that someday you might be able to say: procedure TMyNonGenericClass.MyGenericMethod<T>(List: TList<T>); Or that the random seeming bugs with the existing way generics works is being fixed?

You can see the details of what has already happened by looking at the svn logs for http://svn.freepascal.org/svn/fpc/branches/svenbarth/generics , e.g.:
svn log http://svn.freepascal.org/svn/fpc/branches/svenbarth/ generics | less

The changes specific to that branch start with r17316. As far as {$mode delphi} is concerned, the eventual goal is to have a completely Delphi-compatible generics syntax.

Jonas

Reply by email

Sven Barth
to Anthony Walter: Jump to "So here are some of the problems I've found with fpc generics" if you don't want…

Hi!

As Paul and Jonas already told you, I'm currently working on improving the compilers generic implementation. As you have noticed it currently doesn't support generic methods and the generic
implementation in mode Delphi isn't working correctly as you've noticed. Inline specialications (those without using "type foobar = foo<bar>") don't work either.

For this I have found the following tasks:
* allow declaring and implementing generics with the Delphi syntax while keeping the FPC syntax supported (this is an important point regarding backwards compatibility); state: working
* allow the declarations of multiple generics with the same name (e.g. foo<t> =... foo <t, s> = ...); state: mostly working
* allow inline specializations; state: partly working inside implementations; inside declarations this needs to be tested
* allow nested specializations: not started
* allow nested generics: not started
* implement generic methods: not started
* implement constraints: not started

My plan is to merge the changes from my branch to trunk after the first three points are done (the third one at least mostly as inline specializations are rather fragile) to reduce the differences between those to branches. Then I'll start working on either generic methods or constraints (depends on my mood then).

Please keep in mind that working on the generics feature isn't an easy task, so I can't give you an estimate when the one or other improvement will be available.

If you want you can test around with my branch, but please keep in mind that it's a work in progress and that compilation of the compiler or the RTL might also be broken from time to time (currently the fgl unit doesn't compiler for example).

And if someone ever does intent to fix generics in FPC, there are a whole slew of errors that they already bring into the compiler. I'll make a list of examples of those errors at a later time, but they inner workings of generics should be redone to remove the specialization requirement before those errors are addressed.

If you do such an error list then please provide a single program for each problem you encounter so that it might be integrated into FPC's test suite which will ease fixing the problem.

Regards,
Sven

Reply by email

Anthony Walter
to Sven Barth
Sven Barth wrote 51 lines
Hi! As Paul and Jonas already told you, I'm currently working on improving the compilers generic implementation. As you have noticed it currently doesn't support generic methods and the generic implementation in mode Delphi isn't working correctly as you've noticed. Inline specialications (those without using "type foobar = foo<bar>") don't work either. For this I have found the following tasks: * allow declaring and implementing generics with the Delphi syntax while keeping the FPC syntax supported (this is an important point regarding backwards compatibility); state: working * allow the declarations of multiple generics with the same name (e.g. foo<t> =... foo <t, s> = ...); state: mostly working * allow inline specializations; state: partly working inside implementations; inside declarations this needs to be tested * allow nested specializations: not started * allow nested generics: not started * implement generic methods: not started * implement constraints: not started My plan is to merge the changes from my branch to trunk after the first three points are done (the third one at least mostly as inline specializations are rather fragile) to reduce the differences between those to branches. Then I'll start working on either generic methods or constraints (depends on my mood then). Please keep in mind that working on the generics feature isn't an easy task, so I can't give you an estimate when the one or other improvement will be available. If you want you can test around with my branch, but please keep in mind that it's a work in progress and that compilation of the compiler or the RTL might also be broken from time to time (currently the fgl unit doesn't compiler for example). And if someone ever does intent to fix generics in FPC, there are a whole > slew of errors that they already bring into the compiler. I'll make a list > of examples of those errors at a later time, but they inner workings of > generics should be redone to remove the specialization requirement before > those errors are addressed. > If you do such an error list then please provide a single program for each problem you encounter so that it might be integrated into FPC's test suite which will ease fixing the problem. Regards, Sven

Wow, Sven, thanks for the reply, and most definitely thanks for working on FPC. Anything I can do to help I'll do. Last week I sent patches to fix issues with Lazarus CodeTools and the new generic syntax which basically caused a lot of problems in the IDE.

I'll be on freenode irc in both #fpc and #lararus-ide for the next few weeks at least. Where do you want me to send some unit tests/test program source? The issue tracker? One issue for each test that or all in all tests in one issue?

Also be sure to let me know if there is anything I can do for you. I don't have much experience with OSS collaboration, but am motivated to get things done and done the right way.

Reply by email

Sven Barth
to Anthony Walter
Show quoted text 13 lines
Wow, Sven, thanks for the reply, and most definitely thanks for working on FPC. Anything I can do to help I'll do. Last week I sent patches to fix issues with Lazarus CodeTools and the new generic syntax which basically caused a lot of problems in the IDE. I'll be on freenode irc in both #fpc and #lararus-ide for the next few do you want me to send some unit tests/test program source? The issue tracker? One issue for each test that or all in all tests in one issue? Also be sure to let me know if there is anything I can do for you. I don't have much experience with OSS collaboration, but am motivated to get things done and done the right way.

First it might be good if you'd try to compile my branch and play around with taht. Do you know how to use svn?
Because there you can find all generic related tests that are currently available in the folder tests/test/. They are all named tgenericXX.pp. So if you find a test there that describes one of your problems then I've already tackled it.

Otherwise it might be best if you prepare a list of your problems and post them here. This way I can easily decide which ones are already on my ToDo list, which are already locally fixed or are serious ones that should be put to the bug tracker. With that I'll tell you for which ones I'll need tests.

If you post your problems though, then please try to find a simple test case that by itself could already be compiled with minor modifications/additions, because in your original posting there is quite a bit of "noise" in the code, if you understand what I mean ;)

Regards,
Sven

Reply by email

Anthony Walter
to Sven Barth

Sven,

Tonight I tried to "make all" from your branch and wasn't able to build. This is a the output I got:

http://pastebin.com/GQh5XA2w

I am guessing your build requires a working version of your fpc binary to build itself.

Also, I wrote a few of pretty clean test cases in Delphi 2010 Today when I had some downtime from my other work. I'd be interested in integrating them "cleanly" into your tests and helping the fpc community any way I can.

Finally, I think I looked at your change log. I guess the last check in you did was on July 16? I I reading that right?

------------------------------------------------------------------------ r18005 | svenbarth | 2011-07-16 12:19:33 -0400 (Sat, 16 Jul 2011) | 1 line

Rebase to revision 18000
------------------------------------------------------------------------ r18004 | svenbarth | 2011-07-16 10:13:56 -0400 (Sat, 16 Jul 2011) | 1 line

pexpr.pas, sub_expr: Added support for "as" and "is" operators if the right hand side is an inline specialization (currently detected by the next token being a "<"). This could potentially introduce some problems if the right hand side isn't a specialization but a "<" comparison together with some overloaded operators (I still need to find a case for such a problem)...

Reply by email

Sven Barth
to Anthony Walter
Anthony Walter wrote 7 lines
Tonight I tried to "make all" from your branch and wasn't able to build. This is a the output I got: http://pastebin.com/GQh5XA2w I am guessing your build requires a working version of your fpc binary to build itself.

The errors you see are already when compiling the RTL with the newly compiled compiler. As I said: compiling fgl is currently broken. I'd suggest you to do the following:
Replace fgl.pp in rtl/objpas/ with an empty unit of the same name (empty here means containing "unit fgl; interface implementation end."). You can't use the basic generic classes then, but for tests they aren't necessary. ^^
Then issue the following commands (I assume you start in the base directory of the checkout):

=== commands begin ===
cd compiler
make cycle
cd ../rtl
make clean all
=== commands end ===

You'll have a new compiler as compiler/ppc386 (assuming you're on a x86 system; it will be named approbiately for other systems) and a compiled RTL in rtl/units/PLATFORM-OS (e.g. rtl/units/i386-linux). You can then compile programs with the following command (I assume FPCDIR to be set to the checkout's base directory, and the current platform to be i386-linux ):

=== command begin ===

$FPCDIR/compiler/ppc386 -n -Fu$FPCDIR/rtl/units/i386-linux [OTHER-OPTIONS] yourtest.pas

=== command end ===

Also, I wrote a few of pretty clean test cases in Delphi 2010 Today when I had some downtime from my other work. I'd be interested in integrating them "cleanly" into your tests and helping the fpc community any way I can.

If you write them in a way that they adhere to the same basic pattern as other tests in FPC then they can easily be integrated (just take a look at the existing generic tests to see what I mean). I you have questions regarding this, please feel free to ask :)

Finally, I think I looked at your change log. I guess the last check in you did was on July 16? I I reading that right?

The SVN branch is a bit behind of my local GIT one, but not that much. I didn't have much time the last few weeks (and the coming weeks as well) as I'm working on my bachelor thesis :)

Regards,
Sven

Reply by email