Hi,

What is the state of FPC support for calling generic procedures using inference like Delphi?

For example:

{$mode objfpc}
generic procedure Run<T>(a: T);
begin
end;
begin
  Run(2);
end.

Mattias

Mattias Gaertner via fpc-pascal schrieb
am Do., 3. Okt. 2019, 21:35:

A patch for it by Ryan Joseph exists (
https://bugs.freepascal.org/view.php?id=35261 ), but I've not yet found the time to review it.

Regards,
Sven

On Oct 3, 2019, at 6:07 PM, Sven Barth via fpc-pascal wrote: A patch for it by Ryan Joseph exists ( https://bugs.freepascal.org/view.php?id=35261 ), but I've not yet found the time to review it.

I’m eager to use this also but it’s been sitting there for about 6 months now, and I don’t remember well, but I think it had some questionable design that Sven wasn’t sure about.

The patch for "generics in constants" has been reviewed well and being tested for some months already. Can that be integrated first so we can move forward to implicit function specialization?

Regards,
Ryan Joseph

Ryan Joseph schrieb am Fr., 4. Okt. 2019, 17:10:

My plan is indeed to review that one first.

Regards,
Sven

Ryan, is patch_3_25.diff the latest patch?

I get a lot of merge conflicts.
What fpc revision is needed for this?

And what is the git repo
https://github.com/genericptr/freepascal/tree/generic_implicit ?
It does not contain the modeswitch.

Mattias

Mattias

It’s the latest patch but that’s many months old already so nothing is up to date. I failed multiple times to update my branch from the “upstream" so I can't fix errors easily (there was a long thread some months ago but I gave up).

Look at the changes page on Github and search for m_implicit_function_specialization.

https://github.com/graemeg/freepascal/compare/master...genericptr:generic_implicit

Regards,
Ryan Joseph

My mistake was to use git clone
https://github.com/genericptr/freepascal.git.
The zip contains a valid version.

You implemented it pretty Delphi compatible.
As you know, the Delphi help states wrongly, that type inference uses 32bit "integer" when possible, which would mean that often only one specialization is created. In reality it uses the smallest integer type, creating easily many different specializations, bloating up the exe.
The Delphi compiler is really picky (dumb?) on type inference. That being said, I noticed several differences to Delphi (10.3):

1.
FPC allows default params, Delphi does not. Nice.
I see no tests for this though, so maybe this was not on purpose?

2.
DoThis(1,200) gives range check warning instead of error. A warning means there are some rare cases, where this code is correct. Is this a todo or do you see a useful case?

3.
timpfuncspez2.pp
DoThis<T>
DoThis<T,U>
Delphi gives an error "Ambiguous call to DoThis". FPC silently selects the one with only one param. IMO this is dangerous, it should give an error.

4.
Why does timpfuncspez6 fail?
It works in Delphi.
The comment has an explanation, which looks wrong to me:

generic procedure DoThis<T,U>(msg: T; param1: U; param2: TObject);
begin
end;
begin
        DoThis('aa', 'aa', TObject.Create);
        // wil be specialized as DoThis(msg: integer; param1: TObject;
param2: TObject)
        // so we expect an incompatible type error
        DoThis(1, 1, TObject.Create);
end.

Mattias

Thanks for testing. I don’t have access to Delphi so I need help to make it Delphi compatible.

Probably an oversight if you can believe it. I’ll test that.

Which test? Please post a sample.

This sounds familiar from C# also. Is that the problem case? I feel like we already went over this but it was months ago..

generic procedure DoThis<T>(msg: T);
begin
	writeln('DoThis$1#1:',msg);
end;
generic procedure DoThis<T>(msg: T; param1: T);
begin
	writeln('DoThis$1#2:',msg,' ',param1);
end;

That doesn’t make sense to me either so I need to study it. Both should fail actually as I designed it (for now).

How does Delphi implicitly specialize this?

Regards,
Ryan Joseph

Yeah this didn’t work so I need to think about it. Should it even be allowed though? That means the type can only be specialized the same as the constant?

generic procedure DoThis<T>(param1: string; param2: T = 100);
begin
end;

Regards,
Ryan Joseph

generic procedure DoThis<T>(a:T; b:T);
begin end;
begin
  DoThis(1,200); // 1 sets T to shortint, so the 200 gives a warning
end;

generic function DoThis<T>(a: T): T; overload;
begin end;
generic function DoThis<T,U>(a: T): U; overload;
begin end;
begin
  DoThis(3); // both fits, should give an error
end;

Why?

How does Delphi implicitly specialize this?

DoThis(1,1,nil); // T and U become shortint
DoThis('aa','aa',nil); // T and U become string

Mattias

I meant:

generic procedure DoThis<T>(param1: T; param2: word = 100);
begin
end;

Mattias

What is the preferred behavior? I’m getting an error in ObjFPC mode.

{$mode objfpc}
{$modeswitch implicitfunctionspecialization}
program timpfuncspez16;
generic procedure DoThis<T>(param1: T; param2: T);
begin
end;
begin
  DoThis(1, 200);
end.

This is debatable I think but I understand why "Ambiguous call to DoThis” would make sense. I’m pretty sure C# does this also.

What is the rule then? I’ll have to think about that some more.

I think my logic is different from what you expect though and I’m happy to change it if we need to.

There’s the comment I left in the code for the “try_implicit_specialization” function.

{ find generic procsyms by param count, starting from

          number of parsed params. if a procsym is found then
          validate via tcallcandidates and build list of *unique*
          types for use when specializing.

inferred generic types are evaluated by inserting
non-repating types into the list in linear order.
(1,'string') = <Integer,String>
(1,2,3,4,5,6) = <Integer>
('a','b') = <String>
('string',1) = <String,Integer>
('a',1,'b',2,'c') = <String,Integer>
}

I think for this example my logic does: T = integer, U = TObject because the first 2 integers params are repeating so they both consume the single T and then the TObject consumes the U. That means the second param “1” is being passed for a TObject type.

generic procedure DoThis<T,U>(msg: T; param1: U; param2: TObject);
DoThis(1, 1, TObject.Create); // DoThis(msg: integer; param1: TObject; param2: TObject);

I used this logic because it works for repeating patterns but it’s not very intuitive in cases where the arguments are the same number as the generic parameters.

What should be the rule here?

So Delphi seems to match by index so param 1 = generic param 1. With repeating patterns this breaks downs but maybe it’s a special case if the function parameter count matches the generic parameter count?

Regards,
Ryan Joseph

That works then. Added a test:

{$mode objfpc}
{$modeswitch implicitfunctionspecialization}
program timpfuncspez15;
generic procedure DoThis<T>(param1: T; param2: integer = 100);
begin
  writeln(param1, ' = ', param2);
end;
begin
  DoThis('hello');
end.

Regards,
Ryan Joseph

Ryan Joseph schrieb am Mo., 7. Okt. 2019, 00:16:

There is nothing debatable here. If a generic parameter is not used in the parameter list (cause it's used in the result type (either directly or to specialize something else), only the body of the routine or even not at all), then that routine must not be used for implicit specialization.

Regards,
Sven

[...] That works then. Added a test: [...]

Thanks.

Mattias

I get only a Warning:
Warning: (4110) range check error while evaluating constants (200 must be between -128 and 127)

Mattias

I gave a bad example. Here is a better one:

generic procedure DoThis<T>(a: T; b:T); overload;
begin end;
generic procedure DoThis<T,U>(a: T; b:U); overload;
begin end;
begin
  DoThis(1,2);
end;

fpc selects second, but IMO should give an error .

Mattias

I'm baffled. That's more basic algorithm than I expected. ;)

This means you don't support:
generic procedure Run<S,T>(a: T; b:S); overload;

And the following compiles, but fails on run, as T becomes an array instead of the element type:
generic procedure Run<T>(a: array of T); overload;

IMO the algorithm should look at the declarations instead of blindly mapping the argument types to the generic type params. If you want to keep the simple algorithm, then you should add some checks and give an error if the function does not support the algorithm.

Delphi does not simply match the index. For example this works as expected (I shortened the example):

generic procedure Run<T>(a: array of T); overload;
var
  p: PTypeInfo;
begin
  p:=TypeInfo(T);
  writeln(p^.Name);
end;
var a: array of boolean;
begin
  Run(a); // writes "boolean"
end;

Mattias

Yes, but does that mean to tell the programmer, that he is using these overloads the wrong way (as Delphi does), or should the compiler silently skip invalid overloads. For the sake of don't-point-the-gun-at-my-foot, I would vote for the first.

Mattias

Mattias Gaertner schrieb am Mo., 7. Okt. 2019,
11:59:

I'm also for an error.

However does Delphi pick a suitable non-generic overload if one exists aside from the two generic ones?

Regards,
Sven

Can you give an example what you mean?

Mattias

I think sven means if you have e.g. 3 functions:

generic function DoThis<T>(a: T): T; overload;
begin end;
generic function DoThis<T,U>(a: T): U; overload;
begin end;
function DoThis(aInt : Integer) : Integer;
begin
end;

Should the compiler choose the 3rd non-generic function, or still give an error ?

Michael.

You mean, if a bad overload can be nullified by a good one? That would be insane...

Yes, Delphi eats it without warning.

And it works even with a generic param:

generic function Run<T,U>(a: T; b: U): T; overload;
begin end;
generic function Run<T>(a: T; b: word): T; overload;
begin end;
Run(3,word(4));

Mattias

So do I understand you correctly and Delphi's behaviour is insane ? :-)

Michael.

On Oct 7, 2019, at 5:39 AM, Mattias Gaertner via fpc-pascal wrote: I'm baffled. That's more basic algorithm than I expected. ;)

All contributions are welcome. It sounds like we need make some changes or overhaul the entire thing.

This means you don't support: generic procedure Run<S,T>(a: T; b:S); overload;

That’s correct, this breaks the logic entirely. This to me looks like a sneaky way to trick the compiler and since it’s trying to infer something from the programmer it doesn’t make sense. If you have a better algorithm please post.

Good catch, that’s a bug.

var
  a: array of integer;
begin
  Run(a);

Regards,
Ryan Joseph

Right now this gives an error because you can’t add normal functions AFTER generic functions with the same name in the same scope. I can’t find the bug report but I think Sven fixed this recently or gave a better  function DoThis<T>(a: T): T; overload;
begin
end;
generic function DoThis<T,U>(a: T): U; overload;
begin
end;
// overloaded identifier "DoThis" isn't a function
function DoThis(a : integer): integer; overload;
begin
end;

========================

That bug aside the rule is that non-generic overloads always win over generics. I remember this was discussed at length if we can find the old thread.

{$mode objfpc}
{$modeswitch implicitfunctionspecialization}
program timpfuncspez14;
type
  TMyClass = class
  end;
procedure DoThis(msg: TObject);
begin
  writeln('DoThis:',msg.ClassName);
end;
generic procedure DoThis<T>(msg: T);
begin
  writeln('DoThis<T>:',msg.ClassName);
end;
begin
  DoThis(TMyClass.Create);
  DoThis(TObject.Create);
end.

prints:

DoThis:TMyClass
DoThis:TObject

Regards,
Ryan Joseph

On Oct 7, 2019, at 1:54 AM, Sven Barth via fpc-pascal wrote: There is nothing debatable here. If a generic parameter is not used in the parameter list (cause it's used in the result type (either directly or to specialize something else), only the body of the routine or even not at all), then that routine must not be used for implicit specialization.

Yeah I got it now. I’ll make an ambiguous error message in this case. Right now the cases the will trigger it are:

1) generic functions that have the same name/parameter list, regardless of generic parameters. This is an “Ambiguous call to DoThis” error.

	generic function DoThis<T>(a: T): T;
	generic procedure DoThis<T>(a: T; b: integer = 100);
	generic function DoThis<T,U>(a: T): U;
	generic procedure DoThis<T,U,S>(a: T);
	var
		field: S;
2) Parameter not in function parameter list. Not sure yet but something like “DoThis can not infer specialization from parameter list”.
	generic function DoThis<T>(a: integer): T;

If there’s more let me know.

Regards,
Ryan Joseph

Yes.
Maybe it's just me, every day learning more ugly details of Delphi inference rules.

Mattias

Note that in Delphi the "non-generic-wins" rule is per parameter:

procedure Run<T>(a:T; b: word);
procedure Run<T>(a: word; b: T);
Run(1,word(2)); // calls first
Run(word(1),2); // calls second
Run(1,2); // Error: ambiguous
Run(word(1),word(2)); // Error: ambiguous

Mattias

I don't know the inner workings of FPC, but for pas2js I will probably do something like this:
1. If proc has n templates, start with an array of n elements (all nil).
2. for each passed parameter expression do
2.1 Find the argument declaration of this parameter 2.2 resolve the parameter type
2.2 Traverse recursively the parameter and argument type 2.2.1 If the argument types is a template, assign the parameter type. If the template has already a type then try to find a shared type.
3. Check if all templates have types

[...]

Mattias

Ryan Joseph schrieb am Mo., 7. Okt. 2019, 15:51:

That bug aside the rule is that non-generic overloads always win over generics. I remember this was discussed at length if we can find the old thread.

Here:
https://lists.freepascal.org/pipermail/fpc-pascal/2018-December/055225.html

Luckily I remembered that I had been on a special event back then, thus I remembered the date. ;)

Regards,
Sven

I mean what happens here? The non-generic functions wins right?

procedure Run<T>(a:T; b: word);
procedure Run<T>(a: word; b: T);
procedure Run(a: word; b: word);

Regards,
Ryan Joseph

Only for
Run(word(1),word(2));

Mattias

Mattias Gaertner via fpc-pascal schrieb
am Mo., 7. Okt. 2019, 18:42:

Just to be sure: does the order of the function declarations make a difference?

Regards,
Sven

No. Even if the a,b:word overload is in the ancestor.

Mattias

Ok, then I got this wrong and I need to make significant changes. As I did it if there is a non-generic it always wins regardless of parameters. For example:

procedure Run(a: integer; b: string);
begin
end;
generic procedure Run<S,T>(a: S; b: T; c: integer);
begin
end;
// fails because the non-generic Run() will always be picked and the parameter count will be wrong.
Run(1,1,1);

Regards,
Ryan Joseph

Yes, that works in Delphi.

If you want to be better than Delphi, support this:

{$mode objfpc}
type
  TProc<T> = procedure(a: T);
  TAnt = class
    generic procedure Run<T: class>(a: TProc<T>);
  end;
  TBird = class end;
generic procedure TAnt.Run<T>(a: TProc<T>);
begin
end;
procedure Fly(Bird: TBird);
begin
end;
var obj: TAnt;
begin
  obj.Run(@Fly);
  // when fpc supports anonymous functions:
  obj.Run(procedure Fly(Bird: TBird) begin end);
end.

Mattias

Ryan Joseph schrieb am Fr., 4. Okt. 2019, 17:10:

I checked again. The thing is that in the compiler overload handling is done inside htypechk.tcallcandidates. Your patch weakens/changes that. So you'll have to completely rethink your solution to fit it into tcallcandidates (that should for example collect all suitable generics in addition to the non-generic ones (if any)).

Regards,
Sven

On Oct 8, 2019, at 11:32 AM, Sven Barth via fpc-pascal wrote: I checked again. The thing is that in the compiler overload handling is done inside htypechk.tcallcandidates. Your patch weakens/changes that. So you'll have to completely rethink your solution to fit it into tcallcandidates (that should for example collect all suitable generics in addition to the non-generic ones (if any)).

What is the entry point for this? My design was to intercept the dummy symbol and attempt to perform the specialization at that point and return a valid procsym so the parser could continue with its current code path. Are you saying I should allow the tcallcandidates.create to accept a dummy symbol instead of a procsym?

Sorry if that doesn’t make sense but I’m a little foggy as to how this all works. Any advice would be great but I’ll think about this more later. Btw, the overloading stuff is in try_implicit_specialization but maybe that

Regards,
Ryan Joseph

Ryan Joseph schrieb am Di., 8. Okt. 2019, 23:00:

Yes, exactly that. It could also be that the parser picks up a non generic overload from another unit (due to the order of units in the uses section) and then you can *only* handle that in tcallcandidates.

Regards,
Sven

On Oct 9, 2019, at 1:51 AM, Sven Barth via fpc-pascal wrote: Yes, exactly that. It could also be that the parser picks up a non generic overload from another unit (due to the order of units in the uses section) and then you can *only* handle that in tcallcandidates.

Hmmm, I thought letting that dummy sym pass through would have lots of adverse affects in various places. I’ll have to study this later in the debugger.

Regards,
Ryan Joseph

Ryan Joseph schrieb am Mi., 9. Okt. 2019, 22:53:

In that case the necessary parts would need to be made more resilient.

Regards,
Sven

As a general matter could you explain the reasoning behind resolve_generic_dummysym()?

Regards,
Ryan Joseph

The compiler tries to use as much as possible a non-generic symbol. E.g. if there is a TTest in the current unit and a non-generic TTest in another the compiler uses the latter even if a "<" follows. The real generic symbol will be found in generate_specialization_phase1 (or _phase2?) as only that will know the real amount of generic type parameters.

Regards,
Sven

I hope you're all having a lovely Coronavirus lockdown-vacation and keeping safe. :)

I finally got around to the overhaul of {$modeswitch implicitfunctionspecialization} as according to feedback in this thread. I attached a patch in the bug tracker which should apply without conflicts. In particular, Mattias, if you remember any of this please re-test and see if it works better now (I didn't include any tests yet). As for Sven I've left my design notes for you to review. I think I did what you wanted and the overloading works correctly now.

from the bug tracker:

====================

https://bugs.freepascal.org/view.php?id=35261

I've done an overhaul on the previous design so I'm posting a patch which is a draft version (with debug writeln's as placeholders for proper error messages). If the design is sane I will clean everything up and probably inline some functions.

1) the entire inference process is redone according to feedback (see try_implicit_specialization).
- array types and function pointers can be used for specialization.

   - templates can appear in any order now DoThis<T,U>(a: U; b: T);

2) as was requested by Sven the dummy syms are now valid procsyms and are passed along to tcallcandiates for overloading.
3) The inference is done at one location in do_proc_call but we could possibly move that inside tcallnode.pass_typecheck if it were better for some reason.
4) I added a tprocsym.add_generic_candiate to associate procsyms with dummy procsyms but I wasn't sure if this was good design so it's not cpu compliment and thus specialization from units will fail.
5) non-generic overloads are filtered in htypechk.is_better_candidate but I don't feel like I integrated my check in the right place due to me not really understanding that function.

Regards,
Ryan Joseph