1. On the todo list for fpc's MacPas mode is a feature that allows local procedures (functions) to be passed a procedural parameters. I tentatively put it on my own todo list, since this is the feature that I like most in Pascal compilers, at the same time making a small contribution to fpc and learning about the compiler. Before doing so, I need agreement on the method followed.

2. The problem with local procedures is that there must be a mechanism to address parameters and local variables of parent procedures. In Section 12.2 of "Grundlagen und Techniken des Compilerbaus" Niklaus Wirth points out we can do that with a static link chain pointer to the stack frame of the parent procedure(s). It is important to note that the static link chain can differ from the dynamic link chain because the latter corresponds to the calling chain and that calling chain can differ from the static parent-sub-procedure relation, i.e. if there is more than one local procedure at the same level.

3. FPC passes the static link chain frame pointer as an extra invisible parameter to local procedures (Section 6.4 of the fpc Programmer’s Manual).

4. So, when passing a local procedure (function) as procedural parameter, the static link chain frame pointer must be passed along with the entry point of the local procedure. However, this raises concerns about calling conventions and compatibility with existing code and the current compiler.

5. To solve this dilemma, gnu C, Pascal and Ada (and also MetroWerks CodeWarrior Pascal) use a technique called trampolines. The idea is outlined in <http://people.debian.org/~aaronl/Usenix88-lexic.pdf> and the gnu implementation is discussed at
<http://gcc.gnu.org/onlinedocs/gccint/Trampolines.html>. In short, a transition vector is set up temporarily for the local procedure that contains code from which the static link frame pointer can be obtained.

6. Although conventional wisdom dictates that we always follow the latest fashion in programming without further thinking, I believe that with some analysis we can do better than trampolines. There are two strong arguments that speak against the use of trampolines:

(a) trampolines must be setup at runtime and this involves flushing the instruction/data cache of the processor, a grave (and stupid) slowdown (b) trampolines are very tricky to implement and quite platform-dependent (see e.g.
<http://gcc.gnu.org/bugzilla/show_bug.cgi?id=10900>).

7. Consider the following program:

	program func;
	type tfun = function( x: real): real;
	procedure iso_fun( function f( x: real): real);
	begin
	end;
	procedure typ_fun( pf: tfun);
	begin
	end;
	procedure somefun;
		function f( x: real): real;
		begin
		 	f:= x
		end;
	begin
		iso_fun( f);
		typ_fun( f); {procedural variable can't get nested routiine}
	end;
	begin
	end.

MetroWerks CodeWarrior Pascal allows passing local procedures (functions) as procedural parameter, but there is an interesting point. It allows passing the local function f to iso_fun, but it doesn't allow passing it to typ_fun. The reason is clear - the value of the static link chain frame pointer is valid only within a specific activation of the parent procedure. So, it should not be allowed that the procedural parameter f somehow propagates out of the surrounding parent procedure and this can be guaranteed only if assignment to a typed procedural variable or (typed procedural value parameter) is forbidden.

So, we need to allow local procedures as actual parameters only for the "iso" notation. So, I suggest to change "iso" style procedural parameters only, i.e.

(a) change them from a pointer to a record: the record has two pointer fields, one for the routine's entry point and one for the optional static link chain frame pointer, or
(b) pass the static link chain frame pointer as an extra invisible parameter for each "iso" style procedural parameter.

Something similar has been implemented in fpc already, namely for methods that are passed as parameter. Here, an extra "self" pointer is needed.

The effect on current software should be minimal, because (currently) only MacPas mode allows "iso" style procedural parameters. This eliminates any concerns about calling conventions and compatibility with existing code and the current compiler.

Note that, with this solution, typed procedural variables can be passed as actual procedural parameters to "iso" style formal parameters, but not the other way round.

9. Conclusion.

With the above method, it should be relatively simple to implement local procedures (functions) as procedural parameters, without affecting the code generation of the current compiler.

Regards,

Adriaan van Os

With the above method, it should be relatively simple to implement local procedures (functions) as procedural parameters, without affecting the code generation of the current compiler.

The default calling convention is with registers. How do you want to handle that? At beforehand from the procvar type it must be known if the framepointer must be passed as first argument or not. The methodpointer uses the 'of object' in the declaration.

The problem is not the codegeneration (that can be the same as done by methodpointers) but the implementation in the parser and typechecking.

If I'd to rate this task on a scale from 0 (easy) to 5 (hard) I would give it a 4 :)

This is because, as Peter mentioned, parsing and type checking of proc. vars is really hard and messy. The reason is that proc var expressions are undefined without context and that's why fpc mode requires @ or () to define exactly what's meant if a proc. var or proc name is found.

Note that, with this solution, typed procedural variables can be passed as actual procedural parameters to "iso" style formal parameters, but not the other way round.

How are variables of this type declared?

1. What happens if f would use a variable from somefun, and f is called when somefun is no longer executed ?

2. I see no difference whatsoever between typ_fun and iso_fun, except the use of an extra type, which, in my opinion, does not change anything to the usage or code of these functions. If one is allowed, the other should be allowed as well.

Michael.

That is not possible. If you take the address of f and put it in a procedural variable, you'll get an error (just like when you attempt to assign a method to a regular procedural variable). You can only pass it as parameter to procedures declared with an ISO-style procedure/function parameter, and since its scope is limited to somefun, somefun will always be active when you can do so.

2. I see no difference whatsoever between typ_fun and iso_fun, except the use of an extra type, which, in my opinion, does not change anything to the usage or code of these functions. If one is allowed, the other should be allowed as well.

Well, it currently isn't in either our compiler or in Delphi. You cannot declare a function type inside a parameter list for now. Maybe this was even done on purpose to avoid clashes with ISO-style procedure parameters.

Jonas

But the compiler can never guarantee this ?

It seems to me that the following is perfectly valid code :

Var
   StoredF : Function (x : real) : real
Procedure iso_fun( function f( x: real): real);
begin
   StoredF:=F;
end;

StoredF can then be called at will, even outside somefun.

When compiling this, there is no way the compiler can determine whether f (or storedf) will be a local procedure or a global one. To solve this, it would mean that each procedure variable would have to consist of 2 things; A pointer and a type indicator.

This would break _a lot_ of existing code. If that is the consequence, I am heavily against introducing this possibility.

Yes, but why would one be allowed and the other not ?

Michael.

Michael Van Canneyt wrote

It seems to me that the following is perfectly valid code : Var StoredF : Function (x : real) : real

This is a regular procedural variable, not an ISO-style procedural variable. Just like "var a: array of byte;" is a dynamic array and not an open array.

Procedure iso_fun( function f( x: real): real); begin StoredF:=F; end;

This assignment is invalid, it's the same as trying to assign a method to a regular procedural variable.

That is not necessary.

In general, you cannot declare complex types inside a parameter list (you also can't declare new record, array or set types inside a parameter list). That's why you can (ab)use those things to declare special things, like open arrays (and ISO-style procvars).

Jonas

This is not allowed. Only TP style is allowed with VAR, so

var stored : TSomeFunc;

When compiling this, there is no way the compiler can determine whether f (or storedf) will be a local procedure or a global one. To solve this, it would mean that each procedure variable would have to consist of 2 things; A pointer and a type indicator.

No, they are simply two procvar types, and variables (properties/fields etc) of the new type are not allowed, only params.

TP/BP was all about speed and footprint. The kind of solutions we use now with compiler modi and several kinds of procvars was simply to expensive at the time I think.

Yes this is a problem, but it can be solved by not allowing an iso style procedure parameter to be stored in a variable. It should only be allowed to be passed to another procedure or to be called.

In fact standard pascal does not allow procedure variables, only procedure parameters (note the difference). I suppose the above problem is the reason.

2. I see no difference whatsoever between typ_fun and iso_fun, except the use of an extra type, which, in my opinion, does not change anything to the usage or code of these functions. If one is allowed, the other should be allowed as well.

No, and at present there is no difference in generated code.

On could add support for nestable procedure parameters, by adding a directive 'nestable' or 'isoproc' or the like, to the procedure parameter.

But since traditionally the iso style is used in original pascal where you expect nestability, and the typed style where you want to be able to store the procedure in a var, one could add this meaning to the different syntaxes. This is then also compatible with Metrowerks Pascal.

This do not excludes for adding a directive also, for those cases where one want to combine the clearness of using procedure types with the ability to nest. Or to other language modes of fpc.

Olle

I know.

This is exactly my point: how does the compiler distinguish the two in this case ?

I don't see how, unless you add an extra parameter for the 'iso' way of doing things.

So, basically, to be able to use a local procedure as a callback,
- You must declare a function with the ISO way to specify a callback.
- This function is not usable with a global callback function. -> So if you would want to use such a function for both normal and local procedures, you would have to implement it twice !

If this is indeed the case, then it is totally ridiculous in my opinion to implement such a "feature", because if you must make special amends _anyway_, then you can just as well do it with global functions, without the local variants.

I would only see the use of being able to pass a local function as a callback if the called function can be used for both local and global callback procedures.

I understand that.

I question the usefullness of this whole "feature".

Michael.

That's not true, the above is perfectly valid (but it creates a regular procedural variable, and not an "ISO-style procedural variable", as those simply do not exist).

Jonas

Then I think standard pascal is very handicapped indeed.

Which means that the above example IS a problem.

Michael.

Note that you are talking about MacPas mode now. In the generic fpc mode (or in Delphi or TP mode), you cannot declare a procedural variable type inside a parameter list.

Jonas

var
   f : function : longint of procedure(a : longint);

:)?

The same way it distinguishes between a "procedure x(...) of object" and a "procedure x(...)". When parsing the function header and encountering such a declaration, it sets some sort of flag inside the procdef that this is an ISO-style procvar. Later, when trying to assign this to a regular procvar, you'll get a type clash.

Just like you have to implement functions which you want to be able to use as both "procedure of object" and regular procedural variables twice. We have similar problems in the compiler, with e.g. foreach and foreach_static (and foreachnodestatic).

So we do away with procedure of object also? :)

I question the usefullness of this whole "feature".

The usefulness is mainly existing code which depends on it. There was no procedure of object in those days. This is like saying that nested procedures are useless because you can make a class with methods to achieve the same result (having multiple functions access the same state, while protecting it from outside prying eyes).

Jonas

Only because the compiler currently does not support ISO-style procvars. Olle changed it to accept the syntax (so it works in cases you do not access variables of a parent), but it generates code as if it's a regular procvar (which is indeed wrong). The latter is what Adriaan wants to fix.

Jonas

I dont think this is allowed :-)

Well, you'd have to add a declaration of all local variables of the parent as well :) (and the declaration and local variables of the parents of the parent etc :)

Jonas

This is clear.

No, because normally, one never mixes 'procedure of object' with 'procedural'. You program either linear, either OOP, so you either use one or the other, never both. The distinction is also very clear.

In the case of 'local' or 'global' this distinction is, in my opinion, much less clear.

No, see above.

No, because in this case, you are comparing 2 basically different programming techniques: OOP and linear; they are fundamentally different in their practical use. I don't see much code out there which mixes the two. (I'm talking 50-50%, not 99-1%)

Beware:

I want to make clear that I am not against the concept "an sich". If the compiler was born 'out of the blue', and it supported 'iso' procedures from the very start, the problem would not exist. All procedures (local and global) could be made 'iso-aware', and you would be able to pass both local and global functions for the same procedural parameter.

The problem is that we are in the situation where the majority of existing code does not work like that, and that we have to introduce a 'schisma' in the compiler. I'm not particularly fond of this idea.

Michael.

That's another possibility. I was just naming a way to avoid the problem. You specify/implemented another, but the point was that it doesn't _have_ to be a problem :-)

Possible soultions: They have different datatypes, or a flag is set in the type definition.

It can. When a global proc is passed, the static link is set to nil.

I question the usefullness of this whole "feature".

When using a call back, you dont need to pass data to it as global variables, making code cleaner.

Olle

The credits should go to Peter, who implemented it :-)

Olle

That's not merely a possibility, that's how FPC, TP and Delphi work.

I was just naming a way to avoid the problem. You specify/implemented another, but the point was that it doesn't _have_ to be a problem :-)

You said that "Only TP style is allowed with VAR", but "Var StoredF : Function (x : real) : real;" *is* TP style.

Jonas

That is not possible, because the static link parameter is currently passed as an implicit hidden *first* parameter. If you start passing an extra first parameter with the value "nil" to procedures, you'll get all sorts of strange effects.

The PPC ABI actually prescribes that the static link parameter should always be passed in r11 (which can never be used for any other parameter, since regular parameters stop at r10), so then this works. However, in that respect we don't follow the PPC ABI, and it would also break on other processors so that's not an option.

The only possibility I see for this is to make the hidden static link parameter the last instead of the first parameter.

Jonas

No, because in this case, you are comparing 2 basically different programming techniques: OOP and linear; they are fundamentally different in their practical use.

The iso feature supports a third, namely recursive programming, which has been forgotten after C took over the world and introduced its assembly style programming.

There is a lot of existing code out there which currently are not supported by any live compiler, namely everything written in some of the mac pascals. FPC can be the remedy.

does not work like that, and that we have to introduce a 'schisma' in the compiler. I'm not particularly fond of this idea.

As I see the suggested solution, the iso stuff can be added, without affecting existing code.

Olle

I agree, but comparing the market share of Apple and Windows I think that this codebase is dwarfed by the Delphi code base :)

Please explain the "without affecting existing code" ?

Once more, I don't object to the introduction of this new 'iso' style;
I question it's usefullnes.

I will object against a solution that causes existing code to be altered in any way, such as an extra hidden parameter for all callbacks. For the ISO ones, I don't think there is any other way of doing it. As long as it is restricted to those, there is no problem...

Michael.

If it would cause all types of proc. vars. (which is doesn't as far as I understand) it could be made mac mode only.

Except when programming a compiler, apparently :)

No, because in this case, you are comparing 2 basically different programming techniques: OOP and linear; they are fundamentally different in their practical use.

With nested procedures, you are emulating parts of OOP using procedural programming. They were some form of OOP avant-la-lettre.

I don't see much code out there which mixes the two. (I'm talking 50-50%, not 99-1%)

They all have their own uses.

Then you'd get problems with C callbacks...

The problem is that we are in the situation where the majority of existing code does not work like that, and that we have to introduce a 'schisma' in the compiler. I'm not particularly fond of this idea.

The "majority of existing" code depends on whose code it is. Existing programs will not notice anything, for them declaring a procedural variable type in a parameter list will remain impossible. It's only for people using MacPas mode, who are used to these semantics, that things will change (or rather remain the same compared to other compilers).

Jonas

As far as I understood it, it would only be for ISO type i.e.

Function MyFunction (f : Func(X : Real) : Real) : Real;
begin
end;

If this is so, there is no problem.

Michael.

It's both for now:

a) it does not cause all procvars to become of this special type (only those whose type is fully declared inside a parameter list) b) it will only be available in MacPas mode

Jonas

Indeed :-)

That is a new definition :-)

Absolutely. I agree 100%; for me, it depends on the project; where OOP is more useful, use OOP. Where procedural makes more sense: use procedural.

On top of all other things :-)

Macpas mode and iso callbacks: No problem...

Michael.

It is, and those ISO types are additionally only allowed in MacPas mode :) So people using FPC or Delphi mode can't even accidentally declare an ISO-style procvar parameter (for now at least, we could of course decide to allow it in some other modes as well).

Jonas

I agree. As little ABI dependancy as possible.

The only possibility I see for this is to make the hidden static link parameter the last instead of the first parameter.

What I was thinking of was to check for nil, when the procedure parameter is about to be called. If the static link is nil, it is a global proc, and it is called as such, if not nil it is called as a local proc.

But this mean there must be two separate code chuncks which sets up the parameters, since for the local variant the first must be the static link.

Example Powerpc with static link first:
local
static link r3
param no 1 r4
...

global
param no 1 r3
...

To avoid this problem, it would be better to have the static link last.

For an ABI which passes everything on the stack this would not have been any problem since paremeters are just pushed to the stack, and the push instructions is relative. Perhaps Wirth did have this kinds of architectures in mind when designing Pascal.

Olle

Michael Van Canneyt wrote

1. What happens if f would use a variable from somefun, and f is called when somefun is no longer executed ?

This can not happen when the function parameter cannot be stored anywhere.

2. I see no difference whatsoever between typ_fun and iso_fun, except the use of an extra type, which, in my opinion, does not change anything to the usage or code of these functions. If one is allowed, the other should be allowed as well.

As I understand the ISO convention, it just shall disallow to create variables of the procedural type, which were required to store such a reference. In Pascal two type declarations are different, even if they look the same.

Let me add some more thoughts about procedural types:

- I like the ability to declare procedural types, the ISO convention looks like one of the many incredible C hacks to me :-(

- For the restricted use of local subroutines as procedural parameters I could imagine a "const" prefix in the accepting procedure declaration:

procedure my_fun(const pf: tfun);

The "const" prefix here means that the procedure pf only can be called, but cannot be stored in a variable. The compiler then can assure that local subroutines are passed only as const arguments. This syntax requires no changes to the parser. The compiler message for a missing "const" could be a warning instead of an error, to prevent compatibility/portability problems. Other compilers should ignore the "const", so that accordingly modified source code still should be portable?

Hmm, the hidden frame parameter still will make a difference with local subroutines. At least in Pascal calling convention, where the arguments are popped by the called subroutine, not by the caller...

- I'd appreciate to define procedures based on procedural types as well. Currently a change to the procedural type requires updates of all derived procedure declarations. Something like:

myproc: tfun =
begin
  blabla
end;

IMO such a definition would better implement the strict Pascal typing, instead of only a type equivalence determined by the procedure signature.

Unfortunately(?) this syntax is incompatible with procedural variables, so that it would disallow to create such variables. A "procedure" prefix would make the definition look like a function returning an TProcType. But perhaps somebody has a better idea?

DoDi

Let me add some more thoughts about procedural types: - I like the ability to declare procedural types, the ISO convention looks like one of the many incredible C hacks to me :-(

But it is standard pascal. And we need to support those zillion lines of code out there, written in standard pascal.

- For the restricted use of local subroutines as procedural parameters I could imagine a "const" prefix in the accepting procedure declaration: procedure my_fun(const pf: tfun);

This will be unclear imo, I would prefer a directive which tells what it really is about.

Hmm, the hidden frame parameter still will make a difference with local subroutines. At least in Pascal calling convention, where the arguments are popped by the called subroutine, not by the caller...

The pascal calling convention is not used on most modern processors, since parameters instead are passed in registers.

This would increase the changeability but decrease readabillity of the code. A C programmer would have liked this feature :)

Olle

Agreed, as far as I'm not pressed myself, to use constructs that I don't like ;-)

Such an explicit directive would not be portable, unless introduced by some accepted standard.

I'd be careful with "most", the most frequently used "modern" processor has anything but a modern architecture. But unfortunately the bundling of bad hardware with bad software seems to be what the consumer market appreciates :-(

DoDi

That won't work on x86 when the static link would be on the stack, because there the callee removes the parameters from the stack (so if it's a global function, it will remove sizeof(pointer) bytes too little from the stack.

Jonas

On x86, you can generate code that checks the static link pointer at runtime, to see if it is nil or not (e.g. a global function or not). Then, you pass the static link pointer as the last parameter (for local procedures) or not (for global procedures).

Regards,

Adriaan van Os

Then you can do it just as well on ppc and other processors, but the point of the trick was to avoid having to do this. Implementing this sort of hacks will complicate the code generator (I'm not even immediately sure how it would be implemented).

Jonas

This is a not possible. It is incompatible with Delphi calling conventions and not possible with register calling conventions. Also take into account that modes are per unit. And other units (like the RTL) are not in gpc/macpas mode and don't expect the extra parameter and do not remove it from the stack.

Also generating code at runtime is a no-go for me. It is heavily CPU and OS dependent.

Note: Also thinking with things like "On x86" is the wrong way for current FPC. It should be possible to support things (without hacks) on all supported CPUs.

No, that's not true.

The idea was not to push it in the first place if it's nil. The problem is at the caller side: if that parameter is not nil, that means the procvar is a local parameter and it must be pushed (or loaded into a parameter register) when calling the procvar. If it's nil, then it means the procvar is a global procedure and it must not be pushed (idem).

Also generating code at runtime is a no-go for me. It is heavily CPU and OS dependent.

No code would have to be generated, what would be required is in case of MacPas mode (and only for code compiled in MacPas mode) the insertion of a comparison with nil of a parameter and depending on that push it or not.

Note: Also thinking with things like "On x86" is the wrong way for current FPC. It should be possible to support things (without hacks) on all supported CPUs.

That's why I mentioned x86, because afaik it's the only one where the callee removes the parameters from the stack instead of the caller.

Jonas

But the framepointer parameter must then be the last. That will make macpas local procedures incompatible with the current code where it is passed as the first parameter. Things like the objects unit and FreeVision will then be unusable, without the compiler being able to warn/error you for that.

Also generating code for cdecl on x86 will become more complex:

push para1
..
push paraX
if isoprocvar.framepointernil then
push parentfp
call proc
dec stack,parasize_on_the_stack-sizeof(parentfp)
if isoprocvar.framepointernil then
dec stack sizeof(parentfp)

05-03-16 23.08, skrev Peter Vreman följande:

But the framepointer parameter must then be the last. That will make macpas local procedures incompatible with the current code where it is passed as the first parameter.

Couldn't the framepointer be last parameter in all modes ?

Things like the objects unit and FreeVision will then be unusable, without the compiler being able to warn/error you for that.

Why ? Are these special ?

But cdecl does not allow local functions at all, no ?

You dont have local functions in C, no ?

Olle

That will break existing code and be incompatible with delphi.

They have special routines to call local procedures and therefor expect that the framepointer is the first parameter.

The compiler and also delphi accepts it and generates correct code. A quick 'fix' for this will then be to disallow cdecl in macpas mode.

That would still require some ugly hacks inside the parameter passing code, because of the fact that on x86 the callee removes the parameters from the stack and not the caller (so you'd still need a check comparing that framepointer wil nil and not "pushing" it if it is nil).

Jonas

1) Local subroutines expect an framepointer, regardless of whether being called from their enclosing subroutine, or from outside (as a callback). This parameter must be passed, and possibly be removed from the stack, in every call to, and return from, a local subroutine. All the technical details are already specified, by the current handling of local calls to local subroutines.

2) The compiler must check every call for local/global subroutines, in order to supply or omit the framepointer. The only remarkable difference are static/dynamic checks, where direct calls can be handled at compile time, as is, whereas indirect calls would require a distinction at runtime, based on the local/global kind of the actual callback procedure.

So I wonder why there should be any problem, besides for the generation of the code for the runtime checks?

3) Procedural parameters already come in two flavours, for global subroutines and methods. The "of object" type has an additional Self pointer field. It should not be hard to extend the base type with an according Frame pointer field. This extension only is necessary for parameters, not for variables of procedural type, because the compiler should (already!) reject attempts to store references to local subroutines in variables. Of course this change will break compatibility with older binary code, such code must be recompiled with the new compiler version.

Such a change may allow for really "smart" callbacks, where global or local subroutines can be used for the same parameter, as well as methods can be used without an explicit "of object" clause. In the latter case distinct values must be used for the object (Self) pointer of methods, to distinguish between methods with Self=nil, and ordinary subroutines with no Self extension of the calling conventions may be a nasty task, but the compiler already has implemented a bunch of calling conventions (cdecl, pascal...) and modifications (method vs. ordinary subroutines, direct vs. indirect calls). Only the very last case needs an update, so that indirect calls have to be checked for local/global subroutines at runtime. As mentioned above (3), additional dynamic support can distinguish and handle global/local and subroutine/method variations appropriately, as 4 distinct cases.

Since such "smart" callbacks require some amount of code, increasing the size and runtime of the compiled code, it's up to the compiler writers to add an compiler mode, option, or a flag, to enable/disable smart callbacks and the according code production.

Did I miss anything?

What's your opinion on "smart" callbacks?
I mentioned this new construct based on my current observations with comparison functions in sorting procedures. Wouldn't it be nice if one could supply any kind of comparison function (local/global, subroutine/method) to a single sorting procedure? Then the comparison function could check flags and other values in their own runtime environment, e.g. to distinguish between ascending/descending sort order, in a thread-safe environment.

DoDi