Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / fpc-devel / 2014-07

Blocks support

9 messages · 3 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

Jonas Maebe

Hi,

I've (locally, not yet in svn) implemented support for blocks in FPC ( http://en.wikipedia.org/wiki/Blocks_(C_language_extension) ). Blocks are basically procedural variables with support for capturing local context in various ways. As implemented in C/C++/Objective-C, they also introduce the syntactic sugar of declaring them as inline code fragments (anonymous procdures), but I have not implemented this in FPC. Such functionality is completely orthogonal to the support for blocks, and can be added later if/when support for anonymous methods is added to FPC in general.

The current blocks support is very limited and only supports converting regular procedures/functions to blocks (i.e., not nested procedures/functions, not methods of either Pascal or Objective-Pascal classes/objects, not other procedural variables). It is also only enabled on Darwin ((Mac) OS X, iOS and iphonesim) for now, but since the blocks runtime is available on other platforms too, it can be enabled elsewhere over time.

The way I've implemented it right now is that a block-type is declared like a regular procedure variable, followed by "is block". E.g.:

type
   NSArrayEnumeratorBlock = procedure (obj: id; idx: NSUInteger; var
stop: boolean) is block;

Unlike other procedure variable types, you cannot specify the calling convention of these "is block" types, as it is determined by the blocks runtime. Also unlike other procedure variable types, the compiler will automatically generate wrappers when assigning a procedure/function to such an "is block" type, so that the calling convention of the original procedure/function doesn't matter.

A complete example:

***
{$mode objfpc}
{$modeswitch objectivec2}
{$modeswitch blocks}
uses
   CocoaAll;
type
   NSArrayEnumeratorBlock = procedure (obj: id; idx: NSUInteger; var
stop: boolean) is block;

{ normally part of the NSArray declaration, but not yet in our units since the released compilers don't support blocks yet }
enumcategory = objccategory external (NSArray)

     procedure enumerateObjectsUsingBlock(block:

NSArrayEnumeratorBlock); message 'enumerateObjectsUsingBlock:';

   end;
procedure myenumerate(obj: id; idx: NSUInteger; var stop: boolean);
begin
   NSLog(NSString.alloc.initWithUTF8String('Object at index %lu is %@'),
idx, obj);
end;
var
   arr: NSMutableArray;
   str: NSString;
begin
   arr := NSMutableArray.alloc.init;
   str:=NSString.alloc.initWithUTF8String('abc');
   arr.addObject(str);
   str:=NSString.alloc.initWithUTF8String('123');
   arr.addObject(str);
   str:=NSString.alloc.initWithUTF8String('XYZ');
   arr.addObject(str);
   arr.enumerateObjectsUsingBlock(@myenumerate);
end.

Output:

2014-07-16 09:58:53.886 blockenumerate[17438:507] Object at index 0 is abc 2014-07-16 09:58:53.913 blockenumerate[17438:507] Object at index 1 is 123 2014-07-16 09:58:53.914 blockenumerate[17438:507] Object at index 2 is XYZ

Now, to the hard part. In case of global routines, the block does not capture any local state and everything works perfectly as it is. There is however not much point to using them this way, other than with library APIs that accept blocks rather than regular procedure variables (which forms an increasing part of the OS X/iOS system interface, so it is already useful by itself).

"Capturing local state" means that the block gets a copy or a reference to a variable that is local to the context where it is instantiated (instantiated in our case means "the place where we assign the address of a procedure/function to an "is block" variable). For example, when support is added for assigning an instance method to an "is block" variable, the instance pointer must become part of the block because otherwise we don't know what self parameter to use when the block is actually invoked.

However:

* Methods of Pascal classes

As explained above, the "self" parameter has to be captured by the block in this case. If nothing special is done, then this self pointer will be treated like a regular pointer and you will be responsible for freeing it. However, in case a block is passed to an asynchronous routine, the question will most likely be "when is it safe to free that instance", and I'm not sure how to do that. The blocks runtime supports adding code that will be executed when a block is copied and when it is freed, but since Pascal classes don't have an internal reference count, there's not much we can do automatically here.

There is a way to make a Pascal class reference counted, by making it a descendent of TInterfacedObject (or more generally, by implementing the IUnknown interface) and only handling it via interface variables, so maybe the compiler could generate special code in that case to make sure that the block does update the reference counts in that case. That way, you can have a choice between manual and automatic reference counting. I still have to think this through to determine whether this reference counted approach is in fact feasible to implement.

* Methods of Objective-C classes

Objective-C doesn't support function pointers to Objective-C methods (as far as I know), so right now Objective-Pascal doesn't support procedure variables that reference Objective-C methods either. If this is changed because of blocks, they should be supported in general for orthogonality reasons.

This would also require capturing the "self" parameter, but since Objective-C classes to come with reference counting (in various ways), the reference counting would be handled automatically behind the scenes. On the other hand, such a capture could result in cyclic references (https://developer.apple.com/library/ios/documentation/Cocoa/Conceptual/ProgrammingWithObjectiveC/WorkingwithBlocks/WorkingwithBlocks.html#//apple_ref/doc/uid/TP40011210-CH8-SW16 )

* Local/nested procedures

These most closely resemble blocks as they exist in C, since they can directly access all local variables of their enclosing scope (rather than only a specific value such as the self pointer).

The default in C when accessing a local variable from the enclosing scope within a block depends on the type and declaration of the variable:
1) an Objective-C class or a variable marked to behave as such via __attribute__((NSObject)) (e.g. a CFString), or another block-typed variable: reference counts are automatically updated by the compiler/runtime -- except if that variable is declared as __weak, then no reference counting is performed
2) the variable is marked as "__block". In this case, the compiler and runtime use indirection (and if needed, copying onto the heap and updating all redirections) to make sure that the local variable can survive the scope in which it was declared, so that the block (and copies thereof) can keep using it even after the function in which the local variable was declared, returns
3) all other variables (non-block/object and not marked as __block) accessed by a block are simply copied into local storage of the block when the block is instantiated (in our case that would be: when the procedure pointer is assigned to to the "is block" variable). Modifications to such variables in the original routine after the block has been instantiated are not visible to the block and vice verse. If a copy is made of the block, then the current value of those variables within that first block are copied and afterwards again become private to the copy.

I currently have no idea how to encapsulate this in Pascal. We could add __block and __weak modifiers (the __weak modifier would also be useful for non-block-related Objective-Pascal once we add support for ARC/Automated Reference Counting), but that's not really nice.

Another option is to only support by-value copies, and basically always use cases 1) and 3) above depending on the type of the variable. If you want to change something by reference, then you would have to manually set up a pointer (or object instance) first.

Or maybe someone has another idea that combines the flexibility with a Pascalish syntax/way of working?

Jonas

Reply by email

Sven Barth
to Jonas Maebe
"Jonas Maebe" wrote
Hi, I've (locally, not yet in svn) implemented support for blocks in FPC (

http://en.wikipedia.org/wiki/Blocks_(C_language_extension) ). Blocks are basically procedural variables with support for capturing local context in various ways. As implemented in C/C++/Objective-C, they also introduce the syntactic sugar of declaring them as inline code fragments (anonymous procdures), but I have not implemented this in FPC. Such functionality is completely orthogonal to the support for blocks, and can be added later if/when support for anonymous methods is added to FPC in general.

Woah O.o Didn't expect that ^^ Maybe some of this can be reused for anonymous methods support :)

The way I've implemented it right now is that a block-type is declared

like a regular procedure variable, followed by "is block". E.g.:

type NSArrayEnumeratorBlock = procedure (obj: id; idx: NSUInteger; var stop:
boolean) is block;

As I see below it's part of a modeswitch, right?

Now, to the hard part. In case of global routines, the block does not

capture any local state and everything works perfectly as it is. There is however not much point to using them this way, other than with library APIs that accept blocks rather than regular procedure variables (which forms an increasing part of the OS X/iOS system interface, so it is already useful by itself).

"Capturing local state" means that the block gets a copy or a reference

to a variable that is local to the context where it is instantiated (instantiated in our case means "the place where we assign the address of a procedure/function to an "is block" variable).

Delphi is a bit confusing here: each anonymous function instantiated in a function shares a reference to a copy of the variable... :/ (at least if I remember correctly)

Show quoted text 5 lines
However: * Methods of Pascal classes As explained above, the "self" parameter has to be captured by the block

in this case. If nothing special is done, then this self pointer will be treated like a regular pointer and you will be responsible for freeing it. However, in case a block is passed to an asynchronous routine, the question will most likely be "when is it safe to free that instance", and I'm not sure how to do that. The blocks runtime supports adding code that will be executed when a block is copied and when it is freed, but since Pascal classes don't have an internal reference count, there's not much we can do automatically here.

There is a way to make a Pascal class reference counted, by making it a

descendent of TInterfacedObject (or more generally, by implementing the IUnknown interface) and only handling it via interface variables, so maybe the compiler could generate special code in that case to make sure that the block does update the reference counts in that case. That way, you can have a choice between manual and automatic reference counting. I still have to think this through to determine whether this reference counted approach is in fact feasible to implement.

We would need reference counting support anyway for Pascal strings and arrays if we'd allow it on non-Mac OS X (btw: does invoking in Pascal code also work correctly? Your example does not show this...). And interfaces and ARC classes will "simply" need that as well... For normal classes I'd say that it's the responsibility of the programmer to free them, but maybe we could somehow provide access to that "after use" functionality you mentioned (e.g. another function that can be assigned or something like that, though I don't have a good syntax idea right now)

I currently have no idea how to encapsulate this in Pascal. We could add

__block and __weak modifiers (the __weak modifier would also be useful for non-block-related Objective-Pascal once we add support for ARC/Automated Reference Counting), but that's not really nice.

Another option is to only support by-value copies, and basically always

use cases 1) and 3) above depending on the type of the variable. If you want to change something by reference, then you would have to manually set up a pointer (or object instance) first.

I could (at least on first sight) live with that.

Or maybe someone has another idea that combines the flexibility with a

Pascalish syntax/way of working?

I'll need to think about this a bit...

Regards,
Sven

Reply by email

Jonas Maebe
to Sven Barth
Sven Barth wrote 15 lines
Am 16.07.2014 14:13 schrieb "Jonas Maebe" < >: > > I've (locally, not yet in svn) implemented support for blocks in FPC > ( http://en.wikipedia.org/wiki/Blocks_(C_language_extension) ). Blocks > are basically procedural variables with support for capturing local > context in various ways. As implemented in C/C++/Objective-C, they also > introduce the syntactic sugar of declaring them as inline code fragments > (anonymous procdures), but I have not implemented this in FPC. Such > functionality is completely orthogonal to the support for blocks, and > can be added later if/when support for anonymous methods is added to FPC > in general. Woah O.o Didn't expect that ^^ Maybe some of this can be reused for anonymous methods support :)

Indeed (especially after the changes I've added below).

Show quoted text 8 lines
> The way I've implemented it right now is that a block-type is > declared like a regular procedure variable, followed by "is block". E.g.: > > type > NSArrayEnumeratorBlock = procedure (obj: id; idx: NSUInteger; var > stop: boolean) is block; As I see below it's part of a modeswitch, right?

Yes, since it's an extension.

Show quoted text 8 lines
> "Capturing local state" means that the block gets a copy or a > reference to a variable that is local to the context where it is > instantiated (instantiated in our case means "the place where we assign > the address of a procedure/function to an "is block" variable). Delphi is a bit confusing here: each anonymous function instantiated in a function shares a reference to a copy of the variable... :/ (at least if I remember correctly)

While working on this, I didn't realise at all that Delphi had to have rules already regarding capturing local variables for its own anonymous method support. I've now read what it does and it indeed always has the same behaviour as variables declared as "__block" in C blocks. So I would propose to use this behaviour for consistency and leave out support for capturing local variables as const/snapshot values.

Similarly, they also already have a syntax for "block type" variables (var proc: reference to function(l: longint): longint;"), so I propose to use that one as well instead of my "is block". Possibly with "cdecl;" at the end to differentiate between C blocks and whatever the FPC implementation will be, since some people will probably want to reimplement the blocks runtime, or something similar, in Pascal (definitely not me).

Show quoted text 25 lines
> However: > > * Methods of Pascal classes > > As explained above, the "self" parameter has to be captured by the > block in this case. If nothing special is done, then this self pointer > will be treated like a regular pointer and you will be responsible for > freeing it. However, in case a block is passed to an asynchronous > routine, the question will most likely be "when is it safe to free that > instance", and I'm not sure how to do that. The blocks runtime supports > adding code that will be executed when a block is copied and when it is > freed, but since Pascal classes don't have an internal reference count, > there's not much we can do automatically here. > > There is a way to make a Pascal class reference counted, by making it > a descendent of TInterfacedObject (or more generally, by implementing > the IUnknown interface) and only handling it via interface variables, so > maybe the compiler could generate special code in that case to make sure > that the block does update the reference counts in that case. That way, > you can have a choice between manual and automatic reference counting. I > still have to think this through to determine whether this reference > counted approach is in fact feasible to implement. We would need reference counting support anyway for Pascal strings and arrays if we'd allow it on non-Mac OS X

I don't understand this. As mentioned, there are block-specific callbacks that are automatically invoked whenever a block is copied or destroyed (which has to be performed by ABI-defined helpers). In those callbacks (which are compiler-generated), the compiler can put code to handle the reference counting of reference-counted types. The problem is only with non-reference-counted types such as classes.

An implementation not based on the blocks runtime would indeed need the same functionality, but it would have exactly the same problem (it can't reference count arbitrary class instances). Note that there is an open source blocks run time that is not specific to Mac OS X; there are at least Linux and FreeBSD ports of it, both supported by clang/llvm on those platforms (but of course it depends on libc).

(btw: does invoking in Pascal code also work correctly? Your example does not show this...).

Yes. Invoking a block is the easy part :) You just call the "invoke" function pointer in the block record, passing the address of the block record as hidden first parameter.

Show quoted text 6 lines
And interfaces and ARC classes will "simply" need that as well... For normal classes I'd say that it's the responsibility of the programmer to free them, but maybe we could somehow provide access to that "after use" functionality you mentioned (e.g. another function that can be assigned or something like that, though I don't have a good syntax idea right now)

There's no callback that is called only when the last instance of a block is destroyed. Although I guess we could do that ourselves by inserting a reference count of our own as part of the "captured" data... The syntax could be done similar as with enumeration callbacks: provide a static class method with a particular name that takes an instance of that class as parameter and if it exists, the compiler will generate the cleanup call/reference counting stuff and call the method with the instance as parameter).

This is definitely not something for a first implementation though.

Jonas

PS: the blocks ABI and their behaviour is fairly well documented at http://clang.llvm.org/docs/Block-ABI-Apple.html

Reply by email

Sven Barth
to Jonas Maebe
Jonas Maebe wrote 15 lines
> > "Capturing local state" means that the block gets a copy or a > > reference to a variable that is local to the context where it is > > instantiated (instantiated in our case means "the place where we assign > > the address of a procedure/function to an "is block" variable). > > Delphi is a bit confusing here: each anonymous function instantiated in > a function shares a reference to a copy of the variable... :/ (at least > if I remember correctly) While working on this, I didn't realise at all that Delphi had to have rules already regarding capturing local variables for its own anonymous method support. I've now read what it does and it indeed always has the same behaviour as variables declared as "__block" in C blocks. So I would propose to use this behaviour for consistency and leave out support for capturing local variables as const/snapshot values.

It would however be good if we'd at least have support for const/snapshot values so that we could easily add it once we come up with a suitable syntax.

Show quoted text 7 lines
Similarly, they also already have a syntax for "block type" variables (var proc: reference to function(l: longint): longint;"), so I propose to use that one as well instead of my "is block". Possibly with "cdecl;" at the end to differentiate between C blocks and whatever the FPC implementation will be, since some people will probably want to reimplement the blocks runtime, or something similar, in Pascal (definitely not me).

I'm not that sure whether using the same syntax is a good idea... people want to have Delphi compatible anonymous functions which are basically reference counted interfaces instead of records...

Show quoted text 35 lines
> > However: > > > > * Methods of Pascal classes > > > > As explained above, the "self" parameter has to be captured by the > > block in this case. If nothing special is done, then this self pointer > > will be treated like a regular pointer and you will be responsible for > > freeing it. However, in case a block is passed to an asynchronous > > routine, the question will most likely be "when is it safe to free that > > instance", and I'm not sure how to do that. The blocks runtime supports > > adding code that will be executed when a block is copied and when it is > > freed, but since Pascal classes don't have an internal reference count, > > there's not much we can do automatically here. > > > > There is a way to make a Pascal class reference counted, by making it > > a descendent of TInterfacedObject (or more generally, by implementing > > the IUnknown interface) and only handling it via interface > variables, so > > maybe the compiler could generate special code in that case to make > sure > > that the block does update the reference counts in that case. That way, > > you can have a choice between manual and automatic reference > counting. I > > still have to think this through to determine whether this reference > > counted approach is in fact feasible to implement. > > We would need reference counting support anyway for Pascal strings and > arrays if we'd allow it on non-Mac OS X I don't understand this. As mentioned, there are block-specific callbacks that are automatically invoked whenever a block is copied or destroyed (which has to be performed by ABI-defined helpers). In those callbacks (which are compiler-generated), the compiler can put code to handle the reference counting of reference-counted types. The problem is only with non-reference-counted types such as classes.

Ah, they are block specific. That wasn't that clear to me before. *blush* Ok, then the compiler can generate code for the reference counting (which it needs to do anyway) and we just leave non-ARC classes be... Delphi doesn't handle them specially either.

Show quoted text 15 lines
> And > interfaces and ARC classes will "simply" need that as well... For normal > classes I'd say that it's the responsibility of the programmer to free > them, but maybe we could somehow provide access to that "after use" > functionality you mentioned (e.g. another function that can be assigned > or something like that, though I don't have a good syntax idea right now) There's no callback that is called only when the last instance of a block is destroyed. Although I guess we could do that ourselves by inserting a reference count of our own as part of the "captured" data... The syntax could be done similar as with enumeration callbacks: provide a static class method with a particular name that takes an instance of that class as parameter and if it exists, the compiler will generate the cleanup call/reference counting stuff and call the method with the instance as parameter).

Yes, I mixed that callback stuff up a bit...

PS: the blocks ABI and their behaviour is fairly well documented at http://clang.llvm.org/docs/Block-ABI-Apple.html

Thanks. :)

Regards,
Sven

Reply by email

Jonas Maebe
to Sven Barth
Sven Barth wrote 11 lines
On 16.07.2014 18:45, Jonas Maebe wrote: > While working on this, I didn't realise at all that Delphi had to have > rules already regarding capturing local variables for its own anonymous > method support. I've now read what it does and it indeed always has the > same behaviour as variables declared as "__block" in C blocks. So I > would propose to use this behaviour for consistency and leave out > support for capturing local variables as const/snapshot values. It would however be good if we'd at least have support for const/snapshot values so that we could easily add it once we come up with a suitable syntax.

The fewer kinds of behaviours are possible, the easier it is to see what the code does. I don't mind not supporting can also be easily added later if really necessary (it's mainly a matter of /not/ generating certain metadata and copy/release helpers that we would generate otherwise).

Show quoted text 11 lines
> Similarly, they also already have a syntax for "block type" variables > (var proc: reference to function(l: longint): longint;"), so I propose > to use that one as well instead of my "is block". Possibly with "cdecl;" > at the end to differentiate between C blocks and whatever the FPC > implementation will be, since some people will probably want to > reimplement the blocks runtime, or something similar, in Pascal > (definitely not me). I'm not that sure whether using the same syntax is a good idea... people want to have Delphi compatible anonymous functions which are basically reference counted interfaces instead of records...

Well, the block variable that's passed around is also a pointer. The record it points to can either be non-reference counted (in case it's a block for a global procedure/function, since those don't have context and hence can be reused for all instances) or reference counted. In the latter case, they start life on the stack, and are copied to the heap if necessary by the run time based on the copy/release calls that are made). In a sense, they behave a bit like an interface.

Ah, they are block specific. That wasn't that clear to me before. *blush*

Sorry, I've been with my head in the blocks documentation since the start of the week, so I may not have been explicit regarding some things that I took implicitly for granted in the mean time.

Ok, then the compiler can generate code for the reference counting (which it needs to do anyway) and we just leave non-ARC classes be... Delphi doesn't handle them specially either.

Ok.

Jonas

Reply by email

Jonas Maebe
to Jonas Maebe
Jonas Maebe wrote 12 lines
On 16/07/14 21:19, Sven Barth wrote: > I'm not that sure whether using the same syntax is a good idea... people > want to have Delphi compatible anonymous functions which are basically > reference counted interfaces instead of records... Well, the block variable that's passed around is also a pointer. The record it points to can either be non-reference counted (in case it's a block for a global procedure/function, since those don't have context and hence can be reused for all instances) or reference counted. In the latter case, they start life on the stack, and are copied to the heap if necessary by the run time based on the copy/release calls that are made). In a sense, they behave a bit like an interface.

One definite incompatibility between blocks-based and other "reference to" functions right now is that the former do not accept types that are reference counted in Pascal. The reason is that when those blocks would be called from another language, the reference count wouldn't be updated. While other languages don't know about e.g. FPC-implemented ansistrings, some people would definitely start passing such ansistrings as pointers to outside code, so that this code could then pass it in again to a block invoked as a callback.

Jonas

Reply by email

Jonas Maebe
to Maciej Izak
Maciej Izak wrote 17 lines
Jonas Maebe wrote: > type > NSArrayEnumeratorBlock = procedure (obj: id; idx: NSUInteger; var > stop: boolean) is block; What you think about more oxygene compatible syntax for blocks? : *** type ClickEventDelegate = block (obj: id; idx: NSUInteger; var stop: boolean); *** IMHO more readable :) http://www.elementswiki.com/en/Blocks

As mentioned in a previous mail, I'll go with the Delphi-derived syntax ("reference to procedure (obj: id; idx: NSUInteger; var stop: boolean) is block; cdecl;").

The advantage of "... is block" over "block ..." would have been that it was consistent with the existing way to declare procedure variables for nested procvars ("... is nested"). I also don't think we have any syntactical elements taken from Oxygene yet, so let's not start making the language more complicated by mixing in things from even more sources in case one of the existing influences already has a syntax for it.

Jonas

Reply by email

Jonas Maebe
to Jonas Maebe: I've (locally, not yet in svn) implemented support for blocks in FPC (…
Jonas Maebe wrote
I've (locally, not yet in svn) implemented support for blocks in FPC ( http://en.wikipedia.org/wiki/Blocks_(C_language_extension) ).

I've now committed the initial implementation in the blocks branch on svn. As discussed on the fpc-devel list, the syntax has been changed to:

type
   NSArrayEnumeratorBlock = reference to procedure (obj: id; idx:
NSUInteger; var stop: boolean); cdecl;

The reason is that Delphi already introduced the "reference to" syntax for anonymous functions in Pascal. The extra "cdecl;" indicates that it's the C-variant of an anonymous function (i.e., a "block").

Apart from supporting global routines, the current implementation also already supports assigning the address of a method of a Pascal class/object to a block variable (or passing it to a block parameter). Nested functions/procedures are not yet supported.

Jonas

Reply by email