While trying to find a nice solution for generating interrupt vector tables on ARM, I found that the usually employed solution is to generate weak references to symbols, that are initialized to a single handler. This way an application can override the weak reference, by declaring a new symbol with the same name. While it's possible to do that currently by using some external assembler code, I think a simpler solution could be made by using the weakexternal directive. The infrastructure is already there, but it's used in a slightly different way.

An added bonus of using this new way is that IRQs will branch directly to the handler, instead of first branching to some code that loads the handler address, and then branching to the handler

I've modified the syntax for this directive slightly, so it can take the form "weakexternal ['library'] [name 'name'] [set 'initialvalue'];" instead of "weakexternal ['library'] [name 'name'];"

A short example:
     procedure NoHandler;
     begin
     end;
     procedure TestProc; weakexternal name 'TestProc' set 'NoHandler';
     ...
     begin
        TestProc; // This will call NoHandler
     end.
If we modify the example:
     procedure NoHandler;
     begin
     end;
     procedure TestProcHandler; [public, alias: 'TestProc'];
     begin
     end;
     procedure TestProc; weakexternal name 'TestProc' set 'NoHandler';
     ...
     begin
        TestProc; // This will call TestProcHandler
     end.

All this is done either at compile time, or link time

What do you think about this proposal?

I've put a patch up here: http://j-software.dk/new/weakexternal.txt And a small example of how lpc21x4.pp, for example, could be modified to use the new functionality: http://j-software.dk/new/lpc21x4.txt

As I understand NoHandler will be called if we call a TestProc routine but 'TestProc' symbol is is not found?

If so then:
a) NoHandler must have the same arguments and calling convention as TestProc?
b) What happen if NoHandler is not found?

Best regards,
Paul Ishenin

With this solution, no. This uses only symbol names

b) What happen if NoHandler is not found?

Then TestProc cannot be resolved to any symbol, and will generate a weak undefined reference. Eg. it won't complain about an undefined reference

Jeppe Johansen wrote

> a) NoHandler must have the same arguments and calling convention as > TestProc? With this solution, no. This uses only symbol names

This I don't understand. What happen when I call

DoSomething(1, 'test');

But DoSomething is not resolved and instead
HandleDoSomethingUnresolved() is called? Is there a chance to access my arguments from that handler or no?

What happen if I pass some argument by reference and expect a result or do a function call:

Value := CallSomething(); // What Value equal to if CallSomething is not
resolved?

> b) What happen if NoHandler is not found? Then TestProc cannot be resolved to any symbol, and will generate a weak undefined reference. Eg. it won't complain about an undefined reference

I don't see where you reference the "weakinitial" symbol in your patch that's why I'm asking this question. If NoHandler is not referenced it will be stripped away during the linking process?

Generally I want to have an ability to handle a case where I call an unresolved symbol but I want to be sure that I receive arguments and my handler will not be striped from the executable.

I'd better suggest the next syntax:

procedure NoHandler;
begin
end;
procedure TestProc; weakexternal name 'TestProc' set NoHandler;

In this case I'm sure that:
1. NoHandler is a valid identifier which points to a procedure.
2. I can check that NoHandler has compatible with TestProc definition.
3. I can reference NoHandler so it will not be removed away from the executable.

Although I may be just don't understand your proposal.

Best regards,
Paul Ishenin

Very good idea. It would make alot of sense to do that

Regarding referencing to prevent removal:
weakinitial is referenced through the .set directive here AsmWriteLn('.set'#9+tasmsymbol(current_asmdata.AsmSymbolDict[i]).Name+','+tasmsymbol(current_asmdata.AsmSymbolDict[i]).weakinitial.Name)

This explicitly tells the linker that either the label TestProc has the value weakinitial.name; or if the label TestProc is defined somewhere else take on that value. IfTestProc exists then there would be no need for NoHandler, so it would be fine if it was stripped from the executable. If neither isn't there we would have a problem of course, which your idea would solve

Jeppe Johansen wrote

What do you think about this proposal?

Is the current solution of using procedure variables so bad? Or what does it lack?

I just think it's more elegant :)

Especially on Cortex M3 devices where you usually have over 40 IRQ vector entries(84 for stm32f103). It might seem like a little much work when it's only 5 pointers. But the space required quickly grows when you also calculate the cost for initialization into it, which the new solution doesn't require

12.01.2011 16:34, Florian Klaempfl пишет:

Is the current solution of using procedure variables so bad? Or what does it lack?

I see the next benefits from WeakExternal:

1. Less accidental errors. Which procedural variables you need to care about the initial assignment to nil, then to use correct LoadLibrary handle, then you should not forget to FreeLibrary.

2. Less code to write - you don't need an initialization, finalization code.

3. Someone can accidentally reassign your procedural variable to something else.

4. I don't know exactly how this work but suspect that if your call failed because you don't have some routine in an external library then you can update the library without relaunching an application and the second call to this routine will not fail. I can be wrong regards this but as I understand this weak linking works on os level so some magic can be applied.

Best regards,
Paul Ishenin

That's not correct. You must write all the default handlers.

For each procedure signature you must write a handler. So you must write more code. And if you want a correct error message at runtime, then you must write a handler per weakexternal.

I am not against this feature, but this will definitely result in more code.

Michael.

Only if you choose to use weakexternal with a set default symbol for dynamic linking. I personally don't think that's a good idea in many cases.

This change doesn't require you to to set the default symbol, and if you don't it'll work like a normal weakexternal reference(eg. be nil and not warn you if it's undefined)

How is it normally done in C? At first sight, using the weakexternal directive to /define/ symbols seems like a wrong approach.

Jonas

With C it's done using compiler dependent attributes. With gcc you can do it using __attribute__((weak,alias("SomeFunction")));. Though often it'll just be done using assembler using the .weak and .set/thumb_set directives

The weakexternal directive already "defines" a symbol, it will exist at runtime, but it might be nil. The only difference is that if you set the initial value it'll resolve to something at link time

FPC also has an "alias" procedural directive, so that one should probably be used instead. Giving weakexternal (or anything else) a syntax that is not used for any other procedure directive is not a good idea because it reduces uniformity.

The weakexternal directive already "defines" a symbol, it will exist at runtime, but it might be nil.

What I meant is that there is a semantic difference between referencing a symbol that may or may not be defined somewhere else (regardless of whether a programmer or the linker creates the referenced symbol), and defining a new symbol. This is independent of whether a symbol is weak or not.

Defining a weak symbol also exists, but is not yet supported by FPC. In fact, the example you gave would seem more logical to me as follows (assuming that FPC could define "weak" symbols using the "weak" modifier):

   procedure IRQHandler; weak;
     begin
       { default dummy implementation, the "NoHandler" from your example }
     end;
   procedure MyIRQHandler; public; alias: 'IRQHandler';
     begin
       { real implementation, overrides the weak definition of IRQHandler; if there
         is no such definition, the dummy one is used }
     end;

(and of course, if we'd have a plain "weak" modifier, then "weakexternal;" could be "weak; external;", although this is annoying to still change because "weakexternal" support is already in released FPC versions)

If you have no control over the original definition and it is not weak (maybe the reason you did that is because FPC does not yet support defining weak symbols), then we indeed need some way to define weak symbols whose "default value" is something else than nil, because I guess wrapping is not good enough (although it's not that bad, since it only results in an extra call in the case where the weak symbol is not replaced):

   procedure NoHandler;
     begin
       { dummy implementation }
     end;
   procedure MyIRQHandler; weak;
     begin
       { default behaviour: one extra call }
       NoHandler;
     end;
   { overriding implementation }
   procedure RealIRQHandler; public; alias: 'MyIRQHandler';
     begin
       { real code }
     end;

I can't immediately think of a good syntax to directly define MyIRQHandler as a weak symbol with the address of NoHandler as default value. Maybe this, although I'm not 100% happy with it either:

   procedure MyIRQHandler; weak 'NoHandler';

Jonas

Maybe something like instead? Requiring a symbol reference to a matching implementation of a default value, like Paul suggested, would make the solution more Pascal-like

   procedure NoHandler;
     begin
       { dummy implementation }
     end;
   { Declaration NoHandler must exist and match }
   procedure MyIRQHandler; weak NoHandler;

{ This will emit just a weak reference, the same as weakexternal; Not guaranteed to be resolved after compiletime }

   procedure MyIRQHandler2; weak;

This of course just introduces a new keyword weak, which was why I tried to base it off the weakexternal keyword

The type checking will have to be delayed, since you can only compare the procdef types after the entire second declaration has been parsed (there could e.g. still be calling convention modifiers after that).

{ This will emit just a weak reference, the same as weakexternal; Not guaranteed to be resolved after compiletime } procedure MyIRQHandler2; weak; This of course just introduces a new keyword weak, which was why I tried to base it off the weakexternal keyword

This second example cannot be the same as "weakexternal". Just like a regular "external" declaration does not generate any data, code or reference by itself, a "weakexternal" declaration does not do so either. external/weakexternal are simply Pascal-level aliases that can be used to refer to non-Pascal external symbols, a bit like "absolute" can be used for Pascal symbols.

I don't think that "procedure MyIRQHandler2; weak;" without any implementation should be allowed. It would create a weak symbol in the middle of nowhere, not associated with any particular code or data (and hence probably merged by the linker with the next symbol in the section). It would be the same as allowing "procedure MyIRQHandler2;" without any code or "(weak)external".

Jonas

Okay. Do you have any tip on where the best place to compare them would be? The initial proposal did it in read_proc, but if the implementation of the procedure comes after, this probably wouldn't be the right place to do it

The following is allowed currently, even if no Test symbol exists in the linked objects. The declaration doesn't emit anything, except for a .weak Test assembler directive if it's used. If Test is not used, nothing will be emitted

    procedure Test; weakexternal;
    procedure SomeOtherFunction;
    begin
       Test;
    end.

But I think you're right, that it would make a lot of sense not to allow it

After thinking about this proposal for some time, I've decided that I like the original proposal better than the one that involved typechecked qualified procedure identifiers

I think it might be a bad idea to apply high-level semantics to it, because it's not crossplatform(COFF doesn't seem to support it), and it's already something that only makes sense to use in a limit amount of cases.

I did a little change to the patch though. I changed the syntax to be "weakexternal ['library'] [name 'name'] [default 'initialvalue'];" since I figured default made more sense than set, in the given context. I still think using weakexternal makes sense(over adding a new "weak" keyword), since the meaning and functionality still is the same

You can find the slightly modified patch here:
http://j-software.dk/weakexternal3.patch.txt

It's not the same.

"weak" (without external) would be a definition, which means that it would be used in a situation like this:

   procedure Test; weak;
     begin
       writeln('test');
     end;

And the compiler would then generate something like this:

.weak Test
Test:
<code for test>

This would however be invalid code:

   procedure Test; weak; external; // or "weakexternal" instead of
"weak; external;"
     begin
     end;

And this would be an incomplete declaration (missing function body):

   procedure Test; weak;

That's why it was not a good idea (from me) to introduce "weakexternal", since "weak" combined with "external" would be the same (even if "weak" by itself weren't supported immediately). Furthermore, the whole "default 'initialvalue'" stuff is actually superfluous, since it basically is already part of the weak(external) declaration:

   procedure Test; weakexternal name 'aliasedTest';

could be defined to generate this:

.weak Test
.set Test,aliasedTest

After all, the above states that references to "Test" should be treated as weak references to a symbol called "aliasedTest". It would also be equivalent to what C does. I also don't think it would break much, if any, existing code that uses "weakexternal" if the semantics were to be redefined in this way.

Jonas

Sure, the weak declaration as a procedure directive could be added, and it would indeed be something different from a weakexternal procedure. Weakexternal should stay as it is, a procedure declaration directive. I can see how a "weak" directive could be useful, but I don't need that. I need a linker directive that points to one of two external functions

The reason weakexternal makes sense is because it's a weak undefined symbol reference to some external symbol, which - if it doesn't exist - points to some other external reference. A weak procedure would be almost the same, you just add the default reference as code explicitly

The problem with weak is that it's tagged onto a piece of code. So in a case like the one that originally provoked the proposal

   procedure EmptyFunc;
     begin // Empty function
     end;
procedure Func1; weakexternal name 'Func1' default 'EmptyFunc';
procedure Func2; weakexternal name 'Func2' default 'EmptyFunc';

using weak you would need two procedures. This would waste the space required for the function prologues/epilogues

   procedure Func1; weak;
     begin //Empty function
     end;
   procedure Func2; weak;
     begin //Another empty function
     end;

It's some of the same functionality, but there's still a difference. It's two different use cases

A "weakexternal" declaration by itself is not a reference (weak/ undefined or not), neither at the Pascal, C nor at the assembler/ object file level. It only says that *if* there is a reference to that symbol, then this reference should be treated as a weak reference.

which - if it doesn't exist - points to some other external reference. A weak procedure would be almost the same, you just add the default reference as code explicitly

It's the same as the difference between these two declarations in C:

int SayHello(void) __attribute__((weak))
{
...
}

and

extern int SayHello() __attribute__((weak));
or
extern int SayHello() __attribute__((weak,alias 'OtherSayHello'));

(this second "alias" form is not supported by all object formats though -- e.g. gcc on Mac OS X warns that it's ignored --, but at the Pascal level it can be emulated by the compiler just like for non-weak alias definitions)

No, because you would be able to combine "weak" and "external" to get the same result as with "weakexternal". Furthermore, there's also the "alias" modifier to add symbol aliases to a definition.

I'm also not sure why you made this remark, because at no point I suggested removing the functionality currently offered by "weakexternal". I only proposed to make the syntax more orthogonal, and to extend the semantics of "weakexternal;"/"weak; external;" so that the "default 'xxx'" specifier is not required while still getting the same effect.

Jonas

Some addenda...

Jonas Maebe wrote

A "weakexternal" declaration by itself is not a reference (weak/ undefined or not), neither at the Pascal, C nor at the assembler/ object file level.

This is not correct for all platforms and for all declarations. It's true for function declarations in C, but not for data declarations in C when targetting ELF platforms (there a "weak external" declaration in a C file does result in a reference from the object file's symbol table). It's also not correct for the assembler/object file level (also results in a reference).

This actually doesn't work. It seems gcc only supports weak definitions of data symbols. For functions, you have to say that it's weak when (externally or not) declaring it, it doesn't work anymore as part of the definition:

1) for data, this works:

int global __attribute__((weak)) = 999;

2) for code, only this works:

int SayHello() __attribute__((weak));

int SayHello() {
}

I don't see why we wouldn't support them both for code and data in FPC though.

Jonas

We mean the same here

I think you're right. But aliasing weak external references isn't the same as initializing them

Okay, I understand what you mean. But then something more than "weak" functionality should be added. Aliasing isn't the same, that's what the "default" part does

   procedure EmptyFunc; [public, alias: 'EmptyFunc'];
     begin // Empty function
     end;
procedure Func1; weak; external name 'Func1'; default 'EmptyFunc';
procedure Func2; weak; external name 'Func2'; default 'EmptyFunc';

But I think this way it doesn't make sense. Basically it's just a way to specify a .set assembler directive(which to my knowledge only works for weak external symbols). So there's really no point in having an extra directive. It would be the same as having an extra directive for the import number for windows dll exports in my opinion

The string after "name" always defines the name of what is referred to by the external declaration, not to the name of whatever symbol you are currently declaring. So the "default" actually means the same as what "external" already means. We cannot change that specifically for weak externals (regardless of the used syntax), as it would be inconsistent with how regular external symbols are defined.

To add assembler names to a symbol definition, we use "alias" in FPC: it means "define an alias with name 'name' for the current function".

I.e., you original example would become:

    procedure NoHandler;
    begin
      // translated into:
      //   .globl NoHandler
      //   .NoHandler:
    end;
    procedure TestProcHandler; [public, alias: 'TestProc'];
    begin
      // translated into:
      //   .globl P$TestProcHandler
      //   P$TestProcHandler:
      //   .globl TestProc
      //   TestProc:
    end;

... and then in some other unit, I guess (I'm not sure what the interaction between the earlier regular "TestProc" defintion and the weak "TestProc" declaration now would be otherwise):

    procedure TestProc; weak; external name NoHandler'; alias:
'TestProc';
    // translated into:
    //  a) on Linux
    //   .weak P$TestProc
    //   .set  P$TestProc, NoHandler
    //   .weak TestProc
    //   .set  TestProc, NoHandler
    //  b) on Mac OS X (and possibly on other non-ELF platforms):
    //   fail compilation, since weak symbol aliases don't exist and
there is no way to emulate this in the compiler
    //   (neither the specified name nor "alias" can be supported here
for weak externals)
    ...
    begin
       TestProc; // This will call TestProcHandler
    end;

It has to be handled at the linker level, because the compiler has no view of all code involved (there may be non-Pascal code too).

Jonas

Okay. I get what you mean now

What about allowing a new keyword "weak" and let it have two forms(something like you suggested back in one of the mails from 12th january)? Name mangling is an issue of course. It would be easiest to handle if weak symbols simply weren't name mangled(something like cdecl and co)

procedure TestProc; weak; // This produces a weak symbol
   begin
   end;
procedure TestProc2; weak default 'SomeDefaultFunction'; // Weak+default
makes the declaration external by default
procedure TestProc3; weak default 'SomeDefaultFunction'; external name
'Test_TestProc3'; // Overcome eventual name mangling problems. Symbol is
called Test_TestProc3 and is linked to SomeDefaultFunction if not
resolved at linktime
procedure TestProc4; weak; external; // Replaces/does the same as
weakexternal

Jeppe Johansen wrote

What about allowing a new keyword "weak" and let it have two forms(something like you suggested back in one of the mails from 12th january)? Name mangling is an issue of course. It would be easiest to handle if weak symbols simply weren't name mangled(something like cdecl and co)

weakexternal can already be combined with cdecl. weak+external could also be combined with cdecl. As an aside, cdecl is not the same as "no mangling". It means "mangle the same as the C compiler", which varies from platform to platform.

I don't understand why you again propose to change the meaning of "external name 'xxx'" for weak symbols. As I mentioned in my previous mail, for consistency reasons something like that will not be done. I also don't see the advantage of inventing a new syntax ("weak default") when existing modifiers can already express the required semantics.

Jonas

I don't propose changes. I might just formulate it in a way that make it seem so. Actually, if you go by the example in your previous mail you would have to change the meaning of "external name 'xxx'" external name 'xxx'; means that you "change" the name of an unresolved symbol into xxx so function calls to that procedure will go to xxx(and in that way make it an unresolved symbol). In your example you used external name 'NoHandler'; which of course will be resolved, and all the weak stuff will be ignored

.weak xxx
will then mark an unresolved weak
.set xxx, NoHandler
will then make a weak symbol point to NoHandler if it can't be resolved

The problem I have with the alias solution is that to practically use it you would have to change the way alias works weak symbols. Alias simply declares an extra global label, it doesn't invoke .set

You might also note the problem in the example
    //  a) on Linux
    //   .weak P$TestProc
    //   .set  P$TestProc, NoHandler
    //   .weak TestProc
    //   .set  TestProc, NoHandler

If you then somewhere else defines TestProc, then yes, calls to the assembler symbol TestProc will resolve to TestProcHandler. But if you call TestProc directly, in the pascal sense, then it would call P$TestProc which hasn't been overriden and thus points to NoHandler

Jeppe Johansen wrote

Den 15-02-2011 19:31, Jonas Maebe skrev: > I don't understand why you again propose to change the meaning of "external name 'xxx'" for weak symbols. As I mentioned in my previous mail, for consistency reasons something like that will not be done. I also don't see the advantage of inventing a new syntax ("weak default") when existing modifiers can already express the required semantics. I don't propose changes.

"external name 'xxx'" means more or less the same as "__attribute__((alias "xxx"))" in gcc (it's slightly different, because in gcc the "xxx" must be a symbol known to the compiler at compile time):

1) aliased external declaration
in gcc:

  extern int test(void) __attribute__((alias ("xxx")));

in fpc:

  function test: cint; cdecl; external name 'xxx';

2) regular weak external declaration
in gcc:

  extern int test(void) __attribute__((weak));

in fpc:

  function test: cint; cdecl; weak; external;

3) aliased weak external declaration
in gcc:

  extern int test(void) __attribute__((weak, alias ("xxx")));

in fpc:

  function test: cint; cdecl; weak; external name 'xxx';

While keeping the first two points the same, you propose to make this third case in fpc, :

  function test: cint; cdecl; weak default 'xxx';
or
  function test: cint; weak default 'xxx'; external name 'test'; // not entirely correct, on several platforms this should be '_test' instead

This changes the meaning of "external" when combined with "weak", since it no longer specifies the external symbol to which this particular declaration refers. Please look at it from the standpoint of high level language semantics, not based on the assembler level implementation on ELF platforms.

That's correct, so allowing "alias" in combination with "weak; external;" is probably not a good idea. Requiring cdecl is probably the best way, although I'm not sure yet what to do with non-external weak definitions then, because the symbol names of non-external cdecl routines are normally not mangled C-style by the compiler (after all, nothing prevents you from having overloaded routines that all use the cdecl calling convention). If "public" or "export" is added to a cdecl declaration, then the C-style mangled name is added as an alias, but that's not good enough here because it suffers from the same problem (Pascal-level calls to the routine will use the Pascal-mangled name rather than the C-style mangled one).

Only setting the C-style mangled name in case of "cdecl; weak;" non-external definitions is a possibility, but it's not very nice because it's inconsistent with normal operation.

Jonas

I suppose you mean weakexternal in those examples, or are they just examples of an ideal future case?

That is precisely what I propose. I think that in the low-level cases where I think this feature is beneficial, it's enough to have control over names with the "external name" directive

This changes the meaning of "external" when combined with "weak", since it no longer specifies the external symbol to which this particular declaration refers. Please look at it from the standpoint of high level language semantics, not based on the assembler level implementation on ELF platforms.

Nothing is changed. Absolutely nothing is changed in the way external works

function testimpl: cint; [public, alias: 'test'];
   begin
   end;
function xxx: cint; [public, alias: 'xxx'];
   begin
   end;
function test1: cint; external name 'test';
function test2: cint; weak default 'xxx'; external name 'test';
function test3: cint; weak default 'xxx'; external name 'notest';

test1 and test2 call testimpl, and test3 call xxx

I don't know what you mean about the high level semantics.

I don't think forcing cdecl is a good idea. I think the optimal case would be that weak procedures should just be called by their real name in the default calling convention. I know this is a bit hard to implement(I still haven't found out where that magic happens)

Jeppe Johansen wrote

I don't know what you mean about the high level semantics.

Defining the syntax according to high level semantics (which is what GCC does, and what I also want to do in FPC):

  extern int test(void) __attribute__((weak, alias ("xxx")));
and
  function test: cint; cdecl; weak; external name 'xxx';

Both mean "references to the C/Pascal symbol 'test' should be translated into weak references to the symbol 'xxx'". When removing "weak", the semantics stay identical except that the reference is no longer weak. In other words, the syntax is orthogonal. It's high level because this syntax completely ignores what happens at the assembler level with the different high level constructs.

Defining the syntax according to low level semantics, which is what you propose:
* the consequence at the assembler level on ELF platforms of calling "test" as declared above but *without* "weak", is a reference to an assembler symbol with the name 'xxx' (so the reference goes to the symbol whose name matches whatever comes after 'name')
* when adding "weak", references to the language symbol must keep going to a symbol with name 'xxx' at the assembler level, because that's what happens at the low level if "weak" is not specified
* therefore, in the case "weak" is specified, the 'xxx' can no longer be used to specify the external function that is (weakly) referenced by this declaration. Instead, it must specify the name of the assembler level symbol defined by the ".set" directive that's generated for this weak declaration (since that's where references to this weak symbol will go to)
* as a result, we need some extra modifier (default in your proposal) to define the external function that is weakly referenced by this declaration

And since simply removing "weak" from a declaration does not turn it into an equivalent normal external declaration (since those don't have a "default" modifier), it is not orthogonal. It's low level because it's completely based on how things work at the assembler level.

I don't think forcing cdecl is a good idea. I think the optimal case would be that weak procedures should just be called by their real name in the default calling convention. I know this is a bit hard to implement(I still haven't found out where that magic happens)

It is not that hard to implement, but that will definitely not be done. Adding special cases like that is simply not how programming languages can be designed. It makes both the compiler more complex (special cases to implement the exceptions) and learning the language harder (special cases that are not explicit in the syntax and which the compiler therefore cannot check, and which you just have to "know"). A small typing convenience (not having to add "cdecl") is no reason to change the normal behaviour of the compiler for a particular feature.

Jonas