Hello all

I started some wiki for dynamic loading libraries support discussion.

http://wiki.freepascal.org/Dynamically_loading_headers

I already added an experimental implementation. If you have got some ideas, whises and criticism, don't hesitate to write.

-Ivo Steinmann
<http://www.dict.cc/englisch-deutsch/criticism.html>

I don't like the looks of lines like:
{$IFDEF S}function{$ELSE}var{$ENDIF}foobar_dosomething{$IFDEF D}: function{$ENDIF}(a:cint; b:cdouble): cint; extdecl;{$IFDEF S}external foobarlib;{$ENDIF}

They are hard to read. I understand that this way makes sure that both static and dynamical declaration are in sync, but the loss of readability is too much.

Maybe it is better to generate a foobar_dyn.inc based on the foobar.inc or generate both foobar.inc and foobar_dyn.inc from a common file format (maybe even the original header file).

Vincent

Vincent Snijders пишет:

I second that. If this is intended to be maintained manually, then it is not maintainable. If the intention is to handle the updates using automated tools, then it should not matter how much files to generate.

Sergei

Most libraries are translated by a tool like h2pas from the original headers and then rehashed manually. The best solution would be, if there's a tool that generates the var procedures directly from all external procedures.

But else it's much simpler to maintain a construct as the one above than allways starting from the orignal h file or updating allways both declarations. I know what i'm talking about, since I translated about 15 headers in /fpc/packages ^^

But of course you are right, it's not the "nice" solution ;) The best solution would be to have got some support from the compiler itself ;)

-Ivo Steinmann

Ivo Steinmann wrote

Most libraries are translated by a tool like h2pas from the original headers and then rehashed manually. The best solution would be, if there's a tool that generates the var procedures directly from all external procedures.

What do you exactly mean? A tool which takes for example: http://svn.freepascal.org/trunk/packages/libpng/src/png.pp and outputs a png_dyn.pp.

What kind of help?

Vincent

[..]

Most libraries are translated by a tool like h2pas from the original headers and then rehashed manually. The best solution would be, if there's a tool that generates the var procedures directly from all external procedures.

I think that creating a tool to translate a static header to a dyn header should be easy to create (i'm not talking to convert c header directly ;-)).

Should do the following right? Correct me if i'm wrong:

Static function:

function foobar_dosomething(a:cint; b:cdouble): cint; extdecl;external
foobarlib;

Becomes a dyn function:

var foobar_dosomething: function(a:cint; b:cdouble): cint; extdecl;

I can write such tool.

Luiz

And a statement to initialize the variable.

Vincent

I'd also note that the current approach (loading all functions at once) actually defeats the nature of dynamic linking, and smartlinking as well. If some function is absent in the library, error will occur on loading regardless of whether this function is actually used by the program. With static linking, if the problematic function isn't used, there will be no loading error. For this reason, I use the following approach in my projects (in pseudocode):

interface
function foo(args): integer;
implementation
type
   tfoo = function(args): integer;
const
   bad_ptr = pointer(1);
var
   _foo: pointer = bad_ptr;
function foo(args): integer;
var
   p: PPointer;
begin
   p := @_foo;
   Result := E_NOTIMPL;         // other values may be used, or an exception raised
   if p^ = bad_ptr then
     p^ := GetProcAddress(libHandle, 'foo_name');
   if Assigned(P^) then
     Result := tfoo(p^)(args);
end;

Such approach, of course, introduces somewhat bigger overhead, but allows the compiler to optimize unused parts away.

Sergei

Do you mean this?

sqlite3_symbols: array[0..152{$IFDEF SQLITE_OBSOLETE}+6{$ENDIF}] of TLibSymbol = (
(; name:'sqlite3_libversion'; weak:false), etc..

Luiz

That is one solution, filling the symbols array.

Another solution is simply:
@sqlite3_libversion := GetProcAddress(libHandle, 'sqlite3_libversion');

I was pointing at a missing link.

Vincent

well, there are a lot of possibilities, but that's one:

unit foobar;
{$DYNLOADING ON}
function foobar_dosomething(a:cint; b:cdouble): cint; extdecl;external
foobarlib;

Now when compiler flag {$DYNLOADING ON} is set, the compiler don't do smart/static linking.

Internally it creates an array of all USED!! symbols (similar to the one I recommended) and also generates functions like

InitializeFoobar;
TryInitializeFoobar;  // silent
FinalizeFoobar;

Then the user have to call this functions before he can use one of the external symbols. If one of the used symbols is not found in the library, we get an error similar to when do smart linking, way we could use the same headers as dynamic and static, just by switching {$DYNLOADING ON} or OFF

This mechanism would be also interesting for packages in lazarus.

-Ivo

exactly

Including the list of symbols I suggested.

I have made such a tool, which I used for the database-bindings. I don't know if I still have it tough, and it was very rough.

But it's easy doable, indeed.

Joost

Should we wait to your tool or can i start another one?

Luiz

The more important question is: should we do it this way or should I also write a 2nd implementation by modifying the compiler (branched)? Then we don't need an external ;)

Ivo Steinmann wrote

The more important question is: should we do it this way or should I also write a 2nd implementation by modifying the compiler (branched)? Then we don't need an external prefer the compiler not to be modified for something like this. The

required functionality is easily implementable without doing so, and external linking issues are already hard enough to debug without adding another abstraction layer on top that hides in the source code how a function is actually being imported.

Jonas

you are right of course ;)

but one thing would be nice to have. A compiler switch and/or built in function to check wheter a symbol is used or not. With this information is easy to implement "smart loading".

something like

{$SYMTRACING OFF}
// used symbols in this section aren't marked as used.
{$SYMTRACING ON}
and then
{$IF symused(XYZ)}{$IFEND}
or even
a builtin function symused(XYZ): Boolean;

this way it would be possible to create a structure that contains all symbols including the hint if a symbol is used or not.

const
{$SYMRACING OFF}
  zorba_symbols: array[0..6] of TLibSymbol = (
    (; name:'create_simple_store';
inuse:symused(create_simple_store)),
    (; name:'shutdown_simple_store';
inuse:symused(shutdown_simple_store)),
    (; name:'zorba_implementation';
inuse:symused(zorba_implementation)),
    (;
name:'Zorba_CompilerHints_default';
inuse:symused(Zorba_CompilerHints_default)),
    (;
name:'Zorba_SerializerOptions_default';
inuse:symused(Zorba_SerializerOptions_default)),
    (;
name:'Zorba_SerializerOptions_free';
inuse:symused(Zorba_SerializerOptions_free)),
    (;
name:'Zorba_SerializerOptions_set';
inuse:symused(Zorba_SerializerOptions_set)),
  );
{$SYMTRACING ON}

Then you can do

LoadLibrarySymbols(libhandle, zorba_symbols);

if one of the used symbols is not present, an exception is raised....

The problem is that when compiling a unit, you never know which symbols will be used in the program and which will be not. It only becomes known at the linking stage.

Sergei

+1

Martin

are you also on irc in #fpc @freenode? I'm often online there with nick Aison

Ivo Steinmann wrote

are you also on irc in #fpc @freenode? I'm often online there with nick Aison

I don't use irc often, but should do if necessary.

I will try to do the tool this weekend and keep you informed even if is not used.

Luiz

I think we need it ;)

Delphi have got also a solution:

http://docwiki.embarcadero.com/RADStudio/en/Libraries_and_Packages#Delayed_Loading

In our previous episode, Ivo Steinmann said:

Delphi have got also a solution: http://docwiki.embarcadero.com/RADStudio/en/Libraries_and_Packages#Delayed_Loading

It doesn't really say how it does it?!?

Better start a new one.

Joost

Delphi have got also a solution: http://docwiki.embarcadero.com/RADStudio/en/Libraries_and_Packages#Delayed_Loading

Soon or later people will request this anyways I fear ...

MeMeMe! ;)

This time a feature I like, while I do not like things like 'for..in' (Because it's not an orthogonal feature, another word I just learned in respect to programming)

It's a nightmare to maintain external library-headers when all exported functions have to be declared twice. Once for static loading, and one for dynamic loading. And the fact that all bindings are really done even when they are not used is also very annoying. To solve this adding wrappers for each function..... Means that you have to declare each exported function three times...

I don't see the problem when the exported function is defined 'static' - so a link-error happens when it is not available at compile time, and a error occurs when the program is started while the library is not available.

Or weak. Which means that a library is loaded when the first call to a function is made. Then the error of a missing library/exported function can be given on the moment of the first function call. Nothing special here?

(In fact the compiler generates the necessary wrapper-function:
Function AExternalfunction(param1: type1): type2;
begin
  if not loaded(ExternalLibrary) then
    LoadDynamicLibrary(ExternalLibraryName) // if fails exception
  if not assigned(TempFunctionVariable) then
    @tempFunctionVariable:=GerProcedureAddress(...) // fail->exception
  Result := tempFunctionVariable(param1);
)

Joost

or the compiler even generates that:

var Externalfunction: Function(param1: type1): type2 := @NotInitialized;
Function NotInitialized(param1: type1): type2;
begin
  if not loaded(ExternalLibrary) then
    LoadDynamicLibrary(ExternalLibraryName) // if fails exception
  Externalfunction := GerProcedureAddress(...) // fail->exception
  Result := Externalfunction(param1);
end;

The first call to a function is routed through an initialization function. All later calls are done directly. This avoids unnecessary "singleton" checks.

And it would be also nice that it's possible to initialize the library by the user. So that the user is not forced to check at each library call, if it fails or not because the library is not available.

So the compiler should also generate a public function

function InitializeLibrary[Name]: Boolean;
or even
function InitializeLibrary[Name](const LibraryName: String): Boolean;

-Ivo Steinmann

What about saying that

    procedure proc; external 'libname' name 'proc';

denotes a function that's dynamically loaded implicitly, while

    procedure proc; external name 'proc';

denotes a static linked function.

In my opinion that seems like the only logical solution. Explicit dynamic loading will ofcourse still be possible, but you wouldn't need tools or lots of ifdefs

I'm not aware of how well it would work on Mac platforms, but I'm mostly certain most GNU/Linux and BSD's, and Windows has support for implicit dynamic linking in the executable formats

This is ofcourse a solution that would require modification of the compiler, but I think in the long run that it'll make alot of things more comprehensible

Regards,
Jeppe

In our previous episode, Jeppe Johansen said:

This syntax is not possible. The external 'xxx' name 'proc' is there for a reason, for systems with multiple linker namespaces. Moreover there is already an enormous codebase that uses this.

In my opinion that seems like the only logical solution. Explicit dynamic loading will ofcourse still be possible, but you wouldn't need tools or lots of ifdefs

You still need to define which functions must be there to start.

I don't. I think this dynamically loading of headers has already gone to far. It is a good workaround for a few headers that are versionwise a disaster, like MySQL, but universally, it doesn't solve problems.

I don't know if I'm making a fool of myself, but what systems? Skimming the sources, I couldn't find any mention of it in the ELF writers. An external symbol will simply be linked to the first occurrence of a matching symbol in the first shared or static object(unless there's some name mangling going on that I didn't spot). The COFF writer already does it the way I suggested.

In principle, doing it this way would make cross compiling to an ELF platform with dynamic linking easier, once the internal ELF linker is made. Since the dynamic linked symbols and their locations will be known

In our previous episode, Jeppe Johansen said:

Windows, Darwin, and afaik Solaris. The ELF writers are probably not updated for Solaris yet.

While partially true, this is a brute force approach. Having a few simple map files (that can be generated on target from the libs) over which files contain which symbols might help too.

Regarding a similar subject, I found an article by Hallvards Vassbotn showing a method to delay load dlls in Delphi, its usage is pretty similar to static linking, but the library is loaded on demand (without explicitly calling LoadLibrary).

This is a short version: http://hallvards.blogspot.com/2008/03/tdm8-delayloading-of-dlls.html

The long PDF version: http://vassbotn.googlepages.com/DelayLoadingOfDLLs.pdf

Leonardo M. Ramé
http://leonardorame.blogspot.com

The library name is not used on Darwin. You also can't tell the static linker that a name should only be looked up in a particular library. It simply goes through all libraries/frameworks. Once it finds a library/framework that contains the symbol, I think it does record in the final binary from which library/framework the symbol comes though (so that if at run time multiple libraries export the symbol, the same one will always be chosen).

Jonas

What do you mean by "implicit dynamic linking"?

This is ofcourse a solution that would require modification of the compiler, but I think in the long run that it'll make alot of things more comprehensible

It would also require modifications to several existing header translations, and depending on what "implicit dynamic linking" means, it would also make it impossible to use the same unit for static and for dynamic linking, while this is currently possible. It would also break backwards compatibility. And like Marco I'm not sure what the big problem is that is solved with this.

Jonas

Using a runtime linker. On windows implicit dynamic loaded functions are written in the .idata section. And using dynamic runtime linking on GNU/Linux, with dl

If this method was implemented we could make both methods possible like this

{$ifdef LINKSTATIC}
const
  libname := '';
 {$ifdef LINUX}
  {$l library}
 {$endif}
 {$ifdef WIN32}
  {$l library}
 {$endif}
{$else}
const
 {$ifdef LINUX}
  libname := 'library.so';
 {$endif}
 {$ifdef WIN32}
  libname := 'library.dll';
 {$endif}
{$endif}
procedure ImportedFunction; external libname name 'ImportedFunction';
procedure ImportedFunction2; external libname name 'ImportedFunction2';

I hope I get the point across :)

Offcourse it does. Only with more stable headers you never notice because you never has to change them. Well, that is, after the first convertion to pascal. But as you do this first conversion, this is really usefull.

Now, on the contrary, there are 10 different ways to mimic this behaviour, every library uses it's own way. Which is not usefull for the programmer.

Besides, it is an optimisation. Oracle for example exports more then 500 functions headers. Those are all initialised at start, maybe 20 are used. Those can be eliminated.

Joost.

Offcourse not. For existing headers nothing changes. For new ones one can add the 'lazydynamic' keyword or something like that to the definition.
It it now also impossible to use the same unit for dynamic ans static loading. This is all done using ifdefs and include files generating two units sharing the same code. This all stays the same. Or maybe we can add a flag {$disablelazydynamic} - that way we only need one ifdef. It has no influence on backwards compatibility.

Joost.

Joost van der Sluis wrote on do, 12 nov 2009:

The mail I was reacting to, as quoted above, said:

***
What about saying that procedure proc; external 'libname' name 'proc'; denotes a function that's dynamically loaded implicitly, while

    procedure proc; external name 'proc';

denotes a static linked function.

In my opinion that seems like the only logical solution. Explicit dynamic loading will ofcourse still be possible, but you wouldn't need tools or lots of ifdefs
***

That would break backwards compatibility (it would no longer be possible to use a unit with the first syntax for using a statically linked library). You appear to be talking about some different proposal.

Jonas

---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.

You're right. I missed that, sorry for that.

Joost.