Hello everybody.

I have posted this question in fpc-mailing list but... no answers, so i try here...

Fpc libraries compiled on freeBSD 64 bit are unusable. (32 bit not tested) Python, Java, C and even fpc applications can not use them in freeBSD...

Look at that (works on Linux 32/64): =>

> The fpc library: =>

library test ;
{$mode objfpc}
procedure fpctest(); cdecl;
begin
writeln('OK, it works...');
end;
exports
fpctest name 'fpctest';
begin
end.____________________________________

> The Python script: =>

#file fp_test.py
import string
import os,sys,platform
dia = sys.path[0]
from ctypes import*
fp_lib = "libtest.so"
fppathdll = os.path.join( dia , fp_lib)
fpdll = cdll.LoadLibrary(fppathdll)
fpdll.fpctest
_____________________________

Compile test.pas and run the python script: =>

> python fp_test.py

=> Does not work and same error if fpc library is used by Python, Java (fpc lib compiled with Java-syntax), C or fpc applications:

~/libtest.so: Undefined symbol "operatingsystem_parameter_envp"

How can i help to fix that ?

Thanks.

Fred.

In our previous episode, Fred van Stappen said:

I have posted this question in fpc-mailing list but... no answers, so i try here... Fpc libraries compiled on freeBSD 64 bit are unusable. (32 bit not tested) Python, Java, C and even fpc applications can not use them in freeBSD...

I initially didn't look at it, because you only posted foreign language examples.

For me, the fpc uses fpc case works fine. mainprogram:

{$linklib liblibtest}
procedure fpctest(); cdecl; external;
begin
 fpctest;
end.

and libtester placed somewhere in the LD_LIBRARY_PATH prints out "OK, it works".

But the FPC version has these symbols in the main program, maybe you must make sure that those (operatingsystem*) systems are available from the mainprogram too and preloaded with the proper values.

"Marco van de Voort" wrote

But the FPC version has these symbols in the main program, maybe you must make sure that those (operatingsystem*) systems are available from the mainprogram too and preloaded with the proper values.

The main program should not need to do anything here. It's more likely that there is a problem in our FreeBSD startup files for libraries.

Regards,
Sven

05.08.2014 22:02, Sven Barth пишет:

FreeBSD loader does not pass argc,argv and envp to shared library initialization procedure, like it happens on Linux. Since FPC does not have a shared RTL, such hack with exporting symbols from executable is likely an only way to initialize RTLs compiled into shared libraries.

However starting from version 9.2 IIRC, FreeBSD loader looks more like Linux one, but it passes argc,argv and envp only to initialization procedures pointed at by DT_INIT_ARRAY, not to one pointed at by DT_INIT. I.e. fixing the issue requires changes to startup files/link scripts and means that no support for versions <9.2 will be available.

Regards,
Sergei

Many thanks to take care on freeBSD + fpc libraries...

Please, read that topic, it talks about the same problem and it has a solution: =>

http://lists.freepascal.org/pipermail/fpc-devel/2013-November/032950.html

Hum, i thing also that it could be a problem on FreeBSD startup files of libraries. The same code compiled in Linux and Windows works.

With that library, i have also same problem with Python, C, Java. =>
////////////////////////////
library test ;
{$mode objfpc}
procedure fpctest(); cdecl;
begin
////// NOTHING !
end;
exports
fpctest name 'fpctest';
begin
end.
///////////////////////////

It gives that error : =>

> ~/libtest.so: Undefined symbol > OSerror : "operatingsystem_parameter_envp"

Many thanks.

Fred

Yeeeeeep, Sergei, many many many thanks, you are the superman that i was waiting for. It is more than exactly the problem that i have.

PS: I just post a message who talks about your earlier topics. (and i do not see your new one).

Re many thanks.

Fred

I.e. fixing the issue requires changes to startup files/link scripts and means that no support for versions <9.2 will be available.

Does it mean that actual fpc libraries could be used by freeBSD <9.2 ?

If a script to support freeBSD >=9.2 is done, could it be possible to add, for fpc compiler, something like :

{$IFDEF FREEBSD}
{$DEFINE FREEBSD_VERSION>=92}
{$ENDIF}

So we can compile both version ?

And now, the big question, does it exist some hope to compile libraries that can run on freeBSD >=9.2 (in a short future) ?

Many thanks.

Fred

must

> > make sure that those (operatingsystem*) systems are available from the > > mainprogram too and preloaded with the proper values. > > The main program should not need to do anything here. It's more likely

that there is a problem in

> our FreeBSD startup files for libraries. > FreeBSD loader does not pass argc,argv and envp to shared library

initialization procedure, like it happens on Linux. Since FPC does not have a shared RTL, such hack with exporting symbols from executable is likely an only way to initialize RTLs compiled into shared libraries.

And C code does not have that problem, because the program startup code initializes libc which other libraries then depend on, right? Grrr...

Regards,
Sven

In our previous episode, Sven Barth said:

Because they essentially have a shared RTL in libc, yes.

But note that with a hypothetical shared FPC lib, the problem would be the same if one of the parts was not written in pascal.

IOW the libc case only works for C programs, and programs that build on that.

Hello.

In a old topic, of Michael Walace :=>

> http://lists.freepascal.org/pipermail/fpc-devel/2013-November/032950.html

Michael said :
Hey all.

I had a problem and I fixed it in the compiler, but I'm not sure what the ramifications of my fix are for other people.

My goal was to expose some pascal code to python, as per:

http://wiki.freepascal.org/Developing_Python_Modules_with_Pascal

This works fine for me on windows but failed on freebsd/amd64.

When I would try to load my shared library as a python module, it would fail, saying: "Undefined Symbol: operatingsystem_parameter_envp"

This issue was reported by someone else but it was (mistakenly?) marked as a duplicate issue and closed:

http://bugs.freepascal.org/view.php?id=3D15422

Anyway, the problem appears to be in this file:

http://svn.freepascal.org/cgi-bin/viewvc.cgi/trunk/rtl/freebsd/x86_64/dllpr= t0.as?revision=3D25737&view=3Dmarkup

It has the comment:

# FreeBSD RunTime dynamic loader only
# calls the functions specified by -init
# as functions without parameters
# Thus, it is not possible to retrieve argc, argv and envp # On x86_64 CPU, using weak for the operatingsystem_parameter_XXX

# allows to bind to main program parameters,
# but this does not seem to work for i386 loader.

Then at the bottom:

/* Do not fail linkage if argc, argv and envp are not found. */ .weak operatingsystem_parameter_argc
.weak operatingsystem_parameter_argv
.weak operatingsystem_parameter_envp

My change was to comment these lines out and copy the lines from prt0.as :

.global operatingsystem_parameter_envp
.global operatingsystem_parameter_argc
.global operatingsystem_parameter_argv
.set operatingsystem_parameter_envp,operatingsystem_parameters+0 .set operatingsystem_parameter_envp,operatingsystem_parameters+0 .set operatingsystem_parameter_envp,operatingsystem_parameters+0

Then after recompiling fpc, I was able to compile a *.so file usable from python.

But... It seems like the original .weak version was done quite deliberately, so I suspect that if I submit a patch, it would break things for other people. OTOH, I'd like to get a fix for my issue into trunk.
//////////////////////////////////

Could it be feasible to compile a "custom" fpc, so i could compile libraries for freeBSD 10 ?

Many thanks.

Fred

In our previous episode, Fred van Stappen said:

But... It seems like the original .weak version was done quite deliberately, so I suspect that if I submit a patch, it would break things for other people. OTOH, I'd like to get a fix for my issue into trunk.

Probably fpc->fpc linking then fails.

Could it be feasible to compile a "custom" fpc, so i could compile libraries for freeBSD 10 ?

Startup files can be exchanged without recompiling everything. Just overwrite the .o with the newly assembled ones.

Could it be feasible to compile a "custom" fpc, so i could compile libraries for freeBSD 10 ? Startup files can be exchanged without recompiling everything. Just overwrite the .o with the newly assembled ones.

Ok, many thanks Marco. ;-)

So, according with Michal Wallace solution (hope i understood) :=>

> http://lists.freepascal.org/pipermail/fpc-devel/2013-November/032950.html

I will exchange part of code of dllprt0.as with some of prt0.as =>

Change that :

.weak operatingsystem_parameter_argc
.weak operatingsystem_parameter_argv
.weak operatingsystem_parameter_envp

with that :

.global operatingsystem_parameter_envp
.global operatingsystem_parameter_argc
.global operatingsystem_parameter_argv
.set operatingsystem_parameter_envp,operatingsystem_parameters+0 .set operatingsystem_parameter_envp,operatingsystem_parameters+0 .set operatingsystem_parameter_envp,operatingsystem_parameters+0

////////

=> And recompile fpc ( or faster, recompile only dllprt0.as)

PS : Write you later after test ;-)

Fred

I may be completely wrong, but the last three identical lines look strange to me. I'd assume that either having it just once should be OK, or that the latter two should be in fact different (in particular, refer to argc and argv rather than envp and probably also contain a different offset at the end).

Tomas

Hum, Tomas your advice is extremely clever...

Indeed, i just remark it now.

It seems more logical like this (if there must be 3 lines) :

.set operatingsystem_parameter_envp,operatingsystem_parameters+0 .set operatingsystem_parameter_argc,operatingsystem_parameters+0 .set operatingsystem_parameter_argv,operatingsystem_parameters+0

Other thing, to recompile fpc in freeBSD must i use gmake in place of make ?

Hum, a loooooooong night in perspective....

Wow to have noted it Tomas, thanks.

Write you (much) later.

Fred.

Hello.

Changed that in prt0.as =:>

> .weak operatingsystem_parameter_argc > .weak operatingsystem_parameter_argv > .weak operatingsystem_parameter_envp

with that :

Have re-compiled fpc + recompiled library...

But,...

Same error message :

OSerror: ~/libtest.so: Undefined symbol "operatingsystem_parameter_envp"

Aaaaaaaaarghhh.

I assume that you should have changed dllprt0.as rather than prt0.as?

Moreover - remind me, do you target FreeBSD under x86_64 or x86 (if you change the wrong one, it wouldn't have any effect, of course)? The snippet above seems to suggest x86_64 (considering the offsets for argc and argv addresses)...

Tomas

Yep, Many thanks to take care Tomas.

I assume that you should have changed dllprt0.as rather than prt0.as?

Yes, of course (sorry for bad copy).

Moreover - remind me, do you target FreeBSD under x86_64 or x86 (if you change the wrong one, it wouldn't have any effect, of course)? The snippet above seems to suggest x86_64 (considering the offsets for argc and argv addresses)...

Yes, my target is x86_64 too...

Hum, maybe totally stupid, but if uncomment each line ( remove first "." (dot)) ?

Change dllprt0.as with that :
global operatingsystem_parameter_envp
global operatingsystem_parameter_argc
global operatingsystem_parameter_argv
set operatingsystem_parameter_envp,operatingsystem_parameters+0 set operatingsystem_parameter_argc,operatingsystem_parameters+8 set operatingsystem_parameter_argv,operatingsystem_parameters+16

Many thanks Tomas.

Fred

I'd suggest checking if the file you have changed (dllprt0.o) is really the one used during linking. For that purpose, compile your library with parameter '-s' (avoid linking) and carefully check the created script.res file to make sure that either dllprt0.o is listed with a full path and the full path specification corresponds to the modified file (and no other reference to dllprt0.o appears in the linker script), or none of the search directories listed in the produced linker script contains an older version of dllprt0.o if dllprt0.o is listed in the linker script without path.

.
.

Nope - the dot is not a comment but rather a sign of an assembler directive (as opposed to assembler instructions directly translated to CPU opcodes).

Tomas

Yep, super idea, i wil do it like that.

PS: fpc has lot of chance to have people like you.

Many thanks Tomas. (write you later).

Fred

Hum... Not so easy. I use a script (Code Typhon script) for compiling fpc. That script use gmake clean all blablabla => where to add parameter '-s' ;-(

Other thing, i have done a research of dllprt0.o and find only one: =>

/usr/lib/codetyphon/fpc/units/x86_64-freebsd/rtl/dllprt0.o =>

Last modified : 06/08/2014 => it is the date of compilation after change.

So it seams that it is the modified one...

Aaaaaaarghhhhh. ;-(

.
.

You should not use that parameter for compiling fpc, but only for compiling your library. This shouldn't require any complicated script, IMHO.

Tomas

Ha, ok, i will compile the test library with -s parameter (mine is too compilcated, i begin with a hyper-simple to be sure im not the guilty ;-) ).

Write you later.

And many thanks.

OK, libtest.pas compiled with -s.

link.res: =>

///////////////////////////////

...
blablabla
...

INPUT(
/usr/lib/codetyphon/fpc/units/x86_64-freebsd/rtl/dllprt0.o test.o
/usr/lib/codetyphon/fpc/units/x86_64-freebsd/rtl/system.o /usr/lib/codetyphon/fpc/units/x86_64-freebsd/rtl/objpas.o )

////////////////////////

So, yes, it is the good dllprt0.o used...

Hum, what is this system.o for ? Nothing to change there ?

Thanks. (we will win, sure).

Fred

Compiled code from the system unit. I don't see any reference to the missing symbol from there. You could add the symbols there instead of the loader probably, but that's another story.

However, there's something else (I wonder why there isn't at least a warning during compilation of the modified dllprt0.as) - the modified lines refer to symbol 'operatingsystem_parameters'. However, this symbol is not defined in dllprt0.as (unlike prt0.as - see the lines preceding the operatingsystem_parameter_* definitions there).

BTW, note that even if you succeed with linking, this wouldn't change the fact that the values (pointers) are not initialized. This may be OK as long as your library links to libc itself (directly or indirectly via other libraries) or your library doesn't need to use any environment variables, but it would fail at runtime otherwise. One thing which seemed to be related to potential addressing of this issue which I could find using Google was the following comment (in GLIBC sources):

------------------------
46 #ifdef SHARED
47 /* This function is called to initialize the shared C library. 48 It is called just before the user _start code from sh/sh[34]/elf/start.S,
49 with the stack set up as that code gets it. */ 50
51 /* NOTE! The linker notices the magical name `_init' and sets the DT_INIT
52 pointer in the dynamic section based solely on that. It is convention
53 for this function to be in the `.init' section, but the symbol name is
54 the only thing that really matters!! */
55 /*void _init (int argc, ...) __attribute__ ((unused, section (".init")));*/
=========================

I know virtually nothing about FreeBSD, but if this comment (about linker behaving in special way to 'magical name _init') still applies for the linker used under FreeBSD, this could be the potential solution for initialization of these values for FPC RTL in shared libraries (rather then for shared libc).

Tomas

Yep, this is a excellent opportunity for fpc to become the unique compiler who can produce True Universal Libraries.

freeBSD is growing, freeBSD is used by OSX too..., if fpc can create libraries for freeBSD, accessible for lot of languages (like fpc libraries do for Linux and Windows), fpc becomes the Libraries's King... ;-)

Thanks Tomas.

Fred.

It looks like my old post had an error when I tried to copy and paste the text.
The last line that is repeated 3 times should be three separate lines with an
have an offset for each variable.

For what it's worth, here is the actual patch I used. (I haven't re-tested it
today. If it doesn't work, I would have originally used the latest version of
the svn trunk on the day I posted, so you might try that...)

######### BEGIN PATCH ############################

diff --git a/rtl/freebsd/x86_64/dllprt0.as b/rtl/freebsd/x86_64/dllprt0.as index 7213eae..2a68c6f 100644
--- a/rtl/freebsd/x86_64/dllprt0.as
+++ b/rtl/freebsd/x86_64/dllprt0.as
@@ -68,8 +68,20 @@ _haltproc:
movzwl (%rbx),%edi
syscall
jmp
- /* Do not fail linkage if argc, argv and envp are not found. */
- .weak operatingsystem_parameter_argc
- .weak operatingsystem_parameter_argv
- .weak operatingsystem_parameter_envp
+// /* Do not fail linkage if argc, argv and envp are not found. */ +// .weak operatingsystem_parameter_argc
+// .weak operatingsystem_parameter_argv
+// .weak operatingsystem_parameter_envp

+/* Although these variables are useless, they actually do need to be declared
+ * to create shared libraries usable from other environments. (eg, python).
+ */
+operatingsystem_parameters:
+ .skip 3*8
+
+ .global operatingsystem_parameter_envp
+ .global operatingsystem_parameter_argc
+ .global operatingsystem_parameter_argv
+ .set operatingsystem_parameter_envp,operatingsystem_parameters+0
+ .set operatingsystem_parameter_argc,operatingsystem_parameters+8
+ .set operatingsystem_parameter_argv,operatingsystem_parameters+16

########### END PATCH #############################

Date: Wed, 6 Aug 2014 15:37:19 -0500
From:
To:
Subject: Re: [fpc-devel] fpc libraries do not work in freeBSD...

It looks like my old post had an error when I tried to copy and paste the text.The last line that is repeated 3 times should be three separate lines with anhave an offset for each variable.

For what it's worth, here is the actual patch I used. (I haven't re-tested ittoday. If it doesn't work, I would have originally used the latest version ofthe svn trunk on the day I posted, so you might try that...)

######### BEGIN PATCH ############################

diff --git a/rtl/freebsd/x86_64/dllprt0.as b/rtl/freebsd/x86_64/dllprt0.as index 7213eae..2a68c6f 100644--- a/rtl/freebsd/x86_64/dllprt0.as+++ b/rtl/freebsd/x86_64/dllprt0.as@@ -68,8 +68,20 @@ _haltproc: movzwl (%rbx),%edi syscall jmp /* Do not fail linkage if argc, argv and envp are not found. */- .weak operatingsystem_parameter_argc
- .weak operatingsystem_parameter_argv- .weak operatingsystem_parameter_envp+// /* Do not fail linkage if argc, argv and envp are not found. */+// .weak operatingsystem_parameter_argc +// .weak operatingsystem_parameter_argv+// .weak operatingsystem_parameter_envp +/* Although these variables are useless, they actually do need to be declared
+ * to create shared libraries usable from other environments. (eg, python).+ */+operatingsystem_parameters:+ .skip 3*8++ .global operatingsystem_parameter_envp
+ .global operatingsystem_parameter_argc+ .global operatingsystem_parameter_argv+ .set operatingsystem_parameter_envp,operatingsystem_parameters+0+ .set operatingsystem_parameter_argc,operatingsystem_parameters+8
+ .set operatingsystem_parameter_argv,operatingsystem_parameters+16 ########### END PATCH ############################# Hello Michal.

Many thanks to help us.

I will try your patch now.

Write you lat.

Fred

Date: Wed, 6 Aug 2014 15:37:19 -0500
From:
To:
Subject: Re: [fpc-devel] fpc libraries do not work in freeBSD...

It looks like my old post had an error when I tried to copy and paste the text.The last line that is repeated 3 times should be three separate lines with anhave an offset for each variable.

For what it's worth, here is the actual patch I used. (I haven't re-tested ittoday. If it doesn't work, I would have originally used the latest version ofthe svn trunk on the day I posted, so you might try that...)

######### BEGIN PATCH ############################

diff --git a/rtl/freebsd/x86_64/dllprt0.as b/rtl/freebsd/x86_64/dllprt0.as index 7213eae..2a68c6f 100644--- a/rtl/freebsd/x86_64/dllprt0.as+++ b/rtl/freebsd/x86_64/dllprt0.as@@ -68,8 +68,20 @@ _haltproc: movzwl (%rbx),%edi syscall jmp /* Do not fail linkage if argc, argv and envp are not found. */- .weak operatingsystem_parameter_argc
- .weak operatingsystem_parameter_argv- .weak operatingsystem_parameter_envp+// /* Do not fail linkage if argc, argv and envp are not found. */+// .weak operatingsystem_parameter_argc +// .weak operatingsystem_parameter_argv+// .weak operatingsystem_parameter_envp +/* Although these variables are useless, they actually do need to be declared
+ * to create shared libraries usable from other environments. (eg, python).+ */+operatingsystem_parameters:+ .skip 3*8++ .global operatingsystem_parameter_envp
+ .global operatingsystem_parameter_argc+ .global operatingsystem_parameter_argv+ .set operatingsystem_parameter_envp,operatingsystem_parameters+0+ .set operatingsystem_parameter_argc,operatingsystem_parameters+8
+ .set operatingsystem_parameter_argv,operatingsystem_parameters+16 ########### END PATCH ############################# Hello Michal.

Many thanks to help us.

I will try your patch now.

Write you lat.

Fred

Yeeeeeeeep, Michal, your tip does the trick... :-)

According to your patch (i do not know how to apply), i have changed in: /rtl/freebsd/x86_64/dllprt0.as :
________________________________
.weak operatingsystem_parameter_argc
.weak operatingsystem_parameter_argv .weak operatingsystem_parameter_envp_____________________________________

> with that :

_____________________________________
operatingsystem_parameters: .skip 3*8
.global operatingsystem_parameter_envp
.global operatingsystem_parameter_argc .global operatingsystem_parameter_argv .set operatingsystem_parameter_envp,operatingsystem_parameters+0 .set operatingsystem_parameter_argc,operatingsystem_parameters+8 .set operatingsystem_parameter_argv,operatingsystem_parameters+16 ______________________________________________

After compile fpc, (do not know what assembler command to use to compile only dllprt0.as) the magic appear :

1) I can run Python script that use fpc libraries on freeBSD 10 64 bit.
2) I can run Java app that use fpc libraries on freeBSD 10 64 bit.
3) Complicated GUI fpc libraries tested too on freeBSD 10 64 bit. => It Works !
4) fpc can really do True Universal Libraries.

Morality of that story => fpc is now the unique King of True Universal Libraries. ;-)

Hypra many tetra thanks to Michal and all the developers of King-Magic fpc.

Here screenshots of fpc libraries used by Java and Python on feeBSD 10 64 bits.

> https://sites.google.com/site/fiensprototyping/home/freebsd_java_fpguilib.jpg > https://sites.google.com/site/fiensprototyping/home/freebsd_python_fpguilib.jpg

Fred

PS : I recommend that patch to all freeBSD users.