Hi,

I encountered problems using dynamically loaded sqlite library with lazarus on linux.
My post about this in the sqlite list is here:
http:/msg61565.html
Maybe someone points me to the right direction

It's about loading sqlite library with dlopen, getting all the methods with dlsym and using it. I intentionally use different name and soname to not to conflict with the system name (libsqlite3.so)

It works if I use default libsqlite3 library installed with ubuntu package system in dlopen.

But if I compile the latest sqlite by myself or use configurable sqlite download with default options, I get SIGSEGV errors inside sqlite code and they appear inside dlopen is called (this is because sqlite3_initialize is defined to be executed when the library is beeing loaded). Mostly the faults are always about calling functions accessed from the global structure (sqlite3GlobalConfig) related to mutexes or memory management (I found this when compiled the library with debug info)

If I link the same 'problem' library with the sqlite shell (sqlite3), they both work without any problem (so I can do all db manipulations in the shell).

So I can not understand what is so special about the default installed library and where should I look further.

Thanks in advance,

Max
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20110608/1fbdd604/attachment.html>

Finally found that this was because sqlite3 was in the uses list, so actually it was scenario when fpc linked sqlite dynamically and besides I did my own dynamic loading with my own unit of another version/file of sqlite. So probably there was a conflict with shared data. In my case it looks like everything was ok if the linked library and loaded was the same file (system libsqlite3.so). But if linked was system libsqlite3.so and my loaded was a library with a different name/soname the faults took place.

Just curious, such things occasionally can happen from time to time. Is this about the way sqlite library is compiled so no two independent copies of the library coexist dynamically linked/loaded due to intersetion of the shared data. Or this is about the way linux handles them?

Thanks

Max
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20110609/6c8be621/attachment.html>

09.06.2011 13:17, Max Vlasov ?????:

It is best to ask this question in FPC devel mailing list, this behavior may be a bug.

It is not a FPC bug; you're not supposed to use the 2 units at the same time. There is no way the compiler can detect this. At best the linux linker should detect it.

Michael.

2011/6/10

Michael,
actually what appeared was that two libraries with different filenames and different soname (although compiled from the same sources, one dynamically linked and another dynamically loaded) shared some global data during run-time. I thought that in this case they should look virtually different for the linux loader to separate everything, but actually they were not.

What interesting is that a solution was found. It's a flag RTLD_DEEPBIND for dlopen, I discovered it by chance, wrongly assuming RTLD_LOCAL as having $8 (netware libc) while RTLD_LOCAL=0 in linux. In Linux RTLD_DEEPBIND = $8

RTLD_DEEPBIND is not present in any of lazarus/fpc sources, but after I added it to my dlopen call, dynamically loaded library started to work when there was still a dynamically linked sqlite. So they became independent from each other.

Quote from http://linux.die.net/man/3/dlopen

I'm not sure this will be helpful for everyone, but maybe it is worth to add this constant to rtl\unix\dl.pp file?

Max
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20110610/f69bf3ce/attachment.html>

Yes, I can see how this can help.

I'm not sure this will be helpful for everyone, but maybe it is worth to add this constant to rtl\unix\dl.pp file?

It depends. Does it also exist in FreeBSD and Mac OS ? If not, then I think this is not such a good idea.

Michael.

2011/6/10

Good point,
Looks like it's linux-only patch introduced at around 2004 http://sourceware.org/ml/libc-hacker/2004-09/msg00083.html so was not a good idea to update rtl\unix\dl.pp :)

Max
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20110610/6bcb8dcf/attachment.html>

Max Vlasov wrote

Looks like it's linux-only patch introduced at around 2004 http://sourceware.org/ml/libc-hacker/2004-09/msg00083.html so was not a good idea to update rtl\unix\dl.pp :)

But would be ok to add it to documentation of dynlibs, that someone can use that flag under linux if have similar problem. I had same problem with sqlite3 and double installation of it under fedora 1.system one 2.xulrunner one, always had crashes if firefox is started before I use my app. With that flag it's fixed.

zeljko

The solution you propose to avoid the effect of loading dynamically a library which has already been marked for loading by the linker seems quite dangerous. Suppose the unit that caused the linker to include the library (call it lib1) executes some initialization routines. These will run in lib1. Now you load dynamically the same lib (lib2). From now on all calls to the library functions are executed in lib2. At the end the unit that used lib1 could execute some finalization routines. But this time they run in lib2. Unless you lib2 unloaded completely before the unit finalization. I guess there could be other situations where this causes problems. Possibly everything is fine for sqlite3 but documenting it as a general solution is IMHO not correct.

Why not check at run-time if the library is already loaded:
Mod:=dlopen(nil,RTLD_LAZY);
libloaded:=assigned(dlsym(Mod,'symbol in lib'));
dlclose(Mod);

Ludo

Stuff it in unit "linux", that is meant for linux specific stuff.

(and that is better than putting it in some portable unit with a "platform" modifier or so)

Hmm. CMIIW, but let's look at an example. I took a quote from sqlite3.inc and from my own module

==== this one is from sqlite3.inc (lightly modified)
function sqlite3_open(filename: pchar;ppDb: ppsqlite3): cint; cdecl;external
Sqlite3Lib;
==== this one from my module
  Procs.DLLHandle := dlopen(PAnsiChar('mysqlite3.so'), RTLD_NOW  OR
RTLD_DEEPBIND);
 @Procs.SQLite3_Open := dlsym(Procs.DLLHandle, 'sqlite3_open');

I suppose that if I don't use RTLD_DEEPBIND, both (sqlite3_open and Procs.SQLite3_Open) will point to the same address (what is not so bad), but what worse they both internally will use the same global writable data (what is actually bad). But with RTLD_DEEPBIND both (sqlite3_open and Procs.SQLite3_Open) will get different addresses (maybe not, but it's safe for code or read-only data), but they will definitely get different copies of the global writable data they need. So any current module that uses sqlite3 will use the unchanged, safe context, no matter what I loaded on the second, third or whatever step as long as I use RTLD_DEEPBIND for them.

So what did you mean by saying 'dangerous'?

Max
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20110612/ce6714d6/attachment.html>

So what did you mean by saying 'dangerous'?

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20110612/cfc472b4/attachment-0001.html>

So what did you mean by saying 'dangerous'?

You should have read a little further in the message: "Possibly everything is fine for sqlite3 but documenting it as a general solution is IMHO not correct.". The scenario I was talking about assumed an initialization (before lib2 is loaded) and a finalization routine which would/could run in different librarie. This is not the case in your example. I proposed to test if the library was loaded before loading again as a more general solution to the problem.
Don't get me wrong. I'm happy this is working for you. I'm only saying it is not a good idea to document/promote this as a general solution for avoiding problems with double loaded libraries.

Ludo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20110612/f171a46e/attachment.html>

Typo;) Just found out that Ctrl-Enter sends the message in Outlook.

2011/6/12 Ludo Brands

Ludo, ok, I just tried to see the level of the problem you described. I just recently found this flag so I want to be sure it is working as I expect.

You wrote in your previous post "At the end the unit that used lib1 could execute some finalization routines. But this time they run in lib2". I don't see why a function (let's call it somelibrary_somefinalize()) linked with external directive and used somewhere in the sources should be called from the lib2 this time (if RTLD_DEEPBIND flag is used). For dynamic loading there's always a variable and this new address was obtained with dlsym call and this is the only place that aware of lib2.somelibrary_somefinalize. I may be wrong, but this is how I understood the logic behind it.

I recently saw that I was wrong assuming that read-only code and data is safe, because there are usually different versions of libraries so to ensure imho any symbol should be bind separately.

Max
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20110612/d29d52a6/attachment.html>

You're right. My scenario doesn't cause a problem. All symbols are resolved before lib2 is loaded and loading lib2 won't change the symbols alread resolved. It would only affect new symbol lookups after lib2 is loaded and when using dlopen(nil,...).

Ludo