After hopefully getting parameter passing fixed (I hope, thanks Mark), I'm still facing this problem:
dll using Firebird crashes on Windows and Linux
Using a dummy db unit that does not connect to Firebird: everything ok.

Compiler options on a Windows compile:
-MObjFPC -Scaghi -WR -Cirot -Os -gw2 -gl -vewnhibq -Filib\i386-win32 -Fu3rdparty\csvdocument -Fu3rdparty\fpspreadsheet -Fu. -FUlib\i386-win32 -O-1 -Cg -dFPCONLY -dDPDLLDEBUG

Linux bt shows (the location of the error seems weird - perhaps memory messed up due to Firebird .so problems?)
(gdb) run
Starting program: /home/pascaldev/postcode/Source/dlldemo [Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [New Thread 0x7ffff472d700 (LWP 7731)]
Starting dll demo program.
Looking up house number 68a, postcode 2514GL:

Program received signal SIGSEGV, Segmentation fault. 0x00007ffff785b8e5 in fpc_ansistr_decr_ref () from /usr/lib/libdutchpostcode.so
(gdb) bt
#0 0x00007ffff785b8e5 in fpc_ansistr_decr_ref ()
from /usr/lib/libdutchpostcode.so
#1 0x00007ffff7fe6b00 in ?? ()
#2 0x00007ffff785b975 in fpc_ansistr_assign ()
from /usr/lib/libdutchpostcode.so
#3 0x0000000000000044 in ?? ()
#4 0x0000000000400770 in ()
#5 0x0000000000000044 in ?? ()
#6 0x00007ffff78c33db in SETCITY (this=0x7ffff7fe6ac0, AVALUE=0x7ffff7fdf618 '''s-Gravenhage') at dutchaddress.pas:262 #7 0x00007ffff78bc22e in POSTCODE2ADDRESS (this=0x7ffff7fce960, ADDRESS=0x7ffff7fe6ac0) at businesslogic.pas:390
#8 0x00007ffff7851682 in POSTCODE2ADDRESS (NUMBER=0x474058 '68a', POSTCODE=0x474050 '2514GL') at dutchpostcode.lpr:153 #9 0x0000000000400d01 in main () at dlldemo.lpr:115

If I add cmem first in the .so uses clause, I get this on running my dlldemo application:
Inconsistency detected by ld.so: dl-minimal.c: 136: realloc: Assertion `ptr == alloc_last_block' failed!

Should I enable other options?
Longer term, I think this is a good reason to start PostgreSQL support and check if that works.

Pointers/advice/requests for further info welcome, thanks.

Reinier Olislagers wrote

After hopefully getting parameter passing fixed (I hope, thanks Mark), I'm still facing this problem: dll using Firebird crashes on Windows and Linux Using a dummy db unit that does not connect to Firebird: everything ok.

Can you link the lookup DLL/so statically for test purposes?

If I add cmem first in the .so uses clause, I get this on running my dlldemo application: Inconsistency detected by ld.so: dl-minimal.c: 136: realloc: Assertion `ptr == alloc_last_block' failed!

If adding cmem to one you might need it in both to be effective.

The only thing I can add at this point is that there are a couple of cases where I've written code to interface to multiple databases (Firebird, PostgreSQL) but while the underlying database library (presumably written in C) might be linked dynamically I've not tried interposing a custom demand-loaded library... I presume that in your case you're leaving the OS to do the load when the program starts, but I think that the same points about memory managers etc. apply. I have used demand-loaded libraries to interface to e.g. custom processing for MIDI or to optionally load e.g. libpcap, and once working it's reliable.

Reinier Olislagers wrote

After hopefully getting parameter passing fixed (I hope, thanks Mark), I'm still facing this problem: dll using Firebird crashes on Windows and Linux Using a dummy db unit that does not connect to Firebird: everything ok.

Perhaps a stupid question, but: why are you not simply using sqldb ?

I do a lot of work with firebird embedded, and I've never even come close to the problems you're experiencing ?

Michael.

Why do you think I don't?

My library does use sqldb. I just want to provide postcode lookup/address validation services to other programmers in a .dll/.so.

I do a lot of work with firebird embedded, and I've never even come close to the problems you're experiencing ?

It's not limited to Firebird embedded; I ran tests on Windows & Linux with Firebird client/server; same thing

Mark Morgan Lloyd wrote

Reinier Olislagers wrote: Can you link the lookup DLL/so statically for test purposes?

I'll have a look.

Ah yes. That does fix that ld error.
Backtrace changes
#0 0x00007ffff785be75 in fpc_ansistr_decr_ref ()
from /usr/lib/libdutchpostcode.so
#1 0x00000000006c0ca0 in ?? ()
#2 0x00007ffff785bf05 in fpc_ansistr_assign ()
from /usr/lib/libdutchpostcode.so
#3 0x00000000006add18 in U_$SYSTEM_$$_OUTPUT ()
#4 0x0000000000400ac0 in ()
#5 0x00000000006add18 in U_$SYSTEM_$$_OUTPUT ()
#6 0x00007ffff78c3c8b in SETSTREET (this=0x6c0c78, AVALUE=0x0) at dutchaddress.pas:272
#7 0x00007ffff785178f in CLEARADDRESS () at dutchpostcode.lpr:91 #8 0x0000000000401118 in main () at dlldemo.lpr:116

I presume that in your case you're leaving the OS to do the load when the program starts,

Yes, I am.

but I think that the same points about memory managers etc. apply. I have used demand-loaded libraries to interface to e.g. custom processing for MIDI or to optionally load e.g. libpcap, and once working it's reliable.

May be me, but I have trouble understanding why:
1. A regular program loading Firebird .so/.dlls works without any specified cmem etc. Here applicationFirebird dll
2. An FPC library, coded similarly to 1., fails.
Here applicationdllFirebir dll
The difference between the 2 seems to me application coupled directly to FB dll versus coupling to FPC dll.
Why should that make a difference?
Why should I worry about memory managers etc in the second case but not the first? (Assuming I did not create problems with applicationdll parameter passing)

Everything below is my understanding. I'm happy to be corrected.

I've not used the Firebird interface libraries directly to any great extent (only the PostgreSQL ones), but the thing that occurs to me is that when the database library allocates memory for e.g. a resultset of indeterminate size it is subsequently responsible for freeing it. This is akin to OS (as distinct from application) programming, where the caller of an OS function probably has limited access rights to the memory containing structures that the OS returns, and definitely shouldn't assume that it's got arbitrary allocate/free rights to it.

So in your case (2) there are three separate binaries which potentially allocate and free memory: (i) the database library (ii) the custom DLL/so and (iii) the application program, and in the general case each one should free precisely those areas of memory that it allocated: not a block more and preferably not a block less.

Now if we assume that the database library has its own competent memory manager, and that its callers correctly tell it to free memory as soon as they're done with it (e.g. once a resultset has been parsed); that leaves potential problems in the custom library and application code.

If the custom library never made any attempt to access Pascal-type (long) strings, dynamic arrays, or anything else on the heap which might be reallocated or extended when referenced (i.e. not just at creation), then you'd not need to do anything special. But as soon as the custom library tries to e.g. change the length of a string or dynamic array it makes a call to the memory manager, and if it's not sharing the same memory manager as the main app used to allocate the string/array that's a recipe for disaster.

Now it might, potentially, be possible to tell a DLL/so as it is being loaded to link itself to the heap manager built into the main app. But the current preferred way is for the library and the app to use the same external memory manager, which is where importing cmem comes in.

So if you're getting crashes and your parameters are right, the first thing to check is that your custom library and the main app are both importing and using cmem before anything else such as HeapTrc. My experience is that that works, and if it isn't working then time you put into fixing it will probably be a useful investment.

Here endeth the lesson. Go in peace.

Yes, that sounds like how it should work/what I understand as well.

Now if we assume that the database library has its own competent memory manager, and that its callers correctly tell it to free memory as soon as they're done with it (e.g. once a resultset has been parsed); that leaves potential problems in the custom library and application code.

Agreed.

Yes. 3 things:
1. I hopefully fixed memory allocation issues between applicationmy dll in the previous thread. Of course if you mean my dll modifying strings etc in the firebird .so/.dll yes of course but why don't these problems then occur with the simpler applicationFirebird .dll/.so?
2. Corruption does not occur when I compile my dll with database layer code that does not call the Firebird sqldb layer i.e. the Firebird .so/.dll
3. I've tried both with and without cmem in both application and my dll so they are definitely sharing the same memory manager. What the fb .so uses... well... who knows, probably libc?!?!

Yes, see above.

Here endeth the lesson. Go in peace.

Et lectori salutem & the same to you :)

But when your code modifies strings in the context of the database library they're preallocated C-type strings, i.e. can't be extended or shrunk so don't make unexpected reference to any memory manager.

2. Corruption does not occur when I compile my dll with database layer code that does not call the Firebird sqldb layer i.e. the Firebird .so/.dll

Interesting. So if I understand you correctly it's only when you've got two levels of dynamically-loaded library, i.e. one DLL is responsible for loading another.

Now would probably be a good time to try Postgres as an alternative.

3. I've tried both with and without cmem in both application and my dll so they are definitely sharing the same memory manager. What the fb .so uses... well... who knows, probably libc?!?!

Probably libc, but not really relevant since all the allocate/free operations are done in the same context: the lib giveth, and the lib taketh away :-)

Mark Morgan Lloyd wrote

Reinier Olislagers wrote: But when your code modifies strings in the context of the database library they're preallocated C-type strings, i.e. can't be extended or shrunk so don't make unexpected reference to any memory manager.

Yes, I agree. However, this problem should also appear in regular FPC applications + Firebird library, right? Why don't I see any problems there? As mentioned, I'm using regular sqldb to talk to Firebird, nothing special.

Interesting. So if I understand you correctly it's only when you've got two levels of dynamically-loaded library, i.e. one DLL is responsible for loading another.

Yes. When/if I add Postgresql support we may now more ;)

Now would probably be a good time to try Postgres as an alternative.

Ah, a mind reader ;)

I think the first thing I'd try is a trivial program+DLL checking that the DLL can initialise a database library, in case there's something wrong with multistage initialisation.

If I understand things correctly, your app is linking to the custom library as part of the program-load operation handled by the OS, but the custom library is opening the database library dynamically. It might be worth checking this latter part with utter paranoia, my recollection is that the FPC loader library now saves any error message returned by the OS and I've found this useful in the past.

Yes, thanks, good idea.

custom library is opening the database library dynamically. It might be worth checking this latter part with utter paranoia, my recollection is that the FPC loader library now saves any error message returned by the OS and I've found this useful in the past.

Good pointer, thanks.

Because you are speaking about cmem, low-level pchars and whatnot.

My library does use sqldb. I just want to provide postcode lookup/address validation services to other programmers in a .dll/.so.

Ok.

Could be.

From your story, I doubt your problems are Firebird related.

Firebird (or any other DB client lib) just uses the buffers you provide.

Michael.

Michael Van Canneyt wrote

I do a lot of work with firebird embedded, and I've never even come close to the problems you're experiencing ?

+1

Using Embedded and C/Server in a few projects under Windows and Linux (unofficial support) and it works fine here too. I use FPC's SqlDB components.

Regards,
- Graeme -

Michael Van Canneyt wrote

From your story, I doubt your problems are Firebird related. Firebird (or any other DB client lib) just uses the buffers you provide.

Except that - as I mentioned - replacing my db layer calling Firebird with a dummy layer that does nothing resolves the crash. Anyway, pleae see the post re minimal test case, thanks.