Hi,

I've already asked about this on the community forums on the freepascal.org site and opened a bug on the bugtracker (bugID #4538). But none of those show any progress on a solution to the problem I'm having.

First a little backstory: I'm currently, as part of an internship, porting an application written in Delphi 6 to Free Pascal/Lazarus. The application consists of a main executable and a bunch of (23, to be exact) DLL's. The app uses some of these DLL's and some other are used by the DLL's themselves. To make this more clear, I've schematically illustrated the situation below.

Main-app
/ | \
dll2- | --dll1
/ | \
dll3- dll3 -dll2
\
-dll3

Dll3 could, for example, be a library which is responsible for logging (debug)messages to a file on disk.

Right now I've come to the point where every individual part seems to do what it should do. But when testing the whole package, I came across a problem, which at first i thought to be a bug in our code. But after some debugging and after testing with a bunch of simple testcases I concluded that my problem might be caused by a bug in FPC itself.

The problem is this: When the app calls a function in dll1, which calls a function in dll2, which calls a function in dll3, an AccessViolation is thrown when calling the function in dll3.

This problem is blocking further porting efforts, So you can guess that I'd really like to see this resolved.

What I'd like to ask is, am I doing something wrong? If so, what is it I'm doing wrong? If not, can anybody confirm the bug (if there is one) and maybe give a possible ETA for a fix.

FPC versions I've confirmed the problem with are:
official win32 2.0.0 binaries: Access violation when calling a function in dll3 from dll2 from dll1 from app
home-brew 2.0.3 (built on 6 dec) in win32: AV when calling function in dll3 from dll2 from dll1 from app
2.1.1 bundled with Lazarus nightly build (6 dec ): AV when calling dll2 from dll1 from app.

I'm the one who submitted bug #4538. At that time I thought the problem only occurred on win32 when using fpc 2.1.1, because I only tested with 2 levels of dll's. When I ran into the access violation problem again (with a dll 'tree' of at least 3 levels deep) on Linux using fpc 2.1.1 and 2.0.3, i concluded that probably all the 2.x versions suffer from somewhat the same problem.

Regards,
Stefan van den Berg

Before we can say something, what functions do you call, what params, what calling convention etc.
Does it happen in one specific sequence of calls, to specific functions/methods etc.

Please provide some more info.

Marc

http://www.freepascal.org/bugs/showsource.php3?ID=4538

Jonas

Whoops, I didn't saw that there was a bugrep.

I'm not shure if FPC behaves the same as Delphi, but I think it does.

In Delphi you cannot exchange ansistrings between dlls, unless you have a shared memory manager. Normally each exe (or dll) has its own memory manager. So when you pass a string to a dll and it it goes out of scope there (since it is changed) it is released there. So a allocated memory block from one manager is freed in another. This gives all kinds of unexpected errors at strange places.

IIRC the way constant strings are allocates it changed, so that may be the reason that you didn't see the error on dlls generated by an older fpc.

Marc

To the original person who posed the bug report:
As Marc stated.. it is definitely a bad idea to use an ansistring as a function result in DLL's - unless it was a BPL or you were using shared memory.

Did you include the sharemem unit in the delphi program maybe, which caused it to work there just fine but not in FPC? If not, how long was the program tested and how many users actually used the program without problems? Sometimes you can get away with strings temporarily, but the bugs pop up later or in random occurrences.

The general safe rule is to never return a string into a function result when passing it to a DLL without shared memory. There are ways to use ansistrings without shared memory, such as using setlength calls in certain places, and taking extra caution (like using CONST parameters in some cases).. but I've learned that it takes extra brain effort to try and trick/escape the reference counter, so using pchars makes you extra safe. That is, assuming you study pchars thoroughly (and there are a lot of incorrect information out there so a book needs to be written on pchars).

The other problem with trying to escape/trick the reference counter is you take extra CPU power to do it. The real disadvantage is not the extra cpu that it could cost, but the maintainability. Eventually your tricks catch up with you and you make a small mistake when trying to trick the reference counter. Whereas if you go in knowing what memory you are allocating, you can understand your mistakes and fix them rather than try to play more tricks.

The usage of strings in the samples was just to verify which code was executed and where things went wrong. I wasen't aware of the fact that that alone could cause problems. In the actual application mainly objects are passed to functions as regular and out parameters and a boolean is returned to indicate success or failure.

I don't think the sharemem unit is used anywhere in the origial code. I could be wrong, though. I don't have the code 'll have a look on Monday.

The application is still under development, so it's not actually been used/tested on a large scale. For as far as i know, the application, compiled with Delphi, doesn't show any of the problems I get when the app is compiled with FPC. But that could just as well be dumb luck.

The general safe rule is to never return a string into a function result when passing it to a DLL without shared memory.

I'm not sure what you ment by that. Did you mean it's unsafe to return an ansistring from a function exported by a dll per say? Or did you mean passing an ansistring to a function imported from a dll, manipulating that string and returning it?

What kind of objects.. most objects cannot be passed without taking precaution or using sharemem. If any of the objects use ansistrings, dynamic arrays, there will be problems (also called "automated types", since the memory that the "type" uses is so to say "automated" for you, i.e. reference counted).

Both cases are cause for concern. I meant that sending a string as a paramater can be dangerous, and that returning a string as a function result can be dangerious too. Sometimes sending a string paramater can be done, but only if you take extra precaution.

The DLL does not know what your EXE memory is doing. So by the time the string gets to the DLL it may be cleaned up and gone, kind of like a garbage collector. Same thing with returning a string.. if the string gets returned as the result of a function, it may have already been cleaned up and gone by the time you catch it on the exe side. Some times you can use setlength calls and other tricks to get around this, but I've found you spend too much time trying to trick the reference counter (like a garbage collector) - and its better to just allocate a pchar and KNOW what you are doing.

This is the real advantage of KNOWING your memory, and why C programmers think they are king (but in every day applications that don't use DLL's, knowing your memory is a waste of time - compilers, dlls, operating systems is an exception where knowing your memory is more of a good thing and not a waste of time).

I'm not sure if FPC has a sharing memory unit available - I've not looked into this, because most of the stuff I'm interested in creating, I want other programmers to be able to use too - gotta popularize the Pascal language by making DLL's available to other programmers than just our selfish selves. With a sharemem system I don't think you can share your DLL's with other programmers in other languages - but in your case maybe you are just using DLLs in your local application and would benefit from sharing memory. Again I don't know if FPC has this available - anyone know?

This is a confusing topic though, and no problem being confused about it all, even for weeks or months after researching it.

This does not make sense to me. When you return a string (from a function in a dll or anywhere else), this string will have a reference count of at least 1 and thus will not be cleaned up by anyone.

The only problem I can see that can occur is in case the program and the dll indeed use different memory managers, in which case you can have a situation where memory allocated by the dll's memory manager is freed "to" the program's memory manager, and that will probably confuse the hell out of it (even if they are essentially the same memory manager, just using different data structures).

Jonas

If you open Delphi 5, and say you want to create a dll it will create a file with about three keywords (library, exports, uses) and then a HUGE notice:

{ Important note about DLL memory management: ShareMem must be the first unit in your library's USES clause AND your project's (select Project-View Source) USES clause if your DLL exports any procedures or functions that pass strings as parameters or function results. This applies to all strings passed to and from your DLL--even those that are nested in records and classes. ShareMem is the interface unit to the BORLNDMM.DLL shared memory manager, which must be deployed along with your DLL. To avoid using BORLNDMM.DLL, pass string information using PChar or ShortString parameters. }

So never, ever user Ansistrings on DLLs unless you have a shared memory manager.

If you must use strings just use shortstring. If you need longer strings you can use PChar.

I apologize, I should have said that you'd not have exactly the same problems with returning a string - they are different problems but the basic point and idea remains the same: no matter what you do, strings will cause problems. What I should have said was there would still be all sorts of problems with returning a string - not necessarily the reference count problem, but other issues that cause similar havoc, such as two memory managers not communicating with each other. There are so many problems to consider, that a programmer once again spends more time thinking about the possible problems instead of programming a solution that works reliably.

When it comes time to maintain your application in the future, or modify your DLL, and if you did some how manage to get strings working by using setlength's in the right place, and not returning a string as a result, you may not remember why these workarounds were in place and what workarounds must be left in the code if you change the code. If you go in and modify a DLL one day in the future, with a string that is just barely working as is with workarounds, the application might go BOOM when you make one little change to the DLL. In other words, even if you can get a string working as a parameter being sent, received, and allocated properly (but not as a function result), it may be a dangerous bomb waiting to tick off in the future when you try and maintain or upgrade your DLL. You must spend minutes/hours going through every possible problem that could happen with the modifications you make, every time you upgrade your DLL.

If I change this code in my DLL, does that mess up the work around's that kept the bomb from going off? If I change this other code in my DLL, is the string still safe? There are just too many issues to think about. Unless, of course, there is a sharemem unit for FPC. And with a sharemem unit, since I've never used one in Delphi or FPC, I'm not sure how reliable they are. i.e are there bugs in the sharemem units that cause problems in Delphi, anyone know?

I just tried the bug report source code at
http://www.freepascal.org/bugs/showsource.php3?ID=4538 And I did not get an access violation or any errors.

Which proves that it depends on your random luck on a random day ;)

http://z505.com/images/BrokenDLLisWorking.png

Unrelated note:

The PNG I took was 26KB - a JPEG was 89KB and lower quality. Interesting. http://z505.com/images/BrokenDLLisWorking.jpg In cases where lots of similar colors are in the picture (such as lots of black or lots of white) it appears PNG's are superior. In other cases, JPEGs are smaller and seem to be more efficient.

Also passing objects may couse problems. Operators like "is" will fail, since TObject defined in dll1 is not equal to TObject defined in dll2. They have the same memory layout (if compiled by the same compiler), but the class pointer is different, so they are different classes.

I don't think this is the couse of your exception, but sertenly something to keep in mind.

Marc

Interesting. I'm curious about what version of FPC you used to compile the samples.

I still have not seen a single actual problem mentioned with an explanation by what exactly it is caused, only vague hand waving warning for strange and inexplicable problems. I'm really interested by what these problems are and can be caused. Does every dll have its own memory manager data structures and is indeed the problem that memory allocated by one is free "to" another (as I described in my previous mail)?

That would seem very strange to me though, since this could be easily solved by making the RTL/system unit a dynamically linked library and linking all other libraries to that (so there is only one system unit with only one memory manager).

Or is it something else?

Jonas

Makes sense. I'll definitally keep the points you mentioned in mind.

What I originally thought one of the problems was, when trying to receive a function result, was that the DLL thinks that there is no need to keep the string in the memory. Why should it, if no other code in the DLL uses the string? Unless the DLL has some knowledge about the Exe using the string. Does it? Does it know about the reference count that the exe created? When the exe tries to use the function result, it goes in to the DLL space and tries to grab the function result in memory, but it isn't there because the DLL threw it away (unless hte DLL does know about the reference count the exe created.. does it?). On certain occassions, the DLL hasn't thrown it away and the program runs fine. That was my assumption, but not neccessarily the correct one.

Can we confirm for sure that above is the wrong assumption?

Where is the "reference count" actually stored anyway, and who as access to the reference count - just the dll/exe who created the string, or is it accessible by both of them? I do need to learn more about reference count science.

The reference count is part of the ansistring itself. An ansistring is simply a pointer to a record containing a reference count, the amount of memory currently allocated for the string (i.e., maximum length -1) and the string itself (a 0-terminated string).

So when passing a string from a dll to somewhere else, its reference count is passed along with it.

Jonas

Interesting. I'm curious about what version of FPC you used to compile the samples.

Interesting. I'm curious about what version of FPC you used to compile the samples.

It is an SVN pull out from several months ago, of one of the FPC 2.1.1 releases..

Are you using an SVN pull out or a specific version?

I don't think the code is safe until we discuss this further though, even if it works on my machine as is.

Instead of asking all the newbie questions, I will do more reading about reference count science on my own too.. but if it's easy for you to tell me here, how does the reference count know ahead of time that there will be no usage of the string when it sets the reference count to 0? i.e. how is the reference count decremented? Does the compiler know this at compile time?

Lars

In short (don't pin me on the names or on exact details in special cases):

Assume you have a ansistring and you assign something to it

   S := 'SomeString';

the compiler generates something like

   DecStringRef(S);
   S -> 'SomeString';
   IncStringRef(S);

The DecStringRef() decrements the refcount and checks if the it reaches zero. Ifso, the string is freed. The referencecounter of a strings lies in memory just before the actual text. Thats why you cannot pass a @S to a procedure which expects a textbuf. In that case you pass @S[1], the adress of the first character.

THis decrefcount is also the reason that you shouldn't pass strings to a dll. Suppose the string is created in your main app en passed to a dll. As soon as a DecStringRef is called (and the reference reaches 0) the code executes in your dll, So the freeing of the string is also done in the dll (and a free memblock is added to the dll instead of the app)

Marc

Why does it generate a DecStringRef before you assigned the string? What if it is the first time and you are already at 0, it can't decrement it to -1 can it? Does the reference count start at 0, or 1?

In this case, it decrements 1 and increments 1, so we will always end up with a very simple, easy, solvable problem.

1 - 1 = 0
1 - 1 = 0

That seems like it isn't doing anything useful, since we are incrementing and decrementing by one, every time we assign a string.

What Jonas said was that the reference count would be 1 (one) when you pass it to a DLL. You are saying it will be 0 (zero). We have two people saying opposite things here, so I'm really not sure what is going on. What would be logical to me, is if the reference count was 1, not zero.

I guess when I finally understand this science, I will write a diagram showing how it works in a Pascal procedure or function with comments beside each code snippet saying what the reference count is.

If there is a PDF file that explains this or some good website let me know. I hate to embarrass myself here (maybe other people are confused too, though).

It is first incremented and then decremented. Reason: multi threading safeness.

Does the reference count start at 0, or 1?

1

One string is incremented, the other decremented. Your conclusion applies only to
s:=s;

Read the sources :)

At the time I submitted the bug I was probably using a nightly build from one to a couple of days before. I guess something somewhere changed between the moment of your checkout revision and whatever revision was in the nightly build I used.

I don't think the code is safe until we discuss this further though, even if it works on my machine as is.

I've come to the same conclusion, about the unsafeness of my samples, myself. After reading some of the responses on the list here, I changed the samples somewhat.

When I changed the string parameters and return values to pchars, i could successfully run the samples. And after, just to make sure, adding a third dll (I copied dll2 to dll3, and edited dll2 to call a function in dll3) it seemed the sample still worked.

So my guess is that the bug is probably invalid and in that case could probably be marked as such.

L505 wrote

Why does it generate a DecStringRef before you assigned the string? What if it is the first time and you are already at 0, it can't decrement it to -1 can it?

The reference count can never be zero, because when it becomes zero the string is freed and replaced with a nil pointer. Calling DecStringRef() on a nil pointer does nothing.

Does the reference count start at 0, or 1?

All ansistring variables are initialised with nil.

That seems like it isn't doing anything useful, since we are incrementing and decrementing by one, every time we assign a string.

An ansistring is a pointer which points to a string. Many different ansistring variables can point to the same string, and then the reference count of this ansistring will be equal to the amount of ansistring variables pointing to it.

s := t; (-> reference count of ansistring to which s points is decreased by 1 and freed if reference count became zero, the one to which t points is increased by 1)
q := t; (-> reference count of ansistring to which q points is decreased by 1 and freed if reference count became zero, the one to which t points is again increased by 1)

Jonas

L505 wrote

>In short (don't pin me on the names or on exact details in special cases):

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ there was a reason I said this.

A string is initially initialized as nil. DecStringRef checks if it is a nil, if so, nothing is done.

Nope, since the string was nil, it isn't decremented, so you end up with a refcount of 1

I didn't say that. You assumed that the initial string was empty.

We have two people saying opposite things here, so I'm really not sure what is going on. What would be logical to me, is if the reference count was 1, not zero.

It is 1.

Marc

> In this case, it decrements 1 and increments 1, so we will always end up with a

very

The string is 1, not nil
1 - 1 = 0 from your diagram indicates 1 is the starting refcount, not nil.

When I responded to your mail, I didn't just mean if the string was nil, meant your diagram didn't seem to click in with me when any number was filled in the blank. i.e. 1, 2, 3, 4, 5 or any number.

From your diagram, I got this:

String refcount: 2
Decrement(string) (1 - 1 = 1)
string -> 'some string'
Increment(string) (1 + 1 = 2)
We are at 2 again. What was the point of doing it at all?

Starting string refcount: 3
Decrement(string) (3 - 1 = 2)
string -> 'some string'
Increment(string) (2 + 1 = 3)
We are at 3 again. What was the point of doing it at all?

Starting string refcount: 1
Decrement(string) (1 - 1 = 0)
string -> 'some string'
Increment(string) (0 + 1 = 1)
We are at 1 again. What was the point of doing it at all?

I think we are just having issues communicating what is happening.

The way Jonas explained below makes sense to me, as we get something useful out of the incrementing rather than ending up where we were in the first place. I'm still running it through my head to really understand it on a deeper level with the DLL situation, though - since that's what this thread is trying to solve in the end.

Marc, I think where our communication broke down was that in your diagram it appeared you were incrementing the same string, rather than a different one.. i.e. in Jonas' example he explained it with two string variables - one being decremented, and the other being incremented. Not the same one being incremented and decremented.

The other problem I see with Marc's diagram was that somehow you showed the string being
decremented before anything even happened. Shouldn't it decrement after something happens, such as in Jonas' example? i.e. it just looked magical to me, that decrementing could happen ahead of time.

correction: 1 - 1 does not = 1 ...

Correction:
What I meant was actually this:

String refcount: 2 Decrement(string) (2 - 1 = 1) <------

Other people have already said it is what I suspected: each DLL has its own memory manager structures apparently, and that means that memory allocated by one memory manager can be released to another, which results in very bad things.

In my example there is also a decrease "before anything even happened". If you have

s := t

then the reference count of the ansistring s points to is decreased before the assignment is carried out (because the compiler inserts a call to a routine that does this decreasing), because after you have overwritten s you obviously don't know anymore what it pointed to.

Jonas

Ah now I understand your problem, but you miss the point that we are not refcounting the variable S, but its contents.

So S points to 'somevalue' and suppose this string has a refcount of 2. Calling DecRef(S) will decrement the refcount of the 'somevalue' string.

After we point S to 'some sting', AddRef(S) will set the refcount of 'some string' to 1.

So we have 2 equations:
'somevalue': 2 - 1 = 1
'somestring': 0 + 1 = 1

You see, we are working with 2 different strings, and not the same.

[snip]

The other problem I see with Marc's diagram was that somehow you showed the string being decremented before anything even happened.

Ehm.... I didn't draw any diagram, AFAIK you did.

Marc

One string is incremented, the other decremented. Your conclusion applies only to s:=s;

That's what I was trying to get at.. in order for the example to make sense there must be two strings in action.. i.e. one is incremented, and another is decremented, rather than the same one. Which is basically what Jonas gave in his example.

Ah now I understand your problem, but you miss the point that we are not refcounting the variable S, but its contents.

Thanks for clearing up - I understand, and these messages will be archived for future reference when my brain gets twisted up again.