Hi all,

After upgrading fpc 2.6.4 -> 3.0.0, I'm seeing a bug where (as noted in subject) reference-counted function results are not being initialized to nil.

Is this a known bug?

I am seeing it in ansistrings and also dynamic arrays. Here is an example program to see it with ansistrings:

function new_string : ansistring;
   begin
     new_string := new_string + 'x';
   end;
begin
  writeln (new_string); // outputs: x
  writeln (new_string); // outputs: xx
  writeln (new_string); // outputs: xxx
end.

Reproducing with dynamic arrays seems to be a bit trickier, I haven't made a test case yet, but I think I will be able to if required.

What to do?

Thanks,
David

After upgrading fpc 2.6.4 -> 3.0.0, I'm seeing a bug where (as noted in subject) reference-counted function results are not being initialized to nil.

They were never guaranteed to be initialized to nil.

Reference-counted types (unlike other types) cannot contain memory garbage. But it doesn't mean that they are always initialized empty when the function starts. You need to explicitly do Result := '' if your code reads the Result later.

See similar questions for Delphi:
http://stackoverflow.com/questions/5336863/what-is-the-default-value-of-result-in-delphi http://stackoverflow.com/questions/5314918/do-i-need-to-setlength-a-dynamic-array-on-initialization/5315254#5315254

Luckily, FPC warns about it, at least in my simple test:

$ fpc -vw a.lpr && ./a
a.lpr(5,20) Warning: function result variable of a managed type does not seem to be initialized
blabla
blablablabla
blablablablablabla

Regards,
Michalis

http://stackoverflow.com/questions/5336863/what-is-the-default-value-of-result-in-delphi

http://stackoverflow.com/questions/5314918/do-i-need-to-setlength-a-dynamic-array-on-initialization/5315254#5315254

OUCH!!

I never dreamed that

local_var := some_function (param);

can result in the old value of local_var being used inside some_function. That's what's happening here. I fully expected the result dynamic array (embedded in a record type) to be initialized to nil, and have never taken care to initialize dynamic arrays and ansistrings. I'm astonished that this didn't bite me sooner, actually. Perhaps it is a culprit in occasional crashes.

I can't see any mention of this in the documentation: http://www.freepascal.org/docs-html/ref/refse47.html http://www.freepascal.org/docs-html/ref/refse89.html http://www.freepascal.org/docs-html/ref/refse91.html

Luckily, FPC warns about it, at least in my simple test:

it also warns me in a simple test, but it doesn't give any warning in my not-simple program.

Is there any compiler option that I can use to force nil initialization of ref-counted function results?

Thanks,
David

2016-06-25 5:19 GMT+02:00 Michalis Kamburelis :

They were never guaranteed to be initialized to nil.

Result has special logic for string, dynamic array, method pointer and variant (well documented ;) ):

"For a string, dynamic array, method pointer, or variant result, the effects are the same as if the function result were declared as an additional var parameter following the declared parameters. In other words, the caller passes an additional 32-bit pointer that points to a variable in which to return the function result."

http://docwiki.embarcadero.com/RADStudio/Berlin/en/Program_Control#Handling_Function_Results

variant (well documented ;) ):

"For a string, dynamic array, method pointer, or variant result, the

effects are the same as if the function result were declared as an additional var parameter following the declared parameters. In other words, the caller passes an additional 32-bit pointer that points to a variable in which to return the function result."

http://docwiki.embarcadero.com/RADStudio/Berlin/en/Program_Control#Handling_Function_Results

And them being var-parameters basically states that they retain whatever value they had.

Regards,
Sven

Does that mean that we now need to read the whole documenation of Free Pascal *and* also the whole documentation of Delphi? And what version of Delphi?
Where is this documented within Free Pascal?

This has definitely changed with Free Pascal 3 as my programs suddenly did not work anymore and I had to add Setlength() all over my code. Before we could rely on that managed types where always initialized (well, that's the purpose of managed types, no?). It took me quite a while to find out why my programs failed as I did not expect such a (hidden) change.

variant (well documented ;) ):

> "For a string, dynamic array, method pointer, or variant result, the

effects are the same as if the function result were declared as an additional var parameter following the declared parameters. In other words, the caller passes an additional 32-bit pointer that points to a variable in which to return the function result."

>

http://docwiki.embarcadero.com/RADStudio/Berlin/en/Program_Control#Handling_Function_Results

Does that mean that we now need to read the whole documenation of Free

Pascal *and* also the whole documentation of Delphi?

And what version of Delphi? Where is this documented within Free Pascal?

No one said that FPC's documentation is perfect, we do our best, but that might still mean that there are missing pieces of information that we try to fix as soon as they are known to be missing.

Regards,
Sven

Hi,

Jürgen Hestermann wrote

This has definitely changed with Free Pascal 3 as my programs suddenly did not work anymore and I had to add Setlength() all over my code. Before we could rely on that managed types where always initialized (well, that's the purpose of managed types, no?).

No. Local variables are not initialized by default. Managed type or not, doesn't make a difference. Result is just another local variable in this case. I think the compiler will even warn about using uninitialized local variables, at least with -O3 and above (or when Data Flow Analysis is enabled).

It took me quite a while to find out why my programs failed as I did not expect such a (hidden) change.

It wasn't an intentional change for sure. However, the generated code might have changed (hopefully it became more optimal) which simply exposed some of these issues within the compiled code.

Actually, such issues are much easier to notice on other architectures, aside from i386, because of the different calling convention and register layout they use. I spent the better part of the last 3 years fixing code which was originally intended for i386, to get it working better on ARM (and others), where I ran into this exact "result remains uninitialized" issue, along with many other "my code is good, fix your compiler" issues, where it turned out the code was basically working accidentally, and it was built on false assumptions...

Charlie

Karoly Balogh (Charlie/SGR) wrote

No. Local variables are not initialized by default. Managed type or not, doesn't make a difference.

This is not true.
Local variables of managed types are of course initialized! If you declare

var S : String;

then it is of course initialized to nil.
Otherwise the (random) pointer stored in S would be used to change the reference counter when you assign:

S := '';

The only exception (at least since FPC 3) is the function result which is totally unexpected.
Why such an exception?

And it is not even documented (at least not within FPC)! Which I find a bit strange because it is very important to know. Omitting such information in the documentation is not good. Here
http://www.freepascal.org/docs-html/ref/refsu15.html it says:

"When declaring a variable of a dynamic array type, the initial length of the array is zero."

but no word about that this is not true when this array is a the result of a function.

My understanding (having been bitten by this) is that the requirement is only that S has a "sensible" value, so that the heap is not corrupted on the assignment, as you indicate. The compiler can re-use another variable instead, so the unset value is some other valid string.

Colin

the array is zero."

but no word about that this is not true when this array is a the result

of a function.

But also no word that it is true either, cause you're not declaring a variable for the Result.

Regards,
Sven

Hi,

On Sun, 26 Jun 2016, Jürgen Hestermann wrote:

Ok, scrap what I wrote before... :|

You are right about the managed types initialization. But then the documentation needs to be corrected there too. But actually, result is not a local var, but in fact a "special" parameter of the function. My bad. See below.

The only exception (at least since FPC 3) is the function result which is totally unexpected. Why such an exception?

There's no exception. At least not specifically for Result. It is simply variable passed by reference from the caller side, therefore it's not initialized inside the function. And this is A., consistent with other var parameters B., apparently also how Delphi does it. Actually, since managed types are always passed by reference, this is not really a surprise, nor an exception.

Actually, I tend to agree that it is very misleading. Also, you are also right that this is new behavior in current compiler. At least 2.6.2 (I don't have 2.6.4 at hand) decreased the reference to the result parameter before handing it to the called function, which no longer seems to happen. (I'm looking at the generated code.)

I have vague memories of a lengthy thread related to this, which changed the refcounting behavior of reference passed managed types/strings, maybe we're observing a sideeffect of this? (I think it was related to what happens if the same reference counted variable is passed to multiple var types and/or results, maybe also out parameters were involved. It's a pretty grey area, which is not really documented anywhere...)

To show the problem, what happens with:

var
  a: ansistring;
a:=some_function(a);

If the caller code destroys the result variable (by decreasing the reference count, because it's a result, so lets initialize it) it won't be valid by the time you pass it as parameter "a', because only the function itself will increase its reference then... Also, depending on the order of parameter passing, you might get different results on different CPU platforms... (Fix me?) Of course the function itself could also initialize the result, but we're back to square one with that, if result and one of the parameters are the same, weird things might happen.

I wish someone with real clue of calling conventions would comment. Jonas maybe?

Charlie

I don't think one could treat a function result as a normal local variable. The reason why not to initialise the result variable is because in many cases it is not necessary and reduces performance.

Martin

Sven Barth wrote

But also no word that it is true either, cause you're not declaring a variable for the Result.

Of course I am declaring a variable:

function X : String;

Now the (local) variable "Result" is of type "String". I use it in the same way as any other local variable. And because it is a manged type I could rely on the documentation which says that managed types are initialized (with zero length/nil). "Result" is used in the same way as a (local) variable. What should make me think that it is different?

Karoly Balogh (Charlie/SGR) wrote

There's no exception. At least not specifically for Result. It is simply variable passed by reference from the caller side, therefore it's not initialized inside the function.

You mean when I have the following:

--------------------------
function X : ansistring;
begin
end;
var S : ansistring;
S := X;
--------------------------

Then S is a dangling pointer to nowhere?

And this is A., consistent with other var parameters B., apparently also how Delphi does it. Actually, since managed types are always passed by reference, this is not really a surprise, nor an exception.

Of course it is a surprise because the documenation says that managed types are always initialized. If I use a var parameter I am forced to use a variable (declared somewhere else) but the function result can also be used in expressions without any (direct) assignment to a variable. So it must behave like a locally declared variable IMO.

If I use an intermediate variable like:

--------------------------
function X : ansistring;
var X1 : ansistring;
begin
Result := X1+'A';
end;
--------------------------

then it is initialized while here

--------------------------
function X : ansistring;
begin
Result := Result+'A';
end;
--------------------------

I get garbage?
That's a very strange behaviour and no longer Pascal, it's C with all its funny side effects.

var a: ansistring; a:=some_function(a);

When you have a parameter then of course it depends on how the parameter is declared.

The Result variable within a function is different and not a parameter. You cannot feed in any data here. I would consider this more like an out parameter
but in case of managed types an initialization
must take place somewhere (IMO in the function like for all other local variables too, where else?).

And it is definitly a change with FPC 3 because all my programs relied on that managed types are initialized and suddenly they are not anymore. I am wondering what other surprises lurk here and there which I have just not found out.

Hi,

Jürgen Hestermann wrote

And because it is a manged type I could rely on the documentation which says that managed types are initialized (with zero length/nil). "Result" is used in the same way as a (local) variable. What should make me think that it is different?

as Barry Kelly pointed out there
http://stackoverflow.com/questions/861045/which-variables-are-initialized-when-in-delphi/861178#861178, initialization differs on what mechanism was used to allocate memory . AFAIU, "initialized" "zero-filled" .

regards,

Hi,

No. In this case, S was initialized (to nil), because it's a global var.
This just won't be changed by the X function call. If you had S:='test';
Before S:=X; then it would still contain 'test' after it.

Charlie

Variables are declared with "var", a function result is not declared with "var".

Regards,
Sven

Philosophical arguments aside, I filed a bug against the documentation, since one thing is clear-- that I and others were previously misled into believing something that is not true.

http://mantis.freepascal.org/view.php?id=30321

In the report, I included several references where the documentation could be improved to make the compiler's behavior more clear. If anyone wants to expand on that list, please feel free :)

Also perhaps the wiki could use some work in this regard.

~David.