Hi,

I asked a question about a compiler hint in the fpc-users mailing list. As JoshyFun suggested, is it not maybe better to change FillChar() definition so first parameter is a out parameter - to prevent unnecessary compiler hint in code?

Graeme Geldenhuys wrote

I asked a question about a compiler hint in the fpc-users mailing list. As JoshyFun suggested, is it not maybe better to change FillChar() definition so first parameter is a out parameter - to prevent unnecessary compiler hint in code?

No, that is not possible. I once tried to change move and fillchar to use "out" parameters instead of "var" parameters, and the result was all sorts of crashes. The reason is that "out" has special semantics for reference counted types (they are finalized at the *caller* side), and move/fillchar are sometimes used to zero uninitialised data (so if the run time then tries to finalise garbage, you get crashes).

Jonas

In that case I suspect an implementation insect. The compiler/codegenerator should not differ between var and out, just the parser should distinguish imho. A reference should never influence "reference counting" and should surely not have anything to do with finalisation at all, not even because copy on. Most other languages behave like that I think. What is the reason FPC differs?

Delphi compatibility. And Delphi does that because COM requires this behaviour.

Jonas

Jonas Maebe wrote

Delphi compatibility. And Delphi does that because COM requires this behaviour.

Yes, but.. As I hinted before that is because COM is reference counted on an intermediate level by a certain OS. A simple (but performance cost) change of the memorymanager can fix that for that OS - I wrote one and we use it in real life -. And we can play on with the rest of the OS's. So I suggest the following adjustment, also applicable to widestring: the fact that the com subsystem is reference counting prevents the language from reference counting. That sounds and is silly ;)

Jonas Maebe wrote

Delphi compatibility. And Delphi does that because COM requires this behaviour.

Can't that behaviour be limited to Windows platform only. Now *all* platforms and all non-COM code has to be stuck with the useless compiler hint simply because of some Windows and more specifically Windows + COM behaviour.

Such behaviour & implementation just doesn't seem right for a project that prides itself by being a cross-platform compiler (project).

To me, Windows + COM seem the minority use-case here. But now all other platform and Windows non-COM code seem stuck with the useless hint being reported.

As I stated in the other mailing list. It's not about a obsession to get hint & warning free code. It's about spoting REAL issues in code between all the crap the compiler currently spits out. If your project uses a lot of structure types, you can quickly sit with thousands of lines of compiler hints like this one. There is no easy way of spotting real hint compared to fake (useless) hints. And I definitely don't have the time to seek through my rather large (200k + LOC projects) projects hint output to try and figure out what is real and what is useless hints.

Regards,
- Graeme -

Without COM, FPC wouldn't have out parameters.

Florian Klaempfl wrote

Without COM, FPC wouldn't have out parameters.

Why do you say that? I see many use-cases for out parameters - all without using Windows or Windows+COM.

Not to mention, even Kylix (which for obvious reasons doesn't have COM support) had out parameter support. So even the Object Pascal language creator, Borland, thought out parameters are useful, otherwise Kylix wouldn't have such support either.

Regards,
- Graeme -

Graeme Geldenhuys wrote on do, 19 nov 2009:

It would change the behaviour of code between Windows and non-Windows platforms. One thing it would certainly affect is assembler code.

More importantly, another thing it would affect is that in case someone has written their own fillchar with an out parameter, and *expects* reference counted structures to be finalised at the caller side before they are overwritten with zeroes by their routine. Changing that behaviour woud
a) result in memory leaks when using such a routine b) not be fixable without manually adding finalize calls everywhere this custom fillchar routine is called, since the required typeinfo is not available at the callee side

Such behaviour & implementation just doesn't seem right for a project that prides itself by being a cross-platform compiler (project).

Cross-platform also means having the same behaviour on different platforms.

Jonas

---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.

This does indeed sound like an horrible hack for me, but I do not understand the problem fully.
Can you give an example code which would be (badly) affected by proposed change?

Alexander Klenin wrote on do, 19 nov 2009:

It's not a hack at all, it's relying on 100% defined behaviour.

Can you give an example code which would be (badly) affected by proposed change?

type
   tr = record
     str: ansistring;
   end;
procedure clear(out rec; size: ptruint);
begin
   fillchar(rec,size,0);
end;
var
   r: tr;
begin
   r.str:=paramstr(0);
   clear(r);
end.

If you change the behaviour of "out" so that such parameters are no longer finalised before the routine is entered, then the above will cause a memory leak. You can verify this by changing the "out rec" parameter into "var rec".

Jonas

---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.

Florian Klaempfl wrote: > Without COM, FPC wouldn't have out parameters. Why do you say that?

Because I added it for Delphi compatibility which needs it for COM?

I see many use-cases for out parameters

You mean for VAROUT parameters :)?

Well in this case, the code is actually positively affected by the "out" param (because it avoids the mem leak)

But if you ever handled ref-counted data by hand.....

type TStringArray = Array of string;
procedure InsertOne(a: TStringArray);
var l : integer;
begin
  l := length(a)
  SetLength(a, l + 1)
  Move(a[0], a[1], l*SizeOf(String));
  // now a[0] and a[1] point to the same string
  // but the string has still a ref-count of 1
  // so assigning a[0] := nil => would release the memory used by a[1]
  FillChar(a[0], SizeOf(String), 0);
end;

the above works, because Filchar has a var param. otherwise it would finalize the string, and a[1] would point to released memory....

Similar case happen, if your array was filled with random, none zero data, and now need to be accessed via string functionality. The only way to set the entries to 0 is FillChar. If you use a[]:=nil while a[0] has random data, then it would attempt to free the string at the random location , and that would crash since there is no string

Martin

Florian Klaempfl wrote

> I see many use-cases for out parameters You mean for VAROUT parameters :)?

I search the latest FP Language Reference document, and there is no mention of 'varout'. I also tried to use varout in a procedure as follows - which gave a compiler error with FPC 2.4.0-rc1

 procedure dcpFillChar(varout x; count: SizeInt; Value: Byte); overload;

vs

 procedure dcpFillChar(out x; count: SizeInt; Value: Byte); overload;

First one is not compilable, but the second one is. So no, I don't understand your comment about 'varout'? Please explain more.

Where I use 'out' parameters often is in the case where a function needs to return more than one value. For example the standard return value of the function is Boolean to say if the function as a whole succeeded or not. The out parameters are actual values that will be initialized and what I will work with after the function call.

Regards,
- Graeme -

Alexander Klenin wrote

Can you give an example code which would be (badly) affected by proposed change?

Thanks Alexander, you beat me to that question. :-)

Regards,
- Graeme -

varout could be the name of the new parameter modifier that Jonas mentions[1] and that has the semantics that you think out should have, but hasn't.

Vincent

[1]http://lists.freepascal.org/lists/fpc-pascal/2009-November/023223.html

It is not implemented, but it would be the solution to the fillchar problem.

A VAROUT parameter could have the same semantics as VAR except that the compiler does not expect that it is needed that it is initialized. But be warned: with such a parameter type you can easily create memory leaks with automated types like ansistrings.

The problem with varout is that it easily creates memory leaks with automated types.

Vincent Snijders wrote

varout could be the name of the new parameter modifier that Jonas mentions[1] and that has the semantics that you think out should have, but hasn't.

Ah, now I understand. Thanks Vincent.

Regards,
- Graeme -

Florian Klaempfl wrote

initialized. But be warned: with such a parameter type you can easily create memory leaks with automated types like ansistrings.

Well, isn't that what heaptrc is for? I enable heaptrc by default for all my projects (except on release builds). That way I can catch memory leaks as soon as I introduced them - and much easier to find. I would recommend this to all.

Regards,
- Graeme -

No, the language should protect you from such easy to make mistakes. In theory don't want to know how a type is implemented and if it is safe to pass as parameter (unless the compiler forbids using varout for such types)
Weren't you recently involved in a discussion about knowing the under-the-hood implementation of automated types and their behaviour ?

Marc

Then maybe the solution is to add someting like

{$uninitialized_arguments_check_for_functions off}
procedure FillChar(var ...)
begin
...
end;
{$uninitialized_arguments_check_for_functions on}

Compiler will add a flag for each var/const argument that they dont require that checks.

Best regards,
Paul Ishenin.

This looks really ugly :) Maybe a better solution is to implement a compiler internal counter part to finalize which simply zeros the variable and where the compiler knows that the variable must not be initialized before.

Marc Weustink wrote

> > Well, isn't that what heaptrc is for? No, the language should protect you from such easy to make mistakes.

I guess that's the ideal situation. Currently the language doesn't stop me from creating other memory leaks like not freeing a TStringList or other object instance. Using pointers can easily cause memory leaks too. No language features to prevent those - and we all seem to get along ok. :)

for such types) Weren't you recently involved in a discussion about knowing the under-the-hood implementation of automated types and

Please remind me of the message thread so I can recap. I'm part of a lot of discussions per day, so can't remember the details of them all.

Regards,
- Graeme -

I still do not fully understand.

Martin wrote

Jonas Maebe wrote:

[skipped example]

Well in this case, the code is actually positively affected by the "out" param (because it avoids the mem leak)

Jonas, can you confirm that your example is incorrect one, and Martin's example below is actually what you meant?

Ugh, now this is what I consider to be very ugly.
I did actually wrote exactly this kind of code for Delphi, but it never even occurred to me to rely on that undocumented distinction between out and var parameters. I wrote something like

   Integer(a[0]) := 0;

instead.

So I'd say that this the kind of broken code not worth supporting.

Paul Ishenin wrote

Compiler will add a flag for each var/const argument that they dont require that checks.

Can't the compiler simply initialize structured types by default? Initialize char arrays to empty strings, other structures like pointers to nil, byte arrays to #0 etc..?

After all, this is already done for AnsString, Integer type, real types etc...

Regards,
- Graeme -

Why do you think my example is incorrect? It currently runs without a memory leak, and would result in a memory leak if the semantics of "out" would be changed from "parameters are automatically finalized at the caller side" into "no finalization is performed".

You did ask for an example where changing the semantics of "out" would change the behaviour of existing code, didn't you?

Jonas

Then this is definitely not a sextopod but an octopod. (really serious bug) The "out" should behave like "var" + referencecount and finalization outside the procedure or function.

Jonas is right, but...

I think it got confusing. The thread changed from: "redefine filchar to use out" to "change the behaviour of out, so fillchar can use it"

Jonas example applies to "change the behaviour of out, so fillchar can use it"
My example applies to "redefine filchar to use out"

Martin

In our previous episode, Graeme Geldenhuys said:

No. That is for memory leaks you accidentally forget, AFTER ALL HAS BEEN DONE TO AVOID THEM.

I think that nobody argued against the semantics of "out" parameters -- IMHO the current semantics is good.

You did ask for an example where changing the semantics of "out" would change the behaviour of existing code, didn't you?

No, I asked for an example of code that would be negatively affected by changing "var" to "out" in FillChar parameter.

The compiler already initialises the reference counted fields of structured types by default, because otherwise the code would crash. The rest is not initialised.

Initialize char arrays to empty strings, other structures like pointers to nil, byte arrays to #0 etc..?

Yes, the compiler can do that, at a cost. Here's the speed impact on the compiler itself: http://lists.apple.com/archives/objc-language/2009/Sep/msg00094.html (a more "optimized version" turned out to be slower, because it zeroed more data).

After all, this is already done for AnsString, Integer type, real types etc...

It is done for reference counted types (such as ansistring, widestring, non-corba interfaces, ...). It is /not/ done for integer, real, etc.

You may be confused by the fact that all global variables are zero on startup.

Jonas

That is exactly how it behaves right now, so I don't understand your remark. My example is about what would happen if the behaviour of "out" would be changed into the same as "var" (except that the compiler would not give *hints* about potentially uninitialised data).

Jonas

Alexander Klenin wrote

No, I asked for an example of code that would be negatively affected by changing "var" to "out" in FillChar parameter.

I don't know by heart anymore, but as I mentioned the compiler crashed when I tried that (or if you did that move, which has the same problem). There is probably rtl or generated initialisation code that depends on the current semantics of move/fillchar).

Jonas

I thought it does that?

But FillChar is the same lowlevel like move. (IMHO) Move doesn't care about ref-counts. And if you use move, you must start to take care of refcounts yourself. Ani if you have to care about them, you need a fillchar that doesn't affect them....

But we start discussing 2 different unctiuons
FillChar
FillcharWithPreFinalization

they do differrent things, both have a use case.

The initial problem of this thread was not: Fillchar doesn't finalize ref-counted types

It was about the hints.
What is wanted is a Fillchar (that does not care about ref-counts, as it is currently) but that does not hint neither. => that is behaviour of a var param, but without hint.

If you want Fillchar with previous finalization of ref counted types, then FillChar is not the function you need.

Fillchar is as lowlevel as move / and I wonder: does move give hints too, if the targer is not initalized?

Martin

So, we are unable to fix FillChar because it would uncover bugs hiding in the other places? ;-)
Ok, so how about introducing, say, FillMem procedure, which is identical to FillChar
except that it has "out" parameter, and deprecation of FillChar? I always thought that "FillChar" name is somewhat misleading anyway, so this is a good opportunity to change it.

Why? the other places use fillchar in exactly they way it is supposed to be used. Setting ref-counted types while bypassing the normal ref-counted finalization. That isn't a bug.

fillchar wih var or out are 2 different functions, providing a different functionality (even though they do overlap quite a bit)

Martin

See Martin's reply.

Ok, so how about introducing, say, FillMem procedure, which is identical to FillChar except that it has "out" parameter,

See http://lists.freepascal.org/lists/fpc-devel/2009-November/018559.html

FillMem would be a very bad name though, because it in now way suggests how it would differ from FillChar (a bit like the confusing AllocMem vs GetMem). FillCharFinalized, or FinalizeAndFill or so would be more appropriate.

and deprecation of FillChar?

I don't see any reason to deprecate FillChar.

Jonas

Or simply FillCharOut?
Thinking about it some more, I am wondering -- maybe this is not the real :

var  a: array of String;
begin
  ... fill a with some stings ...
  FillChar(a[0], SizeOf(a) * SizeOf(string), 0);
end;

I suspect that both FillChar and FillCharOut are wrong here. So, maybe what we actually need is FillArray(out array; const value) half-magical function,
so that the above code might be rewritten as FillArray(a, '') ? Another related possibility would be SetLengthInit(out array; len: Integer; const value);

Florian Klaempfl :

Without COM, FPC wouldn't have out parameters.

And because there would be no difference between "var" and "out" then, it also wouldn't have the hint. Case dismissed. ;)

Vinzent.

There would be difference: no out keyword at all :)

it also wouldn't have the hint. Case dismissed. ;)

Yup.

Florian Klaempfl :

A VAROUT parameter could have the same semantics as VAR except that the compiler does not expect that it is needed that it is initialized. But be warned: with such a parameter type you can easily create memory leaks with automated types like ansistrings.

I don't understand the issue. Firstly, it's just a keyword. Secondly, the compiler should know if the actual type is reference counted or not.

So if the programmer can decide which keyword to use, why the hell should it be a problem for the compiler (which usually knows much better than the programmer)?

IMO, something like

-- 8< --
if Is_Reference_Counted (Type) then
   Parameter_Mode := Out_Ref;
else
   Parameter_Mode := Plain_Ref;
end if;
-- 8< --

can't be so hard to accomplish, can it?

I even thought, it's impossible for reference counted types to be uninitialized (in a purely technical way), because the compiler does the initialization automatically? Or how else can it rely on the associated reference count?

Vinzent.

Indeed, I noticed :)

Graeme Geldenhuys :

Can't the compiler simply initialize structured types by default?

[...]

After all, this is already done for AnsString, Integer type, real types etc...

No, it's not (except for AnsiString, because it's reference counted).

Vinzent.

Florian Klaempfl :

Care to explain why the programmer can easily determine something which seems to be an unsolvable problem for the compiler, although both have exactly the same information?

Vinzent.

Because a VAROUT parameter would be simply overwritten by the callee even if it contains a valid automated type:

procedure p(varout v : ansistring);
begin
  v:='asdf';
end;
var
  s : string;
begin
  s:='asdf';
  s:=s+s;
  p(s); <-- memory leak, s+s is never freed
end;

OUT does prevent this because s is cleaned up before p is entered. OTOH, OUT causes a crash if s is filled with garbage.

I don't think that this could cause a memory leak, otherwise this would cause two memory leaks:

procedure p(varout v : ansistring);
begin
 v:='asdf';
 v:=v+v;
 v:='test;
end;

The fact that v is a varout would simply mean that it would not be finalized before the call. It wouldn't necessarily mean that reference counting would no longer be applied inside the callee. Just like with regular "out" parameters the compiler inside the callee still calls FPC_DECR_REF* on the out parameter every time you assign something to it, even though it's finalized before the call.

What would cause a memory leak, is this:

procedure clear(varout v; size: ptruint);
begin
 fillchar(v,sizeof(v),0);
end;
var
 s : string;
begin
 s:='asdf';
 s:=s+s;
 clear(s,sizeof(s));
end;

But that's the same with "var".

Jonas

Florian Klaempfl :

Because a VAROUT parameter would be simply overwritten by the callee even if it contains a valid automated type:

That's a semantic definition, not an explanation. IOW: What stops you from selecting to implement the proper semantics depending on the type?

(Especially when introducing another new keyword for exactly clarifying intended semantics?)

OUT does prevent this because s is cleaned up before p is entered. OTOH, OUT causes a crash if s is filled with garbage.

Assuming that one doesn't play with pointers and typecasts a lot, the probability of an automated type containing garbage should be rather low.

Nonetheless, I still consider a

"FillAnyArrayWithZeros (out X : DynamicArray)"

which deletes the whole array instead of filling it up to it's defined size with zeros rather counterintuitive. I expect the single array elements to contain garbage (which I want to clean out), not the defined array itself...

Vinzent.

Sorry, but it seems you didn't follow up the thread, so I won't continue here.

Florian Klaempfl :

I followed the thread at least up to the point where "varout" was proposed as a possible solution.

So, considering that it's not even implemented, it's rather awkward to discuss semantics which it would have when it were implemented - and then explain why it still would not solve the problem.

I always thought that for a new feature the proper semantics should be discussed first and then how one would implement it. Now it seems, the only possible implementation is already done and cast in stone and now you try to explain why the semantics caused by the implementation still doesn't solve the problem and thus wouldn't be worth to be implemented... is that recursive or circular now?

Vinzent.