Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / fpc-pascal / 2010-06

PChar & AnsiString

26 messages · 11 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

spir ☣

Hello,

The documentation in the ref manual about PChar may have i bit more details: http://www.freepascal.org/docs-html/ref/refsu13.html#x36-390003.2.7

Do the following statements hold true?
* This type is mainly intended to interface with C code (or for low-level needs?). Else AnsiString should be prefered (even for low-level, since AnsiString is also referenced via pointer?).
* Like C strings, and unlike AnsiString-s (even if the latter also are "pointed"), PChar strings cannot hold NULL characters (#0). I just checked this point.

Also:
* How is length computed (traversal?)?

Can AnsiStrings be safely used as dynamic byte arrays? For instance to benefit of ref counting and copy_on_write (if any benefit). Or is it recommended to use Array of Byte?

What is the actual benefit of copy-on-write? I ask because of the following reasoning:
* If a string is just used at several places, for example in output or into bigger strings, then there is no reason reason to copy it into a new variable.
* If a programmer explicitely assigns an existing string to a new variable, the intent is precisely copy-semantics, to make them independent for further changes. If there is no change, there is also no reason for such an assignment. As a consequence, s2:=s1 will nearly always be followed by modification of either string, which will result on copy anyway, according to copy-on-write semantics. So, the initial gain at assignment time is soon lost. While the cost I imagine in terms of type complexity remains (every builtin modification method must ensure copying; no user-defined modification method should be possible without using builtin ones -- else copy-on-write is lost and consequences undefined).

What happens if a programmer indirectly modifies an AnsiString (via a pointer) which ref count is > 1:

Var
    s1,s2 : AnsiString;
    pc    : PChar;
begin
    s1 := 'abcde' ; s2 := s1;
    pc := PChar(s1);
    pc[2] := 'X';
    writeln(pc,' ',s1,' ',s2);	// abXde abXde abXde
end.

There must be an error in my reasoning, else language designers would not bother with such complication. What do you think?

Denis
________________________________

vit esse estrany ☣

spir.wikidot.com

Felipe Monteiro de Carvalho
to spir ☣

2010/6/1 spir ☣ :

* Like C strings, and unlike AnsiString-s (even if the latter also are "pointed")

Sure if you cast an AnsiString to a PChar it will only go until the first #0. You can't magically add capabilities to the PChar type.

* How is length computed (traversal?)?

It isn't computed, it is stored as part of the string in memory. 4 bytes, so a string can go up to 2GB in size.

Felipe Monteiro de Carvalho
Michael Van Canneyt
to spir ☣: The documentation in the ref manual about PChar may have i bit more details:…
spir ☣ wrote 7 lines
Hello, The documentation in the ref manual about PChar may have i bit more details: http://www.freepascal.org/docs-html/ref/refsu13.html#x36-390003.2.7 Do the following statements hold true? * This type is mainly intended to interface with C code (or for low-level needs?). Else AnsiString should be prefered (even for low-level, since AnsiString is also referenced via pointer?).

PChar is for C code.

* Like C strings, and unlike AnsiString-s (even if the latter also are "pointed"), PChar strings cannot hold NULL characters (#0). I just checked this point.

Correct.

Also: * How is length computed (traversal?)?

Strlen traverses till the first null.

Can AnsiStrings be safely used as dynamic byte arrays? For instance to benefit of ref counting and copy_on_write (if any benefit). Or is it recommended to use Array of Byte?

You can use them.

What is the actual benefit of copy-on-write? I ask because of the following reasoning:

Copy on write is needed to preserve the Pascal nature of strings while keeping the benefits of reference counted strings.

After

A:='some string'; // Ref count is 1
B:=A;  // Ref count is 2
B[1]:='S'; // Copy, and ref count of B is 1.

the A[1]='s' should still hold true.

* If a string is just used at several places, for example in output or into bigger strings, then there is no reason reason to copy it into a new variable.
* If a programmer explicitely assigns an existing string to a new variable, the intent is precisely copy-semantics, to make them independent for further changes. If there is no change, there is also no reason for such an assignment.

This is not correct. Many strings are simply referenced several times.

Show quoted text 15 lines
As a consequence, s2:=s1 will nearly always be followed by modification of either string, which will result on copy anyway, according to copy-on-write semantics. So, the initial gain at assignment time is soon lost. While the cost I imagine in terms of type complexity remains (every builtin modification method must ensure copying; no user-defined modification method should be possible without using builtin ones -- else copy-on-write is lost and consequences undefined). What happens if a programmer indirectly modifies an AnsiString (via a pointer) which ref count is > 1: Var s1,s2 : AnsiString; pc : PChar; begin s1 := 'abcde' ; s2 := s1; pc := PChar(s1); pc[2] := 'X'; writeln(pc,' ',s1,' ',s2); // abXde abXde abXde end. There must be an error in my reasoning, else language designers would not bother with such complication. What do you think?

If the programmer does this, it is his own fault; A pchar typecast should be considered read-only and valid only for the duration of the expression. It states as much in the docs.

Michael.

spir ☣
to Michael Van Canneyt

On Tue, 1 Jun 2010 13:05:16 +0200 (CEST)
Michael Van Canneyt wrote:

[...]

Thank you for all answers (all is now clear for me :-).

> * If a programmer explicitely assigns an existing string to a new variable, the intent is precisely copy-semantics, to make them independent for further changes. If there is no change, there is also no reason for such an assignment. This is not correct. Many strings are simply referenced several times.
May I ask in which typical cases? (I can easily find examples of strings _used_ several times, eg concatenated into larger strings and/or output, but this does not make new references.)
Can only think at the special case of loop variables:
    for i:=0 to persons.count do begin
        name := persons[i].name;
        -- do something with name w/o changing it --
    end;

I find this question (from a larger point of view than strings only) very interesting. Some newer languages, such as Clojure, introduce reference semantics while basically preserving basic value immutability: a different approach. (I do not mean one is better than the other; instead try to understand why either is prefered by some language designers.)

Denis
________________________________

vit esse estrany ☣

spir.wikidot.com

Jonas Maebe
to spir ☣
spir ☣ wrote 7 lines
On Tue, 1 Jun 2010 13:05:16 +0200 (CEST) Michael Van Canneyt wrote: > This is not correct. Many strings are simply referenced several > times. May I ask in which typical cases?

The most common one is probably assigning a function result to a variable or field.

Jonas

Martin
to spir ☣: The documentation in the ref manual about PChar may have i bit more details:…
spir ☣ wrote 5 lines
What is the actual benefit of copy-on-write? I ask because of the following reasoning: * If a string is just used at several places, for example in output or into bigger strings, then there is no reason reason to copy it into a new variable. * If a programmer explicitely assigns an existing string to a new variable, the intent is precisely copy-semantics, to make them independent for further changes. If there is no change, there is also no reason for such an assignment. As a consequence, s2:=s1 will nearly always be followed by modification of either string, which will result on copy anyway, according to copy-on-write semantics. So, the initial gain at assignment time is soon lost. While the cost I imagine in terms of type complexity remains (every builtin modification method must ensure copying; no user-defined modification method should be possible without using builtin ones -- else copy-on-write is lost and consequences undefined).

No, an assignment, or increase in ref count is not necessarily followed by a change:

Procedure Foo(a:String);
begin ... end;
Procedure Bar;
var s: string;
begin
   s:= SomeString;
   Foo(s);
end;

Now a copy of s has to be given to Foo, because Foo *may* changes the string. But Foo may also *not* change the string => so leave it till later.

Example 2:

Procedure Bar;
var a, b: string;
begin
   a:= SomeString;
   b:=a;
   if SomeCondition then
      b:=b+"...";
   if SomeOtherCondition then
      b:=b+"???";
end;

So b may never be

spir ☣
to Martin
On 01/06/2010 11:23, spir ☣ wrote: > What is the actual benefit of copy-on-write? I ask because of the following reasoning: > * If a string is just used at several places, for example in output or into bigger strings, then there is no reason reason to copy it into a new variable. > * If a programmer explicitely assigns an existing string to a new variable, the intent is precisely copy-semantics, to make them independent for further changes. If there is no change, there is also no reason for such an assignment.

[...]

Show quoted text 15 lines
No, an assignment, or increase in ref count is not necessarily followed by a change: Procedure Foo(a:String); begin ... end; Procedure Bar; var s: string; begin s:= SomeString; Foo(s); end; Now a copy of s has to be given to Foo, because Foo *may* changes the string. But Foo may also *not* change the string => so leave it till later.
For me, parameter passing is just an implicit assignment "param:=s" (with the variant of var params). The same applies to return values when assigned back in calling code (*). I would have thus expected s the pointer to be passed. Just checked it is not the case. More secure, esp since legacy shortstrings do not behave that way.
This example rather backs my pov, I guess, since here copy-on-write does not apply. But it certainly could, since its application should prevent bad surprises, no? (unexpected change of original variable)
Show quoted text 14 lines
Example 2: Procedure Bar; var a, b: string; begin a:= SomeString; b:=a; if SomeCondition then b:=b+"..."; if SomeOtherCondition then b:=b+"???"; end; So b may never be , I did not mean this cannot happen. I rather spoke of the programmer's intent. My hypothesis is that it does not make sense to copy a string into another var if none will change. Your example is perfect :-) There are cases were this cannot be avoided.

Still, is this worth the complication of copy-on-write.

I also favor a super-simple and explicit coding style, which produces some more assignments, intermediate and not necessary, than clever coding with complex expressions. From this point of view, copy-on-write is an interesting scheme, in that it avoids useless copies (except for references), thus allowing such a coding style with minor penalty (ref copy only). But if the compiler is smart enough, such intermediate vars can probably be eliminated, no?

Denis

(*) And to some more constructs in other languages, like foreach (*the* feature I miss in freepascal):
   foreach name in names do ... end;
________________________________

vit esse estrany ☣

spir.wikidot.com

spir ☣
to Jonas Maebe: The most common one is probably assigning a function result to a variable or field. Jonas
Show quoted text 13 lines
On 01 Jun 2010, at 14:28, spir ☣ wrote: > On Tue, 1 Jun 2010 13:05:16 +0200 (CEST) > Michael Van Canneyt wrote: > >> This is not correct. Many strings are simply referenced several >> times. > > May I ask in which typical cases? The most common one is probably assigning a function result to a variable or field.

Is there increment of ref count (I think it does +1-1)? If no, passing the ref cannot have any side-effect, so the copy_on_write scheme does not apply -- or rather does not make any change if applied. This is plain reference copy. The issue is rather for parameter passing, since here the original var survives the call (see Martin's post).

Denis
________________________________

vit esse estrany ☣

spir.wikidot.com

Martin
to spir ☣: On Tue, 01 Jun 2010 15:00:47 +0100 [...] For me, parameter passing is just an implicit…
spir ☣ wrote 32 lines
On Tue, 01 Jun 2010 15:00:47 +0100 Martin wrote: > On 01/06/2010 11:23, spir ☣ wrote: > >> What is the actual benefit of copy-on-write? I ask because of the following reasoning: >> * If a string is just used at several places, for example in output or into bigger strings, then there is no reason reason to copy it into a new variable. >> * If a programmer explicitely assigns an existing string to a new variable, the intent is precisely copy-semantics, to make them independent for further changes. If there is no change, there is also no reason for such an assignment. >> [...] > No, an assignment, or increase in ref count is not necessarily followed > by a change: > > Procedure Foo(a:String); > begin ... end; > > Procedure Bar; > var s: string; > begin > s:= SomeString; > Foo(s); > end; > > Now a copy of s has to be given to Foo, because Foo *may* changes the > string. But Foo may also *not* change the string => so leave it till later. > For me, parameter passing is just an implicit assignment "param:=s" (with the variant of var params). The same applies to return values when assigned back in calling code (*). I would have thus expected s the pointer to be passed. Just checked it is not the case. More secure, esp since legacy shortstrings do not behave that way. This example rather backs my pov, I guess, since here copy-on-write does not apply. But it certainly could, since its application should prevent bad surprises, no? (unexpected change of original variable)

I don't know all the internals of FPC, but yes to my understanding, your quote:
"parameter passing is just an implicit assignment" is absolutely true.

So why do you then say "copy on write" would not apply? The assignment creates a 2nd reference, and a copy only happens if inside Foo a change is made to the string.
=> So according to your own words, copy-on-write is exactly what happens.

Shortstrings have nothing to do with it, they are not reference counted. They are a completely different data type.

Martin

spir
to Martin
Show quoted text 8 lines
I don't know all the internals of FPC, but yes to my understanding, your quote: "parameter passing is just an implicit assignment" is absolutely true. So why do you then say "copy on write" would not apply? The assignment creates a 2nd reference, and a copy only happens if inside Foo a change is made to the string.

You're right, I reasoned wrongly.

Deins
________________________________

vit esse estrany ☣

spir.wikidot.com

David Emerson
to spir ☣: On Tue, 1 Jun 2010 14:36:36 +0200 Is there increment of ref count (I think it does +1-1)?…
> >> This is not correct. Many strings are simply referenced several > >> times. > > > > May I ask in which typical cases?

In an earlier version of our database (before we had things properly typed) many things were stored as strings. Thus a common ansistring may have had a reference count in the hundreds or even thousands. Most of the time, if I changed one of them, I would not want to change them all! But it saved a lot of memory to not be redundantly storing the same string.

Show quoted text 8 lines
> The most common one is probably assigning a function result to a > variable or field. Is there increment of ref count (I think it does +1-1)? If no, passing the ref cannot have any side-effect, so the copy_on_write scheme does not apply -- or rather does not make any change if applied. This is plain reference copy. The issue is rather for parameter passing, since here the original var survives the call (see Martin's post).

I believe it depends on the circumstance. It's been a while since I tested this, but IIRC, when a variable is passed-by-value the ref count is incremented, whereas if it is passed-by-reference the ref count is not incremented... as one should hope. In the case of the latter, this means that if the ref count is one, then copy-on-write is not needed, as there is only the one instance of the string-- so that can save some processor time. Back when I was dealing with enormous ansistrings that were frequently being changed, I spent considerable time researching this so as to figure out how to avoid unnecessary copy-on-writes. What I found is that the compiler's treatment of them is quite intuitive: there were no unpleasant surprises. I ended up changing a number of functions which took a value parameter and returned a result to instead take a reference parameter and change that.

Have fun...
~David.

spir
to Bee Jay
Show quoted text 8 lines
On 1 Jun 2010, at 22:13, spir ☣ wrote: > (*) And to some more constructs in other languages, like foreach (*the* feature I miss in freepascal): > foreach name in names do ... end; http://wiki.freepascal.org/for-in_loop -Bee-

Great! Thank you very much. (Is this feature mentionned in "official" docs? how could I miss it? The ref guide only mentions "The For..to/downto..do statement", I guess.)

Denis
________________________________

vit esse estrany ☣

spir.wikidot.com

Graeme Geldenhuys
to spir
spir wrote
Great! Thank you very much. (Is this feature mentionned in "official" docs? how could I miss it? The ref guide only mentions "The For..to/downto..do statement", I guess.)

I don't think it's in FPC 2.4.1 either, only Trunk (2.5.1) - so no, the current 2.4.1 documentation will not mention this.

Regards, - Graeme - _______________________________________________ fpGUI - a cross-platform Free Pascal GUI toolkit http://opensoft.homeip.net/fpgui/
Michael Van Canneyt
to spir: On Wed, 2 Jun 2010 15:19:55 +0700 Great! Thank you very much. (Is this feature mentionned…
spir wrote 13 lines
On Wed, 2 Jun 2010 15:19:55 +0700 Bee Jay wrote: > On 1 Jun 2010, at 22:13, spir ☣ wrote: > >> (*) And to some more constructs in other languages, like foreach (*the* feature I miss in freepascal): >> foreach name in names do ... end; > > http://wiki.freepascal.org/for-in_loop > > -Bee- Great! Thank you very much. (Is this feature mentionned in "official" docs? how could I miss it? The ref guide only mentions "The For..to/downto..do statement", I guess.)

The feature is in test in 2.5.1. As soon as it makes it to a stable release, the docs will be updated.

Michael.

Marco van de Voort
to Michael Van Canneyt

In our previous episode, Michael Van Canneyt said:

Show quoted text 9 lines
>> >> http://wiki.freepascal.org/for-in_loop >> >> -Bee- > > Great! Thank you very much. (Is this feature mentionned in "official" docs? how could I miss it? The ref guide only mentions "The For..to/downto..do statement", I guess.) The feature is in test in 2.5.1. As soon as it makes it to a stable release, the docs will be updated.

Afaik it is merged into 2.4.1 already.

Graeme Geldenhuys
to Marco van de Voort
Marco van de Voort wrote
Afaik it is merged into 2.4.1 already.

Did somebody actually test it other than Paul (that also implemented it)? If so, that was a rather quick test for a big compiler change.

Regards, - Graeme - _______________________________________________ fpGUI - a cross-platform Free Pascal GUI toolkit http://opensoft.homeip.net/fpgui/
Michael Van Canneyt
to Marco van de Voort: In our previous episode, Michael Van Canneyt said: Afaik it is merged into 2.4.1 already.
Marco van de Voort wrote 12 lines
In our previous episode, Michael Van Canneyt said: >>> >>> http://wiki.freepascal.org/for-in_loop >>> >>> -Bee- >> >> Great! Thank you very much. (Is this feature mentionned in "official" docs? how could I miss it? The ref guide only mentions "The For..to/downto..do statement", I guess.) > > The feature is in test in 2.5.1. As soon as it makes it to a stable release, > the docs will be updated. Afaik it is merged into 2.4.1 already.

Huh ? Really ? Strange, because it is a rather invasive change.

Michael.

Jonas Maebe
to Graeme Geldenhuys: Did somebody actually test it other than Paul (that also implemented it)? If so, that…
Graeme Geldenhuys wrote 6 lines
On 2 June 2010 14:16, Marco van de Voort wrote: > > Afaik it is merged into 2.4.1 already. Did somebody actually test it other than Paul (that also implemented it)?

Yes (at least other people reported bugs about it on fpc-devel).

If so, that was a rather quick test for a big compiler change.

There was a discussion about it on core. It was merged because a) the merge occurred shortly after the previous release, so there was a lot of time to test its effect on the fixes branch b) it did not affect backwards compatibility
c) the compiler internally simply transforms it into a regular while/ do or for/do loop with temporary variables, and then the existing code takes over. So code was mainly added, which means that that there was little risk of breaking existing functionality

Jonas

Graeme Geldenhuys
to Jonas Maebe

Op 2010-06-02 14:48, Jonas Maebe het geskryf:

Yes (at least other people reported bugs about it on fpc-devel).

If I remember correctly, those discussions were only while it was implemented. No real-world usage afterwards.

There was a discussion about it on core. It was merged because a) the merge occurred shortly after the previous release, so there was a lot of time to test its effect on the fixes branch

Just because it's in Trunk or Fixes branch, doesn't mean it's actually used and thus tested. From this message thread, not many even knew it was implemented. Hence my concern about it moving so quickly into Fixes branch.

Regards,
- Graeme -

fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal http://opensoft.homeip.net/fpgui/
Jonas Maebe
to Graeme Geldenhuys
Graeme Geldenhuys wrote 6 lines
Op 2010-06-02 14:48, Jonas Maebe het geskryf: > > Yes (at least other people reported bugs about it on fpc-devel). If I remember correctly, those discussions were only while it was implemented. No real-world usage afterwards.

Most features only get extensive real-world usage once they appear in a released version.

Show quoted text 10 lines
> There was a discussion about it on core. It was merged because > a) the merge occurred shortly after the previous release, so there > was > a lot of time to test its effect on the fixes branch Just because it's in Trunk or Fixes branch, doesn't mean it's actually used and thus tested. From this message thread, not many even knew it was implemented. Hence my concern about it moving so quickly into Fixes branch.

Hence points b) and c) in my original mail. There are two separate things:
1) does it endanger the stability of existing code? Answer: unlikely in this case (and this is the most important question for the fixes branch).
2) is the feature itself stable? Answer: uncertain, although Paul did add a lot of tests to the testsuite, other people also tested it, I reviewed most of the code for it (and Florian possibly a well), and in general features often experience some growing pains when they are first released because of the reason mentioned above.

Jonas

spir
to Graeme Geldenhuys: Did somebody actually test it other than Paul (that also implemented it)? If so, that…
Show quoted text 7 lines
On 2 June 2010 14:16, Marco van de Voort wrote: > > Afaik it is merged into 2.4.1 already. Did somebody actually test it other than Paul (that also implemented it)? If so, that was a rather quick test for a big compiler change.

I will test it! (even have code ready, since I did not know it was not implemented on the release I currently have). Just need to upgrade (hope it's easy on ubuntu).

denis
________________________________

vit esse estrany ☣

spir.wikidot.com

Paul Ishenin
to Graeme Geldenhuys: Op 2010-06-02 14:48, Jonas Maebe het geskryf: If I remember correctly, those discussions…
Graeme Geldenhuys wrote 5 lines
Just because it's in Trunk or Fixes branch, doesn't mean it's actually used and thus tested. From this message thread, not many even knew it was implemented. Hence my concern about it moving so quickly into Fixes branch.

It will be used by Lazarus only after the release. Before that we need to wait.

Best regards,
Paul Ishenin.