Hello:

I'm working on windows XP, FPC 3.0.0 from stable Lazarus 1.6.

I've come across this issue: When I concatenate two strings in UTF8 they are converted to ansi (Win-1252) .
A bug?
Am I missing something?

I have attached a demo.

Santiago A. wrote

I've come across this issue: When I concatenate two strings in UTF8 they are converted to ansi (Win-1252) .

You have declared all string variables as plain "string", which is the same as AnsiString(CP_ACP). So all string variables have the encoding of your active codepage.

Declare Utf8StrA and related as Utf8String.
In DisplayBytes do not use "String" as parametertype, since this will again automatically convert things.
The AnsiToUtf8 is not necessary anymore if done this way:

procedure DisplayBytes(S:RawByteString);
var
  i:Integer;
begin
  Write('  ');
  for i:=1 to length(s) do
    write(ord(s[i]),' ');
  writeln;
end;
//-----------------------------------
// body
//-----------------------------------
var
  AnsiStrA:string;
  AnsiStrB:string;
  Utf8StrA: utf8string;
  Utf8StrB:utf8string;
  Utf8StrConcat:utf8string;
begin
  AnsiStrA:=' ';
  AnsiStrA[1]:=#243; // o acute win-1252
  AnsiStrB:='A';
  Write('AnsiStrA: ');DisplayBytes(AnsiStrA); // 243
  Write('AnsiStrB: ');DisplayBytes(AnsiStrB); // 65
  Utf8StrA:=(AnsiStrA); // 195 179
  Utf8StrB:=(AnsiStrB); // 65
  writeln;
  Write('Utf8StrA: ');DisplayBytes(Utf8StrA); // 195 179
  Write('Utf8StrB: ');DisplayBytes(Utf8StrB); // 65
  Write('Utf8StrA+Utf8StrB: ');DisplayBytes(Utf8StrA+Utf8StrB);
  writeln;
  Write('Utf8StrA again: ');DisplayBytes(Utf8StrA); // 195 179
  Write('Utf8StrB again: ');DisplayBytes(Utf8StrB); // 65
  Utf8StrConcat:=Utf8StrA+Utf8StrB;
  writeln;
  Write('Utf8StrConcat: ');DisplayBytes(Utf8StrConcat);
end.

Bart

var
  AnsiStrA:string;  // AnsiString(CP_ACP)
  AnsiStrB:string;  // AnsiString(CP_ACP)
  Utf8StrA: string; // AnsiString(CP_ACP)
  Utf8StrB: string; // AnsiString(CP_ACP)
  Utf8StrConcat:string; // AnsiString(CP_ACP)
begin
  AnsiStrA:=' ';
  AnsiStrA[1]:=#243; // o acute win-1252
  AnsiStrB:='A';
  // AnsiStrA is AnsiString(CP_ACP)
  // AnsiStrB is AnsiString(CP_ACP)
  Utf8StrA:=AnsiToUtf8(AnsiStrA); // 195 179
  Utf8StrB:=AnsiToUtf8(AnsiStrB); // 65
  // is Utf8StrA now utf8string? or something similar like Ansistring(UTF_8)
  // is Utf8StrB now utf8string? or something similar like Ansistring(UTF_8)
  Utf8StrConcat:=Utf8StrA+Utf8StrB;
  //  AnsiString(CP_ACP) = UTF8 + UT8
  //  automatic Conversion to ansiString(CP_ACP) ?
end;

Santiago A. wrote

// is Utf8StrA now utf8string? or something similar like Ansistring(UTF_8) // is Utf8StrB now utf8string? or something similar like Ansistring(UTF_8)

Just check the vaue of StringCodePage(Utf8StrA).

procedure DisplayBytes(S:RawByteString);
var
  i:Integer;
begin
  Write('  ');
  for i:=1 to length(s) do
    write(ord(s[i]),' ');
  writeln;
end;
var
  AnsiStrA: String;
begin
  AnsiStrA:=' ';
  AnsiStrA[1]:=#243; // o acute win-1252
  AnsiStrA := AnsiToUtf8(AnsiStrA);
  writeln('StringCodePage(AnsiStrA) now is: ',stringcodepage(ansistra));
  Write('AnsiStrA: ');DisplayBytes(AnsiStrA);
end.

Gives:
StringCodePage(AnsiStrA) now is: 65001
AnsiStrA: 195 179

Notice that your original problem was mainly due to the fact that DisplayBytes used a String parameter, which lead to automatically converting everything back to your Windows codepage.

Bart

Bart wrote

Gives: StringCodePage(AnsiStrA) now is: 65001 AnsiStrA: 195 179

I don't understand, why did AnsiStrA change its codepage type after the 3rd assignment to it?

Here is the results on my Windows 7 system.

==========================================
$ codepagestring.exe
StringCodePage(AnsiStrA) now is: 1252
StringCodePage(AnsiStrA) now is: 65001
AnsiStrA: 195 179
==========================================

All I did to the last code example was duplicate the line of code that calls stringcodepage() as shown here... So I get a before and after result.

==========================================
program codepagestring;
{$mode objfpc}{$H+}
procedure DisplayBytes(S:RawByteString);
var
  i:Integer;
begin
  Write('  ');
  for i:=1 to length(s) do
    write(ord(s[i]),' ');
  writeln;
end;
var
  AnsiStrA: String;
begin
  writeln('StringCodePage(AnsiStrA) now is: ',stringcodepage(ansistra));
  AnsiStrA:=' ';
  AnsiStrA[1]:=#243; // o acute win-1252
  AnsiStrA := AnsiToUtf8(AnsiStrA);
  writeln('StringCodePage(AnsiStrA) now is: ',stringcodepage(ansistra));
  Write('AnsiStrA: ');
  DisplayBytes(AnsiStrA);
end.
==========================================

Regards,
Graeme

Bart wrote

Just check the vaue of StringCodePage(Utf8StrA).

Not Initialized
AnsiStrA: 1252
ResultA: 1252

AnsiStrA:=' '
  AnsiStrA: 0
AnsiStrA[1]:=#243; // o acute win-1252
  AnsiStrA: 0
ResultA:=AnsiStrA
  ResultA: 0
ResultA := AnsiStrA + ' '
  ResultA: 1252
ResultA:=AnsiToUtf8(AnsiStrA);
  ResultA: 65001
ResultA:= AnsiToUtf8(AnsiStrA) + AnsiToUtf8(AnsiStrA);
  ResultA: 1252

I'm definitively completely lost

-------------------------------

program testconvertstr;
var
  AnsiStrA:string;
  ResultA:string;
begin
  writeln('Not Initialized');
  writeln('  AnsiStrA: ',stringcodepage(ansistra));
  writeln('  ResultA: ',stringcodepage(ResultA));
  Writeln;writeln('AnsiStrA:='' ''');
  AnsiStrA:=' ';
  writeln('  AnsiStrA: ',stringcodepage(ansistra));
  Writeln;writeln('AnsiStrA[1]:=#243; // o acute win-1252');
  AnsiStrA[1]:=#243; // o acute win-1252
  writeln('  AnsiStrA: ',stringcodepage(ansistra));
  Writeln;writeln('ResultA:=AnsiStrA');
  ResultA:=AnsiStrA;
  writeln('  ResultA: ',stringcodepage(ResultA));
  Writeln;writeln('ResultA := AnsiStrA + '' ''');
  ResultA:=AnsiStrA+' ';
  writeln('  ResultA: ',stringcodepage(ResultA));
  Writeln;Writeln('ResultA:=AnsiToUtf8(AnsiStrA);');
  ResultA:=AnsiToUtf8(AnsiStrA);
  writeln('  ResultA: ',stringcodepage(ResultA));
  Writeln;writeln('ResultA:= AnsiToUtf8(AnsiStrA) + AnsiToUtf8(AnsiStrA);');
  ResultA:=AnsiToUtf8(AnsiStrA)+AnsiToUtf8(AnsiStrA);
  writeln('  ResultA: ',stringcodepage(ResultA));
  Readln;
end.

Santiago A. wrote

I'm definitively completely lost

:) So am I.

Regards,
Graeme

Santiago A. wrote

program testconvertstr;

You are missing {$h+} here. When posting programs, always include all switches and/or all command line options. The program also compiles with string = shortstring (the default), but has different behaviour in that case.

The string code page of an empty string is always DefaultSystemCodePage.

Writeln;writeln('AnsiStrA:='' '''); AnsiStrA:=' '; writeln(' AnsiStrA: ',stringcodepage(ansistra));

The string code page of constant strings is described at http://wiki.freepascal.org/FPC_Unicode_support#String_constants . In this case, it is CP_ACP (= 0) because no source file code page has been set.

Writeln;writeln('AnsiStrA[1]:=#243; // o acute win-1252'); AnsiStrA[1]:=#243; // o acute win-1252 writeln(' AnsiStrA: ',stringcodepage(ansistra));

Changing an individual byte of a string has no influence on its code page.

Writeln;writeln('ResultA:=AnsiStrA'); ResultA:=AnsiStrA; writeln(' ResultA: ',stringcodepage(ResultA));

Assigning a ansistring to another ansistring with the same declared code page (both AnsiStrA and ResultA have CP_ACP as declared code page) won't change the (dynamic) string code page (see
http://wiki.freepascal.org/FPC_Unicode_support#Dynamic_code_page ).

Writeln;writeln('ResultA := AnsiStrA + '' '''); ResultA:=AnsiStrA+' '; writeln(' ResultA: ',stringcodepage(ResultA));

See http://wiki.freepascal.org/FPC_Unicode_support#String_concatenation : the result of a string concatenation will always be converted to the declared code page of the destination (and CP_ACP represents the current value of DefaultSystemCodePage, see
http://wiki.freepascal.org/FPC_Unicode_support#Code_page_identifiers ).

Writeln;Writeln('ResultA:=AnsiToUtf8(AnsiStrA);'); ResultA:=AnsiToUtf8(AnsiStrA); writeln(' ResultA: ',stringcodepage(ResultA));

AnsiToUtf8() returns a RawByteString with dynamic code page CP_UTF8 (so that the dynamic code page matches the actual string encoding). Assigning a RawByteString to any other string type never results in a string code page conversion (see
http://wiki.freepascal.org/FPC_Unicode_support#RawByteString ).

Writeln;writeln('ResultA:= AnsiToUtf8(AnsiStrA) + AnsiToUtf8(AnsiStrA);'); ResultA:=AnsiToUtf8(AnsiStrA)+AnsiToUtf8(AnsiStrA); writeln(' ResultA: ',stringcodepage(ResultA));

See again
http://wiki.freepascal.org/FPC_Unicode_support#String_concatenations (same as before).

Jonas

Jonas Maebe wrote

See again http://wiki.freepascal.org/FPC_Unicode_support#String_concatenations (same as before).

So

  ResultA := AnsiToUTF8(AnsiStrA + AnsiStrA);  // UTF-8   ResultB :=
AnsiToUTF8(AnsiStrA) + AnsiToUTF8(AnsiStrA); // Win-1252

And resultA is not equal to ResultB

It doesn't look like too intuitive.

I would say that it is closer to "hidden secret knowledge" than to the "Principle of least surprise".

It would be good if someone with access to Delphi 2009+ could test this. It is possible that the result of concatenating two RawByteStrings should again be a RawByteString and that hence no conversion should happen on assignment here either.

I would say that it is closer to "hidden secret knowledge" than to the "Principle of least surprise".

There is no hidden secret knowledge. Everything is documented and the information is linked from the release notes. The "principle of least surprise" has been applied in the sense that we didn't invent our own system that introduces small or large differences compared to how Delphi behaves in the same situation (and if there are differences, then those are bugs in our implementation).

Jonas

With hidden knowledge I don't mean it's not documented, I mean that's like small print in contracts.

Only those who know there is trick with adding are going to check manual. Complex things and pushing the language to its limits could require reading manual, adding two strings shouldn't.

MyString := expression

I really shouldn't need to know if right expression uses '+', or the result of a function, or '*', to guess what MyString type is.

In addition, changing the codepage on the fly if a bad idea. If I cant change the codepage dynamically (I don't like it, but let's live with it), let me assign it explicitly, don't change it on the fly.

SetCodePage(MyString,win1252);
MyString := AnsiToUTF8(Ansi1 + Ansi2);  // Automatically converted to
Win-1252 before assign
MyString := AnsiToUTF8(Ansi1) + AnsiToUTF8(Ansi2);   // Automatically
converted to Win-1252 before assign
SetCodePage(MyString,utf8);
MyString := AnsiToUTF8(Ansi1 + Ansi2);  //   No conversion needed
MyString := AnsiToUTF8(Ansi2) + AnsiToUTF8(Ansi2); // No conversion needed
MyString := Ansi1 + Ansi2;  //  Automatically converted to Utf8 before
assign

None changes the codepage of the String but me

I don't like automatic conversion, but let's live with it. But I think that automatic change of var type is really wrong.

This is Pascal, not bash or PHP.

Santiago A. wrote

In addition, changing the codepage on the fly if a bad idea.

+1
I think that is very much the wrong behaviour too. As I mentioned before is this mailing list. I think the FPC 3.x with these codepage aware AnsiStrings is a damn mess. As far as I can see, FPC is now worse that Delphi 2009 was. And NO, don't tell me everything should work as it did before... It doesn't!! This thread just proves that one more time.

Regards,
Graeme

How much more prominent can it be when it's in the release notes?

Only those who know there is trick with adding are going to check manual. Complex things and pushing the language to its limits could require reading manual, adding two strings shouldn't.

You don't have to read the manual. You have to read the release notes.

MyString := expression I really shouldn't need to know if right expression uses '+', or the result of a function, or '*', to guess what MyString type is.

And maybe you shouldn't, because maybe there is a bug in FPC. But I don't have Delphi 2009 and hence I cannot test.

In addition, changing the codepage on the fly if a bad idea. If I cant change the codepage dynamically (I don't like it, but let's live with it), let me assign it explicitly, don't change it on the fly.

You can do that if you only use RawByteString. I doubt you'll find it very convenient.

I don't like automatic conversion, but let's live with it. But I think that automatic change of var type is really wrong.

I don't understand what you mean with this.

Jonas

I asked on Dutch Delphi forum if someone could test with D2009 or up.

Bart

Hello,
I tested the program listed below in "Delphi 10.1 Berlin" with the results shown in the comments.

PK

program Project3;
{$APPTYPE CONSOLE}
{$R *.res}
var
   AnsiStrA: AnsiString;
   ResultA: AnsiString;
begin
   Writeln('Not Initialized');
   Writeln('  AnsiStrA: ', StringCodePage(AnsiStrA)); // 1250
   Writeln('  ResultA: ', StringCodePage(ResultA));   // 1250
   Writeln;
   Writeln('AnsiStrA := '' ''');
   AnsiStrA := ' ';
   Writeln('  AnsiStrA: ', StringCodePage(AnsiStrA)); // 1250
   Writeln;
   Writeln('AnsiStrA[1] := #243; // o acute win-1252');
   AnsiStrA[1] := #243; // o acute win-1252
   Writeln('  AnsiStrA: ', StringCodePage(AnsiStrA)); // 1250
   Writeln;
   Writeln('ResultA := AnsiStrA');
   ResultA := AnsiStrA;
   Writeln('  ResultA: ', StringCodePage(ResultA));   // 1250
   Writeln;
   Writeln('ResultA := AnsiStrA + '' ''');
   ResultA := AnsiStrA + ' ';
   Writeln('  ResultA: ', StringCodePage(ResultA));   // 1250
   Writeln;
   Writeln('ResultA := AnsiToUtf8(AnsiStrA);');
   ResultA := AnsiToUtf8(AnsiStrA);
   Writeln('  ResultA: ', StringCodePage(ResultA));   // 65001
   Writeln;
   Writeln('ResultA := AnsiToUtf8(AnsiStrA) + AnsiToUtf8(AnsiStrA);');
   ResultA := AnsiToUtf8(AnsiStrA) + AnsiToUtf8(AnsiStrA);
   Writeln('  ResultA: ', StringCodePage(ResultA));   // 65001
   Writeln;
   Writeln('ResultA := AnsiToUtf8(AnsiStrA) + AnsiStrA;');
   ResultA := AnsiToUtf8(AnsiStrA) + AnsiStrA;
   Writeln('  ResultA: ', StringCodePage(ResultA));   // 65001
   Writeln;
   Writeln('ResultA := AnsiStrA + AnsiToUtf8(AnsiStrA);');
   ResultA := AnsiStrA + AnsiToUtf8(AnsiStrA);
   Writeln('  ResultA: ', StringCodePage(ResultA));   // 1250
   Readln;
end.

(*
Not Initialized
AnsiStrA: 1250
ResultA: 1250

AnsiStrA := ' '
   AnsiStrA: 1250
AnsiStrA[1] := #243; // o acute win-1252
   AnsiStrA: 1250
ResultA := AnsiStrA
   ResultA: 1250
ResultA := AnsiStrA + ' '
   ResultA: 1250
ResultA := AnsiToUtf8(AnsiStrA);
   ResultA: 65001
ResultA := AnsiToUtf8(AnsiStrA) + AnsiToUtf8(AnsiStrA);
   ResultA: 65001
ResultA := AnsiToUtf8(AnsiStrA) + AnsiStrA;
   ResultA: 65001

ResultA := AnsiStrA + AnsiToUtf8(AnsiStrA);
ResultA: 1250
*)

------ Původní zpráva ------
Od: "Bart"
Komu: "FPC-Pascal users discussions"
Odesláno: 23.07.2016 0:29:32
Předmět: Re: [fpc-pascal] Weird string behavior

Here is a result of Delphi 10.1:

program DTestConcatenate;
{$APPTYPE CONSOLE}
{$R *.res}
uses
  System.SysUtils;
var
  s1,s2,s3: AnsiString;
  r1,r2,r3: RawByteString;
begin
  s1:='A';   // 1252
  s2:='Aä';  // 1252
  writeln('s1="',s1,'" cp=',StringCodePage(s1));
  writeln('s2="',s1,'" cp=',StringCodePage(s2));
  r1:=AnsiToUTF8(s1); // 65001
  r2:=AnsiToUTF8(s2); // 65001
  writeln('r1="',r1,'" cp=',StringCodePage(r1));
  writeln('r2="',r2,'" cp=',StringCodePage(r2));
  r3:=r1+r2; // 65001
  writeln('r3="',r3,'" cp=',StringCodePage(r3));
  s3:=r1+r2; // 65001
  writeln('s3="',s3,'" cp=',StringCodePage(s3));
end.

Mattias

Mattias Gaertner wrote

Here is a result of Delphi 10.1:

Thank you (also Petr). Maybe one more, to know what happens if you mix rawbytestring and ansistring in the concatenation:

program DTestConcatenate;
{$APPTYPE CONSOLE}
{$R *.res}
uses
   System.SysUtils;
var
   s1,s2,s3: AnsiString;
   r1,r2,r3: RawByteString;
begin
   s1:='A';   // 1252
   s2:='Aä';  // 1252
   writeln('s1="',s1,'" cp=',StringCodePage(s1));
   writeln('s2="',s1,'" cp=',StringCodePage(s2));
   r1:=AnsiToUTF8(s1); // 65001
   r2:=AnsiToUTF8(s2); // 65001
   writeln('r1="',r1,'" cp=',StringCodePage(r1));
   writeln('r2="',r2,'" cp=',StringCodePage(r2));
   r3:=s1+r2; // ??
   writeln('r3="',r3,'" cp=',StringCodePage(r3));
   r3:=r1+s2; // ??
   writeln('r3="',r3,'" cp=',StringCodePage(r3));
   s3:=s1+r2; // ??
   writeln('r3="',r3,'" cp=',StringCodePage(r3));
   s3:=r1+s2; // ??
   writeln('r3="',r3,'" cp=',StringCodePage(r3));
end.

Jonas

Some copy paste errors in the last four lines, that should be

   s3:=s1+r2; // ??
   writeln('s3="',s3,'" cp=',StringCodePage(s3));
   s3:=r1+s2; // ??
   writeln('s3="',s3,'" cp=',StringCodePage(s3));

And maybe also (to check whether there is a difference depending on the actual code page):

   setstringcodepage(rawbytestring(s1),65001,true);
   r3:=s1+r2; // ??
   writeln('r3="',r3,'" cp=',StringCodePage(r3));
   r3:=r1+s1; // ??
   writeln('r3="',r3,'" cp=',StringCodePage(r3));
   s3:=s1+r2; // ??
   writeln('s3="',s3,'" cp=',StringCodePage(s3));
   s3:=r1+s2; // ??
   writeln('s3="',s3,'" cp=',StringCodePage(s3));

Jonas

On 07/23/2016 06:13 AM, Jonas Maebe wrote:
[...]

writeln('s2="',s2,'" cp=',StringCodePage(s2));

you're not the only one to have missed that...

gotta wonder how fubar the test results are now ;)

The only thing that matters for this test is the stringcodepage value, which is the correct one.

Jonas

Hello,
here are results:

...
begin
   Writeln('--------------');
   Writeln;
   s1 := 'A';   // 1250
   s2 := 'Aä';  // 1250
   Writeln('s1 = "', s1, '" cp = ', StringCodePage(s1));
   Writeln('s2 = "', s2, '" cp = ', StringCodePage(s2));
   r1 := AnsiToUTF8(s1); // 65001
   r2 := AnsiToUTF8(s2); // 65001
   Writeln('r1 = "', r1, '" cp = ', StringCodePage(r1));
   Writeln('r2 = "', r2, '" cp = ', StringCodePage(r2));
   r3 := s1 + r2; // 1250
   Writeln('r3 = "', r3, '" cp = ', StringCodePage(r3));
   r3 := r1 + s2; // 65001
   Writeln('r3 = "', r3, '" cp = ', StringCodePage(r3));
   s3 := s1 + r2; // 1250
   Writeln('s3 = "', s3, '" cp = ', StringCodePage(s3));
   s3 := r1 + s2; // 65001
   Writeln('s3 = "', s3, '" cp = ', StringCodePage(s3));
   SetCodePage(RawByteString(s1), 65001, True);
   r3 := s1 + r2; // 65001
   Writeln('r3="', r3, '" cp=', StringCodePage(r3));
   r3 := r1 + s2; // 65001
   Writeln('r3="', r3, '" cp=', StringCodePage(r3));
   s3 := s1 + r2; // 65001
   Writeln('s3="', s3, '" cp=', StringCodePage(s3));
   s3 := r1 + s2; // 65001
   Writeln('s3="', s3, '" cp=', StringCodePage(s3));
   Readln;
end.
(*
--------------

s1 = "A" cp = 1250
s2 = "Aä" cp = 1250
r1 = "A" cp = 65001
r2 = "Aä" cp = 65001
r3 = "AAä" cp = 1250
r3 = "AA?" cp = 65001
s3 = "AAä" cp = 1250
s3 = "AA?" cp = 65001
r3="AAä" cp=65001
r3="AA?" cp=65001
s3="AAä" cp=65001
s3="AA?" cp=65001

*)

------ Původní zpráva ------
Od: "Jonas Maebe"
Komu: "FPC-Pascal users discussions"
Odesláno: 23.07.2016 13:03:33
Předmět: Re: [fpc-pascal] Weird string behavior

Petr Kohut wrote

Hello, here are results:

Thanks a lot. Could you test one more? I think I will have all information I need then.

Jonas

{$APPTYPE CONSOLE}
type
   tcp866 = type ansistring(866);
var
   s1, s2, s3: tcp866;
begin
   s1:='abc';
   setcodepage(rawbytestring(s1),65001,false);
   s2:='def';
   setcodepage(rawbytestring(s2),437,false);
   s3:=s1+s2;
   Writeln('DefaultSystemCodePage = ',DefaultSystemCodePage);
   Writeln('s3 = "', s3, '" cp = ', StringCodePage(s3));
   Readln;
end.

DefaultSystemCodePage = 1252
s3 = "abcdef" cp = 65001

Mattias

Mattias Gaertner wrote

DefaultSystemCodePage = 1252 s3 = "abcdef" cp = 65001

Thanks. So the rule for concatenation appears to be:
* the dynamic code page of the result of a string concatenation is that of the left operand (except if it's an empty string, then it's that of the right operand)
* the declared code page of the final concatenation result is that of the left operand

You then process the assignment as if you are assigning a string with the above declared/dynamic code page to whatever you are assigning the result of the concatenation to (which means no code page conversion in case the declared code pages match, like in the above case).

That's indeed not what FPC does currently. It's mainly complicated by the fact that FPC contains optimised helpers to avoid the final assignment if possible and directly concatenate into the destination when possible (those helpers right now only get passed the declared code page of the final destination, and not that of the concatenation, and hence virtually always convert the result to the dynamic code page of the destination at this time -- there are some exceptions for rawbytestring)

Jonas

Here are some more hints:

{$APPTYPE CONSOLE}
type
   tcp866 = type ansistring(866);
var
   s1, s2: tcp866;
   u1: UTF8String;
   r1: RawByteString;
begin
   s1:='abc';
   setcodepage(rawbytestring(s1),65001,false);
   Writeln('s1 = "', s1, '" cp = ', StringCodePage(s1));
   u1:='nop';
   Writeln('u1 = "', u1, '" cp = ', StringCodePage(u1));
   s2:=s1+u1;
   Writeln('s2 = "', s2, '" cp = ', StringCodePage(s2));
   s2:=u1+s1;
   Writeln('s2 = "', s2, '" cp = ', StringCodePage(s2));
   r1:=s1+u1;
   Writeln('r1 = "', r1, '" cp = ', StringCodePage(r1));
   r1:=u1+s1;
   Writeln('r1 = "', r1, '" cp = ', StringCodePage(r1));
   readln;
end.

s1 = "abc" cp = 65001
u1 = "nop" cp = 65001
s2 = "abcnop" cp = 866
s2 = "nopabc" cp = 866
r1 = "abcnop" cp = 1252
r1 = "nopabc" cp = 1252

Mattias

Mattias Gaertner wrote on Tue, 26 Jul 2016:

Could you try the same program with u1 as plain ansistring instead of utf8string? (with an additional
"setcodepage(rawbytestring(u1),65001,false);" after assigning u1)

Thanks,

Jonas

[...] Could you try the same program with u1 as plain ansistring instead of utf8string? (with an additional "setcodepage(rawbytestring(u1),65001,false);" after assigning u1)

Sure:

{$APPTYPE CONSOLE}
type
   tcp866 = type ansistring(866);
var
   s1, s2: tcp866;
   u1: UTF8String;
   r1: RawByteString;
   a1, a2: AnsiString;
begin
   s1:='cp866';
   setcodepage(rawbytestring(s1),65001,false);
   Writeln('s1 = "', s1, '" cp = ', StringCodePage(s1));
   a1:='acp';
   setcodepage(rawbytestring(a1),65001,false);
   Writeln('a1 = "', a1, '" cp = ', StringCodePage(a1));
   u1:='utf8';
   Writeln('u1 = "', u1, '" cp = ', StringCodePage(u1));
   s2:=s1+u1;
   Writeln('s2:=s1+u1 = "', s2, '" cp = ', StringCodePage(s2));
   s2:=u1+s1;
   Writeln('s2:=u1+s1 = "', s2, '" cp = ', StringCodePage(s2));
   r1:=s1+u1;
   Writeln('r1:=s1+u1 = "', r1, '" cp = ', StringCodePage(r1));
   r1:=u1+s1;
   Writeln('r1:=u1+s1 = "', r1, '" cp = ', StringCodePage(r1));
   a2:=s1+u1;
   Writeln('a2:=s1+u1 = "', a2, '" cp = ', StringCodePage(a2));
   a2:=u1+s1;
   Writeln('a2:=u1+s1 = "', a2, '" cp = ', StringCodePage(a2));
   s2:=s1+a1;
   Writeln('s2:=s1+a1 = "', s2, '" cp = ', StringCodePage(s2));
   s2:=a1+s1;
   Writeln('s2:=a1+s1 = "', s2, '" cp = ', StringCodePage(s2));
   r1:=s1+a1;
   Writeln('r1:=s1+a1 = "', r1, '" cp = ', StringCodePage(r1));
   r1:=a1+s1;
   Writeln('r1:=a1+s1 = "', r1, '" cp = ', StringCodePage(r1));
   a2:=s1+a1;
   Writeln('a2:=s1+a1 = "', a2, '" cp = ', StringCodePage(a2));
   a2:=a1+s1;
   Writeln('a2:=a1+s1 = "', a2, '" cp = ', StringCodePage(a2));
   readln;
end.
s1 = "cp866" cp = 65001
a1 = "acp" cp = 65001
u1 = "utf8" cp = 65001
s2:=s1+u1 = "cp866utf8" cp = 866
s2:=u1+s1 = "utf8cp866" cp = 866
r1:=s1+u1 = "cp866utf8" cp = 1252
r1:=u1+s1 = "utf8cp866" cp = 1252
a2:=s1+u1 = "cp866utf8" cp = 1252
a2:=u1+s1 = "utf8cp866" cp = 1252
s2:=s1+a1 = "cp866acp" cp = 866
s2:=a1+s1 = "acpcp866" cp = 866
r1:=s1+a1 = "cp866acp" cp = 1252
r1:=a1+s1 = "acpcp866" cp = 1252
a2:=s1+a1 = "cp866acp" cp = 1252
a2:=a1+s1 = "acpcp866" cp = 1252

It seems the Delphi rules for non rawbytestrings are:
- Concatenate two same declared strings: append bytes, copy dyn. cp from left operand. Declared cp of result is left operand.
- Assign same declared: no conversion, only refcount.
- Concatenate two different declared strings: convert both to UnicodeString and append. Maybe there is an optimization for same dyn cp.
- Assign different declared strings: convert to LHS.

Mattias

Mattias Gaertner wrote on Tue, 26 Jul 2016:

It seems the Delphi rules for non rawbytestrings are: - Concatenate two same declared strings: append bytes, copy dyn. cp from left operand. Declared cp of result is left operand.

Are you sure it's "append bytes" here and not "append bytes if same dyn cp, otherwise convert to unicodestring, concatenate, and convert back to dyn cp of left operand"?

Jonas

Now I'm sure. See attachments.

Mattias

Mattias Gaertner wrote

a3:=a1+a2 => cp = 1252 a3:=a2+a1 => cp = 65001

Is that the expected behavior?

IMHO the result should be the same. And the only way is to make it depend on a3, no matter what is in the left side. That's the way things are done in Pascal

This is not correct. In pascal the right-hand side of an assignment has a well-defined type. The compiler checks whether the type on the right is assignment-compatible to the left side.

Michael.

Santiago A. wrote on Tue, 26 Jul 2016:

String concatenations are not commutative operations, even if no code pages are involved (even in Pascal). I think it is logical that the code page of the left operand is kept, because it is the "base string" to which you add other data.

The fact that the concatenated data is not converted in this scenario makes some sense in the context of the "same declared code page -> no conversion" rule that is also used for assignments, but it's something I'm less happy about. Maintaining multiple helpers, compiler behaviours and documentation is even less enticing though, so I will implement the Delphi behaviour. It will also slightly speed up string concatenations when all strings have the same declared code page, because no checks need to be performed at run time regarding whether or not data needs to be converted.

Jonas

Sorry I meant no matter what is in the right side. Other way my statment has no sense ;-)

Well, I'm also not sure what left/right sides you are referring to:

left/right of := or left/right of +

But as long as Jonas is on top of things, all will be well =-)

Michael.

Michael Van Canneyt wrote

This is not correct. In pascal the right-hand side of an assignment has a well-defined type. The compiler checks whether the type on the right is assignment-compatible to the left side.

Hmm.

if you do

x := y + z;

with x a real and y and z integers, the type of x will not change to be an integer, but the value will be converted.

Now I understand that with strings the encoding is a kind of "sub-type" and hence (usually) static and not convertible to allow for the compiler do decide if a conversion is necessary.

This has been discussed a long time ago and the argument was that _fully_ dynamically typed strings are to costly regarding CPU demand.

I did not get to know that those design decision has been changed for the normal usage case (while there seems to be ways to sue certain kinds of strings in a fully dynamical way) .

Changing the encoding of the left side operand of ":=" would only be logical if the encoding is never an attribute to the string's type but always a dynamical attribute to the string's content.

-Michael.

And what are the rules for changing left side operand? It looks that they are a little complicated.

Freepascal needed codepages, so string with codepage was needed. Should it need dynamic codepage for backward compatibility? INHO, no. String should had been an alias of rawbytestring, and "codepage aware strings" should be another new type, but codepage should be static.

Legacy programs could compile and run perfectly, and you could start using codepage aware strings type.

Automatic conversions? Well, I'm not for it, but any way, left side shouldn't change its codepage.

Nevertheless, that's my two cents. I looks that there is some pressure to be Delphi XX compatible, I left Delphi long long time ago (Delphi 5), so these compatibility issues are not in my radar.

Santiago A. wrote

And what are the rules for changing left side operand?

Extremely hard to define when trying to follow any decent logic, unless the decision is either "never" (i.e. a strictly static typing) or "always" (a strictly dynamic typing).

A way out could be to set all "normal" string types to behave as strictly static typing (like any other types in Pascal) and a single dedicated (but intended to be used always where appropriate, e. g. in "TStrings") String type to behave as strictly dynamic typing).

Analyze and suggestions see ->
http://wiki.lazarus.freepascal.org/not_Delphi_compatible_enhancement_for_Unicode_Support

-Michael