Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / fpc-pascal / 2020-09

String literals and code page of .pas source file

38 messages · 10 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

LacaK

Hi *,

I would like to have source file in Windows-1250 encoding, where are stored literal strings like 'áéíóčž' in Windows-1250 encoding (I share this one file between FPC/Lazarus and Delphi 7). Windows-1250 is also ANSI code page of my Windows OS. In source file I have:

{$IFDEF FPC}
   {$CODEPAGE cp1250}
{$ENDIF}

and in Lazarus I have set: File settings / Encoding / CP1250 (at this point in .pas file are all characters correctly encoded in Windows-1250 code page)

When I compile application and run it, string literals (characters with accents) are not preserved.
Is there way how to get it working?

Only work-around which worked for me (in both Delphi and Lazarus) was use:

const
   STRINGLITERAL1: WideString = 'áéíóčž';

Thanks

-Laco.

Reply by email

Sven Barth
to LacaK
LacaK wrote 10 lines
Hi *, I would like to have source file in Windows-1250 encoding, where are stored literal strings like 'áéíóčž' in Windows-1250 encoding (I share this one file between FPC/Lazarus and Delphi 7). Windows-1250 is also ANSI code page of my Windows OS. In source file I have: {$IFDEF FPC}   {$CODEPAGE cp1250} {$ENDIF}

A pure string constant will still have CP_ACP as code page (which in Lazarus will by default mean UTF-8) also when you declare it as a typed String constant. The $CodePage directive will only influence the binary encoding of the string. What you need to do is the following:

type
   String1250 = type String(1250);
const
   MyString: String1250 = 'áéíóčž';

This way the string itself will have code page 1250 and if you assign to a normal String or a UnicodeString the compiler will insert the appropriate conversions.

Regards,
Sven

Reply by email

LacaK
to Sven Barth
Show quoted text 14 lines
> Hi *, > > I would like to have source file in Windows-1250 encoding, where are > stored literal strings like 'áéíóčž' in Windows-1250 encoding (I > share this one file between FPC/Lazarus and Delphi 7). Windows-1250 > is also ANSI code page of my Windows OS. In source file I have: > > {$IFDEF FPC} >   {$CODEPAGE cp1250} > {$ENDIF} A pure string constant will still have CP_ACP as code page (which in Lazarus will by default mean UTF-8) also when you declare it as a typed String constant. The $CodePage directive will only influence the binary encoding of the string.

*and probably this is something non intuitive for me. Also after reading https://wiki.freepascal.org/FPC_Unicode_support#String_constants it is hard to understand for me. After reading "The compiler has to know the code page according to which it should interpret string constants, as it may have to convert them at compile time." my expectation is, that when I explicitly define encoding of source file by using **{$CODEPAGE cp1250} then compiler will be aware of string encoding in that file and will handle it appropriate. Now it works in another way which is not clear to me ... can it be better explained in wiki (any example)?*

*
*

Show quoted text 11 lines
What you need to do is the following: type   String1250 = type String(1250); const   MyString: String1250 = 'áéíóčž'; This way the string itself will have code page 1250 and if you assign to a normal String or a UnicodeString the compiler will insert the appropriate conversions.

*yes, this I can understand, but it will not work in Delphi 7 ...*

*TIA*

*L.*

Reply by email

Jonas Maebe
to LacaK: I would like to have source file in Windows-1250 encoding, where are stored literal…
LacaK wrote 16 lines
I would like to have source file in Windows-1250 encoding, where are stored literal strings like 'áéíóčž' in Windows-1250 encoding (I share this one file between FPC/Lazarus and Delphi 7). Windows-1250 is also ANSI code page of my Windows OS. In source file I have: {$IFDEF FPC}   {$CODEPAGE cp1250} {$ENDIF} and in Lazarus I have set: File settings / Encoding / CP1250 (at this point in .pas file are all characters correctly encoded in Windows-1250 code page) When I compile application and run it, string literals (characters with accents) are not preserved. Is there way how to get it working?

The attached program uses the same environment as Lazarus (DefaultSystemCodePage forced to UTF-8) and prints the string correctly for me (under Mac OS X).

Please always provide a compilable program when reporting issues.

Jonas

Reply by email

LacaK
to Jonas Maebe

Hello Jonas,

attached simple Lazarus compilable project (one Form).

...
type
   String1250 = type AnsiString(1250);
const
   c2: AnsiString = 'áéíóčž';
   c3: WideString = 'áéíóčž';
   c4: String1250 = 'áéíóčž';

{ TForm1 }

procedure TForm1.FormShow(Sender: TObject);
begin
   label1.Caption:='áéíóčž'; // FAIL
   label2.Caption:=c2;       // FAIL
   label3.Caption:=c3;       // OK
   label4.Caption:=c4;       // FAIL
end;

TIA

-Laco.

Show quoted text 24 lines
> I would like to have source file in Windows-1250 encoding, where are > stored literal strings like 'áéíóčž' in Windows-1250 encoding (I share > this one file between FPC/Lazarus and Delphi 7). Windows-1250 is also > ANSI code page of my Windows OS. In source file I have: > > {$IFDEF FPC} >   {$CODEPAGE cp1250} > {$ENDIF} > > and in Lazarus I have set: File settings / Encoding / CP1250 > (at this point in .pas file are all characters correctly encoded in > Windows-1250 code page) > > When I compile application and run it, string literals (characters with > accents) are not preserved. > Is there way how to get it working? The attached program uses the same environment as Lazarus (DefaultSystemCodePage forced to UTF-8) and prints the string correctly for me (under Mac OS X). Please always provide a compilable program when reporting issues. Jonas

Reply by email

Jonas Maebe
to LacaK
LacaK wrote 20 lines
attached simple Lazarus compilable project (one Form). ... type   String1250 = type AnsiString(1250); const   c2: AnsiString = 'áéíóčž';   c3: WideString = 'áéíóčž';   c4: String1250 = 'áéíóčž'; { TForm1 } procedure TForm1.FormShow(Sender: TObject); begin   label1.Caption:='áéíóčž'; // FAIL   label2.Caption:=c2;       // FAIL   label3.Caption:=c3;       // OK   label4.Caption:=c4;       // FAIL end;

The attached pure FPC-variant of your project works fine with FPC 3.2.0 for me (except for the very first writeln, because that one gets interpreted as "writeln(shortstring)"), so it may be Lazarus-specific. Best ask it on the Lazarus list or post it on their bug tracker instead.

Also make sure you did *not* build your Lazarus with -dDisableUTF8RTL

Jonas

Reply by email

Mattias Gaertner
to Jonas Maebe
Show quoted text 19 lines
On 07/09/2020 08:51, LacaK via fpc-pascal wrote: > attached simple Lazarus compilable project (one Form). > > ... > type >   String1250 = type AnsiString(1250); > > const >   c2: AnsiString = 'áéíóčž'; >   c3: WideString = 'áéíóčž'; >   c4: String1250 = 'áéíóčž'; > > { TForm1 } > > procedure TForm1.FormShow(Sender: TObject); > begin >   label1.Caption:='áéíóčž'; // FAIL >   label2.Caption:=c2;       // FAIL
label2.Caption:='a'+c2; // OK
>   label3.Caption:=c3;       // OK >   label4.Caption:=c4;       // FAIL > end;

Reason is that LCL TLabel.Caption expects UTF-8.
You can use

label2.Caption:=WinCPToUTF8(c2); // from unit LazUTF8

Maybe SetCaption could check if passed string is not CP_ACP and do a conversion.

Mattias

Reply by email

Sven Barth
to Mattias Gaertner

Mattias Gaertner via fpc-pascal schrieb
am Di., 8. Sep. 2020, 22:32:

Show quoted text 37 lines
On Tue, 8 Sep 2020 20:27:20 +0200 Jonas Maebe via fpc-pascal wrote: > On 07/09/2020 08:51, LacaK via fpc-pascal wrote: > > > attached simple Lazarus compilable project (one Form). > > > > ... > > type > > String1250 = type AnsiString(1250); > > > > const > > c2: AnsiString = 'áéíóčž'; > > c3: WideString = 'áéíóčž'; > > c4: String1250 = 'áéíóčž'; > > > > { TForm1 } > > > > procedure TForm1.FormShow(Sender: TObject); > > begin > > label1.Caption:='áéíóčž'; // FAIL > > label2.Caption:=c2; // FAIL label2.Caption:='a'+c2; // OK > > label3.Caption:=c3; // OK > > label4.Caption:=c4; // FAIL > > end; Reason is that LCL TLabel.Caption expects UTF-8. You can use label2.Caption:=WinCPToUTF8(c2); // from unit LazUTF8 Maybe SetCaption could check if passed string is not CP_ACP and do a conversion.

Why does c4 fail however? Shouldn't the compiler insert a type conversion from codepage 1250 to CP_ACP which is UTF-8 in Lazarus?

Regards,
Sven

Reply by email

LacaK
to Jonas Maebe: The attached pure FPC-variant of your project works fine with FPC 3.2.0 for me (except…
Show quoted text 23 lines
> attached simple Lazarus compilable project (one Form). > > ... > type >   String1250 = type AnsiString(1250); > > const >   c2: AnsiString = 'áéíóčž'; >   c3: WideString = 'áéíóčž'; >   c4: String1250 = 'áéíóčž'; > > { TForm1 } > > procedure TForm1.FormShow(Sender: TObject); > begin >   label1.Caption:='áéíóčž'; // FAIL >   label2.Caption:=c2;       // FAIL >   label3.Caption:=c3;       // OK >   label4.Caption:=c4;       // FAIL > end; The attached pure FPC-variant of your project works fine with FPC 3.2.0 for me (except for the very first writeln, because that one gets interpreted as "writeln(shortstring)"),

*yes for me on Windows it works also (under Lazarus 2.0.8 with FPC 3.0.4) *

*
*

so it may be Lazarus-specific. Best ask it on the Lazarus list or post it on their bug tracker instead.

*I add CC also to Lazarusdev group
(attached simple project)
*

*
*

Also make sure you did *not* build your Lazarus with -dDisableUTF8RTL

*I do not build Lazarus with -dDisableUTF8RTL (I simply use default installation)*

*As Sven wrote there is at least "c4" case, where declared code page (AnsiString(1250)) is different from AnsiString (CP_ACP) codepage and also from dynamic code page (DefaultSystemCodepage which is under Lazarus (UTF8=65001))*

*I am still confused how this CodePage aware stuff is expected to work and when implicit conversios are included (as far as I understand here ONLY declared/static codepage plays role) and when dynamic codepage is taken into account and when not?*

*L.*

Reply by email

Bart
to LacaK
AM LacaK wrote
I am still confused how this CodePage aware stuff is expected to work and when implicit conversios are included (as far as I understand here ONLY declared/static codepage plays role) and when dynamic codepage is taken into account and when not?

You are not the only one.
Given that I am on Lazarus, so all my source files are UTF8 encoded I got confused by this:
unit1
{$codepage utf8}
interface
const

  msCopy = 'Kopiëren...';

....

Than in main program:
Label1.Caption := msgCopy;

This does not show the text correctly.
I have to include {$codepage utf8} in the source code file that contains the "Label1.Caption := msgCopy;" otherwise it does not display correctly.
So basically I have that in all source code that does display text (in a GUI). And if I do so, I can omit the codepage directive in unit1. This is a bit counter intuitive to me.

Bart

Reply by email

Juha Manninen
to Bart

On Wed, Sep 9, 2020 at 11:40 AM Bart via fpc-pascal <

wrote: This is a bit counter intuitive to me.

It is explained here:
https://wiki.freepascal.org/Unicode_Support_in_Lazarus

For LacaK I recommend using UTF-8 encoding as much as possible. Then everything "*just works*".
You could have a unit or .inc file with WinCP constants for Delphi7, and then some IFDEFs. They should be explicitly converted to UTF-8 for LCL. UnicodeString constants can also be a good solution. It triggers an automatic conversion when assigned to a String variable. The conversion is usually fast enough.

Juha

Reply by email

LacaK
to Juha Manninen
Show quoted text 6 lines
> This is a bit counter intuitive to me. It is explained here: https://wiki.freepascal.org/Unicode_Support_in_Lazarus

IMO "c4" case (c4: String1250 = 'áéíóčž';) is not fully explained. Probably what here happens is, that declared code page of c4 = source file code page and in that case compiler does not insert conversion code. StringCodePage(c4)=1250 (also StringCodePage(c2)=1250) so dynamic code page is right, but when assigning to TLabel.Caption current dynamic code page is not taken into account

For LacaK I recommend using UTF-8 encoding as much as possible. Then everything "/just works/".

yes I use this, but I have "one" specific case where I want share one file between Delphi7 and Lazarus

Show quoted text 7 lines
You could have a unit or .inc file with WinCP constants for Delphi7, and then some IFDEFs. They should be explicitly converted to UTF-8 for LCL. UnicodeString constants can also be a good solution. It triggers an automatic conversion when assigned to a String variable. The conversion is usually fast enough.

Yes this is what I use (WideString constants) and it is only solution which works for me in Delphi7/Lazarus

Thanks

L.

Reply by email

Tomas Hajny
to LacaK
LacaK wrote 12 lines
> > > This is a bit counter intuitive to me. > > It is explained here: > https://wiki.freepascal.org/Unicode_Support_in_Lazarus > IMO "c4" case (c4: String1250 = 'áéíóčž';) is not fully explained. Probably what here happens is, that declared code page of c4 = source file code page and in that case compiler does not insert conversion code. StringCodePage(c4)=1250 (also StringCodePage(c2)=1250) so dynamic code page is right, but when assigning to TLabel.Caption current dynamic code page is not taken into account

A simple question - is TLabel.Caption declared as Utf8String (I don't use Lazarus)? _Then_ it should work without any issues and/or special attention from users.

Tomas

Reply by email

Mattias Gaertner
to Tomas Hajny: A simple question - is TLabel.Caption declared as Utf8String (I don't use Lazarus)?…
Show quoted text 5 lines
On 2020-09-09 13:51, LacaK via fpc-pascal wrote: [...] A simple question - is TLabel.Caption declared as Utf8String (I don't use Lazarus)? _Then_ it should work without any issues and/or special attention from users.

Declaring as UTF8String would insert implicit conversion code for all String (CP_ACP), so basically everywhere.

Mattias

Reply by email

Tomas Hajny
to Mattias Gaertner: On Wed, 09 Sep 2020 17:25:09 +0200 Declaring as UTF8String would insert implicit…
Mattias Gaertner wrote 11 lines
On Wed, 09 Sep 2020 17:25:09 +0200 Tomas Hajny via fpc-pascal wrote: > On 2020-09-09 13:51, LacaK via fpc-pascal wrote: > [...] > A simple question - is TLabel.Caption declared as Utf8String (I don't > use Lazarus)? _Then_ it should work without any issues and/or special > attention from users. Declaring as UTF8String would insert implicit conversion code for all String (CP_ACP), so basically everywhere.

Everywhere except when used with other Utf8string variables and constants. The implicit conversion is explicitly expected by the user in this case if I understand it correctly (source in CP1250), right?

Tomas

Reply by email

Mattias Gaertner
to Tomas Hajny
Show quoted text 15 lines
On 2020-09-09 17:37, Mattias Gaertner via fpc-pascal wrote: > On Wed, 09 Sep 2020 17:25:09 +0200 > Tomas Hajny via fpc-pascal wrote: > >> On 2020-09-09 13:51, LacaK via fpc-pascal wrote: >> [...] >> A simple question - is TLabel.Caption declared as Utf8String (I >> don't use Lazarus)? _Then_ it should work without any issues >> and/or special attention from users. > > Declaring as UTF8String would insert implicit conversion code for > all String (CP_ACP), so basically everywhere. Everywhere except when used with other Utf8string variables and constants.

True.

The implicit conversion is explicitly expected by the user in this case if I understand it correctly (source in CP1250), right?

If you mean the "non implicit conversion" requires explicit conversion by the user, then yes.

Mattias

Reply by email

Jonas Maebe
to Mattias Gaertner: On Wed, 09 Sep 2020 17:25:09 +0200 Declaring as UTF8String would insert implicit…
Mattias Gaertner wrote 11 lines
On Wed, 09 Sep 2020 17:25:09 +0200 Tomas Hajny via fpc-pascal wrote: > On 2020-09-09 13:51, LacaK via fpc-pascal wrote: > [...] > A simple question - is TLabel.Caption declared as Utf8String (I don't > use Lazarus)? _Then_ it should work without any issues and/or special > attention from users. Declaring as UTF8String would insert implicit conversion code for all String (CP_ACP), so basically everywhere.

If the dynamic code page of the assigned string was already CP_UTF8 (or CP_ACP with DefaultSystemCodePage = CP_UTF8), no actual conversion will happen though. Is the overhead of that extra bit of code really noticeable? And is it worth the confusion and inconvenience?

Jonas

Reply by email

Tomas Hajny
to Mattias Gaertner: On Wed, 09 Sep 2020 17:44:11 +0200 True. If you mean the "non implicit conversion"…
Mattias Gaertner wrote
On Wed, 09 Sep 2020 17:44:11 +0200 Tomas Hajny via fpc-pascal wrote:

.
.

Show quoted text 5 lines
> The implicit conversion is explicitly expected by the user > in this case if I understand it correctly (source in CP1250), right? If you mean the "non implicit conversion" requires explicit conversion by the user, then yes.

No, I mean that the original poster explicitly mentioned that he expected the implicit conversion to be performed as necessary when he specified that his source code was in CP1250.

Tomas

Reply by email

Santiago A.
to Bart: On Wed, Sep 9, 2020 at 8:37 AM LacaK via fpc-pascal You are not the only one. Given that…
Bart wrote 5 lines
On Wed, Sep 9, 2020 at 8:37 AM LacaK via fpc-pascal wrote: > I am still confused how this CodePage aware stuff is expected to work and when implicit conversios are included (as far as I understand here ONLY declared/static codepage plays role) and when dynamic codepage is taken into account and when not? You are not the only one.

Yeah. In fact, we could set up several self-help groups of codepage victims ;-)

https://wiki.freepascal.org/Unicode_Support_in_Lazarus#Assign_string_literals_to_different_string_types is scaring.

All the doubts, questions, and discussions prove that current system is counter-intuitive and confusing.
I'm sorry, I don't want to bash, but code pages system must be revised almost from scratch, even breaking compatibility. It is a heavy lodge that FPC is carrying. The longer it takes to get rid of it, the worse.

Saludos Santiago A.

Reply by email

Jonas Maebe
to Santiago A.
Santiago A. wrote

The first table is useless. If you don't tell the compiler what the code page is of the source file, it's normal that you will get issues all over the place.

The second table looks more worrying. Where are the bug reports for the cases where it fails? I would have to look at them more in detail to find the causes, but at first sight it indicates several implementation bugs.

All the doubts, questions, and discussions prove that current system is counter-intuitive and confusing.

The issue in this thread is caused by a bug in the LCL: it blindly assumes that the dynamic code page of the caption string is always utf-8. That is simply wrong (unless you put the burden on the user to always assign an utf-8-encoded string to it, but _that_ is counter-intuitive and confusing).

Jonas

Reply by email

Sven Barth
to Jonas Maebe

Jonas Maebe via fpc-pascal schrieb am
Sa., 12. Sep. 2020, 17:47:

Show quoted text 9 lines
> All the doubts, questions, and discussions prove that current system is > counter-intuitive and confusing. The issue in this thread is caused by a bug in the LCL: it blindly assumes that the dynamic code page of the caption string is always utf-8. That is simply wrong (unless you put the burden on the user to always assign an utf-8-encoded string to it, but _that_ is counter-intuitive and confusing).

But shouldn't the compiler insert a conversion if the string is declared as CP_1250 and the destination is CP_ACP?

Regards,
Sven

Reply by email

Jonas Maebe
to Sven Barth
Sven Barth wrote 17 lines
Jonas Maebe via fpc-pascal < > schrieb am Sa., 12. Sep. 2020, 17:47: > All the doubts, questions, and discussions prove that current system is > counter-intuitive and confusing. The issue in this thread is caused by a bug in the LCL: it blindly assumes that the dynamic code page of the caption string is always utf-8. That is simply wrong (unless you put the burden on the user to always assign an utf-8-encoded string to it, but _that_ is counter-intuitive and confusing). But shouldn't the compiler insert a conversion if the string is declared as CP_1250 and the destination is CP_ACP? 

There are two things:
1) regardless of how the static code page of a string is declared, it is never guaranteed that its dynamic code page will match it. The simplest example is when you assign a RawByteString to it. There is, however, also a second case (and this one indeed is counter-intuitive, but needed for backward compatibility):
2) the second bullet under
https://wiki.freepascal.org/FPC_Unicode_support#Dynamic_code_page

That's what gets triggered here: the source file CP is CP_1250 and the string is also ansistring(1250). That case would be solved by declaring Label as UTF8String though.

Jonas

Reply by email

Tomas Hajny
to Jonas Maebe
Jonas Maebe wrote 36 lines
On 12/09/2020 18:44, Sven Barth via fpc-pascal wrote: > Jonas Maebe via fpc-pascal < > > schrieb am Sa., 12. Sep. > 2020, > 17:47: > > > All the doubts, questions, and discussions prove that current > system is > > counter-intuitive and confusing. > > The issue in this thread is caused by a bug in the LCL: it blindly > assumes that the dynamic code page of the caption string is always > utf-8. That is simply wrong (unless you put the burden on the user > to > always assign an utf-8-encoded string to it, but _that_ is > counter-intuitive and confusing). > > > But shouldn't the compiler insert a conversion if the string is > declared > as CP_1250 and the destination is CP_ACP?  There are two things: 1) regardless of how the static code page of a string is declared, it is never guaranteed that its dynamic code page will match it. The simplest example is when you assign a RawByteString to it. There is, however, also a second case (and this one indeed is counter-intuitive, but needed for backward compatibility): 2) the second bullet under https://wiki.freepascal.org/FPC_Unicode_support#Dynamic_code_page That's what gets triggered here: the source file CP is CP_1250 and the string is also ansistring(1250). That case would be solved by declaring Label as UTF8String though.

Yep.

While performing some tests, I came across other things which are not very nice either (those are specific to the Win32/Win64 target due to the difference between process codepage and console codepage). Let's take the following test program:

{$codepage cp1250}
{$IFDEF USECRT}
uses
  Crt;
{$ENDIF USECRT}
const
  S = 'žluťoučký kůň';
var
  T: string;
begin
  T := S;
{$IFDEF USECRT}
  Write ('Using Crt');
{$ELSE USECRT}
  Write ('Not using Crt');
{$ENDIF USECRT}
  WriteLn (S);
  WriteLn (T);
  WriteLn (DefaultSystemCodepage);
  WriteLn (TextRec (Output).Codepage);
end.

Let's compile it _without_ -dUSECRT and _with_ -Mfpc first. The original poster uses the same default codepage as me. If I start cmd.exe and run "chcp" without parameters, it shows codepage 852 as the console codepage. Now run the test program. It shows that the codepage for the default file handle Output matches the console codepage (as it should), but the string output is incorrect for both WriteLn(S) and WriteLn(T) lines. If you perform "chcp 1250" and run the program again, the codepages match and the string output is correct.

If you compile the same program with -dUSECRT, the output is correct for both WriteLn calls regardless from the console codepage setting (i.e. both for "chcp 852" and for "chcp 1250" - and also for "chcp 65001").

If you compile the same program with -dUSECRT and -Mdelphi together and run the program in a console window set to codepage 852 (i.e. the default setting here), the first WriteLn call is wrong, whereas the second gives a correct result (due to the fact that T becomes an ansistring in mode Delphi and dynamic translation is thus performed as opposed to the case when a shortstring or an untyped constant are passed).

Tomas

Reply by email

Tomas Hajny
to Tomas Hajny: Yep. While performing some tests, I came across other things which are not very nice…
Tomas Hajny wrote 6 lines
On 2020-09-12 18:51, Jonas Maebe via fpc-pascal wrote: > On 12/09/2020 18:44, Sven Barth via fpc-pascal wrote: >> Jonas Maebe via fpc-pascal < >> > schrieb am Sa., 12. Sep. >> 2020, >> 17:47:

.
.

Show quoted text 49 lines
While performing some tests, I came across other things which are not very nice either (those are specific to the Win32/Win64 target due to the difference between process codepage and console codepage). Let's take the following test program: {$codepage cp1250} {$IFDEF USECRT} uses Crt; {$ENDIF USECRT} const S = 'žluťoučký kůň'; var T: string; begin T := S; {$IFDEF USECRT} Write ('Using Crt'); {$ELSE USECRT} Write ('Not using Crt'); {$ENDIF USECRT} WriteLn (S); WriteLn (T); WriteLn (DefaultSystemCodepage); WriteLn (TextRec (Output).Codepage); end. Let's compile it _without_ -dUSECRT and _with_ -Mfpc first. The original poster uses the same default codepage as me. If I start cmd.exe and run "chcp" without parameters, it shows codepage 852 as the console codepage. Now run the test program. It shows that the codepage for the default file handle Output matches the console codepage (as it should), but the string output is incorrect for both WriteLn(S) and WriteLn(T) lines. If you perform "chcp 1250" and run the program again, the codepages match and the string output is correct. If you compile the same program with -dUSECRT, the output is correct for both WriteLn calls regardless from the console codepage setting (i.e. both for "chcp 852" and for "chcp 1250" - and also for "chcp 65001"). If you compile the same program with -dUSECRT and -Mdelphi together and run the program in a console window set to codepage 852 (i.e. the default setting here), the first WriteLn call is wrong, whereas the second gives a correct result (due to the fact that T becomes an ansistring in mode Delphi and dynamic translation is thus performed as opposed to the case when a shortstring or an untyped constant are passed).

Questions resulting from my test above and the observed inconsistencies:

1) Wouldn't it be better if shortstrings are treated the same way as ansistrings with CP_ACP? This would make a difference only during assignments to strings with different codepages. Since strings with different codepages didn't exist in the past (and in the current situation they are simply broken), this change shouldn't break compatibility hopefully.

2) Shouldn't WriteLn with a untyped string constant parameter result in calling some Unicode based version of WriteLn rather than the shortstring overloaded version (since the constant is stored in UTF-16 internally)?

3) Shouldn't we try to make the output of Write with and without unit Crt compatible to each other? If we do so, what should be the encoding used for output redirected to a file - should it use DefaultSystemCodePage, or scpConsoleCP, or what (remember that this question doesn't exist with unit Crt, because unit Crt isn't compatible with redirection).

Tomas

Reply by email

Michael Van Canneyt
to Tomas Hajny
Tomas Hajny wrote 67 lines
On 2020-09-12 23:03, Tomas Hajny wrote: > On 2020-09-12 18:51, Jonas Maebe via fpc-pascal wrote: >> On 12/09/2020 18:44, Sven Barth via fpc-pascal wrote: >>> Jonas Maebe via fpc-pascal < >>> > schrieb am Sa., 12. Sep. >>> 2020, >>> 17:47: . . > While performing some tests, I came across other things which are not > very nice either (those are specific to the Win32/Win64 target due to > the difference between process codepage and console codepage). Let's > take the following test program: > > {$codepage cp1250} > {$IFDEF USECRT} > uses > Crt; > {$ENDIF USECRT} > const > S = 'žluťoučký kůň'; > var > T: string; > begin > T := S; > {$IFDEF USECRT} > Write ('Using Crt'); > {$ELSE USECRT} > Write ('Not using Crt'); > {$ENDIF USECRT} > WriteLn (S); > WriteLn (T); > WriteLn (DefaultSystemCodepage); > WriteLn (TextRec (Output).Codepage); > end. > > Let's compile it _without_ -dUSECRT and _with_ -Mfpc first. The > original poster uses the same default codepage as me. If I start > cmd.exe and run "chcp" without parameters, it shows codepage 852 as > the console codepage. Now run the test program. It shows that the > codepage for the default file handle Output matches the console > codepage (as it should), but the string output is incorrect for both > WriteLn(S) and WriteLn(T) lines. If you perform "chcp 1250" and run > the program again, the codepages match and the string output is > correct. > > If you compile the same program with -dUSECRT, the output is correct > for both WriteLn calls regardless from the console codepage setting > (i.e. both for "chcp 852" and for "chcp 1250" - and also for "chcp > 65001"). > > If you compile the same program with -dUSECRT and -Mdelphi together > and run the program in a console window set to codepage 852 (i.e. the > default setting here), the first WriteLn call is wrong, whereas the > second gives a correct result (due to the fact that T becomes an > ansistring in mode Delphi and dynamic translation is thus performed as > opposed to the case when a shortstring or an untyped constant are > passed). Questions resulting from my test above and the observed inconsistencies: 1) Wouldn't it be better if shortstrings are treated the same way as ansistrings with CP_ACP? This would make a difference only during assignments to strings with different codepages. Since strings with different codepages didn't exist in the past (and in the current situation they are simply broken), this change shouldn't break compatibility hopefully.

No idea what to advise here.
I would think shortstring is ASCII or OEM codepage, not even ANSI :/

2) Shouldn't WriteLn with a untyped string constant parameter result in calling some Unicode based version of WriteLn rather than the shortstring overloaded version (since the constant is stored in UTF-16 internally)?

What is the codepage of a constant string ? Should this not be used ?

Show quoted text 7 lines
3) Shouldn't we try to make the output of Write with and without unit Crt compatible to each other? If we do so, what should be the encoding used for output redirected to a file - should it use DefaultSystemCodePage, or scpConsoleCP, or what (remember that this question doesn't exist with unit Crt, because unit Crt isn't compatible with redirection).

I think this last one are in fact 3 questions:
- What to do if output is redirected externally ? (IMHO nothing)
- What to do if output is redirected internally ? (IMHO, the codepage should be kept)
- Whether and how to extend Crt so it works with unicode. (Since Crt is legacy, I would not touch it; You'd need to rewrite it as unicode.)

Michael.

Reply by email

Tomas Hajny
to Michael Van Canneyt
Michael Van Canneyt wrote 9 lines
On Mon, 14 Sep 2020, Tomas Hajny via fpc-pascal wrote: > On 2020-09-12 23:03, Tomas Hajny wrote: >> On 2020-09-12 18:51, Jonas Maebe via fpc-pascal wrote: >>> On 12/09/2020 18:44, Sven Barth via fpc-pascal wrote: >>>> Jonas Maebe via fpc-pascal < >>>> > schrieb am Sa., 12. Sep. >>>> 2020, >>>> 17:47:

.
.

Show quoted text 9 lines
> 1) Wouldn't it be better if shortstrings are treated the same way as > ansistrings with CP_ACP? This would make a difference only during > assignments to strings with different codepages. Since strings with > different codepages didn't exist in the past (and in the current > situation they are simply broken), this change shouldn't break > compatibility hopefully. No idea what to advise here. I would think shortstring is ASCII or OEM codepage, not even ANSI :/

As far as I'm concerned, there's no difference between OEM or ANSI (or ISO 8859-x for that matter) _unless_ somebody targets Win32/Win64 and never anything else. From this point of view, there's no reason why shortstrings should be always OEM. Historically, they were used simply for the default characters set on the particular operaing environment.

Show quoted text 6 lines
> 2) Shouldn't WriteLn with a untyped string constant parameter result > in calling some Unicode based version of WriteLn rather than the > shortstring overloaded version (since the constant > is stored in UTF-16 internally)? What is the codepage of a constant string ? Should this not be used ?

That's what I wrote - internally, the (untyped) constant strings are stored in UTF-16.

Show quoted text 9 lines
> 3) Shouldn't we try to make the output of Write with and without unit > Crt compatible to each other? If we do so, what should be the encoding > used for output redirected to a file - should it use > DefaultSystemCodePage, or scpConsoleCP, or what (remember that this > question doesn't exist with unit Crt, because unit Crt isn't > compatible with redirection). I think this last one are in fact 3 questions: - What to do if output is redirected externally ? (IMHO nothing)

There's no "nothing". Every text file record has an attribute stating the codepage used for that text file. The question is which codepage should be assigned there under which cases.

- What to do if output is redirected internally ? (IMHO, the codepage should be kept)

Kept from what?

- Whether and how to extend Crt so it works with unicode. (Since Crt is legacy, I would not touch it; You'd need to rewrite it as unicode.)

That is a different question and I don't want to raise that one. My question is simply whether WriteLn (shortstring) should behave differently with and without Crt.

Tomas

Reply by email

Michael Van Canneyt
to Tomas Hajny
Tomas Hajny wrote 53 lines
On 2020-09-14 13:39, Michael Van Canneyt wrote: > On Mon, 14 Sep 2020, Tomas Hajny via fpc-pascal wrote: > >> On 2020-09-12 23:03, Tomas Hajny wrote: >>> On 2020-09-12 18:51, Jonas Maebe via fpc-pascal wrote: >>>> On 12/09/2020 18:44, Sven Barth via fpc-pascal wrote: >>>>> Jonas Maebe via fpc-pascal < >>>>> > schrieb am Sa., 12. Sep. >>>>> 2020, >>>>> 17:47: . . >> 1) Wouldn't it be better if shortstrings are treated the same way as >> ansistrings with CP_ACP? This would make a difference only during >> assignments to strings with different codepages. Since strings with >> different codepages didn't exist in the past (and in the current >> situation they are simply broken), this change shouldn't break >> compatibility hopefully. > > No idea what to advise here. I would think shortstring is ASCII or OEM > codepage, not even ANSI :/ As far as I'm concerned, there's no difference between OEM or ANSI (or ISO 8859-x for that matter) _unless_ somebody targets Win32/Win64 and never anything else. From this point of view, there's no reason why shortstrings should be always OEM. Historically, they were used simply for the default characters set on the particular operaing environment. >> 2) Shouldn't WriteLn with a untyped string constant parameter result >> in calling some Unicode based version of WriteLn rather than the >> shortstring overloaded version (since the constant >> is stored in UTF-16 internally)? > > What is the codepage of a constant string ? Should this not be used ? That's what I wrote - internally, the (untyped) constant strings are stored in UTF-16. >> 3) Shouldn't we try to make the output of Write with and without unit >> Crt compatible to each other? If we do so, what should be the encoding >> used for output redirected to a file - should it use >> DefaultSystemCodePage, or scpConsoleCP, or what (remember that this >> question doesn't exist with unit Crt, because unit Crt isn't >> compatible with redirection). > > I think this last one are in fact 3 questions: > - What to do if output is redirected externally ? (IMHO nothing) There's no "nothing". Every text file record has an attribute stating the codepage used for that text file. The question is which codepage should be assigned there under which cases.

Why do you want to assign one ?
It's the responsability of the user to set the correct codepage.

As far as I can see it is initialized to the default one when the file is
opened:
{$ifdef FPC_HAS_FEATURE_ANSISTRINGS}
   { if no codepage is yet assigned then assign default ansi codepage }
   TextRec(t).CodePage:=TranslatePlaceholderCP(TextRec(t).CodePage);
{$else FPC_HAS_FEATURE_ANSISTRINGS}

I see no need to change this ?

> - What to do if output is redirected internally ? (IMHO, the codepage > should be kept) Kept from what?

Things like AssignCRT() should not set the codepage on the passed text record.

Show quoted text 9 lines
> - Whether and how to extend Crt so it works with unicode. > (Since Crt is legacy, I would not touch it; You'd need to rewrite it > as unicode.) That is a different question and I don't want to raise that one. My question is simply whether WriteLn (shortstring) should behave differently with and without Crt.

I think it will differ since Crt is not codepage aware. If you want it to work the same you'll have to make Crt codepage (and hence unicode) aware.

Michael.

Reply by email

Tomas Hajny
to Michael Van Canneyt
Michael Van Canneyt wrote 68 lines
On Mon, 14 Sep 2020, Tomas Hajny via fpc-pascal wrote: > On 2020-09-14 13:39, Michael Van Canneyt wrote: >> On Mon, 14 Sep 2020, Tomas Hajny via fpc-pascal wrote: >> >>> On 2020-09-12 23:03, Tomas Hajny wrote: >>>> On 2020-09-12 18:51, Jonas Maebe via fpc-pascal wrote: >>>>> On 12/09/2020 18:44, Sven Barth via fpc-pascal wrote: >>>>>> Jonas Maebe via fpc-pascal < >>>>>> > schrieb am Sa., 12. >>>>>> Sep. 2020, >>>>>> 17:47: > . > . >>> 1) Wouldn't it be better if shortstrings are treated the same way as >>> ansistrings with CP_ACP? This would make a difference only during >>> assignments to strings with different codepages. Since strings with >>> different codepages didn't exist in the past (and in the current >>> situation they are simply broken), this change shouldn't break >>> compatibility hopefully. >> >> No idea what to advise here. I would think shortstring is ASCII or >> OEM >> codepage, not even ANSI :/ > > As far as I'm concerned, there's no difference between OEM or ANSI (or > ISO 8859-x for that matter) _unless_ somebody targets Win32/Win64 and > never anything else. From this point of view, there's no reason why > shortstrings should be always OEM. Historically, they were used simply > for the default characters set on the particular operaing environment. > > >>> 2) Shouldn't WriteLn with a untyped string constant parameter result >>> in calling some Unicode based version of WriteLn rather than the >>> shortstring overloaded version (since the constant >>> is stored in UTF-16 internally)? >> >> What is the codepage of a constant string ? Should this not be used ? > > That's what I wrote - internally, the (untyped) constant strings are > stored in UTF-16. > > >>> 3) Shouldn't we try to make the output of Write with and without >>> unit Crt compatible to each other? If we do so, what should be the >>> encoding used for output redirected to a file - should it use >>> DefaultSystemCodePage, or scpConsoleCP, or what (remember that this >>> question doesn't exist with unit Crt, because unit Crt isn't >>> compatible with redirection). >> >> I think this last one are in fact 3 questions: >> - What to do if output is redirected externally ? (IMHO nothing) > > There's no "nothing". Every text file record has an attribute stating > the codepage used for that text file. The question is which codepage > should be assigned there under which cases. Why do you want to assign one ? It's the responsability of the user to set the correct codepage. As far as I can see it is initialized to the default one when the file is opened: {$ifdef FPC_HAS_FEATURE_ANSISTRINGS} { if no codepage is yet assigned then assign default ansi codepage } TextRec(t).CodePage:=TranslatePlaceholderCP(TextRec(t).CodePage); {$else FPC_HAS_FEATURE_ANSISTRINGS} I see no need to change this ?

Please, have a look at OpenStdIO (implemented at the end of text.inc). That one changes the default codepage for Input, Output and StdErr, but it doesn't care about possible redirection and/or piping to some other application (not necessarily written in FPC). That approach has both advantages and disadvantages. The advantage is that "type <redirected_file>" will give correct results on the console. The disadvantage is that opening the redirected file with any Win32 GUI application (let's say notepad.exe) will result in garbage. I don't say that it is necessarily bad, but it should be documented at least if we want to keep it that way.

Show quoted text 7 lines
>> - What to do if output is redirected internally ? (IMHO, the codepage >> should be kept) > > Kept from what? Things like AssignCRT() should not set the codepage on the passed text record.

AssignCrt doesn't do that. The translation is performed when calling the console output functions (on MS Windows).

Show quoted text 13 lines
>> - Whether and how to extend Crt so it works with unicode. >> (Since Crt is legacy, I would not touch it; You'd need to rewrite >> it >> as unicode.) > > That is a different question and I don't want to raise that one. My > question is simply whether WriteLn (shortstring) should behave > differently with and without Crt. I think it will differ since Crt is not codepage aware. If you want it to work the same you'll have to make Crt codepage (and hence unicode) aware.

As mentioned by me, Crt is currently more codepage aware than the System unit output as far as output to console is concerned, because Crt provides correct output even for shortstrings (unlike the System unit).

Tomas

Reply by email

Michael Van Canneyt
to Tomas Hajny
Tomas Hajny wrote 18 lines
> opened: > {$ifdef FPC_HAS_FEATURE_ANSISTRINGS} > { if no codepage is yet assigned then assign default ansi codepage } > TextRec(t).CodePage:=TranslatePlaceholderCP(TextRec(t).CodePage); > {$else FPC_HAS_FEATURE_ANSISTRINGS} > > I see no need to change this ? Please, have a look at OpenStdIO (implemented at the end of text.inc). That one changes the default codepage for Input, Output and StdErr, but it doesn't care about possible redirection and/or piping to some other application (not necessarily written in FPC). That approach has both advantages and disadvantages. The advantage is that "type <redirected_file>" will give correct results on the console. The disadvantage is that opening the redirected file with any Win32 GUI application (let's say notepad.exe) will result in garbage. I don't say that it is necessarily bad, but it should be documented at least if we want to keep it that way.

I would definitely keep it that way.

As I see it:
Redirection or not should not matter, the system should assume console output. Things like 'tee' make this concept dubious in any case:

If you pipe output to a program, you don't expect the codepage to change because of the redirection.

Show quoted text 8 lines
> I think it will differ since Crt is not codepage aware. If you want it > to > work the same you'll have to make Crt codepage (and hence unicode) > aware. As mentioned by me, Crt is currently more codepage aware than the System unit output as far as output to console is concerned, because Crt provides correct output even for shortstrings (unlike the System unit).

I would need to check the details, but that sounds more like accidental behaviour as opposed to intended behaviour to me :-)

Michael.

Reply by email

Tomas Hajny
to Michael Van Canneyt
Michael Van Canneyt wrote 31 lines
On Mon, 14 Sep 2020, Tomas Hajny via fpc-pascal wrote: >> opened: >> {$ifdef FPC_HAS_FEATURE_ANSISTRINGS} >> { if no codepage is yet assigned then assign default ansi codepage >> } >> TextRec(t).CodePage:=TranslatePlaceholderCP(TextRec(t).CodePage); >> {$else FPC_HAS_FEATURE_ANSISTRINGS} >> >> I see no need to change this ? > > Please, have a look at OpenStdIO (implemented at the end of text.inc). > That one changes the default codepage for Input, Output and StdErr, > but it doesn't care about possible redirection and/or piping to some > other application (not necessarily written in FPC). That approach has > both advantages and disadvantages. The advantage is that "type > <redirected_file>" will give correct results on the console. The > disadvantage is that opening the redirected file with any Win32 GUI > application (let's say notepad.exe) will result in garbage. I don't > say that it is necessarily bad, but it should be documented at least > if we want to keep it that way. I would definitely keep it that way. As I see it: Redirection or not should not matter, the system should assume console output. Things like 'tee' make this concept dubious in any case: If you pipe output to a program, you don't expect the codepage to change because of the redirection.

No problem, but I'd suggest documenting it at least.

Show quoted text 12 lines
>> I think it will differ since Crt is not codepage aware. If you want >> it to >> work the same you'll have to make Crt codepage (and hence unicode) >> aware. > > As mentioned by me, Crt is currently more codepage aware than the > System unit output as far as output to console is concerned, because > Crt provides correct output even for shortstrings (unlike the System > unit). I would need to check the details, but that sounds more like accidental behaviour as opposed to intended behaviour to me :-)

Not really accidental:

r3606 | florian | 2006-05-20 23:42:58 +0200 (Sat, 20 May 2006) | 2 lines
* fix from Maxim Ganetsky to fix CRT output with non latin code pages, should fix #6785

(there were additional changes performed later, but the primary change was this one)

Tomas

Reply by email

Michael Van Canneyt
to Tomas Hajny
Tomas Hajny wrote 15 lines
>> application (let's say notepad.exe) will result in garbage. I don't >> say that it is necessarily bad, but it should be documented at least >> if we want to keep it that way. > > I would definitely keep it that way. > > As I see it: Redirection or not should not matter, the system should > assume console output. > Things like 'tee' make this concept dubious in any case: > > If you pipe output to a program, you don't expect the codepage to > change > because of the redirection. No problem, but I'd suggest documenting what exactly ? That redirecting does not change the codepage ?
Show quoted text 22 lines
>>> I think it will differ since Crt is not codepage aware. If you want >>> it to >>> work the same you'll have to make Crt codepage (and hence unicode) >>> aware. >> >> As mentioned by me, Crt is currently more codepage aware than the >> System unit output as far as output to console is concerned, because >> Crt provides correct output even for shortstrings (unlike the System >> unit). > > I would need to check the details, but that sounds more like > accidental behaviour as opposed to intended behaviour to me :-) Not really accidental: r3606 | florian | 2006-05-20 23:42:58 +0200 (Sat, 20 May 2006) | 2 lines * fix from Maxim Ganetsky to fix CRT output with non latin code pages, should fix #6785 (there were additional changes performed later, but the primary change was this one)

Does this handle UTF8 ?

Judging by the sources, I would think not:

Interface
{$mode fpc} // Shortstring is assumed
{$i crth.inc}
Const
   { Controlling consts }
   Flushing     = false;               {if true then don't buffer output}
   ConsoleMaxX  = 1024;
   ConsoleMaxY  = 1024;
   ScreenHeight : longint = 25;
   ScreenWidth  : longint = 80;
Type
   TCharAttr=packed record
     ch   : char;
     attr : byte;
   end;
   TConsoleBuf=Array[0..ConsoleMaxX*ConsoleMaxY-1] of TCharAttr;
   PConsoleBuf=^TConsoleBuf;
var
   ConsoleBuf : PConsoleBuf;

Since every screen position handles only a single char (byte) there is no way this can handle UTF8. Maybe other "real" single-byte codepages, yes.

Michael.

Reply by email

Benito van der Zander
to Michael Van Canneyt: I would definitely keep it that way. As I see it: Redirection or not should not matter,…

Hi,

Show quoted text 9 lines
I would definitely keep it that way. As I see it: Redirection or not should not matter, the system should assume console output. Things like 'tee' make this concept dubious in any case: If you pipe output to a program, you don't expect the codepage to change because of the redirection.

In my projects, I set the output codepage to utf-8 when there is a redirection

It is platform agnostic, on identical inputs, it creates identical output files on Linux and Windows. No one needs OEM files on Linux

Best,
Benito

Michael Van Canneyt wrote 50 lines
On Mon, 14 Sep 2020, Tomas Hajny via fpc-pascal wrote: >> opened: >> {$ifdef FPC_HAS_FEATURE_ANSISTRINGS} >>   { if no codepage is yet assigned then assign default ansi codepage } >> TextRec(t).CodePage:=TranslatePlaceholderCP(TextRec(t).CodePage); >> {$else FPC_HAS_FEATURE_ANSISTRINGS} >> >> I see no need to change this ? > > Please, have a look at OpenStdIO (implemented at the end of > text.inc). That one changes the default codepage for Input, Output > and StdErr, but it doesn't care about possible redirection and/or > piping to some other application (not necessarily written in FPC). > That approach has both advantages and disadvantages. The advantage is > that "type <redirected_file>" will give correct results on the > console. The disadvantage is that opening the redirected file with > any Win32 GUI application (let's say notepad.exe) will result in > garbage.  I don't say that it is necessarily bad, but it should be > documented at least if we want to keep it that way. I would definitely keep it that way. As I see it: Redirection or not should not matter, the system should assume console output. Things like 'tee' make this concept dubious in any case: If you pipe output to a program, you don't expect the codepage to change because of the redirection. >> I think it will differ since Crt is not codepage aware. If you want >> it to >> work the same you'll have to make Crt codepage (and hence unicode) >> aware. > > As mentioned by me, Crt is currently more codepage aware than the > System unit output as far as output to console is concerned, because > Crt provides correct output even for shortstrings (unlike the System > unit). I would need to check the details, but that sounds more like accidental behaviour as opposed to intended behaviour to me :-) Michael. _______________________________________________ fpc-pascal maillist  - https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal

Reply by email

Tomas Hajny
to Michael Van Canneyt: Does this handle UTF8 ? Judging by the sources, I would think not: Interface {$mode fpc}…
Michael Van Canneyt wrote 19 lines
On Mon, 14 Sep 2020, Tomas Hajny via fpc-pascal wrote: >>> application (let's say notepad.exe) will result in garbage. I don't >>> say that it is necessarily bad, but it should be documented at least >>> if we want to keep it that way. >> >> I would definitely keep it that way. >> >> As I see it: Redirection or not should not matter, the system should >> assume console output. >> Things like 'tee' make this concept dubious in any case: >> >> If you pipe output to a program, you don't expect the codepage to >> change >> because of the redirection. > > No problem, but I'd suggest documenting it at least. Document what exactly ? That redirecting does not change the codepage ?

1) Document that the following test program results in two different lines under Win32/Win64 (unless you change the console codepage to be equal to the process codepage before running the program):

{$CODEPAGE CP1250}
const
  S = 'Úžasné';
var
  A: ansistring;
  T: text;
begin
  Assign (T, '');
  Rewrite (T);
  A := S;
  WriteLn (A);
  WriteLn (T, A);
  Close (T);
end.

2) Document that the following test program compiled, run and having the output redirected to a file named output1.txt results in two files with different content (again unless you change the console to the process codepage before running the program):

{$CODEPAGE CP1250}
const
  S = 'Úžasné';
var
  A: ansistring;
  T: text;
begin
  Assign (T, 'output2.txt');
  Rewrite (T);
  A := S;
  WriteLn (A);
  WriteLn (T, A);
  Close (T);
end.
  .
  .
Show quoted text 11 lines
> Not really accidental: > > r3606 | florian | 2006-05-20 23:42:58 +0200 (Sat, 20 May 2006) | 2 > lines > * fix from Maxim Ganetsky to fix CRT output with non latin code pages, > should fix #6785 > > (there were additional changes performed later, but the primary change > was this one) Does this handle UTF8 ?

In what sense? It works correctly (under Win32/Win64) if I change the console codepage to 65001 (both for shortstrings and for ansistrings), and it works correctly if I assign a constant to an Utf8string and write it to the console regardless from the console codepage.

Show quoted text 31 lines
Judging by the sources, I would think not: Interface {$mode fpc} // Shortstring is assumed {$i crth.inc} Const { Controlling consts } Flushing = false; {if true then don't buffer output} ConsoleMaxX = 1024; ConsoleMaxY = 1024; ScreenHeight : longint = 25; ScreenWidth : longint = 80; Type TCharAttr=packed record ch : char; attr : byte; end; TConsoleBuf=Array[0..ConsoleMaxX*ConsoleMaxY-1] of TCharAttr; PConsoleBuf=^TConsoleBuf; var ConsoleBuf : PConsoleBuf; Since every screen position handles only a single char (byte) there is no way this can handle UTF8. Maybe other "real" single-byte codepages, yes.

I assume that you're looking at the implementation for Linux, whereas I talk about the implementation for Win32/Win64. I don't know if it makes any difference with regard to UTF-8 string handling on Linux (I'm not aware of any particular issues, but I might be wrong).

Tomas

Reply by email

Tomas Hajny
to Michael Van Canneyt: Does this handle UTF8 ? Judging by the sources, I would think not: Interface {$mode fpc}…
Michael Van Canneyt wrote 19 lines
On Mon, 14 Sep 2020, Tomas Hajny via fpc-pascal wrote: >>> application (let's say notepad.exe) will result in garbage. I don't >>> say that it is necessarily bad, but it should be documented at least >>> if we want to keep it that way. >> >> I would definitely keep it that way. >> >> As I see it: Redirection or not should not matter, the system should >> assume console output. >> Things like 'tee' make this concept dubious in any case: >> >> If you pipe output to a program, you don't expect the codepage to >> change >> because of the redirection. > > No problem, but I'd suggest documenting it at least. Document what exactly ? That redirecting does not change the codepage ?

1) Document that the following test program results in two different lines under Win32/Win64 (unless you change the console codepage to be equal to the process codepage before running the program):

{$CODEPAGE CP1250}
const
  S = 'Úžasné';
var
  A: ansistring;
  T: text;
begin
  Assign (T, '');
  Rewrite (T);
  A := S;
  WriteLn (A);
  WriteLn (T, A);
  Close (T);
end.

2) Document that the following test program compiled, run and having the output redirected to a file named output1.txt results in two files with different content (again unless you change the console to the process codepage before running the program):

{$CODEPAGE CP1250}
const
  S = 'Úžasné';
var
  A: ansistring;
  T: text;
begin
  Assign (T, 'output2.txt');
  Rewrite (T);
  A := S;
  WriteLn (A);
  WriteLn (T, A);
  Close (T);
end.
  .
  .
Show quoted text 11 lines
> Not really accidental: > > r3606 | florian | 2006-05-20 23:42:58 +0200 (Sat, 20 May 2006) | 2 > lines > * fix from Maxim Ganetsky to fix CRT output with non latin code pages, > should fix #6785 > > (there were additional changes performed later, but the primary change > was this one) Does this handle UTF8 ?

In what sense? It works correctly (under Win32/Win64) if I change the console codepage to 65001 (both for shortstrings and for ansistrings), and it works correctly if I assign a constant to an Utf8string and write it to the console regardless from the console codepage.

Show quoted text 31 lines
Judging by the sources, I would think not: Interface {$mode fpc} // Shortstring is assumed {$i crth.inc} Const { Controlling consts } Flushing = false; {if true then don't buffer output} ConsoleMaxX = 1024; ConsoleMaxY = 1024; ScreenHeight : longint = 25; ScreenWidth : longint = 80; Type TCharAttr=packed record ch : char; attr : byte; end; TConsoleBuf=Array[0..ConsoleMaxX*ConsoleMaxY-1] of TCharAttr; PConsoleBuf=^TConsoleBuf; var ConsoleBuf : PConsoleBuf; Since every screen position handles only a single char (byte) there is no way this can handle UTF8. Maybe other "real" single-byte codepages, yes.

I assume that you're looking at the implementation for Linux, whereas I talk about the implementation for Win32/Win64. I don't know if it makes any difference with regard to UTF-8 string handling on Linux (I'm not aware of any particular issues, but I might be wrong).

Tomas

Reply by email