Hello,

when compiling a program with fpc on Windows, fpc seems to have the console output encoding wrong.

Example program (I am using "ä" because I am on german OS, so "ä" is in my DOS Latin and also in my ANSI codepage):

{$codepage utf8}
program TestNote;
begin
{$note ä}
end.

The console output is:

Free Pascal Compiler version 3.3.1 [2023/08/21] for i386 Copyright (c) 1993-2023 by Florian Klaempfl and others (1002) Target OS: Win32 for i386
(3104) Compiling TestNote.lpr
p:\programs\_zk\TestNote\TestNote.lpr(6,2) Note: (2025) User defined: ä (9015) Linking P:\programs\_zk\TestNote\lib\Debug\i386-win32\TestNote.exe (1008) 8 lines compiled, 0.1 sec, 78736 bytes code, 2148 bytes data (1022) 2 hint(s) issued
(1023) 1 note(s) issued

So it looks like the FPC compiler output is raw utf-8 instead of DOS Latin codepage. Is this wanted?

If yes, then Lazarus could take advantage of it and read the console output from FPC as UTF-8. Currently wrong characters are shown in Lazarus messages window on Windows.

If not and console (DOS Latin) codepage should be used by default, would it make sense to add a compiler command line option to set the output codepage? In that case Lazarus could tell FPC to use UTF-8.

Ondrej

On 2023-08-28 15:53, Ondrej Pokorny via fpc-devel wrote:

Hello Ondrej,

.
.

Your output is different from mine (using a freshly compiled trunk compiler):

Free Pascal Compiler version 3.3.1 [2023/08/29] for i386 Copyright (c) 1993-2023 by Florian Klaempfl and others Target OS: Win32 for i386
Compiling t2.pas
t2.pas(4,2) Note: User defined: ä
Linking t2.exe
5 lines compiled, 0.0 sec, 38160 bytes code, 1412 bytes data 1 note(s) issued

However, I run the compiler from regular command line, not from Lazarus. What do you get when running the following test program (you can simulate the same conditions e.g. by naming the resulting binary fpc.exe and configuring Lazarus to use this "compiler"):

uses
  Windows;
begin
  WriteLn (GetACP);
  WriteLn (GetConsoleCP);
end.

Tomas

Then you don't have default windows setup. Maybe you installed the cp 65001?

[...] However, I run the compiler from regular command line, not from Lazarus.

I get the same in cmd.exe and via TProcess.

ACP is 1252
ConsoleCP 437

Mattias

Yes, Tomas, in other words: maybe you enabled "Beta support for UTF-8" in language options on your Windows?

> [...] > However, I run the compiler from regular command line, not from Lazarus. I get the same in cmd.exe and via TProcess.

Agreed.

On my system:

ACP is 1252
ConsoleCP is 850

(437 is DOS-US, 850 is DOS-Latin-1)

https://en.wikipedia.org/wiki/Code_page_437
https://en.wikipedia.org/wiki/Code_page_850

Ondrej

Default setup of what exactly? I didn't install anything special, but my MS Windows installation indeed supports CP 65001. I guess that it may be different for different MS Windows versions - specifically, I tested it on MS Windows 10 Enterprise version 22H2. However, that shouldn't make a difference, because Ondrej stated that he gets compiler output in UTF-8, but I also tried redirecting the output to a file and the result is in CP852 which is my default "ANSI" codepage (and thus also the default console codepage). There's no place where UTF-8 availability might change the result in this. However, I _didn't_ test it under Lazarus. I suspect that Lazarus messing with the codepages may change the behaviour by changing either the system codepage or console codepage to UTF-8 and this then results in compiler emitting output in this different codepage (i.e. UTF-8).

I.e. you open cmd.exe e.g. via the Windows key+R (rather than by shelling from Lazarus)?

Just to be sure - you get this when running the test from Lazarus as a "compiler" (as suggested in my note above)?

Tomas

If you mean
https://stackoverflow.com/questions/56419639/what-does-beta-use-unicode-utf-8-for-worldwide-language-support-actually-do, then no, I don't have this option checked. However, even if I had it checked (I don't), it couldn't make a difference, because I didn't rely on the output displayed on console, but I actually checked the codepage of the compiler output by redirecting it to a file and checking the content of that file as well.

.
.

(437 is DOS-US, 850 is DOS-Latin-1) https://en.wikipedia.org/wiki/Code_page_437 https://en.wikipedia.org/wiki/Code_page_850

Thanks, but I'm quite well aware of the codepage stuff... ;-) (including ISO, Mac codepages, DBCS codepages, EBCDIC codepages, non-standard / custom codepages created in some Central and East European countries for better support of their local languages, etc.)

Tomas

So did I:

See output.txt in the attachment. It is the output from redirection with "> output.txt".

The output is double-encoded on my system (UTF-8 as CP-850) - it means I have to convert it first from CP-850 to ANSI, then I see correct raw UTF-8 byte stream.

Ondrej

On 2023-08-30 16:32, Ondrej Pokorny via fpc-devel wrote: .
.

Interesting, but it doesn't change anything on the fact that my result is different (and correct) and I don't think that my configuration is special in any way...

Could you please change my test program sent earlier by changing "WriteLn (GetConsoleCP);" to "WriteLn (GetConsoleOutputCP);" recompile and check output of that one (GetConsoleCP is not useful in this context, that was my fault :-( ) - just in case?

Do you get a different behaviour if using a stock 3.2.2 compiler (the one distributed by the FPC team, not a Lazarus distribution)?

Tomas

Of course.

No, when running the program in the the console. No Lazarus involved.

Mattias

Yes, it is strange. Mattias and Zeljko have the same output like me (Zeljko wrote about it in the Lazarusdev mailing list). So, currently you are the only one with different result from a group of 4 people involved in the same test :)

Could you please change my test program sent earlier by changing "WriteLn (GetConsoleCP);" to "WriteLn (GetConsoleOutputCP);" recompile and check output of that one (GetConsoleCP is not useful in this context, that was my fault :-( ) - just in case?

GetConsoleOutputCP = GetConsoleCP = 850 on my system.

Do you get a different behaviour if using a stock 3.2.2 compiler (the one distributed by the FPC team, not a Lazarus distribution)?

Stock FPC 3.2.2 has the same output:

Free Pascal Compiler version 3.2.2 [2021/05/15] for i386 Copyright (c) 1993-2021 by Florian Klaempfl and others Target OS: Win32 for i386
Compiling TestNote.lpr
TestNote.lpr(11,3) Note: User defined: ä
Linking TestNote.exe
13 lines compiled, 0.1 sec, 28864 bytes code, 1332 bytes data 1 note(s) issued

Ondrej

Sorry to bother you with something as trivial: is your t2.pas file really encoded in UTF-8?

Because if I compile an ANSI file with the {$codepage utf8} declaration, then I get "correct" output. But obviously this is very wrong.

You can try yourself with the attached files. So maybe this is your mistake?

Ondrej

Ondrej Pokorny wrote

On 30.08.2023 17:04, Tomas Hajny via fpc-devel wrote:

(only sent to Ondrej instead of the list by mistake, resending it to the list now)

.
.

I'm the only one of those 4 not involved in Lazarus. ;-) On the other hand, I wasn't involved in the base RTL codepage handling implementation for Windows at all, i.e. it isn't like that I implemented something in a way only working correctly on my configuration or so.

OK. Please, test it also the way I suggested earlier, i.e. from within Lazarus by pretending that it's the compiler (Mattias already wrote that he ran the program from command line, but I'd like to make sure that there's no difference triggered by Lazarus when running the compiler from it).

Tomas

On 2023-08-30 17:23, Ondrej Pokorny via fpc-devel wrote:

.
.

Well, you're right, this was indeed my mistake, shame on me. :-( Then I can confirm that the compiler behaviour is indeed wrong (although I have no clue why it behaves that way).

Tomas

Having seen the outputs, I think that the compiler just ignores the source file encoding for {$MESSAGE} and {$NOTE}. It reads them always as ANSI and then converts them to DOS-whatever.

That would explain why UTF-8 byte stream is encoded into DOS CP.

So the fix should be quite easy - when {$MESSAGE} or {$NOTE} is read into a string, set the correct codepage of the string.

Ondrej

Try compiling the compiler with the .lpi and turn on the utf-8 option in the lazarus application/manifest screen.

But that makes it Windows 10  1905+ or so.

I was correct in my assumption and I was able to fix it: https://gitlab.com/freepascal.org/fpc/source/-/merge_requests/482

On the other hand, when I read the $CODEPAGE docs: https://www.freepascal.org/docs-html/prog/progsu87.html#x95-940001.3.4 There it is stated that only literal strings follow $CODEPAGE and the actual code must be in US-ASCII.

But you know: Delphi compatibility :) ...and there is no "illegal character" compiler error as it is for:

var
   ä: string;

so one would expect {$note ä} to show up correctly.

Ondrej

Ondrej Pokorny wrote

Am 30.08.2023 um 20:21 schrieb Ondrej Pokorny via fpc-devel:

.
.

Indeed. I'm glad that our discussion (although triggered by my mistake :-( ) led to a positive outcome at last. :-)

Tomas

:) We are not at the finish yet. Positive outcome will be when it gets fixed in the compiler source.

Ondrej

On 2023-08-31 17:08, Ondrej Pokorny via fpc-devel wrote: .
.

:) We are not at the finish yet. Positive outcome will be when it gets fixed in the compiler source.

Yes, I know. I checked your patch briefly now and I don't like the fact that it basically duplicates the code of the two original TScannerFile methods. This isn't very good from maintainability point of view, because if somebody changes/fixes some issue in one, (s)he may easily forget about the other. I see several potential solutions:

1) Keep the original methods and perform codepage translation only in dir_message. I understand that it keeps the current limitation of the message length (which may lead to broken UTF-8 string among others), but it's questionable whether that really matters, IMHO, or whether it could just be documented as a limitation.

2) Always use the ansistring version when parsing the comments. Obviously, this increases compilation time for directives parsing somewhat - we could possibly measure the impact and then decide if the increase is important.

3) Find some smart way how to share the same code for both ansistring and shortstring version (preferably without generics ;-) ). I don't know if this option is feasible, but it might be a good one if so.

Tomas

That is actually what I started with. Then I saw that there are some parts of the code that demand ShortString output and decided to keep both versions.

But basically it is possible to have the 2 ShortString variants use the new Long variants + check the Length and valid characters (to me it looks like there really should be checks for ASCII characters when ShortString is involved). That would be the simplest way to get rid of doubled code. I don't think the compiler performance will be affected that much and AFAIK maintainability is more important than performance anyway.

Or the bigger step would be to make the compiler unicode-ready:

var
   ä: string;

+ file names, paths etc.

Then only the new Long variant would be needed :)

Ondrej

Ondrej Pokorny wrote

On 31.08.2023 17:54, Tomas Hajny via fpc-devel wrote:

.
.

.
.

Yes, that's what I meant by option 2 - I realize that most cases need to end up in shortstrings anyway.

Apparently not only the compiler Unicode-ready, but also the language. Looking forward to identifiers looking like characters having specific meaning in Pascal, but differing in their Unicode codes (e.g. m-dash). Or maybe rather not... I'm sure it would be a good approach to provide enough room for compilers of some other languages to finish their compilation in time comparable to FPC... ;-)

Tomas