Hello,

I am doing some tests and trying to find out what exactly happens with string constants. Suppose this source code:

var
  AString: ansistring;
begin
  AString := 'A acentuação é necessária no Português';
end;

The source is encoded in utf-8, but the compiler isn't told about that in any way:
* There is no BOM marker
* no {$codepage } directive

I took a look at compier/scanner.pas and I see that clearly if we have {$codepage utf-8} the literal string will be stored as UTF-16 (using the charset map in compiler/cp8859_1.pas) in the executable and converted back at run-time if assigned to ansistring, using Wide2AnsiMoveProc.

And what it that doesn't happen? Will it also be encoded in UTF-16?Or will it just be left alone as a group of bytes?

Jonas once said that if there is nothing saying which encoding the source is, it will be treated as ISO 8859-1 and encoded as UTF-16, and then at run-time be converted when assigned to a ansistring to the current locale.

But Paul has a russian windows and he made a lcl application with utf-8 text and it run ok, althougth I would suppose it shouldn't work, because the following conversions would occur:

iso 8859-1 ---> utf-16 ---> locate (maybe iso 8859-5???)

And as LCL expects utf-8 it will only work if the first and last locales on this double transformation are equal. Or if there is no conversion taking place.

Also, what exactly defines how string literals should be handled? Is that compiler specific or are we following delphi, or how is the behavior defined?

thanks,

This applies only if you assign it to a widestring or include chars with ord>255 like #555 or using utf-8. The compiler takes the given code page only to do from/to widestring conversions, no more no less. If no widestring/no char>255 is envolved, nothing happens at compile time. The compiler can't know the encoding of ansistrings at the target system so it's obvious that it shouldn't mess with ansistring constants except if they are assigned to widestrings which are encoding independent.

The BOM/code page directive simply allows (and nothing else!) you to create widestrings using simple string constants even if you want to use chinese or whatever characters. Just tell the compiler what enconding to use to convert the string constant to widestring.

Why?

because the following conversions would occur: iso 8859-1 ---> utf-16 ---> locate (maybe iso 8859-5???)

When? It works correct because:
- the source code uses the same locale for ansistrings as the host system (UTF-8)
- the compiler doesn't mess with the encoding because no widestrings are envolved

Conversion is only done at compile time if string constants are assigned to widestrings everything else would mean that the compiler is guessing.

The BOM/code page directive simply allows (and nothing else!) you to create widestrings using simple string constants even if you want to use chinese or whatever characters. Just tell the compiler what enconding to use to convert the string constant to widestring.

Of course, this might go wrong if you assign such a string constant to an ansistring because this is done at run time because it's obvious that assigning a widestring (that's what you defined using chars>255) to an ansistring must be done at run time because we know the encoding of the ansistring only at runtime.