Hello,

my application uses the AnsiString type to store UTF-8 data. That was totally fine. Now in FPC 3, automatic conversions cause data loss. I get question marks replacing Chinese characters, for example.

I do not fully understand at which points these conversions are done. The FPC 3 Unicode documentation says something about "passing it to a RTL routine".

What about this code:
var a,b:Ansistring;
begin
  a:=Utf8Encode(AWideString);
  b:=Copy(a,1,10);
  end;

Is "Copy" an RTL routine? Is this OK or not?

Best for me would be to be able to turn the conversions off completely. Is that possible?

Cheers,
Tobias

You cannot, but you can set DefaultSystemCodePage to CP_UTF8. Then no conversions will be done for all ansistrings that contain UTF8.

Michael.

You cannot, but you can set DefaultSystemCodePage to CP_UTF8. Then no conversions will be done for all ansistrings that contain UTF8.

Fantastic. Many thanks. That fixes my problem entirely (I think).

Sorry, I was not able to come to that conclusion from the existing docs.

Cheers,
Tobias

Sorry, I was not able to come to that conclusion from the existing docs.

Your question was not about Lazarus but maybe you should read this: http://wiki.freepascal.org/Better_Unicode_Support_in_Lazarus It works also without LCL.
The bottom line is: remove all explicit conversion functions.

Juha

[...] You cannot, but you can set DefaultSystemCodePage to CP_UTF8.

I think it is important to note how to do this properly:

  SetMultiByteConversionCodePage(CP_UTF8);
  SetMultiByteRTLFileSystemCodePage(CP_UTF8);

You should add these lines in an early initialization section. The beginning of your program might be too late.

Then no conversions will be done for all ansistrings that contain UTF8.

And this really means AnsiString, not AnsiString(something).

Mattias

The latter cannot contain UTF8 unless you do some really nasty tricks... :-)

Michael

Your question was not about Lazarus but maybe you should read this: http://wiki.freepascal.org/Better_Unicode_Support_in_Lazarus

Very interesting, but apparently there is some wrong info.

It says "On Linux and Mac OS X UTF-8 is typically the system codepage, so the RTL uses here by default CP_UTF8."

Not on my Mac. DefaultSystemCodePage is 20127 before I set it to 65001.

Should I fix this bit on the documentation page?

Apparently the LCL assumes that FCL sets it to UTF-8, but it does not.

I think the LCL should explicitly set it to CP_UTF8 on Mac and Linux.

Cheers,
Tobias

No. It says 'typically'.

That doesn't necessarily mean it is so, but is mostly correct.

Apparently the LCL assumes that FCL sets it to UTF-8, but it does not. I think the LCL should explicitly set it to CP_UTF8 on Mac and Linux.

No, that would be wrong. it normally is CP_UTF8 on linux. But it can differ. I know people that still use ISO-8895 (or something similar). Only the programmer can know what is correct.

Michael.

That's the first part of the sentence.

The second part says: "the RTL uses here by default CP_UTF8."

That is wrong. It does not (at least not on Mac). This must be fixed.

Cheers,
Tobias

UTF8String is type AnsiString(CP_UTF8) and if you mix that with AnsiString the compiler adds conversions code, because at compile time CP_ACP is not UTF-8.
These kind of traps confuse people.

Mattias

tobiasgiesen wrote on Mon, 04 Apr 2016:

That's because you don't use the cwstring unit nor another widestring manager. In that case, the DefaultSystemCodePage is set to ASCII because the default string conversion routines on Unix platforms do not support anything else (neither for converting from ansi/shortstring to wide/unicodestring, nor between ansistrings using different codepages). This is not new with FPC 3.0, it has always been like that.

Jonas

Why do you think it is wrong ?
It is perhaps wrong on your computer. Maybe it checks the environment ? Did you include clocale ? etc. There are maybe a 100 things that can change this.

Don't be too hasty in changing things.
Jonas (who created that page) is usually very careful in such matters.

Michael.

Michael Van Canneyt wrote on Mon, 04 Apr 2016:

Don't be too hasty in changing things. Jonas (who created that page) is usually very careful in such matters.

I did forget to mention that the Default*CodePage variables are only initialised with the "real" values on *nix platforms if you include a widestring manager unit. I have now updated the page to reflect this fact.

Jonas

I will update the docs with such info for the upcoming 3.0.2. It clearly needs mentioning...

Michael.

It says "On Linux and Mac OS X UTF-8 is typically the system codepage, so the RTL uses here by default CP_UTF8." Not on my Mac. DefaultSystemCodePage is 20127 before I set it to 65001.

Indeed, and on my FreeBSD system DefaultSystemCodePage returns 0.

So the "typically" seems more like "unlikely in most cases".

Regards,
- Graeme -

Not necessarily:
if you are dealing with UTF8/UnicodeString and other codepages, it is quite likely, and preferred, that you have unit cwstring included.

Michael.

Michael Van Canneyt wrote

No. It says 'typically'. That doesn't necessarily mean it is so, but is mostly correct.

It is still a very vague assumption. As I mentioned in my previous reply, that statement is false on my FreeBSD system too. So I should read the wiki as "typically incorrect" ;-)

No, that would be wrong. it normally is CP_UTF8 on linux. But it can differ. I know people that still use ISO-8895 (or something similar). Only the programmer can know what is correct.

Just curious, how do you change the default codepage on a Linux system? By exporting a new value to the LANG environment variable?

Regards,
- Graeme -

Michael Van Canneyt wrote

if you are dealing with UTF8/UnicodeString and other codepages, it is quite likely, and preferred, that you have unit cwstring included.

OK, I just confirmed. Adding clocale to my 5-line test program doesn't affect the DefaultSystemCodePage result, but as soon as I add cwstring to the uses clause, then DefaultSystemCodePage returns 65001.

I never use the UnicodeString type, and was under the impression that only UnicodeString (and WideString) requires the cwstring unit to function correctly. I use UTF-8 encoded text everywhere (stored inside String), so never bothered with the cwstring unit. But clearly in FPC 3.0 it is vital to include cwstring if you do anything text related.

Something funny:
I thought it really funny when Delphi introduced Unicode support, and how everybody struggled. I thought: "what a mess". Now FPC developers seem to be in the same boat. :-/

Regards,
- Graeme -

Jonas Maebe wrote

I have now updated the page to reflect this fact.

Thanks Jonas, that's an important point to note.

Regards,
- Graeme -

OK, I just confirmed. Adding clocale to my 5-line test program doesn't affect the DefaultSystemCodePage result, but as soon as I add cwstring to the uses clause, then DefaultSystemCodePage returns 65001.

On Mac, not even cwstring does that. It sets the DefaultSystemCodePage to 20127.

So, on Mac, the DefaultSystemCodePage is not "typically" set to UTF_8. It is never set to UTF_8 unless you do it yourself.

Cheers,
Tobias

Yes. And various LC_ environment vars.

Michael.

On Mac, not even cwstring does that. It sets the DefaultSystemCodePage to 20127.

I just installed FPC 3.0 on my Macbook Pro (bought in the UK) and did the same test. Here DefaultSystemCodePage returned 65001. So I guess it depends on your OSX installation and which default locale settings was set up during install.

Regards,
- Graeme -

All my Macs since 10.4 had UTF-8 as default and I can't remember a setting during install to change it.

How did you get a codepage 20127 Mac?

Mattias

How did you get a codepage 20127 Mac?

The Mac is UTF-8, but cwstring or whatever does not realize it.

Since I cannot easily step into it with the debugger, I can't tell you why.

Terminal has LC_CTYPE=UTF-8.

Well I will just set the default codepages manually.

Cheers,
Tobias

tobiasgiesen wrote on Mon, 04 Apr 2016:

> How did you get a codepage 20127 Mac? The Mac is UTF-8,

This statement makes no sense. There is no "UTF-8" or "non-UTF-8" Mac. The Unix environment on OS X can use any OS-supported code page.

but cwstring or whatever does not realize it.

cwstring simply sets the code page based on how it is defined in your environment.

Terminal has LC_CTYPE=UTF-8.

What about LC_ALL? LC_ALL overrides LC_CTYPE, because that is how the meaning of these environment variables is defined by POSIX (see e.g. http://pubs.opengroup.org/onlinepubs/7908799/xbd/envvar.html )

Well I will just set the default codepages manually.

Then you will probably be back in a few months with another message complaining how FPC 3.0 supposedly breaks X or Y, because you are merely hiding the real issue (e.g. when calling an external program, which may then try to interpret your UTF-8 command line arguments as plain ASCII).

Jonas

> Terminal has LC_CTYPE=UTF-8. What about LC_ALL?

My Mac OS installations do not have LC_ALL.

But I just noticed that Carbon GUI programs do not get LC_CTYPE in their environment either.

So maybe cwstring needs to be fixed for Carbon GUI Mac OS X programs.

What I see in the environment is
__CF_USER_TEXT_ENCODING=0x1F5:0x0:0x0

I think Carbon apps should override DefaultSytemCodePage, because the Carbon interfaces always use UTF-8, they do not care about any environment strings.

Cheers,
Tobias

tobiasgiesen wrote on Mon, 04 Apr 2016:

If none of the environment variables related to code pages are set, FPC falls back to UTF-8 for (a.o.) OS X.

So maybe cwstring needs to be fixed for Carbon GUI Mac OS X programs.

If you get ASCII, it means that one of the LC_ALL, LC_CTYPE and/or LANG environment variables is set to a setting that corresponds to ASCII (such as "C"), or set to a value that is not recognised as or translatable into a Windows code page number.

On OS X, unlike on Windows, there is no inherent difference between "GUI" (be it Carbon, Cocoa, or --most likely-- a mixture of the two) and "non-GUI" applications. You can have command line applications linking to a Carbon framework to deal with aliases, and a GUI application calling into libutil to open a pseudo tty.

The above environment variable is also unrelated to Carbon, but comes from CoreFoundation. 0x1F5 is the hexadecimal value of your user ID. At least one of the 0x0's indeed refers to the default/ansi encoding of CoreFoundation, but it's definitely not the value you want to use. It's the value of the MacRoman text encoding.

That said, FPC 3.1.1 also contains an OS X/iOS-specific widestring manager unit that you can use instead of cwstring (iosxwstr), and that one will always default to UTF-8 (because the "ansi" code page of CoreFoundation only makes sense from a classic Mac backward compatibility standpoint, which we don't have to care about because we don't have a legacy code base that depends on this default setting -- if someone would want to port code that depends on this to FPC, they would have to set this themselves).

Jonas

Jonas

Not necessarily: if you are dealing with UTF8/UnicodeString and other codepages, it is quite likely, and preferred, that you have unit cwstring included. Michael.

Question about this: If I use cthreads, cwstring, and load my own memory manager, what should the order in the uses clause be? I know if they are loaded/unloaded in the wrong order you can get SIGSEGVs during shutdown.

From my experience cthreads needs to be first. So should it be:

cthreads, my_memory_manager, cwstring

or

cthreads, cwstring, my_memory_manager?

Thanks,

-SG

FWIW: i keep seeing the argument "on Mac" but never has the OS on that Mac been mentioned... AFAIK there is more than one OS for Mac or at least more than one version of the OS... it is possible that the default has been changed plus there's whatever was selected for the language during the installation... this really should be clarified for your Mac and its OS...

Hello,

Jonas wrote

If you get ASCII, it means that one of the LC_ALL, LC_CTYPE and/or LANG environment variables is set to a setting that corresponds to ASCII (such as "C"), or set to a value that is not recognised as or translatable into a Windows code page number.

That was it - only when running under Lazarus via debugger, my project has LANG=C.

I wonder where LANG=C comes from?

Note that it is a very old project. A newly created project file does not have LANG=C in the environment.

Cheers,
Tobias

tobiasgiesen wrote on Wed, 06 Apr 2016:

Then it's probably from the info.plist of that old project.

Jonas

Check Run / Run Parameters / Environment.

Note that it is a very old project. A newly created project file does not have LANG=C in the environment.

Mattias