Hi,

I have a slight issue with a patch made in FPC, and will be included in the 3.0 release.

https://github.com/graemeg/freepascal/commit/aef238f18a04bce621c452fbdcfac46613001e6c

============================================
[src (master)]$ git log aef238f18a04bce6
commit aef238f18a04bce621c452fbdcfac46613001e6c
Author: Jonas Maebe
Date: Thu Dec 11 19:11:11 2014 +0000

* fall back to UTF-8 for darwin in unixcp.GetSystemCodePage if no LC_*/LANG
environment variables are set
* if no LC_*/LANG environment variables are set, let cwstring fall back to
unixcp.GetSystemCodePage
o this is mainly to avoid problems on OS X, where under certain circumstances these variables are not set for GUI programs and as a result we set the system code page to ASCII

git-svn-id:
3ad0048d-3df7-0310-abae-a5850022a9f2
============================================

See the last file changed.... rtl/unix/unixcp.pp
It seems it excludes FreeBSD in that $IF statement. Darwin is after all a FreeBSD fork. This results to cwstrings under FreeBSD to default to ASCII. :-/

----------[ rtl/unix/unixcp.pp ]---------------
function GetSystemCodepage: TSystemCodePage;
var
  p: SizeInt;
  lang: ansistring;
  cp: TSystemCodePage;
begin
  // Get one of non-empty environment variables in the next order:
  // LC_ALL, LC_CTYPE, LANG. Default is UTF-8 or ASCII.
{$if defined(linux) or defined(darwin)}
  Result:=CP_UTF8;
{$else}
  Result:=CP_ASCII;
{$endif linux}
-----------------------------------------------

Any specific reason why FreeBSD is excluded from the $IF statement?

Regards,
- Graeme -

Graeme Geldenhuys wrote

See the last file changed.... rtl/unix/unixcp.pp It seems it excludes FreeBSD in that $IF statement. Darwin is after all a FreeBSD fork.

Darwin's libc is based on FreeBSD's and it shares a few user land utilities, but Darwin is far from a FreeBSD fork.

This results to cwstrings under FreeBSD to default to ASCII. :-/

That is how all Unix platforms are defined to behave: if the LANG or LC_CTYPE environment variable is not set, the "C" locale is what you have to fall back to.

OS X' GUI environment does not follow these conventions, so there we have to use another heuristic. The problem with falling back to ASCII there, is that the OS X kernel interfaces for file system APIs all use UTF-8. This is unlike all other Unix platforms, which don't define any encoding whatsoever for file names, and where all file names are simply arbitrary arrays of bytes that should be interpreted according to the current locale.

If we fall back to ASCII on OS X, then all results from OS file APIs will be converted from UTF-8 to ASCII and you get data loss (due to a difference between system.DefaultFileSystemCodePage, which is UTF-8, and system.DefaultSystemCodePage, which is ASCII). If we fall back to ASCII on another unix platform, both system.DefaultFileSystemCodePage and system.DefaultSystemCodePage will be ASCII and no data loss occurs (not inside the low level file handling routines of the RTL anyway).

That said: I have a very hard time believing that any contemporary Linux or *BSD system would not come preconfigured with an UTF-8 locale for every user, so you should never end up in the fallback situation unless deliberately unset those environment variables yourself.

Jonas

Jonas Maebe wrote

That said: I have a very hard time believing that any contemporary Linux or *BSD system would not come preconfigured with an UTF-8 locale for every user, so you should never end up in the fallback situation unless deliberately unset those environment variables yourself.

Many thanks Jonas for the explanation. I understand what you are saying.

My bad, I should have posted a follow-up message.

My reason for the thread was an issue I found in MSEide where it uses cwstring, and even though my LANG environment variable was set to 'en_GB.UTF-8', it used US_ASCII. Initially I thought the issue was with FPC, but with more investigations and a sample C program I found the problem. MSEgui has a copy of FPC's cwstring unit, but the platform constants were out of date (compared to FPC) and actually a copy of Linux's platform constants were used under FreeBSD. Martin updated his copy of cwstring with new FreeBSD platform constants and the problem is now solved.

Regards,
- Graeme -