[....moved the conversation from Lazarus mailing list....]

Bart wrote

Yes, i see. I have those files on Suse too. Not sure what package is responsible. An rpm -q locale (or locales) reveals nothing.

I'm pretty sure rpm must have a way to query it by giving the file and asking what package it belongs to.

I was under the impression that actually the LC_xxx files in /usr/lib/locales/ determined the locale setting?

I think the /usr/share/i18n/locales/* files are simply text based versions of those files - I have not confirmed this though, on speculating.

See that "grouping" differs. (I have not extensively checked other LC_xxx files)

I have tried that test on my side using en_GB, en_ZA and af_ZA and the "grouping" is the same in either text or LC_xxx files. Maybe your text files and LC_xxx files are differing versions?

Either way, parsing the /usr/share/i18n/locales/* should give much better locale results than the current FPC implementation under Unix systems. If you don't use the libc locale unit, everything is hard-coded to en_US which is not so nice.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

In order to know what is the active local in Unix/Linux you should check the environment variables.
If it was not defined or it stand on "C", then en_US is inplace.

Ido

ik wrote

In order to know what is the active local in Unix/Linux you should check the environment variables. If it was not defined or it stand on "C", then en_US is inplace.

Currently FPC doesn't populate the locale variable on unix systems (if you don't compile your application with the libc related units).

$> locale
LANG=en_GB.UTF-8
LANGUAGE=en_GB:en
LC_CTYPE="en_GB.UTF-8"
LC_NUMERIC="en_GB.UTF-8"
LC_TIME=en_DK.UTF-8
LC_COLLATE="en_GB.UTF-8"
LC_MONETARY="en_GB.UTF-8"
LC_MESSAGES="en_GB.UTF-8"
LC_PAPER="en_GB.UTF-8"
LC_NAME="en_GB.UTF-8"
LC_ADDRESS="en_GB.UTF-8"
LC_TELEPHONE="en_GB.UTF-8"
LC_MEASUREMENT="en_GB.UTF-8"
LC_IDENTIFICATION="en_GB.UTF-8"
LC_ALL=

Yet if I create a simple console application the date formats etc are all en_US. I want to write a new unit that doesn't rely on libc (seeing that it isn't available on all UNIX-style platforms) that will parse the correct locale text files (base on LC_xxx environment variables) and populate the locale variables in the RTL.

It's on my todo list, but I have a few more things to finish before I can start this project.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

In our previous episode, Graeme Geldenhuys said:

Yet if I create a simple console application the date formats etc are all en_US. I want to write a new unit that doesn't rely on libc (seeing that it isn't available on all UNIX-style platforms)

libc what and where? Note that clocale depends on iconv, not (strictly) libc, though in some system iconv is bundled into the libc library.

The locale itself is not just ascii based files, there are compiled files, so you will require to re-implement the way of reading the data inside the compiled files.
If you wish to make rules on your own, well it's way too much information to add to RTL.
For example here in Israel our first work day in the week is Sunday and not Monday, so you will need to work with it. Also our date format is dd/mm/yyyy and not mm/dd/yyyy or yyyy/mm/dd
etc.. Currency separator is comma, the actual names of things are in Hebrew, so for each locale you will require to re-implement all of this rules. That's why using the libc functions is better.

Or am I missing here something in what you are saying ?

Ido

Marco van de Voort wrote

Note that clocale depends on iconv, not (strictly) libc, though in some system iconv is bundled into the libc library.

That's what I meant. :)

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

ik wrote

The locale itself is not just ascii based files, there are compiled files, so you will require to re-implement the way of reading the data inside the compiled files.

I'm not talking about the "compiled binary files", I am talking about the text based locale files. On my Linux Ubuntu 7.10 system they are include in the 'locales' package and are install in the following directory location:
/usr/share/i18n/locales/*

eg:
usr/share/i18n/locales/wo_SN
/usr/share/i18n/locales/xh_ZA
/usr/share/i18n/locales/yi_US
/usr/share/i18n/locales/zh_CN
/usr/share/i18n/locales/zh_HK
/usr/share/i18n/locales/zh_SG
/usr/share/i18n/locales/zh_TW
/usr/share/i18n/locales/zu_ZA

For example here in Israel our first work day in the week is Sunday and not Monday, so you will need to work with it. Also our date format is dd/mm/yyyy and not mm/dd/yyyy or yyyy/mm/dd etc.. Currency separator is comma, the actual names of things are in

Here is the details of the /usr/share/i18n/locales/he_IL file:

----------------[ he_IL ]-----------------------
<.....snip......>
LC_MONETARY
int_curr_symbol "<U0049><U004C><U0053><U0020>"
currency_symbol "<U05E9><U05D7>"
mon_decimal_point "<U002E>"
mon_thousands_sep "<U002C>"
mon_grouping 3;3
positive_sign ""
negative_sign "<U002D>"
int_frac_digits 2
frac_digits 2
p_cs_precedes 1
p_sep_by_space 1
n_cs_precedes 1
n_sep_by_space 1
p_sign_posn 2
n_sign_posn 2
END LC_MONETARY

<.....snip....>

LC_TIME
<...snip....>
d_fmt "<U0025><U0064><U002F><U0025><U006D><U002F><U0025><U0079>" t_fmt "<U0025><U0048><U003A><U0025><U004D><U003A><U0025><U0053>" first_weekday 1
END LC_TIME

<....snip....>
----------------------[ end ]---------------

For example: South Africa's date format is also dd/mm/yyyy and is defined as follows:

% Date representation to be referenced by the "%x" field descriptor - % "%d/%m/%Y", day/month/year as decimal numbers (01/01/2000). d_fmt "<U0025><U0064><U002F><U0025><U006D><U002F><U0025><U0059>"

Hebrew, so for each locale you will require to re-implement all of this rules. That's why using the libc functions is better.

All the locale information seems to be in those text files (date, time, paper size, day;week;month names, measurements, names, addresses etc.). It's just a matter of parsing them and populating the locale variables in FPC. No external programs or libraries are required.

As per fpGUI Toolkit's design rules: I hate 3rd party requirements! :-)

BTW: Does FPC's locale variables cover all the locale categories? For example:

category "en_ZA:2003";LC_IDENTIFICATION
category "en_ZA:2000";LC_CTYPE
category "en_ZA:2000";LC_COLLATE
category "en_ZA:2003";LC_TIME
category "en_ZA:2000";LC_NUMERIC
category "en_ZA:2000";LC_MONETARY
category "en_ZA:2003";LC_MESSAGES
category "en_ZA:2000";LC_PAPER
category "en_ZA:2000";LC_MEASUREMENT
category "en_ZA:2003";LC_NAME
category "en_ZA:2003";LC_ADDRESS
category "en_ZA:2003";LC_TELEPHONE

If not, can we extend the FPC locale variable to include salutation (names), telephone, measurement, paper sizes etc..

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

In that case I think we better start a 'locale' unit. I don't want to burden SysUtils with even more stuff.

Michael.

Would that then mean we could fix issues like Russian locales that use unicode characters which do not fit into a Char size (as is used in the current local variables)? Or would that be a totally different can of worms.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

In our previous episode, Graeme Geldenhuys said:

I think that would be a different can of worms. Format et al are ansistring, not widestring/unicodestring. It doesn't know about unicode, only the local (non mbcs) charset.

Format has a widestring flavour?

In our previous episode, Florian Klaempfl said:

True. Missed that. I quickly looked, and noticed two things:

- Does it (still) work, since it seems to include an includefile with functions as implementation that return a "char"?
- The tformatsettings is still ansistring. However if the default locale is UTF-8 this might translate to unicode?

I noticed that I had more direectories in /usr/lib/locale (echh with it's compiles LC_xxx files) than I have /usr/share/118n/locales directory.

For instance I don't have any xx_XX.utf8 dir in
/usr/share/118n/locales, but my LANG setting is nl_NL.UTF-8

Bart

In our previous episode, Bart said:

I've some in /usr/share/local and /usr/local/share/locale

It could be that the ones in /usr/local are older, they use a single level (xx) country designation instead of the threelevel (xx_yy.zz) ISO one.

2009/3/19 Bart :

the files in /usr/lib/locale are the compiled locale definition files (using localdef). i've only got one file there: locale-archive. these files are used by the libc locale functions (like nl_langinfo).

henry

2009/3/19 Marco van de Voort :

the ones in /usr/share/locale are .mo files for the different programs (on debian, at least). i don't have any files under /usr/local/share/locale/, but the directory structure is there.

henry

On the rules how to decide which file to process.

First check for LC_ALL environmentvariable, if it is set it overrules any LC_xxx env.var. set
If no LC_ALL check for env.var. LC_xxx
If no LC_xxx check for env.var LANG
(If nothing there, defualt to US system settings)

The env.var found has the following format:

To get a fallback do (in this order)
1. remove codeset
2. remove normalised codeset (I have no Idea what that means)
3. remove territory
4. remove modifier.
(fallback method found at http://ftp.gnu.org/gnu/glibc/glibc-2.5.tar.bz2)

However I found that the locale program (/usr/bin/locale) does not exactly behave like this.

My $LANG = nl_NL.UTF-8
The nl_NL.UTF-8 directory (in /usr/lib/locale) does not exist and it falls back on nl_NL.utf8

Any comments appreciated.

Bart

In our previous episode, Henry Vermaak said:

I should have mentioned that it is FreeBSD. /usr/local is probably the same, writable by ports. I have .mo files in /usr/share/locale/<code>/LC_MESSAGES but also other LC_ files with locale info are there.

I don't have any of the other dirs mentioned.

2009/3/20 Marco van de Voort :

where does localedef put the compiled files on bsd? localedef is part of the single unix spec, so we should probably parse the compiled files (like libc), since this will be _much_ faster.

henry

In our previous episode, Henry Vermaak said:

Does the single unix spec define the exact format for these files? Since otherwise they might vary between the 3/4 supported unices (solaris, OS X/FreeBSD and Linux) (OS X and FreeBSD sometimes share details)

since this will be _much_ faster.

Personally I don't see the use of going this way. Why not simply use iconv?

2009/3/20 Marco van de Voort :

Does the single unix spec define the exact format for these files? Since otherwise they might vary between the 3/4 supported unices (solaris, OS X/FreeBSD and Linux) (OS X and FreeBSD sometimes share details)

you are right, according to sus website the format of the created output (of localedef) is unspecified.

Personally I don't see the use of going this way. Why not simply use iconv?

but that's just for char set conversion, how would we get the locale data (like time/date format, etc)? i guess we have no choice but to parse the definition files, then.

henry

In our previous episode, Henry Vermaak said:

Correct, it is apparantly libc itself, not iconv, but the point is if clocale can, why not abstract it that way.

Because the textmode versions might not be installed on all systems, the compiled versions might be OS specific, and if they ever change for linux distributions you have another problem at your hand.

And IMHO it is not crucial enough to try to bypass libc.

2009/3/20 Marco van de Voort :

i agree. my opinion has always been that since it's a libc invention in the first place , libc will always be available on systems with this locale information, so it's a lot easier just linking to libc and using nl_langinfo to set up the global variables in fpc.

henry

In our previous episode, Henry Vermaak said:

That's what clocale already does. Note that the way how this happens is not straight forward, since the exact interpretation of what libc returns and which fields are filled differ. See clocale.

So please start with inventorizing what clocale doesn't do, and see if this can be amended some other way before embarking on great adventures into the unknown, because those are never _that_ simple. Specially when done multiplatform and then maintained over time (hence: multiversion).

2009/3/20 Marco van de Voort :

i know, i've seen it before.

i think you misunderstand me, i've looked at this thing ages ago and come to the same conclusion. i'm just trying to convince the people that scream "dependency" that it's not worth going through the effort. it would really be reinventing _someone else's_ wheel.

henry

In our previous episode, Henry Vermaak said:

> multiplatform and then maintained over time (hence: multiversion). i think you misunderstand me

Seems I have.

i've looked at this thing ages ago and come to the same conclusion. i'm just trying to convince the people that scream "dependency" that it's not worth going through the effort. it would really be reinventing _someone else's_ wheel.

Well, a bit of reinvention is not always a bad thing, but only if it is _really_ necessary.

Note that a bulky "read externally" alternative won't be enabled by default in FPC anyway, so it doesn't even alleviate the "need to include clocale" issue. The point is if in this case it is worth the trouble if most people then happily link to X or Lazarus.

I can imagine that e.g. somebody doing point of sales systems or so or has mixed libc/uclibc systems wants to avoid it, but IMHO that is a specialist solution and doesn't belong in the greater FPC distribution.

Note that a bulky "read externally" alternative won't be enabled by default in FPC anyway, so it doesn't even alleviate the "need to include clocale" issue. The point is if in this case it is worth the trouble if most people then happily link to X or Lazarus.

Personally I would like to see the formatsettings being localised by default (by SysUtils), since on Windows it is (Delphi compatibility) and I'd expect it ot be the same on all platforms. For this reason a "bulky read external" alternative might be useful if we do not want SysUtils be dependent on libc.

In that case I'd opt for parsing the text-based versions, since the compiled ones are libc dependent (their format changes). Anyone who links his app to libc (like a standard Lazarus app) can the hapilly include clocale (because the text-based parsing might not be 100% accurate).

Bart

Exactly. It is currently very annoying that my Windows apps have correct locale settings and the exact same application under Linux doesn't. To resolve this, I had to build a config screen into my application where the user can specify correct formatting for various things.

In that case I'd opt for parsing the text-based versions, since the compiled ones are libc dependent (their format changes).

Seeing that correct locale information is totally missing on anything but Windows, I don't see why we can't enable "correct" locale information on systems we can support via parsing text files for now. And incrementally add support for other platforms like FreeBSD etc as we figure out what to do - without linking to libc.

My alternative is to build locale information into fpGUI's translation files. But that would only solve my problems (any fpGUI based apps) and not anybody else's.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

In our previous episode, Bart said:

It's really doubtful it would be done that way. It just adds a possible reason for breakage to (in practice) each binary. The timezone stuff didn't really make me happy. Since it is only a global (and not per unit) decision, Delphi compatibility doesn't say much, since project-wide information is not Delphi compatible in the first place.

A plugin is the more logical route, for the ones that want to remain libc free, but want to risk maintainance problems.

In that case I'd opt for parsing the text-based versions, since the compiled ones are libc dependent (their format changes).

Are they guaranteed default installed on most OSes ? It's pretty hard to parse something that is not there. Also start inventorizing all possible solutions on all possible distro's and OSes.

Anyone who links his app to libc (like a standard Lazarus app) can the hapilly include clocale (because the text-based parsing might not be 100% accurate).

But has an unneeded size penalty (and breakage, since it is always run!).

In our previous episode, Graeme Geldenhuys said:

Well, if you are so strung up on orthogonality, maybe we should move the windows detection to clocale also :_)

Just include clocale.

I don't see why we can't enable "correct" locale information on systems we can support via parsing text files for now.

Make a libc-free plugin unit, and it can be added as package. I see no problem in that, as long as it is suitably multiplatform.

And incrementally add support for other platforms like FreeBSD etc as we figure out what to do - without linking to libc.

No "other platforms" as afterthought please!

2009/3/20 Bart :

you can't do this automatically, since there will be a performance penalty and you're not even sure that the locale definitions are installed.

on debian (for example), the locale definitions are bundled up in the "locales" package, which depends on glibc. i'm willing to bet that you won't find these definitions bundled without localedef, since that's the only program that uses them.

henry

Marco van de Voort wrote

Well, if you are so strung up on orthogonality, maybe we should move the windows detection to clocale also :_)

Well it's simple for me really. I moved away from Lazarus LCL simply because of inconsistencies between platforms. Hence the reason I use fpGUI Toolkit. I would prefer FPC to be more consistent between platforms compared to LCL.

Just include clocale.

Part of using fpGUI Toolkit is to greatly minimise library dependencies (compared to other GUI toolkits like GTK, QT, LCL etc.). This allows fpGUI based apps to run on a lot more "bare boned" systems and platforms (new and old).

Make a libc-free plugin unit, and it can be added as package. I see no problem in that, as long as it is suitably multiplatform.

Yes, this is how I was going to start it off. Simply include the unit in your project and locale files are parse and variables populated.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

2009/3/20 Marco van de Voort :

Are they guaranteed default installed on most OSes ? It's pretty hard to parse something that is not there. Also start inventorizing all possible solutions on all possible distro's and OSes.

no, none of my embedded systems have the definition files. also, they depend on glibc under debian.

exactly.

henry

2009/3/20 Graeme Geldenhuys :

but fpgui depends on x, right? and x depends on libc? you might as well include clocales by default under an ifdef (like lazarus does with cthreads).

henry

Marco van de Voort wrote

A plugin is the more logical route, for the ones that want to remain libc free, but want to risk maintainance problems.

Like an fplocale unit, that users can include if they do not want to use clocale?

Are they guaranteed default installed on most OSes ?

Probably not...

... It's pretty hard to parse something that is not there.

If it's not there, there's nothing to parse, so the code would exit immediately at that point. Nothing would get broken (more than it is now): formatsettings would remain the same as before trying to localise them (as they are now).

Bart

In our previous episode, Bart said:
[ Charset ISO-8859-1 unsupported, converting... ]

Yes, I don't see a problem with that, if the quality is suitable. (read: the code quality is good and multiplatform).

I won't accept a linux own package, see
http://wiki.freepascal.org/libc_unit#But_I_only_want....

for a rationale.

It is not a problem for a plugin unit (though it makes it less generically appliable, and more a specialism).

I was talking about the default situation, where such empty attempts are not acceptable, because it has the potential to totally break a FPC release.

OK, I can understand that.
However, AFAIK formatsettings on Windows are already adjusted for locale settings in SysUtils, so an fplocale would (in it's basic form) not apply to Windows...

I could help writing the Linux (only x86 32-bit) part of it, I only have acces to my own (almost ancient) Linux and WinME system for development.

Bart

Henry Vermaak wrote

but fpgui depends on x, right? and x depends on libc? you might as well include clocales by default under an ifdef (like lazarus does with cthreads).

fpGUI depends on libx11 for X11 based systems. Also on embedded ARM systems (tested by you).

But last time I tested libc was not available for FreeBSD. Hence the reason I have code as follows (which only includes libc when the OS is Linux on cpu386).

getpwuid(...)
getgrgid(...)

did not work under FreeBSD or DesktopBSD.

==========================
implementation

uses
baseunix,

  // Graeme: temporary. libc is not available for FreeBSD.
  {$if defined(linux) and defined(cpu386)}libc,{$endif}
  ...

[.....snip.....]
{$if defined(linux) and defined(cpu386)}
function TfpgFileListImpl.GetUserName(uid: integer): string;
var
  p: PPasswd;
begin
  p := getpwuid(uid);
  if p  nil then
    result := p^.pw_name
  else
    result := '';
end;
{$else}
// Still need to find an alternative for FreeBSD as we can't use the libc unit.
function TfpgFileListImpl.GetUserName(uid: integer): string;
begin
  result := IntToStr(uid);
end;
{$endif}
{$if defined(linux) and defined(cpu386)}
function TfpgFileListImpl.GetGroupName(gid: integer): string;
var
  p: PGroup;
begin
  p := getgrgid(gid);
  if p  nil then
    result := p^.gr_name;
end;
{$else}
// Still need to find an alternative for FreeBSD as we can't use the libc unit.
function TfpgFileListImpl.GetGroupName(gid: integer): string;
begin
  result := IntToStr(gid);
end;
{$endif}
==========================

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

OK, maybe I am confusing myself - between the libc unit and the libc library as documented in:
http://wiki.freepascal.org/libc_unit#But_I_only_want..

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

In our previous episode, Bart said:

I could help writing the Linux (only x86 32-bit) part of it, I only have acces to my own (almost ancient) Linux and WinME system for development.

It was mostly a hint at the other unices, since it would indeed be *nix only. VMs get you a long way, and you probably can get some Mac users to test for you (e.g. by asking on IRC)

In our previous episode, Graeme Geldenhuys said:

(note that it is still better to use general units if they exist instead of declararing your own, those will in time become hopelessly ifdeffed, and it is easier for e.g. Beos/Haiku managers to check the packages/ tree than your local source)

Well, just for the fun of it, I started on an fplocale.pp unit. But since I only have linux and windows available (and windows is already localized by means of sysutils) it applies omly to linux now. It is not complet yet for at least following reasons:
- I don't know if the location of (text versions) is always /usr/share/i18n/locales/
- EscapeChar not implemented yet (just using '/' as default)

If not I need possible other locations to search (I'm not gonna search the whole /usr/* tree to find it, this will slow down things considerably)

For now it parses all that clocale unit does (LC_TIME, LC_MONETARY, LC_NUMERIC), and on my system has same results, but we can add parsing other LC_xxx categories, the template is there.

But has an unneeded size penalty

So far approx. 220 kb size penalty.

To extend it's use I need input/help from users with other systems that provide locale specific settings:
- FreeBSD
- MacOS classic
- MacOS X
- others?

Bart