Hi everybody,

I know we had so many discussions on how to implement Unicode support in FPC in the past. From what i remember, lots was based on "lets see what CodeGear does with D2009".

So now that D2009 is out, is there any further working being done on Unicode support in FPC? Is anybody working on it at the moment? If so, is there something I can help test? Anybody know what CodeGear did with Unicode enabled locale strings like ThousandSeparator etc...? I remember FPC has a major issue with locale information and Unicode support.

Also, if FPC is not going to follow Delphi's implementation to the letter, could somebody summarize what has already been decided for FPC+Unicode?

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

I found that the current FPC does have Unicode support, but there are some problems.

- WideStrings work fine with Unicode UCS-2 but they (of course) have similar issues as UTF8-Strings when surrogate codes are used (which is rarely necessary in Europe and America).

- FPC does not have a dedicated type "UTF8String", but the type defined as "UTF8String" is just the same as ANSIString and thus the compiler can't decide which is meant by the programmer and can't create the appropriate code when it's necessary to distinguish between them (e.g when it automatically should converting between locale-coded ANSIString, UTF8String and WideString)

- by design (for speed sake), UTF8String (and WideString when surrogate codes are used) count in subcodes and not in Unicode-Characters, so the behavior is "unexpected" when doing things like s[i], pos(s), copy(), delete(), ... There are not _slow_ functions that do the "expected" versions of s[i], pos(s), copy(), delete(), ... (I've yet to find out how I can print just the first character of an UTF8String :)

- there is no decent "character" type for UTF8 or UTF16 coded Characters (WideChar (UCS2 code) works if no surrogate codes are used.)

- there are different option on how the compiler expects the coding of the source file. Seemingly if it detects it to be UTF8 coded and a certain (otherwise correct) option is set, even "s := 'hallo äöü'; " does not work correctly as expected if s is a WideString. (Lazarus with default settings suffers from this problem).

-Michael

Michael Schnell wrote

- there are different option on how the compiler expects the coding of the source file. Seemingly if it detects it to be UTF8 coded

The compiler only sets the encoding of the source to UTF-8 if the file identifies itself as "I am UTF-8 encoded" (by starting with an UTF-8 BOM). The compiler does not guess about the encoding in any way.

and a certain (otherwise correct) option is set,

Which option?

even "s := 'hallo äöü'; " does not work correctly as expected if s is a WideString. (Lazarus with default settings suffers from this problem).

Jonas

Michael Schnell wrote

I found that the current FPC does have Unicode support, but there are some problems.

I am going to give it another try, maybe it helps somebody.

Create a new lazarus project, drop a memo, button and edit on a form and add the lclproc unit. Create a button on click handler and add the following code:

procedure TForm1.Button1Click(Sender: TObject);
var
   w: widestring;
   i: integer;
begin
   w := UTF8Decode('hallo äöü');
   Edit1.Caption := UTF8Encode(w);
   Memo1.Clear;
   for i := 1 to UTF8Length(Edit1.Caption) do
     Memo1.Lines.Add(UTF8Copy(Edit1.Caption, i,1));
end;

IMHO, this is working fine.

Vincent

Note that if the file has been saved using an UTF-8 BOM, then the compiler will at compile time create a widestring containing the UTF-16 version of 'hallo äöü'. If you then pass this to a function expecting an ansistring (such as UTF8Decode), then the widestring manager will be used to decode that string and this decoded string will be passed to UTF8Decode. So then you'll pass an ansi-encoded string to UTF8Decode rather than an UTF-8-encoded string (unless ansi = utf-8 for the current execution).

It seems much more advisable to me to save the file with an UTF-8 BOM, or even better to add {$encoding utf-8} (and/or to pass -Fcutf-8 to the compiler) and then just use

Edit1.Caption := UTF8Encode('hallo äöü');

Jonas

Jonas Maebe wrote

It seems much more advisable to me to save the file with an UTF-8 BOM, or even better to add {$encoding utf-8}

Well, {$codepage utf-8}

Jonas

Michael Schnell wrote

, ... There are not _slow_ functions that do the "expected" versions of s[i], pos(s), copy(), delete(), ... (I've yet to find out how I can print just the first character of an UTF8String :)

Lazarus has a set of utf-8 ready routines, using utf-8 inside of a ansistring.

Yes.
Keep in mind that they work in unicode code points. Composed characters are treated as several units. For example umlaute can be 2 characters (3 bytes). Same problem for UTF-16 with widestrings. We still need a normalize function.

Mattias

Graeme Geldenhuys wrote

I know we had so many discussions on how to implement Unicode support in FPC in the past. From what i remember, lots was based on "lets see what CodeGear does with D2009".

OK, so here goes again yet another discussion... :-)

What I meant is, does Delphi 2009 solve all these issue you guys have just mentioned?

* Unicode source code?
* Copy, Pos etc functions?
* Nomalization?
* Does Vincent's example works as follows in D2009:

procedure TForm1.Button1Click(Sender: TObject);
var
 i: integer;
begin
 Edit1.Caption := 'hallo äöü';
 Memo1.Clear;
 for i := 1 to Length(Edit1.Caption) do
   Memo1.Lines.Add(Copy(Edit1.Caption, i,1));
end;

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Yes

* Copy, Pos etc functions?

No. That functions return code units not code points. They introduced new functions to count and to find code points.

See (including comments) http://www.jacobthurman.com/?p=30

Luiz

Which option?

I don't remember right now. It's the default option in Lazarus ;) (This has already been discussed in another thread.).

Lazarus seems to need to set this option because the LCL API is strictly UTF8 (and the UTF8String [ =ANSIString] ) type used otherwise would not get correct initialization from a string constant in the source code, as without this option, string constants seem to be correct WideStrings which <without manual conversion> are incorrectly converted to UTF8String as the compiler mistakes UTF8String for ANSIString).

-Michael

IMHO, this is working fine.

(Thanks for pointing out that there in fact _are_ (slow) function that can manually called to do this with UTF8Strings.)

I don't doubt that it is working fine, but the usual Pascal programmer does not "expect" that (s)he manually needs to call a special function to extract a character from a string ( (s)he is used to just do s[i]) and that (s)he needs to do something other than just "length()" to find out the count of characters in a string. (S)He "expects" that the compiler can handle this as it does have the type UTF8String that tells it about the coding of the information. The compiler instead just ignores this information. (Other than C, Pascal is more largely supposed to use the operant type for defining the kind of an operation. E.g. in C you do "&" for "bitwise and", and "&&" for "boolean and", while with Pascal you always do "and", while the type (integer vs boolean) decides what kind of operation is to be done. Similarly, you do "+" for integer and for String with completely different operations. That is why I think you are allowed to suppose that the compiler knows how to differently handle ANIString and UTF8String. )

I am aware that there _are_ (speed-) reasons that suggest not to ´work with character-counted (instead of subcode-counted) unicode strings. But I would rather work with UCS2 codes WideStrings (trading memory for speed) in that case.

-Michael

Edit1.Caption := UTF8Encode('hallo äöü');

Grrrrrrr, how ugly !

No "old school" Delphi user will understand/accept that you can't just
do "Edit1.Caption := 'hallo äöü';"

-Michael

Lazarus has a set of utf-8 ready routines, using utf-8 inside of a ansistring.

I see. But it's really ugly that you need to use those instead of just writing clean old school code and have the compiler care for the nasty details.

-Michael

OK, so here goes again yet another discussion... :-)

No wonder, as the current state is working, but rather disappointing :). (No idea if D2009 is different / better: this seems to be the cause of the new thread.)

-Michael

2008/11/11 Michael Schnell :

I agree... When I think Unicode support I think the following should be valid and handled correctly:

var
  s: string;
begin
   s := 'hallo äöü';
   Edit1.Caption := 'hallo äöü';
   Edit2.Caption := s;
   ....
end;

Surely this is allowed and works correctly under D2009, otherwise I really misunderstood Unicode support in D2009.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Michael Schnell wrote

needs to do something other than just "length()" to find out the count of characters in a string. From memory, the Delphi and FPC documentation says that Length()

returns the number of bytes, NOT the number of characters....

As for the rest of you comments, I agree. Object Pascal is a beautiful language because it does type handling for you - like your example of '+' for Integer or String types. In the same way I would hope that FPC can handle the String type seamlessly for UTF-16 or UTF-8 - whichever encoding the FPC developers decide String type should be.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

See (including comments) http://www.jacobthurman.com/?p=30

So it seems that the Type "String" in D2009 in fact is "WideString" and same _does_ use surrogate pairs. This asks for even more unexpected behavior that with FPC, with String seemingly still being ANSIString ;).

-Michael

Yes. In D2009 String is UTF16String and Char is WideChar, sizeof(Char)=2. I personally do not like this solution.

Petr

See (including comments) http://www.jacobthurman.com/?p=30

IMHO it's a bad decision to have the standard unicode string (be it WideString or UTF8String) functionality redefined to "Code units" (subcodes) instead of "code points" (characters). I feel it would have been better to have the old notation (s[i], length(s), pos(s), copy(), delete(), ...) work like expected and introduce new functions for the work on subcodes. This would allow for creating the same code as it does with the actual new stuff, while preserving the compatibility to old school coding style and making porting software a lot easier. You _can_ additionally modify the code if you want to optimize the unicode-enabled version of your software for speed.

-Michael

Yes. In D2009 String is UTF16String and Char is WideChar, sizeof(Char)=2. I personally do not like this solution.

Same here,

IMHO FPC could do this in and "Delphi 20089 string compatibility" mode. But it should be configurable to use other ways (e.g. (a) String = WideString but have it counted in characters ("code points") and offer additional functions for speedy subcode ("code units") counting behavior or (b) String = UTF8String, counted in characters ("code points") and offer additional functions for speedy subcode ("code units") counting behavior), or (c) String = ANSIString old school behavior, or (d) String = UCS2 coded WideString.

In any case IMHO the compiler should know the difference between the types UTF8String and locale-depending ANSIString (and maybe between the WideString types UTF16String and UCS2String), so that conversion can be done automatically.

-Michael

We can implement a D2009 like solution and break a lot of old code :)

Having unicode support in any way is not free. You've to rewrite your code somehow.

Surely this is allowed and works correctly under D2009, otherwise I really misunderstood Unicode support in D2009.

In D2009, "String" is WideString, and the VCL API is done with this (Wide)String. So this of course works. With Lazarus things are more complex, as they need to support a lot of environments for the LCL. Even small embedded environments need to be taken into account, so generally using WideString is not a good idea.

I feel that this is why FPC should be configurable to have "String" be either of ANSIString, UTF8String or WideString, and WideString to be configurable to be either of UTF16String or UCS2String, while of course all these types always do exist with their proper names and conversion between them is automatically done as needed.

Moreover, IMHO, it should be configurable if in "D2009 compatible mode", with s[i], length(s), pos(), copy, delete(), ..., Strings are counted in subcodes (fast behavior) or in "whatever mode" they are counted in characters ("friendly old school behavior"), Of course appropriate functions (overloaded for all types) need to exist to do the "friendly" and the "fast" functionality an all modes.

-Michael

I feel that this _is_ possible without bad effects to FPC's / Lazarus' multi-platform paradigm (see the other mail in this thread). But of course it's a lot of work to make it happen ;).

-Michael

Op Tue, 11 Nov 2008, schreef Michael Schnell:

There will have full compatibility with old code. It quite likely FPC will have a Win32 platform where string=ansistring and a WinNT platform where string=unicodestring. Other platforms will be decided on a case by case basis, i.e. there is little point in having string=unicodestring on Dos.

Daniël

Of course Lazarus' LCL would be needed to be recompiled according to the way the user wants to handle "String", as calling for conversion with any LCL in and out transfer is not a good idea. Maybe the LCL could define a type "LCLString" that can be set when compiling it.

Internally I suppose the LCL uses WideString to access the Windows API and UTF8String to access GTK, so it depends on the Platform if a conversion is necessary, anyway.

-Michael

You are mixing two things here:
a) you said that

"Seemingly if [FPC] detects it to be UTF8 coded and a certain (otherwise correct) option is set, even "s := 'hallo äöü'; " does not work correctly as expected if s is a WideString"

That is apparently plain wrong and that is what I reacted to.

b) Vincent showed how in Lazarus, which is now fully based on UTF-8, you can work around the lack of built-in UTF-8 support in FPC, and I replied with a safer way (which moreover introduces less overhead -- although it is unproductive to complain about the fact that the function you have to call for this is slow, because if the compiler does this automatically it can easily get even slower without you noticing because all conversions are hidden).

You don't have to convince anyone of the fact that these manual conversions are not an optimal situation, but please keep "existing functionality does not work correctly" (e.g., assigning constant strings to widestrings, which does work correctly under all circumstances) and "feature x is not implemented" (e.g., automatic support for UTF-8 strings at run time) clearly separated.

Jonas

Having unicode support in any way is not free. You've to rewrite your code somehow.

Right, but this should only necessary with the code that explicitly is intended to benefit from unicode features. "Old school" code - using String (= ANSIString in locale-dependent coding) - should just work.

-Michael

We can implement a D2009 like solution and break a lot of old code :)

My impression always was the FPC is supposed to be better than Delphi :) :) :) .

-Michael

Right you are, but IMHO, there should not be an explicit or implicit dependency on platforms in the compile process, but it the user should be allowed to define a set of options as he likes. Of course for any target platform (that might be different from the build platform) certain option setting will produce faster or smaller platform code than others.

-Michael

Op Tue, 11 Nov 2008, schreef Michael Schnell:

Of course we coudl extend the $H option for unicodestrings, so yuou can choose whatever string string is your default. However, the "platform" part in it, depends on the string type used that all libraries have been compiled with. I.e. regardless of your setting, "assign" would accept a ansistring or unicodestring depending on the platform, and this will be mostly dependand on wether the platform has an actual unicode API.

Daniël

Michael Schnell wrote

Moreover, IMHO, it should be configurable if in "D2009 compatible mode", with s[i], length(s), pos(), copy, delete(), ..., Strings are counted in subcodes (fast behavior) or in "whatever mode" they are counted in characters ("friendly old school behavior")

It will at best be "friendly old school behaviour which works most of the time, but which fails as soon as the strings are not completely normalised because then you can have decomposed characters and whatnot" (which in turn easily leads to security holes due to incomplete checks, hard to reproduce bugs and "write once, debug everywhere"-style behaviour).

Jonas

IMHO, all libraries should use the individual string types, so that the compiler knows that it needs to convert "String" to/from whatever type the library API uses if the user uses another string type explicitly or by setting "String" to whatever type.

Moreover the user should have the possibility to recompile a library so that it uses the string type he desires in it's API. Hey we do use open source here !!! Recompiling on a per project base should not be an issue.

-Michael

Sorry, I don't understand. What not normalized behavior needs to be taken into account ?

With "friendly old school behavior" I mean counting in characters (not subcode) with whatever string type. This of course is only sensible if all conversion is done automatically. And of course this might result in really slow code with Unicode string types, that is why the explicitly subcode-based functions need to be provided to the users.

-Michael

Because e.g. on the ext3 file system, you can have two files with the name "ü" in the same directory. One named using the single character "ü" and one named using as the string "u¨" (both in utf-8). If you make the compiler automatically normalise everything, you lose information (and get the security holes etc).

Jonas

Remember that an individual code point does not nessacerally represent what a user would consider a character. Indeed one character may be representable in more than one way (either as a precomposed character or a sequence of base character and combining diacritic). And even if we ignore combining diacritics the number of console positions a string takes is not nessacerally equal to the code point either since many CJK characters take two console positions.

Given theese facts code point counts and indexes are not much more usefull than code unit indexes and counts.

And if you need something better than either code point count or code unit count then you have little choice but to pull in an external library. Pulling in an external library with a relatively unstable interface is not something the compiler or RTL should be doing IMO.

I see, but as this is not handled decently with good old ANSIStrings, anyway, there is not "friendly old school" way that a compiler would be able to offer. In these special cases, the user of course needs to explicitly handle the upgrade of his project to unicode.

OTOH, in this special case, I don't see why the compiler should "normalize" "u¨" to "ü". If the software is supposed to be handling unicode, the unicode string "u¨" should be considered a perfectly legal two-code-point information consisting of a "u" (a single sub-code in UTF-8) and a double-dot (supposedly two subcodes in UTF-8). If the user wants to handle this as a single "ü", he should write appropriate code for that. Any automation on that is dangerous.

-Michael

Note that I was simplifying. It's not actually "u¨", but "u" followed by the code point meaning "put ¨ on top of the preceding character". In other words, there is (all in UTF-8)

a) "ü": "LATIN SMALL LETTER U WITH DIAERESIS", encoded as $C3 $BC b) "ü": "LATIN SMALL LETTER U", encoded as $75, followed by "COMBINING DIAERESIS", which is encoded as $CC $88 c) "u¨": "LATIN SMALL LETTER U", encoded as $75, followed by "DIAERESIS", which is encoded as $C2 $A8

If the user wants to handle this as a single "ü", he should write appropriate code for that. Any automation on that is dangerous.

The character combination actually literally means "ü" in both cases. It's not a decision of a user whether or not it means "ü".

Jonas

Remember that an individual code point does not nessacerally represent what a user would consider a character. ...

Again, there is no compatible handling of this with good old ANSIStrings, anyway, so there is not "friendly old school" way that a compiler would be able to offer. In these special cases, the user of course needs to explicitly handle the upgrade of his project to unicode. If the differences or ambiguities are relevant in the project in question, the user of course needs to write code for that. Only if handling the issue in the same "old school" way as ANSIStrings provide, the user should be able to write the same code as he is used to do.

Given theese facts code point counts and indexes are not much more usefull than code unit indexes and counts.

I never said that "code point counts and indexes are more useful", but forcing the programmer to create much more complex and less straight forward code only because the underlying system now uses unicode, even if he would be perfectly happy with ANSI, does not make much sense to me. If he really needs unicode specifics (or wants to optimize his code for space or time) he of course should be aware that things are complex and he needs to dig deeper.

Do you really want to explain a newcomer that and why he can't any more do

s: String;
s := edit1.text;
case s[1] of
 'a': do_this();
 'ä': do that();
end;

(with String being ANSIString; UTF8String or WideString.).

Wait a minute: UTF8Char in case triggers another problem :) :) :) ...... But at least ANSIString and WideString should work as expected (independent of the source code being code as ANSI or UTF8 !)

-Michael

a) "ü": "LATIN SMALL LETTER U WITH DIAERESIS", encoded as $C3 $BC b) "ü": "LATIN SMALL LETTER U", encoded as $75, followed by "COMBINING DIAERESIS", which is encoded as $CC $88

I see, but I fail to see the sense of providing two different UTF8 code variants for the same unicode character.

-Michael

Probably because different kinds of string processing can work more efficiently with one or the other encoding. Anyway, why it is the case is moot: the fact is that this is possible (regardless of whether you use UTF-8, UTF-16 or UTF-32) and therefore you have to deal with it when you use unicode.

Jonas

OK,

If this really is two codes for the same unicode character, the "friendly old school" handling function should normalize it. If someone really needs to take the differences into account (like with the case you described), he ought to do the appropriate code (handling subcodes).

-Michael

It was just one example. Look at Peter Green's mail for others. The run time library (with the help of complex routines) can indeed just destroy all data of which it thinks that you do not care about when using "friendly string" handling (because it is not representable in some ISO ansi encoding that the developer is familiar with), but in that case you will probably end up with something that turns out to be useful for only a very limited number of people, and which in the real world comes crashing down horribly all the time. Yet, it will require an enormous amount of implementation work to actually support all this alleged friendliness

If people want to rely on what they are used to in non-unicode environments, then they cannot directly use unicode strings. They'll first have to assign it or typecast it to a non-unicode string and then operate on that string. At least if there's any data loss in that case, the programmer has full responsibility and knows (or should know) the risks.

Jonas

Op Tue, 11 Nov 2008, schreef Michael Schnell:

IMO widestrings with precomposed characters, just like ansistrings, can fullfill the needs of a newcomer. That there exists decomposed characters, surrogates, and more, does not need to be explained in chapter 1 of a programming for beginners book.

Daniël

In response and to support Daniël:

Also, remember unicode is/are a computerlanguage specific specification(s): you may assume that a lot of thought has gone into it to be able to use it with programming languages. That was the design goal. The specification is, alas, rather complex but it contains every bit of information to be able to implement high level handling without breaking properly designed older high level languages. This goes at the expense of lower level string handling (and asm constructs that were introduced just for string handling in the instruction sets of most modern processors!) . I guess that is just what we call progress.

From your writing I understood that the issue is a UTF8 -> 21-bit-unicode decoding issue and has nothing to do with ISO/ANSI (which would render the problem thoroughly unsolvable, not only for the compiler builder but also for the application programmer, who wants to do a unicode aware program.

-Michael

Yep, but with s: WideString the example does not work in unicode aware Lazarus.

-Michael

That is because FPC has no unicode string type yet (as must have been repeated about 20 times by now).

Jonas

Yes, that might be confusing. Therefore I don't recommend to save if with an UTF8-BOM of compile -Fcutf-8

As an extra bonus of not adding the UTF-8 BOM, you don't have to use conversions to assign the UTF8 string in the source, translated by the compiler to a UTF16 string, to an UTF8 encoded ansistring. It saves a conversion at compile time and a conversion at run time.

Edit1.Caption := 'hallo äöü'.

Is there an explicit way to tell the compile not to convert widestring string constants, even if the file contains an UTF-8 BOM? The UTF-8 BOM might be usefull, if you want to edit the file with another text editor.

Vincent

What do you want to say by this ? (Of course the general user is not supposed to do ASM code for string handling.)

With Lazarus even:

s: WideString;
s := 'äüö';

does not work (i.e. result in an utf-16 (or UCS2) coded word array containing the appropriate three unicode characters

and if you create an utf-16 coded WideString s that contains the three unicode characters äöü

caption := s;

does not work either (i.e. the From caption does not show äöü)

We all know why this is the case, but the knowledge does not really help.

The "case" example shows more issues.

-Michael

Op Tue, 11 Nov 2008, schreef Michael Schnell:

We were talking of a world where strings consist of widechars, not about the current Lazarus, weren't we?

Daniël

That is because FPC has no unicode string type yet (as must have been repeated about 20 times by now).

We are not discussing what it has, but what it should have and how this can be done in a way that provides decent performance on all platforms, is easy to use, compatible to D2009 and optimally versatile (see the first message in this thread). This of course only can be provided with user-settable options, that can be set according to the needs of the specific project.

Nobody is complaining that the currant state is like it is but that does not mean that it should stay this way.

-Michael

Your example shows just how accurate Widestring should be interpreted: It just only shows it is a two ore more byte sequence to represend a single character. It doesn't say anything about the content or the use of a specific (meta) encoding like UCS2 or Unicode16. If the discussion is about those two issues, you may have a point. Or two, because those are separate from a computer language point of view. The difference being: 1. compiles and 2. actually works. What I mean to say is: if the specs aren't ready, you'll have a fair chance of being mislead by assuming things that are not there (yet).

And breaks all widestring usage.

Is there an explicit way to tell the compile not to convert widestring string constants, even if the file contains an UTF-8 BOM?

No.

Jonas

We were talking of a world where strings consist of widechars, not about the current Lazarus, weren't we?

I'm not sure. Of course WideStrings and WideChars are easier to be used, as in Europe and America problems with surrogate pairs will seldom arise, but IMHO the user should be enabled to define what kind of string encoding he uses (UTF-8, UTF-16 or UCS2, trading speed against memory). The compiler should do the dirty work of converting automatically.

With that the current Lazarus LCL would be perfectly usable (even if the user decides to use UCS2 WideStrings and thus can use WideChars).

OTOH, regarding an improved Lazarus, IMHO he should be enabled to choose or compile an LCL version with a UTF16 or UCS2 WideStrings API, for improved speed with GUI handling.

-Michael

Breaking *all* widestring usage is maybe a little exagerated, but it gets more complicated, certainly. All widestring usages have to use UTF8Decode. Fortunately, in Lazarus code widestrings are not used that often, I think. At least less often than UTF-8 encoded strings in ansistrings.

Vincent

Your example shows just how accurate Widestring should be interpreted: It just only shows it is a two ore more byte sequence to represend a single character. It doesn't say anything about the content or the use of a specific (meta) encoding like UCS2 or Unicode16.

Of course I did mean that in the each string type (UTF8String, ANSIString, (UTF16)Widestring and (UCS2)WideString, the content is encoded as the native string name requests (of course there right now is no type UTF16String and UCS2String, but with all the current examples (not using surrogate pairs) the bits in same (and in WideString) are the same.)

-Michael

AFAIK no one measured a noticeable speed difference between UTF8/16 when handling GUI.

Mattias

2008/11/11 Michael Schnell :

I believe it's got to do with case conversion, sorting etc... I think Mac OS always uses the non-normalized form and Linux the normalized one. But yes, those combining characters are a pain to handle. I tried in fpGUI Toolkit and still having issues with things like cursor position calculations etc.. :-(

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Michael Schnell wrote

With Lazarus even:

I don't know about Lazarus, but in fpGUI Toolkit the following works just fine.

var
  s1: string
  s2: TfpgString;  // simply an alias to string
begin
  s1 := 'äüö';
  s2 := 'äüö';
  Button.Text := s1;
  WindowTitle := s2;
end;

fpGUI takes care of the conversion for you. It internally uses UTF-8. But manipulating strings etc becomes a pain because you can't use the standard Pos, Length etc functions, you have to use the fpGUI equivalent. Same for things like FileExists() for example. You have to use fpgFileExits() instead.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

I agree with that.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

So I suppose it does not set the FPC option to use UTF8String instead of WideString for non-ASCII string constants. MSEGUI works here, too, because of this.

-Michael

AFAIK no one measured a noticeable speed difference between UTF8/16 when handling GUI.

So I don't understand why the LCL designers for the unicode upgrade decided to use an UTF8 API instead of a WideString API (like MSEGUI does seemingly successfully).

-Michael

Michael Schnell wrote

So I suppose it does not set the FPC option to use UTF8String instead of WideString for non-ASCII string constants. MSEGUI works here, too, because of this.

I set no special options in FPC and I don't use fits perfectly in the standard String type.

That's why I don't understand all the discussion around FPC and Unicode. Why can't we just choose an encoding (for example UTF-8) and use it internally in FPC. Why must FPC support native UTF-16, UTF-8, ANSI etc.... UTF-8 manages backward compatibility perfectly!

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

I set no special options in FPC

Lazarus does.

and I don't use WideString at all. UTF-8 fits perfectly in the standard String type.

Some conversions are correct or seem to be correct in that case.

-Michael

It has been already pointed out several times that lazarus abuses the anstring type to store utf-8 and this breaks several stuff.

It has been already pointed out several times that lazarus abuses the anstring type to store utf-8 and this breaks several stuff.

Of course we do know this.

But as the compiler does not tell ANSIString from UTF8String anyway (to do automatic conversions), what exactly does this mean ?

-Michael

Michael Schnell wrote

So I don't understand why the LCL designers for the unicode upgrade decided to use an UTF8 API instead of a WideString API (like MSEGUI does seemingly successfully).

I can't speak for the Lazarus team, but I can speak for fpGUI Toolkit. It's very little effort moving from ANSI to UTF-8. Plus you can assign any UTF-8 string to a standard AnsiString type. No special coding or types are required. That's one of the huge benefits of UTF-8. Other benefits being that it's already in byte format, no BOM markers required, seemly handling of all Unicode characters because you can't "cheat" with UTF-16 by using UCS-2 as a start (which MSEgui is doing).

All I do know is that only recently did the WideString manager become usable in FPC. Martin had until recently some issues with bugs in the WideString manager.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

I must have missed that thread. Have you got the thread title or estimated date of the thread? For interest sake, I would like to read those discussions.

I'm curious what Lazarus does to abuse ansistring. Ansistring is in byte format. So is utf-8, so I can't see any immediate issues. That's why I would like to read those discussions. :-)

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Lazarus assumes that an ansistring contains always utf-8. This is not generally true.

Graeme Geldenhuys wrote

All I do know is that only recently did the WideString manager become usable in FPC. Martin had until recently some issues with bugs in the WideString manager.

No functional changes have been made to the unix widestring manager since 2.2.2 (except for Florian's changes preparing for unicodestring support). In fact, the previous changes date from December 2007. For the win32 widestring manager it's August 2007.

Jonas

???
The last widestring manager bug I remember was in January 2007 in FPC 2.0.5.

Martin

Martin Schreiber wrote

> ??? The last widestring manager bug I remember was in January 2007 in FPC 2.0.5.

I can't remember the exact details, but I read it a few months back on the MSEgui newsgroup. I remember the bug was widestring related (and at the time you tried to convince me fpGUI should have gone for UTF-16 which I found funny at the time). I can't find the exact posting - I can't search the newsgroup and ain't going to download the whole newsgroup for something unimportant. :-)

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

So could somebody from the core FPC team summarize or give some roadmap as to what is happing or planned for FPC + Unicode support?

These Unicode discussions seem to go round and round and never seems to reach a conclusion. :-(

So some or other roadmap or feature list for FPC on this matter would be nice.

eg:
* One encoding internally (utf8 maybe)?
* A new default string type like Delphi 2009 did?
* Something totally different to D2009?
* Is anybody actually working or Unicode support?
* Could I write some unit tests for something already?

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

- both UTF-8 and UTF-16 has multi character problems and lazarus should support the full unicode (at least it should not be limited by design). That's why arguments that widestrings are easier, because you can use UCS-2 tricks, are irrelevant concerning the LCL.
- we made some tests and didn't found any speed gain/loose concerning LCL.
- Compatibility with existing code (var parameters) (MSEGui does not have this problem)
- widestrings had some bugs at the time and were not reference counted (Martin did a great job over the years to improve that)
- at the time (four years ago) there was no one with enough time and compiler knowledge to extend fpc.
- We had not the manpower to do both

Speaking for myself: I'm glad for the decision for UTF-8. So far I had much more problems with widestrings than with UTF-8 ansistrings.

Of course, I expect that the pressure for D2009 compatibility rises. And if the FPC team decides to create a unicode-able RTL with UTF-16 for windows, then maybe the LCL will be extended/changed to support that. Knowing the FPC devels this can take years or a weekend. ;)

Mattias

Graeme Geldenhuys wrote

So could somebody from the core FPC team summarize or give some roadmap as to what is happing or planned for FPC + Unicode support?

If anyone can, it's Florian, since he has done all the work in this area until now. There is no roadmap document or official planning or so.

Jonas

In our previous episode, Jonas Maebe said:

I summarized some earlier summer discussions here:

http://www.stack.nl/~marcov/unicode.pdf

but I haven't it updated since that date.

Just for information Delphi introduces RawByteString type to handle such case:

http://www.micro-isv.asia/2008/08/using-rawbytestring-effectively/

Luiz

Op Tue, 11 Nov 2008, schreef Luiz Americo Pereira Camara:

The string type to use in non-Unicode environment remains ansistring, which has their system encoding. Rawbytestring is for strings with unknown encoding, mainly usefull for procedures that work on any encoding (i.e. seems usefull for copy, insert, delete).

Daniël

Lazarus assumes that an ansistring contains always utf-8. This is not generally true.

While this might be true, I think it's a consequence of a shortcoming of FPC, which simply identifies the types ANSIString and UTF8String. IMHO (in a future version) it should take care of the encoding of string types.

I suggest that there should be the native string types ANSIString, UFT8String, UCS2String and UTF16String together with the appropriate character types ANSIChar, UFT8Char, UCS2Char and UTF16Char (UTF8Char and UTF16Char in fact being the appropriate strings. The compiler and RTL should take care of any conversion between those (and do the appropriate constant assignment when needed).

Now there should be compiler options to have the use select which type he wants to use for the generic "String" type (all four applicable) and which he want to use for the generic WideString Type (UCS2String and UTF16String applicable). The generic char and WideChar type is assigned appropriately.

Moreover this version should for all native string types provide as well Unicode-character("code point")-counted as submode("code unit")-counted functions and procedures for what we know as s[i], length(s), pos(), copy(), delete(), ...

There should be compiler options (for all native string types) to have the user select which of the two he wants use for the generic s[i], length(s), pos(), copy(), delete(), ... notation.

With this provided, Lazarus would be able to provide whichever API for LCL they want in a decent and highly compatible way (They _should_ allow the user to select if he wants to link in an ANSIString, UTF8String or UCS2String version).

Moreover this would allow for tuning a project for space of for speed according to the platform we want to compile it for (e.g 32 Bit PC or ARM based Cellphone).

-Michael

- widestrings had some bugs at the time and were not reference counted (Martin did a great job over the years to improve that)

Of course this is a serious argument. I feel that any decent string type (but pchar :) ) needs to be reference counting. In my "suggestion" mail 10 minutes ago in this thread I presumed the UTF16String and UCS2String types to be reference counting. Is the infrastructure for this already in place ?

Speaking for myself: I'm glad for the decision for UTF-8. So far I had much more problems with widestrings than with UTF-8 ansistrings.

OK. I fully accept this argument. That is why I vote for having the compiler do decent automatic conversion between all possible string types (even if "lossy" when converting to ANSI). This would allow for any kind of API for LCL. (maybe some overloaded functions need to be provided).

Of course, I expect that the pressure for D2009 compatibility rises.

Of course FPC and Lazarus need to provide a Delphi 2009 compatible mode. I suppose we agree that the D2009 way of string handling is not the best possible (at least not for all users/projects). So I vote for providing the appropriate compiler switches.

-Michael

I' not in that team ;) but I think I did provide a workable suggestion 15 minutes ago in this thread. Of course I have no idea at all if it's possible to implement this with not too much work for the team members :( .

eg: * One encoding internally (utf8 maybe)?

I vote for allowing the user to decide.

* A new default string type like Delphi 2009 did?

I vote for allowing the user to decide.

* Something totally different to D2009?

I vote for allowing the user to decide.

Of course the Lazarus team would need to get involved, too.

-Michael

Unicode support in FPC?

Sorry for jumping in. I am not so much interested in Unicode because I mostly use shortstrings and ansistrings for performance reasons. But may be it is worth to look how the gcc people have solved these issues.

Regards, Bernd.

Op Wed, 12 Nov 2008, schreef Bernd Mueller:

They didn't. C does not support strings. For any string operation a C programmer calls a library function. For Unicode this can be a Win32 API call, but also a libc call, or an iconv call. It's an unportable mess.

Daniël

They didn't.

Perhaps he means C#.

But here, AFAIK, string support is very different to what Pascal programmers are accustomed to. Strings are always constant and can't be modified, They are created once and discarded completely later. Something like s[3] := 'x'; is not portable to C#.

-Michael

Nice thread - as always when unicode handling is brought up :)

There should really be a decision and feature list as soon as possible so the implementation can go on. There is really no need to spend a year on these decisions.

Proper unicode handling strategy/implementation with backward and Delphi compatibility as well as a useable Lazarus debugger (Property watch, CPU window, etc.) should be the #1 priority now.