Hi Experts,

There has been a long winding discussion on this in the "German Lazarus Forum" and I have been very dissatisfied with the result.

Maybe this already has been discussed in one of the "Unicode" threads here, but I did not follow all of them down to the latest twig and leaf. So I start a new thread hoping for a more comprehensive result.

When using UTF8String I found that if s is an UTF8String containing "ö2", length(s) is 3 and s[3] is "2". Obviously, UTF8Strings content is counted regarding the 8 bit sub-codes and not the "visible" characters. While I don't like this "un-String-like" behavior at all, I am aware that this is by design to guarantee a decent speed.

But happily we don't need to use UTF8Strings to handle Unicode, as we do have WideStrings, which suffer from this queer behavior only when we try to store extremely strange characters (Unicode > $FFFF) using "surrogate pairs". I feel that I am very unlikely to ever need to do this.

So I did some tests with WideStrings and found strange things with them, too. While some of them are Lazarus issues, one quite obviously is introduced by the compiler.

When I want to simply assign a constant text "ö2" to a WideString I would think that I just write s := 'ö2'; . But I found that this does not work, but that it creates a WideString of length 3 that contains the three 8-Bit subcodes of the utf8-coded string "ö2", zero-extended to 16 Bits, each in one WideChar element. For me this is very surprising and incompatible to the same code (s := 'ö2'; ) used in a Turbo-Delphi program.

Obviously - other than Turbo-Delphi that uses ANSIString here - a constant string gets UTF8String as it's intermediate type. This might be a useful definition, but if that is done this way why does an assignment WideString := UTF8String inot implicitly call UTF8Decode as a type conversion ? In my example it calls fpc_ansistr_to_widestr instead, just as if the UTF8String would be an ANSIString.

Is there some compiler setting to change this ?

-Michael

Hello Michael,

Wednesday, October 22, 2008, 3:36:22 PM, you wrote:

[...]
MS> When using UTF8String I found that if s is an UTF8String containing MS> "ö2", length(s) is 3 and s[3] is "2". Obviously, UTF8Strings content is [...]

If anybody say another thing "UTF8String" is just an alias for "ansistring" so they are exactly the same thing, but with different name which in my case I'm using to be clear in code where things are utf-8 encoded.

Michael Schnell wrote

Hi Experts,

[.....]

I am not an expert, but here is what I believe to know:

This is the result of 2 (hidden) "features":

AFAIK the compiler reads the source as non-utf8 (latin or some 8 bit encoding). This leads to other things too, like identifiers cannot contain utf8.

The String within the quotes is a byte sequence to the compiler. And the compiler does not know it to be utf8. From your description I take it the compiler does translate those 3 "8bit chars" into some 16bit chars (correctness of this translation based on the 8bit source encoding is another question)

Lazarus uses UTF8 for everything, it will save your string as utf8. If Your string was kept as ansistring, the compiler would treat it as bytes, and pass it through, so any code wanting to see the utf8 would be fine.

You can try and tell Lazarus to save you file as latin1. As long as all you strings fit into latin1, this may work; IF and only if the compiler will translate the latin1 into correct Widechars.

It will not work for anything not in utf8. AFAIK Lazarus currently doesn't save in ucs2 (or any 16 bit encoding). But even if Lazarus did, since the compiler wants 8bit encoding, your whole source would be broken.

Not much help, I know. Maybe some one else does have more ideas / knowledge.

I'm nevertheless going to point you to a thread which was started Saturday and concluded today.

http://lists.freepascal.org/lists/fpc-pascal/2008-October/018967.html

Please read the entire thread, and if you have more question afterwards, then ask them.

Jonas

I do know that in the current implementation "UTF8String" is just an alias for "ansistring". Of course this is why the compiler can't do any special tricks when assigning an utf8 coded value to a WideString.

So for a decent handling of UTF8, it is necessary to implement UTF8String as a basic type the compiler is aware of.

While the constant assignment problem is something we seem to see in any use of FPC, when doing Lazarus, the problem is even a lot more prominent, as the new LCL has been changed to use UTF8 instead of ANSI in it's user interface. Thus using any LCL function in a WideString based project uses the wrong (ANSI) default type conversion (-> no error message but wrong behavior as soon as - and only if - Unicode characters

$7F are used.) .

(I have no idea why the Lazarus people decided to use utf8 instead of plain WideString for Unicode support. At least half the Lazarus projects are on Windows and same does have a WideString API so another type conversion is necessary in within the LCL, I don't know how the GTK API works.)

-Michael

AFAIK the compiler reads the source as non-utf8 (latin or some 8 bit encoding). This leads to other things too, like identifiers cannot contain utf8.

This was discussed in the German Lazarus Forum. Here I got a funny result: when I right-click the Lazarus-Code-Editor I see that the file coding is set to utf8. There is no option for UCS2 or UFT16. But when I take a look at the source code file with a hex editor I see that it is a word-coding and starts with a BOM $FFFE, which (AFAIK) means UCS2-coded file.

Should the compiler not take care of this automatically, thus creating the same result independently of the code the source file is saved in (of course regarding the current code page it it's ANSI instead of one of the Unicode flavors (UTF8, UCS2, UTF16, UCS4).

(Disregarding the coding of the source file) I do know what happens and why the result I see is produced by the compiler, but I feel that it is highly undesirable that the compiler is not aware of the type UTF8String and thus can't do a decent conversion when assigning it form and to a WideString.

You can try and tell Lazarus to save you file as latin1. As long as all you strings fit into latin1, this may work; IF and only if the compiler will translate the latin1 into correct Widechars.

I'll try this, but this does not solve the underlying problem. And of course it is highly desirable to use Lazarus in a decent Unicode mode.

It will not work for anything not in utf8. AFAIK Lazarus currently doesn't save in ucs2 (or any 16 bit encoding).

It does in my test. But nobody seems to know how/why. Nonetheless, the compiler should be aware of the coding the source is stored in (it does have a correct BOM in my test) and act appropriately.

But even if Lazarus did, since the compiler wants 8bit encoding, your whole source would be broken.

In my test the compiler does compile the UCS2 coded file correctly (seemingly as designed: the text constants get an "internal" type UTF8String, thus they are somehow converted UCS2->UTF8). Funny: the lpr file is stored as ASCII (might of course be either of ANSI, Latin1 or UTF8).

Not much help, I know. Maybe some one else does have more ideas / knowledge.

I in fact don't need help, but want to discuss what I think is as real problem - if non an error.

Inserting the string type conversions manually makes my test project work just fine. But this is a real PITA, WideString Projects ported from Delphi don't work, and new users who try to use WideStrings (IMHO _the_ decent method to do Unicode programming) will be taken aback as the of course do expect that using WideStrings in their code would "just work".

-Michael

Please read the entire thread, and if you have more question afterwards, then ask them.

In fact I don't have questions, but in this regard the way the compiler (in Lazarus with default settings) works is very dissatisfying. IMHO the only cure is to make the compiler aware of the UTF8Type, so that a conversion UTF8 from and to Widestring can be done correctly and not (incorrectly) erroneously assuming ANSIString instead and calling the conversion function for same.

I suppose this might solve the constant assignment on the fly, but in fact I feel that the compiler should generate a WideString constant at compile time instead of calling a conversion function at run time.

-Michael

Op Thu, 23 Oct 2008, schreef Michael Schnell:

I suppose this might solve the constant assignment on the fly, but in fact I feel that the compiler should generate a WideString constant at compile time instead of calling a conversion function at run time.

Then you don't understand it yet, I think. If the compiler knows your source file is UTF-8 (by BOM or directive), the compiler generates a widestring constant and no conversion function is called when assigning to a widestring.

However, if you assing this constant to an utf8string, the compiler does a wide->ansi conversion, which is done according to the system code page, as the compiler does not know the difference between an ansistring and an utf8string. In this case you would need to utf8decode your widestring constant to get in in your ansistring in UTF-8 encoding.

Daniël

Then you don't understand it yet, I think.

May be....

If the compiler knows your source file is UTF-8 (by BOM or directive), the compiler generates a widestring constant and no conversion function is called when assigning to a widestring.

In my test the source code is not UTF8 but UCS2 and does have a correct BOM. The compiler does work correctly with that and (it seems to be designed that way) converts the string constant source UCS2->UTF8, supposedly temporarily assigning a type like UTF8String to the constant. As the compiler is not aware of the type UTDF8String and errineously identifies it with ANSIString, now the resulting code interprets the constant as ANSIString and the conversion to WideString results in a false text.

I am aware of what is happening here, but it in fact is not the way a decent system should work.

A decent system should be able to do the necessary conversions automatically:

var
  ws1, ws2: widestring;
  us1, us2: utf8string;
begin
 ..
 ws1 := 'ö2';
 us1 := 'ü3';
 ws2 := us1;
 us2 := ws1;
 memo1.lines.add(inttostr(length(w1));  // should show 2
 memo1.lines.add(inttostr(length(w2));  // should show 2
 memo1.lines.add(inttostr(length(u1));  // should show 3 (even if  I
would like 2, but counting in subcodes is much quicker)
 memo1.lines.add(inttostr(length(u2));  // should show 3
 memo1.lines.add(ws1);  // should show ö2
 memo1.lines.add(us1);  // should show ö2
 memo1.lines.add(ws1);  // should show ü3
 memo1.lines.add(us1);  // should show ü3
end;

This should hold independently of the code the source code is stored in (ANSI: no BOM and no other directive, UZF8: BOM=?, UCS2: BOM=FFFF, UTF8: BOM=?, UCS4: BOM=?)

-Michael

Op Thu, 23 Oct 2008, schreef Michael Schnell:

No no, a string with unicode characters is interpreted by the compiler as widestring constant, never as UTF-8 ansistring constant. If it does otherwise, the compiler probably does not interpret your source code as Unicode.

The issue might be the UCS-2 encoding of your source, perhaps try to feed the compiler UTF-8, I didn't even know the compiler accepts UCS-2, it may not work correctly.

Daniël

The issue might be the UCS-2 encoding of your source, perhaps try to feed the compiler UTF-8, I didn't even know the compiler accepts UCS-2, it may not work correctly.

The compiler definitively eats no ucs-2 encoded sources.

A decent system should be able to do the necessary conversions automatically:

This is a simplified view which ignores the resource wasting of this apporoach not visible in the academical example below. The conversion utf-8<->utf-16 is a very expensive operation and the compiler has to insert it all over the place and people would cry about the performance of their programs.

Maybe you pinpointed the cause of the constant assigning issue I see here. There is no selectable option in Lazarus to use UCS-2 coded source files, it seems to be set to utf-8 on default and I never changed it. Nonetheless my source file is stored as clean UTF-2 including correct BOM. So maybe the compiler is fooled by Lazarus in some way.

But even if this in fact is a Lazarus- and not an FPC- problem, not decently converting UTF8String <-> WideString seems an essential issue, nonetheless, even if this problem in many cases is triggered by the LCL using UTF-8 for everything.

-Michael

As has been said before: the compiler itself simply does not support UCS-2. Regardless of any BOM, compiler setting or Lazarus setting, it will not understand it.

Jonas

The compiler definitively eats no ucs-2 encoded sources.

I did check several times: My source file looks like this when I open it with Ultra-Edit and tell to show it in Hex:
FF FE 75 00 6E 0069 00 74 00 20 00 55 00 6E 00 ..u.n.i.t. .U.n.

Now I created a Delphi program and read the file with TFileStream.

Now I found a utf-8 coded information without a BOM.

So Windows seems to play some nasty tricks on us. Supposedly FPC reads the file in a similar way as TFileStream and thus the compiler in fact sees utf8.

All this is really nasty stuff !!!

-Michael

Op Thu, 23 Oct 2008, schreef Michael Schnell:

The compiler uses blockread and performs no such conversions.

Daniël

As has been said before: the compiler itself simply does not support UCS-2. Regardless of any BOM, compiler setting or Lazarus setting, it will not understand it.

See ,y other post in this thread: Windows XP seems to play some tricks on us here so that Ultraedit sees the UCS2 coded file while the compiler gets fed with utf8.

Don't think that I understand why/how this happens.

-Michael

The conversion utf-8<->utf-16 is a very expensive operation and the compiler has to insert it all over the place and people would cry about the performance of their programs.

Of course I do agree.

If you want to care about performance you need to know what to do: Either use WideString "all over the place" and beware of the LCL API, or use UTF8String all over the place.

But if you use UTF8String you need to be aware that you can't do simple and totally normal things like s := copy(s, 3); to get the first three characters of a string. Really finding the first three characters of a string is an interesting and time consuming task with utf8 ;) .

That is why I feel that it would be a lot better if the LCL would use a WideString API.

-Michael

If you want widestring, then maybe mseide is a better option for you.

Vincent

If you want widestring, then maybe mseide is a better option for you.

Again I do know this, and I in fact don't have a project that needs Unicode. But the cause why I started this thread is to help making Lazarus / FPC even more useful.

-Michael

This is also a simplified view.
- firstly, which real world (!) task really requires to execute an operation like this, mostly it's something like copy(s,pos(...),...);
- secondly, a properly coded utf-16 application shouldn't do this either: it doesn't handle surrogates properly and e.g. umlauts can be encoded in all utf flavours as two "chars": base letter plus the umlaut (the two dots).

Ultraedit might fool you here. Id edits either ansi or usc2. If you have a utf8 encoded file, it will show the contents in hex as being ucs2

Marc

utf-16 application shouldn't do this either: it doesn't handle surrogates properly

Right you are. For me WideString is UCS2 and not UTF16, as I regard it as a sequence of WideChar so that the Unicode user code can be done using WideChar and WideString. WideChar only has 16 Bits. So this restrict us to Unicode Characters < $FFFF.

I doubt that I ever will need to use Unicode Characters > $FFFF, but of course there _are_ other projects.

-Michael

In our previous episode, Florian Klaempfl said:

More importantly, most of such routines will be implicitely tied to a certain language or language group already.

The idea that UCS2 simply expands the character range, and the rest stays the same is naieve.

Ultraedit might fool you here. Id edits either ansi or usc2. If you have a utf8 encoded file, it will show the contents in hex as being ucs2

That might be. But it would even virtually insert a BOPM ?!?!?!? Why should it do this when using the hex editor ?

-Michael

I doubt that you will never need to support decomposed characters (such as ä being encoded as basically "a¨"). It's not that uncommon.

Jonas

More importantly, most of such routines will be implicitely tied to a certain language or language group already.

Which kind of UCS2 based function do you think are tied to a language(group) ?

-Michael

Bidi stuff? You are aware of the fact that unicode strings can contain e.g. bidi markers?

Since it converts the UTF8 file internally to UCS2 on read before editing.

Marc

Bidi stuff? You are aware of the fact that unicode strings can contain e.g. bidi markers?

Sorry, never heard of bidi :(

-Michael

One should normalize unicode text before processing. If normalized to fully composed form there will be no problems with UCS2 single character processing in Western Europe. The GUI kit should return fully composed characters when ever possible to simplify the users life.

Martin

http://www.unicode.org/reports/tr9/

Since it converts the UTF8 file internally to UCS2 on read before editing.

Seems really silly to me.

But the file length really indicated that it's utf8 coded and when looking at the file with WinCommander's hex viewer it's utf-8. So I suppose that you are right and the nasty trick is Ultraedit's.

-Michael

I doubt that you will never need to support decomposed characters (such as ä being encoded as basically "a¨"). It's not that uncommon.

This is the nasty old stuff Unicode should be useful to get rid of....

-Michael

Bidirectional text. Much more important than the hypothetical codepoints above the BMP. MSEgui does not support bidi BTW, too difficult. ;-)

Martin

Michael Schnell wrote

> Since it converts the UTF8 file internally to UCS2 on read before > editing. Seems really silly to me.

No it's not. This way you have internally only to support 2 editors. One with bytechars and one with wordchars (ignoring surrogates and other stuff)

But the file length really indicated that it's utf8 coded and when looking at the file with WinCommander's hex viewer it's utf-8. So I suppose that you are right and the nasty trick is Ultraedit's.

Yes, since auto conversion by the OS i find very unlikely

(yes I once tripped over this with ultraedit too)

Marc

http://www.unicode.org/reports/tr9/

Thanks. I see. (In fact I even did do embedded software for a display that can show Hebrew text. But this was with ANSI code.)

-Michael

Hello Michael,

you wrote

> More importantly, most of such routines will be implicitely tied to a > certain language or language group already. >

MS> Which kind of UCS2 based function do you think are tied to a MS> language(group) ?

UpperCase, LowerCase, CapitalCase, WordBreak, ParagraphBreak, ... almost all have some language exceptions.

Op Thu, 23 Oct 2008, schreef JoshyFun:

Don't overexagerate, this is true with plain ASCII as well. Non-English software exists already for over 5 decades and nothing has stopped us to write code that performs the functions you name.

Example: In Dutch uppercase characters generally do not get tremas: Daniël becomes DANIEL. Should an uppercase routine worry? No, this is a spelling convention, the correct uppercase of ë is Ë, we should not confuse spelling with uppercasing.

If someone wants to write language spelling convention specific uppercase routines he is free, FPC's runtime library should simply return the correct uppercase of the characters of a string and so should software working with lower/uppercase characters not bother in most circumstances.

Daniël

Hello Daniël,

Thursday, October 23, 2008, 5:34:59 PM, you wrote:

DM> Don't overexagerate, this is true with plain ASCII as well. Non-English DM> software exists already for over 5 decades and nothing has stopped us to DM> write code that performs the functions you name.

I'm not overexagerating, the question:

MS> Which kind of UCS2 based function do you think are tied to a MS> language(group) ?

needs this answer.

DM> Example: In Dutch uppercase characters generally do not get tremas: DM> Daniël becomes DANIEL. Should an uppercase routine worry? No, this is a DM> spelling convention, the correct uppercase of ë is Ë, we should not DM> confuse spelling with uppercasing.

Example also, in spanish "sólo" is different than "SOLO" and meaning is different ( alone only ). This does not means that I and others will not accept a general case routines in most cases and if a proper working is needed anybody can write its own routines, but this does not change the fact that most unicode routines must have language tailoring to be conformant.

DM> If someone wants to write language spelling convention specific uppercase DM> routines he is free, FPC's runtime library should simply return the DM> correct uppercase of the characters of a string and so should software DM> working with lower/uppercase characters not bother in most circumstances.

I'm not asking for it (I have already written it for my needs, but it is a bit incomplete for a good donation), I'm only answering a precise question.

Op Thu, 23 Oct 2008, schreef JoshyFun:

You indeed answered the question, but my point is the answer could just as well have been "none", since many software is correctly functioning with non language specific implementations of these procedures. Wether or not language specific code has to be used depends on the desired functionality of the software. I.e. a Word processor will use language specific codes, a Unix command line text processing utility uses language independend code.

Daniël

DM> Example: In Dutch uppercase characters generally do not get tremas: Daniël becomes DANIEL. Should an uppercase routine worry? No, this is a spelling convention, the correct uppercase of ë is Ë, we should not confuse spelling with uppercasing.

No. This is not a spelling convention. It is a rule dictated by the language the word is written in.

If the word "Daniël" is Dutch, then its uppercase is:

UpperCase("Daniël", langDutch) --> "DANIEL"

Fine.

Yet, if we don"t know what lang it is written, then the uppercase is:

UpperCase("Daniël", langUndefined) --> "DANIËL"

Now.. as I don't know Dutch at all, I wonder what the LowerCase transforms would be for the same uppercased word, "DANIEL"

LowerCase("DANIEL", langDutch) --> "daniel"

or,

LowerCase("DANIEL", langDutch) --> "daniël"

or both?

If both, how do you pick the correct one?

Example also, in spanish "sólo" is different than "SOLO" and meaning

is different ( alone only ).

Yes, it is impretative that we know the language of the word is in, so that

UpperCase("sólo", langSpanish) --> "SÓLO"
UpperCase("solo", langSpanish) --> "SOLO"

Otherwise, we may end up altering the meaning of the text.

UpperCase(), LowerCase() should not alter the meaning of the text.

This would be a crime in any other context.

Hello listmember,

Thursday, October 23, 2008, 11:58:51 PM, you wrote:

l> Yes, it is impretative that we know the language of the word is in, so that l> UpperCase("sólo", langSpanish) --> "SÓLO"
l> UpperCase("solo", langSpanish) --> "SOLO"
l> Otherwise, we may end up altering the meaning of the text. l> UpperCase(), LowerCase() should not alter the meaning of the text.

In the other hand non tailored functions have a high precision in most common languages, so default case is ok for almost all needs, except the word processing ones (including precise spelling, wordbreak, etc.).

I agree with Daniël on this one. Simplify. ë --> Ë always

If you need something which takes into consideration the language then
build another routine with more parameters.

Felipe Monteiro de Carvalho wrote

I agree with Daniël on this one. Simplify. ë --> Ë always If you need something which takes into consideration the language then build another routine with more parameters.

It's not that simple.

How would you uppercase this piece of string

"In Dutch uppercase characters generally do not get tremas: Daniël becomes DANIEL."

correctly unless you knew the substring "Daniël" is in Dutch and while the rest is in English

UpperCase, LowerCase, CapitalCase, WordBreak, ParagraphBreak, ... almost all have some language exceptions.

I don't doubt that you are right here, but I don't think that there is any support for this in the RTL. So it seems to be a lot less relevant than general Unicode handling.

So I thing we first should have decent Unicode support (e.g. assigning string constants to WideStrings correctly independent from the code the source file is stored in <maybe this is a Lazarus bug and FPC can't help it as it gets a called erroneously> ) correct automatic conversions when assigning an UTFString to a WideString and vice-versa.

-Michael

Felipe Monteiro de Carvalho wrote

I agree with Daniël on this one. Simplify. ë --> Ë always If you need something which takes into consideration the language then build another routine with more parameters.

UpperCase and LowerCase are mapped to OS routines which do take into account the current locale (at least on *nix). So unless those routines always do ë -> Ë regardless of the locale, you will not get this behaviour.

Jonas

Hello Jonas,

Friday, October 24, 2008, 10:27:23 AM, you wrote:

Felipe Monteiro de Carvalho wrote

> I agree with Daniël on this one. Simplify. ë --> Ë always > > If you need something which takes into consideration the language then > build another routine with more parameters.

JM> UpperCase and LowerCase are mapped to OS routines which do take into JM> account the current locale (at least on *nix). So unless those JM> routines always do ë -> Ë regardless of the locale, you will not get JM> this behaviour.

Last time I cheked it on Windows no UpperCase is performed for WideString for codepoints > 127, maybe it has been changed recently (1 month). And yes default case conversion for "ë" is "Ë" as default conversions are right in around 99% cases in most common languages, only a few exceptions are available. Sort engine is much more complicated and much more language tied (To be conforman with each language) but default case always do a reasonable result for most cases.

Last time I cheked it on Windows no UpperCase is performed for WideString for codepoints > 127, maybe it has been changed recently (1 month).

With which software ?

In my tests, If I create the WideString correctly (which is needed to be done with an explicit call to a conversion from utf8String), uppercase works for the German Umlauts äöü..
With Turbo Delphi it easily works, as WideStrings are correctly converted from ANSIStrings automatically.

-Michael

They are in FPC as well, at least if your ansistrings contain ansi- encoded strings. I think you are mixing Lazarus and FPC in the above (because Lazarus puts utf-8 encoded stuff in ansistrings).

Jonas

They are in FPC as well, at least if your ansistrings contain ansi-encoded strings. I think you are mixing Lazarus and FPC in the above (because Lazarus puts utf-8 encoded stuff in ansistrings).

Sorry, I indeed forgot to write "using Lazarus", here Unicode constants seem to be not WideStrings (as seemingly with "pure" FPC) but UFT8Strings and this causes the problem.

-Michael