Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / lazarus / 2008-08

Any chance of changing the LCL Unicode encoding to UTF-16?

34 messages · 11 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

Luiz Americo Pereira Camara

This is just a question not a request.

In the recent discussion about fpc support to unicode it seems that delphi compatibility is a must for fpc and since delphi will use UTF-16 as the main encoding in the next release, probably fpc will follow it.

So, any chance of changing the LCL Unicode encoding to UTF-16?

I'm asking this because i develop a component (VirtualTreeView) that uses WideString/UTF-16 to support unicode and i'm planning to switch to UTF-8. If LCL change to UTF-16 it will be lost of time.

Luiz

Reply by email

Ivan P. Gan
to Luiz Americo Pereira Camara

Why not use a flag to state the application encoding? then by using utf-8 to utf-16 conversion routines Lazarus could in theory use both
UTF-8 is now one of the most common encoding standards for Non-English websites, it is generally more efficient on bandwidth

Delphi compatibility is a valid point but from what I have loaded on both systems it is clear that Lazarus is not compatible with Delphi The designers have done a great job but the cross platform & wider language support enjoyed by Lazarus comes at a price

Lazarus is far better in several respects

1. Better cross platform graphic support
2. Wider range of processors
3. Wider range of operating systems
4. Fully open source, we can modify the compiler & IDE to improve it much more than is possible with Delphi

Down sides

1. Delphi has better help system
2. Delphi supports packages
3. Delphi has more components & packages available

Luiz Americo Pereira Camara wrote 18 lines
This is just a question not a request. In the recent discussion about fpc support to unicode it seems that delphi compatibility is a must for fpc and since delphi will use UTF-16 as the main encoding in the next release, probably fpc will follow it. So, any chance of changing the LCL Unicode encoding to UTF-16? I'm asking this because i develop a component (VirtualTreeView) that uses WideString/UTF-16 to support unicode and i'm planning to switch to UTF-8. If LCL change to UTF-16 it will be lost of time. Luiz _______________________________________________ Lazarus mailing list http://www.lazarus.freepascal.org/mailman/listinfo/lazarus

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.lazarus.freepascal.org/pipermail/lazarus/attachments/20080802/3d438731/attachment.html

Reply by email

Graeme Geldenhuys
to Luiz Americo Pereira Camara: This is just a question not a request. In the recent discussion about fpc support to…

Hi Luiz,

Your timing couldn't be better!! I was having the same discussion with a user in the fpGUI newsgroups. Should fpGUI change it's internal string encoding from UTF-8 to UTF-16? Like you said... FPC seems to be very pro-Delphi compatible. And from the discussions in the fpc-devel mailing list it seems they will follow Delphi to the UTF-16 string encoding.

So where does that leave the frameworks that rely on UTF-8 as the internal string encoding? Lazarus LCL and fpGUI? Also where does it leave the applications developed with those UTF-8 enabled toolkits? How would such a change suggested by FPC affect all those projects?

Regards,
- Graeme -

Luiz Americo Pereira Camara wrote 18 lines
This is just a question not a request. In the recent discussion about fpc support to unicode it seems that delphi compatibility is a must for fpc and since delphi will use UTF-16 as the main encoding in the next release, probably fpc will follow it. So, any chance of changing the LCL Unicode encoding to UTF-16? I'm asking this because i develop a component (VirtualTreeView) that uses WideString/UTF-16 to support unicode and i'm planning to switch to UTF-8. If LCL change to UTF-16 it will be lost of time. Luiz _______________________________________________ Lazarus mailing list http://www.lazarus.freepascal.org/mailman/listinfo/lazarus
Regards, - Graeme - _______________________________________________ fpGUI - a cross-platform Free Pascal GUI toolkit http://opensoft.homeip.net/fpgui/

Reply by email

Martin Friebe
to Graeme Geldenhuys

IMHO, it would be best if a string knew it's encoding.

AFAIK, at the moment a string is prefixed by a record, containing data such us the amount of allocated mem, and a ref-count. This should include a field encoding. Then you could always request the string in the encoding you need, and it would be converted if needed.

There was another discussion recently, about data aware components and the encoding from the DB. Many applications today may have to deal with input in more than one encoding, and then deal with this data. This would be much easier if the string had a memory for the encoding it has.

This may be a topic for the FPC mailing lists.

Regards
Martin

Graeme Geldenhuys wrote 44 lines
Hi Luiz, Your timing couldn't be better!! I was having the same discussion with a user in the fpGUI newsgroups. Should fpGUI change it's internal string encoding from UTF-8 to UTF-16? Like you said... FPC seems to be very pro-Delphi compatible. And from the discussions in the fpc-devel mailing list it seems they will follow Delphi to the UTF-16 string encoding. So where does that leave the frameworks that rely on UTF-8 as the internal string encoding? Lazarus LCL and fpGUI? Also where does it leave the applications developed with those UTF-8 enabled toolkits? How would such a change suggested by FPC affect all those projects? Regards, - Graeme - On Sat, Aug 2, 2008 at 6:34 PM, Luiz Americo Pereira Camara wrote: > This is just a question not a request. > > In the recent discussion about fpc support to unicode it seems that > delphi compatibility is a must for fpc and since delphi will use UTF-16 > as the main encoding in the next release, probably fpc will follow it. > > So, any chance of changing the LCL Unicode encoding to UTF-16? > > I'm asking this because i develop a component (VirtualTreeView) that > uses WideString/UTF-16 to support unicode and i'm planning to switch to > UTF-8. If LCL change to UTF-16 it will be lost of time. > > Luiz > _______________________________________________ > Lazarus mailing list > > http://www.lazarus.freepascal.org/mailman/listinfo/lazarus > >

Reply by email

Graeme Geldenhuys
to Martin Friebe
Martin Friebe wrote 6 lines
IMHO, it would be best if a string knew it's encoding. AFAIK, at the moment a string is prefixed by a record, containing data such us the amount of allocated mem, and a ref-count. This should include a field encoding. Then you could always request the string in the encoding you need, and it would be converted if needed.

This is exactly what my Russian friend suggested in the fpGUI newsgroups. We already use a fpGUI specific string type (type TfpgString = String). So if we had the encoding option we could simply change that string type definition to TfpgString = String[(utf8)] and continue assuming inside fpGUI that we are working with UTF-8 encoded strings. Well, something like that.

PS:
I know the new Delphi is going to have some feature like that, specifying the encoding. But I don't know the syntax or how FPC plans to handle this.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Reply by email

Marco van de Voort
to Luiz Americo Pereira Camara: This is just a question not a request. In the recent discussion about fpc support to…
Luiz Americo Pereira Camara wrote 5 lines
This is just a question not a request. In the recent discussion about fpc support to unicode it seems that delphi compatibility is a must for fpc and since delphi will use UTF-16 as the main encoding in the next release, probably fpc will follow it.

This is not true. All options are still open, and currently it is highly unlikely that FPC won't be able to support multiple encodings in one binary.

To anybody that wants to start with unicode before that,

Note also that uptake of Tiburon is not known yet. Till now, the 200x versions haven't been that popular, it will be interesting what major features like this unicode support and generics do to that.

So, any chance of changing the LCL Unicode encoding to UTF-16?

That is not sane. I think if you are going to make assumptions, unicodestring being *DEFAULT* utf-8 on Linux and utf-16 on windows (or whatever native encoding the respective platform has) is the more likely outcome.

Maybe Tiburon code can kept working by adding extra {$H+/-} like directives, but this is all pending the Tiburon release, and the increase of knowledge about it. If this works, this means that UTF16-only tiburon code might be mixed with UTF8 Unix LCL code.

I'm asking this because i develop a component (VirtualTreeView) that uses WideString/UTF-16 to support unicode and i'm planning to switch to UTF-8. If LCL change to UTF-16 it will be lost of time.

The best thing to say is the age old truth that people that abstract it to switch either way, hurt themselves less when it goes the other way. IOW try to avoid to make a choice for encoding.

Again, currently the official state is *WAIT*. These are major decisions and shouldn't be made rash or with a lack of information.

Reply by email

Graeme Geldenhuys
to Marco van de Voort
Marco van de Voort wrote
Again, currently the official state is *WAIT*. These are major decisions and shouldn't be made rash or with a lack of information.

Thanks for that information Marco. That is currently what we decided for fpGUI. We are going to continue using UTF-8 for now and simply wait and see what happens after the Tiburon release. I will definitely be joining in the FPC-devel conversations when the new string encoding discussions start. :-)

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Reply by email

Felipe Monteiro de Carvalho
to Marco van de Voort: This is not true. All options are still open, and currently it is highly unlikely that…
Marco van de Voort wrote 6 lines
> So, any chance of changing the LCL Unicode encoding to UTF-16? That is not sane. I think if you are going to make assumptions, unicodestring being *DEFAULT* utf-8 on Linux and utf-16 on windows (or whatever native encoding the respective platform has) is the more likely outcome.

Varying encodings are a very unlikely option for the LCL, because a single encoding is much easier for developers to deal with, and we try to keep a consistent path.

We could teorically at some point discuss changing from UTF-8 to UTF-16, but there are no plans to implement this at the moment. The only advantage of changing is Delphi compatibility, and we started our unicode support before they announced theirs, so they are the ones breaking compatibility. The disadvantage of changing is that we will break compatibility with ourselves, and a lot of developers are relying in our utf-8 support, so a lot of discussion would be necessary before this could happen.

Honestly Delphi has hat a very unstable history in the last years. I think noone is in favor of blindly following them.

Keeping 2 libraries compatible doesn't really look that easy when one of them can change freely. A more realistic solution could be making LCL compilable with Delphi, so that one can use it with both compilers, instead of using LCL with FPC and VCL with Delphi.

Felipe Monteiro de Carvalho

Reply by email

Ivan Gan
to Felipe Monteiro de Carvalho

UTF-8 is the better encoding system to use
It is more compact than UTF-16 & why make applications more bloated than they already are?
I agree with Felipe, a single encoding is better
UTF-8 is also very widely supported on the net with an increasing number of web pages using it
Existing ANSI encoded applications can be upgraded easily using UTF-8, UTF-16 requires a lot more work
Delphi VCL is Windows, not cross platform, Lazarus LCL is a better solution anyhow
Delphi was a great tool & the creators done a fantastic job but it no longer leads the way as it used to
Lazarus is now in my view a better environment to work with & is beginning to take the lead anyhow
Is Delphi simply following Lazarus by introducing Unicode? Using UTF-8 on linux & UTF-16 on windows is a mistake, it would force code changes & loose cross platform abilities which are the real strength of Lazarus
What I need to see when developing is consistency across platforms, anything else costs money & time
I would far rather see time being spent on addressing the weak areas like better Internet support, the INDY component set don't work properly & I haven't found any other set as easy to use
We have effective Unicode support in Lazarus, ok, next problem please, thats my view

Felipe Monteiro de Carvalho wrote 32 lines
On Sun, Aug 3, 2008 at 6:43 AM, Marco van de Voort wrote: >> So, any chance of changing the LCL Unicode encoding to UTF-16? >> > That is not sane. I think if you are going to make assumptions, > unicodestring being *DEFAULT* utf-8 on Linux and utf-16 on windows (or > whatever native encoding the respective platform has) is the more likely > outcome. > Varying encodings are a very unlikely option for the LCL, because a single encoding is much easier for developers to deal with, and we try to keep a consistent path. We could teorically at some point discuss changing from UTF-8 to UTF-16, but there are no plans to implement this at the moment. The only advantage of changing is Delphi compatibility, and we started our unicode support before they announced theirs, so they are the ones breaking compatibility. The disadvantage of changing is that we will break compatibility with ourselves, and a lot of developers are relying in our utf-8 support, so a lot of discussion would be necessary before this could happen. Honestly Delphi has hat a very unstable history in the last years. I think noone is in favor of blindly following them. Keeping 2 libraries compatible doesn't really look that easy when one of them can change freely. A more realistic solution could be making LCL compilable with Delphi, so that one can use it with both compilers, instead of using LCL with FPC and VCL with Delphi.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.lazarus.freepascal.org/pipermail/lazarus/attachments/20080804/b7d33980/attachment.html

Reply by email

Marco van de Voort
to Felipe Monteiro de Carvalho: Varying encodings are a very unlikely option for the LCL, because a single encoding is…
Felipe Monteiro de Carvalho wrote 11 lines
On Sun, Aug 3, 2008 at 6:43 AM, Marco van de Voort wrote: >> So, any chance of changing the LCL Unicode encoding to UTF-16? > > That is not sane. I think if you are going to make assumptions, > unicodestring being *DEFAULT* utf-8 on Linux and utf-16 on windows (or > whatever native encoding the respective platform has) is the more likely > outcome. Varying encodings are a very unlikely option for the LCL, because a single encoding is much easier for developers to deal with, and we try to keep a consistent path.

IMHO it is insane no not work in a native encoding

Show quoted text 5 lines
We could teorically at some point discuss changing from UTF-8 to UTF-16, but there are no plans to implement this at the moment. The only advantage of changing is Delphi compatibility, and we started our unicode support before they announced theirs, so they are the ones breaking compatibility.

They are not compatible to us, so they probably don't consider that breaking. This kind of reasoning is a bit juvenile.

The disadvantage of changing is that we will break compatibility with ourselves, and a lot of developers are relying in our utf-8 support, so a lot of discussion would be necessary before this could happen.

Well, that will break anyhow with full unicode support, since that would change the base string type away from (something emulated with ansistring).

Honestly Delphi has hat a very unstable history in the last years. I think noone is in favor of blindly following them.

Who says anything about blindly following? There is a very big "WAIT till more info" in my message.

Keeping 2 libraries compatible doesn't really look that easy when one of them can change freely.

I do not know what this is about? If you mean delphi libs, there are more than a dozen.

A more realistic solution could be making LCL compilable with Delphi, so that one can use it with both compilers, instead of using LCL with FPC and VCL with Delphi.

Unrealistic, and unclear what this actually solves?

Reply by email

Graeme Geldenhuys
to Felipe Monteiro de Carvalho: Varying encodings are a very unlikely option for the LCL, because a single encoding is…

On Mon, Aug 4, 2008 at 12:33 AM, Felipe Monteiro de Carvalho >> That is not sane. I think if you are going to make assumptions,

Show quoted text 7 lines
> unicodestring being *DEFAULT* utf-8 on Linux and utf-16 on windows (or > whatever native encoding the respective platform has) is the more likely > outcome. Varying encodings are a very unlikely option for the LCL, because a single encoding is much easier for developers to deal with, and we try to keep a consistent path.

I also can't see a dual encoding working out....

Honestly Delphi has hat a very unstable history in the last years. I think noone is in favor of blindly following them.

I second that!

of them can change freely. A more realistic solution could be making LCL compilable with Delphi, so that one can use it with both compilers, instead of using LCL with FPC and VCL with Delphi.

That doesn't seem very realistic either. LCL relies heavily on many FCL features and classes. Delphi doesn't have a FCL!

-
Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Reply by email

Graeme Geldenhuys
to Ivan Gan: UTF-8 is the better encoding system to use It is more compact than UTF-16 & why make…

2008/8/4 Ivan Gan :

UTF-8 is the better encoding system to use It is more compact than UTF-16 & why make applications more bloated than they already are?

Unfortunately the argument isn't that easy! There are many compelling reasons to implement utf-8 or utf-16. It's not a black & white argument.

And please don't read this as I'm against utf-8. On the contrary, I implemented UTF-8 as our internal encoding in fpGUI as well.

Lazarus is now in my view a better environment to work with & is beginning to take the lead anyhow

The Free Pascal Compiler can be placed in the same context. For my line of work, I must agree with our statement. ;-)

We have effective Unicode support in Lazarus, ok, next problem please, thats my view

Again a narrow point of view... It may work fine if you only write English applications. A simple example where FPC (and Lazarus) is failing with unicode support (and no easy solution is available), is in locale information.

For example:
The ThousandSeparater variable is a Char type (1 byte type), yet the Russian character for a thousand separater (U+00A0 character) requires 2 bytes (the utf-8 encoding would be $C2 $A0).

There are a few other such cases with locale information alone. Simply changing the ThousandSeparater type or other locale types is also not an option, because the gets used in many SysUtils functions like FormatDateTime(), FormatFloat() etc...

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Reply by email

Michael Van Canneyt
to Graeme Geldenhuys: On Mon, Aug 4, 2008 at 12:33 AM, Felipe Monteiro de Carvalho >> That is not sane. I think…
Graeme Geldenhuys wrote 11 lines
On Mon, Aug 4, 2008 at 12:33 AM, Felipe Monteiro de Carvalho >> That is not sane. I think if you are going to make assumptions, >> unicodestring being *DEFAULT* utf-8 on Linux and utf-16 on windows (or >> whatever native encoding the respective platform has) is the more likely >> outcome. > > Varying encodings are a very unlikely option for the LCL, because a > single encoding is much easier for developers to deal with, and we try > to keep a consistent path. I also can't see a dual encoding working out....

There is no need for a dual encoding in LCL.

Just define

  TLCLString = String(thelazaruslclcodepage);

(or whatever declaration format will be used) and use that consistently everywhere.

This way, the LCL programmer knows per definition what the codepage is. In his own routines, he's still free to do as he pleases. FPC will take care of the conversion anyway.

But for the FPC RTL, the situation is totally different, and a varying encoding is the only sensible thing to do. The implementation details are not yet sure, but this is the path that will be taken with 99% probability, even if only for Delphi compatibility and to keep all parties on the mailing lists satisfied.

But as Marco said: the WAIT state is the only correct state for the FPC users (and that includes Lazarus/fpGUI/mseIDE)

Regardless of that, if the lazarus/fpGUI devels want to make it to Lazarus/fpGUI 1.0 still during this millennium, I would not recommend switching encoding (again) yet. You must draw the line somewhere.

Michael.

Reply by email

Graeme Geldenhuys
to Michael Van Canneyt
Michael Van Canneyt wrote
Regardless of that, if the lazarus/fpGUI devels want to make it to Lazarus/fpGUI 1.0 still during this millennium, I would not recommend switching encoding (again) yet. You must draw the line somewhere.

I'll be playing the WAIT game. :-) In the mean time I'm starting to prepare fpGUI as much as I can. TfpgString type must be used everywhere instead of String. UTF8Pos(), UTF8xxx() functions need to be renamed to encoding neutral names like fpgPos, fpgInsert etc...

At least that way fpGUI stays neutral in design. And whatever FPC decides to make String, it should have 'relatively' minimal affect on fpGUI and fpGUI based applications. Well, that's in theory of course! ;-)

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Reply by email

Ivan P. Gan
to Graeme Geldenhuys: 2008/8/4 Ivan Gan : Unfortunately the argument isn't that easy! There are many compelling…
Graeme Geldenhuys wrote 14 lines
2008/8/4 Ivan Gan : > UTF-8 is the better encoding system to use > It is more compact than UTF-16 & why make applications more bloated than > they already are? > Unfortunately the argument isn't that easy! There are many compelling reasons to implement utf-8 or utf-16. It's not a black & white argument. And please don't read this as I'm against utf-8. On the contrary, I implemented UTF-8 as our internal encoding in fpGUI as well.

I never suggested that you are against UTF-8, I understand both arguments, on Windows from a performance point of view the use of UTF-16 is likely to prove faster as it is the native encoding on that platform It is however better to have a single encoding format for all platforms that Free pascal & Lazarus support
The use of UTF-8 allows easier upgrading of existing ANSI encoded applications, this backward compatibility is one I consider a strong plus Updating existing components should be easier, though the variant character would have to be taken into account
Like you say its not black & white

Show quoted text 7 lines
> Lazarus is now in my view a better environment to work with & is beginning > to take the lead anyhow > The Free Pascal Compiler can be placed in the same context. For my line of work, I must agree with our statement. ;-)

Lazarus is impressive indeed as is the Free Pascal Compiler

Show quoted text 9 lines
> We have effective Unicode support in Lazarus, ok, next problem please, thats > my view > Again a narrow point of view... It may work fine if you only write English applications. A simple example where FPC (and Lazarus) is failing with unicode support (and no easy solution is available), is in locale information.

Actually I work with Non English, Mixed language content daily, I am aware of many of the difficulties though the Russian character is one I was unaware of
As my work is web based, I find UTF-8 very effective providing a good balance on size & function
Does using UTF-16 correct the Russian separator problem?

Show quoted text 11 lines
For example: The ThousandSeparater variable is a Char type (1 byte type), yet the Russian character for a thousand separater (U+00A0 character) requires 2 bytes (the utf-8 encoding would be $C2 $A0). There are a few other such cases with locale information alone. Simply changing the ThousandSeparater type or other locale types is also not an option, because the gets used in many SysUtils functions like FormatDateTime(), FormatFloat() etc...

I have noticed one common failing on many projects, people tie language to locale, they need to be independent to allow immigrants for example to choose Chinese in the USA or French in Israel where the locale information would follow local encoding for dates etc. A difficult task, even for those of us who work with it frequently Most locale setting systems however tie language to the location which totally fails to take into account immigration, for example, the second most popular language in the state of Texas, USA is Vietnamese

Regards
Ivan

Show quoted text 12 lines
Regards, - Graeme - _______________________________________________ fpGUI - a cross-platform Free Pascal GUI toolkit http://opensoft.homeip.net/fpgui/ _______________________________________________ Lazarus mailing list http://www.lazarus.freepascal.org/mailman/listinfo/lazarus

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.lazarus.freepascal.org/pipermail/lazarus/attachments/20080804/80c499ae/attachment.html

Reply by email

Hess, Philip J
to Ivan P. Gan

"for example, the second most popular language in the state of Texas, USA is Vietnamese"

Please cite your source for this claim. I think the many Spanish speakers in Texas would be scratching their heads over this one.

Thanks.

-Phil

Reply by email

Ivan P. Gan
to Hess, Philip J
Philip J wrote 16 lines
"for example, the second most popular language in the state of Texas, USA is Vietnamese" Please cite your source for this claim. I think the many Spanish speakers in Texas would be scratching their heads over this one. Thanks. -Phil _______________________________________________ Lazarus mailing list http://www.lazarus.freepascal.org/mailman/listinfo/lazarus

CIA World factbook if I remember rightly though I will have to check on it as I use several sources
Its not Spanish anymore, it sure surprised me when I read it

Reply by email

JoshyFun
to Hess, Philip J: "for example, the second most popular language in the state of Texas, USA is Vietnamese"…

Hello Philip,

Monday, August 4, 2008, 10:38:35 PM, you wrote:

HPJ> "for example, the second most popular language in the state of Texas, HPJ> USA is Vietnamese"
HPJ> Please cite your source for this claim. I think the many Spanish HPJ> speakers in Texas would be scratching their heads over this one.

OT: from census.gov
-----8<------------8<---------8<----------
NATIVITY AND LANGUAGE: Sixteen percent of the people living in Texas in 2005 were foreign born. Eighty-four percent were native, including 61 percent who were born in Texas.

Among people at least five years old living in Texas in 2005, 34 percent spoke a language other than those speaking a language other than English at home, 86 percent spoke Spanish and 14 percent spoke some other language; 44 percent reported that they did not speak English "very well."

xref: http://www.census.gov/population/cen2000/phc-t20/tab05.pdf -----8<------------8<---------8<----------

cut and drop to the trash bin.

Best regards, JoshyFun

Reply by email

Graeme Geldenhuys
to Ivan P. Gan: I never suggested that you are against UTF-8, I understand both arguments, on Windows…

2008/8/4 Ivan P. Gan :

It is however better to have a single encoding format for all platforms that Free pascal & Lazarus support

I agree. While researching the issue with unicode I came across a very good article the other day. Unicode was design using need for variable amounts of bytes to present a character. 2 bytes is all that was needed. It proved a better idea that UTF-8 which required 1-4 bytes. Then to there surprise 65000 code points was not enough so UTF-16 ended up have encoding pairs - just the thing they tried to avoid! Oops.

The other compelling reason I think for UTF-8 is that it always represents data in byte form. No need for a byte order mark. Plus many people consider the fact that it is very unlikely to use characters above the BMP so they normally don't bother implementing support for it. So they intend to have UTF-16 support, but in fact only have UCS2. MSEgui is such a case, but at least they only claim UCS2 support which is good in a way (no false advertising). Also because the higher BMP characters are seldom used, the code path for working with those are hardly used and normally not well tested. So many implementation bugs can slip though the cracks. Because UTF-8 works with variable byte data from the start, it's implementation is normally well tested.

I'll try and find that article again and post the URL here... It was interesting reading.

As my work is web based, I find UTF-8 very effective providing a good balance on size & function

And because it's byte based it lends itself well to data transfer and exchange. Probably with the W3C recommends UTF-8 for web pages and XML.

Does using UTF-16 correct the Russian separator problem?

No. Some of the locale variables used in FPC are Char type (AnsiString), so they can only store 1 byte of data. The Russian thousand separator requires 2 bytes.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Reply by email

Ivan P. Gan
to JoshyFun: HPJ> "for example, the second most popular language in the state of Texas, HPJ> USA is…

JoshyFun wrote:
I stand, hanging my head in shame, humbled & corrected I will look for the source of my info & email them cause they was wrong Government census should be fairly accurate
I search for info on languages & as soon as I open my big mouth, it promptly gets my foot shoved in it
At least I get more accurate information
Thanks

Show quoted text 27 lines
Hello Philip, Monday, August 4, 2008, 10:38:35 PM, you wrote: HPJ> "for example, the second most popular language in the state of Texas, HPJ> USA is Vietnamese" HPJ> Please cite your source for this claim. I think the many Spanish HPJ> speakers in Texas would be scratching their heads over this one. OT: from census.gov -----8<------------8<---------8<---------- NATIVITY AND LANGUAGE: Sixteen percent of the people living in Texas in 2005 were foreign born. Eighty-four percent were native, including 61 percent who were born in Texas. Among people at least five years old living in Texas in 2005, 34 percent spoke a language other than those speaking a language other than English at home, 86 percent spoke Spanish and 14 percent spoke some other language; 44 percent reported that they did not speak English "very well." xref: http://www.census.gov/population/cen2000/phc-t20/tab05.pdf -----8<------------8<---------8<---------- cut and drop to the trash bin.

Reply by email

Martin Schreiber
to Graeme Geldenhuys: 2008/8/4 Ivan P. Gan : I agree. While researching the issue with unicode I came across a…
Graeme Geldenhuys wrote 12 lines
2008/8/4 Ivan P. Gan : > It is however better to have a single encoding format for all platforms > that Free pascal & Lazarus support I agree. While researching the issue with unicode I came across a very good article the other day. Unicode was design using UTF-16 at first. No need for variable amounts of bytes to present a character. 2 bytes is all that was needed. It proved a better idea that UTF-8 which required 1-4 bytes. Then to there surprise 65000 code points was not enough so UTF-16 ended up have encoding pairs - just the thing they tried to avoid! Oops.

[...]

Graeme,
I think you should not mix up external and internal character representation of a GUI framework.
For external character representation utf-8 is best for cross platform storage of text in files and to send and receive text by communication channels. MSEgui uses utf-8 to store initialization and user state data and the like and in the MSEifi remote executable framework. utf-8 is also used to store the source code in files in MSEide.
For internal character representation you should first and formost choose an encoding which will be easy to handle by the users. Because for most of the users a codepoint can not be expressed by a single utf-8 byte, they can not use character access by string index in their code and bytelength(utf8string) charactercount.
With UCS2/utf-16 on the other hand *all* current users of MSEgui can store *all* the codepoints they need in a single utf-16 word and therefore can work with the Unicode MSEgui framework as they are used to work with ansistrings. Even our active Chinese MSEide+MSEgui user didn't report any Unicode issue since a long time.
Parsing of utf-16 is much simpler than parsing of utf-8, processing of UCS2/utf-16 in framework core is probably faster than utf-8 because there are less addressing operations for codepoints > 127.
FPC supports transparent conversion widestring system encoding since a long time. For number formatting, filesystem access, string manipulation and the like MSEgui has implemented its own optimized widestring functions. Widestring versions of all Pascal standard string functions (pos, copy...) are supported by FPC. The only missing piece is a reference counted widestring type on Windows.

Martin

Reply by email

Marco van de Voort
to Martin Schreiber
Martin Schreiber wrote
With UCS2/utf-16 on the other hand *all* current users of MSEgui can store *all* the codepoints they need in a single utf-16 word

?? Martin, can you make a formal statement about what MSEGUI now supports? I keep getting the feeling you only support UCS-2.

Reply by email

Graeme Geldenhuys
to Martin Schreiber: [...] I think you should not mix up external and internal character representation of a…
Martin Schreiber wrote
For external character representation utf-8 is best for cross platform storage of text in files and to send and receive text by communication channels. MSEgui uses utf-8 to store initialization and user state data and the like

That knew that, and that's what I said... utf-8 represents it's data in byte form, to it's ideal for streaming. The reason I think W3C chose utf-8 as a recommendations for XML and web pages.

Show quoted text 5 lines
For internal character representation you should first and formost choose an encoding which will be easy to handle by the users. Because for most of the users a codepoint can not be expressed by a single utf-8 byte, they can not use character access by string index in their code and bytelength(utf8string) charactercount.

And do a search on the Internet... over and over you get articles refuting that point. Random character access is very seldom required. If character access is used, it normally coincides with looping/iterating through the string, which utf-8 has no problems with either. Random character access is not the norm in program code, and if so, it's a very small part.

http://weblogs.mozillazine.org/roc/archives/2008/01/string_theory.html

With UCS2/utf-16 on the other hand *all* current users of MSEgui can store *all* the codepoints they need in a single utf-16 word and therefore can work

I'm being cynical I guess, but that sounds just like what the Unicode guys thought with their first draft. 65000 code points would be more than enough for all written languages. That assumption was proven wrong quickly. ;-)

Parsing of utf-16 is much simpler than parsing of utf-8, processing of UCS2/utf-16 in framework core is probably faster than utf-8 because there are less addressing operations for codepoints > 127.

Yes and with utf-16 you need to worry about the BOM marker and endianness. utf-8 you don't.

http://en.wikipedia.org/wiki/UTF-16#Byte_order_encoding_schemes

"The UTF-16 and UCS-2 encoding forms produce a sequence of 16-bit words or code units. These are not directly usable as a byte or octet sequence because the endianness of these words varies according to the computer architecture; either big-endian or little-endian. To account for this choice of endianness each encoding form defines three related encoding schemes: for UTF-16 there are the schemes UTF-16, UTF-16BE, and UTF-16LE, and for UCS-2 there are the schemes UCS-2, UCS-2BE, and UCS-2LE."

So that means you can have three utf-16 encodings!

Widestring versions of all Pascal standard string functions (pos, copy...) are supported by FPC.

Does the widestring version of Copy take into account surrogate pairs? Or would it simply split the bytes? As for Pos, Copy - LCL and fpGUI has their own utf-8 versions. Why would those be any slower than the widestring versions included in FPC?

As for Length()... What should it return? Bytes or Character count? If bytes (which I believe is what the documentation says it must return), then the standard ANSI Length() included in FPC will suffice. If it's character count you want, LCL and fpGUI has their own versions of that.

As far as I understand, the assumption that Length() returns character count is coincidence, simply because ANSI Char = 1 byte.

from Kylix 3 help: "For single-byte and multibyte strings, Length returns the number of bytes used by the string."

I couldn't find FPC documentation on the Length() function.

Like I said... the argument between utf-8 and utf-16 is not clear cut. It's not black & white. I have read so much documentation on both sides of the argument that my brain started to hurt (that seems to happen often lately). ;-)

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Reply by email

Graeme Geldenhuys
to Marco van de Voort: ?? Martin, can you make a formal statement about what MSEGUI now supports? I keep getting…
Marco van de Voort wrote 6 lines
On Tue, Aug 05, 2008 at 08:50:29AM +0200, Martin Schreiber wrote: > With UCS2/utf-16 on the other hand *all* current users of MSEgui can store > *all* the codepoints they need in a single utf-16 word ?? Martin, can you make a formal statement about what MSEGUI now supports? I keep getting the feeling you only support UCS-2.

That's my point! MSEgui states on their website that they only support UCS-2. But because UCS-2 and UTF-16 are similar for the BMP only, may projects (MSEgui excluded) suggest they are UTF-16 compliant, which they are NOT! Even in the above statement from Martin, he makes that mistake... quoting UCS2/utf-16 which makes people assume he supports the full UTF-16, but doesn't.

This I consider a major downfall for utf-16. You never have this issue with UTF-8 because you have to handle multibyte character from the start! The code path for utf-8 will be well tested.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Reply by email

Martin Schreiber
to Marco van de Voort: ?? Martin, can you make a formal statement about what MSEGUI now supports? I keep getting…
Marco van de Voort wrote 7 lines
On Tue, Aug 05, 2008 at 08:50:29AM +0200, Martin Schreiber wrote: > With UCS2/utf-16 on the other hand *all* current users of MSEgui can > store *all* the codepoints they need in a single utf-16 word ?? Martin, can you make a formal statement about what MSEGUI now supports? I keep getting the feeling you only support UCS-2.

MSEgui doesn't care about surrogate pairs at the moment, they are handled as two consecutive (unknown) single characters. If the users need codepoints above the BMP they can store it in widestrings in utf-16 encoding but MSEgui will not show it correctly on the screen. BiDi isn't supported neither BTW. Surrogate pair handling can be implemented in the MSEgui screen and edit functions later if necessary.
My point is that none of the current MSEide+MSEgui users need codepoints above the BMP so I don't want to complicate and slowdown internal stringhandling for a feature nobody needs at the moment.
BiDi is probably more important but difficult to implement for me because I have no knowledge in right to left languages.

Martin

Reply by email

Graeme Geldenhuys
to Marco van de Voort: ?? Martin, can you make a formal statement about what MSEGUI now supports? I keep getting…
Marco van de Voort wrote 6 lines
On Tue, Aug 05, 2008 at 08:50:29AM +0200, Martin Schreiber wrote: > With UCS2/utf-16 on the other hand *all* current users of MSEgui can store > *all* the codepoints they need in a single utf-16 word ?? Martin, can you make a formal statement about what MSEGUI now supports? I keep getting the feeling you only support UCS-2.

I actually asking him that exact question a few days ago.... Here is his official reply.

Earlier message 37 lines
-------- Original Message -------- Subject: Re: msegui and unicode support Date: Wed, 30 Jul 2008 08:05:45 +0100 From: Martin Schreiber Newsgroups: public.mseide-msegui.talk References: <efi2kml61ljp0cy.300720080212 at dxserver1> Graeme Geldenhuys schrieb: > Hi Martin, > > Does msegui still only support UCS2 or have you implemented full UTF-16 > support in later versions? Last time I checked was around msegui 1.2. > MSEgui doesn't care about surrogate pairs. This is no problem as long you don't enter characters in edit widgets where the codepoint doesn't fit into the Unicode basic multilingual plane, nothing which will happen soon I assume... liuzg, our Chinese user, reported no Unicode issues since a long time. For other routines (filename handling and the like) surrogate characters can be handled like like normal characters without problems. Although I doubt the usefulness of filenames with codepoints > BMP. [....snipped irrelevant part....] Martin ------------------[ end ]---------------------- Regards, - Graeme - _______________________________________________ fpGUI - a cross-platform Free Pascal GUI toolkit http://opensoft.homeip.net/fpgui/

Reply by email

Ivan P. Gan
to Martin Schreiber: MSEgui doesn't care about surrogate pairs at the moment, they are handled as two…
Martin Schreiber wrote 29 lines
On Tuesday 05 August 2008 09.47:01 Marco van de Voort wrote: > On Tue, Aug 05, 2008 at 08:50:29AM +0200, Martin Schreiber wrote: > >> With UCS2/utf-16 on the other hand *all* current users of MSEgui can >> store *all* the codepoints they need in a single utf-16 word >> > ?? Martin, can you make a formal statement about what MSEGUI now supports? > I keep getting the feeling you only support UCS-2. > > MSEgui doesn't care about surrogate pairs at the moment, they are handled as two consecutive (unknown) single characters. If the users need codepoints above the BMP they can store it in widestrings in utf-16 encoding but MSEgui will not show it correctly on the screen. BiDi isn't supported neither BTW. Surrogate pair handling can be implemented in the MSEgui screen and edit functions later if necessary. My point is that none of the current MSEide+MSEgui users need codepoints above the BMP so I don't want to complicate and slowdown internal stringhandling for a feature nobody needs at the moment. BiDi is probably more important but difficult to implement for me because I have no knowledge in right to left languages. Martin _______________________________________________ Lazarus mailing list http://www.lazarus.freepascal.org/mailman/listinfo/lazarus

Hi Martin,
If you need string samples, Hebrew & Mixed text, I can get that for you Also, as my web sites are multi-lingual, I have some knowledge of the subject & will be happy to help, if I can
Regards
Ivan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.lazarus.freepascal.org/pipermail/lazarus/attachments/20080805/0ab797a3/attachment.html

Reply by email

Martin Schreiber
to Graeme Geldenhuys: That knew that, and that's what I said... utf-8 represents it's data in byte form, to…

On Tuesday 05 August 2008 10.10:27 Graeme Geldenhuys wrote: [...]

Show quoted text 15 lines
> For internal character representation you should first and formost choose > an encoding which will be easy to handle by the users. > Because for most of the users a codepoint can not be expressed by a > single utf-8 byte, they can not use character access by string index in > their code and bytelength(utf8string) charactercount. And do a search on the Internet... over and over you get articles refuting that point. Random character access is very seldom required. If character access is used, it normally coincides with looping/iterating through the string, which utf-8 has no problems with either. Random character access is not the norm in program code, and if so, it's a very small part. http://weblogs.mozillazine.org/roc/archives/2008/01/string_theory.html

Most MSEgui users don't program a webbrowser but some string and character manipulation is often used and often the assumption codepoint = storage unit simplifies the task a lot. All current users of MSEgui can take that assumption for widestrings, most can not for utf-8 encoded strings.

[...]

Show quoted text 8 lines
> Parsing of utf-16 is much simpler than parsing of utf-8, processing of > UCS2/utf-16 in framework core is probably faster than utf-8 because there > are less addressing operations for codepoints > 127. Yes and with utf-16 you need to worry about the BOM marker and endianness. utf-8 you don't.

MSEgui uses UCS2 encoding for internal character representation only, there is no BOM and the endianness is the endianess of the system. Converting from/to the system encoding is handled by FPC transparently.

[...]

Show quoted text 15 lines
> Widestring versions of all Pascal standard string functions (pos, > copy...) are supported by FPC. Does the widestring version of Copy take into account surrogate pairs? Or would it simply split the bytes? As for Pos, Copy - LCL and fpGUI has their own utf-8 versions. Why would those be any slower than the widestring versions included in FPC? As for Length()... What should it return? Bytes or Character count? If bytes (which I believe is what the documentation says it must return), then the standard ANSI Length() included in FPC will suffice. If it's character count you want, LCL and fpGUI has their own versions of that.

Length(), Copy()... should work with storage units and not with codepoints or glyph entities, it is unpracticable to use display entities, think on glyphs which are composed by several codepoints. I recently read a similar statement from Allen Bauer (CodeGear).
http://groups.google.ch/group/borland.public.delphi.non-technical/browse_thread/thread/cf4398427aa0358c/838865b5fef0cbfc

Martin

Reply by email

Graeme Geldenhuys
to Martin Schreiber: [...] Most MSEgui users don't program a webbrowser but some string and character…
Martin Schreiber wrote
manipulation is often used and often the assumption codepoint = storage unit simplifies the task a lot. All current users of MSEgui can take that

Which then tells me that if you one day decide to update MSEgui from UCS-2 to UTF-16, all your users applications have potential problems. Like the old saying goes.... "Assumption is the cause of all major f*ckups! Never assume anything." ;-)

assumption for widestrings, most can not for utf-8 encoded strings.

Which is a good thing!

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Reply by email

Martin Schreiber
to Graeme Geldenhuys
Graeme Geldenhuys wrote 9 lines
On Tue, Aug 5, 2008 at 10:48 AM, Martin Schreiber wrote: > manipulation is often used and often the assumption codepoint = storage > unit simplifies the task a lot. All current users of MSEgui can take that Which then tells me that if you one day decide to update MSEgui from UCS-2 to UTF-16, all your users applications have potential problems. Like the old saying goes.... "Assumption is the cause of all major f*ckups! Never assume anything." ;-)

There will be no problems if the used codepoints are from the BMP only. I don't expect that a business application which is developed for the European market for example suddenly has to work with scripts of dead languages or ancient Chinese names.

Martin

Reply by email

Graeme Geldenhuys
to Martin Schreiber
Martin Schreiber wrote 5 lines
There will be no problems if the used codepoints are from the BMP only. I don't expect that a business application which is developed for the European market for example suddenly has to work with scripts of dead languages or ancient Chinese names.

Hey, Klingon is up and coming language! :-)

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Reply by email

Graeme Geldenhuys
to Graeme Geldenhuys
Graeme Geldenhuys wrote
Hey, Klingon is up and coming language! :-)

Oh yeah! I was told to always backup my claims.... ;-) So here goes...

Klingon Language Institute
http://www.kli.org/

Klingon Language Certification Program
http://higbee.cots.net/Holtej/klingon/klcp.htm

See the section on "Klingon language support"
http://www.lanana.org/docs/unicode/unicode.txt

Still not sure... If it's on WikiPedia it must be true! ;-) http://en.wikipedia.org/wiki/Klingon_language

Proposal to encode Klingon in Plane 1 of ISO/IEC 10646-2 http://std.dkuug.dk/JTC1/SC2/WG2/docs/n1643/n1643.htm

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Reply by email

Ivan P. Gan
to Martin Schreiber: There will be no problems if the used codepoints are from the BMP only. I don't expect…
Martin Schreiber wrote 18 lines
On Tuesday 05 August 2008 11.31:29 Graeme Geldenhuys wrote: > On Tue, Aug 5, 2008 at 10:48 AM, Martin Schreiber wrote: > >> manipulation is often used and often the assumption codepoint = storage >> unit simplifies the task a lot. All current users of MSEgui can take that >> > Which then tells me that if you one day decide to update MSEgui from > UCS-2 to UTF-16, all your users applications have potential problems. > Like the old saying goes.... "Assumption is the cause of all major > f*ckups! Never assume anything." ;-) > > There will be no problems if the used codepoints are from the BMP only. I don't expect that a business application which is developed for the European market for example suddenly has to work with scripts of dead languages or ancient Chinese names.

Worst case scenario the user wants to use the application in relation to ancient languages, the best design method is to start out with support for all languages possible with Unicode support
I see someone wants Klingon, is it even assigned a listing in the Unicode tables yet?
If it is then my web site can support it :)
What about when I go crazy & get funky with dynamic language support & you find that all Lazarus applications suddenly support hundreds of languages
I have already done this with web sites & CAN do it with Lazarus too If anybody wants to work with me to do it please contact me for further details on how this can be achieved

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.lazarus.freepascal.org/pipermail/lazarus/attachments/20080805/cd181046/attachment-0001.html

Reply by email