Hi,

Does FPC have UTF-8 versions of the Copy() and Length() functions?

I've started implementing support for UTF-8 in my fpGUI widget set - more specifically text painting in all components (the easy bit) and text handling in the TFEdit (edit box widget).

For now I have used the UTF8Copy() and UTF8Length() functions as designed in the Lazarus LCL project and it works fine. I'm pretty sure FPC must have such functions, but I searched everywhere and couldn't find anything.

Op Sat, 19 May 2007, schreef Graeme Geldenhuys:

Does FPC have UTF-8 versions of the Copy() and Length() functions?

They don't exist. FPC has been designed to either use the system encoding (which can be utf8). In this case, the string routines from sysutils do what you want. The other option is to use widestrings; length(utf8decode(s)) will return the length of an utf-8 string.

Daniël

Sorry, I'm very new to Unicode support. Wouldn't it be useful to have UTF-8 and UTF-16 (and all the other encodings) functions in FPC? For example the Lazarus LCL (LCLProc unit) has loads of such functions.

The Length function is easy to get around, but the Copy, Pos ,etc functions are not.

Op Sat, 19 May 2007, schreef Graeme Geldenhuys:

I think a utf8 unit would be usefull.

The Length function is easy to get around, but the Copy, Pos ,etc functions are not.

Why?

Daniël

Daniël Mantione wrote

I think a utf8 unit would be usefull.

Glad you agree. I searched through the mailing list and found quite a few people that required such function and implemented their own. So yes, why don't we include a new unit with those functions for everybody to use. The Lazarus project (LCLProc unit) has been tried and tested for ages. We can create a new unit based on the functions in there - just remove any LCL specific functions.

Seeing that the LCLProc unit has been in use for ages in Lazarus, any chance that we can get such a unit fast-tracked and included in trunk and fixes_2_2 branch? Ready for the next release. It shouldn't interfere with any existing applications, because it will be a new and separate unit. It's also been well tested in Lazarus.

So what name could we give such a unit. unicodeutils.pas? utfutils.pas?

> The Length function is easy to get around, but the Copy, Pos ,etc > functions are not. Why?

I meant not easy by inline'ing things like your Length example. The UTF8Copy() function requires a bit more coding, that's all.

Daniël Mantione wrote

I think a utf8 unit would be usefull.

Some things need to be choosen:

* which functions from lclproc would be included

* the position of the unit on fpc (maybe fcl? but then, which directory from fcl)

* the exact name of the unit

thanks,

No, it should go in the RTL. It's very basic.

* the exact name of the unit

utf8 seems appropriate, or utf if more than one encoding is supported.

Michael.

Op Sat, 19 May 2007, schreef Felipe Monteiro de Carvalho:

Not FCL. FCL is for classes based OOP code, while utf-8 has a much broader scope. I'd say RTL.

* the exact name of the unit

It depends on the first point. If it is just utf-8, utf8 would be a good name.

Daniël

Op Sat, 19 May 2007, schreef Graeme Geldenhuys:

As far as I am concerned, it can only go in fixes_2_2 after it has proven to be stable and should not disrupt the 2.2 release. In other words, it might not make 2.2.0, but could make 2.2.2.

It is mostly usefull if you use widestrings internally. If you use utf-8 internally I agree an utf-8 would be more comfortable.

Daniël

Michael Van Canneyt wrote

utf8 seems appropriate, or utf if more than one encoding is supported.

I think that having a generic name is better, even if at first we only add utf-8 functions, because when we need more functions for other encodings we would need to create another unit, or rename this one, and that's unnecessary work

Outer RTL? It is not needed for cycling :_)

----- Original Message -----

Rimgaudas Laucius wrote

It is not useful to have functions for both encodings, because these encodings are interconvertable and it is more effective to use UTF-16 for data processing

I disagree. The conversion impacts performance heavely. It will also require memory to store the converted string, and after you perform a operation you need to convert back.

Further, UTF-16 contains both 2-byte characters and 4-byte characters, so I don't see how it would be any faster to process it in comparison to process a utf-8 string.

About being easier to implement, that's irrelevant, because the functions are already done.

Hi,

I've started to create a unit for UTF-8, but lack of time is keeping me from developing it at the moment. I have some implementation already if you are interested, and wish to continue to develop it.

Ido

Storage:
UTF8<UTF16 only for most of latin scripts,
all other scripts (Chinese, Greek, Cylilic, Arabic, Indic, ...) UTF8>UTF16.

Performance:
Length (UTF8) = UTF8->UTF16
2*Lenth(UTF8)> UTF8->UTF16

4-byte characters are used by UTF32. UTF16 uses sequences of 2 code points from surrogates area to expess charactes outside basic multilingual plane that are very rarely used (actully i do not know any program that implements that).

Rimgaudas Laucius wrote

Performance: Length (UTF8) = UTF8->UTF16 2*Lenth(UTF8)> UTF8->UTF16

I don't understand what you are trying to say here.

I mean that any operation done on UTF8 string (deleting or inserting of part of string) consume as much as converting it to UTF16. So it will be less effective to do that operation twice with UTF8 string than to convert it to UTF16 where these operations may be done instantly.

I just grabbed what I needed from the LCLProc unit (part of Lazarus) which has a lot of UTF-8 and UTF-16 functions.

I'll put all those functions in a single unit and post it to fpc-devel mailing list for review before we include it into FPC.

If you have any more functions, please pass them on, so I can include them as well.

Regards,
- Graeme -

Op Sat, 19 May 2007, schreef Felipe Monteiro de Carvalho:

For most operations, it is not necessary to process characters outside the BMP separately, i.e.:

for i:=1 to length(s) do
  s[i]:=upcase(i);

... is valid UTF-16 code, and much faster than the same operation in UTF-8.

Daniël

Hi,

Here is what I made so far,
http://ik.homelinux.org/projects/files/ikunicode.tar.gz

You can take anything you wish. Please note that when I'll have time, I'll continue working on the unit itself. I also wanted it to be included on the rtl of FPC, but I don't see it happens anytime soon... :(

Ido

It's not that easy, for example upper/lower casing, comparing etc. of unicode requires usually external big tables which we don't want to pull into fpc.

----- Original Message -----

Florian Klaempfl wrote

It's not that easy, for example upper/lower casing, comparing etc. of unicode requires usually external big tables which we don't want to pull into fpc.

Yes, but if the unicode functions are all in a separate unit, they won´t be included in your executable, unless you really want it.

Plus I don´t remember any tables on the utf function on Lclproc, but maybe I didn´t pay enougth attention

Only simple helper functions are in LCLProc.
A lower/uppercase Unicode function is not simple and when eventually the LCL needs it, it would simply delegate the implementation to the LCL interfaces, which can simply call the appropriate library functions.

Mattias

All these unicode functions should code through the widestring mananger so they get plugable.

Felipe Monteiro de Carvalho wrote

Yes, but if the unicode functions are all in a separate unit, they won´t be included in your executable, unless you really want it.

Yup, I agree on this one...

Which is why you must love Open Source software. :-) Another Object Pascal based GUI Framework has already done all the work for us and they used a BSD license so we are free to use what we need.

Have a look at the LPTK project on Source Forge. Two units called schar16.pas and unitkeys.pas

It contains a lot of functions: Upper, Lower, Pos, Length, conversions from UTF-8 to UTF-16 and vice versa, character translations for Latin1-Latin4, Arabic, Kana, Hungarian, Cyrillic, Hebrew, Thai, etc... Lots we can use!

The reason I started writing my own library is because I wanted pure Pascal libraries to use things. I dislike bindings to other libraries written in different languages (such as QT). "I have a dream" to see native tools for all common needs written in pure (Object) Pascal.

Secondly, I'm non Latin native speaker, so I find the need for Unicode on all of it's glory (and bad things) a must rather then something that can only be "good" additional support.

On my code, you can see that I prepared the ground for working with TWideStringManager.
Oh, and it's MIT license (not as restrictive as BSD license ;))

Ido

ik wrote

The reason I started writing my own library is because I wanted pure Pascal libraries to use things. I dislike bindings to other libraries written in different languages (such as QT). "I have a dream" to see native tools for all common needs written in pure (Object) Pascal.

I like that dream... Hence the reason I work on the fpGUI Widget Set. A GUI Framework implemented in Object Pascal, no huge 3rd Party library requirements and easy portable to other systems. Currently runs on Linux (Xlib) and Windows (GDI) and the graphics backend tested on Windows Mobile.
See http://opensoft.homeip.net/fpgui/

Secondly, I'm non Latin native speaker, so I find the need for Unicode on all of it's glory (and bad things) a must rather then something that can only be "good" additional support.

I don't require Unicode support in fpGUI but others do, like Felipe. I also don't want one platform having features another doesn't. Hence the reason I started with Unicode support while running under X11 (Felipe already did it for GDI).