Hi list,

I'm diving into properly dealing with UTF8 encoded data coming in from my Twitter application/library (see FPC list announcement yesterday).

Decided to go over the Laz wiki page on unicode and cleaned up some typos etc:
http://wiki.lazarus.freepascal.org/LCL_Unicode_Support

I've got some questions:
- Widestrings and Ansistrings section
What is a widestring? An UTF-16 encoded string? Or a string that can hold multiple bytes that is further undefined?

-Searching a substring section
Searching UTF 8 substrings has this code
uses lazutf8;
...

  BytePos:=Pos(SearchFor,aText);
  CharacterPos:=UTF8Length(PChar(aText),BytePos-1);
  writeln('The substring "',SearchFor,'" is in the text "',aText,'"',
    ' at byte position ',BytePos,' and at character position

',CharacterPos);

It says:
"Due to the special nature of UTF8 you can simply use the normal string functions"
=> is the special nature the fact that there is only a single way to decode UTF8 characters because multi-byte encodings have different high bits set (as explained in the description of UTF8 on the bottom of the page)

- No Unicode support on Win9x section
"Windows platforms <=Win9x [..] only partially support Unicode" however:
"Win 9x and NT offer two parallel sets of API functions:[..]the new, Unicode enabled *W."
Presumably not all *A functions are available as *W functions on Win9x, which is why Unicode is not fully supported?
(I can also imagine the default fonts etc do not support showing Unicode characters)

Thanks for any clarification; I'll update the wiki...

Thanks,
Reinier

Hello, I clarified all points mentioned here directly in the wiki.

Felipe Monteiro de Carvalho wrote

Hello, I clarified all points mentioned here directly in the wiki.

Thanks, Felipe, it's clear now!

Reinier

Reinier Olislagers wrote

It says: "Due to the special nature of UTF8 you can simply use the normal string functions"

It also says:
" Note also that UTF-16, like UTF-8, may have decomposed characters. The character "?" for example might be encoded as a single character or as 2 characters: "A" + a modifying accent.

Thus I understand, the same printout of a Unicode string can have multiple encoding variants.

So the use of "pos" and "=" needs to be handled with care.

-Michael

Michael Schnell wrote

Thus I understand, the same printout of a Unicode string can have multiple encoding variants. So the use of "pos" and "=" needs to be handled with care.

Yes, but in practice the only place that I know that uses everything in decomposed unicode is the Mac OS X filesystem. So for dealing with Mac OS X filenames we need special care and for the rest just suppose composed mode (but remembering that some chars have no composed mode!).

Felipe Monteiro de Carvalho wrote

Yes, but in practice the only place that I know that uses everything in decomposed unicode is the Mac OS X filesystem. So for dealing with Mac OS X filenames we need special care and for the rest just suppose composed mode (but remembering that some chars have no composed mode!).

Anyway it might be helpful to note this in the wiki page at http://wiki.lazarus.freepascal.org/LCL_Unicode_Support#Searching_a_substring.

The text "For searching in a substring they do work perfectly" might cause misunderstanding on that behalf. (Of course it technically does work fine, but the ambiguity of Unicode itself introduces potential problems).

-Michael

Well, it's a wiki. You can edit it...

Reinier Olislagers wrote

Well, it's a wiki. You can edit it...

done.
-Michael

IMO the meaning is clear: searching for an sub*string* means an *exact* match. Who wants case insensitive or otherwise inexact matches, has to use specialized *text* functions.

Ambiguous character representations are a general problem with characters, where e.g. line endings or whitespace can be represented in different ways. The accent/umlaut composition is not a Unicode problem, it occurs in all codepages.

DoDi

Reinier Olislagers wrote

What is a widestring?

On Windows and FPC 2.4+? COM compatible UTF16. (UCS2 on win2000).

On the 2.4, and 2.6. non Windows and general FPC 2.2-, plain UTF16. (aka Kylix widestring).

- No Unicode support on Win9x section "Windows platforms <=Win9x [..] only partially support Unicode" however:

No. Afaik fefault they *don't*. Only with manually installed extensions they *partially* support unicode.

Presumably not all *A functions are available as *W functions on Win9x, which is why Unicode is not fully supported?

Afaik that is again win9x+unicode extensions(unicows) AND a suitable adapted RTL, not default win9x.

Thanks, updated
http://wiki.lazarus.freepascal.org/Widestrings

> - No Unicode support on Win9x section > "Windows platforms <=Win9x [..] only partially support Unicode" > however:

<snip>

So my supposition was correct: Win9x (however spruced up) does not support all *W functions.

Hearing about this reinforces my opinion that patches that work on everything except Win9x should be applied and that people wanting to support Win9x should commit workarounds themselves if available.... (Looking at you, bug 21659)

Thanks,
Reinier

Marco van de Voort wrote

No. Afaik fefault they *don't*. Only with manually installed extensions they *partially* support unicode.

Some W functions are supported in Win9x, please read:

http://wiki.lazarus.freepascal.org/LCL_Unicode_Support#Wide_functions_present_on_Windows_9x

Microsoft killed the original page which had the list of such routines, like they killed MSDN documentation for Windows 9x API =( But there was such a list and when I wrote the original LCL Win9x Unicode support I verified that almost all of the routines in the list worked correctly (some routines in the list didn't work, despite being documented as working in Win9x).

Anyway, we should let Win9x die already.

Some W functions are supported in Win9x, please read:

http://wiki.lazarus.freepascal.org/LCL_Unicode_Support#Wide_functions_presen t_on_Windows_9x

http://support.microsoft.com/kb/210341 has also a list of "natively" supported W functions.

+1 for killing win9x support.

Ludo

Reinier Olislagers wrote

Hearing about this reinforces my opinion that patches that work on everything except Win9x should be applied and that people wanting to support Win9x should commit workarounds themselves if available.... (Looking at you, bug 21659)

I've hassled people on a couple of occasions about things that don't work in elderly OSes (I've still got a few NT4 systems), but in general I agree.

Elsewhere, somebody's pointed out http://kernelex.sourceforge.net/ which is a compatibility layer that updates/implements windows '98 APIs to the extent that it's supposed to be able to run recent copies of Mozilla etc. I've not tried it, there's some depths I'm reluctant to plumb :-)

I've a Win98 VM running my mail, and I cannot install in it new versions of many programs (Thunderbird, Acrobat...). IMO we should not try to make an Unicode version of FPC/Lazarus work in such an environment. Win98 uses the Ansi API internally, so that the use of Wide/UnicodeString in a Lazarus program will result in many string conversions in API calls *and* in all components.

DoDi

Of course, but that does not mean that it is worth to be mentioned in this case where it is obvious that users might be unaware of the ambiguity of printable characters.

The accent/umlaut composition is not a Unicode problem, it occurs in all codepages.

I did not know that ANSI code of any page provides combining multiple bytes to a single printable character. (But I did not check many of them.)

-Michael