Hi,

Has anybody written these conversions functions yet? If not, I am about to port the ConvertUTF.c file from Unicode.org to Object Pascal.

UTF-32 to UTF-16
UTF-32 to UTF-8
UTF-16 to UTF-32
UTF-16 to UTF-8
UTF-8 to UTF-16
UTF-8 to UTF-32

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Please check first if the rtl routines

function UnicodeToUtf8(Dest: PChar; Source: PUnicodeChar; MaxBytes:
SizeInt): SizeInt;{$ifdef SYSTEMINLINE}inline;{$endif}
function UnicodeToUtf8(Dest: PChar; MaxDestBytes: SizeUInt; Source:
PUnicodeChar; SourceChars: SizeUInt): SizeUInt;
function Utf8ToUnicode(Dest: PUnicodeChar; Source: PChar; MaxChars:
SizeInt): SizeInt;{$ifdef SYSTEMINLINE}inline;{$endif}
function Utf8ToUnicode(Dest: PUnicodeChar; MaxDestChars: SizeUInt;
Source: PChar; SourceBytes: SizeUInt): SizeUInt;
function UTF8Encode(const s : Ansistring) : UTF8String; inline;
function UTF8Encode(const s : UnicodeString) : UTF8String;
function UTF8Decode(const s : UTF8String): UnicodeString;
function AnsiToUtf8(const s : ansistring): UTF8String;{$ifdef
SYSTEMINLINE}inline;{$endif}
function Utf8ToAnsi(const s : UTF8String) : ansistring;{$ifdef
SYSTEMINLINE}inline;{$endif}
function UnicodeStringToUCS4String(const s : UnicodeString) : UCS4String;
function UCS4StringToUnicodeString(const s : UCS4String) : UnicodeString;
function WideStringToUCS4String(const s : WideString) : UCS4String;
function UCS4StringToWideString(const s : UCS4String) : WideString;

are sufficient or how they can be extended.

Florian Klaempfl wrote

function UTF8Decode(const s : UTF8String): UnicodeString;

Is there some hard-coded limit on UTF8Decode? I am writing some unit tests for these methods and on my 3rd test, it already fails. I'm using testing sample data from unicode.org.

var
  s8: UTF8String;
  s16: UnicodeString;
begin
  // U+289A8 CJK UNIFIED IDEOGRAPH-289A8
  s8 := Char($F0) + Char($A8) + Char($A6) + Char($A8);
  s16 := UTF8Decode(s8);
  AssertEquals('Failed on 4', 4, Length(s16));
  AssertEquals('Failed on 5', UnicodeChar($D862) + UnicodeChar($DDA8), s16)
end;

Test 4 fails: Expected 4 but was 0.
Test 5 fails: to due to Test 4 failure...

---------------[ from unicode description file ]----------------------- U+289A8 CJK UNIFIED IDEOGRAPH-289A8

General Character Properties

Unicode category: Letter, Other

Various Useful Representations

UTF-8: 0xF0 0xA8 0xA6 0xA8
UTF-16: 0xD862 0xDDA8
------------------------[ end ]----------------------------

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

See http://bugs.freepascal.org/view.php?id=11791

Florian Klaempfl wrote

See http://bugs.freepascal.org/view.php?id=11791

OK, but that bug report is about an invalid sequence and getting out NULL. I'm testing perfectly valid UTF-8 sequence and still getting NULL result.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Hello Graeme,

Sunday, November 23, 2008, 12:42:45 AM, you wrote:

GG> On Sat, Nov 22, 2008 at 9:07 PM, Florian Klaempfl

GG> wrote

> > See http://bugs.freepascal.org/view.php?id=11791

GG> OK, but that bug report is about an invalid sequence and getting out GG> NULL. I'm testing perfectly valid UTF-8 sequence and still getting GG> NULL result.

Not checked right now, but as no valid sequences, no surrogates support.

JoshyFun wrote

Not checked right now, but as no valid sequences, no surrogates support.

So UTF8Decode only supports UCS2 output! Now this is why I think supporting UTF-8 in fpGUI and Lazarus LCL was a good idea. By design (utf-8), you have to support the whole unicode range. With UTF-16, many people take shortcuts and actually only support UCS2 - and it goes unnoticed like this case for many years!

I'm busy writing unit tests for all the conversion functions and implementing some new helper functions as well. Hopefully this will highlight all the UCS2 shotcuts in UTF-16 implementation and other possible conversion issues.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Hello Graeme,

Sunday, November 23, 2008, 9:21:09 AM, you wrote:

GG> So UTF8Decode only supports UCS2 output! Now this is why I think GG> supporting UTF-8 in fpGUI and Lazarus LCL was a good idea. By design GG> (utf-8), you have to support the whole unicode range. With UTF-16, GG> many people take shortcuts and actually only support UCS2 - and it GG> goes unnoticed like this case for many years!

It no so worst, technically it is easy to solve all of those issues, as you seen in the bug report also Delphi has the same problems and it has not been fixed to keep Delphi compatibility. My UTF8ToUnicode takes care of all that problems and the surrogate pairs with a 25% speed penalty (not the same version posted in the bug report, other that I had optimized a bit).

GG> I'm busy writing unit tests for all the conversion functions and GG> implementing some new helper functions as well. Hopefully this will GG> highlight all the UCS2 shotcuts in UTF-16 implementation and other GG> possible conversion issues.

There are many test case files in unicode.org but most of them are quite complex to be coded as a test case :(

Also I whish to know which basic unicode functions will be supported by FPC, only upper/lower, or maybe some more like decompose, normalize, char-word-line-paragraph iterators... I have some of them written if the FPC team wants them.

Possible, however I remember no other bug report.

Afaik we decided to apply the patch, however, I'd no time yet to do so.

It mainly depends if it needs external libs or huge tables.

Hello Florian,

you wrote

> It no so worst, technically it is easy to solve all of those issues, > as you seen in the bug report also Delphi has the same problems and it > has not been fixed to keep Delphi compatibility.

FK> Afaik we decided to apply the patch, however, I'd no time yet to do so.

Well, maybe I should say "not fixed till now for Delphi compatibility". I think when UnicodeString is a fact for stable releases many functions should be fixed, overloaded, written, etc.

> Also I whish to know which basic unicode functions will be supported > by FPC, only upper/lower, or maybe some more like decompose, > normalize, char-word-line-paragraph iterators... I have some of them > written if the FPC team wants them.

FK> It mainly depends if it needs external libs or huge tables.

My functions do not used any external lib but tables are quite big. Most tables has been written using a pascal code to balance between some "if/case" lines and binary search on tables, but tables are still big but I think that they can not be too much shorter and keep functionality. Examples:

------------------------------------------------
//LowerCase arrays size: 2376 bytes
const UnicodeLowerCaseArraySource: array [0..593] of WORD=( //UpperCase arrays size: 2408 bytes
const UnicodeUpperCaseArraySource: array [0..601] of WORD=( //TitleCase arrays size: 2424 bytes
const UnicodeTitleCaseArraySource: array [0..605] of WORD=(

type TUnicodeWordBreakEntry=packed record
  BeginRange: LongInt;
  EndRange: LongInt;
  WordBreakClass: TUnicodeWordBreak;
end;
const TUnicodeWordBreakArray: array [0..737] of TUnicodeWordBreakEntry =(
------------------------------------------------

This tables also need some functions like:

function UnicodeUCS4ToLowerCase(const UC: integer): integer;
begin
  case UC of
  $000041..$00005A: Result:=UC+32;
  $0000C0..$0000D6: Result:=UC+32;
  $0000D8..$0000DE: Result:=UC+32;
  $000391..$0003A1: Result:=UC+32;
  $0003A3..$0003AB: Result:=UC+32;
  $000400..$00040F: Result:=UC+80;
  $000410..$00042F: Result:=UC+32;
  $000531..$000556: Result:=UC+48;
  $0010A0..$0010C5: Result:=UC+7264;
  $001F08..$001F0F: Result:=UC+-8;
  $001F18..$001F1D: Result:=UC+-8;
  $001F28..$001F2F: Result:=UC+-8;
  $001F38..$001F3F: Result:=UC+-8;
  $001F48..$001F4D: Result:=UC+-8;
  $001F68..$001F6F: Result:=UC+-8;
  $001F88..$001F8F: Result:=UC+-8;
  $001F98..$001F9F: Result:=UC+-8;
  $001FA8..$001FAF: Result:=UC+-8;
  $002160..$00216F: Result:=UC+16;
  $0024B6..$0024CF: Result:=UC+26;
  $002C00..$002C2E: Result:=UC+48;
  $00FF21..$00FF3A: Result:=UC+32;
  $010400..$010427: Result:=UC+40;
// In offsets calculated 406 items of 1023
  else begin
    Result:=UnicodeCaseBS(UnicodeLowerCaseArraySource,UC,0,High(UnicodeLowerCaseArraySource));
    if Result=-1 Then Result:=UC else Result:=UnicodeLowerCaseArrayTarget[Result];
    end;
  end;
end;

That makes me wonder? How does D2009 handle upper/lower case? Table lookups, or do they pass that on to Windows to handle?

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

How does this work, given that upper/lower case sometimes depends on the language? (e.g., in Turkish the upper case version of "i" is "İ" -- LATIN CAPITAL LETTER I WITH DOT ABOVE)

Jonas

Most likely the latter.

Michael.

I would guess that too.... Which also means they are making it so much harder to ever support other platforms. Unless they decide to use FPC as their compiler for other platforms. Just like they are using Chrome (or whatever it is called now) for Delphi Prism. :-)

Wouldn't that be cool! ;-)

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Hello Jonas,

JM> How does this work, given that upper/lower case sometimes depends on JM> the language? (e.g., in Turkish the upper case version of "i" is "I" JM> -- LATIN CAPITAL LETTER I WITH DOT ABOVE)

Only general case, language tailoring is a completly different beast. Basic unicode functions are language agnostic and for sure will produce some bad results in some circunstances. It's almost impossible to cover all tailoring even using the database about tailorings, not for the upper/lower but for other operations like word breaking.

Libraries that cover a lot of language particularities are around 30-40 (or more) megabytes in runtime data and I think this kind of dependencies are a no, no, for FPC.

I think we should simply depend on the OS in this case like the cwstring unit does though linux doesn't make life easy in this case, it requires always a conversion to ucs-4 to get a string upper/lower cased.

Not really. Pretty much every platform sports this sort of functionality in the system libraries. The reason that FPC doesn't use them on non-Windows, non-Mac OS X and non-Solaris, is because on these other platforms FPC remains libc-independent as much as possible for various reasons. It's very easy to abstract this though (as is done by FPC's widestring manager) so you can use whatever you want under the hood.

Jonas

From the marco catu doc:

"The traditional Delphi code (the UpCase on the AnsiChar version) handles ASCII characters
only, so it won't convert the character (The same is true for the UpperCase function, which
handles only ASCII, while AnsiUpperCase handles everything in Unicode, despite the
name.). The behavior doesn't change (probably for backward compatibility reasons) if you pass
a WideChar to it. The ToUpper function works properly (its ends up calling the CharUpper
function of the Windows API)"

Luiz

Hello Florian,

you wrote

> Only general case, language tailoring is a completly different beast.

[...]

FK> I think we should simply depend on the OS in this case like the cwstring FK> unit does though linux doesn't make life easy in this case, it requires FK> always a conversion to ucs-4 to get a string upper/lower cased.

Of course if OS provides functions, use them! they should be properly implemented and will be even faster without memory serious impact, but I'm quite sure that not all functions will be provided by all OSs. Maybe a small subset should be available for platforms that do not provide native support like DOS and a WideString or UnicodeString is available, maybe as a separate unit to be linked in only if needed.

Well, if anybody volounteers, something similiar to cwstrings for unix could be done :)

In our previous episode, Florian Klaempfl said:

Worse, the Dos users are the worst with respect to binary size syndrome.

Well, as long as it's optional (as suggested by the original post), noone should complain... ;-)

Tomas

Indeed, the cwstrings unit is optional as well.