Hello,

I read the code for UTF8Encode and UTF8Decode routines and they seam to suppose that the widestring encoding is UCS-2! Instead of UTF-16

Is this the expected behavior or is it only partially implemented?

Someone added UTF8 <-> UTF16 conversion routines in the LCL, so they could be used to repair UTF8Encode / Decode.

thanks,

Op Tue, 26 Aug 2008, schreef Felipe Monteiro de Carvalho:

It is a broken implementation, it should be fixed.

Daniël

Hello Felipe,

Tuesday, August 26, 2008, 12:28:45 PM, you wrote:

FMdC> I read the code for UTF8Encode and UTF8Decode routines and they seam FMdC> to suppose that the widestring encoding is UCS-2! Instead of UTF-16 FMdC> Is this the expected behavior or is it only partially implemented? FMdC> Someone added UTF8 <-> UTF16 conversion routines in the LCL, so they FMdC> could be used to repair UTF8Encode / Decode.

http://bugs.freepascal.org/view.php?id=11791

The LCL implementation of UTF8ToUTF16 (default one) does not pass the UTF8 stress test. My version does not handle UTF16 intentionally as the proposed replace function can not make grow the widestring which comes allocated, but... hmmm... oh! it comes at 2*bytes in UTF8, so there must be space always to fit any character even the doubles ones.

:) It's time to add the UTF16 to my code.

That doesn't say much... :-)

I couldn't find a single program or editor that could display that file correctly. Pretty amazing that one file can defeat all programs. :-)

Programs I tried:
Kate (kde editor)
gEdit (Gnome editor)
Firefox 3 (did the best out of the lot)
vi (disastrous)
OpenOffice 2.3 (probably the best out of the lot)
AbiWord 2.4.6

Do you know of any program that passes that test?

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

I viewed that file with Opera 9.21 and it looked fine for me (all out-of-range chars replaced by a triangle symbol), although the question how it *should* look remains open.

Regards,
Sergei

Hello Graeme,

Tuesday, August 26, 2008, 3:48:39 PM, you wrote:

GG> That doesn't say much... :-)

It's the usual status :) most ones are focused in speed instead given a bit of security, special against overlong sequences which can bypass security barriers... check the attached file and decode it with FPC UTF8Decode ;) and DO NOT RUN IT! ;)

GG> I couldn't find a single program or editor that could display that GG> file correctly. Pretty amazing that one file can defeat all programs. GG> :-)
GG> Programs I tried:
GG> Kate (kde editor)
GG> gEdit (Gnome editor)
GG> Firefox 3 (did the best out of the lot)
GG> vi (disastrous)
GG> OpenOffice 2.3 (probably the best out of the lot) GG> AbiWord 2.4.6
GG> Do you know of any program that passes that test?

Nope, I had checked some of them and Firefox is maybe the better. You can also try the UTF8ToUnicode that I'd posted in the bug filled.

Sergei Gorelkin wrote

> I viewed that file with Opera 9.21 and it looked fine for me (all out-of-range chars replaced by a triangle symbol), although the question how it *should* look remains open.

You are right. :-) Just tried Opera 9.5 (after setting my fonts to mono). It looks by far the best and only tests 2.1.x and 2.2.x fails - right hand column must be in line except for 2.1.1 and I think 2.2.1

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Hello Sergei,

Tuesday, August 26, 2008, 4:16:00 PM, you wrote:

SG> I viewed that file with Opera 9.21 and it looked fine for me (all SG> out-of-range chars replaced by a triangle symbol), although the question SG> how it *should* look remains open.

Opera 9.50 (Windows) fails for 5.3.x tests. Also it seens to fail at 2.2.2 eating the " simbol after the character, but it could be a render error due the carrier rendering NULL character. Even I do not know it is should be considered OK the 2 chars render of 2.1.4 and 2.3.4, but again seens to be a render problem, not a decode error.