Hi,

Can you take a look at this issue reported at the lazarus bug tracker: http://www.lazarus.freepascal.org/mantis/view.php?id=888

It contains a fixed version of the Utf8ToUnicode function. Since it is part of the rtl, I close this lazarus issue and send you this message. I did not test the fixed version.

Regards,
Vincent.

The files in the zip file are not usable; They're in some unicode format, which I can't use nor check on Linux.

Michael.

They're plain UTF-8. I know for a fact there are editors under Linux which support that (at least emacs does, and it would surprise me immensely if vim doesn't). Anyway, here's the plain ascii version of the "-fixed" file.

Jonas

function Utf8ToUnicode(Dest: PWideChar; MaxDestChars: SizeUInt; Source:
PChar; SourceBytes: SizeUInt): SizeUInt;
   var
     i,j : SizeUInt;
     w: SizeUInt;
     b : byte;
   begin
     if not assigned(Source) then
     begin
       result:=0;
       exit;
     end;
     result:=SizeUInt(-1);
     i:=0;
     j:=0;
     if assigned(Dest) then
       begin
         while (j<MaxDestChars) and (i<SourceBytes) do
           begin
             b:=byte(Source[i]);
             w:=b;
             inc(i);
             // 2 or 3 bytes?
             if b>=$80 then
               begin
                 w:=b and $3f;
                 if i>=SourceBytes then
                   exit;
                 // 3 bytes?
                 if (b and $20)0 then
                   begin
                     b:=byte(Source[i]);
                     inc(i);
                     if i>=SourceBytes then
                       exit;
                     if (b and $c0)$80 then
                       exit;
                     w:=(w shl 6) or (b and $3f);
                   end;
                 b:=byte(Source[i]);
                 w:=(w shl 6) or (b and $3f);
                 if (b and $c0)$80 then
                   exit;
                 inc(i);
               end;
             Dest[j]:=WideChar(w);
             inc(j);
           end;
         if j>=MaxDestChars then j:=MaxDestChars-1;
         Dest[j]:=#0;
       end
     else
       begin
         while i<SourceBytes do
           begin
             b:=byte(Source[i]);
             inc(i);
             // 2 or 3 bytes?
             if b>=$80 then
               begin
                 if i>=SourceBytes then
                   exit;
                 // 3 bytes?
                 b := b and $3f;
                 if (b and $20)0 then
                   begin
                     b:=byte(Source[i]);
                     inc(i);
                     if i>=SourceBytes then
                       exit;
                     if (b and $c0)$80 then
                       exit;
                   end;
                 if (byte(Source[i]) and $c0)$80 then
                   exit;
                 inc(i);
               end;
             inc(j);
           end;
       end;
     result:=j+1;
   end;

I already got it from Vincent and applied the patch. But:
- diff doesn't grok unicode.
- "joe" doesn't grok it either.
- "kate" & "kwrite" - no grok.
- PFE (Windows) doesn't grok it either.
- Apparently vim converts to plain ascii before editing.
- Last but not least: The FPC Compiler also doesn't grok unicode :-)

So much for unicode :/

Michael.

On Wed, May 4, 2005 12:07, Jonas Maebe said:

Aren't they rather UCS-2 (as used under Win32)? I think that UTF-8 only uses 1 byte for ASCII 0-127, whereas these files have 0 after every single character.

Tomas

I didn't look at this patch, but if it's really UTF-8 then you can open the patch in Emacs (use `C-x <RET> c utf-8 <RET> C-x C-f FILENAME <RET>' to make sure that utf-8 will be understood), make sure you're in diff-mode (`M-x diff-mode' if necessary), then `C-c C-a' (or `M-x diff-apply-hunk') will apply hunks of the patch. I don't know does it solve the problems here (especially since FPC doesn't grok unicode...) but it's a painless way to apply patch encoded in UTF-8.

Except for a small flaw: I don't have emacs installed :-)

Michael.

Hi,

diff doesn't need to, its line-based anyways. If you mean the displaying of unicode, check widechar xterm compilation option (uxterm) [or for text console, the unicode_start script and loadkeys -u].

- "joe" doesn't grok it either.

joe 3 does (joe 3 rocks in other ways too :))

- "kate" & "kwrite" - no grok. - PFE (Windows) doesn't grok it either. - Apparently vim converts to plain ascii before editing. - Last but not least: The FPC Compiler also doesn't grok unicode :-)

I didnt notice it doesnt yet ... but maybe the char<->character weirdness appears with utf-8 (which cannot be fixed except by fixing the language definition itself).

So much for unicode :/

It used to be a hell of a lot of work to set up unicode to work with linux, but nowadays distros support it out of the box :) (except gentoo :P although I have a patch with all the neccessary bits already done open with them)

What distro do you use ?

Michael.

cheers,
Danny

Jonas Maebe wrote

Anyway, here's the plain ascii version of the "-fixed" file. Jonas

I've several questions and comments:

function Utf8ToUnicode(Dest: PWideChar; MaxDestChars: SizeUInt; Source: PChar; SourceBytes: SizeUInt): SizeUInt;

How can -1 be a return value of this function?
I understand that this value can be created as SizeUInt(-1), and that it is a special error code, but it's everything but strict typing as I expect in Pascal code.

Using such conventions every SizeUInt (above: MaxDestChars...) has to checked for -1 before further use, in order to catch error codes from some function that provided the actual value.

With such conventions and functions, with PChar parameters, I suspect that "FPC" really stands for "Fits Pascal or C" :-(

About the source code, I'd appreciate to have an unoptimized version of the code as well, that can be used to validate the manual optimization. Then an Optimization switch could be used to select the optimized or safe version.

if assigned(Dest) then begin while (j<MaxDestChars) and (i<SourceBytes) do

...

end else begin while i<SourceBytes do

...

end;

The ... sections should contain equivalent code, but this is a fragile assumption, impossible to verify in a reliable way.

In a further step of optimization the function should be split into two distinct functions, running even faster due to eliminated checks like assigned(Dest), and without superfluous parameters.

DoDi