How to determine whether a string is UTF-8 or not?

Hi Ant?nio,

Ant?nio wrote

How to determine whether a string is UTF-8 or not?

Distinguish UTF-8 from what?
ANSI, UTF16 or whatever?

Bart

Mainly from ANSI, but whatever.

Antonio

2010/2/27 Bart :

Hello Lazarus-List,

Saturday, February 27, 2010, 5:43:24 PM, you wrote:

A> Mainly from ANSI, but whatever.

You can not determine if a string is an UTF8 one, only if it is an UTF8 well formed sequence. In other words, you can not detect if a string IS an UTF8, only if it does not have UTF8 errors.

You can use the LCLProc FindInvalidUFT8Character function.

There is no way to determine whether a string is in ANSI format or not?

Ant?nio

2010/2/27 JoshyFun :

There is no way to determine whether a string is in ANSI format or not?

In general: No, because every 8bit combination is valid. In specific case: Not every combination makes sense. So you can write heuristics.

You should save/define the encoding of a string. For example all strings in the LCL are UTF-8.

Mattias

I refer to a text loaded in a SynEdit by the user in which I need to make some string manipulations.

Ant?nio

2010/2/28 Mattias Gaertner :

I refer to a text loaded in a SynEdit by the user in which I need to make some string manipulations.

Ah, so not a string, but a whole text. See here

http://wiki.lazarus.freepascal.org/Multiplatform_Programming_Guide#Text_encoding

GuessEncoding checks for UTF8, UTF16 and defaults to the system encoding. It does not yet contain heuristics to distinguish the various Ansi encodings.

Mattias

Not every ansi string is a valid UTF8 string, but every UTF8 string is a valid ansi string. So you can be sure that a string is not a valid UTF8 string, if it's a valid, it could be a UTF8 or an ansi string.

You can make some guess, for latin alphabet (western or eastern), if it is a valid UTF8 and it has several char(195) then probably you have an UTF8 string with several above char(128) characters. But you are just guessing, it could be an UTF8 with no acute or tilde letters, or it could be an ansi string with several char(195).

El 27/02/2010 2:17, Ant?nio escribi?:

Hi, Antonio

Pls check the function I used for check UTF8 string. Hope it helpful

function IsUTF8(UnknownStr:string):boolean;
var
    i    :Integer;
begin
    if length(UnknownStr)=0 then exit(true);
    i:=1;
    while i<length(UnknownStr) do
    begin
        // ASCII
        if  (UnknownStr[i] = #$09) or
            (UnknownStr[i] = #$0A) or
            (UnknownStr[i] = #$0D) or
            (UnknownStr[i] in [#$20..#$7E]) then
        begin
            inc(i);
            continue;
        end;
        // non-overlong 2-byte
        if  (UnknownStr[i] in [#$C2..#$DF]) and
            (UnknownStr[i+1] in [#$80..#$BF]) then
        begin
            inc(i,2);
            continue;
        end;
        // excluding overlongs
        if ((UnknownStr[i]=#$E0) and
              (UnknownStr[i+1] in [#$A0..#$BF]) and
            (UnknownStr[i+2] in [#$80..#$BF]))
        or
            // straight 3-byte
            (((UnknownStr[i] in [#$E1..#$EC]) or
            (UnknownStr[i] = #$EE) or
            (UnknownStr[i] = #$EF))
            and
            (UnknownStr[i+1] in [#$80..#$BF]) and
            (UnknownStr[i+2] in [#$80..#$BF]))
        or
             // excluding surrogates
             ((UnknownStr[i]=#$ED) and
              (UnknownStr[i+1] in [#$80..#$9F]) and
              (UnknownStr[i+2] in [#$80..#$BF])) then
        begin
                inc(i,3);
                continue;
        end;
        // planes 1-3
        if ((UnknownStr[i]=#$F0) and
            (UnknownStr[i+1] in [#$90..#$BF]) and
            (UnknownStr[i+2] in [#$80..#$BF]) and
            (UnknownStr[i+3] in [#$80..#$BF]))
        or
        // planes 4-15
           ((UnknownStr[i] in [#$F1..#$F3]) and
            (UnknownStr[i+1] in [#$80..#$BF]) and
            (UnknownStr[i+2] in [#$80..#$BF]) and
            (UnknownStr[i+3] in [#$80..#$BF]))
        or
        // plane 16
           ((UnknownStr[i]=#$F4) and
            (UnknownStr[i+1] in [#$80..#$8F]) and
            (UnknownStr[i+2] in [#$80..#$BF]) and
            (UnknownStr[i+3] in [#$80..#$BF])) then
        begin
            inc(i,4);
            continue;
        end;
        exit(false);
    end;
    exit(true);
end;

2010/2/27 Ant?nio

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20100303/2baa419a/attachment.html>

Hi, Antonio Pls check the function I used for check UTF8 string. Hope it helpful

You combine a IsText (no special characters in #0-#31) and IsUTF8 - good idea.

function IsUTF8(UnknownStr:string):boolean;

Maybe better name it IsUTF8Text ?

#12 is a valid character too in texts (form feed).

Mattias

Hello Lazarus-List,

Wednesday, March 3, 2010, 12:24:35 AM, you wrote:

RH> Pls check the function I used for check UTF8 string. Hope it helpful
RH> function IsUTF8(UnknownStr:string):boolean;

Well, there is a lot of UTF8 strings that do not pass your checks ;) If you remove low ascii control chars what happend with UTF8 control chars ?

RH> var
RH> i :Integer;
RH> begin
RH> if length(UnknownStr)=0 then exit(true);
RH> i:=1;
RH> while i<length(UnknownStr) do
RH> begin
RH> // ASCII
RH> if (UnknownStr[i] = #$09) or
RH> (UnknownStr[i] = #$0A) or
RH> (UnknownStr[i] = #$0D) or
RH> (UnknownStr[i] in [#$20..#$7E]) then RH> begin
RH> inc(i);
RH> continue;
RH> end;
RH> // non-overlong 2-byte
RH> if (UnknownStr[i] in [#$C2..#$DF]) and RH> (UnknownStr[i+1] in [#$80..#$BF]) then RH> begin

It should crashes here with strings like:

var
 s: string;
begin
 s:=$C2;
 IsUTF8(s);
end;

which is not valid UTF8.

RH> // excluding surrogates
RH> ((UnknownStr[i]=#$ED) and
RH> (UnknownStr[i+1] in [#$80..#$9F]) and RH> (UnknownStr[i+2] in [#$80..#$BF])) then

Surrogates are not UTF8 valid codepoints.

Hi, JoshyFun

Thanks for pointing out the bug in my coding, yes you are right. I forgot to put some checking before every inc(i,k) and continue; there should a judgement statement
*if i>length(UnknownStr) then exit(false);*

2010/3/3 JoshyFun

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20100303/5bcd2fe8/attachment.html>

That's correct, a possible workaround were the use of PChars, which can safely access the appended #0.

I'd suggest a state machine or the like for the implementation:
   while True do
     case p^ of
     #0: break; //done, okay if past the end of the string
     #8, #10, #12, #13, ' '..#$7E: inc(p); //okay
     #$C0..#$DF: ... //2 bytes
     #$E0..#$EF: ... //3 bytes
     #$F0..#$F4: ... //4 bytes
     else exit(False); //not valid text
     end;

DoDi

Hello Lazarus-List,

Wednesday, March 3, 2010, 2:05:48 PM, you wrote:

RH> Hi, JoshyFun

RH> Thanks for pointing out the bug in my coding, yes you are right. I forgot to RH> put some checking before every inc(i,k) and continue; there should a RH> judgement statement
*if i>>length(UnknownStr) then exit(false);*

You can check the UTF8ToUnicode function in FPC sources located at:

rtl\inc\wstrings.inc