rtl/inc/ustrings.inc

function UTF8ToUnicode(Dest: PUnicodeChar; MaxDestChars: SizeUInt;
Source: PChar; SourceBytes: SizeUInt): SizeUInt;

a) it has "If (PreChar13) and FALSE then" and later some big block. with a comment which tells that "and FALSE" is on purpose and block is ignored. ?????

b) after I removed "that block" I got such trimmed src

   while (OutputUnicode<MaxDestChars) and (InputUTF8<SourceBytes) do
     begin
       IBYTE:=byte(Source[InputUTF8]);
       if (IBYTE and $80) = 0 then
         begin
           //One character US-ASCII, convert it to unicode
           if IBYTE = 10 then
             begin
               Dest[OutputUnicode]:=WideChar(IBYTE);
               inc(OutputUnicode);
               PreChar:=IBYTE;
             end
           else
             begin
               Dest[OutputUnicode]:=WideChar(IBYTE);
               inc(OutputUnicode);
               PreChar:=IBYTE;
             end;
           inc(InputUTF8);

here we have 2 identical small blocks after "if IBYTE=10". ?????

Alexey Torgashin

Hello,

As the author of that piece of code, UTF8 recommends expanding LF line endings to CR+LF but this will generate some troubles and to try to 100% conform the spec. the conversion has been added, but disabled so code maintainers can easily remove them if needed. If you remove the piece of code about #13 you can safely remove the check about #10 which is now nonsense so you end with:

-------------------
while (OutputUnicode<MaxDestChars) and (InputUTF8<SourceBytes) do
     begin
       IBYTE:=byte(Source[InputUTF8]);
       if (IBYTE and $80) = 0 then
         begin
           //One character US-ASCII, convert it to unicode
           Dest[OutputUnicode]:=WideChar(IBYTE);
           inc(OutputUnicode);
           PreChar:=IBYTE;
           inc(InputUTF8);
------------------

Have a nice day.

 So can you COMMENT that dead part with "and FALSE" and dead part II with "if ... =#10"? It will make things faster for people!

Alexey

AlexeyT wrote

 So can you COMMENT that dead part with "and FALSE" and dead part II with "if ... =#10"? It will make things faster for people!

It will make things faster for people as well if you point at the right unit or mail on the right mailing list.

This code is already commented out in FPC trunk.

Michael.

Michael Van Canneyt via fpc-pascal
schrieb am Do., 29. Okt. 2020, 09:36:

Though one could probably exclude the check at line 1981 of rtl/inc/ustrings.inc as well as the two existing branches will result in the same...

Regards,
Sven