Hi,

Using FPC 2.6.1 and Laz trunk on Windows, there is a better way to not to do such codes below?

Examples
1- I'm on the LCL and I have an Edit (edtFileName):
var
  lFileName: string;
  lStrings: TStrings;
begin
  lStrings := TStrings.Create;
  try
    lFileName := Utf8ToSys(edtFileName.Text);
    lStrings.SaveToFile(lFileName);
  finally
    lStrings.Free;
  end;
end;
2- If I want to add the path delimiter using an Edit (edtPath)
var
  lPath: string;
begin
  lPath :=
SysToUtf8(IncludeTrailingPathDelimiter(Utf8ToSys(edtPath.Directory)));
  edtPath.Directory := lPath;
end;

Marcos Douglas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20120407/a4520ec3/attachment.html>

The path delimiter is the same in all system encodings.

   lPath :=IncludeTrailingPathDelimiter(edtPath.Directory);

edtPath.Directory := lPath; end;

Mattias

Ops... you're right.
I was confused because IncludeTrailingPathDelimiter is part of RTL and it not works with UTF8. The code uses Length and Result[l]... so, I thought this is would be problem.

Well, this problem do not exists, but the names of files etc still a problem.

Marcos Douglas

Marcos Douglas wrote

Ops... you're right. I was confused because IncludeTrailingPathDelimiter is part of RTL and it not works with UTF8. The code uses Length and Result[l]... so, I thought this is would be problem.

Some routines made for Ansi work automatically with UTF-8 without changes. Simply using Length and Result[] by itself might sometimes be correct, if those are ment to access only chars <= #$7F like #10 and #13 are.

Lenght function do not works with UTF8 correctly.

procedure TForm1.Button1Click(Sender: TObject);
begin
  Edit1.Text := '?b?';
  ShowMessage(IntToStr(Length(Edit1.Text)));
end;

The result is 5 not 3, but you did know this of course.

Marcos Douglas

Marcos Douglas wrote

The result is 5 not 3, but you did know this of course.

If you study carefully UTF-8 and follow the code of the RTL routines you will find out that some routines work with UTF-8 without changes, regardless of the fact that Length returns the number of bytes in the string for ansistring.

Both are valid results. The first is needed when you copy the string (bytes to move) the second to display the string (since it will probably be around 3 glyphs (yes, I know that is not guaranteed))

Agree.
But I did not wanted to check every single function if will work or not. That would be very dangerous.

Marcos Douglas

Yes, both are valid and do not have problem with the RTL. It was designed to work using AnsiString, we know that... but these problems, on Windows, is very dangerous to developer, don't you think?

Unfortunately I did not followed all discussion about Unicode, cpstrnew, FPC trunk, etc because I could not help at this level so, I'm waiting a solution since.

Marcos Douglas

The Lazarus developers already did this for you for the Lazarus sources.

Mattias

For the Lazarus sources yes, but not for RTL.
I still think about:
DirectoryExists or DirectoryExistsUTF8
ForceDirectoriesUTF8 or ForceDirectories
Pos or UTF8Pos
etc

Depends what part of code you are...

Marcos Douglas

That's why FPC postponed making everything unicode till proper support was added.

Yes, it's a real pain. I once used the Windows API function "CopyFile" and because I knew that this API function uses Ansi coding I prepared my code like this. But after some months my program suddenly crashed because someone named a file my program reads with german umlauts. I then discovered that a CopyFile function exists within the LCL too! Who did build this trap? You cannot use the same name for an existing API function and then silently convert it to UTF8 in the background (not even mention this on
http://lazarus-ccr.sourceforge.net/docs/lcl/fileutil/copyfile.html ). This kind of issue hits me very often.

Yes. For Unicode encoding we would need new functions to distinguish between number of bytes and number of (visible) glyphs:

LengthInBytes()
LengthInGlyphs()

because sometimes you need the first and sometimes (but seldom) you need the latter. For one byte encodings both are identical but not for Unicode. Just think of LENGTH as beeing the first version.

Because this I do not like use functions/procedures without a prefix or unit (as a prefix).
IMHO, every function should be used with a prefix (unit or real prefix)... but this is other discussion. ;-)

You cannot use the same name for an existing API function and then silently convert it to UTF8 in the background (not even mention this on http://lazarus-ccr.sourceforge.net/docs/lcl/fileutil/copyfile.html ). This kind of issue hits me very often.

Do your own functions (like I said above) or use unit.function syntax.

Marcos Douglas

CopyFile was added 2002-12-23 to the LCL FileCtrl with two string parameters. The Windows CopyFile takes two PChar and one boolean, so there was no chance to use the wrong one.
In 2004-09-27 an overloaded CopyFile was added to the LCL to preserve time. This can indeed be misused when passing two constants as file names. I never saw a bug report/note about it, so I guess this is not a common case.

The change of all LCL strings to UTF-8 was all but silent.

This kind of issue hits me very often.

Where else?

Mattias

Length = size in bytes
UTF8Length = number of code points

There is no generic function LengthInGlyphs, because this depends on the font engine and some code points do not have a glyph.

Mattias

In the future, we will transform UTF8Length deprecated, only use Length function and the FPC will know wich the correct codepage use, or it is not going happen never? =)

Marcos Douglas

Marcos Douglas wrote

Agree. But I did not wanted to check every single function if will work or not. That would be very dangerous.

You could start a section in the wiki and help us document this. A section in this page:

http://wiki.lazarus.freepascal.org/LCL%20Unicode%20Support

Also lot of routines have UTF-8 equivalents in various LazUtils units.

For UTF-8 and UTF-16 you will always need both size in bytes and number of codepoints.

My hope some day we can all forget codepages.

Mattias

Such problems may (should) go away with the new Unicode- and AnsiString types, where AnsiString contains an Encoding field. Then the conversion between UTF-8 and the system codepage are done automatically, whenever required, and the xyUTF8 functions can be dropped then.

I discourage the use of UTF8Pos, in detail together with the new (encoded) AnsiString type. When such a string is auto-converted, for some reason, the index returned by UTF8Pos will become invalid. This is one of the downsides of encoded strings, which suggest to use UnicodeString in future code. Delphi enforced that move, by changing String and Char to UnicodeString and WideChar, and Delphi compatibility propagated that pressure into FPC. The continued use of UTF-8 strings (AnsiString) will result in a speed and memory usage penalty, unless the system codepage is UTF-8. If your code only contains String type strings, not AnsiString or UTF8String, then all your strings will become UnicodeStrings (UTF-16), for which the xyUTF8 functions are either inapplicable or will result only in superfluous implicit string conversions.

Now every user has the choice to stay with a specific FPC/Lazarus version, that does not yet support the new string types, or to drop UTF-8 strings in favor of the new UTF-16 strings. Since most code has to deal with the Unicode BMP (BasicMappingPage) only, the difference between the length of an string in (UTF-8)chars and characters has gone away with UTF-16. Do you really see a need for finding the position of a non-BMP character in an string, and changing exactly that character in the string? Then you are on the safe side by using StringReplace, which already worked with UTF-8 and will continue to work with UTF-16 and whatever other encoding. The use of Char variables has been dangerous already with UTF-8, where exotic ("astral") characters can consist of up to 6 bytes. In so far I don't understand why Delphi now uses WideChar for Char, instead of UnicodeChar, where it is guaranteed that every codepoint (except ligatures and similar text-processing stuff) can be stored in a UnicodeChar variable.

DoDi

It should be mentioned that Unicode allows for different encodings of composed/decomposed characters. E.g. '?' can be stored as '?' (single composed codepoint) or as '?e' (two decomposed codepoints). Even if both encodings look the same on screen, Pos (or UTF8Pos) will only find the encoding as given in the search string, and it has to be specified what LengthInGlyphs really should return - the number of really visible glyphs, what in case of ligatures etc.?

Every user has to know which kind of "length" he really wants to get:
- number of bytes for storage in a fixed-size variable or streaming
- number of glyphs for length-restricted user input
- number of pixels for GUI layout (TextWidth)
...

DoDi

OK. I will read and try to help where I can.

Marcos Douglas

+1

Marcos Douglas

When the new Unicode and AnsiString types (that contains an Encoding field) arrive to us, users of FPC 2.6.1? Is this done?

Marcos Douglas

You do know that WideChar and UnicodeChar are the same? Namely two byte characters.

Regards,
Sven

This is part of 2.7.1 and will become part of the next main release.

Regards,
Sven

AFAIK there is no UTF-32 string type.
The LCL uses cardinal or dword for codepoints.

Mattias

You do know that WideChar and UnicodeChar are the same? Namely two byte characters.

Hmm, you are right, but somewhere I saw it declared differently. Does there exist another char type of 4 bytes?

DoDi