Hi and much kudos for those who made the UTF8 RTL.

GetCurrentDir, FindFirst, FileExist, TStringList, etc. all work well. :)

ParamStr is not yet converted and only supports system codepage. I would like to help improving it.

Has someone already started it?

Mattias

In our previous episode, Mattias Gaertner said:

In {$mode delphiunicode} it returns unicodestring.

A UnicodeString function would be sufficient.
But it seems this uses the argv PChar in system codepage.

Paramstr:=UnicodeString(Argv[Param])

It needs a unicode version of Argv.
For example a function like GetCommandLineW could be used to create a ArgvW. Is this the right approach?

Mattias

Mattias Gaertner wrote

Hi and much kudos for those who made the UTF8 RTL.

Thanks, but there is no UTF-8 RTL.

GetCurrentDir, FindFirst, FileExist, TStringList, etc. all work well. :)

They are not guaranteed to use UTF-8, and you must not assume that they do (and TStringList in particular has not been changed in any way).

See http://wiki.freepascal.org/FPC_Unicode_support for the details, and http://wiki.freepascal.org/FPC_Unicode_support#RTL_changes for the list of RTL routines that currently work correctly with arbitrary code pages (or that at least should do so).

ParamStr is not yet converted and only supports system codepage. I would like to help improving it.

It will always only support system code page on Unix platforms, because everything that comes from the shell must be assumed to be in that code page. On Windows, it can be changed though.

Has someone already started it?

Not that I know.

Jonas

On 18/11/14 16:59, Mattias Gaertner wrote: > Hi and much kudos for those who made the UTF8 RTL. Thanks, but there is no UTF-8 RTL.

That's what I thought too a week ago.
FPC 2.7 made an old dream come true. :)

Yes, this page is great. I had to read it twice to understand the flexibility of the new RTL.

True.
From Lazarus point of view most Unicode problems are on Windows. And now most of them are solved. FPC 2.7 will be a big improvement for Lazarus users.

Mattias

In our previous episode, Mattias Gaertner said:

As Jonas said, not using utf8 on Windows. A TStringlist with a ansistrings in them passed to an RTL routine will be seen as ansi.

Marco van de Voort wrote

As Jonas said, not using utf8 on Windows.

No, that's not what I said. There is no problem with using UTF-8 on Windows.

A TStringlist with a ansistrings in them passed to an RTL routine will be seen as ansi.

That is incorrect (although right now there are no RTL routines that accept stringlists and that are also codepage-aware).

Jonas

In our previous episode, Jonas Maebe said:

> As Jonas said, not using utf8 on Windows. No, that's not what I said. There is no problem with using UTF-8 on Windows.

As long as you explicitely use utf8string.

But I meant that even if you use utf8string in many places as soon as you stuff it in a container like tstringlist, that is gone. (forced ansi conversion, since tstringlist's interface is defined using plain string(0))

An ansistring with a dynamic code page of UTF-8 will also work fine with the adapted RTL routines. What will of course not work is an ansistring containing UTF-8 data while its dynamic codepage is different from CP_UTF8 (that includes CP_ACP in case DefaultSystemCodePage happens to be CP_UTF8), such as the current Lazarus convention.

That is however wrong on all platforms, not just on Windows (no one must ever assume that the system code page on a unix platform is UTF-8 either; it's the same as assuming that the keyboard layout is qwerty or that the processor is little endian).

If you use utf8string, yes, but then the problem indeed occurs when the string is added to the tstringlist, not when the tstringlist is passed to an RTL routine (which is how I understood your initial comment).

Jonas

The RTL does not use UTF8String, so this would create a lot of codepage checks and/or conversions.

For Lazarus there is a better solution:
The RTL on Windows now uses the "W" functions and the AnsiString and ShortString are encoded in CP_ACP. Changing the DefaultSystemCodePage to CP_UTF8 does the trick. All UnicodeString to AnsiString assignments convert to UTF8 and vice versus.
This breaks code that still uses the non Unicode functions or reads/writes non ASCII strings in system code page. Which is why this change will be optional in Lazarus.
The whole Lazarus code didn't need a change for this, so I guess big parts of users code will happily run as well. With the exception, that aStringList.LoadFromFile now works with Unicode paths, instead of only characters from the system code page.
Users can then get rid of many UTF8 calls like UTF8ToAnsi or FileExistsUTF8. This will make porting to the coming UnicodeString RTL easier as well.

True. Although it is rare nowadays that someone uses non ASCII characters on a non UTF-8 Unix. Especially for graphical systems.

Mattias

"Mattias Gaertner" wrote

The RTL on Windows now uses the "W" functions and the AnsiString and ShortString are encoded in CP_ACP. Changing the DefaultSystemCodePage to CP_UTF8 does the trick.

AFAIK we don't use the "W" functions yet on non-CE Windows.

Regards,
Sven

We do (in trunk) for the RTL routines mentioned at http://wiki.freepascal.org/FPC_Unicode_support#RTL_changes

Jonas

I tested with FPC 2.7.1 from last week with file names containing a mix of German Umlaute and Hebrew letters.
Lazarus Find declaration takes me to functions calling internally the UnicodeString versions. Maybe codetools don't have the right defines for the IFDEFs, but the results fit the code I found.

Mattias

In our previous episode, Sven Barth said:

As Jonas said, most file I/O does. File execution doesn't yet.

Unit Windows can be built experimentally with -W default by defining FPC_OS_UNICODE

Marco van de Voort wrote

But I meant that even if you use utf8string in many places as soon as you stuff it in a container like tstringlist, that is gone. (forced ansi conversion, since tstringlist's interface is defined using plain string(0))

AFAI Understand (having done some decent research on this) the "STringList forcing time consuming conversions) with Codepage aware Strings is only solvable by inventing an additional fully-code-dynamic String type usable as the basic interface for TStrings. IMHO, this is rather easily doable, but not Delphi compatible (See some former discussions.)

-Michael

FPC_OS_UNICODE

Ah, right. So I mixed this up with the Windows unit :)

Regards,
Sven

In our previous episode, Michael Schnell said:

Not old Delphi compatible. One can go new delphi compatible, and make everything 2-byte as much as possible.

Marco van de Voort wrote

Not old Delphi compatible. One can go new delphi compatible, and make everything 2-byte as much as possible.

Of course I did mean compatibility to "New" Delphi Strings.

But here (seemingly) TStrings (and with this TStringList) works on a single pre-defined (2-Byte) encoding. And hence any other encoding (as well 2 Byte as 1 Byte) needs time consuming conversions in and out.

Changing this would need a new "versatile" String Type that is not available in Delphi and hence not compatible.

-Michael

What has this to do with UTF8 RTL?

Mattias

In our previous episode, Michael Schnell said:

Which other 2 byte encodings are you talking about? Anyway the solution is simple: don't do that, keep it utf16.

TStringlist is not for the performance sensitive in the first place.

In our previous episode, Mattias Gaertner said:

> Changing this would need a new "versatile" String Type that is not > available in Delphi and hence not compatible. What has this to do with UTF8 RTL?

Nothing. The versatile string type is vaporware. There is no designdocument, nothing.

Marco van de Voort wrote

The versatile string type is vaporware. There is no designdocument,

Do you want me to create a document ? I could easily do it.

But as it would need compiler magic to work with, I don't have any chance to do the necessary patches. So I think it would be not nice at all to force any documentation on you without intending to do the work myself.

-Michael

Your argument for not doing something understand that you cannot some of the work involved. Neither can I. But if you don't specify clearly what you want, we definitely cannot work on it. (it is VaporWare as Marco puts it)

By creating for example an entry in the wiki, you achieve at least 2 goals:
1. There is a concrete spec to work with (if work ever starts).
2. You invite people to think about it and comment on it.

Yes, it may be so that you create an entry which no-one will ever work on. This is always a possibility, and it is in a sense 'work wasted', but that is the danger of investing your time.

But as it is, without anything for anyone to work with, your ideas are simply dead-born childs.

Michael.

Michael Van Canneyt wrote

But as it is, without anything for anyone to work with, your ideas are simply dead-born childs.

Right you are and Marco is right that it is nothing but Vaporware right now.

I did post a kind of "documentation draft" here to provide a base to start a discussion on.

The discussion never led to a feasible point, but as there also was an agreement that it would not be doable of viable to go this way, and as you seem to be interested, I will do a Wiki article in the weekend.

Thanks,
-Michael

In our previous episode, Michael Schnell said:

On 11/21/2014 09:53 AM, Marco van de Voort wrote: > The versatile string type is vaporware. There is no designdocument, Do you want me to create a document ? I could easily do it.

The point was that even though odds of implementation are low, even with document, keep trying to push an empty one liner is even more useless.

But as it would need compiler magic to work with, I don't have any chance to do the necessary patches. So I think it would be not nice at all to force any documentation on you without intending to do the work

You don't need to be able to create compiler magic, but to design language extensions you do need to know what is possible with it and what not. Despite the word "magic", not everything is possible or doable.

Mattias Gaertner wrote

On Tue, 18 Nov 2014 18:17:25 +0100 > > Thanks, but there is no UTF-8 RTL. That's what I thought too a week ago. FPC 2.7 made an old dream come true. :)

Nevertheless, please stop calling it the UTF-8 RTL. It will give people the wrong impression, because there is no and may well never be an UTF-8 RTL. It is no way whatsoever specific to UTF-8, nor will it automagically work correctly with UTF-8 encoded data in all scenarios (e.g. when DefaultSystemCodePageCP_UTF8, or when the file system API doesn't support all characters that can be encoded using UTF-8, ...) or on all platforms (some platforms don't even support multiple code pages).

Jonas

Hmm, maybe you have a point there.
It is similar to normal RTL and RTL with CurrencyString:='Euro'. When I say "Euro RTL" it does not work correctly in all scenarios, but I'm sure some people think it does.

I doubt that I can come up with a name, that everyone understands correct, but I have to call it something.

What about "RTL with UTF8 as default"?

Mattias

That seems wrong, since UTF-8 is not the default on Windows ?

Let's try to understand first why do you insist on the "UTF-8" in the name ?

Maybe "UTF-8 aware" is better, if you really want the UTF-8 in the name.

Michael.

In our previous episode, Mattias Gaertner said:

It is not, and neither is it "aware". It is only so when you set defaultsystemcodepage to utf8 and ignore the problems that causes. That is not intended functionality.

It is UTF-8 aware in the sense that now you can safely define a 1-byte string containing UTF-8 and expect the compiler to do the necessary conversions more or less automatically when needed, such as when passing it to a routine that expects widestrings.

That was simply impossible prior to the work of Jonas and others.

Michael.

Maybe there is a misunderstanding. At least I can't follow you here.

I started the thread about ParamStr, which only supports the system codepage. I would like to improve it so that it supports DefaultSystemCodepage. Or at least add an Unicode version of ParamStr.
The reason I need this, is that when I set
DefaultSystemCodepage to CP_UTF8, most RTL functions work with UTF8 as default.
I also want to update the UTF-8 wiki pages. For example "If you use the <insert name here> then ...". For this I need a short name.

Some people has called the RTL with UnicodeString the "Unicode RTL", and the Ansistring RTL with system codepage "Ansi RTL". I thought "UTF8 RTL" is analog, short and unambiguous. Obviously I was wrong.

(And, yes, I know, that all three names "Unicode|Ansi|UTF8 RTL" are not 100% correct from a technical point of view.)

Mattias

In our previous episode, Michael Van Canneyt said:

> It is not, and neither is it "aware". It is only so when you set > defaultsystemcodepage to utf8 and ignore the problems that causes. That is > not intended functionality.

To make things clear:
I meant in the way Mattias proposed, continueing making the default "string" type utf8 on Windows. Utf8string is fine, but limited.

That basically perpetuates the current hack, just slightly more elegant.

That being said, FPC is not offering a complete solution yet, and 2.8.x will be a transition version. (or one in a series of transitional versions)

It is certainly progress, and hopefully a start on at least removing the OS specific aspects of Lazarus *utf8 routines.

I would at least try to get as far as possible without changing defaultsystemcodepage (etc) though.

I don't know about that.

It is a perfectly valid solution to choose UTF-8 as the unicode encoding of choice. Now at least the compiler makes this a more viable decision. So from this point on, I would no longer speak of a "hack".

That being said, FPC is not offering a complete solution yet, and 2.8.x will be a transition version. (or one in a series of transitional versions)

Correct.

Fully agreed.

Michael.

In our previous episode, Mattias Gaertner said:

And the 2-byte unicode version exists, in unit uuchar. (the "objpas" of $mode delphiunicode). For now, simply make a utf8 wrapper that returns an utf8string.

Some people has called the RTL with UnicodeString the "Unicode RTL", and the Ansistring RTL with system codepage "Ansi RTL".

Well, that's what Windows calls them that (-W are unicode, -A are ansi). Delphi follows that terminology (D2007- being ansi, D2009+ being unicode).

I thought "UTF8 RTL" is analog, short and unambiguous. Obviously I was wrong.

The RTL leans somewhat to the prefered encoding on each target, so 1-byte on *nix and 2-byte on Windows. That means that there is no real utf8 support on Windows other than generic codepage aware string type (that goes for all 1-byte encodings).

Setting defaultsystemcodepage will make all autoconversions to ansistring(0) return utf8, so also when calling e.g. unit windows functions. I think it would be very wise to be careful with that, and have an extensive trial period.

You might want to keep the current -utf8 routines as mere codepage correcting wrappers.

(And, yes, I know, that all three names "Unicode|Ansi|UTF8 RTL" are not 100% correct from a technical point of view.)

The filesystem routines are now encoding agnostic. But that assumes you use a type that the compiler knows the associated encoding.

But filesystem routines are only a small part of the system libraries.

In our previous episode, Michael Van Canneyt said:

So now we support changing defaultsystemcodepage formally by endusers? I was not made aware of that.

And aside from formal status, that is IMHO a hack, and an incompatible (to both old and new delphi) to start with.

Now at least the compiler makes this a more viable decision. So from this point on, I would no longer speak of a "hack".

It moves removes the conversions from a few RTL routines, but keeps the "insert conversion here" barrier to all other interfacing to the system firmly intact. And I already mentioned it is incompatible.

I assume this will also make default I/O utf8 etc, which makes no sense.

(I would suggest keeping them for now, just make them simple codepage wrappers in case the defaultsystemcodepage doesnt' work out).

Whether or not to change defaultsystemcodepage is entirely irrelevant to the discussion: the fact that now lazarus *can* use a real UTF8String type.

One does not necessarily imply the other.

How they choose to use this UTF8 type (i.e. how to set the various variables controlling the behaviour of the RTL), is IMHO a second design decision, which comes AFTER the decision to use UTF8..

But anyway, it's not my intention to argue about this. There is progress, that is what matters.

Michael.

Marco van de Voort wrote

So now we support changing defaultsystemcodepage formally by endusers? I was not made aware of that.

There is the SetMultiByteConversionCodePage() routine in the interface of the system unit that does exactly that. It's kind of strange to have such a routine if using it is not supported.

And aside from formal status, that is IMHO a hack, and an incompatible (to both old and new delphi) to start with.

That routine comes from Delphi and does the same there.

It depends on what you mean by "default I/O". It won't change standard input/output to UTF-8 on Windows, as every textrec has its own codepage settings and those for standard input/output are initialised with concrete code page numbers on startup (see OpenStdIO in rtl/inc/textrec.inc, and Win32GetStandardCodePage() in rtl/win/syswin.inc.

On Unix, it will change the code page of stdio, because there the stdio textrecs are initialised with the value of defaultsystemcodepage (DefaultGetStandardCodePage() in rtl/inc/ustrings.inc), which is still CP_ACP at that point. We could fix this by adding code to the initialisation of cwstring to set the codepage of those textrecs to the concrete code pages though (which we probably should do anyway).

Jonas

Mattias Gaertner wrote

I started the thread about ParamStr, which only supports the system codepage. I would like to improve it so that it supports DefaultSystemCodepage. Or at least add an Unicode version of ParamStr.

We need both, as Marco mentioned. The main issue is that on Unix platforms, to fix it properly you must interpret the parameters according to "DefaultSystemCodePage at startup, or in case a separate widestring manager is used then right after that one was initialised". Maybe we need another Default*CodePage variable which indicates the "real" system code page...

I also want to update the UTF-8 wiki pages. For example "If you use the <insert name here> then ...". For this I need a short name.

"If you use FPC 2.7.1 or later then ..."

Jonas

Michael Van Canneyt wrote

> > What about "RTL with UTF8 as default"?

I am nearly done with the would-be "designdocument" Wiki text, you seemed to want to see done on a suggestion for an extension of the string type variants, that allow for e.g. (1) TStrings siblings and (2) GUI libraries to work for all string encoding "brands" without unnecessary conversions. I'll extend the examples to the RTL...

-Michael

Marco van de Voort wrote

I meant in the way Mattias proposed, continueing making the default "string" type utf8 on Windows.

As with Windows, the OS requires API access with UTF-16 encoded strings this would force lots of auto-conversions, not only in the RTL but with any GUI Library, as well.

Same would be true for making the default "string" type utf16 (or whatever) on Linux.

OTOH using different default "string" types on different platforms auks for severe compatibility problems when trying to do portable programs.

Catch 22 :-( .

(I myself vote for killing the type "String" completely and allow/force the user to do a Type definition for "String" himself that is used in his code, e.g. in a central unit.)

-Michael

How do you reconcile this with the fact that pascal should be easy, and it should be usable for teaching ?

In my personal experience, 99% of people really do not care or even know about encodings. The reason is quite simple: lots of programs are written in 1 language.

Let's not get carried away, please... keep it simple.

Michael.

I do know that this is a decent argument

That is why I put my text in brackets with an "I myself vote for...".

Maybe a way to allow the user to define the project-wide default encoding branding for the unqualified type "String" can be invented (e.g. OS-Default if not explicitly set)

-Michael

On 23/11/14 17:23, Mattias Gaertner wrote: [...] Maybe we need another Default*CodePage variable which indicates the "real" system code page...

+1

> I also want to update the UTF-8 wiki pages. For example "If you use the > <insert name here> then ...". For this I need a short name. "If you use FPC 2.7.1 or later then ..."

I meant something short for
"FPC 2.7.1+ with
SetMultiByteConversionCodePage(CP_UTF8);
SetMultiByteRTLFileSystemCodePage(CP_UTF8);"

Mattias

Obviously this asks for a versatile String type (not existing in Delphi) for the interface of any code that is accessed by but not included in the user project. I'll post the wiki paper on this tomorrow.

-Michael

[...] And the 2-byte unicode version exists, in unit uuchar. (the "objpas" of $mode delphiunicode). For now, simply make a utf8 wrapper that returns an utf8string.

At the moment uuchar ParamStr only contains a typecast:

    if (Param=0) then
      Paramstr:=System.Paramstr(0)
    else if (Param>0) and (Param<argc) then
      paramstr:=UnicodeString(Argv[Param])
    else
      paramstr:='';

Unit LazUTF8 contains a procedure SetupCommandlineParametersWide, that uses GetCommandLineW and creates an ArgW array, similar to the Argv array.
Should I adapt that for uuchar?

Mattias

Mattias Gaertner wrote on ma, 24 nov 2014:

Probably, yes. However:
* since we're close to branching/releasing 2.8, I am not very much in favour of still modifying core RTL routines like this for inclusion in 2.8 (this also goes for the defaultformatsettings related patch you posted to the bug tracker). I understand that this will make it harder to use the current RTL with Lazarus, but that can be improved for 2.8.2 yet. It's more important (to me) that what's there is stable, than supporting everything but the kitchen sink and having annoying bugs.
* make functionality codepag-aware should be done for all platforms at the same time, otherwise it will become impossible to write cross-platform programs that have similar behaviour everywhere (and the list at http://wiki.freepascal.org/FPC_Unicode_support#RTL_changes will become unreadable, just like the documentation). In general you need a specific implementation for Windows and for Unix, and for all other platforms (which don't have codepage support yet) it's a mechanical change that is more or less identical. See the commits to the cpstrrtl branch for examples.

Jonas

I guessed so. No problem. I can add workarounds for these issues in the Lazarus code.

Ok.

Mattias

Michael Schnell wrote

. I'll post the wiki paper on this tomorrow.

Please see a preliminary version of the text ->
http://wiki.freepascal.org/not_Delphi_compatible_enhancement_for_Unicode_Support#Three_more_RAW_types .

(Please use a new thread for any discussion on this... )

-Michael