Hello,

Following from a discussion on mac-pascal, I'd like to propose a solution for Unicode support.

I know that hundreds of messages were already spent on the topic, but was there any conclusion in the end?

Basically I think that we could support both choices (utf-8 and unicode string or whatever it is called) by duplicating methods, procedures and TStrings.

We could change:

Function FileOpen (Const FileName : string; Mode : Integer) : THandle;

into

Function FileOpen (Const FileName : utf8string; Mode : Integer) :
THandle; overload;
Function FileOpen (Const FileName : unicodestring; Mode : Integer) :
THandle; overload;

and similarly for other places and everyone should be happy. One of the versions could be the real one and the other could use a conversion. Each platform could decide for itself which is the real version, so it would avoid unnecessary conversions.

It is not really that different from what Delphi did. They changed: Function FileOpen (Const FileName : string; Mode : Integer) : THandle; into: Function FileOpen (Const FileName : unicodestring; Mode : Integer) : THandle;
We just support 1 more option.

The ansi version I would simply let it die. It's time people migrate to Unicode anyway. The change from ansi to utf-8 is very easy.

TStrings seams a special case, because any implementation which supports both will be extremely inefficient, so for it I propose having 2 classes:

TStrings (for utf-8)
TWideStrings (for delphi-style)

For all others I think just overloading some methods should be fine.

Of course it is not a perfect solution, but I am convinced that there is no perfect solution here.

Felipe Monteiro de Carvalho wrote

Hello,

[snip]

The ansi version I would simply let it die. It's time people migrate to Unicode anyway. The change from ansi to utf-8 is very easy.

About the rest I will not comment. But the above certainly will not happen.

The ansistring version must remain for backwards compatibility.

Current strategy on fpc core seems to be to have 2 RTLs:

One with unicode string, one with ansistring.
They will have the same code, but will be compiled twice, each time with a different compiler define to decide which version it must be.

Michael.

Graeme Geldenhuys, sometime, written that string type shoud be depended of plataform. I agree with him, but I don't know if this is the easiest way... but this is be best way to codify our programs, don't you think?

Marcos Douglas

Marcos Douglas wrote

Graeme Geldenhuys, sometime, written that string type shoud be depended of plataform. I agree with him, but I don't know if this is

+1 ;-)

the easiest way... but this is be best way to codify our programs, don't you think?

UnicodeString should meant UTF-8 on Linux, *BSD, MacOSX etc Unicodestring should mean UTF-16 on Windows.

That way you will have the best performance on each platform, because the string type is native to each platform.

All string function in the RTL should be unicode aware, and not make stupid assumptions as is done so often with AnsiString (the whole 1 byte being 1 char).

PS:
I still can't believe Delphi was that stupid making the assumption that Unicode (as in UnicodeString) = UTF-16. Especially now that they are starting to support other platforms too, the name "unicodestring" now doesn't always mean the native string type. :-( FPC should make this mistake.

Perhaps the middle way(?)

Take what Michael said:

... and Graeme...

UnicodeString should meant UTF-8 on Linux, *BSD, MacOSX etc Unicodestring should mean UTF-16 on Windows.

...and we have 2 RTL:
1. one to AnsiString (backwards compatibility);
2. one to UnicodeString to especific plataform.

=)

Marcos Douglas

I propose that the above behavior be implemented as a type named RTLString

and

keep UnicodeString as UTF-16 for compatibility

The string parameters of the RTL functions should be declared with RTLString

So the RTL under unix will have functions compiled with UTF8 strings giving no overhead interacting with native API
The RTL under Windows will have compiled functions with UTF16 strings giving no overhead with native API

If a program is pass a UnicodeString to a RTL function under Windows no conversion is made
When this same program is compiled under unix the UnicodeString should be converted to UTF8 automatically using the encoding info of the string

Luiz

This approach has the advantage of leading to minimum conversion between user string type and native string type

Scenario - RTL with a fixed string type (UTF16/UnicodeString) both in windows and unix:

A) User code with UnicodeString
0 conversions - Under windows: no conversion between user code and rtl and between rtl and native api
1 conversion - Under unix: no conversion between user code and rtl but conversion between rtl and native api

B) User code with UTF-8
1 conversion - Under windows: conversion between user code and rtl but no conversion between rtl and native api
2 conversions - Under unix: conversion between user code and rtl and between rtl and native api

The same applies for a RTL that compiles a fixed string parameter type like UTF8 or ansi

Using RTLString the max number of conversions is 01 (user code and rtl) since always rtl and native string will have the same encoding

Luiz

I didn't know that backward compatibility to ansi was important. I previously thought that a fully Delphi-style solution would be persued (only UTF-16).

So I guess we go back to my previous proposal from the mac-pascal mailling list instead:

* Make file-handling routines which take filenames as parameters from the RTL modular so that the LCL can implement them with UTF-8 support. This plus a UTF-8 widestring manager and the Ansi RTL can be fully UTF-8.

Michael Van Canneyt wrote

One with unicode string, one with ansistring. They will have the same code, but will be compiled twice, each time with a different compiler define to decide which version it must be.

Is this possible in UNIX? I can see that in Windows you can use the trick to use W versions which are identical except for the string type and drop Windows 9x support, but is this really possible for the UNIX syscalls? They expect UTF-8 not UTF-16 which is what UnicodeString uses.

Linux expects an array of bytes in filenames (no encoding, no utf-8) AFAIK.

Martin

Luiz Americo Pereira Camara wrote

I propose that the above behavior be implemented as a type named RTLString

The Object Pascal language already has enough damn string types. I really don't think we should be adding fuel to the fire, by adding yet more string types!

So the RTL under unix will have functions compiled with UTF8 strings giving no overhead interacting with native API The RTL under Windows will have compiled functions with UTF16 strings giving no overhead with native API

That's exactly what I said.

If a program is pass a UnicodeString to a RTL function under Windows no conversion is made When this same program is compiled under unix the UnicodeString should be converted to UTF8 automatically using the encoding info of the string

No, why must unix environments take a performance hit?? This is not needed if UnicodeString is really what the same suggests. Any unicode type string. Unicode standard is defined as UTF-8, UTF-16 and UTF-32. So UnicodeString should really be any of those encodings - living up to it's name.

If FPC has true unicode support, then all functions should work correct with just the UnicodeString type. That type's encoding is based on the native encoding of each platform. NO performance hit required.

I'd even be happier if UnicodeString was dropped too, and String becomes unicode enabled. One less string type to worry about.

String could be define as follows... [ignore the syntax]

IFDEF unix

   String = String(utf8);

ENDIF
IFDEF windows
String = String(utf16)
ENDIF
IFDEF OldDelphi
String = AnsiString // of if some String(xxx) could be used ENDIF

Then if you wanted your project to use some other specific encoding, then you can simply define your own string type and use that. The various string types know what encoding they are in, so auto-conversion is possible too (with possibility of data loss in case of unicode -> ansi) eg:
type
{ say I want to use UTF-32 in my apps for some reason }

  TfpgString = String(utf32);

var
   s: String;  //  as defined above - could be utf8, utf16 etc..
   m: TfpgString;
   a: AnsiString;
begin
  m := 'Hello world!';
  s := m;  // automatic conversion happens here
  a := s;  // auto conversion, with data loss (compiler warning)
end;

Regards,
- Graeme -

Michael Van Canneyt wrote

Current strategy on fpc core seems to be to have 2 RTLs: One with unicode string, one with ansistring.

Isn't that somewhat nasty for people currently using UTF-8?

I mean, lets say that we can divide everyone using FPC into 3 groups:

1st> People using ansi that don't want to change any line of code -> They get a path forward with this proposal, even if temporary (the Ansi half of the RTL really seams like the definition of deprecated to me)
2nd> People using UTF-8 -> They get no love at all and can choose from using the old RTL with no Unicode and put some tape to fix some holes or migrate to something incompatible.
3rd> People that want to use UTF-16 -> They get a new RTL to move forward

But how many percent of FPC users, libraries and applications are on each group?

1st> I really can't imagine anyone who would want to stay stuck to the pre-Unicode world forever...
2nd> The vast majority of users, libraries and applications through Lazarus 3rd> msegui and possibly Delphi 2009+ users

Lazarus is by far the most widely way to use FPC, so I would guess that the group 2 has more then 75% of all users, and still it gets no real path forward is provided for these users?

Of course one path is migrating everything, the LCL, the IDE, SynEdit, all packages, etc, to UTF-16, but that's a huge, immense work with zero advantages over what we are doing up to now, it's just migrate to migrate, who will be motivated to do that? My point is that it is not very reasonable to migrate so much working code for no advantage at all, so the Unicode RTL could provide something to easy interfacing with UTF-8, for example:

* overloaded versions of routines and methods for utf8string
* A TStrings and TStringList for utf8

These would need to be ifdefed so they are not present in the Ansi RTL. Without even a TStrings for utf-8 one cannot really expect Lazarus to be able to use the Unicode URL without doing a full migration to UTF-16 ...

My final point is just: why not? If code in the RTL could fix things for Lazarus why impose the need to migrate so much working code?

If the Unicode RTL provides UTF-8 support too then Lazarus projects could be migrated by just doing 2 things:

1> Change all places which use TStrings and TStringList to TStringsUTF8 and TStringListUTF8
2> Change all places which add utf-8 to ansi conversions to the RTL with no conversion at all

On the other hand if we have no path forward except for migrating to UTF-16 I can imagine we will still be talking about how to move forward in 5 years from now...

Martin Schreiber wrote

Linux expects an array of bytes in filenames (no encoding, no utf-8) AFAIK.

That's a nice theory, but:

All Linux distributions that I know use utf-8
Android uses utf-8
Meego uses utf-8

So, do you have any concrete example of new releases of Linux using something different from UTF-8 for filenames?

And why would this not be possible ?

Michael.

No, why do you think so ?

They should use the unicode version. All will work as-is.

Michael.

Some Samba shares for example and there still are many "old" Linux systems in the wild. Anyway, I simply wanted to remember the fact.

Martin

No, why do you think so ?

Well, at the very least:

1> All var parameters from the RTL will no longer be directly usable with UTF-8 strings

http://www.freepascal.org/docs-html/rtl/sysutils/appendstr.html

How can I pass a UTF-8 string to AppendStr in the Unicode RTL?

The Example62 from the docs will no longer compile =D

2> TStrings will be in a different encoding from the rest of the LCL, this will surely be very nasty.
MyForm.Caption := MyStrings.Strings[9]; and you get an encoding conversion...

Basically it will be a salad of automatic conversions done by the compiler...

3> FileOpen(MyForm.Caption, whatever_mode);
You get first utf-8 -> utf-16 to call FileOpen and then FileOpen does utf-16->utf-8 on UNIXes

On Wed, Sep 14, 2011 at 5:50 AM, Martin Schreiber wrote: > Linux expects an array of bytes in filenames (no encoding, no utf-8) AFAIK. That's a nice theory, but:

It's more than theory.
You can use file names under Linux that are no valid UTF-8. At work I see it every week.

Mattias

Michael Van Canneyt wrote

Current strategy on fpc core seems to be to have 2 RTLs: One with unicode string, one with ansistring.

Can you clarify a bit. When you say "unicode string" to you mean UTF-16 (Delphi's definition of a unicode string), or do you mean a Unicode string in the true sense - it can be utf-8 or utf-16 etc depending on the platform's native encoding.

Regards,
- Graeme -

Another good example: FAT. I'm now as far to avoid umlauts and such when I copy files from Linux to FAT or the other way round, because with the default mount settings they are invalid characters in one of the two... (and I didn't yet bother to fiddle around with that ^^)

Regards,
Sven

That depends on what the compiler will do for you :-)

2> TStrings will be in a different encoding from the rest of the LCL, this will surely be very nasty. MyForm.Caption := MyStrings.Strings[9]; and you get an encoding conversion...

That will always be the case, even if we decided for UTF-16.

Basically it will be a salad of automatic conversions done by the compiler...

This will be so in each case where different codepages or encodings are used.

3> FileOpen(MyForm.Caption, whatever_mode); You get first utf-8 -> utf-16 to call FileOpen and then FileOpen does utf-16->utf-8 on UNIXes

Once more, why do you think so ?

In each case:

1. It will be messy whatever we do.
Thinking there is an easy migration path is wishful thinking.

2. Backwards compatibility is a big concern.
Code that compiled and worked should compile and work.

Michael.

This has not yet been decided.

Michael.

Mattias Gaertner wrote

It's more than theory. You can use file names under Linux that are no valid UTF-8. At work I see it every week.

In this case then for sure we cannot only have file routines only in UTF-16, because that would make it impossible to identify many files in Linux...

IMHO a platform-dependent string would be the worse solution of all ... far worse then migrating to UTF-16.

It adds tiny bit of speed while it puts a large development complexity burdain ... I imagine how one would explain that kind of thing to newbies ...

Just recently I had a student from my university implement a routine which converts HTML text from utf-8 to braille in utf-8 ... I didn't have to explain anything and she could implement it without Pascal previous experience. I wonder if I had to say: ops, the main string type is unknown =D To do any operation on it you need to first convert to something known and then convert back to unknown, while unknown conversions might take place ...

In our previous episode, Felipe Monteiro de Carvalho said:

Following from a discussion on mac-pascal, I'd like to propose a solution for Unicode support.

First and for all. Backwards compat dropping is not going to happen. If we were planning that, we had changed everything to something unicode years ago.

This is not a solution. This is a temporary hack to alieve some perceived Lazarus pain, and doesn't fix my main gripe of the manual conversions everywhere. It is is a hack for 0.01% of the unicode problem.

IMHO the objective should be to mimize manual conversions (and with that the fact that generic code becomes encoding specific).

AFAIK RTLString already exist in the cpstr branch. Anyway is just a alias to a real type.

[]

This is not desirable simply because at each platform (windows / unix) the user code of the same program will have a different encoding increasing the possibility of subtle errors. Some functions like string streaming requires the same encoding between platforms otherwise it will require code change to work properly.

Another advantage of using RTLString as i proposed is that Lazarus will require almost no code change since the encoding of string in LCL will be the same (UTF8) across platforms. The conversion will take place only when interacting with the RTL

Luiz

Felipe Monteiro de Carvalho hat am 14.
September 2011 um 10:51 geschrieben:

Well, "many" is a bit exaggerated. It does not happen on most Linux systems. And, yes, UTF-16 is not enough under Linux. 
But this is nothing new. This was explained several times on this list. See the many threads about unicode strings. 
 
Mattias
 

On 14/9/2011 03:48, Felipe Monteiro de Carvalho wrote:

[..]

Using the approach i described (RTLString) in other mail this (massive LCL code change) is not required. Probably just "load from file" functions like TStrings etc.

Lazarus/LCL could stay as is (UTF8) and would work as today:

Under unix: no conversion is done since the LCL and RTL encodings are the same
Under Windows: conversion UTF8 -> RTLString (UTF16) is done once

Because if someone for some reason, like porting Delphi code, stays with a UTF16 string, under windows, when using RTL functions TWO conversions will be made:

User Code (UTF16) > RTL (UTF8) > WINAPI (UTF16)

Always using the same encoding in RTL and Native API will keep the maximum conversion number at 01

Luiz

Felipe Monteiro de Carvalho wrote

IMHO a platform-dependent string would be the worse solution of all ... far worse then migrating to UTF-16.

I don't see why? Use the RTL functions to manipulate your text strings. Both the string and RTL functions will use the same encoding on each platform - so no problems, no conversions.

If you really needed to know the encoding, the RTL could include a helper function to tell you the encoding of any string (just like Delphi 2009+ has).

Just recently I had a student from my university implement a routine which converts HTML text from utf-8 to braille in utf-8 ... I didn't

Again, no problem. The HTML should have specified the encoding it is in. Normally that would be UTF-8. So under Linux, MacOSX etc it will already be in the native encoding. Under Windows, text is normally stored in UTF-8, contrary to UTF-16 being the encoding off the native Windows API. So loading the file you can compare the HTML file encoding to the current RTL encoding and do a conversion if needed (same as is required in Delphi).

As for the text-to-braille functionality, that is outside the scope of the FPC and RTL. But common sense should prevail, use RTL string functions to implement your conversion - don't assume 1 byte = 1 character. A unicode aware string iterator could be implemented to help you step through the characters one at a time. Such a string iterator could even become part of the RTL as it will probably be used often for many parsers.

Regards,
- Graeme -

Why would the internal enconding of a RTLString/UnicodeString have to affect any effect how you program if the RTL/API is done right?

-Flávio

This would not happen because I proposed to have 2 versions of the routines in the RTL. Not 1 UTF-8 version. There would be both UTF-8 and UTF-16 versions and one would naturally use the one which matches his preferred encoding ... and the RTL would only convert the non-native version.

Luiz Americo Pereira Camara wrote

This is not desirable simply because at each platform (windows / unix) the user code of the same program will have a different encoding increasing the possibility of subtle errors.

Why? Not every program is a text manipulation program or text parser. Most programs simply assign one string to another.

eg:

   Button1.Caption := 'Click me';
   lMyString := Button1.Caption;

Under unix systems 'Click me', Button1.Caption and lMyString will be a UTF-8 encoded. Under Windows 'Click me', Button1.Caption and lMyString will be UTF-16 encoding.

When Lazarus saves this information in a .lfm file, it will be stored as UTF-8 irrespective of the platform. This is normal behaviour on all platforms already, and already done in Lazarus too.

As for streaming, the same applies as for saving to file. UTF-8 is ideally suited for (and was designed for simplifying) streaming, hence the W3C promotes the usage of UTF-8 in HTML, XML etc.

Another advantage of using RTLString as i proposed is that Lazarus will require almost no code change since the encoding of string in LCL will be the same (UTF8) across platforms.

Lazarus, like fpGUI will have to decide what they want to do. Stick to having UTF-8 forced on all platforms, or use a native encoding on each platform. Currently UTF-8 was choosen in both project because it is so compatible (think easy here) with AnsiString - so least amount of work was required and it was pretty efficient because most programs already used AnsiString.

If I was to change fpGUI to use a native encoding on each platform, I would simply change my definition of TfpgString as described in a similar example before. All string manupulation inside fpGUI (and LCL) should already have adhered to the rule that 1 byte 1 character, so the rest of the framework should continue to work as normal. In the case of fpGUI, I would also be able to get rid of all the UTF8Copy(), UTF8Length() calls and simply use the RTL Copy() and Length() functions again - after all, they were only introduced because FPC's RTL lacked Unicode (any encoding) support.

Regards,
- Graeme -

In our previous episode, Felipe Monteiro de Carvalho said:

I'm not as opposed to this as to the other. At least the interfaces stay the same.

But again, that is no unicode solution, just minor damage control to make the current situation bearable.

Yes, but this solution would only work in the Ansi RTL ...

And why would the interfaces change in the other proposal? It is only 1 more overloaded option for the routines, it does not change anything which will use UnicodeString.

In our previous episode, Felipe Monteiro de Carvalho said:

Afaik QT and many other higher level libs always use UTF-16. MSE does too. Might also be useful for the JVM port.

The main reason is that users can pick the RTL that is closest to the codebases that they have, and depending on the platform that they focus on.

UTF16 e.g. when leaning toward Delphi 2009 model, or for MSE, and UTF-8 for the rest.

It also provides transitional ability. Everybody can make changes (and test with the "other") in his own pace, and nobody gets forced into a situation that is disadvanteous for them.

In theory you could even make one byte ansi and one byte utf8 versions on windows (so three RTLs) though I think longterm only ansi and utf16 are viable on windows, since those are the native encodings.

Note that the encoding type of the RTL is mainly the main stringtype used in FPC and the default "char/pchar/STRING" in delphi/unicode mode. (a modeswitch over objfpc/delphi?)

It does not mean that you can't keep parts of the source code UTF8 or ansi. It just might be less optimal, and will have to be properly typed. This will also happen with packages where upgrading to unicode doesn't make sense.

Note that these are just the big lines of a possible solution. There is only one big disadvantage : The parts of the RTL that are encoding dependent are double (relatively minor since many conversions will go automatic when everything is properly typed), and the release angle.

But it will be beneficial to everybody, and it is clear to everybody how something should behave, so there will be no endless bickering over details and workarounds like this thread. It is a structured approach.

BTW: I explained all this to you, including the not dropping legacy, over some Chinese food a few months ago. Don't you remember?

In our previous episode, Felipe Monteiro de Carvalho said:

And why would the interfaces change in the other proposal? It is only 1 more overloaded option for the routines,

Which is just 1 more interface change. And for something that is a temporary workaround.

That is what I like on Mattias proposal, it is mostly hidden in implementation, and the declaration and setting of the "manager" can have a lot of platform and deprecated directives
around it to make it clear that it won't last, and it is not just for lazarus. I assume it is windows only.

But I really do wonder if this is necessary, since it will already not make the 2.6 cycle anymore, and I hope 2.8 can be really unicode.

In our previous episode, Martin Schreiber said:

It is a bit agnostic yes. But does that really matter if all other programs write filenames in utf8 encoding? It might as well be specified to be utf-8 then, there is no difference in approach.

I think I wasn't clear enough. I wanted to say that I don't see how you can have both a Ansi and a UTF-16 RTL in UNIXes with the same codebase, without ifdefs. I think this is not possible and one of the previous messages seamed to indicate that the RTL would be able to use the same codebase regardless of the output version (ansi vs utf-16), so without ifdefs.

But it will be beneficial to everybody, and it is clear to everybody how something should behave, so there will be no endless bickering over details and workarounds like this thread. It is a structured approach.

Well, there is still uncertainty over the question brought up by Graeme: Always UTF-16 or the unknown string type?

BTW: I explained all this to you, including the not dropping legacy, over some Chinese food a few months ago. Don't you remember?

Yes, I remember, but the way you spoke about it, it sounded something like a proposal, not 100% sure it would end up like this =) Now it is really put as the way forward...

AFAIK, the EXT system does not care about the code the file-name byte-arrays are done in. only 0x00 (end of name) and '\' are interpreted.

-Michael

Michael Schnell wrote

AFAIK, the EXT system does not care about the code the file-name byte-arrays are done in. only 0x00 (end of name) and '\' are interpreted.

Sorry, (typing too fast again)

Only 0x00 (end of name) and '/' are interpreted.
AFAIK, even '\" is allowed in a file name. just the command line representation uses backslash as an escape symbol.

-Michael

Felipe Monteiro de Carvalho wrote

In this case then for sure we cannot only have file routines only in UTF-16, because that would make it impossible to identify many files in Linux...

Who says that file names are supposed to be human readable and this done in some character encoding ?

AFAIK:

With EXT they are just streams of up to 512 bytes (with 0x00 and '/' disallowed)

With old style FAT they are just arrays of 11 bytes (maybe with 0x00, '.' and '\' disallowed) and ASCII characters of lower case and upper case identical

With "long filename FAT" I fear it's quite complicated (e.g. short and long file name of a file need to be recognized as identical). But no unicode here.

-Michael

Marco van de Voort wrote

First and for all. Backwards compat dropping is not going to happen.

It already has and supposedly can't be avoided. Take a look of what Lazarus was forced to make out of the identity of ANSIString and UTF8String seemingly forced by FPC. e.g.:

Old programs assuming local ANSI 8 bit code retrieved from LCL GUI components, compiled with the new version don't work (e.g. if doing myChar := myString[3]; )

Doing My16BitString = 'my constant text containing umlauts as äöü';
provides an erroneous result.

-Michael

A few topics:

The NT WinAPI (not 9x) *implements* everything in the Wide (UTF-16) routines, the Ansi versions do the string *conversion* before calling the Wide version. Unix API (most probably - dunno) has no such dual interface with internal conversion.

The NT filesystems store names in UTF-16, while Unix filesystems store UTF-8. This means that access to an NTFS or FAT32 drive under Unix will require a string conversion, in the filesystem handler.

On Windows, Ansi means any (byte-char) encoding, with different (national) codepages on every machine. This can cause trouble to Ansi applications (using Ansi strings), when filenames do not convert losslessly into that codepage. Unix IMO uses UTF-8 as the Ansi encoding, eliminating possible losses, and that's why FPC also prefers UTF-8 encoding.

But let's not forget the user!

Many users still want simple string handling, with direct mapping between logical and physical chars (SBCS). This is not possible at all with UTF-8, while UTF-16 works fine with the BMP, at least. This "want of simple string handling" suggests the use of UTF-16 for Unicode strings in *user* code.

WRT the latter argument, FPC IMO should follow the Delphi implementation of Unicode strings as UTF-16. This choice is independent from the (platform dependent) RTL conventions, but it affects the standard components (string lists...) in the FCL, and the other components in the LCL. Here again the average user will prefer UTF-16 component libraries, compatible with his own code, while more experienced users may be happier with the current UTF-8 libraries.

English (ASCII) users also may prefer UTF-8, as long as they do not have to (or want to) deal with strings in foreign languages. Once they have to face the existence of non-ASCII strings in their applications, they will most probably prefer switching to UTF-16, with few changes to their existing codebase and coding habits(!). Really *processing* Unicode text, with all its bells and whistles, is so complicated that it should be left to dedicated software and libraries, while typical application code will ignore everything beyond char level.

IMO the number of required conversions is of little importance to the runtime behaviour of an application. File access is always expensive, so that a single conversion into the platform specific filename representation is not same for GUI components,
which typically store all strings twice: once for their own (and application) use, and another copy in the widgets. Here again transfers of strings between widgets and components are rare, with neglectable slowdown by eventual conversions during message handling.

More important IMO is the external storage of Unicode, where I see no reasonable way around UTF-8, considering codepage dependencies and UTF-16 byte-order problems.

Another note: a "set of char" is quite incompatible with Unicode/UTF-16. This should be taken into account with *every* introduction of an Unicode string type.

DoDi

If FPC has true unicode support, then all functions should work correct with just the UnicodeString type. That type's encoding is based on the native encoding of each platform. NO performance hit required.

Can you specify, *which* strings ever *require* platform specific encoding? Beyond filenames and environment strings?

UI strings (GUI, console) are more thightly bound to user code and component/widgetset libraries, than to a platform API (see my other comment)

DoDi

How many users will have to deal with chars outside the Unicode BMP? IMO UTF-16 can make 99% of the (current) users happy with simple string handling, while the rest would prefer UTF-32 instead of UTF-8, because outside the BMP UTF-8 is a waste of space, and lacks indexed char access in any case.

DoDi

How many bytes must a char have, when it shall allow to store any (logical) character? Unicode users have no use for an char type, instead they have to use substrings for every logical character. A Unicode BMP user could be happy with a 2-byte char, of course, at his own (low) risk.

DoDi

OK. The drawback is increasing file size of executables (that are already big).

Luiz

Given that Button1.Caption will be different under windows and unix, even if the compiler provides automatically conversion, at least some changes will be required to the default classes that handles things like (de)serialization etc and the places where these methods should be used must be checked.

Moreover having different encodings in different platforms will give no gain to libraries like LCL/Lazarus like stated by DoDi.

All in all my proposition is similar to yours. The only difference is that by default i suggest to be used only in RTL but nothing stops to users like you using it in a broader scope.

The other difference is the name that i dont care (can be xString, MultiString, FPCString). Just i think that using UnicodeString to a variable encoding per platform will loose Delphi compatibility for no good and more: will be floods of bugreports asking why Delphi code does not work this way and asking for a "fix".

Luiz

[...]

Agreed. And so it is made in MSEgui:

- On user side all stringhandling uses type "msestring" which is defined as the existing Free Pascal 16bit UnicodeString.
- The MSEgui widgetset works with UnicodeString too.
- For file and directory access MSEgui has a set of functions which convert from/to system encoding to/from type "filenamety" which is defined as existing Free Pascal 16bit UnicodeString.
- MSEgui has own 16bit functions and classes for lists, maps, sorting and the like.
- Text files are stored in utf-8 by default.

From my point of view there is no need for a complicated encoding aware unicode string type which possibly is slower, needs more memory and introduces new bugs.

Martin

Hans-Peter Diettrich wrote

The NT WinAPI (not 9x) *implements* everything in the Wide (UTF-16) routines

without reference-counting: Yak !

-Michael

Hans-Peter Diettrich wrote

The NT filesystems store names in UTF-16, while Unix filesystems store UTF-8.

there are lots of file systems in Linux. They can work differently. i.e. FAT works case insensitive while ext* works cases sensitive. IMHO ext completely ignores character coding and just works on byte arrays.

-Michael

This is why new Delphi versions provide "New Strings" that know about thee coding their content is done in and do automatic conversion when necessary.

I was told this does help in many situations but produces confusion and incompatibilities in others.

The FPC "cpstrnew" implementation seems to be stalled right now.

-Michael

Hans-Peter Diettrich wrote

Can you specify, *which* strings ever *require* platform specific encoding?

If not strings, Chars do:

MyString := 'Öse';
MyChar := MyString[1];

-Michael

Hans-Peter Diettrich wrote

Unicode users have no use for an char type, instead they have to use substrings for every logical character.

Yep.

The problem is that a "normal" programmer (especially a beginner) does not know (and does not want to know, and IMHO should not be forced to know), that (s)he is a "Unicode User".

A good programming system should hide this "deeply technical stuff" as far as possible. Otherwise we can go back to ASM programming.

-Michael

Hans-Peter Diettrich wrote

Many users still want simple string handling, with direct mapping between logical and physical chars (SBCS). This is not possible at all with UTF-8, while UTF-16 works fine with the BMP, at least.

What rubbish! The only "utf-8 limit" is that the current FPC and Delphi RTL's don't cater for it due to the legacy ANSI support that came before.

But then you also forgot that most string manipulation that requires individual "characters" normally process such strings from left-to-right, so again there is NO problem doing that with UTF-8 encoded strings. I've done enough UTF-8 string manipulation to know.

(platform dependent) RTL conventions, but it affects the standard components (string lists...) in the FCL, and the other components in the LCL.

Please give a concrete example where using platfrom dependent encodings (eg: UnicodeString = UTF-8 on Linux, but UTF-16 on Windows) will cause problems? I really cannot see any issues here, only positives like better performance for each platform due to no need for auto-conversions.

Here again the average user will prefer UTF-16 component libraries, compatible with his own code, while more experienced users may be happier with the current UTF-8 libraries.

What the hell has "experience" got to do with the preference between UTF-8 and UTF-16? To the developer (and more so to the end-user) a Unicode string should act like any other Unicode string. What encoding is used to represent "hello world" shouldn't even come into question.

English (ASCII) users also may prefer UTF-8, as long as they do not have to (or want to) deal with strings in foreign languages.

Rubbish once again! Our applications use UTF-8, I have no problems writing application that support multiple foreign language - as long as those languages are left-to-right (I don't understand RTL languages, so can't comment). Neither have my users any problems using such application. Hell, they don't even know what Unicode is, or what the word Encoding means - they just see it as text on the screen. And that is how it should be.

Regards,
- Graeme -

Martin Schreiber wrote

Agreed. And so it is made in MSEgui:

Yeah, and everything you said applies to fpGUI, except I use TfpgString, TfpgChar and the UTF-8 encoding. Though I would prefer having the native encoding on each platform - thus less conversions and improved performance.

From my point of view there is no need for a complicated encoding aware unicode string type which possibly is slower, needs more memory and introduces new bugs.

There shouldn't be anything complicated for the developer or enduser. UnicodeString under Linux and Mac should be UTF-8, under Windows it should be UTF-16. Everything else is exactly the same as you already do in your code.

As for your comment on memory usage. UTF-8 on average uses less memory than UTF-16. There was a few studies done - Google should find them for you.

Regards,
- Graeme -

Michael Schnell wrote

there are lots of file systems in Linux. They can work differently. i.e. FAT works case insensitive while ext* works cases sensitive. IMHO ext completely ignores character coding and just works on byte arrays.

You also forgot to mention that most file systems under Linux (and Mac) can be tuned. Case sensitive or not, what encoding they should use etc... So making a generic statement like the original post about *nix files systems is just wrong.

Regards,
- Graeme -

Luiz Americo Pereira Camara wrote

OK. The drawback is increasing file size of executables (that are already big).

And disk storages are getting each time bigger. Any modern smartphone comes with at least 8GB of storage ...

In our previous episode, Hans-Peter Diettrich said:

According to unicode n*codepoints. A codepoint is now 20 or 21 bits, but can be expanded in the more distant future

IIRC n is in the range of 5-8 or so, the maximum amount of codepoints that can be combined to a printable character.

So if you want to do it up to spec, a character is +/- 256bit.

Unicode users have no use for an char type, instead they have to use substrings for every logical character. A Unicode BMP user could be happy with a 2-byte char, of course, at his own (low) risk.

Probably. But while a good point for a application builder based in the West, it is IMHO not acceptable to cut corners in the unicode implementation in system and development tools.

Graeme Geldenhuys wrote

What the hell has "experience" got to do with the preference between UTF-8 and UTF-16?

In fact users want to deal with "decently coded characters" and not with "cryptic bytes some of which together are representing a character". (e.g. when doing MyChar := MyString[1]; )

Here UTF-8 code in a type called "ANSISting" is a no-go.

UTF-16 code in WideChar, and WideString come quite close to this in the Western world, but I think this is not good enough.

Full 4 byte Unicode comes even closer, but there still are some hard to handle double-DWord codes plus the memory usage is huge.

So Delphi NewStrings (cpstrnew) and a 32 Bit Char type seems like the most user-friendly way to go. (But of course eating a lot of performance.)

-Michael

Hans-Peter Diettrich wrote

How many users will have to deal with chars outside the Unicode BMP?

You are very narrow minded! It depends on the application you are developing. Lets take a Science application as an example. Many scientific symbols fall outside the BMP. Now lets take a another example. Egyptian Hieroglyphs. Or a Music program. Or your next version of Skype or some IM app using all those emoticons.

Looking at the following chart [see url below], Emoticons, Transport and Map symbols, Alchemical Symbols, Pictographs, Playing Cards, Mathematical symbols etc all fall outside the BMP. There are all symbols that could quite easily be used in a variety of applications - so yes, accessing symbols in Plane 1- 14 is rather important!

http://en.wikipedia.org/wiki/Basic_Multilingual_Plane#Supplementary_Multilingual_Plane

The applications we develop at our work use symbols outside the BMP - Maths, Science, Alchemical etc..

outside the BMP UTF-8 is a waste of space, and lacks indexed char access in any case.

Yeah, and indexed access for UTF-16 encoded strings needs to check for surrogate pairs too! Otherwise your app is not Unicode enabled but rather UCS-2 only.

And considering the amount of text processing apps I have written (plenty of them), indexed character access is really not a top priority or a often used feature.

Regards,
- Graeme -

Michael Schnell wrote

In fact users want to deal with "decently coded characters" and not with "cryptic bytes some of which together are representing a character". (e.g. when doing MyChar := MyString[1]; )

None of our company's users using our products would even know what the hell you are talking about. :-)

But to convert your example to UTF-8 as would be done in a fpGUI application (which I again is never really require in real world applications), it would look as follows:

var
  MyChar: TfpgChar;
  MyString: TfpgString;
begin
  MyString := 'some unicode string even with chars outside BMP';
  MyChar := UTF8Copy(MyString, 1, 1);

The above example is safe, and will always give the correct result, even for Unicode characters above the BMP. And I don't need to worry about endianness (byte order) - unlike UTF-16 which could be in UTF-16BE or UTF-16LE and a Unicode "character" could be a surrogate pair - so your example would fail there too.

Regards,
- Graeme -

2011/9/15 Michael Schnell :

In fact users want to deal with "decently coded characters" and not with "cryptic bytes some of which together are representing a character". (e.g. when doing MyChar := MyString[1]; )

I think of Unicode text as a stream of Unicode characters in some format. I trust that functions that deal with the string Copy, Length, Find, etc., know how to handle it.

When doing:
MyChar := MyString[1]

appropriate function retrieves first unicode character, regardless of encoding.

Whether it's utf8, utf16, utf32 or any other future encoding the code should work the same.

Graeme, please have look into:
http://www.lazarusforum.de/
and
http://www.freepascal.ru/

There are plenty of user problems with utf-8 character access and string length. I assume 100% of them would be solved with utf-16. BTW, do you remember how long it took until the current FPC WideString and UnicodeString were reliable? And now there should be an even more complex string type implemented?
shudder....

Martin

Martin Schreiber wrote

There are plenty of user problems with utf-8 character access and string length. I assume 100% of them would be solved with utf-16.

It would solve for those writing buggy software which ignores part of the Unicode characters.

On the other hand 100% of them would be solved correctly, for all Unicode characters by using LCLProc.UTF8CharacterLength

http://lazarus-ccr.sourceforge.net/docs/lcl/lclproc/utf8characterlength.html

With this routine you can safely iterate through all characters of a utf-8 string.

Michael Schnell wrote

If not strings, Chars do: MyString := 'Öse'; MyChar := MyString[1];

and to show you AGAIN how flawed your "direct index access to a character" example is. How is that 'Öse' entered into the system.

Is the Ö a

U+00D6 LATIN CAPITAL LETTER O WITH DIAERESIS

or

U+004F LATIN CAPITAL LETTER O with
U+0308 COMBINING DIAERESIS

or

what if the Ö was a "character" above the BMP. MyChar would only hold the first part (2 bytes) of a surrogate pair (4 bytes).

Oops! You are thinking in terms of ANSI, *not* Unicode.

Regards,
- Graeme -

+1 on both counts.

:-D

Regards,
- Graeme -

Martin Schreiber wrote

There are plenty of user problems with utf-8

Then they are not well versed in Unicode are they...

character access

in fpGUI:
UTF8Copy(...)
UTF8CharAtByte(...)

and string length.

in fpGUI:
Length(...) result is in bytes
UTF8Length(...) result is in "characters"

I assume 100% of them would be solved with utf-16.

Unless they treat UTF-16 text as UCS-2 - forgetting about endianness and surrogate pairs.

And now there should be an even more complex string type implemented?

UTF-8 is not more good news is that if you write a UTF-8 string handling routine, it works with ALL UTF-8 encoded strings, from 1 byte to 4 byte code points. No special treatment or considerations are required. Streaming is simple and easy as it is already an array of bytes. You can't get easier than that!

Regards,
- Graeme -

Of course. But generations of Pascal programmers have been trained to do

MyChar := MyString[1];

So it would at least be candid to abolish the String[i] notation as a syntax feature, as it does not provide the result "normal" (Unicode unaware) users expect. (And provide an appropriate aromatically correctly coding aware library call - hopefully supported by appropriate syntax - candy instead). This of course would need either a dynamically coded "NewDelphiString" Type or different types for different coding variants.

-Michael

cobines wrote

MyChar := MyString[1] appropriate function retrieves first unicode character, regardless of encoding.

MyChar is an 8 bit thingy and thus is not even able to hold a Unicode 'ä' (in what ever UTF).

-Michael

Graeme Geldenhuys wrote

+1 on both counts.

Hoping for complex things to automatically be solved by just ignoring the complexity usually leads into hell.

-Michael

Graeme Geldenhuys wrote

and to show you AGAIN how flawed your "direct index access to a character" example is.

It's not "my" intend to use it. I'll never use it as I do know that it is bound to create problems. But it is what generations of pascal programmers are trained to do. They all need to be re-trained. In fact this is just "Syntax-Candy" (as here native Array-syntax is used for a non-array type). So it could be removed or modified to better support the expectation of the "generations of pascal programmers" even in times of Unicode.

How is that 'Öse' entered into the system.

I was assuming a current Lazarus version as an IDE and here UTF-8 is used. (older Lazarus versions in fact did use locale ANSI code, thus supporting the "generations of pascal programmers")

-Michael

Felipe Monteiro de Carvalho wrote

On Thu, Sep 15, 2011 at 10:59 AM, Martin Schreiber

I don't think so. If there are characters out of the BMP the "buggy" application will take a single out of BMP character as two unknown characters. No problem for Russian and German hobby programs IMHO. :)

Martin

2011/9/15 Michael Schnell :

Of course. But generations of Pascal programmers have been trained to do MyChar := MyString[1];

Such people should retrain if they want to switch to Unicode using some instructions how to convert your application. If they do not want, they should stay with Ansi. There is no automatic switch your application to Unicode.

2011/9/15 Michael Schnell :

If you assume MyChar is 8-bit, then you are using Ansi. MyChar should be the same type as MyString. At best it could be UTF-32 encoded 32-bit value representing character but then you may lose info about how the character is composed, unless you normalize. Plus in future 32-bit may be not enough.

Graeme Geldenhuys wrote

> And now there should be an even more complex string type implemented? UTF-8 is not more new encoding aware FPC string type is more complex.

Martin

Hans-Peter Diettrich wrote

Unicode users have no use for an char type, instead they have to use substrings for every logical character.

"Unicode" is 32 Bis and allowing for (nearly) any supported character. So a 32 Bit UnicodeCharacter in fact is very viable.

-Michael

Michael Schnell wrote

is bound to create problems. But it is what generations of pascal programmers are trained to do. They all need to be re-trained.

Just like all Delphi developers had to since Delphi 2009+. Obviously, the RTL could cater for it, or implement more obviously named functions to hint to the developer what is going to be returned.

> How is that 'Öse' entered into the system. I was assuming a current Lazarus version as an IDE and here UTF-8 is used.

Still makes no difference. You could have entered it as one combined code point, or as two "characters" displaying as one. :-)

It is a pity Lazarus IDE doesn't support Ctrl+Shift+U to enter Unicode characters - like normal GTK2 edit widgets do. It is very convenient. But they do have the Character Map I guess.

Regards,
- Graeme -

Which imho makes utf8 far more preferable than utf16

in UTF8 the error is bound to happen far easier, which gives people a far better chance to catch it before release, even before creating to much code relying on the buggy implementation.
Certain errors will be made, the importance is not, to create an environment in which they can not be made. The importance is to create an environment in which they will be caught as early as possible.

And besides that "generations of pascal programmers". Well the older ones are 9or at least should be) experienced => they should have no problem learning it. programming requires a developer to keep up to date. (I for example have long given up to directly access the video memory of a VGA adapter, despite my generation was trained to do so)

As for "newbies" => they come with all kind of wrong experiences. A lot of them come from VBA, and yet we aren't discussing introduction of VBS like object instantiation? ( variable.Create instead of variable:=class.create).
so again, utf8 is good, they make the error, they see the error, they learn.

cobines wrote

Such people should retrain if they want to switch to Unicode using some instructions how to convert your application.

In fact they don't know about Unicode and thus don't even know that they need training :-) .

If they do not want, they should stay with Ansi. There is no automatic switch your application to Unicode.

If thy use Lazarus they are forced to use Unicode unless they wand to stick with a very old version.

-Michael

Martin wrote

Which imho makes utf8 far more preferable than utf16 in UTF8 the error is bound to happen

ROFL.

-Michael

2011/9/15 Michael Schnell :

If thy use Lazarus they are forced to use Unicode unless they wand to stick with a very old version.

Then this is a problem of Lazarus. They want to make applications with Lazarus (which is always UTF-8?) and they are unaware of using Unicode where they wanted to use Ansi. I don't know if in Delphi you can use just Ansi, but if you cannot use Ansi in Lazarus then people should be aware of it before they start using it and learn. As far as I know there is a page on Lazarus Wiki explaining it.

Regardless if it will be utf8 or utf16 I don't think any specific format is easier to learn than the other. You know you cannot use it like Ansi anymore anyway and direct access to characters is gone.

cobines wrote

I don't know if in Delphi you can use just Ansi,

Delphi is a completely different matter as (by default) old versions use ANSI and new versions use Unicode in NewDelphiStrins with dynamic encoding.

-Michael

2011/9/15 Michael Schnell :

Delphi is a completely different matter as (by default) old versions use ANSI and new versions use Unicode in NewDelphiStrins with dynamic encoding.

I see. Then people switching from Delphi/Ansi to Lazarus/UTF-8 will have problems if they are not aware they have to adjust their applications. I just think the decision of how to implement Unicode in FPC should not be based on it.

cobines wrote

I see. Then people switching from Delphi/Ansi to Lazarus/UTF-8 will have problems if they are not aware they have to adjust their applications.

Switching from Lazarus-pre-Unicode, Delphi pre-Unicode and Delphi-Unicode will cause this problems (for not completely English applications).

I don't have a Delphi-Unicode version, so I don't know what problems will arrive when switching from Delphi pre-Unicode to Delphi-Unicode, when the application is automatically converted to use NewDelphiStgrings (and supposedly the 32 Bit Unicode Char type). But I was told there are some.

-Michael

Well, here we see why I started this thread. fpGUI and the LCL are implementing things which should be on the RTL ...

But the RTL is not UTF-8 friendly and the result is things like these, or the worse variants:

// file operations
function FileExistsUTF8(const Filename: string): boolean;
function FileAgeUTF8(const FileName: string): Longint;
function DirectoryExistsUTF8(const Directory: string): Boolean;
function ExpandFileNameUTF8(const FileName: string): string;
function ExpandUNCFileNameUTF8(const FileName: string): string;
function ExtractShortPathNameUTF8(Const FileName : String) : String;
function FindFirstUTF8(const Path: string; Attr: Longint; out Rslt:
TSearchRec): Longint;
function FindNextUTF8(var Rslt: TSearchRec): Longint;
procedure FindCloseUTF8(var F: TSearchrec);
function FileSetDateUTF8(const FileName: String; Age: Longint): Longint;
function FileGetAttrUTF8(const FileName: String): Longint;
function FileSetAttrUTF8(const Filename: String; Attr: longint): Longint;
function DeleteFileUTF8(const FileName: String): Boolean;
function RenameFileUTF8(const OldName, NewName: String): Boolean;
function FileSearchUTF8(const Name, DirList : String;
ImplicitCurrentDir : Boolean = True): String;
function FileIsReadOnlyUTF8(const FileName: String): Boolean;
function GetCurrentDirUTF8: String;
function SetCurrentDirUTF8(const NewDir: String): Boolean;
function CreateDirUTF8(const NewDir: String): Boolean;
function RemoveDirUTF8(const Dir: String): Boolean;
function ForceDirectoriesUTF8(const Dir: string): Boolean;

Lazarus is literally being forced to implement it's own RTL...

With the currently planned Unicode RTL it will just get worse, we will then need to either migrate to UnicodeString (a lot of work, no benefit at all) or give up using the RTL at all and simply finish rolling our own set of functions. We will even need to roll our own TStringsUTF8, TFileStreamUTF8, etc...

Same for MSEGui and fpGUI.
I guess all three have the opinion that a x-platform library is easier to use/maintain with only one string type across all platforms. On the other hand is the FPC team, that has to solve an impossible puzzle: Compatibility with old code requires AnsiStrings in system encoding, compatibility to new Delphi requires UTF-16 under Windows, Linux requires UTF-8 byte encoding and automatic conversion will slow down all uses of TStrings.
Call me a pessimist, but my guess is, whatever the FPC team does, the three libraries will still provide their own functions.

It seems to me, that all of the above functions and classes could be solved for applications using UTF-8 ansistrings with a special widestring manager and filefunction manager, right?

Mattias

There shouldn't be anything complicated for the developer or enduser. UnicodeString under Linux and Mac should be UTF-8, under Windows it should be UTF-16.

You're a bit inconsequent ;-)

UTF-8 is much more complicated to handle by the user, than e.g. UTF-16.

DoDi

Mattias Gaertner wrote

It seems to me, that all of the above functions and classes could be solved for applications using UTF-8 ansistrings with a special widestring manager and filefunction manager, right?

Theorically, it could, but it would work only in the Ansi RTL, basically it would not work at all in the Unicode RTL and LCL users would be locked out of it.

And I say more, two RTLs will immediately cause problems in all kinds of libraries. Will the FCL work with the Ansi RTL, with the Unicode RTL, with both? Won't that duplicate the difficulty in maintaining it if it has to be tested to work in both Ansi and Unicode mode? Will all packages from FPC work perfectly in both modes? Is that really feasable? How? {$ifdef UNICODE} everywhere that uses strings?

And what if people start writing packages for the Unicode RTL, without IFDEFs to be compatible with Ansi, then LCL users cannot use them? What if you have a package for Ansi and another for Unicode, they cannot be used at the same time?

This is not a stable solution, for Lazarus I see only 2 stable solutions assuming that my proposal is completely off the table:

1> Migrate to UnicodeString
or
2> Implement everything that we need in the LCL to substitute RTL routines which include string or file handling (and we are not really that far away from finishing that at all...)

Porting all of the libraries and applications which I use for UnicodeString will likely be so huge of a task that I don't see any feasability in this kind of thing. We need to port SynEdit, the LCL, fpvectorial, regexpr... lots of applications which I wrote and nearly all of them do some kind of text parsing or handling...

Even worse would be keeping all those projects with IFDEF UNICODE everywhere ... Just imagine how many regressions linked to not testing both possibilities...

Just implementing ourselves what we need for file handling is orders of magnitude less work for me, and then Lazarus can work regardless of which RTL packages rely on, either Ansi or Unicode.

What data type would you use, to store an UTF-8 character? And how to access the n-th character in an UTF-8 string? ...

As already pointed out, string encoding conversions between application and widgets are rare, consequently performance depends more on string handling in application code. Now the new Delphi string types, with automatic conversion when required, can cause a slowdown. In FPC character-based access to strings also can cause a slowdown (iterators...).

When a multi-platform application must be aware of possible UTF-8 strings, depending on the platform, the code must be MBCS aware. This again is complicated string handling, when otherwise immediate indexed access is possible :-(

This applies only to constant string literals, where the user never has to care for string encoding and conversion.

You better should understand ;-)

RTL is a mere *display* feature, the chars still are stored from first to last. More important is the SBCS/MBCS difference, which must be reflected in user code. Even if *you* have no problems with MBCS (like UTF-8), other users have.

DoDi

Have a look at the many languages in the Unicode BMP (or easier: look for languages with characters *outside* the BMP). All these languages are comfortable with UTF-16, when a char is an UTF-16 WideChar. E.g. <http://en.wikipedia.org/wiki/Plane_%28Unicode%29>

DoDi

I see no reason to change application code a lot, when strings only are read and displayed - no need ever to deal with single characters. But even more sophisticated apps can be happy with every string encoding, when they e.g. break strings at whitespace or other separators, and string concatenation is no special string sorting and comparison (table/db lookup) is independent from the encoding.

Only when an application must *interpret* strings in foreign languages, to e.g. break words, the coder has to learn everything about every *language* to be supported, before he can start to write according code. Such code requires *more* than only knowledge about Unicode and encodings.

DoDi

There exist few situations where Ansi code really deserves updates, when ported to FPC/Lazarus. Filename conversion comes into mind, and eventually conversion in string exchange with LCL controls.

UTF-16 strings will remove all these problems, because all 8-bit Ansi codepages are part of the Unicode BMP, and consequently deserve no further conversions.

DoDi

Right.

When doing: MyChar := MyString[1] appropriate function retrieves first unicode character, regardless of encoding.

This is just wrong :-(

MyString[1] accesses the first element of the *physical* character array, regardless of any encoding. Also Length returns the array size, not the number of *logical* characters in it.

Whether it's utf8, utf16, utf32 or any other future encoding the code should work the same.

Very new functions are required for dealing with *logical* characters, in every MBCS encoding.

DoDi

Any change of RTL types does.

Yes.
Or 3 - migrate LCL to UTF8String

Same here.

The LCL uses TStrings. If TStrings is changed to another string type, then the LCL must follow or define its own class.

Mattias

So what?

(which platform function would require such coding?)

DoDi

Just the contrary is true. UTF-16 extends the range of usable charsets, with only changing the size of a char.

This is true only for formal errors, not for all the other problems with foreign languages. The assumption, that Unicode will turn every coder into an linguist, capable of dealing with all (human) languages, is simply wrong.

VBA uses UTF-16, proofing your assumptions as wrong again.

DoDi

Thank you for ripping the above out of context. Now that it stands outside it's original context => you are right, now the contrary is true. Of course, when it still was in it's context.....

This is true as far as the comparison between utf8 and utf16 goes. All other problems were ignored, because the point was ONLY to compare between the 2. Any othr alternative, or any change caused by either of them was not part of the statement.

Utf16 does have surrogate pairs as well as chars not in the bmp

That means any developper who is serious about it's work must implement the handling of those. Which makes the effort comparable to utf8.

Yes indeed you are right, the chances that something goes wrong, if a developper doesn't deal with them are considerable lower in utf16 than in utf8 (but they do exist) => that was exactly my point.

If there is a likelyhood for a certain error to be made (by accident) then it is better if th chnaces are high that you notice the error => otherwise you ship your software with the error, and spend long time figuring out the bug reports from you users.

Of course, if a programmer INTENTIONALLY puts this error in his code, hoping to get away with it, then utf16 is the better choice. I can and will however not see such bad practice as an argument for the case.

please re-read my statement. Your statement seems to confirm not contradict mine

Hans-Peter Diettrich wrote

Graeme Geldenhuys schrieb: > On 14/09/2011 19:17, Hans-Peter Diettrich wrote: >> How many users will have to deal with chars outside the Unicode BMP? >

Any app that loads a text from disk.

it can never know what the text contains.

I'm well aware of that, but you failed to answer my question: How many users?

And which of these users has already written code for your examples, using 8-bit chars?

Feel free to use symbols as you like :-)

But what's the difference in code, between using astral characters in UTF-8 and UTF-16?

See above. If you honestly *want* to use the full range of Unicode characters, you have to write appropriate code. But who will *force* other coders to do the same?

When I want a program for German or French users, I'll hire an coder with experience in those *languages*, not with experience only in Unicode.

And considering the amount of text processing apps I have written (plenty of them), indexed character access is really not a top priority or a often used feature.

Right, text processing deserves special coding, and again much more experience than only with Unicode. Now tell me the number of coders, which are *capable* of writing a text processing application, and how many of these have problems with Unicode and encodings?

DoDi

FPC also allows to use Complex values - but nobody is forced to use such numbers without any good reason and technical (mathematical) background. The same for the use of astral Unicode characters, IMO.

DoDi

Mattias Gaertner wrote

Yes. Or 3 - migrate LCL to UTF8String

Indeed. And we can aditionally provide UTF-8 versions of routines like we do now for people to use, or alternatively people can also use Unicode RTL routines if the extra conversion is not a problem for them.

The LCL uses TStrings. If TStrings is changed to another string type, then the LCL must follow or define its own class.

Well, I think the RTL should introduce a TStringsUTF8 at the very least. I also need this to keep fpvectorial running in the Unicode RTL. Will really everyone have to roll their own TStringsUTF8? This would be really nasty because they would be incompatible from one another ... Let's see what FPC developers think about this idea...

In our previous episode, Felipe Monteiro de Carvalho said:

And I say more, two RTLs will immediately cause problems in all kinds of libraries.

Why?

Will the FCL work with the Ansi RTL, with the Unicode RTL, with both?

Generally both, and problematic packages are not coded in "string" but in "utf8string" or "unicodestring"

Won't that duplicate the difficulty in maintaining it if it has to be tested to work in both Ansi and Unicode mode?

No. The few cases where it really matters will be excempted. The whole idea is that the _general_ code follows suit, not _all_ code.

Will all packages from FPC work perfectly in both modes? Is that really feasable? How? {$ifdef UNICODE} everywhere that uses strings?

This is totally ignorant FUD. Please, if you don't understand the proposal, or Delphi's unicode implementation refrain from commenting on it.

And what if people start writing packages for the Unicode RTL, without IFDEFs to be compatible with Ansi, then LCL users cannot use them? What if you have a package for Ansi and another for Unicode, they cannot be used at the same time?

The compiler and remains fully UTF8 and UTF16 capable. Using the default "string" type only means statistically less automated conversions. Properly tagged string remain working but are less optimal outside their natural realm. (read UTF8 on Windows and vice versa)

RTL routines like assign(file) will probably be rawbytestring, which allows to pass utf8string and utf16string in a way that remains zero copy for utf16 on Windows and for utf8 on unix (assuming that is the default encoding on Unix)

Only for the classes lib this is harder, due to the fact that rawbytestring has a cumbersome implementation, and thus is probably not fit for virtual methods.

IMHO these are not the only options. IMHO Lazarus should simply flag everything as UTF8 and use the UTF8 RTLs as a base, and then reconsider if they stay that way long term, or migrate to a native encoding for each platform or not. The whole idea of the dual RTLs is that I'm not arrogant enough to pretend how everything with Lazarus will work out(or MSEGUI, or how long people will keep wanting to port ansi code). And don't worry, I can actually muster quite some arrogance if necessary, just not THAT much arrogance.

It is just plain common sense that we can't develop as quickly as Delphi and do the whole transition into every corner of the project within maximally an year. (even IF we all faced the same direction, we all have real lives and practical matters with existing codebases)

The dual RTL model allows to do this in a careful and planned manner, and without a bigbang period where Lazarus is unusable for half an year.

Porting all of the libraries and applications which I use for UnicodeString will likely be so huge of a task that I don't see any feasability in this kind of thing.

IMHO if we don't go for the dual RTL model (which is mainly conceived because of Lazarus' needs), then full Delphi/Unicode compatibility with string=unicodestring is the only longterm solution.

While I prefer to avoid rough breaks, like Embarcadero did by declaring all existing codebases legacy, it is the only route that is sustainable long term if we want to keep being Delphi compatible.

An own course will just be painless at first, and cost us dearly in the end. Remaining ansi only actually might be an easier course then.

We need to port SynEdit, the LCL, fpvectorial, regexpr... lots of applications which I wrote and nearly all of them do some kind of text parsing or handling...

You will to change all string types that MUST contain utf8 to utf8string and kill all conversions from other types.

That was the original idea, but that will bring you into trouble with inheritance on the UTF16 platforms (where the classes unit will be UTF16). Hence the idea of a dual RTL, at least for the transition period.

The assignfile() etc routines are actually not the problem. The classes in the classes unit are.

In our previous episode, Hans-Peter Diettrich said:

IMHO you are right, but the big difference is of course that the rest of the whole computer industry pretty much hasn't decided to stick to complex values as the standard.

Hans-Peter Diettrich wrote

UTF-8 is much more complicated to handle by the user, than e.g. UTF-16.

I don't see this. Please give an example?

Felipe Monteiro de Carvalho wrote

// file operations function FileExistsUTF8(const Filename: string): boolean; function FileAgeUTF8(const FileName: string): Longint;

....

fpGUI has similar, for file handling functions.

With the currently planned Unicode RTL it will just get worse, we will then need to either migrate to UnicodeString (a lot of work,

I guess that's the price we pay to stay with the times. ;-)

Hans-Peter Diettrich wrote

What data type would you use, to store an UTF-8 character? And how to access the n-th character in an UTF-8 string?

I already showed how in a previous post. For more details on how fpGUI does this, have a look at the fpg_base.pas and fpg_stringutils.pas units in the fpGUI repository.

  TfpgChar        = type string[4];

The maximum size any UTF-8 code point can be. Basically a character is treated just like a string - only a small string.

RTL is a mere *display* feature, the chars still are stored from first to

My problem is not the implementation, but the fact that I can't read or understand any right-to-left languages. :)

Please don't.
This was discussed a trillion times and the result was always that every encoding has its pros and cons.

Don't feed the trolls.

Mattias

Hans-Peter Diettrich wrote

When I want a program for German or French users, I'll hire an coder with experience in those *languages*, not with experience only in Unicode.

Why? We simply hired average people (not programmers) to translate your English resource strings into German, Portuguese, French etc.. No other modifications where required. No code or programs had to be recompiled. Our resource strings are stored in external .po file that ship with our application.

And yes, fpGUI has an option to change the default language from English to any other language. So we could recompile our app to default to German - thus no external German .po file is needed. But we have never had a need to do that.

errrm, utf-8 can have 6 octets representing one character, not forgetting those dioretics that are separate characters. (so I doubt representing it as above is so correct, with a short(ansi)string)

kind regards,
Dimitri Smits

Graeme,

while I concur with your first part as a way to go, it is maybe "not-so-average" people (linguists/interpreters), but other than that I agree.

Although there is a whole lot more involved with multilingual apps than that in my experience. Sometimes a form is to be redesigned slightly/heavily due to real-estate and constraints (either by system like screen resolution; or directly on demand by the client).

That is why I never really grasped the whole .po stuff as in my experience that is not always enough.

but I digress...

kind regards,
Dimitri Smits

2011/9/15 Hans-Peter Diettrich :

Right. My point was if I come from Ansi knowing MyString[1] retrieves first character and know nothing about Unicode, I might still think it continues to retrieve first character in Unicode regardless of string encoding (RTL handles that). It is as you say wrong, therefore the need to adapt the code by developer if he uses such access, but people might don't know this. Having UTF-16 RTL might help them in a sense they they will never have to learn, until they deal with characters outside of the BMP.

Hence the need to remove indexed access like MyString[1].

Which means they will have to learn it immediately.

That is of course, unless the application does not have any user input at all. As soon as an text input from a user is processed, never mind what language the user speaks => this user may for some reason enter text, where string[x] will return half a char/surrogate.

If it was utf8, the developer would probably encounter the error fairly soon, and learn before creating tons of wring code. with utf16, the developer may get away for many month, creating tons of code, that he needs to correct.

there are use cases where utf16 beats utf8 and vice versa.

but the argument "easier to learn" is a fake. Trying to hide a problem, and hoping it will not surface, has never been a good idea, why should it here?

Who will be the first to write a UnicodeString object that uses an AnsiString as buffer so we can start doing some tests? What is in the cpstrnew and other unicode branches of FPC? (sorry, I'm using a 3G limited connection and FPC doesn't have a viewvc...) Can we start putting well thought-out specs and use-cases in the wiki instead of the endless swinging discussion?

Best regards,
Flávio

Again: please answer my question first: How many *users*?

it can never know what the text contains.

The *user* may know that, and he will see the text, when loaded e.g. into a memo. When he wants to do more with the text, he has to choose the right application for the job.

Honestly, I don't understand why everybody assumes that every Unicode-aware application has to be a full-fledged text processor :-(

DoDi

There exist math apps, doing computations with complex numbers, and there exist text processors, dealing with Unicode. But what about the 99.9% of *other* applications?

DoDi

No.

Lazarus can continue to use UTF-8.

Just there will be an implicit conversion when using those functions. The overhead is minimum.

Lazarus is doing the (explicit) conversion already for some functions with no fact the code will be clearer

A conversion when calling those functions is necessary anyway because:

- if LCL changes to UnicodeString the conversion will be needed in unix (UTF-16 -> UTF-8)
- creating own UTF8 RTL functions will do the conversion internally anyway

So no problem at all with the proposition of Marco

Luiz

> RTL is a mere *display* feature, the chars still are stored from first to My problem is not the implementation, but the fact that I can't read or understand any right-to-left languages. :)

Don't worry, a compiler also doesn't understand any but his own language, other apps understand even less ;-)

DoDi

I also had a Chinese version of my VB3 decompiler, without any modification to the code. In so far you seem to agree with me, that most applications do not deserve any special text processing code?

DoDi

Take the example of FileExists:

The current LCL implementation - the UTF8 -> UTF16 conversion is done with the need of auxiliary code:

FileGetAttr_     : function (const FileName: String): Longint =
@FileGetAttrWide;
function FileGetAttrWide(const FileName: String): Longint;
begin
Result:=Integer(Windows.GetFileAttributesW(PWideChar(UTF8Decode(FileName))));
end;
function FileExistsUTF8(const Filename: string): boolean;
var
   Attr: Longint;
begin
   Attr:=FileGetAttrUTF8(FileName);
   if Attr  -1 then
     Result:= (Attr and FILE_ATTRIBUTE_DIRECTORY) = 0
   else
     Result:=False;
end;
function FileGetAttrUTF8(const FileName: String): Longint;
begin
   Result:=FileGetAttr_(FileName);
end;

The pure UTF-16 RTL: one implicit conversion is done conversion with clean/direct code

function FileExists(const Filename: unicodestring): boolean;
var
   Attr: Longint;
begin
   Attr:=FileGetAttr(FileName);
   if Attr  -1 then
     Result:= (Attr and FILE_ATTRIBUTE_DIRECTORY) = 0
   else
     Result:=False;
end;
function FileGetAttr(const FileName: unicodeString): Longint;
begin
   Result:=Integer(Windows.GetFileAttributesW(PWideChar(FileName)));
end;

The duplicate UTF8/UTF16 RTL - a conversion still necessary but internally and explicit

function FileExists(const Filename: utf8string): boolean;
var
   Attr: Longint;
begin
   Attr:=FileGetAttr(FileName);
   if Attr  -1 then
     Result:= (Attr and FILE_ATTRIBUTE_DIRECTORY) = 0
   else
     Result:=False;
end;
function FileGetAttr(const FileName: utf8String): Longint;
begin
Result:=Integer(Windows.GetFileAttributesW(PWideChar(UTF8Decode(FileName))));
end;

So:

- It will be a conversion anyway as is current. In fact current is worse because of temporary var created by chained typecasts/function calls
- The double multi RTL UTF8/UTF16 only adds extra code with no performance gain
- The fpc core/Marco proposition will help Lazarus have clearer/smaller code (at least regarding those RTL functions , TStrings etc is other story and where the real problem resides)

Luiz

2011/9/16 Martin :

I agree with you. Yet I don't think anyone is trying to hide a problem. It is the responsibility of each developer to learn before they use Unicode, instead of forcing FPC to have such encoding so that developers will be forced to learn sooner rather than later. From what I understand that argument is not "easier to learn" but "easier to transition to from Ansi if you don't care to learn". It's not an argument for me, but I'm sure there are other reasons as well. And since no encoding is easier to learn then it shouldn't matter which one is chosen.

2011/9/16 Luiz Americo Pereira Camara :

Lazarus can continue to use UTF-8. Just there will be an implicit conversion when using those functions. The overhead is minimum.

Currently UTF8String is just an alias for AnsiString, i.e., the implicit conversion UTF8String -> WideString does ansi -> utf-16 conversion.

Will UTF8String be a separate type recognizable by the compiler and UTF8String -> WideString will do utf8 -> utf-16 conversion?

Hans-Peter Diettrich wrote

Only when an application must *interpret* strings in foreign languages,

With UTF-8 German is such a foreign language :(

-Michael

cobines wrote

I understand that argument is not "easier to learn" but "easier to transition to from Ansi if you don't care to learn".

ANSI means: each element you get is a character.

With Unicode this is only (close to) true when using a 32 Bit encoding.

When using 8 or 16 bit Unicode encoding or (even worse) when mixing different encodings, in case there are "simple" language features that provide code bytes and not Unicode characters, life is obviously much harder. This is true even if you never did any programming using ANSI.

So the only user friendly solution seems to be _not_ to provide "simple" language features that deal with code bytes instead of Unicode characters _at_all_.

-Michael

What do you mean by the statement that FPC doesn't have a viewvc? Are you aware of "http://svn.freepascal.org/cgi-bin/viewvc.cgi/"?

Tomas

Regarding viewvc: http://svn.freepascal.org/cgi-bin/viewvc.cgi/

Regarding cpstrnew: That branch contains the infrastructure that is neccessary to think about a change of the string type at all: the code page aware string type.

Regards,
Sven

Felipe Monteiro de Carvalho wrote

Well, I think the RTL should introduce a TStringsUTF8 at the very least.

and/or (better ?!? ) introduce a basic string type name TStringUTF8.

I understand that cpstrnew is at least considered on the long run.

Is migrating to multiple string types (each denoting a certain encoding) and migrating to cpstrnew (a single string type with dynamical encoding) a contradiction or can it be consolidated ?

What is supposed to happen to the nasty legacy types called "String" and "Char" in that case ?

IMHO at best it would just be killed. A programmer can easily do "Type String = TStringUTF8; Char = TCharUTF8 (or choose using some different string type) in some central unit's interface part.

This approach would allow for continuing the "naive" use of "MyChar:=MyString[1];" (as would migrating to cpstrnew, but with less possible performance degradation).

-Michael

cobines wrote

Currently UTF8String is just an alias for AnsiString,

Which obviously is bound to produce a lot of confusion UTF-8 code in a thing explicitly called "ANSIString" ???

-Michael

Hans-Peter Diettrich wrote

FPC also allows to use Complex values - but nobody is forced to use such numbers

German (and French end, ...) Lazarus programmers are Forced to deal with Unicode if the accept user input. (Newer versions of) Lazarus can't be set to work in ANSI mode.

-Michael

Felipe Monteiro de Carvalho wrote

Well, I think the RTL should introduce a TStringsUTF8 at the very least.

and/or (better ?!? ) introduce a basic string type name TStringUTF8.

I understand that cpstrnew is at least considered on the long run.

Is migrating to multiple string types (each denoting a certain encoding) and migrating to cpstrnew (a single string type with dynamical encoding) a contradiction or can it be consolidated ?

What is supposed to happen to the nasty legacy types called "String" and "Char" in that case ?

IMHO at best it would just be killed. A programmer can easily do "Type String = TStringUTF8; Char = TCharUTF8 (or choose using some different string type) in some central unit's interface part.

This approach would allow for continuing the "naive" use of "MyChar:=MyString[1];" (as would migrating to cpstrnew, but with less possible performance degradation).

-Michael

Dimitri Smits wrote

errrm, utf-8 can have 6 octets representing one character,

Last time I checked, that was only in the very early stages of developing the utf-8 specification. Since then, the maximums size of a utf-8 code point is 4 bytes.

If you know otherwise, please post a URL. Here is the information I have:

"The original specification allowed for sequences of up to six bytes, covering numbers up to 31 bits (the original limit of the Universal Character Set). In November 2003 UTF-8 was restricted by RFC 3629 to four bytes covering only the range U+0000 to U+10FFFF, in order to match the constraints of the UTF-16 character encoding."

http://en.wikipedia.org/wiki/UTF-8#History

not forgetting those dioretics that are separate characters.

I'm representing a code point in TfpgChar. If you want the "completed character as is displayed on the screen", then simply normalize your TfpgString first, then extract the "character".

Regards,
- Graeme -

Just counted....

2,345,237,925

;-)

Regards,
- Graeme -

Are your trying to prove your point by answering questions, that can not be answered ( I don't even know how many users use FPC) ? If you need to go that way, that rather proves the weakness of your point.

My guess more than 90% of all apps deal with some form of user input [1]? So the answer is more than 90% of the users/developers/apps.

-
[1]
Even if the user input does not always contain such data, even if the user input in most cases does not contain such data. The point is the user input can contain such data (even if by accident/unintentional). As for writing an app, if it *can* happen, then the app *must* be able to deal with it.
So for any application: even a 0.0001% chance that it will happen, means the app must 100% deal with it.

That is unless you document: "if any of the input contains ..., then your app will [crash|fail| display nonsense]" => because of course if a crash is documented, it isn't a bug, it becomes a feature ;)

Marco van de Voort wrote

The assignfile() etc routines are actually not the problem. The classes in the classes unit are.

Ok, I may have exaggerated about the problems, but I still don't understand 100% your position. Where exactly is the frontier of how much utf8 support in the RTL is acceptable?

What do you think about adding TStringsUTF8/TStringListUTF8 to classes.pas?

What about stuff like this in classes:

  TReader = class(TFiler)
    function ReadString: string;
    function ReadWideString: WideString;
    function ReadUnicodeString: UnicodeString;

Can a utf8string method be added where there are already multiple methods for various string types?

And regardless of the answers above, I think we should start a new package in fpc called libutf8 which can add all kinds of classes and routines for utf8.

I'm clearly not understanding your (and a few others) logic here. Why must classes like that become obfuscated with such "extra" methods?

What is wrong with changing this class definition...

   TReader = class(TFiler)
     function ReadString: string;
     function ReadWideString: WideString;
     function ReadUnicodeString: UnicodeString;

...into this...

   TReader = class(TFiler)
     function ReadString: UnicodeString;

Where UnicodeString will be whatever the native encoding is on each platform. The class interface stays neat and readable, no conversions required on any platforms etc..

Adding a new overloaded method to each class that handles strings, or multiple versions of classes handling different encodings just seems the wrong way to go. Talk about code duplication!

Regards,
- Graeme -

AFAIK, yes

Luiz

good to know.
I've learned about unicode/utf8 from the following links http://www.joelonsoftware.com/articles/Unicode.html http://www.cl.cam.ac.uk/~mgk25/unicode.html

never bothered to look into the rfc's and/or official unicode site(s).

when I follow the link to the rfc you mentioned in the second link above, I indeed see that it is 4 octets according to the rfc. However, when I follow the link to the unicode appendix (http://www.cl.cam.ac.uk/~mgk25/ucs/ISO-10646-UTF-8.html), mentioned in that second page (anchored link: http://www.cl.cam.ac.uk/~mgk25/unicode.html#utf-8), I see that according to the iso spec, it still is (was?) 6.

kind regards,
Dimitri Smits

In our previous episode, Felipe Monteiro de Carvalho said:

Note that this is all my, not necessarily core's opinion.

IMHO none. We have never made a choice for UTF8, and decided to go in a Delphi compatible way that is both more compatible and powerful.

And I believe that route is also more avantageous long term for Lazarus.

What do you think about adding TStringsUTF8/TStringListUTF8 to classes.pas?

I think this is a slippery slope. These kinds of hacks are slipped in one by one, and each one is only a small concession, but in the end it is a disaster. I don't want to create and maintain UTF8 versions of nearly every class, even when the class doesn't actually do anything UTF8 specific.

So in my opinion: NO!

The constant pressure from the Lazaurs team was the main rationale to come up with two RTLs. Since the original unicode discussion on core (early 2009, just before 2009 came out) came up with a type that was mostly UTF16 on Windows.

People always whined about overloading as a solution, but that won't work because of virtual methods with string parmaeters or returnvalues in it.

Moreover even if Lazarus decided to migrate to that, it would be totally broken for a long while till it caught up. The current route is friendlier.

In the UTF8 RTL, all "string"s _ARE_ utf8, unless specified otherwise (by naming them unicodestring or ansistring(..encoding) or shortstrings).

So the same virtual method with a STRING parameter will be TUnicodestring in the UTF16 rtl and UTF8string in the utf8 RTL.

An additional advantage is that Delphi code that just passes strings along won't need modification (by replacing everything with -UTF8) versions.

Most simple RTL routines that accept a string, but are not string type specific (think fileopen createdir etc) accept rawbytestring, a type that accepts all ansistring types and unicodestring. IOW you can also pass an UTF8 to it, even in the UTF16 rtl.

I hope though that Lazarus in time will see the light and change the Windows port to the UTF16 RTL, since when the manual conversions are removed, the places where encoding matters decreases significantly. (and the places where the automatic ones happen can vary without codechanges)

But that discussion is for after the big transitions, when all the current manual hacks are removed, and as said that might only be when 2.8 is released.

Actually I see only two solutions, either this or follow full Delphi/unicode compatibility. It's a pity for Lazarus, but full Delphi/unicode compatibility is an absolute must for me. If not, we set us totally apart, since the "old" delphi's will decrease in amount every year.

No UTF8 method. But "string" is utf8string in the UTF8 RTL.

But filer is special I guess, these methods might not be just converting aliases. It's too soon to say how all this work out in (how I see) 2.8. The rough designlines are visible and would be introduced bottom to top.

And regardless of the answers above, I think we should start a new package in fpc called libutf8 which can add all kinds of classes and routines for utf8.

I don't see the point of that. I don't see the reason to move the workarounds of Lazarus manual UTF8 conventions into the FPC repository that doesn't support those conventions. Specially since it is only for the 2.6 series that is already branched, because after that new solutions will remain available.

I can understand the confusion. There are lots of old and outdated information regarding UTF-8 on the net.

Regards,
- Graeme -

If we support an UTF-8 version of the RTL, then either the code must work both for UTF-16 and UTF-8, or it has to be separately maintained anyway. And if the code works for both types, generics should enable having both without source code duplication.

Note that I'm not specifically arguing here for adding them or not, but I don't think the maintenance will be much higher than what will be imposed already by requiring that the RTL be compilable for both UTF-8 and UTF-16.

Jonas

Marco van de Voort wrote

In our previous episode, Felipe Monteiro de Carvalho said:

.
.

.
.

Sorry, one thing I'm missing in this point - where exactly is the indexed (SBCS codepage based) version in this if string always means either UnicodeString or UTF8String depending on the context / defines? Would there be no SBCS version any longer, or is this a third option, or what? Was your point about "string", or "RTLString"?

Tomas

In our previous episode, Jonas Maebe said:

The -UTF8 hack is simply because of inheritance and the because NOW they still need to insert manual conversions.

In the UTF8 and UTF16 rtl, the inheritance problems are gone by selecting the proper RTL and the conversions are automated due to cpnewstr.

Note that I'm not specifically arguing here for adding them or not, but I don't think the maintenance will be much higher than what will be imposed already by requiring that the RTL be compilable for both UTF-8 and UTF-16.

I'm not sure that is really the case, both because only really encoding sensitive places are affected in that case. Moreover, most of the remaining changes will be necessary anyway because of Delphi2009 functionality, also if we only do one RTL.

Stuff that gets input like TStringList.Loadfromfile will receive encoding loading options anyway as per Delphi/unicode compat. (this becuase the encoding of the file that you load must be runtime selectable, and is not necessarily the same as your default encoding)

So I don't expect the recompiling of the RTL with for UTF8 and UTF16 to require that much encoding specific changes. Basically it just parameterises the classes trees with a stringtype, and maybe changes the names of some RTL string routines which must be made for utf8 and utf16 anyway. (so more or less that in the UTF* rtl e.g. trimleft is utf8 and trimleftutf16 is utf16, while in the UTF16 rtl it is the otherway around, but one still has to create both anyway in any solution)

The whole point is to avoid messing too much with overloading umpteen stringtypes and mimimize changing existing code (both FPC, Delphi and delphi/unicode) with suffixes (-UTF8 etc). Of course when one wants to blend UTF16 Delphi code in a UTF8 rtl, fixes might be needed, but at least if your code is _mostly_ UTF16, you have the option to go to the utf16 rtl.

And then speicifically avoid modifications to virtual methods. Having multiple versions of virtual methods overloaded (same name or not) is very dangerous, since people might only override the wrong one ( just see the seek32/64 case)

Moreover, nobody is wronged in the sense that "his" choice is second rate and must go through conversion layers, and the general principles are clear for everbody, without exhausting discussion at every single modification.

At the expense of a few more release binaries to build, and dealing with bugreports in a different RTL then you would typically use. I admit that, but still think that it is a netto plus by a wide margin, and the _extra_ work that needs to be done on the code itself is consistently overestimated.

If we remain delphi compat and only allow UTF16, we will over time get bugreports to make UTF8 variants for everything. Better require people to submit a working solution for both from the beginning.

Every project (Lazarus, Delphi/Unicode compats, ansi legacy) picks a RTL fit for their purpose, and just uses STRING for the bulk of the code.

Marco van de Voort wrote

In the UTF8 RTL, all "string"s _ARE_ utf8, unless specified otherwise (by naming them unicodestring or ansistring(..encoding) or shortstrings).

This is somewhat interesting, but then Lazarus and fpvectorial would only work in the UTF-8 RTL. They would not work with the Unicode RTL and they would not be compatible with libraries written to work only for the Unicode RTL. I think it is more stable in the long run if the LCL, fpvectorial and fpspreadsheet simply no longer use TStringList and use an utf-8 variant from libutf8.

I hope though that Lazarus in time will see the light and change the Windows port to the UTF16 RTL, since when the manual conversions are removed, the places where encoding matters decreases significantly. (and the places where the automatic ones happen can vary without codechanges)

I am totally against using a string type that changes from platform to platform.

For me it is completely the opposite, this is mostly useful for 2.8+

The LCL and fpvectorial (and fpspreadsheet and etc) should work with the Unicode RTL. fpvectorial and fpspreadsheet should theorically even be able to work with Delphi if necessary one day. So being locked in a UTF-8 RTL which other people are not using is not a long term solution.

fpvectorial and fpspreadsheet use TStringList to pre-parse text which is separated in lines. Therefore they cannot use a UTF-16 TStringList, much less a unknown string TStringList. It must be a raw UTF8string TStringList (raw as in supporting zero conversions, conversions could also be added, but zero is a must), not more, not less. If the RTL cannot provide this class, they need to use another one. I don't want to duplicate code in the LCL and in fpvectorial and in fpspreadsheet, so I propose that I be allowed to start a libutf8 which will be a dependency for all 3.

In our previous episode, Tomas Hajny said:

It is a third option but only maybe for a while on Windows and the only option platforms that can't or won't support unicode. (like Dos)

The idea is more or less that this trick can be employed for any ansi/unicodestring type. The shorstring overloads are already there and probably can stay. So Dos or OS/2 are not in danger.

It also means three possible options for Windows. But ascii is temporary, and Windows/utf8 is only for Lazarus. (which I hope will see the light and migrate to utf16 in time too)

Was your point about "string", or "RTLString"?

I'm thinking about "string", but that is more directed towards the OOP parts, which assume a objfpc{$h+} or Delphi mode.

So the base RTL functions like fileopen will be rawbytestring that accepts _all_ encodings, (so also ansi/utf8/utf16 in ansi/utf8/utf16 mode) and runtime convert if necessary and possible or explicitely typed. It depends on the amount of routines that don't fall into these categories if something like RTLSTRING is possible.

Of course the "accepting" of all encodings is on interface levels. Implementations for platforms are dos are not supposed to support them all, just the ones they always did.

In our previous episode, Luiz Americo Pereira Camara said:

All the routines you name (fileexists, filegetattr etc) will become rawbytestring and accept both utf8 and utf16, and then only in the implementation figure out if conversion is necessary. (to the encoding of the OS)

And they will accept both those encodings in both RTLs. The different RTLs are more for the classes hierarchy.

On 09/15/2011 07:39 PM, Hans-Peter Diettrich wrote: > > Only when an application must *interpret* strings in foreign languages, With UTF-8 German is such a foreign language :(

That's why European users will be happier with UTF-16 (meaning UCS2).

An UCS2String type could be implemented to substitute astral characters in the conversion from full Unicode, as is the well known procedure with Ansi strings. But such a string type will only make users happy, not the FPC maintainers ;-)

BTW, I actually don't know whether or where the FCL or LCL uses Ansi or UTF-8, never came across problems with string encodings in my own code. In so far I could live with UTF-8 in the future, and with components that don't require different Ansi/Unicode versions and conversions.

The same ignorance may apply to POSIX filesystems, which do not have to care about case insensitive (Windows) names and upper/lower conversions. Everything a filename/path parser requires are unique separators, and all these are SBC ASCII chars :-)

DoDi

"string" has been a generic type since long, also "char" (at least in Delphi). This would allow to map these to any decent type, as Delphi did in the move to Unicode.

DoDi

I also don't understand that, because external files must have a unique format, or they are not exchangeable. Here UTF-8 has the big advantage of supporting full Unicode, and the disadvantage that the length of the string in a different encoding is not known in advance. I'd leave it to the compiler, to insert eventually required conversions, instead of bloating the reader and writer classes with extra conversions.

What is wrong with changing this class definition...

[...]

TReader = class(TFiler) function ReadString: UnicodeString;

This IMO should read UTF8String, usable in every environment without external (platform, string implementation...) dependencies. Two more classes that are not affected in any way by other decisions about string types, immediately usable even in a split {$IFDEF Unicode} model.

DoDi

Understood. I was not asking about shortstring but mostly about "ansistring" and "string".

It also means three possible options for Windows. But ascii is temporary, and Windows/utf8 is only for Lazarus. (which I hope will see the light and migrate to utf16 in time too)

.
.

I guess that it's to be seen how much "temporary" is it for Windows, but that does not matter really for me personally.

OK, I see. I assume that an option to override the "string" meaning (similarly to current $H+/-) among ansi/utf8/utf16 would be created then (if not already available in cpstrnew), right?

Tomas

At least in the JVM branch it exists already when using "{$modeswitch unicodestrings}{$H+}" (I don't know whether Jonas implement it generally or only for JVM though). See:
http://wiki.freepascal.org/FPC_JVM/Language#String.2C_unicodestring_and_java.lang.String

Regards,
Sven

This also doesn't answer my question :-(

Perhaps I should have stressed *deal with* astral characters. Input and output require no explicit code at all, so that the *chance* of occurence of such characters is unimportant.

Which real-life code will have to e.g. parse such strings, in a way that astral characters can get into the way at all? When strings are split at specific locations (whitespace, list or path separators...), these places can be determined by Pos or similar functions, regardless of the chars in between these delimiters. Likewise upper/lowercase conversions can be done with according (Unicode aware) standard functions, working on entire strings, not on char level.

Please give a reasonable (practical) example, where an application *must* deal with (such) chars. (see above)

So for any application: even a 0.0001% chance that it will happen, means the app must 100% deal with it.

*Not* dealing with string *contents* will reduce that chance to definitely zero.

That is unless you document: "if any of the input contains ..., then your app will [crash|fail| display nonsense]" => because of course if a crash is documented, it isn't a bug, it becomes a feature ;)

Looks like your apps also will toggle bits in Real values, and tend to crash with zero-divide later ;-)

SCNR

DoDi

You also confuse *accept* Unicode and *deal with* Unicode :-(

DoDi

Better.

And they will accept both those encodings in both RTLs. The different RTLs are more for the classes hierarchy.

OK.

Much of this discussion is because of lack of understanding of the fpc core / proposition.

It would be good add the current plan to
http://wiki.freepascal.org/FPC_Unicode_support even if is a draft

Luiz

I took a look at the Delphi An Unicode document by Marco Cantu and a get doubt about the implementation with RawByteString:

Assume the implementation under windows of FileGetAttr:

with UnicodeString (there's no need to conversion - all good):

function FileGetAttr(const FileName: UnicodeString): Longint;
begin
   Result:=Integer(Windows.GetFileAttributesW(PWideChar(FileName)));
end;

with RawByteString (need the conversion - but how ?????):

function FileGetAttr(const FileName: RawByteString): Longint;
begin
   // how to convert?
   // UnicodeString(FileName) -> will not work because dont know if is
was a UTF8 or UTF16 or whatever before passing as RawByteString
parameter. SetCodePage will also not work.
   Result:=Integer(Windows.GetFileAttributesW(PWideChar(FileName)));
end;

Luiz

Marco van de Voort wrote

Most simple RTL routines that accept a string, but are not string type specific (think fileopen createdir etc) accept rawbytestring, a type that accepts all ansistring types and unicodestring. IOW you can also pass an UTF8 to it, even in the UTF16 rtl.

Please take a look at other mail i sent. AFAIK RawByteString will not work here since it looses the encoding info so can't know if a conversion is required and if yes don't know what conversion do

Luiz

Of course this is known. The code page field of the string describes it. RawByteString means only: eat any string without conversion.

[..]

with RawByteString (need the conversion - but how ?????):

Nevermind i found the answer at
http://www.micro-isv.asia/2008/08/using-rawbytestring-effectively/

"Then |ShowCodePage(A)| will display 1252, and |ShowCodePage(F)| will display 65001. The StringCodePage function gets the code page information stored in the string at runtime, not the declared code page, which is 65535 for RawByteString. This means that StringCodePage is only meaningful when calling it on a string variable that actually holds a string. If it doesn’t hold a string, StringCodePage cannot retrieve the code page, and it will always return the default code page."

The codepage of a RawByteString at runtime will keep the previous CodePage (65001 for UTF8, 1200 for UTF16) as opposed to change to the RawbyteString CodePage (65535) as a though previously

So the implementation would be:

function FileGetAttr(const FileName: RawByteString): Longint;
begin
SetCodePage(FileName, 1200, True);
Result:=Integer(Windows.GetFileAttributesW(PWideChar(FileName)));
end;

This way the version using UnicodeString parameter would have the benefit of being less verbose and use the possible optimizations of the implicit encoding conversion.

Luiz

+100000000000000000~

this is something that i'm having to deal with with 30+ years of pascal programing... i'm still trying to wrap my head around this GUI coding stuff... while i do have some similar experiences with other languages from way back (dBIII, dBIV and such that have/had forms) forms style coding is still alien to me... i'm used to simply clearing an 80x25 screen and then drawing my next screen... if i need to easily return to a previous screen, i might redraw it or i might restore it from a saved buffer and "blit" it back onto the screen...

i don't know the difference between thisstring and thatstring and there are times that this is one of the worst problems i face... my first hurdle was clearing the 255 character strings that i'm so used to dealing with... it used to be that i used a custom written ACSIIZ convertor routine but now it seems that these are in the run time libraries and i need only to choose the proper strings to convert between... even then, it can be quite the chore :?

moew old school stuff here... a BMP is a windows style graphic... what are you guys calling a BMP???

and agreeing (fully!) there really should be some sort of "hidden(?)" overrides taking place so that folks like myself don't really have to worry about this stuff... but then again, maybe? i dunno? i'm not sure, these days, after a year+, if i'm no the right track or not...

removing that is not a GoodThing<tm> is it? really? as i say above, it would seem to me to be best for the library to handle all of this stuff in the same way that borland handled similar things way back when... on the one hand, it would seem easy enough to handle with automatic overrides but then again, i may not be thinking of things in the same way as others used to working with this OO oriented method of coding...

ROTFLMAO!!!
LMGTFY: "Basic Multilingual Plane"... :-D

hth,

Ralf ;-)

Clearly, in 30 years, not only the pascal language has changed....

Ludo

This might be total crap, so bear with me a moment, In an object like a Stringlist, there is a default property such as Strings, such that List.Strings[1] is equivalent to List[1], is there not? If, as in .NET or Java, all strings become objects, then you could have a String object whose default property is Chars, whose type isn't really a char, but another String whose length is one entity. Thus code that was written as MyString[1] would essentially behave the same as for old shortstrings (ignoring for a moment the difference between ' and " in specifying a literal char or string). This string object could have a property called Encoding that determined if it was UTF8 or UTF16 or ANSI or raw bytes, and methods to trigger a direct conversion, and AsXXX properties to convert for use in routines that needed a different encoding, the need for which can be determined at run time.

This may not be even feasible, and it will break some legacy code, but I think those used to old style strings could get used to it very quickly, and most of the details would be buried in the RTL where people who didn't want to see them would not. Of course, any code using move or treating a section of shortstring as an open array would still break, but perhaps that is a good thing. We aren't running on 8088's any more so dangerous tricks in search of speed shouldn't be needed as often, who are looking for super efficiency won't find it here, but I think months of discussions and thousands of messages pretty much prove this is a complex task and only a complex solution will solve it. Even if the solution is nothing like the above, it is fairly obvious that any attempt at a simple "one type fits all" solution won't work, and if you are going to have to have multiple solutions, to me it is better to bite the bullet and implement a full solution.

Jeff.

The codepage of a RawByteString at runtime will keep the previous CodePage (65001 for UTF8, 1200 for UTF16) as opposed to change to the RawbyteString CodePage (65535) as a though previously

Delphi defines RawByteString=AnsiString, so there is no room for UTF-16 in such an string.

Won't work, because of "const", and because UTF-16 is not a Byte (AnsiChar) string :-(

Result:=Integer(Windows.GetFileAttributesW(PWideChar(FileName)));

Delphi would use
   Result:=Integer(Windows.GetFileAttributesW(PWideChar(string(FileName))));

with a temporary UnicodeString variable and an according try-finally block.

At best it *hides* the temporary variables and implicit conversions, but makes stringhandling more expensive. As I understand the FPC developers, they want to reduce the number of implicit string conversions, what can be achieved best with dedicated string types.

DoDi

No. I was wrong. See Florian email. RawByteString will keep the codepage (1200 = UTF16) and the data of the assigned string be UTF8, be UTF8.

Yes

and because UTF-16 is not a Byte (AnsiChar) string :-(

No. See above. Look in net for Delphi and Unicode doc by marco cantu

Yes

I'm talking about:

function FileGetAttr(const FileName: UnicodeString): Longint;
begin
   Result:=Integer(Windows.GetFileAttributesW(PWideChar(FileName)));
end;

Inside the procedure there will be no conversion since is already UTF16, just a typecast to PWideChar which in fact is a function

The conversion will be done before the function call only if necessary (eg UTF8 -> UTF16). The decision to convert or not is done at compiler time.

With RawByteString

function FileGetAttr(const FileName: RawByteString): Longint;
begin
Result:=Integer(Windows.GetFileAttributesW(PWideChar(UnicodeString(FileName))));
end;

Here the decision to convert or not is done at runtime by checking the CodePage of FileName. Also there's one more temp variable due to UnicodeString typecast.

In summary:
With UnicodeString decision to convert at design time With RawByteString decision to convert at run time + one more temp variable

As I understand the FPC developers, they want to reduce the number of implicit string conversions, what can be achieved best with dedicated string types.

I just saying that ;-) UnicodeString better than RawByteString

Luiz

That's somewhat what I was thinking. Actually something like

  UnicodeString = object
  strict private
    FEncoding: Integer;
    FBuffer: AnsiString;
    function GetCodePointAt(AIndex: SizeInt): Integer;
    procedure SetCodePoint(AIndex: SizeInt; p_Value: Integer);
  public
    property CodePoint[AIndex: SizeInt]: Integer read GetCodePointAt
write SetCodePoint; default;
  end;

I just don't whether something like this is already implemented in the test branches, at least for -err- testing...

-Flávio

Well... you can now take a look at trunk as well, because the changes from cpstrnew have been merged yesterday.

Regards,
Sven

Sven Barth wrote

Well... you can now take a look at trunk as well, because the changes from cpstrnew have been merged yesterday.

[...]
make[7]: Entering directory `/home/mse/packs/standard/svn/fp/trunk/rtl/linux' /home/mse/packs/standard/svn/fp/trunk/compiler/ppc2 -Ur -Ur -Xs -O2 -n - Fi../inc -Fi../i386 -Fi../unix -Fii386 -FE. -
FU/home/mse/packs/standard/svn/fp/trunk/rtl/units/i386-linux -di386 -dRELEASE -Us -Sg system.pp
This binary has no unicodestrings support compiled in. Recompile the application with a unicodestrings-manager in the program uses clause.
Fatal: Compilation aborted
An unhandled exception occurred at $08056B3F :
ENoWideStringSupport : SIGQUIT signal received.
$08056B3F
$08092A8C
$080926DD
$080F121E
$080F48D5
$080F5CEF
$08067532

Martin

Currently the POSIX-based systems seem to be broken (the Windows ones work). That is already known. The other devs are working on that.

Regards,
Sven

In our previous episode, Fl?vio Etrusco said:

Such ability is not unique for an object. One can also do something like that with a native type. It was discussed and rejected.

The trouble is that it is not that easy, consider the first thing a long time pascal user will do is fix his existing code which has many constructs that loop over a string:

setlength(s2,s1);
for i:=1 to length(s1) do
  s2[i]:=s1[i];

Now, to return codepoint[i], you need to parse all codepoints before [i].

So instead of O(n) this loop suddenly becomes O(n^2)....

For now you can apply the following patch as a workaround. The compiler (and fpmake) will depend on the C-library then (which should not be the case in the final solution).

Regards,
Sven

--- fpc/compiler/pp.pas 2011-06-21 20:29:50.340618090 +0200 +++ fpc-build/compiler/pp.pas 2011-09-18 12:11:46.823598604 +0200 @@ -166,6 +166,9 @@

  {$ifdef profile}
    profile,
  {$endif profile}

+{$ifdef unix}
+ cwstring,
+{$endif}

  {$ifndef NOCATCH}
    {$if defined(Unix) or defined(Go32v2) or defined(Watcom)}
      catch,

--- fpc/packages/fpmkunit/src/fpmkunit.pp 2011-09-18 11:44:03.176488622 +0200
+++ fpc-build/packages/fpmkunit/src/fpmkunit.pp 2011-09-18 12:14:34.279871962 +0200
@@ -45,6 +45,9 @@

uses
SysUtils, Classes, StrUtils
+{$ifdef unix}
+ ,cwstring
+{$endif}

  {$ifdef HAS_UNIT_PROCESS}
    ,process
  {$endif HAS_UNIT_PROCESS}

Sven Barth wrote

For now you can apply the following patch as a workaround. The compiler (and fpmake) will depend on the C-library then (which should not be the case in the final solution).

Not only that: even with cwstring (and under Windows) the result is wrong, because doing ansistr1=ansistr2 will currently be evaluated as a case-insensitive string comparison under certain circumstances.

Jonas

shudder. ;-)

Martin

Sven Barth wrote

Currently the POSIX-based systems seem to be broken (the Windows ones work). That is already known. The other devs are working on that.

And it boggles the mind why something so broken / incomplete was merged into Trunk in the first place? Isn't that the whole point of Branches. Make sure it works after being synced with latest Trunk. If the branch works successful, then merge into trunk.

Graeme Geldenhuys wrote

And it boggles the mind why something so broken / incomplete was merged into Trunk in the first place?

Yes, we suck from time to time (in this case: testsuite runs were performed, but the sync and merge were done by a person who only had access to a Windows system). Can we move on now?

Jonas

Also the developers decided that it's better to continue the work on that feature in trunk as errors are then spotted more easily and thus can also be fixed faster (you remember how orphaned cpstrnew basically was, Graeme?).

Regards,
Sven

Of course. That wasn't meant as a real implementation, I just decided to write some code instead of explaining in words. Basically my point was to people discussing endlessly without any data or observations, that FPC already provides much of the tools for a non-native implementation to be made and gather real and practical data.

I hope then that either I'm wrong or that you change your mind ;-) IMHO what must be changed is the way to deal with strings. I must assume from this preoccupation that you're talking about a a directive to make the String keyword instantiate a UnicodeString? Also IMVHO in that compiler mode the code just needs to work, not fast, and the user code be updated/fixed.
One obvious way to mitigate this would be to store the last CodePoint->Char in the string record, so that at least the most common case is covered.

Best regards,
Flávio

PS. Sorry for the double post, Marco.

Can you give me a link? I checked the XE documentation and RTL, and could not find that RawByteString can hold UTF-16, and my test confirms that:

var
   a: AnsiString;
   u: UnicodeString;
procedure test(r: RawByteString; cp: word);
begin
   WriteLn('in:  ', StringElementSize(r), ' cp: ', StringCodePage(r), '
len=', length(r));
   WriteLn('"', r, '"'); //writes garbage for non-OEM chars, of course
   SetCodePage(r, cp, true);
   WriteLn('out: ', StringElementSize(r), ' cp: ', StringCodePage(r), '
len=', length(r));
   a := r; //use the result, so that nothing can be optimized away
   WriteLn('"', r, '"');
end;

This reveals the following behaviour:

1) UnicodeString is converted to AnsiString, before passed to test.
2) Setting codepage to 1200 doesn't change anything.
3) Conversion to UTF-8 seems to work (length changed).
4) Conversion from UTF-8 to Ansi results in an empty string.

I'll ask in an Embarcadero group, in detail for [4].

DoDi

http://edn.embarcadero.com/article/38980

You may read also:

http://www.micro-isv.asia/2008/08/using-rawbytestring-effectively/

Are you using Delphi XE or fpc?

I dont have Delphi XE. What i know is from that docs and these discussions

Luiz

Flávio Etrusco wrote

One obvious way to mitigate this would be to store the last CodePoint->Char in the string record, so that at least the most common case is covered.

... and so that the common case is broken in multithreaded environments.

Directly indexing a string will most likely always work using fixed-length steps (8, 16, 32 bit). If you want to iterate based on anything else (such as code points), use some kind of iterator model instead.

Jonas

Sure it does. So what? The point is, it will do what the user expects. And for most users, the fact that it does it slowly won't even matter. For those whom it does matter, it is a chance for them to learn the right way. Like I said in my first post, this is an extremely complex subject. I think trying to optimize user code before they even write it adds even more complexity, which slows implementation down. Get something that works and gives the expected results first, worry about speed later. By the time you finish, the CPU speed will have caught up to you.

In our previous episode, DaWorm said:

> > So instead of O(n) this loop suddenly becomes O(n^2).... Sure it does. So what?

So much!

The point is, it will do what the user expects.

No it doesn't. The user has no clue, and will just stumble on the next detail (like codepoints not being characters).

Like I said in my first post, this is an extremely complex subject. I think trying to optimize user code before they even write it adds even more complexity, which slows implementation down.

As often repeated: IMHO users can make such decisions for their app logic.

The libraries (and most of Lazarus) can't make such speed and Unicode subset assumptions, and they are heavy "string" users too.

And of course, finally, there is the matter with Delphi compatibility.

Let me quote a saying by Pascal's father Wirth:

"Software is getting slower more rapidly than hardware becomes faster." (see also here: http://en.wikipedia.org/wiki/Wirth%27s_law )

I personally see no reason in Pascal becoming (much) slower only because we want to support code page aware strings (and O(n^2) IS much slower than O(n)).

Regards,
Sven

Sven Barth wrote

On 18.09.2011 17:48, DaWorm wrote:

But isn't it O(n^2) only when actually using unicode strings? Wouldn't you also be able to do something like String.Encoding := Ansi and then all String[i] accesses would then be o(n) + x (where x is the overhead of run time checking that it is safe to just use a memory offset, presumably fairly short)? Of course it would be up to the user to choose to reencode some string he got from the RTL or FCL that way and understand the consequences.

What assumptions are the typical String[i] user going to make about what is returned? There will be the types that are seeing if the fifth character is a 'C' or something like that, and for those there probably isn't too much that is going to go wrong, they might have to switch to "C" instead, or the compiler can make the 'C' literal a "unicode char which is really a string" conversion at compile time. There may be the ones that want to turn a 'C' into a 'c' by flipping the 6th bit, and that will indeed break, and in a Unicode world, perhaps that should break, forcing using LowerCase as needed. And there are those (such as myself) who often use strings as buffers for things like serial comms. That code will totally break if I were to try to use a unicode string buffer, but a simple addition of String.Encoding := ANSI or RawByteString or ShortString in the first line would fix that, or I could bite the bullet and recode that quick and dirty code the right way. My point is that trying to keep the bad habits of a single byte string world in a unicode world is counterproductive. They aren't the same, and all attempts to make them the same just cause more problems than they solve.

As for the RTL and FCL, presumably they wouldn't be doing any of this Sting[i] stuff in the first place, would they? So they aren't going to suffer that speed penalty. Just because one type of code is slow, doesn't mean everything is slow.

Jeff.

In our previous episode, DaWorm said:

It is possible, but that state can't be in the string/object because for read-only access strings are shared. (not doing so incurs a lot of copying overhead)

So that means that you need to allocate that state locally, either explicitely by manually allocating an iterator object (as Jonas already explained) or implicitely on the stack. The latter requires a native string type though, and is therefore hard with objects.

Implicit methods also have the disadvantage that the compiler must recognize the access pattern. So usually that means only the simplest of cases (or e.g. only when for..in is used)

What assumptions are the typical String[i] user going to make about what is returned?

IMHO development should not be driven by the users assumptions.

If so, we would now have UIs with one red button with the text "do what I think", since that seems to be what most users want and expect :-)

This is a very rare case. With the increasing internationalization of
applications operations on such literals are even rarer.

2011/9/18 Marco van de Voort :

Correct me if I'm wrong, but length(s1) wouldn't return the number of code points anyway?

All MBCS encodings, with no fixed character size, suffer from that problem.

Calling subroutines for indexed access, instead of direct array access, will add another factor (10..100?) to single character access - including register save/restore and disallowed optimizations.

The simple upper/lower conversion works only for ASCII, not for Ansi chars.

Delphi introduced TBytes for non-character byte data.

My point is that trying to keep the bad habits of a single byte string world in a unicode world is counterproductive. They aren't the same, and all attempts to make them the same just cause more problems than they solve.

That's why I still suggest to use UTF-16 in user code. When the user skips all unknown chars, nothing can go wrong.

As for the RTL and FCL, presumably they wouldn't be doing any of this Sting[i] stuff in the first place, would they? So they aren't going to suffer that speed penalty. Just because one type of code is slow, doesn't mean everything is slow.

It's absolutely safe, even with UTF-8 strings, to e.g. search for all '\' separators, and to replace these in place with '/'. It's also safe to search for an set of (ASCII) separator chars, and to split strings at these positions (e.g. CSV). Bytewise case-insensitive comparison also works for all encodings, at least for equality. Other comparisons are much slower, due to the required lookup of the sort order values (maybe alphabetic, dictionary etc.), and again with every encoding. Even with ASCII there exists a choice of sorting 'a' like 'A', after 'A' or after 'Z'.

DoDi

Thanks, but that's nothing new to me in general, and the RawByteString handling doesn't work as documented.

Are you using Delphi XE or fpc?

I tested with XE, because I wanted to learn more about the possible use or RawByteString in the RTL. The result was not very enlightening, except for the fact that many years after introduction of string encodings and RawByteString the Delphi RTL still contains severe bugs :-(

I dont have Delphi XE. What i know is from that docs and these discussions

That's why I wanted to test with Delphi first, before I start finding bugs or incompatibilities in the FPC implementation.

DoDi

procedure ShowCodePage(const S: RawByteString);
begin
   Form1.Caption := IntToStr(StringCodePage(S));
end;

Strange

What value you get passing and UTF8 and UTF16?

According to that site you should get 65001 and 1200

You may try also with UTF16 and UTF8 it should implicitly convert to UnicodeString

procedure ShowWithAnyCodePage(const S: RawByteString);
begin
   Form1.Caption := S;
end;

Luiz

end;

UTF-8 comes in as 65001, and UTF-16 as Ansi.

You may try also with UTF16 and UTF8 it should implicitly convert to UnicodeString

Right.

DoDi

By "the most common case" I meant non-threaded ;-) But no, I don't see any trivial and efficient solution to avoid the worst case (but among threadvars, per-string fixed lookup table, shared lookup caches, per-reference data (like Object), etc, there must be a good solution). Basically I think the UnicodeString should move farther (than AnsiString) away from PChar, from the compiler/RTL POV. I think that the user should (have to) use the iterator model to *efficiently* iterate over the string, but I see indexed access as a compatibility feature, and as such should care more about correctness and ease-of-use rather than performance. I thought the endless bugs WRT to char vs codepoint indexes, even in Java-developed software, would buy my argument...

-Flávio

In our previous episode, Fl?vio Etrusco said:

compatibility feature, and as such should care more about correctness and ease-of-use rather than performance. I thought the endless bugs WRT to char vs codepoint indexes, even in Java-developed software, would buy my argument...

IMHO you are seeking the problems in the tools, while the problem is PEBKAC

The base principle should be to mess with strings as little as possible, in that sense you are right.

I don't like the Java/C# way that you have to manually allocate extra objects (stringbuilders etc) to get(performant) access to the characters though.

Marco van de Voort wrote

I don't like the Java/C# way that you have to manually allocate extra objects (stringbuilders etc) to get(performant) access to the characters though.

In Java that's only the case for changing characters. Reading characters happens via a simple accessor and since the java.lang.String class is final it can be inlined from the start without any problems.

Jonas

I partly agree it's PEBKAC, but why make it easy to get wrong when you can avoid it? Isn't that the point of Pascal? Isn't that the point of AnsiStrings? Isn't that the point of strong typed languages in general?

At least we are not tied by bytecode & VM, so I think we can make better ;-) Is it unthinkable to have the basic/native string type an object?

Best regards,
Flávio

Flávio Etrusco wrote

I partly agree it's PEBKAC, but why make it easy to get wrong when you can avoid it? Isn't that the point of Pascal? Isn't that the point of AnsiStrings? Isn't that the point of strong typed languages in general?

Yes, but supporting unicode processing in a way that the user does not have to know about unicode is not possible. Even if everything were UTF-32, you could still have characters where the diacritics are separated from the characters they belong with (or should the iteration also temporarily normalize the string?).

Adding band aids that make plain indexing extremely slow to solve some problems and then still requiring people to write different code to get things working right in general is not the point of Pascal or strongly typed languages. Generally, there is a quick&easy way that is fragile and a somewhat slower and more difficult way that is correct. Having something slower that is still fragile does not belong in this picture. Especially not since indexing strings has always accessed the individual bytes/widechars, so it would also make things more confusing for people who have been using Pascal for a long time.

Furthermore it would be Delphi-incompatible (yet more confusion) and it would require "char" to become a 32 bit type since otherwise it would not be possible to represent every indexed string code point using a char (if any sort of consistency is desired, it should be possible to assign a string element to a char variable without data loss, since a basic Pascal convention that has held forever is that a string is conceptually a packed array of char).

Jonas

In our previous episode, Fl?vio Etrusco said:

> IMHO you are seeking the problems in the tools, while the problem is PEBKAC I partly agree it's PEBKAC, but why make it easy to get wrong when you can avoid it?

The point is you can't. You only keep the illusion you can marginally longer at a gigantic price.

Isn't that the point of Pascal?

No. IMHO the point has always been to find a sweet spot. Delphi is not Visual Basic. Delphi is native and fast.

Isn't that the point of AnsiStrings? Isn't that the point of strong typed languages in general?

IMHO you are just dragging in random terms here. Variable encodings were not forseen in the design, so one can't tell how to deal with it from history.

> I don't like the Java/C# way that you have to manually allocate extra > objects (stringbuilders etc) to get(performant) access to the characters though. At least we are not tied by bytecode & VM, so I think we can make better ;-)

Not everything is black/white.

Is it unthinkable to have the basic/native string type an object?

Probably not. C++ has it, but a rough idea is next to useless. The devil (and the brilliance) is in the details and borderconditions. See e.g.

http://www.freepascal.org/faq.var#extensionselect

and then specially the second half on how to make a proposal.

> IMHO you are seeking the problems in the tools, while the problem is PEBKAC I partly agree it's PEBKAC, but why make it easy to get wrong when you can avoid it? Isn't that the point of Pascal?

Many people think that Pascal is an educational (toy) language, and inefficient code will proof their opinion :-(

DoDi

Marco van de Voort wrote

No. IMHO the point has always been to find a sweet spot. Delphi is not Visual Basic. Delphi is native and fast.

Isn't this nicely provided by "new Strings" ?

If you are naive and just use them as you have been acquainted to at
ANSI times, your programs might get slow when using MyString[i];

But if you want to speed things up you can either try to replace the MyString[i];by some more decent (and complex) iterator implementation or try to set the type of the strings in question (and if necessary convert them) to an appropriate ANSI variant.

-Michael

Marco van de Voort wrote

And of course, finally, there is the matter with Delphi compatibility.

This can't even be discussed regarding Unicode programming as long as FPC does not have "new Strings".

(AFAIK there even are or have been discussions about not doing new strings at all, as they don't completely solve the problems they are supposed to solve. This finally would cancel Delphi compatibility - may this be a problem or not. )

-Michael

DaWorm wrote

But isn't it O(n^2) only when actually using unicode strings?

Allowing the compiler or library decide _if_ this is a Unicode string would require either a dedicated sting types for each encoding or "New Strings" with programmable encoding.

-Michael