Hi

I was asked the title's question but I don't really know the answer. So, compared to Tnt- unicode components for Delphi, or to the new Delphi unicode support. How is it?

I know the FPC's UnicodeString type is not compatible with Delphi's but it is close anyway, right (?)

Regards,
Juha

2010/12/31 Juha Manninen :

It is the same as the new Delphi unicode support, I think. All GUI components support unicode out of the box, it uses UTF8 encoded strings.
AFAIK Delphi uses UTF16, so in that way it is different.

Vincent

Vincent Snijders kirjoitti perjantai 31 joulukuu 2010 11:30:22:

It is the same as the new Delphi unicode support, I think. All GUI components support unicode out of the box, it uses UTF8 encoded strings. AFAIK Delphi uses UTF16, so in that way it is different.

... so UTF8 is also unicode?
I am still confused with all the encodings and string types.

Thanks.

Juha

Yes both are. And both are variable width, where UTF16 encodes characters in 16bit words and UTF8 in 8bit bytes.

For more info see
http://en.wikipedia.org/wiki/Unicode

Marc

I must ask a newbie question again. I never needed to pay attention to this because char encodings in GUIs have worked well for my purposes.

The GUI text properties have type "string" which is ansistring with the normal H+ setting.
TCaption is defined as "string", too.
Examples: TEdit.Text, TMemo.Lines[0]
What happens when I do:

  var s: string;
  ...
  s := TMemo.Lines[0];

Is it converted somehow?
The native widget's encoding is either UTF-8 or UTF-16. Is the string actually a Utf8String or Utf16String then? When do I need to pay attention to it?

Juha
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20110101/43982b20/attachment.html>

Currently there is no automatic conversion (it's planned in one of the branches of FPC). For now a String (=AnsiString) can be seen as an "array of byte". You as a developer are responsible that the string contains the correct encoding.

So in your above example the string that is stored in "s" will be UTF8 encoded, because it comes from the GUI. But if that string contains multibyte characters those characters will appear as single "one byte" characters if you access the string using [], Pos, Copy, etc.

Example (note: this is not accurate UTF8 encoding, I'm just making that up here)

TMemo.Lines[0] contains: 'h??!' ( h a-umlaut ? ! ) I now assume that an a-umlaut is encoded as "ae" (which isn't really the case, but it's for the sake of an example ^^)
s now contains: 'h a e ? !'

If you now want to access the second character of s you'd expect that you'd get the a-umlaut, but if you do s[2] you'll get an "a". And if you access the third one (s[3]) you'll get the "e" instead of "?".

You need to convert the UTF8 string to a different one, e.g. UTF16:

var
   us: UnicodeString;
begin
   us := UTF8Encode(s);
end;

Now us[2] will return the a-umlaut.

I hope this example clears that up a bit, if not: just ask more questions ;)

Regards,
Sven

I would suggest using Utf8Copy(s, 2, 1) instead. It helps to avoid conversion and works correctly even for characters that take 4 bytes in UnicodeString/WideString (i.e. 2 wide characters). Utf8Copy is declared in LCLProc unit.

Vladimir Zhirov kirjoitti lauantai 01 tammikuu 2011 22:14:32:

So the conversion is only needed if a char inside the string is accessed by index?

I understand the principle but I didn't understand how the functions UTF8Encode and UTF8Decode work. Of course I don't need to understand such details because I am not FPC developer but anyway ...

UTF8Encode returns UTF8String and the AnsiString parameter is internally typecasted to UnicodeString. How can that work?

Maybe Sven's example should use UTF8Decode. It returns UnicodeString. According to debugger both functions convert the string to uppercase and add some garbage to the beginning and end, but it may be debugger error.

Regards,
Juha

Juha,

... so UTF8 is also unicode? I am still confused with all the encodings and string types.

The best article ever on it ...
http://www.joelonsoftware.com/articles/Unicode.html

Juha Manninen wrote

So the conversion is only needed if a char inside the string is accessed by index?

No, the conversion is completely optional.
Summing up what was suggested, there are two ways to access character by index in UTF-8 string:

1. Convert it to WideString/UnicodeString and use MyWideString[Index];
2. Use Utf8Copy(MyString, Index, 1);

The limitation of the first approach is that it relies on the fact that the character fits in 2 bytes (WideChar). As a result, it works wrong for characters of some languages and some special symbols (see http://en.wikipedia.org/wiki/Supplementary_Multilingual_Plane#Supplementary_Multilingual_Plane for the list of them). So this approach does not support "true" unicode, but works in most cases.

The second approach should handle this right (provided there is no bugs).

UTF8Encode returns UTF8String and the AnsiString parameter is internally typecasted to UnicodeString. How can that work? Maybe Sven's example should use UTF8Decode.

Sure, UTF8Decode should have been used in this case.

The string is Utf8String (= AnsiString for now and in the near future), as Sven pointed out.

You need to pay attention to it when you cross the boundary between FPC's RTL/FCL and LCL. LCL uses utf-8 regardless of underlying widgetset, but RTL/FCL uses platform-specific encoding. IIRC most if not all encoding dependent routines of RTL/FCL are related to file operations and I/O.

LCL provides functions for platform-independent conversion between utf-8 and platform-specific encoding (see FileUtil unit):

1. Utf8ToSys/SysToUtf8 as general-purpose functions;
2. Set of wrappers such as FileExistsUtf8, FindFirstUtf8 etc. to make our code more readable.

If you use the LCL in your application you can also use the UTF8Copy which was mentioned by Vladimir.

Let's say it this way: if your String contains an UTF8 encoded text you should not use [] or the normal Pos, Copy, etc. functions, because they might return garbage. Use functions that can work with that encoding (either by converting the string or working directly on it).

You looked at the wrong function. I meant the one below it which has a UnicodeString as argument. And this also solves the mystery:

Casting from AnsiString to UnicodeString invokes the WideString Manager's Ansi2UnicodeMoveProc which converts the supplied AnsiString to a correct UTF16 string. Then the function which takes an UnicodeString as argument is invoked (it's an overloaded function after all) and the UTF16 string is converted to UTF8.

Maybe Sven's example should use UTF8Decode. It returns UnicodeString. According to debugger both functions convert the string to uppercase and add some garbage to the beginning and end, but it may be debugger error.

Yes, it should have used UTF8Decode. I used the wrong function. -.-

Regards,
Sven

When using the LCL this is indeed a better way.

Regards,
Sven

Sven Barth kirjoitti sunnuntai 02 tammikuu 2011 13:47:28:

Casting from AnsiString to UnicodeString invokes the WideString Manager's Ansi2UnicodeMoveProc which converts the supplied AnsiString to a correct UTF16 string.

Ok, there is some "compiler magic" here. I think I understand it now.

Regards,
Juha

2011/1/2 Vladimir Zhirov :

LCL provides functions for platform-independent conversion between utf-8 and platform-specific encoding (see FileUtil unit): 1. Utf8ToSys/SysToUtf8 as general-purpose functions;

Those only work for characters in current Ansi code page. It is better to use UTF8Decode, UTF8Encode.

Sven Barth wrote

Casting from AnsiString to UnicodeString invokes the WideString Manager's Ansi2UnicodeMoveProc which converts the supplied AnsiString to a correct UTF16 string.

Does that mean FPC and LCL always treats UnicodeString type as a UTF16 encoded type? If so, that is a rather odd "type name" then, because "unicode" is NOT just UTF16, it is also UTF8, UTF16, UTF16-LE, UTF16-BE and UTF32. A better, and more correct, type name would then have been UTF16String, just like there is a UTF8String type (though I don't really know how the latter differs from AnsiString (which is basically an array of bytes).

I just stumbled across this thread and it worries me a little since the way Delphi introduced unicode is by ambush....

What they did was to redefine the type string from AnsiString to something else unicode-ish in Delphi 2009. So all applications doing some string manipulation on data of type string broke severely. I hope I will not see the same here in FPC/Lazarus?

My concern is that I am communicating using RS232 and I use string variables to hold my messages and commands. The protocol used is defined on a byte by byte level and it will not accept some "automatic" conversion being forced on the variables.

So, will FPC stay with the current definition of string and let the developers decide what to handle as unicode strings by using a different type for these strings? For example "widestring" or "unicodestring" or the like?

Yes, UnicodeString (and WideString as well) is treated as UTF16 encoded string.

The type name might come from Delphi compatibility (tada!).

And currently UTF8String is defined as AnsiString, so there is currently no difference (which could change once the cpstrnew branch is mature enough).

Regards,
Sven

There is currently a branch of FPC where a "codepage aware" string type is developed. I don't know how far the RTL and compiler will be modified once that finalizes, but I believe that the developers will pay enough attention to backwards compatibility (as they do know as well) and that they'll listen to community input (and "fears") regarding this as well.

Regards,
Sven

cobines wrote

Those only work for characters in current Ansi code page.

Yes, on Win32. They also work properly for utf-8 characters on Linux and Mac OS X.

It is better to use UTF8Decode, UTF8Encode.

UTF8Decode and UTF8Encode do not provide any platform abstraction, they merely convert single-byte utf-8 string to double-byte utf-16/usc-2 string.

And since RTL/FCL functions on Win32 use single-byte Ansi strings, they cannot handle characters outside of current code page anyway. I do not see how UTF8Decode/UTF8Encode can help here. The only way is to use them in combination with wide version of Windows API (functions ending with "W"), but it is not portable.

To make it clear, when I called Utf8ToSys/SysToUtf8 "general purpose" I did not mean they should be used all over the code. They are only needed when calling RTL/FCL functions if there is no appropriate wrapper function like FileExistsUtf8.

And once again for someone as myself, not using Delphi, it is quite ridiculous too see the errors the Free Pascal project makes (or should that rather be the errors Delphi makes) in such cases/examples. Trying to fool all developers into thinking that "unicode" [as in UnicodeString type] only means UTF-16, because indecently that is what Microsoft uses in its Windows OS. Free Pascal is a cross-platform compiler but it seems Microsoft even dictates what Free Pascal must do. A shame really (and a slap in the face for any developer working on a non-Microsoft platform). UnicodeString should really mean any of the possible unicode encoding types.

Maybe the code-page enabled string type (cpstrnew branch) will use some more "sane" name for its string type, or redefine the standard String type to mean a code page / encoding enabled string type instead of String = AnsiString.

And currently UTF8String is defined as AnsiString, so there is currently no difference

That's what I thought. So why did they [FPC team] actually bother to create the UTF8String type then?

Maybe this is just to help developers, so you can use it to remind yourself (or other people working on your code) that a certain string contains utf8, so that you/they remember to use the conversion functions, etc. Just a guess.

Henry

This is only a temp. solution (till at the end of the world, the cpstrnew branch is ready ;)). The encoding of an unicode string will be dependent on a variable.

2011/1/2 Vladimir Zhirov :

Yes, I was thinking of Win32.

What I wanted to say was that it's not simply a matter of adding UTF8ToSys call:

FileExists(UTF8ToSys(FileName)).

It might work on Linux, but won't always work on Win32.

As you said, either do FileExistsUtf8, or directly use unicode OS API, like on Windows:
Windows.FindFirstFileW(UTF8Decode(FileName))

That is good to know. Thanks Florian.

Graeme Geldenhuys wrote

the code-page enabled string type (cpstrnew branch)

I understand that cpstrnew not necessary uses code pages, but it _can_ use it if it seems appropriate.

A conversion between multiple UTF-types and codepage based coding types is done automatically, if necessary. (Which will both prevent and introduce confusion :-) )

Correct ?

-Michael

Bo Berglund wrote

What they did was to redefine the type string from AnsiString to something else unicode-ish in Delphi 2009.

I don't have D2009+, but from what I've read I understand that it's possible to force a New Delphi String to a certain (e.g. Code page based ANSI) coding and thus "old" software working on it should just run. If a (e.g.) library function is called it should automatically be converted if the callee needs this.

-Michael

Yes, thanks for clarification.

Regards,
Sven

Graeme Geldenhuys wrote

> The type name might come from Delphi compatibility (tada!). And once again for someone as myself, not using Delphi, it is quite ridiculous too see the errors the Free Pascal project makes

It doesn't. You just assume nobody thought about it before you, and that it is an error. Moreover, you also assume that everybody has the same sensitivities as you about certain topics, and again that is wrong.

(or should that rather be the errors Delphi makes) in such cases/examples. Trying to fool all developers into thinking that "unicode" [as in UnicodeString type] only means UTF-16, because indecently that is what Microsoft uses in its Windows OS.

We would never try to "fool" anybody to think such a thing. Please stop putting words in our mouths.

Maybe the code-page enabled string type (cpstrnew branch) will use some more "sane" name for its string type, or redefine the standard String type to mean a code page / encoding enabled string type instead of String = AnsiString.

This is all undecided. I lean towards splitting operating system targets into a utf8 and a utf16 one for most platforms(*), since nobody will ever agree on one encoding. Not even per platform.

(*) and a legacy "ansi" one if need be.

It's an alias for literal programming purposes. You can see from the typename what a procedure expects, and it goes into the documentation

Florian Kl?mpfl wrote

This is only a temp. solution (till at the end of the world, the cpstrnew branch is ready ;)). The encoding of an unicode string will be dependent on a variable.

The encoding of an 1-byte encoding based string.

On 02/12/2011 06:49 PM, Marco van de Voort wrote:And currently UTF8String is defined as AnsiString, so there is currently no

>> difference > That's what I thought. So why did they [FPC team] actually bother to > create the UTF8String type then? It's an alias for literal programming purposes.

While I disagree on much of his wording I so agree with the OP, that with having ANSIString and UTF8String an alias for exactly the same - only to allow for keeping in mind what coding the user manually introduces - is very confusing. These names strongly suggest that a statement myANSIString := MYUTF8String either is detected as illegal or forces appropriate conversion.

-Michael

Why do we need "targets"?

Wouldn't it be better to simply duplicate all string functions for utf8 and utf16 and ansi if necessary?

That was my idea from the start in case the new string was merged. For example:

CompareText
UTF8CompareText
UTF16CompareText

The versions with a fixed encoding could refer to a generic unicode version with undefined encoding.

Op 2011-02-14 13:16, Felipe Monteiro de Carvalho het geskryf:

Why do we need "targets"?

I would imagine that such a new string type (possibly UnicodeString) would default to the encoding used per platform by default.

eg:
* a Linux UnicodeString will default to UnicodeString(utf8)
* a Windows UnicodeString will default to UnicodeString(utf16) etc..

But that doesn't limit the developer, because the developer could simply define a new string type and use that instead.

eg:
  // alias types with their encodings set to something specific
  UTF8String = UnicodeString(utf8);
  UTF16String = UnicodeString(utf16);
  CP850String = UnicodeString(cp850);
  etc...

Wouldn't it be better to simply duplicate all string functions for utf8 and utf16 and ansi if necessary?

Why? CompareText and all other such functions should simply take a UnicodeString types as parameter (in addition to the already existing WideString, AnsiString and ShortString versions). The Unicode enabled version of CompareText will then query the encodings used and do a automatic conversion if needed, then do the comparison and return the result.

eg:

var
  u8: UTF8String;
  u16: UTF16String;
  s850: CP850String;
  r: integer;
begin
  u8 := ...;
  u16 := ...;
  s850 := ...;
  r := CompareText(u8, u16);
  ...
  r := CompareText(u8, s850);
  ...
end;

CompareText UTF8CompareText UTF16CompareText

So for every possible encoding and code page you want to make a new function? That doesn't sound like a good plan to me. The encoding information is inside the string type, so use it to do automatic conversions.

Regards,
- Graeme -

Graeme Geldenhuys wrote

But that doesn't limit the developer, because the developer could simply define a new string type and use that instead.

Maybe I used a bad example, but anyway, var parameters need to be exact

So for every possible encoding and code page you want to make a new function?

Of course not! Only the important ones. For me this means only UTF-8, but I supposed some people might want UTF-16 too.

Op 2011-02-14 14:15, Felipe Monteiro de Carvalho het geskryf:

Maybe I used a bad example, but anyway, var parameters need to be exact

"alias types" are not really new types, so will not affect var parameters. So in my previous example, UTF16String will still be a UnicodeString. The only difference will be that UTF16String has its internal encoding bit set to UTF-16.

Here is a test program using latest FPC 2.4.3 showing that TMyString = String - the compiler sees no difference between the two types. So the same should be valid for UnicodeString vs UnicodeString(...) that have their encoding bit set to something other than the platform default.

---8<-----------8<-----------8<-----------8<-----------8<-----------
program project1;
{$mode objfpc}{$H+}

uses
Classes

type
  TMyText = String;
procedure TestMe(var AText: string);
begin
  writeln(AText);
  AText := 'Hello ' + AText;
end;
var
  s: TMyText;
begin
  s := 'Graeme';
  TestMe(s);
  writeln(s);
end.
---8<-----------8<-----------8<-----------8<-----------8<-----------

Again I don't see why that is needed. You might only require UTF-8, but somebody else wants UTF-16, and somebody else wants CP850 versions etc etc.. Where does it end?

A simple function like:

  function CompareText(const S1: UnicodeString;
                       const S2: UnicodeString): integer;

should be able to work with any UnicodeString parameters (including alias types that have different encoding bits set). So it should work fine for your UTF-8 text, and somebody else's UTF-16 etc. text.

Regards,
- Graeme -

Graeme Geldenhuys wrote

Here is a test program using latest FPC 2.4.3 showing that TMyString = String - the compiler sees no difference between the two types. So the same should be valid for UnicodeString vs UnicodeString(...) that have their encoding bit set to something other than the platform default.

It is pure speculation to assume that this behavior will remain valid.

Op 2011-02-14 16:00, Felipe Monteiro de Carvalho het geskryf:

It is pure speculation to assume that this behavior will remain valid.

Well, it seems obvious to me that it should [and would]. From my previous examples, the alias type is still a UnicodeString type. So why wouldn't methods/procedures/functions that take UnicodeString types as parameters work?

If the FPC implementation of such a UnicodeString (and the baviour I described) cannot handle such cases, then I would have to say that the FPC implementation would be seriously crippled. Lets hope it doesn't go that route.

Not that I care (because Delphi doesn't do everything perfect), but how does Delphi 2010 handle such alias types - especially when passed as parameters (var and const)?

Regards,
- Graeme -

Do you mean that the compiler should convert the strings as needed in the background (as between different integer types and/or floats) so that you can call ListBox1.Items.Add(x) with x beeing UTF8string or UTF16string or...? I am not sure whether this is a good idea because the programmer no longer knows how many conversions take place and therefore cannot judge the performance impact anymore. On the other hand it would make the life easier for beginners.

Hello Lazarus-List,

Monday, February 14, 2011, 7:03:31 PM, you wrote:

JH> Do you mean that the compiler should convert the strings as needed in JH> the background (as between different integer types and/or floats) so JH> that you can call ListBox1.Items.Add(x) with x beeing UTF8string or JH> UTF16string or...? I am not sure whether this is a good idea because the JH> programmer no longer knows how many conversions take place and therefore JH> cannot judge the performance impact anymore. On the other hand it would JH> make the life easier for beginners.

I'm unable to see the "great" problems with "UnicodeString". The conversions should be the minimun needed, and they will be. Problem would be in the RTL, but not at user level. You say that the programmer will not know how many conversions take place, that's right, but I think they are garanteed to be the minimum except in some corner cases like "CompareText(UTF8String,WideString)" as one of both must be converted, but whichever one, could be a fixed situation or platform dependent, I do not know.

Many people are concerned about "speed" due hidden conversions, so can anybody tell me why ? Maybe I'm blind and I can not see something that is absolutly a problem (except some pieces of RTL).

Yes, since for example Linux allows non valid UTF-8 as file names, so any auto conversion of file names to UTF-16 is an error.

For instance searching needs a lot of compares. Comparing two strings normally fails on the very first characters. An auto conversion will always convert the whole string including allocating and releasing memory, easily slowing down the conversion by an order of magnitude.

Mattias

Jos? Mejuto wrote

Many people are concerned about "speed" due hidden conversions, so can anybody tell me why ? Maybe I'm blind and I can not see something that is absolutly a problem (except some pieces of RTL).

Typical example is you mix two codebases which have a different opinion about the string type. Then for every transition between those two codebases you have a fair chance that a conversion is needed. It is throughout possible that if you do an Tstringlist.indexof() that you do as many conversions as elements in the stringlist (if your passed stringtype is different from the tstringlist one).

A minimum conversion scheme uses one type internally and only converts at the bounderies of the system. But even that has worst cases, e.g. like operating on large database exports in a different format than native.

But at least that kind of problems is fairly localised. It is harder if it is everywhere in the codebase, like the former example.

Hello Lazarus-List,

you wrote

> I'm unable to see the "great" problems with "UnicodeString". The > conversions should be the minimun needed, and they will be. Problem > would be in the RTL, but not at user level.

MG> Yes, since for example Linux allows non valid UTF-8 as file names, MG> so any auto conversion of file names to UTF-16 is an error.

Hmmm... To me it looks like a Linux "problem"/"bug" for that kind of access it is logical to me to use low level APIs. OK, that way you can not access those files ? yes, but also in Windows there are similar problems, some files can not be accessed using regular APIs and some tricks must be used.

> Many people are concerned about "speed" due hidden conversions, so can > anybody tell me why ? Maybe I'm blind and I can not see something that > is absolutly a problem (except some pieces of RTL).

MG> For instance searching needs a lot of compares. Comparing two MG> strings normally fails on the very first characters. An auto conversion MG> will always convert the whole string including allocating and releasing MG> memory, easily slowing down the conversion by an order of magnitude.

This are the "some corner cases" which can not be handled in the usual conversion, operation, conversion back, but I think there are not much cases like this. Of course, there are cases like a TStringList with 100000 items in UTF16 and perform a search using an UTF8String, so or a conversion request to the stringlist (convert all elements in one go) or you must use your unicodestring using default unicode format for the platform.

I would like to see an example of such problem (snippet) which could be a headache, but maybe in the fpc mailing lists ?

Hello Lazarus-List,

you wrote

> Many people are concerned about "speed" due hidden conversions, so can > anybody tell me why ? Maybe I'm blind and I can not see something that > is absolutly a problem (except some pieces of RTL).

MvdV> Typical example is you mix two codebases which have a different opinion MvdV> about the string type. Then for every transition between those two codebases MvdV> you have a fair chance that a conversion is needed. It is throughout MvdV> possible that if you do an Tstringlist.indexof() that you do as many MvdV> conversions as elements in the stringlist (if your passed stringtype is MvdV> different from the tstringlist one).

But you are in the same trouble if you use any other approach, or you use your data in the same unicode format as the other codebase or you update the codebase to use your "new" unicode format.

There isn't a solution for such situation. I'm currently working with GeckoPort which uses WideString in every place and other special strings. I know that conversions must happend so when I need to scan for a string first convert my data to the "native" format and them perform the scan.

I think expecting a TStringList in ansi encode to work transparently and optimal using unicodestrings is just a dream, programmers should update their codebase, but at least only for speed (reduce autoconversions) and do not need to decide constantly which encoding format is needed to call this or that function.

Using a different RTL for each encoding is even worst IMHO. But this is just a simple opinion.

Op 2011-02-14 20:03, J?rgen Hestermann het geskryf:

Do you mean that the compiler should convert the strings as needed in the background (as between different integer types and/or floats) so that you can call ListBox1.Items.Add(x) with x beeing UTF8string or UTF16string or...?

Yes, but in reality how often would such conversions happen? TStringList (used inside a TListBox) would use UnicodeString types. The encoding of that type would default to whatever platform you compiled on. ie: under Linux it would default to UTF-8, and under Windows it would default to UTF-16

So if you define a new string of UnicodeString type in code, it would automatically match the encoding type of the TStringList, so when you add a string item to the listbox, no conversion would be needed. This would probably be the case 99% of the time.

The developer would physically have to create a new string with and manually set an encoding different to the platform default, before a auto-conversion would be required.

In day-to-day work and in most cases auto-conversions will be kept to a minimum - automatically.

Regards,
- Graeme -

Op 2011-02-14 21:13, Jos? Mejuto het geskryf:

right, but I think they are garanteed to be the minimum except in some corner cases like "CompareText(UTF8String,WideString)" as one of both must be converted, but whichever one,

Exactly, auto-conversion would be kept to a minimum without any user intervention. In the above example CompareText() could quickly check each strings encoding, and if one matches the platform default, that string doesn't need a conversion.

So in your example above, if you ran that under Linux, only the second parameter would require an encoding conversion.

Regards,
- Graeme -

You realize the problems, that may result from the different char type of such an target-specific string type?

IMO such strings should be encapsulated in a (polymorphic) class, that only allows for char-type independent operations.

DoDi

Graeme Geldenhuys wrote

Yes, but in reality how often would such conversions happen?

Does that really matter? If I don't now beforehand that I can control such issues with some strategic choices, I won't start using it at all.

Graeme Geldenhuys wrote

Not that I care (because Delphi doesn't do everything perfect), but how does Delphi 2010 handle such alias types - especially when passed as parameters (var and const)?

In Delphi such strings with codepage need to be defined with "type" and thus they are different types (not compatible regarding "var").

The following example fails to compile at the call of Test.

====source begin====
program strvartest;
{$APPTYPE CONSOLE}
uses
   SysUtils;
type
   CyrillicString = type AnsiString(1251);
   LatinString = type AnsiString(1252);
procedure Test(var aStr: CyrillicString);
begin
end;
var
   s: LatinString;
begin
   s := 'Foo';
   Test(s);
end.
====source end====

Regards,
Sven

Hello Lazarus-List,

Tuesday, February 15, 2011, 4:27:40 PM, you wrote:

MvdV> There is a solution that you can take FPC and Lazarus libraries out of the MvdV> equation by splitting each target into an one and two byte encoding MvdV> platform. That should fix the bulk of the problem. [...]
MvdV> Update to what? The point is that neither of both most used encodings (UTF8/16) is MvdV> is going away anything soon, and the split is right through platforms.

So you are talking about a "platform" about the encoding, but in any operative system platform, this means you can choose RTL Linux 32 bits WideString or RTL Linux 32 bits UTF8, do not ?

> Using a different RTL for each encoding is even worst IMHO. But this > is just a simple opinion.

MvdV> Let's here the argumentation for that then.

Well, maybe I misunderstood the platform split...

That's sounds like yet another approach. So up to now I see 3 models how strings could be handled:

--------------
1.) Full programmer responsibillity (current model): The programmer is fully responsible for (and has full control about) the strings used in his program. Libraries use mostly UTF8 with some exception like API related libraries. The programmer needs to know about the used string types in all used libraries and if conversions are needed he has to initiate them manually.

Pros:
The programmer knows exactly what happens under the hood so he can judge performance and incompatibilities (at least he should).

Cons:
Much harder to code because he *needs* to know about all the details of string encodings in different libraries. When strings are saved to files they would be compatible accross OS platforms because the programmer can use the same type in all cases so files can be exchanged accross them.

--------------
2.) A generic "UnicodeString" is mapped to different real sting types "under the hood". So the used string type in programs (and libraries like LCL) differs from platform to platform. The programmer does not even know what type is used. If a conversion is still needed for special routines it would be done automatically in the background without the programmer having to know about it. Other real string types like UTF8string are available but it's not encouraged to use them.

Pros:
Easy to code. In general, deeper knowledge about string encodings and their storage is not needed. String conversions are seldom needed.

Cons:
When non-unicode strings are used on a platform (i.e. ANSI on Windows) but unicode is required by the program it becomes clumsy because then the programmer has to use it's own (unicode) string type and then conversion are needed for all library and other functions. When strings are saved to files they may differ on different platforms so files cannot be exchanged accross them. All libraries have to be rewritten to handle different string types.

-------------
3.) A middle course: UTF8 is chosen to be the main string type which should be used whenever possible (within LCL and other libraries) and also programmers are encouraged to use it so that conversions become (more and more) unlikely. When using interfaces with different string trypes (like OS APIs) there would be an automatic conversion in the background.

Pros:
Easy to code. No doubt about the used string type and its capabilities for the programmer, it's always UTF8 for him. When strings are saved to disk they are all UTF8 on all platforms so files can be exchanged between Linux and Windows (and others).

Cons:
Because LCL and other libraries use UTF8 there could be a performace impact when compiling to non-UTF8 OS (where API's use ANSI or other UTF16 or whatever).

I would prefer model 3.)

Op 2011-02-15 16:32, Hans-Peter Diettrich het geskryf:

You realize the problems, that may result from the different char type of such an target-specific string type?

Please do share your thoughts...

I must add, that I would be very surprised if Embarcadero doesn't use native encoded string types for the "unicode string" support in the upcoming Delphi under Windows (UTF-16), Linux (UTF-8), Mac (UTF-8) etc.. I'm not 100% sure about the default Mac encoding, but seeing that it comes from FreeBSD, I would guess UTF-8 there too.

As for saving text to file...It is universally known to use UTF-8 in such cases, because UTF-8 is the perfect encoding for streaming. Hence the W3C also said all HTML, XML etc should be preferably in UTF-8.

Regards,
- Graeme -

Op 2011-02-16 09:12, J?rgen Hestermann het geskryf:

That's sounds like yet another approach. So up to now I see 3 models how strings could be handled:

UnicodeString should become the default. I would even go as far as recommending that String = UnicodeString in a newer FPC version.

UTF8String is not a special type, it is simply a UnicodeString that has it's encoding bit set to UTF-8. UTF8String is still a UnicodeString as far as the compiler is concerned. It's an "alias type".

Just like we now use String everywhere, in future we could use UnicodeString everywhere and don't need to worry about what encoding we are using. This should even be the case when we talk to OS API's or external libraries. By default libraries under Windows use UTF-16. Your UnicodeString under Windows will already be in UTF-16 - thus no conversion needed. Same for Linux. By default most libraries use UTF-8, and UnicodeString will automatically be in UTF-8 encoding anyway - thus again no conversion needed.

All RTL functions which use UnicodeString will obviously be the correct encoding on each platform too. So again by default, simply using UnicodeString will not require any encoding conversions.

Will will work for 99% all all applications. Only when the developer wants to do something specific with a specific encoding, that developer can simply define a new string type as follows:
eg:

   type
     MyCP850StringType = UnicodeString(cp850);

That developer can then use that string knowing the encoding is CP850. But this would be very seldom needed.

As for the LCL or any libraries. They could simply use UnicodeString everywhere (thus my recommendation that String = UnicodeString in the future so our code wouldn't need much changing), and it will work just fine under all platforms, because UnicodeString will default to each platforms default encoding.

Regards,
- Graeme -

Op 2011-02-15 17:38, Sven Barth het geskryf:

In Delphi such strings with codepage need to be defined with "type" and thus they are different types (not compatible regarding "var").

Lets hope FPC doesn't take that stupid Delphi idea to implementation. A UnicodeString with a different encoding bit set is still a UnicodeString type.

Thanks though for testing and letting us know what Delphi does.

Regards,
- Graeme -

First the cpstrnew branch needs to be revived at all...

Thanks though for testing and letting us know what Delphi does.

You're welcome. After all I have bought that Delphi XE Starter for exactly such tests :D

Regards,
Sven

Graeme Geldenhuys ?????:

It's not as stupid as it may seem. When a string being passed to a function is empty, it is actually a nil pointer, and its encoding cannot be for value parameters, but for var/out parameters the caller must know what data to assign. Making each encoding a distinct type and having compiler do the checks is probably the only way out.

Sergei

I personally would prefer the code page aware string to be a new string type (e.g. CodePageString) instead of abusing UnicodeString here (it should be kept as a 2-byte string for backwards compatibility).

But I wouldn't mind if then String is by default an alias to this CodePageString type (or only in certain modes or with certain switches similar to {$H+/-} as this would decrease the changes/overview needed for old code).

Regards,
Sven

Op 2011-02-16 10:28, Sven Barth het geskryf:

I personally would prefer the code page aware string to be a new string type (e.g. CodePageString) instead of abusing UnicodeString here (it

I think FPC (and Delphi) already have enough string types, so I would presonally vote for NOT adding yet another string type. I also think UnicodeString = UTF16 (as in the case of Delphi) is just wrong - Delphi should then have rather introduced a new type called UTF16String. The Unicode Standards defined 8 or so valid encodings described as Unicode, so how can Embarcadero make the assumption that Unicode = UTF-16 only.

But I wouldn't mind if then String is by default an alias to this CodePageString type (or only in certain modes or with certain switches similar to {$H+/-} as this would decrease the changes/overview needed for old code).

+1
Lets not make the same mistake as Delphi. And I do realize it will not magically work for every application, because some code might assume Char = 1byte, so such code would still have to be fixed. But it would be much less effort that renaming ALL String references in our code to UnicodeString.

Regards,
- Graeme -

Op 2011-02-16 10:21, Sergei Gorelkin het geskryf:

When a string being passed to a function is empty, it is actually a nil pointer, and its encoding cannot be new RTL function like...

  function QueryEncoding(S1: UnicodeString): StringEncodingType;

...could easily solve this problem. Plus, even though the string is empty, the internal type structure information for that string variable had to be setup, so the information about the encoding should exist.

Fine for value parameters, but for var/out parameters the caller must know what data to assign.

Again, if you simply use UnicodeString, the var parameter and the local UnicodeString variable inside the function/procedure should have the same encoding by default - so no conversion would be needed.

And even if they were different, the compiler could easily to the auto-conversion to match the var parameter (as I described in my first paragraph above).

Regards,
- Graeme -

Very valid point !

So we could have an auto-converting compare function in the RTL, that (when finding different encodings) does not do a complete conversion first and comparing afterwards but converts each multi-byte character of both strings to the 32 bit Unicode character and compares thee two DWords, and stops after the first difference.

-Michael

J?rgen Hestermann wrote

1.) Full programmer responsibillity (current model):

This not really matches the current model. For a decent implementation of this, there need to be different string types for the different encodings that are handled as incompatible. Assigning does not do auto-conversion but produces a compile time error.

-Michael

Op 2011-02-14 21:29, Mattias Gaertner het geskryf:

Yes, since for example Linux allows non valid UTF-8 as file names, so any auto conversion of file names to UTF-16 is an error.

I have never noticed that, plus that seems more like a bug in Linux's filesystem (whichever one you use - I use JFS only, and haven't noticed such issues). So maybe file a bug report with the Linux project instead or working around the issue forever.

For instance searching needs a lot of compares. Comparing two strings normally fails on the very first characters. An auto conversion will always convert the whole string including allocating and releasing

You are missing the point. If full Unicode support exists in FPC and Lazarus - say via the UnicodeString string type, then the encoding of the search string and the encoding of the text inside the editor will be the same - thus no conversion needed.

The only exception here will be a "Find in files" under Windows, where Windows uses UTF-16, but normally files are stored as UTF-8. Though Mac, Linux and *BSD will not be affected by this as they use UTF-8 pretty much everywhere.

Regards,
- Graeme -

Most people have been sure, in the past, that they use a SBCS, where every character on screen is a char in memory. And consequently they use indexed access to the chars in an string, and for...to loops. The same procedures may work for UTF-16, where also most characters correspond to one widechar, but this code will fail miserably on an UTF-8 platform, where every single (visual) character can consist of any number of chars, with no compiler warnings.

That's one reason why I think that it should be disallowed, in portable code, to use any char type together with strings. Such restrictions cannot be applied to specific string types, unless these are strictly different from the old ShortStrings and AnsiStrings.

It would be nice, of course, for old style code, to have strings with a known (app specific) and *immutable* encoding. String handling with such a target independent string type would work properly on any target, as long as the contents match the coder's expectations. In Cobol such strings were for "usage computational", in constrast to "usage display" with target specific encoding.

AFAIK the UnicodeString allows for any dynamic encoding, be SBCS, MBCS or UTF-8/16. The element (char) size and encoding have become part of every Unicode string descriptor.

As for saving text to file...It is universally known to use UTF-8 in such cases, because UTF-8 is the perfect encoding for streaming. Hence the W3C also said all HTML, XML etc should be preferably in UTF-8.

Right, UTF-8 is the recommended external representation of text. No byte order problems, no conversion losses...

DoDi

I'm pretty sure that such strings will be widely used, by people that prefer to use string indexing with a fixed character size.

In any other model there is no use of any char type (except perhaps UTF32char), instead only (sub)strings should be used in code. IMO it was a wise decision that Pascal literals make no difference between char and string literals.

DoDi

This is wrong.

The following compiles:

type
   UTF8String = type AnsiString(65001);

but the following does not:

type
   UTF8String = type UnicodeString(65001); // ';' expected, but '(' found

Tested using Delphi XE (65001 is the codepage for UTF-8 on Windows).

Regards,
Sven

Hans-Peter Diettrich wrote

I'm pretty sure that such strings will be widely used, by people that prefer to use string indexing with a fixed character size.

Yep. Bus as discussed here earlier, Length() - that needs to be done with the same paradigm as mystr[i] - is a problem. When using e.g. stream-I/O people will want it to be the byte count, when doing a loop along a string, they want it to be the unicode-character count.

-Michael

Op 2011-02-16 12:52, Hans-Peter Diettrich het geskryf:

Most people have been sure, in the past, that they use a SBCS, where every character on screen is a char in memory. And consequently they use indexed access to the chars in an string, and for...to loops.

Yes, and that code accesses string characters 99% of the times in a sequential manner, be that left-to-right (or other way round), hardly ever random. So to overcome this "supposedly" limitation, one simply needs to create a StringIterator (which I already have in my projects where character extraction is needed) will work just fine. So I don't see this as a problem at all.

The same procedures may work for UTF-16,

No, character indexes will not work for UTF-16 either. Not ALL Unicode Characters can fit into a 2-bytes. Also what about screen characters that are made up of multiple code-points (combining diacritics etc)? eg:
U+0041 (A) + U+030A (?) = A?

Depending on how that string is normalized, doing a MyString[1] might only return 'A' and not A? as you would have expected.

one widechar, but this code will fail miserably on an UTF-8 platform,

And so too for UTF-16 - as I have just shown. If you want to use UTF-16 like that (just because *most* of the Unicode code-points can fit into 2-bytes), then it is no better that UCS-2.

Regards,
- Graeme -

Op 2011-02-16 14:25, Michael Schnell het geskryf:

Yep. Bus as discussed here earlier, Length() - that needs to be done with the same paradigm as mystr[i] - is a problem. When using e.g. stream-I/O people will want it to be the byte count, when doing a loop along a string, they want it to be the unicode-character count.

This is why Length() can stay as is - returning a byte count. We can then introduce a new StringIterator class or something that can return one unicode-character at a time - for loop purposes. I have already create such iterators for myself going through strings or any tyes of list classes found in the RTL. Using such iterators instead of dump for..loops, I can now move forward, backwards, skip elements, filter elements etc... Such Iterators are very powerful and flexible.

eg:

var
  itr: IStringIterator;
  c: UnicodeChar;  // or something that can store at least 4 bytes
begin
  itr := TStringHelper.GetUnicodeIterator(MyString);
  while itr.HasNext do
  begin
    i := itr.Next
    ... now do something with the Unicode character stored in i
  end;
end;

This also reduces confusion. The developer knows what is being returned simply by looking at the code, unlike the confusion with Length() [is it number of chars or number of bytes].

Regards,
- Graeme -

Do we already have / plan a dedicated iterator loop syntax for this

Delphi Prism would go

   for each i in itrdo
   begin
      ... now do something with the Unicode character stored in i
   end;

for this

-Michael

Take a look at the tforin*.pp tests here:
http://svn.freepascal.org/cgi-bin/viewvc.cgi/trunk/tests/test/

But basically the syntax is

for enumeratedvar in enumerator do
   // something

It's the Delphi compatible variant of the Prism syntax. Of course you also might need to define a suitable enumerator (e.g. for enumerating a UTF-8 string).

Regards,
Sven

Op 2011-02-16 15:59, Michael Schnell het geskryf:

Do we already have / plan a dedicated iterator loop syntax for this

A similar syntax is already supported in FPC 2.4.2. See the Language Reference documentation, section 10.2.5 (pg 115 in the ref.pdf).

I personally prefer the Iterator interface though, because it supports bi-directional iteration, filtered iteration, skips, resets etc. Basically a lot more functional for me, than the FPC's for..in..do syntax.

Regards,
- Graeme -

Sven Barth wrote

It's the Delphi compatible variant of the Prism syntax.

Sounds great.
-Michael

Sven Barth ?????:

You are right. Likewise, type AnsiString(1200) can be declared, but it won't work (1200 is utf-16 codepage).
In Delphi, UnicodeString is a very separate type, something close to the current FPC design. It has BytesPerChar and Encoding attributes, but they are fixed to 2 and 1200 respectively and their purpose is unclear (to consume memory? to make it look like it's compatible with AnsiString?)

This has a lot of consequences in RTL, e.g. passing them in 'array of const' uses type field vtUnicodeString, not vtAnsiString; assigning to Variant uses varUString; a published property of type UnicodeString has typekind=tkUString and so on. Part of these are already implemented in FPC RTL due to compatibility reasons.

I'm afraid that due to this "compatibility" we're doomed to clone the Delphi implementation whatever crappy it is :(

Sergei

How do you duplicate every usage of "string" in the entire Lazarus tree?

I don't see why, and don't feel responsible for other people's speculation.

There is no unicodestring(...). Ansistring(...) and unicodestring are treated as different types on a Pascal level.

There is a catch-all rawbytestring for deep RTL work, but you really use that as string type. See earlier discussions.

Please test again, with something like
   MyUnicodeString.Encoding := 65001;

DoDi

I'm afraid that due to this "compatibility" we're doomed to clone the Delphi implementation whatever crappy it is :(

I'm not sure whether we have to be compatible with Delphi>7, and whether the Delphi Unicode implementation is crappy. At least the choosen implementation allowed for quite a smooth transition, from Ansi to Unicode.

DoDi

The iteration costs time, so that many users will insist in using "fast" SBCS access. No doubt that proper Unicode coding will require iterators, unless Pos can return an valid index immediately.

When an Unicode string contains the same characters as an Ansi string, then all these BMP characters fit into one widechar.

Also what about screen characters that are made up of multiple code-points (combining diacritics etc)? eg: U+0041 (A) + U+030A (?) = A?

These are special Unicode issues, that never have been an issue with Ansi strings, and should not be in Unicode - as long as dealing with the same content as before. Again the Cobol distinction applies: the user does not have to bother with the internals of strings of "usage display"
- they only are read, written and displayed, and what else can be made in portable "high-level" string handling.

Dealing with *all* the Unicode quirks IMO is beyond "usual" coding, it will be reserved to specialized text processing components or applications.

Perhaps you understand better now, why I suggest an string type with an immutable application defined codepage, for "traditional" coding? This would be "usage computational", where the known rules for low-level string handling apply, just as used with AnsiStrings.

Depending on how that string is normalized, doing a MyString[1] might only return 'A' and not A? as you would have expected.

No difference to the current encoding, isn't it? You should not assume that such non-canonical Unicode is or has ever been translated into a single Ansi char, by automatic conversion.

*Most* users will be happy with the BMP. Those using codepages outside the BMP had to live with all that stuff, since ever.

IMO the most important thing about Unicode is to teach the users the difference between low and high level string handling. Indexed access to characters is a low level operation, that should not be used in Unicode-aware applications without specific knowledge. Low level string handling requires the exact knowledge about the encoding of an string, with eventual branches for the *expected* encodings and char types. High level string handling is not character-based, so that your objections do not apply.

DoDi

That's why such loops should be disallowed with Unicode strings, as kind of low level string handling. For-Each loops may be acceptable as high level string handling, but with what type of the loop variable???

DoDi

Marco van de Voort schrieb:

Can you please make your mailer preserve *spaces* in the subject?

DoDi

Depends on the type of i. Neither Char nor WideChar are capable of holding every Unicode codepoint (32 bit).

DoDi

Indexed access to characters is a low level operation, that should

not be used in Unicode-aware applications without specific knowledge.

I often search for substrings, delete them from the string, insert other strings at certain places, etc.
How can you do all this without knowledge of the internal structure of the string?

Op 2011-02-16 23:15, Marco van de Voort het geskryf:

There is no unicodestring(...). Ansistring(...) and

I know there currently isn't, but are you also saying that we can't extend UnicodeString to support UnicodeString(?) syntax? To me, AnsiString(?) [like is done in Delphi now], makes even less sense.

On a side note:
DON'T bother replying if your answers is going to be the lame excuse "it's Delphi compatible". Because if I hear that one more time, I'm really go nuts! :) Just because Delphi is brain-dead, doesn't mean FPC must also be brain-dead.

I guess the most logical would be String(?) syntax - upgrading the String type to be encoding aware, similar to what was done when String was a shortstring and then became a longstring later. Making the String type encoding-aware would be a natural progression in the language. This functionality could maybe even be trigger by a new compiler directive, similar to what {$H+} does, but could be default enabled if compiler mode objfpc is used. This means FPC would be one-up to Delphi, where we can toggle the unicode functionality, and Delphi can't.

Regards,
- Graeme -

Op 2011-02-16 20:25, Sergei Gorelkin het geskryf:

In Delphi, UnicodeString is a very separate type, something close to the current FPC design. It has BytesPerChar and Encoding attributes, but they are fixed to 2 and 1200 respectively and their purpose is unclear (to consume memory?

Now that to me is just stupid!

I'm afraid that due to this "compatibility" we're doomed to clone the Delphi implementation whatever crappy it is :(

If so, then I wouldn't waist any more of my time on working on the cpstrnew branch (which I have been doing quietly on my own). If I'm going to invest time in something, I would hope to make it as best I can, and intuitive at the same time - fixing mistake found in other products/languages. Simply cloning a rubbish design for the sake of cloning, is not why I got into programming in the first place!

Hans-Peter also raised a valid point. FPC had a goal to be Delphi 7 compatible, so that should leave use open to me inventive and learn from post-Delphi 7 mistake, and make FPC even better than Delphi 7+. If FPC just wants to be a Delphi clone, then why use FPC - just switch to the "real" thing [Delphi]. They'll have cross-platform support soon.

Regards,
- Graeme -

Op 2011-02-17 00:58, Hans-Peter Diettrich het geskryf:

What's the type of the loop variable???

Any time that can store 4-bytes. Be that a string, dynamic array or a custom object/class type.

The iteration costs time, so that many users will insist in using "fast" SBCS access.

That would also insist they can't use Unicode text - which is the whole point of this conversation.

No doubt that proper Unicode coding will require iterators, unless Pos can return an valid index immediately.

There are many ways of implementing fast unicode Pos(), Length(), Copy() etc... I have read numerous implementations - some fast, some not.

When an Unicode string contains the same characters as an Ansi string, then all these BMP characters fit into one widechar.

Yes, but still, not all Unicode characters fit into a widechar (2-bytes). Most [if not all - I'm not sure here] spoken languages fit into the BMP, but that might not always be the case. Maybe some day you want to translate all your text into Klingon or Goa'uld or whatever alien race visits our planet. Being prepared and supporting the full Unicode is the best option at the moment.

These are special Unicode issues, that never have been an issue with Ansi strings, and should not be in Unicode - as long as dealing with the same content as before.

My example might not have been extensive enough to get the point across. The point being that what you see on screen as a "character" might be a combination of code-points. This is not a "issue of Unicode", but a functionality of Unicode - hence the reason there are stacks of information about various Unicode normalizations too. eg: Mac's keep them separated, where under Linux I believe such combined diacritics are replaced with a single code-point that can represent the same information [if it exists].

- they only are read, written and displayed, and what else can be made in portable "high-level" string handling.

Well, for any string handling in your application, you need to know the difference between what is perceived as a Unicode "character" on the screen, and the various ways such a "character" can be presented in a language structure. There is no way around this, unless FPC defines that such Unicode strings are always stored in some specific normalized manner.

Dealing with *all* the Unicode quirks IMO is beyond "usual" coding, it will be reserved to specialized text processing components or applications.

I'm not arguing that point.

*Most* users will be happy with the BMP. Those using codepages outside the BMP had to live with all that stuff, since ever.

Then you should call it UCS-2 support, and not Unicode support. We are talking about implementing Unicode support here.

Regards,
- Graeme -

I'm also trying to fix mistakes Delphi introduced - at least in mode objfpc.
Take my class helper implementation (WIP) for example: Delphi allows "message", "virtual", "override", "published", etc, but all those specifiers are ignored in the end (they just can't work with the concept of class helpers). So I've simply forbidden them in mode objfpc.

You need to have a Windows to even compile for other platforms... No, thank you (and the IDE might not work in Wine, because it's stuffed with .NET things).

Regards,
Sven

[OT]

Graeme Geldenhuys wrote

Maybe some day you want to translate all your text into Klingon or Goa'uld or whatever alien race visits our planet. Being prepared and supporting the full Unicode is the best option at the moment.

If I wouldn't be at work know, I would have laughed aloud. Thanks for this comment :D

Regards,
Sven (a Star Trek and Stargate fan)

[/OT]

Hello Lazarus-List,

Thursday, February 17, 2011, 8:21:07 AM, you wrote:

GG> Well, for any string handling in your application, you need to know the GG> difference between what is perceived as a Unicode "character" on the GG> screen, and the various ways such a "character" can be presented in a GG> language structure. There is no way around this, unless FPC defines that GG> such Unicode strings are always stored in some specific normalized manner.

I think FPC should handle them as codepoints, this means they are normalized. If the string is not normalized is responsability of the user to normalize it via the supported functions or create new functions to support it unnormalized.

In other words strings support do not deal with their representation.

Marco van de Voort wrote

How do you duplicate every usage of "string" in the entire Lazarus tree?

I don't understand your question. I proposed to duplicate RTL string routines. Lazarus would use only the UTF-8 variant and thus Lazarus routines don't need to be duplicated.

UTF-16 routines would be used by msegui for example

Hans-Peter Diettrich wrote

Dealing with *all* the Unicode quirks IMO is beyond "usual" coding, it will be reserved to specialized text processing components or applications.

+1 from here,
but I never did anything with Mac and I once was told that the Mac constantly uses such quirks.

-Michael

This (magically :-) ) does work with UTF8. You just can't use MyString[i] or store a single character. You need to handle any single characters as strings.

-Michael

Hans-Peter Diettrich wrote

Depends on the type of i. Neither Char nor WideChar are capable of holding every Unicode codepoint (32 bit).

I think Graeme wrote:

   c: UnicodeChar;  // or something that can store at least 4 bytes

(meaning i instead of c)

-Michael

Op 2011-02-17 11:28, Michael Schnell het geskryf:

NO, it doesn't! You can't use FPC's Copy(), Pos() etc reliably with UTF-8 text, because thouse RTL functions work purely on ANSI text (1-byte characters - speaking of String type text here) and don't know about multi-byte characters, combining diacritics etc. Hence LCL and fpGUI have special functions similar to RTL, that knows how to work with UTF-8 encoded text. eg: UTF8Pos(), UTF8Length and UTF8Copy() etc functions.

Regards,
- Graeme -

Hans-Peter Diettrich wrote

That's why such loops should be disallowed with Unicode strings, as kind of low level string handling.

Not only this, but the normal user would like to do

   MyChar := MyString[Length(MyString];

to get the last character of a string.

This would only work if length is counted in Unicode characters disallowing the use of length in stream I/O (such as file read) functions.

Thus IMHO the Length function name should be dumped and two new functions (such as CharacterCount and ByteCount) should be introduced.

For-Each loops may be acceptable as high level string handling, but with what type of the loop variable???

We obviously would need a UnicodeChar Type that holds the 32 Bit encoding.

But the said "quirks" can't be handled by this. I up till now don't understand if - technically - these "quirks" are seen as a single Unicode character or as a sequence of Unicode Characters. Nor do I understand how they can be used in a decent way and if they are necessary or just legacy.

-Michael

Thats the "magic" :-) . pos() does finds the correct multi-byte characters.

combining diacritics etc.

This of course does not work, as theses "Unicode quirks" (this name was not introduced by me !) forces that the same visual character can be encoded in different ways. I don't know if it is even possible (and sensible) to support

?
?

Graeme Geldenhuys hat am 17. Februar 2011 um 10:35 geschrieben:

Yes, it does. UTF8Pos simply calls Pos and converts the byte position to code point.
Pos works, because the first byte of an UTF-8 code point is distinct from the following (%111), so if you search for a valid UTF-8 string Pos will return a valid UTF-8 position. Of course this is a byte position. And since copy, insert and delete use byte positions as well you can use them together without trouble.

Hence LCL and fpGUI have special functions similar to RTL, that knows how to work with UTF-8 encoded text. eg: UTF8Pos(), UTF8Length and UTF8Copy() etc functions.

They are useful when you must deal with code points. For example TEdit.SelStart and SelLength are in code points.

Mattias
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20110217/10876fbe/attachment.html>

+1

Marcos Douglas

If we cloned the multi-platform Delphi version(s), then we had to clone the CLX, not the VCL. New support for other platforms is not yet out, so we cannot decide right now whether we can or will follow that branch.

IMO the new Unicode versions broke so much legacy code, that FPC/Lazarus could become a real successor of the last Ansi version, with free choice of the added Lazarus Unicode handling (UTF-8 for now).

WRT the Delphi IDE: it can be run in a VM, what I prefer for all commercial/long term projects anyhow. The debugging of cross-platform applications requires a separate target machine anyhow, so that virtualization is almost a *must* for the new Delphi versions.

DoDi

How *not*?

Just in your mentioned cases the strings are treated only as strings, not as arrays of chars - that's the difference between high and low level string handling. Dealing with chars inside strings, and incrementing/decrementing indices, is kind of pointer arithmetic, which also deserves knowledge about pointers.

DoDi

You can, when you do it in the *right* way.

because thouse RTL functions work purely on ANSI text (1-byte characters - speaking of String type text here) and don't know about multi-byte characters, combining diacritics etc.

Pos() certainly works with MBCS as well, and you cannot expect that combining characters and ligatures are handled by the basic Unicode functions. When Copy requires an byte count, you can compute it from the difference of the index positions of the involved substrings. It would be better, though, when the basic procedures would not deal with counts or sizes at all.

Hence LCL and fpGUI have special functions similar to RTL, that knows how to work with UTF-8 encoded text. eg: UTF8Pos(), UTF8Length and UTF8Copy() etc functions.

This is a stupid idea, IMO. An "UTF8" prefix is inappropriate when it comes to the distinction between physical and logical functionality. E.g. the number of *logical* (maybe visible) characters can be determined from any string encoding, and that function should have an *unique* name and (possibly) overloaded implementations. Likewise a SubString procedure could take two index positions, which can be determined without knowledge of the string encoding. This way string insertion or extraction do not require a re-parse of the strings, in order to translate logical into physical indices and counts.

IMO we simply have to agree that Length() is a physical property, the number of elements in an array. A logical character count has a very different meaning in string handling, and not even a *single* meaning, when we start dealing with ligatures and other Unicode stuff[1].

[1] In a mix of LTR and RTL parts a distinction between sequential physical and logical indices is required as well. The first RTL codepoint physically follows the preceding LTR codepoint, but logically (on screen...) it precedes the *next* LTR codepoint. I only see one proper solution to such quirks, by restricting the arguments of string handling functions to physical (array) indices. Logical increments of such indices are at the discretion of the user, depending on his understanding of the desired result. Library functions only can deal with different encodings, but always will return physical indices.

DoDi

should be treated as single or multiple logical characters. This in fact is another (third) level of Unicode text manipulation :-(

See also my remarks on logical character positions, in a mix of LTR and RTL sequences.

DoDi

String iteration, based on such positions, is quite useless, because then you'll have to determine the size (byte count...) of the indexed element in another explicit call. Proper for-each iteration will return an *element* of the structure, what in the case of Unicode can be neither an AnsiChar nor a WideChar.

In fact you'll need many different string iterator methods, like NextCharIncludingLigatures, NextCharInVisualOrder...

IMO the only feasable iteration is over physical codepoints, taking into account various string encodings. No special iterator objects are required for that purpose, instead something like MoveNext() etc. can be implemented, as known from database procedures.

Right, most users do not really want nor have to deal with full Unicode and foreign characters. My idea of a unique immutable application defined encoding supports just that model. Then all local/national texts can be processed in the well known way (mostly as SBCS).

Yes, but still, not all Unicode characters fit into a widechar (2-bytes). Most [if not all - I'm not sure here] spoken languages fit into the BMP, but that might not always be the case.

When it comes to only store and display general Unicode text, in a lossless way, the actual encoding of such strings is of no interest to the coder. For that purpose UTF-8 or any OS specific encoding is fine.

Maybe some day you want to translate all your text into Klingon or Goa'uld or whatever alien race visits our planet. Being prepared and supporting the full Unicode is the best option at the moment.

Have you ever tried, seriously, to translate texts *semantically*, preserving the *meaning* of the text? Then you know that the *capability* of entering and storing foreign glyphs is the least important issue, in such a task. And again UTF-8 is perfectly sufficient to even deal with future Unicode versions, with more than 2^17 codepoints.

No problem as long as the strings are displayed properly. If somebody wants a *specific* canonical representation, he has to enforce it by using according translation functions, just like with little/big ending. The implementation of such translation functions is beyond the scope of the runtime library, it deserves detailed knowledge about every language to implement, and maybe of the OS conventions, so that it should be delegated to an installed external library. We also don't care about the visual representation of Unicode text, with e.g. ligatures and accents, do we?

Every higher level Unicode string handling is the task of dedicated libraries, not of an compiler and its RTL.

I see no reason why we should *not* distinguish between *physical* Unicode support (like UTF-8), and *logical* Unicode string handling, which may be restricted to specific codepages (e.g. the BMP). Support of more languages should be delegated to dedicated libraries, supplied by experts with detailed knowledge about the Unicode quirks in their own (natural) language.

Let's proceed step by step :-)

DoDi

When a Mac is capable of importing Unicode text, from e.g. the Web, then the according transformation features should be part of the OS. Once the application needs are known, the Mac target support should include whatever may be required, without affecting other targets or portable

What's simply nonsense with UTF-8/16 strings :-(

Thus IMHO the Length function name should be dumped and two new functions (such as CharacterCount and ByteCount) should be introduced.

Length() since ever returned the number of *physical* elements.

> For-Each loops may be acceptable as high level string handling, but > with what type of the loop variable??? > We obviously would need a UnicodeChar Type that holds the 32 Bit encoding.

Iff we ever want to support such functionality.

The big mess starts with combinations of codepoints. No problems as long as the RTL functions deal with the physical storage of the codepoints, and nothing else.

DoDi

Marco van de Voort schrieb: Can you please make your mailer preserve *spaces* in the subject?

Sorry, I seem to address the wrong person :-(

It also may be Michael Schnell, whose mailer inserts the tabs into the subject. It would be nice if everybody would check his mailer, or when the list software would replace tabs in the subjects by spaces. Oherwise the subjects can become unreadable soon, when other mailers again try to convert tabs into spaces - as already observed in other newsgroups.

DoDi

If so, then I wouldn't waist any more of my time on working on the cpstrnew branch

FPC still follows the thumb rule (notice the thumb rule): the person who implements a certain feature, decides. E.g. if somebody comes up with a working code paged string implementation without breaking existing stuff too hard, I'am pretty sure it will be added. Of course, somebody else might pop up and add Delphi like support.

(which I have been doing quietly on my own).

Always bad. Release sources early, release sources often ;)

then why use FPC - just switch to the "real" thing [Delphi].

Everybody is free do to so, at least I'am not dogmatic about this as long as you use some compiled pascal flavour :)

Op 2011-02-17 13:58, Hans-Peter Diettrich het geskryf:

IMO the new Unicode versions broke so much legacy code, that FPC/Lazarus could become a real successor of the last Ansi version,

I don't think so. Yes the Unicode support in Delphi broke a lot of code, but I think that was just a small bump in the road. I believe Embarcadero expected that too. But then a few months after the release, all the flamewars died down. Now it's business as usual again.

commercial/long term projects anyhow. The debugging of cross-platform applications requires a separate target machine anyhow, so that

I have zero experience with remote debugging, but from what I have heard, it never works as well as native/local debugging. So I guess we will just have to wait and see.

Regards,
- Graeme -

Do you actually have a "new cross platform Delphi" ?

-Michael

Hans-Peter Diettrich wrote

IMO we simply have to agree that Length() is a physical property,

This would result in many errors with "old style" trained users and when compiling "old style" software. So I vote for dropping the ambiguously named "length()" function completely and introduce two new functions with different names.

-Michael

Hans-Peter Diettrich wrote

It also may be Michael Schnell, whose mailer inserts the tabs into the subject.

I use Thunderbird on Linux.
I have no knowledge of something like this happening and in fact I don't see any quirky subjects in this thread. Where do you see this ?

-Michael

Hans-Peter Diettrich wrote

What's simply nonsense with UTF-8/16 strings :-(

Of course right now it is. But this is what average programmers are used to and what we will see in tons of legacy code that might one day is needed top be compiled with a new version of Lazarus.

Length() since ever returned the number of *physical* elements.

Length() since ever returned the number of *visible* characters (until Unicode was introduced to Lazarus :).

Why should we not support a Unicode Character Type ?

But this is a quite lame excuse for not dealing with ("visual") Unicode

Michael Schnell wrote

Where do you see this ?

In fact I do see the tabs when looking at the "source code" of the messages.:

=======================================
X-AOL-SCOLL-SCORE: 0:2:384464192:93952408
X-AOL-SCOLL-URL_COUNT: 0
X-AOL-REROUTE: YES
x-aol-sid: 3039ac1d33834d5d43572286
Subject: Re: [Lazarus] Does Lazarus
support a complete Unicode Component Library? X-BeenThere:
X-Mailman-Version: 2.1.11
Precedence: list
=======================================

The count spaces that have been converted to tabs increases with the reply-depth

But I am unable to detect the source of the problem (which sender ?, which mailer ?, the list distribution software ?)

-Michael

On 02/18/2011 09:41 AM, Michael Schnell wrote:

In fact This happens already with the first reply in this thread:

=======================================

Op 2011-02-18 10:36, Michael Schnell het geskryf:

compiling "old style" software. So I vote for dropping the ambiguously named "length()" function completely and introduce two new functions with different names.

+1

Regards,
- Graeme -

Michael Schnell wrote

On 02/17/2011 02:56 PM, Hans-Peter Diettrich wrote: > Length() since ever returned the number of *physical* elements. Length() since ever returned the number of *visible* characters (until Unicode was introduced to Lazarus :).

Nope, from day 1 of lazarus, Length() returned the number of *physical* elements. Only in some cases this happened to be the number of *visible* characters.

Marc

Marc Weustink kirjoitti perjantai 18 helmikuu 2011 12:01:45:

Nope, from day 1 of lazarus, Length() returned the number of *physical* elements. Only in some cases this happened to be the number of *visible* characters.

We are talking about software here so they are not really physical. A coffee cup, a fork and a knife are physical elements. :-)

Juha

Marc Weustink wrote

Nope, from day 1 of lazarus, Length() returned the number of *physical* elements. Only in some cases this happened to be the number of *visible* characters.

Of course you are right, but inviable characters are no real problem here, visible characters that use up more than one physical element are.

-Michael

?

Michael Schnell hat am 18. Februar 2011 um 11:23
geschrieben:

You can not solve visible characters with a simple string type, because for example in arabic some characters are combined depending on the word. There is simple access to the bytes and with a bit more to code points. ?
Please give more details what problem you need to solve. Otherwise this thread will go endless without a solution.
?
Mattias
?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20110218/b99f7a44/attachment.html>

Op 2011-02-18 12:10, Juha Manninen het geskryf:

We are talking about software here so they are not really physical. A coffee cup, a fork and a knife are physical elements. :-)

In the Sims game they are not. ;-)

Regards,
- Graeme -

Mattias Gaertner wrote

Otherwise this thread will go endless without a solution.

This is true with any Unicode discussion, as the hope that Unicode "fits all needs" , and the hope that the conversion of existing software to Unicode could be rather easy, and the hope that retraining programmers to do Unicode enabled software could be quite smooth, is in vain :-( :-( :-( .

-Michael
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20110218/bc8e91da/attachment.html>

Michael Schnell hat geschrieben:

On 02/18/2011 11:40 AM, Mattias Gaertner wrote: >/

/>/> Otherwise this thread will go endless without a solution.

/> //This is true with any Unicode discussion,

...even more so if you are involved in that discussion

s.c.n.r

:-)

Well, the move was quite painless for most users, because the new string model and also the code conversion tool was well designed. Nonetheless it could have been a bit safer, WRT e.g. PCHAR and byte counts in I/O.

Also implied conversions may not be noticeable to many users, and this also may apply to a future FPC/Lazarus model.

The lack of complaints after the move is typical for Unicode moves on Windows, same observation many years ago with VB5. Most coders only use the BMP, so that UTF-16 is not a serious problem, and the non-BMP users don't have to face anything suspect that nobody seriously tries to deal with strings of foreign codepages, apart from storage and display, even in an Unicode enabled application. This might also be an argument for the new Unicode string type/encoding for Lazarus - the major difference is the different system encoding on Max/Linux, that affects more Lazarus than Delphi users.

In former times remote debugging has been dead slow, sometimes. Most probably not a serious problem with nowadays hardware, and certainly not with virtual machines (with no physical transmission lines in between).

BTW I was just about to buy a Delphi XE licencse, just for curiosity, but then I noticed that it neither supports multiple platforms nor 64 bit right now :-(

DoDi

Length() cannot be dropped, because it's required for streaming (byte counts, buffer sizes...). I also see no real need for a CharCount or similar function, in real life code.

DoDi

Software slang has its own idea of "physical", where it addresses the representation in physical memory.

A coffee cup, a fork and a knife are physical elements. :-)

;-)

DoDi

Hans-Peter Diettrich wrote

Length() cannot be dropped

Of course not. It should just be renamed to prevent lots of hard to detect run time problems when used erroneously for "visual character count".

-Michael

Op 2011-02-18 14:19, Hans-Peter Diettrich het geskryf:

Also implied conversions may not be noticeable to many users, and this also may apply to a future FPC/Lazarus model.

I've lately had a renewed interest in working on the Unicode for FPC, so have been silently hammering away at the code in my [limited] spare time. If all goes to plan, the change-over should be relatively painless, and more intuitive than Delphi's was. This should also mean that projects like Lazarus LCL, fpGUI and MSEgui could essentially switch to simply using the String type instead of custom types like TfpgString or TMSEString etc., and removing special UTF8* string functions and replacing them with standard RTL ones. I guess fpGUI will be the first test case on how successful such a string change-over will be. :)

The good news is [or at least that is my plan], is that if LCL, fpGUI and MSEgui still want to use a hard-coded internal string encoding, that should be simple to accomplish - via something like the following. eg: for fpGUI it could be...

type
  TfpgString = String<cp_utf8>; // <-- this should be the only change.
var
  MyString: TfpgString;  // this code is as it always was

Introducing a method that could set the default global encoding for String type (ignoring the OS platform) is also in the plans. I believe Delphi can do this too.

BTW I was just about to buy a Delphi XE licencse, just for curiosity, but then I noticed that it neither supports multiple platforms nor 64 bit right now :-(

Yeah, the wait continues...one delay after the next. :-/

Regards,
- Graeme -

Michael Schnell wrote

Of course not. It should just be renamed to prevent lots of hard to detect run time problems ?when used erroneously for "visual character count".

And cause trouble to everyone that has been using it correctly? I don't think so.

Plus, Lazarus already uses UTF-8 since years, people should already be used to Length not representing chars...

Then the problem sits before the screen ;-)

I see no chance for "hard to detect" problems, when some code has been broken since ever.

DoDi

Hans-Peter Diettrich wrote

I see no chance for "hard to detect" problems, when some code has been broken since ever.

I don't feel that the legacy (non-Unicode aware) instruction

     MyChar := MyString[Lengt(MyString)];

to get the last character of a string is broken since ever, but I do feel that it has been used in thousands of projects.

-Michael

Such code has been and still is valid for ASCII characters, like spaces, directory separators etc., at the end of an string. It *happened* to work with other SBCS encodings, but this was never guaranteed, and it never worked for MBCS.

DoDi

Well that is my first goal. I don't care much about redundant FPC extensions, because despite of the fierce discussion they tend to be hardly used.

But if you want to waste some time, Florian already invited patches.

Because if I hear that one more time, I'm really go nuts! :) Just because Delphi is brain-dead, doesn't mean FPC must also be brain-dead.

Well, people tend to call braindead what they don't understand.

I guess the most logical would be String(?) syntax - upgrading the String type to be encoding aware, similar to what was done when String was a shortstring and then became a longstring later.

It's possible at a runtime penalty (runtime scaling of the [] operator). See the original pre D2009 discussions. Some estimate that factor large, some low, but all we know is that the Delphi devels were scared to go that direction.

And IMHO there is not enough merit in the whole business to go a different course except your Delphi compatibility allergy.

+1

except us long time, old school newbies who are finally attempting to move from TP/BP to FPC/Lazarus ;) it is a huge jump over delphi and into this fancy schmancy new unicode (wotz'at?) world... i won't even mention objects and GUI stuffings :P

indeed! i just used it yesterday to locate the last character of a string in a tstringlist... specifically, massaging mime text back into ASCII 7 or 8 bit text... specifically, for the removal of the trailing mime equals sign indicating line-wrap at a specific spot...

"we don't want/need no mime mess... give us the trademark symbol as '(tm)' and the copyright symbol as '(c)' and we're just fine."

the above comes from several blind folk who still use braille writers for reading their messages because text-to-speech stuff is still too expensive and they simply cannot do GUI stuffs...

When you're still working on that branch (you got it to compile again?) it might indeed be that you fullfill Florian's point of "those who implement it decide how it's done" (more or less).

I just got an idea: What about using a more sane way of the Unicode string feature in mode (Obj)FPC (like you're trying to implement, Graeme), keeping the old way in mode Delphi (String = "RawByteString") and introduce a new mode Delphi2009 that tries to be compatible with Delphi's Code Page aware string type. Or maybe instead of a new mode better add a modeswitch "DelphiCPAwareStrings" (or something like this...) that needs to be enabled for code that should be compatible to Delphi 2009. Thus users of pre-Delphi2009 could be happy, too.

And yes, I'm aware that this means work. ;)

Just my 2 cents...

Regards,
Sven

"we don't want/need no mime mess... give us the trademark symbol as '(tm)' and the copyright symbol as '(c)' and we're just fine."

This can be accomplished in every encoding, by StringReplace.

DoDi

this is, actually, what i'm doing :)

the problem is/was getting at them in their "raw" state and not their "interpreted" state... i figured it out in synapse, though... at least i do have something that works anyway :P

PS: subject line tabs manually converted to spaces in this reply...

Op 2011-02-19 13:50, Sven Barth het geskryf:

When you're still working on that branch (you got it to compile again?) it might indeed be that you fullfill Florian's point of "those who implement it decide how it's done" (more or less).

I've come to realize that discussing a design is fruitless (99% of the time) in the FPC realms. Simply implement it and see what happens - after all, nobody else is working on Unicode support, so it's not like there will be any code or design conflict. If nothing comes off it, oh well, that wouldn't be a big surprise either, because pretty much everything I suggested (with or without patches) get declined in the FPC project. :-O

and introduce a new mode Delphi2009 that tries to be compatible with

I don't use Delphi at all, so I really don't care what Delphi does or doesn't do. I don't follow Delphi development. I think some programmers can think for themselves - no need to just clone code of features from another project. Unicode is nothing new, it's just Object Pascal that has been very late to the game.

Introducing yet another compiler mode also doesn't seem like the way to go. FPC is already complex enough. The String type simply needs to evolve to Unicode support, just like it did from ShortString to LongString. A natural evolution. This should probably be something included in both ObjFPC and Delphi compiler modes, but again, I only every work with ObjFPC mode.

Regards,
- Graeme -

Remember your crying about the different handling of temp. interfaces in FPC which is only a implementation detail?

This should probably be something included in both ObjFPC and Delphi compiler modes, but again, I only every work with ObjFPC mode.

When did String change from ShortString to LongString (actually AnsiString) in ObjFPC? ObjFPC used AnsiString for a String from the beginning ...

ObjFPC used AnsiString for a String from the beginning ...

You mean ShortString, right?

http://www.freepascal.org/docs-html/prog/progse75.html#x274-289000D.4

Sorry Florian, but I need to correct you here:

=== source begin ===

program stringtest;
{$mode objfpc}
procedure TestShort(var aStr: ShortString);
begin
end;
procedure TestAnsi(var aStr: AnsiString);
begin
end;
var
   s: String;
begin
   s := 'Hello World';
   TestAnsi(s);
end.

=== source end ===

=== console begin ===

[sven at artemis oneshots]$ fpc stringtest.pas
Free Pascal Compiler version 2.4.2 [2010/11/08] for i386 Copyright (c) 1993-2010 by Florian Klaempfl
Target OS: Linux for i386
Compiling stringtest.pas
stringtest.pas(19,13) Error: Call by var for arg no. 1 has to match exactly: Got "ShortString" expected "AnsiString"
stringtest.pas(21) Fatal: There were 1 errors compiling module, stopping Fatal: Compilation aborted
Error: /mnt/data/applications/fpc/2.4.2/bin/ppc386 returned an error exitcode (normal if you did not specify a source file to be compiled)

===console end===

(same with 2.5.1 from end of January this year)

Regards,
Sven

PLEASE don't do this. I don't need Unicode support, and I DON'T want to have to worry about its complexities. Just leave a compiler mode or switch that lets me keep "string" as LongString and I will be happy, whatever happens with unicode. And, as far as I understand, you can still set string = shortstring with a compiler switch, so leaving a compiler switch for unicode strings would be "just like it did from ShortString to LongString".

cheers,
John Sunderland

I don't need Unicode support, and I DON'T want to have to worry about

its complexities.

Just leave a compiler mode or switch that lets me keep "string" as

LongString and I will be

happy, whatever happens with unicode. And, as far as I understand,

you can still set

string = shortstring with a compiler switch, so leaving a compiler

switch for unicode strings

would be "just like it did from ShortString to LongString".

But why are you using the generic "string" type at all? If you want to make sure that you use AnsiString or ShortString or whatever then use it directly. That way you have full control about the string type used.

Op 2011-02-21 10:02, Florian Klaempfl het geskryf:

.... the different handling of temp. interfaces in FPC which is only a implementation detail?

Sorry, no idea how that relates to this discussion. Also, please drop the childish remarks. Grow up and act your age!

When did String change from ShortString to LongString (actually AnsiString) in ObjFPC?

I meant the usage of String, and how it's meaning can be toggled with a simple compiler directive - yet no other code changes are required from the developers side. As such, the String type (meaning AnsiString) can naturally evolve to mean "a String with text encoding support" - no need for a new type like EncodingEnabledString or CodePageString etc, etc.

ObjFPC used AnsiString for a String from the beginning ...

Learn your own compiler! ;-) Theo is correct, mode ObjFPC defaults to ShortString, not AnsiString.

Regards,
- Graeme -

Op 2011-02-22 08:11, J?rgen Hestermann het geskryf:

But why are you using the generic "string" type at all? If you want to make sure that you use AnsiString or ShortString or whatever then use it directly. That way you have full control about the string type used.

Exactly. Just like any other types (numeric, float or text). If you expect some specific behaviour from a type, then use a specific type that gives you that behaviour.

String is a generic type (it already has multiple meanings), so it should evolve.

On a side note:
Just like I think Integer type should have evolved (like it did from 16-bit to 32-bit in the past), so too it should have moved to 64-bit. But unfortunately the powers that be, decided that its evolution should magically stop at 32-bits - no idea why really. But that's a whole different discussion. ;-)

Regards,
- Graeme -

J?rgen Hestermann wrote

> If you want to make sure that you use AnsiString...

Oooops, AnsiString and UTF8String are just aliases for the same type.

-Michael

Delphi clone/non delphi clone? Does it ring bells? You always tell us: don't do a delphi clone. Then we don't do it and you cry.

An even better argument not to change it :) Even in ObjFPC mode a string is a shortstring for years.

Op 2011-02-22 10:44, Florian Klaempfl het geskryf:

Delphi clone/non delphi clone? Does it ring bells? You always tell us: don't do a delphi clone. Then we don't do it and you cry.

Well make up your bloody mind??? If you want to make FPC a Delphi clone, then stick to it. Having some areas cloned and some not, makes things a little annoying. That was what I was complaining about that time. An NO, I'll never shed a tear over FPC - that's all your wishful thinking.

An even better argument not to change it :) Even in ObjFPC mode a string is a shortstring for years.

Out of curiosity, why does ObjFPC mode default to ShortString and not AnsiString? Isn't the TP mode enough for those die hard, corner case programmers that still want their Turbo Pascal and Delphi 1 legacy code to work? Why must modern day programs still be limited to 255 characters per string? Why not let those corner case developers rather switch from AnsiString (what should be the default) to the ShortString.

Regards,
- Graeme -

No problem, if you don't troll, somebody else will.

That was what I was complaining about that time.

You followed the generics discussions?

An NO, I'll never shed a tear over FPC - that's all your wishful thinking.

I'am glad to hear that you're always happy about FPC :)

Why break existing code for nothing? Such a change would simply steel people their time while the gain is zero: people interested in string = ansistrings can either add $H/-Sh to their project when migrating from shortstring to ansistring or they can replace explicitly string by ansistring.

Op 2011-02-22 11:28, Florian Klaempfl het geskryf:

I'am glad to hear that you're always happy about FPC :)

:-)

> Out of curiosity, why does ObjFPC mode default to ShortString and not > AnsiString? Why break existing code for nothing?

Supporting legacy code (10-20 years old) with default compiler options just seems a bit nutty. Considering there is a perfectly good compiler mode (mode TP) just for them. So why carry that legacy support over to a more modern compiler mode (mode objfpc)... it just doesn't make a lot of sense.

Regards,
- Graeme -

Probably because that was the "normal" way of doing it, since ansi/long string was added to Dephi a *long* time ago. Also simply less typing, more readable if I always know that a string is just a string. Sure, if you are doing lots of stuff with multiple string types, it makes good sense to be specific. I am not. (If I am fussed about what sort of integer I have, I will work it out and declare it properly. If I just want a counter, I will just use "integer").

I suppose if the ansistring type remains, I can do a global replace on ALL my code, or redeclare string as ansistring, or something, but I don't see why there should not be a compiler switch to do it, seeing that that was the way the last string divergence was handled. What I was really afraid of was that Graeme had declared ansistring to be surplus to his needs, and that therefore there was no need for it to exist.

John Sunderland

Op 2011-02-22 12:08, John het geskryf:

string was added to Dephi a *long* time ago. Also simply less typing, more readable if I always know that a string is just a string.

There is no such thing as "just a string" or "plain text". All text have some encoding defined/associated to them. ;-)

... or redeclare string as ansistring, or something,

I still hope to make this a possibility. Same goes for a compiler switch. Only closer to the time one will be able to say if it is feasible or not. We all hope FPC can be a little more accommodating that Delphi was in this respect.

Regards,
- Graeme -

But why are you using the generic "string" type at all? If you want to make sure that you use AnsiString or ShortString or whatever then use it directly. That way you have full control about the string type used.

The compiler should support string types as *really* different types, and should not try to convert (certain) string types automatically, or to reinterpret Ansi as UTF-8 or the like. Then there exist no problems with character codes, that do not exist in the codepage choosen by the developer.

I wouldn't care when a very new type, e.g. "text" or "Unicode", would be introduced for Unicode encodings, with slightly different rules (low-level support removed). This would allow to use strings *as* and *for what* these types have been introduced, strictly separated from other textual content and types.

I also wouldn't care when e.g. filenames would become a type of their own, with all *required* support for the various file systems and their conventions - because this is the *only* place where Unicode can not be ignored, in no application. This type could become a true class type, with polymorphic implementation of the target platform, networking and web conventions - e.g. tURL.

DoDi

String is a generic type (it already has multiple meanings), so it should evolve.

Generic types should never evolve in a breaking way.

Look how even the FPC developers have banned the use of AnsiString, for performance reasons. [What raises the next question: what can we expect from designers of such an mental background, with regards to type evolution ;-]

On a side note: Just like I think Integer type should have evolved (like it did from 16-bit to 32-bit in the past), so too it should have moved to 64-bit.

Adding encodings to the string type looks to me like turning "integer" into "complex".

But unfortunately the powers that be, decided that its evolution should magically stop at 32-bits - no idea why really.

This specific issue was discussed in depth, across all platforms and languages. When somebody needs a guaranteed "capacity" for his integral numbers, he can choose from many representations (Currency, Int64, BCD...) or, in Pascal, he can use subranges. Where "subrange" does not necessarily mean less than 32 bits, but could be extended to any bitcount.

But that's a whole different discussion. ;-)

Not really ;-)

DoDi

I suppose if the ansistring type remains, I can do a global replace on ALL my code, or redeclare string as ansistring, or something, but I don't see why there should not be a compiler switch to do it, seeing that that was the way the last string divergence was handled.

String handling code in the libraries, and inside the compiler, deserves a unique (fixed, default) string representation. The low-level code for dealing with Ansi encoding can not be exchanged with code dealing with UTF-8 or UTF-16 encoding. That's why a simple recompilation of the RTL and VCL is impossible, because these deal with strings of immutable types in many places. At least the resulting code would suffer from implied string conversions, with the risk of losses, in most library subroutine calls.

Guess how e.g. a "set of char" is implemented for Unicode...

DoDi

Probably because that was the "normal" way of doing it, since ansi/long string was added to Dephi a *long* time ago.

Well, an even longer time ago the type "String" was a ShortString of max 255 characters.
My first contact with the "modern" (generic) meaning of "String" caused me lots of headache
because I was not even aware that the meaning suddenly changed to be generic.

Also simply less typing, more readable if I always know that a string is just a string.

But that's just the problem: If you don't know (mind) about the string type incarnation then you blunder into lots of traps.

I always need to know how a certain type is handled and stored. Otherwise I could not avoid all the special problems that arise. If I don't know that ShortString is limited to 255 characters and if I don't
know that AnsiString is a (reference counted) pointer and if I don't know that assigning two AnsiStrings just copies the pointer (while for ShortStrings the whole string is copied) and if I don't know about copy-on-write for AnsiStrings then I get into real trouble too often.

Trying to avoid all the hassle that comes with different string types (and encodings) by just using the generic string type is wishful thinking IMO.

Sure, if you are doing lots of stuff with multiple string types, it makes good sense to be specific.

I think this is unavoidable. Sooner or later you cannot avoid to learn all the string specials.

If I just want a counter, I will just use "integer").

I always use LongInt because when using integer I am not sure what range of numbers I will get. Who knows what special command line or compiler switche changes this to 16 bit or even less. Then my program would not work anymore. I have to makes sure that my expectations to a certain data type are correct all the time.

I suppose if the ansistring type remains, I can do a global replace on ALL my code, or redeclare string as ansistring, or something, but I don't see why there should not be a compiler switch to do it,

I still would prefer the search and replace. Often the code has to be changed too. A program that was written with ShortStrings in mind may not work properly with AnsiStrings (just think about records with Strings).

What I was really afraid of was that Graeme had declared ansistring to be surplus to his needs, and that therefore there was no need for it to exist.

I did not unterstand him this way. I think he only wanted to point out that the *generic* type string should always be as modern as possible (i.e. AnsiString and if it comes to encoding then UTF8). Nevertheless, I would try to avoid such generic types at all.

Generic types should never evolve in a breaking way.

But that's the whole reason for generic types: They should not map to a specific type but vary dependend on certain environment settings. If the generic type String means the same type all the time than it's just an alias for that type.

I never understood how this can be of advantage but that's the reason for the design of generic types.

Look how even the FPC developers have banned the use of AnsiString,

for performance reasons.

Yes, therefore I would never use generic types. If ShortStrings are enough for my program, I use them (for performance reasons and ease of use). If it comes to strings longer than 255 characters or other features of AnsiStrings I use AnsiStrings. But I would never leave it to some obscure environment setting what is chosen. Still this is the way the generic type String is designed.

Adding encodings to the string type looks to me like turning

"integer" into "complex".

But what exactly should be the type "String"? Because of the generic nature it must map to different string types under certain environment conditions. Otherwise it would be just another name for a fixed string type.

> But unfortunately the powers that be, decided that its evolution should > magically stop at 32-bits - no idea why really. This specific issue was discussed in depth, across all platforms and

languages.

When somebody needs a guaranteed "capacity" for his integral numbers, he can choose from many representations (Currency, Int64, BCD...) or,

in Pascal,

he can use subranges. Where "subrange" does not necessarily mean less

than 32 bits,

but could be extended to any bitcount.

Again, what reason exists for a generic integer type if not providing the "best" integer representation for a certain platform? If you compile for DOS it
should be 16 bit, for Win32 it should be 32 bit and of course for Win64 it needs to be 64 bit.
Either you vary it or not. If there is no variation then what reason exists for a generic type?

Generic types IMO should vary according to the target machine, with the most prominent type "pointer".

If the generic type String means the same type all the time than it's just an alias for that type.

When it's allowed to redefine the string type per unit, using $H, then it's (kind of) an alias.

I never understood how this can be of advantage but that's the reason for the design of generic types.

Strings are somewhat special types. Just the move from ShortString to AnsiString broke the char at index zero, which holds the string length of an ShortString, but is no more accessible in an AnsiString.

A ShortString16 could hold 16 bit (wide) chars, and a ShortString32 could hold the full range of Unicode codepoints. This were compatible with other generic types, where the (element) bytecount is not fixed. When s[0] still would hold the string length, then the maximum length of the strings would have grown "naturally" with the char size. Following this model we had no discussion about future string types, e.g. Unicode issues (like UTF-8) never had been a problem.

But of course such string implementations are incompatible with Delphi, so...

Again, what reason exists for a generic integer type if not providing the "best" integer representation for a certain platform?

IMO not the platform, but the machine architecture and mode is essential for the determination of "integer" (and "pointer").

If you compile for DOS it should be 16 bit, for Win32 it should be 32 bit and of course for Win64 it needs to be 64 bit.

There is no adavantage in the use of 64 bit integers on current 64 bit machines. There is no speed penalty in the use of values shorter than the register size, and then the data and code size have to be taken into account, too. Even if data size doesn't matter with regards to available RAM, reading and writing twice the amount of bytes costs twice the number of RAM cycles, which are much slower than CPU clock cycles.

When the same code has to run on target machines with different architecture, then the use of overly long generic integers is not a good idea.

Either you vary it or not. If there is no variation then what reason exists for a generic type?

Generic types should not affect any (existing) code. That's hard to accomplish when new attributes shall be added to the string type.

Otherwise generic types should be forward-compatible, i.e. code assuming (at least) 16 bit integers will never fail with integers of any higher bitcount. Backwards compatibility can not be achieved, i.e. code assuming 32 bit integers is not guaranteed to work with 16 bit integers, occuring with older or "smaller" targets. A "backport" should adress such issues, by replacing "integer" by more specific types, when there exist allowed values and a chance for targets with integers of less bits.

DoDi

IMO it's sufficient when your code continues to work in future time, with integers of possibly higher, but never of lower precision.

DoDi

Hans-Peter Diettrich wrote

Generic types should never evolve in a breaking way.

The generic type String once was intended to hold entities of the generic type char (=character), and char was meant to hold a complete (maybe) printable thingy in an 8 bit encoding.

So this is definitely broken when the library functions enforce storing multi-byte characters in an (alias of) String.

(I don't want to judge that this way to go is wrong, I only want to state that it absolutely "evolves a generic type in a breaking way".

-Michael

FPC RTL/FCL contains classes too. It is not just a few isolated procedures, but a lot of classes too. And not all are solvable with overloading.

Marco van de Voort wrote

FPC RTL/FCL contains classes too. It is not just a few isolated procedures, but a lot of classes too. And not all are solvable with overloading.

Indeed, I hadn't thought about that. That makes the case for a general migration to the new string type stronger.

And if we are going to make such a large scale incompatible transition, then at least we should make it compatible with the new Delphi.

Op 2011-02-24 12:36, Felipe Monteiro de Carvalho het geskryf:

Indeed, I hadn't thought about that. That makes the case for a general migration to the new string type stronger.

Correct. The generic String type will have a slightly new meaning/functionality in the future.

And if we are going to make such a large scale incompatible transition,

I really don't see much of an issue here. Simply continue using String as normal, and just like Delphi, don't expect mystring[i] to return something very useful like it does for AnsiString types. In the later case, rather use a Unicode Char Iterator or something.

then at least we should make it compatible with the new Delphi.

I can't see how, or rather exactly what that would entail. For one, I haven't used Delphi in a day-to-day basis, since Delphi 7, so I have no clue how Delphi's Unicode support works, other than reading a website article here and there. I also don't intend to buy Delphi any time soon, just to learn how to improve FPC. Also under Delphi, mystring[i] returns a Char (or WideChar I think) which is now 2-bytes large. Under FPC that would be dependant on the platform you are targeting I would think. Remember the plan is to use the native encoding of each platform... So under Linux and Mac the String's encoding will be UTF-8 by default, and under Windows it will be UTF-16 by default, DOS will be US-ASCII etc... So we definitely can't be a 100% Delphi clone in this regard!

Regards,
- Graeme -

My guess is that they will use UTF-16 everywhere. Simply because they have it, always focus on keeping stuff running with minimal options instead of choosing the most long term portable solution and QT is also UTF-16 oriented.