After the test program, sent by Paul, I was playing more with AnsiStrings in Delphi XE, with catastrophic results :-(

At least when MBCS enter the scene, and UTF-8 is widely used in FPC and Lazarus and is the preferred string type on Linux, incredible bugs show up. With
var

   a: AnsiString;
   u, u2: UTF8String;

even a simple assignment of

   u := 'ü';

results in an string of Length 1, the second byte has gone away.

A direct comparison of
   a := 'äöüü';
   u := a;
   WriteLn(a = u);
will show True or False - dunno why.
Also
   Pos('ü', u); //0
   Pos(a[3], u); //0
Since u := a[3] doesn't work,
   u2 := Copy(u, 5, 2); //'ü'
   WriteLn(Pos(u2, u)); //5 - correct!
but
   WriteLn(Pos(string(u2), u)); //3
returns the index in the UnicodeString, into which u was silently converted.

After all these flaws I see no use for Delphi compatible AnsiString procedures, in an environment where MBCS (UTF-8) strings must be handled.

If we ever want to proceed with the new AnsiStrings, we should specify what every RTL procedure exactly *should* do, in a meaningful and usable way, regardless of Delphi compatibility.

E.g. Pos() should convert the first argument (SubStr) to the encoding of the second string, if both are different, before searching for the SubStr. Only then the result can be used as an index into the second string.

Now we can guess whether the flawed handling of AnsiStrings in Delphi is due to sloppyness of the implementors, or necessary conversions are not performed for speed reasons. Whenever a chance exists, that Pos or other standard functions must convert an argument, the use of strings with different encodings becomes very questionable, performance-wise. Perhaps it would be better (and easier to implement) when only one string type is used in an application, with possible values of native/UTF-8/UTF-16. Required conversions then can be restricted to I/O methods (file encoding), ShortString conversions (wich codepage???), and external subroutines (OS, widgetsets).

DoDi

I don't have a Delphi 2009+, so I'm going to discuss this with a colleague, who has all versions. AFAIK,. he'll be back from holiday next Monday.

-Michael

Just think of Pos() as it only compares bytes. AnsiPos() should care about encoding too. As I remember AnsiPos() has AnsiString arguments instead of RawByteString and therefore the conversion will be made before the byte compare.

Best regards,
Paul Ishenin.

Apart from the mentioned implementation flaws, I came across severe problems with the new AnsiString *model* in general. Let's play around with the Pos() function, which certainly is an inevitable part of any stringhandling.

A general function

  function Pos(SubStr: T1; Str: T2): integer;

should return the character index of SubStr in Str, i.e. Str[i] should definitely be the begin of SubStr within Str.

It also should be possible to find the end of SubStr within Str, in order to e.g. return the remainder of the text.

With multiple coexisting string encodings we have to solve the following problems:

A reasonable result, i.e. the index in the given string, of the given encoding T2, will require to convert the search string SubStr into exactly that encoding. This requires two conversions, from T1 into UTF-8 (or UTF-16) and then into T2. Clearly this can be prevented by using strings of only one encoding, but what about string literals? When a string literal has to be converted, it most probably ends up in UTF-8/16 encoding, what would cause the Unicode version of Pos() being called, resulting in a wrong result. Even if we assume that string literals are stored as native (CP_ACP) strings, or as Unicode, what actually depends on compiler directives, a couple of overloaded Pos() functions had to be added, when an unwanted conversion of *both* arguments into UTF-16 shall be avoided.

The only possible solution were IMO a

  function Pos(SubStr: UnicodeString; Str: RawByteString): integer;

in the *hope* that this version takes precedence over the all-Unicode version.

But when we have the begin of the substring, how do we find its end? Here Length(SubStr) is of little help, since it represents the number of bytes in encoding T1, useless with T2. So we need a feature to determine the length of an string in any (supported) encoding, like:

   function EncodedLength(s: string; cp: TEncoding): integer;

Or we add a function

  function EndPos(SubStr: T1; Str: T2): integer;

returning the index of the char following SubStr in Str.

Or we combine both, into
  function Pos2(SubStr: T1; Str: T2; out begIndex, endIndex: integer):
boolean;
with the result eventually indicating whether SubStr was found in Str.

But even if we implement all that, and use it *everywhere* in our code, the chance for any number of implicit encoding conversions remains :-(

Do you see any chance to reduce the number of possible conversions, other than by using only one single encoding throughout RTL and application code?

But what's the use of strings with a stored encoding, then? Except for strict compatibilty with a flawed Delphi model and implementation, that may be dropped again in the next Delphi version?

DoDi

Paul Ishenin wrote

Just think of Pos() as it only compares bytes.

A full-time programmer can be trained to think that way. (He also can be trained to do ASM.)

For a student, a "sometimes" - programmer, a hobbyist, someone who just uses Pascal to create some testing/supporting tools for his hardware devices, web programming or whatever, and lots of others (you name it), thinking that way is out of question. They have no notion whatever of Unicode implementation details.

For them, a string is just a sequence of visible characters (no idea how Chinese think about that). The language constructs need to work accordingly. This includes Pos() and Length() (which of course might come with modified names in a Unicode aware RTL) and language features like MyString[i]. And of course a "char" needs to be able to hold any visible character for them.

-Michael

Please don't discriminate students. At least not all of them ;P

Regards,
Sven

Sven Barth wrote

Please don't discriminate students. At least not all of them ;P

Sorry for inappropriate wording of a not-native speaker.

I did not think about students in general, but anybody learning the Pascal language.

-Michael

Ah, that may be correct for the old (ASCII/Ansi) model. Thanks for the reminder :-)

AnsiPos() should care about encoding too. As I remember AnsiPos() has AnsiString arguments instead of RawByteString and therefore the conversion will be made before the byte compare.

AnsiPos *was* designed for dealing with MBCS encodings, but in Unicode Delphi the implementation changed to UnicodeString arguments, the "Ansi" names only have been retained for compatibility with existing code.

Thus AnsiPos *now* only accepts UnicodeStrings, with obviously wrong results when called with (automatically converted) AnsiStrings :-(

There exists an AnsiStrings unit, with another AnsiPos function, which accepts an AnsiString. Unfortunately I couldn't make it work right now, dunno why, and it only accepts native (CP_ACP) AnsiStrings - no solution for UTF-8 strings :-(

DoDi