1) Why UTF8String made incompatible with AnsiString(CP_UTF8) ( UTF8String = type AnsiString(CP_UTF8); )? Why not an alias?
2) Same question about RawByteString

3) why UnicodeString is separate type? Does it should be AnsiString(CP_UTF16)? If not what is AnsiString(CP_UTF16)?

4) If now ansistring can contain text in any supported encoding, I think that this only type is enough to support both single-byte and Delphi`s UTF16. The only need is modeswitch to map string to UTF16-encoded _AnsiString_ (!). There is no need to have two (or more) RTLs (for UTF8 or UTF16 f.e) because encoding info is already included in _any_ longstring.
4.1)However, for speed reasons, internally there should be separate code for quick handle particular encodings to avoid conversions (for [Lower|Upper]Case for example), but interface of RTL units should remain unchanged.

5) My proposed changes to spstring.
5.1) if string is defined w/o explicit encoding (f.e. just "string", in H+ modeswitch or "ansistring") then variable of that type can contain value with any encoding and preserve encoding of assigned expression (no encoding conversion). Call it "universal string". It not the same _type_ as ansistring(some_codepage).
5.2) In unicode Delphi mode encoding of all string constant values is forced to UTF16, source encoding can be any. strings can be forced to UTF16, but can also be "universal string"(see 5.1). 5.3) all RTL string routines params are "universal string". No need to separate unicode versions, because the`ll be already unicode-aware. 5.4) UTF8String, RawString, UnicodeString are aliases but not unique types.

Alex Shishkin wrote

1) Why UTF8String made incompatible with AnsiString(CP_UTF8) ( UTF8String = type AnsiString(CP_UTF8); )? Why not an alias? 2) Same question about RawByteString

RawByteString is special, because any other string can be assigned to it WITHOUT conversion, but the code page of the assigned string will be kept, so one can still check which code page the original string had.

Regards,
Sven

12.10.2011 13:45, Sven Barth пишет:

Thanks this is what I miss. But the question remains: what is AnsiString(CP_NONE)?
If it is special too, why not just alias and if not what is it?

My proposed changes to spstring.
1) if string is defined w/o explicit encoding (f.e. just "string", in H+ modeswitch or "ansistring") it treated as RawByteString.
2) In unicode Delphi mode encoding of all string constant values is forced to UTF16, source encoding can be any. String variables forced to UTF16.
But most unidode Delphi code could be compiled in simple Delphi mode.
3) all RTL string routines should be encoding aware (accept RawByteString). No need to
separate unicode versions.

4) UTF8String, RawByteString, UnicodeString are aliases but not unique types.
5) concatenation of 2 rawbytestrings converts right operand to left`s encoding.
6) may be use concept of "universal string" from my previous message. "ansistring" (w/o explicit encoding) = RawByteString + clause "5".

In fine, main idea is to use rawbytestings as widely as possible, but avoid data corruption (perform codepage conversion when it absolutely necessary).

1) Why UTF8String made incompatible with AnsiString(CP_UTF8) ( UTF8String = type AnsiString(CP_UTF8); )? Why not an alias?

An alias allows to assign strings of *any* encoding, with possibly fatal consequences. A strict UTF8String type allows for implicit conversion, whenever required, so that such a string can contain nothing but UTF-8 encoded characters.

2) Same question about RawByteString

Variables of type RawByteString have no fixed encoding. Any AnsiString can be assigned to an RawByteString variable, without conversion. But when a RawByteString is assigned to an different string type, an conversion may be necessary.

3) why UnicodeString is separate type? Does it should be AnsiString(CP_UTF16)? If not what is AnsiString(CP_UTF16)?

Delphi only allows for an element size of 1 for AnsiStrings (and RawByteString). The reason is unclear/undocumented. One reason may be the type of str[i], which is an AnsiChar for Ansi encoding, and a WideChar for UTF-16 encoding. Consequently a Char must have 4 bytes, when AnsiStrings of a variable element size would be allowed.

4) If now ansistring can contain text in any supported encoding, I think that this only type is enough to support both single-byte and Delphi`s UTF16.

AnsiString is only a string type with *native* (system) encoding, i.e. type AnsiString(0). UTF8String is already a different type AnsiString(CP_UTF8).

The only need is modeswitch to map string to UTF16-encoded _AnsiString_ (!). There is no need to have two (or more) RTLs (for UTF8 or UTF16 f.e) because encoding info is already included in _any_ longstring.

Right, there exists no *technical* need. But a *practical* need is the reduction of conversions, between strings of different encodings, and the implementation of procedures, that work with strings (e.g. ToUpper).

4.1)However, for speed reasons, internally there should be separate code for quick handle particular encodings to avoid conversions (for [Lower|Upper]Case for example), but interface of RTL units should remain unchanged.

The fastest implementation were an UCS4 string, with no encoding conversions required inside any library. Then conversions are required only in calls of OS or other external library functions.

DoDi

But the question remains: what is AnsiString(CP_NONE)?

CP_NONE is not a codepage, it's listed under Clipping Capabilities.

If it is special too, why not just alias and if not what is it?

CP_ACP (0) is the placeholder for the system Ansi CodePage.

DoDi

AFAIK yes.

Regards,
Sven

12.10.2011 16:34, Hans-Peter Diettrich пишет:

So if I declare "MyString : AnsiString(CP_UTF8)" and assign win1251 encoded string to it no conversion will be made? It`s strange.

procedures, that work with strings should use RawByteStrings, and fpc unlike delphi might allow UTF8 or UTF16 for RawByteStrings.

But they consume significantly more memory.

Huh? No, of course there will be a conversion. I don't know what DoDi is worried about here.

Regards,
Sven

Ooops, so there is not encoding ID "RAW", but a consistent RAW string can have any encoding ID ?

Funny...
-Michael

Alex Shishkin wrote

...

While I feel that something like this migh be a(nother) decent way to handle Unicode strings, obviously different decisions have been taken because

1) Pascal is supposed to be a strictly typed language. This asks for being able to use distinct types for different encoding.
2) Delphi compatibility.

-Michael

12.10.2011 16:52, Michael Schnell пишет:

I not suggest to break Delphi compatibility or disallow distinct types for different encoding.

In fact this seems to avoid the "nice" feature of the current (Delphi-like) Implementation.

Here seemingly inconsistent (or rather "intersexual") strings are provided for: perfectly encoded and distinguishable data that suffers from being locked in a variable that happens to be of a type with the wrong name.

-Michael

Delphi doesn't allow for such syntax. You'll have to write
   type TMyString = type AnsiString(CP_UTF8); // = UTF8String
   var  MyString: TMyString;
Then all assignments to MyString are converted into UTF-8.

DoDi

12.10.2011 15:29, Alex Shishkin пишет:

7) string indexing
if string is "universal" indexing is always byte-based (compatible to delphi xe ansistring and legacy ansistring). So s[i] is alwayse i`th byte. For UTF16 string indexing word-based of course. Indexing of uft8string is the question (i`th byte of i`th unicode - cardinal - character).