In our previous episode, Michael Schnell said:

With Lazarus on Linux, I did some simple tests with UTF strings. I found that the length of an "AnsiString(CP_UTF16)" is given in terms of bytes and not of Words. Is this like it should ?

Yes. Afaik that is not a sane combination, but Delphi compatible.

I found that pchar(s8) with an UTF-8 string works as expected, giving a pointer to the UTF-8 encoded byte array. Anyway: is it obvious, that the encoding of pchar is UTF-8 ? Is this portable ?

pchar should give access to the raw data of the default string type. (be it still 8-bit as in FPC, or 16-bit in Delphi).

p16 = pchar(s16) with an UTF-16 gives a pointer to the first byte of the word array, so (with ASCII text), the second byte is zero, thus a C-String length 1. Is this like it should ?

Yes. This is not sane code (even if you want e.g. the lower byte, this is not endian safe), since s16 is currently not the default string type

Of course re-assigning p16 to an UTF-16 string does not reproduce the original string. What encoding is to be supposed for a pchar ?

pchar's provide access to memory with the granularity of the default string type. Whatever that is, 8-bit or 16-bit, and in whatever encoding it is stored.

When converted to something else, the default system encoding for the corresponding default string is probably used.

To force 8 or 16 bits one should use pansichar or pwidechar. This is Delphi compatible.

The Debugger does not show UTF-16-Strings correctly (it shows the same result as pchar() ). Is this just a Lazarus problem, or does FPC need to provide additional support for this ?

No idea. Both are possible.

With Lazarus on Linux, I did some simple tests with UTF strings.

I found that the length of an "AnsiString(CP_UTF16)" is given in terms of bytes and not of Words. Is this like it should ?

I found that pchar(s8) with an UTF-8 string works as expected, giving a pointer to the UTF-8 encoded byte array.
Anyway: is it obvious, that the encoding of pchar is UTF-8 ? Is this portable ?

p16 = pchar(s16) with an UTF-16 gives a pointer to the first byte of the word array, so (with ASCII text), the second byte is zero, thus a C-String length 1. Is this like it should ?
Of course re-assigning p16 to an UTF-16 string does not reproduce the original string.

What encoding is to be supposed for a pchar ?

The Debugger does not show UTF-16-Strings correctly (it shows the same result as pchar() ). Is this just a Lazarus problem, or does FPC need to provide additional support for this ?

-Michael

.

Recent GDB should support direct display of 16-bit strings using
x /ws @StringVar
or
x /sw StringVar
depending on the type of StringVar ...

I have no idea if this is supported by Lazarus...

Pierre Muller
Pascal language maintainer for GNU Debugger GDB

Marco van de Voort wrote

When converted to something else, the default system encoding for the corresponding default string is probably used. To force 8 or 16 bits one should use pansichar or pwidechar. This is Delphi compatible.

Hmm.

( Would it not be more "sane" to always use the default system encoding (i.e. UTF-8 in Linux) for pchar and auto-convert the string content ? )

After writing this, I suppose this is not possible, as there is no data area where the converted information should be stored. So we are out of luck wanting a sane solution :( .

-Michael

In our previous episode, Michael Schnell said:

( Would it not be more "sane" to always use the default system encoding (i.e. UTF-8 in Linux) for pchar

That depends on how the default string type is defined. Discussions about that is postponed.

and auto-convert the string content ? )

pchars are generally not string types. If you need that, you shouldn't be using pchars for it.

After writing this, I suppose this is not possible, as there is no data area where the converted information should be stored. So we are out of luck wanting a sane solution :( .

First define a solution for what and why.

AFAIK (I'd need to start Delphi for this) AnsiString(CP_UTF16) is not allowed in Delphi.

Regards,
Sven

Ok... one can use the UTF16 code page, but Delphi does NOT define CP_UTF16 (I needed to use 1200, which is the code page ID used by Microsoft).

And the following code:

type
   TestString = type AnsiString(1200);
var
   s: TestString;
begin
   s := #$3428#$3828;
   Writeln(Length(s));
   Readln;
end;

Writes "0" while the same code with AnsiString(CP_UTF8) writes "6".

Regards,
Sven

Marco van de Voort wrote

First define a solution for what and why.

I don't suppose this makes sense, as the Delphi behavior needs to be mimicked.

-Michael

In our previous episode, Michael Schnell said:

I don't suppose this makes sense, as the Delphi behavior needs to be mimicked.

It doesn't make sense if your ultimate object is to get FPC changed or amended.

It does make sense if you want your problem analysed and want alternate suggestions.

I am just watching the proceeding of FPC and doing some test when I have some spare time to do this.

Of course I would like to help making FPC better and - if possible - more sane than Delphi. But I do see the limits.

-Michael

Which Delphi version supports AnsiString(CP_UTF16)??? At least XE and successors only support byte-char (AnsiChar) codepages with AnsiString and RawByteString.

DoDi

In our previous episode, Hans-Peter Diettrich said:

There is a difference between Delphi accepting it and doing something sane.

FPC afaik also accepts and afaik also doesn't do anything sane, so that is compatible :-)

Btw in this thread it was already mentioned that Delphi doesn't define CP_UTF16, but one easily could define it and create the above code.

Hans-Peter Diettrich wrote

Which Delphi version supports AnsiString(CP_UTF16)???

What is the suggested way to handle UTF-16 with Delphi ?

-Michael

In our previous episode, Michael Schnell said:

> Which Delphi version supports AnsiString(CP_UTF16)??? What is the suggested way to handle UTF-16 with Delphi ?

Unicodestring. If you don't know that, please reread all messages with "unicode" in the subject of the last two years :-)

Marco van de Voort wrote

Unicodestring.

Of course this is "suggested" regarding the history of Delphi, before the variable string type had been introduced.

But again, even the name now is ambiguous, as there are several Unicode encoding types and they seem to be available with the new string type. (This is what this discussion is about.) If UnicodeString forces UTF-16 encoding, why not use a more "speaking" name ?

And again, this seems not "sane", again (so it maybe is questionable if it makes sense to adopt the paradigm to FPC) . Why provide yet another type of strings, if we have a string type that is able to hold 1- 2- and 4- byte based encoding types including any variants of those ?

-Michael

Marco van de Voort wrote

In our previous episode, Michael Schnell said: > I found that the length of an "AnsiString(CP_UTF16)" is given in terms > of bytes and not of Words. Is this like it should ? Yes. Afaik that is not a sane combination, but Delphi compatible.

This really is absolutely insane, as in Delphi (I only have "Turbo"), the length of a WideString is given in terms of word and not of bytes.

So why introducing major differences in the handling of 16 bit encoded Strings such as
- AnsiString(CP_UTF16) (provided CP_UTF16 is previously defined to be 1200)
- WideString
- UnicodeString (I can't test this)
?

I feel that much of the Delphi way of handling the "new string"s is hazardous and not worth mimicking and better ways can be found - there have been long discussions on this - (while of course for compatibility, it is important to provide the Delphi-way behavior forced by a compiler option).

-Michael
(BTW.: I still don't know whether Delphi even introduces the insanity that when assigning, the auto-conversion is decided regarding the actual internal dynamical encoding of the target or if the target's statically given Type name is always used.)

This is absolutely sane, because Length() does and always did reflect the number of *elements* of an array, and strings are treated like arrays.

So why introducing major differences in the handling of 16 bit encoded Strings such as - AnsiString(CP_UTF16) (provided CP_UTF16 is previously defined to be 1200)

This one is not supported by Delphi (XE).

- WideString

This one is required for WinAPI compatibility.

- UnicodeString (I can't test this)

That's the new generic string type.

- UCS4String
For people who like it ;-)

The Delphi devlopers have learned what's a good and what's a bad stringhandling model, based on their experience with the Delphi implementation and use. You instead argue based on half-baked ideas, so please stay away from calling other models "insane" :-]

You should understand the difference between (temporary) strings, used in string expression evaluation, and the assignment of an string to a strictly typed variable.

DoDi

What's insane in not accepting (compiling) insane source code? ;-)

FPC afaik also accepts and afaik also doesn't do anything sane, so that is compatible :-)

What *also*?

Btw in this thread it was already mentioned that Delphi doesn't define CP_UTF16, but one easily could define it and create the above code.

Then tell us the Delphi version that accepts an AnsiString(1200). I couldn't find any :-(

DoDi

On 01/05/2012 04:40 PM, Hans-Peter Diettrich wrote: > > Which Delphi version supports AnsiString(CP_UTF16)??? What is the suggested way to handle UTF-16 with Delphi ?

The obvious: in newer versions use String or UnicodeString :-)

Embarcadero discourages the use of AnsiString in newer versions - this string type should be used only for reading/writing existing data sources, but the application should use UnicodeStrings internally. That's okay WRT performance, no further Encoding conversions required.

When FPC implements an really polymorphic string type (of charsize 1..4), the implementors are responsible for all arising problems, like high runtime and resource overhead, PChar quirks etc. Even if the Delphi developers initially wanted to support such an string type, they later came to a strict separation of strings of a different charsize.

DoDi

I don't know about Delphi, but in FPC the UnicodeString type existed before the code page aware String type was merged (the first release with it was 2.4.0 AFAIK). Also there is the usual reason for unintuitive implementations: Delphi compatibility.

Regards,
Sven

Ehm... Delphi XE DOES accept AnsiString(1200) as I've written in my other mail. It just doesn't do anything sane with it (Length of such a strings seems to be always zero).

Regards,
Sven

Hans-Peter Diettrich wrote

You instead argue based on half-baked ideas, so please stay away from calling other models "insane" :-]

I never promoted that the compiler should force us to _always_ make use of the features that are provided by dynamically encoded strings. I only promote the idea that if FPC indeed implements dynamically encoded strings, we should provided with the possibility to make fully use of them if we choose to do so (e.g. by writing a (library) function that can accept any encoding type and does not do any conversion, neither on entry or exit, nor while working with the strings).

If used decently this will not slow down, but in many cases speed up the code.

Of course this right now is half-baked, as the discussion about the functionality in detail had been killed with the argument that Delphi dopes not provide this.

The insane stuff mentioned here was the Length of a AnsiString(1200) that Swen claimed Delphi XE calculates (I can't test it). and this obviously is insane (or half-baked) in Delphi.

-Michael

You're right, the XE compiler lacks some error checks :-(

DoDi

Hans-Peter Diettrich wrote

You're right, the XE compiler lacks some error checks :-(

If this indeed is considered a bug in Delphi, FPC _could_ in fact in a more sane way provide the length of an "AnsiString(CP_UTF16)" in terms of Words (i.e. UTF codes, as done with UTF8),

-Michael

In our previous episode, Michael Schnell said:

An ansistring is always 8-bit. Nothing can be done there, except warn/error if 1200 is used.

Marco van de Voort wrote

An ansistring is always 8-bit.

Sorry I can't follow here.

Of course the term "ANSI" suggests 8 bit, but it also suggest one visible character = 8 bit, thus non UTF.

If a type called "ANSI..." is used to hold UTF codes, the term ANSI is abused anyway and now the handling of the type can be defined in any way that seems appropriate,

-Michael

ANSI also covers MBCS (DBCS).

If a type called "ANSI..." is used to hold UTF codes, the term ANSI is abused anyway and now the handling of the type can be defined in any way that seems appropriate,

For legacy reasons "Ansi" means types with 8 bit AnsiChar(!), in contrast to WideChar or other Char sizes.

DoDi

P.S.:

I found here
http://en.wikipedia.org/wiki/Windows_ANSI_code_page#ANSI_code_page that there in fact

"The following Windows code pages exist:The following Windows code pages exist:

...

65000 UTF-7
65001 UTF-8
"

But this seems to be be a propriety Microsoft definition while AFAIK, "ANSI" denotes "American National Standards Institute".

- Michael

In our previous episode, Michael Schnell said:

No, it means that the encoding granularity is 8-bit. Length returns encoding granularity, not codepoints (always 32-bit, encoded in sequences of 8 (ansistring) or 16 (widestring,uncidoestring) bits) or printable characters (possibly multiple codepoints)

If a type called "ANSI..." is used to hold UTF codes, the term ANSI is abused anyway and now the handling of the type can be defined in any way that seems appropriate,

Whatever the name is, in all current Unicode Delphi versions and FPC ansistring means 8-bit string exclusively.

Marco van de Voort wrote

Whatever the name is, in all current Unicode Delphi versions and FPC ansistring means 8-bit string exclusively.

OK so the definition of

"AnsiString(CP_UTF16)" in FPC and
"AnsiString(1200)" in Delphi means

"an 8 bit string with data Bytes representing UTF-16 codes in low-byte-First notation".

This of course is not very straightforward, sane or portable but not illegal and not ambiguous either.

(Of course this definition should be explicitly given somewhere)

-Michael

But this seems to be be a propriety Microsoft definition while AFAIK,

"ANSI" denotes "American National Standards Institute". While ANSI does denote american national standards institute in general it doesn't really mean that in this context.

A windows machine has two main code pages in use (both language dependent and for some languages they may be the same code page). The "OEM" code page and the "ANSI" code page. The "OEM" code page is one of the original PC code pages and afaict is mostly used for the console. The "ANSI" code page is used for the non-unicode versions of stuff in windows itself.

The term "ANSI" comes from the fact that the initial "ANSI" code page (1252) was based on an ANSI draft of what became ISO-8859-1. 1252 is fairly close to ISO-8859-1 (it just replaces rarely used control characters with more printable characters) but most of the other "ANSI code pages" bear little to no relationship to any ANSI or ISO standard encoding.

Afaict in europe, america and australasia both the "ANSI" and "OEM" code pages are simple encodings with one byte per user-visiable character and all characters drawn left to right. Once you move to asia and africa though that no longer holds with CJK languages being represented by multibyte encodings, vietnamese being represented using combining characters and middle eastern languages bringing the complications of bidirectional text. MS encourages programmers to use unicode nowasays and afiact languages added more recently to windows (like the indic languages) don't have any non-unicode also defines other code page numbers that are used as neither ANSI or OEM code pages. UTF-8 falls into this category.

Delphi is a windows program (yeah there was an abortive linux port but that came much later and didn't stick arround for long) so it inherits windows terminology. FPC/lazarus is essentially a delphi clone but is cross platform so it's put in the position of trying to interpret and stretch windows grounded ideas to fit a cross-platform context.

So it's me who did go insane. But at least I now know why.

-Michael

Nothing sane :-(

DoDi