DXE Strings again :-) ,

What is the relation between Generic Functions and (static) DXE String encoding schemes ?

Can you (e.g.) do a (silly) Generic Function that takes two strings of any (but identical) encoding type, concatenates them and outputs the result in the same (static) encoding type, so that you can call the beast to handle the encoding type specified by the caller from "outside" (e.g. in another unit) without unnecessary conversions.

If this is possible, how is it implemented ?

Could a single "Generic TStringlist" be done that way to allow for storing and retrieving Strings of any - but ubiquitous - (static) encoding Type ?

-Michael

In our previous episode, Michael Schnell said:

DXE Strings again :-) , What is the relation between Generic Functions and (static) DXE String encoding schemes ?

Afaik none. Constraints are interfaced based, and strings are no objects.

"Michael Schnell" wrote

DXE Strings again :-) , What is the relation between Generic Functions and (static) DXE String

encoding schemes ?

Can you (e.g.) do a (silly) Generic Function that takes two strings of

any (but identical) encoding type, concatenates them and outputs the result in the same (static) encoding type, so that you can call the beast to handle the encoding type specified by the caller from "outside" (e.g. in another unit) without unnecessary conversions.

Why would you want such a function? If the two strings have the same encoding and the destination is a string variable of the same static encoding then there won't be any conversions anyway.

Could a single "Generic TStringlist" be done that way to allow for

storing and retrieving Strings of any - but ubiquitous - (static) encoding Type ?

And then? If you want to use them in e.g. a class you need to pick a specialization and thus a code page at compile time.

Regards,
Sven

Sven Barth wrote

If you want to use them in e.g. a class you need to pick a specialization and thus a code page at compile time.

A project might contain different part that handle differently encoded strings. Thus a central functionality to store and retrieve strings independent of the actual encoding seems very appropriate.

-Michael

"Michael Schnell" wrote

On 01/16/2014 12:04 PM, Sven Barth wrote: > > If you want to use them in e.g. a class you need to pick a

specialization and thus a code page at compile time.

A project might contain different part that handle differently encoded

strings. Thus a central functionality to store and retrieve strings independent of the actual encoding seems very appropriate.

Then TFPGList<UTF8String>, TFPGList<UnicodeString>, etc. should be sufficient for now at least as long as one only needs to store a list of strings.

Regards,
Sven

..Continuing the never ending discussion ...

[ In one ore more other threads here I showed that (IMHO) it would be most sensible (and in fact not performance killing (!) ) way out of the multiple (statically defined) "new" String types with different encoding (with auto-conversion) to additionally support a fully dynamically encoded string type (with auto-conversion) - which needs some compiler magic - and use same as the base type for TStrings and it's siblings. ]

The growing popularity of "Generics" might suggest to call this String encoding "Generic", too, (to easily tell it from the Delphi-introduced "Raw", and the dedicated encoding types).

Regarding arguments for it's usefulness I came across that the Lazarus API currently forces TStrings to be UTF8 encoded while Delphi currently forces TStrings to be UTF16 encoded. IMHO the only way to prevent disastrous compatibility problems is to do TStrings being "Generic"encoded and with that allow for decent portability for both.

-Michael

"Michael Schnell" wrote

..Continuing the never ending discussion ... [ In one ore more other threads here I showed that (IMHO) it would be

most sensible (and in fact not performance killing (!) ) way out of the multiple (statically defined) "new" String types with different encoding (with auto-conversion) to additionally support a fully dynamically encoded string type (with auto-conversion) - which needs some compiler magic - and use same as the base type for TStrings and it's siblings. ]

The growing popularity of "Generics" might suggest to call this String

encoding "Generic", too, (to easily tell it from the Delphi-introduced "Raw", and the dedicated encoding types).

Regarding arguments for it's usefulness I came across that the Lazarus

API currently forces TStrings to be UTF8 encoded while Delphi currently forces TStrings to be UTF16 encoded. IMHO the only way to prevent disastrous compatibility problems is to do TStrings being "Generic"encoded and with that allow for decent portability for both.

Problem: there is (currently) no string type that can handle ANSI, UTF-8 and first two are handled by AnsiString and the third by UnicodeString. And those two are not equal which would be important for overrides/overloads/inheritance. Without that your whole idea is useless.

Regards,
Sven

In our previous episode, Sven Barth said:

Problem: there is (currently) no string type that can handle ANSI, UTF-8 and first two are handled by AnsiString and the third by UnicodeString. And those two are not equal which would be important for overrides/overloads/inheritance. Without that your whole idea is useless.

Moreover, will operations that use character access make sense at all if you don't know what the actual encoding is? (not only s[] but also pos,delete,insert etc). The same code can seem to behave differently because different code-paths make the same parameter have different encodings.

It will require a "forceencoding()" function etc, which is incompatible with all current stringcode, and frustrate the general no conversion spirit which is the main advantage.

And of course it is totally incompatible with Delphi.

Of course this only is relevant when "New Delphi" (i.e. "partly" dynamically encoded) Strings" are introduced (I decline to use the terms "AnsiString" and "UnicodeString" due to ambiguity, unless it comes with a clear definition close by).

Here, The Delphi model does not provide a String encoding type (and appropriate "compiler magic") that can be used for that purpose (i.e. "fully dynamically encoded").

Hence the fpc team _might_ decide to enhance the Delphi imposed model appropriately and with that help the Lazarus team to provide a more Delphi compatible - even though OS independent - API for the LCL.

-Michael

In our previous episode, Michael Schnell said:

I think it is better to start implementing it in a fork. A fork can move much faster and make a reconnaissance of all problems and find solutions for them, and see what you can do to improve compatibility.

One the fork has a functioning Lazarus again, you will have a much better bargaining position than now while it is all paper, and some people are strongly in favor, and the others say it won't work.

As long as not stated differently AnsiString and UnicodeString are meant as implemented in FPC trunk.

Here, The Delphi model does not provide a String encoding type (and appropriate "compiler magic") that can be used for that purpose (i.e. "fully dynamically encoded").

Basically it does. In theory the additional record prepended to each string (wich contains the reference count among others) could be used for 1-, 2-, 4- or multi-Byte strings as it carries a "ElementSize" field which is currently fixed to 1 for AnsiString (even with UTF-8) and to 2 for UnicodeString (both strings use the same record layout though they are declared as different ones). Also there is the StringElementSize function which is overloaded for RawByteString and UnicodeString and which already returs the value of ElementSize. So purely in theory the current AnsiString type would already be capable enough. Also the compiler might already handle overloads correctly if we'd have a (for now hypothetical "AnsiString(UTF16)" (which would be equal to UnicodeString)). One of the problematic parts that already Marco mentioned is character access. A possible solution here would be to force the character size depending on the declared string type (2 for AnsiString(UTF16), 4 for AnsiString(UTF32), 1 for any 1-Byte AnsiString encoding and either 1 or 6 for UTF-8 (6 is the maximum number of Bytes that UTF-8 might encode a character with, but it's currently the maximum used is 4)) and not depending on the runtime type. The compiler would then either need to insert approbiate conversions if the runtime type does not match the declared type (for whatever reason) or the compiler would need to assume that the runtime type always matches the declared type. In the former case this might be quite some performance penalty (this could be avoided if the compiler would create approbiate inline code for detecting the runtime encoding).
An open problem left would be RTTI as there currently are tkUString (for UnicodeString) and tkLString (for AnsiString) of which the second contains a codepage field while the first does not. And to keep Delphi code as much compatible as possible the compiler would then again need to handle the RTTI of a AnsiString(UTF16) differently...

Regards,
Sven

Marco van de Voort wrote

Moreover, will operations that use character access make sense at all if you don't know what the actual encoding is?

The administrative record of each "New Delphi string" contains the encoding type and the byte-count for each code. So "you" (the compiler and the RTL) do know it.

The "only" shortcoming in Delphi is that the handling is completely "static":
- if the encoding definition of the type the string is created with is not "RAW", the encoding needs to be known at compile time (i.e. the encoding type is not allowed to be modified at run time)
- if the encoding definition of the type the string is created with is "RAW", auto-conversion from this string to a non-RAW is not done.

Hence (including - but not only - for decent use on multiple OSes) an additional "fully dynamically encoded" type (I suggest to call the type of this Strings "Generic") is necessary.

(not only s[] but also pos,delete,insert etc). The same code can seem to behave differently because different code-paths make the same parameter have different encodings.

I suppose that you are right. But not only the "funny" position numbers pos(), delete(), insert() and friends use, create a problem, but also the the String type they are defined to use does:

- If using any statically encoded type for same, it is close to impossible to create decently fast programs for string manipulation (unless they by chance use the correct encoding type), as auto-conversion to and fro is invisibly introduced.

- If using the suggested dynamically encoded type, we will have problems when combining strings of different types in a code snippet that calls these functions.

I don't know if / how / to_what_extent compiler magic can help here (doing auto-conversion "when necessary" similar to when simply assigning strings of different encoding types).

In the end, I feel it would be very un-desirable but might be the only "easy" solution to go with full Delphi compatibility and handle all strings encoding but UFT16 in a very un-decent way. This would force Lazarus to provide a (Delphi compatible) LCL-API completely done with UTF16. This completely contradicts all they did in the last few years :-) .

-Michael

It is possible as some mails a few months back have shown. If you have a function that has a RawByteString return type then you can assign that to a string of any static encoding even if the dynamic encoding is not the same (and then static and dynamic encoding of the string variable will not match, nevertheless the RTL and the compiler will handle them correctly (at least I hope so ^^).

The non-conversion nature of RawByteString has its use. Namely to implement the conversion mechanism of any dynamically encoded type (like the current AnsiString is) without the need to implement yet another special construct.

Regards,
Sven

In our previous episode, Michael Schnell said:

Yes, but the point is that the programmer that writes the index into the delete() might not oversee all cases. (codepaths that can bring a string to the current point).

Keeping in mind that these functions generally work at the encoding granularity level. So something that works with unicode (utf16) will fail with ansi (utf8) if it contains values that utf8 can't encode in one byte.

- If using any statically encoded type for same, it is close to impossible to create decently fast programs for string manipulation (unless they by chance use the correct encoding type), as auto-conversion to and fro is invisibly introduced.

Well, the trick is not to leave that to chance.

I see that trying different kinds of "Delphi compatibility plus OS-independence" (which obviously is a contradiction in itself) makes a lot of sense for the fpc / Lazarus combination.

But what do you mean by "fork" regarding the compiler and RTL (which would be the prerequisite for any decent LCL implementation).

A) Doing a "Generic" (fully dynamically encoded) String type.

Here, IMHO regarding the compiler a fork is not necessary, as this is a pure addition. If you don't like it, just don't create any string with that type. Defining the type of TStrings (and some other RTL introduced objects and functions would be a simple "ifdef" thingy. Of course the implementation of the functionality is mostly a pure addition, as well, as the actual type needs to be detected and handled appropriately (easy with TStringList, but potentially complex for stuff like TMemo, no idea (yet) with pos(), copy() and friends).

B) another idea of forking on behalf of Lazarus could be to do a "Delphi compatible" branch with a completely UTF16 based RTL (and compiler magic) and a "Lazarus compatible" branch with a completely UTF8 based RTL (and compiler magic)

-Michael

I don't have any Unicode Delphi so I can't test that and as far as I am informed the implementation of "new Delphi strings" in the fpc RTL is still a moving target.

In many discussions I was told that the Delphi "RAW" type does not provide the features necessary to a fully dynamically encoded String, even though when I first read the online Delphi docs, I had the impression that it should.
.

The non-conversion nature of RawByteString has its use.

I don't doubt is why I don't suggest to modify it's behavior but to introduce yet another String encoding type.

The functionality for this is already in place in the (future) RTL, only the (really small) compiler magic to call conversions "when necessary" needs to be added and called if a string of this type definition is to be handled. (Easy with pure assignments, as discussed in that thread pos(), copy() and friends need some more re-thinking.)

-Michael

Marco van de Voort wrote

Yes, but the point is that the programmer that writes the index into the delete() might not oversee all cases.

You are absolutely right.

But since Unicode is usable in Strings, a "normal programmer" is not really allowed to "write an index" in pos(), delete(), and MyString[i]. To do this a decent understanding of Unicode is necessary (this has been discussed at any length in multiple forums)

The only allowable thing is to use the "index" returned by "pos" in copy() and friends on the same string. This works (mostly) as expected for any encoding.

Well, the trick is not to leave that to chance.

I don't understand. When using Lazarus (if it stays similar to what it is now) your strings will be UTF8. If pos() requires UTF16 you are out of luck.

-Michael

In our previous episode, Michael Schnell said:

Fixing that in Lazarus is easier than implementing your case in the compiler.

Sven Barth wrote

As long as not stated differently AnsiString and UnicodeString are meant as implemented in FPC trunk.

Can you point me to a _final_ documentation on this ?

> Here, The Delphi model does not provide a String encoding type (and > appropriate "compiler magic") that can be used for that purpose (i.e. > "fully dynamically encoded"). Basically it does.

Yep.

In theory

:-)

Of course I know how this works (otherwise this discussion would not make any sense at all) that the implementation of the string record allows for this, but Embarcadero does not allow the users to decently take advantage of it.
- "non-Raw" does not allow for modifying the type and code-length fields
- "Raw" does not provide auto-conversion (seemingly for good reasons)

-Michael

Marco van de Voort wrote

Fixing that in Lazarus is easier than implementing your case in the compiler.

Maybe. (Mseide shows how this can be done even without "New Delphi Strings").

But in many discussions I learned that the Lazarus team is not at all happy with the idea to do a UTF16 LCL (API).

And IMHO a versatile compiler and RTL should be able to handle any string encoding type (that is offered to be defined) decently and not (like Delphi) to allow for defining a - say - UTF8 or "European 8 Bit ANSI" String and not provide the normal RTL functions in a way that allows for using it in a fast program.

If multiple string types are available the user programmer should be free in choosing any of them according to the

In our previous episode, Michael Schnell said:

I see that trying different kinds of "Delphi compatibility plus OS-independence" (which obviously is a contradiction in itself) makes a lot of sense for the fpc / Lazarus combination.

I don't really see it. I'm not even sure if I would like it, even if there was no Delphi compatibility to consider.

But what do you mean by "fork" regarding the compiler and RTL (which would be the prerequisite for any decent LCL implementation).

Well, since the choices for FPC have been made, you were working on your own version, a bit like MSEGUI's Martin Schreiber?

A) Doing a "Generic" (fully dynamically encoded) String type. Here, IMHO regarding the compiler a fork is not necessary, as this is a pure addition.

It is necessarily, since you must show it can actually work. Since you want a decision as much as possible, I wouldn't work in a branch but in a separate fork, and just hack it to run on say Linux and Windows. Just as a proof of concept.

If you have a working proof of concept it is much easier to convince people to give it a go. Much easier than with just two paragraphs of descriptions fragmented over 250 unicode discussion messages.

B) another idea of forking on behalf of Lazarus could be to do a "Delphi compatible" branch with a completely UTF16 based RTL (and compiler magic) and a "Lazarus compatible" branch with a completely UTF8 based RTL (and compiler magic)

single source two releases, or two source two releases?

Marco van de Voort wrote

If you have a working proof of concept ...

As this would need some (even if not much) compiler magic, I will never be able to provide something like this. Here the help of someone who understands the internal working of the compiler is necessary.

If nobody is inclined to do such an enhancement, we will never see it :-( .

-Michael

Marco van de Voort wrote

a bit like MSEGUI's Martin Schreiber?

No.

AFAIK, MSEGUI uses the fpc compiler out of the box and "only" provides an alternate library and IDE.

Martin did draft a new fpc-alike compiler (as discussed in his mailing list) but he declined to implement "Delphi News strings" of any kind.

-Michael

Marco van de Voort wrote

single source two releases, or two source two releases?

Hmm. I suppose also compiler magic is affected and thus the compiler would needed to be re-compiled for either release.

-Michael

The string type itself is fully functional as far as I know. There might still be a few bugs regarding Delphi compatibility here and then, but that's nothing new.

There is no special RAW type. There is only AnsiString and the RawByteString is merely a AnsiString(CP_NONE).

> The non-conversion nature of RawByteString has its use. I don't doubt is why I don't suggest to modify it's behavior but to introduce yet another String encoding type.

No. I would not introduce yet another string type, but try to merge the existing string types instead as they all share at least the logic/concept while not the implementation.

Regards,
Sven

Michael Schnell wrote

On 02/28/2014 03:00 PM, Sven Barth wrote: > As long as not stated differently AnsiString and UnicodeString are > meant as implemented in FPC trunk. Can you point me to a _final_ documentation on this ?

Work on final documentation will be started when 2.8/3.0 will be released. Till then wiki.freepascal.org/FPC_Unicode_Support can be considered the documentation. Please note that the non existance of a documentation does not mean that a feature is not considered final. And the new string type is working as it should, what's still left is to adjust the RTL for more Ansi-/UnicodeString awareness.

Yes, but you can always do it by converting to RawByteString:

=== code begin ===

function ChangeCodePage(aStr: RawByteString; aCodePage:
TSystemCodePage): RawByteString;
begin
   Result := aStr;
   SetCodePage(Result, aCodePage, False);
end;
var
   s: UTF8String;
begin
   s := 'Hello World';
   Writeln(StringCodePage(s));
   s := ChangeCodePage(s, CP_ACP);
   Writeln(StringCodePage(s));
end.

=== code end ===

=== output begin ===

65001
0

=== output end ===

Regards,
Sven

Sven Barth wrote

The string type itself is fully functional as far as I know.

Yep.

The meaning of "fully functional" is what we are discussing here.

No. I would not introduce yet another string type, but try to merge the existing string types instead as they all share at least the logic/concept while not the implementation.

When I started thinking about the issue of how to make the "New Delphi Strings" decently work in a multiple OS environment, I mentally tried this, but I doubt that this is possible while maintaining decent Delphi XE compatibility.

-Michael

Its hard to allow anybody to help deciding what "as it should" exactly means, what the "as it should" is not publicly documented.

Of course if "as it should" is agreed to be "nothing but Delphi XE compatible" no discussion makes any sense.

-Michael (sorry if sounding harsh, but that is what simple logic dictated)

Of course but then there is no auto conversion any more.

The start of the discussion was using a type that _both_ allows for modifying the type and code-length fields _and_at_the_same_time_ provides auto-conversion (e.g.) as the base for TStrings to (e.g.) allow Lazarus to do an LCL API that (e.g.) works with UTF16 encoded strings on Windows and with UTF8 encoded strings in Linux without the need to re-compile the compiler and/or the RTL with different options.

-Michael

Michael Schnell wrote

(sorry if sounding harsh, but that is what simple logic dictated)

.. And if you think that I am a troll, as I keep nagging on something I technically can't help to make it happen (because I am not knowledgeable enough to do any modifications to the compiler even in "testing mode", I can't contradict :-(

-Michael