Here, there have been lots of long winding and partly quite fruitless discussions on the implementation of the new Unicode aware string type(s).

IMHO, before trying to decide regarding any implementation details, there should be a _very_explicit_ decision on the general functionality.

Here, I would suggest to first decide between the (IMHO) three sensible mutually exclusive variants:

A)
Have the new string type(s) maintain the very strict typing paradigm Pascal usually imposes. This IMHO asks for multiple explicit string types such as ANSI_1252_String, UTF8_String, UTF16_String, etc plus the appropriate single-character types. This of course would allow for automatic conversion without ambiguity. As the types are mutually exclusive, function calls always will do automatic conversions, unless the appropriate overloaded function is defined. These string types of course will not include fields denoting their encoding and byte count per code element.

B)
Only do a single string type that is decently dynamically typed. These strings of course will include fields denoting their encoding and byte count per code element. Here conversions will happen when two differently typed strings are combined in some operation. An empty string would be handled as having no predefined encoding, so that combining it with any other string will not force a conversion. As there is only one string type, function calls will never trigger a conversion and the encoding of the function result is not predefined. Of course a single character type is defined that can hold any encoding and supposedly will be done in a way that it in fact is dynamically encoded as well. To enforce that a string is provided in some definite encoding, appropriate function (or compiler magic) is available.

C)
Handle these strings similar to Pascal-Objects that allow for inheritance (and provide the appropriate operator-overloading). So there is a "Parent" string type (aka RAW) that has no predefined encoding and multiple "Child" types that define different encoding enforcements. As the parent type of course needs to include fields denoting their encoding and byte count per code element, the child types of course are implemented in the same way (which in theory allows for "intersexual strings that feature data encoded correctly but differently than the type denotes). Here (for non-intersexual strings) conversion can be handled in an unambiguous ways (using either the type <if it is not the "Parent" type> or the dynamically given encoding) and a non RAW target type of ":=" might request for yet another conversion. A function definition can either use the Parent type (RAW) and so will not trigger a conversion when being called or use one of the Child string types and trigger an appropriate automatic conversion. Of course single character types matching the Parent and all Child string types are necessary.

While neither A nor B is Delphi XE compatible in any way, C seems a bit similar to what Emb does. But AFAIK, Delphi does not provide an unambiguous, well defined and understandable paradigm (such as a Object-like Parent/Child relationship) for the features of the different string types. So the FPC team should be free do do a decent definition.

-Michael

Huh? As far as I understand it B is the closest to Delphi's and current trunk's implementation.

Regards,
Sven

In our previous episode, Sven Barth said:

It's a mix of all three actually. It is typed(A), there are two (B) implementations, and the memory layout of both implementations have similarities that makes Rawbytestring possible, making rawbytestring the base memory representation the others "inherit" from (C).

Finding (C) is a bit of a stretch though. A and B are way more true than C.

A is from the type system view. B is from the implementation (runtime helpers) view. C only for rawbytestring (which is very, very limited and way overestimated in all these discussions)

In that case it seems that I have a very implementation centric point of
view :P

Regards,
Sven

What do you mean by "true". If you mean "it's more true that Delphi works similar to that" the question, is if it is really a good design goal to do an implementation that is closest possible to what DXE provides, regarding that (from what I read, I don't have XE to test it) the XE definitions to me seem like a rather chaotic and hardly understandable mix of strong typing and dynamic encoding.

A is from the type system view.

Yep: no dynamic typing. (As there is no dynamic encoding this obviously is not at all Delphi XE compatible, neither regarding the definition nor regarding the implementation.) A RAW type, allowing for multiple encoding types, is not possible here.

B is from the implementation (runtime helpers) view.

Nope it's just a definition of a pure dynamically typed string definition system. Implementation details were only mentioned to try to make clear what I meant. (As there is only one String type and there is no way to have the name of a type denote the encoding, this - regarding the definition - obviously is not at all Delphi XE compatible.)

C only for rawbytestring (which is very, very limited and way overestimated in all these discussions)

Nope. C defines the other new types as object-alike children of a general "RAW" type. Here we have as well a "RAW" type, as multiple types the names of which suggest a certain encoding the definition. This seems to be rather similar to Delphi XE's. The implementation might be done compatible to what XE does, but the intention of my post was to suggest that _first_ a decent (strict, understandable, unambiguous) definition should be done (as "Delphi-XE-compatible" does not seem to provide an understandable straight forward wording) and _afterward_ evaluate the implementation (i.e. take a look at doing how to improve what already has been done).

-Michael

In our previous episode, Michael Schnell said:
[ Charset ISO-8859-1 unsupported, converting... ]

Yes, and FPC/trunk.

Then reread the discussions on the list about unicode. In the past few years it has been explained enough. And before D2009 came out there was a lot of discussions about how we would tackle it if it were a black slate.

Then there were fully dynamic encoding schemes proposed too (e..g by Florian).

Correct. See the discussions mentioned above (say roughly march-may 2008) for more info.

I'm not sure what you meant. I only saw what you wrote. You created three classes of stringtypes, but in reality the implementation is hybrid of the first two (the (C) bit was a bit a joke, since rawbytestring is so limited)

As far as I know C has no object type. What you want with the whole (C) branch is a complete mystery to me. That's mostly caused because you describe proposals with very general terms like "OO" and "paradigm" and vague references to other languages.

If you want to make proposals, make them clear, self contained, to the point and coherent, see also

http://www.freepascal.org/faq.var#extensionselect

Marco van de Voort wrote

Then there were fully dynamic encoding schemes proposed too (e..g by Florian).

I do know this. There seems pros and cons have been discussed, but no real decision has been done (i.e. a strict independent understandable definition of what is supposed to be provided from the view of the user
- not from the view of the implementation).

I'm not sure what you meant. I only saw what you wrote. You created three classes of stringtypes, but in reality the implementation is hybrid of the first two (the (C) bit was a bit a joke, since rawbytestring is so limited)

Sorry if I have not been clear enough. I suggested three (mutually exclusive) alternative ways to define a (supposedly clearly definable and workable) family of String types (OK, "B" suggests a one-member-family).

As far as I know C has no object type.

Correct. That is why I wrote "object-alike". This wording only illustrates that there is a kind of Parent-Child relationship between the RAW string type and the types supposed to hold a specific encoding. With this in mind, it is obvious that when a function uses a "RAW" as a parameter, no conversion is done, as RAW "is" any specific String type.

What you want with the whole (C) branch is a complete mystery to me.

Obviously a single fully dynamically encoded string type (such as B) is not what everybody (but Florian an myself) wants, (e.g. because EMB decided implementing multiple types with different names).

Obviously a system of hard coded string types (such as A) is not what everybody (but some) wants (e.g. as there would need a lot of such types and because EMB decided implementing dynamic typing).

Obviously there is a request for variables that don't require a dedicated encoding (RAW) _and_ for variables that define their encoding by the name of their type.

So IMHO the Parent-Child (alike) relationship between RAW and any other new string type is quite obvious.

Thus I feel that it is a good idea that - wherever sensible - the behavior of the string types should be similar to that of object regarding the Parent-Child relationship. This is especially true with function parameters. (Recently there has been such a "fruitless" discussion on how the different string types should behave when used as formal/actual parameters)

With this definition (that should be easily understandable for those who are trained for working with object inheritance) you can use strict typing with clearly defined effects (conversion when different type names are combined), and (using RAW) you can take advantage of the dynamic encoding (conversion "whenever necessary") with clearly defined effects as well.

The definition of what happens when RAW and it's "children" are combined also is quite obvious:

- function parameters:
- strict -> RAW: no conversion
- RAW -> Strict: conversion whenever necessary

- "combining" (compare, concat) raw and strict: conversion whenever necessary

There still are ambiguous cases:
- "intersexual" variables (strict Typed variable that happens to be correctly done but with an encoding that does not match the type name)
- RAW variables that in fact have a dynamic encoding definition of "NONE/RAW" and are to be converted into a strictly typed variable. Handling this might be considered an "implementation detail". So Delphi compatibility is not necessary.

A result of this definition is that doing e.g.
RAWs1 := STRICTs2;
is independent of the actual current dynamic encoding of the target.

This might or might not be compatible to what Delphi XE does. But IMHO this is what might be considered as "obvious", So IMHO Delphi compatibility is not necessary.

While I do think that the definitions I did in my first post were clear and self contained (but maybe I failed), I don't think it's viable to do a further elaboration on A) and B), as I feel that there is no real chance that something like this can make it into an official release (see above).

Regarding the (more complex) variant C), I hope I succeeded on making myself more clear with the text above. Again I tried not to suggest an _implementation_ but an approach to a _definition_ that finally should be clear, self-contained and easily acceptable and memorizeable for users of an object-language. (Just notice that the currently available implementation seemingly largely does what this definition requests.)

-Michael

Obviously a system of hard coded string types (such as A) is not what everybody (but some) wants (e.g. as there would need a lot of such types and because EMB decided implementing dynamic typing).

It's the best system, performance-wise. An application should not use more than one or two string types, e.g. Delphi uses the native/generic AnsiString(0), and FPC/Lazarus may add UTF-8 strings. All other Ansi encodings will only add a lot of conversions, with no added advantage.

There still are ambiguous cases: - "intersexual" variables (strict Typed variable that happens to be correctly done but with an encoding that does not match the type name)

This should never occur.

- RAW variables that in fact have a dynamic encoding definition of "NONE/RAW" and are to be converted into a strictly typed variable. Handling this might be considered an "implementation detail". So Delphi compatibility is not necessary.

Nobody wants that, except you.

DoDi

Hans-Peter Diettrich wrote

Delphi uses the native/generic AnsiString(0),

A native /generic type is exactly what is _not_ available in the A) suggestion of definitions. Here a type name stands for exactly one encoding variant. No dynamic encoding (implemented by having a variable hold a field that denotes which encoding is used) is possible.

As with a system that provides Type names and additionally does dynamic encoding such beasts _can_ occur (e.g. if you do a diabolic "move()"), So the implementation definition needs to state what has to be done if such a case is detected (e.g. ignore or fire an exception).

RAW strings are exceptionally useful.

E.G.:
You want to do a function that provides the first (ASCII-coded) number in a string.

If you do this function with a RAW parameter you don't need multiple overloaded variants of the function and the code in the function can take care of the actual encoding of the content of the string without doing any actual (time consuming) conversion, as the ASCII numerals to be extracted never need a conversion.

Moreover, in fact, strings can be used for simply holding a sequence of bytes, words or DWORDs, just to take advantage of the reference counting provided by the string types. Here of course it does not make any sense to define any encoding scheme.

In this case a "NONE" encoding-variant for the string type "RAW" is valuable.

Delphi does provide this by the encoding number $FFFF.

Moreover Delphi seems to define the encoding number $0000 as "to be done" as it determines the conversion to be done from the dynamic encoding of the target of a ":=" statement. I am not sure if this really is the case, but IMHO this is an extremely nasty implementation. IMHO the history of the target in ":=" never should affect what is done in this statement.

-Michael

In our previous episode, Michael Schnell said:

There are no decisions about what of the implemented (D2009+) types will be "string" and when (mode and/or OS, multi RTL or not).

Not about what the new stringtypes will be, they will be D2009+ compatible to a very high degree.

Well, since you say you read the original discussions, this is how they went down in my memory:

The original proposal was like (A) but only for base unicode encodings (utf8/16 and maybe 32), but went down due to either excess conversions and need for overloading. The amount of overloading for the current 3-4 stringtypes is already a bit much. (short/ansi/wide/unicodestring)

The Delphi2009 system might actually reduce overloads in 2.8 compared to 2.6 because some ansistring and unicodestring overloading versions can be combined using rawbytestring.

Moreover, overloading only works for simple functions that take a string. It doesn't solve other spots where the declaration matters, e.g. as parameters of virtual methods, VAR/out parameters, typecasting of stringtypes to pointer types, the need to add additional, non compatible members to VARIANT etc etc.

(B) was a counter proposal floated by Florian. The cons were pretty much that you had to guard every encoding sensitive routine (e.g. every API/OS call) to enforce the string contained the encoding you expected. Combining one and two byte types also cast doubt on the [] operator's performance. Some half-baked requests to let [] return codepoints were swiftly debated and rejected.

Some damage control measures wrt the guarding were proposed, mostly based on directives, which I didn't consider very convincing.

Then Yury proposed to combine A and B, in retrospect a bit like the current Delphi implementation but with one and two byte encodings in one type.

The discussion then died out, but not long after that, the embargo on Delphi 2009 was lifted, and more details crept out.

Note that the Delphi2009 definition is theoretically capable of combining one and two bytes in one type (like Yury's). Afaik there is no consensus why Embarcadero kept the two types separate, though I can think of several reasons:

- performance
- backwards compatibility (and thus the hurdle to upgrade)
- While normal code would probably work with an unified type, the big amounts of code that typecast strings, mess with temps or use strings as buffers would cause problems. Maybe they tried, and it was problematic.

> As far as I know C has no object type. Correct. That is why I wrote "object-alike". This wording only illustrates that there is a kind of Parent-Child relationship between

Explain parent-child for explicitely this context. This kind of stuff is what I meant with self contained. Don't use terms that you don't fully describe elsewhere.

the RAW string type and the types supposed to hold a specific encoding.

Explain RAW.

With this in mind, it is obvious that when a function uses a "RAW" as a parameter, no conversion is done, as RAW "is" any specific String type.

Yes, I never really considered (B) a workable solution. It would break existing code, and the ways to deal with the other problems was hackish at best.

Obviously a system of hard coded string types (such as A) is not what everybody (but some) wants (e.g. as there would need a lot of such types and because EMB decided implementing dynamic typing).

I think the A-B hybrid is better than either A or B. And that is what is being implemented.

Obviously there is a request for variables that don't require a dedicated encoding (RAW) _and_ for variables that define their encoding by the name of their type.

Then describe how that should work. What should happen if I pass such marked raw string to a function that wants encoding <y>?

So IMHO the Parent-Child (alike) relationship between RAW and any other new string type is quite obvious.

No it is not. And you don't make the situation any clearer by writing yet another message without a concrete description (either using specs or with examples), and not defining RAW and exactly how the parent-child relation works.

This is total gibberish to me.

I've been doing OOP for 15 years now, but I've no idea whatsoever.

Of course you can try to create some object based stringtype like C++, but then you will have to deal with all its problems, and the fact that Pascal's object model is not the same as C++'s. Also stuff that we take for granted (like copy-on-write) would be hard.

I'm not against debating about that, but then there must be something to actually debate about, not a mere suggestion that a bit of OOP will make all pain go away....

If I don't have the encoding of raw, how can I convert it to strict?

..... At this point I gave up.

Right, you continue to provide suggestions that only result in slow code at runtime :-(

Why not go the Delphi way, that is fully documented, easy and safe to use, and allows for fast applications?

As with a system that provides Type names and additionally does dynamic encoding such beasts _can_ occur (e.g. if you do a diabolic "move()"),

Such code indicates that the coder didn't understand the principles of Unicode string handling :-(

So the implementation definition needs to state what has to be done if such a case is detected (e.g. ignore or fire an exception).

Not all bogus code can be detected by the compiler. It may be helpful to have sanity checks in many places, but only with conditional compilation. Why bother the brave user with checks and consequently slow code, when these checks only are required to prevent the uneducated users from doing bad things?

Right, that's the *only* place where RawByteString makes sense.

But even then the coder is responsible for doing things right, because the compiler cannot assist in finding more or less obvious coding errors. See the Pos() example, that comes with an unsafe and unusable implementation in Delphi :-(

Abusing strings for binary data is a bad idea. Delphi provides an TBytes type for that purpose, and FPC could add more useful features to it.

DoDi

The major point is performance. With separate single and double byte characters/strings the compiler can insert the required conversions before every call. When instead a subroutine has to deal with mixed arguments, or adds strings from other places (literals...), any number of implicit conversions have to be added to the subroutine code, or all combinations must be handled in explicit code.

It also turned out that the best performance (and least code) can be achieved with only one string type and encoding, so that new (Delphi) code should only use UnicodeStrings. The AnsiStrings are supported merely for backwards compatibility (legacy code), and should be used only with native encoding - everything else will not be supported by the Delphi RTL.

In detail Delphi will not support an immediate conversion between Ansi codepages, instead a conversion provides UTF-16 first, which then is re-converted into the target codepage. This makes conversions very expensive. The only possible (expression) optimization again is based on UTF-16, where all sub-expressions are converted into UTF-16, so that only one more re-conversion is required when the result is stored.

DoDi

Al 17/11/2011 2:15, En/na Hans-Peter Diettrich ha escrit:

Abusing strings for binary data is a bad idea.

I use strings extensively as buffers: strings in delphi/fpc {$H+} are so convenient to use that the eventual performance hit doesn't matter to me. You know, I like the fact that I can simply do a

ReceiveBuffer:=ReceiveBuffer+NewData

without risking a buffer overflow, or that I can just use Copy to extract part of the buffer, etc., etc.

Delphi provides an TBytes type for that purpose, and FPC could add more useful features to it.

I didn't know TBytes, I googled it, and it seems it's defined as an array of byte, so it's not really as convenient to use as a string.

Bye

Hans-Peter Diettrich wrote

Right, you continue to provide suggestions that only result in slow code at runtime :-(

I did not say _anything_ about any kind of implementation, nor did I suggest that any of the alternative "suggested variants of definitions" is preferable.

The only suggestion I intended to offer is that _first_ a clear and straight definition of wow the stuff is to behave from the users view should be agreed upon, and _afterward_ the details of an implementation should be discussed.

Right, that's the *only* place where RawByteString makes sense.

IMHO doing functions the interface of which is encoding - independent is a very important case, especially regarding performance, as no conversion needs to be done when the function is called.So any definition that does not allow for this is lacking an important feature, and of course the use of a "general" String Type needs to be well defined.

(Of course the name "RawByteString" is a very bad choice for such a general string type that could be encoded e.g. as UTF-16).-

(As said I don't have XE so I don't know about TBytes.) Here the name "RawByteString" suggests exactly this kind of use.

Does TBytes provide reference counting and lazy copy ?

-Michael

...

This is exactly what I meant to say. (It's a viable definition, but...)

(B) was a counter proposal floated by Florian. The cons were pretty much that you had to guard every encoding sensitive routine (e.g. every API/OS call) to enforce the string contained the encoding you expected. Combining one and two byte types also cast doubt on the [] operator's performance.

...

This is exactly what I meant to say. (It's a viable definition, but...)

Then Yury proposed to combine A and B, in retrospect a bit like the current Delphi implementation but with one and two byte encodings in one type.

...

Yep. But IMHO the wording I proposed by (C): such as "object-alike" IMHO leads to a more "understandable" definition, in effect providing identical (or at least very similar) results and i.e. to at most an identical implementation, as most of the differences might be considered "implementation depending" (not defined by the pure, documented definition of the behavior (such as what happens with "intersexual" variables).

Note that the Delphi2009 definition is theoretically capable of combining one and two bytes in one type (like Yury's).

As I don't have such a Delphi please help me to understand:

Is there a general type dedicated for being able to hold any encoding ? (be it ANSIxyz, UTF-8 or UTF-16) ?

Of course, when assigning something to a "strictly encoded" String (the type denotes the encoding) the definition of what is supposed to happen is clear and obvious. If the Type name or the dynamic encoding of the target (even if Length=0) is used for deciding about a conversion is an "implementation detail".

Is there a clear definition about what happens if the "general" string type is the target ? Here, IMHO, it would be very hard to understand, if the history of the target variable (i.e. has a string of some encoding been assigned to it before) would decide about a conversion. IMHO a General string type needs to be handled as fully dynamically encoded and thus as a target always needs to get the source's encoding.

Such "assignment" can happen with ":=", and with function calls. With function calls there is "value" and "var" parameters. All this should behave identical, any other behavior would be very hard to understand.

And on top of this: what is the type "String" ? Of course the general String type would be an obvious choice, but perhaps (depending on the implementation) this might result in worse performance in certain cases of usage and thus some strict (specifically encoded) Type could be chosen. (In fact I will never again use "String" in any project, but use a propriety type defined in some central unit so that I at any time can do a central change to some specific string type.)

Embarcadero kept the two types separate,

Making a decently clear definition of the behavior (from a user's view) rather complicated.

- backwards compatibility (and thus the hurdle to upgrade)

This did not seem to have worked. Everybody, I asked, who migrated a large project to the new strings, was very unhappy.

Explain parent-child for explicitely this context. This kind of stuff is what I meant with self contained. Don't use terms that you don't fully describe elsewhere.

Sorry that I seemingly failed with my intention to help understanding what I meant by stating the similarity to the objects' parent-child relationship.

I just meant a "General" (or "Raw") string type needs to exist that can hold any encoding and needs no conversion when a strictly encoded variable is assigned to it (via ":=", value parameter or var parameter). Similar as with a parent object it "is" any strictly encoded string type, so that when using it as a nominal parameter of a function, it can
- without conversion - take any strict string type (and of course the general type, too).
Similar as with an object's runtime type (such as via "is" and "as"), the encoding of a General string can be detected and handled when appropriate (e.g. when combining with a another (strict or general) string or assigning to a strict string variable might request for conversion).

> the RAW string type and the types supposed to hold a specific encoding. Explain RAW.

See above. "General" or "not Strict" would be more appropriate (I took the term "Raw" from other recent discussions on the issue.)

Yes, I never really considered (B) a workable solution. It would break existing code, and the ways to deal with the other problems was hackish at best.

Yep. But I was told by unhappy coders that the new Delphi way breaks a lot of existing code, as well. So a new FPC way has a chance of being better. :) This might (or might not) be a way to do this.

I think the A-B hybrid is better than either A or B. And that is what is being implemented.

Yep. Only the definition of it's behavior of course is a lot more complex. In fact with "C" I tried (and failed) to find a proper basic definition of exactly this.

Then describe how that should work. What should happen if I pass such marked raw string to a function that wants encoding<y>?

I hope I did this some lines up. But better see below

I hope I did this some lines up. But better see below

I've been doing OOP for 15 years now, but I've no idea whatsoever.

Obviously it was not as a good idea as I thought to state the similarity between the relation between a single "General" and multiple "strictly encoded" string types regarding a Parent object and multiple Child objects. But I am not at all against dropping this analogy and just using a "self-contained" definition. Moreover I think we agree upon dropping the term "Raw" for the general string type.

So the wording could be similar to:
- There is a General String type that can hold any encoding (and any width of the code elements)
- There are lots of Strict String types that are supposed to hold strings in a predefined encoding (somebody else might describe in detail how these are defined)
- There are the appropriate single-character types corresponding to all of the above string types
- A variable of the General string type or the General character type only has a defined encoding if it before has been the target of an assignment of a not empty string or a character with defined encoding.
- If just using strict types the conversion rules are obvious

  - If assigning a value to a General string or character, (via ":=",

value or var parameter) no conversion is done.
- There are means to detect the actual encoding of a general String or Character variable.
- If combining any Strings/Chars with General String or Chars, the coding of the General ones is fetched from their embedded dynamic encoding definition to decide upon conversion.

I hope this is more like what you'd like to see.

Note that the recent discussion about how variable passing with RAW / not RAW strings is implemented might be decided by such a definition.

Note that Delphi seemingly introduced the encoding types $0000 for "None"/"to be assigned" and $FFFF" for "Raw". This allows for a string variable to be "General" or "Raw" with different meaning. How use this to implement a proper handling of General variables that hold a certain encoding but still are strictly General so that they get a different encoding with the next assignment ? I don't see of / if / if not this is helps implementing the above definition of if this or if it is a contradiction to same and/or provides nasty ambiguity.

Of course you can try to create some object based stringtype like C++, but then you will have to deal with all its problems, and the fact that Pascal's object model is not the same as C++'s. Also stuff that we take for granted (like copy-on-write) would be hard.

Of course I agree.

Thanks,
-Michael

Hans-Peter Diettrich wrote

The only possible (expression) optimization again is based on UTF-16, where all sub-expressions are converted into UTF-16, so that only one more re-conversion is required when the result is stored.

This is what mse does: using UTF-16 for the user code (i.e. the mse-GUI user interface)., This has been bashed a lot in the Lazarus forum. They say this is "Windowish"behavior resulting in bad performance on Linux, while Lazarus' LCL provides an UTF-8 user interface, for best performance on Linux (while supposedly introducing a lot of additional conversions on Windows.)

Moreover UTF-16 has been bashed as "not good enough for general use" as it does not provide good enough means for avoiding conversions in certain Unicode enabled applications.

-Michael

One could implement a similar type for something like this (maybe even use the mentioned TBytes) and define operator overloads for it (at least for "+").

Regards,
Sven

In theory the AnsiString type (which is now the code page aware string type) should be capable of holding UTF-8 and UTF-16 data, but either the direct unconverted storage of 2 byte data (UTF-16) is forbidden or undefined (don't remember which one it is in Delphi).

Such "assignment" can happen with ":=", and with function calls. With function calls there is "value" and "var" parameters. All this should behave identical, any other behavior would be very hard to understand.

Don't forget about "out". As it sets the string to empty I don't know by myself what Delphi does here (e.g. what codepage the string will contain).

In Delphi the type "String" is an alias to "UnicodeString", thus a 2-byte string. In FPC there is no final decision yet and thus currently "String" is an "AnsiString" set to a specific codepage (though I honestly don't know which one it is...).

Regards,
Sven

In our previous episode, Sven Barth said:

The ansistring and unicodestring types have the same memory layout except for the character data (iow the record before the character data is the same).

But that is not the same as ansistring allowing utf-16. Two types with the same memory layout are not equivalent.

The purpose is probably because of rawbytestring, to combine certain runtime procedures and helpers.

Why should one do this, regarding that a normal string type provides exactly what very often is requested: reference counting, lazy copy, dynamic length, extracting single bytes at any location, finding a certain byte or sequence of bytes, concatenation, deletion of a number of bytes at a given position. proven functionality and performance.

OTOH some important feature that very often is necessary with byte buffers is lacking with strings: To do a FiFo you need a _fast_ deletion from position 1. AFAIK, here strings do a copy of the complete content. To avoid this, an implementation would require handling the content in chops and manage the appropriate pointers. Providing such a Type and the appropriate compiler magic (for reference counting and operations line '+' would be really nice. Regarding the current discussion this even might be a dedicated string type (additional to the "General" and the multiple "Strictly encoded" variants). Of course such a specialized FiFo aware type would perform worse than an ordinarily string type in most other operations.

-Michael

Why should a type that is capable of holding multiple different UTF encodings be called "ANSIString". IMHO this is very contra-intuitive. I think FPC should establish a better name (such as "GeneralString" or similar). This would not harm Delphi compatibility as there could be a type alias for this.

but either the direct unconverted storage of 2 byte data (UTF-16) is forbidden or undefined (don't remember which one it is in Delphi).

What do you mean by unconverted ? What I mean is a type that just is able to "be" any of the "Strict" Types and thus provides for fully dynamic encoding for applications (function) that want to handle any encoding by the same code sequence (being aware that they take the appropriate conversion performance hit when combining differently encoded strings).

Of course we need a decent definition for this case. As I never intentionally used "out" parameters yet, I am not aware of the exact implications, but I am sure that there is a way to do a decently compatible definition.

In Delphi the type "String" is an alias to "UnicodeString", thus a 2-byte string.

IMHO, predefining a type named UnicodeString to be encoded as UTF-16 is contra-intuitive. I think FPC should establish a better naming (such as UTF16String for something that is predefined to be coded that way, if it in fact makes sense to define such a type in the language itself). For Delphi compatibility type alializing could be used.

In FPC there is no final decision yet and thus currently "String" is an "AnsiString" set to a specific codepage (though I honestly don't know which one it is...).

So I hope this discussion might help to promote a string Type functionality and naming system that is better than that Delphi currently provides.

I feel that - regarding the current state of the discussion - such types should be defined (I don't intend to define the exact names by this, nor to do any assumption on how to implement this):

- GeneralString (fully dynamic encoding can hold any encoding, 1, 2 and 4 byte code words, no conversion when used as a target of an assignment, automatic conversion whenever necessary)

- RawByteString (on byte code words, never doing a conversion, supposedly triggering an exception when combined with a variable that requires a dedicated encoding)

- Raw Word String (two bytes code words, working like RawByteString)

- RawDWordString (four bytes code words, working like RawByteString)

- UTF8String (one byte code words, behavior is obvious)

- UTF16String (two byte code words, behavior is obvious)

- UTF32String (and/or UCS4String) (four byte code words, behavior is obvious)

- ANSIString(n) (strictly encoded according to an ANSI code page, one byte code words, behavior is obvious)

- ANSIStinrg(and/or LocaleString) = ANSIString(n) n defined by current locale) This thingy should work very much alike the plain old "String", even though the implementation is different.

and as a goody this could be implemented later:

- RawByteFIFOString (behaving exactly as RawByteString, but implemented in a way that deleting from position 1 is much faster, while any other operation might be slower)
- RawWordFIFOString (obvious)
- RawDWordFIFOString (obvious)

Moreover I feel that for some or all of these string types corresponding character types should be provided. Otherwise I don't see how consistent programming could be enabled. This obviously includes a dynamically typed character type.

Moreover, IMHO, the meaning of the position aware functions (MyString[i], pos(), copy(), delete(), ... ) should be reconsidered, to allow the user to somehow declare his will to either work on code-positions (fast) or on visual-character-positions (meaningful, user-friendly).

(I don't think this is a great contradiction to what already is implemented in the svn.)

-Michael

Marco van de Voort wrote

The ansistring and unicodestring types have the same memory layout except for the character data (iow the record before the character data is the same).

My intention with starting this Thread was not to discuss any implementation details, but to try to help finding a decent definition. I do feel that the current FPC implementation (which I did take a look at, rather than that of Delphi) to a great length provides what is necessary to adhere to a properly defined naming and functionality scheme.

I think that a decently defined scheme could largely be Delphi compatible, so that differences (i.e. improvements) can be considered as "implementation detail" from the view of a Delphi influenced user.

--Michael

Michael Schnell wrote

Why should a type that is capable of holding multiple different UTF encodings be called "ANSIString". IMHO this is very contra-intuitive.

Every time I see this used in Delphi too, I start to laugh as well. It makes no sense. Call the damn thing UnicodeString because that is exactly what it is.

The other annoyance of Delphi is the assumption that the term "Unicode String" always mean UTF-16. A real slap in the face for unicode.org guys.

Can't we just have a single damn string type like Java and some other languages. Lets just call it...ummm.... String! ;-)

Regards,
- Graeme -

Graeme Geldenhuys wrote

Can't we just have a single damn string type like Java and some other languages. Lets just call it...ummm.... String! ;-)

This has been discussed at any length here and in many other forums. This is what I tried to describe in (B). It has been turned down as it's not possible to implement this providing predictable performance.

-Michael

This could indeed have been named better. But there are other examples like this: I still can't remember which of SmallInt and Short is the 1 Byte and the 2 Byte variant. Some type names like "Signed8" and "Unsigned16" would simplyfy this... but I won't go more into that direction now ^^

Regards,
Sven

Perhaps a little extra compiler magic could be used? If the base definition of a string (the hidden stuff before the data) contains not only a field with the encoding, but a flag indicating the disposition of the encoding, then when a string type is aliases, that disposition could be overridden. I'm on my phone, so this may be hard to show an example:

type String = record
  encoding: TEncoding;
  disposition: Tdisposition;
  data: Tbytes;
end;

Disposition would have values like:

fixed - will always be in this encoding, anything assigned to a variable of this type will be automatically converted. Any out parameter of this type will be initialized to an empty string of this encoding.

open - encoding only defines initial encoding. Assigning a string of another type will result in this string taking on that type, but NOT that type's disposition. Out parameters initialized to the given encoding.

strict - only compatible with other strings of the same type, period. Try to assign a string of a different type, compile time error. Will force manual conversion between encoding (possibly via typecast that calls internal encoding translation).

Example:
type ANSIString = String encoding UTC16 disposition fixed;

Syntax is probably all wrong, but hopefully you get the idea. If the real lowest level is something other than String, then String itself can be an alias that is different for each platform to match that platforms native type.

It seems something like this would let the people who just want to use strings in a consistent way and not worry about conversion performance to do so, and those who want speed to have the compiler help them out by stopping them from creating code with lots of conversions.

Or, I may be completely off base.

Jeff.

Michael Schnell wrote

Why should a type that is capable of holding multiple different UTF encodings be called "ANSIString". IMHO this is very contra-intuitive.

Yes, I have to agree here. It seams that my understanding in the Unicode discussions was plagued by the moto which was repeated to death in the last decade: Ansistring holds a string in the system encoding.

And now very recently I found out that this is no longer valid in 2.7, ansistring can be configured to hold a UTF-8 value in a valid and supported way, and this changes a lot of things to me.

Sven Barth wrote

like this: I still can't remember which of SmallInt and Short is the 1 Byte and the 2 Byte variant. Some type names like "Signed8" and "Unsigned16" would simplyfy this... but I won't go more into that direction now ^^

For exactly the same reason I have a A5 "cheat sheet" lying on my desk for the last two years. :-) It contains all the Object Pascal primitive types with their ranges and size in bytes. It also has some C/C++ alias types and what they equate to in Object Pascal (eg: DWord -> Cardinal, UInt -> Cardinal etc)

Regards,
- Graeme -

In our previous episode, Graeme Geldenhuys said:

This will have to be fixed out of Delphi compatibility. D2009+ defines (u)int8/16/32/64. I'll see to it in the coming weekend (though it will be too late for 2.6.0)

Finally :D

Regards,
Sven

As already said, my request at this time is not considering implementation before agreeing on a decently clear definition of what should be provided and what is supposed top happen when.

-Michael

This is ambiguous, but with strings the Delphi naming IMHO is plain wrong.

-Michael

Felipe Monteiro de Carvalho wrote

And now very recently I found out that this is no longer valid in 2.7, ansistring can be configured to hold a UTF-8 value in a valid and supported way, and this changes a lot of things to me.

To the worse I gather.

-Michael

I don't see what you mean here. My suggestion regarding "new string types" was to provide all thee Types RawByteString, RawWordString, RawDWordString, for this purpose.

Why implement another String type to ease something strings weren't intended for?

In C it is very common to use an int for a pointer. :)

My understanding of the word "String" just does not include the definition that this is necessarily something providing a "visual" content.

Implement your own type that handles all this and then you don't even have a problem should FPC decide to switch String from AnsiString to UnicodeString.

Why invent yet another time, when an existing one is perfectly suitable ?

I don't see how a user can define a type that provides reference counting and lazy copy. I gather that this needs compiler magic.

With this in mind I feel that (regarding the proceedings regarding the new string type definition) it would be a nice thing to provide RawByteString, RawWordString, RawDWordString that don't ever trigger a conversion and (at a later time) additionally consider to do FiFo String types that, are optimized for allowing deleting from position 0.

-Michael

Quite the contrary, everything is so much better now, because from what I understood ansistring can be configured to hold a utf-8 string globally. So Lazarus can globally initialize it like this so we can be on the safe side for the first time in decades: Still using utf-8 everywhere but now with RTL support and doing it the right way.

And it gets even better: From what I understood we don't even need to request anything to be changed in the RTL, everything required will automatically come with FPC 2.8

In this solution a migration would be needed, although a very simple one, and it will likely take years to complete this migration because we won't jump into requiring FPC 2.8+ so quickly. But it really looks like this solution could be it. A safe and stable road ahead without major compatibility breaks.

The only thing that could stop this solution would be FPC dropping support for the Ansistring-based RTL, which looks very unlikely from what I understood.

Seems like the message you quote here went to you personally as well (that would explain why you sent this answer to me directly first...)

Thus here the original mail I wroted

=== original mail begin ===

Because then you don't need to rely on the point that SizeOf(Char) = 1. Now imagine you have an applications that uses strings as buffers and port that from lets say Delphi 7 to Delphi 2009. Have fun finding the bugs if you don't remember that you used a String as buffer.

Why implement another String type to ease something strings weren't intended for? Implement your own type that handles all this and then you don't even have a problem should FPC decide to switch String from AnsiString to UnicodeString.

Regards,
Sven

=== original mail end ===

It might indeed work as long as you don't do the following to move the received data to the buffer string (the same applies for an operation into the other direction):

SetLength(YourStringBuffer, LengthOfData);
Move(YourBinaryData^, YourStringBuffer[1], LengthOfData);

> > Why implement another String type to ease something strings weren't > intended for? In C it is very common to use an int for a pointer. :)

And its also common to use the Object part in TStrings for integers, but that doesn't make it suitable for that task.

The point is: it's not perfectly suitable. While things like reference counting and copy-on-write are nice, a type for usage as buffer should have fast insert (and maybe delete) operations. You can easily program a custom type once that handles that good while for Strings you always need to do the SetLength-magic every time you want to do this.

I don't see how a user can define a type that provides reference counting and lazy copy. I gather that this needs compiler magic.

COM type interfaces would be a possibility...

I see no need to introduce a special string type that has optimized FiFo operations. One can easily write that once by hand and maybe include that in the FCL, so there is no need for additional compiler magic.

Regards,
Sven

I'm not sure, but UTF encodings are not Ansi codepages, IMO. That's why UTF-16 is not an allowed encoding of an AnsiString, and UTF-8 seems to be accepted only by accident, is not fully supported by Delphi.

The other annoyance of Delphi is the assumption that the term "Unicode String" always mean UTF-16. A real slap in the face for unicode.org guys.

How that? UTF-16 is a valid Unicode encoding, just as is UTF-8. The difference between Ansi and Unicode strings only is the range of characters (codepoints), which are expected in the strings, and which are supported by the supplied stringhandling procedures.

Did you ever notice that stringhandling was restricted to the system codepage, before the invention of AnsiStrings with an encoding? Unicode stringhandling is indpendent from any codepage issues, and can be implemented for any Unicode (UTF) representation in the same way (API). While WideString only was a container for UTF-16, without stringhandling support beyond the WinAPI, such functionality has been added to UnicodeString. BTW such functionality now is available for AnsiStrings, too, at least by implicit conversion to and from Unicode strings.

Can't we just have a single damn string type like Java and some other languages. Lets just call it...ummm.... String! ;-)

Isn't this exactly what Delphi did? There is one generic "string" type, which now finally is a Unicode string - just like in Java?

But Delphi and FPC suffer from legacy types, which still have to be supported.

DoDi

That's not easily feasable, as long as empty strings are implemented as Nil pointers. When reference counting etc. should be preserved, the additional information had to be moved into an static string descriptor, together with the pointer to the dynamic string content. And what about temporary strings, used in string expressions?

I don't think that an added "disposition" will improve anything, because its value has to be checked with every access to a string variable. With strictly typed strings (of fixed encoding) all checks can be performed at compile time.

DoDi

As already said, my request at this time is not considering implementation before agreeing on a decently clear definition of what should be provided and what is supposed top happen when.

IMO such a separation is of no use. If you want an new string type, then make it a class, and add operators and other functionality as required. Then everybody can use his preferred class, and no decision about "the" string type is required.

DoDi

"Hans-Peter Diettrich" wrote

That's not easily feasable, as long as empty strings are implemented as

Nil pointers. When reference counting etc. should be preserved, the additional information had to be moved into an static string descriptor, together with the pointer to the dynamic string content. And what about temporary strings, used in string expressions?

I was thinking that the type for this sort of thing would be done at compile time, not run time, but I suppose I expressed that quite poorly.

I don't think that an added "disposition" will improve anything, because

its value has to be checked with every access to a string variable. With strictly typed strings (of fixed encoding) all checks can be performed at compile time.

The idea was that instead of forcing strings to be one way (dynamically typed) or the other (fixed type), as defined by the compiler writers, that instead both behaviours be supported, and which is in play for a particular piece of code is decided by the developer. How that gets implemented cleanly could be anything, but I think it would have to be in the type declaration.

Jeff.

Hans-Peter Diettrich wrote

IMO such a separation is of no use. If you want an new string type,

I don't want anything dedicated, I just want to help to make FPC the best possible compiler.

then make it a class,

Very obviously strings are not classes (in Pascal)

-Michael

Hans-Peter Diettrich wrote

IMO such a separation is of no use.

While of course doing a viable (written) definition sometimes is less fun than being "creative" and start implementing what you "feel", very obviously this likely leads to sub-optimal results and fruitless discussions.
-Michael