Starting a new thread for a new topic. It's not that hard.

On 15.08.2017 18:33, Mattias Gaertner via Lazarus wrote: > Do you propose a string without the array operator [] ? I don't understand what you mean by this.

Every Delphi/FPC type has a bunch of operators. Strings support :=, =, , >=, <= and [] for read and write.
When you propose a new string type "dynamicstring" you have to define these operators. Otherwise your proposal is not a string, but just a magic unicorn.

Of course an appropriate "char" type for each string encoding brand could to be provided, hence a "CP_QWord Char" as an alias or a QWord.

There is no QWord codepage. That would be confusing.

Mattias

Mattias Gaertner wrote

Every Delphi/FPC type has a bunch of operators. Strings support :=, =, , >=, <= and [] for read and write. When you propose a new string type "dynamicstring" you have to define these operators.

That is easily doable.
The definition of := is discussed in the paper. (Only for := there is no accessible encoding definition for the left operand.) If the encoding branding is one of those that already exist, the current definition is used.
For "new" encoding brandings, such as CP_Byte, CP_Word, CP_DWord, CP_QWord, the working of the operators is obvious. It somebody tries to compare a printable Text string with a string of binary elements, maybe the behavior is undefined.

There is no QWord codepage. That would be confusing.

Of course the term "Codepage" Embarcadero chose for the encoding identification is misleading in this context. That is why in the said paper it's called "encoding style" (which is not a really appropriate wording, either, but hey, it's just an initial suggestion and not yet a final documentation, and it had been clear from the beginning that it's in vain, anyway. )

-Michael

BTW.:
This is off topic in the Lazarus list and would be an fpc issue.

Moving it in one of the fpc lists would have the additional advantage that I am banned there, anyway, for having answered to similar questions.

-Michael

There are no such codepages. You are confusing people if you name your encodings like this.

It somebody tries to compare a printable Text string with a string of binary elements, maybe the behavior is undefined.

Your "dynamicstring" supports char, widechar, byte, word, dword, qword. Why not shortint or smallint?
Why not boolean, single and variant?

What is the intention of your proposal?

Mattias

Mattias Gaertner wrote

You are confusing people if you name your encodings like this.

There also is no "official" Code pages named "Default" or "None", the naming "CP_DEFAULT" and "CP_NONE" has just been invented by Emparcadero.

So I did the same and just brainlessly extended the existing "CP..." naming scheme.

Your "dynamicstring" supports char, widechar, byte, word, dword, qword. Why not shortint or smallint? Why not boolean, single and variant?

As pointed out this is just a draft of a proposal, prone to enhancement and improvement.

What is the intention of your proposal?

That is given in the instructional paragraph "The problem": "The most obvious candidate for pain on that behalf is “TStrings”.

Only a fully dynamically encoded version of TStrings and friends would allow for a solution for many string encoding related problems, as the user can't modify the string encoding brand TStrings uses and hence will face the described problems when he uses TStrings with all but one of the String encoding brandings he can choose from.

Enhancing the count of available encoding brandings is just a logical consequence of a less problem prone and more versatile (not implicitly restricted to printable text) overall string handling.

-Michael (It's rather frustrating to discuss that obviously never will happen :-()

Mattias Gaertner wrote

When you propose a new string type "dynamicstring" you have to define these operators.

BTW, it will be good to have "Cstring" (or another name, not "dynamicstring") :

- [] operator is 0-based like Python/C

- s[i] is DWORD per char (for all Unicode chars from 0 to MaxDWORD codes)

PChar(s)/PWChar(s) wont work for it? so it is not ok idea? But this type can be compressed inside, eg in utf8. S[i] is DWORD outside. It is like some class.

Mattias Gaertner wrote

> Of course an appropriate "char" type for each string encoding brand > could to be provided, hence a "CP_QWord Char" as an alias or a QWord. There is no QWord codepage. That would be confusing.

And that would still not be "char", but "codepoint"

A char can be composed of several combining code points (each of them afaik, in the 32 bit range).
So a char can have 96 or more bits. (And not all of them have a combined form).

See my prev post: i see that each S[i] good to be like QWord (sizeof(one char)= sizeof(Qword)). It can be TextChar. And type can be TextString. internally it can be compressed to utf8. TextString is good if i want to parse text by "chars". If "char" needs more bytes- lets take more (internally it is same utf8)

On 16.08.2017 13:17, Mattias Gaertner via Lazarus wrote: > You are confusing people if you name your encodings like this. There also is no "official" Code pages named "Default" or "None", the naming "CP_DEFAULT" and "CP_NONE" has just been invented by Emparcadero.

It is not about "official". A codepage describes a character set. What has your CP_QWORD to do with any character set?

I read it, but I must admit, I don't understand it. For some unknown reason you want to store different encodings in a TStrings and fear the "time-consuming" and loss-prone auto conversions. And then it sounds as if this is a common problem ("much more urgent").

[...] Enhancing the count of available encoding brandings is just a logical consequence of a less problem prone and more versatile (not implicitly restricted to printable text) overall string handling.

Who wants to have more encodings?
AFAIK everyone wants less, preferably only one.

-Michael (It's rather frustrating to discuss that obviously never will happen :-()

Not if complicated things get more complicated.

Mattias

Unfortunately in Delphi and FPC the appropriate work-alike existing type is called Char (with certain extensions). It would cause major problems to drop that name for something else, even if that would be appropriate.

-Michael

This sounds, as if you want an UTF-32 string type.

Michael's proposal is a multi encoded string type, storing Ansi, UTF-8, UTF-16 and UTF-32.

Mattias

Alexey wrote

BTW, it will be good to have "Cstring" (or another name, not "dynamicstring") : ...

You are missing the point the paper is supposed to be about: enhancing the versatility of the library functions such as those using TStrings. Not just creating another type of strings, which is nothing but a prerequisite for the main purpose.

-Michael

Mattias Gaertner wrote

Not if complicated things get more complicated.

Please leave out the additional encoding brands suggested just as an afterthought in the paper. These are not the purpose at all but ()if the other stuff would be in place) just com as a natural enhancement.

-Michael

Mattias Gaertner wrote

For some unknown reason you want to store different encodings in a TStrings and fear the "time-consuming" and loss-prone auto conversions.

It's obvious that a user using a different encoding brand in a string var than that suggested by TStrings (UTF-8 in fpc, UTF-16 in Delphi) implicitly triggers auto-conversion when handling the string. This has several consequences.

It might be a really good idea when e.g. doing some code that in a loop needs certain operation that might be very fast with UTF-16 but TStringList would store the data in a more compact way.

It might be time consuming when the conversion is done without being necessary.

It might be error pone when the user stores some random stuff in the string that is not able to be handled by the conversion forth and back.

In any case all this happens without the user being aware of, which might cause frustration.

-Michael

No Alexey, you are now explaining codepoints.
Codeunits and codepoints are the easy part in any case. Could you please define character.

Juha

Michael Schnell wrote

-Michael (It's rather frustrating to discuss that obviously never will happen :-()

Why don't you implement such a system. This is all FOSS, free and open source.

You are writing about encodings etc. which are part of codepoints, but you call them "characters". Why?
Is it possible you don't know Unicode beyond codepoints? Do you have plans to tackle also the complex issues of Unicode? If not, then your efforts are useless because codeunits and codepoints are easy in any case. You use energy for a problem that does not exist.

Juha

Juha Manninen wrote

Why don't you implement such a system. This is all FOSS, free and open source.

I would never dare to try to edit the compiler :-[

You are writing about encodings etc. which are part of codepoints, but you call them "characters". Why?

Because the type for this stuff used in Delphi and and FPC is called "char".

Is it possible you don't know Unicode beyond codepoints?

In fact I did not explicitly talk about paper says it: "In this article, a "String" is thought of as a reference counted ordered array of a number of "Things" (aka elements). (I feel that this is what the name String suggests.)" ..."If the elements of the strings are printable characters or partial codes of UTF. OK, this is nice (provided the conversion functions are in place) and makes doing programs handling conventional problems very easy" ...

Do you have plans to tackle also the complex issues of Unicode?

Not at all.

If not, then your efforts are useless because codeunits and codepoints are easy in any case.

I know. The intention was to handle a completely different problem from that you suggest here.

You use energy for a problem that does not exist.

I wrote the paper because I once was requested to do so in the fpc forum.

-Michael

Have a look at
https://www.reddit.com/r/Unicode/comments/4yie0a/tallest_longest_unicode_character/

There is ONE character, that comprises more than 200 codepoints. Only way to store such a char is in a type of dynamic size (aka string)

Well I couldn't find an official doc what makes the boundaries of a char.

But as far as I can see: if ä is one character, and it can be encoded as "none combining codepoint" + "combining codepoint", then a character is any sequence of one "none combining codepoint" + zero or more "combining codepoints" (AFAIK Arabic scripts has chars, that have several "combining codepoints", so this is happening in actual languages.

The example as far as I checked fulfils this definition.

I agree. "char" actually is a "code unit".

But renaming it, would probably be as good as killing the language.

and anyone can do
type codeunit=char;

and use this.

Michael Schnell wrote

> You are writing about encodings etc. which are part of codepoints, but > you call them "characters". Why? Because the type for this stuff used in Delphi and and FPC is called "char".

No, actually the Pascal type "Char" contains a CodeUnit, not CodePoint. It is the smallest fixed width "atom" of Unicode text. It is still extremely useful in Unicode related programming.
The word "character" in Unicode can mean:

1. CodeUnit — Represented by Pascal type "Char".

2. CodePoint — all the arguments about one encoding's supremacy over another deal with CodePoints. Yes, UTF-8, UTF-16, UTF-32 etc. all only encode CodePoints.

3. Abstract Unicode character — like 🍷 'WINE GLASS'.

4. Coded Unicode character — "U" + a unique number, like U+1F377. This is what "character" means in Unicode Standard.

5. User-perceived character — Whatever the end user thinks of as a character. This is language dependent. For instance, ‘ch’ is two letters in English but one letter in Czech and Slovak.
Many more complexities are involved here, including decomposed codepoints.

6. Grapheme cluster

7. Glyph — related to fonts.

So, number 4. is the official Unicode "character". Otherwise the most useful meanings are 1. "CodeUnit" for programmers and 5. "User-perceived character" for everybody else. Note, CodePoint is NOT a useful meaning for "character". It would only confuse things. Yet most people in these Unicode threads write about "character" like it meant CodePoint. It can only mean that those people are ignorant of the complexity of Unicode. :(

In fact I did not explicitly talk about paper says it: ...

Unicode is the standard now. We cannot ignore it, and we don't want to ignore it because it solves so many problems of the earlier solutions. If you create a new string type, you certainly must take Unicode into account.

Juha

Juha Manninen wrote

The word "character" in Unicode can mean: 1. CodeUnit — Represented by Pascal type "Char".

Actually no.

It can overlap. But a codeunit is NOT a character.

For example a codeunit that holds a codepoint of class "combining mark", this is not a character. It is just something that can form a character if combined with other codepoints.

2. CodePoint

Also not a character. Same as above.

Some Codepoints happen to also be a character. But some are not.

Martin Frb wrote

Actually no.

I know CodeUnit and CodePoint are not called "character" officially by the Unicode Standard.
They however are called "character" in normal communication. For example in the "String vs WideString" thread most people used "character" as a synonym for CodePoint.
For CodeUnit the term is very logical for historical reasons as the type "Char" is a short form of "Character". This is a very important meaning because CodeUnit resolution is so useful also with variable width encodings.
For example the following code works perfectly with UTF-8 and UTF-16:

function SplitInHalf(Txt, Separator: string; out Half1, Half2: string): Boolean;
var
  i: Integer;
begin
  i := Pos(Separator, Txt);
  Result := i > 0;
  if Result then
  begin
    Half1 := Copy(Txt, 1, i-1);
    Half2 := Copy(Txt, i+Length(Separator), Length(Txt));
  end;
end;

although Pos(), Copy() and Length() deal with CodeUnit resolution. I wonder how the new fancy string types would handle it without a performance penalty.

Juha

Juha Manninen wrote

Unicode is the standard now. We cannot ignore it, and we don't want to ignore it because it solves so many problems of the earlier solutions. If you create a new string type, you certainly must take Unicode into account.

It is not "ignored", as it is handled by the conversion functions the functionality of which is not touched. The paper is just about storing the information in the strings (including the "encoding brand" and "bytes per element") fields.

So the actual meaning of the stuff that is stored in the strings is beyond the scope of the paper. And supposed to stay as it currently is.

-Michael

Juha Manninen wrote

although Pos(), Copy() and Length() deal with CodeUnit resolution. I wonder how the new fancy string types would handle it without a performance penalty.

This again is not in the scope of the paper, and supposed to stay as it is. S[x], Pos(), and friends work in terms of "bytes per element" bytes.

The only difference to the current status is that with the "dynamic" string brand the content of the "bytes per element" field is not predefined by the variable declaration but can change when something is assigned to that (additional) brand of string variables (I feel that this is clearly stated in the paper). Hence for that (additional) brand of string variables the compiler needs to generate code to read this field when implementing the built-in functions.

-Michael

This "dynamicstring" sounds like Rawbytestring times two. Any function accessing the inner chars of a "dynamicstring" has to handle Rawbytestring codepages and unicodestring and array of byte/word/dword. If this is the price for avoiding some conversions, many programmers will become unhappy.
Michael, please tell me your proposal has some serious advantages. I don't see them.

Mattias

And that is where the problem starts.

As long as people do this, even if they know it is incorrect, others will pick it up, and others will learn the wrong concepts.

Calling codepoints = char, means that newcomers will think s[x] is a valid way to deal with chars.
And that is wrong, even in utf32.

For example in the "String vs WideString" thread most people used "character" as a synonym for CodePoint.

Lots of people used the word character as if they where the same as codeunit.

But the questions is did they use it as synonym? I.e did they know they were substituting with the wrong word?
If so, why would they intentionally use misleading terms?

For CodeUnit the term is very logical for historical reasons as the type "Char" is a short form of "Character".

That is why today it is a misnomer.

So using "char" (the type) as reference to "codepoint" is something we have to do, because today the type "char" is for codepoints.

That is different from the English word "char" and that can cause a huge confusion.

The English word "character" however is unambitious. It is not the name of a type. So it refers to character only, not to codepoint.

Exactly. Discussions where the word "character" is used are very vague and inaccurate.

So using "char" (the type) as reference to "codepoint" is something we have to do, because today the type "char" is for codepoints.

Sorry I didn't understand this one.
"Char" (the type) holds a codeunit, not a codepoint. Char is either 1 byte or 2 bytes depending on if it maps to AnsiChar or WideChar, for UTF-8 or UTF-16 respectively.

Juha

Right yes. Genuine mistake with all the confusion....

Mattias Gaertner wrote

This "dynamicstring" sounds like Rawbytestring times two.

In fact I do suppose that the initial intention the Developers had with Rawbytestring, was something like this, but it never was implemented or documented appropriately. In fact I (no access to XE) don't know hat happens when you do Rawbytestring := SomeDedicatedString. With the DynamicString here the "encoding brand" and "bytes per element" are copied over.

AFAIK Rawbytestring never is prone to autoconversion. The goal of DynamicString is exactly the contrary: do autoconversion when necessary.

Any function accessing the inner chars of a "dynamicstring" has to handleRawbytestring codepages and unicodestring and array of byte/word/dword.

In fact it has to handle the "codepage" etc that is denoted dynamically in the string header. Exactly like with any other string. Only that it can't be determined at compile time. This of course will introduce a small performance degradation (happening only when explicitly DynamicStrings are used).

What do you mean by "accessing the inner chars" ? If you do SomeDedicatedString := Rawbytestring, AutoConversion is done necessary (i.e. the (static) encoding brand of the target is not equal the (dynamic) encoding brand of the source, and both are not "Raw" / "binary"). This of course needs appropriate compiler magic.

If this is the price for avoiding some conversions, many programmers will become unhappy.

I don't see, why.

Michael, please tell me your proposal has some serious advantages. I don't see them.

Matthias, thanks for taking the effort of evaluating the (weired) suggestion.

The paper was the result of several discussions in the fpc forum started by users complaining hat their code did not work any more when compiling with a new version or fpc/Lazarus.

- Code ported from older (not Unicode aware) Delphi or fpc versions: forced usage of Unicode functions (e.g. based on TStrings) was not compatible with their usage of one-byte-strings the coding of which was of no concern for the compiler (because it was not necessary)

- Code ported from the intermediate Unicode aware Lazarus version that ran with the not Unicode aware fpc: all kinds of problems, a very popular problem at that time.

- Code ported from newer (Unicode aware) Delphi versions: problems arising from UTF-8 / UTF-16 differences.

And you could not have them just define their strings appropriately to have the Unicode aware fpc behave like the system they produced their code on, as (e.g.) TStrings forcibly requires UTF-8.

Hence the ability to allow TStrings (and siblings and in fact any other library function) to handle any encoding / bytes_per_element the user choses for his string functionality, could be a solution.

This in place (in fact rethinking the dull encoding-aware-string support Embarcadero forced on their users), the "String" paradigm can be enhanced to allow for additional functionality. E.g.:

- Non-Text String elements (such as Bytes, Words, DWord, Qwords, ....)
- User defined encoding (the user would be enabled to supply the conversion functions with other string encoding brands he intends to use.

-Michael

Maybe, Sven could answer to this mail in the other thread...

What "indexing" do you think of ?
Could you give an example where such a difference is supposed to get important ?

(As you know I wrote a paper where I claimed the contrary. I'd like to revise same if necessary.)

-Michael
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20170817/e36b8438/attachment.html>

Am 17.08.2017 11:11 schrieb "Michael Schnell via Lazarus" <

: Maybe, Sven could answer to this mail in the other thread...

I provided an example in my answer to Tony Whyman in the same subbranch of the thread.

Regards,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20170817/80156a1b/attachment.html>

I suppose you mean this:

This is not applicable to the "DynamicString" discussed here. With that, "pos" would just need to use the "bytes per element" value taken (once) from the the String header, instead of a value supplied at compile time. I don't think that this can introduce a considerable performance hit.

-Michael
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20170817/3b2e3333/attachment.html>

Am 17.08.2017 11:41 schrieb "Michael Schnell via Lazarus" <

of the thread.

(take Pos() for example) this will lead to a significant slowdown as the compiler (without optimizations) will have to insert a call to the lookup function for each access. While I generally don't consider performance degradation as a backwards compatibility issue I do in this case, due to the significant decrease in performance.

This is not applicable to the "DynamicString" discussed here. With that,

"pos" would just need to use the "bytes per element" value taken (once) from the the String header, instead of a value supplied at compile time. I don't think that this can introduce a considerable performance hit.

Yes it does, because without optimizations implemented and enabled the compiler would need to do this for each element access.

Regards,
Sven
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20170817/f8ea2f80/attachment.html>

Sven Barth wrote

Yes it does, because without optimizations implemented and enabled the compiler would need to do this for each element access.

But it does not need to do a function call. If getting the element size of a string is a pascal function it would be built-in.

Moreover I suppose (or suggest) that pos() is a built-in function that in fact can use decently optimized code.

Also a library function could use optimized pascal code that reads the element size, stores it in a variable and uses a pointer within the loop.

Of course a user function would just do s[x] and see some degradation.

-Michael

Martin Frb wrote

There is ONE character, that comprises more than 200 codepoints. Only way to store such a char is in a type of dynamic size (aka string)

Interesting inf, thanks; so that my idea is not ok for new FPC type. it is ok only for some class, which adds enumerator to take "char" (like you told) of type string. This class not needed in fpc

I answer here Tony's post in "String vs WideString" thread.

So, maybe it is the "User-perceived character" from my list above.

You use words "character" and "encoding" in the same sentences, although only codepoints are encoded.
Codepoints are the easy part anyway.
A "character" can be composed of many codepoints and the rules for that are complex and depend on locale. It is not only about combining accent marks, some languages have rules about grapheme clusters etc. (which I don't know well).

I was referring to GB 18030 and that it has one, two and four byte code points.

Ok, I didn't know that one. If I understand right, it is not part of Unicode.

The point I believe that you are missing is to consider that a character is an abstract symbol with a semantic independent of how it is encoded. Collation sequences are independent of encoding and should remain the same regardless of how a character set is encoded.

Above I used concepts and terms of Unicode.
Maybe your idea is to create another abstract system outside of Unicode. The problem is that the complex rules of Unicode are needed to support all the languages in the world.
Your new abstract system would be equally complex in the end.

Now I remember, my unit LazUnicode in LazUtils has classes:
- TCodePointEnumerator and
- TUnicodeCharacterEnumerator which can handle Combining Diacritical Marks. It may be enough for most European + many other languages. Please take a look.
Maybe you could use the idea for a new abstract "character" class. Anyway such a complex logic must be in library code, not in compiler's built-in type.

Or maybe you just want fixed-width indexing of codepoints without dealing with complexities of Unicode. In that case UTF-32 and a new string type UTF32String or similar would do it.

Juha