Hi All,

In the past I used Length, Pos, Delete, for i:=1 to Length(s) do s[i]... and realized yesterday these practices are wrong. But I do not know what the right practice.

Found UTF8* on "LCL Unicode Support" wiki page, later {$Codepage UTF8} on "Better Unicode Support in Lazarus".

The result of below lines:
2
2
1
1

With {$Codepage UTF8}:
2
1
1
1

I see same results with Linux and Windows.

UTF8* is good to me but a compiler directive is easier to use, just don't know why not working properly.

What is the proper way to read through (character after character) the string if use UTF8* procedures?

var
   s:String;
begin
   s:='á';
   Memo1.Lines.Add(IntToStr(Length(s)));
   Memo1.Lines.Add(IntToStr(Length('á')));
   Memo1.Lines.Add(IntToStr(UTF8Length(s)));
   Memo1.Lines.Add(IntToStr(UTF8Length('á')));

Gabor

Gabor Boros wrote

UTF8* is good to me but a compiler directive is easier to use, just don't know why not working properly.

Please the wiki page you found. It is explained there. http://wiki.freepascal.org/Better_Unicode_Support_in_Lazarus#String_Literals

What is the proper way to read through (character after character) the string if use UTF8* procedures?

A "character" in Unicode is an ambiguous term.
Often the good old byte (codeunit) access is very useful. See:
http://wiki.freepascal.org/UTF8_strings_and_characters

Juha

* Please read the wiki page ...

2016. 10. 21. 10:24 keltezéssel, Juha Manninen via Lazarus írta:

A "character" in Unicode is an ambiguous term. Often the good old byte (codeunit) access is very useful. See: http://wiki.freepascal.org/UTF8_strings_and_characters

I started with the wiki pages, but 2 about UTF8 in english is too much for me and you pointed to a 3rd... :-)

On the above link at "Searching a substring" I read "Due to the special nature of UTF8 you can simply use the normal string functions for searching a sub-string.". But the example Where procedure returns this for me: "The substring "á" is in the text "éáó" at byte position 3 and at character position 1". Which incorrect because á is the 2nd character.

Gabor

Thanks for the hint. I fixed it.

Mattias

Indeed this is a serious problem these days, unicode.. which is almost a virus. In GoLang they use something called "Runes" to try and solve the problem. Off topic but I wonder if Lazarus/fpc uses something anything similar to golang's rune's approach or looked into it.

IMO unicode reaches something like Godel's incompleteness problem. You can never actually prove that a unicode program will work properly nor prove that it won't have bugs, because unicode creates infinite gotchyas and unicode is always evolving to have more characters that you didn't know about before.

It makes code inelegant compared to plain english 255 systems like in the 1970's.

There is an interesting article/video about it on Sucksless, and even this guy scares me when he talks about unicode even though he is trying to fix the problems:
"UTF-8 everywhere? Writing Unicode compliant software that sucks less, Laslo Hunhold"

But it of course is not specific to Lazarus. Sorry for slightly off topic.

2016. 10. 21. 10:25 keltezéssel, Juha Manninen via Lazarus írta:

* Please read the wiki page ...

I read, I read but if contains buggy example... ;-)

I need a quick and a rock solid solution. Is it good solution if replace all Length, Copy, Delete with UTF8Length, UTF8Copy, UTF8Delete and read the strings through with this for i:=1 to UTF8Length(s) do Write(UTF8Copy(s,i,1))?

I do not want to think of where Length, Copy, Delete is good and where UTF8* needed.

Gabor

Gabor Boros wrote

2016. 10. 21. 10:25 keltezéssel, Juha Manninen via Lazarus írta: I read, I read but if contains buggy example... ;-)

Mattias fixed the bug.

I need a quick and a rock solid solution. Is it good solution if replace all Length, Copy, Delete with UTF8Length, UTF8Copy, UTF8Delete and read the strings through with this for i:=1 to UTF8Length(s) do Write(UTF8Copy(s,i,1))?

No, it not a good solution!
I predict that most your code can still use byte indexing. At some point you will get a Heureka-moment like "hey, I don't need the codepoint index when doing this task!".

I do not want to think of where Length, Copy, Delete is good and where UTF8* needed.

Well, you must think when coding. There is no shortcut. :)

BTW, if you are worried about Delphi compatibility there is now unit LazUnicode available.
See:
http://wiki.freepascal.org/Better_Unicode_Support_in_Lazarus#CodePoint_functions_for_encoding_agnostic_code

Juha

Gabor Boros wrote

2016. 10. 21. 10:25 keltezéssel, Juha Manninen via Lazarus írta: > * Please read the wiki page ... I read, I read but if contains buggy example... ;-)

Yes, this can be very frustrating...
Documenation is one of the major drawbacks of Free Pascal/Lazarus.

I need a quick and a rock solid solution. Is it good solution if replace all Length, Copy, Delete with UTF8Length, UTF8Copy, UTF8Delete and read the strings through with this for i:=1 to UTF8Length(s) do Write(UTF8Copy(s,i,1))? I do not want to think of where Length, Copy, Delete is good and where UTF8* needed.

I think if you want to use unicode (which is IMO unavoidable today) then UTF8 is a good choice (see http://utf8everywhere.org ) and then you have to cope with the encoding anyway. Byte and character position are not related anymore, neither in UTF-8 nor in UTF-16.
Only UTF-32 provides this but wastes a lot of memory.

But in many cases you do not need the character position. To find a substring, you only need the byte position. You can then delete this character from the byte position and insert another one. Of course, you need to delete as many bytes as the character consists of.

If you realy need the character position/length, then you have to use UTF8Length/UTF8Copy/etc.
But as said: It is only needed in special circumstances. Still you have to know when to use what.

2016. 10. 21. 12:38 keltezéssel, Juha Manninen via Lazarus írta:

Delphi compatibility not needed for me, but I am a silly coder and don't understand why the wiki say you can use Length, Copy, Delete with UTF8. See two examples below. First is a Lazarus project with a simple editbox, if press á (Alt+160) the form caption show 2. The second example is a console project (in Lazarus also), if press á (Alt+160) then Enter see 1 as result. Same FCP same Lazarus. Why is there a difference in the result?

1.
procedure TForm1.Edit1Change(Sender: TObject);
begin
   Caption:=IntToStr(Length(Edit1.Text));
end;
2.
var
   s:string;
begin
   ReadLn(s);
   Write(Length(s));
   ReadLn;
end.

Gabor

2016. 10. 21. 12:48 keltezéssel, Jürgen Hestermann via Lazarus írta:

If you realy need the character position/length, then you have to use UTF8Length/UTF8Copy/etc. But as said: It is only needed in special circumstances. Still you have to know when to use what.

I will know if somebody describe what a difference between á and an á characters in two points of my program. I answered to Juha's reply with two short examples. Don't understand why an á character/string different for example from a ReadLn and from Edit1.Text.

Gabor

Lars wrote

Indeed this is a serious problem these days, unicode.. which is almost a virus. In GoLang they use something called "Runes" to try and solve the problem.

I had to search about what "runes" in GoLang mean. I found: ---
"Code point" is a bit of a mouthful, so Go introduces a shorter term for the concept: rune. The term appears in the libraries and source code, and means exactly the same as "code point", with one interesting addition.
The Go language defines the word rune as an alias for the type int32, so programs can be clear when an integer value represents a code point.
---
So it is a new name for CodePoint. Great. It does not sound very useful to me. I hope they don't do something as stupid as Python 3 does, converting all string data internally to UTF-32.

Off topic but I wonder if Lazarus/fpc uses something anything similar to golang's rune's approach or looked into it.

Yes but we call it "CodePoint" like rest of the world does. CodePoints are the easy part of Unicode, regardless of encoding! Look at the examples here:
http://wiki.freepascal.org/UTF8_strings_and_characters They can handle pretty much any use case dealing with CodePoints. It is not difficult. It is easy.

Your worries about complexity of Unicode are valid but the reason is combining CodePoints into user perceived characters. The rules are complex, there is normalization and its associated problems etc. No, neither FPC nor Lazarus have library code to deal with that yet. The goal is to have an enumerator for user perceived characters, just like LazUnicode unit has for encoding agnostic CodePoints.

Juha

Gabor Boros wrote

Same FCP same Lazarus. Why is there a difference in the result?

You still did not read the wiki page:
http://wiki.freepascal.org/Better_Unicode_Support_in_Lazarus Console programs are mentioned in many places. This is under "Usage in Lazarus": "For console programs (no LCL) a dependency for LazUtils must be added manually.
LCL applications already have it through the LCL dependency."

Juha

2016. 10. 21. 12:38 keltezéssel, Juha Manninen via Lazarus írta:

Juha, thank you for your patience and sorry if I am a completely idiot but... :-)

Why the below example better than a for loop with UTF8Length and UTF8Copy for go through the string?

I hope my "Heureka-moment" coming shortly! :D

http://wiki.freepascal.org/UTF8_strings_and_characters#Iterating_over_string_analysing_individual_codepoints

Gabor

Gabor Boros wrote

Why the below example better than a for loop with UTF8Length and UTF8Copy for go through the string?

Because it is MUCH faster. It scales linearly, O(n). Calling UTF8Length() and UTF8Copy() inside the loop makes it polynomial O(n^2) or worse depending on how many UTF8...() calls you have there.

Yes, we have seen complaints that UTF-8 is unusable because you must use the slow UTF8Length() and UTF8Copy(), and UTF-16 is better because you can use fixed width S[i] indexing.
That is obviously based on misunderstanding of both encodings.

Hint: if you need to iterate CodePoints, you can also use the enumerator from LazUnicode unit. It uses the same concept as the example in wiki page. It allows this code:

  for ch in s do
    writeln('ch=',ch);

and the same code even works in Delphi with UTF-16. Cool, ha!?

Juha

And again we are at the point where you need to understand what goes on under the hood... ;-)

Gabor Boros wrote

I will know if somebody describe what a difference between á and an á characters in two points of my program.

The problem is, that Unicode has a code point for "á" but also allows to compose this characters by having an "a" and an "´" printed over each over.
I will never understand why this was allowed because I thought that Unicode was intruduced to overcome such issues by defining a huge number of code points directly.

Nevertheless, if you have such a situation then you cannot search for a byte sequence as there are 2 possible representations of the same character.

I have given up on taking care about such composed characters and assume that all Unicode strings are normalized.

Jürgen Hestermann wrote

And again we are at the point where you need to understand what goes on under the hood... ;-)

Yes but that is true with any programming.
I am truly happy that we have Unicode instead of the old system codepages. I remember text full of question marks earlier a lot but not any more. Things are getting better...
I don't even know how the codepages worked when one text had many languages. I don't even care now because we have Unicode. :)

That is all true although Gabor's problem was not caused by it. His LCL app used the default UTF-8 strings but the console program used Windows codepage.
Adding to the confusion, Windows console codepage is different from its system codepage (if I have understood right). This is another reason to use the default UTF-8 system, it handles it all behind the scenes.

I have given up on taking care about such composed characters and assume that all Unicode strings are normalized.

I have understood the composed version (many codepoints / character) is the recommended normalized one.
We must support it properly in future.
The combining rules are extremely complex. Benjamin Rosseaux (BeRo in forum) has code for it. There was some other code, too. I must dive into it sometime in future.

In fact we have simple code for combined accented characters in LazUnicode unit, despite of what I wrote earlier in this thread. It was basically copied from SynEdit. I will write another post...

Juha

Juha Manninen wrote

No, neither FPC nor Lazarus have library code to deal with [combined CodePoints] yet. The goal is to have an enumerator for user perceived characters, just like LazUnicode unit has for encoding agnostic CodePoints.

Sorry, that was not accurate.
Unit LazUnicode already has TUnicodeCharacterEnumerator which is able to iterate combined accented Unicode characters.
It calls either function UTF8IsCombining or UTF16IsCombining depending on the default encoding in use. Yes, Delphi and UTF-16 are supported. The code was basically copied from SynEdit and then ported also to UTF-16. It does not support all the complex rules of combining CodePoints, but it apparently works well for accented characters in western languages.

This:
operator Enumerator(A: String): TUnicodeCharacterEnumerator; would enable it for the for-in loop, but it is commented out now. The current for-in loop enumerator works with CodePoints.

There is a test project in components/lazutils/test/LazUnicodeTest.lpi. It includes combining CodePoints, too. Please take a look if you are interested.

Juha

Juha Manninen wrote

UTF-16. It does not support all the complex rules of combining CodePoints, but it apparently works well for accented characters in western languages.

Which ones does it not support?
When I added it to SynEdit it was complete. It had all the combinings that the utf8 standard had back then. (at least that I could find in the documentation)

Of course if a new combining range is added, it will not contain it. If that is needed one needs an external (OS or otherwise) library, that can/will be updated on those occasions.

Mind "combining codepoints" have nothing to do with how many codepoints will be represented by one glyph.

"â" is one character. But it can be a single codepoint (in utf16 one code-unit or word // in utf8 several code-unit or byte), or 2 codepoints ("a" + combining "^").
"fi" are 2 chars. But the may be 2 or 1 glyph (ligature)

It is my understanding (but I do not know for sure) that in some languages (such as Arabic) certain letter combinations form a single glyph (afaik/google see https://en.wikipedia.org/wiki/Hamzah combined with a letter). Though maybe it is considered 2 glyph? I do not know in some scripts glyphs are displayed in an order different from their occurrence in the text.
All of this however has nothing to do with combining codepoints, or what counts a character.

https://en.wikipedia.org/wiki/Typographic_ligature

Scary: It may depend on the font what letters are combined to a ligature. Even English can have them.

Mattias

Mattias Gaertner wrote

Maybe you mean ligatures? Many languages have them, even German: https://en.wikipedia.org/wiki/Typographic_ligature

I thought that ligatures are just a matter of the font but not the unicode representation?
When I write a text which contains the two letters "fi" they should be two separate characters in my unicode string no matter whether they will be printed as a ligature on the printer or screen. So ligatures should not influence string encoding in FPC. Or am I missing something here?

Ok, I was confusing the Unicode terms again.
I guess the biggest complexity is in glyphs and ligatures. I still don't understand their details.
However for a program that must care about Unicode, like a text layout app, the rules for combining codepoints and glyphs are equally important. Codepoints for one glyph should never be split or copied separately. Isn't it so?
SynEdit is a text layout app, too.
In that sense the function IsCombining is not enough for practical purposes. A comprehensive library function should take care of glyphs (+ other rules), too.

I looked at Bero's PUCU and the other links:
http://forum.lazarus.freepascal.org/index.php/topic,33064.msg214342.html#msg214342 but it went over my head. I must study the issue more later.

* A reality check! *
Despite problems and incompleteness of our Unicode support, it is actually better than most other solutions out there. Ok, most programming tools support Unicode somehow but people use them wrong. A good example is our forum SMF software. It deals with text layout and definitely should handle Unicode but it does not. Not even single Codepoints beyond BMP which should be the most easy case! No combining rules needed or anything.
Try to add this text to a forum post: (I hope the mail SW can deal with it...) "Have 🍷 for FPC 💓 Lazarus."

Now the fact is that code made with FPC / Lazarus using the LazUnicode functions and enumerators supports Unicode already much better than most code out there!

Juha

Jürgen Hestermann wrote

So ligatures should not influence string encoding in FPC. Or am I missing something here?

I guess it matters for a text layout software. It should not separate the two characters forming a ligature.
I admit I don't know the issue, figuring out the details myself.

Juha

Ligatures are a group of different issues.
The "fi" ligature is a "stylistic ligature", aka just a font issue and as such is always represented by the two Unicode codepoints.

The wiki page describes various other types of ligatures, where the Unicode representation can vary.

Languages don't care about programmers.

Mattias

[...] I guess the biggest complexity is in glyphs and ligatures. I still don't understand their details.

There is nothing to understand. Some languages have irregular letters. Same as English has irregular verbs. You don't "understand" them, you simply learn them.
As a programmer you don't need to learn them, but you should be aware that many languages can't be mapped to simple arrays of characters.

However for a program that must care about Unicode, like a text layout app, the rules for combining codepoints and glyphs are equally important. Codepoints for one glyph should never be split or copied separately. Isn't it so?

"Never" is wrong here. For example some editors allow to select the single letters of a ligature. Also when comparing words you may want to ignore the diacritical signs using the decomposed form of Unicode. But afaik you are right that most programs never have an issue with ligatures.
Btw, we need a wiki page about collation.

Yes, that is basic Unicode encoding. No ligatures, no bidi. I agree that this is the minimum for supporting Unicode.
Synedit goes much further.
And the native widgets often have pretty good support for the language of the user. So the LCL controls using native widgets have automatically good Unicode support.

Mattias

Mattias Gaertner wrote

Languages don't care about programmers.

True.
But Unicode should have cared.
It was made for its use on computers.
Pressing each and every language peculiarity
into Unicode was a mistake and
made Unicode so hard to use.

No one forces you to consider every "language peculiarity".

Mattias

AFAIK, the only decent advice is never to use the numbers in Pos() / Length() / Copy() / Delete() for anything else tan with these functions. don't try to do any interpretation of these numbers. Never use the term "Character".

-Michael

Jürgen Hestermann wrote

But Unicode should have cared. It was made for its use on computers.

I don't think so.

I suppose it was defined top allow for printing out digital documents in mind, but not with working with them.

At least this i what the outcome suggests: printing works just fine, but even trying to find out a very short information is identical is not decently possible.

-Michael

Non sense. The various normal forms aren't needed for printing, but for "working with them". Same for the various encodings like UTF-8 and UTF-16.
Think about the other type systems with diacritics like TeX. That is made for printing documents, not for working with them.

At least this i what the outcome suggests: printing works just fine, but even trying to find out a very short information is identical is not decently possible.

That depends on what you mean with "identical".
I guess you mean the topic "collation". It would be nice if someone with some knowledge about that topic could start a wiki page or fpdoc topic to list the common functions for them.

Mattias

Mattias Gaertner wrote

That depends on what you mean with "identical".

You are absolutely right. Very sorry for being critical while being vague myself (again typing faster than thinking) ;) .

I meant to point out exactly this ambiguity:

identically coded vs. identically looking (e.g. combining codepoints), vs identical presumed letters if looking differently (ligatures), ...

-Michael

[...] but even trying to find out a very short information is identical is not

decently possible.

About "identically coded":
That is "decently possible" - simple string/byte comparison.

About "identically looking":
I guess you mean composed vs decomposed form. That is converting normal forms. There are functions to normalize, but the information is scattered and it would be nice if someone would write a page.

About "ligatures":
I guess you mean "collation". Same problem. Needs better documentation.

Basically you are asking for various compare and normalization functions. These functions exist.

Mattias

Mattias Gaertner wrote

These functions exist.

This of course is great (while the lack of documentation supposedly makes them hard to use).

In fact I am not asking, but the question is part of the OP's problem. And here I wanted to point out the ambiguity of the term "identical" on that behalf.

-Michael

I think that there is a degree of confusion about the use of ligatures. Ligatures (at least in English) are typographical elements, not language elements. Not all typefaces support them and the code for a ligature should never appear in the source text. It is the function of the display software to combine adjacent characters and display the appropriate ligature if and only if the font that is used supports them.

A proportional typeface may display the character sequence 'fl' by using the appropriate ligature glyph. A monospaced typeface would display the same sequence as two characters, as would any typeface that did not include the ligature glyphs.

Ligatures improve the appearance of text but are strictly a display function and shouldn't actually appear in the text itself. This may not be true for other writing systems and other languages but is certainly true for English and perhaps other western European languages as well.

Yah, it's way off topic but I'll bite just this once.

It's poetry. It's not to be taken literally. If your native language is not English, it may give you some problems.

For the record, Rabindranath Tagore was an Indian writer. The quote is from his book "Stray Birds".

What this couplet is saying is that sometimes it is easier to use a fictional account to express something true. As I've said it is poetry, these are metaphors, figures of speech. I find much of Tagore's language very beautiful. We've chosen another Tagore "Stray Bird" as the epitaph on our grave marker.

Here are a couple more from "Stray Birds":

The mind, sharp but not broad, sticks at every point but does not move.

and

A mind all logic is like a knife all blade.
It makes the hand bleed that uses it.

There are many more...