A while ago, I started adding support for mixed LTR/RTL text in SynEdit.

The actual display of RTL text now works (that is, if you have some arabic chars in the text, they display RTL, and the caret moves accordingly / caret between RTL and LTR always means caret at LTR). uf8 LTR/RTL markers are not supported. This is absolute basics only.

Unfortunately with RTL came other unicode features, that sofar no one had missed. Those are at the very least
- combining codepoints
- ligatures
- maybe reordering of codepoints.
- other?
They are tasks of different extent. And I need to find out what is mandatory, and what optional. So I can then decide, what does fit into my schedule.

The current state is:
- combining: Only Arabic has been done (but they should be complete). So none Arabic RTL will not work.
- ligatures: see below
- reordering: not researched, hopefully optional.

"work"
means, that the text is stable (except ligatures, only with workaround), and does not expand/shrink, when selecting text, or moving the caret. Also that the caret will be at the correct pos. A newly inserted char will be where the caret was. Can be tested by hitting the "end" key, and see if the caret is at the end of visual text. If SynEdit thinks the text is shorter/longer than the actual painted display, then there is an issue.

ligatures:
The editor does not handle ligatures yet. So it calculates 2 screen cells, when only one is needed. However a stable "workaround" exists (currently depends on config)

On windows and windows only (others will be done, if that turns out to be any good). In Options / Editor / Display / set "Extra CHAR spacing" to 1 This will slightly widen the script, ignore that, its temporary. Requires a proper monospaced font. (Deja vu mono)

What it will do: It will tell windows, that the ligature is expected to cover 2 display cells.
Display: Arabic text is a script, glyphs are connected by a continuous line. The ligature will be in one cell, the next cell will be empty, except for the connecting line.
Editing: The caret can be at either cell. Each cell stands for one of the 2 chars in the ligature. So the 2nd char can be edited, if the caret is at the empty cell

------------------
I need feedback from people who actually speak (or at least read and write) Arabic. I need to know, if the above situation is "useable".

If so, then:
- it can be fixed to work without the extra char spacing
- on gtk, carbon, qt (well at least I hope)
- combining can be added for other languages.

If not, well I don't know yet.

Martin wrote

A while ago, I started adding support for mixed LTR/RTL text in SynEdit.

I forgot: trunk 1.1 only

A while ago, I started adding support for mixed LTR/RTL text in SynEdit.

Great!

[...] I need feedback from people who actually speak (or at least read and write) Arabic. I need to know, if the above situation is "useable".

Attached is a small example text and a screenshot showing, how Firefox with font "serif"renders it, how synedit with "Courier New" renders it and how synedit with "monospace" renders it.

Firefox and the "monospace" synedit shows only one ligature (??), combined of the two codepoints alif ? and lam ?. Firefox allows to select the two independently.

The "courier new" font shows another ligature:
The ??? are three letters in "monospace", while with "courier new" it shows the two rightmost combined as one.

Mattias
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed... Name: arabic2.txt
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20121207/c0acceb4/attachment-0001.txt> -------------- next part --------------
A non-text attachment was scrubbed...
Name: arabic.png
Type: image/png
Size: 22265 bytes
Desc: not available
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20121207/c0acceb4/attachment-0001.png>

Was the synedit picture taken with extra char spacing? Because without that, you get the wrong behaviour (select line, char by char)

I know how it should be shown ideally.

The question is how far away from ideal would still be useful? And that I can not tell for a script, that I do not once said, some editors display the ligatures, as 2 chars. And that would be accepted by many people. But I can not judge that.

The proper ligature handling is a lot more work. And it depends on the font (or it may at least) On windows there is an API, that would allow to get that info. However if not carefully done, it may have effect on the speed of synedit (quite possible noticeable).
On others, I wouldn't even know where to look for such an API. But little point, until SynEdit is ready to use an external API in this place. (That is it does not call it excessively)

As for placing the caret in the middle of a char, that would require 90% of the work to render proportional fonts (on the list, but not now)

Below is with extra char spacing, and you can see a stretched bit of horizontal line

Without the etra char spacing, there is a gap, between the RTL and the LTR spaces atthe end (use visible spaces, r type real latin text at the end).
Delete half the ligature, and it will go away, causing the LTR text to move, even so the RTL has still the same length (ligature replaced by remaining normal char)

So at the moment for synEdit the ligature are 2 chars, they must take to cells on screen.

The point is. It might be some time until I can add proper ligature handling.
But I can try to get the current behaviour without the extra char spacing (if that current behaviour is of any use). It might be possible to force the splitting of the ligature, I do not know that for sure. Also GTK2 will probably split it, others I do not know. (GTK2, will have to pain every char on its own.) -------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20121207/33937c7d/attachment.html> -------------- next part --------------
A non-text attachment was scrubbed...
Name: gjadibdc.png
Type: image/png
Size: 521 bytes
Desc: not available
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20121207/33937c7d/attachment.png>

This is ok in cases like the IDE editor where the document is mainly English. I suppose it will be somehow odd for documents with mainly RTL languages. Formatting (like indentation, bullets), where implemented, will suffer.

I am not aware of any need for reordering.

I have tested on Linux/gtk2 (ubuntu 11.04), and courier new only:
- The attached snapshot (lines 29 and 30) shows an extra space before the 456.
- Long connecting lines are not what I would like, but this is a monospaced font afterall.
- The 456 should have come to the left of the Arabic words.
- If you put a shaddah or damma on a character, it gets displayed on top of the character (correct behaviour). Pressing backspace at this stage should only delete that addition, and not the character.

Stephano
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ArabicSynedit.jpg
Type: image/jpeg
Size: 22590 bytes
Desc: not available
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20121207/adea9783/attachment-0001.jpg>

Martin wrote

A while ago, I started adding support for mixed LTR/RTL text in SynEdit.

Thanks a lot, Martin.

I took the liberty of posting to the forum in an "Arabic issues" thread: http://lazarus.freepascal.org/index.php/topic,17903.msg108631.html#msg108631

Thanks,
Reinier

It was taken under Linux/gtk2 with ExtraCharSpace=0. Setting ExtraCharSpace to 1 shows the characters separated.

I know how it should be shown ideally.

Then you are ahead of most scientists. ;)

The question is how far away from ideal would still be useful? And that I can not tell for a script, that I do not use at all. Someone once said, some editors display the ligatures, as 2 chars. And that would be accepted by many people. But I can not judge that.

The ligatures are not important at the beginning. Later at least the alif+lam ligature is needed.
It is crucial that the connected characters are shown connected.

Maybe it can be done by a plugin?
Then the LCL+synedit do not need to implement it fully, but users can implement the subset they need.

As for placing the caret in the middle of a char, that would require 90% of the work to render proportional fonts (on the list, but not now)

IMO the placing in the middle is somewhat strange. Perhaps showing a special caret or icon is doable. That has low priority.

Below is with extra char spacing, and you can see a stretched bit of horizontal line

I see a long connection between the second and third from the left (the alif and the jim).
And I see the alif-lam ligature.
See the attachment.

Mattias
-------------- next part --------------
A non-text attachment was scrubbed...
Name: arabic2.png
Type: image/png
Size: 1095 bytes
Desc: not available
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20121207/491694e4/attachment.png>

Did you use "Extra Char Spacing" = 1 ? This is what happens, if not! (This and a few other real oddities)

And also, it can only be tested on windows. Because on GTK,QT,Carbon "Extra Char Spacing" is faulty in an other way: It splits the combining chars into individuals, but since SynEdit does not know.....

The problem is, that by current design, SynEdit has to calculate the pixel pos of each char on it's own.If it does not calculate the same, as the OS did when painting (SymEdit gives the OS tokens, fragments of the line or the whole line) then obviously things will be odd afterwards.

- Long connecting lines are not what I would like, but this is a monospaced font afterall.

Ok, but can you test them on windows, with "Extra Char Spacing" = 1

See that the caret pos is treadet correct, backspace and delete, insert work (on the correct char) on them, Copying a selection will copy the highlighted part (except column mode selection, which is not done yet)

About editing. (backspace and delete, insert)
- combining chars see below.
- ligatures. Caret and selection-wise the ligature, and the long-connecting-line, are both treaded as one char. One is the 1st, the other the 2nd char of the ligature (in the order they occur in text). The behaviour for editing should reflect this. Does it.

- The 456 should have come to the left of the Arabic words.

Ok, that could dbe fixed. Depends on treating digits as weak or strong LTR. Actually in this case, depends on treating the line end as such)

If the 456 were embedded in the middle of arab, it would have worked. But they border the EOL, and SynEdit treats the EOL strong LTR (and bordering weak 456 follows). This gives better result for pascal, where Arab occurs in strings. "a:='arab';" The '; in the end will and should be LTR due to bordering the EOL.

This will be fixed eventually, when weak handling is made highlighter depending

- If you put a shaddah or damma on a character, it gets displayed on top of the character (correct behaviour). Pressing backspace at this stage should only delete that addition, and not the character.

Ok, Also simple to fix. Not a painting issue so.

Those are combining codepoints. So backspace must act on codepoints.

The editor understands the diff between "Char" and "codepoint". It is a question of assigning the right choice to each action (and that is a question of writing testcases too)

----------------------
About "Long connecting lines are not what I would like."...

I understand. And it would not be the final solution. But if all else works (as described above) then this is a solution, that I believe, I can reach without too much extra work from where I am now (Will still be next year...).

And then we had something at least use-able.

The rest will be on my todo list, and has to await it's time, between other features and debugger.

Mattias Gaertner wrote

> I know how it should be shown ideally. Then you are ahead of most scientists. ;)

Well ok, that was not 100% true.

The utf8 docs are a lot of text, and no I did not read most of it.

But I looked at other apps (LibreOffice, Notepad++, and what windows does, if SynEdit does not give extra instruction)

Also I mainly referred to the handling of ligatures (like the one discussed. Which ideally should occupy one cell in the text.

Hi,
Good feature for me, but my question (Off Topic), why you interested in this feature while there is no many Arabic/RTL Lazarus users?

For me, I will try to test it, and i like to look at the code too.

Best Regards
Zaher Dirkey
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20121208/8f93c9ab/attachment.html>

From the first trying, Wow it works :D, but i need more tests.

Zaher Dirkey wrote

From the first trying, Wow it works :D, but i need more tests.

Main question at current are the ligatures, with the long line.

1) Acceptable?

2) BEhaves as described: the editor treads the long-line, as the 2nd char in the ligature, (if you delete it, it will delete the correct half of the ligature)
(At least windows, with extra-char-spacing=1)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20121208/26a121af/attachment-0001.html>

patspiper wrote

- If you put a shaddah or damma on a character, it gets displayed on top of the character (correct behaviour). Pressing backspace at this stage should only delete that addition, and not the character.

Does that also apply for "delete" (as in delete next char, opposite of backspace).

Since that deletes the actual char, it would leave the shaddah. So I guess in that case better delete all?

Someone did ask for it... No personal interest

(No, I don't automatically all I am ask for. )
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20121208/838eea7d/attachment.html>

Martin wrote

1) Acceptable?

Yes, and it is wonderful :)

2) BEhaves as described: the editor treads the long-line, as the 2nd char

in the ligature, (if you delete it, it will delete the correct half of the ligature) (At least windows, with extra-char-spacing=1)

I tested it and It is work fine until now.

Best Regards
Zaher Dirkey
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20121208/1a3755a5/attachment.html>

I see there is a unit, and you are use it SynEditTextBidiChars.

So What for USE_UTF8BIDI_LCL? and this units FreeBidi and utf8bidi?

Zaher Dirkey wrote

I see there is a unit, and you are use it SynEditTextBidiChars. So What for USE_UTF8BIDI_LCL? and this units FreeBidi and utf8bidi?

Those 2 units seem way incomplete.

SynEdit currently needs 2 things:

1) combining chars (that may move to LazUtils.. Lets see

2) stronk and weak RTL/LTR detection.
That is definetly not gcomplete in the above 2 units. And SynEdit needs to integrate the highlighter in that.

',+; are all weak

but in Pascal:
a:='arab1'+'arab2+''more arab';

the '+' must be LTR
the +'' must be RTL (all in string)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20121208/c8c0431f/attachment.html>

Delete removes both as well.

Since that deletes the actual char, it would leave the shaddah. So I guess in that case better delete all?

I tested with Libre Office, and the backspace removes the shaddah while delete removes both.

I can give a practical example where this is behaviour needed. When one types a character and follows it by a fat7a, but by mistake types a damma, then backspace should not necessitate the typing of the original character once again.

Stephano

I use Ubuntu, and tested only monospaced fonts (source editor). I understand extra char spacing is for Windows only.

To test proportional fonts, I put a TSynEdit on a form, set the font to Arial, but the characters were all detached. Did I miss something here?

Column selection is fantastic for coding, but is not a prime feature for Arabic at this stage.

OK

Longer connecting lines for monospaced fonts is acceptable especially that the characters have a fixed width. But it is less tolerable with proportional fonts.

Excellent work!

Stephano

patspiper wrote

On 07/12/12 16:58, Martin wrote: > Ok, but can you test them on windows, with "Extra Char Spacing" = 1 I use Ubuntu, and tested only monospaced fonts (source editor). I understand extra char spacing is for Windows only.

On GTK, it will break either way.

The whole think can currently only be tested on windows.

Ok, bad news, I did some more testing myself, and it turns out, that WIndows allocates the extra space (song connection line) just anywhere in the word, or at the end of it, but not always where the ligation is.

The problem is synEdit expects the chars (3+1 ligate = 8 chars) *evenly* distributed (every n pixel). So if the caret is placed at the begin of the line , SynEdit expects a char there, and then n pixel further the next...
If you edit (insert, delete) at those points SynEdit will act as if the char it expected had been there. And that in (a far as I can see) not really useable.

Anyway on windows this is now the current behaviour. (*NO* "Extra char spacing" needed any more).
(Note: instead of the long line, you may get empty spaces, at word borders (including letter to digit changes))

On GTK, QT, and carbon, the widgetset drawing support does not deal with that, so results are even worse.
If windows had worked better, the plan was to try and fix them (Though I am not sure about doing QT or carbon myself). Now I am not sure, how much sense this makes...

As for other RTL behaviour (none drawing related):
- I fixed backspace for combining
- If you find an combining chars that do not work, maybe they got missed (SynEdit has it's own list). Let me know, I will check, and add them, (adding is easy, finding takes the time).
This is Arabic ONLY, other combining have not yet added (planned, no need to report, I do know already)
- As for the treatment of weak chars (e.g. digits being part of the RTL or LTR run. This is still beta. It has flaws, I know, no need to report.
- tabs in RTL, are probably not handled very smart. If you know a good algorithm... However, this will be very low prior.
- Column mode selection, is *not* implemented at all (normal selection will work)
- Caret, if between a RTL and LTR run, is always placed next to the LTR char. I know there a many smarter ways to do this, but low prior.
- Anything else, that needs improvement or fixing: let me know.

So what's next.
The only way to support this, is to make SynEdit aware of the ligatures. And that is a lot of work, so it will take more time. And I don't yet know when I will schedule it....

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20121213/31c4adfd/attachment.html> -------------- next part --------------
A non-text attachment was scrubbed...
Name: jjhfaiaf.png
Type: image/png
Size: 702 bytes
Desc: not available
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20121213/31c4adfd/attachment.png>

.....

So what's next. The only way to support this, is to make SynEdit aware of the ligatures. And that is a lot of work, so it will take more time. And I don't yet know when I will schedule it....

Ok, I found a quick way to get a use-able behaviour.

* it is currently WINDOWS ONLY
* It will NOT be enabled by default in the IDE (but if anyone needs it, you can add it for your own projects, or even the IDE)

- define WithSynExperimentalCharWidth and it should work (need to recompile SynEdit package)
- define SynSystemWidthChars for log messages (should there be problems)

- Adds a small slowdown, but hardly noticeable
- not limited to Arabic, should do all languages, if windows does
- according to my tests it works for lines up to 32001 chars. After that the OS does not handle the line (SynEdit would need to split it) In this case SynEdit behaves (for that line) as if the define was not present.
- ligatures are handled as follows.
There is no middle caret
Depending on the caret being before or after (and accordingly backspace or delete being used) the first or 2nd char is deleted # So it is 2 chars, but any caret move will just be translated into skipping the middle pos

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20121214/df6202e0/attachment.html>

:)

* it is currently WINDOWS ONLY

:(

When you mention ligatures, do you mean any 2 connected characters (ex: ???), or characters that combine such as Lam Alef (??)?

I tested the behaviour of Libre Office Writer (Ubuntu):

- Damma or shadda or similar will combine with the affected character and form one character as far as the cursor movement is concerned. Backspace after the combined character will remove the damma. Delete before the character will remove both.

- Lam Alef produced by pressing a single keyboard key (?) acts as a single character in all aspects.

- Lam followed by Alef will combine visually into one character (??) but acts as 2 characters in all aspects.

Stephano
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20121214/708a981b/attachment.html>

For GTK, I believe this functionhttp://developer.gnome.org/pango/stable/pango-Text-Processing.html#pango-get-log-attrs can deliver the information needed. But in addition there may be a need for further fixes/improvements in ExtTextOut (widgetset)

Other WidgetSets: No idea

If anyone wants to give GTK a try, I will mail what needs to be done.

Lam Alef

combining codepoints (such as dots or accents added to chars) are handled diferrently (SynEdit knows them for Arabic Only):
- Backspace (from behind the char), deletes one codepoint
- Delete (from before the char) deletes char + all combining (leaving the dot without the char makes little sense?)

combining codepoints not known to synedit (none arabic), will act like ligatures.

See above

I can not test (I don't know haw to write Arabic, except for hitting random keys on an Arabic layout). If the single keystroke, produces a single utf8 char, then yes.

SynEdit works on Utf8 Codepoints. Except where it finds combining Codepoints, if they are in the list of combining, that SynEdit includes.

----------
On addition, it asks Windows, if any sequence of codepoints is drawn as a single glyph. This is used to calculate the position of the caret, in relation to each glyph, as well as the length of the line, and where to apply highlights.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20121214/19d67963/attachment-0001.html>