Hello,

I'm reading a lot of code of SynEdit, and I find a big logical bug with how things implemented.

Many programming languages such as Python, Ruby, Javascript and few others, are supporting unicode based named for methods, functions, class, variables etc...
It allow everyone that wish to do it, to create for example in non latin chars names, including also umlets and other chars.

The implementation of TSynEdit is to run across all of the ASCII table (0..255) and act only by this. And that's a logical bug imho.

Ido

http://ik.homelinux.org/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20100604/49680db1/attachment.html>

You speak of the SynHighlighter, or the actual SynEdit?

Most of the highlighters are old, and haven't had much maintenance, so they do what was right 10 years ago....

Pascal-Highlighter, can resolve to the trick of treading text of ascii. All pascal relevant keywords are limited to latin chars => and they are the same in Ascii/Ansi or utf8.
If you source contains other chars (multy-byte utf8) in places where this is allowed, the pascal highlighting will continue to work.

SynEdit itself => outside the Highlighter, does full utf8 handling (including (double) full-width Chinese chars).
Otherwise please point out some examples (and make sure you are using SynEdit from SVN/trunk / latest snapshot)

Martin

http://ik.homelinux.org/

Yes I am.

Most of the highlighters are old, and haven't had much maintenance, so they do what was right 10 years ago....

It seems that the only maintained version of syntax highlight is the Pascal version.

On Delphi.NET they also allow multibyte names such as:

procedure ????????(const ?????? : String); ...

If you source contains other chars (multy-byte utf8) in places where this is allowed, the pascal highlighting will continue to work.

There is a need to also add Bi-Directional support for multi-byte support. I want to add it when I'll have some time for it.

Ido
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20100604/bd924d03/attachment.html>

This was an either or question :) But I take it "the highlighter"

Out of the languages yes => because it is used by the IDE

The LFM highlighter should be ok too.
Diff highlighter is new, but again, the relevant tokens are inside the ascii range.. (and it only supports a subset of diff styles anyway)

I have no idea how xml or html react to the full utf8 range => I haven't looked at them, except for adding a rough folding support.

The overall thing is, that 99% of the work on SynEdit applied to the usage inside the IDE. SynEdit as a component for user-app (and that does include all the highlighters) has not yet been much looked after.

Ok, but Lazarus deals with FPC => and FPC doesn't do that yet. And even then => only thing it needs to do, is to make sure it treads them as a single word.

multi-byte as in what?

I know there is multy-byte ascii => but synedit uses utf8. Well in principal it would be nice to support different encodings. But that would require LCL support, because as long as all Keypresses come in utf8.....

bidi:
Nice => but make sure you keep it modular.

I am in the process (long time prospect) to rip that whole one big single class into pieces. => So try not to put new stuff into the main class.

Martin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20100604/1524688c/attachment-0001.html>

More importantly, so does D2009/D2010. So probably something like this will come sooner or later. (but admitted, currently it looks like later :-)

2010/6/4 ik :

The implementation of TSynEdit is to run across all of the ASCII table (0..255) and act only by this. And that's a logical bug imho.

Not really. UTF-8 is implemented in such a way that it is indeed possible to treat it as Extended ASCII in this case.

But there indeed is a bug, it doesn't recognize non-english identifiers and codetools can't handle them too.

On Sat, Jun 5, 2010 at 09:06, Felipe Monteiro de Carvalho <

How is so ? here is a multi-byte char: ? . It takes more then a word to be used, so you can not do S[i] because it will provide you only part of the char (one byte).

But there indeed is a bug, it doesn't recognize non-english identifiers and codetools can't handle them too.

Ido
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20100605/da7f17ed/attachment.html>

2010/6/5 ik :

How is so ? here is a multi-byte char: ? . It takes more then a word to be used,

UTF-8 implements it's text support in such an way that characters which require more then 1 byte are formed by using only valid ASCII or Extended ASCII values. For example, save your ? in a text file and in a UNIX shell use "od -x file.txt"

You will see that this character is represented as 90 d7 in UTF-8

Or in Decimal: 215 144

Now go to the extended ASCII table and you will see that both are valid Extended ASCII values: http://www.asciitable.com/

The same is valid for all other UTF-8 characters.

so you can not do S[i] because it will provide you only part of the char (one byte).

S[i] returns a byte, not a character. If you character has 2 bytes then S[i] will return the first byte and S[i+1] will return the second byte.

So, it doesn't matter if this part of SynEdit thinks that your identifier is actually 2 characters which read "??", the corresponding Extended ASCII for your original character. It works just the same.

Example?
Does fpc handle them ?

I know
var ?ber: Integer
does not work, but it does not compile either (last time I checked)

Martin

Martin wrote

I know ?var ?ber: Integer does not work, but it does not compile either (last time I checked)

I know it doesn't work, I was just saying that you don't need to change a lot of code to fix it. The problem isn't in the logic, it's just a implementation problem. The original mail was claiming it was a big logic error.

I would guess that to build the identifiers he uses a table to check if the byte is a valid char for identifier, this table should also include extended ASCII chars and then this would work in Lazarus.

For FPC no idea how to fix, but probably in a similar way.

Identifier scanning in unicode enabled languages is usually changed from determining which chars are allowed to which ones are explicitely not allowed. (whitespace, first char not a (standard) number) etc.