Hi,

is it only me ore are there others who find the following behaviors of the pascal syntax highlighter in lazarus strange and inconsistent and sometimes even annoying?

* some types (String and AnsiString) are highlighted, all other types (built in or not) are not highlighted. IMHO no type names should be highlighted.

* all reserved words for control flow (for, to, if, then, do, else, repeat, until, while, begin, end, try, except, finally, raise, goto) are highlighted, the only ones that are not highlighted are break, continue and exit. IMHO all such reserved words that describe the program structure should be highlighted. For example break and exit are of the same caliber as raise (or even goto) and also used for similar design patterns, I don't see why they should be treated differently.

I have tried to think about it long and hard but I cannot find any reason that would justify these anomalies other than maybe some historical reasons. For example a (hypothetical) reason that string is internally working in a different way than a Double or an Integer IMHO does not justify any different syntax highlighting, they all still belong into the same *category*: they appear in var declarations as type names and are used like type names everywhere no different than any other type name, so they consequently should look like type names and not like language keywords.

Bernd

This is historical. Delphi did it, so we are doing it, too (btw: to keep it consistent UnicodeString should be highlighted, too ^^). Might be related to the fact that the highlighter (SynEdit) was an original Delphi project thus it inherited this behavior.

Reserved words are highlighted. The ones you listed are all reserved words and it's not possible to use them as identifiers (without escaping them with "&"). But "break", "continue" and "exit" aren't reserved keywords. You can even use them as identifiers (I did it once and wondered later on why "exit" isn't working ^^).

Also there are some "context sensitive" reserved words like "index" and "message".
The first one is used in properties, but it can be used as a normal identifier as well. E.g. "property Foo[Index: Integer]: Bar" here index will be highlighted, but it only should be highlighted if it is used as "property Bar index 39 GetBar;". It won't be highlighted outside of a class. "message" is used for the message dispatching system in Object Pascal (procedure Foo(aMsg: TMsgStruct); message Bar;), but it is highlighted everywhere and can be used as an identifier everywhere. Those two are indeed a bit annoying. ^^

I don't know why Borland chose to highlight ShortString, AnsiString and String (WideString is only highlighted by Lazarus), but normal reserved keywords and normal non reserved identifiers are clear. The only troubles make hybrids like "index" and "message".

Regards,
Sven

string is a keyword, integer not (both are build in types)

you can do:
   var
     integer: byte;
but you can not do:
   var
     string: byte;

Martin

String is a keyword. You can not redefine it.
Ansistring is for historical reasons highlighted as keyword too.

break, continue and exit are not keywords. They could be redefined.

I'm sure Martin is already thinking about making this optional. ;)

Mattias

[...] The only troubles make hybrids like "index" and "message".

Actually they are more simple than you think. They are highlighted by the synedit highlighter correct.
The trouble makers are postfix modifiers like 'default' and 'cvar'.

Mattias

But why do we need to highlight those two like Delphi does? Why does "index" need to be bold if it's in "Foo[Index: Integer]" if I'd expect it to be highlighted only in "Bar index 34"? Why does "message" need to be highlighted everywhere if it is a keyword only in "procedure Foo(aMsg: TMsgRec); message Bar;"?

This is what I don't understand.

Regards,
Sven

oops. I confused that. A few cases were implemented, but not all.

Because comfort costs. This has nothing to do with Delphi compatibility, but that the highlighter is optimized for speed not for correctness.

You can create a bug report.

Mattias

For me it looks like compatibility, because Delphi is doing it the same way ^^

You can create a bug report.

I might do that...

Regards,
Sven

Actually "index" and "message" are probably easy to fix.

"deprecated" and friends are a bit more complex, because they may be behind the ";" (property) or within the declaration/before the ";" (var a:byte deprecated;)
It still is easy to implement, but it adds a lot of extra states that the highlighter (seen as state engine) can be in. (those few alone are probably not enough to cause any issues, but I had speed issues before, when adding too many states)

real bugger though is "class"
    type
      a=class; // forward declaration
because it should not be fold-able, but it is
    type
      a=class
      // have some nice multiline comment
      //
     ; // and the semicolon

because on the line "a=class" you can not tell the difference. (unless you scan forward, but that means you have to scan twice)

---
Overall there are a few other things on my todo, before I revisit the highlighter.

Martin

Martin wrote

Overall there are a few other things on my todo, before I revisit the highlighter.

I've lived with that wrong highlighting a long time now, so I don't need to have it fixed now. But I'll create a bug report so that it is not forgotten.

Regards,
Sven

Sven Barth wrote

I've lived with that wrong highlighting a long time now, so I don't need to have it fixed now. But I'll create a bug report so that it is not forgotten.

Reported as http://bugs.freepascal.org/view.php?id=17577

Regards,
Sven

Right.

Question is, what the users expect from a "syntax" highlighter. IMO a user doesn't expect it to really follow the (currently multiple different) syntax variations.

Removing these few special types would both simplify the highlighter and look more consistent.

break, continue and exit are not keywords. They could be redefined.

I'd appreciate when, at least in mode FPC, these words would become keywords. As already mentioned, a redefinition of these procedures can only cause trouble. When they occur in legacy code, it would do no harm when such modules had to be compiled in an different (Delphi) mode.

I'm sure Martin is already thinking about making this optional. ;)

I'm just about implementing more languages (front-ends) for FPC, like Modula. These will require their own syntax highlighters, and then I'll think about providing alternative highlighters even for Pascal. But don't hold your breath, this project requires an restructured compiler and will take some more months.

DoDi

Yes, this would be a good change, similar to restrictiions on the duplicate variable names in the overlapping scopes.
I think, however, that this change has near-zero chance of getting through fpc core team.

One relevant data point: for many years,
I use Delphi for the "Programming 101" university course I teach. Every year, out of approx. 15 students in the group, 5-6 make the mistake of naming some component or variable "exit" or "break", and spend hours or even days trying to find the problem, until they finally give up and ask me for help.

Alexander Klenin wrote

On Sat, Oct 9, 2010 at 14:24, Hans-Peter Diettrich

I would vote for this this, too. Maybe nobody asked the fpc core team, so I cross-post this also to fpc-devel list.
I just realized how important the duplicate variable name restriction is when I tried to convert some existing code. Clearly the original author didn't always know which variable gets used.

Juha

Hans-Peter Diettrich wrote

Question is, what the users expect from a "syntax" highlighter. IMO a user doesn't expect it to really follow the (currently multiple different) syntax variations.

Yes, this is the real underlying question here: Should the highlighter highlight every minor oddity in the language definition in the boldest colors or should it rather highlight the structure of the program as it is seen from the programmer's perspective?

Maybe an additional mode for the highlighter could be introduced (a simple flag that can be set to put it into the new mode), by default it would behave as always and highlight the keywords exactly as they are defined in the language definition (call this the "strict mode") and another mode that tries to highlight an "ideal" Pascal language in which these oddities do not exist and it would highlight the program structure like it is intended and seen by the programmer (call it "relaxed mode" or maybe somebody can find a more intuitive name for these highlighter modes).

The changes to the highlighter are just handful of lines, I quickly hacked something together yesterday evening to see how this mode would look like and it seems to work and I very much like how my programs appear in this mode.

Bernd

Hans-Peter Diettrich wrote

Question is, what the users expect from a "syntax" highlighter.

(can't resist: Usually they expect the impossible :-)

Bernd Kreuss wrote

The changes to the highlighter are just handful of lines, I quickly hacked something together yesterday evening to see how this mode would look like and it seems to work and I very much like how my programs appear in this mode.

Attached is a diff of my changes:

* do not highlight String
* do not highlight AnsiString
* do not highlight WideString
* highlight break
* highlight continue
* highlight exit
* highlight message only in a procedure defnition (partial fix to the bug mentioned earlier, it does not eliminate all cases, judging from the comments in the source a complete fix would need an improved parser in the highlighter, but at least it now gets rid of highlighted "message" anywhere *outside* the line where a procedure is defined.

the above patch is what I now use here locally and I like it much better than the original one, but I don't think these radical changes would be easily accepted by the core developers. It tries to highlight the ideas and intentions in the programmer's mind and not so much the strict and sometimes odd language definition.

Bernd
-------------- next part --------------
A non-text attachment was scrubbed...
Name: highlighter-modification.diff
Type: text/x-diff
Size: 3181 bytes
Desc: not available
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20101009/80e93ee9/attachment.bin>

Thanks for the patch. I gonna changes to "message" are ok as they are.

The rest should be optional.
Instead of checkbocxes to switch them on/off, I probably just make them a new color setting. so you can set the attributes for break,exit.... under somethink like "Pascal Commands" (better idea welcome) Same for the string types, Yes it seems a waste to have a new color-field just for those three, but I guess some people will like the old way (I do)

Martin

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20101009/5b561cb6/attachment.html>

Me too.

Syntax highlighting is just an aid, these are corner cases.

Michael.

2010/10/9 Martin :

Non-reserved keywords? Built-in procedures?

On 10.10.2010 02:01, Alexander Klenin wrote:

Built-in procedures?

This is a bad name. exit, break and continue don't really do things that one would expect from a procedure, they have more in common with things like raise and goto, they are control flow constructs, I associate a break or a continue really as an optional *part* of the while (repeat or for) loops, (a part of the construct *itself*) and an exit as an optional part of a function or procedure, in the very same way like an else belongs to an if or a raise belongs to a try/except/finally.

"quasi keywords"
"keyword-like words"

Bernd

I ve applied the "message" part (with even more improvements) and also some improvements to false "index" highlights. rev 27645

I'll look at the rest later, as it will also need extended config entries.

Martin

All implemented, options under: editor / display / markup

* highlight message only in a procedure defnition (partial fix to the

implemented, and hopefully catching all cases now