Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / fpc-pascal / 2001-11

OT: Syntax Highlighting

6 messages · 3 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

Unknown author

Sorry for the off topic...

I have a text-mode file viewer that I wrote for Win32 and Go32. I'm trying to add a new feature; syntax highlighting. I think it would be fairly unique to have a viewer that understands a languages syntax (I know any editor of even reasonable power has done this for years -- which I find to be totally indispensable now -- but I thought I could derive a similar benefit from a viewer that had the same ability).

Every implementation that I've coded thus far has either had some type of rendering problem and/or been horribly slow. The one I thought would have the most promise was loading a bunch of keywords in memory and then scanning each line of the display everytime a movement command (pgup/pgdn/arrows etc) was issued for words that matched the ones loaded in memory. But unfortunately pos() has proven to be way too slow for this, seeing as how the list of potential words to be highlighted could be a few hundred, so when I run the program on anything less then a screaming-fast PC it redraws the screen terribly slow and uneven. Sigh...

I've looked at shedit in the FCL, but I'm afraid I don't understand objects enough to be able to make much sense of the syntax highlighting code in their. So, my question is this; does anyone know of another method for syntax highlighting other then using pos() on each line? Is there a way to make pos() faster? Is there something better then pos() to check if a string resides within another string?

Any concepts on how to do this -- or even better, some example code -- would be greatly appreciated. TIA...

Jim

Reply by email

Gabor DEAK JAHN
to Unknown author

At 11/8/01 08:28 AM, you wrote:

Jim,

their. So, my question is this; does anyone know of another method for syntax highlighting other then using pos() on each line? Is there a way to make pos() faster? Is there something better then pos() to check if a

string

resides within another string?

You should stick to simple keyword highlighting in the beginning, real syntax highlighting is a very complicated business, requiring theoretical and practical familiarity with many concepts used in compiler design.

But even only to find keywords, pos() or anything similar is a completely impractical approach. What you have to do is to write a lexical scanner to scan your input and to turn it into tokens. The details also depend on the language you want to display but, basically, it is as follows: you have a character variable to keep the next character not yet consumed. You write a function to return the next token in the input (an abstract value like identifier, number, punctuation, etc). This NextToken function examines the current-character and decides what to do:

Literal := ''
if end of input stream reached then
return END_TOKEN
else if current-character = whitespace then

   repeat
     append current-character to Literal
     read next character into current-character
   until current-character is not whitespace
   return WHITESPACE_TOKEN

else if current-character = alphabetic then

   repeat
     append current-character to Literal
     read next character into current-character
   until current-character is not alphabetic
   return KEYWORD_TOKEN

else if current-character = numeric then

   repeat
     append current-character to Literal
     read next character into current-character
   until current-character is not numeric
   return NUMBER_TOKEN

else if current-character = quote then

   repeat
     append current-character to Literal
     read next character into current-character
   until current-character is quote
   append current-character to Literal
   read next character into current-character
   return QUOTE_TOKEN

else
append current-character to Literal
read next character into current-character
return OTHER_TOKEN

So, for each call of NextToken, you receive back a xxx_TOKEN value describing what kind of token you have just found in the input stream, plus the actual keyword or number in the Literal string. You always have to take care of leaving this function with the next character of the input already read in and stored into current-character so that the next call of NextToken will have something to work on (this is called one character look-ahead). Before you call NextToken for the first time (or, if you scan each line separately, at the beginning of each line), you have to read the first character into current-character yourself.

read first character into current-character
change to Normal color
repeat
   Token := NextToken
   if Token = KEYWORD_TOKEN then
     if Literal is a keyword then
       change to Keyword color
   else if Token = NUMBER_TOKEN then
     change to Number color
   else if Token = QUOTE_TOKEN then
     change to Quote color
   write Literal
   change back to Normal color
until Token = END_TOKEN

When you receive a KEYWORD_TOKEN, you already know that you have a potential keyword in Literal. Now you have to check whether it is one of the actual keywords. Two data structures are practically useful for this, a sorted table of keywords with binary search or a hash table. In the first phase of writing this program, you could simply use TSortedCollection to store your keywords and to look them up. It won't be optimal in terms of speed but it is already written and easy to use. Once it works, you can think about replacing it with a faster solution.

Bye,
Gábor

------------------------------------------------------------------- Gabor DEAK JAHN -- Budapest, Hungary.
E-mail:

Reply by email

Marco van de Voort
to Gabor DEAK JAHN
Show quoted text 13 lines
> their. So, my question is this; does anyone know of another method for > syntax highlighting other then using pos() on each line? Is there a way to > make pos() faster? Is there something better then pos() to check if a string > resides within another string? You should stick to simple keyword highlighting in the beginning, real syntax highlighting is a very complicated business, requiring theoretical and practical familiarity with many concepts used in compiler design. But even only to find keywords, pos() or anything similar is a completely impractical approach. What you have to do is to write a lexical scanner to scan your input and to turn it into tokens.

[snip]

For an example of a lexical scanner, see the source of the ptop source beautifier program, included with FPC.

Reply by email

Gabor DEAK JAHN
to Marco van de Voort

At 11/9/01 12:28 AM, you wrote:

Marco,

For an example of a lexical scanner, see the source of the ptop source beautifier program, included with FPC.

Good point, now that you mention it, I dimly recall that it did not ran for me when I first downloaded FPC, then I completely forgot about it (actually, now I tried, it runs but stops with an error message at the end; although it finishes the ouput file all right).

Bye,
Gábor

------------------------------------------------------------------- Gabor DEAK JAHN -- Budapest, Hungary.
E-mail:

Reply by email

Marco van de Voort
to Gabor DEAK JAHN
Show quoted text 7 lines
> For an example of a lexical scanner, see the source of the ptop source > beautifier program, included with FPC. Good point, now that you mention it, I dimly recall that it did not ran for me when I first downloaded FPC, then I completely forgot about it (actually, now I tried, it runs but stops with an error message at the end; although it finishes the ouput file all right).

That seems to be the way end-of-file is detected. It doesn't interfere with the working of the program though.

Reply by email

Unknown author
Gabor;
Show quoted text 14 lines
You should stick to simple keyword highlighting in the beginning, real syntax highlighting is a very complicated business, requiring theoretical and practical familiarity with many concepts used in compiler design. But even only to find keywords, pos() or anything similar is a completely impractical approach. What you have to do is to write a lexical scanner to scan your input and to turn it into tokens. The details also depend on the language you want to display but, basically, it is as follows: you have a character variable to keep the next character not yet consumed. You write a function to return the next token in the input (an abstract value like identifier, number, punctuation, etc). This NextToken function examines the current-character and decides what to do:

[snip]

As usual, your assistance was not only very explicit but also extremely helpful. But this seems to be how you handle all your replies; you dispense a lot of good advice. I find myself in your debt. Again.

And maybe, if I can get this whole thing working properly (the more I delve into it the more I realize needs to be done) I'll post it on the FPC site! :-)

Thanks again.

Jim

Reply by email