Hi all,

this is more of a generic lex question, but since plex doesn't support the flex extension that could easily fix it, I'm asking here.

Take these lex rules:
---------------------------------------------------------------------- %start INCOMLINE
%%
"//" start(INCOMLINE);
<INCOMLINE>\n start(0);
<INCOMLINE>. yymore;
[a-zA-Z_]([a-zA-Z0-9_])* return(ID);
----------------------------------------------------------------------

The idea is that the comment start sequence "//" puts the lexer in the INCOMLINE state, which eats everything until the next newline, then goes back into the start state. This works for lines like that: Foo//####
But because lex picks the longest matching rule first, it won't work here: Foo//Bar
"Bar" still gets parsed as an ID token, because the identifier rule is valid for any state and considered longer.

Flex has Condition Scopes (see http://dinosaur.compilertools.net/flex/flex_11.html near the end), where one could just put all generic rules in a "INITIAL only" scope. This is a bit awkward if one has to write the prefix every time...

I have solved this in the past by including a mini scanner in the action because comments are easy, but still... does anyone have a better idea to do this right in the lexer?

This is a bit awkward if one has to write the prefix every time...PS - plex doesn't define a name for the builtin default state, so even that would not actually be possible.

What is really missing then is %x for exclusive start conditions, which would solve all of that.

Martok wrote

What is really missing then is %x for exclusive start conditions, which would solve all of that.

Things I never wanted to be able to do: run scanner state machines in my head. But: turns out adding %x was less work than finding an alternative solution, once it was clear how the DFA is actually built.

It currently lives on my work-in-progress-branch for fpcres: https://github.com/martok/freepascal/commit/79873cd1c8 Proper patches will follow once things settle.