I'm currently looking into the FPC parser, in order to make it usable as a general parser, apart from the compiler.

One required change is the handling of used units and identifiers. While a compiler needs all used files, must inline all included files, and must know about the declaration of every identifier, a simple parser (e.g. for syntax highlighting) should do nothing special with such tokens. For a separation of compiler specific actions from other usages of the parser I've introduced a Semantics class, that has virtual methods for all parser events. The base class does almost nothing, another (derived) CompilerSemantics class will be the new home of the compiler specific code in the parser. An application can provide its own class, as required. This separation is the really hairy part, when we ask the FPC team to accept these modifications.

Another change to the scanner is required, so that it does not only skip over comments. This modification is not critical, since the scanner will not normally deserve updates when the language is extended. So we can provide another scanner, that fits our needs.

But moving the compiler specific code out-of-sight is not enough. A parser for syntax-highlighting must handle all tokens, even those in conditional branches. A simple solution could be another Semantics method, that is called when excluded tokens occur, so that the application knows about the type of every token.

Another issue is a parse tree or AST, that already is created by the FPC parser. Unfortunately the nodes in that tree are very compiler specific, so that it may be required to remove that automatic tree construction, and let it happen in Semantics actions.

And finally we'll have to talk about parsing invalid code, during editing. This can be handled by the same hack as for conditionally excluded code, so that after an syntax error the parser stops parsing, and simply returns the tokens provided by the scanner. The only question: how to make the parser resume parsing, once enough tokens have been skipped?

Now the question is: how to proceed?

I can make available an patch for the parser units of the last fpc release, implementing the Semantics separation. It may be more comfortable for the users when a SVN subproject would hold the units to modify, so that these units can be kept in sync with the FPC trunk. Can somebody do that?

Then I can provide a package containing stubs, for all units used by the parser units, so that the parser can be compiled independently from FPC. And we'll need a test application...

DoDi

OTOH, two approaches come to mind:

First one is, introduce another token kind (or several new token kinds, if we need to take care of degrees of severity) for error, and add an 'error node' every time an error occurs and contniue regardless. Then, let the calling application decide what to do.

Second one could be something like this:

Call an event wherever halting occurs:

<pseudo_code>
If SomethingIllegalHappened then FOnReportErrorAndGetFeedback(Self, ErrorKind, PositionInText, MustStop);

If MustStop then DoEmitFatalErrorAndStop;
</pseudo_code>

Hans-Peter Diettrich wrote

I'm currently looking into the FPC parser, in order to make it usable as a general parser, apart from the compiler.

<snip>

<snip>

Now the question is: how to proceed?

I think it is better to discuss this on the fpc-devel mailing list. http://lists.freepascal.org/mailman/listinfo/fpc-devel/

Vincent

The Semantics class will have according method(s).

But when the parser is in an error state, inside some procedure, how can the parser be instructed to exit this and further procedures in the call stack? Exception handling could implement such detailed recovery, but the compiler code does not use SEH for speed reasons.

DoDi

Okay, subscribed.

DoDi

I haven't studied the actual code, so it is entirely possible that I am talking through my (or even someone else's) hat.

I expect, in the current code, there are exit points that emit a reasonably suitable message to the compiler. Could we not replace these exit points with procedure/function calls to that decides whether (in the case of compiler use) it should exit immediately or (in the case of non-compiler use) it should consult with the calling class?

But, then, it's probably not as easy as that.