In the meantime I could make the OO compiler compile, but it does not already work (project ppoo.lpi). Can somebody assist in the separation of the many global variables, which have to become fields in distinct classes (tmodule, tparser...)?

The intended data tree looks like this:

MainModule: T[Main]Module; //can hold the list of all modules
   Module: TModule; //current_module, in module list
     Parser: TParser; //current_scanner (includes scanner)
       ProcInfo: TProcInfo; //current_procinfo
       ...

MainModule
*can* hold application-wide settings, but these can remain in global variables (and different units) as well.

Module
holds all module specific settings and objects. The Parser (and other objects) only exist during parsing and code generation, can be destroyed (or re-used?) when done.
It looks to me as if a ppu reader can provide the information for previously compiled modules, so that a scanner/parser is not always required to load a used module?

Parser
now is derived from the tscannerfile class. This allows to use different parsers (front-ends). For that purpose Compile() has become a class method, that can check the file extension or content, and can create an specific parser accordingly. The same could be done for the selection of the target (machine/system) specific back-end.

Most parser units have been split into .pas and .inc files. The .pas files mostly contain declarations, that can not be moved into the parser unit without causing circular unit references.

All the current_xyz variables must become class elements. Who knows about the scope and lifetime of these "current" objects, and can give hints where to move these references?

In InitCompiler, InitParser and InitScanner a bunch of objects are created, that should be assigned to the appropriate level (object) in the data tree.

See the $IFDEF NoGlobals sections, which enclose the already identified declarations and initializations, that have to be refactored. Unfortunately I already changed parts of that code, so that the original code will have to be reconstructed, in order to make the compiler work again (using global variables).

DoDi

Hans-Peter Diettrich wrote

It looks to me as if a ppu reader can provide the information for previously compiled modules, so that a scanner/parser is not always required to load a used module?

Probably no.

Parser now is derived from the tscannerfile class.

This is a really ugly design OOP wise ...

Most parser units have been split into .pas and .inc files. The .pas files mostly contain declarations, that can not be moved into the parser unit without causing circular unit references.

Why don't you use separate parser classes for different purposes? Declarations, Function bodies, expressions etc.?

Probably nobody, it is a trial and error process.

> Parser > now is derived from the tscannerfile class. This is a really ugly design OOP wise ...

When a parser is designed to work with exactly one scanner, why should it not inherit from it? In the first approach I added delegates for scanner methods to the parser, but since this only adds another level of indirection, it has been almost dropped from later versions.

For performance and management reasons, and to prevent cyclic unit references. I see no real need for subdividing a grammar into multiple (arbitrary) parts. In a future version many syntactical methods can become local subroutines, so that they disappear from the methods. The suggested separation is much more meaningful for the semantical code, i.e. translation into internal data structures etc., up to the final code generation.

IOW I see no need to replace CONSUME(whatever) by e.g. {self.}base_parser.current_scanner.CONSUME(whatever).

DoDi

Florian Klämpfl wrote

Why don't you use separate parser classes for different purposes? Declarations, Function bodies, expressions etc.?

First hurdle is to find a comprehensive enough descriptions of each and every construct there is --including interfaces and generics. This would cut down on guess work a lot. Is there a such a document anywhere --Google didn't help.

Second one is a tricky one: How do you handle code with multipaths? I.e. code that includes in {$IFDEF}{$ELSE}{$ENDIF} blocks. Do you ignore the bits that don't apply, or do you parse all alternative paths too in the name of perfection?

Finally, other than having the whole unit turned into a class structure, would the effort be worth it?

Wouldn't it be memory hog as well as slow as hell?

Well, with the same argument one could stuff all code in one source file and don't separate anything in units.

Having parser and scanner separated might also enable more fine grained multi threading later one: the scanner creates fills a token queue which is read by the parser later on, I know this won't be easy because FPC's scanner is parser context sensitive.

I simply don't like the idea of having one big parser class blob spread over multiple include files.

Florian Klaempfl wrote

I simply don't like the idea of having one big parser class blob spread over multiple include files.

Especially since in the past several people spent a lot of time on breaking cyclic dependencies and organising everything nicely into separate units (both for the parser and for the symbol table).

Jonas

Encapsulation requires some separation.

Having parser and scanner separated might also enable more fine grained multi threading later one: the scanner creates fills a token queue which is read by the parser later on, I know this won't be easy because FPC's scanner is parser context sensitive.

At least the scanner "context" is used in many places in other code. A hypothetical token stream had to include the full scanner context, i.e. file/offset/line/column information, as well as current_settings, for *every single* token. Not very practical :-(

I simply don't like the idea of having one big parser class blob spread over multiple include files.

I don't like it as well :-(

That's why I want to remove the semantics code from the parser, later, and merge the remaining contents of the include files into an <language>.pas parser file.

DoDi

> I simply don't like the idea of having one big parser class blob spread > over multiple include files. Especially since in the past several people spent a lot of time on breaking cyclic dependencies and organising everything nicely into separate units (both for the parser and for the symbol table).

While that separation removed cyclic dependencies, it spread very different functionality across units. In at least one case a procedure exists in two units, so that more work is required to encapsulate functionality in more classes.

DoDi

This might be happen when copy/pasting. Usually such procedures can be easily moved to a common base unit.

Yes, see below ;)

The current_settings can be reused using ref. counting.

But what is the reason against a
TModuleParser
TDeclarationParser
TSubroutineParser
TStatementParser
TExprParser
?

That's not so easy, when global variables shall no more be used. In an OO approach all the replacements of the globals have to become class members, or they must be passed as arguments to all subroutines. Both models require all related types in the interface sections of all units, leading to cyclic references in many cases.

A quite simple solution were threadvars, that can have different values in every thread (e.g. unit to compile). This requires some care to prevent concurrent updates of all related data elements, by arbitrarily called procedures. This can be controlled best when all volatile variables are class members, and every mutable object is assigned to only one worker thread. The current implementation does something similar, but with almost no plan and protection against access to wrong instances - see the many checks for assigned and equal-to-self references. At least we should have dedicated push/pop procedures, that implement safe context switches - mainly in parsing units and procedures.

Code generation is one of the parts that can (and should) be separated from parsing, so that local procedures can be parsed ahead (not blocking the parser), and the code generation can be done in parallel threads.

DoDi

How are these objects created, and how shall they communicate? (where should their references be stored? Even more global variables, for access to each current_xyzparser and his (local) objects?

Every sub-parser will have to reference (create and use) all the other parser types, so that a bunch of temporary parser objects will have to be created and destroyed frequently. Or according references to reusable objects must either go into the parser class (preventing unit references back to the parser unit), or they have to be passed along as subroutine arguments - one argument for each sub-parser.

It would be acceptable to have data container objects for the parsed declarations, subroutines etc., when these objects are put into their own hierarchy (like already done for the nodes), so that the code generation dependencies can be removed entirely from the parser.

DoDi