This sounds good.

I don't think that threadvars really hurt if it resolves the cyclic dependencies.

Why not?

Which commit is it?

After the problems with the global variables, I started another branch for removing global variables and fixing compiler "states".

In the NoGlobals branch I already could make the scanner/parser a child object, owned by the module. It is not yet clear, however, whether the parser or entire modules can be destroyed before the end of compilation
- if memory requirements are of I couldn't merge this working solution for the compiler/parser startup and shutdown into the OO_rewrite branch, so that I'll have to continue with this new branch.

Now it becomes easier to assign further globals and procedures to specific classes, so that finally every parser can work in its own thread. The current_module becomes the first, and hopefully only, candidate for a threadvar. For use in other applications a compiler class/object could be constructed as well, and back-end classes, of course.

Since not all currently global variables can be moved into their "natural" (owning...) classes, without causing circular references, I started to replace e.g. current_scanner by a property.

Nonetheless many assumptions have to be corrected or specified, like a compile_level doesn't make sense in a threaded compilation. I already tried to replace that counter by a comparison
current_module=main_module, but this seems not to work properly. I'll try module.loaded_from=nil in the next round.

I also found an simple workaround for the TImplementedInterfaces problems (getcopy), that can be moved into the trunk immediately.

DoDi

Florian Klaempfl wrote

I don't think that threadvars really hurt if it resolves the cyclic dependencies.

(Many) Threadvars are no performance problem at all unless they are _evaluated_ very often. If (e.g.) the "execute" of the thread creates a class, saves the instance pointer in a threadvar and then calls a function with the class as a parameter, and finally does a free, the threadvar is only evaluated twice, so no

In our previous episode, Hans-Peter Diettrich said:

How would you count an level, when multiple concurrent parser threads create further threads concurrently?

Hmm, I think a producer/consumer system with a queue for modules that can be compiled and N worker threads (specifiable by user)

So on start threads are created, and no new threads are created after?

That depends mostly on the related objects (symbol tables...). It's not yet clear to me, what the many tables belong to.

> Nonetheless many assumptions have to be corrected or specified, like a > compile_level doesn't make sense in a threaded compilation. Why not?

How would you count an level, when multiple concurrent parser threads create further threads concurrently?

> I also found an simple workaround for the TImplementedInterfaces > problems (getcopy), that can be moved into the trunk immediately. Which commit is it?

Please see the log, it might be my second last commit.

DoDi

Your suggestion doesn't solve the problem of level counting, where in fact only the first level (main module, started from the commandline) is of interest, the exact "depth" of other module processing is of no interest.

The observed problem is not the counting itself, but the exact point in the workflow, when the compiler should go back into level one (or zero) state, indicating "generate final binary for the main module". This is not clear from the given code, and led to strange errors in my first test.

Furthermore we should consider to handle multiple main modules (projects), that can be built in one go - since they are all based on the same RTL and other standard units, that deserve loading/compiling only once. That's why we IMO have to figure out, what special "final" mode should be entered, for every main module, at which point in the workflow, and what should happen when that point is reached.

DoDi

That's not a solution :-(

*When* a parser becomes a thread, it can proceed only to the first "uses" clause - then it has to create further threads, one for each used unit, and has to wait for their completion. I.e. most threads are waiting for other threads to complete[1].

That behaviour makes it questionable, whether parsers should become it suggests (to me) a dynamic thread
priorization, based on the number of other threads waiting for their completion[2]. At least we have to distinguish between processing the interface part (or loading the declarations from the ppu file), until which point all dependent threads (using that unit) are blocked. Once the interface information has become available, the dependent threads can continue parsing and generating object code (.o and .ppu files).

Brainstorming: What other models come into mind, for using threads in the compiler?

Your suggestion with a limited number of threads becomes interesting for the code generation, but this IMO deserves another modification of the workflow - currently all procedure bodies are processed "inline", during parsing, so that the procinfo can be destroyed on exit from parsing the procedure block. So we should have threads for parsing a procedure, with the strange result that parsing can happen in the mainstream, as well as in a thread. I just don't know what consequences will arise for the actual code, in such a splitted environment. Parsing inside a thread may require to synchronize several actions, what's not necessary during parsing in the mainstream (main thread).

[1] What synchronization methods exist for threads, suitable to block and resume threads before they have finished? With a fixed number of threads we IMO risk a stall of the entire compiler, as soon as each of these threads is blocked for *other* reasons (memory management, disk I/O...).

[2] How portable is a dynamic thread priorization?

DoDi

It's the only change in symdef.pas, so far.

DoDi

Symtables belong either to modules or procdefs.

Putting compile level in current_module didn't work? compile_level is probably an artefact from times when we even didn't have current_module.

When it leaves the procedure compile?

That's why we've already build units on some OSes where make -j doesn't work well.

BTW: Could you have a look at the clock/timezone of your PC? Since yesterday your mails come with wrong times.

The "final" plan is to use a task queue with dependencies and all needed information. Such tasks may be: parse module X, generated code for module X, load module Y, search for module Y, assemble module X etc. Each task has a list of dependencies which much be done first before the task can be executed.

So not a task per unit or whatever is started but a couple of worker threads depending on the number of cores available, given command line switches etc. These worker threads look for task in the queue with all dependencies solved and when such a task is found, it is executed. The more cores we get during the next years, the finer we could split the tasks.

Hans-Peter Diettrich wrote

Brainstorming: What other models come into mind, for using threads in the compiler?

Theoretically for a fine grained variant, it should be possible to start a thread for each procedure body...

-Michael

One problem with this model currently is inlining: once the implementation of a unit containing an inline procedure is compiled, its interface crc changes (because the code of the inlined routine is part of the interface), and this can trigger recompilations of other units.

An easy solution is to not let the code of inline routine (and the flag that indicates that it's available) influence the interface crc, but in that case it won't be inlinable in units it depends on in its implementation. Doing that would probably solve a lot of crashes and other errors that currently exist with the unit loading though.

Jonas

In our previous episode, Hans-Peter Diettrich said:

No, the parser classes are. There is no reason to have them 1:1 with the threads (except for that one threadvar) You can push the parser class on some completion stack till the requirements are satisfied. The next time a thread finished, the unit dependancy tree is walked to find the next compilable module. (or a control thread could keep this kind of stuff constantly updated, and e.g. prefetch files)

It would save you the overhead of constantly creating/killing threads _and_ make the control over how many (possibly full CPU using() threads run easier.

In a multiuser system (nightly builds, the webservers and php interpreter count as users!) you might not want to stress the system to the knees.

There is a flaw here of course, the control thread can't really see if the workerthreads are mostly blocked on I/O or really working. But users can up the number of cores in typical make -j way if they want full utilization. (though I think less upping is needed since hopefully the I/O cache hit ratio will be higher since the compiler doesn't constantly restart.)

Well, keeping the monster fed will be the major issue. Compiling multiple main modules with their own settings (to do packages/ paralel without make) would be good.

So a mainmodule (for now assumed to be a packages/ buildunit) has its own settings (-Fu dirs etc) associated to it, as well as a few global dirs.

Unfortunately, we can't just throw any unit from such builds into the general unit cache (since e.g. httpd-x have duplicate unit names)

The easiest would be to flag a mainmodule to be added to the global (ppu) cache (and thus be persistent for the rest of the run, hopefully speeding up the many dependancies on fcl-base,fcl-xml), or to keep a local cache for leaf packages and/or packages with duplicate names, to be discarded after completion of the module.

But now I'm rewriting the architecture on a blank sheet of course, something that is always dangerous.

Brainstorming: What other models come into mind, for using threads in the compiler?

I wouldn't go overboard. Having a few threads working is enough, the rest of the work better go into the structures and mechanisms that allow them to be fed, and to keep the compiler running long.

and resume threads before they have finished? With a fixed number of threads we IMO risk a stall of the entire compiler, as soon as each of these threads is blocked for *other* reasons (memory management, disk I/O...).

That can be mitigated by taking twice (or one and a half times) the number of physical cores, as typically done by make -j in such cases. If it is a possibility that really adds an overall noticable delay.

[2] How portable is a dynamic thread priorization?

You don't. Your control thread determines what module is compiled/resumed next when a worker thread comes available. There is no reason to do this via funky thread systems.

Marco van de Voort wrote

> [2] How portable is a dynamic thread priorization? You don't. Your control thread determines what module is compiled/resumed next when a worker thread comes available. There is no reason to do this via funky thread systems.

A single controlling thread does not scale very well. A better solution is that the
workers choose their work by themself. Or to use some sort of distribution queues.

After doing years of multithreaded development in our communication appliance
we found that apple came to a similar solution like our own, and took the time to write it down :-)

http://developer.apple.com/mac/library/documentation/General/Conceptual/ConcurrencyProgrammingGuide/ThreadMigration/ThreadMigration.html#//apple_ref/doc/uid/TP40008091-CH105-SW1

We have an custom written scheduler that operates on lock free distribution queues that feed exactly
or less that amount of threads that the machine is able to do really parallel.
We strive to use no locks at all by distributing the work(chunks) directly to a per worker thread queue,
and where not possible to use lock free algorithms. Ideally is to partition the data if it's known beforehand
or at an early stage. Also it is good to keep the working set data per thread and per core locally and don't move it from core to core.

More infos:

About the speed of locks :
http://transact09.cs.washington.edu/19_paper.pdf

About memory/cache/threading in general:
Google: Ulrich Drepper, What every programmer should know about memory

helmut

In our previous episode, Helmut Hartl said:

> funky thread systems. A single controlling thread does not scale very well. A better solution is that the workers choose their work by themself. Or to use some sort of distribution queues.

So, I'm no thread expert, but does this all really matter that much if the units (literally) to schedule are in the tens of ms magnitude and HDD access is required for each of them ?

The controlling thread regularly updates the queue of stuff to compile. If it finds 10 units to compile, it can sleep( for easily 40-50ms magnitude even), till a worker thread signals that the number of units are under the low watermark (I'm making this up as I go, but I really don't see a bottleneck here, with the jobs being fairly expensive)

[2] How portable is a dynamic thread priorization?

I suppose you will use threads via some thread-pool component. Same might need to be enhanced by priority management, but this should not have any influence on the code of the compiler project itself.

-Michael

Not so easy, unfortunately. Or I did something wrong in my refactoring...

BTW: Could you have a look at the clock/timezone of your PC? Since yesterday your mails come with wrong times.

Thanks. I found wrong times on many machines, physical and virtual. Are we already in [MS] winter time?

DoDi

The unit dependencies are known only after their "uses" clauses have been parsed. After the used units in the interface part have been parsed, the parser can continue until he must wait again for used units in the implementation part.

The current model does a "depth first" parse, switching to every used unit. The unit with all dependencies resolved can run until completion, and a switch back to the previously active parser is made. I.e. parallel processing can only start after a unit with no further dependencies has been found (parsed past the implementation uses). Then the remainder of the unit can be processed (parsed and compiled) in parallel with other tasks.

DoDi

Nice :-)

But another idea came just into mind, something like PERT/GANTT diagrams, used to optimize production plants. What if we instrument the current code with time measurement for all tasks, that can be parallelized in future? Then somebody could write an analytic program for the tasks and their measured times, so that we can learn more about the real chances and possible wins...

From

   A uses B,C;
   B uses D,E;
   C uses D,F;

we get the sequential flow, using e.g. A1/2 for the interface(1)/implementation(2) processing:
A1 B, C; A2 B1 D, E; B2 C1 D, F; C2
D1 D2 E1 E2 <- F1 F2
or flat
A1 B, B1 D, D1 D2 E; E1 E2 C; C1 D, F; F1 F2 C2 A2 that could be parallized into
A1 B, B1 D, D1 E; E1 C; C1 D, F; F1
+D2 +E2+B2 +F2+C2+A2

Here we can see that massive parallelism occurs at the end of the compilation, so that the determination of the shortest pathes (to the first compilable unit) may be the key to best performance, whereas a restriction of the number of parallel threads may result in performance degradation. But we will know more only after an analysis of real-life projects, and with real times for the parallel parts.

DoDi

Marco van de Voort wrote

So, I'm no thread expert, but does this all really matter that much if the units (literally) to schedule are in the tens of ms magnitude and HDD access is required for each of them ?

Of course you are fully right and it depends on the particular problem - we for example do inplace analysis of tcp / udp packets in memory over gb links with no spinning disks.

What I wanted to say is that the work and concepts already done by others (apple and various others)
can be reused as it's unlikely to have a quick win with an own first solution and to provide some
(useful) links.

helmut

The removal of global variables is almost finished now. I know that the code currently doesn't look nice, and that performance can be improved, but these are minor details compared to the attempts to not break the compiler itself.

In the last round a GlobalModule has been added, that plays the role of a not yet existing current_module at compiler initialization. Since the remaining global variables currently are scattered across many units, they could be moved all into the GlobalModule - any opinions?

For performance reasons it were nice to have unit-level properties with variable refences for the getters, instead of subroutines - any reasons why this is not allowed right now?

What should be done in the next step?

In contrast to the preceding OO approach, a bunch of ordinary procedures still waits for becoming methods, with faster access to frequently used class members. Should I try to convert the procedures to methods now, or should I wait until everything else has settled down?

Another task were the separation of the implementation parse and code generation, from interface parsing. This separation would allow for multiple worker threads for e.g. code generation.

Any suggestions?

DoDi

Please polish and finish one self-contained part, get that reviewed and merged into trunk, and then continue (or use different branches, although then you will have to merge things and resolve conflicts). In particular, do not concurrently change different things in one branch (mixing fixes and/or new features in your refactoring, or mixing different kinds of refactoring that can also be done independently), because that makes it harder to review the code and less likely to get anything merged.

Jonas

Then I'd say it's the right time for review just now :-)

DoDi

Hans-Peter Diettrich wrote

Even if the code generation has not been touched, other targets still may deserve minor updates, related to the changed types of some previously global variables (pointers instead of direct references). I don't know how to test that myself

Run the following command in the compiler directory:

make fullcycle

Jonas

Before fixing the other targets, we should first discuss the changes. I'am not sure however, when I can look into it: I'am rather busy this and the next week because the week after I'll be on holiday.

Hans-Peter Diettrich wrote

Such complications can be eliminated by a refactoring of the targets, into dedicated classes. This can be done in the same branch, because the existing units (in the target directories) have conflicting names, and consequently must be replaced by new unit trees - something similar to the systems implementation.

I don't understand why this "consequently must be" done. The compiler only supports generating code for a single architecture, and there are no plans to change this. It would needlessly complicate the compiler source.

Jonas

Revision 15852 can be considered the release candidate of the NoGlobals branch, tested on/for Windows (x86 32/64 bit). A readme file was added with rev. 15858. A tag could be created for this revision...

Even if the code generation has not been touched, other targets still may deserve minor updates, related to the changed types of some previously global variables (pointers instead of direct references). I don't know how to test that myself, please inform me of any such problems.

Such complications can be eliminated by a refactoring of the targets, into dedicated classes. This can be done in the same branch, because the existing units (in the target directories) have conflicting names, and consequently must be replaced by new unit trees - something similar to the systems implementation.

DoDi

Run the following command in the compiler directory: make fullcycle

There seems to exist a problem with the compiler configuration. This call looks good and works as expected:

D:/repos/NoGlobals/compiler/ppc2.exe -Fux86_64 -Fusystems -Fu../rtl/units/x86_64-win64 -Fix86_64 -FE. -FUx86_64/units/x86_64-win64 -dx86_64 -dGDB -dBROWSERLOG -Fux86 pp.pas

in contrast too a later

D:/repos/NoGlobals/compiler/ppcx64.exe -Fu..
-Fu../../rtl/units/x86_64-win64 -FE. -FUunits/x86_64-win64 -dx86_64 ppudump.pp -Fu../x86_64 -Fi..

which results in

Fatal: Can't find unit cpubase used by node

IMO there is one "../" too much in the -Fu paths.

What's wrong?

DoDi

My experience with the compiler sources is different. The many conditional parts, which are not even properly chained by {$ELSEIF ...}, make the maintance and refactoring a mess :-(

Currently the introduction of a new target CPU requires to update many parts of the existing general compiler units, with no reliable search key for these parts. No problem when the critical pieces become (part of) virtual methods, where it's immediately clear that virtual methods may deserve different target-specific implementations.

A proper OO approach will simplify the compiler maintenance, and it does not require that a final compiler really should support multiple targets.

DoDi

E.g.?

I just greped for ifdef sparc in the general compiler directory: 11 hits:
- 5 in ogelf.pas (elf output generator, so quite natural)
- 1 in globals.pas to define the default cpu switches in init_settings
- 2 in options to display sparc only help pages and create sparc specific defines
- 1 in systems.pas to set the sparc fallback target
- 1 in psystems.pas to set the sparc default floating point type
- 1 in fpcdefs.inc to set sparc specific conditionals

with no reliable search key for these parts.

This is usually not needed. Doing a test driven approach, you'll identify the parts to change very quickly.

The problem is: not everything can be solved easily using OOP. And I'am also against trying to make everything fit into OOP.

Yes, this is because i386 is a very special cpu compared with the others: cisc, high register pressure and a fpu stack with strange behaviour compared to other fpus. Nevertheless, it's far from being so bad that it requires to dig upsite down in the compiler. But I'am still waiting for examples!

Try the same with i386, and get hits somewhere else :-(

The problem is: not everything can be solved easily using OOP. And I'am also against trying to make everything fit into OOP.

Again my current experience (with the compiler) is not so bad. I've just started an private branch for OO back-ends, just to find out more...

DoDi

Hans-Peter Diettrich wrote

The many conditional parts, which are not even properly chained by {$ELSEIF ...}, make the maintance and refactoring a mess :-(

While moving to OO paradigms could make the code slightly slower, I suppose that supporting multiple CPUs with a single compiler might be possible, doing away with the "old-style" $ifdefs, by using derived classes for the different CPUs ans OSes.

-Michael

In the meantime I found it doable, to remove the conditional compilation, but at a very high price :-(

When a data structure (e.g. COFF header) differs amongst 32 and 64 bit versions, so that fields have different offsets, all references to such fields have either to become virtual, or the code has to be duplicated for both models. In this case I find it more efficient, with regards to both speed and coding efforts, to leave the conditional parts in place.

Even in a mixed-mode coding model, with such polymorphic parts moved into re-usable include files, the number of such files would become very high.

Now I'm somewhat blocked, as long as nobody reviews the current state of the NoGlobals branch. I understand that it's summertime (in most countries ;-), and many core developers are in vacancies. But if there is a real desire for parallel execution in the compiler, a quick review would allow me to proceed in that direction.

TIA
DoDi

ppudump (and the Makefile that builds it, and hence the "current working directory" when compiling it) is located in compiler/utils, and hence an extra "../" is required compared to when building the compiler itself.

Jonas

Right, looking in the wrong place :-(

The real problem seems to be the difference between -Fux86 (receives .o and .ppu files)
and
-Fu../x86_64 (no .o and .ppu files)
where building the trunk uses /i386 in both cases.

DoDi

The -FU (note the capital U) parameter specifies where the compiled units are stored (-FUx86_64/units/x86_64-win64). If you have .o and .ppu files in the x86 directory, you put them there yourself somehow (either on the command line or by modifying the default Lazarus project file), because the Makefile and the default Lazarus project file in svn do not put any files there. You will have to manually delete them.

Jonas

Hans-Peter Diettrich wrote

D:/repos/fpc-2.5.trunk/compiler/ppc386.exe -Ur -Xs -O2 -n -Fu.. -FuD:/repos/fpc-2.5.trunk/rtl/units/i386-win32 -FE. -FUunits/i386-win32 -di386 -dRELEASE ppudump.pp -Fu../i386 -Fi.. I cannot see any -Fu../x86 here, so how can x86/cpubase be found here?

ppudump probably does not depend on cpubase in trunk. cpubase is part of the code generator, and therefore the ppu loading should in fact not depend on it (directly nor indirectly).

Jonas

Okay, these files are from my ppnew.lpi.

Now I tried again, for the default target. For some reason compiler units are compiled into multiple directories:

D:/FPC/2.4.0/bin/i386-win32/ppc386.exe -Ur -Xs -O2 -n -Fui386 -Fusystems -FuD:/repos/NoGlobals/rtl/units/i386-win32 -Fii386 -FE. -FUi386/units/i386-win32 -dRELEASE -di386 -dGDB -dBROWSERLOG -Fux86 pp.pas

--> x86/cpubase and i386/cpuinfo in i386/units/i386-win32

Later on this one fails:

D:/repos/NoGlobals/compiler/ppc386.exe -Ur -Xs -O2 -n -Fu.. -FuD:/repos/NoGlobals/rtl/units/i386-win32 -FE. -FUunits/i386-win32 -di386 -dRELEASE ppudump.pp -Fu../i386 -Fi..

Fatal: Can't find unit cpubase used by node

--> cpuinfo *also* in utils/units/i386-win32, but *not* cpubase

Inspecting the trunk directory, after make all, both cpubase and cpuinfo binaries are in i386/units/i386-win32 - how that?

D:/repos/fpc-2.5.trunk/compiler/ppc386.exe -Ur -Xs -O2 -n -Fu.. -FuD:/repos/fpc-2.5.trunk/rtl/units/i386-win32 -FE. -FUunits/i386-win32 -di386 -dRELEASE ppudump.pp -Fu../i386 -Fi..

I cannot see any -Fu../x86 here, so how can x86/cpubase be found here?

DoDi

ppudump probably does not depend on cpubase in trunk. cpubase is part of the code generator, and therefore the ppu loading should in fact not depend on it (directly nor indirectly).

That looks like the right explanation.

Consequently the code generation has to be separated strictly from the remaining code, according (global) references can not reside in other commonly used classes. This (currently) is not the case in a threadable compiler, where FModule.TModule contains *all* references to related objects.

DoDi

ppudump probably does not depend on cpubase in trunk. cpubase is part of the code generator, and therefore the ppu loading should in fact not depend on it (directly nor indirectly).

Apart from that (minor) issue, can you review my work? It would be helpful to learn what's considered okay and what not.

DoDi

Hans-Peter Diettrich wrote

Every parallel processing requires that all related data is private to every thread. Since some time I'm trying to eliminate such (currently) global variables.

What's the problem with using threadvars (I did not look at your code yet, I were on holiday)? As I said in another thread: it is not needed to squeeze anything into OOP, sometimes other solutions are better like threadvars (or "ifdefs" ;)).

Just to make sure: This multi-threading would be user selectable, right?

What would be the default value? Multi-threaded or single-threaded?

Personally I enjoy the fact that FPC only goes up to 50% of the CPU usage (using 1 processor fully out of 2 processors) and then I can keep working normally with OpenOffice and other heavy stuff.

thanks,

Felipe Monteiro de Carvalho wrote

Personally I enjoy the fact that FPC only goes up to 50% of the CPU usage (using 1 processor fully out of 2 processors) and then I can keep working normally with OpenOffice and other heavy stuff.

That's the joy of the thread scheduler under Linux. It works so much better than under Windows (don't know Mac's, so can't comment there). Linux lets thread work slightly longer, so less switching and each thread actually get more done than simply setting up data after each quick switch.

PS: I noticed this (and first thought it was a FPC Linux bug) with a Thread Manager Demo I wrote some years back. If you don't make a thread work hard enough, it delays switching even more and acts more like a single threaded app. This is clearly visible if you port the Delphi Thread Sorting demo to FPC with Linux/Windows. Now that I have a quad core setup, I should actually review these demos again, to see how it behaves now.

If you want to know more (and if you use Linux), I can search my mail archive for the internet links explaining this in a lot more detail.

Every parallel processing requires that all related data is private to every thread. Since some time I'm trying to eliminate such (currently) global variables. Now ppudump turned into an show stopper, due to the unsystematic and obscure unit dependencies in the compiler:

In the meantime I've implemented another version, that does not have the above problem - but further development is impossible because the problem reappears with every of the remaining global variables. Therefore I'm stopping any further development now, until somebody can provide a solution or workaround for this problem.

DoDi

Hans-Peter Diettrich wrote

Every parallel processing requires that all related data is private to every thread. Since some time I'm trying to eliminate such (currently) global variables. Now ppudump turned into an show stopper, due to the unsystematic and obscure unit dependencies in the compiler:

I think it is quite systematic and logical that the ppu loading should not depend on code generator internals.

You have to describe in your svn commit logs what you actually do. "NG: fixed make all" is not a useful commit message. You changed 149 files in that commit, and did not describe what was done nor why:
* what does all that {$ifdef fix} code do? What does it fix?
* you deleted the aasmsym unit and changed code that depended on it without any remarks why you did that. The comment in the header of that unit said: "Contains abstract assembler instructions for all processor types, including routines which depend on the symbol table. These cannot be in aasmtai, because the symbol table units depend on that one."

You nevertheless did move that code to aasmtai and now there are indeed (implementation-level) circular dependencies between aasmtai and symsym. Adding extra circular dependencies to the compiler without any argument as to why this is required is not good.
* that commit also contains changes that have nothing to do with fixing "make all", such as fixing a typo in a comment, removing type redefinitions, commenting the assignment of tcgaddnode to caddnode in the init code of ncgadd.pas (and similar assignments in ncgmat), possibly an unrelated change in ppheap (which is wrongly indented), removing an unused local type definition from ppu.pas...
* I assume that all the code you added in psystem.pas is with the long term view of making the parser independent. However, that is a separate project and not part of removing global variables from the compiler

While reviewing your previous commits, I noticed that you also integrated your patch from http://bugs.freepascal.org/view.php?id=16888 although that one has nothing to do with removing the use of globals.

This way of working is not how we can keep the compiler development manageable for multiple people to work on.

Jonas

Then the problem is to tell which variables can stay *ordinary* variables.

DoDi

Right, that's how it *should* be designed. But try to find out why the code generation is added, when variables like current_filepos or current_tokenpos are moved into TModule (current_module) :-(

You have to describe in your svn commit logs what you actually do. "NG: fixed make all" is not a useful commit message. You changed 149 files in that commit, and did not describe what was done nor why:

I only commited these changes so that they don't get lost. Please review the preceding version, as mentioned in the Readme.

The last change was to remove ppudump from the Makefile, and this proved that so far only ppudump is sensitive to changes in the compiler internals.

* what does all that {$ifdef fix} code do? What does it fix?

It's simply a flag for recent changes, which can be made unconditional after verification. The name was choosen to be enough unique, so that all affected locations can be found be searching for "fix".

I wanted to make code generation more abstract, so that it can be separated better from the remaining code. The GlobVar unit is intended as a place for all (previously) global variables, that can be used without importing further dependencies.

You nevertheless did move that code to aasmtai and now there are indeed (implementation-level) circular dependencies between aasmtai and symsym. Adding extra circular dependencies to the compiler without any argument as to why this is required is not good.

I don't remember details. My experimental approach left these units unmodified, and still resulted in importing code generation (see above).

I tried to follow the coding style, to not comment anything in the code.

Please suggest an better and more systematic way of reducing the number of compiler hints, and supply a justification for redefinitions of system unit types.

* I assume that all the code you added in psystem.pas is with the long term view of making the parser independent. However, that is a separate project and not part of removing global variables from the compiler

Both are related, since the "stateful" variables are related to current_module, and that's where the scanner and parser is involved, somehow.

I added some improvements and corrected typos, while waiting for comments on my work. It would be nice to know in *advance*, what should be avoided. The lack of response to my repeated review requests is not what I consider as "multiple people working on the same project".

In the meantime I started several projects, not commited to the repository, one of which (almost only) moves global variables into TModule, without breaking ppudump and affecting unrelated units. If you think that this is a better base for cooperation, I can try to add it as another branch, with no detailed history.

DoDi

Hans-Peter Diettrich wrote

I only commited these changes so that they don't get lost. Please review the preceding version, as mentioned in the Readme.

[... simply stating my observation. If there is a subversion way of doing the same as I describe with git, it might be worth mentioning, so others can learn how to do similar with SVN. I'm not forcing git on anybody, simply stating how I work, in a similar situation. ]

Reading Jonas's comments and your reply above, it seems a git mirror (or a subversion/git dual setup) would be much better suited for your working style. git allows local commits and local branches. Ideal for cases like "so those changes don't get lost" or braking a large set of changes into smaller commits. The 'git gui' tool also allows you to brake a huge amount of changes into smaller commits. Simply run the 'git gui' tool, add the changed code lines you would want in the next commit (git allows per line adding, not just hunks or whole files), add a commit and click commit. Now repeat, until all your changes are in.

If changes don't relate to the current "feature" branch, simply 'git branch <my_other_feature>' the 'git checkout <my_other_feature>' and start committing in there. Switch back to your origin branch and continue working. Switching branch in git doesn't loose uncommitted changes.

When I mentioned subversion/git dual setup. I mean, use your original svn checkout, but initialize a git repository in that same directory. You can then use git to track local changes and local branches. when you want to commit something, switch to the 'svn' git branch and cherry-pick or squash your local git commits onto the 'svn' git branch
- then commit that to SubVersion as normal (one commit at a time, or in the squashed commits case, a single "feature" change set). Then resync your git 'svn' branch and continue working. This not normal git usage, but it works really well if the project is in SubVersion, and you don't want to use a full-blown 'git-svn' mirror. Simply allow git to track your local changes, until such time as you can commit them to SubVersion.

More details on git mirrors of FPC and links to git help. http://wiki.freepascal.org/git_mirrors

Just my 2c worth. Use it or ignore it.

+1
I can recommend it, too.

In some earlier mail I mentioned it would solve DoDi's problems with merging his branch with trunk after they have deviated much from each other. Having your local git branch active, just run :

 $ git rebase master

(assuming master branch is trunk)

I just recently created a git branch from my old Lazarus Build Profiles patch. It took some effort to solve all conflicts but now I am confident I can keep it in sync with trunk easily by running "git rebase master" once a week or so. It is good in case the feature is accepted for Lazarus in future (during next 5 year plan maybe).

Git seems to be a good tool for such things and it integrates well with SVN. Everybody should be happy. :-)

Juha

That's how it is in trunk.

But try to find out why the code generation is added, when variables like current_filepos or current_tokenpos are moved into TModule (current_module) :-(

Nobody ever said that it would be easy to restructure the compiler. In fact, I will be the first to admit that it is incredibly hard, and since the compiler is 15 years old (although several parts have been rewritten over the years), there are bound to be quite a few less than ideal situations and things that simply grew over time. However, I very much dislike the way you are doing the development and find it very hard to see how we are ever going to be able to integrate any of those changes back into trunk.

Maybe I simply should stop being involved in this, since I'm obviously not able to productively deal with this project.

You should never commit unrelated changes in a *single* commit. Even if you don't want to create a separate branch for other changes, at the very least use separate commits (although unrelated changes will make merging more difficult in case later changes also depend on them).

Please review the preceding version, as mentioned in the Readme.

The main problem is that it cannot be merged since it doesn't compile. As I mentioned earlier: the ideal situation are piecemeal changes that form a tested and self-contained whole that can be easily reviewed and merged. While a single change related to global variables can obviously have repercussions all over the compiler, that does not mean that you have to replace all global variables that at first sight should end up in tscanner with fields in one go, especially if you are unable to completely test those changes (make all, make fullcycle, running the testsuite).

The last change was to remove ppudump from the Makefile, and this proved that so far only ppudump is sensitive to changes in the compiler internals.

It also proves that the internal compiler structure has been broken.

Removing an extra abstraction class and folding it into the base class is the opposite of making something more abstract.

The GlobVar unit is intended as a place for all (previously) global variables, that can be used without importing further dependencies.

The aasmsym unit did not contain any global variables. And putting all global variables in a single unit may not be the right approach since the compiler consists of many components. They are obviously interwoven, but making them more interwoven by basically making all types of all global variables globally visible does not sound a like a good idea.

That's why you should write it in the svn log. Note that you can edit the fpc repository's svn log entries even after you have committed, using "svn pe -r <revnr> --revprop svn:log".

My experimental approach left these units unmodified, and still resulted in importing code generation (see above).

That's because of other changes. Making the problem worse does not contribute to the solution.

Oh, please.

Please suggest an better and more systematic way of reducing the number of compiler hints, and supply a justification for redefinitions of system unit types.

I'm not saying that the local type definitions should stay. I'm saying that removing them must not be committed together with completely unrelated changes.

That does not answer my remark, but I don't know how to make my point clearer.

* don't add cyclic dependencies: http://lists.freepascal.org/lists/fpc-devel/2010-July/021065.html
* commit unrelated changes separately (even to separate branches if possible), polish/finish one self-contained part at a time so it can be merged: http://lists.freepascal.org/lists/fpc-devel/2010-August/021558.html

I may indeed not have mentioned before that svn commit messages should explain the idea behind the changes and why some things are done in a certain way.

The lack of response to my repeated review requests is not what I consider as "multiple people working on the same project".

There are more people that you alone, and many people have spent a lot of time on answering many questions from you. Reviewing a bunch of changes with little or no information about why which changes were done is a) a lot of work
b) not very enticing

In the meantime I started several projects, not commited to the repository, one of which (almost only) moves global variables into TModule, without breaking ppudump and affecting unrelated units. If you think that this is a better base for cooperation, I can try to add it as another branch, with no detailed history.

I don't know.

Jonas

The problem is conflict solving and this is usually the equally hard part with any scm.

I don't see how this is different from merging new trunk changes to a feature branch in svn? See e.g. r15421 where Jonas merged all new changes from trunk into the cpstrnew branch.

Why should current_filepos and current_tokenpos go into TModule? They can be perfectly threadvars. Further, I don't see them fitting into TModule, they describe not the state of a module but are part of the compilation state. Even more, consider several threads compiling different procedure of one module: putting current_filepos and current_tokenpos into TModule won't work in this case.

The last change was to remove ppudump from the Makefile, and this proved that so far only ppudump is sensitive to changes in the compiler

internals.

Guess why I'am sceptical that it's usefull to use the compiler parser for other purposes like code tools or documentation: probably once per week a simple compiler change breaks this external usage (we made this experience ten years ago ;) ).

How we can continue? I'll see if I find within the next week time (I were on holiday for one week) to review the noglobals changes and how we can split them into usable parts.

Juha has a valid point about less conflicts. I have dragged (for a lack of a better description) my local branches for the Lazarus repository to match SVN Trunk for over a year with no (or very minimal) conflicts. These local branches contains many personal customizations to Lazarus, which includes new sections in Environment dialog etc - many areas which see lots activity from Lazarus developers.

I don't see how this is different from merging new trunk changes to a feature branch in svn? See e.g. r15421 where Jonas merged all new changes from trunk into the cpstrnew branch.

Rebase is not the same as merge (at least when you are using git). As I explained before, rebase rewrites history, merge creates a new commit with changes from another branch. Rebase is more often used to keep a clean history (no merge commits in history), and more often used with local branches, because then, rewriting history does not affect somebody else's repository like when they cloned from you.

Anyway, we know your stance with Git, so we will stop here, and not go down that path again. :) If DoDi wanted to know more about Git, he is welcome to mail me (and probably Juha too) in private.

How do you know if you don't use svn? I tried hg and sometimes I use git and I never saw any advantage of them regarding conflict resolving. Usually it's even worse than with svn because compared with TortoiseSVN, git and hg gui support is still poor, TortoiseHg/Git are far from being there. Though git/hg are sometimes usefull to move changes quickly between machines.

Did you realize that I wrote about svn :) ?

To be able to get a review of a patch, he needs to publish it in any form anyways. So local branches/rebases etc. are pretty useless in this case, some svn branch is perfectly usable for this scenario.

After a review of psystem, I found the changes related to the weak setup of the nodetype and other classtype variables, in the general and target specific units. It's an independent part of my work, that can be undone and treated as a separate improvement. The general idea is like this:

Before: In order to avoid references to target specific units, a couple of classtype variables have been established in various units. The original implementation initializes these variables in the general compiler units, together with the related class declarations. Later on some values are overridden by CPU specific classes, in CPU specific units, what also happens in the unit initialization section. This leads to dependencies in the *order* of the used units, as I observed after moving unit references between the interface and implementation uses clauses, resulting in an broken compiler.

After: This positional dependency of the units in uses clauses can be eliminated when the variables are left uninitialized by the general units, and instead missing initialization (by target specific units) is detected and handled on their first use, here in psystem.registernodes.

The handling of uninitialized variables can be implemented by using defaults, as implemented in the NoGlobals branch, or by reporting an fatal InternalError indicating that the currently used set of target specific units missed to initialize such a variable.

This is only a suggestion, which can be implemented in one of the indicated ways, or everything can be left as weak as it is.

DoDi

Florian Klämpfl wrote

> > Juha has a valid point about less conflicts. How do you know if you don't use svn? I tried hg and sometimes I use git

I've only used git for the last two years. So no, I don't use svn any more, but for many years before git, I used svn. So indeed, I can comment on my experiences with both tools.

Usually it's even worse than with svn because compared with TortoiseSVN, git and hg gui support is still poor, TortoiseHg/Git are far from being there. Though git/hg are sometimes usefull to move changes quickly between machines.

I'm not a GUI junkie. SVN and GIT started as console tools, and that is where the power normally lies. As for TortoiseXXX... they are not the sole provider for "gui" font-ends to git (or other SCM's for that matter). So don't judge git by simply look at TortoiseGit - it's quite well known that TortoiseGit is far behind in actual git features. Git includes it's own gui tools that are a lot more powerful (features supported), than any others I have seen (and yes, there are plenty available). Most gui fontends don't even come close to what git can actually do. The default gui tools included with git use the Tk widgetset, so it's not the prettiest tools. But when it comes to getting the job done, I'm not after looks, but rather features and usability. 'git gui' and gitk accept any parameters like 'git' itself does (probably unknown to many). So their behaviour and functionality can change depending on those parameters.

To be able to get a review of a patch, he needs to publish it in any form anyways. So local branches/rebases etc. are pretty useless in this

Rebase is still usable in a published branch (git project itself uses it like that too), as long as you make it clear to developers that the history of the branch will change from time to time, and thus its meant for review/test purposes, and not for developing on. So pushing a local branch to a published "review" branch is a simple task - many projects use it and normally reserve a specific branch just for that task.

Anyway, enough said on the subject.

Graeme Geldenhuys wrote

but rather features and

Indeed. And the features of git gui are just ridiculous. It misses even basic stuff like good explorer integration.

usability.

Yes, unfortunatly, git gui is basically unusable on windows: most of the time it's hanging.

Can you give me some hints, on how to restructure the compiler in the most compatible way, based on your experience?

As you can see from my many attempts (public and private branches), to attack the problem from different sides, I'm open to any strategy :-)

Maybe I simply should stop being involved in this, since I'm obviously not able to productively deal with this project.

Please stay involved! Since I'm very new to the fpc project, I need assistance by more experienced developers.

ACK. That's why I asked for timely reviews of my work, so that the trunk can be updated in sync with my progress, and bad attempts can be detected and avoided in an early stage.

> Please review the preceding version, as mentioned in the Readme. The main problem is that it cannot be merged since it doesn't compile.

I only committed changes that compile on my machine (win32/64). The problemw with ppudump - and only there - was brought to my attention only recently, and its solution is not easy for me.

I actually have no clue on how to avoid the observed problems, based on the general decision that module-specific variables have to become fields or properties of TModule. This is a situation where I badly need assistance by the core developers...

All the suggested tests stopped working many weeks ago, so that I could proceed only with the compiler project only. Now that I found out that dropping ppudump from "make all" will allow to perform the tests, I can at least fix the real syntax errors in the code, occuring only in the generation of compilers for other targets.

WRT to the testsuite, the results are of little help (to me) as long as other users acknowledged that the test results don't match the given reference. Can somebody explain that known problem?

A really broken parser should manifest during make cycle, what can be excluded. So I currently only can compare the test suite results against those of the trunk version, not knowing what will happen on other host systems :-(

Can you enlighten me, what exactly broke the structure?

Apart from that I dare to decline, from the existence of a strong "structure" inside the compiler. IMO the structure is so fragile, that every reasonable refactoring attempt will hit a dead end very soon. But I'd be happy when somebody could proof me wrong :-)

Perhaps the description of aasmsym was misleading me. I'll restore aasmsym and test again...

My idea was (and still is) to remove direct references to many units as far as possible, so that the interface parts of all units can be stripped accordingly, reducing the chance for cyclic unit references and increasing the chance for refactoring in other places. The only negative effect may be the inclusion of more units in related tools, e.g. ppudump. This problem may be solved in a later stage, when the new dependencies have settled down.

Thanks for this hint, but that doesn't help when I don't remember details any more :-(

Right, those "other" changes turned out to break something in the existing structure. When we agree to make the mentioned approach the first candidate for inclusion into trunk, and figure out how the remaining required (currently breaking) changes can be made, it would make life easier for all of us. When no solutions for such early-stage refactoring can be found, all attempts towards parallel processing can be stopped immediately :-(

;-)

Actually I should start just another (private) branch, where I can update and reformat the trunk version according to my needs. The coding style(s), found in the trunk, makes it hard to figure out what's going on. Obstacles are deep indentation levels, and enlarged code sections, caused by arbitrary excessive indentation and the requirement to place every "begin" into a new line. My personal style results in more compact code, that can be analyzed easier. No offense intended, coding style in fact is a very personal thing, and a public project deserves *commonly* accepted rules. I only wanted to explain why I updated e.g. comments while working on the code, so that it's near impossible to commit such changes in distinct steps.

Okay, I'll try to commit more often in the future. My intent was to keep the repository small, by not committing every single change. When this is contraproductive, WRT to later merges, I'll have to change my habits. For the same reason I didn't create new experimental branches, after it turned out that most of them end up in dead ends, and nobody ever looked at them.

Maybe git will help to reduce such problems, but the learning curve... :-(

Thanks for spending your time on me :-)

DoDi

Good point. So would it be sufficient to retype all such variables as threadvar, in order to make parallel processing (threads) possible?

I have no real idea, how in such a model the initialization of the threadvars has to be implemented. That's why I try to assign all state-related variables to a definite object, whose reference can be easily copied (or moved?) into any created thread.

Then we also have to draw a border, between possible parallel and still required sequential processing. Sequential processing must be used for all output, be binary or log files. Also encountered errors must be reported from the originating thread to all related (main and other) threads, when they require to stop and shut down compilation in an orderly manner.

Right, but I see no chance for such parallelism, before all related variables have been found. See my questions about just these variables, and the according tfileposinfo values in several objects.

Parallel code generation requires that the cg is separated from parsing, so that the next procedure can be parsed while the previously parsed procedures are compiled.

I've postponed that initial motivation, to the end of all other refactoring. Apart from parallelism I see more chances for the introduction of really new features in other places, like multiple front-ends. Such projects require a separation of the mere parser from the rest of the infrastructure, i.e. the handling of all symbols, creation of nodes, etc. have to be moved into new and commonly usable interfaces. After that step it also would be easy, and could break nothing in the compiler, when a no-cpu target is added to the target specific back-ends.

How we can continue? I'll see if I find within the next week time (I were on holiday for one week) to review the noglobals changes and how we can split them into usable parts.

IMO the most important decision is about the general direction of the refactoring. Do we want more OO (encapsulation), more codegen separation, or what else. IMO encapsulation is the most useful first step, towards any other goal. The current compiler "structure" is dictated by purely *formal* aspects (unit dependencies), and does not reflect the *logical* dependencies between objects, variables, procedures etc. This lack of logical structure, next lack of up-to-date documentation, is the most annoying problem with *every* compiler enhancement attempt.

DoDi

In case of current_filepos, the initialization could be done by the specific worker thread class doing e.g. code generation.

Yes, this won't get easy, but we are far away from something like this.

I think we agreed on this already. And I also agree with you, that the front end needs some redesign and it won't be easy (it took me almost 2 years to find the current back end design btw. ...)

The current compiler "structure" is dictated by purely *formal* aspects (unit dependencies), and does not reflect the *logical* dependencies between objects, variables, procedures etc.

The problem is (very simplified): in a compiler everything depends on everything, so getting some structure is only possible with some formalism, formalism is the last resort when logic does not work anymore. Formalism is also something absolute while logic might be rather subjective.

This lack of logical structure, next lack of up-to-date documentation,

Nobody volounteered yet to continue Carl's work of documenting the compiler.

Florian Klämpfl wrote

Indeed. And the features of git gui are just ridiculous. It misses even basic stuff like good explorer integration.

That's a matter of opinion I guess. I much prefer stand-alone tools, so I think they hit the nail on the head. I hate shell or ide integration for SCM (of even some other types) software - they always have a limited feature set. As for 'git gui' hanging, I haven't experienced that's nothing new on your system,
because as far as I remember you couldn't even clone the fpc git mirror, where others could. I might be your Windows setup - just guessing here.

Yes, I forgot that you don't like easy to use software.

so I think they hit the nail on the head. I hate shell or ide integration for SCM (of even some other types) software - they always have a limited feature set. As for 'git gui' hanging, I haven't experienced that's nothing new on your system,

Oh really? Strangly enough other software works fine on my system :) Sometimes one can guess from the look of a software (yes, git gui looks poor) on its quality.

because as far as I remember you couldn't even clone the fpc git mirror, where others could.

Before your wildly guess, you should either look up the old thread or at least remember correctly. At that time, git failed on a completely different system: it was an Ubuntu 8.04 LTS where the cloning failed. Different OS, different location, different machine, different internet provider.

I might be your Windows setup - just guessing here.

Well, or the crappy software :)

Florian Klaempfl wrote

Yes, I forgot that you don't like easy to use software.

I'm using FPC and Lazarus, aren't I? ;-)

Oh really? Strangly enough other software works fine on my system :)

Nothing strange really. Windows behaviour is so random, nobody can predict how Windows will behave next. My wife (which runs a digital studio) can vouch for that! :)

Sometimes one can guess from the look of a software (yes, git gui looks poor) on its quality.

You should know the old saying: Don't judge a book my it's cover.

Have you ever seen cad engineering software or machine control/monitor software. They all look pretty crap to me (UI interface), yet with the correct person driving those software, it works like a dream to them.

Anyway Florian, nice to see I'm off your "junk mail" filter. ;-)

The install is crazy, it basically installs a whole mingw/msys system (when I last checked). I've already got an mingw/msys install and having two is suicide. I've heard that people use the git install as a base and then install other mingw/msys packages on top of it.

Henry

Replied on fpc-other.

Henry Vermaak wrote

The install is crazy, it basically installs a whole mingw/msys system

Replied on fpc-other.

What's the problem with using threadvars

As discussed some days ago, the current implementation of Threadvars is quite slow (doing a libc call on each access to a threadvar). This _could_ be improved but I think this only makes sense when removing the dependency of the RTL to the libc pthread library, which seems to be planned on the long run..

Thus Hans-Peter's idea to do no or only a few seldom accessed Threadvars as the base of the objects that exist on the heap and usually are addressed by local variables (on the stack) seems more appropriate.

-Michael

Michael Schnell wrote

> What's the problem with using threadvars As discussed some days ago, the current implementation of Threadvars is quite slow (doing a libc call on each access to a threadvar).

And? Those variables aren't used very often. And if needed, it can be improved later.

No.

You still didn't point out which locations you actually mean.

Why do you want to reduce them? Logic or formalism?

so that finally the compiler could be decomposed into a small number of building blocks (packages...), with tiny interfaces to each other.

Counter example: your attempt of the OOP parser :)

Yes.

As mentioned above, I've started an documentation attempt some time ago. But after many arising questions remained unanswered, I had to figure out everything myself.

Indeed, that's what we do as well :) Some parts of the compiler, e.g. parser, are probably not touched for 10 years.

And after I found out that the current compiler implementation is based on formal requirements, not on logical ones, I

As I said before: formalism is absolute, logic is subjective so in complex situation formalism is really helpful.

I see the main problem in the identification of all the global variables, that have to become threadvars. Or do you know of an way, to make the compiler identify variables that have to become threadvars?

IMO it won't take more time than any of my other experiments, to set apart some small part of the compiler for use in threads. The required threadvars have to be identified in that attempt, too, but possibly in a smaller scope.

It was one of my first experiments, to create a new target with no code generation. Unfortunately I forgot to backup the according documentation, so that only the code is still available.

Another more recent experiment was to convert the back-ends into classes. It turned out that much target-specific code resides in the general compiler units, whose separation into virtual back-end methods would involve much work. In another approach the $IFDEFs could be replaced by ordinary IFs, what would both simplify the redesign and preserve the current compiler speed.

I dare to disagree. It's not so hard to encapsulate everything in dedicated (mostly existing) classes. Only the requirement, that *external* tools should be able to use part of the compiler, without importing too many dependencies, results in the implementation of otherwise unneeded hacks and workarounds. In a radical OO redesign the number of involved units (and global variables) could be reduced dramatically, so that finally the compiler could be decomposed into a small number of building blocks (packages...), with tiny interfaces to each other.

> This lack of logical structure, next lack of up-to-date > documentation, Nobody volounteered yet to continue Carl's work of documenting the compiler.

Where can this work be found? Do you mean the 1.x PDF document, that is not helpful with newer versions?

As mentioned above, I've started an documentation attempt some time ago. But after many arising questions remained unanswered, I had to figure out everything myself. And after I found out that the current compiler implementation is based on formal requirements, not on logical ones, I see not much use in providing documentation for that unsystematic state. IOW the current implementation deserves more badly a redesign, in favor of documentation. IMO.

DoDi

Threadvars *are* used frequently, as long as almost all currently global variables have to become threadvars. When I moved the global scanner variables (c, token...) into the scanner object, every access to such a moved variable requirs access to the thread-local scanner object first.

When objects are used as containers for thread-local variables, these object references (to the thread-local objects) should be cached. This will have to be done inside every procedure, that makes use of a reasonable number of threadvars. E.g. current_scanner or current_module can be copied into local (stack) variables, for further use inside that procedure. That's at least better than having a huge number of references to individual threadvars.

I hope that it has become more clear now, why I want to remove as many global variables as possible.

DoDi

I stopped searching after finding such code in aasmtai, aoptbase, aoptobj, assemble, globals, nadd, ncgutil, nld, ogcoff, options, optloop, pdecsub, pmodules, psub, symdef, systems.

As an extreme case, tcgprocinfo.generate_code requires 6 additional virtual methods.

For logical (functional) grouping. Looking at the parser, a single unit would be sufficient for every front-end. The current high number of parser units results from the inclusion of the semantical and framework actions, which should be moved into their own interfaces (or units), or better into the already existing interfaces of symbols, nodes etc.

WRT to multiple front-ends, true functional interfaces (methods) should be added to the basic framework classes. These classes currently are mere collections of fields and basic methods, that do not even encapsulate the management of their owned objects. See InitCompiler, InitParser, parser.compile and all the other parser stuff, that creates and manipulates objects instead of delegating such tasks to the involved classes.

What's wrong with the parser interface in TParser?

After a separation of unrelated parts (see above), the remaining code can be restructured again into local subroutines, that do not appear even in the specialized class declaration.

We seem to have different definitions for "formalism". Of course it's helpful to have parser units start with "p", node units with "n" etc., but just the parser units lack any functional separation into syntactical parsing, semantical actions, target specifics and flow control.

Logical grouping is not necessarily arbitrary or subjective. A compiler can be subdivided into front-ends, back-ends and infrastructure. Back-ends can be subdivided into system and CPU, with further exchangable functional blocks for e.g. object and debug information formats. The compiler infrastructure can be subdivided into parametrization, file/module management, and intermediate data storage. Intermediate data can be subdivided into symbol tables (declarations) and code (AST), with dedicated interfaces for the front- and back-ends, and general optimization.

A formal separation of cpubase from cpuinfo is not helpful, unless it has a logical (functional) justification.

DoDi

Florian Klaempfl wrote

And if needed, it can be improved later.

As discussed recently, it supposedly _can_ be improved even without completely removing the libc-pthreadlib dependency (e.g. for X86/32 using FS in Windows and GS in Linux according to how MS-C and GNU-C do threadvars) but this is a little bit kind of a hack, relying on the (quite secure) assumption that the not officially documented behavior of Windows and the (documented) behavior of libc do not change in Future on all supported architectures.

-Michael
(lets not go into this at any depth right now :) see the other threads on the issue)

Hans-Peter Diettrich wrote

When objects are used as containers for thread-local variables, these object references (to the thread-local objects) should be cached.

Yep. that is what I supposed you would do to eliminate threadvar access: the threads work on their own instances of the object that hold the values that formerly were global variables.

-Michael

I very much support your arguments !

But do you really think you can take on the work of such a huge restructuring of the complete source code of the compiler including all OS and Architecture variants ?

-Michael

Well, I see no problem with it. Of course, one can do an overly complex design to avoid this but as I said before, I see no use in doing so. It makes only maintainance more complex.

This is not logical to me. For me it is logical to group the parser by expression sub parser, declaration sub parser etc. and even split the declaration parser into type etc. one.

That it inherits from TScanner.

As you said multiple times: *can*. One can also subdivide more or less or even choose another grouping.

A formal separation of cpubase from cpuinfo is not helpful, unless it has a logical (functional) justification.

It has for me a logical and formal one: cpuinfo contains info which is used by generic/common parts of the compiler. Cpubase contains the basic stuff which is used by the cpu specific part of the compiler. Of course, there are deviations from this rule due to laziness and simplicity and refactoring over time, but this is how it is used to be.

That's why I suggested solutions other than $IFDEF or virtual methods.

Many $IFDEFs also occur in declarations and uses clauses. These parts should be removed from the general compiler code, with all such details encapsulated in the back-end class(es).

A parser structure should support the maintenance of the language, i.e. syntax. In so far I'd group together e.g. directive handling, to ease the maintenance in case of new directives. Your suggested structure applies to the semantics of the language, which should be implemented outside a concrete parser (front-end). It's also a bad idea to move procedure processing into a different class (tprocinfo), and let it call paser methods from there. That's turning things upside down, breaking the recommended separation between front-ends and compiler :-(

In an OO design the parser methods must reside in one class, so that the means for further structuring are very limited. That's why I suggested local subroutines for parsing specific (non-reusable) parts of the language.

It's only a matter of taste, whether the parser should be a member or the base of the parser class. My choice of a base class allows to hide many methods and fields from the public scanner interface, that shall be used only by the parser. A member also introduces another reference (indirection), and another (global) variable, so I decided to inherit from the scanner class.

The next decision, of turning the parser units into include files, is also only an intermediate step. When the semantics have been separated from the parser, the code in these include files can be moved into the language unit, and the remaining p* units will contain nothing but the semantical interface to the intermediate data structures. *That* is the place where your suggested structure makes sense, no doubt :-)

Thanks, I'll remember that for a later later refactoring. In my attempt to unify the existing back-ends, I came across several abstract classes (tai...), which look like the more "natural" candidates for back-end base classes. When the cpubase (and more) information and procedures are moved into the interface of these classes, unwanted references to concrete cpu units can no more occur in both the compiler and external tools.

DoDi

Using ifs is usually not possible because some symbols simply don't existing on certain architectures.

Well, it should also support the maintaince of the compiler code ;)

In so far I'd group together e.g. directive handling, to ease the maintenance in case of new directives. Your suggested structure applies to the semantics of the language,

The distinction between e.g. expressions and declarations is completly syntax driven and has nothing to with semantics.

which should be implemented outside a concrete parser (front-end). It's also a bad idea to move procedure processing into a different class (tprocinfo), and let it call paser methods from there.

Actually the naming of t(cg)procinfo is probably bad, maybe it should be called now tproccompiler.

That's turning things upside down, breaking the recommended separation between front-ends and compiler :-(

Who recommends this? The writers of books like the dragon books? Those people never wrote a real world compiler and tried to solve the real problems of compiler writing.

Well, you can call a lot a matter of taste then.

Target specific node units should be included only by cpunode. If it is done this way, there is no problem

Target specific node units should be included only by cpunode. If it is done this way, there is no problem

This doesn't solve the order-sensitivity of the used units :-(

I've observed that effect already, and it may occur again at any later time. So the current code works as-is, and *only* as-is, so that it is near a dead end WRT to further development. That's why I feel an urgent need for combing out such bugs, independently from any other goals.

DoDi

> That's why I suggested solutions other than $IFDEF or virtual methods. Using ifs is usually not possible because some symbols simply don't existing on certain architectures.

A solution for such problems was one of the first steps in that refactoring :-)

Life were much easier to me, when all targets could be checked (syntactically and at runtime) in one (Lazarus) project. E.g. I simply defined all implemented targets, and immediately got a bunch of errors, resulting from incompatible (duplicate...) code and declarations. So my primary interest in a new back-end structure is more targeted at ease of development, than on really using multiple back-ends in one application.

> A parser structure should support the maintenance of the language, i.e. > syntax. Well, it should also support the maintaince of the compiler code ;)

This is what I'm doing right now again, based on and extending the previous OO_rewrite stuff.

The NoGlobals branch actually includes most of my intended modifications. Since it's unusable for stepwise merging into trunk, I'm willing to provide patches (or another branch) for such a merge, where distinct patches look easier to merge to me. I only should know what changes are acceptable, so that I can prepare the according patches based on the current trunk.

This can be done as (your) time allows, i.e. as soon as I get concrete OKAYs for concrete steps. This will require a review of the NoGlobals branch, based on *content*, not only based on *formal* considerations. When I have to provide the patches for trunk almost from scratch, I can take into account any deviation from the NoGlobals or other branches, e.g. with regards to the base of the parser class.

When parser "methods" are called from the outside, i.e. from tprocinfo, then this has nothing to do with syntax. It also is a bad idea to call e.g. consume(_POINT) for the final "end." during code generation or other semantical actions; preserving the token position for useful error messages can be achieved by other means.

That's a reasonable change, but not towards multiple front-ends. A parser can provide and deposit all information in a tprocinfo object, for further processing. When that object provides an interface for inserting all related information, then it can be reused with other front-ends, which only differ in the syntax of the language - and this is one of my refactoring goals. It can be accepted or rejected, I only should get a definite answer.

My experience recommends that. Even if I never wrote an compiler of my own, I've implemented a number of decompilers in the last 25 years. Starting with my first such attempt (a C decompiler on my Atari ST) I know a lot about the different code generation of concrete compilers, and the requirements on different platforms. This insight allows me to determine much about the good and bad concepts in concrete compilers, without having ever seen their concrete implementation.

Please understand me right, I do not want to tell anybody how things *must* be done. I only want to suggest general changes, that help to overcome general problems, as known from other concepts and implementations. It's up to the core developers to accept or reject my ideas, and I'm willing to assist in the implementation of every accepted change. Currently we still are before any decision, what should be changed at all, for what win.

DoDi

Hans-Peter Diettrich wrote

So my primary interest in a new back-end structure is more targeted at ease of development, than on really using multiple back-ends in one application.

+1
-Michael