Hi,

I'm using fpmake more and more in my projects (large or small) and from time to time I come across some limitations that need to be resolved. In the past I have supplied a number of small patches which were normally applied without problems, but sometimes I was sent back (mostly with good reasons) to rethink a patch. However as I noted in http://bugs.freepascal.org/view.php?id=18151this approach is sometimes like a "trial and error" strategy and I would prefer if we could create something like a roadmap for fpmake and put that up on the wiki. Because it's a FPC program it's of course always possible to add functionality yourself. The question though is, what will FPC support out of the box.

A couple of things that come to mind:

- generation of documentation (fpdoc and possible other??)
- compilation of example files
- compilation and running of the testsuite
- generation of the classchart
- execute fpmake from command line (see thread by mattias : "run pascal programs as script") for this we would need instantfpc to support win32 and it should be supplied as a standard fpc tool
- implement the creation of installer (scripts) for all platforms
- implement a binary zip option
- managing debug/release builds (also cross-compiling)
- cmake like system (display and store information on builds and test suites, etc)

BTW: I suppose testsuite and example files could be compiled by a separate fpmake file, but it sounds not logical to create a separate package for that, or does it?

Of course these are all nice to haves, but agreeing what is in and what definitely not does not hurt. All "out of scope" functionality could be added to an external fmkextunit.

Regards, Darius

No problem with adding those.

- execute fpmake from command line (see thread by mattias : "run pascal programs as script") for this we would need instantfpc to support win32 and it should be supplied as a standard fpc tool

I doubt this is feasible for windows. As far as I know, no #! comparable construct exists.

- implement the creation of installer (scripts) for all platforms

I doubt a cross-platform installer is feasible within the scope of FPC. If you mean the creation of a skeleton inno setup script or a .rpm or .deb spec, that can be thought of.

- implement a binary zip option

This should be possible.

- managing debug/release builds (also cross-compiling)

What is failing in the current system for this ?

- cmake like system (display and store information on builds and test suites, etc)

fppkg should handle that part of the job, not fpmake.

Michael.

Am 30.03.2011 17:31, schrieb

It is extension dependend on windows: typing just test.pp starts lazarus on my windows.

So it does here, but I doubt this is what we want ? :-)

Michael.

In our previous episode, said:

While I really like the instantfpc gimmick, I don't think it should be a part of any crucial infrastructure. At least on FreeBSD shells need to be registered, there are special rules for their permissions, security implications etc etc.

It is not a shell. For example a perl program requires the perl interpreter.

Mattias

> - managing debug/release builds (also cross-compiling) What is failing in the current system for this ?

I was thinking to be able to do something like :

fpmake build debug

This will add some debug options to the build process. Also it should be possible to create a custom build profile.

> - cmake like system (display and store information on builds and test suites, etc) fppkg should handle that part of the job, not fpmake.

To make fppkg work for my own project I need to add a fppkg.cfg to my project right?

Regards, Darius

It's not. A possible workaround is to use a distinct extension for makefiles and assign this extension to instantfpc.

Regards, Darius

And how would you do this ?

I don't want to complicate fpmake unnecessarily.

>> - cmake like system (display and store information on builds and test suites, etc) > > fppkg should handle that part of the job, not fpmake. To make fppkg work for my own project I need to add a fppkg.cfg to my project right?

There is no such thing as 'fppkg for a project', so I'm not sure what you mean here.

Michael.

I don't think this is needed/wanted. fpmake.pp is intended to be used in tandem with fppkg, and that looks for a fpmake.pp program, not a script.

(not to mention all the possible complications on Windows, probably Mac OS as well)

Michael.

Improve the command line paring algorithm. It should check if there is something behind the build command line option. If so then it is a named build profile. In fpmake you could check the TDefaults.Profile property to determine which options to add.

But I am talking about using fpmake for projects other than FPC. So the cdash (not cmake as I mistakenly wrote in the mail) idea should also work for standalone projects.

Darius

For FPC yes, but for standalone projects this is not the case as you mentioned. Apart from that I don't see an issue even for fppkg, other than the fact that the extension might be hardcoded, which could be modified. Please note that I'm for a more holistic approach for fpmake.

(not to mention all the possible complications on Windows, probably Mac OS as well)

Can you name a potential issue you see by renaming the makefile to some other extension? I thought that FPC does not mind which extension you use, as long as the syntax is correct?

Regards, Darius

I would prefer a named option, i.e.

fpmake build --profile=debug

and I think TDefaults.Profile needs to be
a) A collection.
b) Loadable from some config file, I suppose ?
--profile then just selects one item in the collection.

Other than that, I think this is OK.

In that case, I don't understand what you mean.

fpmake's intended use is to compile your FPC code. I can see the use of trying to run a testsuite, build docs and a binary zip for distribution: they are all FPC related things.

But it is not intended as a general build system (i.e. replace make or ant or rpm or deb).

Michael.

Which means ?

> (not to mention all the possible complications on Windows, probably Mac OS as well) Can you name a potential issue you see by renaming the makefile to some other extension? I thought that FPC does not mind which extension you use, as long as the syntax is correct?

It does not mind.

Well, if you don't intend to use it for FPC, you can do all you want;
you don't need our permission for that ?

Michael.

from the users perspective this is not very friendly.

and I think TDefaults.Profile needs to be a) A collection. b) Loadable from some config file, I suppose ? --profile then just selects one item in the collection.

So you mean have a fpmake.pp and next to that a fpmake.cfg? That does not make sense. The fpmake.pp in itself is a config file (well sort of). I would just prefer to keep it simple, maybe have the user register the profiles in fpmake.pp?

What do you mean with "general". Fpmake is a build system right? So why not use it also for other projects? Ok, perhaps the cdash functionality belongs in a different tool (although it could be in fpmake), but let's not limit fpmake only to FPC please. Let us (end users) also play with it ;)

Darius

Michael Van Canneyt wrote

>> I don't think this is needed/wanted. fpmake.pp is intended to be used in tandem with fppkg, and that looks for a fpmake.pp program, not a script. > For FPC yes, but for standalone projects this is not the case as you mentioned. Apart from that I don't see an issue even for fppkg, other than the fact that the extension might be hardcoded, which could be modified. Please note that I'm for a more holistic approach for fpmake. Which means ?

Which means that 1) when using fpmake outside FPC we can do whatever we want (provided we do not run into a wall because of a hardcoded extension somewhere). 2) even for FPC I see no objective argument not to change the extension.

I was not asking permission or did I? It was you who stated that there were complications. I just asked to name one. Don't get me wrong, there might be some, but I fail to see which.

Darius

It is more clear what is meant.
All other options are also specified with a --option

Not next to it, on a central location on your system.

Rationale:
I imagine that one wants to have some often-used profiles on a central location. These will be machine specific:

debug
profiling
testing

if you download a package and compile it, you'd have to edit the fpmake.pp and add these profiles each time. If the profiles are in a ~/.fpmake
file, which is loaded on start, then you need to define the profiles only once, and they will be available for all packages you wish to compile.

That does not make sense. The fpmake.pp in itself is a config file (well sort of). I would just prefer to keep it simple, maybe have the user register the profiles in fpmake.pp?

See above for the rationale. But obviously, they must be definable in code as well. I see the above as an add-on to manual definition only.

fpmake is intended as a build system for pascal code.

So why not use it also for other projects? Ok, perhaps the cdash functionality belongs in a different tool (although it could be in fpmake), but let's not limit fpmake only to FPC please. Let us (end users) also play with it ;)

You're of course free to do as you please. But when judging your proposals, I will always reason from the point of view of a FPC-only build tool. You're welcome to view it broader, but this is not the intended purpose of fpmake.

The idea is to keep it simple to build and FPC code. As long as your proposals do not conflict with that: feel free. Till now you're doing a good job :-)

Michael.

But I didn't claim to know all possible complications. I just meant to say that are possibly catches which we currently don't know about; so jumping on a system that relies on the "instantfpc" trick for all platforms may be a bit premature. The thing is barely a week old :-)

That said:
Obviously, you can give your fpmake script/program any extension you want for your 'outside fpc' projects, associate whatever you want with it on all platforms you see fit.

But since fppkg is for the express (and only) purpose of installing FPC code packages, I see nothing wrong with requiring .pas or .pp as an extension for the fpmake file if it is to be used with fppkg.

Michael.

I won't make a fuss about this as it's also a matter of taste. More important is the functionality itself.

One remark, with your reasoning, all packages will support each defined profile. This is not necessarily the case, also for FPC. You could even argue quite the opposite, but as long as you have comprehensive messaging it should not be a problem, although it will generate quite some "false" errors on building.

As long as it's possible to register profiles in a fpmake file itself, I'm fine. For a standalone project this makes more sense. Imagine you have debug, profiling and testing setup locally but the project adds "profiling" as a build profile. Then fpmake would not recognize it (and raise some sort of error message because it's an invalid profile) until you added it yourself to the local config. So having both options is best here.

I'm all for that. Let's live peacefully together :)

-D-

You're absolutely right of course. And I didn't want to imply we should switch in 24h :) I was just exploring the possibilities here. Using instantFPC once added as a FPC tool makes usage much more user friendly!

That said: Obviously, you can give your fpmake script/program any extension you want for your 'outside fpc' projects, associate whatever you want with it on all platforms you see fit. But since fppkg is for the express (and only) purpose of installing FPC code packages, I see nothing wrong with requiring .pas or .pp as an extension for the fpmake file if it is to be used with fppkg.

Also there everything you say is logic, although it does not mean that there couldn't be any other ways to reach a goal. However having an extension that is "executable" on all platforms seems like a valuable addition for FPC and Lazarus does it not? If we could reach consensus on something like .pex (for Pascal EXecutable) or whatever is appropriate and does not clash with existing software, then the FPC and Lazarus installers could associate this extension to instantFPC. Of course this is only valid for win32, but having an agreed extension would make things more transparent and uniform.

Regards, Darius

Imagine you have debug, profiling and testing setup locally b

Should be debug, release and testing

Exactly :-)

I think it highly depends on what you will put in the profile.

Standard profiles such as debug and release profiles will probably simply have some -gl -vw -dDebug or -Ur and -O2 options, which are understood by all projects.

if your projects profiles need more specific defines or options, then they must obviously be code in code.

Agreed.

I would definitely load profiles from file only after all user profiles are made, and would not load something again that the user already defined in code.

Like I said: an add-on to manual definition only...

Michael.

In essence you're saying:
now that we have instantfpc, why not agree on a file extension for scripts to be used with it, and register it by default ? If so, then: yes. I can live with .pex; I could equally well live with .pes (Pascal Executable Script) or whatever. I have no real opinion about that.

Mainly since I'm a unix user, and will write my scripts without extension anyway ;-)

Michael.

Darius Blaszyk wrote

> Imagine you have debug, profiling and testing setup locally b Should be debug, release and testing

Nono, I actually meant profiling as different from debug. The difference being that debug has all optimizations off, profiling has them all on.

So "Release" should be added to the list:-)

Michael.

My view on (traditional) Make:

Make is a script-driven shell, with rudimentary build-related functionality that's missing from other shells (mainly file handling). All other functionality sits in external tools, mainly provided by the platform or GNU. Make has been found insufficient for certain tasks, over time, so that another macro-processor (M4) has been added, mainly for the determination of the capabilites of the host system and external tools. That's the typical Unix model: when you cannot master a job in a shell script, using existing tools, then add another tool.

In a new approach I'd provide the interpretation of existing MakeFiles, and extend it to the specific needs and capabilities of the FPC/Lazarus project (package...) model. Then I'd add ports of frequently used tools, as libraries, not only for speed reasons, but also for debugging complex build jobs.

WRT time consuming jobs, observed e.g. in building FPC itself, I'd integrate "clean"ing of the output directories, together with means to reduce the number of such directories, as far as possible. I'd also add FPC as an integrated module, with a reusable file cache to prevent excessive directory scans and loading of configs and modules.

The compiler module can be used in the Lazarus IDE as well, either as a built-in module or as a shared library, with beforementioned ability to share the module cache with the application. Unfortunately parallel compilation will never become available with FPC, so that other chances for parallel processing should be explored. As long as the build tasks are based on input/output files (disk I/O), a shared file cache looks like the most promising way for speeding up an build process.

When fpmake consists of dedicated modules for various tasks, these modules can be used to build other applications, like test suite runners or profile editors.

Just my €0.02

DoDi

I understand, but this was a typo in the previous mail I wrote. I just corrected it as I mentioned "profiling" two times in the example I made. (It was rather late :)

Darius

In our previous episode, Hans-Peter Diettrich said:

(Note that if I say packages here, I mean a logical set of units (and maybe tests, docs and examples though IMHO that is secondary). Nothing more, nothing less. IOW don't get BPL visions)

I agree with your view on the general Unix status.

In a new approach I'd provide the interpretation of existing MakeFiles, and extend it to the specific needs and capabilities of the FPC/Lazarus project (package...) model.

Personally I don't think the core makefile principles are worth preserving at all:

1. they are mostly generated anyway. The .fpc files are the info, not the makefiles themselves.
2. they totally pass-by certain core FPC principles (like being to able to compile multiple files in one go. In the packages this is worked around using buildunits)
3. For e.g. a multithreaded compiler, the number of files to be handed to the compiler should increase, not decrease (*)

(*) there are multiple issues here to fix:
- compiler is not mutithreaded.
- module system is scheduled for rewrite for about a decade now
- packaging (also static, iow DCP) could decrease number of files and speed up compiler. But what about backend tools?

This only works when integrated with FPC, as the compiler finds its own files. But that is all far, far away. The point is more that the current transition horse makeshift situation (fpmake) should be adapatable in the future.

At least currently its data is fairly abstract (though I'm not entirely happy having to specify fileinfo manually. That should be based on compiler feedback. But this is very difficult in practice since the exact buildprocess is OS dependant)

One big step of fpmake is the killing of tools outside the FPC project like make.

Such plans are on the decade scale. If you are interested in it, I'd start creating as many internal assemblers and linkers as possible and further the module-rewrite, and the organizing of units into larger concepts (packages or whatever) and reducing the proliferation of files.

When fpmake consists of dedicated modules for various tasks, these modules can be used to build other applications, like test suite runners or profile editors.

I think the core bit of fpmake/fppkg should be

(1) killing off external tools, and the compromises to package metadata that they force upon up.
(2) start developing a logical package concept, both from a building and packaging concept. Maybe with a file based representation (.lib/.dcp like) (3) provide some hooks for packages outside the current main build tree to integrate with release engineering.

Both 1 and 2 as a start to more involved systems. At future compiler devel implementing some in-compiler package/library concept or unit cache will more easily roll it out.

Point 3 is because there is an enormous rift between package in the tree, and outside of it. By cutting down the administration for this, we might get a more workable external package management.

Marco van de Voort wrote

1. they are mostly generated anyway. The .fpc files are the info, not the makefiles themselves.

I still prefer those Makefile.fpc files very much to the fpmake.pp files. The fact that fpmake.pp files are plain Pascal code makes them extremely flexible, but at the same time that also makes it much harder to quickly get an overview of what they do. Sticking to the Makefile.fpc format (or any other structured format) and then automatically generating something else from that (be it plain Makefiles, fpmake.pp or whatever) would make it much easier to maintain the actual build instructions, I think. It would also make it less likely that over time some fpmake.pp will become harder and harder to read.

Jonas

PS: I assume that currently there already is a way to auto-generate fpmake files like this: http://svn.freepascal.org/svn/fpc/trunk/packages/univint/fpmake.pp (or did you manually derive all those dependencies, or use a quickly hacked together script to parse the uses-clauses of those particular units?)

The idea is to completely get rid of make.

Michael.

In our previous episode, Jonas Maebe said:

(I have to be careful here, I'm not the fpmake/fppkg person, that is Joost, these are my opinions only)

Well, I can be quite short about this: me too.

However I consider this a transitional problem only. We are still in the raw plumbing and discovery phase. In time, I assume some autogeneration helper functionality will emerge to make it at least (way) more bearable.

OTOH the trouble should not be exaggerated. I did the entire packages/ hierarchy in a few hrs spread over 2 days, and that was without any previous fpmake experience at all, and communicating with Joost.

New packages are not added that often, and I once fpmake/fppkg gets more mainstream, directory layout and general procedures will be more tuned to it.

The trouble is that makefile.fpc does not contain all information. It is not a closed system, and has barely enough info to build, and depends on the autobuilding capacity of the compiler. Something that can't be used for packaging and other features.

The fpmake is more closed. It in theory knows enough to copy files to an isolated environment and compile, and then missing files will turn up.

This means that there are ways to potentially validate many aspects of a packages hierachy before a release.

IIRC there were two solutions. I first did parts with own based small filter programs that parsed -vt output. Later Joost pointed me to some script that was slightly better. For the rest I had a bunch of templates ready.

The raw output was then checked and amended for OS and CPU specific cases. It was very semi automatic, but the simple packages (one OS, or no OS/CPU specific data) went fairly fast, and the bulk of the work was customization of the template.

I can't exactly remember if this also handled the include situation. Afaik first the units were done, and later includefiles were added.

P.s. and initially there was quite some resistance against the .fpc scheme too :-) It worked out fine, mostly due to Peter's very hard work.

I know, but I was not discussing that (although I'm not a big fan of doing so either, but I know that's a done deal). What I said was "Sticking to the Makefile.fpc format (or any other structured format) and then automatically generating something else from that (be it plain Makefiles, fpmake.pp or whatever) would make it much easier to maintain the actual build instructions, I think."

It is possible to stop supporting make without switching to manually writing and maintaining fpmake.pp programs themselves. I don't think that allowing the full flexibility of an arbitrary Pascal program as build instructions a good idea, because it makes maintenance harder and makes it less easy to get a quick overview of what is built/done.

Jonas

The problem is that the Makefile.fpc does not contain enough info and logic for our purposes.

I think you're too pessimistic :-)

If you look at the existing fpmake.pp files in the RTL/packages, you'll see that they are still very simple.

Michael.

That's good news :)

That's all I'm asking for. However, auto-generation will exclude directly modifying fpmake.pp (just like you currently can't safely directly edit the Makefiles). The structured format can of course have provisions for injecting Pascal code into the generated fpmake (just like you can insert plain Makefile code via Makefile.fpc), but then at least the parts that are auto-generated and those that are not are strictly separated and cannot be accidently overwritten during an update. Or, in other words: you don't have to spend time on looking for custom code before updating the build instructions.

OTOH the trouble should not be exaggerated. I did the entire packages/ hierarchy in a few hrs spread over 2 days, and that was without any previous fpmake experience at all, and communicating with Joost.

There's still a difference between initial conversions and maintenance afterwards. The first time there are no potential older changes you might accidentally overwrite.

New packages are not added that often,

At least the package I referred to (univint) tends to change significantly over time. The Mac OS X 10.6 SDK split a bunch of headers and removed others compared to previous versions (most C programmers use umbrella headers, so they don't notice that either). As Carbon gets phased out, more and more things will probably change. It's not really feasible to manually go through the current list of dependencies and to update them where necessary. Otoh, replacing them with a newly auto-generated list currently risks removing code that someone else added to manually to fix something.

That's why I mentioned "or any other structured format".

Unless it's fully automatic so that the current set of dependencies can just be replaced with a new set of generated ones, that's not really doable for univint, I think.

Jonas

In our previous episode, Jonas Maebe said:

It remains FPC source :-) The autogenerated part can be in an .inc file, and included from fpmake.pp

Don't forget to include that .inc in the non-source files section though (exact details obviously not worked out yet) :-)

True. Still I don't think it is any different from updating makefile.fpc with autogenerated unit data. In both cases it is including a block of text into the rest of the file. And at least fpcmake has the .inc workaround possibility.

Provided that you can actually generate the content of course. But in straight cases (files compiled/accessed don't depend on target) this should be generatable from the output of the compilation of a buildunit.

And the buildunit can be fed with find . -name "*.pp |awk print ... just like the makefile.fpc can.

Maybe more an apache solution? univint105,univint106 etc, and only install one? (see bits below before you comment on multiple editing)

Speaking of which, I've been thinking about this, and maybe we should installed most packages one level deeper (e.g. units/$fpctarget/pkgs), except for versioned packages like httpd/apache. This way the pkgs patch can be recursively included, and the versioned ones explicitely, with way less chance on duplicate files in the searchpath.

There is a "law of conservation of trouble" (wet van behoud van ellende) going on here. Splitting packages, means doing fixing multiple times, supporting multiple versions with one set means complicated packaging and building. Take your pick, trouble each way

But IMHO this is a problem and a tradeoff of this package. The complications of this package are not the problem of the build system. With any generalized systems there are winners and losers.

Still, univint _DOES_ seem to meet the criteria to generate fpmake data from buildunit compilation output. winunits-jedi is another such case, and probably more OS header sets.

Not generalizable. So only for packages where files used and dependancies on units are not dependant on target. There you can parse compiler output of a buildunit. (and that could be considered the structured format, the uses line can be generated using a simple script)

The information that must be fixated is quite complicated. This means a fully automatic way is very, very complicated.

can just be replaced with a new set of generated ones, that's not really doable for univint, I think.

As said univint (still) falls into the trivial case.

If it gets more complicated, you need source level scanning. IOW parsing uses and {$i } directives, but somehow follow all nodes of $ifdef statements. Afaik the current fcl-passrc doesn't allow this, since it handles preprocessing like the compiler, only evaluating the TRUE parts.

That program should then also get info on include dirs etc. It would be a nice exercise in tree building :-)

But I hope you realize that I make this last bit up as I go. I doubt we ever will earn back such an investment. Damage control with scripting/generation for often used scenarios, and hard labour in the rest is the only real solution.

I step in too late to avoid some long discussions, but here I go: This feature is already there. When you use fpmake directly: -f, if you use fppkg (what I would recommend): fppkg -c.

When I want to do a cross-build I can do:

fppkg build -c win32
fppkg build -c i386-linux
fppkg build -c 244 (use another compiler-version)
fppkg build -c debug (well, not cross, but debug-information...)

Use 'fppkg listsettings'. It shows which configuration-file it uses, and where you can store your custom configuration-files. (Or 'profiles', if you like that term)

Joost.

Darius Blaszyk wrote

A couple of things that come to mind:

Note that you can build your own add-ins. I used it to build a Lazarus-add-in, so that you can install Lazarus-components. It is not ideal yet, but you can use this approach to add functionality, without having to adapt fpmkunit.pp.

Some of these ideas are long standing items on the wish-list. And what I miss is Lazarus-integration. In which we have to decide if we want to create some units that can be used by Lazarus, or a library that can be called...

But if you want a/my road-map:

1: Let the fpc-installers create fppkg configuration files (almost done, needs a lot of testing)
2: Build some packages from fpc itself with fpmake in trunk. Use it as a field-test. (testing that locally, I wanted to commit it last week but I've found a new issue, probably in trunk very soon) 3: Improvement of version-handling
4: Multi-threading

I still want to try to use fpmake to build the packages of fpc 2.6.0.

Of course these are all nice to haves, but agreeing what is in and what definitely not does not hurt. All "out of scope" functionality could be added to an external fmkextunit.

Indeed, as add-ins.

Joost.

It's not a bad idea to see if we can make a structured (and readable) format to create our fpmake.pp files.

I was also thinking about letting fpmake detect from the compiler-output if a dependency is missing, and give a warning if this happens, so it can be added.

Joost.

Thisnfeature is already there. When you use fpmake directly: -f, if you use fppkg (what I would recommend): fppkg -c.

Thanks, I will have a look how it works. I'll try to document it on the wiki.

On the side, fppkg is only for FPC right? So not useful for standalone projects.

Darius

You can use fppkg also to install your own (standalone) application, I don't see why not?

In fact, fppkg is only a wrapper. When you run fppkg, it will search for a file called 'fpmake.pp'. If this file is found it is compiled and run.

There are some extra's, like downloading packages from a repository and installing them. And rebuild all packages which are dependent on the package being re-compiled.

For example: if I make a change in fcl-db, and use fppkg to compile & install the new version, it will automatically rebuild & install all other packages which are depending on fcl-db. With the lazarus-addin it will even mark the Lazarus-package so that it will be recompiled when you compile your program.

Joost.

In our previous episode, Joost van der Sluis said:

> That's why I mentioned "or any other structured format". It's not a bad idea to see if we can make a structured (and readable) format to create our fpmake.pp files.

I think the problem goes deeper than this. Makefile.fpc doesn't list unit dependancies (units and .inc files), and fpmake does.

It's this detail that is the bulk of the work (at least IMHO, Jonas?) not the fact that one has to wrap p.targets.addunit() around it.

ACK. A build system based on the FPC/Lazarus project management, and Pascal "uses", can operate on a higher level (comparable to automake source files, instead of final makefiles).

1. they are mostly generated anyway. The .fpc files are the info, not the makefiles themselves.

[that's the mentioned automake level]

After my tries I'm not sure whether parallel compilation will really speed up compilation. In most cases all but one modules are in "suspended" state, while their "uses" clauses are evaluated by the compiler.

When a (Lazarus) project uses packages, then all required packages can be compiled in parallel. This doesn't require an multi-threaded compiler, when every package can be compiled by a new compiler process.

- module system is scheduled for rewrite for about a decade now

Does there exist a roadmap or wishlist, of the intended changes?

- packaging (also static, iow DCP) could decrease number of files and speed up compiler. But what about backend tools?

Packages must be recompiled as a whole, if only one contained unit must be recompiled. Disk I/O (file cache) usage could be reduced by a map file with the options of all compiled units. But when used units must be loaded anyhow, even if they deserve no recompilation, the benefit of such an added map is questionable. Okay, it could be given a try, in a renewed module system.

I dunno about platform restrictions, but a shared file pool looks feasable to me. Perhaps fpmake (or Lazarus) can hold a module map or entire modules in an MMF, shareable by all compiler instances or processes?

ACK!

My experience with (im)porting tools is, that only the interface should be modified, from reading commandline-arguments to a straight forward configuration of the internal variables. The worker code should stay as is, so that bugfixes and other improvements deserve not more than a recompilation of the updated sources.

Have e.g. a look on Abbrevia vs. 7zip. Abbrevia never comes to an end with even the maintenance of a few archive formats, while 7zip (as a shell) can be extended easily by only adding an interface to every new compression module.

The same concept could be applied to fpc, so that it can be integrated tightly into fpmake - at least for non-cross-platform builds.

I don't understand the packages issue right now. What's the mentioned "tree"?

DoDi

Naive question: is this all about dependency determination?

If so, I'd attack the problem from the other end: an efficient scan of the searchpath (both source and compiled), from which the locations of all later required ppu and source files can be determined easily.

DoDi