Hi!

Here comes my 'official' plan on how I want to implement Delphi-like Package Support for FPC on Win64. Feel free to comment.

Definitions
~~~~~~~~~~~
<File>.ppl Free-Pascal Package Library
(shared module, Delphi: bpl)
<File>.ppk Free-Pascal Package
(source, Delphi: dpk)
<File>.pcp Free-Pascal Compiled Packge
(header, ppu's, Delphi: dcp)
(skip package collections for now)

-K <File>[ <File>]*
Command line arg to tell fpc to compile
program/dll with runtime package support.

-> M1
~~~~~
Goal: - Compile System unit into a "fpcrtl.ppl"
- Create a simple application that uses fpcrtl.ppl. -> Perhaps some simple string sorting program.

Idea: Create a TPackageModule class, might be some descendant of TPpuModule or TModule, that holds the information found in the ppk. Don't include the ppu files in the ppk (yet).

Change pmodules.proc_package to load all units found in the packages of the requires section, create idata for the ppl, create pcp file.

Add command line parameter (-K) so that applications can be compiled with packages. Change pmodules.proc_program to pre-load all units of all referenced packages.

-> M2
~~~~~
Goal: - RTTI in ppk
- Basic OOP support in ppk (ClassInfo etc.)
- Include all ppu information in pcp file

To Consider:
Add Name mangling for RTTI (etc.) if not yet existant.

Find code in the rtl that needs to be changed (if any).

Threads/Thread Local Storage?

Tests.

-> M3
~~~~~
Goal: - Implement missing functionality
(Weakpackage, CompilerSwitches...)
- Provide ppk files for class library
- add tests
- merge into trunk

Cheers,
Willi

On 3 Oct 2010, at 17:43, Willibald Krenn wrote:

With regards to 64bit Windows:

Feel free to comment.

Is there a particular reason why you're sticking with TLA's? Surely, Win64 has progressed past this limitation?

Definitions ~~~~~~~~~~~ <File>.ppl Free-Pascal Package Library (shared module, Delphi: bpl)

<File>.fppl

Or

<File>.fplib ??

Why force a backward facing legacy on the an OS that does not have such a restriction? Even NT4 supported longer file extensions.

M

In our previous episode, Willibald Krenn said:

- is it really necessary to change extensions to ppl/ppk ? These formats are not compatible between Delphi anyway.
- dcp can also contain implementation part of .dcu which is .o in FPC.

- in a systematic way without any duplicate state or manual intervention

If too difficult, break up M1 in multiple milestones. But the fact that the division is wholly automatical (given the same input data as delphi, so a dpk/dpr and a few settings that signal "build with package x,y,z) is crucial for packages.

That is also more important packing everything in .ppk's exactly like Delphi.

A solution could be to give .ppu's (and .o's if necessary) a different extension if generated as part of a package system.

Add command line parameter (-K) so that applications can be compiled with packages. Change pmodules.proc_program to pre-load all units of all referenced packages.

You will need an additional parameter to specify from which compiled packages (.dcp's) .ppu's must be preloaded.

If sufficient clean and doable, this might already be a good point to merge into the trunk.

But the main point of packages is total control (and thus checking) by the compiler, not "just having something dynamic".

With this argument you want to specify which packages to use? Then you should use a syntax that is more compatible to the other FPC options:

-K<File> [other options] -K<File> [...]

Also you might want to add it as a new -F* option (e.g. -Fp or -FP) instead of -K, because the -F* options are for filenames and paths.

If you want this TPackageModule or whatever in system.pp (e.g in the RTL, not in the compiler) you should use records, because objpas.pp is not loaded inside system.pp (and thus TObject isn't available).

What do you want to mangle here?

Find code in the rtl that needs to be changed (if any). Threads/Thread Local Storage?

Yes, that will be another difficult topic...

As it doesn't seem that you're working in a branch on the FPC repository you might want to send small patches (e.g. your milestones) to the FPC team as they can be discussed and merged more easily than one big patch (and you don't want to repeat your complete work only because a important feature at the beginning wasn't implemented correctly ^^)

Regards,
Sven

Am 03.10.2010 20:06, schrieb Sven Barth:
Willibald can get access to a branch to commit his work.

Florian Klämpfl wrote

Am 03.10.2010 20:06, schrieb Sven Barth: Willibald can get access to a branch to commit his work.

Also an idea. But you might outline some rules for commits so that not the same problems happen as with DoDi's branches.

Regards,
Sven

Hi,

To minimize traffic, I'll answer all comments (thanks!) in this e-mail.

> Willibald can get access to a branch to commit his work. Also an idea. But you might outline some rules for commits so that not the same problems happen as with DoDi's branches.

A branch would be nice, I guess. So everyone had a chance to see what I am doing. Just to be clear about my intensions: I don't want to change existing code. Only if there is no other way I will introduce changes, but these will be minimized as much as possible.

Is there a particular reason why you're sticking with TLA's?

None except that practially all files used by FPC have TLAs. Anyways, these file extensions can be changed at any time. For now, I'll stick to my given defs.

is it really necessary to change extensions to ppl/ppk ? These formats are not compatible between Delphi anyway.

Well, I want to avoid confusion: FPC is not Delphi. Probably dpk could stay as it is, but then I tend to think it is beneficial for everyone to know when you are talking about dpk's you are talking about Delphi and if you refer to ppk, it's fpc. But in the end that's not so important.

Also you might want to add it as a new -F* option (e.g. -Fp or -FP) instead of -K, because the -F* options are for filenames and paths.

Sure, that's better than -K! So I will use -Fp<Name> for the package names and -FP for defining a separate pcp search directory (if needed).

dcp can also contain implementation part of .dcu which is .o in FPC.

Yes, they probably do. For the beginning (and with generics in mind) it's probably easier if they do.

Regarding M1
============

That is also more important packing everything in .ppk's exactly like Delphi.

Ok, so I refine my goal for M1 a little bit: Goal for M1 is to have the compiler use the package-system, meaning one RTL (with system only), no other .o/.ppu files, just ppl/pcp and the main.pp. Of course I am talking about a console application (single threaded) here.

Around the same (~M1.5) time it should also be possible to re-do this example with one fpcrtl.ppl with system unit, one other package using exports of fpcrtl and the main application depending only on that other package. (This would test imports for ppl's.)

If you want this TPackageModule or whatever in system.pp (e.g in the RTL, not in the compiler) you should use records, because objpas.pp is not loaded inside system.pp (and thus TObject isn't available).

Good to know. Actually I hope I can hide the package-thing from the RTL, which probably is a bit unrealistic ;-). Well, at least for M1 I'll concentrate more on compiler internals than RTL aspects.

Regarding M2
============

What do you want to mangle here? [RTTI]

RTTI for types needs to be exported, so it needs to have a name. I don't know yet whether there already is one.

Cheers,
Willi

In our previous episode, Hans-Peter Diettrich said:

Then if you have N packages, you need O(N^2) sets of compiled files? Since modules that have code for late binding are different then for early binding.

I don't think this is very practical.

Here comes my 'official' plan on how I want to implement Delphi-like Package Support for FPC on Win64. Feel free to comment.

[...]

-K <File>[ <File>]* Command line arg to tell fpc to compile program/dll with runtime package support.

I don't like the Delph all-or-nothing philosophy here. I'd prefer to specify the packages, that shall be loaded at runtime, while all other packages can be linked into the executable. This would allow to use an minimal set of external modules, which are shared by the exe and the really self-contained packages.

The same effect could be achieved by a separation of packages into (Lazarus-type) static libraries, and packages *designed* for dynamic loading (plug-ins...).

Good luck :-)

DoDi

I fear also a lot of pitfalls with this possibility for example executables using dyn. packages might be never smartlinked because it is unknown at link time which packages will be loaded so all symbols of the statically linked packages must be available.

Hans-Peter Diettrich wrote

I don't like the Delph all-or-nothing philosophy here. I'd prefer to specify the packages, that shall be loaded at runtime, while all other

You can always load packages at runtime.. That is, after the RTL supports that of course.

packages can be linked into the executable. This would allow to use an minimal set of external modules, which are shared by the exe and the really self-contained packages.

What you probably(?) could do is to link all basic packages into the executable but still export all symbols, so that when the shared object is loaded it finds all necessary entry points.

However, at the moment I don't see why this is better than shipping all necessary ppl files. BTW: I realize that ppl might be a bad TLA - since it probably means "people" for quite a number of ppl ;-)

The same effect could be achieved by a separation of packages into (Lazarus-type) static libraries, and packages *designed* for dynamic loading (plug-ins...).

You had to guarantee that no plug-in depends on a static library that is also used by another module (so the exe or another plugin).

Good luck :-)

Thanks, but no luck involved ;)

Cheers,
Willi

Delphi compiler does this as well. One caveat in this discussion is for instance: when you "build with runtime packages" vclXXX.bpl, db*.bpl; even though you didn't specify them, all packages your bpls depend on are loaded runtime only as well. (rtlXXX.bpl for instance)

the reason is simply: there can only be one instance loaded of EVERY unit you use in your program (or that is included in some bpl).

It is therefore simple to say in delphi that you want to build with rtlXXX.bpl only, but still have the vclXXX.dcp statically linked into your executable when you use a TForm and include the Forms unit somewhere in your code.

kind regards,
Dimitri Smits

maybe others want to contribute too. Will this be a Willibald-only branch?

[OT]
one showstopper (besides loads of time) I have is getting the fpc-trunc to compile and create a specific bin folder. as far as I've gathered is that you need to do a "make clean all" and then copy all the resulting exe's to somewhere (find and retrieve yourself strategy). But all those configuration files etc that you get with a binary release download are a mystery to get right. [/OT]

if I got this to work, then I'd be happy to volunteer some time on that branch as well.

don't forget "inline" as well, not just generics.

you'd have a bit more dependency-hell when you want to make your LoadPackage/UnloadPackage methods and need to either re-implement the dynamic-lib loading or use the standard x-platform one.

also: TPackagemodule should indeed better be a record (maybe have a look at versions beyond D7, to TLibModule for instance)

Which reminds me: ALL your build-targets should register their contents through packagemodule and consorts. That means the main exe or shared-lib as well. Otherwise it is impossible to detect duplicate units and double-initialisation and other troubles with the runtime linking.

A tree-structure emerges in the newer delphi's: Module - unit - types A module contains x units
A unit contains y types

careful for smartlinking those structures.

I think you mean replacing "RTTI_" with "@xp@" or something you are more familiar with for starters? RTTI is already provided. Try building with retaining assembler output to see what I mean.

Some link-names are indeed easily becoming too long, others would think the "all uppercase" is unneccecary. I'd say they are there, they work and were the personal taste of the core-team biased to it through experience and other arguments.

kind regards,
Dimitri Smits

Willibald Krenn wrote

-> M3

...

- merge into trunk

I believe this will be too hard. I prefere to make small changes that can't break the trunk and which adds some functionality, then commit them with tests. Merge of big rewrites is very hard if someone also do parralel changes.

Best regards,
Paul Ishenin.

maybe others want to contribute too. Will this be a Willibald-only branch?

I surely don't mind fellow contributors (au contraire!!) - if we agree on some partitioning of the work and one common plan. For example, it would be terrific, if someone could do Linux or Win32 as main platform. And with time, more and more opportunities for contribution will emerge, as all the RTL stuff needs to be implemented....

Well, I too had difficulties getting fpc to compile (with -gl). First thing, you need the FPC 2.4 compiler and then do e.g.,

make clean build install PP=fpc CPU_TARGET=<?> OS_TARGET=<?> DEBUG=1 OPT="-gl <whatever>" UPXPROG=echo COPYTREE=echo INSTALL_PREFIX=<install path> FPC=<path to fpc for bootstrap>

(should be one line). At least that command works for me. BTW, you'll need the mingw files that come with the 2.4 install. This being done, you can use lazarus to do the development and debugging.

Beware: On Win64 gdb 7.2 will crash when you raise an exception. This _might_ be caused by the non-Win64-standard exception mechanism FPC uses.

if I got this to work, then I'd be happy to volunteer some time on that branch as well.

Welcome! :)

> Yes, they probably do. For the beginning (and with generics in > mind) it's probably easier if they do. don't forget "inline" as well, not just generics.

Yep. (For inlined methods/functions we could - theoretically - provide a 'non-inlined' version in the binary package....)

you'd have a bit more dependency-hell when you want to make your LoadPackage/UnloadPackage methods and need to either re-implement the dynamic-lib loading or use the standard x-platform one.

I don't want to re-invent or re-implement things. :)

also: TPackagemodule should indeed better be a record (maybe have a look at versions beyond D7, to TLibModule for instance)

Ok, never had a look at D7+. There however is a difference between the metadata found in a bpl and the dcp. First thing I'd say should be accessible via some record-type defined in the rtl, latter is some compiler-internal thing (and the TPackagemodule I was referring to).

Sure. However, I'd have said that this is more of an M1.5 to M2 type of thing. First I'd like to see what changes need to be incorporated in the compiler to get the very basics done. Afterwards all the stuff on the RTL needs to be added.

On a side note, I've seen that we need to check/change/extend the _DLLMainCRTStartup (etc.) functions which should probably behave different in a package than in a dll.

A tree-structure emerges in the newer delphi's: Module - unit - types A module contains x units A unit contains y types careful for smartlinking those structures.

Hmm, I still have to think about smartlinking and packages. Is there a way to use smartlinking when compiling a package?

I think you mean replacing "RTTI_" with "@xp@" or something you are more familiar with for starters? RTTI is already provided. Try building with retaining assembler output to see what I mean.

Very well, then we just need to include that symbol in the exports/imports section.

Some link-names are indeed easily becoming too long, others would think the "all uppercase" is unneccecary. I'd say they are there, they work and were the personal taste of the core-team biased to it through experience and other arguments.

I really don't care about how names are mangled. If it works, we will use it. As already said, I'm not into a rewrite of the compiler *g*. I just want to use packages ;).

Cheers,
Willi

Delphi seems not to have problems with the binding of the compiled files. What I want is a linker feature, where the required packages can be flagged as link-in or keep-external, as far as their dependencies allow.

DoDi

I fear also a lot of pitfalls with this possibility for example executables using dyn. packages might be never smartlinked because it is unknown at link time which packages will be loaded so all symbols of the statically linked packages must be available.

IMO all symbols, that are usable from dynamic packages, also must reside in dynamic packages. In Windows terms: a DLL can and must know nothing about the using application.

DoDi

What you probably(?) could do is to link all basic packages into the executable but still export all symbols, so that when the shared object is loaded it finds all necessary entry points.

IMO a dynamic package (.DLL or .so) should only require (depend on) other dynamic packages. That's why just the basic packages (RTL...) must exist as dynamic packages.

However, at the moment I don't see why this is better than shipping all necessary ppl files. BTW: I realize that ppl might be a bad TLA - since it probably means "people" for quite a number of ppl ;-)

After some reasoning I assume that TLA (here) is not related to threads ;-)

In this (bad) case the static library might be linked into multiple packages - certainly not a desireable situation.

DoDi

It is therefore simple to say in delphi that you want to build with rtlXXX.bpl only, but still have the vclXXX.dcp statically linked into your executable when you use a TForm and include the Forms unit somewhere in your code.

This sounds like a post-D7 feature?

DoDi

Sent from my iPhone

No, Delphi 5 onwards allowed the developer to choose which BPL's are linked statically and which are dynamic. We used to link the RTL and VCL but have our BPL's external. I forget the reasoning, but it worked.

M

Op 2010-10-04 01:51, Dimitri Smits het geskryf:

fpc-trunc to compile and create a specific bin folder. as far as I've gathered is that you need to do a "make clean all" and then copy all the resulting exe's to somewhere (find and retrieve yourself strategy). But

No, simply specify INSTALL_PREFIX=<path_goes_here> on the install command line, after you did a 'make clean all'. It will install FPC to you chosen path. No need to manually copy exe's and *.ppu files out.

Here is the script I use to compile Trunk and install it to my preferred directory location.

---------[ start-trunk64.sh ]-------------------------- #!/bin/bash

COMPILER=/opt/fpc-2.4.3/x86_64-linux/bin/ppcx64

make clean
make all FPC=$COMPILER
make install INSTALL_PREFIX=/opt/fpc-2.5.1/x86_64-linux/ FPC=$COMPILER -----------------------------------------

Regards,
- Graeme -

Op 2010-10-03 18:56, Matt Emson het geskryf:

:-) I must agree with you 100%! So many still use *.cfg or *.txt etc... for a long time now I have used *.config or *.text without problems. The 8.3 naming convention is so DOS and Microsoft specific! So 80's. OS/2, Linux etc has had long filename support longer that I care to remember, so why not use it.

Regards,
- Graeme -

For generics: the token stream is already included in the PPUs (learned by looking at DoDi's ppudump patches ^^)

For inline: If I remember correctly inlined methods/functions/procedures are always available as non-inlined methods as well. It's up to the compiler to deside whether it inlines it or not.

Regards,
Sven

In our previous episode, Willibald Krenn said:

Runtime loading of loading is quite a different process from startup loading, and only used for plugin system (where the baseclass is statically linked, and the plugins implement the baseclasses). Dynamic loading for enormous amounts of symbols might be costly (since they are resolved with a call per symbol, instead of by one call to the system, as during startup).

The KDE suites startup speed problems have shown that very complicated dynloading schemes are not desirable.

The RTL is always loaded on startup btw.

Problem is that that leads to a dependancy circle (pkg->exe->pkg). Maybe not a blocker problem, but more a complication.

However, at the moment I don't see why this is better than shipping all necessary ppl files. BTW: I realize that ppl might be a bad TLA - since it probably means "people" for quite a number of ppl ;-)

Nah. Just wonder if that is necessary, if Delphi .bpl's are not compatible too eachother anyway.

In our previous episode, Dimitri Smits said:

(this is all in the http://wiki.freepascal.org/packages ovewview)

It is therefore simple to say in delphi that you want to build with rtlXXX.bpl only, but still have the vclXXX.dcp statically linked into your executable when you use a TForm and include the Forms unit somewhere in your code.

But that will be smartlinked, so that can still be a netto win for small apps.

So... what is your suggestion? Keep the platform specific library name?

Regards,
Sven

nope, D7 does this too. (maybe even D5, but cannot verify)

if you say you want to build with runtime packages x, y and z, the linker/compiler take into account that you implicitly dynamically link to the (units contained in) the packages that package x,y and z depend on (and recusively until no additional dependencies remain).

if in none of the above x,y,z (and the bpl's they depend upon) is contained a unit you use in your current build-target (exe, dll, ...), then the unit is implicitly included into that build-target.

kind regards,
Dimitri Smits

Dimitri Smits wrote

nope, D7 does this too. (maybe even D5, but cannot verify)

Yes, Delphi 5+ does this. It might even work in 3 or 4, I haven't tried it. Packages were horribly buggy in 3 from what I remember and I only used 4 briefly as 5 came out rapidly after we'd started using it and then I moved to a company that was a Borland partner and we had all the latest kit there.

M

In our previous episode, Sven Barth said:

bpl is fine with me on Windows. On *nix, what did Kylix use? Or maybe .dso ?

I'd not use *.bpl. People might get the idea to use FPC packages with Delphi (or the other way round). By using a extension that isn't equal to the Delphi ones we might avoid at least some questions about this topic. ^^

So I personally would prefer *.ppl (for Pascal Package Library) on every platform supporting packages.

Regards,
Sven

In our previous episode, Willibald Krenn said:

inlined functions for cross-unit use are already in the .ppu why change that? (and the .ppu becomes part of the .dcp). I don't think there changes so much wrt to this because packages always fall over unit borders.

In our previous episode, Hans-Peter Diettrich said:

That is generally the packages principle yes. Some very lowlevel (dependancy tables and RTL symbols that are also needed before the rtl.bpl initializes excepted maybe)

Sven Barth wrote

So I personally would prefer *.ppl (for Pascal Package Library) on every platform supporting packages.

Scenario - I'm using <someother> Pascal compiler
(GNU/Delphi/Virtual/etc) so I can use the PPL's from Free Pascal with them? After all, they *are* "Pascal Package Library" files?!

No matter how infrequently, this *will* happen at some point. I'd say, pick another more descriptive extension now, save "your - [collective] - selves" reiterating the same disclaimer every time someone asks this question. The assumption that "Free Pascal" is the only "Pascal" that will ever exist in a usable form going forward is probably subjective, but it's not a reliable constant.

Then let's pick *.fppl (Free Pascal Package Library). I don't mind what extension we use as long as it is not *.bpl ^^

Regards,
Sven

only the symbols that are used are imported in the dyn-loaded package. And it exports all publicly available types/procs/consts/vars.

to say even more, when you run with runtime packages, it's units are initialized first (before any other units/bpl's/the main exe).

when you use packages, it requires some discipline, that is right. Consider following scenario:

pkg1: your plugin-base classes, requires RTL package exe: imports pkg1 as a runtime package requirement, statically binds the gui components in lcl/vcl since it was not defined as a runtime package link requirement pkg2: a plugin for runtime loading, requires RTL package and VCL/LCL package

what would be result when you want to import this at runtime through a loadlibrary call?

I would suspect that you then have "duplicate units"-error. Compiler cannot defend against it due to the "runtime" information vs "compile-time" information.

Just to show that packages bring their own set of problems and pitfalls. But definately not all is bad ;-)

that is why versionnrs are included: RTL70.BPL (Delphi 7) vs RTL150.BPL (Delphi XE aka 2011)

kind regards,
Dimitri Smits

In our previous episode, Dimitri Smits said:

No, also the amounts needed from e.g. LCL to FCL and RTL packages. And those have high granularity (and the program->LCL might also be quite high)

I think this discussion should first define some terms wrt "building with packages" vs "safeloadpackage". Dynamic is a too vague term.

I was mostly thinking Hans-Peter meant doing loadpackage() under the hood. I think that is, if viable at all, an extreme specialty which shouldn't be considered at all in the first run.

In later mails I got the impression he didn't mean that, but normal static-dynamic linking. (importlib static, code dynamic in dll/bpl)

((safe)loadpackage?)

I would suspect that you then have "duplicate units"-error. Compiler cannot defend against it due to the "runtime" information vs "compile-time" information.

Correct. Since LCL might hold state, it violates the rule that the whole of the packages system may hold only one instance of a certain unit state (its variables, but also VMT tables, see wiki)

Just to show that packages bring their own set of problems and pitfalls. But definately not all is bad ;-)

This is all known, and described in the wiki afaik.

That's the first order defence yes, to allow multiple installs. But the packages system should be safeguarded against any kind of tampering, based on CRCs or other .ppu version items. Since with the openess and snapshot generation of FPC/Lazarus, even two builds from different periods with the same number will usually be incompatible.

(CRCs of the interface of the relevant ppus I think, as currently is done with ppu's too)

Thanks for the clarification :-)

DoDi

This is interesting. So there might be no need to include the complete .o file after all.

For inline: If I remember correctly inlined methods/functions/procedures are always available as non-inlined methods as well. It's up to the compiler to deside whether it inlines it or not.

This makes life easier.. :)

Cheers,
Willi