Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / fpc-devel / 2011-04

Building compiler, rtl, host, target... (cross compiling)

19 messages · 6 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

Skybuck Flying

Hello,

First of all I would like to write that I failed to compile the RTL of Free Pascal so that might be adding a little bit too the confusion. (Just simple i386/x86 tests to see if the sources build)

The compiler however seems to compile fine.

I am now a bit confused about the whole theory of things and I will try to explain my confusion:

The confusion pretty much starts with the compiler needing some RTL includes and units and especially some types like DWORD or PtrInt to be able to compile itself ?!?

This is a little bit weird to me: Why would the compiler need an RTL to compile itself ? (Perhaps these are accidental, or unnecessary depedancy ?)

I am now a bit confused again about what free pascal is...

As far as I can remember I once build a free pascal compiler which would compile from a certain "host" operating system towards another "target" operating system.

Is free pascal supposed to be able to do this all naturally by itself (as long as it can use external tools) ?

I just consulted my little tutorial on building a free pascal cross compiler which I once wrote and I notice the following things:

It mentions a "cross assembler" and a "cross linker" these where external GNU related tools.

From the looks of it I'm thinking right now that "Free Pascal" might

actually not be a cross compiler and it actually can only compile to it's own platform that the compiler executable was compiled for.

So in other words if this "hypothesis" above is correct then what was happening at the time might have been the following situation:

1. "Free Pascal" ("fake cross") compiler compiles the source from PPC386.EXE (i386 host) towards Linux/PowerPC (made up target) as "x86 assembly".

2. The cross-assembler.exe from GNU tools takes the x86 assembly produced by free pascal compiler and actually "cross-assembles" it for linux/powerpc into powerpc assembly.

3. The cross-linker does the rest.

So suppose that's how it was working at the time then Free Pascal is actually not really a "cross compiler".

However this does not mean it's useless for my project/experiments... I need Free Pascal to be a "cross compiler" which compiles to "abstract assembly" so I can assemble it further into "any assembler" a true cross compiler I would say ;)

Perhaps I am wrong and you guys would like to correct me... but for that I will help you by asking a question:

1. Would it be possible to build the following compiler:

FreePascalCompiler.exe (runs on windows) (compiles for virtual machine instructions/outputs virtual machine assembly).

Or is a "FreePascalCompiler.exe (x86)" only capable of producing "x86 assembly" like I suspect it might be/do...

Suppose I am correct and free pascal compiler can only compile towards the target that it itself was compiled for then I have a little problem:

My virtual machine isn't really ment to access files, or serve as an operating system or be a memory manager or have great capabilities... it's pretty much ment to execute some simple instructions and that's it...

So what I need is a "free pascal compiler" which was compiled for Windows environment/executable but which can still compile for a virtual machine.

I was under the impression that free pascal was a true cross compiler and therefore I was a bit confused about the RTL.... why does the free pascal compiler need an RTL ?

How do I add a new RTL for another target to the compiler ?!? Is it even necessary or can it compile without it with a few modifications (me investigating this ;))

It's all a bit fuzzy...

I think most of my questions where already answered in a previous posting/reply... people said:

"Nope... free pascal can only compile applications towards it's own platform/target that it was compiled for..."

If that's indeed correct then that's a bit unfortunate...

But none-the-less I seem some possibilities to make free pascal do what I want...

I could simply take the i386/x86 version of it with it's internal assemblers... and then simply convert those internal assemblers towards a sort of cross-assemblers... which would output "virtual machine assembly".

I'm looking for some clearification and confirmation on these thoughts of mine so that I understand correctly...

Also a little bit more explanation about the roll of RTL in relation to the compiler would be nice...

Let me ask a question in relation to that...

Let's suppose I want to compile free pascal compiler and/or RTL for a new target.. since free pascal can only compile towards it's own target... it cannot do this correct ?

So to actually "port" pascal towards a new platform would theoretically require other compilers written for the other platforms ?

So in a way this makes free pascal not "cross-platform-self-hosting" ?

So it's still a bit stuck in it's own world ???

Bye,
Skybuck.

Reply by email

Skybuck Flying
to Skybuck Flying

Hello,

Then again when I think about this some more it doesn't seem to make much sense... since free pascal does have RTL for other platforms.

And it also has "cpu's" for other platforms.

So perhaps what is needed is the following:

The RTL which is necessary for the other platform must first be compiled for the current/hosting platform/cpu.

The CPU which is necessary for the other platform must first be compiled for the current/hosting platform/cpu.

This would then produce a FreePascalCompiler.exe which would be able to run on Windows/i386 but which would be capable of compiling towards the RTL/CPU of the target platform. The only difference is that internally that either RTL/CPU was compiled for the host platform but that doesn't matter... that's just so the executable can run the code which will ultimately produce the correct assembly for the target platform...

The correct assembly would then contain the "translated/converted" RTL/CPU code which would be in the applications produced with this "cross compiler".

Thus I think free pascal compiler should be able to function as a "cross compiler" as well ?

So I think this conceptually involves the following steps:

1. Download free pascal sources codes and executables for current host operating system. (For example windows)

2. Specify what the target platform should be. (Tell the compiler this somehow via options/switches)

3. Then simply compile any application source code for the target.

However this conflicts with what other people have been telling me... and it also conflicts with the source code of the compiler itself which includes "defines" and "conditional compilation" for all the different platforms... I am not sure if all those platforms can actually be enabled all at the same time and be all compiled at the same time for a single compiler...

In theory this should be possible... but in practice it seems the sources where created in such a way that cpubase.pas is pretty much an alias for the folder of the target platform, which is specified via "search paths" this is probably done via the make files.

A solution to be able to compile to multiple "true" targets would be the following:

1. Rename all units in the "platform" subfolders so that they have unique names.

2. Rename all their classes and methods/routines so that they have unique names.

3. Use routine pointers/function pointers/method pointers inside the compiler itself which uses them to call a platform.

4. Hook-up the platform routines to these pointer variables.

(An alternative could be virtual functions/methods but I saw one post saying that this is unwanted in the compiler for some reason... why exactly wasn't explained... except that this would be "harder" ? perhaps harder to debug/or patch the compiler itself in case of problems ?)

None the less pointer variables don't seem that complex...

From documentation/manuals/tutorial/and/or postings I also saw some mention

about language specific features/platform specific features/or platform specific code which might be in the compiler/parser itself... I am not completely sure about this and how much of a problem it would be for this new idea... but I supposed some branches/if statements here and there would probably be sufficient to solve such "specializations towards certain platforms".

So I shall now ask the free pascal compiler "core/main" developers or "big bosses" ;) =D the following question:

1. Why is it not possible for a single free pascal compiler executable to support and compile towards multiple targets ? (One target selected out of multiple target options)

Would there be any big technical hurdles to turn the free pascal source code into such a beast ? What would be disadventages and potential problems now or in the future ?

Is it perhaps lazyness ? Not wanting to turn lot's of code into something that can do this ? Or you tired with the sources and don't want to change them towards something like this ? Is it fear for breaking things ?

I can imagine perhaps some technical drawbacks or perhaps some physical/emotional drawbacks.

If it's the latter then at least there is a possibility that this could be done be it by others with a little guidance from you.

I am not saying that this should happen immediatly... but I do want to know what could be possible in the future ;)

Bye,
Skybuck.

Reply by email

Sven Barth
to Skybuck Flying: First of all I would like to write that I failed to compile the RTL of Free Pascal so…

Wow, wow, wow... you are indeed a bit confused here.

Skybuck Flying wrote 18 lines
Hello, First of all I would like to write that I failed to compile the RTL of Free Pascal so that might be adding a little bit too the confusion. (Just simple i386/x86 tests to see if the sources build) The compiler however seems to compile fine. I am now a bit confused about the whole theory of things and I will try to explain my confusion: The confusion pretty much starts with the compiler needing some RTL includes and units and especially some types like DWORD or PtrInt to be able to compile itself ?!? This is a little bit weird to me: Why would the compiler need an RTL to compile itself ? (Perhaps these are accidental, or unnecessary depedancy ?)

RTL is an abbreviation for Runtime Type Library and it's necessary for EVERY Free Pascal program, because it abstracts the functions of the operating system, provides the system unit and types and other units. As the compiler is just a plain Free Pascal program itself (and when FPC compiles itself it just compiles a plain FPC program) it's absolutely naturaly that the compiler needs the RTL. I would have been more surprised if the compiler would not need the RTL...

Show quoted text 9 lines
I am now a bit confused again about what free pascal is... As far as I can remember I once build a free pascal compiler which would compile from a certain "host" operating system towards another "target" operating system. Is free pascal supposed to be able to do this all naturally by itself (as long as it can use external tools) ?

Yes. As free pascal is able to cross compile (see below).

Show quoted text 10 lines
I just consulted my little tutorial on building a free pascal cross compiler which I once wrote and I notice the following things: It mentions a "cross assembler" and a "cross linker" these where external GNU related tools. > From the looks of it I'm thinking right now that "Free Pascal" might actually not be a cross compiler and it actually can only compile to it's own platform that the compiler executable was compiled for.

This assumption is wrong.
The following assumption is right: the compile can only compile for the one target it was configured for. The target the compiler is compiled to and the target the compiler compiles for don't need to be the same. E.g. I could compile a PowerPC cross compiler by compiling an ARM based compiler that is cross compiled by a i386 cross compiler for ARM.

Show quoted text 13 lines
So in other words if this "hypothesis" above is correct then what was happening at the time might have been the following situation: 1. "Free Pascal" ("fake cross") compiler compiles the source from PPC386.EXE (i386 host) towards Linux/PowerPC (made up target) as "x86 assembly". 2. The cross-assembler.exe from GNU tools takes the x86 assembly produced by free pascal compiler and actually "cross-assembles" it for linux/powerpc into powerpc assembly. 3. The cross-linker does the rest.

Let's assume you want a native ARM compiler and start with an i386 compiler then the steps are like this:

1. compile a i386=>ARM cross compiler by compiling pp.pas using the correct defines
2. compile the RTL for ARM using the new cross compiler
3. compile a ARM=>ARM compiler by compiling pp.pas using the correct defines and by using the i386=>ARM compiler and the ARM RTL => tada! you have a native ARM compiler (and a i386 => ARM compiler as well)

[of course this assumes that you have binutils and eventually needed libraries to do the cross compile]

So suppose that's how it was working at the time then Free Pascal is actually not really a "cross compiler".

You are supposing wrong ;)

Show quoted text 17 lines
However this does not mean it's useless for my project/experiments... I need Free Pascal to be a "cross compiler" which compiles to "abstract assembly" so I can assemble it further into "any assembler" a true cross compiler I would say ;) Perhaps I am wrong and you guys would like to correct me... but for that I will help you by asking a question: 1. Would it be possible to build the following compiler: FreePascalCompiler.exe (runs on windows) (compiles for virtual machine instructions/outputs virtual machine assembly).

Yes this would be possible. Does that virtual machine you want to target FPC for provide its own virtual instruction set or does it use the instruction set of one of the already supported platforms?

If the latter then you don't need to do much.

If the first then you either need to find cross binutils that allow you to assemble code to that virtual instruction set or you must write an internal assembler for FPC. In both cases though you must add a new cpu (parallel to ARM, i386, PowerPC, etc.) that contains the needed methods for the compiler to call (that generate the assembler code for the assembler) and handles calling conventions, variable locations, etc.

Show quoted text 23 lines
Or is a "FreePascalCompiler.exe (x86)" only capable of producing "x86 assembly" like I suspect it might be/do... Suppose I am correct and free pascal compiler can only compile towards the target that it itself was compiled for then I have a little problem: My virtual machine isn't really ment to access files, or serve as an operating system or be a memory manager or have great capabilities... it's pretty much ment to execute some simple instructions and that's it... So what I need is a "free pascal compiler" which was compiled for Windows environment/executable but which can still compile for a virtual machine. I was under the impression that free pascal was a true cross compiler and therefore I was a bit confused about the RTL.... why does the free pascal compiler need an RTL ?

As said above: every FPC program needs a RTL.

How do I add a new RTL for another target to the compiler ?!? Is it even necessary or can it compile without it with a few modifications (me investigating this ;))

You could take a look at the "embedded" target. It's for writing code directly on CPUs without an operating system in between. So once you've added your new virtual platform you can add an embedded target and then you should be (nearly) ready to go...

Show quoted text 9 lines
It's all a bit fuzzy... I think most of my questions where already answered in a previous posting/reply... people said: "Nope... free pascal can only compile applications towards it's own platform/target that it was compiled for..."

You got the sense of "compiled for" a bit wrong. I didn't mean to say that a compiler compiled for i386 can only generate i386 code. I meant that a compiler that has the "i386 code generation" enabled can only generate i386 code no matter for which processor to run on it was compiled.

Show quoted text 21 lines
If that's indeed correct then that's a bit unfortunate... But none-the-less I seem some possibilities to make free pascal do what I want... I could simply take the i386/x86 version of it with it's internal assemblers... and then simply convert those internal assemblers towards a sort of cross-assemblers... which would output "virtual machine assembly". I'm looking for some clearification and confirmation on these thoughts of mine so that I understand correctly... Also a little bit more explanation about the roll of RTL in relation to the compiler would be nice...

I hope my explanation helps you.

Show quoted text 14 lines
Let me ask a question in relation to that... Let's suppose I want to compile free pascal compiler and/or RTL for a new target.. since free pascal can only compile towards it's own target... it cannot do this correct ? So to actually "port" pascal towards a new platform would theoretically require other compilers written for the other platforms ? So in a way this makes free pascal not "cross-platform-self-hosting" ? So it's still a bit stuck in it's own world ???

Perhaps it would help if you'd answer directly to other people's mails and add your specific questions inline below a statement that you'd like to comment on like I'm doing it at this very moment...

Regards,
Sven

Reply by email

Skybuck Flying
to Sven Barth: Wow, wow, wow... you are indeed a bit confused here. RTL is an abbreviation for Runtime…

Ok,

I thought about it a bit and I think I now logically understand what's going on and what needs to happen when one wants to build a new compiler, possibly for cross compiling and such in relation to the RTL.

First of all "free pascal compiler" is not a "multi-target cross compiler" but it is a "dual-target cross compiler" (dual meaning host and target can be different ;))

So starting with the basics: Suppose somebody would get the free pascal source codes only and now executables then that person would first need to compile:

1. The compiler
and
2. The run time library

For let's say windows.

The compiler is a little bit like an application, applications need a system unit, the runtime library provides that system unit.

However the "external/user compiler" might already have a system unit... so for that purposes the "external/user compiler run time library/system unit might be used". The free pascal compiler dependancies on it's own RTL could be fixed/modified/temporarely changed to use the external compiler's system unit to get the first compiler build working.

So step 1 would then produce:

A free pascal compiler executable which runs on windows.

The target to keep things easy was set to windows to first produce a "native compiler" (which runs on windows and compiles to windows)

Now the mission/job/idea is to produce a cross-compiler which can cross-compile to a new architecture/processor/platform let's called it NewCPU and NewRTL in short: NewTarget.

Step 2 is a repeat of step 1. However it would be helpfull if the RTL itself was also compileable to the new target... but let's see how far we can go without it.

Step 2 specifies somehow to the build enviroment that the compiler sources are to be re-compiled but this time the compiler which is build from it should compile to the NewTarget... for that the new compiler needs "new architecture support"... So that's where the new instructions and new assembler come in... and perhaps new linker... but let's start with new assembler and such...

Since the executable to be produced must still run on windows it can and probably must still use the windows RTL to be able to produce the executable.

If the same RTL was used to specify the target build... then it would be a bit strange because then NewCPU would try to use a Windows RTL...

So this would then produce a "half-cross compiler" which would be kinda funny... it would probably be "defunct"/"not working" perhaps such executables produce by such a half compiler could be patched etc... but that's probably a bit stupid to do.

What is needed is a "full-cross compiler" which also uses the NewRTL for the NewPlatform.

Apperently specifieing the NewRTL can be done by compiler command line options or configuration files.

Thus there are now probably two RTL's involved:

One RTL for the host, One RTL for the target.

How the compiler makes sense of this I am not yet sure... Perhaps it's a question of linking in the NewRTL when a NewPlatform application is being build by the new compiler.

So perhaps it's the "internal linker" which needs the NewRTL to be present to do it's work ?!? How else could it work ??

However perhaps the new compiler also needs to do type checking against the NewRTL for NewPlatform applications...

Thus somehow the new compiler must be using the new RTL.

The new RTL didn't exist yet... so it needs to be compiled.

Apperently this is where a "half-cross compiler comes in ?"

The new RTL cannot depend on "other platform RTL's" ?

So if I am correct in my assumption then the any RTL for that matter must be "self-standing" and not rely on anything else except it's own platform that it was written for.

It must also not depend on RTL functionality itself ?

How would otherwise a compiler be able to compile it ? (without creating some kind of platform depedency)

So I guess this is where a "half-cross-compiler" comes in... The half-cross compiler is capable of compiling a NewRTL and producing some kind of assembly text which can then be used to compile the NewRTL to some kind of binary (a pascal unit binary?)

This is perhaps where the linker also comes into play or maybe not...

Anyway once the NewRTL is compiled into pascal unit binaries it can be used/include in a new compiler build.

So the compiler sources are again recompiled... but this time the compiler sources are "told" or "modified via includes/search paths" to use the NewRTL units so that it itself is compiled towards the new platform and uses the new rtl for that platform.

This will then produce a "full cross compiler" which is capable of compiling from running on host towards a target.

Once that's done... it's possible to recompile the compiler towards a native compiler... produce native executable and native target (compilation mode)... so that last step is the easy step.

So if I am correct "porting free pascal compiler and rtl towards a new platform" requires the following steps:

1. Step 1 (preparing for first native free pascal compiler)

Install a "external pascal compiler" so that free pascal compiler sources can be compiled.
(ignore free pascal's own rtl if necessary), make changes to the free pascal compiler source code if necessary to use the external pascal compiler's RTL.

2. Step 2 (building free pascal compiler)

Build the free pascal compiler with the "external pascal compiler". This produces a free pascal compiler executable for whatever platform step1 was running on.

(These two steps can be skipped if "free pascal compiler executable" already available (in this case external compiler=free pascal compiler executable))

3. Step 3 (building half-cross compiler)

Add new target's cpu instructions/assembler to free pascal compiler sources so that the free pascal compiler can compile source code into free pascal unit binaries.

Re-build the compiler. (This creates what I would call a "half-cross compiler")

A half-cross compiler is a compiler which cannot yet produce "target-executables" but it can produce "target-free pascal binary units".

(Seems to make logical sense: target-executables not yet possible because target-rtl missing)

4. Step 4 (building full-cross compiler)

Add new target's rtl to free pascal compiler sources so that the free pascal compiler can compile source into into target-(executable or package/dll)-binaries.

Re-build the compiler and tell it to use the new-target's-rtl as it's uses clausule. (This can be via "target-free pascal binary units in combination with the interface" or simply new-target's RTL full source code, since the compiler should now be able to handle/compile it.

This will produce a "full-cross compiler". The full cross compiler knows how to compile towards target-cpu and it also knows how to use the target-rtl.

Optional repeating steps (fixing up):

These steps can then be repeated to undo the changes that where made at step 1, a native RTL could first be made which works with the native/external compiler and perhaps with free pascal compiler itself as well.

Perhaps these steps are even manditory to be able to easily switch targets... all code should in principe follow the same RTL layout/interfaces ;)
(But this might be conflicting with external compilers which already have their own RTL, so that's why step1 might/would be necessary)

It might also be necessary to change all other target code's rtl's to use a new-non-conflicting api/layout for external compilers.

5. Step 5 (building any native compiler)

Use the cross compiler to rebuild the free pascal compiler source codes.

Specify the target's cpu and target's rtl. (Which in this case should be the cpu and rtl for which a native compiler should be build).

The reason for the native compiler is so that user's of this new target can now be used to develop on itself... (assuming the target has such capabilities)

(Some targets might not have such capabilities like embedded system, perhaps console, gameboys, phones, refrigderators/whatever so form them step 4 is already enough ;))

Missing from this list is ofcourse:

debuggers/cross debuggers which would still be needed for serious platform development but that is beyond the scope of this posting ;)

And unfortunately me very little to almost no experience with writing debuggers myself... perhaps a nice thing for me to explore in the future ;)

Bye,
Skybuck ;)

Reply by email

Skybuck Flying
Show quoted text 10 lines
4. Step 4 (building full-cross compiler) Add new target's rtl to free pascal compiler sources so that the free pascal compiler can compile source into into target-(executable or package/dll)-binaries. Re-build the compiler and tell it to use the new-target's-rtl as it's uses clausule. (This can be via "target-free pascal binary units in combination with the interface" or simply new-target's RTL full source code, since the compiler should now be able to handle/compile it.

I am not so sure if the full source code of the new RTL could be used... probably not...

My original hunch was that it indeed needs to be in binary (pascal compiled units or assembly units)... and needs to be passed towards the linker so that the linker can do "the great switchero-trick".

The compiler used it's own rtl to compile, but the linker will say "nope, not gonna use your rtl... instead I am gonna do a switchero trick and use the new rtl".

Which means the "hook-in" must be the same... so the swap goes easy...

This seems to make most sense to me... (!?)

So my guess is: It's indeed necessary to have a new RTL in binary form... so it can be swapped-in with the old one during the building/linking phase of a new compiler... (?!)

Or perhaps it doesn't matter and the compiler can simply compile with the new rtl full sources since it can compile those on the fly... when the reach the linker they already in target-binary form...

But how to know for sure and how to make the compiler not use it's own RTL ? This is probably my biggest/main questions... and biggest doubt how this works... hmm...

So this is my biggest question and hunch: At some point during this whole step/phase the RTL needs to be switched/swapped with the new RTL... however the own/old RTL is still being used by the active process (the compiler that's running and compiling the new compiler) so what part of the chain is responsible for the "switchero trick" lol ?!? Is it indeed the linker ???

Bye,
Skybuck.

Reply by email

Hans-Peter Diettrich
to Skybuck Flying: I thought about it a bit and I think I now logically understand what's going on and what…
First of all "free pascal compiler" is not a "multi-target cross compiler" but it is a "dual-target cross compiler" (dual meaning host and target can be different ;))

FPC is a single target cross compiler. "cross" indicates that the target can be different from the host system.

Show quoted text 7 lines
So starting with the basics: Suppose somebody would get the free pascal source codes only and now executables then that person would first need to compile: 1. The compiler and 2. The run time library

No. First the RTL is compiled, then the compiler is built with that RTL.

The compiler is a little bit like an application, applications need a system unit, the runtime library provides that system unit.

That's why the RTL is compiled before. The compiler is built *exactly* like any other application, in this step.

In the next step the new compiler is used to compile itself. This way new features can be included, which the old (bootstrap) compiler could not compile (e.g. class helpers...).

This step is repeated until the last two executables are binary identical. This procedure allows for further extensions, that even could not be compiled by an earlier stage compiler, without the need to specify an exact number of required recompilations.

Show quoted text 6 lines
So step 1 would then produce: A free pascal compiler executable which runs on windows. The target to keep things easy was set to windows to first produce a "native compiler" (which runs on windows and compiles to windows)

... and which can compile the new language version, including all added features.

Show quoted text 20 lines
Now the mission/job/idea is to produce a cross-compiler which can cross-compile to a new architecture/processor/platform let's called it NewCPU and NewRTL in short: NewTarget. Step 2 is a repeat of step 1. However it would be helpfull if the RTL itself was also compileable to the new target... but let's see how far we can go without it. Step 2 specifies somehow to the build enviroment that the compiler sources are to be re-compiled but this time the compiler which is build from it should compile to the NewTarget... for that the new compiler needs "new architecture support"... So that's where the new instructions and new assembler come in... and perhaps new linker... but let's start with new assembler and such... Since the executable to be produced must still run on windows it can and probably must still use the windows RTL to be able to produce the executable.

Right. Every application must use the RTL of its host platform.

What is needed is a "full-cross compiler" which also uses the NewRTL for the NewPlatform.

No, why should the compiler need to use a different RTL, for its *own* operation? Every compiler can compile the RTL for its target, as well as other applications.

Apperently specifieing the NewRTL can be done by compiler command line options or configuration files.

As is, the specification of the target (hard coded in the compiler) is sufficient to locate target-specific parts of the source code, in target-specific directories. I really like that idea :-)

Show quoted text 8 lines
Thus there are now probably two RTL's involved: One RTL for the host, One RTL for the target. How the compiler makes sense of this I am not yet sure... Perhaps it's a question of linking in the NewRTL when a NewPlatform application is being build by the new compiler.

Every target e.g. has its own data types, like 32 or 64 bit pointers. That's why the compiler has to use the target-specific version of the RTL, in cross-compiling any code.

So perhaps it's the "internal linker" which needs the NewRTL to be present to do it's work ?!? How else could it work ??

Every linker needs binaries that have been compiled for the target, and includes them into the final executable (for the same target).

However perhaps the new compiler also needs to do type checking against the NewRTL for NewPlatform applications... Thus somehow the new compiler must be using the new RTL.

That's fully automatic. The compiler compiles or loads precompiled units recursively, by evaluating the Uses clauses, so that it traverses the target specific RTL and ends up in the System unit.

All parts of the target, that are required for a compilation, are already built into the compiler. That includes code generation for the target CPU, and linking executables for the target OS (what may actually occur in an external assembler and linker).

Show quoted text 23 lines
So if I am correct "porting free pascal compiler and rtl towards a new platform" requires the following steps: 1. Step 1 (preparing for first native free pascal compiler) Install a "external pascal compiler" so that free pascal compiler sources can be compiled. (ignore free pascal's own rtl if necessary), make changes to the free pascal compiler source code if necessary to use the external pascal compiler's RTL. 2. Step 2 (building free pascal compiler) Build the free pascal compiler with the "external pascal compiler". This produces a free pascal compiler executable for whatever platform step1 was running on. (These two steps can be skipped if "free pascal compiler executable" already available (in this case external compiler=free pascal compiler executable)) 3. Step 3 (building half-cross compiler)

This builds any compiler, for the same or a different platform. Nothing "half" here, the second half only adds the RTL for the target OS.

Show quoted text 6 lines
Add new target's cpu instructions/assembler to free pascal compiler sources so that the free pascal compiler can compile source code into free pascal unit binaries. Re-build the compiler. (This creates what I would call a "half-cross compiler")
Show quoted text 5 lines
4. Step 4 (building full-cross compiler) Add new target's rtl to free pascal compiler sources so that the free pascal compiler can compile source into into target-(executable or package/dll)-binaries.

That's all. The new RTL can be compiled now, but it would be compiled as well with the first project.

Show quoted text 6 lines
5. Step 5 (building any native compiler) Use the cross compiler to rebuild the free pascal compiler source codes. Specify the target's cpu and target's rtl. (Which in this case should be the cpu and rtl for which a native compiler should be build).

This is not different from step 1, only with a different target machine and/or OS.

The reason for the native compiler is so that user's of this new target can now be used to develop on itself... (assuming the target has such capabilities)

The same reason for cross-building any application for the target system.

Missing from this list is ofcourse: debuggers/cross debuggers which would still be needed for serious platform development but that is beyond the scope of this posting ;)

The debugger is part of the binutils, as are assemblers and linkers. While it may make sense to add assembler and linker functionality to the compiler itself, it doesn't make sense to add an debugger to the compiler. But a debugger-interface can be added to a development system (Lazarus), for integrated debugging.

DoDi

Reply by email

Skybuck Flying
to Skybuck Flying: I thought about it a bit and I think I now logically understand what's going on and what…

----- Original Message -----

Earlier message 255 lines
From: "Skybuck Flying" To: "FPC developers' list" Sent: Thursday, 7 April, 2011 03:06 AM Subject: Re: [fpc-devel] Building compiler, rtl, host,target... (cross compiling) > Ok, > > I thought about it a bit and I think I now logically understand what's > going on and what needs to happen when one wants to build a new compiler, > possibly for cross compiling and such in relation to the RTL. > > First of all "free pascal compiler" is not a "multi-target cross compiler" > but it is a "dual-target cross compiler" (dual meaning host and target can > be different ;)) > > So starting with the basics: Suppose somebody would get the free pascal > source codes only and now executables then that person would first need to Little typo here... gonna correct it: source codes only and no executables then that person would first need to > compile: > > 1. The compiler > and > 2. The run time library > > For let's say windows. > > > The compiler is a little bit like an application, applications need a > system unit, the runtime library provides that system unit. > > However the "external/user compiler" might already have a system unit... > so for that purposes the "external/user compiler run time library/system > unit might be used". The free pascal compiler dependancies on it's own RTL > could be fixed/modified/temporarely changed to use the external compiler's > system unit to get the first compiler build working. > > > So step 1 would then produce: > > A free pascal compiler executable which runs on windows. > > The target to keep things easy was set to windows to first produce a > "native compiler" (which runs on windows and compiles to windows) > > > Now the mission/job/idea is to produce a cross-compiler which can > cross-compile to a new architecture/processor/platform let's called it > NewCPU and NewRTL in short: NewTarget. > > > Step 2 is a repeat of step 1. However it would be helpfull if the RTL > itself was also compileable to the new target... but let's see how far we > can go without it. > > > Step 2 specifies somehow to the build enviroment that the compiler sources > are to be re-compiled but this time the compiler which is build from it > should compile to the NewTarget... for that the new compiler needs "new > architecture support"... So that's where the new instructions and new > assembler come in... and perhaps new linker... but let's start with new > assembler and such... > > Since the executable to be produced must still run on windows it can and > probably must still use the windows RTL to be able to produce the > executable. > > If the same RTL was used to specify the target build... then it would be a > bit strange because then NewCPU would try to use a Windows RTL... > > So this would then produce a "half-cross compiler" which would be kinda > funny... it would probably be "defunct"/"not working" perhaps such > executables produce by such a half compiler could be patched etc... but > that's probably a bit stupid to do. > > > What is needed is a "full-cross compiler" which also uses the NewRTL for > the NewPlatform. > > Apperently specifieing the NewRTL can be done by compiler command line > options or configuration files. > > Thus there are now probably two RTL's involved: > > One RTL for the host, One RTL for the target. > > > How the compiler makes sense of this I am not yet sure... Perhaps it's a > question of linking in the NewRTL when a NewPlatform application is being > build by the new compiler. > > So perhaps it's the "internal linker" which needs the NewRTL to be present > to do it's work ?!? How else could it work ?? > > > However perhaps the new compiler also needs to do type checking against > the NewRTL for NewPlatform applications... > > Thus somehow the new compiler must be using the new RTL. > > The new RTL didn't exist yet... so it needs to be compiled. > > Apperently this is where a "half-cross compiler comes in ?" > > The new RTL cannot depend on "other platform RTL's" ? > > So if I am correct in my assumption then the any RTL for that matter must > be "self-standing" and not rely on anything else except it's own platform > that it was written for. > > It must also not depend on RTL functionality itself ? > > How would otherwise a compiler be able to compile it ? (without creating > some kind of platform depedency) > > > > So I guess this is where a "half-cross-compiler" comes in... The > half-cross compiler is capable of compiling a NewRTL and producing some > kind of assembly text which can then be used to compile the NewRTL to some > kind of binary (a pascal unit binary?) > > This is perhaps where the linker also comes into play or maybe not... > > > Anyway once the NewRTL is compiled into pascal unit binaries it can be > used/include in a new compiler build. > > > So the compiler sources are again recompiled... but this time the compiler > sources are "told" or "modified via includes/search paths" to use the > NewRTL units so that it itself is compiled towards the new platform and > uses the new rtl for that platform. > > This will then produce a "full cross compiler" which is capable of > compiling from running on host towards a target. > > > Once that's done... it's possible to recompile the compiler towards a > native compiler... produce native executable and native target > (compilation mode)... so that last step is the easy step. > > > > So if I am correct "porting free pascal compiler and rtl towards a new > platform" requires the following steps: > > > 1. Step 1 (preparing for first native free pascal compiler) > > Install a "external pascal compiler" so that free pascal compiler sources > can be compiled. > (ignore free pascal's own rtl if necessary), make changes to the free > pascal compiler source code if necessary to use the external pascal > compiler's RTL. > > 2. Step 2 (building free pascal compiler) > > Build the free pascal compiler with the "external pascal compiler". This > produces a free pascal compiler executable for whatever platform step1 was > running on. > > (These two steps can be skipped if "free pascal compiler executable" > already available (in this case external compiler=free pascal compiler > executable)) > > 3. Step 3 (building half-cross compiler) > > Add new target's cpu instructions/assembler to free pascal compiler > sources so that the free pascal compiler can compile source code into free > pascal unit binaries. > > Re-build the compiler. (This creates what I would call a "half-cross > compiler") > > A half-cross compiler is a compiler which cannot yet produce > "target-executables" but it can produce "target-free pascal binary units". > > (Seems to make logical sense: target-executables not yet possible because > target-rtl missing) > > > 4. Step 4 (building full-cross compiler) > > Add new target's rtl to free pascal compiler sources so that the free > pascal compiler can compile source into into target-(executable or > package/dll)-binaries. > > Re-build the compiler and tell it to use the new-target's-rtl as it's uses > clausule. (This can be via "target-free pascal binary units in combination > with the interface" or simply new-target's RTL full source code, since the > compiler should now be able to handle/compile it. > > This will produce a "full-cross compiler". The full cross compiler knows > how to compile towards target-cpu and it also knows how to use the > target-rtl. > > > Optional repeating steps (fixing up): > > These steps can then be repeated to undo the changes that where made at > step 1, a native RTL could first be made which works with the > native/external compiler and perhaps with free pascal compiler itself as > well. > > Perhaps these steps are even manditory to be able to easily switch > targets... all code should in principe follow the same RTL > layout/interfaces ;) > (But this might be conflicting with external compilers which already have > their own RTL, so that's why step1 might/would be necessary) > > It might also be necessary to change all other target code's rtl's to use > a new-non-conflicting api/layout for external compilers. > > > 5. Step 5 (building any native compiler) > > Use the cross compiler to rebuild the free pascal compiler source codes. > > Specify the target's cpu and target's rtl. (Which in this case should be > the cpu and rtl for which a native compiler should be build). > > > The reason for the native compiler is so that user's of this new target > can now be used to develop on itself... (assuming the target has such > capabilities) > > (Some targets might not have such capabilities like embedded system, > perhaps console, gameboys, phones, refrigderators/whatever so form them > step 4 is already enough ;)) > > > > > Missing from this list is ofcourse: > > debuggers/cross debuggers which would still be needed for serious platform > development but that is beyond the scope of this posting ;) > > And unfortunately me very little to almost no experience with writing > debuggers myself... perhaps a nice thing for me to explore in the future > ;) > > > Bye, > Skybuck ;) > _______________________________________________ > fpc-devel maillist - > http://lists.freepascal.org/mailman/listinfo/fpc-devel >

Reply by email

Sven Barth
to Skybuck Flying: I thought about it a bit and I think I now logically understand what's going on and what…
Skybuck Flying wrote 10 lines
Ok, I thought about it a bit and I think I now logically understand what's going on and what needs to happen when one wants to build a new compiler, possibly for cross compiling and such in relation to the RTL. First of all "free pascal compiler" is not a "multi-target cross compiler" but it is a "dual-target cross compiler" (dual meaning host and target can be different ;))

The definition of "cross compiler" is that a compiler that runs on CPU A with operating system B can generate code for operating system C that runs on CPU D. Not more, not less.

Show quoted text 7 lines
So starting with the basics: Suppose somebody would get the free pascal source codes only and now executables then that person would first need to compile: 1. The compiler and 2. The run time library

It must be done the other way round as the compiler needs the RTL to run itself.

Show quoted text 13 lines
For let's say windows. The compiler is a little bit like an application, applications need a system unit, the runtime library provides that system unit. However the "external/user compiler" might already have a system unit... so for that purposes the "external/user compiler run time library/system unit might be used". The free pascal compiler dependancies on it's own RTL could be fixed/modified/temporarely changed to use the external compiler's system unit to get the first compiler build working.

There is no need to fix up anything. The process is the following:

You have a starting compiler for let's say i386 and you have source code of FPC and its RTL any you want to have a new shiny i386 Windows compiler.
1. Compile the RTL using the starting compiler (after all the RTL is just Pascal source that pulls in operating system specific code)
2. Compile the compiler together with the newly compiled RTL
3. Use that new compiler to compile the RTL again (so that new features are used)
4. Compile the compiler again using the newly compiled RTL.

Now you can Win32 i386 applications using your new compiler (and the RTL)

Show quoted text 7 lines
One RTL for the host, One RTL for the target. How the compiler makes sense of this I am not yet sure... Perhaps it's a question of linking in the NewRTL when a NewPlatform application is being build by the new compiler.

The RTL for the host is used by the compiler as it was compiled with that RTL (as every Free Pascal program is) and the RTL for the target are "just" (compiled) source files that are used(!) by the compiler to build the final executable (like it's done for every Free Pascal program).

So if I am correct in my assumption then the any RTL for that matter must be "self-standing" and not rely on anything else except it's own platform that it was written for.

This assumption is indeed correct. The base RTL unit (system.pp) as no dependency whatsoever and all other units inside the RTL (sysutils, classes, etc) basically only depend on the system unit.

Show quoted text 5 lines
So I guess this is where a "half-cross-compiler" comes in... The half-cross compiler is capable of compiling a NewRTL and producing some kind of assembly text which can then be used to compile the NewRTL to some kind of binary (a pascal unit binary?)

Every FPC compiler is able to compile the RTL. Just do a "make cycle" inside the compiler directory and you'll see that the RTL is compiled mutliple times.

Show quoted text 12 lines
So if I am correct "porting free pascal compiler and rtl towards a new platform" requires the following steps: 1. Step 1 (preparing for first native free pascal compiler) Install a "external pascal compiler" so that free pascal compiler sources can be compiled. (ignore free pascal's own rtl if necessary), make changes to the free pascal compiler source code if necessary to use the external pascal compiler's RTL.

The "external pascal compiler" MUST be a Free Pascal compiler as it needs to compile the RTL for the compiler first. More specifically that procedure is only supported by the compiler of the last release (currently 2.4.2 although 2.4.0 works as well, but not 2.2.4 or older). It also used to be possible to compile FPC and its RTL with Turbo Pascal (the first FPC was produced this way) or using Delphi. But that time is long gone.

Regards,
Sven

Reply by email

Skybuck Flying
to Hans-Peter Diettrich: FPC is a single target cross compiler. "cross" indicates that the target can be different…

I have to go do other things now so I will keep this short, maybe I'll get back to this posting later on...

Show quoted text 5 lines
> What is needed is a "full-cross compiler" which also uses the NewRTL for > the NewPlatform. No, why should the compiler need to use a different RTL, for its *own* operation?

It already has an RTL embedded inside it which it is currently using to run itself.

(Side question: can an RTL actually be a runtime library instead of being compiled into applications ?)

However for the application which it is trying to compile needs to run on a different platform and thus
the application needs to either be linked to that platform RTL or it needs to have that platform RTL embedded into it.

And that's what free pascal more or less seems to do... it embeds "a target platform RTL" into the application that it tries to compile.

Every compiler can compile the RTL for its target, as well as other applications.

No the compiler cannot compile the RTL for it's target because the compiler does not have
the assembly yet to output the RTL ?

So at the very minimum the compiler needs to produce assembler output and for that it needs
some kind of internal assembler-outputter ;)

An alternative could be a plug-in system where the necessary assembler-outputter is plugged into.

But currently free pascal works with "embedded assembler-outputters" thus the compiler needs to be re-compiled
before it can use these new "assembler outputters".

(Ok, you had no further comments, so this can probably be kept short ;))

Bye,
Skybuck.

Reply by email

Skybuck Flying
to Hans-Peter Diettrich: No swapping required. The host-RTL is linked *into* the compiler itself, the target-RTL…

----- Original Message -----

Earlier message 26 lines
From: "Hans-Peter Diettrich" To: "FPC developers' list" Sent: Thursday, 7 April, 2011 06:21 AM Subject: Re: [fpc-devel] Building compiler, rtl, host,target... (cross compiling) > Skybuck Flying schrieb: > >> So this is my biggest question and hunch: At some point during this whole >> step/phase the RTL needs to be switched/swapped with the new RTL... > > No swapping required. The host-RTL is linked *into* the compiler itself, > the target-RTL is used (read, compiled) *by* the compiler, at runtime. Can you or anybody else give an example of the necessary command line parameters and/or a configuration file so I can try this out to see how it works ? (Please try to keep it as minimal as possible ;) :)) (For the example host could be windows/i386/x86 and target could be something ?) Bye, Skybuck.

Reply by email

Michael Schnell
to Skybuck Flying: First of all I would like to write that I failed to compile the RTL of Free Pascal so…
Skybuck Flying wrote
First of all I would like to write that I failed to compile the RTL of Free Pascal so that might be adding a little bit too the confusion.

AFAIK, not all versions of the compiler are tested to compile the compiler and the RTL. I have a separate (old) FPC installation to be able to always compile the latest svn versions.

-Michael

Reply by email

Michael Schnell
to Skybuck Flying: I thought about it a bit and I think I now logically understand what's going on and what…
Skybuck Flying wrote 8 lines
Missing from this list is ofcourse: debuggers/cross debuggers which would still be needed for serious platform development but that is beyond the scope of this posting ;) And unfortunately me very little to almost no experience with writing debuggers myself... perhaps a nice thing for me to explore in the future ;)

Do you have a gdb for the new arch ? Same should be usable to debug FPC generated programs.

AFAIK, someone has started a project doing a debugger in Pascal code, but I have no idea if this already works.

Also one the native compiler (and command-line debugger) works, it should be possible to compile Lazarus for that arch. But why does it make sense to do the native development tools after all ? IMHO, usually, less effort and more desirable is to do a decent cross-compiling SDK: cross compiler, remote cross-debugger (needs only a "stub" on the target) and running Lazarus on a Windows or Linux PC.

BTW, regarding the RTL: what OS-API does the arch provide ? some "open standard" like Linux Mono or Java, something more cryptic like the Windows or .NET API ore even something completely propriety ?

-Michael

Reply by email

Michael Schnell
to Skybuck Flying: Then again when I think about this some more it doesn't seem to make much sense... since…
Skybuck Flying wrote
1. Why is it not possible for a single free pascal compiler executable to support and compile towards multiple targets ? (One target selected out of multiple target options)

It does not make much sense to request this. It's no hassel at all to use different compiler executables and RTL files for different targets. And a multi-Target compiler would be slower, as at very may lines case-switches between the different target would need to be handled, and the much bigger compiler-executable would be less efficient regarding the CPU's cache use.

-Michael

Reply by email

Skybuck Flying
to Hans-Peter Diettrich: FPC is a single target cross compiler. "cross" indicates that the target can be different…

----- Original Message -----

Earlier message 87 lines
From: "Hans-Peter Diettrich" To: "FPC developers' list" Sent: Thursday, 7 April, 2011 06:10 AM Subject: Re: [fpc-devel] Building compiler, rtl, host,target... (cross compiling) > Skybuck Flying schrieb: > >> First of all "free pascal compiler" is not a "multi-target cross >> compiler" but it is a "dual-target cross compiler" (dual meaning host and >> target can be different ;)) > > FPC is a single target cross compiler. "cross" indicates that the target > can be different from the host system. Perhaps you look at it from the binary level, but that's not fair, you should look at it from the source code level. Single refers to target. A zero-target cross compiler would be a cross compiler under development it has the framework to become a cross compiler but it does not yet have any supported targets. A single-target cross compiler is probably still a cross compiler under development but already has support for a single target. A dual-target cross compiler has support for two targets. A 6-target cross compiler has support for six targets. I would define a target as follows: Target = harware (cpu) + operating system (assembler + linker + rtl (so that application can hook into basic OS functionality)) To be able to cross compile from one target to another target actually requires two targets. It requires: 1. A host/local target 2. A guest/remote target (The source code of the compiler is compiled towards the host target, this requires some sort of host rtl.) (However the source code of the compiler also requires the source code of the guest target/rtl.) So in a way it needs support for both targets otherwise it will somehow not fully compile or it can't cross compile. And thus when I say a "dual target cross compiler" this would mean it has support in it's source code for two targets. The remaining question is: what is free pascal ? Free pascal I would describe as: a x-way self-hosting retarded (/brain-dead) (non-portable) cross compiler". (Perhaps I should even add internal assemblers etc but let's not for now). x should be replaced by the number of target-sub-folders. self-hosting refers to the fact that it can compile itself once it's build. retarded (/brain dead) refers to it's inability to switch targets during runtime. (only one guest-target seems to be selected during the compiler-self-build process). (non-portable) refers to it's inability of being compiled by other compilers. (For example Delphi can't compile the current state of the free pascal compiler source code) cross-compile refers to it's ability to cross compile to other targets. I hope I clearified that a little bit for you ! ;) =D Bye, Skybuck.

Reply by email

Sven Barth
to Skybuck Flying
Skybuck Flying wrote 21 lines
Perhaps you look at it from the binary level, but that's not fair, you should look at it from the source code level. Single refers to target. A zero-target cross compiler would be a cross compiler under development it has the framework to become a cross compiler but it does not yet have any supported targets. A single-target cross compiler is probably still a cross compiler under development but already has support for a single target. A dual-target cross compiler has support for two targets. A 6-target cross compiler has support for six targets. I would define a target as follows: Target = harware (cpu) + operating system (assembler + linker + rtl (so that application can hook into basic OS functionality))

By that definition FPC is a "multi target compiler", because every FPC compiler for CPU A can compile code for all operating systems that are supported inside the compiler. E.g. the i386 compiler can compile code for Win32, Linux, FreeBSD, DOS, OS/2, Mac OS X, NativeNT, ..., while the ARM compiler can compile for WinCE, Linux, Mac OS X, ... .

Regards,
Sven

Reply by email

Sven Barth
to Skybuck Flying: I have to go do other things now so I will keep this short, maybe I'll get back to this…
Skybuck Flying wrote
(Side question: can an RTL actually be a runtime library instead of being compiled into applications ?)

Runtime packages are yet a todo and not implemented.

Show quoted text 8 lines
However for the application which it is trying to compile needs to run on a different platform and thus the application needs to either be linked to that platform RTL or it needs to have that platform RTL embedded into it. And that's what free pascal more or less seems to do... it embeds "a target platform RTL" into the application that it tries to compile.

Because that RTL is just Pascal code as well and the to-be-compiled application uses that units and thus they need to be compiled in. Simple as that.

Show quoted text 6 lines
> Every compiler can compile the RTL for its target, as well as other > applications. No the compiler cannot compile the RTL for it's target because the compiler does not have the assembly yet to output the RTL ?

Yes, the compiler can compile the RTL for its target if the compiler is configured for the same CPU as the target has.

So at the very minimum the compiler needs to produce assembler output and for that it needs some kind of internal assembler-outputter ;)

A compiler always needs to output assembler at the end (in one form or the other...), that the use of a compiler.

Regards,
Sven

Reply by email