Hello,

I would like to experiment with the pascal programming language at the binary/assembler/machine level.

For example compile pascal sources to virtual machine instruction sets.

I took a look at the free pascal sources and there seem to be some assemblers inside the folders.

I see classes with names like TExternalAssember and TInternalAssember

I also see files called: aasmcpu, aasmtai.

Some processor architectures seem to derive from TExternalAssembler but then still implement lot's of stuff themselfes inside the class instead of calling a real external assembler (perhaps a trick ? ;))

This is confusing me a little bit so I have some question about what free pascal exaclty is, how it functions and what it does and can do or cannot do:

So some questions:

1. Are all assemblers compiled into the final free pascal executable ? Or is only one assembler possible which is selected by conditional defines ?

Suppose the answer to question 1 is: yes, multiple assemblers are supported.

(This seems to be the case since free pascal has the -a and -A directive with different asm output options... (?!?))

Then my next question is 2:

2. This would mean free pascal is by default a cross compiler, which can cross-compile to any assembly/intermediate code.

So then my main/subject question: How to add a new assember to free pascal compiler makes sense...

Assuming it makes sense I continue my story:

I searched the documentation about how to do this... but there seems to be no documentation for this ?

So if somebody could create some (short?) documentation for this or some tips how/where to start and how to proceed that would be helpfull.

Perhaps a simple dummy assembler or template might be nice... concerning this adding for a new assembler my main question would be where to add this unit, how to make free pascal notice it and use it and such ? Where to add the compiler directive and perhaps initialization and finalization of classes/create/destroy that sort of thing.

3. Assuming free pascal is a cross compiler, and looking at the code... it seems free pascal actually compiles pascal code into an abstract assembler structure, is this indeed correct ? Can free pascal be considered a "pascal to abstract assembler compiler ?" (which would be pretty cool) (which ultimately gets assembled into real assembler...)

So in short what I am looking for is a pascal compiler which takes care of all the work like "lexing/tokenizing/parsing/parse tree/syntax tree/symantec checks/perhaps even optimizing the intermediate code" and gives me something to work with like a syntax tree or some form of abstract assembler (structures) like free pascal seems to do...

To me it seems free pascal could be the correct choice... either by modifieing it or by extending it... So if I have to modify it... I would for example modify the intel or powerpc or sparc source code and pretend it's my own thing... but it would probably be better if I can stuff my own assembler/platform/units into it to start from scratch just to play nice with free pascal and prevent confusing things later on...

So I think I need to some help with: is this possible and how to do
it/documentation/short tutorial/some tips ?!?

Bye,
Skybuck.

Answering my own question ;)

I guess the answer to my question is more or less the same as the following question:

"How to add a new target to free pascal compiler".

And I found this webpage which explains it a bit:

http://wiki.freepascal.org/Porting_Free_Pascal#Adding_a_new_target

I'll quote webpage here in case it ever goes lost ;)

"
Bellow is a example of how a new target is added. This Patch was responsible for adding SymbianOS target to the compiler. Be careful that this adds a i386-symbian target. If your target is for another architecture of course you should change the files on the directories for your particular architecture.

Index: compiler/compiler.pas
=================================================================== --- compiler/compiler.pas (revision 6039)
+++ compiler/compiler.pas (working copy)
@@ -107,6 +107,9 @@
{$ifdef win32}
,i_win
{$endif win32}
+{$ifdef symbian}
+ ,i_symbian
+{$endif symbian}

   ;

function Compile(const cmd:string):longint;
Index: compiler/i386/cputarg.pas
=================================================================== --- compiler/i386/cputarg.pas (revision 6039)
+++ compiler/i386/cputarg.pas (working copy)
@@ -71,6 +71,9 @@

     {$ifndef NOTARGETWATCOM}
       ,t_watcom
     {$endif}

+ {$ifndef NOTARGETSYMBIAN}
+ ,t_symbian
+ {$endif}

{**************************************
Assemblers
Index: compiler/systems.pas
=================================================================== --- compiler/systems.pas (revision 6039)
+++ compiler/systems.pas (working copy)
@@ -136,7 +136,9 @@
system_x86_64_embedded, { 55 }
system_mips_embedded, { 56 }
system_arm_embedded, { 57 }
- system_powerpc64_embedded { 58 }
+ system_powerpc64_embedded, { 58 }
+ system_i386_symbian, { 59 }
+ system_arm_symbian { 60 }

        );

tasm = (as_none
Index: rtl/Makefile.fpc
=================================================================== --- rtl/Makefile.fpc (revision 6038)
+++ rtl/Makefile.fpc (working copy)
@@ -28,6 +28,7 @@
dirs_solaris=solaris
dirs_gba=gba
dirs_nds=nds
+dirs_symbian=symbian

[install]
fpcpackage=y
Index: utils/fpcm/fpcmake.ini
=================================================================== --- utils/fpcm/fpcmake.ini (revision 6038)
+++ utils/fpcm/fpcmake.ini (working copy)
@@ -908,6 +908,12 @@
SHORTSUFFIX=gba
endif

+# Symbian OS
+ifeq ($(OS_TARGET),symbian)
+SHAREDLIBEXT=.dll
+SHORTSUFFIX=symbian
+endif
+
else
# long version for 1.0.x - target specific extensions

Index: utils/fpcm/fpcmmain.pp
=================================================================== --- utils/fpcm/fpcmmain.pp (revision 6038)
+++ utils/fpcm/fpcmmain.pp (working copy)
@@ -70,7 +70,7 @@
o_linux,o_go32v2,o_win32,o_os2,o_freebsd,o_beos,o_netbsd, o_amiga,o_atari, o_solaris, o_qnx, o_netware, o_openbsd,o_wdosx, o_palmos,o_macos,o_darwin,o_emx,o_watcom,o_morphos,o_netwlibc,
- o_win64,o_wince,o_gba,o_nds,o_embedded
+ o_win64,o_wince,o_gba,o_nds,o_embedded,o_symbian

       );

       TTargetSet=array[tcpu,tos] of boolean;

@@ -88,14 +88,14 @@
'linux','go32v2','win32','os2','freebsd','beos','netbsd', 'amiga','atari','solaris', 'qnx', 'netware','openbsd','wdosx', 'palmos','macos','darwin','emx','watcom','morphos','netwlibc',
- 'win64','wince','gba','nds','embedded'
+ 'win64','wince','gba','nds','embedded','symbian'

       );

OSSuffix : array[TOS] of string=(
'_linux','_go32v2','_win32','_os2','_freebsd','_beos','_netbsd', '_amiga','_atari','_solaris', '_qnx',
'_netware','_openbsd','_wdosx',
'_palmos','_macos','_darwin','_emx','_watcom','_morphos','_netwlibc',
- '_win64','_wince','_gba','_nds','_embedded'
+ '_win64','_wince','_gba','_nds','_embedded','_symbian'

       );

{ This table is kept OS,Cpu because it is easier to maintain (PFV) } @@ -126,7 +126,8 @@
{ wince }( true, false, false, false, false, true, false), { gba } ( false, false, false, false, false, true, false), { nds } ( false, false, false, false, false, true, false),
- { embedded }( true, true, true, true, true, true, true)
+ { embedded }( true, true, true, true, true, true, true),
+ { symbian } ( true, false, false, false, false, true, false)

       );

     type

You will also need to add a i_target.pas and t_target.pas files to the compiler directory. Just copy an existing file for a similar platform and start working from there.

If you want your target to use an internal linker (ELF, COFF, etc.) you need to add your target to that one, too. For the target i386-nativent the following change was done in the COFF output generator:

Index: compiler/ogcoff.pas
=================================================================== --- compiler/ogcoff.pas (Revision 14564)
+++ compiler/ogcoff.pas (Revision 14565)
@@ -2906,7 +2918,7 @@

             idtxt  : 'PECOFF';
             asmbin : '';
             asmcmd : '';

- supported_targets : [system_i386_win32];
+ supported_targets : [system_i386_win32,system_i386_nativent];

             flags : [af_outputbinary,af_smartlink_sections];
             labelprefix : '.L';
             comment : '';

You might also need to add your target to one of the assemblers for the target platform. E. g. if you're targeting an i386 system you might add your target like the following (the line breaks annotated with {cr} were added to keep the layout of the article intact):

Index: compiler/x86/agx86att.pas
=================================================================== --- compiler/x86/agx86att.pas (Revision 15151)
+++ compiler/x86/agx86att.pas (Arbeitskopie)
@@ -380,7 +380,8 @@

             asmcmd : '--32 -o $OBJ $ASM';
             supported_targets :

[system_i386_GO32V2,system_i386_linux,system_i386_Win32,{cr} system_i386_freebsd,system_i386_solaris,system_i386_beos, system_i386_netbsd,system_i386_Netware,system_i386_qnx, system_i386_wdosx,system_i386_openbsd,{cr}
-
system_i386_netwlibc,system_i386_wince,system_i386_embedded,{cr}

                                 system_i386_symbian,system_i386_haiku,system_x86_6432_linux];

+
system_i386_netwlibc,system_i386_wince,system_i386_embedded,{cr} system_i386_symbian,system_i386_haiku,system_x86_6432_linux,
+ system_i386_nativent]; flags :
[af_allowdirect,af_needar,af_smartlink_sections,af_supports_dwarf];

             labelprefix : '.L';
             comment : '# ';

Note: In this case the change is only needed to get the "-al" option of the compiler, which needs the GNU assembler on Windows, working for i386-nativent.

[edit] Compiling your target
Now that you have a target you will want to see if it works and spot compilation problems by compiling the compiler. To do this, open a console, navigate to your fpc subversion source code directory and build the compiler:

PATH=C:\Path_latest_fpc_stable
cd compiler
make arm
And to build your RTL:

cd rtl\my_new_os
make FPC=c:\fpc_subversion\compiler\ppcarm -XP$FPCTARGET- While trying to fix (or implement) RTL for your new target, information found in System unit structure may be useful.

[edit] Porting to a new Architecture
[edit] Step 1 - The Binutils
The first step in porting to a new architecture is getting and understanding how the binutils of the platform work.

[edit] Step 2 - Code Skeleton
You can start porting Free Pascal to a new Architecture by creating a new Lazarus project and a new skeleton for it. You can get the Lazarus project for ARM for example, rename it, lets say to ppcmipsel.lpi, modify it in a text editor to refer to mips directory instead of arm and then modify it in Lazarus in the Compiler Options Dialog in the Others tab and set the compiler directive to -dmipsel instead of -darm.
"

This is probably enough information for me for now to go on ;)

Bye,
Skybuck.

Hmm... one of the reasons why I might not have found this documentation is because of this:

At the wiki/documentation link:

"
Online documentation

Remark:you must have javascript and style sheets enabled in order to view the html documentation correctly.

The following documents are online available:
"

(Followed by nothing)

I guess either javascript or style sheets are disabled on my IE8 browser because of security risks :|

Bye,
Skybuck.

If you just need "a pascal compiler" and not the advanced ""Object Pascal" stuff that FP provides, maybe gcc is more suitable. gcc (= "Gnu Compiler Collection") provides multiple front ends (c is the most known, Pascal is one of them) and as theses create a well defined intermediate code that is usable by all backends, it can compile Pascal to binaries for all supported archs.

-Michael

[snip]

" This is probably enough information for me for now to go on ;)

This information is only useful once the compiler can generate code for your target platform. A FPC target consists of two "bits": the cpu and the OS. You are currently at the "cpu" step and that is documented much less complete here:
http://wiki.freepascal.org/Porting_Free_Pascal#Porting_to_a_new_Architecture

See also my (yet to be written) answers to your first mail.

Regards,
Sven

Yes and no. One FPC compiler (ppc386, ppcarm, ppcx86_64, etc.) can only compile code for its own cpu (i386, arm, x86_64, etc), but multiple assemblers for this platform can be used. E.g. the i386 version of the compiler supports the GNU assembler, NASM and its own internal assembler. Other targets might only support the GNU assembler or some other third party assembler.

As you want to add support for a virtual machine which contains a virtual instruction set (aka not compatible to a existing instruction set) you'd first need to implement a new CPU target for FPC which will contain your assembler as well. Take a look at MIPS or ARM in the subdirectories of compiler as those are the "smaller" and perhaps easier CPUs to copy.

Best is to dive into the code yourself and try whether you can get a own CPU to run (as said: take a look at MIPS and copy what was done to implement MIPS for your own (virtual) CPU).

Free Pascal compiles directly native assembler code. It's just nicely wrapped up by the backend units (those found in the cpu subdirectories).

Regards,
Sven

I would like to experiment with the pascal programming language at the binary/assembler/machine level. For example compile pascal sources to virtual machine instruction sets.

Only one machine (instruction set) is supported by an FPC executable. This is because some conditional compilation is used to handle machine or platform specific issues in the parser code.
Look at the various compiler/ppc*.lpi projects, and the related subdirectories.

I took a look at the free pascal sources and there seem to be some assemblers inside the folders.

Multiple assemblers may be usable in one FPC executable.

Then my next question is 2: 2. This would mean free pascal is by default a cross compiler, which can cross-compile to any assembly/intermediate code.

Right.

"make fullcycle" builds all (registered) cross-compilers. fpc[.exe] is the stub that selects the concrete compiler to use. But every single compiler can be invoked directly, too.

So then my main/subject question: How to add a new assember to free pascal compiler makes sense...

Starting with
http://wiki.freepascal.org/Porting_Free_Pascal#Porting_to_a_new_Architecture

you'll find a lack of documentation about this topic :-(

I've been starting once by copying the m68k directory as mycpu, because I was more familiar with that cpu. You also can use the "vis" directory for your first steps, it's the smallest (and incomplete) port. Then I registered the new directory for it (dunno remember how/where). In all files and identifiers the old cpu name should be replaced by the new one.

Then the instructions (enum) and register sets have to be specified for the new cpu. The vis cpu turned out to be not very helpful, I had to look at various cpu's in order to find out what the register data structures mean.

When you try to compile the rudimentary cpu, you'll get errors about missing units and more, telling you where more has to be done. Have a look at the initialization sections of the cpu's, they often call kind of registration functions or directly initialize global variables, and let them point to cpu-specific tables.

Unfortunately I cannot locate my last-year experiments with a new cpu, and the docs I wrote then :-(

But if you want to continue with your project, we can work together and document what we found out.

The pascal code is translated into nodes (TNode), which represent the abstract code flow. You must derive your own node classes, and add the concrete code generation to them. Your cpunode.pas lists (uses) all your node units, typically named n<cpu>*.pas.

See above. I'm working more on the front-end side, to add parsers for other languages.

See above. New platforms and processors can be added to Free Pascal freely, once the related hooks and procedures are known. Modifying existing assemblers IMO should be done by conditional compilation, so that eventual updates of the original code are integrated automatically.

DoDi

Hello,

(This posting/reply is mostly ment for: Hans-Peter Diettrich)

I saw you mentioned you are working on the "front end part" of the compiler.

I also saw you want to introduce either syntaxes or perhaps even languages (?)

I guess you want to experiment with alternative syntaxes or introduce other languages, correct (?)

Could ofcourse also be learning experience for you...

But where are your primary goals ? ;)

I can imagine alternative syntaxes...

I can also understand the wish to perhaps introduce C syntax or C++ syntax... perhaps for the speed of the compiler ?

You also seem to want to make the front end more flexible... I can understand that in a way ;)

But why exactly ?

Perhaps you can shed some more light on what kind of language experimentation or other languages you had in mind ?

Currently I see that the free pascal compiler was not written in purely "turbo/borland/delphi/oo pascal"... instead it uses some new language/syntaxes here and there... which will probably cause some transpiration/headaches down the line/in the near future...

So seeing these little "problems" I am not so sure if I like your kind of experimentation ! LOL.

But then again "parallel language extensions" or perhaps "cuda/ptx" language extensions in the future might be interesting.

One thing I haven't mentioned so far is even support for "opengl" and "nvidia cg", pascal shading languages.

I would also be a little bit interested in a pascal compiler which could compile to a shading language.

Perhaps some of your front-end work might come in handy to introduce new pascal language syntax specially tailored for shading language... which could then be enabled or disabled in project settings or even better "compiler-understand conditionals/directives" that would be real nice...

Bye,
Skybuck.

Here is one little idea which might be obvious:

1. The assembler could notify to the rest of the system like parser or symantic checker which languages features are supported and which are not supported by the assembler.

This way certain "language elements" of the language could be turned off. This might be helpfull to introduce pascal to new hardware which might not support certain things like "pointers" or "virtual pointer/method tables/whatever" or "calls" or "integer support" or "floating point support".

So each element of the pascal language could get a boolean "supported/not supported" so that an assembler can "cherry-pick" what it wants to support at first...
then later it can start supporting other elements.

The benefit of this is that the symantic checker or parser can throw warnings/errors into the face of the user to notify him/her that he/she is using non-supported language features and such re-consider for the time being ;)

2. It would/could also be beneficial to the user... if the user simply doesn't want to use certain new language features and wants to make absolutely sure that they are not accidently being used then he/she could also disable certain language features and have the compiler automatically check if the user source code is complieing towards the settings/language element rules.

This could be helpfull in later porting/re-writing the code to other languages and/or systems. For example: first developing software on PC/CPU and later porting it to PC/GPU/Shaders/CG/OpenCL/PTX or anything new that comes along ;) =D

3. It might even be helpfull to disable "buggy language features or types" and quickly diagnose source code to see if it contains non-allowed language features/types.

4. Last but not least, handy new languages features could be turned on and be used ;) :) =D

Bye,
Skybuck.

Both, but this project is OT here.
Have a look at my SourceForce project FpcExt...

DoDi