Apologies that I've broken the thread, but some messages don't get through our gateway- for some reason I'm seeing the message I'm quoting arriving as an encoded attachment.

[Sven said]

If you'd now only explain what a cross-reference tool is I might even understand what you're trying to tell me here...

I think what he means is that he wants a tool that will annotate each procedure/function/method call in the source with the file and line number that it transfers control to. That's obviously going to be a problem in a language like Object Pascal which supports virtual methods, where the actual target isn't known until execution time.

With the caveat here that as I understand it experienced IBM programmers avoid the GNU assembler like the plague, since it doesn't have anything like the sort of macro facilities they're used to. By implication, that would imply that they prefer to avoid the GNU linker and related tools as well.

Just to name a few: you'll need to get parameter passing for functions correctly

Which leads to another issue: the 370 is a register-based system without a stack as understood today. Parameters are mostly passed in registers, but this is largely hidden since supervisor calls etc. are usually hidden in macros.

My own feeling is that it would be best to start targeting a late-model 390, which does have a stack etc., and to use the standard GNU assembler and linker (as and ld) /initially/ targeting Linux. Any other combination (i.e. a proprietary assembler etc. with an antique MVS as target) is going to cause nothing but grief, since it makes it very difficult for developers skilled with FPC but not with IBM mainframes to give any practical help.

Ah ok. In that case that indeed not a good idea to do especially with the compiler of which the code generator relies heavily on virtual methods (and exactly this part is what Paul is interested in). For the parser it might work however, but there you could also add a "Writeln({$I %file%}, ' ', {$I %line%});" at the beginning of each function :P

A compiler is something different than a "experienced IBM programmer". If a programmer who writes assembly code directly tries to shortcut things through macros that is one thing, but for a compiler it doesn't matter that much whether it can make use of the macros or not. Let's assume there is a assembler macro for the System.Move functionality. A IBM programmer might prefer to use that, while the compiler has no problem with translating the Pascal implementation of System.Move to "low level" s370.

Mark Morgan Lloyd wrote

There is a problem inherent in this discussion; zArch is not one environment! It's one architecture supporting multiple operating systems, much like the 386/486/586/686 etc supports Linux or DOS or OS/2 or Windows (all 37 versions) etc. zArch has MVS (in all it's varieties) VM, DOS, LINUX, MFT, MVT, MUSIC/SP, and loads of other more niche stuff. In addition to all this, later versions of MVS supply a POSIX compliant shell called OMVS. GNU anything is available in hardly any of these environments even if we can handle the brain-dead assembler.

I am an MVS person so I can't speak for the other lot but parameter passing is mostly done in storage. The standard linkage conventions used allow for two 32-bit signed or unsigned integers (64-bit in later models). Anything else is passed in a storage area pointed to by register 1. Where this storage area comes from is complex and variable. My guess that the other IBM systems have similar models. A further guess is that Linux based code allocates an area of storage, points a register at it, write a couple of mickey mouse macros and bingo, a stack.

Late-model 390's have a stack, but not as you know it. It's not something you can go around lobbing arbitrary data at. It is reserved for data saved during subroutine linkage using the appropriate hardware instruction (Branch and Stack). This includes various register sets, PSW status info etc) and an additional two 32-bit signed or unsigned integers (64-bit in later models).

Steve

GCC plus the basic utilities are definitely ported to VM/370 and MUSIC/SP (i.e. I've run them), and as far as I know to the others. In all cases possibly subject to memory restrictions, i.e. might need more than the standard 16Mb supplied by older operating systems for large jobs (that's where the /380 patch to Hercules comes into it).

Don't guess, see http://wiki.lazarus.freepascal.org/ZSeries which apart from anything else has links to the original GCC implementation notes which in part determine what architecture variants Linux supports. However there also appear to be later ports which support older CPU variants, see link below.

The examples at
http://wiki.lazarus.freepascal.org/Assembler_and_ABI_Resources#zSeries_.28S.2F390.29 show a significant simplification of the code when a newer architecture is specified.

First, let me apologise for that last email. The subject and the formatting was awful. I blame my eMail client, not the fact that I am incompetent. The latter is a calumny that I will defend to the utter limits of the law. :) At least it wasn't HTML.

(Brain the size of a planet but thunderbird just totally baffles me I'm afraid)

To business...

Sven wrote

If 360 assembly code can be used on modern processor variants as well I see no problem with targeting that at first only.

Modern processors provide new instructions that would certainly make life much easier, but IBM put a load of effort into keeping forward compatibility intact. This is true even at object code level. I have worked in environments that lifted Fortran compilers that were 25 years old and run them successfully on modern OSes without recompilation. The only fly in the ointment is Linux (isn't it always the case!). IBM do not supply an assembler for Linux. They use gas. But there are "proper" Assemblers available.

The point with using Linux as a first target is that you would not need to implement a new RTL, but only the code generator and the processor specific RTL code.

Hmmm! I think it will be more complex than that... Certainly, anything related to I/O will need to be drastically rethought, even before it's rewritten. The I/O architecture is just too different.

The point for gas/ld was simply that we have existing writers for these two, but writing your own writers for IBM specific tools isn't rocket science either... But it's another thing you'd need to implement.

You don't have existing writers! They may be familiar with the basic syntax that gas/ld use, but IBM assembler, gas, Assembler F or High Level Assembler has a totally different vocabulary. The semantics are also unique. CALL exists but is different, RET doesn't exist. JMP and it's variants are called B (B for Branch) and the variants are different. What is the difference between L and LA? What are L and LA? MOV doesn't exist, there a whole host of different instructions for moving data depending upon source, destination and format. Do they understand what the 16M line and the 2G bar are and their implications for code? And a whole other list longer than my arm. The manual that defines the Assembler language is, at last count, 1292 pages of A4! All of it different from what your pool of gas coders is familiar with, unless they have previously written IBM Assembler, in which case they probably used Assembler F or one of it's successors.

And it's not just the assembly language that's different, the target hardware bears absolutely no relation to what they are used to apart from the fact that it uses 8-bit bytes; The I/O subsystem is radically different, even the character set is different.

Sorry, isn't this a universal problem, or is it just me? Anyway.. If I am writing code in Delphi and I want to use, say, MySQL; I have a problem; Delphi provides no mechanism to call MySQL. If I am a C programmer, I just include the .h file and MySQL is available. Instead, I have to troll round the internet looking for a component or set of components to provide this support. Often especially in the early years of Delphi, there wasn't one. So I have to write my own. I sit down with the manual and the MySQL.h header file and reverse engineer the relevant code from it's C equivalent. I bought Delphi so I wouldn't have to mess around with C. I hate C. I consign it the same hell that 380 belongs in... No, I consign it to the lowest levels of hell. (Deep breath).

But everything is defined in terms of C and, ultimately has to be reverse engineered into a Pascal equivalent. Wouldn't it be nice if I could say "Uses MySQL;" and away it goes. If it can't find a Delphi unit it looks for MySQL.lib or MySQL.h or whatever and generates the code for me.

I'm sure you're aware of the process, and, yes, I am aware of the problems, but you asked me to elaborate.

The same thing exists in MVS, and the other OSes too but I am an MVS man so I will stick to that. If I want to link to a system facility or another product there are a number of different ways it can be achieved.

1) Use a macro. All base OS facilities are defined by Macro interfaces. Some of the actual binary interfaces are documented. Some are not. Without a varying amount of reverse engineering, gas won't allow us to use them. You called them a "shortcut" in another post. Sometimes they are; Sometimes they are the only sensible way to go because the long way round is far too complex. To take 1 example; Memory management in MVS has 4 or 5 (I think) Supervisor Calls to allocate and deallocate a block of memory. Which one is required depends on the parameters passed to the macro, it's is not a trivial task. Gas doesn't allow macros.

2) Use CALL to link to an interface module. In these cases the CALL interface is documented, but the actual code to link to the product is complex. It may involve Authorisation changes or Address Space switches or finding addresses of Object Code only storage areas. The CALLed code is linked automatically by the linkage editor. Ld can't read MVS load libraries.

3) DB2 and CICS, along with IMS these are the war-horses of MVS application development. They use a pre-processor. The assembler source code is fed into a pre-processor, it generates the required assembler and the output is fed into the assembler. The pre-processors don't understand gas assembler, except, possibly, for the latest and greatest versions. These interfaces are complex command-oriented ones and the assembler equivalents are not documented.

Elaborate enough? :)

Regards,
Steve

This is not really true. I have an C-to-Pascal translator, that translates the header files perfectly. Well, some constructs (with unions) cannot be translated as they are declared, but these are rare and do not affect API calls, only subsequent access to the struct fields.

But that translator requires the right header files of the compiler, used to translate the library, compiler settings etc. If you didn't compile the library on your system, even a C compiler may fail to properly connect to the library ABI. This is not a big problem on systems with only one C compiler and library, but how many C compilers exist for Windows? Even worse with C++, where no compiler produces binaries usable with a different C++ compiler.

In fact my translator can convert C code of many compilers into OPL, where some compilers require very special translation. But configuring it is so complicated, that only few people ever used it. Most are happy with h2pas or similar tools, which only deal with header files and can produce unusable results in certain cases. You have the choice...

DoDi

It's not. Linux on 390 (caveat: my experience is on Hercules, but I have no reason to believe that the distro I was using had been tailored for emulation) looks like Linux on SPARC, Linux on MIPS, Linux on PPC... and so on. The only significant departure is that they don't see Ethernet hardware, but instead talk SLIP to a gateway provided by Hercules or use some sort of offload engine.

STOP. I think that between us we've got a terminology problem: Sven wasn't using "writer" to refer to somebody writing assembler code, but to an object embedded in the compiler that is tailored for emitting assembler statements.

In other words, when we say that an existing assembler writer outputting gas-format statements for (say) MIPS or 68K can probably be adapted without too much trouble for s390, we aren't referring to coders on the development team but to a body of code that comprises a significant proportion of the compiler's backend.

I'm skipping over the remainder of the message with minimal comment, since the significant detail is that if the initial FPC implementation is for Linux than it will obviously have to have the ability to link to standard libraries on the same system.

Exactly. On Linux we'd simply use the Linux I/O API which is already implemented in FPC. This is why it's a good idea to use Linux at first: less work on the RTL.

Exactly. Thank you for explaining. :)

Elaborate enough?

Yes, it was. But Pascal is a nieche language currently, so we need to be grateful already that we can use C libraries without much problems... also FPC comes with some header conversions already included, like MySQL including a database connection wrapper for TDataset ;)

Regards,
Sven

Sorry, I don't follow some of the comments in this thread.

First, what is a stack? A stack IMO consists of a stack pointer provided by the
hardware, that is, a register. And even the old IBM machines have many registers
that can be seen as stack pointers; you only have to select one. That is in fact the
way that the stack has been implemented in "old" compilers on "old" IBM hardware,
be it Fortran, Pascal, PL/1 or C. All those languages needed kind of stacks for
local variables of procedures etc. So IMO there is no need to target new hardware;
this will be very expensive and excludes the use of Hercules simulators and free versions of IBM operating systems for first tests. Even todays IBM compilers
don't use the "new" hardware stack.

And then: I have some doubts about the linkage between FPC and the GNU tools,
like as and ld. Why is it easier to port FPC to a Linux based system? If the compiler translates into an abstract intermediate language first and then into an abstract assembly language maybe - for an abstract machine like the P-machine - then the nature of the assembler and linker used should be irrelevant. Maybe there is some misunderstanding on my - or our - part;
I have the Wirth compilers in mind, and there is a clear separation between the machine independent parts - phase 1 - which generates P-code and the machine dependent parts - phase 2 and runtime. Even if there is no such separation in FPC, it should IMO be possible to develop and test the code generation separately.

I, too, had the difficulties, like Paul Robinson, that I did not get the cross-compiler
working. My goal was, for example, to have a cross compiler running on Windows, that produces Assembler output for MIPS for example, and for a second target S370, which is at the beginning simply a copy of MIPS, producing the identical output, but then I could make changes to the S370 code generation and try to get the results running on a simulated or real 370 hardware.
Could you maybe outline the steps that are necessary to create such an environment?

Kind regards

Bernd

This is correct, but it should be possible to cover most of those systems with
one RTL - much the same way as LE (language environment) does it in the "big" IBM world, too.

Ok. So what? This makes parameter passing just easier, because you don't have to deal with special cases for - say - low number of parameters. It's all the
same method. And Pascal has call by value and call by reference and - for example -
not the strange descriptor and dope vector things that PL/1 has. The stack is simulated
by simply incrementing and decrementing the stack pointer on procedure entry and
return and addressing relatively to it. At least that is what is done in the "classical"
compilers. I recently learned a bit about the MIPS architecture, and it seemed to
me to have some similarities to IBM's, so I guess, a kind of "translation" from MIPS
to IBM could be successful. Many people say that the IBM platform is the only
remaining platform today that reasonably can be programmed using assembly language,
because of the clear structure of the instruction set.

I would strongly suggest not to use the "new" hardware stack, but instead to
rely on the OS linkage conventions (save area chaining, register 13), which in fact are used by most of the operating systems mentioned above - with minor differences.

You are not free to select one register for that purpose, unless it's guaranteed (by calling conventions) that all software preserves this register.

Such subroutines deserve 3 addresses, to their parameters, local variables, and a return address. The latter can not be choosen freely, it's implemented in machine CALL/RETURN instructions, which manage kind of an *return stack*. When these instructions predate the hardware stack, the return addresses obviously have been managed in some different way, not using the new hardware stack. This means that a calling convention must be established, for passing subroutine arguments and return values, for local storage allocation, and for stack unwinding. Some convention may already exist, for calling system services or library functions. Apart from that every compiler can choose its own model, for further (private) calling conventions. In most cases a single register (Base or Frame Pointer) is used for that purpose, which *can* point to some stack location, but it also can point to any other convenient memory frame. Let's call it the *data stack* for disambiguation.

Both kinds of stacks become very important when a debugger is added. The debugger must not only know about the return stack, which is machine specific, but it also must know about the calling conventions, how to access subroutine parameters and local variables, and how these frames are linked together during subroutine calls. This means that, when an already existing debugger shall be used, the available calling conventions are already specified!

DoDi

This will not help you because you don't learn anything by this. As said previously, look at the steps how I started the aarch64 port and try to do the same for S370 or whatever.

I think that an initial port to Linux (targeting a late-model 390 emulated by Hercules) is unavoidable, and that also has the advantage of being of immediate use to the wider community. That implies that it has to be able to use the standard calling convention on that platform, as excerpted at
http://wiki.lazarus.freepascal.org/Assembler_and_ABI_Resources#zSeries_.28S.2F390.29_2 Also see the assembler examples at
http://wiki.lazarus.freepascal.org/Assembler_and_ABI_Resources#zSeries_.28S.2F390.29 which show actual assembler sequences generated (one for the Linux kernel, others for a "Hello, World").

So at the very least, we have to consistently simulate a stack- apart from anything else, that's mandated by Pascal's use of recursion. But we don't necessarily have to use the same calling convention for Linux and for "classic" OSes (i.e. including those which are freely-available, running on Hercules etc.).

Question (to save me digging into the manuals right now): where a recent machine uses the dedicated stack instructions, is the stack pointer one of the standard registers? In other words, can push/pop operations be trivially and exactly simulated for older hardware?

And then: I have some doubts about the linkage between FPC and the GNU tools, like as and ld. Why is it easier to port FPC to a Linux based system?

Because the FPC sources are ASCII, all the RTL code assumes ASCII (including collation order, behaviour of Pred() and Succ(), and so on) and- most importantly- because almost all of the developers are either using it or are familiar with it by necessity.

Yes, but the compiler /doesn't/. The one possible getout here would be to use one of the variants that already generates Java bytecodes or (potentially) something like WebJS, but that would immediately exclude any of the "classic" OSes unless a custom backend were written from scratch.

I don't think that would work, there are too many quasi-circular dependencies (I'm intermittently wrestling with a potential code-generation problem, and hoo-boy).

I agree that the MIPS target is very similar, including Linux calling conventions etc. (noting that there are several possible calling conventions, and that information describing these might only be available under NDA).

However note Florian's comment about there being an optimal sequence for bringing things up, and also note that Sven has recently worked on the 68K port which might be sufficiently-similar to be relevant.

I suspect that targeting "classic" OSes which use EBCDIC would need a branch of the RTL, even if the sourcecode in that branch were in ASCII. I also suspect that- however attractive this is- trying to port the compiler so that it would self-host on EBCDIC-based OSes would effectively fork the project, unless somebody could come up with a neat hack to automatically convert patches back to ASCII when they're sent to the Subversion repository.

Thought for everybody even remotely interested: if we could standardise on the first line of every unit being a non-essential comment (and specifically, not a compiler directive) then it would be possible for the parser to decide whether it were looking at ASCII or EBCDIC before hitting anything important.

A postcompilation translation probably wouldn't work since e.g. there are different ranges of parameters used for compiler passing, but I was wondering whether the MIPS directory could be moved en masse. But again, note Florian's warning about this.

In fairness, the x86 probably falls into that category as well. However I think most people abandoned x86 assembler when both Linux and the NT family of OSes demonstrated that C was adequate for portable device drivers etc.

Noted, but I think we have to go along with Linux practice for the initial port.

If someone wants to port the compiler to a new target processor it is advisable to look whether there exists an OS that is already supported by FPC, because then "only" the code generator and the CPU specific parts of the RTL need to be implemented while the remaining RTL can be reused which simplifies the first steps of the port. Otherwise you'd need to implement the code generator and a more or less complete RTL. So as Linux seems to be available at least for some variants of the CPU I would strongly suggest to target Linux first and other OSes later.

Also the compile process of FPC is roughly this:
- for each used unit:
- parse the unit and generate a node tree for each procedure/function/method (basically platform independant)
- generate a CPU specific linear assembler representation of each node tree (this representation is independant of the specific assembler used)
- if an external assembler (e.g. GNU as) is used: convert the assembler lists to assembler files
- call the assembler (internal assemblers work on the assembler lists directly) to generate the object file
- for external linkers (e.g. GNU ld): write a linkscript to instrument the external linker
- call the linker (internal linkers work directly on the in memory information the compiler has gathered)

For a new port it is advisable to not implement internal assemblers and linkers, because that means more work, but to use existing tools for the given platform. And here like for the RTL assembler/linker interfaces for the GNU assembler and linker already exist which simplify a port. Later on additional assembler/linker interfaces for other assemblers/linkers can be added or even an internal assembler and/or linker can be written.

Regards,
Sven

Presumably that implies that it's not feasible to have an internal assembler followed by a standard platform-specific linker. The reason I highlight this combination is that I've had no success finding a free assembler other than GNU as (gas) which covers all development platforms, in particular "The free Tachyon Legacy Assembler is available for MVS 3.8 and Linux/390" which obviously makes it NBG for the initial stages of development on x86 Linux or Windows.

One of the links I posted earlier demonstrates that GCC is generally tailored to suit the prevalent assembler on each operating system (Linux, MUSIC/SP, CMS or whatever). That obviously doesn't imply that the macro packages that serious assembler programmers would expect are available.

Thank you very much for that, that made things much clearer for me.

So the compiler relies heavily on the external assembler and the syntax it supports,
as long as you don't want to do changes to step 2 (that is, change the linear assembler
representation, which IMO should not be done in the first step).

And: the assembler is not called once, but for every unit.

So here, I think, we have some problems or issues, because, as already pointed out,
the z-Arch doesn't have PUSH and POP instructions, and I guess that the outcome
of the linear assembler representation will not be very suitable to the things that the
z-Arch instruction set provides, although in the meantime there are some 1500 instructions.

Understanding that, I would now like to have some description of the linear assembler
representation that FPC generates, that is: it is of course not target-specific, but it does of
course do some assumptions on the type of the underlying hardware. Maybe, for example,
it assumes the existence of PUSH and POP instructions and some number of registers
which can hold fixed point and floating point values and which are the target of the
PUSH and POP instructions (and, of course, ASCII). The z-Arch Hardware, in contrast, normally
does not access parameters - for example - by issuing individual PUSH or POP instructions,
but - for example - if there are ten parameters requiring - say - 64 bytes of storage,
it increments the "stack pointer" only once (by 64) and accesses the parameters
by relative offsets to the value of the "stack pointer". Simulating the individual
PUSHs and POPs by Assembler macros would by possible, but it would be a waste of time.

So my question is: is it possible to modify the outcome of step 2 (the linear
assembler representation) depending on the target platform - which is desirable
in my opinion for performance reasons - or: should we stick with the outcome
of step 2 - which contains probably PUSH and POP instructions in gas syntax etc -
and simply try to convert them in the one or other way to z-Arch instructions -
z Assembler Macros or a more intelligent assembler writer function which converts
gas syntax to z Assembler syntax - and accept the performance degradation in the first place?

BTW: is it possible to print the linear assembler representation - the outcome of step 2 -
which in my understanding is NOT target-specific - and compare it - for example - to
the assembler code generated for MIPS? I believe that from that mapping there could
indeed be learned something ...

I, too, would try to rely on existing assemblers and linkers, but I have the feeling that HLASM (or free Assemblers supporting the z-Arch vocabulary) is a better choice than gas. I believe that there are some people out there waiting for FPC being available on z-Arch, and they are not all Unix-aware, but they read and understand the classical z-machinery well and would like it, if FPC was available there, without additional prerequisites (from their point of view).

Of course, providing an internal assembler/linker is too much work and even not necessary, in my opinion.

In our previous episode, Mark Morgan Lloyd said:

That cuts both ways. The 16-bit target uses nasm, and it is really noticable that it operates much slower than (G)AS.

Speed is also the main reason for an internal assembler. It is yet noticably faster compared to AS. (and then I'm not even discussing old styled (non section based) smartlinking, when it is several magnitudes)

The Linux and *BSD x86 platforms by default all use an internal assembler and an external linker. Even an internal linker combined with an external assembler works. The internal linker can load external object files (otherwise it would not be able to link in static libraries).

Jonas

Bernd Oppolzer wrote:

I'm about to head out, so have to be extremely brief.

Look at the output when using FPC's -a options, for example -aln... that might in practice need the EXTDEBUG setting during compilation but I can't go into more detail now.

Push will typically be used to put parameters onto the stack, otherwise they'll be accessed by indexed operation. The stack frame is discarded by target-specific code.

As Sven stated above, the linear assembler representation is completely independent of the external assembler and the syntax it supports.

So here, I think, we have some problems or issues, because, as already pointed out, the z-Arch doesn't have PUSH and POP instructions, and I guess that the outcome of the linear assembler representation will not be very suitable to the things that the z-Arch instruction set provides, although in the meantime there are some 1500 instructions.

The compiler does not in any way rely on the presence of push and pop instructions.

The JVM does not have any registers at all and is supported by the compiler.

So my question is: is it possible to modify the outcome of step 2 (the linear assembler representation) depending on the target platform

The linear assembler representation is already 100% platform-specific (as Sven mentioned above). FPC does not have a platform-independent internal RTL (register transfer language) representation.

Jonas

On 01.09.2013 15:49, Bernd Oppolzer wrote:

Most points were already answered by Jonas, so I'll only tackle the remaining ones:

There is currently no way to write this linear assembler representation as is. Only through the assembler writers they can be written.

The entries of that linear presentation are mostly located inside the aasmtai.pas and $CPUTARGET/aasmcpu.pas units (each tai is one entry inside the linear list) with one of the more important ones being taicpu as this is the one which encapsulates the platform specific CPU instructions.

The point is still that one should choose Linux as a first target OS for that new platform and thus the assembler must be supported there as well. As GNU as is normally always there where there is a Linux this would mean that for the first steps of the implementation no new assembler writer needs to be implemented (though this isn't necessarily rocket science). If HLASM supports assembling code for Linux you can of course implement that as first.
My suggestion is however to only implement as few things as possible as the first step which means "only" implementing the code generator and no new RTL, no assembler writer and no linker interface. Later on once Linux works good enough (not everything needs to be implemented for this!) additional assembler writers or target OSes can be added. You are working inside an unfamiliar environment (the compiler's code) and thus you should it make yourself as easy as possible. Otherwise you might experience frustration when doing this port...

Regards,
Sven

Thank you for that; I will take a look at it, although I have some doubts, if the output is "target-specific" or "not target-specific" - and if my understanding
of the linear assembler representation being "not target-specific" is right.

For that question I would like some statement from the core developers: how would you deal with a machine that has no built in PUSH instruction? For example if a function call puts five parameters on the stack, which is

LD A
PUSH
LD B
PUSH
LD C
PUSH
LD D
PUSH
LD E
PUSH
CALL FUNC

given an accumulator which is target of LD and a PUSH instruction which PUSHes
the content of the accumulator to the stack.

In my understanding this could be the not-target specific representation of the calling sequence

The z-Arch could produce something like

L R5,A
AHI R1,4
ST R5,0(R1)
L R5,B
AHI R1,4
ST R5,0(R1)
L R5,C
AHI R1,4
ST R5,0(R1)
L R5,D
AHI R1,4
ST R5,0(R1)
L R5,E
AHI R1,4
ST R5,0(R1)

here evere PUSH is emulated by the AHI (increment of the "stack pointer" R1)
and then the indirect store.

But more efficient would be:

L R5,A
ST R5,0(R1)
L R5,B
ST R5,4(R1)
L R5,C
ST R5,8(R1)
L R5,D
ST R5,12(R1)
L R5,E
ST R5,16(R1)
AHI R1,20

still more efficient, if you use other registers (not only R5); if so, you can maybe store all the values into the stack using only one instruction (STM) - if the variables are loaded into consecutive registers (R5, R6, R7 and so on).

That's what the existing compilers on z-Arch normally do - they don't compile the PUSH instructions one by one as in the first example, but in contrast,
as there are no PUSH/POP instructions provided by the hardware, they do some efforts
to do at least only one increment to the stack pointer (like outlined above) which
is done in the procedure or function prologue.

Now my question is:

do you think that this is a major problem for a FPC port to z-Arch?

Are my assumptions right so far?

Should we start with an easy solution and check the performance implications later?
Maybe there is a clever solution to that ...

Kind regards

Bernd

Thank you.

So my assumptions so far were plain wrong.

The difference between platforms is already in the stages above stage 2.

That is (cited from Sven's mail):

<citation>
Also the compile process of FPC is roughly this:
- for each used unit:
- parse the unit and generate a node tree for each procedure/function/method (basically platform independant)
- generate a CPU specific linear assembler representation of each node tree (this representation is independant of the specific assembler used)
- if an external assembler (e.g. GNU as) is used: convert the assembler lists to assembler files
- call the assembler (internal assemblers work on the assembler lists directly) to generate the object file
- for external linkers (e.g. GNU ld): write a linkscript to instrument the external linker
- call the linker (internal linkers work directly on the in memory information the compiler has gathered)
</citation>

only the first step - node tree - is platform independant, and the translations from there is already CPU specific - oh, I see, it's written there - I looked at the word "independant" in the paranthese - my fault ...

Sorry for that ...

Then the main effort is to understand what the contents of the node tree mean
and to build another variant of step 2 (for z-Arch).

Kind regards

Bernd

No need to answer to that ... I understood in the meantime that FPC does NOT rely on
PUSH and POP instructions. Instead the linear assembler representation is already fully
CPU specific.

(which makes porting a bigger effort)

Kind regards

Bernd

Proof?

You really want to study the compiler sources (e.g. starting with the AARCH64 support as Florian suggested) before making such statements. There is still an abstraction between the node representation and the assembler output, it's just not an architecture-independent RTL but instead generic code generation methods that can be overridden where necessary.

Jonas

At this point I'd throw in that one of the things the higher levels of the compiler knows is the overall properties of the registers, i.e. things like which ones are available for procedure parameters. This is one of the things that the lower level has to specify, so the lower-level units aren't there solely to do a macro-style substitution converting the compiler's internal representation to a sequence of assembler lines.

The corollary of this is that it's fairly common for a new target CPU to necessitate higher-level changes, and these then have to be propagated to all of the other targets. Which is why it's important to keep people like Florian and Jonas happy :-)

No, but it's important to adhere to the calling conventions for the various target operating systems. As a particular example, on Linux compiled code /has/ to be able to link (statically or dynamically) to libraries such as the resolver that maps computer names to IP addresses.

Are my assumptions right so far?

I think so. The important thing is that Pascal mandates a stack due to the prevalence of recursion, and (in the case of Linux) the ABI specifies a combined parameter/return-address stack. The question at this point is obviously what registers the ABI reserves as stack and frame pointers.

Should we start with an easy solution and check the performance implications later? Maybe there is a clever solution to that ...

It's my opinion. If the compiler translates the source language to machine code for an abstract target machine that is not too complicated (but well suited to the needs of the source language), you only have to translate
the operations of this abstract machine one by one to your real target machine,
which seems to me to be an easier task.

Optimization can occur already in the stages, which are target-independent, but later, too.

That's in my understanding Niklaus Wirth's P-Code approach. Furthermore, there is a file interface between the two stages, so that the first part of the
compiler is very easy to port to new platforms. Only the second part has to be modified (in theory). My port of the McGill compiler (which is a descendent
of Stanford, which is a descendent of the P4 compiler) works this way. In fact,
I only ported it from one IBM opsys to another, so there was no problem at all.
More interesting is, if I will succeed in porting it to an ASCII and Intel based
platform (Windows, Linux, OS/2) and what sort of problems I will see when doing that.

When I did some extensions to the compiler (new statement types like continue, break, return, which didn't exist before), I only had to change the
first part of the compiler; the second one was not affected, because I didn't
need new P-Code instructions.

But maybe the same idea is present in FPC, too. I have to take a closer look
at the tree representation of the FPC units after the first compile stage - when my time allows me to do so. I don't have a real feeling at the moment, if there is more work in the stage before that tree representation or after that -
that is, if the tree representation looks more like the source code or already
more like the linear assembly representation.

Kind regards

Bernd

You miss an important point: an intermediate abstract assembler form exists, but it is never generated explicitly. Please study first the compiler sources before making statements as above. E.g. the sparc specific part of the compiler is approx. 7700 lines only.

ok, so to keep Florian, Jonas and Sven (and others) happy, I would like to tell you that I am deeply impressed by the great work you all have done here.

I appreciate that you discuss those things with me, and I'd like to discuss things a little further, because before investing time here, I would like to know as much as possible about the environment. The last few hours gave me much insight. Thank you for that.

As I understand it now, we have several levels of the compiler:

a) scanning and parsing the source code, which leads to a "tree representation"
of the units, which IMO is a kind of in-storage-representation of the source code
(and tables etc.), without much dependencies of the target, if any. I don't think,
that informations about available registers etc. of the target machine are necessary
at this level.

b) the "tree representation" is translated into a "linear assembly list", which is
target specific; from previous posts it sounds as if there are generic methods
which help with this, and those methods of course need information about the
target platform - but there is no "intermediate language" at this stage like
in the P-Code case. (I know of other compilers, namely the IBM commercial ones, which translate in this stage for an abstract target machine which has
an arbitrary number of registers, and the later "real" code generator puts it
down to the real number, for example 16, and the missing registers are simulated in storage). This needs to be examined more.

c) the "linear assembly list" is written to files, more or less without changes

d) the files are assembled using an external assembler (in our case);
it must be callable from the FPC compiler. There exists an interface for
gas;
interfaces to other assemblers have to be built.

e) in the same way an external linker is used to link the units together.

Is this correct so far?

I'm not sure if and when I will find the time to jump really into this doing real work,
but anyway: if we discuss these things now, it will remain in the archives of the mailing
list and if others (like for example Paul Robinson) read this, they don't have to
discuss it again. So IMO it's useful anyway.

Thank you, kind regards

Bernd

The problem here is that compiler design has moved on a lot since Wirth's day. It's not difficult to write a compiler using e.g. recursive descent or Meta-II which emits instructions for an abstract stack-based machine, and that might be a good match for a CPU with a small number of general-purpose registers. However it can be extremely difficult to optimise this for a modern CPU with a large register file, it's far more effective to give the frontend a rough idea of how many registers the backend has available to it and to warn it about known peculiarities.

Optimization can occur already in the stages, which are target-independent, but later, too.

Agreed, but:

you write:

it's far more effective to give the frontend a rough idea ...

the frontend?

In my understanding the frontend of the compiler is the part that reads the source (scanner, parser) and generates a compact representation of the source - and different tables with identifiers, structure layouts etc.

The commercial IBM compilers even have different frontends - one for PL/1, one for C - but when looking at the internals, it is the same compiler.

The need to know about the properties of the target machine appears later, when it comes to code generation. This is in my opinion what the backend does ...

but we will see ... I guess, the proper limit between frontend and backend and the correct definition will always be subject of discussions.

Kind regards

Bernd

Any comments from me below probably need checking by Florian et al.

As a general point, I think it's worth considering that FPC (as a particular example of a modern compiler) supports a lot of things that weren't in e.g. classic Pascal (as described by Jensen & Wirth) which are implemented at this level rather than necessarily requiring fancy opcode sequences at the lowest levels. Dynamic strings and arrays, reference counting associated with both of these, generics, custom definitions of opcodes, and so on.

I think that register-based compilers are now fairly standard. It's still worth considering other things that the compiler might usefully understand, e.g. the sliding window mechanism used by SPARC (and possibly by the Itanium).

It's possible to get the compiler to output script files to control assembly and linkage, so the compiler and assembler don't /have/ to be on the same system. Also there are already interfaces other than gas, but I think a significant thing is that gas is much more widely implemented than the alternatives.

I noticed comment elsewhere this morning from Paul saying that he'd been working on a new compiler, without any further detail.

See my note on the OS specific/supplied debugger.

Question (to save me digging into the manuals right now): where a recent machine uses the dedicated stack instructions, is the stack pointer one of the standard registers? In other words, can push/pop operations be trivially and exactly simulated for older hardware?

You mean thread safety?

As long as only one thread is running, the push/pop instructions must not be atomic.

Multiple threads introduce many more problems, because their return stacks must never get mixed. Furthermore each thread must have its own stack, again no conflicts.

DoDi

For the purpose of this discussion: the compiler source files in e.g. /usr/local/src/fpc/fpcbuild/fpcsrc/compiler (but not in subdirectories). Note assembler writers in ag*pas, which are somewhat scattered.

The CPU-specific backends are in subdirectories, e.g. ../compiler/sparc, .../compiler/mips and so on.

I wanted to avoid using terms like "portable", "non-machine-specific" and so on because the point has been made that there isn't an entirely clean division.

No, I meant that Bernd suggested R1 earlier as a simulated stack pointer. Does IBM use R1 for this on variants of the architecture that have push/pop opcodes, or some other general-purpose register, or a dedicated register?

As long as only one thread is running, the push/pop instructions must not be atomic.

I hate to correct language usage, but "/need/ not be atomic" would be clearer.

Multiple threads introduce many more problems, because their return stacks must never get mixed. Furthermore each thread must have its own stack, again no conflicts.

Correct. There are only very few exceptions, e.g. on m68k-amiga you can use the "location" keyword to specify a register location for a parameter or the result value (whereby of course you need to know which registers are available).

It looks like this: all nodes are classes that descend from the tnode class. E.g. for adding two values (ordinals or floats) there is the taddnode which descends from tbinarynode which in turn descends from tnode. This taddnode does the general type checking and often also determines whether the result will be passed in a register, as memory reference or whatever. Then there is the tcgaddnode which descends from taddnode and is either able to provide a platform independant implementation (not the case for taddnode) which uses the general interface of the code generator or at least provides a few platform specific helpers and splits the implementation up a bit (in this case tcgaddnode provides methods "second_addfloat", "second_cmpfloat", "second_addordinal", etc. which can be implemented by the platform specific node which is (in case of ARM) called tarmaddnode and is a descendant of tcgaddnode.
This node usually uses the platform specific taicpu to generate the abstract assembler instructions.

This seems a bit abstract at first, but with time you get a feeling for it...

c) the "linear assembly list" is written to files, more or less without changes

Whereby the specified assembler writer is used.

d) the files are assembled using an external assembler (in our case); it must be callable from the FPC compiler. There exists an interface for gas; interfaces to other assemblers have to be built.

gas is only one example. There are interfaces for a few other assemblers as well (mostly on i386, but e.g. on ppc-macos (Classic Mac OS) we use the Apple specific assembler tool)

e) in the same way an external linker is used to link the units together.

Correct.

Regards,
Sven

The tree representation is more like the source code. You can get a feeling for it by calling the compiler with the -vp parameter which will write the tree for the complete compilation (so better use it only in small test programs) to a tree.log file in the current directory.

Regards,
Sven

R1 was only meant as an example.

The true linkage and stack conventions of (most) IBM OSes are like this:

R13 points to the save areas of the current procedure or function (that is the current stack frame of this procedure; at the beginning of this stack frame
there is always a register save area for the 16 general purpose registers, which contain the return adress, entry point and parameter base adress, too).
Following this save area, we have the local (automatic) variables of the current procedure or function. If a parameter list for the call of a subsequent
procedure has to be built, this is also done in a work area which is part of
the stack frame of this function, and before calling the next function, R1 is set to point to the beginning of this area.

R15 always contains the entry point address for the new procedure, and R14 always contains the return address. All 16 registers are saved in the prologue of the new procedure and restored upon return - with the exception of R13 itself, which is handled separately - the save areas are chained together using the 2nd and 3rd word (backward and foreward pointer). This is the way how the contents of R13 are saved.

The are machine instructions to do the register saves and restores in a convenient way - all the registers from R0 to R15 (with the exception of R13) are saved and restored using one machine instruction. It looks like this:

STM R14,R12,12(R13) - that is, R14,R15, and R0 thru R12 are saved at offset 12 from R13.

Kind regards

Bernd

Dunno, sorry. Perhaps a new register has been introduced with the new instructions?

Much appreciated :-)

DoDi

That's obviously far friendlier to a language like Pascal than the examples in most assembler-level treatises- I wonder how compatible it is with the description of the Linux ABI informally at http://linuxvm.org/present/SHARE99/S8139db.pdf ?

I notice that Paul has added updates to
http://wiki.lazarus.freepascal.org/ZSeries over the last week or so but haven't had time to read them in detail yet.

Mark Morgan Lloyd wrote

Bernd Oppolzer wrote: > Am 01.09.2013 21:42, schrieb Mark Morgan Lloyd:

.
.

I notice that Paul has added updates to http://wiki.lazarus.freepascal.org/ZSeries over the last week or so but haven't had time to read them in detail yet.

Looking at Paul's articles in Wiki, I've noticed that he has used 2.6.0 or 2.6.2 as his basis. This doesn't look like a good idea to me, because new platforms are always added in trunk and thus the modifications will need to be updated when merging them to trunk. Moreover, I believe that the latest additions related to UnicodeString support, etc., may be especially useful to z370 when tackling the EBCDIC versus ASCII challenge. Just my 2 cents...

Tomas

Some remarks regarding code generation:

if you have a language like C which doesn't support nested procedure definitions,
it's perfectly simple. You can reach the local (auto) variables using register R13, and
the parameters using R1. You only need another register to access the static data
and some kind of heap management to support malloc() etc., and that's about all.

With "classical" 360 machines, you had the problem that the offsets in the machine
instructions only were 12 bits long, so you could only address 4 k from the position
of a base register directly. That is, if your automatic data (of one procedure) was
larger than 4 k, you were in trouble. Data after the 4 k barrier had to be addressed
using two steps; first compute the address and the fetch the data - for example.

With new z-Arch instructions, this is no problem any more.

Same goes for the procedure code itself; PASCAL/VS (an old IBM compiler of the 1980 years) limited the size of procedures to 8 k - which is some hundred
lines of source code. This, too, is no problem today any more, because, when
you use the new relative branches, you don't need base registers for the code area.

A language like Pascal, which allows the nesting of procedures and the access
of auto variables that are defined outside the local procedure (that is: in procedures
above the local procedure), you need to retrieve the stack frame of that procedure
first. This is done by walking up the chain of the save areas and load the base address
of the stack frame of the interesting procedure into another base register, different
from R13 (for example). This is a problem, that a C compiler doesn't have - but it's
well known to PL/1, too.

This all is very well known and can be derived from the other compilers that deal
with the z-Arch environment, for example the IBM commercial ones.

The z-Arch has evolved very much in the last 10 years, compared to 360 and 370 days,
and this makes code generation and the life of compiler builders much easier.

Kind regards

Bernd

"Bernd Oppolzer" wrote

With "classical" 360 machines, you had the problem that the offsets in

the machine

instructions only were 12 bits long, so you could only address 4 k from

the position

of a base register directly. That is, if your automatic data (of one

procedure) was

larger than 4 k, you were in trouble. Data after the 4 k barrier had to

be addressed

using two steps; first compute the address and the fetch the data - for

example.

With new z-Arch instructions, this is no problem any more.

FPC already supports some CPUs which have such restrictions as well. It's no real problem to split up single instruction branches/moves to do a calculation of an address before branching/moving. I'm doing this for example for the Coldfire m68k variant.

A language like Pascal, which allows the nesting of procedures and the

access

of auto variables that are defined outside the local procedure (that is:

in procedures

above the local procedure), you need to retrieve the stack frame of that

procedure

first. This is done by walking up the chain of the save areas and load

the base address

of the stack frame of the interesting procedure into another base

register, different

from R13 (for example). This is a problem, that a C compiler doesn't have

- but it's

well known to PL/1, too.

In FPC we AFAIK pass the parent frame to a nested function as an additional parameter.

Regards,
Sven

I believe this sort of thing is also an issue when a function recurses.

Anything that artificially limits function size, or forces the code generator to jump through hoops to work around architecture issues, is a problem /particularly/ when an external tool has automatically generated (Pascal) source. I'm not sure how many people use that technique these days but it was fairly popular in the era when Fortran was dominant, and there are still tools that generate C or Java source.

[Nod] I believe that most of the above- as well as things like IEEE floating point support- is in late-model 390s and in the Hercules emulator.

On the other hand, since he was in the position of trying to work out how everything hung together without feeling that he was already "an insider" I think that he might be forgiven for working on a stable version. This is, after all, how the MIPS target was implemented (it's regrettable that it wasn't folded back into the trunk a bit more promptly, which might have saved work).

I've just been reading through his recent additions to the wiki and I note that he recognises that (comparatively) recent additions to the architecture might be useful. If he's actually making progress on this it would be very interesting to have a status report.

C supports recursion of functions, and it is without problems possible with the scheme outlined above
and in the previous mails.

Agreed. But although the limit of 8 k per procedure seems strange from todays point of view, there were significant projects done with this compiler, for example the first TCP/IP stack for the 370 platform in the early 1980s (inside IBM, in Pascal !!).

I still have lots of tools generating source code, for example PL/1 and C. XML validator definitions, derived from XSDs (to speed up my XML parser) - well, that's definitions, not executable code, but ocassionnally very large, if the
XSD is large - and database access routines ...

yes !!