Episode 4.  Addressing and it's limits Part Two

So we have seen that Base / Displacement will handle addresses up to a reasonable size.  This should cover most of the requirements.  But there are still issues that have to be resolved, we need the ability to address more.  The way we do this is with Index registers.  We get an advantage of unlimited addressing using index registers but we pay a price that we need to generate more code.

PROG     CSECT              defines the name of our function         £START             set up standard linkage         LR    R12,R15      R15 has the address of PROG, copy it to R12         USING PROG,R12     Tell the assembler to use R12 as base         B     LABELB       we want to branch to the exit here           <61440 bytes of code goes here>
LABELB   EQU   *          
        £END               return to caller SAVEAREA DS    18F          save area for standard linkage           <working storage goes here>
LITPOOL  LTORG
          <constants get defined here>
        END

Taking the simple example we had before, I have added a B (branch or jump to a label) instruction at the beginning.  It needs to branch over the arbitrarily large amount of code between itself and the label, but it is more than 4k and we don't have a base register.
        LA    R1,(LABELB-PROG)/4096
        SLL   R1,12
        B     LABELA-((LABELA-PROG)/4096)*4096(R1) Explanations first.  Instruction 1 splits the code into notional 4k chunks and determines which of these chunks holds the label we want.  It places that value into register 1.  In our case 61440 / 4096 = 15 (/ is integer division, div in Pascal), so the first instruction loads 15 into R1.  Instruction 2 multiplies R1 by 4096 (Shift Left Logical - 12 bit positions).  R1 now holds the notional address of the  4k page offset. Instruction 3 is just calculating Address mod 4096 in pascal terms, the remainder from division by 4096.  Looks ghastly doesn't it. We wouldn't do it like this in real life, it would be encapsulated into a macro. So we would call something like $BLONG  LABELB.  The macro would generate
        LA    R1,15
        SLL   R1,2
        B     PROG+0x022(R1)

The R1 specified on the end is the index register.  What happens is that the cpu  calculates the address as the base register plus the index register plus the offset.  We now have the methodology for infinite (well, as infinite as storage allows) branches.

This complication would need to be considered for other instructions and for constants too. And not all instructions handle index registers, so we would have to point temporary all of this is addressable (sorry for the pun).

Regardless of what you may believe, FreePascal is not the first compiler to be implemented on 370 architecture.  Should I tell tell their developers that 370 architecture is too much like a dinosaur to write a 32 bit compiler.  IBM had 32 bit compilers available in the 1960's.  Should I tell them that the architecture is "broken". It's been around for 50 years and there are hundreds of compilers available for it.  From FORTRAN to GCC, from COBOL to ADA or from PASCAL/VS to APL.  All of these were (for the ones that were available from the late 60's) 32 bit compilers.

Regardless of what you may believe, FreePascal is not the first compiler to be implemented on 370 architecture.

I.e. you can study how they create code, before you reinvent the wheel :-)

DoDi

:-) Had occurred to me as well, although obviously a lot is dictated by FPC's higher-level parsing and structure. Might still be very useful for getting calling conventions etc. right.

I've seen comment that VS/Pascal was used by IBM to write at least some system software, and then there is obviously P4 Pascal which has been ported to at least some of the readily-available S/3x0 operating systems.

This should not be a problem. It's up to the code generator to define its own strategies for allocating space for code and data items, and how to address them in generated code. The parse tree tells the back-end *what* should be done, the back-end then has to know *how* to do it.

Might still be very useful for getting calling conventions etc. right.

The implementation of calling conventions is up to the code generator as well. You can invent your own conventions, and map them to any predefined one. As a starting point it's sufficient to support only one calling convention, the one for system calls. More conventions for calling other external libraries may be required, depending on the target OS conventions.

DoDi

Hans-Peter Diettrich wrote the following on 08/02/12 11:28:49:

Where can I find details of the input to the back-end? I'd like to have a crack at generating assembler code for VM/SP.

Am 08.02.2012 15:16, schrieb

Best (and AFAIK only) available source: the compiler's source code. Sorry.

Regards,
Sven

Sven Barth wrote the following on 08/02/12 14:24:17:

Ouch!! OK, thanks.

Bob

I think that that's one of the things that Paul Robinson was trying to write up: the overall collection of files and how they related to each other. It's also something that I started working through with regards to the MIPS backend, and I suspect that there might have been others.

My understanding is that as far as the compiler is concerned almost of the CPU-specific stuff is in e.g. ./compiler/mips. There are obviously other things e.g. in the makefiles that glue a particular target into the overall framework, but once those are in place the developer(s) working on a particular target are nicely fenced off (for their own protection, as much as everybody else's :-)

One thing that could be useful: is it possible to build a compiler with a dummy backend? In other words, such that it can emit all of the node information (fpc's -an option) even though it doesn't generate any code?

Where can I find details of the input to the back-end?

The node tree contains the parsed instructions, and their argument descriptors should refer to the related variables and types (see node.pas for an start). You'll have to implement the code generation at least for the basic node types (assignment, operators, calls), see cgbase.pas.

Hint: the first few characters of the unit names describe the type: aasm - assembler (aasmtai.pas)
cg - code generation (cgobj.pas)
n - nodes (node.pas)
p - parser

DoDi

Maybe I've misunderstood the logic here, but it's my understanding that the back-end produces assembler statements to produce executable code for the processor.  So if I want to move something from field a to field b say, or add two numbers together, the backend generated the MVC and it's related code or the Add instruction, Handles registers and addressability and stuff.  Anyway, whatever this stuff is called it lives in the COMPILER/VMSP (apologies to you case-sensitive types) sub-directory.

The bit that provides code for VM/SP as opposed to say MVS, is the RTL, so if I want to open a file, there is a routine in the RTL called OpenFile (or whatever) that issues the VM code for open.  This lives in System.pp or Sysfile.pp or whatever it's called in the RTL/VMSP sub-directory

Now, in order for the compiler to function it needs access to a certain level of base functionality in the VM/SP RTL.  Things like Allocate / Deallocate memory, Open and Close a file, Read and Write from and to a file.  Create and Delete a file etc.  These may, or may not, be written totally or partially in assembler.

The RTL is compiled before the compiler, so that it can be used inside the compiler. This is required for cross-compilation, where the compiler has to use the file access routines of the *host* system, and is compiled into code for the host system.

The RTL may contain assembler code for the target machine, if you don't trust your own compiler to generate appropriate code.

DoDi

I've added example FPC output at
http://wiki.lazarus.freepascal.org/ZSeries#Digression:_example_FPC_output which might be useful to the current discussion. I've used David Zhang's MIPS compiler to do this, since the register etc. model is probably closer to zSeries than any of the other v2 compilers.

Note that to get this level of detail the compiler has to be built with -dEXTDEBUG in order to enable the -an option, despite the fact that this always appears in fpc's -h output. If it were my choice I'd make that bit of help output conditional.

Mark Morgan Lloyd wrote

steve smithers wrote: >> wrote on Wed, 8 Feb 2012 14:16:50 +0000 (GMT)

.
.

Note that to get this level of detail the compiler has to be built with -dEXTDEBUG in order to enable the -an option, despite the fact that this always appears in fpc's -h output. If it were my choice I'd make that bit of help output conditional.

Making it conditional isn't possible without modifying the structure used in the message files (plain text files to allow easy translation / localization to other languages) or their concept / the way how they are used.

As it works now, the default (English) text file is translated into an include file containing all texts and compiled into the compiler executable (but another file with the same structure but written in a different language may be supplied on the command line and override the default on run-time). The only conditional parts in the message files are related to platforms and targets (which is a limited set of dependencies by definition); -dEXTDEBUG based dependency doesn't fit into that. The easier solution would be probably adding a note about this dependency directly into the help page (I or anybody else in the core team can do that if it really works that way - i.e., if there is no difference between -an and -a in a compiler compiled without -dEXTDEBUG).

Tomas

Thanks for the expiation :-) Would it be possible to do something like (a) annotating non-standard options with e.g. "(requires EXTDEBUG)" and (b) having a line in the banner showing what non-default options were used during build?

Free Pascal Compiler version 2.7.1 [2012/02/06] for mipsel Copyright (c) 1993-2011 by Florian Klaempfl and others Built with EXTDEBUG, fvm32.
/usr/local/src/fpc/fpc-trunk/compiler/ppcXmipsel [options] <inputfile> [options]
Put + after a boolean switch option to enable it, - to disable it

As a slight aside, I've got a shell script for Linux/Solaris which saves this sort of thing and also generates a cross-reference of all files by parsing -vt output. I've considered doing something like putting it on lazarus-ccr but don't relish handling any "why doesn't it work on Windows?" questions.

Linux pye-dev-07 2.6.38-custom #2 SMP Thu Jun 9 11:07:18 GMT 2011 i686 GNU/Linux
Using Free Pascal Compiler version 2.7.1 [2012/02/04] for i386 make NOGDB=1 OPT='-O- -gl -vt' all |/usr/local/bin/fpc-filter-vt

= /usr/local/src/fpc/fpc-trunk/rtl/linux/system.pp
+ /lib/ld-linux.so.2

I feel I have to respond to this after a couple of things I've read over the last day or so. I for one have never attempted to belittle "big iron", since it has always seemed clear to me that that type of kit has its uses: if nothing else then to do things like running the name and certificate servers that keep distributed systems going. It's also worth noting that IBM and Burroughs did engage in controlled decentralisation quite early, putting a significant amount of "smarts" in their terminals well in advance of anything done by their "trendier" competitors such as DEC.

In the current case I was relying on the precedent set by the GCC porters and the Linux maintainers to say "OK, we need to have some policy to determine what vintage of hardware is supported". However noting the availability of old IBM operating systems and the interest people have in running them, and in particular noting the amount of work being put into the OS/380 project, I'm fairly rapidly coming to the conclusion that the S/370 is worth supporting, even if we brush the S/360 under the carpet.

However I'm disturbed by comments like this from the mainflame brigade:

"...strong suggestions that IBM should provide their own EBCDIC-based operating system and integrated-circuit microprocessor chip for use in the IBM Personal Computer as a CICS intelligent terminal (instead of the incompatible ASCII-based Intel chip, and immature Microsoft 1980 DOS)."

and

"Also, the standard character set on the 360/370/Z-System is EBCDIC, while the Pentium uses ASCII."

If the community can't get its head around the idea that character encoding is much more an operating system than a hardware issue, that the Intel/AMD range of processors could happily run an EBCDIC-based operating system, and that IBM gleefully supports ASCII-based Linux and ASCII-based Internet services then it's going to be damn difficult to get this (sub)project off the ground.

Mark Morgan Lloyd wrote the following on 09/02/12 14:08:24:

IBM earned over $15 billion last year from the sale of mainframes.

To an application programmer there is (was?) little difference between 360 and 370.

I'm puzzled by this whole idea of Free Pascal supporting 360/370. Who is it aimed at? Who needs it?

Yes, this is what I suggested to do above; for -an, not in general, because I don't know of such dependencies myself (I wasn't aware of it for -an either).

I'm afraid that this is "a bit" more difficult _if_ we want a general solution. Addressing individual options explicitly is possible (probably line by line rather than as a list though). Obviously, this wouldn't be a general solution. However, I'm not aware of any compiler macro allowing to list all conditional defines (that would be still the easier part, because these are obviously known within the compiler so adding a new macro should be possible, but it may be a long list), and even less a macro allowing to list just compiler defines added explicitly on the command line (rather than defined internally for a particular target, implied from some other command line options or defined within the respective source file)... I'd wait for opinion of other core team members whether we should add support for explicit "Built with EXTDEBUG" or do something else (but I'm certainly not the one who'd add a macro necessary for the general solution).

Tomas

Mark Morgan Lloyd wrote

steve smithers wrote:

.
.

Just a comment on this: While I understand your statement and the Linux port obviously confirms that an ASCII based operating system is possible on S370 too, I wouldn't consider the character set being so completely independent from the underlying hardware (all IBM PC compatible graphic adapters can show ASCII characters directly but not EBCDIC, and also Intel CPU instruction set includes support for BCD arithmetics based on the ASCII character set if I understand it correctly).

Tomas

I agree: when taking terminals and- in the general case- other I/O devices into account. But both the examples I gave- one from Wikipaedia and the other from Wikibooks- specifically associated ASCII with the CPU type ("Intel" in one case, "Pentium" in the other) and I really don't think that's healthy.

I was just checking the BCD arithmetic situation a few minutes ago, and I believe that there's a "half carry" flag for carry out of the first four bits: and that will work equally well for both ASCII and EBCDIC. I might have come across other cases, but I can't remember whether they apply to x86.

I've worked for a pretty unpleasant mainframe outfit, and am very much used to the way that that type of corporation attempts to rewrite history to their advantage and forces their position on their employees and customers. "The company that invented virtual memory and the virtual machine" to which I added "and the intermittent fault"... didn't last very long, but they had some interesting kit.

I suppose another possibility would be to have something in the makefile that captured the shell/environment variables, in the same way that the Lazarus build captures the revision number if available. But that's not in the same league as putting useful info in the banner.

I think a good starting point would be a line that showed any definitions that were known to have an effect on the interpretation of the options, e.g. EXTDEBUG and- in particular- the exact target CPU when this wasn't the default.

I never intended to imply you did.  If I failed in this, I apolgise.  However these comments were made by someone.  If you look back at the posts from january you can see them for yourself.  It was these posts and their tone that made me write this mammoth rebuttal.  If I'd known how much time that it would consume, I probably wouldn't have started!

...  since it has always seemed clear to me that that type of kit has its uses: if nothing else then to do things like running the name and certificate servers that keep distributed systems going.

You might not intend to belittle big-iron, but it is statements such as this that do just that.  There are millions lines of mainframe code out there written in Cobol or Fortran or, yes, even Assembler that continue to form the backbone of production systems of most of the big banks and financial institions, cashpoint networks, supermarket admin systems, travel companies, sales and administration systems and a host of others.  Not to mention the government!

In the current case I was relying on the precedent set by the GCC porters and the Linux maintainers to say "OK, we need to have some policy to determine what vintage of hardware is supported".

True.  But as I pointed out these systems are produced by IBM.   They may be produced for "the open source community", but the bottom line is that they are there to sell tin.  IBM have never really got the idea that there is more to being a computer company than selling boxes.  It is not in their interest to support old processors.

I don't care which end it comes from, it is nobodies business what IBM produce but IBM and their customers.  Similarly, I don't give a tinkers about whether you run MAC OS or Windows or Linux or even Solaris.  It's nobodies business but yours.  The same applies to me.

steve smithers wrote the following on 09/02/12 18:47:03:

 IBM have never really got the idea that there is more to being a computer company than selling boxes.

That might have been true 20-30 years ago but certainly not now! Hardware accounted for less than 20% of revenue last year. The big earner is services.

What about this (using an example application instead of the compiler):

=== source begin ===

program envtest;
const
   crossopt = {$include %CROSSOPT%};
   opt = {$include %OPT%};
begin
   Writeln('CROSSOPT: ', crossopt);
   Writeln('OPT: ', opt);
end.

=== source end ===

The Makefile used below just calls "fpc envtest.pp" for "all".

=== shell begin ===

oneshots]% make all OPT="-dEXTDEBUG" CROSSOPT="-CfSSE2" fpc envtest.pp
Free Pascal Compiler version 2.6.0 [2011/12/23] for i386 Copyright (c) 1993-2011 by Florian Klaempfl and others Target OS: Linux for i386
Compiling envtest.pp
Linking envtest
/usr/bin/ld: warning: link.res contains output sections; did you forget -T? 10 lines compiled, 0.5 sec
oneshots]% ./envtest
CROSSOPT: -CfSSE2
OPT: -dEXTDEBUG

=== shell end ===

For the explanation: make moves all variables into the environment which is inherited by the processes that are called by it. FPC can include the values of a environment variable at compile time (it will be '' if not set, though a warning will occur which needs to be disabled with $warn) and thus at least the values of OPT and CROSSOPT could be printed by the compiler (which should normally be sufficient...).

Regards,
Sven

The Intel x86 (and thus also Pentium) opcodes like "AAD" ("ASCII adjust before division"), etc., suggest some dependency, but I admit that I didn't try to analyze what results it would have with EBCDIC.

Tomas

One thing I'd caution about the environment is that Debian- and probably others- predefines a significant number of shell functions, which shows up in the output of the set command.

If OPT or CROSSOPT is set in the environment already you might have problems with compiling FPC at all (because it might contain incompatible options) no matter whether we use it for output or not...

Regards,
Sven

The Intel x86 (and thus also Pentium) opcodes like "AAD" ("ASCII adjust before division"), etc., suggest some dependency, but I admit that I didn't try to analyze what results it would have with EBCDIC.

BCD values are stored as "nibbles" (4 bit binary values), which are independent from character codes. A nibble happens to represent the low 4 bits of the digits, in both ASCII and EBCDIC.

DoDi

In any event, I'd not expect somebody who'd spent their career focussed on mainframes to be sufficiently familiar with x86 assembler to be able to make informed criticism at this level of detail. Similarly I'd not expect somebody who'd spent their career focussed on PCs to be able to make informed criticism of mainframe I/O architecture and the claimed throughput levels.

As I've already said, there's a lot of disinformation been disseminated, and I think it behooves both sides to admit that both mainframes and PCs (and their derivatives, which appear to include everything from large servers down to smartphones) have their place and aren't going away.

Sorry Bob, I find that some of your postings aren't getting past our gateway, so I only see them in the archive.

To answer your question: I'm not entirely sure, but I suspect anybody who wants to run on any of the freely-available "classic" operating systems /particularly/ VM where the available version is VM/370 which could be expected to have problems if asked to support virtualised guests including S/390-specific opcodes.

My initial suggestion was to support anything from the S/390 G5 onwards, i.e. the same as Linux and (at least approximately) GCC. Paul's gone very quiet, Steve is making a fairly good case that there are sufficient well-understood workarounds that supporting S/370 if not S/360 should be possible, so I for one am happy to go with the flow on the target CPU (even if I express a strong preference for Linux and ASCII as the initial operating system targets).

I'm afraid that you're missing my point, which is that devotees of PCs (and before that of minis) would argue that all the examples you give above could "easily" be moved away from mainframes onto their favoured platform. In some cases they might be right- provided that you are prepared to ignore the cost and difficulty of doing the port and testing the result. In others, such as the examples I gave, it is quite simply not possible: some sort of centralised system is mandated by what you're trying to do. So my examples were not trying to belittle the platform, but were intended to be cases that the PC brigade quite simply couldn't argue against.

Now let's stop arguing at cross purposes about whether the platform is still used and what governs things like character set selection, and get on with trying to work out how to do the job.

Mark Morgan Lloyd wrote the following on 10/02/12 19:04:55:

Now let's stop arguing at cross purposes about whether the platform is still used and what governs things like character set selection, and get on with trying to work out how to do the job.

I'm pretty sure I know how to do it, I just don't see why it's necessary ;-) But I'm willing to have a go, just for fun, which is why I'm trying to find out what is fed into the "back-end".

rvmartin2 wrote on Fri Feb 10 20:55:27 CET 2012

How about for Linux/390 support.  Do we want to condemn them to C?

:-) I'm a very long way from being a core programmer, but my understanding is that FPC doesn't use a serialised intermediate format which the backend translates. Rather, it constructs one or more internal trees of objects which are walked, at least some of those objects translate into assembler sequences. In other words,

procedure secondpass(var p : tnode);
begin
   p.pass_generate_code;
end;

which is called from... frankly, all over the place, and which via a couple of not-very-thick layers tells subclasses of tnode in the processor-specific directories to emit code.

About the best I can do at the moment is
http://wiki.lazarus.freepascal.org/ZSeries#Digression:_example_FPC_output which can, initially, be read in conjunction with ./compiler/pass_2.pas etc. as above. However that is still rather incomplete, in particular it doesn't show enough of the objects' state to reveal to the casual reader why some sequences generate code and others don't, and it doesn't really describe the logic behind emission of the data sections in the latter part of the listing.

To let Mark (and Sven) know - I tried to add support for this to the compiler but I couldn't make it to work for OPT. The approach is OK in general (apart from the limitation that the compiler may also be compiled without Makefiles and potential use of whatever options wouldn't be visible in that case, but that scenario is probably not important). However, our Makefiles process these variables in very complex way and the result is that OPT is always empty during the compiler compilation (probably due to line 360 in compiler/Makefile). I thought that LOCALOPT might be used instead (see line 355) but that didn't work either. See the attachment (it's the stage when I wanted to try printing LOCALOPT too). Even if I use the original sample program from Sven and rename it to pp.pas and replace the compiler source file with it (and fix the warnings which are treated as errors due to -Sew added by the Makefile ;-) ), I only get CROSSOPT printed and OPT showed as empty.

So I ended up by only adding the note to the help page regarding -an dependency on -dEXTDEBUG...

Tomas

Well... thanks for trying anyways ;)

Regards,
Sven

Definitely, and thanks for tweaking the help text :-)