From: Paul Robinson

As has probably been a bit of a scare to some people, I've more-or-less "unofficially announced" by putting up an article on the Free Pascal Wiki regarding how I've decided I am going to work on porting the Free Pascal compiler to the IBM 370 series mainframe, otherwise known as the zSeries.  And that, having spoken to a few people here, I've been made aware this has sort of been scary or incredible to some of the other people working on this project because they realize that there is a hell of a lot of work involved and I may not be aware of what I'm up against.  I can understand, and I can sympathize with their thoughts. 

I want to do this because I want to get involved in something I can do, because I am familiar with IBM mainframe assembler, I'm not really that familiar with Intel/AMD assembly and because (for reasons I might go into later) I think it needs to be done, I can do this and I've decided I want to.  There are also a couple of other things I want to get involved in doing which will be a significant improvement to Free Pascal even if this does not get done; I'll post that in a separate message.

So in response to those who are incredulous at my audacity or think I'm not aware of what's involved, I'll include a statement I've added to the Wiki about my efforts to do the port / cross-compiler for the IBM 370 (zSeries), which is at http://wiki.freepascal.org/ZSeries/Part_1

- Start Story -

I am aware, first, that this will be a huge undertaking; this is not a weekend project.  I'm probably looking at a minimum six months work, possibly longer.  I have a story.  When I wasn't doing programming I did mobile notary.  I was, then, a commissioned Notary Public for the Commonwealth of Virginia (I'm now commissioned in Virginia and Maryland), and one of the things that was developed was, when people refinanced their mortgages, the company would send a notary right to the person's home or office.  Well, one of the customers happened to discuss how they wanted to upgrade the application that their people ran to handle booking of services.  It was a DOS application, they wanted to do more things with it and they wanted it to be a GUI under windows.  At that time Free Pascal did not have the equivalent of Lazarus, so I unfortunately had to use Visual Basic to do so.  (Please do not send me hate mail, I have to use what I have available.)

Well, anyway, so I had several meetings with the customer and arranged terms including price.  I worked on it on a regular basis, implemented it through "accretion" in which you get part of it working and they can see how it's coming along.  We had to change things along the way and various fits and starts, but in the end, they were very pleased with the program that they can use for their employees who book orders on their laptop computers.  It wasn't a very big program in terms of what was going on, they have it running on about 60 computers, but from when we first sat down to decide what they wanted, until it was fully running and distributed out to everyone took one solid year. 

So the point of this shaggy dog story is that I'm aware that this is a long-term project and will probably take months.  I am also aware that the compiler won't work when I first make changes because I have to learn where the main part of the compiler hands off control to the machine-specific and OS-specific routines so that it can ask that routine, "Hey, the user is implementing a FOR statement, you need to create the code to do this." or "Hey, the guy is declaring the start of a procedure that has an integer argument, here's the information on how it's supposed to be defined."

So anyway, I know there's a lot involved, there will be fits, and starts, and things won't always go right.  But it's a learning experience, and, if you follow this as it goes along over the months as it progresses, maybe you'll learn something too.

- End of Story -

 
The Lessons of history teach us - if they teach us anything - that no one learns the lessons that history teaches us. The following is to allow readers of the listing digest to know where this article ends. -- End of Message by Paul Robinson --

AFAI learned:
I suppose the code generator should be doable, regarding that there already are several supported CPUs. At least a working compiler might come into existence in a decent amount of time, adding optimizations is another project.

OTOH I suppose that a porting the RTL to a mainframe OS will not be easy and without this the compiler is quite useless.

Is there a Java virtual machine for zSystem ?

Maybe using the currently evolving FPC for Java target might be another way to go.

-Michael

I do not think it is more difficult than any other OS.

Michael.

I've got reservations because of the EBCDIC issue. My own opinion is that this effort would be far better targeting Linux on a comparatively recent model of machine, since these are well-supported by the Hercules emulator which gives developers an absolutely standard working environment: which is, obviously, why I wrote it up a few months ago.

I'm sure Paul has his reasons that we all need to respect, but I don't think that using a 370 emulator (i.e. at least two architectures out of date) plus an obsolete operating system is the place to start.

I admit to having considered this as a long-term project, but what I don't have is a good understanding of the architecture's assembler-level programming. However I think I need to make the general point here that "IBM Assembler" is understood in the industry to encompass that which is needed to write application-level programs in CISC assembler (plus macro support etc.), and that even somebody skilled in that particular black art might need to enter unfamiliar territory when it comes to the operations required by a compiler's code generation (e.g. commercial programming doesn't do large amounts of stack-relative addressing).

...except for the EBCDIC stuff, because the common parts of our RTL assume ASCII in many places (most of them probably not that difficult to fix by adding some platform specific constants changing the behaviour from ASCII only to consider EBCDIC too, but scattered around many places and thus difficult to find). That doesn't mean it shouldn't be doable, of course, it will just require debugging even parts which didn't have to be touched during ports to other operating systems.

Still, starting with the Linux for S/390 (or are S/390 and S/370 completely different animals?) as suggested by Mark would be probably a very good idea to avoid having to tackle both a different HW platform and a different OS at the same time (the Linux port to S/390 apparently uses ASCII according to what I found on Internet). Obviously, the OS may follow once the new HW platform is in working state.

Tomas

Tomas Hajny wrote

Still, starting with the Linux for S/390...

A nice thing being that - AFAI read - the more recent versions of the OS allow for starting multiple virtualized Linux instances.

-Michael

I used a macro processor called Stage2 for years which was completely character-set agnostic, assuming only that 1..9 were contiguous and in sequence and putting them (plus a small number of other special characters) on the first line of any file; I was looking at an APL implementation a few weeks ago which did something similar. Thirty years ago it might have been worth writing an RTL like that, but these days ASCII and UTF-8 are so utterly dominant that I see little point.

Still, starting with the Linux for S/390 (or are S/390 and S/370 completely different animals?) as suggested by Mark would be probably a

The S/390 is an incremental change from the S/370, in the same way that zSeries is an incremental change from S/390. However in any one of those architectures there is a significant number of models, and- crucially- there is a change at one model of the 390 which allows it to accept inline literals rather than insisting that these be in a table which is within 4K of the current PC. As I understand it, GCC doesn't support models that predate that change, and frankly I don't see why FPC should either; however I don't know the model's formal designation or how to check for it in RTL code.

very good idea to avoid having to tackle both a different HW platform and a different OS at the same time (the Linux port to S/390 apparently uses ASCII according to what I found on Internet). Obviously, the OS may follow once the new HW platform is in working state.

In my experience, Linux running on an (emulated) S/390 looks almost exactly like Linux running on any other CPU. There are no character set issues, and no obvious issues relating to endianness or floating-point format (I've had no need to examine FP binary representation, but I think that these days the machines support IEEE).

I'd also point out that IBM have a developer relations program, which makes a (Red Hat?) VM available to anybody who needs it. I'm sure this project would qualify for that.

2012/1/18 Tomas Hajny :

...except for the EBCDIC stuff, because the common parts of our RTL assume ASCII in many places

If it was me implementing I would simply convert between both types everywhere necessary and leave the RTL using ASCII.

? It just means you must convert ascii to ebcdic in OS calls that require strings. All these calls must be re-implemented anyway, so a generic routine to do this conversion seems like the obvious path. I doubt this will be the real bottleneck :-)

As for using Linux: I have no idea how this works in practice, so I can't comment on that. I know that people here use Hercules inside linux to develop for zSeries, but then again running a linux inside that seems a bit like overkill :-)

Michael.

As for using Linux: I have no idea how this works in practice, so I can't comment on that. I know that people here use Hercules inside linux to develop for zSeries, but then again running a linux inside that seems a bit like overkill :-)

It's trivial and performs well. See the timing figures I put up at http://wiki.lazarus.freepascal.org/Qemu_and_other_emulators#Relative_Performance

Paul Robinson wrote

From: Paul Robinson As has probably been a bit of a scare to some people, I've more-or-less "unofficially announced" by putting up an article on the Free Pascal Wiki regarding how I've decided I am going to work on porting the Free Pascal compiler to the IBM 370 series mainframe, otherwise known as the zSeries. And that, having spoken to a few people here, I've been made aware this has sort of been scary or incredible to some of the other people working on this project because they realize that there is a hell of a lot of work involved and I may not be aware of what I'm up against. I can understand, and I can sympathize with their thoughts.

It's not that I'm scared that you want to tackle this it's more that I've seen that you based your How-To on assumptions that are wrong or at least "not good" and wanted to help you in this regard.

I am aware, first, that this will be a huge undertaking; this is not a weekend project. I'm probably looking at a minimum six months work, possibly longer.

I have not yet started a new CPU target myself (only an OS target), but I can imagine that you're not that far off with the time needed...

So anyway, I know there's a lot involved, there will be fits, and starts, and things won't always go right. But it's a learning experience, and, if you follow this as it goes along over the months as it progresses, maybe you'll learn something too.

Well, then let's get started with bringing you on track:

1.

You wrote that you use the source code of FPC 2.6.0 and copied the "compiler" and "rtl" directories to some other "workplace". I suggest you to use a SVN checkout of the development version (you can find the links here: http://www.freepascal.org/develop.var (at "Connect to Source Repository with SVN")), because than you can more easily revert back something. Also there were quite a few changes from 2.6.0 to 2.7.1 and thus it's better to work with the most recent source.

2.

I suggest you to use the Lazarus IDE to do compiler development (you might use the text mode IDE ("fp") as well, but I myself only use Lazarus for that). You can then copy one of the project files (*.lpi) in the compiler directory (e.g. ppi386.lpi) and paste it as "ppi370.lpi". You can then open this project in Lazarus and adjust the defines (in tab "Other" replace "-dI386" with "-dI370") and output directory (exchange "i386/units" with "i370/units") in the project settings. Afterwards you should be able to compile the compiler using (Ctrl+)F9 (ok, you won't at first, because you yet need to adjust the source for i370)

3.

I noticed that you wanted to add a unit "i_i370" to the "compiler" unit and to the directory "systems". This is wrong. The units listed in the "compiler" unit are the target operating systems (e.g. Windows, Linux, DOS), not the underlying architecture. Thus you'd only add a "i_osvs1" (if I remember the name correctly) to that unit and directory, but you don't need to do this know as you first need to implement the architecture support.
This architecture support is located in the corresponding CPU unit (e.g. ARM, PowerPC, i386, x86_64 (the latter two share the x86 directory as well)) and is included by using the "cputarg" in the unit "compiler". This unit is located in the corresponding CPU directory. Thus I'd suggest you to copy the directory of an architecture (e.g. MIPS as this is a smaller and younger one) and name that "i370" (now the unit output path I mentioned in "2" will make sense ;) ).

4.

I read in your How-To that you stumpled upon units like "sysutils" and stubbed them. You don't need to care about these yet. These are only needed if you compile programs for the target platform which use that units. The only unit you might want to stub at the beginning is the "System" unit which is located in $fpcdir/rtl/$target/system.pp (where $fpcdir is your source directory and $target is one of the supported targets (e.g. linux, win32, etc.)). The most important part you'll need there is a function or procedure (depending on the operating system) which will contain your real entry point (the "begin...end." in a "program" file is NOT the real entry point, this part is called "PASCALMAIN" and is called from within the real entry point). So at the beginning you can mostly concentrate on your "i370" directory with adding some types here and there (like the assemblertype that you already added).

I myself can't help with porting FPC to a new architecture, as I haven't done that yet (only to a new operating system target). But AFAIK Mark Morgan Lloyd worked on the MIPS target and of course there are compiler developers who implemented new architectures.

Regarding general compiler questions and those regarding the implementation of a new OS target I might be able to help you, so just mail to the list and I'll try to see what I can do.

BTW: It might indeed be the best - as suggested by others already - to first target a Linux system on your architecture as then you can split the work into "implementing the architecture support" and "implementing the OS support", because after you've successfully implemented the architecture support you can of course add other OS targets to your architecture as well.

Regards,
Sven

It should not be a bottleneck, but I'm afraid that you underestimate it a bit. As an example, searching for #10 and #9 across files (just) in rtl/inc (there's much more in rtl/objpas) shows quite a few places which would need to be changed for EBCDIC support and which are not touched otherwise during a RTL port (control characters have completely different layout in EBCDIC compared to ASCII). Also case insensitive search for "'a'" (just as an example - there are more ways how this can appear in the code) finds several places containing code assuming either certain position of the standard alphabet ('a'..'z') in the character set - both assumptions regarding the absolute value of 'a' (or 'A') used e.g. for translation of hexadecimal numbers, or assumptions about the whole alphabet being in one consecutive range (which is not the case for EBCDIC).

Sorry, I don't understand this comment (probably because I know nothing about Hercules). My point was simply that Linux port to S/390 exists so one could start by creating support for the S/390 platform and testing this new platform support with the existing Linux RTL first before continuing with the RTL for zSeries (including necessary adaptations of the common parts of our standard RTL).

Tomas

Would you like to bet on that? What if it also meant that all the sources had to be rendered unto EBCDIC, and digraphs defined for characters that don't exist such as ^[]

But then you are assuming the RTL should be using EBCDIC internally as well ? Obviously, that will be a lot more work.

But I don't think this should be so.

Michael.

Also there are things in trunk that aren't in released versions, for example when I started looking at MIPS I didn't appreciate that somebody (Florian?) had already backported much of David Zhang's (?) implementation.

I definitely found Lazarus indispensable once I started stepping through part-working code to find out where it was going wrong, I was able to run MIPS and PPC side-by-side on a PC.

Perhaps now would be a good time for a core developer to add appropriate definitions to trunk, so that anybody working on the IBM port only needs to modify files in a directory off ./compiler.

MIPS is OK, but there are some defines etc. that need to be sorted out. I've done a bit of work on this but I think it needs somebody with more experience to sort out my fixes and commit them.

I myself can't help with porting FPC to a new architecture, as I haven't done that yet (only to a new operating system target). But AFAIK Mark Morgan Lloyd worked on the MIPS target and of course there are compiler developers who implemented new architectures.

Bearing in mind that what I was looking at was code that had already been working in version 1 of the compiler, which had been massaged into some sort of shape for v2. I can claim very little credit, it was entirely "grunt work". More relevant might be Jonas's recent work targeting the JVM, but I suspect that the "classic" IBM mainframe architecture is probably nearer MIPS or 68K than to JVM or a modern RISC.

Historical note: I worked for a mainframe manufacturer in the early 80s after getting my first degree, so while I'm not intimate with the S/360...zSeries range I think I've got a bit of relevant experience. The important thing to appreciate is that at least some mainframe applications were written in assembler up until the late 70s, so "IBM Assembler" or "Burroughs Medium System Assembler" maps fairly trivially on a line-by-line basis to high-level languages such as COBOL. I think that about the last system designed to look like that was the VAX, even CISC microprocessors have a far more primitive repertoire of operations and data types.

HEARTFELT PLEA: If Paul (or anybody else) starts making significant progress on this, I think that the community very badly needs a blow-by-blow account for future reference. Paul's notes in the wiki are very much to be applauded.

I may be overlooking something, of course. However: Our RTL is based on common (target specific) routines for reading (and writing) text and binary files (do_read & do_write). You cannot translate between ASCII and EBCDID in these target specific routines because you don't know how the input would be used at that point (not even mentioning the fact that there is nothing like translation between ASCII and EBCDIC because there are multiple different character sets for both and real conversion isn't possible without taking this into account and knowing the real character sets which again depends on the context which is again not known at this low level). Unless I'm mistaken, this implies that you indeed need to consider the (basic) EBCDIC layout as an alternative to the (basic) ASCII layout directly within the RTL. This should be obviously done in a way not impacting the RTL behaviour for other targets. However, that is certainly possible using either constants differentiating between ASCII and EBCDIC (thanks to elimination of the unused code identified at compile time by the compiler), or using IFDEFs, or using special versions for certain routines (located in different include files, etc.) - although I personally believe that the last one is the least viable one in this case due to likely duplication of majority of the code within such routines in the two implementations.

Not even mentioning the additional "minor" issue with certain characters (critical for Pascal source codes) not necessarily directly available in _some_ (!) EBCDIC character sets as pointed out by Mark - again something which cannot be handled in the general I/O routines because it only becomes important when interpreting a general text as Pascal source code (in this case, special support on the compiler side will be probably necessary, i.e. this should have no impact to RTL, but it will again have impact to the common parts of the compiler, namely scanner, not to target specific units).

Tomas

IMO the only bottleneck could be the availability of 8 bit bytes (dunno). At least I remember mainframes with byte sizes of 6..16 bits. When at least 8 bits are available, then a program can use ASCII/ANSI/UTF-8 internally, and strings have to be converted into EBCDIC in system calls, like they have to be converted for every OS with specific character sets or encodings.

It would be a real challenge to adopt FPC and all ordinal type handling (and pointers!) to a different byte size. E.g. DG machines only can address 16 bit words, so that access to 8 bit entities requires explicit machine code, and PByte or PChar is very tricky to implement. On such machines UTF-16 would be a good choice for the native string type.

DoDi

I wouldn't suggest to use 2.7 (trunk) as the development base, with Unicode and other features under development. Once a machine and RTL has been implemented for 2.6, it should not be so hard to merge it into trunk.

I also would suggest git for all experiments, with better branch, merge and revert capabilites than SVN offers.

2. I suggest you to use the Lazarus IDE to do compiler development

+1

DoDi

If the RTL were fully Unicode-aware then possibly this could be handled by host localisation, presumably on a classic IBM OS the JCL will state unambiguously which variant of EBCDIC is expected.

I think we need to wait for some input from Paul on this one, after all he's the project's instigator.

I can't remember the source, but my understanding is that Wirth originally worked with an IBM 029 keypunch, possibly connected preparing decks for a CDC. He specifically defined (* and *) as digraphs for { and }, and I think there were others including (. and .) for [ and ] Did FPC /ever/ fully-support these?

These should be fine (as easily confirmed by a short test program). I actually use (* and *) for comments quite often to distinguish from other comments using { and }. ;-) I'm not aware of a digraph for the caret sign though which is (as correctly pointed out by you) missing in the EBCDIC character sets specifically targetting English (and also Portuguese apparently ;-) ) - it seems to exist in the "International" character sets.

Tomas

Once more: I do not think this is the task of the RTL. FPC and the RTL works with ASCII. That includes the expected contents of text files and whatnot.

That a hypothetical FPC program opening an EBCDIC file on OS/370 needs to convert this to ASCII first is in my view the responsability of the programmer.

The same goes for the FPC sources.
I do not suppose that a FPC port would first convert the current ASCII sources to EBCDIC, but would continue to use the ASCII version.

If you define the RTL as being able to handle EBCDIC on OS/370, it is a different matter, and a lot more work.

Michael.

Some of the {$ directives apparently don't have (*$ equivalents. MACRO if I recall correctly.

OK, could be, but not being able to work with text files created on that platform without additional effort on the _programmer_ side (not the RTL porter side because the RTL porter cannot address this easily within our current RTL structure as pointed out above) would hinder usability of such a compiler _a_lot_. It's certainly possible to start with that, but keeping it that way in longer term would considerably decrease the value of such undergoing in my opinion.

Tomas

If some of the directives don't work with (*$ instead of {$ then these should be reported as bugs.

Regards,
Sven

What kind of definitons do you think of?

By intuition I suggested MIPS as the base, because its a smaller target in FPC (and m68k is in no shape to be used as a base...).

I don't know whether the high level code generator Jonas implemented for the JVM branch will be of any help here.

Regards,
Sven

I personally prefer to work in trunk no matter what. This way I can easily retrieve fixes for problems I encountered while developing my own feature, target, whatever.

Additionally trunk has become quite stable again (here and there some hiccups because of the fpmake conversion, but they are usually fixed at the next day).

I also would suggest git for all experiments, with better branch, merge and revert capabilites than SVN offers.

Nevertheless the SVN server should be used as a base. I used git-svn quite successfully when working on generics.

Regards,
Sven

Absolute minimum, probably mostly makefile stuff so that make in ./compiler would correctly pick up ./compiler/s390, possibly initially populated with the MIPS stuff from trunk + my fudges.

I think you're probably right. Of the architectures even part-implemented I think MIPS is about the nearest- certainly in terms of its register model.

As pointed out in my other e-mail, "everywhere necessary" implies either "dear user, convert all your files from the original encoding before you want programs created in FPC to touch them" or "dear programmer targetting OS/370, make sure that your programs are limited in what RTL functions you use, or convert all locally stored files to ASCII and only use the RTL functions for text processing on the converted copies". Otherwise even stuff like line by line reading or field by field reading of the input text file using standard RTL routines may not work as expected with the current RTL.

Tomas

2012/1/18 Tomas Hajny :

As pointed out in my other e-mail, "everywhere necessary" implies either "dear user, convert all your files from the original encoding before you want programs created in FPC to touch them"

Yes, no problem here. I assume there must be some program in this platform which can edit ASCII text and another one (or the same) to convert text files between encodings. If not, just use the new port to cross-compile such a program =)

I don't see why. A text encoding is just a text encoding, of the hundreds of obsolete ones in existence, and the only sane way of handling text in cross-platform applications is Unicode.

The RTL could ship with UTF-8 <-> EBCDIC convertor and define UTF-8 as the platform encoding. Detect which exact format the platform is using at runtime if necessary and convert everywhere necessary. This should cover all characters possibly imaginable and all control characters too.

What could go wrong here? This is what Java does in all its platforms.

As for WriteLn / ReadLn if one really wants to allow inputing directly control codes, one could either make them use UTF-8 and offer an alternative RawWriteLn / RawReadLn for raw input of control codes or leave them sending raw text and expose the routines to convert UTF-8 <-> EBCDIC

.
.

If Mark meant also addition of constants for a new architecture + OS combination(s), it's quite a bit of places already (I could find at least the following: compiler/systems.inc, compiler/systems.pas, compiler/utils/ppudump.pp, packages/fpmkunit/src/fpmkunit.pp and utils/fpcm/fpcmmain.pp plus creation of the corresponding subdirectory in the compiler directory; then there is also stuff like compiler/options.pas where TOption.WriteHelpPages and other places are currently still missing MIPS, compiler/msg/errore.msg, etc.).

Tomas

Just a moment. Granted that everybody's favourite reference site is unavailable today, but minimal Googling suggests that MUSIC/SP- which I believe was Paul's OS of choice- supports web browsing etc.

I think we need a reality check from Paul here, in case EBCDIC operation is mandated only by the emulator he was planning to use (whether I think his choice is good is irrelevant- lets try to get things like character set resolved before it causes any more sound and fury).

I don't know the typical situation on S/370 machines nowadays, so the following comment may not be valid. However: how likely would you be to use some new program if any interchange of files between this program and whatever other software used on your machine would require you to convert these files back and forth manually all the time (considering that the existing software may be the main reason for still using this kind of platform at all)?

My point is not about cross-platform applications here. My point is about applications running on zOS / OS/370 / ... natively (whether these applications should be cross-platform or not is the second step in my opinion).

I don't know how it works for Java, but I know that it cannot work transparently in current FPC RTL without making at least some changes in the common parts (platform independent so-far). Most likely something similar (i.e. changes to otherwise platform independent RTL parts) happened to Java too when ported to S/370, or the run-time part design included such kind of considerations from the beginning (which I personally doubt ;-) ).

The point is not about the programmer interested in inputting the control codes directly (although that may be a valid scenario too if the programmer wants to work the same way he is used to on the other platforms), the point is about common parts of FPC RTL having e.g. hard-coded #9 as the tab character (which in turn controls how fields in text files may be separated from each other) or #10 as the line feed character and that this stuff happens at some point during transition from the generic "binary file I/O" to "text file processing" in common parts of the standard FPC RTL (as it stands now).

BTW, even if one would consider e.g. translation from EBCDIC to UTF-8 in FileReadFunc (modified from its current standard platform independent implementation) because that is the place where generic files become to be interpreted as textual content in our implementation of standard Pascal RTL functions, it would fail in other routines in rtl/inc/text.inc due to difference between length read from the original file stored in EBCDIC and the size necessary in the text file buffer after the translation to UTF-8).

Again - I'm sure multiple solutions exist, but I cannot imagine how it could work reasonably well without touching the common parts of the FPC RTL at least a bit (but unfortunately at multiple places which may be hard to find) except for a limited 'proof of concept' solution not meant to be used in standard ways (obviously, limiting all text file I/O to files encoded in ASCII or Unicode and not doing any console I/O if it cannot support ASCII or Unicode directly may be perfectly acceptable in such 'proof of concept' mode).

Tomas

It appears that MUSIC/SP requires a particular hardware opcode (IUCV) to access TCP/IP, and that this is not provided directly by the Hercules emulator. IUCV can be provided by either IBM VM running between Hercules and a guest OS such as MUSIC/SP or Linux, or alternatively by running the Sim390 emulator instead of Hercules. For completeness, Linux gets around this by using SLIP to encapsulate IP rather than assuming that IP (and lower-level protocols) is handled by the hardware or host OS.

I see MUSIC/SP described as "unix-like", but I don't know whether it is sufficiently close to industry norms to run standard tools such as make, and even if the compiler is intended to be PC-hosted I think that would be unfortunate. MUSIC/SP running on Sim390 should have networking etc., but I for one have reservations about an initial port to an OS that might turn out to be significantly removed from POSIX etc. If I have time I might have a bit more info on this later, as well as on the character set issue.

One important point, and I don't much like raising this, is that it turns out that the principal maintainer of both MUSIC/SP (through his work at McGill) and Sim390 died in 2008. I agree with Paul that a mainframe port is worth attempting, but I've got reservations about his choice of target hardware and OS unless he can make a case that there is a significant number of S/370 and older (i.e. without inline operands) S/390 still in the field.

MUSIC/SP internal character set appears to be EBCDIC, there are specific library functions for converting between this and ASCII with the latter described as for use on the Internet. However, this is a feature either of MUSIC/SP or of older mainframes, Linux running on Hercules is definitely ASCII:

$ xxd hello.p
0000000: 5052 4f47 5241 4d20 4865 6c6c 6f3b 0a0a PROGRAM Hello;.. 0000010: 4245 4749 4e0a 2020 5772 6974 6528 2748 BEGIN. Write('H 0000020: 656c 6c6f 2c20 2729 3b0a 2020 5772 6974 ello, ');. Writ 0000030: 654c 6e28 2757 6f72 6c64 2127 290a 454e eLn('World!').EN 0000040: 442e 0a0a D...

I guess I should make a few qualifiers here.

I think that a mainframe port should be made, if for no other reason than to keep C from being the only choice for developing programs on mainframes (other than Cobol, of course, and dinosaurs like me that know it are rapidly becoming petroleum!)

I know the 370.  I know more-or-less how file I/O is done under OS/VS1 which MUSIC/SP provides an emulation mode for.  I don't (yet) know the machine code for the 390 processor and I do not know the interface for which services from Linux/390 are implemented, I mean, is it a set of different supervisor calls (the way OS/VS1 is) or is it a single supervisor call with lots of parameters (the way MS-DOS with (mostly) INT 21 and Windows 98/Me/XP and Linux on Intel are)?

Certainly I'll eventually learn them.  But for right now, for me to know it works right, I have to implement it for what I know works.  Then I can go on to improve what it does or expand the port to the more powerful mainframe.  As I said - or if I didn't say it, it needs saying - I have to crawl before I can walk or run.

There's another thing.  In at least a few parts of the compiler there are uses of naked, manifest constants which is always a bad idea.  I think someone once said that the only manifest constants that should be used in a program are 0 and 1 and maybe not even them. 

Changing from manifest constants to CONST definitions does two things, it ensures consistency, and, if your language has cross-reference capability, the cross-reference utility can tell you every place you've used an identifier; It can't tell you where you've used a constant.

I remember this from back when I first learned Assembly Language, probably around 1976-77, one of the things instructor Don Lacy taught us is to define registers by creating a set of names on the order of

R0               EQU     0
R1               EQU     1
thru
R15             EQU     15

So when you use

PROGRAM  START
                   BALR   R12,R0                    USING *,R12

The assembler will list the usage of R12 and R0 here in the cross-reference.  It would not list anything if you'd just used 0 and 12.

As anyone who read this can remember a few messages ago when one guy read my error when I wanted to show how the $DEFINE parser only supports {$DEFINE and that (*$DEFINE doesn't work, but thought I didn't know you can't close {* with *) and thought when I did it wrong as {* instead of (* and thought I meant the { when I meant ( just goes to show how easy it is to confuse { and (.  

It's easy to confuse 0 (zero) and O (capital o) 1 and I (upper case i), 1 and l (lower case L), and sometimes 8 and 0.  Not using them as constants but as constant identifiers means it's easier to spot and to be consistent.  

There's also on piece of code - it's probably in SCANNER.PAS - where there's a construct of something like

    if CH in ['A'..'Z']
which won't work on some processors; EBCDIC doesn't define A-Z (or a-z) as a continuous set; there are non-alphabetic characters between I and J (and between i and J) and another intervening point.  This goes back to the days when code was entered on 029 keypunches.  (And yes, I remember those days, and NOT fondly!)

So I guess the question is, for submitting proposed changes, do I ask someone - and if so whom - for SVN access and a branch to use to put corrections in, or do I pick a specific person - and again, if so, whom - or place (and where) to submit source code changes to provide some fixes as I make them?  Not necessarily just for this, I noticed some things I think should be cleaned up.

Just under the presumption of always following rule #1: DON'T BREAK THE BUILD.  If it worked (built without an error) when you put your dirty hands on it, it had damn well better still work when you've pulled your greasy, sticky fingers away!

 
The Lessons of history teach us - if they teach us anything - that no one learns the lessons that history teaches us. 

--- Begin Message ---
From: Paul Robinson

Sven Barth writes (edited for brevity):

It's not that I'm scared that you want to tackle this it's more that I've seen that you based your How-To on assumptions that are wrong or at least "not good" and wanted to help you in this regard.

That was part of the reason I started looking at this first, before "formally" making an announcement.  I wanted to see what kind of mess I was getting myself into.  This is not a trivial task and it's something I've wanted to do for several years.  I want to do this right, and it will take time.  To quote from some movie I can't remember, "If it takes six months, it takes six months.  If it takes a year - or a year and a half - it takes that long."

Well, then let's get started with bringing you on track: 1.  Also there were quite a few changes from 2.6.0 to 2.7.1 and thus it's better to work with the most recent source.

I didn't know the sources were up to 2.7 already.  I installed Tortoise SVN - I was using Mercurial because that's what a different project is using, and I'm an agnostic, so I'll use anyone's source code repository.  I'm writing a (fiction) book about life after death and I was using CVS for its releases I think.   So anyway, I used Tortoise SVN and retrieved the Compiler directory, and (I just looked) PP doesn't mention what version it is, VERSION does, and it says 2.6.0.   I do want to keep a 2.6.0 branch to keep sources for the existing compiler, I will find where the bleeding edge branch is and create a 2.7.1 version there.

2.  I suggest you to use the Lazarus IDE to do compiler development

Once I found you can build console-mode apps using it I switched back to it.  Interesting with all the crapola to implement windows support, even console, e.g. DOS WINDOW - applications are looking more and more like full Windows ones.  I've discovered that Lazarus has raised my standards in IDEs from the dead (pun intentional!)    It is an order of magnitude much easier to work with than the Turbo Vision clone version.  Since I found interest in including cross-reference in the compiler to be less than lukewarm I decided to write an external cross-reference program.  One quick question which the manual doesn't seem to answer, is whitespace allowed between { and $ in the case of {$INCLUDE 'X'}?  I sometimes see it in some of the compiler sources and I'm not sure if that's allowed or it is effectively 'diking out' the $ command.  The manual does not say one way or the other.

3. add unit "i_i370" to the "compiler" unit and directory "systems"..  "compiler" unit .. OS .. not architecture. add a "i_osvs1" to that unit... first need the architecture support...  corresponding CPU unit ... "cputarg" in the unit "compiler". ...suggest you copy the directory of an architecture... name that "i370"

As I've said, I'll take all the help I can get.  Anything to make the work easier is appreciated.

4. stumpled upon "sysutils"  don't need to care ... yet. These are only needed if you compile programs for the target platform ... might want to stub "System" ... most important part .. real entry point "begin...end." in "program" NOT the real entry point, this part is called "PASCALMAIN" and is called from within the real entry point

This makes sense, when you run a compiled program usually you do not want to start the program first, you want to start the run-time-library and have it, after it's been initialized, to start the program. 

That (PASCALMAIN) will have to change, the object file format for 370 machines limits a CSECT to 8 characters, with the optional characters @ # and $.  So I might call it MAIN or possibly #MAIN.  (On the PDP/11, most operating systems (RT-11, RSTS/E, RSX) defined the main program in any language as ".MAIN." )

So at the beginning you can mostly concentrate on your "i370" directory with adding some types here and there (like the assemblertype that you already added). might be best to first target Linux ... split into "architecture support" "OS support"

I may-probably will - put it in at the same time but in stub form for the reasons I stated above; I don't know Linux/370 and might not know why, if a program doesn't work, why it doesn't.  The first step is to at least generate "Hello World" and have it actually execute on the target.  Possibly insert "readln" so I know it correctly reads and writes I/O.  Then going on from there.  Plus - this is sort of a cheat - I have compiler sources that can tell me how to implement some of the run-time library.  Then I can go on and add what Linux requires for this implementation vs. what it does on (I almost said Windows!) I386 hardware or whatever has to be done.

Paul

The Lessons of history teach us - if they teach us anything - that no one learns the lessons that history teaches us. --- END MESSAGE---

WITH message^Paul.Robinson DO
BEGIN

Mark Morgan LLoyd wrote

I've got reservations because of the EBCDIC issue.

That's not a problem.  You write a program on the 370, the machine does the internal conversion.  Most people never notice the difference.  The standard characters are the same in all versions of EBCDIC, the conversions for A-Z, a-z, 0-9, space and most symbols @!#$&*()-+ etc. are pretty much cut and dried.  I actually wrote an ASCII <- ->EBCDIC conversion to be able to view some text files from a mainframe PDS  (their version of unix's tarball files) in order to understand what is going on.

I'm sure Paul has his reasons that we all need to respect, but I don't think that using a 370 emulator (i.e. at least two architectures out of date) plus an obsolete operating system is the place to start.

Again, I have to target what I know.  A program written for the 370 will run on a 390.  Having gotten the simpler version up and running, I can put other things in place and potentially add hooks later.  If I have to learn the extensions for the 390 and how Linux/370 works, well, I guess I can probably add another 2 years on top of the conversion.  I'd like potentially to see something running by Summer 2012, just in case Michel Nostradamus and the Aztecs were interpreted correctly and there'll be no tomorrow on December 22!  (I'm going to be real upset if that happens, I just bought a five-year extension on my Magic Jack phone service before the price goes up.)

I admit to having considered this as a long-term project, but what I don't have is a good understanding of the architecture's assembler-level programming.

I've been updating the Wikibook (another one of Wikipedia's projects) on IBM 370-series assembler, just to remind myself of what was involved and to keep my skills sharp.  Having gotten back into this I'm starting to remember things I'd forgotten. 

They didn't tell us that at the time, but a lot of people did and didn't realize it.  By convention, Register 13 is used by a called program as a save area to store the user's registers.  If you're a subroutine (or if you prefer, you wrote a subroutine or wrote your program so that it would operate as one) the caller hands you R13 with an address of a 72-byte save area to store a spare word (used by Fortran and PL/I), plus all of the registers in the order 13-15 and 0-12 so you can return them as they were given to you and you don't have to worry about how you use them.  So what some people did was use the register version of the OS/VS1 GETMAIN macro, which tells the operating system to return a block of storage space in register 1.  Then they'd save the current value of register 13 at address 4 in this block, then put the block's address in R13.  Just like doing a linked list. 

Now you have a place for any routines you might call to store your registers, plus, if you store your variables as offsets in register 13, you've created a stack to put your variables on and your program, since it uses no local writable memory, is fully reentrant and position independent.  So you got it, you got a stack, reentrancy and PIC all for free and most of us who wrote code doing it that way never even knew it.

Tomas Hajny wrote

Still, starting with the Linux for S/390 (or are S/390 and S/370 completely different animals?)

Similar to the differences between Pentium and IAS-64, the 390 is a 64-bit version of the 370.

probably a very good idea to avoid having to tackle both a different HW platform and a different OS at the same time (the Linux port to S/390 apparently uses ASCII according to what I found on Internet). Obviously, the OS may follow once the new HW platform is in working state.

I may take a look at it, but I figured getting a Linux port was a substantial amount of work I didn't want to do right now; I have code I can use for the OS target I am using, I can know if it works or know why it doesn't.  Not so for the 390 or for Linux.   I can, conceivably include it as an option with it stubbed for the moment, then as I figure out how to implement the interface to it I can.

Michael Schnell wrote

As for using Linux: I have no idea how this works in practice, so I can't comment on that.

Which more-or-less makes two of us. 

I know that people here use Hercules inside linux to develop for zSeries, but then again running a linux inside that seems a bit like overkill

I would disagree, it's available, its standard, and thus it allows you to test programs which might otherwise run on a mainframe on your PC.  I mean, IBM ran a commercial a few years ago where someone thought they had been robbed, when racks and racks of servers were replaced by a single IBM zSystem mainframe about the size of one server rack.  If you spend, oh say US$200,000 on a single computer, as opposed to spending $300,000 or more to put up 1,000 servers, you might find that the architecture provides better results where it's optimized.  Mainframes did very well on block I/O in that respect, and most data transfers on the internet are block I/O, e.g. web pages, data files and other resources.  They suck on character I/O which is why terminals are always connected as block devices; you'd never have single character at a time or mouse controlled apps running on a 370.  They shine with large volumes of data and transaction processing using blocks at a time.   I've heard on a real zSystem you can run something like 22,000 virtual Linux sessions before the underlying VM partition ran out of resources.  Which implies that one zSystem could conceivably handle the load of 22,000 servers.

Plus you can't infect one the way you can to a microcomputer.   You can do a tremendous amount of things running as a non-privileged user, moreso than on, say Linux, because you can decide to let it be able to do some things and implement supervisor calls where it needs to access privileged functions.  So, first, your average random J hacker isn't going to know mainframe assembly language to even be able to inject code to pown someone's box, if a program has an error that allows someone to exploit a buffer overflow, most likely the process running that web server crashes and is restarted, they do not get the ability to do shell access (because there might not even be shell access if a web server is marked as being like a non-interactive task) and it's never running as a privileged task so it has no privileges an attacker can steal.

The Lessons of history teach us - if they teach us anything - that no one learns the lessons that history teaches us.

END .

Paul Robinson wrote the following on 19/01/12 23:28:01:

I guess I should make a few qualifiers here. I think that a mainframe port should be made, if for no other reason than to keep C from being the only choice for developing programs on mainframes (other than Cobol, of course, and dinosaurs like me that know it are rapidly becoming petroleum!)

I used IBM's VS/Pascal and Pascal/VS for many years and really liked them. (IUO and public versions of the same compiler.)
And there's always PL/1!

Or Java. And while it's conceivable that Oracle could make themselves sufficiently unpopular that the rest of the World junked Java, as I understand it IBM has so much invested in e.g. hardware assist engines that they'd be the last to jump ship.

I know the 370. I know more-or-less how file I/O is done under OS/VS1 which MUSIC/SP provides an emulation mode for. I don't (yet) know the machine code for the 390 processor

My understanding is that the 390 is backwards-compatible with the 370 and 360. However you don't need to understand the binary code, since FPC emits assembler source which is processed by the standard GNU assembler as a backend. There are FPC command-line options that allow you to save this with varying levels of explanatory comments for later inspection, although some of the more juicy stuff requires that FPC be recompiled with EXTDEBUG=1 on the make command line.

I'll see what I can dig out. However I think that
http://linuxvm.org/present/SHARE102/S9236uw.pdf is a good starting point, and most if not all other Linux ports work on the basis of a single supervisor call with multiple parameters (nota bene: the constants identifying syscalls vary across architectures).

I want to be quite unambiguous about this: from the point of view of implementing a compiler, it is not that the 390 is more powerful but that the 370 is broken. Referring to
ftp://sourceware.org/pub/gcc/summit/2003/Porting%20GCC%20to%20the%20IBM%20S390.pdf section 3.1, it was at least partially fixed in the 390 circa 1992, and if I recall correctly GCC doesn't support older systems. Now you're the guy with the relevant industry experience: are there still pre-'92 mainframes in commercial or academic service? My understanding is that "big iron" of that vintage has been recycled for its precious metal, and that contemporaneous "micro mainframes" are woefully underpowered by today's standards- and I write as somebody who's got some fairly old kit, and who prefers not waiting 24 hours for a trial compilation to throw bugs.

There's another thing. In at least a few parts of the compiler there are uses of naked, manifest constants which is always a bad idea. I think someone once said that the only manifest constants that should be used in a program are 0 and 1 and maybe not even them.

Yes, but I'd suggest making sure that your architecture-specific code is clean before moving on to the generic stuff.

[Snip]

OK. Important point here: FPC uses gas (GNU Assembler) or equivalent as a backend, and so has to generate assembler that it's happy with. I came across exactly the same issue a few months ago with MIPS: it turns out that symbolic register names were only added to gas comparatively recently so if FPC wants to generate code that will assemble reliably it can't use them... at least for a few more years.

"The senator's still wrong." :-)

If you really want to be strict, I'd suggest that it would be reasonable to be able to close { with *) or (* with }, since (* and *) are digraphs for { and }. {$ works, (*$ should work, //$ /*$ -- $ and ⍝$ don't work and probably never will. {*$ is just plain /wrong/.

However there are additional concerns here related to Pascal's inability to nest comments (unlike Modula-2 which by definition counts nesting depth) so let's just hope for a good compromise.

It's easy to confuse 0 (zero) and O (capital o) 1 and I (upper case i), 1 and l (lower case L), and sometimes 8 and 0. Not using them as constants but as constant identifiers means it's easier to spot and to be consistent.

Agreed. But it's also good practice for a programmer to select typefaces that make the differences stand out. Or to put a new ribbon in his '029.

Yes. And I'm the one who's trying to get you to avoid EBCDIC, at least to start with.

I think we need a core developer's input on this. I'd also like to be able to do something with the MIPS fixes I've done, particularly since they're likely to be of some use to you (i.e. you could start off with a compiler that would build on a PC, and then modify the code generator (in extremis I'll mail you).

Just under the presumption of always following rule #1: DON'T BREAK THE BUILD. If it worked (built without an error) when you put your dirty hands on it, it had damn well better still work when you've pulled your greasy, sticky fingers away!

For project as big as FPC+Lazarus (plus all the other tools and libraries they depend on) you start off with rule #0: DON'T UPSET ANYBODY.

I think you mean x86-64 there, but if you really do mean the IA-64, then that's... not entirely correct. The IA-64, AKA Itanium or in some quarters Itanic, is profoundly different from the Pentium at all levels.

As I understand it the 390 and later the zSeries are evolutionary steps from the 370 and earlier 360. The correct analogy there is that x86-64 (x64) is an evolutionary step from the Pentium, and from the earlier '486 and '386.

More to the point, post-'92 390s are an evolutionary step from the 370, plus they have facilities for inline literals that older machines lack. Insisting on using /only/ 370 facilities when there are one or two things in 390s that make life much easier is comparable to insisting on writing a compiler for the original 8086 when the '186 and all successor chips had the pusha/popa opcodes that made life much easier.

Start off with Linux, using the existing RTL as a starting point. There isn't that much in it that's written in assembler, and trying to redo swathes of it based on your existing code is going to make life difficult for everybody- both future maintainers and people who are trying to help you in the initial port.

Mr Paul Robinson wrote

> 1. Also there were quite a few changes from 2.6.0 to 2.7.1 and thus it's better to work with the most recent source. I didn't know the sources were up to 2.7 already. I installed Tortoise SVN - I was using Mercurial because that's what a different project is using, and I'm an agnostic, so I'll use anyone's source code repository. I'm writing a (fiction) book about life after death and I was using CVS for its releases I think. So anyway, I used Tortoise SVN and retrieved the Compiler directory, and (I just looked) PP doesn't mention what version it is, VERSION does, and it says 2.6.0. I do want to keep a 2.6.0 branch to keep sources for the existing compiler, I will find where the bleeding edge branch is and create a 2.7.1 version there.

Before 2.6.0 was branched from SVN trunk the version was called 2.5.1. An odd minor number (here the "5") always indicates that it's a development version. After 2.6.0 was branched (which was still named 2.5.1 at that time) trunk immediately became 2.7.1 and is available either through SVN or from the FTP server containing the daily snapshots.

The SVN URL for the 2.7.1 sources (also called development version) is here: http://svn.freepascal.org/svn/fpc/trunk

Please checkout the complete trunk directory and not only the compiler one. It might be that you need some of the additional tools and also you can then test whether your platform can be compiled using the Makefile as well (though the Makefile needs to be adjusted first for this).

The version information of FPC is located in
$fpcdir/compiler/version.pas and there indeed "2.7.1" is encoded.

> 2. I suggest you to use the Lazarus IDE to do compiler development Once I found you can build console-mode apps using it I switched back to it. Interesting with all the crapola to implement windows support, even console, e.g. DOS WINDOW - applications are looking more and more like full Windows ones. I've discovered that Lazarus has raised my standards in IDEs from the dead (pun intentional!) It is an order of magnitude much easier to work with than the Turbo Vision clone version. Since I found interest in including cross-reference in the compiler to be less than lukewarm I decided to write an external cross-reference program. One quick question which the manual doesn't seem to answer, is whitespace allowed between { and $ in the case of {$INCLUDE 'X'}? I sometimes see it in some of the compiler sources and I'm not sure if that's allowed or it is effectively 'diking out' the $ command. The manual does not say one way or the other.

A compiler directive "{$...}" (or "(*$...*)" ) is only detected as one if the "$" follows immediatly the opening comment sign. Thus a space character (or anything else) is often used to disable a directive (it's just a normal comment then).

If I understand you correctly, then you might confuse label names with section names.

PASCALMAIN is a placeholder (a label) for the final address where the code of PASCALMAIN is put to by the linker. You can think of PASCALMAIN as a procedure. In fact e.g. the Windows RTLs contain the following code:

=== code begin ===

procedure PascalMain; external 'PASCALMAIN';
//in the entrypoint procedure:
...
PascalMain;
...

=== code end ===

Regards,
Sven

On Fri, January 20, 2012 01:21, Mr Paul Robinson wrote: .
.

much easier to work with than the Turbo Vision clone version.  Since I found interest in including cross-reference in the compiler to be less than lukewarm I decided to write an external cross-reference program.  One

.
.

You may want to look at pasdoc (http://pasdoc.sourceforge.net) - I believe that it provides (among other things) also the cross-reference generation functionality.

Tomas

---- Start of Message by Paul Robinson ----

WITH MESSAGE DO

BEGIN 

Sven Barth wrote (quoting me):

No, I know exactly what I mean.  Label and section names are under the same rules in an assembler program on the 370, and a CSECT in 370 Assembly language

THENAME    CSECT

 is equivalent to

Procedure THENAME;
(* in Pascal *)

       SUBROUTINE THENAME
C       in Fortran

000010 PROCEDURE DIVISION.
000020 PROGRAM-ID.  THENAME.
000030*  In Cobol

int thename(void) {
/* In C */
}Excepting, of course, for the brain-dead moronic behavior of assigning case-sensitivity to identifiers in C.

Paul

The Lessons of history teach us - if they teach us anything - that no one learns the lessons that history teaches us.

END.
----

I suspect that this will depend on the target operating system, so the Linux ABI may differ from e.g. MUSIC/SP.

This is something you'll definitely need to investigate, because the mangled Pascal identifers (for variables, procedures, etc.) are normally MUCH longer than 8 characters. For procedures they contain the unit name, the procedure's name and the types of all arguments (for overloading). You can see this if you compile a program with the "-al" argument, which will tell the compiler to leave the assembler files so you can take a look at them (normally called "unitname.s").

Another possibility would be to research whether there are other assemblers which don't have that restriction. Is there C++ available for this system? If so you can check what that uses as an assembler, because C++ uses mangled names as well.

Regards,
Sven

ABI means what? for those of us reading and trying to keep up ;) this might even be something new to those attempting to work on such a port task ;)

"Application Binary Interface"

Ralf

Application Binary Interface, it's the overall set of register usage conventions, syscall numbers and so on. I'm unsure the origin of the term but over the last ten years or so it's come into common usage for Linux, and extended to other OSes.

For Linux on IBM mainframes, see
http://linuxvm.org/present/SHARE102/S9236uw.pdf as a starting point. Also I've put info on assembler formats etc. at
http://wiki.lazarus.freepascal.org/Assembler_and_ABI_Resources (I needed comparative info when looking at MIPS a few months ago), I was hoping to add links to ABIs for each architecture but didn't get that far since something else intervened.

I've added a fragment of assembler code (from the Linux kernel) to the above page as an example. Digging around, it looks as though Linux assumes at least an S/390 G5, according to Wp this model was released in 1998, I don't know whether any are still in service.

Noting Paul's intention to target MUSIC/SP, I think that the points that distinguish this are (a) it's compatible with older IBM OSes and (b) in conjunction with a suitable emulator it has TCP/IP. I don't think that postings earlier in this thread make clear that it's (apparently) the only free OS that has this combination of features, and whilst I've still got misgivings over e.g. its use of EBCDIC I can "see where he's coming from".