I have just found the thread discussing a port of FreePascal to the System/370 and I feel I have to correct some misinterpretations, mistakes and other calumnies that have been thrown into the discussion.  First, my qualifications.  I have been a developer of Assembler systems, both applications and systems software since 1981.  I have worked on VS1, MVS (370, XA, ESA), OS/390 and z/OS systems.  I have worked for many large blue chip companies and for software houses (small) and computer manufacturer's (large).

In this first note, I will deal, mostly, with the character set issues;  Other notes will follow - be warned.

Firstly, the easy one, the System/370 and related processors come with a large supply of 8 bit bytes. :)

There seems to a common perception on the internet that EBCDIC does not have codes for things like square or curly brackets.  This is untrue.  My little magic EBCDIC reference card (Dated February 1975) lists them as; '}' - 0xD0, '{' - 0xC0, '[' - 0xAD and ']' - 0xBD.  Square brackets seem to be "new" with System/370 (1970's), but curly brackets are actually built into the naming requirements of many systems modules for some odd reason.  As such, they have always been in the character set.  I think the confusion may have arisen by their absence on the original card punch keyboards.  But that was 50 years ago, let's try and be a little more up to date than that!

A slightly later version of this reference card is available online at http://bitsavers.org/pdf/ibm/370/referenceCard/GX20-1850-3_System370_Reference_Summary_Nov76_2up.pdf

'^' doesn't seem to be available, but my eyesight isn't what it was! (VS/PASCAL, an MVS version of ISO pascal, uses -> as a digraph for this).  I don't think there's anything else.

It should be noted that using Linux/390 doesn't remove the '^' problem.  IBM display unit (3270) keyboards, being EBCDIC devices, won't have the '^' on the keyboard so an alternative must be found.  (I'm assuming that Linux uses 3270 devices, maybe I'm wrong)

Finally, the suggestions about developing FreePascal/370 as an ASCII compiler seem somewhat pointless to me.  Why would anyone want to use an ASCII compiler on an EBCDIC system?  I accept fully that producing an EBCDIC version will present problems, but if this compiler is actually going to be used by anyone, these have to be overcome.

I think the reason for producing an ASCII version first is very simple: All FPC sources - including the compiler - are in ASCII encoding.

Whatever is made, the hypothetical compiler will need to understand ASCII in the first place, or it cannot recompile itself or it's RTL...

Michael.

wrote the following on 30/01/12 14:49:53:

I think the reason for producing an ASCII version first is very simple: All FPC sources - including the compiler - are in ASCII encoding.

I don't understand this statement - ASCII and EBCDIC are just human representations of a computer's internal code. I write my programs in the Latin (or Roman) alphabet and the computer does the rest. When I was writing VS/Pascal programs I used the same source code as input to VS/Pascal on the mainframe and to Virtual Pascal on the PC.

Unless the FP source code is to be fed into a mainframe compiler like IBM's VS/Pascal or the Stanford compiler then the first step is surely to write a backend for the (eg PC) compiler to produce 370 assembler code. Producing EBCDIC rather than ASCII sounds a trivial part of the task.

Op Mon, 30 Jan 2012, schreef

... which can be ASCII or EBCDIC encoded, right?

So if the compiler source files are ASCII encoded... then the compiler has to read ASCII to be able to compile itself.

I write my programs in the Latin (or Roman) alphabet and the computer does the rest.

So the compiler has to care about ASCII/EBCDIC encoding, right?

Yes, but that's a difference topic than the source encoding. The compiler doesn't care how the assembler generator communicates with the assembler; it could be in any format, even binary. One just has to implement the assembler generator correctly.

The compiler does care that the source it processes is ASCII. If it has to be able to read EBCDIC, some conversion has to happen before the scanner. As such a conversion is not necessary for the compiler to compile itself, it may make sense to postpone writing such a conversion and have the initial port be ASCII only.

Daniël

I had in mind the following scenario:

1) Somehow we build - using cross-compilation - a first version of the compiler that actually runs on the 370. This binary is transferred to a 370 machine.

2) The sources of the compiler and RTL are transferred to the 370. I assume that after the file transfer, the sources are still in ASCII format ?

3) At that point the compiler can try to recompile itself on the 370 machine.

Unless you have performed some tranformation of the compiler/RTL sources, the compiler in step 3 will read and compile from ASCII sources, no ?

Michael.

Existing source code frequently assumes ASCII encoding. The obvious are upper/lowercase conversions, by and/or or add/sub constant values to the characters. It will be hell to find and fix all such code in the compiler and RTL, even if only the constants have to be modified for EBCDIC. Even code with the assumed order of common characters (' ' < '0' < 'A' < 'a') has to be found and fixed manually - how would you even *find* code with such implicit assumptions?

Next come character ranges, where letters are assumed contiguous in all existing code and examples. Clearly this is true only for ASCII ('a'..'z'), not for national characters like 'ä' or 'é', but the compiler assumes ASCII source encoding all over. Fixing the set constructor to make Set Of Char work with EBCDIC will be a challenge.

When a user e.g. picks up such example or library code from somewhere, and finds that it doesn't work, he'll blame the compiler for malfunction.

An EBCDIC based compiler will disallow the use of any foreign libraries, because a simple (syntactic) conversion from ASCII to EBCDIC encoding doesn't cover beforementioned (semantic) issues :-(

DoDi

Daniël Mantione wrote the following on 30/01/12 16:29:27:

No!! If I write my name and address - in the roman alphabet of course - on a PC the computer stores it as ASCII, but when I display it on the monitor it is displayed as human-readable roman alphabet (at least in my part of the world). If I then email it from the PC to a 370 the computer stores it as EBCDIC, but displaying it on a monitor it is the same human-readable roman alphabet. Human beings do not read or write ASCII or EBCDIC. Pascal source code can be encoded as ASCII or EBCDIC depending on what sort of computer it is stored on, but we see "human code".

steve smithers wrote

It should be noted that using Linux/390 doesn't remove the '^' problem. IBM display unit (3270) keyboards, being EBCDIC devices, won't have the '^' on the keyboard so an alternative must be found. (I'm assuming that Linux uses 3270 devices, maybe I'm wrong)

Linux on S/390 uses SSH, any of the standard X-based desktops, VNC and so on. I see no reference to EBCDIC, unless it's buried in options to control encoding for (emulations of) classic IBM peripherals.

I repeat: IBM is now happily using ASCII on zSeries. That includes the CDSL system made available to developers
http://www-03.ibm.com/systems/z/os/linux/support/community.html

wrote the following on 30/01/12 16:35:20:

No - sending source code from a PC to a 370 performs an automatic translation to EBCDIC (and vice versa).

IBM 370 doesn't use ASCII, anywhere, but it has a hardware instruction (TRT _ Translate and Test) which can convert between character sets in a single instruction using a suitable table.

wrote the following on 30/01/12 16:35:20:

No - sending source code from a PC to a 370 performs an automatic translation to EBCDIC (and vice versa).

IBM 370 doesn't use ASCII, anywhere, but it has a hardware instruction (TRT _ Translate and Test) which can convert between character sets in a single instruction using a suitable table.

Daniël Mantione wrote the following on 30/01/12 16:29:27:

No!! If I write my name and address - in the roman alphabet of course - on a PC the computer stores it as ASCII, but when I display it on the monitor it is displayed as human-readable roman alphabet (at least in my part of the world). If I then email it from the PC to a 370 the computer stores it as EBCDIC, but displaying it on a monitor it is the same human-readable roman alphabet. Human beings do not read or write ASCII or EBCDIC. Pascal source code can be encoded as ASCII or EBCDIC depending on what sort of computer it is stored on, but we see "human code".

Daniël Mantione wrote the following on 30/01/12 16:29:27:

No!! If I write my name and address - in the roman alphabet of course - on a PC the computer stores it as ASCII, but when I display it on the monitor it is displayed as human-readable roman alphabet (at least in my part of the world). If I then email it from the PC to a 370 the computer stores it as EBCDIC, but displaying it on a monitor it is the same human-readable roman alphabet. Human beings do not read or write ASCII or EBCDIC. Pascal source code can be encoded as ASCII or EBCDIC depending on what sort of computer it is stored on, but we see "human code".

Daniël Mantione wrote the following on 30/01/12 16:29:27:

No!! If I write my name and address - in the roman alphabet of course - on a PC the computer stores it as ASCII, but when I display it on the monitor it is displayed as human-readable roman alphabet (at least in my part of the world). If I then email it from the PC to a 370 the computer stores it as EBCDIC, but displaying it on a monitor it is the same human-readable roman alphabet. Human beings do not read or write ASCII or EBCDIC. Pascal source code can be encoded as ASCII or EBCDIC depending on what sort of computer it is stored on, but we see "human code".

I think that very much depends on the program you use to send the file.

We have a mainframe here, and sending ASCII files using FTP in binary mode, most certainly does not do any translation. I suppose the same goes for SCP, although I never tested that protocol myself.

And that is what I meant: unless you do the translation explicitly (and by this I also understand email, ASCII mode FTP transfer and whatnot) the compiler will see ASCII sources, even on the IBM.

Michael.

I seem to have messed up the subject lines on some of these posts.  Sorry.

Yeah.  Hence the repeated references to OS :)  I have had no contact with Linux/390 so I can't possibly comment.

That sounds useful.

> Once a initial port to 370 has been made and the assembler output > generator is implemented (which means that the compiler can basically > generate 370 capable assembler files) this is rather easy as FPC can > generate "link on target" scripts (and AFAIK also "assemble on target").

That too!

JCL IS an executable script.  But there are terminal based scripting facilities available too.  Running such a task on a terminal however, would have less storage available to it.

steve smithers wrote

I seem to have messed up the subject lines on some of these posts. Sorry.

Don't worry about it.

Although Linux/390 is closer to what the bulk of us are used to, so please humour us.

One question if I may, and I hope this doesn't sound too stupid. When Paul Robinson first raised an S/390 port a few days ago, he proposed using MUSIC/SP as his target operating system. This has the advantages that it's free and has TCP/IP, but the major disadvantage that the prime maintainer is... no longer active. I find it difficult keeping track of things on account of the name changes over the years: what is the situation as regards OS, is there a free version that can be run locally under (e.g.) Hercules, or would anybody who wanted to look at what you were up to have to have a login account on a mainframe somewhere? Or does MUSIC/SP have any/adequate binary compatibility?

But as I understand it JCL is distinct from binary programs- different areas of application, different facilities. On the unix-derived systems that most of us are more used to there is no such distinction: a shell script has access to all the facilities that a binary has, you could (in principle) write a compiler in it.

Mark Morgan Lloyd wrote on Mon, 30 Jan 2012 21:46:28 +0000 Although Linux/390 is closer to what the bulk of us are used to, so please humour us.

I am a Linux user so I am sympathetic.  It's just that I really don't do development on Linux and am therefore unaware of it's requirements.

Why should it sound stupid?  I've spent 25 years doing this stuff, I can't reasonably expect everyone else to know how it works.

MUSIC/SP as I understand it (from wikipedia I'm afraid) was a proprietry time sharing system writtem by McGill university in the states.  It's sole use to us in this discussion was that it had an OS/VS1 emulation mode.  OS/VS1 had no terminal based development environment at all (at least not a standard IBM one.  It did, later, have systems called TONE and ROSCOE).  I think Paul chose this because he either knew it was available or because it was the system he used to use.  OS/VS1 has hideous storage limitations by todays standards. However, this emulation mode should give us a good binary comatibilty level storage limitations allowing.

Hercules can run all, or almost all, IBM OS systems. from OS/MFT - OS/MVT from the 1960's up to the latest z/OS 64 bit systems.  The problem is with licensing.  The ones we can run legally are OS/MFT, OS/MVT, OS/VS2, MVS 3.8 along with various others such as early DOS and VM systems and MUSIC/SP.  All of these are free.

These, generally, don't have TCP/IP yet, but Hercules provides us with TCP/IP connection using TN3270 (telnet in 3270 mode) so we can use Linux or Windows based terminal emulators such as x3270 to connect to the guests. People are working on a TCP/IP stack for MVS.

It's also possible to have a logon to a remote MVS system running inside a window running in your favourite web browser.  I can't find it on the web at the moment and I don't know what version of MVS it used, but I will keep an eye out for it.

But we don't need to write a compiler, we just need to feed various source programs into a compiler and assemble and linkedit the resulting output with a modicum of "intelligence"  to stop on errors.  This is what JCL does. Think of it as using a pipe to link the output of one program into the input of another, just not as straightforward.  

You can use the terminal script languages to write more complicated stuff, CLIST the original would probably be capable of this, I wouldn't want to write it though because it's ghastly.  But REXX is available for MVS 3.8 and REXX can do anything! :)

I think in the current context the binary compatibility is the important thing.

With the major limitation here that MUSIC/SP- and possibly other operating systems- don't have TCP/IP unless the underlying platform supports IUCV; Hercules (freely available to everyone) doesn't implement IUCV unless VM (not freely available) is running on top of it. Linux appears to get around this restriction by using a simulated SLIP connection.

Sorry, you've missed my point. I've come across systems where compilers have to be "blessed" by the local security administrator before they can mark code as executable, and there's a progressively stronger chain up to the point where nobody except that manufacturer can bless a compiler such that it can generate the operating system kernel. The objective is to try to avoid the situation described by Ken Thompson in his 1984 "Trusting Trust" paper http://cm.bell-labs.com/who/ken/trust.html

Unix does not have this mechanism: anybody can build a compiler which can then build a new kernel.

This is how Unix and Linux evolved - everybody could play around with it, and add new functionality. Blaming an compiler for buggy source code IMO helps nothing. Recompiled kernels have to be booted, somehow, what is nothing that ordinary users can do on an mainframe. And when every user must manage his own system(s), what can he do but allow a just installed compiler to do its job?

Trusting code is a different thing. With open source code you can be halfways sure that the code has been tested by many people, and MD5 checksums prevent malicious modification of the downloaded sources. This is how malicious modifications, also to the compiler itself, can be detected and avoided.

Hackers are kind of evolutionary force, which makes the system software coders aware of backdoors and other flaws. Monetary losses have two benefits: they make intrusions visible, and they enforce improvement of flawed software (to be paid for). Ask companies how much money they invest in safe software, and compare that amount with the (claimed) losses caused by hackers :-]

DoDi

Please note that I'm not being critical, simply attempting to summarise the situation for somebody who might not appreciate the nuances, particularly in view of an earlier comment that it might not be possible to do the final build on a PC.

I'm not sure that an authenticate-by-source-digest approach would prevent the sort of problem that Thompson described, unless there was also a mechanism to validate that a particular binary was accurately described by its professed source package.