Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / fpc-devel / 2012-01

Re: EBCDIC ( was On a port of Free Pascal to the IBM 370)

7 messages · 6 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

steve smithers
Show quoted text 8 lines
Hans-Peter Diettrich wrote on Mon, 30 Jan 2012 17:40:27 +0100 Existing source code frequently assumes ASCII encoding. The obvious are upper/lowercase conversions, by and/or or add/sub constant values to the characters. It will be hell to find and fix all such code in the compiler and RTL, even if only the constants have to be modified for EBCDIC. Even code with the assumed order of common characters (' ' < '0' < 'A' < 'a') has to be found and fixed manually - how would you even *find* code with such implicit assumptions?

It does indeed.  I am aware of the problems inherent in this.  But the RTL has to be more or less rewritten anyway to support OS.  OS is a very different animal to Windows or Linux.

But, you would start with various searches using grep or something and scan for bits of the code that use constants like '#7' and change them to fpc_Char_Bell or something similar that would live in an fpcASCII or fpcEBCDIC unit or something similar.  You would search for all the combinations you could think of '['a'..'z']', '['A'..'Z']' etc.  Finally, exhausting your ingenuity you would be left with the old stand-by of testing.

A God-awful task I know.  But what's the alternative?  A note in the documentation for FreePascal/MVS that whenever you reference any external data it is the user's responsibility to convert from ASCII to EBCDIC.  Really?  AssignFile(f,'SYS1.PARMLIB'), sorry doesn't work, you forgot the ASCII conversion;  WRITELN('Hello World') produces garbage on the user's terminal.  Who will they blame then.  JobSubmit(asciifile) will disappear from the face of the planet because JES won't have a clue what to do with an ASCII file.

You can't convert automatically because you don't necessarily know whether the user is writing ASCII, EBCDIC or binary.  What happens to

  MyRec = record
     Field1 : string;
     Field2 : char;
     Field3 : integer;
     end;

If we are using ASCII should we be using Little-Endian numbers too!

Show quoted text 12 lines
Next come character ranges, where letters are assumed contiguous in all existing code and examples. Clearly this is true only for ASCII ('a'..'z'), not for national characters like 'ä' or 'é', but the compiler assumes ASCII source encoding all over. Fixing the set constructor to make Set Of Char work with EBCDIC will be a challenge. When a user e.g. picks up such example or library code from somewhere, and finds that it doesn't work, he'll blame the compiler for malfunction. An EBCDIC based compiler will disallow the use of any foreign libraries, because a simple (syntactic) conversion from ASCII to EBCDIC encoding doesn't cover beforementioned (semantic) issues :-(

A compiler is not just a tool for syntax analysis.  It has semantic routines built into already.  It's up to us to use enough ingenuity to cater for as many of these as possible.  Surely it should be possible to pick up stuff like 'a'..'z' at compile- time

Regards
Steve

Show quoted text 6 lines
Mark Morgan Lloyd wrote: I repeat: IBM is now happily using ASCII on zSeries. That includes the CDSL system made available to developers http://www-03.ibm.com/systems/z/os/linux/support/community.html

Yes. The Community Software Development for Linux on System/Z would use ASCII.  As we have already ascertained, Linux/390 is an ASCII system;  Using EBCDIC would be slightly south of stupid.

CDSL doesn't run on OS.  Except possibly under USS.  Does anyone know if USS is ASCII or EBCDIC?

(USS is Unix System Services, formerly known as Open/MVS.  It's a sort of Unix type Look-alike ish sort of thing that runs under versions of OS from MVS/ESA SP 4.3 onwards)

I think the reason for producing an ASCII version first is very simple:

Converting the source from ASCII to EBCDIC isn't a huge problem.  Their are many much larger problems ahead :)

No - sending source code from a PC to a 370 performs an automatic translation to EBCDIC (and vice versa).

It depends on what you use to do the transfer and what options you specify.  These utilities are normally configurable.  FTP and IND£FILE are.  They're the two I've used in the past.

IBM 370 doesn't use ASCII, anywhere, but it has a hardware instruction (TRT _ Translate and Test) which can convert between character sets in a single instruction using a suitable table.

Translate and Test wouldn't help.  Despite the name it doesn't actually do any translation as such. The instruction you meant was TR (Translate).  To quote "Sorry, pedantry strong this one runs"

Regards Steve

Reply by email

Sven Barth
to steve smithers
steve smithers wrote 12 lines
> Hans-Peter Diettrich wrote on Mon, 30 Jan 2012 17:40:27 +0100 > Existing source code frequently assumes ASCII encoding. The obvious are > upper/lowercase conversions, by and/or or add/sub constant values to the > characters. It will be hell to find and fix all such code in the > compiler and RTL, even if only the constants have to be modified for > EBCDIC. Even code with the assumed order of common characters (' '< '0' > < 'A'< 'a') has to be found and fixed manually - how would you even > *find* code with such implicit assumptions? It does indeed. I am aware of the problems inherent in this. But the RTL has to be more or less rewritten anyway to support OS. OS is a very different animal to Windows or Linux.

The RTL consists of two parts (though the border is not easily visible): a platform independant one and a platform dependant one. A port to a different target normally only includes touching the platform dependant one, but a port to 370 also requires touching the platform independant one. This is what DoDi talks about.

@other devs: Could the code page aware AnsiString type be of any help here?

Regards,
Sven

Reply by email

Daniël Mantione
to steve smithers: It does indeed.  I am aware of the problems inherent in this.  But the RTL has to be…

Op Mon, 30 Jan 2012, schreef steve smithers:

Show quoted text 12 lines
> Hans-Peter Diettrich wrote on Mon, 30 Jan 2012 17:40:27 +0100 > Existing source code frequently assumes ASCII encoding. The obvious are > upper/lowercase conversions, by and/or or add/sub constant values to the > characters. It will be hell to find and fix all such code in the > compiler and RTL, even if only the constants have to be modified for > EBCDIC. Even code with the assumed order of common characters (' ' < '0' > < 'A' < 'a') has to be found and fixed manually - how would you even > *find* code with such implicit assumptions? It does indeed.  I am aware of the problems inherent in this.  But the RTL has to be more or less rewritten anyway to support OS.  OS is a very different animal to Windows or Linux.

If you try to achieve a port by modifying all code that deals with characters you will fail. The amount of work becomes then far too big for a single person, and the modifications become too huge and wide-spread that you will raise objections for merging it with the SVN trunk.

In other words, do yourself a favour and keep ALL processing in ASCII. You can convert between ASCII & EBCDIC on input/output. That way the modifications in order to support EBCDIC are well concentrated in a single piece of code, which can be easily merged without risk of destabilizing the code base.

You can then use your manpower, and you still need *a* *lot* of it, to write a code generator & RTL.

Daniël

Reply by email

Tomas Hajny
to Sven Barth: The RTL consists of two parts (though the border is not easily visible): a platform…
Sven Barth wrote 21 lines
On 30.01.2012 20:31, steve smithers wrote: >> Hans-Peter Diettrich wrote on Mon, 30 Jan 2012 17:40:27 +0100 >> Existing source code frequently assumes ASCII encoding. The obvious are >> upper/lowercase conversions, by and/or or add/sub constant values to the >> characters. It will be hell to find and fix all such code in the >> compiler and RTL, even if only the constants have to be modified for >> EBCDIC. Even code with the assumed order of common characters (' '< '0' >> < 'A'< 'a') has to be found and fixed manually - how would you even >> *find* code with such implicit assumptions? > > It does indeed. I am aware of the problems inherent in this. But the RTL > has to be more or less rewritten anyway to support OS. OS is a very different > animal to Windows or Linux. The RTL consists of two parts (though the border is not easily visible): a platform independant one and a platform dependant one. A port to a different target normally only includes touching the platform dependant one, but a port to 370 also requires touching the platform independant one. This is what DoDi talks about. @other devs: Could the code page aware AnsiString type be of any help here?

I suspect that this would not be enough since there are still hard- coded assumption about control codes to be located in #0..#31 on all SBCS encodings, etc.

Tomas

Reply by email

Tomas Hajny
to Daniël Mantione: Op Mon, 30 Jan 2012, schreef steve smithers: If you try to achieve a port by modifying…
Daniël Mantione wrote 27 lines
Op Mon, 30 Jan 2012, schreef steve smithers: >> Hans-Peter Diettrich wrote on Mon, 30 Jan 2012 17:40:27 +0100 >> Existing source code frequently assumes ASCII encoding. The obvious are >> upper/lowercase conversions, by and/or or add/sub constant values to the >> characters. It will be hell to find and fix all such code in the >> compiler and RTL, even if only the constants have to be modified for >> EBCDIC. Even code with the assumed order of common characters (' ' < '0' >> < 'A' < 'a') has to be found and fixed manually - how would you even >> *find* code with such implicit assumptions? > > It does indeed.  I am aware of the problems inherent in this.  But the RTL > has to be more or less rewritten anyway to support OS.  OS is a very different > animal to Windows or Linux. If you try to achieve a port by modifying all code that deals with characters you will fail. The amount of work becomes then far too big for a single person, and the modifications become too huge and wide-spread that you will raise objections for merging it with the SVN trunk. In other words, do yourself a favour and keep ALL processing in ASCII. You can convert between ASCII & EBCDIC on input/output. That way the modifications in order to support EBCDIC are well concentrated in a single piece of code, which can be easily merged without risk of destabilizing the code base. You can then use your manpower, and you still need *a* *lot* of it, to write a code generator & RTL.

Personally, I'd say that it depends on the goals. If the goals are primarily oriented to use of existing codebases on the new platform, ASCII might be the only real option (there are lots of such assumptions in most of the existing code designed for ASCII only). However, even in that case I'm still convinced that conversion on input/output _within_ the Pascal program means either restricting the programmer to using just certain parts of the standard RTL, or certain amount of changes in otherwise common files (mostly due to the fact that you need to restrict the conversion to text files, but some low-level parts of RTL do not distinguish text and binary I/O and treat both equally).

However, if creating programs specifically targetting the new platform is an important part of the goals (and the goal is targetting that mystic "OS" ;-) or MUSIC/SP, etc.) rather than Linux for S/390, I don't think that restricting I/O to ASCII is a viable option for achieving this goal.

Tomas

Reply by email

Hans-Peter Diettrich
to Sven Barth: The RTL consists of two parts (though the border is not easily visible): a platform…
Show quoted text 21 lines
On 30.01.2012 20:31, steve smithers wrote: >> Hans-Peter Diettrich wrote on Mon, 30 Jan 2012 17:40:27 +0100 >> Existing source code frequently assumes ASCII encoding. The obvious are >> upper/lowercase conversions, by and/or or add/sub constant values to the >> characters. It will be hell to find and fix all such code in the >> compiler and RTL, even if only the constants have to be modified for >> EBCDIC. Even code with the assumed order of common characters (' '< '0' >> < 'A'< 'a') has to be found and fixed manually - how would you even >> *find* code with such implicit assumptions? > > It does indeed. I am aware of the problems inherent in this. But the > RTL > has to be more or less rewritten anyway to support OS. OS is a very > different > animal to Windows or Linux. The RTL consists of two parts (though the border is not easily visible): a platform independant one and a platform dependant one. A port to a different target normally only includes touching the platform dependant one, but a port to 370 also requires touching the platform independant one. This is what DoDi talks about.

It's not anything the compiler could solve. Find out what will happen on e.g.

   for c := 'A' to 'Z' do ...
   for c := '0' to 'Z' do ...

(where the literals 'A' etc. could be named constants, or computed values)

With EBCDIC encoding the second loop will never be entered!

@other devs: Could the code page aware AnsiString type be of any help here?

Only at the I/O side, when files are read/written, or when strings (filenames!) are sent or received via the OS API. The latter reminds me of the Windows OEM charset, used in console I/O, which could be exchanged to mean EBCDIC in IBM consoles.

Unfortunately the Encoding is available only with *strings*, not with single characters. New types like EBCDICchar could be introduced, different from AnsiChar, and a directive telling the compiler "literals are EBCDIC" or "Char is EBCDICchar".

DoDi

Reply by email

Mark Morgan Lloyd
to Hans-Peter Diettrich
Hans-Peter Diettrich wrote 45 lines
Sven Barth schrieb: > On 30.01.2012 20:31, steve smithers wrote: >>> Hans-Peter Diettrich wrote on Mon, 30 Jan 2012 17:40:27 +0100 >>> Existing source code frequently assumes ASCII encoding. The obvious are >>> upper/lowercase conversions, by and/or or add/sub constant values to >>> the >>> characters. It will be hell to find and fix all such code in the >>> compiler and RTL, even if only the constants have to be modified for >>> EBCDIC. Even code with the assumed order of common characters (' '< >>> '0' >>> < 'A'< 'a') has to be found and fixed manually - how would you even >>> *find* code with such implicit assumptions? >> >> It does indeed. I am aware of the problems inherent in this. But >> the RTL >> has to be more or less rewritten anyway to support OS. OS is a very >> different >> animal to Windows or Linux. > > The RTL consists of two parts (though the border is not easily > visible): a platform independant one and a platform dependant one. A > port to a different target normally only includes touching the > platform dependant one, but a port to 370 also requires touching the > platform independant one. This is what DoDi talks about. It's not anything the compiler could solve. Find out what will happen on e.g. for c := 'A' to 'Z' do ... for c := '0' to 'Z' do ... (where the literals 'A' etc. could be named constants, or computed values) With EBCDIC encoding the second loop will never be entered! > @other devs: Could the code page aware AnsiString type be of any help > here? Only at the I/O side, when files are read/written, or when strings (filenames!) are sent or received via the OS API. The latter reminds me of the Windows OEM charset, used in console I/O, which could be exchanged to mean EBCDIC in IBM consoles. Unfortunately the Encoding is available only with *strings*, not with single characters. New types like EBCDICchar could be introduced, different from AnsiChar, and a directive telling the compiler "literals are EBCDIC" or "Char is EBCDICchar".

I'd suggest that the thing to do is to first target the compiler at Linux, i.e. ASCII, hosted on a PC. Once that is adequately working branch the RTL for EBCDIC, with the intention that this is basically a set of conversion patches and that the master remains ASCII.

Or of this isn't acceptable because the IBM developers feel we're trying to force them into our image, let's meet half way and use Solaris which nobody really enjoys.

Mark Morgan Lloyd markMLl .AT. telemetry.co .DOT. uk [Opinions above are the author's, not those of his employers or colleagues]

Reply by email