Hello FPC list,

I would like to inform you, that I ported an improved version of the Stanford Pascal compiler (a descendant of the Wirth P4 compiler) to Windows, OS/2 and Linux.

I improved this compiler, which comes from the IBM mainframe, by adding several features that I needed and that I found useful, in the past few months.

Then I started to write a P-Code interpreter on Windows, which interprets the generated P-Code; while doing this, I discovered some severe portability issues, which required additional changes in the compiler, for example: branch tables which were based on the EBCDIC char sets; difficulties with sets of char or sets of subranges of chars and so on. For all those problems, I found portable solutions, that is: extensions or changes to the P-Code, that eliminated those dependencies on the code page of the target machine.

Now the compiler runs with identical results on all the target platforms, including IBM mainframe; the generated P-Code files may be transferred freely between the platforms. Running them on the different platforms will yield the same results.

On the mainframe, the P-Code is translated to machine instructions; on all the other platforms, the P-Code is interpreted. A true P-Code translator on the non-mainframe platforms may follow later. Maybe the P-Code to machine code translater from the mainframe can be used as a starting point to do this, we will see; it does some significant optimizations which are IMO indipendent of the target machine's machine code.

BTW: the P-Code interpreter is written in ANSI-C.

Because I am working with an emulated mainframe (Hercules), the interpreted compiler on Windows is in fact faster than the machine code based compiler on Hercules (which will be not true on a real IBM machine, of course). The compiler compiles itself in 2 to 3 seconds on both platforms (using a laptop running Windows 10 or the Hercules emulator; the laptop is not brand new).

You find some stories about my efforts of the last years and months on my web site:

http://bernd-oppolzer.de/job9.htm

There are still some parts of the language missing on Windows etc., for example binary files (files, which are not files of char), and modules (that is, linking of seperately compiled program units). This will be added in the next weeks and months.

If you want to know more about this project, feel free to contact me offline.

Kind regards,
merry Christmas and a happy new year

Bernd

Well done Bernd. I remember that you were having some difficulty with character array compatibility with FPC, are any of those still a problem?

Hello Mark,

on several occasions, I looked what FPC does, when I extended the Stanford compiler, for example when I added support for direct write of scalars.

At one time I recall that I decided explicitly to take another direction; that was when I allowed shorter string constants to be assigned to longer ones, for example:

var x: array [1 .. 200] of char;
x := 'some string';

IIRC, FPC fills with hex zeroes, but I prefer blanks - the blank representation
of the target system ... which is different on the target systems; this should
show to some of the readers here which are not familiar with IBM mainframes some of the difficulties I had to get the P-Code really portable ... all chars in
the (character) P-Code file had to be converted to character constants; all places where character A - for example - was represented as numeric 193 (which is EBCDIC 'A') had to be found and corrected. Even such places where the reference to 193 was not recognized at first sight, that is: offsets in branch tables and bit strings representing sets.

This changes had to be done in the first pass of the compiler, which translates
the Pascal source to P-Code; the second pass (P-Code to machine instructions)
and the P-Code interpreter had to be adjusted accordingly.

See more information (including Pascal and P-Code examples) at the web site mentioned below.

Regarding your question: There are - of course - some more incompatibilities between FPC and Stanford Pascal; first of all, Stanford Pascal, as of now, is a
very small language, compared to FPC. My extensions of the last months are motivated by my own needs and interests (and by my C and Pascal/VS experience,
to some degree); I wanted before all a Pascal based language that works the same on
Windows, OS/2, Linux and IBM Mainframe. I didn't want to compete with FPC, not even in the long term. This looks impossible to me, from today's view.

Kind regards

Bernd

I think the issue of padding partially-initialised data structures is something that merits wider discussion. Provided of course that we can avoid the sort of arcana that Paul/Kerravon is enmired in :-)

I think you've made creditable progress in a difficult area. What are you doing about PRED() and SUCC() as applied to CHARs?

Anybody with any sort of interest in mainframes is going to have to consider EBCDIC for quite some while, but unfortunately there are still people who insist that it's flawless. One of our wiki pages has somebody confirm that EBCDIC has ^, but he then goes on to admit that it's not in all codepages...

Thank you.

I think about "portability" in a certain way; to make it clear:

of course it is possible to write programs that are not portable using my "new" compiler.

You are mentioning SUCC and PRED with CHAR; that is a very cood example. These functions are implemented based on the underlying character set; that means, that SUCC('R') is not 'S' on EBCDIC, because there is a gap between
'R' and 'S in the EBCDIC codepage (six other characters between 'R' and 'S' which are not alphabetic).

This is a portability problem which appears at the source code level (!) and cannot be healed by the compiler. It is the same with the C language, and the sensible programmer has to deal with this, if he or she wants to have his or her programs really portable.

My problems with the Stanford compiler were different; if the compiler generates
code which will not run on a platform using a different code page, because it generates
branch tables when implementing case statements that imply a certain code page,
this is a big problem and has to be fixed. The compiler implementor has to find a
representation (in the P-Code, in this case), which will work on every platform, that is:
which is independent of the code base - and does not prevent the optimizations
done by the later stages of the compiler.

Same goes for the bit string representation of sets of char; in this case, the construction
of the bit string has to be deferred until the code page can be determined (P-Code translation or interpretation time). On Windows etc., the P-Code interpreter
"translates" the P-Code to an internal representation on startup, and that's the
time when the "portable" representation of set constants (of char) are translated
to the bit string representation. See my web site for details.

Regarding ^:

"my" compiler supports different representations for the pointer symbol, and for other
critical symbols, too:

^ @ -> for the pointer symbol (I use -> most of the time) [ (. (/ for arrays
{ (* /* for comments ("comment" is supported, too, for historic reasons)

no problem with EBCDIC. I do the editing on Windows most of the time and move the
files to Hercules using the socket reader.

BTW: you find the compiler sources and a history of the extensions that I applied in the last
months (or years) on the web site, too.

Kind regards

Bernd

Bernd Oppolzer wrote

Am 24.12.2016 um 12:50 schrieb Mark Morgan Lloyd: You are mentioning SUCC and PRED with CHAR; that is a very cood example.

It's not so much a good example as a specific sticking point that was mentioned in earlier discussion on the ML.

At least a-f are consecutive, and it's still possible to switch letter case with an addition. I've used a tool in the past, written by what I have to assume was one of the Three Wise Men, which made no assumptions other than that 0-9 were consecutive: everything else was specified on the first line of every file.

Ouch :-(

/* was the form used in the first edition of Wirth's description of Pascal (might have been before Jensen was there to help out). However I'd strongly suggest deprecating @ and replacing it with another digraph, it's used as the address-of operator in Turbo Pascal and its successors.

I don't like @, too. What you told me about the address-of operator in Turbo Pascal etc. is a very strong reason to get rid of @; I will think about it.
I converted all relevant sources (including the compiler itself) to ->, so this would be ok for me.

I have a function ADDR, which gets the address of every variable (not only heap elements), automatic and static variables (which I added), too. ADDR returns an untyped pointer, which is compatible to every other pointer type, much the same as the constant NIL. There is a function PTRCAST, which casts every pointer type to this untyped pointer type. And functions PTRADD, PTRDIFF and so on ... I borrowed some of the names from PL/1.

Maybe pandora's box, but I need this to do some of the more systems related work. For example: I rewrote the LE storage management in Pascal and made it
available to the Stanford Pascal runtime (new functions ALLOC and FREE); this works perfectly with Hercules and VM; it still has to be tested with Windows - Linux - OS/2.

I added /* to the list of allowable symbols for comments, because it's what I knew
from PL/1 and C. Comments have to be terminated by the same symbol which started them, and: they may be nested (as with FPC).

Kind regards

Bernd

Leave it in the compiler as long as possible, but explicitly mark it as deprecated (/not/ depreciated :-)

The TopSpeed family used something like ADR() and SIZE() as functions with ADDRESS as an untyped pointer; ADDRESS() acted as a cast. Their Pascal compiler used :: as a type transfer that changed bit patterns where necessary, and :> as one that didn't change bit patterns.

This is indeed a very well known C style.

PTRADD, PTRDIFF and so on ...

This is not C style but with a minor macro helper something like this is quickly added to a typical C program.
The difference between two pointers is defined as the number of elements of the pointer type.
The addition or sum of two pointers is missing a bit of a straight forward logic for me.
Advanced coding style recommendations (e.g. MISRA standards) would tell you not to use such pointer math
because such constructs have a higher chance to sometimes mislead the performing programmer and thus would more likely lead up to codes with functional bugs.
The first answer for such operators often is: use the right container for the value you are in need and avoid all those operators. And the second answer is: If you are really in the true rare need (e.g. for system programming, ring buffers, atomic operations, code sizes + register savings + extreme efficiency) then keep these codes and data very isolted, very well documented and finally deeply tested for all operating conditions.

Is the your lower end of those module touching on a system API level for that?
(e.g. the Linux ABI, OS/2 system peronality, WIN32 api - rather than malloc()/free() as a C standard library provides it)

I added /* to the list of allowable symbols for comments, because it's what I knew from PL/1 and C. Comments have to be terminated by the same symbol which started them, and: they may be nested (as with FPC).

From a C program portability view I would futher "tick" for the support of the "//" comment operator as a single sided one. On the other side, even for ages there is the option of pascal-2-c converters (p2c) that do the other way around, but would have some smaller problem if such a token was suddenly added. If sources of the tool are available then someone that is in need for support could solve it easily.

just my 2 euro cents.

regards and good christmas times.
-Alex.

Alexander Stohr wrote

Am 2016-12-24 um 15:20 schrieb Bernd Oppolzer: The first answer for such operators often is: use the right container for the value you are in need and avoid all those operators.

Bear in mind that this is very old code that Bernd is maintaining, and long predates any concept of objects or of containers other than Pascal records.

Is the your lower end of those module touching on a system API level for that? (e.g. the Linux ABI, OS/2 system peronality, WIN32 api - rather than malloc()/free() as a C standard library provides it)

Again, bear in mind that one of his major targets is a late-70s vintage IBM CMS, which predates most APIs as we know them (in fact, long predates the terms API and ABI themselves :-)

Thank you for your feedback.

I'm aware of all the concerns you have.
That why I wrote about "pandora's box".
Indeed I'd prefer to use the pointer arithmetic facilities only in certain restricted circumstances, when this seems the best way to go and all other (more indirect) ways to get the work done are still more clumsy and more difficult to understand.

Anyway, it's now possible - for example - to access an array by moving a pointer over the elements, but I, too, believe, that that's not how it should
be done, regularly. BTW, I had to remove some sort of self check from the Stanford Pascal runtime; it checked (before), that pointers only pointed to the valid range of heap addresses; because the "traditional" heap consisted
of one contiguous segment, this was very easy to implement. But I had to remove this check, because pointers now can point to auto variables, static variables, "new" alloc areas etc. as well.

For your second question regarding the system API:

on the mainframe, the lower end of the LE storage management consists of the GETMAIN and FREEMAIN macros; that is, the storage management provided by the operating system. I provided for test purposes two functions ALLOCX and FREEX which map directly to GETMAIN and FREEMAIN; this worked, but was very slow. The LE storage management is optimized for C and C++ and tries to support many small areas of storage, minimizes fragmentation and optimizes performance. See this IBM presentation, which is mirrored on my web site:

http://bernd-oppolzer.de/stackheap.pdf

The problem, why I had to add this to the Pascal runtime, was: the original Stanford Pascal runtime only had functions new, mark, and release. Release releases all storage which was required since the last mark call - but there is no invidual "free" of areas. But I wanted to port an application to Stanford
Pascal which required invidiual allocs and frees (like C malloc/free). I decided
to add alloc/free and leave new/mark/release untouched for the moment, because it is used in the compiler.

ALLOCX/FREEX goes directly to GETMAIN/FREEMAIN (which is slow), but ALLOC/FREE implements the LE solution for malloc/free, which is significantly faster.

The port of ALLOC/FREE to WIndows etc. still has to be done; maybe there is no need to re-implement the LE solution, because the normal malloc/free which is available to the interpreter may be acceptable regarding performance;
so ALLOCX / FREEX could simply be the same as ALLOC / FREE on the PC.

You can find the source of PASLIBX, which implements the LE solution in Pascal, here:

http://bernd-oppolzer.de/paslibx.pas

See resources paragraph of my Stanford Pascal related web site:

http://bernd-oppolzer.de/job9.htm

Feel free to ask if you have more questions or suggestions.

Kind regards
and good christmas times to you, too

Bernd

Bernd Oppolzer wrote

Thank you for your feedback.

Thank you for your kind andswers.

It would still work for some applications. But they would be few. The vast majority of real use projects probably tend to break those box. So if it was an option with an enable switch, most would disable it. Probably for that reason it would never be worth keeping it at all.

So it was created more like a stack allocator. The allocations were local to the
functions or context they were allocated in and at some waypoint (exit/return)
all of them were released. Thats not a generic universal heap design but rather
goes to the level where gaps of sometimes named but then unused items increase over time
and on persistent would need some garbage collection (time loss!) any now and then.
But for that design as above growth and shrinkage is determined by the code path.
Under some conditions the growth might be very determined but the shrinkage is always very fixed whilst beeing much rarer. (I feel a little similarity to older stack rewinding concepts in C exception/resume features.)

I see you did wise to keep those items out of yournew codes for the project whilst keeping it untouched for the moment in the existing codes that dont interfere.

Do you see a good chance to use some larger existing code bases and test suites for verifying the compiler?
Do you have some heap tracking functionality inside so that e.g. "1234 heap Bytes lost" it printed at exit?
Is there a some debug option for stack max size tracking? Is there something for stack/heap object out of bounds writes/access? (thinking of magic word fences in between, and of heap sanity checking)

Regards, Alex.

You're welcome; I'm happy to meet someone who is interested in my work :-)

The Pascal documents say that they implemented it in a very basic manner, just to be sufficient for the needs of the compiler; and that future implementors
are free to replace the storage management by more sophisticated solutions. The heap elements allocated by the compiler are - for example - all freed, when
a certain block is completely compiled, that is: the internal lists of definitions
are freed, because they are no longer needed. I kept the mark/release logic,
because I wanted to keep this logic. The new/mark/release areas are completely
different from the alloc/free areas. The whole mark/release area is allocated
at startup and cannot be enlarged (4 MB minus stack at the moment, can be configured). The alloc/free area is allocated on an as-needed base in 64 k chunks;
limited to ca. 8 to 10 MB due to address range limitations - may grow much larger, if 31-bit addressing were possible.

The first test for the compiler is always the compiler itself (first and second pass);
it should compile itself again and again and yield the same results. Then I collected
over the time some 30 testcases, which cover different areas; esspecially the new
features that I added. I am a big fan of test driven development, so I often added new
statements and features which first lead to a compiler error, and then I implemented
them, until they worked as the should. Now these test cases are kept for regression testing.

For the LE heap management, which is a sort of addendum to the Pascal runtime
(the compiler doesn't need it):

there are functions that give statistics on heap usage at the end of the process or
at any point in time in between

there are functions that check the heap for integrity (same checks as suggested by
the IBM paper - the LE heap management technology is a product of IBM Watson
Research Center, see the presentation link some days ago)

I wrote a program to check for memory leaks (in ANSI C), which works with the
"normal" LE heap management (as provided by IBM); you call this program twice
at different points in time, and the program tells you, which areas have been
allocated and not freed in the meantime; this was very helpful when finding memory
leaks for my customers (insurance companies with PL/1 code base ... this tool and
the effort that I had to understand the LE heap management gave me the idea to
rewrite it in Pascal) - I believe, it should be possible to rewrite that in Pascal, too.

in fact, it took me some weeks and very much tests to get the heap management functions
running without errors (hopefully); there are many special cases which need special
treatment (when combining adjacent free areas etc. and maintaining the trees of
free areas). This was one of the most difficult pieces of software I wrote in the last years.

Stack and heap max size is no problem; the values are collected regularly (and printed
at process end, if you wish).

There are magic words etc. and several security measures regarding overwrite of
heap control blocks etc., but with normal processing, errors will be discovered at
free processing, only. And: not all heap areas are regularly checked at every alloc
and free request. The LE system has a HEAPCHK option, which does exactly that
(check the entire heap at every request). If needed, it would be easy to add this, because
the function to check the heap is already available.

If you want more information, see the IBM presentation: http://bernd-oppolzer.de/stackheap.pdf
or the Source code of the Heap Management:
http://bernd-oppolzer.de/paslibx.pas
or: feel free to ask further questions.

Kind regards

Bernd

On 26/12/16 19:00, Bernd Oppolzer wrote:

A thought without its being a direct comment on any part of what you've written. Programming texts from Wirth onwards tended to muddle pointers and dynamic memory allocation together, so in effect assumed that pointers would /only/ come out of the heap manager hence could be checked for consistency in that context. Early Pascal compilers neither had an addressof() function nor expected that the underlying OS would return a pointer as a syscall result.

When teaching/supporting I've found it useful to make the opposite assumption, i.e. starting off with a hypothetical OS call that returned a pointer and showing students what they could do with it, and only later introducing trees etc. on the heap.

Apart from that, I've certainly seen mark/release being the dominant mechanism in 1980s compilers, and even Turbo Pascal 3.0 (circa 1986) supports it with the explicit

"NOTICE that Dispose and Mark/Release use entirely different approaches to heap management - and never shall the twain meet! Any one program must use either Dispose or Mark/Release to manage the heap. Mixing them will produce unpredictable results."

Turbo Pascal 5.0 (circa 1989, and not long preceding Delphi) appears to adopt a complex scheme to handle both of the approaches, and observes

"Mark and Release cannot be used interchangeably with Dispose and FreeMem unless certain rules are observed [...] ."

Glad to hear that FPC is not the only oss/free pascal compiler being developed :)