Now I started another branch, that is not as "intrusive" as my other branches. The AltParser branch makes multiple parsers available, each of which sits in its own class derived from TParser. The parsers register themselves for specific file types and the compiler selects an applicable one, depending on the extension of the source file. The original parser is not affected and will be used for all file types without an registered parser.

For debugging purposes a compiler option (-m2) has been added, that selects an alternative parser, if one is registered. This allows to test the new parser by adding OPT=-m2 to the Make commandlines. Currently a message is output whenever the alternative OPL parser is used to compile an file.

In the first round the parser modules are cloned into the parserOPL unit. The original parser units will export their helper procedures, for use by every parser. The currently inlined semantic code will be moved out of the parser procedures, in multiple steps, and shall end up in its own unit(s). The extracted procedures shall represent logical tasks, that occur in every parser, e.g. dealing with declarations or common actions on program/module/subroutine/block begin and end.

Any comments?

DoDi

Now I started another branch, that is not as "intrusive" as my other branches. The AltParser branch makes multiple parsers available, each of which sits in its own class derived from TParser.

After some tries I've arrived at a working model. The semantic actions, implemented in semantic objects of various kinds, are currently structured according to their occurence in the parser code. In a later step they should be broken down into logical steps, that better reflect the *needs* of an parser (registering encountered variables...).

Some parser methods have been virtualized, so that existing code can be reused. E.g. calls to statement() or expr() are caught in a procedure, that defers the call to the according virtual method of the current parser. The original reusable procedures have been renamed, so that all calls are routed through the parser methods. Inside an parser the specific procedures can be called directly, bypassing the virtualization.

Now it should be possible to implement a modified OPL syntax, e.g. with modulaic statement sequences instead of statement blocks, block-local variables, initialized local variables, statement sequences separated by commas (C style) and whatever comes into mind. Feel free to implement your own language, based on your TParser descendant in the AltParser branch :-)

Further development should be synced better with the trunk. Please let me know about the chances, that the required hooks are merged into the trunk. When the semantic code is already separated in the general parser units, as demonstrated in the cloned p*OPL units, that code could be maintained in fully reusable semantic objects; these objects can become part of the existing parser units, so that no clones have to be maintained further.

DoDi

Hans-Peter Diettrich wrote

Further development should be synced better with the trunk. Please let me know about the chances, that the required hooks are merged into the trunk.

I won't merge it and take the burden to be responsible for a broken compiler for no gain.

What should break the compiler, in this approach? The logic is unchanged, only the control flow has become a bit different.

DoDi

See e.g. the exit problem from a few days ago how quickly something is broken.

This problem showed up immediately, and since the reason is known, it can be cured easily (check the semantic code for "exit"...).

I implemented the new parser model in parallel to the old one, so that different behaviour can be determined by a (binary) comparison of the result of both versions, what can be automated like for the compiler bootstrap. The parser units and even single procedures can be refactored one by one, unlike in my first approach (OO_parser).

Once a part (procedure) has been tested, it can replace the original procedure. All the $ifdefs can be removed in this step, they have been introduced only for easy detection of changes to the original (trunk) code. The same for the virtual parser methods, which only have been introduced to allow for parallel implementations of the very same parser. Once the switch to the new code structure has been made, the parser procedures can be called immediately, just like before.

One goal of this refactoring is the determination and documentation of the actions, required in certain pieces of the grammar. In the first step the semantic code simply reflects the actions in the existing code, i.e. *how* something is done. In further steps documentation can be added, that also reflects *why* things are done in exactly this way, possibly reveiling flaws and duplicate (maybe differerent) implementation of the essentially same tasks. Such multiple implementations can be replaced by unique semantic methods, in a layer on top of the basic procedures and methods of the involved classes. The result is a more stable compiler, where changes in one part can not break the code in other parts any more.

DoDi

One goal of this refactoring is the determination and documentation of the actions, required in certain pieces of the grammar.

Why should we need these actions?

In the first step the semantic code simply reflects the actions in the existing code, i.e. *how* something is done. In further steps documentation can be added,

Yes, can ...

I still don't see a need to get the parser more robust, if it's possible at all. The front end including the parser is one of the simplest, robust and stable parts of the compiler.

With all the drawbacks like:
- people knowing the old code for 15 years having to dig into new code
- slower
- code spread over multiple locations
- lost svn blame history
- last but not least, you coding style does not follow the compiler code style

And?

... which that makes things really hard to overlook: code spread over multiple locations for nothing.

Yes. The current parser fulfills its purpose perfectly. FPC doesn't need multiple parser or whatever.

> - slower how much?

Even if it small, multiple small changes add up.

> - code spread over multiple locations yes, the existing code could be concentrated into less units.

No, I wrote locations, see above.

> - lost svn blame history not necessarily.

You reformatted and moved a lot of code, so a lot of it is lost.

See existing code and http://wiki.freepascal.org/Coding_style and you should also know that it is changed because you re-formatted code.

> One goal of this refactoring is the determination and documentation of > the actions, required in certain pieces of the grammar. Why should we need these actions?

Yes, that's one of the big questions. What is that funny call to XYZ good for, in this particular place? What has to be changed, when a new attribute is introduced? etc...

I still don't see a need to get the parser more robust, if it's possible at all. The front end including the parser is one of the simplest, robust and stable parts of the compiler.

You seem to have missed that the parser itself is unchanged - only the connection to the related data structures (symbol tables...) is sorted out.

With all the drawbacks like: - people knowing the old code for 15 years having to dig into new code

never change a running program? ;-)

- slower

how much?

- code spread over multiple locations

yes, the existing code could be concentrated into less units.

- lost svn blame history

not necessarily.

- last but not least, you coding style does not follow the compiler code style

that is?

DoDi

That's what deserves documentation.

For good reason: a clear interface between the parser syntax and semantic. That interface finally can be located in the most appropriate location, e.g. tprocinfo for procedure attributes, tparamanager for parameters. I kept it separate for now, in order to keep the modifications to the existing units at a minimum.

See above.

I did some reformatting on the extracted semantic code, for analysis purposes. This code will become part of the new interface later, so that it will be moved in any case.

All in all I understand that you mean:
- the project deserves no documentation, SVN history is perfectly sufficient for that purpose
- since nobody knows how all the many pieces fit together, only really necessary punctual changes are allowed
- changes should be restricted to code removal, everything else might decrease compilation speed
- parallel execution is not necessary and never will be implemented, because it would cause too many changes to the existing code
- the existing code deserves no restructuring
- leading whitespace is an essential part of the code

So there's left nothing what I could do for FPC.

DoDi

Hans-Peter Diettrich wrote

So there's left nothing what I could do for FPC.

I suggest you start a git-maintained fork.
This way, developers can transition to your version gradaully, avoiding unnecessary disruption and conflicts with traditional FPC team.

Op 2010-10-19 07:19, Hans-Peter Diettrich het geskryf:

So there's left nothing what I could do for FPC.

Welcome to the club of uphill battles. :)

PS:
[this comment applies in general, not necessarily to this specific thread's feature]
I agree that code clean-up, code restructuring and refactoring are normal steps in software development (at least this is true in our company software). Code is a living thing, it evolves over time. So if someone says you can't implement that feature because code might need to be changed, I start laughing. If somebody says there is no need for that feature, I start laughing more - no one person's vision can fulfill everybody's needs. The whole point of open source software is so that various people can scratch their own itch, and fulfill their needs.

I guess that doesn't mean Florian or some other core developer must accept your patch or new features, but that's the beauty of open source software. Simply fork the project and continue with your own Object Pascal compiler. Many projects have done that in the past (the latest large project being OpenOffice, now named LibreOffice), and many times the new forks are more successful than the old - sometimes new management is all that is needed.

Regards,
- Graeme -

2010/10/19 Alexander Klenin :

Yes, when it is clear that your solution is superior, people will start to use your fork instead of the official FPC releases. Look for example at what happened with EGCS:
http://en.wikipedia.org/wiki/GNU_Compiler_Collection#EGCS_fork

So maybe you cannot do anything for FPC, except creating an Enhanced FPC.

Vincent

Vincent Snijders wrote

So maybe you cannot do anything for FPC, except creating an Enhanced FPC.

About a year ago I created a document which describes what features FPC requires to compile recent Delphi code. If someone does not know what to do for FPC then he is welcome to implement one of missing features:

http://wiki.lazarus.freepascal.org/delphi_language_features_which_fpc_does_not_have

The most wanted feature is "Unicode string support".

Best regards,
Paul Ishenin.

2010/10/19 Graeme Geldenhuys :

Yes, but I doubt this possible fork will be, but feel free to prove me wrong.

Vincent

Op 2010-10-19 08:59, Paul Ishenin het geskryf:

The most wanted feature is "Unicode string support".

What Hans-Peter meant is that even if he does implement Unicode String support, that would require changes in the compiler. Florian apparently doesn't like any changes to the compiler, plus he sees no need for Unicode String support, so why would others. ;-)

[OK, I'll stop here...]

Regards,
- Graeme -

Op 2010-10-19 09:07, Vincent Snijders het geskryf:

Yes, but I doubt this possible fork will be, but feel free to prove me wrong.

Nobody will know, until a fork has been made. As for your opinion that it will simply fail is a bit of thumb sucking. Some commercial entity could easily fork FPC, throw 20 full times developers at it, have a nice huge budget, and take Embarcadero head-on with a cross-platform compiler and with 32/64-bit support already done. You still think that would fail?

Just look at Qt today - it started with 2 developers that thought they could do better than what was currently out there. Mozilla, a fork from Netscape - a pretty good success I would think. Eclipse - an idea IBM had, thinking it would do better that what Sun offered. Firebird DB (from old Interbase), PostgreSQL, MS SQL Server (comes from the same original codebase as PostgresSQL did), MS-DOS, MS Office,... Many, many more such success cases exist. All examples of one person/entity thinking they can do a better job than the original person/entity.

Regards,
- Graeme -

Alexander Klenin wrote

I suggest you start a git-maintained fork.

Of course git is superior to svn for this purpose.

Is the "base" repository hosted on a git server ?

-Michael

Sent from my iPhone

I have been biting my tongue a lot this week and almost posted the same thing earlier on Sunday. It strikes me that DoDi wants to do some interesting, experimental and radical changes and refactoring to the codebase. A fork is the only real option. This removes the burden from the core developers, makes DoDi able to prove his theories without constantly arguing with the core developers and stops the arguments on this list. As I see it, Florian et al are less likely than ever to accept patches from him. He does not want to conform to their coding style and is rocking the status quo. DoDi sees Florian et al as a giant road block in his attempts to move FPC forwards as an academic exercise. A fork will solve this issue completely.

When did I say this? What I said was: I won't merge changes into trunk which implement support for alternative parsers (this is the topic of this thread!) because nobody could explain me yet the use of alternative parsers for FPC users. So I would spend time in something with no gain for users resulting in even less time for other problems. Maybe somebody else will merge it, but he takes the burden to fix any regressions and maintain the changed code.

The alternative is: we merge any patch we get and if it's broken, well, we can hope that somebody provides another patch. But believe me, with this approach FPC will be unusable within a few months because the typical patch supplier does not check regression tests but we currently do it and fix the stuff we took responsibility by merging.

Paul Ishenin wrote

The most wanted feature is "Unicode string support".

Which means "new Delphi Strings" with dynamic code support ?

There is a branch in the svn. How is the state and the pan ?

Other stuff recently discussed is thread support (not or not perfectly functioning TThread.Synchronize, TThread.Suspend, SendMessage, non-existing TThread.Queue, ...) there are some hints in the forums and the Bugtracker.

While most of this in fact are Lazarus-LCL originated issues, IMHO FPC is deeply involved, as it _should_ be able to use all this in projects that don't use the LCL.

-Michael

To be honest (as someone who hasn't yet posted to this topic): You sounded very much like "we don't want to merge this" with "we" as in "the core devs". It wasn't really clear (at least to me and maybe some others as well) that you only spoke for yourself.

And regarding Graeme's comment about Unicode String support: I believe that was meant in a sarcastic way.

Regards,
Sven

Graeme Geldenhuys wrote

'plus he sees no need for Unicode String support, so why would others. ;-)

Some colleagues of mine need to port their projects from non-Delphi to the new Unicode enabled version.

They are "not amused".

I did try to work with the Unicode-enabled Lazarus (that abuses "ANSI"-Strings as UTF-8 Strings".

Yak !

And I in the German Lazarus Forum watched several beginners trying to use this Lazarus version for doing programs dealing with German text.

They were a lot more than "not amused" but rather deeply frustrated.

-Michael

I meant to write:

-->> Some colleagues of mine need to port their projects from non-Unicode Delphi to the new Unicode enabled Delphi version.

Sorry
-Michael

When did I write "we"? I think my initial statement was pretty clear: http://lists.freepascal.org/lists/fpc-devel/2010-October/022335.html

Changes to the compiler are allowed and have been implemented at various times with very good results. Paul is living proof of this.

But in all cases the changes are
- smaller in scope.
- desired features.

I don't think you would take it kindly if I showed up with an interior house designer and re-decorated your whole house in some flashy 70-ies hippy look, just because I think it looks better.
You'd probably first ask him to do your son's room - as a test - and see how that turns out. Then maybe you'd let him work on the rest of your house.

Same is true here.

Currently all we got from Hans Peter were unmanageable patches supposedly fixing things that were not broken or in need of fixing.

If he had first done some smaller things - take your pick in the bugtracker
- and we had seen that this goes well, then maybe such pervasive changes would be discussable.

You also would't let the handyman from around the corner construct a 10-story appartment building just like that, I suppose, and then go and happily live in the penthouse ?

In each case, this is the "chronicle of a death foretold". Not because we don't want changes, but because all guiding road signs we put up have been ignored since day 1. Just read the mail threads. It starts with the discussion that the fcl-passrc parser doesn't handle all compiler constructs. Instead of fixing that, Hans-Peter went off on a completely different angle, for which none of the core developers saw any future, and on more than one occasion pointed this out. And here we are now...

That we would be accused of not allowing changes, this also was foretold.

Michael.

Yes, you didn't write "we". But it came along as such. That might be, because you - as the initial developer of FPC - are more or less the leader of this project and as such an "I" from you is often perceived as a "we", although you might not mean it in that way. That's a psychological problem ^^

Regards,
Sven

Op 2010-10-19 10:18, Michael Schnell het geskryf:

> The most wanted feature is "Unicode string support". Which means "new Delphi Strings" with dynamic code support ? There is a branch in the svn. How is the state and the pan ?

I still have local changes that add more unicode support and improved unicode support under Linux, but the unicode branch is currently not compilable on my system - due to the latest merge done in that branch. So my work regarding that branch is on hold.

Regards,
- Graeme -

Michael Van Canneyt wrote

[...]

Agreed 100%. FPC is used in *production*!
We can't accept experimental changes of the compiler architecture in trunk. Also refactoring because of personal preferences, aesthetics or new design patterns or paradigms must be handled very carefully so the quality of the compiler really increases over time. There is a big danger of degrading the stability by such changes which are not planned by the core developers.
Many thanks to the FPC core team that they try to steadily improve the quality of the Free Pascal compiler and do not allow scientific experiments with the codebase.

Martin

Graeme Geldenhuys wrote

I still have local changes that add more unicode support and improved unicode support under Linux, but the unicode branch is currently not compilable on my system - due to the latest merge done in that branch. So my work regarding that branch is on hold.

You still have not said a single time what the actual problem is that you are having.

Jonas

+1
The current situation is frustrating for everybody.

I understood DoDi's one goal is to refactor the front end so that other language parsers can be added. First Pascal-like languages and later C. FPC's main developer doesn't see a need for it.
That is a fundamental difference. A fork is the only solution.

Competition can be good and beneficial, too.
If DoDi really comes up with a stable compiler that supports many languages, people start asking why FPC didn't do the same. Then maybe things get merged. It will not happen soon though. DoDi's changes are so fundamental that they inevitably break something first.

I promise follow closely and test the fork if it becomes a reality.

Juha

Op 2010-10-19 10:37, Michael Van Canneyt het geskryf:

Changes to the compiler are allowed and have been implemented at various times with very good results. Paul is living proof of this.

...and you totally missed my sarcastic intent with that message.

DoDi's path forward seems clear... Fork the project to experiment without obstructions to prove his worth, or use an "official" SVN branch to experiment, or use a Git branch/repository to experiment. Personally I use the later quite frequent with Lazarus and fpGUI - I have some 15+ local branches with various experimental changes for Lazarus and fpGUI (and it's many sub projects).

I have given up contributing my changes to Lazarus, for the same reason Dodi is struggling now (irrespective if the feature is big or small). This doesn't bothers me any more, because I still benefit from my efforts and changes, while others don't - it's their loss not mine. Thank you for open source software!

[...DoDi, here is my Git workflow...]
I simply reset my local custom lazarus branch to the head svn branch (this way I pick up all the new bugs, bugfixes and features), merge in the various feature branches and be done with it. Occasionally I rebase my feature branch to keep them in sync with svn head development - this near eliminates chances for conflicts (99.9% of the time) when I do merges with my feature branches. I've been working like this for months with great success. It keeps me happy, and I mostly stay out of the way of the lazarus team.

Regards,
- Graeme -

Op 2010-10-19 10:51, Jonas Maebe het geskryf:

You still have not said a single time what the actual problem is that you are having.

I believe I have, but I can't remember all the errors, and I have no time to search the mail archive.... Just tried to compile that branch again under 64-bit linux.

-----------------------------------------
Free Pascal Compiler version 2.4.3 [2010/08/02] for x86_64 Copyright (c) 1993-2009 by Florian Klaempfl
Target OS: Linux for x86-64
Compiling pp.pas
Compiling catch.pas
Compiling verbose.pas
Compiling cutils.pas
Compiling globtype.pas
Compiling finput.pas
Compiling cclasses.pas
Compiling cstreams.pas
Compiling comphook.pas
Compiling systems.pas
Compiling globals.pas
Compiling cfileutl.pas
Compiling globals.pas
Compiling ./x86_64/cpuinfo.pas
Compiling version.pas
Compiling cmsgs.pas
Compiling fmodule.pas
Compiling ogbase.pas
Compiling owbase.pas
Compiling aasmbase.pas
Compiling fmodule.pas
Compiling symbase.pas
Compiling symconst.pas
Compiling symsym.pas
Compiling widestr.pas
Compiling cp8859_1.pas
Compiling cp850.pas
Compiling cp437.pas
Compiling cp1251.pas
Compiling cp866.pas
Compiling cp8859_5.pas
Compiling constexp.pas
Compiling symtype.pas
Compiling ppu.pas
Compiling fpccrc.pas
Compiling crefs.pas
Compiling fmodule.pas
Compiling symsym.pas
Compiling symdef.pas
Compiling tokens.pas
Compiling node.pas
Compiling ./x86/cpubase.pas
Compiling cgbase.pas
Compiling rgbase.pas
Compiling cgutils.pas
Compiling optbase.pas
Compiling nutils.pas
Compiling symsym.pas
Compiling symdef.pas
Compiling aasmtai.pas
Compiling aasmdata.pas
Compiling aasmtai.pas
Compiling fmodule.pas
Compiling symsym.pas
Compiling symdef.pas
Compiling parabase.pas
Compiling symsym.pas
Compiling symdef.pas
Compiling ./x86/aasmcpu.pas
Compiling aasmsym.pas
Compiling symsym.pas
Compiling defcmp.pas
Compiling symtable.pas
Compiling symsym.pas
Compiling symnot.pas
Compiling defutil.pas
Compiling symtable.pas
Compiling symutil.pas
Compiling procinfo.pas
Compiling optutils.pas
Compiling nbas.pas
Compiling ./x86/aasmcpu.pas
Compiling ./x86/itcpugas.pas
Compiling pass_1.pas
Compiling procinfo.pas
Compiling tgobj.pas
Compiling cgobj.pas
Compiling rgobj.pas
Compiling fmodule.pas
Compiling wpobase.pas
Compiling scanner.pas
Compiling switches.pas
Compiling dbgbase.pas
Compiling options.pas
Compiling comprsrc.pas
Compiling script.pas
Compiling rabase.pas
Compiling ./systems/i_bsd.pas
Compiling rescmn.pas
Compiling paramgr.pas
Compiling ncgrtti.pas
Compiling nutils.pas
Compiling ncon.pas
Compiling nld.pas
Compiling htypechk.pas
Compiling ncnv.pas
Compiling ncal.pas
Compiling ninl.pas
Compiling nadd.pas
Compiling nmat.pas
Compiling nset.pas
Compiling nflw.pas
Compiling optloop.pas
Compiling nflw.pas
Compiling nmem.pas
Compiling nset.pas
Compiling ncgmem.pas
Compiling pass_2.pas
Compiling ncgutil.pas
Compiling export.pas
Compiling regvars.pas
Compiling psub.pas
Compiling optdfa.pas
Compiling import.pas
Compiling gendef.pas
Compiling pbase.pas
Compiling pstatmnt.pas
Compiling pexpr.pas
Compiling pinline.pas
Compiling ptype.pas
Compiling nobj.pas
Compiling pdecsub.pas
Compiling ./x86_64/cpupara.pas
Compiling objcutil.pas
Compiling objcdef.pas
Compiling pdecl.pas
Compiling ptconst.pas
Compiling pdecvar.pas
Compiling asmutils.pas
Compiling pdecobj.pas
Compiling pexports.pas
Compiling ./x86_64/cgcpu.pas
Compiling ./x86/cgx86.pas
Compiling ./x86/rgx86.pas
Compiling ./x86_64/rgcpu.pas
Compiling opttail.pas
Compiling optcse.pas
optcse.pas(171,76) Warning: Conversion between ordinals and pointers is not portable
optcse.pas(168,24) Warning: Converting pointers to signed integers may result in wrong comparison results and range errors, use an unsigned type instead.
optcse.pas(234,24) Warning: Converting pointers to signed integers may result in wrong comparison results and range errors, use an unsigned type instead.
optcse.pas(266,29) Warning: Converting pointers to signed integers may result in wrong comparison results and range errors, use an unsigned type instead.
optcse.pas(284,22) Warning: Converting pointers to signed integers may result in wrong comparison results and range errors, use an unsigned type instead.
Compiling aopt.pas
Compiling aoptobj.pas
Compiling aoptbase.pas
Compiling ./x86_64/aoptcpub.pas
Compiling aoptda.pas
Compiling aoptobj.pas
Compiling ./x86_64/aoptcpu.pas
Compiling ./x86_64/aoptcpud.pas
Compiling objcgutl.pas
Compiling nopt.pas
Compiling nobjc.pas
symdef.pas(5128,10) Warning: User defined: : this is completely wrong on so many levels...
Compiling ogmap.pas
Compiling compiler.pas
Compiling parser.pas
Compiling fppu.pas
Compiling wpoinfo.pas
Compiling scandir.pas
Compiling psystem.pas
Compiling pmodules.pas
Compiling link.pas
Compiling owar.pas
Compiling assemble.pas
Compiling cresstr.pas
cresstr.pas(149,178) Error: Identifier not found "DefaultSystemCodePage" cresstr.pas(165,117) Error: Identifier not found "DefaultSystemCodePage" cresstr.pas(170,126) Error: Identifier not found "DefaultSystemCodePage" cresstr.pas(320) Fatal: There were 3 errors compiling module, stopping

-----------------------------------------

Regards,
- Graeme -

Op 2010-10-19 10:06, Michael Schnell het geskryf:

Yes, I maintain the mirror on GitHub.

http://github.com/graemeg/freepascal

For more details see...

http://wiki.lazarus.freepascal.org/git_mirrors

Regards,
- Graeme -

Michael Schnell wrote

And I in the German Lazarus Forum watched several beginners trying to use this Lazarus version for doing programs dealing with German text. They were a lot more than "not amused" but rather deeply frustrated.

?????

They were deeply frustrated that we have a simple and reliable Unicode support which supports any language in the world?

Please move this branch of the discussion to the lazarus (or lazarus-other) list.

Jonas
FPC mailing lists admin

^^^^^ ^^^^^^^^^^

What if you use a released compiler as starting compiler ? (other versions are not supported)

Marc

Op 2010-10-19 11:31, Marc Weustink het geskryf:

What if you use a released compiler as starting compiler ? (other versions are not supported)

Like I said, this was just a quick test so I could post some output. Originally I did use the latest released compiler - 2.4.0 I believe. I'll try later today with 2.4.0 and see what it does, just to confirm.

Regards,
- Graeme -

Graeme Geldenhuys wrote

cresstr.pas(149,178) Error: Identifier not found "DefaultSystemCodePage" cresstr.pas(165,117) Error: Identifier not found "DefaultSystemCodePage" cresstr.pas(170,126) Error: Identifier not found "DefaultSystemCodePage" cresstr.pas(320) Fatal: There were 3 errors compiling module, stopping

I'm not sure how that's related to the merge I did. DefaultSystemCodePage is a system unit variable that was introduced in the cpstrnew branch. The error message suggests that you are trying to compile the cpstrnew compiler against an RTL from a different version. The cpstrnew compiler depends on the cpstrnew RTL.

Jonas

Can it be then that the compiler (or at least that part of the code) is not checking whether its compiled using an older or a newer RTL?

Isn't the process normally "compile new compiler with old RTL, compile new RTL with this compiler, compile new compiler with new RTL"?

Regards,
Sven

The normal process is "make all" in the top level fpc directory, or "make cycle" in the fpc/compiler directory. Both first compile the new RTL. It's the opposite that sometimes has to be done: the RTL always has to be compilable by older compilers.

Jonas

Ah ok. So I got this one wrong. Thanks for clarification.

Regards,
Sven

I personally wanted to do some things for FPC (although Unicode was not one of them),
but hesitated, because the current code is too messy and fragile to touch, and refactorings are not welcome.
So if there would be a fork with better code structure, it is quite plausible that I would contribute.

Op 2010-10-19 10:49, Martin Schreiber het geskryf:

FPC is used in *production*!

I agree, but this shouldn't prevent enhancements, forward thinking and innovation.

We can't accept experimental changes of the compiler architecture in trunk.

For production use, and for "stability", that is what the releases are for. Stick to those then. Trunk is a development branch, you should *not* be using in for production work, because it is unstable, and can break in any commit.

Like I've said a million times before, the management of the FPC project seems a bit off (sorry if this offends some of you, don't take it personally). Hardly anybody looks at the various branches, so to truly test something new, a workflow similar to the git.git repository needs to be adopted. Feature branches are used, a experimental/development branch exists where feature branches are merged together. This is the branch most would test (in terms of SubVersion speak, consider this Trunk), and inherently test those features too. Features in this branch are not guaranteed to make it into the release branch, but importantly, it gives them a time to shine and played with. A more stable branch should exist. This contains features that have been tried and tested, and got promoted to the next level. It is this branch that is used to cut new releases from. At that point "fixes" branches could be cut from the release branch. Fixes applied to the fixes branch can be merged back into the stable branch and vice-versa.

If svn is not up to the job (I'm not saying it is), or seems like to much work using svn, then the SubVersion software is the wrong tool for the job, because what I described above is done with very little effort using Git. In the git.git repository, all this is managed by a single person in his spare time, and the git development is a 1000x faster than FPC's, with hundreds of contributors.

Also refactoring because of personal preferences, aesthetics or new design patterns or paradigms must be handled very carefully

Yes, there goal is normally to make code more manageable and easier understood by others - indirectly making contributions easier.

There is a big danger of degrading the stability by such changes which are not planned by the core developers.

That is what unit testing frameworks where designed for. Unit test the compiler code, so there is no surprises and no guessing required to know if something broke something else or not. FPC has some tests, but I have no idea how extensive they are. Last time I looked at the various tests results published on the web (I can't remember that URL of the top of my head), the outcome looks rather sad - with many many failed tests. I'm not sure of all those failures were false negatives though.

Regards,
- Graeme -

Op 2010-10-19 11:43, Jonas Maebe het geskryf:

I'm not sure how that's related to the merge I did. DefaultSystemCodePage is a system unit variable that was introduced in the cpstrnew branch.

What good is a merge that breaks the compiling of the compiler?

The error message suggests that you are trying to compile the cpstrnew compiler against an RTL from a different version. The cpstrnew compiler depends on the cpstrnew RTL.

I simply loaded up the compiler/ppx86_64.lpi project, set the compiler to be used by Lazarus IDE, and clicked Build All. I didn't even get as far as compiling the RTL - I'm just trying to build the compiler itself, which worked before the merge. I have a separate script which uses the new compiler to build the RTL of that branch, but like I said, I haven't even gotten that far.

Regards,
- Graeme -

As I said, your problem is unrelated to the merge, and the merge did not break compiling the compiler. Prior to the merge, DefaultSystemCodePage was a) already in the RTL of the cpstrnew branch
b) already used in the compiler of the cpstrnew branch.

If everything compiled fine for you before, then maybe you had a compiled version of the cpstrnew branch somewhere that is no longer used now (maybe because a ppu version changed, or because it wasn't compiled with -Ur and the compiler is finding new versions of RTL sources, ...).

As mentioned in other mails, the compilation order is "first the new RTL, then the new compiler"

Jonas

2010/10/19 Graeme Geldenhuys :

Given the fact that 20 full time developer work on it, I think they have a reasonable chance to succeed.

It can be done, but it must be done, to prove it. Just talk won't do it.

Vincent

Op 2010-10-19 12:33, Jonas Maebe het geskryf:

As mentioned in other mails, the compilation order is "first the new RTL, then the new compiler"

I saw your reply to Sven. I didn't know that, and never used it like that before. I guess I was lucky.

Anyway, just downloaded and installed a fresh copy of FPC 2.4.0 from SourceForge. "make cycle FPC=..." still fails.... I when I have the time, I'll take another look, but I can't spend more time on it now.

Did you actually test the cpstrnew branch by compiling it yourself - after the merge?

--------------------------------------------
~ $ cd /opt/git/fpc_cpstrnew/compiler/
compiler $ make cycle FPC=/opt/fpc-2.4.0/bin/fpc
make tempclean ppc3
make[1]: Entering directory `/opt/git/fpc_cpstrnew/compiler' /bin/rm -f ppcrossx64 ppc ppc1 ppc2 ppc3 ./msg2inc pp1.wpo pp2.wpo make 'OLDFPC=' next
make[2]: Entering directory `/opt/git/fpc_cpstrnew/compiler' make rtlclean rtl
make[3]: Entering directory `/opt/git/fpc_cpstrnew/compiler' make -C ../rtl clean
make[4]: Entering directory `/opt/git/fpc_cpstrnew/rtl' /bin/rm -f fpcmade.x86_64-linux Package.fpc ppas.sh script.res link.res /bin/rm -f *.s *_ppas.sh
make -C linux clean
make[5]: Entering directory `/opt/git/fpc_cpstrnew/rtl/linux' /bin/rm -f ../../rtl/units/x86_64-linux/prt0.o
../../rtl/units/x86_64-linux/dllprt0.o ../../rtl/units/x86_64-linux/cprt0.o ../../rtl/units/x86_64-linux/gprt0.o
/bin/rm -f ../../rtl/units/x86_64-linux/system.ppu ../../rtl/units/x86_64-linux/unixtype.ppu
../../rtl/units/x86_64-linux/ctypes.ppu
../../rtl/units/x86_64-linux/baseunix.ppu
../../rtl/units/x86_64-linux/strings.ppu
../../rtl/units/x86_64-linux/objpas.ppu
../../rtl/units/x86_64-linux/macpas.ppu
../../rtl/units/x86_64-linux/syscall.ppu
../../rtl/units/x86_64-linux/unixutil.ppu
../../rtl/units/x86_64-linux/fpintres.ppu
../../rtl/units/x86_64-linux/heaptrc.ppu
../../rtl/units/x86_64-linux/lineinfo.ppu
../../rtl/units/x86_64-linux/lnfodwrf.ppu
../../rtl/units/x86_64-linux/termio.ppu
../../rtl/units/x86_64-linux/unix.ppu
../../rtl/units/x86_64-linux/linux.ppu
../../rtl/units/x86_64-linux/initc.ppu
../../rtl/units/x86_64-linux/cmem.ppu ../../rtl/units/x86_64-linux/x86.ppu ../../rtl/units/x86_64-linux/ports.ppu ../../rtl/units/x86_64-linux/crt.ppu ../../rtl/units/x86_64-linux/printer.ppu
../../rtl/units/x86_64-linux/linuxvcs.ppu
../../rtl/units/x86_64-linux/sysutils.ppu
../../rtl/units/x86_64-linux/typinfo.ppu
../../rtl/units/x86_64-linux/math.ppu
../../rtl/units/x86_64-linux/matrix.ppu
../../rtl/units/x86_64-linux/varutils.ppu
../../rtl/units/x86_64-linux/charset.ppu
../../rtl/units/x86_64-linux/ucomplex.ppu
../../rtl/units/x86_64-linux/getopts.ppu
../../rtl/units/x86_64-linux/errors.ppu
../../rtl/units/x86_64-linux/sockets.ppu
../../rtl/units/x86_64-linux/gpm.ppu ../../rtl/units/x86_64-linux/ipc.ppu ../../rtl/units/x86_64-linux/serial.ppu
../../rtl/units/x86_64-linux/terminfo.ppu
../../rtl/units/x86_64-linux/dl.ppu
../../rtl/units/x86_64-linux/dynlibs.ppu
../../rtl/units/x86_64-linux/video.ppu
../../rtl/units/x86_64-linux/mouse.ppu
../../rtl/units/x86_64-linux/keyboard.ppu
../../rtl/units/x86_64-linux/variants.ppu
../../rtl/units/x86_64-linux/types.ppu
../../rtl/units/x86_64-linux/dateutils.ppu
../../rtl/units/x86_64-linux/sysconst.ppu
../../rtl/units/x86_64-linux/fmtbcd.ppu
../../rtl/units/x86_64-linux/cthreads.ppu
../../rtl/units/x86_64-linux/classes.ppu
../../rtl/units/x86_64-linux/fgl.ppu
../../rtl/units/x86_64-linux/convutils.ppu
../../rtl/units/x86_64-linux/stdconvs.ppu
../../rtl/units/x86_64-linux/strutils.ppu
../../rtl/units/x86_64-linux/rtlconsts.ppu
../../rtl/units/x86_64-linux/dos.ppu
../../rtl/units/x86_64-linux/objects.ppu
../../rtl/units/x86_64-linux/cwstring.ppu
../../rtl/units/x86_64-linux/fpcylix.ppu
../../rtl/units/x86_64-linux/clocale.ppu
../../rtl/units/x86_64-linux/exeinfo.ppu
../../rtl/units/x86_64-linux/syslinux.ppu
../../rtl/units/x86_64-linux/linux.ppu
/bin/rm -f ../../rtl/units/x86_64-linux/system.o
../../rtl/units/x86_64-linux/unixtype.o
../../rtl/units/x86_64-linux/ctypes.o
../../rtl/units/x86_64-linux/baseunix.o
../../rtl/units/x86_64-linux/strings.o
../../rtl/units/x86_64-linux/objpas.o ../../rtl/units/x86_64-linux/macpas.o ../../rtl/units/x86_64-linux/syscall.o
../../rtl/units/x86_64-linux/unixutil.o
../../rtl/units/x86_64-linux/fpintres.o
../../rtl/units/x86_64-linux/heaptrc.o
../../rtl/units/x86_64-linux/lineinfo.o
../../rtl/units/x86_64-linux/lnfodwrf.o
../../rtl/units/x86_64-linux/termio.o ../../rtl/units/x86_64-linux/unix.o ../../rtl/units/x86_64-linux/linux.o ../../rtl/units/x86_64-linux/initc.o ../../rtl/units/x86_64-linux/cmem.o ../../rtl/units/x86_64-linux/x86.o ../../rtl/units/x86_64-linux/ports.o ../../rtl/units/x86_64-linux/crt.o ../../rtl/units/x86_64-linux/printer.o
../../rtl/units/x86_64-linux/linuxvcs.o
../../rtl/units/x86_64-linux/sysutils.o
../../rtl/units/x86_64-linux/typinfo.o ../../rtl/units/x86_64-linux/math.o ../../rtl/units/x86_64-linux/matrix.o
../../rtl/units/x86_64-linux/varutils.o
../../rtl/units/x86_64-linux/charset.o
../../rtl/units/x86_64-linux/ucomplex.o
../../rtl/units/x86_64-linux/getopts.o
../../rtl/units/x86_64-linux/errors.o
../../rtl/units/x86_64-linux/sockets.o ../../rtl/units/x86_64-linux/gpm.o ../../rtl/units/x86_64-linux/ipc.o ../../rtl/units/x86_64-linux/serial.o ../../rtl/units/x86_64-linux/terminfo.o ../../rtl/units/x86_64-linux/dl.o ../../rtl/units/x86_64-linux/dynlibs.o ../../rtl/units/x86_64-linux/video.o ../../rtl/units/x86_64-linux/mouse.o
../../rtl/units/x86_64-linux/keyboard.o
../../rtl/units/x86_64-linux/variants.o
../../rtl/units/x86_64-linux/types.o
../../rtl/units/x86_64-linux/dateutils.o
../../rtl/units/x86_64-linux/sysconst.o
../../rtl/units/x86_64-linux/fmtbcd.o
../../rtl/units/x86_64-linux/cthreads.o
../../rtl/units/x86_64-linux/classes.o ../../rtl/units/x86_64-linux/fgl.o ../../rtl/units/x86_64-linux/convutils.o
../../rtl/units/x86_64-linux/stdconvs.o
../../rtl/units/x86_64-linux/strutils.o
../../rtl/units/x86_64-linux/rtlconsts.o ../../rtl/units/x86_64-linux/dos.o ../../rtl/units/x86_64-linux/objects.o
../../rtl/units/x86_64-linux/cwstring.o
../../rtl/units/x86_64-linux/fpcylix.o
../../rtl/units/x86_64-linux/clocale.o
../../rtl/units/x86_64-linux/exeinfo.o ../../rtl/units/x86_64-linux/linux.o /bin/rm -f ../../rtl/units/x86_64-linux/math.rst
../../rtl/units/x86_64-linux/varutils.rst
../../rtl/units/x86_64-linux/typinfo.rst
../../rtl/units/x86_64-linux/variants.rst
../../rtl/units/x86_64-linux/sysconst.rst
../../rtl/units/x86_64-linux/rtlconsts.rst
../../rtl/units/x86_64-linux/stdconvs.rst
/bin/rm -f fpcmade.x86_64-linux Package.fpc ppas.sh script.res link.res /bin/rm -f *.s *_ppas.sh
make[5]: Leaving directory `/opt/git/fpc_cpstrnew/rtl/linux' make[4]: Leaving directory `/opt/git/fpc_cpstrnew/rtl' make -C ../rtl 'OPT=' all
make[4]: Entering directory `/opt/git/fpc_cpstrnew/rtl' make -C linux all
make[5]: Entering directory `/opt/git/fpc_cpstrnew/rtl/linux' as --64 -o ../../rtl/units/x86_64-linux/prt0.o x86_64/prt0.as as --64 -o ../../rtl/units/x86_64-linux/dllprt0.o x86_64/dllprt0.as as --64 -o ../../rtl/units/x86_64-linux/cprt0.o x86_64/cprt0.as as --64 -o ../../rtl/units/x86_64-linux/gprt0.o x86_64/gprt0.as /opt/fpc-2.4.0/bin/fpc -Fi../inc -Fi../x86_64 -Fi../unix -Fix86_64 -FE. -FU../../rtl/units/x86_64-linux -Cg -dx86_64 -Us -Sg system.pp Free Pascal Compiler version 2.4.0 [2009/12/18] for x86_64 Copyright (c) 1993-2009 by Florian Klaempfl
thread.inc(411,10) Warning: Function result does not seem to be set /opt/fpc-2.4.0/bin/fpc -Fi../inc -Fi../x86_64 -Fi../unix -Fix86_64 -FE. -FU../../rtl/units/x86_64-linux -Cg -dx86_64 ../unix/unixtype.pp Free Pascal Compiler version 2.4.0 [2009/12/18] for x86_64 Copyright (c) 1993-2009 by Florian Klaempfl
Fatal: Can't find unit objpas used by unixtype
Fatal: Compilation aborted
Error: /opt/fpc-2.4.0/bin/ppcx64 returned an error exitcode (normal if you did not specify a source file to be compiled)
make[5]: *** [unixtype.ppu] Error 1
make[5]: Leaving directory `/opt/git/fpc_cpstrnew/rtl/linux' make[4]: *** [linux_all] Error 2
make[4]: Leaving directory `/opt/git/fpc_cpstrnew/rtl' make[3]: *** [rtl] Error 2
make[3]: Leaving directory `/opt/git/fpc_cpstrnew/compiler' make[2]: *** [next] Error 2
make[2]: Leaving directory `/opt/git/fpc_cpstrnew/compiler' make[1]: *** [ppc1] Error 2
make[1]: Leaving directory `/opt/git/fpc_cpstrnew/compiler' make: *** [cycle] Error 2

--------------------------------------------

Regards,
- Graeme -

Yes, but it was on Mac OS X. At first sight, there are however no differences between the Makefile.fpc of the cpstrnew branch and the one in trunk (other than that trunk as an extra iso7185 unit for iso mode support), so I'm not sure what's going on in what you posted. I have to run now, I'll look into it later.

Jonas

Ah, just to close the issue. The original text from Michael Schnell was ambiguous and I understood he was talking about the Lazarus unicode support, but now I see that he could be talking about the Free Pascal unicode support ... which are 2 different things.

Indeed, if he was about Lazarus Unicode support the discussion bellongs elsewhere.

If I see this correctly, the makefile is trying to compile "unixtype" before "objpas" is compiled. Shouldn't "objpas" be the second unit that is compiled? Or was something added to "unixtype" that makes it depend on "objpas" now while that wasn't the case before?

Regards,
Sven

In our previous episode, Graeme Geldenhuys said:

http://www.stack.nl/~marcov/buildfaq/#toc-Section-2.2

http://www.stack.nl/~marcov/buildfaq/#toc-Section-1.7

Indeed, it's just a matter of time which seems nobody to have, see e.g. cpstrnew branch: very few people test, nobody contributes or why is compilation broken? Even more it even doesn't affect the compiler, the rtl needs fixing, we need tests etc. Where are the fixes and enhancments for the rtl? Where are the tests?

a workflow similar to the git.git repository needs to be adopted.

It has nothing to do with workflow.

spare time, and the git development is a 1000x faster than FPC's, with hundreds of contributors.

See above about the cpstrnew branch. But if git helps a lot, where can I pull the fixes to the cpstrnew branch? Where are the other git repositories containing the great patches to FPC? You maintain your git mirror already for years, there must be some cloned repositories and forks with a lot of contributions. I guess a lot of it could go into trunk, git format-patch -XX together with svn-apply makes this really easy, just tell me the git urls ;)

See cpstrnew branch, it has nothing to do with the compiler, the rtl needs fixing.

For the compiler they are very extensive, not for the rtl/packages/fcl.

Last time I looked at the various tests results published on the web (I can't remember that URL of the top of my head),

http://www.freepascal.org/testsuite/cgi-bin/testsuite.cgi

the outcome looks rather sad - with many many failed tests. I'm not sure of all those failures were false negatives though.

Some tests are just added to remember that something is broken. Actually, what we do are regression testing. Newly broken stuff must be fixed immediatly, tests which worked never before will be fixed if possible.

In which areas?

and refactorings are not welcome.

Who says this? Usefull refactoring *is* welcome.

Juha Manninen (gmail) wrote

First Pascal-like languages and later C. FPC's main developer doesn't see a need for it.

Indeed. Please tell what a C front end to FPC helps? FPC has no advantage over existing C compilers except being a pascal compiler ;). Do you think any C developer will work on a C compiler written in pascal? It won't make interfacing C headers easier as explained multiple times. So what do you expect from a C front end?

I have had plans/ideas in many areas,
from fixing case and reformatting of Math unit
to finishing/extending Paul's work on for..in loop.

> and refactorings are not welcome. Who says this? Usefull refactoring *is* welcome.

But here I see, again and again, that good refactorings are rejected as "not useful". Well, maybe I should try myself but the experience of others is definitely discouraging.

Actually, I certainly think (and hope) that DoDi's goal is _not_ a "C frontend", but to find _any_ possible venue to upgrade the code structure of FPC code. I admire his persistence and productivity and my suggestion for a fork was not a trolling attempt (although it did start a flame, for which I am sorry).
To get going, a fork needs some momentum and dedication and maybe he is the person able to pull it off.

This does not help anybody ;)

to finishing/extending Paul's work on for..in loop.

Does it still miss stuff? Actually, I never used it :)

So you looked yourself into the refactored code so you can judge it as good? Did you send an email with your comment why it is good? Where did you comment on the critism I made?

Well, maybe I should try myself but the experience of others is definitely discouraging.

Who is others? I remember very few *compiler* patches we rejected.

Florian Klaempfl wrote

> to finishing/extending Paul's work on for..in loop. Does it still miss stuff? Actually, I never used it :)

Perhaps Alexander is talking about the points mentioned here: http://wiki.freepascal.org/for-in_loop#Proposed_extensions

I am currently starting to use them myself (with a 2.4.3 at work and maybe a 2.4.2 at home :D ). For standard tasks they are indeed a very useful kind of sugar :P

Regards,
Sven

The topic of this thread and the patch was/is alternative parsers. As you worked on the case of string stuff (or your student) you might also know that the fpc parser is written straight forward and anybody knowing the dragon books might be able to work on it and far from being fragile. This why I see no use in the patch: the parser is understandable, maintainable and reasonable fast.

Fragile are other parts: e.g. unit loading.

I'm not going to discuss (again) what a beginner will do and what he will see when stepping character by character through a String containing German text ;)

-Michael

Florian Klaempfl wrote

Am 19.10.2010 13:54, schrieb Alexander Klenin: > from fixing case and reformatting of Math unit This does not help anybody ;)

See, you have already rejected it ;-)
The initial motivation was that current unit have lower-case procedure names, which is not Delphi-compatible
and messes up code completion.

> to finishing/extending Paul's work on for..in loop. Does it still miss stuff? Actually, I never used it :)

It does not "miss" stuff, but could be extended.
I meant http://wiki.freepascal.org/for-in_loop#Proposed_extensions

So you looked yourself into the refactored code so you can judge it as good? Did you send an email with your comment why it is good?

Actually, I tried to, although svn made it extremely difficult and I am not aware of git/hg mirror of branches.
The changes are very promising (although IFDEF's are a mess, but as I understand they are planned for removal). I did not test and have not enough understanding of FPC internals to spot any breakage by code inspection, but the readability and modularization improvements are quite obvious to me.

Where did you comment on the critism I made?

I have really restrained myself from commenting on them because I it is hard to do in a polite manner.
Since you insist, I will try:

- people knowing the old code for 15 years having to dig into new code

Do you even understand how backwards this sounds?
Using the outdatedness of code as an excuse not to refactor it is usually reserved for big corporations, not modern open-source projects :-)

- slower

Or maybe faster? Did you benchmark it?
Of did you see a specific location of code which will cause slowdown?

- code spread over multiple locations

Which is called modularization aka OO, and is IMO the main point of the branch

- lost svn blame history

use git

- last but not least, you coding style does not follow the compiler code

style
Which may be trivially fixed (btw, compiler code style is really ... um... exotic).

> Well, maybe I should try myself > but the experience of others is definitely discouraging. Who is others? I remember very few *compiler* patches we rejected.

Sorry, but I do not believe you have forgotten all the recent flame about the rejected changes ;-)

Oh, do not remind me of that. Do you know that said student had quit working on FPC because he found the code base to be a <insert polite adjective here> mess?
(And which is sadder to me, he also took a different adviser for his next year course work).
I actually teach compiler construction at the university, and he correctly noted that compilers produced by some students at the end of the course are better structured than FPC.

Ok, this is a point. I'am pretty sure that a patch which fixes casing of exported symbols will be accepted and I'am the first one who will apply it. However, this is no argument to do reformatting. Casing yes, but reformatting?

Ask Graeme. But I felt reviewing with tortoisesvn pretty easy while tortoisegit even doesn't allow me to diff pulled changes quickly in the pull window.

Politeness is not important as long as the arguments are reasonable ;)

Who says the code is outdated? I said only that people doing the work on FPC have to dig into new code for no gain. And at least I'am not willing to fix such new code if I see no gain.

> - slower Or maybe faster? Did you benchmark it? Of did you see a specific location of code which will cause slowdown?

Dislocation of code causes less efficient cache usage. Added calls, added class instance accesses.

> - code spread over multiple locations Which is called modularization aka OO, and is IMO the main point of the branch

Only if it serves a purpose. There is no use on modularization or OO for itself. In German you call something like this over-engineered.

> - lost svn blame history use git

I cannot see how git solves this for code being moved around and being reindented, not to mention upper/lower casing changes.

This is out of discussion yet, the compiler uses its coding style and it should be consistent as far as possible.

Oh really? Where can I download these great compilers? I hope they are ready for production use. BTW: We spend a lot of time to fix his mess of code :) No problem he was student, but I consider him far from being able to judge about code quality considering his work :)

http://imcs.dvgu.ru/works/home?name=%D0%9A%D0%BE%D0%BC%D0%BF;wt=1567;do_select=1 Sorry, russian interface only.

I hope they are ready for production use.

Of course not, they are written in 4 months by the 3rd year students spending 3-4 hours per week.

BTW: We spend a lot of time to fix his mess of code :) No problem he was student, but I consider him far from being able to judge about code quality considering his work :)

I know, he is certainly not the best student, but not a bad one too. Still, two points:
1) While he may be a bad judge, but he is certainly a good case study of an average programmer trying to make sense of FPC codebase.
2) I'd say that at least half of his mistakes could be avoided by better code structure of FPC -- he was really uncomfortable using strange records and pchars instead of normal objects and strings.

That has nothing to do with code structure, but with efficiency.

pchar and records are much faster than objects and strings. This has been established empirically a long time ago by the compiler developers. Speed of the compiler is an important factor, so records and pchars are used.

So with your last argument you actually strengthen Florians case.

Michael.

While it may be true that records/pchars increase efficiency, they also certainly _are_ an essential part of the code structure. And the price paid for whatever efficiency gains they bring is reduced maintainability of code.

pchar and records are much faster than objects and strings. This has been established empirically a long time ago by the compiler developers. Speed of the compiler is an important factor, so records and pchars are used.

1) I have serious suspicions that compile time on modern processors is dominated by linking and I/O.
At least this is certainly true for FPC on Windows case.
2) I may believe that records are faster then classes, but if they are faster then objects, then this is a compiler problem, since there should be literally no difference except the VMT pointer memory overhead. Actual usage of virtual calls may cause a slowdown, but it also may cause a speedup if it replaces a 'case' statement, which is usually so.
3) As for strings, I can see that the overhead may be substantial, but that does not mean one have to roll back all the way down to raw pointers. Atomic strings with memory pools, for example, might be a good alternative.

So with your last argument you actually strengthen Florians case.

I am not so sure, it looks a lot like a premature optimization to me.

Alexander Klenin wrote

2) I'd say that at least half of his mistakes could be avoided by better code structure of FPC -- he was really uncomfortable using strange records and pchars instead of normal objects and strings.

Hmm, strange records and pchars are often necessary for good performance. A professional programmer or a hobby programmer with the needed skills to do compiler development on the necessary level has no problems to read and work with such code. I did not much reading in the FPC sources up to now but sometimes I saw possibilities to improve performace by using a little bit more pchars and strange records. ;-)

Martin

This is hilarious. I could just as well say -- he was really uncomfortable using strange objects and strings instead of normal records and pchars.

Regards,

Adriaan van Os

Alexander Klenin wrote

1) I have serious suspicions that compile time on modern processors is dominated by linking and I/O. At least this is certainly true for FPC on Windows case.

Do you remember the factor 10 compiling speed advangage of Delphi7 compared with FPC 2.4?

Martin

I do. I also remember that one of the goals of DoDi's previous set of refactorings
was to address that discrepancy.
And since they were rejected, FPC will continue to be slower than Delphi. This is a typical showcase for the premature micro-optimization -- for a few percentages of speed in the short term,
you pay in reduced code maintainability,
which precludes high-level optimizations in the long term.

But not on Linux. Even so, it doesn't matter. If linking is slow, it doesn't mean the rest should also be slow. What can be optimized, must be optimized.

The VMT is not the point.

Objects are zeroed out. Records are not. There have been various pleas for a faster fillchar() on this mailing list, and they are not without reason.

3) As for strings, I can see that the overhead may be substantial, but that does not mean one have to roll back all the way down to raw pointers. Atomic strings with memory pools, for example, might be a good alternative.

There was no 'roll back'. Originally, there were no ansistrings, only shortstrings, so the compiler was written with pchars.

Ansistrings have serious overhead associated with them, so when they appeared, they were not used.

If you need more convincing: the contnrs unit contains 2 hash classes. one based on ansistrings, one with pchars and shortstrings. The pchar version is easily 20% faster. It comes from the compiler code and was written specially by Peter Vreman.

Try and improve that class with ansistrings. If you succeed, only then we can start making a case for using them in the compiler.

> So with your last argument you actually strengthen Florians case. I am not so sure, it looks a lot like a premature optimization to me.

Not 'premature'. It was done after heavy profiling of the compiler, and the speed improvements were very noticeable.

It sounds like you give us little credit. Most things happen for a reason;
You just don't know the reason.

Michael.

IIRC one of the goals are to enable muti threaded compiling. I have a horror from that scenario. It is difficult enough to produce a stable single threaded compiler. I can't believe that it is possible to make a good multi core aware compiler without a team of highest skilled fulltime developers working several years...

Martin

Op 2010-10-19 17:06, Martin Schreiber het geskryf:

core aware compiler without a team of highest skilled fulltime developers working several years...

Why do you think we are not that already? :-)

[...I have learned from past experience that assuming something is very bad, especially if your only medium of communication has been emails...]

Regards,
- Graeme -

Really? I didn't see any benchmarks?

And since they were rejected, FPC will continue to be slower than Delphi.

Cum hoc ergo propter hoc.

This is a typical showcase for the premature micro-optimization -- for a few percentages of speed in the short term, you pay in reduced code maintainability, which precludes high-level optimizations in the long term.

A very small group of unpaid people maintaining a compiler over several architectures and operating systems? Seems like the fpc team is doing quite well.

Henry

Michael Van Canneyt wrote

What can be optimized, must be optimized.

Sorry, but does not this sound slightly exaggerated? :-)

You are wrong. The code below outputs random garbage, not zeroes.

type T1 = object x, y: Integer; end;
procedure P;
var obj: T1;
begin Writeln(obj.x, obj.y); end;
begin P; end.

It sounds like you give us little credit.

Sorry for that and thanks you for your work.
But, by the same token, don't you think that you also give a little credit to Hans' work? Maybe his reasons are also good?

Most things happen for a reason; You just don't know the reason.

I fully understand that the reason for degrading the code structure was efficiency.
I just doubt that it was/is a good trade-off.

Aha! Maybe that's the reason for the lower performance than Delphi . ;-)

Martin

Sorry, I was talking about classes. I don't use objects from TP;
I consider them deprecated.

Besides my personal opinion, the compiler already uses classes for some other things, and it's not very desirable to mix the two concepts.

> It sounds like you give us little credit. Sorry for that and thanks you for your work. But, by the same token, don't you think that you also give a little credit to Hans' work? Maybe his reasons are also good?

As I explained in another mail, I think he tries too much too soon.

First do some small and desired things, to build up credibility. Check the bugtracker for things to do. You'll be invited on the core team by itself if you do a good job. Then take on the big tasks.

This is how everyone else made it on the FPC team.

Not by wanting do reorganize everything is just not realistic, simply from a human point of view.

If I would attempt to do what he did, I would also get into serious trouble, even though I'm one of the oldest members on the team.

Michael.

Ok, I went ahead and have taken look at the code.
I assume you speak about TFPHashList vs TFPCustomHashTable. The classes are not really comparable, because they use quite different internal data structures.
So instead I converted TFPHashList list to use ansistrings.

Benchmarking included 3*10^6 calls to Add and Find methods with the arguments of various lengths.

Average string length 5 characters:
ShortString: 1.15 s
AnsiString: 1.56 s

Average string length 45 characters:
ShortString: 12.0 s
AnsiString: 3.2 s

I agree that the first case is more relevant for the compiler, but still you can see that ShortStrings are clearly not always faster.

Just curious: Would you, by any chance, have a figure for this:

In a (self) compile operation, how many times are either of these these routines called?

Alexander Klenin пишет:

I believe it's not ShortStrings per se to blame, but storing them in a large single memory block as it is being done in TFPHashList. Adding a lot of keys will cause many reallocations. Large size of the block forces memory manager to use variable-size pool, which is less efficient than using fixed-size blocks in case of small chunks.

Regards,
Sergei

Very possible. I have even managed to get a few OutOfMemory errors while testing ShortStrings version.

Thanks Helmut for your encouraging message. I enjoy lurking and learning by reading these extensive threads.

Back in March 2010 there was a similar thread about Lazarus development and in the end several people posted their success stories which are now here:
http://www.lazarussupport.com/lazarus/Testimonials

I'm glad you came along to help us look at the "big picture" with your FPC success story! :)

But they perform the same function; hash on string with an associated data structure.

So instead I converted TFPHashList list to use ansistrings.

That was the point :-)

I'm always open for benchmark tests.

Can you please send me your test and the implementation ? I'd like to see when the balance shifts.

Recently I had an implementation for TStringList.Sort which the author claimed was 25-45 times (!) faster than the quicksort. I've never been able to reproduce his claim - rather the opposite. Not even with Delphi (which, strangely enough, performed worse than FPC)

Michael.

Strange.
The whole idea of the implementation was to improve memory efficiency. And at the time, it was definitely and consistently faster. If Alexander sends me his code, I will check, and then I should ask Peter Vreman, who wrote the code. (if he'll still answer).

Michael.

Michael Van Canneyt пишет:

It is indeed very efficient regarding number of bytes occupied. Storing all strings in one block has no storage this implementation with huge amount of Add's won't show the best speed, but it has little common with the intended usage scenario, which is "single Add, multiple Find's". Also in the compiler, the average number of items in a single TFPHashList is not huge. It can grow huge in units like Windows (many procedures in one namespace), but these are more like extreme cases.

Regards,
Sergei

In our previous episode, Sergei Gorelkin said:

I actually thought that too, and did a benchmark (cclasses')hashlist against an own implementation based on array of array (to reduce that by a magnitude) a few years back

Mine was slower.

Well, DoDi is not willing to fix the flaws of his patches so they were rejected, see e.g.: http://bugs.freepascal.org/view.php?id=17584 And now I shall apply patches which turn the compiler upside down and then I shall trust that he fixes the bugs the code potentially introduces?

Alexander Klenin wrote

I fully understand that the reason for degrading the code structure was efficiency. I just doubt that it was/is a good trade-off.

Dear FPC Team,

After finishing porting the bullet physics library from C++ to FPC in the last 2 months (http://bulletphysics.org/wordpress/). I had the chance to compare Xcode, MS Visual C++ and MinGW GCC regarding compilation speed.
FPC is about 3-20 times(!) faster then those (modern) compilers on pure compilation speed. That is a fact at least for me and my employees. I had to debug the library on multiple archs with multipe tools side by side,
step by step to find my porting errors - and I had to add lots of debug code in the original code, so I was able to "feel" and measure the turn around times.

I am happy that you all have a deep understanding of writing fast and memory efficient code, because I can use the compiler as spell checker.

I am happy that I can use FPC to do 10 GBit Line speed network analysis and real time protocol identification under FreeBSD, with a small team of developers because they can read and debug pascal code fast and have superior
turn around times.

I am happy that I can also use "degraded" datastructures in conjunctions with
lock free algorithms and that I have the choice to use them if I understand them.

Thank you all for the exceptional stability of the FPC trunk on Windows, FreeBSD (32)
and OSX because we use the latest trunk version to do our development (generics).

Thank you that my company was/is able to write a cross platform OpenGL Engine
for data visualization which now works with OpenGL 4, OpenCL and optimized Shadercode, completly driven by FPC. (Shadercode and OpenCL code is in C) Proving that FPC is able to feature recent technology projects.

Thank you for your long time support of language features - using objects (TP objects)
as generics has proven to be sucessfully fast for vector, matrix,quaternion and memory
management operations while maintaining a FPU needed layout easily.

Thank you all for your time reading, thinking, answering often stupid/lazy mails and
bug reports from me.

Please continue your good judgement about trunk codebase improvements.

helmut

C may be difficult but there could be more pascal-like languages like Modula and ADA. My idea is that they can use the same libraries and can work together even inside one project. I don't know how far the compiler modes in FPC can take you for supporting other languages.

It would be a similar concept that .NET and Java VM's are for typed, garbage collected languages and Parrot is for dynamic languages. Every language under those VMs can call the same libraries. The FPC supported languages would do the same (use common libs and variables defined in those libs). IIRC there is no such thing now for compiled languages. GCC supports many languages but they can't interact transparently. Some clumsy glue code is needed for that.

C language support for FPC frontend would be possible but compiled (binary) C libraries could not be used and existing C source not used directly. Difference in default calling convention is solvable I guess. Anyway I don't see this as a very important feature neither.

Multi-language support was mentioned earlier in DoDi's mails as part of the plan, after refactoring. That's why I wrote about it. The fact still is that his plan is fundamentally different from your plan.

I don't really have an opinion about if DoDi's patches are good or bad. Maybe you did the right thing by not letting such big changes into FPC trunk. However, I am curious to see what he can achieve. He clearly likes to experiment and refactor a lot, and a fork is the only way to do that.

The options are about like:

1. He will send some more mails and big refactoring patches which are again rejected, probably for good reason. After lots of frustration he gives up and the energy is vasted. Nobody is happy.

2. He will freely modify and refactor the code in a separate branch or fork. When there is a stable compiler available, features are compared and a healthy competition is born (hopefully). Everybody is happy.

I am just saying that from those options I prefer to see number 2.

Regards,
Juha

> and refactorings are not welcome. Who says this? Usefull refactoring *is* welcome.

Umm, please specify what you understand by "refactoring".

Something that does not move code around, or into new subroutines? Something that can be achieved in a sequence of only small local changes?

DoDi

According to the preceding discussion.

I have a horror from that scenario. It is difficult enough to produce a stable single threaded compiler.

Well, it's a matter of models, patterns and structure. Something not found in the current compiler sources :-(

I can't believe that it is possible to make a good multi core aware compiler without a team of highest skilled fulltime developers working several years...

I could have done it alone, in about half a year. Once the evitable global variables have been removed, most of the job is done.

DoDi

Well, DoDi is not willing to fix the flaws of his patches so they were rejected, see e.g.: http://bugs.freepascal.org/view.php?id=17584

You're kidding. The same result in a branch, with intermediate steps, would be rejected as well (NoGlobal branch). Why break one patch into smaller patches, that do not compile until all have been applied?

And now I shall apply patches which turn the compiler upside down and then I shall trust that he fixes the bugs the code potentially introduces?

Show the bugs and give me a chance...

DoDi

Sorry, I was talking about classes. I don't use objects from TP; I consider them deprecated.

I use Object in the semantic stuff, just to avoid try-finally blocks. As long as Class objects must reside on the heap, Object still has advantages.

I only tried to contribute the best of my education and knowledge, resulting from decades of code analysis and redesign. I admit that my coding style, that evolved in that time, is partially incompatible with certain "standards" - but only for good reasons.

First do some small and desired things, to build up credibility. Check the bugtracker for things to do. You'll be invited on the core team by itself if you do a good job. Then take on the big tasks.

That's a very dangerous proposition. I've encountered several projects where the maintainers could not fix certain bugs, and left these to others. When such a fix required restructuring of the code, it typically is rejected, because the maintainers don't understand it. A plenty of such bugs will typically disappear as a sideeffect of a general refactoring.

Please note that I do *not* say that FPC already is in such a state...

DoDi

E.g. the elimination of the translation of C headers or entire libraries (bzip...), quick evaluation of sample code (usually in and for C libraries or APIs)...

DoDi

Actually, I certainly think (and hope) that DoDi's goal is _not_ a "C frontend", but to find _any_ possible venue to upgrade the code structure of FPC code.

Well - both ;-)

I would like to see my C parser working as an FPC front-end, instead of producing intermediate Pascal code (in the SF ToPas project). Such a front-end would allow to compare both code compilation and execution speed of FPC with existing C compilers, and I wouldn't be surprised if FPC shows up quite good in such a comparison. But that's only a hobby task, a proof of concept, and a chance to learn (and document) more about the compiler internals.

I'll start an according discussion in fpc-other, as soon as I get access to that list. I hope that such a fork can be kept in sync with the SVN trunk, at least for the compiler back-ends, somehow - Graeme?

BTW, the "Overview of all lists.freepascal.org mailing lists" link seems to be broken, in all web pages?

DoDi

But as long as FPC is in the hands of the current team (and I hope that this will last for a long time ^^) we have to obey their coding style if we want our code in the compiler. I'm not a fan of the style either, but I adhere to it.
Look at it as a sign of good will: you show that you really mean it by acknowledging the standards and then you might be treated better. :)

Regards,
Sven

Op 2010-10-19 18:22, Michael Van Canneyt het geskryf:

Sorry, I was talking about classes. I don't use objects from TP; I consider them deprecated.

I still use objects instead of classes in certain parts of my code. They definitely have their uses. They're like records with methods (which by the way is a recent addition to Delphi - though objects does that already, so not sure why they extended the record structure for that).

Regards,
- Graeme -

Op 2010-10-19 18:38, Alexander Klenin het geskryf:

I did the same thing last night and came to the same conclusion. Those two classes cannot be compared as-is.

Regards,
- Graeme -

Op 2010-10-20 02:04, Hans-Peter Diettrich het geskryf:

to that list. I hope that such a fork can be kept in sync with the SVN trunk, at least for the compiler back-ends, somehow - Graeme?

Yes, an easy task. Add a new remote repository (current github mirror of FPC's SubVersion repository). Then cherry-pick/merge whatever you want from there into your repository. A straight forward task.

Regards,
- Graeme -

Because .NET has it, and the current team at Borland probably never knew about TP-style objects in the first place.

Other than that, I did mention that it is my personal opinion;
I definitely don't want to stop anyone from using them :-)

Michael.

I'm sure it has.

Just as Florian has good reasons for his coding style. Since he is senior to you (in the FPC community), his style goes.

His style is also not my style, I have totally different style, but on the rare occasions that I do something in the compiler, I try to use his style, simply because he does the bulk of the work in the compiler.

It's not about bug fixing "an sich". It's about human attitude: Trust must be earned.

You can say that you have years of education, knowledge and whatnot, but we don't know that. You must prove this first. Just like all the others had to do, just like I had to do.

Like I said in another mail:
One doesn't let a stranger in and let him redecorate ones house to his taste.

You and the FPC team got off on a wrong footing. I regret this - god knows that we need manpower - but it has happened.

Michael.

This in fact is not a native Lazarus issue. The Lazarus team can't do a "decent" Unicode implementation, unless FPC and the RTL provide the appropriate infrastructure. As Delphi decided to go with dynamically coded strings for Unicode (what we should not discuss here) I feel that there is no other way to go for Lazarus.

They could have decided not to provide Unicode at all, but they - supposedly for good reasons - decided to do the interim step we see now, which necessarily introduces a lot of quirks for those who had been happy with the non-Unicode versions.

-Michael

Op 2010-10-20 09:21, Michael Van Canneyt het geskryf:

Because .NET has it, and the current team at Borland probably never knew about TP-style objects in the first place.

Idiots! They are a bunch of Microsoft drones.

Other than that, I did mention that it is my personal opinion; I definitely don't want to stop anyone from using them :-)

I was concerned about your "deprecated" statement - as in you maybe speaking for the FPC team. I wouldn't want object types to be removed from FPC or FPC's objfpc compiler mode.

Regards,
- Graeme -

Hans-Peter Diettrich wrote

I would like to see my C parser working as an FPC front-end,

I feel that there in fact is a decent cause for doing a C(++) front-end for the FPC compiler: importing existing C++ code into a Pascal project. I suppose this is not possible completely without modifying the C++ code this will not be possible in many cases (please don't start discussing multiple inheritance ;-) ), but this would be a lot better than the current situation where linking FPC and C++ is completely impossible due to the different ABI.

-Michael

In our previous episode, Michael Schnell said:

As said thousands of times, it still will be different.

See e.g. Delphi and BCB, where C++ classes needed to be explicitely declared (pureclass) to be usable from Delphi.

80% of the world is :(

Have no fears, that won't happen.
e.g. the FP IDE (textmode) is written using objects.

Michael.

Marco van de Voort wrote

As said thousands of times, it still will be different.

Of course I do know this. (I should have said "a kind of c++" to be more accurate)

It would need to a kind of different/incompatible/propriety c++ language that could only live in #ifdefs and with help of precompiler macros, if you want to be able to compile the source file with a c++ compiler. But this _could_ be used to create files that can be used as well in c++ projects as in Pascal projects and thus _could_ provide a more efficient way to use (some ) existing c++ than completely re-coding it in Pascal.

-Michael

In our previous episode, Michael Schnell said:

Essentially you haven't answered my concern, just spread more vaguery around, as usual.

Again: Combining two languages into one compiler doesn't magically make them interoperable. And the only existing example doesn't exactly have a crack record on using C++ FROM Delphi (the otherway around is different, I'm told, not elegant, but doable).

Basically it would mean implementing the innerards of C++ into the whole existing (pascal) compiler too, so that the pascal compiler can access C++ objects, not just a two bit extra parser. (which is also not so simple as it sounds, since probably backends mod to execute C++ in a compliant way are also possible).

Please stop echoing sentiment without any technical underpinnings.

Marco van de Voort wrote

Essentially you haven't answered my concern, just spread more vaguery around,

Agreed. (Any innovation starts on a very vague level.)

I did not want to start or suggest any development, just wanted to state that DiDo's idea is not completely senseless.

-Michael

In our previous episode, Michael Schnell said:

That's my point. What DiDo does (add parser) and what you expect (interoperations) are two different subjects.

No, it would be the other way around. The "kind of C++" or "pseudo C++" or whatever would map to the existing compiler's data and object structure. I think Michael Schnell's idea is about the same as mine (see the other mail).

However, C++ is a poor example because its syntax is a deep swamp. Ok, I don't start with multiply inheritance or templates... The frontend would need to ban lots of syntax, practically creating a new C++ related language ... No thanks for me ...
I see C++ as an example of how NOT to do things.

Other language frontends would be realistic IMO.

Juha

In our previous episode, Juha Manninen (gmail) said:

No, it would be the other way around. The "kind of C++" or "pseudo C++" or whatever would map to the existing compiler's data and object structure.

Then the avg C++ code would not run, since it uses boost and friends that heavily use every detail of the C++ object model.

However, C++ is a poor example because its syntax is a deep swamp. Ok, I don't start with multiply inheritance or templates...

Yes, but you can't discount the difficult ones, because they are the only ones that could make it useful (even though it is not as easy by far as the painting that you picture). One wouldn't want to butcher up the compiler just so you can compile some forgotten Basic dialect just because it was "clean" enough.

The frontend would need to ban lots of syntax, practically creating a new C++ related language ... No thanks for me ... I see C++ as an example of how NOT to do things.

That is exactly the kind of circle reasoning that haunts these discussions. The actual target is a constant flux, and adapts to suit the reasoning:

MS> We need a C++ frontend
MV> Why ?
MS> porting to Pascal is such a horror, and then we could C++ code as is. MV> implementing a C++ does not mean interoperability JH> Then change C++ so that is interoperable.
MV> But then you are can't compile foreign C++ code anymore!

Juha Manninen wrote

However, C++ is a poor example because its syntax is a deep swamp. The frontend would need to ban lots of syntax, practically creating a new C++

Yep. That is why it would be essential to use the original GNU preprocessor to keep the "pseudo c++" frontent for FPC as small as possible. Macros can do some magic tricks and might help to translate real c++ to "pseudo c++" in some place. Of course there is lots of stuff that can't be ported (such as multiple inheritance).

-Michael

In our previous episode, Michael Schnell said:

macros do nothing more than substitution, which is not a problem since you can control the frontend anyway. The problem is the connection of the frontend with backend and with Pascal.

I already assumed the frontend would be as perfect as could be. If the frontend is assumed perfect, there is nothing you can "fix" with macros.

IMHO you are both just trying to weasel out of the discussion, by keeping both goals and implementation principles deliberately vague.

Keep in mind that I actually am a proponent of multiple frontends (I've wanted a M2 version for over 10 years now, but unfortunately I'm quite realistic), I just don't like Hans-Peter's approach (which is essentially not cooperating, and just throw raw patches to core, and then trying to blackmail core into accepting them by raising noise on the maillists)

The first principle of working in a team is assuming that all code is good, unless proven otherwise.

To be honest, I was responsible for at least half the noise in this thread. Also, what do you mean by "raw" patches?
Do you want a "cooking" model similar to the one used in git.git development? That would be good, but practically impossible to do using svn.

The first principle of working in a team is assuming that all code is good, unless proven otherwise.

Don't you violate this principle by assuming Hans-Peter's code is bad?

Yes, I admit C++ is a bad example. I would also say there is no practical way to integrate it with FPC. It was not listed in DoDi's plans either. Somehow the topic shifted here.

Still, C would be doable, for porting SOME existing code to co-operarate directly with pascal code. If the C code uses lots of library calls it can't be used directly. But, there is code for math and compression etc. which don't call libraries much.

Languages like ADA, Modula (and Fortran ? and Basic ? ) would be realistic.

Juha

In our previous episode, Juha Manninen (gmail) said:

Still, C would be doable, for porting SOME existing code to co-operarate directly with pascal code. If the C code uses lots of library calls it can't be used directly. But, there is code for math and compression etc. which don't call libraries much.

The implementation is much easier I guess yes. But the interoperability remains the same. You still won't be able to access C headers directly from Pascal.

Moreover I think the goals of such project can be much easier accomplished by having a C compiler that outputs (FPC) pascal source.

Languages like ADA, Modula (and Fortran ? and Basic ? ) would be realistic.

Modula2 is a special case since very related to Pascal. It needs some work in the module handling (but that work can be recycled for other pascal dialects that were inspired by that, like ISO Pascal too), but the main problem is the lexer/parser, because M2 is case sensitive and has a different block structure (which goes beyond merely being a pascal dialect).

The only other problem I can think of is that is primarily has unsigned arithmetic, which might require enhancements/debugging on the semantic parts.

I think the same goes for Ada as for C++. The language is so huge, that it is probably better to fork, throw away the pascal part, and then later when the ada support is ok, backport the pascal part back into the Ada compiler. (that will probably be somewhere near 2022)

I don't know fortran enough to comment, and Basic is so balkanized dialect mess that nothing can be said about that without specification of what dialect you mean. I can only image VB here, but VB is very complex internally, and the existing codebases are horribly windows specific). Moreover when it finally would be production ready, VB6 would probably be near extinction.

In our previous episode, Alexander Klenin said:

The diffs of his working repo, with unnecessarily refactoring and formatting removed so that it makes the job on the reviewer easy.

I didn't assume, I 's hard not to, since his commit messages roll over the IRC channel.

I never expected Hans-Peter to really come up with something in the first try, but I more or less expected it would be a scheme like:

1. A new developer first wilde ambitions plan fails because ideas are too unfocussed, on a too big scale, too idealistic and requiring too much up front commitment.

2. The first effort fails, but parts of it that do work are cleaned up and integrated. The developer keeps working and from time to time, finds some subsystem he is interested in, starts maintaining it and gaining experience.

3. After an year, or two, he offers a patch with a certain rewrite or a feature branch that covers the more realistic part of his original ideas, and it gets incorporated, and he stays maitnaining it.

The problem with Hans-Peter is that 1 happened more or less, but the second part didn't because he didn't want to compromise and stick to the FPC core teams rules. Moreover the plans were awfully unfocussed, and tried to delegate quite some of the work and responsibility to core beforehand. (by requiring every step to be merged in before he had anything working, based on just principles. Worse he didn't even make it easy for Core to merge it)

IMHO, he should have toned down a bit, and continued working in a branch to achieve something tangible with as minimal possible intrusion.

That's how you get things done. Not by letting your self get dragged into all kinds of refactorings because "otherwise I can't carry out my real plans". In short : "Don't blame the tools, work with what you have to do the job".

One can hardly expect core to radically change the philosophy of the project for every would-be developer that thinks he has an idea.

People keep blaming core in these threads because they are supposed to be against HP's changes. While that may be true, such meanings can change if you show something tangible, preferably if it can be incorporated easily. And that has happened before. Moreover, opinions differe with time.

True. DoDi must have some idea to solve it with the frontend. Automatic header conversion as part of the frontend maybe?

So that requires changes also to compiler internals and not only to frontend? Ok then.

I have to agree. A frontend can't solve those.
Fortran would be the most realistic I guess but only few people would need it.

Juha

In our previous episode, Juha Manninen (gmail) said:

> Moreover I think the goals of such project can be much easier accomplished > by having a C compiler that outputs (FPC) pascal source. True. DoDi must have some idea to solve it with the frontend.

I only heard him about mapping the C frontend to FPC's backend. I had some concerns about that, but HP answered that plausible enough (on an high level). This is more or less because that operates mostly on C AFTER preprocessing. Afaik the interoperability question is still open (or I didn't buy the answer)

Automatic header conversion as part of the frontend maybe?

If this was feasible, it would be here. Unless somebody really _shows_ me otherwise, I think that what is here, is as much as we are going to get.

I personally think that fully automated C header conversion is not possible.

Yes of course. But that part (unit/module handling) is already destined to be written sooner or later. It is one of the parts that wasn't rewritten after the 1.0.x->2.0.x transition.

The lexer/parser stuff is more about why you probably can't treat Modula2 as yet another Pascal dialect and keep it as a configurable option in the current parser.

And keep in mind that modula2 is about as little intrusive as they come. It basically is Pascal, with slightly different syntax and unsigned arithmetic, and e.g. the whole unit system is a backport from Modula2 to pascal.

That is the main problem. The main people that want fortran are people shopping for easy to use codebases. They are not interested in Fortran themselves.

The actualy Fortran crowd is scientifically oriented and extremely conservative, and unlikely to use a fortran compiler that is ducktaped on to a pascal compiler.

The question is not just what is possible, but also who is going to do it (both separately for the initial effort and maintenance/expansion)

The problem is that the only sane target (C/C++) are so different that interoperability is a bitch, and if you "just" want a C compiler, grab gcc. (interoperability would be the reason to add it to FPC, I don't really expect for C users to drop gcc in droves and flock to FPC).

People constantly try to pooh-pooh these problems with workarounds ( macros, subsets and auto conversion), but these do not fix the problem. Moreover some of these solutions (like auto-conversion) are Loch Ness monsters, in the sense that quite often people say that they saw it, but somehow never in circumstances that can be replicated :-)

If the workarounds start changing the input, and it is not essentially:

- download C code. (e.g. zlib)
- compile C part with FPC/C
- compile own program with FPC/Pascal that uses C compiled part.

then one should wonder what it is actually good for.

Juha Manninen wrote

Still, C would be doable, for porting SOME existing code to co-operarate directly with pascal code.

AFAIK, flat C sources can just can be compiled with GNU and linked with FPC projects.

-Michael

I believe so too.

Still, you may want to have a look at the design goals and library-based architecture of the CLang <http://clang.llvm.org/features.html> Language Family Front-end.

Regards,

Adriaan van Os

and let's not forget the other argument "Delphi does it too" :-)))))))))

just kidding, different compiler is invoked

as for the linking of C++/FPC:
good argument, but then your fpc-compiled c++ will be incompatible with everything else. Come to think of it, no 2 C++ compilers mangle the same due to VMT and other abi-stuff anyway.

kind regards,
Dimitri Smits

Michael Schnell wrote

but this would be a lot better than the current situation where linking FPC and C++ is completely impossible due to the different ABI.

Really? How does accessing Qt then work? Do you know more than me? Did I dream that Lazarus has a Qt widget set? Even more: who does the compilation of e.g.
http://svn.freepascal.org/svn/fpc/trunk/tests/test/cg/tcppcl1.pp in the daily testsuite runs?

The tests (which cover only part of the cppclass support in FPC) are one of the reality checks for your hypotheses about the usefulness of a crippled C++ front end. And they show: crippled C++ support is not needed, cppclasses is basically not used by anybody as far as I know.

In our previous episode, Florian Klaempfl said:

(euh, it doesn't link to C++, but to a C library that wraps the C++ QT?)

So there is no linking involved ;)? It is a poor man's solution like a crippled C++ front end.

In our previous episode, Florian Klaempfl said:

No linking of a pascal interface to a C++ interface.

It is a poor man's solution like a crippled C++ front end.

True.

You mean no direct access to C++ classes ;) The library libqtXpas (or how it is called) is compiled by a C++ compiler actually btw.

cppclass is not used, because it's not even remotly ready for real world stuff. Currently only static methods/classes are supported. For instantiated classes I must first implement a C++ runtime binding.

Regards,
Sven

Indeed that's exactly my point: it is currently as usefull as a crippled C++ front end for FPC because a crippled front end would probably fail on compiling any only slightly complex C++ library. But: getting the cppclass support etc. into shape is much easier than a full fledged C++ front end.

That's what I hope. :D

Regards,
Sven

Why not?

When currently the compiler searches for .pas and .pp units, in order to evaluate a Uses or Requires clause, a future version could search for .c and .h files as well.

Moreover I think the goals of such project can be much easier accomplished by having a C compiler that outputs (FPC) pascal source.

Why do you think that this way would be easier? I've already implemented such an compiler, in Delphi, so that it will not be a big deal to use its front-end. Remains to connect it to the FPC internal data structures and API.

The separated approch has the drawback that a recompilation of every update of the C sources is required, i.e. that compiler must become part of the build process (Makefiles...).

The readability of the generated files is questionable, and eventual manual post-processing (e.g. typing int parameters into sets or enums) is required after every recompilation. Workarounds have to be used for C elements, which do not map straight to Pascal elements. Such workarounds can be eliminated in a direct translation into nodes and other internal data structures.

I like just that different block structure, because it eliminates any fruitless discussion about the placement of the no more required Begins :-)

Case sensitivity can be subject to an mode switch, and the compiler can at least warn about conflicting declarations in case-insensitive mode.

DoDi

I personally think that fully automated C header conversion is not possible.

It is, but at the same cost of e.g. fully automated Delphi project conversion. In an FPC project the related items (modes, targets, search paths...) are configurable in several ways - the equivalent parametrization is required for every (other) compiler and its standard library and header files. In ToPas all this can be specified in a C-compatible way, using pragmas and defines, in a synthetic "project" module. One problem is the determination of exactly that compiler (including its library and header files), which must be used in order to produce a correct result from the inevitable compiler specific idioms and assumptions.

Some translations can require a deep analysis of the related code, in order to determine e.g. the translation of PCHAR parameters, into Pascal PCHAR or VAR char parameters. This is where most other translators apply wild guesses, which will work most of the time, but leave the user absolutely clueless when the generated translation doesn't work.

The problem is that the only sane target (C/C++) are so different that interoperability is a bitch, and if you "just" want a C compiler, grab gcc.

This will work for most GNU sources on Linux, but not for Windows (MSVC, BCB) related stuff. There exists no full-fledged gcc port for use on/for Windows targets, MinGW and even CygWin don't track all the Windows versions, features and quirks :-(

(interoperability would be the reason to add it to FPC, I don't really expect for C users to drop gcc in droves and flock to FPC).

+1

You'll know what it's good for, when bugfixes or other patches are applied to such a C library. In my own experiments with the compression libraries I found an bug in the C code, and when I contacted the maintainer, it turned out that exactly that bug had been fixed just a few weeks ago - after the code had been in use already for over a decade. This IMO is a strong argument against one-time (manual) translations from C into Pascal, that will have to be updated, again manually, for every update of the C sources.

Another problem exists in the beforementioned lack of portability of C code. While FPC/Lazarus attempt to achieve that portability by an intermediate layer (RTL, VCL...) and selectable targets (built into the compiler), C code has to be tweaked specifically for every target, that the developer decided to support. An FPC/C compiler could allow to apply the same support to C code, in target specific implementations of the C standard libraries, so that the compiled code will use the same basic functionality (memory management, file handling...), regardless of the source language. These libraries have to be adopted only once, for every supported target, like already done in the many supplied FPC packages.

As long as C libraries are already supplied as binaries (dll, so...), it may be sufficient to only translate their header files, or to make these usable otherwise. But else it will be necessary to translate the library sources into binaries, that either can be included into an FPC project, or can be used to create the platform specific shareable binaries, for use with further FPC applications or libraries.

DoDi

as for the linking of C++/FPC: good argument, but then your fpc-compiled c++ will be incompatible with everything else. Come to think of it, no 2 C++ compilers mangle the same due to VMT and other abi-stuff anyway.

+1

C++ is not only a different syntax, it also comes with its own standard libraries, object model, templates and more. Java and C# also come with their own memory management (garbage collection). Such features are incompatible with the FPC model, and IMO would require something like P++ or P# languages and compilers, before we can think about such language somebody wants to start his own FPC++ or FPC# project, he should know that there is near nothing that could be borrowed from the FPC implementation or libraries.

DoDi