Hi all,
I'm one of the developer of Hedgewars, an open source video game all written with FreePascal. Please see the full site at http://www.hedgewars.org/

One of the Hedgewars project is to make a WebGL/Javascript version of the game so that it can run in a browser. To do that we are using a tool named 'emscripten' which takes LLVM bytecode and generates Javascript, without affecting performance too much. Yes, we had to write a horribly hacked converter that took the small subset of Pascal used by Hedgewars and convert it to C (on a side node, the converter is written in Haskell) and reimplement the RTL.

So, this got the job done (if you are using an old browser the demo might still work http://hedgewars.org/hwjs/hwjs.html ) but all this process is *highly* unmaintainable and very prone to breakage. So the right way (tm) to achieve this result would be to directly integrate LLVM bytecode generation inside Freepascal.

I think we could make this work thanks to the Google Summer of Code! This program (*if* they announce it) basically introduces students to the world of FOSS development by having them work on projects for an open source organization during the summer.

I don't know if Freepascal wants to participate on its own, but if not, if there is one or more developer willing to act as mentor for the students, Hedgewars (*if* selected) would happily allocate some of its students to work on this task and act as a "umbrella" for this project targeting Freepascal sources. Of course the students would still have to write a sensible project proposal for being accepted.

What do you think of this idea?
Unfortunately I do not know Freepascal sources much and cannot fully evaluate the load of work, but I still think that we could collaborate on this level. Also do not underestimate the skills of GSoC students, the demo I posted above was compiled by one of them.

Cheers,
Vittorio (koda on #hedgewars)

Jonas already did some work on a LLVM code generator. You can find it in SVN: http://svn.freepascal.org/svn/fpc/branches/llvm/

So I'm sure we'll be happy to cooperate on this.

Michael.

I still don't understand what's the use of generating LLVM code in between instead of generating directly Javascript ... The process pascal->llvm->js looks as well very fragile to me.

While I agree that if the end target is Javascript it would be better to generate it directly, but generating LLVM may be useful in its own right.

(otherwise Jonas would not have started it to begin with ;))

Michael

Vittorio Giovara wrote

I think we could make this work thanks to the Google Summer of Code! This program (*if* they announce it) basically introduces students to the world of FOSS development by having them work on projects for an open source organization during the summer.

This is of some interest to me -- I am a university teacher, and I actively encourage
students to contribute to some FOSS project as a topic of their bachelor thesis.

In the past, my students participated in both FPC and Hedgewars (I believe the student used nickname "mbait").

I am not suitable to the role of mentor in GSoC sense, since I am not committer to
either of these projects, but I still can provide some help to my students, if they choose to participate.

Having a LLVM backend would not only benefit HedgeWar's JavaScript case, but also all others that would like to use the LLVM backend for one purpose or the other. And in my opinion a pure JS backend would be much harder to implement than a LLVM backend as one also needs to think about how certain things should be implemented in JS (which the emscripten developers already thought of).

Regards,
Sven

Well, I doubt that this will work smoothless but since I don't care about js or llvm support I won't argue about it :)

Indeed, a fpc->js code generator would have a rather limited use. A LLVM backend instead could be use on many more levels, and for example could improve (or replace) the compilation process on iOS. Plus I would like that this collaboration would be about something that could be useful for many projects, not something hedgewars-only.

Vittorio

Inviato da iPhone

Il giorno 10/feb/2013, alle ore 11:09, Sven Barth
ha scritto:

Vittorio Giovara wrote

Indeed, a fpc->js code generator would have a rather limited use. A LLVM backend instead could be use on many more levels, and for example could improve (or replace) the compilation process on iOS.

Improve in which regard? Experience showed that the more tools involved the less comfortable the use of FPC is. Within a few minutes you can build on any platform a win32 cross compiler (if the CPU is fast enough) because no external tools are involved. Compare this with cross compiling to linux (no to talk about darwin): takes usually 1-2 hours to setup: first, download, configure and build binutils, then sort out the libraries and copy them to host system, and then with some luck (if binutils are not broken), building fpc might work.

Vittorio Giovara wrote

Indeed, a fpc->js code generator would have a rather limited use. A LLVM backend instead could be use on many more levels, and for example

Not to mention that I estimate that full llvm support with debugging and extending llvm to support everything required by fpc might be more something like 1-2 years of full time work than a few summer months of work of a student.

Well the existence of external tools is what has allowed technology to foster and advance, eg have some common grounds and entry points... Your (limited) example is not exactly perfect, the tools that fpc bundle on windows are often outdated and conflict which the other tools (off the top of my head mingw or ld). And I successfully crosscompiled Darwin from Linux, it was a much less painful process than setting up a build environment under Windows.

But I digress.
The iOS experience could be improved in many ways, for example in Xcode you cannot set a breakpoint and when you do so with gdb all you get is an assembly viewer; having a LLVM backend could actually allow a better debugging and other tricks such as those provided by www.piecable.com

Not to mention that I estimate that full llvm support with debugging and extending llvm to support everything required by fpc might be more something like 1-2 years of full time work than a few summer months of work of a student.

I know that this is not going to be an easy project but please as I said don't underestimate students, GSoC people have carried out tasks that revolutionized more than one organization. Moreover I think that trying to offer help to achieve this long term goal is better than pestering on IRC with "is the LLVM-fpc branch ready yet??"

Cheers,
Vittorio

_______________________________________________ fpc-devel maillist - http://lists.freepascal.org/mailman/listinfo/fpc-devel

Vittorio Giovara wrote

The iOS experience could be improved in many ways, for example in Xcode you cannot set a breakpoint and when you do so with gdb all you get is an assembly viewer

I don't remember ever hearing about this or seeing a bug report about this. It's unlikely that this is related to FPC vs LLVM code or debug info generation though. And adding support for LLVM is something different than adding support for LLVM + LLVM debug information. The two are completely decoupled, and one will not automatically get you the other.

That said, regarding some other things that have been mentioned in this thread: a) the fpc llvm branch is completely out of date, predates the high level code generator that now should be used to add LLVM support, and at best can be used to get some ideas or code snippets from. I'm not sure how much is really usable though. b) I do think that adding an LLVM backend to FPC is useful. As I've mentioned before, it's however extremely unlikely that it will ever replace existing code generators (for any platform), it would simply become an additional option c) from the GSoC mentors list, I've understood that the best students are generally people who were already involved with the project (on the mailing lists, by submitting patches, ...). They are familiar with the project, the way contributions are handled, expectations of the existing developers regarding new code, and there is more chance that they are interested in working on the project in the long term (contributed code that is not maintained afterwards is much less valuable, because we already have a lot of code to maintain; if you have to maintain code, it's generally easier to do so if you wrote it yourself)

Jonas

Vittorio Giovara wrote

Well the existence of external tools is what has allowed technology to foster and advance,

In a perfect world, with unlimited cpu resources, yes.

eg have some common grounds and entry points... Your (limited) example is not exactly perfect, the tools that fpc bundle on windows are often outdated and conflict which the other tools

And you think this would change with llvm? The llvm is another tool chain which might cause conflicts and being outdated. No matter if windows or linux or whatever. Actually, linux might even be a bigger pain because fpc would depend on the llvm offered by the distro.

(off the top of my head mingw or ld).

This is exactly my point: cross compiling *to* windows is easy because FPC does not depend on any external tool when building windows programs. To any other platform is hard because of outdated *external* tools we depend on. It does not matter what's the host system: the pain starts as soon as external tools are involved.

And you think llvm provides magically source code debugging info? When just outputting llvm intermediate code, *all* source info is lost. Actually, an llvm port requires another complete debugging info output module to get debugging info.

In our previous episode, Vittorio Giovara said:

While I don't agree with your rant, I think it is futile to comment.

Note though that one of the reasons why FPC tools are old is because they are the most up to date mingw versions. Before they abandonned the tools we were using, and set up everything in MSYS.

Marco van de Voort wrote

Note though that one of the reasons why FPC tools are old is because they are the most up to date mingw versions. Before they abandonned the tools we were using, and set up everything in MSYS.

At least the mingw binutils are still maintained: http://sourceforge.net/projects/mingw/files/MinGW/Base/binutils/binutils-2.22. The same goes for "make" (FPC 2.6.x has problems with make 3.82 though, not specific to Windows/mingw) and abandoned tools do you mean?

Jonas

cp, rm, install, etc.

Regards,
Sven

In our previous episode, Jonas Maebe said:

Sven already replied, but roughly in the coreutils package. (the former findutils and diffutils iirc)

Sorry if this is a daft question, but can't you use the msys coreutils? Or do they have some limitation?

Last time I built fpc in windows, I didn't use the tools shipped with built it inside an msys terminal (with patched fpc Makefiles).

Henry

I wasn't fully aware of that, thanks for the explanation. Yeah as first project, a LLVM backend support is more than enough, isn't it? I'll try to reproduce the bug I mentioned with a more recent build of fpc and open a ticket if it's still present.

For (b) it fully makes sense. For (c) there is a long period between student selection and start of the project, where student can approach organizations and contribute to them; generally the students who are already capable of providing patches have higher chances of being accepted (although that's not always the case). If you think having a student is something useful we can highlight the "collaboration" on each organisation website in order to get maximum visibility, but it's the mentor who has to pick the right student in the end.
What about (d) is this project feasible? If not, is it can you split in (parallel) sub projects so that more people can tackle it?

Jonas

To everyone, it was not my intention to rant/start a flamewar on bundled tools, sorry if I touched a nerve there.
Please let's focus on building something together, foss-style.

Cheers,
Vittorio

In our previous episode, Henry Vermaak said:

IIRC they were less adapted to windows. I got a cygwin vibe there, the state within a state principle, with shared memory space and all.

A quick test seemed to not really be a substitution, but probably would need a whole process to support them and get it stable enough.

That also means that other cygwin problems (multiple versions conflict due to clashing shared version) could reappear too.

I never spent more than an evening on the test though, since I rather get rid of all the mingw parts instead (think fpmake here)

Last time I built fpc in windows, I didn't use the tools shipped with built it inside an msys terminal (with patched fpc Makefiles).

If you need to patch makefiles, doesn't that say enough ? :)

This would indeed be the best solution, thanks for the background.

I already had a full mingw/msys installation with up to date binutils and coreutils and didn't want to install fpc's bundled outdated tools. Patching the Makefiles was the only sane thing to do, since I spend most of my time in windows working with the msys terminal (mintty, actually).

Henry

Marco van de Voort wrote

I never spent more than an evening on the test though, since I rather get rid of all the mingw parts instead (think fpmake here)

This might be the best. Let's see that fpmake can handle all that and then get rid of the remaining tool dependencies.

Regards,
Sven

As I've said before: I think the compiler and RTL should remain Makefile-based (whether or not that is in addition to fpmake support for them, doesn't matter to me), to make porting to new platforms easier. "make cycle" is a very nice and easy test, and it would be quite annoying if fpmkunit and all of it dependencies would have to be compilable/working and installed before that could be performed. It would also make the bootstrapping process in general much more complex.

Jonas

Please also test with a clean project from http://dl.dropbox.com/u/28343282/ObjP/index.html to make sure it's not due to some project setting.

I really don't know whether it is feasible. There is definitely a lot of work that needs to be done to get a clean result, some of it fairly tedious (such as creating complete type definitions for types that are currently opaque in the compiler, like VMT layouts and RTTI). I'm now leaving on holidays for 3 weeks, maybe I'll have a better idea when I come back :)

Jonas

Then we would still have the problem of the outdated tools. Maybe we could write Pascal based substitutes for them that only need to handle the cases of compiler/rtl compilation...

Regards,
Sven

The only problem I'm aware of is cp copying the read-only attribute under Windows and refusing to overwrite such files, resulting in problems when installing examples from the packages dir from a non-exported svn checkout over a previous install. That problem does not affect either the compiler or the RTL.

Jonas

In our previous episode, Jonas Maebe said:

I spoke mainly for the release situation. Primarily I would want to get rid/minimize of own distributions of toolchains on non *nix. (3x windows, go32v2 and OS/2).

"make cycle" is a very nice and easy test, and it would be quite annoying if fpmkunit and all of it dependencies would have to be compilable/working and installed before that could be performed. It would also make the bootstrapping process in general much more complex.

Does anything really get more compilated then compiler/Makefile ? :-) (3800 lines, .fpc is 900 lines, but I bet there is quite some in the template that is for compiler+rtl dirs only too)

I agree that one has to be very conservative, and probably keep makefiles for rtl+compiler around a long, long time, and even longer as for *nix, but that is something else then saying "forever".

I don't think it is wise to say either keep makefile forever OR force fpmake long term for everything before the fpmake system is actually there, and ripened, and cross/bootstrap scenarios covered. Currently there is nothing to compare.

In our previous episode, Jonas Maebe said:

That, and its consequence should not be underestimated, it requires an export that can be terribly slow on a work machine with anti-virus. Luckily if you don't build real releases, you can avoid it by equating it to echo (as described in the buildfaq)

Some others:

1) The need to have manifests for install and patch, since anything with install or setup or patch triggers UAC (fixed, but can be confusing for end-users that copy these files over a network drive, binaries with "downloaded from network" bit get the UAC prompt again, which means a horrible GPF for a non GUI program.

2) the fact that nearly every toolchain (commercial or not) installs an own version of make, and typically adds its own dir to the $PATH. Not in the least our direct competitor, Embarcadero that has a totally incompatible make. This is also why the bootstrap for fpmake, if we ever come to that should not be make based.

3) Same in the case Mingw, cygwin also for other tools than make. Also 3rd party development systems as dev-* and freebasic that package these tools.

4) In some circumstances, even if cygwin is in the path AFTER fpc, suddenly parts may be executed under cygwin's shell which fails because cygwin's shell prefers its own tools. I think that this is because make (our makefile's?) behaviour changes when sh.exe is found. Never really got the details of this.

Is that still a problem now that packages are fpmake based? Also in current versions of the SVN client the .svn files are only stored in the root directory of the working copy... (yes, that would still mean a problem for older client versions, but I hope that it will get less :) )

Regards,
Sven

That's what I did for rm with rmwait indeed. However, people writing such replacements need to be careful to restrict themselves to plain RTL (and preferably the cross-platform parts only) rather than using packages simplifying the work on one hand but leading to many dependencies on the other.

Tomas

I'd say we have plenty of time to evaluate, we have until 18th of March before we have to submit papers.
Let's seen if we can work this out when you get back!

Cheers,
Vittorio

Hi all,
So some time has passed, do you think we could reach an agreement on the GSoC collaboration?
We have to submit our application by the end of this month but I would like to set up a tentative schedule beforehand.

Can we work on this together?
Vittorio

Sent from my iPad Mini

On 21/feb/2013, at 10:37, Vittorio Giovara
wrote:

I'd say we have plenty of time to evaluate, we have until 18th of March before we have to submit papers.
Let's seen if we can work this out when you get back!

Cheers,
Vittorio