Hi,

Any FPC developer willing to comment on the status of some of these issues (that have been years overdue)?

Graeme Geldenhuys wrote

It seems to me, main target of FPC development today is compatibility to the "modern" Delphi language constructs, I don't want to go this route.

And? Some people want to have these features (especially I want generic, I want helpers, I want closures and I want the extented RTTI, so I implement them!). If you don't need them, don't use them. We try to implement them in a way that the compiler's performance (parsing speed, etc.) or the executable size (to be implemented extended RTTI) is not influenced negatively too much. But somewhere one has to pay for it. Somewhen in the future we'll need to refactor the compiler a bit (maybe add multithreading support) and then things should be better again...

And we need more flexibility, I can't fight days or weeks with Michael and Marco for every little change which is not on Lazarus or their own benefit.

I don't know what flexibility you are talking about, but including everything in the RTL is not the answer. One needs to judge what can be made part of it and what not, because otherwise it will just be a chaos (yes, it already is a bit, mostly because of backwards compatibility).

Ouh, and there still is no unicode support for resource strings,

Currently not planned AFAIK. But maybe it's also not really necessary if the unit containing the resourcestrings is compiled in UTF8 format (I don't know though, this is not a topic I'm familiar with).

no official statement how unicode will be implemented in RTL,

There is no official statement, because we don't know ourselves really where we want to go and everytime the discussions comes up it dies down a bit later without much results...

the compiler becomes slower and slower,

I commented on this at the top already

smart linking MSEide on 64 bit Linux is not possible with 2GB ram,

Linking is a problem of ld, not of Free Pascal. ld was not designed for the workload we put on it (especially regarding smart linking). You could try however if the new internal ELF linker solves this problem for you (I don't know whether it already supports smart linking; this is something that Sergei should answer)

still no Delphi-like packages...

They are planned long term, but they are no cheesecake feature. We need to implement them in a way so that at least the major platforms (Windows, Linux, Mac OS X) can make use of it and already the difference regarding procedure name resolution between Windows and Unix can cause headaches...

Regards,
Sven

17.12.2012 13:55, Sven Barth пишет:

The problem here is that smartlinking is still being done with archives (.a) on x86 Linux targets, while ld's preferred mode of operation is section-based smartlinking. Switching to section-based smartlinking will probably reduce memory requirements to acceptable level. However, FPC does not handle exported symbols correctly and they get lost if section-based smartlinking is used.

The internal linker of course supports smartlinking, moreover it is not sensitive to the mentioned exported symbol issue.

Regards,
Sergei.

We should wait for and explore how the multi-target Delphi handles that.

DoDi

Hans-Peter Diettrich wrote

We should wait for and explore how the multi-target Delphi handles that.

Probably not even implemented, because Delphi IDE is Windows only - and there are no plans to make a cross-platform IDE by Embarcadero.

Regards,
- Graeme -

No. We need to think about this ourselves as we support more targets than Delphi ever will support.

Regards,
Sven

Nonetheless it would be wise to wait for an Embarcadero solution. They might invent a new cross-platform model for their packages, in which case we ended up in yet another package model.

DoDi

From what I see so far of the route that Embarcadero takes (complete reimplementation of compiler, sudden appearence of "type helpers" and things like "zero based strings") I predict that we'll reach a point in the near(!) future where we won't (be able to) follow Delphi's lead anymore. And thus I don't think that we'll need to wait for Embarcadero for this.

Regards,
Sven

Sven Barth wrote

I predict that we'll reach a point in the near(!) future where we won't (be able to) follow Delphi's lead anymore.

And what a good day that will be. FPC will can innovate again. Delphi hasn't been leading for years, and Embarcadero is milking a dead cow!!!

Regards,
- Graeme -

Embarcadero has a dead cow propped up in the barn, and is telling everybody that they still milk it.

Summed up eloquently by Ms Stob
http://www.theregister.co.uk/2012/12/11/verity_stob_sons_of_kahn/

For what my opinion is worth, I believe that we can learn from what happened when Clarion bought JPI/TopSpeed: they were interested in owning the technology rather than in ongoing sales of the compilers etc.

We have SPARC-based systems manufactured by Sun here, and I am dismayed at the extent to which /somebody/ has been leaning heavily on web sites who up till now have got away with harbouring copies of system manuals etc.

Right now I'd be extremely wary about pinning my livelihood to Java, lest Oracle succeed in locking it down as their personal property. I can't suggest any alternative, but I'd suggest watching IBM's position carefully since they're undoubtedly thinking the same thing.

Graeme Geldenhuys wrote

Probably not even implemented, because Delphi IDE is Windows only - and there are no plans to make a cross-platform IDE by Embarcadero. Regards, - Graeme -

IMHO Delphi-like IDE (design-time) packages are not useful in Lazarus, as adding a source package and recompiling does the trick just as well.

What does make sense are Delphi like pre-compiled dynamically linked (Runtime)-packages, allowing for creating "plugins".

-Michael

Sven Barth wrote

No. We need to think about this ourselves as we support more targets than Delphi ever will support.

Even cross-platform runtime packages might make sense (e.g. linking a native "plugin" package to an Android/Java Program)

Something I could (supposedly in vain) dream of: Remote plugin-packages attachable via TCP/IP.

-Michael

I suppose Unicode strings could be done better in a (slightly) incompatible way. (e.g. allowong for as well decently code-agnostic (never converted) RawByte, RawWord and RawDWord strings as for decently dynamically coded "Flexy-Strings" that are auto-converted when necessary), with a decent definition of what happens when Normal (predefined coding) String Types, "Raw" and "Flexi" strings are used in combination.

-Michael

Michael Schnell wrote

IMHO Delphi-like IDE (design-time) packages are not useful in Lazarus, as adding a source package and recompiling does the trick just as well.

Tell that to component developers and companies like Devx, TMS etc! With the current way things work, there is 0% chance of getting a trial version of any component.

Regards,
- Graeme -

It does not really matter whether the packages are used as design-time packages or as runtime packages in the end as the compiler must know to handle both, because the RTL package must ALWAYS be supplied as a statically added package (shared linking!) if runtime packages are used (otherwise you could not use runtime packages either).

Regards,
Sven

In our previous episode, Graeme Geldenhuys said:

They could deliver ppu's and .o's.

Marco van de Voort wrote

They could deliver ppu's and .o's.

True, but widely untested. As a previous conversation from a few month back ended... it should work in theory.

Also Lazarus Packages are designed to work with source only. There is no option to install .ppu's and .o's in Lazarus IDE. But this is another issue, best left to be discussed in the Lazarus mailing list.

Regards,
- Graeme -

As Sven wrote, I guess the FPC community needs to think about this. XE2 already supported MacOS X (but through fpc). Haven't checked the OS-X in XE3 though.

As for the multi-platform: they are rewriting their compiler to be a front for llvm. Probably they will make a few code-generators and optimisations as well for the middle and back-end.

But I doubt the "bpl" idiom was or will get ported past Windows variants (if it even gets done in the new compiler format).

Don't forget that with FireMonkey^2, and a Firemonkey component model, they plan to tackle facade-type implementations to make Android/IOS an output target as well from the same code-base. As is today with the MacOSX target in XE2 and XE3, there are restrictions as to what you can use (i.e. must be firemonkey HD or 3D app if you want Mac or "x-platform", meaning 2 platforms ... well, 3 if you count Win32 and Win64 as separate :-))

That said, what is stopping us (the community) to at least start on this? a branch? core-buy-in?

kind regards,
Dimitri Smits

----- Oorspronkelijk e-mail -----

Dimitri Smits wrote

As Sven wrote, I guess the FPC community needs to think about this. XE2 already supported MacOS X (but through fpc). Haven't checked the OS-X in XE3 though.

Only iOS was done through FPC. Mac OS X itself was already done through Embarcadero's compiler in XE 2.

They don't need to implement it for each target like we won't implement it for each target either (thinking about DOS, NDS, GBA, Wii and embedded here).

That said, what is stopping us (the community) to at least start on this? a branch? core-buy-in?

From my side: time :) (and a huge list of other features I want to have implemented first)

Regards,
Sven

Graeme Geldenhuys wrote

Hi, Any FPC developer willing to comment on the status of some of these issues (that have been years overdue)?

It narrows basically down to the fact that fpc lacks developers and contributors, or do I miss something?

2012/12/18 Florian Klämpfl :

That, combined with the fact that the existing developers and contributors spend their resources on different things than the original poster seems desirable.

Vincent

Well, the more contributors, the more likely it is that several features are implemented, so this is imo already a consequence.

Don't you think after more than a year of decision-making about Unicode it is necessary for a serious compiler to actually make and communicate a decision? Don't you think for a serious compiler there should be communicated design goals and the missing pieces and necessary boring improvements should be done before adding sexy new things?
Or is FPC simply a playground for the FPC-developers? Then that should be communicated too and I probably was wrong to invest so much time into the development of MSEide+MSEgui.

Martin

Graeme Geldenhuys wrote

Hi, Any FPC developer willing to comment on the status of some of these issues (that have been years overdue)?

For some reason this thread ended up in my spam box. Hence the late reply.

As for the question:

I've planned to do some work on the unicode string type.

Then:

Please don't bother obfuscating names, the mailing lists are public:
- John Doe 1 is Martin Schreiber himself,
- John Doe 2 is Graeme Geldenhuys.

Concerning "Fighting with Michael or Marco:"

Martin has virtually sole control over MSEide-MSEgui. In that position, he does not need to compromise.
This is quite different from the situation in Free Pascal:

From our perspective, Martin is simply impossible to work with.

It's either his way or not at all, always has been so.

In the last discussion I've gone out of my way to find solutions for his problems that are acceptable for our own goals, and found them too. I even implemented some.

In the end he just closed the discussion with "I don't like this approach".

I have no problem with that, but then be honest and do not put the blame on us.

I also sometimes must compromise in the FPC team. I also do not like all decisions. I also do not want all the latest Delphi features.

But we are cooperating in a group, and then compromise is necessary.

If Martin would be more cooperative and sometimes accepted a compromise, we would probably have solved many of the issues that are bothering him a long time ago, simply because we'd have more manpower and he'd be included in the FPC core group. The first attempt to do so, on the database design, failed for exactly the reasons outlined above: Compromise was not acceptable.

A pity, because there are lots of ideas in MSEgui that I like and plan to copy, time permitting. That is a waste of time and double work that could have been avoided, but alas...

Let us not forget that FPC is gcc+libc+gnu-binutils combined.

All those projects have entire corporations backing them up if need be. We do not. FPC is a small team consisting of volunteers who do this after hours. We do not get payed for this.

It would be good to keep those facts in mind before ranting.

Michael.

Michael Van Canneyt wrote

It would be good to keep those facts in mind before ranting.

I was simply bringing some of those questions (which I had too) to light. Unicode has been "under development" for many years, and has come to a halt - with no final decisions being made. This is very frustrating for those using FPC. And even if we wanted to contribute in that regard, how could we, if the FPC team itself doesn't know what it wants.

I would also like to point out that I am well aware that FPC is a "part time project" for you guys. I never demanded anything with my original post, simply asking what the progress was.

In the same breath, you guys work on FPC - that's your hobby project. Others work on Lazarus, MSEide, fpGUI, tiOPF, FPTest, FP Debugger, OnGuard, etc etc. So comments like Florian's - suggesting that "if you want a feature, implement it your self" is often not an option. I'm skilled in certain programming, definitely not compiler design. So it seems quite logical to leave such compiler work to those that know how to do it, or that are already familiar with the code base. I do contribute to the FPC project where I can, eg like in the fpdoc tool, documentation updates etc. This might mean jack-shit to somebody like Florian, but we are not all compiler designers, and I'm already swamped with other open-source projects I work on.

Anyway, good to hear that Unicode progress might actually happen.

Regards,
- Graeme -

It does.

We know what to do. What we lack, is time.

Status currently:

- Inoussa has made a native unicode string manager. A large effort. There were some issues with licensing, namely the unicode.org files are under license. We tried to get these resolved, but there was never an answer. This took time.

- There was waiting for the merge of the JavaVM branch to trunk. It contains several parts needed to get the codepage aware string working fully (notably the ability to specify what the 'string' keyword actually means)

All that is left to do is make the unicode string manager part of RTL, and create the unicode-string RTL. This is a non-trivial task, which takes time and needs to be worked on preferably in one fell swoop.

Do not underestimate this.
The RTL in trunk will probably be highly instable during that period.

After that there will be 2 RTLs:
1. The classical RTL, compatible with what you have now.
2. The unicode-string RTL which will use the namespaces of Delphi.

There is no other way to implement it without breaking backwards compatibility and a reasonable amount of keeping up with Delphi.

(and at the same time not have 4 RTLs: unicode/not unicode vs. namespaces/no namespaces)

I would also like to point out that I am well aware that FPC is a "part time project" for you guys. I never demanded anything with my original post, simply asking what the progress was.

I understand. What made me react is the original rant in your forwarded mail.

I'm fed up of being treated like a stubborn bastard, opposed to change.

No, we are not opposed to change.

But

Yes, we have our own goals.

A reasonable amount of Delphi compatibility happens to be one of them.

That may not matter to Martin, but it does to us. So compromise will be needed, and any proposed change is always viewed in the light of our own goals.

People that want to help with dynamically loadable packages, are always welcome. But the subject is NOT easy. Any solution will have to work on ALL platforms.

In particular, your contributions in any area you can help with, were and are valued.

Michael.

Michael Van Canneyt wrote

We know what to do. What we lack, is time. Status currently:

Thanks for the update. Most of what you mentioned was unknown to any person outside the FPC core team. So to us outsiders, it seems like progress has halted.

Regards,
- Graeme -

Graeme Geldenhuys hat am 21. Dezember 2012 um 13:49 geschrieben:

Huh?
I'm not part of the FPC core team. I knew all that from the mailing lists.

Granted, with all the lists, chats, forums, repositories and trackers it's hard to know the current state.
It would be nice if someone of the FPC team could update the status page:

http://wiki.freepascal.org/FPC_Unicode_support

So to us outsiders, it seems like progress has halted.

Mattias

It would be good to keep those facts in mind before ranting.

Cutting out a whole lot of crap: as somebody very much on the periphery of the project, I'm disappointed to see sentiments of this tenor being aired in public.

First, in traditional debate it's not considered necessary to be quite so explicit naming names: anybody who follows the project would probably understand "certain persons are difficult to get on with because they have their own agenda" entirely adequately.

Second, there are a number of other projects and commercial products with which we are in friendly competition. If there's internal friction, there's no need to give them a grandstand view.

I'd suggest some sort of resolution mechanism is needed, where somebody can request permission to address the core developers on a private list, and the core developers can "call somebody onto the carpet". Provided that the public was aware that attempts were being made to resolve something off-list, and provided that a summary was posted when the dust had settled, I think something like that would be adequately transparent.

Me, I've no complaints about the project except for the difficulty of working out the details of the compiler etc.

Apart from that, happy Midwinter Solstice everyone.

See mail from Michael.

Don't you think for a serious compiler there should be communicated design goals

The mission goal of FPC is: develop an open source pascal compiler written in pascal in a community effort.

and the missing pieces and necessary boring improvements should be done before adding sexy new things?

Maybe people should indeed first work on the compiler instead of developing another gui and ide. I cannot tell you neither any other developer what they have to develop. We could close down all write accesses to fpc svn till the features you need are developed but I fear you won't get them either this way. People would just fork fpc or fpc would die. So what do you propose?

Or is FPC simply a playground for the FPC-developers? Then that should be communicated too and I probably was wrong to invest so much time into the development of MSEide+MSEgui.

If nobody is interested in features you need, bad luck for you, you have three possibilities: develop them yourself, pay somebody to develop them or use another compiler.

Well, then you depend on those people.

or that are already familiar with the code base.

Believe me, the compiler contains a lot of code I never have touched or even seen ...

documentation updates etc. This might mean jack-shit to somebody like Florian, but we are not all compiler designers,

Me neither, I'am electrical engineer and twenty years ago I needed a 32 bit pascal compiler.

and I'm already swamped with other open-source projects I work on.

This applies to everybody I guess :)

Michael Van Canneyt wrote

- Inoussa has made a native unicode string manager. A large effort.

Is this code publicly available somewhere?

ListMember wrote

On 2012-12-21 14:26, Michael Van Canneyt wrote: > - Inoussa has made a native unicode string manager. A large effort. Is this code publicly available somewhere?

It's attached to a bugreport in Mantis somewhere.

Michael.

I did a search for 'Inoussa' and I got these 4 items --all closed. Neither one contains any significantly sized attachment.

cpnewstr's charray_to_ansistr conversion is not ok (due to codepage parameter)
http://mantis.freepascal.org/view.php?id=17754

Access Violation when assigning "Self" to local interface Reference in constructor
http://mantis.freepascal.org/view.php?id=16901

Apache Bindings - apr.pas
http://mantis.freepascal.org/view.php?id=11460

Memory leak with interfaces
http://mantis.freepascal.org/view.php?id=7281

Can you (or someone else, of course) think of a better search string to locate it?

Neither one contains any significantly sized attachment.

cpnewstr's charray_to_ansistr conversion is not ok (due to codepage

parameter)

http://mantis.freepascal.org/view.php?id=17754 Access Violation when assigning "Self" to local interface Reference in

constructor

locate it?

Go to View Issues, click on the + before the search bix, click in the appearing entries in the top left for "reporter" and select the user "Inoussa OUEDRAOGO" in the list (strangely the user exists twice, I used the first one) and click on Apply Filter. The second entry should be the correct one (you should be able to judge this from the issue's description).

Regards,
Sven

Thank you for that detailed navigation; I got it now. [ http://mantis.freepascal.org/view.php?id=22909 ]

Does anyone know if the license issue has been discussed in any public maillist/wiki etc.

Reason I am asking is this: Having read (now and several times in the past) unicode.org's license [
http://www.unicode.org/copyright.html#Exhibit1 ] I simply cannot see what it is that is so (or, rather, at all) restrictive.

It would require every FPC made program to include the unicode license. By itself maybe not a problem, but this contrasts with the fact that for years, you could make an FPC program without any additional licenses, if you didn't use any third-party libraries.

Inclusion in the RTL would make this an obligation for every FPC program.

However, last status/opinion is that this is only so if you were to copy the files verbatim. If the data contained in the files is somehow recoded, then it would probably not apply.

We didn't get any answers to our inquiries. But we found that Delphi also uses these files, and they put forward the above argument on the Delphi forums when Paul Ishenin inquired.

It boils down to: Only the form is copyrighted, not the actual data.

We hope they are right, otherwise every Delphi program as of Delphi 2009 is in violation of the unicode license :-)

Note that I am not a lawyer, the above are therefor not rigorous legal truths.

Michael.

Thanks.

Thanks.

In 1999? There was no other useful gui and ide for Free Pascal IIRC.

I propose to extend and render more precisely the mission goals of FPC and to concentrate the power on the defined goals.

Thanks for the message.
I must say, in MSEide+MSEgui project the things are handled a little bit different. For example I never planned to internationalize MSEide because it complicates things, is a boring task and I did not see a benefit. Now a Russian speaking user convinced me, that MSEide can be used in Russian schools only, if translated to Russian. I liked the idea to use our loved Free Pascal in schools, so I took the boring task and moved the string constants to tstringcontainer so it can be translated. Remember, FPC does not support Unicode resource strings up to now. ;-)

Martin

That would mean telling people what to do: not to do what they want, but what is needed for the goals of FPC.

That said, the mission goals are quite known:
- Free, cross-platform Object Pascal compiler.
- As Delphi compatible as possible.

All within the restrictions of available manpower.

You say it yourself: 'I liked the idea'.

This is not so different from what we do:
The bugtracker is filled with evidence of this.

Once more, the main restriction is lack of time. Not lack of direction. If it was easy it would have been done already.

Michael.

Please note that the message has not been posted to the list by me. Additionally many members of the Free Pascal community, including me, don't know English language so writing explicit statements is unavoidable.

[...]

Apart from that, happy Midwinter Solstice everyone.

Yup. :-)

Martin

I am not a lawyer either, but I did notice that they were quite pedantic (or, a better word might be meticulous) with their wording: In the license text they state that "Data Files *or* Software" must contain their license text.

Unicode.org guys are as much coders as linguists, so I believe they have used '*/or/*' (as opposed to '/*and*/' or '/*and/or*/') for a reason.

So, as an addition to what you have said, my take is that including their copyright in the data alone will suffice --programs/software need not have to bear the same text.

[Plus, of course, there should be a clear statement that it is 'modified'. And, the documentation should bear license text. Wiki should do.]

Martin Schreiber wrote

> Please note that the message has not been posted to the list by me.

My apologies Martin. I should have taken your questions and rephrased them in a list form. To save time, I simply obfuscated the names - probably not the best idea. The names where not the important bit anyway
- the listed items and their status in the FPC project was.

Regards,
- Graeme -

Mark Morgan Lloyd wrote

Cutting out a whole lot of crap: as somebody very much on the periphery of the project, I'm disappointed to see sentiments of this tenor being aired in public.

Mark, much of what happens with the FPC project seems to be done in secrecy, or a private "core only" mailing list. So it takes message threads like this to actually find out (for the rest of us) what is going on.

It's an open-source project (not commercial), so I would have thought open discussions should be a given.

Regards,
- Graeme -

Florian Klämpfl wrote

The mission goal of FPC is: develop an open source pascal compiler written in pascal in a community effort.

You forgot the last bit.... "and be Delphi compatible!"

Maybe people should indeed first work on the compiler instead of developing another gui and ide.

A compiler on its own is a pretty useless item. It needs advanced GUI frameworks, IDE's and other large apps to fully test the compiler and drive its features.

Regards,
- Graeme -

They most definitely are.

But sometimes you need to discuss with knowledgeable people, without other people interfering.

Search the list archives:
The string type has been discussed many times on the mailing lists. We have heard and listened to the wildest ideas.

Nothing wrong with wild ideas, who knows, maybe there is somewhere this brilliant gem amidst of all the rough...

But in the end, it is us that will do the work.

So then we have some focused conversations about merits and drawbacks in a private list, with some implementation details maybe, but definitely without the noise of the public mailing list.

Otherwise we'd never reach a conclusion, let alone an implementation. Which is what people are waiting for, after all.

So if you don't mind, we'll close the thread, otherwise we'll never start on an implementation.

Michael.

In our previous episode, Graeme Geldenhuys said:

I think you have a wrong idea on what the core list contains. For 90% it is technical internal discussion of the compiler devels, and of the rest a lot is about new platforms/targets.

I don't think direction on unicode (or even general) came up since the last unicode discussions on fpc-devel/pascal.

IMO this is actually a consequence of the community effort. Most people are simply interested in this and spent time in it. The community effort means also that one tries not to break other people's work that's why we try to keep backward compatibility as much as possible.

Martin Schreiber wrote

I propose to extend and render more precisely the mission goals of FPC and to concentrate the power on the defined goals.

And you think people will work on this defined goals instead of maybe completely other projects? Or just fork FPC?

In 1999? There was no other useful gui and ide for Free Pascal IIRC.

That's true but FPC was also far behind at this time.

And what had happened if you didn't like the idea?

It depends if I see that someone actually works for the Free Pascal community. If it does not impair the quality of MSEide+MSEgui I help independent whether I like the idea or not.

Martin

Yes. It is a pleasure to work for a project which reaches the goals.

Martin

If I had to adhere to goals that I don't see fit I'd not have the enjoyment I currently have when working on FPC. Remember: I (and the other devs) work on this in their free time. At least I have enough goals to adhere to either in university don't need another project where I need to do this which I additionally work voluntarily on.

Reagrds,
Sven

Marco van de Voort wrote

I think you have a wrong idea on what the core list contains.

LOL. And how is anybody supposed to know what goes on - it is a PRIVATE mailing list.

I don't think direction on unicode (or even general) came up since the last unicode discussions on fpc-devel/pascal.

OK, so once again it is proven that Unicode is just not "sexy" enough for the core team, so it will stay stagnant for a few more years. That's unless a member ignores all discussions and does his own thing [or gets paid for the job]. As Florian likes to says so often, whoever implements it decides. Unfortunately that courtesy is not extended to non-members, because what good is a patch [of such magnitude and effort] with no chance of ever being committed. So we are at the mercy of the FPC gods.

Many of use non-core developers have given our input with multiple solutions, to try and help the private discussions along. But I guess all of us are not knowledgeable enough people. What a nice F*** Y** to the community.

Well, let me just say that the idea of two RTL's is rather ridiculous too!! You guys keep bitching about not having enough developers, so how on earth do you think you are going to be able to maintain developing two RTL's, patching too RTL's when bugs are reported, inform the public to remember to mention which RTL they are using when reporting bugs, keeps those two RTL's in sync over time etc. Yeah, it seams you guys are sometimes not to knowledgeable either. All you are going to do is create more work for yourselves. But hey, who are we to state the obvious.

Anyway, I can see where this thread is heading... just another waist of time. So I'll stop here.

Regards,
- Graeme -

Well, let me just say that the idea of two RTL's is rather ridiculous too!!

It's not different from Delphi, where the introduction of UnicodeString required a renewed RTL, VCL and IDE. Who should do the same for FPC and Lazarus, and tell the users that they either have to stay with an old (frozen) Ansi release, or must upgrade all their code and component libraries to the new strings, RTL and LCL? IMO a typical loose-loose situation :-(

Anyway, I can see where this thread is heading... just another waist of time. So I'll stop here.

Yeah

DoDi

Which is why I suggested that a semi-formal way of taking disputes to it might be in order, /provided/ that the results were summarised for the public. I don't think that public name-calling is at all in the interest of the community of Object Pascal users as a whole, and note that I'm including both the dwindling band of classic Borland customers and the users of FPC and Lazarus in this.

Many of use non-core developers have given our input with multiple solutions, to try and help the private discussions along. But I guess all of us are not knowledgeable enough people. What a nice F*** Y** to the community.

Graeme, if you don't like it then consider forking the project. However, I'd suggest that if you even start along that path the prerequisite is better documentation, and to get anywhere with that you'll need to work cordially with the existing developers for the foreseeable future. Which is something that, frankly, I think you find difficult.

What makes you think so? Of course, one does not decide if he forgets about the community approach which means implicitly: break other people's code and work as little as possible. This is indeed one of the policies of the current svn maintainers: patches should cause as little as possible regressions.

because what good is a patch [of such magnitude and effort] with no chance of ever being committed.

It would if you realized that "community[Graeme Geldenhuys]".

Did you know that my addition of target NativeNT was published as patch to the bug tracker? Did you know that I wrote patches for the cppclass feature to get it a bit more working than before? It was only the class helpers where I got access to a personal branch in SVN and only the generics when I got access to trunk.

You need to show the others that they can trust you and that you mean no harm and then they'll treat you accordingly.

The best example is this: I had problems commenting on closed/resolved bugs which were assigned to me, so Florian simply made me from "developer" to "manager". It's all about trust....

The two RTLs isn't as difficult as you think:

=== System.System.pp begin ===

{$define USE_UNICODE}
{$include system.pp}

=== System.System.pp end ===

=== system.pp begin ===

// where the mode is set:
{$mode objfpc}
{$ifdef USE_UNICODE}
{$modeswitch UNICODESTRINGS}
{$endif}

=== system.pp end ===

The same for the other units.

Then one just needs to pay attention whether USE_UNICODE is defined or not inside those units and write the code accordingly. I don't say it's a pencake, but it isn't "ridiculous" and the only approach that is really viable for us as - as you said - we only have so much developers.

Regards,
Sven

No, but I think you hugely overestimate what goes on in that list. It's simply a list for some highly technical discussions with people who know how things work in the compiler and RTL.

It's not a list where we plot and scheme.

No, this is not so. It is really a matter of time. I think people underestimate the impact of the implementation.

There are almost 1500 units, which potentially need checking and/or changing.

The task is simply daunting.

And that the prospect of 2 RTLs is not appealing is something we know, but it's either that or throw overboard backwards compatibility.

The Delphi engineers have chosen the latter.
It has cost my company lots of time and money to digest that. You yourself are affected by this in tiOPF: Version 3 vs. Version 2 ?

Michael.

Hey, don't minimalize the work I will do ;-)

Sven is right about how we'll deal with it.
The real work is dealing with the consequences of this apparently little change.

As he wrote

Then one just needs to pay attention whether USE_UNICODE is defined or not inside those units and write the code accordingly.

And then go over all packages to see if all still compiles/works. for example the DB units may prove to be fun, as they mixe low with high-level code.

Michael.

IMO, I wouldn't support wide-character (UnicodeString) strings for anything new.
In the beginning the wide-character string had the advantage of being able to represent
all characters with 2 bytes, but this is no longer the case. I would switch to UTF-8
instead and keep characters 1 byte long. A switch to UTF-8 only affects a small amount of the code-base, and doesn't break string references.

In fact, I introduced UTF-8 in my OS recently, and it was easy to do. So not supporting wide-character strings does not mean you must keep an old ansi state.

Leif Ekblad

? No, the old RTL will remain maintained. It's the same codebase, just recompiled.

Michael.

Sorry. I just couldn't let Graeme's "ridiculous" stand around uncommented :)

Regards,
Sven

Leif Ekblad wrote

IMO, I wouldn't support wide-character (UnicodeString) strings for anything new.

I don't like to read that. ;-)

Martin

UTF-8 is fine for external representation, and for code that's near it. After all, it's merely a form of compression in the same way that HTTP etc. uses compression for content.

I think your point about two bytes now being insufficient to represent all possible Unicode codepoints is valid, but since things like expression parsing are made much more efficient by being able to iterate an array that's an argument for moving to a wider internal representation- not a narrower one.

In our previous episode, Graeme Geldenhuys said:

OK, so once again it is proven that Unicode is just not "sexy" enough for the core team, so it will stay stagnant for a few more years.

No. Simply getting older.

No. Core members have some freedom in doing their own thing because of a proven track record, and the knowledge they will generally see a feature or a development to the end.

And indeed, external patches and committers first need to prove themselves.

But there are many precedents (like Giuliano doing the resource strings), and committers are being added all the time. There probably near an handful of committers who are newer to the project than you. More if you count db-core.

But then you have to work within the team, and respect some traditions and sentiments. And if there is one thing I hope others get from this discussion is that you and Martin never got that. But at least Martin _tried_ during the 2.2.2 tcomponent rewrite.

Same goes for Mr. Schnell.

Many of use non-core developers have given our input with multiple solutions, to try and help the private discussions along.

I haven't seen actual input. Mostly I only have seen some simplistic outlines of radical new solutions that weren't fleshed out enough to fill the backside of a beercoaster. And most of them were thrown in together with some anti-Delphi sentiment.

I have never even seen serious attempts from each one of you (like categorizing the objections, adapting to it, keeping documentation or points lists). (for the unicode 2.2.2 work by Martin was certainly valued)

But I guess all of us are not knowledgeable enough people.

I wouldn't say that. But the two of you have the problem you always choose radical solutions, don't have a history of working on core, never done major projects inside

What a nice F*** Y** to the community.

So basically you just go on bullying in the maillists till you get a carte blanche up front. Well, IT WON'T HAPPEN.

I dare you to bring up one piece of proposal that is detailed, consistent, and mustn't be extracted from some maillist discussion, and that shows some signs that criticism on it was processed.

Well, let me just say that the idea of two RTL's is rather ridiculous too!!

It was my idea actually, to repair the worst problem with delphi compatibility; the fact that Delphi broke backwards compatibility.

But I abandonned it, because there seemed to be little support, and people kept believing an overload here and there would solve the problem. IOW there is no acceptance of the fact that the default stringtype is much more a global change than a per unit change.

So in spring I gave up, and am now in favour of a 100% delphi compatible utf16 solution, compatibility break included. (maintain 2.6.x a bit longer though, if people care)

You guys keep bitching about not having enough developers, so how on earth do you think you are going to be able to maintain developing two RTL's, patching too RTL's when bugs are reported

Again you show your lack of knowledge. It _was_ about two releases built for every target from a single codebase. One 1-byte as default, one 2-byte.

I/we hoped we could polish away the different 1-byte codepages away in the RTL initialization. (so the 1-byte RTL could be used for both the old Delphi 1-byte as one where it is hard utf8. There are some problems with stdio there though)

But that means one codebase compiled twice (once for 1-byte, one for twobyte), only with different default types, so that inheriting methods with string types keep working.

inform the public to remember to mention which RTL they are using when reporting bugs, keeps those two RTL's in sync over time etc.

It is that, or rewrite it. Note that originally the two RTL solution was a temporary solution, to allow Lazarus to gradually change from manual to compiler supported unicode types. (and thus not stick to an old release for an extended period of time).

I then still assumed that over time on windows the 1-byte one would disappear and on *nix the two byte one. Only when protests came, it became somewhat permanent.

Yeah, it seams you guys are sometimes not to knowledgeable either. All you are going to do is create more work for yourselves. But hey, who are we to state the obvious.

This message is a perfect example how you let yourself guide by sentiment. You seem to have started to believe your own advocatism, and think it is evil core that is holding you back, rather than the fact that doing a few minor bugs and spelling corrections in the docs will not give you carte blanche up front to outline major compiler features that will haunt the project for ten+ years.

And everytime that is pointed out to you, and are advised to start working, you mumble something non committal about the great (but irrelevant) work you say you do outside the FPC/Lazarus project, and keep numb for a while.

Of course after a while you restart with your usual arrogance. I've been timing it, and usually the period is about 4 months.

Anyway, I can see where this thread is heading... just another waist of time. So I'll stop here.

For 4 months.

In our previous episode, Leif Ekblad said:

all characters with 2 bytes, but this is no longer the case. I would switch to UTF-8 instead and keep characters 1 byte long. A switch to UTF-8 only affects a small amount of the code-base, and doesn't break string references.

Any solution will need a complete check. Since old code will probably store multiple encodings in the "ansistring" type that must be checked. See e.g. the work done on Zeos by Michael Hiergeist.

That goes both for the case that the default type is 1-byte and 2-byte.

So that is appearances only (ah, they are both onebyte, so not much will change).

Any failure to do that will result in an infinite adding (and maintaining) of ad hoc conversions.

Michael Van Canneyt wrote

? No, the old RTL will remain maintained. It's the same codebase, just recompiled.

It was impossible to deduce that from your earlier reply

With the new information at hand, it seems a lot more manageable than I first envisioned.

Regards,
- Graeme -

Correct.

With the new information at hand, it seems a lot more manageable than I first envisioned.

I sincerely hope so, given the time constraints :-)

Michael.

Obviously, once somebody submits a patch it's there for anybody to consult and use. So people who are into vanity publication shouldn't feel left out: if it's good, the community will read it :-)

I think it's a pity though that, particularly once an issue is closed, it can be difficult to find the attached discussion etc. Some way of finding all patches which have been submitted against a particular unit could be useful, since whether accepted or not the discussion could be enlightening and in some cases rejected code could be a useful starting point if somebody later has a related problem.

Do you know how RTTI will be encoded?

Martin

The RTTI will stay ASCII as we (at least as far as I know) don't plan to extend the compiler to allow non-ASCII identifiers.

For accessing property values the same applies as for any other unit. If you use unit "typinfo" the unit will use AnsiString parameters and "System.TypInfo" will use UnicodeString parameters (at least for those methods where the parameter is just a "String"). Internally then the parameters need to be converted accordingly of course.

Regards,
Sven

I would guess short/ansistrings, since pascal identifiers must be a subset of ASCII anyway. It makes no sense to start using Unicode here.

The extended attributes (recently implemented by Joost) may be an exception, I'll need to consult with Joost on that.

Michael.

Just in case: they are not in trunk yet, but only a branch to simplyfy review through other core devs.

Regards,
Sven

In our previous episode, Michael Van Canneyt said:

> Do you know how RTTI will be encoded? I would guess short/ansistrings, since pascal identifiers must be a subset of ASCII anyway.

Not Delphi 2009+ btw, which allow UTF8 identifiers.

That's how Unicode currently is handled, e.g. in Lazarus, and the mix of codepages, e.g. for filenames, is a source of eternal trouble :-(

I'd be happy with AnsiStrings with an Encoding, but a good implementation would require UnicodeStrings for the codepage conversion.

DoDi

Sorry, I doubt that it is so easy :-(

DoDi

What will FPC do?

Martin

As I already wrote there are currently no plans to change that FPC supports only ASCII identifiers. And I hope it stays that way as the other might be harder ti be fully supported on other, more exotic platforms... But of course if someone wants to change it...

Regards,
Sven

Hopefully not a thing :)

I don't think we can trust on that. I hoped that FPC will not use cpstrnew too. So if somebody implements non ASCII identifiers because he needs a second source Delphi compiler it will be merged because the addition does not break existing code. I assume utf-8 identifiers would not be very difficult to do in compiler.

Martin

But what will the rule be as to whether something's a valid identifier? Will it have to start with something known to be a letter, or something not known to be a digit or reserved character?

In our previous episode, Mark Morgan Lloyd said:

Afaik the latter. You specify what is not allowed rather than which are allowed.

But sourcecode edited on multiple platforms might be a problem, (e.g. ligatures, denormalization and other forms of slightly different characters), this could lead to making the comparison of identifiers expensive, which is what you don't want in a compiler.

But I don't know how big that problem would be. Maybe it is negiable.

Hm that makes it easy to have an incomplete list, that could later become a problem

half-width spaces etc..., control chars (RTL/LTR...), currently unused codepoints (that could become anything in future...)

In our previous episode, Martin said:

Still shorter than what is allowed. And I'm pretty sure Delphi does it that way, it was said during the Delphi 2009 sales presentation iirc. (when they demonstrated unicode in source and object inspector)

Martin wrote

half-width spaces etc..., control chars (RTL/LTR...), currently unused codepoints (that could become anything in future...)

As Marco said, the list will be smaller than the "allowed list".

Also the Unicode specification defines blocks or categories for code points, so that could be used too. eg: Take a look at TCharacter.IsNumeric(..) implementation. It doesn't do actual code point comparisons, it simply checks the Unicode category of the passed in code point.

Regards,
- Graeme -

In our previous episode, Hans-Peter Diettrich said:

Just a quick memory dump (as said it has been 6months since issues like this were discussed in depth):

It is not a blind recompile of course. Everywhere where character access is done has to be reviewed.

But any FPC rtl has some 2-byte and 1-byte support, so the amount of code to be added is not that much. At least not till e.g. compiler structures like RTTI will follow.

Like always with Delphi compatibility, the level of compatibility differs greatly with what you expect.

Usually first the main support is added (the ability to have unicodestring as a basetype for RTL and class hierarchy), other things follow later. That is something else then being fully D2009 compatible.

So initially there will probably conversions when you pass a string to units that have many stringroutines (like dateutils,strutils, sysutils, fmtbcd etc), new utility units in Delphi will be missing etc.

But I hope it will be enough to allow Lazarus (and other big projects) to start working on conversion.

The process then continues (with every issue discussed, checked and doublechecked) with the other remaining defects, and probably some effort will need to be made to kill excessive conversions (like on windows utf16 -> utf8 -> utf16), RTTI, Literals
(if delphi literals are utf8, it will
probably be only a matter of time till a bug is found because sb is poking with pointers in it), new delphi units etc etc.

You can also imagine that just the task of checking and fixing the windows headers for all $ifdef unicode cases is daunting. (we are still fixing heaps of 64-bit issues in them, 9 years after the initial 64-bit port)

The current logical next step is to start revising the OS interface to be unicode, mainly on Windows.

That alone (the fact that the windows unit swaps to unicode calls by default) is a horrible incompatible change.

If you use a utf8 lib, you can check attributes of each codepoint.. though that may be slower

Florian Klämpfl wrote

If nobody is interested in features you need, bad luck for you, you have three possibilities: develop them yourself, pay somebody to develop them or use another compiler.

BTW, I actually think about to fork Free Pascal. Not in the near future but the primary goals are defined already:

- Back to the roots.
- Add the necessary to build the most productive universal programming language.
- Remove the not necessary.
- Produce at least as good code as Delphi 7.
- Compile at least as fast as Delphi 7.

Martin

Martin Schreiber wrote

BTW, I actually think about to fork Free Pascal. Not in the near future but the primary goals are defined already: - Back to the roots.

What are the roots?

- Add the necessary to build the most productive universal programming language. - Remove the not necessary.

Subjective, requirements differ. I assume this definition of "productive" and "necessary" will be dictated by you and drive other people away from the project. Just like e.g. your coding style has previously.

- Produce at least as good code as Delphi 7. - Compile at least as fast as Delphi 7.

You know that sacrifices need to be made to make a compiler cross platform and easily portable. You can't have it all.

It will be a shame to see your considerable talent wasted on another project with a bus factor of approximately 1.

Henry

Yup.

We will see.

It will be a shame to see your considerable talent wasted on another project with a bus factor of approximately 1.

Correct, but for me an important 1. ;-)

Martin

Leif Ekblad wrote

A switch to UTF-8 only affects a small amount of the code-base, and doesn't break string references.

I do agree with your point of view,
I also see that major Pascal/Delphi frameworks have chosen UTF-8 for string such as XMLRAD and mORMot.

Jyv.

It would be good if, prior to forking off, you would do some study and post some proposals here.

As long as your proposals do not include reducing the number of platforms/CPUs (an obvious optimization), I'm sure they will get serious consideration:

Everybody is aware of the speed difference between Delphi and FPC.

The compiling itself (parsing/producing assembler code) is not slow.

From what I remember, the problems you (and everyone else) experience

with smartlinking and so on are largely due to the GNU linker being slow and memory hungry.

What concerns produced code: I think that the largest speed gain there will come from a reorganisation of the exception handling.

What concerns 'stripping the unnecessary' : if you are talking about language features, I doubt you will gain much speed by that.

I am also fairly confident that if you would create a patch to introduce a new compiler mode switch {$MODE D7} which would selectively enable/disable some language features to "go back to the roots", it would be accepted. (I myself would use it ;))

In each case, by doing so, a wider community would benefit from your efforts. (I am assuming, of course, that this interests you).

Michael.

Although I think the time is approaching when some CPUs- IA-64 etc.- and OSes could usefully be moved into an "attic" subtree.

They aren't compiled anyway, so they don't affect the compiler's performance negatively.
So it does not matter whether they are inside the compiler tree or something else. And after all there could somebody who takes enough interest to implement such a CPU... *cough* m68k *cough* ;)

Regards,
Sven

AFAIK there are significant differences in parsing and code producing too. Last time I checked on Windows with the internal FPC linker Delphi 7 compiled and linked about 10 times as fast as FPC.

The goal here is to reduce the count of the used different concepts so one has not to learn much to tap the full potential of the language.

It would unfortunately blow up the code base again.

Martin

AFAIK Delphi's command line compiler does not allow you to skip the assembling and linking phase, so the fairest comparison would be to compare the compilation of a single unit as there the linking phase is skipped. If Delphi is still better there then there are two possibilities: improve the internal assembler or the parser/code generator. Please keep in mind though that FPC's code generator is written in such a way that the backend can be switched rather easily. As we learned with the developer's blog entries about Delphi XE2 this wasn't the case with Delphi XE and older. So it could be that you can never reach the speed of Delphi 7's compiler as FPC is more portable.

Regards,
Sven

I suggest that Florian answers this, he has more exact numbers.

Last time I checked on Windows with the internal FPC linker Delphi 7 compiled and linked about 10 times as fast as FPC.

AFAIK it is still linker related: the assembler files are on disk, the structure of the files is geared towards the GNU toolchain etc.

This is a goal I fully support.

And I consider it a perfect fit for a mode switch.

By about 10 lines in the compiler, so this is hardly an argument.

And - assuming you don't insist on forking - there will be more people to manage these 10 lines.

As we've explained elsewhere, not breaking things is always an important thing, so no-one would take this switch away.

Michael.

Which is why I wrote that limiting the number of CPUs/Platforms would be the obvious optimisation, but one which is incompatible with FPC's goals.

Michael.

In our previous episode, Sven Barth said:

The numbers Martin names (up till 10 times slower, even without linking) are the numbers I have in mind too. IMHO denial without tests is unfair.

I had a setup where I compiled zeos with FPC and Delphi (which doesn't link), and iirc the results were also 7-10times, with delphi XE.

When I'm home after Christmas I'll play a bit and see if I can duplicate the experiments with Delphi XE3.

It would not only be limiting the amount amount of CPUs/Platforms. As long as at least two CPUs exist that use the compiler's current backend system it can't be optimized (at least not in that sense). You'd need to basically bring the code generator back to a pre-1.9 state to gain significant performance with reducing the number of target CPUs.

Regards,
Sven

I'm not denying them, I'd just like to what the performance of the phases of the compiler is compared to the corresponding phases of the Delphi compiler. Then we can optimize that part. Or if it comes out that the problem is that we are more platform independant than e.g. Delphi 7 than we can either try to improve other parts of the compiler or we are simply out of luck, because this platform independance is the main benefit of Free Pascal and I believe not many users are willing to give that up...

When I'm home after Christmas I'll play a bit and see if I can duplicate the experiments with Delphi XE3.

Yes, XE3 numbers would definitely be welcome. If possible could you also compare x86_64 of XE3 and FPC?

Regards,
Sven

Agreed. But combinations that don't compile meaningfully (e.g. the compiler targeting IA-64) or at all without at least backported patches (various RTLs including MacOS Classic, Amiga etc.) could IMO usefully be in compiler/attic and rtl/attic. In other words, anything in the main directories should compile and run, and if somebody spots that that is no longer the case then it's a candidate for being fixed.

Obviously this is a personal opinion. I for one am not trying to hector, badger, buffalo or otherwise irritate the coee team :-)

From: "Marco van de Voort"

I also got similar results few years ago. Also I made some profiling and found a bottleneck in FPC. It is HUGE number of creations of small (or even tiny) objects (various compiler nodes). Each object creation allocates small chunk of memory and zero fills it. It is very time consuming.
It is possible to seed-up compilation by allocating memory for nodes from some zero pre-filled memory pool to avoid costly calls to heap manager and avoid zero filling of small memory chunks. A base class for various FPC nodes should be modified to handle aloocation from the pool...

Yury Sidorov,

Although I think the time is approaching when some CPUs- IA-64 etc.- and OSes could usefully be moved into an "attic" subtree.

Agreed.

I see no way to speed up the 2.x FPC significantly only by stripping something down or whatever. The whole 2.x design is geared towards portability and maintainability, speed is only 2nd level goal (and being honest, self compilation in ~10 sec even on my >5 year old Core 2 Duo is not that bad ;)).

The only approach I see to speed it up is to kick the whole back end and generate directly some close to i386 intermediate code directly in the parser.

Yury Sidorov wrote

It is possible to seed-up compilation by allocating memory for nodes from some zero pre-filled memory pool to avoid costly calls to heap manager and avoid zero filling of small memory chunks. A base class for various FPC nodes should be modified to handle aloocation from the pool...

The heap manager itself does already pooling so I don't see much gain in doing so ...

What's the advantage in doing so? The code hangs around and does not hurt in any way.

You can cross Amiga from that list. At least the PowerPC RTL compiled all time long and recently the m68k RTL was fixed and a 'hello world' app worked.

Regards,
Sven

From: "Florian Klaempfl"

Still zero-filling a million of small memory chunks is very time consuming. It is better to pre-zerofill big pool blocks first and then assueme that memory already zero fillled in custom object init code.

Although FPC heap manager is good, but custom pool memory allocation will be much faster, since it will be very simple:

  Result:=CurPoolPtr;
  Inc(CurPoolPtr, BlockSize);
  if CurPoolPtr > MaxPoolPtr then AllocNewZeroFilledPool();

It is not needed to handle memory releases during object destruction. Since all nodes are available during whole compiling phase (parsing, code generation, etc) and released only at the end of a phase, it is possible just to release whole pool blocks at the end of a phase.

Such pooled base class for compiler nodes should inrease performanse a lot.

Yury Sidorov,

The only approach I see to speed it up is to kick the whole back end and generate directly some close to i386 intermediate code directly in the parser.

Is this close i386 intermediate code similar to IR LLVM generation ? if so, this may become a good option for a new strategy ?

Agreed. Similar experience in XMLRAD code base, and it offered huge profit rapidly and easily.

I do agree,
you should consider his position.

From: "Yury Sidorov"

Hmm, Seems to be a false alarm :(

I've made some tests just now with memory allocation and found that such pooling will not speed up the compiler too much. Only minor improvement such as 10-20% :(

Yury Sidorov,

The total time FPC spend in memory manangement is 20% iirc. So I don't see much optimization potential here.

The total time FPC spend in memory manangement is 20% iirc. So I don't see much optimization potential here.

OK.

No. It must be much closer to i386 else there is no speed advantage over the current approach.

OK, understood.

Florian Klaempfl wrote

The only approach I see to speed it up is to kick the whole back end and generate directly some close to i386 intermediate code directly in the parser.

Ewww... please not...

Regards,
Sven

No, we would be back in pre 1.0 times :) Not to mention the fact that writing such a backend probably takes one year of work.

Ops, messages crossed :) I'd expect no more. One thing might be to rewrite the whole unit handling so that symtables etc. can be read/written as one block to PPUs but doing so might be also pretty time consuming and I doubt the maintainability of such an approach.

25.12.12, 21:59, Yury Sidorov пишет:

10-20% is still much better than nothing.

Best regards,
Paul Ishenin

Yes, but you might also pay with a higher memory footprint which might even eat this speed advantage under certain circumstances.

From: "Paul Ishenin"

Sure, but I would be happy to expect at least 2X increase :) In such case I would start coding the fix right now :)

Yury Sidorov,

Believe me, if it was so easy, I would have it done for years ;)

[...] The numbers Martin names (up till 10 times slower, even without linking) are the numbers I have in mind too. IMHO denial without tests is unfair.

Maybe the parallelization could be improved? This would allow to keep the many features of fpc.

Mattias

Yes, this might be a good solution for the future but last time this topic came up, Martin still wanted to run FPC on some almost 10 years old machine so parallelization does not help. Another thing would be an fpc compiler daemon which stays in memory between compilations and keeps also ppus loaded.

If the memory footprint was less than was needed for the linkage stage would this matter?

Obviously this isn't comparing "like with like", since the compiler footprint would presumably be per-unit rather than for the entire program. But any realistic development system has to be prepared to allocate enough memory for linkage, 512Mb is realistic for building e.g. Lazarus so it's reasonable to argue that that space is also available for compilation.

Presumably over the last week or so. When I was looking at Max OS (classic) a few days ago I was usefully comparing the difference between the Amiga and Morphos RTL.

In our previous episode, Mark Morgan Lloyd said:

Doesn't that only inflate SVN backup sizes? Better have some textfile (in SVN for that matter) that explains a bit more.

----- Oorspronkelijk e-mail -----

I'm guessing that is NOT on a Windows platform? Every full build (it has been a while) I've ever done of the compiler on windows was at least a few minutes.

what I *think* is a large hog on compilation/linking is the spawning of multiple executables in the different steps as opposed to the monolithic exe(+dll) that dcc32.exe is.

lovely, but isn't that what Embarcadero is doing for their future compiler? Using the llvm toolchain, they will provide a front compiler to an intermediate "language" or interpreted code. Then they have a few optimisations for the middle layer and use the back-ends that are already available to make stuff architecturally compile to whatever back-end (ARM v?, intel x86, powerpc, motorola 68k, jvm, dalvik, .net il, compiled php, ... or any community or custom backend)

kind regards,
Dimitri Smits

Do you mean as separate process?

Mattias

Yes, it was at the beginning of last week from the one developer who had also implemented the Morphos RTL (you can take a look at the SVN log if you want).

Regards,
Sven

If you compile a project from scratch (let's keep aside the RTL, FCL and LCL) then the compiler will keep more data in memory than when the project is recompiled and e.g. no of the dependant units were changed.

Regards,
Sven

Dimitri Smits wrote

I'm guessing that is NOT on a Windows platform? Every full build (it has been a while) I've ever done of the compiler on windows was at least a few minutes.

It is windows: "self compilation". No rtl building or whatever.

No. It is exactly the counterpart. llvm intermediate language is for sure as much as possible platform independent and not "close to i386" as I wrote. If one wants to accelarate FPC by one magnitude one cannot fiddle with some abstract intermediate layer but output already from the parser something as close as possible to i386 machine code, do register allocations and optimizations on it and write the machine code to disk.

... and this costs time. And it is similiar to FPC's approach.

Yes. ppcxyz would be only a front end to this daemon process/service.

That's likely because of the slower process startup time of Windows. Also the GNU utilities we use (make, etc.) aren't the fastest on Windows either. Also command line output can slow down things dramatically (cmd.exe or the PowerShell aren't that fast).

what I *think* is a large hog on compilation/linking is the spawning of multiple executables in the different steps as opposed to the monolithic exe(+dll) that dcc32.exe is.

If you use the internal assembler and linker on Windows you get the same.

But it is very likely that they won't gain a speed advantage by that. They need to decouple their frontend from their backend and here it is very likely that they get a performance penalty if they need to go through some indirections or callbacks to get the necessary information from frontend to backend (or they need to stuff all information potentially needed by the backend into the backend). Basically the same that FPC does...

Also LLVM does not support that many backends. Currently the most important ones are x86(_64), ARM and PowerPC. What LLVM definitely does not support is m68k...

Regards,
Sven

I use this machine often so I am forced to write efficient code for MSEgui and my projects.

Another thing would be an fpc compiler daemon which stays in memory between compilations and keeps also ppus loaded.

AFAIK Delphi 7 does not need such an approach so I hope there is room for optimizations in FPC.

Martin

I've got machines which are older than that, but they've almost all got multiple processors. I can't remember the exact timing, but a Sun machine with 16x 80MHz chips would build the Linux kernel in a minute or so, so parallelisation at the make -j level can be a big win.

Obviously that doesn't help if somebody's running native builds on an obscure development board, or if a multi-CPU/core architecture's got very poor aggregate I/O performance, or if certain aspects of SMP quite simply aren't reliable (lack of cache coherence on some MIPS systems).

2012/12/25 Sven Barth :

I always wondered if adapting fpmake to produce make files with native commands like DEL or COPY instead of rm.exe and cp.exe in windows would have any performance benefit. Is this worth the trouble?

Luiz

Indeed, it probably (judging from the difficulties embarcadero apparently had to have x86-64 and arm support) sacrificed flexibility, portability (e.g. still no arm yet afaik, 64 bit support took years) and maintainability instead.

so I hope there is room for optimizations in FPC.

Better start from scratch then. FPC is not designed to be fast at any cost. Tiny C might be a good starting point.

Did you even read what we wrote? The mere fact that FPC's backend is more seperated from it's frontend than Delphi's was till XE2 is very likely one of the biggest performance problems FPC has. And this must be compensated in unusal ways like Florian described. Otherwise you'd need to go back to the way that pre-1.9 FPC worked and I don't think anyone of the FPC devs wants that. You'd basically need to rewrite the complete compiler as it is currently geared towards that seperation.

Regards,
Sven

Note: what's producing the Makefiles is fpcmake not fpmake (this are two different programs)

I'd better wait until fpmake(!) is useful enough to (nearly) completely replace the Makefiles and then we can check how to optimize stuff.

Regards,
Sven

Sven, no offending meant, but I know how the FPC RTL, FCL and compiler is programmed. There *is* room for improvement. ;-)

Martin

If you think so, please make some suggestions.

We've had a suggestion about pooling (which gains us 10-20%), and it didn't come from you.

Florian suggested a daemon kind of approach. Also not your proposal.

Till now you present only critique, claims that you can do better. But you offer no constructive advice, unless I have missed it.

Michael.

I've got machines which are older than that, but they've almost all got multiple processors. I can't remember the exact timing, but a Sun machine with 16x 80MHz chips would build the Linux kernel in a minute or so, so parallelisation at the make -j level can be a big win.

C is very different from Pascal, WRT parallel compilation. C/C++ use independent header files, so that *all* modules can be compiled in parallel. OPL instead must compile all used units, before compilation of a module can proceed. This will at least result in a long startup phase, where the basic units are compiled almost sequentially, before parallel compilation may begin.

Both languages use similar means for speeding up compilation. C compilers use precompiled headers, while Pascal compilers use the DCU/PPU files to reduce parsing efforts. This will give C an big kick, because preprocessing and parsing the source code is a lot of work. I'm not sure about the impact on Pascal, though.

DoDi

In our previous episode, luiz americo pereira camara said:

Since trunk works with fpmake (and not with fpcmake), there is no reason to invest in fpcmake. fpmake already implements this, though currently it only works for packages/

But the slowliness is not just these tools. Just compiling an unit (that uses a bunch of other units) in one compiler run is the said magnitudes slower. Without make (and thus no tools)

So in which regard?

Martin Schreiber wrote

Then that should be communicated too and I probably was wrong to invest so much time into the development of MSEide+MSEgui.

Similar with Lazarus.

At the moment a developer needs to know if a certain function (s)he uses is located in the LCL (so "ANSI" strings are to be assumed to be UTF8) or in the RTL (here "ANSI" strings might be coded in locale-depending ANSI or whatever)

-Michael

Martin Schreiber wrote

- Compile at least as fast as Delphi 7

IMHO hard to do for a portable system and not very important regarding modern hardware. I only feel the linking stage is a viable goal here, as in most cases the by far most of the already compiled units need not to be recompiled when doing a "make" after editing some source code.

-Michael

Florian Klaempfl wrote

What's the advantage in doing so? The code hangs around and does not hurt in any way.

+1

-Michael