Hi,

Does the Lazarus IDE use multi-threading for anything? Will it take advantage of a Quad Core processor?

Examples where I think it could be used (if not already)
- background parsing of units
- fpdoc lookups
- compile project in separate thread (currently the IDE is almost completely non responsive when compiling)

I'm sure there are many other areas as well?

Does anybody know if FPC takes multiple cores in consideration when compiling?

How can I see the thread-count per process / application under Linux? Under windows that's pretty easy with the Task Manager from what I remember. Ubuntu / Gnome's "System Monitor" doesn't have a thread count column. :-(

Regards,
- Graeme -

Graeme Geldenhuys wrote

Hi, Does the Lazarus IDE use multi-threading for anything? Will it take advantage of a Quad Core processor?

The IDE itself is single threaded.

Compiler runs in a separate process. On windows this process is run asynchronous, on linux there is some issue with TAsyncProcess, Mattias and Marc know more about it.

Under windows this works not perfectly though, compile a project with -va, it takes a long time, but CPU is used very little (<10%).

AFAIK, not yet.

Vincent

Don't know about the others, but compiling is done by a separate process, so there is a chance that your os runs it in a different core.

Marc

Marc Weustink wrote

Don't know about the others, but compiling is done by a separate process, so there is a chance that your os runs it in a different core.

That's not the same as running it in a separate thread though - is it?

Quoted from a Posix Thread tutorial I found on the internet. http://www.yolinux.com/TUTORIALS/LinuxTutorialPosixThreads.html

"Threads require less overhead than "forking" or spawning a new process because the system does not initialize a new system virtual memory space and environment for the process."

So wouldn't it be more efficient to create a new thread for the compiler instead of a new process?

Regards,
- Graeme -

Yes, it is more efficient. The fp ide also doesn't create a new process.

The disadvantage of using the same process is that the coupling is much tighter: No easy switching between compiler versions, and crashing the compiler crashes the IDE.

Vincent

Efficient, yes; Desirable: absolutely not.

You cannot do that except by including the compiler in the IDE. This opens up a whole other can of worms, finally resulting in a less stable IDE.

As it is now, you can switch compilers on the fly; you can't do that if the compiler is compiled-in in the IDE.

Michael.

Graeme Geldenhuys wrote

How can I see the thread-count per process / application under Linux?

To answer my own question...

   $ watch -n.1 'ps -C thunderbird-bin -L -o pid,tid,pcpu,state'

This will execute the ps command every second showing a "live" update of the threads for the 'thunderbird-bin' process. Pretty cool! :)

For more such handy command line functions have a look at: http://www.pixelbeat.org/cmdline.html#monitor

Regards,
- Graeme -

> So wouldn't it be more efficient to create a new thread for the > compiler instead of a new process? Efficient, yes;

As I pointed out on fpc-core a few days ago, I'd like to create an fpc which stays in memory and takes commands via pipes/sockets/ipc whatever to reduce process startup times.

I published and provided source for Manager and Worker Threads. It includes advanced knowledge on how to design a highly efficient system with little to no wait-states.

See: http://wiki.freepascal.org/Manager_Worker_Threads_System

Let me know what you think,

- Andrew

Let me just say that the engine I provided would be ideal for compiling - and would result in a dramatic reduction in compile time and perhaps even result in instantaneous compling of some project...

FPC would need to arrange the entire project as a hierarchical tree and that tree would be walked by the Manager Thread and the compiler would be utilized by the worker threads... I can voutch for this system... You would all be very pleased with going multi-threaded in the build process.

IMO it's not a good idea to let the IDE and the compiler share the same address space. Even if the compiler could reuse the data (files...), loaded into the IDE memory, the synchronized access can cost much runtime.

So wouldn't it be more efficient to create a new thread for the compiler instead of a new process?

Since Lazarus already includes an parser, a built-in compiler should be feasable. But I'm not sure how far the FPC developers will support a threaded version of the compiler, that would integrate neatly into Lazarus.

DoDi

Aha. I found it. The thread title was rather confusing. If it is using IPC, then it should be quite a good solution. We have the necessary things in place, the simple IPC implementation I made is more than sufficient for this...

Michael.

Specially it would employ an unit cache.

Still I'm not entirely convinced if a built-in compiler wouldn't be better. For the problems that you get, you also loose some, since the compiler is essentially fully synchronous to the calling Lazarus thread.

The multiple compiler argument (for crosscompilation) could be resolved by using packages based plugin architecture in time.

It would cause a tighter coupling between FPC and Lazarus, and require changes in project structure and releasing, for sure. But maybe it would be more desirable long term.

99% of the users strictly use "as installed" anyway, and the 1% is capable of compiling their own.

Marc Weustink wrote

Don't know about the others, but compiling is done by a separate process, so there is a chance that your os runs it in a different core.

OK, after reading all the other responses I now understand the flexibility of keeping FPC and Lazarus separate, but why then is Lazarus non-responsive while the compiler is compiling in a separate process?

Simply clicking on the editor tabs take 10-15 seconds before they react. I currently have a Intel P4 2.4Ghz with 1.5GB memory, running Ubuntu 8.04.2. Lazarus v0.9.27 from last week.

Regards,
- Graeme -

OK, after reading all the other responses I now understand the flexibility of keeping FPC and Lazarus separate, but why then is Lazarus non-responsive while the compiler is compiling in a separate process?

because IDE is reading FPC output?
and gives only a few .ProcessMessages for user interface?

thanks,
dmitry

Vincent Snijders wrote

Yes, it is more efficient. The fp ide also doesn't create a new process.

Interesting, I'll take a peak at that.

The disadvantage of using the same process is that the coupling is much tighter: No easy switching between compiler versions, and crashing the compiler crashes the IDE.

Can somebody that is more familiar with the internals of the Lazarus IDE tell me, would it be hard to implement the IDE with FPC built in, so that compiling can be done in a separate thread instead of separate process? I'll do the work of course - if it's not a major job. ;-)

This would simply be an exercise to fulfill my own curiosity. Maybe if it works well and does not require to much code changes, it could be wrapped in IFDEF's for more advanced users that want to try it for themselves.

Possible or not (in the Lazarus IDE), I'll take a look at the FP IDE to see how such an integration works. All in the name of learning multi-threading. ;-)

PS:
Does Delphi or Kylix have the compilers built-in or do they work similar to Lazarus IDE?

Regards,
- Graeme -

________________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Can somebody that is more familiar with the internals of the Lazarus IDE tell me, would it be hard to implement the IDE with FPC built in, so that compiling can be done in a separate thread instead of separate process? I'll do the work of course - if it's not a major job. ;-)

The most hardest job, is that Lazarus treats compiler as a separate command tool.
There's no TCompiler "wrapper" class, that would hide actually compiler implementation (i.e. calling for external process, or using built-in lazarus).

AFAIK, there're two components that would require change:
- IDE (building, compilation, compiler options)
- CodeTools (they're using information about units used from the compiler).

If you're able to modify the code, to use some abstract TFPCCompiler class. Then it would be possible to implement TFPCExternalCompiler and TFPCInternalCompiler, without much troubles.

Of course, it would be even more efficient, if CodeTools share/re-use some information about sources with the internal compiler, but this could be done later. Because even now, Codetools are working great! (thanks Mattias)!

thanks,
dmitry

The FP IDE does not use multi-threading. While the compiler is compiling, the IDE is stuck.

PS: Does Delphi or Kylix have the compilers built-in or do they work similar to Lazarus IDE?

As far as I know:

They have the compiler in a DLL which is loaded when the IDE starts.

Michael.

Probably simple. The compiler is started by the compile function in fpc/compiler/compiler.pas and takes simply a command line arguments. You need to set some hooks though to catch messages, see e.g. fpc/ide/fpcompile.pas (especially DoCompile starting at 868) and fpc/ide/fpintf.pas.

Hi, Does the Lazarus IDE use multi-threading for anything?

It is single threaded.

Will it take advantage of a Quad Core processor?

Not yet.

Examples where I think it could be used (if not already) - background parsing of units

Yes, this is planned. Some expensive parts can be done in worker threads. But it is no simple task. Keep in mind that the codetools do not parse in one run like the compiler and they update a lot of caches.

- fpdoc lookups

That should be easier. This already works in several separate parts executed on idle.

- compile project in separate thread (currently the IDE is almost completely non responsive when compiling)

That should be the easiest:
The compiler is an external process and TAsyncProcess is already available. The only problem is that TAsyncProcess bites the debugger under Linux. Compile LCL and IDE with -dUseAsyncProcess. Run an application. Close the IDE. The gui will close, but lazarus is still running.

I'm sure there are many other areas as well?

I'm not so sure.
For example: An expensive function is "Find in files" which could be multithreaded. But the bottleneck here is the memory system and the disk, not the cpu.

Does anybody know if FPC takes multiple cores in consideration when compiling?

AFAIK Florian did some tests years ago and the bottleneck was the memory system and the disk.

Mattias

Hooray. \O/

Can I help somehow?

Mattias

Well, first we have to decide which method to use. I even think a slightly modified fpc which can be controlled through stdin is the easiest.

That's enough. But the ProcessMessages must come regularly, which is not possible with TProcess and FPC. See my mail about TAsyncProcess.

Mattias

Ok. Don't forget proper encoding. ;)

Mattias

Does Delphi or Kylix have the compilers built-in or do they work similar to Lazarus IDE?

AFAIK the Delphi IDE comes with an built-in compiler. This may change with the announced 64-bit/cross compiler, and may be the reason for the long release delay of that new feature.

BTW, when the compiler has to read and write disk files, a separate thread will not speed up much. Then the compiler thread will be idle much time, waiting for disk I/O, and the main thread will be idle, waiting for the compiler. IMO.

DoDi

Michael Van Canneyt wrote

They have the compiler in a DLL which is loaded when the IDE starts.

Clever. I would imagine their command line compiler also uses that DLL then. Simple code reuse.

Regards,
- Graeme -

________________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Thanks for the hints Florian, I'll investigate this further. I'll probably play around with FP IDE first.

Regards,
- Graeme -

________________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Excellent.

> - fpdoc lookups That should be easier. This already works in several separate parts executed on idle.

Nice.

available. The only problem is that TAsyncProcess bites the debugger under Linux. Compile LCL and IDE with -dUseAsyncProcess. Run an application. Close the IDE. The gui will close, but lazarus is still running.

I probably run my applications 1% of the times via the IDE. All other uses and testing is by running them from a command line. I few years ago when debugging via the IDE really sucked, I wrote various logging methods which I got so used to and always works. Logging gets done via a separate thread and I can log to various outputs. cached logging to file, cached logging to GUI or non-cached logging to console (stdout).

I'll recompile tomorrow with-dUseAsyncProcess and see how it goes.

For example: An expensive function is "Find in files" which could be multithreaded. But the bottleneck here is the memory system and the disk, not the cpu.

That is an excellent candidate. We did some testing with a multi-threaded application that does CRC calculation on files, recursively through directories. It was one of the multi-threading entries to a Borland Multi-Threading competition held some years ago. We adapted the program for our needs and by using 2-4 threads on a multi-core system improved overall throughput a lot. The app did not work so well when the files were on CD-ROM though. :-) A major disk bottleneck, but hard drives or raid system is much faster.

AFAIK Florian did some tests years ago and the bottleneck was the memory system and the disk.

I believe GCC can use threads for parallel compiling, so there must be some advantage. With faster hard drives or SSD drives it should make a difference - I guess. But yes, I understand your point regarding disk IO being slower than in-memory computations.

Regards,
- Graeme -

________________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

BTW, when the compiler has to read and write disk files, a separate thread will not speed up much. Then the compiler thread will be idle much time, waiting for disk I/O, and the main thread will be idle, waiting for the compiler. IMO.

That's true. The general problem is not that the IDE is not responsive when compiling/linking but that compiling/linking takes so long. If it would be as fast as in Delphi noone would complain.

And having compiler and linker in separate (asynchronous) threads would be the same as starting them as a new process in asynchronous mode. It would open a can of worms because of synchronising problems.

I would not mind to wait until compiling has finished if it would only be faster.

J?rgen Hestermann wrote

That's true. The general problem is not that the IDE is not responsive when compiling/linking but that compiling/linking takes so long.

FPC is a fast compiler and more speed is obviously also nice. The compiling is not the issue to me, it's the unresponsive IDE that gets me. I would like it to be similir to other users experiences. For example, you can print a large document and continue browsing your text or changing some setting in the menus etc... It doesn't block you from continuing. The Lazarus IDE does.

I compile very often - probably once every minute or two. I also practise Test-Drive Development, which makes sure you compile often. I would like to start a full build (which could take a while for large projects) and continue writing fpdoc documentation or start a new unit test (obviously without saving before the compile is complete) etc...

It's called "multi-tasking". Ever heard of it? ;-)

Regards,
- Graeme -

It takes so long, because compiler and linker have to read all ppus on every run. A persistent compiler and linker could keep some things.

And having compiler and linker in separate (asynchronous) threads would be the same as starting them as a new process in asynchronous mode. It would open a can of worms because of synchronising problems.

They only share files. The synchronising is easy.

The are good reasons to not put the compiler into the same process: -a process can be aborted/killed easily
-various compiler versions
-compiler crashes do not effect the IDE

I would not mind to wait until compiling has finished if it would only be faster.

Mattias

Good enough for me NOT to do it. If the compiler was internal to the IDE, I could not easily compile for 32/64 bit or a different version, without including 2+ compilers in the IDE. As it is now, all I must do is set the compiler to use.

Florian's option seems a better approach, it would preserve this possibility.

A sort of 'daemonized fpc'. You could use this as a back-end for a website: pascal server pages instead of java server pages :-)

Michael.

But an async. process doesn't work for lazarus, right? So controlling through stdin/stdout is not possible?

Mattias Gaertner wrote

-compiler crashes do not effect the IDE

I like Florian's idea too. As for your statement above, how often does this really happen? In the last 4-5 years that I have been using FPC, the compiler has not once "crashed" on me. Yes it might report that there is a compiler error in my project, but I do not consider that a crash.

Regards,
- Graeme -

It's one thing if the compiler compiles once and exits.

It's a whole other matter if the compiler compiles multiple times, and must properly reset itself prior to recompiling.

Michael.

Try

ide\outputfilter.pas around line 295

if (Applicationnil) and (abs(LastProcessMessages-Now)>((1/86400)/3))

make it
if (Applicationnil) and
(abs(LastProcessMessages-Now)>((1/86400)/10))
and maybe add an "application.idle(flase);" too

Martin

It happens from time to time. For example: http://bugs.freepascal.org/view.php?id=13323

It may happen to me more than for you, because I may use the trunk compiler more often, so that these things get noticed before it gets in the release and more people are hurt.

Vincent

Martin wrote

make it if (Applicationnil) and (abs(LastProcessMessages-Now)>((1/86400)/10))

This does not make much difference on my system. :-(

Regards,
- Graeme -

Graeme Geldenhuys wrote

- compile project in separate thread (currently the IDE is almost completely non responsive when compiling)

The other downside of how the IDE currently works is the following... Even though the IDE has the option to abort a compile, the IDE is so non-responsive that even if you press Escape to abort the compile, by the time the IDE tries to process that message, the compile or linking in probably done.

Now at least I understand why aborting a compile/build never worked for me. :)

Regards,
- Graeme -

Zitat von Graeme Geldenhuys :

The compiler is started on every compile which is the cleanest start you can get. A threaded compiler must have a restart, which is less clean. There will be more crashes and bugs.
What happens if the restart fails?

Mattias

Vincent Snijders wrote

It may happen to me more than for you, because I may use the trunk compiler more often, so that these things get noticed before it gets in the release and more people are hurt.

That would be it. I stopped using the trunk compiler for our day-to-day development. Plus what Michael and Mattias mentioned makes perfect sense too. Thanks for the info guys.

Regards,
- Graeme -

For the debugger, we have reset debugger.
For the compiler, we would add a reset compiler option.

Vincent

I guess you need the TAsyncProcess support. This part works OK on windows.

Vincent

That's easy: it stops and restarts gdb. an external process.

For the compiler, we would add a reset compiler option.

If the compiler is internal, what would 'reset compiler' do ? If the compiler people can't guarantee a proper reset, then neither can the IDE...

With the proposal of Florian, the 'reset compiler' is possible, just as with GDB...

Michael.

Hm I tested on windows only. Try adding the 2 lines I inserted with comments (I have not tested this!!)

Just seeing
      if (TheAsyncProcess=nil) and (fProcess.Outputnil) then begin
        // using a blocking TProcess
//        Count := 0;
//        if fProcess.Output.NumBytesAvailable > 0 then
          Count:=fProcess.Output.Read(Buf[1],length(Buf));
        if Count=0 then begin
          // no output on blocking means, process has ended
          break;
        end;
      end;

One more remark, if it works, it will allow normal editing,etc.

But you can not do:
- search and replace
- find in files
...

Because those themself are long running, and while being inside the compile's processMessages, the compile will not read the output, and that may block compiling.

This is because all this stuff is not event-driven, so it cannnot be mixed. It would work in a single thread and all this, but it must be fully event-driven.

Sorry, I was reading it as part of an usage question about Florians proposal (external in memory compiler), not about an internal compiler.

Vincent

sorry tested now.
doesn't work, outside windows world...

So even a blocking pipe should have some kind of CanRead that returns immediately?

You can not really test it on windows, because on windows TAsyncProcess is used.

Vincent

Zitat von Florian Klaempfl :

TAsyncProcess needs to be fixed anyway.
Maybe Graeme will find the solution.

Mattias

Zitat von Martin :

Added by default.

This only have an effect on windows, where TAsyncProcess is used.

Mattias

Zitat von Graeme Geldenhuys :

Do you have time to debug TAsyncProcess?

Mattias

Zitat von Vincent Snijders :

I meant, when the compiler is in another thread and the restart fails, what should happen? You can only restart the IDE.

Mattias

Just out of curiosity, what are we trying to solve?

How much benefit would a build in or preloaded fpc inside Lazarus be? Most Operating System cache disk access. So once you used, it is in memory, never mind if that memory belongs to lazarus, or the OS.

Of course another thread would solve issues with unresponsiveness. But that could be done by having another thread, that runs the external process. Or if the AsyncProcess worked, it could be done, by using this
+ moving to an event-driven model.

Or am I missing some crucial point?

Martin

Martin wrote

Just out of curiosity, what are we trying to solve?

<snip>

Or am I missing some crucial point?

3 idle cores on a quad core CPU. Why pay for a quad core, if Lazarus + FPC use only one? I am still waiting during a compile and those 3 other cores are idling. Let them to part of the work to get the job done faster. ;-)

Vincent

P.S. I couldn't resist not helping in this discussion. It is nice to have an active mailing list not about unicode.

+10 :-) My new computer is on order and will arrive next week. Quad Core with 4GB memory. A huge upgrade from my current P4 2.4Ghz with 1GB memory.

I would like all my cpu cores to work as hard as they can! So I can code faster. ;-)

P.S. I couldn't resist not helping in this discussion. It is nice to have an active mailing list not about unicode.

Yes, it is a pleasure not having to do the SVN vs Git thing either. :) Did I mention Git uses multiple threads for log, diff and delta packing if a multi-core cpu is detected. [...Graeme runs and hides behind his desk...]

Regards,
- Graeme -

Zitat von Martin :

Then it would be non blocking.

Mattias

Mattias G?rtner wrote

Do you have time to debug TAsyncProcess?

No harm in trying...

Regards,
- Graeme -

I guess lazarus uses one, fpc uses one and the os uses one, so that leaves one for me to answer mails or forummessages or browse or chat. No idle cores here ;-)

Marc

Zitat von Martin :

Well, the crucial points speak all against an internal compiler as thread. But an internal compiler can have some benefits:
Access to the compiler structures may allow to find out, why a "unit was not found". But I doubt that.
Access to the compiler structures may allow to do a find declaration in ppu files. But reading ppu files is not a big problem. I already started that in the codetools. The problem is to make the information usable to the existing tools (identifier completion). And this is a lot of work.

My conclusion:
Let's use an external compiler as Florian proposed and one day the ppus don't need to be parsed on every compile. This should reduce the compile time dramatically.

Mattias

and maybe add an "application.idle(flase);" too

Lazalus nevel makes things flase!

SCNL
DoDi

Same for me, FPC fixes_2_2 crashes about 20 times a day with EList or EAcessViolation if I forget to use -B to compile some modified kernel units which are used in many other units.

Martin

So does MSEide with the compile and the search in files function. The checking of the compiler output and the file search run in separate threads, so MSEide is fully functional while compiling or searching.

Martin

Last time I compared FPC and Delphi 7 Delphi was ten times faster. Yes, more speed would be nice.

Martin

Florian Klaempfl wrote

Probably simple. The compiler is started by the compile function in fpc/compiler/compiler.pas and takes simply a command line arguments.

Florian, you were 100% correct, it is VERY easy! I looked at the FP IDE, and within 5 minutes I had my own fpGUI based "ide". :-) Actually just a GUI application with one edit field that takes the compiler parameters and then a big COMPILE button that starts the compile process in a separate thread.

Currently all output goes to stdout, but the compiling works perfectly. Next step is to place that thread in the Lazarus IDE. :)

Regards,
- Graeme -

Yes, but Delphi 7 has 10 times less platforms to support, and 10 times more people for it's development, people working full time on nothing else than Delphi.

So where are the compiler geniuses, and the super patches to improve FPC ?

Michael.

Martin Schreiber wrote

checking of the compiler output and the file search run in separate threads, so MSEide is fully functional while compiling or searching.

Nice! :-)
From the MSEide config dialog, I gather MSEide also launches the FPC compiler in a separate process and not built into the MSEide?

Regards,
- Graeme -

It probably requires to rewrite FPC completely :) FPC chooses always maintainability and portability over speed and compared with gcc or even VS, FPC is still fast ;)

True, but from the point of using FPC and the point of faster compilation => this needs a multithreaded fpc.

Running a single threaded fpc in it's own thread (never mind if compiled into lazarus, or loaded as process from a thread) will not do much (except for one core could do the updates of the message window.... A multi-threade fpc can still be used in a forked process.

As for thinks happening inside lazarus (codetools, search in files,....) different story

Martin

Well, currently this does not help much either because FPC is mainly bound to memory throughput if the disk cache is already hot. Resistent FPC keeping units loaded would be the largest gain for projects like MSE or Lazarus.

Graeme Geldenhuys wrote

From the MSEide config dialog, I gather MSEide also launches the FPC compiler in a separate process and not built into the MSEide?

Yes. It can also call gcc and parse the error messages. Also possible to call make or a script.

Martin

True, but the distance will be less with every FPC release if compiling speed is not observed I fear.
The possibility of Delphi to compile big projects in some seconds and small projects in less than a second on simple PC's has already gone in FPC. :-(

Martin

Really? And why is Delphi 7 so much faster than FPC under same circumstances? I used dcc32 and not the Delphi IDE for comparison. Again, you are right, FPC is fast but it should not become much slower or it will loose a big benefit over gcc or other c++ compilers.

Martin

2009/8/14 Martin Schreiber :

The problem is that you really compare apples to pears. And, as Michael explained, resources (of the human kind) are limited. hopefully this will change when Delphi finally crumbles.

Henry

Martin Schreiber wrote

> From the MSEide config dialog, I gather MSEide also launches the > FPC compiler in a separate process and not built into the MSEide? > Yes. It can also call gcc and parse the error messages. Also possible

So how did you resolve the async process issue that Lazarus is experiencing under Linux? Or does running the compiler process (TProcess style) via a separate thread bypass the issue of TAsyncProcess that Lazarus IDE is using. That means MSEide doesn't need a TAsyncProcess type component?

If so... Mattias, could Lazarus IDE be modified to run the compiler TProcess in a separate thread. And then another thread which filters the compiler output to the Messages window. Or is that a huge task?

Regards,
- Graeme -

I don't know TAsyncProcess. What does it compared to TProcess? What are the Problems? I use the MSEgui version of exec() and the tpipereader class to run the compiler and read the messages.

Martin

Martin Schreiber wrote

So does MSEide with the compile and the search in files function. The checking of the compiler output and the file search run in separate

I just tried with a oldish MSEide 2.1 (unstable) and I monitored the thread usage for the mseide process. You seem to use a few threads for things. One for the main process, one for the editor (though I don't know what it does) and 'search in files' also uses a separate thread.

Very nice to see that even with a long running search, the main ide still responded to all user interactions as normal. I could even compile a project while doing a search. All tested under Linux. Don't you want to move to Lazarus IDE development. ;-)

Regards,
- Graeme -

Zitat von Martin Schreiber :

Huh?

I have some small programs with one thousand lines in the main source and using units with about 30k loc. Recompile and linking the program takes less than a second.
The main work is spent in loading the ppus and of course the gnu slow linker. I'm confident that a compiler kept in memory reusing the parsed ppus should be able to compile even medium sized LCL projects in less than a second.
Of course linking and writing a 20mb exe with debugging information to disk will take some time.

Mattias

Zitat von Graeme Geldenhuys :

No. Just write a ThreadedProcess. That would work pretty much like TAsynProcess.
And I guess it will suffer the same bug.

Mattias

ups, I forgot

uses Typos;
----
unit Typos;
const
  flase = false;
type
  Integel = Integer;
  Integre = Integer;

very useful...

SCNR ;->

Isn't there a callback to report progres (lines etc) ?

I don't know what you mean by "ThreadedProcess" unless you meant yes, but wanted to indicate a different form of implementation?

IMHO, if the process running fpc is started from a different thread, then the IDE can be kept as functional, as it can with TAsyncProcess. In fact if the TProcess is in started from a thread of it's own, it does not want to be Async, it wants to be blocking, so you don't need to put a lot of sleeps into your code.

As separate thread *can* also solve one more issue.

Currently even the AsyncProcess is done in a loop. It does use the OnDataRead event, but only to fill a cache, which is then used in a loop. It should not be in a should be purely event driven. Then it would continue to run, even if a "search in files" is started while compiling....

If you create a class TThreaded Process (just to simulate an AsyncProcess within a (hidden) Thread), then you still must transform the current loop into an event driven model.

However this 2 issues can be addressed on their own. even using a blocking thread, you can go (almost) event drive. Put the read on a timer (every 10 milliseconds) or OnIdle, The read will block, but after each read, it will return to the event loop => events will be processed normally (instead of ProcessMessages), and then the next read happens on timer, or idle

In fact if you make everything eventdriven, then you could introduce some method to mark events as thread-save, and the event-loop could automatically deploy threads, if a lot of events occur => that would be cool.

Martin

A dll. Not statically built in. Afaik cmdline and IDE use the same DLL

Yes, but basic units could be cached. Also the cache of dirsearches would persist.

Moreover you win process startup time (which can be quite significant on windows)

On my Core Duo , FPC itself (230k lines) builds in 5.5s, so this is ok imo ;)

Disadvantages:
- compiler versions might not match (entirely)
- Multiple processes that can only be made sequential over I/O. Potential for race conditions.
- (some) platforms might not support multiple processes at all, or miss the relevant 100% primitives. (also e.g. faulty pipe implementations)

FPC simply needs memory bandwidth due to design?

You probably see the thread which reads messages from gdb. Syntax highlighting is done in onidle in packages of 100 lines.

Martin

There is, but the user cannot do anything except look at the progress indicator.

Michael.

Mattias G?rtner wrote

I have some small programs with one thousand lines in the main source and using units with about 30k loc. Recompile and linking the program takes less than a second.

With small I mean 100'000 lines. ;-)

Martin

Good enough to not do it before a good packages system is developed so it can be loaded dynamically. But that is something else then to chuck it all together.

(or simply not accelerate cross-compiling with the built-in till packages emerge)

MSEide (330k lines) needs 25s on Athlon 64 X2. Commandline: ppc386 -B
apps/ide/mseide.pas -Fulib/common/* -Fulib/common/kernel/i386-linux -Fi/lib/common/kernel

25 s is too much to wait and too little to sleep. ;-)

Martin

2009/8/14 Martin Schreiber :

How long does fpc -sh take?

Henry

21.8 s.

Martin

And?

- Multiple processes that can only be made sequential over I/O. Potential for race conditions.

Can you give an example?

- (some) platforms might not support multiple processes at all, or miss the relevant 100% primitives. (also e.g. faulty pipe implementations)

Are we still talking about the IDE?

Mattias

Martin Schreiber wrote

25 s is too much to wait and too little to sleep. ;-)

I so agree with that! :-)

Same command line params as yours - on a Intel P4 2.4Ghz. 322187 lines compiled, 33.6 sec

And with the -sh parameter.
322187 lines compiled, 28.7 sec

Martin, how does your results compare when you use Delphi's bcc32 compiler?

Regards,
- Graeme -

I have looked a bit more why outputfilter is so slow at parsing a lot of compiler output (e.g. compiling a simple LCL app with -va). The pipe buffer is rather small, NumBytesAvailable is not bigger than 1280 bytes. So OnAsyncReadData reads only 1280 bytes at a time. OnAsyncReadData is only called after an application.handlemessage. So on line 319 of outputfilter.pas, the buffer is most times empty, but NumBytesAvailable = 1280, and we are sleeping until Application.HandleMessage is called.

I had some success with the following patch. It does a sleep(0) to give the compiler
the chance to write to the pipe again and tries to read it again.
--- ide/outputfilter.pas
+++ ide/outputfilter.pas
@@ -1206,9 +1206,12 @@ var
    Count: LongWord;
  begin
    if fProcess=nil then exit;
-  Count:=TAsyncProcess(fProcess).NumBytesAvailable;
-  if Count>0 then
-    FAsyncOutput.Push(TStream(TAsyncProcess(fProcess).Output),Count);
+  repeat
+    Count:=TAsyncProcess(fProcess).NumBytesAvailable;
+    if Count>0 then
+      FAsyncOutput.Push(TStream(TAsyncProcess(fProcess).Output),Count);
+    sleep(0);
+  until Count=0;
  end;
  function TOutputFilter.CreateScanners(ScannerOptions: TStrings): boolean;

It is not clear to me why we can't check NumBytesAvailable (available since fpc 2.2.2 or earlier) on unix and windows and just forget about TAsyncProcess for compiling?

Vincent

Makes sense. Why the sleep(0)?
Maybe the buffersize can also be increased?

NumButesAvailable seems to block on TProcess.
From my understanding on a none blocking socket/pipe you can call all the operations like read/write/peek and they will always return immediately. On a blocking socket they will always wait until data is there. On a blocking socket one has to call select (fpselect), which should indicate the availabilty of data without blocking.

But Imho we should avoid sitting in a loop and sleeping.

That is what TASyncProcess.OnReadData is there fore. Read and handle all data in the event, exit the event and wait for the event to fire again. Then you can even to find in files in parallel with compiling (it wil be slow, since both access the disk) and it will work.

Martin

If I understand it correctly the problem is that TAsyncProcess, currently only reads data, when the event is triggered, which is in a ProcessMessages. Ouch.

Either TAsyncProcess.OnReadData is transferred into the main loop, in a similar way as for the normal TProcess.
I have not tested this, but it should work, as NumBytesAvailable seems to be none blocking.
=> This could simplify the overall loop as the handling of the 2 projects would be more or less the same.

Alternatively, before we sleep(3), we could check for data, and process it.
        end else begin
          // no new input, but process still running
          if TheAsyncProcess.NumBytesAvailable > 0 then
            Application.ProcessMessages
          else
            Sleep(30);
        end;

These numbers tell you exactly nothing. At best they give you an indication of how complex your project is.

I have Delphi compile times of over 2 minutes on projects that have maybe 5.000 lines. It takes this long, because there are a lot of dependencies on external packages and whatnot, and Delphi takes a long time loading the packages and correctly resolving the symbols.

Just for fun, I once managed to have delphi compiling 5 minutes (!) on a single form project. (granted, I did put up some messy dependencies :))

The only way to measure the compiler speed is to create 1 very large unit (say 500.000 lines) and let the compilers compile that single unit.

Michael.

The loop in outputfilter can be simplified quite a bit

All the extra handling of the asyncprocess can be reduced to the following lines (the event handlers can be removed):

        if fProcess is TAsyncProcess then begin
          Count:=TAsyncProcess(fProcess).NumBytesAvailable;
          if Count > 0 then
            Count:=fProcess.Output.Read(Buf[1], Min(Count, length(Buf)));
        end
        else
          // using a blocking TProcess
          Count:=fProcess.Output.Read(Buf[1], length(Buf));
        if (Count=0) and (not fProcess.Running) then
          break
        else if count = 0 then
          sleep(30);

Works on windows / shouldn't change anything on *nix

Michael Van Canneyt wrote

These numbers tell you exactly nothing. At best they give you an indication of how complex your project is.

In the mentioned test FPC compiled the 185'000 lines in 17.9s, Delphi 7 in 1.28s.

http://www.mail-archive.com/fpc-devel%40lists.freepascal.org/msg08029.html

Martin

http://www.mail-archive.com/fpc-devel%40lists.freepascal.org/msg08029.html

Martin

Martin Schreiber wrote

> > Martin, how does your results compare when you use Delphi's bcc32 compiler? > http://www.mail-archive.com/fpc-devel%40lists.freepascal.org/msg08029.html

Wow, that is a sizeable difference. I did not realize the Delphi compiler was that fast.

Regards,
- Graeme -

________________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Requiring the compiler used to compile lazarus, and the one to

> - Multiple processes that can only be made sequential over I/O. > Potential for race conditions. Can you give an example?

Because you have wire time and context switches, and a binary that can generate evens, you have nearly an infinitely big amount of possibilities of the state machine that manages events from lazarus (e.g. set breakpoint) and gdb (break point met), risking an undefined state.

In an library, when you synchronously call gdb, gdb can't callback you.

About both IDEs yes, but I agree this is the weakest of the three, and a bit outdated. (Dos, win95 etc)

Probably, but that is something that can be resolved, not a fundamental problem.