Hi,

[Just as well the Lazarus IDE didn't get the thread update mentioned a few days ago... where a thread was doing work, and the main thread was waiting for that thread to finish... it wouldn't have worked under Linux it seems.]

I just stumbled across this issue in fpGUI, and I see MSEgui and LCL-GTK2 has the
exact same problem. See attached project. I haven't tried LCL-QT, but I presume that same issue will persist.

A simple program (just for illustrative purposes), where a thread updates the progress bar (the thread could be doing any long running process). Click the button, it creates a thread and
starts it off. If then waits for the thread to finish, then displays that it has finished in the GUI, and free's the thread manually.

If you use the MyThread.WaitFor to wait for the thread to finish, then the application is instantly frozen (even the thread), nothing gets updated and you have to kill the process. For some reason WaitFor blocks the main thread's event loop. It seems this issue is only under FPC apps and under X11 (no matter the GUI toolkit), it works fine under Windows.

The work around, is to use a boolean variable in the thread class, and a while loop which checks if the boolean variable is true, if not it calls Application.ProcessMessages (to keep the app alive).

Like I said, this is a stupid example, but it illustrates that WaitFor cannot be used under X11 (for our GUI toolkits). Any ideas as to why WaitFor blocks the main event loop? Is it a Linux thing, X11 thing, or just that we made a mistake somewhere?

So what is wrong? Is it the TThread.WaitFor implementation that is flawed? Is it X11 that is flawed, or just GUI toolkit developers that don't know how to code (thought I doubt it's the latter). :)

As an experiment (I'm all out of ideas now), I dived into the TThread.WaitFor implemenation, as saw that it calls WaitForThreadTerminate() with no timeout value. I though, well, let me try calling WaitForThreadTerminate() directly with say a 2 second timeout. No luck there. The instant you call that, the application is frozen. It never times out either, so the FPC+Unix WaitForThreadTerminate() functions seems broken too. :-/

I guess I need to file a FPC bug report about WaitForThreadTerminate() being broken under Linux, but I wanted others to confirm this first, before I do.

Did anybody know about this? Anybody know how we can get TThread.WaitFor to actually work under Linux/Unix?

Hi!

I got the same when doing some classes for my project. My solution is add a TEvent in my thread that will be signaled in the last line of the thread loop.

On my application instead use TThread.WaitFor, I use TThread.MyWaitFor that internals do:

While TEvent.WaitFor(1)wrSignaled do
   Application.ProcessMessages;

It solves my problem with threads finalization on windows and linux.

Fabio

2010/10/8 Graeme Geldenhuys

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20101008/7ad2cfe5/attachment-0001.html>

I forget: It works only with FPC 2.4, because TEvent.WaitFor works correctly with timeouts.

Fabio

2010/10/8

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20101008/60ff438b/attachment.html>

Graeme Geldenhuys wrote

... Did anybody know about this? Anybody know how we can get TThread.WaitFor to actually work under Linux/Unix?

It is the same old song: absense of centralized event handling in Linux. In Windows, the main thread waits with MsgWaitForMultipleObjects, so it remains capable of processing messages, no matter where they come from. In Linux, there's no centralized event source, so one always has to specify particular source(s) (For GUI apps, it will be X event socket). In well-known toolkits, this is handled by "Main loop" (e.g. GMainLoop in glib).
So to get it working, one needs either to implement something similar to GMainLoop in RTL, or implement some mechanism that can hook into an external "main loop" which, in case of GUI application, will be provided by GUI toolkit.

Regards,
Sergei

This is normal, since WaitFor() is a blocking call.

You should never use WaitFor() in a GUI app, instead, you should set a callback on the thread Terminate, and run the message queue while you're waiting. It's not different from showing a modal form with ShowModal().

Procedure MyForm.OnThreadDone(Sender : TObject);
begin
   FThreadDone:=true;
end;
Procedure MyForm.DoSomethingInThread;
begin
   T:=TMythread.Create(True);
   T.OnTerminate:=OnThreadDone;
   DoSomethingElse;
   While not FThreadDone do
     begin
     CheckSynchronize(10); // normally called in ProcessMessages.
     Application.ProcessMessages;
     end;
   // Now thread is finished.
end;

We could alleviate this by allowing to hook into WaitForThreadTerminate and with this provide 'run message queue' callbacks.

Michael.

Subject: Re: [Lazarus] Threads in Lazarus code base Date: Wed, 15 Sep 2010 13:18:18 +0200

On 15 September 2010 13:00, Juha Manninen (gmail)

The main trend in programming now is to encourage threading, not to ban it. I am little puzzled...

+1
As I said, threading is an "advanced" feature, and clearly here are many developers that are afraid of something they don't understand, so simply think banning the feature is the solution. Simply learn multi-threading, get some experience with it, then using threads are not a problem. It's the same deal as using Pointers. IF YOU DON'T KNOW WHAT YOU ARE DOING, YOU ARE F**KED.

Just a note:
Since this monday the IDE scans the FPC sources in a separate thread, with a progress bar.

Mattias

Al 08/10/10 15:17, En/na Graeme Geldenhuys ha escrit:

I'm surprised it works under windows: using synchronize and waitfor has always been a no-no.

Bye

Just a note: Since this monday the IDE scans the FPC sources in a separate thread, with a progress bar.

It is just that Graeme was bitten by threading and he was the one saying that you should know what you do, which made me smile. In general threads have their purpose (for decoding video and audio streams I use threads myself). I am not against using threads in Lazarus if they are the best option for a given case.

R.

No need to be surprised.

Windows RTL uses MsgWaitForMultipleObjects to wait on the thread's handle and the handle of the Synchronize event (that event is set if a thread calls Synchronize). If the Synchronize event becomes signaled it calls CheckSynchronize (thus no deadlock in this scenario) and continues waiting. Also the use of MSGWaitFor* allows that window messages are still processed allow the (main) thread is waiting.

Regards,
Sven

It is the same old song: absense of centralized event handling in Linux. In Windows, the main thread waits with MsgWaitForMultipleObjects, so it remains capable of processing messages, no matter where they come from.

In the current trunk version Sending a Message to the main thread and handling it with "procedure...message" does not work any more in Linux. I'm sure it did work with some previous version. With that thread generated events could be used in a GUI program.

I thought, TThread.Waitfor would (should) be implemented using this mechanism (e.g doing a loop with "Application.PreocessMessages"). With that it would not be blocking other event handlers. But Maybe I'm wrong and TThread.Waitfor in fact should block the main thread event handlers...

-Michael

Reimar Grabowski wrote

As I said, threading is an "advanced" feature, and clearly here are many developers that are afraid of something they don't understand, so simply think banning the feature is the solution.

In fact IMHO "Object Oriented" "Event Driven" programming such as provided by Delphi/FPC is all about avoiding the complexity of threads and still being able to handle asynchronous events (such as GUI events) by having the RTL transform ()and the OS them into "run-to-completion" Language-Events by means of queuing them in an event queue.

Of course the leads to restriction with latency and (when running on a multi-Core system) performance.

Simply learn multi-threading, get some experience with it, then using threads are not a problem. It's the same deal as using Pointers. IF YOU DON'T KNOW WHAT YOU ARE DOING, YOU ARE F**KED.

Yep ! The language and RTL should decently support threading for those that need it and are willing to deal with the implications and complexity. This includes providing means to notify the main thread from worker threads as well in a synchronous way (such a s TThread.Synchronize) as in an asynchronous way (such as TThread.Queue or (better) "procedure...message" ore (even better) a decent event queue such as provided by MSE. This of course automatically includes TThread.Waitfor.

Of course in this event queue GUI events need to be automatically merged in case a GUI Widget-Type is used.

-Michael

Reimar Grabowski wrote

In general threads have their purpose (for decoding video and audio streams I use threads myself). I am not against using threads in Lazarus if they are the best option for a given case.

IMHO nowadays threads serve two purposes:

1) reduce latency of certain aspects of a program: 1a) with a GUI program improving the responsiveness of the GUI by reducing the latency of the main thread by doing long winding calculations in a worker thread
1b) with embedded or server-type application preventing one action (the Main thread and other worker threads) from blocking important actions by doing these important actions in their own threads 1c) handling blocking devices (such as serial ports) doing this in the main thread would destroy the GUI latency completely.

2) improving performance by doing long winding parallelizable calculations in multiple threads to allow for taking advantage of multiple CPUs

Luca Olivetti wrote

I'm surprised it works under windows: using synchronize and waitfor has always been a no-no.

Seemingly Synchronize is not implemented decently in Lazarus (while it seems to work perfectly as documented with Delphi).

I suppose WaitFor is done with Messages and this seems to be completely broken for Linux in the latest svn version while it did work in former versions and seems still to be working with the Windows Widget Type.

-Michael

It would be good if you could provide a patch, instead of continuously restating your desire of this feature. You manage to bring it up quite often. That means you think about it often, so why not try an implementation while you're at it ?

If it wasn't implemented till now, it's because none of the core developers sees the need for it, and by the looks of it, this will not change in the near future. You can change this by providing a patch. It does not need to be perfect, but it should provide a working - cross-platform - start of an implementation. Then we can see where we can fit it in. You can start by creating an descendent of TCustomApplication or so: TCustomEventApplication.

Michael.

Michael Van Canneyt wrote

It would be good if you could provide a patch, instead of continuously restating your desire of this feature.

I'm not stating my desire of this (set of) features). I'm just confirming that others stating this desire are not alone.

At the moment I am not actively working with Lazarus at all (just evaluating chances for future projects). So actively working on improving it would just be a charity project and unfortunately I don't have the time for this right now. (both facts can change any day in the future)

In fact, some months ago (when I had more spare time) I did invest some two weeks of work on implementing MSE's event queue mechanism in Lazarus, but I found the correct way to implement this would be to do it in the FPC RTL (to allow for an event queue independent of the GUI) and to modify the Lazarus LCL to implement it's event queue based on this (merging the GUI events similar to how it right now it's done in all Widget Types but "Windows"). But i feel providing interdepending patches for both FPC and Lazarus is fay beyond my political power.

You manage to bring it up quite often. That means you think about it often, so why not try an implementation while you're at it ?

In fact I did (see above). There are some discussions on the here in the back log and in the MSE discussion group.

In fact 90 % of the discussion threads here (and in some more groups) brining up this topic have not been started by myself. So I feel that there seems to be a decent "public" request.

You can change this by providing a patch.

Unfortunately not (see above)

It does not need to be perfect, but it should provide a working - cross-platform - start of an implementation.

Of course the cross-Platform and with that GUI independence is the core of a decent patch.

Then we can see where we can fit it in. You can start by creating an descendent of TCustomApplication or so: TCustomEventApplication.

Thanks for remembering my dream of providing an event queue even for worker Threads :) AFAIK, MSE in fact can do this. As I did not see any need (but that of my brain wanting perfect structuring) on that, this would be just a nice add-on, that supposedly would not need much additional effort by doing a thread-save implementation of TCustomEventApplication. :).

But here again, the FPC implementation is involved as "Procedure ... Message" would need to be implemented ion top of the event queue and in fact the Windowism of the Message type should be dropped in favor of a decent Object based Event Type (such as provided by MSE).

-Michael

Reimar Grabowski wrote

Oh sweet irony. Emphasis mine. Have a nice day

:-) So true! My mistake was simply that I applied the "Windowism" (behaviour of Windows) to other platforms. This is bad practice, but I guess something that occurs often because my Windows programming experience (10+ years) is still far more than Linux (4+ years). FPC makes it so easy to switch between platforms, but I must still keep in mind that there ARE difference between platforms. A lesson to all! :)

I think Thread.WaitFor should trigger an event each time in it's loop. Thus a developer could hook into that event and call things like Application.ProcessMessages or whatever else is required to keep the application from deadlock.

There is no 'political power' needed.

1. Make sure you make something that can be integrated with the FPC rtl/FCL.

2. Once that is done and it is performing as expected, it should not be hard to convince the Lazarus people to use it.

If of course you expect to achieve a total shift in paradigm, then
you'll be disappointed, because Delphi compatibility is paramount.

I beg to differ. From my perspective, people bring up issues, you bring up the event mechanism in one of the replies.

At least, it would be a start. Now there is nothing, so we cannot even verify that your mechanism is indeed a superior mechanism, as you claim.

While I agree that an object is a clean approach, for most purposes, an object is simply unnecessary overhead, because it requires additional heap storage.

Michael.

Michael Van Canneyt wrote

1. Make sure you make something that can be integrated with the FPC rtl/FCL.

This would supposedly be doable be based on MSE's code. But it would need to include quite a lot of MSE code lines, as the Event queue type is nicely structured and derived form more basic queue types that would need to be used.

it would not be really easy to test it as it does not synchronize with GUI events when used with Lazarus.

2. Once that is done and it is performing as expected, it should not be hard to convince the Lazarus people to use it.

I doubt that, as it would need to replace the basic event management in all Widget sets, They even two manage complete different KDE Widget Type code bases because they don't want to take the risk of breaking old code when having it use using common functions with the new KDE Widget Type. Moreover all other Widget Types (GNU, Mac, fpGUI) have their own implementation of an Event queue in Pascal code, and the Windows Widget Type uses the OS'es Message queue. Replacing all of them with the methods provided by the RTL would be necessary (and GUI events would need to be merged in the way all but Windows Widget Set code does)

If of course you expect to achieve a total shift in paradigm, then you'll be disappointed, because Delphi compatibility is paramount.

Of course Delphi code would need to run in the same way as it does at the moment, I don't think this is difficult to be achieved.

I beg to differ. From my perspective, people bring up issues, you bring up the event mechanism in one of the replies.

Only if I the event mechanism is the only I see how to solve the problem.

At least, it would be a start. Now there is nothing, so we cannot even verify that your mechanism is indeed a superior mechanism, as you claim.

In fact Lazarus does provide Windowistic Messages from a thread to the Main Thread as a basic event mechanism. I did test this some months ago and it did work fine. But it seems to be broken for Linux (KDE) with the current svn version. "My mechanism" is superior in that it would be usable without a GUI (that is why I did the work on that some months ago) and would be implemented in a centralized way independent of the GUI and thus the code should be better manageable.

Regarding functionality it _could_ be better: Windowish messages provide three free queued 32 Bit parameters to the event handler procedure. An implementations like Delphi's "TThread.Queue" would provide no parameter (but self). A decent event queue _could_ be implemented like an an RPC scheduler and provide any number of marshalized queued parameters.

While I agree that an object is a clean approach, for most purposes, an object is simply unnecessary overhead, because it requires additional heap storage.

A simple implementation for something like TThread.Queue would need to do a queue of recored with a pointer to the procedure and it's self pointer. A more advanced implementation would additionally need marshalized parameters. Maybe a sensible approach is doing a queue of three words: procedure pointer, self pointer and additional pointer. This would allow for (1) reproducing TThread Queue and (2) allow for parameters on the heap with the additional pointer we want to reproduce the Windowish Message stuff (for Delphi compatibility) I suppose it would be good to use four Integers as queue records (avoiding the term Object).

-Michael

I thought the one of the points was to have a message queue also when there is no GUI ? This should be testable, no ?

That's what I meant with a paradigm shift.

In that case, you see this quite often. Or, maybe more accurate, more than I would.

Since integer is 32-bit, and you'd need 64-bit for pointers, you'll need to test what size windows messages have.

Once more: please provide a patch.

Michael.

Michael Van Canneyt wrote

Once more: please provide a patch.

As you pointed out yourself, it would be a paradigm shift in Lazarus, so it does not make sense to invest any work.

This is exactly what I meant with me not having the political power.

-Michael

It does, because we would then have a non-gui event queue for those that need/want it.

Now we have none.

Michael.

Indeed.

Plus, even if it is a paradigm shift, it does not sound as if it will immediately bring the whole house down.

Once it gets incorporated into FPC and proves to be a superior solution, GUIs can adopt (or adapt for) it in their strides.

If you take a look at rtl/unix/tthread.inc (
http://svn.freepascal.org/cgi-bin/viewvc.cgi/trunk/rtl/unix/tthread.inc?revision=15599&view=markup ) you'll see that WaitFor simply calls "WaitForThreadTerminate" which is a Thread Manager function. This is set to CWaitForThreadTerminate on Unix systems ( see
http://svn.freepascal.org/cgi-bin/viewvc.cgi/trunk/rtl/unix/cthreads.pp?revision=15557&view=markup ) and just calls pthread_join, which waits for the thread to terminate (without a timeout or other gimmicks like on Windows).

Regards,
Sven

Adem wrote

Plus, even if it is a paradigm shift, it does not sound as if it will immediately bring the whole house down.

That is right, but it would need more than a one person team. It of course would need a new svn branch as well in FPC as in Lazarus. So the appropriate experts would need to help. I did (most of) an implementation using code taken from MSE (I consider the quality of this code extremely high and Martin was very helpful when I worked on his code. He even refactured his code base to avoid the name "Event" that was used in a different definition as it is in Lazarus.)

Once it gets incorporated into FPC and proves to be a superior solution, GUIs can adopt (or adapt for) it in their strides.

What do you think should be considered superior.

This is what I think would be improved;

- an Event Queue would be available in non-Lazarus and in Lazarus-non-GUI applications (so even MSE could benefit from moving part of it's code into the FPC RTL)

- additional to the current LCL event mechanism "SendMessage" / "Procedure ... Message" (which would be implemented on top of the new one) we would have an RTL based "FireEvent" / "Application.Event" mechanism that not only fires "Windowish" Events with three 32 Bit Parameters but Event handlers with multiple marshalized/queued parameters. A "Procedure...Event" syntax candy could be done on top of that.

- The implementation could be done in a thread save way so that an Application object could be instantiated by a worker thread and so same gets its own event queue allowing for inter-thread notification and RTL-based TTimers, thus allowing for a decent event-driven programming paradigm in all threads

- the "Event-Queue" code is centralized in the RTL and not reduplicated in any Widget Type, improving manageability and thus helping enhancing and debugging it (see: currently broken implementation for Synchronize and SendMessage in Linux, not yet existing Delphi-compatible TThread.Queue)

- this could be the base of thread-pool driven implementation of Prism-compatible parallel loops / future variables by means of the RTL and appropriate syntax candy. (Other ways of providing parallel execution can be considered as well)

What will not be improved:

- performance (it supposedly will not be degraded, either), unless we in fact think about parallel loops or other ways of parallelizing)

- current projects and "Delphi"-code will not benefit (but it of course will not need any modification)

I don't have much hope that the maintainers of the different Widget Type implementations agree that this is a viable improvement.

-Michael

Sven Barth wrote

just calls pthread_join, which waits for the thread to terminate (without a timeout or other gimmicks like on Windows).

Sound nice and straight forward. In Windows of course this needs to be emulated (supposedly using Messages).

So -> OP:
does it work decently in Linux ?
does it work decently in Windows ?

-Michael

Michael Schnell wrote

So -> OP: does it work decently in Linux ?

No! No matter where I call MyThread.WaitFor() I get a instand deadlock under Linux. I had to reintroduce a FPC only WaitFor method in a TTHread descendant, that instead uses the Finised boolean variable. A virtual Execute() method is also implemented that uses a interface object to initially set Finised to False, and then when that interface goes out of scope (when Execute is completed), it automatically sets Finished to True. So my reintroduced WaitFor is implemented as follows:

procedure  TMyBaseThread.WaitFor;
begin
  while not FFinished do CheckSynchronize(100);
end;

Unfortunately WaitFor is not virtual, so I had to use reintroduce, and wrapped in in IFDEF FPC because Delphi doesn't require this.

I actually think my solution should be the default implementation for TThread.WaitFor() in FPC - this will prevent deadlocks under platforms other than Windows.

Huh? The problem is this (as I've sent on numerous other mails last week):

On Windows WaitFor does not use WaitForThreadTerminate, but uses MsgWaitForMultipleObjects on two handles: the thread handle and the handle of the event which is set when a thread calls Synchronize. Now if the thread handle is signaled all is well and the job of WaitForThreadTerminate is done.
If the synchronize event is signaled the method calls CheckSynchronize (thus all threads waiting on a Synchronize have a satisfied wait now) and continues waiting on the thread handle after that. But MsgWaitForMultipleObjects has another speciality: If a new message arrives in Windows' message queue of the waiting thread (normally the main thread) this message is handled by calling PeekMessage and thus the message pump will keep working!

This is a significient difference to the way WaitFor in cthreads works and behaves (this one just blocks and sits there waiting for the thread).

Regards,
Sven

I see (silly me just made wrong assumptions instead of looking at the code :(. )

So all this sound good.

Why does it not work as the OP expected ? (Other Main Thread events should still be handled while hanging in TThread.WaitFor)

-Michael

I am all for it, but first explain the following:

WaitFor by itself cannot cause deadlock.

There must be another call as well, I assume Synchronize(), in the thread that you are waiting on. Is that correct ?

Because, if it is so, then you are simply programming wrong. Your coding style causes deadlock.

Thread 1 (main thread) says: I'll wait for thread 2;
Thread 2 says: I'll wait for thread 1 (implicitly, in sycnhronize)

Typical deadlock situation.

So, your logic contains a flaw, *IF* we take into account the current specs of WaitFor() and Synchronize().

So what you propose is a change in specs, namely: WaitFor() is blocking *except* in the main thread, where synchronize calls are still executed. Main reason for this change is to avoid potential deadlocks.

Is this correct ?

Michael.

Why should Synchronize deadlock a Waitfor ? I understand that in Waitfor other main thread events (e.g. Gui generated ones) should run. Synchronize stalls the thread until the appropriate main thread event handler (given as a parameter with Sychronize) has been run. If MainThread Events do run in WaitFor there is no need for an (additional) chance for a deadlock.

-Michael

Op 2010-10-12 11:43, Michael Van Canneyt het geskryf:

Correct.

Please take a look at tiOPF's tiLog and tiThread units to see the usage. Taking the logging units as an example. So loggers need synchronize (eg; when writing to a GUI log window), other loggers don't need synchronize (eg; when outputing to the console).

Threading 101: If you access the GUI from a thread, you need to use synchronize.

Now my application starts up with a GUI log window (tiOPF normally uses -lv parameter for that). I instantly created 1 million log events. The GUI logger is cached to not slow down the GUI (main thread), so it takes a while before all those 1 million log events appear in the log window. I then decide to quit the application. The GUI logging thread is still busy processing the 1 million log events, so the main thread (application) has to wait for the GUI logger to finish, because it terminates the application. The thread uses synchronize because it it's writing to a GUI window. There is your DEADLOCK under Linux! My application doesn't need to keep track of such logger thread, the tiOPF frame handles that all for me, by creating and destroying those logger classes when needed. So I have no OnTerminate event handlers or Boolean IsLoggerFinished in my application.

This is just one example of where tiOPF uses threads. There is pooled threads, logger threads, thread list, app server thread etc... Simply search the tiOPF source for 'WaitFor' and you'll find lots of different examples. I can't think all of them is just wrong??

Yes, that sounds about right.

Regards,
- Graeme -

Op 2010-10-12 12:14, Graeme Geldenhuys het geskryf:

> Thread 1 (main thread) says: I'll wait for thread 2; > Thread 2 says: I'll wait for thread 1 (implicitly, in sycnhronize) Please take a look at tiOPF's tiLog and tiThread units to see the usage.

Also tiPool, and many of the tiOPF unit tests...

Regards,
- Graeme -

[snip]

threads, logger threads, thread list, app server thread etc... Simply search the tiOPF source for 'WaitFor' and you'll find lots of different examples. I can't think all of them is just wrong??

Of course. It _IS_ wrong programming. Under windows you get away with it, because the messaging system bypasses the threads.

But strictly speaking in thread API terms, you have a classical deadlock situation. Nowhere does it say that synchronize() will work if the main thread waits for a particular thread.

From the Delphi docs:

"Synchronize causes the call specified by Method to be executed using the main thread, thereby avoiding multi-thread conflicts. If you are unsure whether a method call is thread-safe, call it from within the Synchronize method to ensure that it executes in the main thread.

Execution of the thread current is suspended while Method executes in the main thread."

Note that it doesn't say WHEN it will be executed. Just that, if executed, it is in the main thread.

and in "WaitFor" it says:

"Call WaitFor to obtain the value of ReturnValue when the thread finishes executing. WaitFor doesn't return until the thread terminates, so the thread must exit either by finishing the Execute method or by exiting when the Terminated property is true."

There you have it. Nowhere does it say that if the main thread is blocked, your synchronize() method will be executed nonetheless. Indeed, both specs explicitly say that the calling thread is suspended or blocked (!) till the call returns.

So strictly speaking, your (or tiOPF's) code is assuming something which is in no way guaranteed by the API, and thus create deadlock.

So you agree this is a change in specs ?

We can argue that this change is necessary (there is a strong point for it).

But as documented, I think I demonstrated that the FPC RTL works correct, and the programs are simply wrong.

I will see about changing the specs.

Michael.

There is not a single line of documentation that justifies this expectation.

Synchronize stalls the thread until the appropriate main thread event handler (given as a parameter with Sychronize) has been run. If MainThread Events do run in WaitFor there is no need for an (additional) chance for a deadlock.

Events bypass normal threading mechanisms, in particular synchronize(). You get away with it on Windows because of this.

Now, before you discuss: I understand that the 'expected behaviour' is desirable, but I just want to argue that your expectations are not backed up by the API documentation, and that, therefor, you are responsible for creating deadlock.

See also my reply to Graeme.

Michael.

Op 2010-10-12 13:06, Michael Van Canneyt het geskryf:

Of course. It _IS_ wrong programming. Under windows you get away with it, because the messaging system bypasses the threads.

OK what do you propose is the "correct thread programming" solution to tiOPF's "log to GUI" example I gave? I'm eager to learn. :) Just don't say: Introduce multiple boolean FThread1Finished, FThread2Finished variables in each of my applications, and surface all internal thread instances in the various tiOPF classes to we could access their OnTerminate events? This sounds all wrong to me.

Thanking about what you said, I can only guess the Windows developers (from Delphi and even FPC) saw the frequent deadlock problem in TThread.WaitFor, and decided to simply change its implementation from WaitForSingleObject() to rather use MsgWaitForMultipleObjects() to work around the issue... and more importantly, cater for what developers expected. Unfortunately such work-arounds are not available in all FPC supported platforms, so currently the "expected behaviour" and "actual behaviour" of TThread.WaitFor is very different on other platforms. Having your application freeze is a rather serious issue.

If your "correct programming with threads" was strictly adhered to, then FPC developers should change the Windows implementation of TThread.WaitFor to rather use WaitForSingleObject(). That way, at least there is consistency across platforms. "single source, compiler everywhere" with nice "consistent behaviour" and the all important "correct usage of thread programming" is forced onto all. ;-)

I will see about changing the specs.

I wouldn't hold by breath, that's why I simply opted to implemented a new base class (TThread descendant) and implemented a new WaitFor() in there. At least now it works as expected on all the platforms I use.

But unfortunately many other FPC developers will first have to experience the problems, then have to jump through the same hoops I did, to get their applications working.

Weren't you one of those that said a little while ago that TThread is ultimately flawed anyway... so maybe a new implementation of WaitFor (and a new definition of what it does) is a tiny step in the right direction.

Regards,
- Graeme -

No need to get desperate; I already discussed it, and we'll change the implementation as you propose. I'll do it personally.

Michael.

Op 2010-10-12 14:38, Michael Van Canneyt het geskryf:

I already discussed it, and we'll change the implementation as you propose. I'll do it personally.

Thanks Michael.

PS:
I got the tiOPF v2 unit tests failures way down now! Just a few more minor fixes to get the remainder to pass. Before I had something like 300-400 failures, now only 18 failures remain. :-)

http://opensoft.homeip.net:8080/tiopf/fpcunit/index64.html

Regards,
- Graeme -

Hello Lazarus-List,

Tuesday, October 12, 2010, 1:06:09 PM, you wrote:

MVC> But as documented, I think I demonstrated that the FPC RTL works correct, MVC> and the programs are simply wrong.
MVC> I will see about changing the specs.

With the change, what happends if the user press the "X" to close window while the code is in a WaitFor ? Message will be processed and window destroyed ?

Michael Van Canneyt wrote

Now, before you discuss: I understand that the 'expected behaviour' is desirable,

Hmm. If a behavior is as well desirable as observable in all good old Delphi versions and not documented otherwise anywhere, I consider different behavior a bug.

-Michael

That depends on what happens in synchronize, of course.

The change as planned will be the following:

If the main thread does a waitfor() in a loop, it will be with a timeout. If timeout is reached, checksynchronize() will be called.

I will see about adding a hook mechanism for this, in which 'processmessages' or so can be done.

Michael.

These things usually do.

But, you seem to have made some headway and that appears to a good point to start from/with.

Can't sepak on behalf of anyone but I doubt you will be denied a brach --taht's what I gatehered from Vincent's words ["no 'political power' needed."]

> Once it gets incorporated into FPC and proves to be a superior > solution, GUIs can adopt (or adapt for) it in their strides. What do you think should be considered superior.

What you've listed below --plus that a solution will be available for non-Lazarus and in Lazarus-non-GUI applications.

Good.

Good.

Good.

Good.

Good.

What will not be improved: - performance (it supposedly will not be degraded, either), unless we in fact think about parallel loops or other ways of parallelizing)

So, no loss/damage there.

- current projects and "Delphi"-code will not benefit (but it of course will not need any modification)

That's good enough for me. If anyone wants to use this new code, s/he would have to add the necessary glue code using IFDEFs, right? If so, I can't see any damage there either.

I don't have much hope that the maintainers of the different Widget Type implementations agree that this is a viable improvement.

My hope/guess is that they may want to modify/extend it in time to cover their specialized needs (if any); but, having a universal/common code should benefit them/everybody too.

All in good time.

Cheers,

Adem

En/na Michael Van Canneyt ha escrit:

And in older versions of the documentation (delphi 2) it explicitly says

"Don't call WaitFor in the context of the main VCL thread if your thread uses Synchronize. Doing so will either cause a deadlock, making it appear that your application has hung, or cause an EThread exception to be raised. (Synchronize waits for the main VCL thread to enter the message loop before allowing the method it is trying to synchronize to execute. If the main VCL thread has called WaitFor, it won't enter the message loop and Synchronize will never return. TThread detects that case and will raise an EThread exception in the thread, causing it to terminate and, if not caught in the Execute method, the application will terminate as well. If Synchronize is already waiting on the main VCL thread when WaitFor is called, TThread can't intervene, and your application will deadlock.)"

They probably changed the implementation later on to avoid this, but old farts like me still avoid to mix synchronize and waitfor.

Bye

... And Delphi 7 help explicitly mentions that Synchronize() does not work in console applications (VCL and CLX), since it uses the message system.

Seems to me that there is plenty of cause for complaints :)

Michael.

Well... then the person who implemented WaitFor on Windows thought about all those problems and circumvented them...

Regards,
Sven

Op 2010-10-13 09:27, Luca Olivetti het geskryf:

They probably changed the implementation later on to avoid this, but old farts like me still avoid to mix synchronize and waitfor.

That just proves that my assumption was correct. Borland noticed all the deadlocks occur in spite of there efforts to document it (nice a clearly by the way). Thus they worked around the problem by using a different Windows API (MsgWaitForMultipleObjects), to allow for the developers expected usage.

So it is just fitting that FPC changes its implementation under non-Windows platforms too. ;-)

Regards,
- Graeme -

You can all rejoice and merrily start coding: rev. 16145 contains a fix, It's been confirmed by Graeme.

Michael.

Adem wrote

My hope/guess is that they may want to modify/extend it in time to cover their specialized needs (if any); but, having a universal/common code should benefit them/everybody too.

I don't suppose that it will run without any modification in the Widget Type code (e.g. as "procedure..message" would be hooked to the RTL code instead of the LCL code).

A decent modification would be deleting the Widget Types' own event queues and use the one in the RTL and merge the GUI messages (Windows) / GUI callbacks (Linux/MAC) into same.

A less perfect way would be to do the Widget Types' event queue object as a descendant of some virtual Object done in the RTL and hook it in instead of the RTL code.

Right now no common interface for the Widget Types' event queues seems to exist, and so they are not "hookable".

BTW.: as MSE would benefit from the RTL improvement, as well, Maybe Martin would be inclined to help here. He created the appropriate code do make it happen.

-Michael

Obviously you are right. But I don't see how WaitFor can at all be used with a thread that does Synchronize in it's loop, if the implementation does not provide for having synchronized methods executed while the Main Thread is in a WaitFor. So such an implementation would render WaitFor useless and it should be removed or documented "don't use".

-Michael

Michael Van Canneyt wrote

I already discussed it, and we'll change the implementation as you propose. I'll do it personally.

GREAT !

Thanks,
-Michael

I was dreaming the Open Source Dream that everybody could use the best available code for a given purpose .

But I found that not only the open source Object language "Event Queue" code is forked in at least 7 different versions (Lazarus/Windows, Lazarus/KDE2, Lazarus/KDE3, Lazarus/Gnome, Lazarus/Mac, Lazarus/fpGUI, MSE), but also Martin Has forked the FPC RTL so that he can't easily move his (IMHO the best available, as most versatile) code into the official RTL, even with help of the FPC "powers" and my tiny assistance.

So this is the Open Source Nightmare of uncontrollable diversification.

Maybe I'd rather stick with what Microsoft might offer.

-Michael

... code is forked in at least 7 different versions (..)

Ooops. sorry for typing faster than thinking :-[ .

Seemingly at least 9 versions( Lazarus/carbon, Lazarus/cocoa, Lazarus/fpgui, Lazarus/gtk, Lazarus/gtk2, Lazarus/qt, Lazarus/win32, Lazarus/wince, MSE).

@ all: Sorry for having bothered you with my silly suggestions. :-(

-Michael

You still can.

But I found that not only the open source Object language "Event Queue" code is forked in at least 7 different versions (Lazarus/Windows, Lazarus/KDE2, Lazarus/KDE3, Lazarus/Gnome, Lazarus/Mac, Lazarus/fpGUI, MSE)

Eh ?

How else can you do this ? Each widget system uses it's own mechanisms, the LCL must somehow unify this, and they did. To remain delphi compatible, they based the 'public interface' on the Windows mechanism. There was no point in adding an additional layer in between.

That they succeeded in doing so is quite an achievement.

Martin doesn't care about Delphi compatibility. This is his decision, and allows him more freedom in his choice of event system, and he has maximised the options he had. You are free to prefer his code over the LCL. Open source is about having the choice.
But, as the saying goes, choosing is losing.

Michael.

"Windows" is the only OS that provides an Event "Message" queue to the running applications (in fact Windows even does provide it to any thread, while Delphi and Lazarus only provide means to use it in the Main Thread). With all other "Widget Types" (that allow for Event Driven programming) the Event queue is done in pascal code in the LCL (or MSE library). All these Widget Types are binding to a GUI Tool set that uses call backs to fire the GUI events. Theses GUI events need to be scheduled together with Timers and thread based events, so an Event Queue is implemented in Pascal code.

A decently versatile central Event Queue implementation (best: in the RTL) would be usable for all Widget Types. With Windows it needs to be checked if this leads to a performance degradation (using user space code instead of doing some additional system calls might in fact even improve the performance, but with all other Widget types the resulting code flow should be quite similar and thus performance should stay the same.

There was no point in adding an additional layer in between.

I explained several points that could be improved by this in the mail above to which Adem replied with the multiple "good"s :) .

That they succeeded in doing so is quite an achievement.

Yep. I am very happy that there have been all these knowledgeable and helpful people that made th current implementation work ! But the forking of the Event Queue code of course is a result of historical reasons and not of a previous plan. I am sure if the same people had been in a position to do a plan for the current extent of the LCL before starting the first line of code they would decide to do a central Event Queue implementation.

Martin doesn't care about Delphi compatibility.

This is true, but IMHO it's not really relevant. The makeup of the message queue does not harm nor force Delphi compatibility. A decent Event Queue can schedule Windows-Compatible GUI events and Delphi-compatible "Procedure... Message" (e.g. by emulating Windows Messages), schedule parameter-less Events (for Delphi compatible "Synchronize" and "TThread Queue". Additionally it can schedule any "advanced" GUI events another Widget Type might ask for.

This is his decision, and allows him more freedom in his choice of event system, and he has maximised the options he had.

In fact I did a deep look at his Event Queue implementation when I ported it to Lazarus/Linux. I did not see anything that would prevent Delphi compatibility but the use of the identifier name "TEvent". Upon my request, Martin already changed this name in is snv, so that I could do the porting just by copying unmodified snippets of his (then) current svn code, as was my intention.

You are free to prefer his code over the LCL. Open source is about having the choice.

Yep, if I would prefer his complete system (which I don't for obvious reasons as stated above) I would just use it, but do not wish for an additional ninth implementation of the event Queue mechanism for a non-GUI Widget Type (that was my original intention) forking whatever code available.

This would increase the said Open Source nightmare that currently prevents doing a (IMHO viable) improvement.

-Michael

Op 2010-10-18 13:22, Michael Schnell het geskryf:

In fact I did a deep look at his Event Queue implementation when I ported it to Lazarus/Linux. I did not see anything that would prevent

Where is this implementation of yours? How can I see what you have done?

Regards,
- Graeme -

How about a DBUS compatible way (if no DBUS there, have a native surrogate, but if there is DBUS, use it)?

Would it not work ?

L.
[...]

-Michael

D-BUS does not provide an event loop, but it must be integrated with existing event loops. It also requires linking to libdbus, introducing again a dependency on some external tool.

Michael.

Graeme Geldenhuys wrote

Where is this implementation of yours? How can I see what you have done?

It's a really lame thing. I just took the appropriate code from MSE Event Queue and Timer and made in run in a Lazarus no-gui project. The only interesting task was to find which code lines were necessary and which were not, as I wanted to use as little "foreign" code as possible.

In a second project I tried to strip down the KDE2 widget set to drop the GUI binding and make it usable for a no-GUI project. I got into that for quite an extent but in the end I failed understanding some of the deepest details of the setup procedure of the "interface"es interface to the GUI.

I am not fixed on using the MSE code, I just were more successful understanding it then with the Lazarus code.

Of course I would be just as happy with the extracts of the Lazarus code moved into the RTL as with extracts of the MSE code.

-Michael

Lukasz Sokol wrote

How about a DBUS compatible way

What is DBUS ?

-Michael

introducing again

a dependency on some external tool.

As another issue (important regarding threads) already discussed is dropping the libc interface unit and with that the libc binding in Linux (and with that the need of doing thread creation, threadvar implementation and Futex stuff in the RTL) this really would be a step in the wrong direction.

-Michael

Michael Schnell wrote

On 10/18/2010 03:23 PM, Lukasz Sokol wrote: > > How about a DBUS compatible way What is DBUS ?

A very nice inter-process messaging and RPC system. Quite powerful. It's the standard these days on *nix desktops
(inasmuch standards exist at all on *nix)

See

http://www.freedesktop.org/wiki/Software/dbus

Michael.

Ehm... "unit libc" "interfacing/using libc". The libc unit is a relict from Kylix times and is basically i386 only. But interfacing with/using libc is not a bad thing (TM) per se.

Regards,
Sven

I do agree. I was not completely convinced that not using the pthreadlib in the libc for managing the thread stuff. I suppose there are pros and cons (see the "Threadvar" discussion). I suppose when getting at that a discussion what and how stuff from *libc*.so is to be accessed.

-Michael

2010/10/18 Michael Schnell :

Michael, please read
http://wiki.lazarus.freepascal.org/libc_unit#The_problem_with_unit_libc, so you know what you are talking about and understand the difference between library libc and unit libc.

Vincent

I do know, but anyway while it does make sense to use some functions from libc, there might be others that maybe better are done in the RTL without accessing the libc equivalent.

Thread support _might_ be a candidate, as threadvar access could be improved (but maybe it could be improved even without dropping the threadlib binding). OTOH threadlib automatically provides FUTEX access for archs that provide it.

OTOH, on archs like the different ARM flavors, the RTL implementation of "atomic" functions is not perfect.here using or looking at libc might lead to an improvement, as same automatically uses the appropriate Kernel-helper-functions or the advanced hardware instructions if available in the sub-arch without the need of recompiling the project.

-Michael

Michael Schnell wrote

Thread support _might_ be a candidate, as threadvar access could be improved (but maybe it could be improved even without dropping the threadlib binding). OTOH threadlib automatically provides FUTEX access for archs that provide it.

No it could not, since this is non-portable. The current threadlib is global for all *nix, not *nix flavour-architecture specific.

And unit libc has never had any part in the RTL.

Marco van de Voort wrote

The current threadlib is global for all *nix, not *nix flavour-architecture specific.

I suppose you are right,

C program use appropriate header files that provide inline functions for atomic. Same are "flavor-architecture specific", providing different algorithms (with some sub-archs supported by Kernel tricks) to allow for atomicness. When compiling an so library it can use the appropriate header file (e.g. to do Futex).

Seemingly, to take advantage from this we could use libc to use Futex ("pthread_mutex()"). But for static atomic functions the sub-arch needs to be known when compiling.

-Michael

That is all handcrafted hackery. Sure, you can do that with FPC too, if you can get used to configure like tricks and preprocessors.

Seemingly, to take advantage from this we could use libc to use Futex ("pthread_mutex()").

pthread_mutex is used by tbasicevent for ages. If pthreads maps futexes under normal mutex if available, it should all work out of the box. (and afaik already since 2.4.0)

IOW, just use syncobjs.