Hi,

Is it the line with InterlockedExchange() call? The reason I ask, I'm using FPC 2.4.3 under 64-bit linux and using fpGUI. All other platforms has a clear 'IsMultiThreaded := True' in the RTL code, but not for Unix. It seems CheckSynchronized() is never called in fpGUI/X11 because IsMultiThread is never True, even though I have 'cthreads' unit defined as the first unit in my fpGUI program.

-------------[ rtl/unix/cthreads.pp]------------------
....
      { Initialize multithreading if not done }
      if not IsMultiThread then
        begin
          if (InterLockedExchange(longint(IsMultiThread),ord(true)) = 0) then
            begin
              { We're still running in single thread mode, setup the TLS }
              pthread_key_create(@TLSKey,nil);
              InitThreadVars(@CRelocateThreadvar);
            end
        end;
....
-------------------------------

In fpGUI, CheckSynchronized is executed as follows in the main event loop.

....
  if IsMultiThreaded then
    CheckSynchronized();
...

InterlockExchange does exactly that. It writes "ord(true)" to "longint(IsMultiThread)" and returns its old value (0 = ord(false)).

Note: You must start at least one thread (besides the main thread) to turn "IsMultiThread" to true.

Btw: the check for IsMultiThread inside of RTL code is a bit dangerous as ported Delphi applications might set it manually to true out of old habit (which is bad if the multithreading initialization checks that var).

Regards,
Sven

Graeme Geldenhuys wrote

Is it the line with InterlockedExchange() call?

Yes.

The reason I ask, I'm using FPC 2.4.3 under 64-bit linux and using fpGUI. All other platforms has a clear 'IsMultiThreaded := True' in the RTL code,

Multiple threads may be started simultaneously, so setting/checking IsMultiThreaded must be done atomically if it's done inside a new thread's execution context.

but not for Unix. It seems CheckSynchronized() is never called in fpGUI/X11 because IsMultiThread is never True, even though I have 'cthreads' unit defined as the first unit in my fpGUI program.

As Sven mentioned, IsMultiThread only becomes true after the first thread has been started.

Jonas

Sven Barth wrote

Btw: the check for IsMultiThread inside of RTL code is a bit dangerous as ported Delphi applications might set it manually to true out of old habit (which is bad if the multithreading initialization checks that var).

Such applications are simply guaranteed to break on FPC. It is wrong to manually set IsMultiThread to true.

Jonas

Hi,

Sven Barth wrote

Note: You must start at least one thread (besides the main thread) to turn "IsMultiThread" to true.

I am, and changing my fpGUI code to the following does indeed prove that you are right. As soon as my thread is started, I see the '*****' output, but unfortunately only once. Let me explain.

   if IsMultiThreaded then
       begin
         writeln('******');
        CheckSynchronized();
       end;

What I am trying to debug is that as soon as I call mythread.waitfor, then WaitFor() blocks the applications main event loop, thus the application just hangs. This only occurs under X11, not under Windows.

I can work around that problem, by doing the following.... I have to introduce a boolean variable in my thread class, set it to True at the end of the thread's Execute method, as follows:

procedure TBarThread.Execute;
begin
  FFinished := False;    // work-around variable
  while not Terminated do
  begin
    Synchronize(@UpdateProgressBar);
  end;
  FFinished := True;     // work-around variable
end;

....Then use a while loop (instead of WaitFor), to check for that variable, and if not True, call fpgApplication.ProcessMessages instead.

procedure TMainForm.ButtonClicked(Sender: TObject);
begin
  ProgressBar1.Position := ProgressBar1.Min;
  if not Assigned(FThread) then
  begin
    writeln('program: creating thread...');
    FThread := TBarThread.Create(True);
    FThread.ProgressBar := ProgressBar1;
  end;
  writeln('program: starting the thread...');
  FThread.Start;
  writeln('program: waiting for thread...');
  FThread.WaitFor;    // Can use this, it blocks the main event loop
  // my work-around
  while not FThread.Finished do
  begin
    sleep(100);
    fpgApplication.ProcessMessages;
  end;
  writeln('program:  thread is finished!');
  Label1.Text := 'Thread is done';
  FreeAndNil(FThread);
end;

Obviously this is a hack, and damn ugly. I shouldn't need to do this. But still I haven't figured out why the application's main event loop is blocked by WaitFor.

Writing a console app, that does use a thread to count down from 20 to 0, I can use a WaitFor, and it works (even under Linux). So clearly there is a problem in fpGUI, and more specifically in the X11 support code. Unbelievable that I haven't picked up on this sooner.... then again, I have never had a need for WaitFor usage until now. :-/

I can live with that ^^

Regards,
Sven

1. If you do a WaitFor the calling thread will be suspended until the thread you called WaitFor on terminates (or the timeout elapses, but our WaitFor doesn't have a timeout parameter...). Thus it is normal that your main loop appears to hang.
2. Does your thread really terminate till then? Did you test a thread that does nothing and terminates immediatly?

I can't help you much with X11, cause I've not yet found the time and the need to dig into it.

Regards,
Sven

What about dropping that loop and put it in UpdateProgressBar

TForm1.UpdateProgressBar;
begin
  Something here
  Application.ProcessMessages
end;

And drop the waitfor all together?

I thought about the timeout, and dived into the WaitFor code. So I changed my code from.

    FThread.WaitFor;

..to...

    WaitForThreadTerminate(FThread.Handle, 2000);

I waited for 10 seconds and still nothing. The whole app was frozen. So it seems the timeout logic in TThread (under unix) is broken too. :-(

2. Does your thread really terminate till then? Did you test a thread that does nothing and terminates immediatly?

It freezes immediately at the point where I call Syncronize() in the thread's Execute method. With both options shown above.

If I remove the Syncronize(), then the thread keeps running the loop inside Execute. Trying to use WaitForThreadTerminate() with the timeout value > 0, it never happens. So the timeout logic clearly seems broken.

I can't help you much with X11, cause I've not yet found the time and the need to dig into it.

No problems... just thought I would ask in case you knew.

Andrew Brunner wrote

> What about dropping that loop and put it in UpdateProgressBar And drop the waitfor all together?

Ignore what the thread does... it's a dumb example. The thread can do any long running job. I simply wanted to wait for it to finish before I do other processing. So I wanted to use the TThread.WaitFor to see how it works. Then I stumbled upon the problem that it freezes up my app. :-/

The timeout parameter is ignored when using cthreads, because the pthreads API does not support waiting for a thread termination with a timeout.

Furthermore, the whole point of WaitFor() is to block the current execution thread from continuing until the thread has finished (it's a synchronisation point). If you just want to quickly check whether the thread has finished, check FThread.Finished (followed by FThread.Waitfor to clean it up in case it's true).

Jonas

Jonas Maebe wrote

If you just want to quickly check whether the thread has finished, check FThread.Finished (followed by FThread.Waitfor to clean it up in case it's true).

I just noticed that's a private field that's not exposed. I guess the only way is to install an OnTerminate event in your tthread, or to expose the value of the "Terminated" property in a child class (it's a protected property).

Jonas

Jonas Maebe wrote

The timeout parameter is ignored when using cthreads, because the pthreads API does not support waiting for a thread termination with a timeout.

So what does Qt do different, so they have a wait option. From there docs it doesn't explicitly mention your application will be frozen until the thread terminates. They also support a timeout, so it must be possible in FPC too?

http://doc.trolltech.com/4.7/qthread.html#wait

See, currently it's a double whammy under Linux. WaitFor (under unix) has an infinite timeout, yet timout is not implemented, so as soon as you call TThread.WaitFor (in any GUI app: fpGUI, MSEgui, LCL-GTK2 etc.), you can just as well go kill your process, because it never returns. You app is frozen! This is what I'm trying to explain.... and find a possible way of getting it to work in GUI apps.

This seems like a discussion of semantics more than anything. In an application thread if you say WAITFOR something then the app cannot process messages b/c the execution is put on hold until WAITFOR returns. This is normal behaviour, IMO.

So if this is just an exercise, I really don't see a problem. What's the issue you're trying to solve? Create a two thread system and have the second thread WAITFOR the third thread to complete. Don't do this with the main application thread for GUI apps. Service apps aren't even an issue b/c it would just create a blocking condition - again normal.

In our previous episode, Andrew Brunner said:

Or wrap such horrible threads (that hang long time, e.g. due to external programs) in an additional mechanism using an TEvent. TEvent has a waitfor with a timeout.

As Marco mentioned, it can probably be implemented by wrapping the entire thread in something that sets an rtlevent when it's finished. That would require quite a few changes to cthreads though, and I'm personally not convinced it's worth the trouble. If someone wants to submit a patch for that however, be my guest.

See, currently it's a double whammy under Linux. WaitFor (under unix) has an infinite timeout, yet timout is not implemented,

I don't understand what you mean by "yet timeout is not implemented". WaitFor() on all platforms has an infinte timeout. Maybe Windows' messaging system keeps running in the background while your main thread is blocked, but that is unrelated to any kinds of timeouts. WaitFor() by definition blocks the current thread of execution until the thread you are waiting for has finished.

so as soon as you call TThread.WaitFor (in any GUI app: fpGUI, MSEgui, LCL-GTK2 etc.), you can just as well go kill your process, because it never returns.

It will return when the thread you are waiting for is finished, and that's the same on all platforms. If the thread does not finish, it should not return on any platform. E.g., this is part of the Windows code for WaitFor():

     WaitForSingleObject(ulong(FHandle), INFINITE);

Another thing to keep in mind is that you cannot safely WaitFor() a thread that has FreeOnTerminate=true. If you want to wait for a thread, make sure that property remains false (which is the default).

Jonas

OK, my example just just that... to highlight the problem of using TThread.WaitFor.

My real usage is in tiOPF. tiOPF has various cached logging classes, all done via threads. It's cached
because we don't want to slow down the main application while output is written to a file or log window. So logging is written out to disk (or a GUI screen) when the thread gets time slices. Now say you instantly generated 1 million log entries (so the thread has a huge backlog, and because you have a slow hard drive...in the case of logging to a file), and now you quit your application.
The application needs to wait for the logging threads to complete before it can terminate.
This is where the problem comes in. As soon as I call SomeThread.WaitFor, the application is frozen - it never gets to terminate completely because WaitFor never returns. All our apps are database GUI apps, but strangely enough, it is not only limited to GUI apps. The unit testing framework of tiOPF is Text or GUI. I run the daily build under our Linux server and it is non-gui based. The test runs never complete/terminate because there are five logging threads frozen which never returned from WaitFor. So I had to implement that boolean field and a while loop hack, instead of calling WaitFor [when I use FPC and Linux].

A better way of achieving this is looking at the collection of threads you have. Set your own booleans for Finished or if they freeonterminate you will need to create an
TCompletes=Array[0..threadcount] of boolean. And as the threads free set the variable of TCompletes to true. Then you can poll the Completes to see if they are all true.

There are a million ways to accomplish safe thread usage without waitfor...

Yes, but the alternatives above are only safe if you only use atomic operations (which could be quite slow in case of an array of booleans due to the atomic reservation granularity), or if you add proper memory barriers everywhere.

Jonas

That comes down to the same thing I have done with the single boolean variable. All logging thread classes descend from a base log thread class. It's in that base thread class that I implemented the boolean variable.

There are a million ways to accomplish safe thread usage without waitfor...

Yes, but my point is that FPC's "single source, compile everywhere" doesn't works so well in this case, because TThread.WaitFor works completely different under Windows to what it currently does under Unix/Linux. So the code now has to contain IFDEF's - which all tiOPF team members hate.

Here is the offending bit of code in tiOPF's tiLog unit.

procedure TtiLogToCacheAbs.Terminate;
begin

  FThrdLog.Terminate;
  {$IFDEF FPC}
  { graemeg: 2009-05-04
    FPC handles WaitFor slightly different under Unix environments, so I rather
    do the following which seems safer. Delphi could probably also use this
    method.
    When FThrdLog's Execute() method is done, it will set Finished to True,
    which we can then detect to know that the thread is done.

    See the following URL for a more detailed explanation:
      http://free-pascal-general.1045716.n5.nabble.com/TThread-WaitFor-not-returning-td2820297.html
    }
  while not FThrdLog.Finished do sleep(100);
  {$ELSE}
  FThrdLog.WaitFor;
  {$ENDIF}
end;

TtiLogToCacheAbs hold the cached log events, and feeds them to the thread which does the actual logging (to file, to screen, to OS system events, to debug server, etc)

tiOPF code must run under D7-Delphi XE, and under latest stable FPC. So I'm limited to common features, and trying to reduce IFDEF's as much as possible.

I disagree. I've never had problems with setting values inside boolean arrays in multi-threaded environments. The memory is already allocated. Setting the value does not cause memory re-allocation, which would be the only source of problems. But of course just because I didn't experience problems in the past with this does not mean I won't with a future. ;-)

Andrew Brunner wrote

I disagree. I've never had problems with setting values inside boolean arrays in multi-threaded environments. The memory is already allocated. Setting the value does not cause memory re-allocation, which would be the only source of problems.

No, the main source of problems is that the cpu may reordering memory operations resulting in those booleans in the arrays becoming true before all memory locations written in the thread are visible. The result is that you may try to use data that the thread should have written earlier, but that in fact is not yet available even though the boolean is already true.

In fact, using atomic operations won't help to solve that problem, you do need memory barriers to flush the memory writes (most pthread functions and ending a pthread guarantee memory barrier behaviour, which is why WaitFor() itself is safe).

On x86 you're unlikely to encounter many problems because of the strongly consistent memory model, but at least x86-64 now also has read and write barrier instructions. On non-x86 architectures, you definitely need memory barriers in such cases.

Jonas

It is impossible to make it work like Windows, since other OSes simply do not have a single event queue (as mentioned by Sergei Gorelkin).

Rather, one should not take Windows as the reference platform, something that the tiOPF developers do. Doing so will always create problems.

I strongly suspect that the code will also not function on windows if you have a windows non-gui, non-event based program, but that would have to be investigated.

Michael.

[snip]

[snip]

Looking again at your example code explains very clearly what happens (I assume the CheckSynchronize call is in your ProcessMessages method):

In the thread you call Synchronize which will stop the thread until the main thread has found time to process all queued Synchronize messages (by calling CheckSynchronize). Read again: the thread is SUSPENDED, it's in a waiting state.

Now you call .WaitFor during an event handler. So you wait until the thread is finished.

There is the problem: The main thread is waiting for the second thread to finish and the second thread is waiting for the main thread to find time to process its synchronized message. This is a classical DEADLOCK.

The "workaround" is not really a workaround, but more a "clean solution" (there are other ones that others already mentioned like running an OnTerminate-event, which is supported on Delphi as well).

And it doesn't appear on Windows, because WaitFor uses MsgWaitForMultipleObjects which waits for either the thread's handle, the synchronize timeout (in which case CheckSynchronize is run) or a message that is send from Windows' message system (which is then processed). So a Windows GUI won't hang (Sergei has mentioned this on the Lazarus list).

So from the following statement:

So what is wrong? Is it the TThread.WaitFor implementation that is flawed? Is it X11 that is flawed, or just GUI toolkit developers that don't know how to code (thought I doubt it's the latter).

I'll choose the third option. ;P (please don't be offended)

Regards,
Sven

It will work because the Windows RTL waits for the thread's handle OR the synchronize event. So it will call CheckSynchronize automatically if a thread calls Synchronize and then continue waiting on the two handles again.

Regards,
Sven

So are you referring to the topic of this conversation regarding data flushing? The notion of creating an index of completions to poll and waiting for all the values to turn true as a means to determine actual job completion without using locking mechanisms is just one viable solution. Are you implying that the log data (which is the purpose for this/these worker threads) needs to be flushed prior to tripping the boolean or are you saying that the boolean array needs to be flushed? As a solution I just don't see any problems with boolean values coming back as being true *before* they were assigned as complete. Pre-emptive completion - I just don't see that.

Sven Barth wrote

There is the problem: The main thread is waiting for the second thread to finish and the second thread is waiting for the main thread to find time to process its synchronized message. This is a classical DEADLOCK.

OK, with that clear explanation I now understand the problem. :-)

I'll choose the third option. ;P (please don't be offended)

least I now understand the problem better, and why it doesn't occur under Windows (Delphi or FPC).

Yes.

or are you saying that the boolean array needs to be flushed?

That's not necessary if all the various threads do is setting the values to true. If you reset them to false inside newly started/ resumed threads (although that would be strange), then you may have to flush it.

As a solution I just don't see any problems with boolean values coming back as being true *before* they were assigned as complete. Pre-emptive completion - I just don't see that.

If you have in a thread:

buffer:=nil;
finishedarray[threadnr]:=true;

Then another thread (including the main program) running on a different core can see both (finishedarray[threadnr]=true) and (buffernil) at the same time, except probably on x86. This can be prevented by adding memory barrier between those statements. A memory barrier orders whatever memory operations come before it with respect to those that come after it.

Depending on the internal state that these logging threads use and whether or not its reused or read later on, that may or may not be important.

Jonas

Right. So you *were* talking about the actual log data being flushed

That may be true with regard to actual disk writing, but when polling occurs and the value is read as true, any thread can assume it was already tripped. And I agree that other attributes' values may or may not be accurate to observing threads, but I (assume you agree as well) false positives are impossible for the purpose of polling as stated. If not, I missed your point :-(

Hello FPC-Pascal,

Friday, October 8, 2010, 3:49:52 PM, you wrote:

GG> The application needs to wait for the logging threads to complete GG> before it can terminate.
GG> This is where the problem comes in. As soon as I call GG> SomeThread.WaitFor, the application is frozen - it never gets to GG> terminate completely because WaitFor never returns. All our apps are GG> database GUI apps, but strangely enough, it is not only limited to GUI GG> apps. The unit testing framework of tiOPF is Text or GUI. I run the GG> daily build under our Linux server and it is non-gui based. The test GG> runs never complete/terminate because there are five logging threads GG> frozen which never returned from WaitFor. So I had to implement that GG> boolean field and a while loop hack, instead of calling WaitFor [when GG> I use FPC and Linux].

Are you calling Synchronize inside the thread ? Synchronize will only happend when Application.ProcessMessages is being internally called, which not happend if you are WaitingFor. I think that if the thread calls Synchronize while you are in the WaitFor you will be in a race condition, Synchronize waiting for Application.ProcessMessage and the application waiting for the thread to finish.

I think Jonas was talking about out of order execution in the cpu, not disk i/o.

Henry

This isn't a race condition, but a dead lock (but both are bad things ^^).

Also on Windows this works, because on Windows WaitFor handles Synchronize events as well.

Regards,
Sven

Hello FPC-Pascal,

Friday, October 8, 2010, 8:03:01 PM, you wrote:

SB> This isn't a race condition, but a dead lock (but both are bad things ^^).

You are right :) I'm used to use the "Deadlock" term when explicit locks are being used, but yes, that's a DeadLock.

SB> Also on Windows this works, because on Windows WaitFor handles SB> Synchronize events as well.

Right... this explains a bit of code I was looking at two days ago :) "Why is it using Multiobjects waitfor ?" and here is the answer :)

Henry Vermaak wrote

I think Jonas was talking about out of order execution in the cpu, not disk i/o.

No, it was out of order writing to memory (RAM), modern CPUs/Memory controllers can do strange things reordering writes to RAM.

Paulo Costa

Right. But in the case of an array of booleans that will be polled and an exit condition will only be met if all booleans are set to true. The order in which a polling thread sees them as being true has no bearing on the functionality.

In our previous episode, Andrew Brunner said:

Right. But in the case of an array of booleans that will be polled and an exit condition will only be met if all booleans are set to true. The order in which a polling thread sees them as being true has no bearing on the functionality.

No, but the last one might be set before the condition coupled to that last one might be carried out. See Jonas' example.

Marco van de Voort wrote

No, but the last one might be set before the condition coupled to that last one might be carried out. See Jonas' example.

The ONLY way that could happen is if the boolean happens to be set in a worker thread's method that still has more work to do. It is understood that a thread would complete it's work before tripping the Complete flag :-)

Threads aren't really that difficult to manage.

That is incorrect. Even if you set the boolean only to true at the very end, problems can still occur. See e.g. http://en.wikipedia.org/wiki/Memory_barrier#An_illustrative_example for an example and explanation.

Jonas

In our previous episode, Andrew Brunner said:
[ Charset ISO-8859-1 unsupported, converting... ]

No, as said before, the relevant memory writes might be differently ordered on some architectures (and bus configurations)

Guys, this is simply a semantic problem. I've got a six core system here. The boolean array initialised all false. thdArray[0..9] threads executing simultaneously. Corresponding values in baArray[0..9]=false. baArray[n] = true iff thdArray[n] Terminated (method assigned to each member of thdArray

ObserverThread.Execute
repeat
  AllTrue:=CHECK_IF_ALL_IN_baArray_IS_TRUE;
  WaitForSingleObject(Handle,300);
until AllTrue or Terminated

ObserverThread WILL ALWAYS terminate AFTER each and every thread has completed. There is no worry about order of completion with this method.