We have a problem!!!

The test project /fcl/tests/threads.pp as well as a Sort Demo (Bubble, Section and Quick Sort) all execute the threads in sequence, waiting for the previous thread to complete, before the next one executes. Kind of defeats the point of using threads.

The Sort Demo, which is a port of the Threads demo included with Delphi 7, shows this clearly. I can archive and post the source, but you would need Lazarus LCL to run it as it is visual. The sort demo has three columns with identical random data. The three sort methods are applied to each column respectively and gives visual feedback as it sorts the data (horizontal lines of different lengths).

The threads.pp test project in fcl, shown a lot of a's on the screen and after a while, starts displaying b's on the screen. Looking at the source, I gather the a's and b's are supposed to be mixed as each thread gets processor time.

I am using FPC 2.1.1 (svn version from a week ago) and running it under Ubuntu Linux 6.10 on x86 CPU (Pentium 4).

Regards,
- Graeme -

The test project /fcl/tests/threads.pp

I see random alteration of a and b here.

The threads.pp test project in fcl, shown a lot of a's on the screen and after a while, starts displaying b's on the screen. Looking at the source, I gather the a's and b's are supposed to be mixed as each thread gets processor time.

If you touch a key, only a's will be displayed. This is the programs behaviour.

ok, I see hundreds of a's - press b and quite a bit later I see hundreds of b's.

Do you have lazarus? I can then post the Sort Demo to show that one as well.

Regards,
- Graeme -

Attached is the Sort Demo - for incase you have Lazarus.

Graeme.

Just to give some more feedback. Using that same Sort App, I compiled it under Delphi 7, and ran it under Linux using WINE and the threading worked correctly. So somewhere in FPC or Lazarus lies the problem. Yeah I know, not very specific. :-)

Graeme.

Graeme Geldenhuys wrote

Just to give some more feedback. Using that same Sort App, I compiled it under Delphi 7, and ran it under Linux using WINE and the threading worked correctly. So somewhere in FPC or Lazarus lies the problem. Yeah I know, not very specific. :-)

Can you try running the example on windows using fpc/lazarus?

Vincent

The only FPC/Lazarus for windows I have is under the same VMWare session as I used for Delph 7. And yes it works there, using FPC 2.0.2 with a Lazarus 0.9.15 (9152). I'll be at home in a hour, so then I can try under FPC 2.1.1 and latest Lazarus.

Regards,
- Graeme -

So far using FPC 2.0.2 and 2.0.4 and latest Lazarus (svn) it works fine under Windows. I am now busy building FPC 2.1.1 (svn) under Windows and will redo the whole process. Will post the result shortly....

Regards,
- Graeme -

Ok, I have tried this under Windows and Linux, with FPC 2.0.2, 2.0.4 and 2.1.1.

This seems to only be a problem under Linux. Threading works fine under Windows (via WINE or native), but is screwed under Linux native! Under Linux, it has been tested with 2.0.4 and 2.1.1 (I saw no point testing it with 2.0.2 as I don't use that version anymore under Linux). Threading runs sequential (one thread, after the other) in Linux, instead of in parallel which totally defeats the point of threads. I take this as a critical bug with FPC - unless someone can proof otherwise. More testing under Linux revealed that I have quite a few apps, that are affected by this.

I have tested this on 4 different machines all with the same outcome. Is any body else using threading in their applications? Are you experiencing the same issues under Linux? If so, I will submit a bug report with a sample app showing the issue tomorrow.

This is the output of 'uname -a' to show my kernel version.

uname -a
Linux graemeg-laptop 2.6.15-27-386 #1 PREEMPT Sat Sep 16 01:51:59 UTC 2006 i686 GNU/Linux

Regards,
- Graeme -

I know about this report:
http://www.freepascal.org/mantis/view.php?id=1906

Threading related, but not the same, as you are not using synchronize, are you?

Vincent

Do you mean like below - which then means, I do use synchronize. I use synchronize to draw on the main form (running in the main thread). I don't get any lockups as that bug report says, but then I am running under Ubuntu that uses Gnome desktop. Again, no such issues under Windows.

TSortThread is the base class for my three different Sorting threads, so they all use this function.

procedure TSortThread.VisualSwap(A, B, I, J: Integer);
begin
  FA := A;
  FB := B;
  FI := I;
  FJ := J;
  Synchronize(@DoVisualSwap);
end;

They never attached a sample application, so I can't try it on my system to see if I get the same issues as they do, but as I stated, I don't get lockups, only that the threads run in sequence - one after the other.

What platform are you running on, Windows?

Regards,
- Graeme -

What is also very interresting, which I thought I would share. I added three more Bubble Sort columns to my test app (total of 6 sort threads - knowing bubble sort is the slowest).

I then created all 6 threads suspended (t1 thru t6 in order). I then resumed the threads in the order t1, t6, t2 thu t5. They still executed in the *creation order* ( t1 thru t6.) and not the resumed order. Meaning t1 started and completed, t2 started and complete, etc.

Not sure if this might help the debug process...

Regards,
- Graeme -

All gui stuff I run on windows. Non visual test apps, I can run on linux too.

If I really want I could do GUI in linux too, but I don't know too much about gtk and I rather spent time on improving things on the windows platform.

I thought in your initial mail your were talking about having a console test app with threads. Synchronize is harder then, because you have to call CheckSynchronize yourself.

Vincent.

Vincent Snijders wrote

All gui stuff I run on windows. Non visual test apps, I can run on linux too. If I really want I could do GUI in linux too, but I don't know too much about gtk and I rather spent time on improving things on the windows platform.

No problem - maybe someone else in the mailing will spot this and try it under Linux with a GUI. Please post your results if you read this.

I thought in your initial mail your were talking about having a console test app with threads. Synchronize is harder then, because you have to call CheckSynchronize yourself.

Yes, after the Sort Demo didn't work, I looked for test apps based on threads supplied with FPC. I found one in the fcl/tests directory. It didn't run as advertised of my system either. Not a great demo though. I will try and knock up a better console demo and see if it still does what the GUI Sort demo does. I just need a good idea for a console demo.

So far all my tests point to a problem with FPC compiled apps and threading under Linux. Kylix 3 apps using threading works fine as well.

Regards,
- Graeme -

Below is a text (console) thread demo. The one thread counts from 0 to 1k and the other thread counts down from 1k to 0. Again, under Linux, one thread executes and teminates, then the next thread executes and terminates. Under windows, both threads run at the same time (output is mixed as expected). No sychronize() is used in this demo. See attached screenshots for my output.

Regards,
- Graeme -

---------------------------------------
program demo1;
{$mode objfpc}{$H+}
uses
  {$IFDEF UNIX}{$IFDEF UseCThreads}
  cthreads,
  {$ENDIF}{$ENDIF}
  Classes, SysUtils;
type
  // counts up till 1k
  TIncrementer = class(TThread)
  protected
    procedure Execute; override;
  end;
  // counts down from 1k
  TDecrementer = class(TThread)
  protected
    procedure Execute; override;
  end;
  TRunThreads = class(TObject)
    procedure ThreadTerminated(Sender: TObject);
  private
    t1, t2: TThread;
    FThreadCount: integer;
  public
    constructor Create;
    procedure RunNow;
  end;

{ TRunThreads }

procedure TRunThreads.ThreadTerminated(Sender: TObject);
begin
  Dec(FThreadCount);
end;
constructor TRunThreads.Create;
begin
  FThreadCount := 2;
  t1 := TIncrementer.Create(True);
  t1.OnTerminate := @ThreadTerminated;
  t1.Priority := tpLower;
  t1.FreeOnTerminate := True;
  t2 := TDecrementer.Create(True);
  t2.OnTerminate := @ThreadTerminated;
  t2.Priority := tpLower;
  t2.FreeOnTerminate := True;
end;
procedure TRunThreads.RunNow;
begin
  writeln('RunNow');
  t1.Resume;
  t2.Resume;
  repeat
    sleep(100);
  until FThreadCount = 0;
  WriteLn('All threads completed!');
end;

{ TIncrementer }

procedure TIncrementer.Execute;
var
  i: integer;
begin
  for i := 0 to 1000 do
    Writeln(Classname + ': ' + IntToStr(i));
  Terminate;
end;

{ TDecrementer }

procedure TDecrementer.Execute;
var
  i: integer;
begin
  for i := 1000 downto 0 do
    Writeln(Classname + ': ' + IntToStr(i));
  Terminate;
end;
var
  lRunThreads: TRunThreads;
begin
  lRunThreads := TRunThreads.Create;
  lRunThreads.RunNow;
  writeln('Done...');
end.

---------------------------------------

Hi - I read enough of this topic to compel me to respond.

I have been working with Multi-threading in Free Pascal for a only a month or so now and I learned a great deal - and I have resigned to accepting there are design considerations I have had to adopt to get the results I wanted - that some times seem a bit less than ideal - but the pay offs have been wonderful. Let me digress a little...

First of all - THREADS running in a SERIAL fashion... Yup...

I had that problem and I resorted to the "SAMPLE" tthreads.pp I think its called, as a starting point. It is an extremely simple example of multi-threading working that comes with either FPC or the FPC source download.

I was fighting the serial execution - and then I ran that little tiny test program - and it worked. I tried to fix my code some more - same problem - but the little test program worked... so I finally (this is what I mean when I say I resigned to design changes) adopted that EXACT example as the foundation for how I would implement my THREADS.

I create my threads like a thread pool, I create them all suspended. (Because that parts works pretty easy)...

I set up the EXECUTE MY THREAD... as a loop... BUT - Each iteration calls its own SUSPEND method.... So it does a cycle and all I have to do is re initialize the "parameters" it works with BEFORE I resume it.... So its... as I call it "ready to fire" again, and then I just resume from the main program that is handling all these threads.

Basically the test program has a working example of two threads looping at the same time and the simple output shows they are truly multi-tasking... I built my app into this same idea - with the addition that I suspend the loop each go round.... the suspended threads in turn become my "ready to do something" ones. If they aren't idle - they are either busy or trying to find the meaning of life.

One of the biggest "issues" which is hard to debug if you aren't looking for it ... is WHY does a thread stay working when it should have finished the loop? It COULD be a bug in some dumb routine - but in many of my own cases -

- it is accessing data that I wanted "SHARED" by all the threads... I know there is this critical section stuff and all that... but I found it simpler to make a RECORD STRUCTURE INSIDE the Threaded class that has all the DATA relevant to that INSTANCE - and then write "shared routines" that PASS this structure by value - so there is less need to worry about sharing data across the threads.

I follow two rules - if it MUST be shared - treat it read only. If it can be copied to the thread's instance and does NOT really need to actively traded between the threads, I put it in this record structure - I refer to as "Thread CONTEXT".

Now - to handle the ABSOLUTELY MUST allow main app to talk with thread etc.

I keep it simple... each thread has a "structure" in the main program's space allocated - and the main program can read an write to it, and the thread can also... I try to keep the data simple - and less consequential to synchronization isn't a big deal - like an IDLE FLAG - or sending a message to the main program. I may write something out there, to a round robin like queue, and manage the pointers, and set up a way for the main program to read the info - when it can in a manner that doesn't interfere with the thread and vice versa.

The moral of this story is four fold in my experience so far.

1: ITS Definitely worth it - the speed attained from multi threading, and in many cases the simplicity of having "specific" tasks go away and work and return when finished - well... I enjoy it.

2: Debugging a shared memory thing or one thread stomping another somehow - is a REAL pain to debug... have a great error logging system you can include or exclude with a compiler switch - they are priceless - never too big a log file when debugging -- speaking for myself. My favorite thing is a NEST COUNTER.. and my logs indent the nest level as periods "." so I can see when routines are entered and exited. (I don't use the integrated environment cause I can never get it configured right... I'm a command line junkie)

3: Keep it simple Silly Principle. Advanced methods not working? Resort to something you KNOW works (like the little demo) and build from there. It's often better to go around a mountain than climb the thing. Even though I'm always up for challenges - sometimes ya just need to (Maybe you've heard the famous comedian "Larry the Cable Guy" say this now famous slang term: "Git-R-Dun" (Get it Done ;)

4: Threading IS NOT always the best solution. This has been mentioned in a Lazarus WIKI I read. I'm building multi-user web applications and database stuff etc. So its great for me. If I was making a FTP server like FileZilla
- I would do it also - even for the client - separate threads downloading different files at once... But frankly - as many know - sometimes alternatives can actually be faster - like multiplexing - or well managed messaging systems - where program gathers work to do in a loop, then processes each little task in order (or priority) one at time - ... depends on the application however.

Have a great day all I have a ton of actual programming to do today - I love my work - I love Free Pascal Too... I'm such a fan. Jason P Sage

Some rambling observations:

I think your issue was possibly related to misuse of API. Certainly, on the WIN32 platform, unless you use the WaitForXXXXX or Sleep functions, you'll end up with serial processing because your task is too processor intensive and will starve the other process of any time slices. The scheduler can't context switch as easily if there is a contiguous task taking up processor time, and if you're doing things that can't be interrupted (e.g. copying large chanks of data from one file to another) it can't always (ever?) interrupt the task safely.

As mentioned before, use Events, Sepahores and such to create a "state machine" alike. This will then break the work up into chunks, hopefully.

Multithreaded programming is a completely different paragdim to standard single threaded programming.

I create my threads like a thread pool, I create them all suspended. (Because that parts works pretty easy)...

Okay, that will work.

Eeek!! Suspending and resuming threads on Win32 is extremely unreliable IMO. Far better to have an event (as in win32 API, not Delphi "Event"), have the thread wait on it being signalled (e.g. WaitForSingleObject) and then restart the thread at that point!!

Suspending a running thread is a great way to cause a Deadlock too.

Creating threads suspended is a very good idea. This way you can start them in a specific sequence. If you create them as the thread object is created, there is a chance you'll create a race condition because of the potential lag of the data initialization with in your TThread descendent.

Stay away from Critical Sections. They are a complete hack IMO. They breed logic races. You'll end up debugging weird deadlocks because one thread is waiting to enter a piece of code in a critical section whilst another is in the critical section waiting on the first to complete the task it is waiting on the second to exit the critical section on. And that is just with two threads ;-) Use PostThreadMessage for comms, and don't share non essential data between threads. Create a multiread single write mechanism for your stored shared data.

Now - to handle the ABSOLUTELY MUST allow main app to talk with thread etc.

Use a locking mechanism... Create two methods:

function Lock(const timeout: cardinal): boolean;
procedure Unlock;

make Unlock only Unlock the locker if the lock is owned by the calling process. Make the Lock attempt to lock for up to the timeout (miliseconds) and if the lock fails handle that. You then can gain exclusive access tot he thread.

All you then do is make the thread use the locker whenever it accessed the shared data.

I suggest using a Semaphone in win32, though I guess a Mutex would potentially work too. Using a semaphore you can extend at a later date to allow mutiple locks.

HTH

M

There is no bug. The time slice management is simply different. Try attached demo (it's your demo modified). You will notice that output will be mixed.

Linux simply gives longer timeslices to threads (for efficiency reasons).

Micha

Graeme Geldenhuys wrote

The Sort Demo, which is a port of the Threads demo included with Delphi 7, shows this clearly. I can archive and post the source, but

Try inserting some Sleep(0) to force the kernel to switch to another thread ? In real worker threads you wouldn't do this of course, but here just for the visual effect.

Micha

Thanks Jason - I don't recall you mentioning what platform you are working on, but I gather in is some *nix platform. The only thing you just proofed with your post is - threading is not working correctly!

There shouldn't be a need for workarounds like suspending threads to get others to work. The whole point of threads, is let them run all at once in parallel. Delphi does it, Kylix does it, FPC under win32 does it! Linux doesn't! Instead of figuring out a workaround, rather submit a bug report with an example.

My apps follow the fire and forget method and it works 100% under Delphi and Kylix, but since I moved to FPC, this is causing more and more headaches. Take on of my apps as an example. I start my app, and get to the login prompt. While I am logging in, the app fires off a thread to populate the lookup tables from a database, another thread loads the application/system settings. I also have another thread that does logging - to file, to console, to a log window or any combination of the three. Each logging type is another thread, and letting them run is serial is not a solution. Neither is playing round-robin with all those threads to suspend one, to get the next one in the queue to run. How does that take advantage of multiprocessor pc's or hyper threading or the new Core 2 Due processors? It doesn't.

Make no mistake, I love Free Pascal too and it offers a lot for a great price. All my previous posts, was just me trying to debug the threading and pinpoint the problem. All to make it easier for the person that knows the internals of FPC to fix something like that. I am not a Compiler developer, I'm a Application developer. ;-) After all my tests, I now know it is a bug, and submitted a report on Mantis with two demos.

Now the interesting part is how did I get started with all this threading business under FPC.? :-)
I needed to port a Kylix app to FPC and wanted to take advantage of threads as the app lends itself perfectly for threading. I thought to test the qualify of FPC threading support, I would first port one or two of the top 10 threading applications that was submitted to Borland in a Threading Contest which Borland held in 2002. Search Borland's CodeCentral or I could post the link. There are some good threading examples available. :-)

FWIW, maybe only to take the general suspicion from unix-like systems:

On FreebBSD 4.11 I ran the GUI-threading demo posted by you without any problem in the expected quasi-parallel manner.

Marc

I have troubles searching. Please, post the link (to the list of rated submissions or something).

Ah, finally some explanation that makes sence! Thanks Micha, your modified demo does in fact make Linux switch between threads. I also tried the Sleep(0) you mentiond in the Sort Demo. Sleep() didn't do the trick, but using the BurnCPU() method from Demo1 did.

Where can I read more about the timeslices to threads under Linux. It obviously works vastly different to Windows (and FreeBSD as someone pointed out). This sounds like an interesting topic and something work understanding. After all, it gave me enough headaches and a false positive - thinking it is a bug.

Sorry for all the trouble... :-)

Regards,
- Graeme -

Try the following:

Aboute the Contest
http://bdn.borland.com/article/28448

Fun with Threads
http://www.thedelphimagazine.com/samples/1712/1712.htm

The winners
http://bdn.borland.com/article/0,1410,29786,00.html

Regards,
- Graeme -

Graeme Geldenhuys wrote

Ah, finally some explanation that makes sence! Thanks Micha, your modified demo does in fact make Linux switch between threads. I also tried the Sleep(0) you mentiond in the Sort Demo. Sleep() didn't do the trick, but using the BurnCPU() method from Demo1 did.

Ok, maybe Sleep(0) in sysutils has to use the sched_yield syscall for linux. This may be a feature request to be reported in bug tracker IMHO.

http://kerneltrap.org/node/517/print is an interview with one of the authors, Ingo Molnar. Search for 'timeslice length' if you don't want to read it all :-).

Micha

Graeme Geldenhuys wrote

pointed out). This sounds like an interesting topic and something work understanding. After all, it gave me enough headaches and a false positive - thinking it is a bug.

Btw, here is a quote from the interview, default length seems to be 150ms: (it also explains the usefulness of longer timeslices: caches)

<quote>
another property of SCHED_BATCH scheduling is the use of much longer timeslices. Eg. right now it's 3 seconds for a default priority SCHED_BATCH task - while normal tasks have 150 msec timeslices. For things like numeric calculations it's good to have as long timeslices as possible, to minimize the effect of cache trashing. Eg. on a sufficiently powerful CPU with a 2 MB L2 cache, the 'population time' of the cache can be as high as 10 milliseconds. So if there are two numeric calculation tasks that both fully utilize the L2 cache (in nonobvious patterns), and which context-switch every 150 milliseconds, then they will waste 10 milliseconds on cache-rebuilding in the first 6% of their timeslice. This shows up as a direct 6% slowdown of the numeric calculation jobs. Now, if SCHED_BATCH is used, and each task has a 3000 milliseconds timeslice, then the cache-rebuild overhead can be at most 0.3% - a far more acceptable number.
</quote>

Micha

Ugly. Better define a procedure TryYieldTimeSlice() or so. No need to follow Windowisms with archaic reasons.

Yes agree, thought of this just after sending the email. The implementation might/will call Sleep(0) on the windows platform.

Micha

Greetings, Graeme.

I think I see the problem. On today's fast machines, a count of 1000 just isn't enough processing for a meaningful test, at least not under Linux. I really don't know much about Windows, but my conjecture is that perhaps under that system the function calls to write the output cause the other thread to receive processing time.

The following modified version of your program shows that threads work under Linux. The execute loops have been modified to continually count up/down (as appropriate) until terminated. The RunNow procedure was modified to let the threads run for three seconds before terminating them itself, and the FThreadCount thing was taken out (along with the OnTerminate handlers).

To properly see the output, you should redirect it to a file, unless you really have a LOT of scrollback buffer set up. :-) On my Athlon XP 2200 I get a file 35.5 megabytes in size!

~/tmp $./demo1 > demo1.txt
~/tmp $ls -l demo1.txt
-rw-r--r-- 1 pcervasio users 39566886 2006-10-02 12:31 demo1.txt

After looking at the contents of demo1.txt, I can see that the increment thread actually got to its eleventh count up before the decrement thread got its first share of time. This should help explain why in your original program it appeared that one thread was executing to completion before the other... it WAS, but only because there wasn't enough to do.

The above was done using version 2.04 of the compiler on a Slackware 10.1 machine with a 2.4.32 kernel. I appear to get similar results on my 2.6.17.6 box after copying the executable over (running an Athlon 2600). The decrementor first shows up after incrementor loop 15. The redirected output file is 53 megabytes, though... much bigger than I expected from the machine speed difference alone.

I hope this is helpful.

Best regards,
Pete C.

---------------------------------------

program demo1a;
{$mode objfpc}{$H+}
uses
  {$IFDEF UNIX}
  {$IFDEF UseCThreads}
  cthreads,
  {$ENDIF}
  {$ENDIF}
  Classes, SysUtils;
type
  // counts up till 1k until terminated
  TIncrementer = class(TThread)
  protected
    procedure Execute; override;
  end;
  // counts down from 1k until terminated
  TDecrementer = class(TThread)
  protected
    procedure Execute; override;
  end;
  TRunThreads = class(TObject)
  private
    t1, t2: TThread;
  public
    constructor Create;
    procedure RunNow;
  end;

{ TRunThreads }

constructor TRunThreads.Create;
begin
  t1 := TIncrementer.Create(True);
  t1.Priority := tpLower;
  t1.FreeOnTerminate := True;
  t2 := TDecrementer.Create(True);
  t2.Priority := tpLower;
  t2.FreeOnTerminate := True;
end;
procedure TRunThreads.RunNow;
var
  donetime: TDateTime;
begin
  { run for 3 seconds }
  donetime := now + encodetime(0, 0, 3, 0);
  writeln('RunNow');
  t1.Resume;
  t2.Resume;
  repeat
    sleep (100);
  until now > donetime;
  t1.terminate;
  t2.terminate;
  sleep (10); { give threads a chance to end }
  WriteLn('All threads completed!');
end;

{ TIncrementer }

procedure TIncrementer.Execute;
var
  i, j: integer;
begin
  j := 0;
  while not terminated do
  begin
    writeln (ClassName, ': --- Loop ', j);
    for i := 0 to 1000 do
    begin
      if terminated then break;
      Writeln(Classname, ': ', i);
    end;
  end;
end;

{ TDecrementer }

procedure TDecrementer.Execute;
var
  i, j: integer;
begin
  j := 0;
  while not terminated do
  begin
    writeln (ClassName, ': --- Loop ', j);
    for i := 1000 downto 0 do
    begin
      if terminated then break;
      Writeln(Classname, ': ', i);
    end;
  end;
end;
var
  lRunThreads: TRunThreads;
begin
  lRunThreads := TRunThreads.Create;
  lRunThreads.RunNow;
  writeln('Done...');
end.

---------------------------------------

Thanks Pete,

Even though Micha did manage to explain why my demos ran as they did, I will try you demo anyway. Purely to see how big the output file grows. :-)

Quick recap from Micha. The time slicing in Linux is much larger that under Windows. This improves performance (especially on calculations). See the link he posted for a full explanation. I modified my Sorting Demo and added three more Bubble Sort threads. Incremented each threads workload with a lengthy calculation. This did the trick to show the time slicing in action. The first sort thread fired and worked for about 3 seconds, then the others kicked in. :-) Very different to Windows.

Regards,
- Graeme -