I've had the "honor" of looking at current TThread/pthreads/cthreads implementation in unix (FreeBSD to be precise) and found it extremely bad.

Not to criticize, I'm here to ask for permission/toughts on adding a few standard methods to TThreadManager.

I would like to add semaphore functions to it, to be able to use them in Classes/TThread implementation. The reason for this is, that Classes is compiled BEFORE pthreads and as such I cannot use pthreads directly (and shouldn't for linking issues to libc) to implement threading in TThread.

I see basicly 3 approaches:

1. add semaphore/mutex functions to TThreadManager record
2. add a new "semaphores" record
3. move the TThread - threading specific functions out of tthread and into a "TThreadTThreadManager" kind of thing. Eg: "start thread with this callback" and such which are currently used in tthread.inc

Not sure which one is best..

Another thing.

I noticed that FreeBSD has 3 implementations of pthreads.

libpthread, Libc_r (reentrant) and "libthr" (1:1 kernel threads).

Currently fpc does some sort of automagical "descision" between libc and libpthread. I think users should be able to specify this somehow.

I've cleaned up unices I know off old {$IFDEF Linux} but I also want to add proper threading and remove the idiotic pipe/semaphore hack.

Ales

Op Mon, 9 Oct 2006, schreef Ales Katona:

Go ahead. I noticed this as well, we need proper semaphores and the logical place to add them is the thread manager.

Daniël

Ales Katona wrote

I've had the "honor" of looking at current TThread/pthreads/cthreads implementation in unix (FreeBSD to be precise) and found it extremely bad.

About the TThread: this is known.

We should have one (and only one) TThread implementation which is OS-Independent, and which uses ONLY thread functions from the system unit. (and hence from the threadmanager)

Permission granted (if it is mine to grant), provided you add it for all platforms which currently have threads.

I see basicly 3 approaches: 1. add semaphore/mutex functions to TThreadManager record

This is best.

2. add a new "semaphores" record

They will not be used independen of TThreads, so I see no use for this.

3. move the TThread - threading specific functions out of tthread and into a "TThreadTThreadManager" kind of thing. Eg: "start thread with this callback" and such which are currently used in tthread.inc

Nono, just expand the current thread support.
Although I'm surprised you need to add stuff, in principle it should have everything to work for TThread...

The linker currently decides this, and there is no way to 'specify' this except of dynamically loading the library, which is kind of a loop, since libc must be loaded in order for dynlibs to work.

I've cleaned up unices I know off old {$IFDEF Linux} but I also want to add proper threading and remove the idiotic pipe/semaphore hack.

Absolutely. In principle, TThread should be implemented in a system-independent way. The system unit thread support should be made complete enough to enable you to write TThread in a system-independent way.

Michael.

I've implemented the semaphores stuff (with a big bad bug, btw thanks Jonas for fixing).

Right now I'm thinking about how to go next.

The choices I see right now are:

1. Basicly just clean the current pthreads implementation, make sure types are not redefined in baseunix/unixtypes and pthreads etc. and be happy.

2. Add proper "interface" style pthreads with chosable (how is a question) backend lib (eg: freeBSD has 3 libs to which a pthread interface can link).

There's also a choice #3 which is a complete threading design overhaul, but since that'd break old code I don't even consider it to be an option (I actualy do but I know you wouldn't).

There's also still the hackish NPTL vs processes linux situation in TThread 2.4 vs 2.6 kernel. Not sure if it's considered "clean enough" now that pipes are gone or if that should be somehow addressed as well.

If the linux tthread.inc could be cleaned, I'm 100% sure that tthread.inc could be merged into one file and put into rtl/unix.

Write your ideas on the subject please.

Ales

Op Mon, 16 Oct 2006, schreef Ales Katona:

Write your ideas on the subject please.

Short answer: Kick pthreads and use kernel threads.

Daniël

I would only do that once the pthreads stuff is overhauled, since it requires separate and thus extra testing and maintenance.

Jonas

That's a nice idea but there are a few problems.

Kernel threads for example in freeBSD require a "scheduler" in userlevel along with other userlevel stuff which is normaly implemented by one of the pthreads libs. This means we'd have to implement a full blown threading lib.

I already added kse_ syscalls in freeBSD, if we want to go this route, these can be used to add proper SMP support to our threads, but I think using pthreads would be more sane.

Daniël _______________________________________________ fpc-devel maillist - http://lists.freepascal.org/mailman/listinfo/fpc-devel

Op Mon, 16 Oct 2006, schreef Michael Van Canneyt:

Sure, why would we remove the "cthreads" unit. A kernel thread implementation can be done with Linux 2.2+ requirements.

Daniël

What do you mean with this ?

There's also a choice #3 which is a complete threading design overhaul, but since that'd break old code I don't even consider it to be an option (I actualy do but I know you wouldn't).

If you keep everything working, I don't see why this would be a problem.

That is the minimum which should be done, IMHO.

In fact, it should go in rtl/objpas and be independent of the OS. it should only use TThreadManager functions...

When porting to a new OS, only TThreadManager should be implemented with a suitable implementation. That is why TThreadManager was introduced.

Michael.

They only exist on newer kernels, so at least the current implementation should be kept as an option for the user !

Michael.

Op Mon, 16 Oct 2006, schreef Ales Katona:

A scheduler is normally for userspace threads? I thought FreeBSD finally had kernel threads with 5.0?

I already added kse_ syscalls in freeBSD, if we want to go this route, these can be used to add proper SMP support to our threads, but I think using pthreads would be more sane.

I don't. Tthread doesn't fit well on the Posix thread model. At least for Linux, the kernel does not expose Posix threads, those are already emulated over the Linux thread model. In other works, pthreads results in subpar performance, and an emulation on top of emulation doesn't sound so good for robustness either.

Daniël

Daniël Mantione wrote

In other works, pthreads results in subpar performance,

Is the overhead of a few user level routines really that big? Once the threads are setup, they automatically become kernel threads anyway. Having a user level layer in between might even help in some cases (like e.g. if the threading subsystem decides that a spinlock is more appropriate in a particular situation based on the semaphore's properties).

Jonas

Op Tue, 17 Oct 2006, schreef Jonas Maebe:

If I compare my implementation of the Chameneos benchmark with the one from Marc (which uses Pthreads directly), mine is about two times slower. This is propably caused that our thread functions often require multiple Pthread calls, which might result in multiple kernel calls. So it is mainly the synchronisation that is the issue.

Of course, if the threads are running independend code with no need for accessing shared resources, it all depends on the kernel scheduling and there should be no significant overhead. In practise, this is seldom the case.

A spinlock is mainly fast in the case the lock is not held, in such case no kernel call is necessary. If the lock starts spinning, on uniprocessor it won't be released until the kernel schedules the other thread. This exactly the idea behind kernel futexes, if the lock is not held, no kernel call is done. If the lock is held, a schedule is necessary anyway, so the kernel is called. The kernel is in a much better position in this case, because it exactly knows which threads it should not schedule.

On SMP it might be the case both threads are scheduled at the same time, and here a spinlock might prevent a kernel call altogether. However, even here nobody has ever beaten the futex system in performance.

Daniël

Where? I only see this in starting/initialising/stopping/destructing things. The actual locking etc use one pthread call in all cases afaics. Of course, starting a Pascal thread requires initialising all the threadvars etc, but that is not going to change if you use something else than pthreads.

Jonas

Op Tue, 17 Oct 2006, schreef Jonas Maebe:

For example:

procedure intRTLEventSetEvent(AEvent: PRTLEvent);
var p:pintrtlevent;
begin
  p:=pintrtlevent(aevent);
  pthread_mutex_lock(@p^.mutex);
  pthread_cond_signal(@p^.condvar);
  pthread_mutex_unlock(@p^.mutex);
end;
procedure intRTLEventStartWait(AEvent: PRTLEvent);
var p:pintrtlevent;
begin
  p:=pintrtlevent(aevent);
  pthread_mutex_lock(@p^.mutex);
end;
procedure intRTLEventWaitFor(AEvent: PRTLEvent);
var p:pintrtlevent;
begin
  p:=pintrtlevent(aevent);
  pthread_cond_wait(@p^.condvar, @p^.mutex);
  pthread_mutex_unlock(@p^.mutex);
end;

I'm not 100% sure yet, but I believe you can implement a set-event with a FUTEX_WAKE, and a waitfor with an unconditional FUTEX_WAIT, which would mean a lot less calls.

Daniël

This was exactly the reason why I choose to use pthreads directly. For the given situation one single semaphore call could replace this.

Marc

Marc Weustink wrote

This was exactly the reason why I choose to use pthreads directly. For the given situation one single semaphore call could replace this.

Well, events semaphores :-). We need semaphore abstraction as well for "proper" RTL, besides events.

Micha

Op Tue, 17 Oct 2006, schreef Micha Nelissen:

Yes, but Tthread makes a lot use of both semaphores and events, so we need fast implementations for both.

Daniël

thats why I wrote: for the given situation ;)
It was for the cheap-concurrency benchmark. Later it turned out that a semaphore was more suitable that I was used to with (windows) events.

We need semaphore abstraction as well for "proper" RTL, besides events.

Eh, yes, that would be nice.

Marc

Daniël Mantione wrote

no kernel call is necessary. If the lock starts spinning, on uniprocessor it won't be released until the kernel schedules the other thread. This exactly the idea behind kernel futexes, if the lock is not held, no kernel

Aren't futexes 2.6+ only ?

Micha

This last pthread_mutex_unlock does not make sense to me. From the pthread_cond_wait man page:

The pthread_cond_wait() function atomically unlocks the mutex argument
and waits on the cond argument.

So the mutex should already be unlocked afterwards. Further, what is the lock/unlock for in the intRTLEventSetEvent? pthread_cond_signal can perfectly deal with multiple threads simultaneously signalling the same condition variable (pthreads is a multithreading api, not a random non-reentrant piece of code).

Jonas

Jonas Maebe wrote

This last pthread_mutex_unlock does not make sense to me. From the pthread_cond_wait man page:

Read the man page *completely*.

A condition variable must always be associated with a mutex, to avoid the race condition where a thread prepares to wait on a condition variable and another thread signals the condition just before the first thread actually waits on it.

and:

Before returning to the calling thread, !pthread_cond_wait! re-acquires |mutex| (as per !pthread_lock_mutex!).

Micha

If you would have read a couple of lines further you also would have found:

|Before returning to the calling thread, pthread_cond_wait re-acquires |mutex (as per pthread_lock_mutex).

Oh, is it? Read the f*cking man page:

|A condition variable must always be associated with a mutex, to avoid |the race condition where a thread prepares to wait on a condition |variable and another thread signals the condition just before the first |thread actually waits on it.

Vinzent.

That is not on the Mac OS X man page for pthread_cond_wait. See http://www.hmug.org/man/3/pthread_cond_wait.php

Again, this is not on the Mac OS X man page:
http://www.hmug.org/man/3/pthread_cond_signal.php

Not sure if this means it's not necessary in the Mac OS X (and possibly FreeBSD) versions, or that the Mac OS X man pages are incomplete.

Jonas

Jonas Maebe wrote

Not sure if this means it's not necessary in the Mac OS X (and possibly FreeBSD) versions, or that the Mac OS X man pages are incomplete.

It does say: The pthread_cond_signal() function unblocks one thread waiting for the condition variable cond.

By implication this means you need a mutex to protect against race conditions. Those manual pages are incomplete/too concise I would say.

Micha

[ Charset ISO-8859-1 unsupported, converting... ]

FreeBSD text for cond_wait:

DESCRIPTION
The pthread_cond_wait() function atomically blocks the current thread waiting on the condition variable specified by cond, and unblocks the mutex specified by mutex. The waiting thread unblocks only after another
thread calls pthread_cond_signal(3), or pthread_cond_broadcast(3) with the same condition variable, and the current thread reacquires the lock on mutex.

The last line seems to say the same as Linux.

Not necessarily, it is at least possible to implement an atomic linked list without requiring a mutex-style lock.

Those manual pages are incomplete/too concise I would say.

That indeed seems to be the case.

Jonas

Jonas Maebe wrote

Not necessarily, it is at least possible to implement an atomic linked list without requiring a mutex-style lock.

... using the atomic primitives of the PPC processor. Maybe it's not possible in general...

Jonas

I see.

Not sure if this means it's not necessary in the Mac OS X (and possibly FreeBSD) versions, or that the Mac OS X man pages are incomplete.

They are most definitely incomplete.

Vinzent.

That's irrelevant. If thread 1 lets thread 2 do something, and thread 2 would signal thread 1 after completion, then thread 1 might not have started to wait yet (descheduled just before cond_wait). Could happen for example with TThread.Synchronize.

Windows events do not have this problem since they are stateful. In hindsight, a semaphore might have been a better emulation of an 'Event' (although I've not thought this through completely), but at the time condition variables seemed to be the logical way to go.

Micha

Micha Nelissen wrote

Windows events do not have this problem since they are stateful.

To be more precise: Windows signals are persistent, not transient like Unix signals are.

Vinzent.

Op Tue, 17 Oct 2006, schreef Micha Nelissen:

They are. For backward compatibility you use the semctl API. This is not problematic, pthreads uses semctl for <=2.4 as well.

Daniël