In another topic (now closed) Andrew described that a code similar to HansPeter's example did run correctly on a dual core machine, but produced errors on a machine with more cores.

Now I understand that threaded FPC user programs are supposed to be done in a Posix compliant way and FPC/RTL/LCL provide Posix compatible MUTEX behavior e.g. with TCtriticalSection.

I understand that regarding threads and SMP, Posix (while not explicitly handling SMP) requires a user program to assume that unsynchronized code sequences in multiple threads can work truly in parallel (an indefinite number of cores needs to be assumed, priority does not enforce a sequential behavior).

Moreover regarding memory / cache synchronization Posix (not explicitly handling SMP and cache) requires the user program not to assume that any variable is set by another thread unless some synchronization ensures that the appropriate code sequences that sets it already has been executed..

OTOH if some synchronization ensures that the appropriate code sequences that sets it has been executed, IMHO, Posix compatibility of the infrastructure (compiler, libraries, OS, Hardware) ensures that another thread in fact gets that correct value of that variable (i.e. SMP Cache synchronization after the synchronization is granted).

in the other topic we found that
- my original "volatile" question is invalid, as the function calls done with list.lock and list.unlock are a "volatile barrier" preventing the compiler from caching some value inappropriately in a register, simply because they are function calls.
- the MUTEX operations done with list.lock and list.unlock (at least on PCs in Linux and Windows) use library calls that includes memory barriers
- if the potential cache incoherency would not be handled by Hardware / OS / Libraries on behalf of user land programs (not done with FPC), I feel that this would so disastrous and ubiquitous that it results in so many programs not working on SMP systems that it would be a really well known issue in the (C-) programmer community.

So, if Andrew's test case really fails, I think, there is some kind of bug _somewhere_ and I think it should be located to verify that it is not in the FPC compiler or the FPC RTL.

(It it fails only on Andrew's machine, my suggestion is a hardware problem.)

Who has a >2 Core machine to test and debug this ? -Michael

Michael Schnell wrote

In another topic (now closed) Andrew described that a code similar to HansPeter's example did run correctly on a dual core machine, but produced errors on a machine with more cores.

I've not been reading every message. Definitive URL?

So the really big question is whether any given computer has a robust SMP implementation, or if in actual fact it's some species of NUMA with unreliable cache coherency.

It might also be necessary to distinguish between SMP, multicore and hyperthreading.

(It it fails only on Andrew's machine, my suggestion is a hardware problem.) Who has a >2 Core machine to test and debug this ?

Biggest x86 system I've got here is a Compaq with 2x Xeons which I think are 2x multicore rather than HT. More interestingly I've got a 12-way SMP Sun, and they've been in this game for a while. Linux-only in both cases.

Topics:
[fpc-devel] volatile variables
and over in the lazarus mailing list:
[Lazarus] thread safe

While this of course is correct, this is not something a user program should know about. It the user program adheres to the Posix specs, I think the infrastructure (compiler, libraries, OS, Hardware) is supposed to correctly handle the dirty details.

Sound like interesting test cases. Andrew himself did not provide the code that failed. Maybe some information from the other topics might be helpful.

Thanks,
-Michael

2011/7/1 Michael Schnell :

I won't call that a definitive URL, but some vague indication. That is more than 200 'd expected you to give a link to the email in the archives where "Andrew described a code". So, which link to click on
http://lists.freepascal.org/lists/fpc-devel/2011-June/subject.html or which link to click on
http://lists.lazarus.freepascal.org/pipermail/lazarus/2011-June/subject.html ?

Vincent

I have no idea how to find a certain mail in the backlog. But it would not help anyway, as AFAIR, no additional information can be found there.
Maybe Andrew can jump in and give some hints. (I cc'ed him).

-Michael

2011/7/1 Michael Schnell :

So how you expect us to find the description *you* want us to read in all those mails, if even you cannot find it.

But it would not help anyway, as AFAIR, no additional information can be found there.

Oh, it was just another piece of useless information...

Vincent

Vincent Snijders wrote

So how you expect us to find the description *you* want us to read in all those mails, if even you cannot find it.

I can't find it in the backlog website. I did find it in my mailstore (no idea if this helps, though):

This is the message of Andrew's (06/28/2011 02:47 PM):

------------------------------------------------------------------------

This is the initial message of Andrew's (06/23/2011 02:02 PM):

Here he goes on (06/27/2011 04:58 PM):

(06/27/2011 08:03 AM):

(06/28/2011 01:41 AM):

(06/28/2011 04:41 AM):

Fair play, he /has/ said he's mailed Andrew looking for source.

However under the circumstances we've got to have something definitive to inspect and test- I for one would be very unhappy if I spent my time running up a rarely-used system and updating its compiler, only to be told that I didn't have the right version of the test program.

Mark Morgan Lloyd wrote

Fair play, he /has/ said he's mailed Andrew looking for source.

I never received such email. But to Michael's defense, Google mail isn't the product it once was.

However under the circumstances we've got to have something definitive to inspect and test- I for one would be very unhappy if I spent my time running up a rarely-used system and updating its compiler, only to be told that I didn't have the right version of the test program.

I did all of my research on memory barriers a while ago. The best site that describes this "phenomena" is
http://en.wikipedia.org/wiki/Memory_barrier

There are others here that helped walk me through this a while ago, but to me, this is a non-issue. I intend to use interlocked statements for mission critical memory assignments.

I would suggest the poster do more questions than suggest there is a problem with either FPC, POSIX, or pthreads implementations, or kernel bugs, or OOP principals regarding inter-thread memory access to LCL objects. Anyone designing systems should already know how to employ techniques.

I cannot provide an URL (from old Thunderbird), so let me cite the essential part of Andrew's message:

>

Simply protecting a loop with a futex is not sufficient. Assigning variables in a multi-threaded environment is not sufficient.

Interlocked/Exchange/ExchangeAdd/Increment features must be used whenever dealing with variables (with at least one exception) In my experience arrays of boolean are atomic.

I ran tests on a HexCore AMD system which had unexplained anomalies with pointers in an bidirectional linked list. I replaced all the pointer assignments with InterlockedExchange and the system worked flawlessly. Some of the assignments were taking place in a criticalsection (btw).
<<

This scenario was reflected in my example (bi-linked list update).

The very last sentence deserves clarification. When not *all* assignments are protected by a CS, the use of Interlocked assignments only can improve cache coherence, but it IMO *can not* prevent (logical) race conditions. IMO Andrew observed just such race conditions, resulting from a mix of sequential writes from multiple threads.

IMO a linked list can not work flawlessly, until at least a multiple-read-exclusive-write lock is used. While an exclusive-write lock can ensure list consistency, every unsynchronized reader will be fooled by concurrent changes to the list structure :-(

Consider what will happen when an (unsynchronized) reader determined an list node, before or after which a new node shall be inserted. When the write-synchronized update will happen, the list structure already can have changed, so that the referenced node can have changed its position in the list, or can have been moved into a very different list :-(

DoDi

Hans-Peter Diettrich wrote

IMO a linked list can not work flawlessly, until at least a multiple-read-exclusive-write lock is used. While an exclusive-write lock can ensure list consistency, every unsynchronized reader will be fooled by concurrent changes to the list structure :-(

I agree. Without a memory barrier, coupled with interlocked exchanges per node attachment, it would make memory synchronization impossible.

Impossible to comprehend, that's why we have barriers. But in one specific case, Interestingly enough - the problem I experienced was with the 1 owning thread accessing the list. There was no subsequent access via additional threads, but across cores on a single cpu.

Hello FPC,

Friday, July 1, 2011, 6:44:44 PM, you wrote:

HPD> This scenario was reflected in my example (bi-linked list update). HPD> The very last sentence deserves clarification. When not *all* HPD> assignments are protected by a CS, the use of Interlocked assignments HPD> only can improve cache coherence, but it IMO *can not* prevent (logical) HPD> race conditions. IMO Andrew observed just such race conditions, HPD> resulting from a mix of sequential writes from multiple threads.

When not all assignements/reads are not protected by a CS (or similar) there is not a problem of coherence, there is a logical problem. In that case the use of interlocked operations only add overhead without adding any kind of security/stability.

You can write lock free algorithms with interlocked operations, but you can not write any other kind of procedures, usually more complex, without a CS.

Tiny example:

Core/Thread 1:
--------------------
while true do begin
  Critical.Enter;
  a:=b;
  b:=b+1;
  if a=b then Raise Exception.Create('KBOOM!');
  Critical.Leave;
end;
Core/Thread 2:
-------------------
while true do begin
  InterlockedDecrement(b);
end;

This code will crash at a given time, maybe 1 millisecond, maybe 2 days, but it will crash.

Cache coherence is maintained by the hardware, interlocked only provide atomicity. How the atomicity is provided is beyond the scope of pascal and even the OS.

José Mejuto wrote

This code will crash at a given time, maybe 1 millisecond, maybe 2 days, but it will crash.

Yes. It will fail. Access to a,b are forbidden without acquiring a lock.

Cache coherence is maintained by the hardware, interlocked only provide atomicity. How the atomicity is provided is beyond the scope of pascal and even the OS.

IMO, this statement is pretty bold/broad. Meaning interlocked assignments as defined will ensure cache coherence as apposed to CrticialSections which are meant to minimally block *write* access if not read/write access.

I can tell you that a,b in my case were not so trivial. a,b happen to be reference pointers in a collection that without going into too much detail were certainly paged since the collection they resided in was large.

Hello,

This thread is now also moderated. See http://lists.freepascal.org/lists/fpc-other/2011-July/000637.html for the reason why.

Please start/continue any discussions about how to safely program in a multithreaded way on the fpc-other list (unless it's about concrete proposals for language extensions like the topic Hans-Peter started, concrete bugs in the compiler/RTL, etc).

Thanks,

Jonas
FPC mailing lists admin