Hi,

I normally do not agree with anything Microsoft does, but at least this time it looks like they are doing something worth while, or at least steering in the right direction.

Anybody that's ever programmed multi-threaded applications knows it's a daunting task and there is a lot as scope for errors. With all these new multi-core processors coming into the market, application developer need to starting shifting their mindset from serial programing to parallel programming. We all know that's no easy task!

Microsoft is trying to ease that shift for developers by trying to come up with ideas to help the developers build multi-threaded apps. It would be interesting to see what they come up with.

http://www4.osnews.com/story/18993/Parallel_Computing_with_.NET

I definitely agree with their idea... Developers will have to shift their mindset from serial to parallel thinking to take advantage of multi-core processors. Otherwise we will waist a lot of those other CPU cores - all sitting in idle. IBM got it right many years back with the design of OS/2 - now lets see if the rest of the world can follow....many years later.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

I definitely agree with their idea... Developers will have to shift their mindset from serial to parallel thinking to take advantage of multi-core processors.

That is an open door that has been kicked in by all vendors. The problem is how to implement it without making formerly "bulk software" a specialty to be made by higher skilled programmers?

I don't have time to plough through all this, but what do they really offer, beyond the usual "process this array in paralel" primitives that are as the road to Rome?

Marco van de Voort wrote

That is an open door that has been kicked in by all vendors. The problem is

I kind-of understand that... Borland tried to encapsulate the whole threading thing in a TThread class. It made it easier and every bit helps! I simply thought I should mention it here, so that FPC developers know about this new API. I haven't had a look at their API docs yet, but thought if they come up with some neat idea not done before, maybe FPC could borrow some of those ideas..... Piggy-back on Microsoft's research findings.

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Yeah, right. And back in 1983, Ada already tried to encapsulate that in a task object.

If any of the FPC developers had time to implement something more nifty, there would have been plenty of time to come up with something. Quite a while ago there were even some ideas shared for the OMP API etc. pp.

I can't imagine that Microsoft's ideas of adding multi processing to dot-net will change the overall picture. Especially not when I think of all the parallel language( construct)s invented in the past 30 years to accomplish just that.

Vinzent.

I kind-of understand that... Borland tried to encapsulate the whole

threading thing in a TThread class. It made it easier and every bit helps! I simply thought I should >mention it here, so that FPC developers know about this new API. I haven't had a look at their API docs yet, but thought if they come up with some neat idea not done

before, maybe FPC could borrow some of those ideas..... Piggy-back on

Microsoft's research findings.

Regards, - Graeme -

Hello,

If someone is truly interested in parallel computing - "lock free" datstructures are what realy counts -
We program multithreades servercode with pthreads, NPTL threads, windows threads now for around 6 years -

(- and programm in assembler, c, c++, delphi, freepascal since 21 Years (PDP 11,C64,C167,ARM uC,8051 uC,PIC uC,Amiga,x86)
- in all that time object pascal was the most effective language in terms of error probability and turn around times ...)

It is a immense hassle to debug parallel problems like deadlocks, priority inversion and the like.
So for really cooperative porgramms the design phase on paper should be taken into account even more.

The best solution for this problems by now is IMHO the lock free paradigm - but has to be judged on the problem.
Several doctoral thesis, research findings, and papers for this exist - if someone is interested -
join the ACM, and the ACM Digital Library (http://www.acm.org/)
- and the technique is industry recogniced - you maybe don't know FirmOS
- (my firm) but you maybe
know "Xen" (a cambridge project) which is now owned by Citrix, or you maybe know "Half Life 2"
from Valve (which was computer game of the year, i think in 2006) - they uses this techniques extensively.
(The Xen project, is IMHO the first project which uses this technique practical and with immense success)
For Java and a were short primer/starting point read here: http://www.ibm.com/developerworks/java/library/j-jtp11234/

Also "Open-MPI" sounds very interesting. But i fear a "no brain solution API" for developers does not exist -
and without digging deeper into the problems of that field someone would have many hours of time searching
bloody phenomenons and bugs ....

So my personal findings are - nearly every technique "Microsoft" invented, buyed or acquired was cancelled some
years later just to bind the customers by inventing something revolutionary new, so a good stomach decision is
not to bind yourself too thight to this firm -
(Access Runtime for Developers, WinG, DirectX Hardware support for sound cards, the QoS Api (different for W2000, XP and Vista) are all examples where i personally got a bloody nose running into a wall :-(

I hope i could share some experience,

helmut

There are implementations of "parallel" and "sequential" sections in some programming languages. Of course FPGA description languages like VHDL or Verilog do this from ground up, but there are also C derivate s that allow for parallelism in appropriately defined sections of code. I suppose it's not very difficult to define keywords for "parallel" and "sequential" sections for Delphi language. Of course the implementation (that would need multiple threads assigned automatically) is a decent challenge.

-Michael

I kind-of understand that... Borland tried to encapsulate the whole threading thing in a TThread class. It made it easier and every bit helps!

Agreed.

But while the current implementation is absolutely appropriate for many tasks, it _could_ be a lot more usable if in many other cases when using a different paradigm.

TThreads don't have an event queue and thus while in the main thread you do "event driven programming" (doing multiple "run to completion" code snippets that are fired by certain events), while the TThreads only can run as a "one shot" or in a cycle. Giving each thread an event queue could make the thread programming more transparent to the "average" programmer, as the same paradigm could be used in the main thread and the "worker threads"

An even more advanced way would be combining the paradigms of event driven programming and threaded projects by introducing "thread events". I.e. when an event is fired, it can use an additional thread (and thus an additional CPU) instead of running as a subroutine of the code that fires the event.

-Michael

Here is a start:
http://wiki.lazarus.freepascal.org/OpenMP_support

Mattias

Very true words (all of them, technical an political ones).

Multithreading (an thus multiprocessing which requires same) does not help a bit if the problems of protecting data accesses is not decently solved.

-Michael

There are already bindings for MPICH2 and afaik Daniël plans to create OpenMPI bindings.
I started a mpich tutorial here
http://wiki.lazarus.freepascal.org/MPICH and I hope to create something similar to MPE for profiling.
Every help to get FPC on HPC is welcome.

Mattias

Here is a start: http://wiki.lazarus.freepascal.org/OpenMP_support

Thanks for the link.

I'll add my "thread event" idea as another proposal.

-Michael

I've got limited time to follow up the links in that summary story, but I'd be interested to know what it purports to do that isn't done by either OpenMP or OpenMPI, both of which are fairly mature and are supported by- for example- gfortran.

CPU cores - all sitting in idle. IBM got it right many years back with the design of OS/2 - now lets see if the rest of the world can follow....many years later.

IBM implemented threads but as I understand it had no support for parallelisation across multiple processors, either SMP/NUMA or cluster. What's more I don't recall a mainstream language implementation from those days that had implicit support for starting custom threads, it was pretty trivial to do using a function call but there was nothing in the languages themselves.

Actually, there are two distinct issues here: MIMD which can be handled adequately by threads and SIMD which is where things like FORTRAN support for OpenMP comes in. I don't think that there is a single solution that will satisfy both camps.

.
.

Well, OS/2 supports SMP on up to 64 CPUs... If I remember correctly, the first version supporting SMP was OS/2 v2.11 dating back to July 1994.

Tomas

Tomas, I remember something like that as well. At that time OS/2 was miles ahead of anybody: 32bit desktop OS, a very well thought out and (I think) responsive desktop (Workplace Shell), true multi-tasking, usage of threads to help responsiveness etc...

I remember when I was actively part of Team OS/2, we did a simple demo of a directory listing running in a loop or a Find File (recursively). Drag the window, and the content keeps updating/scrolling (remember this was mid 90's). On a Windows XP computer (as ridiculous as it sounds) we still can't do that! Hold the mouse down while some hard drive action occurs and suddenly the hard drive is quiet.... The mouse blocks the process/OS!! Thank goodness Linux is okay in this regard. :-) [...but now I'm getting off topic...]

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Graeme Geldenhuys wrote

> . > Well, OS/2 supports SMP on up to 64 CPUs... If I remember correctly, the > first version supporting SMP was OS/2 v2.11 dating back to July 1994. >

A quick Google search confirms our thoughts. SMP was indeed in OS/2 2.11.

http://www.byte.com/art/9406/sec11/art2.htm
http://www.edm2.com/index.php/OS/2's_Symmetrical_Multiprocessing_Demystified http://www.os2bbs.com/os2news/OS2Warp.html

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Graeme Geldenhuys wrote

A quick Google search confirms our thoughts. SMP was indeed in OS/2 2.11.

OK, I'll gladly concede the point about SMP on OS/2, but the fact remains that SMP didn't really take off on PCs until around 2000.

I think the point stands however about poor language support for the concept in the fairly early days. However from the point of view of FPC I think we've got to work out whether there's scope for improvement of threads etc. (e.g. adding processor affinity support to the basic TThread class) or if something could be done to tell the compiler to spread a computation across multiple processors.

On a positive note, courtesy of a number of eBay transactions I've now got a reasonably big SPARC system specifically to look at this sort of thing and I can confirm that the tmt1 test- previously a problem on SPARC- runs and is distributed over all available processors.

Mark Morgan Lloyd wrote

I think the point stands however about poor language support for the concept in the fairly early days. However from the point of view of FPC I think we've got to work out whether there's scope for improvement of threads etc. (e.g.

That's exactly the point I was trying to make with this message thread. By implying that *if* Microsoft (with all their money and research) found a better way to use threads and multicore processors, maybe FPC could take some hints and implement it in RTL.

That's only if Microsoft came up with something new that the wonderful FPC developers haven't thought of yet. :-)

Regards,
- Graeme -

_______________________________________________
fpGUI - a cross-platform Free Pascal GUI toolkit
http://opensoft.homeip.net/fpgui/

Mattias Gaertner wrote

Here is a start: http://wiki.lazarus.freepascal.org/OpenMP_support

Thanks, interesting. I'll tack any thoughts I have into that page's discussion.

Interesting idea.

OpenMP is about lightweight threads. The 'thread events' proposal is the opposite: normal threads+events queues. Both techniques can live happily together, so I think, the thread events should get a page of its own.

Mattias

OpenMP is about lightweight threads. The 'thread events' proposal is the opposite: normal threads+events queues.

Of course you are right here.

IMHO the "parallel" idea is hard to implement, as the compiler needs to heavily support it. And it's completely outside of what a user expects from a Delphi language compiler. The implications of "parallel" sections might be hard to understand to the user of the compiler, especially if the language does not implement additional synchronizing methods. (Issues: ThreadVars versus normal Vars unprotected normal vars will fail, even a normal counter fails with multiple CPUs if the compiler does not create a bus lock instruction).

Nonetheless it's a good investment in the future, regarding multiple CPUs to become very common (and "light wight" threads in Linux supporting this a lot better than the Windows thread model that seems to impose a lot more overhead).

OTOH "Event Threads" is just an RTL enhancement and thus it's quite easily implemented. And "thread events" only adds a single "procedure of thread" keyword or slightly extends the implementation of "procedure of object". This should not be to hard to implement either. (Of course the user needs to take care of synchronizing issues as well.))

Both techniques can live happily together, so I think, the thread events should get a page of its own.

I can do this. But to me it looks like things in a wicki get lost very easily as they will only be found if someone explicitly searches for that issue. So doing another stand-alone article only makes sense when knowing decent locations where to place a link to same.

-Michael

Michael Schnell wrote

IMHO the "parallel" idea is hard to implement, as the compiler needs to heavily support it.

Nonetheless, this was what this wiki page was about: Proposals for parallel *language* constructs (especially WRT to the C-centric OMP spec).

BTW, I didn't have time to look at your proposal, but once I got "procedure of thread" you're quoting here, I'd expect the language to take care of (most) synchronization issues, much like the "protected object" in the Ada language does.

Vinzent.

Think about the alternative: It is much harder to implement the same parallel loop with TThread. So OpenMP makes parallel loops much easier to implement. For me this is the 'Delphi' way: Makes things easy and readable.

The 'of thread' is a language change, not just an RTL enhancement.

That's why you must add links pointing to the page.

About search optimization: The page is called OpenMP and this is what users see. They don't see Eventhreads.

Mattias

Mattias Gaertner wrote

Think about the alternative: It is much harder to implement the same parallel loop with TThread. So OpenMP makes parallel loops much easier to implement. For me this is the 'Delphi' way: Makes things easy and readable.

It's also very much a case of user expectation. I know of somebody who's writing some number-crunching stuff in VB: I suspect that like me he's aware of the parallelisation facilities in FORTRAN but learned to dislike the we can implement something that has the same
"shape" as other OpenMP-aware languages people might be interested, if it's radically different they won't.

I've taken the liberty of suggesting in
http://wiki.lazarus.freepascal.org/Talk:OpenMP_support that a "sequential" modifier would be useful as a guard around the parallel procedure. This might also be useful for the existing thread model.

BTW, I didn't have time to look at your proposal, but once I got "procedure of thread" you're quoting here, I'd expect the language to take care of (most) synchronization issues, much like the "protected object" in the Ada language does.

This is not part of the proposal. Here I suggested a quite simple RTL enhancement (providing Event driven threads "TEventThread") that _can_ be enhanced further with _some_ language support to provide "Thread Events". Maybe this again can be enhanced by language support for protecting and synchronization, but that is not mandatory. Of course decent OS independent sync Objects/procedures in the RTL should be provided anyway.

-Michael

Of course you are right. In the example of "parallel" loops It's _a_lot_ easier to to use for the programmer. "Lightweight-threaded" stuff like parallel loops was not the original aim of ThreadEvents. The original target was a more "standard" use of threads. But it _can_ be used for parallel loops, too, and it follows the "Delphi-language-paradigms" a lot tighter than using TThread.

I don't suppose that anybody will start implementing real parallel loops like suggested on the wiki page any time soon. But TEventThread and "Thread Events" is quite easy to do.

The 'of thread' is a language change, not just an RTL enhancement.

Sorry if I have been vague here. Of course it is (as is the alternative to enhancing the meaning of "of object"). But this is only another enhancement that can be done as a second step. Most of the functionality can be used (or at least tested) by just implementing TEventThread (and TMainEventThread) in the RTL (and maybe LCL).

That's why you must add links pointing to the page.

Of course. But where to add them ?

About search optimization: The page is called OpenMP and this is what users see. They don't see Eventhreads.

Of course you are right again: It's a good idea to do a separate page. I'll do that ASAP,

-Michael

Zitat von Michael Schnell :

The examples are very artificial to demonstrate the problems. Some real world examples / tutorials should be added on a new page. Parallel algorithms are seldom taught in books/schools/university, so programmers are not used to them. This is slowly improving.

Everywhere. :)

For example search for 'thread' and add at the bottom or in between a short sentence with a link, like 'Another approach is ...' or 'Threads can communicate via TEventThreads too, see ...'.

Mattias

By "implement" I mean the creation of the _FP_compiler_. Of course for FP _users_ it would be _great_ to be be able to use parallel sections and I do suppose there are very decent applications for this. But I don't suppose that anybody any time soon has enough spare time to update the compiler to be able to automatically create parallel code. That is why IMHO "thread events" are more likely to be born.

OK. I'll try to do this.
I had _hoped_ for a more automated feature of the wiki software :(.

-Michael

For me the whole OpenMP approach is really artifical, I don't see a real use in real world code for it honestly.

Op Tue, 11 Dec 2007, schreef Florian Klaempfl:

It is common in the scientistware I benchmark daily. The evil geniuses parallelize their applications by placing a some OpenMP hints in their source code.

However, MPI is de facto the standard for parallelization and this is most used. There are people who believe OpenMP is more efficient than MPI (directly reading from another threads its data is more efficient than sending messages between processes). However, modern MPI implementations use shared memory and RDMA between processes to counter this; I have seen little practical evidence that OpenMP is faster than MPI.

Daniël

For me the whole OpenMP approach is really artifical, I don't see a real use in real world code for it honestly

IMHO, it _would_ be nice to do a matrix multiplication like this (or according to proposal 2 with "parallel for"):

procedure MultMatrix(const mr, m1, m2: TMatrix;);
var
  ll: integer;
  procedure MultLine (l: integer); parallel;
  var
    i, j: Integer;
    sum: TMatrixElement;
  begin
    for i := 0 to length(m1)-1 do begin
      sum := 0;
      for j := 0 to length(m2)-1 do begin
        sum := sum + m1[l, j] * m2[j, i];
      end;
      mr[l, i] := sum;
    end;
  end;
begin
    for l = 0 to length(m1)-1 do begin
      MultLine(l);
    end;
end;

But as said, I suppose implementation of a compiler that can handle this might be too much work for anybody to start with that project.

-Michael

Yes, but only some loops. But just imagine how OpenMP could help in parallelization of the compiler? It can't imo.

But in fairness a compiler is neither more nor less specialist than heavy numbercrunching code. OpenMP parallel procedures should, as one particular simple example, be able to improve Mandelbrot speed significantly since there is minimal interaction between the areas being calculated.

There's lots more people running big finite difference or TLM models than are doing systems programming. Getting them interested in a language that is more friendly than C or FORTRAN would be to everybody's advantage.

I've got some cellular automaton stuff that I want to look at, and while explicit threads are probably the way to go having implicit parallelisation would be a very interesting alternative.

Op Tue, 11 Dec 2007, schreef Mark Morgan Lloyd:

Manual threading is not very comfortable, that is not the point. The point is, if we were to spend a lot of time on a parallelization feature, we want to spend it well.

OpenMP is just one way and has its limitations.

Daniël

True.
Maybe it would be good to first start the OpenMP library part - a unit providing some functions to easily use light weight threads.

Mattias

Maybe it would be good to first start the OpenMP library part - a unit providing some functions to easily use light weight threads.

I just read that as of v4.2, GNU C supports OpenMP by language constructs (seemingly preprocessor directives). When looking at the ALIOTH "shootout" website this seems to result in a serious disadvantage of FPC's.

For things like "Mandelbrot" an OpenMP based implementation of the "parallel loop" syntax (as suggested by Oxygen) seems like a perfect way to handle SMP.

-Michael

I don't see how Mandelbrot requires OpenMP. One can easily launch n threads depending on the number of the available cores.

I don't see how Mandelbrot requires OpenMP. One can easily launch n threads depending on the number of the available cores.

(We are discussing this in the German Lazarus Forum right now...)

I feel that OpenMP is never _required_ to optimize a problem, but just adds a high level of programming convenience to a language.

In fact the "Mandelbrot" test is not even done with threads (or OpenMP) in GNU-C. Someone else in the German Lazarus Forum said he found threading in some tests done in C and on none done in Pascal, leading to an unfair result on multicore machines.

But doing _explicit_ threads in FPC would not be allowed according to the rules in ALIOTH: they only allow simple programming and not complex tricks so that the source code for a test case is similar with among all languages. I suppose adding a "#pragma omp_parallel for" line in the C source or a "parallel" keyword in a for statement in FPC or Oxygen would be nice, while explicitly creating multiple threads would be denied.

-Michael

It solves none of the real problems of MT programming.

I don't care about this then :) IIRC the Mandelbrot benchmark is full of vector macro hacking, if they deny then proper thread starting they should better spent their time in usefull things.

Several years ago I had a look at OpenMP and I consider it as one of the most useless extensions for real world applications. It adds no benefit to the real problems of MT programming.

It solves none of the real problems of MT programming.

Neither does Pascal. We _can_ always do ASM if we want to :).

I don't insist on caring about the Mandelbrot example at all. (As said I found that same even is not one of the examples the poster in The German Lazarus forum was talking about, as here the C code does not use threads either.)

Several years ago I had a look at OpenMP and I consider it as one of the most useless extensions for real world applications. It adds no benefit to the real problems of MT programming.

I don't insist on looking at OpenMP either. I did not do any investigation on same myself. I just _thought_ that it would be a useful thing, as the GCC community has adopted it.

IMHO any technology that enables FPC to compile a loop like (using Oxygen syntax):

for parallel i := 0 to 10 do begin
  a[i] := a[i] + b[i];
end;

in a way that it on a multicore processor runs as fast as the appropriate GNU C construct:

#pragma ocm_parallel for
for (i=0; i<=10; i++) {
  a[i] = a[i] + b[i];
};

would be fine.

-Michael

Great and you really believe this accelerates a program? Starting a thread takes a lot of time and such loops are usually memory throughput bound.

Nice toy example without any real use.

Op Mon, 3 Nov 2008, schreef Florian Klaempfl:

While I largely agree with you, it's nice to show some numbers.

Let's do a comparison using Stream en an Opteron 2354 system. This is on 1 core:

---------------------------------------------------- Function Rate (MB/s) Avg time Min time Max time Copy: 5881.0033 0.0218 0.0218 0.0218 Scale: 5873.7327 0.0218 0.0218 0.0219 Add: 5594.5421 0.0343 0.0343 0.0343 Triad: 5521.5076 0.0348 0.0348 0.0348 ----------------------------------------------------

This is Stream on 8 cores:

------------------------------------------------------------- Function Rate (MB/s) Avg time Min time Max time Copy: 15748.1714 0.0073 0.0081 0.0081 Scale: 15767.5971 0.0073 0.0081 0.0081 Add: 15498.8812 0.0112 0.0124 0.0124 Triad: 15557.8681 0.0111 0.0123 0.0124 -------------------------------------------------------------

So we get a 3-fold speedup for an 8-fold increase in processing capacity. The reason is memory bandwidth is the bottleneck. On Intel the situation is much worse. This is a Xeon 5420 on Blackford. 1 core:

------------------------------------------------------------- Function Rate (MB/s) Avg time Min time Max time Copy: 3923.3765 0.0294 0.0326 0.0328 Scale: 3928.0555 0.0294 0.0326 0.0327 Add: 3929.8382 0.0441 0.0489 0.0490 Triad: 3944.2933 0.0440 0.0487 0.0489 -------------------------------------------------------------

8 cores:

------------------------------------------------------------- Function Rate (MB/s) Avg time Min time Max time Copy: 6117.1414 0.0189 0.0209 0.0210 Scale: 6116.7929 0.0189 0.0209 0.0210 Add: 6081.3180 0.0285 0.0316 0.0317 Triad: 6124.9809 0.0283 0.0313 0.0315 -------------------------------------------------------------

Daniël

Well, those tests even don't take care of thread starting time :) Taking advantage of MT requires always deep knowledge about the used architecture and the code being executed and this is something OpenMP ignores. For a big vector operation the number of used threads should be adapted to the memory architecture, for computational intensive applications like Mandelbrot the number of threads must be adapted to the number of available virtual cores.

Great and you really believe this accelerates a program? Starting a thread takes a lot of time and such loops are usually memory throughput bound.

OK, I should not have used 10 as the element but 1000000 :). And of course it makes more sense if a more complex calculation is to be done in each loop. But exactly this issue (lots of independent calculations in parallel) is a widely seen task (e.g points in physical fields).

Of course the RTL is not supposed to start a thread for each of the loop runs but assign a number of them to a couple of threads.

-Michael

Op Mon, 3 Nov 2008, schreef Florian Klaempfl:

Well, those tests even don't take care of thread starting time :)

Threads are started at application startup, in fact my command lines were:

~]$ OMP_NUM_THREADS=1 ./stream_omp
~]$ OMP_NUM_THREADS=8 ./stream_omp

Theads not needed are simply blocked until an OpenMP loop activates them.

Taking advantage of MT requires always deep knowledge about the used architecture and the code being executed and this is something OpenMP ignores. For a big vector operation the number of used threads should be adapted to the memory architecture

... and bound to the correct cores, i.e.:

stream]$ OMP_NUM_THREADS=2 numactl --physcpubind=0,4 ./stream_omp

... gives:

------------------------------------------------------------- Function Rate (MB/s) Avg time Min time Max time Copy: 8294.5171 0.0139 0.0154 0.0155 Scale: 8191.0001 0.0141 0.0156 0.0157 Add: 7920.1633 0.0218 0.0242 0.0244 Triad: 7990.9738 0.0217 0.0240 0.0241 -------------------------------------------------------------

But:

stream]$ OMP_NUM_THREADS=2 numactl --physcpubind=0,4 ./stream_omp

... gives:

------------------------------------------------------------- Function Rate (MB/s) Avg time Min time Max time Copy: 11603.7546 0.0099 0.0110 0.0110 Scale: 11152.7465 0.0104 0.0115 0.0116 Add: 10795.5704 0.0160 0.0178 0.0179 Triad: 10881.7832 0.0159 0.0176 0.0177 -------------------------------------------------------------

So, you need knowledge about the underlying NUMA architecture to get the best performance.

for computational intensive applications like Mandelbrot the number of threads must be adapted to the number of available virtual cores.

Exactly.

By the way, GCC is totally unsuitable for this benchmark, both its OpenMP implementation as it's loop vectorizers are too weak. You need Intel or Pathscale to reproduce these results.

Daniël

Op Mon, 3 Nov 2008, schreef Daniël Mantione:

Copy/paste error. Correct command lines are:

stream]$ OMP_NUM_THREADS=2 numactl --physcpubind=0,1 ./stream_omp

versus:

stream]$ OMP_NUM_THREADS=2 numactl --physcpubind=0,4 ./stream_omp

Daniël

For a big vector operation the number of used threads should be adapted to the memory architecture,

With the C #pragma opm_.... this can be done statically by the programmer using the "schedule" keyword, defining "chunks" (there seem to be more sophisticated ways using the "static" "dynamic" and "guided" keywords, but I am in no way an expert on this

for computational intensive applications like Mandelbrot the number of threads must be adapted to the number of available virtual cores.

With the C #pragma opm_.... this is done automatically ("dynamic"). The OpenMP library seems to do this and assign the loop runs to the threads. This seems to be the default behavior.

-Michael

I looked into this and it doesn't offer anything sophisticated what cannot be done by simple classes and doesn't solve the real problems of MT like syncing.

I looked into this and it doesn't offer anything sophisticated what cannot be done by simple classes and doesn't solve the real problems of MT like syncing.

Sorry, I don't want to be a PITA, but I really don't understand

Of course you can do parallel processing with FPC by creating threads using TThread and manually distribute the tasks to be done. But the user will need to write a lot of code and even using a single TThread is more than the average Pascal programmer ever does. Distributing identical tasks on multiple thread will go far beyond their scope.

And writing "parallel" together with "for" is just adding a single keyword. Providing easy support for common tasks (such as exploiting the benefit of an SMP system) is exactly what everybody expects from a decent programming language.

-Michael

... and as said before, helps usually nothing. As Daniel's examples has shown MT programming requires deep knowledge about the architecture, adding a simple parallel keyword usually calls only for trouble because of race conditions, memory trashing etc. Exploiting the benefits of an SMP system is _really_ hard and cannot solved easily by the primitive means of OpenMP.

Michael Schnell wrote

And writing "parallel" together with "for" is just adding a single keyword. Providing easy support for common tasks (such as exploiting the benefit of an SMP system) is exactly what everybody expects from a decent programming language.

Can you show a real-world program with source that significantly benefits by using OpenMP (alone)?

Micha

Hi all
I would like to point out that these days there is a huge hype around multicore systems. As a result one sees stupid parallel demonstrations, such as the Mandelbrot one. This is a purely graphics demo with no other utility and remote from any reallistic parallel applications. Moreover, this examples is one of the few which can be done efficiently on present day multicore system. So, if FPC team is really intent on bringing parallel constructs in the language, one has to look at things from a broader perspective. It is necessary to think of what reallistic parallel applications would look like.

Unfortunately, the utility of multicore systems has been largely exagerated by their manufacturers. The main problem is that multiple cores share the same memory bandwith. As a result it is highly unlikely that one can have COMPLEX programs running in a concurrent way in a multicore system without clogging the memory bus and using up all the cache. Multiple cores are usefull if there is little memory transfer (does not happen often, except of course if you compute fractals), or if memory transfer is done in a predictable fashion. About the only example of the later is linear algebra subroutines (scientific computing) and certain multimedia applications (concurrent MPEG decoders, for example).

Now, it is true that a dot-product or vectur sum can be done very ellegantly with a prallel loop. However, these are very low-level operations, ones that (at least in the scientific computing community) are typically optimized for each particular architecture and provided as user api. Consequently no one would go around write matrix-vector multiplication in a high level language. Linear algerba is the usual bottleneck and if you do real applications this has already been written and optimized. Consequently, parallel loops look beautiful, but they are of little practially utility. In summary the programming style that lead to assembly level loop-unrolling for superscalar processors is likely to be the same programming style that will be used for multicore machines.

So typicall parallel code revolves around higher level algorithms. For example, if you want to compute the heat distribution in a automobile engine you would go and first partition your engine in a lot of smaller components. The you would perform complex, memory intensive computations on each piece, then you would patch them together. It is quesionable if milticore systems are usefull in such a scenario as it involves large memory transfers. However, if it is (or you have a real multi-processor shared memory machine), then what you would need from the language is a nice enapsulation of threads. This implies (local) parallel procedures, arrays of (local) parallel procedures, parallel class methods, semaphores and critical sections.

The present obstacle with object pascal is that for each (class) method that implements a parallel algorithm, one has to separately implement a thread object. Moreover, parallel algorithms typically need global vars (the one they operate in prallel to), so you need to move the local method variables to the thread object to. In the end, the implementation of you algorithm is shared between the method and the thread object. Finally synchronization is provided by classes (TEvent, TCriticalSection) which have to be constructed and destructed explicitly, with the necessary resource protection (try..finally) overhead. This is not convenient.

I hope this will helps the discussion.

Peter Popov

I hope this will helps the discussion.

Thanks for this well written explanation. If you've precise ideas how OP could be extended to solve the mentioned problems, I'd to be happy to know them and we can also discuss if it can be implemented.

I just completed another example, where threads helped less than expected:
A program that loads tiff images (about 60mb each), resizes and apply some filters to them, then saves them. The disk io makes less than 10% of the time. The filters are mostly simple loops - perfect for OpenMP. There is almost no speed difference from one thread or two threads on a two core machine. :(
I will soon test on a 2 x quad core with 8 threads.

Mattias

Op Mon, 3 Nov 2008, schreef Peter Popov:

Actually I disagree with this, it overexagerates in the other direction. There are plenty of applications who can use hundreds of cores, so 4 or 8 should not be a problem. Parallelizing is a much more sane way of scaling performance than going for more GHz.

Regarding memory bandwidth, remember that NUMA systems like the Opteron architecture get more memory bandwidth the more sockets there are.

However, as you point out there is no free lunch, parallelization of software is complex.

Daniël

If the machine has a unified cache for both cores like a core duo there is indeed little chance for improvement by MT.

Can you show a real-world program with source that significantly benefits by using OpenMP (alone)?

A task that should benefit from multiple threads on an SMP system should be Matrix multiplication another is doing DSP-like things and FFT. Those are really common tasks when handling physical and mathematical calculations. All these feature independent outer loops that can be done as independent threads that only need synchronization when all are ready. Thus in GNU C a simple line "#pragma opm_parallel for" would be enough to run them in parallel via OpenMP.

It would be nice to compare the speed of such a program on a modern four core PC with and without this line. (Should be easy, but I don't have such a machine myself:( and I didn't ever do C for PC programs yet :) :) , but I'll see if I can find something somewhere....)

-Michael

... and as said before, helps usually nothing.

As much as I do appreciate your comments, I really can't believe that the GNU GCC community took the pain to fully integrate OpenMP that deeply in their system that it can be used by just issuing a single #pragma line, if there would be no benefit to be expected.

-Michael

Very nice wrap on this one Michael.
While architectures such as OpenMP or CUDA do not solve and don't even try to solve the general case of MT development, they are very powerful in the areas of image processing and DSP. Indeed we are already researching CUDA for usage in our libraries. For those unaware of it, CUDA is the MT API for the NVidia GPUs . My NVIdia comes with 240 cores, and can perform with mind blowing speed fairly complex DSP.
Each tool simply has its place and usage ;-) , so does OpenMP . That does not mean that it is the magic wand for solving MT, it is just another tool that we can use.

With best regards,
Boian Mitov

------------------------------------------------------- Mitov Software
www.mitov.com
-------------------------------------------------------

Also all the current game consoles such as PlayStation and XBox 360 use the same approach for ray tracing. This is the reason for their impressive performance.

With best regards,
Boian Mitov

------------------------------------------------------- Mitov Software
www.mitov.com
-------------------------------------------------------

There is almost no speed difference from one thread or two threads on a two core machine. :( I will soon test on a 2 x quad core with 8 threads.

Great example. Thanks for sharing the results with us !

You said "less than expected", so you did expect a better result. Do you have an idea what the technical reason could be ?

I tool a look at the C code of one of the example projects in the ALIOTH test. Here they seemingly used SIMD vector operations via "typedef int v4si __attribute__ ((vector_size(16))); // vector of four ints". Maybe this helps more with such calculations, but I have no idea how this could be done using FPC.

-Michael

If the machine has a unified cache for both cores like a core duo there is indeed little chance for improvement by MT.

Sounds like a reasonable explanation. So a this kind of CPU Chip is more or less a hoax ?

-Michael

Thanks a lot for providing this broader view on the issue ! So the current bottleneck is the hardware architecture and not the programming languages shortcomings.

-Michael

Op Tue, 4 Nov 2008, schreef Michael Schnell:

That's too strong. The Intel multicore architecture is a stopgap solution, AMD systems scale usually much better. However, that does not mean Intel cannot win any benchmark, in fact, they win many benchmarks. Some applications simply put less stress on the memory architecture than others.

Daniël

However, as you point out there is no free lunch, parallelization of software is complex.

Of course you are right.

But providing a starting point to the application programmer (like GNU C does with the basic OpenMP pragma stuff ) would help to do a first step, that - depending on the problem to be solved - might only result in a very small or no improvement, but maybe can encourage the application programmer to to a deeper analysis and enhance his software appropriately (if the language constructs provided are tunable appropriately).

-Michael

So you just add a parallel keyword to your C code and you've CUDA capable code ;)? CUDA (which is a great thing) has no unified memory, the available operations are limited and requires usually a major overhaul of the used algorithms. If you take care of this, you can take advantage of CUDA.

Each tool simply has its place and usage ;-) , so does OpenMP . That does not mean that it is the magic wand for solving MT, it is just another tool that we can use.

OpenMP might have it's use but I consider the impact in the compiler too big for real use.

Each tool simply has its place and usage ;-) , so does OpenMP . That does not mean that it is the magic wand for solving MT, it is just another tool that we can use.

That is my POV, as well.

OpenMP seems to be a relatively quite easy "add-on" for compiler makers, so the GCC people decided to tightly implement it. It seems helpful with a number of common tasks.

If you want to have _really_ good optimization, have to solve a a really complex or non-standard problem, you of course are lost with OpenMP and need to use more appropriate methods. Supposedly will not use FPC here but a more low-level programming language (like GCC without OpneMP support).

-Michael

OpenMP might have it's use but I consider the impact in the compiler too big for real use.

What do you mean by "impact" ?

Using a "parallel" keyword should introduce a similar complexity as the #pragma ocm_... of in GCC (of course i don't know the implications it imposes).

-Michael

Op Tue, 4 Nov 2008, schreef Michael Schnell:

If you want to have _really_ good optimization, have to solve a a really complex or non-standard problem, you of course are lost with OpenMP and need to use more appropriate methods. Supposedly will not use FPC here but a more low-level programming language (like GCC without OpneMP support).

That's too optimistic. If you want to have good multicore scaling you are lost with OpenMP. As Peter Popov said, good multicore scaling usually involves partitioning your problem. And by the way, FPC is much more suitable for low-level programming than GCC :)

Daniël

You still didn't show any example which shows the real power of parallel which cannot be solved by a T(Pooled)Thread.Create and T(Pooled)Thread.WaitFor statements.

What low level constructs does GCC provide FPC hasn't?

If you want to have good multicore scaling you are lost with OpenMP.

As I said, that is my POV, too.

I did not start this discussion for preaching for OpenMP. In fact I know close to nothing about it. My starting point were the rather poor results for FPC in the "shootout" tests and "parallel" keyword you can use with "for" in the Oxygen/Prism Object Pascal language.

I am not in a position to evaluate any implementation details, but I feel that not supporting parallelism as a language construct looks like a shortcoming of FPC's and so I think there might be room for improvement.

-Michael

Moving things in the language without a notable advantage which can be solved by a library is not a good thing.

You still didn't show any example which shows the real power of parallel which cannot be solved by a T(Pooled)Thread.Create and T(Pooled)Thread.WaitFor statements.

As I said several times, I don't suggest that any other implementation (be it OpenMP or whatever) would provide better results than you can achieve with using what we already have (TThread and friends).

But using TThread - especially for this kind of "technical improvement" stuff - is beyond the scope of the average application programmer and now Oxygen/Prism with the "parallel" (and other) keywords and GCC with the "#pragma ocm_..." are here to help the common application programmer with that. This might - or might not - be considered a challenge.

-Michael

Michael. How many MT programs you really made?

Petr

People not being able to use TThread properly should avoid MT programming. I don't think OpenMP prevents you from creating race condititions in parallel for loops.

Michael,

You really start to make more and more sense ;-) . As for parallel implementation that makes it easy, a look at CUDA may be beneficial. Submitting a multidimensional matrix and function to be performed across cores on it, has probably a better chance than using loops, although both can solve the problem. The NVidia also will queue the requests.
On another hand, the modern Intel/AMD CPUs come with only maximum of 8 cores/half cores, and they will barely reach 16 sometime next year. This type of processing may not offer as much advantage as on Sparks or NVidia GPUs where you have hundreds of cores. It may be a bit early to add this functionality in the compilers today, but you never know. For 16 or less cores the traditional Threading approach is probably a better fit than matrix/loop type of queued execution.

With best regards,
Boian Mitov

------------------------------------------------------- Mitov Software
www.mitov.com
-------------------------------------------------------

In our previous episode, Boian Mitov said:

Well, we stuff lighter cards (think 8500,8600) in our vision systems (passively cooled is better for factory use).

We are also playing with CUDA, but one of the problems we have is that you must do pretty heavy transformations to earn back just the transfer of data to and from videocard memory. Those are too rare for us to dive into that technology quickly, but I'm keeping notes :)

However if one is more pattern recognition oriented, I can understand that it could get interesting.

btw, is CUDA MT, as in that you can have several independant calculations on the same GPU ?

In our previous episode, Michael Schnell said:

No. But one must be very careful to say

new app performance = old app performance * cores

it doesn't work that way. Keep in mind that multicores were NOT developed for the desktop but for the heavier servers that used multiple processors (heavier database, web and fileservers). Multicore pretty much started out as that typical setup integrated cost efficiently, nothing more.

And servers work on several connections/workloads at once, often with not THAT great datasets.

Zitat von Michael Schnell :

Yes, I expected more, because the filters were not yet optimized. I thought the filters were still below the memory throughput and that's why multi threading should give a good speed gain. Both cores are at 100% and the critical sections are only called between big blocks. I will investigate this further, but it is already a good example, that 'simply' adding "parallel" keyword does not necessarily accelerate programs, even those with loops that look like perfect OMP examples.

Do you have an idea what the technical reason could be ?

I guess Florian gave a reason. Although I think there must be more than that. I will tell, when I find out.

I did some tests with MMX vectors and the results were surprising: Adding two 4xsingle vectors can be done with one single MMX command. FPC already uses MMX instructions, but is not clever enough to combine the four additions into one. But todays cpus are smart enough to do that themselves. And because assembler functions can not be inlined by FPC yet, while pascal functions can be inlined, the pascal function runs at top speed, while the assembler function with the SIMD opcode is much slower because of the call/ret.

Conclusion:
Multithreading and multi cores could accelerate lots of things, but the above examples (mt and simd) show that contrary to popular believe simply adding 'parallel' or using MMX does not accelerate.

Mattias

I don't know about Michael, but almost every single development I have done in the last 13 or so years had to have multithreading, be it for handling RS 232, sockets, background compilation, image processing, process control, communications, artificial intelligence, data acquisition, or running medical procedures and measurements. Each of those tasks ended up having a lot of support threads. Most of this stuff surely needs writing synchronization and carefully controlling the threads, however there is a large number of tasks that can easily be solved with CUDA or parallel loop approach - DSP, Image processing, AI calculations, process control.

With best regards,
Boian Mitov

------------------------------------------------------- Mitov Software
www.mitov.com
-------------------------------------------------------

CUDA works best if you load the image and then perform multiple processing/transformations on it. This is the approach we take in our architecture.
CUDA uses task queue. It adds the tasks to the queue and assigns a core as soon as one is available. It uses some form of matrix for preparing the tasks.
The best approach IMHO is to use the CPU in a multithreaded filtering application to prepare the tasks, and add them to the queue, while the GPU just executes the tasks. then each CPU thread waits for its result while other threads are preparing the next set of data. This is the approach we take currently. In essence there is no timesharing in GPU Core. Each core will start working on a task and will not interrupt before completing it. This is different than a CPU MT. The standard MT is hardly needed when you have 240 cores in a single board, and you can join boards with external bus to share the queue ;-) . I think the current limit is 3 boards so you can scale up to 720 cores in a single system. I have only one board, but it is probably enough for me to do development.

With best regards,
Boian Mitov

------------------------------------------------------- Mitov Software
www.mitov.com
-------------------------------------------------------

Moving things in the language without a notable advantage which can be solved by a library is not a good thing.

Don't you think "ease of use" _is_ a notable advantage of a programming language ?

-Michael

Michael. How many MT programs you really made?

I don't know what you exactly mean here.

If MT just stands for Multi-Threading. Lots of them as well in Pascal (usually Delphi) as in GNU C. Multi-Processing or Calculation speed never was an issue but providing low latency to some of the tasks (GUI serial and TCP/IP communication tasks, you name it).

As I already said, I never used OpenMP or similar tools but just TThread (in Pascal) or my own stuff (in C).

-Michael

It is. But as pointed out, parallel does not solve any of the problems of MT programming.

In our previous episode, Michael Schnell said:

No, it is not specific enough. Moreover "ease of use" is usually not applied to simple overseeable examples, but more generally to the fact that you can harder make a mess.

I don't think OpenMP prevents you from creating race condititions in parallel for loops.

I guess this is _very_ true and if using OpenMP with less standard stuff than Matrix-Multiplication etc, you will quickly run into this kind of problems (as we all know).

-Michael

a look at CUDA may be beneficial.

I _never_ suggested that OpenMP is the way to go. I just stated that there _should_be_ a way to improve FPC on behalf of easy use of MP.

-Michael

Florian, actually has some points here. Being mainly C++ developer I used to ne big proponent for "clean" languages, and criticized Java for adding threading support into the language. On the other hand, it does make development easier, so there are arguments in both directions.... Florian is right that this has to be carefully weighted.

With best regards,
Boian Mitov

------------------------------------------------------- Mitov Software
www.mitov.com
-------------------------------------------------------

My question was only for thought-provoking. Every programer which works with threads, knows that is necessary give attention many things -- shared memory, locking, deadlocks, race conditions, only thread safe calls...

I think that some keyword _parallel_ cannot solve this. In the hand of unexperienced programer brings more problems than profit.

_parallel_ can be maybe used for lightweight threads. But here we must keep in mind that thread creation is quite expensive. Depending on memory architecture, number of CPUs and selected number of threads we can get different speedup or slowdown too.

Petr

BTW: The CUDA as example simply uses a thread pool. Maintaining a number of cores pool will very much eliminate the thread creation. Please note that I am not advocating this addition. There are better ways to add MT support IMHO, but that is another discussion.

With best regards,
Boian Mitov

------------------------------------------------------- Mitov Software
www.mitov.com
-------------------------------------------------------

In our previous episode, Boian Mitov said:

BTW: The CUDA as example simply uses a thread pool. Maintaining a number of cores pool will very much eliminate the thread creation.

True, but CUDA single threaded. So you would need a threadpool per thread that can do _paralel_

True, in their case it is core pool. Another concern is that in multicore situation the operating systems tend to use spinlocks for the threads instead of context switching locks. This probably will lead to CPU waste if locks are used to suspend the threads in the pool.

With best regards,
Boian Mitov

------------------------------------------------------- Mitov Software
www.mitov.com
-------------------------------------------------------

In our previous episode, Boian Mitov said:

True, in their case it is core pool. Another concern is that in multicore situation the operating systems tend to use spinlocks for the threads instead of context switching locks. This probably will lead to CPU waste if locks are used to suspend the threads in the pool.

Well, that is the responsibility of the programmer. He can choose between a _paralel_ for and a non paralel one. But indeed it could be a waste if some iterations of the for loop can last longer than others.

I think that some keyword _parallel_ cannot solve this.

This of course is correct. It's only meant for very simple cases (e.g. like Matrix Multiplication) where mutual interactions are excluded. Normally this does not help with _latency_ issues (what I normally use threads for) but with parallelism for speed (which I never did yet).

In the hand of unexperienced programer brings more problems than profit.

If he does not stick to this paradigm, he of course will fail.

-Michael

Actually here a "good" implementation of _parallel_ for can be beneficial, if it implements the pool for sure without Spin Locks. Most developers (including me until recently), are probably unaware that Windows, Linux, and Solaris, switch to Spin Locks when you have multiple cores. I thought a context switch was always done on Lock, but it appears I was wrong. This means that thread pools will waste CPU cycles if they are blocked by locks, a good _parallel_ for implementation can ensure proper pool implementation and thus improve performance.

With best regards,
Boian Mitov

------------------------------------------------------- Mitov Software
www.mitov.com
-------------------------------------------------------

Just observing the thread I'd like to add some real world examples where OpenMP like additions can be very effective.
Not because something can't be done otherwise, but because it has syntactical advantages.
That said, even with the addition of a "parallel for" syntax and the like (paralell while or parallel until might be a bit nicer; or parallel for in) MT programming is never easy and an easy syntax might fool a programmer who lacks a good understanding of MT.

Examples (I am with Boian here) are DSP and webservers. DSP often requires multiple sequential operations over buffers (effects processing). This is multi core kintergarten! If I can avoid the typing overhead by writing something like:

    Parallel For Effect in Effectchain do
          ProcessBuffer;

This is where I do a lot of MT as a hobby.
Compare the operator overloading syntax that I use here as well: I can write EndBuffer := EndBuffer + Effect[i] where EndBuffer is a record or class that encapsulates the buffer (I am experimenting with rewriting DC-DSP like that).

Florian, you probably agree this is a more favorable syntax than accessing all samples with loops all the time.

That would be very comfortable...and more efficient than all sequential programming, although it can be done in the way Florian describes (at the cost of a higher RSI risk ;) )

Also: in the case of heavily loaded webservers or SOA applets/servlets/services, where the bus is no bandwdth issue, a "paralell for" syntax might be beneficial in writing concise code. This is where I do a lot of MT processing in actual machines are 8 cores and most apps are single in, distributed out.

Michael Schnell wrote

I tool a look at the C code of one of the example projects in the ALIOTH test. Here they seemingly used SIMD vector operations via "typedef int

This has nothing to do with MT or OpenMP though. GCC can vectorize automatically and takes advantage of SSE.

Micha

Thaddy schrieb:

I consider parallel as syntactic sugar:
- the same effect can be achived by some thread classes
- it gives the compiler no opportunity to generate better code
- the use cases are limited, it is nothing being used hundred of times in a typical program
- it solves none of the real MT problems

Florian Klaempfl wrote

Thaddy schrieb: I consider parallel as syntactic sugar:

Yes it is sugar ;)

Florian Klaempfl wrote

- it solves none of the real MT problems

No. I agree: if a programmer or architect doesn't understand MP the syntax "sugar" might be misleading and prone to misinterpretation. But that is already the case with TThread. You wouldn't believe the times I had to correct code from customers or from my teams because all "threaded" code was written inside a synchronize call. And I found it very hard to explain they did write a timer like routine, nothing to do with threading.

But it seems the opinion of the c++ comunity that - for properly educated users - OpenMP adds value to the language. Enough to integrate the concept.

This is indeed not a feature that makes MT programming easier. But I think you will recognize the example I just described as just as furiating/desparate.

Actually, most people who are doing parallel computing for scientific purposes (the vast majority of prallel programs written so far) use MPI (http://en.wikipedia.org/wiki/Message_Passing_Interface). Language extensions are never too popular as far as shared memory machines/clusters/supercomputers are involved. The main reason is that such equipment is usally unique and the manufacturer typically provides only FORTRAN/C compiler, sometimes C++, and always an optimized MPI. There is virtually no chance that GCC would run well on a custom supercomputer. Moreover, parallel programmig is quite difficult, so usually there is no time to explore new language concepts.

Let me give you a flavour of how concurrency is implemented in ADA, using pascal syntax
(see http://www.seas.gwu.edu/~csci51/fall99/ada_task.html).

procedure TSomeClass.DoSomething;
type
   local_task1 = task; // Declaration of parallel task
   local_task2 = task(x: integer); // Declaration of parallel task
var
   res: Integer;
   // Implementation of first local task
   task local_task1;
   begin
   end;
   // Implementation of second local task
   task local_task2(x: integer);
   begin
   end;
var
   A: local_task1;
   B1: local_task2(10);
   B2: local_task2(20);
begin
// At the start of DoSometing, four processes run concurrently:
// DoSomething, 1 copy of local_task1 (A) and two copies of local_task2
(B(10), B(20))
...
// DoSomething exits only after both A and B(01) and B(20) all are done
end;

The difference between a normal local procedure and a parallel task is that tasks are started at declaration time. Up to this point, the semantics is tha same as in ADA. Passing parameters to a task is done at declaration or by messages
(http://www.seas.gwu.edu/~csci51/fall99/ada_task.html).

However, one can instead define a pointer to a task which is no longer started automatically:

type
   plocal_task2 = ^local_task2;

and then, start a new task at any time by doing:

var
   pB: plocal_task;
begin
   ....
   New(pB, plocal_task(30));
   ...
   Dispose(pB); // wait for pB to terminate
end;

In this way tasks can be made equiavalent to local procedures/functions. The idea is to encapsulate TThread. The advantage is that the following programming paradigm is significanlty easire to code:

We have a class with method DoSomething. DoSomething executes a series of complex (higher level) parallel tasks. These tasks need input from DoSomething and need access to its local variables (global with respect to each task). Whatever the local tasks do, eventually DoSomething patches the results and returns to the caller.

The main advantage is the the task will have access not only to DoSomething's local declarations but also to all the methods, fields and properties of TSomeClass. With threads, the descendant of TTHread which implements the task needs a pointer to TSomeClass in order to have access. This means additional work in constructors, etc and it is not clean. Relatively simple algortithms get spread along several methods which is not desireable.

I will try to refresh further my memories of ADA's parallel constructs and try to see if it could be useful.
I would also like to mention one language particularly good for distributed heterogeneous networks (clouds, as it is called these days): mpC.

Peter Popov