I wanted to start a discussion based on current state of FPC and High Performance Computing (HPC).

The premise of the thread is to discuss high scale threads. Currently FPC has matured to the point where it might be beneficial to start targeting a particular market for users. The first thing that comes to mind is the university community where coding is second to theory. Pascal offers ease of code readability and in turn brings the ability to easily organize scientific formulas into computer code that when arranged properly can be executed in a high scale environment, delivering faster, near instant or instant results.

After reading a few responses in a separate discussion regarding threading I came to realize that some work remains to FPC with regard to High Performance Threading. FPC is advanced in that it offers native cross platform access to kernel threads. FPC however, on Linux, relies on a libc Posix Threads (pthreads) library that is a widely adopted implement of the POSIX specification.

Microsoft has spent a lot of resources on parallel computing and it may be beneficial for the FPC project to at least begin to explore the possibility of FPC to be used in a professional high performance environment.

IMO - it may be smarter to develop of a new threading paradigm. A HPC threading system that addresses issues with waitfor/suspend/resume and resolve them for *nix/windows/mac but do so knowing that there are most likely tens of thousands of threads running.

I would like to offer my commitment to this issue since FPC is ideal for real-time computational engines. With regard to HPC, the native TThread object has been implemented (under *nix) with redundant barriers in creation, suspend, and resume features. This redundancy makes for slow thread creation times and as such is not ideal as is.

Re-writing the current TThread implement may not prove nearly impossible as is, because under windows Resume/Suspend Thread is merely an API kernel call and *nix has no such API. Even if we were to re-write the TThreads unit to remove dependency on the pThreads library, we would still be stuck with creating a semaphore under linux, one per thread instance to gain Suspend/Resume functionality.

Suggested Reading:
http://ftp.gnu.org/gnu/glibc/glibc-linuxthreads-2.3.6.tar.gz http://openmp.org/wp/openmp-specifications/
http://www.gnu.org/software/libc/
http://msdn.microsoft.com/en-us/concurrency/default.aspx

Anyone interested in consulting or helping?

Just one suggestion:

If the current TThread uses too many security facilities for your purpose, then I think that it may not be possible to make a single thread manager which fits all purposes.

So you could create your own high-performance thread manager, and for most users which don't need this feature, but prefere safer threads, can keep using the current thread manager, cthreads in UNIXes.

So you can still use the standard TThread object, just add your own high_performace_thread_manager unit to your uses clause instead of cthreads.

In our previous episode, Andrew Brunner said:

What is exactly the problem? A problem that a 5 second thread pool implementation can't solve?

Andrew Brunner wrote

Suggested Reading:   http://ftp.gnu.org/gnu/glibc/glibc-linuxthreads-2.3.6.tar.gz

Why would you want to look at this? It's not used any more (except for really old kernels).

Henry

Do you have a link to the latest version of pthreads?

It was my understanding that libc took over the pthreads project. I just took up this issue a few days ago so I recognize my info may be stale or even wrong.

In our previous episode, Andrew Brunner said:

pthreads means "posix threads". And you said yourself that posix is a specification not an implementation.

Nearly all userland thread systems are called pthreads, kernel supported or not.

Modern m:n implementations are vastly different to the older attempts.

Andrew Brunner wrote

Do you have a link to the latest version of pthreads?

You probably mean NPTL (Native Posix Threading Library).

It was my understanding that libc took over the pthreads project. I just took up this issue a few days ago so I recognize my info may be stale or even wrong.

Well, according to
<http://en.wikipedia.org/wiki/Native_POSIX_Thread_Library> it is fully integrated into the libc, yet it does not seem to be a separate file and I won't download 17 MB with a GPRS connection just to look if it's in there. ;) It probably is.

Vinzent.

Andrew Brunner wrote

Do you have a link to the latest version of pthreads?

I've told you before: NPTL is part of the glibc sources. Download the latest glibc sources and look in the nptl directory. The latest glibc release is 2.11.2.

It was my understanding that libc took over the pthreads project.  I just took up this issue a few days ago so I recognize my info may be stale or even wrong.

pthreads is the standard, not the implementation. The current most widely used implementation is called NPTL. Your link points to LinuxThreads (judging by the name, I didn't download):

http://en.wikipedia.org/wiki/LinuxThreads

The date of the file that you linked is 2005.

Henry

Henry Vermaak wrote

I've told you before: NPTL is part of the glibc sources.  Download the latest glibc sources and look in the nptl directory.  The latest glibc release is 2.11.2.

FPC source to cThreads unit and the thread manager associated with threading under linux presently calls a pthreads.so object. When I did my research I thought I was correct in making an assumption that pthreads.so would require a pthreads project with it's own make file ;-) That is the latest and greatest pthreads project which is now associated with glibc.

pthreads is the standard, not the implementation.  The current most widely used implementation is called NPTL.  Your link points to LinuxThreads (judging by the name, I didn't download):

pthreads is not the standard. pthreads.so is the implementation of POSIX thread specification to be precise :-)

Ok. You mentioned NPTL. What can you tell me about that? Is that the project that makes pthreads?

I'm looking to get the latest and greatest source of posix threads so I draw upon that code and perhaps even come up with a native version for FPC. Any help would be greatly appreciated.

http://en.wikipedia.org/wiki/POSIX_Threads says:

"POSIX Threads, or Pthreads, is a POSIX standard for threads."

libpthread.so is the name of the library, not the name of the implementation.

Ok. You mentioned NPTL.  What can you tell me about that?  Is that the project that makes pthreads?

http://en.wikipedia.org/wiki/Native_POSIX_Thread_Library

Henry Vermaak wrote

libpthread.so is the name of the library, not the name of the implementation.

I updated FPC from SVN and it is pointing to this library now :-). Only problem is someone took some changes I made to remove semaphore on thread creation... If I would have known someone was going to commit those changes I would have taken the time to move FSEM into a block where createsuspended was true this way it wouldn't have event effected others who need that barrier :-)

> Ok. You mentioned NPTL.  What can you tell me about that?  Is that the > project that makes pthreads? http://en.wikipedia.org/wiki/Native_POSIX_Thread_Library

This was interesting "There exists a tracing tool for NPTL, called POSIX Thread Trace Tool (PTT). And an Open POSIX Test Suite (OPTS) was written for testing the NPTL library against the POSIX standard. "

^^^ Could this possibly be a solution or a lead in to getting GDM-GDB to stop locking up on entry to breakpoint?!? I know debugging threads is nearly impossible under Linux (at least with GDB and Ubuntu)

I wonder if I could possibly make a suggestion here. NPTL is fairly well established in the Linux kernel- it's been a bit of a long haul, with some architectures getting it much later than others. However a lot of unix programmers are still very unfamiliar with the concept of threads, and "diluting the brand" with a competing implementation isn't going to help matters.

If you want to do something which is implemented inside FPC then don't mess with threads. Instead get an efficient coroutine implementation working, and then build whatever else you want on top of that- but call it something like Preemptive Coroutines :-)