Good day.
There is a need to establish some process affinity using free pascal and ubuntu. Do not tell that on this occasion you can read?

Thanks in advance

See
man sched_setaffinity
for the call to do this.

Free Pascal does not have this call predefined, but you can make this call.

Using the do_syscall you should be able to set up the call to the kernel.

Something like:

do_Syscall(syscall_nr_sched_setaffinity,fpgetpid,setsize,@cpu_set);

Michael.

Just a huge thank you:)

2011/1/21 :

Alexey Voytsehovich wrote

Request for additional info :

do_Syscall(syscall_nr_sched_setaffinity,fpgetpid,setsize,@cpu_set);

The above generalized suggestion is very helpful , but can someone provide or point me to a link which describes the type cpu_set_t which @cpu_set points to.

sched.h doesn't really explain the type definition or how to use it.

Many thanks in advance.
Brian

Request for additional info :

do_Syscall(syscall_nr_sched_setaffinity,fpgetpid,setsize,@cpu_set);

The function do_SysCall() Arg 4 expects longint not a pointer but even when typecast do_SysyCall() doesn't appear to work.

... a good description here.

http://www.ibm.com/developerworks/linux/library/l-affinity/index.html

Any suggestions
Thanks in advance
Brian

After a bit of research , the issue of setting the cpu affinity has been solved , which may be of use to other folks.

A bit of info here concerning the data type cpu_set_t , which as far as I can determine (no help from the mess that is the c library source) is essentially an array of DWORD. For my purposes running on an x86 CPU , allowance for 32 cores is sufficient , using cpu_set_1 as one unsigned long word.

http://www.cyberciti.biz/tips/setting-processor-affinity-certain-task-or-process.html

Here are some code fragments which may be of help.

The function do_SysCall(0) basically doesn't work for setting the CPU core.

Although the Linux documentation claims sched_setaffinity() amd sched_getaffinity() work for threads , it only works for processes.

pthread_getaffinity_np() and pthread_setaffinity_np() work properly for threads.

FPC has some sloppy code as there is a note not to use PtrInt , instead use Ptruint , but the function BeginThread() returns PtrInt , but as long as you are not aritmetically manipulating the pointer it works ok.

With a dual core Intel CPU with all cores enabled in the BIOS , sched_getaffinity() returns $03 (0011) and if you set process 0 to use only 1 core , sched_getaffinity returns $01 correctly.

Same for thread_getaffinity_np() and pthread_setaffinity_np() , although if you set the CPU process 0 (current process) to one core , all the threads run on that core.

// ------- external functions from c-library libc (not the UNIT libc !
)----------------------------------------
function sched_getaffinity(pid : Ptruint; cpusetsize : longint; cpuset :
pointer) : longint; cdecl; external;
function sched_setaffinity(pid : Ptruint; cpusetsize : longint; cpuset :
pointer) : longint; cdecl; external;
function pthread_setaffinity_np(pid : Ptruint; cpusetsize : longint; cpuset
: pointer) : longint; cdecl; external;
function pthread_getaffinity_np(pid : Ptruint; cpusetsize : longint; cpuset
: pointer) : longint; cdecl; external;
//-----------------------------------------------------------------------------------------------------------------------
procedure  Set_Thread_CPU_Core(ThreadID : PtrInt; CPU_Core : longword);
const
 cpu_SetSize = 4; // 32 cores max
var
 cpu_Mask : longword;
 cpu_set : longword;  //cpu_set_type sufficient for 32 core CPU
 ResultX : longint;
begin
 ResultX := pthread_getaffinity_np(ThreadID,cpu_SetSize,@cpu_set);
 writeln('Result ',ResultX,' ');
 writeln('Mask   ',Hexl(cpu_set));
 cpu_set := $01; // CPU 0
 ResultX := pthread_setaffinity_np(ThreadID,cpu_SetSize,@cpu_set) ;
 cpu_set := 0; // clear it in case pthread_setAffiniti_np() above = -1
(fail)
 ResultX := pthread_getaffinity_np(ThreadID,cpu_SetSize,@cpu_set) ;
 writeln('Result ',ResultX,' ');
 writeln('Mask   ',Hexl(cpu_set));
end;
procedure  Set_Process_CPU_Core(ProcessID : PtrInt; CPU_Core : longword);
const
 cpu_SetSize = 4; // 32 cores max
var
 cpu_Mask : longword;
 cpu_set : longword;  //cpu_set_type sufficient for 32 core CPU
 ResultX : longint;
begin
writeln('------ Process -----------');
 ResultX := sched_getaffinity(ProcessID,cpu_SetSize,@cpu_set);
 writeln('Result ',ResultX,' ');
 writeln('Mask   ',Hexl(cpu_set));
 cpu_set := CPU_Core;
 ResultX := sched_setaffinity(ProcessID,cpu_SetSize,@cpu_set) ;
 cpu_set := 0; // clear it in case sched_setAffinity() above = -1 (fail)
 ResultX := sched_getaffinity(ProcessID,cpu_SetSize,@cpu_set) ;
 writeln('Result ',ResultX,' ');
 writeln('Mask   ',Hexl(cpu_set));
 writeln('-------------------------');
end;

Thanks for that, interesting. I'd have expected affinity to have been oriented towards processes rather than threads, since I think it largely originated when people started working on NUMA systems- 's also interrupt affinity, which becomes significant when (the bus associated with) an I/O device is "nearer" some CPUs than others. I've had hardware of this type in the past, but in those days kernel support for these calls was patchy.

Mark,

All the documentation seems to indicate that processes and threads are treated alike in Linux , however , even having established that I can apparently select a core to run a thread , I haven't yet been able to make it work.

I explain the findings from a dual core Intel CPU.

Using the sched_setaffinity() I can run a program on either core1 or core2 as long as all threads are set to the same core.

If the main process (program) is set to core2 and all the threads are set to core2 everything is fine and dandy.

However if the main process (program) is set to Core1 and any thread is set to Core2 using pthread_setaffinity_np() , when the Core2 thread starts to run it generates an immediate GP fault.

The intent is to be able to run a thread which polls a serial or Ethernet port , rather like one would do with an ISR , but using a separate core for the task , while the main program runs on a different core.

At this point I am not certain if there is something I am missing.

Brian wrote

Mark, All the documentation seems to indicate that processes and threads are treated alike in Linux ,

Remember that historically, there were two different threading models: LinuxThreads, where a thread was implemented as a process, and the newer NPTL. Somewhere I've got code that can test for this: the difference is detectable and different architectures made the transition with significant time separation.

My understanding is that ownership of resources, in particular memory, is still a property of processes rather than threads.

If the API permits, try setting the affinity of the process to both cores, and then adjusting the affinity of the individual threads within this. Never try to move the affinity of a thread outside that of the process that owns its resources.

The intent is to be able to run a thread which polls a serial or Ethernet port , rather like one would do with an ISR , but using a separate core for the task , while the main program runs on a different core.

Unless you are going directly to the hardware, bear in mind that the actual device access is going to be done by the kernel using interrupts and buffers. So while I can sympathise with wanting to dedicate a core for the "heavy lifting", do you /really/ have to poll the buffers with a tight loop, rather than letting the OS do the job for which it was designed? And are your timing requirements really so stringent that you /have/ to do this?

At this point I am not certain if there is something I am missing.

At this point I'd be digging into the kernel for my architecture of choice and finding out what the restrictions are. Bearing in mind that it's a long time since I've tinkered with this, but my understanding is that much of it originated inside SGI, where their model was to have a high-speed network between the "chipsets" of a large number of- in effect- separate boxes (I've got /one/ SGI system here with that type of port). With that sort of architecture, you'd want to try to stop processes from moving between boxes gratuitously, and you might, as a separate setting, want to pin individual threads to one processor in each box. I've also got systems comprising multiple boards, where each board has an identical collection of ports, and in that type of architecture you'd want to try to keep a process near to any ports it was servicing (i.e. with suitable interrupt affinity) even if traffic between boards was comparatively efficient.

Mark ,

Thanks for the tips and info.

There are two reasons for running threads on specific cores :

1) The possible need for a tight loop for receiving serial data.

2) The program is serial in nature receiving data, does not use X11 , controls the mouse and keyboard using libusb , and writes directly to the graphics card frame buffer (very fast). I have found that when multicore is enabled in the BIOS , or if I set the process to use more than one core with sched_setaffinit() , it appears the Linux scheduler is causing a problem and somehow mucks up some of the graphics when the data load increases , as the scheduler then starts to try and spread the load between the two cores.

If only one core is used everything works fine.

Regards
Brian

Attached is a simple test program with two threads. When the threads run , one displays '+' and the other displays '2'.

If both threads run on the same core , the threads work , but if the threads are set to different cores , it generates a GP fault.

If anyone is interested , download the code . It was tested on Ubuntu 12.04lts , but should be portable .

Would be interested hear any results to confirm it is not a hardware issue on this PC.

coret.pas
<http://free-pascal-general.1045716.n5.nabble.com/file/n5717544/coret.pas> test_threads.pas
<http://free-pascal-general.1045716.n5.nabble.com/file/n5717544/test_threads.pas>

Regards
Brian

The only x86 systems conveniently to hand are multi-CPU rather than multi-core, but I'll happily test that provided you can make it into a standalone program. As it stands coret.pas requires non-standard units, and test_threads.pas isn't standalone.

Mark ... sorry for the mixup. The program and unit should now be stand alone. The unit links libc (rev6).

Regards
Brian

coret.pas
<http://free-pascal-general.1045716.n5.nabble.com/file/n5717549/coret.pas>

test_threads.pas
<http://free-pascal-general.1045716.n5.nabble.com/file/n5717549/test_threads.pas>

Attached binary executables which run on Ubuntu 12.04lts , but should be portable.

coret_ok sets the process and two threads to core1

coret_fail sets to process to cores1 and 2 , one thread to core 1 and one thread to core 2. It soemtimes runs ok , but usually fails with a GP fault.

coret_ok
<http://free-pascal-general.1045716.n5.nabble.com/file/n5717550/coret_ok>

coret_fail
<http://free-pascal-general.1045716.n5.nabble.com/file/n5717550/coret_fail>

Both appear to run OK here, for at least several minutes. Test system is x86, P3x2 with no ht etc., Debian "Squeeze" with standard kernel (2.6.32).

Your code is broken.

You are using the console output from two threads simultaneously without proper locking. That's most likely what's causing the crash.

The critical sections you are using only protect each thread against itself, meaning they basically do nothing. Being inside a critical section is not a guarantee that you won't get preempted, but just that no other threads can enter the same section.

"Jeppe Græsdal Johansen" wrote

Your code is broken. You are using the console output from two threads simultaneously without

proper locking. That's most likely what's causing the crash.

(Disclaimer: I haven't looked at the code, so I'm just assuming that he's using Write(ln))
Console I/O shouldn't be the problem as each thread has its own Output variable (it's a threadvar in FPC unlike Delphi).

Regards,
Sven

"... The critical sections you are using only protect each thread against itself, meaning they basically do nothing. Being inside a critical section is not a guarantee that you won't get preempted, but just that no other threads can enter the same section. ..."

The two threads do not share and variables and do not interact with one another. The question is ... as Mark has previously pointed out ... does Linux x86 allow setting hard affinity with threads where the program (process) runs on one CPU core and one or more threads run on another core?

It looks like all the threads must run on the same core as the block diagram at this link seems to imply.
https://computing.llnl.gov/linux/slurm/mc_support.html

Mark : I found that sometimes coret_fail would run several times correctly , and then when launched on launch N+1 would fail. It may just be luck of the draw that everything was running on one core when it ran correctly.

But they do share state in the context of the RTL :-/

I think what I'd try is skipping the RTL and going direct to a syscall for the output, since I think threading should be safe at that level. Possibly one of the compiler/RTL team would like to comment.

I'm afraid that I'm planning to spend the next few hours, and possibly days, catching up with some long-overdue SPARC and PPC testing- there's some fixed bugs on Mantis that I should have checked and my conscience is catching up with me.

Mark ... many thanks.

Your last comment prompted me to avoid using Write/Writeln to monitor what was happening. By removing Write/Writeln from the threads , ... everything works correctly with two threads , one running on core1 and the other thread running on core2.

I think this boils down to the original issue that I found writing directly to the graphics card frame buffer , which is the same as Write/Writeln , in that there is only one physical display device which is not easily shared.

Regards
Vern

Problem solved ... ironically the solution was from a Windows guy.

Faulty Code

  Core2Thread_ID := BeginThread(@Core2_Thread_Test);
  InitCriticalSection(CriticalSection_Core2);
   Set_Thread_CPU_Core(Core2Thread_ID,$01);
   writeln('Core2 ok');
   Core22Thread_ID := BeginThread(@Core22_Thread_Test);
   InitCriticalSection(CriticalSection_Core22);
    Set_Thread_CPU_Core(Core22Thread_ID,$02);

=====================================================

Corrected Code : InitCriticalSection() BEFORE BeginThread()

 InitCriticalSection(CriticalSection_Core2);
  Core2Thread_ID := BeginThread(@Core2_Thread_Test);
   Set_Thread_CPU_Core(Core2Thread_ID,$01);
   writeln('Core2 ok');
   InitCriticalSection(CriticalSection_Core22);
   Core22Thread_ID := BeginThread(@Core22_Thread_Test);
    Set_Thread_CPU_Core(Core22Thread_ID,$02);

...

Corrected Code : InitCriticalSection() BEFORE BeginThread()

Ah yes. Always keep in mind that the thread could already be happily running (or in extreme cases even finished) when BeginThread (or in extension TThread.Create(False)) returns.

Regards,
Sven