hi,

to create hard realtime linux programs with freepascal and xenomai [1] in userspace
I'm installing a custom ThreadManager and a MemoryMutexManager on program init like this:
  SetThreadManager( XenoThreadManager );
  SetMemoryMutexManager( XenoMemoryMutexManager );

This worked until 2.2.4. With fpc 2.4.0, TMemoryMutexManager was removed with commit "r7407 20.5.2007 + heap manager now per thread, reduce heap lock contention"

Looking at rtl/inc/heap.inc from 2.4.0 it seems that the locking is now done using "CriticalSections", is that true ?

To adapt my ThreadManager for 2.4.0 I think I have to:
- remove the SetMemoryMutexManager() call
- implement the *CriticalSection() calls in my XenoThreadManager

Has someone any hints, comments or concerns about the approach ? Anyone else interested in building hard realtime programs using fpc and xenomai ?

thank you
kisda

[1] Xenomai Real-Time Framework for Linux - www.xenomai.org

Stefan Kisdaroczi wrote

to create hard realtime linux programs with freepascal and xenomai [1] in userspace

Ooops ????

Userspace means Linux and Linux means no hard realtime possible (with the official definition of hard realtime: reaching a predefined timing deadline with 100% certainty).

-Michael

Did you look at the xenomai website?

Stefan Kisdaroczi wrote

Looking at rtl/inc/heap.inc from 2.4.0 it seems that the locking is now done using "CriticalSections", is that true ?

Yes.

It sounds perfectly fine to me. You should also experience much better performance with the new heap manager, as the critical section is seldom used in normal operation. If you want a program in which this scenario is stress tested, you can use http://svn.freepascal.org/svn/fpc/trunk/tests/test/theapthread.pp

Jonas

Florian Klaempfl wrote

Did you look at the xenomai website?

Seemingly you need to do your own device drivers and not use any Linux system calls in your realtime process, that seems to run Linux in a kind of virtualization.

So FPC programming for realtime would require to prevent FPC from creating Linux system calls. Thus you need a modified RTL (e.g. for memory management).

-Michael

Hard real time causes a lot of limitation, indeed.

So FPC programming for realtime would require to prevent FPC from creating Linux system calls. Thus you need a modified RTL (e.g. for memory management).

Guess why the topic of this thread is "custom ThreadManager and MemoryMutexManager" ;)

You can call normal linux system calls without problems, that is one of the big features of xenomai, but of course you have to take care, it can have an impact on the realtime performance.

I try to explain the concept:
(There are much better documents on xenomai.org)

Tasks:
- Xenomai adds a realtime scheduler -> two cooperating schedulers.
- ALL Tasks are 'normal' linux tasks, realtime task are additionaly registered in the realtime scheduler.
- A realtime Task can be scheduled by linux AND xenomai: -> The Tasks can migrate while running from on to the other scheduler: running on realtime scheduler -> "primary mode"
running on linux scheduler -> "secondary mode"
On a linux system call in primary mode, the task will migrate to secondary mode automatically, with a blocking realtime call it will switch automatically back to primary mode.
A lot of information is in the /proc-Interface, so you can see which task migrated how often to secondary mode. You can install a Trap-Handler too, which is great to find unwanted mode-switches.

IRQ's:
- Xenomai adds a Interrupt-Pipeline:
Interrupts are handled first by the xenomai scheduler and then passed to the linux scheduler.
--> Linux interrupt handling is stalled while the realtime scheduler has tasks to run

Device Drivers are programmed in kernel-space using RTDM:
- RTDM is a realtime driver Interface supported by Xenomai and RTAI (another linux realtime extension), maybe others. There is also a RTDM interface for linux "native" realtime support using the "preempt_rt-patch" probably in mainline "soon" ...

Future:
If you build your realtime apps as Xenomai userspace apps and make RTDM-Drivers, in about two years (just guessed), you will be able to run your realtime apps with the Xenomai Duo-Kernel approach or on the linux native preempt_rt realtime support without changing anything in the application or driver code, just recompile.

kisda

Stefan Kisdaroczi wrote

You can call normal linux system calls without problems, that is one of the big features of xenomai, but of course you have to take care, it can have an impact on the realtime performance.

Meaning that when doing a Linux system call, at this point a potentially huge delay might be introduced. As the standard FPC memory management in the RTL potentially does Linux system calls, you either need to modify the RTL on that behalf (AFAIK the memory manager is done as a plugin and thus you easily _can_ provide your own one), or avoid memory management completely (no idea how this might be possible).

That is what I meant with "running Linux in a kind of virtualization".

Even closer to "Virtualization". Of course this is very interesting when dual core hardware is provided (which gets more and more "standard".

I do suppose Xenomai offers some memory management paradigm, which you can use instead of Linux system calls in your alternative FPC memory manager.

-Michael

True, but for realtime apps the most important part is the application design. You never create tasks realtime, you never alloc mem in realtime. You dont take a "normal" app, recompile it with xenomai and everything works realtime.

AFAIK the RTL allocs the heap on app startup, or on thread-creation with 2.4.0. In my code im using getmem() to alloc mem from the heap, this will not generate syscalls because the heap is already allocated and the RTL has its own MemoryManager. Regarding the memory management I think using fpc with Xenomai is even better than using C, where you have to call malloc(), which will probably make a linux syscall.

The only "problem" is the locking of the MemoryManager. It should not use Linux-Mutexes, it should use Xenomai-Mutexes. This is what I did with the custom MemoryMutexManager.

If i'm wrong here please cry loud "STOP", thx :-)

The custom ThreadManager is like the cthreads implementation, but it creates tasks over the xenomai library instead of the pthreads lib. Important here is to setup the TLS after creating the tasks over xenomai for fpc.

Its interesting on a singlecore too :-) But yes, on a duo-core you can use one processor exclusively for the realtime scheduler and the other one for the linux scheduler. Which variant you choose, duo-scheduler or single-scheduler with preempt_rt, depends on the worstcase latency your app needs.

The worstcase latency on recent x86 machines is something like:
- 20us : duo-kernel xenomai
- 200us : linux preempt_rt
- >2ms : linux native
Of course that numbers are wrong, it depends on a lot of things, they are just to give an idea.

I do suppose Xenomai offers some memory management paradigm, which you can use instead of Linux system calls in your alternative FPC memory manager.

You can alloc realtime heaps, I use them for shared mems between multiple apps.

Xenomai supports multiple API's (Native,Posix,VxWorks,pSOS+,VRTX,uITRON), i'm using the native skin for some reasons. If you want to use fpc for realtime with the posix skin, i think you just have to adapt cthreads that it links against pthread_rt instead of pthread. Not tested.

kisda

This is not entirely correct: while the rtl does pre-allocate some memory from the system on startup, it can still allocate more later if necessary, and it can also free memory back to the system.

Jonas

To complement what Jonas said,
if there is no room in the currently allocated
Free Pascal heap, the system memory manager calls the system specific function SysOSAlloc
defined in rtl/unix/sysheap.inc, for unix systems. This in turn calls Fpmmap function
which does a syscall syscall_nr_mmap...
code is in rtl/linux/ossysc.inc.

None of these two functions are function variables, which would mean that you need to modify the rtl,
make SysOSAlloc return nil everytime,
alloc enough memory at startup to avoid
running out of memory within a run,
and recompile it if your memory manager still
calls the standard RTL underneath.

Of course, if your own memory manager does not
call the standard one, you should be safe.

Pierre Muller

Thank you for that important hint. Are there any other options than a custom MemoryManager to disable that ?

For the custom MemoryManager way I think I just copy the original one, try to disable the reallocating and freeing, and preallocate more on startup.

On the other side I think its not really grave for my project, as I never allocate anything in the realtime loops. And there are no Objects/Classes, just plain pascal. The same code still needs to compile on a 16Bit Compiler from 1989 (DOS based realtime system)...

Btw, about circular references :
This old compiler uses C-like header files, no interface and implementation section. So you don't have any circular reference problems. To get the 15 apps with 500+ units compatible to fpc and to avoid circular references it was a lot of work, but its doable.

kisda

Jonas Maebe wrote

while the rtl does pre-allocate some memory from the system on startup, it can still allocate more later if necessary, and it can also free memory back to the system.

That is why I suggested that he needs to create a Xenomai aware Memory Manager Plugin.

Moreover he'd need Xenomai aware components for Timer, TCP/IP, serial interface, file operations, ...

-Michael

Thanks for this suggestion, that was really important.

Moreover he'd need Xenomai aware components for Timer, TCP/IP, serial interface, file operations, ...

No, I don't need that, really. You are not wrong here, but in my case we have something like an own RTL, because our old compiler (1989) didnt had that.

Timer:
No pascal timing functions like sleep() are used, my own timing unit calls rt_task_sleep() in the xenomai library. We have other timing functions calling xenomai functions.

TCP/IP:
It's not realtime anyway, so there are no calls in time critical code. I simply use the fpc sockets unit, the thread is running under the linux scheduler when doing socket calls, thats no problem. Because the thread is known to the realtime scheduler too, i can use xenomai message functions (queues, pipes, sem...) to send the received bytes to a task which is always scheduled by the realtime scheduler.

Serial Interface:
Im calling directly the the xenomai library functions rt_dev_open().

File operations:
Not used in time critical parts of the code.

Another good example to show that the scheduler migration is a nice feature is writing the log files:
- The realtime task writes the log data in a buffer and signals that with a realtime semaphore to the writer task. The writer-task is blocking on that realtime semaphore. The writer-task now calls the normal writeln() function, this switches the task to the linux scheduler. After that the task blocks again on the realtime sema, this will switch the task back to the realtime scheduler. If there is no place in the buffer, the realtime task's dont block, they discard the data.

kisda

Sounds nice.

It would be a beautiful o the RTL to allow for Xenomai aware memory management and to provide some Xenomai-aware standard functions (timer, thread locking, ...)

-Michael