Hi RTL/libc experts.

I'm trying top port the TNoGUIApplication (Linux version) I found in MSEIDE to Lazarus.

Here Martin uses a function called "m_sigprocmask". In his code in "mselibc.pas" it is defined as

function m_sigprocmask(__how:longint; var SigSet : TSigSet;
            var oldset: Tsigset):longint;cdecl;external clib name
'sigprocmask';

Now in the FPC RTL there is a function "Fpsigprocmask" in the RTL, "libc.pp" it is defined as

function Fpsigprocmask(how:longint; act:Psigset_t;
oldact:Psigset_t):longint;cdecl;external libc_nlm name 'sigprocmask';

Which in effect seems to be the same.

Only when "using" libc, the identifier "Fpsigprocmask" is not available, even though I am able to use a lot other stuff from the libc library in my ported code (e.g. "signal()").

When I try to define the function"Fpsigprocmask" in that way myself I get some type conflicts when calling it with Martin's code. (I'll try to solve this when I am able to point it to the proper RTL function.

Any help ?

-Michael

Michael Schnell wrote

When I try to define the function"Fpsigprocmask" in that way myself I get some type conflicts when calling it with Martin's code. (I'll try to solve this when I am able to point it to the proper RTL function.

You can use the minimal MSEgui libc bindings in lib/common/kernel/mselibc.pas. They work for i386-linux and x86_84-linux and can be ported to other systems too if necessary.

Martin

I found that the correct libc.pp function name of course (silly me) is just "sigprocmask" without any prefix. Now this part does compile

Thanks,
-Michael

I did this with the first step (no testing, just trying to get the source compiled), but I'd like to add as little code to the packet I'm trying to do, as possible.

I'll try to preserve your directory structure, to allow for a combined Linux/Windows packet, but up till now I have no idea how in Lazarus the compiler knows for which system if the source (dwelling inn a packet) is (cross-) compiled.

Thanks,
-Michael

Do you see any problems with using the libc function defined in the RTL libc.pp ?

-Michael

Michael Schnell wrote

Do you see any problems with using the libc function defined in the RTL libc.pp ?

You should not use it. It's linux-x86 only, and meanwhile quite outdated. It's definitely not maintained to match the latest libc.

Michael.

Michael Van Canneyt wrote

You should not use it.

Martin, why is your code better ?

I don't see the difference. You just seem to have used the same coder as in libc.pp in your own source file....

-Michael

I copied the bits used by MSEgui, fixed the bugs which affected MSEgui and made it working on x86_64-linux. libc version compatibility is not so bad IMHO.
MSEgui works with libc on Suse 9.3 and on 2.4 kernels up to Suse 11.2 and the newest Ubuntu.

Martin

In our previous episode, Michael Schnell said:

Do you see any problems with using the libc function defined in the RTL libc.pp ?

There is no libc.pp in the RTL. There is some old Kylix legacy package in packages/libc, maybe you mean that?

If so, read this http://wiki.freepascal.org/libc_unit

Note that MSEGUI doesn't support FreeBSD or Apple yet to my knowledge, while the fp* ones support that perfectly.

In our previous episode, Martin Schreiber said:
[ Charset ISO-8859-1 unsupported, converting... ]

That's still no reason to not use the supported routines. One can even simply recompile the RTL to have the normal routines use libc (a feature that is in production for e.g. OS X)

As far as I can see, providing own headers is just a compatibility hazard with no real benefit.

Marco van de Voort wrote

That's still no reason to not use the supported routines. One can even simply recompile the RTL to have the normal routines use libc (a feature that is in production for e.g. OS X)

I trust more in libc than in FPC routines, libc is propably better adapted to the actual kernel. MSEgui tries to be Kylix3 compatible where the FPC routines are missing. Using the libc compiler switch is no option because users may depend on the default non libc RTL.
If you answer: "There is no difference" - many software features are OK until real usage only. ;-)

As far as I can see, providing own headers is just a compatibility hazard with no real benefit.

FPC libc does not support x84_64-linux.

Martin

In our previous episode, Martin Schreiber said:

fpsigprocmask exists, and from this thread, I gather the signature is not compatible there too. Moreover libc and kernel are versioned differently.

There are differences, but they are very rare, and more in the realm of the user (which is libc based btw) and netdb packages.

Using the libc compiler switch is no option because users may depend on the default non libc RTL. If you answer: "There is no difference" - many software features are OK until real usage only. ;-)

The fp* routines have been in use since 2003. They are in realtime use.

> As far as I can see, providing own headers is just a compatibility hazard > with no real benefit. FPC libc does not support x84_64-linux.

Over the supported units obviously. unit libc was always a Kylix legacy interface.

I don't understand

mselibc:
function m_sigprocmask(__how:longint; var SigSet : TSigSet;
            var oldset: Tsigset):longint;cdecl;external clib name
'sigprocmask';
FPC RTL singnalh.inc:
function sigprocmask(__how:longint; var SigSet : TSigSet;
           var oldset: Tsigset):longint;cdecl;external clib name
'sigprocmask';

What is the differemce ?

-Michael

OK.

What to use instead of signal(), sigemptyset(), sigaddset(), sigprocmask(), ... ?

in fact I did not find out what to "use" to get a reference to (e.g.) sigprocmask, that indeed does dwell in several .inc files.

-Michael

In our previous episode, Michael Schnell said:

(baseunix.)
fpsignal, fpsigemptyset fpsigaddset fpsigprocmask

http://www.freepascal.org/docs-html/rtl/baseunix/index-5.html

Note that it has been that way for 6 years already, and nearly 5 years in released versions.

in fact I did not find out what to "use" to get a reference to (e.g.) sigprocmask, that indeed does dwell in several .inc files.

The whole libc unit has been unsupported for new development for the same time. Don't use it.

Marco van de Voort wrote

(baseunix.)

Great ! Found it, Thanks.

The whole libc unit has been unsupported for new development for the same time. Don't use it.

OK, I'll try to convert Martin's code to use the fp* functions in the Lazarus package.

-Michael

I don't remember. Please check the sources.

Martin

???? These _are_ the sources. There is not more to a library call.....

-Michael

Michael Schnell wrote

???? These _are_ the sources. There is not more to a library call.....

Maybe there was other code before or it was in order to ensure the right unit was used or there was a problem with linking or a name clash, I really don't remember...

Possibly you will find more information in the SVN history.

In our previous episode, Martin Schreiber said:

(sigset 2.4 ->2.6 change maybe? IIRC it was kept the same in libc for Kylix compatibility)

<http://mseide-msegui.svn.sourceforge.net/viewvc/mseide-msegui/trunk/lib/common/kernel/msepipestream.pas?sortdir=down&view=log>

I d/ldedf the current version if MSEIDE mseide_i386_linux_2_2.tar.gz and found that code.

mselibc:
function m_sigprocmask(__how:longint; var SigSet : TSigSet;
            var oldset: Tsigset):longint;cdecl;external clib name
'sigprocmask';

-Michael

Marco van de Voort wrote

(baseunix.) fpsignal, fpsigemptyset fpsigaddset fpsigprocmask

Thanks !

Martins source now does compile with
uses baseunix and
fpsignal(), fpsigemptyset(), fpsigaddset() and fpsigprocmask(), fpwrite(), ... :)

but there are some more libc functions used, that I did not yet find an equivalent for::
e.g.:

libc.setitimer(),

As sem_post does not seem to be exported somewhere, I defined sem_post like this:

    function sem_post(__sem:Psem_t):longint;cdecl;external;

(also alike for sem_init() and friends)

it does compiler and link, but I_m not sure if it will work. I'd need to do some additional porting, as this function has different parameters from the sem_init() Martin uses (var parameter for the __sem argument).

Any further tips ?

@Martin:
what is the complicated thread save stuff with sys_semdestroy() necessary for ?
Would it not be useful to use a Futex instead of a sema ?

@Marco:
Direct Futex calls of course are only available with libc, but I suppose there are some baseunix things, as futex implicitly is used by by the ptrhead library with thread-semaphores.

Thanks to both of you,
-Michael

Possible, I don't touch such code if it works - or if I think it works. ;-) Do you think a futex is cheaper than an interlockedincrement()?

Martin

Other that System V "sema", Futex ("Fast Userspace mUTEX") does not do a system call if a program tries to gain the semaphore (= "MUTEX" = "MUTual EXclusion") but just (with an atomic instruction) writes to a word in the userspace to mark the semaphore as taken. Only if the semaphore already is taken by another thread, the FUTEX library call does a system call to have the process wait and queue multiple threads that request for the semaphore. When releasing the appropriate happens: if no other process is waiting for the semaphore, no system call is done but the code stays in user space.

Thus with the in most applications by far most common case of "waiting not necessary", no system calls are done (thus no MMU reprogramming etc.) and thus the code might run up to some hundred times faster.

Of course this is (usually) only done with multiple threads (sharing the same user space memory) but not with multiple processes (each having it's own user space memory mapping, unless they explicitly map a shared area).

If a threaded program uses the PThread ("Posix Thread") library, same provides "pthread mutex" functions that automatically use FUTEX if the system provides this (all PC systems do that, only some embedded architectures lack the FUTEX implementation), otherwise PThread falls back to using the sema provided by the OS.

I don't know if FPC RTL uses the PThread library to implement TThread (but it would be the straight forward thing to do) and if it uses pthread mutex to implement things like WaitFor (straight forward again). I'll check this ASAP.

-Michael

Martin Schreiber wrote

Do you think a futex is cheaper than an interlockedincrement()?

Sorry, I forgot to answer to your question in the first mail.

Of course any FUTEX implementation needs to use something like interlockedincrement() to do the user space implementation.

But FUTEX is a _very_ complicated and malicious issue. I do know that there were several generations of Futex implementations until a really working one was released (the first one was done by Linus Torwalds himself ... and proved to be not working correctly). AFAIK: The main problems arise when a mutex is freed and a single waiting threads is to be freed and given the mutex while multiple other threads are supposed to go on waiting.

So _if_ the implementation with interlockedincrement and sema in fact works in the code you use it for, I believe it only works in certain cases (perhaps exactly those your code produces), but will fail in others.

I only came across this issue, as your "sema" implementation uses the "discouraged" old libc unit, and while porting your code I of course wanted to do that along the lines the FPC developers suggest. But your semty Type seems to be quite different from what is provided in the RTL e.g. with syncobjs.

Maybe we can work together to improve your NoGUIApplication class: (1) not using that libc unit and (2) use Futex. I'll come back with what I find.

Could you please explain what exactly the interlockedincrement in sys_semdestroy is used for and what sys_semdestroy is necessary for with NoGUIApplication. Perhaps you could suggest an idea how to implement your sys_sema...() functions using syncobj (or whatever) instead of direct libc calls.

-Michael

In our previous episode, Michael Schnell said:

Note that syncobjs is libc based (via cthreads basicevent*) on *nix, and wholly portable (also to windows).

Michael Schnell wrote

But FUTEX is a _very_ complicated and malicious issue.

Regarding Futex, there have been some 100 patches published in the Linux Kernel Mailing List only within the last three Months !

And what you are doing with interlockedincrement and sema just _is_ another Futex implementation !

-Michael

Marco van de Voort wrote

Note that syncobjs is libc based (via cthreads basicevent*) on *nix, and wholly portable (also to windows).

Ooops, that would mean I would not improve things when replacing direct libc binding by syncobj binding ?

Many thanks for pointing this out !

What do you suggest for doing mutex and similar things ?

-Michael

In our previous episode, Michael Schnell said:

If you mean that it would still link to libc yes.

Many thanks for pointing this out ! What do you suggest for doing mutex and similar things ?

1) syncobjs
2) if it _has_ to be libc free, unit ipc
3) explain here why the above two are not good enough.

I think that you are confusing two things:
a) using the libc unit is always bad (except for keeping old Kylix code running on Linux/i386) b) the C *library* is perfectly fine to depend on for stuff like this

Jonas

Marco van de Voort wrote

1) syncobjs

OK for me, if the developers here don't discourage this, as they discourage using libc.pp

(see up in this thread:

>> On 01/20/2010 02:12 PM, Michael Van Canneyt wrote: >>>> You should not use it.

)

2) if it _has_ to be libc free, unit ipc

For me it has not to be libc-free, but Michel, what is your opinion ? I don't know the ipc unit. I'll take a look.
I suppose IPC is "Inter Process Communication" so supposedly it's not perfectly suited for inter Thread communication (see FUTEX vs. MUTEX)

3) explain here why the above two are not good enough.

I'd like to

(1) do the code according to what the RTL developers say is the recommended way and
(2) use the most suited algorithm (thus FUTEX) in the most agreeable way (the one I know is via PThreadLib that AFAIK is done within libc, but maybe the RTL implements something else and the developers recommend this for the long run).

thanks
-Michael

Jonas Maebe wrote

I think that you are confusing two things: a) using the libc unit is always bad (except for keeping old Kylix code running on Linux/i386) b) the C *library* is perfectly fine to depend on for stuff like this

Of course you are right that I am confused with that.

So using syncobj is "allowed" as same encapsulates the libc.so calls in a "compatible" way.
also using basunix and friends is "allowed" as same encapsulates the libc.so calls in a "compatible" way.

Correct ?

-Michael

In our previous episode, Michael Schnell said:
[ Charset ISO-8859-1 unsupported, converting... ]

As Jonas (and the libc_unit link I posted earlier) clearly stated, they use LIBRARY libc, not UNIT libc.

Good.

I don't know the ipc unit. I'll take a look. I suppose IPC is "Inter Process Communication" so supposedly it's not perfectly suited for inter Thread communication (see FUTEX vs. MUTEX)

IPC is a class of functions. If futex was a portable concept and would become POSIX, it would probably all under the IPC class.

Use as portable units as possible, until you have proven that there is either something wrong with them, or a measurable performance problem in your app.

(2) use the most suited algorithm (thus FUTEX) in the most agreeable way (the one I know is via PThreadLib that AFAIK is done within libc

There is also pthreads, but afaik it also mostly contains portable calls.

but maybe the RTL implements something else and the developers recommend this for the long run).

If the whole world uses mutexes, except some highspeed devices in the kernel, what are you doing that this is really needed? IMHO you are high on a concept, and haven't really researched if you really need it.

In our previous episode, Michael Schnell said:

Rougly correct. But the geenral idea is that the FPC RTL tries to abstract calls and typing in a portable, supportable way. In short, you don't need to know what is beneath them, as long as it works.

For some functionality libc is used, for some not, and this exact implementation might differ according to platform, and might change over time.

Certain complex, risky kinds of calls (like dynamic library loading, threading, locale and user authentication) are generally implemented via libc and iconv.

Some more general ones (baseunix, unix,IPC, termio) can be syscalls or libc.so based, depending on the nature of the target, or the target maintainer's choice.

In general the idea is that if you need unit libc or declare your own exports to libc, you are doing something atypical. That can be because you really are atypical (e.g. in the case of extreme requirements or deep OS dependant system interfacing), but more generally you are doing something wrong.

See also
http://www.stack.nl/~marcov/unixrtl.pdf

On 01/25/2010 05:16 PM, Michael Schnell wrote:

Can somebody provide a commend to this ?

I have no idea what code is called with that external reference (I suppose it's not an so library, but something statically linked to the program.)

Is this a recommended way to do this or is something else more appropriate ?

-Michael

Marco van de Voort wrote

IPC is a class of functions. If futex was a portable concept and would become POSIX, it would probably all under the IPC class.

As I see it, Futex already is a portable Posix concept, as the PThread Library automatically uses it with the pthread_mutex_lock() and pthread_mutex_unlock() functions. I already used that with programs done in C). So IMHO it would be appropriate to support them with the IPC class. I'll take a look at that ASAP.

Use as portable units as possible, until you have proven that there is either something wrong with them, or a measurable performance problem in your app.

That is exactly what I'm trying to do right now (with your help :) ).

I'll take a look ASAP.

If the whole world uses mutexes, except some highspeed devices in the kernel,

Why do you think so ? IMHO the whole world of C programs uses pthreadlib to do threaded applications and with that they use the pthread_mutex calls that automatically use FUTEX if same is available (on PCs with a decent Linux distribution, its always available) Otherwise it automatically uses some kind of OS-based MUTEX. In the same way pthreadlib automatically uses NTPL threads if this is available (on PCs with a decent Linux distribution, its always available) Otherwise it automatically uses "Linux Threads".

what are you doing that this is really needed? IMHO you are high on a concept, and haven't really researched if you really need it.

In fact right now there is no performance need for me, but as I intend to publish this as an open source "packet" that anybody who wants this can easily install in Lazarus and then use it, I of course want to provide the most decent function, portability and performance possible.

-Michael

What I'm doing here is a kind of RTL-extension (to be implemented as a "packet" that can be loaded in Lazarus and then allows for selecting a new type of "application" (i.e. "NoGUIApplication" that works without any binding to any Graphic system like the Windows User Interface API or GTK/QT...), but allows for Event driven programming for things like Timers, TCP/IP, serial interfaces etc. (Martin's MSEGUI happily does provide this and he allowed me to use his code for that.)

I don't know how atypical it is to do an RTL extension :), but of course it should integrate with the current (and future) RTL code as smoothly as possible.

See also http://www.stack.nl/~marcov/unixrtl.pdf

I'm printing this for reference...

-Michael

I finally succeeded in getting a "proof of concept noguiapplication" project compiled and running using the Lazarus IDE.

Seemingly, the timer event is working correctly.

It still uses the depreciated old libc unit.

Martin, can we work together to create a common code base for both the msegui project and a Lazarus add-on packet that avoids to use the libc unit ?

-Michael

???
MSEgui does not use the libc unit.

Martin

Martin Schreiber wrote

MSEgui does not use the libc unit.

Of course it does not, as in the msesys unit the same code (just "external ....") as in libc is available and thus introduced the same potential problems. I'd like to have the project benefit from the work the FPC RTL designers did when moving away from the libc unit towards something more "compatible". In fact I did not yet take a look at what in fact has been changed on that behalf, but I am studying the3 document "LIBC UNIT" in the Freepascal wiki and the "New Unix RTL by Marco.

-Michael

The potential problems of FPC libc unit (packages/libc/src/libc.pp) is that it is not included in .deb packages by default, that it is i386-linux only, that it is not maintained maybe and that it is possible that FPC team will remove the file from the distribution in future.
mselibc.pas is maintained by me, it is included in MSEgui distribution, it is Linux i386 and x84_64 and it will be ported to other systems by me if necessary.

Martin

In our previous episode, Martin Schreiber said:

Well, baseunix is maintained by the FPC core, works on all *nix platforms already is packaged in all distro's, and still you won't use it :-)

Great for mseide, but not good for my Lazarus add-on package which obviously will not be maintained by you. Same should use what the FPC-RTL offers. At the moment I simply use the functions in the libc unit instead of those with the same name from msesys, but on the long run this should be replaced by the "new Linux RTL" as described by Marco. I feel this would not harm when introduced in the mseide code tree freeing you from part of the maintenance obligation.

Moreover a main the target of this package that enables applications for "headless" hardware, obviously is non-X86 archs.

-Michael