Is it feasible to use lnet for a simple TCP daemon on Linux, i.e. wait for a connection on a predefined port, read as much data as is available, repeat? And if so, what does the SerSock parameter to Accept() represent?

I'd normally use a thread for this, but I've already got lnet's telnet client running in the program so would rather stick to the same library if possible.

Mark Morgan Lloyd wrote

I'd normally use a thread for this, but I've already got lnet's telnet client running in the program so would rather stick to the same library if possible.

AFAIK, Lnet does not do threading internally (as does AsyncPro or - partly - Indy), so if you need to wait for TCP input while doing other things in your program, you need to do your own thread, anyway.

I switched from Lnet to Synapse lately, as same seems to a lot of provide usable protocols. Synapse does not provide internal threads, either.

-Michael

[Nod] Works adequately for a telnet-based terminal though.

I switched from Lnet to Synapse lately, as same seems to a lot of provide usable protocols. Synapse does not provide internal threads, either.

True. Synapse's protocol support is second to none, but in the current case there's really no protocol involved- grab/store data and that's it.

Mark,

I have done that before.

Basically, in Lnet, if you have more than 1 LTcp object, you must create your own TEventer and override its CallAction method.

Then assume your Ltcp object is myLTCP
   myLTCP.Eventer := TMyEventer.Create;
In my thread.execute
    repeat
      myLTCP.Eventer.CallAction
   until terminated;
   if you have just one ltcp then just
    repeat
      myLtcp.CallAction
   until terminated

I don't know about SerSock. Never used it and seemed have no need for it. Dennis

If I understand from more Googling etc., lnet server support works by the program first setting up a server socket and applying Listen, then setting up a connection socket and calling Accept() with the server socket (possibly the server socket's handle, and I'm unsure how to get this) as parameter. Both of these would need CallAction calls on a regular basis.

I've concluded that using a thread is, in fact, preferable since I don't know how much data will arrive from the far end, how it will be paced, and whether the far end might attempt multiple simultaneous connections. While a polling scheme is OK for the foreground terminal stuff there's too much change that its proliferation will mess up either foreground or background transfer.

Mark Morgan Lloyd wrote

I've concluded that using a thread is, in fact, preferable

True.

It's a pity that Synapse and friend (especially SynaSer) does does not implement internal threads that throw appropriate events in the Main Thread when something comes in (or an error appears). This is a very common request, and since the fpc-Team some time ago enabled TThread.Queue in the RTL, this is doable in a straight forward, "fpc-ish" way (portable but Delphi compatible).

-Michael

The very reason I use Synapse is that it DOES NOT use threads. Synapse is simple to use. Keep it so.

The way it is now, threads are still an option. If threads are incorporated in the design there is no way to get them out again.

Michael.

Although for certain types of usage being able to get directly to the hardware (or strictly, to the lowest level the OS supports for normal users) is absolutely essential, e.g. if timing information has to be appended to each character or if records of control signal changes have to be inserted.

In our previous episode, Mark Morgan Lloyd said:

True, but that is rarely done, since you would have to disable FIFOs too then, since you usually can't queue such signals (like e.g. BREAK).

Michael Van Canneyt wrote

The very reason I use Synapse is that it DOES NOT use threads. Synapse is simple to use. Keep it so.

Of course I don't vote for Synapse always using threads, but to provide _additional_ functions / objects (maybe units) that work similar to AsyncPro (TCP/IP and serial streams). Plus maybe more sophisticated _additional_ objects that in internal threads handle complete protocols and provide appropriate MainThread events to the user program.

-Michael

Of course the current functionality of Synapse needs to stay in place as it is.

With Delphi, I very often happily used the asynchronous MainThread Events AsyncPro (for TCP/IP and serial data streams) provides.

-Michael

In our previous episode, Michael Schnell said:

E.g. TComport only starts an helper thread if you register receive events.

If you do not, you can use it in a select/waitformultiple like way in a thread.

Marco van de Voort wrote

E.g. TComport only starts an helper thread if you register receive events. If you do not, you can use it in a select/waitformultiple like way in a thread.

Sounds good.

I'm going to test this ASAP.

-Michael

would this hamper or cause problems with normal non-gui programs using the library?

would this hamper or cause problems with normal non-gui programs using the library?

1) This feature of course should only be provided additionally to the functions we know and love.

2) Other than the work-alike "Application.QueueAsyncCall" which is provided by Lazarus long since, the rather new "TThread.Queue" is not only Delphi-compatible, but it also is located in the RTL and thus can be used in LCL based nongui applications and even without linking to the need to do calls to a "synchronize" function which the RTL provides, to pull the event queue. This _can_ be done in a close loop (e.g. containing a sleep() call), which of course increases latency and processor overhead. Better it is done by a decent triggering mechanism (e.g. using a semaphore or self-piping) the loop waits for and each queue push triggers.

This is another improvement I hope for: enhancing the synchronize and TThread.Queue/TThread.Synchronize features in the RTL in an OS-depending way that (optionally) automatically pulls the queue for the main thread.

-Michael

"Michael Schnell" wrote

On 09/13/2013 06:06 AM, wrote: > > would this hamper or cause problems with normal non-gui programs using

the library?

1) This feature of course should only be provided additionally to the

functions we know and love.

2) Other than the work-alike "Application.QueueAsyncCall" which is

provided by Lazarus long since, the rather new "TThread.Queue" is not only Delphi-compatible, but it also is located in the RTL and thus can be used in LCL based nongui applications and even without linking to the LCL at all.

You need to do calls to a "synchronize" function which the RTL provides,

to pull the event queue. This _can_ be done in a close loop (e.g. containing a sleep() call), which of course increases latency and processor overhead. Better it is done by a decent triggering mechanism (e.g. using a semaphore or self-piping) the loop waits for and each queue push triggers.

This is another improvement I hope for: enhancing the synchronize and

TThread.Queue/TThread.Synchronize features in the RTL in an OS-depending way that (optionally) automatically pulls the queue for the main thread.

As we have already written in some previous mails to there is a global event procedure to wake up the main thread that is triggered when something is queued. The LCL uses this already and other programs could use a TEvent or whatever.

Regards,
Sven

Could you elaborate on this ? Is this "triggering" done by some kind of self-piping or using an OS semaphore ? Is it provided for all OSes and Archs

Or do you mean a function call is provided that allows the user (or the LCL) to implement such a functionality.

-Michael

In our previous episode, Michael Schnell said:

Afaik it is implemented in the checksynchronize call in a platform dependent manner. It assumes that you run checksynchronize in the idle event of your eventloop though. (which should be the case for LCL implementations)

Or do you mean a function call is provided that allows the user (or the LCL) to implement such a functionality.

If you mean integrate with your own eventloop, then yes, make sure that your idle event in your eventloop calls checksyncrhonize.

Marco van de Voort wrote

If you mean integrate with your own eventloop, then yes, make sure that your idle event in your eventloop calls checksyncrhonize

OK, so "checksynchronize()" is the correct name of the RTL-provided function that pulls the event queue. (Sorry that the correct name escaped from my silly brain.) AFAIU this supposedly is not platform-dependent, but simply pulls the queue and calls the event handlers as sub-functions of checksynchronize(). (I need to check the sources ASAP...)

Did I understand you correctly that there is a "notifying" procedure, that is called by the Event queuing mechanism each time an event is pushed onto the queue ? This procedure then would potentially be called from Worker threads and thus here, we need use the OS to schedule the (later to be done) queue poll by the main thread via checksynchronize(). Could you tell me the name of that procedure, so that I can find it in the sources and do a testing project using it ?

Obviously the LCL needs to use both procedures to merge the RTL's event queue with the queue it handles the GUI events in.

Obviously a non-LCL based user code can implement it's own OS-depending mechanism to schedule a call to checksynchronize() triggered by the said "notifying" procedure.

The "hope" I expressed in the recent mail was, that the RTL might allow for enabling the described OS-depending Thread to MainThread scheduling mechanism, so that any user code does not need to implement it new with each project (but just set an enable flag or something similar). This should be possible as the RTL does know which OS it is built for and the maintainers do have the necessary expertize to do it in the optimum way.

Thanks,
-Michael

In our previous episode, Michael Schnell said:

OK, so "checksynchronize()" is the correct name of the RTL-provided function that pulls the event queue. (Sorry that the correct name escaped from my silly brain.)

Yes, it is the connection between VCL and RTL for this kind of stuff.

AFAIU this supposedly is not platform-dependent, but simply pulls the queue and calls the event handlers as sub-functions of checksynchronize(). (I need to check the sources ASAP...)

It blocks for a maximum time on the event.

Did I understand you correctly that there is a "notifying" procedure, that is called by the Event queuing mechanism each time an event is pushed onto the queue ?

That was not me, but Sven or sb else. But as I understood it, the queue method does that internally(thus platform dependent). He never said it was a public option. You trigger it by queue()ing or synchronize()ing.

This procedure then would potentially be called from Worker threads and thus here, we need use the OS to schedule the (later to be done) queue poll by the main thread via checksynchronize().

No. The mainthread simply should always loop around checksynchronize while idle. Checksynchronize is the delay in the event loop so to say, by blocking on some internal event. Thus if something is queued, the event is triggered and the mainthread will wake up immediately (if it was indeed blocked on checksynchronize)

Obviously the LCL needs to use both procedures to merge the RTL's event queue with the queue it handles the GUI events in.

Obviously a non-LCL based user code can implement it's own OS-depending mechanism to schedule a call to checksynchronize() triggered by the said "notifying" procedure.

The non-LCL one must simply call checksynchronize as the blocking part of the mainthread's eventloop.

The "hope" I expressed in the recent mail was, that the RTL might allow for enabling the described OS-depending Thread to MainThread scheduling mechanism, so that any user code does not need to implement it new with each project (but just set an enable flag or something similar).

See above. It is there, there is no LCL dependence in theory, but other eventloop systems might assume they themselves implement the blocking part. (e.g. to wake up because of OS messages)

Services, console network server implementations etc probably already do that both on Delphi and Lazarus.

Marco van de Voort wrote

The non-LCL one must simply call checksynchronize as the blocking part of the mainthread's eventloop.

Thus, my wish already seems to be fulfilled.

Great !

there is no LCL dependence in theory, but other eventloop systems might assume they themselves implement the blocking part. (e.g. to wake up because of OS messages) Services, console network server implementations etc probably already do that both on Delphi and Lazarus.

In fact the LCL seems to rely on the "It blocks for a maximum time" (as you said).

While this might be "good enough" for GUI events, there might be instances where we either would want either (1) a faster reaction (than the said "maximum time") on events that are not to be pushed through the RTL event queue or (2) avoid the processor overhead that is imposed by the main thread leaving the block after the "maximum time".

Regarding (2), I suppose it would be just nice to be able to define this time (I need to check the sources or the docs to find out).

Regarding (1), I suppose this will be a really rare request (not even required by my foreseeable queer projects). I would rather do this kind of stuff in worker threads and use TThread.Queue (which hurray !!! is available since some time) to schedule main thread actions.

Thanks a lot !
-Michael

The latter:
http://www.freepascal.org/docs-html/rtl/classes/wakemainthread.html It is called by the internal procedure ThreadQueueAppend which is used by both TThread.Queue and TThread.Synchronize. Additionally if the the main thread is already waiting inside a CheckSynchronize() call it is notified using an event.

Regads,
Sven

I did not say it explicitely, but meant it that way. See WakeMainThread ( http://www.freepascal.org/docs-html/rtl/classes/wakemainthread.html ).

WakeMainThread is used by the LCL to initiate an idle-event (whatever an "event" is for the underlying widgetset) so that CheckSynchronize gets executed.

Regards,
Sven

CheckSynchronize has a timeout parameter:
http://www.freepascal.org/docs-html/rtl/classes/checksynchronize.html

Regards,
Sven

Sven Barth wrote

CheckSynchronize has a timeout parameter: http://www.freepascal.org/docs-html/rtl/classes/checksynchronize.html

Great !

Maybe the docs should be updated and mention TThread.Queue, which for me is most important, as it - finally - allows for decent parallel work of Worker threads and mainthread in applications that don't use the (GUI binding of the) LCL.

-Michael

Sven Barth wrote

WakeMainThread

Thanks for the pointer.

I'll take another look there.

-Michael

Sven Barth wrote

( http://www.freepascal.org/docs-html/rtl/classes/wakemainthread.html ).

IIRC, TThread.Queue does this, as well.

Thus updating this page might be appropriate...

Thanks,
-Michael

Thanks for the clear information.

Enhanced by the knowledge about TThread.queue It will allow me to do some nice tricks.

-Michael

Conclusion:

Thus we could draft a "living noGUI" TCustomApplication sibling (aka "LCL WidgetType"), that allows for firing MainThread Events triggered from (a newly implemented) TTimer, TThread.Queue(), TThread.Synchronize(), TApplication.QueueAsyncCall() and the legacy windowish PostMessage() -> "procedure..message" mechanism in a low latency and low processor-overhead way:

- The Application Object contains a MainLoop simply calling CheckSynchronize()

- TTimer is implemented including defining the timeout constant for calls to CheckSynchronize() as the greatest common denominator of the "Time" property of all enabled TTimer instances in the project (i.e. a simple timer that accumulates delays imposed by MainThread activities)

- TThread.Queue() and TThread.Synchronize() seemingly just work

- TApplication.QueueAsyncCall() and PostMessage() could be implemented using "NotifyMainThread()" (and maybe slightly enhancing the queuing and de-queuing mechanism provided in the RTL for additionally allowing for this type of Events, containing procedure pointer, self-pointer and an Integer that can be pointer to a parameter or holds the Windows-Message parameters. )

This would provide the mechanism for porting "embedded" (thus without GUI - or with the GUI configured as "disabled" at compile time) Delphi applications to non-GUI Linux gadgets.

It is really frustrating to see, that this now seems to solves the issue I am hunting for since some five years. Over the time, there had been multiple discussions on this, initially in the fpc devel mailing list and later in the Lazarus devel mailing list. Up till now, I never was told "this already works" or "this is easy" (followed by a decent description - such as yours - of CheckSynchronize() and WakeMainThread(). Instead I was told that I should look at the LCL source code and that "in Windows, the Event queuing mechanism is done by Windows itself and in Linux it is done by a queue in the LCL that is managed by the underlying Widget Set". This (some years ago) misled me to invest a considerable amount of time in unsuccessfully trying to strip off the GUI binding from an existing LCL Widget Type.

Unfortunately until end of this year I will am busy with a complicated (non pascal and not even Linux) project and I'll be not able to do some other programming work (and hopefully provide a patch enabling the functionality described above). But If somebody wants we can go on discussing the issue here, at "fpc devel" or at "lazarus devel".

Thanks for listening,
-Michael

Michael Schnell wrote

- TTimer is implemented including defining the timeout constant for calls to CheckSynchronize() as the greatest common denominator of the "Time" property of all enabled TTimer instances in the project (i.e. a simple timer that accumulates delays imposed by MainThread activities)

I wouldn't use the timeout constant for this. If you have two timers of which the greatest common denominator is 1, but nevertheless rather large (e.g. two primes) then you'd loop unnecessarily (I know this is a constructed example, but nevertheless one should care for this!). I'd instead suggest to implement the timerloop of each timer as a thread that waits its timeout time on a event (so that it can also be stopped) and then notifies the mainthread using Queue().

Regards,
Sven

Right you are. (I just wanted to keep this as simple as possible for a starter.)

You simply could use a sleep in the timer loop,. This would be arch independent out of the box. But it would impose accumulative delays depending on the CPU performance.

An more advanced timer implementation would use arch depending OS-based timers. This would allow for a much more exact timing.

-Michael

[...] It is really frustrating to see, that this now seems to solves the issue I am hunting for since some five years.

Has it ever come to your mind that "hunting" might not be sufficient to create code?

See for example:
http://lists.lazarus.freepascal.org/pipermail/lazarus/2012-August/075593.html

This is true for PostMessage.

This (some years ago) misled me to invest a considerable amount of time in unsuccessfully trying to strip off the GUI binding from an existing LCL Widget Type.

There are many ways to implement timers and queues. But if you want the whole event system of the LCL then you need do more. Have you ever asked how to program a timer in a console application on this list?

Mattias

I would not use Sleep as you need to be able to cancel the timer (e.g. Timer.Enabled := False or the application terminating). At least on *nix based systems you'd need to artificially send a signal which is why I'd prefer the event way which you can simply set in SetEnabled and Free.

An more advanced timer implementation would use arch depending OS-based timers. This would allow for a much more exact timing.

Of course. The timer component should be written in a way that it can be implemented in a platform specific way as well.

Regards,
Sven

Preferably, a standard timer implementation does not use a fptimer unit implements a timer with a thread, but this forces the use of threads on your application which is not always desirable.

Michael.

(and supposedly for Application.QueueAsyncCall)

Of course I do know this and this is what makes it impossible to port the Delphi applications, my colleagues did, to a non-GUI environment.

The new idea now is, that implementing PostMessage (and supposedly Application.QueueAsyncCall) might be done using the existing RTL Event queue (which only recently came to my full awareness), instead of creating (or in Windows attaching to) an additional queuing mechanism.

There are many ways to implement timers and queues. But if you want the whole event system of the LCL then you need do more.

What non-GUI events - additionally to the events I mentioned - would be needed for "the whole event system of the LCL" ?

Have you ever asked how to program a timer in a console application on this list?

Of course I do know that this is doable (and I already did it several times). But (as I mentioned multiple times) I don't need this for any concrete application, but for creating an "SDK" that allows for porting existing Delphi applications in a way as easy to use as possible. Thus an "Application" and "Postmessage" (thus an LCL Widget Type implementation) is necessary.

-Michael

Sven Barth wrote

I would not use Sleep as you need to be able to cancel the timer

A decent point ...

-Michael

Michael Van Canneyt wrote

The fptimer unit implements a timer with a thread, but this forces the use of threads on your application which is not always desirable.

Should be doable, as well. AFAIK, mse (for Linux) uses signals on that behalf. We might want to steal some ideas there.

-Michael

No. QueueAsyncCall is implemented in the LCL without widgetset code.

[...] What non-GUI events - additionally to the events I mentioned - would be needed for "the whole event system of the LCL" ?

For example PostMessage can send messages to other applications on Windows.

[...] I don't need this for any concrete application, but for creating an "SDK" that allows for porting existing Delphi applications in a way as easy to use as possible.

Really?

Mattias

Of course this is true but is has noting to do with "the event system of the LCL".

-Michael

Al 18/09/13 09:32, En/na Michael Schnell ha escrit:

It is really frustrating to see, that this now seems to solves the issue I am hunting for since some five years.

Perhaps if you spent your time actually using fpc instead of hunting for non-problems you'd have realized that this worked five (and more) years ago as it does now.

Bye

Luca Olivetti wrote

Perhaps if you spent your time actually using fpc instead of hunting for non-problems ...

As already pointed out, I have no intention to do a Project of such kind myself. I am just trying to construct a toolbox to help my colleagues to port their Delphi applications.

-Michael

Says who?
Some users asked about messages from/to other applications for nogui LCL applications on Windows.

Mattias

In our previous episode, Michael Schnell said:

While there is sigalarm, but can you have multiple independent timers in an application that way?

Mattias Gaertner wrote

Some users asked about messages from/to other applications for nogui LCL applications on Windows.

OK.

In fact to me it has been a completely silly concept of Borland's that in early Delphi versions TThread.Queue was not documented (or in very early versions maybe not even implemented) and thus the users were forced to "by hand" use Windows messages to schedule asynchronous Main Thread events.

By that the concepts of Thread to Mainthread signaling has been mixed with the concept of inter-Application signaling (and even the decently ugly windowish Application to GUI Widget signaling).

But this is not your or my fault ;-) .

Maybe when implementing PostMessage for Thread to Mainthread signaling for windows OS we could take care of inter application messages.

-Michael

Marco van de Voort wrote

While there is sigalarm, but can you have multiple independent timers in an application that way?

I don't know. We would need to ask mse.

-Michael

I think Martin used setitimer and handles multiple timers by calculating which one expires first, and subtracting that amount from the other timers, etc.

Henry

mse inverts the problem.

The timer implementation relies on an external 'tick' routine.

How this tick is delivered depends entirely on the event loop.

Michael.

In our previous episode, Henry Vermaak said:

That's a trick commonly done on embedded platforms with limited timers (I actually did it once as part of a course that created a small RTOS for 8051 chips), but on *nix systems where libraries might also reserve certain itimers, that could fail miserably.

Michael Van Canneyt wrote

The fptimer unit implements a timer with a thread, but this forces the use of threads on your application

The way I like it! ;-) Especially considering how thread friendly these [common place] multi-core CPU's are these days.

Regards,
- Graeme -

TThread.Queue is a rather new addition. Delphi 2009 if I'm correct. So that's definitely not "very early".

Regards,
Sven

In our previous episode, Sven Barth said:

D2006 afaik.

Sven Barth wrote

TThread.Queue is a rather new addition. Delphi 2009 if I'm correct. So that's definitely not "very early".

OK, at least four years ;-) .

I have "Turbo Delphi". Here it is implemented but not to be found in the help.

-Michael

Yes, the POSIX interval timers are a much better solution for this. Dealing with signals are still a bit annoying, but things like timerfd and signalfd are linux specific.

Henry