Hi Lazarus guru's,

I have stumbled into a problem with a Lazarus application. At random times, from 5 minutes to several hours, it crashes suddenly, and silently. The only way to get a clue has been to run it with gdb.

It turns out that the crash occurs because of a Segmentation Fault in QTextEngine, triggered by a PostMessage, but apparently the cause was an endless loop in a previous PostMessage, which was executed some 1723 times. Attached the gdb stack trace (with 1722 identical items suppressed).

The scenario is a main thread using LCL, which is used, among other things not relevant here, to display information to the user, and another thread which communicates with a real-time process, and uses a PostMessage to wake up the main thread and make it show what required.

It appears that in some circumstances, the Application.ProcessMessages invoked in the main thread restarts the handling of the same message, thus creating and endless loop.

Is Application.ProcessMessages the vilain, or PostMessage or what?

Snippet of auxiliary thread code:

Snippet of main thread code:

Thanks for any help

Giuliano

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PostMessage.log
Type: text/x-log
Size: 8152 bytes
Desc: not available
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20160209/2e6a952f/attachment-0001.bin>

Giuliano Colla wrote

PostMessage(Form1.Handle,LM_ERROR,Integer(ErrPut),0);

Is your LM_ERROR larger than LM_USER?

PostMessage is thread safe.

You create a recursive scenario yourself by calling Application.ProcessMessages inside of ShowError inside of HandleError - which is the message handling procedure itself, which is triggered by Application.ProcessMessages.

1.Your ErrList access does not look thread safe.
2.You are casting postMessage param as integer, IMO it's wrong, shouldn't it be PtrInt or LPARAM ?
PostMessage(Form1.Handle,LM_ERROR, LPARAM(ErrPut),0); 3.Application.ProcessMessages probably recurse.

I'm using PostMessage() with QtLCL inside one app which heavily uses threads for gui controls updates and it works w/o problems, atm, one of our users have that app running, uptime is cca 90 days (since last update). I don't use such lists, but creating record pointer and free it in main thread message queue....

zeljko

Il 10/02/2016 00:18, Denis Kozlov ha scritto:

Is your LM_ERROR larger than LM_USER?

Yes :

const LM_ANSWER = LM_USER + 100; LM_MESSAGE = LM_USER + 101; LM_ERROR = LM_USER + 102;

Giuliano

Il 10/02/2016 00:22, Denis Kozlov ha scritto:

You create a recursive scenario yourself by calling Application.ProcessMessages inside of ShowError inside of HandleError - which is the message handling procedure itself, which is triggered by Application.ProcessMessages.

That's what I had guessed, but having it confirmed makes me feel better.

Moreover my ShowError is not protected from recursion, because Application.ProcessMessages is called before incrementing ErrGet. Therefore in case of recursion, it will attempt to process again the same message.

I would appear that also Application.ProcessMessages is not protected against recursion, which is possibly a bug.

Thanks a lot,

Giuliano

Il 10/02/2016 08:44, zeljko ha scritto:

1.Your ErrList access does not look thread safe.

It's a poor man's thread safe queue, implemented as a ring buffer, with two indexes, ErrGet and ErrPut. It's fully safe, provided the buffer is guaranteed not to overflow, and that ErrPut is updated only after the buffer has been written. Only the sending thread writes the buffer and ErrPut, while the receiving thread only writes ErrGet, and reads just once ErrPut.
This mechanism has been tested by some billion messages passing in the years through it, by Kylix applications running 7/7 24h/24h in several hundreds locations.

It's not that there have been billions if errors:-) , but an identical setup is used to update informations at a rate of 8/10 per second. Only, in Kylix I couldn't use PostMessage, and it was processed in an OnIdle handler.

2.You are casting postMessage param as integer, IMO it's wrong, shouldn't it be PtrInt or LPARAM ? PostMessage(Form1.Handle,LM_ERROR, LPARAM(ErrPut),0);

PostMessage param is not actually used.

3.Application.ProcessMessages probably recurse.

This I believe to be the problem. From the observations from Denis I realized that my code isn't protected against recursion, but I believe that Application.ProcessMessages should protect itself too, avoiding to handle more than once the same message.

However thanks a lot,

Giuliano

Giuliano Colla wrote

This I believe to be the problem. From the observations from Denis I realized that my code isn't protected against recursion, but I believe that Application.ProcessMessages should protect itself too, avoiding to handle more than once the same message.

But where ? In LCL or per widgetset ?

zeljko

Il 10/02/2016 12:31, zeljko ha scritto:

I don't know. I did look into the code a long time ago, and I seem to remember that the main loop is handled differently: some widgetsets do rely on LCL, some on their main loop. I was hoping that someone more knowledgeable than me would say "oh yes, the right spot is just here!". If such is not the case, I'll try, as soon as I have some time (which I don't have at the moment), to look again into the matter, and possibly suggest a patch.

Giuliano

Qt relies on LCL main loop.

zeljko

Il 10/02/2016 13:48, zeljko ha scritto:

I skimmed through the backtrace. It would appear that the recursion originates in g_main_context_iteration() which calls g_main_context_dispatch(), which will continue to dispatch the same message. I don't believe that there's an easy way to protect Application.ProcessMessages from recursion unless someone is willing to debug and patch Glib (which is used not only by Qt, but also from GTK, if I'm not wrong). Personally I give up.

Giuliano

My understanding is Application.ProcessMessages has to be reentrant for the clipboard to work, at least under X, as clipboard content is retrieved by receiving messages. See
http://bugs.freepascal.org/view.php?id=26242 for a (fixed) bug that showed this up. The sequence is something along the lines of (for example) ProcessMessages -> Handle paste button -> clipboard -> ProcessMessages.

Colin

Il 10/02/2016 22:12, C Western ha scritto:

Well, in general it appears to behave, otherwise a lot of code would fail. I just stumbled in a special case, when Application.ProcessMessages is called from within the handler of a message posted by a PostMessage from a different thread. It would appear that, under some circumstances (because the error appears randomly), there's a bug which causes it to pass again the same message, thus causing and endless loop. It's the recursion in this specific case which appears to be buggy.

Giuliano

Il 10/02/2016 21:13, Giuliano Colla ha scritto:

I don't believe that there's an easy way to protect Application.ProcessMessages from recursion unless someone is willing to debug and patch Glib (which is used not only by Qt, but also from GTK, if I'm not wrong).

A snippet from gmain.c:

It would appear that they realized the dispatch recursion problems but where unable to find a clean solution!
This code appears both in glib-2.28.8 (the one coming with CentOs 6, which I've been using) and in the current version i.e. glib-2.47.5. My feeling is that it's been left there as a reminder that the matter is still pending.
I'm afraid we must live with it.

Giuliano

C Western wrote

My understanding is Application.ProcessMessages has to be reentrant ...

As calling Application.ProcessMessages is only allowed from the main thread it does not need to be reentrant = thread-save.

Of course it can be called recursively and it hence is reentrant in that meaning.

-Michael

IMHO, Michael, you should not redefine words. It confuses people. Reentrant means it can be called again, while it is still running. It does not need to be thread-safe. And a thread-safe function does not need to be reentrant.
Application.ProcessMessages itself is reentrant, but it is not thread-safe. In fact, as already noted by Michael, it must only be called by the main thread.
But Application.ProcessMessages can call almost any event handler, which might not support changing certain properties while executing.

Of course it can be called recursively and it hence is reentrant in that meaning.

True.

Mattias

Mattias Gaertner wrote

IMHO, Michael, you should not redefine words. It confuses people. Reentrant means it can be called again, while it is still running. It does not need to be thread-safe.

Sorry that was not me. The subject here is ".... thread save ...." ,so I tried to explain exactly the difference you mention.

-Michael

Il 12/02/2016 15:35, Michael Schnell ha scritto:

It was possibly me causing the confusion.

I was experiencing random crashes triggered from a PostMessage executed in a different thread.

My first assumption was that possibly PostMessage wasn't thread safe, and this is the reason of the subject of my first post.

Further analysis has shown that this was not the case. It was not a matter of thread-safe, but a matter of recursion.

At least under Qt widgetset, it may happen that calling Application.ProcessMessages from within a Message Handler will cause the same message which was being processed to be processed again, the handler called again and Application.ProcessMessages being called again, thus originating an endless loop and a consequent stack overflow.

All of that apparently originates in the g_main_context_iteration() procedure of Glib2, which, in some cases, doesn't properly terminate a dispatch operation, and in case of recursion will execute again a g_main_context_dispatch() for a message already dispatched. As it is g_main_context_iteration() responsibility to decide which procedures to call and when terminate, it is its responsibility to provide appropriate protection from recursion.

Looking into the code of gmain.c of Glib2, and in different versions, it can be seen that attempts have been made to avoid the problem, but apparently there's still a hole left.

The result is to make Application.ProcessMessage not 100% reentrant, because of a Glib2 bug.

Giuliano

When calling Application.ProcessMessages from within a Message Handler the *next* event in the event queue should be handled. If *the*event*already*being*handled* pops up again, that would be a bug. But of course "the same" message might have pushed by some thread and also needs handling. Here the same handler might be called recursively. This of course does need additional stack space, but not infinitely .

The result is to make Application.ProcessMessage not 100% reentrant, because of a Glib2 bug.

Would you suggest not to handle "the same" event ? How should you detect that its the event already in the works or the next occurrence ?

-Michael

Il 15/02/2016 11:54, Michael Schnell ha scritto:

I'm positively sure that the message being handled is *the same* for a number of reasons.
It is identified by an "errorstring" which from the stack dump is always the same.
It cannot be sent more than once, because where it originates it waits for a user error reset, before sending the next. Moreover, the error messages originate in a real-time part, which has a pool of just 40 messages, therefore it can't possibly send 1722 messages. The same code has been field tested in hundreds of application which have been running for many years (since 2005), with the only difference that the messages were handled in the OnIdle Handler, waking up the main thread periodically with a timer event, instead of waking up the main thread with a PostMessage.

In the case I've stumbled into, when next message eventually comes (after 1722 iterations over the previous one) the program crashes for a SEGFAULT, most likely generated by a stack overflow.

Giuliano

Anyway, IMHO it's not a good idea to fight a (potential) bug (in "Glib2" whatever this exactly means) by implementing a completely queer behavior in Application.ProcessMessage.

-Michael

Il 16/02/2016 10:04, Michael Schnell ha scritto:

Anyway, IMHO it's not a good idea to fight a (potential) bug (in "Glib2" whatever this exactly means) by implementing a completely queer behavior in Application.ProcessMessage.

I fully agree with you. Seeing a unexpected recursion, I had started believing that somehow Application.ProcessMessages could be the responsible. But if the problem is elsewhere, it should be fixed elsewhere, by someone knowing what he's doing.

Giuliano

Il 16/02/2016 10:04, Michael Schnell ha scritto:

a (potential) bug (in "Glib2" whatever this exactly means)

Glib2 means version 2 of GTK Glib, which is used by Qt (and by GTK) to implement the main event loop.
libglib-2.0.so.0

The full picture (from stack trace) is as follows:

1. The main loop of Lazarus (under Qt WS) calls AppWaitMessage in qtobject.inc
2. qtobject calls QCoreApplication::processEvents in qcoreapplication_c.cpp
3. qcoreapplication_c.cpp calls QEventDispatcherGlib in libQtGui.so.4
4. QEventDispatcherGlib calls g_main_context_iteration in libglib-2.0.s0.0 (the Glib2)
5. if g_main_context_iteration has pending messages, it calls g_main_context_dispatch
6. g_main_context_dispatch calls QCoreApplicationPrivate::sendPostedEvents

When the user code calls Application.ProcessMessages the same sequence occurs:
AppProcessMessages in QtObjects.inc is called, which then calls QCoreApplication::processEvents, and all the steps from 2 on are identical.

My feeling is that the bug is sort of this: if the call to Application.ProcessMessages causes a recursion, and a new different message has arrived in the meantime, g_main_context_iteration() understands rightly that there is a pending message to dispatch, but it decides wrongly to dispatch again the old one, instead of the new one. -------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20160216/13de7f83/attachment.html>