Hello,

Current use of Timeout parameter in TSimpleIPCServer.PeekMessage:

* >0 -- number of milliseconds to wait.
* 0 -- not documented and inconsistent!
* Windows -- wait infinitely (forced via MsgWaitForMultipleObjects)
* Unix -- return immediately (implicit via fpSelect)
* OS/2 -- return immediately (implicit via DosWaitEventSem)
* Amiga -- wait 25 ms (forced via Sleep)
* -1 -- not documented, but consistent!
* Windows -- wait infinitely (implicit via MsgWaitForMultipleObjects)
* Unix -- wait infinitely (implicit via fpSelect)
* OS/2 -- wait infinitely (implicit via DosWaitEventSem)
* Amiga -- wait for MaxInt ms (implicit via Sleep)
* <-1 -- not documented and inconsistent!

The biggest problem is that it is impossible to get PeekMessage to return immediately in Windows, making SimpleIPC unusable in GUI thread (or other high performance threads).

I propose the following consistent use of Timeout parameter:

* >0 -- number of milliseconds to wait.
* 0 -- return immediately.
* -1 -- wait infinitely.
* <-1 -- wait infinitely (force to -1).

I can produce a patch and a test project, if there are no objections.

Regards,
Denis Kozlov

There are no objections.

Michael.

There is another problem with SimpleIPC on Windows.

This same problem was experienced in @Juha's post "SimpleIPC and TWinMsgServerComm.PeekMessage":
http://lists.freepascal.org/pipermail/fpc-devel/2013-October/032871.html

TWinMsgServerComm.PeekMessage never waits for a Timeout because of this line:

MsgWaitForMultipleObjects(1,FHWND,False,TimeOut,QS_SENDMESSAGE);

It always fails and returns immediately WAIT_FAILED with GetLastError saying "Invalid handle". As per documentation at MSDN, MsgWaitForMultipleObjects does not work with window handles. Funny thing is that I am pretty sure that this line used to work in the past, on much older OS, perhaps XP before SP3 or earlier. It used to be able to wait, but definitely not any more (tested on 7 SP1 and XP SP3).

I suspect that the reason why it might have worked in the past is because MsgWaitForMultipleObjects behind the scene (according to MS engineers) also waits for a signal on the current thread (in addition to signals on whatever handles you provide). I also suspect that on older OS versions MsgWaitForMultipleObjects did not fail and return immediately if one of those handles was invalid, meaning that it will continue waiting for a signal to the current thread, which would then get signalled when a thread message arrived and it actually would perform as intended.

On newer OS, it seems that MsgWaitForMultipleObjects always fails and returns immediately if one of the supplied handles was invalid for its purposes, e.g. window handle is supplied. It won't bother waiting for a signal on thread behind the scene.

It can be resolved in one of 2 ways:
A) MsgWaitForMultipleObjects(0, nil, False, TimeOut, QS_SENDMESSAGE); // wait for thread messages, window must be owned by the same thread. B) SetTimer(HWND, Timeout); GetMessage(HWND); KillTimer; // use blocking GetMessage, unblock it via WM_TIMER.

I found some other oddities but not ready to formalize them yet, I'll keep experimenting.

Denis

I think B is the way to go; So the code doesn't assume anything about threads.

Michael.

I got the Plan B working (SetTimer+GetMessage) for SimpleIPC.

However, it is not possible at all to get TSimpleIPCServer.PeekMessage to NOT read the message, i.e. to respect DoReadMessage=False parameter.

Every message queue accessing function in Windows (e.g. PeekMessage, GetMessage) automatically dispatches all incoming *sent* messages by directly calling the associated WindowProc method. So it's not possible to check if there are incoming *sent* messages in the queue without delivering all of only way to make TSimpleIPCServer.PeekMessage respect DoReadMessage=False parameter is to store locally all sent data streams and then feed them to user on ReadMessage call. Meanwhile, TSimpleIPCServer.PeekMessage with DoReadMessage=True can loop trough ReadMessage call until there are no more buffered messages left.

There is a concern that locally buffered data streams will keep growing if server stops processing them while clients keep sending them. We can either (A) delegate this concern to the user of the IPC server, or (B) hard code some limit on the maximum number of buffered data streams. Any recommendations?

There are alternative IPC mechanisms available in Windows (Pipes, Sockets, DDE, etc), but that might be like opening another can of worms. I'm still confident that we can get a fully functioning IPC going using WM_COPYDATA, it just might get a bit more complicated.

Denis

Denis Kozlov wrote

There are alternative IPC mechanisms available in Windows (Pipes, Sockets, DDE, etc), but that might be like opening another can of worms. I'm still confident that we can get a fully functioning IPC going using WM_COPYDATA, it just might get a bit more complicated.

If you don't persist in using SimpleIPC, there is also AdvancedIPC that uses the same approach (temporary files) across all targets and so it behaves consistently.

Ondrej

In fact, to address some other issues in simpleipc, my plan was to do just that for all platforms: keep all messages in a linked list or queue, so that peek message can just check in the list, and if it is empty, try to get the next message.

So please go ahead and implement this. I wanted to introduce a property MaxMessageBufferSize which keeps the maximum amount of messages that can be held in memory, and if more messages arrive, discard old messages. (or introduce a enumerated property that tells the class what todo TBufferOverFlowAction = (baDiscardOld,baDiscardNew,baError)

Well, WM_COPYDATA is - under normal circumstances - the most efficient mechanism.

Michael.

Denis Kozlov wrote

The only way to make TSimpleIPCServer.PeekMessage respect DoReadMessage=False parameter is to store locally all sent data streams ....

TThread does contain a queue for "TThread.Synchronize" and "TThread.Queue". It is decently managed for the main thread and polled by "checksynchronize()" with automatically is used by "event programming" environments such as Lazarus or mse ide.

Perhaps it would make sens to (optionally) forward the messages there that weirdly are pre-fetched by Windows.

In fact it might be useful to optionally allow to forward all received events (as properly formatted event functions) in that queue and so to use checksynchronize() and with that the Lazarus and mse ide to receive messages from other applications in a way compatible to the "event programming" paradigm .

-Michael

Ondrej Pokorny wrote

If you don't persist in using SimpleIPC, there is also AdvancedIPC that uses the same approach (temporary files) across all targets and so it behaves consistently.

I think SimpleIPC and AdvancedIPC complement each other. SimpleIPC uses native IPC mechanisms, while AdvancedIPC uses a generic (file based) mechanism. Both have advantages and disadvantages, so it's great to have both available.

Denis

+1

Michael.

Interesting point! From this point-of-view I don't have to create the connection layer between SimpleIPC and AdvancedIPC as me&you suggested here:
http://lists.freepascal.org/pipermail/fpc-devel/2015-September/035958.html It makes sense.

Ondrej

Well, my initial enthousiasm cooled somewhat when I understood that you are using a file-based approach.

if the current simpleipc implementation can be fixed, then I think that is to be preferred above re-implementing it on top of advancedipc, because I am not very comfortable with the files approach.

The only thing that remains to be fixed in simpleipc is the multiple clients to server on unix. It needs a semaphore to be able to function correctly.

The file-based approach is still a valid approach, since it offers other things:
- the possibility of sending a message to a non-running server and
- receiving a response.
But for simple one-way traffic, I think simpleipc is to be preferred...

But you are right that you should not put any more effort in emulating simpleipc, I think it would be wasted effort.

Michael.

+1.

The only thing that remains to be fixed in simpleipc is the multiple clients to server on unix. It needs a semaphore to be able to function correctly.

There is also the issue that you can start multiple servers with the same ID on Windows, IIRC.

Yes, if the limitations of SimpleIPC aren't a problem.

Btw. I started AdvancedIPC because of the IDEInstances feature in Lazarus 1.6. I decided to use the file-based approach because I thought this is the simplest and most stable approach - I basically took a look at SimpleIPC and made a choice from its approaches. Nevertheless SimpleIPC uses files on Linux as well.
Unfortunately with file-based approach the IPC Server has to check new files (messages) regularly and thus could restrict power management, as Mattias pointed out. Some kind of socket communication would be probably better in this respect and probably also easier to fine-tune :(

Ondrej

El 08/12/15 a les 08:55, Ondrej Pokorny ha escrit:

Nevertheless SimpleIPC uses files on Linux as well.

FIFOs, not regular files.

Bye

I have rewritten SimpleIPC implementation for Windows.

Summary of changes:
1. Using SetTimer + GetMessage method for handling PeekMessage.
2. Store all received incoming messages in a message queue (per IPC server instance).
3. Implemented message queue (PeekMessage receives messages, ReadMessage picks from the queue)
4. Implemented optional message queue limit and overflow handling (more on that below).
5. Timeout<-1 is forced to -1; Documented meaning of various Timeout values in comments.
6. Minor refactoring and code formatting changes.

All these changes currently affect only the Windows implementation. The main IPC interface has not changed. This allows to fix all major issues on Windows and test the new concept of buffering/queueing IPC messages.

A new global type was introduced:

TIPCMessageOverflowAction = (ipcmoaNone, ipcmoaDiscardOld, ipcmoaDiscardNew, ipcmoaError);

Added two global variables which configure message queue limit and overflow handling (affect newly created IPC servers):

DefaultIPCMessageOverflowAction: TIPCMessageOverflowAction = ipcmoaNone; DefaultIPCMessageQueueLimit: Integer = 0;

Currently these are used only in Windows implementation, but they need to be in the interface section to allow unit testing and perhaps later reuse on other platforms.

The second part is a comprehensive test suite for Windows implementation of SimpleIPC. Tests spawn client mode instances as needed for proper testing of IPC. Tests check handling of messages, processing delays, synchronicity and message queue overflow actions. Read comments for each test routine for additional information. All timing based checks are very relaxed and can be controlled by IPC_BASETIME constant (a base line time per message in milliseconds), which can be increased for really-really slow machines. Simply compile and run. Exception is raised if any errors occur.

Other platforms will fail most tests due to not implemented featured, such as message queue and overflow handling. Perhaps it could be added to FPC test suite for Windows platform only, but I'm not sure how suitable it is due to complexity of multiple processes and timing checks.

I also tested separately for compatibility with Lazarus message loop by using PeekMessage(0, True) in Application.OnIdle, it works perfectly.

Attached files (full copies included for convenience; patches against FPC trunk revision 32628 from 10 Dec 2015):
1) testipc.pas/testipc.out.txt - tests for SimpleIPC and example output.
2) simpleipc.pp/simpleipc.pp.patch - patch and a full copy of main SimpleIPC unit.
3) simpleipc.inc/simpleipc.inc.patch - patch and a full copy of Windows include file (not Windows CE).

Next steps:
1) If everyone is happy - commit SimpleIPC changes (should I create a bug report in mantis?).
2) I can merge "wince" with "win" implementation since it is identical except for wide types/winapi, but won't be able to test on Windows CE.
3) See if implementations on other platforms can use/benefit from message queue and overflow handling.

How's that sound?

Denis