Greetings,

I am familiar with the basic underlying methods available for transferring data between processes on Windows and Unix, i.e. Pipes, Shared memory, and TCP/IP - but what I am not familiar with is any higher level functionality that may be available on FPC.

As an example: I have one application with a daemon that uses the IPC component to write to a file in CSV format, and then the user application reads this (GPS location) data via IPC. Then I have to re-convert this string data into a series of floating point values manually, though. The IPC component doesn't seem to be reliable on all platforms either (it sometimes blocks on OS X, and at least the debug client doesn't seem to work at all on Windows 7).

Another disadvantage is that the sequence of launching the applications matters, and what's more, it seems there can only be one "client" per "server" in many cases.

More to the point, if I want to pass around structures, records, etc.between processes - what is the best day? For example, Javascript uses JSON. I know there are JSON libraries and various other libraries for Pascal - but I don't know what is the most standard way. (I think this kind of marshalling/unmarshalling may be more standard in Java and C#, but I don't know the "normal" way in Delphi/FPC).

I would prefer to use built-in functionality, rather than learn yet another library - and if learning a library, I would prefer to use one with lots of users that is actively maintained. Likewise, I would prefer to actually "pass" the data, rather than just pass a pointer to it. I plan to have the processes run on the same machine, so I don't need a solution that works with networking, though that would be fine, of course.

Along the same lines, a convenient way to call functions/procedures with parameters from the "other" process would be greatly appreciated. (i.e. something like RPC that handles OOP).

Thank you,
Noah Silva

印場 乃亜 wrote

Along the same lines, a convenient way to call functions/procedures with parameters from the "other" process would be greatly appreciated. (i.e. something like RPC that handles OOP).

Take a look at MTDORB [http://sourceforge.net/projects/mtdorb/]. It is a free and opensource ORB for Delphi and Kylix. It is intended to be fully compliant implementation of CORBA 2.6 standard.

At one stage I was working on getting it FPC compatible, but never finished the job. CORBA is a OOP way of working with remote objects and remote procedure calls and lots more.

Regards,
- Graeme -

I'm writing at the moment support for msgpack (http://msgpack.org/) in plain FPC (object pascal style):
https://github.com/ik5/fp-msgpack

I will also implement the rpc version.

MsgPack allow you to take data, including JSON and convert it into bytes that represent the data.
It contain smaller version of the same way that JSON does it for example.

You should try WST.

It's perfectly suitable for all this. It is actively used and maintained. You can choose which encoding (binary, soap, JSON) is used, and which protocol (Direct library, TCP socket, http).

Michael.

I have to second Michael's response. Inoussa discussed on some thread some time back and in fact I am lookiing to read it again. Does anyone recall that thread?

I have a very simple unit that uses RTTI to stream class instances to and from strings/streams.

i believe this is an easy foundation on wich to develop something along the lines of what you are trying to do

2012/10/29 Frank Church :

Hi,

I will take a look at WST and see how it fits. Thanks for the tip.

Thank you,
Noah Silva

Hi,

I read about MSGPACK before, but it seems to be a binary version of JSON, right? I actually don't care about the protocol much (and since it's local, Binary vs. text wouldn't make a lot of difference in transfer speed - though using FPC native data structures would be faster since they wouldn't have to be converted to an external format).

The problem isn't just the transfer format, but the whole process.

For example, I want to have something like the unix message (Signal) system that allows me to send not just an Integer, but any random data.

For example, let's say I have a daemon running that receives chat messages (or something) and then it wants to send the data to the GUI, so it would call something like:

SendDataToApplication(Gui_App, ChatDataRecord);

Obviously, I could manually encode that data into JSON or CSV (or msgpack) and manually send it to the other app by IPC and manually decode it on the other side. What I want is some system that will know that the incoming message is of type TCharDataRecord (or whatever) and automatically let me use that type (and not forced typecasting with pointers, etc. - but properly). Obviously there would be limitations to any solution - for example, I would have to have the same types available on both sides, etc.

I will take a look at msgpack, though, to make sure it can't do the above, though. What little JSON I have done in FPC, I have done manually thus far, so perhaps I need to learn the capabilities of the real units before asking more.

Mainly my question was asking if there is a "standard" way to do this, since it seems like a common thing to need to do.

Thank you,
Noah Silva

Simple solution :

Use TSimpleIPC components + my own RTTIObjects unit.

All messages must be descendants of TRTTIObject. (You can use TPersistent, but i believe my unit is neater for this specific task).

Well, using only standard components i believe you should use TSimpleIPC components + TPersistent classes

All you need is a way to stream the TPersistent into a string or stream then pass that resulting string/stream to the simple ipc components.

That will do what you are asking, without much fuss...

2012/10/30 印場 乃亜 :

Ah, and you will need a class factory to register the tpersistent class creators...

And i said my unit is neater ?

here it is :

http://code.google.com/p/pascal-actor-model/source/browse/trunk/rttiobjects.pas

I use it on a lot of things...

All you need is to declare published properties that you want streamed and use TRTTIObject.SaveRTTIToStream.

It will save on a binary format where the first string is the class name. You should have a class factory on the other side that registers all classes that can be exchanged between applications.

Pascal being statically typed (and binary creating instead of bytecode creating) means that you cannot "slurp" an object from one application to another (IE.: Stream an unknown object around).

Each class should be know in advance by both applications.

2012/10/30 Jorge Aldo G. de F. Junior :

Hi,

Wow, Class Factories - I am having nightmarish flashbacks to my college C++ classes.

Okay, Thank you for all of the suggestions. I will try to compare the suggestions, and see what I can come up with for my situation. Just for further information:
1. I wasn't really planning on using the network, so using something that has to pack/unpack the data from FPC to other formats might be overkill.
2. I am happy to pass records, Objects aren't really necessary. (If you pass an object in binary, the internal pointers would be referencing the old copy anyway).

I was actually considering using the GUI messaging system like exists in Windows, but:
1. It doesn't really exists on non-windows systems (I read some about one being created for Lazarus on Linux)
2. I seem to recall that it only allowed passing relatively simple messages. I thought perhaps I could use shared memory and then send a message telling the client application to read the record.

Maybe once I examine all the options, I will make a FAQ ;)

Thank you,
Noah Silva

2012/10/30 印場 乃亜 :

formats might be overkill.

For the data delivery part, you can use TSimpleIPCServer/TSimpleIPCClient

2. I am happy to pass records, Objects aren't really necessary. (If you pass an object in binary, the internal pointers would be referencing the old copy anyway).

The problem with records is that the data sending/receiving function must know the record structure in advance, or use a very inflexible record strucuture (one size fits all), or use a much more complex protocol with multiple entry points selected by a prefix byte or something similar.

in the end of the day, classes tends to be simpler to pass around.

Think about this :

Class TA has a property named pname, another property named pvalue.

Instance A is an instance of TA.
Instance B is an instance of TA.

Instance A has the property pname set to 'somename' Instance A has the property pvalue set to 10

Instance B has the property pname set to 'somename' Instance B has the property pvalue set to 10

Is A=B ?

If you simply look at the pointer (classes are internally pointers to data structures in memory), then no, they are different.

But if you look at the values, then they are the same.

What i want to mean ?

If you can, using some IPC/whatever, make a new instance on the other application, of the same class, with the same values, then (for a very wide interpretation of the meaning of transporting) you transported the class from one application to another...even if they are not exactly the same instance...

that means that you dont need shared memory nor any fancy trick, just a simple protocol like

<classname> <property1value> <property2value> ... <etc>

where classname is used as an index on a class factory and property1values are used to set the class properties

will do the trick.

(There are some languages that can LITERALLY transfer an instance between two applications, but Pascal cant due to being statically typed).

I was actually considering using the GUI messaging system like exists in Windows, but: 1. It doesn't really exists on non-windows systems (I read some about one being created for Lazarus on Linux)

TSimpleIPCServer/TSimpleIPCClient is portable across all FPC targets.

2. I seem to recall that it only allowed passing relatively simple messages. I thought perhaps I could use shared memory and then send a message telling the client application to read the record.

Shared memory is usefull when you need to transfer large blocks of data fast between two instances. I believe pipes are simpler for simpler tasks (And thats what TSimpleIPCServer/TSimpleIPCClient use).

TSimpleIPCServer/Client are standard components being part of the FCL. (Freepascal component library). At least they come standard with the compiler.

Hi,

Wow I am surprised this is continuing so long - I take it other people have interest in the subject too.

Which is what I have been doing so far - though I had to look into the implementation and use lower level calls in some cases to stop the app from blocking on reading - at least on OS X. (It would sometimes think there was data waiting, and try to read it, which would freeze the app). Also I have mainly been using CSV. It doesn't seem easily possible to have multiple clients hook to a single server either - though maybe I am missing something.

I think that last one if what the debug server protocol does.

in the end of the day, classes tends to be simpler to pass around.

wow... but if you pass a class, then the VMT, etc. needs to be re-done somehow. (i.e. the new copy needs to be actually instantiated, the constructor re-run, etc.).

By my definition, yes, but not by pointer logic. (Unless the language does some fancy COW stuff).

Right, if you can easily do this.

Well, it could with some compiler magic I think, if both sides had access to the same type. Actually, I suppose it would be more RTL magic than compiler magic, I suppose. The types are known at compile time (plus there is RTTI), so if the applications shared a common type library, it should seemingly be possible. Of course this approach would be relatively brittle since if anything changed about the type in one place, everything would have to be recompiled with the same change. Not much different from linking, I suppose.

> I was actually considering using the GUI messaging system like exists in Windows, but: > 1. It doesn't really exists on non-windows systems (I read some about one being created for Lazarus on Linux) TSimpleIPCServer/TSimpleIPCClient is portable across all FPC targets.

Supposedly, but see above.

Right, and I am talking about relatively small amounts of data, not movies or something. The reason I thought of SHM is that there would be no need to decode the structure into a string and re-encode it, etc. Instead, the application could somehow pass a message and a pointer. The receiving application then takes over management of that record (or whatever), and the sending application forgets about it (and allocates a new one next time it needs to send something).

Of course the devil is in the details, which is why I was asking about standard solutions.

Which is why I am using them now.

Actually, in case anyone is searching this mail archives for similar solutions in the future, let me say a little more about *why* I am trying to communicate between two applications on the same machine:
1. Using threads in FPC is a pain and has proven unreliable for me. (There needs to be some auto-sunchronization...)
2. I like to separate the logic between the core and the GUI (and what better way than to compile them separately?)
3. I might want to compile the core as 64bit with the GUI as 32bit (which is required on OS X right now...)
4. I might want to have multiple applications (or multiple users) to access the same core without weighing down the machine by running multiple copies of it.
5. The core might be in C or such, while the client is in FPC. (I do this on Mac OS where I write the core in ObjectiveC for easy access to the Cocoa libraries, and then export the data I need to FPC via a named pipe).
6. I might want to have different security on the two "applications". Interestingly OS X has taken this approach recently, but this is slightly different special type of app, and I don't think we have direct support for it in FPC yet.

In all but one of those cases, it's FPC to FPC, so I don't much care the format. I suppose in reality, something like RecordToBlob -> IPC -> BlobToRecord would work fine, as long as the BlobToRecord could detect the original type and return that type.

>

Thank you,
Noah Silva

Right, and I am talking about relatively small amounts of data, not movies or something. The reason I thought of SHM is that there would be no need to decode the structure into a string and re-encode it, etc. Instead, the application could somehow pass a message and a pointer. The receiving application then takes over management of that record (or whatever), and the sending application forgets about it (and allocates a new one next time it needs to send something).

This wouldn't work. You cannot pass a pointer to process A that references memory of process B. You would receive signal 11 (SIGSEGV). You could of course copy the block of memory (the object you're trying to transfer) into shared memory, which you then can access from both programs, but then I do not see the use of a message with a pointer (just start the object at offset 0 in the shared memory block).

Either way, this approach wouldn't be `safe`, as any pointer contained into the object/record/<whatever you call it> also needs te be re-allocated in the shared memory block. Thus strings and dynamic arrays would provide for quite a royal pain in the arse to `share` them.

Encoding the object solves this problem. This does not mean that the data cannot be in binary form however (-> no need for IntToStr and vice versa!). All integers/floats can still be stored in identical the same way (i.e. 1/2/4/8 bytes fixed length), and need not be encoded/decoded in any way, they just need to be copied over from the buffer to the appropriate location. Strings can easlly be stored as an integer (the length field), followed by the data of the string itself; just as dynamic arrays.

Thus if you have a class which contains a number and a string, you would encode it something like this (think as if you're saving to a file, but do not optimize for size):
<type:fixed length (say, one byte?)><integer from class: 4 bytes for objfpc><length field of string: also 4 bytes, could also be 2 if the strings aren't longer than 65535 chars><data in the string>

The type field is obviously needed for constructing the right class. Note that this type cannot be a `class of ....` type, as this is in essence also a pointer (to the VMT if I recall correctly). Then, as Jorge mentioned you only need to provide a neat little factory that `converts` these buffers to classes with which you can just continue in the usual OOP way.

Hope this helps.

Hm...

if you gave up using threads because of the problem with synchronization then you might have a look at my pascal-actor-model framework...

its a set of classes that implements Hewitt's Actor Model Concurrency and its able to solve exactly that...

http://code.google.com/p/pascal-actor-model/

Code can run assynchronously or synchronously, depending on your needs.

theres a mainthreadqueue that lets actors talk to the main thread (where the GUI elements usually reside) using blocking (with timeout) protocol.

All messages are actors and can be streamed across the network, etc.. (i am in the process of implementing distributed computing based on that actor model) etc...

there are already a lot of components and the basic actor functionality is already working.

I am familiar with the basic underlying methods available for transferring data between processes on Windows and Unix, i.e. Pipes, Shared memory, and TCP/IP - but what I am not familiar with is any higher level functionality that may be available on FPC.

If I'd do something in this scope, I'd adopt either a D-BUS technique - no worries about maintaining -, or go the easy way with shared memory - again no worries, you'll be the maintainer... ;-).

Just my 2c on it.

Cheers, A.

Hi Aldo,

Well it's not just synch problems with threads, I've found threads to not be so reliable under FPC for anything but trivial test cases. Sometimes the program is incredibly slowed when using threads. Also, Unix and Windows have to be treated differently, etc. - which is not entirely FPC's fault. Making separate applications gives a number of advantages (which I listed in my last mail). The need to send data structures back and forth is the only real disadvantage - but that's just trading for synchronization issues.

As an aside, threads are one area where it seems ObjC has a huge advantage. I wish there was a way to just say something like
"RunInBackground(procedure)" in FPC. Background threads not being able to touch the GUI, etc. makes it all but useless for many purposes. Of course for scientific computing type applications where problems can be partitioned neatly, threads make perfect sense and work relatively well. For things like "I want to save this file in the background while the user continues to use the word processor", I've found they aren't anywhere near worth the trouble to implement in FPC - yet.

For example, if I want only one instance of a daemon running, then I have to make it a separate process (reasonably anyway). If I want it to be 64bit, but the GUI has to be 32bit, then it has to be a separate process, etc.

I will take a look at your framework, because I am interested in all sorts of new developments, but it's unlikely I will re-code everything to use a particular design pattern or framework.

The original question I meant to ask was basically "Is there an easy/mostly automatic way to transport data structures between processes", and from all the discussion on this list - the answer seems to be "no." Encoding is one piece, data transfer is one piece, and the glue in-between (Class factories, etc.) is something one probably has to put together themselves. Either way, it makes everything more complicated to do something that is in principle relatively simple.

Thank you,
Noah Silva

2012/10/31 Jorge Aldo G. de F. Junior

i reiterate that my Pascal-Actor-Model can do exactly what you are saying...

writing a "save file in background" type of actor is trivial, and the synchronization is already done...

2012/10/31 Noah Silva :

Hi Jorge,

Even if so, it won't solve many of the other problems I am trying to solve. On the other hand, do you have a Tarball available for download somewhere? (I checked the Google Code page, but no downloads, only SVN access it looks like).

Thank you,
Noah Silva

2012/11/1 Jorge Aldo G. de F. Junior

Noah Silva wrote

As an aside, threads are one area where it seems ObjC has a huge advantage. I wish there was a way to just say something like "RunInBackground(procedure)" in FPC.

Once anonymous methods are implemented (I don't give an estimate here though) you should be able to do something like that.

Background threads not being able to touch the GUI, etc. makes it all but useless for many purposes.

That you can't access the GUI is not a problem of FPC, but a design decision of the LCL (or better: the VCL which it is based upon). If you'd write a GUI toolkit yourself you could write it in a way that allows access from multiple threads (if the underlying GUI system (WinAPI, X11, etc.) allows it of course)...

Regards,
Sven

Or even better: the graphical toolkit the LCL is based on. E.g. on Mac OS X, Carbon requires that GUI manipulations happen from the main thread. Cocoa is more lenient, but even that one has a lot of restrictions and caveats when interacting with the GUI or graphics from secondary threads. Of course, the LCL can/could try to hide this from the programmer at the expense of a bunch of overhead.

Jonas

I've added a source zip to the downloads section of the pascal-actor-model google code page.

while it cannot do "runinbackground(procedure)" as you want, i believe it can be mimmicked using the TCreateInstanceAndConfigActorMessage message and a functor actor (functor is a kind of object that holds "functions/methods", its a way to mimmick first class functions in a non-functional language like pascal because it allows to pass function references/contexts around as classes).

This message can create an actor and pass config info to it in one go, with guaranteed message delivery and correct sequence. (This avoids the unbounded non-determinism problem).

Actors usually sit in background doing work or idling. Theres an idle procedure that can be overridden and you can write message receiving methods that are triggered upon specific messages.

Actors wont stop until told to stop.

There are a lot of shortcut procedures in the actors.pas unit that avoids too much verbosity in the code.

If an actor should stop by his own volition, you simply have to set Running to false after all needed work is done :

Running = False;

There is a request/reply model of message passing, but it should be avoided at all costs, because it basically stops the thread until the receiving thread replies (or the timeout expires), the end result is an almost sequential execution (Makes no sense usually).

The main thread can wait for messages comming from the actors (with timeout). Its not an active wait (in the sense of a loop) but a passive wait done using TEventObject.

Usually the correct point to wait for messages comming from the background actors is inside the application idle loop, via the idle timer.

Example :

lMessage := ReceiveMessage(500); // waits 500ms for a message to arrive

To send message to actors :

SendMessage(lMessage);

In your example about saving files in background, you can write a file writer actor and send the content to him as a message. It will happly save the data to the file while the main thread is free to do whatever it should do, in parallel.

I believe i should write documentation but i dont know how to use pasdoc yet :P

2012/11/1 Noah Silva :