Hello,

in http://bugs.freepascal.org/view.php?id=1866 is described, that forms in dlls are not working at this moment. I have coded a dll which opens a form and it works. But I am not sure if there are some hidden problems or side-effects. Before I implement this in a real application, I want to read some opinions.

Here is the code for the DLL:

---->8---->8---->8---->8---->8---->8---->8---->8----
    library server;
    {$mode objfpc}
    {$H+}
    {$DEFINE USE_BIN_STR}
    uses
      Classes, SysUtils, Interfaces, Windows, LCLType, Unit1, Forms;
    {$R *.res}
    procedure ShowText(AText: String); stdcall;
    begin
      try
        Form1 := TForm1.Create(Application);
        Form1.Memo1.Text := AText;
        Form1.ShowModal;
      finally
        FreeAndNil(Form1);
      end;
    end;
    exports
      ShowText;
    begin
      Application.Initialize;
    end.
---->8---->8---->8---->8---->8---->8---->8---->8----

The Unit1 contains a form with a TMemo on it. I can show the form and load some text into it via:

---->8---->8---->8---->8---->8---->8---->8---->8----
procedure DllShowText(AText: String); stdcall; external 'server.dll'
name 'ShowText';
// ...
DllShowText('Blafasel');
---->8---->8---->8---->8---->8---->8---->8---->8----

Is this a bad idea?

Thanks for any help.
Michael

2012/4/30 Michael Fuchs :

I have once also experimented with this but the host application was *not* written with FPC/LCL (it was a proprietary trading and charting software (MetaTrader4) that had a built in scripting language with rudimentary support for importing dll functions that would load my dll and call functions). The problem was that all these function calls into my dll did not originate from the main GUI thread, of the host application. I was able to show a form but it was "frozen", it did not receive any events.

I ended up starting a separate thread in my dll, told the RTL that this new thread is now the main thread (and was surprised that this actually worked) and then wrote my own message loop (because application.run did not seem to work properly). I also had to synchronize all other function calls from the host with this thread because they came from even more different threads, At the end it worked but it all looked so ridiculously complicated and fragile and I had the impression it was sheer coincidence that it actually worked so that I finally decided to abandon these experiments and instead make the gui part of my dll a separate .exe file that communicates via some sort of IPC (I ended up starting the GUI exe with a TProcess and send/receive simple commands via stdin/stdout). This still seemed somehow complicated but at least it did no dirty undocumented tricks with the LCL and seemed quite robust (and enough for my needs).

Bernd

This is almost exactly the way I wrote plugin architecture for my apps. The difference is that the dll function call (which opens a form) is blocked within Application.Initialize and Application.Terminate, so the dll function (and the form) lifetime is only along the executed function. The only currently missing feature (that I need) using this approach is that if the dll contains form that must be shown as modal, I can't pass my main form's instance to make it modal (the dll form is still non-modal).

I ended up starting a separate thread in my dll, told the RTL that this new thread is now the main thread (and was surprised that this actually worked) and then wrote my own message loop (because application.run did not seem to work properly).

Up to here I'd think that a new TApplication type for DLLs would be a nice solution.

I also had to synchronize all other function calls from the host with this thread because they came from even more different threads,

This sounds a bit weird. Or is the DLL *assumed* to synchronize all these calls with the GUI thread (be application or DLL main thread)?

I'd thought that a SendMessage would serve the same purpose (thread switch).

DoDi

2012/5/1 Hans-Peter Diettrich :

This sounds a bit weird. Or is the DLL *assumed* to synchronize all these calls with the GUI thread (be application or DLL main thread)?

No, I didn't expect the LCL to do this, this was just the last straw that made me stop these experiments, when I discovered that due to the chaotic way this specific host application worked I had to jump through even more complicated hoops.

The main reason really was the main thread and my message loop, I just didn't trust my undocumented LCL (mis)usage. It just felt wrong, maybe also because I did not completely understand all aspects of the problem.

If there were an official documented way to to this and it would be robust and just worked under all cicumstances then it would have been ok.

Without doing something special a DLL does have it's own memory manager (because the calling program might be not an FPC program and uses an incompatible memory manager) but it does not have it's own Thread (because the original purpose of a DLL is just providing callable functions).

So you can't (carelessly) pass FPC constructs lice objects or strings to a DLL, because the memory management might fail. There are ways (at least with Delphi) to make a DLL attach to a main programs memory management.

A Form uses the LCL "Application" and thus the "main Thread". Creating a second Main Form (-Thread) is not supported by the LCL.

-Michael

Michael Fuchs wrote

Is this a bad idea?

No, I think you made it the proper way. There are no objects being passed between the executable and the library and you also utilized ShowModal. I think that this approach has everything for it to work in all desktop platforms.

Felipe Monteiro de Carvalho wrote

No, I think you made it the proper way. There are no objects being passed between the executable and the library and you also utilized ShowModal. I think that this approach has everything for it to work in all desktop platforms.

So, there are no interference betwenn the two Application objects?

Good, I will follow this way for my application.

Thanks to all.

Michael

Michael Fuchs wrote

So, there are no interference betwenn the two Application objects?

While I do believe, that it is possible to have two TApplication instances, one in the Main application and one in the DLL, I feel that there are a lot or problems that need to be handled.

In fact I once suggested that it would be very nice to have, if Lazarus would provide a "DLL/so Application" Widget Type", But I think it would be a lot of work to create same in a decent way. So beware !

Besides the obvious benefit of "two main Forms", this additionally would allow for the (then duplicated) LCL GUI functions to take advantage of multiple processors.

Some of the problems are:
- You need a dedicate "main"-thread for each TApplication instance, to allow communication with the OS GUI.
- You need a common memory manager to allow for a decent communication between the main application an the DLL.
- Communication between the two mainthreads needs to be done via a mechanism based on "Synchronize" and/or "QueueAsyncCall".

Rather hard stuff :) .

-Michael

So using the LCL, is it possible to e.g. put the code associated with a right-button menu in a DLL, and have it interact with e.g. a SynEdit on the main form?

Mark Morgan Lloyd wrote

So using the LCL, is it possible to e.g. put the code associated with a right-button menu in a DLL, and have it interact with e.g. a SynEdit on the main form?

I do believe this is possible, but I am sure that nobody has tried this, yet and there of course will be lots of issues to handle.

The LCL (i.e. the GUI-related functions) is not reentrant: it can't be used by multiple threads. As it is instantiated by by the programs the main thread in the startup code of an "Application", it only can be used by the main thread, (A DLL does not have a "main thread" by default.)

This means (besides other thread- and external-widget-set- issues) that there are static variables in memory that are (and need to be) preserved between calls to functions in the LCL. Same are not (and need not be) protected against concurrent calls from multiple threads.

The DLL has it's own static memory, that is why all the static variables of the LCL instance in the DLL are independent of those in the main program. This it is "dual entry" (or in fact completely independent as the code also is duplicated) regarding two different threads.

Moreover when (with it's own thread) instantiating this copy of the LCL TApplication, it on it's own attaches to the OS Widget Set and thus I _believe_ the Widget Set handles it as if it would be a completely different program. In fact the OS widget set _could_ find out that this call is done by a program that already did attach and create a main Window and act appropriately (either by creating a somehow depending sub-main window or somehow unexpectedly so that the intended function can't be accomplished. But I don't see why it even should do so.

Good luck !

-Michael

OK, but if the DLL was specially written, i.e. we're not talking about an arbitrary mix of programs and techniques, and if it specifically did not encapsulate an explicit thread, and possibly if it could only be called from a single app, what then? Am I correct in believing that the major issue would be that the DLL had its own memory manager and globals, so the issues would boil down to manipulating strings and objects (e.g. the text in an editor)?

I'm way too low on the learning curve to be stupid enough to try this, but I was musing on whether a collaborative app could detect that part of itself had been updated in SVN, and recompile the relevant bits on the fly.

Sometime ago I've tested .so stuff under qt and gtk2 and it works fine. New form from dll attaches itself into main loop w/o problems, so it can be modal,non-modal ... whatever. Cannot remember atm what is needed to be done inside library (cannot find doc I wrote about it) .. but think that Application.Initialize should be called in library and that's all. I've explained how it works or on mantis (and dll related issues) or on forum. In this case (under gtk2 and qt) ,application recognizes library "application", so it attaches it to it's main loop.

zeljko

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20120503/d191568b/attachment-0001.html>

Mark Morgan Lloyd wrote

specifically did not encapsulate an explicit thread,

Without a dedicated thread the GUI events are not handled and thus you can't have a dedicated

zeljko wrote

Sometime ago I've tested .so stuff under qt and gtk2 and it works fine.

In fact I did test this with Delphi on Windows. Works fine as well.

In fact did not do it in a DLL but just created a thread and had it create a new TApplication instance. Of course this crashes because of reentrance issues, but the GUI/OS attaching seems to have worked easily.

-Michael
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20120503/04fbefbc/attachment.html>

Michael Schnell wrote

On 05/03/2012 11:11 AM, Mark Morgan Lloyd wrote: > specifically did not encapsulate an explicit thread, Without a dedicated thread the GUI events are not handled and thus you can't have a dedicated , but what if the form was actually declared in the main program?

Could code resident in a DLL (and called from the main thread) interact with the form to e.g. set up main- or context-menu entries, or at least work around the memory manager issues sufficiently to be able to return strings representing menuitem captions and entry points representing click handlers? And could the handlers then interact with the main-program form(s)?

The DLL has it's own static memory, that is why all the static variables of the LCL instance in the DLL are independent of those in the main program. This it is "dual entry" (or in fact completely independent as the code also is duplicated) regarding two different threads.

That's why I suggested an special TApplication type for DLLs. When used by a Lazarus application, the DLL could be instructed to use the main Application object and memory manager.

This were close to the Delphi handling of packages, except that shared code resides in the application, while it resides in further packages in Delphi. This model doesn't prevent code duplication, because the DLL could be used by non-Lazarus apps as well. Using another DLL for the LCL would solve this, but most probably will fail due to other (RTTI...) problems.

DoDi

On 05/03/2012 01:40 PM, Mark Morgan Lloyd wrote:

So you want to get along with a single thread, but want to define the Form in the DLL ?

I don't think that this is possible using the LCL in the DLL. But it should be possible to pass the resources, stored the DLL, that are used to construct the form and it's content to the main program and have the LCL of same construct the (now non-main) form. But as finally the appropriate GUI events will need to call functions in the DLL, I suppose using a combined memory management is absolutely necessary.

-Michael

Mark Morgan Lloyd wrote

Could code resident in a DLL...

Here you will need to do Object-based mutual calls between the main program and the DLL. This of course is possible, but needs a combined memory management.

-Michael

Yes. How's it done? :-)

Michael Schnell wrote

On 05/03/2012 01:40 PM, Mark Morgan Lloyd wrote: So you want to get along with a single thread, but want to define the Form in the DLL ?

No, I want to control a preexisting form from a DLL: tell it what menu structure to display etc., even if that meant that they were built up step-by-step under code control rather than constructed from a description in resources.

In the limiting case a- plugin, for want of a better description- adds a single item to a context-sensitive menu, and provides the functionality to implement that item.

Mark Morgan Lloyd wrote

even if that meant that they were built up step-by-step under code control rather than constructed from a description in resources.

Yep. If you don't want to create a tool that uses the resources in the DLL and constructs the GUI from same, you can't use the Lazarus form designer and need to write code to manually add the controls you need for your plugin. IMHO it should not be too hard to do such a tool, as this is exactly what is done by the program startup code, anyway. You just need to take a look at the LCL part that does this (TApplication.CreateForm, when creating a non-Main form) and take out what you think is useful for that purpose. But (as said) you need to use the LCL of the main project, not that of the DLL. So the tool should be done in the main program, so that it attaches to the correct Application instance. You then need to pass the address of the resource to same when the DLL is "started" (e.g. by providing a function that returns that address). Beware, that the memory management should be unified before.

-Michael

Mark Morgan Lloyd wrote

Yes. How's it done? :-)

I do do this with Delphi, but did not try with Lazarus.

AFAIK there is a page on this in the Lazarus Wiki.

-Michael

One thing does occur to me, and that is that if the IDE was used to create a form stored in a DLL (or, presumably, a shared object library) then it even if it's not animated it should be usable as a template for copying to a blank form in the main program. And code to do that, once written, should be fairly general.

FOUL! :-)

but did not try with Lazarus. AFAIK there is a page on this in the Lazarus Wiki.

Is this what you're thinking of, or is there more that I've so far overlooked?

http://wiki.lazarus.freepascal.org/shared_library#Sharemem_implementation_details

At least in Delphi ReadComponentResEx can be used to load the form components into another form (or compatible control). The links to event handlers must be handled somehow, of course.

DoDi

Mark Morgan Lloyd wrote

> I do do this with Delphi, FOUL! :-)

Only for testing, of course. I have but the old free Turbo Delphi. ;-)

Yep.

My colleagues who are very professional Delphi programmers, very happily use this with Delphi:
http://sourceforge.net/projects/fastmm/

This memory manager makes many Delphi applications a lot faster (it manages short, medium and big chunks separately, for "short" it allocates a pool of memory blocks of equal size that can be found by multiplying the number by a constant instead of walking a linked list). Moreover it allows for tracing and synchronizes DLL and main application memory managers.

I don't know if/how it can be used with Lazarus.

-Michael

AFAIK the standard heap manager of FPC uses chunking too.

I don't know if/how it can be used with Lazarus.

Mattias

Great !

So supposedly a replacement is not necessary for speed purpose.

Maybe it also can do tracing on request...

-Michael

Thanks, I'll bear all of this in mind. I've also seen discussion and an example from a Metatrader forum.

I'm slowly working towards a novel collaborative dedevelopment app, and if there's a "root" manager coordinating overall work then it seems reasonable that he should have the ability to tweak "leaf" workers apps- for example to add a new context-sensitive highlight. If that can be done without everybody having to interrupt their session, possibly losing state, so much the better.

/If/ I decide to play with this, then I've got the choice of either waiting until I rough out the collaborative stuff, or trying it in one of the interim apps that I'm using to test techniques. I'll either be asking for help, or reporting stunning success :-)