Hi,

In the light of recent fppkg and fcl-web discussions:

I've committed simple HTTP server (threaded or not) and HTTP Client components to FCL-Web. They work using the fcl-net ssockets unit. (for which a fix has been committed as well). Both use blocking sockets.

The server component is integrated in the rest of fcl-web, the client is rather stand-alone.
Both are designed to be easily extensible.

There are examples of how to use both. The server example for instance allows files to be served and will properly report a 404 when a file is not found.

To do this, I committed a fpmimetypes unit to fcl-base, so the server can correctly determine the mime type of a file. It can read the contents of a mime.types file as found on unix.

if someone knows where this file can be found (or related information) on a Windows OS, please let me know.

This is an initial implementation. Known to-dos:
Client:
* Maybe add proxy support ?
* Easy access methods for POST/DELETE etc. commands. (currently only Get exists as a method, the rest can easily be done with HTTPMethod)

Server:
* Create a TWebHandler and THTTPWebApplication for use as a Lazarus project.

Comments and suggestions welcome. Please report bugs in the bugtracker.

Michael.

Great stuff Michael. I can see many uses for the HTTPClient alone without having to use a third libraries.

Any idea when THTTPWehHandler/THTTPWebApplication objects will be included? I imagine its a bit of work. It would make development of fpweb applications a lot easier and more convenient though. I'm still trying to wrap my head around the innards of fpweb unfortunately.

Well, I expect it to be ready this or next week, so no need to wrap your head around it afer that ;-)

Michael.

Sweet. I'm looking forward to it. Let me know if you want help testing.

Will this allow to create a project with a browser based remote GUI ? Will this help to create a Lazarus Application running as a Daemon on an Android device and have the user access it via the standard Android browser ?

And my all-time favorite question ;-)
Will this (on Linux) allow for TTimer, TApplication.QueueAsyncCall, and TThread.Synchronize without a binding to an external Widget Set (or X-Server) ?

Thanks for enhancing Lazarus !
-Michael

Yes.

Will this help to create a Lazarus Application running as a Daemon on an Android device and have the user access it via the standard Android browser ?

Yes.

Well, TThread.Synchronize works already as-is without widget-set, provided you call CheckSynchronize in the main thread on a regular basis.

And for the other features: no, this will not allow for it, but also will not interfere with it. They are just components that implement the HTTP protocol: sending and receiving text messages. No more, no less.

Michael.

> Will this allow to create a project with a browser based remote GUI ? Yes.

Great. looking forward to test it.

> Will this help to create a Lazarus Application running as a Daemon on > an Android device and have the user access it via the standard > Android browser ? Yes.

Great. My son does have an Android device, and he also did succeed in setting up Android/386 in a virtual machine on his PC. So we might be able to test this, too.

Polling of course is not a decent way to handle
inter-Thread-communication, especially as "on a regular basis" (e.g. each 200 msecs, while the main Thread continues to be responsive) is not possible without a TTimer.

And for the other features: no, this will not allow for it, but also will not interfere with it. They are just components that implement the HTTP protocol: sending and receiving text messages. No more, no less.

Anyway, this seems to be a good starting point for a full featured "Widget Type". I'm sure that its possible to add an Event Queue re-using and improving the code provided in the FPGui Windget Type, which already (maybe partly) implements the Event Queue for TTimer, TApplication.QueueAsyncCall, and TThread.Synchronize based on the native Linux (no-GUI) API.

As always, I vote for an implementation outside of a certain Widget Type's structure so that the same code can be used for all appropriate Widget Type implementations. Hopefully, Greame might be able to help with this. Unfortunately, when I try to start my testing project compiled against FPGui, same crashes immediately after starting. So I can't see what the actual state of FPGui is.

Thanks again,
-Michael

Any implementation will always use polling. You may hide it under some kind of abstraction, but it's always polling. On unix with select() or epoll(), on Windows with PeekMessage/GetMessage and friends. Your Delphi windows timer uses PeekMessage/GetMessage by the message pump as well.

I think you're mistaken there. Till a couple of months ago, fpGUI's TTimer was buggy, it didn't react unless you moved the mouse in the app, both in Windows and Linux if I recall corretly.

While TTimer has been fixed almost immediatly by Graeme after I reported it, the above shows that it is tightly intertwined with the GUI.

Michael.

buggy, it didn't react unless you moved the mouse in the app, both in Windows and Linux if I recall corretly.

I remember it being Windows only - but that's not important.

While TTimer has been fixed almost immediatly by Graeme after I reported it, the above shows that it is tightly intertwined with the GUI.

Correct. While that design may not be ideal (please mention new designs if they exist), it is perfectly acceptable in fpGUI because after all, fpGUI is a 'gui toolkit' for 'gui applications'. ;-)

Regards,
- Graeme -

Lee Jenkins wrote

Great stuff Michael. I can see many uses for the HTTPClient alone without having to use a third libraries.

Definitely! As soon as FPC 2.6.0 is out, I'll use it in tiOPF's multi-tier support as well.

Nice work Michael.

Regards,
- Graeme -

Sorry, but you are wrong here.

Linux: select() and epoll() don't do polling (even if the name "epoll" is misleading). Theses API calls make the thread sleep with zero CPU time until one of the appropriate events is fired (by a Kernel driver or another thread) and then the Kernel wakes up the thread and same is scheduled according its priority setting (i.e. ASAP).

So the problems of polling are avoided. (Such as latency when doing a slow polling e.g. by means of "sleep()", and high performance overhead when doing fast polling).

With the GUI based Widget Types, the zero-overhead waiting is performed by the external GUI framework which is called by the user program after any main thread event is handled. Here the additional problem is, that the GUI generated events and the thread generated events need to be merged. With Windows, this is done by the System as the thread will generate an event as a a Windows Message so that the main thread is waked up and can receive the message with PeekMessage/GetMessage. This is not polling, as the GUI framework call returns only if some message is received. AFAIK, in Linux this merging is more complicated.

But with FPGui and the possible Widget Type discussed here, there are no GUI Framework events, and so e.g. epoll() can be used to do a zero-overhead latency free waiting for all possible events. (I suppose Graeme did his implementation similar to this.)

-Michael

Graeme Geldenhuys wrote

Correct. While that design may not be ideal (please mention new designs if they exist), it is perfectly acceptable in fpGUI because after all, fpGUI is a 'gui toolkit' for 'gui applications'. ;-)

I really would like to test the newest version :-) .

How is TTimer implemented ? Does it really have a binding to the underlying X ?

When I once was able to test it, I did see the problem that events scheduled by threads only were acknowledged by the main thread when a GUI event had been fired, too. So there was a binding to X. I have no idea how X events are received by the main thread, so I can't see what might be happening here.

OTOH, a GUI-free Linux Widget set could be done by using nothing but a single epoll() to wait for all events (e.g. Timers and Threads). I suppose your expertize on the implementation of the event queue itself should allow for removing the GUI binding and do an implementation that get's the main thread notifications from epoll(). Making the message queue itself a dedicated object in a dedicated unit outside of the Widget Type code would allow for a single implementation used as well by fpGUI as by GUI free Widget Types.

-Michael

Michael Schnell wrote

I really would like to test the newest version :-) .

I just fetched the newest update and not I can't compile it any more,.

I get
../../../../fpGUI/src/corelib/x11/fpg_interface.pas(26,3) Fatal: Can't find unit fpg_x11 used by fpg_interface

even if I explicitely include the directory with the fpg_x11.pas file in the unit search path and even it I add this file to the project :(.

Last time some months ago, when I was able to compile the project, it crashed immediately after starting.

-Michael

That's just name picking. Polling = asking a question.

From the point of view of the application, it is polling the kernel for availability

of data: the call returns as soon as there is data or the kernel decides there will be no data in a reasonable time frame.

So there is no 'event' involved at the application level, just a call to the kernel.

What you describe as 'polling' is the crudest form of user-land polling, which is of course very CPU intensive and to be avoided at all costs.

Michael.

Michael Schnell wrote

I just fetched the newest update and not I can't compile it any more,.

I haven't tried to compile or use LCL-fpGUI widgetset in ages... I'll see if I can take a look in the next few days.

Regards,
- Graeme -

That's just name picking. Polling = asking a question.

OK, to be clear we should use "busy polling" (according to a "busy loop" or spinlock).

But even not busy polling (that avoids performance killing for other threads and processes) is only appropriate, if all possible main thread events from all possible sources immediately take the main thread out of the loop, so that no unnecessary latency is introduced to any of them.

From the point of view of the application, it is polling the kernel for availability of data: the call returns as soon as there is data ...

Hmm.... As I see it, from the view of the application it is a function call that returns as soon as one of the attached events is scheduled by whomever. But supposedly this is exactly the same.

So there is no 'event' involved at the application level, just a call to the kernel.

For me "there is data" is exactly the same as an "Event". If this "data" is generated by a thread (and then transferred to the Kernel <e.g. using a semaphore or a pipe> ), for me, this is an "Event at application level".

An obvious Linux implementation for application events (such as necessary for TThread.Synchronize and Application.QueueAsycCall) could be using epoll() (as you already mentioned). epoll() could monitor one or more pipes. Another way would be to use a single semaphore (supposedly best done by pthread_mutex_...() calls) that manages the waiting on any events to be pushed to an event queue implemented in Pascal.

What you describe as 'polling' is the crudest form of user-land polling, which is of course very CPU intensive and to be avoided at all costs.

I was mistaken that you meant exactly this by the term of "polling", as I did not know the meaning of the term you obviously were thinking of.

-Michael

Exactly, I had thought about that as well :-)

Will you look at it, or would you like me to do it ?

Michael.

Yes, nice work... but what is the problem about third libraries?

Marcos Douglas

Marcos Douglas wrote

Yes, nice work... but what is the problem about third libraries?

Nothing per se. But having some vital functionality included in FPC just makes it so much easier to get a new developer system up and running (less to install and configure). Not to mention the fact that if it is included in FPC, it stands a better chance of being maintained and tested with each new FPC release. There are down sides as well... It might be harder to get commit access to some code living in the same repository as FPC.

Regards,
- Graeme -

OK.

Michael.

Graeme,
OK, I understood... you're right in these points.
But it's also reinventing the wheel, no?

Marcos Douglas wrote

But it's also reinventing the wheel, no?

That could be considered another down side, especially if the 3rd party library has a lot of functionality.

Other benefits:
- A "lite" version could be implemented in FPC - reducing time to implement, and easier to maintain. If the develop then wishes to have more functionality, they can always opt to use the 3rd party library instead.
- Sometimes re-implementing something is not all bad. After all the best code comes from doing something two or three times over - learning from your prior mistakes. I've seen this quite often in real-world projects.[*]

[*]
In one company I worked this was mandatory. We used to implement a prototype for clients. If the prototype was accepted, we deleted the prototype source code (keeping the binary only), then started from scratch with the source code. This might sound strange, but it worked amazing well.

vantages in this project to migrate, eg, the Synapse can do same, no? From what I have read, Michael implemented the bare basics of file/data

transfer. If you want any advanced features, or a different protocol (eg: ftp, nntp components etc), I would imagine Synapse would still be the better choice.

Regards,
- Graeme -

Right. I personally like and enjoy using synapse sockets library, but for simple http client access, I don't have to include it solely for that purpose. In a nutshell, my position is if I don't have to use a 3rd party library, I don't.

In addition to Graeme's excellent points, the need to prevent bloat notwithstanding, I think its a common pattern among programming platforms to support at least some usable high level objects for common coding in today's world, out of the box. Built in high level tcp/http abstractions are expected if you're coming from just about any other modern programming language/platform out there. High level abstractions for common things like XML, JSON, encryption, sockets are necessary and everyday tools in much of today's programming projects and other languages have helped set the bar in terms of what core functionality is expected for a modern programming platform.

Of course we understand this is all dictated by the specific dynamic of fpc/lazarus's respective developments, contributors' time and expertise, etc.

One thing I'd like to raise is SSL/TSL. For apps development via FPC as an industrial viable option, FPC needs SSL/TLS support. Even if you use openSSL- it still needs to be included.

It can.

But then we must distribute synapse with FPC. or lnet, or Indy if we want to use them in core technologies. We do not wish to choose.

So, for things which are considered core technologies, it is not possible to rely on third-party packages. The components I created are intended for the specific purposes of FPC (simple HTTP download and/or embedded web serving).

With these components, we do not wish to replace synapse, lnet or Indy. They are full-blown TCP/IP suites, supporting many protocols and options. If your needs are as simple as the ones we have, then you can use the components we distribute by default. If you have more complex needs, you'll have to use one of the external packages.

Michael.

Exactly.

Michael.

I couldn't have said it better myself.

Michael.

OK, right.

Marcos Douglas

Very good explanation.

Marcos Douglas

Andrew Brunner wrote

One thing I'd like to raise is SSL/TSL. For apps development via FPC as an industrial viable option, FPC needs SSL/TLS support. Even if you use openSSL- it still needs to be included.

I would also very much like to see that as well. It's become apparent to me that web app UI's are perfectly suited to task-based, worker-bee applications of most stripes. Having an embedded server with SSL/TSL means easy installation without the need to dick around with apache/iis and still have one point of upgrade/maintenance, etc.

Neat stuff.

Marcos Douglas wrote

But it's also reinventing the wheel, no?

You say it like you think that is a bad thing :-)

Marco van de Voort wrote

On Tue, May 17, 2011 at 10:45:45AM -0300, Marcos Douglas wrote: > But it's also reinventing the wheel, no? You say it like you think that is a bad thing :-)

Not a bad thing, but in open sources projects, that you can share already done codes, I think that could be a waste of time... not all time, not in this particulary project, but some times.

Do you not agree?

Marcos Douglas

I think that while this cliche has some truth in it. At best it should be an hint to look into sharing code, but it is made into some hard law, no matter what.