I am looking for a pascal implementation of the MQTT protocol. I could not find anything inside Lazarus' OnLinePackageManager.

Is there something like that available?

Hi,

Bo Berglund wrote

I am looking for a pascal implementation of the MQTT protocol. I could not find anything inside Lazarus' OnLinePackageManager. Is there something like that available?

I made a front-end Pascal library, and and Object Pascal wrapper for libmosquitto a couple of years ago. It's available here. It's know to work on Linux, Windows and 's against libmosquitto 1.x. Not sure if 2.x got released meanwhile. I can update things if needed. (Pull requests are welcomed.)

It's available here:
https://github.com/chainq/mosquitto-p

Before I started working on this, I've also seen some Delphi library/component that was a fully Pascal implementation of the MQTT protocol but it was full of newschool Delphi-isms, which I disliked, so I did my own solution instead. But what I want to say with this, there are definitely other options out there too with different pros/cons.

Charlie

Well, the version of the mosquitto broker server that gets installed on my RaspberryPi4 is:

$ mosquitto --version
Error: Unknown option '--version'.
mosquitto version 2.0.11
mosquitto is an MQTT v5.0/v3.1.1/v3.1 broker.

So this is what I have to communicate with as a client.

I need to be able to publish as well as subscribe to topics.

In the Lazarus forum I have seen people having problems with this when they use various FPC packages while the native command line works OK (on Linux), probably because they are using old packages not updated for the changes made in MQTT in recent years.

So this is why I am asking if there is some MQTT client package available for FPC/Lazarus that complies to the updated protocols?

I found this on the Lazarus forum:
https://forum.lazarus.freepascal.org/index.php/topic,54397.msg435050.html#msg435050

Not a poackage, but rather advice on how to utilize binary "drivers" installed on the system via translated header files. Not optimal, but might be doable anyway.

I will try that path going forward, just now dealing with the MQTT source devices.

I am looking for a pascal implementation of the MQTT protocol.

https://github.com/cutec-chris/TMQTTClient
https://github.com/jamiei/Delphi-TMQTT2
https://github.com/songshuang8/MqttServer
https://github.com/AndersonGaitolini/MQTT-Gaitolini https://github.com/ZiCog/mqtt-free-pascal
https://github.com/quini27/MQTT-client-application-for-Windows https://github.com/crossrw/mqttClient
https://github.com/apelsinovv/MQTT
https://github.com/bkeevil/mqtt
https://github.com/sigmdel/mosquitto-p
https://github.com/sigmdel/lazmqttc
https://github.com/GRIDSystemSAS/GS.GRID
https://github.com/join2017/dmqtt
https://github.com/wizinfantry/delphi-mqtt-client
https://github.com/pjde/delphi-mqtt
https://github.com/windflee/delphimqtt
https://github.com/Indemsys/Delphi_MQTT_mosquitto

Hi,

FWIW,
I'm using MQTT via cmd-line. I.e. I call

  ExecuteProcess(MQTTExec,MQTTStrArr);

while MQTTExec is /usr/bin/mosquitto_pub

That's on a linux-server.

Op 20/10/2022 om 15:57 schreef Bo Berglund via fpc-pascal:

I can use mosquitto for the pub part because that is a one-timer for each message to send.
The problem is the other side, the client subscribing to the data...

Here the mosquitto_cli is running constantly and printing data as they come in to the terminal window.
How can I pick that up from an application that will handle the data and do something other than wayching the screen?

If I could get at each output that mosquitto_cli writes into a script or such then I would be good to go.

Le 22/10/2022 à 23:37, Bo Berglund via fpc-pascal a écrit :

If I could get at each output that mosquitto_cli writes into a script or such then I would be good to go.

Maybe with a redirection ( > ) to a file in bash ?

Jean SUZINEAU wrote

Le 22/10/2022 à 23:37, Bo Berglund via fpc-pascal a écrit : > If I could get at each output that mosquitto_cli writes into a script or such > then I would be good to go. Maybe with a redirection ( > ) to a file in bash ?

Do you mean something like this then:

Scriptfile checkmqtt:

#!/bin/bash
COMMFILE="$HOME/mqtt/messages"
eval "mosquitto_sub -h <ip of broker> -F '@@d @@S ; %t ; %p' -t '#' >> $COMMFILE"

And then an FPC program that essentially starts the script and then loops checking for the existence of file $HOME/mqtt/messages and if found opens it and processes each line of data and then erases the file?

Or even cleaner the FPC program could launch the command itself as a separate process (in a thread maybe) and then loop as above checking for the output file and so on?

Seems a bit roundabout but would use the existing working mosquitto program on the Linux box to do the MQTT work...

I know that there will be about 5-10 seconds between each MQTT message arriving so there is ample time to process each arrival.

TESTING.....
No it does not work!
The mosquitto output cannot be redirected like this because it seems to be printed to an output screen which is not redirectable... All that happens is that an empty file is created at start of the command and then nothing more...

So how can one make mosquitto write output to a file instead?

Well, I was wrong, it *does* work if done properly...

First of all, it seems like the call to mosquitto_sub with redirection somehow opens that file for writing *once* and then you cannot delete that file from the outside because mosquitto will keep writing to it but now it is invisble to all others...
This is on an ext4 file system where deleted files still exist such that an application can keep working with it even when the directory system has lost it. I have tested this on Ubuntu 20.04 server:

mosquitto_sub -h 192.168.117.131 -F '@@d @@S ; %t ; %p' -t '#' >> mqttlog.txt

And this happens:

- every new message coming in is logged to that file as a new line
- if I delete that file (from another terminal window) then the file is *not* recreated when a new message arrives and there is no longer any accessible logging. The file still exists, but is not visible anywhere else any longer.
- but if I echo a blank to the file (echo "" > mqttlog.txt) then the file *contents* gets erased but it accepts new messages.

So the procedure needs to be:
- start mosquitto_cli as shown with >> redirection
- then look for the log file to change size
- open it for read/write
- copy out all the lines
- erase the content and then save the file and close the handle
- it is now an empty file waiting for new data

Then one can process the saved content of the file and do whatever is needed depening on that.
And finally return to waiting for the file size change again....

All of this can be done with an FPC command line program, I believe. But...

QUESTION:
---------
It seems like I should not have to start a shell with the mosquitto_cli program when there is TProcess, which could run it directly and read the data without using an intermediate log file...

I have found this:
https://wiki.freepascal.org/Executing_External_Programs#Reading_large_output

How should I accomplish the above using TProcess?
- The called application mosquitto_cli must run until my program itself exits
- The data produced by the process should be checked while the process runs
- Using a TMemoryStream to retrieve the output is apparently the way

But how can I handle this situation:
- The incoming data is arriving one msg at a time with seconds in between
- The messages may contain 10-1500 characters each
- How to check that the stream has received all data for one transmission?

Can I do this (using the example from the wiki link above):

// All generated output from AProcess is read in a loop until no more data...
repeat
  repeat
    BytesRead :=  AProcess.Read(Buffer, BUF_SIZE);
    OutputStream.Write(Buffer, BytesRead)
  until BytesRead = 0;  // Stop if no more data is available
  ProcessIncomingMessages(OutputStream);
until 1 //Run forever... How do I get out eventually ??

What happens if the data in the AProcess output stream is less than my buffer size, will it then stall until as many messages have arrived as will fill the buffer?

This cannot be allowed to happen because there may be hours before the next burst of messages.

May be with a pipe |  ?

Something like

mosquitto_sub -h 192.168.117.131 -F '@@d @@S ; %t ; %p' -t '#' | SomePascalProgram

This way you'll receive the output of mosquitto_sub on the standard input of SomePascalProgram.

I've tested locally on an Apache log with
   tail -f /var/log.apache2/access.log | grep http

Le 23/10/2022 à 15:31, Bo Berglund via fpc-pascal a écrit :

What happens if the data in the AProcess output stream is less than my buffer size, will it then stall until as many messages have arrived as will fill the buffer?

I think you can useAProcess.Output.NumBytesAvailable to avoid stalling and read just what is available :

https://www.freepascal.org/docs-html/fcl/pipes/tinputpipestream.numbytesavailable.html

Le 23/10/2022 à 22:50, Jean SUZINEAU via fpc-pascal a écrit :

This way you'll receive the output of mosquitto_sub on the standard input of SomePascalProgram.

A very small example for SomePascalProgram for  use with pipe, reading on standard input and writing on standard output :

var
    s: String;
begin
      repeat
            readln( s);
            writeln( s);
      until false;
end.

Might get stuck at readln(s) if no messages are sent, i.e. no carriage-return coming along?
AFAIK readln is blocking, right?

I will try things out tomorrow, it's midnight now...

Le 23/10/2022 à 23:59, Bo Berglund via fpc-pascal a écrit :

Might get stuck at readln(s) if no messages are sent, i.e. no carriage-return coming along? AFAIK readln is blocking, right?

I've made a test with mosquitto_pub, it seems there is a line ending at the end of each message, so you will be blocked  only when there are no new message available.

If your program needs to do something else during the wait for a new message, you'll need to dedicate a thread or a separate program.

Maybe dedicate a thread, putting your Readln (in the case of mosquitto_sub ... |SomePascalProgram) or  the loop (in the case of use of TProcess) in your TThread descendant Execute method.

There you when you receive a new mqtt message you can call TThread.Synchronize (on a method of your forms for example) to process the message in your main thread.

If your main program is a console program, it's likely that you'll need to call CheckSynchronize( from unit Class) in your main program/main thread from time to time in order to allow the execution of calls of methods specified in Synchronize in your thread (which reads mqtt messages).

I'm not 100% sure for recent freepascal, but I needed to do this in a pascal dll (with threads) called by a C++ program, the all running in console.