Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / fpc-pascal / 2017-06

Serial to TCP gateway in FPC?

42 messages · 11 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

Bo Berglund

I need to implement a serial to TCP gateway in order to communicate between a Windows based control software and a system controller box. The software and the controller only talk serial to each other and the protocol is proprietary and binary.
Now I have a situation where the controller is on a remote location many thousands of km away from here.
I have managed to get some help on the remote site to hook up the controller serial port to a serial to USB cable plugged into a Raspberry Pi, which I can reach via VPN from here. So I installed serial2net on the RPi, making it publish the serial port as a TCP port (#2091). This part works OK, so whatever is sent on the TCP socket connection gets transferred to the RS232 serial line and vice versa.

So far so good, but now I also need some kind of serial port to TCP converter on the Windows 7 PC I use in my end, which is handling the VPN connection.

So I wonder if there are any FPC examples around for conversion serial to TCP (where the TCP part is a client rather than a server), which can compile on Windows. I cannot use any RPi in this end because I cannot get the RPi to connect by VPN to the remote location...

Ready made example or suggestions on how to go about writing a converter in FPC are welcome!

Bo Berglund Developer in Sweden
Snorkl e
to Bo Berglund

Couldn't you read the serial port data and then stream it with Synapse to the remote location?
You can use Synapse to read the serial port data and sent via TCP/IP.

"Bo Berglund" wrote 33 lines
I need to implement a serial to TCP gateway in order to communicate between a Windows based control software and a system controller box. The software and the controller only talk serial to each other and the protocol is proprietary and binary. Now I have a situation where the controller is on a remote location many thousands of km away from here. I have managed to get some help on the remote site to hook up the controller serial port to a serial to USB cable plugged into a Raspberry Pi, which I can reach via VPN from here. So I installed serial2net on the RPi, making it publish the serial port as a TCP port (#2091). This part works OK, so whatever is sent on the TCP socket connection gets transferred to the RS232 serial line and vice versa. So far so good, but now I also need some kind of serial port to TCP converter on the Windows 7 PC I use in my end, which is handling the VPN connection. So I wonder if there are any FPC examples around for conversion serial to TCP (where the TCP part is a client rather than a server), which can compile on Windows. I cannot use any RPi in this end because I cannot get the RPi to connect by VPN to the remote location... Ready made example or suggestions on how to go about writing a converter in FPC are welcome! -- Bo Berglund Developer in Sweden _______________________________________________ fpc-pascal maillist - http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Bo Berglund
to Snorkl e
Snorkl e wrote
Couldn't you read the serial port data and then stream it with Synapse to the remote location? You can use Synapse to read the serial port data and sent via TCP/IP.

Probably, but I was hoping for some ready-made example to get going a bit faster...
I did another piece of software a year or so ago only involving TCP/IP, also for remote control, but then I used an Indy TCP client component.
My problem is getting my head around serial port handlers, which I have so-so experience of in the past. No RS232 work with FPC/Lazarus earlier and in Delphi I used AsyncPro components, but these are so complex that one is lost amongst all properties to set...

I would like a dead simple serial component where I could just specify the comm parameters (baud, start/stop bits, handshaking and port) then just open the connection and wait for receive *events*.

Yes, it is easier for me to have components with event generation on data reception rather than using blocking components, which I must make threads for and then figure out how to channel the data from the thread into the main application...

Bo Berglund Developer in Sweden
Koenraad Lelong
to Bo Berglund

Op 18-06-17 om 18:16 schreef Bo Berglund:

Hi,

Do you know the SDPO-package ? That has a serial component (SDPOSerial). This has a OnRxData event.

Just a thought. I used it in the past without problems.

HTH

Koenraad.

Snorkl e
to Bo Berglund: On Sun, 18 Jun 2017 09:46:06 -0500, Snorkl e Probably, but I was hoping for some…

Synapse and fpc works great with serial port, I have a daemon I wrote 4 years ago that reads serial data from a phone PBX runs 24X7 and never an issue.

"Bo Berglund" wrote 35 lines
On Sun, 18 Jun 2017 09:46:06 -0500, Snorkl e wrote: >Couldn't you read the serial port data and then stream it with Synapse to >the remote location? >You can use Synapse to read the serial port data and sent via TCP/IP. > Probably, but I was hoping for some ready-made example to get going a bit faster... I did another piece of software a year or so ago only involving TCP/IP, also for remote control, but then I used an Indy TCP client component. My problem is getting my head around serial port handlers, which I have so-so experience of in the past. No RS232 work with FPC/Lazarus earlier and in Delphi I used AsyncPro components, but these are so complex that one is lost amongst all properties to set... I would like a dead simple serial component where I could just specify the comm parameters (baud, start/stop bits, handshaking and port) then just open the connection and wait for receive *events*. Yes, it is easier for me to have components with event generation on data reception rather than using blocking components, which I must make threads for and then figure out how to channel the data from the thread into the main application... -- Bo Berglund Developer in Sweden _______________________________________________ fpc-pascal maillist - http://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Paul Breneman
to Bo Berglund: I need to implement a serial to TCP gateway in order to communicate between a Windows…
Bo Berglund wrote 24 lines
I need to implement a serial to TCP gateway in order to communicate between a Windows based control software and a system controller box. The software and the controller only talk serial to each other and the protocol is proprietary and binary. Now I have a situation where the controller is on a remote location many thousands of km away from here. I have managed to get some help on the remote site to hook up the controller serial port to a serial to USB cable plugged into a Raspberry Pi, which I can reach via VPN from here. So I installed serial2net on the RPi, making it publish the serial port as a TCP port (#2091). This part works OK, so whatever is sent on the TCP socket connection gets transferred to the RS232 serial line and vice versa. So far so good, but now I also need some kind of serial port to TCP converter on the Windows 7 PC I use in my end, which is handling the VPN connection. So I wonder if there are any FPC examples around for conversion serial to TCP (where the TCP part is a client rather than a server), which can compile on Windows. I cannot use any RPi in this end because I cannot get the RPi to connect by VPN to the remote location... Ready made example or suggestions on how to go about writing a converter in FPC are welcome!

Why do you need FPC code? Here are free drivers that work in Windows: http://com0com.sourceforge.net/

Bo Berglund
to Paul Breneman: Why do you need FPC code? Here are free drivers that work in Windows:…
Paul Breneman wrote
Why do you need FPC code? Here are free drivers that work in Windows: http://com0com.sourceforge.net/

Because I am programming in Pascal (Delphi and FPC) and I had failed to find any working ready-made solutions.

It looks like I have already tested com0com a year ago or so:

When I look in my PC Win7 registry I find that two ports that are enumerated but non-working seems to have been set up by com0com and by what I see in the registry com0com seems to be a service. However the indicated home dir for com0com does not exist and still there are all these entries in the registry. There is no registered service named com0com either...
Looks like leftovers from an incomplete uninstall.

Anyway, I would like to test this again but the question is if it really works on Windows 7 and up (X64)?
The interesting part would be com0tcp in my case.

Bo Berglund Developer in Sweden
Bo Berglund
to Paul Breneman: Why do you need FPC code? Here are free drivers that work in Windows:…
Paul Breneman wrote
Why do you need FPC code? Here are free drivers that work in Windows: http://com0com.sourceforge.net/

I tried again and failed again.
I installed com0com and had an error in my face during installation: SetupUninstallOEMInf(oem11.inf)
ERROR: 0xE000023D -

But it was possible to continue anyway, so I did that.

Afterwards I used the setup utility to make the com pair COM91-COM92.

Then I started com2tcp to connect to the remote port and make it possible for my local application to use the virtual com port COM91 to connect to the remote system. But:

com2tcp.exe --baud 38400 --data 8 --parity n --stop 1 --ignore-dsr COM92 10.0.0.26 2091
OpenC0C(): CreateFile("COM92") ERROR No such file or directory (2)

And my application, which enumerates available serial ports on startup, does not even show the COM91, COM92 virtual port pair...

So the local application cannot connect to one side of the virtual com pair and com2tcp cannot connect to the other side...

I would really like to use an available program rather than spening time to develop something myself, but it seems like com0com does not cut it...

Bo Berglund Developer in Sweden
Paul Breneman
to Bo Berglund
Bo Berglund wrote 6 lines
On Sun, 18 Jun 2017 19:16:05 -0400, Paul Breneman wrote: > Why do you need FPC code? Here are free drivers that work in Windows: > http://com0com.sourceforge.net/

...

I would really like to use an available program rather than spening time to develop something myself, but it seems like com0com does not cut it...

I've used com0com *and* hub4com successfully for several things in the past. See #3 here:
http://www.ctrlterm.com/related.htm

I'll try to take a look at com2tcp in the next few days and compare that to the Digi One SP (where Win2K drivers are *still* available) which I've used recently:
https://www.digi.com/products/serial-servers/serial-device-servers/digionesp

BTW, Digi used Delphi & AsyncPro years ago for some test software...

Regards,
Paul
www.ControlPascal.com

Bo Berglund
to Paul Breneman
Paul Breneman wrote
I've used com0com *and* hub4com successfully for several things in the past. See #3 here: http://www.ctrlterm.com/related.htm

In the README file with com0com they write:

"INSTALLING
==========

NOTE (x64-based Windows Vista/Windows Server 2008/Windows 7/8): The com0com.sys is a test-signed kernel-mode driver that will not load by default. To enable test signing, enter command:

bcdedit.exe -set TESTSIGNING ON

and reboot the computer.

NOTE:
Enabling test signing will impair computer security."

This does not sound good...

During installation I saw several warnings from Windows about unknown signing of a driver, and Windows recommended not to install...

Bo Berglund Developer in Sweden
Mark Morgan Lloyd
to Michael Schnell: AsyncPro is free and easy to use but not perfectly bug-free. Unfortunately no fpc/Lazarus…
Michael Schnell wrote 5 lines
On 18.06.2017 18:16, Bo Berglund wrote:> No RS232 work with FPC/Lazarus> earlier and in Delphi I used AsyncPro components, but these are so> complex that one is lost amongst all properties to set...AsyncPro is free and easy to use but not perfectly bug-free. Unfortunately no fpc/Lazarus port yet.

I tend to use the lower-level serial.pp unit. The patches I contributed a couple of years ago specifically added a couple of read-with-timeout functions.

Mark Morgan Lloyd markMLl .AT. telemetry.co .DOT. uk [Opinions above are the author's, not those of his employers or colleagues]
Michael Schnell
to Mark Morgan Lloyd
Mark Morgan Lloyd wrote
I tend to use the lower-level serial.pp unit. The patches I contributed a couple of years ago specifically added a couple of read-with-timeout functions

IMHO, freeing the user from the necessity to poll or do a thread to wait for serial events is a very beneficial strategy.

But Koenraad Lelong said in the other mail that "SDPOSerial" has a OnRxData event. Hopefully by "event" a main thread event is meant.

-Michael

Paulo Costa
to Michael Schnell
Michael Schnell wrote 10 lines
On 20.06.2017 11:18, Mark Morgan Lloyd wrote: > I tend to use the lower-level serial.pp unit. The patches I > contributed a couple of years ago specifically added a couple of > read-with-timeout functions IMHO, freeing the user from the necessity to poll or do a thread to wait for serial events is a very beneficial strategy. But Koenraad Lelong said in the other mail that "SDPOSerial" has a OnRxData event. Hopefully by "event" a main thread event is meant.

Yes. That is the case.

Paulo Costa

Bo Berglund
to Mark Morgan Lloyd: I tend to use the lower-level serial.pp unit. The patches I contributed a couple of years…
Mark Morgan Lloyd wrote
I tend to use the lower-level serial.pp unit. The patches I contributed a couple of years ago specifically added a couple of read-with-timeout functions.

Mark, thanks for your input!

I had a look at the serial unit and I found that there are two overloaded functions SerReadTimeout():

function SerReadTimeout(Handle: TSerialHandle; var Buffer; mSec:
LongInt): LongInt;
function SerReadTimeout(Handle: TSerialHandle; var Buffer: array of
byte; count, mSec: LongInt): LongInt;

In the second of them there is a count argument, presumably indicating the number of bytes to read from the port. The first is missing this but sets the value to 1 in the call to ReadFile.

Does this mean that it will only read a single byte and return it whereas the second version reads the indicated number of bytes and then returns?
I assume that for both there is a return forced when the timeout expires.
In both cases the number of actually read bytes are returned in the result.

Questions:
----------
Do I have to create a thread in order to constantly monitor the state of the reception?
And I could not find a way to set the comm buffer size, what happens if I do not check incoming data for a while when there is an inbound stream. Is there an overflow somewhere?

Anyway it looks promising so far, now I just have to find the proper TCP client component to use as the far end of the link. I have used Indy components in the past but I would like something a bit simpler to use...

Bo Berglund Developer in Sweden
Bo Berglund
to Mark Morgan Lloyd: I tend to use the lower-level serial.pp unit. The patches I contributed a couple of years…
Mark Morgan Lloyd wrote
I tend to use the lower-level serial.pp unit. The patches I contributed a couple of years ago specifically added a couple of read-with-timeout functions.

I'm back with a question:
I used the serial example in the hardware access wiki: http://wiki.freepascal.org/Hardware_Access#FPC_built_in_Serial_unit

as a template and I have a button on my form to open/close the serial port, but it does not work properly. Below is my button code to open and close the port depending on the state it is in, which is set by the global boolean var FComOpen.
The port number is taken from a spinedit speComPort and the baud rate is from a preloaded combobox cbxBaud:

procedure TForm1.btnConnectComClick(Sender: TObject);
begin
  if FComOpen then
  begin
    //Close com port
    SerSync(FComH); { flush out any remaining before closure }
    SerFlushOutput(FComH); { discard any remaining output }
    SerClose(FComH);
    FComOpen := false;
    TButton(Sender).Caption:= 'Open Serial';
    ledSerial.Brush.Color:= clRed;
  end
  else
  begin
    //Open com port
    FComPortName := 'COM' + IntToStr(speComPort.Value) + ':';
    FComFlags := [ ];
    FBaud := StrToInt(cbxBaud.Text);
    FComH := SerOpen(FComPortName);
    if FComH  0 then
    begin
      FComOpen := true;
      SerSetParams(FComH,FBaud,8,NoneParity,1,FComFlags);
      TButton(Sender).Caption:= 'Close Serial';
      ledSerial.Brush.Color:= clLime;
    end;
  end;
end;

The problem I have is that it seems only to work for COM1 or COM3, as soon as I use a higher numbered port like COM33 then it fails. What have I missed here?
Is serial only able to work with the low numbered ports (single digit)?

Bo Berglund Developer in Sweden
José Mejuto
to Bo Berglund
Bo Berglund wrote 5 lines
The problem I have is that it seems only to work for COM1 or COM3, as soon as I use a higher numbered port like COM33 then it fails. What have I missed here? Is serial only able to work with the low numbered ports (single digit)?

Hello,

COM ports in windows can only be 1-9, to open high numbered COM ports you must use the extended name syntax (without quotes) "\\.\COM99" you can also use "\\.\COM1" for COM1-9.

Bo Berglund
to José Mejuto
José Mejuto wrote 9 lines
> The problem I have is that it seems only to work for COM1 or COM3, as > soon as I use a higher numbered port like COM33 then it fails. > What have I missed here? > Is serial only able to work with the low numbered ports (single > digit)? COM ports in windows can only be 1-9, to open high numbered COM ports you must use the extended name syntax (without quotes) "\\.\COM99" you can also use "\\.\COM1" for COM1-9.

It turns out that I also must not use the trailing colon in the name which is present in the wiki example...
But then I get a non-zero handle. I have yet to verify that data can flow.

Is there a way to enumerate the available com ports so that they can be listed in a combobox for selection? This would have the advantage that it would work on both Windows and Linux and it limits the possible errors by the user. Of course the enumerator must know about the differences in syntax between Windows and Linux!

What about Linux (Raspberry Pi)? Is the com port name simply ttyUSB1 or do I have to use the full /dev/ttyUSB1?

Bo Berglund Developer in Sweden
Mark Morgan Lloyd
to Bo Berglund
Bo Berglund wrote 5 lines
On Wed, 21 Jun 2017 01:57:36 +0200, José Mejuto wrote: >> The problem I have is that it seems only to work for COM1 or COM3, as>> soon as I use a higher numbered port like COM33 then it fails.>> What have I missed here?>> Is serial only able to work with the low numbered ports (single>> digit)?>>COM ports in windows can only be 1-9, to open high numbered COM ports >you must use the extended name syntax (without quotes) "\\.\COM99" you >can also use "\\.\COM1" for COM1-9. It turns out that I also must not use the trailing colon in the namewhich is present in the wiki example...But then I get a non-zero handle. I have yet to verify that data canflow. Is there a way to enumerate the available com ports so that they canbe listed in a combobox for selection? This would have the advantagethat it would work on both Windows and Linux and it limits thepossible errors by the user. Of course the enumerator must know aboutthe differences in syntax between Windows and Linux!

Not reliably in unix. You have to know the various potential name patterns, i.e. /dev/ttyS*, /dev/ttyAMA* and so on, and then enumerate what's currently visible in the filesystem. It might be possible to get "hints" as to the device type from the /sys filesystem, but I'm not sure it's safe to rely on this.

What about Linux (Raspberry Pi)? Is the com port name simply ttyUSB1or do I have to use the full /dev/ttyUSB1?

You *definitely* have to use the name as given. You're opening a file in the filesystem tree, not an internally-reserved name.

Mark Morgan Lloyd markMLl .AT. telemetry.co .DOT. uk [Opinions above are the author's, not those of his employers or colleagues]
Bo Berglund
to Mark Morgan Lloyd
Mark Morgan Lloyd wrote
> What about Linux (Raspberry Pi)? Is the com port name simply ttyUSB1or do I have to use the full /dev/ttyUSB1? You *definitely* have to use the name as given. You're opening a file in the filesystem tree, not an internally-reserved name.

OK,
thanks!
I am now creating a super-simple proxy app which will open a serial port and connect to a TCP server. Then it will send all incoming data on the serial port to the TCP server and all incoming TCP server data back to the serial port.

Right now I am doing a Lazarus GUI app on Windows, but eventually I will move the code into a command line app.

I have used the serial unit in FPC to handle the serial port connection and am now looking for the TCP Client component to use.

I have used Indy's TIdTcpClient in the past, but it required me to create a thread to handle the data transfers, and I don't really like complicating matters like that for this (seemingly) simple project. I looked at the socket unit, which has the advantage that it is part of FPC. But the documentation is very confusing... For example I was looking at fpconnect:
https://www.freepascal.org/docs-html/rtl/sockets/fpconnect.html and noted that the example does NOT use fpconnect at all, instead it uses a depreciated Connect method with strange arguments... Is there some updated documentation somewhere?

Anyway:
The functions of this proxy app are only:
- Open the specified serial port and set it to the required baudrate
- Open a connection to the TCP server port
- Now somehow run a loop as follows:
- Serial data available? If so read them and send to the TCP server
- TCP data available? If so read and send through the serial port
- Stop condition set? If so close down TCP and serial connections and exit the program. (Don't know how this will work)
- Repeat loop again

Two issues here:
1) How do I enter a loop in a GUI application? In a command line application it is just going to be part of he main program code.

2) How do I stop the program from running at 100% CPU utilization (on Windows)?

And of course how to use the sockets unit properly?

Bo Berglund Developer in Sweden
Bo Berglund
to Bo Berglund
Bo Berglund wrote 7 lines
I looked at the socket unit, which has the advantage that it is part of FPC. But the documentation is very confusing... For example I was looking at fpconnect: https://www.freepascal.org/docs-html/rtl/sockets/fpconnect.html and noted that the example does NOT use fpconnect at all, instead it uses a depreciated Connect method with strange arguments... Is there some updated documentation somewhere?

OOPS! After I sent my earlier message I realized that the fpconnect example uses code that attaches the socket to textfiles Sin and Sout. But my use is specifically NOT a text based system, all data are binary and no iterpretation whatsoever of the content is allowed.

Can I still use the sockets unit with fpconnect? Or must I use the depreciated Connect method so I can specify the files as binary or text files? Connect is overloaded....

Bo Berglund Developer in Sweden
Mark Morgan Lloyd
to Bo Berglund
Bo Berglund wrote
On Thu, 22 Jun 2017 00:30:27 +0200, Bo Berglund wrote: > I looked at the socket unit, which has the advantage that it is part>of FPC. But the documentation is very confusing...>For example I was looking at fpconnect:>https://www.freepascal.org/docs-html/rtl/sockets/fpconnect.html>and noted that the example does NOT use fpconnect at all, instead it>uses a depreciated Connect method with strange arguments...>Is there some updated documentation somewhere? OOPS! After I sent my earlier message I realized that the fpconnectexample uses code that attaches the socket to textfiles Sin and Sout.But my use is specifically NOT a text based system, all data arebinary and no iterpretation whatsoever of the content is allowed. Can I still use the sockets unit with fpconnect? Or must I use thedepreciated Connect method so I can specify the files as binary ortext files? Connect is overloaded....

I've not looked at the example but sockets should be transparent.

Mark Morgan Lloyd markMLl .AT. telemetry.co .DOT. uk [Opinions above are the author's, not those of his employers or colleagues]
Mark Morgan Lloyd
to Bo Berglund: On Wed, 21 Jun 2017 08:02:00 +0000, Mark Morgan Lloyd thanks! I am now creating a…
Bo Berglund wrote
Two issues here:1) How do I enter a loop in a GUI application? In a command lineapplication it is just going to be part of he main program code.

I'm not saying it's the right or the best way, but in the interest of giving you something to work with I do this:

procedure TListForm.FormCreate(Sender: TObject);
CONST   startParam= 0;
begin
..
     Application.QueueAsyncCall(@OnAfterShow, startParam) (* Keep at end *)
end { TListForm.FormCreate } ;
PROCEDURE TListForm.OnAfterShow(afterShowParam: PtrInt);
(* guaranteed to be after all form creation etc. has completed.         *)
BEGIN
..
END { TListForm.OnAfterShow } ;

That can loop, but make sure you put Application.ProcessMessages in it.

Mark Morgan Lloyd markMLl .AT. telemetry.co .DOT. uk [Opinions above are the author's, not those of his employers or colleagues]
Bo Berglund
to Bo Berglund: I need to implement a serial to TCP gateway in order to communicate between a Windows…
Bo Berglund wrote
So I installed serial2net on the RPi, making it publish the serial port as a TCP port (#2091). This part works OK, so whatever is sent on the TCP socket connection gets transferred to the RS232 serial line and vice versa.

My statement above seems to be correct, ser2net does work properly on my RPi machine as witnessed by the fact that I can use a TCP-enabled client software (written using Lazarus) against the controller and operate it as intended.
The data passes through ser2net exactly as expected.

But when I hook up the serial-only client (a Delphi XE5 program) via a relaying program I made myself (again in Lazarus) using the FPC serial unit and Indy TcpClient objects it fails on initial connect and does not pass into the main control.
The very strange thing is that the ser2net logs show the incoming data from the network as being passed on to the serial port and there is some response but then it just stops.
The protocol transfer should look as follows:

Client S/W Controller Action
--------------------------------------------------------- 0x10 => Switch to remote control <= 0x06 (ACK) Command accepted)
0xD1 => Switch to data transfer mode <= 0x06 (ACK) Command accepted
<cmd> => Any data transfer command ----------------------------------------------------------

I.e:
10 06 D1 06 <some command>
But it turns out to be:
10 06 D1 D1 D1

Where D1 is inbound from the client and is repeated because there is no ACK response.

So in this case the ser2net seems to receive the command (and therefore transmitting it via the serial port), but it does not recognize any response...

Is there any similar experience here with ser2net?

NEXT:
It looks like I will have to rewrite my relaying program now used in Windows for the conversion to TCP from RS232 to also work as the replacement for ser2net on RPi3. This way I can monitor the data better and check time intervals etc.

My problem here is the fact that the RPi is headless so the GUI program I have written with Lazarus will not work. Instead I have to make a console version that can run on RPi for the test and here I have the concern that I am using Application.Processmessages, which I believe cannot be used in a console program.
How can I replace it?

Bo Berglund Developer in Sweden
Mark Morgan Lloyd
to Bo Berglund
Bo Berglund wrote
My problem here is the fact that the RPi is headless so the GUIprogram I have written with Lazarus will not work. Instead I have tomake a console version that can run on RPi for the test and here Ihave the concern that I am using Application.Processmessages, which Ibelieve cannot be used in a console program.How can I replace it?

You're obviously correct that APM doesn't exist in a text-only program because it's part of the LCL. But I think you need to ask yourself why you need to /replace/ it, rather than just removing it.

Mark Morgan Lloyd markMLl .AT. telemetry.co .DOT. uk [Opinions above are the author's, not those of his employers or colleagues]
Lukasz Sokol
to Bo Berglund: On Sun, 18 Jun 2017 09:04:49 +0200, Bo Berglund My statement above seems to be correct,…

On 26/06/17 23:28, Bo Berglund wrote:
[...]

Show quoted text 15 lines
NEXT: It looks like I will have to rewrite my relaying program now used in Windows for the conversion to TCP from RS232 to also work as the replacement for ser2net on RPi3. This way I can monitor the data better and check time intervals etc. My problem here is the fact that the RPi is headless so the GUI program I have written with Lazarus will not work. Instead I have to make a console version that can run on RPi for the test and here I have the concern that I am using Application.Processmessages, which I believe cannot be used in a console program. How can I replace it?

If you are comfortable with thread programming, you could have a thread handle the hardware / serial handling (by keeping the 'messages' in a 'queue' or 'queues' (one outbound, one inbound). The serial handling thread would poll the serial port and handle serial sending timing etc, and put the received data into the rx 'queue'. Depending on how fast the serial communications are, the thread could then ThreadSwitch to give up the reminder of its time slice if not needed, and sleep for the rest of the time.

Then your main program would periodically check the queues and depending on whether it needs to do something or not, will do it and then threadswitch() and sleep(),

The main thread would then be responsible for network / hi level 'protocol' communication, data conversion
(e.g. you could assume network data 'text' encoded in base64, or even text based json, that is then converted to/from binary form for serial)

IIRC some network handling components/libraries use threads of their own.

If you wanted to use 'just' network sockets, you'd probably need to handle network 'low level' communications in threads anyway.

hope this makes (some) sense
_l.

Michael Schnell
to Bo Berglund: On Sun, 18 Jun 2017 09:04:49 +0200, Bo Berglund My statement above seems to be correct,…
Bo Berglund wrote
I have the concern that I am using Application.Processmessages, which I believe cannot be used in a console program. How can I replace it?

TThread.Queue is available in an appropriately done "console application".

As already discussed several times in the past, I did a draft for an "active" version of a "no-GUI application" that allows for a main thread executing such queued TThread and TTimer Events. (It also has Application.Processmessages, Application.QueueAsyncCall,... ).

You can test it, but I am not forcing this project much, as with modern hardware (such as the Pi3) there is not much pressure any more to avoid installing a GUI framework.

-Michael

Michael Schnell
Lukasz Sokol wrote
But I'd put the serial port handling calls, blocking or not, into a thread separate from main anyway.

Of course. (The "short message says this, too.)

(p.s. why not reply on the group still? :) )

I did send the message to the group, as well, but those messages mostly are delayed by a day or more.

-Michael

Bo Berglund
to turro75: I don't know if useful, here You can find a terminal + serial to eth or reversed based on…
turro75 wrote 6 lines
I don't know if useful, here You can find a terminal + serial to eth or reversed based on lnet, custom serial component and richmemo https://github.com/Turro75/TCPLogger_V2.git <https://github.com/Turro75/TCPLogger_V2.git>

Thanks,
but I have two problems:
1) I have failed to install LNet into my Lazarus 1.6 on Windows. See thread titled "Installing LNet in Lazarus 1.6 fails" in the Lazarus list.
So I turned to Indy instead, but I think LNet would serve me better.

2) I need the relay gateway to be running also on Raspberry Pi3, i.e. the system must be cross-platform and it seems like the example is for Windows (COM ports with strange syntax etc).

3) I also noted in the comments in TurSeriale.pas: "The Windows port driver does receive callbacks in chunks that are typically 8 bytes long.
With ReceiveMode = rmRAW, TSeriale will simply pass this chunks on to the application without any processing."
Does this mean that if a single character is received it is not passed along? My protocol is exchanging single bytes for state changes and these have rather tight timeouts as well, so a long delay in forwarding the bytes will break the protocol.

State of my effort as of now:
=============================
Server side:
I am using ser2net on a RPi3 on the remote location to translate the serial port connection into a socket connection. This seems to work but there are issues.

Client side:
Here I have written a relaying application myself with Lazarus 1.6 and FPC 3.0.0. I used the FPC built-in serial unit as adviced in the hardware wiki for the serial port. For the TCP client I finally went with Indy's TIdTcpClient and it seemed to work a bit.

Issues:
For some unknown reason (so far) the remote side loses connection to the target system (no response after initially being able to talk to it). In fact this is reproducible 100%, the first handshake works and the system can be set to the new state, but on the next state change the 1-byte command is not responded to so the connection fails.

In order to trace this I need to have better control at the remote side so I wanted to expand the Windows side client to be able to run in server mode so I can install it on the RPi3 instead of using ser2net. Then I think I could examine all data traffic with timing to figure out the problem.

So my current aim is to convert the current program into an application that can run either as a server or a client on the network and do this:

SERVER
--------
- Listen for connection on a specific TCP port (only one allowed)
- When client connects open the predefined serial port
- Enter relay mode where all bytes coming in on the socket connection are sent to the serial port and all bytes coming in on the serial port are sent to teh socket.
- Keep doing this until the TCP client disconnects, when the serial port is also closed and the system waits for another connect.
- To not lock up forever, there should be a (long) timeout in the relay mode such that in the absence of incoming data the timeout is decremented until triggered. Then the ports are closed and the system goes to wait mode.

CLIENT MODE
-----------
- Wait for some connect command (not sure how to do this)
- Connect to local serial port
- Connect to remote TCP port
- Enter relay mode as above if both connections succeed
- Exit relay mode on some disconnect command
- Disconnect both TCP and serial ports

I would like to also add some data logging to this system so I can see what is happening on the lines.

Using the FPC built-in serial unit I can handle the serial connection, but I have trouble with the TCP part since the Indy components are a bit too involved and hard to understand.

So in order for the program to run as a server I need a TCP server component I can integrate into this without too much fussing about. There is only one single client allowed and data should not be processed at all, just transferred...

(And I failed to install LNet in Lazarus...)

Bo Berglund Developer in Sweden
José Mejuto
to Bo Berglund
Bo Berglund wrote 7 lines
3) I also noted in the comments in TurSeriale.pas: "The Windows port driver does receive callbacks in chunks that are typically 8 bytes long. With ReceiveMode = rmRAW, TSeriale will simply pass this chunks on to the application without any processing." Does this mean that if a single character is received it is not passed along? My protocol is exchanging single bytes for state changes and

Hello,

I'm quite sure no, it means that the driver will try to queue 8 bytes before sending back data, but the tricky word is "try", probably after a context switch or similar the available data will be reported by the driver whichever the amount of bytes are.

Kernel to userland data pass is very expensive so drivers usually try to queue a data amount limit or a data time limit whichever it happens first.

Also forget the usual 15 bytes buffering in serial comms, it is expected that the kernel driver can buffer much more data before overflow, probably around 8 Kbs.

Koenraad Lelong
to Bo Berglund: On Sun, 18 Jun 2017 20:36:11 +0200, Koenraad Lelong Do you mean SDPO or 5DPO (letter S or…

Op 29-06-17 om 10:25 schreef Bo Berglund:

Show quoted text 10 lines
On Sun, 18 Jun 2017 20:36:11 +0200, Koenraad Lelong wrote: > Do you know the SDPO-package ? That has a serial component (SDPOSerial). > This has a OnRxData event. Do you mean SDPO or 5DPO (letter S or number 5 at the start)? I googled for SDPO but the hits were for 5DPO...

It's the same project. Sometimes it's called 5dpo sometimes sdpo. In the component-list there is a tab 5dpo, but the component itself is called TSdpoSerial. If you want to install it, you have to install SdpoSerialLaz.

HTH,
Koenraad

Paulo Costa
to Koenraad Lelong
Koenraad Lelong wrote 15 lines
Op 29-06-17 om 10:25 schreef Bo Berglund: > On Sun, 18 Jun 2017 20:36:11 +0200, Koenraad Lelong > wrote: > >> Do you know the SDPO-package ? That has a serial component (SDPOSerial). >> This has a OnRxData event. > > Do you mean SDPO or 5DPO (letter S or number 5 at the start)? > I googled for SDPO but the hits were for 5DPO... > > It's the same project. Sometimes it's called 5dpo sometimes sdpo. In the component-list there is a tab 5dpo, but the component itself is called TSdpoSerial. If you want to install it, you have to install SdpoSerialLaz.

That is correct.

The longer story is that most of those components came from our work on a team called 5dpo. Our team was active from 1998 to 2007 in the Robocup competitions (Small Size League ans Middle Size League) The trouble is that you can not have an identifier starting with a number, so for component names we replaced the 5 with a S.

Paulo Costa

Bo Berglund
to Bo Berglund: On Wed, 28 Jun 2017 10:55:47 -0700 (MST), turro75 but I have two problems: 1) I have…
Bo Berglund wrote 6 lines
Issues: For some unknown reason (so far) the remote side loses connection to the target system (no response after initially being able to talk to it). In fact this is reproducible 100%, the first handshake works and the system can be set to the new state, but on the next state change the 1-byte command is not responded to so the connection fails.

Just to close this thread...
----------------------------
I finally found the cause of the problem, which was a handshaking timeout inside the PC control software.

The in-process state changes (1-byte commands) were supposed to be acknowledged within 200 ms, and this is long enough for a physical RS232 connection but too short when dealing with a TCP/IP translation and transfer from Europe to USA and back.

When I added logging of the transmt and receive times of the data in the relayer program I saw that the turn-around delays were about 300 ms normally, which is just a bit longer than what the software allowed for.
So we have to get someone revise the PC control software and increase the timeout.

The Lazarus relaying program using FPC serial and Indy TIdTcpclient components works just fine! Now also with millisecond logging.

Bo Berglund Developer in Sweden
Mark Morgan Lloyd
to Bo Berglund
Bo Berglund wrote
The Lazarus relaying program using FPC serial and Indy TIdTcpclientcomponents works just fine! Now also with millisecond logging.

Assuming you mean the standard serial.pp: yes, it was because of accurate logging etc. that I worked on it a few years ago. However be warned that as part of that effort I was looking at it on various hardware, and I found that USB serial introduces a lot of inaccuracy presumably because it's trying to bundle stuff up into packets.

You were asking a few days ago about Application.ProcessMessages but didn't respond to my attempt to tease out your understanding of what was happening. Have you sorted that out to your satisfaction?

Mark Morgan Lloyd markMLl .AT. telemetry.co .DOT. uk [Opinions above are the author's, not those of his employers or colleagues]
Bo Berglund
to Mark Morgan Lloyd
Mark Morgan Lloyd wrote
You were asking a few days ago about Application.ProcessMessages but didn't respond to my attempt to tease out your understanding of what was happening. Have you sorted that out to your satisfaction?

I made a Lazarus GUI program to convert a serial stream to TCP to a remote server where it was again converted to serial. The question was raised because I have used APM inside the main loop of the application in order for it to process stuff like mouse clicks etc while running the main task.
Then I figured that I should make a console program with the same functions, but APM is part of the Forms unit and this is not used by a console. So I was thinking that maybe the data handling wouyld not work if it was missing...

Now I decided to keep the GUI app and use it interactively when I need to run against the remote system.

Bo Berglund Developer in Sweden
Mark Morgan Lloyd
to Bo Berglund
Bo Berglund wrote
On Sun, 2 Jul 2017 12:27:45 +0000, Mark Morgan Lloyd wrote: > You were asking a few days ago about Application.ProcessMessages but >didn't respond to my attempt to tease out your understanding of what was >happening. Have you sorted that out to your satisfaction? I made a Lazarus GUI program to convert a serial stream to TCP to aremote server where it was again converted to serial.The question was raised because I have used APM inside the main loopof the application in order for it to process stuff like mouse clicksetc while running the main task.Then I figured that I should make a console program with the samefunctions, but APM is part of the Forms unit and this is not used by aconsole. So I was thinking that maybe the data handling wouyld notwork if it was missing... Now I decided to keep the GUI app and use it interactively when I needto run against the remote system.

What's happening there is that an LCL-based program is single threaded, and if you're in any sort of tight loop you need to call APM to get any of the usual GUI stuff to work.

When I coded those two extra reads the Linux/Solaris implementation had a comparatively tight loop, so I added the callback so that if they were being used in the main thread the GUI would carry on working. The Windows implementation isn't so careful with that.

If they were instead being called from a background thread you'd obviously /not/ call APM, since in general that can only be safely called from the main (foreground) thread. However they can still be called usefully to put a Sleep(0) in the loop, i.e. a hint to the kernel that it can reschedule if it wants to.

Mark Morgan Lloyd markMLl .AT. telemetry.co .DOT. uk [Opinions above are the author's, not those of his employers or colleagues]
Michael Schnell
to Mark Morgan Lloyd
Mark Morgan Lloyd wrote
... if you're in any sort of tight loop you need to call APM to get any of the usual GUI stuff to work.

Including any messages sent by worker threads such as "TThread.Queue", "TThread.Synchronize", "Application.QueueAsnycCall" and "SendThreadMessage".

With the additional afterthought that TThread.Synchronize will stall the worker thread until the event is able to be handled in the GUI Thread.

-Michael