I have created a simple application which relays data from an application I am debugging in Windows7 to/from a remotely located embedded data system.
I use the built-in fpc Serial unit to handle the serial data and an Indy10 TIdTcpClient for the network part.
Data coming in on the serial port are mirrored out to the remote TCP server and data returned from that server are mirrored to the serial port.
The Windows application only uses RS232 for this type of communication.
At the remote location I have a Raspberry Pi uint where I have installed and configured ser2net to do the same job in the remote location.

My relaying application displays the number of bytes received and transmitted on the interfaces so I can see what is going on.

This scheme has worked fine for most commands and data transfers I have checked, but now I have run against a brick wall...

There is a pair of commands designed to read and write a large section of the system CMOS RAM memory (where the data file system resides). When I try to write a 1 Mbytes big buffer the byte count in my relayer does not reach the correct number. The Windows application I am debugging sends all of the bytes out the serial port (I have logged this), but the relayer seems to lose some data and therefore the transfer fails. The binary protocol specifies at the start how many bytes are to be transferred (0x0FF000 or 1044486 decimal), then it sends the data followed by a two-byte checksum. The data system shall respond with NAK or ACK depending on the outcome of the checksum verification.
The problem is that when the Windows app is done sending the data system is still missing many kilobytes....
So no ACK is returned, it is still in receive mode.

Now I am looking at the Serial unit in order to figure out how buffer sizes influence the performance. Apparently both Tx and Rx buffers are set (hardcoded) to 2048, which feels like a bit low to me.

Is there some hidden property that makes it possible to increase this value?
What is the maximum size one can set it too?

Not that I was responsible for. Refer to the Windows API for any limits.

Mark Morgan Lloyd wrote

> Is there some hidden property that makes it possible to increase thisvalue? >What is the maximum size one can set it too? Not that I was responsible for. Refer to the Windows API for any limits.

I might add that the problem manifests itself on transmission from the Windows application (Delphi XE5).
I.e. the relaying Lazarus/FPC application loses incoming bytes after some time of processing the streams. Small data sizes work just fine but with the 1 Mbytes packet some kbytes are lost.

And here is the core procedure where the relaying happens:

procedure TfrmMain.RelayData;
{This procedure is responsible for transferring the data from one side
to the other.
It will run until one of the ports are closed as indicated by the
flags FTcpConnected
and FComOpen}
var
  Buf: TIdBytes;
  LenRd, BufLen: integer;
  RxSer, TxSer, RXTcp, TxTcp: integer;
begin
  RxSer := 0;
  TxSer := 0;
  RxTcp := 0;
  TxTcp := 0;
  BufLen := 50;
  FTcpComm.ReadTimeout := 30;
  while (FComOpen and FTcpConnected and FRelayOn) do
  begin
    //First check TCP data
    try
      SetLength(Buf, 0);
      if FTcpComm.IOHandler.CheckForDataOnSource(10) then
      begin
        FTcpComm.IOHandler.ReadBytes(Buf, -1, false);
        if Length(Buf) > 0 then
        begin
          Inc(RxTcp, Length(Buf));
          SerWrite(FComH, Buf[0], Length(Buf));
          Inc(TxSer, Length(Buf));
          LogHex('Rx', Buf);
          SetLength(Buf, 0);
        end;
      end;
    except
      on E: Exception do
         Showmessage('TCP read exception: '#13 + E.Message);
    end;
    //Next check serial data
    SetLength(Buf, BufLen);
    LenRd := SerRead(FComH, Buf[0], BufLen);
    if LenRd > 0 then
    begin
      SetLength(Buf, LenRd);
      Inc(RxSer, LenRd);
      try
        FTcpComm.IOHandler.Write(Buf, LenRd);
        Inc(TxTcp, LenRd);
        LogHex('Tx', Buf);
        SetLength(Buf, 0);
      except
        on E: Exception do
           Showmessage('TCP write exception: '#13 + E.Message);
      end;
    end;
    //Finally show data and check events
    stxSerRx.Caption := IntToStr(RxSer);
    stxSerTx.Caption := IntToStr(TxSer);
    stxTcpRx.Caption := IntToStr(RxTcp);
    stxTcpTx.Caption := IntToStr(TxTcp);
    Application.ProcessMessages;
  end;
end;

Where could I be losing incoming serial data?

Bo Berglund wrote

Where could I be losing incoming serial data?

Only for completeness:

Can you rule out problems reagrding the hardware side? Mabe there are some weak line drivers or a faulty cable involved?

Worth checking, at least.

HTH anyhow,
Marc

Well,
I am using a 4-way USB to Serial unit and the cable (nullmodem) connects two adjacent ports and is about 50 mm long... This cable has never failed ever before.

I have now added extra binary logging to the relaying program only to find that the data received and processed are missing about 50 k of the 1M transmission.
Next I will edit serial.pp and up the buffers from 2K to 60K or so.

Bo Berglund wrote

Next I will edit serial.pp and up the buffers from 2K to 60K or so.

Question:
If I edit the content of serial.pp (the one that is brought up by right clicking and "Find declaration") such that it looks like this:

const   bufSize= 61440; //2048;

will it be used the next time I test run my application or must I do some kind of "installation" like updating Lazarus or similar for it to be used?

Bo Berglund wrote

I have now added extra binary logging to the relaying program only to find that the data received and processed are missing about 50 k of the 1M transmission. Next I will edit serial.pp and up the buffers from 2K to 60K or so.

By the way:
I have sent the Delphi test program I am debugging over to people at the remote location and they have connected their laptop directly to the data system and with that setup there is no data loss. So this leaves me with only the Lazarus/FPC network relaying app to fix...

I could also create another test program, a serial sniffer that could be set up to just log whatever appears on the serial lines. I need a 3-way cable for that of course.

Could that be some sort of timeout?

You're setting

  FTcpComm.ReadTimeout := 30;

but I have no idea what order of magnitude this 30 is. Maybe streching the timeout a little bit can help?

I had several cases of timeout problems using RS232 over TCP.

I could also create another test program, a serial sniffer that could be set up to just log whatever appears on the serial lines. I need a 3-way cable for that of course.

Such thing does exists, I only forgot the name(s), ;)

HTH,
Marc

Bo Berglund wrote

Inc(RxTcp, Length(Buf)); SerWrite(FComH, Buf[0], Length(Buf)); Inc(TxSer, Length(Buf)); Where could I be losing incoming serial data?

Hello,

Where do you check that "Length(Buf)" has been sent ?

if SerWrite(FComH, Buf[0], Length(Buf))Length(Buf) then Raise
Exception.Create("KBOOM");

There are two links in the bottom paragraph on this page: http://ctrlterm.com/custom.htm

Regards,
Paul
www.ControlPascal.com

Bo Berglund wrote

I could also create another test program, a serial sniffer that couldbe set up to just log whatever appears on the serial lines. I need a3-way cable for that of course.

I'd also suggest that that sort of thing works better with on-board or at least PCI-connected serial ports rather than relying on USB-connected ports.

Bo Berglund wrote

Where could I be losing incoming serial data?

FTcpComm.IOHandler.ReadBytes(Buf, -1, false);
..
   SerWrite(FComH, Buf[0], Length(Buf));

SerWrite() returns the actual number of bytes it's written, which could be anywhere from the number passed as the parameter down to zero (if hardware handshaking's being used at a lower level). An intermediate value would indicate that a buffer's full somewhere, and that you should adjust your source address and length and have another shot.

TCP is a stream protocol and you can't make any assumptions about the amount of data you get in one call. It's not reasonable to assume that buffers for serial comms are limitless, 2K is entirely reasonable to support most serial comms protocols or an 80x25 screen.

No, the Tcp/IP part is not involved at all here. When the transfer to the server starts there is no response expected until all data have been transferred.
The problem is that the Win7 application sends RS232 data out a serial port and another application (my relayer) is using another serial port which is connected by wire to the first serial port. This relaying application seems to lose incoming data, which are sent at a speed of 38400 baud.
Before the data arrives to the TCP/IP part they have been corrupted. The test on location proves that the case where the relayer is not involved (wire goes from Win7 application directly to the data system) is working fine.
But I cannot use that because I am in Sweden and the data system is in Texas...

Also,
before I wrote my own relaying application in Lazarus I tried to find something for Windows on-line like ser2net for Linux, which I use in the receiving end on a Raspberry Pi. But I had no success. The connection to the remote system goes via a VPN tunnel from my Win7 computer so I cannot use a Raspberry Pi in my end to handle the relaying. It does not see the remote network...

Well as you can see I don't.
However, I have not had any problems *receiving* data from the remote server (this is where the SerWrite would be used to send it into my Win7 program).
What I see is losing data when sending to the server, i.e. not all bytes on the wire from the Win7 program are actually received by my relaying code:

    SetLength(Buf, BufLen);
    LenRd := SerRead(FComH, Buf[0], BufLen);

Here data are arriving but some are lost, so the total number of bytes relayed are too few.
THey are not lost on the TCP/IP outgoing side because I am counting and showing the number of received bytes and this counter does not ever go above 1000000, it stops somewhere at 970000 or so.

if SerWrite(FComH, Buf[0], Length(Buf))Length(Buf) then Raise Exception.Create("KBOOM");

But I am losing data in the *receive* end, not while sending back to Win7.
As described I am sending 0x0FF000 bytes of data out of the Win7 serial port but only less than a million is counted in the serial receive end of the relaying program. So too few bytes are sent over the TCP/IP link.

I think I will have to create a bogus image file where I have a regular pattern of bytes so I can see when the hickups occur in the log. In a normal memory file most of the data is zero, so there are long stretches of zero bytes sent. Therefore it is not easy to see where the losses occur, I only see that the last part of the file (which contains some data) has been displaced to lower addresses.

What structure does the Windows side have, ist your application running on the same machine as the remote control program for the data system or is there a second computer attached using cables?

The most important part is to see where exactly data is vanishing. Which I don't at the moment. There is only one small part of the transmission path involved, running from the RS232 out of the windows app to the relaying app, as I understand. And this only happens in the direction from the win app to the remote system. Did I get that right?

I did so and discovered that loss of data started after some 48-50 Kbytes had been transferred. At that point a few bytes were missing. But the losses got progressively worse as the transfer went on and in the end there was about 50 K lost.

Then I tested a few things (like adding delays etc) but with no real difference.

FINALLY FOUND A SOLUTION...
----------------------------
Finally I realized that the relayer was very busy during the transfer since it was displaying the relayed data in a log window (a listbox) I had put on the screen!

In the RelayData procedure there are two calls like this for the two data directions:

  LogHex('Rx', Buf); //Incoming TCP/IP data
and
  LogHex('Tx', Buf); //Incoming RS232 data

The LogHex procedure does this:

procedure TfrmMain.LogHex(Prefix: AnsiString; var Buf: TIdBytes);
var
  i, l: integer;
  sLine,
  sTime: AnsiString;
begin
  sTime := FormatDateTime('yyyy-mm-dd hh:nn:ss.zzz', Now()) + '  ';
  l := Length(Buf);
  sLine := Prefix + ': ';
  for i := 0 to l-1 do
  begin
    sLine := sLine + IntToHex(Buf[i],2);
  end;
  lbxLog.Items.Add(sTime + ' ' + sLine);
  lbxLog.ItemIndex := lbxLog.Items.Count-1;
end;

Then I added a checkbox on the application form to switch on/off this logging to the listbox.
I put this check as the first line in the LogHex function:

  if not ckbShowLogData.Checked then exit;

Next I unchecked the checkbox before doing the long transfer, and lo-and-behold(!) now there are no losses anymore!

So the action of writing text to the listbox in the LogHex function was actually causing the application to lose incoming serial data!

Serial buffer size?
-------------------
I thought that upping the serial buffers to 60 K would have helped but apparently not. Maybe the change I did was not used after all?

What I did regarding buffer change:
1) Rightclicked SerSetParams() and selected "Find declaration"
2) This brought up the serial.pp unit at the declaration
3) Used Ctrl-Shft-downarrow to find the implementation
4) This had to be repeated for SerSetParamsPrivate()
5) In this function body the first line was this:
const bufSize= 2048;

which I changed to:
const   bufSize= 61440;

6) Then I rebuilt my application, but I have no idea if this actually caused any buffer change. I know too little about the inner workings of the Lazarus/FPC system...

So it might still use the tiny 2K buffer???

I would decouple the log display from the serial handling routines - by delegating the serial task to a thread, only display of log is handled in the main thread (this is true whether your application is cli-only or gui - have the tasks that should not be interrupted, like bulk serial communication, handled by a thread, and whatever the thread wants to display, use a buffer (like TThreadList) to communicate to the main thread.

-L.

Yes, when I saw this I figured that the data handling should be done in a thread.
However, I am not at all sure how this can be accomplished. Basically all of the activity is handled inside my RelayData() function. This is where the Tcp and RS232 data are crossfed and also where I call the logging function.

But if I create a thread and put this function into it, how would that work concerning access to the IdTcpClient and Serial objects? They were set up and connected in the main application and nothing but the actual data handling needs to be put inside the thread if it can communicate the logging info in a way that makes it non-disturbed.

But how?
I have 4 TStaticText controls that show the accumulated byte counts in the Rx and Tx channels of the socket and comport. These are updated in RelayData(). And of course I have the listbox displaying the data going back and forth. I think that the problem is in the listbox, which will get more and more data as time goes by... Probably I should have some other logging/display system here.

I have very limited experience with threads and then these did not interact at all with any GUI functions.

Hello,

This is a big "NO, NO!" for quite large buffers, that code can take a lot of time for a 50K buffer, as it will render in the expected 50K inttohex conversions, but also (in worst and impossible case scenario) 50K memory moves, 50K thread locks, 50K memory allocations and 50K memory frees.

Much better something like:

SetLength(sLine, Length(buf) * 2);
Bin2Hex(@Buf[0],sLine[1],Length(Buf));
sLine := Prefix + ':' + sLine;

Please recheck params syntax for Bin2Hex, my memory is not as it used to be :-)

A large listbox is not also very good.

Bo Berglund wrote

But how?

Never mind this...
I found the wiki article:
http://wiki.freepascal.org/Multithreaded_Application_Tutorial

this tells you right off that you should have at least XON/XOFF flow control so the receiver can tell the sender to hold up a second and let it get the buffer worked down... at least this with good buffer level monitoring...

Bo Berglund wrote

The problem is that when the Windows app is done sending the data system is still missing many kilobytes.... So no ACK is returned, it is still in receive mode.

On serial lines bits and bytes always can get lost or be distorted, independent from OS, hardware, ...

Any data transfer software needs to use appropriate means (i.e. appropriately small packets. counting packet number. ACK block containing the packet number it answers to, timeout, retransmit, ...) to handle this.

Moreover, with USB serial adapters I rather often found that the loose the connection to the OS and can't be re-established without a power cycle.

-Michael

On 06.09.2017 20:35, Bo Berglund wrote:....

...

(i.e. appropriately small packets. counting packet number. ACK block containing the packet number it answers to, timeout, retransmit, ...)

Not to forget: save detection of block start and block end (and hence introducing a decent transparentizing scheme, and using a block check (e.g. CRC).

-Michael