In serial components I have used with Delphi before there was always a property to set the buffer sizes (both Tx and Rx). But I cannot find a way to do this with SdpoSerial.
Is it not needed (does SdpoSerial manage the buffers by itself) or have I overlooked some property? I have not installed SdpoSerial in Lazarus as a component since it is non-visual. I create it in code instead.

I am wondering about this because I am now reaching the point in my project where I have to send large data arrays (up to 1 Mbytes) and I am not sure if I have to check the transmit buffer before I submit the data to the SdpoSerial.

What will happen if I have not set any buffer size (so they are at default values, whatever that is) and then use the method

  FComm.WriteData(cmd);

with cmd being a string containing 1 Mbytes of data? (FComm is of course a TSdpoSerial instance)

Bo Berglund

Bo Berglund wrote

In serial components I have used with Delphi before there was always a property to set the buffer sizes (both Tx and Rx). But I cannot find a way to do this with SdpoSerial.

Does SdpoSerial use buffers at all ? With AsyncPro, Buffers necessary to convert the blocking System interface into the event driven AsyncPro user API by doing the blocking system calls in threads. With the receiver, the collected data needs to be buffered until the main thread is available to handle it, With the sender, the data generated by the main thread needs to be buffered until the system API unblocks the interface it blocked to prevent overrun of the system buffers.

Without using threads the user just sees a blocking interface and no buffers are necessary.

-Michael

Michael Schnell wrote

Without using threads the user just sees a blocking interface and no buffers are necessary.

Don't do this in the main thread, otherwise the complete application hangs.

-Michael

AFAICT SdpoSerial is a wrapper for the Synaser blocking serial component to make it more "user friendly" by providing receive events when data arrive.
And my question is really about the existence of buffers and in that case how to adjust them?

I don't want to mess with the component files myself, they are way too intricate, I just want to use the serial comm functions. But I need to know if there will be loss of data if I use a write command with a very big string argument and the communications buffers are not big enough to store this string. What will happen?
- Loss of data?
- Reverting to blocking operations (ouch!)?

Of course I can test it by actually sending a large chunk of data, but then I might use test data that are just a tad too short to reveal a buffer problem. Better to know how it will react.

And another important thing should I not need to set a buffer size: How can I know when the data I submitted to SdpoSerial has actually left the serial port on to the wire? A TxBuffer.DataCount property or similar would come in handy here if it existed....

Bo Berglund

i don't know if this makes any difference but i'm trudging thru the synapse library attempting to work with its mime coding stuff... in my travels thru the documentation i note that synapse is synchronous instead of asynchronous (see: http://www.ararat.cz/synapse/doku.php/about )... the third and fourth paragraphs are important...

since the synapse library has synaser as one of its sub(?)libraries, i assume that we're talking about the same library...

the reason i mention the above is because i'm not sure how it might affect what you are attempting to do and if buffers are really needed... it has been a while since i did any comm coding which, at that time, was done directly to the com port or via a FOSSIL driver (which greatly alleviated and abstracted much)...

i hope this helps in some small way...

Gosh, I did not know this....

For TCP/IP work I always use Indy, which is now also available and maintained for freepascal.

But in the serial comm projects I am used to components that are non-blocking, which means that a send method returns immediately even before all data have been sent. Then the data are handled by a thread maintained by the component internals (or Windows perhaps). The net effect is that the main application is not blocked and non-responsive while the transfer takes place.

In my case I am using SdpoSerial because it is included with FPC/Lazarus. I am sending a data block that potentially can be 1 Mbytes in size over a channel operating at 38400 baud or slower. Theoretically this means a transfer time of 273 seconds (4.5 min) or more. My application must meanwhile give some feedback to the user that the transfer is happening and also how it is proceeding.

So I need to know in the case of SdpoSerial that it is possible without data loss to write a string of 1 Mbytes size. I also need to know how I can examine the progress of the transfer so I can control a progress bar or similar on screen while the transfer is running.

While the transfer runs my program sits and waits for an ACK coming back from the other end after it has checked the checksum. That is a loop with Application.Processmessages, Sleep and a timeout...

In a blocking design this wait loop would of course not have been needed becaus ethe Write would not return until all data have been written to the output.

Bo Berglund

Hello Lazarus-List,

Monday, February 14, 2011, 8:32:37 PM, you wrote:

BB> While the transfer runs my program sits and waits for an ACK coming BB> back from the other end after it has checked the checksum. That is a BB> loop with Application.Processmessages, Sleep and a timeout... BB> In a blocking design this wait loop would of course not have been BB> needed becaus ethe Write would not return until all data have been BB> written to the output.

So you can send in small chuncks (64 bytes) and give feedback every 250ms.

Or use a "send" thread.

On 14/02/2011 19:32, Bo Berglund wrote:

First one thing, your message subject is too broad: Are we talking about the RX or thr TX buffer?
Then, are we talking about the component, the library or the OS buffers? And then are we talking about Windows 32, 64 bits or Linux? It is hard to answer for all cases...

Synaser is blocking both for writing and reading.
The SdpoSerial component creates a thread to listen and generates an event when bytes are received. So reading can be non-blocking. The write methods are direct calls to the Synaser ones and they are blocking.

In a blocking way (the function will only return when most of the bytes are already sent), all bytes will be transmitted.

I also need to know how I can examine the progress of the transfer so I can control a progress bar or similar on screen while the transfer is running.

For that, you must break your transmission in small block and use the completion of each transfer to signal the progress.

While the transfer runs my program sits and waits for an ACK coming back from the other end after it has checked the checksum. That is a loop with Application.Processmessages, Sleep and a timeout...

I don't know if that is an option but I would break such big transmission in smaller packets each one with its own checksum...

Paulo Costa

Paulo Costa wrote

On 14/02/2011 19:32, Bo Berglund wrote: First one thing, your message subject is too broad:

Sorry about that, I just meant to ask about the way the Tx and Rx buffers are handled since I did not find any property for them to be set...

Are we talking about the RX or thr TX buffer?

Both really.

Then, are we talking about the component, the library or the OS buffers? And then are we talking about Windows 32, 64 bits or Linux? It is hard to answer for all cases...

I have turned to FPC/Lazarus because I need the software to be portable between Windows, Linux and embedded Linux on ARM. I had the impression that I did not have to bother with the distinctions you are making (write once compile everywhere...).

Does that mean that until all of the bytes have left the serial port the Write does not return to my program? This is exactly why I asked about the buffers...

What does "most" mean? And do you mean when they have actually left the UART on to the RS232 wires?

Are you really sure that the Write in SdpoSerial is blocking? So there are no internal transmit buffers that receive the data and then the call returns and the work is done by a transmit thread????

I can do that for the general memory transfer function because it is defined such that the sender defines what address range is to be sent. But it will add handshaking overhead to the transfer.

The instrument protocol for this is as follows:
Memory transfer:
Tx: <command byte><destination address><number of bytes> Rx: ACK (if the instrument can receive the data, NAK otherwise Tx: <length of data><binary data><checksum>
Rx: ACK if all bytes received and checksum is OK, NAK otherwise (Of course if not all bytes according to the length are send the instrument waits forever for the remaining data so there will be no NAK in this case).

But file transfers are different, they can only be transferred complete.
And I cannot change the protocol...

Sorry in advance for piping in so late in the discussion...

Windows and Linux serial implementations are very different in design from a buffering standpoint. In simple terms, the windows serial was a afterthought whereby serialtty in linux is as simple as reading and writing. No controls can be expected to be near flawless to seamlessly work between these platforms.

To keep it short... As long as you are not doing visual work (qt,gtk,etc), cross compiling with fpc is very well built. I have a Ubuntu 10.10 virtual machine (vmware) I use for (i386/arm9) linux cross compiling that you can use if you want it. Its out of the box functionality. Just send an email to if you want a copy of it. I am currently having issues getting the qt sdk (4.6.3) interfacing, and would smile greatly for anyone who would want to join efforts to solve the problems. Otherwise, I will just keep working away at it...

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus.freepascal.org/pipermail/lazarus/attachments/20110215/e49ae19d/attachment-0001.html>

I just checked by setting up a transfer of 100.000 bytes at 38400 baud. With a breakpoint at the FComm.WriteData method I could time this function using the step-over command. It returned in 30 s when the packet was 100006 bytes long, an effective baudrate of 33300 baud. So it really looks like the WriteData *is* blocking...

For that, you must break your transmission in small block and use the completion of each transfer to signal the progress.

Sadly this seems to be the case...

I would have to send in chunks of 1500 bytes or so to get 2 progress ticks per second. The application will probably be stone dead and nonresponsive too, which is one of the things I really like to *always* avoid.

What one would need to do to fix this is to put the SdpoSerial into a thread and let the thread send data out from a buffer that can be loaded by the main application. But then we have the problem of transferring the Tx data into the thread environment in a threadsafe way...

My project just grew. :-(

Bo Berglund

Even if it were non-blocking, a block is required when you try to overflow the output buffer.

DoDi

may i suggest looking at elecomm for an ideal it's a old pascal serial library but would fit your needs

Bo Berglund wrote

That is a loop with Application.Processmessages, Sleep and a timeout...

Thus completely ignoring the event driven Object Pascal programming paradigm.

Yak.

-Michael

Bo Berglund wrote

In a blocking design this wait loop would of course not have been needed becaus ethe Write would not return until all data have been written to the output.

But here the program's GUI stops working, as the software hangs in the blocking read or write,

So IMHO, the only decent way to handle a serial interface is doing blocking communications calls in threads and fire events in the main thread (e.g. by Application.QueuAsyncCall (avoiding the "Windowish" PostMessage() stuff ) ) .

And here, of course, you do need buffers to transfer data between the main thread and the threads

So I feel that SdpoSerial needs a buffer to hold the data read from the interface until the main thread is inclined to bother. The size of this buffer should somehow be configurable.

-Michael

Bo Berglund wrote

Does that mean that until all of the bytes have left the serial port the Write does not return to my program? This is exactly why I asked about the buffers...

Regarding sending, there still is a FiFo (usually some 4..64 Bytes ) in the serial hardware (hundreds or thousands of bytes) and a software Fifo in the OS driver. So the software can't tell when something has left the serial port. For such deeply embedded issues the specifics of the driver and hardware needs to be considered

-Michael

Hello Lazarus-List,

Wednesday, February 16, 2011, 10:59:02 AM, you wrote:

Bo Berglund wrote

> Does that mean that until all of the bytes have left the serial port > the Write does not return to my program? This is exactly why I asked > about the buffers...

MS> Regarding sending, there still is a FiFo (usually some 4..64 Bytes ) in MS> the serial hardware (hundreds or thousands of bytes) and a software Fifo MS> in the OS driver. So the software can't tell when something has left the MS> serial port. For such deeply embedded issues the specifics of the MS> driver and hardware needs to be considered

AFAIK the FIFO is in the receive, in sent the FIFOs are filled but function does not return until the hardware sends the last byte.

Jos? Mejuto wrote

AFAIK the FIFO is in the receive, in sent the FIFOs are filled but function does not return until the hardware sends the last byte.

If this is true somewhere, it is not portable.

-Michael

Hello Lazarus-List,

Wednesday, February 16, 2011, 1:26:38 PM, you wrote:

Jos? Mejuto wrote

> AFAIK the FIFO is in the receive, in sent the FIFOs are filled but > function does not return until the hardware sends the last byte.

MS> If this is true somewhere, it is not portable.

Glups! What ?

MSDN about RS232: Nonoverlapped I/O is familiar to most developers because this is the traditional form of I/O, where an operation is requested and is assumed to be complete when the function returns.

In non-overlapped mode, block mode, or whichever you like to call it, the program sends, in example, 100 bytes to rs-232, the operative system sends 16 of them to the UART, when FIFO bytes has been sent the UART generates an interrupt to tell to the OS to refill the FIFO buffer. This procedure is repeated several times, when no more data is pending in the OS buffers to be sent, the OS waits for the interrupt or timeout interrupt, and when the transmission buffer is empty (interrupt received) the OS exits the send function. FIFO in transmission alleviate the overhead in interrupts 16:1.

The problem is in the timeout case as you will not know at which byte the transmission timedout as the FIFO can only be asked about empty or not.

Non blocking is pretty the same at the OS level, except that the send function returns inmediatly and you have some helper functions to inquiry the current transmission status/state/progress.

Exactly...

I'm not sure, but IMO a FIFO for outgoing data is used at best in a *synchronous* protocol, not in a UART. More likely synchronous I/O is done by DMA, not in code. But which nowadays hardware is capable of synchronous serial transmission at all?

DoDi

Hello Lazarus-List,

you wrote

> AFAIK the FIFO is in the receive, in sent the FIFOs are filled but > function does not return until the hardware sends the last byte.

HPD> I'm not sure, but IMO a FIFO for outgoing data is used at best in a HPD> *synchronous* protocol, not in a UART. More likely synchronous I/O is HPD> done by DMA, not in code. But which nowadays hardware is capable of HPD> synchronous serial transmission at all?

Now you know why PCSpeaker has been removed from Windows ;) 8250 chip which carries RS232 has been removed and it also controls the PCSpeaker. The 8250 is interrupt driven without FIFO and the 16550A is interrupt driven and FIFO 16 bytes.
16550A and alike are no more integrated in PC motherboards because it is the last real IRQ and port mapped device so it overcomplex circuitry and OS to handle it. Anyway the APIs are based in the same concept as when it was interrupt and I/O driven (around 3 years ago).

Of course the 16550A chip was not included since a lot of years, but the circuitry in replacement was compatible in operation mode (IRQs / I/O).

In my application I use an object that encapsulates the communication with the instrument. I create an SdpoSerial object to do the actual serial comm, but I also had to create a TCommBuffer class to handle the data *from* the RS232 receive. So in the receive event of the SdpoSerial I just dump incoming characters into this TCommBuffer object, which keeps track of write and read indices.

The data are stuffed into a dynamic byte array, which is enlarged on demand (but in large chunks so the overhead of expansion will not be noticed).
I did this for receive in order to handle the "strange" protocol used by the instrumentation.

Maybe I should expand this and use the same for transmit also, such that the main application merely adds data to the buffer and a thread sends the data onwards into the blocking WriteData method? That way I can monitor the difference of the read and write indices as a "remaining to be sent" value in a wait loop in my main application.

I might even let the thread create the SdpoSerial object so it is only touched by the thread itself...
But I don't know what would happen because the SdpoSerial object has internal threads of its own, I believe.

Bo Berglund wrote

I might even let the thread create the SdpoSerial object so it is only touched by the thread itself...

If you create an object in a thread that does not mean that it's code will run in that thread.

But I don't know what would happen because the SdpoSerial object has internal threads of its own, I believe.

So maybe it might not work at all or crash when created in a thread.

-Michael

At least in the embedded world UART:s have *both* a transmit and receive FIFO. You usually are able to set the interrupt trigger such that it fires when there are a certain number of bytes in the Rx FIFO and when it reduces below a certain number of bytes in the transmit FIFO. This way one can fill the FIFO with outgoing data and then go away doing something else until the iterrupt fires at which time the interrupt routine will move available data from the buffer to the FIFO.

Hello Hans-Peter,

you wrote

> the 16550A is interrupt driven and FIFO 16 bytes.

HPD> And the FIFO resides in the receiver, the transmitter doesn't have a HPD> FIFO. At least it was so, 20 years ago...

Both have FIFOs, in fact the FIFO was implemented for the transmitter (sender) to achive a speed higher than 38400 bps and before the chip goes out a receive FIFO was added also.

Quote: --------------------

RBR : Receiver buffer register (RO) The RBR, receiver buffer register contains the byte received if no FIFO is used, or the oldest unread byte with FIFO's. If FIFO buffering is used, each new read action of the register will return the next byte, until no more bytes are present. Bit 0 in the LSR line status register can be used to check if all received bytes have been read. This bit wil change to zero if no more bytes are present.

THR : Transmitter holding register (WO) The THR, transmitter holding register is used to buffer outgoing characters. If no FIFO buffering is used, only one character can be stored. Otherwise the amount of characters depends on the type of UART. Bit 5 in the LSR, line status register can be used to check if new information must be written to THR. The value 1 indicates that the register is empty. If FIFO buffering is used, more than one character can be written to the transmitter holding register when the bit signals an empty state. There is no indication of the amount of bytes currently present in the transmitter FIFO.
--------------------------

You can get further information from http://www.lammertbies.nl/comm/info/serial-uart.html

Hello Lazarus-List,

you wrote

>AFAIK the FIFO is in the receive, in sent the FIFOs are filled but >function does not return until the hardware sends the last byte.

BB> At least in the embedded world UART:s have *both* a transmit and BB> receive FIFO. You usually are able to set the interrupt trigger such BB> that it fires when there are a certain number of bytes in the Rx FIFO BB> and when it reduces below a certain number of bytes in the transmit BB> FIFO. This way one can fill the FIFO with outgoing data and then go BB> away doing something else until the iterrupt fires at which time the BB> interrupt routine will move available data from the buffer to the BB> FIFO.

Right, I do not correctly write the idea. I was trying to say that the FIFO in acts like a buffer that you can forget about it (until no receive event/interrupt) is received, while in the send FIFO the OS waits for a full empty/transmission before return from call.

Are we completly off-topic ?

Jos? Mejuto wrote

Are we completly off-topic ?

Maybe a tad.... ;-)
But it is about handling serial data in the best way for the application, though.

Jos? Mejuto schrieb:

Thanks for your FIFO research :-)

Quite a bit. Applications don't have to care about such hardware details, because the drivers of most systems will buffer all I/O, and will allow for both synchronous and asynchronous I/O.

DoDi

Bo Berglund wrote

At least in the embedded world UART:s have *both* a transmit and receive FIFO.

Same with PCs. And AFAIK, the OS-driver adds a huge Fifo buffer in both directions to allow for the quite latency userland software to work as good as possible without degrading the transmission performance.

-Michael

Hans-Peter Diettrich wrote

Quite a bit. Applications don't have to care about such hardware details,

Yep and this is why user land software can't (and should not) detect the point in time the data it issued towards the serial port actually has left the box. These systems (Windows, Linux, Mac) are not supposed to support hard realtime after all.

-Michael

There exists no guarantee for undisturbed transmission. Secure software must be prepared to re-transmit the data, in its protocol, regardless of realtime requirements. It's perfectly sufficient when the software is informed that the next data packet can be supplied for transmission.

DoDi

But then we have the problem I have already detected when test running my SdpoSerial based application:
If I set it up to send a large chunk of data and put a breakpoint at the WriteData call so I can time the process when I click step-over I see this:
Data size 100.000 bytes
Baudrate: 38400
Estimated shortest time for transfer: 26 s
Time to return from call: about 30 s

So this means that even if the OS has a "huge" buffer for outgoing data it fails to improve the call time of the blocking call to WriteData.
My initial question was about adjustment of the size of the transmu?t buffer such that I could ensure safe transfer of the data and snappy response in my application.
As it is now the GUI will die when the transfer is made. I have to somehow adapt to this fact and add a buffer layer myself in a thread that feeds data to the WriteData method.

Hans-Peter Diettrich wrote

It's perfectly sufficient when the software is informed that the next data packet can be supplied for transmission.

IMHO the decent way for user land software is to use a timeout that is _a_lot_ greater than the time the hardware needs to toggle out the serial bits plus to receive biggest possible the answer and resend the block if no answer is detected up till then. This is not hard realtime. The user land software can at any time be delayed for multiple seconds by other proceedings on the PC.

Of course a decent protocol should use a running block counter so that the receiver can detect a block it received a second time.

-Michael

Bo Berglund wrote

I have to somehow adapt to this fact and add a buffer layer myself in a thread that feeds data to the WriteData method.

That is why I suggested that the best way to go is porting the appropriate parts of AsyncPro, that perfectly does this.

Some time a go I did a quite in-depth research on this issue, yielding this result, and I of course would have already done it, unless the project would have been delayed for other reasons.

-Michael

Michael Schnell wrote

porting the appropriate parts of AsyncPro, that perfectly does this.

BTW be warned: The necessary Event-mechanism (e.g
Apllication.QueuAsyncCall) does not work correctly with the current svn Lazarus version and "gtk2" (maybe it does work with the "Windows" and/or the "qt" Widget Type)

This might hit you when using threads for data transfer and trying to do
it in a "decent" way (without polling loops).

-Michael

Hello Lazarus-List,

you wrote

> Both have FIFOs, in fact the FIFO was implemented for the transmitter > (sender) to achive a speed higher than 38400 bps and before the chip > goes out a receive FIFO was added also.

HPD> Correction: the FIFO was implemented first for the receiver, because HPD> lost bytes can not be recovered, and not all lines or senders allow for HPD> hardware handshake (RTS/CTS) - that was a big problem with the old HPD> devices. The baudrate is generated by the device, so that sending never HPD> suffered from timing problems. [In synchronous protocol the device HPD> inserts sync codes automatically, when no new data is supplied in time]

Well, maybe we can talk for hours about this :) as I can not show you a document about this topic, but I'll show you my background about why transmission buffer FIFO was more important in the beginning. In the past times most communications in serial devices was synchronous, so in the receiver part due the possible errors in transferences and, of course, buffer overruns some protocols can have some simple error correction techniques. The problem appears when serial communications want to be far away from 38400 bps as the time needed to raise the interrupt and just put a new byte in the buffer (1 byte buffer) is more than the character frame time which in fact raises a buffer underrun error and a serial comunication break (break, not error). What a change in less than 20 years! :)

Hello Lazarus-List,

Friday, February 18, 2011, 4:16:31 PM, you wrote:

MS> BTW be warned: The necessary Event-mechanism (e.g MS> Apllication.QueuAsyncCall) does not work correctly with the current svn MS> Lazarus version and "gtk2" (maybe it does work with the "Windows" and/or MS> the "qt" Widget Type)

You can use Syncronize, the syncronized procedure raises the event in your code.

Jos? Mejuto wrote

You can use Syncronize, the syncronized procedure raises the event in your code.

Unfortunately, with these Widget sets, TThread.Synchronize does not work, either :( (see FPC bug tracker)

Application.QueueAsyncCall is more suitable, as it
1) allows to transfer the data block to the main thread in a single step and
2) does not block the thread while it waits for handling by the main thread (Which is done in "TThread.Synchronize" (but not in the Delphi-only "TThread.Queue").

-Michael

As I think I mentioned I once tried to just see which AP units were involved with the serial port stuff and it was almost all of them... Furthermore I want a simple cross-platform working serial port handler, because in the end the target is embedded Linux for ARM and not Windows or Linux on x86 platforms....

I am hoping that SdpoSerial is just this cross-platform one so I am working around its problems rather than embarking on something I am not good enough doing.