I want to change the _haltproc for freertos ESP controllers to use normal write/writeln functionality to send information to standard output. On these targets the output is typically buffered serial provided by the OS. Text output written just prior to setting the cpu to sleep doesn't get sent, hence a flush is needed to complete data transmission before putting the cpu to sleep. there doesn't seem to be a
convenient way to provide a flush helper function. Also the RTL Flush procedure calls InOutFunc and not FlushFunc.

What is the proper way to hook a flush helper function into the existing RTL infrastructure?

Simply do this in InoutFunc ?

Michael.

Thanks for your response Michael. Using InOutFunc to also flush the output buffer will work, but that seems inefficient, since the flush needs to wait until the transmit buffer is empty (at slow UART speeds this could potentially take several ms to complete). Is there a specific reason why the RTL Flush procedure does not call the FlushFunc method?

I checked; That code is so old, no idea.

From what I can see the flush code could be changed to

if Assigned((TextRec(t).FlushFunc) then
   FileFunc(TextRec(t).FlushFunc)(TextRec(t))
else
   FileFunc(TextRec(t).InOutFunc)(TextRec(t));

But keep in mind that the InoutFunc() is only called when actually writing data, meaning: when the internal text buffer is full or on a terminal with every writeln(), so I don't think it is inefficient. Writeln() will do a flush. Whether this happens in .InOutFunc or .FlushFunc is largely irrelevant.

Michael.

Michael Van Canneyt wrote

On Mon, 8 Jun 2020, Christo Crause via fpc-devel wrote:

.
.

The question is whether the potential change would make any difference. IMHO, the important points are:

1) The point of flushing is making sure the I/O is really performed (the data are sent to the operating system _and_ flushed from internal buffers of the underlying operating system / platform). Doing the latter without the former makes no sense.

2) From semantic point of view, all data should be sent to the final place (block device / console / ...) before the call to Flush is finished (otherwise the following actions may have incorrect results - the program might finish before all data are transmitted, other I/O may be invoked, etc.).

From this point of view, the proposed change would only lead to code duplication between FlushFunc and InOutFunc and / or to increased code - e.g. checking FlushFunc being nil before calling InOutFunc just to call InOutFunc (or to perform functionality currently included in InOutFunc) from within FlushFunc anyway.

Tomas

That is why I said 'could'. I didn't intend to change anything.

Michael.

Tomas and Michael, thank you for this discussion. I'm starting to think the use case I'm considering is perhaps not covered by the current design. At the moment I directly call the OS flush function before putting the cpu to sleep and that works; I was just wondering if there is a Pascal way to accomplish this so that a user doesn't have to remember to call a nonstandard function when a Flush function already exist in the RTL.

I don't understand why you don't do the flush in the InOutFunc ?

That's what it is for: the text filetype already does the buffering for you. So any performance impact is IMO unavoidable: when InOutFunc is called, the RTL really needs it to clear the buffer.

Michael.

Michael, it is entirely possible that I am misunderstanding your suggestion, I probably don't grasp the complexities of the RTL texmode IO system and also not explaining my intention clearly, for this I apologise. Take for example the esp32 unit provided by Florian, here the link between the underlying OS output and the RTL is configured by calling OpenIO and providing readchar and writechar functions which in turn calls getchar and putchar (
https://github.com/ccrause/freepascal/blob/76911c528251b48e18c348de1758c098ee9fb763/rtl/freertos/xtensa/esp32.pp#L106). In this unit Florian uses a custom printpchar procedure which pushes characters to the OS and at the end directly calls the OS fflush function. This ensures that the haltproc message gets transmitted before the cpu goes to sleep.

I'm trying to change the controller unit to use standard RTL write functionality, since this gets configured during unit initialization anyway. My attempt at "Pascalizing" Florian's version ( https://github.com/ccrause/freepascal/blob/76911c528251b48e18c348de1758c098ee9fb763/rtl/freertos/xtensa/esp8266.pp#L59) calls the normal write functionality and works, except for the last string before esp_deep_sleep which doesn't reach the remote terminal. At the moment I fix this by directly calling the OS fflush function.

The only configuration option exposed via consoleio is OpenIO, which doesn't have a parameter for a low level flush function. I could create an overloaded OpenIO function with an extra parameter for a flush function pointer, and call this, if assigned, at the end of Console_Write. Is this kind of the lower level detail behind your suggestion Michael?

Best regards,
Christo

I would add a flushfunc to TUserData and to the OpenIO call in consoleIO:

Type
   TFlushIOFunc = function(UserData: pointer): boolean;
and change
       PUserData = ^TUserData;
       TUserData = record
         WriteChar: TWriteCharFunc;
         ReadChar: TReadCharFunc;
         FlushFunc : TFlushIOFunc;
         UserData: Pointer;
       end;
change
  procedure OpenIO(var f: Text; AWrite: TWriteCharFunc; ARead: TReadCharFunc; AMode: word; AUserData: pointer);
to
  procedure OpenIO(var f: Text; AWrite: TWriteCharFunc; ARead: TReadCharFunc; aFlush : TFlushIOFunc; AMode: word; AUserData: pointer);
and add
   userdata^.FlushFunc := AFlush;

At the end of Console_Write I would call this new flush function:

        if it.BufPos then
           InOutRes:=101
         else
           begin
           if userdata^.FlushFunc(userdata^.UserData) then
             InOutRes:=0
           else
             InOutRes:=101;
           end;
         t.BufPos:=0;
       end;

Console_Write is only called when the internal buffer is full or when the output must be forced to terminal, so that should be sufficient.

Hope this helps,

Michael.

The flushfunc is called from fpc_write_end and fpc_writeln_end. Those helpers are automatically called after all of the individual write operations from a write(ln) statement have been processed. So if you implement/assign flushfunc and make it call your OS flush function, it should work fine.

Jonas

Thank you Michael and Jonas for your help and suggestions. I think I ended up merging your suggestions, which does work for my corner case. If anyone is interested this change can be viewed here:
https://github.com/ccrause/freepascal/commit/b28ab72ed07f11677493d5d0e37ee451d2313323

I must admit that I was thoroughly confused by the apparent lack of semantic difference in use between InOutFunc and FlushFunc of TextRec structure.

Best wishes,
Christo

Could you please submit a patch with this modification to the bugtracker ?

We don't as a rule follow up on such things in github.

Michael.

AM Michael Van Canneyt wrote

Could you please submit a patch with this modification to the bugtracker ? We don't as a rule follow up on such things in github.

I will of course submit a patch once I'm satisfied it is good enough. My concern with the current patch is that a low level flush is called after every write statement, so a simple loop like the following:

for c := 'A' to 'Z" do write(c);

will incur the burden of a low level flush after each iteration. In normal use the low level flush call is not needed and causes unnecessary blocks. This is what Tomas mentioned, so I'm looking for a different implementation that can fit in with the rest of the use cases in the RTL.

It would make more sense to me if there was a distinct difference in use case between InOutFunc and FlushFunc (for fmoutput case), to my naive view it seems as if they are mostly used interchangeably and in many cases even share the same implementation function. This makes sense for the traditional targets (Windows, Linux etc.), but for lower level targets a distinction between the internal RTL buffer and the external OS/hardware buffer is useful. For a low level target I would think an InOutfunc that flushes the internal buffer to the OS/hardware and a FlushFunc that ensures the internal buffer is empty and then blocks until the OS/hardware has completed transmission would make sense.

Note that there is an (intended) difference between output to console and output to files (Do_IsDevice provides this differentiation in standard RTL). Output to console is supposed to be flushed after every Write(Ln) statement, because it should provide immediate feedback to the user. Output to files should not be flushed immediately. Obviously, it's up to the target maintainer to decide what's the proper behaviour in case of output devices connected to your target, but the semantics should be preferably kept, IMHO. Also note the difference between outputting a series of individual characters using individual Write calls (your loop above calling flush after every character) compared to outputting a whole string (flush called after outputting all the characters in the string).

Finally, note that TP-compatible part of the standard RTL does _not_ call the ("low-level") flush call of the underlying operating system. Such calls exist in API for most platforms (DOS: int 21h, AH=68h; OS/2: DosResetBuffer; Unix: (fp)fsync; WinXX: FlushFileBuffers), but they are not used within our RTL implementation as far as I know (flushing is always performed automatically when closing a file, but the operating systems take care of that).

Tomas

/me has tons of code that does this... especially with logging specifically to ensure that all the logging is actually written to disk in case of a crash which could result in hundreds of lines of logging data being lost... no problems of any kind noted...

Christo Crause wrote

for c := 'A' to 'Z" do write(c);

oops! i failed to note that the above is character by character whereas what i spoke of in my previous post is line by line :/

This already works well enough, the normal use case doesn't require a low level flush.

Indeed, the functionality implemented in the RTL is about managing the RTL buffers and leaving the downstream work in the hands of the OS (or whatever custom function is hooked to InOutFunc). I cannot see a simple way to re-use one of the fields in TextRec to provide what would amount to new functionality, this leaves the option of adding another field to TextRec for a low level flush. This is however not moving in the appropriate direction, since this would lead to higher memory consumption on lower end targets. Perhaps a separate platform specific low level flush is the best compromise for the few use cases requiring low level control.

Indeed, the first modification attempt will result in a low level flush per character for this case.