Since some time FPC doesn't flush output on pipes after writeln. On linux I can use a pseudo terminal, what can I do on win32 to get flushed output from FPC into the message window of MSEide?

Martin

Op Sun, 14 May 2006, schreef Martin Schreiber:

Since some time FPC doesn't flush output on pipes after writeln. On linux I can use a pseudo terminal, what can I do on win32 to get flushed output from FPC into the message window of MSEide?

It is strongly recommended against to parse the compiler output, as it is designed for humans, not computers. The formatting of the compiler output can therefore change in time and can also be configured by the user in any language. Some people are doing it anyway, like Lazarus.

The compiler has an interface designed for computers and that is through the compiler and comphook units. This allows you to retrieve the messages neatly. If you want process separation, please write a wrapper that puts the messages in a computer friendly way, and we'll include it in FPC.

About flushing the output, I think it is impossible to receive the output interactively currently, it will output in bursts. You should still be able to receive the output, though. If you want to receive it interactively, I think we can modify the compiler to flush after each message. But again, I do not recommend this approach.

Daniël

Unix (GNU?) tools usually have a --line-buffered option, that will set output buffering to flush-per-line, or even unbuffered. Such an option could be a good idea for the compiler as well, but: Martin, is time between first message and compiler termination so long that it needs flushing ?

Micha

What I would appreciate, is that the output I get remains in the same order as it is generated. So, first the compiler saying linking and then the linker giving error messages. See
http://www.freepascal.org/bugs/showrec.php3?ID=4996

Vincent.

I do not parse the output of FPC, I display it in a message window for humans. It is possible to continue working with MSEide while compiling (very useful because I must use -B the most of time with 2.0.3 because of compiler crashes and memory overloads). But I wish to see what's going on compiling in realtime.

Same problem with FPC console programs on win32, the MSEide target window uses a pseudo terminal on Linux -> output is flushed, on win32 I can only use pipes -> program output is only visible if output buffer overflows or on program termination.

Suggestion:
On win32 switch automatic output writeln flush only off by regular files (FILE_TYPE_DISK) but not by pipes (FILE_TYPE_PIPE):

rtl/win32/sysfile.inc

function do_isdevice(handle:thandle):boolean;
begin
  do_isdevice:=(getfiletype(handle)=2);
end;

change to:

function do_isdevice(handle:thandle):boolean;
var
  wo1: dword;
begin
  wo1:= getfiletype(handle);
  do_isdevice:= (wo1 = FILE_TYPE_CHAR) or (wo1 = FILE_TYPE_PIPE);
end;

Martin

Op Sun, 14 May 2006, schreef Michael Van Canneyt:

I think a command line output for computers needs to be a bit more advanced, take for example this error message:

testc.pas(11,1) Fatal: Syntax error, "BEGIN" expected but "identifier Z" found

We have:
* The source file
* The error location
* The error severity
* The error message, containing two parameters.

Both the severity and error message are currently unpredictable, i.e. they can be translated for example. Some IDE's may want to have access to the error messages their parameters as well.

A command line output for computers also needs to be extendable, if we want to output more information, this should not break existing applications.

For example, a good way to output this error message in a computer friendly format would be:

error:(file='testc.pas';line=11;column=1;message_code='02003_F';
       message='Syntax error, $1 expected but $2 found';
       parameters=('BEGIN','IDENTIFIER Z'));

Daniël

I would at least make it

   do_isdevice:=(getfiletype(handle)FILE_TYPE_DISK);

so it includes 'unknown' file types and future file types.

Michael.

We could do this ourselves by assigning a unique number to each error. (and keeping these numbers)

Michael.

Parsing the current error messages isn't that hard?

How can it find out, which message file to use?

Mattias

For the quick fix items (like eclipse and lazarus), we need the message ID and parameters, independent of current localization.

Mattias

We could do this ourselves by assigning a unique number to each error. (and keeping these numbers)

All messages already have an unique number. It is only not used in the output.

Peter

Op Sun, 14 May 2006, schreef Florian Klaempfl:

Parsing the current error messages isn't that hard?

Perhaps it is even easier than the format I proposed, but it is bad software design and asking for breakage.

Daniël

I'm all for such a message form; it should be easy to add with some -v switch... But
- This is no use if the message code changes over time, because that is still the unique identifier for the error...
- The message itself should not be given, IMHO.
The program can look it up in the error message file...

Michael.

There is only 1 message file ? errore.msg for english... located next to the compiler binary...

Michael.

So, in principle they don't change ?
I seem to remember occasions where the number changed ?

And so they could also be used to selectively switch off some messages ?

Michael.

I thought also this way but I didn't dare...

Martin

Michael.

I guess that this should be easy to fix by always
flushing Output before launching external
utilities. Am I overlooking something? If not, I
can probably add this.

Tomas

Op Sun, 14 May 2006, schreef Michael Van Canneyt:

I am starting to come to the conclusion that this is not the right approach. Of course such a fix is Windows specific, while pipes are used on Unix as well, imagine using less to paginate the compiler output.

Doing it on all platforms would help, but even a file is often used interactively, for example with "tail -f". Therefore, it seems much more logical to make the compiler flush after each message.

Daniël

The error numbers didn't change, they are keep unique.

I propose to do the following:
- add an option -vm (machine readable output)
Using this option the compiler outputs the messages csv like and flushes after each message:
message type <tab> file <tab> line <tab> column <tab> error number <tab> complete error message <tab> message template (i.e. Type $1 expected, $2 found <tab> parameter 1 <tab> parameter 2 <tab> parameter n

To avoid trouble with future versions, a header line will describe each column.

On Sun, 14 May 2006 12:10:14 +0200 (CEST)
Michael Van Canneyt wrote:

[...]

Some users are using the translated message files. How can a program find out, which one the compiler is using?

Mattias

That one you told the compiler with -Fe to use :)

I don't think that it is wise to flush always when writing to pipes only for the compiler. -vm could for now force an additional flush.

I need no machine readable output (up to now), I need a way to get flushed writeln output from the RTL on widows to display it in the MSEide message window (compiler output) and the target window (program output). On linux I use a PTY to connect compiler and target -> all is OK, on win32 I can only use pipes AFAIK where the RTL doesn't flush output on writeln.

Martin

But that doesn't matter if the message itself is not included for machine-readable output ?! The code uniquely defines the message, and you can look it up in any language file you want, no ?

That's the whole idea of machine-readable output ?

Michael.

Why not output XML, then ?

<fpc-output>
<error file='name.pas' line='xyz' col='nm' code='ABC' params="X,Y,Z">Message text</error> <warning file='name.pas' line='xyz' col='nm' code='ABC' params="X,Y,Z">warning text</warning> <info file='name.pas' line='xyz' col='nm' code='ABC' params="X,Y,Z">info text</info> </fpc-output>

It is extensible. Many parsing tools... Just a thought.

Michael.

I think that is indeed what needs to be done.

Vincent

error:(file='testc.pas';line=11;column=1;message_code='02003_F'; message='Syntax error, $1 expected but $2 found'; parameters=('BEGIN','IDENTIFIER Z'));

Or even an actual array of strings so you don't have to parse the semi colon delimiter - the semi colon delimiter will also have to be escaped if the error has a semi colon in it, i.e.

';' found but identifier expected

The problem exists not only for the compiler but for all console programs. I found no way to display the interactive output in a IDE window on win32. On Linux I can use a pseudo terminal to get flushed output from target. There are no pseudo terminals on win32, I must use pipes. The current 2.0.3 RTL does not flush output on writeln if the filehandle is a pipe (2.0.2 does!). So the IDE target console window is unusable on win32 with FPC 2.0.3 RTL and works OK with FPC 2.0.2 RTL.

text</error>

<warning file='name.pas' line='xyz' col='nm' code='ABC' params="X,Y,Z">warning

text</warning>

<info file='name.pas' line='xyz' col='nm' code='ABC' params="X,Y,Z">info text</info> </fpc-output> It is extensible. Many parsing tools... Just a thought.

Because an IDE must be fast fast fast - this is case where speed really is an issue (where in many other cases, speed is the last issue people should be worrying about) Maybe even a binary array format (I know people cringe about this).

Yes and no :) That's one of speed vs. usability decisions. Believe me, if flush pipes always, people will complain soon that FPC is too slow when writing to pipes.

Good idea.

Mattias

Lazarus parses the output, splits the message into several parts, let it parse by plugins and extend the filenames. This costs almost no time versus the work of the compiler and TListBox and the gnome-terminal. For example using an xterm instead speeds up the output noticeable. And yes, lazarus writes in chunks, not line by line.
Please no binary format. xml is ok. The first proposal was ok too.

Mattias

Op Sun, 14 May 2006, schreef Michael Van Canneyt:

XML requires an advanced parser and existing parsers will also produce complicated parser trees. (All that is needed is a sequence of key/value pairs, XML is a recursive tree). I'm not against it, but I have a light preference for something simple.

Daniël

I'd prefer something less verbose as XML. XML is text, but due to its verbosity (labeled fields) a bit less readable (and thus debugable) than alternative text forms.

I don't think FPC's errormsgs go over the minimal threshold to make XML worthwhile, and neither does it have a complex multi-vendor situation that would justify it.

For me, your or Florian's proposal is perfectly OK. I was just suggesting. After all, it's just 1 or 2 routines that need to be changed... I'm not really in favour of CSV style, but all this is a matter of taste... In the end, it doesn't really matter, as long as it is structured...

Michael.

Done (in trunk). Please, check whether it fixes
bug 4996 for you and let me know the results, so
I could possibly close that bug report.

Tomas

How do FPIDE and Lazarus solve the problem?

Op Mon, 15 May 2006, schreef Martin Schreiber:

The IDE retrieves the messages through the comphook unit.

Daniël

And the output from a console program as debuggee?

Op Mon, 15 May 2006, schreef Martin Schreiber:

You mean the Debug -> Output window?

The IDE reads the user screen contents and displays it inside the output window. Under Linux the output window is only functional under vcsa display, because otherwise you cannot read from the screen.

Daniël

Yes.

The IDE reads the user screen contents and displays it inside the output window. Under Linux the output window is only functional under vcsa display, because otherwise you cannot read from the screen.

Does it work on win32 with GDB?
Do you think it is possible without libgdb (GDB is connected via pipes)?

Probable I have to build my own PTY substitute for win32 with AllocConsole, AttachConsole and ConsoleScreenBuffers. As far as I know MS technologies this is a work of several weeks...
Arghh, I see AttachConsole does not work for win98 **%&*&**%!!!!!

Martin

Op Mon, 15 May 2006, schreef Martin Schreiber:

Using libgdb, the compiler can communicate with gdb without mixing program output and debugger output. I think this is also possible if you connect to gdb via pipes, since gdb can attach to already started programs.

Probable I have to build my own PTY substitute for win32 with AllocConsole, AttachConsole and ConsoleScreenBuffers. As far as I know MS technologies this is a work of several weeks... Arghh, I see AttachConsole does not work for win98 **%&*&**%!!!!!

Doesn't look that funny, indeed :(

Daniël