Hello all,

I have been experimenting with redirection of stan- dard I/O of child processes using FreePascal 2.4.2 on Windows. I have succeeded in capturing standard input and standard output, but failed to feed my own data to the child's standard input.

The child process doesn't seem to receive any input at all, staying blocked forever.

At first I thought it was a problem with my usage of WinApi, so I rewrote my program using the TProcess class... just to get the same result. The program feeds 'Anton' to the more.com program and captures part of its output:

Program StrRedir;
uses Classes, Process, Sysutils;
const MaxByte = 255;
type
      TStrBuf = packed record {As a way to read buffers into strings}
      case Boolean of
         true: (  size: Byte;
                  buf:  array[0..MaxByte] of Char;
               );
         false:(  txt:  ShortString;  );
      end;
var
   MoreProcess: TProcess;
   readCount:   integer;
   strBuf:      TStrBuf;
begin
   MoreProcess := TProcess.Create(nil);
   MoreProcess.CommandLine := 'C:\WINDOWS\system32\cmd.exe /C more';
   MoreProcess.Options := [poUsePipes];
   MoreProcess.Execute;
   strBuf.txt := 'Anton';
   MoreProcess.Input.Write(strBuf.buf, strBuf.size);
   MoreProcess.CloseInput();
   writeLn('Waiting...');    //This never ends
   while MoreProcess.Running do begin Sleep(50); end;
   writeLn('Wait finished.');
   Sleep(100);
   strBuf.size := MoreProcess.Output.Read(strBuf.buf, 255);
   writeLn(strBuf.txt);
end.

Could you please help me to make it work?

Thanks in advance,
Anton

Anton - I have attached an extract from some working code (hopefully I didnt remove anything necessary)
It may help you move on a bit further

SteveG

SteveG:

Anton - I have attached an extract from some work- ing code (hopefully I didnt remove anything neces- sary) It may help you move on a bit further

I turned your piece of code into a complete program, but it didn't work either. In fact, there is not much difference between your program and mine, except that in generality.

But I am creating a process with redirected standard handles (poUsePipes) and am sending a tiny five-byte input to it, and expecitng a tiny output, so there's no need in the loop and in the various checks that your program has.

Also, your program does not call CloseInput(), which makes me wonder how the process can ever end, because that way it is not going to get a EOF or error 109 ("Pipe has been ended") on its standard input.

What OS is the code you sent me working on?

Anton

This is working on Linux and WinXP
I cut what I was hoping was just the relevant code from the unit, so it is missing a bit :)

In our previous episode, Anton Shepelev said:

MoreProcess.CommandLine := 'C:\WINDOWS\system32\cmd.exe /C more';

Why do you execute over the shell? You now pipe your information into the shell (since that is what you execute directly), not the program.

SteveG:

This is working on Linux and WinXP I cut what I was hoping was just the relevant code from the unit, so it is missing a bit :)

Here's a full program made from this code:

http://pastebin.com/id90J1rY

It doesn't work for me in WinXP. What about you?

Anton

Marco van de Voort:

Why do you execute over the shell? You now pipe your information into the shell (since that is what you execute directly), not the program.

Thanks for the note, Marco. Unfortunately, replacing that line with the direct path to more.com had no effect.

When executing command-line programs via the shell, isn't the shell's input redirected to the program's? I have verified that the program's output is redi- rected to the shell's.

Anton

In our previous episode, Anton Shepelev said:

When executing command-line programs via the shell, isn't the shell's input redirected to the program's?

Maybe. I'm pretty sure about batchfiles, but never tried programs, that's why I thought I'd mention the observation.

Marco van de Voort:

My program captures both the standard and error out- put of batchfiles and progams called therefrom just the same way as with regular programs run using the shell.

So you don't see any evident errors?

Anton

In our previous episode, Anton Shepelev said:

I played a bit with it, but was unable to make it work. I added a writeln of outstream.size to the "running" while, and it seems output is written.

It seems the program somehow doesn't get the signal to close. (since I can see more.com is still running in the taskmgr)

Marco van de Voort:

I played a bit with it, but was unable to make it work. I added a writeln of outstream.size to the "running" while, and it seems output is written.

Many thanks :)

If I understood you correctly, you put

    WriteLn(MoreProcess.Output.Size);

inside the waiting loop and it printeda nonzero value. I did the same, but it said zero. Hmmm..... I tried actually read- ing the output inside the loop and the Read() function blocked, as expected when there's nothing to read.

It seems the program somehow doesn't get the signal to close. (since I can see more.com is still running in the taskmgr)

My hypothesis is that the more.com process is blocked at reading standard output, which it doesn't seem to get for some reason. This problem persists with any application expecting standard input, including my own test app.

My next step will be to rewrite my original program, that uses plain WinAPI, in C and see what it yields.

Anton

If you run 'more' in a cmd window you'll notice that 'more' echoes the input but only sends to stdout when a return is entered. I modified the program to send 'Anton'#10 and the program reads back 'Anton'#10 from stdout.

Ludo

Correction: 'Anton'#13#10 is read from stdout. More adds the CR. Windows style EOL ...

Ludo

Ludo Brands:

Thank you, Ludo, this is great!

But why doesn't the process finish? I thought that the clos- ing of the write end of the StdIn pipe handle should cause a EOF to be 'seen' at the read end, but this does not happen.

Maybe these's some explicit way to signal end of input?

Anton

Ludo Brands:

Connected with this remark, is there a way to get the under- lying StdIn handle in a FreePascal program, so as to read from it per-character, or any other way using the OS's rou- tines?

Anton

Anton,

I don't see where you close the mProcess.input.
I tried several things myself which didn't work:
- ctrl Z Return is an eof when you run more in a cmd window.
- call explicitly CloseHandle(mProcess.Input.Handle);

Pipes.pp doesn't have any routines to clean up the pipe handles which is according to msdn not good. Pipes close when one of the 2 processes stops but in larger applications this could lead to memory leaks.

Still looking further into it.

Ludo

The handles are mProcess.Input.Handle and mProcess.Output.Handle

Ludo

how about sending the ^Z line the CRLF is being sent to indicate line breaks?? that should indicate the EOF which should trigger the ending of the process and closing of the pipe... shouldn't it?

That's what I tried already. It simply echoes the ^Z (right arrow). I tried also to close the pipe explicitly from the sender side without success:
mProcess.CloseInput;
CloseHandle(mProcess.Input.Handle);

I'm afraid we're looking at a problem with more.com waiting until the process releases the pipe. Closing the handle from one side of the pipe isn't enough to close the pipe. Or as
http://msdn.microsoft.com/en-us/library/aa365152%28v=vs.85%29.aspx says: "An instance of a pipe is always deleted when the last handle to the instance of the named pipe is closed".

Ludo

Anton,

According to msdn
http://msdn.microsoft.com/en-us/library/aa365782%28v=vs.85%29.aspx the pipe handles should be created non-inheritable in order to detect EOF. Looking at fcl-process/src/win/pipes.inc, pipes are created inheritable... The msdn sample code to redirect input and output also creates 2 pipes (http://msdn.microsoft.com/en-us/library/ms682499%28v=vs.85%29.aspx).

Ludo

waldo kitty:

how about sending the ^Z line the CRLF is being sent to indicate line breaks??

This I had tried even before posting the question :)

Ludo Brands:

I don't see where you close the mProcess.input.

In my original post I call:

    MoreProcess.CloseInput();

As for the program based on SteveG's code, I tried to keep his code as intact as possible, so I didn't close the input handle in it.

Pipes.pp doesn't have any routines to clean up the pipe handles which is according to msdn not good.

You mean reading from the pipes after the closing of the child process?

According to msdn... the pipe handles should be created non-inheritable in order to detect EOF.

Oh, I missed this. My WinApi program creates pipes with:

    SECURITY_ATTRIBUTES.bInheritHandle = true;

in order for the handles to be properly inherited in the child process. That must be my problem. The only other way to do it is to use the DuplicateHandle function which will create in inherarable duplicate of the original handle, that can then be passed to the CreateProcess. The original handle will remain non-inheritable and should trigger EOF on clo- sure.

Thank you for your investigation. I'll try duplicating the handles and report the results.

Anton

Ludo Brands:

I meant the program's own handles. Like instead of doing:

    Read(input, c);

read directly from a handle:

    Windows.ReadFile(inputHandle, ...);

Anton

I'll try duplicating the handles and report the results.

Yes. It works now. I am creating a non-inheritable handle using CreatePipe, then duplicating it to an inheritable one using DuplicateHandle and pass the duplicate to the CreateProcess funtion.

Does this mean that the Windows implementation of TProcess should be rewritten using DuplicateHandle?

Thanks everybody for your help.
Anton

i wasn't sure and the posts didn't seem to indicate this so i thought i'd ask ;)

now i'll go back to reading and see what you guys come up with for the solution :)

Pipe handles can be used directly with readfile:
Var BytesRead:Longword;
windows.Readfile(mProcess.Output.Handle,str1[1],BytesAvailable,BytesRead,nil
);
can replace
        mProcess.OutPut.Read(str1[1], BytesAvailable);

Ludo

Yes. It works now.

Great.

Does this mean that the Windows implementation of TProcess should be

rewritten

using DuplicateHandle?

You can file a bug at http://bugs.freepascal.org/ . Dupicating handles isn't apparently the only solution. The msdn sample creates non-inheritable pipes and hands these handles simply over to createprocess, without duplicating. You want to pass inheritable handles if you want the child to have full control of the pipe, which apparently isn't needed for RW. Note also that in most applications where you are in control (the development I mean) of both sides of the pipe, you would use more robust process control mechanisms than waiting for EOF.

If you need these handles from the child process use:
InHnd:=GetStdHandle(STD_INPUT_HANDLE);
OutHnd:=GetStdHandle(STD_OUTPUT_HANDLE);
ErrHnd:=GetStdHandle(STD_ERROR_HANDLE);

Ludo

Ludo Brands:

You can file a bug at http://bugs.freepascal.org/

OK.

Dupicating handles isn't apparently the only solution. The msdn sample creates non-inheritable pipes and hands these handles simply over to createprocess, without duplicating.

There are two solutions, but only of them allows detection of EOF.

On MSDN there are two examples. In this one:

http://msdn.microsoft.com/en-us/library/ms682499(v=vs.85).aspx

inheritable pipe handles are created using CreatePipe, just as I was doing in my initial WinApi code and as TProcess is doing. These pipes don't support the detection of EOF.

In this one:

http://support.microsoft.com/kb/190351/en-us?fr=1

non-inheritable pipes are created by CreatePipe, which are then duplicated into inheritable ones using DuplicateHandle. The duplicates are passed to CreateProcess, while the non- inheritable originals are used by the parent process, allow- ing it to issue EOF by closing them.

This is what made my WinApi code work with more.com.

You want to pass inheritable handles if you want the child to have full control of the pipe, which apparently isn't needed for RW.

A handle must be inheritable in order to be passed from one process to another. There are two ways to make it so:

1. Specify a flag in CreateProcess

2. Duplicate a non-inheritable handle using DuplicateHandle

The first MSDN example goes the former way and second the latter way.

Anton

Ludo Brands:

If you need these handles from the child process use: InHnd:=GetStdHandle(STD_INPUT_HANDLE); OutHnd:=GetStdHandle(STD_OUTPUT_HANDLE); ErrHnd:=GetStdHandle(STD_ERROR_HANDLE);

Thank you, that's what I need.

By the way, it does work in a simple single-process applica- tion:

    Program Test;
    uses Windows;
    var
       hIn:       LongWord;
       buf:       ShortString;
       bytesRead: LongWord;
    begin
       hIn := GetStdHandle(STD_INPUT_HANDLE);
       while(true) do begin
          ReadFile(hIn, buf[1], 5, bytesRead, nil);
          buf[0] := Chr(bytesRead);
          write(buf);
       end;
    end.

It is interesting to note that even this code reads input only after Enter is pressed.

Anton

It is interesting to note that even this code reads input only after

Enter is

pressed.

Sounds logical. Stdin from console supports line editing (BackSpace, arrows, F3, F1, ...). This can only be accomplished by buffering complete lines.

Ludo

Look at SetConsoleMode:

http://msdn.microsoft.com/en-us/library/ms686033%28v=VS.85%29.aspx

You should be able to read single characters if you disable the ENABLE_LINE_INPUT mode.

Henry

A little update on the subject.

After fixing an error in my WinApi program, it is no longer needed to terminate input with #10. Now that the write-end of the child process's input handle is correctly closed, more.com echoes whatever has been written to it requiring no additional terminal symbols.

Anton

Ludo Brands:

You can file a bug at http://bugs.freepascal.org/.

This is bug #0019325.

...
I thought about writing a patch, but it seems that it is not enough to modify the implementation of the Windows-specific pipes.inc.

The correct way to create pipes for the three channels (out, in and err) on Windows is to create non-inheritable handles and duplicate only the child process's ends of them into inheritable ones, which means that the CreatePipeHandles function must accept a third parameter indicating which end should be inherited: for OUT and ERR channels it will be the write-end and for IN -- the read-end.

Here is a possible implementation:

{--------------- changes to pipes.inc -----------------}
Interface
   {...}
   type
      TDupOpt  = (dupIn, dupOut);
      TDupOpts = Set of TDupOpt;
Implemenation
   {...}
   function DuplicateHandleFP(var handle: THandle): Boolean;
   var
      oldHandle: THandle;
   begin
      oldHandle := handle;
      Result := DuplicateHandle
      (  GetCurrentProcess(),
         oldHandle,
         GetCurrentProcess(),
         @handle,
         0,
         true,
         DUPLICATE_SAME_ACCESS
      );
      if Result then
         Result := CloseHandle(oldHandle);
   end;
   Function CreatePipeHandles
   (Var Inhandle,OutHandle : THandle; DupOpts: TDupOpts) : Boolean;
   begin
      Result := CreatePipe (Inhandle,OutHandle,@piNonInheritablePipe,0);
      if Result and (dupIn in DupOpts) then
         Result := DuplicateHandleFP(Inhandle);
      if Result and (dupOut in DupOpts) then
         Result := DuplicateHandleFP(OutHandle);
   end;
   {----------- process.inc -------------------------}
   {CreatePipes will have these calls:}
   begin
     CreatePipeHandles(SI.hStdInput ,HI, [dupIn ]);
     CreatePipeHandles(HO,Si.hStdOutput, [dupOut]);
     if CE then
       CreatePipeHandles(HE,SI.hStdError, [dupOut])
   {...}

Unfortunately, the third parameter will have to be added to this function in all other platform-specific pipes.inc files, but these will just ignore it and work as before.

If it is preferable to keep the interface of the pipes unit unchanged, the windows-specific part could just be chaged to create non-inheritable pipes, and the duplication could be moved into the windows-specific process.inc.

Will you accept one of such patches?

Anton

Yes.

The interface of the pipes unit can be changed with a parameter with a default value, so existing code continues to work.

Michael.

Michael Van Canneyt:

The interface of the pipes unit can be changed with a parameter with a default value, so existing code continues to work.

Yes, but all the platform-specific pipe.inc files and pipes.pp will have to be changed anyway because the CreatePipeHandles will have a different proto- type. I wasn't sure if it would be OK to add a parameter to the definition of a cross-platform function that would be used only on one one platform and ignored on all the other ones.

I will write a patch then.

Anton

Here's the patch to the Windows-specific pipes.inc and process.inc.

In order to keep the unit interfaces unchanged I am dupli- cating pipe handles inside process.inc. Otherwise, the interface of the Pipes unit would have had to be changed.

Also included is a test program that didn't work (never ter- minated) and works.

Please, see if it is acceptable.

Anton

Sorry, I forgot to say the patch is against verstion 2.4.2.

Anton