Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / lazarus / 2012-02

Run 3 extrenal program, pipeline

8 messages · 5 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

Unknown author

Hi, I need run 3 external program and redirected output: first | second | third

For example:
df -hT | grep -v -E 'tmpfs|ecryptfs' | grep 'sda2' it gives
/dev/sda2 ext4 19G 11G 7,6G 58% /

I use this http://wiki.lazarus.freepascal.org/Executing_External_Programs#How_to_redirect_output_with_TProcess sample code. It works fine for first | second.

For three programs I try edit it:

function threePipe(inputA, inputB, inputC: string): string;
var
 FirstProcess, SecondProcess, ThirdProcess: TProcess;
 Buffer: array[0..127] of char;
 ReadCount: Integer;
 ReadSize: Integer;
begin
 FirstProcess  := TProcess.Create(nil);
 SecondProcess := TProcess.Create(nil);
 ThirdProcess := TProcess.Create(nil);
 FirstProcess.Options  := [poUsePipes];
 SecondProcess.Options := [poUsePipes];
 ThirdProcess.Options  := [poUsePipes,poStderrToOutPut];
 FirstProcess.CommandLine  := inputA;
 SecondProcess.CommandLine := inputB;
 ThirdProcess.CommandLine := inputC;
 FirstProcess.Execute;
 SecondProcess.Execute;
 ThirdProcess.Execute;
 while FirstProcess.Running or (FirstProcess.Output.NumBytesAvailable > 0) do
 begin
   if FirstProcess.Output.NumBytesAvailable > 0 then
   begin
     ReadSize := FirstProcess.Output.NumBytesAvailable;
     if ReadSize > SizeOf(Buffer) then
       ReadSize := SizeOf(Buffer);
     ReadCount := FirstProcess.Output.Read(Buffer[0], ReadSize);
     SecondProcess.Input.Write(Buffer[0], ReadCount);
   end;
 end;
 SecondProcess.CloseInput;
 while SecondProcess.Running or (SecondProcess.Output.NumBytesAvailable > 0) do
 begin
   if SecondProcess.Output.NumBytesAvailable > 0 then
   begin
     ReadSize := SecondProcess.Output.NumBytesAvailable;
     if ReadSize > SizeOf(Buffer) then
       ReadSize := SizeOf(Buffer);
     ReadCount := SecondProcess.Output.Read(Buffer[0], ReadSize);
     ThirdProcess.Input.Write(Buffer[0], ReadCount);
   end;
 end;
 ThirdProcess.CloseInput;
 while ThirdProcess.Running do
   Sleep(1);
 ReadSize := ThirdProcess.Output.NumBytesAvailable;
 if ReadSize > SizeOf(Buffer) then
   ReadSize := SizeOf(Buffer);
 if ReadSize > 0 then
 begin
   ReadCount := ThirdProcess.Output.Read(Buffer, ReadSize);
   //WriteLn(Copy(Buffer,0, ReadCount));
 end
 else
   WriteLn('grep did not find what we searched for. ', ThirdProcess.ExitStatus);
 FirstProcess.Free;
 SecondProcess.Free;
 ThirdProcess.Free;
 threePipe := Copy(Buffer,0, ReadCount);
end;

But after i call

   Label1.Caption:=threePipe('df -hT', 'grep -v -E ''tmpfs|ecryptfs''', 'grep ''sda2''');

function my application end with infinite loop.

Where I do an error?

Reply by email

Terry A. Haimann
to Unknown author

All you should have to do is have each program read from stdin & write to stdout. I've done it before. There is no reason to write it in Lazarus at all, just use FreePascal. If you are trying to execute this all from a tprocess, be aware that this in't bash. I would put it in a script and try it there.

Show quoted text 95 lines
Hi, I need run 3 external program and redirected output: first | second | third For example: df -hT | grep -v -E 'tmpfs|ecryptfs' | grep 'sda2' it gives /dev/sda2 ext4 19G 11G 7,6G 58% / I use this http://wiki.lazarus.freepascal.org/Executing_External_Programs#How_to_redirect_output_with_TProcess sample code. It works fine for first | second. For three programs I try edit it: function threePipe(inputA, inputB, inputC: string): string; var FirstProcess, SecondProcess, ThirdProcess: TProcess; Buffer: array[0..127] of char; ReadCount: Integer; ReadSize: Integer; begin FirstProcess := TProcess.Create(nil); SecondProcess := TProcess.Create(nil); ThirdProcess := TProcess.Create(nil); FirstProcess.Options := [poUsePipes]; SecondProcess.Options := [poUsePipes]; ThirdProcess.Options := [poUsePipes,poStderrToOutPut]; FirstProcess.CommandLine := inputA; SecondProcess.CommandLine := inputB; ThirdProcess.CommandLine := inputC; FirstProcess.Execute; SecondProcess.Execute; ThirdProcess.Execute; while FirstProcess.Running or (FirstProcess.Output.NumBytesAvailable> 0) do begin if FirstProcess.Output.NumBytesAvailable> 0 then begin ReadSize := FirstProcess.Output.NumBytesAvailable; if ReadSize> SizeOf(Buffer) then ReadSize := SizeOf(Buffer); ReadCount := FirstProcess.Output.Read(Buffer[0], ReadSize); SecondProcess.Input.Write(Buffer[0], ReadCount); end; end; SecondProcess.CloseInput; while SecondProcess.Running or (SecondProcess.Output.NumBytesAvailable> 0) do begin if SecondProcess.Output.NumBytesAvailable> 0 then begin ReadSize := SecondProcess.Output.NumBytesAvailable; if ReadSize> SizeOf(Buffer) then ReadSize := SizeOf(Buffer); ReadCount := SecondProcess.Output.Read(Buffer[0], ReadSize); ThirdProcess.Input.Write(Buffer[0], ReadCount); end; end; ThirdProcess.CloseInput; while ThirdProcess.Running do Sleep(1); ReadSize := ThirdProcess.Output.NumBytesAvailable; if ReadSize> SizeOf(Buffer) then ReadSize := SizeOf(Buffer); if ReadSize> 0 then begin ReadCount := ThirdProcess.Output.Read(Buffer, ReadSize); //WriteLn(Copy(Buffer,0, ReadCount)); end else WriteLn('grep did not find what we searched for. ', ThirdProcess.ExitStatus); FirstProcess.Free; SecondProcess.Free; ThirdProcess.Free; threePipe := Copy(Buffer,0, ReadCount); end; But after i call Label1.Caption:=threePipe('df -hT', 'grep -v -E ''tmpfs|ecryptfs''', 'grep ''sda2'''); function my application end with infinite loop. Where I do an error? -- _______________________________________________ Lazarus mailing list http://lists.lazarus.freepascal.org/mailman/listinfo/lazarus

Reply by email

Sven Barth
to Unknown author: Hi, I need run 3 external program and redirected output: first | second | third For…
Show quoted text 88 lines
Hi, I need run 3 external program and redirected output: first | second | third For example: df -hT | grep -v -E 'tmpfs|ecryptfs' | grep 'sda2' it gives /dev/sda2 ext4 19G 11G 7,6G 58% / I use this http://wiki.lazarus.freepascal.org/Executing_External_Programs#How_to_redirect_output_with_TProcess sample code. It works fine for first | second. For three programs I try edit it: function threePipe(inputA, inputB, inputC: string): string; var FirstProcess, SecondProcess, ThirdProcess: TProcess; Buffer: array[0..127] of char; ReadCount: Integer; ReadSize: Integer; begin FirstProcess := TProcess.Create(nil); SecondProcess := TProcess.Create(nil); ThirdProcess := TProcess.Create(nil); FirstProcess.Options := [poUsePipes]; SecondProcess.Options := [poUsePipes]; ThirdProcess.Options := [poUsePipes,poStderrToOutPut]; FirstProcess.CommandLine := inputA; SecondProcess.CommandLine := inputB; ThirdProcess.CommandLine := inputC; FirstProcess.Execute; SecondProcess.Execute; ThirdProcess.Execute; while FirstProcess.Running or (FirstProcess.Output.NumBytesAvailable> 0) do begin if FirstProcess.Output.NumBytesAvailable> 0 then begin ReadSize := FirstProcess.Output.NumBytesAvailable; if ReadSize> SizeOf(Buffer) then ReadSize := SizeOf(Buffer); ReadCount := FirstProcess.Output.Read(Buffer[0], ReadSize); SecondProcess.Input.Write(Buffer[0], ReadCount); end; end; SecondProcess.CloseInput; while SecondProcess.Running or (SecondProcess.Output.NumBytesAvailable> 0) do begin if SecondProcess.Output.NumBytesAvailable> 0 then begin ReadSize := SecondProcess.Output.NumBytesAvailable; if ReadSize> SizeOf(Buffer) then ReadSize := SizeOf(Buffer); ReadCount := SecondProcess.Output.Read(Buffer[0], ReadSize); ThirdProcess.Input.Write(Buffer[0], ReadCount); end; end; ThirdProcess.CloseInput; while ThirdProcess.Running do Sleep(1); ReadSize := ThirdProcess.Output.NumBytesAvailable; if ReadSize> SizeOf(Buffer) then ReadSize := SizeOf(Buffer); if ReadSize> 0 then begin ReadCount := ThirdProcess.Output.Read(Buffer, ReadSize); //WriteLn(Copy(Buffer,0, ReadCount)); end else WriteLn('grep did not find what we searched for. ', ThirdProcess.ExitStatus); FirstProcess.Free; SecondProcess.Free; ThirdProcess.Free; threePipe := Copy(Buffer,0, ReadCount); end; But after i call Label1.Caption:=threePipe('df -hT', 'grep -v -E ''tmpfs|ecryptfs''', 'grep ''sda2'''); function my application end with infinite loop. Where I do an error?

You need to read from all three processes in one loop as they will block if they've reached the limit of the output pipe. Try something like this (pseudo code):

while FirstProcess.Running or SecondProcess.Running or
ThirdProcess.Running do begin
   FromFirstToSecond;
   FromSecondToThird;
   FromThirdToOutput;
end;
FromThirdToOutput; // <= you need to process the leftover output from
the process as well

It might be more complex than the above, but in essence this should do it (maybe you'll need to play with the buffer sizes, etc).

Regards,
Sven

Reply by email

Maxi
to Unknown author: Hi, I need run 3 external program and redirected output: first | second | third For…

In this case is only 1 process and 1 result so no need to call three separate processes
Executes a single TProcess with "df-hT | grep-v-E 'tmpfs | ecryptfs' | grep 'sda2' "

El que pregunta aprende, y el que contesta aprende a responder. No a la obsolecencia programada: http://dimensionargentina.blogspot.com/2011/01/todo-esta-disenado-para-romperse.html Linux User #495070 http://miexperiencialinux.blogspot.com/

Reply by email

Sven Barth
to Maxi
Maxi wrote 6 lines
In this case is only 1 process and 1 result so no need to call three separate processes Executes a single TProcess with "df-hT | grep-v-E 'tmpfs | ecryptfs' | grep 'sda2' "

This won't work as TProcess uses "fork"/"exec" to execute the new process and not "system". But because of the pipes the "system" call would be needed. Thus the manual solution.

Regards,
Sven

Reply by email

Lukasz Sokol
to Unknown author: Hi, I need run 3 external program and redirected output: first | second | third For…
Show quoted text 35 lines
Hi, I need run 3 external program and redirected output: first | second | third For example: df -hT | grep -v -E 'tmpfs|ecryptfs' | grep 'sda2' it gives /dev/sda2 ext4 19G 11G 7,6G 58% / I use this http://wiki.lazarus.freepascal.org/Executing_External_Programs#How_to_redirect_output_with_TProcess sample code. It works fine for first | second. For three programs I try edit it: function threePipe(inputA, inputB, inputC: string): string; var FirstProcess, SecondProcess, ThirdProcess: TProcess; Buffer: array[0..127] of char; ReadCount: Integer; ReadSize: Integer; begin FirstProcess := TProcess.Create(nil); SecondProcess := TProcess.Create(nil); ThirdProcess := TProcess.Create(nil); FirstProcess.Options := [poUsePipes]; SecondProcess.Options := [poUsePipes]; ThirdProcess.Options := [poUsePipes,poStderrToOutPut]; FirstProcess.CommandLine := inputA; SecondProcess.CommandLine := inputB; ThirdProcess.CommandLine := inputC; FirstProcess.Execute; SecondProcess.Execute; ThirdProcess.Execute;
while (Firstprocess.Running or SecondProcess.Running or ThirdProcess.Running) do
begin
  if Firstprocess.Running then
    begin
      if (FirstProcess.Output.NumbytesAvailable > 0) then
        begin
          ReadSize := FirstProcess.Output.NumBytesAvailable;
          if ReadSize > SizeOf(Buffer) then
            ReadSize := SizeOf(Buffer);
          ReadCount := FirstProcess.Output.Read(Buffer[0], ReadSize);
          SecondProcess.Input.Write(Buffer[0], ReadCount);
          // at the end of this, we have written max. 128 bytes to SecondProcess' input
        end;  // if no more bytes, we could terminate the FirstProcess but having it here
              // may race when the lot starts: when FirstProcess just started but has no output yet
              // or if for some reason it stops on the way;
              // some checking for first process's errors may be a good idea too, here.
    end
  else
    SecondProcess.CloseInput;  // when first process ends, input to second also ends
  if SecondProcess.Running then
    begin
      if (SecondProcess.Output.NumBytesAvailable > 0) then
        begin
          ReadSize := SecondProcess.Output.NumBytesAvailable;
          if ReadSize > SizeOf(Buffer) then
            ReadSize := SizeOf(Buffer);
          ReadCount := SecondProcess.Output.Read(Buffer[0], ReadSize);
          ThirdProcess.Input.Write(Buffer[0], ReadCount);
        end;
    end
  else
    ThirdProcess.CloseInput;
  if ThirdProcess.Running then
    begin
      if (ThirdProcess.Output.NumbytesAvailable >0) then
        begin
          ReadSize := ThirdProcess.Output.NumBytesAvailable;
          if ReadSize > SizeOf(Buffer) then
            ReadSize := SizeOf(Buffer);
          if ReadSize > 0 then
            begin
              ReadCount := ThirdProcess.Output.Read(Buffer, ReadSize);
              Write(Copy(Buffer,0, ReadCount));  // i think no writeln, but write (grep gives you ^M on each line IIRC)
            end;
        end;
    end;
end;
Show quoted text 40 lines
while FirstProcess.Running or (FirstProcess.Output.NumBytesAvailable > 0) do begin if FirstProcess.Output.NumBytesAvailable > 0 then begin ReadSize := FirstProcess.Output.NumBytesAvailable; if ReadSize > SizeOf(Buffer) then ReadSize := SizeOf(Buffer); ReadCount := FirstProcess.Output.Read(Buffer[0], ReadSize); SecondProcess.Input.Write(Buffer[0], ReadCount); end; end; SecondProcess.CloseInput; while SecondProcess.Running or (SecondProcess.Output.NumBytesAvailable > 0) do begin if SecondProcess.Output.NumBytesAvailable > 0 then begin ReadSize := SecondProcess.Output.NumBytesAvailable; if ReadSize > SizeOf(Buffer) then ReadSize := SizeOf(Buffer); ReadCount := SecondProcess.Output.Read(Buffer[0], ReadSize); ThirdProcess.Input.Write(Buffer[0], ReadCount); end; end; ThirdProcess.CloseInput; while ThirdProcess.Running do Sleep(1); ReadSize := ThirdProcess.Output.NumBytesAvailable; if ReadSize > SizeOf(Buffer) then ReadSize := SizeOf(Buffer); if ReadSize > 0 then begin ReadCount := ThirdProcess.Output.Read(Buffer, ReadSize); //WriteLn(Copy(Buffer,0, ReadCount)); end else WriteLn('grep did not find what we searched for. ', ThirdProcess.ExitStatus);
//  here;
Show quoted text 14 lines
FirstProcess.Free; SecondProcess.Free; ThirdProcess.Free; threePipe := Copy(Buffer,0, ReadCount); // and this will be nonsense - better have another variable accounting all bytes out from third process end; But after i call Label1.Caption:=threePipe('df -hT', 'grep -v -E ''tmpfs|ecryptfs''', 'grep ''sda2'''); function my application end with infinite loop. Where I do an error?

Thought compiled only.

But:
- uses same buffer to pass between stages, adjusts buffer to size between reads, does not stop to read the whole buffer output from first process or second process, just passes buffer as it comes

Reply by email

Lukasz Sokol
to Lukasz Sokol
Lukasz Sokol wrote 144 lines
On 02/02/2012 11:00, wrote: > Hi, I need run 3 external program and redirected output: first | second | third > > For example: > df -hT | grep -v -E 'tmpfs|ecryptfs' | grep 'sda2' > it gives > /dev/sda2 ext4 19G 11G 7,6G 58% / > > I use this http://wiki.lazarus.freepascal.org/Executing_External_Programs#How_to_redirect_output_with_TProcess sample code. It works fine for first | second. > > For three programs I try edit it: > > > function threePipe(inputA, inputB, inputC: string): string; > var > FirstProcess, SecondProcess, ThirdProcess: TProcess; > Buffer: array[0..127] of char; > ReadCount: Integer; > ReadSize: Integer; > begin > FirstProcess := TProcess.Create(nil); > SecondProcess := TProcess.Create(nil); > ThirdProcess := TProcess.Create(nil); > > FirstProcess.Options := [poUsePipes]; > SecondProcess.Options := [poUsePipes]; > ThirdProcess.Options := [poUsePipes,poStderrToOutPut]; > > FirstProcess.CommandLine := inputA; > SecondProcess.CommandLine := inputB; > ThirdProcess.CommandLine := inputC; > > FirstProcess.Execute; > SecondProcess.Execute; > ThirdProcess.Execute; > while (Firstprocess.Running or SecondProcess.Running or ThirdProcess.Running) do begin if Firstprocess.Running then begin if (FirstProcess.Output.NumbytesAvailable > 0) then begin ReadSize := FirstProcess.Output.NumBytesAvailable; if ReadSize > SizeOf(Buffer) then ReadSize := SizeOf(Buffer); ReadCount := FirstProcess.Output.Read(Buffer[0], ReadSize); SecondProcess.Input.Write(Buffer[0], ReadCount); // at the end of this, we have written max. 128 bytes to SecondProcess' input end; // if no more bytes, we could terminate the FirstProcess but having it here // may race when the lot starts: when FirstProcess just started but has no output yet // or if for some reason it stops on the way; // some checking for first process's errors may be a good idea too, here. end else SecondProcess.CloseInput; // when first process ends, input to second also ends if SecondProcess.Running then begin if (SecondProcess.Output.NumBytesAvailable > 0) then begin ReadSize := SecondProcess.Output.NumBytesAvailable; if ReadSize > SizeOf(Buffer) then ReadSize := SizeOf(Buffer); ReadCount := SecondProcess.Output.Read(Buffer[0], ReadSize); ThirdProcess.Input.Write(Buffer[0], ReadCount); end; end else ThirdProcess.CloseInput; if ThirdProcess.Running then begin if (ThirdProcess.Output.NumbytesAvailable >0) then begin ReadSize := ThirdProcess.Output.NumBytesAvailable; if ReadSize > SizeOf(Buffer) then ReadSize := SizeOf(Buffer); if ReadSize > 0 then begin ReadCount := ThirdProcess.Output.Read(Buffer, ReadSize); Write(Copy(Buffer,0, ReadCount)); // i think no writeln, but write (grep gives you ^M on each line IIRC) end; end; end; end; > while FirstProcess.Running or (FirstProcess.Output.NumBytesAvailable > 0) do > begin > if FirstProcess.Output.NumBytesAvailable > 0 then > begin > ReadSize := FirstProcess.Output.NumBytesAvailable; > if ReadSize > SizeOf(Buffer) then > ReadSize := SizeOf(Buffer); > ReadCount := FirstProcess.Output.Read(Buffer[0], ReadSize); > SecondProcess.Input.Write(Buffer[0], ReadCount); > end; > end; > SecondProcess.CloseInput; > > while SecondProcess.Running or (SecondProcess.Output.NumBytesAvailable > 0) do > begin > if SecondProcess.Output.NumBytesAvailable > 0 then > begin > ReadSize := SecondProcess.Output.NumBytesAvailable; > if ReadSize > SizeOf(Buffer) then > ReadSize := SizeOf(Buffer); > ReadCount := SecondProcess.Output.Read(Buffer[0], ReadSize); > ThirdProcess.Input.Write(Buffer[0], ReadCount); > end; > end; > ThirdProcess.CloseInput; > > > while ThirdProcess.Running do > Sleep(1); > ReadSize := ThirdProcess.Output.NumBytesAvailable; > if ReadSize > SizeOf(Buffer) then > ReadSize := SizeOf(Buffer); > if ReadSize > 0 then > begin > ReadCount := ThirdProcess.Output.Read(Buffer, ReadSize); > //WriteLn(Copy(Buffer,0, ReadCount)); > end > else > WriteLn('grep did not find what we searched for. ', ThirdProcess.ExitStatus); > // here; > FirstProcess.Free; > SecondProcess.Free; > ThirdProcess.Free; > threePipe := Copy(Buffer,0, ReadCount); // and this will be nonsense - better have another variable accounting all bytes out from third process > end; > > > > But after i call > Label1.Caption:=threePipe('df -hT', 'grep -v -E ''tmpfs|ecryptfs''', 'grep ''sda2'''); > function my application end with infinite loop. > > Where I do an error? >

Sorry, I hit 'send' button before completing my thoughts.

With program as you wrote it, with your example chain, you wait for df to complete, and then write its complete output to grep; This df output can be quite large.

Show quoted text 8 lines
Thought compiled only. Look what I think you need to do: - use same buffer to pass between stages, does not stop to read the whole buffer output from first process or second process, just passes buffer as it comes

Possible improvement of what I wrote : abstract the chunk where we pass chunks of output of one process to another process' input, possibly merge it with the writing to stdout case too (selectable/overload?), let the whole procedure create as many tprocesses as need be, pass into it a TStringList containing one command per string, the associated Object can hold the TProcess reference, result : you can dynamically create pipelines as long as you need...

L.

Reply by email

Sven Barth
to Unknown author: Hi, I need run 3 external program and redirected output: first | second | third For…
Show quoted text 8 lines
Hi, I need run 3 external program and redirected output: first | second | third For example: df -hT | grep -v -E 'tmpfs|ecryptfs' | grep 'sda2' it gives /dev/sda2 ext4 19G 11G 7,6G 58% / I use this http://wiki.lazarus.freepascal.org/Executing_External_Programs#How_to_redirect_output_with_TProcess sample code. It works fine for first | second.

Completely different solution: Why do you need to run three processes at all? Just get the output of "df -hT" and then search the corresponding strings using a for loop...

e.g.

function GetDFOfSda2: String;
var
   slist: TStringList;
   s: String;
begin
   slist := TStringList.Create;
   try
     GetOutputOfProcess('df', ['-hT'], slist); // hypothetic function
that returns the output of a process in a TStringList
     for s in slist do begin
       if (Pos('tmpfs', s) = 0) and (Pos('ecryptfs', s) = 0) then
         Continue;
       if Pos('sda2', s) > 0 then
         Exit(s);
     end;
   finally
     slist.Free;
   end;
end.

Of course this can be improved by using regular expressions and setting the drive to look for through a parameter, but it should show you the basic concept. :)

Regards,
Sven

Reply by email