Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / fpc-devel / 2010-02

TProcess, bash, and escaping quotes

9 messages · 4 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

Seth Grover

I'm working with TProcess, which for the most part I love, but I'm scratching my head right now because of a problem I'm experiencing.

Suppose I have the following command:

ifconfig | grep -E -e "inet6? addr:" | grep -v -E -e "(127.0.0.1|::1|Scope:Link)" | sed -e
"s/^.*addr:\s*\(\S*\)\(\/\d+\)*.*$/\1/g"

I can run that as-is from the command-line, since my shell hooks up all of the pipes and whatnot for me. Also, I can execute it like this:

bash -c "ifconfig | grep -E -e \"inet6? addr:\" | grep -v -E -e \"(127.0.0.1|::1|Scope:Link)\" | sed -e
\"s/^.*addr:\s*\(\S*\)\(\/\d+\)*.*$/\1/g\""

By escaping the double-quotes and then passing the whole thing to bash as the argument for the -c, it works.

However, if I pass the above command (the one using bash -c with the double-quotes of the original command escaped), as the CommandLine of a TProcess, it doesn't work. TProcess execs bash, but the argument doesn't seem to make it there correctly. Bash spits out:

$ ./tprocquotes
addr:\" | grep -v -E -e \"(127.0.0.1|::1|Scope:Link)\" | sed -e \"s/^.*addr:\s*\(\S*\)\(\/\d+\)*.*$/\1/g\"": -c: line 0: unexpected EOF while looking for matching `"'
addr:\" | grep -v -E -e \"(127.0.0.1|::1|Scope:Link)\" | sed -e \"s/^.*addr:\s*\(\S*\)\(\/\d+\)*.*$/\1/g\"": -c: line 1: syntax error: unexpected end of file

Yes, I know I could use single quotes instead of double quotes to the -c argument, and then I wouldn't have to escape the double-quotes. And, yes, I know there are other ways to accomplish what that command does by parsing the output of ifconfig. I've got workarounds for this particular example, but I still don't know why it shouldn't work with TProcess with me escaping the double-quotes with a backslash and then passing the whole thing double-quote enclosed to bash -c as the TProcess CommandLine.

What am I doing wrong?

I've got an example program illustrating what I'm doing on PasteBin at http://pastebin.com/f336d384b, and I'll also include the contents of the program in this email in case anyone in the future's ever searching the archives and the pastebin link doesn't exist any more.

-SG

-----------------------------------------------------------------------------------------------------------------
program tprocquotes;
{$mode objfpc}{$H+}

uses

  {$IFDEF UNIX}
  cthreads,
  {$ENDIF}
  Classes,
  SysUtils,
  dateutils,
  process,
  pipes;
  function ExecuteProcess(const command : string;
                          out   commandExitCode : integer;
                          const timeoutSec : integer;
                          const stdout_stream : TStream;
                          const stderr_stream : TStream;
                          const redirectErr : boolean) : boolean;
  const
    READ_BYTES = 2048;
  var
   P : TProcess;
   buffer : array of byte;
   tmpStdOutput : TMemoryStream;
   tmpStdError : TMemoryStream;
   stdOutToUse : TStream;
   stdErrToUse : TStream;
   startTime : TDateTime;
   stdOutBytesRead : longword;
    function ReadAvailableBytes(const source_stream : TInputPipeStream;
                                const dest_stream : TStream) : longword;
    var
      bytes_available : longword;
      bytes_read      : longint;
      bytes_to_read   : longint;
    begin
      result := 0;
      if Assigned(source_stream) and Assigned(dest_stream) then begin
        bytes_available := source_stream.NumBytesAvailable;
        while (bytes_available > 0) do begin
          if (bytes_available >= READ_BYTES) then begin
            bytes_to_read := READ_BYTES;
          end else begin
            bytes_to_read := bytes_available;
          end;
          bytes_read := source_stream.Read(buffer[0], bytes_to_read);
          if (bytes_read > 0) then begin
            dest_stream.Write(buffer[0], bytes_read);
            result := result + longword(bytes_read);
          end;
          bytes_available := source_stream.NumBytesAvailable;
        end;
      end;
    end;
  begin
    result := true;
    commandExitCode := 1;
    if Assigned(stdout_stream) then begin
      tmpStdOutput := nil;
      stdOutToUse := stdout_stream;
    end else begin
      tmpStdOutput := TMemoryStream.Create;
      stdOutToUse := tmpStdOutput;
    end;
    stdOutToUse.Size := 0;
    stdOutToUse.Seek(0, soFromBeginning);
    if Assigned(stderr_stream) then begin
      tmpStdError := nil;
      stdErrToUse := stderr_stream;
    end else begin
      tmpStdError := TMemoryStream.Create;
      stdErrToUse := tmpStdError;
    end;
    stdErrToUse.Size := 0;
    stdErrToUse.Seek(0, soFromBeginning);
    SetLength(buffer, READ_BYTES);
    P := TProcess.Create(nil);
    try
      P.CommandLine := command;
      P.Options := [poUsePipes];
      if redirectErr then P.Options := P.Options + [poStderrToOutPut];
      P.InheritHandles := false;
      startTime := now;
      P.Execute;
      { read stdout while the process runs }
      while P.Running do begin
        stdOutBytesRead := ReadAvailableBytes(P.Output, stdOutToUse);
        if (stdOutBytesRead = 0) and
           (ReadAvailableBytes(P.Stderr, stdErrToUse) = 0) then begin
          sleep(100);
        end;
        { check timeout }
        if (timeoutSec > 0) and (SecondsBetween(now, startTime) >
timeoutSec) then begin
          result := false;
          P.Terminate(1);
          P.WaitOnExit; // avoid a zombie process
          break;
        end;
      end;
      commandExitCode := P.ExitStatus;
      { read what's left over in stdout }
      ReadAvailableBytes(P.Output, stdOutToUse);
      { read what's left over in stderr }
      ReadAvailableBytes(P.Stderr, stdErrToUse);
    finally
      if Assigned(tmpStdError) then FreeAndNil(tmpStdError);
      if Assigned(tmpStdOutput) then FreeAndNil(tmpStdOutput);
      FreeAndNil(P);
      SetLength(buffer, 0)
    end;
  end;
const
  COMMAND = 'bash -c "ifconfig | grep -E -e \"inet6? addr:\" | grep -v
-E -e \"(127.0.0.1|::1|Scope:Link)\" | sed -e
\"s/^.*addr:\s*\(\S*\)\(\/\d+\)*.*$/\1/g\""';
var
  stdOutStream : THandleStream;
  stdErrStream : THandleStream;
  commandExitCode : integer;
begin
  stdOutStream := THandleStream.Create(StdOutputHandle);
  stdErrStream := THandleStream.Create(StdErrorHandle);
  try
    if ExecuteProcess(COMMAND, commandExitCode, 120, stdOutStream,
stdErrStream, false) then begin
      ExitCode := commandExitCode;
    end else begin
      ExitCode := 1;
    end;
  finally
    FreeAndNil(stdOutStream);
    FreeAndNil(stdErrStream);
  end;
end.
-----------------------------------------------------------------------------------------------------------------
This email is fiction. Any resemblance to actual events or persons living or dead is purely coincidental. Seth Grover sethdgrover[at]gmail[dot]com

Reply by email

Seth Grover

Sigh, looking closer I realize it's probably because there's other characters in there I need to escape (like the \ and the $). I'll play around with that and see if that solves my problem.

I always see these things AFTER I send and email to the mailing list. Apologies.

-SG

This email is fiction. Any resemblance to actual events or persons living or dead is purely coincidental. Seth Grover sethdgrover[at]gmail[dot]com

Reply by email

Marco van de Voort
to Seth Grover

In our previous episode, Seth Grover said:
[ Charset UTF-8 unsupported, converting... ]

Show quoted text 6 lines
Sigh, looking closer I realize it's probably because there's other characters in there I need to escape (like the \ and the $). I'll play around with that and see if that solves my problem. I always see these things AFTER I send and email to the mailing list. Apologies.

Don't forget to search mantis too.

<cough> http://bugs.freepascal.org/view.php?id=14446 </uncough>

Reply by email

Seth Grover
to Seth Grover: Sigh, looking closer I realize it's probably because there's other characters in there I…

Well, even after playing with escaping various characters (like \, `, and "), it still wouldn't work, so I came up with a hack that will work for my needs.

In the case where I need to "bash -c" some command with TProcess, I am now writing the command out to a file, then just passing "bash /path/to/file" as the TProcess command line, then deleting the file.

Kind of a hack, but at least it works the way I want it to.

-SG

This email is fiction. Any resemblance to actual events or persons living or dead is purely coincidental. Seth Grover sethdgrover[at]gmail[dot]com

Reply by email

Seth Grover
to Seth Grover

I also just realized I sent all of these messages to the wrong mailing list (devel instead of pascal).

My apologies, it's been one of those days.

-SG

This email is fiction. Any resemblance to actual events or persons living or dead is purely coincidental. Seth Grover sethdgrover[at]gmail[dot]com

Reply by email

cobines
to Seth Grover: Well, even after playing with escaping various characters (like \, `, and "), it still…

If you look at how TProcess parses command line into parameters (fcl-process/src/unix/process.inc CommandToList) you see that it doesn't support escaping with backslash (inside or outside double quotes). Your command is parsed as (using FPC 2.5.1):

1: bash
2: -c
3: "ifconfig | grep -E -e \"inet6?
4: addr:\" | grep -v -E -e\"(127.0.0.1|::1|Scope:Link)\" | sed -e\"s/^.*addr:\s*\(\S*\)\(\/\d+\)*.*$/\1/g\""

cobines

Reply by email

Seth Grover
to Seth Grover: I also just realized I sent all of these messages to the wrong mailing list (devel…

Thank you! It makes sense now. I'll keep an eye on the enhancement request Marco pointed me to and use the workaround I came up with in the meantime.

-SG

Show quoted text 11 lines
If you look at how TProcess parses command line into parameters (fcl-process/src/unix/process.inc CommandToList) you see that it doesn't support escaping with backslash (inside or outside double quotes). Your command is parsed as (using FPC 2.5.1): 1: bash 2: -c 3: "ifconfig | grep -E -e \"inet6? 4: addr:\" | grep -v -E -e\"(127.0.0.1|::1|Scope:Link)\" | sed -e\"s/^.*addr:\s*\(\S*\)\(\/\d+\)*.*$/\1/g\"" -- cobines
This email is fiction. Any resemblance to actual events or persons living or dead is purely coincidental. Seth Grover sethdgrover[at]gmail[dot]com

Reply by email

Graeme Geldenhuys
to Seth Grover: I'm working with TProcess, which for the most part I love, but I'm scratching my head…
Seth Grover wrote 8 lines
I'm working with TProcess, which for the most part I love, but I'm scratching my head right now because of a problem I'm experiencing. Suppose I have the following command: ifconfig | grep -E -e "inet6? addr:" | grep -v -E -e "(127.0.0.1|::1|Scope:Link)" | sed -e "s/^.*addr:\s*\(\S*\)\(\/\d+\)*.*$/\1/g"

Do you mean the same as this issue:

http://bugs.freepascal.org/view.php?id=14446

TProcess does not support bash style syntax where parameters need to be placed in quotes etc. This issue confuses the hell out of me as well. Marco suggested a good solution - but it's not implemented yet. TProcess parameters must be an array and not a string. anyway, read the bug report to understand more of how TProcess interprets things.

Regards,
- Graeme -

fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal http://opensoft.homeip.net/fpgui/

Reply by email

Seth Grover
Graeme wrote
Do you mean the same as this issue: http://bugs.freepascal.org/view.php?id=14446 TProcess does not support bash style syntax

Yes, I can see that now. I agree that being able to specify the parameters as an array would be the ideal solution. Thanks.

-SG

This email is fiction. Any resemblance to actual events or persons living or dead is purely coincidental. Seth Grover sethdgrover[at]gmail[dot]com

Reply by email