Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / fpc-devel / 2014-07

UTF8 support for win\process.inc

2 messages · 1 person ·

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

Michael Thompson

G'day,

I'm on Windows 8/Trunk FPC

Been banging my head against the wall all weekend trying to get some UTF8 working with the LazUtils TProcessUTF8. By the end of that I realized no matter what I did there, I wasn't going to get anywhere.

I've now solved my problem using only TProcess. See below for the modified TProcess.Execute (win\process.inc)

Procedure TProcess.Execute;
Var
  i : Integer;
  PName,PDir,PCommandLine : PWideChar;     // Changed from PChar
  FEnv: pointer;
  FCreationFlags : Cardinal;
  FProcessAttributes : TSecurityAttributes;
  FThreadAttributes : TSecurityAttributes;
  FProcessInformation : TProcessInformation;
  FStartupInfo : STARTUPINFO;
  HI,HO,HE : THandle;
  Cmd : String;
begin
  PName:=Nil;
  PCommandLine:=Nil;
  PDir:=Nil;
  if (FApplicationName='') and (FCommandLine='') and (FExecutable='') then
    Raise EProcess.Create(SNoCommandline);
  if (FApplicationName'') then
    begin
    PName:=PWideChar(UTF8Decode(FApplicationName));   // Added UTF8Decode
    PCommandLine:=PWideChar(UTF8Decode(FCommandLine));// Added UTF8Decode
    end
  else If (FCommandLine'') then
    PCommandLine:=PWideChar(UTF8Decode(FCommandLine)) // Added UTF8Decode
  else if (Fexecutable'') then
    begin
    Cmd:=MaybeQuoteIfNotQuoted(Executable);
    For I:=0 to Parameters.Count-1 do
      Cmd:=Cmd+' '+MaybeQuoteIfNotQuoted(Parameters[i]);
    PCommandLine:=PWideChar(UTF8Decode(Cmd));         // Added UTF8Decode
    end;
  If FCurrentDirectory'' then
    PDir:=PWideChar(UTF8Decode(FCurrentDirectory));   // Added UTF8Decode
  if FEnvironment.Count0 then
    FEnv:=StringsToPChars(FEnvironment)
  else
    FEnv:=Nil;
  Try
    FCreationFlags:=GetCreationFlags(Self);
    InitProcessAttributes(Self,FProcessAttributes);
    InitThreadAttributes(Self,FThreadAttributes);
    InitStartupInfo(Self,FStartUpInfo);
    If poUsePipes in FProcessOptions then
      CreatePipes(HI,HO,HE,FStartupInfo,Not(poStdErrToOutPut in
FProcessOptions), FPipeBufferSize);
    Try
      If Not
CreateProcessW(PName,PCommandLine,@FProcessAttributes,@FThreadAttributes,
                   FInheritHandles,FCreationFlags,FEnv,PDir,FStartupInfo,
                   fProcessInformation) then
             // Changed to CreateProcessW
        Raise
EProcess.CreateFmt(SErrCannotExecute,[FCommandLine,GetLastError]);
      FProcessHandle:=FProcessInformation.hProcess;
      FThreadHandle:=FProcessInformation.hThread;
      FProcessID:=FProcessINformation.dwProcessID;
    Finally
      if POUsePipes in FProcessOptions then
        begin
        FileClose(FStartupInfo.hStdInput);
        FileClose(FStartupInfo.hStdOutput);
        if Not (poStdErrToOutPut in FProcessOptions) then
          FileClose(FStartupInfo.hStdError);
        CreateStreams(HI,HO,HE);
        end;
    end;
    FRunning:=True;
  Finally
    If FEnvNil then
      FreeMem(FEnv);
  end;
  if not (csDesigning in ComponentState) and // This would hang the IDE !
     (poWaitOnExit in FProcessOptions) and
      not (poRunSuspended in FProcessOptions) then
    WaitOnExit;
end;

Basically I changed 3 x PChar to PWideChar, then when the PWideChars are being set I do it via PWideChar(UTF8Decode(<string>)) Finally I changed CreateProcess to CreateProcessW.

I had thought I was going to need to change the datatypes for FCommandline etc, but this wasn't the case.

With the limited testing I've done, this seems to successfully enable UTF8 support within TProcess, without the end user needing to change any of their code. Also, this doesn't seem to break any existing code that uses TProcess.

Before I submit this as a patch though, I have concerns. Mainly around a) not knowing your development plans in this area and b) me not really knowing unicode (just because this works, doesn't mean it's right).

A summary of my concerns:
* I note in redef.inc there is already a redirect for CreateProcess to CreateProcessW. This implies to me there is already a plan in mind, and I might be cutting across that by directly calling CreateProcessW.
* My code doesn't take UTF16 into account. In truth, I'm not sure how to do that.
* I've only done limited testing on a Win 8 64bit PC set to Australian regional settings. Really I think I need to test this on a wide variety of regional settings and across a variety of Win OS's...

Before I submit a patch based on this can anyone tell me if I'm on the wrong track? (and if so, where)

Many thanks

Mike Thompson

Reply by email

Michael Thompson
to Michael Thompson

Apologies, I forgot to include a link to just the patch...

https://dl.dropboxusercontent.com/u/59503182/fpc/process.inc.patch

Mike T

Michael Thompson wrote 125 lines
G'day, I'm on Windows 8/Trunk FPC Been banging my head against the wall all weekend trying to get some UTF8 working with the LazUtils TProcessUTF8. By the end of that I realized no matter what I did there, I wasn't going to get anywhere. I've now solved my problem using only TProcess. See below for the modified TProcess.Execute (win\process.inc) Procedure TProcess.Execute; Var i : Integer; PName,PDir,PCommandLine : PWideChar; // Changed from PChar FEnv: pointer; FCreationFlags : Cardinal; FProcessAttributes : TSecurityAttributes; FThreadAttributes : TSecurityAttributes; FProcessInformation : TProcessInformation; FStartupInfo : STARTUPINFO; HI,HO,HE : THandle; Cmd : String; begin PName:=Nil; PCommandLine:=Nil; PDir:=Nil; if (FApplicationName='') and (FCommandLine='') and (FExecutable='') then Raise EProcess.Create(SNoCommandline); if (FApplicationName'') then begin PName:=PWideChar(UTF8Decode(FApplicationName)); // Added UTF8Decode PCommandLine:=PWideChar(UTF8Decode(FCommandLine));// Added UTF8Decode end else If (FCommandLine'') then PCommandLine:=PWideChar(UTF8Decode(FCommandLine)) // Added UTF8Decode else if (Fexecutable'') then begin Cmd:=MaybeQuoteIfNotQuoted(Executable); For I:=0 to Parameters.Count-1 do Cmd:=Cmd+' '+MaybeQuoteIfNotQuoted(Parameters[i]); PCommandLine:=PWideChar(UTF8Decode(Cmd)); // Added UTF8Decode end; If FCurrentDirectory'' then PDir:=PWideChar(UTF8Decode(FCurrentDirectory)); // Added UTF8Decode if FEnvironment.Count0 then FEnv:=StringsToPChars(FEnvironment) else FEnv:=Nil; Try FCreationFlags:=GetCreationFlags(Self); InitProcessAttributes(Self,FProcessAttributes); InitThreadAttributes(Self,FThreadAttributes); InitStartupInfo(Self,FStartUpInfo); If poUsePipes in FProcessOptions then CreatePipes(HI,HO,HE,FStartupInfo,Not(poStdErrToOutPut in FProcessOptions), FPipeBufferSize); Try If Not CreateProcessW(PName,PCommandLine,@FProcessAttributes,@FThreadAttributes, FInheritHandles,FCreationFlags,FEnv,PDir,FStartupInfo, fProcessInformation) then // Changed to CreateProcessW Raise EProcess.CreateFmt(SErrCannotExecute,[FCommandLine,GetLastError]); FProcessHandle:=FProcessInformation.hProcess; FThreadHandle:=FProcessInformation.hThread; FProcessID:=FProcessINformation.dwProcessID; Finally if POUsePipes in FProcessOptions then begin FileClose(FStartupInfo.hStdInput); FileClose(FStartupInfo.hStdOutput); if Not (poStdErrToOutPut in FProcessOptions) then FileClose(FStartupInfo.hStdError); CreateStreams(HI,HO,HE); end; end; FRunning:=True; Finally If FEnvNil then FreeMem(FEnv); end; if not (csDesigning in ComponentState) and // This would hang the IDE ! (poWaitOnExit in FProcessOptions) and not (poRunSuspended in FProcessOptions) then WaitOnExit; end; Basically I changed 3 x PChar to PWideChar, then when the PWideChars are being set I do it via PWideChar(UTF8Decode(<string>)) Finally I changed CreateProcess to CreateProcessW. I had thought I was going to need to change the datatypes for FCommandline etc, but this wasn't the case. With the limited testing I've done, this seems to successfully enable UTF8 support within TProcess, without the end user needing to change any of their code. Also, this doesn't seem to break any existing code that uses TProcess. Before I submit this as a patch though, I have concerns. Mainly around a) not knowing your development plans in this area and b) me not really knowing unicode (just because this works, doesn't mean it's right). A summary of my concerns: * I note in redef.inc there is already a redirect for CreateProcess to CreateProcessW. This implies to me there is already a plan in mind, and I might be cutting across that by directly calling CreateProcessW. * My code doesn't take UTF16 into account. In truth, I'm not sure how to do that. * I've only done limited testing on a Win 8 64bit PC set to Australian regional settings. Really I think I need to test this on a wide variety of regional settings and across a variety of Win OS's... Before I submit a patch based on this can anyone tell me if I'm on the wrong track? (and if so, where) Many thanks Mike Thompson

Reply by email