Hi,

Whats the differences or pros and cons between using TProcess vs RunProcess().

Regards,
Graeme

Graeme Geldenhuys wrote

Hi, Whats the differences or pros and cons between using TProcess vs RunProcess().

TProcess gives you full access to the process. You can kill it, pause it, write to stdin, read from stdout.

These things cannot be done with RunProcess.
That's one-shot and wait till it exits.

Michael.

Michael Van Canneyt wrote

These things cannot be done with RunProcess. That's one-shot and wait till it exits.

Perfect and very handy to know the difference. Thanks for the explanation.

Regards,
Graeme

Another difference.... simply so others might learn from this discussion (like I did).

If you have a huge list of parameters, RunProcess() - at least under Windows - will most likely hit a 260 character limit, and then fail to actually execute. I don't actually know where that 260 limit originates from (Windows API, Command Prompt, FPC etc).

Whereas if you use TProcess and the Parameters (TStrings) property, you don't have such a limitation.

This was what I experienced.

Regards,
Graeme

In our previous episode, Graeme Geldenhuys said:

Which runprocess are you talking about? I know a process.runcommand and a sysutils.executeprocess, but not a runprocess.

Whereas if you use TProcess and the Parameters (TStrings) property, you don't have such a limitation. This was what I experienced.

if you are talking about executeprocess, test again with trunk.

Marco van de Voort wrote

Which runprocess are you talking about? I know a process.runcommand and a sysutils.executeprocess, but not a runprocess.

Apologies, our code is RunProcess(), which internally calls SysUtils.ExecuteProcess()

if you are talking about executeprocess, test again with trunk.

At the moment we are using FPC 2.6.4 and might move to FPC 3.0.2 when that is final. Not using Trunk I'm afraid.

Was the 260 char limit removed/fixed in FPC trunk?

Regards,
Graeme

In our previous episode, Graeme Geldenhuys said:

First, I'm not aware of a different between the two, so if you have tests for that I'm interested. (at least if you are testing 3.0 or newer, but that I can easily do myself if you have simple reproduction case).

The strange thing is that all variants use CreateProcess, but afaik only executeprocess in trunk uses the -w variant. I started upgrading tprocess, but it got out of hand, and I shelved the work.

Afaik by prefixing with \\.\ you can use larger paths with -w variants.

Similar to FpSystem() and ExecuteProcess, and then there is AssignStream but assignstream/fpsystem unix only

http://www.freepascal.org/docs-html/rtl/sysutils/executeprocess.html

http://www.freepascal.org/docs-html/rtl/unix/assignstream.html

http://www.freepascal.org/docs-html/rtl/unix/fpsystem.html

I like fpsystem because it does not require absolute paths to commands, too bad fpsystem not available on Windows.

All the above is used in webcmd (part of powtils) and also fpcRun ;-)

But I may try RunProcess/TProcess soon!

One issue I discovered with ExecuteProcess, and fpSystem, is I cannot figure out a way to capture StdErr... TProcess likely has more control over this. Tried

Assign(StdErr, 'somefile.txt')
Rewrite(StdErr)

And didn't seem to work

So if you run a process and nothing prints to stdout, and there is some data printed to stderr, I do not know how to capture it. Which makes me want to try TProcess instead, but if I could do it with FpSystem and ExecuteProcess that would be nice

In our previous episode, Lars said:

Executeprocess doesn't support piping. It orignally was mainly meant as a portable dos.exec with parameters passed separately and without string length limits, implementable on RTL level.

If I were you I would simply look at tprocess sources how it pipes stderr.

Lars wrote

So if you run a process and nothing prints to stdout, and there is some data printed to stderr, I do not know how to capture it. Which makes me want to try TProcess instead

Indeed, that is one of the big differences between TProcess and ExecuteProcess. I use TProcess often to execute something and then capture the output results. fpGUI's example IDE (Maximus) shows how it can be done with TProcess.

Regards,
Graeme

Graeme Geldenhuys wrote

If you have a huge list of parameters, RunProcess() - at least under Windows - will most likely hit a 260 character limit, and then fail to actually execute. I don't actually know where that 260 limit originates from (Windows API, Command Prompt, FPC etc).

it is easy to use more that 260 if you pass file names with long paths, and people nowadays love to name directories and files with strings that instead of titles look like abstracts.

In delphi and other languages I've hit may times the 260 character limit in windows running a Shell or using ShellApi (I supposed the limit was 255, not 260, there are always new things to learn) , I have also hit that limit using the command line. Nevertheless using windows API createprocess, there is not such limit (but the call is less handy and more verbose), or maybe I haven't hit it because parameters and call function are separated strings , so they are shorter strings. On the other hand, CreateProcess must call an executable file, can't execute internal commands like 'dir' or 'del', not a big deal but there it is. I think that the limit is in the command line interpreters command.exe and cmd.exe.

As far as I see, ExecuteProcess finally calls to an external symbol named 'FPC_SYSCALLx'. And TProcess finally calls an external symbol 'FPC_SYSC_FORK'. Maybe ExecuteProcess is calling some kind of shell, Freepascal developers will tell.

In my delphi days we implemented a procedure ("RunCommand" was it's name, If memory serves), that was a warp to a complex call to createprocess to avoid these problems. Maybe executeProcess could be implemented this way. Don't know what are the drawbacks (slower, more memory..?)

In our previous episode, Santi said:

There is a lot of confusion about this. (only -W calls can do larger versions, and only if \\.\, but there are also reports that it works with -A as long as the system is capable)

Internal commands are not internal to the OS, but internal to the shell, so you need to execute a shell for that. On both Linux and Windows.

As far as I see, ExecuteProcess finally calls to an external symbol named 'FPC_SYSCALLx'. And TProcess finally calls an external symbol 'FPC_SYSC_FORK'. Maybe ExecuteProcess is calling some kind of shell, Freepascal developers will tell.

On *nix, both will call fork followed by execve. On Windows it will call CreateProcess, though executeprocess will call -W in trunk, and afaik TProcess still -A.

ExecuteProcess is to have some portable execution functionality in the base RTL. For more complex cases there is the TProcess class that also supports piping etc.

Some likely scenarios (e.g. run and capture stdout) have been prepared in RunCommand, also in unit Process.

The main motivation for adding these is that 90+% of the TProcess and CreateProcess usage in the forums was faulty (specially for larger outputs or when the process also printed a lot on stderr (thanks Pierre!)).

Abstracting them in RunCommand reduces the amount of faulty code in play.

Santi wrote

Maybe ExecuteProcess is calling some kind of shell, Freepascal developers will tell.

To call the shell you send in cmd /C

FpSystem does this, but it's on unix

Calling the shell is useful for not requiring absolute paths to commands... which is why I like fpSystem...

And to call a dos command you need a shell like cmd /C because calling a dos command won't do anything, it's not an exe (there is no DIR.exe file for example AFAIK).. so to call DIR (same as LS on unix) you need to invoke a shell.

So people were having issues with stderr going crazy (lots of prints to stderr)?

And this is the problem I have with executeprocess: I want the results from stderr, and need them, but can't figure out how to access them.. only stdout is what I retrieve writes from.

Well I may modify executeprocess/fpsystem to deal with stderr if that is a missing feature (may not be possible, haven't looked into it much).

Or, get assignstream working on windows and not just unix, but that will be a bit of a big undertaking, and again may not be possible.

mseide/msegui may also have some tips in its source code as it captures a lot of stuff and puts it into the ide in its custom widgets that trap outputs, actually kind of a bit like a Blackbox ComponentPascal IDE from switzerland, but still different ;-)

But doesn't all processes report to stderr?
Ones that are created in a program do not report to stderr?

The OS makes it optional if you want to report to stderr? Interesting..

Another idea is to modify AssignStream to work on windows as AFAIK it only works on unix from the unix units of rtl..

Quote:
function AssignStream(var StreamIn, StreamOut, StreamErr: Text; const prog: String): LongInt;
{
Starts the program in 'prog' and makes its input, output and error output the other end of three pipes, which are the stdin, stdout and stderr of a program specified in 'prog'.

^^^ very useful function!

Reason I want to avoid TProcess is simply because cgi programs being around 30-100K are easier to upload for people with limited internet connection speeds. As soon as I pull in large units that involve lots of classes the cgi programs become over 200-500K.

However I will use tprocess if it's a time save, which indeed it is instead of rewriting assignstream for windows.

Plus, golang cgi exe's are about 2MB in size, so golang ain't got anything on fpc at this moment (fpc still creates small programs compared to GoLang runtime programs!)

Lars wrote

But doesn't all processes report to stderr? Ones that are created in a program do not report to stderr? The OS makes it optional if you want to report to stderr? Interesting..

Every process (on Windows: every non-GUI process) on maintstream OSes (and on many non-mainstream OSes too) has by default two standard file handles available to which it can write output: stdout and stderr.

Jonas

Lars wrote

Calling the shell is useful for not requiring absolute paths to commands... which is why I like fpSystem...

Never ship a program that uses fpSystem for that purpose, because it's one of the most basic security holes you can have: https://www.securecoding.cert.org/confluence/pages/viewpage.action?pageId=2130132

Jonas

In our previous episode, Lars said:

Not without effectively changing it into TProcess.

Or, get assignstream working on windows and not just unix, but that will be a bit of a big undertaking, and again may not be possible.

Again, duplicating TProcess.

mseide/msegui may also have some tips in its source code as it captures a lot of stuff and puts it into the ide in its custom widgets that trap outputs, actually kind of a bit like a Blackbox ComponentPascal IDE from switzerland, but still different ;-)

So thus Lazarus. It uses.... TProcess.

Hint: use TProcess :_)

In our previous episode, Lars said:

Afaik there are only conventions. And the convention to return a non zero returnvalue on error is much stronger than parsing stderr.

Another idea is to modify AssignStream to work on windows as AFAIK it only works on unix from the unix units of rtl..

Yes, Unix only, and the abstracted OS independent pipes already are in unit pipes used by TProcess.

Reason I want to avoid TProcess is simply because cgi programs being around 30-100K are easier to upload for people with limited internet connection speeds. As soon as I pull in large units that involve lots of classes the cgi programs become over 200-500K.

Yes. And they might fit on one side of a 5.25" inch dd floppy too! Yeah, let's party like it's 1999!

However I will use tprocess if it's a time save, which indeed it is instead of rewriting assignstream for windows.

Good.

Except assignstream doesn't require TProcess.Create and Free and uses old school programming, which, for some reason, I'm addicted to. Maybe just stuck in 1970's even though I was born in 1980s

You don't seem to understand the concept of RAD web development ;-)

If it takes me 2 minutes to upload a massive 2mb cgi program through FTP, since I pay half price for my internet connection for half speed DSL/Cable...... That's not rad.

Sometimes I develop a cgi program and upload it every few minutes, so waiting 1-2 minutes for a cgi to upload with total commander is just pathetic.

Uploading a 50kb or 30kb cgi is where it is at.

I live in a first world country but in third world countries with dial up or whatever slow dsl they use, I can imagine a 2MB golang cgi (or lazarus zeos/classes/other cgi) taking more than 2 minutes. That's not RAD

Just try compling some hello world or simple programs in lazarus and imagine they take 2 minutes to compile. That's what C++ compilers do. If you are a c++ fan I guess 2MB/2minute upload cgi programs are satisfactory...

Some of us do RAD cgi programming where everything is instant.

Good for you living in a country where dsl/cable is very fast and you don't care about FTP upload times. I think they call that being spoiled, or being white or something.

RAD is for Rapid Application *Development*, not Deployment. For developing/testing applications one should use a local server anyway as that also allows to use a debugger if necessary.

Regards,
Sven