Reinforcing the subject, this is using Linux - to be specific, FPC 2.4.2-0 under 64 bit Mint 9.

I'm trying to convert a large number of MP3 files to Ogg Vorbis.

This is a two step process, first I run mpg321 to switch them to a .wav file, then oggenc to convert to a .ogg

What's driving me crazy is that running the two commands via ExecuteProcess does the first step OK, but oggenc fails with an exit code of 1, operation not permitted.

If I replace the ExecuteProcess with a call to fpSystem, concatenating the CommandString and ParamString with a space as separator and passing that as the command, both steps work just fine.

I've tried putting a Delay(5000) between the two commands using ExecuteProcess, just in case there was some type of problem with data still being written from a cache. No difference whatever.

This is reproducible on a whole batch of files, and yes, I've checked all the file permissions.

It's got to be something obvious, or some quirk of Linux programming that I haven't met up with yet (I'm still a novice with FreePascal and Linux, though I've many years experience with Delphi and Windows). What am I missing?

Thanks,

Brian.

I can't really answer you regarding why ExecuteProcess doesn't work. But just wanted to ask: Have you thought of trying TProcess instead? I normally execute any external programs via TProcess with good results
- no matter the platform.

Regards,
- Graeme -

Nope. I'll give TProcess a whirl. Thanks for the suggestion.

Brian.

In our previous episode, Graeme Geldenhuys said:

I can't really answer you regarding why ExecuteProcess doesn't work. But just wanted to ask: Have you thought of trying TProcess instead? I normally execute any external programs via TProcess with good results - no matter the platform.

The inner plumbing of executeprocess and tprocess should be the same. TProcess just has several options (shell and piping)

In our previous episode, brian said:

First: what does a typical command look like?
second: maybe ogenc is simply faulty and won't work unless directly called from shell.

Marco van de Voort wrote

The inner plumbing of executeprocess and tprocess should be the same. TProcess just has several options (shell and piping)

I thought of that, right after I sent my message. But maybe there is a slim chance that he is simply not using ExecuteProcess() correctly when he uses it directly. Maybe TProcess hides some of the complexity and setup code, which will make things work. Like I said, maybe.... or maybe not. :)

In our previous episode, Graeme Geldenhuys said:

I think the smartest avenue would be to executeprocess with shell, to test if it is shell or not.

executeprocess is the bare plumbing of tprocess and is quite stable (e.g. it has been safe for the commandline parsing problems that have haunted TProcess since it emerged in 1.9.6)

Using executeprocess to diagnose TProcess problems makes sense. (to see if it is in the core features or the "fluff"), the other way around not really, unless you really need piping.

try

strace -o logfile.txt -f yourprogram

It will emit all system calls. Near to the end, there should be one that fails with 'operation not permitted'. (assuming your program exits straight away after the failed command)

This way you'll know the actual cause of the failure.

Michael.

from what I understand, you are trying to run 2 commands?

one question that I ask myself then is: why does your driving program have to be a fpc program? What is wrong with a shell script?

while you may have valid reasons for doing so, have you tried to put that flow (calling of the 2 commands) into a single shell script and executing that instead?

kind regards,
Dimitri Smits

Brian:

In the case of ExecuteProcess() parameter separation takes place on FPC side, while with fpSystem() the shell is responsible for it. Could a problem with FPC's parameter separation algorithm be causing this?

Can you post both the commands you are executing?

When I was learing processes, I wrote a unit to exe- cute a command using fpExecv -- just as ExecuteProcess() does. I'll dig it out and we'll see how (and whether) it works with your examples.

Anton

In our previous episode, Anton Shepelev said:

In the case of ExecuteProcess() parameter separation takes place on FPC side, while with fpSystem() the shell is responsible for it.

Executeprocess has two forms. One does parameter separation, and one not, and directly passes the separated parameters to fpexec.

This was actually the reason for executeprocess.

Could a problem with FPC's parameter separation algorithm be causing this?

Maybe, but only when you use the wrong variant.

Can you post both the commands you are executing?

Yes, that would be the best.

When I was learing processes, I wrote a unit to exe- cute a command using fpExecv -- just as ExecuteProcess() does. I'll dig it out and we'll see how (and whether) it works with your examples.

There should be no need for that.

OK, sorry for the delay in catching up with all the responses. I'll roll everything into one message.

Why am I using a subprocess rather than writing a batch file? There's a bunch of other processing going on, but none of it involves manipulating the files other than a possible rename. It's selection of which files to convert according to various criteria, elimination of some duplicates, that sort of thing. It just seemed tidier to me to do the whole lot in the program rather than doing most of the work then write a batch file for the actual conversions.

What are the commands?

mpg321 -q -w "tempfile.wav" "inputfile.mp3"

oggenc -Q --output="outputfile.ogg" "tempfile.wav"

It makes no difference whether or not I use a full pathname for mpg321 and oggenc, and all the other files are in the current working directory. Yes, there really are quotes round the filenames, the vast majority of the real names have embedded spaces.

As regards oggenc being faulty and not working unless called from the shell, no, that one hadn't occurred to me, not sure how to test it.

Running with strace. Eek! 6.5 MB of trace output. :) I think it's found the problem, though. I get a bunch of

"ERROR: Multiple files specified when using stdin"

messages.

It's difficult to see why this should be the case. I have the program name and the parameters set up as strings within my program. To switch between the ExecuteProcess and fpSystem calls, I just commented out the one I wasn't trying. The two lines of code are

	Status := ExecuteProcess(CommandString, ParamString);

and

	Status := fpSystem(CommandString+' '+ParamString);

CommandString just contains mog321 or oggenc (with or without the full path) and ParamString is the rest of the commands I gave above.

Do I need to add a separator to the front of ParamString when using ExecuteProcess? It wouldn't seem sensible to require it, since there are the two parameters to ExecuteProcess versus the one for fpSystem, but maybe that's it.

Thanks for the assistance so far, everyone.

Brian.

I suspect that the oggenc command is failing because TProcess behind the scenes is splitting --output="outputfile.ogg" into two params.

You can try something like
oggenc -Q '--output="outputfile.ogg"' "tempfile.wav" to see if that works or if you use TProcess directly I understand that options have been added recently to overcome the problem quotes can have on the arguments.

Regards,

Andrew

or consider using -o "outputfile.ogg" which won't have the problem of begin split to two params since it already is.

Regards, Andrew

In our previous episode, brian said:

That's probably the problem. You use ", which might be problematic with commandline separation and don't offer the commands to exexuteprocess in a separated matter.

Try
Status:=ExecuteProcess('/path/to/oggenc',['-Q','--output="outputfile.org"','tempfile.wav']);

You must remove the quotes around 'outputfile.org':

  Status:=ExecuteProcess('/path/to/oggenc',['-Q','--output=outputfile.org','tempfile.wav']);

Michael.

The quotes are bash commands, they are not part of command line. I wrote on this topic not long ago, but for TProcess:

http://wiki.lazarus.freepascal.org/Executing_External_Programs#Parameters_which_contain_spaces_.28Replacing_Shell_Quotes.29

Andrew Haines:

I suspect that the oggenc command is failing because TProcess behind the scenes is splitting --output="outputfile.ogg" into two params.

Yes, indeed. It is also mentioned on the WIKI:

http://wiki.freepascal.org/Executing_External_Programs (see section
"Parameters which contain spaces (Replacing Shell Quotes)"

While the documentation on TProcess and fpExecV is more ambiguous:

TProcess:
If the command to be executed or any of the
arguments contains whitespace (space, tab
character, linefeed character) it should be
enclosed in single or double quotes.

fpExecV:
In case ComLine is a single string, it will
be split out in an array of strings, taking
into account common whitespace and quote
rules.

Anton

In our previous episode, Anton Shepelev said:

It should now be possible to pass the parameters to tprocess separated. I don't remember if that change made it to 2.4.4 though, otherwise it will be in 2.6.0.

OK, I've read your notes for TProcess, I'll give that a try. Thanks.

I have to say that when you have two overloaded declarations for ExecuteProcess, one with an array of parameters and one with a single parameter string, it's not clear to me why you then break the single string into individual parameters, but I'm sure there's some good reason for it. Maybe that single parameter string form of the command should be deprecated. I certainly think that the documentation is directly misleading :-

<paste>

In case ComLine is a single string, it will be split out in an array of strings, taking into account common whitespace and quote rules.

</paste>

At least IMHO, having to quote the entire parameter string, not just the value, is anything but "common whitespace and quote rules".

However, thanks again for the assistance, folks. You can be sure this one has been carefully noted - the idea of needing to wrap parameter AND value in quotes wouldn't have occurred to me in a month of Sundays.

Brian.

brian wrote

At least IMHO, having to quote the entire parameter string, not just the value, is anything but "common whitespace and quote rules".

I am pretty sure that I have had to deal with this in the past on unix, writing shell or Perl scripts i.e. not in the context of FPC. It might, in the general case, be easier to enforce a rule that filenames don't contain characters which have special significance, at which point option values aren't going to need to be quoted except for cases like OPT= passed to the FPC build.

In our previous episode, said:

You must remove the quotes around 'outputfile.org': Status:=ExecuteProcess('/path/to/oggenc',['-Q','--output=outputfile.org','tempfile.wav']);

While better, this is afaik not required per se, the executing binary can strip them also while doing cmdline parsing. It doesn't matter for executeprocess.

Marco van de Voort:

It should now be possible to pass the parameters to tprocess separated. I don't remember if that change made it to 2.4.4 though, otherwise it will be in 2.6.0.

Thanks for letting me know.

Is the parameter separation algorithm intentionally made to split partially quoted arguments like '--file="my file"'? If not, then, maybe the StringToPPChar() function should be fixed?

Anton

Yes, the executing binary can do a lot, but the trouble is that it doesn't expect that it should need to do that (and thus it probably doesn't do it) since the Unix shell would probably strip these quotes _if_ the command is started from shell. That is probably the whole reason why it fails for the original poster right now.

Tomas

Marco van de Voort wrote

While better, this is afaik not required per se, the executing binary can strip them also while doing cmdline parsing.  It doesn't matter for executeprocess.

I wrote that section in the wiki when I had a real issue where the app did not work if you have quotes in a parameter (I don't remember which one it was anymore). Similarly I'd suppose that many other applications won't work with quotes. Maybe some will work....

In general, you can't expect that from the executing binary. It's safe to assume that most don't do this.
(I've never seen one that does, in fact)

Michael.

I don't know any programs other than shells that would do so. FPC itself doesn't do anything like that either, and hence would complain that it couldn't find a file with such a name (quotes are valid characters in file names, at least on Unix-like OSes).

In general, such things are only done if the parameters are passed on to a shell, because that one will again process quotes (and then it could be required if "outputfile.org" contains spaces).

Jonas

Anton Shepelev wrote

Is the parameter separation algorithm  intentionally made   to  split  partially  quoted  arguments  like '--file="my  file"'?   If  not,  then,   maybe   the StringToPPChar() function should be fixed?

It does not split partially quoted arguments AFAIK Not sure about Free Pascal Trunk however.

If FPC Trunk also does not support this, then I'm sure a patch to improve StringToPPChar would be welcome.

In our previous episode, Jonas Maebe said:

I was thinking about --xx="y y" being windows style of quoting and windows originating apps stripping, and Unix style quoting is more over the entire parameter ( "--xx=y y" ), with the shell stripping (though the shell does support nesting iirc)

But forget about it, for the thread, Michael's comment is perfectly valid, and I didn't meant to say this was better. It was meant as a side remark.

Felipe Monteiro de Carvalho:

Version 2.4.4 does split, say

--name="Anton Shepelev"

into two parametes:

a. --name="Anton
b. Shepelev"

The latest revision of the file /rtl/unix/ unixutil.pp dates to 2010, so I think this has never been fixed.

Anton

Felipe Monteiro de Carvalho:

If FPC Trunk also does not support this, then I'm sure a patch to improve StringToPPChar would be welcome.

I have created issue # 0020279.

Anton

2011/9/17 Anton Shepelev :

If this must be like shell quoting then you also need to allow quoting with backslash, inside single quotes escaping is not allowed and not mix single and double quotes.

For example:

param\ eter1 'param\"'eter2' param"eter3' "parameter'4"

should do:

1: param eter1
2: param\"eter2 param"eter3
3: parameter'4

There is a good guide here which I used for similar purpose: http://www.grymoire.com/Unix/Quote.html

In our previous episode, cobines said:

> I have created issue # 0020279. If this must be like shell quoting

These routines were never meant to be shell quoting (which is silly since it is a Runtime lib call, not a shell). I also don't see the point why it should be expanded to do shell quoting.

The original executeprocess was made as replacement for dos.exec, which was before 1.9.2 the only portable way to call a function, but had a 255 char limit.

The original string,string version of executeprocess solved that, and to better suit Unix, a version with an open array was made because it suits Unix api conventions (execve) more, and because since the Unix restructure that option was open.

So basically this bugreport is about adding *nix shell quoting to a routine that was meant to mimic WINDOWS API (iow not even Windows shell) behaviour.

Some minor quoting support that was already there was added for easy conversion, but I'm very reluctant to expand that and get even more corner cases.

I added comments to that end to above bugreport.

Marco van de Voort:

So basically this bugreport is about adding *nix shell quoting to a routine that was meant to mimic WINDOWS API (iow not even Windows shell) be- haviour.

I only wanted to fix the parsing of partially quoted parameters. Of course, it is best to deprecate this function and avoid all confusion.

Anton