Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / fpc-pascal / 2021-07

TProcess read buffer size

4 messages · 2 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

Ryan Joseph

I have some code the basically does:

while bytesRead > 0 do
	bytesRead := process.Output.Read(buffer^, kBufferSize);

but bytesRead is only ever 512 per call to Read.

Is this a system imposed limit or something that's part of TProcess? Setting kBufferSize to anything larger than 512 has no affect.

Regards,
Ryan Joseph

Reply by email

Marco van de Voort
to Ryan Joseph

Op 2021-07-15 om 20:26 schreef Ryan Joseph via fpc-pascal:

Show quoted text 8 lines
I have some code the basically does: while bytesRead > 0 do bytesRead := process.Output.Read(buffer^, kBufferSize); but bytesRead is only ever 512 per call to Read. Is this a system imposed limit or something that's part of TProcess? Setting kBufferSize to anything larger than 512 has no affect.

Not of TProcess as far as I know, it probably depends on the OS pipe implementation and maybe the granularity that the processes that you run outputs data.  (buffers its I/O iow calls write()  in 512 byte increments )

Reply by email

Ryan Joseph
to Marco van de Voort
On Jul 15, 2021, at 3:07 PM, Marco van de Voort via fpc-pascal wrote: Not of TProcess as far as I know, it probably depends on the OS pipe implementation and maybe the granularity that the processes that you run outputs data. (buffers its I/O iow calls write() in 512 byte increments )

In this case the command is a PHP script that reads a file in 64k chunks and writes the text to stdout (in PHP fread returns 64k chunks as expected). TProcess looks like a pretty thin wrapper around pipes and common unix system calls so I don't know where the 512 limit could come from.

How could I test this further? I think something is wrong but I don't know where to look. I'm on macOS btw.

Regards,
Ryan Joseph

Reply by email

Ryan Joseph
Show quoted text 16 lines
On Jul 15, 2021, at 8:33 PM, Dennis Lee Bieber via fpc-pascal wrote: https://flylib.com/books/en/3.126.1.111/1/ """ Only writes below PIPE_BUF bytes in size are guaranteed to be atomic. PIPE_BUF is 512 bytes on Mac OS X. The fpathconf() system call can be used to retrieve the value of PIPE_BUF given a pipe descriptor. """ Granted you seem to be reading -- but the other end may be what is controlling the size of each packet (and may somehow even have set the pipe overall size to just one "buffer"). Unless you can examine the writing side, or source code of your read method...

Yes I think you're right.

~$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 256
pipe size (512 bytes, -p) 1
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 2784
virtual memory (kbytes, -v) unlimited

It's just a PHP script that dumps 64k chunks but if the pipe buffer is 512b then I'm stuck. Is there a lower level library I can use to set the pipe size? I was curious if this is a bottle neck for me. Currently it takes about 7 seconds to read a 600MB file from one process to another, which is insanely slow.

Regards,
Ryan Joseph

Reply by email