Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / fpc-pascal / 2012-04

Compiling from and to memory

16 messages · 8 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

OBones

Hello all,

I'm currently creating a build chain where FPC is the last step, the one producing a DLL for Windows, x86 and x64.
In front of it in the chain, I have designed a generator that creates a program and a set of units from its own representation. Right now, the program and units are written to disk and then FPC is called on those files.
This works fine but is somewhat slow because of disk writing and reading. I was thus wondering if there was a way to send the program and unit contents directly to FPC without writing to disk, and then get FPC to output the final DLL to a memory location as well. The last part is optional, being able to send the program and units without disk access would be a nice thing.
I looked around in the documentation for "standard input", "pipe", but apart from ways to access them from within a program, I was not able to find any information.
Is this possible? If yes, how should I proceed?

Thank you for your answers
OBones

Reply by email

Unknown author
to OBones
OBones wrote 17 lines
Hello all, I'm currently creating a build chain where FPC is the last step, the one producing a DLL for Windows, x86 and x64. In front of it in the chain, I have designed a generator that creates a program and a set of units from its own representation. Right now, the program and units are written to disk and then FPC is called on those files. This works fine but is somewhat slow because of disk writing and reading. I was thus wondering if there was a way to send the program and unit contents directly to FPC without writing to disk, and then get FPC to output the final DLL to a memory location as well. The last part is optional, being able to send the program and units without disk access would be a nice thing. I looked around in the documentation for "standard input", "pipe", but apart from ways to access them from within a program, I was not able to find any information. Is this possible? If yes, how should I proceed?

It is currently not possible.

FPC always reads from and writes to files.

Michael.

Reply by email

Mattias Gaertner
to OBones: I'm currently creating a build chain where FPC is the last step, the one producing a DLL…
Show quoted text 18 lines
Hello all, I'm currently creating a build chain where FPC is the last step, the one producing a DLL for Windows, x86 and x64. In front of it in the chain, I have designed a generator that creates a program and a set of units from its own representation. Right now, the program and units are written to disk and then FPC is called on those files. This works fine but is somewhat slow because of disk writing and reading. I was thus wondering if there was a way to send the program and unit contents directly to FPC without writing to disk, and then get FPC to output the final DLL to a memory location as well. The last part is optional, being able to send the program and units without disk access would be a nice thing. I looked around in the documentation for "standard input", "pipe", but apart from ways to access them from within a program, I was not able to find any information. Is this possible? If yes, how should I proceed?

Maybe you can create a ramdisk.

Mattias

Reply by email

Mark Morgan Lloyd
to OBones
OBones wrote 6 lines
Mattias Gaertner wrote: > Maybe you can create a ramdisk. Thank you for your answer. I have thought about that, but this would create deployment problems as there is no default ramdisk driver under Windows. Might be worth considering should performance become a really big issue.

This is what heavyweight database servers use solid state drives for. In any event, I think it's worth remembering that pipes etc. will always go through the operating system, so there's a limit to how much you can slim things.

Mark Morgan Lloyd markMLl .AT. telemetry.co .DOT. uk [Opinions above are the author's, not those of his employers or colleagues]

Reply by email

Florian Klaempfl
to OBones: I'm currently creating a build chain where FPC is the last step, the one producing a DLL…
OBones wrote 18 lines
Hello all, I'm currently creating a build chain where FPC is the last step, the one producing a DLL for Windows, x86 and x64. In front of it in the chain, I have designed a generator that creates a program and a set of units from its own representation. Right now, the program and units are written to disk and then FPC is called on those files. This works fine but is somewhat slow because of disk writing and reading. I was thus wondering if there was a way to send the program and unit contents directly to FPC without writing to disk, and then get FPC to output the final DLL to a memory location as well. The last part is optional, being able to send the program and units without disk access would be a nice thing. I looked around in the documentation for "standard input", "pipe", but apart from ways to access them from within a program, I was not able to find any information. Is this possible? If yes, how should I proceed?

Do you have any proof that writing/reading to the files to disk is an issue? On modern harddisks and OSes the plain I/O speed is normally not the issue when compiling.

Reply by email

Florian Klaempfl
to OBones: I'm currently creating a build chain where FPC is the last step, the one producing a DLL…
OBones wrote 18 lines
Hello all, I'm currently creating a build chain where FPC is the last step, the one producing a DLL for Windows, x86 and x64. In front of it in the chain, I have designed a generator that creates a program and a set of units from its own representation. Right now, the program and units are written to disk and then FPC is called on those files. This works fine but is somewhat slow because of disk writing and reading. I was thus wondering if there was a way to send the program and unit contents directly to FPC without writing to disk, and then get FPC to output the final DLL to a memory location as well. The last part is optional, being able to send the program and units without disk access would be a nice thing. I looked around in the documentation for "standard input", "pipe", but apart from ways to access them from within a program, I was not able to find any information. Is this possible? If yes, how should I proceed?

Do you have any proof that writing/reading to the files to disk is an issue? On modern harddisks and OSes the plain I/O speed is normally not the issue when compiling.

Reply by email

Mattias Gaertner
to Florian Klaempfl
Show quoted text 23 lines
Am 19.04.2012 14:11, schrieb OBones: > Hello all, > > I'm currently creating a build chain where FPC is the last step, the one > producing a DLL for Windows, x86 and x64. > In front of it in the chain, I have designed a generator that creates a > program and a set of units from its own representation. > Right now, the program and units are written to disk and then FPC is > called on those files. > This works fine but is somewhat slow because of disk writing and reading. > I was thus wondering if there was a way to send the program and unit > contents directly to FPC without writing to disk, and then get FPC to > output the final DLL to a memory location as well. The last part is > optional, being able to send the program and units without disk access > would be a nice thing. > I looked around in the documentation for "standard input", "pipe", but > apart from ways to access them from within a program, I was not able to > find any information. > Is this possible? If yes, how should I proceed? Do you have any proof that writing/reading to the files to disk is an issue? On modern harddisks and OSes the plain I/O speed is normally not the issue when compiling.

True.
Lazarus make clean all under Linux 64 bit:

ramdisk:
real 1m8.377s
user 1m2.584s
sys 0m4.212s

disk:
real 1m11.661s
user 1m5.488s
sys 0m4.352s

Mattias

Reply by email

waldo kitty
to OBones: I'm currently creating a build chain where FPC is the last step, the one producing a DLL…
OBones wrote 12 lines
Hello all, I'm currently creating a build chain where FPC is the last step, the one producing a DLL for Windows, x86 and x64. In front of it in the chain, I have designed a generator that creates a program and a set of units from its own representation. Right now, the program and units are written to disk and then FPC is called on those files. This works fine but is somewhat slow because of disk writing and reading. I was thus wondering if there was a way to send the program and unit contents directly to FPC without writing to disk, and then get FPC to output the final DLL to a memory location as well.

i'm old school so please forgive this if is is "out of bounds"... but... how about a RAM disk? do the compiling on that and then copy off to physical media when completed?

Reply by email

waldo kitty
to Mattias Gaertner: On Fri, 20 Apr 2012 18:26:28 +0200 True. Lazarus make clean all under Linux 64 bit:…
Mattias Gaertner wrote
On Fri, 20 Apr 2012 18:26:28 +0200 Florian Klaempfl wrote:

[...]

Show quoted text 16 lines
> Do you have any proof that writing/reading to the files to disk is an > issue? On modern harddisks and OSes the plain I/O speed is normally not > the issue when compiling. True. Lazarus make clean all under Linux 64 bit: ramdisk: real 1m8.377s user 1m2.584s sys 0m4.212s disk: real 1m11.661s user 1m5.488s sys 0m4.352s

AFAIK, linux also has much better caching capabilities, too... at least as compared to winwhatever... in one project i work with, there are magnitudes of difference when sufficiently large cache memory is available... i'm talking about compiling a complete and stripped linux based OS with all associated tools and apps for a network security device aimed at dumpster diving acquired hardware, though... still, when we're talking about 100 minutes of RAM based IO compared to 200+ or more minutes of strictly hardware based IO writes, it makes a huge difference ;)

Reply by email

OBones
to Mattias Gaertner: On Fri, 20 Apr 2012 18:26:28 +0200 True. Lazarus make clean all under Linux 64 bit:…
Mattias Gaertner wrote 23 lines
On Fri, 20 Apr 2012 18:26:28 +0200 Florian Klaempfl wrote: > Am 19.04.2012 14:11, schrieb OBones: >> I looked around in the documentation for "standard input", "pipe", but >> apart from ways to access them from within a program, I was not able to >> find any information. >> Is this possible? If yes, how should I proceed? > Do you have any proof that writing/reading to the files to disk is an > issue? On modern harddisks and OSes the plain I/O speed is normally not > the issue when compiling. True. Lazarus make clean all under Linux 64 bit: ramdisk: real 1m8.377s user 1m2.584s sys 0m4.212s disk: real 1m11.661s user 1m5.488s sys 0m4.352s

Ah but I don't have any doubt regarding Linux performance, problem is that I'm building under Windows where performances are an order of magnitude lower. And because I have to split my generated files so that FPC does not hit the 2GB memory limit (which it does for a 15M source file), the disk performances are getting more critical.

Thanks for the testing anyway.

Reply by email

Jonas Maebe
to OBones

OBones wrote on Mon, 23 Apr 2012:

And because I have to split my generated files so that FPC does not hit the 2GB memory limit (which it does for a 15M source file)

What is the structure of that source code? (a few giant routines, large constant arrays, ... ?)

Jonas

Reply by email

OBones
to Jonas Maebe
Jonas Maebe wrote 8 lines
OBones wrote on Mon, 23 Apr 2012: > And because I have to split my generated files so that FPC does not > hit the 2GB memory limit (which it does for a 15M source file) What is the structure of that source code? (a few giant routines, large constant arrays, ... ?)

Thousands of small routines. Splitting them across multiple units allows to compile, with the "write to disk" issue.

Reply by email

Florian Klaempfl
to OBones
OBones wrote 11 lines
Jonas Maebe wrote: > > OBones wrote on Mon, 23 Apr 2012: > >> And because I have to split my generated files so that FPC does not >> hit the 2GB memory limit (which it does for a 15M source file) > > What is the structure of that source code? (a few giant routines, > large constant arrays, ... ?) Thousands of small routines. Splitting them across multiple units allows to compile, with the "write to disk" issue.

You can use a compiler compiled with EXTDEBUG to get some statistics where the memory goes.

Reply by email