Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / fpc-pascal / 2013-06

Placing binary data (resources) in object files?

22 messages · 8 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

Anthony Walter

I am trying to come up with an embedded binary resource system and was considering placing data in object files when I found this question on stack overflow:
http://stackoverflow.com/questions/4158900/embedding-resources-in-exe-using-gcc

I see that it's possible to directly turn any file into an object file and reference the memory of that as a resource in a C program using:

extern char binary_foo_bar_start[];
extern char binary_foo_bar_end[];

My question, is it possible to link extern char in free pascal in a way similar to above? Perhaps something similar to:

var binary_foo_bar_start: pointer; external;
{$L foo.bar.o}

Or some permutation of the above?

Reply by email

Ewald
to Anthony Walter

Once upon a time, Anthony Walter said:

Show quoted text 20 lines
I am trying to come up with an embedded binary resource system and was considering placing data in object files when I found this question on stack overflow: http://stackoverflow.com/questions/4158900/embedding-resources-in-exe-using-gcc I see that it's possible to directly turn any file into an object file and reference the memory of that as a resource in a C program using: extern char binary_foo_bar_start[]; extern char binary_foo_bar_end[]; My question, is it possible to link extern char in free pascal in a way similar to above? Perhaps something similar to: var binary_foo_bar_start: pointer; external; {$L foo.bar.o} Or some permutation of the above?

You can pass an option directly to the link ( add -k--format=binary -k<binary files you wish to embed> -k--format=default to the fpc command line), or you could use objcopy.

These two aproaches aren't very portable though, so it is best you write a script or something that generates code that contains this file. Something like this:

===php code begin===
<?php

function ObjCopy($file, $name, $outputname = 0)
{
    $contents = file_get_contents($file);
    $size = strlen($contents);
    if ($outputname === 0)
        $outputname = "blob_$name.o";
    $invalidsymbolcharacters = array('!', '@', '#', '$', '%', '^', '&',
'*', '(', ')', '_', '+', '-', '=', '[', ']', '{', '}', '\'', '\\', '"',
'|', ';', ':', ',', '.', '/', '<', '>', '?', '`', '~');
    $name = str_replace($invalidsymbolcharacters, '_', $name);
    $data = '';
    for ($cnt=0; $cnt < $size; $cnt++)
        $data.= '0x'.dechex(ord($contents[$cnt])).',';
    $data = substr($data, 0, -1);
    $tmpfile = "#include <cstddef>\n";
    $tmpfile.= "extern \"C\" const size_t _binary_{$name}_size = $size;\n";
    $tmpfile.= "extern \"C\" const char _binary_{$name}_start[] =
{{$data}};\n";
    $tmpfile.= "extern \"C\" const char* _binary_{$name}_end =
&_binary_{$name}_start[0] + $size-1;\n";
    fwrite($handle = popen("gcc -o '$outputname' -c -x c++ - -O3 -Wall
-pipe", 'w'), $tmpfile);
    pclose($handle);
}

?>
===php code end===

Usage: `ObjCopy('binaryinputfile', 'symol name',
'outputobjectfile.o');`. Then add `-koutputobjectfile.o` to the command line of fpc. Next you can define your symbols in pascal as follows:

{$escapedpascalname}_start: array[0..<$length-1>] of cuint8; external
name '_binary_{$escapedname}_start';
{$escapedpascalname}_end: array[<1-$length>."..0] of cuint8; external
name '_binary_{$escapedname}_end';
{$escapedpascalname}_size: array[0..0] of cuint8; external name
'_binary_{$escapedname}_size';

Where {$escapedpascalname} is the name you gave as the second argument to the function above. `$length` is the length of the original data in bytes. To read the size, just use `csize_t(@{$escapedpascalname}_size)`.

Note that the above is a very quick draft (kind of a quickfix for some other project -- but is works) and thus contains errors and lacks error checking. Use at your own risk ;-)

PS: Sorry about the php code there; I know it's not the php mailing list, but my buildtool happens to be written in php ;-)

Ewald

Reply by email

Tomas Hajny
to Ewald
Ewald wrote 29 lines
Once upon a time, Anthony Walter said: > I am trying to come up with an embedded binary resource system and was > considering placing data in object files when I found this question on > stack overflow: > http://stackoverflow.com/questions/4158900/embedding-resources-in-exe-using-gcc > > > I see that it's possible to directly turn any file into an object file > and reference the memory of that as a resource in a C program using: > > extern char binary_foo_bar_start[]; > extern char binary_foo_bar_end[]; > > My question, is it possible to link extern char in free pascal in a > way similar to above? Perhaps something similar to: > > var binary_foo_bar_start: pointer; external; > > {$L foo.bar.o} > > Or some permutation of the above? You can pass an option directly to the link ( add -k--format=binary -k<binary files you wish to embed> -k--format=default to the fpc command line), or you could use objcopy. These two aproaches aren't very portable though, so it is best you write a script or something that generates code that contains this file. Something like this:

.
.

PS: Sorry about the php code there; I know it's not the php mailing list, but my buildtool happens to be written in php ;-)

Well, you can obviously use tool bin2obj created with/for FPC and distributed with FPC releases - see fpcsrc/utils/bin2obj.pp.

Tomas

Reply by email

Ewald
to Tomas Hajny

Once upon a time, Tomas Hajny said:

On Mon, June 3, 2013 18:09, Ewald wrote: Well, you can obviously use tool bin2obj created with/for FPC and distributed with FPC releases - see fpcsrc/utils/bin2obj.pp.

Seriously? I should really start looking at included tools... I've got this feeling that it will save me some time.

Anyway, reinventing the weel was quite fun ;-)

Ewald

Reply by email

Rainer Stratmann
to Anthony Walter: I am trying to come up with an embedded binary resource system and was considering…
you wrote 14 lines
I am trying to come up with an embedded binary resource system and was considering placing data in object files when I found this question on stack overflow: http://stackoverflow.com/questions/4158900/embedding-resources-in-exe-using -gcc I see that it's possible to directly turn any file into an object file and reference the memory of that as a resource in a C program using: extern char binary_foo_bar_start[]; extern char binary_foo_bar_end[]; My question, is it possible to link extern char in free pascal in a way similar to above? Perhaps something similar to:

There was a similar discussion about that.

See topic: 'Including binary data making easy'

The result of this discussion was that you can do this (inconvenient) with existing tools.

And that it is not necessary to make improvements so that the compiler can do (simple) things other compilers already can do for a long time.

Show quoted text 5 lines
var binary_foo_bar_start: pointer; external; {$L foo.bar.o} Or some permutation of the above?

Reply by email

Rainer Stratmann
to Anthony Walter: I am trying to come up with an embedded binary resource system and was considering…
you wrote 20 lines
I am trying to come up with an embedded binary resource system and was considering placing data in object files when I found this question on stack overflow: http://stackoverflow.com/questions/4158900/embedding-resources-in-exe-using -gcc I see that it's possible to directly turn any file into an object file and reference the memory of that as a resource in a C program using: extern char binary_foo_bar_start[]; extern char binary_foo_bar_end[]; My question, is it possible to link extern char in free pascal in a way similar to above? Perhaps something similar to: var binary_foo_bar_start: pointer; external; {$L foo.bar.o} Or some permutation of the above?

For example: (see mentioned topic)

// get access to the data of file.dat
// have 1 executable file with containing all the data
{$ir file.dat mydata} // include resource or similar
// fpc compiler makes this
const
 mydata : array[ 0 .. 4 ] of byte =
 ($45,$44,$63,$76,$55); // file data
// the length can find out with sizeof( mydata );

Reply by email

Anthony Walter
to Rainer Stratmann

Wow, thanks for all the interesting replies. In the mean time here is what I came up with:

ld -r -b binary -o binary.o helloworld.txt

#include <stdint.h>
extern char _binary_helloworld_txt_start[];
extern char _binary_helloworld_txt_end[];
void* helloworld(uintptr_t* i) {
uintptr_t a = (uintptr_t)&_binary_helloworld_txt_start;
uintptr_t b = (uintptr_t)&_binary_helloworld_txt_end;
(*i) = b - a;
return &_binary_helloworld_txt_start;
}

gcc -c -o test.o test.c
ar rcs libtest.a test.o binary.o

program testresources;
type
  TExternResource = function(out Size: UIntPtr): Pointer;
function ResString(Res: TExternResource): string;
var
  P: Pointer;
  I: UIntPtr;
begin
  P := Res(I);
  SetLength(Result, I);
  Move(PChar(P)[0], PChar(Result)[0], I);
end;
function helloworld(out Size: UIntPtr): Pointer; cdecl; external;
{$linklib test}
var
  S: string;
begin
  S := ResString(@helloworld);
  WriteLn(S);
end.

Reply by email

Rainer Stratmann
to Anthony Walter

For me this seems (too) complicated to do an easy thing (including some data).

you wrote 45 lines
Wow, thanks for all the interesting replies. In the mean time here is what I came up with: ld -r -b binary -o binary.o helloworld.txt #include <stdint.h> extern char _binary_helloworld_txt_start[]; extern char _binary_helloworld_txt_end[]; void* helloworld(uintptr_t* i) { uintptr_t a = (uintptr_t)&_binary_helloworld_txt_start; uintptr_t b = (uintptr_t)&_binary_helloworld_txt_end; (*i) = b - a; return &_binary_helloworld_txt_start; } gcc -c -o test.o test.c ar rcs libtest.a test.o binary.o program testresources; type TExternResource = function(out Size: UIntPtr): Pointer; function ResString(Res: TExternResource): string; var P: Pointer; I: UIntPtr; begin P := Res(I); SetLength(Result, I); Move(PChar(P)[0], PChar(Result)[0], I); end; function helloworld(out Size: UIntPtr): Pointer; cdecl; external; {$linklib test} var S: string; begin S := ResString(@helloworld); WriteLn(S); end.

Reply by email

Anthony Walter
to Ewald

I think I'll use Rainer's example:

{$ir file.dat mydata}

I just hope the final exe won't get bloated if resource are included in this manner and never used and also that it does slow down compilation every time unless I add to new resources or change code in the unit containing the resource.

What I basically have is some large jpgs, test scripts, compressed sound files and fonts, and I want them built into my library but discarded if no code references them. That is, if an app doesn't use a skybox (there are no references to the skybox class), then the default skybox jpeg resource is not linked into the exe.

http://www.mapcore.org/topic/2829-second-release-5-skies/ 1024*1024*6 pixel jpgs

Reply by email

Rainer Stratmann
to Anthony Walter
you wrote
I think I'll use Rainer's example: {$ir file.dat mydata}

You can try, but I am not aware that this will work.

It was a suggestion. :-)

I just hope the final exe won't get bloated if resource are included in this manner and never used and also that it does slow down compilation

Why should this slow down compilation.
It is only copy some data from a to b.

Show quoted text 11 lines
every time unless I add to new resources or change code in the unit containing the resource. What I basically have is some large jpgs, test scripts, compressed sound files and fonts, and I want them built into my library but discarded if no code references them. That is, if an app doesn't use a skybox (there are no references to the skybox class), then the default skybox jpeg resource is not linked into the exe. http://www.mapcore.org/topic/2829-second-release-5-skies/ 1024*1024*6 pixel jpgs

Reply by email

Graeme Geldenhuys
to Ewald: Once upon a time, Tomas Hajny said: Seriously? I should really start looking at included…
Ewald wrote
> Well, you can obviously use tool bin2obj created with/for FPC and > > distributed with FPC releases - see fpcsrc/utils/bin2obj.pp. Seriously? I should really start looking at included tools... I've got this feeling that it will save me some time.

That's how fpGUI includes all image and sound binaries into executables for years. I do the same for my commercial apps. fpGUI includes a GUI tool where you can simply drag-n-drop a file(s) onto the app, and it converts that into binary array's or exports it to a file that can easily be added to any project.

Regards,
- Graeme -

Reply by email

Anthony Walter
to Rainer Stratmann: You can try, but I am not aware that this will work. It was a suggestion. :-) Why should…

Okay I found some interesting results after a bit of testing...

When using bin2obj you get a byte of array typed constant which you can include in your sources. The compiler will always include the resource whether it is referenced from code or not. Stripping the app (strip -s -g appname) does not remove the size the constants add to the app even if they aren't referenced by any other code.

Using an untyped constant such as "const Image = {$r image.base64};" still adds to the app size even if it's not referenced, but it can only be stripped out if no units are included which reference access to the constant. This means if a "function A" sets a string using part of the constant, then Image (in the example above) will be embedded and cannot be stripped, even if no code calls "function A".

image.base64 contains 480kb of encoded text
'/9j/4AAQSkZJRgABAQEASABIAAD//gATQ3JlYXRlZCB3.....

What this means is that all string constants and typed constants are take up space in your executable whether your use them or not, which is not ideal IMO. The compiler ought to be able to detect when these constants are not referenced (much like not including the machine code of unused functions) and exclude them from the final app.

Reply by email

Jonas Maebe
to Anthony Walter
Anthony Walter wrote 5 lines
What this means is that all string constants and typed constants are take up space in your executable whether your use them or not, which is not ideal IMO. The compiler ought to be able to detect when these constants are not referenced (much like not including the machine code of unused functions) and exclude them from the final app.

If you turn on "smart linking" (compile with -CX -XX), they will be removed by the linker.

Jonas

Reply by email

Anthony Walter
to Jonas Maebe

Jonas,

It's not happening. I have a 700kb image base64 encoded in a file called orangesky.res. I never use "procedure TestResource;" and yet the exe is 700kb larger. The only way to get rid of that extra 700k is to delete the procedure. The smart linking option or using strip do not remove the 700k.

{ TestResource procedure is not called from any code }
procedure TestResource;
const
  Data = {$i orangesky.res}; // 700kb constant is always built into the app
var
  S: string;
  I: Integer;
begin
  S := Data;
  for I := 1 to 10 do
    WriteLn(S[I * 10]);
end;

Reply by email

Rainer Stratmann
to Anthony Walter: It's not happening. I have a 700kb image base64 encoded in a file called orangesky.res. I…
you wrote 11 lines
Jonas, It's not happening. I have a 700kb image base64 encoded in a file called orangesky.res. I never use "procedure TestResource;" and yet the exe is 700kb larger. The only way to get rid of that extra 700k is to delete the procedure. The smart linking option or using strip do not remove the 700k. { TestResource procedure is not called from any code } procedure TestResource; const Data = {$i orangesky.res}; // 700kb constant is always built into the app

This will include binary data???

Did I miss something?
How can you find out the legth of the included data?

Show quoted text 8 lines
var S: string; I: Integer; begin S := Data; for I := 1 to 10 do WriteLn(S[I * 10]); end;

Reply by email

Rainer Stratmann
to Rainer Stratmann
you wrote 20 lines
On Monday 03 June 2013 23:49:07 you wrote: > Jonas, > > It's not happening. I have a 700kb image base64 encoded in a file called > orangesky.res. I never use "procedure TestResource;" and yet the exe is > 700kb larger. The only way to get rid of that extra 700k is to delete the > procedure. The smart linking option or using strip do not remove the > 700k. > > { TestResource procedure is not called from any code } > procedure TestResource; > const > > Data = {$i orangesky.res}; // 700kb constant is always built into the > app This will include binary data??? Did I miss something? How can you find out the legth of the included data?

I see already. You have to make a resourcefile.

It would be too good if that would work as easy as possible.

Show quoted text 17 lines
> var > > S: string; > I: Integer; > > begin > > S := Data; > for I := 1 to 10 do > > WriteLn(S[I * 10]); > > end; _______________________________________________ fpc-pascal maillist - http://lists.freepascal.org/mailman/listinfo/fpc-pascal

Reply by email

Anthony Walter
to Rainer Stratmann

I came with a nice solution to all this and thought I'd share...

I wrote a tool named makeres. In Lazarus edit the default project configuration adding Compilation | Execute Before | makeres $(CompPath). Then when ever you build/run a project the following happens:

* Two directories are created if they don't already exist, one named raw the other resources.
* Any files in raw get compressed and converted to base64 encoded res files placed in the resources folder
* A unit is automatically maintained for your project named ProjectName.Resources
* The ProjectName.Resources unit contains the following (example resources from raw)

unit ProjectName.Resources;
{$mode delphi}
interface

{doc off}
{ To strip unused resources turn off debug symbols and compile with -CX -XX }

const
  ResLargeFont = {$i resources/large.font.base64};
  ResShaderFrag = {$i resources/shader.frag.base64};
  ResShaderVert = {$i resources/shader.vert.base64};
  ResSkyboxJpeg = {$i resources/skybox.jpg.base64};
{doc on}
implementation
end.

* Then to decompressed and read a resources in your app you call call any of these functions (built into my library)

{ Convert a resource into a memory location }
function ResMemory(constref Resource: string): Pointer;
{ Convert a resource into a stream }
function ResStream(constref Resource: string): TStream;
{ Convert a resource into a string }
function ResString(constref Resource: string): string;

So for example you can say:

Font.LoadFromStream(ResStream(ResLargeFont));
or
Shader.Compile(ResString(ResShaderFrag), shaderFragment);

So in summary, you just drop files into your raw folder, and use the above two lines of code. That's all you have to do.

Reply by email

Anthony Walter
to luiz americo pereira camara

Yes, it will be. I am working on releasing the next version of my open source game library which. It includes many new features including static linking (no external dependencies, lib sdl2 rc1, lib tesselation, lib openssl on windows, full opengles support, android support is getting closer) which all go well linking resources inside your apps (I'll have at least two new instructional videos).

http://baregame.org

The makeres tool will be bundled with it (along with sources) and the Lazarus IDE will use it by default when you start a bare game project.

I expect it to be maybe next week will be when I get the new release out.

Reply by email

Reinier Olislagers
to Anthony Walter: I came with a nice solution to all this and thought I'd share... I wrote a tool named…
Anthony Walter wrote 6 lines
I came with a nice solution to all this and thought I'd share... I wrote a tool named makeres. In Lazarus edit the default project configuration adding Compilation | Execute Before | makeres $(CompPath). Then when ever you build/run a project the following happens:

Interesting. FYI, with the fpcres code it's easy to work with resources cross-platform.
See patch http://bugs.freepascal.org/view.php?id=23533 for a proposal to extend fpcres to a general resource compiler.

Reply by email