I'm working on implementing zip64 support and trying to test things by compressing large files or streams using FPC's zip code.

Currently using this to get a huge memorystream filled with the same
data - so it should compress very well:
...
  ContentStream:=TMemoryStream.Create();
  ContentStream.SetSize(1+$FFFFFFFF); //4 bytes+1
  ContentStream.Position:=0;
  Buffer:='A'; //pchar
  for i:=0 to ContentStream.Size-1 do
  begin
    ContentStream.Write(Buffer,1);
  end;
  ContentStream.Position:=0;
  writeln('Buffer created');
  try
    Zipper.Entries.AddFileEntry(ContentStream, ArchiveFile);
    writeln('entry added');
    Zipper.ZipAllFiles;
  finally
    ContentStream.Free;
...

However, I'm sure there must be quicker ways (e.g. less newbie code; using OS functions to quickly create empty memor, or alternatively create a sparse file and zip the file instead of the stream).

Suggestions?

Thanks,
Reinier

Once upon a time, Reinier Olislagers said:

However, I'm sure there must be quicker ways (e.g. less newbie code; using OS functions to quickly create empty memor, or alternatively create a sparse file and zip the file instead of the stream).

Have you tried FillChar(), FillDWord(), ... I not, have a look at http://www.freepascal.org/docs-html/rtl/system/fillchar.html

Actually 4 GB : 4294967295 + 1 = 4294967296 = 4 * 1024 * 1024 * 1024

Now you have 4G write calls, resulting in 4G System.Move calls. Maybe it is faster to create a 1GB datastructure (StringOfChar('A', 1024*1024*1024) and do 4 Write's?

(Untested code!)

Bart

In our previous episode, Reinier Olislagers said:

However, I'm sure there must be quicker ways (e.g. less newbie code; using OS functions to quickly create empty memor, or alternatively create a sparse file and zip the file instead of the stream).

Do not create the data, but generate it in an overriden tstream read() call?

If you can define a function that can generate "count" bytes of data at position n (in this case the function is constant), then you don't have to write it out.

Ken in mind that TStream is an abstraction, not necessarily a block of (RAM or disk) memory.

Arggh. Thanks!
Have even used FillChar yesterday or the day before. Just getting a bit too numb I suppose.

Bart wrote

On 9/13/13, Reinier Olislagers wrote: > ContentStream.SetSize(1+$FFFFFFFF); //4 bytes+1 Actually 4 GB : 4294967295 + 1 = 4294967296 = 4 * 1024 * 1024 * 1024

Yes... but I had trouble counting Fs, so that reminder makes more sense to me ;)

Now you have 4G write calls, resulting in 4G System.Move calls. Maybe it is faster to create a 1GB datastructure (StringOfChar('A', 1024*1024*1024) and do 4 Write's?

Thanks, I'll look into it.

Meanwhile, it turns out (surprisingly ;) ) that the creation time of the 4GB memorystream pales in comparison the compression code takes... Ah well.

Mmm, yes, that sounds very smart, thanks!

Attached a NullStream implementation.

Michael.

Michael Van Canneyt wrote

Attached a NullStream implementation.

Thanks ;)

Thanks a lot; I extended it a bit so it looks a bit more like a TMemoryStream and use it in the tests for zipping. Works nicely.

Oh, in case somebody wants to have a look, I've uploaded a patch for zip64 support ;)
http://bugs.freepascal.org/view.php?id=23533

Thanks to everybody here and on the forum for the help.