Hi,

I have streams compressed using GZip. I get these streams from WebServices. So, how can I decompress these streams only in memory?

I've tried to use zstream unit, that contains TGZFileStream class. Unfortunately this class works only using files. Then I saved the stream in a file (data.txt) and tryed to use TGZFileStream, but I got an error.

I tried to use Abbrevia to:
http://forum.lazarus.freepascal.org/index.php/topic,26240.msg160989.html?PHPSESSID=j1bh8dkl8p8qj2uqk0bcqqtcb4#msg160989

...but nothing happend. No errors, but the result is empty.

So, I have a stream and I need to decompress this stream and send to another process, just in memory.

Could you help me?

Best regards,
Marcos Douglas

Marcos Douglas wrote

Hi, I have streams compressed using GZip. I get these streams from WebServices. So, how can I decompress these streams only in memory?

Perhaps have a main, and only, unit contains these functions:

function CompressStream(InStream: TStream; OutStream: TStream): Integer;
function ExtractStream(InStream: TStream; OutStream: TStream): Integer;

I think this is what you are looking for. I don't know about file headers. At least it would form a good starting point.

Good luck!

Hi and thanks.

I think you're talking about this lib, right?
http://wiki.freepascal.org/Zlibar

I installed and tried, but didn't work.

See my code:

=== begin ===

procedure TMainForm.Button2Click(Sender: TObject);
var
  Buf,Buf2: TMemoryStream;
begin
  Buf := TMemoryStream.Create;
  Buf2 := TMemoryStream.Create;
  try
    Buf.LoadFromFile('data.txt');
    showmessage(inttostr(ExtractStream(Buf, Buf2))); //<< result is 0, ie, OK
    Buf2.SaveToFile('result.txt');
  finally
    Buf.Free;
    Buf2.Free;
  end;
end;

=== end ===

There is no error, but Buf2 is empty.

Regards,
Marcos Douglas

Hello,

I'm not sure but I think that code at least should look like:

   try
     Buf.LoadFromFile('data.txt');
--->>>>  Buf.Position:=0;
     showmessage(inttostr(ExtractStream(Buf, Buf2)));
     Buf2.SaveToFile('result.txt');

Because I think the stream position is just at the end after LoadFromFile.

Hi,

Nope, still the same, Buf2 is empty. :(

Marcos Douglas

Marcos Douglas wrote

Hi, I have streams compressed using GZip. I get these streams from WebServices. So, how can I decompress these streams only in memory?

[...]

Some time ago I needed something like this, and I found the following article that helped a lot:

http://www.gocher.me/GZIP

BTW I shared it here at list, but I think that it wasn't so useful for more users.

Anyway, in attachment a small example that uncompress (in memory) a small compressed file named "content.tar.gz", it is just the GNU license compressed with Linux tar tool, the sample is going to save the uncompressed content in a file named "content.txt".

HTH

I tried your example, works, using your .gz file... but didn't works using my 'stream'.
Then I open your .gz file in Notepad++ and looks like: ‹ \[sÛȱ~Ÿ:?bJ/–ªhnì\w•J %Q6 ‰RH...

But my file (stream) looks like:
H4sIAAAAAAAEAOy9TXPjSJrneV+z/Q5uedkqs0gl/N3RNrXTIAUp...

So, I think my stream has more than Gzip compress... But I tested my stream on this site: http://www.txtwizard.net/compression

...and worked. Then I thought: maybe I need to DecodeBase64 first... but didn't work.

I have no ideas right now.

Marcos Douglas

Ok, it works now.

For the records, I've used the lib http://www.gocher.me/GZIP (Silvio's tip) and the function Base64Decode from WST framework http://svn.code.sf.net/p/lazarus-ccr/svn/wst/trunk/

For some reason the function DecodeBase64 from Synapse doesn't work. It receives an AnsiString. Instead, the Base64Decode function receives a TByteDynArray.

The TDataStream class doesn't matter...

So, the code below is ugly, but shows how it works now (before the refactoring).

Thank you all for the help.

=== begin ===

procedure TMainForm.Button2Click(Sender: TObject);
var
  Buf,Buf2: TMemoryStream;
  B: TByteDynArray;
begin
  Buf := TMemoryStream.Create;
  Buf2 := TMemoryStream.Create;
  try
    Buf.LoadFromFile('result.txt');
    B := Base64Decode(TDataStream.New(Buf).AsString,
[xoDecodeIgnoreIllegalChar]);
    Buf.SaveToFile('bytes.txt');
    Buf.Clear;
    Buf.WriteBuffer(Pointer(B)^, Length(B));
    if ZUncompressStream(Buf, Buf2) then
      Buf2.SaveToFile('content.txt');
  finally
    Buf.Free;
    Buf2.Free;
  end;
end;

=== end ===

Best regards,
Marcos Douglas

"Marcos Douglas" wrote

Ok, it works now.

Great!

refactoring).

I'm on the phone, but looking this part it seems that base64 unconversion can be done directly in memory too. I can try it after...

2016-09-04 18:08 GMT+02:00 Marcos Douglas :

I have streams compressed using GZip. I get these streams from WebServices. So, how can I decompress these streams only in memory?

It seems you already have a solution, but here's another one:)

In Castle Game Engine, I have a unit CastleZStream that provides TGZFileStream that can compress/decompress in memory. It uses the CastleGzioInternal unit, which is a modified version of Gzio unit from FPC, modified to be able to work with TStream (not just a file). You can take these two files to have it working:

https://github.com/castle-engine/castle-engine/blob/master/src/base/castlegziointernal.pas https://github.com/castle-engine/castle-engine/blob/master/src/base/castlezstream.pas

For base64, the TBase64EncodingStream and TBase64DecodingStream from standard FPC Base64 unit do the job nicely:)

They all (TBase64EncodingStream, TBase64DecodingStream, TGZFileStream) descend from TOwnerStream which is a common useful class for a stream that processes another stream.

Regards,
Michalis

As far as I know these things can be done with all the FPC classes as well ? The implementation in base64 unit can perfectly do everything in memory. And ZUncompressStream can be done with TDecompressionStream ?

Var
   Dest : TFileStream;
   B : TBase64DecodingStream;
   Z : TDecompressionStream;
begin
   Z:=Nil;
   B:=Nil;
   Dest:=Nil;
   try
     B:=TBase64DecodingStream.Create(TFileStream.Create('src',fmOpenRead));
     B.SourceOwner:=True;
     Z:=TDecompressionStream.create(B);
     Z.SourceOwner:=True;
     Dest:=TFileStream.Create('result.txt',fmCreate);
     Dest.CopyFrom(Z,0);
   Finally
     Z.Free;
     Dest.Free;
   end;
end;

Michael.

On Mon, Sep 5, 2016 at 2:50 AM, Michael Van Canneyt wrote:
[...]

Perfect! :-)

Var Dest : TFileStream; B : TBase64DecodingStream; Z : TDecompressionStream;

[...]

silvioprog wrote

I'm on the phone, but looking this part it seems that base64 unconversion can be done directly in memory too. I can try it after...

I'm working directly in memory. The example is using files, but just in example.

Thanks.

Marcos Douglas

Looks good, thank you!

For base64, the TBase64EncodingStream and TBase64DecodingStream from standard FPC Base64 unit do the job nicely:)

I didn't know that FPC had classes for Base64. Well, I didn't search because I've always used Synapse::synacode.
Thanks for this tip.

They all (TBase64EncodingStream, TBase64DecodingStream, TGZFileStream) descend from TOwnerStream which is a common useful class for a stream that processes another stream.

I also didn't know TOwnerStream. Thanks.

Regards,
Marcos Douglas

Michael Van Canneyt wrote

As far as I know these things can be done with all the FPC classes as well ? The implementation in base64 unit can perfectly do everything in memory. And ZUncompressStream can be done with TDecompressionStream ?

Well, if FPC can do the job without 3rd dependencies... I will try. Thank you.

Regards,
Marcos Douglas

No, I got an error:
================================
[Debugger Exception Notification]

Project project1 raised exception class 'Edecompressionerror' with message: data error

In file '.\paszlib\src\zstream.pp' at line 302:
raise Edecompressionerror.create(zerror(err));
================================

Regards,
Marcos Douglas

What is the stream content ? TGZFile or simply compressed ? If it is TGZFile, then the header must be skipped before uncompressing data.

Michael.

Michael Van Canneyt wrote

What is the stream content ? TGZFile or simply compressed ? If it is TGZFile, then the header must be skipped before uncompressing data.

Simply compressed.
The example is using a file, but the real app uses streams in memory.

I can't use TBase64DecodingStream either because it not implements Seek method:

function TBase64DecodingStream.Seek(Offset: Longint; Origin: Word): Longint;
begin
  // TODO: implement Seeking in TBase64DecodingStream
  raise EStreamError.Create('Invalid stream operation');
end;

I'm trying to use like this:

=== begin ===
var
  B: TBase64DecodingStream;
  Src, Dest: TMemoryStream;
begin
  Src := TMemoryStream.Create;
  Dest := TMemoryStream.Create;
  B := TBase64DecodingStream.Create(TFileStream.Create('result.txt',fmOpenRead));
  try
    B.SourceOwner:=True;
    Src.CopyFrom(B,0);
    if ZUncompressStream(Src, Dest) then
      Dest.SaveToFile('content2.txt');
  Finally
    Src.Free;
    Dest.Free;
    B.Free;
  end;
=== end ===

So, for now I'll continue using the first solution but I would like to use FPC sources if possible, in the future.

Regars,
Marcos Douglas

Ok, now I understand how TBase64DecodingStream works. The code below is working.

=== begin ===
var
  Decoder: TBase64DecodingStream;
  F: TFileStream;
  Src, Dest: TMemoryStream;
begin
  F := TFileStream.Create('result.txt', fmOpenRead);
  Decoder := TBase64DecodingStream.Create(F);
  Src := TMemoryStream.Create;
  Dest := TMemoryStream.Create;
  try
    Src.CopyFrom(Decoder, Decoder.Size);
    if ZUncompressStream(Src, Dest) then
      Dest.SaveToFile('content.txt');
  Finally
    F.Free;
    Src.Free;
    Dest.Free;
    Decoder.Free;
  end;
end;
=== end ===

Regards,
Marcos Dougas

Marcos Douglas wrote

I have streams compressed using GZip. I get these streams from WebServices. So, how can I decompress these streams only in memory?

See the tiCompressZLib.pas unit included with tiOPF. It supports Stream, Buffer, String and File compress/decompress functionality using ZLib. I believe GZip and ZLib use the same algorithms.

https://sourceforge.net/p/tiopf/code/ci/tiopf2/tree/Options/tiCompressZLib.pas

Regards,
Graeme

Marcos Douglas wrote

Ok, now I understand how TBase64DecodingStream works.

Don't feel alone. I always get my first implementation of using TBase64Decoding/Encoding wrong. So much so, that I annotated my FCL INF documentation with a working code example. :-)

Regards,
Graeme

Thanks, I will take a look.

Marcos Douglas

:)

When this happens I code a wrapper for these "problems".

Regards,
Marcos Douglas