objpas/classes/classesh.inc

TCustomMemoryStream
     function Read(var Buffer; Count: LongInt): LongInt; override;

Having Read result declared as LongInt is problematic on 64 bit systems with large streams.
Read result MUST be either PtrInt or Int64 but it cannot be LongInt.

Please fix, Thanks

Same problem with TMemoryStream.Write :-(

Since you can only read 2GB (Count is a longint) in a single read operation, it makes no sense to return a 64-bit result.

Note that the operating system only allows 2Gb reads anyway, even on 64-bit systems.

Michael.

Hi Michael,

Thanks but that's not the problem. I can't even write 2bytes to a 2GB memory/file stream.

See: http://bugs.freepascal.org/view.php?id=23284 for attached test case shows simple math exactly like read/write in streams

This is another glitch that I must overcome. My sync app can't handle zip files >=~2GB

The problem is with overflow. FPC raises an exception (as it should). I realize this is going to affect a couple base classes too.

All you demonstrate is that it is possible to create overflows.

There is nothing wrong with the base classes. They work perfectly. I've read/written 4Gb files. Probably you do something with the result which is overflowing your variables.

Michael.

See the below program.

It writes 4Gb to disk and writes the resulting number of bytes, which is larger than integer, but which I store in an Int64.

When I run the program:
araminta: >./testw
Wrote 4294968320 bytes

As expected. With Range checking.

-----------------------------------------------------------------------
{$mode objfpc}{$h+}
program testw;
uses classes;
var
   F : TFileStream;
   // 4 Mb buffer.
   Buf : Array[0..4*1024*1024] of byte;
   I : Integer;
   S : Int64;
  begin
    FillChar(Buf,Sizeof(Buf),' ');
    f:=TFileStream.Create('test.dat',fmCreate);
    try
      For I:=1 to 1024 do
        S:=S+F.Write(Buf,SizeOf(Buf));
    Finally
      F.Free;
    end;
    Writeln('Wrote ',S,' bytes');
end.

On 12-11-06 05:07 PM, Michael Van Canneyt wrote:

Please re-open the issue and I will post an updated program that fails EVERY TIME.

Or just use the attached Widen.lpr... :-)

Works perfectly here. 64-bit linux, 8Gb mem.
Are you trying this on a 32-bit system ?
Because then it will of course fail.

Also, it contains a flaw, since the buffer is not
initialized, and will contain random data.
So the tests at the end will fail and cause an error. But this has nothing to do with range checking.

Michael.

Fails EVERY time here. And in production too. I exclusively use memory streams.

Range Checking / Overflow off line 54 returns -2147483648 when I read only 2 bytes and MyBuffer should be [0,0] since that's what I wrote to the memory stream and shows it was never read into.

Michael Van Canneyt wrote

Also, it contains a flaw, since the buffer is not initialized, and will contain random data. So the tests at the end will fail and cause an error. But this has nothing to do with range checking.

const
     MyBuffer:Array[0..1] of byte=(0,0);

It should be [0,0] in debugger. @ Runtime before I change it. Check it out.

Try this
FSize:=int64(1024*1024*1024)*Factor;

1024*1024*1024 is handled as a longint on a 32bit machine. Multiplied by a byte will still give a longint, which will overflow if multiplied by 2.

It also generates a hint about that:
widen.lpr(44,12) Hint: Converting the operands to "Int64" before doing the multiply could prevent overflow errors.

Jeppe Græsdal Johansen wrote

It also generates a hint about that: widen.lpr(44,12) Hint: Converting the operands to "Int64" before doing the multiply could prevent overflow errors.

Right, keep in mind this example was to simulate the read operation of the TMemoryStream when I traced into the problem. I'm on Ubuntu 12.10 x64. I just re-installed. So this is a real problem.

It is a problem with the implementation of TCustomMemoryStream.Read, I fixed it.

The other streams are unaffected by this error, and no change to the API is needed.

Thanks for persisting and pushing me to dig deeper :-)

Permission to go to sleep now ? It's 1 AM...

Michael.

Thanks for fixing. You're awesome!