Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / fpc-devel / 2024-06

Concatenating huge AnsiStrings

7 messages · 6 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

Bart

Hi,

Possibly not a real-world problem, but some pet-project of mine has to deal with it (at least in theory).

What happens if you try to concatenate 2 huge ansistrings, such that Length(String 1) + Length(String 2) > High(SizeInt)? All this assuming you have memory enough to perform such an operation ;-)

Will this trigger some sort of error, or will it silently ditch all characters beyond High(SizeInt), like concatenating two long shortstrings do if the combined length would be > 255?

Bart

Reply by email

Mattias Gaertner
to Bart
Bart wrote 8 lines
Hi, Possibly not a real-world problem, but some pet-project of mine has to deal with it (at least in theory). What happens if you try to concatenate 2 huge ansistrings, such that Length(String 1) + Length(String 2) > High(SizeInt)? All this assuming you have memory enough to perform such an operation ;-)

The GetMem or ReAllocMem will fail. So EOutOfMemory.

Will this trigger some sort of error, or will it silently ditch all characters beyond High(SizeInt), like concatenating two long shortstrings do if the combined length would be > 255?

Mattias

Reply by email

Marco van de Voort
to Bart: Possibly not a real-world problem, but some pet-project of mine has to deal with it (at…

Op 21-6-2024 om 19:39 schreef Bart via fpc-devel:

Show quoted text 11 lines
Possibly not a real-world problem, but some pet-project of mine has to deal with it (at least in theory). What happens if you try to concatenate 2 huge ansistrings, such that Length(String 1) + Length(String 2) > High(SizeInt)? All this assuming you have memory enough to perform such an operation ;-) Will this trigger some sort of error, or will it silently ditch all characters beyond High(SizeInt), like concatenating two long shortstrings do if the combined length would be > 255?

Probably terminate with a heap out of memory error.

Reply by email

Virgo Pärna
to Marco van de Voort: Op 21-6-2024 om 19:39 schreef Bart via fpc-devel: Probably terminate with a heap out of…
Marco van de Voort wrote
Probably terminate with a heap out of memory error.

Also depends of platform...

program tests;
var
  s, s1, s2: ansistring;
begin
  SetLength(s1, High(int32));
  SetLength(s2, High(int32));
  WriteLn(Length(s1));
  WriteLn(Length(s2));
  s := s1 + s2;
  WriteLn(Length(s));
end.

Works as 64 bit Windows program:
2147483647
2147483647
4294967294

But gets "Runtime error 203 at $00404A01" on 32 bit Windows (out of memory).

I did not actually know, that string length was 64 bits on 64 bit platform (at least on Windows), because even uint worked. uint32 version of same code generated warning at compile time. And uint32 version of same code results in strings with length 0. Because sizeint is signed.

I do not have access to 64 bit Delphi, but Embarcadero RAD Studio Athena documentation claims, that there ansistring length is limitud to "~2^31 characters". And there is no implication, that 64 bit has different maximum.

Virgo Pärna

Reply by email

Sven Barth
to Virgo Pärna

Virgo Pärna via fpc-devel schrieb am Fr.,
28. Juni 2024, 08:41:

Show quoted text 31 lines
On Fri, 21 Jun 2024 20:03:56 +0200, Marco van de Voort via fpc-devel < > wrote: > Probably terminate with a heap out of memory error. Also depends of platform... program tests; var s, s1, s2: ansistring; begin SetLength(s1, High(int32)); SetLength(s2, High(int32)); WriteLn(Length(s1)); WriteLn(Length(s2)); s := s1 + s2; WriteLn(Length(s)); end. Works as 64 bit Windows program: 2147483647 2147483647 4294967294 But gets "Runtime error 203 at $00404A01" on 32 bit Windows (out of memory). I did not actually know, that string length was 64 bits on 64 bit platform (at least on Windows), because even uint worked. uint32 version of same code generated warning at compile time. And uint32 version of same code results in strings with length 0. Because sizeint is signed.

The size of the dynamic string types and of dynamic arrays is SizeInt. Thus it matches whatever the size of the underlying architecture is. Thus a UInt32 can lead to problems (and warnings) on 32-bit systems, but not on 64-bit systems.

Show quoted text 5 lines
I do not have access to 64 bit Delphi, but Embarcadero RAD Studio Athena documentation claims, that there ansistring length is limitud to "~2^31 characters". And there is no implication, that 64 bit has different maximum.

It doesn't matter what Delphi does here. FPC had 64-bit support before Delphi did and back then we picked the rule set that we thought was best. Delphi also decided to follow the LP64 vs LLP64 rules regarding the Integer type (it's 32-bit on 64-bit Windows, but 64-bit on 64-bit macOS and Linux) while we decided to keep the size of Integer at 32-bit on 32- and 64-bit systems.
These are differences that will stay this way.

Regards,
Sven

Reply by email

Michalis Kamburelis
to Sven Barth
Delphi also decided to follow the LP64 vs LLP64 rules regarding the Integer type (it's 32-bit on 64-bit Windows, but 64-bit on 64-bit macOS and Linux) while we decided to keep the size of Integer at 32-bit on 32- and 64-bit systems.

Just a clarification to this statement about Delphi: In Delphi the size is inconsistent for LongInt and LongWord. They are 32-bit on 32-bit platforms and (as a special exception) they are also 32-bit on 64-bit Windows platforms . They are 64-bit everywhere else. Yes, this is a bit crazy (I'm thankful FPC didn't follow this inconsistent rule). For me, in practice this means you should not use LongInt / LongWord for code that should be cross-platform and Delphi-compatible, because LongInt / LongWord size is so weirdly defined that it invites bugs IMHO.

The Integer / Cardinal size remains always 32-bit, in both Delphi and FPC.

See https://castle-engine.io/coding_conventions#no_longint_longword , see our testcase
https://github.com/castle-engine/castle-engine/blob/6164c2645fe522751fe7ff48080d07f88c1df5c4/tests/code/testcases/testcompiler.pas#L115-L116 (that is checked with all FPC and Delphi versions we support, so also Delphi on Linux).

Regards,
Michalis

Reply by email