Hi,
I read that we should use TBytes instead of AnsiString in order to implement combined binary/character buffers with automatic memory management. With AnsiString we used setlength() in order to allocate not initialized memory.
TBytes is defined as
"

    TBytes = array of Byte;

"
where setlength() fills the buffer with zeros. What should we use instead?

Thanks, Martin

Because of Delphi compatibility and code cleanness.

To me ("Ansi"-) strings seem a lot more convenient in many cases (e.g. when implementing a kind of protocol on a Byte Stream.

Me too. :-)
Often we need to combine buffer data from binary pieces and character data/strings.

Martin

Martin Schreiber wrote

I read that we should use TBytes instead of AnsiString in order to implement combined binary/character buffers with automatic memory management.

Why ?

To me ("Ansi"-) strings seem a lot more convenient in many cases (e.g. when implementing a kind of protocol on a Byte Stream.

-Michael

This has been sent to the wrong address, sorry Michael.

Or why not let it be as it is in FPC 2.6 and use UnicodeString for visible GUI purpose?

Martin

This is probably because in your testcase the memory is still clean, try with -gh. Additionally this behavior would not be Delphi compatible: "
When you use SetLength, existing characters in the string are preserved, but the contents of any newly allocated space is undefined. "
http://docwiki.embarcadero.com/RADStudio/en/Declaring_and_Initializing_Strings

Martin

Martin Schreiber wrote

Because of Delphi compatibility and code cleanness.

I see: The "Unicode Threat" that makes "String" a moving target, forcing the poor "String" type to be "visible Text" i.e. holding a (somehow) known encoding of its binary content.

Of course I do see both aspects: GUI programmers will hope that "strings" are just printable and thus the name of the "somehow encoded" type should stay "String", while technically, a String is just a decently managed sequence of "something".

Thus to me it's clear that a decent compiler should provide both: one or more types for appropriately encoded text and one or more types for sequences of binary objects. that both should be easily usable and provide fast operation as similar as possible to the legacy String handling.

Often we need to combine buffer data from binary pieces and character data/strings.

Obviously.

So why not define decent types like "ByteString", "WordString", "DWordString", (maybe CharString, PointerString or whatever, too) allowing for the legacy string operations in an appropriately overloaded way ?

-Michael

Same in Delphi?

Martin

For strings SetLength also fills the string with zeros as the following example shows:

=== example begin ===

program setlengthtest;
var
   barr: array of Byte;
   b: Byte;
   s: AnsiString;
   c: Char;
begin
   SetLength(barr, 20);
   for b in barr do
     Write(b, ' ');
   Writeln;
   SetLength(s, 20);
   for c in s do
     Write(Ord(c), ' ');
   Writeln;
end.

=== example end ===

=== output begin ===

PS P:\tests\oneshots> .\setlengthtest.exe
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

=== output end ===

So SetLength is the operation to use in each case.

Regards,
Sven

So using TBytes instead of FPC 2.6 AnsiString as buffer has a performance penalty.
It has a convenience penalty too because TBytes can not be concatenated by "+". Using operator overloading is maybe not Delphi compatible and probably has less performance than the optimized concat*() string compiler procs. There AFAIK also is no convenient way to concatenate TBytes with string constants and variables.

Martin

Martin Schreiber wrote

This has been sent to the wrong address, sorry Michael.

So I'm not the only one who often has this problem when answering to one of Michael's mails ^^ (suggested solution "answer to all" or equivalent solution for your client)

Regards,
Sven

That's because the OS gives you on first time a lot of zeroed mem. SetLength does not zero memory. Try this:

var
  i: Integer;
  s: string;
begin
  for i:=1 to 100000 do begin
    SetLength(s,1000000); // size does not matter for speed
    SetLength(s,1);
  end;
end.

Mattias

Sven Barth wrote

So I'm not the only one who often has this problem when answering to one of Michael's mails

Ooops. Why are my contributions different from others ? I am just using Thunderbird and do "answer to all" when replying to the list (as the list daemons often provide the list as a "cc" rather than a "from"

-Michael

You are right, I checked the code of SetLength for arrays and strings and only arrays use FillChar. Nevertheless there is no other operation you can use for arrays.

Regards,
Sven

Yup, MSEgui has allocuninitedarray() for the purpose. I don't think it is more "clean" or "secure" than the old ansistring method because it depends on compiler internals.

Or pool arrays of byte. (I do this in my Delphi app, where I have a nice generic pool class).

I like handy tools. Using FPC 2.6 AnsiString as buffer is handy.

But I don't think TBookmark is very speed-sensitive.

We don't talk about TBookmark only here. AFAIK all "string as databuffer" is prohibited in future FPC.

Martin

Maybe you should do an "answer to list" yourself? At least the FPC and Lazarus list daemons seem to be configured properly...

Regards,
Sven

In the end I preferred looking at the source of SetLength as written to Martin already ;)

Regards,
Sven

I can't tell whether Delphi also uses FillChar (because I won't look at the source of Delphi's RTL), but there is also no other operation.

Regards,
Sven

In our previous episode, Martin Schreiber said:

If you care, you can do manual management. Or pool arrays of byte. (I do this in my Delphi app, where I have a nice generic pool class).

But I don't think TBookmark is very speed-sensitive.

Sven Barth wrote

Maybe you should do an "answer to list" yourself? At least the FPC and Lazarus list daemons seem to be configured properly...

So you think the additional mail to the original poster is not appropriate.

If so I'll happily do "answer list"

-Michael

You should at least try.

Let's make a test: Answer to this mail once using "answer all" and once "answer list". Then I will answer to you using "answer list" and we'll see where the answers go.

Regards,
Sven

Sven Barth wrote

Let's make a test: Answer to this mail once using "answer all" and once "answer list". Then I will answer to you using "answer list" and we'll see where the answers go.

This is the "reply all" test message for Sven.

Those who are not concerned: please ignore.

-Michael

Sven Barth wrote

Let's make a test: Answer to this mail once using "answer all" and once "answer list". Then I will answer to you using "answer list" and we'll see where the answers go.

This is the "reply to sender" test message for Sven. (_Here_ in fact it goes to the list. There are many other mailing lists that are not usable with "reply to sender". )

Those who are not concerned: please ignore.

-Michael

Sven Barth wrote

Let's make a test: Answer to this mail once using "answer all" and once "answer list". Then I will answer to you using "answer list" and we'll see where the answers go.

This is the "reply to list" test message for Sven. (_Here_ this obviously word as desired. Not sure about other mailing lists.) Lets see if Thunderbird can be configured to show a "reply to List" Button.

Those who are not concerned: please ignore.

-Michael

At least in FPC 2.7.1 a "+" on strings can(!) have a higher performance impact as a custom overloaded operator for an array, because a code page conversion might be done, but also if that would not be the case (e.g. in FPC 2.6) the concat functions basically also boil down to the following code (copied from my mail to Michael on Lazarus list regarding "TByteString"):

=== code begin ===

SetLength(Result, Length(aLeft) + Length(aRight));
if Length(aLeft) > 0 then
   Move(aLeft[0], Result[0], Length(aLeft) * SizeOf(Byte));
if Length(aRight) > 0 then
   Move(aRight[0], Result[Length(aLeft)], Length(aRight) * SizeOf(Byte));

=== code end ===

It's true though that you can't concat arrays using "+" in Delphi, because you can't globally overload operators...

Regards,
Sven

Test answer with "reply to list" ("to" is indeed the list)

Regards,
Sven

So it seems that you should use "reply to sender" as this was the only mail that went to the list with "reply to list"... Now it would be interesting to know why this is the case ^^

Regards,
Sven

This in fact came back through the list.
-Michael

Sven Barth wrote

So it seems that you should use "reply to sender" as this was the only mail that went to the list with "reply to list"... Now it would be interesting to know why this is the case ^^

Seemingly the list daemons can be configured regarding what to use as from and what as cc. Seemingly this is complicated and/or a matter of taste, as the different list work differently.

I do know that with several lists "reply to sender" does not work.

In fact I am quite sure that I more than never by chance did "reply to sender" in this list and the mail only was delivered to the original sender. So it seems to depend on his mailer, as well.

-Michael

And the layout of the variable array record.

[...]

Then please implement such a handy character string compatible automatic memory managed buffer type.
And please don't wait until Delphi has made it or Delphi changed string handling completely again which seems to be possible according a thread with Allen Bauer in Embarcadero forum. And please implement it before AnsiStrings can't be used for the purpose anymore.

Thanks, Martin

Exactly. An automatically memory managed buffer type where binary and character data can be combined in a convenient manner and optimal performance. As FPC 2.6 AnsiString is.

Martin

In our previous episode, Martin Schreiber said:

getmem/freemem ?

> Or pool arrays of byte. (I do > this in my Delphi app, where I have a nice generic pool class). > I like handy tools. Using FPC 2.6 AnsiString as buffer is handy.

IMHO it is an abuse. One could discuss if internal usage is a big problem (I think that is still bad though, but if you switch typing string to explicit ansistring, there is no rush), but I think at least public programming interfaces should be free of such gotchas.

In cases where tbytes behaviour is prohibitive due to performance, one probably shouldn't be using automated types in the first place.

> But I don't think TBookmark is very speed-sensitive. We don't talk about TBookmark only here. AFAIK all "string as databuffer" is prohibited in future FPC.

In the early 1.9.x days, when I ported Delphi code to FPC, half of the difficult bugs were related to this kind of abusing internals, specially the relying on autoconversions (of pchar, arrays of char etc). But then, pchar based items were the only overindexable type in Delphi, and that was a must nack then. Nowadays, other types are overindexable too, or at least can be made too, so there is no reason to continue these practices.

Yes, if it is just one person with complete oversight doing it himself, one can easily get away with it, specially since changes like the D2009 unicode transition don't happen every day. But this kind of hidden assumptions is errorprone if you have multiple persons working on the code, specially if some don't know the ins and outs of the code 100%, and very long term increases the number of spots where maintenance must be done significantly.

There have been such cases (and heated discussions) in the past, e.g. when people abused shortstring, and code stopped working when code was switched to ansistring because characters were accessed that were not <=length(s). Nobody argues that ansistring "should be fixed" for that anymore, and in 5 years nobody will care about this either :-)

In our previous episode, Martin Schreiber said:

If it IS 1-byte character data, ansistring is fairly ok. The whole discussion was about stuffing binary data in a character type.

If you care about the repeated addition stuff, you should probably be check out tstringbuilder. It is said to performance better with string concatenation then ansistring. I never use it though, but that is partially also because currently I don't work much with character data.

Martin Schreiber wrote

Then please implement such a handy character string compatible automatic memory managed buffer type.

For alternative "Data-String" type we of course would need things like TStringlist, as well !

-Michael

Do we already have a TStringBuilder?

Regards,
Sven

In our previous episode, Sven Barth said:

I don't know. As said I don't use it :)

28.02.2012 16:31, Marco van de Voort пишет:

I use something like that in fcl-xml package, not in the form of a class but a record and a set of procedures (see xmlutils.pp: TWideStringBuf, BufAppend(), etc).

Regards,
Sergei

The interface of TStringBuilder is documented rather well in the Embarcadero wiki (it's described as an equivalent implementation of the StringBuilder class of .NET). So I might give it a try once I've finished my last exam for this term on Thursday...

Regards,
Sven