I am working on moving away from using AnsiString as communications buffer in an old application, so I need to write efficient replacements for certain string functions (Delete, Insert, Copy etc).

Now I am wondering if there is a better way to do the Insert() command than this:

procedure InsertBytes(var Dest: TBytes; Src: TBytes; Index: integer);
var
  Len, LenA: integer;
begin
  Len := Length(Dest);
  LenA := Length(Src);
  if LenA = 0 then exit; //Do nothing
  if Index > Len then Index := Len;  //Make an append instead
  SetLength(Dest, Len + LenA);
  Move(Dest[Index], Dest[Index + LenA], LenA);  //Create space
  Move(Src[0], Dest[Index], LenA);  //Insert data
end;

Ideally the function should be portable between FPC and Delphi XE5...

There is no better way if you only want to use TBytes. If you want less reallocations, you must reserve more memory for the array from the start (so length is a "capacity"), and keep a second count, the count of the number of bytes actually used.

Then you're maybe better off working with a TMemory stream, which does all this for you. Or write a TBuffer class.

Michael.

Michael Van Canneyt wrote

There is no better way if you only want to use TBytes.

Thanks, then I will continue using Move...

Ideally the function should be portable between FPC and Delphi XE5...

You'd only need your own functions for Delphi, FPC's intrinsics such as Insert() can already work with arrays:

var
  b, c: TBytes;
begin
  b:= TBytes.Create(1,2,3);
  c:= TBytes.Create(10,11);
  Insert(c,b,2);

-> b is now [1,2,10,11,3]

"Martok" wrote

> Ideally the function should be portable between FPC and Delphi XE5... You'd only need your own functions for Delphi, FPC's intrinsics such as

Insert()

But only in trunk. And Delphi XE8 and newer also support them.

Regards,
Sven

The fpc wiki should probably be updated?

I cannot find "insert" on this page:
http://wiki.freepascal.org/Array

I cannot find it here either:
https://www.freepascal.org/docs-html/ref/refsu14.html

For the last 10 years or so I've been criticizing occasionally the fact that everyone reinvents their own array algorithms for each array! So it seems that may solve it. In fact this problem goes back to the 1970's and 1980's when arrays were being used.

Please list other functions other than insert that are available? Maybe someone add it to the wiki/docs

Oh, that would explain why it is not in the wiki/docs yet

Only issue I can see, is with all Create's, programmers expect a matching Free.
So that is a bit confusing, but so long as someone remembers arrays are garbage (reference rather) collected...

Insert()

Then update it.

I cannot find it here either: https://www.freepascal.org/docs-html/ref/refsu14.html

The documentation is always for the latest release (currently 3.0.2, soon 3.0.4), but only trunk supports this as well as Delete() (Concat() is not yet implemented however).

Regards,
Sven

Martok wrote

... FPC's intrinsics such as Insert() can already work with arrays:

Nonetheless, IMHO using single-Byte Strings (UTF-8 or RawByte, as a proper "uncoded" string type brand does not exist), seems more convenient, especially, as here we have lazy copy on top of ref counting.

-Michael

AFAIK, that is avaible only in the trunk version of FPC. You need to build the trunk version of the documents by yourself.

Bo Berglund wrote

so I need to write efficient replacements for certain string functions (Delete, Insert, Copy etc).

What I am now up aginst after a few days work is to find how to code the Pos() function for strings...
The comm buffer apparently contains the 2-char string as indication of the last and the existing code uses Pos() to find it.

What about something like:
function PosBytes(SearchItem, Target: TBytes): integer;
begin
  ?????
end;
What could I use inside the body of this function to mimik the string
function Pos (for AnsiStrings)?

Bo Berglund wrote

so I need to write efficient replacements for certain string functions (Delete, Insert, Copy etc).

Why do you think the string function (if using strictly just a single UTF-8 or RawByte branded String type) are not efficient ?

If all string encoding brands in an operation are the same there will be no conversion.

-Michael

Sorry,
I was not clear in my question...

What I meant to say is that I need to convert an existing application that uses strings (AnsiString) as binary data buffers to instead use TBytes buffers. Thus my quest is for functions operating like the convenient string functions but on TBytes arrays.

All of this is because I have found that using AnsiString is triggering data changes when the application is running in certain countries (locales) and processing certain data values...

The application was started back in Delphi7 times when "string" actually meant AnsiString and was a 1-byte per element container.

By efficient I meant that I probably would write inefficient code if I had to create things like the mentioned functions myself and if the compiler did the job it would be more efficient.

functions (Delete, Insert, Copy etc).

You could always use RawByteString or a string with a fixed codepage instead if plain AnsiString.

Just declare a type alias so that you can keep it compatible with older Delphi versions as well.

Regards,
Sven

I asked about this problem over at Embarcadero too, but was flamed for even thinking about using any kind of string for storing binary data. THeyn did suggest RawByteString too, though so I went over the application changing all buffers to RawByteString and it did work. I also tried to set a fixed codepage for the application, but it is hard to check if it actually does help.

The root problem is what happens with string conversions on different locales and some functions could not be easily modified to use RawByteString.
So I decided to bite the bullet and convert the whole unit to TBytes instead. Some 7000 code lines to go over...

It has taken a good many hours now and I still have some intricate problems to solve. It still won't compile without errors.

Just declare a type alias so that you can keep it compatible with older Delphi versions as well.

Do you mean:
type
  TMyString = RawByteString;

Not a fixed codepage for the application, but a string with fixed codepage, e.g. "String(CP_1242)" or so...

Yes.

Regards,
Sven

Bo Berglund wrote

All of this is because I have found that using AnsiString is triggering data changes when the application is running in certaincountries (locales) and processing certain data values...

I am rather sure that this can be avoided.

-Michael

Bo Berglund wrote

I asked about this problem over at Embarcadero too, but was flamed foreven thinking about using any kind of string for storing binary data.

They are silly and defending their completely silly implementation of Code aware strings, forcing UTF-16 for any TStrings based classes.

The fpc implementation (forcing UTF-8 for any TStrings based classes) is slightly less silly.

-Michael

Small correction:

FPC "forces" nothing. It offers a single-byte string TStrings implementation. If you want to use that for Unicode, you indeed have no choice but to use UTF8 or use/write a separate class that uses UTF16.

Michael.

That is why I said that it is less silly ! :)

But it does force TStrings siblings to handle single-byte strings (While Delphi forces TStrings siblings to handle 16-bit-word Strings).

As a huge count of classes in the RTL (and the LCL and MSEGUI and ..., and in user code) are or use TStrings siblings the compatibility issues and certain other problems are due to happen.

(I hope I will not completely banned now, as I already am under monitoring for this kind of posts...)

-Michael

Reading that... All I can think of is the 1000's of Linked-List implementations I had to implement when I was studying Turbo Pascal all that years ago. Delete, Insert and Copy would be so easy to do. :-)

What does TBytes use internally to store its data?

Regards,
Graeme

It is an array. Hence a continuous block of memory.

Michael.