Just say, I have a dynamic array with size=5, and I want to delete elements from the index 0 to 2. Is there a trick (the fastest way) to delete those elements (0 to 2) without moving activities?

I've tried to make the dynamic array just pointing to 3rd element and set a new length for it, but fpc generates an unhandled exception.

var
  a: array of string;
SetLength(a, 5);
a[0] := 'aa'; a[1] := 'bb'; a[2] := 'cc'; a[3] := 'dd'; a[4] := 'ee';
a := @sl[3];       // this is the trick, but it doesn't work.
SetLength(a, 2)

Thank's in advance.

Bihar Anwar wrote

Just say, I have a dynamic array with size=5, and I want to delete elements from the index 0 to 2. Is there a trick (the fastest way) to delete those elements (0 to 2) without moving activities?

You won't do it faster than this:

var
 a: array of string;
 i : integer;
SetLength(a, 5);
a[0] := 'aa'; a[1] := 'bb'; a[2] := 'cc'; a[3] := 'dd'; a[4] := 'ee';
a[0]:=a[3];
a[1]:=a[4];
SetLength(a,2);

thanks,
dmitry

I think the right way to do this is:

a := copy(a, 3, length(a) - 3);

Presumably copy optimizes this adequately.

Henry

----- Original Message ----

Bihar Anwar wrote

Would Move() be faster, instead of Copy()?

I guess so, have you tried to benchmark? If you really need speed, why don't you allocate the memory directly?

Henry

Ok, I'll try to observe the compiled code using debugger and benchmark it if needed. BTW, what do you mean by allocating memory directly? Is there another way to allocate memory for dynamic array besides using SetLength()?

Bihar Anwar wrote

Ok, I'll try to observe the compiled code using debugger and benchmark it if needed. BTW, what do you mean by allocating memory directly? Is there another way to allocate memory for dynamic array besides using SetLength()?

Copy() (from the existing dynamic array).

I've tried to use Move() instead of Copy(). Any objection with the following code?

var
  a: array of string;
SetLength(a, 5);
a[0] := 'aa'; a[1] := 'bb'; a[2] := 'cc'; a[3] := 'dd'; a[4] := 'ee';
System.Move(a[3], a[0], 2 * SizeOf(string) );    // instead of  a := Copy(a, 3, 2);
SetLength(a, 2);

Bihar Anwar wrote

I've tried to use Move() instead of Copy(). Any objection with the following code?

Yes, at first glance without much thinking, I don't think it is safe. Did you think through the consequences of copying reference counted types (ansistring in this case presumably)?

var a: array of string;

shortstring of ansistring?

Hmm, maybe you are lucky and this does work without crashes, because the contents of the ansistrings are constants. If they were variable strings, then then the move would make a[3] and a[0] point to the same string (with reference count 1). The setlength would free it, because it frees the memory of the ansistring in a[3].

Vincent

Bihar Anwar wrote

Ok, I'll try to observe the compiled code using debugger and benchmark it if needed. BTW, what do you mean by allocating memory directly? Is there another way to allocate memory for dynamic array besides using SetLength()?

I meant using allocmem, reallocmem, and friends, but that's tricky and bug-prone. And I can't see that it's going to give a massive speed increase.

Henry

The above works great for me for non-reference counted types; I use it extensively (most common usage is adding an element into the middle of a sorted list. Incidentally, if doing this, it's also good to manage the length intelligently, so it does not need to reserve memory each time it adds one element. anyway......)

For ref-counted types, a couple extra operations are needed in order to keep all the reference counts correct.

Before the move: for all the data that is going to be destroyed, set it to nil (setting an ansistring/dynarray to nil does the same thing as setlength to 0 -- namely, it adjusts the reference count, and frees it if needed)

for i := 0 to 2 do a[i] := nil;
System.Move(a[3], a[0], 2 * SizeOf(ansistring) );

After the move: duplicated data will now have an inaccurate ref-count (e.g. ref count = 1 instead of the actual 2), so we have to use fillbyte to zero out the old refs.

fillbyte (a[2], 3*sizeof(ansistring), 0);
SetLength(a, 2);

Note: you're playing with fire here, be really really careful. If you make the tiniest error, your data will become corrupt and your program will crash, and there will be no warnings or anything to help you fix the horrific mess you've created. And I haven't tested the code above... I've written the same stuff many times, but I always scrutinize it super-carefully and test the living crud out of it.

Consider whether your potential performance increase will be worth hours of debugging -- I have spent those hours!! -- and potential crashes from unauthorized memory accesses.

Cheers,
David

You'll have to finalize the strings you are overwriting.

Henry

Thank you very much Henry, David, and Vincent. I think, I must go back to the basic of reference counting. Special thanks to David for the detail explanation.

Delphi has StringRefCount() function (I'm not aware since what version it was introduced). I just curious, is there a function equivalent to it in FPC? Currently, I just steal a portion of code from System unit:

type
  PAnsiRec = ^TAnsiRec;
  TAnsiRec = packed record
    Ref,
    Len   : SizeInt;
    First : Char;
  end;
const
  FirstOff   = SizeOf(TAnsiRec) - 1;
var
  S: ansistring;
  RefCount: SizeInt;
RefCount := PAnsiRec(S - FirstOff)^.Ref

shouldn't that read e.g.



const FirstOff = SizeOf(TAnsiRec) - SizeOf(Char);

SCNR

Anton

Thanks Anton for your suggestion, I will apply it. I hope FPC developers will include StringRefCount() to the RTL.

By the way, how do I call internal compiler functions directly (maybe using a tricky way) such as system.fpc_ansistr_incr_ref() and system.fpc_ansistr_decr_ref? I couldn't call those kind of functions from my program, I guest it is caused by compilerproc definition.

You must never call these, because their behaviour can be changed or they can be removed altogether at any time.

Jonas

Bihar Anwar wrote

I hope FPC developers will include StringRefCount() to the RTL.

Why would you need that?

thanks,
Dmitry

Thanks Jonas for your advice.

Because there are few certain condition when we need to observe what happened to ansistring/unicodestring variable. In my case now, I need it to learn how reference counting works. For example, using Move() on ref-counted string type is considered danger unless we do it properly, and I need to verify that everythings is correct after such operation.

I need StringRefCount() equivalence only for convenience, instead of writing a special codes/function in my program to achieve this. Well, if Delphi care about this need, why FPC not? :-)

Bihar Anwar wrote

Because there are few certain condition when we need to observe what happened to ansistring/unicodestring variable. In my case now, I need it to learn how reference counting works. For example, using Move() on ref-counted string type is considered danger unless we do it properly, and I need to verify that everythings is correct after such operation.

I'd suggest you to use your own move()-safe types. So you don't need to mess with rtl-level language features (like ref-counted types).

thanks,
dmitry

There are use cases for this.

A while ago I thinking about a string-hash table as a cache for strings.

A text parser would keep track of certain words in the text => pchar to the original text wasn't possible, as the original text could be relocated. So each time a word needed to be remembered it was copy(otext,,a,b).
If the same word needed to be remembered 10 times, then 10 copies would be made.

The idea was to keep existing words in a hash table (as proper strings, so the entry in the hash table causes the ref to be increased by one). Then when a word needed to be remembered, it could be looked up, before making a new copy.

If the hash table detects a ref-count of 1, on any word, it can delete it, because this is it's own reference.

Martin

Hi,

There are use cases for this. A while ago I thinking about a string-hash table as a cache for strings.

I have done that. Then I needed to inspect the reference count for debugging and configuration purposes. I didn't know there is StringRefCount() function so I also counted the string type offsets.

The application itself was an optimized, minimalistic in-memory DB, supporting only string type.
It worked together with VirtualTreeView and clever indexing of linking fields to build a tree structure dynamically. It could keep and browse huge amounts of data without any visible delays.
A laptop with 1 GB memory could handle a big company's PDM (Product Data Management) DB field contents in memory. It means around million records and 2 million relations. The data could be searched and the results always came in 1-2 seconds, no matter if there are 10 or 100000 matches.

I could port it from Delphi to Lazarus and publish a component based on that work. It is made on my ex-employer's company time but the program is dumped and nobody maintains it (or even knows about Delphi). So, nobody would care really.
If only I could find a big tree-data to demonstrate it.

Regards,
Juha