As far as the compiler is concerned what’s the difference between clearing an array using a for-loop vs. FillChar? It seems like iterating the array would be slower but what does FillChar do exactly and is it faster? The primary concern here is that the memory originally allocated (using SetLength right?) remains in the same location.

var
  list: array of integer;
SetLength(list, 10);
for i := 0 to high(list) do
  list[i] := 0;
FillChar(list[0], Length(list) * sizeof(integer), 0);

Regards,
Ryan Joseph

"Ryan Joseph" wrote

As far as the compiler is concerned what’s the difference between

clearing an array using a for-loop vs. FillChar? It seems like iterating the array would be slower but what does FillChar do exactly and is it faster? The primary concern here is that the memory originally allocated (using SetLength right?) remains in the same location.

It totally depends on the type. In case of primitive types like integers there is indeed only the performance difference (though if you know that the element size is four FillDWord could be even faster, depending on the implementation in the RTL).
If you have managed types however or class instances the result of the FillChar (or equivalent) would be memory leaks.

Plase also note that after a SetLength the new elements are already 0 (or equivalent).

Regards,
Sven

Yeah, I was concerned with just compiler types or weakly retained classes where I’m just keeping the reference.

Thanks.

Regards,
Ryan Joseph

On Apr 1, 2017, at 2:50 PM, Ryan Joseph wrote: Yeah, I was concerned with just compiler types or weakly retained classes where I’m just keeping the reference.

Another question. Is it more efficient/faster to reallocate a new array of the same size or call FillChar on the existing array?

Regards,
Ryan Joseph

"Ryan Joseph" wrote

> On Apr 1, 2017, at 2:50 PM, Ryan Joseph

> > Yeah, I was concerned with just compiler types or weakly retained

classes where I’m just keeping the reference.

Another question. Is it more efficient/faster to reallocate a new array

of the same size or call FillChar on the existing array?

I think that FillChar should be faster as reallocating would need setting the array to Nil and recreating it (SetLength with the same length won't touch the existing elements) thus loosing any gain of reallocate and then there'd still be the internal FillChar of the array.

Regards,
Sven

I am wondering what the purpose of filling all
array elements with zero's could be.
If I want to discard all elements I would simply delete the whole array (setlength(MyArray,0) ).

But when should it be useful to keep all elements and just overwrite them all with zero's (which is also very time consuming)? If the exisiting values are no longer valid then why not simply delete these elements?
It's the fastest way to get rid of them.
If elements were zero before overwriting them with zero's I cannot even distinguish between original and overwritten elements.

If you wouldn't change the size of the array then merely doing FillChar (if not working with managed types) is more performant then freeing the array and allocating it again (which will also zero out the elements). Normally the performance difference isn't *that* important, but sometimes that difference can make or break something.

Regards,
Sven

Because the array is being iterated and I need to know which values are set. Basically I have a dynamic array I grow to a certain size and this process happens in a loop. The options are to allocate/free the array every cycle or clear memory and allocate the array once.

Regards,
Ryan Joseph

Ryan Joseph wrote

Because the array is being iterated and I need to know which values

are set.

Basically I have a dynamic array I grow to a certain size and this

process happens in a loop.

The options are to allocate/free the array every cycle or clear

memory and allocate the array once.

If you just need to reuse the same array and only need to zero its elements then of course fillchar would be the fastest approach (it saves the memory reallocation step).

But this works only for standard array element types (like integer etc.). Managed types have to be reset element by element.

Jürgen Hestermann wrote

I am wondering what the purpose of filling all array elements with zero's could be. If I want to discard all elements I would simply delete the whole array (setlength(MyArray,0) ).

consider this: when using a string var and writing to a binary file... you reuse the string var for each value written... if you don't clear the string var between fills then the binary will contain "garbage" in the unused positions of the string var... you can see this "garbage" when you view the binary file in hex mode... the "garbage" being older longer string data than the current string data in the var... this var may be a global throwaway string var...

we've seen the above since back in the TP3 days, at least... in some cases, we've been able to extract enough "garbage" to break security on a system... especially when there was enough left to suss out the leading part...

Am 2017-04-01 um 19:42 schrieb

data in the var...

this var may be a global throwaway string var...

This can only happen for older string types where the actual length can differ from the reserved memory (maximum string length). With managed strings, this should never be possible as it would write only the bytes of the current length.

I also don't see how this applies to an array of integers.

some situations call for fixed length records... this same "garbage" effect will be seen in them...

I also don't see how this applies to an array of integers.

it may not... the subject doesn't differentiate... it only says "array" so i provided an example to your query about why this clearing may be desired... i just happened to use an array of char that we loving know as string[255]...

Why is this “of course”? What’s the implementation of FillChar exactly? For all I know it has to iterate over a range of bytes and this could be slower than allocation of memory. Maybe some of the compiler people know these details.

Regards,
Ryan Joseph

Allocating new memory via setlength also clears the memory (+ the overhead of allocating the memory).

Jonas

The question is not about the implementation of FillChar. It's about reallocating a dynamic array as to reallocate it you'd need to do

=== code begin ===

SetLength(arr, 0);
SetLength(arr, newlen);

=== code end ===

And that this is more complex than

=== code begin ===

FillChar(arr, SizeOf(arr[0]) * Length(arr), 0);

=== code end ===

can be easily seen by looking at the implementation of SetLength() in $fpcdir/rtl/inc/dynarr.inc, fpc_dynarray_setlength(). Especially since line 220 (at least in the current revision) contains a call to FillChar(), the whole ordeal can only be *more* complex than FillChar().

Regards,
Sven

That’s good to know about SetLength.

Is it possible use FillChar on a multidimensional arrays?

arr: array of array of array of integer.
SetLength(arr, 3, 3, 3);
FillChar(arr[0], (3*3*3)*sizeof(integer), false);

I’m just getting crashes.

Regards,
Ryan Joseph

That does only work with static arrays as dynamic arrays are in reality managed pointers, thus you override the pointers themselves with your FillChar. Not to mention that you're overwriting the memory behind the outermost array as that only has a size of Length(arr) * SizeOf(Pointer).

Regards,
Sven

You can always use FillChar and its kin on specific 'nested' arrays like this

type
   TIntArray = array of Integer;
   TIntIntArray = array of TIntArray;
   TIntIntIntArray = array of TIntIntArray;
   procedure FillArray(const anArray: TIntIntIntArray; aValue: DWord);
   var
     x, y: integer;
   begin
     for x:=0 to High(anArray) do
       for y:=0 to High(anArray[x]) do
         FillDWord(anArray[x][y][0], Length(anArray[x][y]), aValue);
   end;

Jonas Maebe wrote

Allocating new memory via setlength also clears the memory (+ the overhead of allocating the memory).

Jonas, is it still the case that if SetLength() results in existing data being moved that the original- which might be e.g. an unencrypted password- isn't cleared?

Am 04.04.2017 12:52 schrieb "Mark Morgan Lloyd" <

being moved that the original- which might be e.g. an unencrypted password- isn't cleared?

The reallocation is delegated to the memory manager, thus SetLength() can not know what is really done with the data area.

Regards,
Sven

Doesn’t iterating the array default the purpose of FillChar? The goal was the most efficient way clear the array with zero’s. Even if the array if nested 3 levels deep (anArray[x][y][z]) it should (I hope) be a contiguous block of memory that was allocated (correct me if I wrong please).

Regards,
Ryan Joseph

Mark Morgan Lloyd wrote

On 02/04/17 10:00, Jonas Maebe wrote: > Allocating new memory via setlength also clears the memory (+ the > overhead of allocating the memory). Jonas, is it still the case that if SetLength() results in existing data being moved that the original- which might be e.g. an unencrypted password- isn't cleared?

Only formerly not existing elements are overwritten via fillchar. Already existing data is not changed of course (otherwise you would loose them).
If you have a (dynamic) array of 5 elements and you extend it to 7 elements via Setlength(MyArray,7) then only the last 2 (new) elements are cleared (because otherwise they would contain garbage).

This is (and must be) independend from moving data or just extending the existing array. If the existing elements are moved (because the allocated memory cannot be extended) then they are not cleared (but moved).
Only formerly not existing elements are filled with zeros and they are never moved because they did not exist before the SetLength command.

like this

the most efficient way clear the array with zero’s. Even if the array if nested 3 levels deep (anArray[x][y][z]) it should (I hope) be a contiguous block of memory that was allocated (correct me if I wrong please).

As I wrote that isn't the case. Each dynamic array is allocated independently (and thus you could also resize each subelement independently).

If you want continuous memory areas you need to use static arrays or develop your own dynamic data structure that uses array properties.

Regards,
Sven

Ryan Joseph wrote

Doesn’t iterating the array default the purpose of FillChar? The goal was the most efficient way clear the array with zero’s. Even if the array if nested 3 levels deep (anArray[x][y][z]) it should (I hope) be a contiguous block of memory that was allocated (correct me if I wrong please).

No. If you have a dynamic array of multiple levels it does not generate a continuous block for all levels. Only the last level (if it is an array of integer or other non-managed type) is just that. A (dynamic) array of array if an array of pointers. Only for static arrays all levels are a continuous block.

The first level of a multiple level (dynamic) array is an array of pointers (array of array). So you have a continuous block of pointers on the first level.

Each (!) second level array also is an array of pointers (array of array).

Only the last level is an array of integer (or whatever).

Therefore, for a 3x3x3 array you have 3 pointers on the first level. Each of these pointers (MyArray[x]) again points to an array of 3 pointers. And on the last level (MyArray[x,y]) each pointer points to an array of integer. So you have 3x3=9 continuous blocks of 3 integers which can be located anywhere in memory.

BTW:
Each level of nested dynamic arrays can have individuell size. On a 2-dimensional array

var MyArray : array of array of integer;

you can do:

SetLength(MyArray,3);
SetLength(MyArray[0],2);
SetLength(MyArray[1],3);
SetLength(MyArray[2],4);

So MyArray[0] points to an array of 2 integers,
MyArray[1] points to an array of 3 integers and
MyArray[2] points to an array of 4 integers.

I’m glad I asked because of arrays of pointers is bad news for performance. Does SetLength on a single level dynamic array not even allocate a continuous block of memory? I could use GetMem and array[0..0] but it seems like dynamic arrays should do basically that anyways if they’re not nested.

Regards,
Ryan Joseph

The syntax of dynamic and static arrays are the same although they should differ (because the involved pointers). Therefore it is confusing and misleading.

For dynamic arrays
MyArray[0] should be MyArray^[0] and
MyArray[0,1] should be MyArray^[0]^[1]
and so on...

This would make it possible to distinguish between the pointer (MyArray) and the data it points to (MyArray^).
Also MyArray^[0]^[1] would be a pointer while
MyArray^[0]^[1]^ would be the (static) array of integers.

But as you seldom access the pointers themself (they are managed by the compiler) and because this automatic derefencing of pointers for managed types has been defined decades ago we have to live with this now...

As I understand it accessing continuous blocks of memory in tight loops increases the chance that it will be in cache L1/L2 cache. It’s probably trivial for my use but I’m making some game code which is highly performance sensitive and I’m trying to learn some best practices when writing data types. I watched a couple presentations from a compiler engineer from Google (works on clang and LLVM) that explained how this worked and it’s something worth learning and paying attention to imo.

I was allocating using SetLength(MyArray, 3, 3, 3) for a 3x3x3 matrix for example. Maybe it depends on the memory manager and the state of heap but if you called early in the programs execution it should be allocate all that memory in one block I would think.

Another bottle neck of not using arrays of pointers is you need to perform some math if you want to know which element 2,2,2 is. I’m probably getting in over my head now. :)

Regards,
Ryan Joseph

SetLength() allocates a single block of memory, cause array access is ordinary pointer arithmetic. However if you have an array of array then the first array stores a pointer to each sub array.

E.g. the following would be valid, too:

=== code begin ===

var
  arr: array of array of Integer;
begin
  SetLength(arr, 10, 10);
  SetLength(arr[3], 5);
  arr[6] := Nil;
  SetLength(arr[8], 15);
end.

=== code end ===

Regards,
Sven

“then the first array stores a pointer to each sub array.”

Could you illustrate this is code? I don’t think I’m understanding this exactly like it’s represented in memory. There’s only one “sub array” in this 2x2 array so how does that look in memory?

Regards,
Ryan Joseph

You could use something along these lines instead:

=== code begin ===

program tarrtest;
{$mode objfpc}
{$modeswitch advancedrecords}
type
  generic TTwoDimArray<T> = record
  private
    fData: array of T;
    { Note: Length1 and Length2 are not initialized by default, but you
            could use trunk's management operators for that }
    fLength1,
    fLength2: LongInt;
    function GetElement(Index1, Index2: LongInt): T; inline;
    procedure SetElement(Index1, Index2: LongInt; aValue: T); inline;
  public
    { using SetLength() would lead to us needing to use
      "System.SetLength()" for the array which in turn would complain
      about usage of the static symtable; that's a problem that yet
      needs to be solved inside generics, for non-generics that would
      work however }
    procedure AdjustLength(aLength1, aLength2: LongInt);
    property Length1: LongInt read fLength1;
    property Length2: LongInt read fLength2;
    property Element[Index1, Index2: LongInt]: T read GetElement write
SetElement; default;
  end;

{ TTwoDimArray }

function TTwoDimArray.GetElement(Index1, Index2: LongInt): T;
begin
  { ToDo: Length check }
  Result := fData[Index1 * fLength1 + Index2];
end;
procedure TTwoDimArray.SetElement(Index1, Index2: LongInt; aValue: T);
begin
  { ToDo: Length check }
  fData[Index1 * fLength1 + Index2] := aValue;
end;
procedure TTwoDimArray.AdjustLength(aLength1, aLength2: LongInt);
begin
  SetLength(fData, aLength1 * aLength2);
  fLength1 := aLength1;
  fLength2 := aLength2;
end;
var
  arr: specialize TTwoDimArray<LongInt>;
  i: LongInt;
begin
  arr.AdjustLength(10, 5);
  i := arr[3, 2];
  arr[7, 1] := i;
end.

=== code end ===

Regards,
Sven

Let's look at a smaller array, let's say 3 x 4, then it would look like this:

arr = @arr[0], @arr[1], @arr[2]

arr[0] = 0, 0, 0, 0
arr[1] = 0, 0, 0, 0
arr[2] = 0, 0, 0, 0

Essentially you'd have four arrays, namely the outer array arr which basically contains pointers to the other arrays, and the inner arrays which are each three separate four element arrays. So all four of these arrays could reside at completely different locations of the heap.

Is this clearer?

Regards,
Sven

Ryan Joseph wrote

I’m glad I asked because of arrays of pointers is bad news for performance.

I don't think that you will notice a performance issue with dynamic arrays (though it highly depends on the sizes and levels you use...)

Does SetLength on a single level dynamic array not even allocate a continuous block of memory?

Yes, it does (as explained in all the other mails). A (dynamic) array of integer will be allocated as a single block by SetLength. So if you only have one level of a dynamic array as in

var MyArray : array of integer;

then SetLength(MyArray,1000) will allocate a single block of 1000 integers. But with

var MyArray : array of array of integer;
SetLength(MyArray,1000);

will allocate a single block of 1000 pointers (to an array of integer each). Then, SetLength(MyArray[0],1000) will allocate one (!) block of 1000 integers. SetLength(MyArray[1],1000) will allocate another block of 1000 integers and so on....

[...] SetLength() allocates a single block of memory,

To avoid misunderstanding: SetLength(a,dim1,dim2) allocates one block for the dim1 array and then for each element another block.

Mattias

While your statement regarding allocation might be true you must not forget that a dynamic array consists of a meta data block (length, reference count) that is located directly in front of the data block. So even if the memory blocks would be allocated consecutively then there'd still be the meta data blocks inbetween.

If you already know that your dynamic arrays only have a specific size (for matrices used in games that should usually be the case) then you're better off with using a static array:

=== code begin ===

type
  TMatrix = array[0..2, 0..2, 0..2] of LongInt;

=== code end ===

There you can use FillChar() as much as you want as that is indeed a single memory block containing 9 LongInt elements.

Regards,
Sven

Addendum: If you look at fpc_dynarray_setlength() again then you'll see at line 289 that it's calling itself recursively for nested arrays.

Regards,
Sven

Why 2x2?
SetLength(arr,10,10) creates a 10x10 array.

I am trying to show the memory allocation for the 10x10 array as a "graphic":

arr --> arr[0],arr[1],arr[2],arr[3],arr[4],arr[5],arr[6],arr[7],arr[8],arr[9] | | | | | | | | | | V V V V V V V V V V arr[0,0], . . . . . . . arr[9,0], arr[0,1], . . . . . . . arr[9,1], arr[0,2], . . . . . . . arr[9,2], arr[0,3], arr[9,3],
arr[0,4], arr[9,4],
arr[0,5], arr[9,5],
... ...

arr is a single pointer (that points to arr[0]).
arr[0] to arr[9] are (10) pointers located in a continuous memory block each pointing to arr[0,0],
arr[1,0],
arr[3,0], and so on...
arr[0,0] is a single integer.
arr[0,0] to arr[0,9] are (10) integers located in a continuous memory block. The same applies for
arr[1,0] to arr[1,9],
arr[2,0] to arr[2,9],
arr[3,0] to arr[3,9], and so on.

Yeah after all this talk, I’m going to use array[0..0, 0..0, 0..0] and allocate the memory myself to avoid overhead and confusion. Thanks for explaining everything to me but this time going low level makes the most sense.

Regards,
Ryan Joseph

Thanks that helps. Indeed this is not what I need and I’m not even taking advantage of the resizable elements so I better not use dynamic arrays for my matrix.

Regards,
Ryan Joseph

Ryan Joseph wrote

Yeah after all this talk, I’m going to use array[0..0, 0..0, 0..0] and allocate the memory myself to avoid overhead and confusion. Thanks for explaining everything to me but this time going low level makes the most sense.

Then you lose the ability to turn on range checking in your program, since any access with an index greater than 0 will cause an error.

Jonas

Since fillchar is not a general way to clear an item it almost seems like pascal needs a way to clear things generically, such as Array.clear But I guess this would be reinventing OOP, or Ruby where everything is an object and can have .Something tacked on to it.

I've never been a fan of things like fillchar as it seems like a reinvention of something more like Clear() and mistakes can easily be made when you use complex function like this:

fillchar(something, something * something, 0)

instead of a more simpler:

Array.Clear
or
Clear(array)

You could, put the array inside a class/object I suppose and make a clear method... or use an old pascal object with a clear method on the stack so no free/create is required.

All this seems to be maybe either reinventing GoLang or Ruby though, or possibly python.

To me, fillchar doesn't actually say anything sensible about what the code is doing... "Clear" or "Default" (I think delphi is implementing something like this in later versions) says exactly what the code is doing. FillChar is some obscure old 1970's/80's pascal function that really doesn't tell what you are doing, unless you are an old pascal programmer who knows it from long ago. I'm skeptical of these old ways of doing things.. Imagine a newcomer to pascal who did not program in the 1980's... He sees "fillchar" in the program... Does that code even tell this learner of pascal what is happening? What's a "char" have to do with an array that may not have anything to do with chars?

Just my opinion and, possibly rant. Apologies. Rant not directed at anyone in particular, just I'm skeptical of this whole fillchar thing... Even though I use it myself. Usually in a clear() wrapper of my own, to make the code more clear... Or a setdefault() or default() like function

On Apr 12, 2017, at 8:05 PM, wrote: Since fillchar is not a general way to clear an item it almost seems like pascal needs a way to clear things generically, such as Array.clear But I guess this would be reinventing OOP, or Ruby where everything is an object and can have .Something tacked on to it.

I agree. Dynamic arrays only advantage is they are managed by the compiler but they’re missing obvious functionality like adding/removing elements so they’re usually pretty useless. Why was the functionality of appending an element to the end never added? That’s one the most basic operations for arrays.

Regards,
Ryan Joseph

Am 2017-04-12 um 15:05 schrieb

Since fillchar is not a general way to clear an item

Define "clear". There are many things that can be named "clearing" (see below).

It's true that the wording "char" is not ideal.
Instead you can use FillByte ( http://www.freepascal.org/docs-html/rtl/system/fillbyte.html ) which does the same (it has just another name).

FillByte does what it says: It fills a given memory area with bytes (which you can define). Because of the nature of certain types it can damage i.e. managed types so you have to know what you do (you have to know about the memory usage of the used type).

If you have a static array things are completely different as for dynamic arrays. If you have a static array the memory it occupies is static (never changes) and in one continuous block.
Dynamic arrays are completely different although their syntax is (nearly) the same. So what should Array.Clear do in case of a static and in case of a dynamic array?

Clearing a static array can only mean filling it with bytes. But do they have to be zero in all cases?
Array of integer may require different bytes than array of char. In the end you as the programmer have to decide what is appropiate. Array.Clear would not be able to guess what you intend.

Clearing a dynamic array can mean to free all memory it occupies or fill its used memory with bytes.
Again you have to decide what to do
and which bytes should be used (if you fill everything).

To me, fillchar doesn't actually say anything sensible about what the code is doing...

Well, if you use FillByte (instead of FillChar) then nothing can be clearer. It just fills memory with a chosen byte.

"Clear" or "Default" (I think delphi is implementing something like this in later versions) says exactly what the code is doing.

What exactly does it do when "clearing" a dynamic array? There are multiple ways to "clear" it.

FillChar is some obscure old 1970's/80's pascal function that really doesn't tell what you are doing, unless you are an old pascal programmer who knows it from long ago.

It's true that those who grow up with it know that FillChar is actually a Fillbyte. The first versions of FillChar just excepted the char type as the fill byte.

I'm skeptical of these old ways of doing things.. Imagine a newcomer to pascal who did not program in the 1980's... He sees "fillchar" in the program... Does that code even tell this learner of pascal what is happening? What's a "char" have to do with an array that may not have anything to do with chars?

That applies to all Pascal functions.
You need to lookup what they actually do.
In case of FillChar/FillByte it is quite easy to unterstand.

Much more problematic are things that the compiler does "under the hood" which is often not well documented. If you don't know how the compiler handles dynamic arrays you have a real problem as you cannot predict performance nor do you know exactly how to handle them.
But that has nothing to do with FillChar...

On Apr 12, 2017, at 9:07 PM, Jürgen Hestermann wrote: What exactly does it do when "clearing" a dynamic array? There are multiple ways to "clear" it.

I see your point but there could be a good API to make this clear. SetLength resizes the array in memory and “clear” means setting all values to default() for that type. Not a difficult distinction to remember imo. The whole API is incomplete though so it doesn’t even make sense to add this unless your going to overhaul everything and make a complete list of operations.

Regards,
Ryan Joseph

Ryan Joseph wrote

I agree. Dynamic arrays only advantage is they are managed by the compiler but they’re missing obvious functionality like adding/removing elements so they’re usually pretty useless. Why was the functionality of appending an element to the end never added? That’s one the most basic operations for arrays.

It exists:

SetLength(Array,Length(Array)+1);

You will get an additional element (filled with zeros). Then you can set

Array[High(Array)] := whateveryouwant;

Because dynamic arrays are implemented after Delphi's implementation.

Only recently the new array constructors were implemented. Copy exists.

Adding a pop/push requires compiler magic, and could be implemented; but the question is whether it is worth it, given the plethora of other options at your disposal.

Michael.

Well sure, but this kind of code is exactly what programmers abstract and reuse so it’s a perfect candidate for a function.

SetLength(Array,Length(Array)+1);
Array[High(Array)] := …

In the real world is anyone actually going to type that out every time? I’d bet most of us have an array class we use and ignore dynamic arrays all together because we all need basic operations like “add” and “remove”.

On Apr 12, 2017, at 9:25 PM, Michael Van Canneyt wrote: Adding a pop/push requires compiler magic, and could be implemented; but the question is whether it is worth it, given the plethora of other options at your disposal.

Why magic? It seems silly the operator += doesn’t exist or even a basic operation set like “add”, “remove”, “insert” that every array implementation in every language has. Aren’t those just function around the existing dynamic array implementation in the compiler?

I have no idea if it’s worth it though. I personally don’t use dynamic arrays directly because they’re too bare bones so adding them at all wasn’t worth it for me. However, if the API was more robust I would start using them instead of my own class/record wrappers.

Regards,
Ryan Joseph

Ryan Joseph wrote

SetLength resizes the array in memory and “clear” means setting all values to default() for that type.

"clear" is not very clear to me ;-)
IMO SetLength(array,0) is clearing too.

The function you want should be called "Fill".
For example a function

Array.Fill(DefaultValue);

But what happens to a multi-dimensional (dynamic) array? For example, how to handle this case:

Type MyType = record
               A : Integer;
               B : array of char;
               C : array of Float;
               end; // of record
      ArrayA = array of MyType;
      ArrayB = array of ArrayA.

What would be the DefaultValue for

ArrayB.Fill(DefaultValue);

?
I don't see how this can be achieved in a solid and easy to unterstand Array.Fill routine....

I would think Nil is the only option here, ArrayB is an array of arrays

Michael.

On Apr 12, 2017, at 9:43 PM, Jürgen Hestermann wrote: "clear" is not very clear to me ;-) IMO SetLength(array,0) is clearing too.

SetLength resizes memory so there’s that distinction. The fact there is an API is what’s important because there’s at least a reference point.

Good idea. Any defined function is preferable.

That’s one of the reasons we won’t be seeing this function any time soon. ;)

Regards,
Ryan Joseph

Yes, but that's not what Ryan and others want to achieve. As far as I understand, ArrayB.Fill should fill all the lowest (non-managed) types like Integer, Char and Float. Otherwise it would not be "filling" but "clearing" because some memory allocation is lost.

So you would need multiple DefaultValues but I don't see how this can be done.

Ryan Joseph wrote

Why magic? It seems silly the operator += doesn’t exist

Well, are we using Pascal or C? ;-)

or even a basic operation set like “add”, “remove”, “insert” that every array implementation in every language has. Aren’t those just function around the existing dynamic array implementation in the compiler?

I aggree that these functions could be added.
I always create such functions myself but I need to respect the type of my arrays and rewrite these functions for each new type.
If they were available as standard routines it would avoid these rewrites.

"Ryan Joseph" wrote

> On Apr 12, 2017, at 9:25 PM, Michael Van Canneyt

> > Adding a pop/push requires compiler magic, and could be implemented;

but the question is whether it is worth it, given the plethora of other

> options at your disposal. Why magic? It seems silly the operator += doesn’t exist or even a basic

operation set like “add”, “remove”, “insert” that every array implementation in every language has. Aren’t those just function around the existing dynamic array implementation in the compiler?

Delete() is already supported in trunk, Insert() and the +-operator (or Concat()) are on the near term ToDo list (mainly out of Delphi compatibility, though I definitely agree with their existence).

Regards,
Sven

That is why you need "the magic". The compiler needs to check the type, call some helper routines etc.

Michael.

Yes, I do this. Typing is not much work for me.
My main focus is on *readable* code (effort for writing is of low priority). And the 2 lines show very clear what realy happens.

I can also omit the second line and set a value later while with an Add-function I have to specify a value for the new array element even if I just want to extend the array only. I have finer control about what happens.

I’d bet most of us have an array class we use and ignore dynamic arrays all together because we all need basic operations like “add” and “remove”.

I never use classes.
I like dynamic arrays very much.

Hi Ryan,

In the real world is anyone actually going to type that out every time? I’d bet most of us have an array class we use and ignore dynamic arrays all together because we all need basic operations like “add” and “remove”.

I, for one, have a huge list of array utility functions: http://hg.benibela.de/bbutils/file/tip/bbutils.inc

I use arrays rather than classes, because latter are not reference counted

Bye,
BeniBela

Regarding this code:

SetLength(Array,Length(Array)+1); Array[High(Array)] := …

as I understood from (http://wiki.freepascal.org/Dynamic_array <http://wiki.freepascal.org/Dynamic_array>), SetLength will create a copy of the array and free the memory of the shorter array. In this case, a lot of memory operations and copy operations are performed thus degrading the performances of the code. Then it would be unwise to use such strategy: better allocate the size of the array correctly at the beginning of the run, or resize the array by block, to decrease the frequency of such operation:

SetLength(Array,Length(Array)+N);

Can somebody confirm (or invalidate) this?

Ciao,
Gilles Marcou

Correct.

Then it would be unwise to use such strategy: better allocate the size of the array correctly at the beginning of the run, or resize the array by block, to decrease the frequency of such operation:

The usual solution is to grow exponentially:

SetLength(Array,Length(Array)+Max(4,length(Array) div 4))

Mattias

Why is it copying the array and freeing instead of resizing the existing block (realloc)? That sounds crazy but I don’t know how memory managers work.

Regards,
Ryan Joseph

SetLength will create a copy of the array and free the memory of the shorter array. In this case, a lot of memory operations and copy operations are performed thus degrading the performances of the code. Then it would be unwise to use such strategy: better allocate the size of the array correctly at the beginning of the run, or resize the array by block, to decrease the frequency of such operation:

SetLength() uses reallocation for the memory. So it *might* work without copying though that is not guaranteed.
But in general it's recommended anyway not to do "SetLength(..., ... + 1)" loops as that will lead to fragmentation of the heap.

Regards
Sven

It does not free the elements.
It frees the array.

Mattias

Yes, that's my knowledge about dynamic arrays too. Although it *can* happen that the array is not copied (see answer mail from Sven) in general it will (is forced to) allocate a new (larger) memory chunk and copy the data over to it. This is IMO unavoidable if the existing memory area cannot be extended.

There are 2 strategies against it:

1.) If you already know the correct size at the end of an operation you can simply create the array with the final size in the first place.

2.) If you know that the array will be extended many times but you do not know in prior what the final size is, then you can extend the array in larger slices (16, 32 or whatever). Then, if the final size is reached, you can remove empty elements with SetLength(Array,FinalSize);

But very often the performance degrade of Setlength is overvalued. If the Length variation is done only seldom (compared to other code) then it's not worth thinking about such optimizations.

Also, I often have dynamic arrays of pointers (strings or other managed types). In this case the array elements consists of pointers and only the pointers will be copied but not the data they point to.

"Ryan Joseph" wrote

> On Apr 13, 2017, at 7:08 PM, Mattias Gaertner

> >> as I understood from (http://wiki.freepascal.org/Dynamic_array<

http://wiki.freepascal.org/Dynamic_array>), SetLength will create a copy of the array and free the memory of the shorter array. In this case, a lot of memory operations and copy operations are performed thus degrading the performances of the code.

> > Correct. Why is it copying the array and freeing instead of resizing the existing

block (realloc)? That sounds crazy but I don’t know how memory managers work.

SetLength *does* use a reallocate for this, but since it doesn't give you any guarantee for its success as its the task of the memory manager to deal with this, it's easier to assume that the array is indeed copied.

Regards,
Sven

On Apr 13, 2017, at 10:29 PM, Sven Barth via fpc-pascal wrote: SetLength *does* use a reallocate for this, but since it doesn't give you any guarantee for its success as its the task of the memory manager to deal with this, it's easier to assume that the array is indeed copied.

So the real point here is that ReAllocMem will copy and allocate a new block if the old one can’t be resized, therefore this really isn’t about SetLength as much as the memory manager itself.

Regards,
Ryan Joseph

you any guarantee for its success as its the task of the memory manager to deal with this, it's easier to assume that the array is indeed copied.

> So the real point here is that ReAllocMem will copy and allocate a new

block if the old one can’t be resized, therefore this really isn’t about SetLength as much as the memory manager itself.

Correct. And since the memory manager is changeable and might implement whatever behavior it wants nothing fixed can be done for this.

Regards,
Sven