I have a class that has a field,

f_data : array of byte;  // a dynamic array

Within a method of this class (which takes as a parameter "src", another instance of the class) I have the following code:

move (src.f_data, self.f_data, length(self.f_data) * sizeof(byte));

It appears that this move operation is going far beyond the length of self.f_data, and overwriting other data.

Is there something obviously wrong here?

thanks,
David

2010/4/24 David Emerson :

move (src.f_data, self.f_data, length(self.f_data) * sizeof(byte));

I think it should be:

SetLength(self.f_data, length(src.f_data));
move (src.f_data[0], self.f_data[0], length(self.f_data) * sizeof(byte));

Well, the setlength is not necessary, and very nearly prevented me from noticing the real solution here, f_data[0] -- thanks very much, it is working now. (I have this feeling like I have asked this question before. d'oh.)

~David.

David,

I don't see how you are not getting memory leaks by doing a move before making sure that the destination buffer has the memory allocated. You see, in Delphi and FPC, move d/n actually move the memory... It just copies it.

You were warned :-)

No he doesn't get leaks. If at all he would get crashes, by overwriting random memory.

But in this case, it's (almost !) save.
He uses " length(self.f_data)* sizeof(byte)" => so he never copies, more bytes to the target than the target can hold.

He may however copy bytes from behind the source, reading random memory behind the source. He does not modify this random memory, so far so good => as long as the random meory behind the source is accesible to his process. If it's not accessible, then access violation.

Martin

I *am* checking sizes before the move, which is why it surprised me that things were going out of bounds -- the trouble is I forgot the array index [0]

David Emerson schrieb: I *am* checking sizes before the move, which is why it surprised me that things were going out of bounds -- the trouble is I forgot the array index [0]

Yes, that happens quite often. Unfortunately, Borland gave up the clear und predictable context independend Pascal syntax when implementing dynamic data structures. Dynamic arrays are actually *pointers* to the data while static arrays are the data itself. But the pointers of dynamic arrays are dereferenced automatically when used except for fundamental byte functions like move or fillchar. It would have been better to dereference the pointer for these functions too. I don't know why Borland did not do that. Now we have to live with this mess only because of Delphi compatibility.

Simply because dereferencing a dyn. arrays makes no sense because it is more than just a simply data collection.

Dynamic arrays are actually *pointers* to the data while static arrays are the data itself. But the pointers of dynamic arrays are dereferenced automatically when used except for fundamental byte functions like move or fillchar.

This is plainly wrong. When indexing a dyn. array, the compiler generates the correct code to access the data, no more no less.

Florian Klaempfl schrieb: Simply because dereferencing a dyn. arrays makes no sense because it is more than just a simply data collection.

Realy? What else is happening in the background?

This is plainly wrong. When indexing a dyn. array, the compiler generates the correct code to access the data, no more no less.

So what is the difference between 'MyArray[0]' and 'MyArray^'? Don't they mean the same address? Isn't it possible to use 'MayArray^' as a parameter for move or fillchar? I thought so.

Reference counting (also of the stored data), automatic length storage.

> This is plainly wrong. When indexing a dyn. array, the compiler > generates the correct code to access the data, no more no less. So what is the difference between 'MyArray[0]' and 'MyArray^'?

MyArray[0] enforces range checking if it is enabled, further differences, see below.

Don't they mean the same address?

I would expect to get @MyArray[0]-sizeof(pointer)*2 when doing MyArray^ (which is not allowed), because this is the start of the actual array information.

There is: there is no reason why move should be handled differently than other procedures.

The compiler would generate the correct code as you mentioned. Every newbie is struggling with this syntax. I remember struggling with it myself. It is not you mess with move and have no clue about internals, then you get

burned. Period. Proper dyn. array code uses copy(...) instead of move.

This is plainly wrong. When indexing a dyn. array, the compiler generates the correct code to access the data, no more no less.

It would be possible to allow this syntax :
move(data ...
in addition to the current :
move(data[0] ...

There is no fundamental reason why it wouldn't be possible. The compiler would generate the correct code as you mentioned.
Every newbie is struggling with this syntax. I remember struggling with it myself. It is not ,
Juha

But currently move *does* handle dynamic arrays differently than other procedures. You can use an array as parameter for a function or procedure and it does not matter whether it's dynamic or not. So the identifier means the array in both cases. But this does *not* apply to move or fillchar! That's illogical.

If you mess with move

What do you mean with "mess". It's a standard function since decades. The only thing that was "messed up" was the syntax of dynamic arrays.

and have no clue about internals, then you get burned. Period. Proper dyn. array code uses copy(...) instead of move.

That's nonsense. The only thing that is wrong here is that the identifier for an dynamic array means different things dependend from the context.

>> Simply because dereferencing a dyn. arrays makes no sense because >> it is more than just a simply data collection. > Realy? What else is happening in the background? Reference counting (also of the stored data), automatic length storage.

But that has nothing to do with the fact that the identifier "MyArray" should have the same address as "MyArray[0]" so that both should be usable for move or fillchar. The first means the whole array while the latter means the first element only.

> So what is the difference between 'MyArray[0]' and 'MyArray^'? MyArray[0] enforces range checking if it is enabled, further differences, see below.

Still this is not relevant when using MyArray^ instead of MyArray[0]. Both should mean the same address (and that's the only thing that move/fillchar need). In case the used pointer is nil a runtime error could occur when trying to dereference it (as is done for other pointers too).

> Don't they mean the same address? I would expect to get @MyArray[0]-sizeof(pointer)*2 when doing MyArray^ (which is not allowed), because this is the start of the actual array information.

I don't. I think it is the same as for AnsiStrings: The pointer points to the first element and the reference counter is stored at lower addresses. So you should be able to move x bytes from the beginning of MyArray without the need to know the first element number.

The identifier of a dynamic array should either be a pointer (so that it has to be dereferenced with ^ when accessing elements) or it should mean the beginning of the array (starting with the first element) *in all cases* (not only in some as it's now).

Hello FPC-Pascal,

Sunday, April 25, 2010, 5:38:48 PM, you wrote:

JH> But currently move *does* handle dynamic arrays differently JH> than other procedures. You can use an array as parameter for a JH> function or procedure and it does not matter whether it's dynamic JH> or not. So the identifier means the array in both cases. But this JH> does *not* apply to move or fillchar! That's illogical.

I think there is a great difference between normal arrays and dynamic ones, in normal arrays:

A: array [0..1] of integer;
move (a[0],somewhere^,8);

is OK, but in dynamic arrays it should be:

A: array of integer;
SetLength(A,2);
move (a[0],somewhere^,4);
move (a[1],(somewhere+4)^,4);

Because I think dynamic arrays does not garantee that all elements are together in a big chunck. So something like "move
(A,somewhere^,Length(A)*sizeof(AElement))" is nonsense.

It handles it exactly the same as any other function/procedure that takes a untyped/formal "var" parameter. Formal var parameters always take on the address of whatever you pass.

The confusion comes from the fact that a dynamic array, just like ansi/wide/unicodestrings and a Delphi-style classes/interfaces, is an implicit pointer to the actual data. Hence, if you pass a variable of such type to a formal var parameter, you are passing a pointer to this implicit pointer as the parameter value (rather than a pointer to the actual data).

> If you mess with move What do you mean with "mess". It's a standard function since decades.

fillchar() and move() are equivalent to direct assembly level programming as far as the abstraction level is concerned because they ignore all type info. The fact that they ignore all type info is a mess.

Jonas

> But currently move *does* handle dynamic arrays differently than other procedures. It handles it exactly the same as any other function/procedure that takes a untyped/formal "var" parameter. Formal var parameters always take on the address of whatever you pass.

That's not true! There is a lot of ambigouity with dynamic arrays. Although an underlying pointer is used internally you use it as if it was the array this pointer is pointing to in your code. If you first have a static array "MyArray" then convert it to a dynamic array you don't have to change your code (except if you use fillchar and/or move). Why that? What exactly does "MyArray" mean? You can't tell because it is context dependent. That's not in the spirit of Pascal.

The confusion comes from the fact that a dynamic array, just like ansi/wide/unicodestrings and a Delphi-style classes/interfaces, is an implicit pointer to the actual data. Hence, if you pass a variable of such type to a formal var parameter, you are passing a pointer to this implicit pointer as the parameter value (rather than a pointer to the actual data).

True. As I said: There is no common logic what the identifier "MyArray" means if it is a dynamic array (but there is for static arrays).

fillchar() and move() are equivalent to direct assembly level programming as far as the abstraction level is concerned because they ignore all type info. The fact that they ignore all type info is a mess.

It's true that it ignores all *type* info. Still there was no ambigouity about the starting address of a variable you give as parameter until dynamic arrays were introduced (including AnsiStrings).

I think there is a great difference between normal arrays and dynamic ones

Yes, but it should not be like that. If you first declare a static array and then decide to make it a dynamic array then *no* code change should be required. But now it is for move and fillchar.

is OK, but in dynamic arrays it should be: A: array of integer; SetLength(A,2);

Why change the array length here? The array has a length already. What if the array had 10 elements before?

move (a[0],somewhere^,4); move (a[1],(somewhere+4)^,4); Because I think dynamic arrays does not garantee that all elements are together in a big chunck.

I think that this *is* guaranteed.

See below.

You can't tell because it is context dependent. That's not in the spirit of Pascal.

File types are also less or more opaque. As soon as you start using move/fillchar on them without knowing exactly what you do, you're in trouble.

True. As I said: There is no common logic what the identifier "MyArray" means if it is a dynamic array (but there is for static arrays).

It means the whole array (which is an opaque type). No more no less.

The same applies to classes.

Jürgen Hestermann wrote

>> But currently move *does* handle dynamic arrays differently than other procedures. > It handles it exactly the same as any other function/procedure that takes a untyped/formal "var" parameter. Formal var parameters always take on the address of whatever you pass. That's not true!

Well, it is.

There is a lot of ambigouity with dynamic arrays.

There certainly is if you expect them to work like regular arrays. They are a quite different datatype.

Although an underlying pointer is used internally you use it as if it was the array this pointer is pointing to in your code. If you first have a static array "MyArray" then convert it to a dynamic array you don't have to change your code (except if you use fillchar and/or move).

Or any other routine that takes an untyped var (or out, or const) parameter. Or if you assign one array to another and modify elements (assigning dynamic arrays creates shallow copies, and unlike with ansistrings these are not made unique when you change individual elements). Dynamic arrays are different from regular arrays in several respects.

> fillchar() and move() are equivalent to direct assembly level programming as far as the abstraction level is concerned because they ignore all type info. The fact that they ignore all type info is a mess. It's true that it ignores all *type* info.

Type info is quite important when type-deendent implicit dereferencing enters the picture.

Still there was no ambigouity about the starting address of a variable you give as parameter until dynamic arrays were introduced (including AnsiStrings).

And classes. And interfaces (although these don't really point to any data, other than internal stuff).

Jonas

PS: could you correct the clock of your system? Several of your replies have time stamps that predate the mails your are replying to (not this one, but others).

> True. As I said: There is no common logic what the identifier "MyArray" > means if it is a dynamic array (but there is for static arrays). It means the whole array (which is an opaque type). No more no less.

Then it should be possible to use "MyArray" as a parameter for move/fillchar but obviously this is not the case.

> It's true that it ignores all *type* info. Still there was no ambigouity > about the starting address of a variable you give as parameter until > dynamic arrays were introduced (including AnsiStrings). The same applies to classes.

Maybe, but why not simply use the starting address of the array when using "MyArray" as parameter for fillchar/move? It would have avoided a lot of trouble.

No. Because it does not mean the *array's data* but the *array*.

If people don't pretent to be clever and just use copy instead of the low level procedure move, this would avoid *real* trouble.

>>> But currently move *does* handle dynamic arrays differently than other procedures. >> It handles it exactly the same as any other function/procedure that takes a untyped/formal "var" parameter. Formal var parameters always take on the address of whatever you pass. > That's not true! Well, it is.

In the help for fillchar it says:

"Fillchar fills the memory starting at X with Count bytes or characters with value equal to Value."

So what should a user think if he uses the identifier "MyArray" of a dynamic array as parameter? How should he know that *in this case* "MyArray" is taken as the pointer to the array while in all other cases (indexing etc.) it is meant to be the array itself?

> There is a lot of ambigouity with dynamic arrays. There certainly is if you expect them to work like regular arrays. They are a quite different datatype.

At least it should always mean the same independent from context.

> Although an underlying pointer is used internally you use it as if it was the array this pointer is pointing to in your code. If you first have a static array "MyArray" then convert it to a dynamic array you don't have to change your code (except if you use fillchar and/or move). Or any other routine that takes an untyped var (or out, or const) parameter.

In all these cases the meaning of the identifier of a variable should stay the same. Only handling is different.

> Then it should be possible to use "MyArray" as a parameter for > move/fillchar but obviously this is not the case. No. Because it does not mean the *array's data* but the *array*.

So it doesn't mean anything? What exactly is the *array* if not it's data?

Jürgen Hestermann wrote

>> There is a lot of ambigouity with dynamic arrays. > There certainly is if you expect them to work like regular arrays. They are a quite different datatype. At least it should always mean the same independent from context.

It does always mean the same, just like a class instance always means the same. It's just that implicit dereferencing is applied as soon as you try to access an element (again, like with classes).

Jonas

A variable with a certain type where certain operations can be applied. To access the array's data, you've to use [...], to set the length of the data, pass it to setlength etc.

Hello FPC-Pascal,

you wrote

> I think there is a great difference between normal arrays and dynamic ones

JH> Yes, but it should not be like that. If you first declare a JH> static array and then decide to make it a dynamic array then *no* JH> code change should be required. But now it is for move and JH> fillchar.

And many more are not identical:

A: array of integer;
B: array of integer;
setlength(A,1);
A[0]:=12;
B:=A;
A[0]:=13;
writeln(B[0]);

Which values will be printed ? 12 or 13 ?

> is OK, but in dynamic arrays it should be: > A: array of integer; > SetLength(A,2);

JH> Why change the array length here? The array has a length JH> already. What if the array had 10 elements before?

Because if had defined the array in the line just before "SetLengh" and I need to give it a size.

> move (a[0],somewhere^,4); > move (a[1],(somewhere+4)^,4); > Because I think dynamic arrays does not garantee that all elements are > together in a big chunck.

JH> I think that this *is* guaranteed.

In fact I'm quite sure that it is not garanteed in none of both arrays, static and dynamic, but it happends always in static as it is convenient in all terms. In fact I think this code in static arrays is wrong:

A: array [0..1] of byte;
....
move(a,somewhere,2);

It could or could not work in different architectures.

Florian Klaempfl wrote

If you mess with move and have no clue about internals, then you get burned. Period. Proper dyn. array code uses copy(...) instead of move.

I wish there was a type-checked equivalent to what move does. copy creates a new array, and I don't want to do that: I already have a big block of memory reserved in the dynarray. I just want to move some data into that space -- sometimes only a part of it. Is there any alternative?

It would be nice if move would give a compiler warning or note if it receives a pointer-pointer. I guess that would require move doing type-checking... but at compile-time, is there anything wrong with move doing such a type-check and issuing a warning?

~D.

Yes:
a) the compiler has no idea about "move" in particular, so it would give this warning for all pointers passed to formal var/const/out parameters
b) this warning would also be triggered if you'd use move on a regular array of pointers, which would be wrong

Jonas

What nonsense is this? I fear you've simply no clue about the usage, internals and purpose of dyn. arrays so I stop this discussion.

>>> There is a lot of ambigouity with dynamic arrays. >> There certainly is if you expect them to work like regular arrays. They are a quite different datatype. > At least it should always mean the same independent from context. It does always mean the same, just like a class instance always means the same. It's just that implicit dereferencing is applied as soon as you try to access an element (again, like with classes).

That means that identifiers for open arrays are always pointers, 'just' with an automatic dereferencing when using square brackets for indexing? Then it should be possible to dereference such identifiers with ^ but it's not. Why? That's the inconcistency I mean: There is no strict logic behind it.

An identifier should always refer to a memory address/location. If the type of the data at that memory location is a pointer then it should be possible to change the expression to the memory address of the data by dereferencing it with ^. But for dynamic arrays (and AnsiStrings and other "modern" structures) this is no longer the case which leads to a lot of confusion. To me it's a design flaw.

And the help does not tell about the internals (motto: "you don't need to bother about the internals") but if someone follows this advice and it does not work as expected then here in the list all say "don't use it if you don't know about the internals". But then why not tell the internals in the documentation? If I construct a heavily nested data structure with records and pointers to records how do I know that these "special" things like AnsiStrings etc. don't work as all the other types that made the Pascal language so popular. It was clear and easy to understand but now it is no longer.

I dit not follow this thread, but maybe it helps.
Dynamic arrays are located on the heap and are reference counted. The command System.Move will bypass the reference counting. For this reason and to be sure the data is available when you need it use System.Move on stack located arrays an strings only.

A file type is actually also a record and you cannot access the fields without an explict type cast.

Why? That's the inconcistency I mean: There is no strict logic behind it.

Logic is nice but sometimes it makes no sense especially since you're not supposed to access dyn. arrays by ^ becuase it usually breaks the automatic management of dyn. arrays.

I agree that the introduction of managed data types is inconsistent with previous Pascal data types. Perhaps it is a design flaw to have changed the semantics of reference-counted data types. Perhaps not. Nevertheless, it has been this way for many, many years and will not be changed at this point.

Correct. The language is no longer as simple and clear and easy as it was.

The rule now is that you need to know if the data types you use are managed (reference-counted) or not. If they are, you must be aware that lower-level operations such as Move and FillChar will not work the way they did with earlier data types. That is one of the things you trade for the convenience of having some details managed for you when using managed data.

A file type is actually also a record and you cannot access the fields without an explict type cast.

At least a file identifier always means the same memory address (where the record data is stored). If needed (for some special purpose) I *can* typecast it because I know the address of the data (@myfile). But when using the identifier of a dynamic array then this identifier does not always mean the same address of the array but sometimes means the pointer to it. That makes it illogical.

Logic is nice but sometimes it makes no sense especially since you're not supposed to access dyn. arrays by ^ becuase it usually breaks the automatic management of dyn. arrays.

That's something the user should decide. The language should not build up artificial obstacles to obfuscate what is happening in the background. If you only copy the data to somewhere else (which is *not* a dynamic array) then there is no need to bother about reference count etc.

I am not sure whether this now cast in stone forever. Errors *can* be corrected sometimes. ;-)

A simple solution would be to use the address the pointer is pointing to (address of first element) for fillchar/move too. Then it would be consistent and the identifier would mean the array in all cases. Because it was stated that these functions should not be used for dynamic arrays anyway only a few code changes would be required to existing code but it would save generations of programmers the anger they have with it now.

1.) Where are "the usage, internals and purpose of dyn. arrays" documented?

2.) It's easy to say "nonsense" without giving any reason. I would claim the same to what you wrote.

Jürgen Hestermann wrote

Where are "the usage, internals and purpose of dyn. arrays" documented?

I dunno. But I find them incredibly useful.

they have some quirks, and over the years I've been using dynarrays, I've documented these on my own computer -- I even have a dynamic array test program that I periodically use to test for dynarray quirks. They're weird and unpredictable.

However I'll take all the quirks, for the convenience of being able to call setlength (rather than getmem or whatever) -- and moreso, for the even larger convenience of automatic freeing of memory -- including automatically freeing embedded dynarrays and ansistrings! Those two points alone are, to me, worth all the quirks that I have to carefully test and document, and the occasional usage error (e.g. the mistake I made that caused me to start this thread). When I started using dynamic arrays instead of making calls to getmem/freemem, my main project (about 25k LOC now) became substantially simpler and easier to maintain.

If you're curious about some of the quirks you can look at my test program, which includes some notes I made to myself after running other tests.

http://david9.freepgs.com/fpc/dynamic-arrays.pas

But if you don't like them for all their quirks and impurity, then don't use them. I'll be the first to admit that I frequently forget aspects of how they work, and have to refer back to my test program.

Cheers,
David

Jürgen Hestermann wrote

I am not sure whether this now cast in stone forever. Errors *can* be corrected sometimes. ;-)

The behaviour of dynamic arrays will not be changed. At best, one day we may add GPC-compatible array schema types (see http://www.gnu-pascal.de/gpc/Schema-Types.html , the part under "As a GNU Pascal extension, the above can also be written as"). These don't hide anything.

A simple solution would be to use the address the pointer is pointing to (address of first element) for fillchar/move too.

It would break code like this:

type
   tbytedynarr = array of byte;
var
   a: array of tbytedynarr;
begin
   setlength(a,10);
   { initialise all dynamic array elements of a}
   ...
   { now delete one of these elements }
   finalise(a[4]);
   { and compact the array }
   move(a[3],a[4],(length(a)-3)*sizeof(a[0]));
end.

With your change, move() would operate on the contents of a[3] tbytedynarr instead of on the contents of "a" itself.

Jonas

Jonas Maebe wrote

{ now delete one of these elements } finalise(a[4]); { and compact the array } move(a[3],a[4],(length(a)-3)*sizeof(a[0]));

Well, actually:

move(a[5],a[4],(length(a)-5)*sizeof(a[0]));

Jonas

At best, one day we may add GPC-compatible array schema types (see http://www.gnu-pascal.de/gpc/Schema-Types.html, the part under "As a GNU Pascal extension, the above can also be written as"). These don't hide anything.

Yes, that looks good as it does not violate the rule that an identifier should mean the same memory address independent from context.

Yes, that's a problem because a[0] and a[0]^ are not distinguishable. Although I would have expected that special procedures exist to insert and remove array elements so that there is no need to do such things manually which also requires knowledge of the exact handling of dynamic arrays which is not fully documented so you are never sure. For example, in the above example, a[10] still exists and points to the same data as a[9], correct? And how do I *insert* elements? I would need to extend the allocated memory but how to do that without distroying reference counters etc.?

It could also be the other way round and the correct syntax could be at least *allowed* (besides the automatic dereferencing):

A dynamic array would then be stated to be a *pointer* to an array (as it's already the case for move/fillchar etc).

"MyArray" would be (the address of) the pointer and "MyArray^" would be (the address) of the data.
The latter would be the same address as "MyArray[0]" (although it would mean the whole array).

"MyArray^[0]" could be allowed and could still be abreviated to "MyArray[0]" so both could be made syntactical correct (though "MyArray^[0]" would raise an expection if "MyArray" is nil).

Length(MyArray) would be the same as Length(MyArray^).

SizeOf(MyArray) would be pointer size (4 or 8 bytes) as it's now.

SizeOf(MyArray^[0]) and SizeOf(MyArray[0]) would be the size of an array element (dependend on type) as it's now.

SizeOf(MyArray^) would be the size of the allocated raw data (Length(MyArray^)*sizeof(MyArray^[0]) without auxilliary data like reference counter etc.).

I am not sure whether this would be doable or whether this raises new problems within the parser or so. It would allow programmers to work with a clear, logical and unambiguous syntax.

Jürgen Hestermann wrote

Yes, that looks good as it does not violate the rule that an identifier should mean the same memory address independent from context.

As has been explained umpteen times already in this thread: a dynamic array identifier always identifies exactly the same memory address (namely the address of the pointer to the array). It's the "[]" operator that has been overloaded for dynamic arrays to basically mean "^[]", similarly to how a) it behaves for pchars
b) "." for classes means "^." (and also for pointers to records in Delphi mode)

And no, the syntax for dynamic arrays will not be changed nor extended (unless Embarcadero would do so).

This is my final post in this thread.

Jonas

Jürgen Hestermann wrote

Although I would have expected that special procedures exist to insert and remove array elements so that there is no need to do such things manually ...

These must be written on a case-by-case basis, for each type of array element. Perhaps I'll tackle this someday as my first contribution to the fpc codebase.

As for the rest, I have addressed these very issues in the last week...

... which also requires knowledge of the exact handling of dynamic arrays which is not fully documented so you are never sure.

See my test program, http://david9.freepgs.com/fpc/dynamic-arrays.pas also: http://lists.freepascal.org/lists/fpc-pascal/2010-April/024840.html

You can figure out how all this stuff works relatively easily -- if it is so important to you to be able to use system.move on dynamic array contents.

For example, in the above example, a[10] still exists and points to the same data as a[9], correct? And how do I *insert* elements? I would need to extend the allocated memory but how to do that without distroying reference counters etc.?

http://lists.freepascal.org/lists/fpc-pascal/2010-April/024845.html

Of course you need to use setlength to extend allocated memory.

Cheers,
David

I'm glad to hear this and thanks for that. I suggest there are DeleteArrayItems() and ShiftUpArrayItems(). The ShiftUpArrayItems() will just shift up items without resizing the array.

I fear you've simply no clue about the usage, internals and purpose of dyn. arrays so I stop this discussion.

As you always do once someone does not share your point of view, even after you tried so hard to convince him. :P Well, never mind.

Still, dynamic arrays are a bitch, because they only *pretend* to be arrays up to a certain point. Comparing this data type with classes, or even file type (thus, records) is pointless, because those types do not have a counterpart like the dynamic array has in the form of the "static" array.

A class is a class - is a pointer^Wreference type. Always. Ever. In every context. In contrast, an array is an array is - yes, either an array or a pointer. Maybe.

Is it really that hard to admit to the fact, that the design of dynamic arrays makes promises which the abstraction can't keep, rather than neglecting this fact each and every time this discussion comes up?

Yes, of course, a dynamic array is a precise data type you should know how to handle. This would be the end of discussion if it were the only array-like data type. Unfortunately, the point of reference is the old style static array which the dynamic counter-part tries to resemble - and fails to do so.

The dynamic array simply fails the principle of least surprise. No more, no less.

Vinzent.

> I fear you've simply no clue about the usage, > internals and purpose of dyn. arrays so I stop this discussion. As you always do once someone does not share your point of view,

Indeed, because I don't want to waste too much of my "fpc time" with writing useless emails.

The dynamic array simply fails the principle of least surprise. No

more, no less.

If you think so, don't waste time with writing emails, just don't use them. Period.

"Vinzent Höfler" wrote

The dynamic array simply fails the principle of least surprise. No more, no less.

OK. You are correct. Now can we move on?

Doug C.
-----
A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?

Then the problem is the name, even so it has a different name "dynamic array" versus "static array".

They have many difference. And I never heard any one complaining about the other differences:

Size:
Fixed for static
Variable for dynamyc

Initialization
None for static (declare it, and access it)
SetLength for dynamic (access, it before (via []) and get an error)

So why is it a problem, that one is a pointer, and the other not? Because it is easier, or more often overlooked.

Then maybe that's a fault of the documentation, not of the data-type? To me the data-type itself has no it did need
proper learning of what it is supposed to be.

You say "array is an array " and I ask where is that broken?

An array is a collection of data elements, which can be accessed via an index (kill me, if I used the wrong words in this sentence, I am sure you can pick them and turn them against me, but I am equaly sure, you can (willingness assumed) detuct what I tried to express.)

I have never found anything that says, that an array (not a static-array, just an array), always and under any circumstances must ensure that the identifier of the whole array is equal to the first data-element in the array?
I don't even think there is a guarantee that all the data elements are found at consecutive addresses in memory. (which is also required in order to use move)

It may be (or maybe not) that such promises are made for "static array"; but should that mean they must be made for every other type of array, for any future type of array too?

Yes, of course, a dynamic array is a precise data type you should know how to handle. This would be the end of discussion if it were the only array-like data type. Unfortunately, the point of reference is the old style static array which the dynamic counter-part tries to resemble - and fails to do so.

So all similar data types have to stick to the same principles?

1) then what between an
- instance of a class
- object (as in "type foo=object .... end")

2) Why bother having 2 data they must be so equal, then they either both *must* use SetLength, or none of them must.

procedure Foo;
var a: Array of integer
begin
   a[1] := 7;
end;

why does that crash? It doesn't crash with a static array?, and they are both array-like types. According to your words, I would expect the above to work...
And now wait, before some one says that's silly. Why can they not auto-expand, auto-shrink? If I access an element, then setlength is called automatically....

Martin

Hello FPC-Pascal,

Tuesday, May 4, 2010, 9:41:27 PM, you wrote:

M> I have never found anything that says, that an array (not a M> static-array, just an array), always and under any circumstances must M> ensure that the identifier of the whole array is equal to the first M> data-element in the array?
M> I don't even think there is a guarantee that all the data elements are M> found at consecutive addresses in memory. (which is also required in M> order to use move)

The problem is a bunch of literature that many people encumber but are wrong in definition. Some definitions are fine, like the wikipedia one:

"In computer science, an array data structure or simply array is a data structure consisting of a collection of elements (values or variables), each identified by one or more integer indices, stored so that the address of each element can be computed from its index tuple by a simple mathematical formula."

And of course the "pascal gurus" that shows us that to copy an array (static) the proper way is
"move(array1,array2,elements*sizeof(element))" when as far as I know the result of this operation is undefined but luckily works in most architectures. Much better would be "move
(array1[0],array2[0],elements*sizeof(element))" but even in this case the result is undefined, but it is a bit more exact as the programmer is not assuming that the array pointer is the same address as the first element.

So nobody can copy an array accessing its identifier, but at least accessing the first element, assuming that data is contiguous.

So why is it a problem, that one is a pointer, and the other not? Because it is easier, or more often overlooked.

The problem is not that one is a pointer (to an array) and the other one is the array itself. That would be fine if it is told to the programmer that way and if it would be consistent. But then why am I not able to derefference this pointer with MyArray^? Why can I use an index on that pointer? In these cases it does *not* behave like a pointer but like an array. That's what makes it ambiguous.

So at least I would expect that these realy confusing things are clearely explained in the help so that a programmer is not lead to false assumptions. Say that it is a pointer to an array (and do not let the user think it is "an array"!). And say that although it is a pointer you still cannot derefference it but you need to use the first index to get the beginning of the array.

José Mejuto het geskryf:

architectures. Much better would be "move (array1[0],array2[0],elements*sizeof(element))" but even in this case the result is undefined, but it is a bit more exact as the programmer

Now this is a perfect example, of why I love Java (at least what I have seen of it - I am no Java expert). In your example, the developer need to have knowledge of the internal structure of an array before they can use the move() function. Java and good programming practices like OOP states that you should try and hide the internal structure from a client using your structure. This gives you the freedom to change the internal structure without the need to change client code. This is the reason why things like Iterator exists in design patterns.

What if move() and the developer assumes the data is one continuous block of data, when in fact it might not be. [I don't know internal structures of FPC] Seeing that dynamic arrays are a collection of pointers (I think I understood this correctly), each memory block pointed to by each pointer could be in a totally different memory location. So copying a continuous block of memory will result in rubbish data as a result.

--------[ Extract from Java Documentation ]------------

static void arraycopy(Object src, int src_position, Object dst, int dst_position, int length)

Copies an array from the specified source array, beginning at the specified position, to the specified position of the destination array.

usage example:
--------------
  // playerList is existing array
  int newSize = 2 * playerList.length;  // Size of new array.
  Player[] temp = new Player[newSize];  // The new array.
  System.arraycopy(playerList, 0, temp, 0, playerList.length);
---------------------------------------------

Note: No need to know internal data structures like an array item is actually a pointer to the actual data in memory etc... Much easier to work with.

Requirements like having to know the internal structure, simply to be able to use that structure is not a cleaver design practise. But lucky for me, I hardly ever use arrays - I much prefer classes. ;-)

Maybe the solution is as simply as introducing a new function in FPC called arraycopy(), which can then hide the complexity of internal data structures?

As always, just my 2c worth of info. Use it, don't use it - I don't care.

Regards,
- Graeme -

You do not use index on *pointer* but on *dynamic array* (as already explained). As of copying/moving array elements, why don't you use simple for loop for that, because that's what you actually want to do? I have used dynamic arrays for years, and didn't have these problems, because, when I want to say "address of the first element", I write it: "a[low(a)]". I don't *guess* nor I want compiler to *guess*. And thinking that address of dynamic array variable should be address of it's first element is plain *guess*. Unfortunately, wrong in this case.

Move and Fillchar are low-level system functions, and they are perfectly logical, although little hard to understand :-) Just try to create dynamic array of dynamic arrays and used those functions on them, and you will understand.

I, also, agree that FPC is far from simple Pascal language, but that fact is that FPC team needs to: 1) follow
Borland/Inprise/Embercadero/... in their decisions; 2) try to add new language features which will give more power to the language; 3) maintain crossplatform-ness of the compiler (including adding platform-specific features); 4) keep RTL, FCL and other packages in a good and usable shape; ... Not really nice environment to mess around with such low level functions by adding special cases, isn't it? :-)

In computer science, an array data structure or simply array is a data structure consisting of a collection of elements (values or variables) [...]

???
What does this mean? As far as I know, "variable" in CS can have 3 senses, depending on the context:

* Data that can take different values on different runs (ie input or depend on input). This is the math sense (also mine) ( constant which is known at design time).
* Data that can change at runtime ( constant again). Like in Pascal or C.
* Named data in a program. Here "symbol", "link" or "binding" may be better terms.

In any case, I don't understand the above.

Denis
________________________________

vit esse estrany ☣

spir.wikidot.com

From: Aleksa Todorovic on May 5, 2010 3:13:12 PM

As of copying/moving array elements, why don't you use

simple for loop
for that?

Because of performance optimization. TStringlist.Delete() and TStringlist.InsertItem() itself uses Move() instead of such loop.

Say that it is a pointer to an array (and do not let the user think it is "an array"!). And say that although it is a pointer you still cannot derefference it but you need to use the first index to get the beginning of the array.

I agree that if it is said to be "a pointer to an array", dereferencing should eg allow copying the array globally (to another variable of the same kind, also dereferenced: "a2^:=a1^"). Which indeed is wrong. But i cannot find this the the fpc ref.

On another point, maybe because I have a rather shallow C culture, I'm not tempted to use array variables as pointers to element #0, nore to do direct computations on their actual integer values. The [] syntax is here to provide all what I need. (Maybe that's only me.)

Denis
________________________________

vit esse estrany ☣

spir.wikidot.com

Hello FPC-Pascal,

you wrote

> architectures. Much better would be "move > (array1[0],array2[0],elements*sizeof(element))" but even in this case > the result is undefined, but it is a bit more exact as the programmer

GG> Now this is a perfect example, of why I love Java (at least what I have GG> seen of it - I am no Java expert). In your example, the developer need to GG> have knowledge of the internal structure of an array before they can use GG> the move() function. Java and good programming practices like OOP states GG> that you should try and hide the internal structure from a client using GG> your structure. This gives you the freedom to change the internal structure GG> without the need to change client code. This is the reason why things like GG> Iterator exists in design patterns.
GG> What if move() and the developer assumes the data is one continuous block GG> of data, when in fact it might not be. [I don't know internal structures of GG> FPC] Seeing that dynamic arrays are a collection of pointers (I think I GG> understood this correctly), each memory block pointed to by each pointer GG> could be in a totally different memory location. So copying a continuous GG> block of memory will result in rubbish data as a result.

The same applies to Pascal, my previous example is badly designed, but better than the first one. The problem here is try to use "move" as a Pascal way of do things. Move is low level function that should be used to optimize code when you known underalying structure, if you do not known this one you should copy/move element by element:

for j:=Low(ArraySource) to High(ArraySource) do
  TargetArray[j]:=ArraySource[j];

Is the same as complaint about fpc not assembling X86 ASM in a PPC platform :-? Yes, both are assembler but you must use the assembler of the underlaying architecture.

Hello FPC-Pascal,

you wrote

> In computer science, an array data structure or simply array is a > data structure consisting of a collection of elements (values or > variables) [...]

s> ???
s> What does this mean? As far as I know, "variable" in CS can s> have 3 senses, depending on the context:
s> * Data that can take different values on different runs (ie s> input or depend on input). This is the math sense (also mine) ( s> constant which is known at design time).
s> * Data that can change at runtime ( constant again). Like in Pascal or C. s> * Named data in a program. Here "symbol", "link" or "binding" may be better terms. s> In any case, I don't understand the above.

Constants (values) or variable values (variables).

Anyway in the definition I was trying to remark that elements in an array should be accessed using [] operator, any other access method is undefined.

Constants (values) or variable values (variables).

Let's not forget assignable constants that must be typed :-)

Let me rephrase my own words: "That one of the 2 arrays is implemented using a pointer"

Of course you are right it is an array, both of them, and both of them behave like that.

And as also explained in my original mail: An array does not guarantee that the identifier(stand alone) can be used for the first element. So it is pure luck that for some arrays you can use the identifier on it's own, to indicate the first element

Oh and to make another point.

people seem to believe that because on static arrays you can do "move(src, dest, len)", that would mean the identifier represents the whole data of the array?
Because that assumptions fails even for static arrays. If the identifier really represented the whole array (as in the whole data), then why does it not include it's len

As for operators represented by the same char, but having different meanings:
"." on a record or the instance of a class => exactly the same thing => the instance of a class is internally a pointer.
"+" on a pointer: it doesn't just add the number, it adds the size of the pointed-to-type as often as specified
"()" on a method-pointer or procedure pointer: it acts like dereferencing it, after all you access (call) the procedure to which it points; but there is no "^"

So many data types have there own behaviour for various operators....

Well the problem is that it isn;t guranteed. someone could implement a compiler, where it was not a pointer.

I haven't looked at the documentation, but it should probably point out for *all* arrays, that you should not use the stand alone identifier, when really you mean the first element
And IIRC the doc at least contains that assignment between two array, creates a shallow copy => so you could check out what that means.

But as I said, I havent recently looked at the doc. maybe it needs improvement.
Maybe this is better solved, by adding warning to procedures like "move".

but then again, you get the same results, when doing:
procedure x(var a: TIntArraray);

except, no one is probably going to use that, unless they already understood what it means. (I can't thing of a case where it was needed, but then neither a case where it would hurt.

procedure x(var a: TSomeClass);  is useful by the way

Martin

Is this the way one should implement *really* dynamic arrays: in which one can remove or insert elements and still have index and count corrects? Are there other methods than moving all elements of higher index up or down? If no, is the correct method analog to the above loop? (But inside a unique dynarray)? Or would it be better to implement such semantics from static arrays directly?

Denis
________________________________

vit esse estrany ☣

spir.wikidot.com

And as also explained in my original mail: An array does not guarantee that the identifier(stand alone) can be used for the first element.

That's just the root cause of the problem: An idenifier should *always* mean the same address in memory. Otherwise you cannot use such low level functions like move and fillchar (or have to learn it the hard way). And these functions (used rarely in special situations) let you do things much more powerful as for example in Java. Pascal has this in common with C. But if you don't get any information about the internal data structure then you cannot use them. Pascal was always very clear about internal data structures so in general it is a surprise that suddenly *some* data types no longer obey the rule that an identifier means the same address (context independent).

So it is pure luck that for some arrays you can use the identifier on it's own, to indicate the first element

That's not true. Static arrays are well defined and this definition should never change.

people seem to believe that because on static arrays you can do "move(src, dest, len)", that would mean the identifier represents the whole data of the array?

Yes, that's true. You can assign (compatible) arrays directly. Then all elements will be copied.

If the identifier really represented the whole array (as in the whole data), then why does it not include it's len

Well, it *does* include the length. For shortStrings element zero contains the length and will be part of the array.

Yes, all these things have been added later by violating the rule of context independent semantic.

So many data types have there own behaviour for various operators....

They shouldn't. It's a confusion of the programmer and makes everything harder to read/write.

Well the problem is that it isn;t guranteed. someone could implement a compiler, where it was not a pointer.

Then it should be a different data type. The same happened to strings: ShortStrings still behave the same since the beginning of Pascal. Noone would change it's internal data structure. If you need something else, create a new one. But please be consistent!

I haven't looked at the documentation, but it should probably point out for *all* arrays, that you should not use the stand alone identifier, when really you mean the first element

I don't mean the first element but the beginning of the array! This mostly starts with the first element of course. ;-)

Jürgen Hestermann wrote

That's just the root cause of the problem

Another problem is that this thread has become too long and nobody is saying anything new anymore. So please continue it on the fpc-other list.

Thanks,

Jonas
FPC mailing lists admin