I just wanted to post a warning for everybody who wants to (block)write/read dynamic arrays to/from file. All the documentation about dynamic arrays I found only told me what I *can* do with them but none said what I can *not* do. And there was no detailed information about the internal works. That left room for (mis)interpretation and costed me many days of headache. Maybe my experience is of interest for others (and maybe it can be added to the documentation):

If you Blockwrite a dynamic array pointer to file (i.e. because it is part of a large record) and read it back later with Blockread then the pointer value is invalid of course. BUT, if you now try to set it to nil as in

DynArray := nil;

then Free Pascal seems to free the memory where the pointer points to! This creates hard to find access violations later in your program. This cost me days of headache because never in my life I would have expected that a simple assignment with nil would do more than just setting bytes to zero. Instead of the above assignment I had to use

fillchar(DynArray,sizeof(DynArray),#0);

This solved all problems.

That is in fact more or less documented: http://www.freepascal.org/docs-html/ref/refsu15.html#x39-450003.3.1

"As remarked earlier, dynamic arrays are reference counted: if in one of the previous examples A goes out of scope and B does not, then the array is not yet disposed of: the reference count of A (and B) is decreased with 1. As soon as the reference count reaches zero the memory, allocated for the contents of the array, is disposed of."

Setting a dynamic array pointer to nil also decreases the reference count, since it removes a reference to the array data.

It mainly creates memory leaks. If you want to keep an extra reference to a dynamic array's data, simply use another dynamic array.

Jonas

> If you Blockwrite a dynamic array pointer to file (i.e. because it

is part of a large record) and read it back later with Blockread then the pointer value is invalid of course. BUT, if you now try to set it to nil as in

> DynArray := nil; > then Free Pascal seems to free the memory where the pointer points to! That is in fact more or less documented:

http://www.freepascal.org/docs-html/ref/refsu15.html#x39-450003.3.1

"As remarked earlier, dynamic arrays are reference counted: if in one

of the previous examples A goes out of scope and B does not, then the array is not yet disposed of: the reference count of A (and B) is decreased with 1. As soon as the reference count reaches zero the memory, allocated for the contents of the array, is disposed of."

Setting a dynamic array pointer to nil also decreases the reference

count, since it removes a reference to the array data.

I would never expect an assignmet to nil changing anything else than just the value of the pointer. And for (all?) other pointers this expectation is valid.

> This creates hard to find access violations later in your program.

This cost me days of headache because never in my life I would have expected that a simple assignment with nil would do more than just setting bytes to zero. Instead of the above assignment I had to use

> fillchar(DynArray,sizeof(DynArray),#0); > This solved all problems. It mainly creates memory leaks. If you want to keep an extra

reference to a dynamic array's data, simply use another dynamic array.

No. Be aware that my dyn array pointer is part of a larger record structure which I blockwrite/-read to and from file. When I read back the structure the pointer is not valid. It's just the pointer at the time I wrote the file (could be months ago). Now I have to rebuild the stucture again. To do this I have to the pointer to nil so that following calls to SetLength have the correct assumption that no data has been allocated yet.

That expectation is not valid for any reference counted type (be it a dynamic array, ansistring, unicodestring, COM-style interfaces). That's the whole point of reference counted types: they keep track of how many references still exist to the data, and once that reaches zero the data is freed.

No. Be aware that my dyn array pointer is part of a larger record structure which I blockwrite/-read to and from file. When I read back the structure the pointer is not valid. It's just the pointer at the time I wrote the file (could be months ago). Now I have to rebuild the stucture again. To do this I have to the pointer to nil so that following calls to SetLength have the correct assumption that no data has been allocated yet.

The correct way to do so is to call initialize() on the field.

Jonas

> I would never expect an assignmet to nil changing anything else than

just the value of the pointer. And for (all?) other pointers this expectation is valid.

That expectation is not valid for any reference counted type (be it a

dynamic array, ansistring, unicodestring, COM-style interfaces). That's the whole point of reference counted types: they keep track of how many references still exist to the data, and once that reaches zero the data is freed.

But I just assign a value to a pointer, then I expect it to do just that: Assign a nil to the pointer. This should be low level and not create any side effects. Of course, SetLength(0) should take care of references etc. but not a simple assignment. It's ignoring the will of the programmer.

> No. Be aware that my dyn array pointer is part of a larger record

structure which I blockwrite/-read to and from file. When I read back the structure the pointer is not valid. It's just the pointer at the time I wrote the file (could be months ago). Now I have to rebuild the stucture again. To do this I have to the pointer to nil so that following calls to SetLength have the correct assumption that no data has been allocated yet.

The correct way to do so is to call initialize() on the field.

It's not mentioned in the documentation you refered to (http://www.freepascal.org/docs-html/ref/refsu15.html#x39-450003.3.1). And I don't need any other routine because fillchar does the job. The irritation just was, that I expected an assignment to nil doing exactly the same as fillchar(..,#0). And I am still astonished that it does more. What other traps are lurking under the hood? It's hard to program if you have to fight with undocumented "features" instead of concentrating on my own mistakes...

IMO, a dynamic array should never be part of a structure that is passed to BlockWrite in the first place. Not that I use many dynamic arrays in the first place, but to me they pass over the border between simple data types (that are fine for BlockWrite) and managed data types (that have quite a lot of "gotchas"). I lump them in with TObjects in that respect.

Jürgen Hestermann wrote

>> I would never expect an assignmet to nil changing anything else than just the value of the pointer. And for (all?) other pointers this expectation is valid. > That expectation is not valid for any reference counted type (be it a dynamic array, ansistring, unicodestring, COM-style interfaces). That's the whole point of reference counted types: they keep track of how many references still exist to the data, and once that reaches zero the data is freed. But I just assign a value to a pointer,

No, you assign a value to a dynamic array. A dynamic array is not a plain pointer, just like a COM-style interface is not a plain pointer. Assigning nil to a COM-style interface does not set its value simply to nil either. There is also no difference between assigning the constant nil to a dynamic array variable and assigning a dynamic array with length 0 to a dynamic array variable (since empty arrays are also represented by a nil pointer, even if they are stored inside a variable).

> The correct way to do so is to call initialize() on the field. It's not mentioned in the documentation you refered to (http://www.freepascal.org/docs-html/ref/refsu15.html#x39-450003.3.1).

The documentation for the initialize routine indeed seems to be missing.

And I don't need any other routine because fillchar does the job.

Fillchar does the job just like adding inline assembler would do the job. It's a hack.

The irritation just was, that I expected an assignment to nil doing exactly the same as fillchar(..,#0). And I am still astonished that it does more. What other traps are lurking under the hood? It's hard to program if you have to fight with undocumented "features" instead of concentrating on my own mistakes...

If you don't want the compiler to interfere with anything you do, do not use automated types such as ansistring, unicodestring, dynamic array and COM-style interfaces. The whole point of these types is that the compiler will do lots of stuff behind your back, because without that the reference counting cannot work.

Jonas

The problem with all these "managed data types" is, that their internal structure and behaviour is not fully documented which makes it hard to judge whether and how to use them.

I created a recursive record structure to represent a directory branch (files and directories) on disk:

---------------------------------------------------------
type TreeListZType  = ^TreeListType;
     DirContentType = array of TreeListZTyp;
     TreeListType   = record
                      CreationTime,
                      LastAccessTime,
                      LastWriteTime : FileTime;
                      FileSizeHigh,
                      FileSizeLow : LongInt;
                      IsFile  : boolean;
                      DirContent : DirContentType;
                      Name  : AnsiString;
                      end; // of record
---------------------------------------------------------

When storing all the data I Blockwrite each record except the last AnsiString (which I handle extra). When reading it back from file the dynamic array pointer "DirContent" serves as an automatic boolean because it tells me, whether the array was empty (=nil) or not. When not empty, I have to read back the number of elements of the array, otherwise nothing has to be read. Before doing so, I have to clear the old pointer so that SetLength works correctly. The problem was, that it never came into my mind that assigning nil will do more than just assigning a value. I did this just to prepare the pointer for the later use with SetLength.

If you don't want the compiler to interfere with anything you do, do

not use automated types such as ansistring, unicodestring, dynamic array and COM-style interfaces. The whole point of these types is that the compiler will do lots of stuff behind your back, because without that the reference counting cannot work.

You are missing the whole point: It is just the problem that some ("automated"?) data types are not fully documented! How do I know which type behaves how? Do I have to guess it (as I did now)? You are always saying that you cannot do this and that with these and other types but there is no clear documentation what exactly you can do and what you can *not* do and what the compiler does in the background.

If all this knowledge would be visible *before* trapping into the pitfalls (after painful debug sessions) then the power of these types could be very useful. But now it is blind flying.

Jürgen Hestermann wrote

Jonas Maebe schrieb: > If you don't want the compiler to interfere with anything you do, do not use automated types such as ansistring, unicodestring, dynamic array and COM-style interfaces. The whole point of these types is that the compiler will do lots of stuff behind your back, because without that the reference counting cannot work. You are missing the whole point: It is just the problem that some ("automated"?) data types are not fully documented! How do I know which type behaves how? Do I have to guess it (as I did now)?

There are probably always things that can be added to the documentation, and you're welcome to file bug reports to suggest that. However, in this case it appears that you were not fully aware of the documentation on dynamic arrays that does exist, because it does say that assigning a new value to a dynamic array variable will decrease its current reference count.

You are always saying that you cannot do this and that with these and other types but there is no clear documentation what exactly you can do and what you can *not* do and what the compiler does in the background.

What the compiler does exactly in the background cannot be documented, because it is implementation-dependent and can change at any time, or differ on different platforms.

Regarding what you can and cannot do: you cannot do anything that makes it impossible for the compiler to see that you are taking a copy of or changing the value of an automated/reference counted type. That includes fillchar, move, blockread, and any other routine that operates on untyped "var" parameters, since the type is lost in that case. Similarly, explicit typecasts (e.g., to pointer) etc and assembler will have the same result.

The compiler can however not give an error if you do that anyway, because internal helpers for those types in the RTL require such functionality to perform the actual reference-counting operations (and of course there is much ugly code in existence that performs all sorts of ugly hacks and that does happen to and is expected to work).

Jonas