Hello together!

I'm pleased to announce that after nearly a year various extensions for dynamic arrays have been finished. This includes the following features:

- support for array constructors using "[...]" syntax
- support for Insert(), Delete() and Concat()
- support for "+" operator
- support for dynamic array constants (and variable initializations)

In the following the features will be explained in detail:

## Array Constructors

It is now possible to initialize a dynamic array variable or parameter using the "[...]" which is also known from open array parameters. Unlike the ArrayType.Create() syntax which had been introduced with 3.0 this also works with unnamed array types. Nested dynamic arrays are also supported.

=== code begin ===

var
   t: array of LongInt;
   t2: array of array of LongInt;
begin
   t := [1, 2, 3, 4];
   t2 := [[1, 2, 3], [4, 5]];
end.

=== code end ===

## Insert(), Delete(), Concat()

The intrinsics Insert(), Delete() and Concat() can now be used with dynamic arrays as well.

Insert() allows to insert a single element, a dynamic array or a static array into a dynamic array at a specific index (zero based). If the dynamic array is empty it will simply contain the content to be inserted. If the index to insert is larger than the array's High() then the content will be added at the end.

Delete() allows to remove a range of elements from a dynamic array (the start index is zero based). The range will be capped to the array's boundaries.

Concat() allows to concatenate two (or more) dynamic arrays together. The second array content starts directly after the first and those of the third after the second and so on.

## "+" operator

The compiler now implements a "+" operator for arrays which is the same as if Concat() would be called on the arrays.

Note regarding backwards compatibility: existing "+" operator overloads for dynamic arrays no longer compile.

## Dynamic array constants

It is now possible to initialize dynamic array constants and variables with concrete contents in contrast to merely Nil. The syntax for this is the same as for static array constant and variable initializations.

=== code begin ===

const
   Test1: array of LongInt = (1, 2, 3);
var
   Test2: array of String = ('Alpha', 'Beta', 'Gamma');

=== code end ===

For constants the dynamic array constants adhere to the writable constants switch $J. E.g. the following will result in an exception:

=== code begin ===

{$J-}
const
   Test1: array of LongInt  = (1, 2, 3);
begin
   Test1[1] := 42;
end.

=== code end ===

It is possible to nest dynamic and static array constants.

Delphi compatibility:

- in Delphi modes the syntax for dynamic array constant and variable initialization is "[...]" instead of "(...)"
- Delphi does not adhere to the writable constants switch regarding the array's contents (so the example above - when adjusted for Delphi's syntax - will not result in an exception)
- Delphi does not correctly parse static array constants inside dynamic arrays; as we don't know what the syntax would be for Delphi then, this is prohibited in Delphi modes

Regards,
Sven

Congratulations, job well done. This makes life a lot easier !

Michael.

:)

I hope 3.2 is coming soon.

This seems to be inconsistent with

procedure Test(const a: TStringDynArray);
begin
  a[1]:='42';
end;

Isn't it?

Mattias

The const in the parameters indicates that the pointer a cannot be assigned, but the contents can be assigned when inside the procedure. (we can discuss whether this behaviour is logical, but I suppose we inherited this from Delphi)

Whereas a constant is something that is eternally constant throughout the application. Normally, a constant is something that is replaced by its value when the compiler encounters it the sources.

So in a sense, it does not exist at runtime, hence you cannot assign to it either.

Michael.

Holy crap Sven. This is so awesome! I was going to write a request last week to allow for a simplified array constant declaration, and you've already got it done.

Thanks for your work. I'll test it momentarily and report any actual problems to Mantis.

By the way, what's the reason for differences in () / [] syntax between objpas and delphi mode?

I just got fpc from svn trunk (http://svn.freepascal.org/svn/fpc/trunk/) and tested. Before I submit to Mantis, can you tell me if I made a mistake here?

program hello;
{$mode delphi}
procedure Test;
var
  Data: array of Integer = [0, 15, 24];
  I: Integer;
begin
  WriteLn('Testing new array features');
  for I in Data do
    WriteLn(Data[I]);
end;
begin
  Test;
end.

Outputs:

Testing new array features
0
0
-2235258

Never mind, I found the problem.

program hello;
{$mode delphi}
procedure Test;
var
  Data: array of Integer = [0, 15, 24];
  I: Integer;
begin
  WriteLn('Testing new array features');
  for I in Data do
    WriteLn(I); // fixed
end;
begin
  Test;
end.

Outputs:

Testing new array features
0
15
24

Anthony Walter schrieb am So., 20. Mai 2018, 19:09:

By the way, what's the reason for differences in () / [] syntax between objpas and delphi mode?

For the non-Delphi modes I decided to keep consistency with static array constants (which use the same syntax in Delphi btw!). For the Delphi modes I followed what Delphi does. And for some reason the Delphi devs thought it better to use a different syntax than that for static arrays. 🤷‍♀️

Regards,
Sven

Mattias Gaertner schrieb am So., 20. Mai 2018,
17:58:

But it is consistent with static array constants and that is what counts...

Regards,
Sven

En Sun, 20 May 2018 07:23:25 -0500, Sven Barth via fpc-pascal

Wow!
Thanks :)

Jesus Reyes A.

2018-05-20 20:44 GMT+02:00 Sven Barth via fpc-pascal <

Gratz, seems that I will need to update NewPascal soon to get this fresh feature :).

They decide for [] probably because it is similar syntax like for open arrays. IMO "[]" syntax has more dynamic meaning/nature so looks more proper for this feature (as you said "()" syntax is used for static arrays), but this is probably individual taste.

"[]" also might be better for something like this :

var
  f: array of TSomeRec = [(a: 10; b: ['1', '2']), (a: 20; b: ['6'])];

instead of

var
  f: array of TSomeRec = ((a: 10; b: ('1', '2')), (a: 20; b: ('6')));

The main gain of different syntax between Delphi and ObjFpc is more troubles for Pascal Programmers. :(

Why you decide to prohibit static array constants inside dynamic array constants for Delphi modes in r39045? I do not understand this decision. It is some punishment for using Delphi mode? -,- IMO bad decision.

For the consequence the following code

var
  f: array of TSomeRec = [(a: 10; b: ['1', '2']), (a: 20; b: ['6'])];

should be also prohibited in Delphi mode, because it not works in Delphi like static array constants inside dynamic array constants...

Maciej Izak schrieb am So., 20. Mai 2018, 21:49:

Why you decide to prohibit static array constants inside dynamic array constants for Delphi modes in r39045? I do not understand this decision. It is some punishment for using Delphi mode? -,- IMO bad decision.

I prohibited it because Delphi does not support it correctly and I don't know whether they'd use the usual "(...)"-syntax for the static arrays or switch to the dynamic array one. To avoid potential backwards compatibility problems I decided to simply disallow it. And to be fair: how often do you declare dynamic arrays that consist of static arrays?

For the consequence the following code

I had not tested that one. In consequence that should indeed be disallowed as well. Though here the syntax probably wouldn't provide problems in the future.

Regards,
Sven

2018-05-20 22:50 GMT+02:00 Sven Barth via fpc-pascal <

I do not buy this ;) no one knows what they will use (if ever), but still I see no real reason for blocking this. Following yours logic any of management operators should be also prohibited in Delphi mode because some similar feature for nullable types is planned for Delphi (but no one knows the final form of their Initialize/Finalize equivalent). Code in r39045 is IMO too strict.

I had not tested that one. In consequence that should indeed be disallowed as well. Though here the syntax probably wouldn't provide problems in the future.

and the same sentence is true for the syntax for static array constants inside dynamic array constants, the more Delphied syntax for arrays probably wouldn't provide problems in the future too :)

If you like very strict approach and you want to make our life simpler probably you should implement some "modeswitch" for allowing both examples (example with records and arrays).

btw. the problem for different syntax between Delphi and ObjFpc is still valid. I don't see any real gain for different syntax. It was consulted with community or this is just individual decision? (and here sorry if I overlooked something).

Maciej Izak schrieb am So., 20. Mai 2018, 23:32:

The thing is that I can easily imagine the Delphi devs to be so stoned to use the "[...]" for static arrays inside dynamic arrays as well which could lead to constructs like the following because they forget that dynamic arrays could be part of static arrays as well:

=== code begin ===

var
  v: array[0..2] of array of array[0..2] of LongInt = ([[1, 2, 3], [4, 5,
6]], [], [[7, 8, 9]]);

=== code end ===

Yes, I have absolutely no confidence in the Delphi developers.

I don't want to use a modeswitch for every tiny little bit of feature. What we could do however would be a modeswitch for a strict Delphi mode that prohibits extensions and code that does not work in Delphi (like global operators and global generic functions). This could be used for code that is shared with Delphi to avoid potentially breaking compilability with Delphi if one is mainly using FPC for testing.

The gain is consistency with static array constants. You don't need to care about whether it's a dynamic array or a static one, you simply use the same syntax.
Also what do you care about mode ObjFPC anyway? One purpose of the mode *is* that it can go a different route than those modes that copy existing implementations.

Regards,
Sven

2018-05-21 0:52 GMT+02:00 Sven Barth via fpc-pascal <

Sometimes they have fantasy ^^. Don't underestimate them, I am sure they can do something more unexpected ;) For NewPascal mode I see this like :

var
  v: array[0..2] of array of array[0..2] of LongInt = ([(1, 2, 3), (4, 5,
6)], [], [(7, 8, 9)]);

IMO potentially breaking compatibility is better than lock/prohibition for missing features in Delphi (or waiting for Delphi implementation). Programmers are not stupid and most of Delphi users knows what Delphi have and what can be potentially risky. We have tons of examples which should be disabled in Delphi mode but it works for comfort. r39045 is for sure not comfortable IMO without real important reason, but this is just my opinion.

I am worried about all Pascal modes including ObjFPC and Delphi mode. :)

That is at least what I would *expect* for Delphi as well. And no, I'm definitely not underestimating their ability to come up with strange syntaxes... Do I do prefer not to think about it too much to avoid nightmares :P

I've decided to lift the restriction again in r39078. Happy now?

Regards,
Sven

Does that only apply when both operands are compatible dynamic arrays, or does it also break definition of + to append an element to an array?

Maciej Izak wrote

IMO potentially breaking compatibility is better than lock/prohibition for missing features in Delphi (or waiting for Delphi implementation).

Breaking backwards compatibility is one of the the worst things you can do in a programming language/compiler, and the primary concern when making changes to a language should always be to avoid this (except when you're developing a new language that's still in its infancy).

Even if you design everything with that mindset, you know that you will still have to do it occasionally because you missed certain things or could not foresee them. Especially since no one cares to formally specify and analyse standards for Pascal anymore, so nothing gets vetted properly. Those cases are enough, there is no need to intentionally add more (even if only potential ones).

Breaking backwards compatibility wastes time (both of the developer that has document the change and the users that have to adapt their —or worse, other people's— code), wastes energy on friction (users annoyed that their code no longer works, compiler developers annoyed that users didn't read the changelog, noise on fora/mailing lists, extra bug reports), and breaks the most fundamental expectation of using a high level language (unambiguous/formally correct code that used to compile/work should keep compiling/working; that's the whole point of using a high level language).

Some of the above things can also happen if a particular feature is not available in a certain language mode, but that is bound to the nature of language modes: not all language modes provide the same features and/or features in the same way.

Jonas

2018-05-21 10:49 GMT+02:00 Jonas Maebe :

Generally I fully agree with that... Anyway Delphi mode/case is special. AFAIK all significant Delphi compiler developers are lay off. The progress for classic Delphi is almost dead and Delphi NEXTGEN totally breaks backward compatibility.

We have interesting time : FPC Delphi mode with FPC RTL is more backward compatible with Delphi than new Delphi :).

I don't like the totally prohibition for all new things in Delphi mode, this is because not everyone is happy with ObjFPC and personally I prefer more Delphi syntax (XE2 level/features) but I want to have new features too (like fully working "Dynamic array extensions" without any prohibitions). I like to use some elements from Oxygene dialect (which probably can not be 100% ported into FPC). I have many voices abut this from community and many programmers agree with me.

The solution for keeping Delphi mode in right place and leave ObjFPC in FPC core hands, is new mode : NewPascal.

This was discussed few years ago that I will be able to do new mode when I will done few modeswitches, but now I am outside core team, situation is a little complicated but I am more determined. :)

2018-05-21 10:12 GMT+02:00 Sven Barth via fpc-pascal <

xD

I've decided to lift the restriction again in r39078. Happy now?

I can't be fully happy because I am not part of FPC core anymore but for sure I am more happy than before :)

On May 20, 2018, at 7:23 PM, Sven Barth via fpc-pascal wrote: The compiler now implements a "+" operator for arrays which is the same as if Concat() would be called on the arrays.

I haven’t built it yet but I’m curious, does += now push an element to the array or does this just work to concat 2 arrays? I would expect:

a: array of string = ();
a += ‘foo’;

to push a single string on the array.

Also can you use type helpers with dynamic arrays yet? Thinking about how to extend them if possible.

Thanks.

Regards,
Ryan Joseph

This is currently not supported. And to avoid backwards compatibility problems with existing operator overloads you'd probably need to convert it to a dynamic array first:

=== code begin ===

a += ['foo'];

=== code end ===

Also can you use type helpers with dynamic arrays yet? Thinking about how to extend them if possible.

Type helpers for dynamic arrays are already possible in 3.0, however they must be named array types (both for the declaration of the helper as well as the array) and they must be the same type (so equal type declarations in different units do not work).

=== code begin ===

{$modeswitch typehelpers}
type
   TLongIntArray = array of LongInt;
   TLongIntArrayHelper = type helper for TLongIntArray
     procedure Test;
   end;
procedure TLongIntArrayHelper.Test;
begin
   Writeln(Length(Self));
end;
var
   lia: TLongIntArray = (1, 2, 3);
begin
   lia.Test;
end.

=== code end ===

Regards,
Sven

I just wanted to drop this trick to work around problems with record helpers ..

If you wanted to create a record helper for a type that is not normally allowed, or perhaps you want to have multiple record helpers for the same type, consider defining your own separate type and make it contain said type with implicit conversions to work around those problem.

Example:

TArray<T> = array of T;
TArrayList<T> = record
public
{ The array acting as a list }
var Items: TArray<T>;
{ Convert an array to a list }
class operator Implicit(const Value: TArray<T>): TArrayList<T>;
{ Convert an open array to a list }
class operator Implicit(const Value: array of T): TArrayList<T>;
{ Convert a list to an array }
class operator Implicit(const Value: TArrayList<T>): TArray<T>;

Then in this example you can add all the methods you want to dynamic arrays, because you have your own type which is nothing more than an alias to a dynamic array (array of T). Methods such as:

{ Performs a simple safe copy of up to N elements }
procedure Copy(out List: TArrayList<T>; N: Integer);
{ Performs a fast unsafe copy of up to N elements }
procedure CopyFast(out List: TArrayList<T>; N: Integer);
{ Returns the lower bounds of the list }
function Lo: Integer;
{ Returns the upper bounds of the list }
function Hi: Integer;
{ Reverses the items in the list }
procedure Reverse;
{ Swap two items in the list }
procedure Exchange(A, B: Integer);
{ Adds and item to the end of the list }
procedure Push(const Item: T);
{ Appends an array of items to the list }
procedure PushRange(const Collection: array of T);
{ Remove an item from the end of the list }
function Pop: T;
{ Remove an item randomly from the list }
function PopRandom: T;
{ Return a copy of the list with items passing through a filter }
function Filter(Func: TFilterFunc<T>): TArrayList<T>;
{ Resurn the first item matching a condition }
function FirstOf(Func: TFilterFunc<T>): T;
{ Removes an item by index from the list and decresaes the count by one }
procedure Delete(Index: Integer);
{ Removes all items setting the count of the list to 0 }
procedure Clear;
{ Sort the items using a comparer }
procedure Sort(Order: TSortingOrder = soAscend; Comparer: TCompare<T> =
nil);
{ Attempt to find the item using DefaultCompare }
function IndexOf(const Item: T): Integer; overload;

What is that doing being the scenes? If it’s creating a whole new array and appending it then it’s probably pretty inefficient and hopefully can be optimized away.

Since dynamic array helpers work could you expose a function that adds an element to the array (an grows it if needed) so we could make helpers for it? Pushing a value to an array is perhaps the most common function used for lists so it makes sense to get that right imo.

Regards,
Ryan Joseph

I think the above is "right", and completely equivalent to sets, which are in some ways like an array: a "collection" of same typed values.

To add an element to a set you also do
   MySet:=MySet+[aValue];

That said, sets do have

   Include(MySet,AValue);

to include a single value.

Maybe this syntax can be extended to dynamic arrays.

Michael.

We can do + operator overloads on array classes though so it’s not totally crazy to see this syntax in code. Ultimately if it’s optimized well doing [xx] is no big problem I suppose.

Having said that my feeling with dynamic arrays is that they’re more difficult to use than collection classes and lack basic functionality (until perhaps just now) so why use them? I just got burned really bad by some call to FPC_INTERLOCKEDINCREMENT64 from dynamic arrays which killed performance (who knows why) so I had to pull them out of all my code just in case.

The procedural syntax of Insert(arr, 0, value) is less enticing than arr.Insert(0, value) also but again we can fix that with type helpers.

Regards,
Ryan Joseph

Ryan Joseph schrieb am Fr., 25. Mai 2018,
04:35:

Yes, that is what it does. Though a possible optimization would be that the compiler detects "dynarr := dynarr + [elem]" and converts it to "Insert(elem, dynarr, High(dynarr))".

That function already exists and is called Insert() (see above).

Regards,
Sven

Dynamic arrays are reference counted. The interlocked increment is there to make the operations thread safe. I do not know whether the classes you speak of are thread safe. If there is no FPC_INTERLOCKEDINCREMENT64, most likely they are not.

Dynamic arrays have their reason for existence. For one thing, they are a continuous block in memory, properly aligned and whatnot. The collection classes make no such guarantees.

The procedural syntax of Insert(arr, 0, value) is less enticing than arr.Insert(0, value) also but again we can fix that with type helpers.

Arrays are not objects. They do not have methods.

I understand where the tendency to make everything an object comes from but the basic pascal types (including dynamic arrays) have always been handled procedurally. Nothing wrong with that. I often write small programs procedurally.

And that is why type helpers exist, to tack on an object-like approach.

Michael.

Ryan Joseph schrieb am Fr., 25. Mai 2018,
07:55:

The procedural syntax of Insert(arr, 0, value) is less enticing than arr.Insert(0, value) also but again we can fix that with type helpers.

You are overestimating what type helpers are capable of doing for dynamic arrays - see my remark about named types and matching types in my first reply to you.

Regards,
Sven

I made my own class because I wanted to guarantee nothing silly was going on behind the scenes. I didn’t need thread safe so opting out of all the extra work done to make them safe is important sometimes. That would be nice if you could do the same with dynamic arrays honestly.

Agreed but sometimes we like to just type arr.insert() because it’s faster or more consistent with other code. I suspect FPC users are already making their own type helper wrappers for this.

One of the worst things I’ve encountered recently is C# forcing EVERYTHING to be a class. Total nightmare.

Regards,
Ryan Joseph

On May 25, 2018, at 1:02 PM, Sven Barth via fpc-pascal wrote: Yes, that is what it does. Though a possible optimization would be that the compiler detects "dynarr := dynarr + [elem]" and converts it to "Insert(elem, dynarr, High(dynarr))”.

100%

Oh Insert on the last element works like Push. There really should be a Push/append etc.. function in there. I know it’s redundant but doing

Insert(arr, High(arr), value) is more work than Push(arr, value) and it’s not clear if we can add this ourselves easily without making tons of overloads.

Regards,
Ryan Joseph

Nice. I didn't know Insert() could be used to append to the end of a string.

One learns every day ! One for the documentation :-)

To be clear: for strings you can do
var
   S : String;
begin
   S:='123';
   Insert('456',S,Length(S)+2);
   Writeln(S);
end.

and it will append. Is the same true for dynamic arrays ? i.e. any insert position >= length will do an append ?

Michael.

I'm guessing you are not using strings either then, for performance reasons ? Because exactly the same happens there.

Michael.

Ryan Joseph wrote

Oh Insert on the last element works like Push. There really should be a Push/append etc.. function in there. I know it’s redundant but doing Insert(arr, High(arr), value) is more work than Push(arr, value) and it’s not clear if we can add this ourselves easily without making tons of overloads.

Add() would sound more logical to me then.

Bart

On May 25, 2018, at 3:18 PM, Michael Van Canneyt wrote: I'm guessing you are not using strings either then, for performance reasons ? Because exactly the same happens there.

really? I had a collection class with a dynamic array for the storage and that’s when I got the high CPU usage in the thread safe stuff. I replaced the storage with a pointer to a block of memory which I managed myself and the problem went away. I just wanted to a dynamic array that was fast, minimal and could append elements.

I use lots of strings but I never had that problem. Are strings reference counted like dynamic arrays? I thought a string was just an array of 255 chars.

Regards,
Ryan Joseph

Michael Van Canneyt schrieb am Fr., 25. Mai 2018, 10:15:

Correct :)

Similar for indices <= 0: they will always prepend (for strings <= 1 of course).

Regards,
Sven

Ryan Joseph schrieb am Fr., 25. Mai 2018,
10:24:

ShortString is limited to 255 chars. AnsiString and UnicodeString don't have that limitation and they are reference counted just like dynamic arrays.

The type of "String" depends on the current settings:
- $H-: String = ShortString (this is the default for all modes except the Delphi ones)
- $H+ and "$modeswitch unicodestrings-": String = AnsiString (or AnsiString(CP_ACP) to be precise) (this is the default for mode Delphi)
- $H+ and "$modeswitch unicodestrings+": String = UnicodeString (this is the default for mode DelphiUnicode)

If I had to guess you were probably doing something with the arrays they weren't designed for. :/

Regards,
Sven

On May 25, 2018, at 3:57 PM, Sven Barth via fpc-pascal wrote: If I had to guess you were probably doing something with the arrays they weren't designed for. :/

It’s a little foggy now but I think I had a dynamic array in a class which was being passed around some functions or the class (with the dynamic array) was in a record which was being passed around some functions. I think this was happening within tight loops also.

Bottom line is if the that thread lock function does anything expensive then passing around dynamic arrays within tight loops could be a deal breaker unless you could manually declare a dynamic array as not thread safe.

Regards,
Ryan Joseph

Ryan Joseph schrieb am Fr., 25. Mai 2018,
12:45:

When passing around dynamic arrays in such situations var and constref parameters should be used and assignments inside those loops should be avoided.

Regards,
Sven

FPC now optimizes "dynarr := dynarr + [elem]" to "Insert(elem, dynarr,
high(sizeint))" and "dynarr := [elem] + dynarr" to "Insert(elem, dynarr,
0)".

Regards,
Sven

Are you serious ? I have been using dynamic arrays a lot for processing of vectors and matrices containing floating point values, and implemented the + operator to do the obvious, add the elements of two vectors, NOT to concat them. This is just natural for floating point vectors, similar as with other basic mathemetical operators as *,-, /.
Even if the "+" has always been used to concat strings, I find it a really really bad idea to extend that to danymic arrays.

Why are you using dynamic arrays for vectors/matricies? If what you have is an actual array you wish to grow then + would likely be an append operation.

var
  v: TVec2;
begn
  v := V2(1,1);
  v += 10; // add 10 to x/y  and get: v = 11,11
var
  v: array of float;
begn
  SetLength(v, 1);
  v[0] := 1;
  v += 10; // append 10 and get : v = 1,10

Regards,
Ryan Joseph

Why are you using dynamic arrays for vectors/matricies? If what you have is

an actual array you wish to grow then + would likely be an append operation.

Dynamic arrays are incredibly convenient for purposes like signal processng, whe you need to handle large blocks of numeric data in variable size. What an annoyace that was in times of turbo pascal, when I had to fuff with pointers, and use procedures for simple mathametical operations. With FPC that was all gone, and I can now handle vectors the same as simple numbers. Just like matlab does: a=bxc provides the expected result regardless whether the variables are scalars, vectors or matrices.

I strongly disagree with the opionon that "+" is natural to be the append operation. That may be true for strings, where it is the *only * useful operator. But for numeric vectors, *all * mathematical operators +-*/ are meaningful, and it is madness to steal one of them just for conatenation, and to obliterate all other operators for use with dynamic arrays at the same time. Obviously, if I cannot use "+", I can throw all others into the bin as well, but rather think of a different concept using procedures, like I am back in the 90s with turbo pascal.

By all means, please reconsider this, and leave me the choice to define the operators. If I want "+" for concatting, it is trivial to define it myself. I don't need the language to force that and eseentially destroy operator overloading for mathematical operations on dynamic arrays.

On Jun 2, 2018, at 1:35 PM, Nitorami wrote: I strongly disagree with the opionon that "+" is natural to be the append operation.

Not even in the context of a list? I’m still not sure what exactly the operations you are doing but it sounds like you have a vector of numbers and you want to overload + so you can add to the value of all numbers. That makes perfect sense but it’s in conflict with the other (more common) use of arrays.

Why aren’t you using a class or record and making a custom data type btw?

Regards,
Ryan Joseph

By all means, please reconsider this, and leave me the choice to define the operators. If I want "+" for concatting, it is trivial to define it myself. I don't need the language to force that and eseentially destroy operator overloading for mathematical operations on dynamic arrays.

Same here.

The semantics for vector operations on arrays was thoroughly explored in vector languages (APL, A+, J, K, etc). Doing per-element dyadic function application is the basis for it. Having proper operators overloads is crucial.

@Sven
Please reconsider "+" operator for arrays if your changes really forbid to overload operators for arrays now.

Regards,
Denis Golovan

Why not make a record for vectors that has a dynamic array backend? Then you can add all the methods and various operators you want. That of course doesn’t help for legacy code bases that sound like they’re going to be broken.

Regards,
Ryan Joseph

denisgolovan schrieb am Sa., 2. Juni 2018, 09:18:

@Sven Please reconsider "+" operator for arrays if your changes really forbid to overload operators for arrays now.

It wasn't me who implemented that part. I personally had planned to do it with a warning for existing overloads, but Florian beat me to it and implemented it this way. Though when asked by me he did say that we'll wait and see if people complain... So *maybe* we'll change this.

Regards,
Sven

On Jun 2, 2018, at 2:42 PM, Sven Barth via fpc-pascal wrote: It wasn't me who implemented that part. I personally had planned to do it with a warning for existing overloads, but Florian beat me to it and implemented it this way. Though when asked by me he did say that we'll wait and see if people complain... So *maybe* we'll change this.

btw why can’t there be both? You can have multiple + operators for any given dynamic array type can’t you?

type
	TArrayOfInteger = array of integer;
operator + (left: TArrayOfInteger; right: integer): TArrayOfInteger;
var
	i: integer;
begin
	for i := 0 to high(left) do
		left[i] += 1;
end;
operator + (left: TArrayOfInteger; right: TArrayOfInteger): TArrayOfInteger;
begin
	result := Concat(left, right);
end;

Regards,
Ryan Joseph

It's technically possible.

But for vector operations to be valid/consistent both of them should work the same way. That is perform arithmetic per-element addition.

BTW, you first overload is not implemented properly. You need to clone "left" first and return it as a result.

BR,
Denis

Yes, I strongly support removing that functionality in favor of user operator overloads or vector-compatible way.

Moreover, SSE/AVX vector extensions also should work per-element. I mean those vectors as in https://bugs.freepascal.org/view.php?id=27870

BR,
Denis

Agreed, both are extremely useful and have an intuitively unambiguous meaning (unlike - which can be useful but doesn't have an unambiguous meaning).

However as Dennis points out + is also essential for vector operations. Perhaps either leaving it to the programmer to define what's needed would be the best approach, or alternatively splitting dynamic arrays into mathematical vectors and non-mathematical collections. Or relaxing the requirement that only predefined operators can be redefined, so that something like _ could be used for concatenation.

+1 for the possibility to overload this operator.

Mattias

denisgolovan schrieb am Sa., 2. Juni 2018, 10:28:

Yes, I strongly support removing that functionality in favor of user operator overloads or vector-compatible way.

To clear something up: this new operator will definitely not be removed. Period.

What might be done however (and what I had planned) is that existing overloads of the "+"-operator take precedence to the internal operator.

Though I wouldn't mind introducing a syntax that can be used to force a element wise operation on a array. This way one wouldn't need to do the overload for the array, but the compiler would pick the operator of the element type instead.

Regards,
Sven

Mark Morgan Lloyd schrieb am Sa., 2.
Juni 2018, 10:53:

That needlessly complicates the parser as the compiler still needs to know them and they also need to be part of its operator precedence rules. Don't complicate the language for nothing! And in the end operator overloads are one of the best examples for syntactic sugar as you can easily achieve the same result with functions and methods.

Regards,
Sven

On Jun 2, 2018, at 3:19 PM, denisgolovan wrote: BTW, you first overload is not implemented properly. You need to clone "left" first and return it as a result.

That’s probably the functionality you want but as an aside why can’t + overload mutate the caller (left side) and not return anything?

I would expect there to an operator like below in addition to the usual that returns the left side and assigns to the left aside on return.

class operator + (var left: TIntArray; const right: T);

if you’re appending to an array for example:

a += 1;

why should the compiler return the array (a) and then copy it back to the caller in the same place? It’s a just a waste and if the caller is a record it could be costly.

Regards,
Ryan Joseph

Ryan Joseph schrieb am So., 3. Juni 2018,
05:35:

Because operator overloads are static methods not normal methods.

Regards,
Sven

Yes, if you want destructive updates you need a special syntax for that.

I believe existing syntax is not suitable for destructive update, hence the error I mentioned.

BR,
Denis

On Jun 3, 2018, at 3:15 PM, Sven Barth via fpc-pascal wrote: Because operator overloads are static methods not normal methods.

I don’t understand. Why aren’t both those variants possible? They’re both static I believe. The first is mutating the left side value and the second is a clone (the most common operation for += )

class operator + (var left: TIntArray; const right: T);
class operator + (constref left: TIntArray; const value: T): TIntArray;

Regards,
Ryan Joseph

This is somehow off topic of course,
but IMO it is strange to use + for string concatenation; I always have bad feelings about this. This whole thread would not exist, if FreePascal had gone another direction like PL/1, for example, where the string concatenation operator is ||
(and DB2, and - probably - other SQL dialects).

Where does this + for string concat come from?

(Of course, || has some codepage issues, but that's another story, and it means logical or in other languages ...)

BTW:

this is (part of) the input for the scanner generator for the New Stanford Pascal compiler:

SYLPARENT := '(';
SYRPARENT := ')';
SYLBRACK := '[' OR '(.' OR '(/';
SYRBRACK := ']' OR '.)' OR '/)';
SYCOMMA := ',';
SYSEMICOLON  := ';';
SYARROW := '->' OR '@' OR '^';
SYPERIOD := '.';
SYDOTDOT := '..';
SYCOLON := ':';
SYPLUS := '+';
SYMINUS := '-';
SYMULT := '*';
SYSLASH := '/';
SYEQOP := '=';
SYNEOP := '';
SYGTOP := '>';
SYLTOP := '<';
SYGEOP := '>=';
SYLEOP := '<=';
SYOROP := '|';
SYANDOP := '&';
SYASSIGN := ':=';
SYCONCAT := '||';

if you want to change the representation of the symbols, you only change here.

Kind regards

Bernd

On Jun 3, 2018, at 4:56 PM, Bernd Oppolzer wrote: but IMO it is strange to use + for string concatenation; I always have bad feelings about this.

It’s actually pretty common to use the phrase “adding two things together” in English. For example: “I’m going to add some more strawberries to my milkshake”, or, “add this to my list of things to do”. Maybe it’s a language thing but adding is certainly pretty typical for the idea of concatenation.

Regards,
Ryan Joseph

Mattias Gaertner wrote

> Sven Barth via fpc-pascal hat am 2. Juni 2018 um 09:42 geschrieben: > > denisgolovan < > schrieb am Sa., 2. Juni 2018, 09:18: > > @Sven > > Please reconsider "+" operator for arrays if your changes really forbid to overload operators for arrays now.> >  > It wasn't me who implemented that part. I personally had planned to do it with a warning for existing overloads, but Florian beat me to it and implemented it this way. Though when asked by me he did say that we'll wait and see if people complain... So *maybe* we'll change this. +1 for the possibility to overload this operator.

It would be reasonable to assume that the predefined + might be substantially more efficient than a programmer-defined one could be.

Yes, that's one of the reasons I vote for keeping the new feature and allow to overload the operator.

Mattias

There is still concat which provides exactly the same functionality.

True.
Same as in Delphi.

Is there a problem having the intrinsic '+' and custom overloads for arrays?

Mattias

This is so far not allowed: overriding intrinsic operators with custom overloads, so it would be another language exception.

In our previous episode, Ryan Joseph said:

But nobody + an item to their item list. The operator is "+" not "add".

On Jun 3, 2018, at 9:07 PM, Marco van de Voort wrote: But nobody + an item to their item list. The operator is "+" not "add".

What other operators could be used then? There’s only so many characters on the keyboard after all.

My only interest is because it’s fast to type and looks nice. Swift allows you to overload any operator in the unicode set so we could make up our own crazy stuff. :)

list • 1; // list.Add(1);

Regards,
Ryan Joseph

That is a horrible idea.

Think of it:
You will read someone else's code, and have absolutely no clue what it does.

Michael.

For sure. Apple even showed how you can use emojis as operators. :) The best I can say is some interesting operators could be discovered and standardized if given enough time and experimentation. Math people probably has some fun ideas that could work out in the end.

Regards,
Ryan Joseph

Bernd Oppolzer schrieb am So., 3. Juni 2018,
11:56:

FPC inherited the +-operator for concatenation from the base language: Pascal. So there simply was no other route to take (not that anyone would have thought to take a different route).

Where does this + for string concat come from?

Ask Wirth, he is the one who invented Pascal...

Regards,
Sven

Ryan Joseph schrieb am So., 3. Juni 2018,
10:22:

Let me make one thing perfectly clear: the C-operators (+=, -=, *=, /=) do not exist as explicit operators in FPC. They are merely syntactic sugar for the more expressive variants and converted to those as soon as possible.

Regards,
Sven

Ryan Joseph wrote

> On Jun 3, 2018, at 9:07 PM, Marco van de Voort wrote:> > But nobody + an item to their item list. The operator is "+" not "add". What other operators could be used then? There’s only so many characters on the keyboard after all. My only interest is because it’s fast to type and looks nice. Swift allows you to overload any operator in the unicode set so we could make up our own crazy stuff. :) list • 1; // list.Add(1);

All of the predefined ones have the problem of clashing with arithmetic use in numeric vectors. I'd note that Perl uses . but my preference would be _ (followed by whitespace, or at least not by something that would make it look like a variable name).

People with deeply masochistic tendencies might appreciate the link below, if unavailable Google might have a cache.

http://www.dlugosz.com/Perl6/web/APL.html

Having worked with APL for almost 6 years and continuing with Q/KDB, I would say APL syntax is much more readable than Perl 6.

Though being able to modify language like that is really impressive.

BR,
Denis

In our previous episode, Sven Barth via fpc-pascal said:

Well, + is also addition in pascal. It depends on what is considered more important, the fact that array of integer has a base type that suppers adding, or the simularity with strings(that are also dynamic array like) that uses it for concat.

denisgolovan wrote

Having worked with APL for almost 6 years and continuing with Q/KDB, I would say APL syntax is much more readable than Perl 6.

Horses for courses. I think the impressive thing about APL is that the necessary operations were worked out (and used for Blackboard demonstrations) /before/ it was converted into a computer language, and by and large weren't added to. However the functional nature of the language was vastly overrused, and students who thought they were being smart would on occasion find themselves with mainframe runtimes of /months/ because they'd created an enormous array instead of using a simple control structure.

Nod here to Vector Pascal as well. And as a bit of history, the first computer implementation of Iverson's notation was done at Stanford under the watchful eye of Niklaus Wirth.

Though being able to modify language like that is really impressive.

I agree, which is the main reason I posted the link.

Yes, sure.
But things slowly change in compiler/interpreter world. Now we have loop/streams fusion techniques and other devectorization tricks, so I think smart APL interpreter would be of real help here.

Nod here to Vector Pascal as well. And as a bit of history, the first computer implementation of Iverson's notation was done at Stanford under the watchful eye of Niklaus Wirth.

Hm, I didn't know that. Thanks for sharing.
Looks like computer scientist community was really small those days.

Vector Pascal is much closer to our current discussion though :)

AFAIK, the Pascal Standard (ISO) does not tell anything about strings and concatenation. Wirth did not include varying length strings into the language in the 197x years. Every Pascal compiler which contains strings and concatenation does this by extending the Pascal Standard and can choose its own way to do it.

IBMs Pascal/VS uses || for string concatenation (inspired by PL/1, probably),
and that's what I implemented in the New Stanford Pascal compiler, too.

The + for concatenation must be an invention of Turbo Pascal or UCSD Pascal or something like that.

Regards, Sven

I don't think that argument holds water. Concatenation of dynamic structures is a slow function in the first place. I am not a core developer, but I consider that regardless how you do it, it will require a call to setlength, which in turn will call the operating system to allocate memory for the now larger structure. That takes much more time than a few assembly instructions for a normal pascal procedure call.

Nitorami wrote

>> It would be reasonable to assume that the predefined + might be>> substantially more efficient than a programmer-defined one could be. > Yes, that's one of the reasons I vote for keeping the new feature>and allow to overload the operator. I don't think that argument holds water. Concatenation of dynamic structuresis a slow function in the first place. I am not a core developer, but Iconsider that regardless how you do it, it will require a call to setlength,which in turn will call the operating system to allocate memory for the nowlarger structure. That takes much more time than a few assembly instructionsfor a normal pascal procedure call.

The obvious workaround is to preallocate a dynamic array to a nominal size and then to trim it using SetLength(0), I'd assumed that it would retain ownership of the memory but I've just checked.

What actually happens is that the memory is released back to the heap (but *not* to the OS, at least on Linux), with the result that concatenating elements will introduce a substantial hit particularly if space for a new element allocated from the heap isn't contiguous.

What actually happens is that the memory is released back to the heap (but *not* to the OS, at least on Linux), with the result that concatenating elements will introduce a substantial hit particularly if space for a new element allocated from the heap isn't contiguous.

Writing a preallocating wrapper *where needed* for heavily grown arrays is fairly simple.

The thing is: it's rarely worth it ;-)
I recently did that for a wavefront mesh loader that appends vertices one-by-one to an array, and for a scene with some 400k vertices, the difference was just some milliseconds out of several seconds overall.
Turns out the allocator usually finds a spot where the array doesn't need to actually be copied around for a while, and the pure bookkeeping of realloc is very cheap.

Martok wrote

> What actually happens is that the memory is released back to the heap > (but *not* to the OS, at least on Linux), with the result that > concatenating elements will introduce a substantial hit particularly if > space for a new element allocated from the heap isn't contiguous.Writing a preallocating wrapper *where needed* for heavily grown arrays isfairly simple. The thing is: it's rarely worth it ;-)I recently did that for a wavefront mesh loader that appends vertices one-by-oneto an array, and for a scene with some 400k vertices, the difference was justsome milliseconds out of several seconds overall.Turns out the allocator usually finds a spot where the array doesn't need toactually be copied around for a while, and the pure bookkeeping of realloc isvery cheap.

The bit of code I was looking at was a mainframe emulator reading a tape to find the block positions, I was appending elements individually using a user-defined + and it wasn't exactly fast (resulting in a tape load time probably slower than the hardware would have had 60 years ago)... although obviously not as bad as Nitorami's suggestion that every concatenation would result in a syscall.

I'd been expecting that SetLength() would do the same as it does with a string, i.e. prune the valid range without releasing memory.

What I've done so far is only interim code and I've got lots of ways round it, but the reality check was useful :-)

Mark Morgan Lloyd schrieb am Do., 7.
Juni 2018, 11:46:

Of course SetLength(0) releases the memory as the value of a dynamic array with length 0 is Nil, so nothing can be saved there.

Regards,
Sven

It would be worth though to think about extending the memory manager to make use of the realloc syscall (not sure about its actual name) on some OSes as the OS can "copy" memory to a different virtual address by just re-mapping pages. I think at least linux really does this.

Dear Sir:

I don´t like your way of answering. Period.

Mathematical operators on matrices (+, -, *, ~, etc.) have being defined centuries ago, and all the scientific community uses the same conventions on them. It´s not proper that somebody now would like to redefine the conventions on his own will, regarless of the user´s community opinions.

If you like an operator for concatenation, which is A WHOLE NEW FEATURE, please use a different or invented operator. You can use concatenator operarators borrowed from PHP ("."), Excel - VB ("&"), or invent your own one ( +. +~ +* etc)

I firmly oppose to any of your intentions (define the + operator for concatenation and allow user defined operators to take precedence). It doesn´t make sense. It´s not logical.

Thank you for hearing at me, and my apologies for my rough language.

Best regards,

Roger Rivero

---
El software de antivirus Avast ha analizado este correo electrónico en busca de virus. https://www.avast.com/antivirus

If I can, I would like to vote for "&" instead of "+". As I use Free Pascal with math (neural networks), I would vote for "&" with concatenation and "||" with "union" (in the case it's ever required).

On Mon, Jun 18, 2018 at 5:24 AM, Roger Rivero Jr. (SAMPA WebMaster) <

Joao Schuler wrote

If I can, I would like to vote for "&" instead of "+". As I use Free Pascal with math (neural networks), I would vote for "&" with concatenation and "||" with "union" (in the case it's ever required).

There won't be any voting as the "+" operator was chosen for Delphi compatibility.

However I've worked out a solution using modeswitches that should probably satisfy most people, now I only need to find the time to commit it.

Regards,
Sven

Addendum: the support for the "+" operator is now coupled to a new modeswitch "ArrayOperators".

If the modeswitch is enabled, then the operator can not be overloaded and it also won't be used even if an overload from a unit without the modeswitch is in scope (the compiler however issues a warning if this is the case, so you can either remove the overload or disable the modeswitch). If the modeswitch is not enabled then everything behaves as before.

The modeswitch is enabled by default in Delphi modes as this feature was added for Delphi compatibility. If you want to disable it in those modes then you need to do this:

=== code begin ===

{$mode delphi}
{$modeswitch arrayoperators-} // Note the "-"

=== code end ===

Regards,
Sven

Sven, please could we have a further update when you decide what version of the compiler will get the new facility, so that those of us with code that might clash can make sure we have version-specific conditionals in place.

[...] The modeswitch is enabled by default in Delphi modes as this feature was added for Delphi compatibility.

I would like to use that for objfpc.
Is there a command line flag to enable a modeswitch?

Mattias

On Jun 21, 2018, at 2:56 AM, Sven Barth via fpc-pascal wrote: Addendum: the support for the "+" operator is now coupled to a new modeswitch "ArrayOperators".

Finally got to try this. It’s really satisfying being able to do:

var
	arr: array of integer = (1, 2, 3);
	i: integer;
begin
	arr += [4];
	for i in arr do
		writeln(i);

thanks!

Regards,
Ryan Joseph

[...] Finally got to try this. It’s really satisfying being able to do: [...] for i in arr do

Note: This was supported for a long time.

Mattias

I mean the += [] part. :)

On Jun 21, 2018, at 3:37 PM, Mattias Gaertner wrote: Note: This was supported for a long time.

Regards,
Ryan Joseph

Mark Morgan Lloyd schrieb am Do., 21.
Juni 2018, 09:11:

New language features are normally not merged to a fixes branch (there were exceptions in the past, but they weren't too big). So if you check for >= 3.1.1 you're set.

Regards,
Sven

Mattias Gaertner schrieb am Do., 21. Juni 2018,
09:24:

Not tested, but -MArrayOperators *should* work (it takes both modes and modeswitches).
One point to keep in mind however: any single modeswitch setting is overwritten once a mode is encountered, this includes both the $mode directive and the -M option (or e.g. -S2).

Regards,
Sven