Hello, FPC developers' list.

I think we can close 'sealed/abstract class' page and open another interesting topc.

It is obvious you are full of energy to discuss new language features.

The next feature which we should think of is for-in loop: http://wiki.lazarus.freepascal.org/for-in_loop

There you can find:
1. Delphi implementation details
2. Proposed extended syntax
3. Proposed compiler implementation

While discussion please try to have delphi compatibility in the mind.

At least one of the reasons we never did implement for-in is the absolutely horrible and totally wrong idea to use classes/interfaces for this, to which I seriously objected.

The guy who thought of that should be shot, hanged, guillotined and poisoned. (you pick your order) It is a total negation of everything that pascal stands for: let a language feature (for in) depend on a user-class implementation.

I can understand

   for y in enumerated do

or

   For y in set do

or

   For Y in range do

Because this seems a natural extension of loops to enumerates (and even then) But not ever the class-based one.

Michael.

2009/10/20 Paul Ishenin :

It is obvious you are full of energy to discuss new language features.

:-)

The next feature which we should think of is for-in loop: http://wiki.lazarus.freepascal.org/for-in_loop

As an alternative to the very limited "for-in", I have a better and more flexible design, using the Iterator design pattern. I have already written iterations for most of FPC's list components. It would be nice if the iterators could be part of the default list component and not as an external unit like I currently have. I'm willing to do the work in merging my external iterators with the various list components. Just say the word.

It allows you to write code as follows, irrespective of what list component you are working with.

  MyIterator := MyList.GetIterator;
  while MyIterator.HasNext do
    ShowMessage(MyIterator.Next);

The iterator interfaces has a lot more functions that just .HasNext and .Next.

For example, I have a String Iterator that use regular expressions to "filter" the return results. So now you have the exact same code as above, but ShowMessage() will only act on the filtered return results. As you can see, a lot more flexible that the for-in construct.

Additional methods that the iterator can support.

Add(item)
Inserts a specified item into the collection. (optional modifier operation)
HasNext()
Returns true if the collection has more items when traversing the collection in the forward direction.
HasPrevious()
Returns true if the collection has more items when traversing the collection in the reverse direction.
Next()
Returns the next item in the collection.
Previous()
Returns the previous item in the collection.
Reset()
Jump to the beginning of the collection, setting the cursor/index before the first item in the collection. The iterator is now in the same state as if you just created it.
ToBack()
Jump to the end off the collection, setting the cursor/index after the last item in the collection. This needs to be called before you want to traverse the collection in the reverse order.
PeekNext()
Returns the next item without moving the iterator's cursor/index. PeekPrevious()
Returns the previous item without moving the iterator's cursor/index.
Remove()
Removes from the collection the last item that was returned by the Next() or Previous() calls. (optional modifier operation) SetItem(item)
Replaces the last item returned by the Next() or Previous() calls with the specified item. (optional modifier operation)

BTW: Java and .NET have such type of iterators on just about every list component and it's very handy (I can only comment here on Java implementation because I don't use .NET).

I have written a article about this - in a lot more details. You can get a copy of the article at the following url:

http://opensoft.homeip.net/articles/

Graeme Geldenhuys wrote

As an alternative to the very limited "for-in", I have a better and more flexible design, using the Iterator design pattern.

for-in is just a syntax sugar plus standard interface for the cost common usage of iterators. So they are not mutually
Alexander S. Klenin

Exactly.

I think that Graeme's approach is much better than the very unpascal-ish for Y in Someclass do
and would much prefer this to be implemented in the RTL.

Michael.

2009/10/20 Alexander Klenin :

and direction. You can't peak forward, you can't move back, you can't filter returned content.

So you, for-in is VERY LIMITED in my eyes, and a total waist of time.

2009/10/20 Alexander Klenin :

for-in is just a syntax sugar plus standard interface for the cost common usage of iterators. So they are not mutually flaw in the for-in concept...

What will this do:

  for y in MyList do
  begin
    if y = XXX then
      MyList.Add(YYY);
    if y = ZZZ then
      MyList.Insert(1, AAA);
  end;

With my iterator implementation I can handle this with no problems, and ever make the resulting behaviour user-selectable with parameters to the MyList.GetIterator method.

Well, I suggest you to read at least the link provided in the original post, and maybe also some documention about Delphi/C# enumerators.

Graeme Geldenhuys wrote

BTW: Java and .NET have such type of iterators on just about every list component and it's very handy (I can only comment here on Java implementation because I don't use .NET).

Re .NET: Delphi Prism does have iterators.

( http://prismwiki.codegear.com/en/Iterators )

-Michael

In our previous episode, Paul Ishenin said:

I had a discussion with Marc W. about this on the last BBQ (Micha and Vincent might have been there too). IIRC we agreed that the set functionality is the most important by far, suggest to prioritize that

btw the earlier discussions about D2005+ functionality are archived at:

http://www.stack.nl/~marcov/delphilater.txt

I wanted to use this for an article, so the log are sligtly processed and annotated.

Michael Van Canneyt wrote

At least one of the reasons we never did implement for-in is the absolutely horrible and totally wrong idea to use classes/interfaces for this, to which I seriously objected.

Reading this I conclude you are against 'for-in' implementation in fpc.

So you are not against it but you want to allow them only for the base types?

I hope you understand that ppl will use them for classes anyway?

var
   P: Pointer;
begin
   for P in GetComponents(Component) do
     TComponent(P).DoSomething;
end;

And what is so special in the class type?

Best regards,
Paul Ishenin.

2009/10/20 Alexander Klenin :

Well, I suggest you to read at least the link provided in the original post, and maybe also some documention about Delphi/C# enumerators.

I just read the docs in the lazarus wiki. The problems/limitations still hold true. For-in only goes in ONE direction and can't peak or filter returned results etc..

What good is an enumerator that only ever goes in one direction and visits all items. That's not a very flexible design.

2009/10/20 Michael Schnell :

Re .NET: Delphi Prism does have iterators. ( http://prismwiki.codegear.com/en/Iterators )

In all respects, Delphi Prism is not a Embarcadero product - except for the slapped on "Delphi" prefix, Embarcadero has nothing to do with Delphi Prism.

But yes I know iterators exist in .NET, that's what I said in my previous post. Java & .NET, but I have only every worked with the Java code.

This is IMO similar to using a for loop vs. using a while loop. In a for loop you are not allowed to modify the loop variable. In this case I can imagine its not allowed to modify the list. If you want to, use a while and iterators.
Yes I know iterators, I use them myself too on my own lists. As said I before, I think too that tose can be combined.

Marc

Hm, did you read the about reverseEnumerators and various irders of tree traversals?

and can't peak

That is true, but this is required in maybe 1% of the cases, or even less. For these cases, certainly slightly longer while loop is justified.

filter returned results etc..

Well, there is no such sample currently on the wiki, but it is still quite possible.

What good is an enumerator that only ever goes in one direction and visits all items. That's not a very flexible design.

Please read again with more attention ;-)

No.

Yes.

I hope you understand that ppl will use them for classes anyway?

Those people should be re-educated to use iterators.

First:
For a for loop, you are guaranteed that the loop logic behaves correctly. A loop is equivalent to the following:

I:=StartValue;
E:=EndValue;
While I<=EndValue do
   begin
   // loop stuff
   Inc(I);
   end;

And you cannot change I manually during the loop.

You don't know this with an iterator since you depend on the implementation of the iterator. The loop could loop forever then...

Secondly:
You promote a certain class/interface to a language feature. The compiler then depends on the presence of a certain class with some 'known' methods in the RTL.

Both points are simply wrong from a language point of view.

It is bad enough that the second point is already so for interfaces and even TObject, (a very serious design flaw by Borland) but extending this even further to include actual language features such as the for loop is 2 bridges too far as far as I'm concerned.

Especially when you can have perfectly the same functionality with iterators.

Borland could have perfectly satisfied people by implementing a set of iterators in the classes unit. Instead they raped - yet again - the pascal language. (they are a commercial company, so what else can we expect...)

Michael.

2009/10/20 Alexander Klenin :

-------------------
// for basic type we will call only the apropriate function
procedure TraverseString(S: String);
var
  C: Char;
begin
  for C in S using GetReverseStringEnumerator(S) do
    DoSomething(C);
end;
-------------------

Now your for-in syntax changes because you want to use different enumerators. Why? It's not very elegant having to change your code, based on what you are iterating. That tells me it's not flexible enough. With the Iterators I wrote, you only have ONE interface you have to work with, no matter what type of list you are iterating or if the returned results are filtered or not.

> and can't peak That is true, but this is required in maybe 1% of the cases, or even less. For these cases, certainly slightly longer while loop is justified.

Lets say you want to parse a UTF-8 string. Or you want to parse text for a documentation generator, or you want to strip text from an HTML file or Wiki page. All uses of where Peak is very often used. And that's just a short list of the top of my head.

Please read again with more attention ;-)

BTW: The wiki pages does not contain the word "reverseEnumerators" as you suggested earlier.

Yes. For-in loop is not a for-to loop. While and Repeat loops can also loop forever. You can't protect a developer if his hands grow from the wrong place.

Secondly: You promote a certain class/interface to a language feature. The compiler then depends on the presence of a certain class with some 'known' methods in the RTL.

Yes, I see this very bad too. That's why this discussion is started. But what can be suggested instead?

Maybe new type?

TSomeIterator = iterator(<iterated type>)
...
end;

or we can use some basic class:

TIterator = class
public
   function MoveNext: Boolean;
   property Current: <what type to return?>
end;

I don't like non-constructive criticism. If you don't like something please be ready to suggest something instead. And in this case something what can be used to implement for-in loop for classes and implements this delphi compatible (so 'for S in MyStringList do' will work).

Best regards,
Paul Ishenin.

This is just one of the proposed syntax variants (and not the one I personally favor).
Current Delphi iterators do it with the same syntax:

for C in ReverseEnumerator(S) do
  DoSomething(C);

Hm. I can not see where lookahead aka peeking
is required in using any of the enumerators you mention. UTF8String enumerator is even among the samples on the wiki page.

> Please read again with more attention ;-) BTW: The wiki pages does not contain the word "reverseEnumerators" as you suggested earlier.

My bad. I skipped space between words.

Michael Van Canneyt wrote

And you cannot change I manually during the loop. You don't know this with an iterator since you depend on the implementation of the iterator. The loop could loop forever then...

Maybe the iterator function should be a "const" function. A const function is not allowed to change the class members it's operating on.

Micha

IMO too

> I hope you understand that ppl will use them for classes anyway? Those people should be re-educated to use iterators.

Exactly. This all start to look a bit to much like the VB iterator magic. Where some under the hood functions (names) were used for iterating. If you didn't declare a container with the right function names you couldn't loop.

Exactly. You extend a language construct with something defined with that language (i don't know how to explain better)

I can see a use for using iterators in a for loop, however they should be declared with some keyword.

Something like

type
   TListIterator = iterator(TList, init_func, next_func, check_func)
     function init_func: Boolean;
     function next_func: <element type>
     function check_func: Boolean;
   end;
begin
   for element in list using TListIterator do...

IMO this is more pascal than using some interface or predefined function names.

Marc

IMHO nothing. Maybe marc's proposal.

This is the same as what Borland does: you promote a certain class to a language feature...

I don't like non-constructive criticism. If you don't like something please be ready to suggest something instead. And in this case something what can be used to implement for-in loop for classes and implements this delphi compatible (so 'for S in MyStringList do' will work).

I did suggest something instead: implement iterators in the classes unit, as Graeme suggests.

Yes, I know this is not Delphi compatible. But I'm OK with that. I think the importance and gain of the for in loop is far too small to spend time on it, since there are plenty of alternatives possible today.

Of course, if you want to be 100% delphi compatible, then there is simply no way we can ever agree on this particular feature, no matter how constructive I am or you are.

So then, in the end, someone will have to back down and put aside his wishes :-)

But I really don't see the advantage of being able to type

   For F in Someclass.Iterator do
     F.Something

over

   While SomeClass.HaveValue do
     SomeClasss.NextValue.Something;

It's not clearer, at most it saves you a couple of keystrokes.

Raping a clean language for this is simply not correct. It's not because Borland - in its desire to make money - implemented every idiots' request, that we should do the same.

Michael.

Michael Van Canneyt wrote

You don't know this with an iterator since you depend on the implementation of the iterator. The loop could loop forever then...

Just as it can when using iterators explicitly. No difference here.

Well, for better or worse, tight integration of language and RTL is an intrinsic feature of Pascal language since it's inception.

Just look at the original spec:
Write, Writeln, Read, Readln, Pred, Succ, Reset -- almost all standard functions are "magic"!
Borland just continued the tradition started by Wirth. Now that is so ingrained in ObjectPascal that there is no point in resisting: arrays/SetLength, RTTI/GetXXXProp, the whole TObject, string/Copy, the list goes on...

Almost every feature in modern Pascal is broken this sense. Still, this might not be such a bad thing -- see C++/STL for the opposite method driven to it's logical conclusion. So I guess a measured dose of language/RTL integration is actually good.

The difference is that the language guarantees that the for loop will run OK. It makes no such guarantees with a while or repeat loop.

Michael.

The whole interface ref. counting depends on that as well as e.g. the "as" operator.

My opinion about the for-in loop:
- I won't implement it
- if somebody else implements it in a delphi compatible manner: fine
- if somebody else implements it in a delphi compatible manner solving the flaws: great
- if somebody else has ideas how the existing classes can be extended without drawbacks by more sophisticated iterators, go ahead, this won't interfere with Paul's stuff

This is because the while you provided is not equivalent to the for loop above.
The correct translation would be:
var
  it: TSomethingIterator;
...
  it := SomeClass.Iterator;
  try
    while it.HaveValue do
      it.NextValue.Something;
  finally
    it.Free;
  end;

Now, that is quite e few keystrokes to save, not to mention that if item value is used more than once in the loop, SomeClass.NextValue must be stored in a variable, further bloating code.

Borland - in its desire to make money - implemented every idiots' request, that we should do the same.

Well, I have the same attitude regarding sealed classes ;-)

However, for..in is not like those.
Let me provide a few over-generalized statements in support of for..in feature.

1) Convenient syntax matters -- otherwise we would be programming in C
2) Language level matters -- otherwise we would be programming in assembler language
3) for..in is both a convenient syntax and a very important high-level feature. It is actually more fundamental and important than for..to loop, and it is only a matter of weak compiler implementations that for..to loop was introduced first.

Now this is great attitude ;-)
What are the flaws of Delph implementation in your opinion?

I just completed a parser for a custom RichText component I wrote for fpGUI Toolkit. This will probably apply to HTML, XML etc as well. Say you have the following string

"Hello <b>World</b>!:

When I parse that and I get the the "<" characters, I peak ahead to see if I am working with a actual styling tag, or if I am working with the actual "<" character in the contents. So when I tokenize that string, with the help of "peak ahead" I know what token type to set.

Please pay more attention to the wiki article! :-) Neither "reverse Enumerators" nor "reverseEnumerators" are text in the for-in loop wiki page. At least that is what Firefox tells me when I do a Find.

I know, I said this also in my remark that they did this already for some interfaces (ref counting) and TObject...

If the largest part of the compiler team has no problem with it, I will of course not refuse it. Although I would definitely request to allow the delphi-compatible solution only Delphi mode, and only to allow a *decent* implementation in ObjFPC mode.

Michael.

The iterator itself have to look ahead, yes, but this should be hidden from the caller -- i.e. the user of HTMLIterator have no need to know that the iterators looks ahead.

Please pay more attention to the wiki article! :-) Neither "reverse Enumerators" nor "reverseEnumerators" are text in the for-in loop wiki page. At least that is what Firefox tells me when I do a Find.

Ok, ok, I did not mean to search a literal string, just the topic.

Graeme Geldenhuys :

Well, with for-in you can only loop through a list is a specific order and direction. You can't peak forward, you can't move back, you can't filter returned content.

At least you know it will terminate. That always was a guaranteed property of the for-loop. With being able to move back and forth as some iterator wishes, this property will be gone forever.

Iterators: Good idea.
Using for-loops for that: Bad idea.

for-loops are for "static" things, be it enumerations, sets, or ranges. I would vote for that, but not for (disguised) iterators in for-loops. Those are more suited for while-loops.

Vinzent.

In our previous episode, Paul Ishenin said:

Yes, I see this very bad too. That's why this discussion is started. But what can be suggested instead?

Does there need something to be suggested per se?

What do you think of the DECAL solution ? It's ackward, but that is because it is pre-generics, and I think generics would solve many of the problems.

I don't like non-constructive criticism. If you don't like something please be ready to suggest something instead. And in this case something what can be used to implement for-in loop for classes and implements this delphi compatible (so 'for S in MyStringList do' will work).

I don't like for-in, but if sb wants to waste time on it, better do it Delphi compat. I don't see the need for anything else, and if people see it, it would be nice to hear what they think of the decal principle (but then fixed with generics)

The Prism solution might be overly costly since it depends on a continuation concept.

Paul Ishenin wrote

I don't like non-constructive criticism. If you don't like something please be ready to suggest something instead. And in this case something what can

By no means to I suggest that my code should be the final product, but here I am attaching what I wrote for the Iterator article I previously mentioned.

Please read the accompanied article as to why I did what I did. Due to the iterators not being part of the internal list classes and because I did not what to create descendant list classes for everything, I followed the following approach.

I have used a Factory method to determine which Iterator should be returned, based on the class type in question. I have also not implement all suggested iterator interface methods - only a select few are currently implement. I also followed the Java iterator principle (different from Microsoft's idea) that the iterator pointer points between elements and make a lot more sense to me
when you start the iteration (currently pointing at nothing because the list could be empty) and when you do things like inserting elements.

Please see attached code.

Note:
If iterators get integrated with all list classes, then the factor method could disappear and a few other stuff could be simplified.

Marc Weustink :

[...]

This is IMO similar to using a for loop vs. using a while loop. In a for loop you are not allowed to modify the loop variable. In this case I can imagine its not allowed to modify the list.

But how's the compiler supposed to know which method modifies MyList and which one does not? Disallowing all calls to the methods of MyList doesn't seem a proper solution, if you're not allowed to do that what is the sense of iterating through it? Another option: Raise an exception if the code detects that the list changed during the loop? Can it be guaranteed that a changed list is detected (usually the answer is: No). At what run-time costs can it be detected at all? Etc. pp.

There seems more to that than just syntactic sugar.

Vinzent.

Graeme Geldenhuys wrote

In all respects, Delphi Prism is not a Embarcadero product - except for the slapped on "Delphi" prefix, Embarcadero has nothing to do with Delphi Prism.

They do sell it, so I suppose if they plan language extension (like iterators) for the native Delphi flavor, they will do it along the lines of Prism.

-Michael

Michael Schnell wrote

They do sell it,

As a curtsey because they gave up on Delphi.NET.
RemObjects are now in control of Delphi Prism.

Graeme Geldenhuys wrote

RemObjects are now in control of Delphi Prism.

No wonder, as they created it :)

I understand that Codegear or Embarcadero pay(ed) them for the license to sell it.

-Michael

That's why I suggested the use of 'const functions' in other message in this thread. IIRC they are also part of C++; const functions are functions that never do an assignment (directly or indirectly) to a member field.

Regards, Micha

+1

"using" seems far better than a "fixed name GetIterator procedure".

IMHO the other option is to implement the feature for ase types (e.g. enum, array, maybe string) and allow operator overriding for all other types.

Then you can specify an enum operator for classes, or even for simple/scalar types like integer, and define your for..in behaviour for them.

Martin

Good idea. What is iterator internally? Is this an object with the special header?

Is it internall the same as:
TListIterator = object
  function init_func(AList: TList): Boolean;
  function next_func: Pointer;
  function check_func: Boolean;
end;

Best regards,
Paul Ishenin.

Yes, 'using' and 'operator GetEnumerator' are suggested in the wiki page here: http://wiki.lazarus.freepascal.org/for-in_loop#Proposed_extensions So at least here we are thinking the same :)

The big problem question is "what is enumerator" :

a) A new type very like to a specially defined object. If so then will we allow it usage outside for-in loop?
b) A class with one magic method and one magic property c) something else

Best regards,
Paul Ishenin.

Paul Ishenin пишет:

Don't comlicate it more than necessary:

1) An object (or record with methods, provided you implement it first :-) is preferable over class, since using it can help avoiding construction/destruction, memory allocations, and implicit try..finally.
2) Using objects/records also enables you to use operators to encode the iterator. This isn't possible for classes.
3) next_func and check_func above can be coalesced to a single

   function next(out obj: TItem): Boolean;

4) The syntax 'for element in list using TListIterator' is kind of redundant. I'd suggest:
for element in list.GetForwardIterator do ..., or even for element in list.GetSubItems(arguments) do ...

Regards,
Sergei

-------- Original-Nachricht --------

Yes, I read that later on. Is that implemented in FPC? Actually I read of that feature the first time, but I like it already. ;) In that case this part of my argument may be ignored, of course. OTOH I've never stumbled upon source code using it, so it may seem not widespread enough yet to be useable (basic iterable classes like TList spring to mind)...?

(Currently, I'm reading my mails online, so I answer one at a time at the risk of repeating somebody else's argument. Sorry if that causes inconviniences.)

Vinzent.

Sergei Gorelkin :

4) The syntax 'for element in list using TListIterator' is kind of redundant. I'd suggest: for element in list.GetForwardIterator do ..., or even for element in list.GetSubItems(arguments) do ...

Wat's the return type of those functions? TIterator?

Vinzent.

Vinzent Höfler пишет:

Probably yes. A possibility to use any type which has certain methods/operators instead looks attractive, but I'm afraid it would be hard to fit that into existing compiler model.

Sergei.

Sergei Gorelkin wrote

Probably yes. A possibility to use any type which has certain methods/operators instead looks attractive, but I'm afraid it would be hard to fit that into existing compiler model.

Sorry, what is TIterator? Is this like to the previosly suggested

TIterator = interator(<Type>, MoveNext, Current)
end;

Btw, we can use some new 'enumerator'/'iterator' directive to mark methods which return enumerators.

And only in delphi mode we will search for the GetEnumerator method.

Best regards,
Paul Ishenin.

There is no much threre regards 'for-in'. I would like if you/somebody else improve the current wiki 'for-in' article with your/somebody else ideas.

Best regards,
Paul Ishenin.

I did not try to specify what exactly TIterator is. What I tried to say is: "some type that is already known at the point of parsing the for..in".

The question is, what advantage all this specific syntax could give over simple searching the methods by name?

Sergei

Sergei Gorelkin wrote

The question is, what advantage all this specific syntax could give over simple searching the methods by name?

Some people need less compiler magic, some does not care. New directive can reduce the magic level :)

Btw, we use similar compiler magic every day. Just think of:

1. functions which all have special 'Result' variable :)
2. message methods which calls 'TObject.DefaultHandler' when you call inherited inside.
3. tobject methods related to interfaces

I suspect there are more examples.

In all the listed examples compiler searches for the special identifier. Therefore GetEnumerator identifier looks as very native solution for pascal.

Best regards,
Paul Ishenin.

The more I think about the more headaches it causes. Now I agree it needs to be done for delphi mode

If you do it for objpas/fpc mode it starts the question how far? Let's start with the simpliest for-in, and add each next step, and someone please tell me how far for-in shopuld go...
- for in for ranges [1..10], sets and enum types
- for arrays
- for strings
- for any object via iterator
- with reverse iteration...
=> Now reverse iteration can be done by a different iterator class,
- with filtered iteration...
=> what if you want to iterate through all entries of a list (stringlist) that have "length(entry) > 10" ?
To do that (and specify differnt filters) you need to instantiate the iterator yourself, and set the values for the filter condition. But if you pass an instantiated iterator to for-in => should it still auto destroy it? And if not, you are back to "while (iterator)"

I don't think the last one is really desired... but where to draw the line? Except for the last one: does iterator have to be a class or object at all?

I hadn't had time to think it all through => so I just throw some half finish thoughts in... maybe ....

It could be done as an overloaded operator => a function with an additional var-param that is used for the state
operator ITR = function IterateFoo(FooValue: TFooType; ItrState: Pointer; out ItrNoMoreValues: Boolean): FooEntryType

for-in would call it the first time with a nil for ItrState => the function can then store a number, or create a StateObject, or whatever to remember it;s position.
For all other calls to the function, the param would be passed back in, as it was set on the previous call. Just as if it was a local variable in the calling code)
- It has a serious flaw in this version => if the loop is aborted early "break", then there is no way to tell the iterator. If it was an object then it would just get destroyed. It would need the last param to be var as well, and be passed in true, for required stop
- And of course this would reduce the possibilities a lot, and be far less elegant. (but then again where in the above list should the line be drawn?)
- on the pro:
- It is the simpliest solution. Some iterators do not need to be an object.
- For Iterators that need to be an object, it can act as a wrapper and store it in pointer
- "using" could still be done, but need to point to a function of this type
- no hardcoded function/method names (see below)
- compatible with how other operators are overloaded (as functions)

The problem I have with the iterator being an object is that it would currently end up with having certain method names (such as "next") hardcoded in the compiler => I dislike this idea. (actually "dislike" is understating it)
So if the iterator was to be an class/object, there would first be a need to introduce keywords or modifiers, that allow the programmer to mark the correct methods for the operation.

My 2 cent
Martin

Note that all these cases are introduced by Borland, and they are not exactly an example of clean design. They are is what I referred to when I said that the compiler already relies too much on TObject/IInterface methods.

Ideally, the compiler has no knowledge at all of specific classes, and a new keyword such as Iterator (or whatever) helps in ensuring that the compiler is not contaminated with knowledge of specific classes or methods.

Michael.

I don't think so; but I think it would be a useful part of the iterator proposal. The iterator case really gives it some strength. If not used by other features it may seem more like a "OO-purity" thing (although that is not per definition bad) ;-).

Micha

Michael Van Canneyt wrote

Ideally, the compiler has no knowledge at all of specific classes, and a new keyword such as Iterator (or whatever) helps in ensuring that the compiler is not contaminated with knowledge of specific classes or methods.

That is a very conservative (but of course viable) view on a language. Plain old C is clean in that way, but I suppose even C++ (which I never use) is not.

I have the impression that modern implementation of languages increasingly do provide "syntax sugar" to improve the usability of certain library functions.

-Michael

Michael Van Canneyt wrote

Ideally, the compiler has no knowledge at all of specific classes, and a new keyword such as Iterator (or whatever) helps in ensuring that the compiler is not contaminated with knowledge of specific classes or methods.

I'm not sure how things are helped by "slapping" a keyword onto it? If we call the thing a "class" then it's evil, but if it's called an "iterator" which is actually the exact same thing as a class, it's fine? Correct me if wrong.

Micha

Because with something like

Type
   MyIterator = Iterator(TSomeResultType,Func1,Func2,Func3);
Function MyIterator.Func1 : TSomeResultType;
begin
end;

Etc.

You're free to choose the names of the functions, so nothing has to be hardcoded in the compiler. It's a bit like operators, where you can also choose the names of the operands.

Michael.

Michael Van Canneyt wrote

Because with something like Type MyIterator = Iterator(TSomeResultType,Func1,Func2,Func3);

So the place in this list determines its function?

Micha

The syntax is nice and simple, but I would have to agree. Maybe a two step would be better? Like with converting an enum to a set?

type
  MyIterator = Iterator of TSomeResultType;
  MyIteratiorDefinition = class(MyIterator) //class might be another keyword
    iteration Func1; MoveNext;
    iteration Func1; MovePrior;
    {etc}
  end;

and then tag methods as so? I think though, that the original suggestion would work if very well documented or allowing for additional tags somehow?

MyIterator = Iterator(TSomeResultType, Func1::Next, Func2::Prior);

The list could also then be variable. If it isn't variable with default implementations for missing members, I don't see the reason why the method names can't be fixed - or am I missing something obvious? Having a fixed list may well be confusing should it ever need to be expanded or adapted.

M

Afaik Delphi's implementation only depends on the guid: it's the same as interface ref. counting.

Yes. Just like in an operator...

Michael.

Nono, A::B is very unpascalish. The location uniquely determines whatever you need to make the iterator work.

Just like in operators:

operator + (A : Real; B TSomeRecord) : Z : TSomeRecord;

A,B, and Z have fixed positions and meanings. Only the names vary.

1. The whole point of the exercise is to avoid fixed class or method names.

2. You need only 2 or 3 methods. A for loop in C also has 3 "fields" or "statements" or wwhatever, and the location determines univocally what it means.

It's pascal. If we really must introduce iterators - I still think they are unnecessary bloat - Let's at least keep it simple and clean and similar to existing features.

Michael.

Michael Van Canneyt wrote

On Wed, 21 Oct 2009, Micha Nelissen wrote: > So the place in this list determines its function? Yes. Just like in an operator...

Hmm that's not comparable, for operators it's much more intuitive what
to expect as the context is forced, Result := <expr> <op> <expr>;

Micha

A loop also has a 'forced' context of 3 expressions:

for i:=expr1 to expr2 do
   begin
   // Implicit I:=I+1;
   end;

C makes this even more explicit.

This is all just nitpicking. Iterators as a language construct are a very ugly hack to save some typing, no matter how you turn it. Nice maybe for languages with dynamic typing and so on, but really not on it's place in Pascal.

Michael.

In our previous episode, Michael Van Canneyt said:

I've a bit doubts, even aside from any direct opinion on the feature itself, to add functionality to emulate certain features from other language's very extensive libraries (like Java/.NET/Boost), since the development of the last very major feature (and its use), generics has stalled.

You can always let development occur in a branch (such as with Paul's changes currently) and only merge to trunk once it's more or less finished.

Jonas

Michael Van Canneyt wrote

This is all just nitpicking. Iterators as a language construct are a very ugly hack to save some typing, no matter how you turn it. Nice maybe for languages with dynamic typing and so on, but really not on it's place in Pascal.

I beg to disagree.This is completely backwards -- like saying that "while" loop is an
ugly hack to save some typing of "goto" operators. Iterators and foreach loops are very important tools of structured coding. The fact that "for..in" may be expressed in terms of lower-level primitives does not decrease its importance, just like existence of "goto" does not decrease
the importance of "while".

In our previous episode, Jonas Maebe said:

That goes for the language feature maybe. The point is to make it worthwhile, and used, it has to be integrated into everything. Which, as that would introduce Delphi incompatibilities probably won't.

Just like generics that still consist mostly out of FGL.

Alexander Klenin пишет:

One can always say vice versa, that 'goto' is an ugly hack and 'while' is the solution.

In general, the more I read the discussion, the more I agree with Michael. We should actually discuss:
1) What is the problem we need to solve
2) What existing solutions already exist and can be used
3) What are the advantages and downsides of each existing solution
4) And after that, we could come up to a design of the new feature.

Instead, the discussion is almost ultimately about how to modify the language.

The strategic view of the situation is also very desirable. For example, things can change considerably once we have generics are completed, and maybe there's simply no need trying to fit the iterators into today's state.

Regards,
Sergei

Well, I think you can't compare the two cases.

I see little gain in changing

   while Something(f) do
     F.Somethingelse

to

   For f in Something do
     F.SomethingElse

The number of lines is equal, the amount of typed characters also is equal. It's not more readable, either, IMHO.

When there is a big difference between the following goto and the while loop it is supposed to be equal to:

   :jumphere;
   If Something(F) then
     begin
     F.SomeThingElse;
     Goto jumphere;
     end;

Here you need quite a few more lines and even an extra label definition, and I don't think anyone will dispute that the while loop is more readable...

Michael.

Michael Van Canneyt wrote

I see little gain in changing while Something(f) do F.Somethingelse

This is not quite equal, it's more like:

Start(f);
while not Last(f) do
   F.DoWork;

In your case, the function 'Something' must know about a generic F.

There are also recursive state problems to be thought about. I guess this is why in the STL an iterator is a separate object on the local stack, but able to iterate through (another) object of the defined type.

Micha

Ok. There is not a problem to make this feature available only for the delphi mode. Moreover it is much easier to implement them without all that extensions.

Best regards,
Paul Ishenin.

I understand what this thread is very long and hard to follow. However, could you please at least read direct the answers to your mails:

There is another thing:

The main advantage of "while" as compared to "if + goto" is not the code size (there are times when using "goto" leads to shorter code), but in the fact that it represents a more structured approach to coding, reducing a probability of mistakes and easing code maintenance. This is the main argument in favor of "for..in" as compared to "try + while + manual iterator calling".

Generics still don't work 100%, so it's normal that they are not widely used. I also don't don't have a problem with a language feature that's available but not widely used, as long as it doesn't complicate compiler development/maintenance too much.

Jonas

The above is only for classes. No-one said that the iterator must be a class (e.g. objects are stored on the stack), so it can be reduced to:

var
   it: TSomethingIterator;
   it := SomeClass.Iterator;
   while it.HaveValue do
      it.NextValue.Something;

If the try/finally is not needed (see above), then your argument is almost void, and IMHO not worth the effort.

Let's be serious: it should be clear that I am not a fan of iterators
- just like I think generics is a waste of time - but it looks like they will be implemented, since there seems to be quite some momentum built up in favour of them. Just as it was for generics.

My only worry now is to make sure that if they are implemented, that we make the design as clean as possible: e.g. No hardcoded dependencies on class or interface names.

Michael.

In our previous episode, Michael Van Canneyt said:

Lightcontainers does something like this, but uses function syntax based on records. HaveValue and NextValue are inline.

One could see such constructs using D2006 methods-in-records syntax though, and afaik I have even seen it in the Delphi fora.

Michael Van Canneyt wrote

My only worry now is to make sure that if they are implemented, that we make the design as clean as possible: e.g. No hardcoded dependencies on class or interface names.

We need to count the pros and contras first regards hardcoded names and maybe 'hard coded' code. For example for me the hardcoded 'Result' identifier for all functions is a big advantage. But you are still using the function names to set the result values I suppose. So during this clean design designing we need to find a compromise between 2 approaches.

for example, would you argue if iterator will be any container type: object/class with the next declaration:

TMyIterator = object
public
  function StepNext: Boolean;
  property TheCurrentValue: Integer;
  iterator MoveNext = StepNext;
  iterator Current = TheCurrentValue;
end;

When 'iterator MoveNext' or 'iterator Current' are not defined then compiler uses default hardcoded identifiers 'MoveNext' and 'Current'.

This is very like to how you can bind interface methods to the class methods if they are differ.

Best regards,
Paul Ishenin.

Hm. I like this direction of thinking, yes...

What about

   function StepNext: Boolean; iterator 'movenext';
   property TheCurrentValue: Integer; iterator 'current';

or better yet, because it is more strict:

   function StepNext: Boolean; iterator nextvalue;
   property TheCurrentValue: Integer; iterator currentvalue;

In this way, it's more like some extra modifiers.
(so like safecall, default, stored and whatnot)

It just adds some modifiers, and we're free of hard-coded names. With one of these I could live :-)

Michael.

I agree that _allowing_ iterators to be "plain" objects is useful, in particular as
an optimization technique (and Delphi supports that). However, I am against _requiring_ that. Classes is an important feature of Delphi language, and I see no reason to arbitrarily forbid iterators to be class objects.

Nevertheless, using "plain" objects does not make any difference in the replacement code size,
since plain objects have destructors too. Unless you go C++ way and guarantee the calling of destructor for local objects at the end of the block, try..finally block is still required.

My only worry now is to make sure that if they are implemented, that we make the design as clean as possible: e.g. No hardcoded dependencies on class or interface names.

As I already argued in this thread, such dependencies may be considered the distinguishing
feature of the current Pascal language, so avoiding them at this stage is strange indeed.

Still, how about this proposal then:

1) In FPC mode, introduce 'iterator' attribute to the class functions, like this:

type TMyIterator = class // or object
  function GetNext: Boolean; iterator 'MoveNext';
  function CurrentValue: TMyType; iterator 'Current';
  funtcion Index: Integer; iterator 'CurrentIndex';
end;

2) In FPC mode, require that in "for a in c do" loop, c have to be the iterator, not the container:
for t in Tree.InOrderIterator

  for ch in Chars(MyString) do
  for a in Elements(MyArray) do

where Chars is a normal function defined in RTL,
and Elements is built-in half-magic function similar to SetLength.

3) In Delphi mode, additionally auto-decorate each function with 'magic' name with corresponding 'iterator' attribute

4) In Delphi mode, if non-iterator is a second argument of "for a in c", try to call function with 'magic' name GetEnumerator to get the default iterator.

Note that this design gets rid of the noting of "default" iterator in FPC mode. If wanted, it can be restored by introducing something like "operator iterator" (aka "operator GetEnumerator" in the current wiki proposal).

Michael Van Canneyt wrote

What about  function StepNext: Boolean; iterator 'movenext';  property TheCurrentValue: Integer; iterator 'current';

You beat me by two minutes ;-)

 function StepNext: Boolean; iterator nextvalue;  property TheCurrentValue: Integer; iterator currentvalue;

I do not care much, but putting names in quotes may slightly simplify parser, IMO.

Having an implicit result variable or Self variable is something completely different than having the knowledge of a class and scanning its function names and assign a function to a certain name.

The first is defined by the compiler, the second is declared by you, the code writer.

Again, here you mix names and functionality.
Compare it to a default propery, you declare it as:

   property Foo[index:integer]: String read GetFoo; default;

and not as

   property Default[index:integer]: String read GetFoo;

Marc

I think exactly the opposite, but I don't think the quotes are so relevant. I have a preference for no quotes, but that is a matter of taste.

With quotes it would more resemble the "message" modifier...

Michael.

As I tried to say earlier, having distinct StepNext() and Current() functions is somewhat redundant, except the purpose of Delphi compatibility. Rationale: the for..in loop manages the iterator object itself, and does not allow user code inside the loop to access the iterator and call its methods. Therefore, it is possible to reduce the whole thing to:

function Next(out value: SomeType): Boolean; iterator;

thus eliminating the need for additional qualifiers after 'iterator'.

Sergei

If this is possible, it is even better...

The main thing is: no hardcoded names...

Michael.

This is slightly better, but IMO does not worth additional incompatibility. Also, your proposal is far less extendable:
1) Delphi iterators have also magic 'Reset' function which is easy to express by attribute "iterator reset",
but hard to express in your proposal
2) See the end of the for-in wiki page for my proposal for optional "iterator index" extension,
which is also non-trivial to do in your proposal.

Reset is probably targeted for using iterators 'standalone'. I don't see how it fits into the for..in loop, which does not provide access to the iterator object.

2) See the end of the for-in wiki page for my proposal for optional "iterator index" extension, which is also non-trivial to do in your proposal.

I doubt in usefulness of the iterator index. The iterators are, by definition, targeted for containters that are not indexable, otherwise just access container elements by index and be done with it. Attempting to introduce the index, at best you get a simple counter, at worst there will be some number dependent on particular iteration method, so that a single element gets different index in different situations. Anyway, if the container is non-indexable, this index has no further use.

Regards,
Sergei

In what way is Generics not 100%?

Dave Copeland.

David B Copeland wrote

In what way is Generics not 100%? Dave Copeland.

For me at least, the gravest problem is a lack of function-level generics.

Micha Nelissen pisze:

why not
 f.start;
 while not f.last do
   f.doWork;

There are also recursive state problems to be thought about. I guess this is why in the STL an iterator is a separate object on the local stack, but able to iterate through (another) object of the defined type.

Second problem is multithreading, when we try to iterate concurrent on the same collection.

Paul Ishenin pisze:

But where is current state of iterator.
next_func need to know old state and then easy move to next, but were state is saved.

second solution receive old result, find it in collection, decode past state and compute next, but it hurt performance

Michael Van Canneyt wrote

Hm. I like this direction of thinking, yes...

Hm. I like that after 100 mails of 'yes, I like them', 'no, I don't like them' we finnaly moved to the initial idea of the tread - design and implementation discussion :)

What about function StepNext: Boolean; iterator 'movenext'; property TheCurrentValue: Integer; iterator 'current';

string are bad.

or better yet, because it is more strict: function StepNext: Boolean; iterator nextvalue; property TheCurrentValue: Integer; iterator currentvalue;

I like this one. Only not 'iterator nextvalue' and 'iterator currentvalue' but 'iterator movenext' and 'iterator current' :)

Or 'enumerator' instead of 'iterator' to be compatible with delphi terminology ;)

Best regards,
Paul Ishenin.

Sergei Gorelkin wrote

2) What existing solutions already exist and can be used

... maybe not meaning "using the implementation code", but just "providing the same syntax".

( I do know that I might be bashed <again> for this. Anyway: )

I think it's always worth to take a of course works along the lines of .NET, but many things can be implemented in the RTL to have them work in a "native" environment. A good and very useful - but supposedly not at all trivial to implement - candidate could be "parallel" and friends.

-Michael

David B Copeland wrote

On Wed, 2009-10-21 at 15:54 +0200, Jonas Maebe wrote: > Generics still don't work 100%, In what way is Generics not 100%?

There are problems with using inheritance, e.g. http://bugs.freepascal.org/view.php?id=12249 and http://svn.freepascal.org/svn/fpc/trunk/tests/test/tgeneric16.pp

Another compiler crash: http://bugs.freepascal.org/view.php?id=12810

It's not possible to use references to not yet fully specialised types, e.g. http://bugs.freepascal.org/view.php?id=11777

Assembler does not work in generics.

There are some problems with inline and generics.

Jonas

Don't forget the remark by Sergei Gorelkin, that in fact a single call could be enough.

Michael.

Michael Van Canneyt wrote

Don't forget the remark by Sergei Gorelkin, that in fact a single call could be enough.

In fact a single call could not be compatible with delphi solution.

Best regards,
Paul Ishenin.

That is so, but none of what is discussed is compatible with Delphi, since Delphi uses hardcoded names, so in Delphi mode you'll have to add some separate checks anyway.

In objfpc code, the single call could be enough.

In each case, you'll always need 2 calls: 1 call in class itself to Create Iterator instance, and the iterator call in the iterator instance.

I suggest you think about it when writing the specs, then we can discuss further...

Michael.

Paul Ishenin wrote

The next feature which we should think of is for-in loop: http://wiki.lazarus.freepascal.org/for-in_loop

At the moment implementation is almost complete and it is possible to test and review it (although many checks still needs to be added). It unites 2 approaches: delphi - search enumerators by symbol names and fpc
- search using operators and method/property modifiers.

Enumerator can be found for type using:

1. GetEnumerator method to the class/object (delphi way). For example:

type
  TSomeClass = class
  public
     function GetEnumerator: TSomeEnumerator;
  end;

2. operator enumrator for any type (fpc extension). For example:

operator enumerator (const s: string): TStringEnumerator;
begin
  Result := TStringEnumerator.Create(s);
end;

Enumerator is a class or object which need to have:

1. function MoveNext: Boolean; and property Current: sometype; (delphi way) For example:

  TStringEnumerator = class
  private
    function GetCurrent: Char; inline;
  public
    function MoveNext: Boolean;
    property Current: Char read GetCurrent;
  end;

2. function with any name and Boolean return type marked by 'enumerator MoveNext' directive and a property with any name marked by 'enumerator Current' directive (fpc extension). For example:

  TStringEnumerator = class
  private
    function GetValue: Char; inline;
  public
    function StepForward: Boolean; enumerator MoveNext;
    property Value: Char read GetValue; enumerator Current;
  end;

Enumerator search for the paticular type performs in the next order:
1) search available operator
2) if it is a object/class instance then search for GetEnumerator method
3) if it is a enumeration/range/array/string/set then use built-in enumerator support for this type

A branch to test is:
http://svn.freepascal.org/svn/fpc/branches/paul/features Enumerator tests are: tests\test\tforin1.pp ... tests\test\tforin7.pp A brief explanation of how it works can be found here: http://wiki.lazarus.freepascal.org/for-in_loop#Proposed_implementation

Suggestions and comments are welcome except the new discussion about enumerators necessity.

Best regards,
Paul Ishenin.

2009/10/24 Paul Ishenin :

Just to recap... What exactly does the function GetEnumerator return? Specifically for list type classes. Also what interface would that enumerator support?

The reason I ask... I would still like the full power of Iterator (as discussed before) for all List type classes in the RTL (similar to my code attachment of 4 days ago). I'm not sure if what you implemented will conflict with what I want to implement in the RTL. The major difference would be that mine is a implementation detail in the list classes, whereas your implementation is a language feature. When I use iterators, I would like to have the ability to use Current, HasNext, HasPrevious, PeakNext, PeakPrevious, Reset, JumpToBack, pass a filter to GetIterator... etc.

Also, I referred to them in my implementation as Iterators - as per the Iterator design pattern. How does Enumerator differ from Iterator?

Graeme Geldenhuys wrote

Just to recap... What exactly does the function GetEnumerator return? Specifically for list type classes. Also what interface would that enumerator support?

Valid return for now is object or class instance. I did not implement interface support yet. But delphi has the next interface for this:

  IEnumerator = interface(IInterface)
    function GetCurrent: TObject;
    function MoveNext: Boolean;
    procedure Reset;
    property Current: TObject read GetCurrent;
  end;

I don't argue but how do you think to use HasPrevious, PeakNext, PeakPrevios, Reset, JumpToBack in the for-in loop? Can you suggest a syntax?

Also, I referred to them in my implementation as Iterators - as per the Iterator design pattern. How does Enumerator differ from Iterator?

Delphi enumerator pattern is a structure with one boolean function MoveNext and one property Current.
More info is here:
http://17slon.com/blogs/gabr/2007/03/fun-with-enumerators.html

Best regards,
Paul Ishenin.

Graeme Geldenhuys wrote

Also, I referred to them in my implementation as Iterators - as per the Iterator design pattern. How does Enumerator differ from Iterator?

I have looked at you interfaces. They are needs to be changed a bit to use then in the for-in loop. For example:

  ITBIterator = interface(IInterface)
  ['{9C2BC10D-54C8-4B59-88B5-A564921CF0E3}']
    function    HasNext: Boolean;
    function    Next: TObject;
    function    HasPrevious: Boolean;
    function    Previous: TObject;
  end;

To use it in the for-in loop you need to add a function MoveNext: Boolean (you can choose any other name but you need to mark it using 'enumerator MoveMext' modifier) and property Current: TObject (you can choose any other name but you need to mark it using 'enumerator Current' modifier).

Look at
http://svn.freepascal.org/svn/fpc/branches/paul/features/tests/test/tforin8.pp and
http://svn.freepascal.org/svn/fpc/branches/paul/features/tests/test/tforin9.pp for examples.

Best regards,
Paul Ishenin.

2009/10/25 Paul Ishenin :

I don't argue but how do you think to use HasPrevious, PeakNext, PeakPrevios, Reset, JumpToBack in the for-in loop? Can you suggest a syntax?

I did not expect for-in loop to support all the methods I require (or use), hence the reason I ask, and why I still would like to extend the list/container classes in the RTL to support those. I googled a bit more and found that java actually supports both Iterator and Enumerator in it's container classes. Clearly there is a use-case for each, and one does not necessarily replace the other.

So if the for-in loop uses GetEnumerator, I can still implement the GetIterator without conflicting with your implementation. Looking more at the Java class documentation, it seems Enumerator is a "lite" version of Iterator an only supports a small subset of Iterator features. This same idea could be applied to the RTL, and for users that don't want to use for-in loop syntax.

Delphi enumerator pattern is a structure with one boolean function MoveNext and one property Current. More info is here:

Thanks for the link, I'll go take a look at it shortly...

IMHO, Reset/JumpToBack should not be used inside "for..in" loop -- they are analogous to "goto" inside for loop.
If "for .. in .. index" extension is implemented, it may be used to call HasPrevious, PeakNext, PeakPrevious and other functions that do not change iterator position.

> Also, I referred to them in my implementation as Iterators - as per > the Iterator design pattern. How does Enumerator differ from Iterator?

It does not. I dislike the "Enumerator" term too, but this is how it is called in Delphi :-(

2009/10/25 Paul Ishenin :

Thanks Paul, now I have a much better idea of for-in loop (I started getting confused during this message thread discussion and lost track). As I mentioned in my previous post, I did not expect my idea of Iterators to work with for-in. Looking at your examples, I still agree with my assessment.

for-in can be considered a "lite" version of Iterators. The for-in syntax however does not replace the need for full Iterator support. By "full Iterator" I mean having a large set of methods that the user can use, like: PeakNext, PeakPrevious, Remove, JumpToBack, etc...

So I will continue with my work of Iterator support in all RTL container classes and use the GetIterator method to create an Iterator instance. This means the user can decide which method they want to use to iterate over their container classes and how much control they need in the looping code. It they only need a front-to-back loop, they can use for-in syntax,but if they want more control, they can request a Iterator via GetIterator method.

BTW:
Reading that Delphi article about Enumerators, I must say that your Implementation seems a lot more flexible by not hard-coding method names etc... Clearly it pays to discuss ideas first before jumping into the implementation. CodeGear certainly can learn something from FPC developers! :-)

Graeme Geldenhuys wrote

use, like: PeakNext, PeakPrevious, Remove, JumpToBack, etc...

I would call it PeekNext, PeekPrevious, ...

Vincent

Alexander Klenin wrote

IMHO, Reset/JumpToBack should not be used inside "for..in" loop -- they are analogous to "goto" inside for loop. If "for .. in .. index" extension is implemented, it may be used to call

As I explained in my previous reply to Paul - I did not expect for-in to support everything I use with Iterators. I simply wanted to find out if the for-in implementation is going to clash with my idea of implementing GetIterator() in RTL's container classes. So far I don't see any problems, and I can't see why we can't implement both. There are use-cases for both. I see for-in as a "lite" version of Iterator, but I still have a need for a more "full" implementation as well - supporting a larger method interface.

It does not. I dislike the "Enumerator" term too, but this is how it is called in Delphi :-(

Java uses both, that is why I asked. But yes, I kind-of expected them to be the same thing, but just wanted to make sure.

Alexander Klenin wrote

IMHO, Reset/JumpToBack should not be used inside "for..in" loop -- they are analogous to "goto" inside for loop. If "for .. in .. index" extension is implemented, it may be used to call

As I explained in my previous reply to Paul - I did not expect for-in to support everything I use with Iterators. I simply wanted to find out if the for-in implementation is going to clash with my idea of implementing GetIterator() in RTL's container classes. So far I don't see any problems, and I can't see why we can't implement both. There are use-cases for both. I see for-in as a "lite" version of Iterator, but I still have a need for a more "full" implementation as well - supporting a larger method interface.

It does not. I dislike the "Enumerator" term too, but this is how it is called in Delphi :-(

Java uses both, that is why I asked. But yes, I kind-of expected them to be the same thing, but just wanted to make sure.

Vincent Snijders wrote

I would call it PeekNext, PeekPrevious, ...

You are correct...

AFAIU, the whole point of the previous discussion is that you do not have to implement "GetIterator" separately from "GetEnumerator" -- you can have the same class filling both roles.

Alexander Klenin wrote

AFAIU, the whole point of the previous discussion is that you do not have to implement "GetIterator" separately from "GetEnumerator" -- you can have the same class filling both roles.

Yes. I think we can use some common interfaces/classes/objects in RTL both for the for-in loop enumerators and as iterators with all that methods which Graeme implemented.

I already created delphi compatible enumerator classes for the basic RTL lists: TFpList, TList, TStrings, TComponent and TCollection. We can inherit all them from the TInterfacedObject and Graeme interfaces. And list classes will have 2 methods: GetEnumerator: TSomeEnumerator and GetIterator: ISomeIterator where TSomeEnumerator implements ISomeIterator interface.

Graeme, what do you think?

Best regards,
Paul Ishenin.

When the discussion started about for-in, I could not clearly see how. But after reading the Delphi article (link you posted earlier) and looking at your examples for testing for-in loop, I think we could defiantly share some common iterator code.

Micha Nelissen :

I thought about it [no, not that long I needed to answer this mail...] and there are certain issues with this:

At first, "const function" would tell us that the function does not change the object in any way, right? But this is bad for an iterator: Suppose, you want to replace some value in your iteration set, it wouldn't be allowed via the "const function" and calling "non-const" functions wouldn't be allowed inside the for-loop, because it could destabilize the iterator.

For an iterator to work properly, the called functions should guaranteed that neither size nor order (perhaps in a stronger version) of the iterated set does change. Other values can be changed without problems, so the garantees made by a "const"-function would be too strong here, I think...

If not used by other features it may seem more like a "OO-purity" thing (although that is not per definition bad) ;-).

It could easily be used as optimization. :) If the function is constant (and does not access global variables), its result can be replaced by the result of a previous call with the same argument (IOW: some notion of a "pure" function).

Vinzent.

Vinzent Höfler wrote

At first, "const function" would tell us that the function does not change the object in any way, right?

What is "the object"? Usually there are 2 objects: the list (TList e.g.) and the object being iterated, e.g. TList.Objects[I]. Those referenced objects can be volatile, the const-ness only applies to the TList. Ideally the iterator state and the state of the items iterated over are separated. (Maybe this is not so with e.g. TStringList).

Micha

In that case, the list, of course.

Usually there are 2 objects: the list (TList e.g.) and the object being iterated, e.g. TList.Objects[I]. Those referenced objects can be volatile, the const-ness only applies to the TList.

Yes, that's what I thought.

Ideally the iterator state and the state of the items iterated over are separated. (Maybe this is not so with e.g. TStringList).

Suppose, the list items have a property "Owner" and the iterator calls some method that says "Owner.Delete (self);" somewhere. How's the compiler supposed to detect that?

In some languages, this problem is circumvented by creating a copy of the list before iterating the set, but I don't think that this is a viable solution for Pascal (or any compiled language, for that matter).

Vinzent.

Vinzent Hoefler wrote

Suppose, the list items have a property "Owner" and the iterator calls some method that says "Owner.Delete (self);" somewhere. How's the compiler supposed to detect that?

It can't. In general, all references to the (iterator) list need to be "const" at this time, but I guess it's impossible to prove whether some reference is a reference to the particular list being iterated. Let alone a function that "might" be called by an iterating for-in loop.

In some languages, this problem is circumvented by creating a copy of the list before iterating the set, but I don't think that this is a viable solution for Pascal (or any compiled language, for that matter).

That solution seems too costly to me.

Micha

If nobody want to test/review and nobody argue then maybe we can merge the changes? The changes are made so that they almost does not affect the other compiler/rtl code. Therefore there is no risk for the trunk compiler/rtl to be broken by that changes.

Best regards,
Paul Ishenin.

Paul Ishenin wrote

If nobody want to test/review and nobody argue then maybe we can merge the changes? The changes are made so that they almost does not affect the other compiler/rtl code. Therefore there is no risk for the trunk compiler/rtl to be broken by that changes.

I support this -- it is awkward for me to test the branch, since it is not mirrored in Mercurial,
but I will test once it hits the mirror.

Alexander Klenin wrote

I support this -- it is awkward for me to test the branch, since it is not mirrored in Mercurial, but I will test once it hits the mirror.

Ok, the feature is merged to the trunk now.

Best regards,
Paul Ishenin.