Abstract

Python 2.1 added new functionality to support iterators (PEP 234).Iterators have proven to be useful and convenient in many codingsituations. It is noted that the implementation of Python’sfor-loop control structure uses the iterator protocol as ofrelease 2.1. It is also noted that Python provides iterators forthe following builtin types: lists, tuples, dictionaries, strings,and files. This PEP proposes the addition of an iterator for thebuiltin type int (types.IntType). Such an iterator would simplifythe coding of certain for-loops in Python.

BDFL Pronouncement

This PEP was rejected on 17 June 2005 with a note to python-dev.

Much of the original need was met by theenumerate() function whichwas accepted for Python 2.3.

Also, the proposal both allowed and encouraged misuses such as:

>>>foriin3:printi012

Likewise, it was not helpful that the proposal would disable thesyntax error in statements like:

x,=1

Specification

Define an iterator for types.intType (i.e., the builtin type“int”) that is returned from the builtin function “iter” whencalled with an instance of types.intType as the argument.

The returned iterator has the following behavior:

• Assume that object i is an instance oftypes.intType (thebuiltin type int) and that i > 0
• iter(i) returns an iterator object
• said iterator object iterates through the sequence of ints0,1,2,…,i-1

  Example:

  iter(5) returns an iterator object that iterates through thesequence of ints 0,1,2,3,4

• if i <= 0,iter(i) returns an “empty” iterator, i.e., one thatthrows StopIteration upon the first call of its “next” method

In other words, the conditions and semantics of said iterator isconsistent with the conditions and semantics of therange() andxrange() functions.

Note that the sequence 0,1,2,…,i-1 associated with the int i isconsidered “natural” in the context of Python programming becauseit is consistent with the builtin indexing protocol of sequencesin Python. Python lists and tuples, for example, are indexedstarting at 0 and ending at len(object)-1 (when using positiveindices). In other words, such objects are indexed with thesequence 0,1,2,…,len(object)-1

Rationale

A common programming idiom is to take a collection of objects andapply some operation to each item in the collection in someestablished sequential order. Python provides the “for in”looping control structure for handling this common idiom. Casesarise, however, where it is necessary (or more convenient) toaccess each item in an “indexed” collection by iterating througheach index and accessing each item in the collection using thecorresponding index.

For example, one might have a two-dimensional “table” object where onerequires the application of some operation to the first column ofeach row in the table. Depending on the implementation of the tableit might not be possible to access first each row and then eachcolumn as individual objects. It might, rather, be possible toaccess a cell in the table using a row index and a column index.In such a case it is necessary to use an idiom where one iteratesthrough a sequence of indices (indexes) in order to access thedesired items in the table. (Note that the commonly usedDefaultTableModel class in Java-Swing-Jython has this very protocol).

Another common example is where one needs to process two or morecollections in parallel. Another example is where one needs toaccess, say, every second item in a collection.

There are many other examples where access to items in acollection is facilitated by a computation on an index thusnecessitating access to the indices rather than direct access tothe items themselves.

Let’s call this idiom the “indexed for-loop” idiom. Someprogramming languages provide builtin syntax for handling thisidiom. In Python the common convention for implementing theindexed for-loop idiom is to use the builtinrange() orxrange()function to generate a sequence of indices as in, for example:

forrowcountinrange(table.getRowCount()):printtable.getValueAt(rowcount,0)

or

forrowcountinxrange(table.getRowCount()):printtable.getValueAt(rowcount,0)

From time to time there are discussions in the Python communityabout the indexed for-loop idiom. It is sometimes argued that theneed for using therange() orxrange() function for this designidiom is:

• Not obvious (to new-to-Python programmers),
• Error prone (easy to forget, even for experienced Pythonprogrammers)
• Confusing and distracting for those who feel compelled to understandthe differences and recommended usage ofxrange() vis-a-visrange()
• Unwieldy, especially when combined with thelen() function,i.e.,xrange(len(sequence))
• Not as convenient as equivalent mechanisms in other languages,
• Annoying, a “wart”, etc.

And from time to time proposals are put forth for ways in whichPython could provide a better mechanism for this idiom. Recentexamples includePEP 204, “Range Literals”, andPEP 212, “LoopCounter Iteration”.

Most often, such proposal include changes to Python’s syntax andother “heavyweight” changes.

Part of the difficulty here is that advocating new syntax impliesa comprehensive solution for “general indexing” that has toinclude aspects like:

• starting index value
• ending index value
• step value
• open intervals versus closed intervals versus half opened intervals

Finding a new syntax that is comprehensive, simple, general,Pythonic, appealing to many, easy to implement, not in conflictwith existing structures, not excessively overloading of existingstructures, etc. has proven to be more difficult than one mightanticipate.

The proposal outlined in this PEP tries to address the problem bysuggesting a simple “lightweight” solution that helps the mostcommon case by using a proven mechanism that is already available(as of Python 2.1): namely, iterators.

Because for-loops already use “iterator” protocol as of Python2.1, adding an iterator for types.IntType as proposed in this PEPwould enable by default the following shortcut for the indexedfor-loop idiom:

forrowcountintable.getRowCount():printtable.getValueAt(rowcount,0)

The following benefits for this approach vis-a-vis the currentmechanism of using therange() orxrange() functions are claimedto be:

• Simpler,
• Less cluttered,
• Focuses on the problem at hand without the need to resort tosecondary implementation-oriented functions (range() andxrange())

And compared to other proposals for change:

• Requires no new syntax
• Requires no new keywords
• Takes advantage of the new and well-established iterator mechanism

And generally:

• Is consistent with iterator-based “convenience” changes alreadyincluded (as of Python 2.1) for other builtin types such as:lists, tuples, dictionaries, strings, and files.

Backwards Compatibility

The proposed mechanism is generally backwards compatible as itcalls for neither new syntax nor new keywords. All existing,valid Python programs should continue to work unmodified.

However, this proposal is not perfectly backwards compatible inthe sense that certain statements that are currently invalidwould, under the current proposal, become valid.

Tim Peters has pointed out two such examples:

1. The common case where one forgets to includerange() orxrange(), for example:

   forrowcountintable.getRowCount():printtable.getValueAt(rowcount,0)

   in Python 2.2 raises a TypeError exception.

   Under the current proposal, the above statement would be validand would work as (presumably) intended. Presumably, this is agood thing.

   As noted by Tim, this is the common case of the “forgottenrange” mistake (which one currently corrects by adding a calltorange() orxrange()).

2. The (hopefully) very uncommon case where one makes a typingmistake when using tuple unpacking. For example:

   x,=1

   in Python 2.2 raises aTypeError exception.

   Under the current proposal, the above statement would be validand would set x to 0. The PEP author has no data as to howcommon this typing error is nor how difficult it would be tocatch such an error under the current proposal. He imaginesthat it does not occur frequently and that it would berelatively easy to correct should it happen.

Issues

Extensive discussions concerningPEP 276 on the Python interestmailing list suggests a range of opinions: some in favor, someneutral, some against. Those in favor tend to agree with theclaims above of the usefulness, convenience, ease of learning,and simplicity of a simple iterator for integers.

Issues withPEP 276 include:

• Using range/xrange is fine as is.

  Response: Some posters feel this way. Other disagree.

• Some feel that iterating over the sequence “0, 1, 2, …, n-1”for an integer n is not intuitive. “for i in 5:” is considered(by some) to be “non-obvious”, for example. Some dislike thisusage because it doesn’t have “the right feel”. Some dislike itbecause they believe that this type of usage forces one to viewintegers as a sequences and this seems wrong to them. Somedislike it because they prefer to view for-loops as dealingwith explicit sequences rather than with arbitrary iterators.

  Response: Some like the proposed idiom and see it as simple,elegant, easy to learn, and easy to use. Some are neutral onthis issue. Others, as noted, dislike it.

• Is it obvious thatiter(5) maps to the sequence 0,1,2,3,4?

  Response: Given, as noted above, that Python has a strongconvention for indexing sequences starting at 0 and stopping at(inclusively) the index whose value is one less than the lengthof the sequence, it is argued that the proposed sequence isreasonably intuitive to the Python programmer while being usefuland practical. More importantly, it is argued that once learnedthis convention is very easy to remember. Note that the docstring for the range function makes a reference to thenatural and useful association betweenrange(n) and the indicesfor a list whose length is n.

• Possible ambiguity

  foriin10:printi

  might be mistaken for

  foriin(10,):printi

  Response: This is exactly the same situation with strings incurrent Python (replace 10 with ‘spam’ in the above, forexample).

• Too general: in the newest releases of Python there arecontexts – as with for-loops – where iterators are calledimplicitly. Some fear that having an iterator invoked foran integer in one of the context (excluding for-loops) mightlead to unexpected behavior and bugs. The “x, = 1” examplenoted above is an a case in point.

  Response: From the author’s perspective the examples of theabove that were identified in thePEP 276 discussions didnot appear to be ones that would be accidentally misusedin ways that would lead to subtle and hard-to-detect errors.

  In addition, it seems that there is a way to deal with thisissue by using a variation of what is outlined in thespecification section of this proposal. Instead of addingan__iter__ method to class int, change the for-loop handlingcode to convert (in essence) from

  foriinn:# when isinstance(n,int) is 1

  to

  foriinxrange(n):

  This approach gives the same results in a for-loop as an__iter__ method would but would prevent iteration on integervalues in any other context. Lists and tuples, for example,don’t have__iter__ and are handled with special code.Integer values would be one more special case.

• “i in n” seems very unnatural.

  Response: Some feel that “i in len(mylist)” would be easilyunderstandable and useful. Some don’t like it, particularlywhen a literal is used as in “i in 5”. If the variantmentioned in the response to the previous issue is implemented,this issue is moot. If not, then one could also address thisissue by defining a__contains__ method in class int that wouldalways raise a TypeError. This would then make the behavior of“i in n” identical to that of current Python.

• Might dissuade newbies from using the indexed for-loop idiom whenthe standard “for item in collection:” idiom is clearly better.

  Response: The standard idiom is so nice when it fits that itneeds neither extra “carrot” nor “stick”. On the other hand,one does notice cases of overuse/misuse of the standard idiom(due, most likely, to the awkwardness of the indexed for-loopidiom), as in:

  foriteminsequence:printsequence.index(item)

• Why not propose even bigger changes?

The majority of disagreement withPEP 276 came from those whofavor much larger changes to Python to address the more generalproblem of specifying a sequence of integers where sucha specification is general enough to handle the starting value,ending value, and stepping value of the sequence and alsoaddresses variations of open, closed, and half-open (half-closed)integer intervals. Many suggestions of such were discussed.

These include:

• adding Haskell-like notation for specifying a sequence ofintegers in a literal list,
• various uses of slicing notation to specify sequences,
• changes to the syntax of for-in loops to allow the use ofrelational operators in the loop header,
• creation of an integer-interval class along with methods thatoverload relational operators or division operatorsto provide “slicing” on integer-interval objects,
• and more.

It should be noted that there was much debate but not anoverwhelming consensus for any of these larger-scale suggestions.

Clearly,PEP 276 does not propose such a large-scale changeand instead focuses on a specific problem area. Towards theend of the discussion period, several posters expressed favorfor the narrow focus and simplicity ofPEP 276 vis-a-vis the moreambitious suggestions that were advanced. There did appear to beconsensus for the need for a PEP for any such larger-scale,alternative suggestion. In light of this recognition, details ofthe various alternative suggestions are not discussed here further.

Implementation

An implementation is not available at this time but is expectedto be straightforward. The author has implemented a subclass ofint with an__iter__ method (written in Python) as a means to testout the ideas in this proposal, however.

Copyright

This document has been placed in the public domain.