Abstract

For Python 3,PEP 3106 changed the design of thedict builtin and themapping API in general to replace the separate list based and iterator basedAPIs in Python 2 with a merged, memory efficient set and multiset viewbased API. This new style of dict iteration was also added to the Python 2.7dict type as a new set of iteration methods.

This means that there are now 3 different kinds of dict iteration that mayneed to be migrated to Python 3 when an application makes the transition:

• Lists as mutable snapshots:d.items() ->list(d.items())
• Iterator objects:d.iteritems() ->iter(d.items())
• Set based dynamic views:d.viewitems() ->d.items()

There is currently no widely agreed best practice on how to reliably convertall Python 2 dict iteration code to the common subset of Python 2 and 3,especially when test coverage of the ported code is limited. This PEPreviews the various ways the Python 2 iteration APIs may be accessed, andlooks at the available options for migrating that code to Python 3 by way ofthe common subset of Python 2.6+ and Python 3.0+.

The PEP also considers the question of whether or not there are anyadditions that may be worth making to Python 3.5 that may ease thetransition process for application code that doesn’t need to worry aboutsupporting earlier versions when eventually making the leap to Python 3.

PEP Withdrawal

In writing the second draft of this PEP, I came to the conclusion thatthe readability of hybrid Python 2/3 mapping code can actually be bestenhanced by better helper functions rather than by making changes toPython 3.5+. The main value I now see in this PEP is as a clear recordof the recommended approaches to migrating mapping iteration code fromPython 2 to Python 3, as well as suggesting ways to keep things readableand maintainable when writing hybrid code that supports both versions.

Notably, I recommend that hybrid code avoid calling mapping iterationmethods directly, and instead rely on builtin functions where possible,and some additional helper functions for cases that would be a simplecombination of a builtin and a mapping method in pure Python 3 code, butneed to be handled slightly differently to get the exact same semantics inPython 2.

Static code checkers like pylint could potentially be extended with anoptional warning regarding direct use of the mapping iteration methods ina hybrid code base.

Mapping iteration models

Python 2.7 provides three different sets of methods to extract the keys,values and items from adict instance, accounting for 9 out of the18 public methods of thedict type.

In Python 3, this has been rationalised to just 3 out of 11 public methods(as thehas_key method has also been removed).

>>>d=dict(a=1)>>>forkind:...deld[k]...Traceback (most recent call last):  File"<stdin>", line1, in<module>RuntimeError:dictionary changed size during iteration

>>>d=dict(a=1)>>>fork,vind.items():...deld[k]...Traceback (most recent call last):  File"<stdin>", line1, in<module>RuntimeError:dictionary changed size during iteration

Migrating directly to Python 3

The2to3 migration tool handles direct migrations to Python 3 inaccordance with the semantic equivalents described above:

• d.keys() ->list(d.keys())
• d.values() ->list(d.values())
• d.items() ->list(d.items())
• d.iterkeys() ->iter(d.keys())
• d.itervalues() ->iter(d.values())
• d.iteritems() ->iter(d.items())
• d.viewkeys() ->d.keys()
• d.viewvalues() ->d.values()
• d.viewitems() ->d.items()

Rather than 9 distinct mapping methods for iteration, there are now only the3 view methods, which combine in straightforward ways with the two relevantbuiltin functions to cover all of the behaviours that are available asdict methods in Python 2.7.

Note that in many casesd.keys() can be replaced by justd, but the2to3 migration tool doesn’t attempt that replacement.

The2to3 migration tool alsodoes not provide any automatic assistancefor migrating references to these objects as bound or unbound methods - itonly automates conversions where the API is called immediately.

Migrating to the common subset of Python 2 and 3

When migrating to the common subset of Python 2 and 3, the abovetransformations are not generally appropriate, as they all either result inthe creation of a redundant list in Python 2, have unexpectedly differentsemantics in at least some cases, or both.

Since most code running in the common subset of Python 2 and 3 supportsat least as far back as Python 2.6, the currently recommended approach toconversion of mapping iteration operation depends on two helper functionsfor efficient iteration over mapping values and mapping item tuples:

• d.keys() ->list(d)
• d.values() ->list(itervalues(d))
• d.items() ->list(iteritems(d))
• d.iterkeys() ->iter(d)
• d.itervalues() ->itervalues(d)
• d.iteritems() ->iteritems(d)

Bothsix andfuture.utils provide appropriate definitions ofitervalues() anditeritems() (along with essentially redundantdefinitions ofiterkeys()). Creating your own definitions of thesefunctions in a custom compatibility module is also relativelystraightforward:

try:dict.iteritemsexceptAttributeError:# Python 3defitervalues(d):returniter(d.values())defiteritems(d):returniter(d.items())else:# Python 2defitervalues(d):returnd.itervalues()defiteritems(d):returnd.iteritems()

The greatest loss of readability currently arises when converting code thatactuallyneeds the list based snapshots that were the default in Python2. This readability loss could likely be mitigated by also providinglistvalues andlistitems helper functions, allowing the affectedconversions to be simplified to:

• d.values() ->listvalues(d)
• d.items() ->listitems(d)

The corresponding compatibility function definitions are as straightforwardas their iterator counterparts:

try:dict.iteritemsexceptAttributeError:# Python 3deflistvalues(d):returnlist(d.values())deflistitems(d):returnlist(d.items())else:# Python 2deflistvalues(d):returnd.values()deflistitems(d):returnd.items()

With that expanded set of compatibility functions, Python 2 code wouldthen be converted to “idiomatic” hybrid 2/3 code as:

• d.keys() ->list(d)
• d.values() ->listvalues(d)
• d.items() ->listitems(d)
• d.iterkeys() ->iter(d)
• d.itervalues() ->itervalues(d)
• d.iteritems() ->iteritems(d)

This compares well for readability with the idiomatic pure Python 3code that uses the mapping methods and builtins directly:

• d.keys() ->list(d)
• d.values() ->list(d.values())
• d.items() ->list(d.items())
• d.iterkeys() ->iter(d)
• d.itervalues() ->iter(d.values())
• d.iteritems() ->iter(d.items())

It’s also notable that when using this approach, hybrid code wouldneverinvoke the mapping methods directly: it would always invoke either abuiltin or helper function instead, in order to ensure the exact samesemantics on both Python 2 and 3.

Migrating from Python 3 to the common subset with Python 2.7

While the majority of migrations are currently from Python 2 either directlyto Python 3 or to the common subset of Python 2 and Python 3, there are alsosome migrations of newer projects that start in Python 3 and then lateradd Python 2 support, either due to user demand, or to gain access toPython 2 libraries that are not yet available in Python 3 (and porting themto Python 3 or creating a Python 3 compatible replacement is not a trivialexercise).

In these cases, Python 2.7 compatibility is often sufficient, and the 2.7+only view based helper functions provided byfuture.utils allow the bareaccesses to the Python 3 mapping view methods to be replaced with code thatis compatible with both Python 2.7 and Python 3 (note, this is the onlymigration chart in the PEP that has Python 3 code on the left of theconversion):

• d.keys() ->viewkeys(d)
• d.values() ->viewvalues(d)
• d.items() ->viewitems(d)
• list(d.keys()) ->list(d)
• list(d.values()) ->listvalues(d)
• list(d.items()) ->listitems(d)
• iter(d.keys()) ->iter(d)
• iter(d.values()) ->itervalues(d)
• iter(d.items()) ->iteritems(d)

As with migrations from Python 2 to the common subset, note that the hybridcode ends up never invoking the mapping methods directly - it only callsbuiltins and helper methods, with the latter addressing the semanticdifferences between Python 2 and Python 3.

Possible changes to Python 3.5+

The main proposal put forward to potentially aid migration of existingPython 2 code to Python 3 is the restoration of some or all of thealternate iteration APIs to the Python 3 mapping API. In particular,the initial draft of this PEP proposed making the following conversionspossible when migrating to the common subset of Python 2 and Python 3.5+:

• d.keys() ->list(d)
• d.values() ->list(d.itervalues())
• d.items() ->list(d.iteritems())
• d.iterkeys() ->d.iterkeys()
• d.itervalues() ->d.itervalues()
• d.iteritems() ->d.iteritems()

Possible mitigations of the additional language complexity in Python 3created by restoring these methods included immediately deprecating them,as well as potentially hiding them from thedir() function (or perhapseven defining a way to makepydoc aware of function deprecations).

However, in the case where the list output is actually desired, the endresult of that proposal is actually less readable than an appropriatelydefined helper function, and the function and method forms of the iteratorversions are pretty much equivalent from a readability perspective.

So unless I’ve missed something critical, readily availablelistvalues()andlistitems() helper functions look like they will improve thereadability of hybrid code more than anything we could add back to thePython 3.5+ mapping API, and won’t have any long-term impact on thecomplexity of Python 3 itself.

Discussion

The fact that 5 years in to the Python 3 migration we still have usersconsidering the dict API changes a significant barrier to migration suggeststhat there are problems with previously recommended approaches. This PEPattempts to explore those issues and tries to isolate those cases whereprevious advice (such as it was) could prove problematic.

My assessment (largely based on feedback from Twisted devs) is thatproblems are most likely to arise when attempting to used.keys(),d.values(), andd.items() in hybrid code. While superficially itseems as though there should be cases where it is safe to ignore thesemantic differences, in practice, the change from “mutable snapshot” to“dynamic view” is significant enough that it is likely betterto just force the use of either list or iterator semantics for hybrid code,and leave the use of the view semantics to pure Python 3 code.

This approach also creates rules that are simple enough and safe enough thatit should be possible to automate them in code modernisation scripts thattarget the common subset of Python 2 and Python 3, just as2to3 convertsthem automatically when targeting pure Python 3 code.

Acknowledgements

Thanks to the folks at the Twisted sprint table at PyCon for a veryvigorous discussion of this idea (and several other topics), and especiallyto Hynek Schlawack for acting as a moderator when things got a little tooheated :)

Thanks also to JP Calderone and Itamar Turner-Trauring for their emailfeedback, as well to the participants in thepython-dev review ofthe initial version of the PEP.

Copyright

This document has been placed in the public domain.