Abstract

This PEP proposes a new mechanism in thefunctools standard librarymodule that provides a simple form of generic programming known assingle-dispatch generic functions.

Ageneric function is composed of multiple functions implementingthe same operation for different types. Which implementation should beused during a call is determined by the dispatch algorithm. When theimplementation is chosen based on the type of a single argument, this isknown assingle dispatch.

Rationale and Goals

Python has always provided a variety of built-in and standard-librarygeneric functions, such aslen(),iter(),pprint.pprint(),copy.copy(), and most of the functions in theoperator module.However, it currently:

1. does not have a simple or straightforward way for developers tocreate new generic functions,
2. does not have a standard way for methods to be added to existinggeneric functions (i.e., some are added using registrationfunctions, others require defining__special__ methods, possiblyby monkeypatching).

In addition, it is currently a common anti-pattern for Python code toinspect the types of received arguments, in order to decide what to dowith the objects.

For example, code may wish to accept either an objectof some type, or a sequence of objects of that type.Currently, the “obvious way” to do this is by type inspection, but thisis brittle and closed to extension.

Abstract Base Classes make it easierto discover present behaviour, but don’t help adding new behaviour.A developer using an already-written library may be unable to change howtheir objects are treated by such code, especially if the objects theyare using were created by a third party.

Therefore, this PEP proposes a uniform API to address dynamicoverloading using decorators.

User API

To define a generic function, decorate it with the@singledispatchdecorator. Note that the dispatch happens on the type of the firstargument. Create your function accordingly:

>>>fromfunctoolsimportsingledispatch>>>@singledispatch...deffun(arg,verbose=False):...ifverbose:...print("Let me just say,",end=" ")...print(arg)

To add overloaded implementations to the function, use theregister() attribute of the generic function. This is a decorator,taking a type parameter and decorating a function implementing theoperation for that type:

>>>@fun.register(int)...def_(arg,verbose=False):...ifverbose:...print("Strength in numbers, eh?",end=" ")...print(arg)...>>>@fun.register(list)...def_(arg,verbose=False):...ifverbose:...print("Enumerate this:")...fori,eleminenumerate(arg):...print(i,elem)

To enable registering lambdas and pre-existing functions, theregister() attribute can be used in a functional form:

>>>defnothing(arg,verbose=False):...print("Nothing.")...>>>fun.register(type(None),nothing)

Theregister() attribute returns the undecorated function. Thisenables decorator stacking, pickling, as well as creating unit tests foreach variant independently:

>>>@fun.register(float)...@fun.register(Decimal)...deffun_num(arg,verbose=False):...ifverbose:...print("Half of your number:",end=" ")...print(arg/2)...>>>fun_numisfunFalse

When called, the generic function dispatches on the type of the firstargument:

>>>fun("Hello, world.")Hello, world.>>>fun("test.",verbose=True)Let me just say, test.>>>fun(42,verbose=True)Strength in numbers, eh? 42>>>fun(['spam','spam','eggs','spam'],verbose=True)Enumerate this:0 spam1 spam2 eggs3 spam>>>fun(None)Nothing.>>>fun(1.23)0.615

Where there is no registered implementation for a specific type, itsmethod resolution order is used to find a more generic implementation.The original function decorated with@singledispatch is registeredfor the baseobject type, which means it is used if no betterimplementation is found.

To check which implementation will the generic function choose fora given type, use thedispatch() attribute:

>>>fun.dispatch(float)<function fun_num at 0x104319058>>>>fun.dispatch(dict)# note: default implementation<function fun at 0x103fe0000>

To access all registered implementations, use the read-onlyregistryattribute:

>>>fun.registry.keys()dict_keys([<class 'NoneType'>, <class 'int'>, <class 'object'>,          <class 'decimal.Decimal'>, <class 'list'>,          <class 'float'>])>>>fun.registry[float]<function fun_num at 0x1035a2840>>>>fun.registry[object]<function fun at 0x103fe0000>

The proposed API is intentionally limited and opinionated, as to ensureit is easy to explain and use, as well as to maintain consistency withexisting members in thefunctools module.

Implementation Notes

The functionality described in this PEP is already implemented in thepkgutil standard library module assimplegeneric. Because thisimplementation is mature, the goal is to move it largely as-is. Thereference implementation is available on hg.python.org[1].

The dispatch type is specified as a decorator argument. An alternativeform using function annotations was considered but its inclusionhas been rejected. As of May 2013, this usage pattern is out of scopefor the standard library[2], and the best practices forannotation usage are still debated.

Based on the currentpkgutil.simplegeneric implementation, andfollowing the convention on registering virtual subclasses on AbstractBase Classes, the dispatch registry will not be thread-safe.

>>>_compose_mro(dict,[])[<class 'dict'>, <class 'object'>]

>>>_compose_mro(dict,[Sized,MutableMapping,str,...Sequence,Iterable])[<class 'dict'>, <class 'collections.abc.MutableMapping'>, <class 'collections.abc.Mapping'>, <class 'collections.abc.Sized'>, <class 'collections.abc.Iterable'>, <class 'collections.abc.Container'>, <class 'object'>]

>>>fromcollections.abcimportIterable,Container>>>classP:...pass>>>Iterable.register(P)<class '__main__.P'>>>>Container.register(P)<class '__main__.P'>

>>>@singledispatch...defg(arg):...return"base"...>>>g.register(Iterable,lambdaarg:"iterable")<function <lambda> at 0x108b49110>>>>g.register(Container,lambdaarg:"container")<function <lambda> at 0x108b491c8>>>>g(P())Traceback (most recent call last):...RuntimeError:Ambiguous dispatch: <class 'collections.abc.Container'>or <class 'collections.abc.Iterable'>

>>>classTen(Iterable,Container):...def__iter__(self):...foriinrange(10):...yieldi...def__contains__(self,value):...returnvalueinrange(10)...>>>g(Ten())'iterable'

>>>classQ:...def__contains__(self,value):...returnFalse...>>>issubclass(Q,Container)True>>>Iterable.register(Q)>>>g(Q())Traceback (most recent call last):...RuntimeError:Ambiguous dispatch: <class 'collections.abc.Container'>or <class 'collections.abc.Iterable'>

Usage Patterns

This PEP proposes extending behaviour only of functions specificallymarked as generic. Just as a base class method may be overridden bya subclass, so too a function may be overloaded to provide customfunctionality for a given type.

Universal overloading does not equalarbitrary overloading, in thesense that we need not expect people to randomly redefine the behaviorof existing functions in unpredictable ways. To the contrary, genericfunction usage in actual programs tends to follow very predictablepatterns and registered implementations are highly-discoverable in thecommon case.

If a module is defining a new generic operation, it will usually alsodefine any required implementations for existing types in the sameplace. Likewise, if a module is defining a new type, then it willusually define implementations there for any generic functions that itknows or cares about. As a result, the vast majority of registeredimplementations can be found adjacent to either the function beingoverloaded, or to a newly-defined type for which the implementation isadding support.

It is only in rather infrequent cases that one will have implementationsregistered in a module that contains neither the function nor thetype(s) for which the implementation is added. In the absence ofincompetence or deliberate intention to be obscure, the fewimplementations that are not registered adjacent to the relevant type(s)or function(s), will generally not need to be understood or known aboutoutside the scope where those implementations are defined. (Except inthe “support modules” case, where best practice suggests naming themaccordingly.)

As mentioned earlier, single-dispatch generics are already prolificthroughout the standard library. A clean, standard way of doing themprovides a way forward to refactor those custom implementations to usea common one, opening them up for user extensibility at the same time.

Alternative approaches

InPEP 3124 Phillip J. Eby proposes a full-grown solutionwith overloading based on arbitrary rule sets (with the defaultimplementation dispatching on argument types), as well as interfaces,adaptation and method combining. PEAK-Rules[4] isa reference implementation of the concepts described in PJE’s PEP.

Such a broad approach is inherently complex, which makes reachinga consensus hard. In contrast, this PEP focuses on a single piece offunctionality that is simple to reason about. It’s important to notethis does not preclude the use of other approaches now or in the future.

In a 2005 article on Artima[5] Guido van Rossum presentsa generic function implementation that dispatches on types of allarguments on a function. The same approach was chosen in Andrey Popp’sgeneric package available on PyPI[6], as well as DavidMertz’sgnosis.magic.multimethods[7].

While this seems desirable at first, I agree with Fredrik Lundh’scomment that “if you design APIs with pages of logic just to sort outwhat code a function should execute, you should probably hand over theAPI design to someone else”. In other words, the single argumentapproach proposed in this PEP is not only easier to implement but alsoclearly communicates that dispatching on a more complex state is ananti-pattern. It also has the virtue of corresponding directly with thefamiliar method dispatch mechanism in object oriented programming. Theonly difference is whether the custom implementation is associated moreclosely with the data (object-oriented methods) or the algorithm(single-dispatch overloading).

PyPy’s RPython offersextendabletype[8], a metaclass whichenables classes to be externally extended. In combination withpairtype() andpair() factories, this offers a form ofsingle-dispatch generics.

Acknowledgements

Apart from Phillip J. Eby’s work onPEP 3124 andPEAK-Rules, influences include Paul Moore’s original issue[9] that proposed exposingpkgutil.simplegeneric as partof thefunctools API, Guido van Rossum’s article on multimethods[5], and discussions with Raymond Hettinger on a generalpprint rewrite. Huge thanks to Alyssa Coghlan for encouraging me to createthis PEP and providing initial feedback.

References

Copyright

This document has been placed in the public domain.