Rejection Notice

I’m rejecting this PEP. Something much better is about to happen;it’s too early to say exactly what, but it’s not going to resemblethe proposal in this PEP too closely so it’s better to start a newPEP. GvR.

Abstract

This proposal puts forth an extensible cooperative mechanism forthe adaptation of an incoming object to a context which expects anobject supporting a specific protocol (say a specific type, class,or interface).

This proposal provides a built-in “adapt” function that, for anyobject X and any protocol Y, can be used to ask the Pythonenvironment for a version of X compliant with Y. Behind thescenes, the mechanism asks object X: “Are you now, or do you knowhow to wrap yourself to provide, a supporter of protocol Y?”.And, if this request fails, the function then asks protocol Y:“Does object X support you, or do you know how to wrap it toobtain such a supporter?” This duality is important, becauseprotocols can be developed after objects are, or vice-versa, andthis PEP lets either case be supported non-invasively with regardto the pre-existing component[s].

Lastly, if neither the object nor the protocol know about eachother, the mechanism may check a registry of adapter factories,where callables able to adapt certain objects to certain protocolscan be registered dynamically. This part of the proposal isoptional: the same effect could be obtained by ensuring thatcertain kinds of protocols and/or objects can accept dynamicregistration of adapter factories, for example via suitable custommetaclasses. However, this optional part allows adaptation to bemade more flexible and powerful in a way that is not invasive toeither protocols or other objects, thereby gaining for adaptationmuch the same kind of advantage that Python standard library’s“copy_reg” module offers for serialization and persistence.

This proposal does not specifically constrain what a protocolis, what “compliance to a protocol” exactlymeans, nor whatprecisely a wrapper is supposed to do. These omissions areintended to leave this proposal compatible with both existingcategories of protocols, such as the existing system of type andclasses, as well as the many concepts for “interfaces” as suchwhich have been proposed or implemented for Python, such as theone inPEP 245, the one in Zope3[2], or the ones discussed inthe BDFL’s Artima blog in late 2004 and early 2005[3]. However,some reflections on these subjects, intended to be suggestive andnot normative, are also included.

Motivation

Currently there is no standardized mechanism in Python forchecking if an object supports a particular protocol. Typically,existence of certain methods, particularly special methods such as__getitem__, is used as an indicator of support for a particularprotocol. This technique works well for a few specific protocolsblessed by the BDFL (Benevolent Dictator for Life). The same canbe said for the alternative technique based on checking‘isinstance’ (the built-in class “basestring” exists specificallyto let you use ‘isinstance’ to check if an object “is a [built-in]string”). Neither approach is easily and generally extensible toother protocols, defined by applications and third partyframeworks, outside of the standard Python core.

Even more important than checking if an object already supports agiven protocol can be the task of obtaining a suitable adapter(wrapper or proxy) for the object, if the support is not alreadythere. For example, a string does not support the file protocol,but you can wrap it into a StringIO instance to obtain an objectwhich does support that protocol and gets its data from the stringit wraps; that way, you can pass the string (suitably wrapped) tosubsystems which require as their arguments objects that arereadable as files. Unfortunately, there is currently no general,standardized way to automate this extremely important kind of“adaptation by wrapping” operations.

Typically, today, when you pass objects to a context expecting aparticular protocol, either the object knows about the context andprovides its own wrapper or the context knows about the object andwraps it appropriately. The difficulty with these approaches isthat such adaptations are one-offs, are not centralized in asingle place of the users code, and are not executed with a commontechnique, etc. This lack of standardization increases codeduplication with the same adapter occurring in more than one placeor it encourages classes to be re-written instead of adapted. Ineither case, maintainability suffers.

It would be very nice to have a standard function that can becalled upon to verify an object’s compliance with a particularprotocol and provide for a wrapper if one is readily available –all without having to hunt through each library’s documentationfor the incantation appropriate to that particular, specific case.

Requirements

When considering an object’s compliance with a protocol, there areseveral cases to be examined:

1. When the protocol is a type or class, and the object hasexactly that type or is an instance of exactly that class (nota subclass). In this case, compliance is automatic.
2. When the object knows about the protocol, and either considersitself compliant, or knows how to wrap itself suitably.
3. When the protocol knows about the object, and either the objectalready complies or the protocol knows how to suitably wrap theobject.
4. When the protocol is a type or class, and the object is amember of a subclass. This is distinct from the first case (a)above, since inheritance (unfortunately) does not necessarilyimply substitutability, and thus must be handled carefully.
5. When the context knows about the object and the protocol andknows how to adapt the object so that the required protocol issatisfied. This could use an adapter registry or similarapproaches.

The fourth case above is subtle. A break of substitutability canoccur when a subclass changes a method’s signature, or restrictsthe domains accepted for a method’s argument (“co-variance” onarguments types), or extends the co-domain to include returnvalues which the base class may never produce (“contra-variance”on return types). While compliance based on class inheritanceshould be automatic, this proposal allows an object to signalthat it is not compliant with a base class protocol.

If Python gains some standard “official” mechanism for interfaces,however, then the “fast-path” case (a) can and should be extendedto the protocol being an interface, and the object an instance ofa type or class claiming compliance with that interface. Forexample, if the “interface” keyword discussed in[3] is adoptedinto Python, the “fast path” of case (a) could be used, sinceinstantiable classes implementing an interface would not beallowed to break substitutability.

Specification

This proposal introduces a new built-in function,adapt(), whichis the basis for supporting these requirements.

Theadapt() function has three parameters:

• obj, the object to be adapted
• protocol, the protocol requested of the object
• alternate, an optional object to return if the object couldnot be adapted

A successful result of theadapt() function returns either theobject passedobj, if the object is already compliant with theprotocol, or a secondary objectwrapper, which provides a viewof the object compliant with the protocol. The definition ofwrapper is deliberately vague, and a wrapper is allowed to be afull object with its own state if necessary. However, the designintention is that an adaptation wrapper should hold a reference tothe original object it wraps, plus (if needed) a minimum of extrastate which it cannot delegate to the wrapper object.

An excellent example of adaptation wrapper is an instance ofStringIO which adapts an incoming string to be read as if it was atextfile: the wrapper holds a reference to the string, but dealsby itself with the “current point of reading” (fromwhere in thewrapped strings will the characters for the next, e.g., “readline”call come from), because it cannot delegate it to the wrappedobject (a string has no concept of “current point of reading” noranything else even remotely related to that concept).

A failure to adapt the object to the protocol raises anAdaptationError (which is a subclass ofTypeError), unless thealternate parameter is used, in this case the alternate argumentis returned instead.

To enable the first case listed in the requirements, theadapt()function first checks to see if the object’s type or the object’sclass are identical to the protocol. If so, then theadapt()function returns the object directly without further ado.

To enable the second case, when the object knows about theprotocol, the object must have a__conform__() method. Thisoptional method takes two arguments:

• self, the object being adapted
• protocol, the protocol requested

Just like any other special method in today’s Python,__conform__is meant to be taken from the object’s class, not from the objectitself (for all objects, except instances of “classic classes” aslong as we must still support the latter). This enables apossible ‘tp_conform’ slot to be added to Python’s type objects inthe future, if desired.

The object may return itself as the result of__conform__ toindicate compliance. Alternatively, the object also has theoption of returning a wrapper object compliant with the protocol.If the object knows it is not compliant although it belongs to atype which is a subclass of the protocol, then__conform__ shouldraise aLiskovViolation exception (a subclass ofAdaptationError).Finally, if the object cannot determine its compliance, it shouldreturnNone to enable the remaining mechanisms. If__conform__raises any other exception, “adapt” just propagates it.

To enable the third case, when the protocol knows about theobject, the protocol must have an__adapt__() method. Thisoptional method takes two arguments:

• self, the protocol requested
• obj, the object being adapted

If the protocol finds the object to be compliant, it can returnobj directly. Alternatively, the method may return a wrappercompliant with the protocol. If the protocol knows the object isnot compliant although it belongs to a type which is a subclass ofthe protocol, then__adapt__ should raise aLiskovViolationexception (a subclass ofAdaptationError). Finally, whencompliance cannot be determined, this method should return None toenable the remaining mechanisms. If__adapt__ raises any otherexception, “adapt” just propagates it.

The fourth case, when the object’s class is a sub-class of theprotocol, is handled by the built-inadapt() function. Undernormal circumstances, if “isinstance(object, protocol)” thenadapt() returns the object directly. However, if the object isnot substitutable, either the__conform__() or__adapt__()methods, as above mentioned, may raise anLiskovViolation (asubclass ofAdaptationError) to prevent this default behavior.

If none of the first four mechanisms worked, as a last-ditchattempt, ‘adapt’ falls back to checking a registry of adapterfactories, indexed by the protocol and the type ofobj, to meetthe fifth case. Adapter factories may be dynamically registeredand removed from that registry to provide “third party adaptation”of objects and protocols that have no knowledge of each other, ina way that is not invasive to either the object or the protocols.

Intended Use

The typical intended use of adapt is in code which has receivedsome object X “from the outside”, either as an argument or as theresult of calling some function, and needs to use that objectaccording to a certain protocol Y. A “protocol” such as Y ismeant to indicate an interface, usually enriched with somesemantics constraints (such as are typically used in the “designby contract” approach), and often also some pragmaticalexpectation (such as “the running time of a certain operationshould be no worse than O(N)”, or the like); this proposal doesnot specify how protocols are designed as such, nor how or whethercompliance to a protocol is checked, nor what the consequences maybe of claiming compliance but not actually delivering it (lack of“syntactic” compliance – names and signatures of methods – willoften lead to exceptions being raised; lack of “semantic”compliance may lead to subtle and perhaps occasional errors[imagine a method claiming to be threadsafe but being in factsubject to some subtle race condition, for example]; lack of“pragmatic” compliance will generally lead to code that runscorrectly, but too slowly for practical use, or sometimes toexhaustion of resources such as memory or disk space).

When protocol Y is a concrete type or class, compliance to it isintended to mean that an object allows all of the operations thatcould be performed on instances of Y, with “comparable” semanticsand pragmatics. For example, a hypothetical object X that is asingly-linked list should not claim compliance with protocol‘list’, even if it implements all of list’s methods: the fact thatindexingX[n] takes time O(n), while the same operation would beO(1) on a list, makes a difference. On the other hand, aninstance ofStringIO.StringIO does comply with protocol ‘file’,even though some operations (such as those of module ‘marshal’)may not allow substituting one for the other because they performexplicit type-checks: such type-checks are “beyond the pale” fromthe point of view of protocol compliance.

While this convention makes it feasible to use a concrete type orclass as a protocol for purposes of this proposal, such use willoften not be optimal. Rarely will the code calling ‘adapt’ needALL of the features of a certain concrete type, particularly forsuch rich types as file, list, dict; rarely can all those featuresbe provided by a wrapper with good pragmatics, as well as syntaxand semantics that are really the same as a concrete type’s.

Rather, once this proposal is accepted, a design effort needs tostart to identify the essential characteristics of those protocolswhich are currently used in Python, particularly within thestandard library, and to formalize them using some kind of“interface” construct (not necessarily requiring any new syntax: asimple custom metaclass would let us get started, and the resultsof the effort could later be migrated to whatever “interface”construct is eventually accepted into the Python language). Withsuch a palette of more formally designed protocols, the code using‘adapt’ will be able to ask for, say, adaptation into “a filelikeobject that is readable and seekable”, or whatever else itspecifically needs with some decent level of “granularity”, ratherthan too-generically asking for compliance to the ‘file’ protocol.

Adaptation is NOT “casting”. When object X itself does notconform to protocol Y, adapting X to Y means using some kind ofwrapper object Z, which holds a reference to X, and implementswhatever operation Y requires, mostly by delegating to X inappropriate ways. For example, if X is a string and Y is ‘file’,the proper way to adapt X to Y is to make aStringIO(X),NOT tocallfile(X) [which would try to open a file named by X].

Numeric types and protocols may need to be an exception to this“adaptation is not casting” mantra, however.

Guido’s “Optional Static Typing: Stop the Flames” Blog Entry

A typical simple use case of adaptation would be:

deff(X):X=adapt(X,Y)# continue by using X according to protocol Y

In[4], the BDFL has proposed introducing the syntax:

deff(X:Y):# continue by using X according to protocol Y

to be a handy shortcut for exactly this typical use of adapt, and,as a basis for experimentation until the parser has been modifiedto accept this new syntax, a semantically equivalent decorator:

@arguments(Y)deff(X):# continue by using X according to protocol Y

These BDFL ideas are fully compatible with this proposal, as areother of Guido’s suggestions in the same blog.

Reference Implementation and Test Cases

The following reference implementation does not deal with classicclasses: it consider only new-style classes. If classic classesneed to be supported, the additions should be pretty clear, thougha bit messy (x.__class__ vstype(x), getting boundmethods directlyfrom the object rather than from the type, and so on).

-----------------------------------------------------------------adapt.py-----------------------------------------------------------------classAdaptationError(TypeError):passclassLiskovViolation(AdaptationError):pass_adapter_factory_registry={}defregisterAdapterFactory(objtype,protocol,factory):_adapter_factory_registry[objtype,protocol]=factorydefunregisterAdapterFactory(objtype,protocol):del_adapter_factory_registry[objtype,protocol]def_adapt_by_registry(obj,protocol,alternate):factory=_adapter_factory_registry.get((type(obj),protocol))iffactoryisNone:adapter=alternateelse:adapter=factory(obj,protocol,alternate)ifadapterisAdaptationError:raiseAdaptationErrorelse:returnadapterdefadapt(obj,protocol,alternate=AdaptationError):t=type(obj)# (a) first check to see if object has the exact protocoliftisprotocol:returnobjtry:# (b) next check if t.__conform__ exists & likes protocolconform=getattr(t,'__conform__',None)ifconformisnotNone:result=conform(obj,protocol)ifresultisnotNone:returnresult# (c) then check if protocol.__adapt__ exists & likes objadapt=getattr(type(protocol),'__adapt__',None)ifadaptisnotNone:result=adapt(protocol,obj)ifresultisnotNone:returnresultexceptLiskovViolation:passelse:# (d) check if object is instance of protocolifisinstance(obj,protocol):returnobj# (e) last chance: try the registryreturn_adapt_by_registry(obj,protocol,alternate)-----------------------------------------------------------------test.py-----------------------------------------------------------------fromadaptimportAdaptationError,LiskovViolation,adaptfromadaptimportregisterAdapterFactory,unregisterAdapterFactoryimportdoctestclassA(object):'''    >>> a = A()    >>> a is adapt(a, A)   # case (a)    True    '''classB(A):'''    >>> b = B()    >>> b is adapt(b, A)   # case (d)    True    '''classC(object):'''    >>> c = C()    >>> c is adapt(c, B)   # case (b)    True    >>> c is adapt(c, A)   # a failure case    Traceback (most recent call last):        ...    AdaptationError    '''def__conform__(self,protocol):ifprotocolisB:returnselfclassD(C):'''    >>> d = D()    >>> d is adapt(d, D)   # case (a)    True    >>> d is adapt(d, C)   # case (d) explicitly blocked    Traceback (most recent call last):        ...    AdaptationError    '''def__conform__(self,protocol):ifprotocolisC:raiseLiskovViolationclassMetaAdaptingProtocol(type):def__adapt__(cls,obj):returncls.adapt(obj)classAdaptingProtocol:__metaclass__=MetaAdaptingProtocol@classmethoddefadapt(cls,obj):passclassE(AdaptingProtocol):'''    >>> a = A()    >>> a is adapt(a, E)   # case (c)    True    >>> b = A()    >>> b is adapt(b, E)   # case (c)    True    >>> c = C()    >>> c is adapt(c, E)   # a failure case    Traceback (most recent call last):        ...    AdaptationError    '''@classmethoddefadapt(cls,obj):ifisinstance(obj,A):returnobjclassF(object):passdefadapt_F_to_A(obj,protocol,alternate):ifisinstance(obj,F)andissubclass(protocol,A):returnobjelse:returnalternatedeftest_registry():'''    >>> f = F()    >>> f is adapt(f, A)   # a failure case    Traceback (most recent call last):        ...    AdaptationError    >>> registerAdapterFactory(F, A, adapt_F_to_A)    >>> f is adapt(f, A)   # case (e)    True    >>> unregisterAdapterFactory(F, A)    >>> f is adapt(f, A)   # a failure case again    Traceback (most recent call last):        ...    AdaptationError    >>> registerAdapterFactory(F, A, adapt_F_to_A)    '''doctest.testmod()

Relationship To Microsoft’s QueryInterface

Although this proposal has some similarities to Microsoft’s (COM)QueryInterface, it differs by a number of aspects.

First, adaptation in this proposal is bi-directional, allowing theinterface (protocol) to be queried as well, which gives moredynamic abilities (more Pythonic). Second, there is no special“IUnknown” interface which can be used to check or obtain theoriginal unwrapped object identity, although this could beproposed as one of those “special” blessed interface protocolidentifiers. Third, with QueryInterface, once an object supportsa particular interface it must always there after support thisinterface; this proposal makes no such guarantee, since, inparticular, adapter factories can be dynamically added to theregistried and removed again later.

Fourth, implementations of Microsoft’s QueryInterface must supporta kind of equivalence relation – they must be reflexive,symmetrical, and transitive, in specific senses. The equivalentconditions for protocol adaptation according to this proposalwould also represent desirable properties:

# given, to start with, a successful adaptation:X_as_Y=adapt(X,Y)# reflexive:assertadapt(X_as_Y,Y)isX_as_Y# transitive:X_as_Z=adapt(X,Z,None)X_as_Y_as_Z=adapt(X_as_Y,Z,None)assert(X_as_Y_as_ZisNone)==(X_as_ZisNone)# symmetrical:X_as_Z_as_Y=adapt(X_as_Z,Y,None)assert(X_as_Y_as_ZisNone)==(X_as_Z_as_YisNone)

However, while these properties are desirable, it may not bepossible to guarantee them in all cases. QueryInterface canimpose their equivalents because it dictates, to some extent, howobjects, interfaces, and adapters are to be coded; this proposalis meant to be not necessarily invasive, usable and to “retrofit”adaptation between two frameworks coded in mutual ignorance ofeach other without having to modify either framework.

Transitivity of adaptation is in fact somewhat controversial, asis the relationship (if any) between adaptation and inheritance.

The latter would not be controversial if we knew that inheritancealways implies Liskov substitutability, which, unfortunately wedon’t. If some special form, such as the interfaces proposed in[4], could indeed ensure Liskov substitutability, then for thatkind of inheritance, only, we could perhaps assert that if Xconforms to Y and Y inherits from Z then X conforms to Z… butonly if substitutability was taken in a very strong sense toinclude semantics and pragmatics, which seems doubtful. (For whatit’s worth: in QueryInterface, inheritance does not require norimply conformance). This proposal does not include any “strong”effects of inheritance, beyond the small ones specificallydetailed above.

Similarly, transitivity might imply multiple “internal” adaptationpasses to get the result ofadapt(X,Z) via some intermediate Y,intrinsically likeadapt(adapt(X,Y),Z), for some suitable andautomatically chosen Y. Again, this may perhaps be feasible undersuitably strong constraints, but the practical implications ofsuch a scheme are still unclear to this proposal’s authors. Thus,this proposal does not include any automatic or implicittransitivity of adaptation, under whatever circumstances.

For an implementation of the original version of this proposalwhich performs more advanced processing in terms of transitivity,and of the effects of inheritance, see Phillip J. Eby’sPyProtocols[5]. The documentation accompanyingPyProtocols iswell worth studying for its considerations on how adapters shouldbe coded and used, and on how adaptation can remove any need fortypechecking in application code.

Questions and Answers

• Q: What benefit does this proposal provide?

  A: The typical Python programmer is an integrator, someone who isconnecting components from various suppliers. Often, tointerface between these components, one needs intermediateadapters. Usually the burden falls upon the programmer tostudy the interface exposed by one component and required byanother, determine if they are directly compatible, or developan adapter. Sometimes a supplier may even include theappropriate adapter, but even then searching for the adapterand figuring out how to deploy the adapter takes time.

  This technique enables suppliers to work with each otherdirectly, by implementing__conform__ or__adapt__ asnecessary. This frees the integrator from making their ownadapters. In essence, this allows the components to have asimple dialogue among themselves. The integrator simplyconnects one component to another, and if the types don’tautomatically match an adapting mechanism is built-in.

  Moreover, thanks to the adapter registry, a “fourth party” maysupply adapters to allow interoperation of frameworks whichare totally unaware of each other, non-invasively, and withoutrequiring the integrator to do anything more than install theappropriate adapter factories in the registry at start-up.

  As long as libraries and frameworks cooperate with theadaptation infrastructure proposed here (essentially bydefining and using protocols appropriately, and calling‘adapt’ as needed on arguments received and results ofcall-back factory functions), the integrator’s work therebybecomes much simpler.

  For example, consider SAX1 and SAX2 interfaces: there is anadapter required to switch between them. Normally, theprogrammer must be aware of this; however, with thisadaptation proposal in place, this is no longer the case –indeed, thanks to the adapter registry, this need may beremoved even if the framework supplying SAX1 and the onerequiring SAX2 are unaware of each other.

• Q: Why does this have to be built-in, can’t it be standalone?

  A: Yes, it does work standalone. However, if it is built-in, ithas a greater chance of usage. The value of this proposal isprimarily in standardization: having libraries and frameworkscoming from different suppliers, including the Python standardlibrary, use a single approach to adaptation. Furthermore:

  0. The mechanism is by its very nature a singleton.
  1. If used frequently, it will be much faster as a built-in.
  2. It is extensible and unassuming.
  3. Once ‘adapt’ is built-in, it can support syntax extensionsand even be of some help to a type inference system.
• Q: Why the verbs__conform__ and__adapt__?

  A: conform, verb intransitive

  1. To correspond in form or character; be similar.
  2. To act or be in accord or agreement; comply.
  3. To act in accordance with current customs or modes.

  adapt, verb transitive

  1. To make suitable to or fit for a specific use or situation.

  Source: The American Heritage Dictionary of the EnglishLanguage, Third Edition

Backwards Compatibility

There should be no problem with backwards compatibility unlesssomeone had used the special names__conform__ or__adapt__ inother ways, but this seems unlikely, and, in any case, user codeshould never use special names for non-standard purposes.

This proposal could be implemented and tested without changes tothe interpreter.

Credits

This proposal was created in large part by the feedback of thetalented individuals on the main Python mailing lists and thetype-sig list. To name specific contributors (with apologies ifwe missed anyone!), besides the proposal’s authors: the mainsuggestions for the proposal’s first versions came from PaulPrescod, with significant feedback from Robin Thomas, and we alsoborrowed ideas from Marcin ‘Qrczak’ Kowalczyk and Carlos Ribeiro.

Other contributors (via comments) include Michel Pelletier, JeremyHylton, Aahz Maruch, Fredrik Lundh, Rainer Deyke, Timothy Delaney,and Huaiyu Zhu. The current version owes a lot to discussionswith (among others) Phillip J. Eby, Guido van Rossum, Bruce Eckel,Jim Fulton, and Ka-Ping Yee, and to study and reflection of theirproposals, implementations, and documentation about use andadaptation of interfaces and protocols in Python.

References and Footnotes

Copyright

This document has been placed in the public domain.