Safe Refactoring

Python’s type system does not provide a way to identify call sites that need tobe changed to stay consistent when an overridden function API changes. Thismakes refactoring and transforming code more dangerous.

Consider this simple inheritance structure:

classParent:deffoo(self,x:int)->int:returnxclassChild(Parent):deffoo(self,x:int)->int:returnx+1defparent_callsite(parent:Parent)->None:parent.foo(1)defchild_callsite(child:Child)->None:child.foo(1)

If the overridden method on the superclass is renamed or deleted, type checkerswill only alert us to update call sites that deal with the base type directly.But the type checker can only see the new code, not the change we made, so ithas no way of knowing that we probably also needed to rename the same method onchild classes.

A type checker will happily accept this code, even though we are likelyintroducing bugs:

classParent:# Rename this methoddefnew_foo(self,x:int)->int:returnxclassChild(Parent):# This (unchanged) method used to override `foo` but is unrelated to `new_foo`deffoo(self,x:int)->int:returnx+1defparent_callsite(parent:Parent)->None:# If we pass a Child instance we’ll now run Parent.new_foo - likely a bugparent.new_foo(1)defchild_callsite(child:Child)->None:# We probably wanted to invoke new_foo here. Instead, we forked the methodchild.foo(1)

This code will type check, but there are two potential sources of bugs:

• If we pass aChild instance to theparent_callsite function, it willinvoke the implementation inParent.new_foo. rather thanChild.foo.This is probably a bug - we presumably would not have writtenChild.foo inthe first place if we didn’t need custom behavior.
• Our system was likely relying onChild.foo behaving in a similar way toParent.foo. But unless we catch this early, we have now forked themethods, and in future refactors it is likely no one will realize that majorchanges to the behavior ofnew_foo likely require updatingChild.foo aswell, which could lead to major bugs later.

The incorrectly-refactored code is type-safe, but is probably not what weintended and could cause our system to behave incorrectly. The bug can bedifficult to track down because our new code likely does execute withoutthrowing exceptions. Tests are less likely to catch the problem, and silenterrors can take longer to track down in production.

We are aware of several production outages in multiple typed codebases caused bysuch incorrect refactors. This is our primary motivation for adding an@overridedecorator to the type system, which lets developers express the relationshipbetweenParent.foo andChild.foo so that type checkers can detect the problem.

Subclass Implementations Become More Explicit

We believe that explicit overrides will make unfamiliar code easier to read thanimplicit overrides. A developer reading the implementation of a subclass thatuses@override can immediately see which methods are overridingfunctionality in some base class; without this decorator, the only way toquickly find out is using a static analysis tool.

Precedent in Other Languages and Runtime Libraries

Disadvantages

Using@override will make code more verbose.

No New Rules for Override Compatibility

This PEP is exclusively concerned with the handling of the new@override decorator,which specifies that the decorated method must override some attribute inan ancestor class. This PEP does not propose any new rules regarding the typesignatures of such methods.

Motivation

The primary reason for a strict mode that requires@override is that developerscan only trust that refactors are override-safe if they know that the@overridedecorator is used throughout the project.

There is another class of bug related to overrides that we can only catch using a strict mode.

Consider the following code:

classParent:passclassChild(Parent):deffoo(self)->int:return2

Imagine we refactor it as follows:

classParent:deffoo(self)->int:# This method is newreturn1classChild(Parent):deffoo(self)->int:# This is now an override!return2defcall_foo(parent:Parent)->int:returnparent.foo()# This could invoke Child.foo, which may be surprising.

The semantics of our code changed here, which could cause two problems:

• If the author of the code change did not know thatChild.foo alreadyexisted (which is very possible in a large codebase), they might be surprisedto see thatcall_foo does not always invokeParent.foo.
• If the codebase authors tried to manually apply@override everywhere whenwriting overrides in subclasses, they are likely to miss the fact thatChild.foo needs it here.

At first glance this kind of change may seem unlikely, but it can actuallyhappen often if one or more subclasses have functionality that developers laterrealize belongs in the base class.

With a strict mode, we will always alert developers when this occurs.

Precedent

Most of the typed, object-oriented programming languages we looked at have aneasy way to require explicit overrides throughout a project:

• C#, Kotlin, Scala, and Swift always require explicit overrides
• TypeScript has a–no-implicit-overrideflag to force explicit overrides
• In Hack and Java the type checker always treats overrides as opt-in, butwidely-used linters can warn if explicit overrides are missing.

Set__override__=True when possible

At runtime,@typing.override will make a best-effort attempt to add anattribute__override__ with valueTrue to its argument. By “best-effort”we mean that we will try adding the attribute, but if that fails (for examplebecause the input is a descriptor type with fixed slots) we will silentlyreturn the argument as-is.

This is exactly what the@typing.final decorator does, and the motivationis similar: it gives runtime libraries the ability to use@override. As aconcrete example, a runtime library could check__override__ in orderto automatically populate the__doc__ attribute of child class methodsusing the parent method docstring.

Limitations of setting__override__

As described above, adding__override__ may fail at runtime, in whichcase we will simply return the argument as-is.

In addition, even in cases where it does work, it may be difficult for users tocorrectly work with multiple decorators, because successfully ensuring the__override__ attribute is set on the final output requires understanding theimplementation of each decorator:

• The@override decorator needs to executeafter ordinary decoratorslike@functools.lru_cache that use wrapper functions, since we want toset__override__ on the outermost wrapper. This means it needs togoabove all these other decorators.
• But@override needs to executebefore many special descriptor-baseddecorators like@property,@staticmethod, and@classmethod.
• As discussed above, in some cases (for example a descriptor with fixedslots or a descriptor that also wraps) it may be impossible to set the__override__ attribute at all.

As a result, runtime support for setting__override__ is best effortonly, and we do not expect type checkers to validate the ordering ofdecorators.

Rely on Integrated Development Environments for safety

Modern Integrated Development Environments (IDEs) often provide the ability toautomatically update subclasses when renaming a method. But we view this asinsufficient for several reasons:

• If a codebase is split into multiple projects, an IDE will not help and thebug appears when upgrading dependencies. Type checkers are a fast way to catchbreaking changes in dependencies.
• Not all developers use such IDEs. And library maintainers, even if they do usean IDE, should not need to assume pull request authors use the same IDE. Weprefer being able to detect problems in continuous integration withoutassuming anything about developers’ choice of editor.

Runtime enforcement

We considered having@typing.override enforce override safety at runtime,similarly to how@overrides.overridesdoes today.

We rejected this for four reasons:

• For users of static type checking, it is not clear this brings any benefits.
• There would be at least some performance overhead, leading to projectsimporting slower with runtime enforcement. We estimate the@overrides.overrides implementation takes around 100 microseconds, whichis fast but could still add up to a second or more of extra initializationtime in million-plus line codebases, which is exactly where we think@typing.override will be most useful.
• An implementation may have edge cases where it doesn’t work well (we heardfrom a maintainer of one such closed-source library that this has been aproblem). We expect static enforcement to be simple and reliable.
• The implementation approaches we know of are not simple. The decoratorexecutes before the class is finished evaluating, so the options we know ofare either to inspect the bytecode of the caller (as@overrides.overridesdoes) or to use a metaclass-based approach. Neither approach seems ideal.

Mark a base class to force explicit overrides on subclasses

We considered including a class decorator@require_explicit_overrides, whichwould have provided a way for base classes to declare that all subclasses mustuse the@override decorator on method overrides. TheOverrides library has a mixin class,EnforceExplicitOverrides, which provides similar behavior in runtime checks.

We decided against this because we expect owners of large codebases will benefitmost from@override, and for these use cases having a strict mode whereexplicit@override is required (see the Backward Compatibility section)provides more benefits than a way to mark base classes.

Moreover we believe that authors of projects who do not consider the extra typesafety to be worth the additional boilerplate of using@override should notbe forced to do so. Having an optional strict mode puts the decision in thehands of project owners, whereas the use of@require_explicit_overrides inlibraries would force project owners to use@override even if they prefernot to.

Include the name of the ancestor class being overridden

We considered allowing the caller of@override to specify a specificancestor class where the overridden method should be defined:

classParent0:deffoo(self)->int:return1classParent1:defbar(self)->int:return1classChild(Parent0,Parent1):@override(Parent0)# okay, Parent0 defines foodeffoo(self)->int:return2@override(Parent0)# type error, Parent0 does not define bardefbar(self)->int:return2

This could be useful for code readability because it makes the overridestructure more explicit for deep inheritance trees. It also might catch bugs byprompting developers to check that the implementation of an override still makessense whenever a method being overridden moves from one base class to another.

We decided against it because:

• Supporting this would add complexity to the implementation of both@override and type checker support for it, so there would need tobe considerable benefits.
• We believe that it would be rarely used and catch relatively few bugs.
  • The author of theOverrides package hasnotedthat early versions of his library included this capability but it wasrarely useful and seemed to have little benefit. After it was removed, theability was never requested by users.