Introduction

This PEP describes theaugmented assignment proposal for Python 2.0. ThisPEP tracks the status and ownership of this feature, slated for introductionin Python 2.0. It contains a description of the feature and outlines changesnecessary to support the feature. This PEP summarizes discussions held inmailing list forums[1], and provides URLs for further information whereappropriate. The CVS revision history of this file contains the definitivehistorical record.

Proposed Semantics

The proposed patch that adds augmented assignment to Python introduces thefollowing new operators:

+=-=*=/=%=**=<<=>>=&=^=|=

They implement the same operator as their normal binary form, except that theoperation is donein-place when the left-hand side object supports it, andthat the left-hand side is only evaluated once.

They truly behave as augmented assignment, in that they perform all of thenormal load and store operations, in addition to the binary operation they areintended to do. So, given the expression:

x+=y

The objectx is loaded, theny is added to it, and the resultingobject is stored back in the original place. The precise action performed onthe two arguments depends on the type ofx, and possibly ofy.

The idea behind augmented assignment in Python is that it isn’t just an easierway to write the common practice of storing the result of a binary operationin its left-hand operand, but also a way for the left-hand operand in questionto know that it should operateon itself, rather than creating a modifiedcopy of itself.

To make this possible, a number of newhooks are added to Python classes andC extension types, which are called when the object in question is used as theleft hand side of an augmented assignment operation. If the class or typedoes not implement thein-place hooks, the normal hooks for the particularbinary operation are used.

So, given an instance objectx, the expression:

x+=y

tries to callx.__iadd__(y), which is thein-place variant of__add__ . If__iadd__ is not present,x.__add__(y) is attempted,and finallyy.__radd__(x) if__add__ is missing too. There is noright-hand-side variant of__iadd__, because that would require fory to know how to in-place modifyx, which is unsafe to say the least.The__iadd__ hook should behave similar to__add__, returning theresult of the operation (which could beself) which is to be assigned tothe variablex.

For C extension types, thehooks are members of thePyNumberMethods andPySequenceMethods structures. Some special semantics apply to make theuse of these methods, and the mixing of Python instance objects and C types,as unsurprising as possible.

In the generic case ofx<augop>y (or a similar case using thePyNumber_InPlace API functions) the principal object being operated on isx. This differs from normal binary operations, wherex andycould be consideredco-operating, because unlike in binary operations, theoperands in an in-place operation cannot be swapped. However, in-placeoperations do fall back to normal binary operations when in-place modificationis not supported, resulting in the following rules:

• If the left-hand object (x) is an instance object, and it has a__coerce__ method, call that function withy as the argument. Ifcoercion succeeds, and the resulting left-hand object is a different objectthanx, stop processing it as in-place and call the appropriate functionfor the normal binary operation, with the coercedx andy asarguments. The result of the operation is whatever that function returns.

  If coercion does not yield a different object forx, orx does notdefine a__coerce__ method, andx has the appropriate__ihook__for this operation, call that method withy as the argument, and theresult of the operation is whatever that method returns.

• Otherwise, if the left-hand object is not an instance object, but its typedoes define the in-place function for this operation, call that functionwithx andy as the arguments, and the result of the operation iswhatever that function returns.

  Note that no coercion on eitherx ory is done in this case, andit’s perfectly valid for a C type to receive an instance object as thesecond argument; that is something that cannot happen with normal binaryoperations.

• Otherwise, process it exactly as a normal binary operation (not in-place),including argument coercion. In short, if either argument is an instanceobject, resolve the operation through__coerce__,__hook__ and__rhook__. Otherwise, both objects are C types, and they are coercedand passed to the appropriate function.
• If no way to process the operation can be found, raise aTypeError withan error message specific to the operation.
• Some special casing exists to account for the case of+ and*,which have a special meaning for sequences: for+, sequenceconcatenation, no coercion what so ever is done if a C type definessq_concat orsq_inplace_concat. For*, sequence repeating,y is converted to a C integer before calling eithersq_inplace_repeat andsq_repeat. This is done even ify is aninstance, though not ifx is an instance.

The in-place function should always return a new reference, either to theoldx object if the operation was indeed performed in-place, or to a newobject.

Rationale

There are two main reasons for adding this feature to Python: simplicity ofexpression, and support for in-place operations. The end result is a tradeoffbetween simplicity of syntax and simplicity of expression; like most newfeatures, augmented assignment doesn’t add anything that was previouslyimpossible. It merely makes these things easier to do.

Adding augmented assignment will make Python’s syntax more complex. Insteadof a single assignment operation, there are now twelve assignment operations,eleven of which also perform a binary operation. However, these eleven newforms of assignment are easy to understand as the coupling between assignmentand the binary operation, and they require no large conceptual leap tounderstand. Furthermore, languages that do have augmented assignment haveshown that they are a popular, much used feature. Expressions of the form:

<x>=<x><operator><y>

are common enough in those languages to make the extra syntax worthwhile, andPython does not have significantly fewer of those expressions. Quite theopposite, in fact, since in Python you can also concatenate lists with abinary operator, something that is done quite frequently. Writing the aboveexpression as:

<x><operator>=<y>

is both more readable and less error prone, because it is instantly obvious tothe reader that it is<x> that is being changed, and not<x> that isbeing replaced by something almost, but not quite, entirely unlike<x>.

The new in-place operations are especially useful to matrix calculation andother applications that require large objects. In order to efficiently dealwith the available program memory, such packages cannot blindly use thecurrent binary operations. Because these operations always create a newobject, adding a single item to an existing (large) object would result incopying the entire object (which may cause the application to run out ofmemory), add the single item, and then possibly delete the original object,depending on reference count.

To work around this problem, the packages currently have to use methods orfunctions to modify an object in-place, which is definitely less readable thanan augmented assignment expression. Augmented assignment won’t solve all theproblems for these packages, since some operations cannot be expressed in thelimited set of binary operators to start with, but it is a start.PEP 211is looking at adding new operators.

New methods

The proposed implementation adds the following 11 possiblehooks whichPython classes can implement to overload the augmented assignment operations:

__iadd____isub____imul____idiv____imod____ipow____ilshift____irshift____iand____ixor____ior__

Thei in__iadd__ stands forin-place.

For C extension types, the following struct members are added.

ToPyNumberMethods:

binaryfuncnb_inplace_add;binaryfuncnb_inplace_subtract;binaryfuncnb_inplace_multiply;binaryfuncnb_inplace_divide;binaryfuncnb_inplace_remainder;binaryfuncnb_inplace_power;binaryfuncnb_inplace_lshift;binaryfuncnb_inplace_rshift;binaryfuncnb_inplace_and;binaryfuncnb_inplace_xor;binaryfuncnb_inplace_or;

ToPySequenceMethods:

binaryfuncsq_inplace_concat;intargfuncsq_inplace_repeat;

In order to keep binary compatibility, thetp_flags TypeObject member isused to determine whether the TypeObject in question has allocated room forthese slots. Until a clean break in binary compatibility is made (which mayor may not happen before 2.0) code that wants to use one of the new structmembers must first check that they are available with thePyType_HasFeature() macro:

if(PyType_HasFeature(x->ob_type,Py_TPFLAGS_HAVE_INPLACE_OPS)&&x->ob_type->tp_as_number&&x->ob_type->tp_as_number->nb_inplace_add){/*...*/

This check must be made even before testing the method slots forNULLvalues! The macro only tests whether the slots are available, not whetherthey are filled with methods or not.

Implementation

The current implementation of augmented assignment[2] adds, in addition tothe methods and slots already covered, 13 new bytecodes and 13 new APIfunctions.

The API functions are simply in-place versions of the current binary-operationAPI functions:

PyNumber_InPlaceAdd(PyObject*o1,PyObject*o2);PyNumber_InPlaceSubtract(PyObject*o1,PyObject*o2);PyNumber_InPlaceMultiply(PyObject*o1,PyObject*o2);PyNumber_InPlaceDivide(PyObject*o1,PyObject*o2);PyNumber_InPlaceRemainder(PyObject*o1,PyObject*o2);PyNumber_InPlacePower(PyObject*o1,PyObject*o2);PyNumber_InPlaceLshift(PyObject*o1,PyObject*o2);PyNumber_InPlaceRshift(PyObject*o1,PyObject*o2);PyNumber_InPlaceAnd(PyObject*o1,PyObject*o2);PyNumber_InPlaceXor(PyObject*o1,PyObject*o2);PyNumber_InPlaceOr(PyObject*o1,PyObject*o2);PySequence_InPlaceConcat(PyObject*o1,PyObject*o2);PySequence_InPlaceRepeat(PyObject*o,intcount);

They call either the Python class hooks (if either of the objects is a Pythonclass instance) or the C type’s number or sequence methods.

The new bytecodes are:

INPLACE_ADDINPLACE_SUBTRACTINPLACE_MULTIPLYINPLACE_DIVIDEINPLACE_REMAINDERINPLACE_POWERINPLACE_LEFTSHIFTINPLACE_RIGHTSHIFTINPLACE_ANDINPLACE_XORINPLACE_ORROT_FOURDUP_TOPX

TheINPLACE_* bytecodes mirror theBINARY_* bytecodes, except thatthey are implemented as calls to theInPlace API functions. The other twobytecodes areutility bytecodes:ROT_FOUR behaves likeROT_THREEexcept that the four topmost stack items are rotated.

DUP_TOPX is a bytecode that takes a single argument, which should be aninteger between 1 and 5 (inclusive) which is the number of items to duplicatein one block. Given a stack like this (where the right side of the list isthetop of the stack):

[1,2,3,4,5]

DUP_TOPX3 would duplicate the top 3 items, resulting in this stack:

[1,2,3,4,5,3,4,5]

DUP_TOPX with an argument of 1 is the same asDUP_TOP. The limit of 5is purely an implementation limit . The implementation of augmentedassignment requires onlyDUP_TOPX with an argument of 2 and 3, and coulddo without this new opcode at the cost of a fair number ofDUP_TOP andROT_*.

Open Issues

ThePyNumber_InPlace API is only a subset of the normalPyNumber API:only those functions that are required to support the augmented assignmentsyntax are included. If other in-place API functions are needed, they can beadded later.

TheDUP_TOPX bytecode is a conveniency bytecode, and is not actuallynecessary. It should be considered whether this bytecode is worth having.There seems to be no other possible use for this bytecode at this time.

Copyright

This document has been placed in the public domain.

References