Background

PEP 472 was opened in 2014. The PEP detailed various use cases and was created byextracting implementation strategies from a broad discussion on thepython-ideas mailing list, although no clear consensus was reached on which strategyshould be used. Many corner cases have been examined more closely and feltawkward, backward incompatible or both.

The PEP was eventually rejected in 2019[1] mostlydue to lack of interest for the feature despite its 5 years of existence.

However, with the introduction of type hints inPEP 484 thesquare bracket notation has been used consistently to enrich the typingannotations, e.g. to specify a list of integers as Sequence[int]. Additionally,there has been an expanded growth of packages for data analysis such as pandasand xarray, which use names to describe columns in a table (pandas) or axis inan nd-array (xarray). These packages allow users to access specific data bynames, but cannot currently use index notation ([]) for this functionality.

As a result, a renewed interest in a more flexible syntax that would allow fornamed information has been expressed occasionally in many different threads onpython-ideas, recently by Caleb Donovick[2] in 2019 and AndrasTantos[3] in 2020. These requests prompted a strong activity on thepython-ideas mailing list, where the various options have been re-discussed anda general consensus on an implementation strategy has now been reached.

Use cases

The following practical use cases present different cases where a keywordspecification would improve notation and provide additional value:

1. To provide a more communicative meaning to the index, preventing e.g. accidentalinversion of indexes:

   >>>grid_position[x=3,y=5,z=8]>>>rain_amount[time=0:12,location=location]>>>matrix[row=20,col=40]

2. To enrich the typing notation with keywords, especially during the use of generics:

   deffunction(value:MyType[T=int]):

3. In some domain, such as computational physics and chemistry, the use of anotation such asBasis[Z=5] is a Domain Specific Language notation to representa level of accuracy:

   >>>low_accuracy_energy=computeEnergy(molecule,BasisSet[Z=3])

4. Pandas currently uses a notation such as:

   >>>df[df['x']==1]

   which could be replaced withdf[x=1].

5. xarray has named dimensions. Currently these are handled with functions .isel:

   >>>data.isel(row=10)# Returns the tenth row

   which could also be replaced withdata[row=10]. A more complex example:

   >>># old syntax>>>da.isel(space=0,time=slice(None,2))[...]=spam>>># new syntax>>>da[space=0,time=:2]=spam

   Another example:

   >>># old syntax>>>ds["empty"].loc[dict(lon=5,lat=6)]=10>>># new syntax>>>ds["empty"][lon=5,lat=6]=10>>># old syntax>>>ds["empty"].loc[dict(lon=slice(1,5),lat=slice(3,None))]=10>>># new syntax>>>ds["empty"][lon=1:5,lat=6:]=10

6. Functions/methods whose argument is another function (plus itsarguments) need some way to determine which arguments are destined forthe target function, and which are used to configure how they run thetarget. This is simple (if non-extensible) for positional parameters,but we need some way to distinguish these for keywords.[4]

   An indexed notation would afford a Pythonic way to pass keywordarguments to these functions without cluttering the caller’s code.

   >>># Let's start this example with basic syntax without keywords.>>># the positional values are arguments to `func` while>>># `name=` is processed by `trio.run`.>>>trio.run(func,value1,value2,name="func")>>># `trio.run` ends up calling `func(value1, value2)`.>>># If we want/need to pass value2 by keyword (keyword-only argument,>>># additional arguments that won't break backwards compatibility ...),>>># currently we need to resort to functools.partial:>>>trio.run(functools.partial(func,param2=value2),value1,name="func")>>>trio.run(functools.partial(func,value1,param2=value2),name="func")>>># One possible workaround is to convert `trio.run` to an object>>># with a `__call__` method, and use an "option" helper,>>>trio.run.option(name="func")(func,value1,param2=value2)>>># However, foo(bar)(baz) is uncommon and thus disruptive to the reader.>>># Also, you need to remember the name of the `option` method.>>># This PEP allows us to replace `option` with `__getitem__`.>>># The call is now shorter, more mnemonic, and looks+works like typing>>>trio.run[name="func"](func,value1,param2=value2)

7. Availability of star arguments would benefitPEP 646 Variadic Generics,especially in the formsa[*x] anda[*x,*y,p,q,*z]. The PEP detailsexactly this notation in its “Unpacking: Star Operator” section.

It is important to note that how the notation is interpreted is up to theimplementation. This PEP only defines and dictates the behavior of Pythonregarding passed keyword arguments, not how these arguments should beinterpreted and used by the implementing class.

Current status of indexing operation

Before detailing the new syntax and semantics to the indexing notation, it isrelevant to analyse how the indexing notation works today, in which contexts,and how it is different from a function call.

Subscriptingobj[x] is, effectively, an alternate and specialised form offunction call syntax with a number of differences and restrictions compared toobj(x). The current Python syntax focuses exclusively on position to expressthe index, and also contains syntactic sugar to refer to non-punctiformselection (slices). Some common examples:

>>>a[3]# returns the fourth element of 'a'>>>a[1:10:2]# slice notation (extract a non-trivial data subset)>>>a[3,2]# multiple indexes (for multidimensional arrays)

This translates into a__(get|set|del)item__ dunder call which is passed a singleparameter containing the index (for__getitem__ and__delitem__) or two parameterscontaining index and value (for__setitem__).

The behavior of the indexing call is fundamentally different from a function callin various aspects:

The first difference is in meaning to the reader. A function call says“arbitrary function call potentially with side-effects”. An indexing operationsays “lookup”, typically to point at a subset or specific sub-aspect of anentity (as in the case of typing notation). This fundamental difference meansthat, while we cannot prevent abuse, implementors should be aware that theintroduction of keyword arguments to alter the behavior of the lookup mayviolate this intrinsic meaning.

The second difference of the indexing notation compared to a functionis that indexing can be used for both getting and setting operations.In Python, a function cannot be on the left hand side of an assignment. Inother words, both of these are valid:

>>>x=a[1,2]>>>a[1,2]=5

but only the first one of these is valid:

>>>x=f(1,2)>>>f(1,2)=5# invalid

This asymmetry is important, and makes one understand that there is a naturalimbalance between the two forms. It is therefore not a given that the twoshould behave transparently and symmetrically.

The third difference is that functions have names assigned to theirarguments, unless the passed parameters are captured with*args, in which casethey end up as entries in the args tuple. In other words, functions alreadyhave anonymous argument semantic, exactly like the indexing operation. However,__(get|set|del)item__ is not always receiving a tuple as theindex argument(to be uniform in behavior with*args). In fact, given a trivial class:

classX:def__getitem__(self,index):print(index)

The index operation basically forwards the content of the square brackets “as is”in theindex argument:

>>>x=X()>>>x[0]0>>>x[0,1](0, 1)>>>x[(0,1)](0, 1)>>>>>>x[()]()>>>x[{1,2,3}]{1, 2, 3}>>>x["hello"]hello>>>x["hello","hi"]('hello', 'hi')

The fourth difference is that the indexing operation knows how to convertcolon notations to slices, thanks to support from the parser. This is valid:

a[1:3]

this one isn’t:

f(1:3)

The fifth difference is that there’s no zero-argument form. This is valid:

f()

this one isn’t:

a[]

Indexing behavior in standard classes (dict, list, etc.)

None of what is proposed in this PEP will change the behavior of the currentcore classes that use indexing. Adding keywords to the index operation forcustom classes is not the same as modifying e.g. the standard dict type tohandle keyword arguments. In fact, dict (as well as list and other stdlibclasses with indexing semantics) will remain the same and will continue not toaccept keyword arguments. In other words, ifd is adict, thestatementd[1,a=2] will raiseTypeError, as their implementation willnot support the use of keyword arguments. The same holds for all other classes(list, dict, etc.)

Corner case and Gotchas

With the introduction of the new notation, a few corner cases need to be analysed.

1. Technically, if a class defines their getter like this:

   def__getitem__(self,index):

   then the caller could call that using keyword syntax, like these two cases:

   obj[3,index=4]obj[index=1]

   The resulting behavior would be an error automatically, since it would be likeattempting to call the method with two values for theindex argument, andaTypeError will be raised. In the first case, theindex would be3,in the second case, it would be the empty tuple().

   Note that this behavior applies for all currently existing classes that rely onindexing, meaning that there is no way for the new behavior to introducebackward compatibility issues on this respect.

   Classes that wish to stress this behavior explicitly can define theirparameters as positional-only:

   def__getitem__(self,index,/):

2. a similar case occurs with setter notation:

   # Given type(obj).__setitem__(obj, index, value):obj[1,value=3]=5

   This poses no issue because the value is passed automatically, and the Python interpreter will raiseTypeError:gotmultiplevaluesforkeywordargument'value'

3. If the subscript dunders are declared to use positional-or-keywordparameters, there may be some surprising cases when arguments are passedto the method. Given the signature:

   def__getitem__(self,index,direction='north')

   if the caller uses this:

   obj[0,'south']

   they will probably be surprised by the method call:

   # expected type(obj).__getitem__(obj, 0, direction='south')# but actually get:type(obj).__getitem__(obj,(0,'south'),direction='north')

   Solution: best practice suggests that keyword subscripts should beflagged as keyword-only when possible:

   def__getitem__(self,index,*,direction='north')

   The interpreter need not enforce this rule, as there could be scenarioswhere this is the desired behaviour. But linters may choose to warnabout subscript methods which don’t use the keyword-only flag.

4. As we saw, a single value followed by a keyword argument will not be changed into a tuple, i.e.:d[1,a=3] is treated as__getitem__(d,1,a=3), NOT__getitem__(d,(1,),a=3). It would beextremely confusing if adding keyword arguments were to change the type of the passed index.In other words, adding a keyword to a single-valued subscript will not change it into a tuple.For those cases where an actual tuple needs to be passed, a proper syntax will have to be used:

   obj[(1,),a=3]# calls type(obj).__getitem__(obj, (1,), a=3)

   In this case, the call is passing a single element (which is passed as is, as from rule above),only that the single element happens to be a tuple.

   Note that this behavior just reveals the truth that theobj[1,] notation is shorthand forobj[(1,)] (and alsoobj[1] is shorthand forobj[(1)], with the expected behavior).When keywords are present, the rule that you can omit this outermost pair of parentheses is nolonger true:

   obj[1]# calls type(obj).__getitem__(obj, 1)obj[1,a=3]# calls type(obj).__getitem__(obj, 1, a=3)obj[1,]# calls type(obj).__getitem__(obj, (1,))obj[(1,),a=3]# calls type(obj).__getitem__(obj, (1,), a=3)

   This is particularly relevant in the case where two entries are passed:

   obj[1,2]# calls type(obj).__getitem__(obj, (1, 2))obj[(1,2)]# same as aboveobj[1,2,a=3]# calls type(obj).__getitem__(obj, (1, 2), a=3)obj[(1,2),a=3]# calls type(obj).__getitem__(obj, (1, 2), a=3)

   And particularly when the tuple is extracted as a variable:

   t=(1,2)obj[t]# calls type(obj).__getitem__(obj, (1, 2))obj[t,a=3]# calls type(obj).__getitem__(obj, (1, 2), a=3)

   Why? because in the caseobj[1,2,a=3] we are passing two elements (whichare then packed as a tuple and passed as the index). In the caseobj[(1,2),a=3]we are passing a single element (which is passed as is) which happens to be a tuple.The final result is that they are the same.

Previous PEP 472 solutions

PEP 472 presents a good amount of ideas that are now all to be consideredRejected. A personal email from D’Aprano to the author specifically said:

I have now carefully read through PEP 472 in full, and I am afraid Icannot support any of the strategies currently in the PEP.

We agree that those options are inferior to the currently presented, for onereason or another.

To keep this document compact, we will not present here the objections forall options presented inPEP 472. Suffice to say that they were discussed,and each proposed alternative had one or few dealbreakers.

Adding new dunders

It was proposed to introduce new dunders__(get|set|del)item_ex__that are invoked over the__(get|set|del)item__ triad, if they are present.

The rationale around this choice is to make the intuition around how to add kwdarg support to square brackets more obvious and in line with the functionbehavior. Given:

def__getitem_ex__(self,x,y):...

These all just work and produce the same result effortlessly:

obj[1,2]obj[1,y=2]obj[y=2,x=1]

In other words, this solution would unify the behavior of__getitem__ to the traditionalfunction signature, but since we can’t change__getitem__ and break backward compatibility,we would have an extended version that is used preferentially.

The problems with this approach were found to be:

• It will slow down subscripting. For every subscript access, this new dunderattribute gets investigated on the class, and if it is not present then thedefault key translation function is executed.Different ideas were proposed to handle this, from wrapping the methodonly at class instantiation time, to add a bit flag to signal the availabilityof these methods. Regardess of the solution, the new dunder would be effectiveonly if added at class creation time, not if it’s added later. This wouldbe unusual and would disallow (and behave unexpectedly) monkeypatching of themethods for whatever reason it might be needed.
• It adds complexity to the mechanism.
• Will require a long and painful transition period during which timelibraries will have to somehow support both calling conventions, because mostlikely, the extended methods will delegate to the traditional ones when theright conditions are matched in the arguments, or some classes will supportthe traditional dunder and others the extended dunder. While this will notaffect calling code, it will affect development.
• it would potentially lead to mixed situations where the extended version isdefined for the getter, but not for the setter.
• In the__setitem_ex__ signature, value would have to be made the firstelement, because the index is of arbitrary length depending on the specifiedindexes. This would look awkward because the visual notation does not matchthe signature:

  obj[1,2]=3# calls type(obj).__setitem_ex__(obj, 3, 1, 2)

• the solution relies on the assumption that all keyword indices necessarily mapinto positional indices, or that they must have a name. This assumption may befalse: xarray, which is the primary Python package for numpy arrays withlabelled dimensions, supports indexing by additional dimensions (so called“non-dimension coordinates”) that don’t correspond directly to the dimensionsof the underlying numpy array, and those have no position to match up to.In other words, anonymous indexes are a plausible use case that this solutionwould remove, although it could be argued that using*args would solvethat issue.

Adding an adapter function

Similar to the above, in the sense that a pre-function would be called toconvert the “new style” indexing into “old style indexing” that is then passed.Has problems similar to the above.

create a new “kwslice” object

This proposal has already been explored in “New arguments contents” P4 inPEP 472:

obj[a,b:c,x=1]# calls type(obj).__getitem__(obj, a, slice(b, c), key(x=1))

This solution requires everyone who needs keyword arguments to parse the tupleand/or key object by hand to extract them. This is painful and opens up to theget/set/del function to always accept arbitrary keyword arguments, whether theymake sense or not. We want the developer to be able to specify which argumentsmake sense and which ones do not.

Using a single bit to change the behavior

A special class dunder flag:

__keyfn__=True

would change the signature of the__get|set|delitem__ to a “function like” dispatch,meaning that this:

>>>d[1,2,z=3]

would result in a call to:

>>>type(obj).__getitem__(obj,1,2,z=3)# instead of type(obj).__getitem__(obj, (1, 2), z=3)

This option has been rejected because it feels odd that a signature of a methoddepends on a specific value of another dunder. It would be confusing for bothstatic type checkers and for humans: a static type checker would have to hard-codea special case for this, because there really is nothing else in Pythonwhere the signature of a dunder depends on the value of another dunder.A human that has to implement a__getitem__ dunder would have to look if in theclass (or in any of its subclasses) for a__keyfn__ before the dunder can be written.Moreover, adding a base classes that have the__keyfn__ flag set would breakthe signature of the current methods. This would be even more problematic if theflag is changed at runtime, or if the flag is generated by calling a functionthat returns randomly True or something else.

Allowing for empty index notation obj[]

The current proposal preventsobj[] from being valid notation. Howevera commenter stated

We haveTuple[int,int] as a tuple of two integers. And we haveTuple[int]as a tuple of one integer. And occasionally we need to spell a tuple ofnovalues, since that’s the type of(). But we currently are forced to writethat asTuple[()]. If we allowedTuple[] that odd edge case would beremoved.

So I probably would be okay with allowingobj[] syntactically, as long as thedict type could be made to reject it.

This proposal already established that, in case no positional index is given, thepassed value must be the empty tuple. Allowing for the empty index notation wouldmake the dictionary type accept it automatically, to insert or refer to the value withthe empty tuple as key. Moreover, a typing notation such asTuple[] can easilybe written asTuple without the indexing notation.

However, subsequent discussion with Brandt Bucher during implementation has revealedthat the caseobj[] would fit a natural evolution for variadic generics, givingmore strength to the above comment. In the end, after a discussion between D’Aprano,Bucher and the author, we decided to leave theobj[] notation as a syntaxerror for now, and possibly extend the notation with an additional PEP to holdthe equivalenceobj[] asobj[()].

Sentinel value for no given positional index

The topic of which value to pass as the index in the case of:

obj[k=3]

has been considerably debated.

One apparently rational choice would be to pass no value at all, by making use ofthe keyword only argument feature, but unfortunately will not work well withthe__setitem__ dunder, as a positional element for the value is alwayspassed, and we can’t “skip over” the index one unless we introduce a very weird behaviorwhere the first argument refers to the index when specified, and to the value whenindex is not specified. This is extremely deceiving and error prone.

The above consideration makes it impossible to have a keyword only dunder, andopens up the question of what entity to pass for the index position when no indexis passed:

obj[k=3]=5# would call type(obj).__setitem__(obj, ???, 5, k=3)

A proposed hack would be to let the user specify which entity to use when anindex is not specified, by specifying a default for theindex, but thisforces necessarily to also specify a (never going to be used, as a value isalways passed by design) default for thevalue, as we can’t havenon-default arguments after defaulted one:

def__setitem__(self,index=SENTINEL,value=NEVERUSED,*,k)

which seems ugly, redundant and confusing. We must therefore accept that someform of sentinel index must be passed by the Python implementation when theobj[k=3] notation is used. This also means that default arguments to thoseparameters are simply never going to be used (but it’s already thecase with the current implementation, so no change there).

Additionally, some classes may want to use**kwargs, instead of a keyword-onlyargument, meaning that having a definition like:

def__setitem__(self,index,value,**kwargs):

and a user that wants to pass a keywordvalue:

x[value=1]=0

expecting a call like:

type(obj).__setitem__(obj,SENTINEL,0,**{"value":1})

will instead accidentally be caught by the namedvalue, producing aduplicatevalueerror. The user should not be worried about the actuallocal names of those two arguments if they are, for all practical purposes,positional only. Unfortunately, using positional-only values will ensure thisdoes not happen but it will still not solve the need to pass bothindex andvalue even when the index is not provided. The point is that the user should notbe prevented to use keyword arguments to refer to a columnindex,value(orself) just because the class implementor happens to use those namesin the parameter list.

Moreover, we also require the three dunders to behave in the same way: it wouldbe extremely inconvenient if only__setitem__ were to receive thissentinel, and__get|delitem__ would not because they can get away with asignature that allows for no index specification, thus allowing for auser-specified default index.

Whatever the choice of the sentinel, it will make the following casesdegenerate and thus impossible to differentiate in the dunder:

obj[k=3]obj[SENTINEL,k=3]

The question now shifts to which entity should represent the sentinel:the options were:

1. Empty tuple
2. None
3. NotImplemented
4. a new sentinel object (e.g. NoIndex)

For option 1, the call will become:

type(obj).__getitem__(obj,(),k=3)

therefore makingobj[k=3] andobj[(),k=3] degenerate and indistinguishable.

This option sounds appealing because:

1. The numpy community was inquired[5], and the general consensusof the responses was that the empty tuple felt appropriate.
2. It shows a parallel with the behavior of*args in a function, whenno positional arguments are given:

   >>>deffoo(*args,**kwargs):...print(args,kwargs)...>>>foo(k=3)() {'k': 3}

   Although we do accept the following asymmetry in behavior compared to functionswhen a single value is passed, but that ship has sailed:

   >>>foo(5,k=3)(5,) {'k': 3}   # for indexing, a plain 5, not a 1-tuple is passed

For option 2, usingNone, it was objected that NumPy uses it to indicateinserting a new axis/dimensions (there’s anp.newaxis alias as well):

arr=np.array(5)arr.ndim==0arr[None].ndim==arr[None,].ndim==1

While this is not an insurmountable issue, it certainly will ripple onto numpy.

The only issues with both the above is that both the empty tuple and None arepotential legitimate indexes, and there might be value in being able to differentiatethe two degenerate cases.

So, an alternative strategy (option 3) would be to use an existing entity that isunlikely to be used as a valid index. One option could be the current built-in constantNotImplemented, which is currently returned by operators methods toreport that they do not implement a particular operation, and a different strategyshould be attempted (e.g. to ask the other object). Unfortunately, its name andtraditional use calls back to a feature that is not available, rather than thefact that something was not passed by the user.

This leaves us with option 4: a new built-in constant. This constantmust be unhashable (so it’s never going to be a valid key) and have a clearname that makes it obvious its context:NoIndex. Thiswould solve all the above issues, but the question is: is it worth it?

From a quick inquire, it seems that most people on python-ideas seem to believeit’s not crucial, and the empty tuple is an acceptable option. Hence theresulting series will be:

obj[k=3]# type(obj).__getitem__(obj, (), k=3). Empty tupleobj[1,k=3]# type(obj).__getitem__(obj, 1, k=3). Integerobj[1,2,k=3]# type(obj).__getitem__(obj, (1, 2), k=3). Tuple

and the following two notation will be degenerate:

obj[(),k=3]# type(obj).__getitem__(obj, (), k=3)obj[k=3]# type(obj).__getitem__(obj, (), k=3)