Specialness ofNone

TheNone object denotes the lack of a value. For the purposes of theseoperators, the lack of a value indicates that the remainder of the expressionalso lacks a value and should not be evaluated.

A rejected proposal was to treat any value that evaluates as “false” in aBoolean context as not having a value. However, the purpose of these operatorsis to propagate the “lack of value” state, rather than the “false” state.

Some argue that this makesNone special. We contend thatNone isalready special, and that using it as both the test and the result of theseoperators does not change the existing semantics in any way.

See theRejected Ideas section for discussions on alternate approaches.

Grammar changes

The following rules of the Python grammar are updated to read:

augassign:('+='|'-='|'*='|'@='|'/='|'%='|'&='|'|='|'^='|'<<='|'>>='|'**='|'//='|'??=')power:coalesce['**'factor]coalesce:atom_expr['??'factor]atom_expr:['await']atomtrailer*trailer:('('[arglist]')'|'['subscriptlist']'|'?['subscriptlist']'|'.'NAME|'?.'NAME)

a, b = None, Nonedef c(): return Nonedef ex(): raise Exception()(a ?? 2 ** b ?? 3) == a ?? (2 ** (b ?? 3))(a * b ?? c // d) == a * (b ?? c) // d(a ?? True and b ?? False) == (a ?? True) and (b ?? False)(c() ?? c() ?? True) == True(True ?? ex()) == True(c ?? ex)() == c()

a = Noneb = ''c = 0a ??= 'value'b ??= undefined_namec ??= shutil.rmtree('/')    # don't try this at home, kidsassert a == 'value'assert b == ''assert c == 0 and any(os.scandir('/'))

_v=a_v=_v.b_v=_v(c)_v=_v.d_v=_v[e]await_v

_v=aif_visnotNone:_v=_v.b_v=_v(c)_v=_v.dif_visnotNone:_v=_v[e]await_v

# a?.b_v=aif_visnotNone:_v=_v.b# ... ?? cif_visNone:_v=c# (...).d?.e_v=_v.dif_visnotNone:_v=_v.e

(a?.b ?? d).c = 1

Reading expressions

For the maybe-dot and maybe-subscript operators, the intention is thatexpressions including these operators should be read and interpreted as for theregular versions of these operators. In “normal” cases, the end results aregoing to be identical between an expression such asa?.b?[c] anda.b[c], and just as we do not currently read “a.b” as “read attribute bfrom aif it has an attribute a or else it raises AttributeError”, there isno need to read “a?.b” as “read attribute b from aif a is not None”(unless in a context where the listener needs to be aware of the specificbehaviour).

For coalescing expressions using the?? operator, expressions should eitherbe read as “or … if None” or “coalesced with”. For example, the expressiona.get_value()??100 would be read “call a dot get_value or 100 if None”,or “call a dot get_value coalesced with 100”.

Standard Library

Using thefind-pep505.py script[5] an analysis of the Python 3.7 standardlibrary discovered up to 678 code snippets that could be replaced with use ofone of theNone-aware operators:

$ find /usr/lib/python3.7 -name '*.py' | xargs python3.7 find-pep505.py<snip>Total None-coalescing `if` blocks: 449Total [possible] None-coalescing `or`: 120Total None-coalescing ternaries: 27Total Safe navigation `and`: 13Total Safe navigation `if` blocks: 61Total Safe navigation ternaries: 8

Some of these are shown below as examples before and after converting to use thenew operators.

Frombisect.py:

definsort_right(a,x,lo=0,hi=None):# ...ifhiisNone:hi=len(a)# ...

After updating to use the??= augmented assignment statement:

def insort_right(a, x, lo=0, hi=None):    # ...    hi ??= len(a)    # ...

Fromcalendar.py:

encoding=options.encodingifencodingisNone:encoding=sys.getdefaultencoding()optdict=dict(encoding=encoding,css=options.css)

After updating to use the?? operator:

optdict = dict(encoding=options.encoding ?? sys.getdefaultencoding(),               css=options.css)

Fromemail/generator.py (and importantly note that there is no way tosubstituteor for?? in this situation):

mangle_from_=TrueifpolicyisNoneelsepolicy.mangle_from_

After updating:

mangle_from_ = policy?.mangle_from_ ?? True

Fromasyncio/subprocess.py:

defpipe_data_received(self,fd,data):iffd==1:reader=self.stdouteliffd==2:reader=self.stderrelse:reader=NoneifreaderisnotNone:reader.feed_data(data)

After updating to use the?. operator:

def pipe_data_received(self, fd, data):    if fd == 1:        reader = self.stdout    elif fd == 2:        reader = self.stderr    else:        reader = None    reader?.feed_data(data)

Fromasyncio/tasks.py:

try:awaitwaiterfinally:iftimeout_handleisnotNone:timeout_handle.cancel()

After updating to use the?. operator:

try:    await waiterfinally:    timeout_handle?.cancel()

Fromctypes/_aix.py:

iflibpathsisNone:libpaths=[]else:libpaths=libpaths.split(":")

After updating:

libpaths = libpaths?.split(":") ?? []

Fromos.py:

ifentry.is_dir():dirs.append(name)ifentriesisnotNone:entries.append(entry)else:nondirs.append(name)

After updating to use the?. operator:

if entry.is_dir():    dirs.append(name)    entries?.append(entry)else:    nondirs.append(name)

Fromimportlib/abc.py:

deffind_module(self,fullname,path):ifnothasattr(self,'find_spec'):returnNonefound=self.find_spec(fullname,path)returnfound.loaderiffoundisnotNoneelseNone

After partially updating:

def find_module(self, fullname, path):    if not hasattr(self, 'find_spec'):        return None    return self.find_spec(fullname, path)?.loader

After extensive updating (arguably excessive, though that’s for the styleguides to determine):

def find_module(self, fullname, path):    return getattr(self, 'find_spec', None)?.__call__(fullname, path)?.loader

Fromdis.py:

def_get_const_info(const_index,const_list):argval=const_indexifconst_listisnotNone:argval=const_list[const_index]returnargval,repr(argval)

After updating to use the?[] and?? operators:

def _get_const_info(const_index, const_list):    argval = const_list?[const_index] ?? const_index    return argval, repr(argval)

jsonify

This example is from a Python web crawler that uses the Flask framework as itsfront-end. This function retrieves information about a web site from a SQLdatabase and formats it as JSON to send to an HTTP client:

classSiteView(FlaskView):@route('/site/<id_>',methods=['GET'])defget_site(self,id_):site=db.query('site_table').find(id_)returnjsonify(first_seen=site.first_seen.isoformat()ifsite.first_seenisnotNoneelseNone,id=site.id,is_active=site.is_active,last_seen=site.last_seen.isoformat()ifsite.last_seenisnotNoneelseNone,url=site.url.rstrip('/'))

Bothfirst_seen andlast_seen are allowed to benull in thedatabase, and they are also allowed to benull in the JSON response. JSONdoes not have a native way to represent adatetime, so the server’s contractstates that any non-null date is represented as an ISO-8601 string.

Without knowing the exact semantics of thefirst_seen andlast_seenattributes, it is impossible to know whether the attribute can be safely orperformantly accessed multiple times.

One way to fix this code is to replace each conditional expression with anexplicit value assignment and a fullif/else block:

classSiteView(FlaskView):@route('/site/<id_>',methods=['GET'])defget_site(self,id_):site=db.query('site_table').find(id_)first_seen_dt=site.first_seeniffirst_seen_dtisNone:first_seen=Noneelse:first_seen=first_seen_dt.isoformat()last_seen_dt=site.last_seeniflast_seen_dtisNone:last_seen=Noneelse:last_seen=last_seen_dt.isoformat()returnjsonify(first_seen=first_seen,id=site.id,is_active=site.is_active,last_seen=last_seen,url=site.url.rstrip('/'))

This adds ten lines of code and four new code paths to the function,dramatically increasing the apparent complexity. Rewriting using theNone-aware attribute operator results in shorter code with more clearintent:

class SiteView(FlaskView):    @route('/site/<id_>', methods=['GET'])    def get_site(self, id_):        site = db.query('site_table').find(id_)        return jsonify(            first_seen=site.first_seen?.isoformat(),            id=site.id,            is_active=site.is_active,            last_seen=site.last_seen?.isoformat(),            url=site.url.rstrip('/')        )

Grab

The next example is from a Python scraping library calledGrab:

classBaseUploadObject(object):deffind_content_type(self,filename):ctype,encoding=mimetypes.guess_type(filename)ifctypeisNone:return'application/octet-stream'else:returnctypeclassUploadContent(BaseUploadObject):def__init__(self,content,filename=None,content_type=None):self.content=contentiffilenameisNone:self.filename=self.get_random_filename()else:self.filename=filenameifcontent_typeisNone:self.content_type=self.find_content_type(self.filename)else:self.content_type=content_typeclassUploadFile(BaseUploadObject):def__init__(self,path,filename=None,content_type=None):self.path=pathiffilenameisNone:self.filename=os.path.split(path)[1]else:self.filename=filenameifcontent_typeisNone:self.content_type=self.find_content_type(self.filename)else:self.content_type=content_type

This example contains several good examples of needing to provide defaultvalues. Rewriting to use conditional expressions reduces the overall lines ofcode, but does not necessarily improve readability:

classBaseUploadObject(object):deffind_content_type(self,filename):ctype,encoding=mimetypes.guess_type(filename)return'application/octet-stream'ifctypeisNoneelsectypeclassUploadContent(BaseUploadObject):def__init__(self,content,filename=None,content_type=None):self.content=contentself.filename=(self.get_random_filename()iffilenameisNoneelsefilename)self.content_type=(self.find_content_type(self.filename)ifcontent_typeisNoneelsecontent_type)classUploadFile(BaseUploadObject):def__init__(self,path,filename=None,content_type=None):self.path=pathself.filename=(os.path.split(path)[1]iffilenameisNoneelsefilename)self.content_type=(self.find_content_type(self.filename)ifcontent_typeisNoneelsecontent_type)

The first ternary expression is tidy, but it reverses the intuitive order ofthe operands: it should returnctype if it has a value and use the stringliteral as fallback. The other ternary expressions are unintuitive and solong that they must be wrapped. The overall readability is worsened, notimproved.

Rewriting using theNone coalescing operator:

class BaseUploadObject(object):    def find_content_type(self, filename):        ctype, encoding = mimetypes.guess_type(filename)        return ctype ?? 'application/octet-stream'class UploadContent(BaseUploadObject):    def __init__(self, content, filename=None, content_type=None):        self.content = content        self.filename = filename ?? self.get_random_filename()        self.content_type = content_type ?? self.find_content_type(self.filename)class UploadFile(BaseUploadObject):    def __init__(self, path, filename=None, content_type=None):        self.path = path        self.filename = filename ?? os.path.split(path)[1]        self.content_type = content_type ?? self.find_content_type(self.filename)

This syntax has an intuitive ordering of the operands. Infind_content_type,for example, the preferred valuectype appears before the fallback value.The terseness of the syntax also makes for fewer lines of code and less code tovisually parse, and reading from left-to-right and top-to-bottom more accuratelyfollows the execution flow.

No-Value Protocol

The operators could be generalised to user-defined types by defining a protocolto indicate when a value represents “no value”. Such a protocol may be a dundermethod__has_value__(self) that returnsTrue if the value should betreated as having a value, andFalse if the value should be treated as novalue.

With this generalization,object would implement a dunder method equivalentto this:

def__has_value__(self):returnTrue

NoneType would implement a dunder method equivalent to this:

def__has_value__(self):returnFalse

In the specification section, all uses ofxisNone would be replaced withnotx.__has_value__().

This generalization would allow for domain-specific “no-value” objects to becoalesced just likeNone. For example, thepyasn1 package has a typecalledNull that represents an ASN.1null:

>>> from pyasn1.type import univ>>> univ.Null() ?? univ.Integer(123)Integer(123)

Similarly, values such asmath.nan andNotImplemented could be treatedas representing no value.

However, the “no-value” nature of these values is domain-specific, which meanstheyshould be treated as a value by the language. For example,math.nan.imag is well defined (it’s0.0), and so short-circuitingmath.nan?.imag to returnmath.nan would be incorrect.

AsNone is already defined by the language as being the value thatrepresents “no value”, and the current specification would not precludeswitching to a protocol in the future (though changes to built-in objects wouldnot be compatible), this idea is rejected for now.

Boolean-aware operators

This suggestion is fundamentally the same as adding a no-value protocol, and sothe discussion above also applies.

Similar behavior to the?? operator can be achieved with anorexpression, howeveror checks whether its left operand is false-y and notspecificallyNone. This approach is attractive, as it requires fewer changesto the language, but ultimately does not solve the underlying problem correctly.

Assuming the check is for truthiness rather thanNone, there is no longer aneed for the?? operator. However, applying this check to the?. and?[] operators prevents perfectly valid operations applying

Consider the following example, whereget_log_list() may return either alist containing current log messages (potentially empty), orNone if loggingis not enabled:

lst = get_log_list()lst?.append('A log message')

If?. is checking for true values rather than specificallyNone and thelog has not been initialized with any items, no item will ever be appended. Thisviolates the obvious intent of the code, which is to append an item. Theappend method is available on an empty list, as are all other list methods,and there is no reason to assume that these members should not be used becausethe list is presently empty.

Further, there is no sensible result to use in place of the expression. Anormallst.append returnsNone, but under this idealst?.append mayresult in either[] orNone, depending on the value oflst. As withthe examples in the previous section, this makes no sense.

As checking for truthiness rather thanNone results in apparently validexpressions no longer executing as intended, this idea is rejected.

Exception-aware operators

Arguably, the reason to short-circuit an expression whenNone is encounteredis to avoid theAttributeError orTypeError that would be raised undernormal circumstances. As an alternative to testing forNone, the?. and?[] operators could instead handleAttributeError andTypeErrorraised by the operation and skip the remainder of the expression.

This produces a transformation fora?.b.c?.d.e similar to this:

_v=atry:_v=_v.bexceptAttributeError:passelse:_v=_v.ctry:_v=_v.dexceptAttributeError:passelse:_v=_v.e

One open question is which value should be returned as the expression when anexception is handled. The above example simply leaves the partial result, butthis is not helpful for replacing with a default value. An alternative would beto force the result toNone, which then raises the question as to whyNone is special enough to be the result but not special enough to be thetest.

Secondly, this approach masks errors within code executed implicitly as part ofthe expression. For?., anyAttributeError within a property or__getattr__ implementation would be hidden, and similarly for?[] and__getitem__ implementations.

Similarly, simple typing errors such as{}?.ietms() could go unnoticed.

Existing conventions for handling these kinds of errors in the form of thegetattr builtin and the.get(key,default) method pattern established bydict show that it is already possible to explicitly use this behaviour.

As this approach would hide errors in code, it is rejected.

None-aware Function Call

TheNone-aware syntax applies to attribute and index access, so it seemsnatural to ask if it should also apply to function invocation syntax. It mightbe written asfoo?(), wherefoo is only called if it is not None.

This has been deferred on the basis of the proposed operators being intendedto aid traversal of partially populated hierarchical data structures,notfor traversal of arbitrary class hierarchies. This is reflected in the factthat none of the other mainstream languages that already offer this syntaxhave found it worthwhile to support a similar syntax for optional functioninvocations.

A workaround similar to that used by C# would be to writemaybe_none?.__call__(arguments). If the callable isNone, theexpression will not be evaluated. (The C# equivalent uses?.Invoke() on itscallable type.)

? Unary Postfix Operator

To generalize theNone-aware behavior and limit the number of new operatorsintroduced, a unary, postfix operator spelled? was suggested. The idea isthat? might return a special object that could would override dundermethods that returnself. For example,foo? would evaluate tofoo ifit is notNone, otherwise it would evaluate to an instance ofNoneQuestion:

classNoneQuestion():def__call__(self,*args,**kwargs):returnselfdef__getattr__(self,name):returnselfdef__getitem__(self,key):returnself

With this new operator and new type, an expression likefoo?.bar[baz]evaluates toNoneQuestion iffoo is None. This is a niftygeneralization, but it’s difficult to use in practice since most existing codewon’t know whatNoneQuestion is.

Going back to one of the motivating examples above, consider the following:

>>> import json>>> created = None>>> json.dumps({'created': created?.isoformat()})

The JSON serializer does not know how to serializeNoneQuestion, nor willany other API. This proposal actually requireslots of specialized logicthroughout the standard library and any third party library.

At the same time, the? operator may also betoo general, in the sensethat it can be combined with any other operator. What should the followingexpressions mean?:

>>> x? + 1>>> x? -= 1>>> x? == 1>>> ~x?

This degree of generalization is not useful. The operators actually proposedherein are intentionally limited to a few operators that are expected to make iteasier to write common code patterns.

Built-inmaybe

Haskell has a concept calledMaybe thatencapsulates the idea of an optional value without relying on any specialkeyword (e.g.null) or any special instance (e.g.None). In Haskell, thepurpose ofMaybe is to avoid separate handling of “something” and nothing”.

A Python package calledpymaybe provides arough approximation. The documentation shows the following example:

>>>maybe('VALUE').lower()'value'>>>maybe(None).invalid().method().or_else('unknown')'unknown'

The functionmaybe() returns either aSomething instance or aNothing instance. Similar to the unary postfix operator described in theprevious section,Nothing overrides dunder methods in order to allowchaining on a missing value.

Note thator_else() is eventually required to retrieve the underlying valuefrompymaybe’s wrappers. Furthermore,pymaybe does not short circuit anyevaluation. Althoughpymaybe has some strengths and may be useful in its ownright, it also demonstrates why a pure Python implementation of coalescing isnot nearly as powerful as support built into the language.

The idea of adding a builtinmaybe type to enable this scenario is rejected.

Just use a conditional expression

Another common way to initialize default values is to use the ternary operator.Here is an excerpt from the popularRequests package:

data=[]ifdataisNoneelsedatafiles=[]iffilesisNoneelsefilesheaders={}ifheadersisNoneelseheadersparams={}ifparamsisNoneelseparamshooks={}ifhooksisNoneelsehooks

This particular formulation has the undesirable effect of putting the operandsin an unintuitive order: the brain thinks, “usedata if possible and use[] as a fallback,” but the code puts the fallbackbefore the preferredvalue.

The author of this package could have written it like this instead:

data=dataifdataisnotNoneelse[]files=filesiffilesisnotNoneelse[]headers=headersifheadersisnotNoneelse{}params=paramsifparamsisnotNoneelse{}hooks=hooksifhooksisnotNoneelse{}

This ordering of the operands is more intuitive, but it requires 4 extracharacters (for “not “). It also highlights the repetition of identifiers:dataifdata,filesiffiles, etc.

When written using theNone coalescing operator, the sample reads:

data = data ?? []files = files ?? []headers = headers ?? {}params = params ?? {}hooks = hooks ?? {}