Offset semantics

These offsets are propagated by the compiler from the ones stored currently inall AST nodes. The output of the public APIs (co_positions andPyCode_Addr2Location)that deal with these attributes use 0-indexed offsets (just like the AST nodes), but the underlyingimplementation is free to represent the actual data in whatever form they choose to be most efficient.The error code regarding information not available isNone for theco_positions() API,and-1 for thePyCode_Addr2Location API. The availability of the information highly dependson whether the offsets fall under the range, as well as the runtime flags for the interpreterconfiguration.

The AST nodes useint types to store these values. The current implementation, however,utilizesuint8_t types as an implementation detail to minimize storage impact. This decisionallows offsets to go from 0 to 255, while offsets bigger than these values will be treated asmissing (returning-1 on thePyCode_Addr2Location andNone API in theco_positions() API).

As specified previously, the underlying storage of the offsets should beconsidered an implementation detail, as the public APIs to obtain this valueswill return either Cint types or Pythonint objects, which allows toimplement better compression/encoding in the future if bigger ranges would needto be supported. This PEP proposes to start with this simpler version anddefer improvements to future work.

Displaying tracebacks

When displaying tracebacks, the default exception hook will be modified toquery this information from the code objects and use it to display a sequenceof carets for every displayed line in the traceback if the information isavailable. For instance:

File"test.py",line6,inlelreturn1+foo(a,b,c=x['z']['x']['y']['z']['y'],d=e)~~~~~~~~~~~~~~~~^^^^^TypeError:'NoneType'objectisnotsubscriptable

When displaying tracebacks, instruction offsets will be taken from thetraceback objects. This makes highlighting exceptions that are re-raised worknaturally without the need to store the new information in the stack. Forexample, for this code:

deffoo(x):1+1/0+2defbar(x):try:1+foo(x)+foo(x)exceptExceptionase:raiseValueError("oh no!")fromebar(bar(bar(2)))

The printed traceback would look like this:

Traceback(mostrecentcalllast):File"test.py",line6,inbar1+foo(x)+foo(x)^^^^^^File"test.py",line2,infoo1+1/0+2~^~ZeroDivisionError:divisionbyzeroTheaboveexceptionwasthedirectcauseofthefollowingexception:Traceback(mostrecentcalllast):File"test.py",line10,in<module>bar(bar(bar(2)))^^^^^^File"test.py",line8,inbarraiseValueError("oh no!")frome^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ValueError:ohno

While this code:

deffoo(x):1+1/0+2defbar(x):try:1+foo(x)+foo(x)exceptException:raisebar(bar(bar(2)))

Will be displayed as:

Traceback(mostrecentcalllast):File"test.py",line10,in<module>bar(bar(bar(2)))^^^^^^File"test.py",line6,inbar1+foo(x)+foo(x)^^^^^^File"test.py",line2,infoo1+1/0+2~^~ZeroDivisionError:divisionbyzero

Maintaining the current behavior, only a single line will be displayedin tracebacks. For instructions that span multiple lines (the end offsetand the start offset belong to different lines), the end line number mustbe inspected to know if the end offset applies to the same line as thestarting offset.

Opt-out mechanism

To offer an opt-out mechanism for those users that care about thestorage and memory overhead and to allow third party tools and otherprograms that are currently parsing tracebacks to catch up the followingmethods will be provided to deactivate this feature:

• A new environment variable:PYTHONNODEBUGRANGES.
• A new command line option for the dev mode:python-Xno_debug_ranges.

If any of these methods are used, the Python compiler willnot populatecode objects with the new information (None will be used instead) and anyunmarshalled code objects that contain the extra information will have it strippedaway and replaced withNone). Additionally, the traceback machinery will notshow the extended location information even if the information was present.This method allows users to:

• Create smallerpyc files by using one of the two methods when said filesare created.
• Don’t load the extra information frompyc files if those were created withthe extra information in the first place.
• Deactivate the extra information when displaying tracebacks (the caret charactersindicating the location of the error).

Doing this has avery small performance hit as the interpreter state needsto be fetched when code objects are created to look up the configuration.Creating code objects is not a performance sensitive operation so this shouldnot be a concern.

Use a single caret instead of a range

It has been proposed to use a single caret instead of highlighting the fullrange when reporting errors as a way to simplify the feature. We have decidedto not go this route for the following reasons:

• Deriving the location of the caret is not straightforward using the currentlayout of the AST. This is because the AST nodes only record the start and endline numbers as well as the start and end column offsets. As the AST nodes donot preserve the original tokens (by design) deriving the exact location of sometokens is not possible without extra re-parsing. For instance, currently binaryoperators have nodes for the operands but the type of the operator is storedin an enumeration so its location cannot be derived from the node (this is justan example of how this problem manifest, and not the only one).
• Deriving the ranges from AST nodes greatly simplifies the implementation and reducesa lot the maintenance cost and the possibilities of errors. This is because usingthe ranges is always possible to do generically for any AST node, while any othercustom information would need to be extracted differently from different types ofnodes. Given how error-prone getting the locations manually was when this used tobe a manual process when generating the AST, we believe that a generic solution isa very important property to pursue.
• Storing the information to highlight a single caret will be very limiting for toolssuch as coverage tools and profilers as well as for tools like IPython and IDEs thatwant to make use of this new feature. Asthis message from the author of “friendly-traceback”mentions, the reason is that without the full range (including end lines) these toolswill find very difficult to highlight correctly the relevant source code. For instance,for this code:

  something=foo(a,b,c)ifbar(a,b,c)elseother(b,c,d)

  tools (such as coverage reporters) want to be able to highlight the totality of the callthat is covered by the executed bytecode (let’s sayfoo(a,b,c)) and not just a singlecharacter. Even if is technically possible to re-parse and re-tokenize the source codeto re-construct the information, it is not possible to do this reliably and wouldresult in a much worse user experience.

• Many users have reported that a single caret is much harder to read than a full range,and this motivated using ranges to highlight syntax errors, which was very well received.Additionally, it has been noted that users with vision problems can identify the rangesmuch easily than a single caret character, which we believe is a great advantage ofusing ranges.

Have a configure flag to opt out

Having a configure flag to opt out of the overhead even when executing Pythonin non-optimized mode may sound desirable, but it may cause problems whenreading pyc files that were created with a version of the interpreter that wasnot compiled with the flag activated. This can lead to crashes that would bevery difficult to debug for regular users and will make different pyc filesincompatible between each other. As this pyc could be shipped as part oflibraries or applications without the original source, it is also not alwayspossible to force recompilation of said pyc files. For these reasons we havedecided to use the -O flag to opt-out of this behaviour.

Lazy loading of column information

One potential solution to reduce the memory usage of this feature is to notload the column information from the pyc file when code is imported. Only if anuncaught exception bubbles up or if a call to the C-API functions is made willthe column information be loaded from the pyc file. This is similar to how weonly read source lines to display them in the traceback when an exceptionbubbles up. While this would indeed lower memory usage, it also results in afar more complex implementation requiring changes to the importing machinery toselectively ignore a part of the code object. We consider this an interestingavenue to explore but ultimately we think is out of the scope for this particularPEP. It also means that column information will not be available if the user isnot using pyc files or for code objects created dynamically at runtime.

Implement compression

Although it would be possible to implement some form of compression over thepyc files and the new data in code objects, we believe that this is out of thescope of this proposal due to its larger impact (in the case of pyc files) andthe fact that we expect column offsets to not compress well due to the lack ofpatterns in them (in case of the new data in code objects).