Overloading Literals

While this can be done as an AST transformation, we will often need to executethe constructor for the literal multiple times. Also, we need to be sure thatany additional names required to run our code are provided when we run. Withcodetransformer, we can pre compute our new literals and emit code that isas fast as loading our unmodified literals without requiring any additionalnames be available implicitly.

In the following block we demonstrate overloading dictionary syntax to result incollections.OrderedDict objects.OrderedDict is like adict;however, the order of the keys is preserved.

>>>fromcodetransformer.transformers.literalsimportordereddict_literals>>>@ordereddict_literals...deff():...return{'a':1,'b':2,'c':3}>>>f()OrderedDict([('a',1),('b',2),('c',3)])

This also supports dictionary comprehensions:

>>>@ordereddict_literals...deff():...return{k:vfork,vinzip('abc',(1,2,3))}>>>f()OrderedDict([('a',1),('b',2),('c',3)])

The next block overridesfloat literals withdecimal.Decimalobjects. These objects support arbitrary precision arithmetic.

>>>fromcodetransformer.transformers.literalsimportdecimal_literals>>>@decimal_literals...deff():...return1.5>>>f()Decimal('1.5')

Pattern Matched Exceptions

Pattern matched exceptions are a good example of aCodeTransformer thatwould be very complicated to implement at the AST level. This transformationextends thetry/except syntax to accept instances ofBaseException aswell subclasses ofBaseException. When excepting an instance, theargsof the exception will be compared for equality to determine which exceptionhandler should be invoked. For example:

>>>@pattern_matched_exceptions()...deffoo():...try:...raiseValueError('bar')...exceptValueError('buzz'):...return'buzz'...exceptValueError('bar'):...return'bar'>>>foo()'bar'

This function raises an instance ofValueError and attempts to catch it. Thefirst check looks for instances ofValueError that were constructed with anargument of'buzz'. Because our custom exception is raised with'bar',these are not equal and we do not enter this handler. The next handler looks forValueError('bar') which does match the exception we raised. We then enterthis block and normal python rules take over.

We may also pass their own exception matching function:

>>>defmatch_greater(match_expr,exc_type,exc_value,exc_traceback):...returnmath_expr>exc_value.args[0]>>>@pattern_matched_exceptions(match_greater)...deffoo():...try:...raiseValueError(5)...except4:...return4...except5:...return5...except6:...return6>>>foo()6

This matches on when the match expression is greater in value than the firstargument of any exception type that is raised. This particular behavior would bevery hard to mimic through AST level transformations.

Instructions

TheInstruction object represents an atomic operation that can be performedby the CPython virtual machine. These are things likeLOAD_NAME which loadsa name onto the stack, orROT_TWO which rotates the top two stack elements.

Some instructions accept an argument, for exampleLOAD_NAME, which modifiesthe behavior of the instruction. This is much like a function call where somefunctions accept arguments. Because the bytecode is always packed as raw bytes,the argument must be some integer (CPython stores all arguments two in bytes).This means that things that need a more rich argument system (likeLOAD_NAMEwhich needs the actual name to look up) must carry around the actual argumentsin some table and use the integer as an offset into this array. One of the keyabstractions of theInstruction object is that the argument is always somepython object that represents the actual argument. Any lookup table managementis handled for the user. This is helpful because some arguments share this tableso we don’t want to add extra entries or forget to add them at all.

Another annoyance is that the instructions that handle control flow use theirargument to say what bytecode offset to jump to. Some jumps use the absoluteindex, others use a relative index. This also makes it hard if you want to addor remove instructions because all of the offsets must be recomputed. Incodetransformer, the jump instructions all accept anotherInstruction asthe argument so that the assembler can manage this for the user. We also providean easy way for new instructions to “steal” jumps that targeted anotherinstruction so that can manage altering the bytecode around jump targets.

Code

Code objects are a nice abstraction over python’stypes.CodeType. Quoting theCodeType constructor docstring:

code(argcount, kwonlyargcount, nlocals, stacksize, flags, codestring,      constants, names, varnames, filename, name, firstlineno,      lnotab[, freevars[, cellvars]])Create a code object.  Not for the faint of heart.

Thecodetransformer abstraction is designed to make it easy to dynamicallyconstruct and inspect these objects. This allows us to easy set things like theargument names, and manipulate the line number mappings.

TheCode object provides methods for converting to and from Python’s coderepresentation:

1. from_pycode

2. to_pycode.

This allows us to take an existing function, parse the meaning from it, modifyit, and then assemble this back into a new python code object.