Usage 1: AST optimizer

Transforming an Abstract Syntax Tree (AST) is a convenientway to implement an optimizer. It’s easier to work on the AST thanworking on the bytecode, AST contains more information and is more highlevel.

Since the optimization can done ahead of time, complex but slowoptimizations can be implemented.

Example of optimizations which can be implemented with an AST optimizer:

• Copy propagation:replacex=1;y=x withx=1;y=1
• Constant folding:replace1+1 with2
• Dead code elimination

Using guards (seePEP 510), it is possible toimplement a much wider choice of optimizations. Examples:

• Simplify iterable: replacerange(3) with(0,1,2) when usedas iterable
• Loop unrolling
• Call pure builtins: replacelen("abc") with3
• Copy used builtin symbols to constants
• See alsooptimizations implemented in fatoptimizer,a static optimizer for Python 3.6.

The following issues can be implemented with an AST optimizer:

• Issue #1346238: A constant foldingoptimization pass for the AST
• Issue #2181:optimize out local variables at end of function
• Issue #2499:Fold unary + and not on constants
• Issue #4264:Patch: optimize code to use LIST_APPEND instead of calling list.append
• Issue #7682:Optimisation of if with constant expression
• Issue #10399: ASTOptimization: inlining of function calls
• Issue #11549:Build-out an AST optimizer, moving some functionality out of thepeephole optimizer
• Issue #17068:peephole optimization for constant strings
• Issue #17430:missed peephole optimization

Usage 2: Preprocessor

A preprocessor can be easily implemented with an AST transformer. Apreprocessor has various and different usages.

Some examples:

• Remove debug code like assertions and logs to make the code faster torun it for production.
• Tail-call Optimization
• Add profiling code
• Lazy evaluation:seelazy_python(bytecode transformer) andlazy macro of MacroPy (AST transformer)
• Change dictionary literals into collection.OrderedDict instances
• Declare constants: see@asconstants of codetransformer
• Domain Specific Language (DSL) like SQL queries. ThePython language itself doesn’t need to be modified. Previous attemptsto implement DSL for SQL likePEP 335 - Overloadable BooleanOperators was rejected.
• Pattern Matching of functional languages
• String Interpolation, butPEP 498was merged into Python3.6.

MacroPy has a long list ofexamples and use cases.

This PEP does not add any new code transformer. Using a code transformerwill require an external module and to register it manually.

See alsoPyXfuscator: Pythonobfuscator, deobfuscator, and user-assisted decompiler.

Usage 3: Disable all optimization

Ned Batchelder asked to add an option to disable the peephole optimizerbecause it makes code coverage more difficult to implement. See thediscussion on the python-ideas mailing list:Disable all peepholeoptimizations.

This PEP adds a new-onoopt command line option to disable thepeephole optimizer. In Python, it’s as easy as:

sys.set_code_transformers([])

It will fix theIssue #2506: Addmechanism to disable optimizations.

Usage 4: Write new bytecode optimizers in Python

Python 3.6 optimizes the code using a peephole optimizer. Bydefinition, a peephole optimizer has a narrow view of the code and socan only implement basic optimizations. The optimizer rewrites thebytecode. It is difficult to enhance it, because it written in C.

With this PEP, it becomes possible to implement a new bytecode optimizerin pure Python and experiment new optimizations.

Some optimizations are easier to implement on the AST like constantfolding, but optimizations on the bytecode are still useful. Forexample, when the AST is compiled to bytecode, useless jumps can beemitted because the compiler is naive and does not try to optimizeanything.

Interactive interpreter

It will be possible to use code transformers with the interactiveinterpreter which is popular in Python and commonly used to demonstratePython.

The code is transformed at runtime and so the interpreter can be slowerwhen expensive code transformers are used.

Build a transformed package

It will be possible to build a package of the transformed code.

A transformer can have a configuration. The configuration is not storedin the package.

All.pyc files of the package must be transformed with the same codetransformers and the same transformers configuration.

It is possible to build different.pyc files using differentoptimizer tags. Example:fat for the default configuration andfat_inline for a different configuration with function inliningenabled.

A package can contain.pyc files with different optimizer tags.

Install a package containing transformed .pyc files

It will be possible to install a package which contains transformed.pyc files.

All.pyc files with any optimizer tag contained in the package areinstalled, not only for the current optimizer tag.

Build .pyc files when installing a package

If a package does not contain any.pyc files of the currentoptimizer tag (or some.pyc files are missing), the.pyc arecreated during the installation.

Code transformers of the optimizer tag are required. Otherwise, theinstallation fails with an error.

Execute transformed code

It will be possible to execute transformed code.

Raise anImportError exception on import if the.pyc file of thecurrent optimizer tag is missing and the code transformers required totransform the code are missing.

The interesting point here is that code transformers are not needed toexecute the transformed code if all required.pyc files are alreadyavailable.

code_transformer() method

Prototype:

defcode_transformer(self,code,context):...new_code=......returnnew_code

Parameters:

• code: code object
• context: an object with anoptimize attribute (int), the optimizationlevel (0, 1 or 2). The value of theoptimize attribute comes from theoptimize parameter of thecompile() function, it is equal tosys.flags.optimize by default.

Each implementation of Python can add extra attributes tocontext. Forexample, on CPython,context will also have the following attribute:

• interactive (bool): true if in interactive mode

XXX add more flags?

XXX replace flags int with a sub-namespace, or with specific attributes?

The method must return a code object.

The code transformer is run after the compilation to bytecode

ast_transformer() method

Prototype:

defast_transformer(self,tree,context):...returntree

Parameters:

• tree: an AST tree
• context: an object with afilename attribute (str)

It must return an AST tree. It can modify the AST tree in place, orcreate a new AST tree.

The AST transformer is called after the creation of the AST by theparser and before the compilation to bytecode. New attributes may beadded tocontext in the future.

API to get/set code transformers

Add new functions to register code transformers:

• sys.set_code_transformers(transformers): set the list of codetransformers and updatesys.implementation.optim_tag
• sys.get_code_transformers(): get the list of codetransformers.

The order of code transformers matter. Running transformer A and thentransformer B can give a different output than running transformer B anthen transformer A.

Example to prepend a new code transformer:

transformers=sys.get_code_transformers()transformers.insert(0,new_cool_transformer)sys.set_code_transformers(transformers)

All AST transformers are run sequentially (ex: the second transformergets the input of the first transformer), and then all bytecodetransformers are run sequentially.

Optimizer tag

Changes:

• Addsys.implementation.optim_tag (str): optimization tag.The default optimization tag is'opt'.
• Add a new-oOPTIM_TAG command line option to setsys.implementation.optim_tag.

Changes onimportlib:

• importlib usessys.implementation.optim_tag to build the.pyc filename to importing modules, instead of always usingopt. Remove also the special case for the optimizer level0with the default optimizer tag'opt' to simplify the code.
• When loading a module, if the.pyc file is missing but the.pyis available, the.py is only used if code optimizers have thesame optimizer tag than the current tag, otherwise anImportErrorexception is raised.

Pseudo-code of ause_py() function to decide if a.py file canbe compiled to import a module:

deftransformers_tag():transformers=sys.get_code_transformers()ifnottransformers:return'noopt'return'-'.join(transformer.namefortransformerintransformers)defuse_py():return(transformers_tag()==sys.implementation.optim_tag)

The order ofsys.get_code_transformers() matter. For example, thefat transformer followed by thepythran transformer gives theoptimizer tagfat-pythran.

The behaviour of theimportlib module is unchanged with the defaultoptimizer tag ('opt').

Peephole optimizer

By default,sys.implementation.optim_tag isopt andsys.get_code_transformers() returns a list of one code transformer:the peephole optimizer (optimize the bytecode).

Use-onoopt to disable the peephole optimizer. In this case, theoptimizer tag isnoopt and no code transformer is registered.

Using the-oopt option has not effect.

AST enhancements

Enhancements to simplify the implementation of AST transformers:

• Add a new compiler flagPyCF_TRANSFORMED_AST to get thetransformed AST.PyCF_ONLY_AST returns the AST before thetransformers.

.pyc filenames

Example of.pyc filenames of theos module.

With the default optimizer tag'opt':

With the'fat' optimizer tag:

Bytecode transformer

Scary bytecode transformer replacing all strings with"Ni!Ni!Ni!":

importsysimporttypesclassBytecodeTransformer:name="knights_who_say_ni"defcode_transformer(self,code,context):consts=['Ni! Ni! Ni!'ifisinstance(const,str)elseconstforconstincode.co_consts]returntypes.CodeType(code.co_argcount,code.co_kwonlyargcount,code.co_nlocals,code.co_stacksize,code.co_flags,code.co_code,tuple(consts),code.co_names,code.co_varnames,code.co_filename,code.co_name,code.co_firstlineno,code.co_lnotab,code.co_freevars,code.co_cellvars)# replace existing code transformers with the new bytecode transformersys.set_code_transformers([BytecodeTransformer()])# execute code which will be transformed by code_transformer()exec("print('Hello World!')")

Output:

Ni! Ni! Ni!

AST transformer

Similarly to the bytecode transformer example, the AST transformer alsoreplaces all strings with"Ni!Ni!Ni!":

importastimportsysclassKnightsWhoSayNi(ast.NodeTransformer):defvisit_Str(self,node):node.s='Ni! Ni! Ni!'returnnodeclassASTTransformer:name="knights_who_say_ni"def__init__(self):self.transformer=KnightsWhoSayNi()defast_transformer(self,tree,context):self.transformer.visit(tree)returntree# replace existing code transformers with the new AST transformersys.set_code_transformers([ASTTransformer()])# execute code which will be transformed by ast_transformer()exec("print('Hello World!')")

Output:

Ni! Ni! Ni!

AST optimizers

The Issue #17515“Add sys.setasthook() to allow to use a custom AST”optimizer was a first attempt ofAPI for code transformers, but specific to AST.

In 2015, Victor Stinner wrote thefatoptimizer project, an AST optimizerspecializing functions using guards.

In 2014, Kevin Conway created thePyCCoptimizer.

In 2012, Victor Stinner wrote theastoptimizer project, an AST optimizerimplementing various optimizations. Most interesting optimizations breakthe Python semantics since no guard is used to disable optimization ifsomething changes.

In 2011, Eugene Toder proposed to rewrite some peephole optimizations ina new AST optimizer: issue #11549,Build-out an AST optimizer, movingsome functionality out of the peephole optimizer. The patch addsast.Lit (itwas proposed to rename it toast.Literal).

Python Preprocessors

• MacroPy: MacroPy is animplementation of Syntactic Macros in the Python Programming Language.MacroPy provides a mechanism for user-defined functions (macros) toperform transformations on the abstract syntax tree (AST) of a Pythonprogram at import time.
• pypreprocessor: C-stylepreprocessor directives in Python, like#define and#ifdef

Bytecode transformers

• codetransformer:Bytecode transformers for CPython inspired by theast module’sNodeTransformer.
• byteplay: Byteplay lets youconvert Python code objects into equivalent objects which are easy toplay with, and lets you convert those objects back into living Pythoncode objects. It’s useful for applying crazy transformations on Pythonfunctions, and is also useful in learning Python byte codeintricacies. Seebyteplay documentation.

See also:

• BytecodeAssembler