Framework design

Framework consists of mainly the following kinds of modules:

• parser

• abstract syntax tree (AST) generalizer

• unparser

• compiler

• binder

At least some of the modules are expected to be implemented for each languagesupported by the framework.

The modules are responsible for transforming the data between the following states:

• language-specific code

• language-specific AST

• extended Python AST

• compiled binary

• Python interface for compiled binary

And thus:

• parser transforms language-specific code into language-specific AST

• AST generalizer transforms language-specific AST into extended Python AST

• unparser transforms extended Python AST into language-specific code

• compiler transforms language-specific code into compiled binary

• binder transforms compiled binary into Python interface for compiled binary

The intermediate meeting point which effectively allows code to actually be transpiled betweenlanguages, is the extended Python AST.

Features

Using Python AST as the intermediate representation, enables the AST to be directly manipulated,and certain performance-oriented transformations can be applied. Current transpiler implementationaims at:

• inlining selected calls

• decorating selected loops with compiler-extension pragmas

More optimizations will be introduced in the future.

Some (if not all) of the above optimizations may have very limited (if not no) performance impactin Python, however when C, C++ or Fortran code is generated, the performance gains can bemuch greater.

importtranspyle@transpyle.transpile('Fortran')defmy_function(a:int,b:int)->int:returna+b

importpathlibimporttranspylepath=pathlib.Path('my_script.py')code_reader=transpyle.CodeReader()code=code_reader.read_file(path)from_language=transpyle.Language.find('Python 3.6')to_language=transpyle.Language.find('Fortran 95')translator=transpyle.AutoTranslator(from_language,to_language)fortran_code=translator.translate(code,path)print(fortran_code)

Requirements

Python 3.5 or later.

Python libraries as specified inrequirements.txt.

Building and running tests additionally requires packages listed indev_requirements.txt.

Support for transpilation from/to specific language requires additional Python packagesspecified inextras_requirements.json, which can be installed using the pip extrasinstallation formulapip3installtranspyle[extras] where thoseextrascan be one or more of the following:

• All supported languages:all

• C:c

• C++:cpp

• Cython:cython

• Fortran:fortran

• OpenCL:opencl

Therefore to enable support for all languages, executepip3installtranspyle[all].Alternatively, to enable support for C++ and Fortran only, executepip3installtranspyle[cpp,fortran].

Additionally, full support for some languages requires the following software to be installed:

• C++:

  • a modern C++ compiler – fully tested with GNU’sg++ versions 7 and 8and partially tested with LLVM’sclang++ version 7

  • SWIG (Simplified Wrapper and Interface Generator) – tested with version 3

• Fortran:

  • a modern Fortran compiler – fully tested with GNU’sgfortran versions 7 and 8and partially tested with PGI’spgfortran version 2018

The core functionality of transpyle is platform-independent. However, as support of some languagesdepends on presence of additional software, some functionality might be limited/unavailableon selected platforms.

Transpyle is fully tested on Linux, and partially tested on OS X and Windows.

Installation

pip3installtranspyle[all]

dockerpull"mbdevpl/transpyle"dockerrun-htransmachine-it"mbdevpl/transpyle"

python3-mjupyternotebook--ip="$(hostname-i)"--port=8080

Related publications

Below is the list of papers describing various aspects of transpyle and/or principles behind it.Further research is ongoing, so the list might be extended in the future.

• M. Bysiek, A. Drozd and S. Matsuoka,Migrating Legacy Fortran to Python While Retaining Fortran-Level PerformanceThrough Transpilation and Type Hints,PyHPC 2016: 6th Workshop on Python for High-Performance and Scientific Computing @ SC16,Salt Lake City, Utah, United States of America, 2016, pp. 9-18

  Abstract:

  We propose a method of accelerating Python code by just-in-time compilation leveraging typehints mechanism introduced in Python 3.5. In our approach performance-critical kernels areexpected to be written as if Python was a strictly typed language, however without the needto extend Python syntax. This approach can be applied to any Python application, however wefocus on a special case when legacy Fortran applications are automatically translated intoPython for easier maintenance. We developed a framework implementing two-way transpilationand achieved performance equivalent to that of Python manually translated to Fortran, andbetter than using other currently available JIT alternatives (up to 5x times faster thanNumba in some experiments).

  https://doi.org/10.1109/PyHPC.2016.006

• M. Bysiek, M. Wahib, A. Drozd and S. Matsuoka,Towards Portable High Performance in Python: Transpilation, High-Level IR,Code Transformations and Compiler Directives (Unreferred Workshop Manuscript),2018-HPC-165: 研究報告ハイパフォーマンスコンピューティング,Kumamoto, Kumamoto, Japan, 2018, pp. 1-7

  Abstract:

  We present a method for accelerating the execution of Python programs. We rely onjust-in-time automatic code translation and compilation with Python itself being used as ahigh-level intermediate representation. We also employ performance-oriented codetransformations and compiler directives to achieve high performance portability whileenabling end users to keep their codebase in pure Python. To evaluate our method, weimplement an open-source transpilation framework with an easy-to-use interface thatachieves performance better than state-of-the-art methods for accelerating Python.

  http://id.nii.ac.jp/1001/00190591/