title:"RustPython"

description:"An open source Python-3 (CPython >= 3.5.0)Interpreter written in Rust 🐍 😱 🤘"

description:"An open source Python3 (CPython >= 3.5.0)interpreter written in Rust 🐍 😱 🤘"

url:"https://rustpython.github.io"# the base hostname & protocol for your site, e.g. http://example.com

github_username:RustPython

gitter:https://gitter.im/rustpython/Lobby

show_excerpts:true

contributor_excerpt:""# TODO: write something here, goes right under "Contributors" heading

blog-intro:Create an issue if youread something wrong. Edit posts or create new ones via PR on <a href="https://github.com/RustPython/rustpython.github.io">github.com/RustPython/rustpython.github.io</a>

blog-intro:Create an issue if yousee something wrong. Edit posts or create new ones via PR on <a href="https://github.com/RustPython/rustpython.github.io">github.com/RustPython/rustpython.github.io</a>

navigation:

-title:Blog

This post goes over the RustPython parser. You can see the source code at[RustPython/parser/](https://github.com/RustPython/RustPython/tree/master/parser).

When you write code inpython and run it, an interpreter, such as the RustPython interpreter, acts as the translator between you and your machine.

When you write code inPython and run it, an interpreter, such as the RustPython interpreter, acts as the translator between you and your machine.

The interpreter has the job of turning your human code intobyte codethat apython virtual machine can run. Bytecode is an intermediate code between source code and machine code. This makes it portable across multiple hardware and operating systems. Bytecode "works" as long as you implement a virtual machine(vm) that can run it. There is a performance penalty for this flexibility. RustPython has a vm under[RustPython/vm/](https://github.com/RustPython/RustPython/tree/master/vm). Other posts, will go into the details of that vm but now let's figure out how to turn code into bytecode.

The interpreter has the job of turning your human code intobytecodethat aPython virtual machine can run. Bytecode is an intermediate code between source code and machine code. This makes it portable across multiple hardware and operating systems. Bytecode "works" as long as you implement a virtual machine(vm) that can run it. There is a performance penalty for this flexibility. RustPython has a vm under[RustPython/vm/](https://github.com/RustPython/RustPython/tree/master/vm). Other posts will go into the details of that vm but now let's figure out how to turn code into bytecode.

Seeing is believing. To see what bytecode looks like, you can use a Python module called[`dis`](https://docs.python.org/3/library/dis.html). dis standsfordisassembler. You can write source code then see how its bytecode looks like. Here is an example:

Seeing is believing. To see what bytecode looks like, you can use a Python module called[`dis`](https://docs.python.org/3/library/dis.html)."dis" is short offor_dis_assembler. You can write source code then see how its bytecode looks like. Here is an example:


Here are the main steps that RustPython currentlydoes:

Here are the main steps that RustPython currentlygoes through:

- parse the line of source code into tokens

- determine if the tokens have a valid syntax

- create an Abstract Syntax Tree (AST)

The fastest way to understand what tokens are, is to see them. Conveniently, Python comes with a[tokenizer](https://docs.python.org/3/library/tokenize.html).Here is whathappen if I run the tokenizer on the function that I createdearlier.

The fastest way to understand what tokens are, is to see them. Conveniently, Python comes with a[tokenizer](https://docs.python.org/3/library/tokenize.html). Here is whathappens if I run the tokenizer on the function that I createdabove.


A picture IS worth a thousandword 😛 Those are the tokens. They are the basic "units"in the programming language. They are the keywords and operators that you typed. Even new lines andidentations count.

A picture IS worth a thousandwords 😛 Those are the tokens. They are the basic "units"of the programming language. They are the keywords and operators that you typed. Even new lines andidentation count.

If you want to sound fancy:

- The tokens are the basic "lexical components"

- The parsing process is called "lexical analysis"

- The thing that doesthe process is a "lexer"

- The thing that doesthis is a "lexer"

Here is the link to the RustPython lexer.

**`RustPython/parser/lexer.rs`** >>

[source code](https://github.com/RustPython/RustPython/blob/master/parser/src/lexer.rs)

If you want dive into the details of lexical analysis, check out[Python in a nutshell / Lexical structure](https://learning.oreilly.com/library/view/python-in-a/9781491913833/ch03.html#python_language-id00003)

If you wanttodive into the details of lexical analysis, check out[Python in a nutshell / Lexical structure](https://learning.oreilly.com/library/view/python-in-a/9781491913833/ch03.html#python_language-id00003)

So don't hate on the whole interpreter when you get error messages! or at least don't hate on the tokenizer!

To determine if the tokens are valid syntax, first you need a definition of what a valid syntax is. Python has a defined "grammar" or set of rules. The official reference is on[this link](https://docs.python.org/3/reference/grammar.html). There, you will find a machine readable file. You may read a book to know the rules ofpython, but words are too "fluffy", an algorithm that verifies if rules are followed needs a very strict set of rules encoded in a file.[This video](https://www.youtube.com/watch?v=KGMFvy2d5OI) explains the Python grammar and the file's notation.

As the presenter puts it, this is the spirit of the beast (python) and it is only~10KB 😭 (compare that to the size ofpython books you had to read!)

To determine if the tokens are valid syntax, first you need a definition of what a valid syntax is. Python has a defined "grammar" or set of rules. The official reference is on[this link](https://docs.python.org/3/reference/grammar.html). There, you will find a machine readable file. You may read a book to know the rules ofPython, but words are too "fluffy", an algorithm that verifies if the rules are followed needs a very strict set of rules encoded in a file.[This video](https://www.youtube.com/watch?v=KGMFvy2d5OI) explains the Python grammar and the file's notation.

As the presenter puts it, this is the spirit of the beast (Python) and it is only~10KB 😭 (compare that to the size ofthe Python books you had to read!)

So, we have the rules or grammar of a programming language in a machine encoded format... now we need to write something that verifies that those rules were followed... This sounds like something that other people could use and like something that should exist as an open source project! 🤔

As a recap, when you write a line ofpython code and "run it", here is what the RustPython interpreter does:

As a recap, when you write a line ofPython code and "run it", here is what the RustPython interpreter does:

**INPUT: your code** (in`file.py` or interactive shell)

⬇️ parse the line of source code into tokens

layout:post

title:"How to contribute to RustPythonby CPython unittest"

title:"How to contribute to RustPythonusing CPython's unit tests"

date:2020-04-05 01:45:00 +0900

categories:[guideline, featured]

permalink:guideline/2020/04/04/how-to-contribute-by-cpython-unittest.html

At the very end of 2019, we finally reached one ofthe short-term goals: CPython unittest support. Due to this enhancement,finding CPython compatibilityis easier thanbefore.

Probably thiswill be the major source ofcontribution spotsfor the new contributors this year. Here is a simple guideline.

At the very end of 2019, we finally reached one ofour short-term goals: CPython`unittest` support which makesfinding CPython compatibilityerrors easier thanever.

Thiswillprobablybe the major source ofcontributionsfor new contributors this year. Here is a simple guideline.

Let's find an incompatibility issue and fix it.

1. See`Lib/test` directory of the project. There are many`test_` prefixed files like`test_unicode.py`.

1. Look at the`Lib/test` directory of the project. There are many`test_` prefixed files like`test_unicode.py`.

2. Try to open one of them. It might look just fine at a glance - but search for`TODO: RUSTPYTHON` in the files. There are tons of skipped, marked as an expected failure or commented out tests.

1. Alternatively, try looking at the[regressiontests results]({% link pages/regression-tests-results.markdown %}) to find skipped or expected failure tests; some of them have

1. Alternatively, try looking at the[regressiontest results]({% link pages/regression-tests-results.markdown %}) to find skipped or expected failure tests; some of them have

notes for a way to resolve the issue.

3. Choose one or two interesting bugs. Remove the test blocker - skip,expectedFailure or comments.

3. Choose one or two interesting bugs. Remove the test blocker -`skip`,`expectedFailure` or comments.

4. Try to fix them.

Here is a quick tip torun single unittest file.

Here's how yourunasingle unittest file:

Because CPython unittestis notperfectlyworkingin RustPython, we aredoing thisone by one with editings.

1. Download CPython source code.

2. Check out a specific version of CPython.For now,3.8.2 is recommended. (We are testing against CPython3.8and 3.8.2 isthe most recent version for now)

3. Copy a file from CPython`Lib/test`

4. Commit the file without editing. Specifycopied CPython versionto commit message.

Because CPython unittestdoesn't work workperfectly in RustPython, we areadding test filesone by one. Here's how:

1. DownloadtheCPython source code with`git clone https://github.com/python/cpython.git`.

2. Check out a specific version of CPython.We test against CPython3.8, so the most recent release of3.8([currently 3.8.7](https://www.python.org/doc/versions/)) isrecommended.

3. Copy a file from CPython's`Lib/test`

4. Commit the file without editing it. Specifythe CPython versionyou copied from in the commit message.

5. Try to edit it until it runs without a crash or failure.

6. Commit the changes to make it run. This is the core contribution.

Because RustPython is not perfect, "try to edit it until it runs" doesn't mean to make 100%successful running. The commoneditingmethodshere:

1.At least it must be able to start to run thetest. Fix the test code or bug until it runs at least a single unit of the test. Typically, unimplemented stdlib or missing filesofunittest canmake issues. Sometimes RustPython bugsmake issues too.

2. If any testis not loadable by`SyntaxError`,that part is requiredtobe commented out.

3. If any test leads to a crash of RustPython, this codeis not possible torun. Mark the test to skip.

4. If any testis runbut fails, this is an incompatibility issue. Mark the test as an expected failure.

Because RustPython is not perfect, "try to edit it until it runs" doesn't mean to makeit run100%of the tests successfully. The common methodsto make the test file pass are:

1.It must at least be able to start to run thetests. Fix the test code or bug until it runs at least a single unit of the test. Typically, unimplemented stdlib or missing filesin`unittest` cancause issues. Sometimes RustPython bugscause issues too.

2. If any testcan't be loaded because of a`SyntaxError`,you'll havetocomment that part out.

3. If any test leads to a crash of RustPython, this codecan't berun. Mark the testwith`@unittest.skip('TODO: RUSTPYTHON')`to skip it.

4. If any testrunsbut fails, this is an incompatibility issue. Mark the test as an expected failure with`@unittest.expectedFailure`.

We prefer the reversed order ofupper methods. The later the more strict soeasy to detect any progress or regression.

When we temporarily disable parts of unittest due to RustPython caveats, we mark them tofind itout easilylater. Pleasecheck the examples below or search for`TODO: RUSTPYTHON` in`Lib/test` directory to check actual usage.

We prefer the reversed order ofabove methods. The later, the more strict, soit's easier to detect any progress or regression.

When we temporarily disable parts of`unittest` due to RustPython caveats, we mark them tomake iteasier to find (and re-enable!) themlater. Pleasesee the examples below or search for`TODO: RUSTPYTHON` in the`Lib/test` directory to check actual usage.

Comment out:

skip:

`skip`:

@unittest.skip("TODO: RUSTPYTHON")

expectedFailure:

`expectedFailure`:

Forthe generalsource of the development, please visit the[RustPython development guide](https://github.com/RustPython/RustPython/blob/master/DEVELOPMENT.md)

Fora generalintroduction to RustPython development, please visit the[RustPython development guide](https://github.com/RustPython/RustPython/blob/master/DEVELOPMENT.md)

goals:

-goal:

progress:

-[PyPy](https://www.pypy.org/) (Python)

-[Stackless](http://www.stackless.com/)

Each of these implementations offer some benefits: Jython, for example, compiles Python source code to Java byte code, then routes it to the Java Virtual Machine. Because Python code is translated to Javabyte code, it looks and feels like a true Java program at runtime and so it integrates well with Java applications.

Each of these implementations offer some benefits: Jython, for example, compiles Python2source code to Java byte code, then routes it to the Java Virtual Machine. Because Python code is translated to Javabytecode, it looks and feels like a true Java program at runtime and so it integrates well with Java applications.

IronPython is well-integrated with .NET, which means IronPython can use the .NET framework and Python libraries or vice versa.

We want to unlock the same possibilities that Jython and IronPython enable, but for the Rust programming language. In addition, thanks to Rusts' minimal runtime, we're able to compile RustPython to WebAssembly and allow users to run their Python code easily in the browser.

Check the "learn more" section for an explainer of all those jargon-y words, or read the blog for more in-depth technical discussion.

Checkoutthe "learn more" section for an explainer of all those jargon-y words, or read the blog for more in-depth technical discussion.

"start_url":".",

"display":"standalone",

"theme_color":"#F74C00",

"description":"An open source Python-3 (CPython >= 3.5.0) Interpreter written in Rust 🐍 😱 🤘"

"description":"An open source Python3 (CPython >= 3.5.0) Interpreter written in Rust 🐍 😱 🤘"

}