N.B. Ruff now supports over 200 lint rules and is used in major open-source projects likeFastAPI,Bokeh,Zulip, andPydantic. It’s also about ~50% faster than the benchmarks advertised in this blog post.Try it today!

Over the past few years, there’s been a mindset shift in JavaScript ecosystem, best summarized as: “our tools should be extremely fast”.

As projects grew in complexity, and builds slowed down, we saw the emergence of new bundlers, and even new runtimes, with the common theme being “rewrite these tools in more performant languages”.

Our current build tools for the web are 10-100x slower than they could be. The main goal of the esbuild bundler project is to bring about a new era of build tool performance. —esbuild

I sometimes call this (perhaps unfairly) the ‘Rustification’ of the JavaScript toolchain:swc is written in Rust,esbuild is written in Go,Bun is written in Zig,Rome is being written in Rust. The core developer of swc is even working on a new TypeScript type-checker, again written in Rust.

The Python ecosystem could benefit from a similar mindset shift. Python tooling could be much, much faster.

As a proof-of-concept, I’m releasingRuff, an extremely fast Python linter, written in Rust.

Ruff is ~150x faster than Flake8 on macOS (~25x faster if youhack Flake8 to enable multiprocessing), ~75x faster thanpycodestyle, ~50x faster thanpyflakes andpylint, and so on.

Even a conservative 25x is the difference between ~real-time feedback (~300-500ms) and sitting around for 12+ seconds. With a 150x speed-up, it’s ~300-500ms vs. 75 seconds. If you edit a single file in CPython and re-runruff, it’s 60ms total, increasing the speed-up by another order of magnitude.

You can try Ruff today:pip install ruff. If you have a sufficiently large Python codebase, I think you’ll be surprised. I was!

(Ruff isnot a drop-in replacement for those other tools. It’s not even production-ready — it’s “rough”. See “What ruff is missing” below.)

I think Ruff is a useful anchor for framing the conversation around developer tooling in the Python ecosystem, where it could be headed, and the tradeoffs we might face.

How Ruff works

Ruff is written in Rust.

It leveragesRustPython’s AST parser, and from there, implements its own AST traversal, visitor abstraction, and lint-rule logic. (Much of that logic traces back to existing tools likepycodestyle.) It supports Python 3.10, including the newpattern matching syntax.

Despite being written in Rust, Ruff ispip install-able (facilitated bymaturin), just like your other command-line tools. As a user, you shouldn’t even notice that it’snot written in Python.

In building Ruff, I tried to incorporate the features I missed most when moving between JavaScript and Python:

• Ruff supportsESLint-like caching. So, if you’re working in CPython, edit a single file, and re-run Ruff, we’ll only lint thatsingle file, linting the “entire” CPython codebase in ~60ms.
• Ruff supportsTypeScript-like file watching. So, runningruff --watch path/to/src will drop you into a persistent linter that re-runs whenever your source code changes.
• Ruff supportspyproject.toml-based configuration, which is increasingly the standard in the Python ecosystem.

The Ruff hypotheses

Ruff is based on two core hypotheses:

1. Python tooling could be rewritten in more performant languages.
2. An integrated toolchain can tap into efficiencies that aren’t available to a disparate set of tools.

I’ve talked about that first hypothesis a lot. I want to spend some time on the second (which is itself inspired byRome’s philosophy).

Let’s take Flake8 as an example. Flake8 is really a wrapper around other tools, likepyflakes andpycodestyle. When you run Flake8, bothpyflakes andpycodestyle are reading every file from disk, tokenizing the code, and traversing the tree (I might be wrong on some of the details, but you get the idea). If you then useautoflake to automatically fix some of your lint violations, you’re runningpycodestyleyet again. How many times, in your pre-commit hooks, do you read your source code from disk, parse it, and traverse the parse tree?

An integrated toolchain could read every fileexactly once, generate the ASTexactly once, and leverage that representation throughout.

This is one of Ruff’s goals: generate all violations in a single pass, and even autofix the low-hanging fruit without a noticeable performance penalty.

What Ruff is missing

Mostly: lint rules.

Though it does a similar amount of work (AST parsing, AST traversal, scope + binding tracking), and thereby should be a fair comparison for benchmarking, I’ve only implemented a small subset of Flake8’s supported checks. Eventhose checks are missing a few edge-cases.

This is, of course, a big limitation — though I’ve come to believe that with the advent of autoformatting (see:Black,Prettier), stylistic lint rules are becoming less and less relevant, and so I plan to keep style checks to a minimum anyway.

Ruff will crash in some specific cases (#39), though I’ve run it successfully over large, diverse Python codebases.

Unlike Flake8 and others, Ruff doesn’t support plugins. It’s not extensible. It’s also missing some aspirational features, likeautofix andincremental computation in watch mode.

Tradeoffs

While Ruff is fast, there are of course benefits to writing Python developer tools in Python. For example:

1. Developers in the Python ecosystem can contribute to those tools without learning a new language.
2. Debugging is straightforward. If Flake8 or Black fail, you’ll get a Python trace, and might find yourself reading Python source code. This won’t be the case with Ruff. (I already feel this pain withMypy, which is compiled viamypyc — so crashes don’t yield Python traces.)
3. Developers can write plugins and extensions in Python.

The first two are hard to argue with — they’re tradeoffs!

The third has some wiggle room. Bun, for example, is written in Zig, but will support writing plugins in TypeScript. Similarly, given the state of Rust-Python interoperability, it should be possible to support writing Ruff plugins in Pythonor Rust.

Implications

Ruff is just a linter. But any tool that’s 100, 50, or even 10x faster is worthy of consideration — or, at least, the curiosity to ask “Why?”

The question I keep asking myself is: could we take the Ruff model and apply it to other tooling? You could probably give autoformatters (likeBlack andisort) the same treatment. But what about type checkers? I’m not sure!Mypy is already compiled withmypyc, and so is much faster than pure Python; andPyright is written in Node. It’s something I’d like to put to the test.

Ultimately, my goal with Ruff is to get the Python ecosystem to question the status quo. How long should it take to lint, say, a million lines of code? In my opinion: it should be instant.

And if your developer tools wereinstant, what would that unlock?

Published on August 30, 2022.