Writing A Compiler In Go
This is the sequel toWriting An Interpreter In Go.
We're picking up right where we left off and writea compiler and a virtual machine for Monkey.
Runnable and tested code front and center, built from the ground up, step by step — just like before.
But this time, we're going to define bytecode, compile Monkey and execute it in our very own virtual machine.
It's the next step in Monkey's evolution.
Buy the eBook
- The book inePub (iBook), Mobi (Kindle), PDF and HTML
- Around 260 pages of content with code and illustrations
- Code and tests included
Buy the eBook bundle
- Two books for a reduced bundle price!
- This book andWriting An Interpreter In Go
- Both books inePub (iBook), Mobi (Kindle), PDF and HTML
- Around 480 pages of content!
- Code and tests included
Buy the paperback
- The book in a beautiful 360 pages paperback version
- Code and tests downloadable
- Amazon Support: it's distributed through Amazon so you get all the money-back-guarantees and shipping Amazon offers
Included in theeBook packages:
- All the code presented in the book: runnable, organized by chapters, MIT licensed, including the test suite.
- Free updates: Once you buy them, you will receive free updates for the lifetime of books.
- Money-Back-Guarantee: I want you to enjoy this book. If you, for any reason, are not happy with it just send me an email. You'll keep what you bought and your money back.
Current version:1.2. Last updated:7. May 2020.See changelog.
It's the sequel to … a programming language
Writing A Compiler In Go is thesequel toWriting An Interpreter InGo. It starts right where the first one stopped, with a fully-working,fully-tested Monkey interpreter in hand, connecting both books seamlessly,ready to build a compiler and a virtual machine for Monkey.
In this book, we use the codebase (included in the book!) from the first partand extend it. We take the lexer, the parser, the AST, the REPL and the objectsystem and use them to build a new, faster implementation of Monkey, right nextto the tree-walking evaluator we built in the first book.
The approach is unchanged, too. Working, tested code is the focus, we buildeverything from scratch, do baby steps, write tests firsts, use no3rd-party-libraries and see and understand how all the pieces fit together.
It's a continuation in prose and in code.
Do you need to read the first part before this one? If you're okay withtreating the code from the first book as a black box, then no. But that's notwhat these books are about; they're aboutopening up black boxes, lookinginside and shining a light. You'll have the best understanding of where we'regoing in this book, if you know where we started.
Learn how to write a compiler and a virtual machine
Our main goal in in this book is to evolve Monkey. We change itsarchitecture and turn it intoa bytecode compiler and virtualmachine.
We'll take the lexer, the parser, the AST and the object system wewrote in the first book and use them to build our own Monkey compilerand virtual machine …from scratch! We'll build themside-by-side so that we'll always have a running system we cansteadily evolve.
What we end up with is not only much closer to the programminglanguages we use every day, giving us a better understanding of howthey work, but also3x faster. And that's without explicitlyaiming for performance.
Here's what we'll do:
- We define our ownbytecode instructions, specifying theiroperands and their encoding. Along the way, we also build amini-disassembler for them.
- We write acompiler that takes in a Monkey AST and turns it intobytecode by emitting instructions
- At the same time we build astack-based virtual machine that executes thebytecode in its main loop
We'll learn a lot about computers, how they work, what machine codeand opcodes are, whatthe stack is and how to work with stackpointers and frame pointers, what it means to define acallingconvention, and much more.
We also
- build asymbol table and aconstant pool
- dostack arithmetic
- generatejump instructions
- buildframes into our VM to executefunctions with local bindings and arguments!
- addbuilt-in functions to the VM
- getreal closures working in the virtual machine and learn why closure-compilation is so tricky
The Monkey Programming Language
The implementation of Monkey we build inthis book is markedlydifferent from the tree-walking interpreter we built inWriting AnInterpreter In Go, but Monkeystays the same.
At the end, Monkey will still look and work like this:
// Integers & arithmetic expressionsletversion=1+(50/2)-(8*3);// Stringsletname="The Monkey programming language";// BooleansletisMonkeyFastNow=true;// Arrays & Hashesletpeople=[{"name":"Anna","age":24},{"name":"Bob","age":99}];// FunctionsletgetName=fn(person){person["name"];};getName(people[0]);// => "Anna"getName(people[1]);// => "Bob"And it will still supportrecursive functions,conditionals,implicit and explicit returning of values:
letfibonacci=fn(x){if(x==0){0}else{if(x==1){return1;}else{fibonacci(x-1)+fibonacci(x-2);}}};The crown-jewel we'll build into our new Monkey implementation areclosures:
// `newAdder` returns a closure that makes use of the free variables `a` and `b`:letnewAdder=fn(a,b){fn(c){a+b+c};};// This constructs a new `adder` function:letadder=newAdder(1,2);adder(8);// => 11Yes, we'llcompile all of that tobytecode and execute it in astack-based virtual machine. That not only works, but is also tremendousfun to build.
What readers are saying about this book:
"This is a very polished pair of books that together give a steady path to follow for learning some of the real techniques that are used to implement programming languages. They're both well above average for their depth, technical clarity, and accessibility. I've been recommending them to everyone I work with who wants to get involved in compilers."
Chris Seaton (@ChrisGSeaton)
Researcher atShopify, PhD in Ruby, Founder ofTruffle Ruby
"Thorsten has a wonderful gift — as you read his books, you'll feel as thoughyou've discovered compilers, all on your own! As a self-taught engineerwithout a CS degree, I found myself led on a journey of learning, driven by thenuggets of 'why' that Thorsten shares alongside code.
I've recommendedthese books to many, and I won't hesitate to continue. Compilers, interpretersand programming languages may seem esoteric, but if you look closely, they'reeverywhere. The Monkey language is my go-to project whenever I learn a newprogramming language. There's no better way to take a new language through itspaces!"
Lauren Tan (@potetotes)
Software Engineer
"Longing for implementing a compiler, but lacking the guidance? Then this book is for you. Thorsten offers invaluable mentorship for tackling the multiple challenges that arise when writing a single-pass compiler and a virtual machine to execute the resulting code, in a pragmatic and test-driven approach. After going over this book, you'll certainly feel inspired to go further, such as extending the compiler with code transformations of your own."
Ricardo Nabinger Sanchez (@rnsanchez)
Performance Engineer atTaghos Tecnologia
"Making is understanding. The book carefully teaches us how VM-backed interpreters work, while building our own compiler and virtual machine. Written in a TDD style, a step-by-step manner and with a humane tone. I've enjoyed the wonderful journey!"
Yoji Shidara (@darashi)
Programmer, English/Japanese Translator ofWriting An Interpreter In Go
"I think that it's one of the best self-published books on the market I've ever read, not only in Go area, but whole technical market."
Bart Smykla (@bartsmykla)
Open Source engineer at VMWare
"Finally, a compiler book that you don't have to fight each and every page to get through! Other books burden the reader with so many details and subtleties that you'll never make progress. But if you want to make an actual compiler, read this. I'll definitely be recommending this to my students."
Austin Henley (@austinzhenley)
Assistant Professor, Department of Electrical Engineering & Computer Science, University of Tennessee
"If you are scared ofDragons but want to understand how compilers work, this book is for you. Not only does it make a daunting subject approachable but it also make it fun and enjoyable."
Franck JEANNIN (@frje)
Founder DataNuage
"The Monkey books are everything I've ever wanted in programming books, but could never find until now. They are a real labor of love and it shows on every page. The books are fun to read and showcase the Go programming language, but what makes them extraordinary is the fact that they're completely focused on giving the reader an accessible, conceptual understanding of the implementation of interpreters and compilers. It's hard to overstate just how rare yet extremely valuable that focus on understanding is in an educational book."
Brandon Ramos
Product Owner/Developer
"Just awesome books. If we had books like this on other subjects (search engines, DB systems), I think developers would have less fear of contributing to large open source projects. - Thank you@thorstenball. This is already paying off in my DB internals course."
Evance Soumaoro (@evanxg852000)
Software Engineer | System Programming Enthusiast
What readers said about the first book:
"Compilers was the most surprisingly useful university course I ever took. Learning to write a parser and runtime for a toy language helps take away a lot of "magic" in various parts of computer science. I recommend any engineer who isn't familiar with lexers, parsers, and evaluators to read Thorsten's book."
Mitchell Hashimoto (@mitchellh)
Founder ofHashiCorp
"Thorsten took a topic that is usually very dry and CS heavy and made it accessible and easy to understand. After reading this book I felt confident enough to writePlush, the templating language I’ve always wanted in Go! If you have yet to read Thorsten's book, I can't recommend it enough. Please go and buy it!"
Mark Bates (@markbates)
Creator ofgobuffalo.io
"Great book. I loved it because everything is built by hand, so you get to think about all the details, and it does so in a gradual way, which is didactic. The implementation itself is also nice and simple 🙌"
Xavier Noria (@fxn)
Everlasting student, Rails Core Team, Ruby Hero, Freelance, Live lover
"This book demystifies and makes the topic of interpreters approachable and fun. Don't be surprised if you become a better Go programmer after working your way through it."
Johnny Boursiquot (@jboursiquot)
Go Programmer,@BaltimoreGolang Organizer,@GolangBridge Core Member
"Thorsten's writing style is fun and easy to understand with detailed explanations and even better code. Even if you've written an interpreter before, this book is a great refresher. I picked it up asa project to learn Rust, translating the example Go code into Rust as I read through. Lexers, parsers, and interpreters are such a fundamental part of CS, these skills are valuable to more than just programmers implementing programming languages. As a project for picking up a new language, this book is perfect because it only requires the standard library and has extensive test driven development, which means you get quick feedback as you go along. I highly recommend it for programmers wanting to learn more about lexers, parsers, and interpreters or Go programmers picking up a new language looking for a project to learn through."
Paul Dix (@pauldix)
CTO ofInfluxDB
Buy the eBook
- The book inePub (iBook), Mobi (Kindle), PDF and HTML
- Around 260 pages of content with code and illustrations
- Code and tests included
Buy the eBook bundle
- Two books for a reduced bundle price!
- This book andWriting An Interpreter In Go
- Both books inePub (iBook), Mobi (Kindle), PDF and HTML
- Around 480 pages of content!
- Code and tests included
Buy the paperback
- The book in a beautiful 360 pages paperback version
- Code and tests downloadable
- Amazon Support: it's distributed through Amazon so you get all the money-back-guarantees and shipping Amazon offers
Included in theeBook packages:
- All the code presented in the book: runnable, organized by chapters, MIT licensed, including the test suite.
- Free updates: Once you buy them, you will receive free updates for the lifetime of books.
- Money-Back-Guarantee: I want you to enjoy this book. If you, for any reason, are not happy with it just send me an email. You'll keep what you bought and your money back.
Current version:1.2. Last updated:7. May 2020.See changelog.
FAQ
Do I need to read the first book,Writing An Interpreter In Go,before this one?
The short answer:Yes.
The slightly longer version is this:Writing A Compiler In Go is thesequel toWriting An Interpreter In Go and starts right where the otherone stopped. It also uses the code from the first book — the lexer, theparser, the AST, the object system — and extends it.
If you're okay with treating the code as a black box, then no, you don't needto start with the first book. But that's not what I recommend or what thesebooks are about; they're aboutopening up black boxes, looking inside andshining a light. You'll have the best understanding of where we're going inthis book, if you know where we started.
Can I buy a bundle of the eBook and the paperback?
I'm sorry to say it, but no, I cannot bundle eBooks with paperbacks. It's notthat I don't want to (I do!) but I can't. The eBooks are sold and distributedthrough Gumroad, where I have a lot of influence on the process, but thepaperback editions are being printed, sold and shipped by Amazon and I don'thave many options there. I can't tell Amazon to bundle digital files with thepaperback. Sorry!
I found a typo/mistake/error in the book. What now?
Take a look at thechangelog to see whether I've already fixedit. If I haven't or you're not sure I have, please send me an email tome @ thorstenball.com — I really appreciate it!
The books are too expensive for me. Can you help me out?
Sure, just send me a picture! I'm always fascinated by new places and loveseeing where people live, so here's my proposal.
You go outside, take a picture of where you live and send it to me tome @ thorstenball.com. Tell me what you feel comfortable payingfor the book(s) and we'll make that happen.
Want to stay up to date?
Sign up to my personal newsletter and I'll email you when I have updates on the book or produced something else. I won't bother you, you can unsubscribe at any time and your email address stays with me.
About the author
Hello there! My name is Thorsten Ball. I'm a writer and programmerliving in Aschaffenburg, Germany. In my professional life as asoftware developer, I build web applications and platforms. I havedeployed Ruby, JavaScript, Go and C code to production systems.
In my spare time, though, I like to do what I callrecreationalprogramming, where I dig deep into various topics and theircodebases, taking them apart, recreating them in my own code, tryingto peel away layers of what seems like magic, hoping to come out theother end with a better understanding of what it is that I do when Iprogram.
For the last few years, the two topics that kept my attention aresystems programming and programming languages: interpreters,compilers, virtual machines, JIT compilers, assembly language – Ican't get enough of it.
Writing an interpreter from scratch and writinga book aboutthat has been one of the most wonderful and satisfying thingsI ever did as a programmer. It has been so much fun, in fact, that Icouldn't stop doing it, which is why you're looking at the next book.
You can follow@thorstenball on Twitter to know what I'm upto. If you want to know more about me, visitmy blog andwebsite or take a look at myGitHub profile.
Any questions?
If you have any questions, send me an email. I promise, you'll make my day. Ilove getting email from you:me @ thorstenball.com
