Introduction

Don't block the event loop! Don't poll in a busy loop! Increase throughput! Use async I/O! Concurrency is not parallelism!

You've most likely heard and read claims like these many times before,and maybe, at some point you've thought you understood everythingonly to find yourself confused a moment later. Especially when you want tounderstand how it works on a fundamental level.

Me too.

So I spent a couple of hundred hours trying to fix that for myself. I wrotethis book as a result of that research, and now I invite you to join me as we tryto unveil the secrets of async programming.

This book aims to take a look at thewhy andhow of concurrent programming. First we builda good foundation of basic knowledge, before we use that knowledge to investigate how Node.js worksby building a Node-inspired runtime.

This book is developed in the open and hasits repository here.The book and theaccompanying code is MIT licensed so feel free to clone awayand play with it.

I warn you though, we need to venture from philosophical heights where we try toformally define a "task" all the way down to the deep waters where firmware andother strange creatures rule (I believe some of the more wicked creatures thereare tasked with naming low level OS syscalls and structures on Windows. However, Ihave yet to confirm this).

Everything in this book will cover the topics for the three major Operating SystemsLinux, macOS and Windows. We'll also only cover the details on how this workson 64 bit systems.

Note that this book has a companion book calledExploring Epoll, Kqueue and IOCP with Rust where we create the "minimio" library we depend on later in this book.

Who is this book for?

I originally started out wanting to explore the fundamentals and inner workingsof Rust's Futures. Reading through RFCs, motivations and discussions I realizedthat to really understand thewhy andhow of Rust's Futures, I needed a very goodunderstanding of how async code works in general, and the different strategies to handle it.

This book might be interesting if you:

• Want to take a deep dive into what concurrency is and strategies on how to deal with it

• Are curious on how to make syscalls on three different platforms, and do it on three different abstraction levels.

• Want to know more about how the OS, CPU and hardware handles concurrency.

• Want to learn the basics of Epoll, Kqueue and IOCP.

• Think using our research to write atoy node.js runtime is pretty cool.

• Want to know more about what the Node event loop really is, and why most diagrams of it on the web are pretty misleading.

• Already know some Rust but want to learn more.

So, what do you think? Is the answer yes to some of these questions? Well, then join me on this ventureas we try to get a better understanding of all these subjects.

We'll only use Rust's standard library. The reason for this is that we really want to know how thingswork, and Rust's standard library strikes the perfect balance for this task providing abstractionsbut they're thin enough to let us easily peek under the covers and see what really happens.

Following along

Even though I usemdbook, which has the nice benefit of being able to runthe code we write directly in the book, we're working with I/O and crossplatform syscalls in this book which is not a good fit for the Rust playground.

My best recommendation is to create a project on your localcomputer and follow along by copying the code over and run it locally.

You can also clone or download the example code from the git repository

Prerequisites

You don't have to be a Rust programmer to follow along. This book will have numerous chapters wherewe explore concepts, and where the code examples are small and easy to understand, but it willbe more code towards the end and you'll get the most out of it by learning the basics first. In this caseThe Rust Programming Language is the best place to start.

I do recommend that you read my book preceding thisGreen threads explained in 200 lines of Rustsince I cover quite a bit about Rust basics, stacks, threads and inline assembly there andwill not repeat everything here. However, it's definitely not a must.

You will find everything you need to set up Rust here

Disclaimer

1. We'll implement atoy version of the Node.js event loop (a bad, but working and conceptually similar event loop)

2. We'llnot primarily focus on code quality and safety.I will focus on understanding the concepts and ideas behind the code. We will have to makemany shortcuts to keep this concise and short.

3. I will however do my best to point out hazards and the shortcuts we make.I will try to point out obvious places we could do a better job or take bigshortcuts.

Even though wecover some complex topics we'll have to simplify them significantly to be ableto learn anything from them in a small(ish) book. You can probably spend the betterpart of a career becoming an expert in several of the fields we cover, so forgiveme already now for not being able to cover all of them with the precision,thoroughness and respect they deserve.

Credits

Substantial contributions will be credited here.

Contributing

I have no other interest in this than to share knowledge that can be hard tocome by and make it easier for the next curious person to understand. If you want tocontribute to make this better there are two places to go:

1. The base repo for this book for all feedback and content changes
2. The base repo for the code example we use for all improvements to the example code

Everything from spelling mistakes to correcting errors or inaccuracies are greatly appreciated. It will only make this book better for the next person reading it.

Why I wrote this and its companion books

This started as a wish to write an article about Rust's Futures 3.0. The result so far is3 books about concurrency in general and hopefully, at some point a fourth about Rust's Futures exclusively.

This process has also made me realize why I have vague memories from my childhoodof threats being made about stopping the car and letting me off if I didn't stopasking "why?" to everything.

Basically, the list below is a result of this urge to understandwhy whilereading the RFCs and discussions about Rust's async story:

• Green threads explained in 200 lines of Rust

• The Node Experiment - Exploring Async Basics with Rust (this book)

• Exploring Epoll, Kqueue and IOCP with Rust a companion book to the "Async Basics" book where we create the "minimio" library we depend on in this book.

• Exploring Rust's Futures (TBD) - a different look on thewhy andhow of Rust's futures