Repository files navigation

This repo contains a selection of projects designed to learn to write Rustprocedural macros — Rust code that generates Rust code.

Each of these projects is drawn closely from a compelling real use case. Out ofthe 5 projects here, 3 are macros that I have personally implemented inindustrial codebases for work, and the other 2 exist as libraries on crates.ioby other authors.

• Suggested prerequisites
• Projects — Introduction to each of the projects
  • Derive macro:derive(Builder)
  • Derive macro:derive(CustomDebug)
  • Function-like macro:seq!
  • Attribute macro:#[sorted]
  • Attribute macro:#[bitfield]
  • Project recommendations — What to work ondepending on your interests
• Test harness — Explanation of how testing is set up
• Workflow — Recommended way to work through the workshop
• Debugging tips

This workshop covers attribute macros, derive macros, and function-likeprocedural macros.

Be aware that the content of the workshop and the explanations in this repo willassume a working understanding of structs, enums, traits, trait impls, genericparameters, and trait bounds. You are welcome to dive into the workshop with anylevel of experience with Rust, but you may find that these basics are far easierto learn for the first time outside of the context of macros.

Here is an introduction to each of the projects. At the bottom, I giverecommendations for what order to tackle them based on your interests. Note thateach of these projects goes into more depth than what is described in theintroduction here.

This macro generates the boilerplate code involved in implementing thebuilderpattern in Rust. Builders are a mechanism for instantiating structs, especiallystructs with many fields, and especially if many of those fields are optional orthe set of fields may need to grow backward compatibly over time.

There are a few different possibilities for expressing builders in Rust. Unlessyou have a strong pre-existing preference, to keep things simple for thisproject I would recommend following the example of the standard library'sstd::process::Command builder in which the setter methods each receive andreturn&mut self to allow chained method calls.

Callers will invoke the macro as follows.

use derive_builder::Builder;#[derive(Builder)]pubstructCommand{executable:String,#[builder(each ="arg")]args:Vec<String>,current_dir:Option<String>,}fnmain(){let command =Command::builder().executable("cargo".to_owned()).arg("build".to_owned()).arg("--release".to_owned()).build().unwrap();assert_eq!(command.executable,"cargo");}

This project covers:

• traversing syntax trees;
• constructing output source code;
• processing helper attributes to customize the generated code.

Project skeleton is located under thebuilder directory.

This macro implements a derive for the standard librarystd::fmt::Debugtrait that is more customizable than the similarDebug derive macro exposed bythe standard library.

In particular, we'd like to be able to select the formatting used for individualstruct fields by providing a format string in the style expected by Rust stringformatting macros likeformat! andprintln!.

use derive_debug::CustomDebug;#[derive(CustomDebug)]pubstructField{name:String,#[debug ="0b{:08b}"]bitmask:u8,}

Here, one possible instance of the struct above might be printed by itsgeneratedDebug impl like this:

Field { name: "st0", bitmask: 0b00011100 }

This project covers:

• traversing syntax trees;
• constructing output source code;
• processing helper attributes;
• dealing with lifetime parameters and type parameters;
• inferring trait bounds on generic parameters of trait impls;
• limitations of derive's ability to emit universally correct trait bounds.

Project skeleton is located under thedebug directory.

This macro provides a syntax for stamping out sequentially indexed copies of anarbitrary chunk of code.

For example our application may require an enum with sequentially numberedvariants likeCpu0Cpu1Cpu2 ...Cpu511. But note that the sameseq!macro should work for any sort of compile-time loop; there is nothing specificto emitting enum variants. A different caller might use it for generating anexpression liketuple.0 + tuple.1 + ... + tuple.511.

use seq::seq;seq!(N in0..512{    #[derive(Copy,Clone,PartialEq,Debug)]pubenumProcessor{        #(Cpu~N,)*}});fnmain(){let cpu =Processor::Cpu8;assert_eq!(cpuasu8,8);assert_eq!(cpu,Processor::Cpu8);}

This project covers:

• parsing custom syntax;
• low-level representation of token streams;
• constructing output source code.

Project skeleton is located under theseq directory.

A macro for when your coworkers (or you yourself) cannot seem to keep enumvariants in sorted order when adding variants or refactoring. The macro willdetect unsorted variants at compile time and emit an error pointing out whichvariants are out of order.

#[sorted]#[derive(Debug)]pubenumError{BlockSignal(signal::Error),CreateCrasClient(libcras::Error),CreateEventFd(sys_util::Error),CreateSignalFd(sys_util::SignalFdError),CreateSocket(io::Error),DetectImageType(qcow::Error),DeviceJail(io_jail::Error),NetDeviceNew(virtio::NetError),SpawnVcpu(io::Error),}

This project covers:

• compile-time error reporting;
• application of visitor pattern to traverse a syntax tree;
• limitations of the currently stable macro API and some ways to work aroundthem.

Project skeleton is located under thesorted directory.

This macro provides a mechanism for defining structs in a packed binaryrepresentation with access to ranges of bits, similar to the language-levelsupport forbit fields in C.

The macro will conceptualize one of these structs as a sequence of bits 0..N.The bits are grouped into fields in the order specified by a struct written bythe caller. The#[bitfield] attribute rewrites the caller's struct into aprivate byte array representation with public getter and setter methods for eachfield.

The total number of bits N is required to be a multiple of 8 (this will bechecked at compile time).

For example, the following invocation builds a struct with a total size of 32bits or 4 bytes. It places fielda in the least significant bit of the firstbyte, fieldb in the next three least significant bits, fieldc in theremaining four most significant bits of the first byte, and fieldd spanningthe next three bytes.

use bitfield::*;#[bitfield]pubstructMyFourBytes{a:B1,b:B3,c:B4,d:B24,}

                               least significant bit of third byte                                 ┊           most significant                                 ┊             ┊                                 ┊             ┊║  first byte   ║  second byte  ║  third byte   ║  fourth byte  ║╟───────────────╫───────────────╫───────────────╫───────────────╢║▒ ▒ ▒ ▒ ▒ ▒ ▒ ▒ ▒ ▒ ▒ ▒ ▒ ▒ ▒ ▒ ▒ ▒ ▒ ▒ ▒ ▒ ▒ ▒ ▒ ▒ ▒ ▒ ▒ ▒ ▒ ▒║╟─╫─────╫───────╫───────────────────────────────────────────────╢║a║  b  ║   c   ║                       d                       ║                 ┊                                             ┊                 ┊                                             ┊               least significant bit of d         most significant

The code emitted by the#[bitfield] macro for this struct would be as follows.Note that the field getters and setters use whichever ofu8,u16,u32,u64 is the smallest while being at least as large as the number of bits inthe field.

implMyFourBytes{// Initializes all fields to 0.pubfnnew() ->Self;// Field getters and setters:pubfnget_a(&self) ->u8;pubfnset_a(&mutself,val:u8);pubfnget_b(&self) ->u8;pubfnset_b(&mutself,val:u8);pubfnget_c(&self) ->u8;pubfnset_c(&mutself,val:u8);pubfnget_d(&self) ->u32;pubfnset_d(&mutself,val:u32);}

This project covers:

• traversing syntax trees;
• processing helper attributes;
• constructing output source code;
• interacting with traits and structs other than from the standard library;
• techniques for compile-time assertions that require type information, byleveraging the trait system in interesting ways from generated code;
• tricky code.

Project skeleton is located under thebitfield directory.

If this is your first time working with procedural macros, I would recommendstarting with thederive(Builder) project. This will get you comfortable withtraversing syntax trees and constructing output source code. These are the twofundamental components of a procedural macro.

After that, it would be equally reasonable to jump to any ofderive(CustomDebug),seq!, or#[sorted].

• Go forderive(CustomDebug) if you are interested in exploring how macrosmanipulate trait bounds, which is one of the most complicated aspects ofcode generation in Rust involving generic code likeSerde. This projectprovides an approachable introduction to trait bounds and digs into many ofthe challenging aspects.

• Go forseq! if you are interested in parsing a custom input syntax yourself.The other projects will all mostly rely on parsers that have already beenwritten and distributed as a library, since their input is ordinary Rustsyntax.

• Go for#[sorted] if you are interested in generating diagnostics (customerrors) via a macro. Part of this project also covers a different way ofprocessing input syntax trees; the other projects will do most things throughif let. The visitor approach is better suited to certain types of macrosinvolving statements or expressions as we'll see here when checking thatmatch arms are sorted.

I would recommend starting on#[bitfield] only after you feel you have astrong grasp on at least two of the other projects. Note that completing thefull intended design will involve writing at least one of all three types ofprocedural macros and substantially more code than the other projects.

Testing macros thoroughly tends to be tricky. Rust and Cargo have a built-intesting framework viacargo test which can work for testing the success cases,but we also really care that our macros produce good error message when theydetect a problem at compile time; Cargo isn't able to say that failing tocompile is considered a success, and isn't able to compare that the errormessage produced by the compiler is exactly what we expect.

The project skeletons in this repository use an alternative test harness calledtrybuild.

The test harness is geared toward iterating on the implementation of aprocedural macro, observing the errors emitted by failed executions of themacro, and testing that those errors are as expected.

Every project has a test suite already written under itstestsdirectory. (But feel free to add more tests, remove tests for functionality youdon't want to implement, or modify tests as you see fit to align with yourimplementation.)

Runcargo test inside any of the 5 top-level project directories to run thetest suite for that project.

Initially every projects starts with all of its tests disabled. Open up theproject'stests/progress.rs file and enable tests one at a time as you workthrough the implementation.The test files (for exampletests/01-parse.rs)each contain a comment explaining what functionality is tested and giving sometips for how to implement it. I recommend working through tests in numberedorder, each time enabling one more test and getting it passing before moving on.

Tests come in two flavors: tests that should compile+run successfully, and teststhat should fail to compile with a specific error message.

If a test should compile and run successfully, but fails, the test runner willsurface the compiler error or runtime error output.

For tests that should fail to compile, we compare the compilation output againsta file of expected errors for that test. If those errors match, the test isconsidered to pass. If they do not match, the test runner will surface theexpected and actual output.

Expected output goes in a file with the same name as the test except with anextension of*.stderr instead of*.rs.

If there is no*.stderr file for a test that is supposed to fail to compile,the test runner will save the compiler's output into a directory calledwip adjacent to thetests directory. So the way to updatethe "expected" output is to delete the existing*.stderr file, run the testsagain so that the output is written towip, and then move the new output fromwip totests.

To look at what code a macro is expanding into, install thecargo expand Cargosubcommand and then runcargo expand in the repository root (outside of any ofthe project directories) to expand the main.rs file in that directory. You cancopy any of the test cases into this main.rs and tweak it as you iterate on themacro.

If a macro is emitting syntactically invalid code (not just code that failstype-checking) then cargo expand will not be able to show it. Instead have themacro print its generated TokenStream to stderr before returning the tokens.

eprintln!("TOKENS: {}", tokens);

Then acargo check in the repository root (if you are iterating using main.rs)orcargo test in the corresponding project directory will display this outputduring macro expansion.

Stderr is also a helpful way to see the structure of the syntax tree that getsparsed from the input of the macro.

eprintln!("INPUT: {:#?}", syntax_tree);

Note that in order for Syn's syntax tree types to provide Debug impls, you willneed to setfeatures = ["extra-traits"] on the dependency on Syn. This isbecause adding hundreds of Debug impls adds an appreciable amount of compiletime to Syn, and we really only need this enabled while doing development on amacro rather than when the finished macro is published to users.