(This article iscross-posted frommy blog.)

TheFlatBuffers project is an extremely efficient schema-versioned serialization library. In this tutorial, you'll learn how to use it in Rust.

To learn more about why we need yet another way to encode data, go read my postWhy FlatBuffers.

FlatBuffers is a serialization format from Google. It's really fast at reading and writing your data: much quicker than JSON or XML, and often faster than Google's other format, Protocol Buffers. It's schema-versioned, which means your data has integrity (like in a relational database). FlatBuffers supports thirteen programming languages: C++, C#, C, Dart, Go, Java, JavaScript, Lobster, Lua, PHP, Python, Rust, and TypeScript.

This post will show you how to set up FlatBuffers and then use it in a demo Rust program.

(Full disclosure: I maintain the Golang, Python, and Rust FlatBuffers codebases.)

This tutorial has seven short parts:

1. Install the FlatBuffers compiler
2. Create a new Cargo project (if needed)
3. Write a FlatBuffers schema definition
4. Generate Rust accessor code from the schema
5. Install the FlatBuffers Rust runtime library
6. Write a demo Rust program to encode and decode example data
7. Learn more and get involved

If you'd like to see all of the code in one place, I've put the project up ata GitHub repository.

1. Install the FlatBuffers compiler

First things first: let's install the compiler.

The compiler is used only in development. That means you have no new system dependencies to worry about in production environments!

Installation with Homebrew on OSX

On my OSX system, I useHomebrew to manage packages. To update the Homebrew library and install FlatBuffers, run:

$brew update$brewinstallflatbuffers

(As is usual on my blog, I indicate CLI input with the prefix$.)

Personally, I like to install the latest development version from the official Git repository:

$brew update$brewinstallflatbuffers--HEAD

If successful, you will have theflatc program accessible from your shell. To verify it's installed, executeflatc:

$flatcflatc: missing input files...

Other installation methods

If you'd like to install from source, install a Windows executable, or build for Visual Studio, head over to my postInstalling FlatBuffers for more.

2. Create a new Cargo project (if needed)

(If you're adding FlatBuffers to an existing Rust project, you can skip this step.)

Create a basic Cargo configuration with the following command:

$cargo new rust_flatbuffers_example     Created binary(application)`rust_flatbuffers_example` package

There will now be a directory calledrust_flatbuffers_example. Change the current working directory to that:

$cdrust_flatbuffers_example

Now, note that the directory contains the following files:

$tree.|-- Cargo.toml`-- src`-- main.rs

Finally, check that the Cargo package is properly configured. Do this by running the example program that Cargo automatically generated:

$ cargo run --quietHello, world!

If you do not see this output, then please have a look at theofficial documentation on setting up Rust and Cargo to troubleshoot your configuration.

3. Write a FlatBuffers schema definition

All data in FlatBuffers are defined by schemas. Schemas in FlatBuffers are plain text files, and they are similar in purpose to schemas in databases like Postgres.

We'll work with data that make up user details for a website. It's a trivial example, but good for an introduction. Here's the schema:

// myschema.fbsnamespace users;table User {  name:string;  id:ulong;}root_type User;

Place the above code in a file calledmyschema.fbs, in the root of your Cargo project.

This schema definesUser, which holds one user'sname andid. The namespace for these types isusers (which will be the generated Rust package name). The topmost type in our object hierarchy is the root typeUser.

Schemas are a core part of FlatBuffers, and we're barely scratching the surface with this one. It's possible to have default values, vectors, objects-within-objects, enums, and more. If you're curious, go read thedocumentation on the schema format.

4. Generate Rust accessor code from the schema

The next step is to use theflatc compiler to generate Rust code for us. It takes a schema file as input, and outputs ready-to-use Rust code.

In the directory with themyschema.fbs file, run the following command:

$flatc--rust-o src myschema.fbs

This will generate Rust code in a new file calledmyschema_generated.rs in the pre-existingsrc directory. Here's what our project looks like afterwards:

$tree.|-- Cargo.lock|-- Cargo.toml|-- myschema.fbs`-- src    |-- main.rs`-- myschema_generated.rs1 directory, 6 files

Note that one file is generated for each schema file.

A quick browse ofsrc/myschema_generated.rs shows that there are three sections to the generated file. Here's how to think about the different function groups:

• Type definition and initializer for readingUser data

pubstructUser{...}pubfnget_root_as_user(buf:&[u8])->User{...}pubfnget_size_prefixed_root_as_user(buf:&[u8])->User{...}pubfninit_from_table(table:flatbuffers::Table)->Self{...}

• Instance methods providing read access toUser data

pubfnname(&self)->Option<&'astr>{...}pubfnid(&self)->u64{...}

• Functions used to create newUser objects

pubfncreate(_fbb:&mutflatbuffers::FlatBufferBuilder,args:&UserArgs)->flatbuffers::WIPOffset{...}

(Note that I've elided some of the lifetime annotations in the above code.)

We'll use these functions when we write the demo program.

5. Install the FlatBuffers Rust runtime library

The official FlatBuffers Rust runtime package is hosted on crates.io:Official FlatBuffers Runtime Rust Library.

To use this in your project, addflatbuffers to your dependencies manifest inCargo.toml. The file should now look similar to this:

[package]name="rust_flatbuffers_example"version="0.1.0"authors=["rw <me@rwinslow.com>"]edition="2018"[dependencies]flatbuffers="*"

I use* to fetch the latest package version. In general, you'll want to pick a specific version. You can learn more about this in the documentation for theCargo dependencies format.

6. Write a demo Rust program to encode and decode example data

Now, we'll overwrite the default Cargo "Hello World" program with code to write and read our FlatBuffers data.

(We do this for the sake of simplicity, so that I can avoid explaining the Cargo build system here. To learn more about Cargo projects, head over to theofficial documentation on Cargo project file layouts.)

Imports

To begin, we will import the generated module using themod statement. Place the following code insrc/main.rs:

// src/main.rs part 1 of 4: importsexterncrateflatbuffers;modmyschema_generated;useflatbuffers::FlatBufferBuilder;usemyschema_generated::users::{User,UserArgs,finish_user_buffer,get_root_as_user};

This usage of themod keyword instructs the Rust build system to make the items in the file calledmyschema_generated.rs accessible to our program. Theuse statement makes two generated types,User andUserArgs, accessible to our code with convenient names.

Writing

FlatBuffer objects are stored directly in byte slices. Each Flatbuffers object is constructed using the generated functions we made with theflatc compiler.

Append the following snippet to yoursrc/main.rs:

// src/main.rs part 2 of 4: make_user functionpubfnmake_user(bldr:&mutFlatBufferBuilder,dest:&mutVec<u8>,name:&str,id:u64){// Reset the `bytes` Vec to a clean state.dest.clear();// Reset the `FlatBufferBuilder` to a clean state.bldr.reset();// Create a temporary `UserArgs` object to build a `User` object.// (Note how we call `bldr.create_string` to create the UTF-8 string// ergonomically.)letargs=UserArgs{name:Some(bldr.create_string(name)),id:id,};// Call the `User::create` function with the `FlatBufferBuilder` and our// UserArgs object, to serialize the data to the FlatBuffer. The returned// value is an offset used to track the location of this serializaed data.letuser_offset=User::create(bldr,&args);// Finish the write operation by calling the generated function// `finish_user_buffer` with the `user_offset` created by `User::create`.finish_user_buffer(bldr,user_offset);// Copy the serialized FlatBuffers data to our own byte buffer.letfinished_data=bldr.finished_data();dest.extend_from_slice(finished_data);}

This function takes a FlatBuffersBuilder object and uses generated methods to write the user's name and ID.

Note that thename string is created with thebldr.create_string function. We do this because, in FlatBuffers, variable-length data like strings need to be createdoutside the object that references them. In the code above, this is still ergonomic because we can callbldr.create_string inline from theUserArgs object.

Reading

FlatBuffer objects are stored as byte slices, and we access the data inside using the generated functions (that theflatc compiler made for us inmyschema_generated.rs).

Append the following code to yoursrc/main.rs:

// src/main.rs part 3 of 4: read_user functionpubfnread_user(buf:&[u8])->(&str,u64){letu=get_root_as_user(buf);letname=u.name().unwrap();letid=u.id();(name,id)}

This function takes a byte slice as input, and initializes a FlatBuffer reader for theUser type. It then gives us access to the name and ID values in the byte slice.

The main function

Now we tie it all together. This is themain function:

// src/main.rs part 4 of 4: main functionfnmain(){letmutbldr=FlatBufferBuilder::new();letmutbytes:Vec<u8>=Vec::new();// Write the provided `name` and `id` into the `bytes` Vec using the// FlatBufferBuilder `bldr`:make_user(&mutbldr,&mutbytes,"Arthur Dent",42);// Now, `bytes` contains the serialized representation of our User object.// To read the serialized data, call our `read_user` function to decode// the `user` and `id`:let(name,id)=read_user(&bytes[..]);// Show the decoded information:println!("{} has id {}. The encoded data is {} bytes long.",name,id,bytes.len());}

This function writes, reads, then prints our data. Note thatbytes is the byte vector with encoded data. This is the serialized data you could send over the network, or save to a file.

Running it

$cargo runArthur Dent hasid42. The buffer is 48 bytes long.

To recap, what we've done here is write a short program that uses generated code to write, then read, a byte slice in which we encoded data for an example User. This User has name "Arthur Dent" and ID 42.

7. Learn more and get involved

FlatBuffers is an active open-source project, with backing from Google. It's Apache-licensed, and available for C++, C#, C, Dart, Go, Java, JavaScript, Lobster, Lua, PHP, Python, Rust, and TypeScript (with more languages on the way!).

Here are some resources to get you started:

• GitHub repository
• Issue tracker
• Official documentation

And myadditional blog posts on FlatBuffers.