Updating Rust crates used by Chromium

This document describes how Chromium updates crates.io Rust crates that Chromium depends on.

Staffing

We have aweekly rotation of Google engineers responsible for creating and landing CLs that update Rust crates.

Google engineers can join the rotation by emailingchrome-safe-coding@google.com.

Initial setup

The “Rust: periodic update of 3rd-party crates” rotation requires access to an up-to-date Chromium repo. One way to start a shift is to rungit fetch,git checkout origin/main, andgclient sync (but other workflows should also work - e.g. ones based ongit-new-workdir).

Checking the state of the world

Before creating a CL stack, check for open CLs with thecratesio-autoupdate tag. Such CLs tend to conflict, so coordinate with owners of any open CLs.

You may also check a doc with notes from previous rotations, where we may note known issues and their workarounds. See (Google-internal, sorry):https://docs.google.com/document/d/1S7gsrJFsgoU5CH0K7-X_gL55zIIgd6UsFpCGrJqjdAg/edit?usp=sharing

Automated step:create_update_cl.py

The first actual step of the rotation is runningcreate_update_cl.py. You must invoke it from within thesrc/ directory of a Chromium repository checkout, and it depends ondepot_tools andgit being present in thePATH.

$ cd~/chromium/src# or wherever you have your checkout$ tools/crates/create_update_cl.py auto

Inauto mode, it runsgnrt update to discover crate updates and then for each update creates a new local git branch (and a Gerrit CL unless invoked with--no-upload). Each branch contains an update created bygnrt update <old crate id>,gnrt vendor, andgnrt gen. Depending on how many crates are updated, the script may need 10-15 minutes to run.

The script should Just Work in most cases, but sometimes it may fail when dealing with a specific crate update. SeeRecovering from script failures below for what to do when that happens.

Before the auto-generated CLs can be landed, you will need to get an LGTM from//third_party/rust/OWNERS. A review checklist can be found at//third_party/rust/OWNERS-review-checklist.md.

New transitive dependencies

Notes from//third_party/rust/OWNERS-review-checklist.md apply:

• The dependency will need to go through security review.
• An FYI email should be sent tochrome-atls-discuss@google.com in order to record the addition.

Optional: Adding the transitive dependency in its own CL

If the new crate is non-trivial, it's possible to split the additional crate into its own CL, however then it will default to global visibility and allowing non-test use.

• gnrt add andgnrt vendor can add the dependency to a fresh checkout.
• Mark the crate as being for third-party code only by settingallow_first_party_usage tofalse for the crate inthird_party/rust/chromium_crates_io/gnrt_config.toml.
• If the crates making use of the transitive dependency are only allowed in tests, then setgroup = 'test' for the crate inthird_party/rust/chromium_crates_io/gnrt_config.toml. This reduces the level of security review required for the library.
• gnrt gen will then generate the GN rules.
• Rebase the roll CL on top of the changes to make sure the choices made above are correct.gn gen will fail in CQ if the crate was placed in the'test' group but needs to be visible outside of tests.

Potential additional steps

• Thecreate_update_cl.py script may stop early if it detects thatgnrt vendor orgnrt gen have reported any warnings or errors (e.g. a “License file not found for crate foo” warning). In this case, manual intervention is needed to finish the update CL. It's probably best to finish and land the CLs created so far before trying to restart the script in order to create the remaining CLs.

Landing the CL

Other than the above, the CL can go through the normal, plain-vanilla, manual review and landing process.

1. git cl upload
2. Get a review from one of//third_party/rust/OWNERS
3. Land the CL using CQ+2

Checking for new major versions

Note thatcreate_update_cl.py auto will by default only handle minor version updates (e.g. 123.1 => 123.2, or 0.123.1 => 0.123.2). Major version changes (e.g. 1.0 => 2.0, which may include breaking API changes and other breaking changes) need to be handled separately - this section describes what to do.

Detecting available major version updates

As part of the rotation, one should attempt to check for new major versions ofdirect Chromium dependencies (i.e. dependencies directly listed inthird_party/rust/chromium_crates_io/Cargo.toml). To discover directand transitive dependencies with a new major version, you can use the command below (running it in the final update CL branch - after all the minor version updates):

$ tools/crates/run_gnrt.py update----verbose--dry-run...Unchanged serde_json_lenient v0.1.8(latest: v0.2.0)Unchanged syn v1.0.109(latest: v2.0.53)...

Major version update: Workflow A: Single update CL

If updating to a new major version doesn‘t require lots of Chromium changes, then it may be possible to land the update in a single CL. This is typically possible when the APIs affected by the major version’s breaking change either weren't used by Chromium, or were used only in a handful of places.

Warning: Sometimes a new major version may be API compatible, but may introduce breaking changes in thebehavior of the existing APIs.

To update:

1. tools/crates/create_update_cl.py auto -- some_crate_name --breaking
2. Follow the manual steps from the minor version update rotation for review, landing, etc.

Major version update: Workflow B: Incremental transition

When lots of first-party code depends on the old major version, then the transition to the new major version may need to be done incrementally. In this case the transition can be split into the following steps:

1. Open a new bug to track the transition
   • TODO: Figure out how to tag/format the bug to make it easy to discover in future rotations
2. Land the new major version, so that the old and the new versions coexist. To do this follow the process for importing a new crate as described indocs/rust.md (i.e. editCargo.toml to add the new version, rungnrt vendor, and so forth).
3. Incrementally transition first-party code to the new major version
4. Remove the old major version. To do this follow a similar process as above (i.e. editCargo.toml to remove the old version, rungnrt vendor, and so forth). Any leftover files in//third_party/rust/<crate>/<old epoch> should also be removed.

Note that the followingCargo.toml syntax allows two versions of a crate to coexist:

[dependencies.serde_json_lenient_old_epoch]package="serde_json_lenient"version="0.1"[dependencies.serde_json_lenient]version="0.2"

Other ways to usecreate_update_cl.py

auto mode

Extra arguments passed tocreate_update_cl.py auto end up being passed tocargo update. For a complete list of available options, seeCargo documentation here), but the most common scenarios are covered in the sections below.

Updating all crates during the weekly rotation

tools/crates/create_update_cl.py auto with no extra arguments will attempt to discoverminor version updates forall crates that Chromium depends on and for their transitive dependencies.

Updating the minor version of a single crate

tools/crates/create_update_cl.py auto -- some_crate_name can be used to trigger aminor version update of a single crate.

Updating the major version of a single crate

tools/crates/create_update_cl.py auto -- some_crate_name --breaking can be used to trigger amajor version update of a single crate

manual mode

For maximal control, the script can be used inmanual mode:

1. PrepareCargo.toml change:
   1. git checkout origin/main
   2. git checkout -b manual-update-of-foo
   3. Editthird_party/rust/chromium_crates_io/Cargo.toml to change the crate version of the crate (or crates) you want to update.Important: Do not editCargo.lock (e.g. don't rungnrt vendor etc.).
   4. git add third_party/rust/chromium_crates_io/Cargo.toml
   5. git commit -m "Manual edit of Cargo.toml"
   6. git cl upload -m "Manual edit of Cargo.toml" --bypass-hooks --skip-title --force
2. Run the helper script as follows:tools/crates/create_update_cl.py manual --title "Roll foo crate to new version X"
   • This will rungnrt vendor to discover and execute updates that were requested by the manual edits ofCargo.toml in the previous steps.
   • This will automatically add more details to the CL description
   • To make the review easier, one of the patchsets covers just the path changes. For example - seethe delta here.

Recovering from script failures

Sometimes thecreate_update_cl.py script will fail when dealing with a specific crate update. The general workflow in this case is to

1. fix the issue in a separate CL, and 2) restart the tool from the middle by using--upstream-branch that points to the last successful update branch (or to the fix CL) rather than defaulting toorigin/main.

Examples of a few specific situations that may lead to script failure:

• An update brought in a new crate, butgnrt didn‘t recognize new crate’s license kind or license file. In that case a prerequisite CL needs to be landed first, teachinggnrt about the new license kinds/files (in readme.rs). You can seean example CL with such a fix.
• Patches from//third_party/rust/chromium_crates_io/patches/ no longer apply cleanly to the new version of a crate. In that case the crate update CL needs to 1) first update the patches, and then 2) update the crate as usual. This is not very well supported by the script... But something like this should work:
  • Checkout a new branch:

    $ git checkout rust-crates-update--last-successful-update$ git checkout-b fix-patches-for-foo$ git branch--set-upstream-to=rust-crates-update--last-successful-update

  • Fix the patches and upload as a temporary / throw-away CL (this CL can't be landed on its own - it needs to be combined with the actual update CL):

    $# Fix the patches$ git commit-a-m...$ git cl upload

  • Restart the script (the CL created by the script can't be landed as-is / on its own - it needs to be combined with the fixed patches in the step below) with--upstream-branch parameter:

    $ tools/crates/create_update_cl.py auto \--upstream-branch=fix-patches-for-foo \-- name-of-failed-crate

  • Combine the branches:

    $ git map-branches-v# to orient yourself$ git checkout rust-crates-update--new-successful-update$ git branch--set-upstream-to=rust-crates-update--last-successful-update$ git cl upload-mRebasing...# --bypass-hooks as needed

• //third_party/rust/chromium_crates_io/gnrt_config.toml needs to be updated to work with a new crate version. The same workflow should work as for fixing//third_party/rust/chromium_crates_io/patches/ (see the item above).