Bisecting a regression

This document describes how to use agitbisect to find the source codechange that broke something -- for example when some functionality stoppedworking after upgrading from Linux 6.0 to 6.1.

The text focuses on the gist of the process. If you are new to bisecting thekernel, better followHow to verify bugs and bisect regressionsinstead: it depicts everything from start to finish while covering multipleaspects even kernel developers occasionally forget. This includes detectingsituations early where a bisection would be a waste of time, as nobody wouldcare about the result -- for example, because the problem happens after thekernel marked itself as ‘tainted’, occurs in an abandoned version, was alreadyfixed, or is caused by a .config change you or your Linux distributor performed.

Finding the change causing a kernel issue using a bisection

Note: the following process assumes you prepared everything for a bisection.This includes having a Git clone with the appropriate sources, installing thesoftware required to build and install kernels, as well as a .config file storedin a safe place (the following example assumes ‘~/prepared_kernel_.config’) touse as pristine base at each bisection step; ideally, you have also worked outa fully reliable and straight-forward way to reproduce the regression, too.

  • Preparation: start the bisection and tell Git about the points in the historyyou consider to be working and broken, which Git calls ‘good’ and ‘bad’:

    git bisect startgit bisect good v6.0git bisect bad v6.1

    Instead of Git tags like ‘v6.0’ and ‘v6.1’ you can specify commit-ids, too.

  1. Copy your prepared .config into the build directory and adjust it to theneeds of the codebase Git checked out for testing:

    cp ~/prepared_kernel_.config .configmake olddefconfig
  2. Now build, install, and boot a kernel. This might fail for unrelated reasons,for example, when a compile error happens at the current stage of thebisection a later change resolves. In such cases rungitbisectskip andgo back to step 1.

  3. Check if the functionality that regressed works in the kernel you just built.

    If it works, execute:

    git bisect good

    If it is broken, run:

    git bisect bad

    Note, getting this wrong just once will send the rest of the bisectiontotally off course. To prevent having to start anew later you thus want toensure what you tell Git is correct; it is thus often wise to spend a fewminutes more on testing in case your reproducer is unreliable.

    After issuing one of these two commands, Git will usually check out anotherbisection point and print something like ‘Bisecting: 675 revisions left totest after this (roughly 10 steps)’. In that case go back to step 1.

    If Git instead prints something like ‘cafecaca0c0dacafecaca0c0dacafecaca0c0dais the first bad commit’, then you have finished the bisection. In that casemove to the next point below. Note, right after displaying that line Git willshow some details about the culprit including its patch description; this caneasily fill your terminal, so you might need to scroll up to see the messagementioning the culprit’s commit-id.

    In case you missed Git’s output, you can always rungitbisectlog toprint the status: it will show how many steps remain or mention the result ofthe bisection.

  • Recommended complementary task: put the bisection log and the current .configfile aside for the bug report; furthermore tell Git to reset the sources tothe state before the bisection:

    git bisect log > ~/bisection-logcp .config ~/bisection-config-culpritgit bisect reset
  • Recommended optional task: try reverting the culprit on top of the latestcodebase and check if that fixes your bug; if that is the case, it validatesthe bisection and enables developers to resolve the regression through arevert.

    To try this, update your clone and check out latest mainline. Then tell Gitto revert the change by specifying its commit-id:

    git revert --no-edit cafec0cacaca0

    Git might reject this, for example when the bisection landed on a mergecommit. In that case, abandon the attempt. Do the same, if Git fails to revertthe culprit on its own because later changes depend on it -- at least unlessyou bisected a stable or longterm kernel series, in which case you want tocheck out its latest codebase and try a revert there.

    If a revert succeeds, build and test another kernel to check if revertingresolved your regression.

With that the process is complete. Now report the regression as described byReporting issues.

Bisecting linux-next

If you face a problem only happening in linux-next, bisect between thelinux-next branches ‘stable’ and ‘master’. The following commands will startthe process for a linux-next tree you added as a remote called ‘next’:

git bisect startgit bisect good next/stablegit bisect bad next/master

The ‘stable’ branch refers to the state of linux-mainline that the currentlinux-next release (found in the ‘master’ branch) is based on -- the formerthus should be free of any problems that show up in -next, but not in Linus’tree.

This will bisect across a wide range of changes, some of which you might haveused in earlier linux-next releases without problems. Sadly there is no simpleway to avoid checking them: bisecting from one linux-next release to a laterone (say between ‘next-20241020’ and ‘next-20241021’) is impossible, as theyshare no common history.

Additional reading material