Backporting and conflict resolution

Author:

Vegard Nossum <vegard.nossum@oracle.com>

Introduction

Some developers may never really have to deal with backporting patches,merging branches, or resolving conflicts in their day-to-day work, sowhen a merge conflict does pop up, it can be daunting. Luckily,resolving conflicts is a skill like any other, and there are many usefultechniques you can use to make the process smoother and increase yourconfidence in the result.

This document aims to be a comprehensive, step-by-step guide tobackporting and conflict resolution.

Applying the patch to a tree

Sometimes the patch you are backporting already exists as a git commit,in which case you just cherry-pick it directly usinggitcherry-pick. However, if the patch comes from an email, as itoften does for the Linux kernel, you will need to apply it to a treeusinggitam.

If you’ve ever usedgitam, you probably already know that it isquite picky about the patch applying perfectly to your source tree. Infact, you’ve probably had nightmares about.rej files and trying toedit the patch to make it apply.

It is strongly recommended to instead find an appropriate base versionwhere the patch applies cleanly andthen cherry-pick it over to yourdestination tree, as this will make git output conflict markers and letyou resolve conflicts with the help of git and any other conflictresolution tools you might prefer to use. For example, if you want toapply a patch that just arrived on LKML to an older stable kernel, youcan apply it to the most recent mainline kernel and then cherry-pick itto your older stable branch.

It’s generally better to use the exact same base as the one the patchwas generated from, but it doesn’t really matter that much as long as itapplies cleanly and isn’t too far from the original base. The onlyproblem with applying the patch to the “wrong” base is that it may pullin more unrelated changes in the context of the diff when cherry-pickingit to the older branch.

A good reason to prefergitcherry-pick overgitam is that gitknows the precise history of an existing commit, so it will know whencode has moved around and changed the line numbers; this in turn makesit less likely to apply the patch to the wrong place (which can resultin silent mistakes or messy conflicts).

If you are usingb4. and you are applying the patch directly from anemail, you can useb4am with the options-g/--guess-baseand-3/--prep-3way to do some of this automatically (see theb4 presentation for more information). However, the rest of thisarticle will assume that you are doing a plaingitcherry-pick.

Once you have the patch in git, you can go ahead and cherry-pick it intoyour source tree. Don’t forget to cherry-pick with-x if you want awritten record of where the patch came from!

Note that if you are submitting a patch for stable, the format isslightly different; the first line after the subject line needs to beeither:

commit <upstream commit> upstream

or:

[ Upstream commit <upstream commit> ]

Resolving conflicts

Uh-oh; the cherry-pick failed with a vaguely threatening message:

CONFLICT (content): Merge conflict

What to do now?

In general, conflicts appear when the context of the patch (i.e., thelines being changed and/or the lines surrounding the changes) doesn’tmatch what’s in the tree you are trying to apply the patchto.

For backports, what likely happened was that the branch you arebackporting from contains patches not in the branch you are backportingto. However, the reverse is also possible. In any case, the result is aconflict that needs to be resolved.

If your attempted cherry-pick fails with a conflict, git automaticallyedits the files to include so-called conflict markers showing you wherethe conflict is and how the two branches have diverged. Resolving theconflict typically means editing the end result in such a way that ittakes into account these other commits.

Resolving the conflict can be done either by hand in a regular texteditor or using a dedicated conflict resolution tool.

Many people prefer to use their regular text editor and edit theconflict directly, as it may be easier to understand what you’re doingand to control the final result. There are definitely pros and cons toeach method, and sometimes there’s value in using both.

We will not cover using dedicated merge tools here beyond providing somepointers to various tools that you could use:

To configure git to work with these, seegitmergetool--help orthe officialgit-mergetool documentation.

Prerequisite patches

Most conflicts happen because the branch you are backporting to ismissing some patches compared to the branch you are backportingfrom.In the more general case (such as merging two independent branches),development could have happened on either branch, or the branches havesimply diverged -- perhaps your older branch had some other backportsapplied to it that themselves needed conflict resolutions, causing adivergence.

It’s important to always identify the commit or commits that caused theconflict, as otherwise you cannot be confident in the correctness ofyour resolution. As an added bonus, especially if the patch is in anarea you’re not that familiar with, the changelogs of these commits willoften give you the context to understand the code and potential problemsor pitfalls with your conflict resolution.

git log

A good first step is to look atgitlog for the file that has theconflict -- this is usually sufficient when there aren’t a lot ofpatches to the file, but may get confusing if the file is big andfrequently patched. You should rungitlog on the range of commitsbetween your currently checked-out branch (HEAD) and the parent ofthe patch you are picking (<commit>), i.e.:

git log HEAD..<commit>^ -- <path>

Even better, if you want to restrict this output to a single function(because that’s where the conflict appears), you can use the followingsyntax:

git log -L:'\<function\>':<path> HEAD..<commit>^

Note

The\< and\> around the function name ensure that thematches are anchored on a word boundary. This is important, as thispart is actually a regex and git only follows the first match, soif you use-L:thread_stack:kernel/fork.c it may only give youresults for the functiontry_release_thread_stack_to_cache eventhough there are many other functions in that file containing thestringthread_stack in their names.

Another useful option forgitlog is-G, which allows you tofilter on certain strings appearing in the diffs of the commits you arelisting:

git log -G'regex' HEAD..<commit>^ -- <path>

This can also be a handy way to quickly find when something (e.g. afunction call or a variable) was changed, added, or removed. The searchstring is a regular expression, which means you can potentially searchfor more specific things like assignments to a specificstructmember:

git log -G'\->index\>.*='

git blame

Another way to find prerequisite commits (albeit only the most recentone for a given conflict) is to rungitblame. In this case, youneed to run it against the parent commit of the patch you arecherry-picking and the file where the conflict appeared, i.e.:

git blame <commit>^ -- <path>

This command also accepts the-L argument (for restricting theoutput to a single function), but in this case you specify the filenameat the end of the command as usual:

git blame -L:'\<function\>' <commit>^ -- <path>

Navigate to the place where the conflict occurred. The first column ofthe blame output is the commit ID of the patch that added a given lineof code.

It might be a good idea togitshow these commits and see if theylook like they might be the source of the conflict. Sometimes there willbe more than one of these commits, either because multiple commitschanged different lines of the same conflict areaor because multiplesubsequent patches changed the same line (or lines) multiple times. Inthe latter case, you may have to rungitblame again and specify theolder version of the file to look at in order to dig further back inthe history of the file.

Prerequisite vs. incidental patches

Having found the patch that caused the conflict, you need to determinewhether it is a prerequisite for the patch you are backporting orwhether it is just incidental and can be skipped. An incidental patchwould be one that touches the same code as the patch you arebackporting, but does not change the semantics of the code in anymaterial way. For example, a whitespace cleanup patch is completelyincidental -- likewise, a patch that simply renames a function or avariable would be incidental as well. On the other hand, if the functionbeing changed does not even exist in your current branch then this wouldnot be incidental at all and you need to carefully consider whether thepatch adding the function should be cherry-picked first.

If you find that there is a necessary prerequisite patch, then you needto stop and cherry-pick that instead. If you’ve already resolved someconflicts in a different file and don’t want to do it again, you cancreate a temporary copy of that file.

To abort the current cherry-pick, go ahead and rungitcherry-pick--abort, then restart the cherry-picking processwith the commit ID of the prerequisite patch instead.

Understanding conflict markers

Combined diffs

Let’s say you’ve decided against picking (or reverting) additionalpatches and you just want to resolve the conflict. Git will haveinserted conflict markers into your file. Out of the box, this will looksomething like:

<<<<<<< HEADthis is what's in your current tree before cherry-picking=======this is what the patch wants it to be after cherry-picking>>>>>>> <commit>... title

This is what you would see if you opened the file in your editor.However, if you were to rungitdiff without any arguments, theoutput would look something like this:

$ git diff[...]++<<<<<<<< HEAD +this is what's in your current tree before cherry-picking++========+ this is what the patch wants it to be after cherry-picking++>>>>>>>> <commit>... title

When you are resolving a conflict, the behavior ofgitdiff differsfrom its normal behavior. Notice the two columns of diff markersinstead of the usual one; this is a so-called “combined diff”, hereshowing the 3-way diff (or diff-of-diffs) between

  1. the current branch (before cherry-picking) and the current workingdirectory, and

  2. the current branch (before cherry-picking) and the file as it looksafter the original patch has been applied.

Better diffs

3-way combined diffs include all the other changes that happened to thefile between your current branch and the branch you are cherry-pickingfrom. While this is useful for spotting other changes that you need totake into account, this also makes the output ofgitdiff somewhatintimidating and difficult to read. You may instead prefer to rungitdiffHEAD (orgitdiff--ours) which shows only the diffbetween the current branch before cherry-picking and the current workingdirectory. It looks like this:

$ git diff HEAD[...]+<<<<<<<< HEAD this is what's in your current tree before cherry-picking+========+this is what the patch wants it to be after cherry-picking+>>>>>>>> <commit>... title

As you can see, this reads just like any other diff and makes it clearwhich lines are in the current branch and which lines are being addedbecause they are part of the merge conflict or the patch beingcherry-picked.

Merge styles and diff3

The default conflict marker style shown above is known as themergestyle. There is also another style available, known as thediff3style, which looks like this:

<<<<<<< HEADthis is what is in your current tree before cherry-picking||||||| parent of <commit> (title)this is what the patch expected to find there=======this is what the patch wants it to be after being applied>>>>>>> <commit> (title)

As you can see, this has 3 parts instead of 2, and includes what gitexpected to find there but didn’t. It ishighly recommended to usethis conflict style as it makes it much clearer what the patch actuallychanged; i.e., it allows you to compare the before-and-after versionsof the file for the commit you are cherry-picking. This allows you tomake better decisions about how to resolve the conflict.

To change conflict marker styles, you can use the following command:

git config merge.conflictStyle diff3

There is a third option,zdiff3, introduced inGit 2.35,which has the same 3 sections asdiff3, but where common lines havebeen trimmed off, making the conflict area smaller in some cases.

Iterating on conflict resolutions

The first step in any conflict resolution process is to understand thepatch you are backporting. For the Linux kernel this is especiallyimportant, since an incorrect change can lead to the whole systemcrashing -- or worse, an undetected security vulnerability.

Understanding the patch can be easy or difficult depending on the patchitself, the changelog, and your familiarity with the code being changed.However, a good question for every change (or every hunk of the patch)might be: “Why is this hunk in the patch?” The answers to thesequestions will inform your conflict resolution.

Resolution process

Sometimes the easiest thing to do is to just remove all but the firstpart of the conflict, leaving the file essentially unchanged, and applythe changes by hand. Perhaps the patch is changing a function callargument from0 to1 while a conflicting change added anentirely new (and insignificant) parameter to the end of the parameterlist; in that case, it’s easy enough to change the argument from0to1 by hand and leave the rest of the arguments alone. Thistechnique of manually applying changes is mostly useful if the conflictpulled in a lot of unrelated context that you don’t really need to careabout.

For particularly nasty conflicts with many conflict markers, you can usegitadd orgitadd-i to selectively stage your resolutions toget them out of the way; this also lets you usegitdiffHEAD toalways see what remains to be resolved orgitdiff--cached to seewhat your patch looks like so far.

Dealing with file renames

One of the most annoying things that can happen while backporting apatch is discovering that one of the files being patched has beenrenamed, as that typically means git won’t even put in conflict markers,but will just throw up its hands and say (paraphrased): “Unmerged path!You do the work...”

There are generally a few ways to deal with this. If the patch to therenamed file is small, like a one-line change, the easiest thing is tojust go ahead and apply the change by hand and be done with it. On theother hand, if the change is big or complicated, you definitely don’twant to do it by hand.

As a first pass, you can try something like this, which will lower therename detection threshold to 30% (by default, git uses 50%, meaningthat two files need to have at least 50% in common for it to consideran add-delete pair to be a potential rename):

git cherry-pick -strategy=recursive -Xrename-threshold=30

Sometimes the right thing to do will be to also backport the patch thatdid the rename, but that’s definitely not the most common case. Instead,what you can do is to temporarily rename the file in the branch you’rebackporting to (usinggitmv and committing the result), restart theattempt to cherry-pick the patch, rename the file back (gitmv andcommitting again), and finally squash the result usinggitrebase-i(see therebase tutorial) so it appears as a single commit when youare done.

Gotchas

Function arguments

Pay attention to changing function arguments! It’s easy to gloss overdetails and think that two lines are the same but actually they differin some small detail like which variable was passed as an argument(especially if the two variables are both a single character that lookthe same, like i and j).

Error handling

If you cherry-pick a patch that includes agoto statement (typicallyfor error handling), it is absolutely imperative to double check thatthe target label is still correct in the branch you are backporting to.The same goes for addedreturn,break, andcontinuestatements.

Error handling is typically located at the bottom of the function, so itmay not be part of the conflict even though could have been changed byother patches.

A good way to ensure that you review the error paths is to always usegitdiff-W andgitshow-W (AKA--function-context) wheninspecting your changes. For C code, this will show you the wholefunction that’s being changed in a patch. One of the things that oftengo wrong during backports is that something else in the function changedon either of the branches that you’re backporting from or to. Byincluding the whole function in the diff you get more context and canmore easily spot problems that might otherwise go unnoticed.

Refactored code

Something that happens quite often is that code gets refactored by“factoring out” a common code sequence or pattern into a helperfunction. When backporting patches to an area where such a refactoringhas taken place, you effectively need to do the reverse whenbackporting: a patch to a single location may need to be applied tomultiple locations in the backported version. (One giveaway for thisscenario is that a function was renamed -- but that’s not always thecase.)

To avoid incomplete backports, it’s worth trying to figure out if thepatch fixes a bug that appears in more than one place. One way to dothis would be to usegitgrep. (This is actually a good idea to doin general, not just for backports.) If you do find that the same kindof fix would apply to other places, it’s also worth seeing if thoseplaces exist upstream -- if they don’t, it’s likely the patch may needto be adjusted.gitlog is your friend to figure out what happenedto these areas asgitblame won’t show you code that has beenremoved.

If you do find other instances of the same pattern in the upstream treeand you’re not sure whether it’s also a bug, it may be worth asking thepatch author. It’s not uncommon to find new bugs during backporting!

Verifying the result

colordiff

Having committed a conflict-free new patch, you can now compare yourpatch to the original patch. It is highly recommended that you use atool such ascolordiff that can show two files side by side and colorthem according to the changes between them:

colordiff -yw -W 200 <(git diff -W <upstream commit>^-) <(git diff -W HEAD^-) | less -SR

Here,-y means to do a side-by-side comparison;-w ignoreswhitespace, and-W200 sets the width of the output (as otherwise itwill use 130 by default, which is often a bit too little).

Therev^- syntax is a handy shorthand forrev^..rev, essentiallygiving you just the diff for that single commit; also seethe officialgit rev-parse documentation.

Again, note the inclusion of-W forgitdiff; this ensures thatyou will see the full function for any function that has changed.

One incredibly important thing that colordiff does is to highlight linesthat are different. For example, if an error-handlinggoto haschanged labels between the original and backported patch, colordiff willshow these side-by-side but highlighted in a different color. Thus, itis easy to see that the twogoto statements are jumping to differentlabels. Likewise, lines that were not modified by either patch butdiffer in the context will also be highlighted and thus stand out duringa manual inspection.

Of course, this is just a visual inspection; the real test is buildingand running the patched kernel (or program).

Build testing

We won’t cover runtime testing here, but it can be a good idea to buildjust the files touched by the patch as a quick sanity check. For theLinux kernel you can build single files like this, assuming you have the.config and build environment set up correctly:

make path/to/file.o

Note that this won’t discover linker errors, so you should still do afull build after verifying that the single file compiles. By compilingthe single file first you can avoid having to wait for a full buildincase there are compiler errors in any of the files you’ve changed.

Runtime testing

Even a successful build or boot test is not necessarily enough to ruleout a missing dependency somewhere. Even though the chances are small,there could be code changes where two independent changes to the samefile result in no conflicts, no compile-time errors, and runtime errorsonly in exceptional cases.

One concrete example of this was a pair of patches to the system callentry code where the first patch saved/restored a register and a laterpatch made use of the same register somewhere in the middle of thissequence. Since there was no overlap between the changes, one couldcherry-pick the second patch, have no conflicts, and believe thateverything was fine, when in fact the code was now scribbling over anunsaved register.

Although the vast majority of errors will be caught during compilationor by superficially exercising the code, the only way toreally verifya backport is to review the final patch with the same level of scrutinyas you would (or should) give to any other patch. Having unit tests andregression tests or other types of automatic testing can help increasethe confidence in the correctness of a backport.

Submitting backports to stable

As the stable maintainers try to cherry-pick mainline fixes onto theirstable kernels, they may send out emails asking for backports whenencountering conflicts, see e.g.<https://lore.kernel.org/stable/2023101528-jawed-shelving-071a@gregkh/>.These emails typically include the exact steps you need to cherry-pickthe patch to the correct tree and submit the patch.

One thing to make sure is that your changelog conforms to the expectedformat:

<original patch title>[ Upstream commit <mainline rev> ]<rest of the original changelog>[ <summary of the conflicts and their resolutions> ]Signed-off-by: <your name and email>

The “Upstream commit” line is sometimes slightly different depending onthe stable version. Older version used this format:

commit <mainline rev> upstream.

It is most common to indicate the kernel version the patch applies toin the email subject line (using e.g.gitsend-email--subject-prefix='PATCH6.1.y'), but you can also putit in the Signed-off-by:-area or below the--- line.

The stable maintainers expect separate submissions for each activestable version, and each submission should also be tested separately.

A few final words of advice

  1. Approach the backporting process with humility.

  2. Understand the patch you are backporting; this means reading boththe changelog and the code.

  3. Be honest about your confidence in the result when submitting thepatch.

  4. Ask relevant maintainers for explicit acks.

Examples

The above shows roughly the idealized process of backporting a patch.For a more concrete example, see this video tutorial where two patchesare backported from mainline to stable:Backporting Linux Kernel Patches.