1.3.1.Why is LTS required?

LTS is needed for commercial reasons. More specifically, on the device side,when a product is released, the companies have to support that in-market productsuch that the amount of changes to the firmware are kept to a minimum to avoidthe risk of regression. At the same time the companies don’t want to excludecritical patches such as those for security advisories. Similarly on the server side,companies want to minimize the churn when deploying fixes during incidentresponse, e.g. due to critical security bugs.

Also, the European Cyber Resilience Act (CRA) is a new EU legislation that mandatescybersecurity standards for products containing digital elements, aiming toprotect consumers and businesses by ensuring manufacturers build security intotheir hardware and software throughout their lifecycle, including automaticupdates and incident reporting; essentially requiring all digital productssold in the EU to meet specific cybersecurity requirements.

This means that companies have to maintain and backport critical updates toold branches internally. As this effort is duplicated across different companiesusing TF-A, it makes sense to factor out this effort into a community-wide LTS.

1.3.2.What does LTS mean for TF-A?

In this section we will define exactly what constitutes LTS for TF-A.Specifically, we will define the following characteristics:

• criteria for selecting patches which will be backported to LTS branches

• lifetime and frequency of LTS branches

Criteria

We must have an objective criterion for selecting patches to be backported toLTS branches. This will make maintenance easy because:

1. there will be less – ideally no – discussion when selecting patches to backport

2. large parts of the process can be automated

Below is the criteria

1. No features will be backported.

2. Security advisories: Any patch that makes it intoSecurity Advisoriesis automatically selected for back porting. This includes patches to externalcomponents too, e.g. libfdt.

3. Workarounds for CPU and other ARM IP errata

4. Workarounds for non-ARM IP errata, e.g. TI UART

5. Fixes for platform bugs. These patches must not modify any code outside ofthe specific platform that the fix applies to.

6. Patches can only be backported from the master branch. In other words, themaster branch will be a superset of all the changes in any LTS branch.

Lifetime and frequency

This section approaches three questions: for how long should an LTS release besupported, how frequently should LTS releases be made and at which time(s) ofthe year should the releases be made.

1. For how long should an LTS release be supported?

The Linux kernel maintainers supports an LTS branch for 2 years. Since firmwaretends to have less churn and longer lifetime than a HLOS, TF-A is trying tosupport at-least 7 years for its LTS. Initially it was intended to support5 years but there has been no objections to extend LTS support to 7 years.There are many challenges that may influence the 7 year support from CIinfrastructure to availability of maintainers.

2. How frequently should LTS releases be made?

Given that many products that have a release cycle, have a yearly releasecycle, it would make sense to have yearly TF-A releases.

3. Which time(s) of the year should the releases be made?

TF-A releases are cut twice a year: May and November. Basing LTS releaseon the November TF-A release has a few benefits. First, it aligns with LinuxLTS releases which happen towards the end of each year. Second, it alignswith Android releases which tend to fall in Q3 each year. Since productreleases are timed with Android release, this gives enough time to hardenthe TF-A LTS release during development so that it’s ready for launch inQ3 following year. On the other hand, if the May release of TF-A is chosen asthe basis for LTS then developers will have little time – about a month,taking into account the test-and-debug phase before LTS is cut (see below) –before Android release.

To summarize, there will be one LTS release per year. It will be supported for5 years and we can discuss extending it to 7 years later on. The LTS releasewill be based on the November release of TF-A.

Testing Criteria

Every patch merged to the LTS branch will complete the following tests beforegetting approved.

1. TFTF tests currently running in the testing farm

2. CI/CD static analysis scans

3. Coverity scans

4. Platform tests

Platforms that are not maintained upstream will undergo testing downstream in apre-defined window. The platform maintainer will complete the testing and providea verified score on the patch once testing is completed.

** A note about test coverage from TF.org **

Currently TF.org maintains a CI system to run TF-A automated tests on aselection of HW boards donated by TF.org members (a benefit reserved to projectmembers, see the project charter for more details). This automated test coveragewill be extended to cover testing for LTS as well for boards that are part ofthe CI system.

TFTF Branching

A note about testing here. After a patch is backported to an LTS branch, thatbranch will need to be regression tested. Since TFTF moves forward with latestTF-A changes, newer TFTF tests may not apply to old LTS branches. ThereforeTFTF will also need to be branched, in-sync with TF-A LTS branches. In otherwords, there will be one TFTF LTS branch corresponding to each TF-A LTS branch.The TFTF LTS branch will be used to regression test the corresponding TF-A LTSbranch.

As we work with the LTS branch of TFTF, we might also need fixes for TFTFitself to be ported to LTS. However, decision-making about those patches neednot be as stringent as for TF-A.

CI Scripts

CI Scripts moves forward with TF-A changes, since we need to checkout thecorresponding release version of CI scripts for LTS.

Though we are unlikely to update CI scripts, but time to time migrating a newerFVP version or deprecating certain tests due to unavailability of platforms mayinfluence updates to CI Scripts.

Hafnium / OP-TEE

Both Hafnium and OP-TEE move forward with TF-A changes so we need to freeze theircorresponding version from TF-A release for a LTS.

MbedTLS

Updates to the version of MbedTLS used with LTS will happen time to time based onmaintainers call to update them or not.

1.3.3.Release details

This section goes into details of what the LTS release process will look like.

Test-and-debug period

Since the LTS branch will be used in product releases, it is expected that moretesting and debugging will be done on the November release of TF-A. Thereforeit would make sense to leave at least a month after the November release andthen cut the LTS branch. We recommend two months, given that one of the monthsis December which tends to be slower due to holidays. So, an end-of-NovemberTF-A release would result in a beginning-of-February LTS release. Note thatthe LTS branch will be created at the same time as the TF-A November release,but it will be officially released at the end of January or early February.Going forward we should strive to make the period smaller and smaller untilideally it coincides with TF-A November release which means that our testand CI/CD infra is good enough to allow that to happen.

Example timeline

Below is an example timeline starting from the November 2022 release of TF-A.

• Nov 2022: TF-A 2.8 is released towards the end of Nov, 2022. Not shown in thediagram, at the same time LTS release candidate branch is made which is basedon TF-A 2.8. This means new features going in 2.8 won’t go in the LTS branch.We can call itLTS 2.8-rc.

• Feb 2023: After testing and debugging LTS 2.8-rc for a couple of months,LTS 2.8.0 is officially released in early Feb 2023.

• May 2023: TF-A 2.9 is released but since this is not an LTS branch it doesn’taffect LTS.

• Somewhere between May and Nov of 2023: A security advisory comes up and therelated patches go into TF-A master branch. Since these patches fall underLTS criteria, they are backported to LTS 2.8.0 which results in LTS 2.8.1being released. Note that here we don’t allow the extra testing and debuggingtime that we had between Nov 2022 and early Feb 2023. This is because thereisn’t as much to test and debug as an annual LTS release has. Also companiesmight want to deploy critical patches soon.

• Nov 2023: TF-A 2.10 is released. Not shown in the diagram, at the same timeLTS 2.10-rc is made. It’s tested by partners for a couple of months.

• Feb 2024: LTS 2.10.1 is released in early Feb. Now there are two LTSbranches: 2.8.1 and 2.10.1.

Note that TFTF will follow similar branching model as TF-A LTS, i.e. there willbe TFTF LTS 2.8.0 in Feb 2023, 2.8.1 (if new TFTF tests need to be added forthe security advisory) when there is TF-A LTS 2.8.1 and so on.

1.3.4.Maintainership

Guidelines & Responsibilities

1. Maintainers shall be impartial and strive to work for the benefit ofthe community

2. Objective and well-defined merge criteria to avoid confusion and discussionsat random points in time when there is a “candidate” patch

3. The maintainers shall explain the lifecycle of a patch to the community,with a detailed description of the maximum time spent in each step

4. Automate, automate, automate

5. Reviewers should not focus too much on “what” and instead focus on “how”

6. Constantly refine the merge criteria to include more partner use cases

7. Ensure that all candidate patches flow from the main branch to all LTS branches

8. Maintainers collaborate in the following discord channel -https://discord.com/channels/1106321706588577904/1162029539761852436

9. Maintainers discuss and provide updates about upcoming LTS releases in the abovementioned discord channel.

Options

These are some options in the order of preference.

1. Current set ofLTS Maintainers from tf.org(or hired contractor) take care of the LTS

2. From the community, create a set of maintainers focused solely on the LTS branches

1.3.5.Playbook for new releases

To make a new minor release (e.g. 2.x.y → 2.x.y+1), follow these steps.

1. Every Friday, LTS maintainers receive a triage report email (subject: “TF-ALTS Triage report”) that contains attached CSV files, 1 percurrently-supported LTS major release branch (e.g. lts-2.8, lts-2.10,lts-2.12, etc.). It contains a list of patches to be cherry-picked into a newminor release of each supported LTS branch.

2. Rungitfetchorigin.

3. Rungitcheckout-blts-v2.x.y+1--trackorigin/lts-v2.x.

4. Rungitlog and verify that the most recent commit is the changelog forthe v2.x.y release, and that it has the origin/lts-v2.x tag.

5. For the version 2.x for which you want to create a new release, open its CSVfile. For each patch listed,from the bottom to the top, rungitcherry-pick-xsha1-hash.

6. Some of the patches of this list may not be taken, mainly due to falsepositive. If in doubt, that can be discussed either in the “tf-a-lts” channelon Discord or during the LTS weekly meeting.

7. Some dependency patches, not listed in the CSV file, may have to be taken, to ease theapplication of the LTS patches. This can also be discussed with the other LTS maintainers.

8. Push the stack of changes:gitpushoriginHEAD:refs/for/lts-v2.x%topic=for-lts-v2.x.y+1. You might need the--no-verify option:gitpushorigin--no-verifyHEAD:refs/for/lts-v2.x%topic=for-lts-v2.x.y+1.

9. The AllowCI+2 job runs automatically on each LTS branch once a newcherry-picked patch/patch-stack is pushed to the corresponding branch. Ifthis CI run passes, it automatically applies the Verified+1 (V+1) label tothe patch/all patches in the stack. The other LTS maintainers will provideMR+1 and COR+1 votes. If the CI is OK and votes V+1, and if theMaintainer-Review+1 (MR+1), Code-Owner-Review+1 (COR+1), and V+1 votes arepresent, Gerrit will automatically merge the patch. LTS maintainers will thentrigger a Jenkins job that will take care of the release (tag, mail, andreadthedocs update).

10. Some features may also require updates in other repositories (tf-a-ci-scripts,tf-a-job-configs or tf-a-tests…). For tf-a-job-configs, there are no LTS branches, butdedicated scripts for each LTS version which have to be updated manually. This is the casefor e.g. MbedTLS updates. For tf-a-ci-scripts and tf-a-tests, there are LTS branches and patcheswill be cherry-picked from master branch to the LTS branch the same way it is done for TF-A.There is no automation for those repositories. So the patches will have to be merged manually,and for tf-a-ci-scripts and tf-a-tests, tags will also have to be set manually.