Repository files navigation

This repository contains references to tests (in theregistrydirectory) that are run with every commit to Flutter to verify that nobreaking changes have been introduced (in the "customer_testing"shards). The tests referenced by this repository are typicallymaintained by people outside of the Flutter team, as part of thedevelopment of their applications. They are intended to give theFlutter team visibility into how their changes affect real-worlddevelopers using Flutter.

You are welcome to add a new batch of tests. To do so, copy the fileregistry/template.test to create a new file in theregistry/directory. Fill in the fields and delete all the comments. Then,submit a PR with this new file.

Tests must fulfill the following criteria to be added:

• All the code must be available publicly on GitHub under a licensecompatible with this effort.

• Tests must be hermetic. For example, a test should not involvenetwork activity, spawn processes, or access the local file systemexcept to access files that are packaged with the test.

• Tests must be resilient to being run concurrently with other tests,including concurrent runs of themselves.

• Tests must be reliable. A test must not claim to pass if it isfailing. Running a test multiple times in a row must always have thesame result.

• Tests must have no output when they are passing.

• Tests must be as fast as possible given the hardware. For example,tests must not use real timers or depend on the wall clock.

• The time taken by tests must be proportional to their value. A fewthousands tests are expected to run within a few minutes. An upperlimit of about five minutes will be applied to each contributed testsuite (not including the time to download the tests), but it isexpected that most suites will complete in seconds.

• The tests must be compatible with any tools for automaticallyupdating Flutter code (e.g. they cannot rely on custom codegeneration unless such code generation can hook into the automaticupdate mechanism).

• The tests must represent good practices as described in Flutter'sdocumentation. For example, using an object after calling itsdispose method violates the contract described by that method.Accessing the fields of a privateState subclass from anotherpackage by casting it to dynamic is similarly sketchy and would notbe supported behaviour.

• The tests must pass even if there are analysis 'info' level issuesin the code. Generally, this means that if the test performs staticanalysis, it does so ignoring info level items (i.e.,flutter analyze --no-fatal-infos).

• The tests must pass at the time they are contributed.

• The upstream repository that hosts the tests must be able to receive patchesto support themain channel of Flutter. This means that CI on theupstream repository should use themain channel Flutter SDK.

• Dependencies must be pinned. (Generally, checking in thepubspec.lock file is sufficient for this purpose.) However,please avoid pinning packages such asintl that are also pinnedby the Flutter SDK, otherwise when Flutter updates the dependencythe tests will fail. Consider usingany for the packages that arepinned by the Flutter framework (that way they are automaticallyupdated when Flutter updates).

To run these tests locally, run:

./scripts/verify_tests_on_main.sh --local-flutter

This will run all the tests using theflutter command that is in thePATH.Use./scripts/verify_tests_on_main.sh --help to see more options.

The point of these tests is to make sure we don't break existing code,while still being able to make improvements to Flutter APIs.

If you find that a PR you have created in flutter/flutter causes onethese tests to fail, you have the following options:

1. Change your PR so that the test no longer fails. This is thepreferred option, so long as the result is one we can be proud of.Is the resulting API something that you would plausibly come upwith without the backwards-compatibility constraint? That's good.Is the resulting API something that, as soon as you see it, youthink "why?" or "that's weird"? That's bad. Consider the advice inthe Style guide:https://github.com/flutter/flutter/blob/master/docs/contributing/Style-guide-for-Flutter-repo.md

2. Go through the breaking change process, as documented here:https://github.com/flutter/flutter/blob/master/docs/contributing/Tree-hygiene.md#handling-breaking-changesIf you're going to do this, you will need to contact the relevantpeople responsible for the breaking test(s) (see the relevant .testfiles), help them fix their code, and update this repository to usethe new version of their tests, in addition to the steps describedon the wiki. You will also need to land your change in two parts,so that people have time to migrate (a "soft-breaking" change).

3. Remove the test in question. This is by far the least idealsolution. To go down this path, we must first establish that one ofthe following is true:

   • the people listed as contacts for the test are not responsive (within 72 hours).

   • the test is poorly written (e.g. it contains a race condition orrelies on assumptions that violate clearly documented APIcontracts), and the people listed as contacts are not willing tofix the test or accept fixes for the test.

   • we have gone through the breaking change process cited above,but are unable to update the test accordingly (e.g. the peoplelisted as contacts are not willing to work with us to updatetheir code).

If you have a significant body of tests that you would like to contribute,but are unable to do so using this registry either because they are proprietary,or because the volume of tests is too great for our CI, please consider reachingout on ourDiscord server.

We are willing to add such tests to our CI, under the following conditions:

• you must commit to being very responsive, promptly responding to relevantdiscussions on our Discord during working hours in your time zone.
• you must take responsibility for helping contributors land patches whichyour tests flag as problematic, including providing debugging help.
• you must fund and maintain any tooling necessary for the integration ofyour tests into our system.
• you agree that the integration will be disabled if it has a non-trivial rateof false-positives, suffers regular infrastructure failures, or is otherwisedisruptive to the project.

Theskp_generator directory contains a Flutter program (andassociated shell scripts) to generate test SKPs for the Skia team.

Contributions in the form of stateless widgets showing scenes fromyour application are welcome. See the README.md in that directory.