Repository files navigation

1. Building the lib and applications
   • Prerequisites
   • Get the code
   • Basics
   • Configuration options
   • Dylib builds
   • Debugging
   • Cross compiling
   • Sanitizer support
   • MSVC builds
   • Xcode builds
   • Emscripten builds
   • Extra Build Flags
   • Build with VMAF support
2. Testing the library
   • Basics
     • Unit tests
     • Example tests
     • Encoder tests
   • IDE hosted tests
   • Downloading test data
   • Adding a new test data file
   • Additional test data
   • Sharded testing
     • Running tests directly
     • Running tests via CMake
3. Coding style
4. Submitting patches
   • Login cookie
   • Contributor agreement
   • Testing your code
   • Commit message hook
   • Upload your change
   • Incorporating Reviewer Comments
   • Submitting your change
   • Viewing change status
5. Support
6. Bug reports

1. CMake version 3.6 or higher.
2. Git.
3. Perl.
4. For x86 targets,yasm, which is preferred, or arecent version ofnasm. If you download yasm withthe intention to work with Visual Studio, please download win32.exe orwin64.exe and rename it into yasm.exe. DO NOT download or use vsyasm.exe.
5. Building the documentation requiresdoxygen version 1.8.10 or newer.
6. Building the unit tests requiresPython.
7. Emscripten builds require the portableEMSDK.

The AV1 library source code is stored in the Alliance for Open Media Gitrepository:

    $ git clone https://aomedia.googlesource.com/aom    # By default, the above command stores the source in the aom directory:    $ cd aom

CMake replaces the configure step typical of many projects. Running CMake willproduce configuration and build files for the currently selected CMakegenerator. For most systems the default generator is Unix Makefiles. The basicform of a makefile build is the following:

    $ cmake path/to/aom    $ make

The above will generate a makefile build that produces the AV1 library andapplications for the current host system after the make step completessuccessfully. The compiler chosen varies by host platform, but a general ruleapplies: On systems where cc and c++ are present in $PATH at the time CMake isrun the generated build will use cc and c++ by default.

The AV1 codec library has a great many configuration options. These come in twovarieties:

1. Build system configuration options. These have the formENABLE_FEATURE.
2. AV1 codec configuration options. These have the formCONFIG_FEATURE.

Both types of options are set at the time CMake is run. The following exampleenables ccache and disables the AV1 encoder:

    $ cmake path/to/aom -DENABLE_CCACHE=1 -DCONFIG_AV1_ENCODER=0    $ make

The available configuration options are too numerous to list here. Build systemconfiguration options can be found at the top of the CMakeLists.txt file foundin the root of the AV1 repository, and AV1 codec configuration options cancurrently be found in the filebuild/cmake/aom_config_defaults.cmake.

A dylib (shared object) build of the AV1 codec library can be enabled via theCMake built in variableBUILD_SHARED_LIBS:

    $ cmake path/to/aom -DBUILD_SHARED_LIBS=1    $ make

This is currently only supported on non-Windows targets.

Depending on the generator used there are multiple ways of going aboutdebugging AV1 components. For single configuration generators like the UnixMakefiles generator, settingCMAKE_BUILD_TYPE to Debug is sufficient:

    $ cmake path/to/aom -DCMAKE_BUILD_TYPE=Debug

For Xcode, mainly because configuration controls for Xcode builds are buried twoconfiguration windows deep and must be set for each subproject within the XcodeIDE individually,CMAKE_CONFIGURATION_TYPES should be set to Debug:

    $ cmake path/to/aom -G Xcode -DCMAKE_CONFIGURATION_TYPES=Debug

For Visual Studio the in-IDE configuration controls should be used. Simply setthe IDE project configuration to Debug to allow for stepping through the code.

In addition to the above it can sometimes be useful to debug only C and C++code. To disable all assembly code and intrinsics setAOM_TARGET_CPU togeneric at generation time:

    $ cmake path/to/aom -DAOM_TARGET_CPU=generic

For the purposes of building the AV1 codec and applications and relative to thescope of this guide, all builds for architectures differing from the native hostarchitecture will be considered cross compiles. The AV1 CMake build handlescross compiling via the use of toolchain files included in the AV1 repository.The toolchain files available at the time of this writing are:

• arm64-ios.cmake
• arm64-linux-gcc.cmake
• arm64-mingw-gcc.cmake
• armv7-ios.cmake
• armv7-linux-gcc.cmake
• armv7-mingw-gcc.cmake
• armv7s-ios.cmake
• mips32-linux-gcc.cmake
• mips64-linux-gcc.cmake
• x86-ios-simulator.cmake
• x86-linux.cmake
• x86-macos.cmake
• x86-mingw-gcc.cmake
• x86_64-ios-simulator.cmake
• x86_64-mingw-gcc.cmake

The following example demonstrates use of the x86-macos.cmake toolchain file ona x86_64 MacOS host:

    $ cmake path/to/aom \      -DCMAKE_TOOLCHAIN_FILE=path/to/aom/build/cmake/toolchains/x86-macos.cmake    $ make

To build for an unlisted target creation of a new toolchain file is the bestsolution. The existing toolchain files can be used a starting point for a newtoolchain file since each one exposes the basic requirements for toolchain filesas used in the AV1 codec build.

As a temporary work around an unoptimized AV1 configuration that builds only Cand C++ sources can be produced using the following commands:

    $ cmake path/to/aom -DAOM_TARGET_CPU=generic    $ make

In addition to the above it's important to note that the toolchain filessuffixed with gcc behave differently than the others. These toolchain filesattempt to obey the $CROSS environment variable.

Sanitizer integration is built-in to the CMake build system. To enable asanitizer, add-DSANITIZE=<type> to the CMake command line. For example, toenable address sanitizer:

    $ cmake path/to/aom -DSANITIZE=address    $ make

Sanitizers available vary by platform, target, and compiler. Consult yourcompiler documentation to determine which, if any, are available.

Building the AV1 codec library in Microsoft Visual Studio is supported. VisualStudio 2017 (15.0) or later is required. The following example demonstratesgenerating projects and a solution for the Microsoft IDE:

    # This does not require a bash shell; Command Prompt (cmd.exe) is fine.    # This assumes the build host is a Windows x64 computer.    # To build with Visual Studio 2019 for the x64 target:    $ cmake path/to/aom -G "Visual Studio 16 2019"    $ cmake --build .    # To build with Visual Studio 2019 for the 32-bit x86 target:    $ cmake path/to/aom -G "Visual Studio 16 2019" -A Win32    $ cmake --build .    # To build with Visual Studio 2017 for the x64 target:    $ cmake path/to/aom -G "Visual Studio 15 2017" -T host=x64 -A x64    $ cmake --build .    # To build with Visual Studio 2017 for the 32-bit x86 target:    $ cmake path/to/aom -G "Visual Studio 15 2017" -T host=x64    $ cmake --build .

NOTE: The build system targets Windows 7 or later by compiling files with-D_WIN32_WINNT=0x0601.

Building the AV1 codec library in Xcode is supported. The following exampledemonstrates generating an Xcode project:

    $ cmake path/to/aom -G Xcode

Building the AV1 codec library with Emscripten is supported. Typically this isused to hook into the AOMAnalyzer GUI application. These instructions focus onusing the inspector with AOMAnalyzer, but all tools can be built withEmscripten.

It is assumed here that you have already downloaded and installed the EMSDK,installed and activated at least one toolchain, and setup your environmentappropriately using the emsdk_env script.

1. DownloadAOMAnalyzer.

2. Configure the build:

    $ cmake path/to/aom \        -DENABLE_CCACHE=1 \        -DAOM_TARGET_CPU=generic \        -DENABLE_DOCS=0 \        -DENABLE_TESTS=0 \        -DCONFIG_ACCOUNTING=1 \        -DCONFIG_INSPECTION=1 \        -DCONFIG_MULTITHREAD=0 \        -DCONFIG_RUNTIME_CPU_DETECT=0 \        -DCONFIG_WEBM_IO=0 \        -DCMAKE_TOOLCHAIN_FILE=path/to/emsdk-portable/.../Emscripten.cmake

3. Build it: run make if that's your generator of choice:

    $ make inspect

4. Run the analyzer:

    # inspect.js is in the examples sub directory of the directory in which you    # executed cmake.    $ path/to/AOMAnalyzer path/to/examples/inspect.js path/to/av1/input/file

Three variables allow for passing of additional flags to the build system.

• AOM_EXTRA_C_FLAGS
• AOM_EXTRA_CXX_FLAGS
• AOM_EXTRA_EXE_LINKER_FLAGS

The build system attempts to ensure the flags passed through the above variablesare passed to tools last in order to allow for override of default behavior.These flags can be used, for example, to enable asserts in a release build:

    $ cmake path/to/aom \        -DCMAKE_BUILD_TYPE=Release \        -DAOM_EXTRA_C_FLAGS=-UNDEBUG \        -DAOM_EXTRA_CXX_FLAGS=-UNDEBUG

After installinglibvmaf.a,you can use it with the encoder:

    $ cmake path/to/aom -DCONFIG_TUNE_VMAF=1

Please note that the default VMAF model("/usr/local/share/model/vmaf_v0.6.1.json")will be used unless you set the following flag when running the encoder:

    # --vmaf-model-path=path/to/model

There are several methods of testing the AV1 codec. All of these methods requirethe presence of the AV1 source code and a working build of the AV1 library andapplications.

The unit tests can be run at build time:

    # Before running the make command the LIBAOM_TEST_DATA_PATH environment    # variable should be set to avoid downloading the test files to the    # cmake build configuration directory.    $ cmake path/to/aom    # Note: The AV1 CMake build creates many test targets. Running make    # with multiple jobs will speed up the test run significantly.    $ make runtests

The example tests require a bash shell and can be run in the following manner:

    # See the note above about LIBAOM_TEST_DATA_PATH above.    $ cmake path/to/aom    $ make    # It's best to build the testdata target using many make jobs.    # Running it like this will verify and download (if necessary)    # one at a time, which takes a while.    $ make testdata    $ path/to/aom/test/examples.sh --bin-path examples

When making a change to the encoder run encoder tests to confirm that yourchange has a positive or negligible impact on encode quality. When running thesetests the build configuration should be changed to enable internal encoderstatistics:

    $ cmake path/to/aom -DCONFIG_INTERNAL_STATS=1    $ make

The repository contains scripts intended to make running these tests as simpleas possible. The following example demonstrates creating a set of baseline clipsfor comparison to results produced after making your change to libaom:

    # This will encode all Y4M files in the current directory using the    # settings specified to create the encoder baseline statistical data:    $ cd path/to/test/inputs    # This command line assumes that run_encodes.sh, its helper script    # best_encode.sh, and the aomenc you intend to test are all within a    # directory in your PATH.    $ run_encodes.sh 200 500 50 baseline

After making your change and creating the baseline clips, you'll need to runencodes that include your change(s) to confirm that things are working asintended:

    # This will encode all Y4M files in the current directory using the    # settings specified to create the statistical data for your change:    $ cd path/to/test/inputs    # This command line assumes that run_encodes.sh, its helper script    # best_encode.sh, and the aomenc you intend to test are all within a    # directory in your PATH.    $ run_encodes.sh 200 500 50 mytweak

After creating both data sets you can usetest/visual_metrics.py to generate areport that can be viewed in a web browser:

    $ visual_metrics.py metrics_template.html "*stt" baseline mytweak \      > mytweak.html

You can view the report by opening mytweak.html in a web browser.

By default the generated projects files created by CMake will not include theruntests and testdata rules when generating for IDEs like Microsoft VisualStudio and Xcode. This is done to avoid intolerably long build cycles in theIDEs-- IDE behavior is to build all targets when selecting the build projectoptions in MSVS and Xcode. To enable the test rules in IDEs theENABLE_IDE_TEST_HOSTING variable must be enabled at CMake generation time:

    # This example uses Xcode. To get a list of the generators    # available, run cmake with the -G argument missing its    # value.    $ cmake path/to/aom -DENABLE_IDE_TEST_HOSTING=1 -G Xcode

The fastest and easiest way to obtain the test data is to use CMake to generatea build using the Unix Makefiles generator, and then to build only the testdatarule. By default the test files will be downloaded to the current directory. TheLIBAOM_TEST_DATA_PATH environment variable can be used to set acustom one.

    $ cmake path/to/aom -G "Unix Makefiles"    # 28 is used because there are 28 test files as of this writing.    $ make -j28 testdata

The above make command will only download and verify the test data.

First, add the new test data file to theaom-test-data bucket of theaomedia-testing project on Google Cloud Platform. You may need to ask someonewith the necessary access permissions to do this for you.

NOTE: When a new test data file is added to theaom-test-data bucket, its"Public access" is initially "Not public". We need to change its"Public access" to "Public" by using the followinggsutil command:

    $ gsutil acl ch -g all:R gs://aom-test-data/test-data-file-name

This command grants theAllUsers group READ access to the file named"test-data-file-name" in theaom-test-data bucket.

Once the new test data file has been added toaom-test-data, create a CL toadd the name of the new test data file totest/test_data_util.cmake and addthe SHA1 checksum of the new test data file totest/test-data.sha1. (The SHA1checksum of a file can be calculated by running thesha1sum command on thefile.)

The test data mentioned above is strictly intended for unit testing.

Additional input data for testing the encoder can be obtained from:https://media.xiph.org/video/derf/

The AV1 codec library unit tests are built upon gtest which supports sharding oftest jobs. Sharded test runs can be achieved in a couple of ways.

   # Set the environment variable GTEST_TOTAL_SHARDS to control the number of   # shards.   $ export GTEST_TOTAL_SHARDS=10   # (GTEST shard indexing is 0 based).   $ seq 0 $(( $GTEST_TOTAL_SHARDS - 1 )) \       | xargs -n 1 -P 0 -I{} env GTEST_SHARD_INDEX={} ./test_libaom

To create a test shard for each CPU core available on the current system setGTEST_TOTAL_SHARDS to the number of CPU cores on your system minus one.

    # For IDE based builds, ENABLE_IDE_TEST_HOSTING must be enabled. See    # the IDE hosted tests section above for more information. If the IDE    # supports building targets concurrently tests will be sharded by default.    # For make and ninja builds the -j parameter controls the number of shards    # at test run time. This example will run the tests using 10 shards via    # make.    $ make -j10 runtests

The maximum number of test targets that can run concurrently is determined bythe number of CPUs on the system where the build is configured as detected byCMake. A system with 24 cores can run 24 test shards using a value of 24 withthe-j parameter. When CMake is unable to detect the number of cores 10 shardsis the default maximum value.

We are using the Google C Coding Style defined by theGoogle C++ Style Guide.

The coding style used by this project is enforced with clang-format using theconfiguration contained in the.clang-formatfile in the root of the repository.

You can download clang-format using your system's package manager, or directlyfromllvm.org. You can also view thedocumentation on llvm.org.Output from clang-format varies by clang-format version, for best results yourversion should match the one used on Jenkins. You can find the clang-formatversion by reading the comment in the.clang-format file linked above.

Before pushing changes for review you can format your code with:

    # Apply clang-format to modified .c, .h and .cc files    $ clang-format -i --style=file \      $(git diff --name-only --diff-filter=ACMR '*.[hc]' '*.cc')

Check the .clang-format file for the version used to generate it if there is anydifference between your local formatting and the review system.

Some Git installations have clang-format integration. Here are some examples:

    # Apply clang-format to all staged changes:    $ git clang-format    # Clang format all staged and unstaged changes:    $ git clang-format -f    # Clang format all staged and unstaged changes interactively:    $ git clang-format -f -p

We manage the submission of patches using theGerrit code review tool. This toolimplements a workflow on top of the Git version control system to ensure thatall changes get peer reviewed and tested prior to their distribution.

Browse toAOMedia Git index and login withyour account (Gmail credentials, for example). Next, follow theGenerate Password Password link at the top of the page. You’ll be giveninstructions for creating a cookie to use with our Git repos.

You will be required to execute acontributor agreement to ensure that the AOMediaProject has the right to distribute your changes.

The testing basics are covered in thetesting sectionabove.

In addition to the local tests, many more (e.g. asan, tsan, valgrind) will runthrough Jenkins instances upon upload to gerrit.

Gerrit requires that each submission include a unique Change-Id. You can assignone manually using git commit --amend, but it’s easier to automate it with thecommit-msg hook provided by Gerrit.

Copy commit-msg to the.git/hooks directory of your local repo. Here's anexample:

    $ curl -Lo aom/.git/hooks/commit-msg https://chromium-review.googlesource.com/tools/hooks/commit-msg    # Next, ensure that the downloaded commit-msg script is executable:    $ chmod u+x aom/.git/hooks/commit-msg

See the Gerritdocumentationfor more information.

The command line to upload your patch looks like this:

    $ git push https://aomedia-review.googlesource.com/aom HEAD:refs/for/master

If you previously uploaded a change to Gerrit and the Approver has asked forchanges, follow these steps:

1. Edit the files to make the changes the reviewer has requested.
2. Recommit your edits using the --amend flag, for example:

   $ git commit -a --amend

3. Use the same git push command as above to upload to Gerrit again for anotherreview cycle.

In general, you should not rebase your changes when doing updates in response toreview. Doing so can make it harder to follow the evolution of your change inthe diff view.

Once your change has been Approved and Verified, you can “submit” it through theGerrit UI. This will usually automatically rebase your change onto the branchspecified.

Sometimes this can’t be done automatically. If you run into this problem, youmust rebase your changes manually:

    $ git fetch    $ git rebase origin/branchname

If there are any conflicts, resolve them as you normally would with Git. Whenyou’re done, reupload your change.

To check the status of a change that you uploaded, openGerrit, sign in, and click My >Changes.

This library is an open source project supported by its community. Pleaseplease emailaomediacodec@jointdevelopment.kavi.com for help.

Bug reports can be filed in the Alliance for Open Mediaissue tracker.