Architecture overview

Benchmark hardwares

Benchmark results format

// The list of all benchmark metrics[  {    // Information about the benchmark    benchmark: Tuple(      name,  // Required. The name of the benchmark      mode,  // Training or inference      dtype,  // The dtype used by the benchmark      extra_info: {},  // Any additional information about the benchmark    ),    // Information about the model or the test    model: Tuple (      name,  // Required. The model or the test name      type,  // Additional information, for example is this a HF model or a micro-benchmark custom layer      backend,  // Any delegation backend used here, i.e. XNNPACK      origins,  // Tell us where this is from, i.e. HF      extra_info: {},  // Any additional information about the model or the test    ),    // Information about the benchmark result    metric: Tuple(      name,  // Required. The name of the metric. It's a good practice to include its unit here too, i.e. compilation_time(ms)      benchmark_values,  // Float. Required. The metric values. It's a list here because a benchmark is usually run multiple times      target_value,  // Float. The optional target value used to indicate if there is a regression      extra_info: {},  // Any additional information about the benchmark result    ),    // Optional information about any inputs used by the benchmark    inputs: {      name: Tuple(        dtype,  // The dtype of the input        extra_info: {},  // Any additional information about the input      )    },  },  ...]

Upload the benchmark results

GitHub CI

A sample job on AWS self-hosted runners

name: A sample benchmark job that runs on all main commitson:  push:    - mainjobs:  benchmark:    runs-on: linux.2xlarge    steps:      - uses: actions/checkout@v3      - name: Run your own benchmark logic        shell: bash        run: |          set -eux          # Run your benchmark script and write the result to benchmark-results.json whose format is defined in the previous section          python run_my_benchmark_script.py > ${{ runner.temp }}/benchmark-results/benchmark-results.json          # It's also ok to write the results into multiple JSON files, for example          python run_my_benchmark_script.py --output-dir ${{ runner.temp }}/benchmark-results     - name: Upload the benchmark results to OSS benchmark database for the dashboard       uses: pytorch/test-infra/.github/actions/upload-benchmark-results@main       with:         benchmark-results-dir: ${{ runner.temp }}/benchmark-results         dry-run: false         github-token: ${{ secrets.GITHUB_TOKEN }}

A sample job on non-AWS runners

name: A sample benchmark job that runs on all main commitson:  push:    - mainjobs:  benchmark:    runs-on: linux.rocm.gpu.2  // An example non-AWS runner    environment: upload-benchmark-results  // The environment has write access S3 to upload the results    permissions:      id-token: write      contents: read    steps:      - uses: actions/checkout@v3      - name: Authenticate with AWS        uses: aws-actions/configure-aws-credentials@v4        with:          role-to-assume: arn:aws:iam::308535385114:role/gha_workflow_upload-benchmark-results          # The max duration enforced by the server side          role-duration-seconds: 18000          aws-region: us-east-1      - name: Run your own benchmark logic        shell: bash        run: |          # Run your benchmark script and write the result to benchmark-results.json whose format is defined in the previous section          python run_my_benchmark_script.py > ${{ runner.temp }}/benchmark-results/benchmark-results.json          # It's also ok to write the results into multiple JSON files, for example          python run_my_benchmark_script.py --output-dir ${{ runner.temp }}/benchmark-results     - name: Upload the benchmark results to OSS benchmark database for the dashboard       uses: pytorch/test-infra/.github/actions/upload-benchmark-results@main       with:         benchmark-results-dir: ${{ runner.temp }}/benchmark-results         dry-run: false         github-token: ${{ secrets.GITHUB_TOKEN }}

Upload API

export UPLOADER_USERNAME=<REDACT>export UPLOADER_PASSWORD=<REDACT># Any strings work as long as they are consistentexport DEVICE_NAME=cudaexport DEVICE_TYPE=$(nvidia-smi -i 0 --query-gpu=name --format=csv,noheader | awk '{print $2}')git clone https://github.com/pytorch/pytorch-integration-testingcd pytorch-integration-testing/.github/scripts# The script dependenciespip install -r requirements.txt# --repo is where the repo is checkout and built# --benchmark-name is an unique string that is used to identify the benchmark# --benchmark-results is where the JSON benchmark result files are kept# --dry-run is to prepare everything except writing the results to S3python upload_benchmark_results.py \  --repo pytorch \  --benchmark-name "My PyTorch benchmark" \  --benchmark-results benchmark-results-dir \  --device-name "${DEVICE_NAME}" \  --device-type "${DEVICE_TYPE}" \  --dry-run

# Use PyTorch 2.7 releasepython upload_benchmark_results.py \  --repo-name "pytorch/pytorch" \  --head-branch "release/2.7" \  --head-sha "e2d141dbde55c2a4370fac5165b0561b6af4798b" \  --benchmark-name "My PyTorch benchmark" \  --benchmark-results benchmark-results-dir \  --device-name "${DEVICE_NAME}" \  --device-type "${DEVICE_TYPE}" \  --dry-run# Use PyTorch nightlypython upload_benchmark_results.py \  --repo-name "pytorch/pytorch" \  --head-branch "nightly" \  --head-sha "be2ad70cfa1360da5c23a04ff6ca3480fa02f278" \  --benchmark-name "My PyTorch benchmark" \  --benchmark-results benchmark-results-dir \  --device-name "${DEVICE_NAME}" \  --device-type "${DEVICE_TYPE}" \  --dry-run

# This path is an example - any path under the v3 directory is acceptable. If the path already exists, the API will not overwrite its3_path = f"v3/{repo_name}/{head_branch}/{head_sha}/{device_name}/{device_type}/benchmark_results.json"payload = {    "username": UPLOADER_USERNAME,    "password": UPLOADER_PASSWORD,    "s3_path": s3_path,    "content": json.dumps(benchmark_results),}headers = {"content-type": "application/json"}requests.post(    # One current limitation of the API is that AWS limits the maximum size of the JSON to be less than **6MB**    "https://kvvka55vt7t2dzl6qlxys72kra0xtirv.lambda-url.us-east-1.on.aws",    json=payload,    headers=headers)

Explore benchmark results

HUD

Flambeau PyTorch CI agent

Query API

import osimport jsonimport requestsusername = os.environ.get("CLICKHOUSE_API_USERNAME")password = os.environ.get("CLICKHOUSE_API_PASSWORD")params = {    "format": "JSONEachRow",    "queryVariables": {               # REQUIRED: The repo name in org/repo format        "repo": "pytorch/pytorch",        # REQUIRED: The name of the benchmark        "benchmark": "TorchInductor",        # REQUIRED: YYYT-MM-DDThh:mm:ss        "startTime": "2025-06-06T00:00:00",        # REQUIRED: YYYT-MM-DDThh:mm:ss        "stopTime": "2025-06-13T00:00:00",        # OPTIONAL: Only query benchmark results for these models.  Leaving this as an empty array [] will fetch all of them        "models": ["BERT_pytorch"],        # OPTIONAL: Only fetch these metrics.  Leaving this as an empty array [] will fetch all of them        "metrics": ["speedup"],        # OPTIONAL: Filter the benchmark results by device, i.e. cuda, and arch, i.e. H100, leave them empty to get all devices        "device": "",        "arch": "",        # OPTIONAL: Use this when you only care about the benchmark results from a specific branch and commit        "branch": "",        "commit": "",    }}api_url = "https://queries.clickhouse.cloud/run/84649f4e-52c4-4cf9-bd6e-0a105ea145c8"r = requests.post(api_url, json=params, auth=(username, password))with open("benchmark_results.txt", "w") as f:    print(r.text, file=f)

"pytorch-labs/tritonbench": "compile_time""pytorch-labs/tritonbench": "nightly""pytorch/ao": "TorchAO benchmark""pytorch/ao": "micro-benchmark api""pytorch/benchmark": "TorchInductor""pytorch/executorch": "ExecuTorch""pytorch/pytorch": "PyTorch gpt-fast benchmark""pytorch/pytorch": "PyTorch operator benchmark""pytorch/pytorch": "TorchCache Benchmark""pytorch/pytorch": "TorchInductor""pytorch/pytorch": "cache_benchmarks""pytorch/pytorch": "pr_time_benchmarks""vllm-project/vllm": "vLLM benchmark"

Benchmark database

select    head_branch,    head_sha,    benchmark,    model.name as model,    metric.name as name,    arrayAvg(metric.benchmark_values) as valuefrom    oss_ci_benchmark_v3where    tupleElement(benchmark, 'name') = 'TorchAO benchmark'    and oss_ci_benchmark_v3.timestamp < 1733870813    and oss_ci_benchmark_v3.timestamp > 1733784413

Benchmark UI Configuration

This section describes how to onboard a benchmark into the PyTorch Benchmark UI, from the minimum required setup to advanced customization of APIs and UI components.

Onboarding a Benchmark (Minimum Setup)

Required Steps

Example

Advanced: Customizing Benchmark Results

Customized API Responses

Relevant Fetchers

Setting Up a Custom QueryBuilder

Customized UI Rendering

Supported UI Customization Options

Pages

Components

Utilities

Regression notification