Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

chore: tune postgres CI tests#17756

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
hugodutka merged 1 commit intomainfromhugodutka/ci-windows-macos-tuning
May 12, 2025
Merged
Show file tree
Hide file tree
Changes fromall commits
Commits
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion.github/actions/setup-go/action.yaml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -26,13 +26,15 @@ runs:
export GOCACHE_DIR="$RUNNER_TEMP""\go-cache"
export GOMODCACHE_DIR="$RUNNER_TEMP""\go-mod-cache"
export GOPATH_DIR="$RUNNER_TEMP""\go-path"
export GOTMP_DIR="$RUNNER_TEMP""\go-tmp"
mkdir -p "$GOCACHE_DIR"
mkdir -p "$GOMODCACHE_DIR"
mkdir -p "$GOPATH_DIR"
mkdir -p "$GOTMP_DIR"
go env -w GOCACHE="$GOCACHE_DIR"
go env -w GOMODCACHE="$GOMODCACHE_DIR"
go env -w GOPATH="$GOPATH_DIR"

go env -w GOTMPDIR="$GOTMP_DIR"
- name: Setup Go
uses: actions/setup-go@0a12ed9d6a96ab950c8f026ed9f722fe0da7ef32 # v5.0.2
with:
Expand Down
2 changes: 1 addition & 1 deletion.github/workflows/ci.yaml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -454,7 +454,7 @@ jobs:
api-key: ${{ secrets.DATADOG_API_KEY }}

test-go-pg:
runs-on: ${{ matrix.os == 'ubuntu-latest' && github.repository_owner == 'coder' && 'depot-ubuntu-22.04-4' || matrix.os }}
runs-on: ${{ matrix.os == 'ubuntu-latest' && github.repository_owner == 'coder' && 'depot-ubuntu-22.04-8' || matrix.os }}
needs: changes
if: needs.changes.outputs.go == 'true' || needs.changes.outputs.ci == 'true' || github.ref == 'refs/heads/main'
# This timeout must be greater than the timeout set by `go test` in
Expand Down
75 changes: 58 additions & 17 deletions.github/workflows/nightly-gauntlet.yaml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -12,8 +12,9 @@ permissions:

jobs:
test-go-pg:
runs-on: ${{ matrix.os == 'macos-latest' && github.repository_owner == 'coder' && 'depot-macos-latest' || matrix.os == 'windows-2022' && github.repository_owner == 'coder' && 'windows-latest-16-cores' || matrix.os }}
if: github.ref == 'refs/heads/main'
# make sure to adjust NUM_PARALLEL_PACKAGES and NUM_PARALLEL_TESTS below
# when changing runner sizes
runs-on: ${{ matrix.os == 'macos-latest' && github.repository_owner == 'coder' && 'depot-macos-latest' || matrix.os == 'windows-2022' && github.repository_owner == 'coder' && 'depot-windows-2022-16' || matrix.os }}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Can you add a comment here about the constants you have below for parallelism? It'd be easy for these to go out of sync.

hugodutka reacted with thumbs up emoji
# This timeout must be greater than the timeout set by `go test` in
# `make test-postgres` to ensure we receive a trace of running
# goroutines. Setting this to the timeout +5m should work quite well
Expand All@@ -31,22 +32,39 @@ jobs:
with:
egress-policy: audit

# macOS indexes all new files in the background. Our Postgres tests
# create and destroy thousands of databases on disk, and Spotlight
# tries to index all of them, seriously slowing down the tests.
- name: Disable Spotlight Indexing
if: runner.os == 'macOS'
run: |
sudo mdutil -a -i off
sudo mdutil -X /
sudo launchctl bootout system /System/Library/LaunchDaemons/com.apple.metadata.mds.plist

# Set up RAM disks to speed up the rest of the job. This action is in
# a separate repository to allow its use before actions/checkout.
- name: Setup RAM Disks
if: runner.os == 'Windows'
uses: coder/setup-ramdisk-action@79dacfe70c47ad6d6c0dd7f45412368802641439

- name: Checkout
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
fetch-depth: 1

- name: Setup Go
uses: ./.github/actions/setup-go
with:
# Runners have Go baked-in and Go will automatically
# download the toolchain configured in go.mod, so we don't
# need to reinstall it. It's faster on Windows runners.
use-preinstalled-go: ${{ runner.os == 'Windows' }}
use-temp-cache-dirs: ${{ runner.os == 'Windows' }}

- name: Setup Terraform
uses: ./.github/actions/setup-tf

# Sets up the ImDisk toolkit for Windows and creates a RAM disk on drive R:.
- name: Setup ImDisk
if: runner.os == 'Windows'
uses: ./.github/actions/setup-imdisk

- name: Test with PostgreSQL Database
env:
POSTGRES_VERSION: "13"
Expand All@@ -55,6 +73,19 @@ jobs:
LC_ALL: "en_US.UTF-8"
shell: bash
run: |
if [ "${{ runner.os }}" == "Windows" ]; then
# Create a temp dir on the R: ramdisk drive for Windows. The default
# C: drive is extremely slow: https://github.com/actions/runner-images/issues/8755
mkdir -p "R:/temp/embedded-pg"
go run scripts/embedded-pg/main.go -path "R:/temp/embedded-pg"
fi
if [ "${{ runner.os }}" == "macOS" ]; then
# Postgres runs faster on a ramdisk on macOS too
mkdir -p /tmp/tmpfs
sudo mount_tmpfs -o noowners -s 8g /tmp/tmpfs
go run scripts/embedded-pg/main.go -path /tmp/tmpfs/embedded-pg
fi

# if macOS, install google-chrome for scaletests
# As another concern, should we really have this kind of external dependency
# requirement on standard CI?
Expand All@@ -72,19 +103,29 @@ jobs:
touch ~/.bash_profile && echo "export BASH_SILENCE_DEPRECATION_WARNING=1" >> ~/.bash_profile
fi

# Golang's default for these 2 variables is the number of logical CPUs.
# Our Windows and Linux runners have 16 cores, so they match up there.
NUM_PARALLEL_PACKAGES=16
NUM_PARALLEL_TESTS=16
if [ "${{ runner.os }}" == "Windows" ]; then
# Create a temp dir on the R: ramdisk drive for Windows. The default
# C: drive is extremely slow: https://github.com/actions/runner-images/issues/8755
mkdir -p "R:/temp/embedded-pg"
go run scripts/embedded-pg/main.go -path "R:/temp/embedded-pg"
else
go run scripts/embedded-pg/main.go
# On Windows Postgres chokes up when we have 16x16=256 tests
# running in parallel, and dbtestutil.NewDB starts to take more than
# 10s to complete sometimes causing test timeouts. With 16x8=128 tests
# Postgres tends not to choke.
NUM_PARALLEL_PACKAGES=8
fi
if [ "${{ runner.os }}" == "macOS" ]; then
# Our macOS runners have 8 cores. We leave NUM_PARALLEL_TESTS at 16
# because the tests complete faster and Postgres doesn't choke. It seems
# that macOS's tmpfs is faster than the one on Windows.
NUM_PARALLEL_PACKAGES=8
fi

# Reduce test parallelism, mirroring what we do for race tests.
# We'd been encountering issues with timing related flakes, and
# this seems to help.
DB=ci gotestsum --format standard-quiet -- -v -short -count=1 -parallel 4 -p 4 ./...
# We rerun failing tests to counteract flakiness coming from Postgres
# choking on macOS and Windows sometimes.
DB=ci gotestsum --rerun-fails=2 --rerun-fails-max-failures=1000 \
--format standard-quiet --packages "./..." \
-- -v -p $NUM_PARALLEL_PACKAGES -parallel=$NUM_PARALLEL_TESTS -count=1

- name: Upload test stats to Datadog
timeout-minutes: 1
Expand Down
Loading
[8]ページ先頭
©2009-2025 Movatter.jp