Scale Testing

This folder contains CLI commands, Terraform code, and scripts to aid in performing load tests of Coder.At a high level, it performs the following steps:

• Using the Terraform code in./terraform, stands up a preconfigured Google Cloud environmentconsisting of a VPC, GKE Cluster, and CloudSQL instance.

  Note: You must have an existing Google Cloud project available.

• Creates a dedicated namespace for Coder and installs Coder using the Helm chart in this namespace.
• Configures the Coder deployment with random credentials and a predefined Kubernetes template.

  Note: These credentials are stored in${PROJECT_ROOT}/scaletest/.coderv2/coder.env.

• Creates a number of workspaces and waits for them to all start successfully. These workspacesare ephemeral and do not contain any persistent resources.
• Waits for 10 minutes to allow things to settle and establish a baseline.
• Generates web terminal traffic to all workspaces for 30 minutes.
• Directly after traffic generation, captures goroutine and heap snapshots of the Coder deployment.
• Tears down all resources (unless--skip-cleanup is specified).

Usage

The main entrypoint is thescaletest.sh script.

$ scaletest.sh --helpUsage: scaletest.sh --name <name> --project <project> --num-workspaces <num-workspaces> --scenario <scenario> [--dry-run] [--skip-cleanup]

Required arguments:

• --name: Name for the loadtest. This is added as a prefix to resources created by Terraform (e.g.joe-big-loadtest).
• --project: Google Cloud project in which to create the resources (example:my-loadtest-project).
• --num-workspaces: Number of workspaces to create (example:10).
• --scenario: Deployment scenario to use (example:small). Seeterraform/scenario-*.tfvars.

Note: In order to capture Prometheus metrics, you must define the environment variablesSCALETEST_PROMETHEUS_REMOTE_WRITE_USER andSCALETEST_PROMETHEUS_REMOTE_WRITE_PASSWORD.

Optional arguments:

• --dry-run: Do not perform any action and instead print what would be executed.
• --skip-cleanup: Do not perform any cleanup. You will be responsible for deleting any resources this creates.

Environment Variables

All of the above arguments may be specified as environment variables. Consult the script for details.

Prometheus Metrics

To capture Prometheus metrics from the loadtest, two environment variables are required:

• SCALETEST_PROMETHEUS_REMOTE_WRITE_USER
• SCALETEST_PROMETHEUS_REMOTE_WRITE_PASSWORD

Enterprise License

To add an Enterprise license, set theSCALETEST_CODER_LICENSE environment variable to the JWT string

Scenarios

A scenario defines a number of variables that override the default Terraform variables.A number of existing scenarios are provided inscaletest/terraform/scenario-*.tfvars.

For example,scenario-small.tfvars includes the following variable definitions:

nodepool_machine_type_coder      = "t2d-standard-2"nodepool_machine_type_workspaces = "t2d-standard-2"coder_cpu                        = "1000m" # Leaving 1 CPU for system workloadscoder_mem                        = "4Gi"   # Leaving 4GB for system workloads

To create your own scenario, simply add a new fileterraform/scenario-$SCENARIO_NAME.tfvars.In this file, override variables as required, consultingvars.tf as needed.You can then use this scenario by specifying--scenario $SCENARIO_NAME.For example, if your scenario file were namedscenario-big-whopper2x.tfvars, you would specify--scenario=big-whopper2x.

Utility scripts

A number of utility scripts are provided inlib, and are used byscaletest.sh:

• coder_shim.sh: a convenience script to run thecoder binary with a predefined config root.This is intended to allow running Coder CLI commands against the loadtest cluster withoutmodifying a user's existing Coder CLI configuration.
• coder_init.sh: Performs first-time user setup of an existing Coder instance, generatinga random password for the admin user. The admin user is namedadmin@coder.com by default.Credentials are written toscaletest/.coderv2/coder.env.
• coder_workspacetraffic.sh: Runs traffic generation against the loadtest cluster and createsa monitoring manifest for the traffic generation pod. This pod will restart automaticallyafter the traffic generation has completed.

Grafana Dashboard

A sample Grafana dashboard is provided inscaletest_dashboard.json. This dashboard is intendedto be imported into an existing Grafana instance. It provides a number of useful metrics:

• Control Plane Resources: CPU, memory, and network usage for the Coder deployment, as well as the number of pod restarts.
• Database: Rows inserted/updated/deleted/returned, active connections, and transactions per second. Fine-grainedsqlQuerier metrics are provided for Coder's database as well, broken down my query method.
• HTTP requests: Number of HTTP requests per second, broken down by status code and path.
• Workspace Resources: CPU, memory, and network usage for all workspaces.
• Workspace Agents: Workspace agent network usage, connection latency, and number of active connections.
• Workspace Traffic: Statistics related to workspace traffic generation.
• Internals: Provisioner job timings, concurrency, workspace builds, and AuthZ duration.

A subset of these metrics may be useful for a production deployment, but some are only usefulfor load testing.

Note: in particular,sqlQuerier metrics produce a large number of time series and may causeincreased charges in your metrics provider.