- Notifications
You must be signed in to change notification settings - Fork1k
docs: create WIP 10k scale doc#20213
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
base:main
Are you sure you want to change the base?
Changes fromall commits
File filter
Filter by extension
Conversations
Uh oh!
There was an error while loading.Please reload this page.
Jump to
Uh oh!
There was an error while loading.Please reload this page.
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
@@ -0,0 +1,96 @@ | ||||||||||||||||||||||||||
# Reference Architecture: up to 10,000 users | ||||||||||||||||||||||||||
> [!CAUTION] | ||||||||||||||||||||||||||
> This page is a work in progress. | ||||||||||||||||||||||||||
> | ||||||||||||||||||||||||||
> We are actively testing different load profiles for this user target and will be updating | ||||||||||||||||||||||||||
> recommendations. Use these recommendations as a starting point, but monitor your cluster resource | ||||||||||||||||||||||||||
> utilization and adjust. | ||||||||||||||||||||||||||
The 10,000 users architecture targets large-scale enterprises with globally-distributed development | ||||||||||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others.Learn more. I think it'd be nice to detail out the numbers we are expecting when we say 10k users - 600 concurrent builds, 6000 concurrent workspaces, etc. | ||||||||||||||||||||||||||
teams in multiple geographic regions. | ||||||||||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others.Learn more. Suggested change
either | ||||||||||||||||||||||||||
**Geographic Distribution**: For these tests we deploy on 3 Cloud-managed Kubernetes clusters in | ||||||||||||||||||||||||||
different regions. | ||||||||||||||||||||||||||
1. USA - Primary (also contains the PostgreSQL database deployment). | ||||||||||||||||||||||||||
2. Europe - Workspace Proxies | ||||||||||||||||||||||||||
3. Asia - Workspace Proxies | ||||||||||||||||||||||||||
Comment on lines +13 to +18 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others.Learn more. Suggested change
| ||||||||||||||||||||||||||
**High Availability**: Typically, such scale requires a fully-managed HA | ||||||||||||||||||||||||||
PostgreSQL service, and all Coder observability features enabled for operational | ||||||||||||||||||||||||||
purposes. | ||||||||||||||||||||||||||
**Observability**: Deploy monitoring solutions to gather Prometheus metrics and | ||||||||||||||||||||||||||
visualize them with Grafana to gain detailed insights into infrastructure and | ||||||||||||||||||||||||||
application behavior. This allows operators to respond quickly to incidents and | ||||||||||||||||||||||||||
continuously improve the reliability and performance of the platform. | ||||||||||||||||||||||||||
## Load Types | ||||||||||||||||||||||||||
**Workspace Network Traffic**: 6000 concurrent workspaces (2000 per region), all sending 10 kB/s application traffic | ||||||||||||||||||||||||||
**API**: TBD | ||||||||||||||||||||||||||
## Hardware recommendations | ||||||||||||||||||||||||||
### Coderd | ||||||||||||||||||||||||||
These are deployed in the Primary region only. | ||||||||||||||||||||||||||
| vCPU Limit | Memory Limit | Replicas | GCP Node Pool Machine Type | | ||||||||||||||||||||||||||
|----------------|--------------|----------|----------------------------| | ||||||||||||||||||||||||||
| 4 vCPU (4000m) | 12 GiB | 10 | `c2d-standard-16` | | ||||||||||||||||||||||||||
### Provisioners | ||||||||||||||||||||||||||
These are deployed in each of the 3 regions. | ||||||||||||||||||||||||||
| vCPU Limit | Memory Limit | Replicas | GCP Node Pool Machine Type | | ||||||||||||||||||||||||||
|-----------------|--------------|----------|----------------------------| | ||||||||||||||||||||||||||
| 0.1 vCPU (100m) | 1 GiB | 200 | `c2d-standard-16` | | ||||||||||||||||||||||||||
**Footnotes**: | ||||||||||||||||||||||||||
- Each provisioner handles a single concurrent build, so this configuration imples 200 concurrent | ||||||||||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others.Learn more. Suggested change
| ||||||||||||||||||||||||||
workspace builds per region. | ||||||||||||||||||||||||||
- Provisioners are run as a separate Kubernetes Deployment from Coderd, although they may | ||||||||||||||||||||||||||
share the same node pool. | ||||||||||||||||||||||||||
- Separate provisioners into different namespaces in favor of zero-trust or | ||||||||||||||||||||||||||
multi-cloud deployments. | ||||||||||||||||||||||||||
### Workspace Proxies | ||||||||||||||||||||||||||
These are deployed in the non-Primary regions only. | ||||||||||||||||||||||||||
| vCPU Limit | Memory Limit | Replicas | GCP Node Pool Machine Type | | ||||||||||||||||||||||||||
|----------------|--------------|----------|----------------------------| | ||||||||||||||||||||||||||
| 4 vCPU (4000m) | 12 GiB | 10 | `c2d-standard-16` | | ||||||||||||||||||||||||||
**Footnotes**: | ||||||||||||||||||||||||||
- Our testing implies this is somewhat overspecced for the loads we have tried. We are in process of revising these numbers. | ||||||||||||||||||||||||||
### Workspaces | ||||||||||||||||||||||||||
Thse numbers are for each of the 3 regions. We recommend that you use a separate node pool for user Workspaces. | ||||||||||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others.Learn more. Suggested change
| ||||||||||||||||||||||||||
| Users | Node capacity | Replicas | GCP | AWS | Azure | | ||||||||||||||||||||||||||
|-------------|----------------------|-------------------------------|------------------|--------------|-------------------| | ||||||||||||||||||||||||||
| Up to 3,000 | 8 vCPU, 32 GB memory | 256 nodes, 12 workspaces each | `t2d-standard-8` | `m5.2xlarge` | `Standard_D8s_v3` | | ||||||||||||||||||||||||||
**Footnotes**: | ||||||||||||||||||||||||||
- Assumed that a workspace user needs 2 GB memory to perform | ||||||||||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others.Learn more. How realistic is that 2 GB? That's the request for the workspace pod or both request and limit? | ||||||||||||||||||||||||||
- Maximum number of Kubernetes workspace pods per node: 256 | ||||||||||||||||||||||||||
- As workspace nodes can be distributed between regions, on-premises networks | ||||||||||||||||||||||||||
and cloud areas, consider different namespaces in favor of zero-trust or | ||||||||||||||||||||||||||
multi-cloud deployments. | ||||||||||||||||||||||||||
### Database nodes | ||||||||||||||||||||||||||
We conducted our test using the `db-custom-16-61440` tier on Google Cloud SQL. | ||||||||||||||||||||||||||
**Footnotes**: | ||||||||||||||||||||||||||
- This database tier was only just able to keep up with 600 concurrent builds in our tests. |
Uh oh!
There was an error while loading.Please reload this page.