December 16, 2024

This best practices guide is separated into parts to help you secure aspects ofyour Coder deployment.

Each section briefly introduces each threat model, then suggests steps orconcepts to help implement security improvements such as authentication andencryption.

As with any security guide, the steps and suggestions outlined in this documentare not meant to be exhaustive and do not offer any guarantee.

Coder Server

Coder Server is the main control core of a Coder deployment.

If the Coder Server is compromised in a security incident, it can affect everyother part of your deployment. Even a successful read-only attack against theCoder Server could result in a complete compromise of the Coder deployment ifcredentials are stolen.

User authentication

ConfigureOIDC authentication against yourorganization’s Identity Provider (IdP), such as Okta, to allow single-sign on.

1. Enable and require two-factor authentication in your identity provider.
2. EnableIdP Sync to manage users’ roles andgroups in Coder.
3. Use SCIM to automatically suspend users when they leave the organization.

This allows you to manage user credentials according to your company’s centralrequirements, such as password complexity, 2FA, PassKeys, and others.

Using IdP sync and SCIM means that the central Identity Provider is the sourceof truth, so that when users change roles or leave, their permissions in Coderare automatically up to date.

Encryption in transit

Place Coder behind a TLS-capable reverse-proxy/load balancer and enableStrict Transport Securityso that connections from end users are always encrypted.

EnableTLS on Coder Server andencrypt traffic from the reverse-proxy/load balancer to Coder Server, so thateven if an attacker gains access to your network, they will not be able to snoopon Coder Server traffic.

Encryption at rest

Coder Server persists no state locally. No action is required.

Server logs and audit logs

Capture the logging output of all Coder Server instances and persist them.

Retain all logs for a minimum of thirty days, ideally ninety days. Filter auditlogs (which havemsg: audit_log) and retain them for a minimum of two years(ideally five years) in a secure system that resists tampering.

If a security incident with Coder does occur, audit logs are invaluable indetermining the nature and scope of the impact.

Disable path-based apps

For production deployments, we recommend that you disable path-based apps after you've configured a wildcard access URL.

Path-based apps share the same origin as the Coder API, which can be convenient for trialing Coder,but can expose the deployment to cross-site-scripting (XSS) attacks in production.A malicious workspace could reuse Coder cookies to call the API or interact with other workspaces owned by the same user.

1. Enable sub-domain apps with a wildcard DNS record (like*.coder.example.com)

2. Disable path-based apps:

   coderd server --disable-path-apps# orexport CODER_DISABLE_PATH_APPS=true

By default, Coder mitigates the impact of having path-based apps enabled, but we still recommend disabling it to preventmalicious workspaces accessing other workspaces owned by the same user or performing requests against the Coder API.

If you do keep path-based apps enabled:

• Path-based apps cannot be shared with other users unless you start the Coder server with--dangerous-allow-path-app-sharing.
• Users with the siteowner role cannot use their admin privileges to access path-based apps for workspaces unless theserver is started with--dangerous-allow-path-app-site-owner-access.

PostgreSQL

PostgreSQL is the persistent datastore underlying the entire Coder deployment.If the database is compromised, it may leave every other part of your deploymentvulnerable.

Coder session tokens and API keys are salted and hashed, so a read-onlycompromise of the database is unlikely to allow an attacker to log into Coder.However, the database contains the Terraform state for all workspaces, OIDCtokens, and agent tokens, so it is possible that a read-only attack could enablelateral movement to other systems.

A successful attack that modifies database state could be escalated to a fulltakeover of an owner account in Coder which could lead to a complete compromiseof the Coder deployment.

Authentication

1. Generate a strong, random password for accessing PostgreSQL and store itsecurely.

2. Use environment variables to pass the PostgreSQL URL to Coder.

3. If on Kubernetes, use a Kubernetes secret to set the environment variable.

Encryption in transit

Enable TLS on PostgreSQL and setsslmode=verify-full in yourpostgres URL on Coder Server.This configures Coder Server to only establish TLS connections to PostgreSQL andcheck that the PostgreSQL server’s certificate is valid and matches the expectedhostname.

Encryption at rest

Run PostgreSQL on servers with full disk encryption enabled and configured.

Coder supportsencrypting some particularly sensitive dataincluding OIDC tokens using an encryption key managed independently of thedatabase, so even a user with full administrative privileges on the PostgreSQLserver(s) cannot read the data without the separate key.

If you use this feature:

1. Generate a random encryption key and store it in a central secrets managementsystem like Vault.

2. Inject the secret using an environment variable.

   • If you're using Kubernetes, use a Kubernetes secret rather than includingthe secret directly in the podspec.

3. Follow your organization's policies about key rotation on a fixed schedule.

   • If you suspect the key has been leaked or compromised,rotate the key immediately.

Provisioner daemons

Provisioner daemons are deployed with credentials that give them power to makerequests to cluster/cloud APIs.

If one of those credentials is compromised, the potential severity of thecompromise depends on the permissions granted to the credentials, but willalmost certainly include code execution inside the cluster/cloud since the wholepurpose of Coder is to deploy workspaces in the cluster/cloud that can rundeveloper code.

In addition, provisioner daemons are given access to parameters entered by endusers, which could include sensitive data like credentials for additionalsystems.

External provisioner daemons

When Coder workspaces are deployed into multiple clusters/clouds, or workspacesare in a different cluster/cloud than the Coder Server, use external provisionerdaemons.

Running provisioner daemons within the same cluster/cloud as the workspaces theyprovision:

• Allows you to use infrastructure-provided credentials (seeAuthenticationbelow) which are typically easier to manage and have shorter lifetimes thancredentials issued outside the cloud/cluster.
• Means that you don’t have to open any ingress ports on the clusters/cloudsthat host workspaces.
  • The external provisioner daemons dial out to Coder Server.
  • Provisioner daemons run in the cluster, so you don’t need to exposecluster/cloud APIs externally.
• Each cloud/cluster is isolated, so a compromise of a provisioner daemon islimited to a single cluster.

Authentication

1. Use ascoped key toauthenticate the provisioner daemons with Coder. These keys can only be usedto authenticate provisioner daemons (not other APIs on the Coder Server).

2. Store the keys securely and use environment variables to pass them to theprovisioner daemon.

3. If on Kubernetes, use a Kubernetes secret to set the environment variable.

4. Tag provisioners with identifiers for the specific cluster/cloud.

   This allows your templates to target a specific cluster/cloud such as forgeographic proximity to the end user, or for specific features like GPUs ormanaged services.

5. Scope your keys to organizations and the specific cluster/cloud using thesame tags when creating the keys.

   This ensures that a compromised key will not allow an attacker to gain accessto jobs for other clusters or organizations.

Provisioner daemons should have access only to cluster/cloud API credentials forthe specific cluster/cloud they are for. This ensures that compromise of oneProvisioner Daemon does not compromise all clusters/clouds.

Deploy the provisioner daemon to the cloud and leverage infrastructure-providedcredentials, if available:

• Service account tokens on Kubernetes
• IAM roles for EC2 on AWS
• Attached service accounts on Google Cloud

Encryption in transit

Enable TLS on Coder Server and ensure you use anhttps:// URL to access theCoder Server.

See theEncryption in transit subheading of theTemplates section for more about encryptingcluster/cloud API calls.

Encryption at rest

Run provisioner daemons only on systems with full disk encryption enabled.

• Provisioner daemons temporarily persist terraform template files and resourcestate to disk. Either of these could contain sensitive information, includingcredentials.

  This temporary state is on disk only while actively building workspaces, butan attacker who compromises physical disks could successfully read thisinformation if not encrypted.

• Provisioner daemons store cached copies of Terraform provider binaries. Theseare generally not sensitive in terms of confidentiality, but it is importantto maintain their integrity. An attacker that can modify these binaries couldinject malicious code.

Workspace proxies

Workspace proxies authenticate end users and then proxy network traffic toworkspaces.

Coder takes care to ensure the user credentials processed by workspace proxiesare scoped to application access and do not grant full access to the Coder APIon behalf of the user. Still, a fully compromised workspace proxy would be in aprivileged position to phish unrestricted user credentials.

Workspace proxies have unrestricted access to establish encrypted tunnels toworkspaces and can access any port on any running workspace.

Authentication

1. Securely store the workspace proxy token generated bycoder wsproxy create.

2. Inject the token to the workspace proxy process via an environment variable,rather than via an argument.

3. If on Kubernetes, use a Kubernetes secret to set the environment variable.

Encryption in transit

Enable TLS on Coder Server and ensure you use anhttps:// URL to access theCoder Server.

Communication to the proxied workspace applications is always encrypted withWireguard. No action is required.

Encryption at rest

Workspace proxies persist no state locally. No action is required.

Workspace templates

Coder templates are executed on provisioner daemons and can include arbitrarycode via thelocal-exec provisioner.

Furthermore, Coder templates are designed to provision compute resources in oneor more clusters/clouds, and template authors are generally in full control overcode and scripts executed by the Coder agent in those compute resources.

This means that template admins have remote code execution privileges for anyprovisioner daemons in their organization and within any cluster/cloud thoseprovisioner daemons are credentialed to access.

Template admin is a powerful, highly-trusted role that you should not assignlightly. Instead of directly assigning the role to anyone who might need to edita template, useGitOps to allow users to author and edit templates.

Secrets

Never include credentials or any other secrets directly in templates, includingin.tfvars or other files uploaded with the template.

Instead do one of the following:

• Store secrets in a central secrets manager.

  • Access the secrets at build time via a Terraform provider.

    This can be throughVaultorAWS Secrets Manager.

• Place secrets inTF_VAR_* environment variables.

  • Provide the secrets to the relevant Provisioner Daemons and access them viaTerraform variables withsensitive = true.

• Use Coder parameters to accept secrets from end users at build time.

Coder does not attempt to obscure the contents of template files from usersauthorized to view and edit templates, so secrets included directly couldinadvertently appear on screen while template authors do their work.

Template versions are persisted indefinitely in the PostgreSQL database, so ifsecrets are inadvertently included, they should be revoked as soon as practical.Pushing a new template version does not expunge them from the database. Contactsupport if you need assistance expunging any particularly sensitive data.

Encryption in transit

Always use encrypted transport to access any infrastructure APIs. Crucially,this protects confidentiality of the credentials used to access the APIs.

Configuration of this depends on the specific Terraform providers in use and isbeyond the scope of this document.

Encryption at rest

While your most privileged secrets should never be included in template files,they may inevitably contain confidential or sensitive data about your operationsand/or infrastructure.

• Ensure that operators who write, review or modify Coder templates are workingon laptops/workstations with full disk encryption, or do their work inside aCoder workspace with full disk encryption.
• EnsurePostgreSQL is encrypted at rest.
• Ensure anysource code repositories that store templates areencrypted at rest and have appropriate access controls.

GitOps

GitOps is the practice of using a Git repository as the source of truth foroperational config and reconciling the config in Git with operational systemseach time themain (or, archaically,master) branch of the repository isupdated.

1. Store Coder templates in a single Git repository, or a single repository perCoder organization, and use theCoderd Terraform providerto push changes from the main branch to Coder using a CI/CD tool.

   This gives you an easily browsable, auditable history of template changes andwho made them. Coder audit logs establish who and when changes happen, butgit repositories are particularly handy for analyzing exactly what changes totemplates are made.

2. Use a Coder user account exclusively for the purpose of pushing templatechanges and do not give any human users the credentials.

   This ensures any actions taken by the account correspond exactly to CI/CDactions from the repository and allows you to avoid granting the templateadmin role widely in your organization.

3. UseGitHub branch protection,or the equivalent for your source repository to enforce code review ofchanges to templates.

   Code review increases the chance that someone will catch a potential securitybug in your template.

These protections also mitigate the risk of a single trusted insider “goingrogue” and acting unilaterally to maliciously modify Coder templates.

Workspaces

The central purpose of Coder is to give end users access to managed compute inclusters/clouds designated by Coder’s operators (like platform or developerexperience teams). End users are granted shell access and from there can executearbitrary commands.

This means that end users have remote code execution privileges within theclusters/clouds that host Coder workspaces.

It is important to limit Coder users to trusted insiders and/or take steps toconstrain malicious activity that could be undertaken from a Coder workspace.

Example constraints include:

• Network policy or segmentation
• Runtime protections on the workspace host (e.g. SELinux)
• Limiting privileges of the account or role assigned to the workspace such as aservice account on Kubernetes, or IAM role on public clouds
• Monitoring and/or auditing for suspicious activity such as cryptomining orexfiltration

Outbound network access

Identify network assets like production systems or highly confidentialdatastores and configure the network to limit access from Coder workspaces.

If production systems or confidential data reside in the same cluster/cloud, useseparate node pools and network boundaries.

If extraordinary access is required, followZero Trustprinciples:

• Authenticate the user and the workspace using strong cryptography
• Apply strict authorization controls
• Audit access in a tamper resistant secure store

Consider the network assets end users will need to do their job and the level oftrust the company has with them. In-house full-time employees have differentaccess than temporary contractors or third-party service providers. Restrictaccess as appropriate.

A non-exclusive list of network assets to consider:

• Access to the public Internet
  • If end users will access the workspace over the public Internet, you mustallow outbound access to establish the encrypted tunnels.
• Access to internal corporate networks
  • If end users will access the workspace over the corporate network, you mustallow outbound access to establish the encrypted tunnels.
• Access to staging or production systems
• Access to confidential data (e.g. payment processing data, health records,personally identifiable information)
• Access to other clusters/clouds

Inbound network access

Coder manages inbound network access to your workspaces via a set of Wireguardencrypted tunnels. These tunnels are established by sending outbound packets, soon stateful firewalls, disable inbound connections to workspaces to ensureinbound connections are handled exclusively by the encrypted tunnels.

DERP

DERP is a relay protocol developedby Tailscale.

Coder Server and Workspace Proxies include a DERP service by default. Tailcalealso runs a set of public DERP servers, globally distributed.

All DERP messages are end-to-end encrypted, so the DERP service only learns the(public) IP addresses of the participants.

If you consider these addresses or the fact that pairs of them communicate overDERP to be sensitive, stick to the Coder-provided DERP services which run onyour own infrastructure. If not, feel free to configure Tailscale DERP serversfor global coverage.

STUN

STUN is an IETF standard protocol thatallows network endpoints behind NAT to learn their public address and portmappings. It is an essential component of Coder’s networking to enable encryptedtunnels to be established without a relay for best performance.

Coder does not ship with a STUN service because it needs to be run directlyconnected to the network, not behind a reverse proxy or load balancer as Coderusually is.

STUN messages are not encrypted, but do not transmit any tunneled data, theysimply query the public address and ports. As such, a STUN service learns thepublic address and port information such as the address and port on the NATdevice of Coder workspaces and the end user's device if STUN is configured.

Unlike DERP, it doesn’t definitively learn about communicating pairs of IPs.

If you consider the public IP and port information to be sensitive, do not usepublic STUN servers.

You may choose not to configure any STUN servers, in which case most workspacetraffic will need to be relayed via DERP. You may choose to deploy your own STUNservers, either on the public Internet, or on your corporate network andconfigure Coder to use it.

If you do not consider the addresses and ports to be sensitive, we recommendusing the default set of STUN servers operated by Google.

Workspace apps

Coder workspace apps are a way to allow users to access web applications runningin the workspace via the Coder Server or Workspace Proxy.

1. Disable workspace apps on sub-pathsof the main Coder domain name.

2. Use a separate, wildcard domain namefor forwarding.

   Because of the defaultsame-origin policy inbrowsers, serving web apps on the main Coder domain would allow those apps tosend API requests to the Coder Server, authenticated as the logged-in userwithout their explicit consent.

Port sharing

Coder supports the option to allow users to designate specific network ports ontheir workspace as shared, which allows others to access those ports via theCoder Server.

Consider restricting the maximum sharing level for workspaces, located in thetemplate settings for the corresponding template.

Encryption at rest

Deploy Coder workspaces using full disk encryption for all volumes.

This mitigates attempts to recover sensitive data in the workspace by attackerswho gain physical access to the disk(s).