Environment- Coder deployed in Kubernetes via Helm
- Coder chart version: coder-2.20.2
- Deployed behind Teleport Application Service (not using direct ingress)
- Coder is deployed as an internal ClusterIP service at
http://coder.coder.svc.cluster.local - External access through Teleport:
https://coder.org.teleport.sh
Problem DescriptionWe're experiencing two connectivity issues with Coder when accessed through Teleport: WebSocket Connection Failures: - Coder's WebSocket connections fail with "expected handshake response status code 101 but got 200"
- The
/health/websocket health check endpoint fails - Normal HTTP requests to Coder UI work fine, but any WebSocket-dependent functionality fails
DERP Relay Connection Failures: - DERP relay connections are being redirected to Teleport's authentication page
- The
/health/derp health check endpoint fails - This prevents workspace connections from functioning correctly
These issues only occur when Coder is accessed through Teleport. The basic UI loads correctly, but workspace connections and other WebSocket/DERP dependent functionality fail. Error DetailsWebSocket ErrorEWS01: websocket dial: failed to WebSocket dial: expected handshake response status code 101 but got 200
HTML Response Received (instead of WebSocket upgrade)<!DOCTYPE html><htmllang="en"><head><scriptsrc="/web/config.js"></script><metacharset="utf-8"/><metahttp-equiv="X-UA-Compatible"content="IE=edge"/><metaname="referrer"content="no-referrer"/><metaname="viewport"content="width=device-width, initial-scale=1"/><metaname="grv_csrf_token"content="449fc56af0c97e20e5f415a7955f57bbb08d1c5af64e1199c383bcaf6a2f832d"/><metaname="grv_bearer_token"content=""/><metaname="robots"content="noindex"/><linkhref="/web/app/favicon-light.png"rel="icon"type="image/png"media="(prefers-color-scheme: light)"/><linkhref="/web/app/favicon-dark.png"rel="icon"type="image/png"media="(prefers-color-scheme: dark)"/><title></title><scripttype="module"crossoriginsrc="/web/app/app.js"></script><linkrel="stylesheet"crossoriginhref="/web/app/index.css"></head><body><divid="app"></div></body></html> DERP Connection Errorconnect to derp: derphttp.Client.Connect connect to https://coder.org.teleport.sh/derp: GET failed: <nil>: <a href="https://org.teleport.sh:443/web/launch/coder.org.teleport.sh?path=%2Fderp">Found</a>.
DERP Connection Logsderphttp.Client.Connect: connecting to https://coder.org.teleport.sh/derpderphttp.Client.Connect: TLS version 0x304derpclient: got cert org.teleport.shderpclient: got cert R10derphttp.Client.Connect: not using fast startderphttp.Client.Connect: DERP server returned status 302
Health Check FailuresCurrent Coder ConfigurationWe haven't made any special configuration changes to Coder besides deploying it in Kubernetes. We're investigating whether we need specific environment variables or configuration for proxy deployments. URL Configuration ChallengeA key challenge we're facing is related to the CODER_ACCESS_URL configuration. We're specifically concerned about how Coder handles internal communication between components. Current Limitation:
We're currently using a single CODER_ACCESS_URL which points to our Teleport-proxied external URL (https://coder.org.teleport.sh). This creates a problem because: When users access Coder, they go through this URL and authenticate with Teleport, which works fine for basic UI access. However, Coder's internal components (including WebSocket connections and DERP relay) also attempt to use this same URL for their communication. Our Teleport proxy requires authentication for all requests, which creates a conflict for these internal communications that aren't designed to handle Teleport's authentication.
We've researched CODER_DERP_SERVER_RELAY_URL but understand this is specifically for inter-node communication in high-availability setups, not for separating user access from internal component communication. Troubleshooting Steps Taken- Confirmed Coder UI access works through Teleport
- Added WebSocket headers to the Teleport proxy configuration
- Attempted to force WebSocket relay with
CODER_FORCE_WEBSOCKET_RELAY=true, but this alone didn't resolve the issue - Checked the health check endpoints that confirm both issues
- Verified we can access Coder directly (bypassing Teleport) without these issues
Questions- Is Coder's embedded DERP relay compatible with being accessed through a reverse proxy like Teleport?
- What is the recommended configuration for Coder when deployed behind a proxy?
- Would disabling the embedded DERP relay (
CODER_DERP_SERVER_ENABLED=false) resolve the DERP issue? What functionality would be impacted if we disable it? - Are there specific environment variables we should set for Coder when behind a proxy?
- Does the
CODER_ACCESS_URL need to be set to our external Teleport URL? - Are there known compatibility issues between reverse proxies and Coder's WebSocket/DERP requirements?
- Is there a way to configure Coder to use different URLs for different types of communication? Specifically, can we configure a separate URL for internal component communication that bypasses our authentication proxy?
- Are there any undocumented settings that separate the "user access URL" from "internal communication URLs" within Coder?
Requested AssistanceWe need guidance on the correct configuration to make Coder work properly when accessed through a proxy like Teleport. Specifically: - Recommended settings or environment variables for proxy deployments
- Whether to disable the DERP relay or how to configure it properly
- Any way to separate internal communication paths from user-facing paths
- Any other configuration needed for WebSocket support through a proxy
We're very limited by the single CODER_ACCESS_URL configuration because our Teleport proxy needs authentication for all requests, which Coder's internal components can't provide. Any solution that allows components to communicate directly while still allowing user access through our proxy would resolve our issues. Thank you for your assistance. |