DERP

This is the code for theTailscale DERP server.

In general, you should not need to or want to run this code. The overwhelmingmajority of Tailscale users (both individuals and companies) do not.

In the happy path, Tailscale establishes direct connections between peers anddata plane traffic flows directly between them, without using DERP for more thanacting as a low bandwidth side channel to bootstrap the NAT traversal. If youfind yourself wanting DERP for more bandwidth, the real problem is usually thenetwork configuration of your Tailscale node(s), making sure that Tailscale canget direction connections via some mechanism.

If you've decided or been advised to run your ownderper, then read on.

Caveats

• Node sharing and other cross-Tailnet features don't work when using customDERP servers.

• DERP servers only see encrypted WireGuard packets and thus are not useful fornetwork-level debugging.

• The Tailscale control plane does certain geo-level steering features andoptimizations that are not available when using custom DERP servers.

Guide to runningcmd/derper

• You must build and update thecmd/derper binary yourself. There are nopackages. Usego install tailscale.com/cmd/derper@latest with the latestversion of Go. You should update this binary approximately as regularly asyou update Tailscale nodes. If using--verify-clients, thederper binaryandtailscaled binary on the machine must be built from the same git revision.(It might work otherwise, but they're developed and only tested together.)

• The DERP protocol does a protocol switch inside TLS from HTTP to a custombidirectional binary protocol. It is thus incompatible with many HTTP proxies.Do not putderper behind another HTTP proxy.

• Thetailscaled client does its own selection of the fastest/nearest DERPserver based on latency measurements. Do not putderper behind a global loadbalancer.

• DERP servers should ideally have both a static IPv4 and static IPv6 address.Both of those should be listed in the DERP map so the client doesn't need torely on its DNS which might be broken and dependent on DERP to get back up.

• A DERP server should not share an IP address with any other DERP server.

• Avoid having multiple DERP nodes in a region. If you must, they all need to bemeshed with each other and monitored. Having two one-node "regions" in thesame datacenter is usually easier and more reliable than meshing, at the costof more required connections from clients in some cases. If your clientsaren't mobile (battery constrained), one node regions are definitelypreferred. If you really need multiple nodes in a region for HA reasons, twois sufficient.

• Monitor your DERP servers withcmd/derpprobe.

• If using--verify-clients, atailscaled must be running alongside thederper, and all clients must be visible to the derper tailscaled in the ACL.

• If using--verify-clients, atailscaled must also be running alongsideyourderpprobe, andderpprobe needs to use--derp-map=local.

• The firewall on thederper should permit TCP ports 80 and 443 and UDP port3478.

• Only LetsEncrypt certs are rotated automatically. Other cert updates require arestart.

• Don't use a firewall in front ofderper that suppressesRSTs uponreceiving traffic to a dead or unknown connection.

• Don't rate-limit UDP STUN packets.

• Don't rate-limit outbound TCP traffic (only inbound).

Diagnostics

This is not a complete guide on DERP diagnostics.

Running your own DERP services requires exeprtise in multi-layer network andapplication diagnostics. As the DERP runs multiple protocols at multiple layersand is not a regular HTTP(s) server you will need expertise in correlativeanalysis to diagnose the most tricky problems. There is no "plain text" or"open" mode of operation for DERP.

• The debug handler is accessible at URL path/debug/. It is only accessibleover localhost or from a Tailscale IP address.

• Go pprof can be accessed via the debug handler at/debug/pprof/

• Prometheus compatible metrics can be gathered from the debug handler at/debug/varz.

• cmd/stunc in the Tailscale repository provides a basic tool for diagnosingissues with STUN.

• cmd/derpprobe provides a service for monitoring DERP cluster health.

• tailscale debug derp andtailscale netcheck provide additional clientdriven diagnostic information for DERP communications.

• Tailscale logs may provide insight for certain problems, such as if DERPs areunreachable or peers are regularly not reachable in their DERP home regions.There are many possible misconfiguration causes for these problems, butregular log entries are a good first indicator that there is a problem.