Repository files navigation

This repository holds the source for various packages and tools that supportGo's build system and the development of the Go programming language.

Warning: Packages here are internal to Go's build system and its needs.Some may one day be promoted to anothergolang.org/x repository,or they may be modified arbitrarily or even disappear altogether.In short, code in this repository is not subject to the Go 1 compatibilitypromise nor theRelease Policy.

This repository uses Gerrit for code changes. To contribute, seehttps://golang.org/doc/contribute.html.

The main issue tracker for the blog is located athttps://github.com/golang/go/issues. Prefix your issue with"x/build/DIR:" in the subject line.

The main components of the Go build system are:

• Thecoordinator, incmd/coordinator/,serveshttps://farmer.golang.org/ andhttps://build.golang.org/.It runs on GKE and coordinates the whole build system. It finds workto do (both pre-submit "TryBot" work, and post-submit work) and executesbuilds, allocating machines to run the builds. It is the owner of all machines.It holds the state for which builds passed or failed, and the build logs.

• The Go package inbuildenv/contains constants for where the dashboard and coordinator run, for prod,staging, and local development.

• Thebuildlet, incmd/buildlet/, is theHTTP server that runs on each worker machine to execute builds on thecoordinator's behalf. This runs on every possible GOOS/GOARCH value. Thebuildlet binaries are stored on Google Cloud Storage and fetchedper-build, so we can update the buildlet binary independently of theunderlying machine images. The buildlet is the most insecure serverpossible: it has HTTP handlers to read & write arbitrary content to disk,and to execute any file on disk. It also has an SSH tunnel handler. Thebuildlet must never be exposed to the Internet. The coordinator provisionsbuildlets in one of three ways:

  1. by creating VMs on Google Compute Engine (GCE) with custom imagesconfigured to fetch & run the buildlet on boot, listening on port 80 in aprivate network.

  2. by running Linux containers (on either Google Kubernetes Engineor GCE with the Container-Optimized OS image), with the containerimages configured to fetch & run the buildlet on start, alsolistening on port 80 in a private network.

  3. by taking buildlets out of a pool of connected, dedicated machines. Thebuildlet can run in eitherlisten mode (as on GCE and GKE) or inreverse mode. In reverse mode, the buildlet connects out tohttps://farmer.golang.org/ and registers itself with the coordinator. TheTCP connection is then logically reversed (usingrevdial and when thecoordinator needs to do a build, it makes HTTP requests to the coordinatorover the already-open TCP connection.

  These three pools can be viewed at the coordinator'shttps://farmer.golang.org/#pools.

• Theenv/ directory describesbuild environments. It contains scripts to create VM images, Dockerfilesto create Kubernetes containers, and instructions and tools for dedicatedmachines.

• maintner inmaintner/ isa library for slurping all of Go's GitHub and Gerrit state into memory.The daemonmaintnerd inmaintner/maintnerd/runs on GKE and serveshttps://maintner.golang.org/. The daemon watchesGitHub and Gerrit and appends to a mutation log whenever it sees newactivity. The logs are stored on GCS and served to clients.

• Thegodata packageinmaintner/godata/provides a trivial API to let anybody write programs againstGo's maintner corpus (all of our GitHub and Gerrit history), live upto the second. It takes a few seconds to load into memory and a few hundredMB of RAM after it downloads the mutation log from the network.

• pubsubhelper incmd/pubsubhelper/ isa dependency of maintnerd. It runs on GKE, is available athttps://pubsubhelper.golang.org/, and runs an HTTP server to receiveWebhook updates from GitHub on new activity and an SMTP server to receivenew activity emails from Gerrit. It then is a pubsub system for maintnerdto subscribe to.

• Thegitmirror server incmd/gitmirror/ mirrorsGerrit to GitHub, and also serves a mirror of the Gerrit code to thecoordinator for builds, so we don't overwhelm Gerrit and blow our quota.

• The Gogopherbot bot logic runs on GKE. The code is incmd/gopherbot. Itdepends on maintner via the godata package.

• Thedeveloper dashboard athttps://dev.golang.org/ runs on GKE.Its code is indevapp/.It also depends on maintner via the godata package.

• cmd/retrybuilds: a Go client program to delete build results from thedashboard

• Theperfdata server, inperfdata/appengineserveshttps://perfdata.golang.org/. It runs on App Engine andserves the benchmark result storage system.

• Theperf server, inperf/appengineserveshttps://perf.golang.org/. It runs on App Engine and servesthe benchmark result analysis system. SeeitsREADME for how to start a local testing instance.

If you wish to run a Go builder, please emailgolang-dev@googlegroups.com first. Thereis documentation athttps://golang.org/wiki/DashboardBuilders, but dependingon the type of builder, we may want to run it ourselves, after you prepare anenvironment description (resulting in a VM image) of it. See the env directory.