Repository files navigation

Kubernoodles is a framework formanaging custom self-hosted runners for GitHub Actions in Kubernetes at the enterprise-wide scale. The design goal is to easily bootstrap a system where customized self-hosted runners update, build, test, deploy, and scale themselves with minimal interaction from enterprise admins and maximum input from the developers using it.

This is anopinionated reference implementation, designed to be taken and modified to your liking. I use this to test GitHub Actions on my personal account,GitHub Enterprise Cloud (SaaS) orGitHub Enterprise Server (self-hosted) from Docker Desktop, a Raspberry Pi cluster forarm64, a managed Kubernetes provider, and other random platforms as needed. Your implementation may look wildly different, etc.

❓ Are you a GitHub Enterprise admin that's new to GitHub Actions? Don't know how to set up self-hosted runners at scale? Starthere!

Pull requests welcome! ❤️

Theadmin introduction walks you through some key considerations onhow to think about implementing GitHub Actions at the enterprise scale, the implications of those decisions, and why this project is generally built out the way it is.

Theadmin setup is a mostly copy-and-paste exercise to get a basic deployment up and going.

Thecustomization guide has a quick writeup and links to learn more about the ways you can customize things to your needs.

Tips and tricks has a few more considerations if things aren't quite going according to plan.

There are currently 5 images that are "prebuilt" by this project, although you can certainly use others or build your own! All images assume that they are ephemeral. If you're copy/pasting out of thedeployments, you should be set ... provided you give it the right repository/organization/enterprise to use!

There are a few assumptions that go into this that aren't necessarily true or best practices outside of an enterprise "walled garden". Being approachable and readable are the most important goals of all code and documentation. As a reference implementation, this isn't a turn-key solution, but the amount of fiddling needed should be up to you as much as possible. Links to the appropriate documentation, resources to learn more where needed, and explanations of design choices will be included!

Co-tenanted business systems tend to have small admin teams running services (like GitHub Enterprise) available to a large group of diverse internal users. That system places a premium on people-overhead more than computer-overhead. The implication of that is an anti-pattern where there are larger containers capable of lots of different things instead of discrete, "microservices" type containers.

Moving data around locally is exponentially cheaper and easier than pulling data in from external sources, especially in a larger company. Big containers are not scary if the registry, the compute, and the entire network path is all within the same datacenter or availability zone. Caching on-site is important to prevent rate-limiting by upstream providers, as that can take down other services and users that rely on them. This also provides a mechanism for using a "trusted" package registry, common in enterprise environments, using an.env file as outlinedhere.

These are all excellent reads and can provide more insight into the customization options and updates than are available in this repository. This entire repository is mostly gluing a bunch of these other bits together and explaining how/why to make this your own.

• GitHub's officialdocumentation on hosting your own runners.
• Kubernetes controller for self-hosted runners, onGitHub, is the glue that makes this entire solution possible.
• Docker image for runners that can automatically join, which solved a good bit of getting the runner agent started automatically on each pod,write up andGitHub.
• GitHub's repository used to generate the hosted runners' images (GitHub), where I got the idea of using shell scripts to layer discrete dependency management on top of a base image. Thesoftware scripts are (mostly) copy/pasted directly out of that repo.

• Don't know what the whole Kubernetes thing is about? Here's some help:
  • TheKubernetes Aquarium
  • The Cloud Native Computing Foundation's book,The Illustrated Children's Guide to Kubernetes
  • The officialtutorial covering the basics of what Kubernetes is and how it works
  • What helped me to understand this whole concept shift is to think that Kubernetes is to containers as KVM/vSphere/Hyper-V is to virtual machines. It's probably not a perfect metaphor, but it helped. 😄
• Want to see a whole bunch of other ways to solve this problem? You should check outAwesome Runners for a curated list and amazing matrix comparison of all sorts of other self-hosted runner solutions.
• Even if this is 100% on-premises, many of theseantipatterns for cloud applications are very relevant to the architecture of CI at scale and these are all well worth the time to read.
• Rootful versus rootless containerization in Podman is a bit different than in Docker. Learn more at RedHat'sEnable Sysadmin blog post.

• actions-runner-controller
• Helm
• Yelp dumb-init
• Docker engine andDocker Compose for Debian-based images
• Podman,Buildah, andSkopeo for the RedHat-based images
• actions/runner is the runner agent for GitHub Actions