Repository files navigation

	Develop	Master
Build
GoReportCard
Coveralls

• Kubernetes Jenkins as Code management
  • Prerequisites
• Basic concept
  • Advantages
• Build worker
• Configuration
  • Configure alternative configuration with overlays
  • Setup with multiple secret files
• How to use
  • Server only usage / CLI
  • Build hints
  • Dry-Run and Logging
  • Debugging
  • Templates
    • Deployment-only Namespaces
    • Sub-Templates (cloud-templates)
• Execution of Scripts
• IP Management
• Screenshots
  • Deployment
  • Create Project
• Additional tools
  • k8sfullconfigexport
• Helpful links

This project offers a template for managing Jenkins instances on Kubernetes with a JobDSL and Jenkins Configuration asCode (JcasC).

To simplify the installation and the project settings, it has a small helper toolk8s-jcasc-mgmt.go, which can be usedin wizard mode or via arguments to

• create new projects for Jenkins administration
• manage secrets
  • encrypt/decrypt secrets for secure commit to a VCS (version control system)
  • apply secrets to Kubernetes
    • for each project while installation or as an update (applySecrets)
    • for all known namespaces, that are configured in theip_config.cnf file (applySecretsToAll)
  • store secrets globally for easy administration
  • store secrets per project for more security
• manage the Jenkins instances for a namespace with the project configuration
  • install
    • create namespace if it does not exist
    • install Jenkins
    • install nginx-ingress-controller per namespace (if configured)
    • install loadbalancer and ingress for Jenkins
  • uninstall
    • uninstall Jenkins installation
    • uninstall nginx-ingress-controller per namespace (if configured)
    • uninstall loadbalancer and ingress for Jenkins (other ingress routes will not be changed)
  • upgrade
• version check to see a message in the console if a new version is available

If you want to use existing persistent volume claims, then you have to create a persistent volume before you installthe application.

The password for the preconfigured secrets file isadmin. There is no valid data inside this file! Please change itfor your own project!

As default the system uses encrypted passwords instead of using the password from thejenkins_helm_values.yaml.The default users and passwords are:

• administrator
  • User: admin
  • Pass: admin
  • permissions: all
• project user
  • User: project-user
  • Pass: project
  • permissions: read all and execute build

This can be changed on thejcasc_config.yaml file under thejenkins.securityRealm section.

To learn more about the internal processes seehere.

To use this tool, you need to have the following tools installed:

• golang >= 1.14
• shell (sh/bash for scripts support)
• for encryption:
  • gpg
• kubectl
• helm 3

• A namespace contains one Jenkins instance.
• The namespace is more or less equal to a k8s-mgmt project
  • Projects are stored in a separate repository in a VCS
  • The project contains
    • the Jenkins Helm Chart values.yaml overwrites
    • one JCasC file
• The Jenkins for each namespace will be deployed via k8s-mgmt from the cloned Git repository
• Jenkins loads its main configuration from the project repository (and only from this, which means you can play aroundand reload configuration directly from the remote URL)
• This main configuration also contains a very simpleseed-job, which does a scm checkout of a Groovy script to managejobs and a repository, which contains the job definition.
  • you can use thejenkins-jobdsl-remote script as such anseed-job manager.

By having all things stored in VCS repositories, which are normally backed up, it is possible to recreate every instancein no-time.It is impossible to misconfigure a Jenkins instance, because the configuration can be reloaded from this remoterepository and all configurations are completely versioned.

Also, every develops maybe can have admin access to play around with the Jenkins, because they can not destroy thesystem permanently with the beloved "I have nothing done..." statement.

If the K8S cluster or server crashes, it is possible to redeploy everything as it was in minutes, because also the jobdefinition is stored in a VCS repository.

The pre-defined worker-containers will not work directly.Every build worker container needs to set up the jenkins home work directory and jenkins user/groupwithuid/gid1000.

Also, the build worker did not need to have any jenkins agent or something else. Only the user/group and the workdir isneeded.

To resolve the problem, that build containers directly shut down, simply add an entrypoint with atail -f /dev/null.

You can also create a Jenkins build worker base container and build your own build tools container on top of it.

Example of a jenkins-build-worker-base-image:

ARG UBI_CORE_VERSION=ubi8ARG UBI_TAG_VERSION=latestFROM registry.access.redhat.com/${UBI_CORE_VERSION}/ubi-minimal:${UBI_TAG_VERSION}ARG VERSION=1.0.0LABEL Description="Jenkins Node Base Container" Vendor="K8S_MGMT" Version="${VERSION}"# Jenkins user settingsARG user=jenkinsARG group=jenkinsARG uid=1000ARG gid=1000ARG AGENT_WORKDIR=/home/${user}/agent# GLIBC-ENVIRONMENTENV GLIBC_LANG=en_USENV LANG=${GLIBC_LANG}.UTF-8ENV LANGUAGE=${GLIBC_LANG}.UTF-8# install base packagesRUN microdnf update -y && \    microdnf install -y shadow-utils xz tar zip unzip sudo curl wget bash git procps ca-certificates glibc glibc-langpack-en &&  \    microdnf clean all# create jenkins userRUN groupadd -g ${gid} ${group} && useradd -d /home/${user} -u ${uid} -g ${group} ${user}# setup jenkins directories and permissionsRUN mkdir /home/${user}/.jenkins && mkdir -p ${AGENT_WORKDIR}VOLUME /home/${user}/.jenkinsVOLUME ${AGENT_WORKDIR}WORKDIR /home/${user}ENTRYPOINT ["tail","-f","/dev/null"]

A build-worker image for nodejs can look then like this:

FROM jenkins-worker-baseARG VERSION=1.0.0LABEL Description="Jenkins Worker image for NodeJS" Vendor="K8S_MGMT" Version="${VERSION}"# install base packagesRUN microdnf update -y && \    microdnf install -y nodejs && \    microdnf clean all

The system has a basic configuration file to pre-configure some global settings.This file is located underconfig/k8s_jcasc_mgmt.yaml.

It is recommended to change the project base directory to a directory outside of this project.Thecreateproject command will create new projects as subfolders of this directory.All files and directories under this directory should be passed to a git repository which is backed up.

Then your existing Jenkins projects can be fully recovered from this repository.

To use this repository "as-it-is", it is possible to create aconfig/k8s_jcasc_custom.yaml file.

This file can contain the following configuration:

k8sManagement:# Configuration file for project specific overrides. This file must be relative to the `basePath`.configFile:"./config/k8s_jcasc_mgmt_custom.yaml"# Base path for all projects. The path can be specified absolutely.basePath:"/deployments/k8s-jcasc-manaagement"

The script checks, if this file exists.If this is the case, it loads this configuration and checks the argument for the path of the alternative config file.

This means, that thek8sManagement.configFile key can define, where the alternative of thek8s_jcasc_mgmt.cnf islocated.In the.gitignore file, this file is set to ignore, to prevent a commit.

For base paths like templates, the system searches first for the configured base path and if the directory does notexist, it tries to find the directory in the local (./) directory.With this mechanism it is not required to copy for example also the templates into the project directory if they are ok.

To set up multiple secret files, simply add new files in the same directory where thesecrets.sh(.gpg) is located.These files need the prefixsecrets_.

The simplest way is to call the script without arguments. Everything else will be asked by the script.

Hint: Before you install the Jenkins, you have to commit the files of your project directory and to ensure, thatthejasc_config.yaml file is readable for Jenkins (public)

go run k8s-jcasc-mgmt.go

You can also add one of the following flags:

To use the CLI interface only (e.g. on a server) the tagcli deactivates the GUI completely:

go run -tags cli k8s-jcasc-mgmt.go

With this tag the flag-cli is not necessary.In this case Go uses theapp_cli.go file instead ofapp_gui.go to compile.

This was realized with thefyne framework.If you have trouble to compile the project, please visit thefyne developer sitefirst to check the prerequisites.On Windows, it is recommended to installTDM-GCC - tdm-gcc.tdragon.net.

For dry-run with logging it is highly recommended, to use the following option:

go run k8s-jcasc-mgmt.go -logfile=debug.log -logencoding=console -dry-run

This logs the output in console format, which makes it much more readable. With the-dry-run option it renders onlythe Helm Charts and outputs them into the logfile.

It is also a good idea to use the-logfile=<file> and-logencoding=console arguments if there are problems.

To debug this application in JetBrains GoLand (will maybe work also in IntelliJ, but was not tested), you have to opentheActions and to chooseRegistry....There is a point calledgo.run.processes.with.tty, which should be activated. This is relevant because of the UIelements.

Actions can be found with:

• Mac:Command +Shift +A
• Windows:Ctrl +Shift +A

At thetemplates directory contains the following:

• cloud-templates -> support for subsets for Jenkins Cloud templates
• jcasc_config.yaml -> Jenkins Configuration as Code template
• jenkins_helm_values.yaml -> Basic Jenkins configuration of the Helm Charts template
• nginx_ingress_helm_values.yaml -> Nginx Ingress Controller Helm Chart values template
• pvc_claim.yaml -> Template for Persistent Volume Claim
• secrets.sh -> Example of secrets.sh script

To find out more about available placeholders in the template filesseedocs/TemplatePlaceholder.md.

jenkins_helm_values.yaml offers the possibility to add other namespaces for a Jenkins instance, that should deploy.The default for this section is empty:

[ ... ]k8smanagement:rbac:# list of additional namespaces that should be deployable (adds RBAC roles to those namespaces)additionalNamespaces:{ }

To let the Jenkins install applications into other namespaces, these namespaces can be added here.It is not required to add the namespace in which the Jenkins instance is running.The Helm Chart will then add additionalRoles andRoleBindings to these namespaces for this instance.

This is currently a manual process after creating a new project!

Example:In this example we add the namespacesmyapplication-qa,myapplication-preview andmyapplication-production to thisJenkins instance.After this was deployed, Jenkins can now deploy the application into them.

[ ... ]k8smanagement:rbac:# list of additional namespaces that should be deployable (adds RBAC roles to those namespaces)additionalNamespaces:      -myapplication-qa      -myapplication-preview      -myapplication-production

k8s-jcasc-management supports additional sub-templates to create projects with dynamic container configuration.These templates are located in the./templates/cloud-templates/ directory.If this directory does not exist, thecreate project wizard will not ask for other sub-templates.

All files stored there can be selected with the process/menucreate project and will added to thejcasc_config.yaml.

The filejcasc_config.yaml should now have a{{ .JCasc.Clouds.Kubernetes.Templates.AdditionalCloudTemplates }}placeholder:

clouds:    -kubernetes:name:"jenkins-build-worker"serverUrl:""serverCertificate:{ { .JCasc.Clouds.Kubernetes.ServerCertificate } }directConnection:falseskipTlsVerify:truenamespace:"{{ .Base.Namespace }}"jenkinsUrl:"http://{{ .Base.DeploymentName }}:8080"maxRequestsPerHostStr:64retentionTimeout:5connectTimeout:10readTimeout:20templates:{ { .JCasc.Clouds.Kubernetes.Templates.AdditionalCloudTemplates } }

It is important, that the placeholder is at the beginning of the line.

If files are inside of this directory, the user can then select which (none or multiple) sub-templates should be addedto the main template.

These sub-templates must also start on the beginning of the line.For an example have a look here:templates/cloud-templates/node.yaml

It is also possible to create shell scripts for namespaces/directories. This can be helpful, if you want to installother tools besides Jenkins.Thek8s-jcasc-mgmt.go tool first tries to install the secrets, PVC and Jenkins. After this was done it checks, if adirectory calledscripts is inside of the project directory and if it contains*.sh files.

These files have to follow these rules and naming conventions:

• i_*.sh -> Files that should be executed for installation
• d_*.sh -> Files that should be executed for deinstallation

The deinstallation scripts can only be executed if the project directory matches the namespace name. This is necessarybecause normally only the namespace and no directory selection is required for deinstallation.

For greater installations and after a recovery case too, it is helpful to know which Jenkins instance is running behindwhich loadbalancer IP on which namespace.

To provide a simple solution, the system stores this information (namespace and IP) into a configuration file, which canalso be backed up.For every deployment of Jenkins, the system looks into this file and configures the loadbalancer with the IP. This alsoallows static DNS records.

If you create a new project via the wizard, the system also checks, if a IP address already exists to avoid IPconflicts.

Windows screenshot, dark theme

Mac OSX Screenshot, dark themeMac OSX Screenshot, light theme

Mac OSX Screenshot, dark themeMac OSX Screenshot, light theme

Main menu in CLI mode

Namespace selection in CLI mode

You can use this tool to export the complete Kubernetes configuration to a localk8s-manifests directory.This can help to figure out differences between clusters.

• K8S JCasC Management internal Processes:Processes overview
• Kubernetes DNS-Based Service Discovery:https://github.com/kubernetes/dns/blob/master/docs/specification.md
• JCasC Examples:https://github.com/jenkinsci/configuration-as-code-plugin/tree/master/demos
• Jenkins Seed Job script to create jobs from a JSON in a GITrepository:https://github.com/Ragin-LundF/jenkins-jobdsl-remote
• Medium article about thebackground:https://medium.com/@ragin/jenkins-jenkins-configuration-as-code-jcasc-together-with-jobdsl-on-kubernetes-2f5a173491ab
• Medium article about a concept how to work with Jenkins andKubernetes:https://ragin.medium.com/kubernetes-and-ci-cd-how-to-integrate-in-your-development-process-9b483b194975