Repository files navigation

A reference for tools, configurations, and documentation used to monitor CircleCI server.

🚧Under Development

This repository is currently under active development and is not yet a supported resource. Please refer to it at your own discretion until further notice.

• Installing the Monitoring Stack
• Modifying or Adding Grafana Dashboards
• Helm Values
• Helm Releases

A reference Helm chart for setting up a monitoring stack for CircleCI server

To set up monitoring for a CircleCI server instance, you need to configure Telegraf to set up a Prometheus client and expose a metrics endpoint. Add the following configuration to the CircleCIserver Helm chart values:

telegraf:config:outputs:      -file:files:["stdout"]      -prometheus_client:listen:":9273"

First, add the CircleCI Server Monitoring Stack Helm repository:

$ helm repo add server-monitoring-stack https://packagecloud.io/circleci/server-monitoring-stack/helm$ helm repo update

Before installing the full chart, you must first install the dependency subcharts and operators.

Install the Prometheus Custom Resource Definitions (CRDs) and the Grafana operator chart. This assumes you are installing it in the same namespace as your CircleCI server installation:

$ helm install server-monitoring-stack server-monitoring-stack/server-monitoring-stack --set global.enabled=false --set prometheusOperator.installCRDs=true --version 0.1.0-alpha.8 -n<your-server-namespace>

NOTE: It's possible to install the monitoring stack in a different namespace than the CircleCI server installation. If you do so, set theprometheus.serviceMonitor.selectorNamespaces value with the target namespace.

If you plan to enable distributed tracing with Tempo (tempo.enabled=true), you must manually install the Tempo Operator. There is currently no official Helm chart available for the Tempo Operator or its CRDs, so manual installation is required. The Tempo Operator also requires cert-manager to be installed in your cluster. Additionally, this reference chart requires thegrafanaOperator feature gate to be enabled for proper integration with Grafana.

For more detailed installation instructions, refer to theofficial Tempo Operator documentation.

Prerequisites:

• cert-manager must be installed in your cluster

Example installation steps:

1. Install the Tempo Operator:

$ kubectl apply -f https://github.com/grafana/tempo-operator/releases/download/v0.17.0/tempo-operator.yaml

2. Enable thegrafanaOperator feature gate (required for integration with Grafana):

$ kubectl get cm tempo-operator-manager-config -n tempo-operator-system -o yaml| \    sed's/^  *grafanaOperator: false$/      grafanaOperator: true/'| \    kubectl apply -f -

3. Restart the operator deployment to apply the configuration:

$ kubectl rollout restart deployment/tempo-operator-controller -n tempo-operator-system$ kubectlwait --for=condition=available --timeout=120s deployment/tempo-operator-controller -n tempo-operator-system

Next, install the Helm chart using the following command:

$ helm upgrade --install server-monitoring-stack server-monitoring-stack/server-monitoring-stack --reset-values --version 0.1.0-alpha.8 -n<your-server-namespace>

To verify that Prometheus is working correctly and targeting Telegraf, use the following command to port-forward Prometheus:

$ kubectl port-forward svc/server-monitoring-prometheus 9090:9090 -n<your-namespace-here>

Then visithttp://localhost:9090/targets in your browser. Verify that Telegraf appears as a target and that its state is "up".

To verify that Grafana is working correctly and connected to Prometheus, use the following command to port-forward Grafana:

$ kubectl port-forward svc/server-monitoring-grafana-service 3000:3000<your-namespace-here>

Then visithttp://localhost:3000 in your browser. Once logged in with the default credentials, navigate tohttp://localhost:3000/dashboards and verify that the default dashboards are present and populating with data.

After ensuring both Prometheus and Grafana are operational, consider these enhancements:

Secure Grafana by configuring credentials:

grafana:credentials:# Directly set these for quick setupsadminUser:"admin"adminPassword:"<your-secure-password-here>"# For production, use a Kubernetes secret to manage credentials securelyexistingSecretName:"<your-secret-here>"

For external access, modify the service or ingress values. For example:

grafana:service:type:LoadBalancer

Persist data by enabling storage for Prometheus and Grafana:

prometheus:persistence:enabled:truestorageClass:<your-custom-storage-class>grafana:persistence:enabled:truestorageClass:<your-custom-storage-class>

NOTE: Use a custom storage class with a 'Retain' policy to allow for data retention even after uninstalling the chart.

When Tempo is enabled, it's recommended to use object storage instead of in-memory storage for trace persistence. Compatible storage backends for Tempo and CircleCI server include S3, GCS, and MinIO.

Configure object storage using thetempo.storage values detailed in thevalues section below.

NOTE: For production deployments, object storage provides better durability and scalability compared to in-memory storage, which loses traces on pod restarts.

For detailed configuration options, consult theofficial Tempo documentation.

The default dashboards are located in thedashboards directory of the reference chart. To add new dashboards or modify existing ones, follow these steps.

Dashboards are provisioned directly from CRDs, which means any manual edits will be lost upon a refresh. As such, the workflow outlined below is recommended for making changes:

1. Create a Copy:
   • SelectEdit in the upper right corner.
   • ChooseSave dashboard ->Save as copy.
   • After saving, navigate to the copy.
2. Make Edits:
   • Modify the copy as needed and exit edit mode.
3. Export as JSON:
   • SelectExport in the upper right corner and thenExport as JSON.
   • Ensure thatExport the dashboard to use in another instance is toggled on.
4. Update the JSON File:
   • Download the file and replace the./dashboards/server-slis.json file with the updated copy.
   • Run the following command to automatically validate the JSON and apply necessary updates:

     ./do validate-dashboards

5. Commit and Open a PR:
   • Review and commit the changes.
   • Open a pull request for the On-Prem team to review.

Releases are managed by the CI/CD pipeline on the main branch, with an approval job gate calledapprove-deploy-chart. Before releasing, increment the Helm chart version inChart.yaml and regenerate the documentation using./do helm-docs. Once approved, the release will be available in thepackage repository.

This monitoring reference is not part of CircleCI’s Server product. CircleCI provides it as a monitoring tooling and configuration repository for CircleCI Server User(s) that may be referred to when the User(s) plan and deploy their own monitoring implementations.

CircleCI strives to ensure that the monitoring tooling and configurations in this reference are functional and up to date. While CircleCI may provide reference to, answer questions regarding, and/or review contributions to the monitoring tooling and configurations, CircleCI does not make any judgment or recommendation as to the suitability for any customer installation of them with CircleCI Server, nor provide support for their installation and/or management in any customer’s system.

This monitoring reference and the monitoring tooling and configurations are provided on an ‘as-is’ and ‘as available’ basis without any warranties of any kind. CircleCI disclaims all warranties, express or implied, including, but not limited to, all implied warranties of merchantability, title, fitness for a particular purpose, and noninfringement.