- Notifications
You must be signed in to change notification settings - Fork165
Created how to guide - upgrade deprecated docker images#1352
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
Merged
Uh oh!
There was an error while loading.Please reload this page.
Merged
Changes fromall commits
Commits
Show all changes
10 commits Select commitHold shift + click to select a range
0c2e37f Created how to guide - upgrade deprecated docker images
TamirSolomoncb7c4d9 Merge remote-tracking branch 'origin/master' into deprecate-old-docke…
TamirSolomon40d33fa new
TamirSolomon76a0ef9 Trying again
TamirSolomon1e77104 added grafana dashboard as link
TamirSolomon104ce0e added images
TamirSolomonc9076b0 added the screenshots
TamirSolomon9b69d3f fixed numbering
TamirSolomonca3dfee added break lines
TamirSolomon095f602 added links
TamirSolomonFile filter
Filter by extension
Conversations
Failed to load comments.
Loading
Uh oh!
There was an error while loading.Please reload this page.
Jump to
Jump to file
Failed to load files.
Loading
Uh oh!
There was an error while loading.Please reload this page.
Diff view
Diff view
There are no files selected for viewing
326 changes: 326 additions & 0 deletions_docs/kb/articles/upgrade-deprecated-docker-images.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,326 @@ | ||
| --- | ||
| title:"How to: Upgrade Deprecated Docker Images in Classic Pipelines" | ||
| description:"Find and upgrade deprecated Docker v1 and schema 1 images to support docker-27 runner in Classic Pipelines." | ||
| group:kb | ||
| sub-group:articles | ||
| toc:true | ||
| kb:false | ||
| ht:true | ||
| common:false | ||
| categories:[Pipelines] | ||
| support-reviewed:2025-07-14 AA | ||
| --- | ||
| #How to: Upgrade to the newer classic docker-27 runner | ||
| As part of our ongoing platform upgrades, we’re moving to a newer classic runner version based | ||
| on docker ver 27. This change deprecates older Docker image formats, and you may need to | ||
| identify and upgrade deprecated images in your pipelines. | ||
| This guide will help you: | ||
| 1. Identify deprecated Docker images in your Classic Pipelines. (Step 1) | ||
| 2. Upgrade those images to a modern, supported format. (Step 2) | ||
| ###Step 1: Find Deprecated Docker Images | ||
| There are two recommended approaches to identify deprecated images in your Classic | ||
| Pipelines. | ||
| ####Analyze Build Logs | ||
| This method scans historical build logs for deprecation warnings. | ||
| **How it works:** | ||
| Latest versions of dind print deprecation warning into build logs each time Docker daemon | ||
| tries to pull a deprecated image. | ||
| **Example:** | ||
| ``` | ||
| [DEPRECATION NOTICE] Docker Image Format v1 and Docker Image manifest | ||
| version 2, schema 1 support is disabled by default... | ||
| ``` | ||
| Use the following script to scan logs and extract deprecated image names:[Find deprecated images used in previous builds](https://gist.github.com/francisco-cocozza/6046028184cc12b5ee4513bdcb4217c5) | ||
| 💡 For large-scale environments with thousands of builds, we offer a more scalable, | ||
| metrics-based detection method. See Appendix I for details. | ||
| ###Step 2: Upgrade Deprecated Docker Images | ||
| Once you’ve identified deprecated images using engine metrics, the next step is to upgrade | ||
| those images to a modern format (OCI or Docker manifest v2, schema 2). This is required to | ||
| ensure compatibility with docker-27 and future releases. | ||
| ####How It Works | ||
| Re-pushing an image using a modern Docker client will automatically convert its manifest to a | ||
| supported version. | ||
| ####Sample Codefresh Pipeline to Upgrade an Image | ||
| You can automate this process with a Codefresh pipeline like the one below: | ||
| ``` | ||
| 1. version: "1.0" | ||
| 2. | ||
| 3. steps: | ||
| 4. push: | ||
| 5. title: "Re-pushing deprecated image" | ||
| 6. type: push | ||
| 7. candidate: <source-image-name> | ||
| 8. registry: <target-registry> | ||
| 9. tag: <target-tag> | ||
| 10. image_name: <target-image-name> | ||
| 11. | ||
| 12. # Example: | ||
| 13. # push: | ||
| 14. # title: "Re-pushing deprecated image" | ||
| 15. # type: push | ||
| 16. # candidate: docker/whalesay:latest | ||
| 17. # registry: docker | ||
| 18. # tag: new-manifest | ||
| 19. # image_name: codefresh/whalesay | ||
| ``` | ||
| This pulls the deprecated image and re-pushes it under a new name/tag with a compliant | ||
| manifest format. | ||
| 💡 You can repeat this process for each deprecated image discovered via metrics or logs. | ||
| ##Appendix I: | ||
| ###Monitor Engine Metrics | ||
| This method enables scalable detection of deprecated images by using real-time engine | ||
| metrics. Recommended for high-volume or automated environments. | ||
| ####Monitor engine metrics | ||
| Starting from Runtime v7.5.0, engine emits | ||
| codefresh_engine_deprecated_images_pulled_total{ account_name, | ||
| pipeline_id, workflow, image_name } metric, which should be monitored to detect | ||
| deprecated images pulled within DIND. | ||
| This metric is a counter increased by 1 each time Docker daemon pulls a deprecated image. | ||
| ####How to enable | ||
| Set the following values for the Hybrid Runtime Chart: | ||
| ``` | ||
| 1. runtime: | ||
| 2. engine: | ||
| 3. env: | ||
| 4. METRICS_PROMETHEUS_ENABLED: true | ||
| 5. podMonitor: | ||
| 6. main: | ||
| 7. enabled: true | ||
| ``` | ||
| Before normal shutdown, the engine pod waits for the scrape of the metrics. This helps ensure | ||
| that the latest metric values are not missed. | ||
| The timeout is controlled by the METRICS_PROMETHEUS_SCRAPE_TIMEOUT engine variable. If | ||
| you want to be sure that the Prometheus operator will scrape the latest metric values before the | ||
| engine exits, we recommend setting this variable to 4 times your scrape interval. | ||
| To do so, adjust your Hybrid Runtime Chart values: | ||
| ``` | ||
| 1. runtime: | ||
| 2. engine: | ||
| 3. env: | ||
| 4. # Timeout in ms | ||
| 5. METRICS_PROMETHEUS_SCRAPE_TIMEOUT: '120000' | ||
| ``` | ||
| ####How to monitor | ||
| Installation of the monitoring stack is up to the customer and not covered by our support. | ||
| This section provides very basic instructions suitable for the purpose of this doc, however, they | ||
| may not meet customer’s long-term production needs and may require some adjustment by the | ||
| end user. | ||
| If the monitoring stack is not installed in the target cluster, the easiest way is to install | ||
| kube-prometheus-stack Chart: | ||
| [kube-prometheus-stack 75.6.2 · prometheus/prometheus-community](https://artifacthub.io/packages/helm/prometheus-community/kube-prometheus-stack) | ||
| While installing, please consider enabling persistence for the Prometheus operator so that | ||
| collected metrics are not purged on node rotation. Please follow the[official docs](https://artifacthub.io/packages/helm/prometheus-community/kube-prometheus-stack?modal=values&path=prometheus.prometheusSpec.storageSpec) for details | ||
| (check prometheus.prometheusSpec.storageSpec value). | ||
| ####Example installation | ||
| **kube-prom-storage-class.yaml** | ||
| ``` | ||
| 1. # kube-prom-storage-class.yaml | ||
| 2. | ||
| 3. apiVersion: storage.k8s.io/v1 | ||
| 4. kind: StorageClass | ||
| 5. metadata: | ||
| 6. name: kube-prom | ||
| 7. provisioner: kubernetes.io/aws-ebs | ||
| 8. parameters: | ||
| 9. type: gp3 | ||
| ``` | ||
| **kube-prom-values.yaml** | ||
| ``` | ||
| 1. # kube-prom-values.yaml | ||
| 2. | ||
| 3. prometheus: | ||
| 4. prometheusSpec: | ||
| 5. podMonitorSelectorNilUsesHelmValues: false | ||
| 6. serviceMonitorSelectorNilUsesHelmValues: false | ||
| 7. storageSpec: | ||
| 8. volumeClaimTemplate: | ||
| 9. spec: | ||
| 10. storageClassName: kube-prom | ||
| 11. accessModes: ["ReadWriteOnce"] | ||
| 12. resources: | ||
| 13. requests: | ||
| 14. storage: 20Gi | ||
| 15. grafana: | ||
| 16. persistence: | ||
| 17. storageClassName: kube-prom | ||
| 18. enabled: true | ||
| 19. datasources: | ||
| 20. datasources.yaml: | ||
| 21. apiVersion: 1 | ||
| 22. datasources: | ||
| 23. - name: Mimir API | ||
| 24. type: marcusolsson-json-datasource | ||
| 25. uid: Mimir-API | ||
| 26. access: proxy | ||
| 27. url: http://kube-prom-kube-prometheus-prometheus:9090/api/v1 | ||
| 28. plugins: | ||
| 29. - marcusolsson-json-datasource | ||
| ``` | ||
| ###Helm installation | ||
| ``` | ||
| 1. helm repo add prometheus-community https://prometheus-community.github.io/helm-charts | ||
| 2. helm repo update | ||
| 3. helm install kube-prom prometheus-community/kube-prometheus-stack --create-namespace --namespace kube-prom -f ./kube-prom-values.yaml | ||
| ``` | ||
| Please follow instructions provided by Helm after successful installation to get access to the | ||
| newly installed Grafana instance: | ||
| ``` | ||
| 1. kubectl --namespace kube-prom get pods -l "release=kube-prom" | ||
| 2. kubectl --namespace kube-prom get secrets kube-prom-grafana -o jsonpath="{.data.admin-password}" | base64 -d ; echo | ||
| 3. export POD_NAME=$(kubectl --namespace kube-prom get pod -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=kube-prom" -oname) | ||
| 4. kubectl --namespace kube-prom port-forward $POD_NAME 3000 | ||
| ``` | ||
| You can now log in to Grafana athttp://localhost:3000 as an admin user. Double-check the | ||
| instructions above for the password. | ||
| ###Verifying metrics | ||
| To ensure that engine metrics are collected correctly, run some build on the Runtime | ||
| under monitoring. Login to Grafana, open “Explore → Metrics” in the left-hand menu. | ||
| Type codefresh_ in “Search” field, ensure that time range covers at least one running | ||
| build. You will see a list of available metrics. | ||
| {% include | ||
| image.html | ||
| lightbox="true" | ||
| file="/images/troubleshooting/before-verify.png" | ||
| url="/images/troubleshooting/before-verify.png" | ||
| alt="codefresh_no_such_file_directory.png" | ||
| max-width="70%" | ||
| %} | ||
| ##How to use/understand metrics | ||
| codefresh_engine_deprecated_images_pulled_total metric is a counter increased | ||
| by 1 each time Docker daemon pulls a deprecated image. Labels {account_name, | ||
| pipeline_id, workflow, image_name} allow you to group results. | ||
| ##Grafana dashboard | ||
| Easiest way to analyze data — to use the attached Grafana dashboard. | ||
| 1\. Download[this Grafana dashboard json](../../grafana-dashboard.json) <br> | ||
| 2\. Login to Grafana <br> | ||
| 3\. Open “Dashboards → New → Import” <br> | ||
| {% include | ||
| image.html | ||
| lightbox="true" | ||
| file="/images/troubleshooting/import-dashboard.png" | ||
| url="/images/troubleshooting/import-dashboard.png" | ||
| alt="codefresh_no_such_file_directory.png" | ||
| max-width="70%" | ||
| %} | ||
| 4\. Upload Dashboard JSON <br> | ||
| 5\. Select “Select a JSON API data source → Mimir API”. Click “Import” | ||
| {% include | ||
| image.html | ||
| lightbox="true" | ||
| file="/images/troubleshooting/select-api-source.png" | ||
| url="/images/troubleshooting/select-api-source.png" | ||
| alt="codefresh_no_such_file_directory.png" | ||
| max-width="70%" | ||
| %} | ||
| 6\. Done! 🎉 | ||
| Deprecated Images Dashboard overview | ||
| {% include | ||
| image.html | ||
| lightbox="true" | ||
| file="/images/troubleshooting/deprecated-images-overview.png" | ||
| url="/images/troubleshooting/deprecated-images-overview.png" | ||
| alt="codefresh_no_such_file_directory.png" | ||
| max-width="70%" | ||
| %} | ||
| On the screenshot above: | ||
| 1. Filter by account name. | ||
| 2. Time range. | ||
| 3. Account name (build owner). Link leads to the Admin Panel. | ||
| 4. Deprecated image name. | ||
| 5. Pipeline ID where pull was detected. | ||
| ##Query Prometheus metric | ||
| You can query codefresh_engine_deprecated_images_pulled_total metric directly. | ||
| Consider the following query, that returns per second rate of deprecated images pulls, | ||
| aggregated by account ID, pipeline ID, build and image: | ||
| ``` | ||
| 1. sum( | ||
| 2. rate( | ||
| 3. codefresh_engine_deprecated_images_pulled_total | ||
| 4. [$__rate_interval]) | ||
| 5. ) by (account_name, pipeline_id, workflow, image_name) | ||
| ``` | ||
| {% include | ||
| image.html | ||
| lightbox="true" | ||
| file="/images/troubleshooting/deprecated-images.png" | ||
| url="/images/troubleshooting/deprecated-images.png" | ||
| alt="codefresh_no_such_file_directory.png" | ||
| max-width="70%" | ||
| %} | ||
| Deprecated images docker.io/tutum/dnsutils:latest and docker.io/docker/whalesay:latest | ||
| has been pulled in Pipeline #67867cfe8307bd8f9b7b034e | ||
| http://docker.io/tutum/dnsutils:latest | ||
| http://docker.io/docker/whalesay:latest | ||
Oops, something went wrong.
Uh oh!
There was an error while loading.Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.