Thecontainer startup agent in Compute Engine is deprecated. This agent lets youdeploy containers on Compute Engine instances when you create VMs.

This document describes how to migrate existing containers that the startupagent created on your VMs or managed instance groups (MIGs) to otherGoogle Cloud services.

Based on your requirements, choose one of the following options tomigrate the containers that were deployed on VMs using the deprecated method:

• If you want to continue running containers on individual VMs and MIGs, usestartup scripts orcloud-init.
• If you have stateless container applications and small to medium jobs,useCloud Run.
• If your container is a batch job that has a definite end state and requiresadditional computing resources, useBatch.
• If you need advanced control and scalability or if you can't meet yourrequirements with the other options, useGKE on Google Cloud.

For more use cases and alternative solutions, seeCompare the container deployment options.

Deprecated options for configuring containers on VMs

When you configure a container during VM creation,Compute Engine uses the container startup agent to read thegce-container-declaration metadata that stores thecontainer information, and to deploy the container on the VM.

The following options fordeploying containers directly on a VM or MIGthat use the container startup agent and thegce-container-metadata are deprecated.

Identify instances that use the deprecated container metadata

To identify instances in your project that use the deprecated containermetadata, follow these steps to check for thegce-container-declarationmetadata key:

1. Run one of the following commands:

   • To list all instances in your project that use thegce-container-declaration metadata key and value, run the followingGoogle Cloud CLI command:

     gcloudcomputeinstanceslist--filter="metadata.items.key:gce-container-declaration"

     This command provides a list of all VM instances in your configuredproject that contain thegce-container-declaration metadata key.The metadata key uniquely identifies VMs that are in scope of thedeprecation. If you are using multiple projects, run this commandacross all of the active projects.

   • To validate a specific instance, run the following Google Cloud CLI command:

     gcloudcomputeinstancesdescribeVM_NAME--format="(metadata.items)"

     ReplaceVM_NAME with the name of the VMinstance that you want to validate.

2. If the output of the command from the preceding step listsinstances that use thegce-container-declaration metadata key, runthe following command to get more details about the containerdeclaration on your VMs:

   gcloudcomputeinstanceslist\--filter='metadata.items.key:gce-container-declaration AND (metadata.items.value~"image:")'\--format="table(name, zone, metadata.items.filter(key='gce-container-declaration').extract(value).slice(0):label=CONTAINER_DECLARATION)"

3. Based on the output of the command, consider the following:

   • If the metadata contains the definition for the deprecated container startup agent, thenyou must migrate the container deployment to an alternative solution as described inthis document.

   • If thegce-container-declaration metadata key is present, but you arenot using it for the container startup agent, take the following actions:

     • Check if you are reusing this metadata key for other configurations.
     • If you are reusing the key, use a different metadata key for other configurations.

For more information about viewing metadata, seeView and query metadata.

Compare the container deployment options

The following table summarizes the use cases for running containers on VMs andrecommends alternative container solutions for migrating your workloads:

When you transition from the Compute Engine container startup agent to analternative solution, consider the following required changes and the potential effort of implementing them:

• VMs running Container-Optimized OS: Take full ownership of VM and containerruntime setup, configuration, security, and maintenance, which ofteninvolves scripting with startup scripts orcloud-init.
• Cloud Run or Batch: Ensure that your applications arestateless and fit the request-driven or job-based execution model. Thisapproach might involve adaptingapplications to work with external state management services.
• GKE: Adopt Kubernetes principles, define workloads by usingKubernetes manifest files, and manage cluster resources.

Use startup scripts to deploy containers on VMs

You can run a basic container on a VM using a startup script.

Consider the following points when you use a startup script to configure containers:

• You can use a startup script for basic scenarios. For advanced configuration,considerusingcloud-init.
• Because you are creating a new VM with a container configured using the startup script, you must planfor the transition of any workloads deployed on the existing VMs.
• Test and ensure that everything works as expected before you route trafficto your newly created VM with a container.

To create a VM and to deploy a container on a VM or a MIG, do the following:

1. Map current container on VM metadata to startup script command
2. Create a startup script based on the existing metadata configuration
3. Create a VM by using the startup script orCreate a MIG by using the startup script.

Map your container metadata todocker run command

You can map the VM metadata orgcloud flagstodocker run arguments and include this in your startup script for creating VMs.

Somegcloud flags translate directly to VM metadata. These flags alsotranslate directly todocker run flags.If you have an existing container on a VM, you can read the VM metadataconfiguration and build a startup script using the equivalentdocker run commands.

# Get your existing VM instance configuration in yaml formatgcloudcomputeinstancesdescribeVM_NAME--format="(metadata.items)"

The output is similar to the following:

metadata:items:-key:gce-container-declarationvalue:|spec:containers:-args:-'"hello world!"'command:-echoenv:-name:ONEvalue:'1'image:docker.io/library/busyboxname:my-instancesecurityContext:privileged:truestdin:truetty:truerestartPolicy:Always-key:google-logging-enabledvalue:'true'

Use the following table to map existing specification todocker run commands:

Example startup scripts

The following examples show how to include thedocker commands in your startupscript:

• Example 1: runs a standalone container in a VM based onContainer-Optimized OS.

• Example 2: runs a web server container in a VM based onContainer-Optimized OS.

#!/bin/bash# A name for the containerCONTAINER_NAME="my-app-container"# Stop and remove the container if it existsdockerstop$CONTAINER_NAME||truedockerrm$CONTAINER_NAME||true# Pull the latest version of the container image from Docker Hubdockerpullbusybox:latest# Run docker container from image in docker hubdockerrunbusybox:latest\echo"hello world!"

#!/bin/bash# Enable incoming trafficiptables-AINPUT-jACCEPT# A name for the containerCONTAINER_NAME="my-app-container"# Stop and remove the container if it existsdockerstop$CONTAINER_NAME||truedockerrm$CONTAINER_NAME||true# Pull the latest version of the container image from Docker Hubdockerpullnginx:latest# Run docker container from image in docker hubdockerrun\--name=$CONTAINER_NAME\--privileged\--restart=always\--tty\--detach\--network="host"\nginx:latest

Additional configuration options for container deployment

This section describes the additional configuration parametersfor deploying containers on your VMs.

For more information about these options, seeConfigure options to run a container.

Access to Artifact Registry images

If you need access to container images fromgcr.io or pkg.dev,use thedocker-credential-gcr tool, which is preinstalled in Container-Optimized OS,and configure authentication to Artifact Registry for Docker.Run the following command before you run the container:

# Set home directory to save docker credentialsexportHOME=/home/appuser# Configure docker with credentials for gcr.io and pkg.devdocker-credential-gcrconfigure-docker

For more information, seeConfigure authentication to Artifact Registry for Docker.

Configure logging

We recommend using Cloud Loggingby enabling a logging agent on a VM.

Alternatively, if you want to change the logging driver, you can includethe--log-driver parameter with yourdocker run command:

# Use Cloud Logging logging driverdockerrun--log-driver=gcplogsnginx:latest

For more information, seeUsing Cloud Logging with Container-Optimized OS

Configure internal firewall

Container-Optimized OS denies incoming traffic by default, so you must addiptables rules to allow that traffic. Note that these commands configure thehost operating system's internal firewall. Additionally, you must configure yourVirtual Private Cloud firewall to allow that traffic to the new VM

For more information, seeUse VPC firewall rules.

# Enable all incoming and routed trafficiptables-AINPUT-jACCEPTiptables-AFORWARD-jACCEPT

For more information, seeConfiguring the host firewall.

Attach volumes to the container

If volumes are attached to the container, the container metadata includes thevolumes entry and avolumeMounts array. Thename of an entry involumescorresponds to the name of an entry involumeMounts, and the other way around.For each volume that you collect, gather the required information either fromthevolumes or from thevolumeMounts entry.

If no volumes are attached to the container, you can skip this section anddirectlycreate a VM by using the startup script.

For more information about disks and file system on Container-Optimized OS,seeDisks and file system overview.

Mount tmpfs file system

To mount an empty tmpfs file system to a container, specify the--tmpfsargument with yourdocker run command. For example, to mount a cache file systemto your nginx container, run the following command:

# mount a cache file system to the nginx containerdockerrun-d--name=$CONTAINER_NAME--tmpfs/var/cache/nginx:rw,size=512m,noexec,nosuid,nodev--network="host"nginx:latest

For more information about mountingtmpfs file systems, seetmpfs mounts.

Mount a host directory

To mount a directory from a host VM to a container, specify the--mount argumentwith thedocker run command:

# mount a read-only directory to the nginx containerdockerrun-d--name=$CONTAINER_NAME--mounttype=bind,source=/var/www/html,target=/usr/share/nginx/html,ronginx:latest

For more information, seeBind mounts.

Mount a persistent disk to the container

Mounting a disk to the container requires additional steps. To mount a disk,first mount it on the VM, and then mount that disk to the container:

1. To mount the disk to the VM, run the following command:

   #!/bin/bashDISK_DEVICE_NAME="my-persistent-disk"# This name MUST match the 'device-name' in the gcloud --disk flagDISK_BY_ID_PATH="/dev/disk/by-id/google-${DISK_DEVICE_NAME}"HOST_MOUNT_POINT="/mnt/disks/my-persistent-disk"# This is the path where the disk will be mounted on the VMCONTAINER_MOUNT_PATH="/usr/share/my-persistent-disk"# This is the path where the disk will be mounted in the container# format a disk as an ext4 filesystem, if it doesn't already contain onefile-sL$DISK_BY_ID_PATH|grep-qfilesystem||\mkfs.ext4-m0-Elazy_itable_init=0,lazy_journal_init=0,discard$DISK_BY_ID_PATH# create a directory for mounting pointsudomkdir-p"${HOST_MOUNT_POINT}"# mount a disk to the VMsudomount-odefaults,discard"${DISK_BY_ID_PATH}""${HOST_MOUNT_POINT}"

2. After you mount the disk to the VM, add the--mount flag with thedocker run command to mount the disk to the container:

   dockerrun-d--name=$CONTAINER_NAME--mounttype=bind,source="${HOST_MOUNT_POINT}",target="${CONTAINER_MOUNT_PATH}",readonlynginx:latest

Create a VM by using the startup script

After creating a startup script with your container configuration, use thisstartup script to create a VM based on Container-Optimized OS. For moreinformation about creating a VM based on Container-Optimized OS, seeCreate an instance from a public image.

For more information about using startup scripts, seeUsing startup scripts on Linux VMs.

provider "google" {project = "PROJECT_ID"}resource "google_compute_instance" "cos_vm_instance" {name         = "VM_NAME"machine_type = "MACHINE_TYPE"zone         = "ZONE"# Use a Container-Optimized OS image for the boot diskboot_disk {  initialize_params {    image = "IMAGE_PROJECT/IMAGE_FAMILY"  }}# Attaches the instance to the default networknetwork_interface {  network = "default"}# Specify the relative path to the startup script on your local machinemetadata = {  startup-script = file("STARTUP_SCRIPT_FILE")}}

provider "google" {  project = "my-project"}resource "google_compute_instance" "my_container_vm" {  name = "my-container-vm-startup"  machine_type = "e2-medium"  zone = "us-central1-a"  boot_disk {    initialize_params {      image = "cos-cloud/cos-stable"    }  }  network_interface {    network = "default"  }  metadata = {    startup-script = file("./startup_script.sh")  }}

Create a MIG by using the startup script

After creating an instance template using the startup script, use one of thefollowing methods to create a MIG.

For more information about creating MIGs, seeCreate a managed instance group.

  # Create the instance template that uses a startup script  gcloud compute instance-templates create startup-template \      --machine-type=e2-medium \      --image-family=cos-stable \      --image-project=cos-cloud \      --metadata-from-file=startup-script=./startup_script.sh  # Create the managed instance group    gcloud compute instance-groups managed create startup-mig \      --template=startup-template \      --size=2 \      --zone=us-central1-a

resource "google_compute_instance_template" "startup_template" {  name_prefix = "startup-template-"  machine_type = "e2-medium"  disk {    source_image = "cos-cloud/cos-stable"    auto_delete  = true    boot         = true  }  network_interface {    network = "default"  }  metadata = {    startup-script = file("./startup_script.sh")}}resource "google_compute_instance_group_manager" "startup_mig" {  name               = "startup-mig"  base_instance_name = "startup-vm"  zone               = "us-central1-a"  version {    instance_template = google_compute_instance_template.startup_template.id  }  target_size = 2}

Test and clean up

After successful creation of a VM or a MIG, validate that your applicationis running on the container and working as expected. To fix any issues, seeTroubleshooting.

If the application is running successfully on your new VMs created using thestartup script, you can delete theVMsandMIGs that use the deprecated method of deploying containers.

Troubleshooting startup script issues

This section provides troubleshooting information for issues that you mightencounter when using startup scripts.

Unable to save docker configuration

When you run thedocker-credential-gcr configure-dockercommand in a startup script,you might get the following error message:

ERROR: Unable to save docker config: mkdir /root/.docker: read-only file system

This error occurs becausedocker-credential-gcr attempts to write credentialsto the/root/.docker/config.json file. Theroot file system onContainer-Optimized OS is read-only, so you can't write to it.

To resolve this issue, set the environment variable$HOME topoint to a custom home directory before you run thedocker commands.

Example:

  export HOME=/home/appuser  docker-credential-gcr configure-docker

View logs to debug issues

To troubleshoot issues that might occur when you configure containers on VMs usinga startup script, view the startup script logs and the container logs.

• To view startup script logs on the VM instance, run the following command:

  sudo journalctl | grep "startup-script"

• To view logs from the Docker container, run thedocker logs command:

  docker logsCONTAINER_NAME

  ReplaceCONTAINER_NAME with the name of your container.

For troubleshooting other issues, see the following documents:

• Cloud Logging overview
• Using Cloud Logging with Container-Optimized OS
• Troubleshooting the Docker daemon
• Troubleshoot and diagnose
• Troubleshoot Terraform
• Troubleshooting when running a basic web server
• Set up and manage network address translation with Public NAT

Usecloud-init with Container-Optimized OS

You can usecloud-init,an industry-standard and cross-platform solution, to deploy containers on VMsrunning Container-Optimized OS.This tool lets you run custom configuration during the VM creation or startup.For more information, seeUsingcloud-init with the Cloud config format.

Use managed services for container deployment

This section describes the managed services provided by Google Cloud that you canuse to deploy containers.

Cloud Run

Cloud Run is a good option for stateless container applications and small to medium jobs.

Key features of Cloud Run include the following:

• You can choose to only allocate CPUs during request processing, or always allocate CPUs.
• You can run a stateless container application or execute a job as one-off, on a schedule, or as part of a workflow.
• You can configure timeouts for each request or task.
• It's highly scalable and secure.
• It has integrated load balancing and autoscaling.

For more information about deploying containers on Cloud Run, seeDeploying container images to Cloud Run

Batch

Batch is a fully managed service that lets you schedule, queue,and execute batch processing workloads on Google Cloud resources. It's designedfor running batch-style, parallelizable workloads, including those packaged in containers.

For more information about deploying containers on Batch, see the following documents:

• Get started
• Create and run a basic job

Google Kubernetes Engine

If you are running complex applications, microservices, continuous operationand need fine-grained control and scalability, Google Kubernetes Engine (GKE) is the offeringthat is best suited. For more information about deploying containers on GKE.see the following documents:

• GKE overview
• Quickstart: Deploy an app to a GKE cluster
• Deploying a containerized web application

Get support

If you have any questions about the migration process or if you need assistance,review theFAQor contactGoogle Cloud Support.