This document describes the suspend and stop actions on virtual machine (VM)instances in a managed instance group (MIG).It also describes how suspending and stopping VMs in a MIG can help you savecosts and reduce the waiting time when you need more VMs in the group.

MIGs let you suspend and stop VMs to achieve the following:

• Pause an application or a service that you are not using to save costs bynot paying for compute resources.
• Accelerate MIG scale out by starting pre-initialized VMs from thestandby pool of stopped and suspended VMs.

Use cases

The following sections describe typical use cases for the standby pool in a MIG.

Pause an application or a service

You cansuspend or stopVMs in a MIG to pause your application and resume it when needed, accordingto your computation, working hours, peak time, and budget constraints.You can keep the results of your current computations on persistent disks or,in the case of suspended VMs, in memory.

For example, you might want to suspend or stop the VMs in a MIG in thefollowing scenarios:

• You have heavy workloads during weekdays and want to suspend VMs on weekendsto save on costs.
• You have a testing environment that is needed during implementationchanges, and you want to stop it when you are not actively developing.

Accelerate MIG scale out

You can keep a standby pool of pre-initialized VMs ready to start when the MIGresizes up.Instead of creating new VMs and waiting for your app to initialize andbecome ready to run, the MIG starts or resumes VMs from the standby pool.In such case, the VM initialization is completed in advance, not in a criticalmoment of increased load.

Standby pools are helpful for applications that take long time to initialize,for example in the following scenarios:

• Applications that need to download up-to-date content to persistent disks.
• Applications that need to cache extra content in memory—via downloadsfrom external storage, from local computation, or a combination of both.
• Applications that need to install fresh software during the initialization,such as Kubernetes nodes.

Preserved resources

The following table shows the resources that are preserved when you suspendand stop VMs in a MIG.

Preserved	Suspended VM	Stopped VM
VM name
Internal IP
External IP (ephemeral)
External IP (static*)
Disks†
Metadata
Memory

Note: You can keep a VM in suspended state forup to 60 days.After 60 days in the suspended state, the VM is terminated.The MIG then creates a new VM and puts it in the suspended state,according to the configuration of the standby pool.If you haven't created astateful configuration in the MIG,then the MIG doesn't preserve the VM name, IP addresses, and disks whenrecreating the VM.

Behavior and configuration

Standby pool is formed by stopped and suspended pools of VMs. All stoppedVMs become a part of the stopped pool, and all suspended VMs become partof the suspended pool.If you configured autoscaling in a MIG, after you suspend or stop a VM,the MIG immediately creates new VMs to maintain the recommended size of the MIG.

Important: If you have aGoogle Cloud load balancerconfigured in your MIG, stopped or suspended VMs are automatically excludedfrom the backend service and don't receive traffic.

Target sizes of suspended and stopped pools

Similar to the target size of the MIG, stopped and suspended pools havetheir own target sizes.You can control the standby pool target size in the following ways:

• By configuring the values of the stopped and suspended target sizes.
• By manually stopping and suspending VMs, which then automatically changes thetarget sizes.

When you change target sizes for stopped or suspended pools, the MIG behavesas follows:

• When you increase the size of suspended or stopped pools, the MIG createsnew VMs, waits until the VMs are initialized, and then suspends or stopsthe VMs accordingly. For regional MIGs, VMs are created in accordance withthetarget distribution shapeconfigured.
• When you decrease the size of suspended or stopped pools, the MIG arbitrarilyselects which suspended or stopped VMs to delete.
• When you change the MIG target size and the size of suspended or stopped poolsimultaneously, the MIG attempts to minimize the number of operations requiredto apply your changes. This means that the MIG might resume or start VMs fromthe standby pool, or suspend or stop some running VMs.

Standby policy

The standby policy defines the behavior of the standby pool based on thefollowing parameters that you specify:

• Mode: The mode in which the MIG uses suspended andstopped VMs. This can bemanual orscale-out-pool mode.
• Initial delay: The time for which the MIG runs a newly createdVM before suspending or stopping it. Configure the initial delay to allowenough time for your app to pre-initialize and be ready to run when the VMstarts or resumes.

Mode

You can choose how to manage standby pools by setting the operation mode.There are two possible options:manual mode andscale-out-pool mode.

Manual mode (default)

In manual mode, you have full control over which VMs are stopped and suspendedin the MIG.Manual mode is the default mode of standby pool.

Manual mode is useful in the following cases:

• To pause your workload and save on costs of idle running VMs.
• To integrate the MIG with third-party autoscalers that require advancedmanagement of individual VMs.
• To stop selected VMs for debugging purposes.

With manual mode, MIG doesn't apply any automations to the standby pool:

• When you or autoscaler increases the target size of the MIG, the MIGdoesn't automatically start or resume VMs, but creates new ones.
• When you or autoscaler decreases the target size of the MIG, the MIGdoesn't automatically stop or suspend running VMs, but deletes them.

Scale out pool mode

In scale out pool mode, the MIG uses the VMs from the standby pools toaccelerate the scale out by resuming or starting them.Then, the MIG automatically replenishes the standby pool with new VMsto maintain the target sizes.

Scale out pool mode is useful to accelerate the scale out of the MIGin the following cases:

• If you use Compute Engine autoscaler.
• If you use third-party autoscalers and you want to preserve any existingintegration.
• If you manually increase the target size of running VMs.

In scale out pool mode, the MIG behaves as follows:

• When you or autoscaler increases the target size of running VMs in the MIG, theMIG takes action in the following order:

  1. The MIG resumes suspended VMs in case any are available in the zoneswhere the MIG scales out.
  2. After resuming the suspended VMs, if the target size of the MIG is notyet reached, the MIG starts stopped VMs if any are available inthe zones where the MIG scales out.
  3. After starting the VMs, if the target size of the MIG is still notreached, it creates new VMs from scratch.

  After the standby pool is used to accelerate scale out, the MIG does thefollowing:

  1. It creates new VMs to replenish the suspended and stopped pools basedon their target sizes, and in accordance with the target distributionshape in case of a regional MIG.
  2. It puts the new VMs in the running state.
  3. It suspends or stops the new VMs after the initial delay is passed.

• When you or autoscaler decreases the target size of the MIG, the MIGdoesn't automatically stop or suspend running VMs but deletes them.

Note: When a regional MIG scales out, depending on itstarget distribution shape,the scale out might occur in a zone or zones that at that moment don't havestopped or suspended VMs. In that case, the MIG creates VMs from scratch in therequired zone or zones.

Initial delay

To make sure that your VM is initialized correctly, specify the initial delayin the standby policy.The initial delay is the time that VMs wait before stopping or suspending afterthey are created. This gives your initialization script the time to complete.

The initial delay occurs in the following cases:

• A new VM is created with the intended target state ofSUSPENDED orTERMINATED.
• An existing instance in theRUNNING state is suspended or stopped.

In both cases, the instance is allowed to initialize before it is suspendedor stopped.

When you want to use standby pool to accelerate the scale out of MIG it isrecommended that you measure the time required for your application toinitialize on the selected machine type to assure that it is sufficientfor your application to be fully ready before suspending or stopping. Otherwise,resuming or starting VMs from standby pool might take longer that creating VMsfrom scratch.

Target status for VMs in MIGs

MIGs have a declarative API. This means that you declare the target status forthe VMs in the MIG, and the API request is successful when the target statusis saved.The MIG then performs the necessary operations to reach the targetstatus, and you can check thecurrent actionand current status of all the VMs using API.

Suspending and stopping VMs in a MIG works in the same declarative way.When you send a request to suspend or stop VMs, the MIG stores the informationabout the target status for each VM and starts the necessary operations toreach it.

When youlist managed VMsin a MIG, you can see thetargetStatus field.It describes the final status of a VM, when the MIG is stable.It can be one of the following values:

• RUNNING
• STOPPED
• SUSPENDED

VMs in a MIG can have the samelifecycle statuses of single VMs.The following are examples of possible operations on a MIG, and theassociated values of thetargetStatus field:

• Create the new VM and suspend it after the initialization.
  • Target status of the VM:SUSPENDED.
• Resume a previously suspended VM.
  • Target status of the VM:RUNNING
• Stop a previously running VM.
  • Target status of the VM:STOPPED
• Start a previously stopped VM.
  • Target status of the VM:RUNNING

Limitations

• The following limitations forsuspending standalone VMsalso apply to suspending VMs in a MIG:
  • You cannot suspend an instance that uses aGPU.
  • You cannot suspend a bare metal instance.
  • You cannot suspend an instance by using the standard processes that are built into the guest environment. Commands, such as thesystemctl suspend command in Ubuntu 16.04 and later, are not available. The in-guest signal is ignored.
  • You can only suspend an instance for up to 60 days before the VM is automatically stopped.
  • You cannot suspend instances with more than 208 GB of memory.
  • You can suspend preemptible instances, but the preemptible instance might be terminated before it is successfully suspended.
  • You cannot suspend a Confidential VM.
  • You cannot suspend a VM that has CSEK-protected disks attached.
• In a regional MIG withEVEN target distribution shape and instanceredistribution enabled, you cannot suspend, stop, resume, or start specificVMs in the group. To manage a standby pool, set the target sizes of thesuspended and stopped pools.
• You cannot use scale out pool mode if you've configured a second instancetemplate forcanary updatein the MIG.
• You cannot suspend or stop VMs in a MIG ifyou'veturned off repairsin the MIG.
• You can only suspend an instance for up to 60 days before the VM isautomatically stopped.

Pricing

Each stopped and suspended VM is billed for the following items:

• Any persistent disk usage for the boot disk, and any additional disksattached to the VM.For more information, seePersistent Disk pricing.
• Any static IPs attached to the VM.For more information, seeIP pricing.
• In the case of suspended VMs, the VM memory and device state.For more information, seeVM instance pricing.

What's next

• Learn how tomanually suspend or stop VMs in a MIG.
• Learn how toaccelerate scale out with suspended and stopped VMs.
• Learn more aboutupdates for suspended and stopped VMs.