You can build highly available deployments of stateful workloads on VMinstances using statefulmanaged instance groups(stateful MIGs). Statefulworkloads include applications with stateful data or configurations such asdatabases, legacy monolith applications, and long-running batch computationswith checkpointing.

With stateful MIGs, you can improve the uptime and resiliency of such statefulapplications with autohealing (automatic recovery of failed workloads),multi-zone deployments, andautomated rolling updates.

A stateful managed instance group preserves the unique state of each instance(including instance name, attached persistent disks, IP addresses, and metadata)on VM restart, recreation, autohealing, or update.

This page describes when to use stateful MIGs and provides a high-level overviewof how they work. For more information, seeHow stateful MIGs work.

To learn how to set up a stateful MIG, seeConfiguring stateful MIGs.

How stateful workloads are different from stateless workloads

You can use managed instance groups to support both stateful and statelessworkloads. The key difference between stateful and stateless workloads is thatstateful workloads preserve individual VM state (for example, adatabase shard, or app configuration) on the VM's disks, while statelessworkloads, like a web frontend, don't retain any state on the individual VMs.

You treat VMs with stateful workloads like custom-built machinery: you careabout VM identity (name), IP address, metadata, and data on each individualmachine. You cannot easily scale stateful workloads horizontally because scalingcould require data replication, creation or deletion of data shards, or changingthe overall application configuration. When recreating or updating a machinewith a stateful workload, you must preserve the VM's unique state. Examples ofstateful applications include Cassandra, ElasticSearch, mongoDB, MySQL,PostgreSQL, and Kafka.

You treat VMs with stateless workloads as interchangeable and only care aboutthe number of serving VMs that you have. No one VM is treated any differentlythan another. You can quickly scale stateless workloads horizontally by addingor removing VMs. When updating your application, you can delete machines andreplace them with new ones with different names, IP addresses, metadata, anddisks. When a stateless VM is deleted or recreated, all data on the machine islost: the disks are deleted or recreated from scratch. A web frontend is anexample of a stateless application.

All MIGs support custom and preservable instance names.

When to use stateful MIGs

Consider using stateful managed instance groups (stateful MIGs) whenever youdeploy a stateful application or cluster to Compute Engine and would like toimprove its availability withautohealingandmulti-zone deployments,or you want to simplify updates of stateful instances byorchestrating update rolloutsand controlling the allowed level of disruption to the instances.

Stateful MIGs are intended for applications with stateful data orconfiguration, such as:

• Databases. For example: Cassandra, ElasticSearch, mongoDB, andZooKeeper. Before deciding on stateful MIGs, consider using fully managedservices, for example, MySQL and PostgreSQL are available inCloud SQL,to focus on your applications and not have to deal with VMs.
• Data processing applications. For example: Kafka and Flink. Beforedeciding on stateful MIGs, consider using fully managed services, forexample,Dataflow orDataproc,to focus on your data processing tasks and not have to deal with VMs.
• Other stateful applications. For example: TeamCity, Jenkins,Bamboo, DNS servers with stateful IP address, and custom stateful workloads.
• Legacy monolith applications. These applications store applicationstate on a boot disk or additional persistent disks, or they rely onstateful configuration, such as specific VM instance names, IP addresses, ormetadata key values.
• Batch workloads with checkpointing. With this configuration, you canpreserve checkpointed results of long-running computation in anticipationof workload or VM failure or instance preemption. Stateful MIGs canrecreate a failed machine, while preserving its data disk, so that yourcomputation can continue from the last checkpoint.

To achieve resilience against zonal failure, your application must replicatedata across multiple instances at the application level. For example,ElasticSearch and Cassandra support such functionality. You can use a regionalMIG to make such an application resilient to zonal failure by deployingredundant replicas to multiple zones and relying on your application's datareplication functionality. In the event of a zonal failure, your data is servedfrom available replicas in the remaining zones.

Review thelimitationsto verify if a stateful MIG fully meets your requirements.

Subsequent changes to your MIG's stateful configuration or size (for example,decreasing the MIG's size, or deleting or abandoning instances from the MIG) canaffect the preserved states ofthe instances.

Stateful configuration

A stateful managed instance group (MIG) takes its instance configuration froma combination of theinstance template,optionalall-instances configuration,optionalstateful policy, and optionalper-instance configurations that you set. After youset the configuration for your group, the MIG uses that configuration whencreating VMs. To apply an updated configuration to existing VMs, seeApply new VM configurations in a MIG.

• MIGs preserve instance names on recreation and on autohealing bydefault.
• For proactive rolling updates to preserve instance names, you must setthe Updater replacement method toRECREATE.

Stateful policy

A stateful policy defines common stateful items for all instances in a managedinstance group. Each item that you include in the stateful policy must bedefined in the MIG's instance template.

You can make the following changes to a stateful policy:

• Configure disks to become stateful byadding themto the stateful policy.
• Configure disks to become stateless byremoving themfrom the stateful policy.
• Specify that IP addresses must be stateful byaddingnetwork interface configuration to the stateful policy.
• Specify that IP addresses must be treated as stateless byremovingthe configuration from stateful policy.

Per-instance configurations

A per-instance configuration defines stateful items that are unique for aspecificmanaged instance,such as instance-specific disks, metadata key-value pairs, and IP addresses.Instance-specific metadata and disks don't need to be defined in the MIG'sinstance template; however, network interfaces for stateful IPs must bedefined in the MIG's instance template.

You can make the following changes to a per-instance configuration for aspecific instance in a MIG:

• Configure disksthat are defined in the instance template to becomestateful for the instance (by adding those disks to the per-instanceconfiguration) or to become stateless (by removing those disks from theper-instance configuration).
• Configure existing disks,not defined in the instance template, to beattached and become stateful for the instance (by adding those disks to theper-instance configuration) or to be detached from the instance (removing disksfrom the per-instance configuration).
• Add or removestateful metadata key-value pairs that are specific to the instance.
• Configuring IP addressesindividually for instances in a MIG to become stateful or stateless. Per-instance configurations for IP addresses are not supported for Dynamic Network Interfaces.

Example of stateful configuration

Here is an example of a stateful configuration:

In this chart:

• The instance template defines a common configuration for allVM instances in a MIG
• The stateful policy defines a common stateful configuration for diskswith device name,data-disk, which are defined by the instancetemplate, and which are created and attached individually to each VMinstance in the MIG.
• The per-instance configuration defines a stateful configuration for a specificVM instance named,node-1. It specifies to attach an existingdisk,my-legacy-1, to thenode-1 instance and treat it asstateful. It also specifies one metadata key value to preserve individualityfor thenode-1 instance:node-id:xyz273.

When creating thenode-1 VM, the MIG does the following:

1. Uses then2-standard-2 machine type, according to the instance template.
2. Creates and attaches a boot disk with an auto-generated disk name,boot-node-1, and device nameboot-disk, using a Debian GNU/Linuximage, according to the instance template. The MIG treats theboot-node-1boot disk as stateless because it isn't configured in the stateful policyor in the per-instance configuration.
3. Creates and attaches an additional disk with an auto-generated diskname,data-disk-1, and device name,data-disk, using a custom image,according to the instance template. The MIG treats thedata-disk-1additional disk as stateful because its device name is specified in thestateful policy.
4. Attaches an existing disk with the disk name,my-legacy-1, and usesdevice name,legacy-disk, according to the per-instance configuration. The MIGtreats themy-legacy-1 additional disk as stateful because its devicename is specified in the per-instance configuration.
5. Sets three metadata key-value pairs: two from the instance template(app:example-stateful-app,version:1.0) and one from the per-instanceconfig (node-id:xyz273). The MIG treats thenode-id:xyz273 key-valuepair as stateful because it is specified in the per-instance configuration.

When recreating thenode-1 VM, assuming the same config is stilleffective, the MIG recreates the stateless items and preserves the statefulitems:

1. Recreates the boot disk from the original image:

   First, it deletes theboot-node-1 boot disk, and then it recreates it from the Debian GNU/Linuximage, as specified in the instance template.

2. Preserves additional disks,data-disk-1 andmy-legacy-1:

   Detaches the additional disksbefore deleting the VM, and then attaches them to the VMafter it has been recreated.

3. Preserves the individual metadata key-value pair,node-id:xyz273:

   Setsthe metadata after the VM has been recreated. Also sets the commonkey-value pairs from the instance template (app:example-stateful-app andversion:1.0).

Feedback

We want to learn about your use cases, challenges, and feedback about statefulMIGs. We invite you to share your feedback with our team atmig-discuss@google.com.

What's next

• ReadConfiguring stateful MIGsto learn how to support stateful workloads by preserving instance names,persistent disks, and metadata in managed instances.
• Learn how toMigrate an existing workload to a stateful MIG.
• Learn more aboutHow stateful MIGs work.
• Learn more aboutManaged instance groups.
• Read aboutWorking with managed instances.