Repository files navigation

This module handles opinionated Google Cloud Platform Kubernetes Engine cluster creation and configuration with Node Pools, IP MASQ, Network Policy, etc.The resources/services/activations/deletions that this module will create/trigger are:

• Create a GKE cluster with the provided addons
• Create GKE Node Pool(s) with provided configuration and attach to cluster
• Replace the default kube-dns configmap ifstub_domains are provided
• Activate network policy ifnetwork_policy is true
• Addip-masq-agent configmap with providednon_masquerade_cidrs ifconfigure_ip_masq is true

Sub modules are provided for creating private clusters, beta private clusters, and beta public clusters as well. Beta sub modules allow for the use of various GKE beta features. See the modules directory for the various sub modules.

This module is meant for use with Terraform 1.3+ and tested using Terraform 1.10+.If you find incompatibilities using Terraform>=1.3, please open an issue.

If you haven'tupgraded to 1.3 and need a Terraform0.13.x-compatible version of this module, the last released versionintended for Terraform 0.13.x is [27.0.0].

If you haven'tupgraded to 0.13 and need a Terraform0.12.x-compatible version of this module, the last released versionintended for Terraform 0.12.x is12.3.0.

There are multiple examples included in theexamples folder but simple usage is as follows:

# google_client_config and kubernetes provider must be explicitly specified like the following.data"google_client_config""default" {}provider"kubernetes" {host="https://${module.gke.endpoint}"token=data.google_client_config.default.access_tokencluster_ca_certificate=base64decode(module.gke.ca_certificate)}module"gke" {source="terraform-google-modules/kubernetes-engine/google"project_id="<PROJECT ID>"name="gke-test-1"region="us-central1"zones=["us-central1-a","us-central1-b","us-central1-f"]network="vpc-01"subnetwork="us-central1-01"ip_range_pods="us-central1-01-gke-01-pods"ip_range_services="us-central1-01-gke-01-services"http_load_balancing=falsenetwork_policy=falsehorizontal_pod_autoscaling=truefilestore_csi_driver=falsedns_cache=falsenode_pools=[    {      name="default-node-pool"      machine_type="e2-medium"      node_locations="us-central1-b,us-central1-c"      min_count=1      max_count=100      local_ssd_count=0      spot=false      disk_size_gb=100      disk_type="pd-standard"      image_type="COS_CONTAINERD"      enable_gcfs=false      enable_gvnic=false      logging_variant="DEFAULT"      auto_repair=true      auto_upgrade=true      service_account="project-service-account@<PROJECT ID>.iam.gserviceaccount.com"      preemptible=false      initial_node_count=80      accelerator_count=1      accelerator_type="nvidia-l4"      gpu_driver_version="LATEST"      gpu_sharing_strategy="TIME_SHARING"      max_shared_clients_per_gpu=2    },  ]node_pools_oauth_scopes={    all= ["https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/monitoring",    ]  }node_pools_labels={    all= {}    default-node-pool= {      default-node-pool=true    }  }node_pools_metadata={    all= {}    default-node-pool= {      node-pool-metadata-custom-value="my-node-pool"    }  }node_pools_taints={    all= []    default-node-pool= [      {        key="default-node-pool"        value=true        effect="PREFER_NO_SCHEDULE"      },    ]  }node_pools_tags={    all= []    default-node-pool= ["default-node-pool",    ]  }}

Then perform the following commands on the root folder:

• terraform init to get the plugins
• terraform plan to see the infrastructure plan
• terraform apply to apply the infrastructure build
• terraform destroy to destroy the built infrastructure

object({
    enabled                     = bool
    autoscaling_profile         = string
    min_cpu_cores               = optional(number)
    max_cpu_cores               = optional(number)
    min_memory_gb               = optional(number)
    max_memory_gb               = optional(number)
    gpu_resources               = list(object({ resource_type = string, minimum = number, maximum = number }))
    auto_repair                 = bool
    auto_upgrade                = bool
    disk_size                   = optional(number)
    disk_type                   = optional(string)
    image_type                  = optional(string)
    strategy                    = optional(string)
    max_surge                   = optional(number)
    max_unavailable             = optional(number)
    node_pool_soak_duration     = optional(string)
    batch_soak_duration         = optional(string)
    batch_percentage            = optional(number)
    batch_node_count            = optional(number)
    enable_secure_boot          = optional(bool, false)
    enable_integrity_monitoring = optional(bool, true)
  })

{
  "auto_repair": true,
  "auto_upgrade": true,
  "autoscaling_profile": "BALANCED",
  "disk_size": 100,
  "disk_type": "pd-standard",
  "enable_integrity_monitoring": true,
  "enable_secure_boot": false,
  "enabled": false,
  "gpu_resources": [],
  "image_type": "COS_CONTAINERD",
  "max_cpu_cores": 0,
  "max_memory_gb": 0,
  "min_cpu_cores": 0,
  "min_memory_gb": 0
}

[
  {
    "key_name": "",
    "state": "DECRYPTED"
  }
]

[
  "8443",
  "9443",
  "15017"
]

[
  {
    "name": "default-node-pool"
  }
]

{
  "all": "",
  "default-node-pool": ""
}

{
  "all": "",
  "default-node-pool": ""
}

{
  "all": "",
  "default-node-pool": ""
}

{
  "all": {},
  "default-node-pool": {}
}

{
  "all": {},
  "default-node-pool": {}
}

{
  "all": {},
  "default-node-pool": {}
}

{
  "all": [
    "https://www.googleapis.com/auth/cloud-platform"
  ],
  "default-node-pool": []
}

{
  "all": {},
  "default-node-pool": {}
}

{
  "all": {},
  "default-node-pool": {}
}

{
  "all": [],
  "default-node-pool": []
}

{
  "all": [],
  "default-node-pool": []
}

[
  "10.0.0.0/8",
  "172.16.0.0/12",
  "192.168.0.0/16"
]

object({
    enabled            = bool
    logging_enabled    = optional(bool, false)
    monitoring_enabled = optional(bool, false)
  })

{
  "enabled": false,
  "logging_enabled": false,
  "monitoring_enabled": false
}

object({
    enable_insecure_binding_system_unauthenticated = optional(bool, null)
    enable_insecure_binding_system_authenticated   = optional(bool, null)
  })

{
  "enable_insecure_binding_system_authenticated": null,
  "enable_insecure_binding_system_unauthenticated": null
}

object({
    metadata = string
  })

{
  "metadata": "INCLUDE_ALL_METADATA"
}

Use this variable for provisioning linux based node pools. For Windows based node pools usewindows_node_pools

The node_pools variable takes the following parameters:

The windows_node_pools variable takes the same parameters asnode_pools but is reserved for provisioning Windows based node pools only. This variable is introduced to satisfy aspecific requirement for the presence of at least one linux based node pool in the cluster before a windows based node pool can be created.

Before this module can be used on a project, you must ensure that the following pre-requisites are fulfilled:

1. Terraform and kubectl areinstalled on the machine where Terraform is executed.
2. The Service Account you execute the module with has the rightpermissions.
3. The Compute Engine and Kubernetes Engine APIs areactive on the project you will launch the cluster in.
4. If you are using a Shared VPC, the APIs must also be activated on the Shared VPC host project and your service account needs the proper permissions there.

Theproject factory can be used to provision projects with the correct APIs active and the necessary Shared VPC connections.

• kubectl 1.9.x

• Terraform 1.3+
• Terraform Provider for GCP v6.47+

Some submodules use theterraform-google-gcloud module. By default, this module assumes you already have gcloud installed in your $PATH.See themodule documentation for more information.

In order to execute this module you must have a Service Account with thefollowing project roles:

• roles/compute.viewer
• roles/compute.securityAdmin (only required ifadd_cluster_firewall_rules is set totrue)
• roles/container.clusterAdmin
• roles/container.developer
• roles/iam.serviceAccountAdmin
• roles/iam.serviceAccountUser
• roles/resourcemanager.projectIamAdmin (only required ifservice_account is set tocreate)

Additionally, ifservice_account is set tocreate andgrant_registry_access is requested, the service account requires the following role on theregistry_project_ids projects:

• roles/resourcemanager.projectIamAdmin

In order to operate with the Service Account you must activate the following APIs on the project where the Service Account was created:

• Compute Engine API - compute.googleapis.com
• Kubernetes Engine API - container.googleapis.com