bmctl

1. Change the values of the applicable fields in the cluster's configurationfile, which by default is located here:
   bmctl-workspace/CLUSTER-NAME/CLUSTER-NAME.yaml

2. Update the cluster by running thebmctl update command:

   bmctlupdatecluster-cCLUSTER_NAME--kubeconfig=KUBECONFIG

   Replace the following:

   • CLUSTER_NAME: the name of the cluster you want toupdate.
   • KUBECONFIG: for admin, hybrid, or standaloneclusters, enter the path to the cluster's kubeconfig file. For a usercluster, enter the path to theadmin cluster's kubeconfig file.

If you then attempt to make changes to the cluster using the clusterconfiguration with thekubectl apply -f command, thespec.nodeConfig.registryMirrors populated by the reconcileron the cluster is removed. This behavior happens because the configurationdoesn't exist in the cluster configuration spec, but it does in the cluster.

gcloud CLI

1. Specify only the flags for the configuration that you want to modify.

2. Run the applicable update command:

   • Admin clusters:gcloud container bare-metal admin-clusters update

   • User clusters:gcloud container bare-metal clusters update.

   • Node pools on a user cluster:gcloud container bare-metal node-pools update

Terraform

1. Change the values of the applicable fields in the Terraformconfiguration file that you used to create the cluster or node pool. Seethe Terraform reference documentation for detailed field descriptions:

   • google_gkeonprem_bare_metal_admin_cluster

   • google_gkeonprem_bare_metal_cluster

   • google_gkeonprem_bare_metal_node_pool

2. Update the configuration by running theterraform apply command.

bmctl

You add or remove a node from a node pool by adding or removing the IP addressof the node in a specific section of the cluster configuration file. Thefollowing list shows which section to edit for a given node pool:

• Worker node pool: add or remove the IP address of the node in thespec.nodes section of theNodePool spec.
• Control plane node pool: add or remove the IP address of the node in thespec.controlPlane.nodePoolSpec.nodes section of theCluster spec.
• Load balancer node pool: add or remove the IP address of the node in thespec.loadBalancer.nodePoolSpec.nodes section of theCluster spec.

Example: remove a worker node

Here's a sample cluster configuration file that shows the specifications oftwo worker nodes:

---apiVersion:baremetal.cluster.gke.io/v1kind:NodePoolmetadata:name:nodepool1namespace:cluster-cluster1spec:clusterName:cluster1nodes:-address:192.0.2.1-address:192.0.2.2

To remove a node:

1. (Optional) If the node that you're removing is running critical Pods,first put the node intomaintenance mode.

   You can monitor the node draining process for worker nodes by viewing thestatus.nodesDrained andstatus.nodesDraining fields on theNodePool resource.

2. Edit the cluster configuration file todelete the IP address entry for thenode.

3. Update the cluster:

   bmctlupdatecluster1\--kubeconfig=ADMIN_KUBECONFIG

gcloud CLI

You use anupdate command to add or remove nodes. Theupdate command thatyou use and the flag in which you specify the IP address depends on the typeof node pool that you want to update:

• Worker node pool: Rungcloud container bare-metal node-pools update andspecify the IP address in the flag--node-configs 'node-ip=IP_ADDRESS'.

• Control plane node pool on an admin cluster: Rungcloud container bare-metal admin-clusters update and specify the IPaddress in the flag--control-plane-node-configs 'node-ip=IP_ADDRESS'.

• Control plane node pool on a user cluster: Rungcloud container bare-metal clusters update and specify the IPaddress in the flag--control-plane-node-configs 'node-ip=IP_ADDRESS'.

• Load balancer node pool: Rungcloud container bare-metal clusters update and specify the IPaddress in the flag--metal-lb-load-balancer-node-configs 'node-ip=IP_ADDRESS'or
  --bgp-load-balancer-node-configs 'node-ip=IP_ADDRESS'

The flag in which you specify the IP address only accepts onenode-ip. Youinclude the flag for each IP address in the node pool.

Theupdate commands replace all IP addresses with the IP addresses thatyou specify. To add a node, include the IP addresses of the existingnodes and the IP address of the new node in theupdate command. Similarly,you remove nodes by only including the IP addresses of the nodes that youwant to keep.

Example: remove a worker node

This section shows how to remove a worker node from a node pool usingexample data. Additional gcloud CLI commands that you mightfind useful are also included in the following steps.

1. (Optional) If the node that you're removing is running critical Pods,first put the node intomaintenance mode.

   You can monitor the node draining process for worker nodes by viewing thestatus.nodesDrained andstatus.nodesDraining fields on theNodePool resource.

2. Run thelist command to list all the node pools in the cluster:

   gcloud container bare-metal node-pools list \    --cluster=abm-user-cluster1 \    --project=example-project-12345 \    --location=us-central1

   The output is similar to the following:

   NAME         LOCATION     STATEnode-pool-1  us-central1  RUNNINGnode-pool-2  asia-east1   RUNNING

3. Run thedescribe command to list all the IP addresses in the node pool:

   gcloud container bare-metal node-pools describe node-pool-1 \    --cluster=abm-user-cluster1 \    --project=example-project-12345 \    --location=us-central1

   The following example output is truncated for readability:

   annotations:  ...  baremetal.cluster.gke.io/version: 1.33...name: projects/example-project-12345/locations/us-central1/bareMetalClusters/abm-user-cluster1/bareMetalNodePools/node-pool-1nodePoolConfig:  nodeConfigs:  - nodeIp: 192.0.2.1  - nodeIp: 192.0.2.2  operatingSystem: LINUXstate: RUNNING...

   Note the following in the example output:

   • Thename field contains the fully-qualified name of the node pool.When specifying the node pool name in a command, you can either specifythe fully-qualified name, or the node pool name, for example,node-pool-1, along with the--cluster,--project, and the--location flags.

   • ThenodeConfigs section contains twonodeIp fields with the IPaddresses of the nodes.

4. Run the following command to remove the node with the IP address192.0.2.1:

   gcloud container bare-metal node-pools update node-pool-1 \    --cluster=abm-user-cluster1 \    --project=example-project-12345 \    --location=us-central1 \    --node-configs='node-ip=192.0.2.2'

   Theupdate command replaces all IP addresses with the IP addresses thatyou specify. Because 192.0.2.1 isn't included, the node is removed.

   The output of the command is similar to the following:

   Waiting for operation [projects/example-project-12345/locations/us-central1/operations/operation-1697154681749-6078d9def4030-76686d6e-9fcb1de9] to complete

   In the example output, the stringoperation-1697154681749-6078d9def4030-76686d6e-9fcb1de9is theOPERATION_ID of the long-running operation.You can find out the status of the operation by running the followingcommand in another terminal window:

   gcloud container bare-metal operations describe operation-1697154681749-6078d9def4030-76686d6e-9fcb1de9 \    --project= example-project-12345 \    --location=us-central1

   You can rerun the command every so often to check the status.

bmctl

You can usebmctl to replace high availability (HA) control plane nodes inall cluster types.

You replace a node in a cluster by performing the following steps:

1. Remove the node's IP address from the cluster configuration file.
2. Update the cluster.
3. Check the status of nodes in the cluster.
4. Add a new node's IP address to the same cluster configuration file.
5. Update the cluster.

Example: replace a HA control plane node

Here's a sample cluster configuration file that shows three control planenodes in a user cluster:

---apiVersion:baremetal.cluster.gke.io/v1kind:Clustermetadata:name:user-clusternamespace:cluster-user-clusterspec:controlPlane:nodePoolSpec:nodes:-address:192.0.2.11-address:192.0.2.12-address:192.0.2.13

To replace the last node listed in thespec.controlPlane.nodePoolSpec.nodessection, perform the following steps:

1. Remove the node by deleting its IP address entry in the clusterconfiguration file. After making this change, the cluster configuration fileshould look something like this:

   ---apiVersion:baremetal.cluster.gke.io/v1kind:Clustermetadata:name:user-clusternamespace:cluster-user-clusterspec:controlPlane:nodePoolSpec:nodes:-address:192.0.2.11-address:192.0.2.12

2. Update the cluster by running the following command:

   bmctl update cluster -cCLUSTER_NAME \  --kubeconfig=KUBECONFIG

   Make the following changes:

   • ReplaceCLUSTER_NAME with the name of the cluster you want toupdate.
   • If the cluster is a self-managing cluster (such as admin or standalonecluster), replaceKUBECONFIG with the path to the cluster'skubeconfig file. If the cluster is a user cluster, as it is in thisexample, replaceKUBECONFIG with the path to theadmincluster's kubeconfig file.

3. After thebmctl update command has executed successfully, it takes sometime to complete themachine-preflight andmachine-init jobs. You canview the status of nodes and their respective node pools by running thecommands described in theVerify your updates sectionof this document. After the node pool and nodes are in a ready state, youcan proceed to the next step.

4. Add a new control plane node to the node pool by adding the IP address ofthe new control plane node to thespec.controlPlane.nodePoolSpec.nodessection of the cluster configuration file. After making this change, thecluster configuration file should look something like this:

   ---apiVersion:baremetal.cluster.gke.io/v1kind:Clustermetadata:name:user-clusternamespace:cluster-user-clusterspec:controlPlane:nodePoolSpec:nodes:-address:192.0.2.11-address:192.0.2.12-address:192.0.2.14

5. Update the cluster by running the following command:

   bmctl update cluster -cCLUSTER_NAME \  --kubeconfig=KUBECONFIG

gcloud CLI

You can use the gcloud CLI to replace high availability (HA)control plane nodes in admin and user clusters.

You replace a node in a cluster by performing the following steps:

1. Remove the node's IP address by running the applicableupdate command:

   • User cluster:gcloud container bare-metal clusters update
   • Admin cluster:gcloud container bare-metal admin-clusters update

2. Check the status of the node removal in the cluster by runninggcloud container bare-metal operations describeOPERATION_ID.

3. Add the new node's IP address by running the applicableupdate command.

Example: replace a HA control plane node

This section shows how to replace a control plane from a cluster usingexample data. Additional gcloud CLI commands that you mightfind useful are also included in the following steps.

1. Run thelist command to list all the user clusters in aGoogle Cloud project:

   gcloud container bare-metal clusters list \    --project=example-project-12345 \    --location=-

   When you set--location=-, that means to list all clusters in allregions. If you need to scope down the list, set--location to aspecific region.

   The output is similar to the following:

   NAME                 LOCATION      VERSION   ADMIN_CLUSTER        STATEabm-user-cluster1a   us-central1   1.33      abm-admin-cluster1   RUNNINGabm-user-cluster1b   europe-west1  1.33      abm-admin-cluster1   RUNNING

2. Run thedescribe command on the cluster:

   gcloud container bare-metal clusters describe abm-user-cluster1  \    --project=example-project-12345 \    --location=us-central1

   The example output is truncated for readability:

   ...controlPlane:  controlPlaneNodePoolConfig:    nodePoolConfig:      nodeConfigs:      - nodeIp: 192.0.2.11      - nodeIp: 192.0.2.12      - nodeIp: 192.0.2.13      operatingSystem: LINUX...name: projects/example-project-1234567/locations/us-central1/bareMetalClusters/abm-user-cluster1a...

   Note the following in the example output:

   • Thename field contains the fully-qualified name of the cluster.When specifying the cluster name in a command, you can either specifythe fully-qualified name, or the cluster name, for example,abm-user-cluster1a, along with the--project and the--location flags.

   • ThenodeConfigs section contains threenodeIp fields with the IPaddresses of the control plane nodes.

3. Remove the node with the IP address192.0.2.13:

   gcloud container bare-metal cluster update abm-user-cluster1a \    --project=example-project-12345 \    --location=us-central1 \    --control-plane-node-configs 'node-ip=192.0.2.11'    --control-plane-node-configs 'node-ip=192.0.2.12'

   The output of the command is similar to the following:

   Waiting for operation [projects/example-project-12345/locations/us-central1/operations/operation-1956154681749-6078d9def4030-76686d6e-9fcb1d7] to complete

   In the example output, the stringoperation-1956154681749-6078d9def4030-76686d6e-9fcb1de7is theOPERATION_ID of the long-running operation.You can find out the status of the operation by running the followingcommand in another terminal window:

   gcloud container bare-metal operations describe operation-1956154681749-6078d9def4030-76686d6e-9fcb1de7 \    --project= example-project-12345 \    --location=us-central1

   You can rerun the command every so often to check the status.

4. Add the new node with the IP address192.0.2.14:

   gcloud container bare-metal cluster update abm-user-cluster1a \    --project=example-project-12345 \    --location=us-central1 \    --control-plane-node-configs 'node-ip=192.0.2.11'    --control-plane-node-configs 'node-ip=192.0.2.12'    --control-plane-node-configs 'node-ip=192.0.2.14'

kubectl

You can view the status of nodes and their respective node pools with thekubectl get command.

For example, the following command shows the status of the node pools in thecluster namespacecluster-my-cluster:

kubectl-ncluster-my-clustergetnodepools.baremetal.cluster.gke.io

The system returns results similar to the following:

NAME                    READY   RECONCILING   STALLED   UNDERMAINTENANCE   UNKNOWNcluster-my-cluster      3       0             0         0                  0cluster-my-cluster-lb   2       0             0         0                  0np1                     3       0             0         0                  0

Reconciling=1 means that the reconciliation step is still in progress. Youshould wait until the status changes toReconciling=0.

You can also check status of nodes in the a cluster by running followingcommand:

kubectlgetnodes--kubeconfig=KUBECONFIG

gcloud CLI

As described previously, after running anupdate command, you cancheck the status of the operation usinggcloud container bare-metaloperations describeOPERATIONS_ID. The output ofthe command gives the status of the nodes, for example:

  ...  metrics:  - intValue: '1'    metric: NODES_RECONCILING  - intValue: '2'    metric: NODES_HEALTHY  - intValue: '0'    metric: NODES_FAILED  - intValue: '0'    metric: NODES_IN_MAINTENANCE  - intValue: '3'    metric: NODES_TOTAL  stage: HEALTH_CHECK...

No matter which tool you use to update a node pool, you can get the currentstatus of a node pool by running the applicabledescribe command as shownpreviously.

bmctl

TheaddressPoolssection contains fields for specifying load-balancing pools for the MetalLBand Border Gateway Protocol (BGP) bundled load balancers. You can add moreload-balancing address pools at any time, but you can't remove any existingaddress pools. Starting with Google Distributed Cloud version 1.16.0, you canmodify the values foraddressPools.avoidBuggyIPs andaddressPools.manualAssign at any time.

addressPools:-name:pool1addresses:-198.51.100.0-198.51.100.4-198.51.100.240/28-name:pool2addresses:-198.51.100.224/28

gcloud CLI

You can add more load-balancing address pools at any time for the bundledload balancers, but you can't remove any existing address pools. The flag thatyou specify ingcloud container bare-metal clusters update to add an addresspool depends on the type of bundled load balancer:

• MetalLB (layer 2): Use the--metal-lb-address-pools flag.
• Border Gateway Protocol (BGP): Use the--bgp-address-pools flag.

The value for the flags have the following format:

'pool=NAME,avoid-buggy-ips=True|False,manual-assign=True|False,addresses=IP_ADDRESS_RANGE_1;IP_ADDRESS_RANGE_2;...' \

The value has segments that start with the keywordspool,avoid-buggy-ip,manual-assign, andaddresses. Separate each segmentwith a comma.

• pool: A name of your choice for the pool.

• avoid-buggy-ips: If you set this toTrue, the IP address management(IPAM) controller won't assign IP addresses ending in.0 or.255 toServices. This avoids the problem of buggy consumer devices mistakenlydropping traffic sent to those special IP addresses. If not specified,defaults toFalse. Starting with Google Distributed Cloud version 1.16.0,you can modify this value in an existing address pool.

• manual-assign: If you don't want the IPAM controller toautomatically assign IP addresses from this pool to Services, setthis toTrue. Then a developer can create a Service of typeLoadBalancer and manually specify one of the addresses from thepool. If not specified,manual-assign is set toFalse.Starting with Google Distributed Cloud version 1.16.0, you can modifythis value in an existing address pool.

• In the list ofaddresses: Each address must be a range either inCIDR orhyphenated-range format. To specify a single IP address in a pool(such as for the ingress VIP), use/32 in CIDR notation(for example, 192.0.2.1/32).

Note the following syntax rules:

• Surround the entire value in single quotes.
• Whitespace isn't allowed.
• Separate each IP address range with a semicolon.

You can specify more than one instance of the flag, as shown in thefollowing example:

--metal-lb-address-pools='pool=pool2,avoid-buggy-ips=False,manual-assign=True,addresses=198.51.100.0/30;198.51.100.64-198.51.100.72'--metal-lb-address-pools='pool=pool3,avoid-buggy-ips=True,manual-assign=True,addresses=203.0.113.0/28'

bmctl

If you add thebaremetal.cluster.gke.io/prevent-deletion: "true" annotationto your cluster configuration file, you are prevented from deleting thecluster. For example, runningkubectl delete cluster orbmctl resetcluster produces an error.

apiVersion:baremetal.cluster.gke.io/v1kind:Clustermetadata:name:ci-10c3c6f4d9c698enamespace:cluster-ci-10c3c6f4d9c698eannotations:baremetal.cluster.gke.io/prevent-deletion:"true"spec:clusterNetwork:

gcloud CLI

If you specify the--add-annotations flag with the valuebaremetal.cluster.gke.io/prevent-deletion="true", you are prevented fromdeleting the cluster. For example:

1. Add the annotation to prevent accidental cluster deletion:

   gcloud container bare-metal clusters update abm-user-cluster1a \    --project=example-project-12345 \    --location=us-central1 \    --add-annotations=baremetal.cluster.gke.io/prevent-deletion="true"

2. Attempt to delete the user cluster:

   gcloud container bare-metal clusters delete abm-user-cluster1a \    --project=example-project-12345 \    --location=us-central1 \    --force \    --allow-missing

   The response from the command is similar to the following:

   ERROR: (gcloud.container.bare-metal.clusters.delete) INVALID_ARGUMENT:invalid request: admission webhook "vcluster.kb.io" denied the request:annotations[baremetal.cluster.gke.io/prevent-deletion]: Invalid value:"true": Annotation "baremetal.cluster.gke.io/prevent-deletion" should beremoved in order to delete this cluster

   To remove the annotation, specify--remove-annotations=baremetal.cluster.gke.io/prevent-deletion="true"in theupdate command.

bmctl

You can add or remove a user or service account as a cluster administratorfor a user cluster by specifying email addresses in theclusterSecurity.authorization.clusterAdmin.gcpAccountssection of the cluster configuration file. The accounts are granted thecluster-admin role on the cluster, providing full access to the cluster.

apiVersion:baremetal.cluster.gke.io/v1kind:Clustermetadata:name:cluster1namespace:cluster-cluster1spec:clusterSecurity:authorization:clusterAdmin:gcpAccounts:-alex@example.com-hao@example.com-my-sa@example-project-12345.iam.gserviceaccount.com

When updating a user cluster to add an account, be sure to include allaccounts in the list (both existing and new accounts) becausebmctl updateoverwrites the list with what you specify in the configuration file. Toremove an account, remove it from the cluster configuration file and runbmctl update.

gcloud CLI

You can add or remove a user or service account as a cluster administratorby specifying an email addresses in the--admin-users flag. The flagaccepts only one email address. To add multiple users, specify one accountin each flag, for example:

gcloudcontainerbare-metalclustersupdateabm-user-cluster1a\--project=example-project-12345\--location=us-central1\--admin-users=alex@example.com\--admin-users=hao@example.com--admin-users=my-sa@example-project-12345.iam.gserviceaccount.com

Theupdate command overwrites the whole grant list. Specify all existing andnew users who you want to be cluster administrators.

bmctl

apiVersion:baremetal.cluster.gke.io/v1kind:Clustermetadata:name:cluster1namespace:cluster-cluster1annotations:baremetal.cluster.gke.io/private-mode:"true"spec:nodeAccess:loginUser:abm

gcloud CLI

You specify the user that you want to use to access node machines in the--login-user flag, for example:

gcloudcontainerbare-metalclustersupdateabm-user-cluster1a\--project=example-project-12345\--location=us-central1\--login-user=abm

bmctl

SetclusterNetwork.advancedNetworkingtotruein the cluster configuration when you create your cluster:

apiVersion:baremetal.cluster.gke.io/v1kind:Clustermetadata:name:cluster1namespace:cluster-cluster1spec:clusterNetwork:...advancedNetworking:true...

gcloud CLI

Include the--enable-advanced-networking flag inthegcloud container bare-metal clusters create command when you create your cluster.

bmctl

apiVersion:baremetal.cluster.gke.io/v1kind:Clustermetadata:name:cluster1namespace:cluster-cluster1spec:...clusterNetwork:services:cidrBlocks:-192.0.2.0/14...

gcloud CLI

To increase the range of the IPv4 service CIDR on a user cluster, specifythe new range in the--island-mode-service-address-cidr-blocks flag.

gcloudcontainerbare-metalclustersupdatecluster1\--project=example-project-12345\--location=us-central1\--island-mode-service-address-cidr-blocks=192.0.2.0/14

bmctl

The following supported fields control Artifact Registry pull operations forkubelet:

For more information about each of the kubelet configuration fields, see theCluster configuration field reference.

You can specify these fields inkubeletConfig sections of the Cluster spec andthe NodePool spec for the following pools of nodes:

• Cluster spec:
• Control plane nodesspec.controlPlane.nodePoolSpec.kubeletConfig
• Load balancer nodesspec.loadBalancer.nodePoolSpec.kubeletConfig
• NodePool spec:
• Worker nodesspec.kubeletConfig

The following example shows the added fields with theirdefault values inthe cluster configuration file. Note that the annotationpreview.baremetal.cluster.gke.io/custom-kubelet: "enable" is required.

apiVersion:baremetal.cluster.gke.io/v1kind:Clustermetadata:name:cluster1namespace:cluster-cluster1annotations:preview.baremetal.cluster.gke.io/custom-kubelet:"enable"spec:...controlPlane:nodePoolSpec:kubeletConfig:registryBurst:10registryPullQPS:5serializeImagePulls:true...loadBalancer:nodePoolSpec:kubeletConfig:registryBurst:10registryPullQPS:5serializeImagePulls:true...apiVersion:baremetal.cluster.gke.io/v1kind:NodePoolmetadata:name:node-pool-newnamespace:cluster-cluster1spec:clusterName:cluster1...kubeletConfig:registryBurst:10registryPullQPS:5serializeImagePulls:true

In each case, the setting applies to all nodes in the pool.

gcloud CLI

The following flags control Artifact Registry pull operations forkubelet:

• Control plane nodes

  • --control-plane-registry-burst
  • --control-plane-registry-pull-qps
  • --disable-control-plane-serialize-image-pulls
  • --enable-control-plane-serialize-image-pulls

• Load balancer nodes

  • --bgp-load-balancer-registry-burst
  • --bgp-load-balancer-registry-pull-qps
  • --disable-bgp-load-balancer-serialize-image-pulls
  • --enable-bgp-load-balancer-serialize-image-pulls
  • --metal-lb-load-balancer-registry-burst
  • --metal-lb-load-balancer-registry-pull-qps
  • --disable-metal-lb-load-balancer-serialize-image-pull
  • --enable-metal-lb-load-balancer-serialize-image-pulls

• Worker nodes

  • --registry-burst
  • --registry-pull-qps
  • --disable-serialize-image-pulls
  • --enable-serialize-image-pulls

bmctl

• bmctl command reference

• Cluster configuration field reference

gcloud CLI

• gcloud container bare-metal clusters

• gcloud container bare-metal node-pools

• gcloud container bare-metal admin-clusters

Terraform

• google_gkeonprem_bare_metal_cluster

• google_gkeonprem_bare_metal_node_pool

• google_gkeonprem_bare_metal_admin_cluster