Overview

AProcess ID (PID) limit is a Kubernetes resource constraint on nodes and pods to prevent excessive process creation, which can impact node stability. Apigee Hybrid supports the Kubernetes feature to set process ID limits. This document provides instructions on how to set these limits, with recommendations for values for Apigee services on specific platforms.

When Apigee hybrid users manage their own clusters, setting PID limits in Kubernetes can improve system stability, security, and resource management. This is also consistent with Kubernetes best practices.

• Kernel max ID limits: Determined by the operating system and its kernel settings. Since Apigee Hybrid runs exclusively on a Linux kernel, this guide discusses Linus-based limits on Kubernetes nodes. The max number of process ID limits for a Linux kernel is 4194304.
• Kube-reserved andsystem-reserved: For resource reservation for Kubernetes or OS system daemons.
• Eviction threshold: The limit that signals pressure on the node. The node will be evicted when the threshold is met. SeePID based eviction for more details.
• Allocatable: The number of PIDs available. SeeKubernetes: Node Allocatable for more information. Kube-reserved and system-reserved can be configured with Node PID limit settings.

The pod PID limit can be configured for the node and shared across all pods within the node.

Prepare to manage the process ID limits

These procedures uses the following environment variables:

export PROJECT_ID=MY_PROJECT_IDexport CLUSTER_NAME=MY_CLUSTER_NAMEexport LOCATION=MY_CLUSTER_LOCATIONexport APIGEE_NAMESPACE=MY_APIGEE_NAMESPACE # Default: apigee

Verify access

Before configuring the process ID limit, ensure that you have access to edit Kubernetes clusters.

The following steps are for installations on GKE. For other platforms, see the documentation for your platform.

1. Check if you have roles/container.clusterAdmin in the IAM policy:

   gcloud projects get-iam-policy ${PROJECT_ID}  \ --flatten="bindings[].members" \ --format='table(bindings.role)' \ --filter="bindings.members:your_account_email"

2. If you do not have the access, add the role to your account:

   gcloud projects add-iam-policy-binding ${PROJECT_ID} \ --member user:your_account_email \ --role roles/container.clusterAdmin

Verify Existing PID Limits

Verify whether the node has existing PID limits before configuring new limits.

1. Get a node from the cluster to check values. You will need to check nodes from both theapigee-data andapigee-runtime node pools:

   kubectl get nodes -n ${APIGEE_NAMESPACE}

   Your output should look similar to the following:

   NAME                                      STATUS   ROLES    AGE    VERSIONgke-my-hybrid-apigee-data-0a1b2c3d-efgh      Ready   2d8h   v1.31.5-gke.1169000gke-my-hybrid-apigee-runtime-1b2c3d4e-fghi   Ready   2d8h   v1.31.5-gke.1169000

2. Export the node name from the output of the previous step. Run this the next step first for anapigee-data node and again for anapigee-runtime node:

   Code

   export NODE_NAME=MY_NODE_NAME

   Example

   export NODE_NAME="gke-my-hybrid-apigee-data-0a1b2c3d-efgh"

3. Check thenode PID limits. Use the following commands to check the reserved values. If the value is null, the value is not configured:

   kubectl get --raw "/api/v1/nodes/${NODE_NAME}/proxy/configz" | jq '.kubeletconfig.kubeReserved'kubectl get --raw "/api/v1/nodes/${NODE_NAME}/proxy/configz" | jq '.kubeletconfig.systemReserved'kubectl get --raw "/api/v1/nodes/${NODE_NAME}/proxy/configz" | jq '.kubeletconfig.evictionHard'

4. Check thepod PID limit. Use the following command to check the existing pod PID limit. If the returned value is-1 or empty, the limit is not set:

   kubectl get --raw "/api/v1/nodes/${NODE_NAME}/proxy/configz" | jq '.kubeletconfig.podPidsLimit'

Manage the process ID limits

Manage the Node PID limits

For installations on GKE, the infrastructure resources of Kubernetes nodes are managed internally and you do not need to configure them. The current capacity and allocatable resources can be viewed followingNode allocatable resources in the Google Kubernetes Engine documentation.

For non-GKE platforms, refer to corresponding Kubernetes documentation for the platform. If the cluster/node is user-managed (as opposed to Fully-managed), the kube-reserved PID limit and system-reserved PID limit can be configured with Kubelet. SeeNode PID limits in the Kubernetes documentation.

Tooling

This procedure uses Kubelet to manage the process ID limit. Kubelet is the agent that runs on pods and containers that ensures they are running according to their PodSpec. If you need to install Kubelet, follow the instructions in the Kubernetes documentation:Installing kubeadm, kubelet and kubectl.

Procedure

1. Create a Kubelet configuration file calledkubelet-config.yaml.

   apiVersion: kubelet.config.k8s.io/v1beta1kind: KubeletConfigurationkubeReserved: pid:PID_VALUE # Example: 1000

   See more info on the configurations:Kube Reserved in the Kubernetes documentation.

2. Apply the Kubelet configuration:

   kubelet --configPATH_TO_KUBELET_CONFIG_YAML

Manage the Pod PID limits

Choosing the Limit

If the PID limit is set too low, it can prevent a pod from starting up. If it is set too high, it can fail to detect resource misbehaviors. When selecting the appropriate limit, it is important to consider the previous behavior of the node and any service-specific requirements.

GKE has a required range for the value: [1024, 4194304]. On GKE Platforms you can see the status of your Kubernetes service accounts in the Google Cloud consoleMetrics Explorer. SelectKubernetes Node - PID usage metrics and apply filters. This metrics shows the recent usage of process IDs and can be referenced when choosing a PID limit.

Metrics Explorer

On non-GKE platforms, different monitoring options may be available. Please refer to the corresponding platform's Kubernetes documentation to check the metrics.

Process ID requirements for Apigee pods

Apigee Hybrid uses 2 node pools, apigee-data and apigee-runtime. Since some of the Apigee components are deployed in both node pools, the Pod PID limit shall be the same for the two node pools. The Pod PID limit should also be higher than the maximum number of PIDs required across all Apigee pods. The required Apigee Pod PID limit is 1000, which is below the minimum required number for GKE platforms.

Recommended Pod PIDs Limit

Some platforms enforce a minimum value requirement for the number of the Pod PIDs limit, in which case the min value requirement is selected.

Procedure

The procedures for managing the Pod PID limits are different for GKE and non-GKE platforms.

Troubleshooting process ID limits

Pod stuck inPending status with aFailedScheduling error

When the Node or Pod PIDs limit evicts or restricts a pod from starting up, the pod will be stuck inPending status and fail withFailedScheduling error.

1. Get the Node column:

   kubectl get pods -n ${APIGEE_NAMESPACE} ${POD_NAME} -o wide

2. Check if there's aPIDPressure condition:

   kubectl describe node -n apigee ${NODE_NAME} | grep PIDPressure

3. Alternatively, checkApigeeDeployment of the corresponding pod. Get theApigeeDeployment from the result that has the same prefix as the erroring pod.

   kubectl get ApigeeDeployment -n ${APIGEE_NAMESPACE}

4. Check if the recentEvents has PID-related error messages:

   kubectl describe ApigeeDeployment -n ${APIGEE_NAMESPACE} ${APIGEE_DEPLOYMENT_NAME}

5. If the cause is confirmed to be PID limit, follow the steps inManage the Node PID limits to update the PID limit to a higher value.

InvalidpodPidsLimit

When setting the limit for GKE, if thepodPidsLimit is beyond limit, error will prompt:

ERROR: (gcloud.container.node-pools.update) ResponseError: code=400, message=Invalid podPidsLimit: value must be 1024<= podPidsLimit<= 4194304.

Update the podPidsLimit value to within the required range.