Prepare the environment

Clone the example repository and open the tutorial directory:

gitclonehttps://github.com/GoogleCloudPlatform/kubernetes-engine-samples.gitcdkubernetes-engine-samples/ai-ml/t5-model-serving

Create the cluster

Run the following command:

gcloudcontainerclusterscreate-automl-cluster\--release-channel=RELEASE_CHANNEL\--cluster-version=CLUSTER_VERSION\--location=us-central1

Replace the following:

• RELEASE_CHANNEL: the release channel for your cluster.Must be one ofrapid,regular, orstable. Choose a channel that hasGKE version 1.28.3-gke.1203000 or later to use L4 GPUs. Tosee the versions available in a specific channel, seeView the default and available versions for release channels.
• CLUSTER_VERSION: the GKE version touse. Must be1.28.3-gke.1203000 or later.

This operation takes several minutes to complete.

Create an Artifact Registry repository

1. Create a new Artifact Registry standard repository with the Docker formatin the same region as your cluster:

   gcloudartifactsrepositoriescreatemodels\--repository-format=docker\--location=us-central1\--description="Repo for T5 serving image"

2. Verify the repository name:

   gcloudartifactsrepositoriesdescribemodels\--location=us-central1

   The output is similar to the following:

   Encryption: Google-managed keyRepository Size: 0.000MBcreateTime: '2023-06-14T15:48:35.267196Z'description: Repo for T5 serving imageformat: DOCKERmode: STANDARD_REPOSITORYname: projects/PROJECT_ID/locations/us-central1/repositories/modelsupdateTime: '2023-06-14T15:48:35.267196Z'

Package the model

In this section, you package the model and the serving framework in a singlecontainer image using Cloud Build and push the resulting image to theArtifact Registry repository.

1. Review the Dockerfile for the container image:

   # Copyright 2023 Google LLC## Licensed under the Apache License, Version 2.0 (the "License");# you may not use this file except in compliance with the License.# You may obtain a copy of the License at##     https://www.apache.org/licenses/LICENSE-2.0## Unless required by applicable law or agreed to in writing, software# distributed under the License is distributed on an "AS IS" BASIS,# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.# See the License for the specific language governing permissions and# limitations under the License.ARGBASE_IMAGE=pytorch/torchserve:0.12.0-cpuFROMalpine/gitARGMODEL_NAME=t5-smallARGMODEL_REPO=https://huggingface.co/${MODEL_NAME}ENVMODEL_NAME=${MODEL_NAME}ENVMODEL_VERSION=${MODEL_VERSION}RUNgitclone"${MODEL_REPO}"/modelFROM${BASE_IMAGE}ARGMODEL_NAME=t5-smallARGMODEL_VERSION=1.0ENVMODEL_NAME=${MODEL_NAME}ENVMODEL_VERSION=${MODEL_VERSION}COPY--from=0/model/./home/model-server/COPYhandler.py\model.py\requirements.txt\setup_config.json/home/model-server/RUNtorch-model-archiver\--model-name="${MODEL_NAME}"\--version="${MODEL_VERSION}"\--model-file="model.py"\--serialized-file="pytorch_model.bin"\--handler="handler.py"\--extra-files="config.json,spiece.model,tokenizer.json,setup_config.json"\--runtime="python"\--export-path="model-store"\--requirements-file="requirements.txt"FROM${BASE_IMAGE}ENVPATH/home/model-server/.local/bin:$PATHENVTS_CONFIG_FILE/home/model-server/config.properties# CPU inference will throw a warning cuda warning (not error)# Could not load dynamic library 'libnvinfer_plugin.so.7'# This is expected behaviour. see: https://stackoverflow.com/a/61137388ENVTF_CPP_MIN_LOG_LEVEL2COPY--from=1/home/model-server/model-store//home/model-server/model-storeCOPYconfig.properties/home/model-server/

   This Dockerfile defines the following multiple stage build process:

   1. Download the model artifacts from the Hugging Face repository.
   2. Package the model using thePyTorch Serving Archive tool. This creates a model archive (.mar) file that the inference serveruses to load the model.
   3. Build the final image with PyTorch Serve.

2. Build and push the image using Cloud Build:

   gcloudbuildssubmitmodel/\--region=us-central1\--config=model/cloudbuild.yaml\--substitutions=_LOCATION=us-central1,_MACHINE=gpu,_MODEL_NAME=t5-small,_MODEL_VERSION=1.0

   The build process takes several minutes to complete. If you use a larger modelsize thant5-small, the build process might takesignificantly moretime.

3. Check that the image is in the repository:

   gcloudartifactsdockerimageslistus-central1-docker.pkg.dev/PROJECT_ID/models

   ReplacePROJECT_ID with your Google Cloudproject ID.

   The output is similar to the following:

   IMAGE                                                     DIGEST         CREATE_TIME          UPDATE_TIMEus-central1-docker.pkg.dev/PROJECT_ID/models/t5-small     sha256:0cd...  2023-06-14T12:06:38  2023-06-14T12:06:38

Deploy the packaged model to GKE

To deploy the image, this tutorial use Kubernetes Deployments. ADeployment is a Kubernetes API object that lets you run multiple replicas of Pods that are distributed among the nodes in a cluster..

Modify the Kubernetes manifest in the example repository tomatch your environment.

1. Review the manifest for the inference workload:

   # Copyright 2023 Google LLC## Licensed under the Apache License, Version 2.0 (the "License");# you may not use this file except in compliance with the License.# You may obtain a copy of the License at##     https://www.apache.org/licenses/LICENSE-2.0## Unless required by applicable law or agreed to in writing, software# distributed under the License is distributed on an "AS IS" BASIS,# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.# See the License for the specific language governing permissions and# limitations under the License.---apiVersion:apps/v1kind:Deploymentmetadata:name:t5-inferencelabels:model:t5version:v1.0machine:gpuspec:replicas:1selector:matchLabels:model:t5version:v1.0machine:gputemplate:metadata:labels:model:t5version:v1.0machine:gpuspec:nodeSelector:cloud.google.com/gke-accelerator:nvidia-l4securityContext:fsGroup:1000runAsUser:1000runAsGroup:1000containers:-name:inferenceimage:us-central1-docker.pkg.dev/PROJECT_ID/models/t5-small:1.0-gpuimagePullPolicy:IfNotPresentargs:["torchserve","--start","--foreground"]resources:limits:nvidia.com/gpu:"1"cpu:"3000m"memory:16Giephemeral-storage:10Girequests:nvidia.com/gpu:"1"cpu:"3000m"memory:16Giephemeral-storage:10Giports:-containerPort:8080name:http-containerPort:8081name:management-containerPort:8082name:metricsreadinessProbe:httpGet:path:/pingport:httpinitialDelaySeconds:120failureThreshold:10livenessProbe:httpGet:path:/models/t5-smallport:managementinitialDelaySeconds:150periodSeconds:5---apiVersion:v1kind:Servicemetadata:name:t5-inferencelabels:model:t5version:v1.0machine:gpuspec:type:ClusterIPselector:model:t5version:v1.0machine:gpuports:-port:8080name:httptargetPort:http-port:8081name:managementtargetPort:management-port:8082name:metricstargetPort:metrics

2. ReplacePROJECT_ID with your Google Cloudproject ID:

   sed-i"s/PROJECT_ID/PROJECT_ID/g""kubernetes/serving-gpu.yaml"

   This ensures that the container image path in the Deployment specificationmatches the path to your T5 model image in Artifact Registry.

3. Create the Kubernetes resources:

   kubectlcreate-fkubernetes/serving-gpu.yaml

To verify that the model deployed successfully, do the following:

1. Get the status of the Deployment and the Service:

   kubectlget-fkubernetes/serving-gpu.yaml

   Wait until the output shows ready Pods, similar to the following. Dependingon the size of the image, the first image pull might take several minutes.

   NAME                            READY   UP-TO-DATE    AVAILABLE   AGEdeployment.apps/t5-inference    1/1     1             0           66sNAME                    TYPE        CLUSTER-IP        EXTERNAL-IP   PORT(S)                       AGEservice/t5-inference    ClusterIP   10.48.131.86    <none>        8080/TCP,8081/TCP,8082/TCP    66s

2. Open a local port for thet5-inference Service:

   kubectlport-forwardsvc/t5-inference8080

3. Open a new terminal window and send a test request to the Service:

   curl-v-XPOST-H'Content-Type: application/json'-d'{"text": "this is a test sentence", "from": "en", "to": "fr"}'"http://localhost:8080/predictions/t5-small/1.0"

   If the test request fails and the Pod connection closes, check the logs:

   kubectllogsdeployments/t5-inference

   If the output is similar to the following, TorchServe failed to installsome model dependencies:

   org.pytorch.serve.archive.model.ModelException: Custom pip package installation failed for t5-small

   To resolve this issue, restart the Deployment:

   kubectlrolloutrestartdeploymentt5-inference

   The Deployment controller creates a new Pod. Repeat the previous steps toopen a port on the new Pod.

Access the deployed model using the web application

To access the deployed model with theFast Dash web application, complete the following steps:

1. Build and push the Fast Dash web application as a container image inArtifact Registry:

   gcloudbuildssubmitclient-app/\--region=us-central1\--config=client-app/cloudbuild.yaml

2. Openkubernetes/application.yaml in a text editor and replacePROJECT_ID in theimage: field with yourproject ID. Alternatively, run the following command:

   sed-i"s/PROJECT_ID/PROJECT_ID/g""kubernetes/application.yaml"

3. Create the Kubernetes resources:

   kubectlcreate-fkubernetes/application.yaml

   The Deployment and Service might take some time to fully provision.

4. To check the status, run the following command:

   kubectlget-fkubernetes/application.yaml

   Wait until the output shows ready Pods, similar to the following:

   NAME                       READY   UP-TO-DATE   AVAILABLE   AGEdeployment.apps/fastdash   1/1     1            0           1mNAME               TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGEservice/fastdash   NodePort   203.0.113.12    <none>        8050/TCP         1m

5. The web application is now running, although it isn't exposed on an externalIP address. To access the web application, open a local port:

   kubectlport-forwardservice/fastdash8050

6. In a browser, open the web interface:

   • If you're using a local shell, open a browser and go tohttp://127.0.0.1:8050.
   • If you're using Cloud Shell, clickWeb preview, and then clickChange port. Specify port8050.

7. To send a request to the T5 model, specify values in theTEXT,FROM LANG, andTO LANG fields in the web interface and clickSubmit. For a list of available languages, see theT5 documentation.

Enable autoscaling for the model

This section shows you how to enable autoscaling for the model based on metricsfromGoogle Cloud Managed Service for Prometheusby doing the following:

1. Install Custom Metrics Stackdriver Adapter
2. Apply PodMonitoring and HorizontalPodAutoscaling configurations

Google Cloud Managed Service for Prometheus is enabled by default in Autopilotclusters running version 1.25 and later.

Install Custom Metrics Stackdriver Adapter

This adapter lets your cluster use metrics from Prometheus to make Kubernetesautoscaling decisions.

1. Deploy the adapter:

   kubectlcreate-fhttps://raw.githubusercontent.com/GoogleCloudPlatform/k8s-stackdriver/master/custom-metrics-stackdriver-adapter/deploy/production/adapter_new_resource_model.yaml

2. Create an IAM service account for the adapter to use:

   gcloudiamservice-accountscreatemonitoring-viewer

3. Grant the IAM service account themonitoring.viewer role onthe project and theiam.workloadIdentityUser role:

   gcloudprojectsadd-iam-policy-bindingPROJECT_ID\--member"serviceAccount:monitoring-viewer@PROJECT_ID.iam.gserviceaccount.com"\--roleroles/monitoring.viewergcloudiamservice-accountsadd-iam-policy-bindingmonitoring-viewer@PROJECT_ID.iam.gserviceaccount.com\--roleroles/iam.workloadIdentityUser\--member"serviceAccount:PROJECT_ID.svc.id.goog[custom-metrics/custom-metrics-stackdriver-adapter]"

   ReplacePROJECT_ID with your Google Cloudproject ID.

4. Annotate the Kubernetes ServiceAccount of the adapter to let it impersonatethe IAM service account:

   kubectlannotateserviceaccountcustom-metrics-stackdriver-adapter\--namespacecustom-metrics\iam.gke.io/gcp-service-account=monitoring-viewer@PROJECT_ID.iam.gserviceaccount.com

5. Restart the adapter to propagate the changes:

   kubectlrolloutrestartdeploymentcustom-metrics-stackdriver-adapter\--namespace=custom-metrics

Apply PodMonitoring and HorizontalPodAutoscaling configurations

PodMonitoring is a Google Cloud Managed Service for Prometheus custom resource thatenables metrics ingestion and target scraping in a specific namespace.

1. Deploy the PodMonitoring resource in the same namespace as the TorchServeDeployment:

   kubectlapply-fkubernetes/pod-monitoring.yaml

2. Review the HorizontalPodAutoscaler manifest:

   # Copyright 2023 Google LLC## Licensed under the Apache License, Version 2.0 (the "License");# you may not use this file except in compliance with the License.# You may obtain a copy of the License at##     https://www.apache.org/licenses/LICENSE-2.0## Unless required by applicable law or agreed to in writing, software# distributed under the License is distributed on an "AS IS" BASIS,# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.# See the License for the specific language governing permissions and# limitations under the License.apiVersion:autoscaling/v2kind:HorizontalPodAutoscalermetadata:name:t5-inferencespec:scaleTargetRef:apiVersion:apps/v1kind:Deploymentname:t5-inferenceminReplicas:1maxReplicas:5metrics:-type:Podspods:metric:name:prometheus.googleapis.com|ts_queue_latency_microseconds|countertarget:type:AverageValueaverageValue:"30000"

   The HorizontalPodAutoscaler scales the T5 model Pod quantity based on thecumulative duration of the request queue. Autoscaling is based on thets_queue_latency_microseconds metric, which shows cumulative queueduration in microseconds.

3. Create the HorizontalPodAutoscaler:

   kubectlapply-fkubernetes/hpa.yaml

Verify autoscaling using a load generator

To test your autoscaling configuration, generate load for the servingapplication. This tutorial uses a Locust load generator to send requests to theprediction endpoint for the model.

1. Create the load generator:

   kubectlapply-fkubernetes/loadgenerator.yaml

   Wait for the load generator Pods to become ready.

2. Expose the load generator web interface locally:

   kubectlport-forwardsvc/loadgenerator8080

   If you see an error message, try again when the Pod is running.

3. In a browser, open the load generator web interface:

   • If you're using a local shell, open a browser and go tohttp://127.0.0.1:8080.
   • If you're using Cloud Shell, clickWeb preview, and thenclickChange port. Enter port8080.

4. Click theCharts tab to observe performance over time.

5. Open a new terminal window and watch the replica count of yourhorizontal Pod autoscalers:

   kubectlgethpa-w

   The number of replicas increases as the load increases. The scaleup mighttake approximately ten minutes. As new replicas start, the number ofsuccessful requests in the Locust chart increases.

   NAME           REFERENCE                 TARGETS           MINPODS   MAXPODS   REPLICAS   AGEt5-inference   Deployment/t5-inference   71352001470m/7M   1         5        1           2m11s

To observe the model performance, you can use the TorchServe dashboardintegration inCloud Monitoring.With this dashboard, you can view critical performance metrics like tokenthroughput, request latency, and error rates.

To use the TorchServe dashboard, you must enableGoogle Cloud Managed Service for Prometheus,which collects the metrics from TorchServe,in your GKE cluster.TorchServe exposes metrics in Prometheus format by default;you do not need to install an additional exporter.