Access Qwen3 by using Hugging Face

To use Hugging Face to access Qwen3, follow these steps:

1. Sign in to Hugging Face
2. Create a Hugging Faceread access token.ClickYour Profile > Settings > Access Tokens > +Create new token.
3. Specify a name of your choice for the token and then select a role. Theminimum role permission level that you can select for this tutorial isRead.
4. SelectCreate token.
5. Copy and save the generated token to your clipboard. You use it later inthis tutorial.

Prepare your environment

To prepare your environment, set the default environment variables:

gcloud config set projectPROJECT_IDgcloud config set billing/quota_projectPROJECT_IDexport PROJECT_ID=$(gcloud config get project)export RESERVATION_URL=RESERVATION_URLexport REGION=REGIONexport CLUSTER_NAME=CLUSTER_NAMEexport HUGGING_FACE_TOKEN=HUGGING_FACE_TOKENexport NETWORK=NETWORK_NAMEexport SUBNETWORK=SUBNETWORK_NAME

Replace the following:

• PROJECT_ID: the ID of the Google Cloud projectwhere you want to create the GKE cluster.

• RESERVATION_URL: the URL of the reservation that you wantto use to create your GKE cluster. Based on the project inwhich the reservation exists, specify one of the following values:

  • The reservation exists in your project:RESERVATION_NAME

  • The reservation exists in a different project, and your project canuse the reservation:projects/RESERVATION_PROJECT_ID/reservations/RESERVATION_NAME

• REGION: the region where you want to create yourGKE cluster. You can only create the cluster in the regionwhere your reservation exists.

• CLUSTER_NAME: the name of the GKE clusterto create.

• HUGGING_FACE_TOKEN: the Hugging Face access token thatyou created in the previous section.

• NETWORK_NAME: the network that the GKEcluster uses. Specify one of the following values:

  • If you created a custom network, then specify the name of your network.

  • Otherwise, specifydefault.

• SUBNETWORK_NAME: the subnetwork that theGKE cluster uses. Specify one of the following values:

  • If you created a custom subnetwork, then specify the name of yoursubnetwork. You can only specify a subnetwork that exists in the sameregion as the reservation.

  • Otherwise, specifydefault.

Create a GKE cluster in Autopilot mode

To create a GKE cluster in Autopilot mode, run thefollowing command:

gcloudcontainerclusterscreate-auto$CLUSTER_NAME\--project=$PROJECT_ID\--region=$REGION\--release-channel=rapid\--network=$NETWORK\--subnetwork=$SUBNETWORK

Creating the GKE cluster might take some time to complete. Toverify that Google Cloud has finished creating your cluster, go toKubernetes clusterson the Google Cloud console.

Create a Kubernetes secret for Hugging Face credentials

To create a Kubernetes secret for Hugging Face credentials, follow these steps:

1. Configurekubectl to communicate with your GKE cluster:

   gcloudcontainerclustersget-credentials$CLUSTER_NAME\--location=$REGION

2. Create a Kubernetes secret to store your Hugging Face token:

   kubectlcreatesecretgenerichf-secret\--from-literal=hf_token=${HUGGING_FACE_TOKEN}\--dry-run=client-oyaml|kubectlapply-f-

Deploy a vLLM container to your GKE cluster

To deploy the vLLM container to serve the Qwen3 model by using Kubernetes Deployments,do the following:

1. Create aqwen3-235b-deploy.yaml file with your chosen vLLM deployment. :

   apiVersion:apps/v1kind:Deploymentmetadata:name:vllm-qwen3-deploymentspec:replicas:1selector:matchLabels:app:qwen3-servertemplate:metadata:labels:app:qwen3-serverai.gke.io/model:Qwen3-235B-A22B-Instruct-2507ai.gke.io/inference-server:vllmspec:containers:-name:qwen-inference-serverimage:us-docker.pkg.dev/vertex-ai/vertex-vision-model-garden-dockers/pytorch-vllm-serve:20250801_0916_RC01resources:requests:cpu:"10"memory:"1000Gi"ephemeral-storage:"500Gi"nvidia.com/gpu:"8"limits:cpu:"10"memory:"1000Gi"ephemeral-storage:"500Gi"nvidia.com/gpu:"8"command:["python3","-m","vllm.entrypoints.openai.api_server"]args:---model=$(MODEL_ID)---tensor-parallel-size=8---host=0.0.0.0---port=8000---max-model-len=8192---max-num-seqs=4---dtype=bfloat16env:-name:MODEL_IDvalue:"Qwen/Qwen3-235B-A22B-Instruct-2507"-name:HUGGING_FACE_HUB_TOKENvalueFrom:secretKeyRef:name:hf-secretkey:hf_tokenvolumeMounts:-mountPath:/dev/shmname:dshmlivenessProbe:httpGet:path:/healthport:8000initialDelaySeconds:1320periodSeconds:10readinessProbe:httpGet:path:/healthport:8000initialDelaySeconds:1320periodSeconds:5volumes:-name:dshmemptyDir:medium:MemorynodeSelector:cloud.google.com/gke-accelerator:nvidia-b200cloud.google.com/reservation-name:RESERVATION_URLcloud.google.com/reservation-affinity:"specific"cloud.google.com/gke-gpu-driver-version:latest---apiVersion:v1kind:Servicemetadata:name:qwen3-servicespec:selector:app:qwen3-servertype:ClusterIPports:-protocol:TCPport:8000targetPort:8000---apiVersion:monitoring.googleapis.com/v1kind:PodMonitoringmetadata:name:vllm-qwen3-monitoringspec:selector:matchLabels:app:qwen3-serverendpoints:-port:8000path:/metricsinterval:30s

2. Apply theqwen3-235b-deploy.yaml file to your GKE cluster:

   kubectl apply -f qwen3-235b-deploy.yaml

   During the deployment process, the container must download theQwen3-235B-A22B-Instruct-2507 model from Hugging Face. For this reason,deployment of the container might take up to 30 minutes to complete.

3. To see the completion status, run the following command:

   kubectl wait \    --for=condition=Available \    --timeout=1500s deployment/vllm-qwen3-deployment

   The--timeout=1500s flag allows the command to monitor the deployment forup to 25 minutes.

Interact with Qwen3 by using curl

To verify the Qwen3 model that you deployed, do the following:

1. Set up port forwarding to Qwen3:

   kubectl port-forward service/qwen3-service 8000:8000

2. Open a new terminal window. You can then chat with your model by usingcurl:

   curlhttp://127.0.0.1:8000/v1/chat/completions\-XPOST\-H"Content-Type: application/json"\-d'{  "model": "Qwen/Qwen3-235B-A22B-Instruct-2507",  "messages": [    {      "role": "user",      "content": "Describe a GPU in one short sentence?"    }  ]}'

   The output is similar to the following:

   {"id":"chatcmpl-a926ddf7ef2745ca832bda096e867764","object":"chat.completion","created":1755023619,"model":"Qwen/Qwen3-235B-A22B-Instruct-2507","choices":[{"index":0,"message":{"role":"assistant","content":"A GPU is a specialized electronic circuit designed to rapidly process and render graphics and perform parallel computations.","refusal":null,"annotations":null,"audio":null,"function_call":null,"tool_calls":[],"reasoning_content":null},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":16,"total_tokens":36,"completion_tokens":20,"prompt_tokens_details":null},"prompt_logprobs":null,"kv_transfer_params":null}

Observe model performance

If you want to observe your model's performance, then you can use the vLLMdashboard integration inCloud Monitoring. This dashboard helps you viewcritical performance metrics for your model like token throughput, networklatency, and error rates. For information, seevLLM in theMonitoring documentation.