Set up gcloud

To configure the Google Cloud CLI for your Cloud Run service:

1. Set your default project:

   gcloudconfigsetprojectPROJECT_ID

   Click the iconto replace the variablePROJECT_ID with the name of the project youcreated for this tutorial. This ensures that all listings on this page thatreferencePROJECT_ID have the correct value alreadyfilled in.

2. Configure Google Cloud CLI to use the regioneurope-west1 for Cloud Runcommands.

   gcloudconfigsetrun/regioneurope-west1

Use Docker to create a container image with Ollama and Gemma

1. Create a directory for the Ollama service and change your workingdirectory to this new directory:

   mkdirollama-backendcdollama-backend

2. Create aDockerfile file with the following contents:

   FROMollama/ollama:latest# Listen on all interfaces, port 8080ENVOLLAMA_HOST0.0.0.0:8080# Store model weight files in /modelsENVOLLAMA_MODELS/models# Reduce logging verbosityENVOLLAMA_DEBUGfalse# Never unload model weights from the GPUENVOLLAMA_KEEP_ALIVE-1# Store the model weights in the container imageENVMODELgemma3:4bRUNollamaserve &sleep5 &&ollamapull$MODEL# Start OllamaENTRYPOINT["ollama","serve"]

Store model weights in the container image for faster instance starts

Google recommends storing themodel weights forGemma 3 (4B) and similarly sized models directly in the container image.

Model weights are the numerical parameters that define the behavior of an LLM.Ollama must fully read these files and load the weights into GPU memory (VRAM)during container instance startup, before it can start serving inference requests.

On Cloud Run, a fast container instance startup is important for minimizingrequest latency. If your container instance has a slow startup time, the servicetakes longer to scale from zero to one instance, and it needs more time to scaleout during a traffic spike.

To ensure a fast startup, store the model files in the container image itself.This is faster and more reliable than downloading the files from a remote locationduring startup. Cloud Run's internal container image storage is optimizedfor handling traffic spikes, allowing it to quickly set up the container's filesystem when an instance starts.

Note that the model weights for Gemma 3 (4B) take up 8 GB of storage. Larger modelshave larger model weight files, and these might be impractical to store in the container image.Refer toBest practices: AI inference on Cloud Run with GPUs for an overview of the trade-offs.

Build and deploy the Ollama service for LLM inference

Build and deploy the service to Cloud Run:

gcloudrundeployollama-gemma\--source.\--concurrency4\--cpu8\--set-env-varsOLLAMA_NUM_PARALLEL=4\--gpu1\--gpu-typenvidia-l4\--max-instances1\--memory32Gi\--no-allow-unauthenticated\--no-cpu-throttling\--no-gpu-zonal-redundancy\--timeout=600

Note the following important flags in this command:

• --concurrency 4 is set to match the value of the environment variableOLLAMA_NUM_PARALLEL.
• --gpu 1 with--gpu-type nvidia-l4 assigns 1 NVIDIA L4 GPUto every Cloud Run instance in the service.
• --max-instances 1 specifies the maximum number of instances to scale to.It has to be equal to or lower than your project's NVIDIA L4 GPU (Total Nvidia L4 GPU allocation, per project per region) quota.
• --no-allow-unauthenticated restricts unauthenticated access to the service.By keeping the service private, you can rely on Cloud Run's built-inIdentity and Access Management (IAM) authentication for service-to-servicecommunication. Refer toManaging access using IAM.
• --no-cpu-throttling is required for enabling GPU.
• --no-gpu-zonal-redundancy set zonal redundancy options depending on your zonal failover requirements and available quota. SeeGPU zonal redundancy options for details.

Concurrency settings for optimal performance

This section provides context on the recommended concurrency settings. For optimalrequest latency, ensure the--concurrency setting is equal to Ollama'sOLLAMA_NUM_PARALLEL environment variable.

• OLLAMA_NUM_PARALLEL determines how many request slots are available pereach model to handle inference requests concurrently.
• --concurrency determines how many requests Cloud Run sends toan Ollama instance at the same time.

If--concurrency exceedsOLLAMA_NUM_PARALLEL, Cloud Run can sendmore requests to a model in Ollama than it has available request slots for.This leads to request queuing within Ollama, increasing request latency for thequeued requests. It also leads to less responsive auto scaling, as the queuedrequests don't trigger Cloud Run to scale out and start new instances.

Ollama also supports serving multiple models from one GPU. To completelyavoid request queuing on the Ollama instance, you should still set--concurrency to matchOLLAMA_NUM_PARALLEL.

It's important to note that increasingOLLAMA_NUM_PARALLEL also makes parallel requests take longer.

Optimize GPU utilization

For optimalGPU utilization, increase--concurrency, keeping it withintwice the value ofOLLAMA_NUM_PARALLEL. While this leads to request queuing in Ollama, it can help improve utilization: Ollama instances can immediately process requests from their queue, and the queues help absorb traffic spikes.

Test the deployed Ollama service with curl

Now that you have deployed the Ollama service, you can send requests to it. However,if you send a request directly, Cloud Run responds withHTTP 401 Unauthorized.This is intentional, because an LLM inference API is intended for other services tocall, such as a frontend application. For more information on service-to-serviceauthentication on Cloud Run, refer toAuthenticating service-to-service.

To send requests to the Ollama service, add a header with a valid OIDC token tothe requests, for example using the Cloud Rundeveloper proxy:

1. Start the proxy, and when prompted to install thecloud-run-proxy component, chooseY:

   gcloudrunservicesproxyollama-gemma--port=9090

2. Send a request to it in a separate terminal tab, leaving the proxy running. Note that the proxy runs onlocalhost:9090:

   curlhttp://localhost:9090/api/generate-d'{  "model": "gemma3:4b",  "prompt": "Why is the sky blue?"}'

   This command should provide streaming output similar to this:

   {"model":"gemma3:4b","created_at":"2025-03-10T03:02:18.641492408Z","response":"That","done":false}{"model":"gemma3:4b","created_at":"2025-03-10T03:02:18.687529153Z","response":"'","done":false}{"model":"gemma3:4b","created_at":"2025-03-10T03:02:18.753284927Z","response":"s","done":false}{"model":"gemma3:4b","created_at":"2025-03-10T03:02:18.812957381Z","response":" a","done":false}{"model":"gemma3:4b","created_at":"2025-03-10T03:02:18.889102649Z","response":" fantastic","done":false}{"model":"gemma3:4b","created_at":"2025-03-10T03:02:18.925748116Z","response":",","done":false}{"model":"gemma3:4b","created_at":"2025-03-10T03:02:18.958391572Z","response":" decept","done":false}{"model":"gemma3:4b","created_at":"2025-03-10T03:02:18.971035028Z","response":"ively","done":false}{"model":"gemma3:4b","created_at":"2025-03-10T03:02:18.989678484Z","response":" tricky","done":false}{"model":"gemma3:4b","created_at":"2025-03-10T03:02:18.999321940Z","response":" question","done":false}...

   Success: You deployed a GPU-enabled Cloud Run service with Gemma 3 on Ollama and sent aninference request to it.