This document provides a reference architecture that you can use to design theinfrastructure to run a generative AI application withretrieval-augmented generation (RAG) using Google Kubernetes Engine (GKE), Cloud SQL, and open source tools like Ray,Hugging Face, and LangChain. To help you experiment with this referencearchitecture, a sample application and Terraform configuration are provided inGitHub.

This document is for developers who want to rapidly build and deploy RAG-capablegenerative AI applications by using open source tools and models. It assumesthat you have experience with using GKE andCloud SQL and that you have a conceptual understanding of AI, machinelearning (ML), and large language models (LLMs). This document doesn't provideguidance about how to design anddevelop a generative AI application.

Architecture

The following diagram shows a high-level view of an architecture for aRAG-capable generative AI application in Google Cloud:

The architecture contains a serving subsystem and an embedding subsystem.

• Theserving subsystem handles the request-response flow between theapplication and its users. The subsystem includes a frontend server, aninference server, and aresponsible AI (RAI) service. The serving subsystem interacts with the embedding subsystemthrough a vector database.
• Theembedding subsystem enables the RAG capability in thearchitecture. This subsystem does the following:
  • Ingests data from data sources in Google Cloud,on-premises, and other cloud platforms.
  • Converts the ingested data to vector embeddings.
  • Stores the embeddings in a vector database.

The following diagram shows a detailed view of the architecture:

As shown in the preceding diagram, the frontend server, inference server, andembedding service are deployed in a regional GKE cluster inAutopilot mode. Data for RAG is ingested through a Cloud Storage bucket. Thearchitecture uses a Cloud SQL for PostgreSQL instance with thepgvector extension as the vector database to store embeddings and perform semanticsearches.Vector databases are designed to efficiently store and retrieve high-dimensional vectors.

The following sections describe the components and data flow within eachsubsystem of the architecture.

Embedding subsystem

The following is the flow of data in the embedding subsystem:

1. Data from external and internal sources is uploaded to theCloud Storage bucket by human users or programmatically. Theuploaded data might be in files, databases, or streamed data.
2. (Not shown in the architecture diagram.) The data upload activitytriggers an event that's published to a messaging service likePub/Sub. The messaging service sends a notification to theembedding service.
3. When the embedding service receives a notification of a data uploadevent, it does the following:
   1. Retrieves data from the Cloud Storage bucket throughtheCloud Storage FUSE CSI driver.
   2. Reads the uploaded data and preprocesses it usingRay Data.The preprocessing can include chunking the data and transforming itinto a suitable format for embedding generation.
   3. Runs aRay job to create vectorized embeddings of the preprocessed data by using anopen-source model likeintfloat/multilingual-e5-small that's deployed in the same cluster.
   4. Writes the vectorized embeddings to theCloud SQL for PostgreSQL vector database.

As described in the following section, when the serving subsystem processesuser requests, it uses the embeddings in the vector database to retrieverelevant domain-specific data.

Serving subsystem

The following is the request-response flow in the serving subsystem:

1. A user submits a natural-language request to a frontend server througha web-based chat interface. The frontend server runs onGKE.
2. The frontend server runs aLangChain process that does the following:
   1. Converts the natural-language request to embeddings by usingthe same model and parameters that the embedding service uses.
   2. Retrieves relevant grounding data by performing a semanticsearch for the embeddings in the vector database. Semantic search helpsfind embeddings based on the intent of a prompt rather than its textual content.
   3. Constructs a contextualized prompt by combining the originalrequest with the grounding data that was retrieved.
   4. Sends the contextualized prompt to the inference server, whichruns on GKE.
3. The inference server uses theHugging Face TGI serving framework to serve an open-source LLM likeMistral-7B-Instruct or aGemma open model.

4. The LLM generates a response to the prompt, and the inference serversends the response to the frontend server.

   You can store and view logs of the request-response activity inCloud Logging, and you can set up logs-based monitoring by usingCloud Monitoring. You can also load the generated responses intoBigQuery for offline analytics.

5. The frontend server invokes an RAI service to apply the required safetyfilters to the response. You can use tools likeSensitive Data Protection andCloud Natural Language API to discover, filter, classify, and de-identify sensitive content in theresponses.

6. The frontend server sends the filtered response to the user.

Products used

The following is a summary of the Google Cloud and open-source productsthat the preceding architecture uses:

Google Cloud products

• Google Kubernetes Engine (GKE): A Kubernetes service that you can use to deployand operate containerized applications at scale using Google's infrastructure.
• Cloud Storage: A low-cost, no-limit object store for diverse data types.Data can be accessed from within and outside Google Cloud, and it'sreplicated across locations for redundancy.
• Cloud SQL: A fully managed relational database service that helps youprovision, operate, and manage your MySQL, PostgreSQL, and SQL Server databaseson Google Cloud.

Open-source products

• Hugging Face Text Generation Inference (TGI): A toolkit for deploying andserving LLMs.
• Ray: An open-source unified compute framework that helps you scale AI andPython workloads.
• LangChain: A framework for developing and deploying applications powered byLLMs.

Use cases

RAG is an effective technique to improve the quality of output that'sgenerated from an LLM. This section provides examples of use cases for which youcan use RAG-capable generative AI applications.

Personalized product recommendations

An online shopping site might use an LLM-powered chatbot to assist customerswith finding products or getting shopping-related help. The questions from auser can be augmented by using historical data about the user's buying behaviorand website interaction patterns. The data might include user reviews andfeedback that's stored in an unstructured datastore or search-related metricsthat are stored in a web analytics data warehouse. The augmented question canthen be processed by the LLM to generate personalized responses that the usermight find more appealing and compelling.

Clinical assistance systems

Doctors in hospitals need to quickly analyze and diagnose a patient's healthcondition to make decisions about appropriate care and medication. A generativeAI application that uses a medical LLM likeMed-PaLM can be used to assist doctors in their clinical diagnosis process. The responsesthat the application generates can be grounded in historical patient records bycontextualizing the doctors' prompts with data from the hospital's electronichealth record (EHR) database or from an external knowledge base likePubMed.

Efficient legal research

Generative AI-powered legal research lets lawyers quickly query large volumesof statutes and case laws to identify relevant legal precedents or summarizecomplex legal concepts. The output of such research can be enhanced byaugmenting a lawyer's prompts with data that's retrieved from the law firm'sproprietary corpus of contracts, past legal communication, and internal caserecords. This design approach ensures that the generated responses are relevantto the legal domain that the lawyer specializes in.

Design alternatives

This section presents alternative design approaches that you can consider foryour RAG-capable generative AI application in Google Cloud.

Fully managed vector search

If you need an architecture that uses a fully managed vector search product, youcan use Vertex AI and Vector Search, which provides optimizedserving infrastructure for very large-scale vector search. For more information,seeRAG infrastructure for generative AI using Vertex AI and Vector Search.

Vector-enabled Google Cloud database

If you want to take advantage of the vector store capabilities of a fullymanaged Google Cloud database like AlloyDB for PostgreSQL or Cloud SQLfor your RAG application, then seeRAG infrastructure for generative AI using Vertex AI and AlloyDB for PostgreSQL.

Other options

For information about other infrastructure options, supported models, andgrounding techniques that you can use for generative AI applications inGoogle Cloud, seeChoose models and infrastructure for your generative AI application.

Design considerations

This section provides guidance to help you develop and run aGKE-hosted RAG-capable generative AI architecture thatmeets your specific requirements for security and compliance, reliability, cost,and performance. The guidance in this section isn't exhaustive. Depending on thespecific requirements of your application and the Google Cloud productsand features that you use, you might need to consider additional design factorsand trade-offs.

For design guidance related to the open-source tools in this referencearchitecture, like Hugging Face TGI, see the documentation for those tools.

Security, privacy, and compliance

This section describes factors that you should consider when you design andbuild a RAG-capable generative AI application in Google Cloud that meetsyour security, privacy, and compliance requirements.

For security principles and recommendations that are specific to AI and ML workloads, seeAI and ML perspective: Securityin the Well-Architected Framework.

Reliability

This section describes design factors that you should consider to build andoperate reliable infrastructure for a RAG-capable generative AI application inGoogle Cloud.

For reliability principles and recommendations that are specific to AI and ML workloads, seeAI and ML perspective: Reliabilityin the Well-Architected Framework.

Cost optimization

This section provides guidance to help you optimize the cost of setting up andoperating a RAG-capable generative AI application in Google Cloud.

To estimate the cost of your Google Cloud resources, use theGoogle Cloud Pricing Calculator.

For cost optimization principles and recommendations that are specific to AI and ML workloads, seeAI and ML perspective: Cost optimizationin the Well-Architected Framework.

Performance optimization

This section describes the factors that you should consider when you design andbuild a RAG-capable generative AI application in Google Cloud that meetsyour performance requirements.

For performance optimization principles and recommendations that are specific to AI and ML workloads, seeAI and ML perspective: Performance optimizationin the Well-Architected Framework.

Deployment

To deploy a topology that's based on this reference architecture, you candownload and use the open-source sample code that's available in a repository inGitHub.The sample code isn't intended for production use cases. You can use the code toexperiment with setting up AI infrastructure for a RAG-enabled generative AIapplication.

The sample code does the following:

1. Provisions a Cloud SQL for PostgreSQL instance to serve as thevector database.
2. Deploys Ray, JupyterHub, and Hugging Face TGI to a GKE cluster thatyou specify.
3. Deploys a sample web-based chatbot application to your GKE clusterto let you verify the RAG capability.

For instructions to use the sample code, see theREADME for the code. If any errors occur when you use the sample code, and if openGitHub issues don't exist for the errors, then create issues in GitHub.

The sample code deploys billable Google Cloud resources. When you finishusing the code, remove any resources that you no longer need.

What's next

• Review the following GKE best practices guides:
  • Best practices for GKE networking
  • Best practices for running cost-optimized Kubernetes applications on GKE
• Serve Gemma open models using GPUs on GKE with Hugging Face TGI.
• Build RAG infrastructure for generative AI using Vertex AI and Vector Search.
• Build RAG infrastructure for generative AI using Vertex AI and AlloyDB for PostgreSQL.
• Build RAG infrastructure for generative AI using Gemini Enterprise and Vertex AI.
• Build GraphRAG infrastructure for generative AI using Vertex AI and Spanner Graph.
• For an overview of architectural principles and recommendations that are specific to AIand ML workloads in Google Cloud, see theAI and ML perspectivein the Well-Architected Framework.
• For more reference architectures, diagrams, and best practices, explore theCloud Architecture Center.

Contributors

Author:Kumar Dhanagopal | Cross-Product Solution Developer

Other contributors:

• Anna Berenberg | Engineering Fellow
• Ali Zaidi | Solutions Architect
• Bala Narasimhan | Group Product Manager
• Bill Bernsen | Security Engineer
• Brandon Royal | Outbound Product Manager
• Cynthia Thomas | Product Manager
• Geoffrey Anderson | Product Manager
• Gleb Otochkin | Cloud Advocate, Databases
• Jack Wotherspoon | Developer Advocate
• Julie Amundson | Senior Staff Software Engineer
• Kent Hua | Solutions Manager
• Kavitha Rajendran | AI/ML Specialist, Solutions Architect
• Mark Schlagenhauf | Technical Writer, Networking
• Megan O'Keefe | Developer Advocate
• Mofi Rahman | Google Cloud Advocate