- Notifications
You must be signed in to change notification settings - Fork0
Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.
License
RajiRai/GenerativeAIExamples
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
State-of-the-art Generative AI examples that are easy to deploy, test, and extend. All examples run on the high performance NVIDIA CUDA-X software stack and NVIDIA GPUs.
Generative AI Examples uses resources from theNVIDIA NGC AI Development Catalog.
Sign up for afree NGC developer account to access:
- GPU-optimized containers used in these examples
- Release notes and developer documentation
A RAG pipeline embeds multimodal data -- such as documents, images, and video -- into a database connected to a LLM. RAG lets users chat with their data!
The developer RAG examples run on a single VM. They demonstrate how to combine NVIDIA GPU acceleration with popular LLM programming frameworks using NVIDIA'sopen source connectors. The examples are easy to deploy viaDocker Compose.
Examples support local and remote inference endpoints. If you have a GPU, you can inference locally viaTensorRT-LLM. If you don't have a GPU, you can inference and embed remotely viaNVIDIA AI Foundations endpoints.
| Model | Embedding | Framework | Description | Multi-GPU | TRT-LLM | NVIDIA AI Foundation | Triton | Vector Database |
|---|---|---|---|---|---|---|---|---|
| llama-2 | e5-large-v2 | Llamaindex | Canonical QA Chatbot | YES | YES | No | YES | Milvus/PGVector |
| mixtral_8x7b | nvolveqa_40k | Langchain | Nvidia AI foundation based QA Chatbot | No | No | YES | YES | FAISS |
| llama-2 | all-MiniLM-L6-v2 | Llama Index | QA Chatbot, GeForce, Windows | NO | YES | NO | NO | FAISS |
| llama-2 | nvolveqa_40k | Langchain | QA Chatbot, Task Decomposition Agent | No | No | YES | YES | FAISS |
| mixtral_8x7b | nvolveqa_40k | Langchain | Minimilastic example showcasing RAG using Nvidia AI foundation models | No | No | YES | YES | FAISS |
The enterprise RAG examples run as microservies distributed across multiple VMs and GPUs. They show how RAG pipelines can be orchestrated withKubernetes and deployed withHelm.
Enterprise RAG examples include aKubernetes operator for LLM lifecycle management. It is compatible with theNVIDIA GPU operator that automates GPU discovery and lifecycle management in a Kubernetes cluster.
Enterprise RAG examples also support local and remote inference viaTensorRT-LLM andNVIDIA AI Foundations endpoints.
| Model | Embedding | Framework | Description | Multi-GPU | Multi-node | TRT-LLM | NVIDIA AI Foundation | Triton | Vector Database |
|---|---|---|---|---|---|---|---|---|---|
| llama-2 | NV-Embed-QA-003 | Llamaindex | QA Chatbot, Helm, k8s | NO | NO | YES | NO | YES | Milvus |
Example tools and tutorials to enhance LLM development and productivity when using NVIDIA RAG pipelines.
| Name | Description | Deployment | Tutorial |
|---|---|---|---|
| Evaluation | Example open source RAG eval tool that uses synthetic data generation and LLM-as-a-judge | Docker compose file | README |
| Observability | Observability serves as an efficient mechanism for both monitoring and debugging RAG pipelines. | Docker compose file | README |
These are open source connectors for NVIDIA-hosted and self-hosted API endpoints. These open source connectors are maintained and tested by NVIDIA engineers.
| Name | Framework | Chat | Text Embedding | Python | Description |
|---|---|---|---|---|---|
| NVIDIA AI Foundation Endpoints | Langchain | YES | YES | YES | Easy access to NVIDIA hosted models. Supports chat, embedding, code generation, steerLM, multimodal, and RAG. |
| NVIDIA Triton + TensorRT-LLM | Langchain | YES | YES | YES | This connector allows Langchain to remotely interact with a Triton inference server over GRPC or HTTP tfor optimized LLM inference. |
| NVIDIA Triton Inference Server | LlamaIndex | YES | YES | NO | Triton inference server provides API access to hosted LLM models over gRPC. |
| NVIDIA TensorRT-LLM | LlamaIndex | YES | YES | NO | TensorRT-LLM provides a Python API to build TensorRT engines with state-of-the-art optimizations for LLM inference on NVIDIA GPUs. |
In each example README we indicate the level of support provided.
We're posting these examples on GitHub to support the NVIDIA LLM community, facilitate feedback. We invite contributions via GitHub Issues or pull requests!
- In each of the READMEs, we indicate any known issues and encourage the community to provide feedback.
- The datasets provided as part of this project is under a different license for research and evaluation purposes.
- This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use.
About
Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.
Resources
License
Security policy
Uh oh!
There was an error while loading.Please reload this page.
Stars
Watchers
Forks
Releases
Packages0
Languages
- Python57.3%
- Jupyter Notebook29.4%
- Go9.1%
- Makefile1.5%
- Jinja0.8%
- Shell0.8%
- Other1.1%