|
| 1 | +#Rag with OpenAI |
| 2 | + |
| 3 | +This example shows how to use third-party LLM providers like OpenAI to perform RAG with Korvus. |
| 4 | + |
| 5 | +Rag is comoposed of two parts: |
| 6 | +- Retrieval - Search to get the context |
| 7 | +- Augmented Generation - Perform text-generation with the LLM |
| 8 | + |
| 9 | +Korvus can unify the retrieval and augmented generation parts into one SQL query, but if you want to use closed source models, you will have to perform retrieval and augmented generation seperately. |
| 10 | + |
| 11 | +!!! note |
| 12 | + |
| 13 | +Remeber Korvus only writes SQL queries utilizing pgml to perform embeddings and text-generation in the database. The pgml extension does not support closed source models so neither does Korvus. |
| 14 | + |
| 15 | +!!! |
| 16 | + |
| 17 | +Even though Korvus can't use closed source models, we can use Korvus for search and use closed source models ourself. |
| 18 | + |
| 19 | +##RAG Code |
| 20 | + |
| 21 | +In this code block we create a Collection and a Pipeline, upsert documents into the Collection, and instead of calling the`rag` method, we call the`vector_search` method. |
| 22 | + |
| 23 | +We take the results returned from the`vector_search` (in this case we limited it to 1) and format a prompt for OpenAI using it. |
| 24 | + |
| 25 | +See the[Vector Search guide](../guides/vector-search) for more information on using the`vector_search` method. |
| 26 | + |
| 27 | +{% tabs %} |
| 28 | +{% tab title="JavaScript" %} |
| 29 | + |
| 30 | +```js |
| 31 | +constkorvus=require("korvus"); |
| 32 | +constopenai=require("openai"); |
| 33 | + |
| 34 | +// Initialize our Collection |
| 35 | +constcollection=korvus.newCollection("openai-text-generation-demo"); |
| 36 | + |
| 37 | +// Initialize our Pipeline |
| 38 | +// Our Pipeline will split and embed the `text` key of documents we upsert |
| 39 | +constpipeline=korvus.newPipeline("v1", { |
| 40 | + text: { |
| 41 | + splitter: { model:"recursive_character" }, |
| 42 | + semantic_search: { |
| 43 | + model:"mixedbread-ai/mxbai-embed-large-v1", |
| 44 | + } |
| 45 | + }, |
| 46 | +}); |
| 47 | + |
| 48 | + |
| 49 | +// Initialize our client connection to OpenAI |
| 50 | +constclient=newopenai.OpenAI({ |
| 51 | + apiKey:process.env['OPENAI_API_KEY'],// This is the default and can be omitted |
| 52 | +}); |
| 53 | + |
| 54 | + |
| 55 | +constmain=async ()=> { |
| 56 | +// Add our Pipeline to our Collection |
| 57 | +awaitcollection.add_pipeline(pipeline); |
| 58 | + |
| 59 | +// Upsert our documents |
| 60 | +// The `text` key of our documents will be split and embedded per our Pipeline specification above |
| 61 | +let documents= [ |
| 62 | + { |
| 63 | + id:"1", |
| 64 | + text:"Korvus is incredibly fast and easy to use.", |
| 65 | + }, |
| 66 | + { |
| 67 | + id:"2", |
| 68 | + text:"Tomatoes are incredible on burgers.", |
| 69 | + }, |
| 70 | + ] |
| 71 | +awaitcollection.upsert_documents(documents) |
| 72 | + |
| 73 | +// Perform vector_search |
| 74 | +// We are querying for the string "Is Korvus fast?" |
| 75 | +// Notice that the `mixedbread-ai/mxbai-embed-large-v1` embedding model takes a prompt paramter when embedding for search |
| 76 | +// We specify that we only want to return the `id` of documents. If the `document` key was blank it would return the entire document with every result |
| 77 | +// Limit the results to 5. In our case we only have two documents in our Collection so we will only get two results |
| 78 | +constquery="Is Korvus fast?" |
| 79 | +constresults=awaitcollection.vector_search( |
| 80 | + { |
| 81 | + query: { |
| 82 | + fields: { |
| 83 | + text: { |
| 84 | + query: query, |
| 85 | + parameters: { |
| 86 | + prompt: |
| 87 | +"Represent this sentence for searching relevant passages:", |
| 88 | + } |
| 89 | + }, |
| 90 | + }, |
| 91 | + }, |
| 92 | +document: { |
| 93 | + keys: [ |
| 94 | +"id" |
| 95 | + ] |
| 96 | + }, |
| 97 | + limit:5, |
| 98 | + }, |
| 99 | + pipeline); |
| 100 | +console.log("Our search results:") |
| 101 | +console.log(results) |
| 102 | + |
| 103 | +// After retrieving the context, we build our prompt for gpt-4o and make our completion request |
| 104 | +constcontext= results[0].chunk |
| 105 | +console.log("Model output:") |
| 106 | +constchatCompletion=awaitclient.chat.completions.create({ |
| 107 | + messages: [{ role:'user', content:`Answer the question:\n\n${query}\n\nGiven the context:\n\n${context}` }], |
| 108 | + model:'gpt-4o', |
| 109 | + }); |
| 110 | +console.dir(chatCompletion, {depth:10}); |
| 111 | +} |
| 112 | + |
| 113 | +main().then(()=>console.log("DONE!")) |
| 114 | +``` |
| 115 | + |
| 116 | +{% endtab %} |
| 117 | +{% tab title="Python" %} |
| 118 | + |
| 119 | +```python |
| 120 | +from korvusimport Collection, Pipeline |
| 121 | +from richimportprint |
| 122 | +from openaiimport OpenAI |
| 123 | +import os |
| 124 | +import asyncio |
| 125 | + |
| 126 | +# Initialize our Collection |
| 127 | +collection= Collection("openai-text-generation-demo") |
| 128 | + |
| 129 | +# Initialize our Pipeline |
| 130 | +# Our Pipeline will split and embed the `text` key of documents we upsert |
| 131 | +pipeline= Pipeline( |
| 132 | +"v1", |
| 133 | + { |
| 134 | +"text": { |
| 135 | +"splitter": {"model":"recursive_character"}, |
| 136 | +"semantic_search": { |
| 137 | +"model":"mixedbread-ai/mxbai-embed-large-v1", |
| 138 | + }, |
| 139 | + }, |
| 140 | + }, |
| 141 | +) |
| 142 | + |
| 143 | +# Initialize our client connection to OpenAI |
| 144 | +client= OpenAI( |
| 145 | +# This is the default and can be omitted |
| 146 | +api_key=os.environ.get("OPENAI_API_KEY"), |
| 147 | +) |
| 148 | + |
| 149 | + |
| 150 | +asyncdefmain(): |
| 151 | +# Add our Pipeline to our Collection |
| 152 | +await collection.add_pipeline(pipeline) |
| 153 | + |
| 154 | +# Upsert our documents |
| 155 | +# The `text` key of our documents will be split and embedded per our Pipeline specification above |
| 156 | + documents= [ |
| 157 | + { |
| 158 | +"id":"1", |
| 159 | +"text":"Korvus is incredibly fast and easy to use.", |
| 160 | + }, |
| 161 | + { |
| 162 | +"id":"2", |
| 163 | +"text":"Tomatoes are incredible on burgers.", |
| 164 | + }, |
| 165 | + ] |
| 166 | +await collection.upsert_documents(documents) |
| 167 | + |
| 168 | +# Perform vector_search |
| 169 | +# We are querying for the string "Is Korvus fast?" |
| 170 | +# Notice that the `mixedbread-ai/mxbai-embed-large-v1` embedding model takes a prompt paramter when embedding for search |
| 171 | +# We specify that we only want to return the `id` of documents. If the `document` key was blank it would return the entire document with every result |
| 172 | +# Limit the results to 1. In our case we only want to feed the top result to OpenAI as we know the other result is not going to be relevant to our question |
| 173 | + query="Is Korvus Fast?" |
| 174 | + results=await collection.vector_search( |
| 175 | + { |
| 176 | +"query": { |
| 177 | +"fields": { |
| 178 | +"text": { |
| 179 | +"query": query, |
| 180 | +"parameters": { |
| 181 | +"prompt":"Represent this sentence for searching relevant passages:", |
| 182 | + }, |
| 183 | + }, |
| 184 | + }, |
| 185 | + }, |
| 186 | +"document": {"keys": ["id"]}, |
| 187 | +"limit":1, |
| 188 | + }, |
| 189 | + pipeline, |
| 190 | + ) |
| 191 | +print("Our search results:") |
| 192 | +print(results) |
| 193 | + |
| 194 | +# After retrieving the context, we build our prompt for gpt-4o and make our completion request |
| 195 | + context= results[0]["chunk"] |
| 196 | +print("Model output:") |
| 197 | + chat_completion= client.chat.completions.create( |
| 198 | +messages=[ |
| 199 | + { |
| 200 | +"role":"user", |
| 201 | +"content":f"Answer the question:\n\n{query}\n\nGiven the context:\n\n{context}", |
| 202 | + } |
| 203 | + ], |
| 204 | +model="gpt-4o", |
| 205 | + ) |
| 206 | +print(chat_completion) |
| 207 | + |
| 208 | + |
| 209 | +asyncio.run(main()) |
| 210 | +``` |
| 211 | +{% endtab %} |
| 212 | + |
| 213 | +{% endtabs %} |
| 214 | + |
| 215 | +Running the example outputs: |
| 216 | + |
| 217 | +```json |
| 218 | +{ |
| 219 | +id: 'chatcmpl-9kHvSowKHra1692aJsZc3G7hHMZKz', |
| 220 | +object: 'chat.completion', |
| 221 | +created: 1720819022, |
| 222 | +model: 'gpt-4o-2024-05-13', |
| 223 | +choices: [ |
| 224 | + { |
| 225 | +index: 0, |
| 226 | +message: { |
| 227 | +role: 'assistant', |
| 228 | +content: 'Yes, Korvus is fast according to the provided context.' |
| 229 | + }, |
| 230 | +logprobs: null, |
| 231 | +finish_reason: 'stop' |
| 232 | + } |
| 233 | +], |
| 234 | +usage: { prompt_tokens: 30, completion_tokens: 12, total_tokens: 42 }, |
| 235 | +system_fingerprint: 'fp_dd932ca5d1' |
| 236 | +} |
| 237 | +``` |
| 238 | + |
| 239 | +The example above shows how we can use OpenAI or any other third-party LLM to perform RAG. |
| 240 | + |
| 241 | +A bullet point summary: |
| 242 | +- Use Korvus to perform search |
| 243 | +- Use the third party API provider to generate the text |