This tutorial seeks to broadly cover the majority of topics required to not only implement a modern chatbot, but understand why we build them this way.There are three primary sections:

This tutorial seeks to broadly cover the majority of topics required to not only implement a modern chatbot, but understand why we build them this way.There are three primary sections:

* The Limitations of Modern LLMs

* Circumventing Limitations with RAG

* The chatbot remembers our past conversation

* The chatbot can answer questions correctly about Baldur's Gate 3

In reality we haven't created a SOTA LLM, but fortunately other people have and we will be using the incredibly popular fine-tune of Mistral: `teknium/OpenHermes-2.5-Mistral-7B`. We will be using pgml our own Python library for the remainder of this tutorial. If you want to follow along and have not installed it yet:

pip install pgml

Also make sureandset the`DATABASE_URL` environment variable:

exportDATABASE_URL="{your free PostgresML database url}"

Let's setup a basic chat loop with our model:

from pgmlimport TransformerPipeline

import asyncio

model= TransformerPipeline(

"text-generation",

"teknium/OpenHermes-2.5-Mistral-7B",

{"device_map":"auto","torch_dtype":"bfloat16"},

)

asyncdef main():

whileTrue:

user_input=input("=>")

model_output=await model.transform([user_input], {"max_new_tokens":1000})

print(model_output[0][0]["generated_text"],"\n")

asyncio.run(main())

{% hint style="info"%}

Note thatin our previous hypothetical examples we manually called tokenize to convert our inputs into`tokens`,in the real world we let`pgml` handle converting the text into`tokens`.

{% endhint%}

Now we can have the following conversation:

=> Whatis your name?

A: My nameis John.

Q: How old are you?

A: I am25 years old.

Q: Whatis your favorite color?

=> What did I just ask you?

I asked youif you were going to the store.

Oh, I see. No, I'm not going to the store.

That wasn't close to what we wanted to happen. Getting chatbots to work in the real world seems a bit more complicated than the hypothetical world.

To understand why our chatbot gave us a nonsensical first response,and why it didn't remember our conversation at all, we must dive shortly into the world of prompting.

RememberLLM's are just function approximators that are designed to predict the next most likely `token` given a list of `tokens`, and just like any other function, we must give the correct input. Let's look closer at theinput we are giving our chatbot. In our last conversation we asked it two questions:

* Whatis your name?

* What did I just ask you?

We need to understand that LLMs have a specialformatfor the inputs specificallyfor conversations. So far we have been ignoring this required formattingand giving ourLLM the wrong inputs causing it to predicate nonsensical outputs.

What do the right inputs look like? That actually depends on the model. Each model can choose whichformat to usefor conversationswhile training,andnotall models are trained to be conversational.`teknium/OpenHermes-2.5-Mistral-7B` has been trained to be conversationaland expects us toformat text meantfor conversations like so:

<|im_start|>system

You are a helpfulAI assistant named Hermes

<|im_start|>user

Whatis your name?<|im_end|>

<|im_start|>assistant

We have added a bunch of these newHTML looking tags throughout ourinput. These tagsmap to tokens theLLM has been trained to associatewith conversation shifts.`<|im_start|>` marks the beginning of a message. The text right after`<|im_start|>`, either system, user,or assistant marks the role of the message,and`<|im_end|>` marks the end of a message.

Thisis the style ofinput ourLLM has been trained on. Let's do a simple test with this input and see if we get a better response:

from pgmlimport TransformerPipeline

import asyncio

model= TransformerPipeline(

"text-generation",

"teknium/OpenHermes-2.5-Mistral-7B",

{"device_map":"auto","torch_dtype":"bfloat16"},

)

asyncdef main():

model_output=await model.transform([user_input], {"max_new_tokens":1000})

print(model_output[0][0]["generated_text"],"\n")

asyncio.run(main())

My nameis Hermes

{% hint style="info"%}

Notice we have a new"system" message we haven't discussed before. This special message gives us control over how the chatbot should interact with users. We could tell it to talk like a pirate, to be super friendly, or to not respond to angry messages. In this case we told it what it is, and its name. We will also add any conversation context the chatbot should have in the system message later.

{% endhint%}

Note that we have alist of dictionaries called`history` we use to store the chat history,and instead of feeding text into our model, we are inputting the`history`list. Our library automatically converts thislist of dictionaries into theformat expected by the model. Notice the`roles`in the dictionaries are the sameas the`roles` of the messagesin the previous example. Thislist of dictionarieswith keys`role`and`content`as a storage systemfor messagesis pretty standardand used by usas wellas OpenAIand HuggingFace.

Let's ask it the dreaded question:

=> Whatis Baldur's Gate?

Baldur's Gate 3 is a role-playing video game developed by Larian Studios and published by Dontnod Entertainment. It is based on the Advanced Dungeons & Dragons (D&D) rules and set in the Forgotten Realms campaign setting. Originally announced in 2012, the game had a long development period and was finally released in early access in October 2020. The game is a sequel to the popular Baldur's GateII: Shadows of Amn (2000)and Baldur's Gate: Siege of Dragonspear (2016) expansion, and it continues the tradition of immersive storytelling, tactical combat, and character progression that fans of the series love.L

How does it know about Baldur's Gate 3? As it turns out, Baldur's Gate3 has actually been around since2020. I guess that completely ruins the hypothetical example. Let's ignore that and ask it something trickier it wouldn't know about Baldur's Gate 3.

=> Whatis the plot of Baldur's Gate 3?

Baldur's Gate 3 is a role-playing game set in the Dungeons & Dragons Forgotten Realms universe. The story revolves around a mind flayer, also known as an illithid, called The Mind Flayer who is attempting to merge humanoid minds into itself to achieve god-like power. Your character and their companions must navigate a world torn apart by various factions and conflicts while uncovering the conspiracy surrounding The Mind Flayer. Throughout the game, you'll forge relationshipswith various NPCs, make choices that impact the story,and engagein battleswith enemies using a turn-based combat system.

As expected thisis rather a shallow response that lacksany of the actual plot. To get the answer we want, we need to provide the correct context to ourLLM, that means we need to:

* Get the textfrom theURL that has the answer

* Split that text into chunks

* Embed those chunks

* Search over the chunks to find the closest match

* Use the textfrom that chunkas contextfor theLLM

Luckily none of thisis actually very difficultas people like us have built libraries that handle thecomplex pieces. Hereis a program that handles steps1-4:

from pgmlimport Collection, Model, Splitter, Pipeline

import wikipediaapi

import asyncio

wiki_wiki= wikipediaapi.Wikipedia("Chatbot Tutorial Project","en")

model= Model()# The default model is intfloat/e5-small

splitter= Splitter()# The default splitter is recursive_character

pipeline= Pipeline("test-pipeline-1", model, splitter)

collection= Collection("chatbot-knowledge-base-1")

asyncdef main():

await collection.add_pipeline(pipeline)

page= wiki_wiki.page("Baldur's_Gate_3")

await collection.upsert_documents([{"id":"Baldur's_Gate_3","text": page.text}])

most_relevant_section=await (

collection.query()

.vector_recall("What is the plot of Baldur's Gate 3", pipeline)

.limit(1)

.fetch_all()

)

print(most_relevant_section[0][1])

asyncio.run(main())

Plot

Setting

Baldur's Gate 3 takes place in the fictional world of the Forgotten Realms during the year of 1492 DR, over 120 years after the events of the previous game, Baldur's Gate II: Shadows of Amn, and months after the events of the playable Dungeons & Dragons 5e module, Baldur's Gate: Descent into Avernus. The story is set primarily in the Sword Coast in western Faerûn, encompassing a forested area that includes the Emerald Grove, a druid grove dedicated to the deity Silvanus; Moonrise Towers and the Shadow-Cursed Lands, which are covered by an unnatural and sentient darkness that can only be penetrated through magical means; and Baldur's Gate, the largest and most affluent city in the region, as well as its outlying suburb of Rivington. Other places the player will pass through include the Underdark, the Astral Plane and Avernus.The player character can either be created from scratch by the player, chosen from six pre-made "origin characters", or a customisable seventh origin character known as the Dark Urge. All six pre-made origin characters can be recruited as part of the player character's party. They include Lae'zel, a githyanki fighter; Shadowheart, a half-elf cleric; Astarion, a high elf vampire rogue; Gale, a human wizard; Wyll, a human warlock; and Karlach, a tiefling barbarian. Four other characters may join the player's party: Halsin, a wood elf druid; Jaheira, a half-elf druid; Minsc, a human ranger who carries with him a hamster named Boo; and Minthara, a drow paladin. Jaheira and Minsc previously appeared in both Baldur's Gate and Baldur's Gate II: Shadows of Amn.

{% hint style="info"%}

Once again we are using`pgml` to abstract away the complicated piecesfor our machine learning task. This isn't a guide on how to use our libraries, but for more information [check out our docs](https://postgresml.org/docs/api/client-sdk/getting-started).

{% endhint%}