You signed in with another tab or window.Reload to refresh your session.You signed out in another tab or window.Reload to refresh your session.You switched accounts on another tab or window.Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: pgml-cms/docs/guides/chatbots/README.md
+4-4Lines changed: 4 additions & 4 deletions
Original file line number
Diff line number
Diff line change
@@ -30,7 +30,7 @@ Here is an example flowing from:
30
30
31
31
text -> tokens -> LLM -> probability distribution -> predicted token -> text
32
32
33
-
<figure><imgsrc="https://files.gitbook.com/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FrvfCoPdoQeoovZiqNG90%2Fuploads%2FPzJzmVS3uNhbvseiJbgi%2FScreenshot%20from%202023-12-13%2013-19-33.png?alt=media&token=11d57b2a-6aa3-4374-b26c-afc6f531d2f3"alt=""><figcaption><p>The flow of inputs through an LLM. In this case the inputs are "What is Baldur's Gate 3?" and the output token "14" maps to the word "I"</p></figcaption></figure>
33
+
<figure><imgsrc="../../.gitbook/assets/Chatbots_Limitations-Diagram.svg"alt=""><figcaption><p>The flow of inputs through an LLM. In this case the inputs are "What is Baldur's Gate 3?" and the output token "14" maps to the word "I"</p></figcaption></figure>
34
34
35
35
{% hint style="info" %}
36
36
We have simplified the tokenization process. Words do not always map directly to tokens. For instance, the word "Baldur's" may actually map to multiple tokens. For more information on tokenization checkout[HuggingFace's summary](https://huggingface.co/docs/transformers/tokenizer\_summary).
@@ -108,11 +108,11 @@ What does an `embedding` look like? `Embeddings` are just vectors (for our use c
108
108
embedding_1= embed("King")# embed returns something like [0.11, -0.32, 0.46, ...]
109
109
```
110
110
111
-
<figure><imgsrc="../../.gitbook/assets/embedding_king.png"alt=""><figcaption><p>The flow of word -> token -> embedding</p></figcaption></figure>
111
+
<figure><imgsrc="../../.gitbook/assets/Chatbots_King-Diagram.svg"alt=""><figcaption><p>The flow of word -> token -> embedding</p></figcaption></figure>
112
112
113
113
`Embeddings` aren't limited to words, we have models that can embed entire sentences.
114
114
115
-
<figure><imgsrc="../../.gitbook/assets/embeddings_tokens.png"alt=""><figcaption><p>The flow of sentence -> tokens -> embedding</p></figcaption></figure>
115
+
<figure><imgsrc="../../.gitbook/assets/Chatbots_Tokens-Diagram.svg"alt=""><figcaption><p>The flow of sentence -> tokens -> embedding</p></figcaption></figure>
116
116
117
117
Why do we care about`embeddings`?`Embeddings` have a very interesting property. Words and sentences that have close[semantic similarity](https://en.wikipedia.org/wiki/Semantic\_similarity) sit closer to one another in vector space than words and sentences that do not have close semantic similarity.
118
118
@@ -157,7 +157,7 @@ print(context)
157
157
158
158
Thereis a lot going onwith this, let's check out this diagram and step through it.
159
159
160
-
<figure><imgsrc="../../.gitbook/assets/chatbot_flow.png"alt=""><figcaption><p>The flow of taking a document, splitting it into chunks, embedding those chunks,and then retrieving a chunk based off of a users query</p></figcaption></figure>
160
+
<figure><imgsrc="../../.gitbook/assets/Chatbots_Flow-Diagram.svg"alt=""><figcaption><p>The flow of taking a document, splitting it into chunks, embedding those chunks,and then retrieving a chunk based off of a users query</p></figcaption></figure>
161
161
162
162
Step1: We take the documentand split it into chunks. Chunks are typically a paragraphor twoin size. There are many ways to split documents into chunks,for more information check out [this guide](https://www.pinecone.io/learn/chunking-strategies/).