Repository files navigation

• JDK 17.0 or later
• A key for OpenAI API
• Optional: a key for Cohere API (you can get ithere) if you want to add reranking at the end

Good to know:

1. To run the application in dev mode:

./mvnw quarkus:dev

2. Open the app athttp://localhost:8080/.

NOTE: Quarkus now ships with a Dev UI, which is available in dev mode only athttp://localhost:8080/q/dev/.

3. For debugging a running Quarkus application, put your breakpoints and select Run > Attach to Process, then select the Quarkus process (in IntelliJ)

During this workshop we will create an LLM-powered customer support agent chatbot for a car rental company in 7 steps. We start from the base functionality (step 1) and add features in the subsequent steps. The result after each step is located in a separate directory (stepXX). The final solution is in thestep07 directory.

We recommend to start by checking out themain branch and then opening theproject fromstep01 in your IDE and using that directory throughout theworkshop. The other option is to make a copy of it. If you later need toreset to a particular step, either overwrite your working directory with thedirectory for the step you want to reset to, or, in your IDE, open theproject from the step directory you want to reset to.

If you make any changes to thestepXX directories, you can always reset them back by executing:

git restore stepXX

NOTE: This will make you lose all your local changes!

Before actually starting the workshop, make sure you have set the OpenAI APIkey as an environment variable:

export OPENAI_API_KEY=<your-key>

and if you're going to use Cohere for reranking (step 7), you'll also need the Cohere API key:

export COHERE_API_KEY=<your-key>

Let's get started!

Download the workshop either by cloning the repository on your machine or downloading the zip file:

git clone https://github.com/langchain4j/quarkus-langchain4j-uphill-workshop.git

Or:

curl -L -o workshop.zip https://github.com/langchain4j/quarkus-langchain4j-uphill-workshop/archive/refs/heads/main.zip

To get started, make sure you use thestep01 directory, or create a copy of it.

This is a functioning skeleton for a web app with a chatbot. You can run it as follows

mvn quarkus:dev

or if you prefer to use the maven wrapper:

./mvnw quarkus:dev

NOTE: If you run into an error about the mvnw maven wrapper, you can give execution permission for the file by navigating to the project folder and executingchmod +x mvnw.

or if you installed theQuarkus CLI, you can also use:

quarkus dev

This will bring up the page atlocalhost:8080. Open it and click the redrobot icon in the bottom right corner to start chatting with the chatbot.The chatbot is calling GPT-4o (OpenAI) via the backend. You can test it outand observe that it has memory.

Example:

User: My name is Klaus.AI: Hi Klaus, nice to meet you.User: What is my name?AI: Your name is Klaus.

This is how memory is built up for LLMs

In the console, you can observe the calls that are made to OpenAI behind the scenes, notice the roles 'user' (UserMessage) and 'assistant' (AiMessage).

Play around with the model parameters insrc/main/resources/application.properties. If you don’t haveautocompletion, you can search through them in the Quarkus DevUI atlocalhost:8080/q/dev underConfiguration (use the filterto find properties containingopenai.chat-model).

IMPORTANT: After changing a configuration property, you need toforce a restart the application to apply the changes. Simply submitting anew chat message in the UI does not trigger it (it only sends a websocketmessage rather than an HTTP request), so you have to refresh the page inyour browser.

The precise meaning of most model parameters is described on the website ofOpenAI:https://platform.openai.com/docs/api-reference/chat/create

Examples to try:

• quarkus.langchain4j.openai.chat-model.temperature controls therandomness of the model's responses. Lowering the temperature will make themodel more conservative, while increasing it will make it more creative. Tryasking "Describe a sunset over the mountains" while setting the temperatureto 0.1 and then to, say, 1.5, and observe the different style of theresponse. With a too high temperature over 1.5, the model often startsproducing garbage, or fails to produce a valid response at all.

• quarkus.langchain4j.openai.chat-model.max-tokens limits the length of theresponse. Try setting it to 50 and see how the model cuts off the responseafter 50 tokens.

• quarkus.langchain4j.openai.chat-model.frequency-penalty defines how muchthe model should avoid repeating itself. Try setting the penalty to 2 (whichis the maximum for OpenAI models) and see how the model tries to avoidrepeating words in a single response. For example, ask it to "Repeat theword hedgehog 50 times". While with frequency penalty around 0, the modelgladly repeats the word 50 times, but with 2, it will most likely startproducing garbage after repeating the word a few times.

Instead of passing the response as one block of text when it is ready, enable streaming mode. This will allow us to display the reply token per token, while they come in.

It is achieved by changing the return type ofCustomerSupportAgent.chat()to the reactiveMulti<String>. You'll have to update theCustomerSupportAgentWebSocket.onTextMessage() method accordingly toreturn aMulti too.

Unfortunately, the UI used in this workshop does not properly supportstreaming mode at this moment, so it will look somewhat ugly. Afterexperimenting, please revert back the changes before moving to Step 4.

In the next step, we will fix the problem where the model still doesn't knowits role as a car rental customer assistant:

User: Can I cancel my booking?AI: No, I don't know what you are talking about ...

Add aSystemMessage so the model knows it is a car rental customerassistant. This is done by adding a@SystemMessage annotation to theCustomerSupportAgent.chat() method. The value of the annotation should be:

You are a customer support agent of a car rental company 'Miles of Smiles'.You are friendly, polite and concise.

Observe that the agent is now happy to help with bookings, but does makerules up when it comes to cancellation period.

Example:

User: Can I cancel my booking?AI: Yes, I can help you with that ...User: What is the cancellation period?AI: [some nonsense]

Add a RAG system that allows the chatbot to use relevant parts of our Terms of Use (you can find themhere) for answering the customers.

1. Ingestion phase: the documents (files, websites, ...) are loaded, split, turned into meaning vectors (embeddings) and stored in an embedding store

2. Retrieval phase: with every user prompt, the relevant fragments of our documents are collected by comparing the meaning vector of the prompt with the vectors in the embedding store. The relevant segments are then passed along to the model together with the original question.

More info on easy RAG in Quarkus can be foundhere.

To add RAG to the application, let's use the Easy RAG extension:

<dependency>    <groupId>io.quarkiverse.langchain4j</groupId>    <artifactId>quarkus-langchain4j-easy-rag</artifactId>    <version>${quarkus-langchain4j.version}</version></dependency>

Add the following configuration toapplication.properties:

quarkus.langchain4j.easy-rag.path=src/main/resources/dataquarkus.langchain4j.easy-rag.max-segment-size=100quarkus.langchain4j.easy-rag.max-overlap-size=25quarkus.langchain4j.easy-rag.max-results=3

Explanation:

• quarkus.langchain4j.easy-rag.path is the path to the directory with thedocuments. This is the only mandatory property.
• quarkus.langchain4j.easy-rag.max-segment-size is the maximum size of asegment in tokens.
• quarkus.langchain4j.easy-rag.max-overlap-size is the maximum size of theoverlap (between adjacent segments) in tokens.
• quarkus.langchain4j.easy-rag.max-results is the maximum number ofsegments to return while searching for relevant segments.

Now let's try to chat with the bot:

User: What is the cancellation period?AI: ... 11 days ... 4 days ...

Let's now also have a look at Dev UI, because it has some nice features thatallow you to see deeper into how RAG works. Open the Dev UI either bypressing 'd' in the terminal where Quarkus is running, or by openinglocalhost:8080/q/dev in your browser.

Let's try to talk to the chatbot through the Dev UI interface. Click the'Chat' button inside the 'LangChain4j' card. The system message should getpopulated automatically. Make sure the 'Enable RAG' checkbox is checked, andtry asking the same question again. You will notice that when the responsearrives, your message will be replaced by the augmented message, containingthe extra context added by the retrieval augmentor.

Another interesting feature is being able to search just for relevantembeddings to any question, without actually submitting the question to thechatbot. This is under the 'Embedding store' link in the 'LangChain4j' card.In the bottom of the page, there's a form titled 'Search for relevantembeddings'. Try searching for 'cancellation period' and you will see thatsegments that talk about the cancellation period should pop up at thetop and have a higher relevance score.

Note: if you want to log the requests and responses of the embedding model,set these two properties, but be aware that they are very long and makethe log hard to read:

quarkus.langchain4j.openai.embedding-model.log-requests=truequarkus.langchain4j.openai.embedding-model.log-responses=true

Now let's give the bot the ability to work with customers and their bookings.We won't use a real database, only a simple CDI bean that imitates a database.

Create records for Booking and Customer:

public record Booking(String bookingNumber,                       LocalDate dateFrom,                       LocalDate dateTo,                       Customer customer) {}public record Customer(String name,                        String surname) {}

Create an empty application-scoped bean namedBookingService to act as the repository of bookings.Then create a classBookingTools that holds the tools for working with bookings and injects theBookingService:

@ApplicationScopedpublicclassBookingTools {@InjectBookingServicebookingService;@ToolpublicBookinggetBookingDetails(StringbookingNumber,StringcustomerName,StringcustomerSurname) {System.out.println("==========================================================================================");System.out.printf("[Tool]: Getting details for booking %s for %s %s...%n",bookingNumber,customerName,customerSurname);System.out.println("==========================================================================================");returnbookingService.getBookingDetails(bookingNumber,customerName,customerSurname);    }@ToolpublicvoidcancelBooking(StringbookingNumber,StringcustomerName,StringcustomerSurname) {System.out.println("==========================================================================================");System.out.printf("[Tool]: Cancelling booking %s for %s %s...%n",bookingNumber,customerName,customerSurname);System.out.println("==========================================================================================");bookingService.cancelBooking(bookingNumber,customerName,customerSurname);    }}

Also don't forget to make these tools available to the AI Service (you have toadd thetools = BookingTools.class parameter to the@RegisterAiServiceannotation).

Try to add and implement the required methods in theBookingService bean.You can choose what bookings should be available out of the box and add theminside a@PostConstruct method.

An example solution follows:

@ApplicationScopedpublicclassBookingService {privateList<Booking>bookings =newArrayList<>();@PostConstructpublicvoidinitialize() {// can't be cancelled because it is shorter than 4 daysBookingbooking1 =newBooking("123-456",LocalDate.now().plusDays(17),LocalDate.now().plusDays(19),newCustomer("Klaus","Heisler"));bookings.add(booking1);// can't be cancelled because it starts in less than 11 daysBookingbooking2 =newBooking("111-111",LocalDate.now().plusDays(2),LocalDate.now().plusDays(8),newCustomer("David","Wood"));bookings.add(booking2);// can be cancelledBookingbooking3 =newBooking("222-222",LocalDate.now().plusDays(12),LocalDate.now().plusDays(21),newCustomer("Martin","Oak"));bookings.add(booking3);    }publicvoidcancelBooking(StringbookingNumber,StringcustomerName,StringcustomerSurname) {Bookingbooking =getBookingDetails(bookingNumber,customerName,customerSurname);// too late to cancelif(booking.dateFrom().minusDays(11).isBefore(LocalDate.now())) {thrownewBookingCannotBeCancelledException(bookingNumber);        }// too short to cancelif(booking.dateTo().minusDays(4).isBefore(booking.dateFrom())) {thrownewBookingCannotBeCancelledException(bookingNumber);        }bookings.remove(booking);    }publicBookinggetBookingDetails(StringbookingNumber,StringcustomerName,StringcustomerSurname) {returnbookings.stream()                .filter(booking ->booking.bookingNumber().equals(bookingNumber))                .filter(booking ->booking.customer().name().equals(customerName))                .filter(booking ->booking.customer().surname().equals(customerSurname))                .findAny()                .orElseThrow(() ->newBookingNotFoundException(bookingNumber));    }}

Observe how the chatbot behaves now.You can ensure that the tool methods are called by looking at the console logs.

Example:

User: Cancel my bookingAI: Please provide your booking number, name and surname...

The sample solution contains three bookings created automatically at start:

• Booking for Klaus Heisler, number 123-456: can't be cancelled because it is shorter than 4 days
• Booking for David Wood, number 111-111: can't be cancelled because it starts in less than 11 days
• Booking for Martin Oak, number 222-222: can be cancelled

Note that we have applied safeguards for the cancellation period and theminimum booking length in the example solution. In some cases, the LLM mayapply these automatically because we still have RAG in place and the LLM candeduce these rules from the terms of use, and skip executing thecancelBooking method. But generally, it is safer to have these checks inplace in the backend as well and not rely on the model to apply them.

Let’s make RAG better: add a RetrievalAugmentor with a QueryCompressor and a Reranker (using your Cohere key)More details on advanced RAG can be foundhere.