- Notifications
You must be signed in to change notification settings - Fork111
A simple and easy-to-use library for interacting with the Ollama API.
License
pepperoni21/ollama-rs
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
This library was created following theOllama API documentation.
[dependencies]ollama-rs ="0.2.6"
If you absolutely want the latest version, you can use themaster branch by adding the following to yourCargo.toml file:
[dependencies]ollama-rs = {git ="https://github.com/pepperoni21/ollama-rs.git",branch ="master" }
Note that themaster branch may not be stable and may contain breaking changes.
use ollama_rs::Ollama;// By default, it will connect to localhost:11434let ollama =Ollama::default();// For custom values:let ollama =Ollama::new("http://localhost".to_string(),11434);
Feel free to check theChatbot example that shows how to use the library to create a simple chatbot in less than 50 lines of code. You can also check someother examples.
These examples use poor error handling for simplicity, but you should handle errors properly in your code.
use ollama_rs::generation::completion::GenerationRequest;let model ="llama2:latest".to_string();let prompt ="Why is the sky blue?".to_string();let res = ollama.generate(GenerationRequest::new(model, prompt)).await;ifletOk(res) = res{println!("{}", res.response);}
OUTPUTS:The sky appears blue because of a phenomenon called Rayleigh scattering...
Requires thestream feature.
use ollama_rs::generation::completion::GenerationRequest;use tokio::io::{self,AsyncWriteExt};use tokio_stream::StreamExt;let model ="llama2:latest".to_string();let prompt ="Why is the sky blue?".to_string();letmut stream = ollama.generate_stream(GenerationRequest::new(model, prompt)).await.unwrap();letmut stdout = io::stdout();whileletSome(res) = stream.next().await{let responses = res.unwrap();for respin responses{ stdout.write_all(resp.response.as_bytes()).await.unwrap(); stdout.flush().await.unwrap();}}
Same output as above but streamed.
use ollama_rs::generation::completion::GenerationRequest;use ollama_rs::models::ModelOptions;let model ="llama2:latest".to_string();let prompt ="Why is the sky blue?".to_string();let options =ModelOptions::default().temperature(0.2).repeat_penalty(1.5).top_k(25).top_p(0.25);let res = ollama.generate(GenerationRequest::new(model, prompt).options(options)).await;ifletOk(res) = res{println!("{}", res.response);}
OUTPUTS:1. Sun emits white sunlight: The sun consists primarily ...
Every message sent and received will be stored in the library's history.
Example with history:
use ollama_rs::generation::chat::{ChatMessage,ChatMessageRequest};use ollama_rs::history::ChatHistory;let model ="llama2:latest".to_string();let prompt ="Why is the sky blue?".to_string();// `Vec<ChatMessage>` implements `ChatHistory`,// but you could also implement it yourself on a custom typeletmut history =vec![];let res = ollama.send_chat_messages_with_history(&mut history,// <- messages will be saved hereChatMessageRequest::new( model,vec![ChatMessage::user(prompt)],// <- You should provide only one message),).await;ifletOk(res) = res{println!("{}", res.message.content);}
Check chat with history examples fordefault andstream
let res = ollama.list_local_models().await.unwrap();
Returns a vector ofLocalModel structs.
let res = ollama.show_model_info("llama2:latest".to_string()).await.unwrap();
Returns aModelInfo struct.
use ollama_rs::models::create::CreateModelRequest;let res = ollama.create_model(CreateModelRequest::path("model".into(),"/tmp/Modelfile.example".into())).await.unwrap();
Returns aCreateModelStatus struct representing the final status of the model creation.
Requires thestream feature.
use ollama_rs::models::create::CreateModelRequest;use tokio_stream::StreamExt;letmut res = ollama.create_model_stream(CreateModelRequest::path("model".into(),"/tmp/Modelfile.example".into())).await.unwrap();whileletSome(res) = res.next().await{let res = res.unwrap();// Handle the status}
Returns aCreateModelStatusStream that will stream every status update of the model creation.
let _ = ollama.copy_model("mario".into(),"mario_copy".into()).await.unwrap();
let _ = ollama.delete_model("mario_copy".into()).await.unwrap();
use ollama_rs::generation::embeddings::request::GenerateEmbeddingsRequest;let request =GenerateEmbeddingsRequest::new("llama2:latest".to_string(),"Why is the sky blue?".into());let res = ollama.generate_embeddings(request).await.unwrap();
use ollama_rs::generation::embeddings::request::GenerateEmbeddingsRequest;let request =GenerateEmbeddingsRequest::new("llama2:latest".to_string(),vec!["Why is the sky blue?","Why is the sky red?"].into());let res = ollama.generate_embeddings(request).await.unwrap();
Returns aGenerateEmbeddingsResponse struct containing the embeddings (a vector of floats).
use ollama_rs::coordinator::Coordinator;use ollama_rs::generation::chat::{ChatMessage,ChatMessageRequest};use ollama_rs::generation::tools::implementations::{DDGSearcher,Scraper,Calculator};use ollama_rs::models::ModelOptions;letmut history =vec![];letmut coordinator =Coordinator::new(ollama,"qwen2.5:32b".to_string(), history).options(ModelOptions::default().num_ctx(16384)).add_tool(DDGSearcher::new()).add_tool(Scraper{}).add_tool(Calculator{});let resp = coordinator.chat(vec![ChatMessage::user("What is the current oil price?")]).await.unwrap();println!("{}", resp.message.content);
Uses the given tools (such as searching the web) to find an answer, feeds that answer back into the LLM, and returns aChatMessageResponse with the answer to the question.
Thefunction macro simplifies the creation of custom tools. Below is an example of a tool that retrieves the current weather for a specified city:
/// Retrieve the weather for a specified city.////// * city - The city for which to get the weather.#[ollama_rs::function]asyncfnget_weather(city:String) ->Result<String,Box<dyn std::error::Error +Sync +Send>>{let url =format!("https://wttr.in/{city}?format=%C+%t");let response = reqwest::get(&url).await?.text().await?;Ok(response)}
To create a custom tool, define a function that returns aResult<String, Box<dyn std::error::Error + Sync + Send>> and annotate it with thefunction macro. This function will be automatically converted into a tool that can be used with theCoordinator, just like any other tool.
Ensure that the doc comment above the function clearly describes the tool's purpose and its parameters. This information will be provided to the LLM to help it understand how to use the tool.
For a more detailed example, see thefunction call example.
About
A simple and easy-to-use library for interacting with the Ollama API.