Repository files navigation

LLamaWorker is a HTTP API server developed based on theLLamaSharp project. It provides an OpenAI-compatible API, making it easy for developers to integrate Large Language Models (LLM) into their applications.

English |中文

• OpenAI API Compatible: Offers an API similar to OpenAI's / Azure OpenAI, making migration and integration easy.
• Multi-Model Support: Supports configuring and switching between different models to meet the needs of various scenarios.
• Streaming Response: Supports streaming responses to improve the efficiency of processing large responses.
• Embedding Support: Provides text embedding functionality with support for various embedding models.
• chat templates: Provides some common chat templates.
• Auto-Release: Supports automatic release of loaded models.
• Function Call: Supports function calls.
• API Key Authentication: Supports API Key authentication.
• Test UI: Provides a friendly development test UI.
• Gradio UI Demo: Provides a UI demo based on Gradio.NET.

When the server starts, it checks for the existence of thewwwroot directory. If it exists, it enables static file services.

Awwwroot.zip file is provided in theReleases, you can unzip it to the project directory, and then run the project to access the test UI.

This interface provides model switching, chat completion, prompt completion, chat history management, token generation rate display, and other functions. You can test the model effect through this interface.

A Vulkan backend compiled version is provided in the release, you can download the corresponding compiled version fromReleases:

• LLamaWorker-Vulkan-win-x64.zip
• LLamaWorker-Vulkan-linux-x64.zip

After downloading and unzipping, modify the configuration in theappsettings.json file, and you can run the software and start using it.

For other backends, you can also download theVulkan version, go tollama.cpp to download the corresponding compiled version, and replace the relevant libraries. You can also compile thellama.cpp project yourself to get the required libraries.

LLamaWorker supports function calls, and currently provides three templates in the configuration file, and has tested the function call effect ofPhi-3,Qwen2 andLlama3.1.

Function calls are compatible with OpenAI's API, You can test it with the following JSON request:

POST /v1/chat/completions

{"model":"default","messages": [    {"role":"user","content":"Where is the temperature high between Beijing and Shanghai?"    }  ],"tools": [    {"function": {"name":"GetWeatherPlugin-GetCurrentTemperature","description":"Get the current temperature of the specified city。","parameters": {"type":"object","required": ["city"          ],"properties": {"city": {"type":"string","description":"City Name"            }          }        }      },"type":"function"    },    {"function": {"name":"EmailPlugin-SendEmail","description":"Send an email to the recipient.","parameters": {"type":"object","required": ["recipientEmails","subject","body"          ],"properties": {"recipientEmails": {"type":"string","description":"A recipient email list separated by semicolons"            },"subject": {"type":"string"            },"body": {"type":"string"            }          }        }      },"type":"function"    }  ],"tool_choice":"auto"}

1. Clone the repository locally

   git clone https://github.com/sangyuxiaowu/LLamaWorker.git

2. Enter the project directory

   cd LLamaWorker

3. Choose the project file according to your needs. The project provides three versions of the project files:

   • LLamaWorker.Backend.Cpu: For CPU environments.
   • LLamaWorker.Backend.Cuda11: For GPU environments with CUDA 11.
   • LLamaWorker.Backend.Cuda12: For GPU environments with CUDA 12.
   • LLamaWorker.Backend.Vulkan: Vulkan.

   Select the project file that suits your environment for the next step.

4. Install dependencies

   dotnet restore LLamaWorker.Backend.Cpu\LLamaWorker.Backend.Cpu.csproj

   If you are using a CUDA version, replace the project file name accordingly.

5. Modify the configuration fileappsettings.json. The default configuration includes some common open-source model configurations, you only need to modify the model file path (ModelPath) as needed.

6. Start the server

   dotnet run --project LLamaWorker.Backend.Cpu\LLamaWorker.Backend.Cpu.csproj

   If you are using a CUDA version, replace the project file name accordingly.

LLamaWorker offers the following API endpoints:

• /v1/chat/completions: Chat completion requests
• /v1/completions: Prompt completion requests
• /v1/embeddings: Create embeddings
• /models/info: Returns basic information about the model
• /models/config: Returns information about configured models
• /models/{modelId}/switch: Switch to a specified model

This ui is based onGradio.NET.

You can also try the Gradio UI demo by running the following command:

dotnet restore ChatUI\ChatUI.csprojdotnet run --project ChatUI\ChatUI.csproj

Then open the browser and visit the Gradio UI demo.