- Notifications
You must be signed in to change notification settings - Fork403
A web interface for chatting with Alpaca through llama.cpp. Fully dockerized, with an easy to use API.
License
Apache-2.0, MIT licenses found
Licenses found
serge-chat/serge
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
Serge is a chat interface crafted withllama.cpp for running LLM models. No API keys, entirely self-hosted!
- 🌐SvelteKit frontend
- 💾Redis for storing chat history & parameters
- ⚙️FastAPI + LangChain for the API, wrapping calls tollama.cpp using thepython bindings
🎥 Demo:
demo.webm
🐳 Docker:
docker run -d \ --name serge \ -v weights:/usr/src/app/weights \ -v datadb:/data/db/ \ -p 8008:8008 \ ghcr.io/serge-chat/serge:latest
🐙 Docker Compose:
services:serge:image:ghcr.io/serge-chat/serge:latestcontainer_name:sergerestart:unless-stoppedports: -8008:8008volumes: -weights:/usr/src/app/weights -datadb:/data/db/volumes:weights:datadb:
Then, just visithttp://localhost:8008, You can find the API documentation athttp://localhost:8008/api/docs
The following Environment Variables are available:
| Variable Name | Description | Default Value |
|---|---|---|
SERGE_DATABASE_URL | Database connection string | sqlite:////data/db/sql_app.db |
SERGE_JWT_SECRET | Key for auth token encryption. Use a random string | uF7FGN5uzfGdFiPzR |
SERGE_SESSION_EXPIRY | Duration in minutes before a user must reauthenticate | 60 |
NODE_ENV | Node.js running environment | production |
Ensure you have Docker Desktop installed, WSL2 configured, and enough free RAM to run models.
LLaMA will crash if you don't have enough available memory for the model
Need help? Join ourDiscord
Nathan Sarrazin andContributors.Serge is free and open-source software licensed under theMIT License andApache-2.0.
If you discover a bug or have a feature idea, feel free to open an issue or PR.
To run Serge in development mode:
git clone https://github.com/serge-chat/serge.gitcd serge/docker compose -f docker-compose.dev.yml up --buildThe solution will accept a python debugger session on port 5678. Example launch.json for VSCode:
{"version":"0.2.0","configurations": [ {"name":"Remote Debug","type":"python","request":"attach","connect": {"host":"localhost","port":5678 },"pathMappings": [ {"localRoot":"${workspaceFolder}/api","remoteRoot":"/usr/src/app/api/" } ],"justMyCode":false } ]}About
A web interface for chatting with Alpaca through llama.cpp. Fully dockerized, with an easy to use API.