{% extends "admin/base.html" %} {% block title %}Help & Credits{% endblock %} {% block header_title %}Help & Credits{% endblock %} {% block content %}

🚀 Getting Started: Your Workflow

  1. Configure Servers: Go to the Server Management page and add the URLs of all your Ollama instances.
  2. Create a User: Navigate to the User Management page. This is where you'll create and manage all users.
  3. Generate an API Key: From the user list, click "Manage Keys" for a specific user. On their details page, you can generate new keys and set optional rate limits.
  4. Copy the Key: The full API key is shown only once upon creation. Copy it and store it securely.
  5. Use the Key: In your applications, replace your Ollama URL with the proxy's URL (e.g., `http://127.0.0.1:8080`) and provide the API key in the `Authorization` header as a Bearer token.

🖥️ The Dashboard: Your Monitoring Hub

The main dashboard provides a real-time overview of your proxy server and connected Ollama instances.

  • System Status: These gauges show the live CPU, Memory, and Disk usage of the machine running the proxy server. This helps you monitor server health at a glance.
  • Active Ollama Models: This table shows all models that are currently loaded into memory on your backend Ollama servers (equivalent to running `ollama ps`). It includes details like which server is running the model, its size, context length, and when it will expire from memory.

💡 Core Concepts

🛡️ API Key Authentication

Every request to the proxy must include a valid API key. This is the primary security feature, preventing unauthorized access to your models.

⚖️ Load Balancing

When a request is received, the proxy distributes it among all active backend servers in a round-robin fashion. This balances the load and improves availability.

🧠 Smart Model Routing

When you request a specific model (e.g., "llama3"), the proxy is intelligent. It first looks up which of your backend servers have that model available and only routes the request to one of them. This ensures your request never fails because it was sent to a server without the right model.

Important: For Smart Routing to work, you must periodically click the "Refresh Models" button on the Server Management page. This keeps the proxy's catalog of available models up to date.

🌐 Federated Model Listing

The standard /api/tags endpoint is enhanced. When you call it through the proxy, it contacts all active backend servers, gathers their model lists, and returns a single, de-duplicated list of all unique models available across your entire setup.

👨‍💻 Usage Examples

cURL


curl http://127.0.0.1:8080/api/generate \
  -H "Authorization: Bearer op_prefix_secret" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3",
    "prompt": "Why is the sky blue?",
    "stream": false
  }'

⚡ Redis Setup Guide

Why Do I Need Redis?

Redis is an in-memory database that is extremely fast. This application uses it for two optional but important security features:

  • API Rate Limiting: Prevents abuse by limiting the number of requests a single API key can make in a given time frame.
  • Brute-Force Protection: Temporarily locks an IP address after too many failed login attempts to the admin panel.
Note: The proxy will work perfectly without Redis, but these security features will be disabled.

Installation

The easiest way to run Redis on any platform is with Docker. If you don't use Docker, choose your OS below.

🐳 Docker (Recommended)

This single command will download and run a Redis container that restarts automatically.

docker run -d --name redis-stack -p 6379:6379 --restart always redis/redis-stack:latest

🐧 Windows (via WSL 2)

Redis is not officially supported on Windows, but it runs perfectly under the Windows Subsystem for Linux (WSL).

  1. Open PowerShell as an Administrator and run: wsl --install
  2. After it's done, open your new Linux terminal (e.g., Ubuntu) and run:
sudo apt update && sudo apt upgrade
sudo apt install redis-server
sudo service redis-server start

🍏 macOS (via Homebrew)

brew install redis
brew services start redis

🐧 Linux (Ubuntu/Debian)

sudo apt-get install redis-server
sudo systemctl enable --now redis-server

Configuration

Once Redis is running, go to the Settings page in the admin UI and enter your Redis connection details (the defaults are usually fine for a local install). Save the settings, and the proxy will attempt to connect automatically.

💖 Credits & Acknowledgements

This application was developed with passion by the open-source community. It stands on the shoulders of giants and wouldn't be possible without the following incredible projects:

Project built and maintained by ParisNeo with help from AI and cool developers (check the contributors list in the github page).

Visit the project on GitHub to contribute, report issues, or star the repository!

On this page

{% endblock %}