Add support for Ollama (and other compatible endpoints) #10

New Issue

AtHeartEngineer · 2025-07-08T08:36:40-04:00

AtHeartEngineer commented

2025-07-08 08:36:40 -04:00

Originally created by @wbste on 3/7/2025

EDIT: After trying a lot of models on Ollama I couldn't get a single one to work (even though they all support tools). I'll play some more with system prompts and the like, but for now this may not be worthwhile. I know Ollama's SDK supports "forcing" tool use, whereas the OpenAI-compat. endpoint doesn't have the same feature. For anyone else who tries this keep in mind the system prompt is 2600 tokens, so you need to adjust the Modelfile (or env. variable now) to set a larger default than 2048 (for Ollama). Maybe someone else can play with it and see if you get better results?

A lot of the open source community likes to dabble in self-hosted AI models. I'm still looking over the code, but I think if you add the following here:

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY || "",
  baseURL: process.env.OPENAI_BASE_URL || "https://api.openai.com",
});
const model = process.env.OPENAI_MODEL || "gpt-4o";

Then create a .env

OPENAI_API_KEY=doesnotmatter
OPENAI_MODEL=ollamamodelhere
OPENAI_BASE_URL=http://host.docker.internal:11434/v1

and update docker-compose.yml

services:
  probly:
    build:
      context: .
      args:
        - OPENAI_API_KEY=${OPENAI_API_KEY:-}
        - OPENAI_MODEL=${OPENAI_MODEL:-}
        - OPENAI_BASE_URL=${OPENAI_BASE_URL:-}
    ports:
      - "3000:3000"
    environment:
        - OPENAI_API_KEY=${OPENAI_API_KEY:-}
        - OPENAI_MODEL=${OPENAI_MODEL:-}
        - OPENAI_BASE_URL=${OPENAI_BASE_URL:-}
    restart: unless-stopped

You can use Ollama for the AI. I'm still playing around with finding any models smart enough to work with the prompts, however.

*Originally created by @wbste on 3/7/2025* **EDIT**: After trying a lot of models on Ollama I couldn't get a single one to work (even though they all support tools). I'll play some more with system prompts and the like, but for now this may not be worthwhile. I know Ollama's SDK supports "forcing" tool use, whereas the OpenAI-compat. endpoint doesn't have the same feature. For anyone else who tries this keep in mind the system prompt is 2600 tokens, so you need to adjust the Modelfile (or env. variable now) to set a larger default than 2048 (for Ollama). _Maybe someone else can play with it and see if you get better results?_ --- A lot of the open source community likes to dabble in self-hosted AI models. I'm still looking over the code, but I think if you add the following [here](https://github.com/PragmaticMachineLearning/probly/blob/5d9eac76c211e54827c3ee74418c924cd65b545d/src/pages/api/llm.ts#L18C1-L21C24): ```ts const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY || "", baseURL: process.env.OPENAI_BASE_URL || "https://api.openai.com", }); const model = process.env.OPENAI_MODEL || "gpt-4o"; ``` Then create a `.env` ``` OPENAI_API_KEY=doesnotmatter OPENAI_MODEL=ollamamodelhere OPENAI_BASE_URL=http://host.docker.internal:11434/v1 ``` and update `docker-compose.yml` ```yml services: probly: build: context: . args: - OPENAI_API_KEY=${OPENAI_API_KEY:-} - OPENAI_MODEL=${OPENAI_MODEL:-} - OPENAI_BASE_URL=${OPENAI_BASE_URL:-} ports: - "3000:3000" environment: - OPENAI_API_KEY=${OPENAI_API_KEY:-} - OPENAI_MODEL=${OPENAI_MODEL:-} - OPENAI_BASE_URL=${OPENAI_BASE_URL:-} restart: unless-stopped ``` You can use [Ollama](https://ollama.com/) for the AI. I'm still playing around with finding any models smart enough to work with the prompts, however.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github/probly#10