Files
AutoGPT/autogpt_platform/docker-compose.platform.yml
Reinier van der Leer d23248f065 feat(backend/copilot): Copilot Executor Microservice (#12057)
Uncouple Copilot task execution from the REST API server. This should
help performance and scalability, and allows task execution to continue
regardless of the state of the user's connection.

- Resolves #12023

### Changes 🏗️

- Add `backend.copilot.executor`->`CoPilotExecutor` (setup similar to
`backend.executor`->`ExecutionManager`).

This executor service uses RabbitMQ-based task distribution, and sticks
with the existing Redis Streams setup for task output. It uses a cluster
lock mechanism to ensure a task is only executed by one pod, and the
`DatabaseManager` for pooled DB access.

- Add `backend.data.db_accessors` for automatic choice of direct/proxied
DB access

Chat requests now flow: API → RabbitMQ → CoPilot Executor → Redis
Streams → SSE Client. This enables horizontal scaling of chat processing
and isolates long-running LLM operations from the API service.

- Move non-API Copilot stuff into `backend.copilot` (from
`backend.api.features.chat`)
  - Updated import paths for all usages

- Move `backend.executor.database` to `backend.data.db_manager` and add
methods for copilot executor
  - Updated import paths for all usages
- Make `backend.copilot.db` RPC-compatible (-> DB ops return ~~Prisma~~
Pydantic models)
  - Make `backend.data.workspace` RPC-compatible
  - Make `backend.data.graphs.get_store_listed_graphs` RPC-compatible

DX:
- Add `copilot_executor` service to Docker setup

Config:
- Add `Config.num_copilot_workers` (default 5) and
`Config.copilot_executor_port` (default 8008)
- Remove unused `Config.agent_server_port`

> [!WARNING]
> **This change adds a new microservice to the system, with entrypoint
`backend.copilot.executor`.**
> The `docker compose` setup has been updated, but if you run the
Platform on something else, you'll have to update your deployment config
to include this new service.
>
> When running locally, the `CoPilotExecutor` uses port 8008 by default.

### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
  - [x] Copilot works
    - [x] Processes messages when triggered
    - [x] Can use its tools

#### For configuration changes:

- [x] `.env.default` is updated or already compatible with my changes
- [x] `docker-compose.yml` is updated or already compatible with my
changes
- [x] I have included a list of my configuration changes in the PR
description (under **Changes**)

---------

Co-authored-by: Zamil Majdy <zamil.majdy@agpt.co>
2026-02-17 16:15:28 +00:00

373 lines
9.6 KiB
YAML

# Environment Variable Loading Order (first → last, later overrides earlier):
# 1. backend/.env.default - Default values for all settings
# 2. backend/.env - User's custom configuration (if exists)
# 3. environment key - Docker-specific overrides defined below
# 4. Shell environment - Variables exported before running docker compose
# 5. CLI arguments - docker compose run -e VAR=value
# Common backend environment - Docker service names
x-backend-env: &backend-env # Docker internal service hostnames (override localhost defaults)
PYRO_HOST: "0.0.0.0"
AGENTSERVER_HOST: rest_server
SCHEDULER_HOST: scheduler_server
DATABASEMANAGER_HOST: database_manager
EXECUTIONMANAGER_HOST: executor
NOTIFICATIONMANAGER_HOST: notification_server
CLAMAV_SERVICE_HOST: clamav
DB_HOST: db
REDIS_HOST: redis
RABBITMQ_HOST: rabbitmq
# Override Supabase URL for Docker network
SUPABASE_URL: http://kong:8000
# Database connection string for Docker network
# This cannot be constructed like in .env because we cannot interpolate values set here (DB_HOST)
DATABASE_URL: postgresql://postgres:your-super-secret-and-long-postgres-password@db:5432/postgres?connect_timeout=60&schema=platform
DIRECT_URL: postgresql://postgres:your-super-secret-and-long-postgres-password@db:5432/postgres?connect_timeout=60&schema=platform
# Common env_file configuration for backend services
x-backend-env-files: &backend-env-files
env_file:
- backend/.env.default # Base defaults (always exists)
- path: backend/.env # User overrides (optional)
required: false
services:
migrate:
build:
context: ../
dockerfile: autogpt_platform/backend/Dockerfile
target: migrate
command: ["sh", "-c", "prisma generate && python3 gen_prisma_types_stub.py && prisma migrate deploy"]
develop:
watch:
- path: ./
target: autogpt_platform/backend/migrations
action: rebuild
depends_on:
db:
condition: service_healthy
<<: *backend-env-files
environment:
<<: *backend-env
networks:
- app-network
restart: on-failure
healthcheck:
test:
[
"CMD-SHELL",
"prisma migrate status | grep -q 'No pending migrations' || exit 1",
]
interval: 30s
timeout: 10s
retries: 3
start_period: 5s
redis:
image: redis:latest
ports:
- "6379:6379"
networks:
- app-network
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 10s
timeout: 5s
retries: 5
rabbitmq:
image: rabbitmq:4.1.4
container_name: rabbitmq
healthcheck:
test: rabbitmq-diagnostics -q ping
interval: 30s
timeout: 10s
retries: 5
start_period: 10s
<<: *backend-env-files
environment:
<<: *backend-env
ports:
- "5672:5672"
rest_server:
build:
context: ../
dockerfile: autogpt_platform/backend/Dockerfile
target: server
command: ["rest"] # points to entry in [tool.poetry.scripts] in pyproject.toml
develop:
watch:
- path: ./
target: autogpt_platform/backend/
action: rebuild
depends_on:
redis:
condition: service_healthy
db:
condition: service_healthy
migrate:
condition: service_completed_successfully
rabbitmq:
condition: service_healthy
<<: *backend-env-files
environment:
<<: *backend-env
ports:
- "8006:8006"
networks:
- app-network
logging:
driver: json-file
options:
max-size: "10m"
max-file: "3"
executor:
build:
context: ../
dockerfile: autogpt_platform/backend/Dockerfile
target: server
command: ["executor"] # points to entry in [tool.poetry.scripts] in pyproject.toml
develop:
watch:
- path: ./
target: autogpt_platform/backend/
action: rebuild
depends_on:
redis:
condition: service_healthy
rabbitmq:
condition: service_healthy
db:
condition: service_healthy
migrate:
condition: service_completed_successfully
database_manager:
condition: service_started
<<: *backend-env-files
environment:
<<: *backend-env
ports:
- "8002:8002"
networks:
- app-network
logging:
driver: json-file
options:
max-size: "10m"
max-file: "3"
copilot_executor:
build:
context: ../
dockerfile: autogpt_platform/backend/Dockerfile
target: server
command: ["python", "-m", "backend.copilot.executor"]
develop:
watch:
- path: ./
target: autogpt_platform/backend/
action: rebuild
depends_on:
redis:
condition: service_healthy
rabbitmq:
condition: service_healthy
db:
condition: service_healthy
migrate:
condition: service_completed_successfully
database_manager:
condition: service_started
<<: *backend-env-files
environment:
<<: *backend-env
ports:
- "8008:8008"
networks:
- app-network
logging:
driver: json-file
options:
max-size: "10m"
max-file: "3"
websocket_server:
build:
context: ../
dockerfile: autogpt_platform/backend/Dockerfile
target: server
command: ["ws"] # points to entry in [tool.poetry.scripts] in pyproject.toml
develop:
watch:
- path: ./
target: autogpt_platform/backend/
action: rebuild
depends_on:
db:
condition: service_healthy
redis:
condition: service_healthy
migrate:
condition: service_completed_successfully
database_manager:
condition: service_started
<<: *backend-env-files
environment:
<<: *backend-env
ports:
- "8001:8001"
networks:
- app-network
logging:
driver: json-file
options:
max-size: "10m"
max-file: "3"
database_manager:
build:
context: ../
dockerfile: autogpt_platform/backend/Dockerfile
target: server
command: ["db"] # points to entry in [tool.poetry.scripts] in pyproject.toml
develop:
watch:
- path: ./
target: autogpt_platform/backend/
action: rebuild
depends_on:
db:
condition: service_healthy
migrate:
condition: service_completed_successfully
<<: *backend-env-files
environment:
<<: *backend-env
ports:
- "8005:8005"
networks:
- app-network
logging:
driver: json-file
options:
max-size: "10m"
max-file: "3"
scheduler_server:
build:
context: ../
dockerfile: autogpt_platform/backend/Dockerfile
target: server
command: ["scheduler"] # points to entry in [tool.poetry.scripts] in pyproject.toml
develop:
watch:
- path: ./
target: autogpt_platform/backend/
action: rebuild
depends_on:
db:
condition: service_healthy
redis:
condition: service_healthy
rabbitmq:
condition: service_healthy
migrate:
condition: service_completed_successfully
database_manager:
condition: service_started
# healthcheck:
# test:
# [
# "CMD",
# "curl",
# "-f",
# "-X",
# "POST",
# "http://localhost:8003/health_check",
# ]
# interval: 10s
# timeout: 10s
# retries: 5
<<: *backend-env-files
environment:
<<: *backend-env
ports:
- "8003:8003"
networks:
- app-network
logging:
driver: json-file
options:
max-size: "10m"
max-file: "3"
notification_server:
build:
context: ../
dockerfile: autogpt_platform/backend/Dockerfile
target: server
command: ["notification"] # points to entry in [tool.poetry.scripts] in pyproject.toml
develop:
watch:
- path: ./
target: autogpt_platform/backend/
action: rebuild
depends_on:
db:
condition: service_healthy
rabbitmq:
condition: service_healthy
migrate:
condition: service_completed_successfully
database_manager:
condition: service_started
<<: *backend-env-files
environment:
<<: *backend-env
ports:
- "8007:8007"
networks:
- app-network
logging:
driver: json-file
options:
max-size: "10m"
max-file: "3"
frontend:
build:
context: ../
dockerfile: autogpt_platform/frontend/Dockerfile
target: prod
args:
NEXT_PUBLIC_PW_TEST: ${NEXT_PUBLIC_PW_TEST:-false}
depends_on:
db:
condition: service_healthy
migrate:
condition: service_completed_successfully
ports:
- "3000:3000"
networks:
- app-network
logging:
driver: json-file
options:
max-size: "10m"
max-file: "3"
# Load environment variables in order (later overrides earlier)
env_file:
- path: ./frontend/.env.default # Base defaults (always exists)
- path: ./frontend/.env # User overrides (optional)
required: false
environment:
# Server-side environment variables (Docker service names)
# These override the localhost URLs from env files when running in Docker
AUTH_CALLBACK_URL: http://rest_server:8006/auth/callback
SUPABASE_URL: http://kong:8000
AGPT_SERVER_URL: http://rest_server:8006/api
AGPT_WS_SERVER_URL: ws://websocket_server:8001/ws
networks:
app-network:
driver: bridge