mirror of
https://github.com/Significant-Gravitas/AutoGPT.git
synced 2026-01-06 22:03:59 -05:00
This PR addresses multiple critical and medium security vulnerabilities
that could lead to Denial of Service (DoS) attacks. All fixes implement
defense-in-depth strategies with comprehensive testing.
### Changes 🏗️
#### **Critical Security Fixes:**
1. **GHSA-m2wr-7m3r-p52c - ReDoS in CodeExtractionBlock**
- Fixed catastrophic backtracking in regex patterns `\s+[\s\S]*?` and
`\s+(.*?)`
- Replaced with safer patterns: `[ \t]*\n([^\s\S]*?)`
- Files: `backend/blocks/code_extraction_block.py`
2. **GHSA-955p-gpfx-r66j - AITextSummarizerBlock Memory Amplification**
- Added 1MB text size limit and 100 chunk maximum
- Prevents 10K input → 50G memory amplification attacks
- Files: `backend/blocks/llm.py`
3. **GHSA-5cqw-g779-9f9x - RSS Feed XML Bomb DoS**
- Added 10MB feed size limit and 30s timeout
- Prevents deep XML parsing memory exhaustion
- Files: `backend/blocks/rss.py`
4. **GHSA-7g34-7fvq-xxq6 - File Storage Disk Exhaustion**
- Added 100MB per file and 1GB per execution directory limits
- Prevents disk space exhaustion from file uploads
- Files: `backend/util/file.py`
5. **GHSA-pppq-xx2w-7jpq - ExtractTextInformationBlock ReDoS**
- Added 1MB text limit, 1000 match limit, and 5s timeout protection
- Prevents lookahead pattern memory exhaustion
- Files: `backend/blocks/text.py`
6. **GHSA-vw3v-whvp-33v5 - Docker Logging Disk Exhaustion**
- Added log rotation limits at Docker (10MB × 3 files) and application
levels
- Prevents unbounded log growth causing disk exhaustion
- Files: `docker-compose.platform.yml`,
`autogpt_libs/autogpt_libs/logging/config.py`
#### **Additional Security Improvements:**
7. **StepThroughItemsBlock DoS Prevention**
- Added 10,000 item limit and 1MB input size limit
- Prevents large iteration DoS attacks
- Files: `backend/blocks/iteration.py`
8. **XMLParserBlock XML Bomb Prevention**
- Added 10MB XML input size limit
- Files: `backend/blocks/xml_parser.py`
#### **Code Quality:**
- Fixed Python 3.10 typing compatibility issues
- Added comprehensive security test suite
- All code formatted and linted
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] Created comprehensive security test suite covering all
vulnerabilities
- [x] Verified ReDoS patterns are fixed and don't cause timeouts
- [x] Confirmed memory limits prevent amplification attacks
- [x] Tested file size limits prevent disk exhaustion
- [x] Validated log rotation prevents unbounded growth
- [x] Ensured backward compatibility for normal usage
#### For configuration changes:
- [x] `docker-compose.yml` is updated with logging limits
- [x] I have included a list of my configuration changes in the PR
description (under **Changes**)
### Test Plan 🧪
**Security Tests:**
1. **ReDoS Protection**: Tested with malicious regex inputs (large
spaces) - completes without hanging
2. **Memory Limits**: Verified 2MB text input gets truncated to 1MB,
chunk limits enforced
3. **File Size Limits**: Confirmed 200MB files rejected, directory size
limits enforced
4. **Iteration Limits**: Tested 20K item arrays rejected, large JSON
strings rejected
5. **Timeout Protection**: Dangerous regex patterns timeout after 5s
instead of hanging
**Compatibility Tests:**
- Normal functionality preserved for all blocks
- Existing tests pass with new security limits
- Performance impact minimal for typical usage
### Security Impact 🛡️
**Before:** Multiple attack vectors could cause:
- CPU exhaustion (ReDoS attacks)
- Memory exhaustion (amplification attacks)
- Disk exhaustion (file/log bombs)
- Service unavailability
**After:** All attack vectors mitigated with:
- Input validation and size limits
- Timeout protections
- Resource quotas
- Defense-in-depth approach
All fixes maintain backward compatibility while preventing DoS attacks.
🤖 Generated with [Claude Code](https://claude.ai/code)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> Adds robust DoS protections across blocks (regex, memory, iteration,
XML/RSS, file I/O) and enables app/Docker log rotation with
comprehensive tests.
>
> - **Security hardening**:
> - Replace unsafe regex in `backend/blocks/code_extraction_block.py` to
prevent ReDoS; add safer extraction/removal patterns.
> - Constrain LLM summarizer chunking in `backend/blocks/llm.py` (1MB
cap, chunk/overlap validation, chunk count limit).
> - Limit RSS fetching in `backend/blocks/rss.py` (scheme validation,
10MB cap, timeout, bounded read) and return empty on failure.
> - Impose XML size limit (10MB) in `backend/blocks/xml_parser.py`.
> - Add file upload/download limits in `backend/util/file.py`
(100MB/file, 1GB dir quota) and enforce scanning before write.
> - Enable rotating file logs in `autogpt_libs/logging/config.py` (size
+ backups) and Docker json-file log rotation in
`docker-compose.platform.yml`.
> - **Iteration block**:
> - Add item count/string size limits; fix yielded key for dicts; cap
iterations in `backend/blocks/iteration.py`.
> - **Tests**:
> - New `backend/blocks/test/test_security_fixes.py` covering ReDoS,
timeouts, memory/size and iteration limits, XML/file constraints.
> - **Misc**:
> - Typing fallback for `NotRequired` in `activity_status_generator.py`.
> - Dependency updates in `backend/poetry.lock`.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
500e1578b1. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Nicholas Tindle <nicholas.tindle@agpt.co>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Nicholas Tindle <ntindle@users.noreply.github.com>
Co-authored-by: Zamil Majdy <majdyz@users.noreply.github.com>
Co-authored-by: Reinier van der Leer <Pwuts@users.noreply.github.com>
Co-authored-by: Reinier van der Leer <pwuts@agpt.co>
339 lines
8.6 KiB
YAML
339 lines
8.6 KiB
YAML
# Environment Variable Loading Order (first → last, later overrides earlier):
|
|
# 1. backend/.env.default - Default values for all settings
|
|
# 2. backend/.env - User's custom configuration (if exists)
|
|
# 3. environment key - Docker-specific overrides defined below
|
|
# 4. Shell environment - Variables exported before running docker compose
|
|
# 5. CLI arguments - docker compose run -e VAR=value
|
|
|
|
# Common backend environment - Docker service names
|
|
x-backend-env: &backend-env # Docker internal service hostnames (override localhost defaults)
|
|
PYRO_HOST: "0.0.0.0"
|
|
AGENTSERVER_HOST: rest_server
|
|
SCHEDULER_HOST: scheduler_server
|
|
DATABASEMANAGER_HOST: database_manager
|
|
EXECUTIONMANAGER_HOST: executor
|
|
NOTIFICATIONMANAGER_HOST: notification_server
|
|
CLAMAV_SERVICE_HOST: clamav
|
|
DB_HOST: db
|
|
REDIS_HOST: redis
|
|
RABBITMQ_HOST: rabbitmq
|
|
# Override Supabase URL for Docker network
|
|
SUPABASE_URL: http://kong:8000
|
|
# Database connection string for Docker network
|
|
# This cannot be constructed like in .env because we cannot interpolate values set here (DB_HOST)
|
|
DATABASE_URL: postgresql://postgres:your-super-secret-and-long-postgres-password@db:5432/postgres?connect_timeout=60&schema=platform
|
|
DIRECT_URL: postgresql://postgres:your-super-secret-and-long-postgres-password@db:5432/postgres?connect_timeout=60&schema=platform
|
|
|
|
# Common env_file configuration for backend services
|
|
x-backend-env-files: &backend-env-files
|
|
env_file:
|
|
- backend/.env.default # Base defaults (always exists)
|
|
- path: backend/.env # User overrides (optional)
|
|
required: false
|
|
|
|
services:
|
|
migrate:
|
|
build:
|
|
context: ../
|
|
dockerfile: autogpt_platform/backend/Dockerfile
|
|
target: migrate
|
|
command: ["sh", "-c", "poetry run prisma generate && poetry run prisma migrate deploy"]
|
|
develop:
|
|
watch:
|
|
- path: ./
|
|
target: autogpt_platform/backend/migrations
|
|
action: rebuild
|
|
depends_on:
|
|
db:
|
|
condition: service_healthy
|
|
<<: *backend-env-files
|
|
environment:
|
|
<<: *backend-env
|
|
networks:
|
|
- app-network
|
|
restart: on-failure
|
|
healthcheck:
|
|
test:
|
|
[
|
|
"CMD-SHELL",
|
|
"poetry run prisma migrate status | grep -q 'No pending migrations' || exit 1",
|
|
]
|
|
interval: 30s
|
|
timeout: 10s
|
|
retries: 3
|
|
start_period: 5s
|
|
|
|
redis:
|
|
image: redis:latest
|
|
ports:
|
|
- "6379:6379"
|
|
networks:
|
|
- app-network
|
|
healthcheck:
|
|
test: ["CMD", "redis-cli", "ping"]
|
|
interval: 10s
|
|
timeout: 5s
|
|
retries: 5
|
|
rabbitmq:
|
|
image: rabbitmq:management
|
|
container_name: rabbitmq
|
|
healthcheck:
|
|
test: rabbitmq-diagnostics -q ping
|
|
interval: 30s
|
|
timeout: 10s
|
|
retries: 5
|
|
start_period: 10s
|
|
<<: *backend-env-files
|
|
environment:
|
|
<<: *backend-env
|
|
ports:
|
|
- "5672:5672"
|
|
- "15672:15672"
|
|
|
|
rest_server:
|
|
build:
|
|
context: ../
|
|
dockerfile: autogpt_platform/backend/Dockerfile
|
|
target: server
|
|
command: ["python", "-m", "backend.rest"]
|
|
develop:
|
|
watch:
|
|
- path: ./
|
|
target: autogpt_platform/backend/
|
|
action: rebuild
|
|
depends_on:
|
|
redis:
|
|
condition: service_healthy
|
|
db:
|
|
condition: service_healthy
|
|
migrate:
|
|
condition: service_completed_successfully
|
|
rabbitmq:
|
|
condition: service_healthy
|
|
<<: *backend-env-files
|
|
environment:
|
|
<<: *backend-env
|
|
ports:
|
|
- "8006:8006"
|
|
networks:
|
|
- app-network
|
|
logging:
|
|
driver: json-file
|
|
options:
|
|
max-size: "10m"
|
|
max-file: "3"
|
|
|
|
executor:
|
|
build:
|
|
context: ../
|
|
dockerfile: autogpt_platform/backend/Dockerfile
|
|
target: server
|
|
command: ["python", "-m", "backend.exec"]
|
|
develop:
|
|
watch:
|
|
- path: ./
|
|
target: autogpt_platform/backend/
|
|
action: rebuild
|
|
depends_on:
|
|
redis:
|
|
condition: service_healthy
|
|
rabbitmq:
|
|
condition: service_healthy
|
|
db:
|
|
condition: service_healthy
|
|
migrate:
|
|
condition: service_completed_successfully
|
|
database_manager:
|
|
condition: service_started
|
|
<<: *backend-env-files
|
|
environment:
|
|
<<: *backend-env
|
|
ports:
|
|
- "8002:8002"
|
|
networks:
|
|
- app-network
|
|
logging:
|
|
driver: json-file
|
|
options:
|
|
max-size: "10m"
|
|
max-file: "3"
|
|
|
|
websocket_server:
|
|
build:
|
|
context: ../
|
|
dockerfile: autogpt_platform/backend/Dockerfile
|
|
target: server
|
|
command: ["python", "-m", "backend.ws"]
|
|
develop:
|
|
watch:
|
|
- path: ./
|
|
target: autogpt_platform/backend/
|
|
action: rebuild
|
|
depends_on:
|
|
db:
|
|
condition: service_healthy
|
|
redis:
|
|
condition: service_healthy
|
|
migrate:
|
|
condition: service_completed_successfully
|
|
database_manager:
|
|
condition: service_started
|
|
<<: *backend-env-files
|
|
environment:
|
|
<<: *backend-env
|
|
ports:
|
|
- "8001:8001"
|
|
networks:
|
|
- app-network
|
|
logging:
|
|
driver: json-file
|
|
options:
|
|
max-size: "10m"
|
|
max-file: "3"
|
|
|
|
database_manager:
|
|
build:
|
|
context: ../
|
|
dockerfile: autogpt_platform/backend/Dockerfile
|
|
target: server
|
|
command: ["python", "-m", "backend.db"]
|
|
develop:
|
|
watch:
|
|
- path: ./
|
|
target: autogpt_platform/backend/
|
|
action: rebuild
|
|
depends_on:
|
|
db:
|
|
condition: service_healthy
|
|
migrate:
|
|
condition: service_completed_successfully
|
|
<<: *backend-env-files
|
|
environment:
|
|
<<: *backend-env
|
|
ports:
|
|
- "8005:8005"
|
|
networks:
|
|
- app-network
|
|
logging:
|
|
driver: json-file
|
|
options:
|
|
max-size: "10m"
|
|
max-file: "3"
|
|
|
|
scheduler_server:
|
|
build:
|
|
context: ../
|
|
dockerfile: autogpt_platform/backend/Dockerfile
|
|
target: server
|
|
command: ["python", "-m", "backend.scheduler"]
|
|
develop:
|
|
watch:
|
|
- path: ./
|
|
target: autogpt_platform/backend/
|
|
action: rebuild
|
|
depends_on:
|
|
db:
|
|
condition: service_healthy
|
|
redis:
|
|
condition: service_healthy
|
|
rabbitmq:
|
|
condition: service_healthy
|
|
migrate:
|
|
condition: service_completed_successfully
|
|
database_manager:
|
|
condition: service_started
|
|
# healthcheck:
|
|
# test:
|
|
# [
|
|
# "CMD",
|
|
# "curl",
|
|
# "-f",
|
|
# "-X",
|
|
# "POST",
|
|
# "http://localhost:8003/health_check",
|
|
# ]
|
|
# interval: 10s
|
|
# timeout: 10s
|
|
# retries: 5
|
|
<<: *backend-env-files
|
|
environment:
|
|
<<: *backend-env
|
|
ports:
|
|
- "8003:8003"
|
|
networks:
|
|
- app-network
|
|
logging:
|
|
driver: json-file
|
|
options:
|
|
max-size: "10m"
|
|
max-file: "3"
|
|
|
|
notification_server:
|
|
build:
|
|
context: ../
|
|
dockerfile: autogpt_platform/backend/Dockerfile
|
|
target: server
|
|
command: ["python", "-m", "backend.notification"]
|
|
develop:
|
|
watch:
|
|
- path: ./
|
|
target: autogpt_platform/backend/
|
|
action: rebuild
|
|
depends_on:
|
|
db:
|
|
condition: service_healthy
|
|
rabbitmq:
|
|
condition: service_healthy
|
|
migrate:
|
|
condition: service_completed_successfully
|
|
database_manager:
|
|
condition: service_started
|
|
<<: *backend-env-files
|
|
environment:
|
|
<<: *backend-env
|
|
ports:
|
|
- "8007:8007"
|
|
networks:
|
|
- app-network
|
|
logging:
|
|
driver: json-file
|
|
options:
|
|
max-size: "10m"
|
|
max-file: "3"
|
|
frontend:
|
|
build:
|
|
context: ../
|
|
dockerfile: autogpt_platform/frontend/Dockerfile
|
|
target: prod
|
|
args:
|
|
NEXT_PUBLIC_PW_TEST: ${NEXT_PUBLIC_PW_TEST:-false}
|
|
depends_on:
|
|
db:
|
|
condition: service_healthy
|
|
migrate:
|
|
condition: service_completed_successfully
|
|
ports:
|
|
- "3000:3000"
|
|
networks:
|
|
- app-network
|
|
logging:
|
|
driver: json-file
|
|
options:
|
|
max-size: "10m"
|
|
max-file: "3"
|
|
# Load environment variables in order (later overrides earlier)
|
|
env_file:
|
|
- path: ./frontend/.env.default # Base defaults (always exists)
|
|
- path: ./frontend/.env # User overrides (optional)
|
|
required: false
|
|
environment:
|
|
# Server-side environment variables (Docker service names)
|
|
# These override the localhost URLs from env files when running in Docker
|
|
AUTH_CALLBACK_URL: http://rest_server:8006/auth/callback
|
|
SUPABASE_URL: http://kong:8000
|
|
AGPT_SERVER_URL: http://rest_server:8006/api
|
|
AGPT_WS_SERVER_URL: ws://websocket_server:8001/ws
|
|
networks:
|
|
app-network:
|
|
driver: bridge
|