Compare commits

...

7 Commits

Author SHA1 Message Date
Bently
5c01eb4fc8 Merge branch 'dev' into docs/deployment-env-variables 2026-02-23 16:43:20 +00:00
Bently
ef42b17e3b docs: add Podman compatibility warning (#12120)
## Summary
Adds a warning to the Getting Started docs clarifying that **Podman and
podman-compose are not supported**.

## Problem
Users on Windows using `podman-compose` instead of Docker get errors
like:
```
Error: the specified Containerfile or Dockerfile does not exist, ..\..\autogpt_platform\backend\Dockerfile
```

This is because Podman handles relative paths differently than Docker,
causing incorrect path resolution on Windows.

## Solution
- Added a clear warning section after the Windows WSL 2 notes
- Explains the error users might see
- Directs them to install Docker Desktop instead

Closes #11358

<!-- greptile_comment -->

<details><summary><h3>Greptile Summary</h3></summary>

Adds a "Podman Not Supported" warning section to the Getting Started
documentation, placed after the Windows/WSL 2 installation notes. The
section clarifies that Docker is required, shows the typical error
message users encounter when using Podman, and directs them to install
Docker Desktop instead. This addresses issue #11358 where Windows users
using `podman-compose` hit path resolution errors.

- Adds `### ⚠️ Podman Not Supported` section under Manual Setup, after
Windows Installation Note
- Includes the specific error message users see with Podman for easy
identification
- Links to Docker Desktop installation docs as the recommended solution
- Formatting is consistent with existing sections in the document (emoji
headings, code blocks for errors)
</details>


<details><summary><h3>Confidence Score: 5/5</h3></summary>

- This PR is safe to merge — it only adds a documentation warning
section with no code changes.
- The change is a small, well-written documentation addition that adds a
Podman compatibility warning. It touches only one markdown file,
introduces no code changes, and is consistent with the existing document
structure and style. No issues were found.
- No files require special attention.
</details>


<details><summary><h3>Flowchart</h3></summary>

```mermaid
flowchart TD
    A[User wants to run AutoGPT] --> B{Which container runtime?}
    B -->|Docker / Docker Desktop| C[docker compose up -d --build]
    C --> D[AutoGPT starts successfully]
    B -->|Podman / podman-compose| E[podman-compose up -d --build]
    E --> F[Error: Containerfile or Dockerfile does not exist]
    F --> G[New warning section directs user to install Docker Desktop]
    G --> C
```
</details>


<sub>Last reviewed commit: 23ea6bd</sub>

<!-- greptile_other_comments_section -->

<!-- /greptile_comment -->
2026-02-23 15:19:24 +00:00
Ubbe
a18ffd0b21 fix(frontend/copilot): always-visible credentials, inputs, and login prompts (#12194)
Credentials, inputs, and login prompts in copilot tool outputs were
hidden inside collapsible accordions — users could accidentally collapse
them, hiding blocking actionable UI. This PR extracts all blocking
requirements out of accordions so they're always visible.

### Changes 🏗️

- **RunAgent & RunBlock**: Extract `SetupRequirementsCard` (credentials
picker) out of `ToolAccordion` — renders standalone, always visible
- **RunAgent**: Also extract `AgentDetailsCard` (inputs needed) and
`need_login` message out of accordion
- **SetupRequirementsCard (RunBlock)**: Input form always visible
(removed toggle button and animation), unified "Proceed" button disabled
until credentials + inputs are satisfied
- **SetupRequirementsCard (RunAgent)**: "Proceed" button disabled until
all credentials are selected
- **Both cards**: Added titled box with border for credentials section
("Block credentials" / "Agent credentials"), matching the existing
inputs box pattern
- **CredentialsFlatView**: "Add" button uses `variant="primary"` when
user has no credentials (was `secondary`)
- **Styleguide**: Added mock `CredentialsProvidersContext` with two
scenarios:
  - No credentials → shows "add new" flow
  - Has credentials → shows selection list with existing accounts
- **CreateAgent & EditAgent**: Picked up user-initiated styling
refinements

### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
  - [x] `pnpm format && pnpm lint && pnpm types` all pass
  - [ ] Visit `/copilot/styleguide` and verify:
- [ ] "Setup requirements — no credentials" shows add-credential button
(primary variant)
- [ ] "Setup requirements — has credentials" shows credential selection
dropdown
- [ ] Both RunAgent and RunBlock setup requirements render outside
accordion
- [ ] Trigger a copilot agent run that requires credentials — credential
picker always visible
- [ ] Trigger a copilot block run that requires credentials + inputs —
both sections visible, "Proceed" disabled until ready
- [ ] Trigger a copilot agent run that returns "agent details" — card
renders outside accordion
- [ ] Verify other output types (execution_started, error) still render
inside accordions


🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 16:39:21 +07:00
Otto
e40c8c70ce fix(copilot): collision detection, session locking, and sync for concurrent message saves (#12177)
Requested by @majdyz

Concurrent writers (incremental streaming saves from PR #12173 and
long-running tool callbacks) can race to persist messages with the same
`(sessionId, sequence)` pair, causing unique constraint violations on
`ChatMessage`.

**Root cause:** The streaming loop tracks `saved_msg_count` in-memory,
but the long-running tool callback (`_build_long_running_callback`) also
appends messages and calls `upsert_chat_session` independently — without
coordinating sequence numbers. When the streaming loop does its next
incremental save with the stale `saved_msg_count`, it tries to insert at
a sequence that already exists.

**Fix:** Multi-layered defense-in-depth approach:

1. **Collision detection with retry** (db.py): `add_chat_messages_batch`
uses `create_many()` in a transaction. On `UniqueViolationError`,
queries `MAX(sequence)+1` from DB and retries with the correct offset
(max 5 attempts).

2. **Robust sequence tracking** (db.py): `get_next_sequence()` uses
indexed `find_first` with `order={"sequence": "desc"}` for O(1) MAX
lookup, immune to deleted messages.

3. **Session-based counter** (model.py): Added `saved_message_count`
field to `ChatSession`. `upsert_chat_session` returns the session with
updated count, eliminating tuple returns throughout the codebase.

4. **MessageCounter dataclass** (sdk/service.py): Replaced list[int]
mutable reference pattern with a clean `MessageCounter` dataclass for
shared state between streaming loop and long-running callbacks.

5. **Session locking** (sdk/service.py): Prevent concurrent streams on
the same session using Redis `SET NX EX` distributed locks with TTL
refresh on heartbeats (config.stream_ttl = 3600s).

6. **Atomic operations** (db.py): Single timestamp for all messages and
session update in batch operations for consistency. Parallel queries
with `asyncio.gather` for lower latency.

7. **Config-based TTL** (sdk/service.py, config.py): Consolidated all
TTL constants to use `config.stream_ttl` (3600s) with lock refresh on
heartbeats.

### Key implementation details

- **create_many**: Uses `sessionId` directly (not nested
`Session.connect`) as `create_many` doesn't support nested creates
- **Type narrowing**: Added explicit `assert session is not None`
statements for pyright type checking in async contexts
- **Parallel operations**: Use `asyncio.gather` for independent DB
operations (create_many + session update)
- **Single timestamp**: All messages in a batch share the same
`createdAt` timestamp for atomicity

### Changes
- `backend/copilot/db.py`: Collision detection with `create_many` +
retry, indexed sequence lookup, single timestamp, parallel queries
- `backend/copilot/model.py`: Added `saved_message_count` field,
simplified return types
- `backend/copilot/sdk/service.py`: MessageCounter dataclass, session
locking with refresh, config-based TTL, type narrowing
- `backend/copilot/service.py`: Updated all callers to handle new return
types
- `backend/copilot/config.py`: Increased long_running_operation_ttl to
3600s with clarified docstring
- `backend/copilot/*_test.py`: Tests updated for new signatures

---------

Co-authored-by: Zamil Majdy <zamil.majdy@agpt.co>
2026-02-20 15:05:03 +00:00
Bently
2d7431bde6 Merge branch 'dev' into docs/deployment-env-variables 2026-02-19 17:49:09 +00:00
Bentlybro
e934df3c0c fix: address code review feedback
- Add 'text' language identifier to code blocks (MD040)
- Add VAULT_ENC_KEY generation command (openssl rand -hex 16)
- Fix DB_HOST default to 'localhost' (not 'db')
- Add info box clarifying port numbers are internal Docker ports
- Update OAuth callback URL to not include port by default
- Clarify Docker service names are internal container DNS
2026-02-16 12:10:09 +00:00
Bentlybro
8d557d33e1 docs: add deployment environment variables guide
Closes #10961, Closes OPEN-2715

Documents all environment variables that must be configured when deploying
AutoGPT to a new server:

- Quick reference table of critical URLs that must change
- Configuration file locations and loading order
- Security keys that must be regenerated (with generation commands)
- Database, Redis, RabbitMQ configuration
- Default ports for all services
- OAuth callback URLs for all supported providers
- Full deployment checklist
- Docker vs external services guidance
2026-02-16 11:59:34 +00:00
23 changed files with 1208 additions and 371 deletions

View File

@@ -38,8 +38,10 @@ class ChatConfig(BaseSettings):
# Long-running operation configuration
long_running_operation_ttl: int = Field(
default=600,
description="TTL in seconds for long-running operation tracking in Redis (safety net if pod dies)",
default=3600,
description="TTL in seconds for long-running operation deduplication lock "
"(1 hour, matches stream_ttl). Prevents duplicate operations if pod dies. "
"For longer operations, the stream_registry heartbeat keeps them alive.",
)
# Stream registry configuration for SSE reconnection
@@ -47,6 +49,11 @@ class ChatConfig(BaseSettings):
default=3600,
description="TTL in seconds for stream data in Redis (1 hour)",
)
stream_lock_ttl: int = Field(
default=120,
description="TTL in seconds for stream lock (2 minutes). Short timeout allows "
"reconnection after refresh/crash without long waits.",
)
stream_max_length: int = Field(
default=10000,
description="Maximum number of messages to store per stream",

View File

@@ -3,8 +3,9 @@
import asyncio
import logging
from datetime import UTC, datetime
from typing import Any, cast
from typing import Any
from prisma.errors import UniqueViolationError
from prisma.models import ChatMessage as PrismaChatMessage
from prisma.models import ChatSession as PrismaChatSession
from prisma.types import (
@@ -92,10 +93,9 @@ async def add_chat_message(
function_call: dict[str, Any] | None = None,
) -> ChatMessage:
"""Add a message to a chat session."""
# Build input dict dynamically rather than using ChatMessageCreateInput directly
# because Prisma's TypedDict validation rejects optional fields set to None.
# We only include fields that have values, then cast at the end.
data: dict[str, Any] = {
# Build ChatMessageCreateInput with only non-None values
# (Prisma TypedDict rejects optional fields set to None)
data: ChatMessageCreateInput = {
"Session": {"connect": {"id": session_id}},
"role": role,
"sequence": sequence,
@@ -123,7 +123,7 @@ async def add_chat_message(
where={"id": session_id},
data={"updatedAt": datetime.now(UTC)},
),
PrismaChatMessage.prisma().create(data=cast(ChatMessageCreateInput, data)),
PrismaChatMessage.prisma().create(data=data),
)
return ChatMessage.from_db(message)
@@ -132,58 +132,93 @@ async def add_chat_messages_batch(
session_id: str,
messages: list[dict[str, Any]],
start_sequence: int,
) -> list[ChatMessage]:
) -> int:
"""Add multiple messages to a chat session in a batch.
Uses a transaction for atomicity - if any message creation fails,
the entire batch is rolled back.
Uses collision detection with retry: tries to create messages starting
at start_sequence. If a unique constraint violation occurs (e.g., the
streaming loop and long-running callback race), queries the latest
sequence and retries with the correct offset. This avoids unnecessary
upserts and DB queries in the common case (no collision).
Returns:
Next sequence number for the next message to be inserted. This equals
start_sequence + len(messages) and allows callers to update their
counters even when collision detection adjusts start_sequence.
"""
if not messages:
return []
# No messages to add - return current count
return start_sequence
created_messages = []
max_retries = 5
for attempt in range(max_retries):
try:
# Single timestamp for all messages and session update
now = datetime.now(UTC)
async with db.transaction() as tx:
for i, msg in enumerate(messages):
# Build input dict dynamically rather than using ChatMessageCreateInput
# directly because Prisma's TypedDict validation rejects optional fields
# set to None. We only include fields that have values, then cast.
data: dict[str, Any] = {
"Session": {"connect": {"id": session_id}},
"role": msg["role"],
"sequence": start_sequence + i,
}
async with db.transaction() as tx:
# Build all message data
messages_data = []
for i, msg in enumerate(messages):
# Build ChatMessageCreateInput with only non-None values
# (Prisma TypedDict rejects optional fields set to None)
# Note: create_many doesn't support nested creates, use sessionId directly
data: ChatMessageCreateInput = {
"sessionId": session_id,
"role": msg["role"],
"sequence": start_sequence + i,
"createdAt": now,
}
# Add optional string fields
if msg.get("content") is not None:
data["content"] = msg["content"]
if msg.get("name") is not None:
data["name"] = msg["name"]
if msg.get("tool_call_id") is not None:
data["toolCallId"] = msg["tool_call_id"]
if msg.get("refusal") is not None:
data["refusal"] = msg["refusal"]
# Add optional string fields
if msg.get("content") is not None:
data["content"] = msg["content"]
if msg.get("name") is not None:
data["name"] = msg["name"]
if msg.get("tool_call_id") is not None:
data["toolCallId"] = msg["tool_call_id"]
if msg.get("refusal") is not None:
data["refusal"] = msg["refusal"]
# Add optional JSON fields only when they have values
if msg.get("tool_calls") is not None:
data["toolCalls"] = SafeJson(msg["tool_calls"])
if msg.get("function_call") is not None:
data["functionCall"] = SafeJson(msg["function_call"])
# Add optional JSON fields only when they have values
if msg.get("tool_calls") is not None:
data["toolCalls"] = SafeJson(msg["tool_calls"])
if msg.get("function_call") is not None:
data["functionCall"] = SafeJson(msg["function_call"])
created = await PrismaChatMessage.prisma(tx).create(
data=cast(ChatMessageCreateInput, data)
)
created_messages.append(created)
messages_data.append(data)
# Update session's updatedAt timestamp within the same transaction.
# Note: Token usage (total_prompt_tokens, total_completion_tokens) is updated
# separately via update_chat_session() after streaming completes.
await PrismaChatSession.prisma(tx).update(
where={"id": session_id},
data={"updatedAt": datetime.now(UTC)},
)
# Run create_many and session update in parallel within transaction
# Both use the same timestamp for consistency
await asyncio.gather(
PrismaChatMessage.prisma(tx).create_many(data=messages_data),
PrismaChatSession.prisma(tx).update(
where={"id": session_id},
data={"updatedAt": now},
),
)
return [ChatMessage.from_db(m) for m in created_messages]
# Return next sequence number for counter sync
return start_sequence + len(messages)
except UniqueViolationError:
if attempt < max_retries - 1:
# Collision detected - query MAX(sequence)+1 and retry with correct offset
logger.info(
f"Collision detected for session {session_id} at sequence "
f"{start_sequence}, querying DB for latest sequence"
)
start_sequence = await get_next_sequence(session_id)
logger.info(
f"Retrying batch insert with start_sequence={start_sequence}"
)
continue
else:
# Max retries exceeded - propagate error
raise
# Should never reach here due to raise in exception handler
raise RuntimeError(f"Failed to insert messages after {max_retries} attempts")
async def get_user_chat_sessions(
@@ -237,10 +272,20 @@ async def delete_chat_session(session_id: str, user_id: str | None = None) -> bo
return False
async def get_chat_session_message_count(session_id: str) -> int:
"""Get the number of messages in a chat session."""
count = await PrismaChatMessage.prisma().count(where={"sessionId": session_id})
return count
async def get_next_sequence(session_id: str) -> int:
"""Get the next sequence number for a new message in this session.
Uses MAX(sequence) + 1 for robustness. Returns 0 if no messages exist.
More robust than COUNT(*) because it's immune to deleted messages.
Optimized to select only the sequence column using raw SQL.
The unique index on (sessionId, sequence) makes this query fast.
"""
results = await db.query_raw_with_schema(
'SELECT "sequence" FROM {schema_prefix}"ChatMessage" WHERE "sessionId" = $1 ORDER BY "sequence" DESC LIMIT 1',
session_id,
)
return 0 if not results else results[0]["sequence"] + 1
async def update_tool_message_content(

View File

@@ -434,8 +434,6 @@ async def _get_session_from_db(session_id: str) -> ChatSession | None:
async def upsert_chat_session(
session: ChatSession,
*,
existing_message_count: int | None = None,
) -> ChatSession:
"""Update a chat session in both cache and database.
@@ -443,12 +441,6 @@ async def upsert_chat_session(
operations (e.g., background title update and main stream handler)
attempt to upsert the same session simultaneously.
Args:
existing_message_count: If provided, skip the DB query to count
existing messages. The caller is responsible for tracking this
accurately. Useful for incremental saves in a streaming loop
where the caller already knows how many messages are persisted.
Raises:
DatabaseError: If the database write fails. The cache is still updated
as a best-effort optimization, but the error is propagated to ensure
@@ -459,11 +451,8 @@ async def upsert_chat_session(
lock = await _get_session_lock(session.session_id)
async with lock:
# Get existing message count from DB for incremental saves
if existing_message_count is None:
existing_message_count = await chat_db().get_chat_session_message_count(
session.session_id
)
# Always query DB for existing message count to ensure consistency
existing_message_count = await chat_db().get_next_sequence(session.session_id)
db_error: Exception | None = None
@@ -587,9 +576,7 @@ async def append_and_save_message(session_id: str, message: ChatMessage) -> Chat
raise ValueError(f"Session {session_id} not found")
session.messages.append(message)
existing_message_count = await chat_db().get_chat_session_message_count(
session_id
)
existing_message_count = await chat_db().get_next_sequence(session_id)
try:
await _save_session_to_db(session, existing_message_count)

View File

@@ -331,3 +331,96 @@ def test_to_openai_messages_merges_split_assistants():
tc_list = merged.get("tool_calls")
assert tc_list is not None and len(list(tc_list)) == 1
assert list(tc_list)[0]["id"] == "tc1"
# --------------------------------------------------------------------------- #
# Concurrent save collision detection #
# --------------------------------------------------------------------------- #
@pytest.mark.asyncio(loop_scope="session")
async def test_concurrent_saves_collision_detection(setup_test_user, test_user_id):
"""Test that concurrent saves from streaming loop and callback handle collisions correctly.
Simulates the race condition where:
1. Streaming loop starts with saved_msg_count=5
2. Long-running callback appends message #5 and saves
3. Streaming loop tries to save with stale count=5
The collision detection should handle this gracefully.
"""
import asyncio
# Create a session with initial messages
session = ChatSession.new(user_id=test_user_id)
for i in range(3):
session.messages.append(
ChatMessage(
role="user" if i % 2 == 0 else "assistant", content=f"Message {i}"
)
)
# Save initial messages
session = await upsert_chat_session(session)
# Simulate streaming loop and callback saving concurrently
async def streaming_loop_save():
"""Simulates streaming loop saving messages."""
# Add 2 messages
session.messages.append(ChatMessage(role="user", content="Streaming message 1"))
session.messages.append(
ChatMessage(role="assistant", content="Streaming message 2")
)
# Wait a bit to let callback potentially save first
await asyncio.sleep(0.01)
# Save (will query DB for existing count)
return await upsert_chat_session(session)
async def callback_save():
"""Simulates long-running callback saving a message."""
# Add 1 message
session.messages.append(
ChatMessage(role="tool", content="Callback result", tool_call_id="tc1")
)
# Save immediately (will query DB for existing count)
return await upsert_chat_session(session)
# Run both saves concurrently - one will hit collision detection
results = await asyncio.gather(streaming_loop_save(), callback_save())
# Both should succeed
assert all(r is not None for r in results)
# Reload session from DB to verify
from backend.data.redis_client import get_redis_async
redis_key = f"chat:session:{session.session_id}"
async_redis = await get_redis_async()
await async_redis.delete(redis_key) # Clear cache to force DB load
loaded_session = await get_chat_session(session.session_id, test_user_id)
assert loaded_session is not None
# Should have all 6 messages (3 initial + 2 streaming + 1 callback)
assert len(loaded_session.messages) == 6
# Verify no duplicate sequences
sequences = []
for i, msg in enumerate(loaded_session.messages):
# Messages should have sequential sequence numbers starting from 0
sequences.append(i)
# All sequences should be unique and sequential
assert sequences == list(range(6))
# Verify message content is preserved
contents = [m.content for m in loaded_session.messages]
assert "Message 0" in contents
assert "Message 1" in contents
assert "Message 2" in contents
assert "Streaming message 1" in contents
assert "Streaming message 2" in contents
assert "Callback result" in contents

View File

@@ -7,8 +7,10 @@ import os
import uuid
from collections.abc import AsyncGenerator
from dataclasses import dataclass
from typing import Any
from typing import Any, cast
from backend.data.redis_client import get_redis_async
from backend.executor.cluster_lock import AsyncClusterLock
from backend.util.exceptions import NotFoundError
from .. import stream_registry
@@ -61,6 +63,7 @@ from .transcript import (
logger = logging.getLogger(__name__)
config = ChatConfig()
# Set to hold background tasks to prevent garbage collection
_background_tasks: set[asyncio.Task[Any]] = set()
@@ -132,8 +135,12 @@ is delivered to the user via a background stream.
All tasks must run in the foreground.
"""
STREAM_LOCK_PREFIX = "copilot:stream:lock:"
def _build_long_running_callback(user_id: str | None) -> LongRunningCallback:
def _build_long_running_callback(
user_id: str | None,
) -> LongRunningCallback:
"""Build a callback that delegates long-running tools to the non-SDK infrastructure.
Long-running tools (create_agent, edit_agent, etc.) are delegated to the
@@ -142,6 +149,9 @@ def _build_long_running_callback(user_id: str | None) -> LongRunningCallback:
page refreshes / pod restarts, and the frontend shows the proper loading
widget with progress updates.
Args:
user_id: User ID for the session
The returned callback matches the ``LongRunningCallback`` signature:
``(tool_name, args, session) -> MCP response dict``.
"""
@@ -207,7 +217,8 @@ def _build_long_running_callback(user_id: str | None) -> LongRunningCallback:
tool_call_id=tool_call_id,
)
session.messages.append(pending_message)
await upsert_chat_session(session)
# Collision detection happens in add_chat_messages_batch (db.py)
session = await upsert_chat_session(session)
# --- Spawn background task (reuses non-SDK infrastructure) ---
bg_task = asyncio.create_task(
@@ -527,6 +538,9 @@ async def stream_chat_completion_sdk(
f"Session {session_id} not found. Please create a new session first."
)
# Type narrowing: session is guaranteed ChatSession after the check above
session = cast(ChatSession, session)
# Append the new message to the session if it's not already there
new_message_role = "user" if is_user_message else "assistant"
if message and (
@@ -564,6 +578,29 @@ async def stream_chat_completion_sdk(
system_prompt += _SDK_TOOL_SUPPLEMENT
message_id = str(uuid.uuid4())
task_id = str(uuid.uuid4())
stream_id = task_id # Use task_id as unique stream identifier
# Acquire stream lock to prevent concurrent streams to the same session
lock = AsyncClusterLock(
redis=await get_redis_async(),
key=f"{STREAM_LOCK_PREFIX}{session_id}",
owner_id=stream_id,
timeout=config.stream_lock_ttl,
)
lock_owner = await lock.try_acquire()
if lock_owner != stream_id:
# Another stream is active
logger.warning(
f"[SDK] Session {session_id} already has an active stream: {lock_owner}"
)
yield StreamError(
errorText="Another stream is already active for this session. "
"Please wait or stop it.",
code="stream_already_active",
)
yield StreamFinish()
return
yield StreamStart(messageId=message_id, taskId=task_id)
@@ -715,9 +752,6 @@ async def stream_chat_completion_sdk(
accumulated_tool_calls: list[dict[str, Any]] = []
has_appended_assistant = False
has_tool_results = False
# Track persisted message count to skip DB count queries
# on incremental saves. Initial save happened at line 545.
saved_msg_count = len(session.messages)
# Use an explicit async iterator with non-cancelling heartbeats.
# CRITICAL: we must NOT cancel __anext__() mid-flight — doing so
@@ -744,6 +778,8 @@ async def stream_chat_completion_sdk(
if not done:
# Timeout — emit heartbeat but keep the task alive
# Also refresh lock TTL to keep it alive
await lock.refresh()
yield StreamHeartbeat()
continue
@@ -893,13 +929,10 @@ async def stream_chat_completion_sdk(
has_appended_assistant = True
# Save before tool execution starts so the
# pending tool call is visible on refresh /
# other devices.
# other devices. Collision detection happens
# in add_chat_messages_batch (db.py).
try:
await upsert_chat_session(
session,
existing_message_count=saved_msg_count,
)
saved_msg_count = len(session.messages)
session = await upsert_chat_session(session)
except Exception as save_err:
logger.warning(
"[SDK] [%s] Incremental save " "failed: %s",
@@ -922,12 +955,9 @@ async def stream_chat_completion_sdk(
has_tool_results = True
# Save after tool completes so the result is
# visible on refresh / other devices.
# Collision detection happens in add_chat_messages_batch (db.py).
try:
await upsert_chat_session(
session,
existing_message_count=saved_msg_count,
)
saved_msg_count = len(session.messages)
session = await upsert_chat_session(session)
except Exception as save_err:
logger.warning(
"[SDK] [%s] Incremental save " "failed: %s",
@@ -1059,7 +1089,7 @@ async def stream_chat_completion_sdk(
"to use the OpenAI-compatible fallback."
)
await asyncio.shield(upsert_chat_session(session))
session = cast(ChatSession, await asyncio.shield(upsert_chat_session(session)))
logger.info(
"[SDK] [%s] Session saved with %d messages",
session_id[:12],
@@ -1076,10 +1106,11 @@ async def stream_chat_completion_sdk(
raise
except Exception as e:
logger.error(f"[SDK] Error: {e}", exc_info=True)
try:
await asyncio.shield(upsert_chat_session(session))
except Exception as save_err:
logger.error(f"[SDK] Failed to save session on error: {save_err}")
if session:
try:
await asyncio.shield(upsert_chat_session(session))
except Exception as save_err:
logger.error(f"[SDK] Failed to save session on error: {save_err}")
yield StreamError(
errorText="An error occurred. Please try again.",
code="sdk_error",
@@ -1101,7 +1132,7 @@ async def stream_chat_completion_sdk(
if not raw_transcript and use_resume and resume_file:
raw_transcript = read_transcript_file(resume_file)
if raw_transcript:
if raw_transcript and session is not None:
await asyncio.shield(
_try_upload_transcript(
user_id,
@@ -1121,6 +1152,9 @@ async def stream_chat_completion_sdk(
if sdk_cwd:
_cleanup_sdk_tool_results(sdk_cwd)
# Release stream lock to allow new streams for this session
await lock.release()
async def _try_upload_transcript(
user_id: str,

View File

@@ -352,7 +352,8 @@ async def assign_user_to_session(
if not session:
raise NotFoundError(f"Session {session_id} not found")
session.user_id = user_id
return await upsert_chat_session(session)
session = await upsert_chat_session(session)
return session
async def stream_chat_completion(

View File

@@ -303,7 +303,7 @@ class DatabaseManager(AppService):
get_user_chat_sessions = _(chat_db.get_user_chat_sessions)
get_user_session_count = _(chat_db.get_user_session_count)
delete_chat_session = _(chat_db.delete_chat_session)
get_chat_session_message_count = _(chat_db.get_chat_session_message_count)
get_next_sequence = _(chat_db.get_next_sequence)
update_tool_message_content = _(chat_db.update_tool_message_content)
@@ -473,5 +473,5 @@ class DatabaseManagerAsyncClient(AppServiceClient):
get_user_chat_sessions = d.get_user_chat_sessions
get_user_session_count = d.get_user_session_count
delete_chat_session = d.delete_chat_session
get_chat_session_message_count = d.get_chat_session_message_count
get_next_sequence = d.get_next_sequence
update_tool_message_content = d.update_tool_message_content

View File

@@ -1,5 +1,6 @@
"""Redis-based distributed locking for cluster coordination."""
import asyncio
import logging
import threading
import time
@@ -7,6 +8,7 @@ from typing import TYPE_CHECKING
if TYPE_CHECKING:
from redis import Redis
from redis.asyncio import Redis as AsyncRedis
logger = logging.getLogger(__name__)
@@ -126,3 +128,124 @@ class ClusterLock:
with self._refresh_lock:
self._last_refresh = 0.0
class AsyncClusterLock:
"""Async Redis-based distributed lock for preventing duplicate execution."""
def __init__(
self, redis: "AsyncRedis", key: str, owner_id: str, timeout: int = 300
):
self.redis = redis
self.key = key
self.owner_id = owner_id
self.timeout = timeout
self._last_refresh = 0.0
self._refresh_lock = asyncio.Lock()
async def try_acquire(self) -> str | None:
"""Try to acquire the lock.
Returns:
- owner_id (self.owner_id) if successfully acquired
- different owner_id if someone else holds the lock
- None if Redis is unavailable or other error
"""
try:
success = await self.redis.set(
self.key, self.owner_id, nx=True, ex=self.timeout
)
if success:
async with self._refresh_lock:
self._last_refresh = time.time()
return self.owner_id # Successfully acquired
# Failed to acquire, get current owner
current_value = await self.redis.get(self.key)
if current_value:
current_owner = (
current_value.decode("utf-8")
if isinstance(current_value, bytes)
else str(current_value)
)
return current_owner
# Key doesn't exist but we failed to set it - race condition or Redis issue
return None
except Exception as e:
logger.error(f"AsyncClusterLock.try_acquire failed for key {self.key}: {e}")
return None
async def refresh(self) -> bool:
"""Refresh lock TTL if we still own it.
Rate limited to at most once every timeout/10 seconds (minimum 1 second).
During rate limiting, still verifies lock existence but skips TTL extension.
Setting _last_refresh to 0 bypasses rate limiting for testing.
Async-safe: uses asyncio.Lock to protect _last_refresh access.
"""
# Calculate refresh interval: max(timeout // 10, 1)
refresh_interval = max(self.timeout // 10, 1)
current_time = time.time()
# Check if we're within the rate limit period (async-safe read)
# _last_refresh == 0 forces a refresh (bypasses rate limiting for testing)
async with self._refresh_lock:
last_refresh = self._last_refresh
is_rate_limited = (
last_refresh > 0 and (current_time - last_refresh) < refresh_interval
)
try:
# Always verify lock existence, even during rate limiting
current_value = await self.redis.get(self.key)
if not current_value:
async with self._refresh_lock:
self._last_refresh = 0
return False
stored_owner = (
current_value.decode("utf-8")
if isinstance(current_value, bytes)
else str(current_value)
)
if stored_owner != self.owner_id:
async with self._refresh_lock:
self._last_refresh = 0
return False
# If rate limited, return True but don't update TTL or timestamp
if is_rate_limited:
return True
# Perform actual refresh
if await self.redis.expire(self.key, self.timeout):
async with self._refresh_lock:
self._last_refresh = current_time
return True
async with self._refresh_lock:
self._last_refresh = 0
return False
except Exception as e:
logger.error(f"AsyncClusterLock.refresh failed for key {self.key}: {e}")
async with self._refresh_lock:
self._last_refresh = 0
return False
async def release(self):
"""Release the lock."""
async with self._refresh_lock:
if self._last_refresh == 0:
return
try:
await self.redis.delete(self.key)
except Exception:
pass
async with self._refresh_lock:
self._last_refresh = 0.0

View File

@@ -11,6 +11,11 @@ import {
MessageResponse,
} from "@/components/ai-elements/message";
import { Text } from "@/components/atoms/Text/Text";
import {
CredentialsProvidersContext,
type CredentialsProviderData,
type CredentialsProvidersContextType,
} from "@/providers/agent-credentials/credentials-provider";
import { CopilotChatActionsProvider } from "../components/CopilotChatActionsProvider/CopilotChatActionsProvider";
import { CreateAgentTool } from "../tools/CreateAgent/CreateAgent";
import { EditAgentTool } from "../tools/EditAgent/EditAgent";
@@ -97,6 +102,65 @@ function uid() {
return `sg-${++_id}`;
}
// ---------------------------------------------------------------------------
// Mock credential providers for setup-requirements demos
// ---------------------------------------------------------------------------
const noop = () => Promise.reject(new Error("Styleguide mock"));
function makeMockProvider(
provider: string,
providerName: string,
savedCredentials: CredentialsProviderData["savedCredentials"] = [],
): CredentialsProviderData {
return {
provider,
providerName,
savedCredentials,
isSystemProvider: false,
oAuthCallback: noop as CredentialsProviderData["oAuthCallback"],
mcpOAuthCallback: noop as CredentialsProviderData["mcpOAuthCallback"],
createAPIKeyCredentials:
noop as CredentialsProviderData["createAPIKeyCredentials"],
createUserPasswordCredentials:
noop as CredentialsProviderData["createUserPasswordCredentials"],
createHostScopedCredentials:
noop as CredentialsProviderData["createHostScopedCredentials"],
deleteCredentials: noop as CredentialsProviderData["deleteCredentials"],
};
}
/**
* Provider context where the user already has saved credentials
* so the credential picker shows a selection list.
*/
const MOCK_PROVIDERS_WITH_CREDENTIALS: CredentialsProvidersContextType = {
google: makeMockProvider("google", "Google", [
{
id: "cred-google-1",
provider: "google",
type: "oauth2",
title: "work@company.com",
scopes: ["email", "calendar"],
},
{
id: "cred-google-2",
provider: "google",
type: "oauth2",
title: "personal@gmail.com",
scopes: ["email", "calendar"],
},
]),
};
/**
* Provider context where the user has NO saved credentials,
* so the credential picker shows an "add new" flow.
*/
const MOCK_PROVIDERS_WITHOUT_CREDENTIALS: CredentialsProvidersContextType = {
openweathermap: makeMockProvider("openweathermap", "OpenWeatherMap"),
};
// ---------------------------------------------------------------------------
// Page
// ---------------------------------------------------------------------------
@@ -554,45 +618,80 @@ export default function StyleguidePage() {
/>
</SubSection>
<SubSection label="Output available (setup requirements)">
<RunBlockTool
part={{
type: "tool-run_block",
toolCallId: uid(),
state: "output-available",
input: { block_id: "weather-block-123" },
output: {
type: ResponseType.setup_requirements,
message:
"This block requires API credentials to run. Please configure them below.",
setup_info: {
agent_name: "Weather Agent",
requirements: {
inputs: [
{
name: "city",
title: "City",
type: "string",
required: true,
description: "The city to get weather for",
},
],
},
user_readiness: {
missing_credentials: {
openweathermap: {
provider: "openweathermap",
credentials_type: "api_key",
title: "OpenWeatherMap API Key",
description:
"Required to access weather data. Get your key at openweathermap.org",
<SubSection label="Setup requirements — no credentials (add new)">
<CredentialsProvidersContext.Provider
value={MOCK_PROVIDERS_WITHOUT_CREDENTIALS}
>
<RunBlockTool
part={{
type: "tool-run_block",
toolCallId: uid(),
state: "output-available",
input: { block_id: "weather-block-123" },
output: {
type: ResponseType.setup_requirements,
message:
"This block requires API credentials to run. Please configure them below.",
setup_info: {
agent_id: "agent-weather-1",
agent_name: "Weather Agent",
requirements: {
inputs: [
{
name: "city",
title: "City",
type: "string",
required: true,
description: "The city to get weather for",
},
],
},
user_readiness: {
missing_credentials: {
openweathermap_key: {
provider: "openweathermap",
types: ["api_key"],
},
},
},
},
},
},
}}
/>
}}
/>
</CredentialsProvidersContext.Provider>
</SubSection>
<SubSection label="Setup requirements — has credentials (pick from list)">
<CredentialsProvidersContext.Provider
value={MOCK_PROVIDERS_WITH_CREDENTIALS}
>
<RunBlockTool
part={{
type: "tool-run_block",
toolCallId: uid(),
state: "output-available",
input: { block_id: "calendar-block-456" },
output: {
type: ResponseType.setup_requirements,
message:
"This block requires Google credentials. Pick an account below or connect a new one.",
setup_info: {
agent_id: "agent-calendar-1",
agent_name: "Calendar Agent",
user_readiness: {
missing_credentials: {
google_oauth: {
provider: "google",
types: ["oauth2"],
scopes: ["email", "calendar"],
},
},
},
},
},
}}
/>
</CredentialsProvidersContext.Provider>
</SubSection>
<SubSection label="Output available (error)">
@@ -849,34 +948,71 @@ export default function StyleguidePage() {
/>
</SubSection>
<SubSection label="Output available (setup requirements)">
<RunAgentTool
part={{
type: "tool-run_agent",
toolCallId: uid(),
state: "output-available",
input: { username_agent_slug: "creator/my-agent" },
output: {
type: ResponseType.setup_requirements,
message: "This agent requires additional setup.",
setup_info: {
agent_name: "YouTube Summarizer",
requirements: {},
user_readiness: {
missing_credentials: {
youtube_api: {
provider: "youtube",
credentials_type: "api_key",
title: "YouTube Data API Key",
description:
"Required to access YouTube video data.",
<SubSection label="Setup requirements — no credentials (add new)">
<CredentialsProvidersContext.Provider
value={MOCK_PROVIDERS_WITHOUT_CREDENTIALS}
>
<RunAgentTool
part={{
type: "tool-run_agent",
toolCallId: uid(),
state: "output-available",
input: { username_agent_slug: "creator/weather-agent" },
output: {
type: ResponseType.setup_requirements,
message:
"This agent requires an API key. Add your credentials below.",
setup_info: {
agent_id: "agent-weather-1",
agent_name: "Weather Agent",
requirements: {},
user_readiness: {
missing_credentials: {
openweathermap_key: {
provider: "openweathermap",
types: ["api_key"],
},
},
},
},
},
},
}}
/>
}}
/>
</CredentialsProvidersContext.Provider>
</SubSection>
<SubSection label="Setup requirements — has credentials (pick from list)">
<CredentialsProvidersContext.Provider
value={MOCK_PROVIDERS_WITH_CREDENTIALS}
>
<RunAgentTool
part={{
type: "tool-run_agent",
toolCallId: uid(),
state: "output-available",
input: { username_agent_slug: "creator/calendar-agent" },
output: {
type: ResponseType.setup_requirements,
message:
"This agent needs Google credentials. Pick an account or connect a new one.",
setup_info: {
agent_id: "agent-calendar-1",
agent_name: "Google Calendar Agent",
requirements: {},
user_readiness: {
missing_credentials: {
google_oauth: {
provider: "google",
types: ["oauth2"],
scopes: ["email", "calendar"],
},
},
},
},
},
}}
/>
</CredentialsProvidersContext.Provider>
</SubSection>
<SubSection label="Output available (need login)">

View File

@@ -16,7 +16,6 @@ import {
ContentCardDescription,
ContentCodeBlock,
ContentGrid,
ContentHint,
ContentMessage,
} from "../../components/ToolAccordion/AccordionContent";
import { ToolAccordion } from "../../components/ToolAccordion/ToolAccordion";
@@ -24,8 +23,8 @@ import {
ClarificationQuestionsCard,
ClarifyingQuestion,
} from "./components/ClarificationQuestionsCard";
import sparklesImg from "./components/MiniGame/assets/sparkles.png";
import { MiniGame } from "./components/MiniGame/MiniGame";
import sparklesImg from "../../components/MiniGame/assets/sparkles.png";
import { MiniGame } from "../../components/MiniGame/MiniGame";
import { SuggestedGoalCard } from "./components/SuggestedGoalCard";
import {
AccordionIcon,
@@ -93,9 +92,7 @@ function getAccordionMeta(output: CreateAgentToolOutput) {
) {
return {
icon,
title:
"Creating agent, this may take a few minutes. Play while you wait.",
expanded: true,
title: output.message || "Agent creation started",
};
}
return {
@@ -169,15 +166,22 @@ export function CreateAgentTool({ part }: Props) {
/>
</div>
{isStreaming && (
<ToolAccordion
icon={<AccordionIcon />}
title="Creating agent, this may take a few minutes. Play while you wait."
expanded
>
<ContentGrid>
<MiniGame />
</ContentGrid>
</ToolAccordion>
)}
{hasExpandableContent && output && (
<ToolAccordion {...getAccordionMeta(output)}>
{isOperating && (
<ContentGrid>
<MiniGame />
<ContentHint>
This could take a few minutes play while you wait!
</ContentHint>
</ContentGrid>
{isOperating && output.message && (
<ContentMessage>{output.message}</ContentMessage>
)}
{isAgentSavedOutput(output) && (

View File

@@ -4,17 +4,15 @@ import { WarningDiamondIcon } from "@phosphor-icons/react";
import type { ToolUIPart } from "ai";
import { useCopilotChatActions } from "../../components/CopilotChatActionsProvider/useCopilotChatActions";
import { MorphingTextAnimation } from "../../components/MorphingTextAnimation/MorphingTextAnimation";
import { OrbitLoader } from "../../components/OrbitLoader/OrbitLoader";
import {
ContentCardDescription,
ContentCodeBlock,
ContentGrid,
ContentHint,
ContentLink,
ContentMessage,
} from "../../components/ToolAccordion/AccordionContent";
import { ToolAccordion } from "../../components/ToolAccordion/ToolAccordion";
import { MiniGame } from "../CreateAgent/components/MiniGame/MiniGame";
import { MiniGame } from "../../components/MiniGame/MiniGame";
import {
ClarificationQuestionsCard,
ClarifyingQuestion,
@@ -81,9 +79,8 @@ function getAccordionMeta(output: EditAgentToolOutput): {
isOperationInProgressOutput(output)
) {
return {
icon: <OrbitLoader size={32} />,
title: "Editing agent, this may take a few minutes. Play while you wait.",
expanded: true,
icon,
title: output.message || "Agent editing started",
};
}
return {
@@ -148,15 +145,22 @@ export function EditAgentTool({ part }: Props) {
/>
</div>
{isStreaming && (
<ToolAccordion
icon={<AccordionIcon />}
title="Editing agent, this may take a few minutes. Play while you wait."
expanded
>
<ContentGrid>
<MiniGame />
</ContentGrid>
</ToolAccordion>
)}
{hasExpandableContent && output && (
<ToolAccordion {...getAccordionMeta(output)}>
{isOperating && (
<ContentGrid>
<MiniGame />
<ContentHint>
This could take a few minutes play while you wait!
</ContentHint>
</ContentGrid>
{isOperating && output.message && (
<ContentMessage>{output.message}</ContentMessage>
)}
{isAgentSavedOutput(output) && (

View File

@@ -9,7 +9,7 @@ import {
ContentHint,
ContentMessage,
} from "../../components/ToolAccordion/AccordionContent";
import { MiniGame } from "../CreateAgent/components/MiniGame/MiniGame";
import { MiniGame } from "../../components/MiniGame/MiniGame";
import {
getAccordionMeta,
getAnimationText,
@@ -47,14 +47,25 @@ export function RunAgentTool({ part }: Props) {
const isError =
part.state === "output-error" ||
(!!output && isRunAgentErrorOutput(output));
const isOutputAvailable = part.state === "output-available" && !!output;
const setupRequirementsOutput =
isOutputAvailable && isRunAgentSetupRequirementsOutput(output)
? output
: null;
const agentDetailsOutput =
isOutputAvailable && isRunAgentAgentDetailsOutput(output) ? output : null;
const needLoginOutput =
isOutputAvailable && isRunAgentNeedLoginOutput(output) ? output : null;
const hasExpandableContent =
part.state === "output-available" &&
!!output &&
(isRunAgentExecutionStartedOutput(output) ||
isRunAgentAgentDetailsOutput(output) ||
isRunAgentSetupRequirementsOutput(output) ||
isRunAgentNeedLoginOutput(output) ||
isRunAgentErrorOutput(output));
isOutputAvailable &&
!setupRequirementsOutput &&
!agentDetailsOutput &&
!needLoginOutput &&
(isRunAgentExecutionStartedOutput(output) || isRunAgentErrorOutput(output));
return (
<div className="py-2">
@@ -81,24 +92,30 @@ export function RunAgentTool({ part }: Props) {
</ToolAccordion>
)}
{setupRequirementsOutput && (
<div className="mt-2">
<SetupRequirementsCard output={setupRequirementsOutput} />
</div>
)}
{agentDetailsOutput && (
<div className="mt-2">
<AgentDetailsCard output={agentDetailsOutput} />
</div>
)}
{needLoginOutput && (
<div className="mt-2">
<ContentMessage>{needLoginOutput.message}</ContentMessage>
</div>
)}
{hasExpandableContent && output && (
<ToolAccordion {...getAccordionMeta(output)}>
{isRunAgentExecutionStartedOutput(output) && (
<ExecutionStartedCard output={output} />
)}
{isRunAgentAgentDetailsOutput(output) && (
<AgentDetailsCard output={output} />
)}
{isRunAgentSetupRequirementsOutput(output) && (
<SetupRequirementsCard output={output} />
)}
{isRunAgentNeedLoginOutput(output) && (
<ContentMessage>{output.message}</ContentMessage>
)}
{isRunAgentErrorOutput(output) && <ErrorCard output={output} />}
</ToolAccordion>
)}

View File

@@ -1,10 +1,11 @@
"use client";
import { useState } from "react";
import { CredentialsGroupedView } from "@/components/contextual/CredentialsInput/components/CredentialsGroupedView/CredentialsGroupedView";
import { Button } from "@/components/atoms/Button/Button";
import type { CredentialsMetaInput } from "@/lib/autogpt-server-api/types";
import type { SetupRequirementsResponse } from "@/app/api/__generated__/models/setupRequirementsResponse";
import { Button } from "@/components/atoms/Button/Button";
import { Text } from "@/components/atoms/Text/Text";
import { CredentialsGroupedView } from "@/components/contextual/CredentialsInput/components/CredentialsGroupedView/CredentialsGroupedView";
import type { CredentialsMetaInput } from "@/lib/autogpt-server-api/types";
import { useState } from "react";
import { useCopilotChatActions } from "../../../../components/CopilotChatActionsProvider/useCopilotChatActions";
import {
ContentBadge,
@@ -38,40 +39,40 @@ export function SetupRequirementsCard({ output }: Props) {
setInputCredentials((prev) => ({ ...prev, [key]: value }));
}
const isAllComplete =
credentialFields.length > 0 &&
const needsCredentials = credentialFields.length > 0;
const isAllCredentialsComplete =
needsCredentials &&
[...requiredCredentials].every((key) => !!inputCredentials[key]);
const canProceed =
!hasSent && (!needsCredentials || isAllCredentialsComplete);
function handleProceed() {
setHasSent(true);
onSend(
"I've configured the required credentials. Please check if everything is ready and proceed with running the agent.",
);
const message = needsCredentials
? "I've configured the required credentials. Please check if everything is ready and proceed with running the agent."
: "Please proceed with running the agent.";
onSend(message);
}
return (
<div className="grid gap-2">
<ContentMessage>{output.message}</ContentMessage>
{credentialFields.length > 0 && (
{needsCredentials && (
<div className="rounded-2xl border bg-background p-3">
<CredentialsGroupedView
credentialFields={credentialFields}
requiredCredentials={requiredCredentials}
inputCredentials={inputCredentials}
inputValues={{}}
onCredentialChange={handleCredentialChange}
/>
{isAllComplete && !hasSent && (
<Button
variant="primary"
size="small"
className="mt-3 w-full"
onClick={handleProceed}
>
Proceed
</Button>
)}
<Text variant="small" className="w-fit border-b text-zinc-500">
Agent credentials
</Text>
<div className="mt-6">
<CredentialsGroupedView
credentialFields={credentialFields}
requiredCredentials={requiredCredentials}
inputCredentials={inputCredentials}
inputValues={{}}
onCredentialChange={handleCredentialChange}
/>
</div>
</div>
)}
@@ -100,6 +101,18 @@ export function SetupRequirementsCard({ output }: Props) {
</div>
</div>
)}
{(needsCredentials || expectedInputs.length > 0) && (
<Button
variant="primary"
size="small"
className="mt-4 w-fit"
disabled={!canProceed}
onClick={handleProceed}
>
Proceed
</Button>
)}
</div>
);
}

View File

@@ -39,12 +39,19 @@ export function RunBlockTool({ part }: Props) {
const isError =
part.state === "output-error" ||
(!!output && isRunBlockErrorOutput(output));
const setupRequirementsOutput =
part.state === "output-available" &&
output &&
isRunBlockSetupRequirementsOutput(output)
? output
: null;
const hasExpandableContent =
part.state === "output-available" &&
!!output &&
!setupRequirementsOutput &&
(isRunBlockBlockOutput(output) ||
isRunBlockDetailsOutput(output) ||
isRunBlockSetupRequirementsOutput(output) ||
isRunBlockErrorOutput(output));
return (
@@ -57,6 +64,12 @@ export function RunBlockTool({ part }: Props) {
/>
</div>
{setupRequirementsOutput && (
<div className="mt-2">
<SetupRequirementsCard output={setupRequirementsOutput} />
</div>
)}
{hasExpandableContent && output && (
<ToolAccordion {...getAccordionMeta(output)}>
{isRunBlockBlockOutput(output) && <BlockOutputCard output={output} />}
@@ -65,10 +78,6 @@ export function RunBlockTool({ part }: Props) {
<BlockDetailsCard output={output} />
)}
{isRunBlockSetupRequirementsOutput(output) && (
<SetupRequirementsCard output={output} />
)}
{isRunBlockErrorOutput(output) && <ErrorCard output={output} />}
</ToolAccordion>
)}

View File

@@ -6,15 +6,9 @@ import { Text } from "@/components/atoms/Text/Text";
import { CredentialsGroupedView } from "@/components/contextual/CredentialsInput/components/CredentialsGroupedView/CredentialsGroupedView";
import { FormRenderer } from "@/components/renderers/InputRenderer/FormRenderer";
import type { CredentialsMetaInput } from "@/lib/autogpt-server-api/types";
import { AnimatePresence, motion } from "framer-motion";
import { useState } from "react";
import { useCopilotChatActions } from "../../../../components/CopilotChatActionsProvider/useCopilotChatActions";
import {
ContentBadge,
ContentCardDescription,
ContentCardTitle,
ContentMessage,
} from "../../../../components/ToolAccordion/AccordionContent";
import { ContentMessage } from "../../../../components/ToolAccordion/AccordionContent";
import {
buildExpectedInputsSchema,
coerceCredentialFields,
@@ -31,10 +25,8 @@ export function SetupRequirementsCard({ output }: Props) {
const [inputCredentials, setInputCredentials] = useState<
Record<string, CredentialsMetaInput | undefined>
>({});
const [hasSentCredentials, setHasSentCredentials] = useState(false);
const [showInputForm, setShowInputForm] = useState(false);
const [inputValues, setInputValues] = useState<Record<string, unknown>>({});
const [hasSent, setHasSent] = useState(false);
const { credentialFields, requiredCredentials } = coerceCredentialFields(
output.setup_info.user_readiness?.missing_credentials,
@@ -50,27 +42,49 @@ export function SetupRequirementsCard({ output }: Props) {
setInputCredentials((prev) => ({ ...prev, [key]: value }));
}
const needsCredentials = credentialFields.length > 0;
const isAllCredentialsComplete =
credentialFields.length > 0 &&
needsCredentials &&
[...requiredCredentials].every((key) => !!inputCredentials[key]);
function handleProceedCredentials() {
setHasSentCredentials(true);
onSend(
"I've configured the required credentials. Please re-run the block now.",
);
}
const needsInputs = inputSchema !== null;
const requiredInputNames = expectedInputs
.filter((i) => i.required)
.map((i) => i.name);
const isAllInputsComplete =
needsInputs &&
requiredInputNames.every((name) => {
const v = inputValues[name];
return v !== undefined && v !== null && v !== "";
});
function handleRunWithInputs() {
const nonEmpty = Object.fromEntries(
Object.entries(inputValues).filter(
([, v]) => v !== undefined && v !== null && v !== "",
),
);
onSend(
`Run the block with these inputs: ${JSON.stringify(nonEmpty, null, 2)}`,
);
setShowInputForm(false);
const canRun =
!hasSent &&
(!needsCredentials || isAllCredentialsComplete) &&
(!needsInputs || isAllInputsComplete);
function handleRun() {
setHasSent(true);
const parts: string[] = [];
if (needsCredentials) {
parts.push("I've configured the required credentials.");
}
if (needsInputs) {
const nonEmpty = Object.fromEntries(
Object.entries(inputValues).filter(
([, v]) => v !== undefined && v !== null && v !== "",
),
);
parts.push(
`Run the block with these inputs: ${JSON.stringify(nonEmpty, null, 2)}`,
);
} else {
parts.push("Please re-run the block now.");
}
onSend(parts.join(" "));
setInputValues({});
}
@@ -78,119 +92,54 @@ export function SetupRequirementsCard({ output }: Props) {
<div className="grid gap-2">
<ContentMessage>{output.message}</ContentMessage>
{credentialFields.length > 0 && (
{needsCredentials && (
<div className="rounded-2xl border bg-background p-3">
<CredentialsGroupedView
credentialFields={credentialFields}
requiredCredentials={requiredCredentials}
inputCredentials={inputCredentials}
inputValues={{}}
onCredentialChange={handleCredentialChange}
/>
{isAllCredentialsComplete && !hasSentCredentials && (
<Button
variant="primary"
size="small"
className="mt-3 w-full"
onClick={handleProceedCredentials}
>
Proceed
</Button>
)}
<Text variant="small" className="w-fit border-b text-zinc-500">
Block credentials
</Text>
<div className="mt-6">
<CredentialsGroupedView
credentialFields={credentialFields}
requiredCredentials={requiredCredentials}
inputCredentials={inputCredentials}
inputValues={{}}
onCredentialChange={handleCredentialChange}
/>
</div>
</div>
)}
{inputSchema && (
<div className="flex gap-2 pt-2">
<Button
variant="outline"
size="small"
className="w-fit"
onClick={() => setShowInputForm((prev) => !prev)}
>
{showInputForm ? "Hide inputs" : "Fill in inputs"}
</Button>
<div className="rounded-2xl border bg-background p-3 pt-4">
<Text variant="small" className="w-fit border-b text-zinc-500">
Block inputs
</Text>
<FormRenderer
jsonSchema={inputSchema}
className="mb-3 mt-3"
handleChange={(v) => setInputValues(v.formData ?? {})}
uiSchema={{
"ui:submitButtonOptions": { norender: true },
}}
initialValues={inputValues}
formContext={{
showHandles: false,
size: "small",
}}
/>
</div>
)}
<AnimatePresence initial={false}>
{showInputForm && inputSchema && (
<motion.div
initial={{ height: 0, opacity: 0, filter: "blur(6px)" }}
animate={{ height: "auto", opacity: 1, filter: "blur(0px)" }}
exit={{ height: 0, opacity: 0, filter: "blur(6px)" }}
transition={{
height: { type: "spring", bounce: 0.15, duration: 0.5 },
opacity: { duration: 0.25 },
filter: { duration: 0.2 },
}}
className="overflow-hidden"
style={{ willChange: "height, opacity, filter" }}
>
<div className="rounded-2xl border bg-background p-3 pt-4">
<Text variant="body-medium">Block inputs</Text>
<FormRenderer
jsonSchema={inputSchema}
handleChange={(v) => setInputValues(v.formData ?? {})}
uiSchema={{
"ui:submitButtonOptions": { norender: true },
}}
initialValues={inputValues}
formContext={{
showHandles: false,
size: "small",
}}
/>
<div className="-mt-8 flex gap-2">
<Button
variant="primary"
size="small"
className="w-fit"
onClick={handleRunWithInputs}
>
Run
</Button>
<Button
variant="secondary"
size="small"
className="w-fit"
onClick={() => {
setShowInputForm(false);
setInputValues({});
}}
>
Cancel
</Button>
</div>
</div>
</motion.div>
)}
</AnimatePresence>
{expectedInputs.length > 0 && !inputSchema && (
<div className="rounded-2xl border bg-background p-3">
<ContentCardTitle className="text-xs">
Expected inputs
</ContentCardTitle>
<div className="mt-2 grid gap-2">
{expectedInputs.map((input) => (
<div key={input.name} className="rounded-xl border p-2">
<div className="flex items-center justify-between gap-2">
<ContentCardTitle className="text-xs">
{input.title}
</ContentCardTitle>
<ContentBadge>
{input.required ? "Required" : "Optional"}
</ContentBadge>
</div>
<ContentCardDescription className="mt-1">
{input.name} &bull; {input.type}
{input.description ? ` \u2022 ${input.description}` : ""}
</ContentCardDescription>
</div>
))}
</div>
</div>
{(needsCredentials || needsInputs) && (
<Button
variant="primary"
size="small"
className="w-fit"
disabled={!canRun}
onClick={handleRun}
>
Proceed
</Button>
)}
</div>
);

View File

@@ -119,7 +119,7 @@ export function CredentialsFlatView({
) : (
!readOnly && (
<Button
variant="secondary"
variant="primary"
size="small"
onClick={onAddCredential}
className="w-fit"

View File

@@ -1,10 +1,11 @@
import { cn } from "@/lib/utils";
import { RJSFSchema } from "@rjsf/utils";
import { preprocessInputSchema } from "./utils/input-schema-pre-processor";
import { useMemo } from "react";
import { customValidator } from "./utils/custom-validator";
import Form from "./registry";
import { ExtendedFormContextType } from "./types";
import { customValidator } from "./utils/custom-validator";
import { generateUiSchemaForCustomFields } from "./utils/generate-ui-schema";
import { preprocessInputSchema } from "./utils/input-schema-pre-processor";
type FormRendererProps = {
jsonSchema: RJSFSchema;
@@ -12,15 +13,17 @@ type FormRendererProps = {
uiSchema: any;
initialValues: any;
formContext: ExtendedFormContextType;
className?: string;
};
export const FormRenderer = ({
export function FormRenderer({
jsonSchema,
handleChange,
uiSchema,
initialValues,
formContext,
}: FormRendererProps) => {
className,
}: FormRendererProps) {
const preprocessedSchema = useMemo(() => {
return preprocessInputSchema(jsonSchema);
}, [jsonSchema]);
@@ -31,7 +34,10 @@ export const FormRenderer = ({
}, [preprocessedSchema, uiSchema]);
return (
<div className={"mb-6 mt-4"} data-tutorial-id="input-handles">
<div
className={cn("mb-6 mt-4", className)}
data-tutorial-id="input-handles"
>
<Form
formContext={formContext}
idPrefix="agpt"
@@ -45,4 +51,4 @@ export const FormRenderer = ({
/>
</div>
);
};
}

View File

@@ -15,6 +15,7 @@
## Advanced Setup
* [Advanced Setup](advanced_setup.md)
* [Deployment Environment Variables](deployment-environment-variables.md)
## Building Blocks

View File

@@ -0,0 +1,397 @@
# Deployment Environment Variables
This guide documents **all environment variables that must be configured** when deploying AutoGPT to a new server or environment. Use this as a checklist to ensure your deployment works correctly.
## Quick Reference: What MUST Change
When deploying to a new server, these variables **must** be updated from their localhost defaults:
| Variable | Location | Default | Purpose |
|----------|----------|---------|---------|
| `SITE_URL` | `.env` | `http://localhost:3000` | Frontend URL for auth redirects |
| `API_EXTERNAL_URL` | `.env` | `http://localhost:8000` | Public Supabase API URL |
| `SUPABASE_PUBLIC_URL` | `.env` | `http://localhost:8000` | Studio dashboard URL |
| `PLATFORM_BASE_URL` | `backend/.env` | `http://localhost:8000` | Backend platform URL |
| `FRONTEND_BASE_URL` | `backend/.env` | `http://localhost:3000` | Frontend URL for webhooks/OAuth |
| `NEXT_PUBLIC_SUPABASE_URL` | `frontend/.env` | `http://localhost:8000` | Client-side Supabase URL |
| `NEXT_PUBLIC_AGPT_SERVER_URL` | `frontend/.env` | `http://localhost:8006/api` | Client-side backend API URL |
| `NEXT_PUBLIC_AGPT_WS_SERVER_URL` | `frontend/.env` | `ws://localhost:8001/ws` | Client-side WebSocket URL |
| `NEXT_PUBLIC_FRONTEND_BASE_URL` | `frontend/.env` | `http://localhost:3000` | Client-side frontend URL |
---
## Configuration Files
AutoGPT uses multiple `.env` files across different components:
```text
autogpt_platform/
├── .env # Supabase/infrastructure config
├── backend/
│ ├── .env.default # Backend defaults (DO NOT EDIT)
│ └── .env # Your backend overrides
└── frontend/
├── .env.default # Frontend defaults (DO NOT EDIT)
└── .env # Your frontend overrides
```
**Loading Order** (later overrides earlier):
1. `*.env.default` - Base defaults
2. `*.env` - Your overrides
3. Docker `environment:` section
4. Shell environment variables
---
## 1. URL Configuration (REQUIRED)
These URLs must be updated to match your deployment domain/IP.
### Root `.env` (Supabase)
```bash
# Auth redirects - where users return after login
SITE_URL=https://your-domain.com:3000
# Public API URL - exposed to clients
API_EXTERNAL_URL=https://your-domain.com:8000
# Studio dashboard URL
SUPABASE_PUBLIC_URL=https://your-domain.com:8000
```
### Backend `.env`
```bash
# Platform URLs for webhooks and OAuth callbacks
PLATFORM_BASE_URL=https://your-domain.com:8000
FRONTEND_BASE_URL=https://your-domain.com:3000
# Internal Supabase URL (use Docker service name if containerized)
SUPABASE_URL=http://kong:8000 # Docker
# SUPABASE_URL=https://your-domain.com:8000 # External
```
### Frontend `.env`
```bash
# Client-side URLs (used in browser)
NEXT_PUBLIC_SUPABASE_URL=https://your-domain.com:8000
NEXT_PUBLIC_AGPT_SERVER_URL=https://your-domain.com:8006/api
NEXT_PUBLIC_AGPT_WS_SERVER_URL=wss://your-domain.com:8001/ws
NEXT_PUBLIC_FRONTEND_BASE_URL=https://your-domain.com:3000
```
!!! warning "HTTPS Note"
For production, use HTTPS URLs and `wss://` for WebSocket. You'll need a reverse proxy (nginx, Caddy) with SSL certificates.
!!! info "Port Numbers"
The port numbers shown (`:3000`, `:8000`, `:8001`, `:8006`) are internal Docker service ports. In production with a reverse proxy, your public URLs typically won't include port numbers (e.g., `https://your-domain.com` instead of `https://your-domain.com:3000`). Configure your reverse proxy to route external traffic to the internal service ports.
---
## 2. Security Keys (MUST REGENERATE)
These default values are **public** and **must be changed** for production.
### Root `.env`
```bash
# Database password
POSTGRES_PASSWORD=<generate-strong-password>
# JWT secret for Supabase auth (min 32 chars)
JWT_SECRET=<generate-random-string>
# Supabase keys (regenerate with matching JWT_SECRET)
ANON_KEY=<regenerate>
SERVICE_ROLE_KEY=<regenerate>
# Studio dashboard credentials
DASHBOARD_USERNAME=<your-username>
DASHBOARD_PASSWORD=<strong-password>
# Encryption keys
SECRET_KEY_BASE=<generate-random-string>
VAULT_ENC_KEY=<generate-32-char-key> # Run: openssl rand -hex 16
```
### Backend `.env`
```bash
# Must match root POSTGRES_PASSWORD
DB_PASS=<same-as-POSTGRES_PASSWORD>
# Must match root SERVICE_ROLE_KEY
SUPABASE_SERVICE_ROLE_KEY=<same-as-SERVICE_ROLE_KEY>
# Must match root JWT_SECRET
JWT_VERIFY_KEY=<same-as-JWT_SECRET>
# Generate new encryption keys
# Run: python -c "from cryptography.fernet import Fernet;print(Fernet.generate_key().decode())"
ENCRYPTION_KEY=<generated-fernet-key>
UNSUBSCRIBE_SECRET_KEY=<generated-fernet-key>
```
### Generating Keys
```bash
# Generate Fernet encryption key (for ENCRYPTION_KEY, UNSUBSCRIBE_SECRET_KEY)
python -c "from cryptography.fernet import Fernet;print(Fernet.generate_key().decode())"
# Generate random string (for JWT_SECRET, SECRET_KEY_BASE)
openssl rand -base64 32
# Generate 32-character key (for VAULT_ENC_KEY)
openssl rand -hex 16
# Generate Supabase keys (requires matching JWT_SECRET)
# Use: https://supabase.com/docs/guides/self-hosting/docker#generate-api-keys
```
---
## 3. Database Configuration
### Root `.env`
```bash
POSTGRES_HOST=db # Docker service name or external host
POSTGRES_DB=postgres
POSTGRES_PORT=5432
POSTGRES_PASSWORD=<your-password>
```
### Backend `.env`
```bash
DB_USER=postgres
DB_PASS=<your-password>
DB_NAME=postgres
DB_PORT=5432
DB_HOST=localhost # Default is localhost; use 'db' in Docker
DB_SCHEMA=platform
# Connection pooling
DB_CONNECTION_LIMIT=12
DB_CONNECT_TIMEOUT=60
DB_POOL_TIMEOUT=300
# Full connection URL (auto-constructed from above in .env.default)
# Variable substitution is handled automatically; only override if you need custom parameters
DATABASE_URL="postgresql://${DB_USER}:${DB_PASS}@${DB_HOST}:${DB_PORT}/${DB_NAME}?schema=${DB_SCHEMA}"
```
---
## 4. Service Dependencies
### Redis
```bash
REDIS_HOST=redis # Docker: 'redis', External: hostname/IP
REDIS_PORT=6379
# REDIS_PASSWORD= # Uncomment if using authentication
```
### RabbitMQ
```bash
RABBITMQ_DEFAULT_USER=<username>
RABBITMQ_DEFAULT_PASS=<strong-password>
# In Docker, host is 'rabbitmq'
```
---
## 5. Default Ports
| Service | Port | Purpose |
|---------|------|---------|
| Frontend | 3000 | Next.js web UI |
| Kong (Supabase API) | 8000 | API gateway |
| WebSocket Server | 8001 | Real-time updates |
| Executor | 8002 | Agent execution |
| Scheduler | 8003 | Scheduled tasks |
| Database Manager | 8005 | DB operations |
| REST Server | 8006 | Main API |
| Notification Server | 8007 | Notifications |
| PostgreSQL | 5432 | Database |
| Redis | 6379 | Cache/queue |
| RabbitMQ | 5672/15672 | Message queue |
| ClamAV | 3310 | Antivirus scanning |
---
## 6. OAuth Callbacks
When configuring OAuth providers, use this callback URL format:
```text
https://your-domain.com/auth/integrations/oauth_callback
# Or with explicit port if not using a reverse proxy:
# https://your-domain.com:3000/auth/integrations/oauth_callback
```
### Supported OAuth Providers
| Provider | Env Variables | Setup URL |
|----------|---------------|-----------|
| GitHub | `GITHUB_CLIENT_ID`, `GITHUB_CLIENT_SECRET` | [github.com/settings/developers](https://github.com/settings/developers) |
| Google | `GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET` | [console.cloud.google.com](https://console.cloud.google.com/apis/credentials) |
| Discord | `DISCORD_CLIENT_ID`, `DISCORD_CLIENT_SECRET` | [discord.com/developers](https://discord.com/developers/applications) |
| Twitter/X | `TWITTER_CLIENT_ID`, `TWITTER_CLIENT_SECRET` | [developer.x.com](https://developer.x.com) |
| Notion | `NOTION_CLIENT_ID`, `NOTION_CLIENT_SECRET` | [developers.notion.com](https://developers.notion.com) |
| Linear | `LINEAR_CLIENT_ID`, `LINEAR_CLIENT_SECRET` | [linear.app/settings/api](https://linear.app/settings/api/applications/new) |
| Reddit | `REDDIT_CLIENT_ID`, `REDDIT_CLIENT_SECRET` | [reddit.com/prefs/apps](https://reddit.com/prefs/apps) |
| Todoist | `TODOIST_CLIENT_ID`, `TODOIST_CLIENT_SECRET` | [developer.todoist.com](https://developer.todoist.com/appconsole.html) |
---
## 7. Optional Services
### AI/LLM Providers
```bash
OPENAI_API_KEY=
ANTHROPIC_API_KEY=
GROQ_API_KEY=
OPEN_ROUTER_API_KEY=
NVIDIA_API_KEY=
```
### Email (SMTP)
```bash
# Supabase auth emails
SMTP_HOST=smtp.example.com
SMTP_PORT=587
SMTP_USER=<username>
SMTP_PASS=<password>
SMTP_ADMIN_EMAIL=admin@example.com
# Application emails (Postmark)
POSTMARK_SERVER_API_TOKEN=
POSTMARK_SENDER_EMAIL=noreply@your-domain.com
```
### Payments (Stripe)
```bash
STRIPE_API_KEY=
STRIPE_WEBHOOK_SECRET=
```
### Error Tracking (Sentry)
```bash
SENTRY_DSN=
```
### Analytics (PostHog)
```bash
POSTHOG_API_KEY=
POSTHOG_HOST=https://eu.i.posthog.com
# Frontend
NEXT_PUBLIC_POSTHOG_KEY=
NEXT_PUBLIC_POSTHOG_HOST=https://eu.i.posthog.com
```
---
## 8. Deployment Checklist
Use this checklist when deploying to a new environment:
### Pre-deployment
- [ ] Clone repository and navigate to `autogpt_platform/`
- [ ] Copy all `.env.default` files to `.env`
- [ ] Determine your deployment domain/IP
### URL Configuration
- [ ] Update `SITE_URL` in root `.env`
- [ ] Update `API_EXTERNAL_URL` in root `.env`
- [ ] Update `SUPABASE_PUBLIC_URL` in root `.env`
- [ ] Update `PLATFORM_BASE_URL` in `backend/.env`
- [ ] Update `FRONTEND_BASE_URL` in `backend/.env`
- [ ] Update all `NEXT_PUBLIC_*` URLs in `frontend/.env`
### Security
- [ ] Generate new `POSTGRES_PASSWORD`
- [ ] Generate new `JWT_SECRET` (min 32 chars)
- [ ] Regenerate `ANON_KEY` and `SERVICE_ROLE_KEY`
- [ ] Change `DASHBOARD_USERNAME` and `DASHBOARD_PASSWORD`
- [ ] Generate new `ENCRYPTION_KEY` (backend)
- [ ] Generate new `UNSUBSCRIBE_SECRET_KEY` (backend)
- [ ] Update `DB_PASS` to match `POSTGRES_PASSWORD`
- [ ] Update `JWT_VERIFY_KEY` to match `JWT_SECRET`
- [ ] Update `SUPABASE_SERVICE_ROLE_KEY` to match
### Services
- [ ] Configure Redis connection (if external)
- [ ] Configure RabbitMQ credentials
- [ ] Configure SMTP for emails (if needed)
### OAuth (if using integrations)
- [ ] Register OAuth apps with your callback URL
- [ ] Add client IDs and secrets to `backend/.env`
### Post-deployment
- [ ] Run `docker compose up -d --build`
- [ ] Verify frontend loads at your URL
- [ ] Test authentication flow
- [ ] Test WebSocket connection (real-time updates)
---
## 9. Docker vs External Services
### Running Everything in Docker (Default)
The docker-compose files automatically set internal hostnames:
```yaml
# Internal Docker service names (container-to-container communication)
# These are set automatically in docker-compose.platform.yml
DB_HOST: db
REDIS_HOST: redis
RABBITMQ_HOST: rabbitmq
SUPABASE_URL: http://kong:8000
```
### Using External Services
If using managed services (AWS RDS, Redis Cloud, etc.), override in your `.env`:
```bash
# External PostgreSQL
DB_HOST=your-rds-instance.region.rds.amazonaws.com
DB_PORT=5432
# External Redis
REDIS_HOST=your-redis.cache.amazonaws.com
REDIS_PORT=6379
REDIS_PASSWORD=<if-required>
# External Supabase (hosted)
SUPABASE_URL=https://your-project.supabase.co
SUPABASE_SERVICE_ROLE_KEY=<your-service-role-key>
```
---
## Related Documentation
- [Getting Started](getting-started.md) - Basic setup guide
- [Advanced Setup](advanced_setup.md) - Development configuration
- [OAuth & SSO](integrating/oauth-guide.md) - Integration setup

View File

@@ -218,6 +218,17 @@ If you initially installed Docker with Hyper-V, you **dont need to reinstall*
For more details, refer to [Docker's official documentation](https://docs.docker.com/desktop/windows/wsl/).
### ⚠️ Podman Not Supported
AutoGPT requires **Docker** (Docker Desktop or Docker Engine). **Podman and podman-compose are not supported** and may cause path resolution issues, particularly on Windows.
If you see errors like:
```text
Error: the specified Containerfile or Dockerfile does not exist, ..\..\autogpt_platform\backend\Dockerfile
```
This indicates you're using Podman instead of Docker. Please install [Docker Desktop](https://docs.docker.com/desktop/) and use `docker compose` instead of `podman-compose`.
## Development