Compare commits

..

12 Commits

Author SHA1 Message Date
Zamil Majdy
bac7b9efb9 fix(copilot): update shared counter after collision detection
When collision detection in add_chat_messages_batch retries with a higher
sequence number, the actual persisted message count may differ from
len(session.messages). This commit ensures the shared counter
(saved_msg_count_ref) used by the streaming loop and long-running callback
stays synchronized with the actual DB state.

Changes:
- Modified add_chat_messages_batch to return tuple[list[ChatMessage], int]
  where the int is the final message count after collision resolution
- Updated _save_session_to_db and upsert_chat_session to propagate the
  final count up the call chain
- Updated all callers in sdk/service.py to use the returned count instead
  of len(session.messages) when updating saved_msg_count_ref
- Updated all other callers in service.py and tests to handle tuple return
2026-02-20 18:58:02 +07:00
Zamil Majdy
6e1941d7ae feat(copilot): implement session locking to prevent concurrent streams
- Add stream_id (using task_id) to uniquely identify each stream
- Acquire exclusive lock (Redis SET NX EX) when starting a stream
- Release lock in finally block using Lua script (atomic compare-and-delete)
- Return error if another stream is already active for the session
- Lock TTL is 1 hour (matches stream_ttl) with automatic cleanup

This prevents:
- Message duplication from concurrent streams
- Race conditions in message saves
- Confusing UX with multiple AI responses
- Frontend reconnecting while existing stream is active
- Multiple browser tabs streaming to same session
2026-02-20 18:28:35 +07:00
Zamil Majdy
129b992059 feat(copilot): increase long-running operation TTL to 1 hour
- Increase long_running_operation_ttl from 600s (10min) to 3600s (1hour)
- Match stream_ttl duration for consistency
- Add clarifying description about deduplication lock purpose

Some operations (like complex agent runs) can take longer than 10 minutes.
The stream_registry heartbeat (publish_chunk) already keeps operations alive,
so this TTL is just a safety net for deduplication.
2026-02-20 18:22:26 +07:00
Zamil Majdy
1b82a55eca chore: remove obsolete plan file
Plan was completed and changes are now in the PR. No need to keep the plan file.
2026-02-20 18:21:00 +07:00
Zamil Majdy
9d4697e859 refactor(copilot): replace COUNT with MAX for sequence tracking
- Rename get_max_sequence() to get_next_sequence() returning MAX+1
- Replace all get_chat_session_message_count() calls with get_next_sequence()
- Remove old get_chat_session_message_count() function
- Update db_manager.py to export get_next_sequence

Using MAX(sequence)+1 is more robust than COUNT(*) because:
- Immune to deleted messages
- Handles gaps in sequence numbers correctly
- Simpler collision detection logic
2026-02-20 18:20:29 +07:00
Zamil Majdy
366547e448 refactor(copilot): remove confusing 'Layer' comments from code
- Remove '(Layer 3: defense-in-depth)' annotations
- Replace with clearer explanations of what the code does
- Makes the code easier to understand without implementation history
2026-02-20 18:18:25 +07:00
Zamil Majdy
af491b5511 refactor(copilot): replace upsert with collision detection for concurrent message saves
- Use create() with MAX(sequence) retry instead of upsert()
- Query DB only on collision (not every save) for better performance
- Remove Layer 2 DB queries from incremental saves in streaming loop
- Add get_max_sequence() helper using raw SQL for robustness
- Collision detection retries up to 3 times on unique constraint errors

This approach:
- Optimizes common case (no collision) - no extra DB queries
- Handles concurrent writes via automatic retry with correct sequence
- Uses MAX(sequence) instead of COUNT for more robust offset calculation
2026-02-20 18:09:34 +07:00
Zamil Majdy
6acefee6f3 fix(copilot): defense-in-depth for concurrent message saves (all 3 layers)
Implements three complementary layers to prevent unique constraint violations
on (sessionId, sequence) caused by concurrent writers during SDK streaming:

**Layer 1: Upsert (already in PR)**
- add_chat_messages_batch uses upsert() instead of create()
- Explicitly constructs update_data excluding Session and sequence
- Final safety net: duplicate sequences update instead of crash

**Layer 2: Query DB Before Each Save (NEW)**
- Query get_chat_session_message_count() before each save
- DB is source of truth, prevents using stale in-memory counter
- Applied to: long-running callback + 2 incremental saves
- Trade-off: Extra COUNT query (~1-2ms), but prevents race

**Layer 3: Shared Counter (NEW)**
- saved_msg_count_ref as mutable list[int] shared between:
  - Streaming loop (incremental saves)
  - Long-running callback (_build_long_running_callback)
- Both writers update it after successful save
- Keeps in-memory tracking accurate for performance

**Why all three:**
- Layer 2 alone: adds DB queries (performance cost)
- Layer 3 alone: doesn't handle external writers
- Layer 1 alone: may silently overwrite data
- Together: correctness + performance + safety net

Files:
- backend/copilot/db.py - Layer 1 (upsert with explicit update_data)
- backend/copilot/sdk/service.py - Layers 2 & 3

Fixes race where long-running tools (create_agent, edit_agent) would
append messages behind streaming loop's back, causing stale counter.

Addresses PR review comments and Discord analysis.
2026-02-20 18:02:00 +07:00
Zamil Majdy
eb4650fbb8 fix(copilot): explicitly construct update_data for better type safety
Instead of filtering from data dict, explicitly build update_data with
only the fields that should be updated. This is safer and makes it
obvious what fields are being updated in the upsert operation.

Addresses PR review comment about exhaustive field construction.
2026-02-20 17:53:52 +07:00
Zamil Majdy
8bdf83128e fix(copilot): address CodeRabbit review - add type safety and exclude sequence from update
- Add ChatMessageUpdateInput import for type-safe update payload
- Exclude both 'Session' and 'sequence' from update_data (sequence is part of composite key)
- Cast update_data to ChatMessageUpdateInput for type checking
- Update docstring to document upsert semantics and idempotency
2026-02-20 17:49:11 +07:00
Zamil Majdy
a1d5b99226 Merge branch 'dev' into otto/fix-chat-messages-batch-upsert 2026-02-20 17:48:02 +07:00
Otto
0450ea5313 fix(copilot): use upsert in add_chat_messages_batch to handle duplicate sequences
Concurrent writers (incremental streaming saves and long-running tool
callbacks) can race to persist messages with the same (sessionId, sequence)
pair, causing unique constraint violations.

Replace prisma create() with upsert() so duplicate sequences update the
existing row instead of failing. This is safe because later writes always
contain the most complete data (e.g. accumulated assistant text).
2026-02-20 09:59:56 +00:00
23 changed files with 464 additions and 1145 deletions

View File

@@ -49,11 +49,6 @@ class ChatConfig(BaseSettings):
default=3600,
description="TTL in seconds for stream data in Redis (1 hour)",
)
stream_lock_ttl: int = Field(
default=120,
description="TTL in seconds for stream lock (2 minutes). Short timeout allows "
"reconnection after refresh/crash without long waits.",
)
stream_max_length: int = Field(
default=10000,
description="Maximum number of messages to store per stream",

View File

@@ -3,9 +3,8 @@
import asyncio
import logging
from datetime import UTC, datetime
from typing import Any
from typing import Any, cast
from prisma.errors import UniqueViolationError
from prisma.models import ChatMessage as PrismaChatMessage
from prisma.models import ChatSession as PrismaChatSession
from prisma.types import (
@@ -93,9 +92,10 @@ async def add_chat_message(
function_call: dict[str, Any] | None = None,
) -> ChatMessage:
"""Add a message to a chat session."""
# Build ChatMessageCreateInput with only non-None values
# (Prisma TypedDict rejects optional fields set to None)
data: ChatMessageCreateInput = {
# Build input dict dynamically rather than using ChatMessageCreateInput directly
# because Prisma's TypedDict validation rejects optional fields set to None.
# We only include fields that have values, then cast at the end.
data: dict[str, Any] = {
"Session": {"connect": {"id": session_id}},
"role": role,
"sequence": sequence,
@@ -123,7 +123,7 @@ async def add_chat_message(
where={"id": session_id},
data={"updatedAt": datetime.now(UTC)},
),
PrismaChatMessage.prisma().create(data=data),
PrismaChatMessage.prisma().create(data=cast(ChatMessageCreateInput, data)),
)
return ChatMessage.from_db(message)
@@ -132,42 +132,38 @@ async def add_chat_messages_batch(
session_id: str,
messages: list[dict[str, Any]],
start_sequence: int,
) -> int:
) -> tuple[list[ChatMessage], int]:
"""Add multiple messages to a chat session in a batch.
Uses collision detection with retry: tries to create messages starting
at start_sequence. If a unique constraint violation occurs (e.g., the
streaming loop and long-running callback race), queries the latest
sequence and retries with the correct offset. This avoids unnecessary
upserts and DB queries in the common case (no collision).
streaming loop and long-running callback race), queries MAX(sequence)
and retries with the correct next sequence number. This avoids
unnecessary upserts and DB queries in the common case (no collision).
Returns:
Next sequence number for the next message to be inserted. This equals
start_sequence + len(messages) and allows callers to update their
counters even when collision detection adjusts start_sequence.
Tuple of (messages, final_message_count) where final_message_count
is the total number of messages in the session after insertion.
This allows callers to update their counters even when collision
detection adjusts start_sequence.
"""
if not messages:
# No messages to add - return current count
return start_sequence
return [], start_sequence
max_retries = 5
max_retries = 3
for attempt in range(max_retries):
try:
# Single timestamp for all messages and session update
now = datetime.now(UTC)
created_messages = []
async with db.transaction() as tx:
# Build all message data
messages_data = []
for i, msg in enumerate(messages):
# Build ChatMessageCreateInput with only non-None values
# (Prisma TypedDict rejects optional fields set to None)
# Note: create_many doesn't support nested creates, use sessionId directly
data: ChatMessageCreateInput = {
"sessionId": session_id,
# Build input dict dynamically rather than using ChatMessageCreateInput
# directly because Prisma's TypedDict validation rejects optional fields
# set to None. We only include fields that have values, then cast.
data: dict[str, Any] = {
"Session": {"connect": {"id": session_id}},
"role": msg["role"],
"sequence": start_sequence + i,
"createdAt": now,
}
# Add optional string fields
@@ -186,23 +182,31 @@ async def add_chat_messages_batch(
if msg.get("function_call") is not None:
data["functionCall"] = SafeJson(msg["function_call"])
messages_data.append(data)
created = await PrismaChatMessage.prisma(tx).create(
data=cast(ChatMessageCreateInput, data)
)
created_messages.append(created)
# Run create_many and session update in parallel within transaction
# Both use the same timestamp for consistency
await asyncio.gather(
PrismaChatMessage.prisma(tx).create_many(data=messages_data),
PrismaChatSession.prisma(tx).update(
where={"id": session_id},
data={"updatedAt": now},
),
# Update session's updatedAt timestamp within the same transaction.
# Note: Token usage (total_prompt_tokens, total_completion_tokens) is updated
# separately via update_chat_session() after streaming completes.
await PrismaChatSession.prisma(tx).update(
where={"id": session_id},
data={"updatedAt": datetime.now(UTC)},
)
# Return next sequence number for counter sync
return start_sequence + len(messages)
# Return messages and final message count (for shared counter sync)
final_count = start_sequence + len(messages)
return [ChatMessage.from_db(m) for m in created_messages], final_count
except UniqueViolationError:
if attempt < max_retries - 1:
except Exception as e:
# Check if it's a unique constraint violation
error_msg = str(e).lower()
is_unique_constraint = (
"unique constraint" in error_msg or "duplicate key" in error_msg
)
if is_unique_constraint and attempt < max_retries - 1:
# Collision detected - query MAX(sequence)+1 and retry with correct offset
logger.info(
f"Collision detected for session {session_id} at sequence "
@@ -214,7 +218,7 @@ async def add_chat_messages_batch(
)
continue
else:
# Max retries exceeded - propagate error
# Not a collision or max retries exceeded - propagate error
raise
# Should never reach here due to raise in exception handler
@@ -277,15 +281,18 @@ async def get_next_sequence(session_id: str) -> int:
Uses MAX(sequence) + 1 for robustness. Returns 0 if no messages exist.
More robust than COUNT(*) because it's immune to deleted messages.
Optimized to select only the sequence column using raw SQL.
The unique index on (sessionId, sequence) makes this query fast.
"""
results = await db.query_raw_with_schema(
'SELECT "sequence" FROM {schema_prefix}"ChatMessage" WHERE "sessionId" = $1 ORDER BY "sequence" DESC LIMIT 1',
result = await db.prisma.query_raw(
"""
SELECT COALESCE(MAX(sequence) + 1, 0) as next_seq
FROM "ChatMessage"
WHERE "sessionId" = $1
""",
session_id,
)
return 0 if not results else results[0]["sequence"] + 1
if not result or len(result) == 0:
return 0
return int(result[0]["next_seq"])
async def update_tool_message_content(

View File

@@ -434,13 +434,25 @@ async def _get_session_from_db(session_id: str) -> ChatSession | None:
async def upsert_chat_session(
session: ChatSession,
) -> ChatSession:
*,
existing_message_count: int | None = None,
) -> tuple[ChatSession, int]:
"""Update a chat session in both cache and database.
Uses session-level locking to prevent race conditions when concurrent
operations (e.g., background title update and main stream handler)
attempt to upsert the same session simultaneously.
Args:
existing_message_count: If provided, skip the DB query to count
existing messages. The caller is responsible for tracking this
accurately. Useful for incremental saves in a streaming loop
where the caller already knows how many messages are persisted.
Returns:
Tuple of (session, final_message_count) where final_message_count is
the actual persisted message count after collision detection adjustments.
Raises:
DatabaseError: If the database write fails. The cache is still updated
as a best-effort optimization, but the error is propagated to ensure
@@ -451,14 +463,18 @@ async def upsert_chat_session(
lock = await _get_session_lock(session.session_id)
async with lock:
# Always query DB for existing message count to ensure consistency
existing_message_count = await chat_db().get_next_sequence(session.session_id)
# Get existing message count from DB for incremental saves
if existing_message_count is None:
existing_message_count = await chat_db().get_next_sequence(
session.session_id
)
db_error: Exception | None = None
final_count = existing_message_count
# Save to database (primary storage)
try:
await _save_session_to_db(
final_count = await _save_session_to_db(
session,
existing_message_count,
skip_existence_check=existing_message_count > 0,
@@ -489,7 +505,7 @@ async def upsert_chat_session(
f"Failed to persist chat session {session.session_id} to database"
) from db_error
return session
return session, final_count
async def _save_session_to_db(
@@ -497,13 +513,16 @@ async def _save_session_to_db(
existing_message_count: int,
*,
skip_existence_check: bool = False,
) -> None:
) -> int:
"""Save or update a chat session in the database.
Args:
skip_existence_check: When True, skip the ``get_chat_session`` query
and assume the session row already exists. Saves one DB round trip
for incremental saves during streaming.
Returns:
Final message count after save (accounting for collision detection).
"""
db = chat_db()
@@ -535,6 +554,7 @@ async def _save_session_to_db(
# Add new messages (only those after existing count)
new_messages = session.messages[existing_message_count:]
final_count = existing_message_count
if new_messages:
messages_data = []
for msg in new_messages:
@@ -554,12 +574,14 @@ async def _save_session_to_db(
f"roles={[m['role'] for m in messages_data]}, "
f"start_sequence={existing_message_count}"
)
await db.add_chat_messages_batch(
_, final_count = await db.add_chat_messages_batch(
session_id=session.session_id,
messages=messages_data,
start_sequence=existing_message_count,
)
return final_count
async def append_and_save_message(session_id: str, message: ChatMessage) -> ChatSession:
"""Atomically append a message to a session and persist it.

View File

@@ -60,7 +60,7 @@ async def test_chatsession_redis_storage(setup_test_user, test_user_id):
s = ChatSession.new(user_id=test_user_id)
s.messages = messages
s = await upsert_chat_session(s)
s, _ = await upsert_chat_session(s)
s2 = await get_chat_session(
session_id=s.session_id,
@@ -77,7 +77,7 @@ async def test_chatsession_redis_storage_user_id_mismatch(
s = ChatSession.new(user_id=test_user_id)
s.messages = messages
s = await upsert_chat_session(s)
s, _ = await upsert_chat_session(s)
s2 = await get_chat_session(s.session_id, "different_user_id")
@@ -94,7 +94,7 @@ async def test_chatsession_db_storage(setup_test_user, test_user_id):
s.messages = messages # Contains user, assistant, and tool messages
assert s.session_id is not None, "Session id is not set"
# Upsert to save to both cache and DB
s = await upsert_chat_session(s)
s, _ = await upsert_chat_session(s)
# Clear the Redis cache to force DB load
redis_key = f"chat:session:{s.session_id}"
@@ -331,96 +331,3 @@ def test_to_openai_messages_merges_split_assistants():
tc_list = merged.get("tool_calls")
assert tc_list is not None and len(list(tc_list)) == 1
assert list(tc_list)[0]["id"] == "tc1"
# --------------------------------------------------------------------------- #
# Concurrent save collision detection #
# --------------------------------------------------------------------------- #
@pytest.mark.asyncio(loop_scope="session")
async def test_concurrent_saves_collision_detection(setup_test_user, test_user_id):
"""Test that concurrent saves from streaming loop and callback handle collisions correctly.
Simulates the race condition where:
1. Streaming loop starts with saved_msg_count=5
2. Long-running callback appends message #5 and saves
3. Streaming loop tries to save with stale count=5
The collision detection should handle this gracefully.
"""
import asyncio
# Create a session with initial messages
session = ChatSession.new(user_id=test_user_id)
for i in range(3):
session.messages.append(
ChatMessage(
role="user" if i % 2 == 0 else "assistant", content=f"Message {i}"
)
)
# Save initial messages
session = await upsert_chat_session(session)
# Simulate streaming loop and callback saving concurrently
async def streaming_loop_save():
"""Simulates streaming loop saving messages."""
# Add 2 messages
session.messages.append(ChatMessage(role="user", content="Streaming message 1"))
session.messages.append(
ChatMessage(role="assistant", content="Streaming message 2")
)
# Wait a bit to let callback potentially save first
await asyncio.sleep(0.01)
# Save (will query DB for existing count)
return await upsert_chat_session(session)
async def callback_save():
"""Simulates long-running callback saving a message."""
# Add 1 message
session.messages.append(
ChatMessage(role="tool", content="Callback result", tool_call_id="tc1")
)
# Save immediately (will query DB for existing count)
return await upsert_chat_session(session)
# Run both saves concurrently - one will hit collision detection
results = await asyncio.gather(streaming_loop_save(), callback_save())
# Both should succeed
assert all(r is not None for r in results)
# Reload session from DB to verify
from backend.data.redis_client import get_redis_async
redis_key = f"chat:session:{session.session_id}"
async_redis = await get_redis_async()
await async_redis.delete(redis_key) # Clear cache to force DB load
loaded_session = await get_chat_session(session.session_id, test_user_id)
assert loaded_session is not None
# Should have all 6 messages (3 initial + 2 streaming + 1 callback)
assert len(loaded_session.messages) == 6
# Verify no duplicate sequences
sequences = []
for i, msg in enumerate(loaded_session.messages):
# Messages should have sequential sequence numbers starting from 0
sequences.append(i)
# All sequences should be unique and sequential
assert sequences == list(range(6))
# Verify message content is preserved
contents = [m.content for m in loaded_session.messages]
assert "Message 0" in contents
assert "Message 1" in contents
assert "Message 2" in contents
assert "Streaming message 1" in contents
assert "Streaming message 2" in contents
assert "Callback result" in contents

View File

@@ -7,10 +7,9 @@ import os
import uuid
from collections.abc import AsyncGenerator
from dataclasses import dataclass
from typing import Any, cast
from typing import Any
from backend.data.redis_client import get_redis_async
from backend.executor.cluster_lock import AsyncClusterLock
from backend.util.exceptions import NotFoundError
from .. import stream_registry
@@ -63,7 +62,6 @@ from .transcript import (
logger = logging.getLogger(__name__)
config = ChatConfig()
# Set to hold background tasks to prevent garbage collection
_background_tasks: set[asyncio.Task[Any]] = set()
@@ -135,11 +133,64 @@ is delivered to the user via a background stream.
All tasks must run in the foreground.
"""
# Session streaming lock configuration
STREAM_LOCK_PREFIX = "copilot:stream:lock:"
STREAM_LOCK_TTL = 3600 # 1 hour - matches stream_ttl
async def _acquire_stream_lock(session_id: str, stream_id: str) -> bool:
"""Acquire an exclusive lock for streaming to this session.
Prevents multiple concurrent streams to the same session which can cause:
- Message duplication
- Race conditions in message saves
- Confusing UX with multiple AI responses
Returns:
True if lock was acquired, False if another stream is active.
"""
redis = await get_redis_async()
lock_key = f"{STREAM_LOCK_PREFIX}{session_id}"
# SET NX EX - atomic "set if not exists" with expiry
result = await redis.set(lock_key, stream_id, ex=STREAM_LOCK_TTL, nx=True)
return result is not None
async def _release_stream_lock(session_id: str, stream_id: str) -> None:
"""Release the stream lock if we still own it.
Only releases the lock if the stored stream_id matches ours (prevents
releasing another stream's lock if we somehow timed out).
"""
redis = await get_redis_async()
lock_key = f"{STREAM_LOCK_PREFIX}{session_id}"
# Lua script for atomic compare-and-delete (only delete if value matches)
script = """
if redis.call("GET", KEYS[1]) == ARGV[1] then
return redis.call("DEL", KEYS[1])
else
return 0
end
"""
await redis.eval(script, 1, lock_key, stream_id) # type: ignore[misc]
async def check_active_stream(session_id: str) -> str | None:
"""Check if a stream is currently active for this session.
Returns:
The active stream_id if one exists, None otherwise.
"""
redis = await get_redis_async()
lock_key = f"{STREAM_LOCK_PREFIX}{session_id}"
active_stream = await redis.get(lock_key)
return active_stream.decode() if isinstance(active_stream, bytes) else active_stream
def _build_long_running_callback(
user_id: str | None,
saved_msg_count_ref: list[int] | None = None,
) -> LongRunningCallback:
"""Build a callback that delegates long-running tools to the non-SDK infrastructure.
@@ -151,6 +202,9 @@ def _build_long_running_callback(
Args:
user_id: User ID for the session
saved_msg_count_ref: Mutable reference [count] shared with streaming loop
for coordinating message saves. When provided, the callback will update
it after appending messages to prevent counter drift.
The returned callback matches the ``LongRunningCallback`` signature:
``(tool_name, args, session) -> MCP response dict``.
@@ -218,7 +272,10 @@ def _build_long_running_callback(
)
session.messages.append(pending_message)
# Collision detection happens in add_chat_messages_batch (db.py)
session = await upsert_chat_session(session)
_, final_count = await upsert_chat_session(session)
# Update shared counter so streaming loop stays in sync
if saved_msg_count_ref is not None:
saved_msg_count_ref[0] = final_count
# --- Spawn background task (reuses non-SDK infrastructure) ---
bg_task = asyncio.create_task(
@@ -538,9 +595,6 @@ async def stream_chat_completion_sdk(
f"Session {session_id} not found. Please create a new session first."
)
# Type narrowing: session is guaranteed ChatSession after the check above
session = cast(ChatSession, session)
# Append the new message to the session if it's not already there
new_message_role = "user" if is_user_message else "assistant"
if message and (
@@ -556,7 +610,7 @@ async def stream_chat_completion_sdk(
user_id=user_id, session_id=session_id, message_length=len(message)
)
session = await upsert_chat_session(session)
session, _ = await upsert_chat_session(session)
# Generate title for new sessions (first user message)
if is_user_message and not session.title:
@@ -581,22 +635,16 @@ async def stream_chat_completion_sdk(
stream_id = task_id # Use task_id as unique stream identifier
# Acquire stream lock to prevent concurrent streams to the same session
lock = AsyncClusterLock(
redis=await get_redis_async(),
key=f"{STREAM_LOCK_PREFIX}{session_id}",
owner_id=stream_id,
timeout=config.stream_lock_ttl,
)
lock_owner = await lock.try_acquire()
if lock_owner != stream_id:
# Another stream is active
lock_acquired = await _acquire_stream_lock(session_id, stream_id)
if not lock_acquired:
# Another stream is active - check if it's still alive
active_stream = await check_active_stream(session_id)
logger.warning(
f"[SDK] Session {session_id} already has an active stream: {lock_owner}"
f"[SDK] Session {session_id} already has an active stream: {active_stream}"
)
yield StreamError(
errorText="Another stream is already active for this session. "
"Please wait or stop it.",
"Please wait for it to complete or refresh the page.",
code="stream_already_active",
)
yield StreamFinish()
@@ -618,10 +666,16 @@ async def stream_chat_completion_sdk(
sdk_cwd = _make_sdk_cwd(session_id)
os.makedirs(sdk_cwd, exist_ok=True)
# Initialize saved message counter as mutable list so long-running
# callback and streaming loop can coordinate
saved_msg_count_ref: list[int] = [len(session.messages)]
set_execution_context(
user_id,
session,
long_running_callback=_build_long_running_callback(user_id),
long_running_callback=_build_long_running_callback(
user_id, saved_msg_count_ref
),
)
try:
from claude_agent_sdk import ClaudeAgentOptions, ClaudeSDKClient
@@ -752,6 +806,8 @@ async def stream_chat_completion_sdk(
accumulated_tool_calls: list[dict[str, Any]] = []
has_appended_assistant = False
has_tool_results = False
# Track persisted message count. Uses shared ref so long-running
# callback can update it for coordination
# Use an explicit async iterator with non-cancelling heartbeats.
# CRITICAL: we must NOT cancel __anext__() mid-flight — doing so
@@ -778,8 +834,6 @@ async def stream_chat_completion_sdk(
if not done:
# Timeout — emit heartbeat but keep the task alive
# Also refresh lock TTL to keep it alive
await lock.refresh()
yield StreamHeartbeat()
continue
@@ -932,7 +986,9 @@ async def stream_chat_completion_sdk(
# other devices. Collision detection happens
# in add_chat_messages_batch (db.py).
try:
session = await upsert_chat_session(session)
_, final_count = await upsert_chat_session(session)
# Update shared ref so callback stays in sync
saved_msg_count_ref[0] = final_count
except Exception as save_err:
logger.warning(
"[SDK] [%s] Incremental save " "failed: %s",
@@ -957,7 +1013,9 @@ async def stream_chat_completion_sdk(
# visible on refresh / other devices.
# Collision detection happens in add_chat_messages_batch (db.py).
try:
session = await upsert_chat_session(session)
_, final_count = await upsert_chat_session(session)
# Update shared ref so callback stays in sync
saved_msg_count_ref[0] = final_count
except Exception as save_err:
logger.warning(
"[SDK] [%s] Incremental save " "failed: %s",
@@ -1089,11 +1147,12 @@ async def stream_chat_completion_sdk(
"to use the OpenAI-compatible fallback."
)
session = cast(ChatSession, await asyncio.shield(upsert_chat_session(session)))
_, final_count = await asyncio.shield(upsert_chat_session(session))
logger.info(
"[SDK] [%s] Session saved with %d messages",
"[SDK] [%s] Session saved with %d messages (DB count: %d)",
session_id[:12],
len(session.messages),
final_count,
)
if not stream_completed:
yield StreamFinish()
@@ -1106,11 +1165,10 @@ async def stream_chat_completion_sdk(
raise
except Exception as e:
logger.error(f"[SDK] Error: {e}", exc_info=True)
if session:
try:
await asyncio.shield(upsert_chat_session(session))
except Exception as save_err:
logger.error(f"[SDK] Failed to save session on error: {save_err}")
try:
await asyncio.shield(upsert_chat_session(session))
except Exception as save_err:
logger.error(f"[SDK] Failed to save session on error: {save_err}")
yield StreamError(
errorText="An error occurred. Please try again.",
code="sdk_error",
@@ -1132,7 +1190,7 @@ async def stream_chat_completion_sdk(
if not raw_transcript and use_resume and resume_file:
raw_transcript = read_transcript_file(resume_file)
if raw_transcript and session is not None:
if raw_transcript:
await asyncio.shield(
_try_upload_transcript(
user_id,
@@ -1153,7 +1211,7 @@ async def stream_chat_completion_sdk(
_cleanup_sdk_tool_results(sdk_cwd)
# Release stream lock to allow new streams for this session
await lock.release()
await _release_stream_lock(session_id, stream_id)
async def _try_upload_transcript(

View File

@@ -352,7 +352,7 @@ async def assign_user_to_session(
if not session:
raise NotFoundError(f"Session {session_id} not found")
session.user_id = user_id
session = await upsert_chat_session(session)
session, _ = await upsert_chat_session(session)
return session
@@ -464,7 +464,7 @@ async def stream_chat_completion(
)
upsert_start = time.monotonic()
session = await upsert_chat_session(session)
session, _ = await upsert_chat_session(session)
upsert_time = (time.monotonic() - upsert_start) * 1000
logger.info(
f"[TIMING] upsert_chat_session took {upsert_time:.1f}ms",
@@ -690,7 +690,7 @@ async def stream_chat_completion(
f"tool_responses={len(tool_response_messages)}"
)
if messages_to_save_early or has_appended_streaming_message:
await upsert_chat_session(session)
_ = await upsert_chat_session(session)
has_saved_assistant_message = True
has_yielded_end = True
@@ -729,7 +729,7 @@ async def stream_chat_completion(
if tool_response_messages:
session.messages.extend(tool_response_messages)
try:
await upsert_chat_session(session)
_ = await upsert_chat_session(session)
except Exception as e:
logger.warning(
f"Failed to save interrupted session {session.session_id}: {e}"
@@ -770,7 +770,7 @@ async def stream_chat_completion(
if messages_to_save:
session.messages.extend(messages_to_save)
if messages_to_save or has_appended_streaming_message:
await upsert_chat_session(session)
_ = await upsert_chat_session(session)
if not has_yielded_error:
error_message = str(e)
@@ -854,7 +854,7 @@ async def stream_chat_completion(
not has_long_running_tool_call
and (messages_to_save or has_appended_streaming_message)
):
await upsert_chat_session(session)
_ = await upsert_chat_session(session)
else:
logger.info(
"Assistant message already saved when StreamFinish was received, "
@@ -1526,7 +1526,7 @@ async def _yield_tool_call(
tool_call_id=tool_call_id,
)
session.messages.append(pending_message)
await upsert_chat_session(session)
_ = await upsert_chat_session(session)
await _with_optional_lock(session_lock, _save_pending)
logger.info(
@@ -2020,7 +2020,7 @@ async def _generate_llm_continuation(
fresh_session.messages.append(assistant_message)
# Save to database (not cache) to persist the response
await upsert_chat_session(fresh_session)
_ = await upsert_chat_session(fresh_session)
# Invalidate cache so next poll/refresh gets fresh data
await invalidate_session_cache(session_id)
@@ -2226,7 +2226,7 @@ async def _generate_llm_continuation_with_streaming(
fresh_session.messages.append(assistant_message)
# Save to database (not cache) to persist the response
await upsert_chat_session(fresh_session)
_ = await upsert_chat_session(fresh_session)
# Invalidate cache so next poll/refresh gets fresh data
await invalidate_session_cache(session_id)

View File

@@ -58,7 +58,7 @@ async def test_stream_chat_completion_with_tool_calls(setup_test_user, test_user
return pytest.skip("OPEN_ROUTER_API_KEY is not set, skipping test")
session = await create_chat_session(test_user_id)
session = await upsert_chat_session(session)
session, _ = await upsert_chat_session(session)
has_errors = False
has_ended = False
@@ -104,7 +104,7 @@ async def test_sdk_resume_multi_turn(setup_test_user, test_user_id):
return pytest.skip("CLAUDE_AGENT_USE_RESUME is not enabled, skipping test")
session = await create_chat_session(test_user_id)
session = await upsert_chat_session(session)
session, _ = await upsert_chat_session(session)
# --- Turn 1: send a message with a unique keyword ---
keyword = "ZEPHYR42"

View File

@@ -1,6 +1,5 @@
"""Redis-based distributed locking for cluster coordination."""
import asyncio
import logging
import threading
import time
@@ -8,7 +7,6 @@ from typing import TYPE_CHECKING
if TYPE_CHECKING:
from redis import Redis
from redis.asyncio import Redis as AsyncRedis
logger = logging.getLogger(__name__)
@@ -128,124 +126,3 @@ class ClusterLock:
with self._refresh_lock:
self._last_refresh = 0.0
class AsyncClusterLock:
"""Async Redis-based distributed lock for preventing duplicate execution."""
def __init__(
self, redis: "AsyncRedis", key: str, owner_id: str, timeout: int = 300
):
self.redis = redis
self.key = key
self.owner_id = owner_id
self.timeout = timeout
self._last_refresh = 0.0
self._refresh_lock = asyncio.Lock()
async def try_acquire(self) -> str | None:
"""Try to acquire the lock.
Returns:
- owner_id (self.owner_id) if successfully acquired
- different owner_id if someone else holds the lock
- None if Redis is unavailable or other error
"""
try:
success = await self.redis.set(
self.key, self.owner_id, nx=True, ex=self.timeout
)
if success:
async with self._refresh_lock:
self._last_refresh = time.time()
return self.owner_id # Successfully acquired
# Failed to acquire, get current owner
current_value = await self.redis.get(self.key)
if current_value:
current_owner = (
current_value.decode("utf-8")
if isinstance(current_value, bytes)
else str(current_value)
)
return current_owner
# Key doesn't exist but we failed to set it - race condition or Redis issue
return None
except Exception as e:
logger.error(f"AsyncClusterLock.try_acquire failed for key {self.key}: {e}")
return None
async def refresh(self) -> bool:
"""Refresh lock TTL if we still own it.
Rate limited to at most once every timeout/10 seconds (minimum 1 second).
During rate limiting, still verifies lock existence but skips TTL extension.
Setting _last_refresh to 0 bypasses rate limiting for testing.
Async-safe: uses asyncio.Lock to protect _last_refresh access.
"""
# Calculate refresh interval: max(timeout // 10, 1)
refresh_interval = max(self.timeout // 10, 1)
current_time = time.time()
# Check if we're within the rate limit period (async-safe read)
# _last_refresh == 0 forces a refresh (bypasses rate limiting for testing)
async with self._refresh_lock:
last_refresh = self._last_refresh
is_rate_limited = (
last_refresh > 0 and (current_time - last_refresh) < refresh_interval
)
try:
# Always verify lock existence, even during rate limiting
current_value = await self.redis.get(self.key)
if not current_value:
async with self._refresh_lock:
self._last_refresh = 0
return False
stored_owner = (
current_value.decode("utf-8")
if isinstance(current_value, bytes)
else str(current_value)
)
if stored_owner != self.owner_id:
async with self._refresh_lock:
self._last_refresh = 0
return False
# If rate limited, return True but don't update TTL or timestamp
if is_rate_limited:
return True
# Perform actual refresh
if await self.redis.expire(self.key, self.timeout):
async with self._refresh_lock:
self._last_refresh = current_time
return True
async with self._refresh_lock:
self._last_refresh = 0
return False
except Exception as e:
logger.error(f"AsyncClusterLock.refresh failed for key {self.key}: {e}")
async with self._refresh_lock:
self._last_refresh = 0
return False
async def release(self):
"""Release the lock."""
async with self._refresh_lock:
if self._last_refresh == 0:
return
try:
await self.redis.delete(self.key)
except Exception:
pass
async with self._refresh_lock:
self._last_refresh = 0.0

View File

@@ -11,11 +11,6 @@ import {
MessageResponse,
} from "@/components/ai-elements/message";
import { Text } from "@/components/atoms/Text/Text";
import {
CredentialsProvidersContext,
type CredentialsProviderData,
type CredentialsProvidersContextType,
} from "@/providers/agent-credentials/credentials-provider";
import { CopilotChatActionsProvider } from "../components/CopilotChatActionsProvider/CopilotChatActionsProvider";
import { CreateAgentTool } from "../tools/CreateAgent/CreateAgent";
import { EditAgentTool } from "../tools/EditAgent/EditAgent";
@@ -102,65 +97,6 @@ function uid() {
return `sg-${++_id}`;
}
// ---------------------------------------------------------------------------
// Mock credential providers for setup-requirements demos
// ---------------------------------------------------------------------------
const noop = () => Promise.reject(new Error("Styleguide mock"));
function makeMockProvider(
provider: string,
providerName: string,
savedCredentials: CredentialsProviderData["savedCredentials"] = [],
): CredentialsProviderData {
return {
provider,
providerName,
savedCredentials,
isSystemProvider: false,
oAuthCallback: noop as CredentialsProviderData["oAuthCallback"],
mcpOAuthCallback: noop as CredentialsProviderData["mcpOAuthCallback"],
createAPIKeyCredentials:
noop as CredentialsProviderData["createAPIKeyCredentials"],
createUserPasswordCredentials:
noop as CredentialsProviderData["createUserPasswordCredentials"],
createHostScopedCredentials:
noop as CredentialsProviderData["createHostScopedCredentials"],
deleteCredentials: noop as CredentialsProviderData["deleteCredentials"],
};
}
/**
* Provider context where the user already has saved credentials
* so the credential picker shows a selection list.
*/
const MOCK_PROVIDERS_WITH_CREDENTIALS: CredentialsProvidersContextType = {
google: makeMockProvider("google", "Google", [
{
id: "cred-google-1",
provider: "google",
type: "oauth2",
title: "work@company.com",
scopes: ["email", "calendar"],
},
{
id: "cred-google-2",
provider: "google",
type: "oauth2",
title: "personal@gmail.com",
scopes: ["email", "calendar"],
},
]),
};
/**
* Provider context where the user has NO saved credentials,
* so the credential picker shows an "add new" flow.
*/
const MOCK_PROVIDERS_WITHOUT_CREDENTIALS: CredentialsProvidersContextType = {
openweathermap: makeMockProvider("openweathermap", "OpenWeatherMap"),
};
// ---------------------------------------------------------------------------
// Page
// ---------------------------------------------------------------------------
@@ -618,80 +554,45 @@ export default function StyleguidePage() {
/>
</SubSection>
<SubSection label="Setup requirements — no credentials (add new)">
<CredentialsProvidersContext.Provider
value={MOCK_PROVIDERS_WITHOUT_CREDENTIALS}
>
<RunBlockTool
part={{
type: "tool-run_block",
toolCallId: uid(),
state: "output-available",
input: { block_id: "weather-block-123" },
output: {
type: ResponseType.setup_requirements,
message:
"This block requires API credentials to run. Please configure them below.",
setup_info: {
agent_id: "agent-weather-1",
agent_name: "Weather Agent",
requirements: {
inputs: [
{
name: "city",
title: "City",
type: "string",
required: true,
description: "The city to get weather for",
},
],
},
user_readiness: {
missing_credentials: {
openweathermap_key: {
provider: "openweathermap",
types: ["api_key"],
},
<SubSection label="Output available (setup requirements)">
<RunBlockTool
part={{
type: "tool-run_block",
toolCallId: uid(),
state: "output-available",
input: { block_id: "weather-block-123" },
output: {
type: ResponseType.setup_requirements,
message:
"This block requires API credentials to run. Please configure them below.",
setup_info: {
agent_name: "Weather Agent",
requirements: {
inputs: [
{
name: "city",
title: "City",
type: "string",
required: true,
description: "The city to get weather for",
},
],
},
user_readiness: {
missing_credentials: {
openweathermap: {
provider: "openweathermap",
credentials_type: "api_key",
title: "OpenWeatherMap API Key",
description:
"Required to access weather data. Get your key at openweathermap.org",
},
},
},
},
}}
/>
</CredentialsProvidersContext.Provider>
</SubSection>
<SubSection label="Setup requirements — has credentials (pick from list)">
<CredentialsProvidersContext.Provider
value={MOCK_PROVIDERS_WITH_CREDENTIALS}
>
<RunBlockTool
part={{
type: "tool-run_block",
toolCallId: uid(),
state: "output-available",
input: { block_id: "calendar-block-456" },
output: {
type: ResponseType.setup_requirements,
message:
"This block requires Google credentials. Pick an account below or connect a new one.",
setup_info: {
agent_id: "agent-calendar-1",
agent_name: "Calendar Agent",
user_readiness: {
missing_credentials: {
google_oauth: {
provider: "google",
types: ["oauth2"],
scopes: ["email", "calendar"],
},
},
},
},
},
}}
/>
</CredentialsProvidersContext.Provider>
},
}}
/>
</SubSection>
<SubSection label="Output available (error)">
@@ -948,71 +849,34 @@ export default function StyleguidePage() {
/>
</SubSection>
<SubSection label="Setup requirements — no credentials (add new)">
<CredentialsProvidersContext.Provider
value={MOCK_PROVIDERS_WITHOUT_CREDENTIALS}
>
<RunAgentTool
part={{
type: "tool-run_agent",
toolCallId: uid(),
state: "output-available",
input: { username_agent_slug: "creator/weather-agent" },
output: {
type: ResponseType.setup_requirements,
message:
"This agent requires an API key. Add your credentials below.",
setup_info: {
agent_id: "agent-weather-1",
agent_name: "Weather Agent",
requirements: {},
user_readiness: {
missing_credentials: {
openweathermap_key: {
provider: "openweathermap",
types: ["api_key"],
},
<SubSection label="Output available (setup requirements)">
<RunAgentTool
part={{
type: "tool-run_agent",
toolCallId: uid(),
state: "output-available",
input: { username_agent_slug: "creator/my-agent" },
output: {
type: ResponseType.setup_requirements,
message: "This agent requires additional setup.",
setup_info: {
agent_name: "YouTube Summarizer",
requirements: {},
user_readiness: {
missing_credentials: {
youtube_api: {
provider: "youtube",
credentials_type: "api_key",
title: "YouTube Data API Key",
description:
"Required to access YouTube video data.",
},
},
},
},
}}
/>
</CredentialsProvidersContext.Provider>
</SubSection>
<SubSection label="Setup requirements — has credentials (pick from list)">
<CredentialsProvidersContext.Provider
value={MOCK_PROVIDERS_WITH_CREDENTIALS}
>
<RunAgentTool
part={{
type: "tool-run_agent",
toolCallId: uid(),
state: "output-available",
input: { username_agent_slug: "creator/calendar-agent" },
output: {
type: ResponseType.setup_requirements,
message:
"This agent needs Google credentials. Pick an account or connect a new one.",
setup_info: {
agent_id: "agent-calendar-1",
agent_name: "Google Calendar Agent",
requirements: {},
user_readiness: {
missing_credentials: {
google_oauth: {
provider: "google",
types: ["oauth2"],
scopes: ["email", "calendar"],
},
},
},
},
},
}}
/>
</CredentialsProvidersContext.Provider>
},
}}
/>
</SubSection>
<SubSection label="Output available (need login)">

View File

@@ -16,6 +16,7 @@ import {
ContentCardDescription,
ContentCodeBlock,
ContentGrid,
ContentHint,
ContentMessage,
} from "../../components/ToolAccordion/AccordionContent";
import { ToolAccordion } from "../../components/ToolAccordion/ToolAccordion";
@@ -23,8 +24,8 @@ import {
ClarificationQuestionsCard,
ClarifyingQuestion,
} from "./components/ClarificationQuestionsCard";
import sparklesImg from "../../components/MiniGame/assets/sparkles.png";
import { MiniGame } from "../../components/MiniGame/MiniGame";
import sparklesImg from "./components/MiniGame/assets/sparkles.png";
import { MiniGame } from "./components/MiniGame/MiniGame";
import { SuggestedGoalCard } from "./components/SuggestedGoalCard";
import {
AccordionIcon,
@@ -92,7 +93,9 @@ function getAccordionMeta(output: CreateAgentToolOutput) {
) {
return {
icon,
title: output.message || "Agent creation started",
title:
"Creating agent, this may take a few minutes. Play while you wait.",
expanded: true,
};
}
return {
@@ -166,22 +169,15 @@ export function CreateAgentTool({ part }: Props) {
/>
</div>
{isStreaming && (
<ToolAccordion
icon={<AccordionIcon />}
title="Creating agent, this may take a few minutes. Play while you wait."
expanded
>
<ContentGrid>
<MiniGame />
</ContentGrid>
</ToolAccordion>
)}
{hasExpandableContent && output && (
<ToolAccordion {...getAccordionMeta(output)}>
{isOperating && output.message && (
<ContentMessage>{output.message}</ContentMessage>
{isOperating && (
<ContentGrid>
<MiniGame />
<ContentHint>
This could take a few minutes play while you wait!
</ContentHint>
</ContentGrid>
)}
{isAgentSavedOutput(output) && (

View File

@@ -4,15 +4,17 @@ import { WarningDiamondIcon } from "@phosphor-icons/react";
import type { ToolUIPart } from "ai";
import { useCopilotChatActions } from "../../components/CopilotChatActionsProvider/useCopilotChatActions";
import { MorphingTextAnimation } from "../../components/MorphingTextAnimation/MorphingTextAnimation";
import { OrbitLoader } from "../../components/OrbitLoader/OrbitLoader";
import {
ContentCardDescription,
ContentCodeBlock,
ContentGrid,
ContentHint,
ContentLink,
ContentMessage,
} from "../../components/ToolAccordion/AccordionContent";
import { ToolAccordion } from "../../components/ToolAccordion/ToolAccordion";
import { MiniGame } from "../../components/MiniGame/MiniGame";
import { MiniGame } from "../CreateAgent/components/MiniGame/MiniGame";
import {
ClarificationQuestionsCard,
ClarifyingQuestion,
@@ -79,8 +81,9 @@ function getAccordionMeta(output: EditAgentToolOutput): {
isOperationInProgressOutput(output)
) {
return {
icon,
title: output.message || "Agent editing started",
icon: <OrbitLoader size={32} />,
title: "Editing agent, this may take a few minutes. Play while you wait.",
expanded: true,
};
}
return {
@@ -145,22 +148,15 @@ export function EditAgentTool({ part }: Props) {
/>
</div>
{isStreaming && (
<ToolAccordion
icon={<AccordionIcon />}
title="Editing agent, this may take a few minutes. Play while you wait."
expanded
>
<ContentGrid>
<MiniGame />
</ContentGrid>
</ToolAccordion>
)}
{hasExpandableContent && output && (
<ToolAccordion {...getAccordionMeta(output)}>
{isOperating && output.message && (
<ContentMessage>{output.message}</ContentMessage>
{isOperating && (
<ContentGrid>
<MiniGame />
<ContentHint>
This could take a few minutes play while you wait!
</ContentHint>
</ContentGrid>
)}
{isAgentSavedOutput(output) && (

View File

@@ -9,7 +9,7 @@ import {
ContentHint,
ContentMessage,
} from "../../components/ToolAccordion/AccordionContent";
import { MiniGame } from "../../components/MiniGame/MiniGame";
import { MiniGame } from "../CreateAgent/components/MiniGame/MiniGame";
import {
getAccordionMeta,
getAnimationText,
@@ -47,25 +47,14 @@ export function RunAgentTool({ part }: Props) {
const isError =
part.state === "output-error" ||
(!!output && isRunAgentErrorOutput(output));
const isOutputAvailable = part.state === "output-available" && !!output;
const setupRequirementsOutput =
isOutputAvailable && isRunAgentSetupRequirementsOutput(output)
? output
: null;
const agentDetailsOutput =
isOutputAvailable && isRunAgentAgentDetailsOutput(output) ? output : null;
const needLoginOutput =
isOutputAvailable && isRunAgentNeedLoginOutput(output) ? output : null;
const hasExpandableContent =
isOutputAvailable &&
!setupRequirementsOutput &&
!agentDetailsOutput &&
!needLoginOutput &&
(isRunAgentExecutionStartedOutput(output) || isRunAgentErrorOutput(output));
part.state === "output-available" &&
!!output &&
(isRunAgentExecutionStartedOutput(output) ||
isRunAgentAgentDetailsOutput(output) ||
isRunAgentSetupRequirementsOutput(output) ||
isRunAgentNeedLoginOutput(output) ||
isRunAgentErrorOutput(output));
return (
<div className="py-2">
@@ -92,30 +81,24 @@ export function RunAgentTool({ part }: Props) {
</ToolAccordion>
)}
{setupRequirementsOutput && (
<div className="mt-2">
<SetupRequirementsCard output={setupRequirementsOutput} />
</div>
)}
{agentDetailsOutput && (
<div className="mt-2">
<AgentDetailsCard output={agentDetailsOutput} />
</div>
)}
{needLoginOutput && (
<div className="mt-2">
<ContentMessage>{needLoginOutput.message}</ContentMessage>
</div>
)}
{hasExpandableContent && output && (
<ToolAccordion {...getAccordionMeta(output)}>
{isRunAgentExecutionStartedOutput(output) && (
<ExecutionStartedCard output={output} />
)}
{isRunAgentAgentDetailsOutput(output) && (
<AgentDetailsCard output={output} />
)}
{isRunAgentSetupRequirementsOutput(output) && (
<SetupRequirementsCard output={output} />
)}
{isRunAgentNeedLoginOutput(output) && (
<ContentMessage>{output.message}</ContentMessage>
)}
{isRunAgentErrorOutput(output) && <ErrorCard output={output} />}
</ToolAccordion>
)}

View File

@@ -1,11 +1,10 @@
"use client";
import type { SetupRequirementsResponse } from "@/app/api/__generated__/models/setupRequirementsResponse";
import { Button } from "@/components/atoms/Button/Button";
import { Text } from "@/components/atoms/Text/Text";
import { CredentialsGroupedView } from "@/components/contextual/CredentialsInput/components/CredentialsGroupedView/CredentialsGroupedView";
import type { CredentialsMetaInput } from "@/lib/autogpt-server-api/types";
import { useState } from "react";
import { CredentialsGroupedView } from "@/components/contextual/CredentialsInput/components/CredentialsGroupedView/CredentialsGroupedView";
import { Button } from "@/components/atoms/Button/Button";
import type { CredentialsMetaInput } from "@/lib/autogpt-server-api/types";
import type { SetupRequirementsResponse } from "@/app/api/__generated__/models/setupRequirementsResponse";
import { useCopilotChatActions } from "../../../../components/CopilotChatActionsProvider/useCopilotChatActions";
import {
ContentBadge,
@@ -39,40 +38,40 @@ export function SetupRequirementsCard({ output }: Props) {
setInputCredentials((prev) => ({ ...prev, [key]: value }));
}
const needsCredentials = credentialFields.length > 0;
const isAllCredentialsComplete =
needsCredentials &&
const isAllComplete =
credentialFields.length > 0 &&
[...requiredCredentials].every((key) => !!inputCredentials[key]);
const canProceed =
!hasSent && (!needsCredentials || isAllCredentialsComplete);
function handleProceed() {
setHasSent(true);
const message = needsCredentials
? "I've configured the required credentials. Please check if everything is ready and proceed with running the agent."
: "Please proceed with running the agent.";
onSend(message);
onSend(
"I've configured the required credentials. Please check if everything is ready and proceed with running the agent.",
);
}
return (
<div className="grid gap-2">
<ContentMessage>{output.message}</ContentMessage>
{needsCredentials && (
{credentialFields.length > 0 && (
<div className="rounded-2xl border bg-background p-3">
<Text variant="small" className="w-fit border-b text-zinc-500">
Agent credentials
</Text>
<div className="mt-6">
<CredentialsGroupedView
credentialFields={credentialFields}
requiredCredentials={requiredCredentials}
inputCredentials={inputCredentials}
inputValues={{}}
onCredentialChange={handleCredentialChange}
/>
</div>
<CredentialsGroupedView
credentialFields={credentialFields}
requiredCredentials={requiredCredentials}
inputCredentials={inputCredentials}
inputValues={{}}
onCredentialChange={handleCredentialChange}
/>
{isAllComplete && !hasSent && (
<Button
variant="primary"
size="small"
className="mt-3 w-full"
onClick={handleProceed}
>
Proceed
</Button>
)}
</div>
)}
@@ -101,18 +100,6 @@ export function SetupRequirementsCard({ output }: Props) {
</div>
</div>
)}
{(needsCredentials || expectedInputs.length > 0) && (
<Button
variant="primary"
size="small"
className="mt-4 w-fit"
disabled={!canProceed}
onClick={handleProceed}
>
Proceed
</Button>
)}
</div>
);
}

View File

@@ -39,19 +39,12 @@ export function RunBlockTool({ part }: Props) {
const isError =
part.state === "output-error" ||
(!!output && isRunBlockErrorOutput(output));
const setupRequirementsOutput =
part.state === "output-available" &&
output &&
isRunBlockSetupRequirementsOutput(output)
? output
: null;
const hasExpandableContent =
part.state === "output-available" &&
!!output &&
!setupRequirementsOutput &&
(isRunBlockBlockOutput(output) ||
isRunBlockDetailsOutput(output) ||
isRunBlockSetupRequirementsOutput(output) ||
isRunBlockErrorOutput(output));
return (
@@ -64,12 +57,6 @@ export function RunBlockTool({ part }: Props) {
/>
</div>
{setupRequirementsOutput && (
<div className="mt-2">
<SetupRequirementsCard output={setupRequirementsOutput} />
</div>
)}
{hasExpandableContent && output && (
<ToolAccordion {...getAccordionMeta(output)}>
{isRunBlockBlockOutput(output) && <BlockOutputCard output={output} />}
@@ -78,6 +65,10 @@ export function RunBlockTool({ part }: Props) {
<BlockDetailsCard output={output} />
)}
{isRunBlockSetupRequirementsOutput(output) && (
<SetupRequirementsCard output={output} />
)}
{isRunBlockErrorOutput(output) && <ErrorCard output={output} />}
</ToolAccordion>
)}

View File

@@ -6,9 +6,15 @@ import { Text } from "@/components/atoms/Text/Text";
import { CredentialsGroupedView } from "@/components/contextual/CredentialsInput/components/CredentialsGroupedView/CredentialsGroupedView";
import { FormRenderer } from "@/components/renderers/InputRenderer/FormRenderer";
import type { CredentialsMetaInput } from "@/lib/autogpt-server-api/types";
import { AnimatePresence, motion } from "framer-motion";
import { useState } from "react";
import { useCopilotChatActions } from "../../../../components/CopilotChatActionsProvider/useCopilotChatActions";
import { ContentMessage } from "../../../../components/ToolAccordion/AccordionContent";
import {
ContentBadge,
ContentCardDescription,
ContentCardTitle,
ContentMessage,
} from "../../../../components/ToolAccordion/AccordionContent";
import {
buildExpectedInputsSchema,
coerceCredentialFields,
@@ -25,8 +31,10 @@ export function SetupRequirementsCard({ output }: Props) {
const [inputCredentials, setInputCredentials] = useState<
Record<string, CredentialsMetaInput | undefined>
>({});
const [hasSentCredentials, setHasSentCredentials] = useState(false);
const [showInputForm, setShowInputForm] = useState(false);
const [inputValues, setInputValues] = useState<Record<string, unknown>>({});
const [hasSent, setHasSent] = useState(false);
const { credentialFields, requiredCredentials } = coerceCredentialFields(
output.setup_info.user_readiness?.missing_credentials,
@@ -42,49 +50,27 @@ export function SetupRequirementsCard({ output }: Props) {
setInputCredentials((prev) => ({ ...prev, [key]: value }));
}
const needsCredentials = credentialFields.length > 0;
const isAllCredentialsComplete =
needsCredentials &&
credentialFields.length > 0 &&
[...requiredCredentials].every((key) => !!inputCredentials[key]);
const needsInputs = inputSchema !== null;
const requiredInputNames = expectedInputs
.filter((i) => i.required)
.map((i) => i.name);
const isAllInputsComplete =
needsInputs &&
requiredInputNames.every((name) => {
const v = inputValues[name];
return v !== undefined && v !== null && v !== "";
});
function handleProceedCredentials() {
setHasSentCredentials(true);
onSend(
"I've configured the required credentials. Please re-run the block now.",
);
}
const canRun =
!hasSent &&
(!needsCredentials || isAllCredentialsComplete) &&
(!needsInputs || isAllInputsComplete);
function handleRun() {
setHasSent(true);
const parts: string[] = [];
if (needsCredentials) {
parts.push("I've configured the required credentials.");
}
if (needsInputs) {
const nonEmpty = Object.fromEntries(
Object.entries(inputValues).filter(
([, v]) => v !== undefined && v !== null && v !== "",
),
);
parts.push(
`Run the block with these inputs: ${JSON.stringify(nonEmpty, null, 2)}`,
);
} else {
parts.push("Please re-run the block now.");
}
onSend(parts.join(" "));
function handleRunWithInputs() {
const nonEmpty = Object.fromEntries(
Object.entries(inputValues).filter(
([, v]) => v !== undefined && v !== null && v !== "",
),
);
onSend(
`Run the block with these inputs: ${JSON.stringify(nonEmpty, null, 2)}`,
);
setShowInputForm(false);
setInputValues({});
}
@@ -92,54 +78,119 @@ export function SetupRequirementsCard({ output }: Props) {
<div className="grid gap-2">
<ContentMessage>{output.message}</ContentMessage>
{needsCredentials && (
{credentialFields.length > 0 && (
<div className="rounded-2xl border bg-background p-3">
<Text variant="small" className="w-fit border-b text-zinc-500">
Block credentials
</Text>
<div className="mt-6">
<CredentialsGroupedView
credentialFields={credentialFields}
requiredCredentials={requiredCredentials}
inputCredentials={inputCredentials}
inputValues={{}}
onCredentialChange={handleCredentialChange}
/>
</div>
<CredentialsGroupedView
credentialFields={credentialFields}
requiredCredentials={requiredCredentials}
inputCredentials={inputCredentials}
inputValues={{}}
onCredentialChange={handleCredentialChange}
/>
{isAllCredentialsComplete && !hasSentCredentials && (
<Button
variant="primary"
size="small"
className="mt-3 w-full"
onClick={handleProceedCredentials}
>
Proceed
</Button>
)}
</div>
)}
{inputSchema && (
<div className="rounded-2xl border bg-background p-3 pt-4">
<Text variant="small" className="w-fit border-b text-zinc-500">
Block inputs
</Text>
<FormRenderer
jsonSchema={inputSchema}
className="mb-3 mt-3"
handleChange={(v) => setInputValues(v.formData ?? {})}
uiSchema={{
"ui:submitButtonOptions": { norender: true },
}}
initialValues={inputValues}
formContext={{
showHandles: false,
size: "small",
}}
/>
<div className="flex gap-2 pt-2">
<Button
variant="outline"
size="small"
className="w-fit"
onClick={() => setShowInputForm((prev) => !prev)}
>
{showInputForm ? "Hide inputs" : "Fill in inputs"}
</Button>
</div>
)}
{(needsCredentials || needsInputs) && (
<Button
variant="primary"
size="small"
className="w-fit"
disabled={!canRun}
onClick={handleRun}
>
Proceed
</Button>
<AnimatePresence initial={false}>
{showInputForm && inputSchema && (
<motion.div
initial={{ height: 0, opacity: 0, filter: "blur(6px)" }}
animate={{ height: "auto", opacity: 1, filter: "blur(0px)" }}
exit={{ height: 0, opacity: 0, filter: "blur(6px)" }}
transition={{
height: { type: "spring", bounce: 0.15, duration: 0.5 },
opacity: { duration: 0.25 },
filter: { duration: 0.2 },
}}
className="overflow-hidden"
style={{ willChange: "height, opacity, filter" }}
>
<div className="rounded-2xl border bg-background p-3 pt-4">
<Text variant="body-medium">Block inputs</Text>
<FormRenderer
jsonSchema={inputSchema}
handleChange={(v) => setInputValues(v.formData ?? {})}
uiSchema={{
"ui:submitButtonOptions": { norender: true },
}}
initialValues={inputValues}
formContext={{
showHandles: false,
size: "small",
}}
/>
<div className="-mt-8 flex gap-2">
<Button
variant="primary"
size="small"
className="w-fit"
onClick={handleRunWithInputs}
>
Run
</Button>
<Button
variant="secondary"
size="small"
className="w-fit"
onClick={() => {
setShowInputForm(false);
setInputValues({});
}}
>
Cancel
</Button>
</div>
</div>
</motion.div>
)}
</AnimatePresence>
{expectedInputs.length > 0 && !inputSchema && (
<div className="rounded-2xl border bg-background p-3">
<ContentCardTitle className="text-xs">
Expected inputs
</ContentCardTitle>
<div className="mt-2 grid gap-2">
{expectedInputs.map((input) => (
<div key={input.name} className="rounded-xl border p-2">
<div className="flex items-center justify-between gap-2">
<ContentCardTitle className="text-xs">
{input.title}
</ContentCardTitle>
<ContentBadge>
{input.required ? "Required" : "Optional"}
</ContentBadge>
</div>
<ContentCardDescription className="mt-1">
{input.name} &bull; {input.type}
{input.description ? ` \u2022 ${input.description}` : ""}
</ContentCardDescription>
</div>
))}
</div>
</div>
)}
</div>
);

View File

@@ -119,7 +119,7 @@ export function CredentialsFlatView({
) : (
!readOnly && (
<Button
variant="primary"
variant="secondary"
size="small"
onClick={onAddCredential}
className="w-fit"

View File

@@ -1,11 +1,10 @@
import { cn } from "@/lib/utils";
import { RJSFSchema } from "@rjsf/utils";
import { preprocessInputSchema } from "./utils/input-schema-pre-processor";
import { useMemo } from "react";
import { customValidator } from "./utils/custom-validator";
import Form from "./registry";
import { ExtendedFormContextType } from "./types";
import { customValidator } from "./utils/custom-validator";
import { generateUiSchemaForCustomFields } from "./utils/generate-ui-schema";
import { preprocessInputSchema } from "./utils/input-schema-pre-processor";
type FormRendererProps = {
jsonSchema: RJSFSchema;
@@ -13,17 +12,15 @@ type FormRendererProps = {
uiSchema: any;
initialValues: any;
formContext: ExtendedFormContextType;
className?: string;
};
export function FormRenderer({
export const FormRenderer = ({
jsonSchema,
handleChange,
uiSchema,
initialValues,
formContext,
className,
}: FormRendererProps) {
}: FormRendererProps) => {
const preprocessedSchema = useMemo(() => {
return preprocessInputSchema(jsonSchema);
}, [jsonSchema]);
@@ -34,10 +31,7 @@ export function FormRenderer({
}, [preprocessedSchema, uiSchema]);
return (
<div
className={cn("mb-6 mt-4", className)}
data-tutorial-id="input-handles"
>
<div className={"mb-6 mt-4"} data-tutorial-id="input-handles">
<Form
formContext={formContext}
idPrefix="agpt"
@@ -51,4 +45,4 @@ export function FormRenderer({
/>
</div>
);
}
};

View File

@@ -15,7 +15,6 @@
## Advanced Setup
* [Advanced Setup](advanced_setup.md)
* [Deployment Environment Variables](deployment-environment-variables.md)
## Building Blocks

View File

@@ -1,397 +0,0 @@
# Deployment Environment Variables
This guide documents **all environment variables that must be configured** when deploying AutoGPT to a new server or environment. Use this as a checklist to ensure your deployment works correctly.
## Quick Reference: What MUST Change
When deploying to a new server, these variables **must** be updated from their localhost defaults:
| Variable | Location | Default | Purpose |
|----------|----------|---------|---------|
| `SITE_URL` | `.env` | `http://localhost:3000` | Frontend URL for auth redirects |
| `API_EXTERNAL_URL` | `.env` | `http://localhost:8000` | Public Supabase API URL |
| `SUPABASE_PUBLIC_URL` | `.env` | `http://localhost:8000` | Studio dashboard URL |
| `PLATFORM_BASE_URL` | `backend/.env` | `http://localhost:8000` | Backend platform URL |
| `FRONTEND_BASE_URL` | `backend/.env` | `http://localhost:3000` | Frontend URL for webhooks/OAuth |
| `NEXT_PUBLIC_SUPABASE_URL` | `frontend/.env` | `http://localhost:8000` | Client-side Supabase URL |
| `NEXT_PUBLIC_AGPT_SERVER_URL` | `frontend/.env` | `http://localhost:8006/api` | Client-side backend API URL |
| `NEXT_PUBLIC_AGPT_WS_SERVER_URL` | `frontend/.env` | `ws://localhost:8001/ws` | Client-side WebSocket URL |
| `NEXT_PUBLIC_FRONTEND_BASE_URL` | `frontend/.env` | `http://localhost:3000` | Client-side frontend URL |
---
## Configuration Files
AutoGPT uses multiple `.env` files across different components:
```text
autogpt_platform/
├── .env # Supabase/infrastructure config
├── backend/
│ ├── .env.default # Backend defaults (DO NOT EDIT)
│ └── .env # Your backend overrides
└── frontend/
├── .env.default # Frontend defaults (DO NOT EDIT)
└── .env # Your frontend overrides
```
**Loading Order** (later overrides earlier):
1. `*.env.default` - Base defaults
2. `*.env` - Your overrides
3. Docker `environment:` section
4. Shell environment variables
---
## 1. URL Configuration (REQUIRED)
These URLs must be updated to match your deployment domain/IP.
### Root `.env` (Supabase)
```bash
# Auth redirects - where users return after login
SITE_URL=https://your-domain.com:3000
# Public API URL - exposed to clients
API_EXTERNAL_URL=https://your-domain.com:8000
# Studio dashboard URL
SUPABASE_PUBLIC_URL=https://your-domain.com:8000
```
### Backend `.env`
```bash
# Platform URLs for webhooks and OAuth callbacks
PLATFORM_BASE_URL=https://your-domain.com:8000
FRONTEND_BASE_URL=https://your-domain.com:3000
# Internal Supabase URL (use Docker service name if containerized)
SUPABASE_URL=http://kong:8000 # Docker
# SUPABASE_URL=https://your-domain.com:8000 # External
```
### Frontend `.env`
```bash
# Client-side URLs (used in browser)
NEXT_PUBLIC_SUPABASE_URL=https://your-domain.com:8000
NEXT_PUBLIC_AGPT_SERVER_URL=https://your-domain.com:8006/api
NEXT_PUBLIC_AGPT_WS_SERVER_URL=wss://your-domain.com:8001/ws
NEXT_PUBLIC_FRONTEND_BASE_URL=https://your-domain.com:3000
```
!!! warning "HTTPS Note"
For production, use HTTPS URLs and `wss://` for WebSocket. You'll need a reverse proxy (nginx, Caddy) with SSL certificates.
!!! info "Port Numbers"
The port numbers shown (`:3000`, `:8000`, `:8001`, `:8006`) are internal Docker service ports. In production with a reverse proxy, your public URLs typically won't include port numbers (e.g., `https://your-domain.com` instead of `https://your-domain.com:3000`). Configure your reverse proxy to route external traffic to the internal service ports.
---
## 2. Security Keys (MUST REGENERATE)
These default values are **public** and **must be changed** for production.
### Root `.env`
```bash
# Database password
POSTGRES_PASSWORD=<generate-strong-password>
# JWT secret for Supabase auth (min 32 chars)
JWT_SECRET=<generate-random-string>
# Supabase keys (regenerate with matching JWT_SECRET)
ANON_KEY=<regenerate>
SERVICE_ROLE_KEY=<regenerate>
# Studio dashboard credentials
DASHBOARD_USERNAME=<your-username>
DASHBOARD_PASSWORD=<strong-password>
# Encryption keys
SECRET_KEY_BASE=<generate-random-string>
VAULT_ENC_KEY=<generate-32-char-key> # Run: openssl rand -hex 16
```
### Backend `.env`
```bash
# Must match root POSTGRES_PASSWORD
DB_PASS=<same-as-POSTGRES_PASSWORD>
# Must match root SERVICE_ROLE_KEY
SUPABASE_SERVICE_ROLE_KEY=<same-as-SERVICE_ROLE_KEY>
# Must match root JWT_SECRET
JWT_VERIFY_KEY=<same-as-JWT_SECRET>
# Generate new encryption keys
# Run: python -c "from cryptography.fernet import Fernet;print(Fernet.generate_key().decode())"
ENCRYPTION_KEY=<generated-fernet-key>
UNSUBSCRIBE_SECRET_KEY=<generated-fernet-key>
```
### Generating Keys
```bash
# Generate Fernet encryption key (for ENCRYPTION_KEY, UNSUBSCRIBE_SECRET_KEY)
python -c "from cryptography.fernet import Fernet;print(Fernet.generate_key().decode())"
# Generate random string (for JWT_SECRET, SECRET_KEY_BASE)
openssl rand -base64 32
# Generate 32-character key (for VAULT_ENC_KEY)
openssl rand -hex 16
# Generate Supabase keys (requires matching JWT_SECRET)
# Use: https://supabase.com/docs/guides/self-hosting/docker#generate-api-keys
```
---
## 3. Database Configuration
### Root `.env`
```bash
POSTGRES_HOST=db # Docker service name or external host
POSTGRES_DB=postgres
POSTGRES_PORT=5432
POSTGRES_PASSWORD=<your-password>
```
### Backend `.env`
```bash
DB_USER=postgres
DB_PASS=<your-password>
DB_NAME=postgres
DB_PORT=5432
DB_HOST=localhost # Default is localhost; use 'db' in Docker
DB_SCHEMA=platform
# Connection pooling
DB_CONNECTION_LIMIT=12
DB_CONNECT_TIMEOUT=60
DB_POOL_TIMEOUT=300
# Full connection URL (auto-constructed from above in .env.default)
# Variable substitution is handled automatically; only override if you need custom parameters
DATABASE_URL="postgresql://${DB_USER}:${DB_PASS}@${DB_HOST}:${DB_PORT}/${DB_NAME}?schema=${DB_SCHEMA}"
```
---
## 4. Service Dependencies
### Redis
```bash
REDIS_HOST=redis # Docker: 'redis', External: hostname/IP
REDIS_PORT=6379
# REDIS_PASSWORD= # Uncomment if using authentication
```
### RabbitMQ
```bash
RABBITMQ_DEFAULT_USER=<username>
RABBITMQ_DEFAULT_PASS=<strong-password>
# In Docker, host is 'rabbitmq'
```
---
## 5. Default Ports
| Service | Port | Purpose |
|---------|------|---------|
| Frontend | 3000 | Next.js web UI |
| Kong (Supabase API) | 8000 | API gateway |
| WebSocket Server | 8001 | Real-time updates |
| Executor | 8002 | Agent execution |
| Scheduler | 8003 | Scheduled tasks |
| Database Manager | 8005 | DB operations |
| REST Server | 8006 | Main API |
| Notification Server | 8007 | Notifications |
| PostgreSQL | 5432 | Database |
| Redis | 6379 | Cache/queue |
| RabbitMQ | 5672/15672 | Message queue |
| ClamAV | 3310 | Antivirus scanning |
---
## 6. OAuth Callbacks
When configuring OAuth providers, use this callback URL format:
```text
https://your-domain.com/auth/integrations/oauth_callback
# Or with explicit port if not using a reverse proxy:
# https://your-domain.com:3000/auth/integrations/oauth_callback
```
### Supported OAuth Providers
| Provider | Env Variables | Setup URL |
|----------|---------------|-----------|
| GitHub | `GITHUB_CLIENT_ID`, `GITHUB_CLIENT_SECRET` | [github.com/settings/developers](https://github.com/settings/developers) |
| Google | `GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET` | [console.cloud.google.com](https://console.cloud.google.com/apis/credentials) |
| Discord | `DISCORD_CLIENT_ID`, `DISCORD_CLIENT_SECRET` | [discord.com/developers](https://discord.com/developers/applications) |
| Twitter/X | `TWITTER_CLIENT_ID`, `TWITTER_CLIENT_SECRET` | [developer.x.com](https://developer.x.com) |
| Notion | `NOTION_CLIENT_ID`, `NOTION_CLIENT_SECRET` | [developers.notion.com](https://developers.notion.com) |
| Linear | `LINEAR_CLIENT_ID`, `LINEAR_CLIENT_SECRET` | [linear.app/settings/api](https://linear.app/settings/api/applications/new) |
| Reddit | `REDDIT_CLIENT_ID`, `REDDIT_CLIENT_SECRET` | [reddit.com/prefs/apps](https://reddit.com/prefs/apps) |
| Todoist | `TODOIST_CLIENT_ID`, `TODOIST_CLIENT_SECRET` | [developer.todoist.com](https://developer.todoist.com/appconsole.html) |
---
## 7. Optional Services
### AI/LLM Providers
```bash
OPENAI_API_KEY=
ANTHROPIC_API_KEY=
GROQ_API_KEY=
OPEN_ROUTER_API_KEY=
NVIDIA_API_KEY=
```
### Email (SMTP)
```bash
# Supabase auth emails
SMTP_HOST=smtp.example.com
SMTP_PORT=587
SMTP_USER=<username>
SMTP_PASS=<password>
SMTP_ADMIN_EMAIL=admin@example.com
# Application emails (Postmark)
POSTMARK_SERVER_API_TOKEN=
POSTMARK_SENDER_EMAIL=noreply@your-domain.com
```
### Payments (Stripe)
```bash
STRIPE_API_KEY=
STRIPE_WEBHOOK_SECRET=
```
### Error Tracking (Sentry)
```bash
SENTRY_DSN=
```
### Analytics (PostHog)
```bash
POSTHOG_API_KEY=
POSTHOG_HOST=https://eu.i.posthog.com
# Frontend
NEXT_PUBLIC_POSTHOG_KEY=
NEXT_PUBLIC_POSTHOG_HOST=https://eu.i.posthog.com
```
---
## 8. Deployment Checklist
Use this checklist when deploying to a new environment:
### Pre-deployment
- [ ] Clone repository and navigate to `autogpt_platform/`
- [ ] Copy all `.env.default` files to `.env`
- [ ] Determine your deployment domain/IP
### URL Configuration
- [ ] Update `SITE_URL` in root `.env`
- [ ] Update `API_EXTERNAL_URL` in root `.env`
- [ ] Update `SUPABASE_PUBLIC_URL` in root `.env`
- [ ] Update `PLATFORM_BASE_URL` in `backend/.env`
- [ ] Update `FRONTEND_BASE_URL` in `backend/.env`
- [ ] Update all `NEXT_PUBLIC_*` URLs in `frontend/.env`
### Security
- [ ] Generate new `POSTGRES_PASSWORD`
- [ ] Generate new `JWT_SECRET` (min 32 chars)
- [ ] Regenerate `ANON_KEY` and `SERVICE_ROLE_KEY`
- [ ] Change `DASHBOARD_USERNAME` and `DASHBOARD_PASSWORD`
- [ ] Generate new `ENCRYPTION_KEY` (backend)
- [ ] Generate new `UNSUBSCRIBE_SECRET_KEY` (backend)
- [ ] Update `DB_PASS` to match `POSTGRES_PASSWORD`
- [ ] Update `JWT_VERIFY_KEY` to match `JWT_SECRET`
- [ ] Update `SUPABASE_SERVICE_ROLE_KEY` to match
### Services
- [ ] Configure Redis connection (if external)
- [ ] Configure RabbitMQ credentials
- [ ] Configure SMTP for emails (if needed)
### OAuth (if using integrations)
- [ ] Register OAuth apps with your callback URL
- [ ] Add client IDs and secrets to `backend/.env`
### Post-deployment
- [ ] Run `docker compose up -d --build`
- [ ] Verify frontend loads at your URL
- [ ] Test authentication flow
- [ ] Test WebSocket connection (real-time updates)
---
## 9. Docker vs External Services
### Running Everything in Docker (Default)
The docker-compose files automatically set internal hostnames:
```yaml
# Internal Docker service names (container-to-container communication)
# These are set automatically in docker-compose.platform.yml
DB_HOST: db
REDIS_HOST: redis
RABBITMQ_HOST: rabbitmq
SUPABASE_URL: http://kong:8000
```
### Using External Services
If using managed services (AWS RDS, Redis Cloud, etc.), override in your `.env`:
```bash
# External PostgreSQL
DB_HOST=your-rds-instance.region.rds.amazonaws.com
DB_PORT=5432
# External Redis
REDIS_HOST=your-redis.cache.amazonaws.com
REDIS_PORT=6379
REDIS_PASSWORD=<if-required>
# External Supabase (hosted)
SUPABASE_URL=https://your-project.supabase.co
SUPABASE_SERVICE_ROLE_KEY=<your-service-role-key>
```
---
## Related Documentation
- [Getting Started](getting-started.md) - Basic setup guide
- [Advanced Setup](advanced_setup.md) - Development configuration
- [OAuth & SSO](integrating/oauth-guide.md) - Integration setup

View File

@@ -218,17 +218,6 @@ If you initially installed Docker with Hyper-V, you **dont need to reinstall*
For more details, refer to [Docker's official documentation](https://docs.docker.com/desktop/windows/wsl/).
### ⚠️ Podman Not Supported
AutoGPT requires **Docker** (Docker Desktop or Docker Engine). **Podman and podman-compose are not supported** and may cause path resolution issues, particularly on Windows.
If you see errors like:
```text
Error: the specified Containerfile or Dockerfile does not exist, ..\..\autogpt_platform\backend\Dockerfile
```
This indicates you're using Podman instead of Docker. Please install [Docker Desktop](https://docs.docker.com/desktop/) and use `docker compose` instead of `podman-compose`.
## Development