fix(backend/copilot): respect CLAUDE_CONFIG_DIR in SDK_PROJECTS_DIR constant

SDK_PROJECTS_DIR was hardcoded to ~/.claude/projects, ignoring the
CLAUDE_CONFIG_DIR environment variable. This caused path validation
mismatches in environments with custom Claude configurations.
Now consistent with transcript.py's _projects_base() function.

.claude/skills/pr-address/SKILL.md

@@ -0,0 +1,79 @@
+---
+name: pr-address
+description: Address PR review comments and loop until CI green and all comments resolved. TRIGGER when user asks to address comments, fix PR feedback, respond to reviewers, or babysit/monitor a PR.
+user-invocable: true
+args: "[PR number or URL] — if omitted, finds PR for current branch."
+metadata:
+  author: autogpt-team
+  version: "1.0.0"
+---
+
+# PR Address
+
+## Find the PR
+
+```bash
+gh pr list --head $(git branch --show-current) --repo Significant-Gravitas/AutoGPT
+gh pr view {N}
+```
+
+## Fetch comments (all sources)
+
+```bash
+gh api repos/Significant-Gravitas/AutoGPT/pulls/{N}/reviews       # top-level reviews
+gh api repos/Significant-Gravitas/AutoGPT/pulls/{N}/comments      # inline review comments
+gh api repos/Significant-Gravitas/AutoGPT/issues/{N}/comments     # PR conversation comments
+```
+
+**Bots to watch for:**
+- `autogpt-reviewer` — posts "Blockers", "Should Fix", "Nice to Have". Address ALL of them.
+- `sentry[bot]` — bug predictions. Fix real bugs, explain false positives.
+- `coderabbitai[bot]` — automated review. Address actionable items.
+
+## For each unaddressed comment
+
+Address comments **one at a time**: fix → commit → push → inline reply → next.
+
+1. Read the referenced code, make the fix (or reply explaining why it's not needed)
+2. Commit and push the fix
+3. Reply **inline** (not as a new top-level comment) referencing the fixing commit — this is what resolves the conversation for bot reviewers (coderabbitai, sentry):
+
+| Comment type | How to reply |
+|---|---|
+| Inline review (`pulls/{N}/comments`) | `gh api repos/Significant-Gravitas/AutoGPT/pulls/{N}/comments/{ID}/replies -f body="Fixed in <commit-sha>: <description>"` |
+| Conversation (`issues/{N}/comments`) | `gh api repos/Significant-Gravitas/AutoGPT/issues/{N}/comments -f body="Fixed in <commit-sha>: <description>"` |
+
+## Format and commit
+
+After fixing, format the changed code:
+
+- **Backend** (from `autogpt_platform/backend/`): `poetry run format`
+- **Frontend** (from `autogpt_platform/frontend/`): `pnpm format && pnpm lint && pnpm types`
+
+If API routes changed, regenerate the frontend client:
+```bash
+cd autogpt_platform/backend && poetry run rest &
+REST_PID=$!
+trap "kill $REST_PID 2>/dev/null" EXIT
+WAIT=0; until curl -sf http://localhost:8006/health > /dev/null 2>&1; do sleep 1; WAIT=$((WAIT+1)); [ $WAIT -ge 60 ] && echo "Timed out" && exit 1; done
+cd ../frontend && pnpm generate:api:force
+kill $REST_PID 2>/dev/null; trap - EXIT
+```
+Never manually edit files in `src/app/api/__generated__/`.
+
+Then commit and **push immediately** — never batch commits without pushing.
+
+For backend commits in worktrees: `poetry run git commit` (pre-commit hooks).
+
+## The loop
+
+```text
+address comments → format → commit → push
+→ re-check comments → fix new ones → push
+→ wait for CI → re-check comments after CI settles
+→ repeat until: all comments addressed AND CI green AND no new comments arriving
+```
+
+While CI runs, stay productive: run local tests, address remaining comments.
+
+**The loop ends when:** CI fully green + all comments addressed + no new comments since CI settled.

.claude/skills/pr-review/SKILL.md

@@ -0,0 +1,74 @@
+---
+name: pr-review
+description: Review a PR for correctness, security, code quality, and testing issues. TRIGGER when user asks to review a PR, check PR quality, or give feedback on a PR.
+user-invocable: true
+args: "[PR number or URL] — if omitted, finds PR for current branch."
+metadata:
+  author: autogpt-team
+  version: "1.0.0"
+---
+
+# PR Review
+
+## Find the PR
+
+```bash
+gh pr list --head $(git branch --show-current) --repo Significant-Gravitas/AutoGPT
+gh pr view {N}
+```
+
+## Read the diff
+
+```bash
+gh pr diff {N}
+```
+
+## Fetch existing review comments
+
+Before posting anything, fetch existing inline comments to avoid duplicates:
+
+```bash
+gh api repos/Significant-Gravitas/AutoGPT/pulls/{N}/comments
+gh api repos/Significant-Gravitas/AutoGPT/pulls/{N}/reviews
+```
+
+## What to check
+
+**Correctness:** logic errors, off-by-one, missing edge cases, race conditions (TOCTOU in file access, credit charging), error handling gaps, async correctness (missing `await`, unclosed resources).
+
+**Security:** input validation at boundaries, no injection (command, XSS, SQL), secrets not logged, file paths sanitized (`os.path.basename()` in error messages).
+
+**Code quality:** apply rules from backend/frontend CLAUDE.md files.
+
+**Architecture:** DRY, single responsibility, modular functions. `Security()` vs `Depends()` for FastAPI auth. `data:` for SSE events, `: comment` for heartbeats. `transaction=True` for Redis pipelines.
+
+**Testing:** edge cases covered, colocated `*_test.py` (backend) / `__tests__/` (frontend), mocks target where symbol is **used** not defined, `AsyncMock` for async.
+
+## Output format
+
+Every comment **must** be prefixed with `🤖` and a criticality badge:
+
+| Tier | Badge | Meaning |
+|---|---|---|
+| Blocker | `🔴 **Blocker**` | Must fix before merge |
+| Should Fix | `🟠 **Should Fix**` | Important improvement |
+| Nice to Have | `🟡 **Nice to Have**` | Minor suggestion |
+| Nit | `🔵 **Nit**` | Style / wording |
+
+Example: `🤖 🔴 **Blocker**: Missing error handling for X — suggest wrapping in try/except.`
+
+## Post inline comments
+
+For each finding, post an inline comment on the PR (do not just write a local report):
+
+```bash
+# Get the latest commit SHA for the PR
+COMMIT_SHA=$(gh api repos/Significant-Gravitas/AutoGPT/pulls/{N} --jq '.head.sha')
+
+# Post an inline comment on a specific file/line
+gh api repos/Significant-Gravitas/AutoGPT/pulls/{N}/comments \
+  -f body="🤖 🔴 **Blocker**: <description>" \
+  -f commit_id="$COMMIT_SHA" \
+  -f path="<file path>" \
+  -F line=<line number>
+```

.claude/skills/worktree/SKILL.md

@@ -0,0 +1,85 @@
+---
+name: worktree
+description: Set up a new git worktree for parallel development. Creates the worktree, copies .env files, installs dependencies, and generates Prisma client. TRIGGER when user asks to set up a worktree, work on a branch in isolation, or needs a separate environment for a branch or PR.
+user-invocable: true
+args: "[name] — optional worktree name (e.g., 'AutoGPT7'). If omitted, uses next available AutoGPT<N>."
+metadata:
+  author: autogpt-team
+  version: "3.0.0"
+---
+
+# Worktree Setup
+
+## Create the worktree
+
+Derive paths from the git toplevel. If a name is provided as argument, use it. Otherwise, check `git worktree list` and pick the next `AutoGPT<N>`.
+
+```bash
+ROOT=$(git rev-parse --show-toplevel)
+PARENT=$(dirname "$ROOT")
+
+# From an existing branch
+git worktree add "$PARENT/<NAME>" <branch-name>
+
+# From a new branch off dev
+git worktree add -b <new-branch> "$PARENT/<NAME>" dev
+```
+
+## Copy environment files
+
+Copy `.env` from the root worktree. Falls back to `.env.default` if `.env` doesn't exist.
+
+```bash
+ROOT=$(git rev-parse --show-toplevel)
+TARGET="$(dirname "$ROOT")/<NAME>"
+
+for envpath in autogpt_platform/backend autogpt_platform/frontend autogpt_platform; do
+  if [ -f "$ROOT/$envpath/.env" ]; then
+    cp "$ROOT/$envpath/.env" "$TARGET/$envpath/.env"
+  elif [ -f "$ROOT/$envpath/.env.default" ]; then
+    cp "$ROOT/$envpath/.env.default" "$TARGET/$envpath/.env"
+  fi
+done
+```
+
+## Install dependencies
+
+```bash
+TARGET="$(dirname "$(git rev-parse --show-toplevel)")/<NAME>"
+cd "$TARGET/autogpt_platform/autogpt_libs" && poetry install
+cd "$TARGET/autogpt_platform/backend" && poetry install && poetry run prisma generate
+cd "$TARGET/autogpt_platform/frontend" && pnpm install
+```
+
+Replace `<NAME>` with the actual worktree name (e.g., `AutoGPT7`).
+
+## Running the app (optional)
+
+Backend uses ports: 8001, 8002, 8003, 8005, 8006, 8007, 8008. Free them first if needed:
+
+```bash
+TARGET="$(dirname "$(git rev-parse --show-toplevel)")/<NAME>"
+for port in 8001 8002 8003 8005 8006 8007 8008; do
+  lsof -ti :$port | xargs kill -9 2>/dev/null || true
+done
+cd "$TARGET/autogpt_platform/backend" && poetry run app
+```
+
+## CoPilot testing
+
+SDK mode spawns a Claude subprocess — won't work inside Claude Code. Set `CHAT_USE_CLAUDE_AGENT_SDK=false` in `backend/.env` to use baseline mode.
+
+## Cleanup
+
+```bash
+# Replace <NAME> with the actual worktree name (e.g., AutoGPT7)
+git worktree remove "$(dirname "$(git rev-parse --show-toplevel)")/<NAME>"
+```
+
+## Alternative: Branchlet (optional)
+
+If [branchlet](https://www.npmjs.com/package/branchlet) is installed:
+
+```bash
+branchlet create -n <name> -s <source-branch> -b <new-branch>
+```

.github/workflows/platform-backend-ci.yml

@@ -41,13 +41,18 @@ jobs:
         ports:
           - 6379:6379
       rabbitmq:
-        image: rabbitmq:3.12-management
+        image: rabbitmq:4.1.4
         ports:
           - 5672:5672
-          - 15672:15672
         env:
           RABBITMQ_DEFAULT_USER: ${{ env.RABBITMQ_DEFAULT_USER }}
           RABBITMQ_DEFAULT_PASS: ${{ env.RABBITMQ_DEFAULT_PASS }}
+        options: >-
+          --health-cmd "rabbitmq-diagnostics -q ping"
+          --health-interval 30s
+          --health-timeout 10s
+          --health-retries 5
+          --health-start-period 10s
       clamav:
         image: clamav/clamav-debian:latest
         ports:

.github/workflows/platform-frontend-ci.yml

@@ -6,10 +6,16 @@ on:
     paths:
       - ".github/workflows/platform-frontend-ci.yml"
       - "autogpt_platform/frontend/**"
+      - "autogpt_platform/backend/Dockerfile"
+      - "autogpt_platform/docker-compose.yml"
+      - "autogpt_platform/docker-compose.platform.yml"
   pull_request:
     paths:
       - ".github/workflows/platform-frontend-ci.yml"
       - "autogpt_platform/frontend/**"
+      - "autogpt_platform/backend/Dockerfile"
+      - "autogpt_platform/docker-compose.yml"
+      - "autogpt_platform/docker-compose.platform.yml"
   merge_group:
   workflow_dispatch:
 
@@ -143,7 +149,7 @@ jobs:
           driver-opts: network=host
 
       - name: Set up Platform - Expose GHA cache to docker buildx CLI
-        uses: crazy-max/ghaction-github-runtime@v3
+        uses: crazy-max/ghaction-github-runtime@v4
 
       - name: Set up Platform - Build Docker images (with cache)
         working-directory: autogpt_platform

.gitignore

@@ -180,4 +180,6 @@ autogpt_platform/backend/settings.py
 .claude/settings.local.json
 CLAUDE.local.md
 /autogpt_platform/backend/logs
-.next
+.next
+# Implementation plans (generated by AI agents)
+plans/

.nvmrc

@@ -0,0 +1 @@
+22

.pre-commit-config.yaml

@@ -1,3 +1,10 @@
+default_install_hook_types:
+  - pre-commit
+  - pre-push
+  - post-checkout
+
+default_stages: [pre-commit]
+
 repos:
   - repo: https://github.com/pre-commit/pre-commit-hooks
     rev: v4.4.0
@@ -17,6 +24,7 @@ repos:
         name: Detect secrets
         description: Detects high entropy strings that are likely to be passwords.
         files: ^autogpt_platform/
+        exclude: pnpm-lock\.yaml$
         stages: [pre-push]
 
   - repo: local
@@ -26,49 +34,106 @@ repos:
       - id: poetry-install
         name: Check & Install dependencies - AutoGPT Platform - Backend
         alias: poetry-install-platform-backend
-        entry: poetry -C autogpt_platform/backend install
         # include autogpt_libs source (since it's a path dependency)
-        files: ^autogpt_platform/(backend|autogpt_libs)/poetry\.lock$
-        types: [file]
+        entry: >
+          bash -c '
+          if [ -n "$PRE_COMMIT_FROM_REF" ]; then
+            git diff --name-only "$PRE_COMMIT_FROM_REF" "$PRE_COMMIT_TO_REF"
+          else
+            git diff --cached --name-only
+          fi | grep -qE "^autogpt_platform/(backend|autogpt_libs)/poetry\.lock$" || exit 0;
+          poetry -C autogpt_platform/backend install
+          '
+        always_run: true
         language: system
         pass_filenames: false
+        stages: [pre-commit, post-checkout]
 
       - id: poetry-install
         name: Check & Install dependencies - AutoGPT Platform - Libs
         alias: poetry-install-platform-libs
-        entry: poetry -C autogpt_platform/autogpt_libs install
-        files: ^autogpt_platform/autogpt_libs/poetry\.lock$
-        types: [file]
+        entry: >
+          bash -c '
+          if [ -n "$PRE_COMMIT_FROM_REF" ]; then
+            git diff --name-only "$PRE_COMMIT_FROM_REF" "$PRE_COMMIT_TO_REF"
+          else
+            git diff --cached --name-only
+          fi | grep -qE "^autogpt_platform/autogpt_libs/poetry\.lock$" || exit 0;
+          poetry -C autogpt_platform/autogpt_libs install
+          '
+        always_run: true
         language: system
         pass_filenames: false
+        stages: [pre-commit, post-checkout]
+
+      - id: pnpm-install
+        name: Check & Install dependencies - AutoGPT Platform - Frontend
+        alias: pnpm-install-platform-frontend
+        entry: >
+          bash -c '
+          if [ -n "$PRE_COMMIT_FROM_REF" ]; then
+            git diff --name-only "$PRE_COMMIT_FROM_REF" "$PRE_COMMIT_TO_REF"
+          else
+            git diff --cached --name-only
+          fi | grep -qE "^autogpt_platform/frontend/pnpm-lock\.yaml$" || exit 0;
+          pnpm --prefix autogpt_platform/frontend install
+          '
+        always_run: true
+        language: system
+        pass_filenames: false
+        stages: [pre-commit, post-checkout]
 
       - id: poetry-install
         name: Check & Install dependencies - Classic - AutoGPT
         alias: poetry-install-classic-autogpt
-        entry: poetry -C classic/original_autogpt install
+        entry: >
+          bash -c '
+          if [ -n "$PRE_COMMIT_FROM_REF" ]; then
+            git diff --name-only "$PRE_COMMIT_FROM_REF" "$PRE_COMMIT_TO_REF"
+          else
+            git diff --cached --name-only
+          fi | grep -qE "^classic/(original_autogpt|forge)/poetry\.lock$" || exit 0;
+          poetry -C classic/original_autogpt install
+          '
         # include forge source (since it's a path dependency)
-        files: ^classic/(original_autogpt|forge)/poetry\.lock$
-        types: [file]
+        always_run: true
         language: system
         pass_filenames: false
+        stages: [pre-commit, post-checkout]
 
       - id: poetry-install
         name: Check & Install dependencies - Classic - Forge
         alias: poetry-install-classic-forge
-        entry: poetry -C classic/forge install
-        files: ^classic/forge/poetry\.lock$
-        types: [file]
+        entry: >
+          bash -c '
+          if [ -n "$PRE_COMMIT_FROM_REF" ]; then
+            git diff --name-only "$PRE_COMMIT_FROM_REF" "$PRE_COMMIT_TO_REF"
+          else
+            git diff --cached --name-only
+          fi | grep -qE "^classic/forge/poetry\.lock$" || exit 0;
+          poetry -C classic/forge install
+          '
+        always_run: true
         language: system
         pass_filenames: false
+        stages: [pre-commit, post-checkout]
 
       - id: poetry-install
         name: Check & Install dependencies - Classic - Benchmark
         alias: poetry-install-classic-benchmark
-        entry: poetry -C classic/benchmark install
-        files: ^classic/benchmark/poetry\.lock$
-        types: [file]
+        entry: >
+          bash -c '
+          if [ -n "$PRE_COMMIT_FROM_REF" ]; then
+            git diff --name-only "$PRE_COMMIT_FROM_REF" "$PRE_COMMIT_TO_REF"
+          else
+            git diff --cached --name-only
+          fi | grep -qE "^classic/benchmark/poetry\.lock$" || exit 0;
+          poetry -C classic/benchmark install
+          '
+        always_run: true
         language: system
         pass_filenames: false
+        stages: [pre-commit, post-checkout]
 
   - repo: local
     # For proper type checking, Prisma client must be up-to-date.
@@ -76,12 +141,54 @@ repos:
       - id: prisma-generate
         name: Prisma Generate - AutoGPT Platform - Backend
         alias: prisma-generate-platform-backend
-        entry: bash -c 'cd autogpt_platform/backend && poetry run prisma generate'
+        entry: >
+          bash -c '
+          if [ -n "$PRE_COMMIT_FROM_REF" ]; then
+            git diff --name-only "$PRE_COMMIT_FROM_REF" "$PRE_COMMIT_TO_REF"
+          else
+            git diff --cached --name-only
+          fi | grep -qE "^autogpt_platform/((backend|autogpt_libs)/poetry\.lock|backend/schema\.prisma)$" || exit 0;
+          cd autogpt_platform/backend
+          && poetry run prisma generate
+          && poetry run gen-prisma-stub
+          '
         # include everything that triggers poetry install + the prisma schema
-        files: ^autogpt_platform/((backend|autogpt_libs)/poetry\.lock|backend/schema.prisma)$
-        types: [file]
+        always_run: true
         language: system
         pass_filenames: false
+        stages: [pre-commit, post-checkout]
+
+      - id: export-api-schema
+        name: Export API schema - AutoGPT Platform - Backend -> Frontend
+        alias: export-api-schema-platform
+        entry: >
+          bash -c '
+          cd autogpt_platform/backend
+          && poetry run export-api-schema --output ../frontend/src/app/api/openapi.json
+          && cd ../frontend
+          && pnpm prettier --write ./src/app/api/openapi.json
+          '
+        files: ^autogpt_platform/backend/
+        language: system
+        pass_filenames: false
+
+      - id: generate-api-client
+        name: Generate API client - AutoGPT Platform - Frontend
+        alias: generate-api-client-platform-frontend
+        entry: >
+          bash -c '
+          SCHEMA=autogpt_platform/frontend/src/app/api/openapi.json;
+          if [ -n "$PRE_COMMIT_FROM_REF" ]; then
+            git diff --quiet "$PRE_COMMIT_FROM_REF" "$PRE_COMMIT_TO_REF" -- "$SCHEMA" && exit 0
+          else
+            git diff --quiet HEAD -- "$SCHEMA" && exit 0
+          fi;
+          cd autogpt_platform/frontend && pnpm generate:api
+          '
+        always_run: true
+        language: system
+        pass_filenames: false
+        stages: [pre-commit, post-checkout]
 
   - repo: https://github.com/astral-sh/ruff-pre-commit
     rev: v0.7.2

autogpt_platform/.gitignore

@@ -1,2 +1,3 @@
 *.ignore.*
-*.ign.*
+*.ign.*
+.application.logs

autogpt_platform/CLAUDE.md

@@ -60,9 +60,12 @@ AutoGPT Platform is a monorepo containing:
 
 ### Reviewing/Revising Pull Requests
 
-- When the user runs /pr-comments or tries to fetch them, also run gh api /repos/Significant-Gravitas/AutoGPT/pulls/[issuenum]/reviews to get the reviews
-- Use gh api /repos/Significant-Gravitas/AutoGPT/pulls/[issuenum]/reviews/[review_id]/comments to get the review contents
-- Use gh api /repos/Significant-Gravitas/AutoGPT/issues/9924/comments to get the pr specific comments
+Use `/pr-review` to review a PR or `/pr-address` to address comments.
+
+When fetching comments manually:
+- `gh api repos/Significant-Gravitas/AutoGPT/pulls/{N}/reviews` — top-level reviews
+- `gh api repos/Significant-Gravitas/AutoGPT/pulls/{N}/comments` — inline review comments
+- `gh api repos/Significant-Gravitas/AutoGPT/issues/{N}/comments` — PR conversation comments
 
 ### Conventional Commits
 

autogpt_platform/analytics/queries/auth_activities.sql

@@ -0,0 +1,40 @@
+-- =============================================================
+-- View: analytics.auth_activities
+-- Looker source alias: ds49  |  Charts: 1
+-- =============================================================
+-- DESCRIPTION
+--   Tracks authentication events (login, logout, SSO, password
+--   reset, etc.) from Supabase's internal audit log.
+--   Useful for monitoring sign-in patterns and detecting anomalies.
+--
+-- SOURCE TABLES
+--   auth.audit_log_entries  — Supabase internal auth event log
+--
+-- OUTPUT COLUMNS
+--   created_at      TIMESTAMPTZ  When the auth event occurred
+--   actor_id        TEXT         User ID who triggered the event
+--   actor_via_sso   TEXT         Whether the action was via SSO ('true'/'false')
+--   action          TEXT         Event type (e.g. 'login', 'logout', 'token_refreshed')
+--
+-- WINDOW
+--   Rolling 90 days from current date
+--
+-- EXAMPLE QUERIES
+--   -- Daily login counts
+--   SELECT DATE_TRUNC('day', created_at) AS day, COUNT(*) AS logins
+--   FROM analytics.auth_activities
+--   WHERE action = 'login'
+--   GROUP BY 1 ORDER BY 1;
+--
+--   -- SSO vs password login breakdown
+--   SELECT actor_via_sso, COUNT(*) FROM analytics.auth_activities
+--   WHERE action = 'login' GROUP BY 1;
+-- =============================================================
+
+SELECT
+    created_at,
+    payload->>'actor_id'      AS actor_id,
+    payload->>'actor_via_sso' AS actor_via_sso,
+    payload->>'action'        AS action
+FROM auth.audit_log_entries
+WHERE created_at >= NOW() - INTERVAL '90 days'

autogpt_platform/analytics/queries/graph_execution.sql

@@ -0,0 +1,105 @@
+-- =============================================================
+-- View: analytics.graph_execution
+-- Looker source alias: ds16  |  Charts: 21
+-- =============================================================
+-- DESCRIPTION
+--   One row per agent graph execution (last 90 days).
+--   Unpacks the JSONB stats column into individual numeric columns
+--   and normalises the executionStatus — runs that failed due to
+--   insufficient credits are reclassified as 'NO_CREDITS' for
+--   easier filtering.  Error messages are scrubbed of IDs and URLs
+--   to allow safe grouping.
+--
+-- SOURCE TABLES
+--   platform.AgentGraphExecution  — Execution records
+--   platform.AgentGraph           — Agent graph metadata (for name)
+--   platform.LibraryAgent         — To flag possibly-AI (safe-mode) agents
+--
+-- OUTPUT COLUMNS
+--   id                TEXT         Execution UUID
+--   agentGraphId      TEXT         Agent graph UUID
+--   agentGraphVersion INT          Graph version number
+--   executionStatus   TEXT         COMPLETED | FAILED | NO_CREDITS | RUNNING | QUEUED | TERMINATED
+--   createdAt         TIMESTAMPTZ  When the execution was queued
+--   updatedAt         TIMESTAMPTZ  Last status update time
+--   userId            TEXT         Owner user UUID
+--   agentGraphName    TEXT         Human-readable agent name
+--   cputime           DECIMAL      Total CPU seconds consumed
+--   walltime          DECIMAL      Total wall-clock seconds
+--   node_count        DECIMAL      Number of nodes in the graph
+--   nodes_cputime     DECIMAL      CPU time across all nodes
+--   nodes_walltime    DECIMAL      Wall time across all nodes
+--   execution_cost    DECIMAL      Credit cost of this execution
+--   correctness_score FLOAT        AI correctness score (if available)
+--   possibly_ai       BOOLEAN      True if agent has sensitive_action_safe_mode enabled
+--   groupedErrorMessage TEXT       Scrubbed error string (IDs/URLs replaced with wildcards)
+--
+-- WINDOW
+--   Rolling 90 days (createdAt > CURRENT_DATE - 90 days)
+--
+-- EXAMPLE QUERIES
+--   -- Daily execution counts by status
+--   SELECT DATE_TRUNC('day', "createdAt") AS day, "executionStatus", COUNT(*)
+--   FROM analytics.graph_execution
+--   GROUP BY 1, 2 ORDER BY 1;
+--
+--   -- Average cost per execution by agent
+--   SELECT "agentGraphName", AVG("execution_cost") AS avg_cost, COUNT(*) AS runs
+--   FROM analytics.graph_execution
+--   WHERE "executionStatus" = 'COMPLETED'
+--   GROUP BY 1 ORDER BY avg_cost DESC;
+--
+--   -- Top error messages
+--   SELECT "groupedErrorMessage", COUNT(*) AS occurrences
+--   FROM analytics.graph_execution
+--   WHERE "executionStatus" = 'FAILED'
+--   GROUP BY 1 ORDER BY 2 DESC LIMIT 20;
+-- =============================================================
+
+SELECT
+    ge."id"                                                        AS id,
+    ge."agentGraphId"                                              AS agentGraphId,
+    ge."agentGraphVersion"                                         AS agentGraphVersion,
+    CASE
+        WHEN jsonb_exists(ge."stats"::jsonb, 'error')
+         AND (
+               (ge."stats"::jsonb->>'error') ILIKE '%insufficient balance%'
+            OR (ge."stats"::jsonb->>'error') ILIKE '%you have no credits left%'
+             )
+        THEN 'NO_CREDITS'
+        ELSE CAST(ge."executionStatus" AS TEXT)
+    END                                                            AS executionStatus,
+    ge."createdAt"                                                 AS createdAt,
+    ge."updatedAt"                                                 AS updatedAt,
+    ge."userId"                                                    AS userId,
+    g."name"                                                       AS agentGraphName,
+    (ge."stats"::jsonb->>'cputime')::decimal                       AS cputime,
+    (ge."stats"::jsonb->>'walltime')::decimal                      AS walltime,
+    (ge."stats"::jsonb->>'node_count')::decimal                    AS node_count,
+    (ge."stats"::jsonb->>'nodes_cputime')::decimal                 AS nodes_cputime,
+    (ge."stats"::jsonb->>'nodes_walltime')::decimal                AS nodes_walltime,
+    (ge."stats"::jsonb->>'cost')::decimal                          AS execution_cost,
+    (ge."stats"::jsonb->>'correctness_score')::float               AS correctness_score,
+    COALESCE(la.possibly_ai, FALSE)                                AS possibly_ai,
+    REGEXP_REPLACE(
+        REGEXP_REPLACE(
+            TRIM(BOTH '"' FROM ge."stats"::jsonb->>'error'),
+            '(https?://)([A-Za-z0-9.-]+)(:[0-9]+)?(/[^\s]*)?',
+            '\1\2/...', 'gi'
+        ),
+        '[a-zA-Z0-9_:-]*\d[a-zA-Z0-9_:-]*', '*', 'g'
+    )                                                              AS groupedErrorMessage
+FROM platform."AgentGraphExecution" ge
+LEFT JOIN platform."AgentGraph" g
+       ON ge."agentGraphId" = g."id"
+      AND ge."agentGraphVersion" = g."version"
+LEFT JOIN (
+    SELECT DISTINCT ON ("userId", "agentGraphId")
+           "userId", "agentGraphId",
+           ("settings"::jsonb->>'sensitive_action_safe_mode')::boolean AS possibly_ai
+    FROM platform."LibraryAgent"
+    WHERE "isDeleted"  = FALSE
+      AND "isArchived" = FALSE
+    ORDER BY "userId", "agentGraphId", "agentGraphVersion" DESC
+) la ON la."userId" = ge."userId" AND la."agentGraphId" = ge."agentGraphId"
+WHERE ge."createdAt" > CURRENT_DATE - INTERVAL '90 days'

autogpt_platform/analytics/queries/node_block_execution.sql

@@ -0,0 +1,101 @@
+-- =============================================================
+-- View: analytics.node_block_execution
+-- Looker source alias: ds14  |  Charts: 11
+-- =============================================================
+-- DESCRIPTION
+--   One row per node (block) execution (last 90 days).
+--   Unpacks stats JSONB and joins to identify which block type
+--   was run.  For failed nodes, joins the error output and
+--   scrubs it for safe grouping.
+--
+-- SOURCE TABLES
+--   platform.AgentNodeExecution              — Node execution records
+--   platform.AgentNode                       — Node → block mapping
+--   platform.AgentBlock                      — Block name/ID
+--   platform.AgentNodeExecutionInputOutput   — Error output values
+--
+-- OUTPUT COLUMNS
+--   id                    TEXT         Node execution UUID
+--   agentGraphExecutionId TEXT         Parent graph execution UUID
+--   agentNodeId           TEXT         Node UUID within the graph
+--   executionStatus       TEXT         COMPLETED | FAILED | QUEUED | RUNNING | TERMINATED
+--   addedTime             TIMESTAMPTZ  When the node was queued
+--   queuedTime            TIMESTAMPTZ  When it entered the queue
+--   startedTime           TIMESTAMPTZ  When execution started
+--   endedTime             TIMESTAMPTZ  When execution finished
+--   inputSize             BIGINT       Input payload size in bytes
+--   outputSize            BIGINT       Output payload size in bytes
+--   walltime              NUMERIC      Wall-clock seconds for this node
+--   cputime               NUMERIC      CPU seconds for this node
+--   llmRetryCount         INT          Number of LLM retries
+--   llmCallCount          INT          Number of LLM API calls made
+--   inputTokenCount       BIGINT       LLM input tokens consumed
+--   outputTokenCount      BIGINT       LLM output tokens produced
+--   blockName             TEXT         Human-readable block name (e.g. 'OpenAIBlock')
+--   blockId               TEXT         Block UUID
+--   groupedErrorMessage   TEXT         Scrubbed error (IDs/URLs wildcarded)
+--   errorMessage          TEXT         Raw error output (only set when FAILED)
+--
+-- WINDOW
+--   Rolling 90 days (addedTime > CURRENT_DATE - 90 days)
+--
+-- EXAMPLE QUERIES
+--   -- Most-used blocks by execution count
+--   SELECT "blockName", COUNT(*) AS executions,
+--          COUNT(*) FILTER (WHERE "executionStatus"='FAILED') AS failures
+--   FROM analytics.node_block_execution
+--   GROUP BY 1 ORDER BY executions DESC LIMIT 20;
+--
+--   -- Average LLM token usage per block
+--   SELECT "blockName",
+--          AVG("inputTokenCount") AS avg_input_tokens,
+--          AVG("outputTokenCount") AS avg_output_tokens
+--   FROM analytics.node_block_execution
+--   WHERE "llmCallCount" > 0
+--   GROUP BY 1 ORDER BY avg_input_tokens DESC;
+--
+--   -- Top failure reasons
+--   SELECT "blockName", "groupedErrorMessage", COUNT(*) AS count
+--   FROM analytics.node_block_execution
+--   WHERE "executionStatus" = 'FAILED'
+--   GROUP BY 1, 2 ORDER BY count DESC LIMIT 20;
+-- =============================================================
+
+SELECT
+    ne."id"                                                            AS id,
+    ne."agentGraphExecutionId"                                         AS agentGraphExecutionId,
+    ne."agentNodeId"                                                   AS agentNodeId,
+    CAST(ne."executionStatus" AS TEXT)                                 AS executionStatus,
+    ne."addedTime"                                                     AS addedTime,
+    ne."queuedTime"                                                    AS queuedTime,
+    ne."startedTime"                                                   AS startedTime,
+    ne."endedTime"                                                     AS endedTime,
+    (ne."stats"::jsonb->>'input_size')::bigint                         AS inputSize,
+    (ne."stats"::jsonb->>'output_size')::bigint                        AS outputSize,
+    (ne."stats"::jsonb->>'walltime')::numeric                          AS walltime,
+    (ne."stats"::jsonb->>'cputime')::numeric                           AS cputime,
+    (ne."stats"::jsonb->>'llm_retry_count')::int                       AS llmRetryCount,
+    (ne."stats"::jsonb->>'llm_call_count')::int                        AS llmCallCount,
+    (ne."stats"::jsonb->>'input_token_count')::bigint                  AS inputTokenCount,
+    (ne."stats"::jsonb->>'output_token_count')::bigint                 AS outputTokenCount,
+    b."name"                                                           AS blockName,
+    b."id"                                                             AS blockId,
+    REGEXP_REPLACE(
+        REGEXP_REPLACE(
+            TRIM(BOTH '"' FROM eio."data"::text),
+            '(https?://)([A-Za-z0-9.-]+)(:[0-9]+)?(/[^\s]*)?',
+            '\1\2/...', 'gi'
+        ),
+        '[a-zA-Z0-9_:-]*\d[a-zA-Z0-9_:-]*', '*', 'g'
+    )                                                                  AS groupedErrorMessage,
+    eio."data"                                                         AS errorMessage
+FROM platform."AgentNodeExecution" ne
+LEFT JOIN platform."AgentNode" nd
+       ON ne."agentNodeId" = nd."id"
+LEFT JOIN platform."AgentBlock" b
+       ON nd."agentBlockId" = b."id"
+LEFT JOIN platform."AgentNodeExecutionInputOutput" eio
+       ON eio."referencedByOutputExecId" = ne."id"
+      AND eio."name" = 'error'
+      AND ne."executionStatus" = 'FAILED'
+WHERE ne."addedTime" > CURRENT_DATE - INTERVAL '90 days'

autogpt_platform/analytics/queries/retention_agent.sql

@@ -0,0 +1,97 @@
+-- =============================================================
+-- View: analytics.retention_agent
+-- Looker source alias: ds35  |  Charts: 2
+-- =============================================================
+-- DESCRIPTION
+--   Weekly cohort retention broken down per individual agent.
+--   Cohort = week of a user's first use of THAT specific agent.
+--   Tells you which agents keep users coming back vs. one-shot
+--   use. Only includes cohorts from the last 180 days.
+--
+-- SOURCE TABLES
+--   platform.AgentGraphExecution  — Execution records (user × agent × time)
+--   platform.AgentGraph           — Agent names
+--
+-- OUTPUT COLUMNS
+--   agent_id            TEXT   Agent graph UUID
+--   agent_label         TEXT   'AgentName [first8chars]'
+--   agent_label_n       TEXT   'AgentName [first8chars] (n=total_users)'
+--   cohort_week_start   DATE   Week users first ran this agent
+--   cohort_label        TEXT   ISO week label
+--   cohort_label_n      TEXT   ISO week label with cohort size
+--   user_lifetime_week  INT    Weeks since first use of this agent
+--   cohort_users        BIGINT Users in this cohort for this agent
+--   active_users        BIGINT Users who ran the agent again in week k
+--   retention_rate      FLOAT  active_users / cohort_users
+--   cohort_users_w0     BIGINT cohort_users only at week 0 (safe to SUM)
+--   agent_total_users   BIGINT Total users across all cohorts for this agent
+--
+-- EXAMPLE QUERIES
+--   -- Best-retained agents at week 2
+--   SELECT agent_label, AVG(retention_rate) AS w2_retention
+--   FROM analytics.retention_agent
+--   WHERE user_lifetime_week = 2 AND cohort_users >= 10
+--   GROUP BY 1 ORDER BY w2_retention DESC LIMIT 10;
+--
+--   -- Agents with most unique users
+--   SELECT DISTINCT agent_label, agent_total_users
+--   FROM analytics.retention_agent
+--   ORDER BY agent_total_users DESC LIMIT 20;
+-- =============================================================
+
+WITH params AS (SELECT 12::int AS max_weeks, (CURRENT_DATE - INTERVAL '180 days') AS cohort_start),
+events AS (
+  SELECT e."userId"::text AS user_id, e."agentGraphId" AS agent_id,
+         e."createdAt"::timestamptz AS created_at,
+         DATE_TRUNC('week', e."createdAt")::date AS week_start
+  FROM platform."AgentGraphExecution" e
+),
+first_use AS (
+  SELECT user_id, agent_id, MIN(created_at) AS first_use_at,
+         DATE_TRUNC('week', MIN(created_at))::date AS cohort_week_start
+  FROM events GROUP BY 1,2
+  HAVING MIN(created_at) >= (SELECT cohort_start FROM params)
+),
+activity_weeks AS (SELECT DISTINCT user_id, agent_id, week_start FROM events),
+user_week_age AS (
+  SELECT aw.user_id, aw.agent_id, fu.cohort_week_start,
+         ((aw.week_start - DATE_TRUNC('week',fu.first_use_at)::date)/7)::int AS user_lifetime_week
+  FROM activity_weeks aw JOIN first_use fu USING (user_id, agent_id)
+  WHERE aw.week_start >= DATE_TRUNC('week',fu.first_use_at)::date
+),
+active_counts AS (
+  SELECT agent_id, cohort_week_start, user_lifetime_week, COUNT(DISTINCT user_id) AS active_users
+  FROM user_week_age WHERE user_lifetime_week >= 0 GROUP BY 1,2,3
+),
+cohort_sizes AS (
+  SELECT agent_id, cohort_week_start, COUNT(DISTINCT user_id) AS cohort_users FROM first_use GROUP BY 1,2
+),
+cohort_caps AS (
+  SELECT cs.agent_id, cs.cohort_week_start, cs.cohort_users,
+         LEAST((SELECT max_weeks FROM params),
+               GREATEST(0,((DATE_TRUNC('week',CURRENT_DATE)::date-cs.cohort_week_start)/7)::int)) AS cap_weeks
+  FROM cohort_sizes cs
+),
+grid AS (
+  SELECT cc.agent_id, cc.cohort_week_start, gs AS user_lifetime_week, cc.cohort_users
+  FROM cohort_caps cc CROSS JOIN LATERAL generate_series(0, cc.cap_weeks) gs
+),
+agent_names AS (SELECT DISTINCT ON (g."id") g."id" AS agent_id, g."name" AS agent_name FROM platform."AgentGraph" g ORDER BY g."id", g."version" DESC),
+agent_total_users AS (SELECT agent_id, SUM(cohort_users) AS agent_total_users FROM cohort_sizes GROUP BY 1)
+SELECT
+  g.agent_id,
+  COALESCE(an.agent_name,'(unnamed)')||' ['||LEFT(g.agent_id::text,8)||']'  AS agent_label,
+  COALESCE(an.agent_name,'(unnamed)')||' ['||LEFT(g.agent_id::text,8)||'] (n='||COALESCE(atu.agent_total_users,0)||')' AS agent_label_n,
+  g.cohort_week_start,
+  TO_CHAR(g.cohort_week_start,'IYYY-"W"IW')                               AS cohort_label,
+  TO_CHAR(g.cohort_week_start,'IYYY-"W"IW')||' (n='||g.cohort_users||')'  AS cohort_label_n,
+  g.user_lifetime_week, g.cohort_users,
+  COALESCE(ac.active_users,0)                                              AS active_users,
+  COALESCE(ac.active_users,0)::float / NULLIF(g.cohort_users,0)           AS retention_rate,
+  CASE WHEN g.user_lifetime_week=0 THEN g.cohort_users ELSE 0 END         AS cohort_users_w0,
+  COALESCE(atu.agent_total_users,0)                                        AS agent_total_users
+FROM grid g
+LEFT JOIN active_counts     ac  ON ac.agent_id=g.agent_id AND ac.cohort_week_start=g.cohort_week_start AND ac.user_lifetime_week=g.user_lifetime_week
+LEFT JOIN agent_names       an  ON an.agent_id=g.agent_id
+LEFT JOIN agent_total_users atu ON atu.agent_id=g.agent_id
+ORDER BY agent_label, g.cohort_week_start, g.user_lifetime_week;

autogpt_platform/analytics/queries/retention_execution_daily.sql

@@ -0,0 +1,81 @@
+-- =============================================================
+-- View: analytics.retention_execution_daily
+-- Looker source alias: ds111  |  Charts: 1
+-- =============================================================
+-- DESCRIPTION
+--   Daily cohort retention based on agent executions.
+--   Cohort anchor = day of user's FIRST ever execution.
+--   Only includes cohorts from the last 90 days, up to day 30.
+--   Great for early engagement analysis (did users run another
+--   agent the next day?).
+--
+-- SOURCE TABLES
+--   platform.AgentGraphExecution  — Execution records
+--
+-- OUTPUT COLUMNS
+--   Same pattern as retention_login_daily.
+--   cohort_day_start = day of first execution (not first login)
+--
+-- EXAMPLE QUERIES
+--   -- Day-3 execution retention
+--   SELECT cohort_label, retention_rate_bounded AS d3_retention
+--   FROM analytics.retention_execution_daily
+--   WHERE user_lifetime_day = 3 ORDER BY cohort_day_start;
+-- =============================================================
+
+WITH params AS (SELECT 30::int AS max_days, (CURRENT_DATE - INTERVAL '90 days') AS cohort_start),
+events AS (
+  SELECT e."userId"::text AS user_id, e."createdAt"::timestamptz AS created_at,
+         DATE_TRUNC('day', e."createdAt")::date AS day_start
+  FROM platform."AgentGraphExecution" e WHERE e."userId" IS NOT NULL
+),
+first_exec AS (
+  SELECT user_id, MIN(created_at) AS first_exec_at,
+         DATE_TRUNC('day', MIN(created_at))::date AS cohort_day_start
+  FROM events GROUP BY 1
+  HAVING MIN(created_at) >= (SELECT cohort_start FROM params)
+),
+activity_days AS (SELECT DISTINCT user_id, day_start FROM events),
+user_day_age AS (
+  SELECT ad.user_id, fe.cohort_day_start,
+         (ad.day_start - DATE_TRUNC('day',fe.first_exec_at)::date)::int AS user_lifetime_day
+  FROM activity_days ad JOIN first_exec fe USING (user_id)
+  WHERE ad.day_start >= DATE_TRUNC('day',fe.first_exec_at)::date
+),
+bounded_counts AS (
+  SELECT cohort_day_start, user_lifetime_day, COUNT(DISTINCT user_id) AS active_users_bounded
+  FROM user_day_age WHERE user_lifetime_day >= 0 GROUP BY 1,2
+),
+last_active AS (
+  SELECT cohort_day_start, user_id, MAX(user_lifetime_day) AS last_active_day FROM user_day_age GROUP BY 1,2
+),
+unbounded_counts AS (
+  SELECT la.cohort_day_start, gs AS user_lifetime_day, COUNT(*) AS retained_users_unbounded
+  FROM last_active la
+  CROSS JOIN LATERAL generate_series(0, LEAST(la.last_active_day,(SELECT max_days FROM params))) gs
+  GROUP BY 1,2
+),
+cohort_sizes AS (SELECT cohort_day_start, COUNT(DISTINCT user_id) AS cohort_users FROM first_exec GROUP BY 1),
+cohort_caps AS (
+  SELECT cs.cohort_day_start, cs.cohort_users,
+         LEAST((SELECT max_days FROM params), GREATEST(0,(CURRENT_DATE-cs.cohort_day_start)::int)) AS cap_days
+  FROM cohort_sizes cs
+),
+grid AS (
+  SELECT cc.cohort_day_start, gs AS user_lifetime_day, cc.cohort_users
+  FROM cohort_caps cc CROSS JOIN LATERAL generate_series(0, cc.cap_days) gs
+)
+SELECT
+  g.cohort_day_start,
+  TO_CHAR(g.cohort_day_start,'YYYY-MM-DD')                                AS cohort_label,
+  TO_CHAR(g.cohort_day_start,'YYYY-MM-DD')||' (n='||g.cohort_users||')'   AS cohort_label_n,
+  g.user_lifetime_day, g.cohort_users,
+  COALESCE(b.active_users_bounded,0)     AS active_users_bounded,
+  COALESCE(u.retained_users_unbounded,0) AS retained_users_unbounded,
+  CASE WHEN g.cohort_users>0 THEN COALESCE(b.active_users_bounded,0)::float/g.cohort_users END    AS retention_rate_bounded,
+  CASE WHEN g.cohort_users>0 THEN COALESCE(u.retained_users_unbounded,0)::float/g.cohort_users END AS retention_rate_unbounded,
+  CASE WHEN g.user_lifetime_day=0 THEN g.cohort_users ELSE 0 END          AS cohort_users_d0
+FROM grid g
+LEFT JOIN bounded_counts   b ON b.cohort_day_start=g.cohort_day_start AND b.user_lifetime_day=g.user_lifetime_day
+LEFT JOIN unbounded_counts u ON u.cohort_day_start=g.cohort_day_start AND u.user_lifetime_day=g.user_lifetime_day
+ORDER BY g.cohort_day_start, g.user_lifetime_day;

autogpt_platform/analytics/queries/retention_execution_weekly.sql

@@ -0,0 +1,81 @@
+-- =============================================================
+-- View: analytics.retention_execution_weekly
+-- Looker source alias: ds92  |  Charts: 2
+-- =============================================================
+-- DESCRIPTION
+--   Weekly cohort retention based on agent executions.
+--   Cohort anchor = week of user's FIRST ever agent execution
+--   (not first login). Only includes cohorts from the last 180 days.
+--   Useful when you care about product engagement, not just visits.
+--
+-- SOURCE TABLES
+--   platform.AgentGraphExecution  — Execution records
+--
+-- OUTPUT COLUMNS
+--   Same pattern as retention_login_weekly.
+--   cohort_week_start = week of first execution (not first login)
+--
+-- EXAMPLE QUERIES
+--   -- Week-2 execution retention
+--   SELECT cohort_label, retention_rate_bounded
+--   FROM analytics.retention_execution_weekly
+--   WHERE user_lifetime_week = 2 ORDER BY cohort_week_start;
+-- =============================================================
+
+WITH params AS (SELECT 12::int AS max_weeks, (CURRENT_DATE - INTERVAL '180 days') AS cohort_start),
+events AS (
+  SELECT e."userId"::text AS user_id, e."createdAt"::timestamptz AS created_at,
+         DATE_TRUNC('week', e."createdAt")::date AS week_start
+  FROM platform."AgentGraphExecution" e WHERE e."userId" IS NOT NULL
+),
+first_exec AS (
+  SELECT user_id, MIN(created_at) AS first_exec_at,
+         DATE_TRUNC('week', MIN(created_at))::date AS cohort_week_start
+  FROM events GROUP BY 1
+  HAVING MIN(created_at) >= (SELECT cohort_start FROM params)
+),
+activity_weeks AS (SELECT DISTINCT user_id, week_start FROM events),
+user_week_age AS (
+  SELECT aw.user_id, fe.cohort_week_start,
+         ((aw.week_start - DATE_TRUNC('week',fe.first_exec_at)::date)/7)::int AS user_lifetime_week
+  FROM activity_weeks aw JOIN first_exec fe USING (user_id)
+  WHERE aw.week_start >= DATE_TRUNC('week',fe.first_exec_at)::date
+),
+bounded_counts AS (
+  SELECT cohort_week_start, user_lifetime_week, COUNT(DISTINCT user_id) AS active_users_bounded
+  FROM user_week_age WHERE user_lifetime_week >= 0 GROUP BY 1,2
+),
+last_active AS (
+  SELECT cohort_week_start, user_id, MAX(user_lifetime_week) AS last_active_week FROM user_week_age GROUP BY 1,2
+),
+unbounded_counts AS (
+  SELECT la.cohort_week_start, gs AS user_lifetime_week, COUNT(*) AS retained_users_unbounded
+  FROM last_active la
+  CROSS JOIN LATERAL generate_series(0, LEAST(la.last_active_week,(SELECT max_weeks FROM params))) gs
+  GROUP BY 1,2
+),
+cohort_sizes AS (SELECT cohort_week_start, COUNT(DISTINCT user_id) AS cohort_users FROM first_exec GROUP BY 1),
+cohort_caps AS (
+  SELECT cs.cohort_week_start, cs.cohort_users,
+         LEAST((SELECT max_weeks FROM params),
+               GREATEST(0,((DATE_TRUNC('week',CURRENT_DATE)::date-cs.cohort_week_start)/7)::int)) AS cap_weeks
+  FROM cohort_sizes cs
+),
+grid AS (
+  SELECT cc.cohort_week_start, gs AS user_lifetime_week, cc.cohort_users
+  FROM cohort_caps cc CROSS JOIN LATERAL generate_series(0, cc.cap_weeks) gs
+)
+SELECT
+  g.cohort_week_start,
+  TO_CHAR(g.cohort_week_start,'IYYY-"W"IW')                               AS cohort_label,
+  TO_CHAR(g.cohort_week_start,'IYYY-"W"IW')||' (n='||g.cohort_users||')'  AS cohort_label_n,
+  g.user_lifetime_week, g.cohort_users,
+  COALESCE(b.active_users_bounded,0)     AS active_users_bounded,
+  COALESCE(u.retained_users_unbounded,0) AS retained_users_unbounded,
+  CASE WHEN g.cohort_users>0 THEN COALESCE(b.active_users_bounded,0)::float/g.cohort_users END    AS retention_rate_bounded,
+  CASE WHEN g.cohort_users>0 THEN COALESCE(u.retained_users_unbounded,0)::float/g.cohort_users END AS retention_rate_unbounded,
+  CASE WHEN g.user_lifetime_week=0 THEN g.cohort_users ELSE 0 END         AS cohort_users_w0
+FROM grid g
+LEFT JOIN bounded_counts   b ON b.cohort_week_start=g.cohort_week_start AND b.user_lifetime_week=g.user_lifetime_week
+LEFT JOIN unbounded_counts u ON u.cohort_week_start=g.cohort_week_start AND u.user_lifetime_week=g.user_lifetime_week
+ORDER BY g.cohort_week_start, g.user_lifetime_week;

autogpt_platform/analytics/queries/retention_login_daily.sql

@@ -0,0 +1,94 @@
+-- =============================================================
+-- View: analytics.retention_login_daily
+-- Looker source alias: ds112  |  Charts: 1
+-- =============================================================
+-- DESCRIPTION
+--   Daily cohort retention based on login sessions.
+--   Same logic as retention_login_weekly but at day granularity,
+--   showing up to day 30 for cohorts from the last 90 days.
+--   Useful for analysing early activation (days 1-7) in detail.
+--
+-- SOURCE TABLES
+--   auth.sessions  — Login session records
+--
+-- OUTPUT COLUMNS (same pattern as retention_login_weekly)
+--   cohort_day_start          DATE     First day the cohort logged in
+--   cohort_label              TEXT     Date string (e.g. '2025-03-01')
+--   cohort_label_n            TEXT     Date + cohort size (e.g. '2025-03-01 (n=12)')
+--   user_lifetime_day         INT      Days since first login (0 = signup day)
+--   cohort_users              BIGINT   Total users in cohort
+--   active_users_bounded      BIGINT   Users active on exactly day k
+--   retained_users_unbounded  BIGINT   Users active any time on/after day k
+--   retention_rate_bounded    FLOAT    bounded / cohort_users
+--   retention_rate_unbounded  FLOAT    unbounded / cohort_users
+--   cohort_users_d0           BIGINT   cohort_users only at day 0, else 0 (safe to SUM)
+--
+-- EXAMPLE QUERIES
+--   -- Day-1 retention rate (came back next day)
+--   SELECT cohort_label, retention_rate_bounded AS d1_retention
+--   FROM analytics.retention_login_daily
+--   WHERE user_lifetime_day = 1 ORDER BY cohort_day_start;
+--
+--   -- Average retention curve across all cohorts
+--   SELECT user_lifetime_day,
+--          SUM(active_users_bounded)::float / NULLIF(SUM(cohort_users_d0), 0) AS avg_retention
+--   FROM analytics.retention_login_daily
+--   GROUP BY 1 ORDER BY 1;
+-- =============================================================
+
+WITH params AS (SELECT 30::int AS max_days, (CURRENT_DATE - INTERVAL '90 days')::date AS cohort_start),
+events AS (
+  SELECT s.user_id::text AS user_id, s.created_at::timestamptz AS created_at,
+         DATE_TRUNC('day', s.created_at)::date AS day_start
+  FROM auth.sessions s WHERE s.user_id IS NOT NULL
+),
+first_login AS (
+  SELECT user_id, MIN(created_at) AS first_login_time,
+         DATE_TRUNC('day', MIN(created_at))::date AS cohort_day_start
+  FROM events GROUP BY 1
+  HAVING MIN(created_at) >= (SELECT cohort_start FROM params)
+),
+activity_days AS (SELECT DISTINCT user_id, day_start FROM events),
+user_day_age AS (
+  SELECT ad.user_id, fl.cohort_day_start,
+         (ad.day_start - DATE_TRUNC('day', fl.first_login_time)::date)::int AS user_lifetime_day
+  FROM activity_days ad JOIN first_login fl USING (user_id)
+  WHERE ad.day_start >= DATE_TRUNC('day', fl.first_login_time)::date
+),
+bounded_counts AS (
+  SELECT cohort_day_start, user_lifetime_day, COUNT(DISTINCT user_id) AS active_users_bounded
+  FROM user_day_age WHERE user_lifetime_day >= 0 GROUP BY 1,2
+),
+last_active AS (
+  SELECT cohort_day_start, user_id, MAX(user_lifetime_day) AS last_active_day FROM user_day_age GROUP BY 1,2
+),
+unbounded_counts AS (
+  SELECT la.cohort_day_start, gs AS user_lifetime_day, COUNT(*) AS retained_users_unbounded
+  FROM last_active la
+  CROSS JOIN LATERAL generate_series(0, LEAST(la.last_active_day,(SELECT max_days FROM params))) gs
+  GROUP BY 1,2
+),
+cohort_sizes AS (SELECT cohort_day_start, COUNT(DISTINCT user_id) AS cohort_users FROM first_login GROUP BY 1),
+cohort_caps AS (
+  SELECT cs.cohort_day_start, cs.cohort_users,
+         LEAST((SELECT max_days FROM params), GREATEST(0,(CURRENT_DATE-cs.cohort_day_start)::int)) AS cap_days
+  FROM cohort_sizes cs
+),
+grid AS (
+  SELECT cc.cohort_day_start, gs AS user_lifetime_day, cc.cohort_users
+  FROM cohort_caps cc CROSS JOIN LATERAL generate_series(0, cc.cap_days) gs
+)
+SELECT
+  g.cohort_day_start,
+  TO_CHAR(g.cohort_day_start,'YYYY-MM-DD')                                  AS cohort_label,
+  TO_CHAR(g.cohort_day_start,'YYYY-MM-DD')||' (n='||g.cohort_users||')'     AS cohort_label_n,
+  g.user_lifetime_day, g.cohort_users,
+  COALESCE(b.active_users_bounded,0)     AS active_users_bounded,
+  COALESCE(u.retained_users_unbounded,0) AS retained_users_unbounded,
+  CASE WHEN g.cohort_users>0 THEN COALESCE(b.active_users_bounded,0)::float/g.cohort_users END    AS retention_rate_bounded,
+  CASE WHEN g.cohort_users>0 THEN COALESCE(u.retained_users_unbounded,0)::float/g.cohort_users END AS retention_rate_unbounded,
+  CASE WHEN g.user_lifetime_day=0 THEN g.cohort_users ELSE 0 END            AS cohort_users_d0
+FROM grid g
+LEFT JOIN bounded_counts   b ON b.cohort_day_start=g.cohort_day_start AND b.user_lifetime_day=g.user_lifetime_day
+LEFT JOIN unbounded_counts u ON u.cohort_day_start=g.cohort_day_start AND u.user_lifetime_day=g.user_lifetime_day
+ORDER BY g.cohort_day_start, g.user_lifetime_day;

autogpt_platform/analytics/queries/retention_login_onboarded_weekly.sql

@@ -0,0 +1,96 @@
+-- =============================================================
+-- View: analytics.retention_login_onboarded_weekly
+-- Looker source alias: ds101  |  Charts: 2
+-- =============================================================
+-- DESCRIPTION
+--   Weekly cohort retention from login sessions, restricted to
+--   users who "onboarded" — defined as running at least one
+--   agent within 365 days of their first login.
+--   Filters out users who signed up but never activated,
+--   giving a cleaner view of engaged-user retention.
+--
+-- SOURCE TABLES
+--   auth.sessions                  — Login session records
+--   platform.AgentGraphExecution   — Used to identify onboarders
+--
+-- OUTPUT COLUMNS
+--   Same as retention_login_weekly (cohort_week_start, user_lifetime_week,
+--   retention_rate_bounded, retention_rate_unbounded, etc.)
+--   Only difference: cohort is filtered to onboarded users only.
+--
+-- EXAMPLE QUERIES
+--   -- Compare week-4 retention: all users vs onboarded only
+--   SELECT 'all_users' AS segment, AVG(retention_rate_bounded) AS w4_retention
+--   FROM analytics.retention_login_weekly WHERE user_lifetime_week = 4
+--   UNION ALL
+--   SELECT 'onboarded', AVG(retention_rate_bounded)
+--   FROM analytics.retention_login_onboarded_weekly WHERE user_lifetime_week = 4;
+-- =============================================================
+
+WITH params AS (SELECT 12::int AS max_weeks, 365::int AS onboarding_window_days),
+events AS (
+  SELECT s.user_id::text AS user_id, s.created_at::timestamptz AS created_at,
+         DATE_TRUNC('week', s.created_at)::date AS week_start
+  FROM auth.sessions s WHERE s.user_id IS NOT NULL
+),
+first_login_all AS (
+  SELECT user_id, MIN(created_at) AS first_login_time,
+         DATE_TRUNC('week', MIN(created_at))::date AS cohort_week_start
+  FROM events GROUP BY 1
+),
+onboarders AS (
+  SELECT fl.user_id FROM first_login_all fl
+  WHERE EXISTS (
+    SELECT 1 FROM platform."AgentGraphExecution" e
+    WHERE e."userId"::text = fl.user_id
+      AND e."createdAt" >= fl.first_login_time
+      AND e."createdAt" < fl.first_login_time
+          + make_interval(days => (SELECT onboarding_window_days FROM params))
+  )
+),
+first_login AS (SELECT * FROM first_login_all WHERE user_id IN (SELECT user_id FROM onboarders)),
+activity_weeks AS (SELECT DISTINCT user_id, week_start FROM events),
+user_week_age AS (
+  SELECT aw.user_id, fl.cohort_week_start,
+         ((aw.week_start - DATE_TRUNC('week',fl.first_login_time)::date)/7)::int AS user_lifetime_week
+  FROM activity_weeks aw JOIN first_login fl USING (user_id)
+  WHERE aw.week_start >= DATE_TRUNC('week',fl.first_login_time)::date
+),
+bounded_counts AS (
+  SELECT cohort_week_start, user_lifetime_week, COUNT(DISTINCT user_id) AS active_users_bounded
+  FROM user_week_age WHERE user_lifetime_week >= 0 GROUP BY 1,2
+),
+last_active AS (
+  SELECT cohort_week_start, user_id, MAX(user_lifetime_week) AS last_active_week FROM user_week_age GROUP BY 1,2
+),
+unbounded_counts AS (
+  SELECT la.cohort_week_start, gs AS user_lifetime_week, COUNT(*) AS retained_users_unbounded
+  FROM last_active la
+  CROSS JOIN LATERAL generate_series(0, LEAST(la.last_active_week,(SELECT max_weeks FROM params))) gs
+  GROUP BY 1,2
+),
+cohort_sizes AS (SELECT cohort_week_start, COUNT(DISTINCT user_id) AS cohort_users FROM first_login GROUP BY 1),
+cohort_caps AS (
+  SELECT cs.cohort_week_start, cs.cohort_users,
+         LEAST((SELECT max_weeks FROM params),
+               GREATEST(0,((DATE_TRUNC('week',CURRENT_DATE)::date-cs.cohort_week_start)/7)::int)) AS cap_weeks
+  FROM cohort_sizes cs
+),
+grid AS (
+  SELECT cc.cohort_week_start, gs AS user_lifetime_week, cc.cohort_users
+  FROM cohort_caps cc CROSS JOIN LATERAL generate_series(0, cc.cap_weeks) gs
+)
+SELECT
+  g.cohort_week_start,
+  TO_CHAR(g.cohort_week_start,'IYYY-"W"IW')                               AS cohort_label,
+  TO_CHAR(g.cohort_week_start,'IYYY-"W"IW')||' (n='||g.cohort_users||')'  AS cohort_label_n,
+  g.user_lifetime_week, g.cohort_users,
+  COALESCE(b.active_users_bounded,0)     AS active_users_bounded,
+  COALESCE(u.retained_users_unbounded,0) AS retained_users_unbounded,
+  CASE WHEN g.cohort_users>0 THEN COALESCE(b.active_users_bounded,0)::float/g.cohort_users END    AS retention_rate_bounded,
+  CASE WHEN g.cohort_users>0 THEN COALESCE(u.retained_users_unbounded,0)::float/g.cohort_users END AS retention_rate_unbounded,
+  CASE WHEN g.user_lifetime_week=0 THEN g.cohort_users ELSE 0 END         AS cohort_users_w0
+FROM grid g
+LEFT JOIN bounded_counts   b ON b.cohort_week_start=g.cohort_week_start AND b.user_lifetime_week=g.user_lifetime_week
+LEFT JOIN unbounded_counts u ON u.cohort_week_start=g.cohort_week_start AND u.user_lifetime_week=g.user_lifetime_week
+ORDER BY g.cohort_week_start, g.user_lifetime_week;

autogpt_platform/analytics/queries/retention_login_weekly.sql

@@ -0,0 +1,103 @@
+-- =============================================================
+-- View: analytics.retention_login_weekly
+-- Looker source alias: ds83  |  Charts: 2
+-- =============================================================
+-- DESCRIPTION
+--   Weekly cohort retention based on login sessions.
+--   Users are grouped by the ISO week of their first ever login.
+--   For each cohort × lifetime-week combination, outputs both:
+--     - bounded rate: % active in exactly that week
+--     - unbounded rate: % who were ever active on or after that week
+--   Weeks are capped to the cohort's actual age (no future data points).
+--
+-- SOURCE TABLES
+--   auth.sessions  — Login session records
+--
+-- HOW TO READ THE OUTPUT
+--   cohort_week_start   The Monday of the week users first logged in
+--   user_lifetime_week  0 = signup week, 1 = one week later, etc.
+--   retention_rate_bounded   = active_users_bounded / cohort_users
+--   retention_rate_unbounded = retained_users_unbounded / cohort_users
+--
+-- OUTPUT COLUMNS
+--   cohort_week_start         DATE     First day of the cohort's signup week
+--   cohort_label              TEXT     ISO week label (e.g. '2025-W01')
+--   cohort_label_n            TEXT     ISO week label with cohort size (e.g. '2025-W01 (n=42)')
+--   user_lifetime_week        INT      Weeks since first login (0 = signup week)
+--   cohort_users              BIGINT   Total users in this cohort (denominator)
+--   active_users_bounded      BIGINT   Users active in exactly week k
+--   retained_users_unbounded  BIGINT   Users active any time on/after week k
+--   retention_rate_bounded    FLOAT    bounded active / cohort_users
+--   retention_rate_unbounded  FLOAT    unbounded retained / cohort_users
+--   cohort_users_w0           BIGINT   cohort_users only at week 0, else 0 (safe to SUM in pivot tables)
+--
+-- EXAMPLE QUERIES
+--   -- Week-1 retention rate per cohort
+--   SELECT cohort_label, retention_rate_bounded AS w1_retention
+--   FROM analytics.retention_login_weekly
+--   WHERE user_lifetime_week = 1
+--   ORDER BY cohort_week_start;
+--
+--   -- Overall average retention curve (all cohorts combined)
+--   SELECT user_lifetime_week,
+--          SUM(active_users_bounded)::float / NULLIF(SUM(cohort_users_w0), 0) AS avg_retention
+--   FROM analytics.retention_login_weekly
+--   GROUP BY 1 ORDER BY 1;
+-- =============================================================
+
+WITH params AS (SELECT 12::int AS max_weeks),
+events AS (
+  SELECT s.user_id::text AS user_id, s.created_at::timestamptz AS created_at,
+         DATE_TRUNC('week', s.created_at)::date AS week_start
+  FROM auth.sessions s WHERE s.user_id IS NOT NULL
+),
+first_login AS (
+  SELECT user_id, MIN(created_at) AS first_login_time,
+         DATE_TRUNC('week', MIN(created_at))::date AS cohort_week_start
+  FROM events GROUP BY 1
+),
+activity_weeks AS (SELECT DISTINCT user_id, week_start FROM events),
+user_week_age AS (
+  SELECT aw.user_id, fl.cohort_week_start,
+         ((aw.week_start - DATE_TRUNC('week', fl.first_login_time)::date) / 7)::int AS user_lifetime_week
+  FROM activity_weeks aw JOIN first_login fl USING (user_id)
+  WHERE aw.week_start >= DATE_TRUNC('week', fl.first_login_time)::date
+),
+bounded_counts AS (
+  SELECT cohort_week_start, user_lifetime_week, COUNT(DISTINCT user_id) AS active_users_bounded
+  FROM user_week_age WHERE user_lifetime_week >= 0 GROUP BY 1,2
+),
+last_active AS (
+  SELECT cohort_week_start, user_id, MAX(user_lifetime_week) AS last_active_week FROM user_week_age GROUP BY 1,2
+),
+unbounded_counts AS (
+  SELECT la.cohort_week_start, gs AS user_lifetime_week, COUNT(*) AS retained_users_unbounded
+  FROM last_active la
+  CROSS JOIN LATERAL generate_series(0, LEAST(la.last_active_week,(SELECT max_weeks FROM params))) gs
+  GROUP BY 1,2
+),
+cohort_sizes AS (SELECT cohort_week_start, COUNT(DISTINCT user_id) AS cohort_users FROM first_login GROUP BY 1),
+cohort_caps AS (
+  SELECT cs.cohort_week_start, cs.cohort_users,
+         LEAST((SELECT max_weeks FROM params),
+               GREATEST(0,((DATE_TRUNC('week',CURRENT_DATE)::date - cs.cohort_week_start)/7)::int)) AS cap_weeks
+  FROM cohort_sizes cs
+),
+grid AS (
+  SELECT cc.cohort_week_start, gs AS user_lifetime_week, cc.cohort_users
+  FROM cohort_caps cc CROSS JOIN LATERAL generate_series(0, cc.cap_weeks) gs
+)
+SELECT
+  g.cohort_week_start,
+  TO_CHAR(g.cohort_week_start,'IYYY-"W"IW')                                    AS cohort_label,
+  TO_CHAR(g.cohort_week_start,'IYYY-"W"IW')||' (n='||g.cohort_users||')'       AS cohort_label_n,
+  g.user_lifetime_week, g.cohort_users,
+  COALESCE(b.active_users_bounded,0)     AS active_users_bounded,
+  COALESCE(u.retained_users_unbounded,0) AS retained_users_unbounded,
+  CASE WHEN g.cohort_users>0 THEN COALESCE(b.active_users_bounded,0)::float/g.cohort_users END    AS retention_rate_bounded,
+  CASE WHEN g.cohort_users>0 THEN COALESCE(u.retained_users_unbounded,0)::float/g.cohort_users END AS retention_rate_unbounded,
+  CASE WHEN g.user_lifetime_week=0 THEN g.cohort_users ELSE 0 END               AS cohort_users_w0
+FROM grid g
+LEFT JOIN bounded_counts   b ON b.cohort_week_start=g.cohort_week_start AND b.user_lifetime_week=g.user_lifetime_week
+LEFT JOIN unbounded_counts u ON u.cohort_week_start=g.cohort_week_start AND u.user_lifetime_week=g.user_lifetime_week
+ORDER BY g.cohort_week_start, g.user_lifetime_week

autogpt_platform/analytics/queries/user_block_spending.sql

@@ -0,0 +1,71 @@
+-- =============================================================
+-- View: analytics.user_block_spending
+-- Looker source alias: ds6  |  Charts: 5
+-- =============================================================
+-- DESCRIPTION
+--   One row per credit transaction (last 90 days).
+--   Shows how users spend credits broken down by block type,
+--   LLM provider and model.  Joins node execution stats for
+--   token-level detail.
+--
+-- SOURCE TABLES
+--   platform.CreditTransaction   — Credit debit/credit records
+--   platform.AgentNodeExecution  — Node execution stats (for token counts)
+--
+-- OUTPUT COLUMNS
+--   transactionKey        TEXT         Unique transaction identifier
+--   userId                TEXT         User who was charged
+--   amount                DECIMAL      Credit amount (positive = credit, negative = debit)
+--   negativeAmount        DECIMAL      amount * -1 (convenience for spend charts)
+--   transactionType       TEXT         Transaction type (e.g. 'USAGE', 'REFUND', 'TOP_UP')
+--   transactionTime       TIMESTAMPTZ  When the transaction was recorded
+--   blockId               TEXT         Block UUID that triggered the spend
+--   blockName             TEXT         Human-readable block name
+--   llm_provider          TEXT         LLM provider (e.g. 'openai', 'anthropic')
+--   llm_model             TEXT         Model name (e.g. 'gpt-4o', 'claude-3-5-sonnet')
+--   node_exec_id          TEXT         Linked node execution UUID
+--   llm_call_count        INT          LLM API calls made in that execution
+--   llm_retry_count       INT          LLM retries in that execution
+--   llm_input_token_count INT          Input tokens consumed
+--   llm_output_token_count INT         Output tokens produced
+--
+-- WINDOW
+--   Rolling 90 days (createdAt > CURRENT_DATE - 90 days)
+--
+-- EXAMPLE QUERIES
+--   -- Total spend per user (last 90 days)
+--   SELECT "userId", SUM("negativeAmount") AS total_spent
+--   FROM analytics.user_block_spending
+--   WHERE "transactionType" = 'USAGE'
+--   GROUP BY 1 ORDER BY total_spent DESC;
+--
+--   -- Spend by LLM provider + model
+--   SELECT "llm_provider", "llm_model",
+--          SUM("negativeAmount") AS total_cost,
+--          SUM("llm_input_token_count") AS input_tokens,
+--          SUM("llm_output_token_count") AS output_tokens
+--   FROM analytics.user_block_spending
+--   WHERE "llm_provider" IS NOT NULL
+--   GROUP BY 1, 2 ORDER BY total_cost DESC;
+-- =============================================================
+
+SELECT
+    c."transactionKey"                                        AS transactionKey,
+    c."userId"                                                AS userId,
+    c."amount"                                                AS amount,
+    c."amount" * -1                                           AS negativeAmount,
+    c."type"                                                  AS transactionType,
+    c."createdAt"                                             AS transactionTime,
+    c.metadata->>'block_id'                                   AS blockId,
+    c.metadata->>'block'                                      AS blockName,
+    c.metadata->'input'->'credentials'->>'provider'           AS llm_provider,
+    c.metadata->'input'->>'model'                             AS llm_model,
+    c.metadata->>'node_exec_id'                               AS node_exec_id,
+    (ne."stats"->>'llm_call_count')::int                       AS llm_call_count,
+    (ne."stats"->>'llm_retry_count')::int                      AS llm_retry_count,
+    (ne."stats"->>'input_token_count')::int                    AS llm_input_token_count,
+    (ne."stats"->>'output_token_count')::int                   AS llm_output_token_count
+FROM platform."CreditTransaction" c
+LEFT JOIN platform."AgentNodeExecution" ne
+       ON (c.metadata->>'node_exec_id') = ne."id"::text
+WHERE c."createdAt" > CURRENT_DATE - INTERVAL '90 days'

autogpt_platform/analytics/queries/user_onboarding.sql

@@ -0,0 +1,45 @@
+-- =============================================================
+-- View: analytics.user_onboarding
+-- Looker source alias: ds68  |  Charts: 3
+-- =============================================================
+-- DESCRIPTION
+--   One row per user onboarding record.  Contains the user's
+--   stated usage reason, selected integrations, completed
+--   onboarding steps and optional first agent selection.
+--   Full history (no date filter) since onboarding happens
+--   once per user.
+--
+-- SOURCE TABLES
+--   platform.UserOnboarding  — Onboarding state per user
+--
+-- OUTPUT COLUMNS
+--   id                            TEXT         Onboarding record UUID
+--   createdAt                     TIMESTAMPTZ  When onboarding started
+--   updatedAt                     TIMESTAMPTZ  Last update to onboarding state
+--   usageReason                   TEXT         Why user signed up (e.g. 'work', 'personal')
+--   integrations                  TEXT[]       Array of integration names the user selected
+--   userId                        TEXT         User UUID
+--   completedSteps                TEXT[]       Array of onboarding step enums completed
+--   selectedStoreListingVersionId TEXT         First marketplace agent the user chose (if any)
+--
+-- EXAMPLE QUERIES
+--   -- Usage reason breakdown
+--   SELECT "usageReason", COUNT(*) FROM analytics.user_onboarding GROUP BY 1;
+--
+--   -- Completion rate per step
+--   SELECT step, COUNT(*) AS users_completed
+--   FROM analytics.user_onboarding
+--   CROSS JOIN LATERAL UNNEST("completedSteps") AS step
+--   GROUP BY 1 ORDER BY users_completed DESC;
+-- =============================================================
+
+SELECT
+    id,
+    "createdAt",
+    "updatedAt",
+    "usageReason",
+    integrations,
+    "userId",
+    "completedSteps",
+    "selectedStoreListingVersionId"
+FROM platform."UserOnboarding"

autogpt_platform/analytics/queries/user_onboarding_funnel.sql

@@ -0,0 +1,100 @@
+-- =============================================================
+-- View: analytics.user_onboarding_funnel
+-- Looker source alias: ds74  |  Charts: 1
+-- =============================================================
+-- DESCRIPTION
+--   Pre-aggregated onboarding funnel showing how many users
+--   completed each step and the drop-off percentage from the
+--   previous step.  One row per onboarding step (all 22 steps
+--   always present, even with 0 completions — prevents sparse
+--   gaps from making LAG compare the wrong predecessors).
+--
+-- SOURCE TABLES
+--   platform.UserOnboarding  — Onboarding records with completedSteps array
+--
+-- OUTPUT COLUMNS
+--   step             TEXT     Onboarding step enum name (e.g. 'WELCOME', 'CONGRATS')
+--   step_order       INT      Numeric position in the funnel (1=first, 22=last)
+--   users_completed  BIGINT   Distinct users who completed this step
+--   pct_from_prev    NUMERIC  % of users from the previous step who reached this one
+--
+-- STEP ORDER
+--   1  WELCOME               9  MARKETPLACE_VISIT     17  SCHEDULE_AGENT
+--   2  USAGE_REASON         10  MARKETPLACE_ADD_AGENT  18  RUN_AGENTS
+--   3  INTEGRATIONS         11  MARKETPLACE_RUN_AGENT  19  RUN_3_DAYS
+--   4  AGENT_CHOICE         12  BUILDER_OPEN           20  TRIGGER_WEBHOOK
+--   5  AGENT_NEW_RUN        13  BUILDER_SAVE_AGENT     21  RUN_14_DAYS
+--   6  AGENT_INPUT          14  BUILDER_RUN_AGENT      22  RUN_AGENTS_100
+--   7  CONGRATS             15  VISIT_COPILOT
+--   8  GET_RESULTS          16  RE_RUN_AGENT
+--
+-- WINDOW
+--   Users who started onboarding in the last 90 days
+--
+-- EXAMPLE QUERIES
+--   -- Full funnel
+--   SELECT * FROM analytics.user_onboarding_funnel ORDER BY step_order;
+--
+--   -- Biggest drop-off point
+--   SELECT step, pct_from_prev FROM analytics.user_onboarding_funnel
+--   ORDER BY pct_from_prev ASC LIMIT 3;
+-- =============================================================
+
+WITH all_steps AS (
+  -- Complete ordered grid of all 22 steps so zero-completion steps
+  -- are always present, keeping LAG comparisons correct.
+  SELECT step_name, step_order
+  FROM (VALUES
+    ('WELCOME',               1),
+    ('USAGE_REASON',          2),
+    ('INTEGRATIONS',          3),
+    ('AGENT_CHOICE',          4),
+    ('AGENT_NEW_RUN',         5),
+    ('AGENT_INPUT',           6),
+    ('CONGRATS',              7),
+    ('GET_RESULTS',           8),
+    ('MARKETPLACE_VISIT',     9),
+    ('MARKETPLACE_ADD_AGENT', 10),
+    ('MARKETPLACE_RUN_AGENT', 11),
+    ('BUILDER_OPEN',          12),
+    ('BUILDER_SAVE_AGENT',    13),
+    ('BUILDER_RUN_AGENT',     14),
+    ('VISIT_COPILOT',         15),
+    ('RE_RUN_AGENT',          16),
+    ('SCHEDULE_AGENT',        17),
+    ('RUN_AGENTS',            18),
+    ('RUN_3_DAYS',            19),
+    ('TRIGGER_WEBHOOK',       20),
+    ('RUN_14_DAYS',           21),
+    ('RUN_AGENTS_100',        22)
+  ) AS t(step_name, step_order)
+),
+raw AS (
+  SELECT
+      u."userId",
+      step_txt::text AS step
+  FROM platform."UserOnboarding" u
+  CROSS JOIN LATERAL UNNEST(u."completedSteps") AS step_txt
+  WHERE u."createdAt" >= CURRENT_DATE - INTERVAL '90 days'
+),
+step_counts AS (
+  SELECT step, COUNT(DISTINCT "userId") AS users_completed
+  FROM raw GROUP BY step
+),
+funnel AS (
+  SELECT
+      a.step_name                          AS step,
+      a.step_order,
+      COALESCE(sc.users_completed, 0)      AS users_completed,
+      ROUND(
+        100.0 * COALESCE(sc.users_completed, 0)
+        / NULLIF(
+            LAG(COALESCE(sc.users_completed, 0)) OVER (ORDER BY a.step_order),
+            0
+          ),
+        2
+      )                                    AS pct_from_prev
+  FROM all_steps a
+  LEFT JOIN step_counts sc ON sc.step = a.step_name
+)
+SELECT * FROM funnel ORDER BY step_order

autogpt_platform/analytics/queries/user_onboarding_integration.sql

@@ -0,0 +1,41 @@
+-- =============================================================
+-- View: analytics.user_onboarding_integration
+-- Looker source alias: ds75  |  Charts: 1
+-- =============================================================
+-- DESCRIPTION
+--   Pre-aggregated count of users who selected each integration
+--   during onboarding.  One row per integration type, sorted
+--   by popularity.
+--
+-- SOURCE TABLES
+--   platform.UserOnboarding  — integrations array column
+--
+-- OUTPUT COLUMNS
+--   integration            TEXT    Integration name (e.g. 'github', 'slack', 'notion')
+--   users_with_integration BIGINT  Distinct users who selected this integration
+--
+-- WINDOW
+--   Users who started onboarding in the last 90 days
+--
+-- EXAMPLE QUERIES
+--   -- Full integration popularity ranking
+--   SELECT * FROM analytics.user_onboarding_integration;
+--
+--   -- Top 5 integrations
+--   SELECT * FROM analytics.user_onboarding_integration LIMIT 5;
+-- =============================================================
+
+WITH exploded AS (
+  SELECT
+      u."userId" AS user_id,
+      UNNEST(u."integrations") AS integration
+  FROM platform."UserOnboarding" u
+  WHERE u."createdAt" >= CURRENT_DATE - INTERVAL '90 days'
+)
+SELECT
+    integration,
+    COUNT(DISTINCT user_id) AS users_with_integration
+FROM exploded
+WHERE integration IS NOT NULL AND integration <> ''
+GROUP BY integration
+ORDER BY users_with_integration DESC

autogpt_platform/analytics/queries/users_activities.sql

@@ -0,0 +1,145 @@
+-- =============================================================
+-- View: analytics.users_activities
+-- Looker source alias: ds56  |  Charts: 5
+-- =============================================================
+-- DESCRIPTION
+--   One row per user with lifetime activity summary.
+--   Joins login sessions with agent graphs, executions and
+--   node-level runs to give a full picture of how engaged
+--   each user is.  Includes a convenience flag for 7-day
+--   activation (did the user return at least 7 days after
+--   their first login?).
+--
+-- SOURCE TABLES
+--   auth.sessions                    — Login/session records
+--   platform.AgentGraph              — Graphs (agents) built by the user
+--   platform.AgentGraphExecution     — Agent run history
+--   platform.AgentNodeExecution      — Individual block execution history
+--
+-- PERFORMANCE NOTE
+--   Each CTE aggregates its own table independently by userId.
+--   This avoids the fan-out that occurs when driving every join
+--   from user_logins across the two largest tables
+--   (AgentGraphExecution and AgentNodeExecution).
+--
+-- OUTPUT COLUMNS
+--   user_id                   TEXT         Supabase user UUID
+--   first_login_time          TIMESTAMPTZ  First ever session created_at
+--   last_login_time           TIMESTAMPTZ  Most recent session created_at
+--   last_visit_time           TIMESTAMPTZ  Max of last refresh or login
+--   last_agent_save_time      TIMESTAMPTZ  Last time user saved an agent graph
+--   agent_count               BIGINT       Number of distinct active graphs built (0 if none)
+--   first_agent_run_time      TIMESTAMPTZ  First ever graph execution
+--   last_agent_run_time       TIMESTAMPTZ  Most recent graph execution
+--   unique_agent_runs         BIGINT       Distinct agent graphs ever run (0 if none)
+--   agent_runs                BIGINT       Total graph execution count (0 if none)
+--   node_execution_count      BIGINT       Total node executions across all runs
+--   node_execution_failed     BIGINT       Node executions with FAILED status
+--   node_execution_completed  BIGINT       Node executions with COMPLETED status
+--   node_execution_terminated BIGINT       Node executions with TERMINATED status
+--   node_execution_queued     BIGINT       Node executions with QUEUED status
+--   node_execution_running    BIGINT       Node executions with RUNNING status
+--   is_active_after_7d        INT          1=returned after day 7, 0=did not, NULL=too early to tell
+--   node_execution_incomplete BIGINT       Node executions with INCOMPLETE status
+--   node_execution_review     BIGINT       Node executions with REVIEW status
+--
+-- EXAMPLE QUERIES
+--   -- Users who ran at least one agent and returned after 7 days
+--   SELECT COUNT(*) FROM analytics.users_activities
+--   WHERE agent_runs > 0 AND is_active_after_7d = 1;
+--
+--   -- Top 10 most active users by agent runs
+--   SELECT user_id, agent_runs, node_execution_count
+--   FROM analytics.users_activities
+--   ORDER BY agent_runs DESC LIMIT 10;
+--
+--   -- 7-day activation rate
+--   SELECT
+--     SUM(CASE WHEN is_active_after_7d = 1 THEN 1 ELSE 0 END)::float
+--     / NULLIF(COUNT(CASE WHEN is_active_after_7d IS NOT NULL THEN 1 END), 0)
+--     AS activation_rate
+--   FROM analytics.users_activities;
+-- =============================================================
+
+WITH user_logins AS (
+  SELECT
+    user_id::text                                    AS user_id,
+    MIN(created_at)                                  AS first_login_time,
+    MAX(created_at)                                  AS last_login_time,
+    GREATEST(
+      MAX(refreshed_at)::timestamptz,
+      MAX(created_at)::timestamptz
+    )                                                AS last_visit_time
+  FROM auth.sessions
+  GROUP BY user_id
+),
+user_agents AS (
+  -- Aggregate AgentGraph directly by userId (no fan-out from user_logins)
+  SELECT
+    "userId"::text                AS user_id,
+    MAX("updatedAt")              AS last_agent_save_time,
+    COUNT(DISTINCT "id")          AS agent_count
+  FROM platform."AgentGraph"
+  WHERE "isActive"
+  GROUP BY "userId"
+),
+user_graph_runs AS (
+  -- Aggregate AgentGraphExecution directly by userId
+  SELECT
+    "userId"::text                        AS user_id,
+    MIN("createdAt")                      AS first_agent_run_time,
+    MAX("createdAt")                      AS last_agent_run_time,
+    COUNT(DISTINCT "agentGraphId")        AS unique_agent_runs,
+    COUNT("id")                           AS agent_runs
+  FROM platform."AgentGraphExecution"
+  GROUP BY "userId"
+),
+user_node_runs AS (
+  -- Aggregate AgentNodeExecution directly; resolve userId via a
+  -- single join to AgentGraphExecution instead of fanning out from
+  -- user_logins through both large tables.
+  SELECT
+    g."userId"::text                                                   AS user_id,
+    COUNT(*)                                                           AS node_execution_count,
+    COUNT(*) FILTER (WHERE n."executionStatus" = 'FAILED')             AS node_execution_failed,
+    COUNT(*) FILTER (WHERE n."executionStatus" = 'COMPLETED')          AS node_execution_completed,
+    COUNT(*) FILTER (WHERE n."executionStatus" = 'TERMINATED')         AS node_execution_terminated,
+    COUNT(*) FILTER (WHERE n."executionStatus" = 'QUEUED')             AS node_execution_queued,
+    COUNT(*) FILTER (WHERE n."executionStatus" = 'RUNNING')            AS node_execution_running,
+    COUNT(*) FILTER (WHERE n."executionStatus" = 'INCOMPLETE')         AS node_execution_incomplete,
+    COUNT(*) FILTER (WHERE n."executionStatus" = 'REVIEW')             AS node_execution_review
+  FROM platform."AgentNodeExecution" n
+  JOIN platform."AgentGraphExecution" g
+    ON g."id" = n."agentGraphExecutionId"
+  GROUP BY g."userId"
+)
+SELECT
+  ul.user_id,
+  ul.first_login_time,
+  ul.last_login_time,
+  ul.last_visit_time,
+  ua.last_agent_save_time,
+  COALESCE(ua.agent_count, 0)             AS agent_count,
+  gr.first_agent_run_time,
+  gr.last_agent_run_time,
+  COALESCE(gr.unique_agent_runs, 0)       AS unique_agent_runs,
+  COALESCE(gr.agent_runs, 0)              AS agent_runs,
+  COALESCE(nr.node_execution_count, 0)      AS node_execution_count,
+  COALESCE(nr.node_execution_failed, 0)     AS node_execution_failed,
+  COALESCE(nr.node_execution_completed, 0)  AS node_execution_completed,
+  COALESCE(nr.node_execution_terminated, 0) AS node_execution_terminated,
+  COALESCE(nr.node_execution_queued, 0)     AS node_execution_queued,
+  COALESCE(nr.node_execution_running, 0)    AS node_execution_running,
+  CASE
+    WHEN ul.first_login_time < NOW() - INTERVAL '7 days'
+     AND ul.last_visit_time  >= ul.first_login_time + INTERVAL '7 days' THEN 1
+    WHEN ul.first_login_time < NOW() - INTERVAL '7 days'
+     AND ul.last_visit_time  <  ul.first_login_time + INTERVAL '7 days' THEN 0
+    ELSE NULL
+  END AS is_active_after_7d,
+  COALESCE(nr.node_execution_incomplete, 0) AS node_execution_incomplete,
+  COALESCE(nr.node_execution_review, 0)     AS node_execution_review
+FROM user_logins ul
+LEFT JOIN user_agents     ua ON ul.user_id = ua.user_id
+LEFT JOIN user_graph_runs gr ON ul.user_id = gr.user_id
+LEFT JOIN user_node_runs  nr ON ul.user_id = nr.user_id

autogpt_platform/backend/.env.default

@@ -37,6 +37,10 @@ JWT_VERIFY_KEY=your-super-secret-jwt-token-with-at-least-32-characters-long
 ENCRYPTION_KEY=dvziYgz0KSK8FENhju0ZYi8-fRTfAdlz6YLhdB_jhNw=
 UNSUBSCRIBE_SECRET_KEY=HlP8ivStJjmbf6NKi78m_3FnOogut0t5ckzjsIqeaio=
 
+## ===== SIGNUP / INVITE GATE ===== ##
+# Set to true to require an invite before users can sign up
+ENABLE_INVITE_GATE=false
+
 ## ===== IMPORTANT OPTIONAL CONFIGURATION ===== ##
 # Platform URLs (set these for webhooks and OAuth to work)
 PLATFORM_BASE_URL=http://localhost:8000
@@ -190,5 +194,8 @@ ZEROBOUNCE_API_KEY=
 POSTHOG_API_KEY=
 POSTHOG_HOST=https://eu.i.posthog.com
 
+# Tally Form Integration (pre-populate business understanding on signup)
+TALLY_API_KEY=
+
 # Other Services
 AUTOMOD_API_KEY=

autogpt_platform/backend/CLAUDE.md

@@ -58,10 +58,31 @@ poetry run pytest path/to/test.py --snapshot-update
 - **Authentication**: JWT-based with Supabase integration
 - **Security**: Cache protection middleware prevents sensitive data caching in browsers/proxies
 
+## Code Style
+
+- **Top-level imports only** — no local/inner imports (lazy imports only for heavy optional deps like `openpyxl`)
+- **No duck typing** — no `hasattr`/`getattr`/`isinstance` for type dispatch; use typed interfaces/unions/protocols
+- **Pydantic models** over dataclass/namedtuple/dict for structured data
+- **No linter suppressors** — no `# type: ignore`, `# noqa`, `# pyright: ignore`; fix the type/code
+- **List comprehensions** over manual loop-and-append
+- **Early return** — guard clauses first, avoid deep nesting
+- **Lazy `%s` logging** — `logger.info("Processing %s items", count)` not `logger.info(f"Processing {count} items")`
+- **Sanitize error paths** — `os.path.basename()` in error messages to avoid leaking directory structure
+- **TOCTOU awareness** — avoid check-then-act patterns for file access and credit charging
+- **`Security()` vs `Depends()`** — use `Security()` for auth deps to get proper OpenAPI security spec
+- **Redis pipelines** — `transaction=True` for atomicity on multi-step operations
+- **`max(0, value)` guards** — for computed values that should never be negative
+- **SSE protocol** — `data:` lines for frontend-parsed events (must match Zod schema), `: comment` lines for heartbeats/status
+- **File length** — keep files under ~300 lines; if a file grows beyond this, split by responsibility (e.g. extract helpers, models, or a sub-module into a new file). Never keep appending to a long file.
+- **Function length** — keep functions under ~40 lines; extract named helpers when a function grows longer. Long functions are a sign of mixed concerns, not complexity.
+
 ## Testing Approach
 
 - Uses pytest with snapshot testing for API responses
 - Test files are colocated with source files (`*_test.py`)
+- Mock at boundaries — mock where the symbol is **used**, not where it's **defined**
+- After refactoring, update mock targets to match new module paths
+- Use `AsyncMock` for async functions (`from unittest.mock import AsyncMock`)
 
 ## Database Schema
 

autogpt_platform/backend/Dockerfile

@@ -53,63 +53,6 @@ COPY autogpt_platform/backend/backend/data/partial_types.py ./backend/data/parti
 COPY autogpt_platform/backend/gen_prisma_types_stub.py ./
 RUN poetry run prisma generate && poetry run gen-prisma-stub
 
-# ============================== BACKEND SERVER ============================== #
-
-FROM debian:13-slim AS server
-
-WORKDIR /app
-
-ENV POETRY_HOME=/opt/poetry \
-    POETRY_NO_INTERACTION=1 \
-    POETRY_VIRTUALENVS_CREATE=true \
-    POETRY_VIRTUALENVS_IN_PROJECT=true \
-    DEBIAN_FRONTEND=noninteractive
-ENV PATH=/opt/poetry/bin:$PATH
-
-# Install Python, FFmpeg, ImageMagick, and CLI tools for agent use.
-# bubblewrap provides OS-level sandbox (whitelist-only FS + no network)
-# for the bash_exec MCP tool.
-# Using --no-install-recommends saves ~650MB by skipping unnecessary deps like llvm, mesa, etc.
-RUN apt-get update && apt-get install -y --no-install-recommends \
-    python3.13 \
-    python3-pip \
-    ffmpeg \
-    imagemagick \
-    jq \
-    ripgrep \
-    tree \
-    bubblewrap \
-    && rm -rf /var/lib/apt/lists/*
-
-COPY --from=builder /usr/local/lib/python3* /usr/local/lib/python3*
-COPY --from=builder /usr/local/bin/poetry /usr/local/bin/poetry
-# Copy Node.js installation for Prisma
-COPY --from=builder /usr/bin/node /usr/bin/node
-COPY --from=builder /usr/lib/node_modules /usr/lib/node_modules
-COPY --from=builder /usr/bin/npm /usr/bin/npm
-COPY --from=builder /usr/bin/npx /usr/bin/npx
-COPY --from=builder /root/.cache/prisma-python/binaries /root/.cache/prisma-python/binaries
-
-WORKDIR /app/autogpt_platform/backend
-
-# Copy only the .venv from builder (not the entire /app directory)
-# The .venv includes the generated Prisma client
-COPY --from=builder /app/autogpt_platform/backend/.venv ./.venv
-ENV PATH="/app/autogpt_platform/backend/.venv/bin:$PATH"
-
-# Copy dependency files + autogpt_libs (path dependency)
-COPY autogpt_platform/autogpt_libs /app/autogpt_platform/autogpt_libs
-COPY autogpt_platform/backend/poetry.lock autogpt_platform/backend/pyproject.toml ./
-
-# Copy backend code + docs (for Copilot docs search)
-COPY autogpt_platform/backend ./
-COPY docs /app/docs
-RUN poetry install --no-ansi --only-root
-
-ENV PORT=8000
-
-CMD ["poetry", "run", "rest"]
-
 # =============================== DB MIGRATOR =============================== #
 
 # Lightweight migrate stage - only needs Prisma CLI, not full Python environment
@@ -141,3 +84,75 @@ COPY autogpt_platform/backend/schema.prisma ./
 COPY autogpt_platform/backend/backend/data/partial_types.py ./backend/data/partial_types.py
 COPY autogpt_platform/backend/gen_prisma_types_stub.py ./
 COPY autogpt_platform/backend/migrations ./migrations
+
+# ============================== BACKEND SERVER ============================== #
+
+FROM debian:13-slim AS server
+
+WORKDIR /app
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+# Install Python, FFmpeg, ImageMagick, and CLI tools for agent use.
+# bubblewrap provides OS-level sandbox (whitelist-only FS + no network)
+# for the bash_exec MCP tool (fallback when E2B is not configured).
+# Using --no-install-recommends saves ~650MB by skipping unnecessary deps like llvm, mesa, etc.
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    python3.13 \
+    python3-pip \
+    ffmpeg \
+    imagemagick \
+    jq \
+    ripgrep \
+    tree \
+    bubblewrap \
+    && rm -rf /var/lib/apt/lists/*
+
+# Copy poetry (build-time only, for `poetry install --only-root` to create entry points)
+COPY --from=builder /usr/local/lib/python3* /usr/local/lib/python3*
+COPY --from=builder /usr/local/bin/poetry /usr/local/bin/poetry
+# Copy Node.js installation for Prisma and agent-browser.
+# npm/npx are symlinks in the builder (-> ../lib/node_modules/npm/bin/*-cli.js);
+# COPY resolves them to regular files, breaking require() paths.  Recreate as
+# proper symlinks so npm/npx can find their modules.
+COPY --from=builder /usr/bin/node /usr/bin/node
+COPY --from=builder /usr/lib/node_modules /usr/lib/node_modules
+RUN ln -s ../lib/node_modules/npm/bin/npm-cli.js /usr/bin/npm \
+    && ln -s ../lib/node_modules/npm/bin/npx-cli.js /usr/bin/npx
+COPY --from=builder /root/.cache/prisma-python/binaries /root/.cache/prisma-python/binaries
+
+# Install agent-browser (Copilot browser tool) + Chromium runtime dependencies.
+# These are the runtime libraries Chromium/Playwright needs on Debian 13 (trixie).
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    libnss3 libnspr4 libatk1.0-0 libatk-bridge2.0-0 libcups2 libdrm2 \
+    libdbus-1-3 libxkbcommon0 libatspi2.0-0t64 libxcomposite1 libxdamage1 \
+    libxfixes3 libxrandr2 libgbm1 libasound2t64 libpango-1.0-0 libcairo2 \
+    libx11-6 libx11-xcb1 libxcb1 libxext6 libglib2.0-0t64 \
+    fonts-liberation libfontconfig1 \
+    && rm -rf /var/lib/apt/lists/* \
+    && npm install -g agent-browser \
+    && agent-browser install \
+    && rm -rf /tmp/* /root/.npm
+
+WORKDIR /app/autogpt_platform/backend
+
+# Copy only the .venv from builder (not the entire /app directory)
+# The .venv includes the generated Prisma client
+COPY --from=builder /app/autogpt_platform/backend/.venv ./.venv
+ENV PATH="/app/autogpt_platform/backend/.venv/bin:$PATH"
+
+# Copy dependency files + autogpt_libs (path dependency)
+COPY autogpt_platform/autogpt_libs /app/autogpt_platform/autogpt_libs
+COPY autogpt_platform/backend/poetry.lock autogpt_platform/backend/pyproject.toml ./
+
+# Copy backend code + docs (for Copilot docs search)
+COPY autogpt_platform/backend ./
+COPY docs /app/docs
+# Install the project package to create entry point scripts in .venv/bin/
+# (e.g., rest, executor, ws, db, scheduler, notification - see [tool.poetry.scripts])
+RUN POETRY_VIRTUALENVS_CREATE=true POETRY_VIRTUALENVS_IN_PROJECT=true \
+    poetry install --no-ansi --only-root
+
+ENV PORT=8000
+
+CMD ["rest"]

autogpt_platform/backend/backend/api/conftest.py

@@ -1,4 +1,9 @@
-"""Common test fixtures for server tests."""
+"""Common test fixtures for server tests.
+
+Note: Common fixtures like test_user_id, admin_user_id, target_user_id,
+setup_test_user, and setup_admin_user are defined in the parent conftest.py
+(backend/conftest.py) and are available here automatically.
+"""
 
 import pytest
 from pytest_snapshot.plugin import Snapshot
@@ -11,54 +16,6 @@ def configured_snapshot(snapshot: Snapshot) -> Snapshot:
     return snapshot
 
 
-@pytest.fixture
-def test_user_id() -> str:
-    """Test user ID fixture."""
-    return "3e53486c-cf57-477e-ba2a-cb02dc828e1a"
-
-
-@pytest.fixture
-def admin_user_id() -> str:
-    """Admin user ID fixture."""
-    return "4e53486c-cf57-477e-ba2a-cb02dc828e1b"
-
-
-@pytest.fixture
-def target_user_id() -> str:
-    """Target user ID fixture."""
-    return "5e53486c-cf57-477e-ba2a-cb02dc828e1c"
-
-
-@pytest.fixture
-async def setup_test_user(test_user_id):
-    """Create test user in database before tests."""
-    from backend.data.user import get_or_create_user
-
-    # Create the test user in the database using JWT token format
-    user_data = {
-        "sub": test_user_id,
-        "email": "test@example.com",
-        "user_metadata": {"name": "Test User"},
-    }
-    await get_or_create_user(user_data)
-    return test_user_id
-
-
-@pytest.fixture
-async def setup_admin_user(admin_user_id):
-    """Create admin user in database before tests."""
-    from backend.data.user import get_or_create_user
-
-    # Create the admin user in the database using JWT token format
-    user_data = {
-        "sub": admin_user_id,
-        "email": "test-admin@example.com",
-        "user_metadata": {"name": "Test Admin"},
-    }
-    await get_or_create_user(user_data)
-    return admin_user_id
-
-
 @pytest.fixture
 def mock_jwt_user(test_user_id):
     """Provide mock JWT payload for regular user testing."""

autogpt_platform/backend/backend/api/external/middleware.py

@@ -88,20 +88,23 @@ async def require_auth(
     )
 
 
-def require_permission(permission: APIKeyPermission):
+def require_permission(*permissions: APIKeyPermission):
     """
-    Dependency function for checking specific permissions
+    Dependency function for checking required permissions.
+    All listed permissions must be present.
     (works with API keys and OAuth tokens)
     """
 
-    async def check_permission(
+    async def check_permissions(
         auth: APIAuthorizationInfo = Security(require_auth),
     ) -> APIAuthorizationInfo:
-        if permission not in auth.scopes:
+        missing = [p for p in permissions if p not in auth.scopes]
+        if missing:
             raise HTTPException(
                 status_code=status.HTTP_403_FORBIDDEN,
-                detail=f"Missing required permission: {permission.value}",
+                detail=f"Missing required permission(s): "
+                f"{', '.join(p.value for p in missing)}",
             )
         return auth
 
-    return check_permission
+    return check_permissions

autogpt_platform/backend/backend/api/external/v1/routes.py

@@ -1,7 +1,7 @@
 import logging
 import urllib.parse
 from collections import defaultdict
-from typing import Annotated, Any, Literal, Optional, Sequence
+from typing import Annotated, Any, Optional, Sequence
 
 from fastapi import APIRouter, Body, HTTPException, Security
 from prisma.enums import AgentExecutionStatus, APIKeyPermission
@@ -9,15 +9,17 @@ from pydantic import BaseModel, Field
 from typing_extensions import TypedDict
 
 import backend.api.features.store.cache as store_cache
+import backend.api.features.store.db as store_db
 import backend.api.features.store.model as store_model
 import backend.blocks
-from backend.api.external.middleware import require_permission
+from backend.api.external.middleware import require_auth, require_permission
 from backend.data import execution as execution_db
 from backend.data import graph as graph_db
 from backend.data import user as user_db
 from backend.data.auth.base import APIAuthorizationInfo
 from backend.data.block import BlockInput, CompletedBlockOutput
 from backend.executor.utils import add_graph_execution
+from backend.integrations.webhooks.graph_lifecycle_hooks import on_graph_activate
 from backend.util.settings import Settings
 
 from .integrations import integrations_router
@@ -95,6 +97,43 @@ async def execute_graph_block(
     return output
 
 
+@v1_router.post(
+    path="/graphs",
+    tags=["graphs"],
+    status_code=201,
+    dependencies=[
+        Security(
+            require_permission(
+                APIKeyPermission.WRITE_GRAPH, APIKeyPermission.WRITE_LIBRARY
+            )
+        )
+    ],
+)
+async def create_graph(
+    graph: graph_db.Graph,
+    auth: APIAuthorizationInfo = Security(
+        require_permission(APIKeyPermission.WRITE_GRAPH, APIKeyPermission.WRITE_LIBRARY)
+    ),
+) -> graph_db.GraphModel:
+    """
+    Create a new agent graph.
+
+    The graph will be validated and assigned a new ID.
+    It is automatically added to the user's library.
+    """
+    from backend.api.features.library import db as library_db
+
+    graph_model = graph_db.make_graph_model(graph, auth.user_id)
+    graph_model.reassign_ids(user_id=auth.user_id, reassign_graph_id=True)
+    graph_model.validate_graph(for_run=False)
+
+    await graph_db.create_graph(graph_model, user_id=auth.user_id)
+    await library_db.create_library_agent(graph_model, auth.user_id)
+    activated_graph = await on_graph_activate(graph_model, user_id=auth.user_id)
+
+    return activated_graph
+
+
 @v1_router.post(
     path="/graphs/{graph_id}/execute/{graph_version}",
     tags=["graphs"],
@@ -192,13 +231,13 @@ async def get_graph_execution_results(
 @v1_router.get(
     path="/store/agents",
     tags=["store"],
-    dependencies=[Security(require_permission(APIKeyPermission.READ_STORE))],
+    dependencies=[Security(require_auth)],  # data is public; auth required as anti-DDoS
     response_model=store_model.StoreAgentsResponse,
 )
 async def get_store_agents(
     featured: bool = False,
     creator: str | None = None,
-    sorted_by: Literal["rating", "runs", "name", "updated_at"] | None = None,
+    sorted_by: store_db.StoreAgentsSortOptions | None = None,
     search_query: str | None = None,
     category: str | None = None,
     page: int = 1,
@@ -240,7 +279,7 @@ async def get_store_agents(
 @v1_router.get(
     path="/store/agents/{username}/{agent_name}",
     tags=["store"],
-    dependencies=[Security(require_permission(APIKeyPermission.READ_STORE))],
+    dependencies=[Security(require_auth)],  # data is public; auth required as anti-DDoS
     response_model=store_model.StoreAgentDetails,
 )
 async def get_store_agent(
@@ -268,13 +307,13 @@ async def get_store_agent(
 @v1_router.get(
     path="/store/creators",
     tags=["store"],
-    dependencies=[Security(require_permission(APIKeyPermission.READ_STORE))],
+    dependencies=[Security(require_auth)],  # data is public; auth required as anti-DDoS
     response_model=store_model.CreatorsResponse,
 )
 async def get_store_creators(
     featured: bool = False,
     search_query: str | None = None,
-    sorted_by: Literal["agent_rating", "agent_runs", "num_agents"] | None = None,
+    sorted_by: store_db.StoreCreatorsSortOptions | None = None,
     page: int = 1,
     page_size: int = 20,
 ) -> store_model.CreatorsResponse:
@@ -310,7 +349,7 @@ async def get_store_creators(
 @v1_router.get(
     path="/store/creators/{username}",
     tags=["store"],
-    dependencies=[Security(require_permission(APIKeyPermission.READ_STORE))],
+    dependencies=[Security(require_auth)],  # data is public; auth required as anti-DDoS
     response_model=store_model.CreatorDetails,
 )
 async def get_store_creator(

autogpt_platform/backend/backend/api/external/v1/tools.py

@@ -15,9 +15,9 @@ from prisma.enums import APIKeyPermission
 from pydantic import BaseModel, Field
 
 from backend.api.external.middleware import require_permission
-from backend.api.features.chat.model import ChatSession
-from backend.api.features.chat.tools import find_agent_tool, run_agent_tool
-from backend.api.features.chat.tools.models import ToolResponseBase
+from backend.copilot.model import ChatSession
+from backend.copilot.tools import find_agent_tool, run_agent_tool
+from backend.copilot.tools.models import ToolResponseBase
 from backend.data.auth.base import APIAuthorizationInfo
 
 logger = logging.getLogger(__name__)

autogpt_platform/backend/backend/api/features/admin/model.py

@@ -1,8 +1,17 @@
-from pydantic import BaseModel
+from __future__ import annotations
+
+from datetime import datetime
+from typing import TYPE_CHECKING, Any, Literal, Optional
+
+import prisma.enums
+from pydantic import BaseModel, EmailStr
 
 from backend.data.model import UserTransaction
 from backend.util.models import Pagination
 
+if TYPE_CHECKING:
+    from backend.data.invited_user import BulkInvitedUsersResult, InvitedUserRecord
+
 
 class UserHistoryResponse(BaseModel):
     """Response model for listings with version history"""
@@ -14,3 +23,70 @@ class UserHistoryResponse(BaseModel):
 class AddUserCreditsResponse(BaseModel):
     new_balance: int
     transaction_key: str
+
+
+class CreateInvitedUserRequest(BaseModel):
+    email: EmailStr
+    name: Optional[str] = None
+
+
+class InvitedUserResponse(BaseModel):
+    id: str
+    email: str
+    status: prisma.enums.InvitedUserStatus
+    auth_user_id: Optional[str] = None
+    name: Optional[str] = None
+    tally_understanding: Optional[dict[str, Any]] = None
+    tally_status: prisma.enums.TallyComputationStatus
+    tally_computed_at: Optional[datetime] = None
+    tally_error: Optional[str] = None
+    created_at: datetime
+    updated_at: datetime
+
+    @classmethod
+    def from_record(cls, record: InvitedUserRecord) -> InvitedUserResponse:
+        return cls.model_validate(record.model_dump())
+
+
+class InvitedUsersResponse(BaseModel):
+    invited_users: list[InvitedUserResponse]
+    pagination: Pagination
+
+
+class BulkInvitedUserRowResponse(BaseModel):
+    row_number: int
+    email: Optional[str] = None
+    name: Optional[str] = None
+    status: Literal["CREATED", "SKIPPED", "ERROR"]
+    message: str
+    invited_user: Optional[InvitedUserResponse] = None
+
+
+class BulkInvitedUsersResponse(BaseModel):
+    created_count: int
+    skipped_count: int
+    error_count: int
+    results: list[BulkInvitedUserRowResponse]
+
+    @classmethod
+    def from_result(cls, result: BulkInvitedUsersResult) -> BulkInvitedUsersResponse:
+        return cls(
+            created_count=result.created_count,
+            skipped_count=result.skipped_count,
+            error_count=result.error_count,
+            results=[
+                BulkInvitedUserRowResponse(
+                    row_number=row.row_number,
+                    email=row.email,
+                    name=row.name,
+                    status=row.status,
+                    message=row.message,
+                    invited_user=(
+                        InvitedUserResponse.from_record(row.invited_user)
+                        if row.invited_user is not None
+                        else None
+                    ),
+                )
+                for row in result.results
+            ],
+        )

autogpt_platform/backend/backend/api/features/admin/store_admin_routes.py

@@ -24,14 +24,13 @@ router = fastapi.APIRouter(
 @router.get(
     "/listings",
     summary="Get Admin Listings History",
-    response_model=store_model.StoreListingsWithVersionsResponse,
 )
 async def get_admin_listings_with_versions(
     status: typing.Optional[prisma.enums.SubmissionStatus] = None,
     search: typing.Optional[str] = None,
     page: int = 1,
     page_size: int = 20,
-):
+) -> store_model.StoreListingsWithVersionsAdminViewResponse:
     """
     Get store listings with their version history for admins.
 
@@ -45,36 +44,26 @@ async def get_admin_listings_with_versions(
         page_size: Number of items per page
 
     Returns:
-        StoreListingsWithVersionsResponse with listings and their versions
+        Paginated listings with their versions
     """
-    try:
-        listings = await store_db.get_admin_listings_with_versions(
-            status=status,
-            search_query=search,
-            page=page,
-            page_size=page_size,
-        )
-        return listings
-    except Exception as e:
-        logger.exception("Error getting admin listings with versions: %s", e)
-        return fastapi.responses.JSONResponse(
-            status_code=500,
-            content={
-                "detail": "An error occurred while retrieving listings with versions"
-            },
-        )
+    listings = await store_db.get_admin_listings_with_versions(
+        status=status,
+        search_query=search,
+        page=page,
+        page_size=page_size,
+    )
+    return listings
 
 
 @router.post(
     "/submissions/{store_listing_version_id}/review",
     summary="Review Store Submission",
-    response_model=store_model.StoreSubmission,
 )
 async def review_submission(
     store_listing_version_id: str,
     request: store_model.ReviewSubmissionRequest,
     user_id: str = fastapi.Security(autogpt_libs.auth.get_user_id),
-):
+) -> store_model.StoreSubmissionAdminView:
     """
     Review a store listing submission.
 
@@ -84,31 +73,24 @@ async def review_submission(
         user_id: Authenticated admin user performing the review
 
     Returns:
-        StoreSubmission with updated review information
+        StoreSubmissionAdminView with updated review information
     """
-    try:
-        already_approved = await store_db.check_submission_already_approved(
-            store_listing_version_id=store_listing_version_id,
-        )
-        submission = await store_db.review_store_submission(
-            store_listing_version_id=store_listing_version_id,
-            is_approved=request.is_approved,
-            external_comments=request.comments,
-            internal_comments=request.internal_comments or "",
-            reviewer_id=user_id,
-        )
+    already_approved = await store_db.check_submission_already_approved(
+        store_listing_version_id=store_listing_version_id,
+    )
+    submission = await store_db.review_store_submission(
+        store_listing_version_id=store_listing_version_id,
+        is_approved=request.is_approved,
+        external_comments=request.comments,
+        internal_comments=request.internal_comments or "",
+        reviewer_id=user_id,
+    )
 
-        state_changed = already_approved != request.is_approved
-        # Clear caches when the request is approved as it updates what is shown on the store
-        if state_changed:
-            store_cache.clear_all_caches()
-        return submission
-    except Exception as e:
-        logger.exception("Error reviewing submission: %s", e)
-        return fastapi.responses.JSONResponse(
-            status_code=500,
-            content={"detail": "An error occurred while reviewing the submission"},
-        )
+    state_changed = already_approved != request.is_approved
+    # Clear caches whenever approval state changes, since store visibility can change
+    if state_changed:
+        store_cache.clear_all_caches()
+    return submission
 
 
 @router.get(

autogpt_platform/backend/backend/api/features/admin/user_admin_routes.py

@@ -0,0 +1,137 @@
+import logging
+import math
+
+from autogpt_libs.auth import get_user_id, requires_admin_user
+from fastapi import APIRouter, File, Query, Security, UploadFile
+
+from backend.data.invited_user import (
+    bulk_create_invited_users_from_file,
+    create_invited_user,
+    list_invited_users,
+    retry_invited_user_tally,
+    revoke_invited_user,
+)
+from backend.data.tally import mask_email
+from backend.util.models import Pagination
+
+from .model import (
+    BulkInvitedUsersResponse,
+    CreateInvitedUserRequest,
+    InvitedUserResponse,
+    InvitedUsersResponse,
+)
+
+logger = logging.getLogger(__name__)
+
+
+router = APIRouter(
+    prefix="/admin",
+    tags=["users", "admin"],
+    dependencies=[Security(requires_admin_user)],
+)
+
+
+@router.get(
+    "/invited-users",
+    response_model=InvitedUsersResponse,
+    summary="List Invited Users",
+)
+async def get_invited_users(
+    admin_user_id: str = Security(get_user_id),
+    page: int = Query(1, ge=1),
+    page_size: int = Query(50, ge=1, le=200),
+) -> InvitedUsersResponse:
+    logger.info("Admin user %s requested invited users", admin_user_id)
+    invited_users, total = await list_invited_users(page=page, page_size=page_size)
+    return InvitedUsersResponse(
+        invited_users=[InvitedUserResponse.from_record(iu) for iu in invited_users],
+        pagination=Pagination(
+            total_items=total,
+            total_pages=max(1, math.ceil(total / page_size)),
+            current_page=page,
+            page_size=page_size,
+        ),
+    )
+
+
+@router.post(
+    "/invited-users",
+    response_model=InvitedUserResponse,
+    summary="Create Invited User",
+)
+async def create_invited_user_route(
+    request: CreateInvitedUserRequest,
+    admin_user_id: str = Security(get_user_id),
+) -> InvitedUserResponse:
+    logger.info(
+        "Admin user %s creating invited user for %s",
+        admin_user_id,
+        mask_email(request.email),
+    )
+    invited_user = await create_invited_user(request.email, request.name)
+    logger.info(
+        "Admin user %s created invited user %s",
+        admin_user_id,
+        invited_user.id,
+    )
+    return InvitedUserResponse.from_record(invited_user)
+
+
+@router.post(
+    "/invited-users/bulk",
+    response_model=BulkInvitedUsersResponse,
+    summary="Bulk Create Invited Users",
+    operation_id="postV2BulkCreateInvitedUsers",
+)
+async def bulk_create_invited_users_route(
+    file: UploadFile = File(...),
+    admin_user_id: str = Security(get_user_id),
+) -> BulkInvitedUsersResponse:
+    logger.info(
+        "Admin user %s bulk invited users from %s",
+        admin_user_id,
+        file.filename or "<unnamed>",
+    )
+    content = await file.read()
+    result = await bulk_create_invited_users_from_file(file.filename, content)
+    return BulkInvitedUsersResponse.from_result(result)
+
+
+@router.post(
+    "/invited-users/{invited_user_id}/revoke",
+    response_model=InvitedUserResponse,
+    summary="Revoke Invited User",
+)
+async def revoke_invited_user_route(
+    invited_user_id: str,
+    admin_user_id: str = Security(get_user_id),
+) -> InvitedUserResponse:
+    logger.info(
+        "Admin user %s revoking invited user %s", admin_user_id, invited_user_id
+    )
+    invited_user = await revoke_invited_user(invited_user_id)
+    logger.info("Admin user %s revoked invited user %s", admin_user_id, invited_user_id)
+    return InvitedUserResponse.from_record(invited_user)
+
+
+@router.post(
+    "/invited-users/{invited_user_id}/retry-tally",
+    response_model=InvitedUserResponse,
+    summary="Retry Invited User Tally",
+)
+async def retry_invited_user_tally_route(
+    invited_user_id: str,
+    admin_user_id: str = Security(get_user_id),
+) -> InvitedUserResponse:
+    logger.info(
+        "Admin user %s retrying Tally seed for invited user %s",
+        admin_user_id,
+        invited_user_id,
+    )
+    invited_user = await retry_invited_user_tally(invited_user_id)
+    logger.info(
+        "Admin user %s retried Tally seed for invited user %s",
+        admin_user_id,
+        invited_user_id,
+    )
+    return InvitedUserResponse.from_record(invited_user)

autogpt_platform/backend/backend/api/features/admin/user_admin_routes_test.py

@@ -0,0 +1,168 @@
+from datetime import datetime, timezone
+from unittest.mock import AsyncMock
+
+import fastapi
+import fastapi.testclient
+import prisma.enums
+import pytest
+import pytest_mock
+from autogpt_libs.auth.jwt_utils import get_jwt_payload
+
+from backend.data.invited_user import (
+    BulkInvitedUserRowResult,
+    BulkInvitedUsersResult,
+    InvitedUserRecord,
+)
+
+from .user_admin_routes import router as user_admin_router
+
+app = fastapi.FastAPI()
+app.include_router(user_admin_router)
+
+client = fastapi.testclient.TestClient(app)
+
+
+@pytest.fixture(autouse=True)
+def setup_app_admin_auth(mock_jwt_admin):
+    app.dependency_overrides[get_jwt_payload] = mock_jwt_admin["get_jwt_payload"]
+    yield
+    app.dependency_overrides.clear()
+
+
+def _sample_invited_user() -> InvitedUserRecord:
+    now = datetime.now(timezone.utc)
+    return InvitedUserRecord(
+        id="invite-1",
+        email="invited@example.com",
+        status=prisma.enums.InvitedUserStatus.INVITED,
+        auth_user_id=None,
+        name="Invited User",
+        tally_understanding=None,
+        tally_status=prisma.enums.TallyComputationStatus.PENDING,
+        tally_computed_at=None,
+        tally_error=None,
+        created_at=now,
+        updated_at=now,
+    )
+
+
+def _sample_bulk_invited_users_result() -> BulkInvitedUsersResult:
+    return BulkInvitedUsersResult(
+        created_count=1,
+        skipped_count=1,
+        error_count=0,
+        results=[
+            BulkInvitedUserRowResult(
+                row_number=1,
+                email="invited@example.com",
+                name=None,
+                status="CREATED",
+                message="Invite created",
+                invited_user=_sample_invited_user(),
+            ),
+            BulkInvitedUserRowResult(
+                row_number=2,
+                email="duplicate@example.com",
+                name=None,
+                status="SKIPPED",
+                message="An invited user with this email already exists",
+                invited_user=None,
+            ),
+        ],
+    )
+
+
+def test_get_invited_users(
+    mocker: pytest_mock.MockerFixture,
+) -> None:
+    mocker.patch(
+        "backend.api.features.admin.user_admin_routes.list_invited_users",
+        AsyncMock(return_value=([_sample_invited_user()], 1)),
+    )
+
+    response = client.get("/admin/invited-users")
+
+    assert response.status_code == 200
+    data = response.json()
+    assert len(data["invited_users"]) == 1
+    assert data["invited_users"][0]["email"] == "invited@example.com"
+    assert data["invited_users"][0]["status"] == "INVITED"
+    assert data["pagination"]["total_items"] == 1
+    assert data["pagination"]["current_page"] == 1
+    assert data["pagination"]["page_size"] == 50
+
+
+def test_create_invited_user(
+    mocker: pytest_mock.MockerFixture,
+) -> None:
+    mocker.patch(
+        "backend.api.features.admin.user_admin_routes.create_invited_user",
+        AsyncMock(return_value=_sample_invited_user()),
+    )
+
+    response = client.post(
+        "/admin/invited-users",
+        json={"email": "invited@example.com", "name": "Invited User"},
+    )
+
+    assert response.status_code == 200
+    data = response.json()
+    assert data["email"] == "invited@example.com"
+    assert data["name"] == "Invited User"
+
+
+def test_bulk_create_invited_users(
+    mocker: pytest_mock.MockerFixture,
+) -> None:
+    mocker.patch(
+        "backend.api.features.admin.user_admin_routes.bulk_create_invited_users_from_file",
+        AsyncMock(return_value=_sample_bulk_invited_users_result()),
+    )
+
+    response = client.post(
+        "/admin/invited-users/bulk",
+        files={
+            "file": ("invites.txt", b"invited@example.com\nduplicate@example.com\n")
+        },
+    )
+
+    assert response.status_code == 200
+    data = response.json()
+    assert data["created_count"] == 1
+    assert data["skipped_count"] == 1
+    assert data["results"][0]["status"] == "CREATED"
+    assert data["results"][1]["status"] == "SKIPPED"
+
+
+def test_revoke_invited_user(
+    mocker: pytest_mock.MockerFixture,
+) -> None:
+    revoked = _sample_invited_user().model_copy(
+        update={"status": prisma.enums.InvitedUserStatus.REVOKED}
+    )
+    mocker.patch(
+        "backend.api.features.admin.user_admin_routes.revoke_invited_user",
+        AsyncMock(return_value=revoked),
+    )
+
+    response = client.post("/admin/invited-users/invite-1/revoke")
+
+    assert response.status_code == 200
+    assert response.json()["status"] == "REVOKED"
+
+
+def test_retry_invited_user_tally(
+    mocker: pytest_mock.MockerFixture,
+) -> None:
+    retried = _sample_invited_user().model_copy(
+        update={"tally_status": prisma.enums.TallyComputationStatus.RUNNING}
+    )
+    mocker.patch(
+        "backend.api.features.admin.user_admin_routes.retry_invited_user_tally",
+        AsyncMock(return_value=retried),
+    )
+
+    response = client.post("/admin/invited-users/invite-1/retry-tally")
+
+    assert response.status_code == 200
+    assert response.json()["tally_status"] == "RUNNING"

autogpt_platform/backend/backend/api/features/builder/db.py

@@ -1,15 +1,17 @@
 import logging
 from dataclasses import dataclass
-from datetime import datetime, timedelta, timezone
 from difflib import SequenceMatcher
-from typing import Sequence
+from typing import Any, Sequence, get_args, get_origin
 
 import prisma
+from prisma.enums import ContentType
+from prisma.models import mv_suggested_blocks
 
 import backend.api.features.library.db as library_db
 import backend.api.features.library.model as library_model
 import backend.api.features.store.db as store_db
 import backend.api.features.store.model as store_model
+from backend.api.features.store.hybrid_search import unified_hybrid_search
 from backend.blocks import load_all_blocks
 from backend.blocks._base import (
     AnyBlockSchema,
@@ -19,7 +21,6 @@ from backend.blocks._base import (
     BlockType,
 )
 from backend.blocks.llm import LlmModel
-from backend.data.db import query_raw_with_schema
 from backend.integrations.providers import ProviderName
 from backend.util.cache import cached
 from backend.util.models import Pagination
@@ -42,6 +43,16 @@ MAX_LIBRARY_AGENT_RESULTS = 100
 MAX_MARKETPLACE_AGENT_RESULTS = 100
 MIN_SCORE_FOR_FILTERED_RESULTS = 10.0
 
+# Boost blocks over marketplace agents in search results
+BLOCK_SCORE_BOOST = 50.0
+
+# Block IDs to exclude from search results
+EXCLUDED_BLOCK_IDS = frozenset(
+    {
+        "e189baac-8c20-45a1-94a7-55177ea42565",  # AgentExecutorBlock
+    }
+)
+
 SearchResultItem = BlockInfo | library_model.LibraryAgent | store_model.StoreAgent
 
 
@@ -64,8 +75,8 @@ def get_block_categories(category_blocks: int = 3) -> list[BlockCategoryResponse
 
     for block_type in load_all_blocks().values():
         block: AnyBlockSchema = block_type()
-        # Skip disabled blocks
-        if block.disabled:
+        # Skip disabled and excluded blocks
+        if block.disabled or block.id in EXCLUDED_BLOCK_IDS:
             continue
         # Skip blocks that don't have categories (all should have at least one)
         if not block.categories:
@@ -116,6 +127,9 @@ def get_blocks(
         # Skip disabled blocks
         if block.disabled:
             continue
+        # Skip excluded blocks
+        if block.id in EXCLUDED_BLOCK_IDS:
+            continue
         # Skip blocks that don't match the category
         if category and category not in {c.name.lower() for c in block.categories}:
             continue
@@ -255,14 +269,25 @@ async def _build_cached_search_results(
         "my_agents": 0,
     }
 
-    block_results, block_total, integration_total = _collect_block_results(
-        normalized_query=normalized_query,
-        include_blocks=include_blocks,
-        include_integrations=include_integrations,
-    )
-    scored_items.extend(block_results)
-    total_items["blocks"] = block_total
-    total_items["integrations"] = integration_total
+    # Use hybrid search when query is present, otherwise list all blocks
+    if (include_blocks or include_integrations) and normalized_query:
+        block_results, block_total, integration_total = await _hybrid_search_blocks(
+            query=search_query,
+            include_blocks=include_blocks,
+            include_integrations=include_integrations,
+        )
+        scored_items.extend(block_results)
+        total_items["blocks"] = block_total
+        total_items["integrations"] = integration_total
+    elif include_blocks or include_integrations:
+        # No query - list all blocks using in-memory approach
+        block_results, block_total, integration_total = _collect_block_results(
+            include_blocks=include_blocks,
+            include_integrations=include_integrations,
+        )
+        scored_items.extend(block_results)
+        total_items["blocks"] = block_total
+        total_items["integrations"] = integration_total
 
     if include_library_agents:
         library_response = await library_db.list_library_agents(
@@ -307,10 +332,14 @@ async def _build_cached_search_results(
 
 def _collect_block_results(
     *,
-    normalized_query: str,
     include_blocks: bool,
     include_integrations: bool,
 ) -> tuple[list[_ScoredItem], int, int]:
+    """
+    Collect all blocks for listing (no search query).
+
+    All blocks get BLOCK_SCORE_BOOST to prioritize them over marketplace agents.
+    """
     results: list[_ScoredItem] = []
     block_count = 0
     integration_count = 0
@@ -323,6 +352,10 @@ def _collect_block_results(
         if block.disabled:
             continue
 
+        # Skip excluded blocks
+        if block.id in EXCLUDED_BLOCK_IDS:
+            continue
+
         block_info = block.get_info()
         credentials = list(block.input_schema.get_credentials_fields().values())
         is_integration = len(credentials) > 0
@@ -332,10 +365,6 @@ def _collect_block_results(
         if not is_integration and not include_blocks:
             continue
 
-        score = _score_block(block, block_info, normalized_query)
-        if not _should_include_item(score, normalized_query):
-            continue
-
         filter_type: FilterType = "integrations" if is_integration else "blocks"
         if is_integration:
             integration_count += 1
@@ -346,8 +375,122 @@ def _collect_block_results(
             _ScoredItem(
                 item=block_info,
                 filter_type=filter_type,
-                score=score,
-                sort_key=_get_item_name(block_info),
+                score=BLOCK_SCORE_BOOST,
+                sort_key=block_info.name.lower(),
+            )
+        )
+
+    return results, block_count, integration_count
+
+
+async def _hybrid_search_blocks(
+    *,
+    query: str,
+    include_blocks: bool,
+    include_integrations: bool,
+) -> tuple[list[_ScoredItem], int, int]:
+    """
+    Search blocks using hybrid search with builder-specific filtering.
+
+    Uses unified_hybrid_search for semantic + lexical search, then applies
+    post-filtering for block/integration types and scoring adjustments.
+
+    Scoring:
+        - Base: hybrid relevance score (0-1) scaled to 0-100, plus BLOCK_SCORE_BOOST
+          to prioritize blocks over marketplace agents in combined results
+        - +30 for exact name match, +15 for prefix name match
+        - +20 if the block has an LlmModel field and the query matches an LLM model name
+
+    Args:
+        query: The search query string
+        include_blocks: Whether to include regular blocks
+        include_integrations: Whether to include integration blocks
+
+    Returns:
+        Tuple of (scored_items, block_count, integration_count)
+    """
+    results: list[_ScoredItem] = []
+    block_count = 0
+    integration_count = 0
+
+    if not include_blocks and not include_integrations:
+        return results, block_count, integration_count
+
+    normalized_query = query.strip().lower()
+
+    # Fetch more results to account for post-filtering
+    search_results, _ = await unified_hybrid_search(
+        query=query,
+        content_types=[ContentType.BLOCK],
+        page=1,
+        page_size=150,
+        min_score=0.10,
+    )
+
+    # Load all blocks for getting BlockInfo
+    all_blocks = load_all_blocks()
+
+    for result in search_results:
+        block_id = result["content_id"]
+
+        # Skip excluded blocks
+        if block_id in EXCLUDED_BLOCK_IDS:
+            continue
+
+        metadata = result.get("metadata", {})
+        hybrid_score = result.get("relevance", 0.0)
+
+        # Get the actual block class
+        if block_id not in all_blocks:
+            continue
+
+        block_cls = all_blocks[block_id]
+        block: AnyBlockSchema = block_cls()
+
+        if block.disabled:
+            continue
+
+        # Check block/integration filter using metadata
+        is_integration = metadata.get("is_integration", False)
+
+        if is_integration and not include_integrations:
+            continue
+        if not is_integration and not include_blocks:
+            continue
+
+        # Get block info
+        block_info = block.get_info()
+
+        # Calculate final score: scale hybrid score and add builder-specific bonuses
+        # Hybrid scores are 0-1, builder scores were 0-200+
+        # Add BLOCK_SCORE_BOOST to prioritize blocks over marketplace agents
+        final_score = hybrid_score * 100 + BLOCK_SCORE_BOOST
+
+        # Add LLM model match bonus
+        has_llm_field = metadata.get("has_llm_model_field", False)
+        if has_llm_field and _matches_llm_model(block.input_schema, normalized_query):
+            final_score += 20
+
+        # Add exact/prefix match bonus for deterministic tie-breaking
+        name = block_info.name.lower()
+        if name == normalized_query:
+            final_score += 30
+        elif name.startswith(normalized_query):
+            final_score += 15
+
+        # Track counts
+        filter_type: FilterType = "integrations" if is_integration else "blocks"
+        if is_integration:
+            integration_count += 1
+        else:
+            block_count += 1
+
+        results.append(
+            _ScoredItem(
+                item=block_info,
+                filter_type=filter_type,
+                score=final_score,
+                sort_key=name,
             )
         )
 
@@ -472,6 +615,8 @@ async def _get_static_counts():
         block: AnyBlockSchema = block_type()
         if block.disabled:
             continue
+        if block.id in EXCLUDED_BLOCK_IDS:
+            continue
 
         all_blocks += 1
 
@@ -498,47 +643,25 @@ async def _get_static_counts():
     }
 
 
+def _contains_type(annotation: Any, target: type) -> bool:
+    """Check if an annotation is or contains the target type (handles Optional/Union/Annotated)."""
+    if annotation is target:
+        return True
+    origin = get_origin(annotation)
+    if origin is None:
+        return False
+    return any(_contains_type(arg, target) for arg in get_args(annotation))
+
+
 def _matches_llm_model(schema_cls: type[BlockSchema], query: str) -> bool:
     for field in schema_cls.model_fields.values():
-        if field.annotation == LlmModel:
+        if _contains_type(field.annotation, LlmModel):
             # Check if query matches any value in llm_models
             if any(query in name for name in llm_models):
                 return True
     return False
 
 
-def _score_block(
-    block: AnyBlockSchema,
-    block_info: BlockInfo,
-    normalized_query: str,
-) -> float:
-    if not normalized_query:
-        return 0.0
-
-    name = block_info.name.lower()
-    description = block_info.description.lower()
-    score = _score_primary_fields(name, description, normalized_query)
-
-    category_text = " ".join(
-        category.get("category", "").lower() for category in block_info.categories
-    )
-    score += _score_additional_field(category_text, normalized_query, 12, 6)
-
-    credentials_info = block.input_schema.get_credentials_fields_info().values()
-    provider_names = [
-        provider.value.lower()
-        for info in credentials_info
-        for provider in info.provider
-    ]
-    provider_text = " ".join(provider_names)
-    score += _score_additional_field(provider_text, normalized_query, 15, 6)
-
-    if _matches_llm_model(block.input_schema, normalized_query):
-        score += 20
-
-    return score
-
-
 def _score_library_agent(
     agent: library_model.LibraryAgent,
     normalized_query: str,
@@ -645,31 +768,20 @@ def _get_all_providers() -> dict[ProviderName, Provider]:
     return providers
 
 
-@cached(ttl_seconds=3600)
+@cached(ttl_seconds=3600, shared_cache=True)
 async def get_suggested_blocks(count: int = 5) -> list[BlockInfo]:
-    suggested_blocks = []
-    # Sum the number of executions for each block type
-    # Prisma cannot group by nested relations, so we do a raw query
-    # Calculate the cutoff timestamp
-    timestamp_threshold = datetime.now(timezone.utc) - timedelta(days=30)
+    """Return the most-executed blocks from the last 14 days.
 
-    results = await query_raw_with_schema(
-        """
-        SELECT
-            agent_node."agentBlockId" AS block_id,
-            COUNT(execution.id) AS execution_count
-        FROM {schema_prefix}"AgentNodeExecution" execution
-        JOIN {schema_prefix}"AgentNode" agent_node ON execution."agentNodeId" = agent_node.id
-        WHERE execution."endedTime" >= $1::timestamp
-        GROUP BY agent_node."agentBlockId"
-        ORDER BY execution_count DESC;
-        """,
-        timestamp_threshold,
-    )
+    Queries the mv_suggested_blocks materialized view (refreshed hourly via pg_cron)
+    and returns the top `count` blocks sorted by execution count, excluding
+    Input/Output/Agent block types and blocks in EXCLUDED_BLOCK_IDS.
+    """
+    results = await mv_suggested_blocks.prisma().find_many()
 
     # Get the top blocks based on execution count
-    # But ignore Input and Output blocks
+    # But ignore Input, Output, Agent, and excluded blocks
     blocks: list[tuple[BlockInfo, int]] = []
+    execution_counts = {row.block_id: row.execution_count for row in results}
 
     for block_type in load_all_blocks().values():
         block: AnyBlockSchema = block_type()
@@ -679,11 +791,9 @@ async def get_suggested_blocks(count: int = 5) -> list[BlockInfo]:
             BlockType.AGENT,
         ):
             continue
-        # Find the execution count for this block
-        execution_count = next(
-            (row["execution_count"] for row in results if row["block_id"] == block.id),
-            0,
-        )
+        if block.id in EXCLUDED_BLOCK_IDS:
+            continue
+        execution_count = execution_counts.get(block.id, 0)
         blocks.append((block.get_info(), execution_count))
     # Sort blocks by execution count
     blocks.sort(key=lambda x: x[1], reverse=True)

autogpt_platform/backend/backend/api/features/builder/model.py

@@ -27,7 +27,6 @@ class SearchEntry(BaseModel):
 
 # Suggestions
 class SuggestionsResponse(BaseModel):
-    otto_suggestions: list[str]
     recent_searches: list[SearchEntry]
     providers: list[ProviderName]
     top_blocks: list[BlockInfo]

autogpt_platform/backend/backend/api/features/builder/routes.py

@@ -1,5 +1,5 @@
 import logging
-from typing import Annotated, Sequence
+from typing import Annotated, Sequence, cast, get_args
 
 import fastapi
 from autogpt_libs.auth.dependencies import get_user_id, requires_user
@@ -10,6 +10,8 @@ from backend.util.models import Pagination
 from . import db as builder_db
 from . import model as builder_model
 
+VALID_FILTER_VALUES = get_args(builder_model.FilterType)
+
 logger = logging.getLogger(__name__)
 
 router = fastapi.APIRouter(
@@ -49,11 +51,6 @@ async def get_suggestions(
     Get all suggestions for the Blocks Menu.
     """
     return builder_model.SuggestionsResponse(
-        otto_suggestions=[
-            "What blocks do I need to get started?",
-            "Help me create a list",
-            "Help me feed my data to Google Maps",
-        ],
         recent_searches=await builder_db.get_recent_searches(user_id),
         providers=[
             ProviderName.TWITTER,
@@ -151,7 +148,7 @@ async def get_providers(
 async def search(
     user_id: Annotated[str, fastapi.Security(get_user_id)],
     search_query: Annotated[str | None, fastapi.Query()] = None,
-    filter: Annotated[list[builder_model.FilterType] | None, fastapi.Query()] = None,
+    filter: Annotated[str | None, fastapi.Query()] = None,
     search_id: Annotated[str | None, fastapi.Query()] = None,
     by_creator: Annotated[list[str] | None, fastapi.Query()] = None,
     page: Annotated[int, fastapi.Query()] = 1,
@@ -160,9 +157,20 @@ async def search(
     """
     Search for blocks (including integrations), marketplace agents, and user library agents.
     """
-    # If no filters are provided, then we will return all types
-    if not filter:
-        filter = [
+    # Parse and validate filter parameter
+    filters: list[builder_model.FilterType]
+    if filter:
+        filter_values = [f.strip() for f in filter.split(",")]
+        invalid_filters = [f for f in filter_values if f not in VALID_FILTER_VALUES]
+        if invalid_filters:
+            raise fastapi.HTTPException(
+                status_code=400,
+                detail=f"Invalid filter value(s): {', '.join(invalid_filters)}. "
+                f"Valid values are: {', '.join(VALID_FILTER_VALUES)}",
+            )
+        filters = cast(list[builder_model.FilterType], filter_values)
+    else:
+        filters = [
             "blocks",
             "integrations",
             "marketplace_agents",
@@ -174,7 +182,7 @@ async def search(
     cached_results = await builder_db.get_sorted_search_results(
         user_id=user_id,
         search_query=search_query,
-        filters=filter,
+        filters=filters,
         by_creator=by_creator,
     )
 
@@ -196,7 +204,7 @@ async def search(
         user_id,
         builder_model.SearchEntry(
             search_query=search_query,
-            filter=filter,
+            filter=filters,
             by_creator=by_creator,
             search_id=search_id,
         ),

autogpt_platform/backend/backend/api/features/chat/completion_consumer.py

@@ -1,368 +0,0 @@
-"""Redis Streams consumer for operation completion messages.
-
-This module provides a consumer (ChatCompletionConsumer) that listens for
-completion notifications (OperationCompleteMessage) from external services
-(like Agent Generator) and triggers the appropriate stream registry and
-chat service updates via process_operation_success/process_operation_failure.
-
-Why Redis Streams instead of RabbitMQ?
---------------------------------------
-While the project typically uses RabbitMQ for async task queues (e.g., execution
-queue), Redis Streams was chosen for chat completion notifications because:
-
-1. **Unified Infrastructure**: The SSE reconnection feature already uses Redis
-   Streams (via stream_registry) for message persistence and replay. Using Redis
-   Streams for completion notifications keeps all chat streaming infrastructure
-   in one system, simplifying operations and reducing cross-system coordination.
-
-2. **Message Replay**: Redis Streams support XREAD with arbitrary message IDs,
-   allowing consumers to replay missed messages after reconnection. This aligns
-   with the SSE reconnection pattern where clients can resume from last_message_id.
-
-3. **Consumer Groups with XAUTOCLAIM**: Redis consumer groups provide automatic
-   load balancing across pods with explicit message claiming (XAUTOCLAIM) for
-   recovering from dead consumers - ideal for the completion callback pattern.
-
-4. **Lower Latency**: For real-time SSE updates, Redis (already in-memory for
-   stream_registry) provides lower latency than an additional RabbitMQ hop.
-
-5. **Atomicity with Task State**: Completion processing often needs to update
-   task metadata stored in Redis. Keeping both in Redis enables simpler
-   transactional semantics without distributed coordination.
-
-The consumer uses Redis Streams with consumer groups for reliable message
-processing across multiple platform pods, with XAUTOCLAIM for reclaiming
-stale pending messages from dead consumers.
-"""
-
-import asyncio
-import logging
-import os
-import uuid
-from typing import Any
-
-import orjson
-from prisma import Prisma
-from pydantic import BaseModel
-from redis.exceptions import ResponseError
-
-from backend.data.redis_client import get_redis_async
-
-from . import stream_registry
-from .completion_handler import process_operation_failure, process_operation_success
-from .config import ChatConfig
-
-logger = logging.getLogger(__name__)
-config = ChatConfig()
-
-
-class OperationCompleteMessage(BaseModel):
-    """Message format for operation completion notifications."""
-
-    operation_id: str
-    task_id: str
-    success: bool
-    result: dict | str | None = None
-    error: str | None = None
-
-
-class ChatCompletionConsumer:
-    """Consumer for chat operation completion messages from Redis Streams.
-
-    This consumer initializes its own Prisma client in start() to ensure
-    database operations work correctly within this async context.
-
-    Uses Redis consumer groups to allow multiple platform pods to consume
-    messages reliably with automatic redelivery on failure.
-    """
-
-    def __init__(self):
-        self._consumer_task: asyncio.Task | None = None
-        self._running = False
-        self._prisma: Prisma | None = None
-        self._consumer_name = f"consumer-{uuid.uuid4().hex[:8]}"
-
-    async def start(self) -> None:
-        """Start the completion consumer."""
-        if self._running:
-            logger.warning("Completion consumer already running")
-            return
-
-        # Create consumer group if it doesn't exist
-        try:
-            redis = await get_redis_async()
-            await redis.xgroup_create(
-                config.stream_completion_name,
-                config.stream_consumer_group,
-                id="0",
-                mkstream=True,
-            )
-            logger.info(
-                f"Created consumer group '{config.stream_consumer_group}' "
-                f"on stream '{config.stream_completion_name}'"
-            )
-        except ResponseError as e:
-            if "BUSYGROUP" in str(e):
-                logger.debug(
-                    f"Consumer group '{config.stream_consumer_group}' already exists"
-                )
-            else:
-                raise
-
-        self._running = True
-        self._consumer_task = asyncio.create_task(self._consume_messages())
-        logger.info(
-            f"Chat completion consumer started (consumer: {self._consumer_name})"
-        )
-
-    async def _ensure_prisma(self) -> Prisma:
-        """Lazily initialize Prisma client on first use."""
-        if self._prisma is None:
-            database_url = os.getenv("DATABASE_URL", "postgresql://localhost:5432")
-            self._prisma = Prisma(datasource={"url": database_url})
-            await self._prisma.connect()
-            logger.info("[COMPLETION] Consumer Prisma client connected (lazy init)")
-        return self._prisma
-
-    async def stop(self) -> None:
-        """Stop the completion consumer."""
-        self._running = False
-
-        if self._consumer_task:
-            self._consumer_task.cancel()
-            try:
-                await self._consumer_task
-            except asyncio.CancelledError:
-                pass
-            self._consumer_task = None
-
-        if self._prisma:
-            await self._prisma.disconnect()
-            self._prisma = None
-            logger.info("[COMPLETION] Consumer Prisma client disconnected")
-
-        logger.info("Chat completion consumer stopped")
-
-    async def _consume_messages(self) -> None:
-        """Main message consumption loop with retry logic."""
-        max_retries = 10
-        retry_delay = 5  # seconds
-        retry_count = 0
-        block_timeout = 5000  # milliseconds
-
-        while self._running and retry_count < max_retries:
-            try:
-                redis = await get_redis_async()
-
-                # Reset retry count on successful connection
-                retry_count = 0
-
-                while self._running:
-                    # First, claim any stale pending messages from dead consumers
-                    # Redis does NOT auto-redeliver pending messages; we must explicitly
-                    # claim them using XAUTOCLAIM
-                    try:
-                        claimed_result = await redis.xautoclaim(
-                            name=config.stream_completion_name,
-                            groupname=config.stream_consumer_group,
-                            consumername=self._consumer_name,
-                            min_idle_time=config.stream_claim_min_idle_ms,
-                            start_id="0-0",
-                            count=10,
-                        )
-                        # xautoclaim returns: (next_start_id, [(id, data), ...], [deleted_ids])
-                        if claimed_result and len(claimed_result) >= 2:
-                            claimed_entries = claimed_result[1]
-                            if claimed_entries:
-                                logger.info(
-                                    f"Claimed {len(claimed_entries)} stale pending messages"
-                                )
-                                for entry_id, data in claimed_entries:
-                                    if not self._running:
-                                        return
-                                    await self._process_entry(redis, entry_id, data)
-                    except Exception as e:
-                        logger.warning(f"XAUTOCLAIM failed (non-fatal): {e}")
-
-                    # Read new messages from the stream
-                    messages = await redis.xreadgroup(
-                        groupname=config.stream_consumer_group,
-                        consumername=self._consumer_name,
-                        streams={config.stream_completion_name: ">"},
-                        block=block_timeout,
-                        count=10,
-                    )
-
-                    if not messages:
-                        continue
-
-                    for stream_name, entries in messages:
-                        for entry_id, data in entries:
-                            if not self._running:
-                                return
-                            await self._process_entry(redis, entry_id, data)
-
-            except asyncio.CancelledError:
-                logger.info("Consumer cancelled")
-                return
-            except Exception as e:
-                retry_count += 1
-                logger.error(
-                    f"Consumer error (retry {retry_count}/{max_retries}): {e}",
-                    exc_info=True,
-                )
-                if self._running and retry_count < max_retries:
-                    await asyncio.sleep(retry_delay)
-                else:
-                    logger.error("Max retries reached, stopping consumer")
-                    return
-
-    async def _process_entry(
-        self, redis: Any, entry_id: str, data: dict[str, Any]
-    ) -> None:
-        """Process a single stream entry and acknowledge it on success.
-
-        Args:
-            redis: Redis client connection
-            entry_id: The stream entry ID
-            data: The entry data dict
-        """
-        try:
-            # Handle the message
-            message_data = data.get("data")
-            if message_data:
-                await self._handle_message(
-                    message_data.encode()
-                    if isinstance(message_data, str)
-                    else message_data
-                )
-
-            # Acknowledge the message after successful processing
-            await redis.xack(
-                config.stream_completion_name,
-                config.stream_consumer_group,
-                entry_id,
-            )
-        except Exception as e:
-            logger.error(
-                f"Error processing completion message {entry_id}: {e}",
-                exc_info=True,
-            )
-            # Message remains in pending state and will be claimed by
-            # XAUTOCLAIM after min_idle_time expires
-
-    async def _handle_message(self, body: bytes) -> None:
-        """Handle a completion message using our own Prisma client."""
-        try:
-            data = orjson.loads(body)
-            message = OperationCompleteMessage(**data)
-        except Exception as e:
-            logger.error(f"Failed to parse completion message: {e}")
-            return
-
-        logger.info(
-            f"[COMPLETION] Received completion for operation {message.operation_id} "
-            f"(task_id={message.task_id}, success={message.success})"
-        )
-
-        # Find task in registry
-        task = await stream_registry.find_task_by_operation_id(message.operation_id)
-        if task is None:
-            task = await stream_registry.get_task(message.task_id)
-
-        if task is None:
-            logger.warning(
-                f"[COMPLETION] Task not found for operation {message.operation_id} "
-                f"(task_id={message.task_id})"
-            )
-            return
-
-        logger.info(
-            f"[COMPLETION] Found task: task_id={task.task_id}, "
-            f"session_id={task.session_id}, tool_call_id={task.tool_call_id}"
-        )
-
-        # Guard against empty task fields
-        if not task.task_id or not task.session_id or not task.tool_call_id:
-            logger.error(
-                f"[COMPLETION] Task has empty critical fields! "
-                f"task_id={task.task_id!r}, session_id={task.session_id!r}, "
-                f"tool_call_id={task.tool_call_id!r}"
-            )
-            return
-
-        if message.success:
-            await self._handle_success(task, message)
-        else:
-            await self._handle_failure(task, message)
-
-    async def _handle_success(
-        self,
-        task: stream_registry.ActiveTask,
-        message: OperationCompleteMessage,
-    ) -> None:
-        """Handle successful operation completion."""
-        prisma = await self._ensure_prisma()
-        await process_operation_success(task, message.result, prisma)
-
-    async def _handle_failure(
-        self,
-        task: stream_registry.ActiveTask,
-        message: OperationCompleteMessage,
-    ) -> None:
-        """Handle failed operation completion."""
-        prisma = await self._ensure_prisma()
-        await process_operation_failure(task, message.error, prisma)
-
-
-# Module-level consumer instance
-_consumer: ChatCompletionConsumer | None = None
-
-
-async def start_completion_consumer() -> None:
-    """Start the global completion consumer."""
-    global _consumer
-    if _consumer is None:
-        _consumer = ChatCompletionConsumer()
-    await _consumer.start()
-
-
-async def stop_completion_consumer() -> None:
-    """Stop the global completion consumer."""
-    global _consumer
-    if _consumer:
-        await _consumer.stop()
-        _consumer = None
-
-
-async def publish_operation_complete(
-    operation_id: str,
-    task_id: str,
-    success: bool,
-    result: dict | str | None = None,
-    error: str | None = None,
-) -> None:
-    """Publish an operation completion message to Redis Streams.
-
-    Args:
-        operation_id: The operation ID that completed.
-        task_id: The task ID associated with the operation.
-        success: Whether the operation succeeded.
-        result: The result data (for success).
-        error: The error message (for failure).
-    """
-    message = OperationCompleteMessage(
-        operation_id=operation_id,
-        task_id=task_id,
-        success=success,
-        result=result,
-        error=error,
-    )
-
-    redis = await get_redis_async()
-    await redis.xadd(
-        config.stream_completion_name,
-        {"data": message.model_dump_json()},
-        maxlen=config.stream_max_length,
-    )
-    logger.info(f"Published completion for operation {operation_id}")

autogpt_platform/backend/backend/api/features/chat/completion_handler.py

@@ -1,344 +0,0 @@
-"""Shared completion handling for operation success and failure.
-
-This module provides common logic for handling operation completion from both:
-- The Redis Streams consumer (completion_consumer.py)
-- The HTTP webhook endpoint (routes.py)
-"""
-
-import logging
-from typing import Any
-
-import orjson
-from prisma import Prisma
-
-from . import service as chat_service
-from . import stream_registry
-from .response_model import StreamError, StreamToolOutputAvailable
-from .tools.models import ErrorResponse
-
-logger = logging.getLogger(__name__)
-
-# Tools that produce agent_json that needs to be saved to library
-AGENT_GENERATION_TOOLS = {"create_agent", "edit_agent"}
-
-# Keys that should be stripped from agent_json when returning in error responses
-SENSITIVE_KEYS = frozenset(
-    {
-        "api_key",
-        "apikey",
-        "api_secret",
-        "password",
-        "secret",
-        "credentials",
-        "credential",
-        "token",
-        "access_token",
-        "refresh_token",
-        "private_key",
-        "privatekey",
-        "auth",
-        "authorization",
-    }
-)
-
-
-def _sanitize_agent_json(obj: Any) -> Any:
-    """Recursively sanitize agent_json by removing sensitive keys.
-
-    Args:
-        obj: The object to sanitize (dict, list, or primitive)
-
-    Returns:
-        Sanitized copy with sensitive keys removed/redacted
-    """
-    if isinstance(obj, dict):
-        return {
-            k: "[REDACTED]" if k.lower() in SENSITIVE_KEYS else _sanitize_agent_json(v)
-            for k, v in obj.items()
-        }
-    elif isinstance(obj, list):
-        return [_sanitize_agent_json(item) for item in obj]
-    else:
-        return obj
-
-
-class ToolMessageUpdateError(Exception):
-    """Raised when updating a tool message in the database fails."""
-
-    pass
-
-
-async def _update_tool_message(
-    session_id: str,
-    tool_call_id: str,
-    content: str,
-    prisma_client: Prisma | None,
-) -> None:
-    """Update tool message in database.
-
-    Args:
-        session_id: The session ID
-        tool_call_id: The tool call ID to update
-        content: The new content for the message
-        prisma_client: Optional Prisma client. If None, uses chat_service.
-
-    Raises:
-        ToolMessageUpdateError: If the database update fails. The caller should
-            handle this to avoid marking the task as completed with inconsistent state.
-    """
-    try:
-        if prisma_client:
-            # Use provided Prisma client (for consumer with its own connection)
-            updated_count = await prisma_client.chatmessage.update_many(
-                where={
-                    "sessionId": session_id,
-                    "toolCallId": tool_call_id,
-                },
-                data={"content": content},
-            )
-            # Check if any rows were updated - 0 means message not found
-            if updated_count == 0:
-                raise ToolMessageUpdateError(
-                    f"No message found with tool_call_id={tool_call_id} in session {session_id}"
-                )
-        else:
-            # Use service function (for webhook endpoint)
-            await chat_service._update_pending_operation(
-                session_id=session_id,
-                tool_call_id=tool_call_id,
-                result=content,
-            )
-    except ToolMessageUpdateError:
-        raise
-    except Exception as e:
-        logger.error(f"[COMPLETION] Failed to update tool message: {e}", exc_info=True)
-        raise ToolMessageUpdateError(
-            f"Failed to update tool message for tool_call_id={tool_call_id}: {e}"
-        ) from e
-
-
-def serialize_result(result: dict | list | str | int | float | bool | None) -> str:
-    """Serialize result to JSON string with sensible defaults.
-
-    Args:
-        result: The result to serialize. Can be a dict, list, string,
-            number, boolean, or None.
-
-    Returns:
-        JSON string representation of the result. Returns '{"status": "completed"}'
-        only when result is explicitly None.
-    """
-    if isinstance(result, str):
-        return result
-    if result is None:
-        return '{"status": "completed"}'
-    return orjson.dumps(result).decode("utf-8")
-
-
-async def _save_agent_from_result(
-    result: dict[str, Any],
-    user_id: str | None,
-    tool_name: str,
-) -> dict[str, Any]:
-    """Save agent to library if result contains agent_json.
-
-    Args:
-        result: The result dict that may contain agent_json
-        user_id: The user ID to save the agent for
-        tool_name: The tool name (create_agent or edit_agent)
-
-    Returns:
-        Updated result dict with saved agent details, or original result if no agent_json
-    """
-    if not user_id:
-        logger.warning("[COMPLETION] Cannot save agent: no user_id in task")
-        return result
-
-    agent_json = result.get("agent_json")
-    if not agent_json:
-        logger.warning(
-            f"[COMPLETION] {tool_name} completed but no agent_json in result"
-        )
-        return result
-
-    try:
-        from .tools.agent_generator import save_agent_to_library
-
-        is_update = tool_name == "edit_agent"
-        created_graph, library_agent = await save_agent_to_library(
-            agent_json, user_id, is_update=is_update
-        )
-
-        logger.info(
-            f"[COMPLETION] Saved agent '{created_graph.name}' to library "
-            f"(graph_id={created_graph.id}, library_agent_id={library_agent.id})"
-        )
-
-        # Return a response similar to AgentSavedResponse
-        return {
-            "type": "agent_saved",
-            "message": f"Agent '{created_graph.name}' has been saved to your library!",
-            "agent_id": created_graph.id,
-            "agent_name": created_graph.name,
-            "library_agent_id": library_agent.id,
-            "library_agent_link": f"/library/agents/{library_agent.id}",
-            "agent_page_link": f"/build?flowID={created_graph.id}",
-        }
-    except Exception as e:
-        logger.error(
-            f"[COMPLETION] Failed to save agent to library: {e}",
-            exc_info=True,
-        )
-        # Return error but don't fail the whole operation
-        # Sanitize agent_json to remove sensitive keys before returning
-        return {
-            "type": "error",
-            "message": f"Agent was generated but failed to save: {str(e)}",
-            "error": str(e),
-            "agent_json": _sanitize_agent_json(agent_json),
-        }
-
-
-async def process_operation_success(
-    task: stream_registry.ActiveTask,
-    result: dict | str | None,
-    prisma_client: Prisma | None = None,
-) -> None:
-    """Handle successful operation completion.
-
-    Publishes the result to the stream registry, updates the database,
-    generates LLM continuation, and marks the task as completed.
-
-    Args:
-        task: The active task that completed
-        result: The result data from the operation
-        prisma_client: Optional Prisma client for database operations.
-            If None, uses chat_service._update_pending_operation instead.
-
-    Raises:
-        ToolMessageUpdateError: If the database update fails. The task will be
-            marked as failed instead of completed to avoid inconsistent state.
-    """
-    # For agent generation tools, save the agent to library
-    if task.tool_name in AGENT_GENERATION_TOOLS and isinstance(result, dict):
-        result = await _save_agent_from_result(result, task.user_id, task.tool_name)
-
-    # Serialize result for output (only substitute default when result is exactly None)
-    result_output = result if result is not None else {"status": "completed"}
-    output_str = (
-        result_output
-        if isinstance(result_output, str)
-        else orjson.dumps(result_output).decode("utf-8")
-    )
-
-    # Publish result to stream registry
-    await stream_registry.publish_chunk(
-        task.task_id,
-        StreamToolOutputAvailable(
-            toolCallId=task.tool_call_id,
-            toolName=task.tool_name,
-            output=output_str,
-            success=True,
-        ),
-    )
-
-    # Update pending operation in database
-    # If this fails, we must not continue to mark the task as completed
-    result_str = serialize_result(result)
-    try:
-        await _update_tool_message(
-            session_id=task.session_id,
-            tool_call_id=task.tool_call_id,
-            content=result_str,
-            prisma_client=prisma_client,
-        )
-    except ToolMessageUpdateError:
-        # DB update failed - mark task as failed to avoid inconsistent state
-        logger.error(
-            f"[COMPLETION] DB update failed for task {task.task_id}, "
-            "marking as failed instead of completed"
-        )
-        await stream_registry.publish_chunk(
-            task.task_id,
-            StreamError(errorText="Failed to save operation result to database"),
-        )
-        await stream_registry.mark_task_completed(task.task_id, status="failed")
-        raise
-
-    # Generate LLM continuation with streaming
-    try:
-        await chat_service._generate_llm_continuation_with_streaming(
-            session_id=task.session_id,
-            user_id=task.user_id,
-            task_id=task.task_id,
-        )
-    except Exception as e:
-        logger.error(
-            f"[COMPLETION] Failed to generate LLM continuation: {e}",
-            exc_info=True,
-        )
-
-    # Mark task as completed and release Redis lock
-    await stream_registry.mark_task_completed(task.task_id, status="completed")
-    try:
-        await chat_service._mark_operation_completed(task.tool_call_id)
-    except Exception as e:
-        logger.error(f"[COMPLETION] Failed to mark operation completed: {e}")
-
-    logger.info(
-        f"[COMPLETION] Successfully processed completion for task {task.task_id}"
-    )
-
-
-async def process_operation_failure(
-    task: stream_registry.ActiveTask,
-    error: str | None,
-    prisma_client: Prisma | None = None,
-) -> None:
-    """Handle failed operation completion.
-
-    Publishes the error to the stream registry, updates the database with
-    the error response, and marks the task as failed.
-
-    Args:
-        task: The active task that failed
-        error: The error message from the operation
-        prisma_client: Optional Prisma client for database operations.
-            If None, uses chat_service._update_pending_operation instead.
-    """
-    error_msg = error or "Operation failed"
-
-    # Publish error to stream registry
-    await stream_registry.publish_chunk(
-        task.task_id,
-        StreamError(errorText=error_msg),
-    )
-
-    # Update pending operation with error
-    # If this fails, we still continue to mark the task as failed
-    error_response = ErrorResponse(
-        message=error_msg,
-        error=error,
-    )
-    try:
-        await _update_tool_message(
-            session_id=task.session_id,
-            tool_call_id=task.tool_call_id,
-            content=error_response.model_dump_json(),
-            prisma_client=prisma_client,
-        )
-    except ToolMessageUpdateError:
-        # DB update failed - log but continue with cleanup
-        logger.error(
-            f"[COMPLETION] DB update failed while processing failure for task {task.task_id}, "
-            "continuing with cleanup"
-        )
-
-    # Mark task as failed and release Redis lock
-    await stream_registry.mark_task_completed(task.task_id, status="failed")
-    try:
-        await chat_service._mark_operation_completed(task.tool_call_id)
-    except Exception as e:
-        logger.error(f"[COMPLETION] Failed to mark operation completed: {e}")
-
-    logger.info(f"[COMPLETION] Processed failure for task {task.task_id}: {error_msg}")

autogpt_platform/backend/backend/api/features/chat/db.py

@@ -1,288 +0,0 @@
-"""Database operations for chat sessions."""
-
-import asyncio
-import logging
-from datetime import UTC, datetime
-from typing import Any, cast
-
-from prisma.models import ChatMessage as PrismaChatMessage
-from prisma.models import ChatSession as PrismaChatSession
-from prisma.types import (
-    ChatMessageCreateInput,
-    ChatSessionCreateInput,
-    ChatSessionUpdateInput,
-    ChatSessionWhereInput,
-)
-
-from backend.data.db import transaction
-from backend.util.json import SafeJson
-
-logger = logging.getLogger(__name__)
-
-
-async def get_chat_session(session_id: str) -> PrismaChatSession | None:
-    """Get a chat session by ID from the database."""
-    session = await PrismaChatSession.prisma().find_unique(
-        where={"id": session_id},
-        include={"Messages": True},
-    )
-    if session and session.Messages:
-        # Sort messages by sequence in Python - Prisma Python client doesn't support
-        # order_by in include clauses (unlike Prisma JS), so we sort after fetching
-        session.Messages.sort(key=lambda m: m.sequence)
-    return session
-
-
-async def create_chat_session(
-    session_id: str,
-    user_id: str,
-) -> PrismaChatSession:
-    """Create a new chat session in the database."""
-    data = ChatSessionCreateInput(
-        id=session_id,
-        userId=user_id,
-        credentials=SafeJson({}),
-        successfulAgentRuns=SafeJson({}),
-        successfulAgentSchedules=SafeJson({}),
-    )
-    return await PrismaChatSession.prisma().create(data=data)
-
-
-async def update_chat_session(
-    session_id: str,
-    credentials: dict[str, Any] | None = None,
-    successful_agent_runs: dict[str, Any] | None = None,
-    successful_agent_schedules: dict[str, Any] | None = None,
-    total_prompt_tokens: int | None = None,
-    total_completion_tokens: int | None = None,
-    title: str | None = None,
-) -> PrismaChatSession | None:
-    """Update a chat session's metadata."""
-    data: ChatSessionUpdateInput = {"updatedAt": datetime.now(UTC)}
-
-    if credentials is not None:
-        data["credentials"] = SafeJson(credentials)
-    if successful_agent_runs is not None:
-        data["successfulAgentRuns"] = SafeJson(successful_agent_runs)
-    if successful_agent_schedules is not None:
-        data["successfulAgentSchedules"] = SafeJson(successful_agent_schedules)
-    if total_prompt_tokens is not None:
-        data["totalPromptTokens"] = total_prompt_tokens
-    if total_completion_tokens is not None:
-        data["totalCompletionTokens"] = total_completion_tokens
-    if title is not None:
-        data["title"] = title
-
-    session = await PrismaChatSession.prisma().update(
-        where={"id": session_id},
-        data=data,
-        include={"Messages": True},
-    )
-    if session and session.Messages:
-        # Sort in Python - Prisma Python doesn't support order_by in include clauses
-        session.Messages.sort(key=lambda m: m.sequence)
-    return session
-
-
-async def add_chat_message(
-    session_id: str,
-    role: str,
-    sequence: int,
-    content: str | None = None,
-    name: str | None = None,
-    tool_call_id: str | None = None,
-    refusal: str | None = None,
-    tool_calls: list[dict[str, Any]] | None = None,
-    function_call: dict[str, Any] | None = None,
-) -> PrismaChatMessage:
-    """Add a message to a chat session."""
-    # Build input dict dynamically rather than using ChatMessageCreateInput directly
-    # because Prisma's TypedDict validation rejects optional fields set to None.
-    # We only include fields that have values, then cast at the end.
-    data: dict[str, Any] = {
-        "Session": {"connect": {"id": session_id}},
-        "role": role,
-        "sequence": sequence,
-    }
-
-    # Add optional string fields
-    if content is not None:
-        data["content"] = content
-    if name is not None:
-        data["name"] = name
-    if tool_call_id is not None:
-        data["toolCallId"] = tool_call_id
-    if refusal is not None:
-        data["refusal"] = refusal
-
-    # Add optional JSON fields only when they have values
-    if tool_calls is not None:
-        data["toolCalls"] = SafeJson(tool_calls)
-    if function_call is not None:
-        data["functionCall"] = SafeJson(function_call)
-
-    # Run message create and session timestamp update in parallel for lower latency
-    _, message = await asyncio.gather(
-        PrismaChatSession.prisma().update(
-            where={"id": session_id},
-            data={"updatedAt": datetime.now(UTC)},
-        ),
-        PrismaChatMessage.prisma().create(data=cast(ChatMessageCreateInput, data)),
-    )
-    return message
-
-
-async def add_chat_messages_batch(
-    session_id: str,
-    messages: list[dict[str, Any]],
-    start_sequence: int,
-) -> list[PrismaChatMessage]:
-    """Add multiple messages to a chat session in a batch.
-
-    Uses a transaction for atomicity - if any message creation fails,
-    the entire batch is rolled back.
-    """
-    if not messages:
-        return []
-
-    created_messages = []
-
-    async with transaction() as tx:
-        for i, msg in enumerate(messages):
-            # Build input dict dynamically rather than using ChatMessageCreateInput
-            # directly because Prisma's TypedDict validation rejects optional fields
-            # set to None. We only include fields that have values, then cast.
-            data: dict[str, Any] = {
-                "Session": {"connect": {"id": session_id}},
-                "role": msg["role"],
-                "sequence": start_sequence + i,
-            }
-
-            # Add optional string fields
-            if msg.get("content") is not None:
-                data["content"] = msg["content"]
-            if msg.get("name") is not None:
-                data["name"] = msg["name"]
-            if msg.get("tool_call_id") is not None:
-                data["toolCallId"] = msg["tool_call_id"]
-            if msg.get("refusal") is not None:
-                data["refusal"] = msg["refusal"]
-
-            # Add optional JSON fields only when they have values
-            if msg.get("tool_calls") is not None:
-                data["toolCalls"] = SafeJson(msg["tool_calls"])
-            if msg.get("function_call") is not None:
-                data["functionCall"] = SafeJson(msg["function_call"])
-
-            created = await PrismaChatMessage.prisma(tx).create(
-                data=cast(ChatMessageCreateInput, data)
-            )
-            created_messages.append(created)
-
-        # Update session's updatedAt timestamp within the same transaction.
-        # Note: Token usage (total_prompt_tokens, total_completion_tokens) is updated
-        # separately via update_chat_session() after streaming completes.
-        await PrismaChatSession.prisma(tx).update(
-            where={"id": session_id},
-            data={"updatedAt": datetime.now(UTC)},
-        )
-
-    return created_messages
-
-
-async def get_user_chat_sessions(
-    user_id: str,
-    limit: int = 50,
-    offset: int = 0,
-) -> list[PrismaChatSession]:
-    """Get chat sessions for a user, ordered by most recent."""
-    return await PrismaChatSession.prisma().find_many(
-        where={"userId": user_id},
-        order={"updatedAt": "desc"},
-        take=limit,
-        skip=offset,
-    )
-
-
-async def get_user_session_count(user_id: str) -> int:
-    """Get the total number of chat sessions for a user."""
-    return await PrismaChatSession.prisma().count(where={"userId": user_id})
-
-
-async def delete_chat_session(session_id: str, user_id: str | None = None) -> bool:
-    """Delete a chat session and all its messages.
-
-    Args:
-        session_id: The session ID to delete.
-        user_id: If provided, validates that the session belongs to this user
-            before deletion. This prevents unauthorized deletion of other
-            users' sessions.
-
-    Returns:
-        True if deleted successfully, False otherwise.
-    """
-    try:
-        # Build typed where clause with optional user_id validation
-        where_clause: ChatSessionWhereInput = {"id": session_id}
-        if user_id is not None:
-            where_clause["userId"] = user_id
-
-        result = await PrismaChatSession.prisma().delete_many(where=where_clause)
-        if result == 0:
-            logger.warning(
-                f"No session deleted for {session_id} "
-                f"(user_id validation: {user_id is not None})"
-            )
-            return False
-        return True
-    except Exception as e:
-        logger.error(f"Failed to delete chat session {session_id}: {e}")
-        return False
-
-
-async def get_chat_session_message_count(session_id: str) -> int:
-    """Get the number of messages in a chat session."""
-    count = await PrismaChatMessage.prisma().count(where={"sessionId": session_id})
-    return count
-
-
-async def update_tool_message_content(
-    session_id: str,
-    tool_call_id: str,
-    new_content: str,
-) -> bool:
-    """Update the content of a tool message in chat history.
-
-    Used by background tasks to update pending operation messages with final results.
-
-    Args:
-        session_id: The chat session ID.
-        tool_call_id: The tool call ID to find the message.
-        new_content: The new content to set.
-
-    Returns:
-        True if a message was updated, False otherwise.
-    """
-    try:
-        result = await PrismaChatMessage.prisma().update_many(
-            where={
-                "sessionId": session_id,
-                "toolCallId": tool_call_id,
-            },
-            data={
-                "content": new_content,
-            },
-        )
-        if result == 0:
-            logger.warning(
-                f"No message found to update for session {session_id}, "
-                f"tool_call_id {tool_call_id}"
-            )
-            return False
-        return True
-    except Exception as e:
-        logger.error(
-            f"Failed to update tool message for session {session_id}, "
-            f"tool_call_id {tool_call_id}: {e}"
-        )
-        return False

autogpt_platform/backend/backend/api/features/chat/routes.py

@@ -2,33 +2,34 @@
 
 import asyncio
 import logging
-import uuid as uuid_module
+import re
 from collections.abc import AsyncGenerator
 from typing import Annotated
+from uuid import uuid4
 
 from autogpt_libs import auth
-from fastapi import APIRouter, Depends, Header, HTTPException, Query, Response, Security
+from fastapi import APIRouter, Depends, HTTPException, Query, Response, Security
 from fastapi.responses import StreamingResponse
-from pydantic import BaseModel
+from prisma.models import UserWorkspaceFile
+from pydantic import BaseModel, Field, field_validator
 
-from backend.util.exceptions import NotFoundError
-from backend.util.feature_flag import Flag, is_feature_enabled
-
-from . import service as chat_service
-from . import stream_registry
-from .completion_handler import process_operation_failure, process_operation_success
-from .config import ChatConfig
-from .model import (
+from backend.copilot import service as chat_service
+from backend.copilot import stream_registry
+from backend.copilot.config import ChatConfig
+from backend.copilot.executor.utils import enqueue_cancel_task, enqueue_copilot_turn
+from backend.copilot.model import (
     ChatMessage,
     ChatSession,
     append_and_save_message,
     create_chat_session,
+    delete_chat_session,
     get_chat_session,
     get_user_sessions,
+    update_session_title,
 )
-from .response_model import StreamError, StreamFinish, StreamHeartbeat, StreamStart
-from .sdk import service as sdk_service
-from .tools.models import (
+from backend.copilot.response_model import StreamError, StreamFinish, StreamHeartbeat
+from backend.copilot.tools.e2b_sandbox import kill_sandbox
+from backend.copilot.tools.models import (
     AgentDetailsResponse,
     AgentOutputResponse,
     AgentPreviewResponse,
@@ -43,18 +44,25 @@ from .tools.models import (
     ErrorResponse,
     ExecutionStartedResponse,
     InputValidationErrorResponse,
+    MCPToolOutputResponse,
+    MCPToolsDiscoveredResponse,
     NeedLoginResponse,
     NoResultsResponse,
-    OperationInProgressResponse,
-    OperationPendingResponse,
-    OperationStartedResponse,
     SetupRequirementsResponse,
+    SuggestedGoalResponse,
     UnderstandingUpdatedResponse,
 )
-from .tracking import track_user_message
+from backend.copilot.tracking import track_user_message
+from backend.data.redis_client import get_redis_async
+from backend.data.understanding import get_business_understanding
+from backend.data.workspace import get_or_create_workspace
+from backend.util.exceptions import NotFoundError
 
 config = ChatConfig()
 
+_UUID_RE = re.compile(
+    r"^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$", re.I
+)
 
 logger = logging.getLogger(__name__)
 
@@ -83,6 +91,9 @@ class StreamChatRequest(BaseModel):
     message: str
     is_user_message: bool = True
     context: dict[str, str] | None = None  # {url: str, content: str}
+    file_ids: list[str] | None = Field(
+        default=None, max_length=20
+    )  # Workspace file IDs attached to this message
 
 
 class CreateSessionResponse(BaseModel):
@@ -96,10 +107,8 @@ class CreateSessionResponse(BaseModel):
 class ActiveStreamInfo(BaseModel):
     """Information about an active stream for reconnection."""
 
-    task_id: str
+    turn_id: str
     last_message_id: str  # Redis Stream message ID for resumption
-    operation_id: str  # Operation ID for completion tracking
-    tool_name: str  # Name of the tool being executed
 
 
 class SessionDetailResponse(BaseModel):
@@ -120,6 +129,7 @@ class SessionSummaryResponse(BaseModel):
     created_at: str
     updated_at: str
     title: str | None = None
+    is_processing: bool
 
 
 class ListSessionsResponse(BaseModel):
@@ -129,12 +139,25 @@ class ListSessionsResponse(BaseModel):
     total: int
 
 
-class OperationCompleteRequest(BaseModel):
-    """Request model for external completion webhook."""
+class CancelSessionResponse(BaseModel):
+    """Response model for the cancel session endpoint."""
 
-    success: bool
-    result: dict | str | None = None
-    error: str | None = None
+    cancelled: bool
+    reason: str | None = None
+
+
+class UpdateSessionTitleRequest(BaseModel):
+    """Request model for updating a session's title."""
+
+    title: str
+
+    @field_validator("title")
+    @classmethod
+    def title_must_not_be_blank(cls, v: str) -> str:
+        stripped = v.strip()
+        if not stripped:
+            raise ValueError("Title must not be blank")
+        return stripped
 
 
 # ========== Routes ==========
@@ -165,6 +188,28 @@ async def list_sessions(
     """
     sessions, total_count = await get_user_sessions(user_id, limit, offset)
 
+    # Batch-check Redis for active stream status on each session
+    processing_set: set[str] = set()
+    if sessions:
+        try:
+            redis = await get_redis_async()
+            pipe = redis.pipeline(transaction=False)
+            for session in sessions:
+                pipe.hget(
+                    f"{config.session_meta_prefix}{session.session_id}",
+                    "status",
+                )
+            statuses = await pipe.execute()
+            processing_set = {
+                session.session_id
+                for session, st in zip(sessions, statuses)
+                if st == "running"
+            }
+        except Exception:
+            logger.warning(
+                "Failed to fetch processing status from Redis; " "defaulting to empty"
+            )
+
     return ListSessionsResponse(
         sessions=[
             SessionSummaryResponse(
@@ -172,6 +217,7 @@ async def list_sessions(
                 created_at=session.started_at.isoformat(),
                 updated_at=session.updated_at.isoformat(),
                 title=session.title,
+                is_processing=session.session_id in processing_set,
             )
             for session in sessions
         ],
@@ -211,6 +257,92 @@ async def create_session(
     )
 
 
+@router.delete(
+    "/sessions/{session_id}",
+    dependencies=[Security(auth.requires_user)],
+    status_code=204,
+    responses={404: {"description": "Session not found or access denied"}},
+)
+async def delete_session(
+    session_id: str,
+    user_id: Annotated[str, Security(auth.get_user_id)],
+) -> Response:
+    """
+    Delete a chat session.
+
+    Permanently removes a chat session and all its messages.
+    Only the owner can delete their sessions.
+
+    Args:
+        session_id: The session ID to delete.
+        user_id: The authenticated user's ID.
+
+    Returns:
+        204 No Content on success.
+
+    Raises:
+        HTTPException: 404 if session not found or not owned by user.
+    """
+    deleted = await delete_chat_session(session_id, user_id)
+
+    if not deleted:
+        raise HTTPException(
+            status_code=404,
+            detail=f"Session {session_id} not found or access denied",
+        )
+
+    # Best-effort cleanup of the E2B sandbox (if any).
+    # sandbox_id is in Redis; kill_sandbox() fetches it from there.
+    e2b_cfg = ChatConfig()
+    if e2b_cfg.e2b_active:
+        assert e2b_cfg.e2b_api_key  # guaranteed by e2b_active check
+        try:
+            await kill_sandbox(session_id, e2b_cfg.e2b_api_key)
+        except Exception:
+            logger.warning(
+                "[E2B] Failed to kill sandbox for session %s", session_id[:12]
+            )
+
+    return Response(status_code=204)
+
+
+@router.patch(
+    "/sessions/{session_id}/title",
+    summary="Update session title",
+    dependencies=[Security(auth.requires_user)],
+    status_code=200,
+    responses={404: {"description": "Session not found or access denied"}},
+)
+async def update_session_title_route(
+    session_id: str,
+    request: UpdateSessionTitleRequest,
+    user_id: Annotated[str, Security(auth.get_user_id)],
+) -> dict:
+    """
+    Update the title of a chat session.
+
+    Allows the user to rename their chat session.
+
+    Args:
+        session_id: The session ID to update.
+        request: Request body containing the new title.
+        user_id: The authenticated user's ID.
+
+    Returns:
+        dict: Status of the update.
+
+    Raises:
+        HTTPException: 404 if session not found or not owned by user.
+    """
+    success = await update_session_title(session_id, user_id, request.title)
+    if not success:
+        raise HTTPException(
+            status_code=404,
+            detail=f"Session {session_id} not found or access denied",
+        )
+    return {"status": "ok"}
+
+
 @router.get(
     "/sessions/{session_id}",
 )
@@ -222,7 +354,7 @@ async def get_session(
     Retrieve the details of a specific chat session.
 
     Looks up a chat session by ID for the given user (if authenticated) and returns all session data including messages.
-    If there's an active stream for this session, returns the task_id for reconnection.
+    If there's an active stream for this session, returns active_stream info for reconnection.
 
     Args:
         session_id: The unique identifier for the desired chat session.
@@ -240,28 +372,21 @@ async def get_session(
 
     # Check if there's an active stream for this session
     active_stream_info = None
-    active_task, last_message_id = await stream_registry.get_active_task_for_session(
+    active_session, last_message_id = await stream_registry.get_active_session(
         session_id, user_id
     )
     logger.info(
-        f"[GET_SESSION] session={session_id}, active_task={active_task is not None}, "
+        f"[GET_SESSION] session={session_id}, active_session={active_session is not None}, "
         f"msg_count={len(messages)}, last_role={messages[-1].get('role') if messages else 'none'}"
     )
-    if active_task:
-        # Filter out the in-progress assistant message from the session response.
-        # The client will receive the complete assistant response through the SSE
-        # stream replay instead, preventing duplicate content.
-        if messages and messages[-1].get("role") == "assistant":
-            messages = messages[:-1]
-
-        # Use "0-0" as last_message_id to replay the stream from the beginning.
-        # Since we filtered out the cached assistant message, the client needs
-        # the full stream to reconstruct the response.
+    if active_session:
+        # Keep the assistant message (including tool_calls) so the frontend can
+        # render the correct tool UI (e.g. CreateAgent with mini game).
+        # convertChatSessionToUiMessages handles isComplete=false by setting
+        # tool parts without output to state "input-available".
         active_stream_info = ActiveStreamInfo(
-            task_id=active_task.task_id,
-            last_message_id="0-0",
-            operation_id=active_task.operation_id,
-            tool_name=active_task.tool_name,
+            turn_id=active_session.turn_id,
+            last_message_id=last_message_id,
         )
 
     return SessionDetailResponse(
@@ -274,6 +399,51 @@ async def get_session(
     )
 
 
+@router.post(
+    "/sessions/{session_id}/cancel",
+    status_code=200,
+)
+async def cancel_session_task(
+    session_id: str,
+    user_id: Annotated[str | None, Depends(auth.get_user_id)],
+) -> CancelSessionResponse:
+    """Cancel the active streaming task for a session.
+
+    Publishes a cancel event to the executor via RabbitMQ FANOUT, then
+    polls Redis until the task status flips from ``running`` or a timeout
+    (5 s) is reached.  Returns only after the cancellation is confirmed.
+    """
+    await _validate_and_get_session(session_id, user_id)
+
+    active_session, _ = await stream_registry.get_active_session(session_id, user_id)
+    if not active_session:
+        return CancelSessionResponse(cancelled=True, reason="no_active_session")
+
+    await enqueue_cancel_task(session_id)
+    logger.info(f"[CANCEL] Published cancel for session ...{session_id[-8:]}")
+
+    # Poll until the executor confirms the task is no longer running.
+    poll_interval = 0.5
+    max_wait = 5.0
+    waited = 0.0
+    while waited < max_wait:
+        await asyncio.sleep(poll_interval)
+        waited += poll_interval
+        session_state = await stream_registry.get_session(session_id)
+        if session_state is None or session_state.status != "running":
+            logger.info(
+                f"[CANCEL] Session ...{session_id[-8:]} confirmed stopped "
+                f"(status={session_state.status if session_state else 'gone'}) after {waited:.1f}s"
+            )
+            return CancelSessionResponse(cancelled=True)
+
+    logger.warning(
+        f"[CANCEL] Session ...{session_id[-8:]} not confirmed after {max_wait}s, force-completing"
+    )
+    await stream_registry.mark_session_completed(session_id, error_message="Cancelled")
+    return CancelSessionResponse(cancelled=True)
+
+
 @router.post(
     "/sessions/{session_id}/stream",
 )
@@ -291,16 +461,15 @@ async def stream_chat_post(
       - Tool execution results
 
     The AI generation runs in a background task that continues even if the client disconnects.
-    All chunks are written to Redis for reconnection support. If the client disconnects,
-    they can reconnect using GET /tasks/{task_id}/stream to resume from where they left off.
+    All chunks are written to a per-turn Redis stream for reconnection support. If the client
+    disconnects, they can reconnect using GET /sessions/{session_id}/stream to resume.
 
     Args:
         session_id: The chat session identifier to associate with the streamed messages.
         request: Request body containing message, is_user_message, and optional context.
         user_id: Optional authenticated user ID.
     Returns:
-        StreamingResponse: SSE-formatted response chunks. First chunk is a "start" event
-        containing the task_id for reconnection.
+        StreamingResponse: SSE-formatted response chunks.
 
     """
     import asyncio
@@ -316,7 +485,7 @@ async def stream_chat_post(
         f"user={user_id}, message_len={len(request.message)}",
         extra={"json_fields": log_meta},
     )
-    session = await _validate_and_get_session(session_id, user_id)
+    await _validate_and_get_session(session_id, user_id)
     logger.info(
         f"[TIMING] session validated in {(time.perf_counter() - stream_start_time) * 1000:.1f}ms",
         extra={
@@ -327,6 +496,38 @@ async def stream_chat_post(
         },
     )
 
+    # Enrich message with file metadata if file_ids are provided.
+    # Also sanitise file_ids so only validated, workspace-scoped IDs are
+    # forwarded downstream (e.g. to the executor via enqueue_copilot_turn).
+    sanitized_file_ids: list[str] | None = None
+    if request.file_ids and user_id:
+        # Filter to valid UUIDs only to prevent DB abuse
+        valid_ids = [fid for fid in request.file_ids if _UUID_RE.match(fid)]
+
+        if valid_ids:
+            workspace = await get_or_create_workspace(user_id)
+            # Batch query instead of N+1
+            files = await UserWorkspaceFile.prisma().find_many(
+                where={
+                    "id": {"in": valid_ids},
+                    "workspaceId": workspace.id,
+                    "isDeleted": False,
+                }
+            )
+            # Only keep IDs that actually exist in the user's workspace
+            sanitized_file_ids = [wf.id for wf in files] or None
+            file_lines: list[str] = [
+                f"- {wf.name} ({wf.mimeType}, {round(wf.sizeBytes / 1024, 1)} KB), file_id={wf.id}"
+                for wf in files
+            ]
+            if file_lines:
+                files_block = (
+                    "\n\n[Attached files]\n"
+                    + "\n".join(file_lines)
+                    + "\nUse read_workspace_file with the file_id to access file contents."
+                )
+                request.message += files_block
+
     # Atomically append user message to session BEFORE creating task to avoid
     # race condition where GET_SESSION sees task as "running" but message isn't
     # saved yet.  append_and_save_message re-fetches inside a lock to prevent
@@ -343,152 +544,47 @@ async def stream_chat_post(
                 message_length=len(request.message),
             )
         logger.info(f"[STREAM] Saving user message to session {session_id}")
-        session = await append_and_save_message(session_id, message)
+        await append_and_save_message(session_id, message)
         logger.info(f"[STREAM] User message saved for session {session_id}")
 
     # Create a task in the stream registry for reconnection support
-    task_id = str(uuid_module.uuid4())
-    operation_id = str(uuid_module.uuid4())
-    log_meta["task_id"] = task_id
+    turn_id = str(uuid4())
+    log_meta["turn_id"] = turn_id
 
-    task_create_start = time.perf_counter()
-    await stream_registry.create_task(
-        task_id=task_id,
+    session_create_start = time.perf_counter()
+    await stream_registry.create_session(
         session_id=session_id,
         user_id=user_id,
-        tool_call_id="chat_stream",  # Not a tool call, but needed for the model
+        tool_call_id="chat_stream",
         tool_name="chat",
-        operation_id=operation_id,
+        turn_id=turn_id,
     )
     logger.info(
-        f"[TIMING] create_task completed in {(time.perf_counter() - task_create_start) * 1000:.1f}ms",
+        f"[TIMING] create_session completed in {(time.perf_counter() - session_create_start) * 1000:.1f}ms",
         extra={
             "json_fields": {
                 **log_meta,
-                "duration_ms": (time.perf_counter() - task_create_start) * 1000,
+                "duration_ms": (time.perf_counter() - session_create_start) * 1000,
             }
         },
     )
 
-    # Background task that runs the AI generation independently of SSE connection
-    async def run_ai_generation():
-        import time as time_module
+    # Per-turn stream is always fresh (unique turn_id), subscribe from beginning
+    subscribe_from_id = "0-0"
 
-        gen_start_time = time_module.perf_counter()
-        logger.info(
-            f"[TIMING] run_ai_generation STARTED, task={task_id}, session={session_id}, user={user_id}",
-            extra={"json_fields": log_meta},
-        )
-        first_chunk_time, ttfc = None, None
-        chunk_count = 0
-        try:
-            # Emit a start event with task_id for reconnection
-            start_chunk = StreamStart(messageId=task_id, taskId=task_id)
-            await stream_registry.publish_chunk(task_id, start_chunk)
-            logger.info(
-                f"[TIMING] StreamStart published at {(time_module.perf_counter() - gen_start_time) * 1000:.1f}ms",
-                extra={
-                    "json_fields": {
-                        **log_meta,
-                        "elapsed_ms": (time_module.perf_counter() - gen_start_time)
-                        * 1000,
-                    }
-                },
-            )
+    await enqueue_copilot_turn(
+        session_id=session_id,
+        user_id=user_id,
+        message=request.message,
+        turn_id=turn_id,
+        is_user_message=request.is_user_message,
+        context=request.context,
+        file_ids=sanitized_file_ids,
+    )
 
-            # Choose service based on LaunchDarkly flag (falls back to config default)
-            use_sdk = await is_feature_enabled(
-                Flag.COPILOT_SDK,
-                user_id or "anonymous",
-                default=config.use_claude_agent_sdk,
-            )
-            stream_fn = (
-                sdk_service.stream_chat_completion_sdk
-                if use_sdk
-                else chat_service.stream_chat_completion
-            )
-            logger.info(
-                f"[TIMING] Calling {'sdk' if use_sdk else 'standard'} stream_chat_completion",
-                extra={"json_fields": log_meta},
-            )
-            # Pass message=None since we already added it to the session above
-            async for chunk in stream_fn(
-                session_id,
-                None,  # Message already in session
-                is_user_message=request.is_user_message,
-                user_id=user_id,
-                session=session,  # Pass session with message already added
-                context=request.context,
-            ):
-                # Skip duplicate StreamStart — we already published one above
-                if isinstance(chunk, StreamStart):
-                    continue
-                chunk_count += 1
-                if first_chunk_time is None:
-                    first_chunk_time = time_module.perf_counter()
-                    ttfc = first_chunk_time - gen_start_time
-                    logger.info(
-                        f"[TIMING] FIRST AI CHUNK at {ttfc:.2f}s, type={type(chunk).__name__}",
-                        extra={
-                            "json_fields": {
-                                **log_meta,
-                                "chunk_type": type(chunk).__name__,
-                                "time_to_first_chunk_ms": ttfc * 1000,
-                            }
-                        },
-                    )
-                # Write to Redis (subscribers will receive via XREAD)
-                await stream_registry.publish_chunk(task_id, chunk)
-
-            gen_end_time = time_module.perf_counter()
-            total_time = (gen_end_time - gen_start_time) * 1000
-            logger.info(
-                f"[TIMING] run_ai_generation FINISHED in {total_time / 1000:.1f}s; "
-                f"task={task_id}, session={session_id}, "
-                f"ttfc={ttfc or -1:.2f}s, n_chunks={chunk_count}",
-                extra={
-                    "json_fields": {
-                        **log_meta,
-                        "total_time_ms": total_time,
-                        "time_to_first_chunk_ms": (
-                            ttfc * 1000 if ttfc is not None else None
-                        ),
-                        "n_chunks": chunk_count,
-                    }
-                },
-            )
-            await stream_registry.mark_task_completed(task_id, "completed")
-        except Exception as e:
-            elapsed = time_module.perf_counter() - gen_start_time
-            logger.error(
-                f"[TIMING] run_ai_generation ERROR after {elapsed:.2f}s: {e}",
-                extra={
-                    "json_fields": {
-                        **log_meta,
-                        "elapsed_ms": elapsed * 1000,
-                        "error": str(e),
-                    }
-                },
-            )
-            # Publish a StreamError so the frontend can display an error message
-            try:
-                await stream_registry.publish_chunk(
-                    task_id,
-                    StreamError(
-                        errorText="An error occurred. Please try again.",
-                        code="stream_error",
-                    ),
-                )
-            except Exception:
-                pass  # Best-effort; mark_task_completed will publish StreamFinish
-            await stream_registry.mark_task_completed(task_id, "failed")
-
-    # Start the AI generation in a background task
-    bg_task = asyncio.create_task(run_ai_generation())
-    await stream_registry.set_task_asyncio_task(task_id, bg_task)
     setup_time = (time.perf_counter() - stream_start_time) * 1000
     logger.info(
-        f"[TIMING] Background task started, setup={setup_time:.1f}ms",
+        f"[TIMING] Task enqueued to RabbitMQ, setup={setup_time:.1f}ms",
         extra={"json_fields": {**log_meta, "setup_time_ms": setup_time}},
     )
 
@@ -498,7 +594,7 @@ async def stream_chat_post(
 
         event_gen_start = time_module.perf_counter()
         logger.info(
-            f"[TIMING] event_generator STARTED, task={task_id}, session={session_id}, "
+            f"[TIMING] event_generator STARTED, turn={turn_id}, session={session_id}, "
             f"user={user_id}",
             extra={"json_fields": log_meta},
         )
@@ -506,11 +602,12 @@ async def stream_chat_post(
         first_chunk_yielded = False
         chunks_yielded = 0
         try:
-            # Subscribe to the task stream (this replays existing messages + live updates)
-            subscriber_queue = await stream_registry.subscribe_to_task(
-                task_id=task_id,
+            # Subscribe from the position we captured before enqueuing
+            # This avoids replaying old messages while catching all new ones
+            subscriber_queue = await stream_registry.subscribe_to_session(
+                session_id=session_id,
                 user_id=user_id,
-                last_message_id="0-0",  # Get all messages from the beginning
+                last_message_id=subscribe_from_id,
             )
 
             if subscriber_queue is None:
@@ -525,7 +622,7 @@ async def stream_chat_post(
             )
             while True:
                 try:
-                    chunk = await asyncio.wait_for(subscriber_queue.get(), timeout=30.0)
+                    chunk = await asyncio.wait_for(subscriber_queue.get(), timeout=10.0)
                     chunks_yielded += 1
 
                     if not first_chunk_yielded:
@@ -593,19 +690,19 @@ async def stream_chat_post(
             # Unsubscribe when client disconnects or stream ends
             if subscriber_queue is not None:
                 try:
-                    await stream_registry.unsubscribe_from_task(
-                        task_id, subscriber_queue
+                    await stream_registry.unsubscribe_from_session(
+                        session_id, subscriber_queue
                     )
                 except Exception as unsub_err:
                     logger.error(
-                        f"Error unsubscribing from task {task_id}: {unsub_err}",
+                        f"Error unsubscribing from session {session_id}: {unsub_err}",
                         exc_info=True,
                     )
             # AI SDK protocol termination - always yield even if unsubscribe fails
             total_time = time_module.perf_counter() - event_gen_start
             logger.info(
                 f"[TIMING] event_generator FINISHED in {total_time:.2f}s; "
-                f"task={task_id}, session={session_id}, n_chunks={chunks_yielded}",
+                f"turn={turn_id}, session={session_id}, n_chunks={chunks_yielded}",
                 extra={
                     "json_fields": {
                         **log_meta,
@@ -652,17 +749,21 @@ async def resume_session_stream(
     """
     import asyncio
 
-    active_task, _last_id = await stream_registry.get_active_task_for_session(
+    active_session, last_message_id = await stream_registry.get_active_session(
         session_id, user_id
     )
 
-    if not active_task:
+    if not active_session:
         return Response(status_code=204)
 
-    subscriber_queue = await stream_registry.subscribe_to_task(
-        task_id=active_task.task_id,
+    # Always replay from the beginning ("0-0") on resume.
+    # We can't use last_message_id because it's the latest ID in the backend
+    # stream, not the latest the frontend received — the gap causes lost
+    # messages. The frontend deduplicates replayed content.
+    subscriber_queue = await stream_registry.subscribe_to_session(
+        session_id=session_id,
         user_id=user_id,
-        last_message_id="0-0",  # Full replay so useChat rebuilds the message
+        last_message_id="0-0",
     )
 
     if subscriber_queue is None:
@@ -674,7 +775,7 @@ async def resume_session_stream(
         try:
             while True:
                 try:
-                    chunk = await asyncio.wait_for(subscriber_queue.get(), timeout=30.0)
+                    chunk = await asyncio.wait_for(subscriber_queue.get(), timeout=10.0)
                     if chunk_count < 3:
                         logger.info(
                             "Resume stream chunk",
@@ -698,12 +799,12 @@ async def resume_session_stream(
             logger.error(f"Error in resume stream for session {session_id}: {e}")
         finally:
             try:
-                await stream_registry.unsubscribe_from_task(
-                    active_task.task_id, subscriber_queue
+                await stream_registry.unsubscribe_from_session(
+                    session_id, subscriber_queue
                 )
             except Exception as unsub_err:
                 logger.error(
-                    f"Error unsubscribing from task {active_task.task_id}: {unsub_err}",
+                    f"Error unsubscribing from session {active_session.session_id}: {unsub_err}",
                     exc_info=True,
                 )
             logger.info(
@@ -731,7 +832,6 @@ async def resume_session_stream(
 @router.patch(
     "/sessions/{session_id}/assign-user",
     dependencies=[Security(auth.requires_user)],
-    status_code=200,
 )
 async def session_assign_user(
     session_id: str,
@@ -754,227 +854,34 @@ async def session_assign_user(
     return {"status": "ok"}
 
 
-# ========== Task Streaming (SSE Reconnection) ==========
+# ========== Suggested Prompts ==========
+
+
+class SuggestedPromptsResponse(BaseModel):
+    """Response model for user-specific suggested prompts."""
+
+    prompts: list[str]
 
 
 @router.get(
-    "/tasks/{task_id}/stream",
+    "/suggested-prompts",
+    dependencies=[Security(auth.requires_user)],
 )
-async def stream_task(
-    task_id: str,
-    user_id: str | None = Depends(auth.get_user_id),
-    last_message_id: str = Query(
-        default="0-0",
-        description="Last Redis Stream message ID received (e.g., '1706540123456-0'). Use '0-0' for full replay.",
-    ),
-):
+async def get_suggested_prompts(
+    user_id: Annotated[str, Security(auth.get_user_id)],
+) -> SuggestedPromptsResponse:
     """
-    Reconnect to a long-running task's SSE stream.
+    Get LLM-generated suggested prompts for the authenticated user.
 
-    When a long-running operation (like agent generation) starts, the client
-    receives a task_id. If the connection drops, the client can reconnect
-    using this endpoint to resume receiving updates.
-
-    Args:
-        task_id: The task ID from the operation_started response.
-        user_id: Authenticated user ID for ownership validation.
-        last_message_id: Last Redis Stream message ID received ("0-0" for full replay).
-
-    Returns:
-        StreamingResponse: SSE-formatted response chunks starting after last_message_id.
-
-    Raises:
-        HTTPException: 404 if task not found, 410 if task expired, 403 if access denied.
+    Returns personalized quick-action prompts based on the user's
+    business understanding. Returns an empty list if no custom prompts
+    are available.
     """
-    # Check task existence and expiry before subscribing
-    task, error_code = await stream_registry.get_task_with_expiry_info(task_id)
+    understanding = await get_business_understanding(user_id)
+    if understanding is None:
+        return SuggestedPromptsResponse(prompts=[])
 
-    if error_code == "TASK_EXPIRED":
-        raise HTTPException(
-            status_code=410,
-            detail={
-                "code": "TASK_EXPIRED",
-                "message": "This operation has expired. Please try again.",
-            },
-        )
-
-    if error_code == "TASK_NOT_FOUND":
-        raise HTTPException(
-            status_code=404,
-            detail={
-                "code": "TASK_NOT_FOUND",
-                "message": f"Task {task_id} not found.",
-            },
-        )
-
-    # Validate ownership if task has an owner
-    if task and task.user_id and user_id != task.user_id:
-        raise HTTPException(
-            status_code=403,
-            detail={
-                "code": "ACCESS_DENIED",
-                "message": "You do not have access to this task.",
-            },
-        )
-
-    # Get subscriber queue from stream registry
-    subscriber_queue = await stream_registry.subscribe_to_task(
-        task_id=task_id,
-        user_id=user_id,
-        last_message_id=last_message_id,
-    )
-
-    if subscriber_queue is None:
-        raise HTTPException(
-            status_code=404,
-            detail={
-                "code": "TASK_NOT_FOUND",
-                "message": f"Task {task_id} not found or access denied.",
-            },
-        )
-
-    async def event_generator() -> AsyncGenerator[str, None]:
-        heartbeat_interval = 15.0  # Send heartbeat every 15 seconds
-        try:
-            while True:
-                try:
-                    # Wait for next chunk with timeout for heartbeats
-                    chunk = await asyncio.wait_for(
-                        subscriber_queue.get(), timeout=heartbeat_interval
-                    )
-                    yield chunk.to_sse()
-
-                    # Check for finish signal
-                    if isinstance(chunk, StreamFinish):
-                        break
-                except asyncio.TimeoutError:
-                    # Send heartbeat to keep connection alive
-                    yield StreamHeartbeat().to_sse()
-        except Exception as e:
-            logger.error(f"Error in task stream {task_id}: {e}", exc_info=True)
-        finally:
-            # Unsubscribe when client disconnects or stream ends
-            try:
-                await stream_registry.unsubscribe_from_task(task_id, subscriber_queue)
-            except Exception as unsub_err:
-                logger.error(
-                    f"Error unsubscribing from task {task_id}: {unsub_err}",
-                    exc_info=True,
-                )
-            # AI SDK protocol termination - always yield even if unsubscribe fails
-            yield "data: [DONE]\n\n"
-
-    return StreamingResponse(
-        event_generator(),
-        media_type="text/event-stream",
-        headers={
-            "Cache-Control": "no-cache",
-            "Connection": "keep-alive",
-            "X-Accel-Buffering": "no",
-            "x-vercel-ai-ui-message-stream": "v1",
-        },
-    )
-
-
-@router.get(
-    "/tasks/{task_id}",
-)
-async def get_task_status(
-    task_id: str,
-    user_id: str | None = Depends(auth.get_user_id),
-) -> dict:
-    """
-    Get the status of a long-running task.
-
-    Args:
-        task_id: The task ID to check.
-        user_id: Authenticated user ID for ownership validation.
-
-    Returns:
-        dict: Task status including task_id, status, tool_name, and operation_id.
-
-    Raises:
-        NotFoundError: If task_id is not found or user doesn't have access.
-    """
-    task = await stream_registry.get_task(task_id)
-
-    if task is None:
-        raise NotFoundError(f"Task {task_id} not found.")
-
-    # Validate ownership - if task has an owner, requester must match
-    if task.user_id and user_id != task.user_id:
-        raise NotFoundError(f"Task {task_id} not found.")
-
-    return {
-        "task_id": task.task_id,
-        "session_id": task.session_id,
-        "status": task.status,
-        "tool_name": task.tool_name,
-        "operation_id": task.operation_id,
-        "created_at": task.created_at.isoformat(),
-    }
-
-
-# ========== External Completion Webhook ==========
-
-
-@router.post(
-    "/operations/{operation_id}/complete",
-    status_code=200,
-)
-async def complete_operation(
-    operation_id: str,
-    request: OperationCompleteRequest,
-    x_api_key: str | None = Header(default=None),
-) -> dict:
-    """
-    External completion webhook for long-running operations.
-
-    Called by Agent Generator (or other services) when an operation completes.
-    This triggers the stream registry to publish completion and continue LLM generation.
-
-    Args:
-        operation_id: The operation ID to complete.
-        request: Completion payload with success status and result/error.
-        x_api_key: Internal API key for authentication.
-
-    Returns:
-        dict: Status of the completion.
-
-    Raises:
-        HTTPException: If API key is invalid or operation not found.
-    """
-    # Validate internal API key - reject if not configured or invalid
-    if not config.internal_api_key:
-        logger.error(
-            "Operation complete webhook rejected: CHAT_INTERNAL_API_KEY not configured"
-        )
-        raise HTTPException(
-            status_code=503,
-            detail="Webhook not available: internal API key not configured",
-        )
-    if x_api_key != config.internal_api_key:
-        raise HTTPException(status_code=401, detail="Invalid API key")
-
-    # Find task by operation_id
-    task = await stream_registry.find_task_by_operation_id(operation_id)
-    if task is None:
-        raise HTTPException(
-            status_code=404,
-            detail=f"Operation {operation_id} not found",
-        )
-
-    logger.info(
-        f"Received completion webhook for operation {operation_id} "
-        f"(task_id={task.task_id}, success={request.success})"
-    )
-
-    if request.success:
-        await process_operation_success(task, request.result)
-    else:
-        await process_operation_failure(task, request.error)
-
-    return {"status": "ok", "task_id": task.task_id}
+    return SuggestedPromptsResponse(prompts=understanding.suggested_prompts)
 
 
 # ========== Configuration ==========
@@ -1051,14 +958,14 @@ ToolResponseUnion = (
     | AgentPreviewResponse
     | AgentSavedResponse
     | ClarificationNeededResponse
+    | SuggestedGoalResponse
     | BlockListResponse
     | BlockDetailsResponse
     | BlockOutputResponse
     | DocSearchResultsResponse
     | DocPageResponse
-    | OperationStartedResponse
-    | OperationPendingResponse
-    | OperationInProgressResponse
+    | MCPToolsDiscoveredResponse
+    | MCPToolOutputResponse
 )
 
 

autogpt_platform/backend/backend/api/features/chat/routes_test.py

@@ -0,0 +1,310 @@
+"""Tests for chat API routes: session title update, file attachment validation, and suggested prompts."""
+
+from unittest.mock import AsyncMock, MagicMock
+
+import fastapi
+import fastapi.testclient
+import pytest
+import pytest_mock
+
+from backend.api.features.chat import routes as chat_routes
+
+app = fastapi.FastAPI()
+app.include_router(chat_routes.router)
+
+client = fastapi.testclient.TestClient(app)
+
+TEST_USER_ID = "3e53486c-cf57-477e-ba2a-cb02dc828e1a"
+
+
+@pytest.fixture(autouse=True)
+def setup_app_auth(mock_jwt_user):
+    """Setup auth overrides for all tests in this module"""
+    from autogpt_libs.auth.jwt_utils import get_jwt_payload
+
+    app.dependency_overrides[get_jwt_payload] = mock_jwt_user["get_jwt_payload"]
+    yield
+    app.dependency_overrides.clear()
+
+
+def _mock_update_session_title(
+    mocker: pytest_mock.MockerFixture, *, success: bool = True
+):
+    """Mock update_session_title."""
+    return mocker.patch(
+        "backend.api.features.chat.routes.update_session_title",
+        new_callable=AsyncMock,
+        return_value=success,
+    )
+
+
+# ─── Update title: success ─────────────────────────────────────────────
+
+
+def test_update_title_success(
+    mocker: pytest_mock.MockerFixture,
+    test_user_id: str,
+) -> None:
+    mock_update = _mock_update_session_title(mocker, success=True)
+
+    response = client.patch(
+        "/sessions/sess-1/title",
+        json={"title": "My project"},
+    )
+
+    assert response.status_code == 200
+    assert response.json() == {"status": "ok"}
+    mock_update.assert_called_once_with("sess-1", test_user_id, "My project")
+
+
+def test_update_title_trims_whitespace(
+    mocker: pytest_mock.MockerFixture,
+    test_user_id: str,
+) -> None:
+    mock_update = _mock_update_session_title(mocker, success=True)
+
+    response = client.patch(
+        "/sessions/sess-1/title",
+        json={"title": "  trimmed  "},
+    )
+
+    assert response.status_code == 200
+    mock_update.assert_called_once_with("sess-1", test_user_id, "trimmed")
+
+
+# ─── Update title: blank / whitespace-only → 422 ──────────────────────
+
+
+def test_update_title_blank_rejected(
+    test_user_id: str,
+) -> None:
+    """Whitespace-only titles must be rejected before hitting the DB."""
+    response = client.patch(
+        "/sessions/sess-1/title",
+        json={"title": "   "},
+    )
+
+    assert response.status_code == 422
+
+
+def test_update_title_empty_rejected(
+    test_user_id: str,
+) -> None:
+    response = client.patch(
+        "/sessions/sess-1/title",
+        json={"title": ""},
+    )
+
+    assert response.status_code == 422
+
+
+# ─── Update title: session not found or wrong user → 404 ──────────────
+
+
+def test_update_title_not_found(
+    mocker: pytest_mock.MockerFixture,
+    test_user_id: str,
+) -> None:
+    _mock_update_session_title(mocker, success=False)
+
+    response = client.patch(
+        "/sessions/sess-1/title",
+        json={"title": "New name"},
+    )
+
+    assert response.status_code == 404
+
+
+# ─── file_ids Pydantic validation ─────────────────────────────────────
+
+
+def test_stream_chat_rejects_too_many_file_ids():
+    """More than 20 file_ids should be rejected by Pydantic validation (422)."""
+    response = client.post(
+        "/sessions/sess-1/stream",
+        json={
+            "message": "hello",
+            "file_ids": [f"00000000-0000-0000-0000-{i:012d}" for i in range(21)],
+        },
+    )
+    assert response.status_code == 422
+
+
+def _mock_stream_internals(mocker: pytest_mock.MockFixture):
+    """Mock the async internals of stream_chat_post so tests can exercise
+    validation and enrichment logic without needing Redis/RabbitMQ."""
+    mocker.patch(
+        "backend.api.features.chat.routes._validate_and_get_session",
+        return_value=None,
+    )
+    mocker.patch(
+        "backend.api.features.chat.routes.append_and_save_message",
+        return_value=None,
+    )
+    mock_registry = mocker.MagicMock()
+    mock_registry.create_session = mocker.AsyncMock(return_value=None)
+    mocker.patch(
+        "backend.api.features.chat.routes.stream_registry",
+        mock_registry,
+    )
+    mocker.patch(
+        "backend.api.features.chat.routes.enqueue_copilot_turn",
+        return_value=None,
+    )
+    mocker.patch(
+        "backend.api.features.chat.routes.track_user_message",
+        return_value=None,
+    )
+
+
+def test_stream_chat_accepts_20_file_ids(mocker: pytest_mock.MockFixture):
+    """Exactly 20 file_ids should be accepted (not rejected by validation)."""
+    _mock_stream_internals(mocker)
+    # Patch workspace lookup as imported by the routes module
+    mocker.patch(
+        "backend.api.features.chat.routes.get_or_create_workspace",
+        return_value=type("W", (), {"id": "ws-1"})(),
+    )
+    mock_prisma = mocker.MagicMock()
+    mock_prisma.find_many = mocker.AsyncMock(return_value=[])
+    mocker.patch(
+        "prisma.models.UserWorkspaceFile.prisma",
+        return_value=mock_prisma,
+    )
+
+    response = client.post(
+        "/sessions/sess-1/stream",
+        json={
+            "message": "hello",
+            "file_ids": [f"00000000-0000-0000-0000-{i:012d}" for i in range(20)],
+        },
+    )
+    # Should get past validation — 200 streaming response expected
+    assert response.status_code == 200
+
+
+# ─── UUID format filtering ─────────────────────────────────────────────
+
+
+def test_file_ids_filters_invalid_uuids(mocker: pytest_mock.MockFixture):
+    """Non-UUID strings in file_ids should be silently filtered out
+    and NOT passed to the database query."""
+    _mock_stream_internals(mocker)
+    mocker.patch(
+        "backend.api.features.chat.routes.get_or_create_workspace",
+        return_value=type("W", (), {"id": "ws-1"})(),
+    )
+
+    mock_prisma = mocker.MagicMock()
+    mock_prisma.find_many = mocker.AsyncMock(return_value=[])
+    mocker.patch(
+        "prisma.models.UserWorkspaceFile.prisma",
+        return_value=mock_prisma,
+    )
+
+    valid_id = "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"
+    client.post(
+        "/sessions/sess-1/stream",
+        json={
+            "message": "hello",
+            "file_ids": [
+                valid_id,
+                "not-a-uuid",
+                "../../../etc/passwd",
+                "",
+            ],
+        },
+    )
+
+    # The find_many call should only receive the one valid UUID
+    mock_prisma.find_many.assert_called_once()
+    call_kwargs = mock_prisma.find_many.call_args[1]
+    assert call_kwargs["where"]["id"]["in"] == [valid_id]
+
+
+# ─── Cross-workspace file_ids ─────────────────────────────────────────
+
+
+def test_file_ids_scoped_to_workspace(mocker: pytest_mock.MockFixture):
+    """The batch query should scope to the user's workspace."""
+    _mock_stream_internals(mocker)
+    mocker.patch(
+        "backend.api.features.chat.routes.get_or_create_workspace",
+        return_value=type("W", (), {"id": "my-workspace-id"})(),
+    )
+
+    mock_prisma = mocker.MagicMock()
+    mock_prisma.find_many = mocker.AsyncMock(return_value=[])
+    mocker.patch(
+        "prisma.models.UserWorkspaceFile.prisma",
+        return_value=mock_prisma,
+    )
+
+    fid = "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"
+    client.post(
+        "/sessions/sess-1/stream",
+        json={"message": "hi", "file_ids": [fid]},
+    )
+
+    call_kwargs = mock_prisma.find_many.call_args[1]
+    assert call_kwargs["where"]["workspaceId"] == "my-workspace-id"
+    assert call_kwargs["where"]["isDeleted"] is False
+
+
+# ─── Suggested prompts endpoint ──────────────────────────────────────
+
+
+def _mock_get_business_understanding(
+    mocker: pytest_mock.MockerFixture,
+    *,
+    return_value=None,
+):
+    """Mock get_business_understanding."""
+    return mocker.patch(
+        "backend.api.features.chat.routes.get_business_understanding",
+        new_callable=AsyncMock,
+        return_value=return_value,
+    )
+
+
+def test_suggested_prompts_returns_prompts(
+    mocker: pytest_mock.MockerFixture,
+    test_user_id: str,
+) -> None:
+    """User with understanding and prompts gets them back."""
+    mock_understanding = MagicMock()
+    mock_understanding.suggested_prompts = ["Do X", "Do Y", "Do Z"]
+    _mock_get_business_understanding(mocker, return_value=mock_understanding)
+
+    response = client.get("/suggested-prompts")
+
+    assert response.status_code == 200
+    assert response.json() == {"prompts": ["Do X", "Do Y", "Do Z"]}
+
+
+def test_suggested_prompts_no_understanding(
+    mocker: pytest_mock.MockerFixture,
+    test_user_id: str,
+) -> None:
+    """User with no understanding gets empty list."""
+    _mock_get_business_understanding(mocker, return_value=None)
+
+    response = client.get("/suggested-prompts")
+
+    assert response.status_code == 200
+    assert response.json() == {"prompts": []}
+
+
+def test_suggested_prompts_empty_prompts(
+    mocker: pytest_mock.MockerFixture,
+    test_user_id: str,
+) -> None:
+    """User with understanding but no prompts gets empty list."""
+    mock_understanding = MagicMock()
+    mock_understanding.suggested_prompts = []
+    _mock_get_business_understanding(mocker, return_value=mock_understanding)
+
+    response = client.get("/suggested-prompts")
+
+    assert response.status_code == 200
+    assert response.json() == {"prompts": []}

autogpt_platform/backend/backend/api/features/chat/sdk/__init__.py

@@ -1,14 +0,0 @@
-"""Claude Agent SDK integration for CoPilot.
-
-This module provides the integration layer between the Claude Agent SDK
-and the existing CoPilot tool system, enabling drop-in replacement of
-the current LLM orchestration with the battle-tested Claude Agent SDK.
-"""
-
-from .service import stream_chat_completion_sdk
-from .tool_adapter import create_copilot_mcp_server
-
-__all__ = [
-    "stream_chat_completion_sdk",
-    "create_copilot_mcp_server",
-]

autogpt_platform/backend/backend/api/features/chat/sdk/response_adapter.py

@@ -1,203 +0,0 @@
-"""Response adapter for converting Claude Agent SDK messages to Vercel AI SDK format.
-
-This module provides the adapter layer that converts streaming messages from
-the Claude Agent SDK into the Vercel AI SDK UI Stream Protocol format that
-the frontend expects.
-"""
-
-import json
-import logging
-import uuid
-
-from claude_agent_sdk import (
-    AssistantMessage,
-    Message,
-    ResultMessage,
-    SystemMessage,
-    TextBlock,
-    ToolResultBlock,
-    ToolUseBlock,
-    UserMessage,
-)
-
-from backend.api.features.chat.response_model import (
-    StreamBaseResponse,
-    StreamError,
-    StreamFinish,
-    StreamFinishStep,
-    StreamStart,
-    StreamStartStep,
-    StreamTextDelta,
-    StreamTextEnd,
-    StreamTextStart,
-    StreamToolInputAvailable,
-    StreamToolInputStart,
-    StreamToolOutputAvailable,
-)
-from backend.api.features.chat.sdk.tool_adapter import (
-    MCP_TOOL_PREFIX,
-    pop_pending_tool_output,
-)
-
-logger = logging.getLogger(__name__)
-
-
-class SDKResponseAdapter:
-    """Adapter for converting Claude Agent SDK messages to Vercel AI SDK format.
-
-    This class maintains state during a streaming session to properly track
-    text blocks, tool calls, and message lifecycle.
-    """
-
-    def __init__(self, message_id: str | None = None):
-        self.message_id = message_id or str(uuid.uuid4())
-        self.text_block_id = str(uuid.uuid4())
-        self.has_started_text = False
-        self.has_ended_text = False
-        self.current_tool_calls: dict[str, dict[str, str]] = {}
-        self.task_id: str | None = None
-        self.step_open = False
-
-    def set_task_id(self, task_id: str) -> None:
-        """Set the task ID for reconnection support."""
-        self.task_id = task_id
-
-    def convert_message(self, sdk_message: Message) -> list[StreamBaseResponse]:
-        """Convert a single SDK message to Vercel AI SDK format."""
-        responses: list[StreamBaseResponse] = []
-
-        if isinstance(sdk_message, SystemMessage):
-            if sdk_message.subtype == "init":
-                responses.append(
-                    StreamStart(messageId=self.message_id, taskId=self.task_id)
-                )
-                # Open the first step (matches non-SDK: StreamStart then StreamStartStep)
-                responses.append(StreamStartStep())
-                self.step_open = True
-
-        elif isinstance(sdk_message, AssistantMessage):
-            # After tool results, the SDK sends a new AssistantMessage for the
-            # next LLM turn. Open a new step if the previous one was closed.
-            if not self.step_open:
-                responses.append(StreamStartStep())
-                self.step_open = True
-
-            for block in sdk_message.content:
-                if isinstance(block, TextBlock):
-                    if block.text:
-                        self._ensure_text_started(responses)
-                        responses.append(
-                            StreamTextDelta(id=self.text_block_id, delta=block.text)
-                        )
-
-                elif isinstance(block, ToolUseBlock):
-                    self._end_text_if_open(responses)
-
-                    # Strip MCP prefix so frontend sees "find_block"
-                    # instead of "mcp__copilot__find_block".
-                    tool_name = block.name.removeprefix(MCP_TOOL_PREFIX)
-
-                    responses.append(
-                        StreamToolInputStart(toolCallId=block.id, toolName=tool_name)
-                    )
-                    responses.append(
-                        StreamToolInputAvailable(
-                            toolCallId=block.id,
-                            toolName=tool_name,
-                            input=block.input,
-                        )
-                    )
-                    self.current_tool_calls[block.id] = {"name": tool_name}
-
-        elif isinstance(sdk_message, UserMessage):
-            # UserMessage carries tool results back from tool execution.
-            content = sdk_message.content
-            blocks = content if isinstance(content, list) else []
-            for block in blocks:
-                if isinstance(block, ToolResultBlock) and block.tool_use_id:
-                    tool_info = self.current_tool_calls.get(block.tool_use_id, {})
-                    tool_name = tool_info.get("name", "unknown")
-
-                    # Prefer the stashed full output over the SDK's
-                    # (potentially truncated) ToolResultBlock content.
-                    # The SDK truncates large results, writing them to disk,
-                    # which breaks frontend widget parsing.
-                    output = pop_pending_tool_output(tool_name) or (
-                        _extract_tool_output(block.content)
-                    )
-
-                    responses.append(
-                        StreamToolOutputAvailable(
-                            toolCallId=block.tool_use_id,
-                            toolName=tool_name,
-                            output=output,
-                            success=not (block.is_error or False),
-                        )
-                    )
-
-            # Close the current step after tool results — the next
-            # AssistantMessage will open a new step for the continuation.
-            if self.step_open:
-                responses.append(StreamFinishStep())
-                self.step_open = False
-
-        elif isinstance(sdk_message, ResultMessage):
-            self._end_text_if_open(responses)
-            # Close the step before finishing.
-            if self.step_open:
-                responses.append(StreamFinishStep())
-                self.step_open = False
-
-            if sdk_message.subtype == "success":
-                responses.append(StreamFinish())
-            elif sdk_message.subtype in ("error", "error_during_execution"):
-                error_msg = getattr(sdk_message, "result", None) or "Unknown error"
-                responses.append(
-                    StreamError(errorText=str(error_msg), code="sdk_error")
-                )
-                responses.append(StreamFinish())
-            else:
-                logger.warning(
-                    f"Unexpected ResultMessage subtype: {sdk_message.subtype}"
-                )
-                responses.append(StreamFinish())
-
-        else:
-            logger.debug(f"Unhandled SDK message type: {type(sdk_message).__name__}")
-
-        return responses
-
-    def _ensure_text_started(self, responses: list[StreamBaseResponse]) -> None:
-        """Start (or restart) a text block if needed."""
-        if not self.has_started_text or self.has_ended_text:
-            if self.has_ended_text:
-                self.text_block_id = str(uuid.uuid4())
-                self.has_ended_text = False
-            responses.append(StreamTextStart(id=self.text_block_id))
-            self.has_started_text = True
-
-    def _end_text_if_open(self, responses: list[StreamBaseResponse]) -> None:
-        """End the current text block if one is open."""
-        if self.has_started_text and not self.has_ended_text:
-            responses.append(StreamTextEnd(id=self.text_block_id))
-            self.has_ended_text = True
-
-
-def _extract_tool_output(content: str | list[dict[str, str]] | None) -> str:
-    """Extract a string output from a ToolResultBlock's content field."""
-    if isinstance(content, str):
-        return content
-    if isinstance(content, list):
-        parts = [item.get("text", "") for item in content if item.get("type") == "text"]
-        if parts:
-            return "".join(parts)
-        try:
-            return json.dumps(content)
-        except (TypeError, ValueError):
-            return str(content)
-    if content is None:
-        return ""
-    try:
-        return json.dumps(content)
-    except (TypeError, ValueError):
-        return str(content)

autogpt_platform/backend/backend/api/features/chat/sdk/security_hooks_test.py

@@ -1,165 +0,0 @@
-"""Unit tests for SDK security hooks."""
-
-import os
-
-from .security_hooks import _validate_tool_access, _validate_user_isolation
-
-SDK_CWD = "/tmp/copilot-abc123"
-
-
-def _is_denied(result: dict) -> bool:
-    hook = result.get("hookSpecificOutput", {})
-    return hook.get("permissionDecision") == "deny"
-
-
-# -- Blocked tools -----------------------------------------------------------
-
-
-def test_blocked_tools_denied():
-    for tool in ("bash", "shell", "exec", "terminal", "command"):
-        result = _validate_tool_access(tool, {})
-        assert _is_denied(result), f"{tool} should be blocked"
-
-
-def test_unknown_tool_allowed():
-    result = _validate_tool_access("SomeCustomTool", {})
-    assert result == {}
-
-
-# -- Workspace-scoped tools --------------------------------------------------
-
-
-def test_read_within_workspace_allowed():
-    result = _validate_tool_access(
-        "Read", {"file_path": f"{SDK_CWD}/file.txt"}, sdk_cwd=SDK_CWD
-    )
-    assert result == {}
-
-
-def test_write_within_workspace_allowed():
-    result = _validate_tool_access(
-        "Write", {"file_path": f"{SDK_CWD}/output.json"}, sdk_cwd=SDK_CWD
-    )
-    assert result == {}
-
-
-def test_edit_within_workspace_allowed():
-    result = _validate_tool_access(
-        "Edit", {"file_path": f"{SDK_CWD}/src/main.py"}, sdk_cwd=SDK_CWD
-    )
-    assert result == {}
-
-
-def test_glob_within_workspace_allowed():
-    result = _validate_tool_access("Glob", {"path": f"{SDK_CWD}/src"}, sdk_cwd=SDK_CWD)
-    assert result == {}
-
-
-def test_grep_within_workspace_allowed():
-    result = _validate_tool_access("Grep", {"path": f"{SDK_CWD}/src"}, sdk_cwd=SDK_CWD)
-    assert result == {}
-
-
-def test_read_outside_workspace_denied():
-    result = _validate_tool_access(
-        "Read", {"file_path": "/etc/passwd"}, sdk_cwd=SDK_CWD
-    )
-    assert _is_denied(result)
-
-
-def test_write_outside_workspace_denied():
-    result = _validate_tool_access(
-        "Write", {"file_path": "/home/user/secrets.txt"}, sdk_cwd=SDK_CWD
-    )
-    assert _is_denied(result)
-
-
-def test_traversal_attack_denied():
-    result = _validate_tool_access(
-        "Read",
-        {"file_path": f"{SDK_CWD}/../../etc/passwd"},
-        sdk_cwd=SDK_CWD,
-    )
-    assert _is_denied(result)
-
-
-def test_no_path_allowed():
-    """Glob/Grep without a path argument defaults to cwd — should pass."""
-    result = _validate_tool_access("Glob", {}, sdk_cwd=SDK_CWD)
-    assert result == {}
-
-
-def test_read_no_cwd_denies_absolute():
-    """If no sdk_cwd is set, absolute paths are denied."""
-    result = _validate_tool_access("Read", {"file_path": "/tmp/anything"})
-    assert _is_denied(result)
-
-
-# -- Tool-results directory --------------------------------------------------
-
-
-def test_read_tool_results_allowed():
-    home = os.path.expanduser("~")
-    path = f"{home}/.claude/projects/-tmp-copilot-abc123/tool-results/12345.txt"
-    result = _validate_tool_access("Read", {"file_path": path}, sdk_cwd=SDK_CWD)
-    assert result == {}
-
-
-def test_read_claude_projects_without_tool_results_denied():
-    home = os.path.expanduser("~")
-    path = f"{home}/.claude/projects/-tmp-copilot-abc123/settings.json"
-    result = _validate_tool_access("Read", {"file_path": path}, sdk_cwd=SDK_CWD)
-    assert _is_denied(result)
-
-
-# -- Built-in Bash is blocked (use bash_exec MCP tool instead) ---------------
-
-
-def test_bash_builtin_always_blocked():
-    """SDK built-in Bash is blocked — bash_exec MCP tool with bubblewrap is used instead."""
-    result = _validate_tool_access("Bash", {"command": "echo hello"}, sdk_cwd=SDK_CWD)
-    assert _is_denied(result)
-
-
-# -- Dangerous patterns ------------------------------------------------------
-
-
-def test_dangerous_pattern_blocked():
-    result = _validate_tool_access("SomeTool", {"cmd": "sudo rm -rf /"})
-    assert _is_denied(result)
-
-
-def test_subprocess_pattern_blocked():
-    result = _validate_tool_access("SomeTool", {"code": "subprocess.run(...)"})
-    assert _is_denied(result)
-
-
-# -- User isolation ----------------------------------------------------------
-
-
-def test_workspace_path_traversal_blocked():
-    result = _validate_user_isolation(
-        "workspace_read", {"path": "../../../etc/shadow"}, user_id="user-1"
-    )
-    assert _is_denied(result)
-
-
-def test_workspace_absolute_path_blocked():
-    result = _validate_user_isolation(
-        "workspace_read", {"path": "/etc/passwd"}, user_id="user-1"
-    )
-    assert _is_denied(result)
-
-
-def test_workspace_normal_path_allowed():
-    result = _validate_user_isolation(
-        "workspace_read", {"path": "src/main.py"}, user_id="user-1"
-    )
-    assert result == {}
-
-
-def test_non_workspace_tool_passes_isolation():
-    result = _validate_user_isolation(
-        "find_agent", {"query": "email"}, user_id="user-1"
-    )
-    assert result == {}

autogpt_platform/backend/backend/api/features/chat/sdk/service.py

@@ -1,752 +0,0 @@
-"""Claude Agent SDK service layer for CoPilot chat completions."""
-
-import asyncio
-import json
-import logging
-import os
-import uuid
-from collections.abc import AsyncGenerator
-from dataclasses import dataclass
-from typing import Any
-
-from backend.util.exceptions import NotFoundError
-
-from .. import stream_registry
-from ..config import ChatConfig
-from ..model import (
-    ChatMessage,
-    ChatSession,
-    get_chat_session,
-    update_session_title,
-    upsert_chat_session,
-)
-from ..response_model import (
-    StreamBaseResponse,
-    StreamError,
-    StreamFinish,
-    StreamStart,
-    StreamTextDelta,
-    StreamToolInputAvailable,
-    StreamToolOutputAvailable,
-)
-from ..service import (
-    _build_system_prompt,
-    _execute_long_running_tool_with_streaming,
-    _generate_session_title,
-)
-from ..tools.models import OperationPendingResponse, OperationStartedResponse
-from ..tools.sandbox import WORKSPACE_PREFIX, make_session_path
-from ..tracking import track_user_message
-from .response_adapter import SDKResponseAdapter
-from .security_hooks import create_security_hooks
-from .tool_adapter import (
-    COPILOT_TOOL_NAMES,
-    SDK_DISALLOWED_TOOLS,
-    LongRunningCallback,
-    create_copilot_mcp_server,
-    set_execution_context,
-)
-from .transcript import (
-    download_transcript,
-    read_transcript_file,
-    upload_transcript,
-    validate_transcript,
-    write_transcript_to_tempfile,
-)
-
-logger = logging.getLogger(__name__)
-config = ChatConfig()
-
-# Set to hold background tasks to prevent garbage collection
-_background_tasks: set[asyncio.Task[Any]] = set()
-
-
-@dataclass
-class CapturedTranscript:
-    """Info captured by the SDK Stop hook for stateless --resume."""
-
-    path: str = ""
-    sdk_session_id: str = ""
-
-    @property
-    def available(self) -> bool:
-        return bool(self.path)
-
-
-_SDK_CWD_PREFIX = WORKSPACE_PREFIX
-
-# Appended to the system prompt to inform the agent about available tools.
-# The SDK built-in Bash is NOT available — use mcp__copilot__bash_exec instead,
-# which has kernel-level network isolation (unshare --net).
-_SDK_TOOL_SUPPLEMENT = """
-
-## Tool notes
-
-- The SDK built-in Bash tool is NOT available.  Use the `bash_exec` MCP tool
-  for shell commands — it runs in a network-isolated sandbox.
-- **Shared workspace**: The SDK Read/Write tools and `bash_exec` share the
-  same working directory. Files created by one are readable by the other.
-  These files are **ephemeral** — they exist only for the current session.
-- **Persistent storage**: Use `write_workspace_file` / `read_workspace_file`
-  for files that should persist across sessions (stored in cloud storage).
-- Long-running tools (create_agent, edit_agent, etc.) are handled
-  asynchronously.  You will receive an immediate response; the actual result
-  is delivered to the user via a background stream.
-"""
-
-
-def _build_long_running_callback(user_id: str | None) -> LongRunningCallback:
-    """Build a callback that delegates long-running tools to the non-SDK infrastructure.
-
-    Long-running tools (create_agent, edit_agent, etc.) are delegated to the
-    existing background infrastructure: stream_registry (Redis Streams),
-    database persistence, and SSE reconnection.  This means results survive
-    page refreshes / pod restarts, and the frontend shows the proper loading
-    widget with progress updates.
-
-    The returned callback matches the ``LongRunningCallback`` signature:
-    ``(tool_name, args, session) -> MCP response dict``.
-    """
-
-    async def _callback(
-        tool_name: str, args: dict[str, Any], session: ChatSession
-    ) -> dict[str, Any]:
-        operation_id = str(uuid.uuid4())
-        task_id = str(uuid.uuid4())
-        tool_call_id = f"sdk-{uuid.uuid4().hex[:12]}"
-        session_id = session.session_id
-
-        # --- Build user-friendly messages (matches non-SDK service) ---
-        if tool_name == "create_agent":
-            desc = args.get("description", "")
-            desc_preview = (desc[:100] + "...") if len(desc) > 100 else desc
-            pending_msg = (
-                f"Creating your agent: {desc_preview}"
-                if desc_preview
-                else "Creating agent... This may take a few minutes."
-            )
-            started_msg = (
-                "Agent creation started. You can close this tab - "
-                "check your library in a few minutes."
-            )
-        elif tool_name == "edit_agent":
-            changes = args.get("changes", "")
-            changes_preview = (changes[:100] + "...") if len(changes) > 100 else changes
-            pending_msg = (
-                f"Editing agent: {changes_preview}"
-                if changes_preview
-                else "Editing agent... This may take a few minutes."
-            )
-            started_msg = (
-                "Agent edit started. You can close this tab - "
-                "check your library in a few minutes."
-            )
-        else:
-            pending_msg = f"Running {tool_name}... This may take a few minutes."
-            started_msg = (
-                f"{tool_name} started. You can close this tab - "
-                "check back in a few minutes."
-            )
-
-        # --- Register task in Redis for SSE reconnection ---
-        await stream_registry.create_task(
-            task_id=task_id,
-            session_id=session_id,
-            user_id=user_id,
-            tool_call_id=tool_call_id,
-            tool_name=tool_name,
-            operation_id=operation_id,
-        )
-
-        # --- Save OperationPendingResponse to chat history ---
-        pending_message = ChatMessage(
-            role="tool",
-            content=OperationPendingResponse(
-                message=pending_msg,
-                operation_id=operation_id,
-                tool_name=tool_name,
-            ).model_dump_json(),
-            tool_call_id=tool_call_id,
-        )
-        session.messages.append(pending_message)
-        await upsert_chat_session(session)
-
-        # --- Spawn background task (reuses non-SDK infrastructure) ---
-        bg_task = asyncio.create_task(
-            _execute_long_running_tool_with_streaming(
-                tool_name=tool_name,
-                parameters=args,
-                tool_call_id=tool_call_id,
-                operation_id=operation_id,
-                task_id=task_id,
-                session_id=session_id,
-                user_id=user_id,
-            )
-        )
-        _background_tasks.add(bg_task)
-        bg_task.add_done_callback(_background_tasks.discard)
-        await stream_registry.set_task_asyncio_task(task_id, bg_task)
-
-        logger.info(
-            f"[SDK] Long-running tool {tool_name} delegated to background "
-            f"(operation_id={operation_id}, task_id={task_id})"
-        )
-
-        # --- Return OperationStartedResponse as MCP tool result ---
-        # This flows through SDK → response adapter → frontend, triggering
-        # the loading widget with SSE reconnection support.
-        started_json = OperationStartedResponse(
-            message=started_msg,
-            operation_id=operation_id,
-            tool_name=tool_name,
-            task_id=task_id,
-        ).model_dump_json()
-
-        return {
-            "content": [{"type": "text", "text": started_json}],
-            "isError": False,
-        }
-
-    return _callback
-
-
-def _resolve_sdk_model() -> str | None:
-    """Resolve the model name for the Claude Agent SDK CLI.
-
-    Uses ``config.claude_agent_model`` if set, otherwise derives from
-    ``config.model`` by stripping the OpenRouter provider prefix (e.g.,
-    ``"anthropic/claude-opus-4.6"`` → ``"claude-opus-4.6"``).
-    """
-    if config.claude_agent_model:
-        return config.claude_agent_model
-    model = config.model
-    if "/" in model:
-        return model.split("/", 1)[1]
-    return model
-
-
-def _build_sdk_env() -> dict[str, str]:
-    """Build env vars for the SDK CLI process.
-
-    Routes API calls through OpenRouter (or a custom base_url) using
-    the same ``config.api_key`` / ``config.base_url`` as the non-SDK path.
-    This gives per-call token and cost tracking on the OpenRouter dashboard.
-
-    Only overrides ``ANTHROPIC_API_KEY`` when a valid proxy URL and auth
-    token are both present — otherwise returns an empty dict so the SDK
-    falls back to its default credentials.
-    """
-    env: dict[str, str] = {}
-    if config.api_key and config.base_url:
-        # Strip /v1 suffix — SDK expects the base URL without a version path
-        base = config.base_url.rstrip("/")
-        if base.endswith("/v1"):
-            base = base[:-3]
-        if not base or not base.startswith("http"):
-            # Invalid base_url — don't override SDK defaults
-            return env
-        env["ANTHROPIC_BASE_URL"] = base
-        env["ANTHROPIC_AUTH_TOKEN"] = config.api_key
-        # Must be explicitly empty so the CLI uses AUTH_TOKEN instead
-        env["ANTHROPIC_API_KEY"] = ""
-    return env
-
-
-def _make_sdk_cwd(session_id: str) -> str:
-    """Create a safe, session-specific working directory path.
-
-    Delegates to :func:`~backend.api.features.chat.tools.sandbox.make_session_path`
-    (single source of truth for path sanitization) and adds a defence-in-depth
-    assertion.
-    """
-    cwd = make_session_path(session_id)
-    # Defence-in-depth: normpath + startswith is a CodeQL-recognised sanitizer
-    cwd = os.path.normpath(cwd)
-    if not cwd.startswith(_SDK_CWD_PREFIX):
-        raise ValueError(f"SDK cwd escaped prefix: {cwd}")
-    return cwd
-
-
-def _cleanup_sdk_tool_results(cwd: str) -> None:
-    """Remove SDK tool-result files for a specific session working directory.
-
-    The SDK creates tool-result files under ~/.claude/projects/<encoded-cwd>/tool-results/.
-    We clean only the specific cwd's results to avoid race conditions between
-    concurrent sessions.
-
-    Security: cwd MUST be created by _make_sdk_cwd() which sanitizes session_id.
-    """
-    import shutil
-
-    # Validate cwd is under the expected prefix
-    normalized = os.path.normpath(cwd)
-    if not normalized.startswith(_SDK_CWD_PREFIX):
-        logger.warning(f"[SDK] Rejecting cleanup for path outside workspace: {cwd}")
-        return
-
-    # SDK encodes the cwd path by replacing '/' with '-'
-    encoded_cwd = normalized.replace("/", "-")
-
-    # Construct the project directory path (known-safe home expansion)
-    claude_projects = os.path.expanduser("~/.claude/projects")
-    project_dir = os.path.join(claude_projects, encoded_cwd)
-
-    # Security check 3: Validate project_dir is under ~/.claude/projects
-    project_dir = os.path.normpath(project_dir)
-    if not project_dir.startswith(claude_projects):
-        logger.warning(
-            f"[SDK] Rejecting cleanup for escaped project path: {project_dir}"
-        )
-        return
-
-    results_dir = os.path.join(project_dir, "tool-results")
-    if os.path.isdir(results_dir):
-        for filename in os.listdir(results_dir):
-            file_path = os.path.join(results_dir, filename)
-            try:
-                if os.path.isfile(file_path):
-                    os.remove(file_path)
-            except OSError:
-                pass
-
-    # Also clean up the temp cwd directory itself
-    try:
-        shutil.rmtree(normalized, ignore_errors=True)
-    except OSError:
-        pass
-
-
-async def _compress_conversation_history(
-    session: ChatSession,
-) -> list[ChatMessage]:
-    """Compress prior conversation messages if they exceed the token threshold.
-
-    Uses the shared compress_context() from prompt.py which supports:
-    - LLM summarization of old messages (keeps recent ones intact)
-    - Progressive content truncation as fallback
-    - Middle-out deletion as last resort
-
-    Returns the compressed prior messages (everything except the current message).
-    """
-    prior = session.messages[:-1]
-    if len(prior) < 2:
-        return prior
-
-    from backend.util.prompt import compress_context
-
-    # Convert ChatMessages to dicts for compress_context
-    messages_dict = []
-    for msg in prior:
-        msg_dict: dict[str, Any] = {"role": msg.role}
-        if msg.content:
-            msg_dict["content"] = msg.content
-        if msg.tool_calls:
-            msg_dict["tool_calls"] = msg.tool_calls
-        if msg.tool_call_id:
-            msg_dict["tool_call_id"] = msg.tool_call_id
-        messages_dict.append(msg_dict)
-
-    try:
-        import openai
-
-        async with openai.AsyncOpenAI(
-            api_key=config.api_key, base_url=config.base_url, timeout=30.0
-        ) as client:
-            result = await compress_context(
-                messages=messages_dict,
-                model=config.model,
-                client=client,
-            )
-    except Exception as e:
-        logger.warning(f"[SDK] Context compression with LLM failed: {e}")
-        # Fall back to truncation-only (no LLM summarization)
-        result = await compress_context(
-            messages=messages_dict,
-            model=config.model,
-            client=None,
-        )
-
-    if result.was_compacted:
-        logger.info(
-            f"[SDK] Context compacted: {result.original_token_count} -> "
-            f"{result.token_count} tokens "
-            f"({result.messages_summarized} summarized, "
-            f"{result.messages_dropped} dropped)"
-        )
-        # Convert compressed dicts back to ChatMessages
-        return [
-            ChatMessage(
-                role=m["role"],
-                content=m.get("content"),
-                tool_calls=m.get("tool_calls"),
-                tool_call_id=m.get("tool_call_id"),
-            )
-            for m in result.messages
-        ]
-
-    return prior
-
-
-def _format_conversation_context(messages: list[ChatMessage]) -> str | None:
-    """Format conversation messages into a context prefix for the user message.
-
-    Returns a string like:
-        <conversation_history>
-        User: hello
-        You responded: Hi! How can I help?
-        </conversation_history>
-
-    Returns None if there are no messages to format.
-    """
-    if not messages:
-        return None
-
-    lines: list[str] = []
-    for msg in messages:
-        if not msg.content:
-            continue
-        if msg.role == "user":
-            lines.append(f"User: {msg.content}")
-        elif msg.role == "assistant":
-            lines.append(f"You responded: {msg.content}")
-        # Skip tool messages — they're internal details
-
-    if not lines:
-        return None
-
-    return "<conversation_history>\n" + "\n".join(lines) + "\n</conversation_history>"
-
-
-async def stream_chat_completion_sdk(
-    session_id: str,
-    message: str | None = None,
-    tool_call_response: str | None = None,  # noqa: ARG001
-    is_user_message: bool = True,
-    user_id: str | None = None,
-    retry_count: int = 0,  # noqa: ARG001
-    session: ChatSession | None = None,
-    context: dict[str, str] | None = None,  # noqa: ARG001
-) -> AsyncGenerator[StreamBaseResponse, None]:
-    """Stream chat completion using Claude Agent SDK.
-
-    Drop-in replacement for stream_chat_completion with improved reliability.
-    """
-
-    if session is None:
-        session = await get_chat_session(session_id, user_id)
-
-    if not session:
-        raise NotFoundError(
-            f"Session {session_id} not found. Please create a new session first."
-        )
-
-    if message:
-        session.messages.append(
-            ChatMessage(
-                role="user" if is_user_message else "assistant", content=message
-            )
-        )
-        if is_user_message:
-            track_user_message(
-                user_id=user_id, session_id=session_id, message_length=len(message)
-            )
-
-    session = await upsert_chat_session(session)
-
-    # Generate title for new sessions (first user message)
-    if is_user_message and not session.title:
-        user_messages = [m for m in session.messages if m.role == "user"]
-        if len(user_messages) == 1:
-            first_message = user_messages[0].content or message or ""
-            if first_message:
-                task = asyncio.create_task(
-                    _update_title_async(session_id, first_message, user_id)
-                )
-                _background_tasks.add(task)
-                task.add_done_callback(_background_tasks.discard)
-
-    # Build system prompt (reuses non-SDK path with Langfuse support)
-    has_history = len(session.messages) > 1
-    system_prompt, _ = await _build_system_prompt(
-        user_id, has_conversation_history=has_history
-    )
-    system_prompt += _SDK_TOOL_SUPPLEMENT
-    message_id = str(uuid.uuid4())
-    task_id = str(uuid.uuid4())
-
-    yield StreamStart(messageId=message_id, taskId=task_id)
-
-    stream_completed = False
-    # Initialise sdk_cwd before the try so the finally can reference it
-    # even if _make_sdk_cwd raises (in that case it stays as "").
-    sdk_cwd = ""
-    use_resume = False
-
-    try:
-        # Use a session-specific temp dir to avoid cleanup race conditions
-        # between concurrent sessions.
-        sdk_cwd = _make_sdk_cwd(session_id)
-        os.makedirs(sdk_cwd, exist_ok=True)
-
-        set_execution_context(
-            user_id,
-            session,
-            long_running_callback=_build_long_running_callback(user_id),
-        )
-        try:
-            from claude_agent_sdk import ClaudeAgentOptions, ClaudeSDKClient
-
-            # Fail fast when no API credentials are available at all
-            sdk_env = _build_sdk_env()
-            if not sdk_env and not os.environ.get("ANTHROPIC_API_KEY"):
-                raise RuntimeError(
-                    "No API key configured. Set OPEN_ROUTER_API_KEY "
-                    "(or CHAT_API_KEY) for OpenRouter routing, "
-                    "or ANTHROPIC_API_KEY for direct Anthropic access."
-                )
-
-            mcp_server = create_copilot_mcp_server()
-
-            sdk_model = _resolve_sdk_model()
-
-            # --- Transcript capture via Stop hook ---
-            captured_transcript = CapturedTranscript()
-
-            def _on_stop(transcript_path: str, sdk_session_id: str) -> None:
-                captured_transcript.path = transcript_path
-                captured_transcript.sdk_session_id = sdk_session_id
-
-            security_hooks = create_security_hooks(
-                user_id,
-                sdk_cwd=sdk_cwd,
-                max_subtasks=config.claude_agent_max_subtasks,
-                on_stop=_on_stop if config.claude_agent_use_resume else None,
-            )
-
-            # --- Resume strategy: download transcript from bucket ---
-            resume_file: str | None = None
-            use_resume = False
-
-            if config.claude_agent_use_resume and user_id and len(session.messages) > 1:
-                transcript_content = await download_transcript(user_id, session_id)
-                if transcript_content and validate_transcript(transcript_content):
-                    resume_file = write_transcript_to_tempfile(
-                        transcript_content, session_id, sdk_cwd
-                    )
-                    if resume_file:
-                        use_resume = True
-                        logger.info(
-                            f"[SDK] Using --resume with transcript "
-                            f"({len(transcript_content)} bytes)"
-                        )
-
-            sdk_options_kwargs: dict[str, Any] = {
-                "system_prompt": system_prompt,
-                "mcp_servers": {"copilot": mcp_server},
-                "allowed_tools": COPILOT_TOOL_NAMES,
-                "disallowed_tools": SDK_DISALLOWED_TOOLS,
-                "hooks": security_hooks,
-                "cwd": sdk_cwd,
-                "max_buffer_size": config.claude_agent_max_buffer_size,
-            }
-            if sdk_env:
-                sdk_options_kwargs["model"] = sdk_model
-                sdk_options_kwargs["env"] = sdk_env
-            if use_resume and resume_file:
-                sdk_options_kwargs["resume"] = resume_file
-
-            options = ClaudeAgentOptions(**sdk_options_kwargs)  # type: ignore[arg-type]
-
-            adapter = SDKResponseAdapter(message_id=message_id)
-            adapter.set_task_id(task_id)
-
-            async with ClaudeSDKClient(options=options) as client:
-                current_message = message or ""
-                if not current_message and session.messages:
-                    last_user = [m for m in session.messages if m.role == "user"]
-                    if last_user:
-                        current_message = last_user[-1].content or ""
-
-                if not current_message.strip():
-                    yield StreamError(
-                        errorText="Message cannot be empty.",
-                        code="empty_prompt",
-                    )
-                    yield StreamFinish()
-                    return
-
-                # Build query: with --resume the CLI already has full
-                # context, so we only send the new message.  Without
-                # resume, compress history into a context prefix.
-                query_message = current_message
-                if not use_resume and len(session.messages) > 1:
-                    logger.warning(
-                        f"[SDK] Using compression fallback for session "
-                        f"{session_id} ({len(session.messages)} messages) — "
-                        f"no transcript available for --resume"
-                    )
-                    compressed = await _compress_conversation_history(session)
-                    history_context = _format_conversation_context(compressed)
-                    if history_context:
-                        query_message = (
-                            f"{history_context}\n\n"
-                            f"Now, the user says:\n{current_message}"
-                        )
-
-                logger.info(
-                    f"[SDK] Sending query ({len(session.messages)} msgs in session)"
-                )
-                logger.debug(f"[SDK] Query preview: {current_message[:80]!r}")
-                await client.query(query_message, session_id=session_id)
-
-                assistant_response = ChatMessage(role="assistant", content="")
-                accumulated_tool_calls: list[dict[str, Any]] = []
-                has_appended_assistant = False
-                has_tool_results = False
-
-                async for sdk_msg in client.receive_messages():
-                    logger.debug(
-                        f"[SDK] Received: {type(sdk_msg).__name__} "
-                        f"{getattr(sdk_msg, 'subtype', '')}"
-                    )
-                    for response in adapter.convert_message(sdk_msg):
-                        if isinstance(response, StreamStart):
-                            continue
-
-                        yield response
-
-                        if isinstance(response, StreamTextDelta):
-                            delta = response.delta or ""
-                            # After tool results, start a new assistant
-                            # message for the post-tool text.
-                            if has_tool_results and has_appended_assistant:
-                                assistant_response = ChatMessage(
-                                    role="assistant", content=delta
-                                )
-                                accumulated_tool_calls = []
-                                has_appended_assistant = False
-                                has_tool_results = False
-                                session.messages.append(assistant_response)
-                                has_appended_assistant = True
-                            else:
-                                assistant_response.content = (
-                                    assistant_response.content or ""
-                                ) + delta
-                                if not has_appended_assistant:
-                                    session.messages.append(assistant_response)
-                                    has_appended_assistant = True
-
-                        elif isinstance(response, StreamToolInputAvailable):
-                            accumulated_tool_calls.append(
-                                {
-                                    "id": response.toolCallId,
-                                    "type": "function",
-                                    "function": {
-                                        "name": response.toolName,
-                                        "arguments": json.dumps(response.input or {}),
-                                    },
-                                }
-                            )
-                            assistant_response.tool_calls = accumulated_tool_calls
-                            if not has_appended_assistant:
-                                session.messages.append(assistant_response)
-                                has_appended_assistant = True
-
-                        elif isinstance(response, StreamToolOutputAvailable):
-                            session.messages.append(
-                                ChatMessage(
-                                    role="tool",
-                                    content=(
-                                        response.output
-                                        if isinstance(response.output, str)
-                                        else str(response.output)
-                                    ),
-                                    tool_call_id=response.toolCallId,
-                                )
-                            )
-                            has_tool_results = True
-
-                        elif isinstance(response, StreamFinish):
-                            stream_completed = True
-
-                    if stream_completed:
-                        break
-
-                if (
-                    assistant_response.content or assistant_response.tool_calls
-                ) and not has_appended_assistant:
-                    session.messages.append(assistant_response)
-
-                # --- Capture transcript while CLI is still alive ---
-                # Must happen INSIDE async with: close() sends SIGTERM
-                # which kills the CLI before it can flush the JSONL.
-                if (
-                    config.claude_agent_use_resume
-                    and user_id
-                    and captured_transcript.available
-                ):
-                    # Give CLI time to flush JSONL writes before we read
-                    await asyncio.sleep(0.5)
-                    raw_transcript = read_transcript_file(captured_transcript.path)
-                    if raw_transcript:
-                        task = asyncio.create_task(
-                            _upload_transcript_bg(user_id, session_id, raw_transcript)
-                        )
-                        _background_tasks.add(task)
-                        task.add_done_callback(_background_tasks.discard)
-                    else:
-                        logger.debug("[SDK] Stop hook fired but transcript not usable")
-
-        except ImportError:
-            raise RuntimeError(
-                "claude-agent-sdk is not installed. "
-                "Disable SDK mode (CHAT_USE_CLAUDE_AGENT_SDK=false) "
-                "to use the OpenAI-compatible fallback."
-            )
-
-        await upsert_chat_session(session)
-        logger.debug(
-            f"[SDK] Session {session_id} saved with {len(session.messages)} messages"
-        )
-        if not stream_completed:
-            yield StreamFinish()
-
-    except Exception as e:
-        logger.error(f"[SDK] Error: {e}", exc_info=True)
-        try:
-            await upsert_chat_session(session)
-        except Exception as save_err:
-            logger.error(f"[SDK] Failed to save session on error: {save_err}")
-        yield StreamError(
-            errorText="An error occurred. Please try again.",
-            code="sdk_error",
-        )
-        yield StreamFinish()
-    finally:
-        if sdk_cwd:
-            _cleanup_sdk_tool_results(sdk_cwd)
-
-
-async def _upload_transcript_bg(
-    user_id: str, session_id: str, raw_content: str
-) -> None:
-    """Background task to strip progress entries and upload transcript."""
-    try:
-        await upload_transcript(user_id, session_id, raw_content)
-    except Exception as e:
-        logger.error(f"[SDK] Failed to upload transcript for {session_id}: {e}")
-
-
-async def _update_title_async(
-    session_id: str, message: str, user_id: str | None = None
-) -> None:
-    """Background task to update session title."""
-    try:
-        title = await _generate_session_title(
-            message, user_id=user_id, session_id=session_id
-        )
-        if title:
-            await update_session_title(session_id, title)
-            logger.debug(f"[SDK] Generated title for {session_id}: {title}")
-    except Exception as e:
-        logger.warning(f"[SDK] Failed to update session title: {e}")

autogpt_platform/backend/backend/api/features/chat/sdk/tool_adapter.py

@@ -1,363 +0,0 @@
-"""Tool adapter for wrapping existing CoPilot tools as Claude Agent SDK MCP tools.
-
-This module provides the adapter layer that converts existing BaseTool implementations
-into in-process MCP tools that can be used with the Claude Agent SDK.
-
-Long-running tools (``is_long_running=True``) are delegated to the non-SDK
-background infrastructure (stream_registry, Redis persistence, SSE reconnection)
-via a callback provided by the service layer.  This avoids wasteful SDK polling
-and makes results survive page refreshes.
-"""
-
-import itertools
-import json
-import logging
-import os
-import uuid
-from collections.abc import Awaitable, Callable
-from contextvars import ContextVar
-from typing import Any
-
-from backend.api.features.chat.model import ChatSession
-from backend.api.features.chat.tools import TOOL_REGISTRY
-from backend.api.features.chat.tools.base import BaseTool
-
-logger = logging.getLogger(__name__)
-
-# Allowed base directory for the Read tool (SDK saves oversized tool results here).
-# Restricted to ~/.claude/projects/ and further validated to require "tool-results"
-# in the path — prevents reading settings, credentials, or other sensitive files.
-_SDK_PROJECTS_DIR = os.path.expanduser("~/.claude/projects/")
-
-# MCP server naming - the SDK prefixes tool names as "mcp__{server_name}__{tool}"
-MCP_SERVER_NAME = "copilot"
-MCP_TOOL_PREFIX = f"mcp__{MCP_SERVER_NAME}__"
-
-# Context variables to pass user/session info to tool execution
-_current_user_id: ContextVar[str | None] = ContextVar("current_user_id", default=None)
-_current_session: ContextVar[ChatSession | None] = ContextVar(
-    "current_session", default=None
-)
-# Stash for MCP tool outputs before the SDK potentially truncates them.
-# Keyed by tool_name → full output string. Consumed (popped) by the
-# response adapter when it builds StreamToolOutputAvailable.
-_pending_tool_outputs: ContextVar[dict[str, str]] = ContextVar(
-    "pending_tool_outputs", default=None  # type: ignore[arg-type]
-)
-
-# Callback type for delegating long-running tools to the non-SDK infrastructure.
-# Args: (tool_name, arguments, session) → MCP-formatted response dict.
-LongRunningCallback = Callable[
-    [str, dict[str, Any], ChatSession], Awaitable[dict[str, Any]]
-]
-
-# ContextVar so the service layer can inject the callback per-request.
-_long_running_callback: ContextVar[LongRunningCallback | None] = ContextVar(
-    "long_running_callback", default=None
-)
-
-
-def set_execution_context(
-    user_id: str | None,
-    session: ChatSession,
-    long_running_callback: LongRunningCallback | None = None,
-) -> None:
-    """Set the execution context for tool calls.
-
-    This must be called before streaming begins to ensure tools have access
-    to user_id and session information.
-
-    Args:
-        user_id: Current user's ID.
-        session: Current chat session.
-        long_running_callback: Optional callback to delegate long-running tools
-            to the non-SDK background infrastructure (stream_registry + Redis).
-    """
-    _current_user_id.set(user_id)
-    _current_session.set(session)
-    _pending_tool_outputs.set({})
-    _long_running_callback.set(long_running_callback)
-
-
-def get_execution_context() -> tuple[str | None, ChatSession | None]:
-    """Get the current execution context."""
-    return (
-        _current_user_id.get(),
-        _current_session.get(),
-    )
-
-
-def pop_pending_tool_output(tool_name: str) -> str | None:
-    """Pop and return the stashed full output for *tool_name*.
-
-    The SDK CLI may truncate large tool results (writing them to disk and
-    replacing the content with a file reference). This stash keeps the
-    original MCP output so the response adapter can forward it to the
-    frontend for proper widget rendering.
-
-    Returns ``None`` if nothing was stashed for *tool_name*.
-    """
-    pending = _pending_tool_outputs.get(None)
-    if pending is None:
-        return None
-    return pending.pop(tool_name, None)
-
-
-async def _execute_tool_sync(
-    base_tool: BaseTool,
-    user_id: str | None,
-    session: ChatSession,
-    args: dict[str, Any],
-) -> dict[str, Any]:
-    """Execute a tool synchronously and return MCP-formatted response."""
-    effective_id = f"sdk-{uuid.uuid4().hex[:12]}"
-    result = await base_tool.execute(
-        user_id=user_id,
-        session=session,
-        tool_call_id=effective_id,
-        **args,
-    )
-
-    text = (
-        result.output if isinstance(result.output, str) else json.dumps(result.output)
-    )
-
-    # Stash the full output before the SDK potentially truncates it.
-    pending = _pending_tool_outputs.get(None)
-    if pending is not None:
-        pending[base_tool.name] = text
-
-    return {
-        "content": [{"type": "text", "text": text}],
-        "isError": not result.success,
-    }
-
-
-def _mcp_error(message: str) -> dict[str, Any]:
-    return {
-        "content": [
-            {"type": "text", "text": json.dumps({"error": message, "type": "error"})}
-        ],
-        "isError": True,
-    }
-
-
-def create_tool_handler(base_tool: BaseTool):
-    """Create an async handler function for a BaseTool.
-
-    This wraps the existing BaseTool._execute method to be compatible
-    with the Claude Agent SDK MCP tool format.
-
-    Long-running tools (``is_long_running=True``) are delegated to the
-    non-SDK background infrastructure via a callback set in the execution
-    context.  The callback persists the operation in Redis (stream_registry)
-    so results survive page refreshes and pod restarts.
-    """
-
-    async def tool_handler(args: dict[str, Any]) -> dict[str, Any]:
-        """Execute the wrapped tool and return MCP-formatted response."""
-        user_id, session = get_execution_context()
-
-        if session is None:
-            return _mcp_error("No session context available")
-
-        # --- Long-running: delegate to non-SDK background infrastructure ---
-        if base_tool.is_long_running:
-            callback = _long_running_callback.get(None)
-            if callback:
-                try:
-                    return await callback(base_tool.name, args, session)
-                except Exception as e:
-                    logger.error(
-                        f"Long-running callback failed for {base_tool.name}: {e}",
-                        exc_info=True,
-                    )
-                    return _mcp_error(f"Failed to start {base_tool.name}: {e}")
-            # No callback — fall through to synchronous execution
-            logger.warning(
-                f"[SDK] No long-running callback for {base_tool.name}, "
-                f"executing synchronously (may block)"
-            )
-
-        # --- Normal (fast) tool: execute synchronously ---
-        try:
-            return await _execute_tool_sync(base_tool, user_id, session, args)
-        except Exception as e:
-            logger.error(f"Error executing tool {base_tool.name}: {e}", exc_info=True)
-            return _mcp_error(f"Failed to execute {base_tool.name}: {e}")
-
-    return tool_handler
-
-
-def _build_input_schema(base_tool: BaseTool) -> dict[str, Any]:
-    """Build a JSON Schema input schema for a tool."""
-    return {
-        "type": "object",
-        "properties": base_tool.parameters.get("properties", {}),
-        "required": base_tool.parameters.get("required", []),
-    }
-
-
-async def _read_file_handler(args: dict[str, Any]) -> dict[str, Any]:
-    """Read a file with optional offset/limit. Restricted to SDK working directory.
-
-    After reading, the file is deleted to prevent accumulation in long-running pods.
-    """
-    file_path = args.get("file_path", "")
-    offset = args.get("offset", 0)
-    limit = args.get("limit", 2000)
-
-    # Security: only allow reads under ~/.claude/projects/**/tool-results/
-    real_path = os.path.realpath(file_path)
-    if not real_path.startswith(_SDK_PROJECTS_DIR) or "tool-results" not in real_path:
-        return {
-            "content": [{"type": "text", "text": f"Access denied: {file_path}"}],
-            "isError": True,
-        }
-
-    try:
-        with open(real_path) as f:
-            selected = list(itertools.islice(f, offset, offset + limit))
-        content = "".join(selected)
-        # Cleanup happens in _cleanup_sdk_tool_results after session ends;
-        # don't delete here — the SDK may read in multiple chunks.
-        return {"content": [{"type": "text", "text": content}], "isError": False}
-    except FileNotFoundError:
-        return {
-            "content": [{"type": "text", "text": f"File not found: {file_path}"}],
-            "isError": True,
-        }
-    except Exception as e:
-        return {
-            "content": [{"type": "text", "text": f"Error reading file: {e}"}],
-            "isError": True,
-        }
-
-
-_READ_TOOL_NAME = "Read"
-_READ_TOOL_DESCRIPTION = (
-    "Read a file from the local filesystem. "
-    "Use offset and limit to read specific line ranges for large files."
-)
-_READ_TOOL_SCHEMA = {
-    "type": "object",
-    "properties": {
-        "file_path": {
-            "type": "string",
-            "description": "The absolute path to the file to read",
-        },
-        "offset": {
-            "type": "integer",
-            "description": "Line number to start reading from (0-indexed). Default: 0",
-        },
-        "limit": {
-            "type": "integer",
-            "description": "Number of lines to read. Default: 2000",
-        },
-    },
-    "required": ["file_path"],
-}
-
-
-# Create the MCP server configuration
-def create_copilot_mcp_server():
-    """Create an in-process MCP server configuration for CoPilot tools.
-
-    This can be passed to ClaudeAgentOptions.mcp_servers.
-
-    Note: The actual SDK MCP server creation depends on the claude-agent-sdk
-    package being available. This function returns the configuration that
-    can be used with the SDK.
-    """
-    try:
-        from claude_agent_sdk import create_sdk_mcp_server, tool
-
-        # Create decorated tool functions
-        sdk_tools = []
-
-        for tool_name, base_tool in TOOL_REGISTRY.items():
-            handler = create_tool_handler(base_tool)
-            decorated = tool(
-                tool_name,
-                base_tool.description,
-                _build_input_schema(base_tool),
-            )(handler)
-            sdk_tools.append(decorated)
-
-        # Add the Read tool so the SDK can read back oversized tool results
-        read_tool = tool(
-            _READ_TOOL_NAME,
-            _READ_TOOL_DESCRIPTION,
-            _READ_TOOL_SCHEMA,
-        )(_read_file_handler)
-        sdk_tools.append(read_tool)
-
-        server = create_sdk_mcp_server(
-            name=MCP_SERVER_NAME,
-            version="1.0.0",
-            tools=sdk_tools,
-        )
-
-        return server
-
-    except ImportError:
-        # Let ImportError propagate so service.py handles the fallback
-        raise
-
-
-# SDK built-in tools allowed within the workspace directory.
-# Security hooks validate that file paths stay within sdk_cwd.
-# Bash is NOT included — use the sandboxed MCP bash_exec tool instead,
-# which provides kernel-level network isolation via unshare --net.
-# Task allows spawning sub-agents (rate-limited by security hooks).
-# WebSearch uses Brave Search via Anthropic's API — safe, no SSRF risk.
-_SDK_BUILTIN_TOOLS = ["Read", "Write", "Edit", "Glob", "Grep", "Task", "WebSearch"]
-
-# SDK built-in tools that must be explicitly blocked.
-# Bash: dangerous — agent uses mcp__copilot__bash_exec with kernel-level
-#   network isolation (unshare --net) instead.
-# WebFetch: SSRF risk — can reach internal network (localhost, 10.x, etc.).
-#   Agent uses the SSRF-protected mcp__copilot__web_fetch tool instead.
-SDK_DISALLOWED_TOOLS = ["Bash", "WebFetch"]
-
-# Tools that are blocked entirely in security hooks (defence-in-depth).
-# Includes SDK_DISALLOWED_TOOLS plus common aliases/synonyms.
-BLOCKED_TOOLS = {
-    *SDK_DISALLOWED_TOOLS,
-    "bash",
-    "shell",
-    "exec",
-    "terminal",
-    "command",
-}
-
-# Tools allowed only when their path argument stays within the SDK workspace.
-# The SDK uses these to handle oversized tool results (writes to tool-results/
-# files, then reads them back) and for workspace file operations.
-WORKSPACE_SCOPED_TOOLS = {"Read", "Write", "Edit", "Glob", "Grep"}
-
-# Dangerous patterns in tool inputs
-DANGEROUS_PATTERNS = [
-    r"sudo",
-    r"rm\s+-rf",
-    r"dd\s+if=",
-    r"/etc/passwd",
-    r"/etc/shadow",
-    r"chmod\s+777",
-    r"curl\s+.*\|.*sh",
-    r"wget\s+.*\|.*sh",
-    r"eval\s*\(",
-    r"exec\s*\(",
-    r"__import__",
-    r"os\.system",
-    r"subprocess",
-]
-
-# List of tool names for allowed_tools configuration
-# Include MCP tools, the MCP Read tool for oversized results,
-# and SDK built-in file tools for workspace operations.
-COPILOT_TOOL_NAMES = [
-    *[f"{MCP_TOOL_PREFIX}{name}" for name in TOOL_REGISTRY.keys()],
-    f"{MCP_TOOL_PREFIX}{_READ_TOOL_NAME}",
-    *_SDK_BUILTIN_TOOLS,
-]

autogpt_platform/backend/backend/api/features/chat/sdk/transcript.py

@@ -1,356 +0,0 @@
-"""JSONL transcript management for stateless multi-turn resume.
-
-The Claude Code CLI persists conversations as JSONL files (one JSON object per
-line).  When the SDK's ``Stop`` hook fires we read this file, strip bloat
-(progress entries, metadata), and upload the result to bucket storage.  On the
-next turn we download the transcript, write it to a temp file, and pass
-``--resume`` so the CLI can reconstruct the full conversation.
-
-Storage is handled via ``WorkspaceStorageBackend`` (GCS in prod, local
-filesystem for self-hosted) — no DB column needed.
-"""
-
-import json
-import logging
-import os
-import re
-
-logger = logging.getLogger(__name__)
-
-# UUIDs are hex + hyphens; strip everything else to prevent path injection.
-_SAFE_ID_RE = re.compile(r"[^0-9a-fA-F-]")
-
-# Entry types that can be safely removed from the transcript without breaking
-# the parentUuid conversation tree that ``--resume`` relies on.
-# - progress: UI progress ticks, no message content (avg 97KB for agent_progress)
-# - file-history-snapshot: undo tracking metadata
-# - queue-operation: internal queue bookkeeping
-# - summary: session summaries
-# - pr-link: PR link metadata
-STRIPPABLE_TYPES = frozenset(
-    {"progress", "file-history-snapshot", "queue-operation", "summary", "pr-link"}
-)
-
-# Workspace storage constants — deterministic path from session_id.
-TRANSCRIPT_STORAGE_PREFIX = "chat-transcripts"
-
-
-# ---------------------------------------------------------------------------
-# Progress stripping
-# ---------------------------------------------------------------------------
-
-
-def strip_progress_entries(content: str) -> str:
-    """Remove progress/metadata entries from a JSONL transcript.
-
-    Removes entries whose ``type`` is in ``STRIPPABLE_TYPES`` and reparents
-    any remaining child entries so the ``parentUuid`` chain stays intact.
-    Typically reduces transcript size by ~30%.
-    """
-    lines = content.strip().split("\n")
-
-    entries: list[dict] = []
-    for line in lines:
-        try:
-            entries.append(json.loads(line))
-        except json.JSONDecodeError:
-            # Keep unparseable lines as-is (safety)
-            entries.append({"_raw": line})
-
-    stripped_uuids: set[str] = set()
-    uuid_to_parent: dict[str, str] = {}
-    kept: list[dict] = []
-
-    for entry in entries:
-        if "_raw" in entry:
-            kept.append(entry)
-            continue
-        uid = entry.get("uuid", "")
-        parent = entry.get("parentUuid", "")
-        entry_type = entry.get("type", "")
-
-        if uid:
-            uuid_to_parent[uid] = parent
-
-        if entry_type in STRIPPABLE_TYPES:
-            if uid:
-                stripped_uuids.add(uid)
-        else:
-            kept.append(entry)
-
-    # Reparent: walk up chain through stripped entries to find surviving ancestor
-    for entry in kept:
-        if "_raw" in entry:
-            continue
-        parent = entry.get("parentUuid", "")
-        original_parent = parent
-        while parent in stripped_uuids:
-            parent = uuid_to_parent.get(parent, "")
-        if parent != original_parent:
-            entry["parentUuid"] = parent
-
-    result_lines: list[str] = []
-    for entry in kept:
-        if "_raw" in entry:
-            result_lines.append(entry["_raw"])
-        else:
-            result_lines.append(json.dumps(entry, separators=(",", ":")))
-
-    return "\n".join(result_lines) + "\n"
-
-
-# ---------------------------------------------------------------------------
-# Local file I/O (read from CLI's JSONL, write temp file for --resume)
-# ---------------------------------------------------------------------------
-
-
-def read_transcript_file(transcript_path: str) -> str | None:
-    """Read a JSONL transcript file from disk.
-
-    Returns the raw JSONL content, or ``None`` if the file is missing, empty,
-    or only contains metadata (≤2 lines with no conversation messages).
-    """
-    if not transcript_path or not os.path.isfile(transcript_path):
-        logger.debug(f"[Transcript] File not found: {transcript_path}")
-        return None
-
-    try:
-        with open(transcript_path) as f:
-            content = f.read()
-
-        if not content.strip():
-            logger.debug(f"[Transcript] Empty file: {transcript_path}")
-            return None
-
-        lines = content.strip().split("\n")
-        if len(lines) < 3:
-            # Raw files with ≤2 lines are metadata-only
-            # (queue-operation + file-history-snapshot, no conversation).
-            logger.debug(
-                f"[Transcript] Too few lines ({len(lines)}): {transcript_path}"
-            )
-            return None
-
-        # Quick structural validation — parse first and last lines.
-        json.loads(lines[0])
-        json.loads(lines[-1])
-
-        logger.info(
-            f"[Transcript] Read {len(lines)} lines, "
-            f"{len(content)} bytes from {transcript_path}"
-        )
-        return content
-
-    except (json.JSONDecodeError, OSError) as e:
-        logger.warning(f"[Transcript] Failed to read {transcript_path}: {e}")
-        return None
-
-
-def _sanitize_id(raw_id: str, max_len: int = 36) -> str:
-    """Sanitize an ID for safe use in file paths.
-
-    Session/user IDs are expected to be UUIDs (hex + hyphens).  Strip
-    everything else and truncate to *max_len* so the result cannot introduce
-    path separators or other special characters.
-    """
-    cleaned = _SAFE_ID_RE.sub("", raw_id or "")[:max_len]
-    return cleaned or "unknown"
-
-
-_SAFE_CWD_PREFIX = os.path.realpath("/tmp/copilot-")
-
-
-def write_transcript_to_tempfile(
-    transcript_content: str,
-    session_id: str,
-    cwd: str,
-) -> str | None:
-    """Write JSONL transcript to a temp file inside *cwd* for ``--resume``.
-
-    The file lives in the session working directory so it is cleaned up
-    automatically when the session ends.
-
-    Returns the absolute path to the file, or ``None`` on failure.
-    """
-    # Validate cwd is under the expected sandbox prefix (CodeQL sanitizer).
-    real_cwd = os.path.realpath(cwd)
-    if not real_cwd.startswith(_SAFE_CWD_PREFIX):
-        logger.warning(f"[Transcript] cwd outside sandbox: {cwd}")
-        return None
-
-    try:
-        os.makedirs(real_cwd, exist_ok=True)
-        safe_id = _sanitize_id(session_id, max_len=8)
-        jsonl_path = os.path.realpath(
-            os.path.join(real_cwd, f"transcript-{safe_id}.jsonl")
-        )
-        if not jsonl_path.startswith(real_cwd):
-            logger.warning(f"[Transcript] Path escaped cwd: {jsonl_path}")
-            return None
-
-        with open(jsonl_path, "w") as f:
-            f.write(transcript_content)
-
-        logger.info(f"[Transcript] Wrote resume file: {jsonl_path}")
-        return jsonl_path
-
-    except OSError as e:
-        logger.warning(f"[Transcript] Failed to write resume file: {e}")
-        return None
-
-
-def validate_transcript(content: str | None) -> bool:
-    """Check that a transcript has actual conversation messages.
-
-    A valid transcript for resume needs at least one user message and one
-    assistant message (not just queue-operation / file-history-snapshot
-    metadata).
-    """
-    if not content or not content.strip():
-        return False
-
-    lines = content.strip().split("\n")
-    if len(lines) < 2:
-        return False
-
-    has_user = False
-    has_assistant = False
-
-    for line in lines:
-        try:
-            entry = json.loads(line)
-            msg_type = entry.get("type")
-            if msg_type == "user":
-                has_user = True
-            elif msg_type == "assistant":
-                has_assistant = True
-        except json.JSONDecodeError:
-            return False
-
-    return has_user and has_assistant
-
-
-# ---------------------------------------------------------------------------
-# Bucket storage (GCS / local via WorkspaceStorageBackend)
-# ---------------------------------------------------------------------------
-
-
-def _storage_path_parts(user_id: str, session_id: str) -> tuple[str, str, str]:
-    """Return (workspace_id, file_id, filename) for a session's transcript.
-
-    Path structure: ``chat-transcripts/{user_id}/{session_id}.jsonl``
-    IDs are sanitized to hex+hyphen to prevent path traversal.
-    """
-    return (
-        TRANSCRIPT_STORAGE_PREFIX,
-        _sanitize_id(user_id),
-        f"{_sanitize_id(session_id)}.jsonl",
-    )
-
-
-def _build_storage_path(user_id: str, session_id: str, backend: object) -> str:
-    """Build the full storage path string that ``retrieve()`` expects.
-
-    ``store()`` returns a path like ``gcs://bucket/workspaces/...`` or
-    ``local://workspace_id/file_id/filename``.  Since we use deterministic
-    arguments we can reconstruct the same path for download/delete without
-    having stored the return value.
-    """
-    from backend.util.workspace_storage import GCSWorkspaceStorage
-
-    wid, fid, fname = _storage_path_parts(user_id, session_id)
-
-    if isinstance(backend, GCSWorkspaceStorage):
-        blob = f"workspaces/{wid}/{fid}/{fname}"
-        return f"gcs://{backend.bucket_name}/{blob}"
-    else:
-        # LocalWorkspaceStorage returns local://{relative_path}
-        return f"local://{wid}/{fid}/{fname}"
-
-
-async def upload_transcript(user_id: str, session_id: str, content: str) -> None:
-    """Strip progress entries and upload transcript to bucket storage.
-
-    Safety: only overwrites when the new (stripped) transcript is larger than
-    what is already stored.  Since JSONL is append-only, the latest transcript
-    is always the longest.  This prevents a slow/stale background task from
-    clobbering a newer upload from a concurrent turn.
-    """
-    from backend.util.workspace_storage import get_workspace_storage
-
-    stripped = strip_progress_entries(content)
-    if not validate_transcript(stripped):
-        logger.warning(
-            f"[Transcript] Skipping upload — stripped content is not a valid "
-            f"transcript for session {session_id}"
-        )
-        return
-
-    storage = await get_workspace_storage()
-    wid, fid, fname = _storage_path_parts(user_id, session_id)
-    encoded = stripped.encode("utf-8")
-    new_size = len(encoded)
-
-    # Check existing transcript size to avoid overwriting newer with older
-    path = _build_storage_path(user_id, session_id, storage)
-    try:
-        existing = await storage.retrieve(path)
-        if len(existing) >= new_size:
-            logger.info(
-                f"[Transcript] Skipping upload — existing transcript "
-                f"({len(existing)}B) >= new ({new_size}B) for session "
-                f"{session_id}"
-            )
-            return
-    except (FileNotFoundError, Exception):
-        pass  # No existing transcript or retrieval error — proceed with upload
-
-    await storage.store(
-        workspace_id=wid,
-        file_id=fid,
-        filename=fname,
-        content=encoded,
-    )
-    logger.info(
-        f"[Transcript] Uploaded {new_size} bytes "
-        f"(stripped from {len(content)}) for session {session_id}"
-    )
-
-
-async def download_transcript(user_id: str, session_id: str) -> str | None:
-    """Download transcript from bucket storage.
-
-    Returns the JSONL content string, or ``None`` if not found.
-    """
-    from backend.util.workspace_storage import get_workspace_storage
-
-    storage = await get_workspace_storage()
-    path = _build_storage_path(user_id, session_id, storage)
-
-    try:
-        data = await storage.retrieve(path)
-        content = data.decode("utf-8")
-        logger.info(
-            f"[Transcript] Downloaded {len(content)} bytes for session {session_id}"
-        )
-        return content
-    except FileNotFoundError:
-        logger.debug(f"[Transcript] No transcript in storage for {session_id}")
-        return None
-    except Exception as e:
-        logger.warning(f"[Transcript] Failed to download transcript: {e}")
-        return None
-
-
-async def delete_transcript(user_id: str, session_id: str) -> None:
-    """Delete transcript from bucket storage (e.g. after resume failure)."""
-    from backend.util.workspace_storage import get_workspace_storage
-
-    storage = await get_workspace_storage()
-    path = _build_storage_path(user_id, session_id, storage)
-
-    try:
-        await storage.delete(path)
-        logger.info(f"[Transcript] Deleted transcript for session {session_id}")
-    except Exception as e:
-        logger.warning(f"[Transcript] Failed to delete transcript: {e}")

autogpt_platform/backend/backend/api/features/chat/tools/agent_generator/dummy.py

@@ -1,154 +0,0 @@
-"""Dummy Agent Generator for testing.
-
-Returns mock responses matching the format expected from the external service.
-Enable via AGENTGENERATOR_USE_DUMMY=true in settings.
-
-WARNING: This is for testing only. Do not use in production.
-"""
-
-import asyncio
-import logging
-import uuid
-from typing import Any
-
-logger = logging.getLogger(__name__)
-
-# Dummy decomposition result (instructions type)
-DUMMY_DECOMPOSITION_RESULT: dict[str, Any] = {
-    "type": "instructions",
-    "steps": [
-        {
-            "description": "Get input from user",
-            "action": "input",
-            "block_name": "AgentInputBlock",
-        },
-        {
-            "description": "Process the input",
-            "action": "process",
-            "block_name": "TextFormatterBlock",
-        },
-        {
-            "description": "Return output to user",
-            "action": "output",
-            "block_name": "AgentOutputBlock",
-        },
-    ],
-}
-
-# Block IDs from backend/blocks/io.py
-AGENT_INPUT_BLOCK_ID = "c0a8e994-ebf1-4a9c-a4d8-89d09c86741b"
-AGENT_OUTPUT_BLOCK_ID = "363ae599-353e-4804-937e-b2ee3cef3da4"
-
-
-def _generate_dummy_agent_json() -> dict[str, Any]:
-    """Generate a minimal valid agent JSON for testing."""
-    input_node_id = str(uuid.uuid4())
-    output_node_id = str(uuid.uuid4())
-
-    return {
-        "id": str(uuid.uuid4()),
-        "version": 1,
-        "is_active": True,
-        "name": "Dummy Test Agent",
-        "description": "A dummy agent generated for testing purposes",
-        "nodes": [
-            {
-                "id": input_node_id,
-                "block_id": AGENT_INPUT_BLOCK_ID,
-                "input_default": {
-                    "name": "input",
-                    "title": "Input",
-                    "description": "Enter your input",
-                    "placeholder_values": [],
-                },
-                "metadata": {"position": {"x": 0, "y": 0}},
-            },
-            {
-                "id": output_node_id,
-                "block_id": AGENT_OUTPUT_BLOCK_ID,
-                "input_default": {
-                    "name": "output",
-                    "title": "Output",
-                    "description": "Agent output",
-                    "format": "{output}",
-                },
-                "metadata": {"position": {"x": 400, "y": 0}},
-            },
-        ],
-        "links": [
-            {
-                "id": str(uuid.uuid4()),
-                "source_id": input_node_id,
-                "sink_id": output_node_id,
-                "source_name": "result",
-                "sink_name": "value",
-                "is_static": False,
-            },
-        ],
-    }
-
-
-async def decompose_goal_dummy(
-    description: str,
-    context: str = "",
-    library_agents: list[dict[str, Any]] | None = None,
-) -> dict[str, Any]:
-    """Return dummy decomposition result."""
-    logger.info("Using dummy agent generator for decompose_goal")
-    return DUMMY_DECOMPOSITION_RESULT.copy()
-
-
-async def generate_agent_dummy(
-    instructions: dict[str, Any],
-    library_agents: list[dict[str, Any]] | None = None,
-    operation_id: str | None = None,
-    task_id: str | None = None,
-) -> dict[str, Any]:
-    """Return dummy agent JSON after a simulated delay."""
-    logger.info("Using dummy agent generator for generate_agent (30s delay)")
-    await asyncio.sleep(30)
-    return _generate_dummy_agent_json()
-
-
-async def generate_agent_patch_dummy(
-    update_request: str,
-    current_agent: dict[str, Any],
-    library_agents: list[dict[str, Any]] | None = None,
-    operation_id: str | None = None,
-    task_id: str | None = None,
-) -> dict[str, Any]:
-    """Return dummy patched agent (returns the current agent with updated description)."""
-    logger.info("Using dummy agent generator for generate_agent_patch")
-    patched = current_agent.copy()
-    patched["description"] = (
-        f"{current_agent.get('description', '')} (updated: {update_request})"
-    )
-    return patched
-
-
-async def customize_template_dummy(
-    template_agent: dict[str, Any],
-    modification_request: str,
-    context: str = "",
-) -> dict[str, Any]:
-    """Return dummy customized template (returns template with updated description)."""
-    logger.info("Using dummy agent generator for customize_template")
-    customized = template_agent.copy()
-    customized["description"] = (
-        f"{template_agent.get('description', '')} (customized: {modification_request})"
-    )
-    return customized
-
-
-async def get_blocks_dummy() -> list[dict[str, Any]]:
-    """Return dummy blocks list."""
-    logger.info("Using dummy agent generator for get_blocks")
-    return [
-        {"id": AGENT_INPUT_BLOCK_ID, "name": "AgentInputBlock"},
-        {"id": AGENT_OUTPUT_BLOCK_ID, "name": "AgentOutputBlock"},
-    ]
-
-
-async def health_check_dummy() -> bool:
-    """Always returns healthy for dummy service."""
-    return True

autogpt_platform/backend/backend/api/features/chat/tools/agent_generator/service.py

@@ -1,549 +0,0 @@
-"""External Agent Generator service client.
-
-This module provides a client for communicating with the external Agent Generator
-microservice. When AGENTGENERATOR_HOST is configured, the agent generation functions
-will delegate to the external service instead of using the built-in LLM-based implementation.
-"""
-
-import logging
-from typing import Any
-
-import httpx
-
-from backend.util.settings import Settings
-
-from .dummy import (
-    customize_template_dummy,
-    decompose_goal_dummy,
-    generate_agent_dummy,
-    generate_agent_patch_dummy,
-    get_blocks_dummy,
-    health_check_dummy,
-)
-
-logger = logging.getLogger(__name__)
-
-_dummy_mode_warned = False
-
-
-def _create_error_response(
-    error_message: str,
-    error_type: str = "unknown",
-    details: dict[str, Any] | None = None,
-) -> dict[str, Any]:
-    """Create a standardized error response dict.
-
-    Args:
-        error_message: Human-readable error message
-        error_type: Machine-readable error type
-        details: Optional additional error details
-
-    Returns:
-        Error dict with type="error" and error details
-    """
-    response: dict[str, Any] = {
-        "type": "error",
-        "error": error_message,
-        "error_type": error_type,
-    }
-    if details:
-        response["details"] = details
-    return response
-
-
-def _classify_http_error(e: httpx.HTTPStatusError) -> tuple[str, str]:
-    """Classify an HTTP error into error_type and message.
-
-    Args:
-        e: The HTTP status error
-
-    Returns:
-        Tuple of (error_type, error_message)
-    """
-    status = e.response.status_code
-    if status == 429:
-        return "rate_limit", f"Agent Generator rate limited: {e}"
-    elif status == 503:
-        return "service_unavailable", f"Agent Generator unavailable: {e}"
-    elif status == 504 or status == 408:
-        return "timeout", f"Agent Generator timed out: {e}"
-    else:
-        return "http_error", f"HTTP error calling Agent Generator: {e}"
-
-
-def _classify_request_error(e: httpx.RequestError) -> tuple[str, str]:
-    """Classify a request error into error_type and message.
-
-    Args:
-        e: The request error
-
-    Returns:
-        Tuple of (error_type, error_message)
-    """
-    error_str = str(e).lower()
-    if "timeout" in error_str or "timed out" in error_str:
-        return "timeout", f"Agent Generator request timed out: {e}"
-    elif "connect" in error_str:
-        return "connection_error", f"Could not connect to Agent Generator: {e}"
-    else:
-        return "request_error", f"Request error calling Agent Generator: {e}"
-
-
-_client: httpx.AsyncClient | None = None
-_settings: Settings | None = None
-
-
-def _get_settings() -> Settings:
-    """Get or create settings singleton."""
-    global _settings
-    if _settings is None:
-        _settings = Settings()
-    return _settings
-
-
-def _is_dummy_mode() -> bool:
-    """Check if dummy mode is enabled for testing."""
-    global _dummy_mode_warned
-    settings = _get_settings()
-    is_dummy = bool(settings.config.agentgenerator_use_dummy)
-    if is_dummy and not _dummy_mode_warned:
-        logger.warning(
-            "Agent Generator running in DUMMY MODE - returning mock responses. "
-            "Do not use in production!"
-        )
-        _dummy_mode_warned = True
-    return is_dummy
-
-
-def is_external_service_configured() -> bool:
-    """Check if external Agent Generator service is configured (or dummy mode)."""
-    settings = _get_settings()
-    return bool(settings.config.agentgenerator_host) or bool(
-        settings.config.agentgenerator_use_dummy
-    )
-
-
-def _get_base_url() -> str:
-    """Get the base URL for the external service."""
-    settings = _get_settings()
-    host = settings.config.agentgenerator_host
-    port = settings.config.agentgenerator_port
-    return f"http://{host}:{port}"
-
-
-def _get_client() -> httpx.AsyncClient:
-    """Get or create the HTTP client for the external service."""
-    global _client
-    if _client is None:
-        settings = _get_settings()
-        _client = httpx.AsyncClient(
-            base_url=_get_base_url(),
-            timeout=httpx.Timeout(settings.config.agentgenerator_timeout),
-        )
-    return _client
-
-
-async def decompose_goal_external(
-    description: str,
-    context: str = "",
-    library_agents: list[dict[str, Any]] | None = None,
-) -> dict[str, Any] | None:
-    """Call the external service to decompose a goal.
-
-    Args:
-        description: Natural language goal description
-        context: Additional context (e.g., answers to previous questions)
-        library_agents: User's library agents available for sub-agent composition
-
-    Returns:
-        Dict with either:
-        - {"type": "clarifying_questions", "questions": [...]}
-        - {"type": "instructions", "steps": [...]}
-        - {"type": "unachievable_goal", ...}
-        - {"type": "vague_goal", ...}
-        - {"type": "error", "error": "...", "error_type": "..."} on error
-        Or None on unexpected error
-    """
-    if _is_dummy_mode():
-        return await decompose_goal_dummy(description, context, library_agents)
-
-    client = _get_client()
-
-    if context:
-        description = f"{description}\n\nAdditional context from user:\n{context}"
-
-    payload: dict[str, Any] = {"description": description}
-    if library_agents:
-        payload["library_agents"] = library_agents
-
-    try:
-        response = await client.post("/api/decompose-description", json=payload)
-        response.raise_for_status()
-        data = response.json()
-
-        if not data.get("success"):
-            error_msg = data.get("error", "Unknown error from Agent Generator")
-            error_type = data.get("error_type", "unknown")
-            logger.error(
-                f"Agent Generator decomposition failed: {error_msg} "
-                f"(type: {error_type})"
-            )
-            return _create_error_response(error_msg, error_type)
-
-        # Map the response to the expected format
-        response_type = data.get("type")
-        if response_type == "instructions":
-            return {"type": "instructions", "steps": data.get("steps", [])}
-        elif response_type == "clarifying_questions":
-            return {
-                "type": "clarifying_questions",
-                "questions": data.get("questions", []),
-            }
-        elif response_type == "unachievable_goal":
-            return {
-                "type": "unachievable_goal",
-                "reason": data.get("reason"),
-                "suggested_goal": data.get("suggested_goal"),
-            }
-        elif response_type == "vague_goal":
-            return {
-                "type": "vague_goal",
-                "suggested_goal": data.get("suggested_goal"),
-            }
-        elif response_type == "error":
-            # Pass through error from the service
-            return _create_error_response(
-                data.get("error", "Unknown error"),
-                data.get("error_type", "unknown"),
-            )
-        else:
-            logger.error(
-                f"Unknown response type from external service: {response_type}"
-            )
-            return _create_error_response(
-                f"Unknown response type from Agent Generator: {response_type}",
-                "invalid_response",
-            )
-
-    except httpx.HTTPStatusError as e:
-        error_type, error_msg = _classify_http_error(e)
-        logger.error(error_msg)
-        return _create_error_response(error_msg, error_type)
-    except httpx.RequestError as e:
-        error_type, error_msg = _classify_request_error(e)
-        logger.error(error_msg)
-        return _create_error_response(error_msg, error_type)
-    except Exception as e:
-        error_msg = f"Unexpected error calling Agent Generator: {e}"
-        logger.error(error_msg)
-        return _create_error_response(error_msg, "unexpected_error")
-
-
-async def generate_agent_external(
-    instructions: dict[str, Any],
-    library_agents: list[dict[str, Any]] | None = None,
-    operation_id: str | None = None,
-    task_id: str | None = None,
-) -> dict[str, Any] | None:
-    """Call the external service to generate an agent from instructions.
-
-    Args:
-        instructions: Structured instructions from decompose_goal
-        library_agents: User's library agents available for sub-agent composition
-        operation_id: Operation ID for async processing (enables Redis Streams callback)
-        task_id: Task ID for async processing (enables Redis Streams callback)
-
-    Returns:
-        Agent JSON dict, {"status": "accepted"} for async, or error dict {"type": "error", ...} on error
-    """
-    if _is_dummy_mode():
-        return await generate_agent_dummy(
-            instructions, library_agents, operation_id, task_id
-        )
-
-    client = _get_client()
-
-    # Build request payload
-    payload: dict[str, Any] = {"instructions": instructions}
-    if library_agents:
-        payload["library_agents"] = library_agents
-    if operation_id and task_id:
-        payload["operation_id"] = operation_id
-        payload["task_id"] = task_id
-
-    try:
-        response = await client.post("/api/generate-agent", json=payload)
-
-        # Handle 202 Accepted for async processing
-        if response.status_code == 202:
-            logger.info(
-                f"Agent Generator accepted async request "
-                f"(operation_id={operation_id}, task_id={task_id})"
-            )
-            return {
-                "status": "accepted",
-                "operation_id": operation_id,
-                "task_id": task_id,
-            }
-
-        response.raise_for_status()
-        data = response.json()
-
-        if not data.get("success"):
-            error_msg = data.get("error", "Unknown error from Agent Generator")
-            error_type = data.get("error_type", "unknown")
-            logger.error(
-                f"Agent Generator generation failed: {error_msg} (type: {error_type})"
-            )
-            return _create_error_response(error_msg, error_type)
-
-        return data.get("agent_json")
-
-    except httpx.HTTPStatusError as e:
-        error_type, error_msg = _classify_http_error(e)
-        logger.error(error_msg)
-        return _create_error_response(error_msg, error_type)
-    except httpx.RequestError as e:
-        error_type, error_msg = _classify_request_error(e)
-        logger.error(error_msg)
-        return _create_error_response(error_msg, error_type)
-    except Exception as e:
-        error_msg = f"Unexpected error calling Agent Generator: {e}"
-        logger.error(error_msg)
-        return _create_error_response(error_msg, "unexpected_error")
-
-
-async def generate_agent_patch_external(
-    update_request: str,
-    current_agent: dict[str, Any],
-    library_agents: list[dict[str, Any]] | None = None,
-    operation_id: str | None = None,
-    task_id: str | None = None,
-) -> dict[str, Any] | None:
-    """Call the external service to generate a patch for an existing agent.
-
-    Args:
-        update_request: Natural language description of changes
-        current_agent: Current agent JSON
-        library_agents: User's library agents available for sub-agent composition
-        operation_id: Operation ID for async processing (enables Redis Streams callback)
-        task_id: Task ID for async processing (enables Redis Streams callback)
-
-    Returns:
-        Updated agent JSON, clarifying questions dict, {"status": "accepted"} for async, or error dict on error
-    """
-    if _is_dummy_mode():
-        return await generate_agent_patch_dummy(
-            update_request, current_agent, library_agents, operation_id, task_id
-        )
-
-    client = _get_client()
-
-    # Build request payload
-    payload: dict[str, Any] = {
-        "update_request": update_request,
-        "current_agent_json": current_agent,
-    }
-    if library_agents:
-        payload["library_agents"] = library_agents
-    if operation_id and task_id:
-        payload["operation_id"] = operation_id
-        payload["task_id"] = task_id
-
-    try:
-        response = await client.post("/api/update-agent", json=payload)
-
-        # Handle 202 Accepted for async processing
-        if response.status_code == 202:
-            logger.info(
-                f"Agent Generator accepted async update request "
-                f"(operation_id={operation_id}, task_id={task_id})"
-            )
-            return {
-                "status": "accepted",
-                "operation_id": operation_id,
-                "task_id": task_id,
-            }
-
-        response.raise_for_status()
-        data = response.json()
-
-        if not data.get("success"):
-            error_msg = data.get("error", "Unknown error from Agent Generator")
-            error_type = data.get("error_type", "unknown")
-            logger.error(
-                f"Agent Generator patch generation failed: {error_msg} "
-                f"(type: {error_type})"
-            )
-            return _create_error_response(error_msg, error_type)
-
-        # Check if it's clarifying questions
-        if data.get("type") == "clarifying_questions":
-            return {
-                "type": "clarifying_questions",
-                "questions": data.get("questions", []),
-            }
-
-        # Check if it's an error passed through
-        if data.get("type") == "error":
-            return _create_error_response(
-                data.get("error", "Unknown error"),
-                data.get("error_type", "unknown"),
-            )
-
-        # Otherwise return the updated agent JSON
-        return data.get("agent_json")
-
-    except httpx.HTTPStatusError as e:
-        error_type, error_msg = _classify_http_error(e)
-        logger.error(error_msg)
-        return _create_error_response(error_msg, error_type)
-    except httpx.RequestError as e:
-        error_type, error_msg = _classify_request_error(e)
-        logger.error(error_msg)
-        return _create_error_response(error_msg, error_type)
-    except Exception as e:
-        error_msg = f"Unexpected error calling Agent Generator: {e}"
-        logger.error(error_msg)
-        return _create_error_response(error_msg, "unexpected_error")
-
-
-async def customize_template_external(
-    template_agent: dict[str, Any],
-    modification_request: str,
-    context: str = "",
-) -> dict[str, Any] | None:
-    """Call the external service to customize a template/marketplace agent.
-
-    Args:
-        template_agent: The template agent JSON to customize
-        modification_request: Natural language description of customizations
-        context: Additional context (e.g., answers to previous questions)
-
-    Returns:
-        Customized agent JSON, clarifying questions dict, or error dict on error
-    """
-    if _is_dummy_mode():
-        return await customize_template_dummy(
-            template_agent, modification_request, context
-        )
-
-    client = _get_client()
-
-    request = modification_request
-    if context:
-        request = f"{modification_request}\n\nAdditional context from user:\n{context}"
-
-    payload: dict[str, Any] = {
-        "template_agent_json": template_agent,
-        "modification_request": request,
-    }
-
-    try:
-        response = await client.post("/api/template-modification", json=payload)
-        response.raise_for_status()
-        data = response.json()
-
-        if not data.get("success"):
-            error_msg = data.get("error", "Unknown error from Agent Generator")
-            error_type = data.get("error_type", "unknown")
-            logger.error(
-                f"Agent Generator template customization failed: {error_msg} "
-                f"(type: {error_type})"
-            )
-            return _create_error_response(error_msg, error_type)
-
-        # Check if it's clarifying questions
-        if data.get("type") == "clarifying_questions":
-            return {
-                "type": "clarifying_questions",
-                "questions": data.get("questions", []),
-            }
-
-        # Check if it's an error passed through
-        if data.get("type") == "error":
-            return _create_error_response(
-                data.get("error", "Unknown error"),
-                data.get("error_type", "unknown"),
-            )
-
-        # Otherwise return the customized agent JSON
-        return data.get("agent_json")
-
-    except httpx.HTTPStatusError as e:
-        error_type, error_msg = _classify_http_error(e)
-        logger.error(error_msg)
-        return _create_error_response(error_msg, error_type)
-    except httpx.RequestError as e:
-        error_type, error_msg = _classify_request_error(e)
-        logger.error(error_msg)
-        return _create_error_response(error_msg, error_type)
-    except Exception as e:
-        error_msg = f"Unexpected error calling Agent Generator: {e}"
-        logger.error(error_msg)
-        return _create_error_response(error_msg, "unexpected_error")
-
-
-async def get_blocks_external() -> list[dict[str, Any]] | None:
-    """Get available blocks from the external service.
-
-    Returns:
-        List of block info dicts or None on error
-    """
-    if _is_dummy_mode():
-        return await get_blocks_dummy()
-
-    client = _get_client()
-
-    try:
-        response = await client.get("/api/blocks")
-        response.raise_for_status()
-        data = response.json()
-
-        if not data.get("success"):
-            logger.error("External service returned error getting blocks")
-            return None
-
-        return data.get("blocks", [])
-
-    except httpx.HTTPStatusError as e:
-        logger.error(f"HTTP error getting blocks from external service: {e}")
-        return None
-    except httpx.RequestError as e:
-        logger.error(f"Request error getting blocks from external service: {e}")
-        return None
-    except Exception as e:
-        logger.error(f"Unexpected error getting blocks from external service: {e}")
-        return None
-
-
-async def health_check() -> bool:
-    """Check if the external service is healthy.
-
-    Returns:
-        True if healthy, False otherwise
-    """
-    if not is_external_service_configured():
-        return False
-
-    if _is_dummy_mode():
-        return await health_check_dummy()
-
-    client = _get_client()
-
-    try:
-        response = await client.get("/health")
-        response.raise_for_status()
-        data = response.json()
-        return data.get("status") == "healthy" and data.get("blocks_loaded", False)
-    except Exception as e:
-        logger.warning(f"External agent generator health check failed: {e}")
-        return False
-
-
-async def close_client() -> None:
-    """Close the HTTP client."""
-    global _client
-    if _client is not None:
-        await _client.aclose()
-        _client = None

autogpt_platform/backend/backend/api/features/chat/tools/base.py

@@ -1,129 +0,0 @@
-"""Base classes and shared utilities for chat tools."""
-
-import logging
-from typing import Any
-
-from openai.types.chat import ChatCompletionToolParam
-
-from backend.api.features.chat.model import ChatSession
-from backend.api.features.chat.response_model import StreamToolOutputAvailable
-
-from .models import ErrorResponse, NeedLoginResponse, ToolResponseBase
-
-logger = logging.getLogger(__name__)
-
-
-class BaseTool:
-    """Base class for all chat tools."""
-
-    @property
-    def name(self) -> str:
-        """Tool name for OpenAI function calling."""
-        raise NotImplementedError
-
-    @property
-    def description(self) -> str:
-        """Tool description for OpenAI."""
-        raise NotImplementedError
-
-    @property
-    def parameters(self) -> dict[str, Any]:
-        """Tool parameters schema for OpenAI."""
-        raise NotImplementedError
-
-    @property
-    def requires_auth(self) -> bool:
-        """Whether this tool requires authentication."""
-        return False
-
-    @property
-    def is_long_running(self) -> bool:
-        """Whether this tool is long-running and should execute in background.
-
-        Long-running tools (like agent generation) are executed via background
-        tasks to survive SSE disconnections. The result is persisted to chat
-        history and visible when the user refreshes.
-        """
-        return False
-
-    def as_openai_tool(self) -> ChatCompletionToolParam:
-        """Convert to OpenAI tool format."""
-        return ChatCompletionToolParam(
-            type="function",
-            function={
-                "name": self.name,
-                "description": self.description,
-                "parameters": self.parameters,
-            },
-        )
-
-    async def execute(
-        self,
-        user_id: str | None,
-        session: ChatSession,
-        tool_call_id: str,
-        **kwargs,
-    ) -> StreamToolOutputAvailable:
-        """Execute the tool with authentication check.
-
-        Args:
-            user_id: User ID (may be anonymous like "anon_123")
-            session_id: Chat session ID
-            **kwargs: Tool-specific parameters
-
-        Returns:
-            Pydantic response object
-
-        """
-        if self.requires_auth and not user_id:
-            logger.error(
-                f"Attempted tool call for {self.name} but user not authenticated"
-            )
-            return StreamToolOutputAvailable(
-                toolCallId=tool_call_id,
-                toolName=self.name,
-                output=NeedLoginResponse(
-                    message=f"Please sign in to use {self.name}",
-                    session_id=session.session_id,
-                ).model_dump_json(),
-                success=False,
-            )
-
-        try:
-            result = await self._execute(user_id, session, **kwargs)
-            return StreamToolOutputAvailable(
-                toolCallId=tool_call_id,
-                toolName=self.name,
-                output=result.model_dump_json(),
-            )
-        except Exception as e:
-            logger.error(f"Error in {self.name}: {e}", exc_info=True)
-            return StreamToolOutputAvailable(
-                toolCallId=tool_call_id,
-                toolName=self.name,
-                output=ErrorResponse(
-                    message=f"An error occurred while executing {self.name}",
-                    error=str(e),
-                    session_id=session.session_id,
-                ).model_dump_json(),
-                success=False,
-            )
-
-    async def _execute(
-        self,
-        user_id: str | None,
-        session: ChatSession,
-        **kwargs,
-    ) -> ToolResponseBase:
-        """Internal execution logic to be implemented by subclasses.
-
-        Args:
-            user_id: User ID (authenticated or anonymous)
-            session_id: Chat session ID
-            **kwargs: Tool-specific parameters
-
-        Returns:
-            Pydantic response object
-
-        """
-        raise NotImplementedError

autogpt_platform/backend/backend/api/features/chat/tools/bash_exec.py

@@ -1,131 +0,0 @@
-"""Bash execution tool — run shell commands in a bubblewrap sandbox.
-
-Full Bash scripting is allowed (loops, conditionals, pipes, functions, etc.).
-Safety comes from OS-level isolation (bubblewrap): only system dirs visible
-read-only, writable workspace only, clean env, no network.
-
-Requires bubblewrap (``bwrap``) — the tool is disabled when bwrap is not
-available (e.g. macOS development).
-"""
-
-import logging
-from typing import Any
-
-from backend.api.features.chat.model import ChatSession
-from backend.api.features.chat.tools.base import BaseTool
-from backend.api.features.chat.tools.models import (
-    BashExecResponse,
-    ErrorResponse,
-    ToolResponseBase,
-)
-from backend.api.features.chat.tools.sandbox import (
-    get_workspace_dir,
-    has_full_sandbox,
-    run_sandboxed,
-)
-
-logger = logging.getLogger(__name__)
-
-
-class BashExecTool(BaseTool):
-    """Execute Bash commands in a bubblewrap sandbox."""
-
-    @property
-    def name(self) -> str:
-        return "bash_exec"
-
-    @property
-    def description(self) -> str:
-        if not has_full_sandbox():
-            return (
-                "Bash execution is DISABLED — bubblewrap sandbox is not "
-                "available on this platform. Do not call this tool."
-            )
-        return (
-            "Execute a Bash command or script in a bubblewrap sandbox. "
-            "Full Bash scripting is supported (loops, conditionals, pipes, "
-            "functions, etc.). "
-            "The sandbox shares the same working directory as the SDK Read/Write "
-            "tools — files created by either are accessible to both. "
-            "SECURITY: Only system directories (/usr, /bin, /lib, /etc) are "
-            "visible read-only, the per-session workspace is the only writable "
-            "path, environment variables are wiped (no secrets), all network "
-            "access is blocked at the kernel level, and resource limits are "
-            "enforced (max 64 processes, 512MB memory, 50MB file size). "
-            "Application code, configs, and other directories are NOT accessible. "
-            "To fetch web content, use the web_fetch tool instead. "
-            "Execution is killed after the timeout (default 30s, max 120s). "
-            "Returns stdout and stderr. "
-            "Useful for file manipulation, data processing with Unix tools "
-            "(grep, awk, sed, jq, etc.), and running shell scripts."
-        )
-
-    @property
-    def parameters(self) -> dict[str, Any]:
-        return {
-            "type": "object",
-            "properties": {
-                "command": {
-                    "type": "string",
-                    "description": "Bash command or script to execute.",
-                },
-                "timeout": {
-                    "type": "integer",
-                    "description": (
-                        "Max execution time in seconds (default 30, max 120)."
-                    ),
-                    "default": 30,
-                },
-            },
-            "required": ["command"],
-        }
-
-    @property
-    def requires_auth(self) -> bool:
-        return False
-
-    async def _execute(
-        self,
-        user_id: str | None,
-        session: ChatSession,
-        **kwargs: Any,
-    ) -> ToolResponseBase:
-        session_id = session.session_id if session else None
-
-        if not has_full_sandbox():
-            return ErrorResponse(
-                message="bash_exec requires bubblewrap sandbox (Linux only).",
-                error="sandbox_unavailable",
-                session_id=session_id,
-            )
-
-        command: str = (kwargs.get("command") or "").strip()
-        timeout: int = kwargs.get("timeout", 30)
-
-        if not command:
-            return ErrorResponse(
-                message="No command provided.",
-                error="empty_command",
-                session_id=session_id,
-            )
-
-        workspace = get_workspace_dir(session_id or "default")
-
-        stdout, stderr, exit_code, timed_out = await run_sandboxed(
-            command=["bash", "-c", command],
-            cwd=workspace,
-            timeout=timeout,
-        )
-
-        return BashExecResponse(
-            message=(
-                "Execution timed out"
-                if timed_out
-                else f"Command executed (exit {exit_code})"
-            ),
-            stdout=stdout,
-            stderr=stderr,
-            exit_code=exit_code,
-            timed_out=timed_out,
-            session_id=session_id,
-        )

autogpt_platform/backend/backend/api/features/chat/tools/check_operation_status.py

@@ -1,127 +0,0 @@
-"""CheckOperationStatusTool — query the status of a long-running operation."""
-
-import logging
-from typing import Any
-
-from backend.api.features.chat.model import ChatSession
-from backend.api.features.chat.tools.base import BaseTool
-from backend.api.features.chat.tools.models import (
-    ErrorResponse,
-    ResponseType,
-    ToolResponseBase,
-)
-
-logger = logging.getLogger(__name__)
-
-
-class OperationStatusResponse(ToolResponseBase):
-    """Response for check_operation_status tool."""
-
-    type: ResponseType = ResponseType.OPERATION_STATUS
-    task_id: str
-    operation_id: str
-    status: str  # "running", "completed", "failed"
-    tool_name: str | None = None
-    message: str = ""
-
-
-class CheckOperationStatusTool(BaseTool):
-    """Check the status of a long-running operation (create_agent, edit_agent, etc.).
-
-    The CoPilot uses this tool to report back to the user whether an
-    operation that was started earlier has completed, failed, or is still
-    running.
-    """
-
-    @property
-    def name(self) -> str:
-        return "check_operation_status"
-
-    @property
-    def description(self) -> str:
-        return (
-            "Check the current status of a long-running operation such as "
-            "create_agent or edit_agent. Accepts either an operation_id or "
-            "task_id from a previous operation_started response. "
-            "Returns the current status: running, completed, or failed."
-        )
-
-    @property
-    def parameters(self) -> dict[str, Any]:
-        return {
-            "type": "object",
-            "properties": {
-                "operation_id": {
-                    "type": "string",
-                    "description": (
-                        "The operation_id from an operation_started response."
-                    ),
-                },
-                "task_id": {
-                    "type": "string",
-                    "description": (
-                        "The task_id from an operation_started response. "
-                        "Used as fallback if operation_id is not provided."
-                    ),
-                },
-            },
-            "required": [],
-        }
-
-    @property
-    def requires_auth(self) -> bool:
-        return False
-
-    async def _execute(
-        self,
-        user_id: str | None,
-        session: ChatSession,
-        **kwargs,
-    ) -> ToolResponseBase:
-        from backend.api.features.chat import stream_registry
-
-        operation_id = (kwargs.get("operation_id") or "").strip()
-        task_id = (kwargs.get("task_id") or "").strip()
-
-        if not operation_id and not task_id:
-            return ErrorResponse(
-                message="Please provide an operation_id or task_id.",
-                error="missing_parameter",
-            )
-
-        task = None
-        if operation_id:
-            task = await stream_registry.find_task_by_operation_id(operation_id)
-        if task is None and task_id:
-            task = await stream_registry.get_task(task_id)
-
-        if task is None:
-            # Task not in Redis — it may have already expired (TTL).
-            # Check conversation history for the result instead.
-            return ErrorResponse(
-                message=(
-                    "Operation not found — it may have already completed and "
-                    "expired from the status tracker. Check the conversation "
-                    "history for the result."
-                ),
-                error="not_found",
-            )
-
-        status_messages = {
-            "running": (
-                f"The {task.tool_name or 'operation'} is still running. "
-                "Please wait for it to complete."
-            ),
-            "completed": (
-                f"The {task.tool_name or 'operation'} has completed successfully."
-            ),
-            "failed": f"The {task.tool_name or 'operation'} has failed.",
-        }
-
-        return OperationStatusResponse(
-            task_id=task.task_id,
-            operation_id=task.operation_id,
-            status=task.status,
-            tool_name=task.tool_name,
-            message=status_messages.get(task.status, f"Status: {task.status}"),
-        )

autogpt_platform/backend/backend/api/features/chat/tools/create_agent.py

@@ -1,335 +0,0 @@
-"""CreateAgentTool - Creates agents from natural language descriptions."""
-
-import logging
-from typing import Any
-
-from backend.api.features.chat.model import ChatSession
-
-from .agent_generator import (
-    AgentGeneratorNotConfiguredError,
-    decompose_goal,
-    enrich_library_agents_from_steps,
-    generate_agent,
-    get_all_relevant_agents_for_generation,
-    get_user_message_for_error,
-    save_agent_to_library,
-)
-from .base import BaseTool
-from .models import (
-    AgentPreviewResponse,
-    AgentSavedResponse,
-    AsyncProcessingResponse,
-    ClarificationNeededResponse,
-    ClarifyingQuestion,
-    ErrorResponse,
-    ToolResponseBase,
-)
-
-logger = logging.getLogger(__name__)
-
-
-class CreateAgentTool(BaseTool):
-    """Tool for creating agents from natural language descriptions."""
-
-    @property
-    def name(self) -> str:
-        return "create_agent"
-
-    @property
-    def description(self) -> str:
-        return (
-            "Create a new agent workflow from a natural language description. "
-            "First generates a preview, then saves to library if save=true."
-        )
-
-    @property
-    def requires_auth(self) -> bool:
-        return True
-
-    @property
-    def is_long_running(self) -> bool:
-        return True
-
-    @property
-    def parameters(self) -> dict[str, Any]:
-        return {
-            "type": "object",
-            "properties": {
-                "description": {
-                    "type": "string",
-                    "description": (
-                        "Natural language description of what the agent should do. "
-                        "Be specific about inputs, outputs, and the workflow steps."
-                    ),
-                },
-                "context": {
-                    "type": "string",
-                    "description": (
-                        "Additional context or answers to previous clarifying questions. "
-                        "Include any preferences or constraints mentioned by the user."
-                    ),
-                },
-                "save": {
-                    "type": "boolean",
-                    "description": (
-                        "Whether to save the agent to the user's library. "
-                        "Default is true. Set to false for preview only."
-                    ),
-                    "default": True,
-                },
-            },
-            "required": ["description"],
-        }
-
-    async def _execute(
-        self,
-        user_id: str | None,
-        session: ChatSession,
-        **kwargs,
-    ) -> ToolResponseBase:
-        """Execute the create_agent tool.
-
-        Flow:
-        1. Decompose the description into steps (may return clarifying questions)
-        2. Generate agent JSON (external service handles fixing and validation)
-        3. Preview or save based on the save parameter
-        """
-        description = kwargs.get("description", "").strip()
-        context = kwargs.get("context", "")
-        save = kwargs.get("save", True)
-        session_id = session.session_id if session else None
-
-        # Extract async processing params (passed by long-running tool handler)
-        operation_id = kwargs.get("_operation_id")
-        task_id = kwargs.get("_task_id")
-
-        if not description:
-            return ErrorResponse(
-                message="Please provide a description of what the agent should do.",
-                error="Missing description parameter",
-                session_id=session_id,
-            )
-
-        library_agents = None
-        if user_id:
-            try:
-                library_agents = await get_all_relevant_agents_for_generation(
-                    user_id=user_id,
-                    search_query=description,
-                    include_marketplace=True,
-                )
-                logger.debug(
-                    f"Found {len(library_agents)} relevant agents for sub-agent composition"
-                )
-            except Exception as e:
-                logger.warning(f"Failed to fetch library agents: {e}")
-
-        try:
-            decomposition_result = await decompose_goal(
-                description, context, library_agents
-            )
-        except AgentGeneratorNotConfiguredError:
-            return ErrorResponse(
-                message=(
-                    "Agent generation is not available. "
-                    "The Agent Generator service is not configured."
-                ),
-                error="service_not_configured",
-                session_id=session_id,
-            )
-
-        if decomposition_result is None:
-            return ErrorResponse(
-                message="Failed to analyze the goal. The agent generation service may be unavailable. Please try again.",
-                error="decomposition_failed",
-                details={"description": description[:100]},
-                session_id=session_id,
-            )
-
-        if decomposition_result.get("type") == "error":
-            error_msg = decomposition_result.get("error", "Unknown error")
-            error_type = decomposition_result.get("error_type", "unknown")
-            user_message = get_user_message_for_error(
-                error_type,
-                operation="analyze the goal",
-                llm_parse_message="The AI had trouble understanding this request. Please try rephrasing your goal.",
-            )
-            return ErrorResponse(
-                message=user_message,
-                error=f"decomposition_failed:{error_type}",
-                details={
-                    "description": description[:100],
-                    "service_error": error_msg,
-                    "error_type": error_type,
-                },
-                session_id=session_id,
-            )
-
-        if decomposition_result.get("type") == "clarifying_questions":
-            questions = decomposition_result.get("questions", [])
-            return ClarificationNeededResponse(
-                message=(
-                    "I need some more information to create this agent. "
-                    "Please answer the following questions:"
-                ),
-                questions=[
-                    ClarifyingQuestion(
-                        question=q.get("question", ""),
-                        keyword=q.get("keyword", ""),
-                        example=q.get("example"),
-                    )
-                    for q in questions
-                ],
-                session_id=session_id,
-            )
-
-        if decomposition_result.get("type") == "unachievable_goal":
-            suggested = decomposition_result.get("suggested_goal", "")
-            reason = decomposition_result.get("reason", "")
-            return ErrorResponse(
-                message=(
-                    f"This goal cannot be accomplished with the available blocks. "
-                    f"{reason} "
-                    f"Suggestion: {suggested}"
-                ),
-                error="unachievable_goal",
-                details={"suggested_goal": suggested, "reason": reason},
-                session_id=session_id,
-            )
-
-        if decomposition_result.get("type") == "vague_goal":
-            suggested = decomposition_result.get("suggested_goal", "")
-            return ErrorResponse(
-                message=(
-                    f"The goal is too vague to create a specific workflow. "
-                    f"Suggestion: {suggested}"
-                ),
-                error="vague_goal",
-                details={"suggested_goal": suggested},
-                session_id=session_id,
-            )
-
-        if user_id and library_agents is not None:
-            try:
-                library_agents = await enrich_library_agents_from_steps(
-                    user_id=user_id,
-                    decomposition_result=decomposition_result,
-                    existing_agents=library_agents,
-                    include_marketplace=True,
-                )
-                logger.debug(
-                    f"After enrichment: {len(library_agents)} total agents for sub-agent composition"
-                )
-            except Exception as e:
-                logger.warning(f"Failed to enrich library agents from steps: {e}")
-
-        try:
-            agent_json = await generate_agent(
-                decomposition_result,
-                library_agents,
-                operation_id=operation_id,
-                task_id=task_id,
-            )
-        except AgentGeneratorNotConfiguredError:
-            return ErrorResponse(
-                message=(
-                    "Agent generation is not available. "
-                    "The Agent Generator service is not configured."
-                ),
-                error="service_not_configured",
-                session_id=session_id,
-            )
-
-        if agent_json is None:
-            return ErrorResponse(
-                message="Failed to generate the agent. The agent generation service may be unavailable. Please try again.",
-                error="generation_failed",
-                details={"description": description[:100]},
-                session_id=session_id,
-            )
-
-        if isinstance(agent_json, dict) and agent_json.get("type") == "error":
-            error_msg = agent_json.get("error", "Unknown error")
-            error_type = agent_json.get("error_type", "unknown")
-            user_message = get_user_message_for_error(
-                error_type,
-                operation="generate the agent",
-                llm_parse_message="The AI had trouble generating the agent. Please try again or simplify your goal.",
-                validation_message=(
-                    "I wasn't able to create a valid agent for this request. "
-                    "The generated workflow had some structural issues. "
-                    "Please try simplifying your goal or breaking it into smaller steps."
-                ),
-                error_details=error_msg,
-            )
-            return ErrorResponse(
-                message=user_message,
-                error=f"generation_failed:{error_type}",
-                details={
-                    "description": description[:100],
-                    "service_error": error_msg,
-                    "error_type": error_type,
-                },
-                session_id=session_id,
-            )
-
-        # Check if Agent Generator accepted for async processing
-        if agent_json.get("status") == "accepted":
-            logger.info(
-                f"Agent generation delegated to async processing "
-                f"(operation_id={operation_id}, task_id={task_id})"
-            )
-            return AsyncProcessingResponse(
-                message="Agent generation started. You'll be notified when it's complete.",
-                operation_id=operation_id,
-                task_id=task_id,
-                session_id=session_id,
-            )
-
-        agent_name = agent_json.get("name", "Generated Agent")
-        agent_description = agent_json.get("description", "")
-        node_count = len(agent_json.get("nodes", []))
-        link_count = len(agent_json.get("links", []))
-
-        if not save:
-            return AgentPreviewResponse(
-                message=(
-                    f"I've generated an agent called '{agent_name}' with {node_count} blocks. "
-                    f"Review it and call create_agent with save=true to save it to your library."
-                ),
-                agent_json=agent_json,
-                agent_name=agent_name,
-                description=agent_description,
-                node_count=node_count,
-                link_count=link_count,
-                session_id=session_id,
-            )
-
-        if not user_id:
-            return ErrorResponse(
-                message="You must be logged in to save agents.",
-                error="auth_required",
-                session_id=session_id,
-            )
-
-        try:
-            created_graph, library_agent = await save_agent_to_library(
-                agent_json, user_id
-            )
-
-            return AgentSavedResponse(
-                message=f"Agent '{created_graph.name}' has been saved to your library!",
-                agent_id=created_graph.id,
-                agent_name=created_graph.name,
-                library_agent_id=library_agent.id,
-                library_agent_link=f"/library/agents/{library_agent.id}",
-                agent_page_link=f"/build?flowID={created_graph.id}",
-                session_id=session_id,
-            )
-        except Exception as e:
-            return ErrorResponse(
-                message=f"Failed to save the agent: {str(e)}",
-                error="save_failed",
-                details={"exception": str(e)},
-                session_id=session_id,
-            )

autogpt_platform/backend/backend/api/features/chat/tools/customize_agent.py

@@ -1,337 +0,0 @@
-"""CustomizeAgentTool - Customizes marketplace/template agents using natural language."""
-
-import logging
-from typing import Any
-
-from backend.api.features.chat.model import ChatSession
-from backend.api.features.store import db as store_db
-from backend.api.features.store.exceptions import AgentNotFoundError
-
-from .agent_generator import (
-    AgentGeneratorNotConfiguredError,
-    customize_template,
-    get_user_message_for_error,
-    graph_to_json,
-    save_agent_to_library,
-)
-from .base import BaseTool
-from .models import (
-    AgentPreviewResponse,
-    AgentSavedResponse,
-    ClarificationNeededResponse,
-    ClarifyingQuestion,
-    ErrorResponse,
-    ToolResponseBase,
-)
-
-logger = logging.getLogger(__name__)
-
-
-class CustomizeAgentTool(BaseTool):
-    """Tool for customizing marketplace/template agents using natural language."""
-
-    @property
-    def name(self) -> str:
-        return "customize_agent"
-
-    @property
-    def description(self) -> str:
-        return (
-            "Customize a marketplace or template agent using natural language. "
-            "Takes an existing agent from the marketplace and modifies it based on "
-            "the user's requirements before adding to their library."
-        )
-
-    @property
-    def requires_auth(self) -> bool:
-        return True
-
-    @property
-    def is_long_running(self) -> bool:
-        return True
-
-    @property
-    def parameters(self) -> dict[str, Any]:
-        return {
-            "type": "object",
-            "properties": {
-                "agent_id": {
-                    "type": "string",
-                    "description": (
-                        "The marketplace agent ID in format 'creator/slug' "
-                        "(e.g., 'autogpt/newsletter-writer'). "
-                        "Get this from find_agent results."
-                    ),
-                },
-                "modifications": {
-                    "type": "string",
-                    "description": (
-                        "Natural language description of how to customize the agent. "
-                        "Be specific about what changes you want to make."
-                    ),
-                },
-                "context": {
-                    "type": "string",
-                    "description": (
-                        "Additional context or answers to previous clarifying questions."
-                    ),
-                },
-                "save": {
-                    "type": "boolean",
-                    "description": (
-                        "Whether to save the customized agent to the user's library. "
-                        "Default is true. Set to false for preview only."
-                    ),
-                    "default": True,
-                },
-            },
-            "required": ["agent_id", "modifications"],
-        }
-
-    async def _execute(
-        self,
-        user_id: str | None,
-        session: ChatSession,
-        **kwargs,
-    ) -> ToolResponseBase:
-        """Execute the customize_agent tool.
-
-        Flow:
-        1. Parse the agent ID to get creator/slug
-        2. Fetch the template agent from the marketplace
-        3. Call customize_template with the modification request
-        4. Preview or save based on the save parameter
-        """
-        agent_id = kwargs.get("agent_id", "").strip()
-        modifications = kwargs.get("modifications", "").strip()
-        context = kwargs.get("context", "")
-        save = kwargs.get("save", True)
-        session_id = session.session_id if session else None
-
-        if not agent_id:
-            return ErrorResponse(
-                message="Please provide the marketplace agent ID (e.g., 'creator/agent-name').",
-                error="missing_agent_id",
-                session_id=session_id,
-            )
-
-        if not modifications:
-            return ErrorResponse(
-                message="Please describe how you want to customize this agent.",
-                error="missing_modifications",
-                session_id=session_id,
-            )
-
-        # Parse agent_id in format "creator/slug"
-        parts = [p.strip() for p in agent_id.split("/")]
-        if len(parts) != 2 or not parts[0] or not parts[1]:
-            return ErrorResponse(
-                message=(
-                    f"Invalid agent ID format: '{agent_id}'. "
-                    "Expected format is 'creator/agent-name' "
-                    "(e.g., 'autogpt/newsletter-writer')."
-                ),
-                error="invalid_agent_id_format",
-                session_id=session_id,
-            )
-
-        creator_username, agent_slug = parts
-
-        # Fetch the marketplace agent details
-        try:
-            agent_details = await store_db.get_store_agent_details(
-                username=creator_username, agent_name=agent_slug
-            )
-        except AgentNotFoundError:
-            return ErrorResponse(
-                message=(
-                    f"Could not find marketplace agent '{agent_id}'. "
-                    "Please check the agent ID and try again."
-                ),
-                error="agent_not_found",
-                session_id=session_id,
-            )
-        except Exception as e:
-            logger.error(f"Error fetching marketplace agent {agent_id}: {e}")
-            return ErrorResponse(
-                message="Failed to fetch the marketplace agent. Please try again.",
-                error="fetch_error",
-                session_id=session_id,
-            )
-
-        if not agent_details.store_listing_version_id:
-            return ErrorResponse(
-                message=(
-                    f"The agent '{agent_id}' does not have an available version. "
-                    "Please try a different agent."
-                ),
-                error="no_version_available",
-                session_id=session_id,
-            )
-
-        # Get the full agent graph
-        try:
-            graph = await store_db.get_agent(agent_details.store_listing_version_id)
-            template_agent = graph_to_json(graph)
-        except Exception as e:
-            logger.error(f"Error fetching agent graph for {agent_id}: {e}")
-            return ErrorResponse(
-                message="Failed to fetch the agent configuration. Please try again.",
-                error="graph_fetch_error",
-                session_id=session_id,
-            )
-
-        # Call customize_template
-        try:
-            result = await customize_template(
-                template_agent=template_agent,
-                modification_request=modifications,
-                context=context,
-            )
-        except AgentGeneratorNotConfiguredError:
-            return ErrorResponse(
-                message=(
-                    "Agent customization is not available. "
-                    "The Agent Generator service is not configured."
-                ),
-                error="service_not_configured",
-                session_id=session_id,
-            )
-        except Exception as e:
-            logger.error(f"Error calling customize_template for {agent_id}: {e}")
-            return ErrorResponse(
-                message=(
-                    "Failed to customize the agent due to a service error. "
-                    "Please try again."
-                ),
-                error="customization_service_error",
-                session_id=session_id,
-            )
-
-        if result is None:
-            return ErrorResponse(
-                message=(
-                    "Failed to customize the agent. "
-                    "The agent generation service may be unavailable or timed out. "
-                    "Please try again."
-                ),
-                error="customization_failed",
-                session_id=session_id,
-            )
-
-        # Handle error response
-        if isinstance(result, dict) and result.get("type") == "error":
-            error_msg = result.get("error", "Unknown error")
-            error_type = result.get("error_type", "unknown")
-            user_message = get_user_message_for_error(
-                error_type,
-                operation="customize the agent",
-                llm_parse_message=(
-                    "The AI had trouble customizing the agent. "
-                    "Please try again or simplify your request."
-                ),
-                validation_message=(
-                    "The customized agent failed validation. "
-                    "Please try rephrasing your request."
-                ),
-                error_details=error_msg,
-            )
-            return ErrorResponse(
-                message=user_message,
-                error=f"customization_failed:{error_type}",
-                session_id=session_id,
-            )
-
-        # Handle clarifying questions
-        if isinstance(result, dict) and result.get("type") == "clarifying_questions":
-            questions = result.get("questions") or []
-            if not isinstance(questions, list):
-                logger.error(
-                    f"Unexpected clarifying questions format: {type(questions)}"
-                )
-                questions = []
-            return ClarificationNeededResponse(
-                message=(
-                    "I need some more information to customize this agent. "
-                    "Please answer the following questions:"
-                ),
-                questions=[
-                    ClarifyingQuestion(
-                        question=q.get("question", ""),
-                        keyword=q.get("keyword", ""),
-                        example=q.get("example"),
-                    )
-                    for q in questions
-                    if isinstance(q, dict)
-                ],
-                session_id=session_id,
-            )
-
-        # Result should be the customized agent JSON
-        if not isinstance(result, dict):
-            logger.error(f"Unexpected customize_template response type: {type(result)}")
-            return ErrorResponse(
-                message="Failed to customize the agent due to an unexpected response.",
-                error="unexpected_response_type",
-                session_id=session_id,
-            )
-
-        customized_agent = result
-
-        agent_name = customized_agent.get(
-            "name", f"Customized {agent_details.agent_name}"
-        )
-        agent_description = customized_agent.get("description", "")
-        nodes = customized_agent.get("nodes")
-        links = customized_agent.get("links")
-        node_count = len(nodes) if isinstance(nodes, list) else 0
-        link_count = len(links) if isinstance(links, list) else 0
-
-        if not save:
-            return AgentPreviewResponse(
-                message=(
-                    f"I've customized the agent '{agent_details.agent_name}'. "
-                    f"The customized agent has {node_count} blocks. "
-                    f"Review it and call customize_agent with save=true to save it."
-                ),
-                agent_json=customized_agent,
-                agent_name=agent_name,
-                description=agent_description,
-                node_count=node_count,
-                link_count=link_count,
-                session_id=session_id,
-            )
-
-        if not user_id:
-            return ErrorResponse(
-                message="You must be logged in to save agents.",
-                error="auth_required",
-                session_id=session_id,
-            )
-
-        # Save to user's library
-        try:
-            created_graph, library_agent = await save_agent_to_library(
-                customized_agent, user_id, is_update=False
-            )
-
-            return AgentSavedResponse(
-                message=(
-                    f"Customized agent '{created_graph.name}' "
-                    f"(based on '{agent_details.agent_name}') "
-                    f"has been saved to your library!"
-                ),
-                agent_id=created_graph.id,
-                agent_name=created_graph.name,
-                library_agent_id=library_agent.id,
-                library_agent_link=f"/library/agents/{library_agent.id}",
-                agent_page_link=f"/build?flowID={created_graph.id}",
-                session_id=session_id,
-            )
-        except Exception as e:
-            logger.error(f"Error saving customized agent: {e}")
-            return ErrorResponse(
-                message="Failed to save the customized agent. Please try again.",
-                error="save_failed",
-                session_id=session_id,
-            )

autogpt_platform/backend/backend/api/features/chat/tools/edit_agent.py

@@ -1,284 +0,0 @@
-"""EditAgentTool - Edits existing agents using natural language."""
-
-import logging
-from typing import Any
-
-from backend.api.features.chat.model import ChatSession
-
-from .agent_generator import (
-    AgentGeneratorNotConfiguredError,
-    generate_agent_patch,
-    get_agent_as_json,
-    get_all_relevant_agents_for_generation,
-    get_user_message_for_error,
-    save_agent_to_library,
-)
-from .base import BaseTool
-from .models import (
-    AgentPreviewResponse,
-    AgentSavedResponse,
-    AsyncProcessingResponse,
-    ClarificationNeededResponse,
-    ClarifyingQuestion,
-    ErrorResponse,
-    ToolResponseBase,
-)
-
-logger = logging.getLogger(__name__)
-
-
-class EditAgentTool(BaseTool):
-    """Tool for editing existing agents using natural language."""
-
-    @property
-    def name(self) -> str:
-        return "edit_agent"
-
-    @property
-    def description(self) -> str:
-        return (
-            "Edit an existing agent from the user's library using natural language. "
-            "Generates updates to the agent while preserving unchanged parts."
-        )
-
-    @property
-    def requires_auth(self) -> bool:
-        return True
-
-    @property
-    def is_long_running(self) -> bool:
-        return True
-
-    @property
-    def parameters(self) -> dict[str, Any]:
-        return {
-            "type": "object",
-            "properties": {
-                "agent_id": {
-                    "type": "string",
-                    "description": (
-                        "The ID of the agent to edit. "
-                        "Can be a graph ID or library agent ID."
-                    ),
-                },
-                "changes": {
-                    "type": "string",
-                    "description": (
-                        "Natural language description of what changes to make. "
-                        "Be specific about what to add, remove, or modify."
-                    ),
-                },
-                "context": {
-                    "type": "string",
-                    "description": (
-                        "Additional context or answers to previous clarifying questions."
-                    ),
-                },
-                "save": {
-                    "type": "boolean",
-                    "description": (
-                        "Whether to save the changes. "
-                        "Default is true. Set to false for preview only."
-                    ),
-                    "default": True,
-                },
-            },
-            "required": ["agent_id", "changes"],
-        }
-
-    async def _execute(
-        self,
-        user_id: str | None,
-        session: ChatSession,
-        **kwargs,
-    ) -> ToolResponseBase:
-        """Execute the edit_agent tool.
-
-        Flow:
-        1. Fetch the current agent
-        2. Generate updated agent (external service handles fixing and validation)
-        3. Preview or save based on the save parameter
-        """
-        agent_id = kwargs.get("agent_id", "").strip()
-        changes = kwargs.get("changes", "").strip()
-        context = kwargs.get("context", "")
-        save = kwargs.get("save", True)
-        session_id = session.session_id if session else None
-
-        # Extract async processing params (passed by long-running tool handler)
-        operation_id = kwargs.get("_operation_id")
-        task_id = kwargs.get("_task_id")
-
-        if not agent_id:
-            return ErrorResponse(
-                message="Please provide the agent ID to edit.",
-                error="Missing agent_id parameter",
-                session_id=session_id,
-            )
-
-        if not changes:
-            return ErrorResponse(
-                message="Please describe what changes you want to make.",
-                error="Missing changes parameter",
-                session_id=session_id,
-            )
-
-        current_agent = await get_agent_as_json(agent_id, user_id)
-
-        if current_agent is None:
-            return ErrorResponse(
-                message=f"Could not find agent with ID '{agent_id}' in your library.",
-                error="agent_not_found",
-                session_id=session_id,
-            )
-
-        library_agents = None
-        if user_id:
-            try:
-                graph_id = current_agent.get("id")
-                library_agents = await get_all_relevant_agents_for_generation(
-                    user_id=user_id,
-                    search_query=changes,
-                    exclude_graph_id=graph_id,
-                    include_marketplace=True,
-                )
-                logger.debug(
-                    f"Found {len(library_agents)} relevant agents for sub-agent composition"
-                )
-            except Exception as e:
-                logger.warning(f"Failed to fetch library agents: {e}")
-
-        update_request = changes
-        if context:
-            update_request = f"{changes}\n\nAdditional context:\n{context}"
-
-        try:
-            result = await generate_agent_patch(
-                update_request,
-                current_agent,
-                library_agents,
-                operation_id=operation_id,
-                task_id=task_id,
-            )
-        except AgentGeneratorNotConfiguredError:
-            return ErrorResponse(
-                message=(
-                    "Agent editing is not available. "
-                    "The Agent Generator service is not configured."
-                ),
-                error="service_not_configured",
-                session_id=session_id,
-            )
-
-        if result is None:
-            return ErrorResponse(
-                message="Failed to generate changes. The agent generation service may be unavailable or timed out. Please try again.",
-                error="update_generation_failed",
-                details={"agent_id": agent_id, "changes": changes[:100]},
-                session_id=session_id,
-            )
-
-        # Check if Agent Generator accepted for async processing
-        if result.get("status") == "accepted":
-            logger.info(
-                f"Agent edit delegated to async processing "
-                f"(operation_id={operation_id}, task_id={task_id})"
-            )
-            return AsyncProcessingResponse(
-                message="Agent edit started. You'll be notified when it's complete.",
-                operation_id=operation_id,
-                task_id=task_id,
-                session_id=session_id,
-            )
-
-        # Check if the result is an error from the external service
-        if isinstance(result, dict) and result.get("type") == "error":
-            error_msg = result.get("error", "Unknown error")
-            error_type = result.get("error_type", "unknown")
-            user_message = get_user_message_for_error(
-                error_type,
-                operation="generate the changes",
-                llm_parse_message="The AI had trouble generating the changes. Please try again or simplify your request.",
-                validation_message="The generated changes failed validation. Please try rephrasing your request.",
-                error_details=error_msg,
-            )
-            return ErrorResponse(
-                message=user_message,
-                error=f"update_generation_failed:{error_type}",
-                details={
-                    "agent_id": agent_id,
-                    "changes": changes[:100],
-                    "service_error": error_msg,
-                    "error_type": error_type,
-                },
-                session_id=session_id,
-            )
-
-        if result.get("type") == "clarifying_questions":
-            questions = result.get("questions", [])
-            return ClarificationNeededResponse(
-                message=(
-                    "I need some more information about the changes. "
-                    "Please answer the following questions:"
-                ),
-                questions=[
-                    ClarifyingQuestion(
-                        question=q.get("question", ""),
-                        keyword=q.get("keyword", ""),
-                        example=q.get("example"),
-                    )
-                    for q in questions
-                ],
-                session_id=session_id,
-            )
-
-        updated_agent = result
-
-        agent_name = updated_agent.get("name", "Updated Agent")
-        agent_description = updated_agent.get("description", "")
-        node_count = len(updated_agent.get("nodes", []))
-        link_count = len(updated_agent.get("links", []))
-
-        if not save:
-            return AgentPreviewResponse(
-                message=(
-                    f"I've updated the agent. "
-                    f"The agent now has {node_count} blocks. "
-                    f"Review it and call edit_agent with save=true to save the changes."
-                ),
-                agent_json=updated_agent,
-                agent_name=agent_name,
-                description=agent_description,
-                node_count=node_count,
-                link_count=link_count,
-                session_id=session_id,
-            )
-
-        if not user_id:
-            return ErrorResponse(
-                message="You must be logged in to save agents.",
-                error="auth_required",
-                session_id=session_id,
-            )
-
-        try:
-            created_graph, library_agent = await save_agent_to_library(
-                updated_agent, user_id, is_update=True
-            )
-
-            return AgentSavedResponse(
-                message=f"Updated agent '{created_graph.name}' has been saved to your library!",
-                agent_id=created_graph.id,
-                agent_name=created_graph.name,
-                library_agent_id=library_agent.id,
-                library_agent_link=f"/library/agents/{library_agent.id}",
-                agent_page_link=f"/build?flowID={created_graph.id}",
-                session_id=session_id,
-            )
-        except Exception as e:
-            return ErrorResponse(
-                message=f"Failed to save the updated agent: {str(e)}",
-                error="save_failed",
-                details={"exception": str(e)},
-                session_id=session_id,
-            )

autogpt_platform/backend/backend/api/features/chat/tools/helpers.py

@@ -1,29 +0,0 @@
-"""Shared helpers for chat tools."""
-
-from typing import Any
-
-
-def get_inputs_from_schema(
-    input_schema: dict[str, Any],
-    exclude_fields: set[str] | None = None,
-) -> list[dict[str, Any]]:
-    """Extract input field info from JSON schema."""
-    if not isinstance(input_schema, dict):
-        return []
-
-    exclude = exclude_fields or set()
-    properties = input_schema.get("properties", {})
-    required = set(input_schema.get("required", []))
-
-    return [
-        {
-            "name": name,
-            "title": schema.get("title", name),
-            "type": schema.get("type", "string"),
-            "description": schema.get("description", ""),
-            "required": name in required,
-            "default": schema.get("default"),
-        }
-        for name, schema in properties.items()
-        if name not in exclude
-    ]

autogpt_platform/backend/backend/api/features/chat/tools/workspace_files.py

@@ -1,626 +0,0 @@
-"""CoPilot tools for workspace file operations."""
-
-import base64
-import logging
-from typing import Any, Optional
-
-from pydantic import BaseModel
-
-from backend.api.features.chat.model import ChatSession
-from backend.data.workspace import get_or_create_workspace
-from backend.util.settings import Config
-from backend.util.virus_scanner import scan_content_safe
-from backend.util.workspace import WorkspaceManager
-
-from .base import BaseTool
-from .models import ErrorResponse, ResponseType, ToolResponseBase
-
-logger = logging.getLogger(__name__)
-
-
-class WorkspaceFileInfoData(BaseModel):
-    """Data model for workspace file information (not a response itself)."""
-
-    file_id: str
-    name: str
-    path: str
-    mime_type: str
-    size_bytes: int
-
-
-class WorkspaceFileListResponse(ToolResponseBase):
-    """Response containing list of workspace files."""
-
-    type: ResponseType = ResponseType.WORKSPACE_FILE_LIST
-    files: list[WorkspaceFileInfoData]
-    total_count: int
-
-
-class WorkspaceFileContentResponse(ToolResponseBase):
-    """Response containing workspace file content (legacy, for small text files)."""
-
-    type: ResponseType = ResponseType.WORKSPACE_FILE_CONTENT
-    file_id: str
-    name: str
-    path: str
-    mime_type: str
-    content_base64: str
-
-
-class WorkspaceFileMetadataResponse(ToolResponseBase):
-    """Response containing workspace file metadata and download URL (prevents context bloat)."""
-
-    type: ResponseType = ResponseType.WORKSPACE_FILE_METADATA
-    file_id: str
-    name: str
-    path: str
-    mime_type: str
-    size_bytes: int
-    download_url: str
-    preview: str | None = None  # First 500 chars for text files
-
-
-class WorkspaceWriteResponse(ToolResponseBase):
-    """Response after writing a file to workspace."""
-
-    type: ResponseType = ResponseType.WORKSPACE_FILE_WRITTEN
-    file_id: str
-    name: str
-    path: str
-    size_bytes: int
-
-
-class WorkspaceDeleteResponse(ToolResponseBase):
-    """Response after deleting a file from workspace."""
-
-    type: ResponseType = ResponseType.WORKSPACE_FILE_DELETED
-    file_id: str
-    success: bool
-
-
-class ListWorkspaceFilesTool(BaseTool):
-    """Tool for listing files in user's workspace."""
-
-    @property
-    def name(self) -> str:
-        return "list_workspace_files"
-
-    @property
-    def description(self) -> str:
-        return (
-            "List files in the user's persistent workspace (cloud storage). "
-            "These files survive across sessions. "
-            "For ephemeral session files, use the SDK Read/Glob tools instead. "
-            "Returns file names, paths, sizes, and metadata. "
-            "Optionally filter by path prefix."
-        )
-
-    @property
-    def parameters(self) -> dict[str, Any]:
-        return {
-            "type": "object",
-            "properties": {
-                "path_prefix": {
-                    "type": "string",
-                    "description": (
-                        "Optional path prefix to filter files "
-                        "(e.g., '/documents/' to list only files in documents folder). "
-                        "By default, only files from the current session are listed."
-                    ),
-                },
-                "limit": {
-                    "type": "integer",
-                    "description": "Maximum number of files to return (default 50, max 100)",
-                    "minimum": 1,
-                    "maximum": 100,
-                },
-                "include_all_sessions": {
-                    "type": "boolean",
-                    "description": (
-                        "If true, list files from all sessions. "
-                        "Default is false (only current session's files)."
-                    ),
-                },
-            },
-            "required": [],
-        }
-
-    @property
-    def requires_auth(self) -> bool:
-        return True
-
-    async def _execute(
-        self,
-        user_id: str | None,
-        session: ChatSession,
-        **kwargs,
-    ) -> ToolResponseBase:
-        session_id = session.session_id
-
-        if not user_id:
-            return ErrorResponse(
-                message="Authentication required",
-                session_id=session_id,
-            )
-
-        path_prefix: Optional[str] = kwargs.get("path_prefix")
-        limit = min(kwargs.get("limit", 50), 100)
-        include_all_sessions: bool = kwargs.get("include_all_sessions", False)
-
-        try:
-            workspace = await get_or_create_workspace(user_id)
-            # Pass session_id for session-scoped file access
-            manager = WorkspaceManager(user_id, workspace.id, session_id)
-
-            files = await manager.list_files(
-                path=path_prefix,
-                limit=limit,
-                include_all_sessions=include_all_sessions,
-            )
-            total = await manager.get_file_count(
-                path=path_prefix,
-                include_all_sessions=include_all_sessions,
-            )
-
-            file_infos = [
-                WorkspaceFileInfoData(
-                    file_id=f.id,
-                    name=f.name,
-                    path=f.path,
-                    mime_type=f.mimeType,
-                    size_bytes=f.sizeBytes,
-                )
-                for f in files
-            ]
-
-            scope_msg = "all sessions" if include_all_sessions else "current session"
-            return WorkspaceFileListResponse(
-                files=file_infos,
-                total_count=total,
-                message=f"Found {len(files)} files in workspace ({scope_msg})",
-                session_id=session_id,
-            )
-
-        except Exception as e:
-            logger.error(f"Error listing workspace files: {e}", exc_info=True)
-            return ErrorResponse(
-                message=f"Failed to list workspace files: {str(e)}",
-                error=str(e),
-                session_id=session_id,
-            )
-
-
-class ReadWorkspaceFileTool(BaseTool):
-    """Tool for reading file content from workspace."""
-
-    # Size threshold for returning full content vs metadata+URL
-    # Files larger than this return metadata with download URL to prevent context bloat
-    MAX_INLINE_SIZE_BYTES = 32 * 1024  # 32KB
-    # Preview size for text files
-    PREVIEW_SIZE = 500
-
-    @property
-    def name(self) -> str:
-        return "read_workspace_file"
-
-    @property
-    def description(self) -> str:
-        return (
-            "Read a file from the user's persistent workspace (cloud storage). "
-            "These files survive across sessions. "
-            "For ephemeral session files, use the SDK Read tool instead. "
-            "Specify either file_id or path to identify the file. "
-            "For small text files, returns content directly. "
-            "For large or binary files, returns metadata and a download URL. "
-            "Paths are scoped to the current session by default. "
-            "Use /sessions/<session_id>/... for cross-session access."
-        )
-
-    @property
-    def parameters(self) -> dict[str, Any]:
-        return {
-            "type": "object",
-            "properties": {
-                "file_id": {
-                    "type": "string",
-                    "description": "The file's unique ID (from list_workspace_files)",
-                },
-                "path": {
-                    "type": "string",
-                    "description": (
-                        "The virtual file path (e.g., '/documents/report.pdf'). "
-                        "Scoped to current session by default."
-                    ),
-                },
-                "force_download_url": {
-                    "type": "boolean",
-                    "description": (
-                        "If true, always return metadata+URL instead of inline content. "
-                        "Default is false (auto-selects based on file size/type)."
-                    ),
-                },
-            },
-            "required": [],  # At least one must be provided
-        }
-
-    @property
-    def requires_auth(self) -> bool:
-        return True
-
-    def _is_text_mime_type(self, mime_type: str) -> bool:
-        """Check if the MIME type is a text-based type."""
-        text_types = [
-            "text/",
-            "application/json",
-            "application/xml",
-            "application/javascript",
-            "application/x-python",
-            "application/x-sh",
-        ]
-        return any(mime_type.startswith(t) for t in text_types)
-
-    async def _execute(
-        self,
-        user_id: str | None,
-        session: ChatSession,
-        **kwargs,
-    ) -> ToolResponseBase:
-        session_id = session.session_id
-
-        if not user_id:
-            return ErrorResponse(
-                message="Authentication required",
-                session_id=session_id,
-            )
-
-        file_id: Optional[str] = kwargs.get("file_id")
-        path: Optional[str] = kwargs.get("path")
-        force_download_url: bool = kwargs.get("force_download_url", False)
-
-        if not file_id and not path:
-            return ErrorResponse(
-                message="Please provide either file_id or path",
-                session_id=session_id,
-            )
-
-        try:
-            workspace = await get_or_create_workspace(user_id)
-            # Pass session_id for session-scoped file access
-            manager = WorkspaceManager(user_id, workspace.id, session_id)
-
-            # Get file info
-            if file_id:
-                file_info = await manager.get_file_info(file_id)
-                if file_info is None:
-                    return ErrorResponse(
-                        message=f"File not found: {file_id}",
-                        session_id=session_id,
-                    )
-                target_file_id = file_id
-            else:
-                # path is guaranteed to be non-None here due to the check above
-                assert path is not None
-                file_info = await manager.get_file_info_by_path(path)
-                if file_info is None:
-                    return ErrorResponse(
-                        message=f"File not found at path: {path}",
-                        session_id=session_id,
-                    )
-                target_file_id = file_info.id
-
-            # Decide whether to return inline content or metadata+URL
-            is_small_file = file_info.sizeBytes <= self.MAX_INLINE_SIZE_BYTES
-            is_text_file = self._is_text_mime_type(file_info.mimeType)
-
-            # Return inline content for small text files (unless force_download_url)
-            if is_small_file and is_text_file and not force_download_url:
-                content = await manager.read_file_by_id(target_file_id)
-                content_b64 = base64.b64encode(content).decode("utf-8")
-
-                return WorkspaceFileContentResponse(
-                    file_id=file_info.id,
-                    name=file_info.name,
-                    path=file_info.path,
-                    mime_type=file_info.mimeType,
-                    content_base64=content_b64,
-                    message=f"Successfully read file: {file_info.name}",
-                    session_id=session_id,
-                )
-
-            # Return metadata + workspace:// reference for large or binary files
-            # This prevents context bloat (100KB file = ~133KB as base64)
-            # Use workspace:// format so frontend urlTransform can add proxy prefix
-            download_url = f"workspace://{target_file_id}"
-
-            # Generate preview for text files
-            preview: str | None = None
-            if is_text_file:
-                try:
-                    content = await manager.read_file_by_id(target_file_id)
-                    preview_text = content[: self.PREVIEW_SIZE].decode(
-                        "utf-8", errors="replace"
-                    )
-                    if len(content) > self.PREVIEW_SIZE:
-                        preview_text += "..."
-                    preview = preview_text
-                except Exception:
-                    pass  # Preview is optional
-
-            return WorkspaceFileMetadataResponse(
-                file_id=file_info.id,
-                name=file_info.name,
-                path=file_info.path,
-                mime_type=file_info.mimeType,
-                size_bytes=file_info.sizeBytes,
-                download_url=download_url,
-                preview=preview,
-                message=f"File: {file_info.name} ({file_info.sizeBytes} bytes). Use download_url to retrieve content.",
-                session_id=session_id,
-            )
-
-        except FileNotFoundError as e:
-            return ErrorResponse(
-                message=str(e),
-                session_id=session_id,
-            )
-        except Exception as e:
-            logger.error(f"Error reading workspace file: {e}", exc_info=True)
-            return ErrorResponse(
-                message=f"Failed to read workspace file: {str(e)}",
-                error=str(e),
-                session_id=session_id,
-            )
-
-
-class WriteWorkspaceFileTool(BaseTool):
-    """Tool for writing files to workspace."""
-
-    @property
-    def name(self) -> str:
-        return "write_workspace_file"
-
-    @property
-    def description(self) -> str:
-        return (
-            "Write or create a file in the user's persistent workspace (cloud storage). "
-            "These files survive across sessions. "
-            "For ephemeral session files, use the SDK Write tool instead. "
-            "Provide the content as a base64-encoded string. "
-            f"Maximum file size is {Config().max_file_size_mb}MB. "
-            "Files are saved to the current session's folder by default. "
-            "Use /sessions/<session_id>/... for cross-session access."
-        )
-
-    @property
-    def parameters(self) -> dict[str, Any]:
-        return {
-            "type": "object",
-            "properties": {
-                "filename": {
-                    "type": "string",
-                    "description": "Name for the file (e.g., 'report.pdf')",
-                },
-                "content_base64": {
-                    "type": "string",
-                    "description": "Base64-encoded file content",
-                },
-                "path": {
-                    "type": "string",
-                    "description": (
-                        "Optional virtual path where to save the file "
-                        "(e.g., '/documents/report.pdf'). "
-                        "Defaults to '/{filename}'. Scoped to current session."
-                    ),
-                },
-                "mime_type": {
-                    "type": "string",
-                    "description": (
-                        "Optional MIME type of the file. "
-                        "Auto-detected from filename if not provided."
-                    ),
-                },
-                "overwrite": {
-                    "type": "boolean",
-                    "description": "Whether to overwrite if file exists at path (default: false)",
-                },
-            },
-            "required": ["filename", "content_base64"],
-        }
-
-    @property
-    def requires_auth(self) -> bool:
-        return True
-
-    async def _execute(
-        self,
-        user_id: str | None,
-        session: ChatSession,
-        **kwargs,
-    ) -> ToolResponseBase:
-        session_id = session.session_id
-
-        if not user_id:
-            return ErrorResponse(
-                message="Authentication required",
-                session_id=session_id,
-            )
-
-        filename: str = kwargs.get("filename", "")
-        content_b64: str = kwargs.get("content_base64", "")
-        path: Optional[str] = kwargs.get("path")
-        mime_type: Optional[str] = kwargs.get("mime_type")
-        overwrite: bool = kwargs.get("overwrite", False)
-
-        if not filename:
-            return ErrorResponse(
-                message="Please provide a filename",
-                session_id=session_id,
-            )
-
-        if not content_b64:
-            return ErrorResponse(
-                message="Please provide content_base64",
-                session_id=session_id,
-            )
-
-        # Decode content
-        try:
-            content = base64.b64decode(content_b64)
-        except Exception:
-            return ErrorResponse(
-                message="Invalid base64-encoded content",
-                session_id=session_id,
-            )
-
-        # Check size
-        max_file_size = Config().max_file_size_mb * 1024 * 1024
-        if len(content) > max_file_size:
-            return ErrorResponse(
-                message=f"File too large. Maximum size is {Config().max_file_size_mb}MB",
-                session_id=session_id,
-            )
-
-        try:
-            # Virus scan
-            await scan_content_safe(content, filename=filename)
-
-            workspace = await get_or_create_workspace(user_id)
-            # Pass session_id for session-scoped file access
-            manager = WorkspaceManager(user_id, workspace.id, session_id)
-
-            file_record = await manager.write_file(
-                content=content,
-                filename=filename,
-                path=path,
-                mime_type=mime_type,
-                overwrite=overwrite,
-            )
-
-            return WorkspaceWriteResponse(
-                file_id=file_record.id,
-                name=file_record.name,
-                path=file_record.path,
-                size_bytes=file_record.sizeBytes,
-                message=f"Successfully wrote file: {file_record.name}",
-                session_id=session_id,
-            )
-
-        except ValueError as e:
-            return ErrorResponse(
-                message=str(e),
-                session_id=session_id,
-            )
-        except Exception as e:
-            logger.error(f"Error writing workspace file: {e}", exc_info=True)
-            return ErrorResponse(
-                message=f"Failed to write workspace file: {str(e)}",
-                error=str(e),
-                session_id=session_id,
-            )
-
-
-class DeleteWorkspaceFileTool(BaseTool):
-    """Tool for deleting files from workspace."""
-
-    @property
-    def name(self) -> str:
-        return "delete_workspace_file"
-
-    @property
-    def description(self) -> str:
-        return (
-            "Delete a file from the user's persistent workspace (cloud storage). "
-            "Specify either file_id or path to identify the file. "
-            "Paths are scoped to the current session by default. "
-            "Use /sessions/<session_id>/... for cross-session access."
-        )
-
-    @property
-    def parameters(self) -> dict[str, Any]:
-        return {
-            "type": "object",
-            "properties": {
-                "file_id": {
-                    "type": "string",
-                    "description": "The file's unique ID (from list_workspace_files)",
-                },
-                "path": {
-                    "type": "string",
-                    "description": (
-                        "The virtual file path (e.g., '/documents/report.pdf'). "
-                        "Scoped to current session by default."
-                    ),
-                },
-            },
-            "required": [],  # At least one must be provided
-        }
-
-    @property
-    def requires_auth(self) -> bool:
-        return True
-
-    async def _execute(
-        self,
-        user_id: str | None,
-        session: ChatSession,
-        **kwargs,
-    ) -> ToolResponseBase:
-        session_id = session.session_id
-
-        if not user_id:
-            return ErrorResponse(
-                message="Authentication required",
-                session_id=session_id,
-            )
-
-        file_id: Optional[str] = kwargs.get("file_id")
-        path: Optional[str] = kwargs.get("path")
-
-        if not file_id and not path:
-            return ErrorResponse(
-                message="Please provide either file_id or path",
-                session_id=session_id,
-            )
-
-        try:
-            workspace = await get_or_create_workspace(user_id)
-            # Pass session_id for session-scoped file access
-            manager = WorkspaceManager(user_id, workspace.id, session_id)
-
-            # Determine the file_id to delete
-            target_file_id: str
-            if file_id:
-                target_file_id = file_id
-            else:
-                # path is guaranteed to be non-None here due to the check above
-                assert path is not None
-                file_info = await manager.get_file_info_by_path(path)
-                if file_info is None:
-                    return ErrorResponse(
-                        message=f"File not found at path: {path}",
-                        session_id=session_id,
-                    )
-                target_file_id = file_info.id
-
-            success = await manager.delete_file(target_file_id)
-
-            if not success:
-                return ErrorResponse(
-                    message=f"File not found: {target_file_id}",
-                    session_id=session_id,
-                )
-
-            return WorkspaceDeleteResponse(
-                file_id=target_file_id,
-                success=True,
-                message="File deleted successfully",
-                session_id=session_id,
-            )
-
-        except Exception as e:
-            logger.error(f"Error deleting workspace file: {e}", exc_info=True)
-            return ErrorResponse(
-                message=f"Failed to delete workspace file: {str(e)}",
-                error=str(e),
-                session_id=session_id,
-            )

autogpt_platform/backend/backend/api/features/executions/review/review_routes_test.py

@@ -638,7 +638,7 @@ async def test_process_review_action_auto_approve_creates_auto_approval_records(
 
     # Mock get_node_executions to return node_id mapping
     mock_get_node_executions = mocker.patch(
-        "backend.data.execution.get_node_executions"
+        "backend.api.features.executions.review.routes.get_node_executions"
     )
     mock_node_exec = mocker.Mock(spec=NodeExecutionResult)
     mock_node_exec.node_exec_id = "test_node_123"
@@ -936,7 +936,7 @@ async def test_process_review_action_auto_approve_only_applies_to_approved_revie
 
     # Mock get_node_executions to return node_id mapping
     mock_get_node_executions = mocker.patch(
-        "backend.data.execution.get_node_executions"
+        "backend.api.features.executions.review.routes.get_node_executions"
     )
     mock_node_exec = mocker.Mock(spec=NodeExecutionResult)
     mock_node_exec.node_exec_id = "node_exec_approved"
@@ -1148,7 +1148,7 @@ async def test_process_review_action_per_review_auto_approve_granularity(
 
     # Mock get_node_executions to return batch node data
     mock_get_node_executions = mocker.patch(
-        "backend.data.execution.get_node_executions"
+        "backend.api.features.executions.review.routes.get_node_executions"
     )
     # Create mock node executions for each review
     mock_node_execs = []

autogpt_platform/backend/backend/api/features/executions/review/routes.py

@@ -6,10 +6,15 @@ import autogpt_libs.auth as autogpt_auth_lib
 from fastapi import APIRouter, HTTPException, Query, Security, status
 from prisma.enums import ReviewStatus
 
+from backend.copilot.constants import (
+    is_copilot_synthetic_id,
+    parse_node_id_from_exec_id,
+)
 from backend.data.execution import (
     ExecutionContext,
     ExecutionStatus,
     get_graph_execution_meta,
+    get_node_executions,
 )
 from backend.data.graph import get_graph_settings
 from backend.data.human_review import (
@@ -22,6 +27,7 @@ from backend.data.human_review import (
 )
 from backend.data.model import USER_TIMEZONE_NOT_SET
 from backend.data.user import get_user_by_id
+from backend.data.workspace import get_or_create_workspace
 from backend.executor.utils import add_graph_execution
 
 from .model import PendingHumanReviewModel, ReviewRequest, ReviewResponse
@@ -35,6 +41,38 @@ router = APIRouter(
 )
 
 
+async def _resolve_node_ids(
+    node_exec_ids: list[str],
+    graph_exec_id: str,
+    is_copilot: bool,
+) -> dict[str, str]:
+    """Resolve node_exec_id -> node_id for auto-approval records.
+
+    CoPilot synthetic IDs encode node_id in the format "{node_id}:{random}".
+    Graph executions look up node_id from NodeExecution records.
+    """
+    if not node_exec_ids:
+        return {}
+
+    if is_copilot:
+        return {neid: parse_node_id_from_exec_id(neid) for neid in node_exec_ids}
+
+    node_execs = await get_node_executions(
+        graph_exec_id=graph_exec_id, include_exec_data=False
+    )
+    node_exec_map = {ne.node_exec_id: ne.node_id for ne in node_execs}
+
+    result = {}
+    for neid in node_exec_ids:
+        if neid in node_exec_map:
+            result[neid] = node_exec_map[neid]
+        else:
+            logger.error(
+                f"Failed to resolve node_id for {neid}: Node execution not found."
+            )
+    return result
+
+
 @router.get(
     "/pending",
     summary="Get Pending Reviews",
@@ -109,14 +147,16 @@ async def list_pending_reviews_for_execution(
     """
 
     # Verify user owns the graph execution before returning reviews
-    graph_exec = await get_graph_execution_meta(
-        user_id=user_id, execution_id=graph_exec_id
-    )
-    if not graph_exec:
-        raise HTTPException(
-            status_code=status.HTTP_404_NOT_FOUND,
-            detail=f"Graph execution #{graph_exec_id} not found",
+    # (CoPilot synthetic IDs don't have graph execution records)
+    if not is_copilot_synthetic_id(graph_exec_id):
+        graph_exec = await get_graph_execution_meta(
+            user_id=user_id, execution_id=graph_exec_id
         )
+        if not graph_exec:
+            raise HTTPException(
+                status_code=status.HTTP_404_NOT_FOUND,
+                detail=f"Graph execution #{graph_exec_id} not found",
+            )
 
     return await get_pending_reviews_for_execution(graph_exec_id, user_id)
 
@@ -159,30 +199,26 @@ async def process_review_action(
         )
 
     graph_exec_id = next(iter(graph_exec_ids))
+    is_copilot = is_copilot_synthetic_id(graph_exec_id)
 
-    # Validate execution status before processing reviews
-    graph_exec_meta = await get_graph_execution_meta(
-        user_id=user_id, execution_id=graph_exec_id
-    )
-
-    if not graph_exec_meta:
-        raise HTTPException(
-            status_code=status.HTTP_404_NOT_FOUND,
-            detail=f"Graph execution #{graph_exec_id} not found",
-        )
-
-    # Only allow processing reviews if execution is paused for review
-    # or incomplete (partial execution with some reviews already processed)
-    if graph_exec_meta.status not in (
-        ExecutionStatus.REVIEW,
-        ExecutionStatus.INCOMPLETE,
-    ):
-        raise HTTPException(
-            status_code=status.HTTP_409_CONFLICT,
-            detail=f"Cannot process reviews while execution status is {graph_exec_meta.status}. "
-            f"Reviews can only be processed when execution is paused (REVIEW status). "
-            f"Current status: {graph_exec_meta.status}",
+    # Validate execution status for graph executions (skip for CoPilot synthetic IDs)
+    if not is_copilot:
+        graph_exec_meta = await get_graph_execution_meta(
+            user_id=user_id, execution_id=graph_exec_id
         )
+        if not graph_exec_meta:
+            raise HTTPException(
+                status_code=status.HTTP_404_NOT_FOUND,
+                detail=f"Graph execution #{graph_exec_id} not found",
+            )
+        if graph_exec_meta.status not in (
+            ExecutionStatus.REVIEW,
+            ExecutionStatus.INCOMPLETE,
+        ):
+            raise HTTPException(
+                status_code=status.HTTP_409_CONFLICT,
+                detail=f"Cannot process reviews while execution status is {graph_exec_meta.status}",
+            )
 
     # Build review decisions map and track which reviews requested auto-approval
     # Auto-approved reviews use original data (no modifications allowed)
@@ -235,7 +271,7 @@ async def process_review_action(
             )
             return (node_id, False)
 
-    # Collect node_exec_ids that need auto-approval
+    # Collect node_exec_ids that need auto-approval and resolve their node_ids
     node_exec_ids_needing_auto_approval = [
         node_exec_id
         for node_exec_id, review_result in updated_reviews.items()
@@ -243,29 +279,16 @@ async def process_review_action(
         and auto_approve_requests.get(node_exec_id, False)
     ]
 
-    # Batch-fetch node executions to get node_ids
+    node_id_map = await _resolve_node_ids(
+        node_exec_ids_needing_auto_approval, graph_exec_id, is_copilot
+    )
+
+    # Deduplicate by node_id — one auto-approval per node
     nodes_needing_auto_approval: dict[str, Any] = {}
-    if node_exec_ids_needing_auto_approval:
-        from backend.data.execution import get_node_executions
-
-        node_execs = await get_node_executions(
-            graph_exec_id=graph_exec_id, include_exec_data=False
-        )
-        node_exec_map = {node_exec.node_exec_id: node_exec for node_exec in node_execs}
-
-        for node_exec_id in node_exec_ids_needing_auto_approval:
-            node_exec = node_exec_map.get(node_exec_id)
-            if node_exec:
-                review_result = updated_reviews[node_exec_id]
-                # Use the first approved review for this node (deduplicate by node_id)
-                if node_exec.node_id not in nodes_needing_auto_approval:
-                    nodes_needing_auto_approval[node_exec.node_id] = review_result
-            else:
-                logger.error(
-                    f"Failed to create auto-approval record for {node_exec_id}: "
-                    f"Node execution not found. This may indicate a race condition "
-                    f"or data inconsistency."
-                )
+    for node_exec_id in node_exec_ids_needing_auto_approval:
+        node_id = node_id_map.get(node_exec_id)
+        if node_id and node_id not in nodes_needing_auto_approval:
+            nodes_needing_auto_approval[node_id] = updated_reviews[node_exec_id]
 
     # Execute all auto-approval creations in parallel (deduplicated by node_id)
     auto_approval_results = await asyncio.gather(
@@ -280,13 +303,11 @@ async def process_review_action(
     auto_approval_failed_count = 0
     for result in auto_approval_results:
         if isinstance(result, Exception):
-            # Unexpected exception during auto-approval creation
             auto_approval_failed_count += 1
             logger.error(
                 f"Unexpected exception during auto-approval creation: {result}"
             )
         elif isinstance(result, tuple) and len(result) == 2 and not result[1]:
-            # Auto-approval creation failed (returned False)
             auto_approval_failed_count += 1
 
     # Count results
@@ -301,30 +322,31 @@ async def process_review_action(
         if review.status == ReviewStatus.REJECTED
     )
 
-    # Resume execution only if ALL pending reviews for this execution have been processed
-    if updated_reviews:
+    # Resume graph execution only for real graph executions (not CoPilot)
+    # CoPilot sessions are resumed by the LLM retrying run_block with review_id
+    if not is_copilot and updated_reviews:
         still_has_pending = await has_pending_reviews_for_graph_exec(graph_exec_id)
 
         if not still_has_pending:
-            # Get the graph_id from any processed review
             first_review = next(iter(updated_reviews.values()))
 
             try:
-                # Fetch user and settings to build complete execution context
                 user = await get_user_by_id(user_id)
                 settings = await get_graph_settings(
                     user_id=user_id, graph_id=first_review.graph_id
                 )
 
-                # Preserve user's timezone preference when resuming execution
                 user_timezone = (
                     user.timezone if user.timezone != USER_TIMEZONE_NOT_SET else "UTC"
                 )
 
+                workspace = await get_or_create_workspace(user_id)
+
                 execution_context = ExecutionContext(
                     human_in_the_loop_safe_mode=settings.human_in_the_loop_safe_mode,
                     sensitive_action_safe_mode=settings.sensitive_action_safe_mode,
                     user_timezone=user_timezone,
+                    workspace_id=workspace.id,
                 )
 
                 await add_graph_execution(

autogpt_platform/backend/backend/api/features/library/db_test.py

@@ -4,7 +4,6 @@ import prisma.enums
 import prisma.models
 import pytest
 
-import backend.api.features.store.exceptions
 from backend.data.db import connect
 from backend.data.includes import library_agent_include
 
@@ -144,6 +143,7 @@ async def test_add_agent_to_library(mocker):
     )
 
     mock_library_agent = mocker.patch("prisma.models.LibraryAgent.prisma")
+    mock_library_agent.return_value.find_first = mocker.AsyncMock(return_value=None)
     mock_library_agent.return_value.find_unique = mocker.AsyncMock(return_value=None)
     mock_library_agent.return_value.create = mocker.AsyncMock(
         return_value=mock_library_agent_data
@@ -178,7 +178,6 @@ async def test_add_agent_to_library(mocker):
                 "agentGraphVersion": 1,
             }
         },
-        include={"AgentGraph": True},
     )
     # Check that create was called with the expected data including settings
     create_call_args = mock_library_agent.return_value.create.call_args
@@ -218,7 +217,7 @@ async def test_add_agent_to_library_not_found(mocker):
     )
 
     # Call function and verify exception
-    with pytest.raises(backend.api.features.store.exceptions.AgentNotFoundError):
+    with pytest.raises(db.NotFoundError):
         await db.add_store_agent_to_library("version123", "test-user")
 
     # Verify mock called correctly

autogpt_platform/backend/backend/api/features/library/exceptions.py

@@ -0,0 +1,10 @@
+class FolderValidationError(Exception):
+    """Raised when folder operations fail validation."""
+
+    pass
+
+
+class FolderAlreadyExistsError(FolderValidationError):
+    """Raised when a folder with the same name already exists in the location."""
+
+    pass

autogpt_platform/backend/backend/api/features/library/model.py

@@ -26,6 +26,95 @@ class LibraryAgentStatus(str, Enum):
     ERROR = "ERROR"
 
 
+# === Folder Models ===
+
+
+class LibraryFolder(pydantic.BaseModel):
+    """Represents a folder for organizing library agents."""
+
+    id: str
+    user_id: str
+    name: str
+    icon: str | None = None
+    color: str | None = None
+    parent_id: str | None = None
+    created_at: datetime.datetime
+    updated_at: datetime.datetime
+    agent_count: int = 0  # Direct agents in folder
+    subfolder_count: int = 0  # Direct child folders
+
+    @staticmethod
+    def from_db(
+        folder: prisma.models.LibraryFolder,
+        agent_count: int = 0,
+        subfolder_count: int = 0,
+    ) -> "LibraryFolder":
+        """Factory method that constructs a LibraryFolder from a Prisma model."""
+        return LibraryFolder(
+            id=folder.id,
+            user_id=folder.userId,
+            name=folder.name,
+            icon=folder.icon,
+            color=folder.color,
+            parent_id=folder.parentId,
+            created_at=folder.createdAt,
+            updated_at=folder.updatedAt,
+            agent_count=agent_count,
+            subfolder_count=subfolder_count,
+        )
+
+
+class LibraryFolderTree(LibraryFolder):
+    """Folder with nested children for tree view."""
+
+    children: list["LibraryFolderTree"] = []
+
+
+class FolderCreateRequest(pydantic.BaseModel):
+    """Request model for creating a folder."""
+
+    name: str = pydantic.Field(..., min_length=1, max_length=100)
+    icon: str | None = None
+    color: str | None = pydantic.Field(
+        None, pattern=r"^#[0-9A-Fa-f]{6}$", description="Hex color code (#RRGGBB)"
+    )
+    parent_id: str | None = None
+
+
+class FolderUpdateRequest(pydantic.BaseModel):
+    """Request model for updating a folder."""
+
+    name: str | None = pydantic.Field(None, min_length=1, max_length=100)
+    icon: str | None = None
+    color: str | None = None
+
+
+class FolderMoveRequest(pydantic.BaseModel):
+    """Request model for moving a folder to a new parent."""
+
+    target_parent_id: str | None = None  # None = move to root
+
+
+class BulkMoveAgentsRequest(pydantic.BaseModel):
+    """Request model for moving multiple agents to a folder."""
+
+    agent_ids: list[str]
+    folder_id: str | None = None  # None = move to root
+
+
+class FolderListResponse(pydantic.BaseModel):
+    """Response schema for a list of folders."""
+
+    folders: list[LibraryFolder]
+    pagination: Pagination
+
+
+class FolderTreeResponse(pydantic.BaseModel):
+    """Response schema for folder tree structure."""
+
+    tree: list[LibraryFolderTree]
+
+
 class MarketplaceListingCreator(pydantic.BaseModel):
     """Creator information for a marketplace listing."""
 
@@ -76,7 +165,6 @@ class LibraryAgent(pydantic.BaseModel):
     id: str
     graph_id: str
     graph_version: int
-    owner_user_id: str
 
     image_url: str | None
 
@@ -117,9 +205,14 @@ class LibraryAgent(pydantic.BaseModel):
         default_factory=list,
         description="List of recent executions with status, score, and summary",
     )
-    can_access_graph: bool
+    can_access_graph: bool = pydantic.Field(
+        description="Indicates whether the same user owns the corresponding graph"
+    )
     is_latest_version: bool
     is_favorite: bool
+    folder_id: str | None = None
+    folder_name: str | None = None  # Denormalized for display
+
     recommended_schedule_cron: str | None = None
     settings: GraphSettings = pydantic.Field(default_factory=GraphSettings)
     marketplace_listing: Optional["MarketplaceListing"] = None
@@ -232,7 +325,6 @@ class LibraryAgent(pydantic.BaseModel):
             id=agent.id,
             graph_id=agent.agentGraphId,
             graph_version=agent.agentGraphVersion,
-            owner_user_id=agent.userId,
             image_url=agent.imageUrl,
             creator_name=creator_name,
             creator_image_url=creator_image_url,
@@ -259,6 +351,8 @@ class LibraryAgent(pydantic.BaseModel):
             can_access_graph=can_access_graph,
             is_latest_version=is_latest_version,
             is_favorite=agent.isFavorite,
+            folder_id=agent.folderId,
+            folder_name=agent.Folder.name if agent.Folder else None,
             recommended_schedule_cron=agent.AgentGraph.recommendedScheduleCron,
             settings=_parse_settings(agent.settings),
             marketplace_listing=marketplace_listing_data,
@@ -470,3 +564,7 @@ class LibraryAgentUpdateRequest(pydantic.BaseModel):
     settings: Optional[GraphSettings] = pydantic.Field(
         default=None, description="User-specific settings for this library agent"
     )
+    folder_id: Optional[str] = pydantic.Field(
+        default=None,
+        description="Folder ID to move agent to (None to move to root)",
+    )

autogpt_platform/backend/backend/api/features/library/routes/__init__.py

@@ -1,9 +1,11 @@
 import fastapi
 
 from .agents import router as agents_router
+from .folders import router as folders_router
 from .presets import router as presets_router
 
 router = fastapi.APIRouter()
 
 router.include_router(presets_router)
+router.include_router(folders_router)
 router.include_router(agents_router)

autogpt_platform/backend/backend/api/features/library/routes/agents.py

@@ -41,6 +41,14 @@ async def list_library_agents(
         ge=1,
         description="Number of agents per page (must be >= 1)",
     ),
+    folder_id: Optional[str] = Query(
+        None,
+        description="Filter by folder ID",
+    ),
+    include_root_only: bool = Query(
+        False,
+        description="Only return agents without a folder (root-level agents)",
+    ),
 ) -> library_model.LibraryAgentResponse:
     """
     Get all agents in the user's library (both created and saved).
@@ -51,6 +59,8 @@ async def list_library_agents(
         sort_by=sort_by,
         page=page,
         page_size=page_size,
+        folder_id=folder_id,
+        include_root_only=include_root_only,
     )
 
 
@@ -168,6 +178,7 @@ async def update_library_agent(
         is_favorite=payload.is_favorite,
         is_archived=payload.is_archived,
         settings=payload.settings,
+        folder_id=payload.folder_id,
     )
 
 

autogpt_platform/backend/backend/api/features/library/routes/folders.py

@@ -0,0 +1,287 @@
+from typing import Optional
+
+import autogpt_libs.auth as autogpt_auth_lib
+from fastapi import APIRouter, Query, Security, status
+from fastapi.responses import Response
+
+from .. import db as library_db
+from .. import model as library_model
+
+router = APIRouter(
+    prefix="/folders",
+    tags=["library", "folders", "private"],
+    dependencies=[Security(autogpt_auth_lib.requires_user)],
+)
+
+
+@router.get(
+    "",
+    summary="List Library Folders",
+    response_model=library_model.FolderListResponse,
+    responses={
+        200: {"description": "List of folders"},
+        500: {"description": "Server error"},
+    },
+)
+async def list_folders(
+    user_id: str = Security(autogpt_auth_lib.get_user_id),
+    parent_id: Optional[str] = Query(
+        None,
+        description="Filter by parent folder ID. If not provided, returns root-level folders.",
+    ),
+    include_relations: bool = Query(
+        True,
+        description="Include agent and subfolder relations (for counts)",
+    ),
+) -> library_model.FolderListResponse:
+    """
+    List folders for the authenticated user.
+
+    Args:
+        user_id: ID of the authenticated user.
+        parent_id: Optional parent folder ID to filter by.
+        include_relations: Whether to include agent and subfolder relations for counts.
+
+    Returns:
+        A FolderListResponse containing folders.
+    """
+    folders = await library_db.list_folders(
+        user_id=user_id,
+        parent_id=parent_id,
+        include_relations=include_relations,
+    )
+    return library_model.FolderListResponse(
+        folders=folders,
+        pagination=library_model.Pagination(
+            total_items=len(folders),
+            total_pages=1,
+            current_page=1,
+            page_size=len(folders),
+        ),
+    )
+
+
+@router.get(
+    "/tree",
+    summary="Get Folder Tree",
+    response_model=library_model.FolderTreeResponse,
+    responses={
+        200: {"description": "Folder tree structure"},
+        500: {"description": "Server error"},
+    },
+)
+async def get_folder_tree(
+    user_id: str = Security(autogpt_auth_lib.get_user_id),
+) -> library_model.FolderTreeResponse:
+    """
+    Get the full folder tree for the authenticated user.
+
+    Args:
+        user_id: ID of the authenticated user.
+
+    Returns:
+        A FolderTreeResponse containing the nested folder structure.
+    """
+    tree = await library_db.get_folder_tree(user_id=user_id)
+    return library_model.FolderTreeResponse(tree=tree)
+
+
+@router.get(
+    "/{folder_id}",
+    summary="Get Folder",
+    response_model=library_model.LibraryFolder,
+    responses={
+        200: {"description": "Folder details"},
+        404: {"description": "Folder not found"},
+        500: {"description": "Server error"},
+    },
+)
+async def get_folder(
+    folder_id: str,
+    user_id: str = Security(autogpt_auth_lib.get_user_id),
+) -> library_model.LibraryFolder:
+    """
+    Get a specific folder.
+
+    Args:
+        folder_id: ID of the folder to retrieve.
+        user_id: ID of the authenticated user.
+
+    Returns:
+        The requested LibraryFolder.
+    """
+    return await library_db.get_folder(folder_id=folder_id, user_id=user_id)
+
+
+@router.post(
+    "",
+    summary="Create Folder",
+    status_code=status.HTTP_201_CREATED,
+    response_model=library_model.LibraryFolder,
+    responses={
+        201: {"description": "Folder created successfully"},
+        400: {"description": "Validation error"},
+        404: {"description": "Parent folder not found"},
+        409: {"description": "Folder name conflict"},
+        500: {"description": "Server error"},
+    },
+)
+async def create_folder(
+    payload: library_model.FolderCreateRequest,
+    user_id: str = Security(autogpt_auth_lib.get_user_id),
+) -> library_model.LibraryFolder:
+    """
+    Create a new folder.
+
+    Args:
+        payload: The folder creation request.
+        user_id: ID of the authenticated user.
+
+    Returns:
+        The created LibraryFolder.
+    """
+    return await library_db.create_folder(
+        user_id=user_id,
+        name=payload.name,
+        parent_id=payload.parent_id,
+        icon=payload.icon,
+        color=payload.color,
+    )
+
+
+@router.patch(
+    "/{folder_id}",
+    summary="Update Folder",
+    response_model=library_model.LibraryFolder,
+    responses={
+        200: {"description": "Folder updated successfully"},
+        400: {"description": "Validation error"},
+        404: {"description": "Folder not found"},
+        409: {"description": "Folder name conflict"},
+        500: {"description": "Server error"},
+    },
+)
+async def update_folder(
+    folder_id: str,
+    payload: library_model.FolderUpdateRequest,
+    user_id: str = Security(autogpt_auth_lib.get_user_id),
+) -> library_model.LibraryFolder:
+    """
+    Update a folder's properties.
+
+    Args:
+        folder_id: ID of the folder to update.
+        payload: The folder update request.
+        user_id: ID of the authenticated user.
+
+    Returns:
+        The updated LibraryFolder.
+    """
+    return await library_db.update_folder(
+        folder_id=folder_id,
+        user_id=user_id,
+        name=payload.name,
+        icon=payload.icon,
+        color=payload.color,
+    )
+
+
+@router.post(
+    "/{folder_id}/move",
+    summary="Move Folder",
+    response_model=library_model.LibraryFolder,
+    responses={
+        200: {"description": "Folder moved successfully"},
+        400: {"description": "Validation error (circular reference)"},
+        404: {"description": "Folder or target parent not found"},
+        409: {"description": "Folder name conflict in target location"},
+        500: {"description": "Server error"},
+    },
+)
+async def move_folder(
+    folder_id: str,
+    payload: library_model.FolderMoveRequest,
+    user_id: str = Security(autogpt_auth_lib.get_user_id),
+) -> library_model.LibraryFolder:
+    """
+    Move a folder to a new parent.
+
+    Args:
+        folder_id: ID of the folder to move.
+        payload: The move request with target parent.
+        user_id: ID of the authenticated user.
+
+    Returns:
+        The moved LibraryFolder.
+    """
+    return await library_db.move_folder(
+        folder_id=folder_id,
+        user_id=user_id,
+        target_parent_id=payload.target_parent_id,
+    )
+
+
+@router.delete(
+    "/{folder_id}",
+    summary="Delete Folder",
+    status_code=status.HTTP_204_NO_CONTENT,
+    responses={
+        204: {"description": "Folder deleted successfully"},
+        404: {"description": "Folder not found"},
+        500: {"description": "Server error"},
+    },
+)
+async def delete_folder(
+    folder_id: str,
+    user_id: str = Security(autogpt_auth_lib.get_user_id),
+) -> Response:
+    """
+    Soft-delete a folder and all its contents.
+
+    Args:
+        folder_id: ID of the folder to delete.
+        user_id: ID of the authenticated user.
+
+    Returns:
+        204 No Content if successful.
+    """
+    await library_db.delete_folder(
+        folder_id=folder_id,
+        user_id=user_id,
+        soft_delete=True,
+    )
+    return Response(status_code=status.HTTP_204_NO_CONTENT)
+
+
+# === Bulk Agent Operations ===
+
+
+@router.post(
+    "/agents/bulk-move",
+    summary="Bulk Move Agents",
+    response_model=list[library_model.LibraryAgent],
+    responses={
+        200: {"description": "Agents moved successfully"},
+        404: {"description": "Folder not found"},
+        500: {"description": "Server error"},
+    },
+)
+async def bulk_move_agents(
+    payload: library_model.BulkMoveAgentsRequest,
+    user_id: str = Security(autogpt_auth_lib.get_user_id),
+) -> list[library_model.LibraryAgent]:
+    """
+    Move multiple agents to a folder.
+
+    Args:
+        payload: The bulk move request with agent IDs and target folder.
+        user_id: ID of the authenticated user.
+
+    Returns:
+        The updated LibraryAgents.
+    """
+    return await library_db.bulk_move_agents_to_folder(
+        agent_ids=payload.agent_ids,
+        folder_id=payload.folder_id,
+        user_id=user_id,
+    )

autogpt_platform/backend/backend/api/features/library/routes_test.py

@@ -42,7 +42,6 @@ async def test_get_library_agents_success(
                 id="test-agent-1",
                 graph_id="test-agent-1",
                 graph_version=1,
-                owner_user_id=test_user_id,
                 name="Test Agent 1",
                 description="Test Description 1",
                 image_url=None,
@@ -67,7 +66,6 @@ async def test_get_library_agents_success(
                 id="test-agent-2",
                 graph_id="test-agent-2",
                 graph_version=1,
-                owner_user_id=test_user_id,
                 name="Test Agent 2",
                 description="Test Description 2",
                 image_url=None,
@@ -115,6 +113,8 @@ async def test_get_library_agents_success(
         sort_by=library_model.LibraryAgentSort.UPDATED_AT,
         page=1,
         page_size=15,
+        folder_id=None,
+        include_root_only=False,
     )
 
 
@@ -129,7 +129,6 @@ async def test_get_favorite_library_agents_success(
                 id="test-agent-1",
                 graph_id="test-agent-1",
                 graph_version=1,
-                owner_user_id=test_user_id,
                 name="Favorite Agent 1",
                 description="Test Favorite Description 1",
                 image_url=None,
@@ -182,7 +181,6 @@ def test_add_agent_to_library_success(
         id="test-library-agent-id",
         graph_id="test-agent-1",
         graph_version=1,
-        owner_user_id=test_user_id,
         name="Test Agent 1",
         description="Test Description 1",
         image_url=None,

autogpt_platform/backend/backend/api/features/mcp/routes.py

@@ -7,20 +7,24 @@ frontend can list available tools on an MCP server before placing a block.
 
 import logging
 from typing import Annotated, Any
-from urllib.parse import urlparse
 
 import fastapi
 from autogpt_libs.auth import get_user_id
 from fastapi import Security
-from pydantic import BaseModel, Field
+from pydantic import BaseModel, Field, SecretStr
 
 from backend.api.features.integrations.router import CredentialsMetaResponse
 from backend.blocks.mcp.client import MCPClient, MCPClientError
+from backend.blocks.mcp.helpers import (
+    auto_lookup_mcp_credential,
+    normalize_mcp_url,
+    server_host,
+)
 from backend.blocks.mcp.oauth import MCPOAuthHandler
 from backend.data.model import OAuth2Credentials
 from backend.integrations.creds_manager import IntegrationCredentialsManager
 from backend.integrations.providers import ProviderName
-from backend.util.request import HTTPClientError, Requests
+from backend.util.request import HTTPClientError, Requests, validate_url_host
 from backend.util.settings import Settings
 
 logger = logging.getLogger(__name__)
@@ -74,32 +78,20 @@ async def discover_tools(
     If the user has a stored MCP credential for this server URL, it will be
     used automatically — no need to pass an explicit auth token.
     """
+    # Validate URL to prevent SSRF — blocks loopback and private IP ranges.
+    try:
+        await validate_url_host(request.server_url)
+    except ValueError as e:
+        raise fastapi.HTTPException(status_code=400, detail=f"Invalid server URL: {e}")
+
     auth_token = request.auth_token
 
     # Auto-use stored MCP credential when no explicit token is provided.
     if not auth_token:
-        mcp_creds = await creds_manager.store.get_creds_by_provider(
-            user_id, ProviderName.MCP.value
+        best_cred = await auto_lookup_mcp_credential(
+            user_id, normalize_mcp_url(request.server_url)
         )
-        # Find the freshest credential for this server URL
-        best_cred: OAuth2Credentials | None = None
-        for cred in mcp_creds:
-            if (
-                isinstance(cred, OAuth2Credentials)
-                and (cred.metadata or {}).get("mcp_server_url") == request.server_url
-            ):
-                if best_cred is None or (
-                    (cred.access_token_expires_at or 0)
-                    > (best_cred.access_token_expires_at or 0)
-                ):
-                    best_cred = cred
         if best_cred:
-            # Refresh the token if expired before using it
-            best_cred = await creds_manager.refresh_if_needed(user_id, best_cred)
-            logger.info(
-                f"Using MCP credential {best_cred.id} for {request.server_url}, "
-                f"expires_at={best_cred.access_token_expires_at}"
-            )
             auth_token = best_cred.access_token.get_secret_value()
 
     client = MCPClient(request.server_url, auth_token=auth_token)
@@ -134,7 +126,7 @@ async def discover_tools(
         ],
         server_name=(
             init_result.get("serverInfo", {}).get("name")
-            or urlparse(request.server_url).hostname
+            or server_host(request.server_url)
             or "MCP"
         ),
         protocol_version=init_result.get("protocolVersion"),
@@ -173,7 +165,16 @@ async def mcp_oauth_login(
     3. Performs Dynamic Client Registration (RFC 7591) if available
     4. Returns the authorization URL for the frontend to open in a popup
     """
-    client = MCPClient(request.server_url)
+    # Validate URL to prevent SSRF — blocks loopback and private IP ranges.
+    try:
+        await validate_url_host(request.server_url)
+    except ValueError as e:
+        raise fastapi.HTTPException(status_code=400, detail=f"Invalid server URL: {e}")
+
+    # Normalize the URL so that credentials stored here are matched consistently
+    # by auto_lookup_mcp_credential (which also uses normalized URLs).
+    server_url = normalize_mcp_url(request.server_url)
+    client = MCPClient(server_url)
 
     # Step 1: Discover protected-resource metadata (RFC 9728)
     protected_resource = await client.discover_auth()
@@ -182,7 +183,16 @@ async def mcp_oauth_login(
 
     if protected_resource and protected_resource.get("authorization_servers"):
         auth_server_url = protected_resource["authorization_servers"][0]
-        resource_url = protected_resource.get("resource", request.server_url)
+        resource_url = protected_resource.get("resource", server_url)
+
+        # Validate the auth server URL from metadata to prevent SSRF.
+        try:
+            await validate_url_host(auth_server_url)
+        except ValueError as e:
+            raise fastapi.HTTPException(
+                status_code=400,
+                detail=f"Invalid authorization server URL in metadata: {e}",
+            )
 
         # Step 2a: Discover auth-server metadata (RFC 8414)
         metadata = await client.discover_auth_server_metadata(auth_server_url)
@@ -192,7 +202,7 @@ async def mcp_oauth_login(
         # Don't assume a resource_url — omitting it lets the auth server choose
         # the correct audience for the token (RFC 8707 resource is optional).
         resource_url = None
-        metadata = await client.discover_auth_server_metadata(request.server_url)
+        metadata = await client.discover_auth_server_metadata(server_url)
 
     if (
         not metadata
@@ -222,12 +232,18 @@ async def mcp_oauth_login(
     client_id = ""
     client_secret = ""
     if registration_endpoint:
-        reg_result = await _register_mcp_client(
-            registration_endpoint, redirect_uri, request.server_url
-        )
-        if reg_result:
-            client_id = reg_result.get("client_id", "")
-            client_secret = reg_result.get("client_secret", "")
+        # Validate the registration endpoint to prevent SSRF via metadata.
+        try:
+            await validate_url_host(registration_endpoint)
+        except ValueError:
+            pass  # Skip registration, fall back to default client_id
+        else:
+            reg_result = await _register_mcp_client(
+                registration_endpoint, redirect_uri, server_url
+            )
+            if reg_result:
+                client_id = reg_result.get("client_id", "")
+                client_secret = reg_result.get("client_secret", "")
 
     if not client_id:
         client_id = "autogpt-platform"
@@ -245,7 +261,7 @@ async def mcp_oauth_login(
             "token_url": token_url,
             "revoke_url": revoke_url,
             "resource_url": resource_url,
-            "server_url": request.server_url,
+            "server_url": server_url,
             "client_id": client_id,
             "client_secret": client_secret,
         },
@@ -342,7 +358,7 @@ async def mcp_oauth_callback(
     credentials.metadata["mcp_token_url"] = meta["token_url"]
     credentials.metadata["mcp_resource_url"] = meta.get("resource_url", "")
 
-    hostname = urlparse(meta["server_url"]).hostname or meta["server_url"]
+    hostname = server_host(meta["server_url"])
     credentials.title = f"MCP: {hostname}"
 
     # Remove old MCP credentials for the same server to prevent stale token buildup.
@@ -357,7 +373,9 @@ async def mcp_oauth_callback(
             ):
                 await creds_manager.store.delete_creds_by_id(user_id, old.id)
                 logger.info(
-                    f"Removed old MCP credential {old.id} for {meta['server_url']}"
+                    "Removed old MCP credential %s for %s",
+                    old.id,
+                    server_host(meta["server_url"]),
                 )
     except Exception:
         logger.debug("Could not clean up old MCP credentials", exc_info=True)
@@ -375,6 +393,93 @@ async def mcp_oauth_callback(
     )
 
 
+# ======================== Bearer Token ======================== #
+
+
+class MCPStoreTokenRequest(BaseModel):
+    """Request to store a bearer token for an MCP server that doesn't support OAuth."""
+
+    server_url: str = Field(
+        description="MCP server URL the token authenticates against"
+    )
+    token: SecretStr = Field(
+        min_length=1, description="Bearer token / API key for the MCP server"
+    )
+
+
+@router.post(
+    "/token",
+    summary="Store a bearer token for an MCP server",
+)
+async def mcp_store_token(
+    request: MCPStoreTokenRequest,
+    user_id: Annotated[str, Security(get_user_id)],
+) -> CredentialsMetaResponse:
+    """
+    Store a manually provided bearer token as an MCP credential.
+
+    Used by the Copilot MCPSetupCard when the server doesn't support the MCP
+    OAuth discovery flow (returns 400 from /oauth/login).  Subsequent
+    ``run_mcp_tool`` calls will automatically pick up the token via
+    ``_auto_lookup_credential``.
+    """
+    token = request.token.get_secret_value().strip()
+    if not token:
+        raise fastapi.HTTPException(status_code=422, detail="Token must not be blank.")
+
+    # Validate URL to prevent SSRF — blocks loopback and private IP ranges.
+    try:
+        await validate_url_host(request.server_url)
+    except ValueError as e:
+        raise fastapi.HTTPException(status_code=400, detail=f"Invalid server URL: {e}")
+
+    # Normalize URL so trailing-slash variants match existing credentials.
+    server_url = normalize_mcp_url(request.server_url)
+    hostname = server_host(server_url)
+
+    # Collect IDs of old credentials to clean up after successful create.
+    old_cred_ids: list[str] = []
+    try:
+        old_creds = await creds_manager.store.get_creds_by_provider(
+            user_id, ProviderName.MCP.value
+        )
+        old_cred_ids = [
+            old.id
+            for old in old_creds
+            if isinstance(old, OAuth2Credentials)
+            and normalize_mcp_url((old.metadata or {}).get("mcp_server_url", ""))
+            == server_url
+        ]
+    except Exception:
+        logger.debug("Could not query old MCP token credentials", exc_info=True)
+
+    credentials = OAuth2Credentials(
+        provider=ProviderName.MCP.value,
+        title=f"MCP: {hostname}",
+        access_token=SecretStr(token),
+        scopes=[],
+        metadata={"mcp_server_url": server_url},
+    )
+    await creds_manager.create(user_id, credentials)
+
+    # Only delete old credentials after the new one is safely stored.
+    for old_id in old_cred_ids:
+        try:
+            await creds_manager.store.delete_creds_by_id(user_id, old_id)
+        except Exception:
+            logger.debug("Could not clean up old MCP token credential", exc_info=True)
+
+    return CredentialsMetaResponse(
+        id=credentials.id,
+        provider=credentials.provider,
+        type=credentials.type,
+        title=credentials.title,
+        scopes=credentials.scopes,
+        username=credentials.username,
+        host=hostname,
+    )
+
+
 # ======================== Helpers ======================== #
 
 
@@ -400,5 +505,7 @@ async def _register_mcp_client(
             return data
         return None
     except Exception as e:
-        logger.warning(f"Dynamic client registration failed for {server_url}: {e}")
+        logger.warning(
+            "Dynamic client registration failed for %s: %s", server_host(server_url), e
+        )
         return None

autogpt_platform/backend/backend/api/features/mcp/test_routes.py

@@ -11,9 +11,11 @@ import httpx
 import pytest
 import pytest_asyncio
 from autogpt_libs.auth import get_user_id
+from pydantic import SecretStr
 
 from backend.api.features.mcp.routes import router
 from backend.blocks.mcp.client import MCPClientError, MCPTool
+from backend.data.model import OAuth2Credentials
 from backend.util.request import HTTPClientError
 
 app = fastapi.FastAPI()
@@ -28,6 +30,16 @@ async def client():
         yield c
 
 
+@pytest.fixture(autouse=True)
+def _bypass_ssrf_validation():
+    """Bypass validate_url_host in all route tests (test URLs don't resolve)."""
+    with patch(
+        "backend.api.features.mcp.routes.validate_url_host",
+        new_callable=AsyncMock,
+    ):
+        yield
+
+
 class TestDiscoverTools:
     @pytest.mark.asyncio(loop_scope="session")
     async def test_discover_tools_success(self, client):
@@ -56,9 +68,12 @@ class TestDiscoverTools:
 
         with (
             patch("backend.api.features.mcp.routes.MCPClient") as MockClient,
-            patch("backend.api.features.mcp.routes.creds_manager") as mock_cm,
+            patch(
+                "backend.api.features.mcp.routes.auto_lookup_mcp_credential",
+                new_callable=AsyncMock,
+                return_value=None,
+            ),
         ):
-            mock_cm.store.get_creds_by_provider = AsyncMock(return_value=[])
             instance = MockClient.return_value
             instance.initialize = AsyncMock(
                 return_value={
@@ -107,10 +122,6 @@ class TestDiscoverTools:
     @pytest.mark.asyncio(loop_scope="session")
     async def test_discover_tools_auto_uses_stored_credential(self, client):
         """When no explicit token is given, stored MCP credentials are used."""
-        from pydantic import SecretStr
-
-        from backend.data.model import OAuth2Credentials
-
         stored_cred = OAuth2Credentials(
             provider="mcp",
             title="MCP: example.com",
@@ -124,10 +135,12 @@ class TestDiscoverTools:
 
         with (
             patch("backend.api.features.mcp.routes.MCPClient") as MockClient,
-            patch("backend.api.features.mcp.routes.creds_manager") as mock_cm,
+            patch(
+                "backend.api.features.mcp.routes.auto_lookup_mcp_credential",
+                new_callable=AsyncMock,
+                return_value=stored_cred,
+            ),
         ):
-            mock_cm.store.get_creds_by_provider = AsyncMock(return_value=[stored_cred])
-            mock_cm.refresh_if_needed = AsyncMock(return_value=stored_cred)
             instance = MockClient.return_value
             instance.initialize = AsyncMock(
                 return_value={"serverInfo": {}, "protocolVersion": "2025-03-26"}
@@ -149,9 +162,12 @@ class TestDiscoverTools:
     async def test_discover_tools_mcp_error(self, client):
         with (
             patch("backend.api.features.mcp.routes.MCPClient") as MockClient,
-            patch("backend.api.features.mcp.routes.creds_manager") as mock_cm,
+            patch(
+                "backend.api.features.mcp.routes.auto_lookup_mcp_credential",
+                new_callable=AsyncMock,
+                return_value=None,
+            ),
         ):
-            mock_cm.store.get_creds_by_provider = AsyncMock(return_value=[])
             instance = MockClient.return_value
             instance.initialize = AsyncMock(
                 side_effect=MCPClientError("Connection refused")
@@ -169,9 +185,12 @@ class TestDiscoverTools:
     async def test_discover_tools_generic_error(self, client):
         with (
             patch("backend.api.features.mcp.routes.MCPClient") as MockClient,
-            patch("backend.api.features.mcp.routes.creds_manager") as mock_cm,
+            patch(
+                "backend.api.features.mcp.routes.auto_lookup_mcp_credential",
+                new_callable=AsyncMock,
+                return_value=None,
+            ),
         ):
-            mock_cm.store.get_creds_by_provider = AsyncMock(return_value=[])
             instance = MockClient.return_value
             instance.initialize = AsyncMock(side_effect=Exception("Network timeout"))
 
@@ -187,9 +206,12 @@ class TestDiscoverTools:
     async def test_discover_tools_auth_required(self, client):
         with (
             patch("backend.api.features.mcp.routes.MCPClient") as MockClient,
-            patch("backend.api.features.mcp.routes.creds_manager") as mock_cm,
+            patch(
+                "backend.api.features.mcp.routes.auto_lookup_mcp_credential",
+                new_callable=AsyncMock,
+                return_value=None,
+            ),
         ):
-            mock_cm.store.get_creds_by_provider = AsyncMock(return_value=[])
             instance = MockClient.return_value
             instance.initialize = AsyncMock(
                 side_effect=HTTPClientError("HTTP 401 Error: Unauthorized", 401)
@@ -207,9 +229,12 @@ class TestDiscoverTools:
     async def test_discover_tools_forbidden(self, client):
         with (
             patch("backend.api.features.mcp.routes.MCPClient") as MockClient,
-            patch("backend.api.features.mcp.routes.creds_manager") as mock_cm,
+            patch(
+                "backend.api.features.mcp.routes.auto_lookup_mcp_credential",
+                new_callable=AsyncMock,
+                return_value=None,
+            ),
         ):
-            mock_cm.store.get_creds_by_provider = AsyncMock(return_value=[])
             instance = MockClient.return_value
             instance.initialize = AsyncMock(
                 side_effect=HTTPClientError("HTTP 403 Error: Forbidden", 403)
@@ -331,10 +356,6 @@ class TestOAuthLogin:
 class TestOAuthCallback:
     @pytest.mark.asyncio(loop_scope="session")
     async def test_oauth_callback_success(self, client):
-        from pydantic import SecretStr
-
-        from backend.data.model import OAuth2Credentials
-
         mock_creds = OAuth2Credentials(
             provider="mcp",
             title=None,
@@ -434,3 +455,118 @@ class TestOAuthCallback:
 
         assert response.status_code == 400
         assert "token exchange failed" in response.json()["detail"].lower()
+
+
+class TestStoreToken:
+    @pytest.mark.asyncio(loop_scope="session")
+    async def test_store_token_success(self, client):
+        with patch("backend.api.features.mcp.routes.creds_manager") as mock_cm:
+            mock_cm.store.get_creds_by_provider = AsyncMock(return_value=[])
+            mock_cm.create = AsyncMock()
+
+            response = await client.post(
+                "/token",
+                json={
+                    "server_url": "https://mcp.example.com/mcp",
+                    "token": "my-api-key-123",
+                },
+            )
+
+        assert response.status_code == 200
+        data = response.json()
+        assert data["provider"] == "mcp"
+        assert data["type"] == "oauth2"
+        assert data["host"] == "mcp.example.com"
+        mock_cm.create.assert_called_once()
+
+    @pytest.mark.asyncio(loop_scope="session")
+    async def test_store_token_blank_rejected(self, client):
+        """Blank token string (after stripping) should return 422."""
+        response = await client.post(
+            "/token",
+            json={
+                "server_url": "https://mcp.example.com/mcp",
+                "token": "   ",
+            },
+        )
+        # Pydantic min_length=1 catches the whitespace-only token
+        assert response.status_code == 422
+
+    @pytest.mark.asyncio(loop_scope="session")
+    async def test_store_token_replaces_old_credential(self, client):
+        old_cred = OAuth2Credentials(
+            provider="mcp",
+            title="MCP: mcp.example.com",
+            access_token=SecretStr("old-token"),
+            scopes=[],
+            metadata={"mcp_server_url": "https://mcp.example.com/mcp"},
+        )
+        with patch("backend.api.features.mcp.routes.creds_manager") as mock_cm:
+            mock_cm.store.get_creds_by_provider = AsyncMock(return_value=[old_cred])
+            mock_cm.create = AsyncMock()
+            mock_cm.store.delete_creds_by_id = AsyncMock()
+
+            response = await client.post(
+                "/token",
+                json={
+                    "server_url": "https://mcp.example.com/mcp",
+                    "token": "new-token",
+                },
+            )
+
+        assert response.status_code == 200
+        mock_cm.store.delete_creds_by_id.assert_called_once_with(
+            "test-user-id", old_cred.id
+        )
+
+
+class TestSSRFValidation:
+    """Verify that validate_url_host is enforced on all endpoints."""
+
+    @pytest.mark.asyncio(loop_scope="session")
+    async def test_discover_tools_ssrf_blocked(self, client):
+        with patch(
+            "backend.api.features.mcp.routes.validate_url_host",
+            new_callable=AsyncMock,
+            side_effect=ValueError("blocked loopback"),
+        ):
+            response = await client.post(
+                "/discover-tools",
+                json={"server_url": "http://localhost/mcp"},
+            )
+
+        assert response.status_code == 400
+        assert "blocked loopback" in response.json()["detail"].lower()
+
+    @pytest.mark.asyncio(loop_scope="session")
+    async def test_oauth_login_ssrf_blocked(self, client):
+        with patch(
+            "backend.api.features.mcp.routes.validate_url_host",
+            new_callable=AsyncMock,
+            side_effect=ValueError("blocked private IP"),
+        ):
+            response = await client.post(
+                "/oauth/login",
+                json={"server_url": "http://10.0.0.1/mcp"},
+            )
+
+        assert response.status_code == 400
+        assert "blocked private ip" in response.json()["detail"].lower()
+
+    @pytest.mark.asyncio(loop_scope="session")
+    async def test_store_token_ssrf_blocked(self, client):
+        with patch(
+            "backend.api.features.mcp.routes.validate_url_host",
+            new_callable=AsyncMock,
+            side_effect=ValueError("blocked loopback"),
+        ):
+            response = await client.post(
+                "/token",
+                json={
+                    "server_url": "http://127.0.0.1/mcp",
+                    "token": "some-token",
+                },
+            )
+
+        assert response.status_code == 400
+        assert "blocked loopback" in response.json()["detail"].lower()

autogpt_platform/backend/backend/api/features/store/cache.py

@@ -1,5 +1,3 @@
-from typing import Literal
-
 from backend.util.cache import cached
 
 from . import db as store_db
@@ -23,7 +21,7 @@ def clear_all_caches():
 async def _get_cached_store_agents(
     featured: bool,
     creator: str | None,
-    sorted_by: Literal["rating", "runs", "name", "updated_at"] | None,
+    sorted_by: store_db.StoreAgentsSortOptions | None,
     search_query: str | None,
     category: str | None,
     page: int,
@@ -57,7 +55,7 @@ async def _get_cached_agent_details(
 async def _get_cached_store_creators(
     featured: bool,
     search_query: str | None,
-    sorted_by: Literal["agent_rating", "agent_runs", "num_agents"] | None,
+    sorted_by: store_db.StoreCreatorsSortOptions | None,
     page: int,
     page_size: int,
 ):
@@ -75,4 +73,4 @@ async def _get_cached_store_creators(
 @cached(maxsize=100, ttl_seconds=300, shared_cache=True)
 async def _get_cached_creator_details(username: str):
     """Cached helper to get creator details."""
-    return await store_db.get_store_creator_details(username=username.lower())
+    return await store_db.get_store_creator(username=username.lower())

autogpt_platform/backend/backend/api/features/store/content_handlers.py

@@ -9,15 +9,26 @@ import logging
 from abc import ABC, abstractmethod
 from dataclasses import dataclass
 from pathlib import Path
-from typing import Any
+from typing import Any, get_args, get_origin
 
 from prisma.enums import ContentType
 
+from backend.blocks.llm import LlmModel
 from backend.data.db import query_raw_with_schema
 
 logger = logging.getLogger(__name__)
 
 
+def _contains_type(annotation: Any, target: type) -> bool:
+    """Check if an annotation is or contains the target type (handles Optional/Union/Annotated)."""
+    if annotation is target:
+        return True
+    origin = get_origin(annotation)
+    if origin is None:
+        return False
+    return any(_contains_type(arg, target) for arg in get_args(annotation))
+
+
 @dataclass
 class ContentItem:
     """Represents a piece of content to be embedded."""
@@ -188,45 +199,51 @@ class BlockHandler(ContentHandler):
             try:
                 block_instance = block_cls()
 
-                # Skip disabled blocks - they shouldn't be indexed
                 if block_instance.disabled:
                     continue
 
                 # Build searchable text from block metadata
                 parts = []
-                if hasattr(block_instance, "name") and block_instance.name:
+                if block_instance.name:
                     parts.append(block_instance.name)
-                if (
-                    hasattr(block_instance, "description")
-                    and block_instance.description
-                ):
+                if block_instance.description:
                     parts.append(block_instance.description)
-                if hasattr(block_instance, "categories") and block_instance.categories:
-                    # Convert BlockCategory enum to strings
+                if block_instance.categories:
                     parts.append(
                         " ".join(str(cat.value) for cat in block_instance.categories)
                     )
 
-                # Add input/output schema info
-                if hasattr(block_instance, "input_schema"):
-                    schema = block_instance.input_schema
-                    if hasattr(schema, "model_json_schema"):
-                        schema_dict = schema.model_json_schema()
-                        if "properties" in schema_dict:
-                            for prop_name, prop_info in schema_dict[
-                                "properties"
-                            ].items():
-                                if "description" in prop_info:
-                                    parts.append(
-                                        f"{prop_name}: {prop_info['description']}"
-                                    )
+                # Add input schema field descriptions
+                block_input_fields = block_instance.input_schema.model_fields
+                parts += [
+                    f"{field_name}: {field_info.description}"
+                    for field_name, field_info in block_input_fields.items()
+                    if field_info.description
+                ]
 
                 searchable_text = " ".join(parts)
 
-                # Convert categories set of enums to list of strings for JSON serialization
-                categories = getattr(block_instance, "categories", set())
                 categories_list = (
-                    [cat.value for cat in categories] if categories else []
+                    [cat.value for cat in block_instance.categories]
+                    if block_instance.categories
+                    else []
+                )
+
+                # Extract provider names from credentials fields
+                credentials_info = (
+                    block_instance.input_schema.get_credentials_fields_info()
+                )
+                is_integration = len(credentials_info) > 0
+                provider_names = [
+                    provider.value.lower()
+                    for info in credentials_info.values()
+                    for provider in info.provider
+                ]
+
+                # Check if block has LlmModel field in input schema
+                has_llm_model_field = any(
+                    _contains_type(field.annotation, LlmModel)
+                    for field in block_instance.input_schema.model_fields.values()
                 )
 
                 items.append(
@@ -235,8 +252,11 @@ class BlockHandler(ContentHandler):
                         content_type=ContentType.BLOCK,
                         searchable_text=searchable_text,
                         metadata={
-                            "name": getattr(block_instance, "name", ""),
+                            "name": block_instance.name,
                             "categories": categories_list,
+                            "providers": provider_names,
+                            "has_llm_model_field": has_llm_model_field,
+                            "is_integration": is_integration,
                         },
                         user_id=None,  # Blocks are public
                     )

autogpt_platform/backend/backend/api/features/store/content_handlers_test.py

@@ -82,9 +82,10 @@ async def test_block_handler_get_missing_items(mocker):
     mock_block_instance.description = "Performs calculations"
     mock_block_instance.categories = [MagicMock(value="MATH")]
     mock_block_instance.disabled = False
-    mock_block_instance.input_schema.model_json_schema.return_value = {
-        "properties": {"expression": {"description": "Math expression to evaluate"}}
-    }
+    mock_field = MagicMock()
+    mock_field.description = "Math expression to evaluate"
+    mock_block_instance.input_schema.model_fields = {"expression": mock_field}
+    mock_block_instance.input_schema.get_credentials_fields_info.return_value = {}
     mock_block_class.return_value = mock_block_instance
 
     mock_blocks = {"block-uuid-1": mock_block_class}
@@ -309,19 +310,19 @@ async def test_content_handlers_registry():
 
 
 @pytest.mark.asyncio(loop_scope="session")
-async def test_block_handler_handles_missing_attributes():
-    """Test BlockHandler gracefully handles blocks with missing attributes."""
+async def test_block_handler_handles_empty_attributes():
+    """Test BlockHandler handles blocks with empty/falsy attribute values."""
     handler = BlockHandler()
 
-    # Mock block with minimal attributes
+    # Mock block with empty values (all attributes exist but are falsy)
     mock_block_class = MagicMock()
     mock_block_instance = MagicMock()
     mock_block_instance.name = "Minimal Block"
     mock_block_instance.disabled = False
-    # No description, categories, or schema
-    del mock_block_instance.description
-    del mock_block_instance.categories
-    del mock_block_instance.input_schema
+    mock_block_instance.description = ""
+    mock_block_instance.categories = set()
+    mock_block_instance.input_schema.model_fields = {}
+    mock_block_instance.input_schema.get_credentials_fields_info.return_value = {}
     mock_block_class.return_value = mock_block_instance
 
     mock_blocks = {"block-minimal": mock_block_class}
@@ -352,6 +353,8 @@ async def test_block_handler_skips_failed_blocks():
     good_instance.description = "Works fine"
     good_instance.categories = []
     good_instance.disabled = False
+    good_instance.input_schema.model_fields = {}
+    good_instance.input_schema.get_credentials_fields_info.return_value = {}
     good_block.return_value = good_instance
 
     bad_block = MagicMock()

autogpt_platform/backend/backend/api/features/store/db_test.py

@@ -26,7 +26,7 @@ async def test_get_store_agents(mocker):
     mock_agents = [
         prisma.models.StoreAgent(
             listing_id="test-id",
-            storeListingVersionId="version123",
+            listing_version_id="version123",
             slug="test-agent",
             agent_name="Test Agent",
             agent_video=None,
@@ -40,11 +40,11 @@ async def test_get_store_agents(mocker):
             runs=10,
             rating=4.5,
             versions=["1.0"],
-            agentGraphVersions=["1"],
-            agentGraphId="test-graph-id",
+            graph_id="test-graph-id",
+            graph_versions=["1"],
             updated_at=datetime.now(),
             is_available=False,
-            useForOnboarding=False,
+            use_for_onboarding=False,
         )
     ]
 
@@ -68,10 +68,10 @@ async def test_get_store_agents(mocker):
 
 @pytest.mark.asyncio(loop_scope="session")
 async def test_get_store_agent_details(mocker):
-    # Mock data
+    # Mock data - StoreAgent view already contains the active version data
     mock_agent = prisma.models.StoreAgent(
         listing_id="test-id",
-        storeListingVersionId="version123",
+        listing_version_id="version123",
         slug="test-agent",
         agent_name="Test Agent",
         agent_video="video.mp4",
@@ -85,102 +85,38 @@ async def test_get_store_agent_details(mocker):
         runs=10,
         rating=4.5,
         versions=["1.0"],
-        agentGraphVersions=["1"],
-        agentGraphId="test-graph-id",
-        updated_at=datetime.now(),
-        is_available=False,
-        useForOnboarding=False,
-    )
-
-    # Mock active version agent (what we want to return for active version)
-    mock_active_agent = prisma.models.StoreAgent(
-        listing_id="test-id",
-        storeListingVersionId="active-version-id",
-        slug="test-agent",
-        agent_name="Test Agent Active",
-        agent_video="active_video.mp4",
-        agent_image=["active_image.jpg"],
-        featured=False,
-        creator_username="creator",
-        creator_avatar="avatar.jpg",
-        sub_heading="Test heading active",
-        description="Test description active",
-        categories=["test"],
-        runs=15,
-        rating=4.8,
-        versions=["1.0", "2.0"],
-        agentGraphVersions=["1", "2"],
-        agentGraphId="test-graph-id-active",
+        graph_id="test-graph-id",
+        graph_versions=["1"],
         updated_at=datetime.now(),
         is_available=True,
-        useForOnboarding=False,
+        use_for_onboarding=False,
     )
 
-    # Create a mock StoreListing result
-    mock_store_listing = mocker.MagicMock()
-    mock_store_listing.activeVersionId = "active-version-id"
-    mock_store_listing.hasApprovedVersion = True
-    mock_store_listing.ActiveVersion = mocker.MagicMock()
-    mock_store_listing.ActiveVersion.recommendedScheduleCron = None
-
-    # Mock StoreAgent prisma call - need to handle multiple calls
+    # Mock StoreAgent prisma call
     mock_store_agent = mocker.patch("prisma.models.StoreAgent.prisma")
-
-    # Set up side_effect to return different results for different calls
-    def mock_find_first_side_effect(*args, **kwargs):
-        where_clause = kwargs.get("where", {})
-        if "storeListingVersionId" in where_clause:
-            # Second call for active version
-            return mock_active_agent
-        else:
-            # First call for initial lookup
-            return mock_agent
-
-    mock_store_agent.return_value.find_first = mocker.AsyncMock(
-        side_effect=mock_find_first_side_effect
-    )
-
-    # Mock Profile prisma call
-    mock_profile = mocker.MagicMock()
-    mock_profile.userId = "user-id-123"
-    mock_profile_db = mocker.patch("prisma.models.Profile.prisma")
-    mock_profile_db.return_value.find_first = mocker.AsyncMock(
-        return_value=mock_profile
-    )
-
-    # Mock StoreListing prisma call
-    mock_store_listing_db = mocker.patch("prisma.models.StoreListing.prisma")
-    mock_store_listing_db.return_value.find_first = mocker.AsyncMock(
-        return_value=mock_store_listing
-    )
+    mock_store_agent.return_value.find_first = mocker.AsyncMock(return_value=mock_agent)
 
     # Call function
     result = await db.get_store_agent_details("creator", "test-agent")
 
-    # Verify results - should use active version data
+    # Verify results - constructed from the StoreAgent view
     assert result.slug == "test-agent"
-    assert result.agent_name == "Test Agent Active"  # From active version
-    assert result.active_version_id == "active-version-id"
+    assert result.agent_name == "Test Agent"
+    assert result.active_version_id == "version123"
     assert result.has_approved_version is True
-    assert (
-        result.store_listing_version_id == "active-version-id"
-    )  # Should be active version ID
+    assert result.store_listing_version_id == "version123"
+    assert result.graph_id == "test-graph-id"
+    assert result.runs == 10
+    assert result.rating == 4.5
 
-    # Verify mocks called correctly - now expecting 2 calls
-    assert mock_store_agent.return_value.find_first.call_count == 2
-
-    # Check the specific calls
-    calls = mock_store_agent.return_value.find_first.call_args_list
-    assert calls[0] == mocker.call(
+    # Verify single StoreAgent lookup
+    mock_store_agent.return_value.find_first.assert_called_once_with(
         where={"creator_username": "creator", "slug": "test-agent"}
     )
-    assert calls[1] == mocker.call(where={"storeListingVersionId": "active-version-id"})
-
-    mock_store_listing_db.return_value.find_first.assert_called_once()
 
 
 @pytest.mark.asyncio(loop_scope="session")
-async def test_get_store_creator_details(mocker):
+async def test_get_store_creator(mocker):
     # Mock data
     mock_creator_data = prisma.models.Creator(
         name="Test Creator",
@@ -202,7 +138,7 @@ async def test_get_store_creator_details(mocker):
     mock_creator.return_value.find_unique.return_value = mock_creator_data
 
     # Call function
-    result = await db.get_store_creator_details("creator")
+    result = await db.get_store_creator("creator")
 
     # Verify results
     assert result.username == "creator"
@@ -218,61 +154,110 @@ async def test_get_store_creator_details(mocker):
 
 @pytest.mark.asyncio(loop_scope="session")
 async def test_create_store_submission(mocker):
-    # Mock data
+    now = datetime.now()
+
+    # Mock agent graph (with no pending submissions) and user with profile
+    mock_profile = prisma.models.Profile(
+        id="profile-id",
+        userId="user-id",
+        name="Test User",
+        username="testuser",
+        description="Test",
+        isFeatured=False,
+        links=[],
+        createdAt=now,
+        updatedAt=now,
+    )
+    mock_user = prisma.models.User(
+        id="user-id",
+        email="test@example.com",
+        createdAt=now,
+        updatedAt=now,
+        Profile=[mock_profile],
+        emailVerified=True,
+        metadata="{}",  # type: ignore[reportArgumentType]
+        integrations="",
+        maxEmailsPerDay=1,
+        notifyOnAgentRun=True,
+        notifyOnZeroBalance=True,
+        notifyOnLowBalance=True,
+        notifyOnBlockExecutionFailed=True,
+        notifyOnContinuousAgentError=True,
+        notifyOnDailySummary=True,
+        notifyOnWeeklySummary=True,
+        notifyOnMonthlySummary=True,
+        notifyOnAgentApproved=True,
+        notifyOnAgentRejected=True,
+        timezone="Europe/Delft",
+    )
     mock_agent = prisma.models.AgentGraph(
         id="agent-id",
         version=1,
         userId="user-id",
-        createdAt=datetime.now(),
+        createdAt=now,
         isActive=True,
+        StoreListingVersions=[],
+        User=mock_user,
     )
 
-    mock_listing = prisma.models.StoreListing(
+    # Mock the created StoreListingVersion (returned by create)
+    mock_store_listing_obj = prisma.models.StoreListing(
         id="listing-id",
-        createdAt=datetime.now(),
-        updatedAt=datetime.now(),
+        createdAt=now,
+        updatedAt=now,
         isDeleted=False,
         hasApprovedVersion=False,
         slug="test-agent",
         agentGraphId="agent-id",
-        agentGraphVersion=1,
         owningUserId="user-id",
-        Versions=[
-            prisma.models.StoreListingVersion(
-                id="version-id",
-                agentGraphId="agent-id",
-                agentGraphVersion=1,
-                name="Test Agent",
-                description="Test description",
-                createdAt=datetime.now(),
-                updatedAt=datetime.now(),
-                subHeading="Test heading",
-                imageUrls=["image.jpg"],
-                categories=["test"],
-                isFeatured=False,
-                isDeleted=False,
-                version=1,
-                storeListingId="listing-id",
-                submissionStatus=prisma.enums.SubmissionStatus.PENDING,
-                isAvailable=True,
-            )
-        ],
         useForOnboarding=False,
     )
+    mock_version = prisma.models.StoreListingVersion(
+        id="version-id",
+        agentGraphId="agent-id",
+        agentGraphVersion=1,
+        name="Test Agent",
+        description="Test description",
+        createdAt=now,
+        updatedAt=now,
+        subHeading="",
+        imageUrls=[],
+        categories=[],
+        isFeatured=False,
+        isDeleted=False,
+        version=1,
+        storeListingId="listing-id",
+        submissionStatus=prisma.enums.SubmissionStatus.PENDING,
+        isAvailable=True,
+        submittedAt=now,
+        StoreListing=mock_store_listing_obj,
+    )
 
     # Mock prisma calls
     mock_agent_graph = mocker.patch("prisma.models.AgentGraph.prisma")
     mock_agent_graph.return_value.find_first = mocker.AsyncMock(return_value=mock_agent)
 
-    mock_store_listing = mocker.patch("prisma.models.StoreListing.prisma")
-    mock_store_listing.return_value.find_first = mocker.AsyncMock(return_value=None)
-    mock_store_listing.return_value.create = mocker.AsyncMock(return_value=mock_listing)
+    # Mock transaction context manager
+    mock_tx = mocker.MagicMock()
+    mocker.patch(
+        "backend.api.features.store.db.transaction",
+        return_value=mocker.AsyncMock(
+            __aenter__=mocker.AsyncMock(return_value=mock_tx),
+            __aexit__=mocker.AsyncMock(return_value=False),
+        ),
+    )
+
+    mock_sl = mocker.patch("prisma.models.StoreListing.prisma")
+    mock_sl.return_value.find_unique = mocker.AsyncMock(return_value=None)
+
+    mock_slv = mocker.patch("prisma.models.StoreListingVersion.prisma")
+    mock_slv.return_value.create = mocker.AsyncMock(return_value=mock_version)
 
     # Call function
     result = await db.create_store_submission(
         user_id="user-id",
-        agent_id="agent-id",
-        agent_version=1,
+        graph_id="agent-id",
+        graph_version=1,
         slug="test-agent",
         name="Test Agent",
         description="Test description",
@@ -281,11 +266,11 @@ async def test_create_store_submission(mocker):
     # Verify results
     assert result.name == "Test Agent"
     assert result.description == "Test description"
-    assert result.store_listing_version_id == "version-id"
+    assert result.listing_version_id == "version-id"
 
     # Verify mocks called correctly
     mock_agent_graph.return_value.find_first.assert_called_once()
-    mock_store_listing.return_value.create.assert_called_once()
+    mock_slv.return_value.create.assert_called_once()
 
 
 @pytest.mark.asyncio(loop_scope="session")
@@ -318,7 +303,6 @@ async def test_update_profile(mocker):
         description="Test description",
         links=["link1"],
         avatar_url="avatar.jpg",
-        is_featured=False,
     )
 
     # Call function
@@ -389,7 +373,7 @@ async def test_get_store_agents_with_search_and_filters_parameterized():
         creators=["creator1'; DROP TABLE Users; --", "creator2"],
         category="AI'; DELETE FROM StoreAgent; --",
         featured=True,
-        sorted_by="rating",
+        sorted_by=db.StoreAgentsSortOptions.RATING,
         page=1,
         page_size=20,
     )

autogpt_platform/backend/backend/api/features/store/exceptions.py

@@ -57,12 +57,6 @@ class StoreError(ValueError):
     pass
 
 
-class AgentNotFoundError(NotFoundError):
-    """Raised when an agent is not found"""
-
-    pass
-
-
 class CreatorNotFoundError(NotFoundError):
     """Raised when a creator is not found"""
 

autogpt_platform/backend/backend/api/features/store/hybrid_search.py

@@ -568,7 +568,7 @@ async def hybrid_search(
             SELECT uce."contentId" as "storeListingVersionId"
             FROM {{schema_prefix}}"UnifiedContentEmbedding" uce
             INNER JOIN {{schema_prefix}}"StoreAgent" sa
-                ON uce."contentId" = sa."storeListingVersionId"
+                ON uce."contentId" = sa.listing_version_id
             WHERE uce."contentType" = 'STORE_AGENT'::{{schema_prefix}}"ContentType"
             AND uce."userId" IS NULL
             AND uce.search @@ plainto_tsquery('english', {query_param})
@@ -582,7 +582,7 @@ async def hybrid_search(
                 SELECT uce."contentId", uce.embedding
                 FROM {{schema_prefix}}"UnifiedContentEmbedding" uce
                 INNER JOIN {{schema_prefix}}"StoreAgent" sa
-                    ON uce."contentId" = sa."storeListingVersionId"
+                    ON uce."contentId" = sa.listing_version_id
                 WHERE uce."contentType" = 'STORE_AGENT'::{{schema_prefix}}"ContentType"
                 AND uce."userId" IS NULL
                 AND {where_clause}
@@ -605,7 +605,7 @@ async def hybrid_search(
                 sa.featured,
                 sa.is_available,
                 sa.updated_at,
-                sa."agentGraphId",
+                sa.graph_id,
                 -- Searchable text for BM25 reranking
                 COALESCE(sa.agent_name, '') || ' ' || COALESCE(sa.sub_heading, '') || ' ' || COALESCE(sa.description, '') as searchable_text,
                 -- Semantic score
@@ -627,9 +627,9 @@ async def hybrid_search(
                 sa.runs as popularity_raw
             FROM candidates c
             INNER JOIN {{schema_prefix}}"StoreAgent" sa
-                ON c."storeListingVersionId" = sa."storeListingVersionId"
+                ON c."storeListingVersionId" = sa.listing_version_id
             INNER JOIN {{schema_prefix}}"UnifiedContentEmbedding" uce
-                ON sa."storeListingVersionId" = uce."contentId"
+                ON sa.listing_version_id = uce."contentId"
                 AND uce."contentType" = 'STORE_AGENT'::{{schema_prefix}}"ContentType"
         ),
         max_vals AS (
@@ -665,7 +665,7 @@ async def hybrid_search(
                 featured,
                 is_available,
                 updated_at,
-                "agentGraphId",
+                graph_id,
                 searchable_text,
                 semantic_score,
                 lexical_score,

autogpt_platform/backend/backend/api/features/store/model.py

@@ -1,11 +1,14 @@
 import datetime
-from typing import List
+from typing import TYPE_CHECKING, List, Self
 
 import prisma.enums
 import pydantic
 
 from backend.util.models import Pagination
 
+if TYPE_CHECKING:
+    import prisma.models
+
 
 class ChangelogEntry(pydantic.BaseModel):
     version: str
@@ -13,9 +16,9 @@ class ChangelogEntry(pydantic.BaseModel):
     date: datetime.datetime
 
 
-class MyAgent(pydantic.BaseModel):
-    agent_id: str
-    agent_version: int
+class MyUnpublishedAgent(pydantic.BaseModel):
+    graph_id: str
+    graph_version: int
     agent_name: str
     agent_image: str | None = None
     description: str
@@ -23,8 +26,8 @@ class MyAgent(pydantic.BaseModel):
     recommended_schedule_cron: str | None = None
 
 
-class MyAgentsResponse(pydantic.BaseModel):
-    agents: list[MyAgent]
+class MyUnpublishedAgentsResponse(pydantic.BaseModel):
+    agents: list[MyUnpublishedAgent]
     pagination: Pagination
 
 
@@ -40,6 +43,21 @@ class StoreAgent(pydantic.BaseModel):
     rating: float
     agent_graph_id: str
 
+    @classmethod
+    def from_db(cls, agent: "prisma.models.StoreAgent") -> "StoreAgent":
+        return cls(
+            slug=agent.slug,
+            agent_name=agent.agent_name,
+            agent_image=agent.agent_image[0] if agent.agent_image else "",
+            creator=agent.creator_username or "Needs Profile",
+            creator_avatar=agent.creator_avatar or "",
+            sub_heading=agent.sub_heading,
+            description=agent.description,
+            runs=agent.runs,
+            rating=agent.rating,
+            agent_graph_id=agent.graph_id,
+        )
+
 
 class StoreAgentsResponse(pydantic.BaseModel):
     agents: list[StoreAgent]
@@ -62,81 +80,192 @@ class StoreAgentDetails(pydantic.BaseModel):
     runs: int
     rating: float
     versions: list[str]
-    agentGraphVersions: list[str]
-    agentGraphId: str
+    graph_id: str
+    graph_versions: list[str]
     last_updated: datetime.datetime
     recommended_schedule_cron: str | None = None
 
-    active_version_id: str | None = None
-    has_approved_version: bool = False
+    active_version_id: str
+    has_approved_version: bool
 
     # Optional changelog data when include_changelog=True
     changelog: list[ChangelogEntry] | None = None
 
-
-class Creator(pydantic.BaseModel):
-    name: str
-    username: str
-    description: str
-    avatar_url: str
-    num_agents: int
-    agent_rating: float
-    agent_runs: int
-    is_featured: bool
-
-
-class CreatorsResponse(pydantic.BaseModel):
-    creators: List[Creator]
-    pagination: Pagination
-
-
-class CreatorDetails(pydantic.BaseModel):
-    name: str
-    username: str
-    description: str
-    links: list[str]
-    avatar_url: str
-    agent_rating: float
-    agent_runs: int
-    top_categories: list[str]
+    @classmethod
+    def from_db(cls, agent: "prisma.models.StoreAgent") -> "StoreAgentDetails":
+        return cls(
+            store_listing_version_id=agent.listing_version_id,
+            slug=agent.slug,
+            agent_name=agent.agent_name,
+            agent_video=agent.agent_video or "",
+            agent_output_demo=agent.agent_output_demo or "",
+            agent_image=agent.agent_image,
+            creator=agent.creator_username or "",
+            creator_avatar=agent.creator_avatar or "",
+            sub_heading=agent.sub_heading,
+            description=agent.description,
+            categories=agent.categories,
+            runs=agent.runs,
+            rating=agent.rating,
+            versions=agent.versions,
+            graph_id=agent.graph_id,
+            graph_versions=agent.graph_versions,
+            last_updated=agent.updated_at,
+            recommended_schedule_cron=agent.recommended_schedule_cron,
+            active_version_id=agent.listing_version_id,
+            has_approved_version=True,  # StoreAgent view only has approved agents
+        )
 
 
 class Profile(pydantic.BaseModel):
-    name: str
+    """Marketplace user profile (only attributes that the user can update)"""
+
     username: str
+    name: str
     description: str
+    avatar_url: str | None
     links: list[str]
-    avatar_url: str
-    is_featured: bool = False
+
+
+class ProfileDetails(Profile):
+    """Marketplace user profile (including read-only fields)"""
+
+    is_featured: bool
+
+    @classmethod
+    def from_db(cls, profile: "prisma.models.Profile") -> "ProfileDetails":
+        return cls(
+            name=profile.name,
+            username=profile.username,
+            avatar_url=profile.avatarUrl,
+            description=profile.description,
+            links=profile.links,
+            is_featured=profile.isFeatured,
+        )
+
+
+class CreatorDetails(ProfileDetails):
+    """Marketplace creator profile details, including aggregated stats"""
+
+    num_agents: int
+    agent_runs: int
+    agent_rating: float
+    top_categories: list[str]
+
+    @classmethod
+    def from_db(cls, creator: "prisma.models.Creator") -> "CreatorDetails":  # type: ignore[override]
+        return cls(
+            name=creator.name,
+            username=creator.username,
+            avatar_url=creator.avatar_url,
+            description=creator.description,
+            links=creator.links,
+            is_featured=creator.is_featured,
+            num_agents=creator.num_agents,
+            agent_runs=creator.agent_runs,
+            agent_rating=creator.agent_rating,
+            top_categories=creator.top_categories,
+        )
+
+
+class CreatorsResponse(pydantic.BaseModel):
+    creators: List[CreatorDetails]
+    pagination: Pagination
 
 
 class StoreSubmission(pydantic.BaseModel):
+    # From StoreListing:
     listing_id: str
-    agent_id: str
-    agent_version: int
+    user_id: str
+    slug: str
+
+    # From StoreListingVersion:
+    listing_version_id: str
+    listing_version: int
+    graph_id: str
+    graph_version: int
     name: str
     sub_heading: str
-    slug: str
     description: str
-    instructions: str | None = None
+    instructions: str | None
+    categories: list[str]
     image_urls: list[str]
-    date_submitted: datetime.datetime
-    status: prisma.enums.SubmissionStatus
-    runs: int
-    rating: float
-    store_listing_version_id: str | None = None
-    version: int | None = None  # Actual version number from the database
+    video_url: str | None
+    agent_output_demo_url: str | None
 
+    submitted_at: datetime.datetime | None
+    changes_summary: str | None
+    status: prisma.enums.SubmissionStatus
+    reviewed_at: datetime.datetime | None = None
     reviewer_id: str | None = None
     review_comments: str | None = None  # External comments visible to creator
-    internal_comments: str | None = None  # Private notes for admin use only
-    reviewed_at: datetime.datetime | None = None
-    changes_summary: str | None = None
 
-    # Additional fields for editing
-    video_url: str | None = None
-    agent_output_demo_url: str | None = None
-    categories: list[str] = []
+    # Aggregated from AgentGraphExecutions and StoreListingReviews:
+    run_count: int = 0
+    review_count: int = 0
+    review_avg_rating: float = 0.0
+
+    @classmethod
+    def from_db(cls, _sub: "prisma.models.StoreSubmission") -> Self:
+        """Construct from the StoreSubmission Prisma view."""
+        return cls(
+            listing_id=_sub.listing_id,
+            user_id=_sub.user_id,
+            slug=_sub.slug,
+            listing_version_id=_sub.listing_version_id,
+            listing_version=_sub.listing_version,
+            graph_id=_sub.graph_id,
+            graph_version=_sub.graph_version,
+            name=_sub.name,
+            sub_heading=_sub.sub_heading,
+            description=_sub.description,
+            instructions=_sub.instructions,
+            categories=_sub.categories,
+            image_urls=_sub.image_urls,
+            video_url=_sub.video_url,
+            agent_output_demo_url=_sub.agent_output_demo_url,
+            submitted_at=_sub.submitted_at,
+            changes_summary=_sub.changes_summary,
+            status=_sub.status,
+            reviewed_at=_sub.reviewed_at,
+            reviewer_id=_sub.reviewer_id,
+            review_comments=_sub.review_comments,
+            run_count=_sub.run_count,
+            review_count=_sub.review_count,
+            review_avg_rating=_sub.review_avg_rating,
+        )
+
+    @classmethod
+    def from_listing_version(cls, _lv: "prisma.models.StoreListingVersion") -> Self:
+        """
+        Construct from the StoreListingVersion Prisma model (with StoreListing included)
+        """
+        if not (_l := _lv.StoreListing):
+            raise ValueError("StoreListingVersion must have included StoreListing")
+
+        return cls(
+            listing_id=_l.id,
+            user_id=_l.owningUserId,
+            slug=_l.slug,
+            listing_version_id=_lv.id,
+            listing_version=_lv.version,
+            graph_id=_lv.agentGraphId,
+            graph_version=_lv.agentGraphVersion,
+            name=_lv.name,
+            sub_heading=_lv.subHeading,
+            description=_lv.description,
+            instructions=_lv.instructions,
+            categories=_lv.categories,
+            image_urls=_lv.imageUrls,
+            video_url=_lv.videoUrl,
+            agent_output_demo_url=_lv.agentOutputDemoUrl,
+            submitted_at=_lv.submittedAt,
+            changes_summary=_lv.changesSummary,
+            status=_lv.submissionStatus,
+            reviewed_at=_lv.reviewedAt,
+            reviewer_id=_lv.reviewerId,
+            review_comments=_lv.reviewComments,
+        )
 
 
 class StoreSubmissionsResponse(pydantic.BaseModel):
@@ -144,33 +273,12 @@ class StoreSubmissionsResponse(pydantic.BaseModel):
     pagination: Pagination
 
 
-class StoreListingWithVersions(pydantic.BaseModel):
-    """A store listing with its version history"""
-
-    listing_id: str
-    slug: str
-    agent_id: str
-    agent_version: int
-    active_version_id: str | None = None
-    has_approved_version: bool = False
-    creator_email: str | None = None
-    latest_version: StoreSubmission | None = None
-    versions: list[StoreSubmission] = []
-
-
-class StoreListingsWithVersionsResponse(pydantic.BaseModel):
-    """Response model for listings with version history"""
-
-    listings: list[StoreListingWithVersions]
-    pagination: Pagination
-
-
 class StoreSubmissionRequest(pydantic.BaseModel):
-    agent_id: str = pydantic.Field(
-        ..., min_length=1, description="Agent ID cannot be empty"
+    graph_id: str = pydantic.Field(
+        ..., min_length=1, description="Graph ID cannot be empty"
     )
-    agent_version: int = pydantic.Field(
-        ..., gt=0, description="Agent version must be greater than 0"
+    graph_version: int = pydantic.Field(
+        ..., gt=0, description="Graph version must be greater than 0"
     )
     slug: str
     name: str
@@ -198,12 +306,42 @@ class StoreSubmissionEditRequest(pydantic.BaseModel):
     recommended_schedule_cron: str | None = None
 
 
-class ProfileDetails(pydantic.BaseModel):
-    name: str
-    username: str
-    description: str
-    links: list[str]
-    avatar_url: str | None = None
+class StoreSubmissionAdminView(StoreSubmission):
+    internal_comments: str | None  # Private admin notes
+
+    @classmethod
+    def from_db(cls, _sub: "prisma.models.StoreSubmission") -> Self:
+        return cls(
+            **StoreSubmission.from_db(_sub).model_dump(),
+            internal_comments=_sub.internal_comments,
+        )
+
+    @classmethod
+    def from_listing_version(cls, _lv: "prisma.models.StoreListingVersion") -> Self:
+        return cls(
+            **StoreSubmission.from_listing_version(_lv).model_dump(),
+            internal_comments=_lv.internalComments,
+        )
+
+
+class StoreListingWithVersionsAdminView(pydantic.BaseModel):
+    """A store listing with its version history"""
+
+    listing_id: str
+    graph_id: str
+    slug: str
+    active_listing_version_id: str | None = None
+    has_approved_version: bool = False
+    creator_email: str | None = None
+    latest_version: StoreSubmissionAdminView | None = None
+    versions: list[StoreSubmissionAdminView] = []
+
+
+class StoreListingsWithVersionsAdminViewResponse(pydantic.BaseModel):
+    """Response model for listings with version history"""
+
+    listings: list[StoreListingWithVersionsAdminView]
+    pagination: Pagination
 
 
 class StoreReview(pydantic.BaseModel):

autogpt_platform/backend/backend/api/features/store/model_test.py

@@ -1,203 +0,0 @@
-import datetime
-
-import prisma.enums
-
-from . import model as store_model
-
-
-def test_pagination():
-    pagination = store_model.Pagination(
-        total_items=100, total_pages=5, current_page=2, page_size=20
-    )
-    assert pagination.total_items == 100
-    assert pagination.total_pages == 5
-    assert pagination.current_page == 2
-    assert pagination.page_size == 20
-
-
-def test_store_agent():
-    agent = store_model.StoreAgent(
-        slug="test-agent",
-        agent_name="Test Agent",
-        agent_image="test.jpg",
-        creator="creator1",
-        creator_avatar="avatar.jpg",
-        sub_heading="Test subheading",
-        description="Test description",
-        runs=50,
-        rating=4.5,
-        agent_graph_id="test-graph-id",
-    )
-    assert agent.slug == "test-agent"
-    assert agent.agent_name == "Test Agent"
-    assert agent.runs == 50
-    assert agent.rating == 4.5
-    assert agent.agent_graph_id == "test-graph-id"
-
-
-def test_store_agents_response():
-    response = store_model.StoreAgentsResponse(
-        agents=[
-            store_model.StoreAgent(
-                slug="test-agent",
-                agent_name="Test Agent",
-                agent_image="test.jpg",
-                creator="creator1",
-                creator_avatar="avatar.jpg",
-                sub_heading="Test subheading",
-                description="Test description",
-                runs=50,
-                rating=4.5,
-                agent_graph_id="test-graph-id",
-            )
-        ],
-        pagination=store_model.Pagination(
-            total_items=1, total_pages=1, current_page=1, page_size=20
-        ),
-    )
-    assert len(response.agents) == 1
-    assert response.pagination.total_items == 1
-
-
-def test_store_agent_details():
-    details = store_model.StoreAgentDetails(
-        store_listing_version_id="version123",
-        slug="test-agent",
-        agent_name="Test Agent",
-        agent_video="video.mp4",
-        agent_output_demo="demo.mp4",
-        agent_image=["image1.jpg", "image2.jpg"],
-        creator="creator1",
-        creator_avatar="avatar.jpg",
-        sub_heading="Test subheading",
-        description="Test description",
-        categories=["cat1", "cat2"],
-        runs=50,
-        rating=4.5,
-        versions=["1.0", "2.0"],
-        agentGraphVersions=["1", "2"],
-        agentGraphId="test-graph-id",
-        last_updated=datetime.datetime.now(),
-    )
-    assert details.slug == "test-agent"
-    assert len(details.agent_image) == 2
-    assert len(details.categories) == 2
-    assert len(details.versions) == 2
-
-
-def test_creator():
-    creator = store_model.Creator(
-        agent_rating=4.8,
-        agent_runs=1000,
-        name="Test Creator",
-        username="creator1",
-        description="Test description",
-        avatar_url="avatar.jpg",
-        num_agents=5,
-        is_featured=False,
-    )
-    assert creator.name == "Test Creator"
-    assert creator.num_agents == 5
-
-
-def test_creators_response():
-    response = store_model.CreatorsResponse(
-        creators=[
-            store_model.Creator(
-                agent_rating=4.8,
-                agent_runs=1000,
-                name="Test Creator",
-                username="creator1",
-                description="Test description",
-                avatar_url="avatar.jpg",
-                num_agents=5,
-                is_featured=False,
-            )
-        ],
-        pagination=store_model.Pagination(
-            total_items=1, total_pages=1, current_page=1, page_size=20
-        ),
-    )
-    assert len(response.creators) == 1
-    assert response.pagination.total_items == 1
-
-
-def test_creator_details():
-    details = store_model.CreatorDetails(
-        name="Test Creator",
-        username="creator1",
-        description="Test description",
-        links=["link1.com", "link2.com"],
-        avatar_url="avatar.jpg",
-        agent_rating=4.8,
-        agent_runs=1000,
-        top_categories=["cat1", "cat2"],
-    )
-    assert details.name == "Test Creator"
-    assert len(details.links) == 2
-    assert details.agent_rating == 4.8
-    assert len(details.top_categories) == 2
-
-
-def test_store_submission():
-    submission = store_model.StoreSubmission(
-        listing_id="listing123",
-        agent_id="agent123",
-        agent_version=1,
-        sub_heading="Test subheading",
-        name="Test Agent",
-        slug="test-agent",
-        description="Test description",
-        image_urls=["image1.jpg", "image2.jpg"],
-        date_submitted=datetime.datetime(2023, 1, 1),
-        status=prisma.enums.SubmissionStatus.PENDING,
-        runs=50,
-        rating=4.5,
-    )
-    assert submission.name == "Test Agent"
-    assert len(submission.image_urls) == 2
-    assert submission.status == prisma.enums.SubmissionStatus.PENDING
-
-
-def test_store_submissions_response():
-    response = store_model.StoreSubmissionsResponse(
-        submissions=[
-            store_model.StoreSubmission(
-                listing_id="listing123",
-                agent_id="agent123",
-                agent_version=1,
-                sub_heading="Test subheading",
-                name="Test Agent",
-                slug="test-agent",
-                description="Test description",
-                image_urls=["image1.jpg"],
-                date_submitted=datetime.datetime(2023, 1, 1),
-                status=prisma.enums.SubmissionStatus.PENDING,
-                runs=50,
-                rating=4.5,
-            )
-        ],
-        pagination=store_model.Pagination(
-            total_items=1, total_pages=1, current_page=1, page_size=20
-        ),
-    )
-    assert len(response.submissions) == 1
-    assert response.pagination.total_items == 1
-
-
-def test_store_submission_request():
-    request = store_model.StoreSubmissionRequest(
-        agent_id="agent123",
-        agent_version=1,
-        slug="test-agent",
-        name="Test Agent",
-        sub_heading="Test subheading",
-        video_url="video.mp4",
-        image_urls=["image1.jpg", "image2.jpg"],
-        description="Test description",
-        categories=["cat1", "cat2"],
-    )
-    assert request.agent_id == "agent123"
-    assert request.agent_version == 1
-    assert len(request.image_urls) == 2
-    assert len(request.categories) == 2

autogpt_platform/backend/backend/api/features/store/routes.py

@@ -1,16 +1,17 @@
 import logging
 import tempfile
-import typing
 import urllib.parse
-from typing import Literal
 
 import autogpt_libs.auth
 import fastapi
 import fastapi.responses
 import prisma.enums
+from fastapi import Query, Security
+from pydantic import BaseModel
 
 import backend.data.graph
 import backend.util.json
+from backend.util.exceptions import NotFoundError
 from backend.util.models import Pagination
 
 from . import cache as store_cache
@@ -34,22 +35,15 @@ router = fastapi.APIRouter()
     "/profile",
     summary="Get user profile",
     tags=["store", "private"],
-    dependencies=[fastapi.Security(autogpt_libs.auth.requires_user)],
-    response_model=store_model.ProfileDetails,
+    dependencies=[Security(autogpt_libs.auth.requires_user)],
 )
 async def get_profile(
-    user_id: str = fastapi.Security(autogpt_libs.auth.get_user_id),
-):
-    """
-    Get the profile details for the authenticated user.
-    Cached for 1 hour per user.
-    """
+    user_id: str = Security(autogpt_libs.auth.get_user_id),
+) -> store_model.ProfileDetails:
+    """Get the profile details for the authenticated user."""
     profile = await store_db.get_user_profile(user_id)
     if profile is None:
-        return fastapi.responses.JSONResponse(
-            status_code=404,
-            content={"detail": "Profile not found"},
-        )
+        raise NotFoundError("User does not have a profile yet")
     return profile
 
 
@@ -57,98 +51,17 @@ async def get_profile(
     "/profile",
     summary="Update user profile",
     tags=["store", "private"],
-    dependencies=[fastapi.Security(autogpt_libs.auth.requires_user)],
-    response_model=store_model.CreatorDetails,
+    dependencies=[Security(autogpt_libs.auth.requires_user)],
 )
 async def update_or_create_profile(
     profile: store_model.Profile,
-    user_id: str = fastapi.Security(autogpt_libs.auth.get_user_id),
-):
-    """
-    Update the store profile for the authenticated user.
-
-    Args:
-        profile (Profile): The updated profile details
-        user_id (str): ID of the authenticated user
-
-    Returns:
-        CreatorDetails: The updated profile
-
-    Raises:
-        HTTPException: If there is an error updating the profile
-    """
+    user_id: str = Security(autogpt_libs.auth.get_user_id),
+) -> store_model.ProfileDetails:
+    """Update the store profile for the authenticated user."""
     updated_profile = await store_db.update_profile(user_id=user_id, profile=profile)
     return updated_profile
 
 
-##############################################
-############### Agent Endpoints ##############
-##############################################
-
-
-@router.get(
-    "/agents",
-    summary="List store agents",
-    tags=["store", "public"],
-    response_model=store_model.StoreAgentsResponse,
-)
-async def get_agents(
-    featured: bool = False,
-    creator: str | None = None,
-    sorted_by: Literal["rating", "runs", "name", "updated_at"] | None = None,
-    search_query: str | None = None,
-    category: str | None = None,
-    page: int = 1,
-    page_size: int = 20,
-):
-    """
-    Get a paginated list of agents from the store with optional filtering and sorting.
-
-    Args:
-        featured (bool, optional): Filter to only show featured agents. Defaults to False.
-        creator (str | None, optional): Filter agents by creator username. Defaults to None.
-        sorted_by (str | None, optional): Sort agents by "runs" or "rating". Defaults to None.
-        search_query (str | None, optional): Search agents by name, subheading and description. Defaults to None.
-        category (str | None, optional): Filter agents by category. Defaults to None.
-        page (int, optional): Page number for pagination. Defaults to 1.
-        page_size (int, optional): Number of agents per page. Defaults to 20.
-
-    Returns:
-        StoreAgentsResponse: Paginated list of agents matching the filters
-
-    Raises:
-        HTTPException: If page or page_size are less than 1
-
-    Used for:
-    - Home Page Featured Agents
-    - Home Page Top Agents
-    - Search Results
-    - Agent Details - Other Agents By Creator
-    - Agent Details - Similar Agents
-    - Creator Details - Agents By Creator
-    """
-    if page < 1:
-        raise fastapi.HTTPException(
-            status_code=422, detail="Page must be greater than 0"
-        )
-
-    if page_size < 1:
-        raise fastapi.HTTPException(
-            status_code=422, detail="Page size must be greater than 0"
-        )
-
-    agents = await store_cache._get_cached_store_agents(
-        featured=featured,
-        creator=creator,
-        sorted_by=sorted_by,
-        search_query=search_query,
-        category=category,
-        page=page,
-        page_size=page_size,
-    )
-    return agents
-
-
 ##############################################
 ############### Search Endpoints #############
 ##############################################
@@ -158,60 +71,30 @@ async def get_agents(
     "/search",
     summary="Unified search across all content types",
     tags=["store", "public"],
-    response_model=store_model.UnifiedSearchResponse,
 )
 async def unified_search(
     query: str,
-    content_types: list[str] | None = fastapi.Query(
+    content_types: list[prisma.enums.ContentType] | None = Query(
         default=None,
-        description="Content types to search: STORE_AGENT, BLOCK, DOCUMENTATION. If not specified, searches all.",
+        description="Content types to search. If not specified, searches all.",
     ),
-    page: int = 1,
-    page_size: int = 20,
-    user_id: str | None = fastapi.Security(
+    page: int = Query(ge=1, default=1),
+    page_size: int = Query(ge=1, default=20),
+    user_id: str | None = Security(
         autogpt_libs.auth.get_optional_user_id, use_cache=False
     ),
-):
+) -> store_model.UnifiedSearchResponse:
     """
-    Search across all content types (store agents, blocks, documentation) using hybrid search.
+    Search across all content types (marketplace agents, blocks, documentation)
+    using hybrid search.
 
     Combines semantic (embedding-based) and lexical (text-based) search for best results.
-
-    Args:
-        query: The search query string
-        content_types: Optional list of content types to filter by (STORE_AGENT, BLOCK, DOCUMENTATION)
-        page: Page number for pagination (default 1)
-        page_size: Number of results per page (default 20)
-        user_id: Optional authenticated user ID (for user-scoped content in future)
-
-    Returns:
-        UnifiedSearchResponse: Paginated list of search results with relevance scores
     """
-    if page < 1:
-        raise fastapi.HTTPException(
-            status_code=422, detail="Page must be greater than 0"
-        )
-
-    if page_size < 1:
-        raise fastapi.HTTPException(
-            status_code=422, detail="Page size must be greater than 0"
-        )
-
-    # Convert string content types to enum
-    content_type_enums: list[prisma.enums.ContentType] | None = None
-    if content_types:
-        try:
-            content_type_enums = [prisma.enums.ContentType(ct) for ct in content_types]
-        except ValueError as e:
-            raise fastapi.HTTPException(
-                status_code=422,
-                detail=f"Invalid content type. Valid values: STORE_AGENT, BLOCK, DOCUMENTATION. Error: {e}",
-            )
 
     # Perform unified hybrid search
     results, total = await store_hybrid_search.unified_hybrid_search(
         query=query,
-        content_types=content_type_enums,
+        content_types=content_types,
         user_id=user_id,
         page=page,
         page_size=page_size,
@@ -245,22 +128,69 @@ async def unified_search(
     )
 
 
+##############################################
+############### Agent Endpoints ##############
+##############################################
+
+
+@router.get(
+    "/agents",
+    summary="List store agents",
+    tags=["store", "public"],
+)
+async def get_agents(
+    featured: bool = Query(
+        default=False, description="Filter to only show featured agents"
+    ),
+    creator: str | None = Query(
+        default=None, description="Filter agents by creator username"
+    ),
+    category: str | None = Query(default=None, description="Filter agents by category"),
+    search_query: str | None = Query(
+        default=None, description="Literal + semantic search on names and descriptions"
+    ),
+    sorted_by: store_db.StoreAgentsSortOptions | None = Query(
+        default=None,
+        description="Property to sort results by. Ignored if search_query is provided.",
+    ),
+    page: int = Query(ge=1, default=1),
+    page_size: int = Query(ge=1, default=20),
+) -> store_model.StoreAgentsResponse:
+    """
+    Get a paginated list of agents from the marketplace,
+    with optional filtering and sorting.
+
+    Used for:
+    - Home Page Featured Agents
+    - Home Page Top Agents
+    - Search Results
+    - Agent Details - Other Agents By Creator
+    - Agent Details - Similar Agents
+    - Creator Details - Agents By Creator
+    """
+    agents = await store_cache._get_cached_store_agents(
+        featured=featured,
+        creator=creator,
+        sorted_by=sorted_by,
+        search_query=search_query,
+        category=category,
+        page=page,
+        page_size=page_size,
+    )
+    return agents
+
+
 @router.get(
     "/agents/{username}/{agent_name}",
     summary="Get specific agent",
     tags=["store", "public"],
-    response_model=store_model.StoreAgentDetails,
 )
-async def get_agent(
+async def get_agent_by_name(
     username: str,
     agent_name: str,
-    include_changelog: bool = fastapi.Query(default=False),
-):
-    """
-    This is only used on the AgentDetails Page.
-
-    It returns the store listing agents details.
-    """
+    include_changelog: bool = Query(default=False),
+) -> store_model.StoreAgentDetails:
+    """Get details of a marketplace agent"""
     username = urllib.parse.unquote(username).lower()
     # URL decode the agent name since it comes from the URL path
     agent_name = urllib.parse.unquote(agent_name).lower()
@@ -270,76 +200,82 @@ async def get_agent(
     return agent
 
 
-@router.get(
-    "/graph/{store_listing_version_id}",
-    summary="Get agent graph",
-    tags=["store"],
-    dependencies=[fastapi.Security(autogpt_libs.auth.requires_user)],
-)
-async def get_graph_meta_by_store_listing_version_id(
-    store_listing_version_id: str,
-) -> backend.data.graph.GraphModelWithoutNodes:
-    """
-    Get Agent Graph from Store Listing Version ID.
-    """
-    graph = await store_db.get_available_graph(store_listing_version_id)
-    return graph
-
-
-@router.get(
-    "/agents/{store_listing_version_id}",
-    summary="Get agent by version",
-    tags=["store"],
-    dependencies=[fastapi.Security(autogpt_libs.auth.requires_user)],
-    response_model=store_model.StoreAgentDetails,
-)
-async def get_store_agent(store_listing_version_id: str):
-    """
-    Get Store Agent Details from Store Listing Version ID.
-    """
-    agent = await store_db.get_store_agent_by_version_id(store_listing_version_id)
-
-    return agent
-
-
 @router.post(
     "/agents/{username}/{agent_name}/review",
     summary="Create agent review",
     tags=["store"],
-    dependencies=[fastapi.Security(autogpt_libs.auth.requires_user)],
-    response_model=store_model.StoreReview,
+    dependencies=[Security(autogpt_libs.auth.requires_user)],
 )
-async def create_review(
+async def post_user_review_for_agent(
     username: str,
     agent_name: str,
     review: store_model.StoreReviewCreate,
-    user_id: str = fastapi.Security(autogpt_libs.auth.get_user_id),
-):
-    """
-    Create a review for a store agent.
-
-    Args:
-        username: Creator's username
-        agent_name: Name/slug of the agent
-        review: Review details including score and optional comments
-        user_id: ID of authenticated user creating the review
-
-    Returns:
-        The created review
-    """
+    user_id: str = Security(autogpt_libs.auth.get_user_id),
+) -> store_model.StoreReview:
+    """Post a user review on a marketplace agent listing"""
     username = urllib.parse.unquote(username).lower()
     agent_name = urllib.parse.unquote(agent_name).lower()
-    # Create the review
+
     created_review = await store_db.create_store_review(
         user_id=user_id,
         store_listing_version_id=review.store_listing_version_id,
         score=review.score,
         comments=review.comments,
     )
-
     return created_review
 
 
+@router.get(
+    "/listings/versions/{store_listing_version_id}",
+    summary="Get agent by version",
+    tags=["store"],
+    dependencies=[Security(autogpt_libs.auth.requires_user)],
+)
+async def get_agent_by_listing_version(
+    store_listing_version_id: str,
+) -> store_model.StoreAgentDetails:
+    agent = await store_db.get_store_agent_by_version_id(store_listing_version_id)
+    return agent
+
+
+@router.get(
+    "/listings/versions/{store_listing_version_id}/graph",
+    summary="Get agent graph",
+    tags=["store"],
+    dependencies=[Security(autogpt_libs.auth.requires_user)],
+)
+async def get_graph_meta_by_store_listing_version_id(
+    store_listing_version_id: str,
+) -> backend.data.graph.GraphModelWithoutNodes:
+    """Get outline of graph belonging to a specific marketplace listing version"""
+    graph = await store_db.get_available_graph(store_listing_version_id)
+    return graph
+
+
+@router.get(
+    "/listings/versions/{store_listing_version_id}/graph/download",
+    summary="Download agent file",
+    tags=["store", "public"],
+)
+async def download_agent_file(
+    store_listing_version_id: str,
+) -> fastapi.responses.FileResponse:
+    """Download agent graph file for a specific marketplace listing version"""
+    graph_data = await store_db.get_agent(store_listing_version_id)
+    file_name = f"agent_{graph_data.id}_v{graph_data.version or 'latest'}.json"
+
+    # Sending graph as a stream (similar to marketplace v1)
+    with tempfile.NamedTemporaryFile(
+        mode="w", suffix=".json", delete=False
+    ) as tmp_file:
+        tmp_file.write(backend.util.json.dumps(graph_data))
+        tmp_file.flush()
+
+        return fastapi.responses.FileResponse(
+            tmp_file.name, filename=file_name, media_type="application/json"
+        )
+
+
 ##############################################
 ############# Creator Endpoints #############
 ##############################################
@@ -349,37 +285,19 @@ async def create_review(
     "/creators",
     summary="List store creators",
     tags=["store", "public"],
-    response_model=store_model.CreatorsResponse,
 )
 async def get_creators(
-    featured: bool = False,
-    search_query: str | None = None,
-    sorted_by: Literal["agent_rating", "agent_runs", "num_agents"] | None = None,
-    page: int = 1,
-    page_size: int = 20,
-):
-    """
-    This is needed for:
-    - Home Page Featured Creators
-    - Search Results Page
-
-    ---
-
-    To support this functionality we need:
-    - featured: bool - to limit the list to just featured agents
-    - search_query: str - vector search based on the creators profile description.
-    - sorted_by: [agent_rating, agent_runs] -
-    """
-    if page < 1:
-        raise fastapi.HTTPException(
-            status_code=422, detail="Page must be greater than 0"
-        )
-
-    if page_size < 1:
-        raise fastapi.HTTPException(
-            status_code=422, detail="Page size must be greater than 0"
-        )
-
+    featured: bool = Query(
+        default=False, description="Filter to only show featured creators"
+    ),
+    search_query: str | None = Query(
+        default=None, description="Literal + semantic search on names and descriptions"
+    ),
+    sorted_by: store_db.StoreCreatorsSortOptions | None = None,
+    page: int = Query(ge=1, default=1),
+    page_size: int = Query(ge=1, default=20),
+) -> store_model.CreatorsResponse:
+    """List or search marketplace creators"""
     creators = await store_cache._get_cached_store_creators(
         featured=featured,
         search_query=search_query,
@@ -391,18 +309,12 @@ async def get_creators(
 
 
 @router.get(
-    "/creator/{username}",
+    "/creators/{username}",
     summary="Get creator details",
     tags=["store", "public"],
-    response_model=store_model.CreatorDetails,
 )
-async def get_creator(
-    username: str,
-):
-    """
-    Get the details of a creator.
-    - Creator Details Page
-    """
+async def get_creator(username: str) -> store_model.CreatorDetails:
+    """Get details on a marketplace creator"""
     username = urllib.parse.unquote(username).lower()
     creator = await store_cache._get_cached_creator_details(username=username)
     return creator
@@ -414,20 +326,17 @@ async def get_creator(
 
 
 @router.get(
-    "/myagents",
+    "/my-unpublished-agents",
     summary="Get my agents",
     tags=["store", "private"],
-    dependencies=[fastapi.Security(autogpt_libs.auth.requires_user)],
-    response_model=store_model.MyAgentsResponse,
+    dependencies=[Security(autogpt_libs.auth.requires_user)],
 )
-async def get_my_agents(
-    user_id: str = fastapi.Security(autogpt_libs.auth.get_user_id),
-    page: typing.Annotated[int, fastapi.Query(ge=1)] = 1,
-    page_size: typing.Annotated[int, fastapi.Query(ge=1)] = 20,
-):
-    """
-    Get user's own agents.
-    """
+async def get_my_unpublished_agents(
+    user_id: str = Security(autogpt_libs.auth.get_user_id),
+    page: int = Query(ge=1, default=1),
+    page_size: int = Query(ge=1, default=20),
+) -> store_model.MyUnpublishedAgentsResponse:
+    """List the authenticated user's unpublished agents"""
     agents = await store_db.get_my_agents(user_id, page=page, page_size=page_size)
     return agents
 
@@ -436,28 +345,17 @@ async def get_my_agents(
     "/submissions/{submission_id}",
     summary="Delete store submission",
     tags=["store", "private"],
-    dependencies=[fastapi.Security(autogpt_libs.auth.requires_user)],
-    response_model=bool,
+    dependencies=[Security(autogpt_libs.auth.requires_user)],
 )
 async def delete_submission(
     submission_id: str,
-    user_id: str = fastapi.Security(autogpt_libs.auth.get_user_id),
-):
-    """
-    Delete a store listing submission.
-
-    Args:
-        user_id (str): ID of the authenticated user
-        submission_id (str): ID of the submission to be deleted
-
-    Returns:
-        bool: True if the submission was successfully deleted, False otherwise
-    """
+    user_id: str = Security(autogpt_libs.auth.get_user_id),
+) -> bool:
+    """Delete a marketplace listing submission"""
     result = await store_db.delete_store_submission(
         user_id=user_id,
         submission_id=submission_id,
     )
-
     return result
 
 
@@ -465,37 +363,14 @@ async def delete_submission(
     "/submissions",
     summary="List my submissions",
     tags=["store", "private"],
-    dependencies=[fastapi.Security(autogpt_libs.auth.requires_user)],
-    response_model=store_model.StoreSubmissionsResponse,
+    dependencies=[Security(autogpt_libs.auth.requires_user)],
 )
 async def get_submissions(
-    user_id: str = fastapi.Security(autogpt_libs.auth.get_user_id),
-    page: int = 1,
-    page_size: int = 20,
-):
-    """
-    Get a paginated list of store submissions for the authenticated user.
-
-    Args:
-        user_id (str): ID of the authenticated user
-        page (int, optional): Page number for pagination. Defaults to 1.
-        page_size (int, optional): Number of submissions per page. Defaults to 20.
-
-    Returns:
-        StoreListingsResponse: Paginated list of store submissions
-
-    Raises:
-        HTTPException: If page or page_size are less than 1
-    """
-    if page < 1:
-        raise fastapi.HTTPException(
-            status_code=422, detail="Page must be greater than 0"
-        )
-
-    if page_size < 1:
-        raise fastapi.HTTPException(
-            status_code=422, detail="Page size must be greater than 0"
-        )
+    user_id: str = Security(autogpt_libs.auth.get_user_id),
+    page: int = Query(ge=1, default=1),
+    page_size: int = Query(ge=1, default=20),
+) -> store_model.StoreSubmissionsResponse:
+    """List the authenticated user's marketplace listing submissions"""
     listings = await store_db.get_store_submissions(
         user_id=user_id,
         page=page,
@@ -508,30 +383,17 @@ async def get_submissions(
     "/submissions",
     summary="Create store submission",
     tags=["store", "private"],
-    dependencies=[fastapi.Security(autogpt_libs.auth.requires_user)],
-    response_model=store_model.StoreSubmission,
+    dependencies=[Security(autogpt_libs.auth.requires_user)],
 )
 async def create_submission(
     submission_request: store_model.StoreSubmissionRequest,
-    user_id: str = fastapi.Security(autogpt_libs.auth.get_user_id),
-):
-    """
-    Create a new store listing submission.
-
-    Args:
-        submission_request (StoreSubmissionRequest): The submission details
-        user_id (str): ID of the authenticated user submitting the listing
-
-    Returns:
-        StoreSubmission: The created store submission
-
-    Raises:
-        HTTPException: If there is an error creating the submission
-    """
+    user_id: str = Security(autogpt_libs.auth.get_user_id),
+) -> store_model.StoreSubmission:
+    """Submit a new marketplace listing for review"""
     result = await store_db.create_store_submission(
         user_id=user_id,
-        agent_id=submission_request.agent_id,
-        agent_version=submission_request.agent_version,
+        graph_id=submission_request.graph_id,
+        graph_version=submission_request.graph_version,
         slug=submission_request.slug,
         name=submission_request.name,
         video_url=submission_request.video_url,
@@ -544,7 +406,6 @@ async def create_submission(
         changes_summary=submission_request.changes_summary or "Initial Submission",
         recommended_schedule_cron=submission_request.recommended_schedule_cron,
     )
-
     return result
 
 
@@ -552,28 +413,14 @@ async def create_submission(
     "/submissions/{store_listing_version_id}",
     summary="Edit store submission",
     tags=["store", "private"],
-    dependencies=[fastapi.Security(autogpt_libs.auth.requires_user)],
-    response_model=store_model.StoreSubmission,
+    dependencies=[Security(autogpt_libs.auth.requires_user)],
 )
 async def edit_submission(
     store_listing_version_id: str,
     submission_request: store_model.StoreSubmissionEditRequest,
-    user_id: str = fastapi.Security(autogpt_libs.auth.get_user_id),
-):
-    """
-    Edit an existing store listing submission.
-
-    Args:
-        store_listing_version_id (str): ID of the store listing version to edit
-        submission_request (StoreSubmissionRequest): The updated submission details
-        user_id (str): ID of the authenticated user editing the listing
-
-    Returns:
-        StoreSubmission: The updated store submission
-
-    Raises:
-        HTTPException: If there is an error editing the submission
-    """
+    user_id: str = Security(autogpt_libs.auth.get_user_id),
+) -> store_model.StoreSubmission:
+    """Update a pending marketplace listing submission"""
     result = await store_db.edit_store_submission(
         user_id=user_id,
         store_listing_version_id=store_listing_version_id,
@@ -588,7 +435,6 @@ async def edit_submission(
         changes_summary=submission_request.changes_summary,
         recommended_schedule_cron=submission_request.recommended_schedule_cron,
     )
-
     return result
 
 
@@ -596,115 +442,61 @@ async def edit_submission(
     "/submissions/media",
     summary="Upload submission media",
     tags=["store", "private"],
-    dependencies=[fastapi.Security(autogpt_libs.auth.requires_user)],
+    dependencies=[Security(autogpt_libs.auth.requires_user)],
 )
 async def upload_submission_media(
     file: fastapi.UploadFile,
-    user_id: str = fastapi.Security(autogpt_libs.auth.get_user_id),
-):
-    """
-    Upload media (images/videos) for a store listing submission.
-
-    Args:
-        file (UploadFile): The media file to upload
-        user_id (str): ID of the authenticated user uploading the media
-
-    Returns:
-        str: URL of the uploaded media file
-
-    Raises:
-        HTTPException: If there is an error uploading the media
-    """
+    user_id: str = Security(autogpt_libs.auth.get_user_id),
+) -> str:
+    """Upload media for a marketplace listing submission"""
     media_url = await store_media.upload_media(user_id=user_id, file=file)
     return media_url
 
 
+class ImageURLResponse(BaseModel):
+    image_url: str
+
+
 @router.post(
     "/submissions/generate_image",
     summary="Generate submission image",
     tags=["store", "private"],
-    dependencies=[fastapi.Security(autogpt_libs.auth.requires_user)],
+    dependencies=[Security(autogpt_libs.auth.requires_user)],
 )
 async def generate_image(
-    agent_id: str,
-    user_id: str = fastapi.Security(autogpt_libs.auth.get_user_id),
-) -> fastapi.responses.Response:
+    graph_id: str,
+    user_id: str = Security(autogpt_libs.auth.get_user_id),
+) -> ImageURLResponse:
     """
-    Generate an image for a store listing submission.
-
-    Args:
-        agent_id (str): ID of the agent to generate an image for
-        user_id (str): ID of the authenticated user
-
-    Returns:
-        JSONResponse: JSON containing the URL of the generated image
+    Generate an image for a marketplace listing submission based on the properties
+    of a given graph.
     """
-    agent = await backend.data.graph.get_graph(
-        graph_id=agent_id, version=None, user_id=user_id
+    graph = await backend.data.graph.get_graph(
+        graph_id=graph_id, version=None, user_id=user_id
     )
 
-    if not agent:
-        raise fastapi.HTTPException(
-            status_code=404, detail=f"Agent with ID {agent_id} not found"
-        )
+    if not graph:
+        raise NotFoundError(f"Agent graph #{graph_id} not found")
     # Use .jpeg here since we are generating JPEG images
-    filename = f"agent_{agent_id}.jpeg"
+    filename = f"agent_{graph_id}.jpeg"
 
     existing_url = await store_media.check_media_exists(user_id, filename)
     if existing_url:
-        logger.info(f"Using existing image for agent {agent_id}")
-        return fastapi.responses.JSONResponse(content={"image_url": existing_url})
+        logger.info(f"Using existing image for agent graph {graph_id}")
+        return ImageURLResponse(image_url=existing_url)
     # Generate agent image as JPEG
-    image = await store_image_gen.generate_agent_image(agent=agent)
+    image = await store_image_gen.generate_agent_image(agent=graph)
 
     # Create UploadFile with the correct filename and content_type
     image_file = fastapi.UploadFile(
         file=image,
         filename=filename,
     )
-
     image_url = await store_media.upload_media(
         user_id=user_id, file=image_file, use_file_name=True
     )
 
-    return fastapi.responses.JSONResponse(content={"image_url": image_url})
-
-
-@router.get(
-    "/download/agents/{store_listing_version_id}",
-    summary="Download agent file",
-    tags=["store", "public"],
-)
-async def download_agent_file(
-    store_listing_version_id: str = fastapi.Path(
-        ..., description="The ID of the agent to download"
-    ),
-) -> fastapi.responses.FileResponse:
-    """
-    Download the agent file by streaming its content.
-
-    Args:
-        store_listing_version_id (str): The ID of the agent to download
-
-    Returns:
-        StreamingResponse: A streaming response containing the agent's graph data.
-
-    Raises:
-        HTTPException: If the agent is not found or an unexpected error occurs.
-    """
-    graph_data = await store_db.get_agent(store_listing_version_id)
-    file_name = f"agent_{graph_data.id}_v{graph_data.version or 'latest'}.json"
-
-    # Sending graph as a stream (similar to marketplace v1)
-    with tempfile.NamedTemporaryFile(
-        mode="w", suffix=".json", delete=False
-    ) as tmp_file:
-        tmp_file.write(backend.util.json.dumps(graph_data))
-        tmp_file.flush()
-
-        return fastapi.responses.FileResponse(
-            tmp_file.name, filename=file_name, media_type="application/json"
-        )
+    return ImageURLResponse(image_url=image_url)
 
 
 ##############################################

autogpt_platform/backend/backend/api/features/store/routes_test.py

@@ -8,6 +8,8 @@ import pytest
 import pytest_mock
 from pytest_snapshot.plugin import Snapshot
 
+from backend.api.features.store.db import StoreAgentsSortOptions
+
 from . import model as store_model
 from . import routes as store_routes
 
@@ -196,7 +198,7 @@ def test_get_agents_sorted(
     mock_db_call.assert_called_once_with(
         featured=False,
         creators=None,
-        sorted_by="runs",
+        sorted_by=StoreAgentsSortOptions.RUNS,
         search_query=None,
         category=None,
         page=1,
@@ -380,9 +382,11 @@ def test_get_agent_details(
         runs=100,
         rating=4.5,
         versions=["1.0.0", "1.1.0"],
-        agentGraphVersions=["1", "2"],
-        agentGraphId="test-graph-id",
+        graph_versions=["1", "2"],
+        graph_id="test-graph-id",
         last_updated=FIXED_NOW,
+        active_version_id="test-version-id",
+        has_approved_version=True,
     )
     mock_db_call = mocker.patch("backend.api.features.store.db.get_store_agent_details")
     mock_db_call.return_value = mocked_value
@@ -435,15 +439,17 @@ def test_get_creators_pagination(
 ) -> None:
     mocked_value = store_model.CreatorsResponse(
         creators=[
-            store_model.Creator(
+            store_model.CreatorDetails(
                 name=f"Creator {i}",
                 username=f"creator{i}",
-                description=f"Creator {i} description",
                 avatar_url=f"avatar{i}.jpg",
-                num_agents=1,
-                agent_rating=4.5,
-                agent_runs=100,
+                description=f"Creator {i} description",
+                links=[f"user{i}.link.com"],
                 is_featured=False,
+                num_agents=1,
+                agent_runs=100,
+                agent_rating=4.5,
+                top_categories=["cat1", "cat2", "cat3"],
             )
             for i in range(5)
         ],
@@ -496,19 +502,19 @@ def test_get_creator_details(
     mocked_value = store_model.CreatorDetails(
         name="Test User",
         username="creator1",
+        avatar_url="avatar.jpg",
         description="Test creator description",
         links=["link1.com", "link2.com"],
-        avatar_url="avatar.jpg",
-        agent_rating=4.8,
+        is_featured=True,
+        num_agents=5,
         agent_runs=1000,
+        agent_rating=4.8,
         top_categories=["category1", "category2"],
     )
-    mock_db_call = mocker.patch(
-        "backend.api.features.store.db.get_store_creator_details"
-    )
+    mock_db_call = mocker.patch("backend.api.features.store.db.get_store_creator")
     mock_db_call.return_value = mocked_value
 
-    response = client.get("/creator/creator1")
+    response = client.get("/creators/creator1")
     assert response.status_code == 200
 
     data = store_model.CreatorDetails.model_validate(response.json())
@@ -528,19 +534,26 @@ def test_get_submissions_success(
         submissions=[
             store_model.StoreSubmission(
                 listing_id="test-listing-id",
-                name="Test Agent",
-                description="Test agent description",
-                image_urls=["test.jpg"],
-                date_submitted=FIXED_NOW,
-                status=prisma.enums.SubmissionStatus.APPROVED,
-                runs=50,
-                rating=4.2,
-                agent_id="test-agent-id",
-                agent_version=1,
-                sub_heading="Test agent subheading",
+                user_id="test-user-id",
                 slug="test-agent",
-                video_url="test.mp4",
+                listing_version_id="test-version-id",
+                listing_version=1,
+                graph_id="test-agent-id",
+                graph_version=1,
+                name="Test Agent",
+                sub_heading="Test agent subheading",
+                description="Test agent description",
+                instructions="Click the button!",
                 categories=["test-category"],
+                image_urls=["test.jpg"],
+                video_url="test.mp4",
+                agent_output_demo_url="demo_video.mp4",
+                submitted_at=FIXED_NOW,
+                changes_summary="Initial Submission",
+                status=prisma.enums.SubmissionStatus.APPROVED,
+                run_count=50,
+                review_count=5,
+                review_avg_rating=4.2,
             )
         ],
         pagination=store_model.Pagination(

autogpt_platform/backend/backend/api/features/store/test_cache_delete.py

@@ -11,6 +11,7 @@ import pytest
 from backend.util.models import Pagination
 
 from . import cache as store_cache
+from .db import StoreAgentsSortOptions
 from .model import StoreAgent, StoreAgentsResponse
 
 
@@ -215,7 +216,7 @@ class TestCacheDeletion:
             await store_cache._get_cached_store_agents(
                 featured=True,
                 creator="testuser",
-                sorted_by="rating",
+                sorted_by=StoreAgentsSortOptions.RATING,
                 search_query="AI assistant",
                 category="productivity",
                 page=2,
@@ -227,7 +228,7 @@ class TestCacheDeletion:
             deleted = store_cache._get_cached_store_agents.cache_delete(
                 featured=True,
                 creator="testuser",
-                sorted_by="rating",
+                sorted_by=StoreAgentsSortOptions.RATING,
                 search_query="AI assistant",
                 category="productivity",
                 page=2,
@@ -239,7 +240,7 @@ class TestCacheDeletion:
             deleted = store_cache._get_cached_store_agents.cache_delete(
                 featured=True,
                 creator="testuser",
-                sorted_by="rating",
+                sorted_by=StoreAgentsSortOptions.RATING,
                 search_query="AI assistant",
                 category="productivity",
                 page=2,

autogpt_platform/backend/backend/api/features/v1.py

@@ -55,6 +55,7 @@ from backend.data.credit import (
     set_auto_top_up,
 )
 from backend.data.graph import GraphSettings
+from backend.data.invited_user import get_or_activate_user
 from backend.data.model import CredentialsMetaInput, UserOnboarding
 from backend.data.notifications import NotificationPreference, NotificationPreferenceDTO
 from backend.data.onboarding import (
@@ -70,7 +71,6 @@ from backend.data.onboarding import (
     update_user_onboarding,
 )
 from backend.data.user import (
-    get_or_create_user,
     get_user_by_id,
     get_user_notification_preference,
     update_user_email,
@@ -126,6 +126,9 @@ v1_router = APIRouter()
 ########################################################
 
 
+_tally_background_tasks: set[asyncio.Task] = set()
+
+
 @v1_router.post(
     "/auth/user",
     summary="Get or create user",
@@ -133,7 +136,23 @@ v1_router = APIRouter()
     dependencies=[Security(requires_user)],
 )
 async def get_or_create_user_route(user_data: dict = Security(get_jwt_payload)):
-    user = await get_or_create_user(user_data)
+    user = await get_or_activate_user(user_data)
+
+    # Fire-and-forget: backfill Tally understanding when invite pre-seeding did
+    # not produce a stored result before first activation.
+    age_seconds = (datetime.now(timezone.utc) - user.created_at).total_seconds()
+    if age_seconds < 30:
+        try:
+            from backend.data.tally import populate_understanding_from_tally
+
+            task = asyncio.create_task(
+                populate_understanding_from_tally(user.id, user.email)
+            )
+            _tally_background_tasks.add(task)
+            task.add_done_callback(_tally_background_tasks.discard)
+        except Exception:
+            logger.debug("Failed to start Tally population task", exc_info=True)
+
     return user.model_dump()
 
 
@@ -144,7 +163,8 @@ async def get_or_create_user_route(user_data: dict = Security(get_jwt_payload)):
     dependencies=[Security(requires_user)],
 )
 async def update_user_email_route(
-    user_id: Annotated[str, Security(get_user_id)], email: str = Body(...)
+    user_id: Annotated[str, Security(get_user_id)],
+    email: str = Body(...),
 ) -> dict[str, str]:
     await update_user_email(user_id, email)
 
@@ -158,10 +178,16 @@ async def update_user_email_route(
     dependencies=[Security(requires_user)],
 )
 async def get_user_timezone_route(
-    user_data: dict = Security(get_jwt_payload),
+    user_id: Annotated[str, Security(get_user_id)],
 ) -> TimezoneResponse:
     """Get user timezone setting."""
-    user = await get_or_create_user(user_data)
+    try:
+        user = await get_user_by_id(user_id)
+    except ValueError:
+        raise HTTPException(
+            status_code=HTTP_404_NOT_FOUND,
+            detail="User not found. Please complete activation via /auth/user first.",
+        )
     return TimezoneResponse(timezone=user.timezone)
 
 
@@ -172,7 +198,8 @@ async def get_user_timezone_route(
     dependencies=[Security(requires_user)],
 )
 async def update_user_timezone_route(
-    user_id: Annotated[str, Security(get_user_id)], request: UpdateTimezoneRequest
+    user_id: Annotated[str, Security(get_user_id)],
+    request: UpdateTimezoneRequest,
 ) -> TimezoneResponse:
     """Update user timezone. The timezone should be a valid IANA timezone identifier."""
     user = await update_user_timezone(user_id, str(request.timezone))
@@ -428,7 +455,6 @@ async def execute_graph_block(
 async def upload_file(
     user_id: Annotated[str, Security(get_user_id)],
     file: UploadFile = File(...),
-    provider: str = "gcs",
     expiration_hours: int = 24,
 ) -> UploadFileResponse:
     """
@@ -491,7 +517,6 @@ async def upload_file(
     storage_path = await cloud_storage.store_file(
         content=content,
         filename=file_name,
-        provider=provider,
         expiration_hours=expiration_hours,
         user_id=user_id,
     )

autogpt_platform/backend/backend/api/features/v1_test.py

@@ -1,5 +1,5 @@
 import json
-from datetime import datetime
+from datetime import datetime, timezone
 from io import BytesIO
 from unittest.mock import AsyncMock, Mock, patch
 
@@ -43,6 +43,7 @@ def test_get_or_create_user_route(
 ) -> None:
     """Test get or create user endpoint"""
     mock_user = Mock()
+    mock_user.created_at = datetime.now(timezone.utc)
     mock_user.model_dump.return_value = {
         "id": test_user_id,
         "email": "test@example.com",
@@ -50,7 +51,7 @@ def test_get_or_create_user_route(
     }
 
     mocker.patch(
-        "backend.api.features.v1.get_or_create_user",
+        "backend.api.features.v1.get_or_activate_user",
         return_value=mock_user,
     )
 
@@ -514,7 +515,6 @@ async def test_upload_file_success(test_user_id: str):
         result = await upload_file(
             file=upload_file_mock,
             user_id=test_user_id,
-            provider="gcs",
             expiration_hours=24,
         )
 
@@ -532,7 +532,6 @@ async def test_upload_file_success(test_user_id: str):
         mock_handler.store_file.assert_called_once_with(
             content=file_content,
             filename="test.txt",
-            provider="gcs",
             expiration_hours=24,
             user_id=test_user_id,
         )

autogpt_platform/backend/backend/api/features/workspace/routes.py

@@ -3,15 +3,29 @@ Workspace API routes for managing user file storage.
 """
 
 import logging
+import os
 import re
 from typing import Annotated
 from urllib.parse import quote
 
 import fastapi
 from autogpt_libs.auth.dependencies import get_user_id, requires_user
+from fastapi import Query, UploadFile
 from fastapi.responses import Response
+from pydantic import BaseModel
 
-from backend.data.workspace import get_workspace, get_workspace_file
+from backend.data.workspace import (
+    WorkspaceFile,
+    count_workspace_files,
+    get_or_create_workspace,
+    get_workspace,
+    get_workspace_file,
+    get_workspace_total_size,
+    soft_delete_workspace_file,
+)
+from backend.util.settings import Config
+from backend.util.virus_scanner import scan_content_safe
+from backend.util.workspace import WorkspaceManager
 from backend.util.workspace_storage import get_workspace_storage
 
 
@@ -44,11 +58,11 @@ router = fastapi.APIRouter(
 )
 
 
-def _create_streaming_response(content: bytes, file) -> Response:
+def _create_streaming_response(content: bytes, file: WorkspaceFile) -> Response:
     """Create a streaming response for file content."""
     return Response(
         content=content,
-        media_type=file.mimeType,
+        media_type=file.mime_type,
         headers={
             "Content-Disposition": _sanitize_filename_for_header(file.name),
             "Content-Length": str(len(content)),
@@ -56,7 +70,7 @@ def _create_streaming_response(content: bytes, file) -> Response:
     )
 
 
-async def _create_file_download_response(file) -> Response:
+async def _create_file_download_response(file: WorkspaceFile) -> Response:
     """
     Create a download response for a workspace file.
 
@@ -66,38 +80,57 @@ async def _create_file_download_response(file) -> Response:
     storage = await get_workspace_storage()
 
     # For local storage, stream the file directly
-    if file.storagePath.startswith("local://"):
-        content = await storage.retrieve(file.storagePath)
+    if file.storage_path.startswith("local://"):
+        content = await storage.retrieve(file.storage_path)
         return _create_streaming_response(content, file)
 
     # For GCS, try to redirect to signed URL, fall back to streaming
     try:
-        url = await storage.get_download_url(file.storagePath, expires_in=300)
+        url = await storage.get_download_url(file.storage_path, expires_in=300)
         # If we got back an API path (fallback), stream directly instead
         if url.startswith("/api/"):
-            content = await storage.retrieve(file.storagePath)
+            content = await storage.retrieve(file.storage_path)
             return _create_streaming_response(content, file)
         return fastapi.responses.RedirectResponse(url=url, status_code=302)
     except Exception as e:
         # Log the signed URL failure with context
         logger.error(
             f"Failed to get signed URL for file {file.id} "
-            f"(storagePath={file.storagePath}): {e}",
+            f"(storagePath={file.storage_path}): {e}",
             exc_info=True,
         )
         # Fall back to streaming directly from GCS
         try:
-            content = await storage.retrieve(file.storagePath)
+            content = await storage.retrieve(file.storage_path)
             return _create_streaming_response(content, file)
         except Exception as fallback_error:
             logger.error(
                 f"Fallback streaming also failed for file {file.id} "
-                f"(storagePath={file.storagePath}): {fallback_error}",
+                f"(storagePath={file.storage_path}): {fallback_error}",
                 exc_info=True,
             )
             raise
 
 
+class UploadFileResponse(BaseModel):
+    file_id: str
+    name: str
+    path: str
+    mime_type: str
+    size_bytes: int
+
+
+class DeleteFileResponse(BaseModel):
+    deleted: bool
+
+
+class StorageUsageResponse(BaseModel):
+    used_bytes: int
+    limit_bytes: int
+    used_percent: float
+    file_count: int
+
+
 @router.get(
     "/files/{file_id}/download",
     summary="Download file by ID",
@@ -120,3 +153,148 @@ async def download_file(
         raise fastapi.HTTPException(status_code=404, detail="File not found")
 
     return await _create_file_download_response(file)
+
+
+@router.delete(
+    "/files/{file_id}",
+    summary="Delete a workspace file",
+)
+async def delete_workspace_file(
+    user_id: Annotated[str, fastapi.Security(get_user_id)],
+    file_id: str,
+) -> DeleteFileResponse:
+    """
+    Soft-delete a workspace file and attempt to remove it from storage.
+
+    Used when a user clears a file input in the builder.
+    """
+    workspace = await get_workspace(user_id)
+    if workspace is None:
+        raise fastapi.HTTPException(status_code=404, detail="Workspace not found")
+
+    manager = WorkspaceManager(user_id, workspace.id)
+    deleted = await manager.delete_file(file_id)
+    if not deleted:
+        raise fastapi.HTTPException(status_code=404, detail="File not found")
+
+    return DeleteFileResponse(deleted=True)
+
+
+@router.post(
+    "/files/upload",
+    summary="Upload file to workspace",
+)
+async def upload_file(
+    user_id: Annotated[str, fastapi.Security(get_user_id)],
+    file: UploadFile,
+    session_id: str | None = Query(default=None),
+) -> UploadFileResponse:
+    """
+    Upload a file to the user's workspace.
+
+    Files are stored in session-scoped paths when session_id is provided,
+    so the agent's session-scoped tools can discover them automatically.
+    """
+    config = Config()
+
+    # Sanitize filename — strip any directory components
+    filename = os.path.basename(file.filename or "upload") or "upload"
+
+    # Read file content with early abort on size limit
+    max_file_bytes = config.max_file_size_mb * 1024 * 1024
+    chunks: list[bytes] = []
+    total_size = 0
+    while chunk := await file.read(64 * 1024):  # 64KB chunks
+        total_size += len(chunk)
+        if total_size > max_file_bytes:
+            raise fastapi.HTTPException(
+                status_code=413,
+                detail=f"File exceeds maximum size of {config.max_file_size_mb} MB",
+            )
+        chunks.append(chunk)
+    content = b"".join(chunks)
+
+    # Get or create workspace
+    workspace = await get_or_create_workspace(user_id)
+
+    # Pre-write storage cap check (soft check — final enforcement is post-write)
+    storage_limit_bytes = config.max_workspace_storage_mb * 1024 * 1024
+    current_usage = await get_workspace_total_size(workspace.id)
+    if storage_limit_bytes and current_usage + len(content) > storage_limit_bytes:
+        used_percent = (current_usage / storage_limit_bytes) * 100
+        raise fastapi.HTTPException(
+            status_code=413,
+            detail={
+                "message": "Storage limit exceeded",
+                "used_bytes": current_usage,
+                "limit_bytes": storage_limit_bytes,
+                "used_percent": round(used_percent, 1),
+            },
+        )
+
+    # Warn at 80% usage
+    if (
+        storage_limit_bytes
+        and (usage_ratio := (current_usage + len(content)) / storage_limit_bytes) >= 0.8
+    ):
+        logger.warning(
+            f"User {user_id} workspace storage at {usage_ratio * 100:.1f}% "
+            f"({current_usage + len(content)} / {storage_limit_bytes} bytes)"
+        )
+
+    # Virus scan
+    await scan_content_safe(content, filename=filename)
+
+    # Write file via WorkspaceManager
+    manager = WorkspaceManager(user_id, workspace.id, session_id)
+    try:
+        workspace_file = await manager.write_file(content, filename)
+    except ValueError as e:
+        raise fastapi.HTTPException(status_code=409, detail=str(e)) from e
+
+    # Post-write storage check — eliminates TOCTOU race on the quota.
+    # If a concurrent upload pushed us over the limit, undo this write.
+    new_total = await get_workspace_total_size(workspace.id)
+    if storage_limit_bytes and new_total > storage_limit_bytes:
+        await soft_delete_workspace_file(workspace_file.id, workspace.id)
+        raise fastapi.HTTPException(
+            status_code=413,
+            detail={
+                "message": "Storage limit exceeded (concurrent upload)",
+                "used_bytes": new_total,
+                "limit_bytes": storage_limit_bytes,
+            },
+        )
+
+    return UploadFileResponse(
+        file_id=workspace_file.id,
+        name=workspace_file.name,
+        path=workspace_file.path,
+        mime_type=workspace_file.mime_type,
+        size_bytes=workspace_file.size_bytes,
+    )
+
+
+@router.get(
+    "/storage/usage",
+    summary="Get workspace storage usage",
+)
+async def get_storage_usage(
+    user_id: Annotated[str, fastapi.Security(get_user_id)],
+) -> StorageUsageResponse:
+    """
+    Get storage usage information for the user's workspace.
+    """
+    config = Config()
+    workspace = await get_or_create_workspace(user_id)
+
+    used_bytes = await get_workspace_total_size(workspace.id)
+    file_count = await count_workspace_files(workspace.id)
+    limit_bytes = config.max_workspace_storage_mb * 1024 * 1024
+
+    return StorageUsageResponse(
+        used_bytes=used_bytes,
+        limit_bytes=limit_bytes,
+        used_percent=round((used_bytes / limit_bytes) * 100, 1) if limit_bytes else 0,
+        file_count=file_count,
+    )

autogpt_platform/backend/backend/api/features/workspace/routes_test.py

@@ -0,0 +1,359 @@
+"""Tests for workspace file upload and download routes."""
+
+import io
+from datetime import datetime, timezone
+
+import fastapi
+import fastapi.testclient
+import pytest
+import pytest_mock
+
+from backend.api.features.workspace import routes as workspace_routes
+from backend.data.workspace import WorkspaceFile
+
+app = fastapi.FastAPI()
+app.include_router(workspace_routes.router)
+
+
+@app.exception_handler(ValueError)
+async def _value_error_handler(
+    request: fastapi.Request, exc: ValueError
+) -> fastapi.responses.JSONResponse:
+    """Mirror the production ValueError → 400 mapping from rest_api.py."""
+    return fastapi.responses.JSONResponse(status_code=400, content={"detail": str(exc)})
+
+
+client = fastapi.testclient.TestClient(app)
+
+TEST_USER_ID = "3e53486c-cf57-477e-ba2a-cb02dc828e1a"
+
+MOCK_WORKSPACE = type("W", (), {"id": "ws-1"})()
+
+_NOW = datetime(2023, 1, 1, tzinfo=timezone.utc)
+
+MOCK_FILE = WorkspaceFile(
+    id="file-aaa-bbb",
+    workspace_id="ws-1",
+    created_at=_NOW,
+    updated_at=_NOW,
+    name="hello.txt",
+    path="/session/hello.txt",
+    mime_type="text/plain",
+    size_bytes=13,
+    storage_path="local://hello.txt",
+)
+
+
+@pytest.fixture(autouse=True)
+def setup_app_auth(mock_jwt_user):
+    from autogpt_libs.auth.jwt_utils import get_jwt_payload
+
+    app.dependency_overrides[get_jwt_payload] = mock_jwt_user["get_jwt_payload"]
+    yield
+    app.dependency_overrides.clear()
+
+
+def _upload(
+    filename: str = "hello.txt",
+    content: bytes = b"Hello, world!",
+    content_type: str = "text/plain",
+):
+    """Helper to POST a file upload."""
+    return client.post(
+        "/files/upload?session_id=sess-1",
+        files={"file": (filename, io.BytesIO(content), content_type)},
+    )
+
+
+# ---- Happy path ----
+
+
+def test_upload_happy_path(mocker: pytest_mock.MockFixture):
+    mocker.patch(
+        "backend.api.features.workspace.routes.get_or_create_workspace",
+        return_value=MOCK_WORKSPACE,
+    )
+    mocker.patch(
+        "backend.api.features.workspace.routes.get_workspace_total_size",
+        return_value=0,
+    )
+    mocker.patch(
+        "backend.api.features.workspace.routes.scan_content_safe",
+        return_value=None,
+    )
+    mock_manager = mocker.MagicMock()
+    mock_manager.write_file = mocker.AsyncMock(return_value=MOCK_FILE)
+    mocker.patch(
+        "backend.api.features.workspace.routes.WorkspaceManager",
+        return_value=mock_manager,
+    )
+
+    response = _upload()
+    assert response.status_code == 200
+    data = response.json()
+    assert data["file_id"] == "file-aaa-bbb"
+    assert data["name"] == "hello.txt"
+    assert data["size_bytes"] == 13
+
+
+# ---- Per-file size limit ----
+
+
+def test_upload_exceeds_max_file_size(mocker: pytest_mock.MockFixture):
+    """Files larger than max_file_size_mb should be rejected with 413."""
+    cfg = mocker.patch("backend.api.features.workspace.routes.Config")
+    cfg.return_value.max_file_size_mb = 0  # 0 MB → any content is too big
+    cfg.return_value.max_workspace_storage_mb = 500
+
+    response = _upload(content=b"x" * 1024)
+    assert response.status_code == 413
+
+
+# ---- Storage quota exceeded ----
+
+
+def test_upload_storage_quota_exceeded(mocker: pytest_mock.MockFixture):
+    mocker.patch(
+        "backend.api.features.workspace.routes.get_or_create_workspace",
+        return_value=MOCK_WORKSPACE,
+    )
+    # Current usage already at limit
+    mocker.patch(
+        "backend.api.features.workspace.routes.get_workspace_total_size",
+        return_value=500 * 1024 * 1024,
+    )
+
+    response = _upload()
+    assert response.status_code == 413
+    assert "Storage limit exceeded" in response.text
+
+
+# ---- Post-write quota race (B2) ----
+
+
+def test_upload_post_write_quota_race(mocker: pytest_mock.MockFixture):
+    """If a concurrent upload tips the total over the limit after write,
+    the file should be soft-deleted and 413 returned."""
+    mocker.patch(
+        "backend.api.features.workspace.routes.get_or_create_workspace",
+        return_value=MOCK_WORKSPACE,
+    )
+    # Pre-write check passes (under limit), but post-write check fails
+    mocker.patch(
+        "backend.api.features.workspace.routes.get_workspace_total_size",
+        side_effect=[0, 600 * 1024 * 1024],  # first call OK, second over limit
+    )
+    mocker.patch(
+        "backend.api.features.workspace.routes.scan_content_safe",
+        return_value=None,
+    )
+    mock_manager = mocker.MagicMock()
+    mock_manager.write_file = mocker.AsyncMock(return_value=MOCK_FILE)
+    mocker.patch(
+        "backend.api.features.workspace.routes.WorkspaceManager",
+        return_value=mock_manager,
+    )
+    mock_delete = mocker.patch(
+        "backend.api.features.workspace.routes.soft_delete_workspace_file",
+        return_value=None,
+    )
+
+    response = _upload()
+    assert response.status_code == 413
+    mock_delete.assert_called_once_with("file-aaa-bbb", "ws-1")
+
+
+# ---- Any extension accepted (no allowlist) ----
+
+
+def test_upload_any_extension(mocker: pytest_mock.MockFixture):
+    """Any file extension should be accepted — ClamAV is the security layer."""
+    mocker.patch(
+        "backend.api.features.workspace.routes.get_or_create_workspace",
+        return_value=MOCK_WORKSPACE,
+    )
+    mocker.patch(
+        "backend.api.features.workspace.routes.get_workspace_total_size",
+        return_value=0,
+    )
+    mocker.patch(
+        "backend.api.features.workspace.routes.scan_content_safe",
+        return_value=None,
+    )
+    mock_manager = mocker.MagicMock()
+    mock_manager.write_file = mocker.AsyncMock(return_value=MOCK_FILE)
+    mocker.patch(
+        "backend.api.features.workspace.routes.WorkspaceManager",
+        return_value=mock_manager,
+    )
+
+    response = _upload(filename="data.xyz", content=b"arbitrary")
+    assert response.status_code == 200
+
+
+# ---- Virus scan rejection ----
+
+
+def test_upload_blocked_by_virus_scan(mocker: pytest_mock.MockFixture):
+    """Files flagged by ClamAV should be rejected and never written to storage."""
+    from backend.api.features.store.exceptions import VirusDetectedError
+
+    mocker.patch(
+        "backend.api.features.workspace.routes.get_or_create_workspace",
+        return_value=MOCK_WORKSPACE,
+    )
+    mocker.patch(
+        "backend.api.features.workspace.routes.get_workspace_total_size",
+        return_value=0,
+    )
+    mocker.patch(
+        "backend.api.features.workspace.routes.scan_content_safe",
+        side_effect=VirusDetectedError("Eicar-Test-Signature"),
+    )
+    mock_manager = mocker.MagicMock()
+    mock_manager.write_file = mocker.AsyncMock(return_value=MOCK_FILE)
+    mocker.patch(
+        "backend.api.features.workspace.routes.WorkspaceManager",
+        return_value=mock_manager,
+    )
+
+    response = _upload(filename="evil.exe", content=b"X5O!P%@AP...")
+    assert response.status_code == 400
+    assert "Virus detected" in response.text
+    mock_manager.write_file.assert_not_called()
+
+
+# ---- No file extension ----
+
+
+def test_upload_file_without_extension(mocker: pytest_mock.MockFixture):
+    """Files without an extension should be accepted and stored as-is."""
+    mocker.patch(
+        "backend.api.features.workspace.routes.get_or_create_workspace",
+        return_value=MOCK_WORKSPACE,
+    )
+    mocker.patch(
+        "backend.api.features.workspace.routes.get_workspace_total_size",
+        return_value=0,
+    )
+    mocker.patch(
+        "backend.api.features.workspace.routes.scan_content_safe",
+        return_value=None,
+    )
+    mock_manager = mocker.MagicMock()
+    mock_manager.write_file = mocker.AsyncMock(return_value=MOCK_FILE)
+    mocker.patch(
+        "backend.api.features.workspace.routes.WorkspaceManager",
+        return_value=mock_manager,
+    )
+
+    response = _upload(
+        filename="Makefile",
+        content=b"all:\n\techo hello",
+        content_type="application/octet-stream",
+    )
+    assert response.status_code == 200
+    mock_manager.write_file.assert_called_once()
+    assert mock_manager.write_file.call_args[0][1] == "Makefile"
+
+
+# ---- Filename sanitization (SF5) ----
+
+
+def test_upload_strips_path_components(mocker: pytest_mock.MockFixture):
+    """Path-traversal filenames should be reduced to their basename."""
+    mocker.patch(
+        "backend.api.features.workspace.routes.get_or_create_workspace",
+        return_value=MOCK_WORKSPACE,
+    )
+    mocker.patch(
+        "backend.api.features.workspace.routes.get_workspace_total_size",
+        return_value=0,
+    )
+    mocker.patch(
+        "backend.api.features.workspace.routes.scan_content_safe",
+        return_value=None,
+    )
+    mock_manager = mocker.MagicMock()
+    mock_manager.write_file = mocker.AsyncMock(return_value=MOCK_FILE)
+    mocker.patch(
+        "backend.api.features.workspace.routes.WorkspaceManager",
+        return_value=mock_manager,
+    )
+
+    # Filename with traversal
+    _upload(filename="../../etc/passwd.txt")
+
+    # write_file should have been called with just the basename
+    mock_manager.write_file.assert_called_once()
+    call_args = mock_manager.write_file.call_args
+    assert call_args[0][1] == "passwd.txt"
+
+
+# ---- Download ----
+
+
+def test_download_file_not_found(mocker: pytest_mock.MockFixture):
+    mocker.patch(
+        "backend.api.features.workspace.routes.get_workspace",
+        return_value=MOCK_WORKSPACE,
+    )
+    mocker.patch(
+        "backend.api.features.workspace.routes.get_workspace_file",
+        return_value=None,
+    )
+
+    response = client.get("/files/some-file-id/download")
+    assert response.status_code == 404
+
+
+# ---- Delete ----
+
+
+def test_delete_file_success(mocker: pytest_mock.MockFixture):
+    """Deleting an existing file should return {"deleted": true}."""
+    mocker.patch(
+        "backend.api.features.workspace.routes.get_workspace",
+        return_value=MOCK_WORKSPACE,
+    )
+    mock_manager = mocker.MagicMock()
+    mock_manager.delete_file = mocker.AsyncMock(return_value=True)
+    mocker.patch(
+        "backend.api.features.workspace.routes.WorkspaceManager",
+        return_value=mock_manager,
+    )
+
+    response = client.delete("/files/file-aaa-bbb")
+    assert response.status_code == 200
+    assert response.json() == {"deleted": True}
+    mock_manager.delete_file.assert_called_once_with("file-aaa-bbb")
+
+
+def test_delete_file_not_found(mocker: pytest_mock.MockFixture):
+    """Deleting a non-existent file should return 404."""
+    mocker.patch(
+        "backend.api.features.workspace.routes.get_workspace",
+        return_value=MOCK_WORKSPACE,
+    )
+    mock_manager = mocker.MagicMock()
+    mock_manager.delete_file = mocker.AsyncMock(return_value=False)
+    mocker.patch(
+        "backend.api.features.workspace.routes.WorkspaceManager",
+        return_value=mock_manager,
+    )
+
+    response = client.delete("/files/nonexistent-id")
+    assert response.status_code == 404
+    assert "File not found" in response.text
+
+
+def test_delete_file_no_workspace(mocker: pytest_mock.MockFixture):
+    """Deleting when user has no workspace should return 404."""
+    mocker.patch(
+        "backend.api.features.workspace.routes.get_workspace",
+        return_value=None,
+    )
+
+    response = client.delete("/files/file-aaa-bbb")
+    assert response.status_code == 404
+    assert "Workspace not found" in response.text

autogpt_platform/backend/backend/api/model.py

@@ -94,3 +94,8 @@ class NotificationPayload(pydantic.BaseModel):
 
 class OnboardingNotificationPayload(NotificationPayload):
     step: OnboardingStep | None
+
+
+class CopilotCompletionPayload(NotificationPayload):
+    session_id: str
+    status: Literal["completed", "failed"]

autogpt_platform/backend/backend/api/rest_api.py

@@ -19,6 +19,7 @@ from prisma.errors import PrismaError
 import backend.api.features.admin.credit_admin_routes
 import backend.api.features.admin.execution_analytics_routes
 import backend.api.features.admin.store_admin_routes
+import backend.api.features.admin.user_admin_routes
 import backend.api.features.builder
 import backend.api.features.builder.routes
 import backend.api.features.chat.routes as chat_routes
@@ -41,9 +42,9 @@ import backend.data.user
 import backend.integrations.webhooks.utils
 import backend.util.service
 import backend.util.settings
-from backend.api.features.chat.completion_consumer import (
-    start_completion_consumer,
-    stop_completion_consumer,
+from backend.api.features.library.exceptions import (
+    FolderAlreadyExistsError,
+    FolderValidationError,
 )
 from backend.blocks.llm import DEFAULT_LLM_MODEL
 from backend.data.model import Credentials
@@ -55,6 +56,7 @@ from backend.util.exceptions import (
     MissingConfigError,
     NotAuthorizedError,
     NotFoundError,
+    PreconditionFailed,
 )
 from backend.util.feature_flag import initialize_launchdarkly, shutdown_launchdarkly
 from backend.util.service import UnhealthyServiceError
@@ -123,21 +125,9 @@ async def lifespan_context(app: fastapi.FastAPI):
     await backend.data.graph.migrate_llm_models(DEFAULT_LLM_MODEL)
     await backend.integrations.webhooks.utils.migrate_legacy_triggered_graphs()
 
-    # Start chat completion consumer for Redis Streams notifications
-    try:
-        await start_completion_consumer()
-    except Exception as e:
-        logger.warning(f"Could not start chat completion consumer: {e}")
-
     with launch_darkly_context():
         yield
 
-    # Stop chat completion consumer
-    try:
-        await stop_completion_consumer()
-    except Exception as e:
-        logger.warning(f"Error stopping chat completion consumer: {e}")
-
     try:
         await shutdown_cloud_storage_handler()
     except Exception as e:
@@ -277,12 +267,17 @@ async def validation_error_handler(
 
 
 app.add_exception_handler(PrismaError, handle_internal_http_error(500))
+app.add_exception_handler(
+    FolderAlreadyExistsError, handle_internal_http_error(409, False)
+)
+app.add_exception_handler(FolderValidationError, handle_internal_http_error(400, False))
 app.add_exception_handler(NotFoundError, handle_internal_http_error(404, False))
 app.add_exception_handler(NotAuthorizedError, handle_internal_http_error(403, False))
 app.add_exception_handler(RequestValidationError, validation_error_handler)
 app.add_exception_handler(pydantic.ValidationError, validation_error_handler)
 app.add_exception_handler(MissingConfigError, handle_internal_http_error(503))
 app.add_exception_handler(ValueError, handle_internal_http_error(400))
+app.add_exception_handler(PreconditionFailed, handle_internal_http_error(428))
 app.add_exception_handler(Exception, handle_internal_http_error(500))
 
 app.include_router(backend.api.features.v1.v1_router, tags=["v1"], prefix="/api")
@@ -317,6 +312,11 @@ app.include_router(
     tags=["v2", "admin"],
     prefix="/api/executions",
 )
+app.include_router(
+    backend.api.features.admin.user_admin_routes.router,
+    tags=["v2", "admin"],
+    prefix="/api/users",
+)
 app.include_router(
     backend.api.features.executions.review.routes.router,
     tags=["v2", "executions", "review"],

autogpt_platform/backend/backend/app.py

@@ -24,7 +24,7 @@ def run_processes(*processes: "AppProcess", **kwargs):
         # Run the last process in the foreground.
         processes[-1].start(background=False, **kwargs)
     finally:
-        for process in processes:
+        for process in reversed(processes):
             try:
                 process.stop()
             except Exception as e:
@@ -38,7 +38,9 @@ def main(**kwargs):
 
     from backend.api.rest_api import AgentServer
     from backend.api.ws_api import WebsocketServer
-    from backend.executor import DatabaseManager, ExecutionManager, Scheduler
+    from backend.copilot.executor.manager import CoPilotExecutor
+    from backend.data.db_manager import DatabaseManager
+    from backend.executor import ExecutionManager, Scheduler
     from backend.notifications import NotificationManager
 
     run_processes(
@@ -48,6 +50,7 @@ def main(**kwargs):
         WebsocketServer(),
         AgentServer(),
         ExecutionManager(),
+        CoPilotExecutor(),
         **kwargs,
     )
 

autogpt_platform/backend/backend/blocks/_base.py

@@ -418,6 +418,8 @@ class BlockWebhookConfig(BlockManualWebhookConfig):
 
 
 class Block(ABC, Generic[BlockSchemaInputType, BlockSchemaOutputType]):
+    _optimized_description: ClassVar[str | None] = None
+
     def __init__(
         self,
         id: str = "",
@@ -470,6 +472,8 @@ class Block(ABC, Generic[BlockSchemaInputType, BlockSchemaOutputType]):
         self.block_type = block_type
         self.webhook_config = webhook_config
         self.is_sensitive_action = is_sensitive_action
+        # Read from ClassVar set by initialize_blocks()
+        self.optimized_description: str | None = type(self)._optimized_description
         self.execution_stats: "NodeExecutionStats" = NodeExecutionStats()
 
         if self.webhook_config:
@@ -620,6 +624,7 @@ class Block(ABC, Generic[BlockSchemaInputType, BlockSchemaOutputType]):
         graph_id: str,
         graph_version: int,
         execution_context: "ExecutionContext",
+        is_graph_execution: bool = True,
         **kwargs,
     ) -> tuple[bool, BlockInput]:
         """
@@ -648,6 +653,7 @@ class Block(ABC, Generic[BlockSchemaInputType, BlockSchemaOutputType]):
             graph_version=graph_version,
             block_name=self.name,
             editable=True,
+            is_graph_execution=is_graph_execution,
         )
 
         if decision is None:

autogpt_platform/backend/backend/blocks/basic.py

@@ -126,7 +126,7 @@ class PrintToConsoleBlock(Block):
             output_schema=PrintToConsoleBlock.Output,
             test_input={"text": "Hello, World!"},
             is_sensitive_action=True,
-            disabled=True,  # Disabled per Nick Tindle's request (OPEN-3000)
+            disabled=True,
             test_output=[
                 ("output", "Hello, World!"),
                 ("status", "printed"),

autogpt_platform/backend/backend/blocks/code_executor.py

@@ -142,7 +142,7 @@ class BaseE2BExecutorMixin:
                 start_timestamp = ts_result.stdout.strip() if ts_result.stdout else None
 
             # Execute the code
-            execution = await sandbox.run_code(
+            execution = await sandbox.run_code(  # type: ignore[attr-defined]
                 code,
                 language=language.value,
                 on_error=lambda e: sandbox.kill(),  # Kill the sandbox on error

autogpt_platform/backend/backend/blocks/email_block.py

@@ -96,6 +96,7 @@ class SendEmailBlock(Block):
             test_credentials=TEST_CREDENTIALS,
             test_output=[("status", "Email sent successfully")],
             test_mock={"send_email": lambda *args, **kwargs: "Email sent successfully"},
+            is_sensitive_action=True,
         )
 
     @staticmethod

autogpt_platform/backend/backend/blocks/github/_utils.py

@@ -0,0 +1,3 @@
+def github_repo_path(repo_url: str) -> str:
+    """Extract 'owner/repo' from a GitHub repository URL."""
+    return repo_url.replace("https://github.com/", "")

autogpt_platform/backend/backend/blocks/github/commits.py

@@ -0,0 +1,408 @@
+import asyncio
+from enum import StrEnum
+from urllib.parse import quote
+
+from typing_extensions import TypedDict
+
+from backend.blocks._base import (
+    Block,
+    BlockCategory,
+    BlockOutput,
+    BlockSchemaInput,
+    BlockSchemaOutput,
+)
+from backend.data.execution import ExecutionContext
+from backend.data.model import SchemaField
+from backend.util.file import parse_data_uri, resolve_media_content
+from backend.util.type import MediaFileType
+
+from ._api import get_api
+from ._auth import (
+    TEST_CREDENTIALS,
+    TEST_CREDENTIALS_INPUT,
+    GithubCredentials,
+    GithubCredentialsField,
+    GithubCredentialsInput,
+)
+from ._utils import github_repo_path
+
+
+class GithubListCommitsBlock(Block):
+    class Input(BlockSchemaInput):
+        credentials: GithubCredentialsInput = GithubCredentialsField("repo")
+        repo_url: str = SchemaField(
+            description="URL of the GitHub repository",
+            placeholder="https://github.com/owner/repo",
+        )
+        branch: str = SchemaField(
+            description="Branch name to list commits from",
+            default="main",
+        )
+        per_page: int = SchemaField(
+            description="Number of commits to return (max 100)",
+            default=30,
+            ge=1,
+            le=100,
+        )
+        page: int = SchemaField(
+            description="Page number for pagination",
+            default=1,
+            ge=1,
+        )
+
+    class Output(BlockSchemaOutput):
+        class CommitItem(TypedDict):
+            sha: str
+            message: str
+            author: str
+            date: str
+            url: str
+
+        commit: CommitItem = SchemaField(
+            title="Commit", description="A commit with its details"
+        )
+        commits: list[CommitItem] = SchemaField(
+            description="List of commits with their details"
+        )
+        error: str = SchemaField(description="Error message if listing commits failed")
+
+    def __init__(self):
+        super().__init__(
+            id="8b13f579-d8b6-4dc2-a140-f770428805de",
+            description="This block lists commits on a branch in a GitHub repository.",
+            categories={BlockCategory.DEVELOPER_TOOLS},
+            input_schema=GithubListCommitsBlock.Input,
+            output_schema=GithubListCommitsBlock.Output,
+            test_input={
+                "repo_url": "https://github.com/owner/repo",
+                "branch": "main",
+                "per_page": 30,
+                "page": 1,
+                "credentials": TEST_CREDENTIALS_INPUT,
+            },
+            test_credentials=TEST_CREDENTIALS,
+            test_output=[
+                (
+                    "commits",
+                    [
+                        {
+                            "sha": "abc123",
+                            "message": "Initial commit",
+                            "author": "octocat",
+                            "date": "2024-01-01T00:00:00Z",
+                            "url": "https://github.com/owner/repo/commit/abc123",
+                        }
+                    ],
+                ),
+                (
+                    "commit",
+                    {
+                        "sha": "abc123",
+                        "message": "Initial commit",
+                        "author": "octocat",
+                        "date": "2024-01-01T00:00:00Z",
+                        "url": "https://github.com/owner/repo/commit/abc123",
+                    },
+                ),
+            ],
+            test_mock={
+                "list_commits": lambda *args, **kwargs: [
+                    {
+                        "sha": "abc123",
+                        "message": "Initial commit",
+                        "author": "octocat",
+                        "date": "2024-01-01T00:00:00Z",
+                        "url": "https://github.com/owner/repo/commit/abc123",
+                    }
+                ]
+            },
+        )
+
+    @staticmethod
+    async def list_commits(
+        credentials: GithubCredentials,
+        repo_url: str,
+        branch: str,
+        per_page: int,
+        page: int,
+    ) -> list[Output.CommitItem]:
+        api = get_api(credentials)
+        commits_url = repo_url + "/commits"
+        params = {"sha": branch, "per_page": str(per_page), "page": str(page)}
+        response = await api.get(commits_url, params=params)
+        data = response.json()
+        repo_path = github_repo_path(repo_url)
+        return [
+            GithubListCommitsBlock.Output.CommitItem(
+                sha=c["sha"],
+                message=c["commit"]["message"],
+                author=(c["commit"].get("author") or {}).get("name", "Unknown"),
+                date=(c["commit"].get("author") or {}).get("date", ""),
+                url=f"https://github.com/{repo_path}/commit/{c['sha']}",
+            )
+            for c in data
+        ]
+
+    async def run(
+        self,
+        input_data: Input,
+        *,
+        credentials: GithubCredentials,
+        **kwargs,
+    ) -> BlockOutput:
+        try:
+            commits = await self.list_commits(
+                credentials,
+                input_data.repo_url,
+                input_data.branch,
+                input_data.per_page,
+                input_data.page,
+            )
+            yield "commits", commits
+            for commit in commits:
+                yield "commit", commit
+        except Exception as e:
+            yield "error", str(e)
+
+
+class FileOperation(StrEnum):
+    """File operations for GithubMultiFileCommitBlock.
+
+    UPSERT creates or overwrites a file (the Git Trees API does not distinguish
+    between creation and update — the blob is placed at the given path regardless
+    of whether a file already exists there).
+
+    DELETE removes a file from the tree.
+    """
+
+    UPSERT = "upsert"
+    DELETE = "delete"
+
+
+class FileOperationInput(TypedDict):
+    path: str
+    # MediaFileType is a str NewType — no runtime breakage for existing callers.
+    content: MediaFileType
+    operation: FileOperation
+
+
+class GithubMultiFileCommitBlock(Block):
+    class Input(BlockSchemaInput):
+        credentials: GithubCredentialsInput = GithubCredentialsField("repo")
+        repo_url: str = SchemaField(
+            description="URL of the GitHub repository",
+            placeholder="https://github.com/owner/repo",
+        )
+        branch: str = SchemaField(
+            description="Branch to commit to",
+            placeholder="feature-branch",
+        )
+        commit_message: str = SchemaField(
+            description="Commit message",
+            placeholder="Add new feature",
+        )
+        files: list[FileOperationInput] = SchemaField(
+            description=(
+                "List of file operations. Each item has: "
+                "'path' (file path), 'content' (file content, ignored for delete), "
+                "'operation' (upsert/delete)"
+            ),
+        )
+
+    class Output(BlockSchemaOutput):
+        sha: str = SchemaField(description="SHA of the new commit")
+        url: str = SchemaField(description="URL of the new commit")
+        error: str = SchemaField(description="Error message if the commit failed")
+
+    def __init__(self):
+        super().__init__(
+            id="389eee51-a95e-4230-9bed-92167a327802",
+            description=(
+                "This block creates a single commit with multiple file "
+                "upsert/delete operations using the Git Trees API."
+            ),
+            categories={BlockCategory.DEVELOPER_TOOLS},
+            input_schema=GithubMultiFileCommitBlock.Input,
+            output_schema=GithubMultiFileCommitBlock.Output,
+            test_input={
+                "repo_url": "https://github.com/owner/repo",
+                "branch": "feature",
+                "commit_message": "Add files",
+                "files": [
+                    {
+                        "path": "src/new.py",
+                        "content": "print('hello')",
+                        "operation": "upsert",
+                    },
+                    {
+                        "path": "src/old.py",
+                        "content": "",
+                        "operation": "delete",
+                    },
+                ],
+                "credentials": TEST_CREDENTIALS_INPUT,
+            },
+            test_credentials=TEST_CREDENTIALS,
+            test_output=[
+                ("sha", "newcommitsha"),
+                ("url", "https://github.com/owner/repo/commit/newcommitsha"),
+            ],
+            test_mock={
+                "multi_file_commit": lambda *args, **kwargs: (
+                    "newcommitsha",
+                    "https://github.com/owner/repo/commit/newcommitsha",
+                )
+            },
+        )
+
+    @staticmethod
+    async def multi_file_commit(
+        credentials: GithubCredentials,
+        repo_url: str,
+        branch: str,
+        commit_message: str,
+        files: list[FileOperationInput],
+    ) -> tuple[str, str]:
+        api = get_api(credentials)
+        safe_branch = quote(branch, safe="")
+
+        # 1. Get the latest commit SHA for the branch
+        ref_url = repo_url + f"/git/refs/heads/{safe_branch}"
+        response = await api.get(ref_url)
+        ref_data = response.json()
+        latest_commit_sha = ref_data["object"]["sha"]
+
+        # 2. Get the tree SHA of the latest commit
+        commit_url = repo_url + f"/git/commits/{latest_commit_sha}"
+        response = await api.get(commit_url)
+        commit_data = response.json()
+        base_tree_sha = commit_data["tree"]["sha"]
+
+        # 3. Build tree entries for each file operation (blobs created concurrently)
+        async def _create_blob(content: str, encoding: str = "utf-8") -> str:
+            blob_url = repo_url + "/git/blobs"
+            blob_response = await api.post(
+                blob_url,
+                json={"content": content, "encoding": encoding},
+            )
+            return blob_response.json()["sha"]
+
+        tree_entries: list[dict] = []
+        upsert_files = []
+        for file_op in files:
+            path = file_op["path"]
+            operation = FileOperation(file_op.get("operation", "upsert"))
+
+            if operation == FileOperation.DELETE:
+                tree_entries.append(
+                    {
+                        "path": path,
+                        "mode": "100644",
+                        "type": "blob",
+                        "sha": None,  # null SHA = delete
+                    }
+                )
+            else:
+                upsert_files.append((path, file_op.get("content", "")))
+
+        # Create all blobs concurrently. Data URIs (from store_media_file)
+        # are sent as base64 blobs to preserve binary content.
+        if upsert_files:
+
+            async def _make_blob(content: str) -> str:
+                parsed = parse_data_uri(content)
+                if parsed is not None:
+                    _, b64_payload = parsed
+                    return await _create_blob(b64_payload, encoding="base64")
+                return await _create_blob(content)
+
+            blob_shas = await asyncio.gather(
+                *[_make_blob(content) for _, content in upsert_files]
+            )
+            for (path, _), blob_sha in zip(upsert_files, blob_shas):
+                tree_entries.append(
+                    {
+                        "path": path,
+                        "mode": "100644",
+                        "type": "blob",
+                        "sha": blob_sha,
+                    }
+                )
+
+        # 4. Create a new tree
+        tree_url = repo_url + "/git/trees"
+        tree_response = await api.post(
+            tree_url,
+            json={"base_tree": base_tree_sha, "tree": tree_entries},
+        )
+        new_tree_sha = tree_response.json()["sha"]
+
+        # 5. Create a new commit
+        new_commit_url = repo_url + "/git/commits"
+        commit_response = await api.post(
+            new_commit_url,
+            json={
+                "message": commit_message,
+                "tree": new_tree_sha,
+                "parents": [latest_commit_sha],
+            },
+        )
+        new_commit_sha = commit_response.json()["sha"]
+
+        # 6. Update the branch reference
+        try:
+            await api.patch(
+                ref_url,
+                json={"sha": new_commit_sha},
+            )
+        except Exception as e:
+            raise RuntimeError(
+                f"Commit {new_commit_sha} was created but failed to update "
+                f"ref heads/{branch}: {e}. "
+                f"You can recover by manually updating the branch to {new_commit_sha}."
+            ) from e
+
+        repo_path = github_repo_path(repo_url)
+        commit_web_url = f"https://github.com/{repo_path}/commit/{new_commit_sha}"
+        return new_commit_sha, commit_web_url
+
+    async def run(
+        self,
+        input_data: Input,
+        *,
+        credentials: GithubCredentials,
+        execution_context: ExecutionContext,
+        **kwargs,
+    ) -> BlockOutput:
+        try:
+            # Resolve media references (workspace://, data:, URLs) to data
+            # URIs so _make_blob can send binary content correctly.
+            resolved_files: list[FileOperationInput] = []
+            for file_op in input_data.files:
+                content = file_op.get("content", "")
+                operation = FileOperation(file_op.get("operation", "upsert"))
+                if operation != FileOperation.DELETE:
+                    content = await resolve_media_content(
+                        MediaFileType(content),
+                        execution_context,
+                        return_format="for_external_api",
+                    )
+                resolved_files.append(
+                    FileOperationInput(
+                        path=file_op["path"],
+                        content=MediaFileType(content),
+                        operation=operation,
+                    )
+                )
+
+            sha, url = await self.multi_file_commit(
+                credentials,
+                input_data.repo_url,
+                input_data.branch,
+                input_data.commit_message,
+                resolved_files,
+            )
+            yield "sha", sha
+            yield "url", url
+        except Exception as e:
+            yield "error", str(e)

autogpt_platform/backend/backend/blocks/github/pull_requests.py

@@ -1,4 +1,5 @@
 import re
+from typing import Literal
 
 from typing_extensions import TypedDict
 
@@ -20,6 +21,8 @@ from ._auth import (
     GithubCredentialsInput,
 )
 
+MergeMethod = Literal["merge", "squash", "rebase"]
+
 
 class GithubListPullRequestsBlock(Block):
     class Input(BlockSchemaInput):
@@ -558,12 +561,109 @@ class GithubListPRReviewersBlock(Block):
             yield "reviewer", reviewer
 
 
+class GithubMergePullRequestBlock(Block):
+    class Input(BlockSchemaInput):
+        credentials: GithubCredentialsInput = GithubCredentialsField("repo")
+        pr_url: str = SchemaField(
+            description="URL of the GitHub pull request",
+            placeholder="https://github.com/owner/repo/pull/1",
+        )
+        merge_method: MergeMethod = SchemaField(
+            description="Merge method to use: merge, squash, or rebase",
+            default="merge",
+        )
+        commit_title: str = SchemaField(
+            description="Title for the merge commit (optional, used for merge and squash)",
+            default="",
+        )
+        commit_message: str = SchemaField(
+            description="Message for the merge commit (optional, used for merge and squash)",
+            default="",
+        )
+
+    class Output(BlockSchemaOutput):
+        sha: str = SchemaField(description="SHA of the merge commit")
+        merged: bool = SchemaField(description="Whether the PR was merged")
+        message: str = SchemaField(description="Merge status message")
+        error: str = SchemaField(description="Error message if the merge failed")
+
+    def __init__(self):
+        super().__init__(
+            id="77456c22-33d8-4fd4-9eef-50b46a35bb48",
+            description="This block merges a pull request using merge, squash, or rebase.",
+            categories={BlockCategory.DEVELOPER_TOOLS},
+            input_schema=GithubMergePullRequestBlock.Input,
+            output_schema=GithubMergePullRequestBlock.Output,
+            test_input={
+                "pr_url": "https://github.com/owner/repo/pull/1",
+                "merge_method": "squash",
+                "commit_title": "",
+                "commit_message": "",
+                "credentials": TEST_CREDENTIALS_INPUT,
+            },
+            test_credentials=TEST_CREDENTIALS,
+            test_output=[
+                ("sha", "abc123"),
+                ("merged", True),
+                ("message", "Pull Request successfully merged"),
+            ],
+            test_mock={
+                "merge_pr": lambda *args, **kwargs: (
+                    "abc123",
+                    True,
+                    "Pull Request successfully merged",
+                )
+            },
+            is_sensitive_action=True,
+        )
+
+    @staticmethod
+    async def merge_pr(
+        credentials: GithubCredentials,
+        pr_url: str,
+        merge_method: MergeMethod,
+        commit_title: str,
+        commit_message: str,
+    ) -> tuple[str, bool, str]:
+        api = get_api(credentials)
+        merge_url = prepare_pr_api_url(pr_url=pr_url, path="merge")
+        data: dict[str, str] = {"merge_method": merge_method}
+        if commit_title:
+            data["commit_title"] = commit_title
+        if commit_message:
+            data["commit_message"] = commit_message
+        response = await api.put(merge_url, json=data)
+        result = response.json()
+        return result["sha"], result["merged"], result["message"]
+
+    async def run(
+        self,
+        input_data: Input,
+        *,
+        credentials: GithubCredentials,
+        **kwargs,
+    ) -> BlockOutput:
+        try:
+            sha, merged, message = await self.merge_pr(
+                credentials,
+                input_data.pr_url,
+                input_data.merge_method,
+                input_data.commit_title,
+                input_data.commit_message,
+            )
+            yield "sha", sha
+            yield "merged", merged
+            yield "message", message
+        except Exception as e:
+            yield "error", str(e)
+
+
 def prepare_pr_api_url(pr_url: str, path: str) -> str:
     # Pattern to capture the base repository URL and the pull request number
-    pattern = r"^(?:https?://)?([^/]+/[^/]+/[^/]+)/pull/(\d+)"
+    pattern = r"^(?:(https?)://)?([^/]+/[^/]+/[^/]+)/pull/(\d+)"
     match = re.match(pattern, pr_url)
     if not match:
         return pr_url
 
-    base_url, pr_number = match.groups()
-    return f"{base_url}/pulls/{pr_number}/{path}"
+    scheme, base_url, pr_number = match.groups()
+    return f"{scheme or 'https'}://{base_url}/pulls/{pr_number}/{path}"

autogpt_platform/backend/backend/blocks/github/repo.py

@@ -1,5 +1,3 @@
-import base64
-
 from typing_extensions import TypedDict
 
 from backend.blocks._base import (
@@ -19,6 +17,7 @@ from ._auth import (
     GithubCredentialsField,
     GithubCredentialsInput,
 )
+from ._utils import github_repo_path
 
 
 class GithubListTagsBlock(Block):
@@ -89,7 +88,7 @@ class GithubListTagsBlock(Block):
         tags_url = repo_url + "/tags"
         response = await api.get(tags_url)
         data = response.json()
-        repo_path = repo_url.replace("https://github.com/", "")
+        repo_path = github_repo_path(repo_url)
         tags: list[GithubListTagsBlock.Output.TagItem] = [
             {
                 "name": tag["name"],
@@ -115,101 +114,6 @@ class GithubListTagsBlock(Block):
             yield "tag", tag
 
 
-class GithubListBranchesBlock(Block):
-    class Input(BlockSchemaInput):
-        credentials: GithubCredentialsInput = GithubCredentialsField("repo")
-        repo_url: str = SchemaField(
-            description="URL of the GitHub repository",
-            placeholder="https://github.com/owner/repo",
-        )
-
-    class Output(BlockSchemaOutput):
-        class BranchItem(TypedDict):
-            name: str
-            url: str
-
-        branch: BranchItem = SchemaField(
-            title="Branch",
-            description="Branches with their name and file tree browser URL",
-        )
-        branches: list[BranchItem] = SchemaField(
-            description="List of branches with their name and file tree browser URL"
-        )
-
-    def __init__(self):
-        super().__init__(
-            id="74243e49-2bec-4916-8bf4-db43d44aead5",
-            description="This block lists all branches for a specified GitHub repository.",
-            categories={BlockCategory.DEVELOPER_TOOLS},
-            input_schema=GithubListBranchesBlock.Input,
-            output_schema=GithubListBranchesBlock.Output,
-            test_input={
-                "repo_url": "https://github.com/owner/repo",
-                "credentials": TEST_CREDENTIALS_INPUT,
-            },
-            test_credentials=TEST_CREDENTIALS,
-            test_output=[
-                (
-                    "branches",
-                    [
-                        {
-                            "name": "main",
-                            "url": "https://github.com/owner/repo/tree/main",
-                        }
-                    ],
-                ),
-                (
-                    "branch",
-                    {
-                        "name": "main",
-                        "url": "https://github.com/owner/repo/tree/main",
-                    },
-                ),
-            ],
-            test_mock={
-                "list_branches": lambda *args, **kwargs: [
-                    {
-                        "name": "main",
-                        "url": "https://github.com/owner/repo/tree/main",
-                    }
-                ]
-            },
-        )
-
-    @staticmethod
-    async def list_branches(
-        credentials: GithubCredentials, repo_url: str
-    ) -> list[Output.BranchItem]:
-        api = get_api(credentials)
-        branches_url = repo_url + "/branches"
-        response = await api.get(branches_url)
-        data = response.json()
-        repo_path = repo_url.replace("https://github.com/", "")
-        branches: list[GithubListBranchesBlock.Output.BranchItem] = [
-            {
-                "name": branch["name"],
-                "url": f"https://github.com/{repo_path}/tree/{branch['name']}",
-            }
-            for branch in data
-        ]
-        return branches
-
-    async def run(
-        self,
-        input_data: Input,
-        *,
-        credentials: GithubCredentials,
-        **kwargs,
-    ) -> BlockOutput:
-        branches = await self.list_branches(
-            credentials,
-            input_data.repo_url,
-        )
-        yield "branches", branches
-        for branch in branches:
-            yield "branch", branch
-
-
 class GithubListDiscussionsBlock(Block):
     class Input(BlockSchemaInput):
         credentials: GithubCredentialsInput = GithubCredentialsField("repo")
@@ -283,7 +187,7 @@ class GithubListDiscussionsBlock(Block):
     ) -> list[Output.DiscussionItem]:
         api = get_api(credentials)
         # GitHub GraphQL API endpoint is different; we'll use api.post with custom URL
-        repo_path = repo_url.replace("https://github.com/", "")
+        repo_path = github_repo_path(repo_url)
         owner, repo = repo_path.split("/")
         query = """
         query($owner: String!, $repo: String!, $num: Int!) {
@@ -416,564 +320,6 @@ class GithubListReleasesBlock(Block):
             yield "release", release
 
 
-class GithubReadFileBlock(Block):
-    class Input(BlockSchemaInput):
-        credentials: GithubCredentialsInput = GithubCredentialsField("repo")
-        repo_url: str = SchemaField(
-            description="URL of the GitHub repository",
-            placeholder="https://github.com/owner/repo",
-        )
-        file_path: str = SchemaField(
-            description="Path to the file in the repository",
-            placeholder="path/to/file",
-        )
-        branch: str = SchemaField(
-            description="Branch to read from",
-            placeholder="branch_name",
-            default="master",
-        )
-
-    class Output(BlockSchemaOutput):
-        text_content: str = SchemaField(
-            description="Content of the file (decoded as UTF-8 text)"
-        )
-        raw_content: str = SchemaField(
-            description="Raw base64-encoded content of the file"
-        )
-        size: int = SchemaField(description="The size of the file (in bytes)")
-
-    def __init__(self):
-        super().__init__(
-            id="87ce6c27-5752-4bbc-8e26-6da40a3dcfd3",
-            description="This block reads the content of a specified file from a GitHub repository.",
-            categories={BlockCategory.DEVELOPER_TOOLS},
-            input_schema=GithubReadFileBlock.Input,
-            output_schema=GithubReadFileBlock.Output,
-            test_input={
-                "repo_url": "https://github.com/owner/repo",
-                "file_path": "path/to/file",
-                "branch": "master",
-                "credentials": TEST_CREDENTIALS_INPUT,
-            },
-            test_credentials=TEST_CREDENTIALS,
-            test_output=[
-                ("raw_content", "RmlsZSBjb250ZW50"),
-                ("text_content", "File content"),
-                ("size", 13),
-            ],
-            test_mock={"read_file": lambda *args, **kwargs: ("RmlsZSBjb250ZW50", 13)},
-        )
-
-    @staticmethod
-    async def read_file(
-        credentials: GithubCredentials, repo_url: str, file_path: str, branch: str
-    ) -> tuple[str, int]:
-        api = get_api(credentials)
-        content_url = repo_url + f"/contents/{file_path}?ref={branch}"
-        response = await api.get(content_url)
-        data = response.json()
-
-        if isinstance(data, list):
-            # Multiple entries of different types exist at this path
-            if not (file := next((f for f in data if f["type"] == "file"), None)):
-                raise TypeError("Not a file")
-            data = file
-
-        if data["type"] != "file":
-            raise TypeError("Not a file")
-
-        return data["content"], data["size"]
-
-    async def run(
-        self,
-        input_data: Input,
-        *,
-        credentials: GithubCredentials,
-        **kwargs,
-    ) -> BlockOutput:
-        content, size = await self.read_file(
-            credentials,
-            input_data.repo_url,
-            input_data.file_path,
-            input_data.branch,
-        )
-        yield "raw_content", content
-        yield "text_content", base64.b64decode(content).decode("utf-8")
-        yield "size", size
-
-
-class GithubReadFolderBlock(Block):
-    class Input(BlockSchemaInput):
-        credentials: GithubCredentialsInput = GithubCredentialsField("repo")
-        repo_url: str = SchemaField(
-            description="URL of the GitHub repository",
-            placeholder="https://github.com/owner/repo",
-        )
-        folder_path: str = SchemaField(
-            description="Path to the folder in the repository",
-            placeholder="path/to/folder",
-        )
-        branch: str = SchemaField(
-            description="Branch name to read from (defaults to master)",
-            placeholder="branch_name",
-            default="master",
-        )
-
-    class Output(BlockSchemaOutput):
-        class DirEntry(TypedDict):
-            name: str
-            path: str
-
-        class FileEntry(TypedDict):
-            name: str
-            path: str
-            size: int
-
-        file: FileEntry = SchemaField(description="Files in the folder")
-        dir: DirEntry = SchemaField(description="Directories in the folder")
-        error: str = SchemaField(
-            description="Error message if reading the folder failed"
-        )
-
-    def __init__(self):
-        super().__init__(
-            id="1355f863-2db3-4d75-9fba-f91e8a8ca400",
-            description="This block reads the content of a specified folder from a GitHub repository.",
-            categories={BlockCategory.DEVELOPER_TOOLS},
-            input_schema=GithubReadFolderBlock.Input,
-            output_schema=GithubReadFolderBlock.Output,
-            test_input={
-                "repo_url": "https://github.com/owner/repo",
-                "folder_path": "path/to/folder",
-                "branch": "master",
-                "credentials": TEST_CREDENTIALS_INPUT,
-            },
-            test_credentials=TEST_CREDENTIALS,
-            test_output=[
-                (
-                    "file",
-                    {
-                        "name": "file1.txt",
-                        "path": "path/to/folder/file1.txt",
-                        "size": 1337,
-                    },
-                ),
-                ("dir", {"name": "dir2", "path": "path/to/folder/dir2"}),
-            ],
-            test_mock={
-                "read_folder": lambda *args, **kwargs: (
-                    [
-                        {
-                            "name": "file1.txt",
-                            "path": "path/to/folder/file1.txt",
-                            "size": 1337,
-                        }
-                    ],
-                    [{"name": "dir2", "path": "path/to/folder/dir2"}],
-                )
-            },
-        )
-
-    @staticmethod
-    async def read_folder(
-        credentials: GithubCredentials, repo_url: str, folder_path: str, branch: str
-    ) -> tuple[list[Output.FileEntry], list[Output.DirEntry]]:
-        api = get_api(credentials)
-        contents_url = repo_url + f"/contents/{folder_path}?ref={branch}"
-        response = await api.get(contents_url)
-        data = response.json()
-
-        if not isinstance(data, list):
-            raise TypeError("Not a folder")
-
-        files: list[GithubReadFolderBlock.Output.FileEntry] = [
-            GithubReadFolderBlock.Output.FileEntry(
-                name=entry["name"],
-                path=entry["path"],
-                size=entry["size"],
-            )
-            for entry in data
-            if entry["type"] == "file"
-        ]
-
-        dirs: list[GithubReadFolderBlock.Output.DirEntry] = [
-            GithubReadFolderBlock.Output.DirEntry(
-                name=entry["name"],
-                path=entry["path"],
-            )
-            for entry in data
-            if entry["type"] == "dir"
-        ]
-
-        return files, dirs
-
-    async def run(
-        self,
-        input_data: Input,
-        *,
-        credentials: GithubCredentials,
-        **kwargs,
-    ) -> BlockOutput:
-        files, dirs = await self.read_folder(
-            credentials,
-            input_data.repo_url,
-            input_data.folder_path.lstrip("/"),
-            input_data.branch,
-        )
-        for file in files:
-            yield "file", file
-        for dir in dirs:
-            yield "dir", dir
-
-
-class GithubMakeBranchBlock(Block):
-    class Input(BlockSchemaInput):
-        credentials: GithubCredentialsInput = GithubCredentialsField("repo")
-        repo_url: str = SchemaField(
-            description="URL of the GitHub repository",
-            placeholder="https://github.com/owner/repo",
-        )
-        new_branch: str = SchemaField(
-            description="Name of the new branch",
-            placeholder="new_branch_name",
-        )
-        source_branch: str = SchemaField(
-            description="Name of the source branch",
-            placeholder="source_branch_name",
-        )
-
-    class Output(BlockSchemaOutput):
-        status: str = SchemaField(description="Status of the branch creation operation")
-        error: str = SchemaField(
-            description="Error message if the branch creation failed"
-        )
-
-    def __init__(self):
-        super().__init__(
-            id="944cc076-95e7-4d1b-b6b6-b15d8ee5448d",
-            description="This block creates a new branch from a specified source branch.",
-            categories={BlockCategory.DEVELOPER_TOOLS},
-            input_schema=GithubMakeBranchBlock.Input,
-            output_schema=GithubMakeBranchBlock.Output,
-            test_input={
-                "repo_url": "https://github.com/owner/repo",
-                "new_branch": "new_branch_name",
-                "source_branch": "source_branch_name",
-                "credentials": TEST_CREDENTIALS_INPUT,
-            },
-            test_credentials=TEST_CREDENTIALS,
-            test_output=[("status", "Branch created successfully")],
-            test_mock={
-                "create_branch": lambda *args, **kwargs: "Branch created successfully"
-            },
-        )
-
-    @staticmethod
-    async def create_branch(
-        credentials: GithubCredentials,
-        repo_url: str,
-        new_branch: str,
-        source_branch: str,
-    ) -> str:
-        api = get_api(credentials)
-        ref_url = repo_url + f"/git/refs/heads/{source_branch}"
-        response = await api.get(ref_url)
-        data = response.json()
-        sha = data["object"]["sha"]
-
-        # Create the new branch
-        new_ref_url = repo_url + "/git/refs"
-        data = {
-            "ref": f"refs/heads/{new_branch}",
-            "sha": sha,
-        }
-        response = await api.post(new_ref_url, json=data)
-        return "Branch created successfully"
-
-    async def run(
-        self,
-        input_data: Input,
-        *,
-        credentials: GithubCredentials,
-        **kwargs,
-    ) -> BlockOutput:
-        status = await self.create_branch(
-            credentials,
-            input_data.repo_url,
-            input_data.new_branch,
-            input_data.source_branch,
-        )
-        yield "status", status
-
-
-class GithubDeleteBranchBlock(Block):
-    class Input(BlockSchemaInput):
-        credentials: GithubCredentialsInput = GithubCredentialsField("repo")
-        repo_url: str = SchemaField(
-            description="URL of the GitHub repository",
-            placeholder="https://github.com/owner/repo",
-        )
-        branch: str = SchemaField(
-            description="Name of the branch to delete",
-            placeholder="branch_name",
-        )
-
-    class Output(BlockSchemaOutput):
-        status: str = SchemaField(description="Status of the branch deletion operation")
-        error: str = SchemaField(
-            description="Error message if the branch deletion failed"
-        )
-
-    def __init__(self):
-        super().__init__(
-            id="0d4130f7-e0ab-4d55-adc3-0a40225e80f4",
-            description="This block deletes a specified branch.",
-            categories={BlockCategory.DEVELOPER_TOOLS},
-            input_schema=GithubDeleteBranchBlock.Input,
-            output_schema=GithubDeleteBranchBlock.Output,
-            test_input={
-                "repo_url": "https://github.com/owner/repo",
-                "branch": "branch_name",
-                "credentials": TEST_CREDENTIALS_INPUT,
-            },
-            test_credentials=TEST_CREDENTIALS,
-            test_output=[("status", "Branch deleted successfully")],
-            test_mock={
-                "delete_branch": lambda *args, **kwargs: "Branch deleted successfully"
-            },
-        )
-
-    @staticmethod
-    async def delete_branch(
-        credentials: GithubCredentials, repo_url: str, branch: str
-    ) -> str:
-        api = get_api(credentials)
-        ref_url = repo_url + f"/git/refs/heads/{branch}"
-        await api.delete(ref_url)
-        return "Branch deleted successfully"
-
-    async def run(
-        self,
-        input_data: Input,
-        *,
-        credentials: GithubCredentials,
-        **kwargs,
-    ) -> BlockOutput:
-        status = await self.delete_branch(
-            credentials,
-            input_data.repo_url,
-            input_data.branch,
-        )
-        yield "status", status
-
-
-class GithubCreateFileBlock(Block):
-    class Input(BlockSchemaInput):
-        credentials: GithubCredentialsInput = GithubCredentialsField("repo")
-        repo_url: str = SchemaField(
-            description="URL of the GitHub repository",
-            placeholder="https://github.com/owner/repo",
-        )
-        file_path: str = SchemaField(
-            description="Path where the file should be created",
-            placeholder="path/to/file.txt",
-        )
-        content: str = SchemaField(
-            description="Content to write to the file",
-            placeholder="File content here",
-        )
-        branch: str = SchemaField(
-            description="Branch where the file should be created",
-            default="main",
-        )
-        commit_message: str = SchemaField(
-            description="Message for the commit",
-            default="Create new file",
-        )
-
-    class Output(BlockSchemaOutput):
-        url: str = SchemaField(description="URL of the created file")
-        sha: str = SchemaField(description="SHA of the commit")
-        error: str = SchemaField(
-            description="Error message if the file creation failed"
-        )
-
-    def __init__(self):
-        super().__init__(
-            id="8fd132ac-b917-428a-8159-d62893e8a3fe",
-            description="This block creates a new file in a GitHub repository.",
-            categories={BlockCategory.DEVELOPER_TOOLS},
-            input_schema=GithubCreateFileBlock.Input,
-            output_schema=GithubCreateFileBlock.Output,
-            test_input={
-                "repo_url": "https://github.com/owner/repo",
-                "file_path": "test/file.txt",
-                "content": "Test content",
-                "branch": "main",
-                "commit_message": "Create test file",
-                "credentials": TEST_CREDENTIALS_INPUT,
-            },
-            test_credentials=TEST_CREDENTIALS,
-            test_output=[
-                ("url", "https://github.com/owner/repo/blob/main/test/file.txt"),
-                ("sha", "abc123"),
-            ],
-            test_mock={
-                "create_file": lambda *args, **kwargs: (
-                    "https://github.com/owner/repo/blob/main/test/file.txt",
-                    "abc123",
-                )
-            },
-        )
-
-    @staticmethod
-    async def create_file(
-        credentials: GithubCredentials,
-        repo_url: str,
-        file_path: str,
-        content: str,
-        branch: str,
-        commit_message: str,
-    ) -> tuple[str, str]:
-        api = get_api(credentials)
-        contents_url = repo_url + f"/contents/{file_path}"
-        content_base64 = base64.b64encode(content.encode()).decode()
-        data = {
-            "message": commit_message,
-            "content": content_base64,
-            "branch": branch,
-        }
-        response = await api.put(contents_url, json=data)
-        data = response.json()
-        return data["content"]["html_url"], data["commit"]["sha"]
-
-    async def run(
-        self,
-        input_data: Input,
-        *,
-        credentials: GithubCredentials,
-        **kwargs,
-    ) -> BlockOutput:
-        try:
-            url, sha = await self.create_file(
-                credentials,
-                input_data.repo_url,
-                input_data.file_path,
-                input_data.content,
-                input_data.branch,
-                input_data.commit_message,
-            )
-            yield "url", url
-            yield "sha", sha
-        except Exception as e:
-            yield "error", str(e)
-
-
-class GithubUpdateFileBlock(Block):
-    class Input(BlockSchemaInput):
-        credentials: GithubCredentialsInput = GithubCredentialsField("repo")
-        repo_url: str = SchemaField(
-            description="URL of the GitHub repository",
-            placeholder="https://github.com/owner/repo",
-        )
-        file_path: str = SchemaField(
-            description="Path to the file to update",
-            placeholder="path/to/file.txt",
-        )
-        content: str = SchemaField(
-            description="New content for the file",
-            placeholder="Updated content here",
-        )
-        branch: str = SchemaField(
-            description="Branch containing the file",
-            default="main",
-        )
-        commit_message: str = SchemaField(
-            description="Message for the commit",
-            default="Update file",
-        )
-
-    class Output(BlockSchemaOutput):
-        url: str = SchemaField(description="URL of the updated file")
-        sha: str = SchemaField(description="SHA of the commit")
-
-    def __init__(self):
-        super().__init__(
-            id="30be12a4-57cb-4aa4-baf5-fcc68d136076",
-            description="This block updates an existing file in a GitHub repository.",
-            categories={BlockCategory.DEVELOPER_TOOLS},
-            input_schema=GithubUpdateFileBlock.Input,
-            output_schema=GithubUpdateFileBlock.Output,
-            test_input={
-                "repo_url": "https://github.com/owner/repo",
-                "file_path": "test/file.txt",
-                "content": "Updated content",
-                "branch": "main",
-                "commit_message": "Update test file",
-                "credentials": TEST_CREDENTIALS_INPUT,
-            },
-            test_credentials=TEST_CREDENTIALS,
-            test_output=[
-                ("url", "https://github.com/owner/repo/blob/main/test/file.txt"),
-                ("sha", "def456"),
-            ],
-            test_mock={
-                "update_file": lambda *args, **kwargs: (
-                    "https://github.com/owner/repo/blob/main/test/file.txt",
-                    "def456",
-                )
-            },
-        )
-
-    @staticmethod
-    async def update_file(
-        credentials: GithubCredentials,
-        repo_url: str,
-        file_path: str,
-        content: str,
-        branch: str,
-        commit_message: str,
-    ) -> tuple[str, str]:
-        api = get_api(credentials)
-        contents_url = repo_url + f"/contents/{file_path}"
-        params = {"ref": branch}
-        response = await api.get(contents_url, params=params)
-        data = response.json()
-
-        # Convert new content to base64
-        content_base64 = base64.b64encode(content.encode()).decode()
-        data = {
-            "message": commit_message,
-            "content": content_base64,
-            "sha": data["sha"],
-            "branch": branch,
-        }
-        response = await api.put(contents_url, json=data)
-        data = response.json()
-        return data["content"]["html_url"], data["commit"]["sha"]
-
-    async def run(
-        self,
-        input_data: Input,
-        *,
-        credentials: GithubCredentials,
-        **kwargs,
-    ) -> BlockOutput:
-        try:
-            url, sha = await self.update_file(
-                credentials,
-                input_data.repo_url,
-                input_data.file_path,
-                input_data.content,
-                input_data.branch,
-                input_data.commit_message,
-            )
-            yield "url", url
-            yield "sha", sha
-        except Exception as e:
-            yield "error", str(e)
-
-
 class GithubCreateRepositoryBlock(Block):
     class Input(BlockSchemaInput):
         credentials: GithubCredentialsInput = GithubCredentialsField("repo")
@@ -1103,7 +449,7 @@ class GithubListStargazersBlock(Block):
 
     def __init__(self):
         super().__init__(
-            id="a4b9c2d1-e5f6-4g7h-8i9j-0k1l2m3n4o5p",  # Generated unique UUID
+            id="e96d01ec-b55e-4a99-8ce8-c8776dce850b",  # Generated unique UUID
             description="This block lists all users who have starred a specified GitHub repository.",
             categories={BlockCategory.DEVELOPER_TOOLS},
             input_schema=GithubListStargazersBlock.Input,
@@ -1172,3 +518,230 @@ class GithubListStargazersBlock(Block):
         yield "stargazers", stargazers
         for stargazer in stargazers:
             yield "stargazer", stargazer
+
+
+class GithubGetRepositoryInfoBlock(Block):
+    class Input(BlockSchemaInput):
+        credentials: GithubCredentialsInput = GithubCredentialsField("repo")
+        repo_url: str = SchemaField(
+            description="URL of the GitHub repository",
+            placeholder="https://github.com/owner/repo",
+        )
+
+    class Output(BlockSchemaOutput):
+        name: str = SchemaField(description="Repository name")
+        full_name: str = SchemaField(description="Full repository name (owner/repo)")
+        description: str = SchemaField(description="Repository description")
+        default_branch: str = SchemaField(description="Default branch name (e.g. main)")
+        private: bool = SchemaField(description="Whether the repository is private")
+        html_url: str = SchemaField(description="Web URL of the repository")
+        clone_url: str = SchemaField(description="Git clone URL")
+        stars: int = SchemaField(description="Number of stars")
+        forks: int = SchemaField(description="Number of forks")
+        open_issues: int = SchemaField(description="Number of open issues")
+        error: str = SchemaField(
+            description="Error message if fetching repo info failed"
+        )
+
+    def __init__(self):
+        super().__init__(
+            id="59d4f241-968a-4040-95da-348ac5c5ce27",
+            description="This block retrieves metadata about a GitHub repository.",
+            categories={BlockCategory.DEVELOPER_TOOLS},
+            input_schema=GithubGetRepositoryInfoBlock.Input,
+            output_schema=GithubGetRepositoryInfoBlock.Output,
+            test_input={
+                "repo_url": "https://github.com/owner/repo",
+                "credentials": TEST_CREDENTIALS_INPUT,
+            },
+            test_credentials=TEST_CREDENTIALS,
+            test_output=[
+                ("name", "repo"),
+                ("full_name", "owner/repo"),
+                ("description", "A test repo"),
+                ("default_branch", "main"),
+                ("private", False),
+                ("html_url", "https://github.com/owner/repo"),
+                ("clone_url", "https://github.com/owner/repo.git"),
+                ("stars", 42),
+                ("forks", 5),
+                ("open_issues", 3),
+            ],
+            test_mock={
+                "get_repo_info": lambda *args, **kwargs: {
+                    "name": "repo",
+                    "full_name": "owner/repo",
+                    "description": "A test repo",
+                    "default_branch": "main",
+                    "private": False,
+                    "html_url": "https://github.com/owner/repo",
+                    "clone_url": "https://github.com/owner/repo.git",
+                    "stargazers_count": 42,
+                    "forks_count": 5,
+                    "open_issues_count": 3,
+                }
+            },
+        )
+
+    @staticmethod
+    async def get_repo_info(credentials: GithubCredentials, repo_url: str) -> dict:
+        api = get_api(credentials)
+        response = await api.get(repo_url)
+        return response.json()
+
+    async def run(
+        self,
+        input_data: Input,
+        *,
+        credentials: GithubCredentials,
+        **kwargs,
+    ) -> BlockOutput:
+        try:
+            data = await self.get_repo_info(credentials, input_data.repo_url)
+            yield "name", data["name"]
+            yield "full_name", data["full_name"]
+            yield "description", data.get("description", "") or ""
+            yield "default_branch", data["default_branch"]
+            yield "private", data["private"]
+            yield "html_url", data["html_url"]
+            yield "clone_url", data["clone_url"]
+            yield "stars", data["stargazers_count"]
+            yield "forks", data["forks_count"]
+            yield "open_issues", data["open_issues_count"]
+        except Exception as e:
+            yield "error", str(e)
+
+
+class GithubForkRepositoryBlock(Block):
+    class Input(BlockSchemaInput):
+        credentials: GithubCredentialsInput = GithubCredentialsField("repo")
+        repo_url: str = SchemaField(
+            description="URL of the GitHub repository to fork",
+            placeholder="https://github.com/owner/repo",
+        )
+        organization: str = SchemaField(
+            description="Organization to fork into (leave empty to fork to your account)",
+            default="",
+        )
+
+    class Output(BlockSchemaOutput):
+        url: str = SchemaField(description="URL of the forked repository")
+        clone_url: str = SchemaField(description="Git clone URL of the fork")
+        full_name: str = SchemaField(description="Full name of the fork (owner/repo)")
+        error: str = SchemaField(description="Error message if the fork failed")
+
+    def __init__(self):
+        super().__init__(
+            id="a439f2f4-835f-4dae-ba7b-0205ffa70be6",
+            description="This block forks a GitHub repository to your account or an organization.",
+            categories={BlockCategory.DEVELOPER_TOOLS},
+            input_schema=GithubForkRepositoryBlock.Input,
+            output_schema=GithubForkRepositoryBlock.Output,
+            test_input={
+                "repo_url": "https://github.com/owner/repo",
+                "organization": "",
+                "credentials": TEST_CREDENTIALS_INPUT,
+            },
+            test_credentials=TEST_CREDENTIALS,
+            test_output=[
+                ("url", "https://github.com/myuser/repo"),
+                ("clone_url", "https://github.com/myuser/repo.git"),
+                ("full_name", "myuser/repo"),
+            ],
+            test_mock={
+                "fork_repo": lambda *args, **kwargs: (
+                    "https://github.com/myuser/repo",
+                    "https://github.com/myuser/repo.git",
+                    "myuser/repo",
+                )
+            },
+        )
+
+    @staticmethod
+    async def fork_repo(
+        credentials: GithubCredentials,
+        repo_url: str,
+        organization: str,
+    ) -> tuple[str, str, str]:
+        api = get_api(credentials)
+        forks_url = repo_url + "/forks"
+        data: dict[str, str] = {}
+        if organization:
+            data["organization"] = organization
+        response = await api.post(forks_url, json=data)
+        result = response.json()
+        return result["html_url"], result["clone_url"], result["full_name"]
+
+    async def run(
+        self,
+        input_data: Input,
+        *,
+        credentials: GithubCredentials,
+        **kwargs,
+    ) -> BlockOutput:
+        try:
+            url, clone_url, full_name = await self.fork_repo(
+                credentials,
+                input_data.repo_url,
+                input_data.organization,
+            )
+            yield "url", url
+            yield "clone_url", clone_url
+            yield "full_name", full_name
+        except Exception as e:
+            yield "error", str(e)
+
+
+class GithubStarRepositoryBlock(Block):
+    class Input(BlockSchemaInput):
+        credentials: GithubCredentialsInput = GithubCredentialsField("repo")
+        repo_url: str = SchemaField(
+            description="URL of the GitHub repository to star",
+            placeholder="https://github.com/owner/repo",
+        )
+
+    class Output(BlockSchemaOutput):
+        status: str = SchemaField(description="Status of the star operation")
+        error: str = SchemaField(description="Error message if starring failed")
+
+    def __init__(self):
+        super().__init__(
+            id="bd700764-53e3-44dd-a969-d1854088458f",
+            description="This block stars a GitHub repository.",
+            categories={BlockCategory.DEVELOPER_TOOLS},
+            input_schema=GithubStarRepositoryBlock.Input,
+            output_schema=GithubStarRepositoryBlock.Output,
+            test_input={
+                "repo_url": "https://github.com/owner/repo",
+                "credentials": TEST_CREDENTIALS_INPUT,
+            },
+            test_credentials=TEST_CREDENTIALS,
+            test_output=[("status", "Repository starred successfully")],
+            test_mock={
+                "star_repo": lambda *args, **kwargs: "Repository starred successfully"
+            },
+        )
+
+    @staticmethod
+    async def star_repo(credentials: GithubCredentials, repo_url: str) -> str:
+        api = get_api(credentials, convert_urls=False)
+        repo_path = github_repo_path(repo_url)
+        owner, repo = repo_path.split("/")
+        await api.put(
+            f"https://api.github.com/user/starred/{owner}/{repo}",
+            headers={"Content-Length": "0"},
+        )
+        return "Repository starred successfully"
+
+    async def run(
+        self,
+        input_data: Input,
+        *,
+        credentials: GithubCredentials,
+        **kwargs,
+    ) -> BlockOutput:
+        try:
+            status = await self.star_repo(credentials, input_data.repo_url)
+            yield "status", status
+        except Exception as e:
+            yield "error", str(e)

autogpt_platform/backend/backend/blocks/github/repo_branches.py

@@ -0,0 +1,452 @@
+from urllib.parse import quote
+
+from typing_extensions import TypedDict
+
+from backend.blocks._base import (
+    Block,
+    BlockCategory,
+    BlockOutput,
+    BlockSchemaInput,
+    BlockSchemaOutput,
+)
+from backend.data.model import SchemaField
+
+from ._api import get_api
+from ._auth import (
+    TEST_CREDENTIALS,
+    TEST_CREDENTIALS_INPUT,
+    GithubCredentials,
+    GithubCredentialsField,
+    GithubCredentialsInput,
+)
+from ._utils import github_repo_path
+
+
+class GithubListBranchesBlock(Block):
+    class Input(BlockSchemaInput):
+        credentials: GithubCredentialsInput = GithubCredentialsField("repo")
+        repo_url: str = SchemaField(
+            description="URL of the GitHub repository",
+            placeholder="https://github.com/owner/repo",
+        )
+        per_page: int = SchemaField(
+            description="Number of branches to return per page (max 100)",
+            default=30,
+            ge=1,
+            le=100,
+        )
+        page: int = SchemaField(
+            description="Page number for pagination",
+            default=1,
+            ge=1,
+        )
+
+    class Output(BlockSchemaOutput):
+        class BranchItem(TypedDict):
+            name: str
+            url: str
+
+        branch: BranchItem = SchemaField(
+            title="Branch",
+            description="Branches with their name and file tree browser URL",
+        )
+        branches: list[BranchItem] = SchemaField(
+            description="List of branches with their name and file tree browser URL"
+        )
+        error: str = SchemaField(description="Error message if listing branches failed")
+
+    def __init__(self):
+        super().__init__(
+            id="74243e49-2bec-4916-8bf4-db43d44aead5",
+            description="This block lists all branches for a specified GitHub repository.",
+            categories={BlockCategory.DEVELOPER_TOOLS},
+            input_schema=GithubListBranchesBlock.Input,
+            output_schema=GithubListBranchesBlock.Output,
+            test_input={
+                "repo_url": "https://github.com/owner/repo",
+                "per_page": 30,
+                "page": 1,
+                "credentials": TEST_CREDENTIALS_INPUT,
+            },
+            test_credentials=TEST_CREDENTIALS,
+            test_output=[
+                (
+                    "branches",
+                    [
+                        {
+                            "name": "main",
+                            "url": "https://github.com/owner/repo/tree/main",
+                        }
+                    ],
+                ),
+                (
+                    "branch",
+                    {
+                        "name": "main",
+                        "url": "https://github.com/owner/repo/tree/main",
+                    },
+                ),
+            ],
+            test_mock={
+                "list_branches": lambda *args, **kwargs: [
+                    {
+                        "name": "main",
+                        "url": "https://github.com/owner/repo/tree/main",
+                    }
+                ]
+            },
+        )
+
+    @staticmethod
+    async def list_branches(
+        credentials: GithubCredentials, repo_url: str, per_page: int, page: int
+    ) -> list[Output.BranchItem]:
+        api = get_api(credentials)
+        branches_url = repo_url + "/branches"
+        response = await api.get(
+            branches_url, params={"per_page": str(per_page), "page": str(page)}
+        )
+        data = response.json()
+        repo_path = github_repo_path(repo_url)
+        branches: list[GithubListBranchesBlock.Output.BranchItem] = [
+            {
+                "name": branch["name"],
+                "url": f"https://github.com/{repo_path}/tree/{branch['name']}",
+            }
+            for branch in data
+        ]
+        return branches
+
+    async def run(
+        self,
+        input_data: Input,
+        *,
+        credentials: GithubCredentials,
+        **kwargs,
+    ) -> BlockOutput:
+        try:
+            branches = await self.list_branches(
+                credentials,
+                input_data.repo_url,
+                input_data.per_page,
+                input_data.page,
+            )
+            yield "branches", branches
+            for branch in branches:
+                yield "branch", branch
+        except Exception as e:
+            yield "error", str(e)
+
+
+class GithubMakeBranchBlock(Block):
+    class Input(BlockSchemaInput):
+        credentials: GithubCredentialsInput = GithubCredentialsField("repo")
+        repo_url: str = SchemaField(
+            description="URL of the GitHub repository",
+            placeholder="https://github.com/owner/repo",
+        )
+        new_branch: str = SchemaField(
+            description="Name of the new branch",
+            placeholder="new_branch_name",
+        )
+        source_branch: str = SchemaField(
+            description="Name of the source branch",
+            placeholder="source_branch_name",
+        )
+
+    class Output(BlockSchemaOutput):
+        status: str = SchemaField(description="Status of the branch creation operation")
+        error: str = SchemaField(
+            description="Error message if the branch creation failed"
+        )
+
+    def __init__(self):
+        super().__init__(
+            id="944cc076-95e7-4d1b-b6b6-b15d8ee5448d",
+            description="This block creates a new branch from a specified source branch.",
+            categories={BlockCategory.DEVELOPER_TOOLS},
+            input_schema=GithubMakeBranchBlock.Input,
+            output_schema=GithubMakeBranchBlock.Output,
+            test_input={
+                "repo_url": "https://github.com/owner/repo",
+                "new_branch": "new_branch_name",
+                "source_branch": "source_branch_name",
+                "credentials": TEST_CREDENTIALS_INPUT,
+            },
+            test_credentials=TEST_CREDENTIALS,
+            test_output=[("status", "Branch created successfully")],
+            test_mock={
+                "create_branch": lambda *args, **kwargs: "Branch created successfully"
+            },
+        )
+
+    @staticmethod
+    async def create_branch(
+        credentials: GithubCredentials,
+        repo_url: str,
+        new_branch: str,
+        source_branch: str,
+    ) -> str:
+        api = get_api(credentials)
+        ref_url = repo_url + f"/git/refs/heads/{quote(source_branch, safe='')}"
+        response = await api.get(ref_url)
+        data = response.json()
+        sha = data["object"]["sha"]
+
+        # Create the new branch
+        new_ref_url = repo_url + "/git/refs"
+        data = {
+            "ref": f"refs/heads/{new_branch}",
+            "sha": sha,
+        }
+        response = await api.post(new_ref_url, json=data)
+        return "Branch created successfully"
+
+    async def run(
+        self,
+        input_data: Input,
+        *,
+        credentials: GithubCredentials,
+        **kwargs,
+    ) -> BlockOutput:
+        try:
+            status = await self.create_branch(
+                credentials,
+                input_data.repo_url,
+                input_data.new_branch,
+                input_data.source_branch,
+            )
+            yield "status", status
+        except Exception as e:
+            yield "error", str(e)
+
+
+class GithubDeleteBranchBlock(Block):
+    class Input(BlockSchemaInput):
+        credentials: GithubCredentialsInput = GithubCredentialsField("repo")
+        repo_url: str = SchemaField(
+            description="URL of the GitHub repository",
+            placeholder="https://github.com/owner/repo",
+        )
+        branch: str = SchemaField(
+            description="Name of the branch to delete",
+            placeholder="branch_name",
+        )
+
+    class Output(BlockSchemaOutput):
+        status: str = SchemaField(description="Status of the branch deletion operation")
+        error: str = SchemaField(
+            description="Error message if the branch deletion failed"
+        )
+
+    def __init__(self):
+        super().__init__(
+            id="0d4130f7-e0ab-4d55-adc3-0a40225e80f4",
+            description="This block deletes a specified branch.",
+            categories={BlockCategory.DEVELOPER_TOOLS},
+            input_schema=GithubDeleteBranchBlock.Input,
+            output_schema=GithubDeleteBranchBlock.Output,
+            test_input={
+                "repo_url": "https://github.com/owner/repo",
+                "branch": "branch_name",
+                "credentials": TEST_CREDENTIALS_INPUT,
+            },
+            test_credentials=TEST_CREDENTIALS,
+            test_output=[("status", "Branch deleted successfully")],
+            test_mock={
+                "delete_branch": lambda *args, **kwargs: "Branch deleted successfully"
+            },
+            is_sensitive_action=True,
+        )
+
+    @staticmethod
+    async def delete_branch(
+        credentials: GithubCredentials, repo_url: str, branch: str
+    ) -> str:
+        api = get_api(credentials)
+        ref_url = repo_url + f"/git/refs/heads/{quote(branch, safe='')}"
+        await api.delete(ref_url)
+        return "Branch deleted successfully"
+
+    async def run(
+        self,
+        input_data: Input,
+        *,
+        credentials: GithubCredentials,
+        **kwargs,
+    ) -> BlockOutput:
+        try:
+            status = await self.delete_branch(
+                credentials,
+                input_data.repo_url,
+                input_data.branch,
+            )
+            yield "status", status
+        except Exception as e:
+            yield "error", str(e)
+
+
+class GithubCompareBranchesBlock(Block):
+    class Input(BlockSchemaInput):
+        credentials: GithubCredentialsInput = GithubCredentialsField("repo")
+        repo_url: str = SchemaField(
+            description="URL of the GitHub repository",
+            placeholder="https://github.com/owner/repo",
+        )
+        base: str = SchemaField(
+            description="Base branch or commit SHA",
+            placeholder="main",
+        )
+        head: str = SchemaField(
+            description="Head branch or commit SHA to compare against base",
+            placeholder="feature-branch",
+        )
+
+    class Output(BlockSchemaOutput):
+        class FileChange(TypedDict):
+            filename: str
+            status: str
+            additions: int
+            deletions: int
+            patch: str
+
+        status: str = SchemaField(
+            description="Comparison status: ahead, behind, diverged, or identical"
+        )
+        ahead_by: int = SchemaField(
+            description="Number of commits head is ahead of base"
+        )
+        behind_by: int = SchemaField(
+            description="Number of commits head is behind base"
+        )
+        total_commits: int = SchemaField(
+            description="Total number of commits in the comparison"
+        )
+        diff: str = SchemaField(description="Unified diff of all file changes")
+        file: FileChange = SchemaField(
+            title="Changed File", description="A changed file with its diff"
+        )
+        files: list[FileChange] = SchemaField(
+            description="List of changed files with their diffs"
+        )
+        error: str = SchemaField(description="Error message if comparison failed")
+
+    def __init__(self):
+        super().__init__(
+            id="2e4faa8c-6086-4546-ba77-172d1d560186",
+            description="This block compares two branches or commits in a GitHub repository.",
+            categories={BlockCategory.DEVELOPER_TOOLS},
+            input_schema=GithubCompareBranchesBlock.Input,
+            output_schema=GithubCompareBranchesBlock.Output,
+            test_input={
+                "repo_url": "https://github.com/owner/repo",
+                "base": "main",
+                "head": "feature",
+                "credentials": TEST_CREDENTIALS_INPUT,
+            },
+            test_credentials=TEST_CREDENTIALS,
+            test_output=[
+                ("status", "ahead"),
+                ("ahead_by", 2),
+                ("behind_by", 0),
+                ("total_commits", 2),
+                ("diff", "+++ b/file.py\n+new line"),
+                (
+                    "files",
+                    [
+                        {
+                            "filename": "file.py",
+                            "status": "modified",
+                            "additions": 1,
+                            "deletions": 0,
+                            "patch": "+new line",
+                        }
+                    ],
+                ),
+                (
+                    "file",
+                    {
+                        "filename": "file.py",
+                        "status": "modified",
+                        "additions": 1,
+                        "deletions": 0,
+                        "patch": "+new line",
+                    },
+                ),
+            ],
+            test_mock={
+                "compare_branches": lambda *args, **kwargs: {
+                    "status": "ahead",
+                    "ahead_by": 2,
+                    "behind_by": 0,
+                    "total_commits": 2,
+                    "files": [
+                        {
+                            "filename": "file.py",
+                            "status": "modified",
+                            "additions": 1,
+                            "deletions": 0,
+                            "patch": "+new line",
+                        }
+                    ],
+                }
+            },
+        )
+
+    @staticmethod
+    async def compare_branches(
+        credentials: GithubCredentials,
+        repo_url: str,
+        base: str,
+        head: str,
+    ) -> dict:
+        api = get_api(credentials)
+        safe_base = quote(base, safe="")
+        safe_head = quote(head, safe="")
+        compare_url = repo_url + f"/compare/{safe_base}...{safe_head}"
+        response = await api.get(compare_url)
+        return response.json()
+
+    async def run(
+        self,
+        input_data: Input,
+        *,
+        credentials: GithubCredentials,
+        **kwargs,
+    ) -> BlockOutput:
+        try:
+            data = await self.compare_branches(
+                credentials,
+                input_data.repo_url,
+                input_data.base,
+                input_data.head,
+            )
+            yield "status", data["status"]
+            yield "ahead_by", data["ahead_by"]
+            yield "behind_by", data["behind_by"]
+            yield "total_commits", data["total_commits"]
+
+            files: list[GithubCompareBranchesBlock.Output.FileChange] = [
+                GithubCompareBranchesBlock.Output.FileChange(
+                    filename=f["filename"],
+                    status=f["status"],
+                    additions=f["additions"],
+                    deletions=f["deletions"],
+                    patch=f.get("patch", ""),
+                )
+                for f in data.get("files", [])
+            ]
+
+            # Build unified diff
+            diff_parts = []
+            for f in data.get("files", []):
+                patch = f.get("patch", "")
+                if patch:
+                    diff_parts.append(f"+++ b/{f['filename']}\n{patch}")
+            yield "diff", "\n".join(diff_parts)
+
+            yield "files", files
+            for file in files:
+                yield "file", file
+        except Exception as e:
+            yield "error", str(e)