Merge remote-tracking branch 'origin/dev' into fix/sentry-error-cleanup

The `@@agptfile:` expansion system previously used content-sniffing
(trying
`json.loads` then `csv.Sniffer`) to decide whether to parse file content
as
structured data. This was fragile — a file containing just `"42"` would
be
parsed as an integer, and the heuristics could misfire on ambiguous
content.

This PR replaces content-sniffing with **extension/MIME-based format
detection**.
When the file has a well-known extension (`.json`, `.csv`, etc.) or MIME
type
fragment (`workspace://id#application/json`), the content is parsed
accordingly.
Unknown formats or parse failures always fall back to plain string — no
surprises.

> [!NOTE]
> This PR builds on the `@@agptfile:` file reference protocol introduced
in #12332 and the structured data auto-parsing added in #12390.
>
> **What is `@@agptfile:`?**
> It is a special URI prefix (e.g. `@@agptfile:workspace:///report.csv`)
that the CoPilot SDK expands inline before sending tool arguments to
blocks. This lets the AI reference workspace files by name, and the SDK
automatically reads and injects the file content. See #12332 for the
full design.

### Changes 🏗️

**New utility: `backend/util/file_content_parser.py`**
- `infer_format(uri)` — determines format from file extension or MIME
fragment
- `parse_file_content(content, fmt)` — parses content, never raises
- Supported text formats: JSON, JSONL/NDJSON, CSV, TSV, YAML, TOML
- Supported binary formats: Parquet (via pyarrow), Excel/XLSX (via
openpyxl)
- JSON scalars (strings, numbers, booleans, null) stay as strings — only
  containers (arrays, objects) are promoted
- CSV/TSV require ≥1 row and ≥2 columns to qualify as tabular data
- Added `openpyxl` dependency for Excel reading via pandas
- Case-insensitive MIME fragment matching per RFC 2045
- Shared `PARSE_EXCEPTIONS` constant to avoid duplication between
modules

**Updated `expand_file_refs_in_args` in `file_ref.py`**
- Bare refs now use `infer_format` + `parse_file_content` instead of the
  old `_try_parse_structured` content-sniffing function
- Binary formats (parquet, xlsx) read raw bytes via `read_file_bytes`
- Embedded refs (text around `@@agptfile:`) still produce plain strings
- **Size guards**: Workspace and sandbox file reads now enforce a 10 MB
limit
  (matching the existing local file limit) to prevent OOM on large files

**Updated `blocks/github/commits.py`**
- Consolidated `_create_blob` and `_create_binary_blob` into a single
function
  with an `encoding` parameter

**Updated copilot system prompt**
- Documents the extension-based structured data parsing and supported
formats

**66 new tests** in `file_content_parser_test.py` covering:
- Format inference (extension, MIME, case-insensitive, precedence)
- All 8 format parsers (happy path + edge cases + fallbacks)
- Binary format handling (string input fallback, invalid bytes fallback)
- Unknown format passthrough

### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
  - [x] All 66 file_content_parser_test.py tests pass
  - [x] All 31 file_ref_test.py tests pass
  - [x] All 13 file_ref_integration_test.py tests pass
  - [x] `poetry run format` passes clean (including pyright)

.claude/skills/pr-address/SKILL.md

@@ -0,0 +1,79 @@
+---
+name: pr-address
+description: Address PR review comments and loop until CI green and all comments resolved. TRIGGER when user asks to address comments, fix PR feedback, respond to reviewers, or babysit/monitor a PR.
+user-invocable: true
+args: "[PR number or URL] — if omitted, finds PR for current branch."
+metadata:
+  author: autogpt-team
+  version: "1.0.0"
+---
+
+# PR Address
+
+## Find the PR
+
+```bash
+gh pr list --head $(git branch --show-current) --repo Significant-Gravitas/AutoGPT
+gh pr view {N}
+```
+
+## Fetch comments (all sources)
+
+```bash
+gh api repos/Significant-Gravitas/AutoGPT/pulls/{N}/reviews       # top-level reviews
+gh api repos/Significant-Gravitas/AutoGPT/pulls/{N}/comments      # inline review comments
+gh api repos/Significant-Gravitas/AutoGPT/issues/{N}/comments     # PR conversation comments
+```
+
+**Bots to watch for:**
+- `autogpt-reviewer` — posts "Blockers", "Should Fix", "Nice to Have". Address ALL of them.
+- `sentry[bot]` — bug predictions. Fix real bugs, explain false positives.
+- `coderabbitai[bot]` — automated review. Address actionable items.
+
+## For each unaddressed comment
+
+Address comments **one at a time**: fix → commit → push → inline reply → next.
+
+1. Read the referenced code, make the fix (or reply explaining why it's not needed)
+2. Commit and push the fix
+3. Reply **inline** (not as a new top-level comment) referencing the fixing commit — this is what resolves the conversation for bot reviewers (coderabbitai, sentry):
+
+| Comment type | How to reply |
+|---|---|
+| Inline review (`pulls/{N}/comments`) | `gh api repos/Significant-Gravitas/AutoGPT/pulls/{N}/comments/{ID}/replies -f body="Fixed in <commit-sha>: <description>"` |
+| Conversation (`issues/{N}/comments`) | `gh api repos/Significant-Gravitas/AutoGPT/issues/{N}/comments -f body="Fixed in <commit-sha>: <description>"` |
+
+## Format and commit
+
+After fixing, format the changed code:
+
+- **Backend** (from `autogpt_platform/backend/`): `poetry run format`
+- **Frontend** (from `autogpt_platform/frontend/`): `pnpm format && pnpm lint && pnpm types`
+
+If API routes changed, regenerate the frontend client:
+```bash
+cd autogpt_platform/backend && poetry run rest &
+REST_PID=$!
+trap "kill $REST_PID 2>/dev/null" EXIT
+WAIT=0; until curl -sf http://localhost:8006/health > /dev/null 2>&1; do sleep 1; WAIT=$((WAIT+1)); [ $WAIT -ge 60 ] && echo "Timed out" && exit 1; done
+cd ../frontend && pnpm generate:api:force
+kill $REST_PID 2>/dev/null; trap - EXIT
+```
+Never manually edit files in `src/app/api/__generated__/`.
+
+Then commit and **push immediately** — never batch commits without pushing.
+
+For backend commits in worktrees: `poetry run git commit` (pre-commit hooks).
+
+## The loop
+
+```text
+address comments → format → commit → push
+→ re-check comments → fix new ones → push
+→ wait for CI → re-check comments after CI settles
+→ repeat until: all comments addressed AND CI green AND no new comments arriving
+```
+
+While CI runs, stay productive: run local tests, address remaining comments.
+
+**The loop ends when:** CI fully green + all comments addressed + no new comments since CI settled.

.claude/skills/pr-review/SKILL.md

@@ -0,0 +1,74 @@
+---
+name: pr-review
+description: Review a PR for correctness, security, code quality, and testing issues. TRIGGER when user asks to review a PR, check PR quality, or give feedback on a PR.
+user-invocable: true
+args: "[PR number or URL] — if omitted, finds PR for current branch."
+metadata:
+  author: autogpt-team
+  version: "1.0.0"
+---
+
+# PR Review
+
+## Find the PR
+
+```bash
+gh pr list --head $(git branch --show-current) --repo Significant-Gravitas/AutoGPT
+gh pr view {N}
+```
+
+## Read the diff
+
+```bash
+gh pr diff {N}
+```
+
+## Fetch existing review comments
+
+Before posting anything, fetch existing inline comments to avoid duplicates:
+
+```bash
+gh api repos/Significant-Gravitas/AutoGPT/pulls/{N}/comments
+gh api repos/Significant-Gravitas/AutoGPT/pulls/{N}/reviews
+```
+
+## What to check
+
+**Correctness:** logic errors, off-by-one, missing edge cases, race conditions (TOCTOU in file access, credit charging), error handling gaps, async correctness (missing `await`, unclosed resources).
+
+**Security:** input validation at boundaries, no injection (command, XSS, SQL), secrets not logged, file paths sanitized (`os.path.basename()` in error messages).
+
+**Code quality:** apply rules from backend/frontend CLAUDE.md files.
+
+**Architecture:** DRY, single responsibility, modular functions. `Security()` vs `Depends()` for FastAPI auth. `data:` for SSE events, `: comment` for heartbeats. `transaction=True` for Redis pipelines.
+
+**Testing:** edge cases covered, colocated `*_test.py` (backend) / `__tests__/` (frontend), mocks target where symbol is **used** not defined, `AsyncMock` for async.
+
+## Output format
+
+Every comment **must** be prefixed with `🤖` and a criticality badge:
+
+| Tier | Badge | Meaning |
+|---|---|---|
+| Blocker | `🔴 **Blocker**` | Must fix before merge |
+| Should Fix | `🟠 **Should Fix**` | Important improvement |
+| Nice to Have | `🟡 **Nice to Have**` | Minor suggestion |
+| Nit | `🔵 **Nit**` | Style / wording |
+
+Example: `🤖 🔴 **Blocker**: Missing error handling for X — suggest wrapping in try/except.`
+
+## Post inline comments
+
+For each finding, post an inline comment on the PR (do not just write a local report):
+
+```bash
+# Get the latest commit SHA for the PR
+COMMIT_SHA=$(gh api repos/Significant-Gravitas/AutoGPT/pulls/{N} --jq '.head.sha')
+
+# Post an inline comment on a specific file/line
+gh api repos/Significant-Gravitas/AutoGPT/pulls/{N}/comments \
+  -f body="🤖 🔴 **Blocker**: <description>" \
+  -f commit_id="$COMMIT_SHA" \
+  -f path="<file path>" \
+  -F line=<line number>
+```

.claude/skills/worktree/SKILL.md

@@ -0,0 +1,85 @@
+---
+name: worktree
+description: Set up a new git worktree for parallel development. Creates the worktree, copies .env files, installs dependencies, and generates Prisma client. TRIGGER when user asks to set up a worktree, work on a branch in isolation, or needs a separate environment for a branch or PR.
+user-invocable: true
+args: "[name] — optional worktree name (e.g., 'AutoGPT7'). If omitted, uses next available AutoGPT<N>."
+metadata:
+  author: autogpt-team
+  version: "3.0.0"
+---
+
+# Worktree Setup
+
+## Create the worktree
+
+Derive paths from the git toplevel. If a name is provided as argument, use it. Otherwise, check `git worktree list` and pick the next `AutoGPT<N>`.
+
+```bash
+ROOT=$(git rev-parse --show-toplevel)
+PARENT=$(dirname "$ROOT")
+
+# From an existing branch
+git worktree add "$PARENT/<NAME>" <branch-name>
+
+# From a new branch off dev
+git worktree add -b <new-branch> "$PARENT/<NAME>" dev
+```
+
+## Copy environment files
+
+Copy `.env` from the root worktree. Falls back to `.env.default` if `.env` doesn't exist.
+
+```bash
+ROOT=$(git rev-parse --show-toplevel)
+TARGET="$(dirname "$ROOT")/<NAME>"
+
+for envpath in autogpt_platform/backend autogpt_platform/frontend autogpt_platform; do
+  if [ -f "$ROOT/$envpath/.env" ]; then
+    cp "$ROOT/$envpath/.env" "$TARGET/$envpath/.env"
+  elif [ -f "$ROOT/$envpath/.env.default" ]; then
+    cp "$ROOT/$envpath/.env.default" "$TARGET/$envpath/.env"
+  fi
+done
+```
+
+## Install dependencies
+
+```bash
+TARGET="$(dirname "$(git rev-parse --show-toplevel)")/<NAME>"
+cd "$TARGET/autogpt_platform/autogpt_libs" && poetry install
+cd "$TARGET/autogpt_platform/backend" && poetry install && poetry run prisma generate
+cd "$TARGET/autogpt_platform/frontend" && pnpm install
+```
+
+Replace `<NAME>` with the actual worktree name (e.g., `AutoGPT7`).
+
+## Running the app (optional)
+
+Backend uses ports: 8001, 8002, 8003, 8005, 8006, 8007, 8008. Free them first if needed:
+
+```bash
+TARGET="$(dirname "$(git rev-parse --show-toplevel)")/<NAME>"
+for port in 8001 8002 8003 8005 8006 8007 8008; do
+  lsof -ti :$port | xargs kill -9 2>/dev/null || true
+done
+cd "$TARGET/autogpt_platform/backend" && poetry run app
+```
+
+## CoPilot testing
+
+SDK mode spawns a Claude subprocess — won't work inside Claude Code. Set `CHAT_USE_CLAUDE_AGENT_SDK=false` in `backend/.env` to use baseline mode.
+
+## Cleanup
+
+```bash
+# Replace <NAME> with the actual worktree name (e.g., AutoGPT7)
+git worktree remove "$(dirname "$(git rev-parse --show-toplevel)")/<NAME>"
+```
+
+## Alternative: Branchlet (optional)
+
+If [branchlet](https://www.npmjs.com/package/branchlet) is installed:
+
+```bash
+branchlet create -n <name> -s <source-branch> -b <new-branch>
+```

.github/workflows/platform-backend-ci.yml

@@ -41,13 +41,18 @@ jobs:
         ports:
           - 6379:6379
       rabbitmq:
-        image: rabbitmq:3.12-management
+        image: rabbitmq:4.1.4
         ports:
           - 5672:5672
-          - 15672:15672
         env:
           RABBITMQ_DEFAULT_USER: ${{ env.RABBITMQ_DEFAULT_USER }}
           RABBITMQ_DEFAULT_PASS: ${{ env.RABBITMQ_DEFAULT_PASS }}
+        options: >-
+          --health-cmd "rabbitmq-diagnostics -q ping"
+          --health-interval 30s
+          --health-timeout 10s
+          --health-retries 5
+          --health-start-period 10s
       clamav:
         image: clamav/clamav-debian:latest
         ports:

.github/workflows/platform-frontend-ci.yml

@@ -6,10 +6,16 @@ on:
     paths:
       - ".github/workflows/platform-frontend-ci.yml"
       - "autogpt_platform/frontend/**"
+      - "autogpt_platform/backend/Dockerfile"
+      - "autogpt_platform/docker-compose.yml"
+      - "autogpt_platform/docker-compose.platform.yml"
   pull_request:
     paths:
       - ".github/workflows/platform-frontend-ci.yml"
       - "autogpt_platform/frontend/**"
+      - "autogpt_platform/backend/Dockerfile"
+      - "autogpt_platform/docker-compose.yml"
+      - "autogpt_platform/docker-compose.platform.yml"
   merge_group:
   workflow_dispatch:
 
@@ -143,7 +149,7 @@ jobs:
           driver-opts: network=host
 
       - name: Set up Platform - Expose GHA cache to docker buildx CLI
-        uses: crazy-max/ghaction-github-runtime@v3
+        uses: crazy-max/ghaction-github-runtime@v4
 
       - name: Set up Platform - Build Docker images (with cache)
         working-directory: autogpt_platform

.gitignore

@@ -180,4 +180,6 @@ autogpt_platform/backend/settings.py
 .claude/settings.local.json
 CLAUDE.local.md
 /autogpt_platform/backend/logs
-.next
+.next
+# Implementation plans (generated by AI agents)
+plans/

.nvmrc

@@ -0,0 +1 @@
+22

.pre-commit-config.yaml

@@ -1,3 +1,10 @@
+default_install_hook_types:
+  - pre-commit
+  - pre-push
+  - post-checkout
+
+default_stages: [pre-commit]
+
 repos:
   - repo: https://github.com/pre-commit/pre-commit-hooks
     rev: v4.4.0
@@ -17,6 +24,7 @@ repos:
         name: Detect secrets
         description: Detects high entropy strings that are likely to be passwords.
         files: ^autogpt_platform/
+        exclude: pnpm-lock\.yaml$
         stages: [pre-push]
 
   - repo: local
@@ -26,49 +34,106 @@ repos:
       - id: poetry-install
         name: Check & Install dependencies - AutoGPT Platform - Backend
         alias: poetry-install-platform-backend
-        entry: poetry -C autogpt_platform/backend install
         # include autogpt_libs source (since it's a path dependency)
-        files: ^autogpt_platform/(backend|autogpt_libs)/poetry\.lock$
-        types: [file]
+        entry: >
+          bash -c '
+          if [ -n "$PRE_COMMIT_FROM_REF" ]; then
+            git diff --name-only "$PRE_COMMIT_FROM_REF" "$PRE_COMMIT_TO_REF"
+          else
+            git diff --cached --name-only
+          fi | grep -qE "^autogpt_platform/(backend|autogpt_libs)/poetry\.lock$" || exit 0;
+          poetry -C autogpt_platform/backend install
+          '
+        always_run: true
         language: system
         pass_filenames: false
+        stages: [pre-commit, post-checkout]
 
       - id: poetry-install
         name: Check & Install dependencies - AutoGPT Platform - Libs
         alias: poetry-install-platform-libs
-        entry: poetry -C autogpt_platform/autogpt_libs install
-        files: ^autogpt_platform/autogpt_libs/poetry\.lock$
-        types: [file]
+        entry: >
+          bash -c '
+          if [ -n "$PRE_COMMIT_FROM_REF" ]; then
+            git diff --name-only "$PRE_COMMIT_FROM_REF" "$PRE_COMMIT_TO_REF"
+          else
+            git diff --cached --name-only
+          fi | grep -qE "^autogpt_platform/autogpt_libs/poetry\.lock$" || exit 0;
+          poetry -C autogpt_platform/autogpt_libs install
+          '
+        always_run: true
         language: system
         pass_filenames: false
+        stages: [pre-commit, post-checkout]
+
+      - id: pnpm-install
+        name: Check & Install dependencies - AutoGPT Platform - Frontend
+        alias: pnpm-install-platform-frontend
+        entry: >
+          bash -c '
+          if [ -n "$PRE_COMMIT_FROM_REF" ]; then
+            git diff --name-only "$PRE_COMMIT_FROM_REF" "$PRE_COMMIT_TO_REF"
+          else
+            git diff --cached --name-only
+          fi | grep -qE "^autogpt_platform/frontend/pnpm-lock\.yaml$" || exit 0;
+          pnpm --prefix autogpt_platform/frontend install
+          '
+        always_run: true
+        language: system
+        pass_filenames: false
+        stages: [pre-commit, post-checkout]
 
       - id: poetry-install
         name: Check & Install dependencies - Classic - AutoGPT
         alias: poetry-install-classic-autogpt
-        entry: poetry -C classic/original_autogpt install
+        entry: >
+          bash -c '
+          if [ -n "$PRE_COMMIT_FROM_REF" ]; then
+            git diff --name-only "$PRE_COMMIT_FROM_REF" "$PRE_COMMIT_TO_REF"
+          else
+            git diff --cached --name-only
+          fi | grep -qE "^classic/(original_autogpt|forge)/poetry\.lock$" || exit 0;
+          poetry -C classic/original_autogpt install
+          '
         # include forge source (since it's a path dependency)
-        files: ^classic/(original_autogpt|forge)/poetry\.lock$
-        types: [file]
+        always_run: true
         language: system
         pass_filenames: false
+        stages: [pre-commit, post-checkout]
 
       - id: poetry-install
         name: Check & Install dependencies - Classic - Forge
         alias: poetry-install-classic-forge
-        entry: poetry -C classic/forge install
-        files: ^classic/forge/poetry\.lock$
-        types: [file]
+        entry: >
+          bash -c '
+          if [ -n "$PRE_COMMIT_FROM_REF" ]; then
+            git diff --name-only "$PRE_COMMIT_FROM_REF" "$PRE_COMMIT_TO_REF"
+          else
+            git diff --cached --name-only
+          fi | grep -qE "^classic/forge/poetry\.lock$" || exit 0;
+          poetry -C classic/forge install
+          '
+        always_run: true
         language: system
         pass_filenames: false
+        stages: [pre-commit, post-checkout]
 
       - id: poetry-install
         name: Check & Install dependencies - Classic - Benchmark
         alias: poetry-install-classic-benchmark
-        entry: poetry -C classic/benchmark install
-        files: ^classic/benchmark/poetry\.lock$
-        types: [file]
+        entry: >
+          bash -c '
+          if [ -n "$PRE_COMMIT_FROM_REF" ]; then
+            git diff --name-only "$PRE_COMMIT_FROM_REF" "$PRE_COMMIT_TO_REF"
+          else
+            git diff --cached --name-only
+          fi | grep -qE "^classic/benchmark/poetry\.lock$" || exit 0;
+          poetry -C classic/benchmark install
+          '
+        always_run: true
         language: system
         pass_filenames: false
+        stages: [pre-commit, post-checkout]
 
   - repo: local
     # For proper type checking, Prisma client must be up-to-date.
@@ -76,12 +141,54 @@ repos:
       - id: prisma-generate
         name: Prisma Generate - AutoGPT Platform - Backend
         alias: prisma-generate-platform-backend
-        entry: bash -c 'cd autogpt_platform/backend && poetry run prisma generate'
+        entry: >
+          bash -c '
+          if [ -n "$PRE_COMMIT_FROM_REF" ]; then
+            git diff --name-only "$PRE_COMMIT_FROM_REF" "$PRE_COMMIT_TO_REF"
+          else
+            git diff --cached --name-only
+          fi | grep -qE "^autogpt_platform/((backend|autogpt_libs)/poetry\.lock|backend/schema\.prisma)$" || exit 0;
+          cd autogpt_platform/backend
+          && poetry run prisma generate
+          && poetry run gen-prisma-stub
+          '
         # include everything that triggers poetry install + the prisma schema
-        files: ^autogpt_platform/((backend|autogpt_libs)/poetry\.lock|backend/schema.prisma)$
-        types: [file]
+        always_run: true
         language: system
         pass_filenames: false
+        stages: [pre-commit, post-checkout]
+
+      - id: export-api-schema
+        name: Export API schema - AutoGPT Platform - Backend -> Frontend
+        alias: export-api-schema-platform
+        entry: >
+          bash -c '
+          cd autogpt_platform/backend
+          && poetry run export-api-schema --output ../frontend/src/app/api/openapi.json
+          && cd ../frontend
+          && pnpm prettier --write ./src/app/api/openapi.json
+          '
+        files: ^autogpt_platform/backend/
+        language: system
+        pass_filenames: false
+
+      - id: generate-api-client
+        name: Generate API client - AutoGPT Platform - Frontend
+        alias: generate-api-client-platform-frontend
+        entry: >
+          bash -c '
+          SCHEMA=autogpt_platform/frontend/src/app/api/openapi.json;
+          if [ -n "$PRE_COMMIT_FROM_REF" ]; then
+            git diff --quiet "$PRE_COMMIT_FROM_REF" "$PRE_COMMIT_TO_REF" -- "$SCHEMA" && exit 0
+          else
+            git diff --quiet HEAD -- "$SCHEMA" && exit 0
+          fi;
+          cd autogpt_platform/frontend && pnpm generate:api
+          '
+        always_run: true
+        language: system
+        pass_filenames: false
+        stages: [pre-commit, post-checkout]
 
   - repo: https://github.com/astral-sh/ruff-pre-commit
     rev: v0.7.2

autogpt_platform/.gitignore

@@ -1,2 +1,3 @@
 *.ignore.*
-*.ign.*
+*.ign.*
+.application.logs

autogpt_platform/CLAUDE.md

@@ -60,9 +60,12 @@ AutoGPT Platform is a monorepo containing:
 
 ### Reviewing/Revising Pull Requests
 
-- When the user runs /pr-comments or tries to fetch them, also run gh api /repos/Significant-Gravitas/AutoGPT/pulls/[issuenum]/reviews to get the reviews
-- Use gh api /repos/Significant-Gravitas/AutoGPT/pulls/[issuenum]/reviews/[review_id]/comments to get the review contents
-- Use gh api /repos/Significant-Gravitas/AutoGPT/issues/9924/comments to get the pr specific comments
+Use `/pr-review` to review a PR or `/pr-address` to address comments.
+
+When fetching comments manually:
+- `gh api repos/Significant-Gravitas/AutoGPT/pulls/{N}/reviews` — top-level reviews
+- `gh api repos/Significant-Gravitas/AutoGPT/pulls/{N}/comments` — inline review comments
+- `gh api repos/Significant-Gravitas/AutoGPT/issues/{N}/comments` — PR conversation comments
 
 ### Conventional Commits
 

autogpt_platform/analytics/queries/auth_activities.sql

@@ -0,0 +1,40 @@
+-- =============================================================
+-- View: analytics.auth_activities
+-- Looker source alias: ds49  |  Charts: 1
+-- =============================================================
+-- DESCRIPTION
+--   Tracks authentication events (login, logout, SSO, password
+--   reset, etc.) from Supabase's internal audit log.
+--   Useful for monitoring sign-in patterns and detecting anomalies.
+--
+-- SOURCE TABLES
+--   auth.audit_log_entries  — Supabase internal auth event log
+--
+-- OUTPUT COLUMNS
+--   created_at      TIMESTAMPTZ  When the auth event occurred
+--   actor_id        TEXT         User ID who triggered the event
+--   actor_via_sso   TEXT         Whether the action was via SSO ('true'/'false')
+--   action          TEXT         Event type (e.g. 'login', 'logout', 'token_refreshed')
+--
+-- WINDOW
+--   Rolling 90 days from current date
+--
+-- EXAMPLE QUERIES
+--   -- Daily login counts
+--   SELECT DATE_TRUNC('day', created_at) AS day, COUNT(*) AS logins
+--   FROM analytics.auth_activities
+--   WHERE action = 'login'
+--   GROUP BY 1 ORDER BY 1;
+--
+--   -- SSO vs password login breakdown
+--   SELECT actor_via_sso, COUNT(*) FROM analytics.auth_activities
+--   WHERE action = 'login' GROUP BY 1;
+-- =============================================================
+
+SELECT
+    created_at,
+    payload->>'actor_id'      AS actor_id,
+    payload->>'actor_via_sso' AS actor_via_sso,
+    payload->>'action'        AS action
+FROM auth.audit_log_entries
+WHERE created_at >= NOW() - INTERVAL '90 days'

autogpt_platform/analytics/queries/graph_execution.sql

@@ -0,0 +1,105 @@
+-- =============================================================
+-- View: analytics.graph_execution
+-- Looker source alias: ds16  |  Charts: 21
+-- =============================================================
+-- DESCRIPTION
+--   One row per agent graph execution (last 90 days).
+--   Unpacks the JSONB stats column into individual numeric columns
+--   and normalises the executionStatus — runs that failed due to
+--   insufficient credits are reclassified as 'NO_CREDITS' for
+--   easier filtering.  Error messages are scrubbed of IDs and URLs
+--   to allow safe grouping.
+--
+-- SOURCE TABLES
+--   platform.AgentGraphExecution  — Execution records
+--   platform.AgentGraph           — Agent graph metadata (for name)
+--   platform.LibraryAgent         — To flag possibly-AI (safe-mode) agents
+--
+-- OUTPUT COLUMNS
+--   id                TEXT         Execution UUID
+--   agentGraphId      TEXT         Agent graph UUID
+--   agentGraphVersion INT          Graph version number
+--   executionStatus   TEXT         COMPLETED | FAILED | NO_CREDITS | RUNNING | QUEUED | TERMINATED
+--   createdAt         TIMESTAMPTZ  When the execution was queued
+--   updatedAt         TIMESTAMPTZ  Last status update time
+--   userId            TEXT         Owner user UUID
+--   agentGraphName    TEXT         Human-readable agent name
+--   cputime           DECIMAL      Total CPU seconds consumed
+--   walltime          DECIMAL      Total wall-clock seconds
+--   node_count        DECIMAL      Number of nodes in the graph
+--   nodes_cputime     DECIMAL      CPU time across all nodes
+--   nodes_walltime    DECIMAL      Wall time across all nodes
+--   execution_cost    DECIMAL      Credit cost of this execution
+--   correctness_score FLOAT        AI correctness score (if available)
+--   possibly_ai       BOOLEAN      True if agent has sensitive_action_safe_mode enabled
+--   groupedErrorMessage TEXT       Scrubbed error string (IDs/URLs replaced with wildcards)
+--
+-- WINDOW
+--   Rolling 90 days (createdAt > CURRENT_DATE - 90 days)
+--
+-- EXAMPLE QUERIES
+--   -- Daily execution counts by status
+--   SELECT DATE_TRUNC('day', "createdAt") AS day, "executionStatus", COUNT(*)
+--   FROM analytics.graph_execution
+--   GROUP BY 1, 2 ORDER BY 1;
+--
+--   -- Average cost per execution by agent
+--   SELECT "agentGraphName", AVG("execution_cost") AS avg_cost, COUNT(*) AS runs
+--   FROM analytics.graph_execution
+--   WHERE "executionStatus" = 'COMPLETED'
+--   GROUP BY 1 ORDER BY avg_cost DESC;
+--
+--   -- Top error messages
+--   SELECT "groupedErrorMessage", COUNT(*) AS occurrences
+--   FROM analytics.graph_execution
+--   WHERE "executionStatus" = 'FAILED'
+--   GROUP BY 1 ORDER BY 2 DESC LIMIT 20;
+-- =============================================================
+
+SELECT
+    ge."id"                                                        AS id,
+    ge."agentGraphId"                                              AS agentGraphId,
+    ge."agentGraphVersion"                                         AS agentGraphVersion,
+    CASE
+        WHEN jsonb_exists(ge."stats"::jsonb, 'error')
+         AND (
+               (ge."stats"::jsonb->>'error') ILIKE '%insufficient balance%'
+            OR (ge."stats"::jsonb->>'error') ILIKE '%you have no credits left%'
+             )
+        THEN 'NO_CREDITS'
+        ELSE CAST(ge."executionStatus" AS TEXT)
+    END                                                            AS executionStatus,
+    ge."createdAt"                                                 AS createdAt,
+    ge."updatedAt"                                                 AS updatedAt,
+    ge."userId"                                                    AS userId,
+    g."name"                                                       AS agentGraphName,
+    (ge."stats"::jsonb->>'cputime')::decimal                       AS cputime,
+    (ge."stats"::jsonb->>'walltime')::decimal                      AS walltime,
+    (ge."stats"::jsonb->>'node_count')::decimal                    AS node_count,
+    (ge."stats"::jsonb->>'nodes_cputime')::decimal                 AS nodes_cputime,
+    (ge."stats"::jsonb->>'nodes_walltime')::decimal                AS nodes_walltime,
+    (ge."stats"::jsonb->>'cost')::decimal                          AS execution_cost,
+    (ge."stats"::jsonb->>'correctness_score')::float               AS correctness_score,
+    COALESCE(la.possibly_ai, FALSE)                                AS possibly_ai,
+    REGEXP_REPLACE(
+        REGEXP_REPLACE(
+            TRIM(BOTH '"' FROM ge."stats"::jsonb->>'error'),
+            '(https?://)([A-Za-z0-9.-]+)(:[0-9]+)?(/[^\s]*)?',
+            '\1\2/...', 'gi'
+        ),
+        '[a-zA-Z0-9_:-]*\d[a-zA-Z0-9_:-]*', '*', 'g'
+    )                                                              AS groupedErrorMessage
+FROM platform."AgentGraphExecution" ge
+LEFT JOIN platform."AgentGraph" g
+       ON ge."agentGraphId" = g."id"
+      AND ge."agentGraphVersion" = g."version"
+LEFT JOIN (
+    SELECT DISTINCT ON ("userId", "agentGraphId")
+           "userId", "agentGraphId",
+           ("settings"::jsonb->>'sensitive_action_safe_mode')::boolean AS possibly_ai
+    FROM platform."LibraryAgent"
+    WHERE "isDeleted"  = FALSE
+      AND "isArchived" = FALSE
+    ORDER BY "userId", "agentGraphId", "agentGraphVersion" DESC
+) la ON la."userId" = ge."userId" AND la."agentGraphId" = ge."agentGraphId"
+WHERE ge."createdAt" > CURRENT_DATE - INTERVAL '90 days'

autogpt_platform/analytics/queries/node_block_execution.sql

@@ -0,0 +1,101 @@
+-- =============================================================
+-- View: analytics.node_block_execution
+-- Looker source alias: ds14  |  Charts: 11
+-- =============================================================
+-- DESCRIPTION
+--   One row per node (block) execution (last 90 days).
+--   Unpacks stats JSONB and joins to identify which block type
+--   was run.  For failed nodes, joins the error output and
+--   scrubs it for safe grouping.
+--
+-- SOURCE TABLES
+--   platform.AgentNodeExecution              — Node execution records
+--   platform.AgentNode                       — Node → block mapping
+--   platform.AgentBlock                      — Block name/ID
+--   platform.AgentNodeExecutionInputOutput   — Error output values
+--
+-- OUTPUT COLUMNS
+--   id                    TEXT         Node execution UUID
+--   agentGraphExecutionId TEXT         Parent graph execution UUID
+--   agentNodeId           TEXT         Node UUID within the graph
+--   executionStatus       TEXT         COMPLETED | FAILED | QUEUED | RUNNING | TERMINATED
+--   addedTime             TIMESTAMPTZ  When the node was queued
+--   queuedTime            TIMESTAMPTZ  When it entered the queue
+--   startedTime           TIMESTAMPTZ  When execution started
+--   endedTime             TIMESTAMPTZ  When execution finished
+--   inputSize             BIGINT       Input payload size in bytes
+--   outputSize            BIGINT       Output payload size in bytes
+--   walltime              NUMERIC      Wall-clock seconds for this node
+--   cputime               NUMERIC      CPU seconds for this node
+--   llmRetryCount         INT          Number of LLM retries
+--   llmCallCount          INT          Number of LLM API calls made
+--   inputTokenCount       BIGINT       LLM input tokens consumed
+--   outputTokenCount      BIGINT       LLM output tokens produced
+--   blockName             TEXT         Human-readable block name (e.g. 'OpenAIBlock')
+--   blockId               TEXT         Block UUID
+--   groupedErrorMessage   TEXT         Scrubbed error (IDs/URLs wildcarded)
+--   errorMessage          TEXT         Raw error output (only set when FAILED)
+--
+-- WINDOW
+--   Rolling 90 days (addedTime > CURRENT_DATE - 90 days)
+--
+-- EXAMPLE QUERIES
+--   -- Most-used blocks by execution count
+--   SELECT "blockName", COUNT(*) AS executions,
+--          COUNT(*) FILTER (WHERE "executionStatus"='FAILED') AS failures
+--   FROM analytics.node_block_execution
+--   GROUP BY 1 ORDER BY executions DESC LIMIT 20;
+--
+--   -- Average LLM token usage per block
+--   SELECT "blockName",
+--          AVG("inputTokenCount") AS avg_input_tokens,
+--          AVG("outputTokenCount") AS avg_output_tokens
+--   FROM analytics.node_block_execution
+--   WHERE "llmCallCount" > 0
+--   GROUP BY 1 ORDER BY avg_input_tokens DESC;
+--
+--   -- Top failure reasons
+--   SELECT "blockName", "groupedErrorMessage", COUNT(*) AS count
+--   FROM analytics.node_block_execution
+--   WHERE "executionStatus" = 'FAILED'
+--   GROUP BY 1, 2 ORDER BY count DESC LIMIT 20;
+-- =============================================================
+
+SELECT
+    ne."id"                                                            AS id,
+    ne."agentGraphExecutionId"                                         AS agentGraphExecutionId,
+    ne."agentNodeId"                                                   AS agentNodeId,
+    CAST(ne."executionStatus" AS TEXT)                                 AS executionStatus,
+    ne."addedTime"                                                     AS addedTime,
+    ne."queuedTime"                                                    AS queuedTime,
+    ne."startedTime"                                                   AS startedTime,
+    ne."endedTime"                                                     AS endedTime,
+    (ne."stats"::jsonb->>'input_size')::bigint                         AS inputSize,
+    (ne."stats"::jsonb->>'output_size')::bigint                        AS outputSize,
+    (ne."stats"::jsonb->>'walltime')::numeric                          AS walltime,
+    (ne."stats"::jsonb->>'cputime')::numeric                           AS cputime,
+    (ne."stats"::jsonb->>'llm_retry_count')::int                       AS llmRetryCount,
+    (ne."stats"::jsonb->>'llm_call_count')::int                        AS llmCallCount,
+    (ne."stats"::jsonb->>'input_token_count')::bigint                  AS inputTokenCount,
+    (ne."stats"::jsonb->>'output_token_count')::bigint                 AS outputTokenCount,
+    b."name"                                                           AS blockName,
+    b."id"                                                             AS blockId,
+    REGEXP_REPLACE(
+        REGEXP_REPLACE(
+            TRIM(BOTH '"' FROM eio."data"::text),
+            '(https?://)([A-Za-z0-9.-]+)(:[0-9]+)?(/[^\s]*)?',
+            '\1\2/...', 'gi'
+        ),
+        '[a-zA-Z0-9_:-]*\d[a-zA-Z0-9_:-]*', '*', 'g'
+    )                                                                  AS groupedErrorMessage,
+    eio."data"                                                         AS errorMessage
+FROM platform."AgentNodeExecution" ne
+LEFT JOIN platform."AgentNode" nd
+       ON ne."agentNodeId" = nd."id"
+LEFT JOIN platform."AgentBlock" b
+       ON nd."agentBlockId" = b."id"
+LEFT JOIN platform."AgentNodeExecutionInputOutput" eio
+       ON eio."referencedByOutputExecId" = ne."id"
+      AND eio."name" = 'error'
+      AND ne."executionStatus" = 'FAILED'
+WHERE ne."addedTime" > CURRENT_DATE - INTERVAL '90 days'

autogpt_platform/analytics/queries/retention_agent.sql

@@ -0,0 +1,97 @@
+-- =============================================================
+-- View: analytics.retention_agent
+-- Looker source alias: ds35  |  Charts: 2
+-- =============================================================
+-- DESCRIPTION
+--   Weekly cohort retention broken down per individual agent.
+--   Cohort = week of a user's first use of THAT specific agent.
+--   Tells you which agents keep users coming back vs. one-shot
+--   use. Only includes cohorts from the last 180 days.
+--
+-- SOURCE TABLES
+--   platform.AgentGraphExecution  — Execution records (user × agent × time)
+--   platform.AgentGraph           — Agent names
+--
+-- OUTPUT COLUMNS
+--   agent_id            TEXT   Agent graph UUID
+--   agent_label         TEXT   'AgentName [first8chars]'
+--   agent_label_n       TEXT   'AgentName [first8chars] (n=total_users)'
+--   cohort_week_start   DATE   Week users first ran this agent
+--   cohort_label        TEXT   ISO week label
+--   cohort_label_n      TEXT   ISO week label with cohort size
+--   user_lifetime_week  INT    Weeks since first use of this agent
+--   cohort_users        BIGINT Users in this cohort for this agent
+--   active_users        BIGINT Users who ran the agent again in week k
+--   retention_rate      FLOAT  active_users / cohort_users
+--   cohort_users_w0     BIGINT cohort_users only at week 0 (safe to SUM)
+--   agent_total_users   BIGINT Total users across all cohorts for this agent
+--
+-- EXAMPLE QUERIES
+--   -- Best-retained agents at week 2
+--   SELECT agent_label, AVG(retention_rate) AS w2_retention
+--   FROM analytics.retention_agent
+--   WHERE user_lifetime_week = 2 AND cohort_users >= 10
+--   GROUP BY 1 ORDER BY w2_retention DESC LIMIT 10;
+--
+--   -- Agents with most unique users
+--   SELECT DISTINCT agent_label, agent_total_users
+--   FROM analytics.retention_agent
+--   ORDER BY agent_total_users DESC LIMIT 20;
+-- =============================================================
+
+WITH params AS (SELECT 12::int AS max_weeks, (CURRENT_DATE - INTERVAL '180 days') AS cohort_start),
+events AS (
+  SELECT e."userId"::text AS user_id, e."agentGraphId" AS agent_id,
+         e."createdAt"::timestamptz AS created_at,
+         DATE_TRUNC('week', e."createdAt")::date AS week_start
+  FROM platform."AgentGraphExecution" e
+),
+first_use AS (
+  SELECT user_id, agent_id, MIN(created_at) AS first_use_at,
+         DATE_TRUNC('week', MIN(created_at))::date AS cohort_week_start
+  FROM events GROUP BY 1,2
+  HAVING MIN(created_at) >= (SELECT cohort_start FROM params)
+),
+activity_weeks AS (SELECT DISTINCT user_id, agent_id, week_start FROM events),
+user_week_age AS (
+  SELECT aw.user_id, aw.agent_id, fu.cohort_week_start,
+         ((aw.week_start - DATE_TRUNC('week',fu.first_use_at)::date)/7)::int AS user_lifetime_week
+  FROM activity_weeks aw JOIN first_use fu USING (user_id, agent_id)
+  WHERE aw.week_start >= DATE_TRUNC('week',fu.first_use_at)::date
+),
+active_counts AS (
+  SELECT agent_id, cohort_week_start, user_lifetime_week, COUNT(DISTINCT user_id) AS active_users
+  FROM user_week_age WHERE user_lifetime_week >= 0 GROUP BY 1,2,3
+),
+cohort_sizes AS (
+  SELECT agent_id, cohort_week_start, COUNT(DISTINCT user_id) AS cohort_users FROM first_use GROUP BY 1,2
+),
+cohort_caps AS (
+  SELECT cs.agent_id, cs.cohort_week_start, cs.cohort_users,
+         LEAST((SELECT max_weeks FROM params),
+               GREATEST(0,((DATE_TRUNC('week',CURRENT_DATE)::date-cs.cohort_week_start)/7)::int)) AS cap_weeks
+  FROM cohort_sizes cs
+),
+grid AS (
+  SELECT cc.agent_id, cc.cohort_week_start, gs AS user_lifetime_week, cc.cohort_users
+  FROM cohort_caps cc CROSS JOIN LATERAL generate_series(0, cc.cap_weeks) gs
+),
+agent_names AS (SELECT DISTINCT ON (g."id") g."id" AS agent_id, g."name" AS agent_name FROM platform."AgentGraph" g ORDER BY g."id", g."version" DESC),
+agent_total_users AS (SELECT agent_id, SUM(cohort_users) AS agent_total_users FROM cohort_sizes GROUP BY 1)
+SELECT
+  g.agent_id,
+  COALESCE(an.agent_name,'(unnamed)')||' ['||LEFT(g.agent_id::text,8)||']'  AS agent_label,
+  COALESCE(an.agent_name,'(unnamed)')||' ['||LEFT(g.agent_id::text,8)||'] (n='||COALESCE(atu.agent_total_users,0)||')' AS agent_label_n,
+  g.cohort_week_start,
+  TO_CHAR(g.cohort_week_start,'IYYY-"W"IW')                               AS cohort_label,
+  TO_CHAR(g.cohort_week_start,'IYYY-"W"IW')||' (n='||g.cohort_users||')'  AS cohort_label_n,
+  g.user_lifetime_week, g.cohort_users,
+  COALESCE(ac.active_users,0)                                              AS active_users,
+  COALESCE(ac.active_users,0)::float / NULLIF(g.cohort_users,0)           AS retention_rate,
+  CASE WHEN g.user_lifetime_week=0 THEN g.cohort_users ELSE 0 END         AS cohort_users_w0,
+  COALESCE(atu.agent_total_users,0)                                        AS agent_total_users
+FROM grid g
+LEFT JOIN active_counts     ac  ON ac.agent_id=g.agent_id AND ac.cohort_week_start=g.cohort_week_start AND ac.user_lifetime_week=g.user_lifetime_week
+LEFT JOIN agent_names       an  ON an.agent_id=g.agent_id
+LEFT JOIN agent_total_users atu ON atu.agent_id=g.agent_id
+ORDER BY agent_label, g.cohort_week_start, g.user_lifetime_week;

autogpt_platform/analytics/queries/retention_execution_daily.sql

@@ -0,0 +1,81 @@
+-- =============================================================
+-- View: analytics.retention_execution_daily
+-- Looker source alias: ds111  |  Charts: 1
+-- =============================================================
+-- DESCRIPTION
+--   Daily cohort retention based on agent executions.
+--   Cohort anchor = day of user's FIRST ever execution.
+--   Only includes cohorts from the last 90 days, up to day 30.
+--   Great for early engagement analysis (did users run another
+--   agent the next day?).
+--
+-- SOURCE TABLES
+--   platform.AgentGraphExecution  — Execution records
+--
+-- OUTPUT COLUMNS
+--   Same pattern as retention_login_daily.
+--   cohort_day_start = day of first execution (not first login)
+--
+-- EXAMPLE QUERIES
+--   -- Day-3 execution retention
+--   SELECT cohort_label, retention_rate_bounded AS d3_retention
+--   FROM analytics.retention_execution_daily
+--   WHERE user_lifetime_day = 3 ORDER BY cohort_day_start;
+-- =============================================================
+
+WITH params AS (SELECT 30::int AS max_days, (CURRENT_DATE - INTERVAL '90 days') AS cohort_start),
+events AS (
+  SELECT e."userId"::text AS user_id, e."createdAt"::timestamptz AS created_at,
+         DATE_TRUNC('day', e."createdAt")::date AS day_start
+  FROM platform."AgentGraphExecution" e WHERE e."userId" IS NOT NULL
+),
+first_exec AS (
+  SELECT user_id, MIN(created_at) AS first_exec_at,
+         DATE_TRUNC('day', MIN(created_at))::date AS cohort_day_start
+  FROM events GROUP BY 1
+  HAVING MIN(created_at) >= (SELECT cohort_start FROM params)
+),
+activity_days AS (SELECT DISTINCT user_id, day_start FROM events),
+user_day_age AS (
+  SELECT ad.user_id, fe.cohort_day_start,
+         (ad.day_start - DATE_TRUNC('day',fe.first_exec_at)::date)::int AS user_lifetime_day
+  FROM activity_days ad JOIN first_exec fe USING (user_id)
+  WHERE ad.day_start >= DATE_TRUNC('day',fe.first_exec_at)::date
+),
+bounded_counts AS (
+  SELECT cohort_day_start, user_lifetime_day, COUNT(DISTINCT user_id) AS active_users_bounded
+  FROM user_day_age WHERE user_lifetime_day >= 0 GROUP BY 1,2
+),
+last_active AS (
+  SELECT cohort_day_start, user_id, MAX(user_lifetime_day) AS last_active_day FROM user_day_age GROUP BY 1,2
+),
+unbounded_counts AS (
+  SELECT la.cohort_day_start, gs AS user_lifetime_day, COUNT(*) AS retained_users_unbounded
+  FROM last_active la
+  CROSS JOIN LATERAL generate_series(0, LEAST(la.last_active_day,(SELECT max_days FROM params))) gs
+  GROUP BY 1,2
+),
+cohort_sizes AS (SELECT cohort_day_start, COUNT(DISTINCT user_id) AS cohort_users FROM first_exec GROUP BY 1),
+cohort_caps AS (
+  SELECT cs.cohort_day_start, cs.cohort_users,
+         LEAST((SELECT max_days FROM params), GREATEST(0,(CURRENT_DATE-cs.cohort_day_start)::int)) AS cap_days
+  FROM cohort_sizes cs
+),
+grid AS (
+  SELECT cc.cohort_day_start, gs AS user_lifetime_day, cc.cohort_users
+  FROM cohort_caps cc CROSS JOIN LATERAL generate_series(0, cc.cap_days) gs
+)
+SELECT
+  g.cohort_day_start,
+  TO_CHAR(g.cohort_day_start,'YYYY-MM-DD')                                AS cohort_label,
+  TO_CHAR(g.cohort_day_start,'YYYY-MM-DD')||' (n='||g.cohort_users||')'   AS cohort_label_n,
+  g.user_lifetime_day, g.cohort_users,
+  COALESCE(b.active_users_bounded,0)     AS active_users_bounded,
+  COALESCE(u.retained_users_unbounded,0) AS retained_users_unbounded,
+  CASE WHEN g.cohort_users>0 THEN COALESCE(b.active_users_bounded,0)::float/g.cohort_users END    AS retention_rate_bounded,
+  CASE WHEN g.cohort_users>0 THEN COALESCE(u.retained_users_unbounded,0)::float/g.cohort_users END AS retention_rate_unbounded,
+  CASE WHEN g.user_lifetime_day=0 THEN g.cohort_users ELSE 0 END          AS cohort_users_d0
+FROM grid g
+LEFT JOIN bounded_counts   b ON b.cohort_day_start=g.cohort_day_start AND b.user_lifetime_day=g.user_lifetime_day
+LEFT JOIN unbounded_counts u ON u.cohort_day_start=g.cohort_day_start AND u.user_lifetime_day=g.user_lifetime_day
+ORDER BY g.cohort_day_start, g.user_lifetime_day;

autogpt_platform/analytics/queries/retention_execution_weekly.sql

@@ -0,0 +1,81 @@
+-- =============================================================
+-- View: analytics.retention_execution_weekly
+-- Looker source alias: ds92  |  Charts: 2
+-- =============================================================
+-- DESCRIPTION
+--   Weekly cohort retention based on agent executions.
+--   Cohort anchor = week of user's FIRST ever agent execution
+--   (not first login). Only includes cohorts from the last 180 days.
+--   Useful when you care about product engagement, not just visits.
+--
+-- SOURCE TABLES
+--   platform.AgentGraphExecution  — Execution records
+--
+-- OUTPUT COLUMNS
+--   Same pattern as retention_login_weekly.
+--   cohort_week_start = week of first execution (not first login)
+--
+-- EXAMPLE QUERIES
+--   -- Week-2 execution retention
+--   SELECT cohort_label, retention_rate_bounded
+--   FROM analytics.retention_execution_weekly
+--   WHERE user_lifetime_week = 2 ORDER BY cohort_week_start;
+-- =============================================================
+
+WITH params AS (SELECT 12::int AS max_weeks, (CURRENT_DATE - INTERVAL '180 days') AS cohort_start),
+events AS (
+  SELECT e."userId"::text AS user_id, e."createdAt"::timestamptz AS created_at,
+         DATE_TRUNC('week', e."createdAt")::date AS week_start
+  FROM platform."AgentGraphExecution" e WHERE e."userId" IS NOT NULL
+),
+first_exec AS (
+  SELECT user_id, MIN(created_at) AS first_exec_at,
+         DATE_TRUNC('week', MIN(created_at))::date AS cohort_week_start
+  FROM events GROUP BY 1
+  HAVING MIN(created_at) >= (SELECT cohort_start FROM params)
+),
+activity_weeks AS (SELECT DISTINCT user_id, week_start FROM events),
+user_week_age AS (
+  SELECT aw.user_id, fe.cohort_week_start,
+         ((aw.week_start - DATE_TRUNC('week',fe.first_exec_at)::date)/7)::int AS user_lifetime_week
+  FROM activity_weeks aw JOIN first_exec fe USING (user_id)
+  WHERE aw.week_start >= DATE_TRUNC('week',fe.first_exec_at)::date
+),
+bounded_counts AS (
+  SELECT cohort_week_start, user_lifetime_week, COUNT(DISTINCT user_id) AS active_users_bounded
+  FROM user_week_age WHERE user_lifetime_week >= 0 GROUP BY 1,2
+),
+last_active AS (
+  SELECT cohort_week_start, user_id, MAX(user_lifetime_week) AS last_active_week FROM user_week_age GROUP BY 1,2
+),
+unbounded_counts AS (
+  SELECT la.cohort_week_start, gs AS user_lifetime_week, COUNT(*) AS retained_users_unbounded
+  FROM last_active la
+  CROSS JOIN LATERAL generate_series(0, LEAST(la.last_active_week,(SELECT max_weeks FROM params))) gs
+  GROUP BY 1,2
+),
+cohort_sizes AS (SELECT cohort_week_start, COUNT(DISTINCT user_id) AS cohort_users FROM first_exec GROUP BY 1),
+cohort_caps AS (
+  SELECT cs.cohort_week_start, cs.cohort_users,
+         LEAST((SELECT max_weeks FROM params),
+               GREATEST(0,((DATE_TRUNC('week',CURRENT_DATE)::date-cs.cohort_week_start)/7)::int)) AS cap_weeks
+  FROM cohort_sizes cs
+),
+grid AS (
+  SELECT cc.cohort_week_start, gs AS user_lifetime_week, cc.cohort_users
+  FROM cohort_caps cc CROSS JOIN LATERAL generate_series(0, cc.cap_weeks) gs
+)
+SELECT
+  g.cohort_week_start,
+  TO_CHAR(g.cohort_week_start,'IYYY-"W"IW')                               AS cohort_label,
+  TO_CHAR(g.cohort_week_start,'IYYY-"W"IW')||' (n='||g.cohort_users||')'  AS cohort_label_n,
+  g.user_lifetime_week, g.cohort_users,
+  COALESCE(b.active_users_bounded,0)     AS active_users_bounded,
+  COALESCE(u.retained_users_unbounded,0) AS retained_users_unbounded,
+  CASE WHEN g.cohort_users>0 THEN COALESCE(b.active_users_bounded,0)::float/g.cohort_users END    AS retention_rate_bounded,
+  CASE WHEN g.cohort_users>0 THEN COALESCE(u.retained_users_unbounded,0)::float/g.cohort_users END AS retention_rate_unbounded,
+  CASE WHEN g.user_lifetime_week=0 THEN g.cohort_users ELSE 0 END         AS cohort_users_w0
+FROM grid g
+LEFT JOIN bounded_counts   b ON b.cohort_week_start=g.cohort_week_start AND b.user_lifetime_week=g.user_lifetime_week
+LEFT JOIN unbounded_counts u ON u.cohort_week_start=g.cohort_week_start AND u.user_lifetime_week=g.user_lifetime_week
+ORDER BY g.cohort_week_start, g.user_lifetime_week;

autogpt_platform/analytics/queries/retention_login_daily.sql

@@ -0,0 +1,94 @@
+-- =============================================================
+-- View: analytics.retention_login_daily
+-- Looker source alias: ds112  |  Charts: 1
+-- =============================================================
+-- DESCRIPTION
+--   Daily cohort retention based on login sessions.
+--   Same logic as retention_login_weekly but at day granularity,
+--   showing up to day 30 for cohorts from the last 90 days.
+--   Useful for analysing early activation (days 1-7) in detail.
+--
+-- SOURCE TABLES
+--   auth.sessions  — Login session records
+--
+-- OUTPUT COLUMNS (same pattern as retention_login_weekly)
+--   cohort_day_start          DATE     First day the cohort logged in
+--   cohort_label              TEXT     Date string (e.g. '2025-03-01')
+--   cohort_label_n            TEXT     Date + cohort size (e.g. '2025-03-01 (n=12)')
+--   user_lifetime_day         INT      Days since first login (0 = signup day)
+--   cohort_users              BIGINT   Total users in cohort
+--   active_users_bounded      BIGINT   Users active on exactly day k
+--   retained_users_unbounded  BIGINT   Users active any time on/after day k
+--   retention_rate_bounded    FLOAT    bounded / cohort_users
+--   retention_rate_unbounded  FLOAT    unbounded / cohort_users
+--   cohort_users_d0           BIGINT   cohort_users only at day 0, else 0 (safe to SUM)
+--
+-- EXAMPLE QUERIES
+--   -- Day-1 retention rate (came back next day)
+--   SELECT cohort_label, retention_rate_bounded AS d1_retention
+--   FROM analytics.retention_login_daily
+--   WHERE user_lifetime_day = 1 ORDER BY cohort_day_start;
+--
+--   -- Average retention curve across all cohorts
+--   SELECT user_lifetime_day,
+--          SUM(active_users_bounded)::float / NULLIF(SUM(cohort_users_d0), 0) AS avg_retention
+--   FROM analytics.retention_login_daily
+--   GROUP BY 1 ORDER BY 1;
+-- =============================================================
+
+WITH params AS (SELECT 30::int AS max_days, (CURRENT_DATE - INTERVAL '90 days')::date AS cohort_start),
+events AS (
+  SELECT s.user_id::text AS user_id, s.created_at::timestamptz AS created_at,
+         DATE_TRUNC('day', s.created_at)::date AS day_start
+  FROM auth.sessions s WHERE s.user_id IS NOT NULL
+),
+first_login AS (
+  SELECT user_id, MIN(created_at) AS first_login_time,
+         DATE_TRUNC('day', MIN(created_at))::date AS cohort_day_start
+  FROM events GROUP BY 1
+  HAVING MIN(created_at) >= (SELECT cohort_start FROM params)
+),
+activity_days AS (SELECT DISTINCT user_id, day_start FROM events),
+user_day_age AS (
+  SELECT ad.user_id, fl.cohort_day_start,
+         (ad.day_start - DATE_TRUNC('day', fl.first_login_time)::date)::int AS user_lifetime_day
+  FROM activity_days ad JOIN first_login fl USING (user_id)
+  WHERE ad.day_start >= DATE_TRUNC('day', fl.first_login_time)::date
+),
+bounded_counts AS (
+  SELECT cohort_day_start, user_lifetime_day, COUNT(DISTINCT user_id) AS active_users_bounded
+  FROM user_day_age WHERE user_lifetime_day >= 0 GROUP BY 1,2
+),
+last_active AS (
+  SELECT cohort_day_start, user_id, MAX(user_lifetime_day) AS last_active_day FROM user_day_age GROUP BY 1,2
+),
+unbounded_counts AS (
+  SELECT la.cohort_day_start, gs AS user_lifetime_day, COUNT(*) AS retained_users_unbounded
+  FROM last_active la
+  CROSS JOIN LATERAL generate_series(0, LEAST(la.last_active_day,(SELECT max_days FROM params))) gs
+  GROUP BY 1,2
+),
+cohort_sizes AS (SELECT cohort_day_start, COUNT(DISTINCT user_id) AS cohort_users FROM first_login GROUP BY 1),
+cohort_caps AS (
+  SELECT cs.cohort_day_start, cs.cohort_users,
+         LEAST((SELECT max_days FROM params), GREATEST(0,(CURRENT_DATE-cs.cohort_day_start)::int)) AS cap_days
+  FROM cohort_sizes cs
+),
+grid AS (
+  SELECT cc.cohort_day_start, gs AS user_lifetime_day, cc.cohort_users
+  FROM cohort_caps cc CROSS JOIN LATERAL generate_series(0, cc.cap_days) gs
+)
+SELECT
+  g.cohort_day_start,
+  TO_CHAR(g.cohort_day_start,'YYYY-MM-DD')                                  AS cohort_label,
+  TO_CHAR(g.cohort_day_start,'YYYY-MM-DD')||' (n='||g.cohort_users||')'     AS cohort_label_n,
+  g.user_lifetime_day, g.cohort_users,
+  COALESCE(b.active_users_bounded,0)     AS active_users_bounded,
+  COALESCE(u.retained_users_unbounded,0) AS retained_users_unbounded,
+  CASE WHEN g.cohort_users>0 THEN COALESCE(b.active_users_bounded,0)::float/g.cohort_users END    AS retention_rate_bounded,
+  CASE WHEN g.cohort_users>0 THEN COALESCE(u.retained_users_unbounded,0)::float/g.cohort_users END AS retention_rate_unbounded,
+  CASE WHEN g.user_lifetime_day=0 THEN g.cohort_users ELSE 0 END            AS cohort_users_d0
+FROM grid g
+LEFT JOIN bounded_counts   b ON b.cohort_day_start=g.cohort_day_start AND b.user_lifetime_day=g.user_lifetime_day
+LEFT JOIN unbounded_counts u ON u.cohort_day_start=g.cohort_day_start AND u.user_lifetime_day=g.user_lifetime_day
+ORDER BY g.cohort_day_start, g.user_lifetime_day;

autogpt_platform/analytics/queries/retention_login_onboarded_weekly.sql

@@ -0,0 +1,96 @@
+-- =============================================================
+-- View: analytics.retention_login_onboarded_weekly
+-- Looker source alias: ds101  |  Charts: 2
+-- =============================================================
+-- DESCRIPTION
+--   Weekly cohort retention from login sessions, restricted to
+--   users who "onboarded" — defined as running at least one
+--   agent within 365 days of their first login.
+--   Filters out users who signed up but never activated,
+--   giving a cleaner view of engaged-user retention.
+--
+-- SOURCE TABLES
+--   auth.sessions                  — Login session records
+--   platform.AgentGraphExecution   — Used to identify onboarders
+--
+-- OUTPUT COLUMNS
+--   Same as retention_login_weekly (cohort_week_start, user_lifetime_week,
+--   retention_rate_bounded, retention_rate_unbounded, etc.)
+--   Only difference: cohort is filtered to onboarded users only.
+--
+-- EXAMPLE QUERIES
+--   -- Compare week-4 retention: all users vs onboarded only
+--   SELECT 'all_users' AS segment, AVG(retention_rate_bounded) AS w4_retention
+--   FROM analytics.retention_login_weekly WHERE user_lifetime_week = 4
+--   UNION ALL
+--   SELECT 'onboarded', AVG(retention_rate_bounded)
+--   FROM analytics.retention_login_onboarded_weekly WHERE user_lifetime_week = 4;
+-- =============================================================
+
+WITH params AS (SELECT 12::int AS max_weeks, 365::int AS onboarding_window_days),
+events AS (
+  SELECT s.user_id::text AS user_id, s.created_at::timestamptz AS created_at,
+         DATE_TRUNC('week', s.created_at)::date AS week_start
+  FROM auth.sessions s WHERE s.user_id IS NOT NULL
+),
+first_login_all AS (
+  SELECT user_id, MIN(created_at) AS first_login_time,
+         DATE_TRUNC('week', MIN(created_at))::date AS cohort_week_start
+  FROM events GROUP BY 1
+),
+onboarders AS (
+  SELECT fl.user_id FROM first_login_all fl
+  WHERE EXISTS (
+    SELECT 1 FROM platform."AgentGraphExecution" e
+    WHERE e."userId"::text = fl.user_id
+      AND e."createdAt" >= fl.first_login_time
+      AND e."createdAt" < fl.first_login_time
+          + make_interval(days => (SELECT onboarding_window_days FROM params))
+  )
+),
+first_login AS (SELECT * FROM first_login_all WHERE user_id IN (SELECT user_id FROM onboarders)),
+activity_weeks AS (SELECT DISTINCT user_id, week_start FROM events),
+user_week_age AS (
+  SELECT aw.user_id, fl.cohort_week_start,
+         ((aw.week_start - DATE_TRUNC('week',fl.first_login_time)::date)/7)::int AS user_lifetime_week
+  FROM activity_weeks aw JOIN first_login fl USING (user_id)
+  WHERE aw.week_start >= DATE_TRUNC('week',fl.first_login_time)::date
+),
+bounded_counts AS (
+  SELECT cohort_week_start, user_lifetime_week, COUNT(DISTINCT user_id) AS active_users_bounded
+  FROM user_week_age WHERE user_lifetime_week >= 0 GROUP BY 1,2
+),
+last_active AS (
+  SELECT cohort_week_start, user_id, MAX(user_lifetime_week) AS last_active_week FROM user_week_age GROUP BY 1,2
+),
+unbounded_counts AS (
+  SELECT la.cohort_week_start, gs AS user_lifetime_week, COUNT(*) AS retained_users_unbounded
+  FROM last_active la
+  CROSS JOIN LATERAL generate_series(0, LEAST(la.last_active_week,(SELECT max_weeks FROM params))) gs
+  GROUP BY 1,2
+),
+cohort_sizes AS (SELECT cohort_week_start, COUNT(DISTINCT user_id) AS cohort_users FROM first_login GROUP BY 1),
+cohort_caps AS (
+  SELECT cs.cohort_week_start, cs.cohort_users,
+         LEAST((SELECT max_weeks FROM params),
+               GREATEST(0,((DATE_TRUNC('week',CURRENT_DATE)::date-cs.cohort_week_start)/7)::int)) AS cap_weeks
+  FROM cohort_sizes cs
+),
+grid AS (
+  SELECT cc.cohort_week_start, gs AS user_lifetime_week, cc.cohort_users
+  FROM cohort_caps cc CROSS JOIN LATERAL generate_series(0, cc.cap_weeks) gs
+)
+SELECT
+  g.cohort_week_start,
+  TO_CHAR(g.cohort_week_start,'IYYY-"W"IW')                               AS cohort_label,
+  TO_CHAR(g.cohort_week_start,'IYYY-"W"IW')||' (n='||g.cohort_users||')'  AS cohort_label_n,
+  g.user_lifetime_week, g.cohort_users,
+  COALESCE(b.active_users_bounded,0)     AS active_users_bounded,
+  COALESCE(u.retained_users_unbounded,0) AS retained_users_unbounded,
+  CASE WHEN g.cohort_users>0 THEN COALESCE(b.active_users_bounded,0)::float/g.cohort_users END    AS retention_rate_bounded,
+  CASE WHEN g.cohort_users>0 THEN COALESCE(u.retained_users_unbounded,0)::float/g.cohort_users END AS retention_rate_unbounded,
+  CASE WHEN g.user_lifetime_week=0 THEN g.cohort_users ELSE 0 END         AS cohort_users_w0
+FROM grid g
+LEFT JOIN bounded_counts   b ON b.cohort_week_start=g.cohort_week_start AND b.user_lifetime_week=g.user_lifetime_week
+LEFT JOIN unbounded_counts u ON u.cohort_week_start=g.cohort_week_start AND u.user_lifetime_week=g.user_lifetime_week
+ORDER BY g.cohort_week_start, g.user_lifetime_week;

autogpt_platform/analytics/queries/retention_login_weekly.sql

@@ -0,0 +1,103 @@
+-- =============================================================
+-- View: analytics.retention_login_weekly
+-- Looker source alias: ds83  |  Charts: 2
+-- =============================================================
+-- DESCRIPTION
+--   Weekly cohort retention based on login sessions.
+--   Users are grouped by the ISO week of their first ever login.
+--   For each cohort × lifetime-week combination, outputs both:
+--     - bounded rate: % active in exactly that week
+--     - unbounded rate: % who were ever active on or after that week
+--   Weeks are capped to the cohort's actual age (no future data points).
+--
+-- SOURCE TABLES
+--   auth.sessions  — Login session records
+--
+-- HOW TO READ THE OUTPUT
+--   cohort_week_start   The Monday of the week users first logged in
+--   user_lifetime_week  0 = signup week, 1 = one week later, etc.
+--   retention_rate_bounded   = active_users_bounded / cohort_users
+--   retention_rate_unbounded = retained_users_unbounded / cohort_users
+--
+-- OUTPUT COLUMNS
+--   cohort_week_start         DATE     First day of the cohort's signup week
+--   cohort_label              TEXT     ISO week label (e.g. '2025-W01')
+--   cohort_label_n            TEXT     ISO week label with cohort size (e.g. '2025-W01 (n=42)')
+--   user_lifetime_week        INT      Weeks since first login (0 = signup week)
+--   cohort_users              BIGINT   Total users in this cohort (denominator)
+--   active_users_bounded      BIGINT   Users active in exactly week k
+--   retained_users_unbounded  BIGINT   Users active any time on/after week k
+--   retention_rate_bounded    FLOAT    bounded active / cohort_users
+--   retention_rate_unbounded  FLOAT    unbounded retained / cohort_users
+--   cohort_users_w0           BIGINT   cohort_users only at week 0, else 0 (safe to SUM in pivot tables)
+--
+-- EXAMPLE QUERIES
+--   -- Week-1 retention rate per cohort
+--   SELECT cohort_label, retention_rate_bounded AS w1_retention
+--   FROM analytics.retention_login_weekly
+--   WHERE user_lifetime_week = 1
+--   ORDER BY cohort_week_start;
+--
+--   -- Overall average retention curve (all cohorts combined)
+--   SELECT user_lifetime_week,
+--          SUM(active_users_bounded)::float / NULLIF(SUM(cohort_users_w0), 0) AS avg_retention
+--   FROM analytics.retention_login_weekly
+--   GROUP BY 1 ORDER BY 1;
+-- =============================================================
+
+WITH params AS (SELECT 12::int AS max_weeks),
+events AS (
+  SELECT s.user_id::text AS user_id, s.created_at::timestamptz AS created_at,
+         DATE_TRUNC('week', s.created_at)::date AS week_start
+  FROM auth.sessions s WHERE s.user_id IS NOT NULL
+),
+first_login AS (
+  SELECT user_id, MIN(created_at) AS first_login_time,
+         DATE_TRUNC('week', MIN(created_at))::date AS cohort_week_start
+  FROM events GROUP BY 1
+),
+activity_weeks AS (SELECT DISTINCT user_id, week_start FROM events),
+user_week_age AS (
+  SELECT aw.user_id, fl.cohort_week_start,
+         ((aw.week_start - DATE_TRUNC('week', fl.first_login_time)::date) / 7)::int AS user_lifetime_week
+  FROM activity_weeks aw JOIN first_login fl USING (user_id)
+  WHERE aw.week_start >= DATE_TRUNC('week', fl.first_login_time)::date
+),
+bounded_counts AS (
+  SELECT cohort_week_start, user_lifetime_week, COUNT(DISTINCT user_id) AS active_users_bounded
+  FROM user_week_age WHERE user_lifetime_week >= 0 GROUP BY 1,2
+),
+last_active AS (
+  SELECT cohort_week_start, user_id, MAX(user_lifetime_week) AS last_active_week FROM user_week_age GROUP BY 1,2
+),
+unbounded_counts AS (
+  SELECT la.cohort_week_start, gs AS user_lifetime_week, COUNT(*) AS retained_users_unbounded
+  FROM last_active la
+  CROSS JOIN LATERAL generate_series(0, LEAST(la.last_active_week,(SELECT max_weeks FROM params))) gs
+  GROUP BY 1,2
+),
+cohort_sizes AS (SELECT cohort_week_start, COUNT(DISTINCT user_id) AS cohort_users FROM first_login GROUP BY 1),
+cohort_caps AS (
+  SELECT cs.cohort_week_start, cs.cohort_users,
+         LEAST((SELECT max_weeks FROM params),
+               GREATEST(0,((DATE_TRUNC('week',CURRENT_DATE)::date - cs.cohort_week_start)/7)::int)) AS cap_weeks
+  FROM cohort_sizes cs
+),
+grid AS (
+  SELECT cc.cohort_week_start, gs AS user_lifetime_week, cc.cohort_users
+  FROM cohort_caps cc CROSS JOIN LATERAL generate_series(0, cc.cap_weeks) gs
+)
+SELECT
+  g.cohort_week_start,
+  TO_CHAR(g.cohort_week_start,'IYYY-"W"IW')                                    AS cohort_label,
+  TO_CHAR(g.cohort_week_start,'IYYY-"W"IW')||' (n='||g.cohort_users||')'       AS cohort_label_n,
+  g.user_lifetime_week, g.cohort_users,
+  COALESCE(b.active_users_bounded,0)     AS active_users_bounded,
+  COALESCE(u.retained_users_unbounded,0) AS retained_users_unbounded,
+  CASE WHEN g.cohort_users>0 THEN COALESCE(b.active_users_bounded,0)::float/g.cohort_users END    AS retention_rate_bounded,
+  CASE WHEN g.cohort_users>0 THEN COALESCE(u.retained_users_unbounded,0)::float/g.cohort_users END AS retention_rate_unbounded,
+  CASE WHEN g.user_lifetime_week=0 THEN g.cohort_users ELSE 0 END               AS cohort_users_w0
+FROM grid g
+LEFT JOIN bounded_counts   b ON b.cohort_week_start=g.cohort_week_start AND b.user_lifetime_week=g.user_lifetime_week
+LEFT JOIN unbounded_counts u ON u.cohort_week_start=g.cohort_week_start AND u.user_lifetime_week=g.user_lifetime_week
+ORDER BY g.cohort_week_start, g.user_lifetime_week

autogpt_platform/analytics/queries/user_block_spending.sql

@@ -0,0 +1,71 @@
+-- =============================================================
+-- View: analytics.user_block_spending
+-- Looker source alias: ds6  |  Charts: 5
+-- =============================================================
+-- DESCRIPTION
+--   One row per credit transaction (last 90 days).
+--   Shows how users spend credits broken down by block type,
+--   LLM provider and model.  Joins node execution stats for
+--   token-level detail.
+--
+-- SOURCE TABLES
+--   platform.CreditTransaction   — Credit debit/credit records
+--   platform.AgentNodeExecution  — Node execution stats (for token counts)
+--
+-- OUTPUT COLUMNS
+--   transactionKey        TEXT         Unique transaction identifier
+--   userId                TEXT         User who was charged
+--   amount                DECIMAL      Credit amount (positive = credit, negative = debit)
+--   negativeAmount        DECIMAL      amount * -1 (convenience for spend charts)
+--   transactionType       TEXT         Transaction type (e.g. 'USAGE', 'REFUND', 'TOP_UP')
+--   transactionTime       TIMESTAMPTZ  When the transaction was recorded
+--   blockId               TEXT         Block UUID that triggered the spend
+--   blockName             TEXT         Human-readable block name
+--   llm_provider          TEXT         LLM provider (e.g. 'openai', 'anthropic')
+--   llm_model             TEXT         Model name (e.g. 'gpt-4o', 'claude-3-5-sonnet')
+--   node_exec_id          TEXT         Linked node execution UUID
+--   llm_call_count        INT          LLM API calls made in that execution
+--   llm_retry_count       INT          LLM retries in that execution
+--   llm_input_token_count INT          Input tokens consumed
+--   llm_output_token_count INT         Output tokens produced
+--
+-- WINDOW
+--   Rolling 90 days (createdAt > CURRENT_DATE - 90 days)
+--
+-- EXAMPLE QUERIES
+--   -- Total spend per user (last 90 days)
+--   SELECT "userId", SUM("negativeAmount") AS total_spent
+--   FROM analytics.user_block_spending
+--   WHERE "transactionType" = 'USAGE'
+--   GROUP BY 1 ORDER BY total_spent DESC;
+--
+--   -- Spend by LLM provider + model
+--   SELECT "llm_provider", "llm_model",
+--          SUM("negativeAmount") AS total_cost,
+--          SUM("llm_input_token_count") AS input_tokens,
+--          SUM("llm_output_token_count") AS output_tokens
+--   FROM analytics.user_block_spending
+--   WHERE "llm_provider" IS NOT NULL
+--   GROUP BY 1, 2 ORDER BY total_cost DESC;
+-- =============================================================
+
+SELECT
+    c."transactionKey"                                        AS transactionKey,
+    c."userId"                                                AS userId,
+    c."amount"                                                AS amount,
+    c."amount" * -1                                           AS negativeAmount,
+    c."type"                                                  AS transactionType,
+    c."createdAt"                                             AS transactionTime,
+    c.metadata->>'block_id'                                   AS blockId,
+    c.metadata->>'block'                                      AS blockName,
+    c.metadata->'input'->'credentials'->>'provider'           AS llm_provider,
+    c.metadata->'input'->>'model'                             AS llm_model,
+    c.metadata->>'node_exec_id'                               AS node_exec_id,
+    (ne."stats"->>'llm_call_count')::int                       AS llm_call_count,
+    (ne."stats"->>'llm_retry_count')::int                      AS llm_retry_count,
+    (ne."stats"->>'input_token_count')::int                    AS llm_input_token_count,
+    (ne."stats"->>'output_token_count')::int                   AS llm_output_token_count
+FROM platform."CreditTransaction" c
+LEFT JOIN platform."AgentNodeExecution" ne
+       ON (c.metadata->>'node_exec_id') = ne."id"::text
+WHERE c."createdAt" > CURRENT_DATE - INTERVAL '90 days'

autogpt_platform/analytics/queries/user_onboarding.sql

@@ -0,0 +1,45 @@
+-- =============================================================
+-- View: analytics.user_onboarding
+-- Looker source alias: ds68  |  Charts: 3
+-- =============================================================
+-- DESCRIPTION
+--   One row per user onboarding record.  Contains the user's
+--   stated usage reason, selected integrations, completed
+--   onboarding steps and optional first agent selection.
+--   Full history (no date filter) since onboarding happens
+--   once per user.
+--
+-- SOURCE TABLES
+--   platform.UserOnboarding  — Onboarding state per user
+--
+-- OUTPUT COLUMNS
+--   id                            TEXT         Onboarding record UUID
+--   createdAt                     TIMESTAMPTZ  When onboarding started
+--   updatedAt                     TIMESTAMPTZ  Last update to onboarding state
+--   usageReason                   TEXT         Why user signed up (e.g. 'work', 'personal')
+--   integrations                  TEXT[]       Array of integration names the user selected
+--   userId                        TEXT         User UUID
+--   completedSteps                TEXT[]       Array of onboarding step enums completed
+--   selectedStoreListingVersionId TEXT         First marketplace agent the user chose (if any)
+--
+-- EXAMPLE QUERIES
+--   -- Usage reason breakdown
+--   SELECT "usageReason", COUNT(*) FROM analytics.user_onboarding GROUP BY 1;
+--
+--   -- Completion rate per step
+--   SELECT step, COUNT(*) AS users_completed
+--   FROM analytics.user_onboarding
+--   CROSS JOIN LATERAL UNNEST("completedSteps") AS step
+--   GROUP BY 1 ORDER BY users_completed DESC;
+-- =============================================================
+
+SELECT
+    id,
+    "createdAt",
+    "updatedAt",
+    "usageReason",
+    integrations,
+    "userId",
+    "completedSteps",
+    "selectedStoreListingVersionId"
+FROM platform."UserOnboarding"

autogpt_platform/analytics/queries/user_onboarding_funnel.sql

@@ -0,0 +1,100 @@
+-- =============================================================
+-- View: analytics.user_onboarding_funnel
+-- Looker source alias: ds74  |  Charts: 1
+-- =============================================================
+-- DESCRIPTION
+--   Pre-aggregated onboarding funnel showing how many users
+--   completed each step and the drop-off percentage from the
+--   previous step.  One row per onboarding step (all 22 steps
+--   always present, even with 0 completions — prevents sparse
+--   gaps from making LAG compare the wrong predecessors).
+--
+-- SOURCE TABLES
+--   platform.UserOnboarding  — Onboarding records with completedSteps array
+--
+-- OUTPUT COLUMNS
+--   step             TEXT     Onboarding step enum name (e.g. 'WELCOME', 'CONGRATS')
+--   step_order       INT      Numeric position in the funnel (1=first, 22=last)
+--   users_completed  BIGINT   Distinct users who completed this step
+--   pct_from_prev    NUMERIC  % of users from the previous step who reached this one
+--
+-- STEP ORDER
+--   1  WELCOME               9  MARKETPLACE_VISIT     17  SCHEDULE_AGENT
+--   2  USAGE_REASON         10  MARKETPLACE_ADD_AGENT  18  RUN_AGENTS
+--   3  INTEGRATIONS         11  MARKETPLACE_RUN_AGENT  19  RUN_3_DAYS
+--   4  AGENT_CHOICE         12  BUILDER_OPEN           20  TRIGGER_WEBHOOK
+--   5  AGENT_NEW_RUN        13  BUILDER_SAVE_AGENT     21  RUN_14_DAYS
+--   6  AGENT_INPUT          14  BUILDER_RUN_AGENT      22  RUN_AGENTS_100
+--   7  CONGRATS             15  VISIT_COPILOT
+--   8  GET_RESULTS          16  RE_RUN_AGENT
+--
+-- WINDOW
+--   Users who started onboarding in the last 90 days
+--
+-- EXAMPLE QUERIES
+--   -- Full funnel
+--   SELECT * FROM analytics.user_onboarding_funnel ORDER BY step_order;
+--
+--   -- Biggest drop-off point
+--   SELECT step, pct_from_prev FROM analytics.user_onboarding_funnel
+--   ORDER BY pct_from_prev ASC LIMIT 3;
+-- =============================================================
+
+WITH all_steps AS (
+  -- Complete ordered grid of all 22 steps so zero-completion steps
+  -- are always present, keeping LAG comparisons correct.
+  SELECT step_name, step_order
+  FROM (VALUES
+    ('WELCOME',               1),
+    ('USAGE_REASON',          2),
+    ('INTEGRATIONS',          3),
+    ('AGENT_CHOICE',          4),
+    ('AGENT_NEW_RUN',         5),
+    ('AGENT_INPUT',           6),
+    ('CONGRATS',              7),
+    ('GET_RESULTS',           8),
+    ('MARKETPLACE_VISIT',     9),
+    ('MARKETPLACE_ADD_AGENT', 10),
+    ('MARKETPLACE_RUN_AGENT', 11),
+    ('BUILDER_OPEN',          12),
+    ('BUILDER_SAVE_AGENT',    13),
+    ('BUILDER_RUN_AGENT',     14),
+    ('VISIT_COPILOT',         15),
+    ('RE_RUN_AGENT',          16),
+    ('SCHEDULE_AGENT',        17),
+    ('RUN_AGENTS',            18),
+    ('RUN_3_DAYS',            19),
+    ('TRIGGER_WEBHOOK',       20),
+    ('RUN_14_DAYS',           21),
+    ('RUN_AGENTS_100',        22)
+  ) AS t(step_name, step_order)
+),
+raw AS (
+  SELECT
+      u."userId",
+      step_txt::text AS step
+  FROM platform."UserOnboarding" u
+  CROSS JOIN LATERAL UNNEST(u."completedSteps") AS step_txt
+  WHERE u."createdAt" >= CURRENT_DATE - INTERVAL '90 days'
+),
+step_counts AS (
+  SELECT step, COUNT(DISTINCT "userId") AS users_completed
+  FROM raw GROUP BY step
+),
+funnel AS (
+  SELECT
+      a.step_name                          AS step,
+      a.step_order,
+      COALESCE(sc.users_completed, 0)      AS users_completed,
+      ROUND(
+        100.0 * COALESCE(sc.users_completed, 0)
+        / NULLIF(
+            LAG(COALESCE(sc.users_completed, 0)) OVER (ORDER BY a.step_order),
+            0
+          ),
+        2
+      )                                    AS pct_from_prev
+  FROM all_steps a
+  LEFT JOIN step_counts sc ON sc.step = a.step_name
+)
+SELECT * FROM funnel ORDER BY step_order

autogpt_platform/analytics/queries/user_onboarding_integration.sql

@@ -0,0 +1,41 @@
+-- =============================================================
+-- View: analytics.user_onboarding_integration
+-- Looker source alias: ds75  |  Charts: 1
+-- =============================================================
+-- DESCRIPTION
+--   Pre-aggregated count of users who selected each integration
+--   during onboarding.  One row per integration type, sorted
+--   by popularity.
+--
+-- SOURCE TABLES
+--   platform.UserOnboarding  — integrations array column
+--
+-- OUTPUT COLUMNS
+--   integration            TEXT    Integration name (e.g. 'github', 'slack', 'notion')
+--   users_with_integration BIGINT  Distinct users who selected this integration
+--
+-- WINDOW
+--   Users who started onboarding in the last 90 days
+--
+-- EXAMPLE QUERIES
+--   -- Full integration popularity ranking
+--   SELECT * FROM analytics.user_onboarding_integration;
+--
+--   -- Top 5 integrations
+--   SELECT * FROM analytics.user_onboarding_integration LIMIT 5;
+-- =============================================================
+
+WITH exploded AS (
+  SELECT
+      u."userId" AS user_id,
+      UNNEST(u."integrations") AS integration
+  FROM platform."UserOnboarding" u
+  WHERE u."createdAt" >= CURRENT_DATE - INTERVAL '90 days'
+)
+SELECT
+    integration,
+    COUNT(DISTINCT user_id) AS users_with_integration
+FROM exploded
+WHERE integration IS NOT NULL AND integration <> ''
+GROUP BY integration
+ORDER BY users_with_integration DESC

autogpt_platform/analytics/queries/users_activities.sql

@@ -0,0 +1,145 @@
+-- =============================================================
+-- View: analytics.users_activities
+-- Looker source alias: ds56  |  Charts: 5
+-- =============================================================
+-- DESCRIPTION
+--   One row per user with lifetime activity summary.
+--   Joins login sessions with agent graphs, executions and
+--   node-level runs to give a full picture of how engaged
+--   each user is.  Includes a convenience flag for 7-day
+--   activation (did the user return at least 7 days after
+--   their first login?).
+--
+-- SOURCE TABLES
+--   auth.sessions                    — Login/session records
+--   platform.AgentGraph              — Graphs (agents) built by the user
+--   platform.AgentGraphExecution     — Agent run history
+--   platform.AgentNodeExecution      — Individual block execution history
+--
+-- PERFORMANCE NOTE
+--   Each CTE aggregates its own table independently by userId.
+--   This avoids the fan-out that occurs when driving every join
+--   from user_logins across the two largest tables
+--   (AgentGraphExecution and AgentNodeExecution).
+--
+-- OUTPUT COLUMNS
+--   user_id                   TEXT         Supabase user UUID
+--   first_login_time          TIMESTAMPTZ  First ever session created_at
+--   last_login_time           TIMESTAMPTZ  Most recent session created_at
+--   last_visit_time           TIMESTAMPTZ  Max of last refresh or login
+--   last_agent_save_time      TIMESTAMPTZ  Last time user saved an agent graph
+--   agent_count               BIGINT       Number of distinct active graphs built (0 if none)
+--   first_agent_run_time      TIMESTAMPTZ  First ever graph execution
+--   last_agent_run_time       TIMESTAMPTZ  Most recent graph execution
+--   unique_agent_runs         BIGINT       Distinct agent graphs ever run (0 if none)
+--   agent_runs                BIGINT       Total graph execution count (0 if none)
+--   node_execution_count      BIGINT       Total node executions across all runs
+--   node_execution_failed     BIGINT       Node executions with FAILED status
+--   node_execution_completed  BIGINT       Node executions with COMPLETED status
+--   node_execution_terminated BIGINT       Node executions with TERMINATED status
+--   node_execution_queued     BIGINT       Node executions with QUEUED status
+--   node_execution_running    BIGINT       Node executions with RUNNING status
+--   is_active_after_7d        INT          1=returned after day 7, 0=did not, NULL=too early to tell
+--   node_execution_incomplete BIGINT       Node executions with INCOMPLETE status
+--   node_execution_review     BIGINT       Node executions with REVIEW status
+--
+-- EXAMPLE QUERIES
+--   -- Users who ran at least one agent and returned after 7 days
+--   SELECT COUNT(*) FROM analytics.users_activities
+--   WHERE agent_runs > 0 AND is_active_after_7d = 1;
+--
+--   -- Top 10 most active users by agent runs
+--   SELECT user_id, agent_runs, node_execution_count
+--   FROM analytics.users_activities
+--   ORDER BY agent_runs DESC LIMIT 10;
+--
+--   -- 7-day activation rate
+--   SELECT
+--     SUM(CASE WHEN is_active_after_7d = 1 THEN 1 ELSE 0 END)::float
+--     / NULLIF(COUNT(CASE WHEN is_active_after_7d IS NOT NULL THEN 1 END), 0)
+--     AS activation_rate
+--   FROM analytics.users_activities;
+-- =============================================================
+
+WITH user_logins AS (
+  SELECT
+    user_id::text                                    AS user_id,
+    MIN(created_at)                                  AS first_login_time,
+    MAX(created_at)                                  AS last_login_time,
+    GREATEST(
+      MAX(refreshed_at)::timestamptz,
+      MAX(created_at)::timestamptz
+    )                                                AS last_visit_time
+  FROM auth.sessions
+  GROUP BY user_id
+),
+user_agents AS (
+  -- Aggregate AgentGraph directly by userId (no fan-out from user_logins)
+  SELECT
+    "userId"::text                AS user_id,
+    MAX("updatedAt")              AS last_agent_save_time,
+    COUNT(DISTINCT "id")          AS agent_count
+  FROM platform."AgentGraph"
+  WHERE "isActive"
+  GROUP BY "userId"
+),
+user_graph_runs AS (
+  -- Aggregate AgentGraphExecution directly by userId
+  SELECT
+    "userId"::text                        AS user_id,
+    MIN("createdAt")                      AS first_agent_run_time,
+    MAX("createdAt")                      AS last_agent_run_time,
+    COUNT(DISTINCT "agentGraphId")        AS unique_agent_runs,
+    COUNT("id")                           AS agent_runs
+  FROM platform."AgentGraphExecution"
+  GROUP BY "userId"
+),
+user_node_runs AS (
+  -- Aggregate AgentNodeExecution directly; resolve userId via a
+  -- single join to AgentGraphExecution instead of fanning out from
+  -- user_logins through both large tables.
+  SELECT
+    g."userId"::text                                                   AS user_id,
+    COUNT(*)                                                           AS node_execution_count,
+    COUNT(*) FILTER (WHERE n."executionStatus" = 'FAILED')             AS node_execution_failed,
+    COUNT(*) FILTER (WHERE n."executionStatus" = 'COMPLETED')          AS node_execution_completed,
+    COUNT(*) FILTER (WHERE n."executionStatus" = 'TERMINATED')         AS node_execution_terminated,
+    COUNT(*) FILTER (WHERE n."executionStatus" = 'QUEUED')             AS node_execution_queued,
+    COUNT(*) FILTER (WHERE n."executionStatus" = 'RUNNING')            AS node_execution_running,
+    COUNT(*) FILTER (WHERE n."executionStatus" = 'INCOMPLETE')         AS node_execution_incomplete,
+    COUNT(*) FILTER (WHERE n."executionStatus" = 'REVIEW')             AS node_execution_review
+  FROM platform."AgentNodeExecution" n
+  JOIN platform."AgentGraphExecution" g
+    ON g."id" = n."agentGraphExecutionId"
+  GROUP BY g."userId"
+)
+SELECT
+  ul.user_id,
+  ul.first_login_time,
+  ul.last_login_time,
+  ul.last_visit_time,
+  ua.last_agent_save_time,
+  COALESCE(ua.agent_count, 0)             AS agent_count,
+  gr.first_agent_run_time,
+  gr.last_agent_run_time,
+  COALESCE(gr.unique_agent_runs, 0)       AS unique_agent_runs,
+  COALESCE(gr.agent_runs, 0)              AS agent_runs,
+  COALESCE(nr.node_execution_count, 0)      AS node_execution_count,
+  COALESCE(nr.node_execution_failed, 0)     AS node_execution_failed,
+  COALESCE(nr.node_execution_completed, 0)  AS node_execution_completed,
+  COALESCE(nr.node_execution_terminated, 0) AS node_execution_terminated,
+  COALESCE(nr.node_execution_queued, 0)     AS node_execution_queued,
+  COALESCE(nr.node_execution_running, 0)    AS node_execution_running,
+  CASE
+    WHEN ul.first_login_time < NOW() - INTERVAL '7 days'
+     AND ul.last_visit_time  >= ul.first_login_time + INTERVAL '7 days' THEN 1
+    WHEN ul.first_login_time < NOW() - INTERVAL '7 days'
+     AND ul.last_visit_time  <  ul.first_login_time + INTERVAL '7 days' THEN 0
+    ELSE NULL
+  END AS is_active_after_7d,
+  COALESCE(nr.node_execution_incomplete, 0) AS node_execution_incomplete,
+  COALESCE(nr.node_execution_review, 0)     AS node_execution_review
+FROM user_logins ul
+LEFT JOIN user_agents     ua ON ul.user_id = ua.user_id
+LEFT JOIN user_graph_runs gr ON ul.user_id = gr.user_id
+LEFT JOIN user_node_runs  nr ON ul.user_id = nr.user_id

autogpt_platform/backend/.env.default

@@ -37,6 +37,10 @@ JWT_VERIFY_KEY=your-super-secret-jwt-token-with-at-least-32-characters-long
 ENCRYPTION_KEY=dvziYgz0KSK8FENhju0ZYi8-fRTfAdlz6YLhdB_jhNw=
 UNSUBSCRIBE_SECRET_KEY=HlP8ivStJjmbf6NKi78m_3FnOogut0t5ckzjsIqeaio=
 
+## ===== SIGNUP / INVITE GATE ===== ##
+# Set to true to require an invite before users can sign up
+ENABLE_INVITE_GATE=false
+
 ## ===== IMPORTANT OPTIONAL CONFIGURATION ===== ##
 # Platform URLs (set these for webhooks and OAuth to work)
 PLATFORM_BASE_URL=http://localhost:8000
@@ -190,5 +194,8 @@ ZEROBOUNCE_API_KEY=
 POSTHOG_API_KEY=
 POSTHOG_HOST=https://eu.i.posthog.com
 
+# Tally Form Integration (pre-populate business understanding on signup)
+TALLY_API_KEY=
+
 # Other Services
 AUTOMOD_API_KEY=

autogpt_platform/backend/CLAUDE.md

@@ -58,10 +58,31 @@ poetry run pytest path/to/test.py --snapshot-update
 - **Authentication**: JWT-based with Supabase integration
 - **Security**: Cache protection middleware prevents sensitive data caching in browsers/proxies
 
+## Code Style
+
+- **Top-level imports only** — no local/inner imports (lazy imports only for heavy optional deps like `openpyxl`)
+- **No duck typing** — no `hasattr`/`getattr`/`isinstance` for type dispatch; use typed interfaces/unions/protocols
+- **Pydantic models** over dataclass/namedtuple/dict for structured data
+- **No linter suppressors** — no `# type: ignore`, `# noqa`, `# pyright: ignore`; fix the type/code
+- **List comprehensions** over manual loop-and-append
+- **Early return** — guard clauses first, avoid deep nesting
+- **Lazy `%s` logging** — `logger.info("Processing %s items", count)` not `logger.info(f"Processing {count} items")`
+- **Sanitize error paths** — `os.path.basename()` in error messages to avoid leaking directory structure
+- **TOCTOU awareness** — avoid check-then-act patterns for file access and credit charging
+- **`Security()` vs `Depends()`** — use `Security()` for auth deps to get proper OpenAPI security spec
+- **Redis pipelines** — `transaction=True` for atomicity on multi-step operations
+- **`max(0, value)` guards** — for computed values that should never be negative
+- **SSE protocol** — `data:` lines for frontend-parsed events (must match Zod schema), `: comment` lines for heartbeats/status
+- **File length** — keep files under ~300 lines; if a file grows beyond this, split by responsibility (e.g. extract helpers, models, or a sub-module into a new file). Never keep appending to a long file.
+- **Function length** — keep functions under ~40 lines; extract named helpers when a function grows longer. Long functions are a sign of mixed concerns, not complexity.
+
 ## Testing Approach
 
 - Uses pytest with snapshot testing for API responses
 - Test files are colocated with source files (`*_test.py`)
+- Mock at boundaries — mock where the symbol is **used**, not where it's **defined**
+- After refactoring, update mock targets to match new module paths
+- Use `AsyncMock` for async functions (`from unittest.mock import AsyncMock`)
 
 ## Database Schema
 

autogpt_platform/backend/Dockerfile

@@ -53,63 +53,6 @@ COPY autogpt_platform/backend/backend/data/partial_types.py ./backend/data/parti
 COPY autogpt_platform/backend/gen_prisma_types_stub.py ./
 RUN poetry run prisma generate && poetry run gen-prisma-stub
 
-# ============================== BACKEND SERVER ============================== #
-
-FROM debian:13-slim AS server
-
-WORKDIR /app
-
-ENV POETRY_HOME=/opt/poetry \
-    POETRY_NO_INTERACTION=1 \
-    POETRY_VIRTUALENVS_CREATE=true \
-    POETRY_VIRTUALENVS_IN_PROJECT=true \
-    DEBIAN_FRONTEND=noninteractive
-ENV PATH=/opt/poetry/bin:$PATH
-
-# Install Python, FFmpeg, ImageMagick, and CLI tools for agent use.
-# bubblewrap provides OS-level sandbox (whitelist-only FS + no network)
-# for the bash_exec MCP tool.
-# Using --no-install-recommends saves ~650MB by skipping unnecessary deps like llvm, mesa, etc.
-RUN apt-get update && apt-get install -y --no-install-recommends \
-    python3.13 \
-    python3-pip \
-    ffmpeg \
-    imagemagick \
-    jq \
-    ripgrep \
-    tree \
-    bubblewrap \
-    && rm -rf /var/lib/apt/lists/*
-
-COPY --from=builder /usr/local/lib/python3* /usr/local/lib/python3*
-COPY --from=builder /usr/local/bin/poetry /usr/local/bin/poetry
-# Copy Node.js installation for Prisma
-COPY --from=builder /usr/bin/node /usr/bin/node
-COPY --from=builder /usr/lib/node_modules /usr/lib/node_modules
-COPY --from=builder /usr/bin/npm /usr/bin/npm
-COPY --from=builder /usr/bin/npx /usr/bin/npx
-COPY --from=builder /root/.cache/prisma-python/binaries /root/.cache/prisma-python/binaries
-
-WORKDIR /app/autogpt_platform/backend
-
-# Copy only the .venv from builder (not the entire /app directory)
-# The .venv includes the generated Prisma client
-COPY --from=builder /app/autogpt_platform/backend/.venv ./.venv
-ENV PATH="/app/autogpt_platform/backend/.venv/bin:$PATH"
-
-# Copy dependency files + autogpt_libs (path dependency)
-COPY autogpt_platform/autogpt_libs /app/autogpt_platform/autogpt_libs
-COPY autogpt_platform/backend/poetry.lock autogpt_platform/backend/pyproject.toml ./
-
-# Copy backend code + docs (for Copilot docs search)
-COPY autogpt_platform/backend ./
-COPY docs /app/docs
-RUN poetry install --no-ansi --only-root
-
-ENV PORT=8000
-
-CMD ["poetry", "run", "rest"]
-
 # =============================== DB MIGRATOR =============================== #
 
 # Lightweight migrate stage - only needs Prisma CLI, not full Python environment
@@ -141,3 +84,75 @@ COPY autogpt_platform/backend/schema.prisma ./
 COPY autogpt_platform/backend/backend/data/partial_types.py ./backend/data/partial_types.py
 COPY autogpt_platform/backend/gen_prisma_types_stub.py ./
 COPY autogpt_platform/backend/migrations ./migrations
+
+# ============================== BACKEND SERVER ============================== #
+
+FROM debian:13-slim AS server
+
+WORKDIR /app
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+# Install Python, FFmpeg, ImageMagick, and CLI tools for agent use.
+# bubblewrap provides OS-level sandbox (whitelist-only FS + no network)
+# for the bash_exec MCP tool (fallback when E2B is not configured).
+# Using --no-install-recommends saves ~650MB by skipping unnecessary deps like llvm, mesa, etc.
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    python3.13 \
+    python3-pip \
+    ffmpeg \
+    imagemagick \
+    jq \
+    ripgrep \
+    tree \
+    bubblewrap \
+    && rm -rf /var/lib/apt/lists/*
+
+# Copy poetry (build-time only, for `poetry install --only-root` to create entry points)
+COPY --from=builder /usr/local/lib/python3* /usr/local/lib/python3*
+COPY --from=builder /usr/local/bin/poetry /usr/local/bin/poetry
+# Copy Node.js installation for Prisma and agent-browser.
+# npm/npx are symlinks in the builder (-> ../lib/node_modules/npm/bin/*-cli.js);
+# COPY resolves them to regular files, breaking require() paths.  Recreate as
+# proper symlinks so npm/npx can find their modules.
+COPY --from=builder /usr/bin/node /usr/bin/node
+COPY --from=builder /usr/lib/node_modules /usr/lib/node_modules
+RUN ln -s ../lib/node_modules/npm/bin/npm-cli.js /usr/bin/npm \
+    && ln -s ../lib/node_modules/npm/bin/npx-cli.js /usr/bin/npx
+COPY --from=builder /root/.cache/prisma-python/binaries /root/.cache/prisma-python/binaries
+
+# Install agent-browser (Copilot browser tool) + Chromium runtime dependencies.
+# These are the runtime libraries Chromium/Playwright needs on Debian 13 (trixie).
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    libnss3 libnspr4 libatk1.0-0 libatk-bridge2.0-0 libcups2 libdrm2 \
+    libdbus-1-3 libxkbcommon0 libatspi2.0-0t64 libxcomposite1 libxdamage1 \
+    libxfixes3 libxrandr2 libgbm1 libasound2t64 libpango-1.0-0 libcairo2 \
+    libx11-6 libx11-xcb1 libxcb1 libxext6 libglib2.0-0t64 \
+    fonts-liberation libfontconfig1 \
+    && rm -rf /var/lib/apt/lists/* \
+    && npm install -g agent-browser \
+    && agent-browser install \
+    && rm -rf /tmp/* /root/.npm
+
+WORKDIR /app/autogpt_platform/backend
+
+# Copy only the .venv from builder (not the entire /app directory)
+# The .venv includes the generated Prisma client
+COPY --from=builder /app/autogpt_platform/backend/.venv ./.venv
+ENV PATH="/app/autogpt_platform/backend/.venv/bin:$PATH"
+
+# Copy dependency files + autogpt_libs (path dependency)
+COPY autogpt_platform/autogpt_libs /app/autogpt_platform/autogpt_libs
+COPY autogpt_platform/backend/poetry.lock autogpt_platform/backend/pyproject.toml ./
+
+# Copy backend code + docs (for Copilot docs search)
+COPY autogpt_platform/backend ./
+COPY docs /app/docs
+# Install the project package to create entry point scripts in .venv/bin/
+# (e.g., rest, executor, ws, db, scheduler, notification - see [tool.poetry.scripts])
+RUN POETRY_VIRTUALENVS_CREATE=true POETRY_VIRTUALENVS_IN_PROJECT=true \
+    poetry install --no-ansi --only-root
+
+ENV PORT=8000
+
+CMD ["rest"]

autogpt_platform/backend/backend/api/conftest.py

@@ -1,4 +1,9 @@
-"""Common test fixtures for server tests."""
+"""Common test fixtures for server tests.
+
+Note: Common fixtures like test_user_id, admin_user_id, target_user_id,
+setup_test_user, and setup_admin_user are defined in the parent conftest.py
+(backend/conftest.py) and are available here automatically.
+"""
 
 import pytest
 from pytest_snapshot.plugin import Snapshot
@@ -11,54 +16,6 @@ def configured_snapshot(snapshot: Snapshot) -> Snapshot:
     return snapshot
 
 
-@pytest.fixture
-def test_user_id() -> str:
-    """Test user ID fixture."""
-    return "3e53486c-cf57-477e-ba2a-cb02dc828e1a"
-
-
-@pytest.fixture
-def admin_user_id() -> str:
-    """Admin user ID fixture."""
-    return "4e53486c-cf57-477e-ba2a-cb02dc828e1b"
-
-
-@pytest.fixture
-def target_user_id() -> str:
-    """Target user ID fixture."""
-    return "5e53486c-cf57-477e-ba2a-cb02dc828e1c"
-
-
-@pytest.fixture
-async def setup_test_user(test_user_id):
-    """Create test user in database before tests."""
-    from backend.data.user import get_or_create_user
-
-    # Create the test user in the database using JWT token format
-    user_data = {
-        "sub": test_user_id,
-        "email": "test@example.com",
-        "user_metadata": {"name": "Test User"},
-    }
-    await get_or_create_user(user_data)
-    return test_user_id
-
-
-@pytest.fixture
-async def setup_admin_user(admin_user_id):
-    """Create admin user in database before tests."""
-    from backend.data.user import get_or_create_user
-
-    # Create the admin user in the database using JWT token format
-    user_data = {
-        "sub": admin_user_id,
-        "email": "test-admin@example.com",
-        "user_metadata": {"name": "Test Admin"},
-    }
-    await get_or_create_user(user_data)
-    return admin_user_id
-
-
 @pytest.fixture
 def mock_jwt_user(test_user_id):
     """Provide mock JWT payload for regular user testing."""

autogpt_platform/backend/backend/api/external/middleware.py

@@ -88,20 +88,23 @@ async def require_auth(
     )
 
 
-def require_permission(permission: APIKeyPermission):
+def require_permission(*permissions: APIKeyPermission):
     """
-    Dependency function for checking specific permissions
+    Dependency function for checking required permissions.
+    All listed permissions must be present.
     (works with API keys and OAuth tokens)
     """
 
-    async def check_permission(
+    async def check_permissions(
         auth: APIAuthorizationInfo = Security(require_auth),
     ) -> APIAuthorizationInfo:
-        if permission not in auth.scopes:
+        missing = [p for p in permissions if p not in auth.scopes]
+        if missing:
             raise HTTPException(
                 status_code=status.HTTP_403_FORBIDDEN,
-                detail=f"Missing required permission: {permission.value}",
+                detail=f"Missing required permission(s): "
+                f"{', '.join(p.value for p in missing)}",
             )
         return auth
 
-    return check_permission
+    return check_permissions

autogpt_platform/backend/backend/api/external/v1/routes.py

@@ -1,7 +1,7 @@
 import logging
 import urllib.parse
 from collections import defaultdict
-from typing import Annotated, Any, Literal, Optional, Sequence
+from typing import Annotated, Any, Optional, Sequence
 
 from fastapi import APIRouter, Body, HTTPException, Security
 from prisma.enums import AgentExecutionStatus, APIKeyPermission
@@ -9,15 +9,17 @@ from pydantic import BaseModel, Field
 from typing_extensions import TypedDict
 
 import backend.api.features.store.cache as store_cache
+import backend.api.features.store.db as store_db
 import backend.api.features.store.model as store_model
 import backend.blocks
-from backend.api.external.middleware import require_permission
+from backend.api.external.middleware import require_auth, require_permission
 from backend.data import execution as execution_db
 from backend.data import graph as graph_db
 from backend.data import user as user_db
 from backend.data.auth.base import APIAuthorizationInfo
 from backend.data.block import BlockInput, CompletedBlockOutput
 from backend.executor.utils import add_graph_execution
+from backend.integrations.webhooks.graph_lifecycle_hooks import on_graph_activate
 from backend.util.settings import Settings
 
 from .integrations import integrations_router
@@ -95,6 +97,43 @@ async def execute_graph_block(
     return output
 
 
+@v1_router.post(
+    path="/graphs",
+    tags=["graphs"],
+    status_code=201,
+    dependencies=[
+        Security(
+            require_permission(
+                APIKeyPermission.WRITE_GRAPH, APIKeyPermission.WRITE_LIBRARY
+            )
+        )
+    ],
+)
+async def create_graph(
+    graph: graph_db.Graph,
+    auth: APIAuthorizationInfo = Security(
+        require_permission(APIKeyPermission.WRITE_GRAPH, APIKeyPermission.WRITE_LIBRARY)
+    ),
+) -> graph_db.GraphModel:
+    """
+    Create a new agent graph.
+
+    The graph will be validated and assigned a new ID.
+    It is automatically added to the user's library.
+    """
+    from backend.api.features.library import db as library_db
+
+    graph_model = graph_db.make_graph_model(graph, auth.user_id)
+    graph_model.reassign_ids(user_id=auth.user_id, reassign_graph_id=True)
+    graph_model.validate_graph(for_run=False)
+
+    await graph_db.create_graph(graph_model, user_id=auth.user_id)
+    await library_db.create_library_agent(graph_model, auth.user_id)
+    activated_graph = await on_graph_activate(graph_model, user_id=auth.user_id)
+
+    return activated_graph
+
+
 @v1_router.post(
     path="/graphs/{graph_id}/execute/{graph_version}",
     tags=["graphs"],
@@ -192,13 +231,13 @@ async def get_graph_execution_results(
 @v1_router.get(
     path="/store/agents",
     tags=["store"],
-    dependencies=[Security(require_permission(APIKeyPermission.READ_STORE))],
+    dependencies=[Security(require_auth)],  # data is public; auth required as anti-DDoS
     response_model=store_model.StoreAgentsResponse,
 )
 async def get_store_agents(
     featured: bool = False,
     creator: str | None = None,
-    sorted_by: Literal["rating", "runs", "name", "updated_at"] | None = None,
+    sorted_by: store_db.StoreAgentsSortOptions | None = None,
     search_query: str | None = None,
     category: str | None = None,
     page: int = 1,
@@ -240,7 +279,7 @@ async def get_store_agents(
 @v1_router.get(
     path="/store/agents/{username}/{agent_name}",
     tags=["store"],
-    dependencies=[Security(require_permission(APIKeyPermission.READ_STORE))],
+    dependencies=[Security(require_auth)],  # data is public; auth required as anti-DDoS
     response_model=store_model.StoreAgentDetails,
 )
 async def get_store_agent(
@@ -268,13 +307,13 @@ async def get_store_agent(
 @v1_router.get(
     path="/store/creators",
     tags=["store"],
-    dependencies=[Security(require_permission(APIKeyPermission.READ_STORE))],
+    dependencies=[Security(require_auth)],  # data is public; auth required as anti-DDoS
     response_model=store_model.CreatorsResponse,
 )
 async def get_store_creators(
     featured: bool = False,
     search_query: str | None = None,
-    sorted_by: Literal["agent_rating", "agent_runs", "num_agents"] | None = None,
+    sorted_by: store_db.StoreCreatorsSortOptions | None = None,
     page: int = 1,
     page_size: int = 20,
 ) -> store_model.CreatorsResponse:
@@ -310,7 +349,7 @@ async def get_store_creators(
 @v1_router.get(
     path="/store/creators/{username}",
     tags=["store"],
-    dependencies=[Security(require_permission(APIKeyPermission.READ_STORE))],
+    dependencies=[Security(require_auth)],  # data is public; auth required as anti-DDoS
     response_model=store_model.CreatorDetails,
 )
 async def get_store_creator(

autogpt_platform/backend/backend/api/external/v1/tools.py

@@ -15,9 +15,9 @@ from prisma.enums import APIKeyPermission
 from pydantic import BaseModel, Field
 
 from backend.api.external.middleware import require_permission
-from backend.api.features.chat.model import ChatSession
-from backend.api.features.chat.tools import find_agent_tool, run_agent_tool
-from backend.api.features.chat.tools.models import ToolResponseBase
+from backend.copilot.model import ChatSession
+from backend.copilot.tools import find_agent_tool, run_agent_tool
+from backend.copilot.tools.models import ToolResponseBase
 from backend.data.auth.base import APIAuthorizationInfo
 
 logger = logging.getLogger(__name__)

autogpt_platform/backend/backend/api/features/admin/model.py

@@ -1,8 +1,17 @@
-from pydantic import BaseModel
+from __future__ import annotations
+
+from datetime import datetime
+from typing import TYPE_CHECKING, Any, Literal, Optional
+
+import prisma.enums
+from pydantic import BaseModel, EmailStr
 
 from backend.data.model import UserTransaction
 from backend.util.models import Pagination
 
+if TYPE_CHECKING:
+    from backend.data.invited_user import BulkInvitedUsersResult, InvitedUserRecord
+
 
 class UserHistoryResponse(BaseModel):
     """Response model for listings with version history"""
@@ -14,3 +23,70 @@ class UserHistoryResponse(BaseModel):
 class AddUserCreditsResponse(BaseModel):
     new_balance: int
     transaction_key: str
+
+
+class CreateInvitedUserRequest(BaseModel):
+    email: EmailStr
+    name: Optional[str] = None
+
+
+class InvitedUserResponse(BaseModel):
+    id: str
+    email: str
+    status: prisma.enums.InvitedUserStatus
+    auth_user_id: Optional[str] = None
+    name: Optional[str] = None
+    tally_understanding: Optional[dict[str, Any]] = None
+    tally_status: prisma.enums.TallyComputationStatus
+    tally_computed_at: Optional[datetime] = None
+    tally_error: Optional[str] = None
+    created_at: datetime
+    updated_at: datetime
+
+    @classmethod
+    def from_record(cls, record: InvitedUserRecord) -> InvitedUserResponse:
+        return cls.model_validate(record.model_dump())
+
+
+class InvitedUsersResponse(BaseModel):
+    invited_users: list[InvitedUserResponse]
+    pagination: Pagination
+
+
+class BulkInvitedUserRowResponse(BaseModel):
+    row_number: int
+    email: Optional[str] = None
+    name: Optional[str] = None
+    status: Literal["CREATED", "SKIPPED", "ERROR"]
+    message: str
+    invited_user: Optional[InvitedUserResponse] = None
+
+
+class BulkInvitedUsersResponse(BaseModel):
+    created_count: int
+    skipped_count: int
+    error_count: int
+    results: list[BulkInvitedUserRowResponse]
+
+    @classmethod
+    def from_result(cls, result: BulkInvitedUsersResult) -> BulkInvitedUsersResponse:
+        return cls(
+            created_count=result.created_count,
+            skipped_count=result.skipped_count,
+            error_count=result.error_count,
+            results=[
+                BulkInvitedUserRowResponse(
+                    row_number=row.row_number,
+                    email=row.email,
+                    name=row.name,
+                    status=row.status,
+                    message=row.message,
+                    invited_user=(
+                        InvitedUserResponse.from_record(row.invited_user)
+                        if row.invited_user is not None
+                        else None
+                    ),
+                )
+                for row in result.results
+            ],
+        )

autogpt_platform/backend/backend/api/features/admin/store_admin_routes.py

@@ -24,14 +24,13 @@ router = fastapi.APIRouter(
 @router.get(
     "/listings",
     summary="Get Admin Listings History",
-    response_model=store_model.StoreListingsWithVersionsResponse,
 )
 async def get_admin_listings_with_versions(
     status: typing.Optional[prisma.enums.SubmissionStatus] = None,
     search: typing.Optional[str] = None,
     page: int = 1,
     page_size: int = 20,
-):
+) -> store_model.StoreListingsWithVersionsAdminViewResponse:
     """
     Get store listings with their version history for admins.
 
@@ -45,36 +44,26 @@ async def get_admin_listings_with_versions(
         page_size: Number of items per page
 
     Returns:
-        StoreListingsWithVersionsResponse with listings and their versions
+        Paginated listings with their versions
     """
-    try:
-        listings = await store_db.get_admin_listings_with_versions(
-            status=status,
-            search_query=search,
-            page=page,
-            page_size=page_size,
-        )
-        return listings
-    except Exception as e:
-        logger.exception("Error getting admin listings with versions: %s", e)
-        return fastapi.responses.JSONResponse(
-            status_code=500,
-            content={
-                "detail": "An error occurred while retrieving listings with versions"
-            },
-        )
+    listings = await store_db.get_admin_listings_with_versions(
+        status=status,
+        search_query=search,
+        page=page,
+        page_size=page_size,
+    )
+    return listings
 
 
 @router.post(
     "/submissions/{store_listing_version_id}/review",
     summary="Review Store Submission",
-    response_model=store_model.StoreSubmission,
 )
 async def review_submission(
     store_listing_version_id: str,
     request: store_model.ReviewSubmissionRequest,
     user_id: str = fastapi.Security(autogpt_libs.auth.get_user_id),
-):
+) -> store_model.StoreSubmissionAdminView:
     """
     Review a store listing submission.
 
@@ -84,31 +73,24 @@ async def review_submission(
         user_id: Authenticated admin user performing the review
 
     Returns:
-        StoreSubmission with updated review information
+        StoreSubmissionAdminView with updated review information
     """
-    try:
-        already_approved = await store_db.check_submission_already_approved(
-            store_listing_version_id=store_listing_version_id,
-        )
-        submission = await store_db.review_store_submission(
-            store_listing_version_id=store_listing_version_id,
-            is_approved=request.is_approved,
-            external_comments=request.comments,
-            internal_comments=request.internal_comments or "",
-            reviewer_id=user_id,
-        )
+    already_approved = await store_db.check_submission_already_approved(
+        store_listing_version_id=store_listing_version_id,
+    )
+    submission = await store_db.review_store_submission(
+        store_listing_version_id=store_listing_version_id,
+        is_approved=request.is_approved,
+        external_comments=request.comments,
+        internal_comments=request.internal_comments or "",
+        reviewer_id=user_id,
+    )
 
-        state_changed = already_approved != request.is_approved
-        # Clear caches when the request is approved as it updates what is shown on the store
-        if state_changed:
-            store_cache.clear_all_caches()
-        return submission
-    except Exception as e:
-        logger.exception("Error reviewing submission: %s", e)
-        return fastapi.responses.JSONResponse(
-            status_code=500,
-            content={"detail": "An error occurred while reviewing the submission"},
-        )
+    state_changed = already_approved != request.is_approved
+    # Clear caches whenever approval state changes, since store visibility can change
+    if state_changed:
+        store_cache.clear_all_caches()
+    return submission
 
 
 @router.get(

autogpt_platform/backend/backend/api/features/admin/user_admin_routes.py

@@ -0,0 +1,137 @@
+import logging
+import math
+
+from autogpt_libs.auth import get_user_id, requires_admin_user
+from fastapi import APIRouter, File, Query, Security, UploadFile
+
+from backend.data.invited_user import (
+    bulk_create_invited_users_from_file,
+    create_invited_user,
+    list_invited_users,
+    retry_invited_user_tally,
+    revoke_invited_user,
+)
+from backend.data.tally import mask_email
+from backend.util.models import Pagination
+
+from .model import (
+    BulkInvitedUsersResponse,
+    CreateInvitedUserRequest,
+    InvitedUserResponse,
+    InvitedUsersResponse,
+)
+
+logger = logging.getLogger(__name__)
+
+
+router = APIRouter(
+    prefix="/admin",
+    tags=["users", "admin"],
+    dependencies=[Security(requires_admin_user)],
+)
+
+
+@router.get(
+    "/invited-users",
+    response_model=InvitedUsersResponse,
+    summary="List Invited Users",
+)
+async def get_invited_users(
+    admin_user_id: str = Security(get_user_id),
+    page: int = Query(1, ge=1),
+    page_size: int = Query(50, ge=1, le=200),
+) -> InvitedUsersResponse:
+    logger.info("Admin user %s requested invited users", admin_user_id)
+    invited_users, total = await list_invited_users(page=page, page_size=page_size)
+    return InvitedUsersResponse(
+        invited_users=[InvitedUserResponse.from_record(iu) for iu in invited_users],
+        pagination=Pagination(
+            total_items=total,
+            total_pages=max(1, math.ceil(total / page_size)),
+            current_page=page,
+            page_size=page_size,
+        ),
+    )
+
+
+@router.post(
+    "/invited-users",
+    response_model=InvitedUserResponse,
+    summary="Create Invited User",
+)
+async def create_invited_user_route(
+    request: CreateInvitedUserRequest,
+    admin_user_id: str = Security(get_user_id),
+) -> InvitedUserResponse:
+    logger.info(
+        "Admin user %s creating invited user for %s",
+        admin_user_id,
+        mask_email(request.email),
+    )
+    invited_user = await create_invited_user(request.email, request.name)
+    logger.info(
+        "Admin user %s created invited user %s",
+        admin_user_id,
+        invited_user.id,
+    )
+    return InvitedUserResponse.from_record(invited_user)
+
+
+@router.post(
+    "/invited-users/bulk",
+    response_model=BulkInvitedUsersResponse,
+    summary="Bulk Create Invited Users",
+    operation_id="postV2BulkCreateInvitedUsers",
+)
+async def bulk_create_invited_users_route(
+    file: UploadFile = File(...),
+    admin_user_id: str = Security(get_user_id),
+) -> BulkInvitedUsersResponse:
+    logger.info(
+        "Admin user %s bulk invited users from %s",
+        admin_user_id,
+        file.filename or "<unnamed>",
+    )
+    content = await file.read()
+    result = await bulk_create_invited_users_from_file(file.filename, content)
+    return BulkInvitedUsersResponse.from_result(result)
+
+
+@router.post(
+    "/invited-users/{invited_user_id}/revoke",
+    response_model=InvitedUserResponse,
+    summary="Revoke Invited User",
+)
+async def revoke_invited_user_route(
+    invited_user_id: str,
+    admin_user_id: str = Security(get_user_id),
+) -> InvitedUserResponse:
+    logger.info(
+        "Admin user %s revoking invited user %s", admin_user_id, invited_user_id
+    )
+    invited_user = await revoke_invited_user(invited_user_id)
+    logger.info("Admin user %s revoked invited user %s", admin_user_id, invited_user_id)
+    return InvitedUserResponse.from_record(invited_user)
+
+
+@router.post(
+    "/invited-users/{invited_user_id}/retry-tally",
+    response_model=InvitedUserResponse,
+    summary="Retry Invited User Tally",
+)
+async def retry_invited_user_tally_route(
+    invited_user_id: str,
+    admin_user_id: str = Security(get_user_id),
+) -> InvitedUserResponse:
+    logger.info(
+        "Admin user %s retrying Tally seed for invited user %s",
+        admin_user_id,
+        invited_user_id,
+    )
+    invited_user = await retry_invited_user_tally(invited_user_id)
+    logger.info(
+        "Admin user %s retried Tally seed for invited user %s",
+        admin_user_id,
+        invited_user_id,
+    )
+    return InvitedUserResponse.from_record(invited_user)

autogpt_platform/backend/backend/api/features/admin/user_admin_routes_test.py

@@ -0,0 +1,168 @@
+from datetime import datetime, timezone
+from unittest.mock import AsyncMock
+
+import fastapi
+import fastapi.testclient
+import prisma.enums
+import pytest
+import pytest_mock
+from autogpt_libs.auth.jwt_utils import get_jwt_payload
+
+from backend.data.invited_user import (
+    BulkInvitedUserRowResult,
+    BulkInvitedUsersResult,
+    InvitedUserRecord,
+)
+
+from .user_admin_routes import router as user_admin_router
+
+app = fastapi.FastAPI()
+app.include_router(user_admin_router)
+
+client = fastapi.testclient.TestClient(app)
+
+
+@pytest.fixture(autouse=True)
+def setup_app_admin_auth(mock_jwt_admin):
+    app.dependency_overrides[get_jwt_payload] = mock_jwt_admin["get_jwt_payload"]
+    yield
+    app.dependency_overrides.clear()
+
+
+def _sample_invited_user() -> InvitedUserRecord:
+    now = datetime.now(timezone.utc)
+    return InvitedUserRecord(
+        id="invite-1",
+        email="invited@example.com",
+        status=prisma.enums.InvitedUserStatus.INVITED,
+        auth_user_id=None,
+        name="Invited User",
+        tally_understanding=None,
+        tally_status=prisma.enums.TallyComputationStatus.PENDING,
+        tally_computed_at=None,
+        tally_error=None,
+        created_at=now,
+        updated_at=now,
+    )
+
+
+def _sample_bulk_invited_users_result() -> BulkInvitedUsersResult:
+    return BulkInvitedUsersResult(
+        created_count=1,
+        skipped_count=1,
+        error_count=0,
+        results=[
+            BulkInvitedUserRowResult(
+                row_number=1,
+                email="invited@example.com",
+                name=None,
+                status="CREATED",
+                message="Invite created",
+                invited_user=_sample_invited_user(),
+            ),
+            BulkInvitedUserRowResult(
+                row_number=2,
+                email="duplicate@example.com",
+                name=None,
+                status="SKIPPED",
+                message="An invited user with this email already exists",
+                invited_user=None,
+            ),
+        ],
+    )
+
+
+def test_get_invited_users(
+    mocker: pytest_mock.MockerFixture,
+) -> None:
+    mocker.patch(
+        "backend.api.features.admin.user_admin_routes.list_invited_users",
+        AsyncMock(return_value=([_sample_invited_user()], 1)),
+    )
+
+    response = client.get("/admin/invited-users")
+
+    assert response.status_code == 200
+    data = response.json()
+    assert len(data["invited_users"]) == 1
+    assert data["invited_users"][0]["email"] == "invited@example.com"
+    assert data["invited_users"][0]["status"] == "INVITED"
+    assert data["pagination"]["total_items"] == 1
+    assert data["pagination"]["current_page"] == 1
+    assert data["pagination"]["page_size"] == 50
+
+
+def test_create_invited_user(
+    mocker: pytest_mock.MockerFixture,
+) -> None:
+    mocker.patch(
+        "backend.api.features.admin.user_admin_routes.create_invited_user",
+        AsyncMock(return_value=_sample_invited_user()),
+    )
+
+    response = client.post(
+        "/admin/invited-users",
+        json={"email": "invited@example.com", "name": "Invited User"},
+    )
+
+    assert response.status_code == 200
+    data = response.json()
+    assert data["email"] == "invited@example.com"
+    assert data["name"] == "Invited User"
+
+
+def test_bulk_create_invited_users(
+    mocker: pytest_mock.MockerFixture,
+) -> None:
+    mocker.patch(
+        "backend.api.features.admin.user_admin_routes.bulk_create_invited_users_from_file",
+        AsyncMock(return_value=_sample_bulk_invited_users_result()),
+    )
+
+    response = client.post(
+        "/admin/invited-users/bulk",
+        files={
+            "file": ("invites.txt", b"invited@example.com\nduplicate@example.com\n")
+        },
+    )
+
+    assert response.status_code == 200
+    data = response.json()
+    assert data["created_count"] == 1
+    assert data["skipped_count"] == 1
+    assert data["results"][0]["status"] == "CREATED"
+    assert data["results"][1]["status"] == "SKIPPED"
+
+
+def test_revoke_invited_user(
+    mocker: pytest_mock.MockerFixture,
+) -> None:
+    revoked = _sample_invited_user().model_copy(
+        update={"status": prisma.enums.InvitedUserStatus.REVOKED}
+    )
+    mocker.patch(
+        "backend.api.features.admin.user_admin_routes.revoke_invited_user",
+        AsyncMock(return_value=revoked),
+    )
+
+    response = client.post("/admin/invited-users/invite-1/revoke")
+
+    assert response.status_code == 200
+    assert response.json()["status"] == "REVOKED"
+
+
+def test_retry_invited_user_tally(
+    mocker: pytest_mock.MockerFixture,
+) -> None:
+    retried = _sample_invited_user().model_copy(
+        update={"tally_status": prisma.enums.TallyComputationStatus.RUNNING}
+    )
+    mocker.patch(
+        "backend.api.features.admin.user_admin_routes.retry_invited_user_tally",
+        AsyncMock(return_value=retried),
+    )
+
+    response = client.post("/admin/invited-users/invite-1/retry-tally")
+
+    assert response.status_code == 200
+    assert response.json()["tally_status"] == "RUNNING"

autogpt_platform/backend/backend/api/features/builder/db.py

@@ -1,15 +1,17 @@
 import logging
 from dataclasses import dataclass
-from datetime import datetime, timedelta, timezone
 from difflib import SequenceMatcher
-from typing import Sequence
+from typing import Any, Sequence, get_args, get_origin
 
 import prisma
+from prisma.enums import ContentType
+from prisma.models import mv_suggested_blocks
 
 import backend.api.features.library.db as library_db
 import backend.api.features.library.model as library_model
 import backend.api.features.store.db as store_db
 import backend.api.features.store.model as store_model
+from backend.api.features.store.hybrid_search import unified_hybrid_search
 from backend.blocks import load_all_blocks
 from backend.blocks._base import (
     AnyBlockSchema,
@@ -19,7 +21,6 @@ from backend.blocks._base import (
     BlockType,
 )
 from backend.blocks.llm import LlmModel
-from backend.data.db import query_raw_with_schema
 from backend.integrations.providers import ProviderName
 from backend.util.cache import cached
 from backend.util.models import Pagination
@@ -42,6 +43,16 @@ MAX_LIBRARY_AGENT_RESULTS = 100
 MAX_MARKETPLACE_AGENT_RESULTS = 100
 MIN_SCORE_FOR_FILTERED_RESULTS = 10.0
 
+# Boost blocks over marketplace agents in search results
+BLOCK_SCORE_BOOST = 50.0
+
+# Block IDs to exclude from search results
+EXCLUDED_BLOCK_IDS = frozenset(
+    {
+        "e189baac-8c20-45a1-94a7-55177ea42565",  # AgentExecutorBlock
+    }
+)
+
 SearchResultItem = BlockInfo | library_model.LibraryAgent | store_model.StoreAgent
 
 
@@ -64,8 +75,8 @@ def get_block_categories(category_blocks: int = 3) -> list[BlockCategoryResponse
 
     for block_type in load_all_blocks().values():
         block: AnyBlockSchema = block_type()
-        # Skip disabled blocks
-        if block.disabled:
+        # Skip disabled and excluded blocks
+        if block.disabled or block.id in EXCLUDED_BLOCK_IDS:
             continue
         # Skip blocks that don't have categories (all should have at least one)
         if not block.categories:
@@ -116,6 +127,9 @@ def get_blocks(
         # Skip disabled blocks
         if block.disabled:
             continue
+        # Skip excluded blocks
+        if block.id in EXCLUDED_BLOCK_IDS:
+            continue
         # Skip blocks that don't match the category
         if category and category not in {c.name.lower() for c in block.categories}:
             continue
@@ -255,14 +269,25 @@ async def _build_cached_search_results(
         "my_agents": 0,
     }
 
-    block_results, block_total, integration_total = _collect_block_results(
-        normalized_query=normalized_query,
-        include_blocks=include_blocks,
-        include_integrations=include_integrations,
-    )
-    scored_items.extend(block_results)
-    total_items["blocks"] = block_total
-    total_items["integrations"] = integration_total
+    # Use hybrid search when query is present, otherwise list all blocks
+    if (include_blocks or include_integrations) and normalized_query:
+        block_results, block_total, integration_total = await _hybrid_search_blocks(
+            query=search_query,
+            include_blocks=include_blocks,
+            include_integrations=include_integrations,
+        )
+        scored_items.extend(block_results)
+        total_items["blocks"] = block_total
+        total_items["integrations"] = integration_total
+    elif include_blocks or include_integrations:
+        # No query - list all blocks using in-memory approach
+        block_results, block_total, integration_total = _collect_block_results(
+            include_blocks=include_blocks,
+            include_integrations=include_integrations,
+        )
+        scored_items.extend(block_results)
+        total_items["blocks"] = block_total
+        total_items["integrations"] = integration_total
 
     if include_library_agents:
         library_response = await library_db.list_library_agents(
@@ -307,10 +332,14 @@ async def _build_cached_search_results(
 
 def _collect_block_results(
     *,
-    normalized_query: str,
     include_blocks: bool,
     include_integrations: bool,
 ) -> tuple[list[_ScoredItem], int, int]:
+    """
+    Collect all blocks for listing (no search query).
+
+    All blocks get BLOCK_SCORE_BOOST to prioritize them over marketplace agents.
+    """
     results: list[_ScoredItem] = []
     block_count = 0
     integration_count = 0
@@ -323,6 +352,10 @@ def _collect_block_results(
         if block.disabled:
             continue
 
+        # Skip excluded blocks
+        if block.id in EXCLUDED_BLOCK_IDS:
+            continue
+
         block_info = block.get_info()
         credentials = list(block.input_schema.get_credentials_fields().values())
         is_integration = len(credentials) > 0
@@ -332,10 +365,6 @@ def _collect_block_results(
         if not is_integration and not include_blocks:
             continue
 
-        score = _score_block(block, block_info, normalized_query)
-        if not _should_include_item(score, normalized_query):
-            continue
-
         filter_type: FilterType = "integrations" if is_integration else "blocks"
         if is_integration:
             integration_count += 1
@@ -346,8 +375,122 @@ def _collect_block_results(
             _ScoredItem(
                 item=block_info,
                 filter_type=filter_type,
-                score=score,
-                sort_key=_get_item_name(block_info),
+                score=BLOCK_SCORE_BOOST,
+                sort_key=block_info.name.lower(),
+            )
+        )
+
+    return results, block_count, integration_count
+
+
+async def _hybrid_search_blocks(
+    *,
+    query: str,
+    include_blocks: bool,
+    include_integrations: bool,
+) -> tuple[list[_ScoredItem], int, int]:
+    """
+    Search blocks using hybrid search with builder-specific filtering.
+
+    Uses unified_hybrid_search for semantic + lexical search, then applies
+    post-filtering for block/integration types and scoring adjustments.
+
+    Scoring:
+        - Base: hybrid relevance score (0-1) scaled to 0-100, plus BLOCK_SCORE_BOOST
+          to prioritize blocks over marketplace agents in combined results
+        - +30 for exact name match, +15 for prefix name match
+        - +20 if the block has an LlmModel field and the query matches an LLM model name
+
+    Args:
+        query: The search query string
+        include_blocks: Whether to include regular blocks
+        include_integrations: Whether to include integration blocks
+
+    Returns:
+        Tuple of (scored_items, block_count, integration_count)
+    """
+    results: list[_ScoredItem] = []
+    block_count = 0
+    integration_count = 0
+
+    if not include_blocks and not include_integrations:
+        return results, block_count, integration_count
+
+    normalized_query = query.strip().lower()
+
+    # Fetch more results to account for post-filtering
+    search_results, _ = await unified_hybrid_search(
+        query=query,
+        content_types=[ContentType.BLOCK],
+        page=1,
+        page_size=150,
+        min_score=0.10,
+    )
+
+    # Load all blocks for getting BlockInfo
+    all_blocks = load_all_blocks()
+
+    for result in search_results:
+        block_id = result["content_id"]
+
+        # Skip excluded blocks
+        if block_id in EXCLUDED_BLOCK_IDS:
+            continue
+
+        metadata = result.get("metadata", {})
+        hybrid_score = result.get("relevance", 0.0)
+
+        # Get the actual block class
+        if block_id not in all_blocks:
+            continue
+
+        block_cls = all_blocks[block_id]
+        block: AnyBlockSchema = block_cls()
+
+        if block.disabled:
+            continue
+
+        # Check block/integration filter using metadata
+        is_integration = metadata.get("is_integration", False)
+
+        if is_integration and not include_integrations:
+            continue
+        if not is_integration and not include_blocks:
+            continue
+
+        # Get block info
+        block_info = block.get_info()
+
+        # Calculate final score: scale hybrid score and add builder-specific bonuses
+        # Hybrid scores are 0-1, builder scores were 0-200+
+        # Add BLOCK_SCORE_BOOST to prioritize blocks over marketplace agents
+        final_score = hybrid_score * 100 + BLOCK_SCORE_BOOST
+
+        # Add LLM model match bonus
+        has_llm_field = metadata.get("has_llm_model_field", False)
+        if has_llm_field and _matches_llm_model(block.input_schema, normalized_query):
+            final_score += 20
+
+        # Add exact/prefix match bonus for deterministic tie-breaking
+        name = block_info.name.lower()
+        if name == normalized_query:
+            final_score += 30
+        elif name.startswith(normalized_query):
+            final_score += 15
+
+        # Track counts
+        filter_type: FilterType = "integrations" if is_integration else "blocks"
+        if is_integration:
+            integration_count += 1
+        else:
+            block_count += 1
+
+        results.append(
+            _ScoredItem(
+                item=block_info,
+                filter_type=filter_type,
+                score=final_score,
+                sort_key=name,
             )
         )
 
@@ -472,6 +615,8 @@ async def _get_static_counts():
         block: AnyBlockSchema = block_type()
         if block.disabled:
             continue
+        if block.id in EXCLUDED_BLOCK_IDS:
+            continue
 
         all_blocks += 1
 
@@ -498,47 +643,25 @@ async def _get_static_counts():
     }
 
 
+def _contains_type(annotation: Any, target: type) -> bool:
+    """Check if an annotation is or contains the target type (handles Optional/Union/Annotated)."""
+    if annotation is target:
+        return True
+    origin = get_origin(annotation)
+    if origin is None:
+        return False
+    return any(_contains_type(arg, target) for arg in get_args(annotation))
+
+
 def _matches_llm_model(schema_cls: type[BlockSchema], query: str) -> bool:
     for field in schema_cls.model_fields.values():
-        if field.annotation == LlmModel:
+        if _contains_type(field.annotation, LlmModel):
             # Check if query matches any value in llm_models
             if any(query in name for name in llm_models):
                 return True
     return False
 
 
-def _score_block(
-    block: AnyBlockSchema,
-    block_info: BlockInfo,
-    normalized_query: str,
-) -> float:
-    if not normalized_query:
-        return 0.0
-
-    name = block_info.name.lower()
-    description = block_info.description.lower()
-    score = _score_primary_fields(name, description, normalized_query)
-
-    category_text = " ".join(
-        category.get("category", "").lower() for category in block_info.categories
-    )
-    score += _score_additional_field(category_text, normalized_query, 12, 6)
-
-    credentials_info = block.input_schema.get_credentials_fields_info().values()
-    provider_names = [
-        provider.value.lower()
-        for info in credentials_info
-        for provider in info.provider
-    ]
-    provider_text = " ".join(provider_names)
-    score += _score_additional_field(provider_text, normalized_query, 15, 6)
-
-    if _matches_llm_model(block.input_schema, normalized_query):
-        score += 20
-
-    return score
-
-
 def _score_library_agent(
     agent: library_model.LibraryAgent,
     normalized_query: str,
@@ -645,31 +768,20 @@ def _get_all_providers() -> dict[ProviderName, Provider]:
     return providers
 
 
-@cached(ttl_seconds=3600)
+@cached(ttl_seconds=3600, shared_cache=True)
 async def get_suggested_blocks(count: int = 5) -> list[BlockInfo]:
-    suggested_blocks = []
-    # Sum the number of executions for each block type
-    # Prisma cannot group by nested relations, so we do a raw query
-    # Calculate the cutoff timestamp
-    timestamp_threshold = datetime.now(timezone.utc) - timedelta(days=30)
+    """Return the most-executed blocks from the last 14 days.
 
-    results = await query_raw_with_schema(
-        """
-        SELECT
-            agent_node."agentBlockId" AS block_id,
-            COUNT(execution.id) AS execution_count
-        FROM {schema_prefix}"AgentNodeExecution" execution
-        JOIN {schema_prefix}"AgentNode" agent_node ON execution."agentNodeId" = agent_node.id
-        WHERE execution."endedTime" >= $1::timestamp
-        GROUP BY agent_node."agentBlockId"
-        ORDER BY execution_count DESC;
-        """,
-        timestamp_threshold,
-    )
+    Queries the mv_suggested_blocks materialized view (refreshed hourly via pg_cron)
+    and returns the top `count` blocks sorted by execution count, excluding
+    Input/Output/Agent block types and blocks in EXCLUDED_BLOCK_IDS.
+    """
+    results = await mv_suggested_blocks.prisma().find_many()
 
     # Get the top blocks based on execution count
-    # But ignore Input and Output blocks
+    # But ignore Input, Output, Agent, and excluded blocks
     blocks: list[tuple[BlockInfo, int]] = []
+    execution_counts = {row.block_id: row.execution_count for row in results}
 
     for block_type in load_all_blocks().values():
         block: AnyBlockSchema = block_type()
@@ -679,11 +791,9 @@ async def get_suggested_blocks(count: int = 5) -> list[BlockInfo]:
             BlockType.AGENT,
         ):
             continue
-        # Find the execution count for this block
-        execution_count = next(
-            (row["execution_count"] for row in results if row["block_id"] == block.id),
-            0,
-        )
+        if block.id in EXCLUDED_BLOCK_IDS:
+            continue
+        execution_count = execution_counts.get(block.id, 0)
         blocks.append((block.get_info(), execution_count))
     # Sort blocks by execution count
     blocks.sort(key=lambda x: x[1], reverse=True)

autogpt_platform/backend/backend/api/features/builder/model.py

@@ -27,7 +27,6 @@ class SearchEntry(BaseModel):
 
 # Suggestions
 class SuggestionsResponse(BaseModel):
-    otto_suggestions: list[str]
     recent_searches: list[SearchEntry]
     providers: list[ProviderName]
     top_blocks: list[BlockInfo]

autogpt_platform/backend/backend/api/features/builder/routes.py

@@ -1,5 +1,5 @@
 import logging
-from typing import Annotated, Sequence
+from typing import Annotated, Sequence, cast, get_args
 
 import fastapi
 from autogpt_libs.auth.dependencies import get_user_id, requires_user
@@ -10,6 +10,8 @@ from backend.util.models import Pagination
 from . import db as builder_db
 from . import model as builder_model
 
+VALID_FILTER_VALUES = get_args(builder_model.FilterType)
+
 logger = logging.getLogger(__name__)
 
 router = fastapi.APIRouter(
@@ -49,11 +51,6 @@ async def get_suggestions(
     Get all suggestions for the Blocks Menu.
     """
     return builder_model.SuggestionsResponse(
-        otto_suggestions=[
-            "What blocks do I need to get started?",
-            "Help me create a list",
-            "Help me feed my data to Google Maps",
-        ],
         recent_searches=await builder_db.get_recent_searches(user_id),
         providers=[
             ProviderName.TWITTER,
@@ -151,7 +148,7 @@ async def get_providers(
 async def search(
     user_id: Annotated[str, fastapi.Security(get_user_id)],
     search_query: Annotated[str | None, fastapi.Query()] = None,
-    filter: Annotated[list[builder_model.FilterType] | None, fastapi.Query()] = None,
+    filter: Annotated[str | None, fastapi.Query()] = None,
     search_id: Annotated[str | None, fastapi.Query()] = None,
     by_creator: Annotated[list[str] | None, fastapi.Query()] = None,
     page: Annotated[int, fastapi.Query()] = 1,
@@ -160,9 +157,20 @@ async def search(
     """
     Search for blocks (including integrations), marketplace agents, and user library agents.
     """
-    # If no filters are provided, then we will return all types
-    if not filter:
-        filter = [
+    # Parse and validate filter parameter
+    filters: list[builder_model.FilterType]
+    if filter:
+        filter_values = [f.strip() for f in filter.split(",")]
+        invalid_filters = [f for f in filter_values if f not in VALID_FILTER_VALUES]
+        if invalid_filters:
+            raise fastapi.HTTPException(
+                status_code=400,
+                detail=f"Invalid filter value(s): {', '.join(invalid_filters)}. "
+                f"Valid values are: {', '.join(VALID_FILTER_VALUES)}",
+            )
+        filters = cast(list[builder_model.FilterType], filter_values)
+    else:
+        filters = [
             "blocks",
             "integrations",
             "marketplace_agents",
@@ -174,7 +182,7 @@ async def search(
     cached_results = await builder_db.get_sorted_search_results(
         user_id=user_id,
         search_query=search_query,
-        filters=filter,
+        filters=filters,
         by_creator=by_creator,
     )
 
@@ -196,7 +204,7 @@ async def search(
         user_id,
         builder_model.SearchEntry(
             search_query=search_query,
-            filter=filter,
+            filter=filters,
             by_creator=by_creator,
             search_id=search_id,
         ),

autogpt_platform/backend/backend/api/features/chat/completion_consumer.py

@@ -1,368 +0,0 @@
-"""Redis Streams consumer for operation completion messages.
-
-This module provides a consumer (ChatCompletionConsumer) that listens for
-completion notifications (OperationCompleteMessage) from external services
-(like Agent Generator) and triggers the appropriate stream registry and
-chat service updates via process_operation_success/process_operation_failure.
-
-Why Redis Streams instead of RabbitMQ?
---------------------------------------
-While the project typically uses RabbitMQ for async task queues (e.g., execution
-queue), Redis Streams was chosen for chat completion notifications because:
-
-1. **Unified Infrastructure**: The SSE reconnection feature already uses Redis
-   Streams (via stream_registry) for message persistence and replay. Using Redis
-   Streams for completion notifications keeps all chat streaming infrastructure
-   in one system, simplifying operations and reducing cross-system coordination.
-
-2. **Message Replay**: Redis Streams support XREAD with arbitrary message IDs,
-   allowing consumers to replay missed messages after reconnection. This aligns
-   with the SSE reconnection pattern where clients can resume from last_message_id.
-
-3. **Consumer Groups with XAUTOCLAIM**: Redis consumer groups provide automatic
-   load balancing across pods with explicit message claiming (XAUTOCLAIM) for
-   recovering from dead consumers - ideal for the completion callback pattern.
-
-4. **Lower Latency**: For real-time SSE updates, Redis (already in-memory for
-   stream_registry) provides lower latency than an additional RabbitMQ hop.
-
-5. **Atomicity with Task State**: Completion processing often needs to update
-   task metadata stored in Redis. Keeping both in Redis enables simpler
-   transactional semantics without distributed coordination.
-
-The consumer uses Redis Streams with consumer groups for reliable message
-processing across multiple platform pods, with XAUTOCLAIM for reclaiming
-stale pending messages from dead consumers.
-"""
-
-import asyncio
-import logging
-import os
-import uuid
-from typing import Any
-
-import orjson
-from prisma import Prisma
-from pydantic import BaseModel
-from redis.exceptions import ResponseError
-
-from backend.data.redis_client import get_redis_async
-
-from . import stream_registry
-from .completion_handler import process_operation_failure, process_operation_success
-from .config import ChatConfig
-
-logger = logging.getLogger(__name__)
-config = ChatConfig()
-
-
-class OperationCompleteMessage(BaseModel):
-    """Message format for operation completion notifications."""
-
-    operation_id: str
-    task_id: str
-    success: bool
-    result: dict | str | None = None
-    error: str | None = None
-
-
-class ChatCompletionConsumer:
-    """Consumer for chat operation completion messages from Redis Streams.
-
-    This consumer initializes its own Prisma client in start() to ensure
-    database operations work correctly within this async context.
-
-    Uses Redis consumer groups to allow multiple platform pods to consume
-    messages reliably with automatic redelivery on failure.
-    """
-
-    def __init__(self):
-        self._consumer_task: asyncio.Task | None = None
-        self._running = False
-        self._prisma: Prisma | None = None
-        self._consumer_name = f"consumer-{uuid.uuid4().hex[:8]}"
-
-    async def start(self) -> None:
-        """Start the completion consumer."""
-        if self._running:
-            logger.warning("Completion consumer already running")
-            return
-
-        # Create consumer group if it doesn't exist
-        try:
-            redis = await get_redis_async()
-            await redis.xgroup_create(
-                config.stream_completion_name,
-                config.stream_consumer_group,
-                id="0",
-                mkstream=True,
-            )
-            logger.info(
-                f"Created consumer group '{config.stream_consumer_group}' "
-                f"on stream '{config.stream_completion_name}'"
-            )
-        except ResponseError as e:
-            if "BUSYGROUP" in str(e):
-                logger.debug(
-                    f"Consumer group '{config.stream_consumer_group}' already exists"
-                )
-            else:
-                raise
-
-        self._running = True
-        self._consumer_task = asyncio.create_task(self._consume_messages())
-        logger.info(
-            f"Chat completion consumer started (consumer: {self._consumer_name})"
-        )
-
-    async def _ensure_prisma(self) -> Prisma:
-        """Lazily initialize Prisma client on first use."""
-        if self._prisma is None:
-            database_url = os.getenv("DATABASE_URL", "postgresql://localhost:5432")
-            self._prisma = Prisma(datasource={"url": database_url})
-            await self._prisma.connect()
-            logger.info("[COMPLETION] Consumer Prisma client connected (lazy init)")
-        return self._prisma
-
-    async def stop(self) -> None:
-        """Stop the completion consumer."""
-        self._running = False
-
-        if self._consumer_task:
-            self._consumer_task.cancel()
-            try:
-                await self._consumer_task
-            except asyncio.CancelledError:
-                pass
-            self._consumer_task = None
-
-        if self._prisma:
-            await self._prisma.disconnect()
-            self._prisma = None
-            logger.info("[COMPLETION] Consumer Prisma client disconnected")
-
-        logger.info("Chat completion consumer stopped")
-
-    async def _consume_messages(self) -> None:
-        """Main message consumption loop with retry logic."""
-        max_retries = 10
-        retry_delay = 5  # seconds
-        retry_count = 0
-        block_timeout = 5000  # milliseconds
-
-        while self._running and retry_count < max_retries:
-            try:
-                redis = await get_redis_async()
-
-                # Reset retry count on successful connection
-                retry_count = 0
-
-                while self._running:
-                    # First, claim any stale pending messages from dead consumers
-                    # Redis does NOT auto-redeliver pending messages; we must explicitly
-                    # claim them using XAUTOCLAIM
-                    try:
-                        claimed_result = await redis.xautoclaim(
-                            name=config.stream_completion_name,
-                            groupname=config.stream_consumer_group,
-                            consumername=self._consumer_name,
-                            min_idle_time=config.stream_claim_min_idle_ms,
-                            start_id="0-0",
-                            count=10,
-                        )
-                        # xautoclaim returns: (next_start_id, [(id, data), ...], [deleted_ids])
-                        if claimed_result and len(claimed_result) >= 2:
-                            claimed_entries = claimed_result[1]
-                            if claimed_entries:
-                                logger.info(
-                                    f"Claimed {len(claimed_entries)} stale pending messages"
-                                )
-                                for entry_id, data in claimed_entries:
-                                    if not self._running:
-                                        return
-                                    await self._process_entry(redis, entry_id, data)
-                    except Exception as e:
-                        logger.warning(f"XAUTOCLAIM failed (non-fatal): {e}")
-
-                    # Read new messages from the stream
-                    messages = await redis.xreadgroup(
-                        groupname=config.stream_consumer_group,
-                        consumername=self._consumer_name,
-                        streams={config.stream_completion_name: ">"},
-                        block=block_timeout,
-                        count=10,
-                    )
-
-                    if not messages:
-                        continue
-
-                    for stream_name, entries in messages:
-                        for entry_id, data in entries:
-                            if not self._running:
-                                return
-                            await self._process_entry(redis, entry_id, data)
-
-            except asyncio.CancelledError:
-                logger.info("Consumer cancelled")
-                return
-            except Exception as e:
-                retry_count += 1
-                logger.error(
-                    f"Consumer error (retry {retry_count}/{max_retries}): {e}",
-                    exc_info=True,
-                )
-                if self._running and retry_count < max_retries:
-                    await asyncio.sleep(retry_delay)
-                else:
-                    logger.error("Max retries reached, stopping consumer")
-                    return
-
-    async def _process_entry(
-        self, redis: Any, entry_id: str, data: dict[str, Any]
-    ) -> None:
-        """Process a single stream entry and acknowledge it on success.
-
-        Args:
-            redis: Redis client connection
-            entry_id: The stream entry ID
-            data: The entry data dict
-        """
-        try:
-            # Handle the message
-            message_data = data.get("data")
-            if message_data:
-                await self._handle_message(
-                    message_data.encode()
-                    if isinstance(message_data, str)
-                    else message_data
-                )
-
-            # Acknowledge the message after successful processing
-            await redis.xack(
-                config.stream_completion_name,
-                config.stream_consumer_group,
-                entry_id,
-            )
-        except Exception as e:
-            logger.error(
-                f"Error processing completion message {entry_id}: {e}",
-                exc_info=True,
-            )
-            # Message remains in pending state and will be claimed by
-            # XAUTOCLAIM after min_idle_time expires
-
-    async def _handle_message(self, body: bytes) -> None:
-        """Handle a completion message using our own Prisma client."""
-        try:
-            data = orjson.loads(body)
-            message = OperationCompleteMessage(**data)
-        except Exception as e:
-            logger.error(f"Failed to parse completion message: {e}")
-            return
-
-        logger.info(
-            f"[COMPLETION] Received completion for operation {message.operation_id} "
-            f"(task_id={message.task_id}, success={message.success})"
-        )
-
-        # Find task in registry
-        task = await stream_registry.find_task_by_operation_id(message.operation_id)
-        if task is None:
-            task = await stream_registry.get_task(message.task_id)
-
-        if task is None:
-            logger.warning(
-                f"[COMPLETION] Task not found for operation {message.operation_id} "
-                f"(task_id={message.task_id})"
-            )
-            return
-
-        logger.info(
-            f"[COMPLETION] Found task: task_id={task.task_id}, "
-            f"session_id={task.session_id}, tool_call_id={task.tool_call_id}"
-        )
-
-        # Guard against empty task fields
-        if not task.task_id or not task.session_id or not task.tool_call_id:
-            logger.error(
-                f"[COMPLETION] Task has empty critical fields! "
-                f"task_id={task.task_id!r}, session_id={task.session_id!r}, "
-                f"tool_call_id={task.tool_call_id!r}"
-            )
-            return
-
-        if message.success:
-            await self._handle_success(task, message)
-        else:
-            await self._handle_failure(task, message)
-
-    async def _handle_success(
-        self,
-        task: stream_registry.ActiveTask,
-        message: OperationCompleteMessage,
-    ) -> None:
-        """Handle successful operation completion."""
-        prisma = await self._ensure_prisma()
-        await process_operation_success(task, message.result, prisma)
-
-    async def _handle_failure(
-        self,
-        task: stream_registry.ActiveTask,
-        message: OperationCompleteMessage,
-    ) -> None:
-        """Handle failed operation completion."""
-        prisma = await self._ensure_prisma()
-        await process_operation_failure(task, message.error, prisma)
-
-
-# Module-level consumer instance
-_consumer: ChatCompletionConsumer | None = None
-
-
-async def start_completion_consumer() -> None:
-    """Start the global completion consumer."""
-    global _consumer
-    if _consumer is None:
-        _consumer = ChatCompletionConsumer()
-    await _consumer.start()
-
-
-async def stop_completion_consumer() -> None:
-    """Stop the global completion consumer."""
-    global _consumer
-    if _consumer:
-        await _consumer.stop()
-        _consumer = None
-
-
-async def publish_operation_complete(
-    operation_id: str,
-    task_id: str,
-    success: bool,
-    result: dict | str | None = None,
-    error: str | None = None,
-) -> None:
-    """Publish an operation completion message to Redis Streams.
-
-    Args:
-        operation_id: The operation ID that completed.
-        task_id: The task ID associated with the operation.
-        success: Whether the operation succeeded.
-        result: The result data (for success).
-        error: The error message (for failure).
-    """
-    message = OperationCompleteMessage(
-        operation_id=operation_id,
-        task_id=task_id,
-        success=success,
-        result=result,
-        error=error,
-    )
-
-    redis = await get_redis_async()
-    await redis.xadd(
-        config.stream_completion_name,
-        {"data": message.model_dump_json()},
-        maxlen=config.stream_max_length,
-    )
-    logger.info(f"Published completion for operation {operation_id}")

autogpt_platform/backend/backend/api/features/chat/completion_handler.py

@@ -1,344 +0,0 @@
-"""Shared completion handling for operation success and failure.
-
-This module provides common logic for handling operation completion from both:
-- The Redis Streams consumer (completion_consumer.py)
-- The HTTP webhook endpoint (routes.py)
-"""
-
-import logging
-from typing import Any
-
-import orjson
-from prisma import Prisma
-
-from . import service as chat_service
-from . import stream_registry
-from .response_model import StreamError, StreamToolOutputAvailable
-from .tools.models import ErrorResponse
-
-logger = logging.getLogger(__name__)
-
-# Tools that produce agent_json that needs to be saved to library
-AGENT_GENERATION_TOOLS = {"create_agent", "edit_agent"}
-
-# Keys that should be stripped from agent_json when returning in error responses
-SENSITIVE_KEYS = frozenset(
-    {
-        "api_key",
-        "apikey",
-        "api_secret",
-        "password",
-        "secret",
-        "credentials",
-        "credential",
-        "token",
-        "access_token",
-        "refresh_token",
-        "private_key",
-        "privatekey",
-        "auth",
-        "authorization",
-    }
-)
-
-
-def _sanitize_agent_json(obj: Any) -> Any:
-    """Recursively sanitize agent_json by removing sensitive keys.
-
-    Args:
-        obj: The object to sanitize (dict, list, or primitive)
-
-    Returns:
-        Sanitized copy with sensitive keys removed/redacted
-    """
-    if isinstance(obj, dict):
-        return {
-            k: "[REDACTED]" if k.lower() in SENSITIVE_KEYS else _sanitize_agent_json(v)
-            for k, v in obj.items()
-        }
-    elif isinstance(obj, list):
-        return [_sanitize_agent_json(item) for item in obj]
-    else:
-        return obj
-
-
-class ToolMessageUpdateError(Exception):
-    """Raised when updating a tool message in the database fails."""
-
-    pass
-
-
-async def _update_tool_message(
-    session_id: str,
-    tool_call_id: str,
-    content: str,
-    prisma_client: Prisma | None,
-) -> None:
-    """Update tool message in database.
-
-    Args:
-        session_id: The session ID
-        tool_call_id: The tool call ID to update
-        content: The new content for the message
-        prisma_client: Optional Prisma client. If None, uses chat_service.
-
-    Raises:
-        ToolMessageUpdateError: If the database update fails. The caller should
-            handle this to avoid marking the task as completed with inconsistent state.
-    """
-    try:
-        if prisma_client:
-            # Use provided Prisma client (for consumer with its own connection)
-            updated_count = await prisma_client.chatmessage.update_many(
-                where={
-                    "sessionId": session_id,
-                    "toolCallId": tool_call_id,
-                },
-                data={"content": content},
-            )
-            # Check if any rows were updated - 0 means message not found
-            if updated_count == 0:
-                raise ToolMessageUpdateError(
-                    f"No message found with tool_call_id={tool_call_id} in session {session_id}"
-                )
-        else:
-            # Use service function (for webhook endpoint)
-            await chat_service._update_pending_operation(
-                session_id=session_id,
-                tool_call_id=tool_call_id,
-                result=content,
-            )
-    except ToolMessageUpdateError:
-        raise
-    except Exception as e:
-        logger.error(f"[COMPLETION] Failed to update tool message: {e}", exc_info=True)
-        raise ToolMessageUpdateError(
-            f"Failed to update tool message for tool_call_id={tool_call_id}: {e}"
-        ) from e
-
-
-def serialize_result(result: dict | list | str | int | float | bool | None) -> str:
-    """Serialize result to JSON string with sensible defaults.
-
-    Args:
-        result: The result to serialize. Can be a dict, list, string,
-            number, boolean, or None.
-
-    Returns:
-        JSON string representation of the result. Returns '{"status": "completed"}'
-        only when result is explicitly None.
-    """
-    if isinstance(result, str):
-        return result
-    if result is None:
-        return '{"status": "completed"}'
-    return orjson.dumps(result).decode("utf-8")
-
-
-async def _save_agent_from_result(
-    result: dict[str, Any],
-    user_id: str | None,
-    tool_name: str,
-) -> dict[str, Any]:
-    """Save agent to library if result contains agent_json.
-
-    Args:
-        result: The result dict that may contain agent_json
-        user_id: The user ID to save the agent for
-        tool_name: The tool name (create_agent or edit_agent)
-
-    Returns:
-        Updated result dict with saved agent details, or original result if no agent_json
-    """
-    if not user_id:
-        logger.warning("[COMPLETION] Cannot save agent: no user_id in task")
-        return result
-
-    agent_json = result.get("agent_json")
-    if not agent_json:
-        logger.warning(
-            f"[COMPLETION] {tool_name} completed but no agent_json in result"
-        )
-        return result
-
-    try:
-        from .tools.agent_generator import save_agent_to_library
-
-        is_update = tool_name == "edit_agent"
-        created_graph, library_agent = await save_agent_to_library(
-            agent_json, user_id, is_update=is_update
-        )
-
-        logger.info(
-            f"[COMPLETION] Saved agent '{created_graph.name}' to library "
-            f"(graph_id={created_graph.id}, library_agent_id={library_agent.id})"
-        )
-
-        # Return a response similar to AgentSavedResponse
-        return {
-            "type": "agent_saved",
-            "message": f"Agent '{created_graph.name}' has been saved to your library!",
-            "agent_id": created_graph.id,
-            "agent_name": created_graph.name,
-            "library_agent_id": library_agent.id,
-            "library_agent_link": f"/library/agents/{library_agent.id}",
-            "agent_page_link": f"/build?flowID={created_graph.id}",
-        }
-    except Exception as e:
-        logger.error(
-            f"[COMPLETION] Failed to save agent to library: {e}",
-            exc_info=True,
-        )
-        # Return error but don't fail the whole operation
-        # Sanitize agent_json to remove sensitive keys before returning
-        return {
-            "type": "error",
-            "message": f"Agent was generated but failed to save: {str(e)}",
-            "error": str(e),
-            "agent_json": _sanitize_agent_json(agent_json),
-        }
-
-
-async def process_operation_success(
-    task: stream_registry.ActiveTask,
-    result: dict | str | None,
-    prisma_client: Prisma | None = None,
-) -> None:
-    """Handle successful operation completion.
-
-    Publishes the result to the stream registry, updates the database,
-    generates LLM continuation, and marks the task as completed.
-
-    Args:
-        task: The active task that completed
-        result: The result data from the operation
-        prisma_client: Optional Prisma client for database operations.
-            If None, uses chat_service._update_pending_operation instead.
-
-    Raises:
-        ToolMessageUpdateError: If the database update fails. The task will be
-            marked as failed instead of completed to avoid inconsistent state.
-    """
-    # For agent generation tools, save the agent to library
-    if task.tool_name in AGENT_GENERATION_TOOLS and isinstance(result, dict):
-        result = await _save_agent_from_result(result, task.user_id, task.tool_name)
-
-    # Serialize result for output (only substitute default when result is exactly None)
-    result_output = result if result is not None else {"status": "completed"}
-    output_str = (
-        result_output
-        if isinstance(result_output, str)
-        else orjson.dumps(result_output).decode("utf-8")
-    )
-
-    # Publish result to stream registry
-    await stream_registry.publish_chunk(
-        task.task_id,
-        StreamToolOutputAvailable(
-            toolCallId=task.tool_call_id,
-            toolName=task.tool_name,
-            output=output_str,
-            success=True,
-        ),
-    )
-
-    # Update pending operation in database
-    # If this fails, we must not continue to mark the task as completed
-    result_str = serialize_result(result)
-    try:
-        await _update_tool_message(
-            session_id=task.session_id,
-            tool_call_id=task.tool_call_id,
-            content=result_str,
-            prisma_client=prisma_client,
-        )
-    except ToolMessageUpdateError:
-        # DB update failed - mark task as failed to avoid inconsistent state
-        logger.error(
-            f"[COMPLETION] DB update failed for task {task.task_id}, "
-            "marking as failed instead of completed"
-        )
-        await stream_registry.publish_chunk(
-            task.task_id,
-            StreamError(errorText="Failed to save operation result to database"),
-        )
-        await stream_registry.mark_task_completed(task.task_id, status="failed")
-        raise
-
-    # Generate LLM continuation with streaming
-    try:
-        await chat_service._generate_llm_continuation_with_streaming(
-            session_id=task.session_id,
-            user_id=task.user_id,
-            task_id=task.task_id,
-        )
-    except Exception as e:
-        logger.error(
-            f"[COMPLETION] Failed to generate LLM continuation: {e}",
-            exc_info=True,
-        )
-
-    # Mark task as completed and release Redis lock
-    await stream_registry.mark_task_completed(task.task_id, status="completed")
-    try:
-        await chat_service._mark_operation_completed(task.tool_call_id)
-    except Exception as e:
-        logger.error(f"[COMPLETION] Failed to mark operation completed: {e}")
-
-    logger.info(
-        f"[COMPLETION] Successfully processed completion for task {task.task_id}"
-    )
-
-
-async def process_operation_failure(
-    task: stream_registry.ActiveTask,
-    error: str | None,
-    prisma_client: Prisma | None = None,
-) -> None:
-    """Handle failed operation completion.
-
-    Publishes the error to the stream registry, updates the database with
-    the error response, and marks the task as failed.
-
-    Args:
-        task: The active task that failed
-        error: The error message from the operation
-        prisma_client: Optional Prisma client for database operations.
-            If None, uses chat_service._update_pending_operation instead.
-    """
-    error_msg = error or "Operation failed"
-
-    # Publish error to stream registry
-    await stream_registry.publish_chunk(
-        task.task_id,
-        StreamError(errorText=error_msg),
-    )
-
-    # Update pending operation with error
-    # If this fails, we still continue to mark the task as failed
-    error_response = ErrorResponse(
-        message=error_msg,
-        error=error,
-    )
-    try:
-        await _update_tool_message(
-            session_id=task.session_id,
-            tool_call_id=task.tool_call_id,
-            content=error_response.model_dump_json(),
-            prisma_client=prisma_client,
-        )
-    except ToolMessageUpdateError:
-        # DB update failed - log but continue with cleanup
-        logger.error(
-            f"[COMPLETION] DB update failed while processing failure for task {task.task_id}, "
-            "continuing with cleanup"
-        )
-
-    # Mark task as failed and release Redis lock
-    await stream_registry.mark_task_completed(task.task_id, status="failed")
-    try:
-        await chat_service._mark_operation_completed(task.tool_call_id)
-    except Exception as e:
-        logger.error(f"[COMPLETION] Failed to mark operation completed: {e}")
-
-    logger.info(f"[COMPLETION] Processed failure for task {task.task_id}: {error_msg}")

autogpt_platform/backend/backend/api/features/chat/db.py

@@ -1,288 +0,0 @@
-"""Database operations for chat sessions."""
-
-import asyncio
-import logging
-from datetime import UTC, datetime
-from typing import Any, cast
-
-from prisma.models import ChatMessage as PrismaChatMessage
-from prisma.models import ChatSession as PrismaChatSession
-from prisma.types import (
-    ChatMessageCreateInput,
-    ChatSessionCreateInput,
-    ChatSessionUpdateInput,
-    ChatSessionWhereInput,
-)
-
-from backend.data.db import transaction
-from backend.util.json import SafeJson
-
-logger = logging.getLogger(__name__)
-
-
-async def get_chat_session(session_id: str) -> PrismaChatSession | None:
-    """Get a chat session by ID from the database."""
-    session = await PrismaChatSession.prisma().find_unique(
-        where={"id": session_id},
-        include={"Messages": True},
-    )
-    if session and session.Messages:
-        # Sort messages by sequence in Python - Prisma Python client doesn't support
-        # order_by in include clauses (unlike Prisma JS), so we sort after fetching
-        session.Messages.sort(key=lambda m: m.sequence)
-    return session
-
-
-async def create_chat_session(
-    session_id: str,
-    user_id: str,
-) -> PrismaChatSession:
-    """Create a new chat session in the database."""
-    data = ChatSessionCreateInput(
-        id=session_id,
-        userId=user_id,
-        credentials=SafeJson({}),
-        successfulAgentRuns=SafeJson({}),
-        successfulAgentSchedules=SafeJson({}),
-    )
-    return await PrismaChatSession.prisma().create(data=data)
-
-
-async def update_chat_session(
-    session_id: str,
-    credentials: dict[str, Any] | None = None,
-    successful_agent_runs: dict[str, Any] | None = None,
-    successful_agent_schedules: dict[str, Any] | None = None,
-    total_prompt_tokens: int | None = None,
-    total_completion_tokens: int | None = None,
-    title: str | None = None,
-) -> PrismaChatSession | None:
-    """Update a chat session's metadata."""
-    data: ChatSessionUpdateInput = {"updatedAt": datetime.now(UTC)}
-
-    if credentials is not None:
-        data["credentials"] = SafeJson(credentials)
-    if successful_agent_runs is not None:
-        data["successfulAgentRuns"] = SafeJson(successful_agent_runs)
-    if successful_agent_schedules is not None:
-        data["successfulAgentSchedules"] = SafeJson(successful_agent_schedules)
-    if total_prompt_tokens is not None:
-        data["totalPromptTokens"] = total_prompt_tokens
-    if total_completion_tokens is not None:
-        data["totalCompletionTokens"] = total_completion_tokens
-    if title is not None:
-        data["title"] = title
-
-    session = await PrismaChatSession.prisma().update(
-        where={"id": session_id},
-        data=data,
-        include={"Messages": True},
-    )
-    if session and session.Messages:
-        # Sort in Python - Prisma Python doesn't support order_by in include clauses
-        session.Messages.sort(key=lambda m: m.sequence)
-    return session
-
-
-async def add_chat_message(
-    session_id: str,
-    role: str,
-    sequence: int,
-    content: str | None = None,
-    name: str | None = None,
-    tool_call_id: str | None = None,
-    refusal: str | None = None,
-    tool_calls: list[dict[str, Any]] | None = None,
-    function_call: dict[str, Any] | None = None,
-) -> PrismaChatMessage:
-    """Add a message to a chat session."""
-    # Build input dict dynamically rather than using ChatMessageCreateInput directly
-    # because Prisma's TypedDict validation rejects optional fields set to None.
-    # We only include fields that have values, then cast at the end.
-    data: dict[str, Any] = {
-        "Session": {"connect": {"id": session_id}},
-        "role": role,
-        "sequence": sequence,
-    }
-
-    # Add optional string fields
-    if content is not None:
-        data["content"] = content
-    if name is not None:
-        data["name"] = name
-    if tool_call_id is not None:
-        data["toolCallId"] = tool_call_id
-    if refusal is not None:
-        data["refusal"] = refusal
-
-    # Add optional JSON fields only when they have values
-    if tool_calls is not None:
-        data["toolCalls"] = SafeJson(tool_calls)
-    if function_call is not None:
-        data["functionCall"] = SafeJson(function_call)
-
-    # Run message create and session timestamp update in parallel for lower latency
-    _, message = await asyncio.gather(
-        PrismaChatSession.prisma().update(
-            where={"id": session_id},
-            data={"updatedAt": datetime.now(UTC)},
-        ),
-        PrismaChatMessage.prisma().create(data=cast(ChatMessageCreateInput, data)),
-    )
-    return message
-
-
-async def add_chat_messages_batch(
-    session_id: str,
-    messages: list[dict[str, Any]],
-    start_sequence: int,
-) -> list[PrismaChatMessage]:
-    """Add multiple messages to a chat session in a batch.
-
-    Uses a transaction for atomicity - if any message creation fails,
-    the entire batch is rolled back.
-    """
-    if not messages:
-        return []
-
-    created_messages = []
-
-    async with transaction() as tx:
-        for i, msg in enumerate(messages):
-            # Build input dict dynamically rather than using ChatMessageCreateInput
-            # directly because Prisma's TypedDict validation rejects optional fields
-            # set to None. We only include fields that have values, then cast.
-            data: dict[str, Any] = {
-                "Session": {"connect": {"id": session_id}},
-                "role": msg["role"],
-                "sequence": start_sequence + i,
-            }
-
-            # Add optional string fields
-            if msg.get("content") is not None:
-                data["content"] = msg["content"]
-            if msg.get("name") is not None:
-                data["name"] = msg["name"]
-            if msg.get("tool_call_id") is not None:
-                data["toolCallId"] = msg["tool_call_id"]
-            if msg.get("refusal") is not None:
-                data["refusal"] = msg["refusal"]
-
-            # Add optional JSON fields only when they have values
-            if msg.get("tool_calls") is not None:
-                data["toolCalls"] = SafeJson(msg["tool_calls"])
-            if msg.get("function_call") is not None:
-                data["functionCall"] = SafeJson(msg["function_call"])
-
-            created = await PrismaChatMessage.prisma(tx).create(
-                data=cast(ChatMessageCreateInput, data)
-            )
-            created_messages.append(created)
-
-        # Update session's updatedAt timestamp within the same transaction.
-        # Note: Token usage (total_prompt_tokens, total_completion_tokens) is updated
-        # separately via update_chat_session() after streaming completes.
-        await PrismaChatSession.prisma(tx).update(
-            where={"id": session_id},
-            data={"updatedAt": datetime.now(UTC)},
-        )
-
-    return created_messages
-
-
-async def get_user_chat_sessions(
-    user_id: str,
-    limit: int = 50,
-    offset: int = 0,
-) -> list[PrismaChatSession]:
-    """Get chat sessions for a user, ordered by most recent."""
-    return await PrismaChatSession.prisma().find_many(
-        where={"userId": user_id},
-        order={"updatedAt": "desc"},
-        take=limit,
-        skip=offset,
-    )
-
-
-async def get_user_session_count(user_id: str) -> int:
-    """Get the total number of chat sessions for a user."""
-    return await PrismaChatSession.prisma().count(where={"userId": user_id})
-
-
-async def delete_chat_session(session_id: str, user_id: str | None = None) -> bool:
-    """Delete a chat session and all its messages.
-
-    Args:
-        session_id: The session ID to delete.
-        user_id: If provided, validates that the session belongs to this user
-            before deletion. This prevents unauthorized deletion of other
-            users' sessions.
-
-    Returns:
-        True if deleted successfully, False otherwise.
-    """
-    try:
-        # Build typed where clause with optional user_id validation
-        where_clause: ChatSessionWhereInput = {"id": session_id}
-        if user_id is not None:
-            where_clause["userId"] = user_id
-
-        result = await PrismaChatSession.prisma().delete_many(where=where_clause)
-        if result == 0:
-            logger.warning(
-                f"No session deleted for {session_id} "
-                f"(user_id validation: {user_id is not None})"
-            )
-            return False
-        return True
-    except Exception as e:
-        logger.error(f"Failed to delete chat session {session_id}: {e}")
-        return False
-
-
-async def get_chat_session_message_count(session_id: str) -> int:
-    """Get the number of messages in a chat session."""
-    count = await PrismaChatMessage.prisma().count(where={"sessionId": session_id})
-    return count
-
-
-async def update_tool_message_content(
-    session_id: str,
-    tool_call_id: str,
-    new_content: str,
-) -> bool:
-    """Update the content of a tool message in chat history.
-
-    Used by background tasks to update pending operation messages with final results.
-
-    Args:
-        session_id: The chat session ID.
-        tool_call_id: The tool call ID to find the message.
-        new_content: The new content to set.
-
-    Returns:
-        True if a message was updated, False otherwise.
-    """
-    try:
-        result = await PrismaChatMessage.prisma().update_many(
-            where={
-                "sessionId": session_id,
-                "toolCallId": tool_call_id,
-            },
-            data={
-                "content": new_content,
-            },
-        )
-        if result == 0:
-            logger.warning(
-                f"No message found to update for session {session_id}, "
-                f"tool_call_id {tool_call_id}"
-            )
-            return False
-        return True
-    except Exception as e:
-        logger.error(
-            f"Failed to update tool message for session {session_id}, "
-            f"tool_call_id {tool_call_id}: {e}"
-        )
-        return False

autogpt_platform/backend/backend/api/features/chat/routes.py

@@ -2,33 +2,34 @@
 
 import asyncio
 import logging
-import uuid as uuid_module
+import re
 from collections.abc import AsyncGenerator
 from typing import Annotated
+from uuid import uuid4
 
 from autogpt_libs import auth
-from fastapi import APIRouter, Depends, Header, HTTPException, Query, Response, Security
+from fastapi import APIRouter, Depends, HTTPException, Query, Response, Security
 from fastapi.responses import StreamingResponse
-from pydantic import BaseModel
+from prisma.models import UserWorkspaceFile
+from pydantic import BaseModel, Field, field_validator
 
-from backend.util.exceptions import NotFoundError
-from backend.util.feature_flag import Flag, is_feature_enabled
-
-from . import service as chat_service
-from . import stream_registry
-from .completion_handler import process_operation_failure, process_operation_success
-from .config import ChatConfig
-from .model import (
+from backend.copilot import service as chat_service
+from backend.copilot import stream_registry
+from backend.copilot.config import ChatConfig
+from backend.copilot.executor.utils import enqueue_cancel_task, enqueue_copilot_turn
+from backend.copilot.model import (
     ChatMessage,
     ChatSession,
     append_and_save_message,
     create_chat_session,
+    delete_chat_session,
     get_chat_session,
     get_user_sessions,
+    update_session_title,
 )
-from .response_model import StreamError, StreamFinish, StreamHeartbeat, StreamStart
-from .sdk import service as sdk_service
-from .tools.models import (
+from backend.copilot.response_model import StreamError, StreamFinish, StreamHeartbeat
+from backend.copilot.tools.e2b_sandbox import kill_sandbox
+from backend.copilot.tools.models import (
     AgentDetailsResponse,
     AgentOutputResponse,
     AgentPreviewResponse,
@@ -43,18 +44,25 @@ from .tools.models import (
     ErrorResponse,
     ExecutionStartedResponse,
     InputValidationErrorResponse,
+    MCPToolOutputResponse,
+    MCPToolsDiscoveredResponse,
     NeedLoginResponse,
     NoResultsResponse,
-    OperationInProgressResponse,
-    OperationPendingResponse,
-    OperationStartedResponse,
     SetupRequirementsResponse,
+    SuggestedGoalResponse,
     UnderstandingUpdatedResponse,
 )
-from .tracking import track_user_message
+from backend.copilot.tracking import track_user_message
+from backend.data.redis_client import get_redis_async
+from backend.data.understanding import get_business_understanding
+from backend.data.workspace import get_or_create_workspace
+from backend.util.exceptions import NotFoundError
 
 config = ChatConfig()
 
+_UUID_RE = re.compile(
+    r"^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$", re.I
+)
 
 logger = logging.getLogger(__name__)
 
@@ -83,6 +91,9 @@ class StreamChatRequest(BaseModel):
     message: str
     is_user_message: bool = True
     context: dict[str, str] | None = None  # {url: str, content: str}
+    file_ids: list[str] | None = Field(
+        default=None, max_length=20
+    )  # Workspace file IDs attached to this message
 
 
 class CreateSessionResponse(BaseModel):
@@ -96,10 +107,8 @@ class CreateSessionResponse(BaseModel):
 class ActiveStreamInfo(BaseModel):
     """Information about an active stream for reconnection."""
 
-    task_id: str
+    turn_id: str
     last_message_id: str  # Redis Stream message ID for resumption
-    operation_id: str  # Operation ID for completion tracking
-    tool_name: str  # Name of the tool being executed
 
 
 class SessionDetailResponse(BaseModel):
@@ -120,6 +129,7 @@ class SessionSummaryResponse(BaseModel):
     created_at: str
     updated_at: str
     title: str | None = None
+    is_processing: bool
 
 
 class ListSessionsResponse(BaseModel):
@@ -129,12 +139,25 @@ class ListSessionsResponse(BaseModel):
     total: int
 
 
-class OperationCompleteRequest(BaseModel):
-    """Request model for external completion webhook."""
+class CancelSessionResponse(BaseModel):
+    """Response model for the cancel session endpoint."""
 
-    success: bool
-    result: dict | str | None = None
-    error: str | None = None
+    cancelled: bool
+    reason: str | None = None
+
+
+class UpdateSessionTitleRequest(BaseModel):
+    """Request model for updating a session's title."""
+
+    title: str
+
+    @field_validator("title")
+    @classmethod
+    def title_must_not_be_blank(cls, v: str) -> str:
+        stripped = v.strip()
+        if not stripped:
+            raise ValueError("Title must not be blank")
+        return stripped
 
 
 # ========== Routes ==========
@@ -165,6 +188,28 @@ async def list_sessions(
     """
     sessions, total_count = await get_user_sessions(user_id, limit, offset)
 
+    # Batch-check Redis for active stream status on each session
+    processing_set: set[str] = set()
+    if sessions:
+        try:
+            redis = await get_redis_async()
+            pipe = redis.pipeline(transaction=False)
+            for session in sessions:
+                pipe.hget(
+                    f"{config.session_meta_prefix}{session.session_id}",
+                    "status",
+                )
+            statuses = await pipe.execute()
+            processing_set = {
+                session.session_id
+                for session, st in zip(sessions, statuses)
+                if st == "running"
+            }
+        except Exception:
+            logger.warning(
+                "Failed to fetch processing status from Redis; " "defaulting to empty"
+            )
+
     return ListSessionsResponse(
         sessions=[
             SessionSummaryResponse(
@@ -172,6 +217,7 @@ async def list_sessions(
                 created_at=session.started_at.isoformat(),
                 updated_at=session.updated_at.isoformat(),
                 title=session.title,
+                is_processing=session.session_id in processing_set,
             )
             for session in sessions
         ],
@@ -211,6 +257,92 @@ async def create_session(
     )
 
 
+@router.delete(
+    "/sessions/{session_id}",
+    dependencies=[Security(auth.requires_user)],
+    status_code=204,
+    responses={404: {"description": "Session not found or access denied"}},
+)
+async def delete_session(
+    session_id: str,
+    user_id: Annotated[str, Security(auth.get_user_id)],
+) -> Response:
+    """
+    Delete a chat session.
+
+    Permanently removes a chat session and all its messages.
+    Only the owner can delete their sessions.
+
+    Args:
+        session_id: The session ID to delete.
+        user_id: The authenticated user's ID.
+
+    Returns:
+        204 No Content on success.
+
+    Raises:
+        HTTPException: 404 if session not found or not owned by user.
+    """
+    deleted = await delete_chat_session(session_id, user_id)
+
+    if not deleted:
+        raise HTTPException(
+            status_code=404,
+            detail=f"Session {session_id} not found or access denied",
+        )
+
+    # Best-effort cleanup of the E2B sandbox (if any).
+    # sandbox_id is in Redis; kill_sandbox() fetches it from there.
+    e2b_cfg = ChatConfig()
+    if e2b_cfg.e2b_active:
+        assert e2b_cfg.e2b_api_key  # guaranteed by e2b_active check
+        try:
+            await kill_sandbox(session_id, e2b_cfg.e2b_api_key)
+        except Exception:
+            logger.warning(
+                "[E2B] Failed to kill sandbox for session %s", session_id[:12]
+            )
+
+    return Response(status_code=204)
+
+
+@router.patch(
+    "/sessions/{session_id}/title",
+    summary="Update session title",
+    dependencies=[Security(auth.requires_user)],
+    status_code=200,
+    responses={404: {"description": "Session not found or access denied"}},
+)
+async def update_session_title_route(
+    session_id: str,
+    request: UpdateSessionTitleRequest,
+    user_id: Annotated[str, Security(auth.get_user_id)],
+) -> dict:
+    """
+    Update the title of a chat session.
+
+    Allows the user to rename their chat session.
+
+    Args:
+        session_id: The session ID to update.
+        request: Request body containing the new title.
+        user_id: The authenticated user's ID.
+
+    Returns:
+        dict: Status of the update.
+
+    Raises:
+        HTTPException: 404 if session not found or not owned by user.
+    """
+    success = await update_session_title(session_id, user_id, request.title)
+    if not success:
+        raise HTTPException(
+            status_code=404,
+            detail=f"Session {session_id} not found or access denied",
+        )
+    return {"status": "ok"}
+
+
 @router.get(
     "/sessions/{session_id}",
 )
@@ -222,7 +354,7 @@ async def get_session(
     Retrieve the details of a specific chat session.
 
     Looks up a chat session by ID for the given user (if authenticated) and returns all session data including messages.
-    If there's an active stream for this session, returns the task_id for reconnection.
+    If there's an active stream for this session, returns active_stream info for reconnection.
 
     Args:
         session_id: The unique identifier for the desired chat session.
@@ -240,28 +372,21 @@ async def get_session(
 
     # Check if there's an active stream for this session
     active_stream_info = None
-    active_task, last_message_id = await stream_registry.get_active_task_for_session(
+    active_session, last_message_id = await stream_registry.get_active_session(
         session_id, user_id
     )
     logger.info(
-        f"[GET_SESSION] session={session_id}, active_task={active_task is not None}, "
+        f"[GET_SESSION] session={session_id}, active_session={active_session is not None}, "
         f"msg_count={len(messages)}, last_role={messages[-1].get('role') if messages else 'none'}"
     )
-    if active_task:
-        # Filter out the in-progress assistant message from the session response.
-        # The client will receive the complete assistant response through the SSE
-        # stream replay instead, preventing duplicate content.
-        if messages and messages[-1].get("role") == "assistant":
-            messages = messages[:-1]
-
-        # Use "0-0" as last_message_id to replay the stream from the beginning.
-        # Since we filtered out the cached assistant message, the client needs
-        # the full stream to reconstruct the response.
+    if active_session:
+        # Keep the assistant message (including tool_calls) so the frontend can
+        # render the correct tool UI (e.g. CreateAgent with mini game).
+        # convertChatSessionToUiMessages handles isComplete=false by setting
+        # tool parts without output to state "input-available".
         active_stream_info = ActiveStreamInfo(
-            task_id=active_task.task_id,
-            last_message_id="0-0",
-            operation_id=active_task.operation_id,
-            tool_name=active_task.tool_name,
+            turn_id=active_session.turn_id,
+            last_message_id=last_message_id,
         )
 
     return SessionDetailResponse(
@@ -274,6 +399,51 @@ async def get_session(
     )
 
 
+@router.post(
+    "/sessions/{session_id}/cancel",
+    status_code=200,
+)
+async def cancel_session_task(
+    session_id: str,
+    user_id: Annotated[str | None, Depends(auth.get_user_id)],
+) -> CancelSessionResponse:
+    """Cancel the active streaming task for a session.
+
+    Publishes a cancel event to the executor via RabbitMQ FANOUT, then
+    polls Redis until the task status flips from ``running`` or a timeout
+    (5 s) is reached.  Returns only after the cancellation is confirmed.
+    """
+    await _validate_and_get_session(session_id, user_id)
+
+    active_session, _ = await stream_registry.get_active_session(session_id, user_id)
+    if not active_session:
+        return CancelSessionResponse(cancelled=True, reason="no_active_session")
+
+    await enqueue_cancel_task(session_id)
+    logger.info(f"[CANCEL] Published cancel for session ...{session_id[-8:]}")
+
+    # Poll until the executor confirms the task is no longer running.
+    poll_interval = 0.5
+    max_wait = 5.0
+    waited = 0.0
+    while waited < max_wait:
+        await asyncio.sleep(poll_interval)
+        waited += poll_interval
+        session_state = await stream_registry.get_session(session_id)
+        if session_state is None or session_state.status != "running":
+            logger.info(
+                f"[CANCEL] Session ...{session_id[-8:]} confirmed stopped "
+                f"(status={session_state.status if session_state else 'gone'}) after {waited:.1f}s"
+            )
+            return CancelSessionResponse(cancelled=True)
+
+    logger.warning(
+        f"[CANCEL] Session ...{session_id[-8:]} not confirmed after {max_wait}s, force-completing"
+    )
+    await stream_registry.mark_session_completed(session_id, error_message="Cancelled")
+    return CancelSessionResponse(cancelled=True)
+
+
 @router.post(
     "/sessions/{session_id}/stream",
 )
@@ -291,16 +461,15 @@ async def stream_chat_post(
       - Tool execution results
 
     The AI generation runs in a background task that continues even if the client disconnects.
-    All chunks are written to Redis for reconnection support. If the client disconnects,
-    they can reconnect using GET /tasks/{task_id}/stream to resume from where they left off.
+    All chunks are written to a per-turn Redis stream for reconnection support. If the client
+    disconnects, they can reconnect using GET /sessions/{session_id}/stream to resume.
 
     Args:
         session_id: The chat session identifier to associate with the streamed messages.
         request: Request body containing message, is_user_message, and optional context.
         user_id: Optional authenticated user ID.
     Returns:
-        StreamingResponse: SSE-formatted response chunks. First chunk is a "start" event
-        containing the task_id for reconnection.
+        StreamingResponse: SSE-formatted response chunks.
 
     """
     import asyncio
@@ -316,7 +485,7 @@ async def stream_chat_post(
         f"user={user_id}, message_len={len(request.message)}",
         extra={"json_fields": log_meta},
     )
-    session = await _validate_and_get_session(session_id, user_id)
+    await _validate_and_get_session(session_id, user_id)
     logger.info(
         f"[TIMING] session validated in {(time.perf_counter() - stream_start_time) * 1000:.1f}ms",
         extra={
@@ -327,6 +496,38 @@ async def stream_chat_post(
         },
     )
 
+    # Enrich message with file metadata if file_ids are provided.
+    # Also sanitise file_ids so only validated, workspace-scoped IDs are
+    # forwarded downstream (e.g. to the executor via enqueue_copilot_turn).
+    sanitized_file_ids: list[str] | None = None
+    if request.file_ids and user_id:
+        # Filter to valid UUIDs only to prevent DB abuse
+        valid_ids = [fid for fid in request.file_ids if _UUID_RE.match(fid)]
+
+        if valid_ids:
+            workspace = await get_or_create_workspace(user_id)
+            # Batch query instead of N+1
+            files = await UserWorkspaceFile.prisma().find_many(
+                where={
+                    "id": {"in": valid_ids},
+                    "workspaceId": workspace.id,
+                    "isDeleted": False,
+                }
+            )
+            # Only keep IDs that actually exist in the user's workspace
+            sanitized_file_ids = [wf.id for wf in files] or None
+            file_lines: list[str] = [
+                f"- {wf.name} ({wf.mimeType}, {round(wf.sizeBytes / 1024, 1)} KB), file_id={wf.id}"
+                for wf in files
+            ]
+            if file_lines:
+                files_block = (
+                    "\n\n[Attached files]\n"
+                    + "\n".join(file_lines)
+                    + "\nUse read_workspace_file with the file_id to access file contents."
+                )
+                request.message += files_block
+
     # Atomically append user message to session BEFORE creating task to avoid
     # race condition where GET_SESSION sees task as "running" but message isn't
     # saved yet.  append_and_save_message re-fetches inside a lock to prevent
@@ -343,152 +544,47 @@ async def stream_chat_post(
                 message_length=len(request.message),
             )
         logger.info(f"[STREAM] Saving user message to session {session_id}")
-        session = await append_and_save_message(session_id, message)
+        await append_and_save_message(session_id, message)
         logger.info(f"[STREAM] User message saved for session {session_id}")
 
     # Create a task in the stream registry for reconnection support
-    task_id = str(uuid_module.uuid4())
-    operation_id = str(uuid_module.uuid4())
-    log_meta["task_id"] = task_id
+    turn_id = str(uuid4())
+    log_meta["turn_id"] = turn_id
 
-    task_create_start = time.perf_counter()
-    await stream_registry.create_task(
-        task_id=task_id,
+    session_create_start = time.perf_counter()
+    await stream_registry.create_session(
         session_id=session_id,
         user_id=user_id,
-        tool_call_id="chat_stream",  # Not a tool call, but needed for the model
+        tool_call_id="chat_stream",
         tool_name="chat",
-        operation_id=operation_id,
+        turn_id=turn_id,
     )
     logger.info(
-        f"[TIMING] create_task completed in {(time.perf_counter() - task_create_start) * 1000:.1f}ms",
+        f"[TIMING] create_session completed in {(time.perf_counter() - session_create_start) * 1000:.1f}ms",
         extra={
             "json_fields": {
                 **log_meta,
-                "duration_ms": (time.perf_counter() - task_create_start) * 1000,
+                "duration_ms": (time.perf_counter() - session_create_start) * 1000,
             }
         },
     )
 
-    # Background task that runs the AI generation independently of SSE connection
-    async def run_ai_generation():
-        import time as time_module
+    # Per-turn stream is always fresh (unique turn_id), subscribe from beginning
+    subscribe_from_id = "0-0"
 
-        gen_start_time = time_module.perf_counter()
-        logger.info(
-            f"[TIMING] run_ai_generation STARTED, task={task_id}, session={session_id}, user={user_id}",
-            extra={"json_fields": log_meta},
-        )
-        first_chunk_time, ttfc = None, None
-        chunk_count = 0
-        try:
-            # Emit a start event with task_id for reconnection
-            start_chunk = StreamStart(messageId=task_id, taskId=task_id)
-            await stream_registry.publish_chunk(task_id, start_chunk)
-            logger.info(
-                f"[TIMING] StreamStart published at {(time_module.perf_counter() - gen_start_time) * 1000:.1f}ms",
-                extra={
-                    "json_fields": {
-                        **log_meta,
-                        "elapsed_ms": (time_module.perf_counter() - gen_start_time)
-                        * 1000,
-                    }
-                },
-            )
+    await enqueue_copilot_turn(
+        session_id=session_id,
+        user_id=user_id,
+        message=request.message,
+        turn_id=turn_id,
+        is_user_message=request.is_user_message,
+        context=request.context,
+        file_ids=sanitized_file_ids,
+    )
 
-            # Choose service based on LaunchDarkly flag (falls back to config default)
-            use_sdk = await is_feature_enabled(
-                Flag.COPILOT_SDK,
-                user_id or "anonymous",
-                default=config.use_claude_agent_sdk,
-            )
-            stream_fn = (
-                sdk_service.stream_chat_completion_sdk
-                if use_sdk
-                else chat_service.stream_chat_completion
-            )
-            logger.info(
-                f"[TIMING] Calling {'sdk' if use_sdk else 'standard'} stream_chat_completion",
-                extra={"json_fields": log_meta},
-            )
-            # Pass message=None since we already added it to the session above
-            async for chunk in stream_fn(
-                session_id,
-                None,  # Message already in session
-                is_user_message=request.is_user_message,
-                user_id=user_id,
-                session=session,  # Pass session with message already added
-                context=request.context,
-            ):
-                # Skip duplicate StreamStart — we already published one above
-                if isinstance(chunk, StreamStart):
-                    continue
-                chunk_count += 1
-                if first_chunk_time is None:
-                    first_chunk_time = time_module.perf_counter()
-                    ttfc = first_chunk_time - gen_start_time
-                    logger.info(
-                        f"[TIMING] FIRST AI CHUNK at {ttfc:.2f}s, type={type(chunk).__name__}",
-                        extra={
-                            "json_fields": {
-                                **log_meta,
-                                "chunk_type": type(chunk).__name__,
-                                "time_to_first_chunk_ms": ttfc * 1000,
-                            }
-                        },
-                    )
-                # Write to Redis (subscribers will receive via XREAD)
-                await stream_registry.publish_chunk(task_id, chunk)
-
-            gen_end_time = time_module.perf_counter()
-            total_time = (gen_end_time - gen_start_time) * 1000
-            logger.info(
-                f"[TIMING] run_ai_generation FINISHED in {total_time / 1000:.1f}s; "
-                f"task={task_id}, session={session_id}, "
-                f"ttfc={ttfc or -1:.2f}s, n_chunks={chunk_count}",
-                extra={
-                    "json_fields": {
-                        **log_meta,
-                        "total_time_ms": total_time,
-                        "time_to_first_chunk_ms": (
-                            ttfc * 1000 if ttfc is not None else None
-                        ),
-                        "n_chunks": chunk_count,
-                    }
-                },
-            )
-            await stream_registry.mark_task_completed(task_id, "completed")
-        except Exception as e:
-            elapsed = time_module.perf_counter() - gen_start_time
-            logger.error(
-                f"[TIMING] run_ai_generation ERROR after {elapsed:.2f}s: {e}",
-                extra={
-                    "json_fields": {
-                        **log_meta,
-                        "elapsed_ms": elapsed * 1000,
-                        "error": str(e),
-                    }
-                },
-            )
-            # Publish a StreamError so the frontend can display an error message
-            try:
-                await stream_registry.publish_chunk(
-                    task_id,
-                    StreamError(
-                        errorText="An error occurred. Please try again.",
-                        code="stream_error",
-                    ),
-                )
-            except Exception:
-                pass  # Best-effort; mark_task_completed will publish StreamFinish
-            await stream_registry.mark_task_completed(task_id, "failed")
-
-    # Start the AI generation in a background task
-    bg_task = asyncio.create_task(run_ai_generation())
-    await stream_registry.set_task_asyncio_task(task_id, bg_task)
     setup_time = (time.perf_counter() - stream_start_time) * 1000
     logger.info(
-        f"[TIMING] Background task started, setup={setup_time:.1f}ms",
+        f"[TIMING] Task enqueued to RabbitMQ, setup={setup_time:.1f}ms",
         extra={"json_fields": {**log_meta, "setup_time_ms": setup_time}},
     )
 
@@ -498,7 +594,7 @@ async def stream_chat_post(
 
         event_gen_start = time_module.perf_counter()
         logger.info(
-            f"[TIMING] event_generator STARTED, task={task_id}, session={session_id}, "
+            f"[TIMING] event_generator STARTED, turn={turn_id}, session={session_id}, "
             f"user={user_id}",
             extra={"json_fields": log_meta},
         )
@@ -506,11 +602,12 @@ async def stream_chat_post(
         first_chunk_yielded = False
         chunks_yielded = 0
         try:
-            # Subscribe to the task stream (this replays existing messages + live updates)
-            subscriber_queue = await stream_registry.subscribe_to_task(
-                task_id=task_id,
+            # Subscribe from the position we captured before enqueuing
+            # This avoids replaying old messages while catching all new ones
+            subscriber_queue = await stream_registry.subscribe_to_session(
+                session_id=session_id,
                 user_id=user_id,
-                last_message_id="0-0",  # Get all messages from the beginning
+                last_message_id=subscribe_from_id,
             )
 
             if subscriber_queue is None:
@@ -525,7 +622,7 @@ async def stream_chat_post(
             )
             while True:
                 try:
-                    chunk = await asyncio.wait_for(subscriber_queue.get(), timeout=30.0)
+                    chunk = await asyncio.wait_for(subscriber_queue.get(), timeout=10.0)
                     chunks_yielded += 1
 
                     if not first_chunk_yielded:
@@ -593,19 +690,19 @@ async def stream_chat_post(
             # Unsubscribe when client disconnects or stream ends
             if subscriber_queue is not None:
                 try:
-                    await stream_registry.unsubscribe_from_task(
-                        task_id, subscriber_queue
+                    await stream_registry.unsubscribe_from_session(
+                        session_id, subscriber_queue
                     )
                 except Exception as unsub_err:
                     logger.error(
-                        f"Error unsubscribing from task {task_id}: {unsub_err}",
+                        f"Error unsubscribing from session {session_id}: {unsub_err}",
                         exc_info=True,
                     )
             # AI SDK protocol termination - always yield even if unsubscribe fails
             total_time = time_module.perf_counter() - event_gen_start
             logger.info(
                 f"[TIMING] event_generator FINISHED in {total_time:.2f}s; "
-                f"task={task_id}, session={session_id}, n_chunks={chunks_yielded}",
+                f"turn={turn_id}, session={session_id}, n_chunks={chunks_yielded}",
                 extra={
                     "json_fields": {
                         **log_meta,
@@ -652,17 +749,21 @@ async def resume_session_stream(
     """
     import asyncio
 
-    active_task, _last_id = await stream_registry.get_active_task_for_session(
+    active_session, last_message_id = await stream_registry.get_active_session(
         session_id, user_id
     )
 
-    if not active_task:
+    if not active_session:
         return Response(status_code=204)
 
-    subscriber_queue = await stream_registry.subscribe_to_task(
-        task_id=active_task.task_id,
+    # Always replay from the beginning ("0-0") on resume.
+    # We can't use last_message_id because it's the latest ID in the backend
+    # stream, not the latest the frontend received — the gap causes lost
+    # messages. The frontend deduplicates replayed content.
+    subscriber_queue = await stream_registry.subscribe_to_session(
+        session_id=session_id,
         user_id=user_id,
-        last_message_id="0-0",  # Full replay so useChat rebuilds the message
+        last_message_id="0-0",
     )
 
     if subscriber_queue is None:
@@ -674,7 +775,7 @@ async def resume_session_stream(
         try:
             while True:
                 try:
-                    chunk = await asyncio.wait_for(subscriber_queue.get(), timeout=30.0)
+                    chunk = await asyncio.wait_for(subscriber_queue.get(), timeout=10.0)
                     if chunk_count < 3:
                         logger.info(
                             "Resume stream chunk",
@@ -698,12 +799,12 @@ async def resume_session_stream(
             logger.error(f"Error in resume stream for session {session_id}: {e}")
         finally:
             try:
-                await stream_registry.unsubscribe_from_task(
-                    active_task.task_id, subscriber_queue
+                await stream_registry.unsubscribe_from_session(
+                    session_id, subscriber_queue
                 )
             except Exception as unsub_err:
                 logger.error(
-                    f"Error unsubscribing from task {active_task.task_id}: {unsub_err}",
+                    f"Error unsubscribing from session {active_session.session_id}: {unsub_err}",
                     exc_info=True,
                 )
             logger.info(
@@ -731,7 +832,6 @@ async def resume_session_stream(
 @router.patch(
     "/sessions/{session_id}/assign-user",
     dependencies=[Security(auth.requires_user)],
-    status_code=200,
 )
 async def session_assign_user(
     session_id: str,
@@ -754,227 +854,34 @@ async def session_assign_user(
     return {"status": "ok"}
 
 
-# ========== Task Streaming (SSE Reconnection) ==========
+# ========== Suggested Prompts ==========
+
+
+class SuggestedPromptsResponse(BaseModel):
+    """Response model for user-specific suggested prompts."""
+
+    prompts: list[str]
 
 
 @router.get(
-    "/tasks/{task_id}/stream",
+    "/suggested-prompts",
+    dependencies=[Security(auth.requires_user)],
 )
-async def stream_task(
-    task_id: str,
-    user_id: str | None = Depends(auth.get_user_id),
-    last_message_id: str = Query(
-        default="0-0",
-        description="Last Redis Stream message ID received (e.g., '1706540123456-0'). Use '0-0' for full replay.",
-    ),
-):
+async def get_suggested_prompts(
+    user_id: Annotated[str, Security(auth.get_user_id)],
+) -> SuggestedPromptsResponse:
     """
-    Reconnect to a long-running task's SSE stream.
+    Get LLM-generated suggested prompts for the authenticated user.
 
-    When a long-running operation (like agent generation) starts, the client
-    receives a task_id. If the connection drops, the client can reconnect
-    using this endpoint to resume receiving updates.
-
-    Args:
-        task_id: The task ID from the operation_started response.
-        user_id: Authenticated user ID for ownership validation.
-        last_message_id: Last Redis Stream message ID received ("0-0" for full replay).
-
-    Returns:
-        StreamingResponse: SSE-formatted response chunks starting after last_message_id.
-
-    Raises:
-        HTTPException: 404 if task not found, 410 if task expired, 403 if access denied.
+    Returns personalized quick-action prompts based on the user's
+    business understanding. Returns an empty list if no custom prompts
+    are available.
     """
-    # Check task existence and expiry before subscribing
-    task, error_code = await stream_registry.get_task_with_expiry_info(task_id)
+    understanding = await get_business_understanding(user_id)
+    if understanding is None:
+        return SuggestedPromptsResponse(prompts=[])
 
-    if error_code == "TASK_EXPIRED":
-        raise HTTPException(
-            status_code=410,
-            detail={
-                "code": "TASK_EXPIRED",
-                "message": "This operation has expired. Please try again.",
-            },
-        )
-
-    if error_code == "TASK_NOT_FOUND":
-        raise HTTPException(
-            status_code=404,
-            detail={
-                "code": "TASK_NOT_FOUND",
-                "message": f"Task {task_id} not found.",
-            },
-        )
-
-    # Validate ownership if task has an owner
-    if task and task.user_id and user_id != task.user_id:
-        raise HTTPException(
-            status_code=403,
-            detail={
-                "code": "ACCESS_DENIED",
-                "message": "You do not have access to this task.",
-            },
-        )
-
-    # Get subscriber queue from stream registry
-    subscriber_queue = await stream_registry.subscribe_to_task(
-        task_id=task_id,
-        user_id=user_id,
-        last_message_id=last_message_id,
-    )
-
-    if subscriber_queue is None:
-        raise HTTPException(
-            status_code=404,
-            detail={
-                "code": "TASK_NOT_FOUND",
-                "message": f"Task {task_id} not found or access denied.",
-            },
-        )
-
-    async def event_generator() -> AsyncGenerator[str, None]:
-        heartbeat_interval = 15.0  # Send heartbeat every 15 seconds
-        try:
-            while True:
-                try:
-                    # Wait for next chunk with timeout for heartbeats
-                    chunk = await asyncio.wait_for(
-                        subscriber_queue.get(), timeout=heartbeat_interval
-                    )
-                    yield chunk.to_sse()
-
-                    # Check for finish signal
-                    if isinstance(chunk, StreamFinish):
-                        break
-                except asyncio.TimeoutError:
-                    # Send heartbeat to keep connection alive
-                    yield StreamHeartbeat().to_sse()
-        except Exception as e:
-            logger.error(f"Error in task stream {task_id}: {e}", exc_info=True)
-        finally:
-            # Unsubscribe when client disconnects or stream ends
-            try:
-                await stream_registry.unsubscribe_from_task(task_id, subscriber_queue)
-            except Exception as unsub_err:
-                logger.error(
-                    f"Error unsubscribing from task {task_id}: {unsub_err}",
-                    exc_info=True,
-                )
-            # AI SDK protocol termination - always yield even if unsubscribe fails
-            yield "data: [DONE]\n\n"
-
-    return StreamingResponse(
-        event_generator(),
-        media_type="text/event-stream",
-        headers={
-            "Cache-Control": "no-cache",
-            "Connection": "keep-alive",
-            "X-Accel-Buffering": "no",
-            "x-vercel-ai-ui-message-stream": "v1",
-        },
-    )
-
-
-@router.get(
-    "/tasks/{task_id}",
-)
-async def get_task_status(
-    task_id: str,
-    user_id: str | None = Depends(auth.get_user_id),
-) -> dict:
-    """
-    Get the status of a long-running task.
-
-    Args:
-        task_id: The task ID to check.
-        user_id: Authenticated user ID for ownership validation.
-
-    Returns:
-        dict: Task status including task_id, status, tool_name, and operation_id.
-
-    Raises:
-        NotFoundError: If task_id is not found or user doesn't have access.
-    """
-    task = await stream_registry.get_task(task_id)
-
-    if task is None:
-        raise NotFoundError(f"Task {task_id} not found.")
-
-    # Validate ownership - if task has an owner, requester must match
-    if task.user_id and user_id != task.user_id:
-        raise NotFoundError(f"Task {task_id} not found.")
-
-    return {
-        "task_id": task.task_id,
-        "session_id": task.session_id,
-        "status": task.status,
-        "tool_name": task.tool_name,
-        "operation_id": task.operation_id,
-        "created_at": task.created_at.isoformat(),
-    }
-
-
-# ========== External Completion Webhook ==========
-
-
-@router.post(
-    "/operations/{operation_id}/complete",
-    status_code=200,
-)
-async def complete_operation(
-    operation_id: str,
-    request: OperationCompleteRequest,
-    x_api_key: str | None = Header(default=None),
-) -> dict:
-    """
-    External completion webhook for long-running operations.
-
-    Called by Agent Generator (or other services) when an operation completes.
-    This triggers the stream registry to publish completion and continue LLM generation.
-
-    Args:
-        operation_id: The operation ID to complete.
-        request: Completion payload with success status and result/error.
-        x_api_key: Internal API key for authentication.
-
-    Returns:
-        dict: Status of the completion.
-
-    Raises:
-        HTTPException: If API key is invalid or operation not found.
-    """
-    # Validate internal API key - reject if not configured or invalid
-    if not config.internal_api_key:
-        logger.error(
-            "Operation complete webhook rejected: CHAT_INTERNAL_API_KEY not configured"
-        )
-        raise HTTPException(
-            status_code=503,
-            detail="Webhook not available: internal API key not configured",
-        )
-    if x_api_key != config.internal_api_key:
-        raise HTTPException(status_code=401, detail="Invalid API key")
-
-    # Find task by operation_id
-    task = await stream_registry.find_task_by_operation_id(operation_id)
-    if task is None:
-        raise HTTPException(
-            status_code=404,
-            detail=f"Operation {operation_id} not found",
-        )
-
-    logger.info(
-        f"Received completion webhook for operation {operation_id} "
-        f"(task_id={task.task_id}, success={request.success})"
-    )
-
-    if request.success:
-        await process_operation_success(task, request.result)
-    else:
-        await process_operation_failure(task, request.error)
-
-    return {"status": "ok", "task_id": task.task_id}
+    return SuggestedPromptsResponse(prompts=understanding.suggested_prompts)
 
 
 # ========== Configuration ==========
@@ -1051,14 +958,14 @@ ToolResponseUnion = (
     | AgentPreviewResponse
     | AgentSavedResponse
     | ClarificationNeededResponse
+    | SuggestedGoalResponse
     | BlockListResponse
     | BlockDetailsResponse
     | BlockOutputResponse
     | DocSearchResultsResponse
     | DocPageResponse
-    | OperationStartedResponse
-    | OperationPendingResponse
-    | OperationInProgressResponse
+    | MCPToolsDiscoveredResponse
+    | MCPToolOutputResponse
 )
 
 

autogpt_platform/backend/backend/api/features/chat/routes_test.py

@@ -0,0 +1,310 @@
+"""Tests for chat API routes: session title update, file attachment validation, and suggested prompts."""
+
+from unittest.mock import AsyncMock, MagicMock
+
+import fastapi
+import fastapi.testclient
+import pytest
+import pytest_mock
+
+from backend.api.features.chat import routes as chat_routes
+
+app = fastapi.FastAPI()
+app.include_router(chat_routes.router)
+
+client = fastapi.testclient.TestClient(app)
+
+TEST_USER_ID = "3e53486c-cf57-477e-ba2a-cb02dc828e1a"
+
+
+@pytest.fixture(autouse=True)
+def setup_app_auth(mock_jwt_user):
+    """Setup auth overrides for all tests in this module"""
+    from autogpt_libs.auth.jwt_utils import get_jwt_payload
+
+    app.dependency_overrides[get_jwt_payload] = mock_jwt_user["get_jwt_payload"]
+    yield
+    app.dependency_overrides.clear()
+
+
+def _mock_update_session_title(
+    mocker: pytest_mock.MockerFixture, *, success: bool = True
+):
+    """Mock update_session_title."""
+    return mocker.patch(
+        "backend.api.features.chat.routes.update_session_title",
+        new_callable=AsyncMock,
+        return_value=success,
+    )
+
+
+# ─── Update title: success ─────────────────────────────────────────────
+
+
+def test_update_title_success(
+    mocker: pytest_mock.MockerFixture,
+    test_user_id: str,
+) -> None:
+    mock_update = _mock_update_session_title(mocker, success=True)
+
+    response = client.patch(
+        "/sessions/sess-1/title",
+        json={"title": "My project"},
+    )
+
+    assert response.status_code == 200
+    assert response.json() == {"status": "ok"}
+    mock_update.assert_called_once_with("sess-1", test_user_id, "My project")
+
+
+def test_update_title_trims_whitespace(
+    mocker: pytest_mock.MockerFixture,
+    test_user_id: str,
+) -> None:
+    mock_update = _mock_update_session_title(mocker, success=True)
+
+    response = client.patch(
+        "/sessions/sess-1/title",
+        json={"title": "  trimmed  "},
+    )
+
+    assert response.status_code == 200
+    mock_update.assert_called_once_with("sess-1", test_user_id, "trimmed")
+
+
+# ─── Update title: blank / whitespace-only → 422 ──────────────────────
+
+
+def test_update_title_blank_rejected(
+    test_user_id: str,
+) -> None:
+    """Whitespace-only titles must be rejected before hitting the DB."""
+    response = client.patch(
+        "/sessions/sess-1/title",
+        json={"title": "   "},
+    )
+
+    assert response.status_code == 422
+
+
+def test_update_title_empty_rejected(
+    test_user_id: str,
+) -> None:
+    response = client.patch(
+        "/sessions/sess-1/title",
+        json={"title": ""},
+    )
+
+    assert response.status_code == 422
+
+
+# ─── Update title: session not found or wrong user → 404 ──────────────
+
+
+def test_update_title_not_found(
+    mocker: pytest_mock.MockerFixture,
+    test_user_id: str,
+) -> None:
+    _mock_update_session_title(mocker, success=False)
+
+    response = client.patch(
+        "/sessions/sess-1/title",
+        json={"title": "New name"},
+    )
+
+    assert response.status_code == 404
+
+
+# ─── file_ids Pydantic validation ─────────────────────────────────────
+
+
+def test_stream_chat_rejects_too_many_file_ids():
+    """More than 20 file_ids should be rejected by Pydantic validation (422)."""
+    response = client.post(
+        "/sessions/sess-1/stream",
+        json={
+            "message": "hello",
+            "file_ids": [f"00000000-0000-0000-0000-{i:012d}" for i in range(21)],
+        },
+    )
+    assert response.status_code == 422
+
+
+def _mock_stream_internals(mocker: pytest_mock.MockFixture):
+    """Mock the async internals of stream_chat_post so tests can exercise
+    validation and enrichment logic without needing Redis/RabbitMQ."""
+    mocker.patch(
+        "backend.api.features.chat.routes._validate_and_get_session",
+        return_value=None,
+    )
+    mocker.patch(
+        "backend.api.features.chat.routes.append_and_save_message",
+        return_value=None,
+    )
+    mock_registry = mocker.MagicMock()
+    mock_registry.create_session = mocker.AsyncMock(return_value=None)
+    mocker.patch(
+        "backend.api.features.chat.routes.stream_registry",
+        mock_registry,
+    )
+    mocker.patch(
+        "backend.api.features.chat.routes.enqueue_copilot_turn",
+        return_value=None,
+    )
+    mocker.patch(
+        "backend.api.features.chat.routes.track_user_message",
+        return_value=None,
+    )
+
+
+def test_stream_chat_accepts_20_file_ids(mocker: pytest_mock.MockFixture):
+    """Exactly 20 file_ids should be accepted (not rejected by validation)."""
+    _mock_stream_internals(mocker)
+    # Patch workspace lookup as imported by the routes module
+    mocker.patch(
+        "backend.api.features.chat.routes.get_or_create_workspace",
+        return_value=type("W", (), {"id": "ws-1"})(),
+    )
+    mock_prisma = mocker.MagicMock()
+    mock_prisma.find_many = mocker.AsyncMock(return_value=[])
+    mocker.patch(
+        "prisma.models.UserWorkspaceFile.prisma",
+        return_value=mock_prisma,
+    )
+
+    response = client.post(
+        "/sessions/sess-1/stream",
+        json={
+            "message": "hello",
+            "file_ids": [f"00000000-0000-0000-0000-{i:012d}" for i in range(20)],
+        },
+    )
+    # Should get past validation — 200 streaming response expected
+    assert response.status_code == 200
+
+
+# ─── UUID format filtering ─────────────────────────────────────────────
+
+
+def test_file_ids_filters_invalid_uuids(mocker: pytest_mock.MockFixture):
+    """Non-UUID strings in file_ids should be silently filtered out
+    and NOT passed to the database query."""
+    _mock_stream_internals(mocker)
+    mocker.patch(
+        "backend.api.features.chat.routes.get_or_create_workspace",
+        return_value=type("W", (), {"id": "ws-1"})(),
+    )
+
+    mock_prisma = mocker.MagicMock()
+    mock_prisma.find_many = mocker.AsyncMock(return_value=[])
+    mocker.patch(
+        "prisma.models.UserWorkspaceFile.prisma",
+        return_value=mock_prisma,
+    )
+
+    valid_id = "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"
+    client.post(
+        "/sessions/sess-1/stream",
+        json={
+            "message": "hello",
+            "file_ids": [
+                valid_id,
+                "not-a-uuid",
+                "../../../etc/passwd",
+                "",
+            ],
+        },
+    )
+
+    # The find_many call should only receive the one valid UUID
+    mock_prisma.find_many.assert_called_once()
+    call_kwargs = mock_prisma.find_many.call_args[1]
+    assert call_kwargs["where"]["id"]["in"] == [valid_id]
+
+
+# ─── Cross-workspace file_ids ─────────────────────────────────────────
+
+
+def test_file_ids_scoped_to_workspace(mocker: pytest_mock.MockFixture):
+    """The batch query should scope to the user's workspace."""
+    _mock_stream_internals(mocker)
+    mocker.patch(
+        "backend.api.features.chat.routes.get_or_create_workspace",
+        return_value=type("W", (), {"id": "my-workspace-id"})(),
+    )
+
+    mock_prisma = mocker.MagicMock()
+    mock_prisma.find_many = mocker.AsyncMock(return_value=[])
+    mocker.patch(
+        "prisma.models.UserWorkspaceFile.prisma",
+        return_value=mock_prisma,
+    )
+
+    fid = "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"
+    client.post(
+        "/sessions/sess-1/stream",
+        json={"message": "hi", "file_ids": [fid]},
+    )
+
+    call_kwargs = mock_prisma.find_many.call_args[1]
+    assert call_kwargs["where"]["workspaceId"] == "my-workspace-id"
+    assert call_kwargs["where"]["isDeleted"] is False
+
+
+# ─── Suggested prompts endpoint ──────────────────────────────────────
+
+
+def _mock_get_business_understanding(
+    mocker: pytest_mock.MockerFixture,
+    *,
+    return_value=None,
+):
+    """Mock get_business_understanding."""
+    return mocker.patch(
+        "backend.api.features.chat.routes.get_business_understanding",
+        new_callable=AsyncMock,
+        return_value=return_value,
+    )
+
+
+def test_suggested_prompts_returns_prompts(
+    mocker: pytest_mock.MockerFixture,
+    test_user_id: str,
+) -> None:
+    """User with understanding and prompts gets them back."""
+    mock_understanding = MagicMock()
+    mock_understanding.suggested_prompts = ["Do X", "Do Y", "Do Z"]
+    _mock_get_business_understanding(mocker, return_value=mock_understanding)
+
+    response = client.get("/suggested-prompts")
+
+    assert response.status_code == 200
+    assert response.json() == {"prompts": ["Do X", "Do Y", "Do Z"]}
+
+
+def test_suggested_prompts_no_understanding(
+    mocker: pytest_mock.MockerFixture,
+    test_user_id: str,
+) -> None:
+    """User with no understanding gets empty list."""
+    _mock_get_business_understanding(mocker, return_value=None)
+
+    response = client.get("/suggested-prompts")
+
+    assert response.status_code == 200
+    assert response.json() == {"prompts": []}
+
+
+def test_suggested_prompts_empty_prompts(
+    mocker: pytest_mock.MockerFixture,
+    test_user_id: str,
+) -> None:
+    """User with understanding but no prompts gets empty list."""
+    mock_understanding = MagicMock()
+    mock_understanding.suggested_prompts = []
+    _mock_get_business_understanding(mocker, return_value=mock_understanding)
+
+    response = client.get("/suggested-prompts")
+
+    assert response.status_code == 200
+    assert response.json() == {"prompts": []}

autogpt_platform/backend/backend/api/features/chat/sdk/__init__.py

@@ -1,14 +0,0 @@
-"""Claude Agent SDK integration for CoPilot.
-
-This module provides the integration layer between the Claude Agent SDK
-and the existing CoPilot tool system, enabling drop-in replacement of
-the current LLM orchestration with the battle-tested Claude Agent SDK.
-"""
-
-from .service import stream_chat_completion_sdk
-from .tool_adapter import create_copilot_mcp_server
-
-__all__ = [
-    "stream_chat_completion_sdk",
-    "create_copilot_mcp_server",
-]

autogpt_platform/backend/backend/api/features/chat/sdk/response_adapter.py

@@ -1,203 +0,0 @@
-"""Response adapter for converting Claude Agent SDK messages to Vercel AI SDK format.
-
-This module provides the adapter layer that converts streaming messages from
-the Claude Agent SDK into the Vercel AI SDK UI Stream Protocol format that
-the frontend expects.
-"""
-
-import json
-import logging
-import uuid
-
-from claude_agent_sdk import (
-    AssistantMessage,
-    Message,
-    ResultMessage,
-    SystemMessage,
-    TextBlock,
-    ToolResultBlock,
-    ToolUseBlock,
-    UserMessage,
-)
-
-from backend.api.features.chat.response_model import (
-    StreamBaseResponse,
-    StreamError,
-    StreamFinish,
-    StreamFinishStep,
-    StreamStart,
-    StreamStartStep,
-    StreamTextDelta,
-    StreamTextEnd,
-    StreamTextStart,
-    StreamToolInputAvailable,
-    StreamToolInputStart,
-    StreamToolOutputAvailable,
-)
-from backend.api.features.chat.sdk.tool_adapter import (
-    MCP_TOOL_PREFIX,
-    pop_pending_tool_output,
-)
-
-logger = logging.getLogger(__name__)
-
-
-class SDKResponseAdapter:
-    """Adapter for converting Claude Agent SDK messages to Vercel AI SDK format.
-
-    This class maintains state during a streaming session to properly track
-    text blocks, tool calls, and message lifecycle.
-    """
-
-    def __init__(self, message_id: str | None = None):
-        self.message_id = message_id or str(uuid.uuid4())
-        self.text_block_id = str(uuid.uuid4())
-        self.has_started_text = False
-        self.has_ended_text = False
-        self.current_tool_calls: dict[str, dict[str, str]] = {}
-        self.task_id: str | None = None
-        self.step_open = False
-
-    def set_task_id(self, task_id: str) -> None:
-        """Set the task ID for reconnection support."""
-        self.task_id = task_id
-
-    def convert_message(self, sdk_message: Message) -> list[StreamBaseResponse]:
-        """Convert a single SDK message to Vercel AI SDK format."""
-        responses: list[StreamBaseResponse] = []
-
-        if isinstance(sdk_message, SystemMessage):
-            if sdk_message.subtype == "init":
-                responses.append(
-                    StreamStart(messageId=self.message_id, taskId=self.task_id)
-                )
-                # Open the first step (matches non-SDK: StreamStart then StreamStartStep)
-                responses.append(StreamStartStep())
-                self.step_open = True
-
-        elif isinstance(sdk_message, AssistantMessage):
-            # After tool results, the SDK sends a new AssistantMessage for the
-            # next LLM turn. Open a new step if the previous one was closed.
-            if not self.step_open:
-                responses.append(StreamStartStep())
-                self.step_open = True
-
-            for block in sdk_message.content:
-                if isinstance(block, TextBlock):
-                    if block.text:
-                        self._ensure_text_started(responses)
-                        responses.append(
-                            StreamTextDelta(id=self.text_block_id, delta=block.text)
-                        )
-
-                elif isinstance(block, ToolUseBlock):
-                    self._end_text_if_open(responses)
-
-                    # Strip MCP prefix so frontend sees "find_block"
-                    # instead of "mcp__copilot__find_block".
-                    tool_name = block.name.removeprefix(MCP_TOOL_PREFIX)
-
-                    responses.append(
-                        StreamToolInputStart(toolCallId=block.id, toolName=tool_name)
-                    )
-                    responses.append(
-                        StreamToolInputAvailable(
-                            toolCallId=block.id,
-                            toolName=tool_name,
-                            input=block.input,
-                        )
-                    )
-                    self.current_tool_calls[block.id] = {"name": tool_name}
-
-        elif isinstance(sdk_message, UserMessage):
-            # UserMessage carries tool results back from tool execution.
-            content = sdk_message.content
-            blocks = content if isinstance(content, list) else []
-            for block in blocks:
-                if isinstance(block, ToolResultBlock) and block.tool_use_id:
-                    tool_info = self.current_tool_calls.get(block.tool_use_id, {})
-                    tool_name = tool_info.get("name", "unknown")
-
-                    # Prefer the stashed full output over the SDK's
-                    # (potentially truncated) ToolResultBlock content.
-                    # The SDK truncates large results, writing them to disk,
-                    # which breaks frontend widget parsing.
-                    output = pop_pending_tool_output(tool_name) or (
-                        _extract_tool_output(block.content)
-                    )
-
-                    responses.append(
-                        StreamToolOutputAvailable(
-                            toolCallId=block.tool_use_id,
-                            toolName=tool_name,
-                            output=output,
-                            success=not (block.is_error or False),
-                        )
-                    )
-
-            # Close the current step after tool results — the next
-            # AssistantMessage will open a new step for the continuation.
-            if self.step_open:
-                responses.append(StreamFinishStep())
-                self.step_open = False
-
-        elif isinstance(sdk_message, ResultMessage):
-            self._end_text_if_open(responses)
-            # Close the step before finishing.
-            if self.step_open:
-                responses.append(StreamFinishStep())
-                self.step_open = False
-
-            if sdk_message.subtype == "success":
-                responses.append(StreamFinish())
-            elif sdk_message.subtype in ("error", "error_during_execution"):
-                error_msg = getattr(sdk_message, "result", None) or "Unknown error"
-                responses.append(
-                    StreamError(errorText=str(error_msg), code="sdk_error")
-                )
-                responses.append(StreamFinish())
-            else:
-                logger.warning(
-                    f"Unexpected ResultMessage subtype: {sdk_message.subtype}"
-                )
-                responses.append(StreamFinish())
-
-        else:
-            logger.debug(f"Unhandled SDK message type: {type(sdk_message).__name__}")
-
-        return responses
-
-    def _ensure_text_started(self, responses: list[StreamBaseResponse]) -> None:
-        """Start (or restart) a text block if needed."""
-        if not self.has_started_text or self.has_ended_text:
-            if self.has_ended_text:
-                self.text_block_id = str(uuid.uuid4())
-                self.has_ended_text = False
-            responses.append(StreamTextStart(id=self.text_block_id))
-            self.has_started_text = True
-
-    def _end_text_if_open(self, responses: list[StreamBaseResponse]) -> None:
-        """End the current text block if one is open."""
-        if self.has_started_text and not self.has_ended_text:
-            responses.append(StreamTextEnd(id=self.text_block_id))
-            self.has_ended_text = True
-
-
-def _extract_tool_output(content: str | list[dict[str, str]] | None) -> str:
-    """Extract a string output from a ToolResultBlock's content field."""
-    if isinstance(content, str):
-        return content
-    if isinstance(content, list):
-        parts = [item.get("text", "") for item in content if item.get("type") == "text"]
-        if parts:
-            return "".join(parts)
-        try:
-            return json.dumps(content)
-        except (TypeError, ValueError):
-            return str(content)
-    if content is None:
-        return ""
-    try:
-        return json.dumps(content)
-    except (TypeError, ValueError):
-        return str(content)

autogpt_platform/backend/backend/api/features/chat/sdk/security_hooks_test.py

@@ -1,165 +0,0 @@
-"""Unit tests for SDK security hooks."""
-
-import os
-
-from .security_hooks import _validate_tool_access, _validate_user_isolation
-
-SDK_CWD = "/tmp/copilot-abc123"
-
-
-def _is_denied(result: dict) -> bool:
-    hook = result.get("hookSpecificOutput", {})
-    return hook.get("permissionDecision") == "deny"
-
-
-# -- Blocked tools -----------------------------------------------------------
-
-
-def test_blocked_tools_denied():
-    for tool in ("bash", "shell", "exec", "terminal", "command"):
-        result = _validate_tool_access(tool, {})
-        assert _is_denied(result), f"{tool} should be blocked"
-
-
-def test_unknown_tool_allowed():
-    result = _validate_tool_access("SomeCustomTool", {})
-    assert result == {}
-
-
-# -- Workspace-scoped tools --------------------------------------------------
-
-
-def test_read_within_workspace_allowed():
-    result = _validate_tool_access(
-        "Read", {"file_path": f"{SDK_CWD}/file.txt"}, sdk_cwd=SDK_CWD
-    )
-    assert result == {}
-
-
-def test_write_within_workspace_allowed():
-    result = _validate_tool_access(
-        "Write", {"file_path": f"{SDK_CWD}/output.json"}, sdk_cwd=SDK_CWD
-    )
-    assert result == {}
-
-
-def test_edit_within_workspace_allowed():
-    result = _validate_tool_access(
-        "Edit", {"file_path": f"{SDK_CWD}/src/main.py"}, sdk_cwd=SDK_CWD
-    )
-    assert result == {}
-
-
-def test_glob_within_workspace_allowed():
-    result = _validate_tool_access("Glob", {"path": f"{SDK_CWD}/src"}, sdk_cwd=SDK_CWD)
-    assert result == {}
-
-
-def test_grep_within_workspace_allowed():
-    result = _validate_tool_access("Grep", {"path": f"{SDK_CWD}/src"}, sdk_cwd=SDK_CWD)
-    assert result == {}
-
-
-def test_read_outside_workspace_denied():
-    result = _validate_tool_access(
-        "Read", {"file_path": "/etc/passwd"}, sdk_cwd=SDK_CWD
-    )
-    assert _is_denied(result)
-
-
-def test_write_outside_workspace_denied():
-    result = _validate_tool_access(
-        "Write", {"file_path": "/home/user/secrets.txt"}, sdk_cwd=SDK_CWD
-    )
-    assert _is_denied(result)
-
-
-def test_traversal_attack_denied():
-    result = _validate_tool_access(
-        "Read",
-        {"file_path": f"{SDK_CWD}/../../etc/passwd"},
-        sdk_cwd=SDK_CWD,
-    )
-    assert _is_denied(result)
-
-
-def test_no_path_allowed():
-    """Glob/Grep without a path argument defaults to cwd — should pass."""
-    result = _validate_tool_access("Glob", {}, sdk_cwd=SDK_CWD)
-    assert result == {}
-
-
-def test_read_no_cwd_denies_absolute():
-    """If no sdk_cwd is set, absolute paths are denied."""
-    result = _validate_tool_access("Read", {"file_path": "/tmp/anything"})
-    assert _is_denied(result)
-
-
-# -- Tool-results directory --------------------------------------------------
-
-
-def test_read_tool_results_allowed():
-    home = os.path.expanduser("~")
-    path = f"{home}/.claude/projects/-tmp-copilot-abc123/tool-results/12345.txt"
-    result = _validate_tool_access("Read", {"file_path": path}, sdk_cwd=SDK_CWD)
-    assert result == {}
-
-
-def test_read_claude_projects_without_tool_results_denied():
-    home = os.path.expanduser("~")
-    path = f"{home}/.claude/projects/-tmp-copilot-abc123/settings.json"
-    result = _validate_tool_access("Read", {"file_path": path}, sdk_cwd=SDK_CWD)
-    assert _is_denied(result)
-
-
-# -- Built-in Bash is blocked (use bash_exec MCP tool instead) ---------------
-
-
-def test_bash_builtin_always_blocked():
-    """SDK built-in Bash is blocked — bash_exec MCP tool with bubblewrap is used instead."""
-    result = _validate_tool_access("Bash", {"command": "echo hello"}, sdk_cwd=SDK_CWD)
-    assert _is_denied(result)
-
-
-# -- Dangerous patterns ------------------------------------------------------
-
-
-def test_dangerous_pattern_blocked():
-    result = _validate_tool_access("SomeTool", {"cmd": "sudo rm -rf /"})
-    assert _is_denied(result)
-
-
-def test_subprocess_pattern_blocked():
-    result = _validate_tool_access("SomeTool", {"code": "subprocess.run(...)"})
-    assert _is_denied(result)
-
-
-# -- User isolation ----------------------------------------------------------
-
-
-def test_workspace_path_traversal_blocked():
-    result = _validate_user_isolation(
-        "workspace_read", {"path": "../../../etc/shadow"}, user_id="user-1"
-    )
-    assert _is_denied(result)
-
-
-def test_workspace_absolute_path_blocked():
-    result = _validate_user_isolation(
-        "workspace_read", {"path": "/etc/passwd"}, user_id="user-1"
-    )
-    assert _is_denied(result)
-
-
-def test_workspace_normal_path_allowed():
-    result = _validate_user_isolation(
-        "workspace_read", {"path": "src/main.py"}, user_id="user-1"
-    )
-    assert result == {}
-
-
-def test_non_workspace_tool_passes_isolation():
-    result = _validate_user_isolation(
-        "find_agent", {"query": "email"}, user_id="user-1"
-    )
-    assert result == {}

autogpt_platform/backend/backend/api/features/chat/sdk/service.py

@@ -1,752 +0,0 @@
-"""Claude Agent SDK service layer for CoPilot chat completions."""
-
-import asyncio
-import json
-import logging
-import os
-import uuid
-from collections.abc import AsyncGenerator
-from dataclasses import dataclass
-from typing import Any
-
-from backend.util.exceptions import NotFoundError
-
-from .. import stream_registry
-from ..config import ChatConfig
-from ..model import (
-    ChatMessage,
-    ChatSession,
-    get_chat_session,
-    update_session_title,
-    upsert_chat_session,
-)
-from ..response_model import (
-    StreamBaseResponse,
-    StreamError,
-    StreamFinish,
-    StreamStart,
-    StreamTextDelta,
-    StreamToolInputAvailable,
-    StreamToolOutputAvailable,
-)
-from ..service import (
-    _build_system_prompt,
-    _execute_long_running_tool_with_streaming,
-    _generate_session_title,
-)
-from ..tools.models import OperationPendingResponse, OperationStartedResponse
-from ..tools.sandbox import WORKSPACE_PREFIX, make_session_path
-from ..tracking import track_user_message
-from .response_adapter import SDKResponseAdapter
-from .security_hooks import create_security_hooks
-from .tool_adapter import (
-    COPILOT_TOOL_NAMES,
-    SDK_DISALLOWED_TOOLS,
-    LongRunningCallback,
-    create_copilot_mcp_server,
-    set_execution_context,
-)
-from .transcript import (
-    download_transcript,
-    read_transcript_file,
-    upload_transcript,
-    validate_transcript,
-    write_transcript_to_tempfile,
-)
-
-logger = logging.getLogger(__name__)
-config = ChatConfig()
-
-# Set to hold background tasks to prevent garbage collection
-_background_tasks: set[asyncio.Task[Any]] = set()
-
-
-@dataclass
-class CapturedTranscript:
-    """Info captured by the SDK Stop hook for stateless --resume."""
-
-    path: str = ""
-    sdk_session_id: str = ""
-
-    @property
-    def available(self) -> bool:
-        return bool(self.path)
-
-
-_SDK_CWD_PREFIX = WORKSPACE_PREFIX
-
-# Appended to the system prompt to inform the agent about available tools.
-# The SDK built-in Bash is NOT available — use mcp__copilot__bash_exec instead,
-# which has kernel-level network isolation (unshare --net).
-_SDK_TOOL_SUPPLEMENT = """
-
-## Tool notes
-
-- The SDK built-in Bash tool is NOT available.  Use the `bash_exec` MCP tool
-  for shell commands — it runs in a network-isolated sandbox.
-- **Shared workspace**: The SDK Read/Write tools and `bash_exec` share the
-  same working directory. Files created by one are readable by the other.
-  These files are **ephemeral** — they exist only for the current session.
-- **Persistent storage**: Use `write_workspace_file` / `read_workspace_file`
-  for files that should persist across sessions (stored in cloud storage).
-- Long-running tools (create_agent, edit_agent, etc.) are handled
-  asynchronously.  You will receive an immediate response; the actual result
-  is delivered to the user via a background stream.
-"""
-
-
-def _build_long_running_callback(user_id: str | None) -> LongRunningCallback:
-    """Build a callback that delegates long-running tools to the non-SDK infrastructure.
-
-    Long-running tools (create_agent, edit_agent, etc.) are delegated to the
-    existing background infrastructure: stream_registry (Redis Streams),
-    database persistence, and SSE reconnection.  This means results survive
-    page refreshes / pod restarts, and the frontend shows the proper loading
-    widget with progress updates.
-
-    The returned callback matches the ``LongRunningCallback`` signature:
-    ``(tool_name, args, session) -> MCP response dict``.
-    """
-
-    async def _callback(
-        tool_name: str, args: dict[str, Any], session: ChatSession
-    ) -> dict[str, Any]:
-        operation_id = str(uuid.uuid4())
-        task_id = str(uuid.uuid4())
-        tool_call_id = f"sdk-{uuid.uuid4().hex[:12]}"
-        session_id = session.session_id
-
-        # --- Build user-friendly messages (matches non-SDK service) ---
-        if tool_name == "create_agent":
-            desc = args.get("description", "")
-            desc_preview = (desc[:100] + "...") if len(desc) > 100 else desc
-            pending_msg = (
-                f"Creating your agent: {desc_preview}"
-                if desc_preview
-                else "Creating agent... This may take a few minutes."
-            )
-            started_msg = (
-                "Agent creation started. You can close this tab - "
-                "check your library in a few minutes."
-            )
-        elif tool_name == "edit_agent":
-            changes = args.get("changes", "")
-            changes_preview = (changes[:100] + "...") if len(changes) > 100 else changes
-            pending_msg = (
-                f"Editing agent: {changes_preview}"
-                if changes_preview
-                else "Editing agent... This may take a few minutes."
-            )
-            started_msg = (
-                "Agent edit started. You can close this tab - "
-                "check your library in a few minutes."
-            )
-        else:
-            pending_msg = f"Running {tool_name}... This may take a few minutes."
-            started_msg = (
-                f"{tool_name} started. You can close this tab - "
-                "check back in a few minutes."
-            )
-
-        # --- Register task in Redis for SSE reconnection ---
-        await stream_registry.create_task(
-            task_id=task_id,
-            session_id=session_id,
-            user_id=user_id,
-            tool_call_id=tool_call_id,
-            tool_name=tool_name,
-            operation_id=operation_id,
-        )
-
-        # --- Save OperationPendingResponse to chat history ---
-        pending_message = ChatMessage(
-            role="tool",
-            content=OperationPendingResponse(
-                message=pending_msg,
-                operation_id=operation_id,
-                tool_name=tool_name,
-            ).model_dump_json(),
-            tool_call_id=tool_call_id,
-        )
-        session.messages.append(pending_message)
-        await upsert_chat_session(session)
-
-        # --- Spawn background task (reuses non-SDK infrastructure) ---
-        bg_task = asyncio.create_task(
-            _execute_long_running_tool_with_streaming(
-                tool_name=tool_name,
-                parameters=args,
-                tool_call_id=tool_call_id,
-                operation_id=operation_id,
-                task_id=task_id,
-                session_id=session_id,
-                user_id=user_id,
-            )
-        )
-        _background_tasks.add(bg_task)
-        bg_task.add_done_callback(_background_tasks.discard)
-        await stream_registry.set_task_asyncio_task(task_id, bg_task)
-
-        logger.info(
-            f"[SDK] Long-running tool {tool_name} delegated to background "
-            f"(operation_id={operation_id}, task_id={task_id})"
-        )
-
-        # --- Return OperationStartedResponse as MCP tool result ---
-        # This flows through SDK → response adapter → frontend, triggering
-        # the loading widget with SSE reconnection support.
-        started_json = OperationStartedResponse(
-            message=started_msg,
-            operation_id=operation_id,
-            tool_name=tool_name,
-            task_id=task_id,
-        ).model_dump_json()
-
-        return {
-            "content": [{"type": "text", "text": started_json}],
-            "isError": False,
-        }
-
-    return _callback
-
-
-def _resolve_sdk_model() -> str | None:
-    """Resolve the model name for the Claude Agent SDK CLI.
-
-    Uses ``config.claude_agent_model`` if set, otherwise derives from
-    ``config.model`` by stripping the OpenRouter provider prefix (e.g.,
-    ``"anthropic/claude-opus-4.6"`` → ``"claude-opus-4.6"``).
-    """
-    if config.claude_agent_model:
-        return config.claude_agent_model
-    model = config.model
-    if "/" in model:
-        return model.split("/", 1)[1]
-    return model
-
-
-def _build_sdk_env() -> dict[str, str]:
-    """Build env vars for the SDK CLI process.
-
-    Routes API calls through OpenRouter (or a custom base_url) using
-    the same ``config.api_key`` / ``config.base_url`` as the non-SDK path.
-    This gives per-call token and cost tracking on the OpenRouter dashboard.
-
-    Only overrides ``ANTHROPIC_API_KEY`` when a valid proxy URL and auth
-    token are both present — otherwise returns an empty dict so the SDK
-    falls back to its default credentials.
-    """
-    env: dict[str, str] = {}
-    if config.api_key and config.base_url:
-        # Strip /v1 suffix — SDK expects the base URL without a version path
-        base = config.base_url.rstrip("/")
-        if base.endswith("/v1"):
-            base = base[:-3]
-        if not base or not base.startswith("http"):
-            # Invalid base_url — don't override SDK defaults
-            return env
-        env["ANTHROPIC_BASE_URL"] = base
-        env["ANTHROPIC_AUTH_TOKEN"] = config.api_key
-        # Must be explicitly empty so the CLI uses AUTH_TOKEN instead
-        env["ANTHROPIC_API_KEY"] = ""
-    return env
-
-
-def _make_sdk_cwd(session_id: str) -> str:
-    """Create a safe, session-specific working directory path.
-
-    Delegates to :func:`~backend.api.features.chat.tools.sandbox.make_session_path`
-    (single source of truth for path sanitization) and adds a defence-in-depth
-    assertion.
-    """
-    cwd = make_session_path(session_id)
-    # Defence-in-depth: normpath + startswith is a CodeQL-recognised sanitizer
-    cwd = os.path.normpath(cwd)
-    if not cwd.startswith(_SDK_CWD_PREFIX):
-        raise ValueError(f"SDK cwd escaped prefix: {cwd}")
-    return cwd
-
-
-def _cleanup_sdk_tool_results(cwd: str) -> None:
-    """Remove SDK tool-result files for a specific session working directory.
-
-    The SDK creates tool-result files under ~/.claude/projects/<encoded-cwd>/tool-results/.
-    We clean only the specific cwd's results to avoid race conditions between
-    concurrent sessions.
-
-    Security: cwd MUST be created by _make_sdk_cwd() which sanitizes session_id.
-    """
-    import shutil
-
-    # Validate cwd is under the expected prefix
-    normalized = os.path.normpath(cwd)
-    if not normalized.startswith(_SDK_CWD_PREFIX):
-        logger.warning(f"[SDK] Rejecting cleanup for path outside workspace: {cwd}")
-        return
-
-    # SDK encodes the cwd path by replacing '/' with '-'
-    encoded_cwd = normalized.replace("/", "-")
-
-    # Construct the project directory path (known-safe home expansion)
-    claude_projects = os.path.expanduser("~/.claude/projects")
-    project_dir = os.path.join(claude_projects, encoded_cwd)
-
-    # Security check 3: Validate project_dir is under ~/.claude/projects
-    project_dir = os.path.normpath(project_dir)
-    if not project_dir.startswith(claude_projects):
-        logger.warning(
-            f"[SDK] Rejecting cleanup for escaped project path: {project_dir}"
-        )
-        return
-
-    results_dir = os.path.join(project_dir, "tool-results")
-    if os.path.isdir(results_dir):
-        for filename in os.listdir(results_dir):
-            file_path = os.path.join(results_dir, filename)
-            try:
-                if os.path.isfile(file_path):
-                    os.remove(file_path)
-            except OSError:
-                pass
-
-    # Also clean up the temp cwd directory itself
-    try:
-        shutil.rmtree(normalized, ignore_errors=True)
-    except OSError:
-        pass
-
-
-async def _compress_conversation_history(
-    session: ChatSession,
-) -> list[ChatMessage]:
-    """Compress prior conversation messages if they exceed the token threshold.
-
-    Uses the shared compress_context() from prompt.py which supports:
-    - LLM summarization of old messages (keeps recent ones intact)
-    - Progressive content truncation as fallback
-    - Middle-out deletion as last resort
-
-    Returns the compressed prior messages (everything except the current message).
-    """
-    prior = session.messages[:-1]
-    if len(prior) < 2:
-        return prior
-
-    from backend.util.prompt import compress_context
-
-    # Convert ChatMessages to dicts for compress_context
-    messages_dict = []
-    for msg in prior:
-        msg_dict: dict[str, Any] = {"role": msg.role}
-        if msg.content:
-            msg_dict["content"] = msg.content
-        if msg.tool_calls:
-            msg_dict["tool_calls"] = msg.tool_calls
-        if msg.tool_call_id:
-            msg_dict["tool_call_id"] = msg.tool_call_id
-        messages_dict.append(msg_dict)
-
-    try:
-        import openai
-
-        async with openai.AsyncOpenAI(
-            api_key=config.api_key, base_url=config.base_url, timeout=30.0
-        ) as client:
-            result = await compress_context(
-                messages=messages_dict,
-                model=config.model,
-                client=client,
-            )
-    except Exception as e:
-        logger.warning(f"[SDK] Context compression with LLM failed: {e}")
-        # Fall back to truncation-only (no LLM summarization)
-        result = await compress_context(
-            messages=messages_dict,
-            model=config.model,
-            client=None,
-        )
-
-    if result.was_compacted:
-        logger.info(
-            f"[SDK] Context compacted: {result.original_token_count} -> "
-            f"{result.token_count} tokens "
-            f"({result.messages_summarized} summarized, "
-            f"{result.messages_dropped} dropped)"
-        )
-        # Convert compressed dicts back to ChatMessages
-        return [
-            ChatMessage(
-                role=m["role"],
-                content=m.get("content"),
-                tool_calls=m.get("tool_calls"),
-                tool_call_id=m.get("tool_call_id"),
-            )
-            for m in result.messages
-        ]
-
-    return prior
-
-
-def _format_conversation_context(messages: list[ChatMessage]) -> str | None:
-    """Format conversation messages into a context prefix for the user message.
-
-    Returns a string like:
-        <conversation_history>
-        User: hello
-        You responded: Hi! How can I help?
-        </conversation_history>
-
-    Returns None if there are no messages to format.
-    """
-    if not messages:
-        return None
-
-    lines: list[str] = []
-    for msg in messages:
-        if not msg.content:
-            continue
-        if msg.role == "user":
-            lines.append(f"User: {msg.content}")
-        elif msg.role == "assistant":
-            lines.append(f"You responded: {msg.content}")
-        # Skip tool messages — they're internal details
-
-    if not lines:
-        return None
-
-    return "<conversation_history>\n" + "\n".join(lines) + "\n</conversation_history>"
-
-
-async def stream_chat_completion_sdk(
-    session_id: str,
-    message: str | None = None,
-    tool_call_response: str | None = None,  # noqa: ARG001
-    is_user_message: bool = True,
-    user_id: str | None = None,
-    retry_count: int = 0,  # noqa: ARG001
-    session: ChatSession | None = None,
-    context: dict[str, str] | None = None,  # noqa: ARG001
-) -> AsyncGenerator[StreamBaseResponse, None]:
-    """Stream chat completion using Claude Agent SDK.
-
-    Drop-in replacement for stream_chat_completion with improved reliability.
-    """
-
-    if session is None:
-        session = await get_chat_session(session_id, user_id)
-
-    if not session:
-        raise NotFoundError(
-            f"Session {session_id} not found. Please create a new session first."
-        )
-
-    if message:
-        session.messages.append(
-            ChatMessage(
-                role="user" if is_user_message else "assistant", content=message
-            )
-        )
-        if is_user_message:
-            track_user_message(
-                user_id=user_id, session_id=session_id, message_length=len(message)
-            )
-
-    session = await upsert_chat_session(session)
-
-    # Generate title for new sessions (first user message)
-    if is_user_message and not session.title:
-        user_messages = [m for m in session.messages if m.role == "user"]
-        if len(user_messages) == 1:
-            first_message = user_messages[0].content or message or ""
-            if first_message:
-                task = asyncio.create_task(
-                    _update_title_async(session_id, first_message, user_id)
-                )
-                _background_tasks.add(task)
-                task.add_done_callback(_background_tasks.discard)
-
-    # Build system prompt (reuses non-SDK path with Langfuse support)
-    has_history = len(session.messages) > 1
-    system_prompt, _ = await _build_system_prompt(
-        user_id, has_conversation_history=has_history
-    )
-    system_prompt += _SDK_TOOL_SUPPLEMENT
-    message_id = str(uuid.uuid4())
-    task_id = str(uuid.uuid4())
-
-    yield StreamStart(messageId=message_id, taskId=task_id)
-
-    stream_completed = False
-    # Initialise sdk_cwd before the try so the finally can reference it
-    # even if _make_sdk_cwd raises (in that case it stays as "").
-    sdk_cwd = ""
-    use_resume = False
-
-    try:
-        # Use a session-specific temp dir to avoid cleanup race conditions
-        # between concurrent sessions.
-        sdk_cwd = _make_sdk_cwd(session_id)
-        os.makedirs(sdk_cwd, exist_ok=True)
-
-        set_execution_context(
-            user_id,
-            session,
-            long_running_callback=_build_long_running_callback(user_id),
-        )
-        try:
-            from claude_agent_sdk import ClaudeAgentOptions, ClaudeSDKClient
-
-            # Fail fast when no API credentials are available at all
-            sdk_env = _build_sdk_env()
-            if not sdk_env and not os.environ.get("ANTHROPIC_API_KEY"):
-                raise RuntimeError(
-                    "No API key configured. Set OPEN_ROUTER_API_KEY "
-                    "(or CHAT_API_KEY) for OpenRouter routing, "
-                    "or ANTHROPIC_API_KEY for direct Anthropic access."
-                )
-
-            mcp_server = create_copilot_mcp_server()
-
-            sdk_model = _resolve_sdk_model()
-
-            # --- Transcript capture via Stop hook ---
-            captured_transcript = CapturedTranscript()
-
-            def _on_stop(transcript_path: str, sdk_session_id: str) -> None:
-                captured_transcript.path = transcript_path
-                captured_transcript.sdk_session_id = sdk_session_id
-
-            security_hooks = create_security_hooks(
-                user_id,
-                sdk_cwd=sdk_cwd,
-                max_subtasks=config.claude_agent_max_subtasks,
-                on_stop=_on_stop if config.claude_agent_use_resume else None,
-            )
-
-            # --- Resume strategy: download transcript from bucket ---
-            resume_file: str | None = None
-            use_resume = False
-
-            if config.claude_agent_use_resume and user_id and len(session.messages) > 1:
-                transcript_content = await download_transcript(user_id, session_id)
-                if transcript_content and validate_transcript(transcript_content):
-                    resume_file = write_transcript_to_tempfile(
-                        transcript_content, session_id, sdk_cwd
-                    )
-                    if resume_file:
-                        use_resume = True
-                        logger.info(
-                            f"[SDK] Using --resume with transcript "
-                            f"({len(transcript_content)} bytes)"
-                        )
-
-            sdk_options_kwargs: dict[str, Any] = {
-                "system_prompt": system_prompt,
-                "mcp_servers": {"copilot": mcp_server},
-                "allowed_tools": COPILOT_TOOL_NAMES,
-                "disallowed_tools": SDK_DISALLOWED_TOOLS,
-                "hooks": security_hooks,
-                "cwd": sdk_cwd,
-                "max_buffer_size": config.claude_agent_max_buffer_size,
-            }
-            if sdk_env:
-                sdk_options_kwargs["model"] = sdk_model
-                sdk_options_kwargs["env"] = sdk_env
-            if use_resume and resume_file:
-                sdk_options_kwargs["resume"] = resume_file
-
-            options = ClaudeAgentOptions(**sdk_options_kwargs)  # type: ignore[arg-type]
-
-            adapter = SDKResponseAdapter(message_id=message_id)
-            adapter.set_task_id(task_id)
-
-            async with ClaudeSDKClient(options=options) as client:
-                current_message = message or ""
-                if not current_message and session.messages:
-                    last_user = [m for m in session.messages if m.role == "user"]
-                    if last_user:
-                        current_message = last_user[-1].content or ""
-
-                if not current_message.strip():
-                    yield StreamError(
-                        errorText="Message cannot be empty.",
-                        code="empty_prompt",
-                    )
-                    yield StreamFinish()
-                    return
-
-                # Build query: with --resume the CLI already has full
-                # context, so we only send the new message.  Without
-                # resume, compress history into a context prefix.
-                query_message = current_message
-                if not use_resume and len(session.messages) > 1:
-                    logger.warning(
-                        f"[SDK] Using compression fallback for session "
-                        f"{session_id} ({len(session.messages)} messages) — "
-                        f"no transcript available for --resume"
-                    )
-                    compressed = await _compress_conversation_history(session)
-                    history_context = _format_conversation_context(compressed)
-                    if history_context:
-                        query_message = (
-                            f"{history_context}\n\n"
-                            f"Now, the user says:\n{current_message}"
-                        )
-
-                logger.info(
-                    f"[SDK] Sending query ({len(session.messages)} msgs in session)"
-                )
-                logger.debug(f"[SDK] Query preview: {current_message[:80]!r}")
-                await client.query(query_message, session_id=session_id)
-
-                assistant_response = ChatMessage(role="assistant", content="")
-                accumulated_tool_calls: list[dict[str, Any]] = []
-                has_appended_assistant = False
-                has_tool_results = False
-
-                async for sdk_msg in client.receive_messages():
-                    logger.debug(
-                        f"[SDK] Received: {type(sdk_msg).__name__} "
-                        f"{getattr(sdk_msg, 'subtype', '')}"
-                    )
-                    for response in adapter.convert_message(sdk_msg):
-                        if isinstance(response, StreamStart):
-                            continue
-
-                        yield response
-
-                        if isinstance(response, StreamTextDelta):
-                            delta = response.delta or ""
-                            # After tool results, start a new assistant
-                            # message for the post-tool text.
-                            if has_tool_results and has_appended_assistant:
-                                assistant_response = ChatMessage(
-                                    role="assistant", content=delta
-                                )
-                                accumulated_tool_calls = []
-                                has_appended_assistant = False
-                                has_tool_results = False
-                                session.messages.append(assistant_response)
-                                has_appended_assistant = True
-                            else:
-                                assistant_response.content = (
-                                    assistant_response.content or ""
-                                ) + delta
-                                if not has_appended_assistant:
-                                    session.messages.append(assistant_response)
-                                    has_appended_assistant = True
-
-                        elif isinstance(response, StreamToolInputAvailable):
-                            accumulated_tool_calls.append(
-                                {
-                                    "id": response.toolCallId,
-                                    "type": "function",
-                                    "function": {
-                                        "name": response.toolName,
-                                        "arguments": json.dumps(response.input or {}),
-                                    },
-                                }
-                            )
-                            assistant_response.tool_calls = accumulated_tool_calls
-                            if not has_appended_assistant:
-                                session.messages.append(assistant_response)
-                                has_appended_assistant = True
-
-                        elif isinstance(response, StreamToolOutputAvailable):
-                            session.messages.append(
-                                ChatMessage(
-                                    role="tool",
-                                    content=(
-                                        response.output
-                                        if isinstance(response.output, str)
-                                        else str(response.output)
-                                    ),
-                                    tool_call_id=response.toolCallId,
-                                )
-                            )
-                            has_tool_results = True
-
-                        elif isinstance(response, StreamFinish):
-                            stream_completed = True
-
-                    if stream_completed:
-                        break
-
-                if (
-                    assistant_response.content or assistant_response.tool_calls
-                ) and not has_appended_assistant:
-                    session.messages.append(assistant_response)
-
-                # --- Capture transcript while CLI is still alive ---
-                # Must happen INSIDE async with: close() sends SIGTERM
-                # which kills the CLI before it can flush the JSONL.
-                if (
-                    config.claude_agent_use_resume
-                    and user_id
-                    and captured_transcript.available
-                ):
-                    # Give CLI time to flush JSONL writes before we read
-                    await asyncio.sleep(0.5)
-                    raw_transcript = read_transcript_file(captured_transcript.path)
-                    if raw_transcript:
-                        task = asyncio.create_task(
-                            _upload_transcript_bg(user_id, session_id, raw_transcript)
-                        )
-                        _background_tasks.add(task)
-                        task.add_done_callback(_background_tasks.discard)
-                    else:
-                        logger.debug("[SDK] Stop hook fired but transcript not usable")
-
-        except ImportError:
-            raise RuntimeError(
-                "claude-agent-sdk is not installed. "
-                "Disable SDK mode (CHAT_USE_CLAUDE_AGENT_SDK=false) "
-                "to use the OpenAI-compatible fallback."
-            )
-
-        await upsert_chat_session(session)
-        logger.debug(
-            f"[SDK] Session {session_id} saved with {len(session.messages)} messages"
-        )
-        if not stream_completed:
-            yield StreamFinish()
-
-    except Exception as e:
-        logger.error(f"[SDK] Error: {e}", exc_info=True)
-        try:
-            await upsert_chat_session(session)
-        except Exception as save_err:
-            logger.error(f"[SDK] Failed to save session on error: {save_err}")
-        yield StreamError(
-            errorText="An error occurred. Please try again.",
-            code="sdk_error",
-        )
-        yield StreamFinish()
-    finally:
-        if sdk_cwd:
-            _cleanup_sdk_tool_results(sdk_cwd)
-
-
-async def _upload_transcript_bg(
-    user_id: str, session_id: str, raw_content: str
-) -> None:
-    """Background task to strip progress entries and upload transcript."""
-    try:
-        await upload_transcript(user_id, session_id, raw_content)
-    except Exception as e:
-        logger.error(f"[SDK] Failed to upload transcript for {session_id}: {e}")
-
-
-async def _update_title_async(
-    session_id: str, message: str, user_id: str | None = None
-) -> None:
-    """Background task to update session title."""
-    try:
-        title = await _generate_session_title(
-            message, user_id=user_id, session_id=session_id
-        )
-        if title:
-            await update_session_title(session_id, title)
-            logger.debug(f"[SDK] Generated title for {session_id}: {title}")
-    except Exception as e:
-        logger.warning(f"[SDK] Failed to update session title: {e}")

autogpt_platform/backend/backend/api/features/chat/sdk/tool_adapter.py

@@ -1,363 +0,0 @@
-"""Tool adapter for wrapping existing CoPilot tools as Claude Agent SDK MCP tools.
-
-This module provides the adapter layer that converts existing BaseTool implementations
-into in-process MCP tools that can be used with the Claude Agent SDK.
-
-Long-running tools (``is_long_running=True``) are delegated to the non-SDK
-background infrastructure (stream_registry, Redis persistence, SSE reconnection)
-via a callback provided by the service layer.  This avoids wasteful SDK polling
-and makes results survive page refreshes.
-"""
-
-import itertools
-import json
-import logging
-import os
-import uuid
-from collections.abc import Awaitable, Callable
-from contextvars import ContextVar
-from typing import Any
-
-from backend.api.features.chat.model import ChatSession
-from backend.api.features.chat.tools import TOOL_REGISTRY
-from backend.api.features.chat.tools.base import BaseTool
-
-logger = logging.getLogger(__name__)
-
-# Allowed base directory for the Read tool (SDK saves oversized tool results here).
-# Restricted to ~/.claude/projects/ and further validated to require "tool-results"
-# in the path — prevents reading settings, credentials, or other sensitive files.
-_SDK_PROJECTS_DIR = os.path.expanduser("~/.claude/projects/")
-
-# MCP server naming - the SDK prefixes tool names as "mcp__{server_name}__{tool}"
-MCP_SERVER_NAME = "copilot"
-MCP_TOOL_PREFIX = f"mcp__{MCP_SERVER_NAME}__"
-
-# Context variables to pass user/session info to tool execution
-_current_user_id: ContextVar[str | None] = ContextVar("current_user_id", default=None)
-_current_session: ContextVar[ChatSession | None] = ContextVar(
-    "current_session", default=None
-)
-# Stash for MCP tool outputs before the SDK potentially truncates them.
-# Keyed by tool_name → full output string. Consumed (popped) by the
-# response adapter when it builds StreamToolOutputAvailable.
-_pending_tool_outputs: ContextVar[dict[str, str]] = ContextVar(
-    "pending_tool_outputs", default=None  # type: ignore[arg-type]
-)
-
-# Callback type for delegating long-running tools to the non-SDK infrastructure.
-# Args: (tool_name, arguments, session) → MCP-formatted response dict.
-LongRunningCallback = Callable[
-    [str, dict[str, Any], ChatSession], Awaitable[dict[str, Any]]
-]
-
-# ContextVar so the service layer can inject the callback per-request.
-_long_running_callback: ContextVar[LongRunningCallback | None] = ContextVar(
-    "long_running_callback", default=None
-)
-
-
-def set_execution_context(
-    user_id: str | None,
-    session: ChatSession,
-    long_running_callback: LongRunningCallback | None = None,
-) -> None:
-    """Set the execution context for tool calls.
-
-    This must be called before streaming begins to ensure tools have access
-    to user_id and session information.
-
-    Args:
-        user_id: Current user's ID.
-        session: Current chat session.
-        long_running_callback: Optional callback to delegate long-running tools
-            to the non-SDK background infrastructure (stream_registry + Redis).
-    """
-    _current_user_id.set(user_id)
-    _current_session.set(session)
-    _pending_tool_outputs.set({})
-    _long_running_callback.set(long_running_callback)
-
-
-def get_execution_context() -> tuple[str | None, ChatSession | None]:
-    """Get the current execution context."""
-    return (
-        _current_user_id.get(),
-        _current_session.get(),
-    )
-
-
-def pop_pending_tool_output(tool_name: str) -> str | None:
-    """Pop and return the stashed full output for *tool_name*.
-
-    The SDK CLI may truncate large tool results (writing them to disk and
-    replacing the content with a file reference). This stash keeps the
-    original MCP output so the response adapter can forward it to the
-    frontend for proper widget rendering.
-
-    Returns ``None`` if nothing was stashed for *tool_name*.
-    """
-    pending = _pending_tool_outputs.get(None)
-    if pending is None:
-        return None
-    return pending.pop(tool_name, None)
-
-
-async def _execute_tool_sync(
-    base_tool: BaseTool,
-    user_id: str | None,
-    session: ChatSession,
-    args: dict[str, Any],
-) -> dict[str, Any]:
-    """Execute a tool synchronously and return MCP-formatted response."""
-    effective_id = f"sdk-{uuid.uuid4().hex[:12]}"
-    result = await base_tool.execute(
-        user_id=user_id,
-        session=session,
-        tool_call_id=effective_id,
-        **args,
-    )
-
-    text = (
-        result.output if isinstance(result.output, str) else json.dumps(result.output)
-    )
-
-    # Stash the full output before the SDK potentially truncates it.
-    pending = _pending_tool_outputs.get(None)
-    if pending is not None:
-        pending[base_tool.name] = text
-
-    return {
-        "content": [{"type": "text", "text": text}],
-        "isError": not result.success,
-    }
-
-
-def _mcp_error(message: str) -> dict[str, Any]:
-    return {
-        "content": [
-            {"type": "text", "text": json.dumps({"error": message, "type": "error"})}
-        ],
-        "isError": True,
-    }
-
-
-def create_tool_handler(base_tool: BaseTool):
-    """Create an async handler function for a BaseTool.
-
-    This wraps the existing BaseTool._execute method to be compatible
-    with the Claude Agent SDK MCP tool format.
-
-    Long-running tools (``is_long_running=True``) are delegated to the
-    non-SDK background infrastructure via a callback set in the execution
-    context.  The callback persists the operation in Redis (stream_registry)
-    so results survive page refreshes and pod restarts.
-    """
-
-    async def tool_handler(args: dict[str, Any]) -> dict[str, Any]:
-        """Execute the wrapped tool and return MCP-formatted response."""
-        user_id, session = get_execution_context()
-
-        if session is None:
-            return _mcp_error("No session context available")
-
-        # --- Long-running: delegate to non-SDK background infrastructure ---
-        if base_tool.is_long_running:
-            callback = _long_running_callback.get(None)
-            if callback:
-                try:
-                    return await callback(base_tool.name, args, session)
-                except Exception as e:
-                    logger.error(
-                        f"Long-running callback failed for {base_tool.name}: {e}",
-                        exc_info=True,
-                    )
-                    return _mcp_error(f"Failed to start {base_tool.name}: {e}")
-            # No callback — fall through to synchronous execution
-            logger.warning(
-                f"[SDK] No long-running callback for {base_tool.name}, "
-                f"executing synchronously (may block)"
-            )
-
-        # --- Normal (fast) tool: execute synchronously ---
-        try:
-            return await _execute_tool_sync(base_tool, user_id, session, args)
-        except Exception as e:
-            logger.error(f"Error executing tool {base_tool.name}: {e}", exc_info=True)
-            return _mcp_error(f"Failed to execute {base_tool.name}: {e}")
-
-    return tool_handler
-
-
-def _build_input_schema(base_tool: BaseTool) -> dict[str, Any]:
-    """Build a JSON Schema input schema for a tool."""
-    return {
-        "type": "object",
-        "properties": base_tool.parameters.get("properties", {}),
-        "required": base_tool.parameters.get("required", []),
-    }
-
-
-async def _read_file_handler(args: dict[str, Any]) -> dict[str, Any]:
-    """Read a file with optional offset/limit. Restricted to SDK working directory.
-
-    After reading, the file is deleted to prevent accumulation in long-running pods.
-    """
-    file_path = args.get("file_path", "")
-    offset = args.get("offset", 0)
-    limit = args.get("limit", 2000)
-
-    # Security: only allow reads under ~/.claude/projects/**/tool-results/
-    real_path = os.path.realpath(file_path)
-    if not real_path.startswith(_SDK_PROJECTS_DIR) or "tool-results" not in real_path:
-        return {
-            "content": [{"type": "text", "text": f"Access denied: {file_path}"}],
-            "isError": True,
-        }
-
-    try:
-        with open(real_path) as f:
-            selected = list(itertools.islice(f, offset, offset + limit))
-        content = "".join(selected)
-        # Cleanup happens in _cleanup_sdk_tool_results after session ends;
-        # don't delete here — the SDK may read in multiple chunks.
-        return {"content": [{"type": "text", "text": content}], "isError": False}
-    except FileNotFoundError:
-        return {
-            "content": [{"type": "text", "text": f"File not found: {file_path}"}],
-            "isError": True,
-        }
-    except Exception as e:
-        return {
-            "content": [{"type": "text", "text": f"Error reading file: {e}"}],
-            "isError": True,
-        }
-
-
-_READ_TOOL_NAME = "Read"
-_READ_TOOL_DESCRIPTION = (
-    "Read a file from the local filesystem. "
-    "Use offset and limit to read specific line ranges for large files."
-)
-_READ_TOOL_SCHEMA = {
-    "type": "object",
-    "properties": {
-        "file_path": {
-            "type": "string",
-            "description": "The absolute path to the file to read",
-        },
-        "offset": {
-            "type": "integer",
-            "description": "Line number to start reading from (0-indexed). Default: 0",
-        },
-        "limit": {
-            "type": "integer",
-            "description": "Number of lines to read. Default: 2000",
-        },
-    },
-    "required": ["file_path"],
-}
-
-
-# Create the MCP server configuration
-def create_copilot_mcp_server():
-    """Create an in-process MCP server configuration for CoPilot tools.
-
-    This can be passed to ClaudeAgentOptions.mcp_servers.
-
-    Note: The actual SDK MCP server creation depends on the claude-agent-sdk
-    package being available. This function returns the configuration that
-    can be used with the SDK.
-    """
-    try:
-        from claude_agent_sdk import create_sdk_mcp_server, tool
-
-        # Create decorated tool functions
-        sdk_tools = []
-
-        for tool_name, base_tool in TOOL_REGISTRY.items():
-            handler = create_tool_handler(base_tool)
-            decorated = tool(
-                tool_name,
-                base_tool.description,
-                _build_input_schema(base_tool),
-            )(handler)
-            sdk_tools.append(decorated)
-
-        # Add the Read tool so the SDK can read back oversized tool results
-        read_tool = tool(
-            _READ_TOOL_NAME,
-            _READ_TOOL_DESCRIPTION,
-            _READ_TOOL_SCHEMA,
-        )(_read_file_handler)
-        sdk_tools.append(read_tool)
-
-        server = create_sdk_mcp_server(
-            name=MCP_SERVER_NAME,
-            version="1.0.0",
-            tools=sdk_tools,
-        )
-
-        return server
-
-    except ImportError:
-        # Let ImportError propagate so service.py handles the fallback
-        raise
-
-
-# SDK built-in tools allowed within the workspace directory.
-# Security hooks validate that file paths stay within sdk_cwd.
-# Bash is NOT included — use the sandboxed MCP bash_exec tool instead,
-# which provides kernel-level network isolation via unshare --net.
-# Task allows spawning sub-agents (rate-limited by security hooks).
-# WebSearch uses Brave Search via Anthropic's API — safe, no SSRF risk.
-_SDK_BUILTIN_TOOLS = ["Read", "Write", "Edit", "Glob", "Grep", "Task", "WebSearch"]
-
-# SDK built-in tools that must be explicitly blocked.
-# Bash: dangerous — agent uses mcp__copilot__bash_exec with kernel-level
-#   network isolation (unshare --net) instead.
-# WebFetch: SSRF risk — can reach internal network (localhost, 10.x, etc.).
-#   Agent uses the SSRF-protected mcp__copilot__web_fetch tool instead.
-SDK_DISALLOWED_TOOLS = ["Bash", "WebFetch"]
-
-# Tools that are blocked entirely in security hooks (defence-in-depth).
-# Includes SDK_DISALLOWED_TOOLS plus common aliases/synonyms.
-BLOCKED_TOOLS = {
-    *SDK_DISALLOWED_TOOLS,
-    "bash",
-    "shell",
-    "exec",
-    "terminal",
-    "command",
-}
-
-# Tools allowed only when their path argument stays within the SDK workspace.
-# The SDK uses these to handle oversized tool results (writes to tool-results/
-# files, then reads them back) and for workspace file operations.
-WORKSPACE_SCOPED_TOOLS = {"Read", "Write", "Edit", "Glob", "Grep"}
-
-# Dangerous patterns in tool inputs
-DANGEROUS_PATTERNS = [
-    r"sudo",
-    r"rm\s+-rf",
-    r"dd\s+if=",
-    r"/etc/passwd",
-    r"/etc/shadow",
-    r"chmod\s+777",
-    r"curl\s+.*\|.*sh",
-    r"wget\s+.*\|.*sh",
-    r"eval\s*\(",
-    r"exec\s*\(",
-    r"__import__",
-    r"os\.system",
-    r"subprocess",
-]
-
-# List of tool names for allowed_tools configuration
-# Include MCP tools, the MCP Read tool for oversized results,
-# and SDK built-in file tools for workspace operations.
-COPILOT_TOOL_NAMES = [
-    *[f"{MCP_TOOL_PREFIX}{name}" for name in TOOL_REGISTRY.keys()],
-    f"{MCP_TOOL_PREFIX}{_READ_TOOL_NAME}",
-    *_SDK_BUILTIN_TOOLS,
-]

autogpt_platform/backend/backend/api/features/chat/sdk/transcript.py

@@ -1,356 +0,0 @@
-"""JSONL transcript management for stateless multi-turn resume.
-
-The Claude Code CLI persists conversations as JSONL files (one JSON object per
-line).  When the SDK's ``Stop`` hook fires we read this file, strip bloat
-(progress entries, metadata), and upload the result to bucket storage.  On the
-next turn we download the transcript, write it to a temp file, and pass
-``--resume`` so the CLI can reconstruct the full conversation.
-
-Storage is handled via ``WorkspaceStorageBackend`` (GCS in prod, local
-filesystem for self-hosted) — no DB column needed.
-"""
-
-import json
-import logging
-import os
-import re
-
-logger = logging.getLogger(__name__)
-
-# UUIDs are hex + hyphens; strip everything else to prevent path injection.
-_SAFE_ID_RE = re.compile(r"[^0-9a-fA-F-]")
-
-# Entry types that can be safely removed from the transcript without breaking
-# the parentUuid conversation tree that ``--resume`` relies on.
-# - progress: UI progress ticks, no message content (avg 97KB for agent_progress)
-# - file-history-snapshot: undo tracking metadata
-# - queue-operation: internal queue bookkeeping
-# - summary: session summaries
-# - pr-link: PR link metadata
-STRIPPABLE_TYPES = frozenset(
-    {"progress", "file-history-snapshot", "queue-operation", "summary", "pr-link"}
-)
-
-# Workspace storage constants — deterministic path from session_id.
-TRANSCRIPT_STORAGE_PREFIX = "chat-transcripts"
-
-
-# ---------------------------------------------------------------------------
-# Progress stripping
-# ---------------------------------------------------------------------------
-
-
-def strip_progress_entries(content: str) -> str:
-    """Remove progress/metadata entries from a JSONL transcript.
-
-    Removes entries whose ``type`` is in ``STRIPPABLE_TYPES`` and reparents
-    any remaining child entries so the ``parentUuid`` chain stays intact.
-    Typically reduces transcript size by ~30%.
-    """
-    lines = content.strip().split("\n")
-
-    entries: list[dict] = []
-    for line in lines:
-        try:
-            entries.append(json.loads(line))
-        except json.JSONDecodeError:
-            # Keep unparseable lines as-is (safety)
-            entries.append({"_raw": line})
-
-    stripped_uuids: set[str] = set()
-    uuid_to_parent: dict[str, str] = {}
-    kept: list[dict] = []
-
-    for entry in entries:
-        if "_raw" in entry:
-            kept.append(entry)
-            continue
-        uid = entry.get("uuid", "")
-        parent = entry.get("parentUuid", "")
-        entry_type = entry.get("type", "")
-
-        if uid:
-            uuid_to_parent[uid] = parent
-
-        if entry_type in STRIPPABLE_TYPES:
-            if uid:
-                stripped_uuids.add(uid)
-        else:
-            kept.append(entry)
-
-    # Reparent: walk up chain through stripped entries to find surviving ancestor
-    for entry in kept:
-        if "_raw" in entry:
-            continue
-        parent = entry.get("parentUuid", "")
-        original_parent = parent
-        while parent in stripped_uuids:
-            parent = uuid_to_parent.get(parent, "")
-        if parent != original_parent:
-            entry["parentUuid"] = parent
-
-    result_lines: list[str] = []
-    for entry in kept:
-        if "_raw" in entry:
-            result_lines.append(entry["_raw"])
-        else:
-            result_lines.append(json.dumps(entry, separators=(",", ":")))
-
-    return "\n".join(result_lines) + "\n"
-
-
-# ---------------------------------------------------------------------------
-# Local file I/O (read from CLI's JSONL, write temp file for --resume)
-# ---------------------------------------------------------------------------
-
-
-def read_transcript_file(transcript_path: str) -> str | None:
-    """Read a JSONL transcript file from disk.
-
-    Returns the raw JSONL content, or ``None`` if the file is missing, empty,
-    or only contains metadata (≤2 lines with no conversation messages).
-    """
-    if not transcript_path or not os.path.isfile(transcript_path):
-        logger.debug(f"[Transcript] File not found: {transcript_path}")
-        return None
-
-    try:
-        with open(transcript_path) as f:
-            content = f.read()
-
-        if not content.strip():
-            logger.debug(f"[Transcript] Empty file: {transcript_path}")
-            return None
-
-        lines = content.strip().split("\n")
-        if len(lines) < 3:
-            # Raw files with ≤2 lines are metadata-only
-            # (queue-operation + file-history-snapshot, no conversation).
-            logger.debug(
-                f"[Transcript] Too few lines ({len(lines)}): {transcript_path}"
-            )
-            return None
-
-        # Quick structural validation — parse first and last lines.
-        json.loads(lines[0])
-        json.loads(lines[-1])
-
-        logger.info(
-            f"[Transcript] Read {len(lines)} lines, "
-            f"{len(content)} bytes from {transcript_path}"
-        )
-        return content
-
-    except (json.JSONDecodeError, OSError) as e:
-        logger.warning(f"[Transcript] Failed to read {transcript_path}: {e}")
-        return None
-
-
-def _sanitize_id(raw_id: str, max_len: int = 36) -> str:
-    """Sanitize an ID for safe use in file paths.
-
-    Session/user IDs are expected to be UUIDs (hex + hyphens).  Strip
-    everything else and truncate to *max_len* so the result cannot introduce
-    path separators or other special characters.
-    """
-    cleaned = _SAFE_ID_RE.sub("", raw_id or "")[:max_len]
-    return cleaned or "unknown"
-
-
-_SAFE_CWD_PREFIX = os.path.realpath("/tmp/copilot-")
-
-
-def write_transcript_to_tempfile(
-    transcript_content: str,
-    session_id: str,
-    cwd: str,
-) -> str | None:
-    """Write JSONL transcript to a temp file inside *cwd* for ``--resume``.
-
-    The file lives in the session working directory so it is cleaned up
-    automatically when the session ends.
-
-    Returns the absolute path to the file, or ``None`` on failure.
-    """
-    # Validate cwd is under the expected sandbox prefix (CodeQL sanitizer).
-    real_cwd = os.path.realpath(cwd)
-    if not real_cwd.startswith(_SAFE_CWD_PREFIX):
-        logger.warning(f"[Transcript] cwd outside sandbox: {cwd}")
-        return None
-
-    try:
-        os.makedirs(real_cwd, exist_ok=True)
-        safe_id = _sanitize_id(session_id, max_len=8)
-        jsonl_path = os.path.realpath(
-            os.path.join(real_cwd, f"transcript-{safe_id}.jsonl")
-        )
-        if not jsonl_path.startswith(real_cwd):
-            logger.warning(f"[Transcript] Path escaped cwd: {jsonl_path}")
-            return None
-
-        with open(jsonl_path, "w") as f:
-            f.write(transcript_content)
-
-        logger.info(f"[Transcript] Wrote resume file: {jsonl_path}")
-        return jsonl_path
-
-    except OSError as e:
-        logger.warning(f"[Transcript] Failed to write resume file: {e}")
-        return None
-
-
-def validate_transcript(content: str | None) -> bool:
-    """Check that a transcript has actual conversation messages.
-
-    A valid transcript for resume needs at least one user message and one
-    assistant message (not just queue-operation / file-history-snapshot
-    metadata).
-    """
-    if not content or not content.strip():
-        return False
-
-    lines = content.strip().split("\n")
-    if len(lines) < 2:
-        return False
-
-    has_user = False
-    has_assistant = False
-
-    for line in lines:
-        try:
-            entry = json.loads(line)
-            msg_type = entry.get("type")
-            if msg_type == "user":
-                has_user = True
-            elif msg_type == "assistant":
-                has_assistant = True
-        except json.JSONDecodeError:
-            return False
-
-    return has_user and has_assistant
-
-
-# ---------------------------------------------------------------------------
-# Bucket storage (GCS / local via WorkspaceStorageBackend)
-# ---------------------------------------------------------------------------
-
-
-def _storage_path_parts(user_id: str, session_id: str) -> tuple[str, str, str]:
-    """Return (workspace_id, file_id, filename) for a session's transcript.
-
-    Path structure: ``chat-transcripts/{user_id}/{session_id}.jsonl``
-    IDs are sanitized to hex+hyphen to prevent path traversal.
-    """
-    return (
-        TRANSCRIPT_STORAGE_PREFIX,
-        _sanitize_id(user_id),
-        f"{_sanitize_id(session_id)}.jsonl",
-    )
-
-
-def _build_storage_path(user_id: str, session_id: str, backend: object) -> str:
-    """Build the full storage path string that ``retrieve()`` expects.
-
-    ``store()`` returns a path like ``gcs://bucket/workspaces/...`` or
-    ``local://workspace_id/file_id/filename``.  Since we use deterministic
-    arguments we can reconstruct the same path for download/delete without
-    having stored the return value.
-    """
-    from backend.util.workspace_storage import GCSWorkspaceStorage
-
-    wid, fid, fname = _storage_path_parts(user_id, session_id)
-
-    if isinstance(backend, GCSWorkspaceStorage):
-        blob = f"workspaces/{wid}/{fid}/{fname}"
-        return f"gcs://{backend.bucket_name}/{blob}"
-    else:
-        # LocalWorkspaceStorage returns local://{relative_path}
-        return f"local://{wid}/{fid}/{fname}"
-
-
-async def upload_transcript(user_id: str, session_id: str, content: str) -> None:
-    """Strip progress entries and upload transcript to bucket storage.
-
-    Safety: only overwrites when the new (stripped) transcript is larger than
-    what is already stored.  Since JSONL is append-only, the latest transcript
-    is always the longest.  This prevents a slow/stale background task from
-    clobbering a newer upload from a concurrent turn.
-    """
-    from backend.util.workspace_storage import get_workspace_storage
-
-    stripped = strip_progress_entries(content)
-    if not validate_transcript(stripped):
-        logger.warning(
-            f"[Transcript] Skipping upload — stripped content is not a valid "
-            f"transcript for session {session_id}"
-        )
-        return
-
-    storage = await get_workspace_storage()
-    wid, fid, fname = _storage_path_parts(user_id, session_id)
-    encoded = stripped.encode("utf-8")
-    new_size = len(encoded)
-
-    # Check existing transcript size to avoid overwriting newer with older
-    path = _build_storage_path(user_id, session_id, storage)
-    try:
-        existing = await storage.retrieve(path)
-        if len(existing) >= new_size:
-            logger.info(
-                f"[Transcript] Skipping upload — existing transcript "
-                f"({len(existing)}B) >= new ({new_size}B) for session "
-                f"{session_id}"
-            )
-            return
-    except (FileNotFoundError, Exception):
-        pass  # No existing transcript or retrieval error — proceed with upload
-
-    await storage.store(
-        workspace_id=wid,
-        file_id=fid,
-        filename=fname,
-        content=encoded,
-    )
-    logger.info(
-        f"[Transcript] Uploaded {new_size} bytes "
-        f"(stripped from {len(content)}) for session {session_id}"
-    )
-
-
-async def download_transcript(user_id: str, session_id: str) -> str | None:
-    """Download transcript from bucket storage.
-
-    Returns the JSONL content string, or ``None`` if not found.
-    """
-    from backend.util.workspace_storage import get_workspace_storage
-
-    storage = await get_workspace_storage()
-    path = _build_storage_path(user_id, session_id, storage)
-
-    try:
-        data = await storage.retrieve(path)
-        content = data.decode("utf-8")
-        logger.info(
-            f"[Transcript] Downloaded {len(content)} bytes for session {session_id}"
-        )
-        return content
-    except FileNotFoundError:
-        logger.debug(f"[Transcript] No transcript in storage for {session_id}")
-        return None
-    except Exception as e:
-        logger.warning(f"[Transcript] Failed to download transcript: {e}")
-        return None
-
-
-async def delete_transcript(user_id: str, session_id: str) -> None:
-    """Delete transcript from bucket storage (e.g. after resume failure)."""
-    from backend.util.workspace_storage import get_workspace_storage
-
-    storage = await get_workspace_storage()
-    path = _build_storage_path(user_id, session_id, storage)
-
-    try:
-        await storage.delete(path)
-        logger.info(f"[Transcript] Deleted transcript for session {session_id}")
-    except Exception as e:
-        logger.warning(f"[Transcript] Failed to delete transcript: {e}")

autogpt_platform/backend/backend/api/features/chat/tools/agent_generator/dummy.py

@@ -1,154 +0,0 @@
-"""Dummy Agent Generator for testing.
-
-Returns mock responses matching the format expected from the external service.
-Enable via AGENTGENERATOR_USE_DUMMY=true in settings.
-
-WARNING: This is for testing only. Do not use in production.
-"""
-
-import asyncio
-import logging
-import uuid
-from typing import Any
-
-logger = logging.getLogger(__name__)
-
-# Dummy decomposition result (instructions type)
-DUMMY_DECOMPOSITION_RESULT: dict[str, Any] = {
-    "type": "instructions",
-    "steps": [
-        {
-            "description": "Get input from user",
-            "action": "input",
-            "block_name": "AgentInputBlock",
-        },
-        {
-            "description": "Process the input",
-            "action": "process",
-            "block_name": "TextFormatterBlock",
-        },
-        {
-            "description": "Return output to user",
-            "action": "output",
-            "block_name": "AgentOutputBlock",
-        },
-    ],
-}
-
-# Block IDs from backend/blocks/io.py
-AGENT_INPUT_BLOCK_ID = "c0a8e994-ebf1-4a9c-a4d8-89d09c86741b"
-AGENT_OUTPUT_BLOCK_ID = "363ae599-353e-4804-937e-b2ee3cef3da4"
-
-
-def _generate_dummy_agent_json() -> dict[str, Any]:
-    """Generate a minimal valid agent JSON for testing."""
-    input_node_id = str(uuid.uuid4())
-    output_node_id = str(uuid.uuid4())
-
-    return {
-        "id": str(uuid.uuid4()),
-        "version": 1,
-        "is_active": True,
-        "name": "Dummy Test Agent",
-        "description": "A dummy agent generated for testing purposes",
-        "nodes": [
-            {
-                "id": input_node_id,
-                "block_id": AGENT_INPUT_BLOCK_ID,
-                "input_default": {
-                    "name": "input",
-                    "title": "Input",
-                    "description": "Enter your input",
-                    "placeholder_values": [],
-                },
-                "metadata": {"position": {"x": 0, "y": 0}},
-            },
-            {
-                "id": output_node_id,
-                "block_id": AGENT_OUTPUT_BLOCK_ID,
-                "input_default": {
-                    "name": "output",
-                    "title": "Output",
-                    "description": "Agent output",
-                    "format": "{output}",
-                },
-                "metadata": {"position": {"x": 400, "y": 0}},
-            },
-        ],
-        "links": [
-            {
-                "id": str(uuid.uuid4()),
-                "source_id": input_node_id,
-                "sink_id": output_node_id,
-                "source_name": "result",
-                "sink_name": "value",
-                "is_static": False,
-            },
-        ],
-    }
-
-
-async def decompose_goal_dummy(
-    description: str,
-    context: str = "",
-    library_agents: list[dict[str, Any]] | None = None,
-) -> dict[str, Any]:
-    """Return dummy decomposition result."""
-    logger.info("Using dummy agent generator for decompose_goal")
-    return DUMMY_DECOMPOSITION_RESULT.copy()
-
-
-async def generate_agent_dummy(
-    instructions: dict[str, Any],
-    library_agents: list[dict[str, Any]] | None = None,
-    operation_id: str | None = None,
-    task_id: str | None = None,
-) -> dict[str, Any]:
-    """Return dummy agent JSON after a simulated delay."""
-    logger.info("Using dummy agent generator for generate_agent (30s delay)")
-    await asyncio.sleep(30)
-    return _generate_dummy_agent_json()
-
-
-async def generate_agent_patch_dummy(
-    update_request: str,
-    current_agent: dict[str, Any],
-    library_agents: list[dict[str, Any]] | None = None,
-    operation_id: str | None = None,
-    task_id: str | None = None,
-) -> dict[str, Any]:
-    """Return dummy patched agent (returns the current agent with updated description)."""
-    logger.info("Using dummy agent generator for generate_agent_patch")
-    patched = current_agent.copy()
-    patched["description"] = (
-        f"{current_agent.get('description', '')} (updated: {update_request})"
-    )
-    return patched
-
-
-async def customize_template_dummy(
-    template_agent: dict[str, Any],
-    modification_request: str,
-    context: str = "",
-) -> dict[str, Any]:
-    """Return dummy customized template (returns template with updated description)."""
-    logger.info("Using dummy agent generator for customize_template")
-    customized = template_agent.copy()
-    customized["description"] = (
-        f"{template_agent.get('description', '')} (customized: {modification_request})"
-    )
-    return customized
-
-
-async def get_blocks_dummy() -> list[dict[str, Any]]:
-    """Return dummy blocks list."""
-    logger.info("Using dummy agent generator for get_blocks")
-    return [
-        {"id": AGENT_INPUT_BLOCK_ID, "name": "AgentInputBlock"},
-        {"id": AGENT_OUTPUT_BLOCK_ID, "name": "AgentOutputBlock"},
-    ]
-
-
-async def health_check_dummy() -> bool:
-    """Always returns healthy for dummy service."""
-    return True

autogpt_platform/backend/backend/api/features/chat/tools/agent_generator/service.py

@@ -1,549 +0,0 @@
-"""External Agent Generator service client.
-
-This module provides a client for communicating with the external Agent Generator
-microservice. When AGENTGENERATOR_HOST is configured, the agent generation functions
-will delegate to the external service instead of using the built-in LLM-based implementation.
-"""
-
-import logging
-from typing import Any
-
-import httpx
-
-from backend.util.settings import Settings
-
-from .dummy import (
-    customize_template_dummy,
-    decompose_goal_dummy,
-    generate_agent_dummy,
-    generate_agent_patch_dummy,
-    get_blocks_dummy,
-    health_check_dummy,
-)
-
-logger = logging.getLogger(__name__)
-
-_dummy_mode_warned = False
-
-
-def _create_error_response(
-    error_message: str,
-    error_type: str = "unknown",
-    details: dict[str, Any] | None = None,
-) -> dict[str, Any]:
-    """Create a standardized error response dict.
-
-    Args:
-        error_message: Human-readable error message
-        error_type: Machine-readable error type
-        details: Optional additional error details
-
-    Returns:
-        Error dict with type="error" and error details
-    """
-    response: dict[str, Any] = {
-        "type": "error",
-        "error": error_message,
-        "error_type": error_type,
-    }
-    if details:
-        response["details"] = details
-    return response
-
-
-def _classify_http_error(e: httpx.HTTPStatusError) -> tuple[str, str]:
-    """Classify an HTTP error into error_type and message.
-
-    Args:
-        e: The HTTP status error
-
-    Returns:
-        Tuple of (error_type, error_message)
-    """
-    status = e.response.status_code
-    if status == 429:
-        return "rate_limit", f"Agent Generator rate limited: {e}"
-    elif status == 503:
-        return "service_unavailable", f"Agent Generator unavailable: {e}"
-    elif status == 504 or status == 408:
-        return "timeout", f"Agent Generator timed out: {e}"
-    else:
-        return "http_error", f"HTTP error calling Agent Generator: {e}"
-
-
-def _classify_request_error(e: httpx.RequestError) -> tuple[str, str]:
-    """Classify a request error into error_type and message.
-
-    Args:
-        e: The request error
-
-    Returns:
-        Tuple of (error_type, error_message)
-    """
-    error_str = str(e).lower()
-    if "timeout" in error_str or "timed out" in error_str:
-        return "timeout", f"Agent Generator request timed out: {e}"
-    elif "connect" in error_str:
-        return "connection_error", f"Could not connect to Agent Generator: {e}"
-    else:
-        return "request_error", f"Request error calling Agent Generator: {e}"
-
-
-_client: httpx.AsyncClient | None = None
-_settings: Settings | None = None
-
-
-def _get_settings() -> Settings:
-    """Get or create settings singleton."""
-    global _settings
-    if _settings is None:
-        _settings = Settings()
-    return _settings
-
-
-def _is_dummy_mode() -> bool:
-    """Check if dummy mode is enabled for testing."""
-    global _dummy_mode_warned
-    settings = _get_settings()
-    is_dummy = bool(settings.config.agentgenerator_use_dummy)
-    if is_dummy and not _dummy_mode_warned:
-        logger.warning(
-            "Agent Generator running in DUMMY MODE - returning mock responses. "
-            "Do not use in production!"
-        )
-        _dummy_mode_warned = True
-    return is_dummy
-
-
-def is_external_service_configured() -> bool:
-    """Check if external Agent Generator service is configured (or dummy mode)."""
-    settings = _get_settings()
-    return bool(settings.config.agentgenerator_host) or bool(
-        settings.config.agentgenerator_use_dummy
-    )
-
-
-def _get_base_url() -> str:
-    """Get the base URL for the external service."""
-    settings = _get_settings()
-    host = settings.config.agentgenerator_host
-    port = settings.config.agentgenerator_port
-    return f"http://{host}:{port}"
-
-
-def _get_client() -> httpx.AsyncClient:
-    """Get or create the HTTP client for the external service."""
-    global _client
-    if _client is None:
-        settings = _get_settings()
-        _client = httpx.AsyncClient(
-            base_url=_get_base_url(),
-            timeout=httpx.Timeout(settings.config.agentgenerator_timeout),
-        )
-    return _client
-
-
-async def decompose_goal_external(
-    description: str,
-    context: str = "",
-    library_agents: list[dict[str, Any]] | None = None,
-) -> dict[str, Any] | None:
-    """Call the external service to decompose a goal.
-
-    Args:
-        description: Natural language goal description
-        context: Additional context (e.g., answers to previous questions)
-        library_agents: User's library agents available for sub-agent composition
-
-    Returns:
-        Dict with either:
-        - {"type": "clarifying_questions", "questions": [...]}
-        - {"type": "instructions", "steps": [...]}
-        - {"type": "unachievable_goal", ...}
-        - {"type": "vague_goal", ...}
-        - {"type": "error", "error": "...", "error_type": "..."} on error
-        Or None on unexpected error
-    """
-    if _is_dummy_mode():
-        return await decompose_goal_dummy(description, context, library_agents)
-
-    client = _get_client()
-
-    if context:
-        description = f"{description}\n\nAdditional context from user:\n{context}"
-
-    payload: dict[str, Any] = {"description": description}
-    if library_agents:
-        payload["library_agents"] = library_agents
-
-    try:
-        response = await client.post("/api/decompose-description", json=payload)
-        response.raise_for_status()
-        data = response.json()
-
-        if not data.get("success"):
-            error_msg = data.get("error", "Unknown error from Agent Generator")
-            error_type = data.get("error_type", "unknown")
-            logger.error(
-                f"Agent Generator decomposition failed: {error_msg} "
-                f"(type: {error_type})"
-            )
-            return _create_error_response(error_msg, error_type)
-
-        # Map the response to the expected format
-        response_type = data.get("type")
-        if response_type == "instructions":
-            return {"type": "instructions", "steps": data.get("steps", [])}
-        elif response_type == "clarifying_questions":
-            return {
-                "type": "clarifying_questions",
-                "questions": data.get("questions", []),
-            }
-        elif response_type == "unachievable_goal":
-            return {
-                "type": "unachievable_goal",
-                "reason": data.get("reason"),
-                "suggested_goal": data.get("suggested_goal"),
-            }
-        elif response_type == "vague_goal":
-            return {
-                "type": "vague_goal",
-                "suggested_goal": data.get("suggested_goal"),
-            }
-        elif response_type == "error":
-            # Pass through error from the service
-            return _create_error_response(
-                data.get("error", "Unknown error"),
-                data.get("error_type", "unknown"),
-            )
-        else:
-            logger.error(
-                f"Unknown response type from external service: {response_type}"
-            )
-            return _create_error_response(
-                f"Unknown response type from Agent Generator: {response_type}",
-                "invalid_response",
-            )
-
-    except httpx.HTTPStatusError as e:
-        error_type, error_msg = _classify_http_error(e)
-        logger.error(error_msg)
-        return _create_error_response(error_msg, error_type)
-    except httpx.RequestError as e:
-        error_type, error_msg = _classify_request_error(e)
-        logger.error(error_msg)
-        return _create_error_response(error_msg, error_type)
-    except Exception as e:
-        error_msg = f"Unexpected error calling Agent Generator: {e}"
-        logger.error(error_msg)
-        return _create_error_response(error_msg, "unexpected_error")
-
-
-async def generate_agent_external(
-    instructions: dict[str, Any],
-    library_agents: list[dict[str, Any]] | None = None,
-    operation_id: str | None = None,
-    task_id: str | None = None,
-) -> dict[str, Any] | None:
-    """Call the external service to generate an agent from instructions.
-
-    Args:
-        instructions: Structured instructions from decompose_goal
-        library_agents: User's library agents available for sub-agent composition
-        operation_id: Operation ID for async processing (enables Redis Streams callback)
-        task_id: Task ID for async processing (enables Redis Streams callback)
-
-    Returns:
-        Agent JSON dict, {"status": "accepted"} for async, or error dict {"type": "error", ...} on error
-    """
-    if _is_dummy_mode():
-        return await generate_agent_dummy(
-            instructions, library_agents, operation_id, task_id
-        )
-
-    client = _get_client()
-
-    # Build request payload
-    payload: dict[str, Any] = {"instructions": instructions}
-    if library_agents:
-        payload["library_agents"] = library_agents
-    if operation_id and task_id:
-        payload["operation_id"] = operation_id
-        payload["task_id"] = task_id
-
-    try:
-        response = await client.post("/api/generate-agent", json=payload)
-
-        # Handle 202 Accepted for async processing
-        if response.status_code == 202:
-            logger.info(
-                f"Agent Generator accepted async request "
-                f"(operation_id={operation_id}, task_id={task_id})"
-            )
-            return {
-                "status": "accepted",
-                "operation_id": operation_id,
-                "task_id": task_id,
-            }
-
-        response.raise_for_status()
-        data = response.json()
-
-        if not data.get("success"):
-            error_msg = data.get("error", "Unknown error from Agent Generator")
-            error_type = data.get("error_type", "unknown")
-            logger.error(
-                f"Agent Generator generation failed: {error_msg} (type: {error_type})"
-            )
-            return _create_error_response(error_msg, error_type)
-
-        return data.get("agent_json")
-
-    except httpx.HTTPStatusError as e:
-        error_type, error_msg = _classify_http_error(e)
-        logger.error(error_msg)
-        return _create_error_response(error_msg, error_type)
-    except httpx.RequestError as e:
-        error_type, error_msg = _classify_request_error(e)
-        logger.error(error_msg)
-        return _create_error_response(error_msg, error_type)
-    except Exception as e:
-        error_msg = f"Unexpected error calling Agent Generator: {e}"
-        logger.error(error_msg)
-        return _create_error_response(error_msg, "unexpected_error")
-
-
-async def generate_agent_patch_external(
-    update_request: str,
-    current_agent: dict[str, Any],
-    library_agents: list[dict[str, Any]] | None = None,
-    operation_id: str | None = None,
-    task_id: str | None = None,
-) -> dict[str, Any] | None:
-    """Call the external service to generate a patch for an existing agent.
-
-    Args:
-        update_request: Natural language description of changes
-        current_agent: Current agent JSON
-        library_agents: User's library agents available for sub-agent composition
-        operation_id: Operation ID for async processing (enables Redis Streams callback)
-        task_id: Task ID for async processing (enables Redis Streams callback)
-
-    Returns:
-        Updated agent JSON, clarifying questions dict, {"status": "accepted"} for async, or error dict on error
-    """
-    if _is_dummy_mode():
-        return await generate_agent_patch_dummy(
-            update_request, current_agent, library_agents, operation_id, task_id
-        )
-
-    client = _get_client()
-
-    # Build request payload
-    payload: dict[str, Any] = {
-        "update_request": update_request,
-        "current_agent_json": current_agent,
-    }
-    if library_agents:
-        payload["library_agents"] = library_agents
-    if operation_id and task_id:
-        payload["operation_id"] = operation_id
-        payload["task_id"] = task_id
-
-    try:
-        response = await client.post("/api/update-agent", json=payload)
-
-        # Handle 202 Accepted for async processing
-        if response.status_code == 202:
-            logger.info(
-                f"Agent Generator accepted async update request "
-                f"(operation_id={operation_id}, task_id={task_id})"
-            )
-            return {
-                "status": "accepted",
-                "operation_id": operation_id,
-                "task_id": task_id,
-            }
-
-        response.raise_for_status()
-        data = response.json()
-
-        if not data.get("success"):
-            error_msg = data.get("error", "Unknown error from Agent Generator")
-            error_type = data.get("error_type", "unknown")
-            logger.error(
-                f"Agent Generator patch generation failed: {error_msg} "
-                f"(type: {error_type})"
-            )
-            return _create_error_response(error_msg, error_type)
-
-        # Check if it's clarifying questions
-        if data.get("type") == "clarifying_questions":
-            return {
-                "type": "clarifying_questions",
-                "questions": data.get("questions", []),
-            }
-
-        # Check if it's an error passed through
-        if data.get("type") == "error":
-            return _create_error_response(
-                data.get("error", "Unknown error"),
-                data.get("error_type", "unknown"),
-            )
-
-        # Otherwise return the updated agent JSON
-        return data.get("agent_json")
-
-    except httpx.HTTPStatusError as e:
-        error_type, error_msg = _classify_http_error(e)
-        logger.error(error_msg)
-        return _create_error_response(error_msg, error_type)
-    except httpx.RequestError as e:
-        error_type, error_msg = _classify_request_error(e)
-        logger.error(error_msg)
-        return _create_error_response(error_msg, error_type)
-    except Exception as e:
-        error_msg = f"Unexpected error calling Agent Generator: {e}"
-        logger.error(error_msg)
-        return _create_error_response(error_msg, "unexpected_error")
-
-
-async def customize_template_external(
-    template_agent: dict[str, Any],
-    modification_request: str,
-    context: str = "",
-) -> dict[str, Any] | None:
-    """Call the external service to customize a template/marketplace agent.
-
-    Args:
-        template_agent: The template agent JSON to customize
-        modification_request: Natural language description of customizations
-        context: Additional context (e.g., answers to previous questions)
-
-    Returns:
-        Customized agent JSON, clarifying questions dict, or error dict on error
-    """
-    if _is_dummy_mode():
-        return await customize_template_dummy(
-            template_agent, modification_request, context
-        )
-
-    client = _get_client()
-
-    request = modification_request
-    if context:
-        request = f"{modification_request}\n\nAdditional context from user:\n{context}"
-
-    payload: dict[str, Any] = {
-        "template_agent_json": template_agent,
-        "modification_request": request,
-    }
-
-    try:
-        response = await client.post("/api/template-modification", json=payload)
-        response.raise_for_status()
-        data = response.json()
-
-        if not data.get("success"):
-            error_msg = data.get("error", "Unknown error from Agent Generator")
-            error_type = data.get("error_type", "unknown")
-            logger.error(
-                f"Agent Generator template customization failed: {error_msg} "
-                f"(type: {error_type})"
-            )
-            return _create_error_response(error_msg, error_type)
-
-        # Check if it's clarifying questions
-        if data.get("type") == "clarifying_questions":
-            return {
-                "type": "clarifying_questions",
-                "questions": data.get("questions", []),
-            }
-
-        # Check if it's an error passed through
-        if data.get("type") == "error":
-            return _create_error_response(
-                data.get("error", "Unknown error"),
-                data.get("error_type", "unknown"),
-            )
-
-        # Otherwise return the customized agent JSON
-        return data.get("agent_json")
-
-    except httpx.HTTPStatusError as e:
-        error_type, error_msg = _classify_http_error(e)
-        logger.error(error_msg)
-        return _create_error_response(error_msg, error_type)
-    except httpx.RequestError as e:
-        error_type, error_msg = _classify_request_error(e)
-        logger.error(error_msg)
-        return _create_error_response(error_msg, error_type)
-    except Exception as e:
-        error_msg = f"Unexpected error calling Agent Generator: {e}"
-        logger.error(error_msg)
-        return _create_error_response(error_msg, "unexpected_error")
-
-
-async def get_blocks_external() -> list[dict[str, Any]] | None:
-    """Get available blocks from the external service.
-
-    Returns:
-        List of block info dicts or None on error
-    """
-    if _is_dummy_mode():
-        return await get_blocks_dummy()
-
-    client = _get_client()
-
-    try:
-        response = await client.get("/api/blocks")
-        response.raise_for_status()
-        data = response.json()
-
-        if not data.get("success"):
-            logger.error("External service returned error getting blocks")
-            return None
-
-        return data.get("blocks", [])
-
-    except httpx.HTTPStatusError as e:
-        logger.error(f"HTTP error getting blocks from external service: {e}")
-        return None
-    except httpx.RequestError as e:
-        logger.error(f"Request error getting blocks from external service: {e}")
-        return None
-    except Exception as e:
-        logger.error(f"Unexpected error getting blocks from external service: {e}")
-        return None
-
-
-async def health_check() -> bool:
-    """Check if the external service is healthy.
-
-    Returns:
-        True if healthy, False otherwise
-    """
-    if not is_external_service_configured():
-        return False
-
-    if _is_dummy_mode():
-        return await health_check_dummy()
-
-    client = _get_client()
-
-    try:
-        response = await client.get("/health")
-        response.raise_for_status()
-        data = response.json()
-        return data.get("status") == "healthy" and data.get("blocks_loaded", False)
-    except Exception as e:
-        logger.warning(f"External agent generator health check failed: {e}")
-        return False
-
-
-async def close_client() -> None:
-    """Close the HTTP client."""
-    global _client
-    if _client is not None:
-        await _client.aclose()
-        _client = None

autogpt_platform/backend/backend/api/features/chat/tools/base.py

@@ -1,129 +0,0 @@
-"""Base classes and shared utilities for chat tools."""
-
-import logging
-from typing import Any
-
-from openai.types.chat import ChatCompletionToolParam
-
-from backend.api.features.chat.model import ChatSession
-from backend.api.features.chat.response_model import StreamToolOutputAvailable
-
-from .models import ErrorResponse, NeedLoginResponse, ToolResponseBase
-
-logger = logging.getLogger(__name__)
-
-
-class BaseTool:
-    """Base class for all chat tools."""
-
-    @property
-    def name(self) -> str:
-        """Tool name for OpenAI function calling."""
-        raise NotImplementedError
-
-    @property
-    def description(self) -> str:
-        """Tool description for OpenAI."""
-        raise NotImplementedError
-
-    @property
-    def parameters(self) -> dict[str, Any]:
-        """Tool parameters schema for OpenAI."""
-        raise NotImplementedError
-
-    @property
-    def requires_auth(self) -> bool:
-        """Whether this tool requires authentication."""
-        return False
-
-    @property
-    def is_long_running(self) -> bool:
-        """Whether this tool is long-running and should execute in background.
-
-        Long-running tools (like agent generation) are executed via background
-        tasks to survive SSE disconnections. The result is persisted to chat
-        history and visible when the user refreshes.
-        """
-        return False
-
-    def as_openai_tool(self) -> ChatCompletionToolParam:
-        """Convert to OpenAI tool format."""
-        return ChatCompletionToolParam(
-            type="function",
-            function={
-                "name": self.name,
-                "description": self.description,
-                "parameters": self.parameters,
-            },
-        )
-
-    async def execute(
-        self,
-        user_id: str | None,
-        session: ChatSession,
-        tool_call_id: str,
-        **kwargs,
-    ) -> StreamToolOutputAvailable:
-        """Execute the tool with authentication check.
-
-        Args:
-            user_id: User ID (may be anonymous like "anon_123")
-            session_id: Chat session ID
-            **kwargs: Tool-specific parameters
-
-        Returns:
-            Pydantic response object
-
-        """
-        if self.requires_auth and not user_id:
-            logger.error(
-                f"Attempted tool call for {self.name} but user not authenticated"
-            )
-            return StreamToolOutputAvailable(
-                toolCallId=tool_call_id,
-                toolName=self.name,
-                output=NeedLoginResponse(
-                    message=f"Please sign in to use {self.name}",
-                    session_id=session.session_id,
-                ).model_dump_json(),
-                success=False,
-            )
-
-        try:
-            result = await self._execute(user_id, session, **kwargs)
-            return StreamToolOutputAvailable(
-                toolCallId=tool_call_id,
-                toolName=self.name,
-                output=result.model_dump_json(),
-            )
-        except Exception as e:
-            logger.error(f"Error in {self.name}: {e}", exc_info=True)
-            return StreamToolOutputAvailable(
-                toolCallId=tool_call_id,
-                toolName=self.name,
-                output=ErrorResponse(
-                    message=f"An error occurred while executing {self.name}",
-                    error=str(e),
-                    session_id=session.session_id,
-                ).model_dump_json(),
-                success=False,
-            )
-
-    async def _execute(
-        self,
-        user_id: str | None,
-        session: ChatSession,
-        **kwargs,
-    ) -> ToolResponseBase:
-        """Internal execution logic to be implemented by subclasses.
-
-        Args:
-            user_id: User ID (authenticated or anonymous)
-            session_id: Chat session ID
-            **kwargs: Tool-specific parameters
-
-        Returns:
-            Pydantic response object
-
-        """
-        raise NotImplementedError

autogpt_platform/backend/backend/api/features/chat/tools/bash_exec.py

@@ -1,131 +0,0 @@
-"""Bash execution tool — run shell commands in a bubblewrap sandbox.
-
-Full Bash scripting is allowed (loops, conditionals, pipes, functions, etc.).
-Safety comes from OS-level isolation (bubblewrap): only system dirs visible
-read-only, writable workspace only, clean env, no network.
-
-Requires bubblewrap (``bwrap``) — the tool is disabled when bwrap is not
-available (e.g. macOS development).
-"""
-
-import logging
-from typing import Any
-
-from backend.api.features.chat.model import ChatSession
-from backend.api.features.chat.tools.base import BaseTool
-from backend.api.features.chat.tools.models import (
-    BashExecResponse,
-    ErrorResponse,
-    ToolResponseBase,
-)
-from backend.api.features.chat.tools.sandbox import (
-    get_workspace_dir,
-    has_full_sandbox,
-    run_sandboxed,
-)
-
-logger = logging.getLogger(__name__)
-
-
-class BashExecTool(BaseTool):
-    """Execute Bash commands in a bubblewrap sandbox."""
-
-    @property
-    def name(self) -> str:
-        return "bash_exec"
-
-    @property
-    def description(self) -> str:
-        if not has_full_sandbox():
-            return (
-                "Bash execution is DISABLED — bubblewrap sandbox is not "
-                "available on this platform. Do not call this tool."
-            )
-        return (
-            "Execute a Bash command or script in a bubblewrap sandbox. "
-            "Full Bash scripting is supported (loops, conditionals, pipes, "
-            "functions, etc.). "
-            "The sandbox shares the same working directory as the SDK Read/Write "
-            "tools — files created by either are accessible to both. "
-            "SECURITY: Only system directories (/usr, /bin, /lib, /etc) are "
-            "visible read-only, the per-session workspace is the only writable "
-            "path, environment variables are wiped (no secrets), all network "
-            "access is blocked at the kernel level, and resource limits are "
-            "enforced (max 64 processes, 512MB memory, 50MB file size). "
-            "Application code, configs, and other directories are NOT accessible. "
-            "To fetch web content, use the web_fetch tool instead. "
-            "Execution is killed after the timeout (default 30s, max 120s). "
-            "Returns stdout and stderr. "
-            "Useful for file manipulation, data processing with Unix tools "
-            "(grep, awk, sed, jq, etc.), and running shell scripts."
-        )
-
-    @property
-    def parameters(self) -> dict[str, Any]:
-        return {
-            "type": "object",
-            "properties": {
-                "command": {
-                    "type": "string",
-                    "description": "Bash command or script to execute.",
-                },
-                "timeout": {
-                    "type": "integer",
-                    "description": (
-                        "Max execution time in seconds (default 30, max 120)."
-                    ),
-                    "default": 30,
-                },
-            },
-            "required": ["command"],
-        }
-
-    @property
-    def requires_auth(self) -> bool:
-        return False
-
-    async def _execute(
-        self,
-        user_id: str | None,
-        session: ChatSession,
-        **kwargs: Any,
-    ) -> ToolResponseBase:
-        session_id = session.session_id if session else None
-
-        if not has_full_sandbox():
-            return ErrorResponse(
-                message="bash_exec requires bubblewrap sandbox (Linux only).",
-                error="sandbox_unavailable",
-                session_id=session_id,
-            )
-
-        command: str = (kwargs.get("command") or "").strip()
-        timeout: int = kwargs.get("timeout", 30)
-
-        if not command:
-            return ErrorResponse(
-                message="No command provided.",
-                error="empty_command",
-                session_id=session_id,
-            )
-
-        workspace = get_workspace_dir(session_id or "default")
-
-        stdout, stderr, exit_code, timed_out = await run_sandboxed(
-            command=["bash", "-c", command],
-            cwd=workspace,
-            timeout=timeout,
-        )
-
-        return BashExecResponse(
-            message=(
-                "Execution timed out"
-                if timed_out
-                else f"Command executed (exit {exit_code})"
-            ),
-            stdout=stdout,
-            stderr=stderr,
-            exit_code=exit_code,
-            timed_out=timed_out,
-            session_id=session_id,
-        )

autogpt_platform/backend/backend/api/features/chat/tools/check_operation_status.py

@@ -1,127 +0,0 @@
-"""CheckOperationStatusTool — query the status of a long-running operation."""
-
-import logging
-from typing import Any
-
-from backend.api.features.chat.model import ChatSession
-from backend.api.features.chat.tools.base import BaseTool
-from backend.api.features.chat.tools.models import (
-    ErrorResponse,
-    ResponseType,
-    ToolResponseBase,
-)
-
-logger = logging.getLogger(__name__)
-
-
-class OperationStatusResponse(ToolResponseBase):
-    """Response for check_operation_status tool."""
-
-    type: ResponseType = ResponseType.OPERATION_STATUS
-    task_id: str
-    operation_id: str
-    status: str  # "running", "completed", "failed"
-    tool_name: str | None = None
-    message: str = ""
-
-
-class CheckOperationStatusTool(BaseTool):
-    """Check the status of a long-running operation (create_agent, edit_agent, etc.).
-
-    The CoPilot uses this tool to report back to the user whether an
-    operation that was started earlier has completed, failed, or is still
-    running.
-    """
-
-    @property
-    def name(self) -> str:
-        return "check_operation_status"
-
-    @property
-    def description(self) -> str:
-        return (
-            "Check the current status of a long-running operation such as "
-            "create_agent or edit_agent. Accepts either an operation_id or "
-            "task_id from a previous operation_started response. "
-            "Returns the current status: running, completed, or failed."
-        )
-
-    @property
-    def parameters(self) -> dict[str, Any]:
-        return {
-            "type": "object",
-            "properties": {
-                "operation_id": {
-                    "type": "string",
-                    "description": (
-                        "The operation_id from an operation_started response."
-                    ),
-                },
-                "task_id": {
-                    "type": "string",
-                    "description": (
-                        "The task_id from an operation_started response. "
-                        "Used as fallback if operation_id is not provided."
-                    ),
-                },
-            },
-            "required": [],
-        }
-
-    @property
-    def requires_auth(self) -> bool:
-        return False
-
-    async def _execute(
-        self,
-        user_id: str | None,
-        session: ChatSession,
-        **kwargs,
-    ) -> ToolResponseBase:
-        from backend.api.features.chat import stream_registry
-
-        operation_id = (kwargs.get("operation_id") or "").strip()
-        task_id = (kwargs.get("task_id") or "").strip()
-
-        if not operation_id and not task_id:
-            return ErrorResponse(
-                message="Please provide an operation_id or task_id.",
-                error="missing_parameter",
-            )
-
-        task = None
-        if operation_id:
-            task = await stream_registry.find_task_by_operation_id(operation_id)
-        if task is None and task_id:
-            task = await stream_registry.get_task(task_id)
-
-        if task is None:
-            # Task not in Redis — it may have already expired (TTL).
-            # Check conversation history for the result instead.
-            return ErrorResponse(
-                message=(
-                    "Operation not found — it may have already completed and "
-                    "expired from the status tracker. Check the conversation "
-                    "history for the result."
-                ),
-                error="not_found",
-            )
-
-        status_messages = {
-            "running": (
-                f"The {task.tool_name or 'operation'} is still running. "
-                "Please wait for it to complete."
-            ),
-            "completed": (
-                f"The {task.tool_name or 'operation'} has completed successfully."
-            ),
-            "failed": f"The {task.tool_name or 'operation'} has failed.",
-        }
-
-        return OperationStatusResponse(
-            task_id=task.task_id,
-            operation_id=task.operation_id,
-            status=task.status,
-            tool_name=task.tool_name,
-            message=status_messages.get(task.status, f"Status: {task.status}"),
-        )

autogpt_platform/backend/backend/api/features/chat/tools/create_agent.py

@@ -1,335 +0,0 @@
-"""CreateAgentTool - Creates agents from natural language descriptions."""
-
-import logging
-from typing import Any
-
-from backend.api.features.chat.model import ChatSession
-
-from .agent_generator import (
-    AgentGeneratorNotConfiguredError,
-    decompose_goal,
-    enrich_library_agents_from_steps,
-    generate_agent,
-    get_all_relevant_agents_for_generation,
-    get_user_message_for_error,
-    save_agent_to_library,
-)
-from .base import BaseTool
-from .models import (
-    AgentPreviewResponse,
-    AgentSavedResponse,
-    AsyncProcessingResponse,
-    ClarificationNeededResponse,
-    ClarifyingQuestion,
-    ErrorResponse,
-    ToolResponseBase,
-)
-
-logger = logging.getLogger(__name__)
-
-
-class CreateAgentTool(BaseTool):
-    """Tool for creating agents from natural language descriptions."""
-
-    @property
-    def name(self) -> str:
-        return "create_agent"
-
-    @property
-    def description(self) -> str:
-        return (
-            "Create a new agent workflow from a natural language description. "
-            "First generates a preview, then saves to library if save=true."
-        )
-
-    @property
-    def requires_auth(self) -> bool:
-        return True
-
-    @property
-    def is_long_running(self) -> bool:
-        return True
-
-    @property
-    def parameters(self) -> dict[str, Any]:
-        return {
-            "type": "object",
-            "properties": {
-                "description": {
-                    "type": "string",
-                    "description": (
-                        "Natural language description of what the agent should do. "
-                        "Be specific about inputs, outputs, and the workflow steps."
-                    ),
-                },
-                "context": {
-                    "type": "string",
-                    "description": (
-                        "Additional context or answers to previous clarifying questions. "
-                        "Include any preferences or constraints mentioned by the user."
-                    ),
-                },
-                "save": {
-                    "type": "boolean",
-                    "description": (
-                        "Whether to save the agent to the user's library. "
-                        "Default is true. Set to false for preview only."
-                    ),
-                    "default": True,
-                },
-            },
-            "required": ["description"],
-        }
-
-    async def _execute(
-        self,
-        user_id: str | None,
-        session: ChatSession,
-        **kwargs,
-    ) -> ToolResponseBase:
-        """Execute the create_agent tool.
-
-        Flow:
-        1. Decompose the description into steps (may return clarifying questions)
-        2. Generate agent JSON (external service handles fixing and validation)
-        3. Preview or save based on the save parameter
-        """
-        description = kwargs.get("description", "").strip()
-        context = kwargs.get("context", "")
-        save = kwargs.get("save", True)
-        session_id = session.session_id if session else None
-
-        # Extract async processing params (passed by long-running tool handler)
-        operation_id = kwargs.get("_operation_id")
-        task_id = kwargs.get("_task_id")
-
-        if not description:
-            return ErrorResponse(
-                message="Please provide a description of what the agent should do.",
-                error="Missing description parameter",
-                session_id=session_id,
-            )
-
-        library_agents = None
-        if user_id:
-            try:
-                library_agents = await get_all_relevant_agents_for_generation(
-                    user_id=user_id,
-                    search_query=description,
-                    include_marketplace=True,
-                )
-                logger.debug(
-                    f"Found {len(library_agents)} relevant agents for sub-agent composition"
-                )
-            except Exception as e:
-                logger.warning(f"Failed to fetch library agents: {e}")
-
-        try:
-            decomposition_result = await decompose_goal(
-                description, context, library_agents
-            )
-        except AgentGeneratorNotConfiguredError:
-            return ErrorResponse(
-                message=(
-                    "Agent generation is not available. "
-                    "The Agent Generator service is not configured."
-                ),
-                error="service_not_configured",
-                session_id=session_id,
-            )
-
-        if decomposition_result is None:
-            return ErrorResponse(
-                message="Failed to analyze the goal. The agent generation service may be unavailable. Please try again.",
-                error="decomposition_failed",
-                details={"description": description[:100]},
-                session_id=session_id,
-            )
-
-        if decomposition_result.get("type") == "error":
-            error_msg = decomposition_result.get("error", "Unknown error")
-            error_type = decomposition_result.get("error_type", "unknown")
-            user_message = get_user_message_for_error(
-                error_type,
-                operation="analyze the goal",
-                llm_parse_message="The AI had trouble understanding this request. Please try rephrasing your goal.",
-            )
-            return ErrorResponse(
-                message=user_message,
-                error=f"decomposition_failed:{error_type}",
-                details={
-                    "description": description[:100],
-                    "service_error": error_msg,
-                    "error_type": error_type,
-                },
-                session_id=session_id,
-            )
-
-        if decomposition_result.get("type") == "clarifying_questions":
-            questions = decomposition_result.get("questions", [])
-            return ClarificationNeededResponse(
-                message=(
-                    "I need some more information to create this agent. "
-                    "Please answer the following questions:"
-                ),
-                questions=[
-                    ClarifyingQuestion(
-                        question=q.get("question", ""),
-                        keyword=q.get("keyword", ""),
-                        example=q.get("example"),
-                    )
-                    for q in questions
-                ],
-                session_id=session_id,
-            )
-
-        if decomposition_result.get("type") == "unachievable_goal":
-            suggested = decomposition_result.get("suggested_goal", "")
-            reason = decomposition_result.get("reason", "")
-            return ErrorResponse(
-                message=(
-                    f"This goal cannot be accomplished with the available blocks. "
-                    f"{reason} "
-                    f"Suggestion: {suggested}"
-                ),
-                error="unachievable_goal",
-                details={"suggested_goal": suggested, "reason": reason},
-                session_id=session_id,
-            )
-
-        if decomposition_result.get("type") == "vague_goal":
-            suggested = decomposition_result.get("suggested_goal", "")
-            return ErrorResponse(
-                message=(
-                    f"The goal is too vague to create a specific workflow. "
-                    f"Suggestion: {suggested}"
-                ),
-                error="vague_goal",
-                details={"suggested_goal": suggested},
-                session_id=session_id,
-            )
-
-        if user_id and library_agents is not None:
-            try:
-                library_agents = await enrich_library_agents_from_steps(
-                    user_id=user_id,
-                    decomposition_result=decomposition_result,
-                    existing_agents=library_agents,
-                    include_marketplace=True,
-                )
-                logger.debug(
-                    f"After enrichment: {len(library_agents)} total agents for sub-agent composition"
-                )
-            except Exception as e:
-                logger.warning(f"Failed to enrich library agents from steps: {e}")
-
-        try:
-            agent_json = await generate_agent(
-                decomposition_result,
-                library_agents,
-                operation_id=operation_id,
-                task_id=task_id,
-            )
-        except AgentGeneratorNotConfiguredError:
-            return ErrorResponse(
-                message=(
-                    "Agent generation is not available. "
-                    "The Agent Generator service is not configured."
-                ),
-                error="service_not_configured",
-                session_id=session_id,
-            )
-
-        if agent_json is None:
-            return ErrorResponse(
-                message="Failed to generate the agent. The agent generation service may be unavailable. Please try again.",
-                error="generation_failed",
-                details={"description": description[:100]},
-                session_id=session_id,
-            )
-
-        if isinstance(agent_json, dict) and agent_json.get("type") == "error":
-            error_msg = agent_json.get("error", "Unknown error")
-            error_type = agent_json.get("error_type", "unknown")
-            user_message = get_user_message_for_error(
-                error_type,
-                operation="generate the agent",
-                llm_parse_message="The AI had trouble generating the agent. Please try again or simplify your goal.",
-                validation_message=(
-                    "I wasn't able to create a valid agent for this request. "
-                    "The generated workflow had some structural issues. "
-                    "Please try simplifying your goal or breaking it into smaller steps."
-                ),
-                error_details=error_msg,
-            )
-            return ErrorResponse(
-                message=user_message,
-                error=f"generation_failed:{error_type}",
-                details={
-                    "description": description[:100],
-                    "service_error": error_msg,
-                    "error_type": error_type,
-                },
-                session_id=session_id,
-            )
-
-        # Check if Agent Generator accepted for async processing
-        if agent_json.get("status") == "accepted":
-            logger.info(
-                f"Agent generation delegated to async processing "
-                f"(operation_id={operation_id}, task_id={task_id})"
-            )
-            return AsyncProcessingResponse(
-                message="Agent generation started. You'll be notified when it's complete.",
-                operation_id=operation_id,
-                task_id=task_id,
-                session_id=session_id,
-            )
-
-        agent_name = agent_json.get("name", "Generated Agent")
-        agent_description = agent_json.get("description", "")
-        node_count = len(agent_json.get("nodes", []))
-        link_count = len(agent_json.get("links", []))
-
-        if not save:
-            return AgentPreviewResponse(
-                message=(
-                    f"I've generated an agent called '{agent_name}' with {node_count} blocks. "
-                    f"Review it and call create_agent with save=true to save it to your library."
-                ),
-                agent_json=agent_json,
-                agent_name=agent_name,
-                description=agent_description,
-                node_count=node_count,
-                link_count=link_count,
-                session_id=session_id,
-            )
-
-        if not user_id:
-            return ErrorResponse(
-                message="You must be logged in to save agents.",
-                error="auth_required",
-                session_id=session_id,
-            )
-
-        try:
-            created_graph, library_agent = await save_agent_to_library(
-                agent_json, user_id
-            )
-
-            return AgentSavedResponse(
-                message=f"Agent '{created_graph.name}' has been saved to your library!",
-                agent_id=created_graph.id,
-                agent_name=created_graph.name,
-                library_agent_id=library_agent.id,
-                library_agent_link=f"/library/agents/{library_agent.id}",
-                agent_page_link=f"/build?flowID={created_graph.id}",
-                session_id=session_id,
-            )
-        except Exception as e:
-            return ErrorResponse(
-                message=f"Failed to save the agent: {str(e)}",
-                error="save_failed",
-                details={"exception": str(e)},
-                session_id=session_id,
-            )

autogpt_platform/backend/backend/api/features/chat/tools/customize_agent.py

@@ -1,337 +0,0 @@
-"""CustomizeAgentTool - Customizes marketplace/template agents using natural language."""
-
-import logging
-from typing import Any
-
-from backend.api.features.chat.model import ChatSession
-from backend.api.features.store import db as store_db
-from backend.api.features.store.exceptions import AgentNotFoundError
-
-from .agent_generator import (
-    AgentGeneratorNotConfiguredError,
-    customize_template,
-    get_user_message_for_error,
-    graph_to_json,
-    save_agent_to_library,
-)
-from .base import BaseTool
-from .models import (
-    AgentPreviewResponse,
-    AgentSavedResponse,
-    ClarificationNeededResponse,
-    ClarifyingQuestion,
-    ErrorResponse,
-    ToolResponseBase,
-)
-
-logger = logging.getLogger(__name__)
-
-
-class CustomizeAgentTool(BaseTool):
-    """Tool for customizing marketplace/template agents using natural language."""
-
-    @property
-    def name(self) -> str:
-        return "customize_agent"
-
-    @property
-    def description(self) -> str:
-        return (
-            "Customize a marketplace or template agent using natural language. "
-            "Takes an existing agent from the marketplace and modifies it based on "
-            "the user's requirements before adding to their library."
-        )
-
-    @property
-    def requires_auth(self) -> bool:
-        return True
-
-    @property
-    def is_long_running(self) -> bool:
-        return True
-
-    @property
-    def parameters(self) -> dict[str, Any]:
-        return {
-            "type": "object",
-            "properties": {
-                "agent_id": {
-                    "type": "string",
-                    "description": (
-                        "The marketplace agent ID in format 'creator/slug' "
-                        "(e.g., 'autogpt/newsletter-writer'). "
-                        "Get this from find_agent results."
-                    ),
-                },
-                "modifications": {
-                    "type": "string",
-                    "description": (
-                        "Natural language description of how to customize the agent. "
-                        "Be specific about what changes you want to make."
-                    ),
-                },
-                "context": {
-                    "type": "string",
-                    "description": (
-                        "Additional context or answers to previous clarifying questions."
-                    ),
-                },
-                "save": {
-                    "type": "boolean",
-                    "description": (
-                        "Whether to save the customized agent to the user's library. "
-                        "Default is true. Set to false for preview only."
-                    ),
-                    "default": True,
-                },
-            },
-            "required": ["agent_id", "modifications"],
-        }
-
-    async def _execute(
-        self,
-        user_id: str | None,
-        session: ChatSession,
-        **kwargs,
-    ) -> ToolResponseBase:
-        """Execute the customize_agent tool.
-
-        Flow:
-        1. Parse the agent ID to get creator/slug
-        2. Fetch the template agent from the marketplace
-        3. Call customize_template with the modification request
-        4. Preview or save based on the save parameter
-        """
-        agent_id = kwargs.get("agent_id", "").strip()
-        modifications = kwargs.get("modifications", "").strip()
-        context = kwargs.get("context", "")
-        save = kwargs.get("save", True)
-        session_id = session.session_id if session else None
-
-        if not agent_id:
-            return ErrorResponse(
-                message="Please provide the marketplace agent ID (e.g., 'creator/agent-name').",
-                error="missing_agent_id",
-                session_id=session_id,
-            )
-
-        if not modifications:
-            return ErrorResponse(
-                message="Please describe how you want to customize this agent.",
-                error="missing_modifications",
-                session_id=session_id,
-            )
-
-        # Parse agent_id in format "creator/slug"
-        parts = [p.strip() for p in agent_id.split("/")]
-        if len(parts) != 2 or not parts[0] or not parts[1]:
-            return ErrorResponse(
-                message=(
-                    f"Invalid agent ID format: '{agent_id}'. "
-                    "Expected format is 'creator/agent-name' "
-                    "(e.g., 'autogpt/newsletter-writer')."
-                ),
-                error="invalid_agent_id_format",
-                session_id=session_id,
-            )
-
-        creator_username, agent_slug = parts
-
-        # Fetch the marketplace agent details
-        try:
-            agent_details = await store_db.get_store_agent_details(
-                username=creator_username, agent_name=agent_slug
-            )
-        except AgentNotFoundError:
-            return ErrorResponse(
-                message=(
-                    f"Could not find marketplace agent '{agent_id}'. "
-                    "Please check the agent ID and try again."
-                ),
-                error="agent_not_found",
-                session_id=session_id,
-            )
-        except Exception as e:
-            logger.error(f"Error fetching marketplace agent {agent_id}: {e}")
-            return ErrorResponse(
-                message="Failed to fetch the marketplace agent. Please try again.",
-                error="fetch_error",
-                session_id=session_id,
-            )
-
-        if not agent_details.store_listing_version_id:
-            return ErrorResponse(
-                message=(
-                    f"The agent '{agent_id}' does not have an available version. "
-                    "Please try a different agent."
-                ),
-                error="no_version_available",
-                session_id=session_id,
-            )
-
-        # Get the full agent graph
-        try:
-            graph = await store_db.get_agent(agent_details.store_listing_version_id)
-            template_agent = graph_to_json(graph)
-        except Exception as e:
-            logger.error(f"Error fetching agent graph for {agent_id}: {e}")
-            return ErrorResponse(
-                message="Failed to fetch the agent configuration. Please try again.",
-                error="graph_fetch_error",
-                session_id=session_id,
-            )
-
-        # Call customize_template
-        try:
-            result = await customize_template(
-                template_agent=template_agent,
-                modification_request=modifications,
-                context=context,
-            )
-        except AgentGeneratorNotConfiguredError:
-            return ErrorResponse(
-                message=(
-                    "Agent customization is not available. "
-                    "The Agent Generator service is not configured."
-                ),
-                error="service_not_configured",
-                session_id=session_id,
-            )
-        except Exception as e:
-            logger.error(f"Error calling customize_template for {agent_id}: {e}")
-            return ErrorResponse(
-                message=(
-                    "Failed to customize the agent due to a service error. "
-                    "Please try again."
-                ),
-                error="customization_service_error",
-                session_id=session_id,
-            )
-
-        if result is None:
-            return ErrorResponse(
-                message=(
-                    "Failed to customize the agent. "
-                    "The agent generation service may be unavailable or timed out. "
-                    "Please try again."
-                ),
-                error="customization_failed",
-                session_id=session_id,
-            )
-
-        # Handle error response
-        if isinstance(result, dict) and result.get("type") == "error":
-            error_msg = result.get("error", "Unknown error")
-            error_type = result.get("error_type", "unknown")
-            user_message = get_user_message_for_error(
-                error_type,
-                operation="customize the agent",
-                llm_parse_message=(
-                    "The AI had trouble customizing the agent. "
-                    "Please try again or simplify your request."
-                ),
-                validation_message=(
-                    "The customized agent failed validation. "
-                    "Please try rephrasing your request."
-                ),
-                error_details=error_msg,
-            )
-            return ErrorResponse(
-                message=user_message,
-                error=f"customization_failed:{error_type}",
-                session_id=session_id,
-            )
-
-        # Handle clarifying questions
-        if isinstance(result, dict) and result.get("type") == "clarifying_questions":
-            questions = result.get("questions") or []
-            if not isinstance(questions, list):
-                logger.error(
-                    f"Unexpected clarifying questions format: {type(questions)}"
-                )
-                questions = []
-            return ClarificationNeededResponse(
-                message=(
-                    "I need some more information to customize this agent. "
-                    "Please answer the following questions:"
-                ),
-                questions=[
-                    ClarifyingQuestion(
-                        question=q.get("question", ""),
-                        keyword=q.get("keyword", ""),
-                        example=q.get("example"),
-                    )
-                    for q in questions
-                    if isinstance(q, dict)
-                ],
-                session_id=session_id,
-            )
-
-        # Result should be the customized agent JSON
-        if not isinstance(result, dict):
-            logger.error(f"Unexpected customize_template response type: {type(result)}")
-            return ErrorResponse(
-                message="Failed to customize the agent due to an unexpected response.",
-                error="unexpected_response_type",
-                session_id=session_id,
-            )
-
-        customized_agent = result
-
-        agent_name = customized_agent.get(
-            "name", f"Customized {agent_details.agent_name}"
-        )
-        agent_description = customized_agent.get("description", "")
-        nodes = customized_agent.get("nodes")
-        links = customized_agent.get("links")
-        node_count = len(nodes) if isinstance(nodes, list) else 0
-        link_count = len(links) if isinstance(links, list) else 0
-
-        if not save:
-            return AgentPreviewResponse(
-                message=(
-                    f"I've customized the agent '{agent_details.agent_name}'. "
-                    f"The customized agent has {node_count} blocks. "
-                    f"Review it and call customize_agent with save=true to save it."
-                ),
-                agent_json=customized_agent,
-                agent_name=agent_name,
-                description=agent_description,
-                node_count=node_count,
-                link_count=link_count,
-                session_id=session_id,
-            )
-
-        if not user_id:
-            return ErrorResponse(
-                message="You must be logged in to save agents.",
-                error="auth_required",
-                session_id=session_id,
-            )
-
-        # Save to user's library
-        try:
-            created_graph, library_agent = await save_agent_to_library(
-                customized_agent, user_id, is_update=False
-            )
-
-            return AgentSavedResponse(
-                message=(
-                    f"Customized agent '{created_graph.name}' "
-                    f"(based on '{agent_details.agent_name}') "
-                    f"has been saved to your library!"
-                ),
-                agent_id=created_graph.id,
-                agent_name=created_graph.name,
-                library_agent_id=library_agent.id,
-                library_agent_link=f"/library/agents/{library_agent.id}",
-                agent_page_link=f"/build?flowID={created_graph.id}",
-                session_id=session_id,
-            )
-        except Exception as e:
-            logger.error(f"Error saving customized agent: {e}")
-            return ErrorResponse(
-                message="Failed to save the customized agent. Please try again.",
-                error="save_failed",
-                session_id=session_id,
-            )

autogpt_platform/backend/backend/api/features/chat/tools/edit_agent.py

@@ -1,284 +0,0 @@
-"""EditAgentTool - Edits existing agents using natural language."""
-
-import logging
-from typing import Any
-
-from backend.api.features.chat.model import ChatSession
-
-from .agent_generator import (
-    AgentGeneratorNotConfiguredError,
-    generate_agent_patch,
-    get_agent_as_json,
-    get_all_relevant_agents_for_generation,
-    get_user_message_for_error,
-    save_agent_to_library,
-)
-from .base import BaseTool
-from .models import (
-    AgentPreviewResponse,
-    AgentSavedResponse,
-    AsyncProcessingResponse,
-    ClarificationNeededResponse,
-    ClarifyingQuestion,
-    ErrorResponse,
-    ToolResponseBase,
-)
-
-logger = logging.getLogger(__name__)
-
-
-class EditAgentTool(BaseTool):
-    """Tool for editing existing agents using natural language."""
-
-    @property
-    def name(self) -> str:
-        return "edit_agent"
-
-    @property
-    def description(self) -> str:
-        return (
-            "Edit an existing agent from the user's library using natural language. "
-            "Generates updates to the agent while preserving unchanged parts."
-        )
-
-    @property
-    def requires_auth(self) -> bool:
-        return True
-
-    @property
-    def is_long_running(self) -> bool:
-        return True
-
-    @property
-    def parameters(self) -> dict[str, Any]:
-        return {
-            "type": "object",
-            "properties": {
-                "agent_id": {
-                    "type": "string",
-                    "description": (
-                        "The ID of the agent to edit. "
-                        "Can be a graph ID or library agent ID."
-                    ),
-                },
-                "changes": {
-                    "type": "string",
-                    "description": (
-                        "Natural language description of what changes to make. "
-                        "Be specific about what to add, remove, or modify."
-                    ),
-                },
-                "context": {
-                    "type": "string",
-                    "description": (
-                        "Additional context or answers to previous clarifying questions."
-                    ),
-                },
-                "save": {
-                    "type": "boolean",
-                    "description": (
-                        "Whether to save the changes. "
-                        "Default is true. Set to false for preview only."
-                    ),
-                    "default": True,
-                },
-            },
-            "required": ["agent_id", "changes"],
-        }
-
-    async def _execute(
-        self,
-        user_id: str | None,
-        session: ChatSession,
-        **kwargs,
-    ) -> ToolResponseBase:
-        """Execute the edit_agent tool.
-
-        Flow:
-        1. Fetch the current agent
-        2. Generate updated agent (external service handles fixing and validation)
-        3. Preview or save based on the save parameter
-        """
-        agent_id = kwargs.get("agent_id", "").strip()
-        changes = kwargs.get("changes", "").strip()
-        context = kwargs.get("context", "")
-        save = kwargs.get("save", True)
-        session_id = session.session_id if session else None
-
-        # Extract async processing params (passed by long-running tool handler)
-        operation_id = kwargs.get("_operation_id")
-        task_id = kwargs.get("_task_id")
-
-        if not agent_id:
-            return ErrorResponse(
-                message="Please provide the agent ID to edit.",
-                error="Missing agent_id parameter",
-                session_id=session_id,
-            )
-
-        if not changes:
-            return ErrorResponse(
-                message="Please describe what changes you want to make.",
-                error="Missing changes parameter",
-                session_id=session_id,
-            )
-
-        current_agent = await get_agent_as_json(agent_id, user_id)
-
-        if current_agent is None:
-            return ErrorResponse(
-                message=f"Could not find agent with ID '{agent_id}' in your library.",
-                error="agent_not_found",
-                session_id=session_id,
-            )
-
-        library_agents = None
-        if user_id:
-            try:
-                graph_id = current_agent.get("id")
-                library_agents = await get_all_relevant_agents_for_generation(
-                    user_id=user_id,
-                    search_query=changes,
-                    exclude_graph_id=graph_id,
-                    include_marketplace=True,
-                )
-                logger.debug(
-                    f"Found {len(library_agents)} relevant agents for sub-agent composition"
-                )
-            except Exception as e:
-                logger.warning(f"Failed to fetch library agents: {e}")
-
-        update_request = changes
-        if context:
-            update_request = f"{changes}\n\nAdditional context:\n{context}"
-
-        try:
-            result = await generate_agent_patch(
-                update_request,
-                current_agent,
-                library_agents,
-                operation_id=operation_id,
-                task_id=task_id,
-            )
-        except AgentGeneratorNotConfiguredError:
-            return ErrorResponse(
-                message=(
-                    "Agent editing is not available. "
-                    "The Agent Generator service is not configured."
-                ),
-                error="service_not_configured",
-                session_id=session_id,
-            )
-
-        if result is None:
-            return ErrorResponse(
-                message="Failed to generate changes. The agent generation service may be unavailable or timed out. Please try again.",
-                error="update_generation_failed",
-                details={"agent_id": agent_id, "changes": changes[:100]},
-                session_id=session_id,
-            )
-
-        # Check if Agent Generator accepted for async processing
-        if result.get("status") == "accepted":
-            logger.info(
-                f"Agent edit delegated to async processing "
-                f"(operation_id={operation_id}, task_id={task_id})"
-            )
-            return AsyncProcessingResponse(
-                message="Agent edit started. You'll be notified when it's complete.",
-                operation_id=operation_id,
-                task_id=task_id,
-                session_id=session_id,
-            )
-
-        # Check if the result is an error from the external service
-        if isinstance(result, dict) and result.get("type") == "error":
-            error_msg = result.get("error", "Unknown error")
-            error_type = result.get("error_type", "unknown")
-            user_message = get_user_message_for_error(
-                error_type,
-                operation="generate the changes",
-                llm_parse_message="The AI had trouble generating the changes. Please try again or simplify your request.",
-                validation_message="The generated changes failed validation. Please try rephrasing your request.",
-                error_details=error_msg,
-            )
-            return ErrorResponse(
-                message=user_message,
-                error=f"update_generation_failed:{error_type}",
-                details={
-                    "agent_id": agent_id,
-                    "changes": changes[:100],
-                    "service_error": error_msg,
-                    "error_type": error_type,
-                },
-                session_id=session_id,
-            )
-
-        if result.get("type") == "clarifying_questions":
-            questions = result.get("questions", [])
-            return ClarificationNeededResponse(
-                message=(
-                    "I need some more information about the changes. "
-                    "Please answer the following questions:"
-                ),
-                questions=[
-                    ClarifyingQuestion(
-                        question=q.get("question", ""),
-                        keyword=q.get("keyword", ""),
-                        example=q.get("example"),
-                    )
-                    for q in questions
-                ],
-                session_id=session_id,
-            )
-
-        updated_agent = result
-
-        agent_name = updated_agent.get("name", "Updated Agent")
-        agent_description = updated_agent.get("description", "")
-        node_count = len(updated_agent.get("nodes", []))
-        link_count = len(updated_agent.get("links", []))
-
-        if not save:
-            return AgentPreviewResponse(
-                message=(
-                    f"I've updated the agent. "
-                    f"The agent now has {node_count} blocks. "
-                    f"Review it and call edit_agent with save=true to save the changes."
-                ),
-                agent_json=updated_agent,
-                agent_name=agent_name,
-                description=agent_description,
-                node_count=node_count,
-                link_count=link_count,
-                session_id=session_id,
-            )
-
-        if not user_id:
-            return ErrorResponse(
-                message="You must be logged in to save agents.",
-                error="auth_required",
-                session_id=session_id,
-            )
-
-        try:
-            created_graph, library_agent = await save_agent_to_library(
-                updated_agent, user_id, is_update=True
-            )
-
-            return AgentSavedResponse(
-                message=f"Updated agent '{created_graph.name}' has been saved to your library!",
-                agent_id=created_graph.id,
-                agent_name=created_graph.name,
-                library_agent_id=library_agent.id,
-                library_agent_link=f"/library/agents/{library_agent.id}",
-                agent_page_link=f"/build?flowID={created_graph.id}",
-                session_id=session_id,
-            )
-        except Exception as e:
-            return ErrorResponse(
-                message=f"Failed to save the updated agent: {str(e)}",
-                error="save_failed",
-                details={"exception": str(e)},
-                session_id=session_id,
-            )

autogpt_platform/backend/backend/api/features/chat/tools/helpers.py

@@ -1,29 +0,0 @@
-"""Shared helpers for chat tools."""
-
-from typing import Any
-
-
-def get_inputs_from_schema(
-    input_schema: dict[str, Any],
-    exclude_fields: set[str] | None = None,
-) -> list[dict[str, Any]]:
-    """Extract input field info from JSON schema."""
-    if not isinstance(input_schema, dict):
-        return []
-
-    exclude = exclude_fields or set()
-    properties = input_schema.get("properties", {})
-    required = set(input_schema.get("required", []))
-
-    return [
-        {
-            "name": name,
-            "title": schema.get("title", name),
-            "type": schema.get("type", "string"),
-            "description": schema.get("description", ""),
-            "required": name in required,
-            "default": schema.get("default"),
-        }
-        for name, schema in properties.items()
-        if name not in exclude
-    ]

autogpt_platform/backend/backend/api/features/chat/tools/workspace_files.py

@@ -1,626 +0,0 @@
-"""CoPilot tools for workspace file operations."""
-
-import base64
-import logging
-from typing import Any, Optional
-
-from pydantic import BaseModel
-
-from backend.api.features.chat.model import ChatSession
-from backend.data.workspace import get_or_create_workspace
-from backend.util.settings import Config
-from backend.util.virus_scanner import scan_content_safe
-from backend.util.workspace import WorkspaceManager
-
-from .base import BaseTool
-from .models import ErrorResponse, ResponseType, ToolResponseBase
-
-logger = logging.getLogger(__name__)
-
-
-class WorkspaceFileInfoData(BaseModel):
-    """Data model for workspace file information (not a response itself)."""
-
-    file_id: str
-    name: str
-    path: str
-    mime_type: str
-    size_bytes: int
-
-
-class WorkspaceFileListResponse(ToolResponseBase):
-    """Response containing list of workspace files."""
-
-    type: ResponseType = ResponseType.WORKSPACE_FILE_LIST
-    files: list[WorkspaceFileInfoData]
-    total_count: int
-
-
-class WorkspaceFileContentResponse(ToolResponseBase):
-    """Response containing workspace file content (legacy, for small text files)."""
-
-    type: ResponseType = ResponseType.WORKSPACE_FILE_CONTENT
-    file_id: str
-    name: str
-    path: str
-    mime_type: str
-    content_base64: str
-
-
-class WorkspaceFileMetadataResponse(ToolResponseBase):
-    """Response containing workspace file metadata and download URL (prevents context bloat)."""
-
-    type: ResponseType = ResponseType.WORKSPACE_FILE_METADATA
-    file_id: str
-    name: str
-    path: str
-    mime_type: str
-    size_bytes: int
-    download_url: str
-    preview: str | None = None  # First 500 chars for text files
-
-
-class WorkspaceWriteResponse(ToolResponseBase):
-    """Response after writing a file to workspace."""
-
-    type: ResponseType = ResponseType.WORKSPACE_FILE_WRITTEN
-    file_id: str
-    name: str
-    path: str
-    size_bytes: int
-
-
-class WorkspaceDeleteResponse(ToolResponseBase):
-    """Response after deleting a file from workspace."""
-
-    type: ResponseType = ResponseType.WORKSPACE_FILE_DELETED
-    file_id: str
-    success: bool
-
-
-class ListWorkspaceFilesTool(BaseTool):
-    """Tool for listing files in user's workspace."""
-
-    @property
-    def name(self) -> str:
-        return "list_workspace_files"
-
-    @property
-    def description(self) -> str:
-        return (
-            "List files in the user's persistent workspace (cloud storage). "
-            "These files survive across sessions. "
-            "For ephemeral session files, use the SDK Read/Glob tools instead. "
-            "Returns file names, paths, sizes, and metadata. "
-            "Optionally filter by path prefix."
-        )
-
-    @property
-    def parameters(self) -> dict[str, Any]:
-        return {
-            "type": "object",
-            "properties": {
-                "path_prefix": {
-                    "type": "string",
-                    "description": (
-                        "Optional path prefix to filter files "
-                        "(e.g., '/documents/' to list only files in documents folder). "
-                        "By default, only files from the current session are listed."
-                    ),
-                },
-                "limit": {
-                    "type": "integer",
-                    "description": "Maximum number of files to return (default 50, max 100)",
-                    "minimum": 1,
-                    "maximum": 100,
-                },
-                "include_all_sessions": {
-                    "type": "boolean",
-                    "description": (
-                        "If true, list files from all sessions. "
-                        "Default is false (only current session's files)."
-                    ),
-                },
-            },
-            "required": [],
-        }
-
-    @property
-    def requires_auth(self) -> bool:
-        return True
-
-    async def _execute(
-        self,
-        user_id: str | None,
-        session: ChatSession,
-        **kwargs,
-    ) -> ToolResponseBase:
-        session_id = session.session_id
-
-        if not user_id:
-            return ErrorResponse(
-                message="Authentication required",
-                session_id=session_id,
-            )
-
-        path_prefix: Optional[str] = kwargs.get("path_prefix")
-        limit = min(kwargs.get("limit", 50), 100)
-        include_all_sessions: bool = kwargs.get("include_all_sessions", False)
-
-        try:
-            workspace = await get_or_create_workspace(user_id)
-            # Pass session_id for session-scoped file access
-            manager = WorkspaceManager(user_id, workspace.id, session_id)
-
-            files = await manager.list_files(
-                path=path_prefix,
-                limit=limit,
-                include_all_sessions=include_all_sessions,
-            )
-            total = await manager.get_file_count(
-                path=path_prefix,
-                include_all_sessions=include_all_sessions,
-            )
-
-            file_infos = [
-                WorkspaceFileInfoData(
-                    file_id=f.id,
-                    name=f.name,
-                    path=f.path,
-                    mime_type=f.mimeType,
-                    size_bytes=f.sizeBytes,
-                )
-                for f in files
-            ]
-
-            scope_msg = "all sessions" if include_all_sessions else "current session"
-            return WorkspaceFileListResponse(
-                files=file_infos,
-                total_count=total,
-                message=f"Found {len(files)} files in workspace ({scope_msg})",
-                session_id=session_id,
-            )
-
-        except Exception as e:
-            logger.error(f"Error listing workspace files: {e}", exc_info=True)
-            return ErrorResponse(
-                message=f"Failed to list workspace files: {str(e)}",
-                error=str(e),
-                session_id=session_id,
-            )
-
-
-class ReadWorkspaceFileTool(BaseTool):
-    """Tool for reading file content from workspace."""
-
-    # Size threshold for returning full content vs metadata+URL
-    # Files larger than this return metadata with download URL to prevent context bloat
-    MAX_INLINE_SIZE_BYTES = 32 * 1024  # 32KB
-    # Preview size for text files
-    PREVIEW_SIZE = 500
-
-    @property
-    def name(self) -> str:
-        return "read_workspace_file"
-
-    @property
-    def description(self) -> str:
-        return (
-            "Read a file from the user's persistent workspace (cloud storage). "
-            "These files survive across sessions. "
-            "For ephemeral session files, use the SDK Read tool instead. "
-            "Specify either file_id or path to identify the file. "
-            "For small text files, returns content directly. "
-            "For large or binary files, returns metadata and a download URL. "
-            "Paths are scoped to the current session by default. "
-            "Use /sessions/<session_id>/... for cross-session access."
-        )
-
-    @property
-    def parameters(self) -> dict[str, Any]:
-        return {
-            "type": "object",
-            "properties": {
-                "file_id": {
-                    "type": "string",
-                    "description": "The file's unique ID (from list_workspace_files)",
-                },
-                "path": {
-                    "type": "string",
-                    "description": (
-                        "The virtual file path (e.g., '/documents/report.pdf'). "
-                        "Scoped to current session by default."
-                    ),
-                },
-                "force_download_url": {
-                    "type": "boolean",
-                    "description": (
-                        "If true, always return metadata+URL instead of inline content. "
-                        "Default is false (auto-selects based on file size/type)."
-                    ),
-                },
-            },
-            "required": [],  # At least one must be provided
-        }
-
-    @property
-    def requires_auth(self) -> bool:
-        return True
-
-    def _is_text_mime_type(self, mime_type: str) -> bool:
-        """Check if the MIME type is a text-based type."""
-        text_types = [
-            "text/",
-            "application/json",
-            "application/xml",
-            "application/javascript",
-            "application/x-python",
-            "application/x-sh",
-        ]
-        return any(mime_type.startswith(t) for t in text_types)
-
-    async def _execute(
-        self,
-        user_id: str | None,
-        session: ChatSession,
-        **kwargs,
-    ) -> ToolResponseBase:
-        session_id = session.session_id
-
-        if not user_id:
-            return ErrorResponse(
-                message="Authentication required",
-                session_id=session_id,
-            )
-
-        file_id: Optional[str] = kwargs.get("file_id")
-        path: Optional[str] = kwargs.get("path")
-        force_download_url: bool = kwargs.get("force_download_url", False)
-
-        if not file_id and not path:
-            return ErrorResponse(
-                message="Please provide either file_id or path",
-                session_id=session_id,
-            )
-
-        try:
-            workspace = await get_or_create_workspace(user_id)
-            # Pass session_id for session-scoped file access
-            manager = WorkspaceManager(user_id, workspace.id, session_id)
-
-            # Get file info
-            if file_id:
-                file_info = await manager.get_file_info(file_id)
-                if file_info is None:
-                    return ErrorResponse(
-                        message=f"File not found: {file_id}",
-                        session_id=session_id,
-                    )
-                target_file_id = file_id
-            else:
-                # path is guaranteed to be non-None here due to the check above
-                assert path is not None
-                file_info = await manager.get_file_info_by_path(path)
-                if file_info is None:
-                    return ErrorResponse(
-                        message=f"File not found at path: {path}",
-                        session_id=session_id,
-                    )
-                target_file_id = file_info.id
-
-            # Decide whether to return inline content or metadata+URL
-            is_small_file = file_info.sizeBytes <= self.MAX_INLINE_SIZE_BYTES
-            is_text_file = self._is_text_mime_type(file_info.mimeType)
-
-            # Return inline content for small text files (unless force_download_url)
-            if is_small_file and is_text_file and not force_download_url:
-                content = await manager.read_file_by_id(target_file_id)
-                content_b64 = base64.b64encode(content).decode("utf-8")
-
-                return WorkspaceFileContentResponse(
-                    file_id=file_info.id,
-                    name=file_info.name,
-                    path=file_info.path,
-                    mime_type=file_info.mimeType,
-                    content_base64=content_b64,
-                    message=f"Successfully read file: {file_info.name}",
-                    session_id=session_id,
-                )
-
-            # Return metadata + workspace:// reference for large or binary files
-            # This prevents context bloat (100KB file = ~133KB as base64)
-            # Use workspace:// format so frontend urlTransform can add proxy prefix
-            download_url = f"workspace://{target_file_id}"
-
-            # Generate preview for text files
-            preview: str | None = None
-            if is_text_file:
-                try:
-                    content = await manager.read_file_by_id(target_file_id)
-                    preview_text = content[: self.PREVIEW_SIZE].decode(
-                        "utf-8", errors="replace"
-                    )
-                    if len(content) > self.PREVIEW_SIZE:
-                        preview_text += "..."
-                    preview = preview_text
-                except Exception:
-                    pass  # Preview is optional
-
-            return WorkspaceFileMetadataResponse(
-                file_id=file_info.id,
-                name=file_info.name,
-                path=file_info.path,
-                mime_type=file_info.mimeType,
-                size_bytes=file_info.sizeBytes,
-                download_url=download_url,
-                preview=preview,
-                message=f"File: {file_info.name} ({file_info.sizeBytes} bytes). Use download_url to retrieve content.",
-                session_id=session_id,
-            )
-
-        except FileNotFoundError as e:
-            return ErrorResponse(
-                message=str(e),
-                session_id=session_id,
-            )
-        except Exception as e:
-            logger.error(f"Error reading workspace file: {e}", exc_info=True)
-            return ErrorResponse(
-                message=f"Failed to read workspace file: {str(e)}",
-                error=str(e),
-                session_id=session_id,
-            )
-
-
-class WriteWorkspaceFileTool(BaseTool):
-    """Tool for writing files to workspace."""
-
-    @property
-    def name(self) -> str:
-        return "write_workspace_file"
-
-    @property
-    def description(self) -> str:
-        return (
-            "Write or create a file in the user's persistent workspace (cloud storage). "
-            "These files survive across sessions. "
-            "For ephemeral session files, use the SDK Write tool instead. "
-            "Provide the content as a base64-encoded string. "
-            f"Maximum file size is {Config().max_file_size_mb}MB. "
-            "Files are saved to the current session's folder by default. "
-            "Use /sessions/<session_id>/... for cross-session access."
-        )
-
-    @property
-    def parameters(self) -> dict[str, Any]:
-        return {
-            "type": "object",
-            "properties": {
-                "filename": {
-                    "type": "string",
-                    "description": "Name for the file (e.g., 'report.pdf')",
-                },
-                "content_base64": {
-                    "type": "string",
-                    "description": "Base64-encoded file content",
-                },
-                "path": {
-                    "type": "string",
-                    "description": (
-                        "Optional virtual path where to save the file "
-                        "(e.g., '/documents/report.pdf'). "
-                        "Defaults to '/{filename}'. Scoped to current session."
-                    ),
-                },
-                "mime_type": {
-                    "type": "string",
-                    "description": (
-                        "Optional MIME type of the file. "
-                        "Auto-detected from filename if not provided."
-                    ),
-                },
-                "overwrite": {
-                    "type": "boolean",
-                    "description": "Whether to overwrite if file exists at path (default: false)",
-                },
-            },
-            "required": ["filename", "content_base64"],
-        }
-
-    @property
-    def requires_auth(self) -> bool:
-        return True
-
-    async def _execute(
-        self,
-        user_id: str | None,
-        session: ChatSession,
-        **kwargs,
-    ) -> ToolResponseBase:
-        session_id = session.session_id
-
-        if not user_id:
-            return ErrorResponse(
-                message="Authentication required",
-                session_id=session_id,
-            )
-
-        filename: str = kwargs.get("filename", "")
-        content_b64: str = kwargs.get("content_base64", "")
-        path: Optional[str] = kwargs.get("path")
-        mime_type: Optional[str] = kwargs.get("mime_type")
-        overwrite: bool = kwargs.get("overwrite", False)
-
-        if not filename:
-            return ErrorResponse(
-                message="Please provide a filename",
-                session_id=session_id,
-            )
-
-        if not content_b64:
-            return ErrorResponse(
-                message="Please provide content_base64",
-                session_id=session_id,
-            )
-
-        # Decode content
-        try:
-            content = base64.b64decode(content_b64)
-        except Exception:
-            return ErrorResponse(
-                message="Invalid base64-encoded content",
-                session_id=session_id,
-            )
-
-        # Check size
-        max_file_size = Config().max_file_size_mb * 1024 * 1024
-        if len(content) > max_file_size:
-            return ErrorResponse(
-                message=f"File too large. Maximum size is {Config().max_file_size_mb}MB",
-                session_id=session_id,
-            )
-
-        try:
-            # Virus scan
-            await scan_content_safe(content, filename=filename)
-
-            workspace = await get_or_create_workspace(user_id)
-            # Pass session_id for session-scoped file access
-            manager = WorkspaceManager(user_id, workspace.id, session_id)
-
-            file_record = await manager.write_file(
-                content=content,
-                filename=filename,
-                path=path,
-                mime_type=mime_type,
-                overwrite=overwrite,
-            )
-
-            return WorkspaceWriteResponse(
-                file_id=file_record.id,
-                name=file_record.name,
-                path=file_record.path,
-                size_bytes=file_record.sizeBytes,
-                message=f"Successfully wrote file: {file_record.name}",
-                session_id=session_id,
-            )
-
-        except ValueError as e:
-            return ErrorResponse(
-                message=str(e),
-                session_id=session_id,
-            )
-        except Exception as e:
-            logger.error(f"Error writing workspace file: {e}", exc_info=True)
-            return ErrorResponse(
-                message=f"Failed to write workspace file: {str(e)}",
-                error=str(e),
-                session_id=session_id,
-            )
-
-
-class DeleteWorkspaceFileTool(BaseTool):
-    """Tool for deleting files from workspace."""
-
-    @property
-    def name(self) -> str:
-        return "delete_workspace_file"
-
-    @property
-    def description(self) -> str:
-        return (
-            "Delete a file from the user's persistent workspace (cloud storage). "
-            "Specify either file_id or path to identify the file. "
-            "Paths are scoped to the current session by default. "
-            "Use /sessions/<session_id>/... for cross-session access."
-        )
-
-    @property
-    def parameters(self) -> dict[str, Any]:
-        return {
-            "type": "object",
-            "properties": {
-                "file_id": {
-                    "type": "string",
-                    "description": "The file's unique ID (from list_workspace_files)",
-                },
-                "path": {
-                    "type": "string",
-                    "description": (
-                        "The virtual file path (e.g., '/documents/report.pdf'). "
-                        "Scoped to current session by default."
-                    ),
-                },
-            },
-            "required": [],  # At least one must be provided
-        }
-
-    @property
-    def requires_auth(self) -> bool:
-        return True
-
-    async def _execute(
-        self,
-        user_id: str | None,
-        session: ChatSession,
-        **kwargs,
-    ) -> ToolResponseBase:
-        session_id = session.session_id
-
-        if not user_id:
-            return ErrorResponse(
-                message="Authentication required",
-                session_id=session_id,
-            )
-
-        file_id: Optional[str] = kwargs.get("file_id")
-        path: Optional[str] = kwargs.get("path")
-
-        if not file_id and not path:
-            return ErrorResponse(
-                message="Please provide either file_id or path",
-                session_id=session_id,
-            )
-
-        try:
-            workspace = await get_or_create_workspace(user_id)
-            # Pass session_id for session-scoped file access
-            manager = WorkspaceManager(user_id, workspace.id, session_id)
-
-            # Determine the file_id to delete
-            target_file_id: str
-            if file_id:
-                target_file_id = file_id
-            else:
-                # path is guaranteed to be non-None here due to the check above
-                assert path is not None
-                file_info = await manager.get_file_info_by_path(path)
-                if file_info is None:
-                    return ErrorResponse(
-                        message=f"File not found at path: {path}",
-                        session_id=session_id,
-                    )
-                target_file_id = file_info.id
-
-            success = await manager.delete_file(target_file_id)
-
-            if not success:
-                return ErrorResponse(
-                    message=f"File not found: {target_file_id}",
-                    session_id=session_id,
-                )
-
-            return WorkspaceDeleteResponse(
-                file_id=target_file_id,
-                success=True,
-                message="File deleted successfully",
-                session_id=session_id,
-            )
-
-        except Exception as e:
-            logger.error(f"Error deleting workspace file: {e}", exc_info=True)
-            return ErrorResponse(
-                message=f"Failed to delete workspace file: {str(e)}",
-                error=str(e),
-                session_id=session_id,
-            )

autogpt_platform/backend/backend/api/features/executions/review/review_routes_test.py

@@ -638,7 +638,7 @@ async def test_process_review_action_auto_approve_creates_auto_approval_records(
 
     # Mock get_node_executions to return node_id mapping
     mock_get_node_executions = mocker.patch(
-        "backend.data.execution.get_node_executions"
+        "backend.api.features.executions.review.routes.get_node_executions"
     )
     mock_node_exec = mocker.Mock(spec=NodeExecutionResult)
     mock_node_exec.node_exec_id = "test_node_123"
@@ -936,7 +936,7 @@ async def test_process_review_action_auto_approve_only_applies_to_approved_revie
 
     # Mock get_node_executions to return node_id mapping
     mock_get_node_executions = mocker.patch(
-        "backend.data.execution.get_node_executions"
+        "backend.api.features.executions.review.routes.get_node_executions"
     )
     mock_node_exec = mocker.Mock(spec=NodeExecutionResult)
     mock_node_exec.node_exec_id = "node_exec_approved"
@@ -1148,7 +1148,7 @@ async def test_process_review_action_per_review_auto_approve_granularity(
 
     # Mock get_node_executions to return batch node data
     mock_get_node_executions = mocker.patch(
-        "backend.data.execution.get_node_executions"
+        "backend.api.features.executions.review.routes.get_node_executions"
     )
     # Create mock node executions for each review
     mock_node_execs = []

autogpt_platform/backend/backend/api/features/executions/review/routes.py

@@ -6,10 +6,15 @@ import autogpt_libs.auth as autogpt_auth_lib
 from fastapi import APIRouter, HTTPException, Query, Security, status
 from prisma.enums import ReviewStatus
 
+from backend.copilot.constants import (
+    is_copilot_synthetic_id,
+    parse_node_id_from_exec_id,
+)
 from backend.data.execution import (
     ExecutionContext,
     ExecutionStatus,
     get_graph_execution_meta,
+    get_node_executions,
 )
 from backend.data.graph import get_graph_settings
 from backend.data.human_review import (
@@ -22,6 +27,7 @@ from backend.data.human_review import (
 )
 from backend.data.model import USER_TIMEZONE_NOT_SET
 from backend.data.user import get_user_by_id
+from backend.data.workspace import get_or_create_workspace
 from backend.executor.utils import add_graph_execution
 
 from .model import PendingHumanReviewModel, ReviewRequest, ReviewResponse
@@ -35,6 +41,38 @@ router = APIRouter(
 )
 
 
+async def _resolve_node_ids(
+    node_exec_ids: list[str],
+    graph_exec_id: str,
+    is_copilot: bool,
+) -> dict[str, str]:
+    """Resolve node_exec_id -> node_id for auto-approval records.
+
+    CoPilot synthetic IDs encode node_id in the format "{node_id}:{random}".
+    Graph executions look up node_id from NodeExecution records.
+    """
+    if not node_exec_ids:
+        return {}
+
+    if is_copilot:
+        return {neid: parse_node_id_from_exec_id(neid) for neid in node_exec_ids}
+
+    node_execs = await get_node_executions(
+        graph_exec_id=graph_exec_id, include_exec_data=False
+    )
+    node_exec_map = {ne.node_exec_id: ne.node_id for ne in node_execs}
+
+    result = {}
+    for neid in node_exec_ids:
+        if neid in node_exec_map:
+            result[neid] = node_exec_map[neid]
+        else:
+            logger.error(
+                f"Failed to resolve node_id for {neid}: Node execution not found."
+            )
+    return result
+
+
 @router.get(
     "/pending",
     summary="Get Pending Reviews",
@@ -109,14 +147,16 @@ async def list_pending_reviews_for_execution(
     """
 
     # Verify user owns the graph execution before returning reviews
-    graph_exec = await get_graph_execution_meta(
-        user_id=user_id, execution_id=graph_exec_id
-    )
-    if not graph_exec:
-        raise HTTPException(
-            status_code=status.HTTP_404_NOT_FOUND,
-            detail=f"Graph execution #{graph_exec_id} not found",
+    # (CoPilot synthetic IDs don't have graph execution records)
+    if not is_copilot_synthetic_id(graph_exec_id):
+        graph_exec = await get_graph_execution_meta(
+            user_id=user_id, execution_id=graph_exec_id
         )
+        if not graph_exec:
+            raise HTTPException(
+                status_code=status.HTTP_404_NOT_FOUND,
+                detail=f"Graph execution #{graph_exec_id} not found",
+            )
 
     return await get_pending_reviews_for_execution(graph_exec_id, user_id)
 
@@ -159,30 +199,26 @@ async def process_review_action(
         )
 
     graph_exec_id = next(iter(graph_exec_ids))
+    is_copilot = is_copilot_synthetic_id(graph_exec_id)
 
-    # Validate execution status before processing reviews
-    graph_exec_meta = await get_graph_execution_meta(
-        user_id=user_id, execution_id=graph_exec_id
-    )
-
-    if not graph_exec_meta:
-        raise HTTPException(
-            status_code=status.HTTP_404_NOT_FOUND,
-            detail=f"Graph execution #{graph_exec_id} not found",
-        )
-
-    # Only allow processing reviews if execution is paused for review
-    # or incomplete (partial execution with some reviews already processed)
-    if graph_exec_meta.status not in (
-        ExecutionStatus.REVIEW,
-        ExecutionStatus.INCOMPLETE,
-    ):
-        raise HTTPException(
-            status_code=status.HTTP_409_CONFLICT,
-            detail=f"Cannot process reviews while execution status is {graph_exec_meta.status}. "
-            f"Reviews can only be processed when execution is paused (REVIEW status). "
-            f"Current status: {graph_exec_meta.status}",
+    # Validate execution status for graph executions (skip for CoPilot synthetic IDs)
+    if not is_copilot:
+        graph_exec_meta = await get_graph_execution_meta(
+            user_id=user_id, execution_id=graph_exec_id
         )
+        if not graph_exec_meta:
+            raise HTTPException(
+                status_code=status.HTTP_404_NOT_FOUND,
+                detail=f"Graph execution #{graph_exec_id} not found",
+            )
+        if graph_exec_meta.status not in (
+            ExecutionStatus.REVIEW,
+            ExecutionStatus.INCOMPLETE,
+        ):
+            raise HTTPException(
+                status_code=status.HTTP_409_CONFLICT,
+                detail=f"Cannot process reviews while execution status is {graph_exec_meta.status}",
+            )
 
     # Build review decisions map and track which reviews requested auto-approval
     # Auto-approved reviews use original data (no modifications allowed)
@@ -235,7 +271,7 @@ async def process_review_action(
             )
             return (node_id, False)
 
-    # Collect node_exec_ids that need auto-approval
+    # Collect node_exec_ids that need auto-approval and resolve their node_ids
     node_exec_ids_needing_auto_approval = [
         node_exec_id
         for node_exec_id, review_result in updated_reviews.items()
@@ -243,29 +279,16 @@ async def process_review_action(
         and auto_approve_requests.get(node_exec_id, False)
     ]
 
-    # Batch-fetch node executions to get node_ids
+    node_id_map = await _resolve_node_ids(
+        node_exec_ids_needing_auto_approval, graph_exec_id, is_copilot
+    )
+
+    # Deduplicate by node_id — one auto-approval per node
     nodes_needing_auto_approval: dict[str, Any] = {}
-    if node_exec_ids_needing_auto_approval:
-        from backend.data.execution import get_node_executions
-
-        node_execs = await get_node_executions(
-            graph_exec_id=graph_exec_id, include_exec_data=False
-        )
-        node_exec_map = {node_exec.node_exec_id: node_exec for node_exec in node_execs}
-
-        for node_exec_id in node_exec_ids_needing_auto_approval:
-            node_exec = node_exec_map.get(node_exec_id)
-            if node_exec:
-                review_result = updated_reviews[node_exec_id]
-                # Use the first approved review for this node (deduplicate by node_id)
-                if node_exec.node_id not in nodes_needing_auto_approval:
-                    nodes_needing_auto_approval[node_exec.node_id] = review_result
-            else:
-                logger.error(
-                    f"Failed to create auto-approval record for {node_exec_id}: "
-                    f"Node execution not found. This may indicate a race condition "
-                    f"or data inconsistency."
-                )
+    for node_exec_id in node_exec_ids_needing_auto_approval:
+        node_id = node_id_map.get(node_exec_id)
+        if node_id and node_id not in nodes_needing_auto_approval:
+            nodes_needing_auto_approval[node_id] = updated_reviews[node_exec_id]
 
     # Execute all auto-approval creations in parallel (deduplicated by node_id)
     auto_approval_results = await asyncio.gather(
@@ -280,13 +303,11 @@ async def process_review_action(
     auto_approval_failed_count = 0
     for result in auto_approval_results:
         if isinstance(result, Exception):
-            # Unexpected exception during auto-approval creation
             auto_approval_failed_count += 1
             logger.error(
                 f"Unexpected exception during auto-approval creation: {result}"
             )
         elif isinstance(result, tuple) and len(result) == 2 and not result[1]:
-            # Auto-approval creation failed (returned False)
             auto_approval_failed_count += 1
 
     # Count results
@@ -301,30 +322,31 @@ async def process_review_action(
         if review.status == ReviewStatus.REJECTED
     )
 
-    # Resume execution only if ALL pending reviews for this execution have been processed
-    if updated_reviews:
+    # Resume graph execution only for real graph executions (not CoPilot)
+    # CoPilot sessions are resumed by the LLM retrying run_block with review_id
+    if not is_copilot and updated_reviews:
         still_has_pending = await has_pending_reviews_for_graph_exec(graph_exec_id)
 
         if not still_has_pending:
-            # Get the graph_id from any processed review
             first_review = next(iter(updated_reviews.values()))
 
             try:
-                # Fetch user and settings to build complete execution context
                 user = await get_user_by_id(user_id)
                 settings = await get_graph_settings(
                     user_id=user_id, graph_id=first_review.graph_id
                 )
 
-                # Preserve user's timezone preference when resuming execution
                 user_timezone = (
                     user.timezone if user.timezone != USER_TIMEZONE_NOT_SET else "UTC"
                 )
 
+                workspace = await get_or_create_workspace(user_id)
+
                 execution_context = ExecutionContext(
                     human_in_the_loop_safe_mode=settings.human_in_the_loop_safe_mode,
                     sensitive_action_safe_mode=settings.sensitive_action_safe_mode,
                     user_timezone=user_timezone,
+                    workspace_id=workspace.id,
                 )
 
                 await add_graph_execution(

autogpt_platform/backend/backend/api/features/library/db_test.py

@@ -4,7 +4,6 @@ import prisma.enums
 import prisma.models
 import pytest
 
-import backend.api.features.store.exceptions
 from backend.data.db import connect
 from backend.data.includes import library_agent_include
 
@@ -144,6 +143,7 @@ async def test_add_agent_to_library(mocker):
     )
 
     mock_library_agent = mocker.patch("prisma.models.LibraryAgent.prisma")
+    mock_library_agent.return_value.find_first = mocker.AsyncMock(return_value=None)
     mock_library_agent.return_value.find_unique = mocker.AsyncMock(return_value=None)
     mock_library_agent.return_value.create = mocker.AsyncMock(
         return_value=mock_library_agent_data
@@ -178,7 +178,6 @@ async def test_add_agent_to_library(mocker):
                 "agentGraphVersion": 1,
             }
         },
-        include={"AgentGraph": True},
     )
     # Check that create was called with the expected data including settings
     create_call_args = mock_library_agent.return_value.create.call_args
@@ -218,7 +217,7 @@ async def test_add_agent_to_library_not_found(mocker):
     )
 
     # Call function and verify exception
-    with pytest.raises(backend.api.features.store.exceptions.AgentNotFoundError):
+    with pytest.raises(db.NotFoundError):
         await db.add_store_agent_to_library("version123", "test-user")
 
     # Verify mock called correctly

autogpt_platform/backend/backend/api/features/library/exceptions.py

@@ -0,0 +1,10 @@
+class FolderValidationError(Exception):
+    """Raised when folder operations fail validation."""
+
+    pass
+
+
+class FolderAlreadyExistsError(FolderValidationError):
+    """Raised when a folder with the same name already exists in the location."""
+
+    pass

autogpt_platform/backend/backend/api/features/library/model.py

@@ -26,6 +26,95 @@ class LibraryAgentStatus(str, Enum):
     ERROR = "ERROR"
 
 
+# === Folder Models ===
+
+
+class LibraryFolder(pydantic.BaseModel):
+    """Represents a folder for organizing library agents."""
+
+    id: str
+    user_id: str
+    name: str
+    icon: str | None = None
+    color: str | None = None
+    parent_id: str | None = None
+    created_at: datetime.datetime
+    updated_at: datetime.datetime
+    agent_count: int = 0  # Direct agents in folder
+    subfolder_count: int = 0  # Direct child folders
+
+    @staticmethod
+    def from_db(
+        folder: prisma.models.LibraryFolder,
+        agent_count: int = 0,
+        subfolder_count: int = 0,
+    ) -> "LibraryFolder":
+        """Factory method that constructs a LibraryFolder from a Prisma model."""
+        return LibraryFolder(
+            id=folder.id,
+            user_id=folder.userId,
+            name=folder.name,
+            icon=folder.icon,
+            color=folder.color,
+            parent_id=folder.parentId,
+            created_at=folder.createdAt,
+            updated_at=folder.updatedAt,
+            agent_count=agent_count,
+            subfolder_count=subfolder_count,
+        )
+
+
+class LibraryFolderTree(LibraryFolder):
+    """Folder with nested children for tree view."""
+
+    children: list["LibraryFolderTree"] = []
+
+
+class FolderCreateRequest(pydantic.BaseModel):
+    """Request model for creating a folder."""
+
+    name: str = pydantic.Field(..., min_length=1, max_length=100)
+    icon: str | None = None
+    color: str | None = pydantic.Field(
+        None, pattern=r"^#[0-9A-Fa-f]{6}$", description="Hex color code (#RRGGBB)"
+    )
+    parent_id: str | None = None
+
+
+class FolderUpdateRequest(pydantic.BaseModel):
+    """Request model for updating a folder."""
+
+    name: str | None = pydantic.Field(None, min_length=1, max_length=100)
+    icon: str | None = None
+    color: str | None = None
+
+
+class FolderMoveRequest(pydantic.BaseModel):
+    """Request model for moving a folder to a new parent."""
+
+    target_parent_id: str | None = None  # None = move to root
+
+
+class BulkMoveAgentsRequest(pydantic.BaseModel):
+    """Request model for moving multiple agents to a folder."""
+
+    agent_ids: list[str]
+    folder_id: str | None = None  # None = move to root
+
+
+class FolderListResponse(pydantic.BaseModel):
+    """Response schema for a list of folders."""
+
+    folders: list[LibraryFolder]
+    pagination: Pagination
+
+
+class FolderTreeResponse(pydantic.BaseModel):
+    """Response schema for folder tree structure."""
+
+    tree: list[LibraryFolderTree]
+
+
 class MarketplaceListingCreator(pydantic.BaseModel):
     """Creator information for a marketplace listing."""
 
@@ -76,7 +165,6 @@ class LibraryAgent(pydantic.BaseModel):
     id: str
     graph_id: str
     graph_version: int
-    owner_user_id: str
 
     image_url: str | None
 
@@ -117,9 +205,14 @@ class LibraryAgent(pydantic.BaseModel):
         default_factory=list,
         description="List of recent executions with status, score, and summary",
     )
-    can_access_graph: bool
+    can_access_graph: bool = pydantic.Field(
+        description="Indicates whether the same user owns the corresponding graph"
+    )
     is_latest_version: bool
     is_favorite: bool
+    folder_id: str | None = None
+    folder_name: str | None = None  # Denormalized for display
+
     recommended_schedule_cron: str | None = None
     settings: GraphSettings = pydantic.Field(default_factory=GraphSettings)
     marketplace_listing: Optional["MarketplaceListing"] = None
@@ -232,7 +325,6 @@ class LibraryAgent(pydantic.BaseModel):
             id=agent.id,
             graph_id=agent.agentGraphId,
             graph_version=agent.agentGraphVersion,
-            owner_user_id=agent.userId,
             image_url=agent.imageUrl,
             creator_name=creator_name,
             creator_image_url=creator_image_url,
@@ -259,6 +351,8 @@ class LibraryAgent(pydantic.BaseModel):
             can_access_graph=can_access_graph,
             is_latest_version=is_latest_version,
             is_favorite=agent.isFavorite,
+            folder_id=agent.folderId,
+            folder_name=agent.Folder.name if agent.Folder else None,
             recommended_schedule_cron=agent.AgentGraph.recommendedScheduleCron,
             settings=_parse_settings(agent.settings),
             marketplace_listing=marketplace_listing_data,
@@ -470,3 +564,7 @@ class LibraryAgentUpdateRequest(pydantic.BaseModel):
     settings: Optional[GraphSettings] = pydantic.Field(
         default=None, description="User-specific settings for this library agent"
     )
+    folder_id: Optional[str] = pydantic.Field(
+        default=None,
+        description="Folder ID to move agent to (None to move to root)",
+    )

autogpt_platform/backend/backend/api/features/library/routes/__init__.py

@@ -1,9 +1,11 @@
 import fastapi
 
 from .agents import router as agents_router
+from .folders import router as folders_router
 from .presets import router as presets_router
 
 router = fastapi.APIRouter()
 
 router.include_router(presets_router)
+router.include_router(folders_router)
 router.include_router(agents_router)

autogpt_platform/backend/backend/api/features/library/routes/agents.py

@@ -41,6 +41,14 @@ async def list_library_agents(
         ge=1,
         description="Number of agents per page (must be >= 1)",
     ),
+    folder_id: Optional[str] = Query(
+        None,
+        description="Filter by folder ID",
+    ),
+    include_root_only: bool = Query(
+        False,
+        description="Only return agents without a folder (root-level agents)",
+    ),
 ) -> library_model.LibraryAgentResponse:
     """
     Get all agents in the user's library (both created and saved).
@@ -51,6 +59,8 @@ async def list_library_agents(
         sort_by=sort_by,
         page=page,
         page_size=page_size,
+        folder_id=folder_id,
+        include_root_only=include_root_only,
     )
 
 
@@ -168,6 +178,7 @@ async def update_library_agent(
         is_favorite=payload.is_favorite,
         is_archived=payload.is_archived,
         settings=payload.settings,
+        folder_id=payload.folder_id,
     )
 
 

autogpt_platform/backend/backend/api/features/library/routes/folders.py

@@ -0,0 +1,287 @@
+from typing import Optional
+
+import autogpt_libs.auth as autogpt_auth_lib
+from fastapi import APIRouter, Query, Security, status
+from fastapi.responses import Response
+
+from .. import db as library_db
+from .. import model as library_model
+
+router = APIRouter(
+    prefix="/folders",
+    tags=["library", "folders", "private"],
+    dependencies=[Security(autogpt_auth_lib.requires_user)],
+)
+
+
+@router.get(
+    "",
+    summary="List Library Folders",
+    response_model=library_model.FolderListResponse,
+    responses={
+        200: {"description": "List of folders"},
+        500: {"description": "Server error"},
+    },
+)
+async def list_folders(
+    user_id: str = Security(autogpt_auth_lib.get_user_id),
+    parent_id: Optional[str] = Query(
+        None,
+        description="Filter by parent folder ID. If not provided, returns root-level folders.",
+    ),
+    include_relations: bool = Query(
+        True,
+        description="Include agent and subfolder relations (for counts)",
+    ),
+) -> library_model.FolderListResponse:
+    """
+    List folders for the authenticated user.
+
+    Args:
+        user_id: ID of the authenticated user.
+        parent_id: Optional parent folder ID to filter by.
+        include_relations: Whether to include agent and subfolder relations for counts.
+
+    Returns:
+        A FolderListResponse containing folders.
+    """
+    folders = await library_db.list_folders(
+        user_id=user_id,
+        parent_id=parent_id,
+        include_relations=include_relations,
+    )
+    return library_model.FolderListResponse(
+        folders=folders,
+        pagination=library_model.Pagination(
+            total_items=len(folders),
+            total_pages=1,
+            current_page=1,
+            page_size=len(folders),
+        ),
+    )
+
+
+@router.get(
+    "/tree",
+    summary="Get Folder Tree",
+    response_model=library_model.FolderTreeResponse,
+    responses={
+        200: {"description": "Folder tree structure"},
+        500: {"description": "Server error"},
+    },
+)
+async def get_folder_tree(
+    user_id: str = Security(autogpt_auth_lib.get_user_id),
+) -> library_model.FolderTreeResponse:
+    """
+    Get the full folder tree for the authenticated user.
+
+    Args:
+        user_id: ID of the authenticated user.
+
+    Returns:
+        A FolderTreeResponse containing the nested folder structure.
+    """
+    tree = await library_db.get_folder_tree(user_id=user_id)
+    return library_model.FolderTreeResponse(tree=tree)
+
+
+@router.get(
+    "/{folder_id}",
+    summary="Get Folder",
+    response_model=library_model.LibraryFolder,
+    responses={
+        200: {"description": "Folder details"},
+        404: {"description": "Folder not found"},
+        500: {"description": "Server error"},
+    },
+)
+async def get_folder(
+    folder_id: str,
+    user_id: str = Security(autogpt_auth_lib.get_user_id),
+) -> library_model.LibraryFolder:
+    """
+    Get a specific folder.
+
+    Args:
+        folder_id: ID of the folder to retrieve.
+        user_id: ID of the authenticated user.
+
+    Returns:
+        The requested LibraryFolder.
+    """
+    return await library_db.get_folder(folder_id=folder_id, user_id=user_id)
+
+
+@router.post(
+    "",
+    summary="Create Folder",
+    status_code=status.HTTP_201_CREATED,
+    response_model=library_model.LibraryFolder,
+    responses={
+        201: {"description": "Folder created successfully"},
+        400: {"description": "Validation error"},
+        404: {"description": "Parent folder not found"},
+        409: {"description": "Folder name conflict"},
+        500: {"description": "Server error"},
+    },
+)
+async def create_folder(
+    payload: library_model.FolderCreateRequest,
+    user_id: str = Security(autogpt_auth_lib.get_user_id),
+) -> library_model.LibraryFolder:
+    """
+    Create a new folder.
+
+    Args:
+        payload: The folder creation request.
+        user_id: ID of the authenticated user.
+
+    Returns:
+        The created LibraryFolder.
+    """
+    return await library_db.create_folder(
+        user_id=user_id,
+        name=payload.name,
+        parent_id=payload.parent_id,
+        icon=payload.icon,
+        color=payload.color,
+    )
+
+
+@router.patch(
+    "/{folder_id}",
+    summary="Update Folder",
+    response_model=library_model.LibraryFolder,
+    responses={
+        200: {"description": "Folder updated successfully"},
+        400: {"description": "Validation error"},
+        404: {"description": "Folder not found"},
+        409: {"description": "Folder name conflict"},
+        500: {"description": "Server error"},
+    },
+)
+async def update_folder(
+    folder_id: str,
+    payload: library_model.FolderUpdateRequest,
+    user_id: str = Security(autogpt_auth_lib.get_user_id),
+) -> library_model.LibraryFolder:
+    """
+    Update a folder's properties.
+
+    Args:
+        folder_id: ID of the folder to update.
+        payload: The folder update request.
+        user_id: ID of the authenticated user.
+
+    Returns:
+        The updated LibraryFolder.
+    """
+    return await library_db.update_folder(
+        folder_id=folder_id,
+        user_id=user_id,
+        name=payload.name,
+        icon=payload.icon,
+        color=payload.color,
+    )
+
+
+@router.post(
+    "/{folder_id}/move",
+    summary="Move Folder",
+    response_model=library_model.LibraryFolder,
+    responses={
+        200: {"description": "Folder moved successfully"},
+        400: {"description": "Validation error (circular reference)"},
+        404: {"description": "Folder or target parent not found"},
+        409: {"description": "Folder name conflict in target location"},
+        500: {"description": "Server error"},
+    },
+)
+async def move_folder(
+    folder_id: str,
+    payload: library_model.FolderMoveRequest,
+    user_id: str = Security(autogpt_auth_lib.get_user_id),
+) -> library_model.LibraryFolder:
+    """
+    Move a folder to a new parent.
+
+    Args:
+        folder_id: ID of the folder to move.
+        payload: The move request with target parent.
+        user_id: ID of the authenticated user.
+
+    Returns:
+        The moved LibraryFolder.
+    """
+    return await library_db.move_folder(
+        folder_id=folder_id,
+        user_id=user_id,
+        target_parent_id=payload.target_parent_id,
+    )
+
+
+@router.delete(
+    "/{folder_id}",
+    summary="Delete Folder",
+    status_code=status.HTTP_204_NO_CONTENT,
+    responses={
+        204: {"description": "Folder deleted successfully"},
+        404: {"description": "Folder not found"},
+        500: {"description": "Server error"},
+    },
+)
+async def delete_folder(
+    folder_id: str,
+    user_id: str = Security(autogpt_auth_lib.get_user_id),
+) -> Response:
+    """
+    Soft-delete a folder and all its contents.
+
+    Args:
+        folder_id: ID of the folder to delete.
+        user_id: ID of the authenticated user.
+
+    Returns:
+        204 No Content if successful.
+    """
+    await library_db.delete_folder(
+        folder_id=folder_id,
+        user_id=user_id,
+        soft_delete=True,
+    )
+    return Response(status_code=status.HTTP_204_NO_CONTENT)
+
+
+# === Bulk Agent Operations ===
+
+
+@router.post(
+    "/agents/bulk-move",
+    summary="Bulk Move Agents",
+    response_model=list[library_model.LibraryAgent],
+    responses={
+        200: {"description": "Agents moved successfully"},
+        404: {"description": "Folder not found"},
+        500: {"description": "Server error"},
+    },
+)
+async def bulk_move_agents(
+    payload: library_model.BulkMoveAgentsRequest,
+    user_id: str = Security(autogpt_auth_lib.get_user_id),
+) -> list[library_model.LibraryAgent]:
+    """
+    Move multiple agents to a folder.
+
+    Args:
+        payload: The bulk move request with agent IDs and target folder.
+        user_id: ID of the authenticated user.
+
+    Returns:
+        The updated LibraryAgents.
+    """
+    return await library_db.bulk_move_agents_to_folder(
+        agent_ids=payload.agent_ids,
+        folder_id=payload.folder_id,
+        user_id=user_id,
+    )

autogpt_platform/backend/backend/api/features/library/routes_test.py

@@ -42,7 +42,6 @@ async def test_get_library_agents_success(
                 id="test-agent-1",
                 graph_id="test-agent-1",
                 graph_version=1,
-                owner_user_id=test_user_id,
                 name="Test Agent 1",
                 description="Test Description 1",
                 image_url=None,
@@ -67,7 +66,6 @@ async def test_get_library_agents_success(
                 id="test-agent-2",
                 graph_id="test-agent-2",
                 graph_version=1,
-                owner_user_id=test_user_id,
                 name="Test Agent 2",
                 description="Test Description 2",
                 image_url=None,
@@ -115,6 +113,8 @@ async def test_get_library_agents_success(
         sort_by=library_model.LibraryAgentSort.UPDATED_AT,
         page=1,
         page_size=15,
+        folder_id=None,
+        include_root_only=False,
     )
 
 
@@ -129,7 +129,6 @@ async def test_get_favorite_library_agents_success(
                 id="test-agent-1",
                 graph_id="test-agent-1",
                 graph_version=1,
-                owner_user_id=test_user_id,
                 name="Favorite Agent 1",
                 description="Test Favorite Description 1",
                 image_url=None,
@@ -182,7 +181,6 @@ def test_add_agent_to_library_success(
         id="test-library-agent-id",
         graph_id="test-agent-1",
         graph_version=1,
-        owner_user_id=test_user_id,
         name="Test Agent 1",
         description="Test Description 1",
         image_url=None,

autogpt_platform/backend/backend/api/features/mcp/routes.py

@@ -7,20 +7,24 @@ frontend can list available tools on an MCP server before placing a block.
 
 import logging
 from typing import Annotated, Any
-from urllib.parse import urlparse
 
 import fastapi
 from autogpt_libs.auth import get_user_id
 from fastapi import Security
-from pydantic import BaseModel, Field
+from pydantic import BaseModel, Field, SecretStr
 
 from backend.api.features.integrations.router import CredentialsMetaResponse
 from backend.blocks.mcp.client import MCPClient, MCPClientError
+from backend.blocks.mcp.helpers import (
+    auto_lookup_mcp_credential,
+    normalize_mcp_url,
+    server_host,
+)
 from backend.blocks.mcp.oauth import MCPOAuthHandler
 from backend.data.model import OAuth2Credentials
 from backend.integrations.creds_manager import IntegrationCredentialsManager
 from backend.integrations.providers import ProviderName
-from backend.util.request import HTTPClientError, Requests
+from backend.util.request import HTTPClientError, Requests, validate_url_host
 from backend.util.settings import Settings
 
 logger = logging.getLogger(__name__)
@@ -74,32 +78,20 @@ async def discover_tools(
     If the user has a stored MCP credential for this server URL, it will be
     used automatically — no need to pass an explicit auth token.
     """
+    # Validate URL to prevent SSRF — blocks loopback and private IP ranges.
+    try:
+        await validate_url_host(request.server_url)
+    except ValueError as e:
+        raise fastapi.HTTPException(status_code=400, detail=f"Invalid server URL: {e}")
+
     auth_token = request.auth_token
 
     # Auto-use stored MCP credential when no explicit token is provided.
     if not auth_token:
-        mcp_creds = await creds_manager.store.get_creds_by_provider(
-            user_id, ProviderName.MCP.value
+        best_cred = await auto_lookup_mcp_credential(
+            user_id, normalize_mcp_url(request.server_url)
         )
-        # Find the freshest credential for this server URL
-        best_cred: OAuth2Credentials | None = None
-        for cred in mcp_creds:
-            if (
-                isinstance(cred, OAuth2Credentials)
-                and (cred.metadata or {}).get("mcp_server_url") == request.server_url
-            ):
-                if best_cred is None or (
-                    (cred.access_token_expires_at or 0)
-                    > (best_cred.access_token_expires_at or 0)
-                ):
-                    best_cred = cred
         if best_cred:
-            # Refresh the token if expired before using it
-            best_cred = await creds_manager.refresh_if_needed(user_id, best_cred)
-            logger.info(
-                f"Using MCP credential {best_cred.id} for {request.server_url}, "
-                f"expires_at={best_cred.access_token_expires_at}"
-            )
             auth_token = best_cred.access_token.get_secret_value()
 
     client = MCPClient(request.server_url, auth_token=auth_token)
@@ -134,7 +126,7 @@ async def discover_tools(
         ],
         server_name=(
             init_result.get("serverInfo", {}).get("name")
-            or urlparse(request.server_url).hostname
+            or server_host(request.server_url)
             or "MCP"
         ),
         protocol_version=init_result.get("protocolVersion"),
@@ -173,7 +165,16 @@ async def mcp_oauth_login(
     3. Performs Dynamic Client Registration (RFC 7591) if available
     4. Returns the authorization URL for the frontend to open in a popup
     """
-    client = MCPClient(request.server_url)
+    # Validate URL to prevent SSRF — blocks loopback and private IP ranges.
+    try:
+        await validate_url_host(request.server_url)
+    except ValueError as e:
+        raise fastapi.HTTPException(status_code=400, detail=f"Invalid server URL: {e}")
+
+    # Normalize the URL so that credentials stored here are matched consistently
+    # by auto_lookup_mcp_credential (which also uses normalized URLs).
+    server_url = normalize_mcp_url(request.server_url)
+    client = MCPClient(server_url)
 
     # Step 1: Discover protected-resource metadata (RFC 9728)
     protected_resource = await client.discover_auth()
@@ -182,7 +183,16 @@ async def mcp_oauth_login(
 
     if protected_resource and protected_resource.get("authorization_servers"):
         auth_server_url = protected_resource["authorization_servers"][0]
-        resource_url = protected_resource.get("resource", request.server_url)
+        resource_url = protected_resource.get("resource", server_url)
+
+        # Validate the auth server URL from metadata to prevent SSRF.
+        try:
+            await validate_url_host(auth_server_url)
+        except ValueError as e:
+            raise fastapi.HTTPException(
+                status_code=400,
+                detail=f"Invalid authorization server URL in metadata: {e}",
+            )
 
         # Step 2a: Discover auth-server metadata (RFC 8414)
         metadata = await client.discover_auth_server_metadata(auth_server_url)
@@ -192,7 +202,7 @@ async def mcp_oauth_login(
         # Don't assume a resource_url — omitting it lets the auth server choose
         # the correct audience for the token (RFC 8707 resource is optional).
         resource_url = None
-        metadata = await client.discover_auth_server_metadata(request.server_url)
+        metadata = await client.discover_auth_server_metadata(server_url)
 
     if (
         not metadata
@@ -222,12 +232,18 @@ async def mcp_oauth_login(
     client_id = ""
     client_secret = ""
     if registration_endpoint:
-        reg_result = await _register_mcp_client(
-            registration_endpoint, redirect_uri, request.server_url
-        )
-        if reg_result:
-            client_id = reg_result.get("client_id", "")
-            client_secret = reg_result.get("client_secret", "")
+        # Validate the registration endpoint to prevent SSRF via metadata.
+        try:
+            await validate_url_host(registration_endpoint)
+        except ValueError:
+            pass  # Skip registration, fall back to default client_id
+        else:
+            reg_result = await _register_mcp_client(
+                registration_endpoint, redirect_uri, server_url
+            )
+            if reg_result:
+                client_id = reg_result.get("client_id", "")
+                client_secret = reg_result.get("client_secret", "")
 
     if not client_id:
         client_id = "autogpt-platform"
@@ -245,7 +261,7 @@ async def mcp_oauth_login(
             "token_url": token_url,
             "revoke_url": revoke_url,
             "resource_url": resource_url,
-            "server_url": request.server_url,
+            "server_url": server_url,
             "client_id": client_id,
             "client_secret": client_secret,
         },
@@ -342,7 +358,7 @@ async def mcp_oauth_callback(
     credentials.metadata["mcp_token_url"] = meta["token_url"]
     credentials.metadata["mcp_resource_url"] = meta.get("resource_url", "")
 
-    hostname = urlparse(meta["server_url"]).hostname or meta["server_url"]
+    hostname = server_host(meta["server_url"])
     credentials.title = f"MCP: {hostname}"
 
     # Remove old MCP credentials for the same server to prevent stale token buildup.
@@ -357,7 +373,9 @@ async def mcp_oauth_callback(
             ):
                 await creds_manager.store.delete_creds_by_id(user_id, old.id)
                 logger.info(
-                    f"Removed old MCP credential {old.id} for {meta['server_url']}"
+                    "Removed old MCP credential %s for %s",
+                    old.id,
+                    server_host(meta["server_url"]),
                 )
     except Exception:
         logger.debug("Could not clean up old MCP credentials", exc_info=True)
@@ -375,6 +393,93 @@ async def mcp_oauth_callback(
     )
 
 
+# ======================== Bearer Token ======================== #
+
+
+class MCPStoreTokenRequest(BaseModel):
+    """Request to store a bearer token for an MCP server that doesn't support OAuth."""
+
+    server_url: str = Field(
+        description="MCP server URL the token authenticates against"
+    )
+    token: SecretStr = Field(
+        min_length=1, description="Bearer token / API key for the MCP server"
+    )
+
+
+@router.post(
+    "/token",
+    summary="Store a bearer token for an MCP server",
+)
+async def mcp_store_token(
+    request: MCPStoreTokenRequest,
+    user_id: Annotated[str, Security(get_user_id)],
+) -> CredentialsMetaResponse:
+    """
+    Store a manually provided bearer token as an MCP credential.
+
+    Used by the Copilot MCPSetupCard when the server doesn't support the MCP
+    OAuth discovery flow (returns 400 from /oauth/login).  Subsequent
+    ``run_mcp_tool`` calls will automatically pick up the token via
+    ``_auto_lookup_credential``.
+    """
+    token = request.token.get_secret_value().strip()
+    if not token:
+        raise fastapi.HTTPException(status_code=422, detail="Token must not be blank.")
+
+    # Validate URL to prevent SSRF — blocks loopback and private IP ranges.
+    try:
+        await validate_url_host(request.server_url)
+    except ValueError as e:
+        raise fastapi.HTTPException(status_code=400, detail=f"Invalid server URL: {e}")
+
+    # Normalize URL so trailing-slash variants match existing credentials.
+    server_url = normalize_mcp_url(request.server_url)
+    hostname = server_host(server_url)
+
+    # Collect IDs of old credentials to clean up after successful create.
+    old_cred_ids: list[str] = []
+    try:
+        old_creds = await creds_manager.store.get_creds_by_provider(
+            user_id, ProviderName.MCP.value
+        )
+        old_cred_ids = [
+            old.id
+            for old in old_creds
+            if isinstance(old, OAuth2Credentials)
+            and normalize_mcp_url((old.metadata or {}).get("mcp_server_url", ""))
+            == server_url
+        ]
+    except Exception:
+        logger.debug("Could not query old MCP token credentials", exc_info=True)
+
+    credentials = OAuth2Credentials(
+        provider=ProviderName.MCP.value,
+        title=f"MCP: {hostname}",
+        access_token=SecretStr(token),
+        scopes=[],
+        metadata={"mcp_server_url": server_url},
+    )
+    await creds_manager.create(user_id, credentials)
+
+    # Only delete old credentials after the new one is safely stored.
+    for old_id in old_cred_ids:
+        try:
+            await creds_manager.store.delete_creds_by_id(user_id, old_id)
+        except Exception:
+            logger.debug("Could not clean up old MCP token credential", exc_info=True)
+
+    return CredentialsMetaResponse(
+        id=credentials.id,
+        provider=credentials.provider,
+        type=credentials.type,
+        title=credentials.title,
+        scopes=credentials.scopes,
+        username=credentials.username,
+        host=hostname,
+    )
+
+
 # ======================== Helpers ======================== #
 
 
@@ -400,5 +505,7 @@ async def _register_mcp_client(
             return data
         return None
     except Exception as e:
-        logger.warning(f"Dynamic client registration failed for {server_url}: {e}")
+        logger.warning(
+            "Dynamic client registration failed for %s: %s", server_host(server_url), e
+        )
         return None

autogpt_platform/backend/backend/api/features/mcp/test_routes.py

@@ -11,9 +11,11 @@ import httpx
 import pytest
 import pytest_asyncio
 from autogpt_libs.auth import get_user_id
+from pydantic import SecretStr
 
 from backend.api.features.mcp.routes import router
 from backend.blocks.mcp.client import MCPClientError, MCPTool
+from backend.data.model import OAuth2Credentials
 from backend.util.request import HTTPClientError
 
 app = fastapi.FastAPI()
@@ -28,6 +30,16 @@ async def client():
         yield c
 
 
+@pytest.fixture(autouse=True)
+def _bypass_ssrf_validation():
+    """Bypass validate_url_host in all route tests (test URLs don't resolve)."""
+    with patch(
+        "backend.api.features.mcp.routes.validate_url_host",
+        new_callable=AsyncMock,
+    ):
+        yield
+
+
 class TestDiscoverTools:
     @pytest.mark.asyncio(loop_scope="session")
     async def test_discover_tools_success(self, client):
@@ -56,9 +68,12 @@ class TestDiscoverTools:
 
         with (
             patch("backend.api.features.mcp.routes.MCPClient") as MockClient,
-            patch("backend.api.features.mcp.routes.creds_manager") as mock_cm,
+            patch(
+                "backend.api.features.mcp.routes.auto_lookup_mcp_credential",
+                new_callable=AsyncMock,
+                return_value=None,
+            ),
         ):
-            mock_cm.store.get_creds_by_provider = AsyncMock(return_value=[])
             instance = MockClient.return_value
             instance.initialize = AsyncMock(
                 return_value={
@@ -107,10 +122,6 @@ class TestDiscoverTools:
     @pytest.mark.asyncio(loop_scope="session")
     async def test_discover_tools_auto_uses_stored_credential(self, client):
         """When no explicit token is given, stored MCP credentials are used."""
-        from pydantic import SecretStr
-
-        from backend.data.model import OAuth2Credentials
-
         stored_cred = OAuth2Credentials(
             provider="mcp",
             title="MCP: example.com",
@@ -124,10 +135,12 @@ class TestDiscoverTools:
 
         with (
             patch("backend.api.features.mcp.routes.MCPClient") as MockClient,
-            patch("backend.api.features.mcp.routes.creds_manager") as mock_cm,
+            patch(
+                "backend.api.features.mcp.routes.auto_lookup_mcp_credential",
+                new_callable=AsyncMock,
+                return_value=stored_cred,
+            ),
         ):
-            mock_cm.store.get_creds_by_provider = AsyncMock(return_value=[stored_cred])
-            mock_cm.refresh_if_needed = AsyncMock(return_value=stored_cred)
             instance = MockClient.return_value
             instance.initialize = AsyncMock(
                 return_value={"serverInfo": {}, "protocolVersion": "2025-03-26"}
@@ -149,9 +162,12 @@ class TestDiscoverTools:
     async def test_discover_tools_mcp_error(self, client):
         with (
             patch("backend.api.features.mcp.routes.MCPClient") as MockClient,
-            patch("backend.api.features.mcp.routes.creds_manager") as mock_cm,
+            patch(
+                "backend.api.features.mcp.routes.auto_lookup_mcp_credential",
+                new_callable=AsyncMock,
+                return_value=None,
+            ),
         ):
-            mock_cm.store.get_creds_by_provider = AsyncMock(return_value=[])
             instance = MockClient.return_value
             instance.initialize = AsyncMock(
                 side_effect=MCPClientError("Connection refused")
@@ -169,9 +185,12 @@ class TestDiscoverTools:
     async def test_discover_tools_generic_error(self, client):
         with (
             patch("backend.api.features.mcp.routes.MCPClient") as MockClient,
-            patch("backend.api.features.mcp.routes.creds_manager") as mock_cm,
+            patch(
+                "backend.api.features.mcp.routes.auto_lookup_mcp_credential",
+                new_callable=AsyncMock,
+                return_value=None,
+            ),
         ):
-            mock_cm.store.get_creds_by_provider = AsyncMock(return_value=[])
             instance = MockClient.return_value
             instance.initialize = AsyncMock(side_effect=Exception("Network timeout"))
 
@@ -187,9 +206,12 @@ class TestDiscoverTools:
     async def test_discover_tools_auth_required(self, client):
         with (
             patch("backend.api.features.mcp.routes.MCPClient") as MockClient,
-            patch("backend.api.features.mcp.routes.creds_manager") as mock_cm,
+            patch(
+                "backend.api.features.mcp.routes.auto_lookup_mcp_credential",
+                new_callable=AsyncMock,
+                return_value=None,
+            ),
         ):
-            mock_cm.store.get_creds_by_provider = AsyncMock(return_value=[])
             instance = MockClient.return_value
             instance.initialize = AsyncMock(
                 side_effect=HTTPClientError("HTTP 401 Error: Unauthorized", 401)
@@ -207,9 +229,12 @@ class TestDiscoverTools:
     async def test_discover_tools_forbidden(self, client):
         with (
             patch("backend.api.features.mcp.routes.MCPClient") as MockClient,
-            patch("backend.api.features.mcp.routes.creds_manager") as mock_cm,
+            patch(
+                "backend.api.features.mcp.routes.auto_lookup_mcp_credential",
+                new_callable=AsyncMock,
+                return_value=None,
+            ),
         ):
-            mock_cm.store.get_creds_by_provider = AsyncMock(return_value=[])
             instance = MockClient.return_value
             instance.initialize = AsyncMock(
                 side_effect=HTTPClientError("HTTP 403 Error: Forbidden", 403)
@@ -331,10 +356,6 @@ class TestOAuthLogin:
 class TestOAuthCallback:
     @pytest.mark.asyncio(loop_scope="session")
     async def test_oauth_callback_success(self, client):
-        from pydantic import SecretStr
-
-        from backend.data.model import OAuth2Credentials
-
         mock_creds = OAuth2Credentials(
             provider="mcp",
             title=None,
@@ -434,3 +455,118 @@ class TestOAuthCallback:
 
         assert response.status_code == 400
         assert "token exchange failed" in response.json()["detail"].lower()
+
+
+class TestStoreToken:
+    @pytest.mark.asyncio(loop_scope="session")
+    async def test_store_token_success(self, client):
+        with patch("backend.api.features.mcp.routes.creds_manager") as mock_cm:
+            mock_cm.store.get_creds_by_provider = AsyncMock(return_value=[])
+            mock_cm.create = AsyncMock()
+
+            response = await client.post(
+                "/token",
+                json={
+                    "server_url": "https://mcp.example.com/mcp",
+                    "token": "my-api-key-123",
+                },
+            )
+
+        assert response.status_code == 200
+        data = response.json()
+        assert data["provider"] == "mcp"
+        assert data["type"] == "oauth2"
+        assert data["host"] == "mcp.example.com"
+        mock_cm.create.assert_called_once()
+
+    @pytest.mark.asyncio(loop_scope="session")
+    async def test_store_token_blank_rejected(self, client):
+        """Blank token string (after stripping) should return 422."""
+        response = await client.post(
+            "/token",
+            json={
+                "server_url": "https://mcp.example.com/mcp",
+                "token": "   ",
+            },
+        )
+        # Pydantic min_length=1 catches the whitespace-only token
+        assert response.status_code == 422
+
+    @pytest.mark.asyncio(loop_scope="session")
+    async def test_store_token_replaces_old_credential(self, client):
+        old_cred = OAuth2Credentials(
+            provider="mcp",
+            title="MCP: mcp.example.com",
+            access_token=SecretStr("old-token"),
+            scopes=[],
+            metadata={"mcp_server_url": "https://mcp.example.com/mcp"},
+        )
+        with patch("backend.api.features.mcp.routes.creds_manager") as mock_cm:
+            mock_cm.store.get_creds_by_provider = AsyncMock(return_value=[old_cred])
+            mock_cm.create = AsyncMock()
+            mock_cm.store.delete_creds_by_id = AsyncMock()
+
+            response = await client.post(
+                "/token",
+                json={
+                    "server_url": "https://mcp.example.com/mcp",
+                    "token": "new-token",
+                },
+            )
+
+        assert response.status_code == 200
+        mock_cm.store.delete_creds_by_id.assert_called_once_with(
+            "test-user-id", old_cred.id
+        )
+
+
+class TestSSRFValidation:
+    """Verify that validate_url_host is enforced on all endpoints."""
+
+    @pytest.mark.asyncio(loop_scope="session")
+    async def test_discover_tools_ssrf_blocked(self, client):
+        with patch(
+            "backend.api.features.mcp.routes.validate_url_host",
+            new_callable=AsyncMock,
+            side_effect=ValueError("blocked loopback"),
+        ):
+            response = await client.post(
+                "/discover-tools",
+                json={"server_url": "http://localhost/mcp"},
+            )
+
+        assert response.status_code == 400
+        assert "blocked loopback" in response.json()["detail"].lower()
+
+    @pytest.mark.asyncio(loop_scope="session")
+    async def test_oauth_login_ssrf_blocked(self, client):
+        with patch(
+            "backend.api.features.mcp.routes.validate_url_host",
+            new_callable=AsyncMock,
+            side_effect=ValueError("blocked private IP"),
+        ):
+            response = await client.post(
+                "/oauth/login",
+                json={"server_url": "http://10.0.0.1/mcp"},
+            )
+
+        assert response.status_code == 400
+        assert "blocked private ip" in response.json()["detail"].lower()
+
+    @pytest.mark.asyncio(loop_scope="session")
+    async def test_store_token_ssrf_blocked(self, client):
+        with patch(
+            "backend.api.features.mcp.routes.validate_url_host",
+            new_callable=AsyncMock,
+            side_effect=ValueError("blocked loopback"),
+        ):
+            response = await client.post(
+                "/token",
+                json={
+                    "server_url": "http://127.0.0.1/mcp",
+                    "token": "some-token",
+                },
+            )
+
+        assert response.status_code == 400
+        assert "blocked loopback" in response.json()["detail"].lower()

autogpt_platform/backend/backend/api/features/store/cache.py

@@ -1,5 +1,3 @@
-from typing import Literal
-
 from backend.util.cache import cached
 
 from . import db as store_db
@@ -23,7 +21,7 @@ def clear_all_caches():
 async def _get_cached_store_agents(
     featured: bool,
     creator: str | None,
-    sorted_by: Literal["rating", "runs", "name", "updated_at"] | None,
+    sorted_by: store_db.StoreAgentsSortOptions | None,
     search_query: str | None,
     category: str | None,
     page: int,
@@ -57,7 +55,7 @@ async def _get_cached_agent_details(
 async def _get_cached_store_creators(
     featured: bool,
     search_query: str | None,
-    sorted_by: Literal["agent_rating", "agent_runs", "num_agents"] | None,
+    sorted_by: store_db.StoreCreatorsSortOptions | None,
     page: int,
     page_size: int,
 ):
@@ -75,4 +73,4 @@ async def _get_cached_store_creators(
 @cached(maxsize=100, ttl_seconds=300, shared_cache=True)
 async def _get_cached_creator_details(username: str):
     """Cached helper to get creator details."""
-    return await store_db.get_store_creator_details(username=username.lower())
+    return await store_db.get_store_creator(username=username.lower())

autogpt_platform/backend/backend/api/features/store/content_handlers.py

@@ -9,15 +9,26 @@ import logging
 from abc import ABC, abstractmethod
 from dataclasses import dataclass
 from pathlib import Path
-from typing import Any
+from typing import Any, get_args, get_origin
 
 from prisma.enums import ContentType
 
+from backend.blocks.llm import LlmModel
 from backend.data.db import query_raw_with_schema
 
 logger = logging.getLogger(__name__)
 
 
+def _contains_type(annotation: Any, target: type) -> bool:
+    """Check if an annotation is or contains the target type (handles Optional/Union/Annotated)."""
+    if annotation is target:
+        return True
+    origin = get_origin(annotation)
+    if origin is None:
+        return False
+    return any(_contains_type(arg, target) for arg in get_args(annotation))
+
+
 @dataclass
 class ContentItem:
     """Represents a piece of content to be embedded."""
@@ -188,45 +199,51 @@ class BlockHandler(ContentHandler):
             try:
                 block_instance = block_cls()
 
-                # Skip disabled blocks - they shouldn't be indexed
                 if block_instance.disabled:
                     continue
 
                 # Build searchable text from block metadata
                 parts = []
-                if hasattr(block_instance, "name") and block_instance.name:
+                if block_instance.name:
                     parts.append(block_instance.name)
-                if (
-                    hasattr(block_instance, "description")
-                    and block_instance.description
-                ):
+                if block_instance.description:
                     parts.append(block_instance.description)
-                if hasattr(block_instance, "categories") and block_instance.categories:
-                    # Convert BlockCategory enum to strings
+                if block_instance.categories:
                     parts.append(
                         " ".join(str(cat.value) for cat in block_instance.categories)
                     )
 
-                # Add input/output schema info
-                if hasattr(block_instance, "input_schema"):
-                    schema = block_instance.input_schema
-                    if hasattr(schema, "model_json_schema"):
-                        schema_dict = schema.model_json_schema()
-                        if "properties" in schema_dict:
-                            for prop_name, prop_info in schema_dict[
-                                "properties"
-                            ].items():
-                                if "description" in prop_info:
-                                    parts.append(
-                                        f"{prop_name}: {prop_info['description']}"
-                                    )
+                # Add input schema field descriptions
+                block_input_fields = block_instance.input_schema.model_fields
+                parts += [
+                    f"{field_name}: {field_info.description}"
+                    for field_name, field_info in block_input_fields.items()
+                    if field_info.description
+                ]
 
                 searchable_text = " ".join(parts)
 
-                # Convert categories set of enums to list of strings for JSON serialization
-                categories = getattr(block_instance, "categories", set())
                 categories_list = (
-                    [cat.value for cat in categories] if categories else []
+                    [cat.value for cat in block_instance.categories]
+                    if block_instance.categories
+                    else []
+                )
+
+                # Extract provider names from credentials fields
+                credentials_info = (
+                    block_instance.input_schema.get_credentials_fields_info()
+                )
+                is_integration = len(credentials_info) > 0
+                provider_names = [
+                    provider.value.lower()
+                    for info in credentials_info.values()
+                    for provider in info.provider
+                ]
+
+                # Check if block has LlmModel field in input schema
+                has_llm_model_field = any(
+                    _contains_type(field.annotation, LlmModel)
+                    for field in block_instance.input_schema.model_fields.values()
                 )
 
                 items.append(
@@ -235,8 +252,11 @@ class BlockHandler(ContentHandler):
                         content_type=ContentType.BLOCK,
                         searchable_text=searchable_text,
                         metadata={
-                            "name": getattr(block_instance, "name", ""),
+                            "name": block_instance.name,
                             "categories": categories_list,
+                            "providers": provider_names,
+                            "has_llm_model_field": has_llm_model_field,
+                            "is_integration": is_integration,
                         },
                         user_id=None,  # Blocks are public
                     )

autogpt_platform/backend/backend/api/features/store/content_handlers_test.py

@@ -82,9 +82,10 @@ async def test_block_handler_get_missing_items(mocker):
     mock_block_instance.description = "Performs calculations"
     mock_block_instance.categories = [MagicMock(value="MATH")]
     mock_block_instance.disabled = False
-    mock_block_instance.input_schema.model_json_schema.return_value = {
-        "properties": {"expression": {"description": "Math expression to evaluate"}}
-    }
+    mock_field = MagicMock()
+    mock_field.description = "Math expression to evaluate"
+    mock_block_instance.input_schema.model_fields = {"expression": mock_field}
+    mock_block_instance.input_schema.get_credentials_fields_info.return_value = {}
     mock_block_class.return_value = mock_block_instance
 
     mock_blocks = {"block-uuid-1": mock_block_class}
@@ -309,19 +310,19 @@ async def test_content_handlers_registry():
 
 
 @pytest.mark.asyncio(loop_scope="session")
-async def test_block_handler_handles_missing_attributes():
-    """Test BlockHandler gracefully handles blocks with missing attributes."""
+async def test_block_handler_handles_empty_attributes():
+    """Test BlockHandler handles blocks with empty/falsy attribute values."""
     handler = BlockHandler()
 
-    # Mock block with minimal attributes
+    # Mock block with empty values (all attributes exist but are falsy)
     mock_block_class = MagicMock()
     mock_block_instance = MagicMock()
     mock_block_instance.name = "Minimal Block"
     mock_block_instance.disabled = False
-    # No description, categories, or schema
-    del mock_block_instance.description
-    del mock_block_instance.categories
-    del mock_block_instance.input_schema
+    mock_block_instance.description = ""
+    mock_block_instance.categories = set()
+    mock_block_instance.input_schema.model_fields = {}
+    mock_block_instance.input_schema.get_credentials_fields_info.return_value = {}
     mock_block_class.return_value = mock_block_instance
 
     mock_blocks = {"block-minimal": mock_block_class}
@@ -352,6 +353,8 @@ async def test_block_handler_skips_failed_blocks():
     good_instance.description = "Works fine"
     good_instance.categories = []
     good_instance.disabled = False
+    good_instance.input_schema.model_fields = {}
+    good_instance.input_schema.get_credentials_fields_info.return_value = {}
     good_block.return_value = good_instance
 
     bad_block = MagicMock()

autogpt_platform/backend/backend/api/features/store/db_test.py

@@ -26,7 +26,7 @@ async def test_get_store_agents(mocker):
     mock_agents = [
         prisma.models.StoreAgent(
             listing_id="test-id",
-            storeListingVersionId="version123",
+            listing_version_id="version123",
             slug="test-agent",
             agent_name="Test Agent",
             agent_video=None,
@@ -40,11 +40,11 @@ async def test_get_store_agents(mocker):
             runs=10,
             rating=4.5,
             versions=["1.0"],
-            agentGraphVersions=["1"],
-            agentGraphId="test-graph-id",
+            graph_id="test-graph-id",
+            graph_versions=["1"],
             updated_at=datetime.now(),
             is_available=False,
-            useForOnboarding=False,
+            use_for_onboarding=False,
         )
     ]
 
@@ -68,10 +68,10 @@ async def test_get_store_agents(mocker):
 
 @pytest.mark.asyncio(loop_scope="session")
 async def test_get_store_agent_details(mocker):
-    # Mock data
+    # Mock data - StoreAgent view already contains the active version data
     mock_agent = prisma.models.StoreAgent(
         listing_id="test-id",
-        storeListingVersionId="version123",
+        listing_version_id="version123",
         slug="test-agent",
         agent_name="Test Agent",
         agent_video="video.mp4",
@@ -85,102 +85,38 @@ async def test_get_store_agent_details(mocker):
         runs=10,
         rating=4.5,
         versions=["1.0"],
-        agentGraphVersions=["1"],
-        agentGraphId="test-graph-id",
-        updated_at=datetime.now(),
-        is_available=False,
-        useForOnboarding=False,
-    )
-
-    # Mock active version agent (what we want to return for active version)
-    mock_active_agent = prisma.models.StoreAgent(
-        listing_id="test-id",
-        storeListingVersionId="active-version-id",
-        slug="test-agent",
-        agent_name="Test Agent Active",
-        agent_video="active_video.mp4",
-        agent_image=["active_image.jpg"],
-        featured=False,
-        creator_username="creator",
-        creator_avatar="avatar.jpg",
-        sub_heading="Test heading active",
-        description="Test description active",
-        categories=["test"],
-        runs=15,
-        rating=4.8,
-        versions=["1.0", "2.0"],
-        agentGraphVersions=["1", "2"],
-        agentGraphId="test-graph-id-active",
+        graph_id="test-graph-id",
+        graph_versions=["1"],
         updated_at=datetime.now(),
         is_available=True,
-        useForOnboarding=False,
+        use_for_onboarding=False,
     )
 
-    # Create a mock StoreListing result
-    mock_store_listing = mocker.MagicMock()
-    mock_store_listing.activeVersionId = "active-version-id"
-    mock_store_listing.hasApprovedVersion = True
-    mock_store_listing.ActiveVersion = mocker.MagicMock()
-    mock_store_listing.ActiveVersion.recommendedScheduleCron = None
-
-    # Mock StoreAgent prisma call - need to handle multiple calls
+    # Mock StoreAgent prisma call
     mock_store_agent = mocker.patch("prisma.models.StoreAgent.prisma")
-
-    # Set up side_effect to return different results for different calls
-    def mock_find_first_side_effect(*args, **kwargs):
-        where_clause = kwargs.get("where", {})
-        if "storeListingVersionId" in where_clause:
-            # Second call for active version
-            return mock_active_agent
-        else:
-            # First call for initial lookup
-            return mock_agent
-
-    mock_store_agent.return_value.find_first = mocker.AsyncMock(
-        side_effect=mock_find_first_side_effect
-    )
-
-    # Mock Profile prisma call
-    mock_profile = mocker.MagicMock()
-    mock_profile.userId = "user-id-123"
-    mock_profile_db = mocker.patch("prisma.models.Profile.prisma")
-    mock_profile_db.return_value.find_first = mocker.AsyncMock(
-        return_value=mock_profile
-    )
-
-    # Mock StoreListing prisma call
-    mock_store_listing_db = mocker.patch("prisma.models.StoreListing.prisma")
-    mock_store_listing_db.return_value.find_first = mocker.AsyncMock(
-        return_value=mock_store_listing
-    )
+    mock_store_agent.return_value.find_first = mocker.AsyncMock(return_value=mock_agent)
 
     # Call function
     result = await db.get_store_agent_details("creator", "test-agent")
 
-    # Verify results - should use active version data
+    # Verify results - constructed from the StoreAgent view
     assert result.slug == "test-agent"
-    assert result.agent_name == "Test Agent Active"  # From active version
-    assert result.active_version_id == "active-version-id"
+    assert result.agent_name == "Test Agent"
+    assert result.active_version_id == "version123"
     assert result.has_approved_version is True
-    assert (
-        result.store_listing_version_id == "active-version-id"
-    )  # Should be active version ID
+    assert result.store_listing_version_id == "version123"
+    assert result.graph_id == "test-graph-id"
+    assert result.runs == 10
+    assert result.rating == 4.5
 
-    # Verify mocks called correctly - now expecting 2 calls
-    assert mock_store_agent.return_value.find_first.call_count == 2
-
-    # Check the specific calls
-    calls = mock_store_agent.return_value.find_first.call_args_list
-    assert calls[0] == mocker.call(
+    # Verify single StoreAgent lookup
+    mock_store_agent.return_value.find_first.assert_called_once_with(
         where={"creator_username": "creator", "slug": "test-agent"}
     )
-    assert calls[1] == mocker.call(where={"storeListingVersionId": "active-version-id"})
-
-    mock_store_listing_db.return_value.find_first.assert_called_once()
 
 
 @pytest.mark.asyncio(loop_scope="session")
-async def test_get_store_creator_details(mocker):
+async def test_get_store_creator(mocker):
     # Mock data
     mock_creator_data = prisma.models.Creator(
         name="Test Creator",
@@ -202,7 +138,7 @@ async def test_get_store_creator_details(mocker):
     mock_creator.return_value.find_unique.return_value = mock_creator_data
 
     # Call function
-    result = await db.get_store_creator_details("creator")
+    result = await db.get_store_creator("creator")
 
     # Verify results
     assert result.username == "creator"
@@ -218,61 +154,110 @@ async def test_get_store_creator_details(mocker):
 
 @pytest.mark.asyncio(loop_scope="session")
 async def test_create_store_submission(mocker):
-    # Mock data
+    now = datetime.now()
+
+    # Mock agent graph (with no pending submissions) and user with profile
+    mock_profile = prisma.models.Profile(
+        id="profile-id",
+        userId="user-id",
+        name="Test User",
+        username="testuser",
+        description="Test",
+        isFeatured=False,
+        links=[],
+        createdAt=now,
+        updatedAt=now,
+    )
+    mock_user = prisma.models.User(
+        id="user-id",
+        email="test@example.com",
+        createdAt=now,
+        updatedAt=now,
+        Profile=[mock_profile],
+        emailVerified=True,
+        metadata="{}",  # type: ignore[reportArgumentType]
+        integrations="",
+        maxEmailsPerDay=1,
+        notifyOnAgentRun=True,
+        notifyOnZeroBalance=True,
+        notifyOnLowBalance=True,
+        notifyOnBlockExecutionFailed=True,
+        notifyOnContinuousAgentError=True,
+        notifyOnDailySummary=True,
+        notifyOnWeeklySummary=True,
+        notifyOnMonthlySummary=True,
+        notifyOnAgentApproved=True,
+        notifyOnAgentRejected=True,
+        timezone="Europe/Delft",
+    )
     mock_agent = prisma.models.AgentGraph(
         id="agent-id",
         version=1,
         userId="user-id",
-        createdAt=datetime.now(),
+        createdAt=now,
         isActive=True,
+        StoreListingVersions=[],
+        User=mock_user,
     )
 
-    mock_listing = prisma.models.StoreListing(
+    # Mock the created StoreListingVersion (returned by create)
+    mock_store_listing_obj = prisma.models.StoreListing(
         id="listing-id",
-        createdAt=datetime.now(),
-        updatedAt=datetime.now(),
+        createdAt=now,
+        updatedAt=now,
         isDeleted=False,
         hasApprovedVersion=False,
         slug="test-agent",
         agentGraphId="agent-id",
-        agentGraphVersion=1,
         owningUserId="user-id",
-        Versions=[
-            prisma.models.StoreListingVersion(
-                id="version-id",
-                agentGraphId="agent-id",
-                agentGraphVersion=1,
-                name="Test Agent",
-                description="Test description",
-                createdAt=datetime.now(),
-                updatedAt=datetime.now(),
-                subHeading="Test heading",
-                imageUrls=["image.jpg"],
-                categories=["test"],
-                isFeatured=False,
-                isDeleted=False,
-                version=1,
-                storeListingId="listing-id",
-                submissionStatus=prisma.enums.SubmissionStatus.PENDING,
-                isAvailable=True,
-            )
-        ],
         useForOnboarding=False,
     )
+    mock_version = prisma.models.StoreListingVersion(
+        id="version-id",
+        agentGraphId="agent-id",
+        agentGraphVersion=1,
+        name="Test Agent",
+        description="Test description",
+        createdAt=now,
+        updatedAt=now,
+        subHeading="",
+        imageUrls=[],
+        categories=[],
+        isFeatured=False,
+        isDeleted=False,
+        version=1,
+        storeListingId="listing-id",
+        submissionStatus=prisma.enums.SubmissionStatus.PENDING,
+        isAvailable=True,
+        submittedAt=now,
+        StoreListing=mock_store_listing_obj,
+    )
 
     # Mock prisma calls
     mock_agent_graph = mocker.patch("prisma.models.AgentGraph.prisma")
     mock_agent_graph.return_value.find_first = mocker.AsyncMock(return_value=mock_agent)
 
-    mock_store_listing = mocker.patch("prisma.models.StoreListing.prisma")
-    mock_store_listing.return_value.find_first = mocker.AsyncMock(return_value=None)
-    mock_store_listing.return_value.create = mocker.AsyncMock(return_value=mock_listing)
+    # Mock transaction context manager
+    mock_tx = mocker.MagicMock()
+    mocker.patch(
+        "backend.api.features.store.db.transaction",
+        return_value=mocker.AsyncMock(
+            __aenter__=mocker.AsyncMock(return_value=mock_tx),
+            __aexit__=mocker.AsyncMock(return_value=False),
+        ),
+    )
+
+    mock_sl = mocker.patch("prisma.models.StoreListing.prisma")
+    mock_sl.return_value.find_unique = mocker.AsyncMock(return_value=None)
+
+    mock_slv = mocker.patch("prisma.models.StoreListingVersion.prisma")
+    mock_slv.return_value.create = mocker.AsyncMock(return_value=mock_version)
 
     # Call function
     result = await db.create_store_submission(
         user_id="user-id",
-        agent_id="agent-id",
-        agent_version=1,
+        graph_id="agent-id",
+        graph_version=1,
         slug="test-agent",
         name="Test Agent",
         description="Test description",
@@ -281,11 +266,11 @@ async def test_create_store_submission(mocker):
     # Verify results
     assert result.name == "Test Agent"
     assert result.description == "Test description"
-    assert result.store_listing_version_id == "version-id"
+    assert result.listing_version_id == "version-id"
 
     # Verify mocks called correctly
     mock_agent_graph.return_value.find_first.assert_called_once()
-    mock_store_listing.return_value.create.assert_called_once()
+    mock_slv.return_value.create.assert_called_once()
 
 
 @pytest.mark.asyncio(loop_scope="session")
@@ -318,7 +303,6 @@ async def test_update_profile(mocker):
         description="Test description",
         links=["link1"],
         avatar_url="avatar.jpg",
-        is_featured=False,
     )
 
     # Call function
@@ -389,7 +373,7 @@ async def test_get_store_agents_with_search_and_filters_parameterized():
         creators=["creator1'; DROP TABLE Users; --", "creator2"],
         category="AI'; DELETE FROM StoreAgent; --",
         featured=True,
-        sorted_by="rating",
+        sorted_by=db.StoreAgentsSortOptions.RATING,
         page=1,
         page_size=20,
     )

autogpt_platform/backend/backend/api/features/store/exceptions.py

@@ -57,12 +57,6 @@ class StoreError(ValueError):
     pass
 
 
-class AgentNotFoundError(NotFoundError):
-    """Raised when an agent is not found"""
-
-    pass
-
-
 class CreatorNotFoundError(NotFoundError):
     """Raised when a creator is not found"""
 

autogpt_platform/backend/backend/api/features/store/hybrid_search.py

@@ -568,7 +568,7 @@ async def hybrid_search(
             SELECT uce."contentId" as "storeListingVersionId"
             FROM {{schema_prefix}}"UnifiedContentEmbedding" uce
             INNER JOIN {{schema_prefix}}"StoreAgent" sa
-                ON uce."contentId" = sa."storeListingVersionId"
+                ON uce."contentId" = sa.listing_version_id
             WHERE uce."contentType" = 'STORE_AGENT'::{{schema_prefix}}"ContentType"
             AND uce."userId" IS NULL
             AND uce.search @@ plainto_tsquery('english', {query_param})
@@ -582,7 +582,7 @@ async def hybrid_search(
                 SELECT uce."contentId", uce.embedding
                 FROM {{schema_prefix}}"UnifiedContentEmbedding" uce
                 INNER JOIN {{schema_prefix}}"StoreAgent" sa
-                    ON uce."contentId" = sa."storeListingVersionId"
+                    ON uce."contentId" = sa.listing_version_id
                 WHERE uce."contentType" = 'STORE_AGENT'::{{schema_prefix}}"ContentType"
                 AND uce."userId" IS NULL
                 AND {where_clause}
@@ -605,7 +605,7 @@ async def hybrid_search(
                 sa.featured,
                 sa.is_available,
                 sa.updated_at,
-                sa."agentGraphId",
+                sa.graph_id,
                 -- Searchable text for BM25 reranking
                 COALESCE(sa.agent_name, '') || ' ' || COALESCE(sa.sub_heading, '') || ' ' || COALESCE(sa.description, '') as searchable_text,
                 -- Semantic score
@@ -627,9 +627,9 @@ async def hybrid_search(
                 sa.runs as popularity_raw
             FROM candidates c
             INNER JOIN {{schema_prefix}}"StoreAgent" sa
-                ON c."storeListingVersionId" = sa."storeListingVersionId"
+                ON c."storeListingVersionId" = sa.listing_version_id
             INNER JOIN {{schema_prefix}}"UnifiedContentEmbedding" uce
-                ON sa."storeListingVersionId" = uce."contentId"
+                ON sa.listing_version_id = uce."contentId"
                 AND uce."contentType" = 'STORE_AGENT'::{{schema_prefix}}"ContentType"
         ),
         max_vals AS (
@@ -665,7 +665,7 @@ async def hybrid_search(
                 featured,
                 is_available,
                 updated_at,
-                "agentGraphId",
+                graph_id,
                 searchable_text,
                 semantic_score,
                 lexical_score,

autogpt_platform/backend/backend/api/features/store/model.py

@@ -1,11 +1,14 @@
 import datetime
-from typing import List
+from typing import TYPE_CHECKING, List, Self
 
 import prisma.enums
 import pydantic
 
 from backend.util.models import Pagination
 
+if TYPE_CHECKING:
+    import prisma.models
+
 
 class ChangelogEntry(pydantic.BaseModel):
     version: str
@@ -13,9 +16,9 @@ class ChangelogEntry(pydantic.BaseModel):
     date: datetime.datetime
 
 
-class MyAgent(pydantic.BaseModel):
-    agent_id: str
-    agent_version: int
+class MyUnpublishedAgent(pydantic.BaseModel):
+    graph_id: str
+    graph_version: int
     agent_name: str
     agent_image: str | None = None
     description: str
@@ -23,8 +26,8 @@ class MyAgent(pydantic.BaseModel):
     recommended_schedule_cron: str | None = None
 
 
-class MyAgentsResponse(pydantic.BaseModel):
-    agents: list[MyAgent]
+class MyUnpublishedAgentsResponse(pydantic.BaseModel):
+    agents: list[MyUnpublishedAgent]
     pagination: Pagination
 
 
@@ -40,6 +43,21 @@ class StoreAgent(pydantic.BaseModel):
     rating: float
     agent_graph_id: str
 
+    @classmethod
+    def from_db(cls, agent: "prisma.models.StoreAgent") -> "StoreAgent":
+        return cls(
+            slug=agent.slug,
+            agent_name=agent.agent_name,
+            agent_image=agent.agent_image[0] if agent.agent_image else "",
+            creator=agent.creator_username or "Needs Profile",
+            creator_avatar=agent.creator_avatar or "",
+            sub_heading=agent.sub_heading,
+            description=agent.description,
+            runs=agent.runs,
+            rating=agent.rating,
+            agent_graph_id=agent.graph_id,
+        )
+
 
 class StoreAgentsResponse(pydantic.BaseModel):
     agents: list[StoreAgent]
@@ -62,81 +80,192 @@ class StoreAgentDetails(pydantic.BaseModel):
     runs: int
     rating: float
     versions: list[str]
-    agentGraphVersions: list[str]
-    agentGraphId: str
+    graph_id: str
+    graph_versions: list[str]
     last_updated: datetime.datetime
     recommended_schedule_cron: str | None = None
 
-    active_version_id: str | None = None
-    has_approved_version: bool = False
+    active_version_id: str
+    has_approved_version: bool
 
     # Optional changelog data when include_changelog=True
     changelog: list[ChangelogEntry] | None = None
 
-
-class Creator(pydantic.BaseModel):
-    name: str
-    username: str
-    description: str
-    avatar_url: str
-    num_agents: int
-    agent_rating: float
-    agent_runs: int
-    is_featured: bool
-
-
-class CreatorsResponse(pydantic.BaseModel):
-    creators: List[Creator]
-    pagination: Pagination
-
-
-class CreatorDetails(pydantic.BaseModel):
-    name: str
-    username: str
-    description: str
-    links: list[str]
-    avatar_url: str
-    agent_rating: float
-    agent_runs: int
-    top_categories: list[str]
+    @classmethod
+    def from_db(cls, agent: "prisma.models.StoreAgent") -> "StoreAgentDetails":
+        return cls(
+            store_listing_version_id=agent.listing_version_id,
+            slug=agent.slug,
+            agent_name=agent.agent_name,
+            agent_video=agent.agent_video or "",
+            agent_output_demo=agent.agent_output_demo or "",
+            agent_image=agent.agent_image,
+            creator=agent.creator_username or "",
+            creator_avatar=agent.creator_avatar or "",
+            sub_heading=agent.sub_heading,
+            description=agent.description,
+            categories=agent.categories,
+            runs=agent.runs,
+            rating=agent.rating,
+            versions=agent.versions,
+            graph_id=agent.graph_id,
+            graph_versions=agent.graph_versions,
+            last_updated=agent.updated_at,
+            recommended_schedule_cron=agent.recommended_schedule_cron,
+            active_version_id=agent.listing_version_id,
+            has_approved_version=True,  # StoreAgent view only has approved agents
+        )
 
 
 class Profile(pydantic.BaseModel):
-    name: str
+    """Marketplace user profile (only attributes that the user can update)"""
+
     username: str
+    name: str
     description: str
+    avatar_url: str | None
     links: list[str]
-    avatar_url: str
-    is_featured: bool = False
+
+
+class ProfileDetails(Profile):
+    """Marketplace user profile (including read-only fields)"""
+
+    is_featured: bool
+
+    @classmethod
+    def from_db(cls, profile: "prisma.models.Profile") -> "ProfileDetails":
+        return cls(
+            name=profile.name,
+            username=profile.username,
+            avatar_url=profile.avatarUrl,
+            description=profile.description,
+            links=profile.links,
+            is_featured=profile.isFeatured,
+        )
+
+
+class CreatorDetails(ProfileDetails):
+    """Marketplace creator profile details, including aggregated stats"""
+
+    num_agents: int
+    agent_runs: int
+    agent_rating: float
+    top_categories: list[str]
+
+    @classmethod
+    def from_db(cls, creator: "prisma.models.Creator") -> "CreatorDetails":  # type: ignore[override]
+        return cls(
+            name=creator.name,
+            username=creator.username,
+            avatar_url=creator.avatar_url,
+            description=creator.description,
+            links=creator.links,
+            is_featured=creator.is_featured,
+            num_agents=creator.num_agents,
+            agent_runs=creator.agent_runs,
+            agent_rating=creator.agent_rating,
+            top_categories=creator.top_categories,
+        )
+
+
+class CreatorsResponse(pydantic.BaseModel):
+    creators: List[CreatorDetails]
+    pagination: Pagination
 
 
 class StoreSubmission(pydantic.BaseModel):
+    # From StoreListing:
     listing_id: str
-    agent_id: str
-    agent_version: int
+    user_id: str
+    slug: str
+
+    # From StoreListingVersion:
+    listing_version_id: str
+    listing_version: int
+    graph_id: str
+    graph_version: int
     name: str
     sub_heading: str
-    slug: str
     description: str
-    instructions: str | None = None
+    instructions: str | None
+    categories: list[str]
     image_urls: list[str]
-    date_submitted: datetime.datetime
-    status: prisma.enums.SubmissionStatus
-    runs: int
-    rating: float
-    store_listing_version_id: str | None = None
-    version: int | None = None  # Actual version number from the database
+    video_url: str | None
+    agent_output_demo_url: str | None
 
+    submitted_at: datetime.datetime | None
+    changes_summary: str | None
+    status: prisma.enums.SubmissionStatus
+    reviewed_at: datetime.datetime | None = None
     reviewer_id: str | None = None
     review_comments: str | None = None  # External comments visible to creator
-    internal_comments: str | None = None  # Private notes for admin use only
-    reviewed_at: datetime.datetime | None = None
-    changes_summary: str | None = None
 
-    # Additional fields for editing
-    video_url: str | None = None
-    agent_output_demo_url: str | None = None
-    categories: list[str] = []
+    # Aggregated from AgentGraphExecutions and StoreListingReviews:
+    run_count: int = 0
+    review_count: int = 0
+    review_avg_rating: float = 0.0
+
+    @classmethod
+    def from_db(cls, _sub: "prisma.models.StoreSubmission") -> Self:
+        """Construct from the StoreSubmission Prisma view."""
+        return cls(
+            listing_id=_sub.listing_id,
+            user_id=_sub.user_id,
+            slug=_sub.slug,
+            listing_version_id=_sub.listing_version_id,
+            listing_version=_sub.listing_version,
+            graph_id=_sub.graph_id,
+            graph_version=_sub.graph_version,
+            name=_sub.name,
+            sub_heading=_sub.sub_heading,
+            description=_sub.description,
+            instructions=_sub.instructions,
+            categories=_sub.categories,
+            image_urls=_sub.image_urls,
+            video_url=_sub.video_url,
+            agent_output_demo_url=_sub.agent_output_demo_url,
+            submitted_at=_sub.submitted_at,
+            changes_summary=_sub.changes_summary,
+            status=_sub.status,
+            reviewed_at=_sub.reviewed_at,
+            reviewer_id=_sub.reviewer_id,
+            review_comments=_sub.review_comments,
+            run_count=_sub.run_count,
+            review_count=_sub.review_count,
+            review_avg_rating=_sub.review_avg_rating,
+        )
+
+    @classmethod
+    def from_listing_version(cls, _lv: "prisma.models.StoreListingVersion") -> Self:
+        """
+        Construct from the StoreListingVersion Prisma model (with StoreListing included)
+        """
+        if not (_l := _lv.StoreListing):
+            raise ValueError("StoreListingVersion must have included StoreListing")
+
+        return cls(
+            listing_id=_l.id,
+            user_id=_l.owningUserId,
+            slug=_l.slug,
+            listing_version_id=_lv.id,
+            listing_version=_lv.version,
+            graph_id=_lv.agentGraphId,
+            graph_version=_lv.agentGraphVersion,
+            name=_lv.name,
+            sub_heading=_lv.subHeading,
+            description=_lv.description,
+            instructions=_lv.instructions,
+            categories=_lv.categories,
+            image_urls=_lv.imageUrls,
+            video_url=_lv.videoUrl,
+            agent_output_demo_url=_lv.agentOutputDemoUrl,
+            submitted_at=_lv.submittedAt,
+            changes_summary=_lv.changesSummary,
+            status=_lv.submissionStatus,
+            reviewed_at=_lv.reviewedAt,
+            reviewer_id=_lv.reviewerId,
+            review_comments=_lv.reviewComments,
+        )
 
 
 class StoreSubmissionsResponse(pydantic.BaseModel):
@@ -144,33 +273,12 @@ class StoreSubmissionsResponse(pydantic.BaseModel):
     pagination: Pagination
 
 
-class StoreListingWithVersions(pydantic.BaseModel):
-    """A store listing with its version history"""
-
-    listing_id: str
-    slug: str
-    agent_id: str
-    agent_version: int
-    active_version_id: str | None = None
-    has_approved_version: bool = False
-    creator_email: str | None = None
-    latest_version: StoreSubmission | None = None
-    versions: list[StoreSubmission] = []
-
-
-class StoreListingsWithVersionsResponse(pydantic.BaseModel):
-    """Response model for listings with version history"""
-
-    listings: list[StoreListingWithVersions]
-    pagination: Pagination
-
-
 class StoreSubmissionRequest(pydantic.BaseModel):
-    agent_id: str = pydantic.Field(
-        ..., min_length=1, description="Agent ID cannot be empty"
+    graph_id: str = pydantic.Field(
+        ..., min_length=1, description="Graph ID cannot be empty"
     )
-    agent_version: int = pydantic.Field(
-        ..., gt=0, description="Agent version must be greater than 0"
+    graph_version: int = pydantic.Field(
+        ..., gt=0, description="Graph version must be greater than 0"
     )
     slug: str
     name: str
@@ -198,12 +306,42 @@ class StoreSubmissionEditRequest(pydantic.BaseModel):
     recommended_schedule_cron: str | None = None
 
 
-class ProfileDetails(pydantic.BaseModel):
-    name: str
-    username: str
-    description: str
-    links: list[str]
-    avatar_url: str | None = None
+class StoreSubmissionAdminView(StoreSubmission):
+    internal_comments: str | None  # Private admin notes
+
+    @classmethod
+    def from_db(cls, _sub: "prisma.models.StoreSubmission") -> Self:
+        return cls(
+            **StoreSubmission.from_db(_sub).model_dump(),
+            internal_comments=_sub.internal_comments,
+        )
+
+    @classmethod
+    def from_listing_version(cls, _lv: "prisma.models.StoreListingVersion") -> Self:
+        return cls(
+            **StoreSubmission.from_listing_version(_lv).model_dump(),
+            internal_comments=_lv.internalComments,
+        )
+
+
+class StoreListingWithVersionsAdminView(pydantic.BaseModel):
+    """A store listing with its version history"""
+
+    listing_id: str
+    graph_id: str
+    slug: str
+    active_listing_version_id: str | None = None
+    has_approved_version: bool = False
+    creator_email: str | None = None
+    latest_version: StoreSubmissionAdminView | None = None
+    versions: list[StoreSubmissionAdminView] = []
+
+
+class StoreListingsWithVersionsAdminViewResponse(pydantic.BaseModel):
+    """Response model for listings with version history"""
+
+    listings: list[StoreListingWithVersionsAdminView]
+    pagination: Pagination
 
 
 class StoreReview(pydantic.BaseModel):

autogpt_platform/backend/backend/api/features/store/model_test.py

@@ -1,203 +0,0 @@
-import datetime
-
-import prisma.enums
-
-from . import model as store_model
-
-
-def test_pagination():
-    pagination = store_model.Pagination(
-        total_items=100, total_pages=5, current_page=2, page_size=20
-    )
-    assert pagination.total_items == 100
-    assert pagination.total_pages == 5
-    assert pagination.current_page == 2
-    assert pagination.page_size == 20
-
-
-def test_store_agent():
-    agent = store_model.StoreAgent(
-        slug="test-agent",
-        agent_name="Test Agent",
-        agent_image="test.jpg",
-        creator="creator1",
-        creator_avatar="avatar.jpg",
-        sub_heading="Test subheading",
-        description="Test description",
-        runs=50,
-        rating=4.5,
-        agent_graph_id="test-graph-id",
-    )
-    assert agent.slug == "test-agent"
-    assert agent.agent_name == "Test Agent"
-    assert agent.runs == 50
-    assert agent.rating == 4.5
-    assert agent.agent_graph_id == "test-graph-id"
-
-
-def test_store_agents_response():
-    response = store_model.StoreAgentsResponse(
-        agents=[
-            store_model.StoreAgent(
-                slug="test-agent",
-                agent_name="Test Agent",
-                agent_image="test.jpg",
-                creator="creator1",
-                creator_avatar="avatar.jpg",
-                sub_heading="Test subheading",
-                description="Test description",
-                runs=50,
-                rating=4.5,
-                agent_graph_id="test-graph-id",
-            )
-        ],
-        pagination=store_model.Pagination(
-            total_items=1, total_pages=1, current_page=1, page_size=20
-        ),
-    )
-    assert len(response.agents) == 1
-    assert response.pagination.total_items == 1
-
-
-def test_store_agent_details():
-    details = store_model.StoreAgentDetails(
-        store_listing_version_id="version123",
-        slug="test-agent",
-        agent_name="Test Agent",
-        agent_video="video.mp4",
-        agent_output_demo="demo.mp4",
-        agent_image=["image1.jpg", "image2.jpg"],
-        creator="creator1",
-        creator_avatar="avatar.jpg",
-        sub_heading="Test subheading",
-        description="Test description",
-        categories=["cat1", "cat2"],
-        runs=50,
-        rating=4.5,
-        versions=["1.0", "2.0"],
-        agentGraphVersions=["1", "2"],
-        agentGraphId="test-graph-id",
-        last_updated=datetime.datetime.now(),
-    )
-    assert details.slug == "test-agent"
-    assert len(details.agent_image) == 2
-    assert len(details.categories) == 2
-    assert len(details.versions) == 2
-
-
-def test_creator():
-    creator = store_model.Creator(
-        agent_rating=4.8,
-        agent_runs=1000,
-        name="Test Creator",
-        username="creator1",
-        description="Test description",
-        avatar_url="avatar.jpg",
-        num_agents=5,
-        is_featured=False,
-    )
-    assert creator.name == "Test Creator"
-    assert creator.num_agents == 5
-
-
-def test_creators_response():
-    response = store_model.CreatorsResponse(
-        creators=[
-            store_model.Creator(
-                agent_rating=4.8,
-                agent_runs=1000,
-                name="Test Creator",
-                username="creator1",
-                description="Test description",
-                avatar_url="avatar.jpg",
-                num_agents=5,
-                is_featured=False,
-            )
-        ],
-        pagination=store_model.Pagination(
-            total_items=1, total_pages=1, current_page=1, page_size=20
-        ),
-    )
-    assert len(response.creators) == 1
-    assert response.pagination.total_items == 1
-
-
-def test_creator_details():
-    details = store_model.CreatorDetails(
-        name="Test Creator",
-        username="creator1",
-        description="Test description",
-        links=["link1.com", "link2.com"],
-        avatar_url="avatar.jpg",
-        agent_rating=4.8,
-        agent_runs=1000,
-        top_categories=["cat1", "cat2"],
-    )
-    assert details.name == "Test Creator"
-    assert len(details.links) == 2
-    assert details.agent_rating == 4.8
-    assert len(details.top_categories) == 2
-
-
-def test_store_submission():
-    submission = store_model.StoreSubmission(
-        listing_id="listing123",
-        agent_id="agent123",
-        agent_version=1,
-        sub_heading="Test subheading",
-        name="Test Agent",
-        slug="test-agent",
-        description="Test description",
-        image_urls=["image1.jpg", "image2.jpg"],
-        date_submitted=datetime.datetime(2023, 1, 1),
-        status=prisma.enums.SubmissionStatus.PENDING,
-        runs=50,
-        rating=4.5,
-    )
-    assert submission.name == "Test Agent"
-    assert len(submission.image_urls) == 2
-    assert submission.status == prisma.enums.SubmissionStatus.PENDING
-
-
-def test_store_submissions_response():
-    response = store_model.StoreSubmissionsResponse(
-        submissions=[
-            store_model.StoreSubmission(
-                listing_id="listing123",
-                agent_id="agent123",
-                agent_version=1,
-                sub_heading="Test subheading",
-                name="Test Agent",
-                slug="test-agent",
-                description="Test description",
-                image_urls=["image1.jpg"],
-                date_submitted=datetime.datetime(2023, 1, 1),
-                status=prisma.enums.SubmissionStatus.PENDING,
-                runs=50,
-                rating=4.5,
-            )
-        ],
-        pagination=store_model.Pagination(
-            total_items=1, total_pages=1, current_page=1, page_size=20
-        ),
-    )
-    assert len(response.submissions) == 1
-    assert response.pagination.total_items == 1
-
-
-def test_store_submission_request():
-    request = store_model.StoreSubmissionRequest(
-        agent_id="agent123",
-        agent_version=1,
-        slug="test-agent",
-        name="Test Agent",
-        sub_heading="Test subheading",
-        video_url="video.mp4",
-        image_urls=["image1.jpg", "image2.jpg"],
-        description="Test description",
-        categories=["cat1", "cat2"],
-    )
-    assert request.agent_id == "agent123"
-    assert request.agent_version == 1
-    assert len(request.image_urls) == 2
-    assert len(request.categories) == 2

autogpt_platform/backend/backend/api/features/store/routes.py

@@ -1,16 +1,17 @@
 import logging
 import tempfile
-import typing
 import urllib.parse
-from typing import Literal
 
 import autogpt_libs.auth
 import fastapi
 import fastapi.responses
 import prisma.enums
+from fastapi import Query, Security
+from pydantic import BaseModel
 
 import backend.data.graph
 import backend.util.json
+from backend.util.exceptions import NotFoundError
 from backend.util.models import Pagination
 
 from . import cache as store_cache
@@ -34,22 +35,15 @@ router = fastapi.APIRouter()
     "/profile",
     summary="Get user profile",
     tags=["store", "private"],
-    dependencies=[fastapi.Security(autogpt_libs.auth.requires_user)],
-    response_model=store_model.ProfileDetails,
+    dependencies=[Security(autogpt_libs.auth.requires_user)],
 )
 async def get_profile(
-    user_id: str = fastapi.Security(autogpt_libs.auth.get_user_id),
-):
-    """
-    Get the profile details for the authenticated user.
-    Cached for 1 hour per user.
-    """
+    user_id: str = Security(autogpt_libs.auth.get_user_id),
+) -> store_model.ProfileDetails:
+    """Get the profile details for the authenticated user."""
     profile = await store_db.get_user_profile(user_id)
     if profile is None:
-        return fastapi.responses.JSONResponse(
-            status_code=404,
-            content={"detail": "Profile not found"},
-        )
+        raise NotFoundError("User does not have a profile yet")
     return profile
 
 
@@ -57,98 +51,17 @@ async def get_profile(
     "/profile",
     summary="Update user profile",
     tags=["store", "private"],
-    dependencies=[fastapi.Security(autogpt_libs.auth.requires_user)],
-    response_model=store_model.CreatorDetails,
+    dependencies=[Security(autogpt_libs.auth.requires_user)],
 )
 async def update_or_create_profile(
     profile: store_model.Profile,
-    user_id: str = fastapi.Security(autogpt_libs.auth.get_user_id),
-):
-    """
-    Update the store profile for the authenticated user.
-
-    Args:
-        profile (Profile): The updated profile details
-        user_id (str): ID of the authenticated user
-
-    Returns:
-        CreatorDetails: The updated profile
-
-    Raises:
-        HTTPException: If there is an error updating the profile
-    """
+    user_id: str = Security(autogpt_libs.auth.get_user_id),
+) -> store_model.ProfileDetails:
+    """Update the store profile for the authenticated user."""
     updated_profile = await store_db.update_profile(user_id=user_id, profile=profile)
     return updated_profile
 
 
-##############################################
-############### Agent Endpoints ##############
-##############################################
-
-
-@router.get(
-    "/agents",
-    summary="List store agents",
-    tags=["store", "public"],
-    response_model=store_model.StoreAgentsResponse,
-)
-async def get_agents(
-    featured: bool = False,
-    creator: str | None = None,
-    sorted_by: Literal["rating", "runs", "name", "updated_at"] | None = None,
-    search_query: str | None = None,
-    category: str | None = None,
-    page: int = 1,
-    page_size: int = 20,
-):
-    """
-    Get a paginated list of agents from the store with optional filtering and sorting.
-
-    Args:
-        featured (bool, optional): Filter to only show featured agents. Defaults to False.
-        creator (str | None, optional): Filter agents by creator username. Defaults to None.
-        sorted_by (str | None, optional): Sort agents by "runs" or "rating". Defaults to None.
-        search_query (str | None, optional): Search agents by name, subheading and description. Defaults to None.
-        category (str | None, optional): Filter agents by category. Defaults to None.
-        page (int, optional): Page number for pagination. Defaults to 1.
-        page_size (int, optional): Number of agents per page. Defaults to 20.
-
-    Returns:
-        StoreAgentsResponse: Paginated list of agents matching the filters
-
-    Raises:
-        HTTPException: If page or page_size are less than 1
-
-    Used for:
-    - Home Page Featured Agents
-    - Home Page Top Agents
-    - Search Results
-    - Agent Details - Other Agents By Creator
-    - Agent Details - Similar Agents
-    - Creator Details - Agents By Creator
-    """
-    if page < 1:
-        raise fastapi.HTTPException(
-            status_code=422, detail="Page must be greater than 0"
-        )
-
-    if page_size < 1:
-        raise fastapi.HTTPException(
-            status_code=422, detail="Page size must be greater than 0"
-        )
-
-    agents = await store_cache._get_cached_store_agents(
-        featured=featured,
-        creator=creator,
-        sorted_by=sorted_by,
-        search_query=search_query,
-        category=category,
-        page=page,
-        page_size=page_size,
-    )
-    return agents
-
-
 ##############################################
 ############### Search Endpoints #############
 ##############################################
@@ -158,60 +71,30 @@ async def get_agents(
     "/search",
     summary="Unified search across all content types",
     tags=["store", "public"],
-    response_model=store_model.UnifiedSearchResponse,
 )
 async def unified_search(
     query: str,
-    content_types: list[str] | None = fastapi.Query(
+    content_types: list[prisma.enums.ContentType] | None = Query(
         default=None,
-        description="Content types to search: STORE_AGENT, BLOCK, DOCUMENTATION. If not specified, searches all.",
+        description="Content types to search. If not specified, searches all.",
     ),
-    page: int = 1,
-    page_size: int = 20,
-    user_id: str | None = fastapi.Security(
+    page: int = Query(ge=1, default=1),
+    page_size: int = Query(ge=1, default=20),
+    user_id: str | None = Security(
         autogpt_libs.auth.get_optional_user_id, use_cache=False
     ),
-):
+) -> store_model.UnifiedSearchResponse:
     """
-    Search across all content types (store agents, blocks, documentation) using hybrid search.
+    Search across all content types (marketplace agents, blocks, documentation)
+    using hybrid search.
 
     Combines semantic (embedding-based) and lexical (text-based) search for best results.
-
-    Args:
-        query: The search query string
-        content_types: Optional list of content types to filter by (STORE_AGENT, BLOCK, DOCUMENTATION)
-        page: Page number for pagination (default 1)
-        page_size: Number of results per page (default 20)
-        user_id: Optional authenticated user ID (for user-scoped content in future)
-
-    Returns:
-        UnifiedSearchResponse: Paginated list of search results with relevance scores
     """
-    if page < 1:
-        raise fastapi.HTTPException(
-            status_code=422, detail="Page must be greater than 0"
-        )
-
-    if page_size < 1:
-        raise fastapi.HTTPException(
-            status_code=422, detail="Page size must be greater than 0"
-        )
-
-    # Convert string content types to enum
-    content_type_enums: list[prisma.enums.ContentType] | None = None
-    if content_types:
-        try:
-            content_type_enums = [prisma.enums.ContentType(ct) for ct in content_types]
-        except ValueError as e:
-            raise fastapi.HTTPException(
-                status_code=422,
-                detail=f"Invalid content type. Valid values: STORE_AGENT, BLOCK, DOCUMENTATION. Error: {e}",
-            )
 
     # Perform unified hybrid search
     results, total = await store_hybrid_search.unified_hybrid_search(
         query=query,
-        content_types=content_type_enums,
+        content_types=content_types,
         user_id=user_id,
         page=page,
         page_size=page_size,
@@ -245,22 +128,69 @@ async def unified_search(
     )
 
 
+##############################################
+############### Agent Endpoints ##############
+##############################################
+
+
+@router.get(
+    "/agents",
+    summary="List store agents",
+    tags=["store", "public"],
+)
+async def get_agents(
+    featured: bool = Query(
+        default=False, description="Filter to only show featured agents"
+    ),
+    creator: str | None = Query(
+        default=None, description="Filter agents by creator username"
+    ),
+    category: str | None = Query(default=None, description="Filter agents by category"),
+    search_query: str | None = Query(
+        default=None, description="Literal + semantic search on names and descriptions"
+    ),
+    sorted_by: store_db.StoreAgentsSortOptions | None = Query(
+        default=None,
+        description="Property to sort results by. Ignored if search_query is provided.",
+    ),
+    page: int = Query(ge=1, default=1),
+    page_size: int = Query(ge=1, default=20),
+) -> store_model.StoreAgentsResponse:
+    """
+    Get a paginated list of agents from the marketplace,
+    with optional filtering and sorting.
+
+    Used for:
+    - Home Page Featured Agents
+    - Home Page Top Agents
+    - Search Results
+    - Agent Details - Other Agents By Creator
+    - Agent Details - Similar Agents
+    - Creator Details - Agents By Creator
+    """
+    agents = await store_cache._get_cached_store_agents(
+        featured=featured,
+        creator=creator,
+        sorted_by=sorted_by,
+        search_query=search_query,
+        category=category,
+        page=page,
+        page_size=page_size,
+    )
+    return agents
+
+
 @router.get(
     "/agents/{username}/{agent_name}",
     summary="Get specific agent",
     tags=["store", "public"],
-    response_model=store_model.StoreAgentDetails,
 )
-async def get_agent(
+async def get_agent_by_name(
     username: str,
     agent_name: str,
-    include_changelog: bool = fastapi.Query(default=False),
-):
-    """
-    This is only used on the AgentDetails Page.
-
-    It returns the store listing agents details.
-    """
+    include_changelog: bool = Query(default=False),
+) -> store_model.StoreAgentDetails:
+    """Get details of a marketplace agent"""
     username = urllib.parse.unquote(username).lower()
     # URL decode the agent name since it comes from the URL path
     agent_name = urllib.parse.unquote(agent_name).lower()
@@ -270,76 +200,82 @@ async def get_agent(
     return agent
 
 
-@router.get(
-    "/graph/{store_listing_version_id}",
-    summary="Get agent graph",
-    tags=["store"],
-    dependencies=[fastapi.Security(autogpt_libs.auth.requires_user)],
-)
-async def get_graph_meta_by_store_listing_version_id(
-    store_listing_version_id: str,
-) -> backend.data.graph.GraphModelWithoutNodes:
-    """
-    Get Agent Graph from Store Listing Version ID.
-    """
-    graph = await store_db.get_available_graph(store_listing_version_id)
-    return graph
-
-
-@router.get(
-    "/agents/{store_listing_version_id}",
-    summary="Get agent by version",
-    tags=["store"],
-    dependencies=[fastapi.Security(autogpt_libs.auth.requires_user)],
-    response_model=store_model.StoreAgentDetails,
-)
-async def get_store_agent(store_listing_version_id: str):
-    """
-    Get Store Agent Details from Store Listing Version ID.
-    """
-    agent = await store_db.get_store_agent_by_version_id(store_listing_version_id)
-
-    return agent
-
-
 @router.post(
     "/agents/{username}/{agent_name}/review",
     summary="Create agent review",
     tags=["store"],
-    dependencies=[fastapi.Security(autogpt_libs.auth.requires_user)],
-    response_model=store_model.StoreReview,
+    dependencies=[Security(autogpt_libs.auth.requires_user)],
 )
-async def create_review(
+async def post_user_review_for_agent(
     username: str,
     agent_name: str,
     review: store_model.StoreReviewCreate,
-    user_id: str = fastapi.Security(autogpt_libs.auth.get_user_id),
-):
-    """
-    Create a review for a store agent.
-
-    Args:
-        username: Creator's username
-        agent_name: Name/slug of the agent
-        review: Review details including score and optional comments
-        user_id: ID of authenticated user creating the review
-
-    Returns:
-        The created review
-    """
+    user_id: str = Security(autogpt_libs.auth.get_user_id),
+) -> store_model.StoreReview:
+    """Post a user review on a marketplace agent listing"""
     username = urllib.parse.unquote(username).lower()
     agent_name = urllib.parse.unquote(agent_name).lower()
-    # Create the review
+
     created_review = await store_db.create_store_review(
         user_id=user_id,
         store_listing_version_id=review.store_listing_version_id,
         score=review.score,
         comments=review.comments,
     )
-
     return created_review
 
 
+@router.get(
+    "/listings/versions/{store_listing_version_id}",
+    summary="Get agent by version",
+    tags=["store"],
+    dependencies=[Security(autogpt_libs.auth.requires_user)],
+)
+async def get_agent_by_listing_version(
+    store_listing_version_id: str,
+) -> store_model.StoreAgentDetails:
+    agent = await store_db.get_store_agent_by_version_id(store_listing_version_id)
+    return agent
+
+
+@router.get(
+    "/listings/versions/{store_listing_version_id}/graph",
+    summary="Get agent graph",
+    tags=["store"],
+    dependencies=[Security(autogpt_libs.auth.requires_user)],
+)
+async def get_graph_meta_by_store_listing_version_id(
+    store_listing_version_id: str,
+) -> backend.data.graph.GraphModelWithoutNodes:
+    """Get outline of graph belonging to a specific marketplace listing version"""
+    graph = await store_db.get_available_graph(store_listing_version_id)
+    return graph
+
+
+@router.get(
+    "/listings/versions/{store_listing_version_id}/graph/download",
+    summary="Download agent file",
+    tags=["store", "public"],
+)
+async def download_agent_file(
+    store_listing_version_id: str,
+) -> fastapi.responses.FileResponse:
+    """Download agent graph file for a specific marketplace listing version"""
+    graph_data = await store_db.get_agent(store_listing_version_id)
+    file_name = f"agent_{graph_data.id}_v{graph_data.version or 'latest'}.json"
+
+    # Sending graph as a stream (similar to marketplace v1)
+    with tempfile.NamedTemporaryFile(
+        mode="w", suffix=".json", delete=False
+    ) as tmp_file:
+        tmp_file.write(backend.util.json.dumps(graph_data))
+        tmp_file.flush()
+
+        return fastapi.responses.FileResponse(
+            tmp_file.name, filename=file_name, media_type="application/json"
+        )
+
+
 ##############################################
 ############# Creator Endpoints #############
 ##############################################
@@ -349,37 +285,19 @@ async def create_review(
     "/creators",
     summary="List store creators",
     tags=["store", "public"],
-    response_model=store_model.CreatorsResponse,
 )
 async def get_creators(
-    featured: bool = False,
-    search_query: str | None = None,
-    sorted_by: Literal["agent_rating", "agent_runs", "num_agents"] | None = None,
-    page: int = 1,
-    page_size: int = 20,
-):
-    """
-    This is needed for:
-    - Home Page Featured Creators
-    - Search Results Page
-
-    ---
-
-    To support this functionality we need:
-    - featured: bool - to limit the list to just featured agents
-    - search_query: str - vector search based on the creators profile description.
-    - sorted_by: [agent_rating, agent_runs] -
-    """
-    if page < 1:
-        raise fastapi.HTTPException(
-            status_code=422, detail="Page must be greater than 0"
-        )
-
-    if page_size < 1:
-        raise fastapi.HTTPException(
-            status_code=422, detail="Page size must be greater than 0"
-        )
-
+    featured: bool = Query(
+        default=False, description="Filter to only show featured creators"
+    ),
+    search_query: str | None = Query(
+        default=None, description="Literal + semantic search on names and descriptions"
+    ),
+    sorted_by: store_db.StoreCreatorsSortOptions | None = None,
+    page: int = Query(ge=1, default=1),
+    page_size: int = Query(ge=1, default=20),
+) -> store_model.CreatorsResponse:
+    """List or search marketplace creators"""
     creators = await store_cache._get_cached_store_creators(
         featured=featured,
         search_query=search_query,
@@ -391,18 +309,12 @@ async def get_creators(
 
 
 @router.get(
-    "/creator/{username}",
+    "/creators/{username}",
     summary="Get creator details",
     tags=["store", "public"],
-    response_model=store_model.CreatorDetails,
 )
-async def get_creator(
-    username: str,
-):
-    """
-    Get the details of a creator.
-    - Creator Details Page
-    """
+async def get_creator(username: str) -> store_model.CreatorDetails:
+    """Get details on a marketplace creator"""
     username = urllib.parse.unquote(username).lower()
     creator = await store_cache._get_cached_creator_details(username=username)
     return creator
@@ -414,20 +326,17 @@ async def get_creator(
 
 
 @router.get(
-    "/myagents",
+    "/my-unpublished-agents",
     summary="Get my agents",
     tags=["store", "private"],
-    dependencies=[fastapi.Security(autogpt_libs.auth.requires_user)],
-    response_model=store_model.MyAgentsResponse,
+    dependencies=[Security(autogpt_libs.auth.requires_user)],
 )
-async def get_my_agents(
-    user_id: str = fastapi.Security(autogpt_libs.auth.get_user_id),
-    page: typing.Annotated[int, fastapi.Query(ge=1)] = 1,
-    page_size: typing.Annotated[int, fastapi.Query(ge=1)] = 20,
-):
-    """
-    Get user's own agents.
-    """
+async def get_my_unpublished_agents(
+    user_id: str = Security(autogpt_libs.auth.get_user_id),
+    page: int = Query(ge=1, default=1),
+    page_size: int = Query(ge=1, default=20),
+) -> store_model.MyUnpublishedAgentsResponse:
+    """List the authenticated user's unpublished agents"""
     agents = await store_db.get_my_agents(user_id, page=page, page_size=page_size)
     return agents
 
@@ -436,28 +345,17 @@ async def get_my_agents(
     "/submissions/{submission_id}",
     summary="Delete store submission",
     tags=["store", "private"],
-    dependencies=[fastapi.Security(autogpt_libs.auth.requires_user)],
-    response_model=bool,
+    dependencies=[Security(autogpt_libs.auth.requires_user)],
 )
 async def delete_submission(
     submission_id: str,
-    user_id: str = fastapi.Security(autogpt_libs.auth.get_user_id),
-):
-    """
-    Delete a store listing submission.
-
-    Args:
-        user_id (str): ID of the authenticated user
-        submission_id (str): ID of the submission to be deleted
-
-    Returns:
-        bool: True if the submission was successfully deleted, False otherwise
-    """
+    user_id: str = Security(autogpt_libs.auth.get_user_id),
+) -> bool:
+    """Delete a marketplace listing submission"""
     result = await store_db.delete_store_submission(
         user_id=user_id,
         submission_id=submission_id,
     )
-
     return result
 
 
@@ -465,37 +363,14 @@ async def delete_submission(
     "/submissions",
     summary="List my submissions",
     tags=["store", "private"],
-    dependencies=[fastapi.Security(autogpt_libs.auth.requires_user)],
-    response_model=store_model.StoreSubmissionsResponse,
+    dependencies=[Security(autogpt_libs.auth.requires_user)],
 )
 async def get_submissions(
-    user_id: str = fastapi.Security(autogpt_libs.auth.get_user_id),
-    page: int = 1,
-    page_size: int = 20,
-):
-    """
-    Get a paginated list of store submissions for the authenticated user.
-
-    Args:
-        user_id (str): ID of the authenticated user
-        page (int, optional): Page number for pagination. Defaults to 1.
-        page_size (int, optional): Number of submissions per page. Defaults to 20.
-
-    Returns:
-        StoreListingsResponse: Paginated list of store submissions
-
-    Raises:
-        HTTPException: If page or page_size are less than 1
-    """
-    if page < 1:
-        raise fastapi.HTTPException(
-            status_code=422, detail="Page must be greater than 0"
-        )
-
-    if page_size < 1:
-        raise fastapi.HTTPException(
-            status_code=422, detail="Page size must be greater than 0"
-        )
+    user_id: str = Security(autogpt_libs.auth.get_user_id),
+    page: int = Query(ge=1, default=1),
+    page_size: int = Query(ge=1, default=20),
+) -> store_model.StoreSubmissionsResponse:
+    """List the authenticated user's marketplace listing submissions"""
     listings = await store_db.get_store_submissions(
         user_id=user_id,
         page=page,
@@ -508,30 +383,17 @@ async def get_submissions(
     "/submissions",
     summary="Create store submission",
     tags=["store", "private"],
-    dependencies=[fastapi.Security(autogpt_libs.auth.requires_user)],
-    response_model=store_model.StoreSubmission,
+    dependencies=[Security(autogpt_libs.auth.requires_user)],
 )
 async def create_submission(
     submission_request: store_model.StoreSubmissionRequest,
-    user_id: str = fastapi.Security(autogpt_libs.auth.get_user_id),
-):
-    """
-    Create a new store listing submission.
-
-    Args:
-        submission_request (StoreSubmissionRequest): The submission details
-        user_id (str): ID of the authenticated user submitting the listing
-
-    Returns:
-        StoreSubmission: The created store submission
-
-    Raises:
-        HTTPException: If there is an error creating the submission
-    """
+    user_id: str = Security(autogpt_libs.auth.get_user_id),
+) -> store_model.StoreSubmission:
+    """Submit a new marketplace listing for review"""
     result = await store_db.create_store_submission(
         user_id=user_id,
-        agent_id=submission_request.agent_id,
-        agent_version=submission_request.agent_version,
+        graph_id=submission_request.graph_id,
+        graph_version=submission_request.graph_version,
         slug=submission_request.slug,
         name=submission_request.name,
         video_url=submission_request.video_url,
@@ -544,7 +406,6 @@ async def create_submission(
         changes_summary=submission_request.changes_summary or "Initial Submission",
         recommended_schedule_cron=submission_request.recommended_schedule_cron,
     )
-
     return result
 
 
@@ -552,28 +413,14 @@ async def create_submission(
     "/submissions/{store_listing_version_id}",
     summary="Edit store submission",
     tags=["store", "private"],
-    dependencies=[fastapi.Security(autogpt_libs.auth.requires_user)],
-    response_model=store_model.StoreSubmission,
+    dependencies=[Security(autogpt_libs.auth.requires_user)],
 )
 async def edit_submission(
     store_listing_version_id: str,
     submission_request: store_model.StoreSubmissionEditRequest,
-    user_id: str = fastapi.Security(autogpt_libs.auth.get_user_id),
-):
-    """
-    Edit an existing store listing submission.
-
-    Args:
-        store_listing_version_id (str): ID of the store listing version to edit
-        submission_request (StoreSubmissionRequest): The updated submission details
-        user_id (str): ID of the authenticated user editing the listing
-
-    Returns:
-        StoreSubmission: The updated store submission
-
-    Raises:
-        HTTPException: If there is an error editing the submission
-    """
+    user_id: str = Security(autogpt_libs.auth.get_user_id),
+) -> store_model.StoreSubmission:
+    """Update a pending marketplace listing submission"""
     result = await store_db.edit_store_submission(
         user_id=user_id,
         store_listing_version_id=store_listing_version_id,
@@ -588,7 +435,6 @@ async def edit_submission(
         changes_summary=submission_request.changes_summary,
         recommended_schedule_cron=submission_request.recommended_schedule_cron,
     )
-
     return result
 
 
@@ -596,115 +442,61 @@ async def edit_submission(
     "/submissions/media",
     summary="Upload submission media",
     tags=["store", "private"],
-    dependencies=[fastapi.Security(autogpt_libs.auth.requires_user)],
+    dependencies=[Security(autogpt_libs.auth.requires_user)],
 )
 async def upload_submission_media(
     file: fastapi.UploadFile,
-    user_id: str = fastapi.Security(autogpt_libs.auth.get_user_id),
-):
-    """
-    Upload media (images/videos) for a store listing submission.
-
-    Args:
-        file (UploadFile): The media file to upload
-        user_id (str): ID of the authenticated user uploading the media
-
-    Returns:
-        str: URL of the uploaded media file
-
-    Raises:
-        HTTPException: If there is an error uploading the media
-    """
+    user_id: str = Security(autogpt_libs.auth.get_user_id),
+) -> str:
+    """Upload media for a marketplace listing submission"""
     media_url = await store_media.upload_media(user_id=user_id, file=file)
     return media_url
 
 
+class ImageURLResponse(BaseModel):
+    image_url: str
+
+
 @router.post(
     "/submissions/generate_image",
     summary="Generate submission image",
     tags=["store", "private"],
-    dependencies=[fastapi.Security(autogpt_libs.auth.requires_user)],
+    dependencies=[Security(autogpt_libs.auth.requires_user)],
 )
 async def generate_image(
-    agent_id: str,
-    user_id: str = fastapi.Security(autogpt_libs.auth.get_user_id),
-) -> fastapi.responses.Response:
+    graph_id: str,
+    user_id: str = Security(autogpt_libs.auth.get_user_id),
+) -> ImageURLResponse:
     """
-    Generate an image for a store listing submission.
-
-    Args:
-        agent_id (str): ID of the agent to generate an image for
-        user_id (str): ID of the authenticated user
-
-    Returns:
-        JSONResponse: JSON containing the URL of the generated image
+    Generate an image for a marketplace listing submission based on the properties
+    of a given graph.
     """
-    agent = await backend.data.graph.get_graph(
-        graph_id=agent_id, version=None, user_id=user_id
+    graph = await backend.data.graph.get_graph(
+        graph_id=graph_id, version=None, user_id=user_id
     )
 
-    if not agent:
-        raise fastapi.HTTPException(
-            status_code=404, detail=f"Agent with ID {agent_id} not found"
-        )
+    if not graph:
+        raise NotFoundError(f"Agent graph #{graph_id} not found")
     # Use .jpeg here since we are generating JPEG images
-    filename = f"agent_{agent_id}.jpeg"
+    filename = f"agent_{graph_id}.jpeg"
 
     existing_url = await store_media.check_media_exists(user_id, filename)
     if existing_url:
-        logger.info(f"Using existing image for agent {agent_id}")
-        return fastapi.responses.JSONResponse(content={"image_url": existing_url})
+        logger.info(f"Using existing image for agent graph {graph_id}")
+        return ImageURLResponse(image_url=existing_url)
     # Generate agent image as JPEG
-    image = await store_image_gen.generate_agent_image(agent=agent)
+    image = await store_image_gen.generate_agent_image(agent=graph)
 
     # Create UploadFile with the correct filename and content_type
     image_file = fastapi.UploadFile(
         file=image,
         filename=filename,
     )
-
     image_url = await store_media.upload_media(
         user_id=user_id, file=image_file, use_file_name=True
     )
 
-    return fastapi.responses.JSONResponse(content={"image_url": image_url})
-
-
-@router.get(
-    "/download/agents/{store_listing_version_id}",
-    summary="Download agent file",
-    tags=["store", "public"],
-)
-async def download_agent_file(
-    store_listing_version_id: str = fastapi.Path(
-        ..., description="The ID of the agent to download"
-    ),
-) -> fastapi.responses.FileResponse:
-    """
-    Download the agent file by streaming its content.
-
-    Args:
-        store_listing_version_id (str): The ID of the agent to download
-
-    Returns:
-        StreamingResponse: A streaming response containing the agent's graph data.
-
-    Raises:
-        HTTPException: If the agent is not found or an unexpected error occurs.
-    """
-    graph_data = await store_db.get_agent(store_listing_version_id)
-    file_name = f"agent_{graph_data.id}_v{graph_data.version or 'latest'}.json"
-
-    # Sending graph as a stream (similar to marketplace v1)
-    with tempfile.NamedTemporaryFile(
-        mode="w", suffix=".json", delete=False
-    ) as tmp_file:
-        tmp_file.write(backend.util.json.dumps(graph_data))
-        tmp_file.flush()
-
-        return fastapi.responses.FileResponse(
-            tmp_file.name, filename=file_name, media_type="application/json"
-        )
+    return ImageURLResponse(image_url=image_url)
 
 
 ##############################################

autogpt_platform/backend/backend/api/features/store/routes_test.py

@@ -8,6 +8,8 @@ import pytest
 import pytest_mock
 from pytest_snapshot.plugin import Snapshot
 
+from backend.api.features.store.db import StoreAgentsSortOptions
+
 from . import model as store_model
 from . import routes as store_routes
 
@@ -196,7 +198,7 @@ def test_get_agents_sorted(
     mock_db_call.assert_called_once_with(
         featured=False,
         creators=None,
-        sorted_by="runs",
+        sorted_by=StoreAgentsSortOptions.RUNS,
         search_query=None,
         category=None,
         page=1,
@@ -380,9 +382,11 @@ def test_get_agent_details(
         runs=100,
         rating=4.5,
         versions=["1.0.0", "1.1.0"],
-        agentGraphVersions=["1", "2"],
-        agentGraphId="test-graph-id",
+        graph_versions=["1", "2"],
+        graph_id="test-graph-id",
         last_updated=FIXED_NOW,
+        active_version_id="test-version-id",
+        has_approved_version=True,
     )
     mock_db_call = mocker.patch("backend.api.features.store.db.get_store_agent_details")
     mock_db_call.return_value = mocked_value
@@ -435,15 +439,17 @@ def test_get_creators_pagination(
 ) -> None:
     mocked_value = store_model.CreatorsResponse(
         creators=[
-            store_model.Creator(
+            store_model.CreatorDetails(
                 name=f"Creator {i}",
                 username=f"creator{i}",
-                description=f"Creator {i} description",
                 avatar_url=f"avatar{i}.jpg",
-                num_agents=1,
-                agent_rating=4.5,
-                agent_runs=100,
+                description=f"Creator {i} description",
+                links=[f"user{i}.link.com"],
                 is_featured=False,
+                num_agents=1,
+                agent_runs=100,
+                agent_rating=4.5,
+                top_categories=["cat1", "cat2", "cat3"],
             )
             for i in range(5)
         ],
@@ -496,19 +502,19 @@ def test_get_creator_details(
     mocked_value = store_model.CreatorDetails(
         name="Test User",
         username="creator1",
+        avatar_url="avatar.jpg",
         description="Test creator description",
         links=["link1.com", "link2.com"],
-        avatar_url="avatar.jpg",
-        agent_rating=4.8,
+        is_featured=True,
+        num_agents=5,
         agent_runs=1000,
+        agent_rating=4.8,
         top_categories=["category1", "category2"],
     )
-    mock_db_call = mocker.patch(
-        "backend.api.features.store.db.get_store_creator_details"
-    )
+    mock_db_call = mocker.patch("backend.api.features.store.db.get_store_creator")
     mock_db_call.return_value = mocked_value
 
-    response = client.get("/creator/creator1")
+    response = client.get("/creators/creator1")
     assert response.status_code == 200
 
     data = store_model.CreatorDetails.model_validate(response.json())
@@ -528,19 +534,26 @@ def test_get_submissions_success(
         submissions=[
             store_model.StoreSubmission(
                 listing_id="test-listing-id",
-                name="Test Agent",
-                description="Test agent description",
-                image_urls=["test.jpg"],
-                date_submitted=FIXED_NOW,
-                status=prisma.enums.SubmissionStatus.APPROVED,
-                runs=50,
-                rating=4.2,
-                agent_id="test-agent-id",
-                agent_version=1,
-                sub_heading="Test agent subheading",
+                user_id="test-user-id",
                 slug="test-agent",
-                video_url="test.mp4",
+                listing_version_id="test-version-id",
+                listing_version=1,
+                graph_id="test-agent-id",
+                graph_version=1,
+                name="Test Agent",
+                sub_heading="Test agent subheading",
+                description="Test agent description",
+                instructions="Click the button!",
                 categories=["test-category"],
+                image_urls=["test.jpg"],
+                video_url="test.mp4",
+                agent_output_demo_url="demo_video.mp4",
+                submitted_at=FIXED_NOW,
+                changes_summary="Initial Submission",
+                status=prisma.enums.SubmissionStatus.APPROVED,
+                run_count=50,
+                review_count=5,
+                review_avg_rating=4.2,
             )
         ],
         pagination=store_model.Pagination(

autogpt_platform/backend/backend/api/features/store/test_cache_delete.py

@@ -11,6 +11,7 @@ import pytest
 from backend.util.models import Pagination
 
 from . import cache as store_cache
+from .db import StoreAgentsSortOptions
 from .model import StoreAgent, StoreAgentsResponse
 
 
@@ -215,7 +216,7 @@ class TestCacheDeletion:
             await store_cache._get_cached_store_agents(
                 featured=True,
                 creator="testuser",
-                sorted_by="rating",
+                sorted_by=StoreAgentsSortOptions.RATING,
                 search_query="AI assistant",
                 category="productivity",
                 page=2,
@@ -227,7 +228,7 @@ class TestCacheDeletion:
             deleted = store_cache._get_cached_store_agents.cache_delete(
                 featured=True,
                 creator="testuser",
-                sorted_by="rating",
+                sorted_by=StoreAgentsSortOptions.RATING,
                 search_query="AI assistant",
                 category="productivity",
                 page=2,
@@ -239,7 +240,7 @@ class TestCacheDeletion:
             deleted = store_cache._get_cached_store_agents.cache_delete(
                 featured=True,
                 creator="testuser",
-                sorted_by="rating",
+                sorted_by=StoreAgentsSortOptions.RATING,
                 search_query="AI assistant",
                 category="productivity",
                 page=2,

autogpt_platform/backend/backend/api/features/v1.py

@@ -55,6 +55,7 @@ from backend.data.credit import (
     set_auto_top_up,
 )
 from backend.data.graph import GraphSettings
+from backend.data.invited_user import get_or_activate_user
 from backend.data.model import CredentialsMetaInput, UserOnboarding
 from backend.data.notifications import NotificationPreference, NotificationPreferenceDTO
 from backend.data.onboarding import (
@@ -70,7 +71,6 @@ from backend.data.onboarding import (
     update_user_onboarding,
 )
 from backend.data.user import (
-    get_or_create_user,
     get_user_by_id,
     get_user_notification_preference,
     update_user_email,
@@ -126,6 +126,9 @@ v1_router = APIRouter()
 ########################################################
 
 
+_tally_background_tasks: set[asyncio.Task] = set()
+
+
 @v1_router.post(
     "/auth/user",
     summary="Get or create user",
@@ -133,7 +136,23 @@ v1_router = APIRouter()
     dependencies=[Security(requires_user)],
 )
 async def get_or_create_user_route(user_data: dict = Security(get_jwt_payload)):
-    user = await get_or_create_user(user_data)
+    user = await get_or_activate_user(user_data)
+
+    # Fire-and-forget: backfill Tally understanding when invite pre-seeding did
+    # not produce a stored result before first activation.
+    age_seconds = (datetime.now(timezone.utc) - user.created_at).total_seconds()
+    if age_seconds < 30:
+        try:
+            from backend.data.tally import populate_understanding_from_tally
+
+            task = asyncio.create_task(
+                populate_understanding_from_tally(user.id, user.email)
+            )
+            _tally_background_tasks.add(task)
+            task.add_done_callback(_tally_background_tasks.discard)
+        except Exception:
+            logger.debug("Failed to start Tally population task", exc_info=True)
+
     return user.model_dump()
 
 
@@ -144,7 +163,8 @@ async def get_or_create_user_route(user_data: dict = Security(get_jwt_payload)):
     dependencies=[Security(requires_user)],
 )
 async def update_user_email_route(
-    user_id: Annotated[str, Security(get_user_id)], email: str = Body(...)
+    user_id: Annotated[str, Security(get_user_id)],
+    email: str = Body(...),
 ) -> dict[str, str]:
     await update_user_email(user_id, email)
 
@@ -158,10 +178,16 @@ async def update_user_email_route(
     dependencies=[Security(requires_user)],
 )
 async def get_user_timezone_route(
-    user_data: dict = Security(get_jwt_payload),
+    user_id: Annotated[str, Security(get_user_id)],
 ) -> TimezoneResponse:
     """Get user timezone setting."""
-    user = await get_or_create_user(user_data)
+    try:
+        user = await get_user_by_id(user_id)
+    except ValueError:
+        raise HTTPException(
+            status_code=HTTP_404_NOT_FOUND,
+            detail="User not found. Please complete activation via /auth/user first.",
+        )
     return TimezoneResponse(timezone=user.timezone)
 
 
@@ -172,7 +198,8 @@ async def get_user_timezone_route(
     dependencies=[Security(requires_user)],
 )
 async def update_user_timezone_route(
-    user_id: Annotated[str, Security(get_user_id)], request: UpdateTimezoneRequest
+    user_id: Annotated[str, Security(get_user_id)],
+    request: UpdateTimezoneRequest,
 ) -> TimezoneResponse:
     """Update user timezone. The timezone should be a valid IANA timezone identifier."""
     user = await update_user_timezone(user_id, str(request.timezone))
@@ -428,7 +455,6 @@ async def execute_graph_block(
 async def upload_file(
     user_id: Annotated[str, Security(get_user_id)],
     file: UploadFile = File(...),
-    provider: str = "gcs",
     expiration_hours: int = 24,
 ) -> UploadFileResponse:
     """
@@ -491,7 +517,6 @@ async def upload_file(
     storage_path = await cloud_storage.store_file(
         content=content,
         filename=file_name,
-        provider=provider,
         expiration_hours=expiration_hours,
         user_id=user_id,
     )

autogpt_platform/backend/backend/api/features/v1_test.py

@@ -1,5 +1,5 @@
 import json
-from datetime import datetime
+from datetime import datetime, timezone
 from io import BytesIO
 from unittest.mock import AsyncMock, Mock, patch
 
@@ -43,6 +43,7 @@ def test_get_or_create_user_route(
 ) -> None:
     """Test get or create user endpoint"""
     mock_user = Mock()
+    mock_user.created_at = datetime.now(timezone.utc)
     mock_user.model_dump.return_value = {
         "id": test_user_id,
         "email": "test@example.com",
@@ -50,7 +51,7 @@ def test_get_or_create_user_route(
     }
 
     mocker.patch(
-        "backend.api.features.v1.get_or_create_user",
+        "backend.api.features.v1.get_or_activate_user",
         return_value=mock_user,
     )
 
@@ -514,7 +515,6 @@ async def test_upload_file_success(test_user_id: str):
         result = await upload_file(
             file=upload_file_mock,
             user_id=test_user_id,
-            provider="gcs",
             expiration_hours=24,
         )
 
@@ -532,7 +532,6 @@ async def test_upload_file_success(test_user_id: str):
         mock_handler.store_file.assert_called_once_with(
             content=file_content,
             filename="test.txt",
-            provider="gcs",
             expiration_hours=24,
             user_id=test_user_id,
         )

autogpt_platform/backend/backend/api/features/workspace/routes.py

@@ -3,15 +3,29 @@ Workspace API routes for managing user file storage.
 """
 
 import logging
+import os
 import re
 from typing import Annotated
 from urllib.parse import quote
 
 import fastapi
 from autogpt_libs.auth.dependencies import get_user_id, requires_user
+from fastapi import Query, UploadFile
 from fastapi.responses import Response
+from pydantic import BaseModel
 
-from backend.data.workspace import get_workspace, get_workspace_file
+from backend.data.workspace import (
+    WorkspaceFile,
+    count_workspace_files,
+    get_or_create_workspace,
+    get_workspace,
+    get_workspace_file,
+    get_workspace_total_size,
+    soft_delete_workspace_file,
+)
+from backend.util.settings import Config
+from backend.util.virus_scanner import scan_content_safe
+from backend.util.workspace import WorkspaceManager
 from backend.util.workspace_storage import get_workspace_storage
 
 
@@ -44,11 +58,11 @@ router = fastapi.APIRouter(
 )
 
 
-def _create_streaming_response(content: bytes, file) -> Response:
+def _create_streaming_response(content: bytes, file: WorkspaceFile) -> Response:
     """Create a streaming response for file content."""
     return Response(
         content=content,
-        media_type=file.mimeType,
+        media_type=file.mime_type,
         headers={
             "Content-Disposition": _sanitize_filename_for_header(file.name),
             "Content-Length": str(len(content)),
@@ -56,7 +70,7 @@ def _create_streaming_response(content: bytes, file) -> Response:
     )
 
 
-async def _create_file_download_response(file) -> Response:
+async def _create_file_download_response(file: WorkspaceFile) -> Response:
     """
     Create a download response for a workspace file.
 
@@ -66,38 +80,57 @@ async def _create_file_download_response(file) -> Response:
     storage = await get_workspace_storage()
 
     # For local storage, stream the file directly
-    if file.storagePath.startswith("local://"):
-        content = await storage.retrieve(file.storagePath)
+    if file.storage_path.startswith("local://"):
+        content = await storage.retrieve(file.storage_path)
         return _create_streaming_response(content, file)
 
     # For GCS, try to redirect to signed URL, fall back to streaming
     try:
-        url = await storage.get_download_url(file.storagePath, expires_in=300)
+        url = await storage.get_download_url(file.storage_path, expires_in=300)
         # If we got back an API path (fallback), stream directly instead
         if url.startswith("/api/"):
-            content = await storage.retrieve(file.storagePath)
+            content = await storage.retrieve(file.storage_path)
             return _create_streaming_response(content, file)
         return fastapi.responses.RedirectResponse(url=url, status_code=302)
     except Exception as e:
         # Log the signed URL failure with context
         logger.error(
             f"Failed to get signed URL for file {file.id} "
-            f"(storagePath={file.storagePath}): {e}",
+            f"(storagePath={file.storage_path}): {e}",
             exc_info=True,
         )
         # Fall back to streaming directly from GCS
         try:
-            content = await storage.retrieve(file.storagePath)
+            content = await storage.retrieve(file.storage_path)
             return _create_streaming_response(content, file)
         except Exception as fallback_error:
             logger.error(
                 f"Fallback streaming also failed for file {file.id} "
-                f"(storagePath={file.storagePath}): {fallback_error}",
+                f"(storagePath={file.storage_path}): {fallback_error}",
                 exc_info=True,
             )
             raise
 
 
+class UploadFileResponse(BaseModel):
+    file_id: str
+    name: str
+    path: str
+    mime_type: str
+    size_bytes: int
+
+
+class DeleteFileResponse(BaseModel):
+    deleted: bool
+
+
+class StorageUsageResponse(BaseModel):
+    used_bytes: int
+    limit_bytes: int
+    used_percent: float
+    file_count: int
+
+
 @router.get(
     "/files/{file_id}/download",
     summary="Download file by ID",
@@ -120,3 +153,148 @@ async def download_file(
         raise fastapi.HTTPException(status_code=404, detail="File not found")
 
     return await _create_file_download_response(file)
+
+
+@router.delete(
+    "/files/{file_id}",
+    summary="Delete a workspace file",
+)
+async def delete_workspace_file(
+    user_id: Annotated[str, fastapi.Security(get_user_id)],
+    file_id: str,
+) -> DeleteFileResponse:
+    """
+    Soft-delete a workspace file and attempt to remove it from storage.
+
+    Used when a user clears a file input in the builder.
+    """
+    workspace = await get_workspace(user_id)
+    if workspace is None:
+        raise fastapi.HTTPException(status_code=404, detail="Workspace not found")
+
+    manager = WorkspaceManager(user_id, workspace.id)
+    deleted = await manager.delete_file(file_id)
+    if not deleted:
+        raise fastapi.HTTPException(status_code=404, detail="File not found")
+
+    return DeleteFileResponse(deleted=True)
+
+
+@router.post(
+    "/files/upload",
+    summary="Upload file to workspace",
+)
+async def upload_file(
+    user_id: Annotated[str, fastapi.Security(get_user_id)],
+    file: UploadFile,
+    session_id: str | None = Query(default=None),
+) -> UploadFileResponse:
+    """
+    Upload a file to the user's workspace.
+
+    Files are stored in session-scoped paths when session_id is provided,
+    so the agent's session-scoped tools can discover them automatically.
+    """
+    config = Config()
+
+    # Sanitize filename — strip any directory components
+    filename = os.path.basename(file.filename or "upload") or "upload"
+
+    # Read file content with early abort on size limit
+    max_file_bytes = config.max_file_size_mb * 1024 * 1024
+    chunks: list[bytes] = []
+    total_size = 0
+    while chunk := await file.read(64 * 1024):  # 64KB chunks
+        total_size += len(chunk)
+        if total_size > max_file_bytes:
+            raise fastapi.HTTPException(
+                status_code=413,
+                detail=f"File exceeds maximum size of {config.max_file_size_mb} MB",
+            )
+        chunks.append(chunk)
+    content = b"".join(chunks)
+
+    # Get or create workspace
+    workspace = await get_or_create_workspace(user_id)
+
+    # Pre-write storage cap check (soft check — final enforcement is post-write)
+    storage_limit_bytes = config.max_workspace_storage_mb * 1024 * 1024
+    current_usage = await get_workspace_total_size(workspace.id)
+    if storage_limit_bytes and current_usage + len(content) > storage_limit_bytes:
+        used_percent = (current_usage / storage_limit_bytes) * 100
+        raise fastapi.HTTPException(
+            status_code=413,
+            detail={
+                "message": "Storage limit exceeded",
+                "used_bytes": current_usage,
+                "limit_bytes": storage_limit_bytes,
+                "used_percent": round(used_percent, 1),
+            },
+        )
+
+    # Warn at 80% usage
+    if (
+        storage_limit_bytes
+        and (usage_ratio := (current_usage + len(content)) / storage_limit_bytes) >= 0.8
+    ):
+        logger.warning(
+            f"User {user_id} workspace storage at {usage_ratio * 100:.1f}% "
+            f"({current_usage + len(content)} / {storage_limit_bytes} bytes)"
+        )
+
+    # Virus scan
+    await scan_content_safe(content, filename=filename)
+
+    # Write file via WorkspaceManager
+    manager = WorkspaceManager(user_id, workspace.id, session_id)
+    try:
+        workspace_file = await manager.write_file(content, filename)
+    except ValueError as e:
+        raise fastapi.HTTPException(status_code=409, detail=str(e)) from e
+
+    # Post-write storage check — eliminates TOCTOU race on the quota.
+    # If a concurrent upload pushed us over the limit, undo this write.
+    new_total = await get_workspace_total_size(workspace.id)
+    if storage_limit_bytes and new_total > storage_limit_bytes:
+        await soft_delete_workspace_file(workspace_file.id, workspace.id)
+        raise fastapi.HTTPException(
+            status_code=413,
+            detail={
+                "message": "Storage limit exceeded (concurrent upload)",
+                "used_bytes": new_total,
+                "limit_bytes": storage_limit_bytes,
+            },
+        )
+
+    return UploadFileResponse(
+        file_id=workspace_file.id,
+        name=workspace_file.name,
+        path=workspace_file.path,
+        mime_type=workspace_file.mime_type,
+        size_bytes=workspace_file.size_bytes,
+    )
+
+
+@router.get(
+    "/storage/usage",
+    summary="Get workspace storage usage",
+)
+async def get_storage_usage(
+    user_id: Annotated[str, fastapi.Security(get_user_id)],
+) -> StorageUsageResponse:
+    """
+    Get storage usage information for the user's workspace.
+    """
+    config = Config()
+    workspace = await get_or_create_workspace(user_id)
+
+    used_bytes = await get_workspace_total_size(workspace.id)
+    file_count = await count_workspace_files(workspace.id)
+    limit_bytes = config.max_workspace_storage_mb * 1024 * 1024
+
+    return StorageUsageResponse(
+        used_bytes=used_bytes,
+        limit_bytes=limit_bytes,
+        used_percent=round((used_bytes / limit_bytes) * 100, 1) if limit_bytes else 0,
+        file_count=file_count,
+    )

autogpt_platform/backend/backend/api/features/workspace/routes_test.py

@@ -0,0 +1,359 @@
+"""Tests for workspace file upload and download routes."""
+
+import io
+from datetime import datetime, timezone
+
+import fastapi
+import fastapi.testclient
+import pytest
+import pytest_mock
+
+from backend.api.features.workspace import routes as workspace_routes
+from backend.data.workspace import WorkspaceFile
+
+app = fastapi.FastAPI()
+app.include_router(workspace_routes.router)
+
+
+@app.exception_handler(ValueError)
+async def _value_error_handler(
+    request: fastapi.Request, exc: ValueError
+) -> fastapi.responses.JSONResponse:
+    """Mirror the production ValueError → 400 mapping from rest_api.py."""
+    return fastapi.responses.JSONResponse(status_code=400, content={"detail": str(exc)})
+
+
+client = fastapi.testclient.TestClient(app)
+
+TEST_USER_ID = "3e53486c-cf57-477e-ba2a-cb02dc828e1a"
+
+MOCK_WORKSPACE = type("W", (), {"id": "ws-1"})()
+
+_NOW = datetime(2023, 1, 1, tzinfo=timezone.utc)
+
+MOCK_FILE = WorkspaceFile(
+    id="file-aaa-bbb",
+    workspace_id="ws-1",
+    created_at=_NOW,
+    updated_at=_NOW,
+    name="hello.txt",
+    path="/session/hello.txt",
+    mime_type="text/plain",
+    size_bytes=13,
+    storage_path="local://hello.txt",
+)
+
+
+@pytest.fixture(autouse=True)
+def setup_app_auth(mock_jwt_user):
+    from autogpt_libs.auth.jwt_utils import get_jwt_payload
+
+    app.dependency_overrides[get_jwt_payload] = mock_jwt_user["get_jwt_payload"]
+    yield
+    app.dependency_overrides.clear()
+
+
+def _upload(
+    filename: str = "hello.txt",
+    content: bytes = b"Hello, world!",
+    content_type: str = "text/plain",
+):
+    """Helper to POST a file upload."""
+    return client.post(
+        "/files/upload?session_id=sess-1",
+        files={"file": (filename, io.BytesIO(content), content_type)},
+    )
+
+
+# ---- Happy path ----
+
+
+def test_upload_happy_path(mocker: pytest_mock.MockFixture):
+    mocker.patch(
+        "backend.api.features.workspace.routes.get_or_create_workspace",
+        return_value=MOCK_WORKSPACE,
+    )
+    mocker.patch(
+        "backend.api.features.workspace.routes.get_workspace_total_size",
+        return_value=0,
+    )
+    mocker.patch(
+        "backend.api.features.workspace.routes.scan_content_safe",
+        return_value=None,
+    )
+    mock_manager = mocker.MagicMock()
+    mock_manager.write_file = mocker.AsyncMock(return_value=MOCK_FILE)
+    mocker.patch(
+        "backend.api.features.workspace.routes.WorkspaceManager",
+        return_value=mock_manager,
+    )
+
+    response = _upload()
+    assert response.status_code == 200
+    data = response.json()
+    assert data["file_id"] == "file-aaa-bbb"
+    assert data["name"] == "hello.txt"
+    assert data["size_bytes"] == 13
+
+
+# ---- Per-file size limit ----
+
+
+def test_upload_exceeds_max_file_size(mocker: pytest_mock.MockFixture):
+    """Files larger than max_file_size_mb should be rejected with 413."""
+    cfg = mocker.patch("backend.api.features.workspace.routes.Config")
+    cfg.return_value.max_file_size_mb = 0  # 0 MB → any content is too big
+    cfg.return_value.max_workspace_storage_mb = 500
+
+    response = _upload(content=b"x" * 1024)
+    assert response.status_code == 413
+
+
+# ---- Storage quota exceeded ----
+
+
+def test_upload_storage_quota_exceeded(mocker: pytest_mock.MockFixture):
+    mocker.patch(
+        "backend.api.features.workspace.routes.get_or_create_workspace",
+        return_value=MOCK_WORKSPACE,
+    )
+    # Current usage already at limit
+    mocker.patch(
+        "backend.api.features.workspace.routes.get_workspace_total_size",
+        return_value=500 * 1024 * 1024,
+    )
+
+    response = _upload()
+    assert response.status_code == 413
+    assert "Storage limit exceeded" in response.text
+
+
+# ---- Post-write quota race (B2) ----
+
+
+def test_upload_post_write_quota_race(mocker: pytest_mock.MockFixture):
+    """If a concurrent upload tips the total over the limit after write,
+    the file should be soft-deleted and 413 returned."""
+    mocker.patch(
+        "backend.api.features.workspace.routes.get_or_create_workspace",
+        return_value=MOCK_WORKSPACE,
+    )
+    # Pre-write check passes (under limit), but post-write check fails
+    mocker.patch(
+        "backend.api.features.workspace.routes.get_workspace_total_size",
+        side_effect=[0, 600 * 1024 * 1024],  # first call OK, second over limit
+    )
+    mocker.patch(
+        "backend.api.features.workspace.routes.scan_content_safe",
+        return_value=None,
+    )
+    mock_manager = mocker.MagicMock()
+    mock_manager.write_file = mocker.AsyncMock(return_value=MOCK_FILE)
+    mocker.patch(
+        "backend.api.features.workspace.routes.WorkspaceManager",
+        return_value=mock_manager,
+    )
+    mock_delete = mocker.patch(
+        "backend.api.features.workspace.routes.soft_delete_workspace_file",
+        return_value=None,
+    )
+
+    response = _upload()
+    assert response.status_code == 413
+    mock_delete.assert_called_once_with("file-aaa-bbb", "ws-1")
+
+
+# ---- Any extension accepted (no allowlist) ----
+
+
+def test_upload_any_extension(mocker: pytest_mock.MockFixture):
+    """Any file extension should be accepted — ClamAV is the security layer."""
+    mocker.patch(
+        "backend.api.features.workspace.routes.get_or_create_workspace",
+        return_value=MOCK_WORKSPACE,
+    )
+    mocker.patch(
+        "backend.api.features.workspace.routes.get_workspace_total_size",
+        return_value=0,
+    )
+    mocker.patch(
+        "backend.api.features.workspace.routes.scan_content_safe",
+        return_value=None,
+    )
+    mock_manager = mocker.MagicMock()
+    mock_manager.write_file = mocker.AsyncMock(return_value=MOCK_FILE)
+    mocker.patch(
+        "backend.api.features.workspace.routes.WorkspaceManager",
+        return_value=mock_manager,
+    )
+
+    response = _upload(filename="data.xyz", content=b"arbitrary")
+    assert response.status_code == 200
+
+
+# ---- Virus scan rejection ----
+
+
+def test_upload_blocked_by_virus_scan(mocker: pytest_mock.MockFixture):
+    """Files flagged by ClamAV should be rejected and never written to storage."""
+    from backend.api.features.store.exceptions import VirusDetectedError
+
+    mocker.patch(
+        "backend.api.features.workspace.routes.get_or_create_workspace",
+        return_value=MOCK_WORKSPACE,
+    )
+    mocker.patch(
+        "backend.api.features.workspace.routes.get_workspace_total_size",
+        return_value=0,
+    )
+    mocker.patch(
+        "backend.api.features.workspace.routes.scan_content_safe",
+        side_effect=VirusDetectedError("Eicar-Test-Signature"),
+    )
+    mock_manager = mocker.MagicMock()
+    mock_manager.write_file = mocker.AsyncMock(return_value=MOCK_FILE)
+    mocker.patch(
+        "backend.api.features.workspace.routes.WorkspaceManager",
+        return_value=mock_manager,
+    )
+
+    response = _upload(filename="evil.exe", content=b"X5O!P%@AP...")
+    assert response.status_code == 400
+    assert "Virus detected" in response.text
+    mock_manager.write_file.assert_not_called()
+
+
+# ---- No file extension ----
+
+
+def test_upload_file_without_extension(mocker: pytest_mock.MockFixture):
+    """Files without an extension should be accepted and stored as-is."""
+    mocker.patch(
+        "backend.api.features.workspace.routes.get_or_create_workspace",
+        return_value=MOCK_WORKSPACE,
+    )
+    mocker.patch(
+        "backend.api.features.workspace.routes.get_workspace_total_size",
+        return_value=0,
+    )
+    mocker.patch(
+        "backend.api.features.workspace.routes.scan_content_safe",
+        return_value=None,
+    )
+    mock_manager = mocker.MagicMock()
+    mock_manager.write_file = mocker.AsyncMock(return_value=MOCK_FILE)
+    mocker.patch(
+        "backend.api.features.workspace.routes.WorkspaceManager",
+        return_value=mock_manager,
+    )
+
+    response = _upload(
+        filename="Makefile",
+        content=b"all:\n\techo hello",
+        content_type="application/octet-stream",
+    )
+    assert response.status_code == 200
+    mock_manager.write_file.assert_called_once()
+    assert mock_manager.write_file.call_args[0][1] == "Makefile"
+
+
+# ---- Filename sanitization (SF5) ----
+
+
+def test_upload_strips_path_components(mocker: pytest_mock.MockFixture):
+    """Path-traversal filenames should be reduced to their basename."""
+    mocker.patch(
+        "backend.api.features.workspace.routes.get_or_create_workspace",
+        return_value=MOCK_WORKSPACE,
+    )
+    mocker.patch(
+        "backend.api.features.workspace.routes.get_workspace_total_size",
+        return_value=0,
+    )
+    mocker.patch(
+        "backend.api.features.workspace.routes.scan_content_safe",
+        return_value=None,
+    )
+    mock_manager = mocker.MagicMock()
+    mock_manager.write_file = mocker.AsyncMock(return_value=MOCK_FILE)
+    mocker.patch(
+        "backend.api.features.workspace.routes.WorkspaceManager",
+        return_value=mock_manager,
+    )
+
+    # Filename with traversal
+    _upload(filename="../../etc/passwd.txt")
+
+    # write_file should have been called with just the basename
+    mock_manager.write_file.assert_called_once()
+    call_args = mock_manager.write_file.call_args
+    assert call_args[0][1] == "passwd.txt"
+
+
+# ---- Download ----
+
+
+def test_download_file_not_found(mocker: pytest_mock.MockFixture):
+    mocker.patch(
+        "backend.api.features.workspace.routes.get_workspace",
+        return_value=MOCK_WORKSPACE,
+    )
+    mocker.patch(
+        "backend.api.features.workspace.routes.get_workspace_file",
+        return_value=None,
+    )
+
+    response = client.get("/files/some-file-id/download")
+    assert response.status_code == 404
+
+
+# ---- Delete ----
+
+
+def test_delete_file_success(mocker: pytest_mock.MockFixture):
+    """Deleting an existing file should return {"deleted": true}."""
+    mocker.patch(
+        "backend.api.features.workspace.routes.get_workspace",
+        return_value=MOCK_WORKSPACE,
+    )
+    mock_manager = mocker.MagicMock()
+    mock_manager.delete_file = mocker.AsyncMock(return_value=True)
+    mocker.patch(
+        "backend.api.features.workspace.routes.WorkspaceManager",
+        return_value=mock_manager,
+    )
+
+    response = client.delete("/files/file-aaa-bbb")
+    assert response.status_code == 200
+    assert response.json() == {"deleted": True}
+    mock_manager.delete_file.assert_called_once_with("file-aaa-bbb")
+
+
+def test_delete_file_not_found(mocker: pytest_mock.MockFixture):
+    """Deleting a non-existent file should return 404."""
+    mocker.patch(
+        "backend.api.features.workspace.routes.get_workspace",
+        return_value=MOCK_WORKSPACE,
+    )
+    mock_manager = mocker.MagicMock()
+    mock_manager.delete_file = mocker.AsyncMock(return_value=False)
+    mocker.patch(
+        "backend.api.features.workspace.routes.WorkspaceManager",
+        return_value=mock_manager,
+    )
+
+    response = client.delete("/files/nonexistent-id")
+    assert response.status_code == 404
+    assert "File not found" in response.text
+
+
+def test_delete_file_no_workspace(mocker: pytest_mock.MockFixture):
+    """Deleting when user has no workspace should return 404."""
+    mocker.patch(
+        "backend.api.features.workspace.routes.get_workspace",
+        return_value=None,
+    )
+
+    response = client.delete("/files/file-aaa-bbb")
+    assert response.status_code == 404
+    assert "Workspace not found" in response.text

autogpt_platform/backend/backend/api/model.py

@@ -94,3 +94,8 @@ class NotificationPayload(pydantic.BaseModel):
 
 class OnboardingNotificationPayload(NotificationPayload):
     step: OnboardingStep | None
+
+
+class CopilotCompletionPayload(NotificationPayload):
+    session_id: str
+    status: Literal["completed", "failed"]

autogpt_platform/backend/backend/api/rest_api.py

@@ -19,6 +19,7 @@ from prisma.errors import PrismaError
 import backend.api.features.admin.credit_admin_routes
 import backend.api.features.admin.execution_analytics_routes
 import backend.api.features.admin.store_admin_routes
+import backend.api.features.admin.user_admin_routes
 import backend.api.features.builder
 import backend.api.features.builder.routes
 import backend.api.features.chat.routes as chat_routes
@@ -41,9 +42,9 @@ import backend.data.user
 import backend.integrations.webhooks.utils
 import backend.util.service
 import backend.util.settings
-from backend.api.features.chat.completion_consumer import (
-    start_completion_consumer,
-    stop_completion_consumer,
+from backend.api.features.library.exceptions import (
+    FolderAlreadyExistsError,
+    FolderValidationError,
 )
 from backend.blocks.llm import DEFAULT_LLM_MODEL
 from backend.data.model import Credentials
@@ -55,6 +56,7 @@ from backend.util.exceptions import (
     MissingConfigError,
     NotAuthorizedError,
     NotFoundError,
+    PreconditionFailed,
 )
 from backend.util.feature_flag import initialize_launchdarkly, shutdown_launchdarkly
 from backend.util.service import UnhealthyServiceError
@@ -123,21 +125,9 @@ async def lifespan_context(app: fastapi.FastAPI):
     await backend.data.graph.migrate_llm_models(DEFAULT_LLM_MODEL)
     await backend.integrations.webhooks.utils.migrate_legacy_triggered_graphs()
 
-    # Start chat completion consumer for Redis Streams notifications
-    try:
-        await start_completion_consumer()
-    except Exception as e:
-        logger.warning(f"Could not start chat completion consumer: {e}")
-
     with launch_darkly_context():
         yield
 
-    # Stop chat completion consumer
-    try:
-        await stop_completion_consumer()
-    except Exception as e:
-        logger.warning(f"Error stopping chat completion consumer: {e}")
-
     try:
         await shutdown_cloud_storage_handler()
     except Exception as e:
@@ -277,12 +267,17 @@ async def validation_error_handler(
 
 
 app.add_exception_handler(PrismaError, handle_internal_http_error(500))
+app.add_exception_handler(
+    FolderAlreadyExistsError, handle_internal_http_error(409, False)
+)
+app.add_exception_handler(FolderValidationError, handle_internal_http_error(400, False))
 app.add_exception_handler(NotFoundError, handle_internal_http_error(404, False))
 app.add_exception_handler(NotAuthorizedError, handle_internal_http_error(403, False))
 app.add_exception_handler(RequestValidationError, validation_error_handler)
 app.add_exception_handler(pydantic.ValidationError, validation_error_handler)
 app.add_exception_handler(MissingConfigError, handle_internal_http_error(503))
 app.add_exception_handler(ValueError, handle_internal_http_error(400))
+app.add_exception_handler(PreconditionFailed, handle_internal_http_error(428))
 app.add_exception_handler(Exception, handle_internal_http_error(500))
 
 app.include_router(backend.api.features.v1.v1_router, tags=["v1"], prefix="/api")
@@ -317,6 +312,11 @@ app.include_router(
     tags=["v2", "admin"],
     prefix="/api/executions",
 )
+app.include_router(
+    backend.api.features.admin.user_admin_routes.router,
+    tags=["v2", "admin"],
+    prefix="/api/users",
+)
 app.include_router(
     backend.api.features.executions.review.routes.router,
     tags=["v2", "executions", "review"],

autogpt_platform/backend/backend/app.py

@@ -24,7 +24,7 @@ def run_processes(*processes: "AppProcess", **kwargs):
         # Run the last process in the foreground.
         processes[-1].start(background=False, **kwargs)
     finally:
-        for process in processes:
+        for process in reversed(processes):
             try:
                 process.stop()
             except Exception as e:
@@ -38,7 +38,9 @@ def main(**kwargs):
 
     from backend.api.rest_api import AgentServer
     from backend.api.ws_api import WebsocketServer
-    from backend.executor import DatabaseManager, ExecutionManager, Scheduler
+    from backend.copilot.executor.manager import CoPilotExecutor
+    from backend.data.db_manager import DatabaseManager
+    from backend.executor import ExecutionManager, Scheduler
     from backend.notifications import NotificationManager
 
     run_processes(
@@ -48,6 +50,7 @@ def main(**kwargs):
         WebsocketServer(),
         AgentServer(),
         ExecutionManager(),
+        CoPilotExecutor(),
         **kwargs,
     )
 

autogpt_platform/backend/backend/blocks/_base.py

@@ -418,6 +418,8 @@ class BlockWebhookConfig(BlockManualWebhookConfig):
 
 
 class Block(ABC, Generic[BlockSchemaInputType, BlockSchemaOutputType]):
+    _optimized_description: ClassVar[str | None] = None
+
     def __init__(
         self,
         id: str = "",
@@ -470,6 +472,8 @@ class Block(ABC, Generic[BlockSchemaInputType, BlockSchemaOutputType]):
         self.block_type = block_type
         self.webhook_config = webhook_config
         self.is_sensitive_action = is_sensitive_action
+        # Read from ClassVar set by initialize_blocks()
+        self.optimized_description: str | None = type(self)._optimized_description
         self.execution_stats: "NodeExecutionStats" = NodeExecutionStats()
 
         if self.webhook_config:
@@ -620,6 +624,7 @@ class Block(ABC, Generic[BlockSchemaInputType, BlockSchemaOutputType]):
         graph_id: str,
         graph_version: int,
         execution_context: "ExecutionContext",
+        is_graph_execution: bool = True,
         **kwargs,
     ) -> tuple[bool, BlockInput]:
         """
@@ -648,6 +653,7 @@ class Block(ABC, Generic[BlockSchemaInputType, BlockSchemaOutputType]):
             graph_version=graph_version,
             block_name=self.name,
             editable=True,
+            is_graph_execution=is_graph_execution,
         )
 
         if decision is None:

autogpt_platform/backend/backend/blocks/basic.py

@@ -126,7 +126,7 @@ class PrintToConsoleBlock(Block):
             output_schema=PrintToConsoleBlock.Output,
             test_input={"text": "Hello, World!"},
             is_sensitive_action=True,
-            disabled=True,  # Disabled per Nick Tindle's request (OPEN-3000)
+            disabled=True,
             test_output=[
                 ("output", "Hello, World!"),
                 ("status", "printed"),

autogpt_platform/backend/backend/blocks/code_executor.py

@@ -142,7 +142,7 @@ class BaseE2BExecutorMixin:
                 start_timestamp = ts_result.stdout.strip() if ts_result.stdout else None
 
             # Execute the code
-            execution = await sandbox.run_code(
+            execution = await sandbox.run_code(  # type: ignore[attr-defined]
                 code,
                 language=language.value,
                 on_error=lambda e: sandbox.kill(),  # Kill the sandbox on error

autogpt_platform/backend/backend/blocks/email_block.py

@@ -96,6 +96,7 @@ class SendEmailBlock(Block):
             test_credentials=TEST_CREDENTIALS,
             test_output=[("status", "Email sent successfully")],
             test_mock={"send_email": lambda *args, **kwargs: "Email sent successfully"},
+            is_sensitive_action=True,
         )
 
     @staticmethod

autogpt_platform/backend/backend/blocks/github/_utils.py

@@ -0,0 +1,3 @@
+def github_repo_path(repo_url: str) -> str:
+    """Extract 'owner/repo' from a GitHub repository URL."""
+    return repo_url.replace("https://github.com/", "")

autogpt_platform/backend/backend/blocks/github/commits.py

@@ -0,0 +1,408 @@
+import asyncio
+from enum import StrEnum
+from urllib.parse import quote
+
+from typing_extensions import TypedDict
+
+from backend.blocks._base import (
+    Block,
+    BlockCategory,
+    BlockOutput,
+    BlockSchemaInput,
+    BlockSchemaOutput,
+)
+from backend.data.execution import ExecutionContext
+from backend.data.model import SchemaField
+from backend.util.file import parse_data_uri, resolve_media_content
+from backend.util.type import MediaFileType
+
+from ._api import get_api
+from ._auth import (
+    TEST_CREDENTIALS,
+    TEST_CREDENTIALS_INPUT,
+    GithubCredentials,
+    GithubCredentialsField,
+    GithubCredentialsInput,
+)
+from ._utils import github_repo_path
+
+
+class GithubListCommitsBlock(Block):
+    class Input(BlockSchemaInput):
+        credentials: GithubCredentialsInput = GithubCredentialsField("repo")
+        repo_url: str = SchemaField(
+            description="URL of the GitHub repository",
+            placeholder="https://github.com/owner/repo",
+        )
+        branch: str = SchemaField(
+            description="Branch name to list commits from",
+            default="main",
+        )
+        per_page: int = SchemaField(
+            description="Number of commits to return (max 100)",
+            default=30,
+            ge=1,
+            le=100,
+        )
+        page: int = SchemaField(
+            description="Page number for pagination",
+            default=1,
+            ge=1,
+        )
+
+    class Output(BlockSchemaOutput):
+        class CommitItem(TypedDict):
+            sha: str
+            message: str
+            author: str
+            date: str
+            url: str
+
+        commit: CommitItem = SchemaField(
+            title="Commit", description="A commit with its details"
+        )
+        commits: list[CommitItem] = SchemaField(
+            description="List of commits with their details"
+        )
+        error: str = SchemaField(description="Error message if listing commits failed")
+
+    def __init__(self):
+        super().__init__(
+            id="8b13f579-d8b6-4dc2-a140-f770428805de",
+            description="This block lists commits on a branch in a GitHub repository.",
+            categories={BlockCategory.DEVELOPER_TOOLS},
+            input_schema=GithubListCommitsBlock.Input,
+            output_schema=GithubListCommitsBlock.Output,
+            test_input={
+                "repo_url": "https://github.com/owner/repo",
+                "branch": "main",
+                "per_page": 30,
+                "page": 1,
+                "credentials": TEST_CREDENTIALS_INPUT,
+            },
+            test_credentials=TEST_CREDENTIALS,
+            test_output=[
+                (
+                    "commits",
+                    [
+                        {
+                            "sha": "abc123",
+                            "message": "Initial commit",
+                            "author": "octocat",
+                            "date": "2024-01-01T00:00:00Z",
+                            "url": "https://github.com/owner/repo/commit/abc123",
+                        }
+                    ],
+                ),
+                (
+                    "commit",
+                    {
+                        "sha": "abc123",
+                        "message": "Initial commit",
+                        "author": "octocat",
+                        "date": "2024-01-01T00:00:00Z",
+                        "url": "https://github.com/owner/repo/commit/abc123",
+                    },
+                ),
+            ],
+            test_mock={
+                "list_commits": lambda *args, **kwargs: [
+                    {
+                        "sha": "abc123",
+                        "message": "Initial commit",
+                        "author": "octocat",
+                        "date": "2024-01-01T00:00:00Z",
+                        "url": "https://github.com/owner/repo/commit/abc123",
+                    }
+                ]
+            },
+        )
+
+    @staticmethod
+    async def list_commits(
+        credentials: GithubCredentials,
+        repo_url: str,
+        branch: str,
+        per_page: int,
+        page: int,
+    ) -> list[Output.CommitItem]:
+        api = get_api(credentials)
+        commits_url = repo_url + "/commits"
+        params = {"sha": branch, "per_page": str(per_page), "page": str(page)}
+        response = await api.get(commits_url, params=params)
+        data = response.json()
+        repo_path = github_repo_path(repo_url)
+        return [
+            GithubListCommitsBlock.Output.CommitItem(
+                sha=c["sha"],
+                message=c["commit"]["message"],
+                author=(c["commit"].get("author") or {}).get("name", "Unknown"),
+                date=(c["commit"].get("author") or {}).get("date", ""),
+                url=f"https://github.com/{repo_path}/commit/{c['sha']}",
+            )
+            for c in data
+        ]
+
+    async def run(
+        self,
+        input_data: Input,
+        *,
+        credentials: GithubCredentials,
+        **kwargs,
+    ) -> BlockOutput:
+        try:
+            commits = await self.list_commits(
+                credentials,
+                input_data.repo_url,
+                input_data.branch,
+                input_data.per_page,
+                input_data.page,
+            )
+            yield "commits", commits
+            for commit in commits:
+                yield "commit", commit
+        except Exception as e:
+            yield "error", str(e)
+
+
+class FileOperation(StrEnum):
+    """File operations for GithubMultiFileCommitBlock.
+
+    UPSERT creates or overwrites a file (the Git Trees API does not distinguish
+    between creation and update — the blob is placed at the given path regardless
+    of whether a file already exists there).
+
+    DELETE removes a file from the tree.
+    """
+
+    UPSERT = "upsert"
+    DELETE = "delete"
+
+
+class FileOperationInput(TypedDict):
+    path: str
+    # MediaFileType is a str NewType — no runtime breakage for existing callers.
+    content: MediaFileType
+    operation: FileOperation
+
+
+class GithubMultiFileCommitBlock(Block):
+    class Input(BlockSchemaInput):
+        credentials: GithubCredentialsInput = GithubCredentialsField("repo")
+        repo_url: str = SchemaField(
+            description="URL of the GitHub repository",
+            placeholder="https://github.com/owner/repo",
+        )
+        branch: str = SchemaField(
+            description="Branch to commit to",
+            placeholder="feature-branch",
+        )
+        commit_message: str = SchemaField(
+            description="Commit message",
+            placeholder="Add new feature",
+        )
+        files: list[FileOperationInput] = SchemaField(
+            description=(
+                "List of file operations. Each item has: "
+                "'path' (file path), 'content' (file content, ignored for delete), "
+                "'operation' (upsert/delete)"
+            ),
+        )
+
+    class Output(BlockSchemaOutput):
+        sha: str = SchemaField(description="SHA of the new commit")
+        url: str = SchemaField(description="URL of the new commit")
+        error: str = SchemaField(description="Error message if the commit failed")
+
+    def __init__(self):
+        super().__init__(
+            id="389eee51-a95e-4230-9bed-92167a327802",
+            description=(
+                "This block creates a single commit with multiple file "
+                "upsert/delete operations using the Git Trees API."
+            ),
+            categories={BlockCategory.DEVELOPER_TOOLS},
+            input_schema=GithubMultiFileCommitBlock.Input,
+            output_schema=GithubMultiFileCommitBlock.Output,
+            test_input={
+                "repo_url": "https://github.com/owner/repo",
+                "branch": "feature",
+                "commit_message": "Add files",
+                "files": [
+                    {
+                        "path": "src/new.py",
+                        "content": "print('hello')",
+                        "operation": "upsert",
+                    },
+                    {
+                        "path": "src/old.py",
+                        "content": "",
+                        "operation": "delete",
+                    },
+                ],
+                "credentials": TEST_CREDENTIALS_INPUT,
+            },
+            test_credentials=TEST_CREDENTIALS,
+            test_output=[
+                ("sha", "newcommitsha"),
+                ("url", "https://github.com/owner/repo/commit/newcommitsha"),
+            ],
+            test_mock={
+                "multi_file_commit": lambda *args, **kwargs: (
+                    "newcommitsha",
+                    "https://github.com/owner/repo/commit/newcommitsha",
+                )
+            },
+        )
+
+    @staticmethod
+    async def multi_file_commit(
+        credentials: GithubCredentials,
+        repo_url: str,
+        branch: str,
+        commit_message: str,
+        files: list[FileOperationInput],
+    ) -> tuple[str, str]:
+        api = get_api(credentials)
+        safe_branch = quote(branch, safe="")
+
+        # 1. Get the latest commit SHA for the branch
+        ref_url = repo_url + f"/git/refs/heads/{safe_branch}"
+        response = await api.get(ref_url)
+        ref_data = response.json()
+        latest_commit_sha = ref_data["object"]["sha"]
+
+        # 2. Get the tree SHA of the latest commit
+        commit_url = repo_url + f"/git/commits/{latest_commit_sha}"
+        response = await api.get(commit_url)
+        commit_data = response.json()
+        base_tree_sha = commit_data["tree"]["sha"]
+
+        # 3. Build tree entries for each file operation (blobs created concurrently)
+        async def _create_blob(content: str, encoding: str = "utf-8") -> str:
+            blob_url = repo_url + "/git/blobs"
+            blob_response = await api.post(
+                blob_url,
+                json={"content": content, "encoding": encoding},
+            )
+            return blob_response.json()["sha"]
+
+        tree_entries: list[dict] = []
+        upsert_files = []
+        for file_op in files:
+            path = file_op["path"]
+            operation = FileOperation(file_op.get("operation", "upsert"))
+
+            if operation == FileOperation.DELETE:
+                tree_entries.append(
+                    {
+                        "path": path,
+                        "mode": "100644",
+                        "type": "blob",
+                        "sha": None,  # null SHA = delete
+                    }
+                )
+            else:
+                upsert_files.append((path, file_op.get("content", "")))
+
+        # Create all blobs concurrently. Data URIs (from store_media_file)
+        # are sent as base64 blobs to preserve binary content.
+        if upsert_files:
+
+            async def _make_blob(content: str) -> str:
+                parsed = parse_data_uri(content)
+                if parsed is not None:
+                    _, b64_payload = parsed
+                    return await _create_blob(b64_payload, encoding="base64")
+                return await _create_blob(content)
+
+            blob_shas = await asyncio.gather(
+                *[_make_blob(content) for _, content in upsert_files]
+            )
+            for (path, _), blob_sha in zip(upsert_files, blob_shas):
+                tree_entries.append(
+                    {
+                        "path": path,
+                        "mode": "100644",
+                        "type": "blob",
+                        "sha": blob_sha,
+                    }
+                )
+
+        # 4. Create a new tree
+        tree_url = repo_url + "/git/trees"
+        tree_response = await api.post(
+            tree_url,
+            json={"base_tree": base_tree_sha, "tree": tree_entries},
+        )
+        new_tree_sha = tree_response.json()["sha"]
+
+        # 5. Create a new commit
+        new_commit_url = repo_url + "/git/commits"
+        commit_response = await api.post(
+            new_commit_url,
+            json={
+                "message": commit_message,
+                "tree": new_tree_sha,
+                "parents": [latest_commit_sha],
+            },
+        )
+        new_commit_sha = commit_response.json()["sha"]
+
+        # 6. Update the branch reference
+        try:
+            await api.patch(
+                ref_url,
+                json={"sha": new_commit_sha},
+            )
+        except Exception as e:
+            raise RuntimeError(
+                f"Commit {new_commit_sha} was created but failed to update "
+                f"ref heads/{branch}: {e}. "
+                f"You can recover by manually updating the branch to {new_commit_sha}."
+            ) from e
+
+        repo_path = github_repo_path(repo_url)
+        commit_web_url = f"https://github.com/{repo_path}/commit/{new_commit_sha}"
+        return new_commit_sha, commit_web_url
+
+    async def run(
+        self,
+        input_data: Input,
+        *,
+        credentials: GithubCredentials,
+        execution_context: ExecutionContext,
+        **kwargs,
+    ) -> BlockOutput:
+        try:
+            # Resolve media references (workspace://, data:, URLs) to data
+            # URIs so _make_blob can send binary content correctly.
+            resolved_files: list[FileOperationInput] = []
+            for file_op in input_data.files:
+                content = file_op.get("content", "")
+                operation = FileOperation(file_op.get("operation", "upsert"))
+                if operation != FileOperation.DELETE:
+                    content = await resolve_media_content(
+                        MediaFileType(content),
+                        execution_context,
+                        return_format="for_external_api",
+                    )
+                resolved_files.append(
+                    FileOperationInput(
+                        path=file_op["path"],
+                        content=MediaFileType(content),
+                        operation=operation,
+                    )
+                )
+
+            sha, url = await self.multi_file_commit(
+                credentials,
+                input_data.repo_url,
+                input_data.branch,
+                input_data.commit_message,
+                resolved_files,
+            )
+            yield "sha", sha
+            yield "url", url
+        except Exception as e:
+            yield "error", str(e)

autogpt_platform/backend/backend/blocks/github/pull_requests.py

@@ -1,4 +1,5 @@
 import re
+from typing import Literal
 
 from typing_extensions import TypedDict
 
@@ -20,6 +21,8 @@ from ._auth import (
     GithubCredentialsInput,
 )
 
+MergeMethod = Literal["merge", "squash", "rebase"]
+
 
 class GithubListPullRequestsBlock(Block):
     class Input(BlockSchemaInput):
@@ -558,12 +561,109 @@ class GithubListPRReviewersBlock(Block):
             yield "reviewer", reviewer
 
 
+class GithubMergePullRequestBlock(Block):
+    class Input(BlockSchemaInput):
+        credentials: GithubCredentialsInput = GithubCredentialsField("repo")
+        pr_url: str = SchemaField(
+            description="URL of the GitHub pull request",
+            placeholder="https://github.com/owner/repo/pull/1",
+        )
+        merge_method: MergeMethod = SchemaField(
+            description="Merge method to use: merge, squash, or rebase",
+            default="merge",
+        )
+        commit_title: str = SchemaField(
+            description="Title for the merge commit (optional, used for merge and squash)",
+            default="",
+        )
+        commit_message: str = SchemaField(
+            description="Message for the merge commit (optional, used for merge and squash)",
+            default="",
+        )
+
+    class Output(BlockSchemaOutput):
+        sha: str = SchemaField(description="SHA of the merge commit")
+        merged: bool = SchemaField(description="Whether the PR was merged")
+        message: str = SchemaField(description="Merge status message")
+        error: str = SchemaField(description="Error message if the merge failed")
+
+    def __init__(self):
+        super().__init__(
+            id="77456c22-33d8-4fd4-9eef-50b46a35bb48",
+            description="This block merges a pull request using merge, squash, or rebase.",
+            categories={BlockCategory.DEVELOPER_TOOLS},
+            input_schema=GithubMergePullRequestBlock.Input,
+            output_schema=GithubMergePullRequestBlock.Output,
+            test_input={
+                "pr_url": "https://github.com/owner/repo/pull/1",
+                "merge_method": "squash",
+                "commit_title": "",
+                "commit_message": "",
+                "credentials": TEST_CREDENTIALS_INPUT,
+            },
+            test_credentials=TEST_CREDENTIALS,
+            test_output=[
+                ("sha", "abc123"),
+                ("merged", True),
+                ("message", "Pull Request successfully merged"),
+            ],
+            test_mock={
+                "merge_pr": lambda *args, **kwargs: (
+                    "abc123",
+                    True,
+                    "Pull Request successfully merged",
+                )
+            },
+            is_sensitive_action=True,
+        )
+
+    @staticmethod
+    async def merge_pr(
+        credentials: GithubCredentials,
+        pr_url: str,
+        merge_method: MergeMethod,
+        commit_title: str,
+        commit_message: str,
+    ) -> tuple[str, bool, str]:
+        api = get_api(credentials)
+        merge_url = prepare_pr_api_url(pr_url=pr_url, path="merge")
+        data: dict[str, str] = {"merge_method": merge_method}
+        if commit_title:
+            data["commit_title"] = commit_title
+        if commit_message:
+            data["commit_message"] = commit_message
+        response = await api.put(merge_url, json=data)
+        result = response.json()
+        return result["sha"], result["merged"], result["message"]
+
+    async def run(
+        self,
+        input_data: Input,
+        *,
+        credentials: GithubCredentials,
+        **kwargs,
+    ) -> BlockOutput:
+        try:
+            sha, merged, message = await self.merge_pr(
+                credentials,
+                input_data.pr_url,
+                input_data.merge_method,
+                input_data.commit_title,
+                input_data.commit_message,
+            )
+            yield "sha", sha
+            yield "merged", merged
+            yield "message", message
+        except Exception as e:
+            yield "error", str(e)
+
+
 def prepare_pr_api_url(pr_url: str, path: str) -> str:
     # Pattern to capture the base repository URL and the pull request number
-    pattern = r"^(?:https?://)?([^/]+/[^/]+/[^/]+)/pull/(\d+)"
+    pattern = r"^(?:(https?)://)?([^/]+/[^/]+/[^/]+)/pull/(\d+)"
     match = re.match(pattern, pr_url)
     if not match:
         return pr_url
 
-    base_url, pr_number = match.groups()
-    return f"{base_url}/pulls/{pr_number}/{path}"
+    scheme, base_url, pr_number = match.groups()
+    return f"{scheme or 'https'}://{base_url}/pulls/{pr_number}/{path}"

autogpt_platform/backend/backend/blocks/github/repo.py

@@ -1,5 +1,3 @@
-import base64
-
 from typing_extensions import TypedDict
 
 from backend.blocks._base import (
@@ -19,6 +17,7 @@ from ._auth import (
     GithubCredentialsField,
     GithubCredentialsInput,
 )
+from ._utils import github_repo_path
 
 
 class GithubListTagsBlock(Block):
@@ -89,7 +88,7 @@ class GithubListTagsBlock(Block):
         tags_url = repo_url + "/tags"
         response = await api.get(tags_url)
         data = response.json()
-        repo_path = repo_url.replace("https://github.com/", "")
+        repo_path = github_repo_path(repo_url)
         tags: list[GithubListTagsBlock.Output.TagItem] = [
             {
                 "name": tag["name"],
@@ -115,101 +114,6 @@ class GithubListTagsBlock(Block):
             yield "tag", tag
 
 
-class GithubListBranchesBlock(Block):
-    class Input(BlockSchemaInput):
-        credentials: GithubCredentialsInput = GithubCredentialsField("repo")
-        repo_url: str = SchemaField(
-            description="URL of the GitHub repository",
-            placeholder="https://github.com/owner/repo",
-        )
-
-    class Output(BlockSchemaOutput):
-        class BranchItem(TypedDict):
-            name: str
-            url: str
-
-        branch: BranchItem = SchemaField(
-            title="Branch",
-            description="Branches with their name and file tree browser URL",
-        )
-        branches: list[BranchItem] = SchemaField(
-            description="List of branches with their name and file tree browser URL"
-        )
-
-    def __init__(self):
-        super().__init__(
-            id="74243e49-2bec-4916-8bf4-db43d44aead5",
-            description="This block lists all branches for a specified GitHub repository.",
-            categories={BlockCategory.DEVELOPER_TOOLS},
-            input_schema=GithubListBranchesBlock.Input,
-            output_schema=GithubListBranchesBlock.Output,
-            test_input={
-                "repo_url": "https://github.com/owner/repo",
-                "credentials": TEST_CREDENTIALS_INPUT,
-            },
-            test_credentials=TEST_CREDENTIALS,
-            test_output=[
-                (
-                    "branches",
-                    [
-                        {
-                            "name": "main",
-                            "url": "https://github.com/owner/repo/tree/main",
-                        }
-                    ],
-                ),
-                (
-                    "branch",
-                    {
-                        "name": "main",
-                        "url": "https://github.com/owner/repo/tree/main",
-                    },
-                ),
-            ],
-            test_mock={
-                "list_branches": lambda *args, **kwargs: [
-                    {
-                        "name": "main",
-                        "url": "https://github.com/owner/repo/tree/main",
-                    }
-                ]
-            },
-        )
-
-    @staticmethod
-    async def list_branches(
-        credentials: GithubCredentials, repo_url: str
-    ) -> list[Output.BranchItem]:
-        api = get_api(credentials)
-        branches_url = repo_url + "/branches"
-        response = await api.get(branches_url)
-        data = response.json()
-        repo_path = repo_url.replace("https://github.com/", "")
-        branches: list[GithubListBranchesBlock.Output.BranchItem] = [
-            {
-                "name": branch["name"],
-                "url": f"https://github.com/{repo_path}/tree/{branch['name']}",
-            }
-            for branch in data
-        ]
-        return branches
-
-    async def run(
-        self,
-        input_data: Input,
-        *,
-        credentials: GithubCredentials,
-        **kwargs,
-    ) -> BlockOutput:
-        branches = await self.list_branches(
-            credentials,
-            input_data.repo_url,
-        )
-        yield "branches", branches
-        for branch in branches:
-            yield "branch", branch
-
-
 class GithubListDiscussionsBlock(Block):
     class Input(BlockSchemaInput):
         credentials: GithubCredentialsInput = GithubCredentialsField("repo")
@@ -283,7 +187,7 @@ class GithubListDiscussionsBlock(Block):
     ) -> list[Output.DiscussionItem]:
         api = get_api(credentials)
         # GitHub GraphQL API endpoint is different; we'll use api.post with custom URL
-        repo_path = repo_url.replace("https://github.com/", "")
+        repo_path = github_repo_path(repo_url)
         owner, repo = repo_path.split("/")
         query = """
         query($owner: String!, $repo: String!, $num: Int!) {
@@ -416,564 +320,6 @@ class GithubListReleasesBlock(Block):
             yield "release", release
 
 
-class GithubReadFileBlock(Block):
-    class Input(BlockSchemaInput):
-        credentials: GithubCredentialsInput = GithubCredentialsField("repo")
-        repo_url: str = SchemaField(
-            description="URL of the GitHub repository",
-            placeholder="https://github.com/owner/repo",
-        )
-        file_path: str = SchemaField(
-            description="Path to the file in the repository",
-            placeholder="path/to/file",
-        )
-        branch: str = SchemaField(
-            description="Branch to read from",
-            placeholder="branch_name",
-            default="master",
-        )
-
-    class Output(BlockSchemaOutput):
-        text_content: str = SchemaField(
-            description="Content of the file (decoded as UTF-8 text)"
-        )
-        raw_content: str = SchemaField(
-            description="Raw base64-encoded content of the file"
-        )
-        size: int = SchemaField(description="The size of the file (in bytes)")
-
-    def __init__(self):
-        super().__init__(
-            id="87ce6c27-5752-4bbc-8e26-6da40a3dcfd3",
-            description="This block reads the content of a specified file from a GitHub repository.",
-            categories={BlockCategory.DEVELOPER_TOOLS},
-            input_schema=GithubReadFileBlock.Input,
-            output_schema=GithubReadFileBlock.Output,
-            test_input={
-                "repo_url": "https://github.com/owner/repo",
-                "file_path": "path/to/file",
-                "branch": "master",
-                "credentials": TEST_CREDENTIALS_INPUT,
-            },
-            test_credentials=TEST_CREDENTIALS,
-            test_output=[
-                ("raw_content", "RmlsZSBjb250ZW50"),
-                ("text_content", "File content"),
-                ("size", 13),
-            ],
-            test_mock={"read_file": lambda *args, **kwargs: ("RmlsZSBjb250ZW50", 13)},
-        )
-
-    @staticmethod
-    async def read_file(
-        credentials: GithubCredentials, repo_url: str, file_path: str, branch: str
-    ) -> tuple[str, int]:
-        api = get_api(credentials)
-        content_url = repo_url + f"/contents/{file_path}?ref={branch}"
-        response = await api.get(content_url)
-        data = response.json()
-
-        if isinstance(data, list):
-            # Multiple entries of different types exist at this path
-            if not (file := next((f for f in data if f["type"] == "file"), None)):
-                raise TypeError("Not a file")
-            data = file
-
-        if data["type"] != "file":
-            raise TypeError("Not a file")
-
-        return data["content"], data["size"]
-
-    async def run(
-        self,
-        input_data: Input,
-        *,
-        credentials: GithubCredentials,
-        **kwargs,
-    ) -> BlockOutput:
-        content, size = await self.read_file(
-            credentials,
-            input_data.repo_url,
-            input_data.file_path,
-            input_data.branch,
-        )
-        yield "raw_content", content
-        yield "text_content", base64.b64decode(content).decode("utf-8")
-        yield "size", size
-
-
-class GithubReadFolderBlock(Block):
-    class Input(BlockSchemaInput):
-        credentials: GithubCredentialsInput = GithubCredentialsField("repo")
-        repo_url: str = SchemaField(
-            description="URL of the GitHub repository",
-            placeholder="https://github.com/owner/repo",
-        )
-        folder_path: str = SchemaField(
-            description="Path to the folder in the repository",
-            placeholder="path/to/folder",
-        )
-        branch: str = SchemaField(
-            description="Branch name to read from (defaults to master)",
-            placeholder="branch_name",
-            default="master",
-        )
-
-    class Output(BlockSchemaOutput):
-        class DirEntry(TypedDict):
-            name: str
-            path: str
-
-        class FileEntry(TypedDict):
-            name: str
-            path: str
-            size: int
-
-        file: FileEntry = SchemaField(description="Files in the folder")
-        dir: DirEntry = SchemaField(description="Directories in the folder")
-        error: str = SchemaField(
-            description="Error message if reading the folder failed"
-        )
-
-    def __init__(self):
-        super().__init__(
-            id="1355f863-2db3-4d75-9fba-f91e8a8ca400",
-            description="This block reads the content of a specified folder from a GitHub repository.",
-            categories={BlockCategory.DEVELOPER_TOOLS},
-            input_schema=GithubReadFolderBlock.Input,
-            output_schema=GithubReadFolderBlock.Output,
-            test_input={
-                "repo_url": "https://github.com/owner/repo",
-                "folder_path": "path/to/folder",
-                "branch": "master",
-                "credentials": TEST_CREDENTIALS_INPUT,
-            },
-            test_credentials=TEST_CREDENTIALS,
-            test_output=[
-                (
-                    "file",
-                    {
-                        "name": "file1.txt",
-                        "path": "path/to/folder/file1.txt",
-                        "size": 1337,
-                    },
-                ),
-                ("dir", {"name": "dir2", "path": "path/to/folder/dir2"}),
-            ],
-            test_mock={
-                "read_folder": lambda *args, **kwargs: (
-                    [
-                        {
-                            "name": "file1.txt",
-                            "path": "path/to/folder/file1.txt",
-                            "size": 1337,
-                        }
-                    ],
-                    [{"name": "dir2", "path": "path/to/folder/dir2"}],
-                )
-            },
-        )
-
-    @staticmethod
-    async def read_folder(
-        credentials: GithubCredentials, repo_url: str, folder_path: str, branch: str
-    ) -> tuple[list[Output.FileEntry], list[Output.DirEntry]]:
-        api = get_api(credentials)
-        contents_url = repo_url + f"/contents/{folder_path}?ref={branch}"
-        response = await api.get(contents_url)
-        data = response.json()
-
-        if not isinstance(data, list):
-            raise TypeError("Not a folder")
-
-        files: list[GithubReadFolderBlock.Output.FileEntry] = [
-            GithubReadFolderBlock.Output.FileEntry(
-                name=entry["name"],
-                path=entry["path"],
-                size=entry["size"],
-            )
-            for entry in data
-            if entry["type"] == "file"
-        ]
-
-        dirs: list[GithubReadFolderBlock.Output.DirEntry] = [
-            GithubReadFolderBlock.Output.DirEntry(
-                name=entry["name"],
-                path=entry["path"],
-            )
-            for entry in data
-            if entry["type"] == "dir"
-        ]
-
-        return files, dirs
-
-    async def run(
-        self,
-        input_data: Input,
-        *,
-        credentials: GithubCredentials,
-        **kwargs,
-    ) -> BlockOutput:
-        files, dirs = await self.read_folder(
-            credentials,
-            input_data.repo_url,
-            input_data.folder_path.lstrip("/"),
-            input_data.branch,
-        )
-        for file in files:
-            yield "file", file
-        for dir in dirs:
-            yield "dir", dir
-
-
-class GithubMakeBranchBlock(Block):
-    class Input(BlockSchemaInput):
-        credentials: GithubCredentialsInput = GithubCredentialsField("repo")
-        repo_url: str = SchemaField(
-            description="URL of the GitHub repository",
-            placeholder="https://github.com/owner/repo",
-        )
-        new_branch: str = SchemaField(
-            description="Name of the new branch",
-            placeholder="new_branch_name",
-        )
-        source_branch: str = SchemaField(
-            description="Name of the source branch",
-            placeholder="source_branch_name",
-        )
-
-    class Output(BlockSchemaOutput):
-        status: str = SchemaField(description="Status of the branch creation operation")
-        error: str = SchemaField(
-            description="Error message if the branch creation failed"
-        )
-
-    def __init__(self):
-        super().__init__(
-            id="944cc076-95e7-4d1b-b6b6-b15d8ee5448d",
-            description="This block creates a new branch from a specified source branch.",
-            categories={BlockCategory.DEVELOPER_TOOLS},
-            input_schema=GithubMakeBranchBlock.Input,
-            output_schema=GithubMakeBranchBlock.Output,
-            test_input={
-                "repo_url": "https://github.com/owner/repo",
-                "new_branch": "new_branch_name",
-                "source_branch": "source_branch_name",
-                "credentials": TEST_CREDENTIALS_INPUT,
-            },
-            test_credentials=TEST_CREDENTIALS,
-            test_output=[("status", "Branch created successfully")],
-            test_mock={
-                "create_branch": lambda *args, **kwargs: "Branch created successfully"
-            },
-        )
-
-    @staticmethod
-    async def create_branch(
-        credentials: GithubCredentials,
-        repo_url: str,
-        new_branch: str,
-        source_branch: str,
-    ) -> str:
-        api = get_api(credentials)
-        ref_url = repo_url + f"/git/refs/heads/{source_branch}"
-        response = await api.get(ref_url)
-        data = response.json()
-        sha = data["object"]["sha"]
-
-        # Create the new branch
-        new_ref_url = repo_url + "/git/refs"
-        data = {
-            "ref": f"refs/heads/{new_branch}",
-            "sha": sha,
-        }
-        response = await api.post(new_ref_url, json=data)
-        return "Branch created successfully"
-
-    async def run(
-        self,
-        input_data: Input,
-        *,
-        credentials: GithubCredentials,
-        **kwargs,
-    ) -> BlockOutput:
-        status = await self.create_branch(
-            credentials,
-            input_data.repo_url,
-            input_data.new_branch,
-            input_data.source_branch,
-        )
-        yield "status", status
-
-
-class GithubDeleteBranchBlock(Block):
-    class Input(BlockSchemaInput):
-        credentials: GithubCredentialsInput = GithubCredentialsField("repo")
-        repo_url: str = SchemaField(
-            description="URL of the GitHub repository",
-            placeholder="https://github.com/owner/repo",
-        )
-        branch: str = SchemaField(
-            description="Name of the branch to delete",
-            placeholder="branch_name",
-        )
-
-    class Output(BlockSchemaOutput):
-        status: str = SchemaField(description="Status of the branch deletion operation")
-        error: str = SchemaField(
-            description="Error message if the branch deletion failed"
-        )
-
-    def __init__(self):
-        super().__init__(
-            id="0d4130f7-e0ab-4d55-adc3-0a40225e80f4",
-            description="This block deletes a specified branch.",
-            categories={BlockCategory.DEVELOPER_TOOLS},
-            input_schema=GithubDeleteBranchBlock.Input,
-            output_schema=GithubDeleteBranchBlock.Output,
-            test_input={
-                "repo_url": "https://github.com/owner/repo",
-                "branch": "branch_name",
-                "credentials": TEST_CREDENTIALS_INPUT,
-            },
-            test_credentials=TEST_CREDENTIALS,
-            test_output=[("status", "Branch deleted successfully")],
-            test_mock={
-                "delete_branch": lambda *args, **kwargs: "Branch deleted successfully"
-            },
-        )
-
-    @staticmethod
-    async def delete_branch(
-        credentials: GithubCredentials, repo_url: str, branch: str
-    ) -> str:
-        api = get_api(credentials)
-        ref_url = repo_url + f"/git/refs/heads/{branch}"
-        await api.delete(ref_url)
-        return "Branch deleted successfully"
-
-    async def run(
-        self,
-        input_data: Input,
-        *,
-        credentials: GithubCredentials,
-        **kwargs,
-    ) -> BlockOutput:
-        status = await self.delete_branch(
-            credentials,
-            input_data.repo_url,
-            input_data.branch,
-        )
-        yield "status", status
-
-
-class GithubCreateFileBlock(Block):
-    class Input(BlockSchemaInput):
-        credentials: GithubCredentialsInput = GithubCredentialsField("repo")
-        repo_url: str = SchemaField(
-            description="URL of the GitHub repository",
-            placeholder="https://github.com/owner/repo",
-        )
-        file_path: str = SchemaField(
-            description="Path where the file should be created",
-            placeholder="path/to/file.txt",
-        )
-        content: str = SchemaField(
-            description="Content to write to the file",
-            placeholder="File content here",
-        )
-        branch: str = SchemaField(
-            description="Branch where the file should be created",
-            default="main",
-        )
-        commit_message: str = SchemaField(
-            description="Message for the commit",
-            default="Create new file",
-        )
-
-    class Output(BlockSchemaOutput):
-        url: str = SchemaField(description="URL of the created file")
-        sha: str = SchemaField(description="SHA of the commit")
-        error: str = SchemaField(
-            description="Error message if the file creation failed"
-        )
-
-    def __init__(self):
-        super().__init__(
-            id="8fd132ac-b917-428a-8159-d62893e8a3fe",
-            description="This block creates a new file in a GitHub repository.",
-            categories={BlockCategory.DEVELOPER_TOOLS},
-            input_schema=GithubCreateFileBlock.Input,
-            output_schema=GithubCreateFileBlock.Output,
-            test_input={
-                "repo_url": "https://github.com/owner/repo",
-                "file_path": "test/file.txt",
-                "content": "Test content",
-                "branch": "main",
-                "commit_message": "Create test file",
-                "credentials": TEST_CREDENTIALS_INPUT,
-            },
-            test_credentials=TEST_CREDENTIALS,
-            test_output=[
-                ("url", "https://github.com/owner/repo/blob/main/test/file.txt"),
-                ("sha", "abc123"),
-            ],
-            test_mock={
-                "create_file": lambda *args, **kwargs: (
-                    "https://github.com/owner/repo/blob/main/test/file.txt",
-                    "abc123",
-                )
-            },
-        )
-
-    @staticmethod
-    async def create_file(
-        credentials: GithubCredentials,
-        repo_url: str,
-        file_path: str,
-        content: str,
-        branch: str,
-        commit_message: str,
-    ) -> tuple[str, str]:
-        api = get_api(credentials)
-        contents_url = repo_url + f"/contents/{file_path}"
-        content_base64 = base64.b64encode(content.encode()).decode()
-        data = {
-            "message": commit_message,
-            "content": content_base64,
-            "branch": branch,
-        }
-        response = await api.put(contents_url, json=data)
-        data = response.json()
-        return data["content"]["html_url"], data["commit"]["sha"]
-
-    async def run(
-        self,
-        input_data: Input,
-        *,
-        credentials: GithubCredentials,
-        **kwargs,
-    ) -> BlockOutput:
-        try:
-            url, sha = await self.create_file(
-                credentials,
-                input_data.repo_url,
-                input_data.file_path,
-                input_data.content,
-                input_data.branch,
-                input_data.commit_message,
-            )
-            yield "url", url
-            yield "sha", sha
-        except Exception as e:
-            yield "error", str(e)
-
-
-class GithubUpdateFileBlock(Block):
-    class Input(BlockSchemaInput):
-        credentials: GithubCredentialsInput = GithubCredentialsField("repo")
-        repo_url: str = SchemaField(
-            description="URL of the GitHub repository",
-            placeholder="https://github.com/owner/repo",
-        )
-        file_path: str = SchemaField(
-            description="Path to the file to update",
-            placeholder="path/to/file.txt",
-        )
-        content: str = SchemaField(
-            description="New content for the file",
-            placeholder="Updated content here",
-        )
-        branch: str = SchemaField(
-            description="Branch containing the file",
-            default="main",
-        )
-        commit_message: str = SchemaField(
-            description="Message for the commit",
-            default="Update file",
-        )
-
-    class Output(BlockSchemaOutput):
-        url: str = SchemaField(description="URL of the updated file")
-        sha: str = SchemaField(description="SHA of the commit")
-
-    def __init__(self):
-        super().__init__(
-            id="30be12a4-57cb-4aa4-baf5-fcc68d136076",
-            description="This block updates an existing file in a GitHub repository.",
-            categories={BlockCategory.DEVELOPER_TOOLS},
-            input_schema=GithubUpdateFileBlock.Input,
-            output_schema=GithubUpdateFileBlock.Output,
-            test_input={
-                "repo_url": "https://github.com/owner/repo",
-                "file_path": "test/file.txt",
-                "content": "Updated content",
-                "branch": "main",
-                "commit_message": "Update test file",
-                "credentials": TEST_CREDENTIALS_INPUT,
-            },
-            test_credentials=TEST_CREDENTIALS,
-            test_output=[
-                ("url", "https://github.com/owner/repo/blob/main/test/file.txt"),
-                ("sha", "def456"),
-            ],
-            test_mock={
-                "update_file": lambda *args, **kwargs: (
-                    "https://github.com/owner/repo/blob/main/test/file.txt",
-                    "def456",
-                )
-            },
-        )
-
-    @staticmethod
-    async def update_file(
-        credentials: GithubCredentials,
-        repo_url: str,
-        file_path: str,
-        content: str,
-        branch: str,
-        commit_message: str,
-    ) -> tuple[str, str]:
-        api = get_api(credentials)
-        contents_url = repo_url + f"/contents/{file_path}"
-        params = {"ref": branch}
-        response = await api.get(contents_url, params=params)
-        data = response.json()
-
-        # Convert new content to base64
-        content_base64 = base64.b64encode(content.encode()).decode()
-        data = {
-            "message": commit_message,
-            "content": content_base64,
-            "sha": data["sha"],
-            "branch": branch,
-        }
-        response = await api.put(contents_url, json=data)
-        data = response.json()
-        return data["content"]["html_url"], data["commit"]["sha"]
-
-    async def run(
-        self,
-        input_data: Input,
-        *,
-        credentials: GithubCredentials,
-        **kwargs,
-    ) -> BlockOutput:
-        try:
-            url, sha = await self.update_file(
-                credentials,
-                input_data.repo_url,
-                input_data.file_path,
-                input_data.content,
-                input_data.branch,
-                input_data.commit_message,
-            )
-            yield "url", url
-            yield "sha", sha
-        except Exception as e:
-            yield "error", str(e)
-
-
 class GithubCreateRepositoryBlock(Block):
     class Input(BlockSchemaInput):
         credentials: GithubCredentialsInput = GithubCredentialsField("repo")
@@ -1103,7 +449,7 @@ class GithubListStargazersBlock(Block):
 
     def __init__(self):
         super().__init__(
-            id="a4b9c2d1-e5f6-4g7h-8i9j-0k1l2m3n4o5p",  # Generated unique UUID
+            id="e96d01ec-b55e-4a99-8ce8-c8776dce850b",  # Generated unique UUID
             description="This block lists all users who have starred a specified GitHub repository.",
             categories={BlockCategory.DEVELOPER_TOOLS},
             input_schema=GithubListStargazersBlock.Input,
@@ -1172,3 +518,230 @@ class GithubListStargazersBlock(Block):
         yield "stargazers", stargazers
         for stargazer in stargazers:
             yield "stargazer", stargazer
+
+
+class GithubGetRepositoryInfoBlock(Block):
+    class Input(BlockSchemaInput):
+        credentials: GithubCredentialsInput = GithubCredentialsField("repo")
+        repo_url: str = SchemaField(
+            description="URL of the GitHub repository",
+            placeholder="https://github.com/owner/repo",
+        )
+
+    class Output(BlockSchemaOutput):
+        name: str = SchemaField(description="Repository name")
+        full_name: str = SchemaField(description="Full repository name (owner/repo)")
+        description: str = SchemaField(description="Repository description")
+        default_branch: str = SchemaField(description="Default branch name (e.g. main)")
+        private: bool = SchemaField(description="Whether the repository is private")
+        html_url: str = SchemaField(description="Web URL of the repository")
+        clone_url: str = SchemaField(description="Git clone URL")
+        stars: int = SchemaField(description="Number of stars")
+        forks: int = SchemaField(description="Number of forks")
+        open_issues: int = SchemaField(description="Number of open issues")
+        error: str = SchemaField(
+            description="Error message if fetching repo info failed"
+        )
+
+    def __init__(self):
+        super().__init__(
+            id="59d4f241-968a-4040-95da-348ac5c5ce27",
+            description="This block retrieves metadata about a GitHub repository.",
+            categories={BlockCategory.DEVELOPER_TOOLS},
+            input_schema=GithubGetRepositoryInfoBlock.Input,
+            output_schema=GithubGetRepositoryInfoBlock.Output,
+            test_input={
+                "repo_url": "https://github.com/owner/repo",
+                "credentials": TEST_CREDENTIALS_INPUT,
+            },
+            test_credentials=TEST_CREDENTIALS,
+            test_output=[
+                ("name", "repo"),
+                ("full_name", "owner/repo"),
+                ("description", "A test repo"),
+                ("default_branch", "main"),
+                ("private", False),
+                ("html_url", "https://github.com/owner/repo"),
+                ("clone_url", "https://github.com/owner/repo.git"),
+                ("stars", 42),
+                ("forks", 5),
+                ("open_issues", 3),
+            ],
+            test_mock={
+                "get_repo_info": lambda *args, **kwargs: {
+                    "name": "repo",
+                    "full_name": "owner/repo",
+                    "description": "A test repo",
+                    "default_branch": "main",
+                    "private": False,
+                    "html_url": "https://github.com/owner/repo",
+                    "clone_url": "https://github.com/owner/repo.git",
+                    "stargazers_count": 42,
+                    "forks_count": 5,
+                    "open_issues_count": 3,
+                }
+            },
+        )
+
+    @staticmethod
+    async def get_repo_info(credentials: GithubCredentials, repo_url: str) -> dict:
+        api = get_api(credentials)
+        response = await api.get(repo_url)
+        return response.json()
+
+    async def run(
+        self,
+        input_data: Input,
+        *,
+        credentials: GithubCredentials,
+        **kwargs,
+    ) -> BlockOutput:
+        try:
+            data = await self.get_repo_info(credentials, input_data.repo_url)
+            yield "name", data["name"]
+            yield "full_name", data["full_name"]
+            yield "description", data.get("description", "") or ""
+            yield "default_branch", data["default_branch"]
+            yield "private", data["private"]
+            yield "html_url", data["html_url"]
+            yield "clone_url", data["clone_url"]
+            yield "stars", data["stargazers_count"]
+            yield "forks", data["forks_count"]
+            yield "open_issues", data["open_issues_count"]
+        except Exception as e:
+            yield "error", str(e)
+
+
+class GithubForkRepositoryBlock(Block):
+    class Input(BlockSchemaInput):
+        credentials: GithubCredentialsInput = GithubCredentialsField("repo")
+        repo_url: str = SchemaField(
+            description="URL of the GitHub repository to fork",
+            placeholder="https://github.com/owner/repo",
+        )
+        organization: str = SchemaField(
+            description="Organization to fork into (leave empty to fork to your account)",
+            default="",
+        )
+
+    class Output(BlockSchemaOutput):
+        url: str = SchemaField(description="URL of the forked repository")
+        clone_url: str = SchemaField(description="Git clone URL of the fork")
+        full_name: str = SchemaField(description="Full name of the fork (owner/repo)")
+        error: str = SchemaField(description="Error message if the fork failed")
+
+    def __init__(self):
+        super().__init__(
+            id="a439f2f4-835f-4dae-ba7b-0205ffa70be6",
+            description="This block forks a GitHub repository to your account or an organization.",
+            categories={BlockCategory.DEVELOPER_TOOLS},
+            input_schema=GithubForkRepositoryBlock.Input,
+            output_schema=GithubForkRepositoryBlock.Output,
+            test_input={
+                "repo_url": "https://github.com/owner/repo",
+                "organization": "",
+                "credentials": TEST_CREDENTIALS_INPUT,
+            },
+            test_credentials=TEST_CREDENTIALS,
+            test_output=[
+                ("url", "https://github.com/myuser/repo"),
+                ("clone_url", "https://github.com/myuser/repo.git"),
+                ("full_name", "myuser/repo"),
+            ],
+            test_mock={
+                "fork_repo": lambda *args, **kwargs: (
+                    "https://github.com/myuser/repo",
+                    "https://github.com/myuser/repo.git",
+                    "myuser/repo",
+                )
+            },
+        )
+
+    @staticmethod
+    async def fork_repo(
+        credentials: GithubCredentials,
+        repo_url: str,
+        organization: str,
+    ) -> tuple[str, str, str]:
+        api = get_api(credentials)
+        forks_url = repo_url + "/forks"
+        data: dict[str, str] = {}
+        if organization:
+            data["organization"] = organization
+        response = await api.post(forks_url, json=data)
+        result = response.json()
+        return result["html_url"], result["clone_url"], result["full_name"]
+
+    async def run(
+        self,
+        input_data: Input,
+        *,
+        credentials: GithubCredentials,
+        **kwargs,
+    ) -> BlockOutput:
+        try:
+            url, clone_url, full_name = await self.fork_repo(
+                credentials,
+                input_data.repo_url,
+                input_data.organization,
+            )
+            yield "url", url
+            yield "clone_url", clone_url
+            yield "full_name", full_name
+        except Exception as e:
+            yield "error", str(e)
+
+
+class GithubStarRepositoryBlock(Block):
+    class Input(BlockSchemaInput):
+        credentials: GithubCredentialsInput = GithubCredentialsField("repo")
+        repo_url: str = SchemaField(
+            description="URL of the GitHub repository to star",
+            placeholder="https://github.com/owner/repo",
+        )
+
+    class Output(BlockSchemaOutput):
+        status: str = SchemaField(description="Status of the star operation")
+        error: str = SchemaField(description="Error message if starring failed")
+
+    def __init__(self):
+        super().__init__(
+            id="bd700764-53e3-44dd-a969-d1854088458f",
+            description="This block stars a GitHub repository.",
+            categories={BlockCategory.DEVELOPER_TOOLS},
+            input_schema=GithubStarRepositoryBlock.Input,
+            output_schema=GithubStarRepositoryBlock.Output,
+            test_input={
+                "repo_url": "https://github.com/owner/repo",
+                "credentials": TEST_CREDENTIALS_INPUT,
+            },
+            test_credentials=TEST_CREDENTIALS,
+            test_output=[("status", "Repository starred successfully")],
+            test_mock={
+                "star_repo": lambda *args, **kwargs: "Repository starred successfully"
+            },
+        )
+
+    @staticmethod
+    async def star_repo(credentials: GithubCredentials, repo_url: str) -> str:
+        api = get_api(credentials, convert_urls=False)
+        repo_path = github_repo_path(repo_url)
+        owner, repo = repo_path.split("/")
+        await api.put(
+            f"https://api.github.com/user/starred/{owner}/{repo}",
+            headers={"Content-Length": "0"},
+        )
+        return "Repository starred successfully"
+
+    async def run(
+        self,
+        input_data: Input,
+        *,
+        credentials: GithubCredentials,
+        **kwargs,
+    ) -> BlockOutput:
+        try:
+            status = await self.star_repo(credentials, input_data.repo_url)
+            yield "status", status
+        except Exception as e:
+            yield "error", str(e)

autogpt_platform/backend/backend/blocks/github/repo_branches.py

@@ -0,0 +1,452 @@
+from urllib.parse import quote
+
+from typing_extensions import TypedDict
+
+from backend.blocks._base import (
+    Block,
+    BlockCategory,
+    BlockOutput,
+    BlockSchemaInput,
+    BlockSchemaOutput,
+)
+from backend.data.model import SchemaField
+
+from ._api import get_api
+from ._auth import (
+    TEST_CREDENTIALS,
+    TEST_CREDENTIALS_INPUT,
+    GithubCredentials,
+    GithubCredentialsField,
+    GithubCredentialsInput,
+)
+from ._utils import github_repo_path
+
+
+class GithubListBranchesBlock(Block):
+    class Input(BlockSchemaInput):
+        credentials: GithubCredentialsInput = GithubCredentialsField("repo")
+        repo_url: str = SchemaField(
+            description="URL of the GitHub repository",
+            placeholder="https://github.com/owner/repo",
+        )
+        per_page: int = SchemaField(
+            description="Number of branches to return per page (max 100)",
+            default=30,
+            ge=1,
+            le=100,
+        )
+        page: int = SchemaField(
+            description="Page number for pagination",
+            default=1,
+            ge=1,
+        )
+
+    class Output(BlockSchemaOutput):
+        class BranchItem(TypedDict):
+            name: str
+            url: str
+
+        branch: BranchItem = SchemaField(
+            title="Branch",
+            description="Branches with their name and file tree browser URL",
+        )
+        branches: list[BranchItem] = SchemaField(
+            description="List of branches with their name and file tree browser URL"
+        )
+        error: str = SchemaField(description="Error message if listing branches failed")
+
+    def __init__(self):
+        super().__init__(
+            id="74243e49-2bec-4916-8bf4-db43d44aead5",
+            description="This block lists all branches for a specified GitHub repository.",
+            categories={BlockCategory.DEVELOPER_TOOLS},
+            input_schema=GithubListBranchesBlock.Input,
+            output_schema=GithubListBranchesBlock.Output,
+            test_input={
+                "repo_url": "https://github.com/owner/repo",
+                "per_page": 30,
+                "page": 1,
+                "credentials": TEST_CREDENTIALS_INPUT,
+            },
+            test_credentials=TEST_CREDENTIALS,
+            test_output=[
+                (
+                    "branches",
+                    [
+                        {
+                            "name": "main",
+                            "url": "https://github.com/owner/repo/tree/main",
+                        }
+                    ],
+                ),
+                (
+                    "branch",
+                    {
+                        "name": "main",
+                        "url": "https://github.com/owner/repo/tree/main",
+                    },
+                ),
+            ],
+            test_mock={
+                "list_branches": lambda *args, **kwargs: [
+                    {
+                        "name": "main",
+                        "url": "https://github.com/owner/repo/tree/main",
+                    }
+                ]
+            },
+        )
+
+    @staticmethod
+    async def list_branches(
+        credentials: GithubCredentials, repo_url: str, per_page: int, page: int
+    ) -> list[Output.BranchItem]:
+        api = get_api(credentials)
+        branches_url = repo_url + "/branches"
+        response = await api.get(
+            branches_url, params={"per_page": str(per_page), "page": str(page)}
+        )
+        data = response.json()
+        repo_path = github_repo_path(repo_url)
+        branches: list[GithubListBranchesBlock.Output.BranchItem] = [
+            {
+                "name": branch["name"],
+                "url": f"https://github.com/{repo_path}/tree/{branch['name']}",
+            }
+            for branch in data
+        ]
+        return branches
+
+    async def run(
+        self,
+        input_data: Input,
+        *,
+        credentials: GithubCredentials,
+        **kwargs,
+    ) -> BlockOutput:
+        try:
+            branches = await self.list_branches(
+                credentials,
+                input_data.repo_url,
+                input_data.per_page,
+                input_data.page,
+            )
+            yield "branches", branches
+            for branch in branches:
+                yield "branch", branch
+        except Exception as e:
+            yield "error", str(e)
+
+
+class GithubMakeBranchBlock(Block):
+    class Input(BlockSchemaInput):
+        credentials: GithubCredentialsInput = GithubCredentialsField("repo")
+        repo_url: str = SchemaField(
+            description="URL of the GitHub repository",
+            placeholder="https://github.com/owner/repo",
+        )
+        new_branch: str = SchemaField(
+            description="Name of the new branch",
+            placeholder="new_branch_name",
+        )
+        source_branch: str = SchemaField(
+            description="Name of the source branch",
+            placeholder="source_branch_name",
+        )
+
+    class Output(BlockSchemaOutput):
+        status: str = SchemaField(description="Status of the branch creation operation")
+        error: str = SchemaField(
+            description="Error message if the branch creation failed"
+        )
+
+    def __init__(self):
+        super().__init__(
+            id="944cc076-95e7-4d1b-b6b6-b15d8ee5448d",
+            description="This block creates a new branch from a specified source branch.",
+            categories={BlockCategory.DEVELOPER_TOOLS},
+            input_schema=GithubMakeBranchBlock.Input,
+            output_schema=GithubMakeBranchBlock.Output,
+            test_input={
+                "repo_url": "https://github.com/owner/repo",
+                "new_branch": "new_branch_name",
+                "source_branch": "source_branch_name",
+                "credentials": TEST_CREDENTIALS_INPUT,
+            },
+            test_credentials=TEST_CREDENTIALS,
+            test_output=[("status", "Branch created successfully")],
+            test_mock={
+                "create_branch": lambda *args, **kwargs: "Branch created successfully"
+            },
+        )
+
+    @staticmethod
+    async def create_branch(
+        credentials: GithubCredentials,
+        repo_url: str,
+        new_branch: str,
+        source_branch: str,
+    ) -> str:
+        api = get_api(credentials)
+        ref_url = repo_url + f"/git/refs/heads/{quote(source_branch, safe='')}"
+        response = await api.get(ref_url)
+        data = response.json()
+        sha = data["object"]["sha"]
+
+        # Create the new branch
+        new_ref_url = repo_url + "/git/refs"
+        data = {
+            "ref": f"refs/heads/{new_branch}",
+            "sha": sha,
+        }
+        response = await api.post(new_ref_url, json=data)
+        return "Branch created successfully"
+
+    async def run(
+        self,
+        input_data: Input,
+        *,
+        credentials: GithubCredentials,
+        **kwargs,
+    ) -> BlockOutput:
+        try:
+            status = await self.create_branch(
+                credentials,
+                input_data.repo_url,
+                input_data.new_branch,
+                input_data.source_branch,
+            )
+            yield "status", status
+        except Exception as e:
+            yield "error", str(e)
+
+
+class GithubDeleteBranchBlock(Block):
+    class Input(BlockSchemaInput):
+        credentials: GithubCredentialsInput = GithubCredentialsField("repo")
+        repo_url: str = SchemaField(
+            description="URL of the GitHub repository",
+            placeholder="https://github.com/owner/repo",
+        )
+        branch: str = SchemaField(
+            description="Name of the branch to delete",
+            placeholder="branch_name",
+        )
+
+    class Output(BlockSchemaOutput):
+        status: str = SchemaField(description="Status of the branch deletion operation")
+        error: str = SchemaField(
+            description="Error message if the branch deletion failed"
+        )
+
+    def __init__(self):
+        super().__init__(
+            id="0d4130f7-e0ab-4d55-adc3-0a40225e80f4",
+            description="This block deletes a specified branch.",
+            categories={BlockCategory.DEVELOPER_TOOLS},
+            input_schema=GithubDeleteBranchBlock.Input,
+            output_schema=GithubDeleteBranchBlock.Output,
+            test_input={
+                "repo_url": "https://github.com/owner/repo",
+                "branch": "branch_name",
+                "credentials": TEST_CREDENTIALS_INPUT,
+            },
+            test_credentials=TEST_CREDENTIALS,
+            test_output=[("status", "Branch deleted successfully")],
+            test_mock={
+                "delete_branch": lambda *args, **kwargs: "Branch deleted successfully"
+            },
+            is_sensitive_action=True,
+        )
+
+    @staticmethod
+    async def delete_branch(
+        credentials: GithubCredentials, repo_url: str, branch: str
+    ) -> str:
+        api = get_api(credentials)
+        ref_url = repo_url + f"/git/refs/heads/{quote(branch, safe='')}"
+        await api.delete(ref_url)
+        return "Branch deleted successfully"
+
+    async def run(
+        self,
+        input_data: Input,
+        *,
+        credentials: GithubCredentials,
+        **kwargs,
+    ) -> BlockOutput:
+        try:
+            status = await self.delete_branch(
+                credentials,
+                input_data.repo_url,
+                input_data.branch,
+            )
+            yield "status", status
+        except Exception as e:
+            yield "error", str(e)
+
+
+class GithubCompareBranchesBlock(Block):
+    class Input(BlockSchemaInput):
+        credentials: GithubCredentialsInput = GithubCredentialsField("repo")
+        repo_url: str = SchemaField(
+            description="URL of the GitHub repository",
+            placeholder="https://github.com/owner/repo",
+        )
+        base: str = SchemaField(
+            description="Base branch or commit SHA",
+            placeholder="main",
+        )
+        head: str = SchemaField(
+            description="Head branch or commit SHA to compare against base",
+            placeholder="feature-branch",
+        )
+
+    class Output(BlockSchemaOutput):
+        class FileChange(TypedDict):
+            filename: str
+            status: str
+            additions: int
+            deletions: int
+            patch: str
+
+        status: str = SchemaField(
+            description="Comparison status: ahead, behind, diverged, or identical"
+        )
+        ahead_by: int = SchemaField(
+            description="Number of commits head is ahead of base"
+        )
+        behind_by: int = SchemaField(
+            description="Number of commits head is behind base"
+        )
+        total_commits: int = SchemaField(
+            description="Total number of commits in the comparison"
+        )
+        diff: str = SchemaField(description="Unified diff of all file changes")
+        file: FileChange = SchemaField(
+            title="Changed File", description="A changed file with its diff"
+        )
+        files: list[FileChange] = SchemaField(
+            description="List of changed files with their diffs"
+        )
+        error: str = SchemaField(description="Error message if comparison failed")
+
+    def __init__(self):
+        super().__init__(
+            id="2e4faa8c-6086-4546-ba77-172d1d560186",
+            description="This block compares two branches or commits in a GitHub repository.",
+            categories={BlockCategory.DEVELOPER_TOOLS},
+            input_schema=GithubCompareBranchesBlock.Input,
+            output_schema=GithubCompareBranchesBlock.Output,
+            test_input={
+                "repo_url": "https://github.com/owner/repo",
+                "base": "main",
+                "head": "feature",
+                "credentials": TEST_CREDENTIALS_INPUT,
+            },
+            test_credentials=TEST_CREDENTIALS,
+            test_output=[
+                ("status", "ahead"),
+                ("ahead_by", 2),
+                ("behind_by", 0),
+                ("total_commits", 2),
+                ("diff", "+++ b/file.py\n+new line"),
+                (
+                    "files",
+                    [
+                        {
+                            "filename": "file.py",
+                            "status": "modified",
+                            "additions": 1,
+                            "deletions": 0,
+                            "patch": "+new line",
+                        }
+                    ],
+                ),
+                (
+                    "file",
+                    {
+                        "filename": "file.py",
+                        "status": "modified",
+                        "additions": 1,
+                        "deletions": 0,
+                        "patch": "+new line",
+                    },
+                ),
+            ],
+            test_mock={
+                "compare_branches": lambda *args, **kwargs: {
+                    "status": "ahead",
+                    "ahead_by": 2,
+                    "behind_by": 0,
+                    "total_commits": 2,
+                    "files": [
+                        {
+                            "filename": "file.py",
+                            "status": "modified",
+                            "additions": 1,
+                            "deletions": 0,
+                            "patch": "+new line",
+                        }
+                    ],
+                }
+            },
+        )
+
+    @staticmethod
+    async def compare_branches(
+        credentials: GithubCredentials,
+        repo_url: str,
+        base: str,
+        head: str,
+    ) -> dict:
+        api = get_api(credentials)
+        safe_base = quote(base, safe="")
+        safe_head = quote(head, safe="")
+        compare_url = repo_url + f"/compare/{safe_base}...{safe_head}"
+        response = await api.get(compare_url)
+        return response.json()
+
+    async def run(
+        self,
+        input_data: Input,
+        *,
+        credentials: GithubCredentials,
+        **kwargs,
+    ) -> BlockOutput:
+        try:
+            data = await self.compare_branches(
+                credentials,
+                input_data.repo_url,
+                input_data.base,
+                input_data.head,
+            )
+            yield "status", data["status"]
+            yield "ahead_by", data["ahead_by"]
+            yield "behind_by", data["behind_by"]
+            yield "total_commits", data["total_commits"]
+
+            files: list[GithubCompareBranchesBlock.Output.FileChange] = [
+                GithubCompareBranchesBlock.Output.FileChange(
+                    filename=f["filename"],
+                    status=f["status"],
+                    additions=f["additions"],
+                    deletions=f["deletions"],
+                    patch=f.get("patch", ""),
+                )
+                for f in data.get("files", [])
+            ]
+
+            # Build unified diff
+            diff_parts = []
+            for f in data.get("files", []):
+                patch = f.get("patch", "")
+                if patch:
+                    diff_parts.append(f"+++ b/{f['filename']}\n{patch}")
+            yield "diff", "\n".join(diff_parts)
+
+            yield "files", files
+            for file in files:
+                yield "file", file
+        except Exception as e:
+            yield "error", str(e)