Add early return when user has no stripe_customer_id to prevent creating
an orphaned Stripe customer if a subsequent Subscription.list call fails.
Follows the same pattern as cancel_stripe_subscription and
get_proration_credit_cents. Update tests to mock get_user_by_id and add
a test for the no-customer-id path.
## Why
When a user switches from **baseline** (fast) mode to **SDK**
(extended_thinking) mode mid-session, every subsequent SDK turn started
fresh with no memory of prior conversation.
Root cause: two complementary bugs on mode-switch T1 (first SDK turn
after baseline turns):
1. `session_id` was gated on `not has_history`. On mode-switch T1,
`has_history=True` (prior baseline turns in DB) so no `session_id` was
set. The CLI generated a random ID and could not upload the session file
under a predictable path → `--resume` failed on every following SDK
turn.
2. Even if `session_id` were set, the upload guard `(not has_history or
state.use_resume)` would block the session file upload on mode-switch T1
(`has_history=True`, `use_resume=False`), so the next turn still cannot
`--resume`.
Together these caused every SDK turn to re-inject the full compressed
history, causing model confusion (proactive tool calls, forgetting
context) observed in session `8237a27b-45d0-4688-af20-c185379e926f`.
## What
- **`service.py`**: Change `elif not has_history:` → `else:` for the
`session_id` assignment — set it whenever `--resume` is not active.
Covers T1 fresh, mode-switch T1 (`has_history=True` but no CLI session
exists), and T2+ fallback turns where restore failed.
- **`service.py` retry path**: Replace `not has_history` with
`"session_id" in sdk_options_kwargs` as the discriminator, so
mode-switch T1 retries also keep `session_id` while T2+ retries (where
`restore_cli_session` put a file on disk) correctly remove it to avoid
"Session ID already in use".
- **`service.py` upload guard**: Remove `and not skip_transcript_upload`
and `and (not has_history or state.use_resume)` from the
`upload_cli_session` guard. The CLI session file is independent of the
JSONL transcript; and upload must run on mode-switch T1 so the next turn
can `--resume`. `upload_cli_session` silently skips when the file is
absent, so unconditional upload is always safe.
## How
| Scenario | Before | After |
|---|---|---|
| T1 fresh (`has_history=False`) | `session_id` set ✓ | `session_id` set
✓ |
| Mode-switch T1 (`has_history=True`, no CLI session) | ❌ not set —
**bug** | `session_id` set ✓ |
| T2+ with `--resume` | `resume` set ✓ | `resume` set ✓ |
| T2+ retry after `--resume` failed | `session_id` removed ✓ |
`session_id` removed ✓ |
| Mode-switch T1 retry | `session_id` removed ❌ | `session_id` kept ✓ |
| Upload on mode-switch T1 | ❌ blocked by guard — **bug** | uploaded ✓ |
7 new unit tests in `TestSdkSessionIdSelection` document all session_id
cases.
6 new tests in `mode_switch_context_test.py` cover transcript bridging
for both fast→SDK and SDK→fast switches.
## Checklist
- [x] I have read the contributing guidelines
- [x] My changes are covered by tests
- [x] `poetry run format` passes
---------
Co-authored-by: Zamil Majdy <zamilmajdy@gmail.com>
## Why
After merging #12782 to dev, a k8s rolling deployment triggered
infrastructure-level POST retries — nginx detected the old pod's
connection reset mid-stream and resent the same POST to a new pod. Both
pods independently saved the user message and ran the executor,
producing duplicate entries in the DB (seq 159, 161, 163) and a
duplicate response in the chat. The model saw the same question 3× in
its context window and spent its response commenting on that instead of
answering.
Two compounding issues:
1. **No backend idempotency**: `append_and_save_message` saves
unconditionally — k8s/nginx retries silently produce duplicate turns.
2. **Frontend dedup cleared after success**:
`lastSubmittedMsgRef.current = null` after every completed turn wipes
the dedup guard, so any rapid re-submit of the same text (from a stalled
UI or user double-click) slips through.
## What
**Backend** — Redis idempotency gate in `stream_chat_post`:
- Before saving the user message, compute `sha256(session_id +
message)[:16]` and `SET NX ex=30` in Redis
- If key already exists → duplicate: return empty SSE (`StreamFinish +
[DONE]`) immediately, skip save + executor enqueue
- User messages only (`is_user_message=True`); system/assistant messages
bypass the check
**Frontend** — Keep `lastSubmittedMsgRef` populated after success:
- Remove `lastSubmittedMsgRef.current = null` on stream complete
- `getSendSuppressionReason` already has a two-condition check: `ref ===
text AND lastUserMsg === text` — so legitimate re-asks (after a
different question was answered) still work; only rapid re-sends of the
exact same text while it's still the last user message are blocked
## How
- 30 s Redis TTL covers infrastructure retry windows (k8s SIGTERM →
connection reset → ingress retry typically < 5 s)
- Empty SSE response is well-formed (StreamFinish + [DONE]) — frontend
AI SDK marks the turn complete without rendering a ghost message
- Frontend ref kept live means: submit "foo" → success → submit "foo"
again instantly → suppressed. Submit "foo" → success → submit "bar" →
proceeds (different text updates the ref).
## Tests
- 3 new backend route tests: duplicate blocked, first POST proceeds,
non-user messages bypass
- 5 new frontend `getSendSuppressionReason` unit tests: fresh ref,
reconnecting, duplicate suppressed, different-turn re-ask allowed,
different text allowed
## Checklist
- [x] I have read the [AutoGPT Contributing
Guide](https://github.com/Significant-Gravitas/AutoGPT/blob/master/CONTRIBUTING.md)
- [x] I have performed a self-review of my code
- [x] I have added tests that prove the fix is effective
- [x] I have run `poetry run format` and `pnpm format` + `pnpm lint`
## Summary
Fixes stream disconnection bugs where the UI shows "running" with no
output when users switch between copilot chat sessions. The root cause
is that the old SSE fetch is not aborted and backend XREAD listeners
keep running until timeout when switching sessions.
### Changes
**Frontend (`useCopilotStream.ts`, `helpers.ts`)**
- Call `sdkStop()` on session switch to abort the in-flight SSE fetch
from the old session's transport
- Fire-and-forget `DELETE` to new backend disconnect endpoint so
server-side listeners release immediately
- Store `resumeStream` and `sdkStop` in refs to fix stale closure bugs
in:
- Wake re-sync visibility handler (could call stale `resumeStream` after
tab sleep)
- Reconnect timer callback (could target wrong session's transport)
- Resume effect (captured stale `resumeStream` during rapid session
switches)
**Backend (`stream_registry.py`, `routes.py`)**
- Add `disconnect_all_listeners(session_id)` to stream registry —
iterates active listener tasks, cancels any matching the session
- Add `DELETE /sessions/{session_id}/stream` endpoint — auth-protected,
calls `disconnect_all_listeners`, returns 204
### Why
Reported by multiple team members: when using Autopilot for anything
serious, the frontend loses the SSE connection — particularly when
switching between conversations. The backend completes fine (refreshing
shows full output), but the UI gets stuck showing "running". This is the
worst UX bug we have right now because real users will never know to
refresh.
### How to test
1. Start a long-running autopilot task (e.g., "build a snake game")
2. While it's streaming, switch to a different chat session
3. Switch back — the UI should correctly show the completed output or
resume the stream
4. Verify no "stuck running" state
## Test plan
- [ ] Manual: switch sessions during active stream — no stuck "running"
state
- [ ] Manual: background tab for >30s during stream, return — wake
re-sync works
- [ ] Manual: trigger reconnect (kill network briefly) — reconnects to
correct session
- [ ] Verify: `pnpm lint`, `pnpm types`, `poetry run lint` all pass
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: majdyz <zamil.majdy@agpt.co>
## Why
Users have different task complexity needs. Sonnet is fast and cheap for
most queries; Opus is more capable for hard reasoning tasks. Exposing
this as a simple toggle gives users control without requiring
infrastructure complexity.
Opus costs 5× more than Sonnet per Anthropic pricing ($15/$75 vs $3/$15
per M tokens). Rather than adding a separate entitlement gate, the
rate-limit multiplier (5×) ensures Opus turns deplete the daily/weekly
quota proportionally faster — users self-limit via their existing
budget.
## What
- **Standard/Advanced model toggle** in the chat input toolbar (sky-blue
star icon, label only when active — matches the simulation
DryRunToggleButton pattern but visually distinct)
- **`CopilotLlmModel = Literal["standard", "advanced"]`** —
model-agnostic tier names (not tied to Anthropic model names)
- **Backend model resolution**: `"advanced"` → `claude-opus-4-6`,
`"standard"` → `config.model` (currently Sonnet)
- **Rate-limit multiplier**: Opus turns count as 5× in Redis token
counters (daily + weekly limits). Does **not** affect `PlatformCostLog`
or `cost_usd` — those use real API-reported values
- **localStorage persistence** via `Key.COPILOT_MODEL` so preference
survives page refresh
- **`claude_agent_max_budget_usd`** reduced from $15 to $10
## How
### Backend
- `CopilotLlmModel` type added to `config.py`, imported in
routes/executor/service
- `stream_chat_completion_sdk` accepts `model: CopilotLlmModel | None`
- Model tier resolved early in the SDK path; `_normalize_model_name`
strips the OpenRouter provider prefix
- `model_cost_multiplier` (1.0 or 5.0) computed from final resolved
model name, passed to `persist_and_record_usage` → `record_token_usage`
(Redis only)
- No separate LD flag needed — rate limit is the gate
### Frontend
- `ModelToggleButton` component: sky-blue, star icon, "Advanced" label
when active
- `copilotModel` state in `useCopilotUIStore` with localStorage
hydration
- `copilotModelRef` pattern in `useCopilotStream` (avoids recreating
`DefaultChatTransport`)
- Toggle gated behind `showModeToggle && !isStreaming` in `ChatInput`
## Checklist
- [x] Tests added/updated (ModelToggleButton.test.tsx,
service_helpers_test.py, token_tracking_test.py)
- [x] Rate-limit multiplier only affects Redis counters, not cost
tracking
- [x] No new LD flag needed
## Why
When the agent generator LLM builds a graph, it may need to look up
schema details for graph-only blocks like `AgentInputBlock`,
`AgentOutputBlock`, or `OrchestratorBlock`. These blocks are correctly
hidden from regular CoPilot `find_block` results (they can't run
standalone), but that same filter was also preventing the LLM from
discovering them when composing an agent graph.
## What
Added a `for_agent_generation: bool = False` parameter to
`FindBlockTool`.
## How
- `for_agent_generation=false` (default): existing behaviour unchanged —
graph-only blocks are filtered from both UUID lookups and text search
results.
- `for_agent_generation=true`: bypasses `COPILOT_EXCLUDED_BLOCK_TYPES` /
`COPILOT_EXCLUDED_BLOCK_IDS` so the LLM can find and inspect schemas for
INPUT, OUTPUT, ORCHESTRATOR, WEBHOOK, etc. blocks when building agent
JSON.
- MCP_TOOL blocks are still excluded even with
`for_agent_generation=true` (they go through `run_mcp_tool`, not
`find_block`).
## Checklist
- [x] No new dependencies
- [x] Backward compatible (default `false` preserves existing behaviour)
- [x] No frontend changes
## Why
OpenRouter occasionally returns `null` (not `0`) for
`cache_read_input_tokens` and `cache_creation_input_tokens` on the
initial streaming event, before real token counts are available.
Python's `dict.get(key, 0)` only falls back to `0` when the key is
**missing** — when the key exists with a `null` value, `.get(key, 0)`
returns `None`. This causes `TypeError: unsupported operand type(s) for
+=: 'int' and 'NoneType'` in the usage accumulator on the first
streaming chunk from OpenRouter models.
## What
- Replace `.get(key, 0)` with `.get(key) or 0` for all four token fields
in `_run_stream_attempt`
- Add `TestTokenUsageNullSafety` unit tests in `service_helpers_test.py`
## How
Minimal targeted fix — only the four `+=` accumulation lines changed. No
behaviour change for Anthropic-native models (they never emit null
values).
## Checklist
- [x] Tests cover null event, real event, absent keys, and multi-turn
accumulation
- [x] No behaviour change for Anthropic-native models
- [x] No API changes
### Why / What / How
**Why:** The OrchestratorBlock in agent mode makes multiple LLM calls in
a single node execution (one per iteration of the tool-calling loop),
but the executor was only charging the user once per run via
`_charge_usage`. Tools spawned by the orchestrator also bypassed
`_charge_usage` entirely — they execute via `on_node_execution()`
directly without going through the main execution queue, producing free
internal block executions.
**What:**
1. Charge `base_cost * (llm_call_count - 1)` extra credits after the
orchestrator block completes — covers the additional iterations beyond
the first (which is already paid for upfront).
2. Charge user credits for tools executed inside the orchestrator, the
same way queue-driven node executions are charged.
**How:**
**1. Per-iteration LLM charging**
- New `Block.extra_runtime_cost(execution_stats)` virtual method
(default returns `0`)
- `OrchestratorBlock` overrides it to return `max(0, llm_call_count -
1)`
- New `resolve_block_cost` free function in `billing.py` centralises the
block-lookup + cost-calculation pattern (used by both `charge_usage` and
`charge_extra_runtime_cost`)
- New `billing.charge_extra_runtime_cost(node_exec, extra_count)`
function that debits `base_cost * min(extra_count,
_MAX_EXTRA_RUNTIME_COST)` via `spend_credits()`, running synchronously
in a thread-pool worker
- After `_on_node_execution` completes with COMPLETED status,
`on_node_execution` calls `charge_extra_runtime_cost` if
`extra_runtime_cost > 0` and not a dry run
- `InsufficientBalanceError` from post-hoc charging is treated as a
billing leak: logged at ERROR with `billing_leak: True` structured
fields, user is notified via `_handle_insufficient_funds_notif`, but the
run status stays COMPLETED (work already done)
**2. Tool execution charging**
- New public async `ExecutionProcessor.charge_node_usage(node_exec)`
wrapper around `charge_usage` (with `execution_count=0` to avoid
inflating execution-tier counters); also calls `_handle_low_balance`
internally
- `OrchestratorBlock._execute_single_tool_with_manager` calls
`charge_node_usage` after successful tool execution (skipped for dry
runs and failed/cancelled tool runs)
- Tool cost is added to the orchestrator's `extra_cost` so it shows up
in graph stats display
- `InsufficientBalanceError` from tool charging is re-raised (not
downgraded to a tool error) in all three execution paths:
`_execute_single_tool_with_manager`, `_agent_mode_tool_executor`, and
`_execute_tools_sdk_mode`
**3. Billing module extraction**
- All billing logic extracted from `ExecutionProcessor` into
`backend/executor/billing.py` as free functions — keeps `manager.py` and
`service.py` focused on orchestration
- `ExecutionProcessor` retains thin delegation methods
(`charge_node_usage`, `charge_extra_runtime_cost`) for backward
compatibility with blocks that call them
**4. Structured error signalling**
- Tool error detection replaced brittle `text.startswith("Tool execution
failed:")` string check with a structured `_is_error` boolean field on
the tool response dict
### Changes
- `backend/blocks/_base.py`: Add
`Block.extra_runtime_cost(execution_stats) -> int` virtual method
(default `0`)
- `backend/blocks/orchestrator.py`: Override `extra_runtime_cost`; add
tool charging in `_execute_single_tool_with_manager`; add
`InsufficientBalanceError` re-raise carve-outs in all three execution
paths; replace string-prefix error detection with `_is_error` flag
- `backend/executor/billing.py` (new): Free functions
`resolve_block_cost`, `charge_usage`, `charge_extra_runtime_cost`,
`charge_node_usage`, `handle_post_execution_billing`,
`clear_insufficient_funds_notifications` — extracted from
`ExecutionProcessor`
- `backend/executor/manager.py`: Thin delegation to `billing.*`; remove
~500 lines of billing methods from `ExecutionProcessor`
- `backend/data/credit.py`: Update lazy import source from `manager` to
`billing`
- `backend/blocks/test/test_orchestrator.py`: Add `charge_node_usage`
mock + assertion
- `backend/blocks/test/test_orchestrator_dynamic_fields.py`: Add
`charge_node_usage` async mock
- `backend/blocks/test/test_orchestrator_responses_api.py`: Add
`charge_node_usage` async mock
- `backend/blocks/test/test_orchestrator_per_iteration_cost.py`: New
test file — `extra_runtime_cost` hook, `charge_extra_runtime_cost` math
(positive/zero/negative/capped/zero-cost/block-not-found/IBE),
`charge_node_usage` delegation, `on_node_execution` gate conditions
(COMPLETED/FAILED/zero-charges/dry-run/IBE), tool charging guards
(dry-run/failed/cancelled/IBE propagation)
### Checklist
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [ ] I have tested my changes according to the test plan:
- [ ] Run `poetry run pytest
backend/blocks/test/test_orchestrator_per_iteration_cost.py`
- [ ] Verify on dev: an OrchestratorBlock run with
`agent_mode_max_iterations=5` and 5 actual iterations is charged 5x the
base cost
- [ ] Verify tool executions inside the orchestrator are charged
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: majdyz <majdy.zamil@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: majdyz <majdyz@users.noreply.github.com>
get_proration_credit_cents now checks user.stripe_customer_id before calling into
Stripe, matching the same pattern applied to cancel_stripe_subscription. Admin-granted
paid-tier users without a Stripe record previously triggered customer creation on every
billing page load; they now get 0 immediately.
Also adds tests: no_customer_id fast-path for cancel, and three proration scenarios
(zero cost, no customer id, active subscription with proration calculation).
- Guard cancel_stripe_subscription with stripe_customer_id check to prevent
creating orphaned Stripe customers for users who never had a paid subscription
- Fix upgrade dialog showing raw tier key (PRO) instead of human-readable label (Pro)
- Fix misleading footer text that contradicted downgrade-to-FREE behaviour
(downgrades to Free are scheduled at period end, not immediate)
Adds the missing get_proration_credit_cents mock to the three
GET /credits/subscription route tests so they don't attempt a live
Stripe/DB call in unit-test context, and extends assertions to cover
the proration_credit_cents field in the response.
## Why
During a live production session, the copilot lost all conversation
context mid-session. The model stated \"I don't see any implementation
plan in our conversation\" despite 9 prior turns of context. Three
compounding bugs:
**Bug 1 — Self-perpetuating upload gate:** When `restore_cli_session`
fails on a T2+ turn, `state.use_resume=False`. The old gate `and (not
has_history or state.use_resume)` then skips the CLI session upload —
even though the T1 file may exist. Each turn without `use_resume` skips
upload → next turn can't restore → also skips → etc.
**Bug 2 — Blunt message-count cap on retries:** On `prompt-too-long`,
`_reduce_context` retried 3× but rebuilt the same oversized query each
time (transcript was empty, so all 3 attempts were identical). The
`max_fallback_messages` count-cap was a blunt instrument — it threw away
middle turns blindly instead of letting the compressor summarize
intelligently.
**Bug 3 — Gap-empty path returned zero context:** When a transcript
exists but no `--resume` (CLI session unavailable), and the gap is empty
(transcript is current), the code fell through to `return
current_message, False` — the model got no history at all.
## What
1. **Remove upload gate** — upload is attempted after every successful
turn; `upload_cli_session` silently skips when the file is absent.
2. **`transcript_msg_count` set on `cli_restored=False`** — enables the
gap path on the very next turn without waiting for a full upload cycle.
3. **Token-budget compression instead of message-count cap** —
`_reduce_context` now returns `target_tokens` (50K → 15K across
retries). `compress_context` decides what to drop via LLM summarize →
content truncate → middle-out delete → first/last trim. More context
preserved at any budget vs. blindly slicing the list.
4. **Fix gap-empty case** — when transcript is current but `--resume`
unavailable, fall through to full-session compression with the token
budget instead of returning no context.
5. **Transcript seeding after fallback** — after `use_resume=False` with
no stored transcript, compress DB messages to 30K tokens and serialise
as JSONL into `transcript_builder`. Next turn uses the gap path (inject
only new messages) instead of re-compressing full history. Only fires
once per broken session (`not transcript_content` guard).
6. **Seeding guard** — seeding skips when `skip_transcript_upload=True`
(avoids wasted compression work when the result won't be saved).
7. **Structured logging** — INFO/WARNING at every branch of
`_build_query_message` with path variables, context_bytes, compression
results.
## How
**Upload gate** (`sdk/service.py` finally-block): removed `and (not
has_history or state.use_resume)`; added INFO log showing
`use_resume`/`has_history` before upload.
**`transcript_msg_count`**: set from `dl.message_count` in the
`cli_restored=False` branch.
**`_build_query_message`**: `max_fallback_messages: int | None` →
`target_tokens: int | None`; gap-empty case falls through to
full-session compression rather than returning bare message.
**`_reduce_context`**: `_FALLBACK_MSG_LIMITS` → `_RETRY_TARGET_TOKENS =
(50_000, 15_000)`; returns `ReducedContext.target_tokens`.
**`_compress_messages` / `_run_compression`**: both now accept
`target_tokens: int | None` and thread it through to `compress_context`.
**Seeding block**: added `not skip_transcript_upload` guard; uses
`_SEED_TARGET_TOKENS = 30_000` so the seeded JSONL is always compact
enough to pass `validate_transcript`.
## Checklist
- [x] `poetry run format` passes
- [x] No new lint errors introduced (pre-existing pyright errors
unrelated)
- [x] Tests added for `attempt` parameter and `target_tokens` in
`_reduce_context`
Downgrading between paid tiers (e.g. BUSINESS→PRO) previously fell through to
the Checkout Session path, which would cancel the existing subscription immediately
without crediting unused time on the old plan.
For paid→paid tier changes, modify the existing Stripe subscription in-place
via stripe.Subscription.modify with proration_behavior="create_prorations".
Stripe handles the proration automatically (crediting unused time + charging
the pro-rated difference), and the existing customer.subscription.updated
webhook fires to update the DB tier — no new Checkout flow needed.
Adds unit tests for modify_stripe_subscription_for_tier and API-level tests for
the paid→paid code path in update_subscription_tier.
Pass the Stripe invoice ID as transaction_key to _add_transaction in
handle_subscription_payment_failure. This prevents double-charging user
credits when Stripe retries the invoice.payment_failed webhook after a
transient failure (e.g. if stripe.Invoice.pay raises a network error).
_add_transaction silently skips insertion when the key already exists,
so subsequent retries deduct nothing while still attempting to mark the
invoice as paid on the Stripe side.
Adds test to verify the idempotency key is set correctly.
When handle_subscription_payment_failure successfully deducts the invoice
amount from a user's credit balance, Stripe's dunning system would still
retry the failed invoice automatically on its schedule, causing repeated
deductions per billing period. Fix by calling stripe.Invoice.pay() after
the successful balance deduction so Stripe marks the invoice as settled and
stops retrying. Invoice.pay failures are logged as warnings but do not
propagate, matching the existing best-effort pattern for cleanup operations.
Add two tests covering the pay call and the swallowed-error path.
The implementation schedules cancellation at period end via
stripe.Subscription.modify(cancel_at_period_end=True) rather than
stripe.Subscription.cancel. Update the four affected test cases to
patch the correct API call and assert the right arguments.
When a user's paid tier was admin-granted (no associated Stripe subscription),
cancel_stripe_subscription found nothing to cancel and returned. The API then
returned 200 without updating the DB tier, because the design relied on a
customer.subscription.deleted webhook — which would never fire.
Fix: have _cancel_customer_subscriptions return the count of subs cancelled and
cancel_stripe_subscription return a bool. When False (no subs found), the endpoint
now calls set_subscription_tier directly, mirroring the payment-disabled code path.
Added test_update_subscription_tier_free_no_stripe_subscription to cover this case.
- Regenerated openapi.json to include `proration_credit_cents` field added
to `SubscriptionStatusResponse` (was missing, causing check-API-types CI failure)
- Updated SubscriptionTierSection tests to go through the Confirm Upgrade
dialog before asserting mutate/redirect/error behaviour — previously tests
clicked "Upgrade to Pro" and expected a direct mutate call, but the current
code shows a confirmation dialog first (3 tests were failing because of this)
- Added `proration_credit_cents: 0` to `makeSubscription()` helper to match
the now-required field in the generated TypeScript type
- Add get_proration_credit_cents() — fetches active Stripe sub, calculates
unused portion of current billing period in cents
- Expose proration_credit_cents in GET /credits/subscription response
- Show upgrade confirmation dialog: "Your unused $X.XX will be added to
your balance" before redirecting to Stripe Checkout
- Update downgrade dialog copy to reflect cancel-at-period-end behaviour
Cancelling the subscription triggered customer.subscription.deleted
which then downgraded the user to FREE despite the balance having
already covered the invoice. Now the sub is left intact when balance
covers the cost — only cancelled (and tier downgraded) when balance
is insufficient.
User keeps their paid tier for the remainder of the billing period
they already paid for. DB tier is no longer updated immediately on
downgrade — customer.subscription.deleted webhook fires at period end
and downgrades to FREE then.
### Why / What / How
This PR simplifies frontend PR validation to one Playwright E2E suite,
moves redundant page-level browser coverage into Vitest integration
tests, and switches Playwright auth to deterministic seeded QA accounts.
It also folds in the follow-up fixes that came out of review and CI:
lint cleanup, CodeQL feedback, PR-local type regressions, and the flaky
Library run helper.
The approach is:
- keep Playwright focused on real browser and cross-page flows that
integration tests cannot prove well
- keep page-level render and mocked API behavior in Vitest
- remove the old PR-vs-full Playwright split from CI and run one
deterministic PR suite instead
- seed reusable auth states for fixed QA users so the browser suite is
less flaky and faster to bootstrap
### Changes 🏗️
- Removed the workflow indirection that selected different Playwright
suites for PRs vs other events
- Standardized frontend CI on a single command: `pnpm test:e2e:no-build`
- Consolidated the PR-gating Playwright suite around these happy-path
specs:
- `auth-happy-path.spec.ts`
- `settings-happy-path.spec.ts`
- `api-keys-happy-path.spec.ts`
- `builder-happy-path.spec.ts`
- `library-happy-path.spec.ts`
- `marketplace-happy-path.spec.ts`
- `publish-happy-path.spec.ts`
- `copilot-happy-path.spec.ts`
- Added the missing browser-only confidence checks to the PR suite:
- settings persistence across reload and re-login
- API key create, copy, and revoke
- schedule `Run now` from Library
- activity dropdown visibility for a real run
- creator dashboard verification after publish submission
- Increased Playwright CI workers from `6` to `8`
- Migrated redundant page-level browser coverage into Vitest
integration/unit tests where appropriate, including marketplace,
profile, settings, API keys, signup behavior, agent dashboard row
behavior, agent activity, and utility/auth helpers
- Seeded deterministic Playwright QA users in
`backend/test/e2e_test_data.py` and reused auth states from
`frontend/src/tests/credentials/`
- Fixed CodeQL insecure randomness feedback by replacing insecure
randomness in test auth utilities
- Fixed frontend lint issues in marketplace image rendering
- Fixed PR-local type regressions introduced during test migration
- Stabilized the Library E2E run helper to support the current Library
action states: `Setup your task`, `New task`, `Rerun task`, and `Run
now`
- Removed obsolete Playwright specs and the temporary migration planning
doc once the consolidation was complete
- Reverted unintended non-test backend source changes; only backend test
fixture changes remain in scope
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] `pnpm lint`
- [x] `pnpm types`
- [x] `pnpm test:unit`
- [x] `pnpm exec playwright test --list`
- [x] `pnpm test:e2e:no-build` locally
- [ ] PR CI green after the latest push
#### For configuration changes:
- [x] `.env.default` is updated or already compatible with my changes
- [x] `docker-compose.yml` is updated or already compatible with my
changes
- [x] I have included a list of my configuration changes in the PR
description (under **Changes**)
Notes:
- Current local Playwright run on this branch: `28 passed`, `0 flaky`,
`0 retries`, `3m 25s`.
- Latest Codecov report on this PR showed overall coverage `63.14% ->
63.61%` (`+0.47%`), with frontend coverage up `+2.32%` and frontend E2E
coverage up `+2.10%`.
- The backend change in this PR is limited to deterministic E2E test
data setup in `backend/test/e2e_test_data.py`.
- Playwright retries remain enabled in CI; this branch does not add
fail-on-flaky behavior.
---------
Co-authored-by: Zamil Majdy <zamil.majdy@agpt.co>
Co-authored-by: Zamil Majdy <majdy.zamil@gmail.com>
- Use user_id="system" for global LD flag lookups (price IDs don't need user context)
- Skip Supabase lookup silently for non-UUID keys in _fetch_user_context_data
- Block paid tier changes when ENABLE_PLATFORM_PAYMENT is disabled
- Add invoice.payment_failed handler: deduct from balance or downgrade to FREE
- Hide upgrade/downgrade buttons in UI when payment flag is disabled
## Summary
- **Token breakdown in provider table**: Added separate Input Tokens and
Output Tokens columns to the By Provider table, making it easy to see
whether costs are driven by large contexts (input) or verbose
responses/thinking (output)
- **New summary cards (8 total)**: Added Avg Cost/Request, Avg Input
Tokens, Avg Output Tokens, and Total Tokens (in/out split) cards plus
P50/P75/P95/P99 cost percentile cards at the top of the dashboard for
at-a-glance cost analysis
- **Cost distribution histogram**: Added a cost distribution section
showing request count across configurable price buckets ($0–0.50,
$0.50–1, $1–2, $2–5, $5–10, $10+)
- **Per-user avg cost**: Added Avg Cost/Req column to the By User table
to identify users with unusually expensive requests
- **Backend aggregations**: Extended `PlatformCostDashboard` model with
`total_input_tokens`, `total_output_tokens`,
`avg_input_tokens_per_request`, `avg_output_tokens_per_request`,
`avg_cost_microdollars_per_request`,
`cost_p50/p75/p95/p99_microdollars`, and `cost_buckets` fields
- **Correct denominators**: Avg cost uses cost-bearing requests only;
avg token stats use token-bearing requests only — no artificial dilution
from non-cost/non-token rows
## Test plan
- [x] Verify the admin cost dashboard loads without errors at
`/admin/platform-costs`
- [x] Check that the new summary cards display correct values
- [x] Verify Input/Output Tokens columns appear in the By Provider table
- [x] Verify Avg Cost/Req column appears in the By User table
- [x] Confirm existing functionality (filters, export, rate overrides)
still works
- [x] Verify backward compatibility — new fields have defaults so old
API responses still work
## Why
`src/app/api/__generated__/` is listed in `.gitignore` but 4 model files
were committed before that rule existed, so git kept tracking them and
they showed up in every PR that touched the API schema.
## What
Run `git rm --cached` on all 4 tracked files so the existing gitignore
rule takes effect. No gitignore content changes needed — the rule was
already correct.
## How
The `check API types` CI job only diffs `openapi.json` against the
backend's exported schema — it does not diff the generated TypeScript
models. So removing these from tracking does not break any CI check.
After this merges, `pnpm generate:api` output will be gitignored
everywhere and future API-touching PRs won't include generated model
diffs.
When ENABLE_PLATFORM_PAYMENT is off for paid tier requests, return 422
instead of setting the tier directly. Admin tier changes must go through
the /api/admin/ routes, not the self-service endpoint.
Updates the corresponding subscription route test to assert the 422
response and removes the now-invalid set_subscription_tier mock.
## Summary
- Use `SystemPromptPreset` with `exclude_dynamic_sections=True` in the
SDK path so the Claude Code default prompt serves as a cacheable prefix
shared across all users, reducing input token cost by ~90%
- Add `claude_agent_cross_user_prompt_cache` config field (default
`True`) to make this configurable, with fallback to raw string when
disabled
- Extract `_build_system_prompt_value()` helper for testability, with
`_SystemPromptPreset` TypedDict for proper type annotation
> **Depends on #12747** — requires SDK >=0.1.58 which adds
`SystemPromptPreset` with `exclude_dynamic_sections`. Must be merged
after #12747.
## Changes
- **`config.py`**: New `claude_agent_cross_user_prompt_cache: bool =
True` field on `ChatConfig`
- **`sdk/service.py`**: `_SystemPromptPreset` TypedDict for type safety;
`_build_system_prompt_value()` helper that constructs the preset dict or
returns the raw string; call site uses the helper
- **`sdk/service_test.py`**: Tests exercise the production
`_build_system_prompt_value()` helper directly — verifying preset dict
structure (enabled), raw string fallback (disabled), and default config
value
## How it works
The Claude Code CLI supports `SystemPromptPreset` which uses the
built-in Claude Code default prompt as a static prefix. By setting
`exclude_dynamic_sections=True`, per-user dynamic sections (working dir,
git status, auto-memory) are stripped from that prefix so it stays
identical across users and benefits from Anthropic's prompt caching. Our
custom prompt (tool notes, supplements, graphiti context) is appended
after the cacheable prefix.
## Test plan
- [x] CI passes (formatting, linting, unit tests)
- [x] Verify `_build_system_prompt_value()` returns correct preset dict
when enabled
- [x] Verify fallback to raw string when
`CHAT_CLAUDE_AGENT_CROSS_USER_PROMPT_CACHE=false`
## Why
The baseline copilot path (OpenAI-compatible / OpenRouter) did not
record any cost when the `x-total-cost` response header was absent, even
though token counts were always available. The admin cost dashboard also
lacked cache token columns.
## What
- **`x-total-cost` header extraction**: Reads the OpenRouter cost header
per LLM call in the `finally` block (so cost is captured even when the
stream errors mid-way). Accumulated across multi-round tool-calling
turns.
- **Cache token extraction**: Extracts
`prompt_tokens_details.cached_tokens` and `cache_creation_input_tokens`
from streaming usage chunks and passes
`cache_read_tokens`/`cache_creation_tokens` through to
`persist_and_record_usage` for storage in `PlatformCostLog`.
- **Dashboard cache token display**: Adds cache read/write columns to
the Raw Logs and By User tables on the admin platform costs dashboard.
Adds `total_cache_read_tokens` and `total_cache_creation_tokens` to
`UserCostSummary`.
- **No cost estimation**: When `x-total-cost` is absent, `cost_usd` is
left as `None` and `persist_and_record_usage` records the entry under
`tracking_type="tokens"`. Token-based cost estimation was removed — the
platform dashboard already handles per-token cost display, and estimates
would introduce inaccuracy in the reported figures.
## How
- In `_baseline_llm_caller`: extract the `x-total-cost` header in the
`finally` block; accumulate to `state.cost_usd`.
- In `_BaselineStreamState`: add `turn_cache_read_tokens` /
`turn_cache_creation_tokens` counters, populated from streaming usage
chunks.
- In `persist_and_record_usage` / `record_cost_log`: pass through
`cache_read_tokens` and `cache_creation_tokens` to `PlatformCostEntry`.
- Frontend: add `total_cache_read_tokens` /
`total_cache_creation_tokens` fields to `UserCostSummary` and render
them as columns in the cost dashboard.
## Test plan
- [x] Verify baseline copilot sessions log cost when `x-total-cost`
header is present
- [x] Verify `cost_usd` stays `None` and token count is logged when
header is absent
- [x] Verify cache tokens appear in the dashboard logs table for
sessions using prompt caching
- [x] Verify the By User tab shows Cache Read and Cache Write columns
- [x] Unit tests: `test_cost_usd_extracted_from_response_header`,
`test_cost_usd_remains_none_when_header_missing`,
`test_cache_tokens_extracted_from_usage_details`
### Why / What / How
**Why:** The Claude Agent SDK's built-in Write and Edit tools have no
defence against output-token truncation. When the LLM generates a large
`content` or `new_string` argument, the API truncates the response
mid-JSON, causing Ajv to reject it with the opaque `"'file_path' is a
required property"` error. The user's work is silently lost, and
retrying with the same approach loops infinitely.
**What:** Replaces the SDK's built-in Write and Edit tools with unified
MCP equivalents that detect truncation and return actionable recovery
guidance. Adds a new `read_file` MCP tool with offset/limit pagination.
Consolidates all file-tool handlers into a single module
(`e2b_file_tools.py`) covering both E2B (sandbox) and non-E2B (local SDK
working directory) modes.
**How:**
- `file_path` is placed first in every JSON schema so truncation is more
likely to preserve the path
- `"required"` is intentionally omitted from all MCP schemas so the MCP
SDK delivers empty/truncated args to the handler instead of rejecting
them with an opaque error
- Handlers detect two truncation patterns: complete (`{}`) and partial
(other fields present but `file_path` missing), returning actionable
error messages in both cases
- Edit uses a per-path `asyncio.Lock` (keyed by resolved absolute path)
to prevent parallel read-modify-write races when MCP tools are
dispatched concurrently
- Both E2B and non-E2B paths validate via `is_allowed_local_path()` /
`is_within_allowed_dirs()` to block directory traversal
- The SDK built-in Write and Edit are added to `SDK_DISALLOWED_TOOLS`;
the SDK built-in Read remains allowed only for workspace-scoped paths
(tool-results/tool-outputs) via `WORKSPACE_SCOPED_TOOLS`
- E2B write/edit tools are registered with `readOnlyHint=False`
(`_MUTATING_ANNOTATION`) to prevent parallel dispatch
- `bridge_to_sandbox` copies host-side tool-result files into the E2B
sandbox on read so `bash_exec` can process them
### Changes 🏗️
- **`e2b_file_tools.py`** — unified file-tool handlers for Write, Read
(`read_file`), Edit, Glob, Grep covering both E2B and non-E2B modes;
per-path edit locking; truncation detection; sandbox symlink-escape
check; `bridge_to_sandbox` for SDK→E2B file bridging
- **`tool_adapter.py`** — registers unified Write/Edit/read_file MCP
tools (non-E2B only); adds `Read` tool for workspace-scoped SDK-internal
reads (both modes); E2B tools use `_MUTATING_ANNOTATION`;
`get_copilot_tool_names` / `get_sdk_disallowed_tools` updated for both
modes
- **`security_hooks.py`** — `WORKSPACE_SCOPED_TOOLS` checked before
`BLOCKED_TOOLS` so SDK internal Read is allowed on tool-results paths;
Write/Edit removed from workspace scope
- **`prompting.py`** — improved wording for large-file truncation
warning
- **`e2b_file_tools_test.py`** — comprehensive tests for non-E2B
Write/Read/Edit (path validation, truncation detection, offset/limit,
binary rejection, schema validation); E2B sandbox symlink-escape,
`bridge_to_sandbox`, and `_sandbox_write` tests
- **`security_hooks_test.py`** — updated tests for revised tool-blocking
and workspace-scoped Read behaviour
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] Read: normal read, offset/limit, file not found, path traversal
blocked, binary file handling, truncation detection
- [x] Edit: normal edit, old_string not found, old_string not unique,
replace_all, partial truncation, path traversal blocked
- [x] Write: existing tests unchanged; truncation detection, path
validation, large-content warning
- [x] Schema validation: file_path first, required fields intentionally
absent
- [x] CLI built-in Write and Edit are in `SDK_DISALLOWED_TOOLS`; Read is
workspace-scoped only
- [x] E2B write/edit use `_MUTATING_ANNOTATION` (not parallel)
- [x] `black`, `ruff`, `pyright` pass on all modified files
- [ ] CI pipeline passes
## Summary
- Hide the mode toggle button while streaming (instead of disabling it)
to avoid confusing partial-toggle UI
- Remove localStorage mode persistence — mode is now transient in-memory
state only (no stale overrides across sessions)
- The copilot mode indicator now correctly reflects the active session's
mode because it reads from Zustand store which is updated on session
switch
## Changes
- `ChatInput.tsx` — hide `<ModeToggleButton>` when `isStreaming` instead
of passing `isStreaming` prop and showing a disabled button
- `ModeToggleButton.tsx` — remove `isStreaming` prop, disabled state,
and streaming-specific tooltip
- `store.ts` — remove localStorage read/write for `copilotMode`; mode
now defaults to `extended_thinking` and resets on page load
- `local-storage.ts` — keep `COPILOT_MODE` enum entry for backward
compatibility; remove unused `COPILOT_SESSION_MODES`
- `store.test.ts` — update tests to assert mode is NOT persisted to
localStorage
- `ChatInput.test.tsx` / `ModeToggleButton.stories.tsx` — update to
match hide-not-disable behavior
## Test plan
- [x] Create a session in fast mode, create another in extended_thinking
mode
- [x] Switch between sessions and verify the mode indicator updates
correctly
- [x] Mode toggle is hidden (not disabled) while a response is streaming
- [x] Refreshing the page resets mode to extended_thinking (no stale
localStorage override)
### Why / What / How
**Why:** AutoPilot sessions were silently dying with no response. Root
cause: `AsyncSandbox.create()` in the E2B SDK uses
`httpx.AsyncClient(timeout=None)` — infinite wait. When the E2B API
stalled during sandbox provisioning, executor goroutines hung
indefinitely. After 1h42m the RabbitMQ consumer timeout
(`COPILOT_CONSUMER_TIMEOUT_SECONDS = 3600`) killed the pod and all
in-flight sessions were orphaned — user sees no response, no error.
**What:**
1. Added per-attempt timeout + retry loop to `AsyncSandbox.create()`
calls in `e2b_sandbox.py` — 30s/attempt × 3 retries with exponential
backoff (~93s worst case vs infinite)
2. Added recovery enqueue in `AutoPilotBlock.run()` — on unexpected
failure, re-enqueues the session to RabbitMQ so a fresh executor pod
picks it up on the next turn
3. Added `_is_deliberate_block()` guard so recursion-limit errors are
not re-enqueued (they are expected terminations)
4. Unit tests for both new mechanisms
**How:**
- `asyncio.wait_for(AsyncSandbox.create(), timeout=30)` wraps each
attempt; `TimeoutError` triggers retry
- Redis creation sentinel TTL bumped 60→120s to cover the full retry
window (prevents concurrent callers from seeing stale sentinel)
- `_enqueue_for_recovery` calls `enqueue_copilot_turn()` with the
original prompt so the session resumes where it left off; dry-run
sessions are skipped; enqueue failures are logged but never mask the
original error
- `CancelledError` is re-raised after yielding the error output
(cooperative cancellation)
### Changes 🏗️
**`backend/copilot/tools/e2b_sandbox.py`**
- Added `_SANDBOX_CREATE_TIMEOUT_SECONDS = 30`,
`_SANDBOX_CREATE_MAX_RETRIES = 3`
- Bumped `_CREATION_LOCK_TTL` 60 → 120s
- Replaced bare `AsyncSandbox.create()` with `asyncio.wait_for` + retry
loop
**`backend/blocks/autopilot.py`**
- Added `_is_deliberate_block(exc)` — returns True for recursion-limit
RuntimeErrors
- Added `_enqueue_for_recovery(session_id, user_id, message, dry_run)` —
re-enqueues to RabbitMQ; no-ops on dry_run
- Exception handler in `run()` calls `_enqueue_for_recovery` for
transient failures; inner try/except prevents enqueue failure from
masking the original error
**`backend/blocks/test/test_autopilot.py`**
- `TestIsDeliberateBlock` — 4 unit tests for `_is_deliberate_block`
- `TestRecoveryEnqueue` — 5 tests: transient error triggers enqueue,
recursion limit skips, dry_run passes flag through, enqueue failure
doesn't mask original error, `ctx.dry_run` is OR-ed in
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] `poetry run pytest backend/blocks/test/test_autopilot.py -xvs` —
24/24 pass
- [x] Verified retry logic constants: 30s × 3 retries + 1s + 2s = 93s
worst case, sentinel TTL 120s covers it
- [x] Verified `_enqueue_for_recovery` is no-op for dry_run=True (no
RabbitMQ publish)
- [x] Verified `CancelledError` re-raises after yield
## Why
The Claude CLI 2.1.97 (bundled in `claude-agent-sdk 0.1.58`) changed the
`--resume` flag to accept a **session UUID**, not a file path. Our
service was incorrectly passing a temp file path (from
`write_transcript_to_tempfile`), causing the CLI subprocess to crash
with exit code 1 on every T2+ message — breaking all multi-turn CoPilot
conversations.
Additionally, using a file-per-pod approach meant pod affinity was
required for `--resume` to work (the file only existed on the pod that
handled T1).
## What
- Add `upload_cli_session()` to `transcript.py`: after each turn, upload
the CLI's native session JSONL (at
`{projects_base}/{encoded_cwd}/{session_id}.jsonl`) to remote storage
- Add `restore_cli_session()` to `transcript.py`: before T2+, download
and restore the CLI native session file to the expected path
- Pass `--session-id {app_uuid}` via `ClaudeAgentOptions` so the CLI
uses the app session UUID as its session ID → predictable file path
- On T2+: call `restore_cli_session()` and if successful, pass `--resume
{session_uuid}` (UUID, not file path)
- Remove `write_transcript_to_tempfile` from the resume path in
service.py (it only exists in transcript.py for compaction use)
- Keep DB reconstruction as last-resort fallback (populates builder
state only, no `--resume`)
- Compaction retry path now runs without `--resume` (compacted content
cannot be written in CLI native format)
## How
**Normal multi-turn flow (fixed):**
1. T1: SDK runs with `--session-id {app_uuid}` → CLI writes session to
predictable path
2. T1 finally: `upload_cli_session()` uploads native session to storage
(GCS/local)
3. T2+: `restore_cli_session()` downloads and writes the native session
back to disk
4. T2+: `--resume {app_uuid}` → CLI reads the restored session → full
context preserved
**Cross-pod benefit:**
The native session file is now in remote storage, so any pod can restore
it before a turn. Pod affinity for CoPilot is no longer required.
**Backward compatibility:**
- First turn: no native session in storage → runs without `--resume`
(same as before)
- If `restore_cli_session` fails: falls back gracefully to no
`--resume`, logs a warning
- DB reconstruction still available as last resort when no transcript
exists at all
## Checklist
- [x] Tests updated (service_helpers_test, retry_scenarios_test,
transcript_test all pass)
- [x] `poetry run ruff check` clean
- [x] `poetry run black --check` clean
- [x] `poetry run pyright` 0 errors on changed files
## Why
`platform_cost_test.py` had an extra blank line between
`TestUsdToMicrodollars.test_large_value` and `class TestMaskEmail`,
causing black to flag it. This failure was appearing in the CI merge
checks of unrelated PRs that target `dev`.
## What
Remove the extra blank line (3 → 2) to satisfy black's formatting rules.
## How
Single-character diff — no logic changes.
## Why
When the AutoPilot copilot needed to connect credentials for an existing
agent, it was routing users to the Builder — flagged by @Pwuts in [the
AutoPilot Credential UX
thread](https://discord.com/channels/1126875755960336515/1492203735034892471/1492204936056930304).
Two root causes:
1. **Credential race-condition on the run/schedule path.**
`_check_prerequisites` only catches missing creds *before* the
executor/scheduler call. If creds are deleted (or drift) between the
prereq check and the actual call, the executor/scheduler raises
`GraphValidationError`. The tool returned a plain `ErrorResponse`, and
the LLM fell back to `create_agent`/`edit_agent` — whose
`AgentSavedResponse.agent_page_link=/build?flowID=...` is exactly the
Builder redirect the user saw.
2. **`GraphValidationError.node_errors` lost over RPC.** The scheduler
call goes through `get_scheduler_client()` (RPC). The server-side error
handler only preserved `exc.args` — the structured `node_errors` mapping
was stripped, making it impossible for the copilot to distinguish
credential failures from other validation errors on the schedule path.
## What
- **Race-condition handling for both run and schedule paths.**
`_run_agent` and `_schedule_agent` now catch `GraphValidationError`,
detect credential-flavoured node errors, and rebuild the inline
`SetupRequirementsResponse` so the credential setup card renders inline
without leaving chat. Mixed credential+structural errors fall through to
plain `ErrorResponse` so structural errors aren't hidden.
- **`GraphValidationError` round-trips over RPC.** `service.py` now
packs `node_errors` into a typed `RemoteCallExtras` field on
`RemoteCallError`, and the client-side handler re-threads it back into
the reconstructed exception.
- **Shared credential-error matcher.** The credential-string matching
logic is extracted to `is_credential_validation_error_message()` in
`backend/executor/utils.py`, backed by `CRED_ERR_*` module-level
constants that are referenced at both raise sites and in the matcher —
so adding a new credential error string doesn't silently break the
copilot fallback.
- **Tool-description guardrails.** `create_agent` and `edit_agent`
descriptions now explicitly say "Do NOT use this to connect credentials
— call run_agent instead." `agent_generation_guide.md` has the same
guardrail for the agent-building context.
## How
- `backend/copilot/tools/run_agent.py`: new
`_build_setup_requirements_from_validation_error()` helper; try/except
around `add_graph_execution` and `add_execution_schedule` in the
respective `_run_agent`/`_schedule_agent` paths; race-condition warnings
logged.
- `backend/executor/utils.py`: `CRED_ERR_*` constants +
`_CREDENTIAL_ERROR_MARKERS` typed tuple + public
`is_credential_validation_error_message()` exported; old private
`_is_credential_error` lambda replaced.
- `backend/util/service.py`: `RemoteCallExtras` Pydantic model with
`node_errors: Optional[dict[str, dict[str, str]]]`; server handler packs
it for `GraphValidationError`; client handler re-threads it;
`exception_class is GraphValidationError` identity check (not
`issubclass`).
- `backend/copilot/tools/create_agent.py`, `edit_agent.py`: added
credential-routing guardrail to tool descriptions.
- `backend/copilot/sdk/agent_generation_guide.md`: added
credential-routing guardrail.
## Test plan
- [x] Unit tests for `is_credential_validation_error_message` (all four
error templates matched, case-insensitive, non-credential messages
rejected).
- [x] Parity tests in `utils_test.py` that pin all `CRED_ERR_*`
constants against `is_credential_validation_error_message` — drift when
a new credential error is added fails immediately.
- [x] Unit tests for `_build_setup_requirements_from_validation_error`:
credential error → `SetupRequirementsResponse`; non-credential error →
`None`; mixed errors → `None`.
- [x] E2E test for `_schedule_agent` race path:
`get_scheduler_client().add_execution_schedule` mocked to raise
credential `GraphValidationError` → response is `setup_requirements`,
not generic error.
- [x] E2E test for `_run_agent` race path:
`execution_utils.add_graph_execution` mocked with `AsyncMock` to raise
credential `GraphValidationError` → response is `setup_requirements`.
- [x] `RemoteCallError` round-trip tests in `service_test.py`: server
handler packs `node_errors` into `extras`; client handler unpacks; full
round-trip preserves `node_errors`.
- [x] Backwards-compat test: old `RemoteCallError` without `extras`
still deserializes to `GraphValidationError` with empty `node_errors`.
## Summary
- Extract `ThinkingStripper` from `baseline/service.py` into a shared
`copilot/thinking_stripper.py` module
- Apply thinking-tag stripping to the SDK streaming path
(`_dispatch_response`) so `<internal_reasoning>` and `<thinking>` tags
emitted by non-extended-thinking models (e.g. Sonnet) are stripped
before reaching the SSE client
- Flush any buffered text from the stripper at stream end so no content
is lost
- Add unit tests for the shared `ThinkingStripper` and integration tests
for the SDK dispatch path
## Problem
When using Claude Sonnet (which doesn't have extended thinking), the
model sometimes outputs `<internal_reasoning>...</internal_reasoning>`
tags as visible text in the response stream. The baseline path already
stripped these, but the SDK path did not.
## Test plan
- [ ] CI passes (unit tests for ThinkingStripper and SDK dispatch
stripping)
- [ ] Manual test: send a message via Sonnet and verify no
`<internal_reasoning>` tags appear in the response
## Summary
- Fixes duplicate message content shown in CoPilot during SSE
reconnections (page visibility change, network hiccups, wake-resync)
- The `resume_session_stream` backend always replays from `"0-0"`
(beginning of Redis stream), and replayed `UIMessage` objects get new
generated IDs from `useChat`, bypassing the old adjacent-only content
dedup
- Extends `deduplicateMessages` to track all seen `role +
preceding-user-context + content` fingerprints globally, catching
replayed messages regardless of different IDs or position in the list
- Scopes fingerprints by preceding user message text to avoid false
positives when the assistant legitimately gives the same answer to
different prompts
## Test plan
- [ ] Verify new unit tests pass in CI (`helpers.test.ts` - 7 new dedup
test cases)
- [ ] Manual: start a long tool-use session, switch tabs, return - no
duplicate content
- [ ] Manual: refresh page during active session - content loads from DB
without duplicates
- [ ] Manual: ask the same question twice in different turns - both
answers preserved
## Why
We've been pinned at `claude-agent-sdk==0.1.45` (bundled CLI 2.1.63)
since PR #12294 because newer versions had two OpenRouter
incompatibilities:
1. **`tool_reference` content blocks** (CLI 2.1.69+) — OpenRouter's Zod
validation rejects them
2. **`context-management-2025-06-27` beta header** (CLI 2.1.91+) —
OpenRouter returns 400
Both are now resolved:
- **`tool_reference`: Fixed by CLI's built-in proxy detection.** CLI
2.1.70+ detects `ANTHROPIC_BASE_URL` pointing to a non-Anthropic
endpoint and disables `tool_reference` blocks automatically. Verified
working in CLI 2.1.97 — the bare CLI test only XFAILs on the beta
header, NOT on tool_reference.
- **`context-management` beta: Fixed by
`CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS=1` env var.** Injected via
`build_sdk_env()` for all SDK subprocess calls. Verified in CI.
## What
- Upgrades `claude-agent-sdk` from **0.1.45 → 0.1.58** (bundled CLI
2.1.63 → 2.1.97)
- Injects `CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS=1` in
`build_sdk_env()` (all modes)
- Adds `claude_agent_cli_path` config override with executable
validation
- Adds `claude_agent_max_thinking_tokens=8192` (was unlimited — 54% of
$14K/5-day spend was thinking tokens at $75/M)
- Lowers `max_budget_usd` from $100 → $15 and `max_turns` from 1000 → 50
### Features unlocked by the upgrade
| Feature | SDK | Impact |
|---|---|---|
| `exclude_dynamic_sections` | 0.1.57 | Cross-user prompt cache hits
(see #12758) |
| `AssistantMessage.usage` per-turn | 0.1.49 | Cost attribution per LLM
call |
| `task_budget` | 0.1.51 | Per-task cost ceiling at SDK level |
| `get_context_usage()` | 0.1.52 | Live context-window monitoring |
| MCP large-tool-result fix | 0.1.55 | No more silent truncation >50K
chars |
| MCP HTTP/SSE buffer leak fix | CLI 2.1.97 | Production memory creep
~50 MB/hr |
| 429 retry exponential backoff | CLI 2.1.97 | Rate-limit recovery (was
burning all retries in ~13s) |
| `--resume` cache miss fix | CLI 2.1.90 | Prompt cache works after
resume |
| SDK session quadratic-write fix | CLI 2.1.90 | No more slowdown on
long sessions |
| `max_thinking_tokens` | 0.1.57 | Cap extended thinking cost |
## How
- `build_sdk_env()` in `env.py` injects the env var unconditionally (all
3 auth modes)
- `service.py` passes `max_thinking_tokens` to `ClaudeAgentOptions`
- `config.py` adds 3 new fields with env var overrides
- Regression tests verify both OpenRouter compat issues are handled
## Test plan
- [x] CI green on all test matrices (3.11, 3.12, 3.13)
- [x] `test_disable_experimental_betas_env_var_strips_headers` passes —
verifies env var strips both patterns
- [x] `test_bare_cli_*` correctly XFAILs — documents the CLI regression
exists
- [x] `test_sdk_exposes_max_thinking_tokens_option` guards the new param
- [x] Config validation tests use real temp executables
### Why
LLM token costs are significant, especially for the copilot feature. The
system prompt and tool definitions are the two largest static components
of every request — caching them dramatically reduces input token costs
(cache reads cost 10% of the base input price).
Previously, user-specific context (business understanding) was embedded
directly in the system prompt, making it unique per user and preventing
cache sharing across users or sessions. Every request paid full price
for the system prompt even when the content was functionally identical.
A secondary security concern was identified during review: because the
LLM is instructed to parse `<user_context>` blocks, a user could type a
literal `<user_context>…</user_context>` tag in any message and
potentially spoof or suppress their own personalisation context. This PR
includes a full defence-in-depth fix for that injection vector on the
first turn (including new users with no stored understanding), plus
GET-endpoint stripping so injected context is never surfaced back to the
client.
### What
- **`copilot/service.py`**: Added `USER_CONTEXT_TAG` constant (shared by
writer and reader). Added `_USER_CONTEXT_ANYWHERE_RE` /
`_USER_CONTEXT_PREFIX_RE` regexes, `format_user_context_prefix`,
`strip_user_context_prefix`, `sanitize_user_supplied_context`, and
`_sanitize_user_context_field` helpers. Replaced the old
`_build_cacheable_system_prompt` / `_build_system_prompt` pair with a
single `_build_system_prompt` that returns `(static_prompt,
understanding)`. Added `inject_user_context` which sanitizes user input,
optionally wraps trusted understanding, and persists the result to DB.
- **`copilot/sdk/service.py`**: On first turn calls
`inject_user_context` before `_build_query_message` so the query sees
the prefixed content. Passes `user_id if not has_history else None` to
avoid redundant DB lookups on subsequent turns.
- **`copilot/baseline/service.py`**: Same pattern —
`inject_user_context` called before transcript append and OpenAI message
list construction; `openai_messages` loop patches the first user entry
after injection.
- **`blocks/llm.py`**: System prompt sent as a structured block with
`cache_control: {"type": "ephemeral"}`. `cache_control` placed on the
last tool in the tool list. Guards against empty/whitespace-only system
blocks (Anthropic rejects them). Fixed `anthropic.omit` →
`anthropic.NOT_GIVEN` sentinel for the no-tools case.
- **`api/features/chat/routes.py`**: Added `_strip_injected_context`
which returns a shallow copy of each message with the server-injected
`<user_context>` prefix stripped before the GET `/sessions/{id}`
response, so the prefix is invisible to the frontend.
- **`copilot/db.py`**: Added defence-in-depth `result > 1` error log in
`update_message_content_by_sequence`. Added authorization note
documenting why a `userId` join is not required.
- **`data/db_manager.py`**: Registered
`update_message_content_by_sequence` on both the sync and async DB
manager clients.
### How it works
**Static system prompt**: The system prompt is now identical for every
user. The LLM is instructed to look for a `<user_context>` block in the
first user message when present, and to greet new users warmly when no
context is provided.
**User context injection**: On the first turn of a new session, the
caller's business understanding is prepended to the user's message as
`<user_context>…</user_context>`. The prefixed content is also persisted
to the DB so resumed sessions and page reloads retain personalisation.
**`<user_context>` tag sanitization (security)**: `inject_user_context`
calls `sanitize_user_supplied_context` unconditionally — even when
`understanding` is `None` — so new users cannot smuggle a
`<user_context>` tag to the LLM on the first turn. Fields from the
stored `BusinessUnderstanding` object are escaped with
`_sanitize_user_context_field` so user-controlled free-text cannot break
out of the trusted block. The GET endpoint strips the injected prefix
before returning message history to the client.
**All-turn sanitization**: `strip_user_context_tags` (a public alias of
`sanitize_user_supplied_context`) is called unconditionally on every
incoming message in both the SDK and baseline paths — before
`maybe_append_user_message` — so `<user_context>` tags typed by a user
on any turn (not just the first) are stripped before reaching the LLM.
Lone unpaired tags (e.g. `<user_context>spoof` without a closing tag)
are also caught by a second-pass `_USER_CONTEXT_LONE_TAG_RE`
substitution. The system prompt explicitly states the tag is
server-injected, only trusted on the first message, and must be ignored
on subsequent turns.
**Cache placement**: Per Anthropic's caching model, placing
`cache_control` on the system prompt block caches everything up to and
including it. Placing `cache_control` on the last tool definition caches
all tool schemas as a single prefix. Both cache points are set so
repeated requests from any user can hit both caches.
**Langfuse compatibility**: `_build_system_prompt` calls
`prompt.compile(users_information="")` so existing Langfuse prompt
templates remain static and cacheable.
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] Verify system prompt no longer contains user-specific information
- [x] Verify `<user_context>` block appears in the first user message on
new sessions
- [x] Verify returning users still receive personalised responses via
user context
- [x] Verify Langfuse-sourced prompts compile correctly with empty
`users_information`
- [x] Verify Anthropic API calls include `cache_control` on system block
and last tool
- [x] Verify user-supplied `<user_context>` tags are stripped on the
first turn (including when understanding is None)
- [x] Verify user-supplied `<user_context>` tags are stripped on all
turns (turn 2+ sanitization via `strip_user_context_tags`)
- [x] Verify lone unpaired `<user_context>` tags (no closing tag) are
also stripped
- [x] Verify GET `/sessions/{id}` does not expose the injected
`<user_context>` prefix to the client
---------
Co-authored-by: majdyz <majdy.zamil@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
A transient LaunchDarkly failure returned None from get_subscription_price_id,
which was cached for the full 60-second TTL, blocking subscription upgrades
until expiry. Adding cache_none=False ensures None is never stored in the cache
so the next call retries LD immediately.
Adds a regression test verifying that two consecutive calls where the first
returns None (LD transient error) and the second returns the real price ID
both hit LD, confirming the None sentinel is not cached.
Flagged by sentry[bot] (credit.py:1352, Severity: MEDIUM).
Since get_subscription_price_id is now @cached, in-memory cache state can
persist between tests in the same process and cause false cache hits. Call
cache_clear() before and after tests that call the function directly to
ensure each test exercises a fresh LD flag lookup.
