mirror of
https://github.com/Significant-Gravitas/AutoGPT.git
synced 2026-02-10 23:05:17 -05:00
e596ea87cbae55eedba3bc58f144702e9e310504
88 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Nicholas Tindle
|
7668c17d9c |
feat(platform): add User Workspace for persistent CoPilot file storage (#11867)
Implements persistent User Workspace storage for CoPilot, enabling
blocks to save and retrieve files across sessions. Files are stored in
session-scoped virtual paths (`/sessions/{session_id}/`).
Fixes SECRT-1833
### Changes 🏗️
**Database & Storage:**
- Add `UserWorkspace` and `UserWorkspaceFile` Prisma models
- Implement `WorkspaceStorageBackend` abstraction (GCS for cloud, local
filesystem for self-hosted)
- Add `workspace_id` and `session_id` fields to `ExecutionContext`
**Backend API:**
- Add REST endpoints: `GET/POST /api/workspace/files`, `GET/DELETE
/api/workspace/files/{id}`, `GET /api/workspace/files/{id}/download`
- Add CoPilot tools: `list_workspace_files`, `read_workspace_file`,
`write_workspace_file`
- Integrate workspace storage into `store_media_file()` - returns
`workspace://file-id` references
**Block Updates:**
- Refactor all file-handling blocks to use unified `ExecutionContext`
parameter
- Update media-generating blocks to persist outputs to workspace
(AIImageGenerator, AIImageCustomizer, FluxKontext, TalkingHead, FAL
video, Bannerbear, etc.)
**Frontend:**
- Render `workspace://` image references in chat via proxy endpoint
- Add "AI cannot see this image" overlay indicator
**CoPilot Context Mapping:**
- Session = Agent (graph_id) = Run (graph_exec_id)
- Files scoped to `/sessions/{session_id}/`
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [ ] I have tested my changes according to the test plan:
- [ ] Create CoPilot session, generate image with AIImageGeneratorBlock
- [ ] Verify image returns `workspace://file-id` (not base64)
- [ ] Verify image renders in chat with visibility indicator
- [ ] Verify workspace files persist across sessions
- [ ] Test list/read/write workspace files via CoPilot tools
- [ ] Test local storage backend for self-hosted deployments
#### For configuration changes:
- [x] `.env.default` is updated or already compatible with my changes
- [x] `docker-compose.yml` is updated or already compatible with my
changes
- [x] I have included a list of my configuration changes in the PR
description (under **Changes**)
🤖 Generated with [Claude Code](https://claude.ai/code)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Introduces a new persistent file-storage surface area (DB tables,
storage backends, download API, and chat tools) and rewires
`store_media_file()`/block execution context across many blocks, so
regressions could impact file handling, access control, or storage
costs.
>
> **Overview**
> Adds a **persistent per-user Workspace** (new
`UserWorkspace`/`UserWorkspaceFile` models plus `WorkspaceManager` +
`WorkspaceStorageBackend` with GCS/local implementations) and wires it
into the API via a new `/api/workspace/files/{file_id}/download` route
(including header-sanitized `Content-Disposition`) and shutdown
lifecycle hooks.
>
> Extends `ExecutionContext` to carry execution identity +
`workspace_id`/`session_id`, updates executor tooling to clone
node-specific contexts, and updates `run_block` (CoPilot) to create a
session-scoped workspace and synthetic graph/run/node IDs.
>
> Refactors `store_media_file()` to require `execution_context` +
`return_format` and to support `workspace://` references; migrates many
media/file-handling blocks and related tests to the new API and to
persist generated media as `workspace://...` (or fall back to data URIs
outside CoPilot), and adds CoPilot chat tools for
listing/reading/writing/deleting workspace files with safeguards against
context bloat.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
|
||
|
Nicholas Tindle
|
0953983944 |
feat(platform): disable onboarding redirects and add $5 signup bonus (#11862)
Disable automatic onboarding redirects on signup/login while keeping the
checklist/wallet functional. Users now receive $5 (500 credits) on their
first visit to /copilot.
### Changes 🏗️
- **Frontend**: `shouldShowOnboarding()` now returns `false`, disabling
auto-redirects to `/onboarding`
- **Backend**: Added `VISIT_COPILOT` onboarding step with 500 credit
($5) reward
- **Frontend**: Copilot page automatically completes `VISIT_COPILOT`
step on mount
- **Database**: Migration to add `VISIT_COPILOT` to `OnboardingStep`
enum
NOTE: /onboarding/1-welcome -> /library now as shouldShowOnboardin is
always false
Users land directly on `/copilot` after signup/login and receive $5
invisibly (not shown in checklist UI).
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] New user signup (email/password) → lands on `/copilot`, wallet
shows 500 credits
- [x] Verified credits are only granted once (idempotent via onboarding
reward mechanism)
- [x] Existing user login (already granted flag set) → lands on
`/copilot`, no duplicate credits
- [x] Checklist/wallet remains functional
#### For configuration changes:
- [x] `.env.default` is updated or already compatible with my changes
- [x] `docker-compose.yml` is updated or already compatible with my
changes
- [x] I have included a list of my configuration changes in the PR
description (under **Changes**)
No configuration changes required.
---
OPEN-2967
🤖 Generated with [Claude Code](https://claude.ai/code)
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> Introduces a new onboarding step and adjusts onboarding flow.
>
> - Adds `VISIT_COPILOT` onboarding step (+500 credits) with DB enum
migration and API/type updates
> - Copilot page auto-completes `VISIT_COPILOT` on mount to grant the
welcome bonus
> - Changes `/onboarding/enabled` to require user context and return
`false` when `CHAT` feature is enabled (skips legacy onboarding)
> - Wallet now refreshes credits on any onboarding `step_completed`
notification; confetti limited to visible tasks
> - Test flows updated to accept redirects to `copilot`/`library` and
verify authenticated state
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
|
||
|
Zamil Majdy
|
fb58827c61 |
feat(backend;frontend): Implement node-specific auto-approval, safety popup, and race condition fixes (#11810)
## Summary
This PR implements comprehensive improvements to the human-in-the-loop
(HITL) review system, including safety features, architectural changes,
and bug fixes:
### Key Features
- **SECRT-1798: One-time safety popup** - Shows informational popup
before first run of AI-generated agents with sensitive actions/HITL
blocks
- **SECRT-1795: Auto-approval toggle UX** - Toggle in pending reviews
panel to auto-approve future actions from the same node
- **Node-specific auto-approval** - Changed from execution-specific to
node-specific using special key pattern
`auto_approve_{graph_exec_id}_{node_id}`
- **Consolidated approval checking** - Merged `check_auto_approval` into
`check_approval` using single OR query for better performance
- **Race condition prevention** - Added execution status check before
resuming to prevent duplicate execution when approving while graph is
running
- **Parallel auto-approval creation** - Uses `asyncio.gather` for better
performance when creating multiple auto-approval records
## Changes
### Backend Architecture
- **`human_review.py`**:
- Added `check_approval()` function that checks both normal and
auto-approval in single query
- Added `create_auto_approval_record()` for node-specific auto-approval
using special key pattern
- Added `get_auto_approve_key()` helper to generate consistent
auto-approval keys
- **`review/routes.py`**:
- Added execution status check before resuming to prevent race
conditions
- Refactored auto-approval record creation to use parallel execution
with `asyncio.gather`
- Removed obvious comments for cleaner code
- **`review/model.py`**: Added `auto_approve_future_actions` field to
`ReviewRequest`
- **`blocks/helpers/review.py`**: Updated to use consolidated
`check_approval` via database manager client
- **`executor/database.py`**: Exposed `check_approval` through
DatabaseManager RPC for block execution context
- **`data/block.py`**: Fixed safe mode checks for sensitive action
blocks
### Frontend
- **New `AIAgentSafetyPopup`** component with localStorage-based
one-time display
- **`PendingReviewsList`**:
- Replaced "Approve all future actions" button with toggle
- Toggle resets data to original values and disables editing when
enabled
- Shows warning message explaining auto-approval behavior
- **`RunAgentModal`**: Integrated safety popup before first run
- **`usePendingReviews`**: Added polling for real-time badge updates
- **`FloatingSafeModeToggle` & `SafeModeToggle`**: Simplified visibility
logic
- **`local-storage.ts`**: Added localStorage key for popup state
tracking
### Bug Fixes
- Fixed "Client is not connected to query engine" error by using
database manager client pattern
- Fixed race condition where approving reviews while graph is RUNNING
could queue execution twice
- Fixed migration to only drop FK constraint, not non-existent column
- Fixed card data reset when auto-approve toggle changes
### Code Quality
- Removed duplicate/obvious comments
- Moved imports to top-level instead of local scope in tests
- Used walrus operator for cleaner conditional assignments
- Parallel execution for auto-approval record creation
## Test plan
- [ ] Create an AI-generated agent with sensitive actions (e.g., email
sending)
- [ ] First run should show the safety popup before starting
- [ ] Subsequent runs should not show the popup
- [ ] Clear localStorage (`AI_AGENT_SAFETY_POPUP_SHOWN`) to verify popup
shows again
- [ ] Create an agent with human-in-the-loop blocks
- [ ] Run it and verify the pending reviews panel appears
- [ ] Enable the "Auto-approve all future actions" toggle
- [ ] Verify editing is disabled and shows warning message
- [ ] Click "Approve" and verify subsequent blocks from same node
auto-approve
- [ ] Verify auto-approval persists across multiple executions of same
graph
- [ ] Disable toggle and verify editing works normally
- [ ] Verify "Reject" button still works regardless of toggle state
- [ ] Test race condition: Approve reviews while graph is RUNNING
(should skip resume)
- [ ] Test race condition: Approve reviews while graph is REVIEW (should
resume)
- [ ] Verify pending reviews badge updates in real-time when new reviews
are created
|
||
|
Zamil Majdy
|
f31c160043 |
feat(platform): add endedAt field and fix execution analytics timestamps (#11759)
## Summary
This PR adds proper execution end time tracking and fixes timestamp
handling throughout the execution analytics system.
### Key Changes
1. **Added `endedAt` field to database schema** - Executions now have a
dedicated field for tracking when they finish
2. **Fixed timestamp nullable handling** - `started_at` and `ended_at`
are now properly nullable in types
3. **Fixed chart aggregation** - Reduced threshold from ≥3 to ≥1
executions per day
4. **Improved timestamp display** - Moved timestamps to expandable
details section in analytics table
5. **Fixed nullable timestamp bugs** - Updated all frontend code to
handle null timestamps correctly
## Problem Statement
### Issue 1: Missing Execution End Times
Previously, executions used `updatedAt` (last DB update) as a proxy for
"end time". This broke when adding correctness scores retroactively -
the end time would change to whenever the score was added, not when the
execution actually finished.
### Issue 2: Chart Shows Only One Data Point
The accuracy trends chart showed only one data point despite having
executions across multiple days. Root cause: aggregation required ≥3
executions per day.
### Issue 3: Incorrect Type Definitions
Manually maintained types defined `started_at` and `ended_at` as
non-nullable `Date`, contradicting reality where QUEUED executions
haven't started yet.
## Solution
### Database Schema (`schema.prisma`)
```prisma
model AgentGraphExecution {
// ...
startedAt DateTime?
endedAt DateTime? // NEW FIELD
// ...
}
```
### Execution Lifecycle
- **QUEUED**: `startedAt = null`, `endedAt = null` (not started)
- **RUNNING**: `startedAt = set`, `endedAt = null` (in progress)
- **COMPLETED/FAILED/TERMINATED**: `startedAt = set`, `endedAt = set`
(finished)
### Migration Strategy
```sql
-- Add endedAt column
ALTER TABLE "AgentGraphExecution" ADD COLUMN "endedAt" TIMESTAMP(3);
-- Backfill ONLY terminal executions (prevents marking RUNNING executions as ended)
UPDATE "AgentGraphExecution"
SET "endedAt" = "updatedAt"
WHERE "endedAt" IS NULL
AND "executionStatus" IN ('COMPLETED', 'FAILED', 'TERMINATED');
```
## Changes by Component
### Backend
**`schema.prisma`**
- Added `endedAt` field to `AgentGraphExecution`
**`execution.py`**
- Made `started_at` and `ended_at` optional with Field descriptions
- Updated `from_db()` to use `endedAt` instead of `updatedAt`
- `update_graph_execution_stats()` sets `endedAt` when status becomes
terminal
**`execution_analytics_routes.py`**
- Removed `created_at`/`updated_at` from `ExecutionAnalyticsResult` (DB
metadata, not execution data)
- Kept only `started_at`/`ended_at` (actual execution runtime)
- Made settings global (avoid recreation)
- Moved OpenAI key validation to `_process_batch` (only check when LLM
actually runs)
**`analytics.py`**
- Fixed aggregation: `COUNT(*) >= 1` (was 3) - include all days with ≥1
execution
- Uses `createdAt` for chart grouping (when execution was queued)
**`late_execution_monitor.py`**
- Handle optional `started_at` with fallback to `datetime.min` for
sorting
- Display "Not started" when `started_at` is null
### Frontend
**Type Definitions**
- Fixed manually maintained `types.ts`: `started_at: Date | null` (was
non-nullable)
- Generated types were already correct
**Analytics Components**
- `AnalyticsResultsTable.tsx`: Show only `started_at`/`ended_at` in
2-column expandable grid
- `ExecutionAnalyticsForm.tsx`: Added filter explanation UI
**Monitoring Components** - Fixed null handling bugs:
- `OldAgentLibraryView.tsx`: Handle null in reduce function
- `agent-runs-selector-list.tsx`: Safe sorting with `?.getTime() ?? 0`
- `AgentFlowList.tsx`: Filter/sort with null checks
- `FlowRunsStatus.tsx`: Filter null timestamps
- `FlowRunsTimeline.tsx`: Filter executions with null timestamps before
rendering
- `monitoring/page.tsx`: Safe sorting
- `ActivityItem.tsx`: Fallback to "recently" for null timestamps
## Benefits
✅ **Accurate End Times**: `endedAt` is frozen when execution finishes,
not updated later
✅ **Type Safety**: Nullable types match reality, exposing real bugs
✅ **Better UX**: Chart shows all days with data (not just days with ≥3
executions)
✅ **Bug Fixes**: 7+ frontend components now handle null timestamps
correctly
✅ **Documentation**: Field descriptions explain when timestamps are null
## Testing
### Backend
```bash
cd autogpt_platform/backend
poetry run format # ✅ All checks passed
poetry run lint # ✅ All checks passed
```
### Frontend
```bash
cd autogpt_platform/frontend
pnpm format # ✅ All checks passed
pnpm lint # ✅ All checks passed
pnpm types # ✅ All type errors fixed
```
### Test Data Generation
Created script to generate 35 test executions across 7 days with
correctness scores:
```bash
poetry run python scripts/generate_test_analytics_data.py
```
## Migration Notes
⚠️ **Important**: The migration only backfills `endedAt` for executions
with terminal status (COMPLETED, FAILED, TERMINATED). Active executions
(QUEUED, RUNNING) correctly keep `endedAt = null`.
## Breaking Changes
None - this is backward compatible:
- `endedAt` is nullable, existing code that doesn't use it is unaffected
- Frontend already used generated types which were correct
- Migration safely backfills historical data
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> Introduces explicit execution end-time tracking and normalizes
timestamp handling across backend and frontend.
>
> - Adds `endedAt` to `AgentGraphExecution` (schema + migration);
backfills terminal executions; sets `endedAt` on terminal status updates
> - Makes `GraphExecutionMeta.started_at/ended_at` optional; updates
`from_db()` to use DB `endedAt`; exposes timestamps in
`ExecutionAnalyticsResult`
> - Moves OpenAI key validation into batch processing; instantiates
`Settings` once
> - Accuracy trends: reduce daily aggregation threshold to `>= 1`;
optional historical series
> - Monitoring/analytics UI: results table shows/export
`started_at`/`ended_at`; adds chart filter explainer
> - Frontend null-safety: update types (`Date | null`) and fix
sorting/filtering/rendering for nullable timestamps across monitoring
and library views
> - Late execution monitor: safe sorting/display when `started_at` is
null
> - OpenAPI specs updated for new/nullable fields
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
|
||
|
Zamil Majdy
|
8b83bb8647 |
feat(backend): unified hybrid search with embedding backfill for all content types (#11767)
## Summary This PR extends the embedding system to support **blocks** and **documentation** content types in addition to store agents, and introduces **unified hybrid search** across all content types using a single `UnifiedContentEmbedding` table. ### Key Changes 1. **Unified Hybrid Search Architecture** - Added `search` tsvector column to `UnifiedContentEmbedding` table - New `unified_hybrid_search()` function searches across all content types (agents, blocks, docs) - Updated `hybrid_search()` for store agents to use `UnifiedContentEmbedding.search` - Removed deprecated `search` column from `StoreListingVersion` table 2. **Pluggable Content Handler Architecture** - Created abstract `ContentHandler` base class for extensibility - Implemented handlers: `StoreAgentHandler`, `BlockHandler`, `DocumentationHandler` - Registry pattern for easy addition of new content types 3. **Block Embeddings** - Discovers all blocks using `get_blocks()` - Extracts searchable text from: name, description, categories, input/output schemas 4. **Documentation Embeddings** - Scans `/docs/` directory for `.md` and `.mdx` files - Extracts title from first `#` heading or uses filename as fallback 5. **Hybrid Search Graceful Degradation** - Falls back to lexical-only search if query embedding generation fails - Redistributes semantic weight proportionally to other components - Logs warning instead of throwing error 6. **Database Migrations** - `20260115200000_add_unified_search_tsvector`: Adds search column to UnifiedContentEmbedding with auto-update trigger - `20260115210000_remove_storelistingversion_search`: Removes deprecated search column and updates StoreAgent view 7. **Orphan Cleanup** - `cleanup_orphaned_embeddings()` removes embeddings for deleted content - Always runs after backfill, even at 100% coverage ### Review Comments Addressed - ✅ SQL parameter index bug when user_id provided (embeddings.py) - ✅ Early return skipping cleanup at 100% coverage (scheduler.py) - ✅ Inconsistent return structure across code paths (scheduler.py) - ✅ SQL UNION syntax error - added parentheses for ORDER BY/LIMIT (hybrid_search.py) - ✅ Version numeric ordering in aggregations (migration) - ✅ Embedding dimension uses EMBEDDING_DIM constant ### Files Changed - `backend/api/features/store/content_handlers.py` (NEW): Handler architecture - `backend/api/features/store/embeddings.py`: Refactored to use handlers - `backend/api/features/store/hybrid_search.py`: Unified search + graceful degradation - `backend/executor/scheduler.py`: Process all content types, consistent returns - `migrations/20260115200000_add_unified_search_tsvector/`: Add tsvector to unified table - `migrations/20260115210000_remove_storelistingversion_search/`: Remove old search column - `schema.prisma`: Updated UnifiedContentEmbedding and StoreListingVersion models - `*_test.py`: Added tests for unified_hybrid_search ## Test Plan 1. ✅ All tests passing on Python 3.11, 3.12, 3.13 2. ✅ Types check passing 3. ✅ CodeRabbit and Sentry reviews addressed 4. Deploy to staging and verify: - Backfill job processes all content types - Search results include blocks and docs - Search works without OpenAI API (graceful degradation) 🤖 Generated with [Claude Code](https://claude.ai/code) --------- Co-authored-by: Swifty <craigswift13@gmail.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> |
||
|
Swifty
|
3b1b2fe30c |
feat(backend): Extract backend copilot/chat enhancements from hackathon (#11719)
This PR extracts backend changes from the hackathon/copilot branch, adding enhanced chat capabilities, agent management tools, store embeddings, and hybrid search functionality. ### Changes 🏗️ **Chat Features:** - Added chat database layer (`db.py`) for conversation and message persistence - Extended chat models with new types and response structures - New onboarding system prompt for guided user experiences - Enhanced chat routes with additional endpoints - Expanded chat service with more capabilities **Chat Agent Tools:** - `agent_output.py` - Handle agent execution outputs - `create_agent.py` - Tool for creating new agents via chat - `edit_agent.py` - Tool for modifying existing agents - `find_library_agent.py` - Search and discover library agents - Enhanced `run_agent.py` with additional functionality - New `models.py` for shared tool types **Store Enhancements:** - `embeddings.py` - Vector embeddings support for semantic search - `hybrid_search.py` - Combined keyword and semantic search - `backfill_embeddings.py` - Utility for backfilling existing data - Updated store database operations **Admin:** - Enhanced store admin routes **Data Layer:** - New `understanding.py` module for agent understanding/context **Database Migrations:** - `add_chat_tables` - Chat conversation and message tables - `add_store_embeddings` - Embeddings storage for store items - `enhance_search` - Search index improvements ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] Chat endpoints respond correctly - [x] Agent tools (create/edit/find/run) function properly - [x] Store embeddings and hybrid search work - [x] Database migrations apply cleanly #### For configuration changes: - [x] `.env.default` is updated or already compatible with my changes - [x] `docker-compose.yml` is updated or already compatible with my changes - [x] I have included a list of my configuration changes in the PR description (under **Changes**) --------- Co-authored-by: Torantulino <40276179@live.napier.ac.uk> |
||
|
Swifty
|
5ac941fe2f |
feat(backend): add hybrid search for store listings, docs and blocks (#11721)
This PR adds hybrid search functionality combining semantic embeddings with traditional text search for improved store listing discovery. ### Changes 🏗️ - Add `embeddings.py` - OpenAI-based embedding generation and similarity search - Add `hybrid_search.py` - Combines vector similarity with text matching for better search results - Add `backfill_embeddings.py` - Script to generate embeddings for existing store listings - Update `db.py` - Integrate hybrid search into store database queries - Update `schema.prisma` - Add embedding storage fields and indexes - Add migrations for embedding columns and HNSW index for vector search ### Architecture Decisions 🏛️ **Fail-Fast Approach (No Silent Fallbacks)** We explicitly chose NOT to implement graceful degradation when hybrid search fails. Here's why: ✅ **Benefits:** - Errors surface immediately → faster fixes - Tests verify hybrid search actually works (not just fallback) - Consistent search quality for all users - Forces proper infrastructure setup (API keys, database) ❌ **Why Not Fallback:** - Silent degradation hides production issues - Users get inconsistent results without knowing why - Tests can pass even when hybrid search is broken - Reduces operational visibility **How We Prevent Failures:** 1. Embedding generation in approval flow (db.py:1545) 2. Error logging with `logger.error` (not warning) 3. Clear error messages (ValueError explains what's wrong) 4. Comprehensive test coverage (9/9 tests passing) If embeddings fail, it indicates a real infrastructure issue (missing API key, OpenAI down, database issues) that needs immediate attention, not silent degradation. ### Test Coverage ✅ **All tests passing (1625 total):** - 9/9 hybrid_search tests (including fail-fast validation) - 3/3 db search integration tests - Full schema compatibility (public/platform schemas) - Error handling verification ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] Test hybrid search returns relevant results - [x] Test embedding generation for new listings - [x] Test backfill script on existing data - [x] Verify search performance with embeddings - [x] Test fail-fast behavior when embeddings unavailable #### For configuration changes: - [x] `.env.default` is updated or already compatible with my changes - [x] `docker-compose.yml` is updated or already compatible with my changes - [x] Configuration: Requires `openai_internal_api_key` in secrets --------- Co-authored-by: Zamil Majdy <zamil.majdy@agpt.co> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> |
||
|
Zamil Majdy
|
88731b1f76 |
feat(platform): marketplace update notifications with enhanced publishing workflow (#11630)
## Summary This PR implements a comprehensive marketplace update notification system that allows users to discover and update to newer agent versions, along with enhanced publishing workflows and UI improvements. <img width="1500" height="533" alt="image" src="https://github.com/user-attachments/assets/ee331838-d712-4718-b231-1f9ec21bcd8e" /> <img width="600" height="610" alt="image" src="https://github.com/user-attachments/assets/b881a7b8-91a5-460d-a159-f64765b339f1" /> <img width="1500" height="416" alt="image" src="https://github.com/user-attachments/assets/a2d61904-2673-4e44-bcc5-c47d36af7a38" /> <img width="1500" height="1015" alt="image" src="https://github.com/user-attachments/assets/2dd978c7-20cc-4230-977e-9c62157b9f23" /> ## Core Features ### 🔔 Marketplace Update Notifications - **Update detection**: Automatically detects when marketplace has newer agent versions than user's local copy - **Creator notifications**: Shows banners for creators with unpublished changes ready to publish - **Non-creator support**: Enables regular users to discover and update to newer marketplace versions - **Version comparison**: Intelligent logic comparing `graph_version` vs marketplace listing versions ### 📋 Enhanced Publishing Workflow - **Builder integration**: Added "Publish to Marketplace" button directly in the builder actions - **Unified banner system**: Consistent `MarketplaceBanners` component across library and marketplace pages - **Streamlined UX**: Fixed layout issues, improved button placement and styling - **Modal improvements**: Fixed thumbnail loading race conditions and infinite loop bugs ### 📚 Version History & Changelog - **Inline version history**: Added version changelog directly to marketplace agent pages - **Version comparison**: Clear display of available versions with current version highlighting - **Update mechanism**: Direct updates using `graph_version` parameter for accuracy ## Technical Implementation ### Backend Changes - **Database schema**: Added `agentGraphVersions` and `agentGraphId` fields to `StoreAgent` model - **API enhancement**: Updated store endpoints to expose graph version data for version comparison - **Data migration**: Fixed agent version field naming from `version` to `agentGraphVersions` - **Model updates**: Enhanced `LibraryAgentUpdateRequest` with `graph_version` field ### Frontend Architecture - **`useMarketplaceUpdate` hook**: Centralized marketplace update detection and creator identification - **`MarketplaceBanners` component**: Unified banner system with proper vertical layout and styling - **`AgentVersionChangelog` component**: Version history display for marketplace pages - **`PublishToMarketplace` component**: Builder integration with modal workflow ### Key Bug Fixes - **Thumbnail loading**: Fixed race condition where images wouldn't load on first modal open - **Infinite loops**: Used refs to prevent circular dependencies in `useThumbnailImages` hook - **Layout issues**: Fixed banner placement, removed duplicate breadcrumbs, corrected vertical layout - **Field naming**: Fixed `agent_version` vs `version` field inconsistencies across APIs ## Files Changed ### Backend - `autogpt_platform/backend/backend/server/v2/store/` - Enhanced store API with graph version data - `autogpt_platform/backend/backend/server/v2/library/` - Updated library API models - `autogpt_platform/backend/migrations/` - Database migrations for version fields - `autogpt_platform/backend/schema.prisma` - Schema updates for graph versions ### Frontend - `src/app/(platform)/components/MarketplaceBanners/` - New unified banner component - `src/app/(platform)/library/agents/[id]/components/` - Enhanced library views with banners - `src/app/(platform)/build/components/BuilderActions/` - Added marketplace publish button - `src/app/(platform)/marketplace/components/AgentInfo/` - Added inline version history - `src/components/contextual/PublishAgentModal/` - Fixed thumbnail loading and modal workflow ## User Experience Impact - **Better discovery**: Users automatically notified of newer agent versions - **Streamlined publishing**: Direct publish access from builder interface - **Reduced friction**: Fixed UI bugs, improved loading states, consistent design - **Enhanced transparency**: Inline version history on marketplace pages - **Creator workflow**: Better notifications for creators with unpublished changes ## Testing - ✅ Update banners appear correctly when marketplace has newer versions - ✅ Creator banners show for users with unpublished changes - ✅ Version comparison logic works with graph_version vs marketplace versions - ✅ Publish button in builder opens modal correctly with pre-populated data - ✅ Thumbnail images load properly on first modal open without infinite loops - ✅ Database migrations completed successfully with version field fixes - ✅ All existing tests updated and passing with new schema changes 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Lluis Agusti <hi@llu.lu> Co-authored-by: Ubbe <hi@ubbe.dev> Co-authored-by: Reinier van der Leer <pwuts@agpt.co> |
||
|
Reinier van der Leer
|
3dbc03e488 |
feat(platform): OAuth API & Single Sign-On (#11617)
We want to provide Single Sign-On for multiple AutoGPT apps that use the Platform as their backend. ### Changes 🏗️ Backend: - DB + logic + API for OAuth flow (w/ tests) - DB schema additions for OAuth apps, codes, and tokens - Token creation/validation/management logic - OAuth flow endpoints (app info, authorize, token exchange, introspect, revoke) - E2E OAuth API integration tests - Other OAuth-related endpoints (upload app logo, list owned apps, external `/me` endpoint) - App logo asset management - Adjust external API middleware to support auth with access token - Expired token clean-up job - Add `OAUTH_TOKEN_CLEANUP_INTERVAL_HOURS` setting (optional) - `poetry run oauth-tool`: dev tool to test the OAuth flows and register new OAuth apps - `poetry run export-api-schema`: dev tool to quickly export the OpenAPI schema (much quicker than spinning up the backend) Frontend: - Frontend UI for app authorization (`/auth/authorize`) - Re-redirect after login/signup - Frontend flow to batch-auth integrations on request of the client app (`/auth/integrations/setup-wizard`) - Debug `CredentialInputs` component - Add `/profile/oauth-apps` management page - Add `isOurProblem` flag to `ErrorCard` to hide action buttons when the error isn't our fault - Add `showTitle` flag to `CredentialsInput` to hide built-in title for layout reasons DX: - Add [API guide](https://github.com/Significant-Gravitas/AutoGPT/blob/pwuts/sso/docs/content/platform/integrating/api-guide.md) and [OAuth guide](https://github.com/Significant-Gravitas/AutoGPT/blob/pwuts/sso/docs/content/platform/integrating/oauth-guide.md) ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] Manually verify test coverage of OAuth API tests - Test `/auth/authorize` using `poetry run oauth-tool test-server` - [x] Works - [x] Looks okay - Test `/auth/integrations/setup-wizard` using `poetry run oauth-tool test-server` - [x] Works - [x] Looks okay - Test `/profile/oauth-apps` page - [x] All owned OAuth apps show up - [x] Enabling/disabling apps works - [ ] ~~Uploading logos works~~ can only test this once deployed to dev #### For configuration changes: - [x] `.env.default` is updated or already compatible with my changes - [x] `docker-compose.yml` is updated or already compatible with my changes - [x] I have included a list of my configuration changes in the PR description (under **Changes**) |
||
|
Krzysztof Czerwinski
|
bd37fe946d |
feat(platform): Builder search history (#11457)
Preserve user searches in the new builder and cache search results for more efficiency. Search is saved, so the user can see their previous searches. ### Changes 🏗️ - Add `BuilderSearch` column&migration to save user search (with all filters) - Builder `db.py` now caches all search results using `@cached` and returns paginated results, so following pages are returned much quicker - Score and sort results - Update models&routes - Update frontend, so it works properly with modified endpoints - Frontend: store `serachId` and use it for subsequent searches, so we don't save partial searches (e.g. "b", "bl", ..., "block"). Search id is reset when user clears the search field. - Add clickable chips to the Suggestions builder tab - Add `HorizontalScroll` component (chips use it) ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] Search works and is cached - [x] Search sorts results - [x] Searches are preserved properly --------- Co-authored-by: Reinier van der Leer <pwuts@agpt.co> |
||
|
Zamil Majdy
|
e4d0dbc283 |
feat(platform): add Agent Output Demo field to marketplace submission form (#11538)
## Summary - Add Agent Output Demo field to marketplace agent submission form, positioned below the Description field - Store agent output demo URLs in database for future CoPilot integration - Implement proper video/image ordering on marketplace pages - Add shared YouTube URL validation utility to eliminate code duplication ## Changes Made ### Frontend - **Agent submission form**: Added Agent Output Demo field with YouTube URL validation - **Edit agent form**: Added Agent Output Demo field for existing submissions - **Marketplace display**: Implemented proper video/image ordering: 1. YouTube/Overview video (if exists) 2. First image (hero) 3. Agent Output Demo (if exists) 4. Additional images - **Shared utilities**: Created `validateYouTubeUrl` function in `src/lib/utils.ts` ### Backend - **Database schema**: Added `agentOutputDemoUrl` field to `StoreListingVersion` model - **Database views**: Updated `StoreAgent` view to include `agent_output_demo` field - **API models**: Added `agent_output_demo_url` to submission requests and `agent_output_demo` to responses - **Database migration**: Added migration to create new column and update view - **Test files**: Updated all test files to include the new required field ## Test Plan - [x] Frontend form validation works correctly for YouTube URLs - [x] Database migration applies successfully - [x] Backend API accepts and returns the new field - [x] Marketplace displays videos in correct order - [x] Both frontend and backend formatting/linting pass - [x] All test files include required field to prevent failures 🤖 Generated with [Claude Code](https://claude.ai/code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Swifty
|
2cb6fd581c |
feat(platform): Integration management from external api (#11472)
Allow the external api to manage credentials
### Changes 🏗️
- add ability to external api to manage credentials
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
<!-- Put your test plan here: -->
- [x] tested it works
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> Introduces external API endpoints to manage integrations (OAuth
initiation/completion and credential CRUD), adds external OAuth state
fields, and new API key permissions/config.
>
> - **External API – Integrations**:
> - Add router `backend/server/external/routes/integrations.py` with
endpoints to:
> - `GET /v1/integrations/providers` list providers (incl. default
scopes)
> - `POST /v1/integrations/{provider}/oauth/initiate` and `POST
/oauth/complete` for external OAuth (custom callback, state)
> - `GET /v1/integrations/credentials` and `GET /{provider}/credentials`
to list credentials
> - `POST /{provider}/credentials` to create `api_key`, `user_password`,
`host_scoped` creds; `DELETE /{provider}/credentials/{cred_id}` to
delete
> - Wire router in `backend/server/external/api.py`.
> - **Auth/Permissions**:
> - Add `APIKeyPermission` values: `MANAGE_INTEGRATIONS`,
`READ_INTEGRATIONS`, `DELETE_INTEGRATIONS` (schema + migration +
OpenAPI).
> - **Data model / Store**:
> - Extend `OAuthState` with external-flow fields: `callback_url`,
`state_metadata`, `api_key_id`, `is_external`.
> - Update `IntegrationCredentialsStore.store_state_token(...)` to
accept/store external OAuth metadata.
> - **OAuth providers**:
> - Set GitHub handler `DEFAULT_SCOPES = ["repo"]` in
`integrations/oauth/github.py`.
> - **Config**:
> - Add `config.external_oauth_callback_origins` in
`backend/util/settings.py` to validate allowed OAuth callback origins.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
|
||
|
Zamil Majdy
|
7b951c977e |
feat(platform): implement graph-level Safe Mode toggle for HITL blocks (#11455)
## Summary This PR implements a graph-level Safe Mode toggle system for Human-in-the-Loop (HITL) blocks. When Safe Mode is ON (default), HITL blocks require manual review before proceeding. When OFF, they execute automatically. ## 🔧 Backend Changes - **Database**: Added `metadata` JSON column to `AgentGraph` table with migration - **API**: Updated `execute_graph` endpoint to accept `safe_mode` parameter - **Execution**: Enhanced execution context to use graph metadata as default with API override capability - **Auto-detection**: Automatically populate `has_human_in_the_loop` for graphs containing HITL blocks - **Block Detection**: HITL block ID: `8b2a7b3c-6e9d-4a5f-8c1b-2e3f4a5b6c7d` ## 🎨 Frontend Changes - **Component**: New `FloatingSafeModeToggle` with dual variants: - **White variant**: For library pages, integrates with action buttons - **Black variant**: For builders, floating positioned - **Integration**: Added toggles to both new/legacy builders and library pages - **API Integration**: Direct graph metadata updates via `usePutV1UpdateGraphVersion` - **Query Management**: React Query cache invalidation for consistent UI updates - **Conditional Display**: Toggle only appears when graph contains HITL blocks ## 🛠 Technical Implementation - **Safe Mode ON** (default): HITL blocks require manual review before proceeding - **Safe Mode OFF**: HITL blocks execute automatically without intervention - **Priority**: Backend API `safe_mode` parameter takes precedence over graph metadata - **Detection**: Auto-populates `has_human_in_the_loop` metadata field - **Positioning**: Proper z-index and responsive positioning for floating elements ## 🚧 Known Issues (Work in Progress) ### High Priority - [ ] **Toggle state persistence**: Always shows "ON" regardless of actual state - query invalidation issue - [ ] **LibraryAgent metadata**: Missing metadata field causing TypeScript errors - [ ] **Tooltip z-index**: Still covered by some UI elements despite high z-index ### Medium Priority - [ ] **HITL detection**: Logic needs improvement for reliable block detection - [ ] **Error handling**: Removing HITL blocks from graph causes save errors - [ ] **TypeScript**: Fix type mismatches between GraphModel and LibraryAgent ### Low Priority - [ ] **Frontend API**: Add `safe_mode` parameter to execution calls once OpenAPI is regenerated - [ ] **Performance**: Consider debouncing rapid toggle clicks ## 🧪 Test Plan - [ ] Verify toggle appears only when graph has HITL blocks - [ ] Test toggle persistence across page refreshes - [ ] Confirm API calls update graph metadata correctly - [ ] Validate execution behavior respects safe mode setting - [ ] Check styling consistency across builder and library contexts ## 🔗 Related - Addresses requirements for graph-level HITL configuration - Builds on existing FloatingReviewsPanel infrastructure - Integrates with existing graph metadata system 🤖 Generated with [Claude Code](https://claude.ai/code) |
||
|
Swifty
|
00148f4e3d |
feat(platform): add external api routes for store search and tool usage (#11463)
We want to allow external tools to explore the marketplace and use the chat agent tools ### Changes 🏗️ - add store api routes - add tool api routes ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: <!-- Put your test plan here: --> - [x] tested all endpoints work --------- Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> |
||
|
Zamil Majdy
|
3d08c22dd5 |
feat(platform): add Human In The Loop block with review workflow (#11380)
## Summary This PR implements a comprehensive Human In The Loop (HITL) block that allows agents to pause execution and wait for human approval/modification of data before continuing. https://github.com/user-attachments/assets/c027d731-17d3-494c-85ca-97c3bf33329c ## Key Features - Added WAITING_FOR_REVIEW status to AgentExecutionStatus enum - Created PendingHumanReview database table for storing review requests - Implemented HumanInTheLoopBlock that extracts input data and creates review entries - Added API endpoints at /api/executions/review for fetching and reviewing pending data - Updated execution manager to properly handle waiting status and resume after approval ## Frontend Components - PendingReviewCard for individual review handling - PendingReviewsList for multiple reviews - FloatingReviewsPanel for graph builder integration - Integrated review UI into 3 locations: legacy library, new library, and graph builder ## Technical Implementation - Added proper type safety throughout with SafeJson handling - Optimized database queries using count functions instead of full data fetching - Fixed imports to be top-level instead of local - All formatters and linters pass ## Test plan - [ ] Test Human In The Loop block creation in graph builder - [ ] Test block execution pauses and creates pending review - [ ] Test review UI appears in all 3 locations - [ ] Test data modification and approval workflow - [ ] Test rejection workflow - [ ] Test execution resumes after approval 🤖 Generated with [Claude Code](https://claude.ai/code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added Human-In-The-Loop review workflows to pause executions for human validation. * Users can approve or reject pending tasks, optionally editing submitted data and adding a message. * New "Waiting for Review" execution status with UI indicators across run lists, badges, and activity views. * Review management UI: pending review cards, list view, and a floating reviews panel for quick access. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Zamil Majdy
|
4922f88851 |
feat(backend/executor): Implement cascading stop for nested graph executions (#11277)
## Summary Fixes critical issue where child executions spawned by `AgentExecutorBlock` continue running after parent execution is stopped. Implements parent-child execution tracking and recursive cascading stop logic to ensure entire execution trees are terminated together. ## Background When a parent graph execution containing `AgentExecutorBlock` nodes is stopped, only the parent was terminated. Child executions continued running, leading to: - ❌ Orphaned child executions consuming credits - ❌ No user control over execution trees - ❌ Race conditions where children start after parent stops - ❌ Resource leaks from abandoned executions ## Core Changes ### 1. Database Schema (`schema.prisma` + migration) ```sql -- Add nullable parent tracking field ALTER TABLE "AgentGraphExecution" ADD COLUMN "parentGraphExecutionId" TEXT; -- Add self-referential foreign key with graceful deletion ALTER TABLE "AgentGraphExecution" ADD CONSTRAINT "AgentGraphExecution_parentGraphExecutionId_fkey" FOREIGN KEY ("parentGraphExecutionId") REFERENCES "AgentGraphExecution"("id") ON DELETE SET NULL ON UPDATE CASCADE; -- Add index for efficient child queries CREATE INDEX "AgentGraphExecution_parentGraphExecutionId_idx" ON "AgentGraphExecution"("parentGraphExecutionId"); ``` ### 2. Parent ID Propagation (`backend/blocks/agent.py`) ```python # Extract current graph execution ID and pass as parent to child execution = add_graph_execution( # ... other params parent_graph_exec_id=graph_exec_id, # NEW: Track parent relationship ) ``` ### 3. Data Layer (`backend/data/execution.py`) ```python async def get_child_graph_executions(parent_exec_id: str) -> list[GraphExecution]: """Get all child executions of a parent execution.""" children = await AgentGraphExecution.prisma().find_many( where={"parentGraphExecutionId": parent_exec_id, "isDeleted": False} ) return [GraphExecution.from_db(child) for child in children] ``` ### 4. Cascading Stop Logic (`backend/executor/utils.py`) ```python async def stop_graph_execution( user_id: str, graph_exec_id: str, wait_timeout: float = 15.0, cascade: bool = True, # NEW parameter ): # 1. Find all child executions if cascade: children = await _get_child_executions(graph_exec_id) # 2. Stop all children recursively in parallel if children: await asyncio.gather( *[stop_graph_execution(user_id, child.id, wait_timeout, True) for child in children], return_exceptions=True, # Don't fail parent if child fails ) # 3. Stop the parent execution # ... existing stop logic ``` ### 5. Race Condition Prevention (`backend/executor/manager.py`) ```python # Before executing queued child, check if parent was terminated if parent_graph_exec_id: parent_exec = get_db_client().get_graph_execution_meta(parent_graph_exec_id, user_id) if parent_exec and parent_exec.status == ExecutionStatus.TERMINATED: # Skip execution, mark child as terminated get_db_client().update_graph_execution_stats( graph_exec_id=graph_exec_id, status=ExecutionStatus.TERMINATED, ) return # Don't start orphaned child ``` ## How It Works ### Before (Broken) ``` User stops parent execution ↓ Parent terminates ✓ ↓ Child executions keep running ✗ ↓ User cannot stop children ✗ ``` ### After (Fixed) ``` User stops parent execution ↓ Query database for all children ↓ Recursively stop all children in parallel ↓ Wait for children to terminate ↓ Stop parent execution ↓ All executions in tree stopped ✓ ``` ### Race Prevention ``` Child in QUEUED status ↓ Parent stopped ↓ Child picked up by executor ↓ Pre-flight check: parent TERMINATED? ↓ Yes → Skip execution, mark child TERMINATED ↓ Child never runs ✓ ``` ## Edge Cases Handled ✅ **Deep nesting** - Recursive cascading handles multi-level trees ✅ **Queued children** - Pre-flight check prevents execution ✅ **Race conditions** - Child spawned during stop operation ✅ **Partial failures** - `return_exceptions=True` continues on error ✅ **Multiple children** - Parallel stop via `asyncio.gather()` ✅ **No parent** - Backward compatible (nullable field) ✅ **Already completed** - Existing status check handles it ## Performance Impact - **Stop operation**: O(depth) with parallel execution vs O(1) before - **Memory**: +36 bytes per execution (one UUID reference) - **Database**: +1 query per tree level, indexed for efficiency ## API Changes (Backward Compatible) ### `stop_graph_execution()` - New Optional Parameter ```python # Before async def stop_graph_execution(user_id: str, graph_exec_id: str, wait_timeout: float = 15.0) # After async def stop_graph_execution(user_id: str, graph_exec_id: str, wait_timeout: float = 15.0, cascade: bool = True) ``` **Default `cascade=True`** means existing callers get the new behavior automatically. ### `add_graph_execution()` - New Optional Parameter ```python async def add_graph_execution(..., parent_graph_exec_id: Optional[str] = None) ``` ## Security & Safety - ✅ **User verification** - Users can only stop their own executions (parent + children) - ✅ **No cycles** - Self-referential FK prevents infinite loops - ✅ **Graceful degradation** - Errors in child stops don't block parent stop - ✅ **Rate limits** - Existing execution rate limits still apply ## Testing Checklist ### Database Migration - [x] Migration runs successfully - [x] Prisma client regenerates without errors - [x] Existing tests pass ### Core Functionality - [ ] Manual test: Stop parent with running child → child stops - [ ] Manual test: Stop parent with queued child → child never starts - [ ] Unit test: Cascading stop with multiple children - [ ] Unit test: Deep nesting (3+ levels) - [ ] Integration test: Race condition prevention ## Breaking Changes **None** - All changes are backward compatible with existing code. ## Rollback Plan If issues arise: 1. **Code rollback**: Revert PR, redeploy 2. **Database rollback**: Drop column and constraints (non-destructive) --- **Note**: This branch contains additional unrelated changes from merging with `dev`. The core cascading stop feature involves only: - `schema.prisma` + migration - `backend/data/execution.py` - `backend/executor/utils.py` - `backend/blocks/agent.py` - `backend/executor/manager.py` All other file changes are from dev branch updates and not part of this feature. 🤖 Generated with [Claude Code](https://claude.ai/code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Nested graph executions: parent-child tracking and retrieval of child executions * **Improvements** * Cascading stop: stopping a parent optionally terminates child executions * Parent execution IDs propagated through runs and surfaced in logs * Per-user/graph concurrent execution limits enforced * **Bug Fixes** * Skip enqueuing children if parent is terminated; robust handling when parent-status checks fail * **Tests** * Updated tests to cover parent linkage in graph creation <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Swifty
|
d25997b4f2 |
Revert "Merge branch 'swiftyos/secrt-1709-store-provider-names-and-en… (#11225)
Changes to providers blocks to store in db ### Changes 🏗️ - revet change ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: <!-- Put your test plan here: --> - [x] I have reverted the merge |
||
|
Swifty
|
e72ce2f9e7 | Merge branch 'swiftyos/secrt-1709-store-provider-names-and-env-vars-in-db' into dev | ||
|
Swifty
|
bd7a79a920 | Merge branch 'swiftyos/secrt-1706-improve-store-search' into dev | ||
|
Swifty
|
2dc0c97a52 | Add block registry and updated | ||
|
Swifty
|
29895c290f | store providers in db | ||
|
Zamil Majdy
|
73c0b6899a |
fix(backend): Remove advisory locks for atomic credit operations (#11143)
## Problem High QPS failures on `spend_credits` operations due to lock contention from `pg_advisory_xact_lock` causing serialization and seconds of wait time. ## Solution Replace PostgreSQL advisory locks with atomic database operations using CTEs (Common Table Expressions). ### Key Changes - **Add persistent balance column** to User table for O(1) balance lookups - **Atomic CTE-based operations** for all credit transactions using UPDATE...RETURNING pattern - **Comprehensive concurrency tests** with 7 test scenarios including stress testing - **Remove all advisory lock usage** from the credit system ### Implementation Details 1. **Migration**: Adds balance column with backfill from transaction history 2. **Atomic Operations**: All credit operations now use single atomic CTEs that update balance and create transaction in one query 3. **Race Condition Prevention**: WHERE clauses in UPDATE statements ensure balance never goes negative 4. **BetaUserCredit Compatibility**: Preserved monthly refill logic with updated `_add_transaction` signature ### Performance Impact - ✅ Eliminated lock contention bottlenecks - ✅ O(1) balance lookups instead of O(n) transaction aggregation - ✅ Atomic operations prevent race conditions without locks - ✅ Supports high QPS without serialization delays ### Testing - All existing tests pass - New concurrency test suite (`credit_concurrency_test.py`) with: - Concurrent spends from same user - Insufficient balance handling - Mixed operations (spends, top-ups, balance checks) - Race condition prevention - Integer overflow protection - Stress testing with 100 concurrent operations ### Breaking Changes None - all existing APIs maintain compatibility 🤖 Generated with [Claude Code](https://claude.ai/code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Enhanced top‑up flows with top‑up types, clearer credit→dollar formatting, and idempotent onboarding rewards. * **Bug Fixes** * Fixed race conditions for concurrent spends/top‑ups, added integer‑overflow and underflow protection, stronger input validation, and improved refund/dispute handling. * **Refactor** * Persisted per‑user balance with atomic updates for reliable balances; admin history now prefetches balances. * **Tests** * Added extensive concurrency, refund, ceiling/underflow and migration test suites. * **Chores** * Database migration to add persisted user balance; APIKey status extended (SUSPENDED). <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Swifty <craigswift13@gmail.com> |
||
|
Swifty
|
794aee25ab | add full text search | ||
|
Krzysztof Czerwinski
|
9c8652b273 |
feat(backend): Whitelist Onboarding Agents (#11149)
Some agents aren't suitable for onboarding. This adds per-store agent setting to allow them for onboarding. In case no agent is allowed fallback to the former search. ### Changes 🏗️ - Add `useForOnboarding` to `StoreListing` model and `StoreAgent` view (with migration) - Remove filtering of agents with empty input schema or credentials ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] Only allowed agents are displayed - [x] Fallback to the old system in case there aren't enough allowed agents |
||
|
Swifty
|
18e169aa51 |
feat(platform): Log Marketplace Search Terms (#11092)
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Reinier van der Leer <Pwuts@users.noreply.github.com> |
||
|
Krzysztof Czerwinski
|
48a0faa611 |
feat(frontend): Restore onboarding steps (#11027)
Wallet update removed `BUILDER_OPEN` and `BUILDER_RUN_AGENT`. ### Changes 🏗️ - Restore completion codepaths for `BUILDER_OPEN` and `BUILDER_RUN_AGENT` for analytical purposes ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] Tasks are completed silently |
||
|
Krzysztof Czerwinski
|
f33ec1f2ec |
feat(platform): New retention-focused tasks and wallet update (#10977)
### Changes 🏗️ - Rename wallet and update design - Update tasks and add Hidden Tasks section - Update onboarding backend code and related db migration - Add progress bar for some tasks ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] All tasks can be finished - [x] Finished tasks add correct amount of credits --------- Co-authored-by: Zamil Majdy <zamil.majdy@agpt.co> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Nicholas Tindle <nicholas.tindle@agpt.co> Co-authored-by: Reinier van der Leer <pwuts@agpt.co> |
||
|
Zamil Majdy
|
7a331651ba |
feat(backend): enhance database indexes for AgentGraph and AgentGraphExecution performance (#10985)
## Summary Enhances database performance by improving indexes on `AgentGraph` and `AgentGraphExecution` tables for better query efficiency. ### Changes 🏗️ - **Database Schema**: Updated Prisma schema to enhance database indexes - Modified `AgentGraph` index from `[userId, isActive]` to `[userId, isActive, id, version]` for better compound query performance - Enhanced `AgentGraphExecution` index from `[userId]` to `[userId, isDeleted, createdAt]` to support filtered queries with sorting - **Migration**: Auto-generated Prisma migration to implement the index changes - Drops existing indexes: `AgentGraph_userId_isActive_idx` and `AgentGraphExecution_userId_idx` - Creates new compound indexes: `AgentGraph_userId_isActive_id_version_idx` and `AgentGraphExecution_userId_isDeleted_createdAt_idx` ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] Verified migration runs successfully - [x] Confirmed database queries continue to work with new indexes - [x] Tested that existing functionality remains unaffected #### For configuration changes: - [x] `.env.default` is updated or already compatible with my changes - [x] `docker-compose.yml` is updated or already compatible with my changes - [x] I have included a list of my configuration changes in the PR description (under **Changes**) 🤖 Generated with [Claude Code](https://claude.ai/code) Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Nicholas Tindle
|
dd84fb5c66 |
feat(platform): Add public share links for agent run results (#10938)
<!-- Clearly explain the need for these changes: --> This PR adds the ability for users to share their agent run results publicly via shareable links. Users can generate a public link that allows anyone to view the outputs of a specific agent execution without requiring authentication. This feature enables users to share their agent results with clients, colleagues, or the community. https://github.com/user-attachments/assets/5508f430-07d0-4cd3-87bc-301b0b005cce ### Changes 🏗️ #### Backend Changes - **Database Schema**: Added share tracking fields to `AgentGraphExecution` model in Prisma schema: - `isShared`: Boolean flag to track if execution is shared - `shareToken`: Unique token for the share URL - `sharedAt`: Timestamp when sharing was enabled - **API Endpoints**: Added three new REST endpoints in `/backend/backend/server/routers/v1.py`: - `POST /graphs/{graph_id}/executions/{graph_exec_id}/share`: Enable sharing for an execution - `DELETE /graphs/{graph_id}/executions/{graph_exec_id}/share`: Disable sharing - `GET /share/{share_token}`: Retrieve shared execution data (public endpoint) - **Data Models**: - Created `SharedExecutionResponse` model for public-safe execution data - Added `ShareRequest` and `ShareResponse` Pydantic models for type-safe API responses - Updated `GraphExecutionMeta` to include share status fields - **Security**: - All share management endpoints verify user ownership before allowing changes - Public endpoint only exposes OUTPUT block data, no intermediate execution details - Share tokens are UUIDs for security #### Frontend Changes - **ShareButton Component** (`/frontend/src/components/ShareButton.tsx`): - Modal dialog for managing share settings - Copy-to-clipboard functionality for share links - Clear warnings about public accessibility - Uses Orval-generated API hooks for enable/disable operations - **Share Page** (`/frontend/src/app/(no-navbar)/share/[token]/page.tsx`): - Clean, navigation-free page for viewing shared executions - Reuses existing `RunOutputs` component for consistent output rendering - Proper error handling for invalid/disabled share links - Loading states during data fetch - **API Integration**: - Fixed custom mutator to properly set Content-Type headers for POST requests with empty bodies - Generated TypeScript types via Orval for type-safe API calls ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: <!-- Test plan: --> - [x] Enable sharing for an agent execution and verify share link is generated - [x] Copy share link and verify it copies to clipboard - [x] Open share link in incognito/private browser and verify outputs are displayed - [x] Disable sharing and verify share link returns 404 - [x] Try to enable/disable sharing for another user's execution (should fail with 404) - [x] Verify share page shows proper loading and error states - [x] Test that only OUTPUT blocks are shown in shared view, no intermediate data = |
||
|
Zamil Majdy
|
33679f3ffe |
feat(platform): Add instructions field to agent submissions (#10931)
## Summary Added an optional "Instructions" field for agent submissions to help users understand how to run agents and what to expect. <img width="1000" alt="image" src="https://github.com/user-attachments/assets/015c4f0b-4bdd-48df-af30-9e52ad283e8b" /> <img width="1000" alt="image" src="https://github.com/user-attachments/assets/3242cee8-a4ad-4536-bc12-64b491a8ef68" /> <img width="1000" alt="image" src="https://github.com/user-attachments/assets/a9b63e1c-94c0-41a4-a44f-b9f98e446793" /> ### Changes Made **Backend:** - Added `instructions` field to `AgentGraph` and `StoreListingVersion` database models - Updated `StoreSubmission`, `LibraryAgent`, and related Pydantic models - Modified store submission API routes to handle instructions parameter - Updated all database functions to properly save/retrieve instructions field - Added graceful handling for cases where database doesn't yet have the field **Frontend:** - Added instructions field to agent submission flow (PublishAgentModal) - Positioned below "Recommended Schedule" section as specified - Added instructions display in library/run flow (RunAgentModal) - Positioned above credentials section with informative blue styling - Added proper form validation with 2000 character limit - Updated all TypeScript types and API client interfaces ### Key Features - ✅ Optional field - fully backward compatible - ✅ Proper positioning in both submission and run flows - ✅ Character limit validation (2000 chars) - ✅ User-friendly display with "How to use this agent" styling - ✅ Only shows when instructions are provided ### Testing - Verified Pydantic model validation works correctly - Confirmed schema validation enforces character limits - Tested graceful handling of missing database fields - Code formatting and linting completed ## Test plan - [ ] Test agent submission with instructions field - [ ] Test agent submission without instructions (backward compatibility) - [ ] Verify instructions display correctly in run modal - [ ] Test character limit validation - [ ] Verify database migrations work properly 🤖 Generated with [Claude Code](https://claude.ai/code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Reinier van der Leer
|
6a86e70fd6 |
fix(backend/db): Keep CreditTransaction entries on User delete (#10917)
This is a non-critical improvement for bookkeeping purposes. ### Changes 🏗️ - Change `CreditTransaction` <- `User` relation to `ON DELETE NO ACTION` so that `CreditTransactions` are not automatically deleted when we delete a user's data. ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] Migration applies without problems |
||
|
Reinier van der Leer
|
2ffd249aac |
fix(backend/external-api): Improve security & reliability of API key storage (#10796)
Our API key generation, storage, and verification system has a couple of issues that need to be ironed out before full-scale deployment. ### Changes 🏗️ - Move from unsalted SHA256 to salted Scrypt hashing for API keys - Avoid false-negative API key validation due to prefix collision - Refactor API key management code for clarity - [refactor(backend): Clean up API key DB & API code (#10797)](https://github.com/Significant-Gravitas/AutoGPT/pull/10797) - Rename models and properties in `backend.data.api_key` for clarity - Eliminate redundant/custom/boilerplate error handling/wrapping in API key endpoint call stack - Remove redundant/inaccurate `response_model` declarations from API key endpoints Dependencies for `autogpt_libs`: - Add `cryptography` as a dependency - Add `pyright` as a dev dependency ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - Performing these actions through the UI (still) works: - [x] Creating an API key - [x] Listing owned API keys - [x] Deleting an owned API key - [x] Newly created API key can be used in Swagger UI - [x] Existing API key can be used in Swagger UI - [x] Existing API key is re-encrypted with salt on use |
||
|
Zamil Majdy
|
46e0f6cc45 |
feat(platform): Add recommended run schedule for agent execution (#10827)
## Summary <img width="1000" alt="Screenshot 2025-09-02 at 9 46 49 PM" src="https://github.com/user-attachments/assets/d78100c7-7974-4d37-a788-757764d8b6b7" /> <img width="1000" alt="Screenshot 2025-09-02 at 9 20 24 PM" src="https://github.com/user-attachments/assets/cd092963-8e26-4198-b65a-4416b2307a50" /> <img width="1000" alt="Screenshot 2025-09-02 at 9 22 30 PM" src="https://github.com/user-attachments/assets/e16b3bdb-c48c-4dec-9281-b2a35b3e21d0" /> <img width="1000" alt="Screenshot 2025-09-02 at 9 20 38 PM" src="https://github.com/user-attachments/assets/11d74a39-f4b4-4fce-8d30-0e6a925f3a9b" /> • Added recommended schedule cron expression as an optional input throughout the platform • Implemented complete data flow from builder → store submission → agent library → run page • Fixed UI layout issues including button text overflow and ensured proper component reusability ## Changes ### Backend - Added `recommended_schedule_cron` field to `AgentGraph` schema and database migration - Updated API models (`LibraryAgent`, `MyAgent`, `StoreSubmissionRequest`) to include the new field - Enhanced store submission approval flow to persist recommended schedule to database ### Frontend - Added recommended schedule input to builder page (SaveControl component) with overflow-safe styling - Updated store submission modal (PublishAgentModal) with schedule configuration - Enhanced agent run page with schedule tip display and pre-filled schedule dialog - Refactored `CronSchedulerDialog` with discriminated union types for better reusability - Fixed layout issues including button text truncation and popover width constraints - Implemented robust cron expression parsing with 100% reversibility between UI and cron format 🤖 Generated with [Claude Code](https://claude.ai/code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Zamil Majdy
|
2e38f132e7 |
feat(backend/store): implement sub-agent approval support (#10842)
## Summary Implement comprehensive sub-agent approval flow following the business requirements from the flow diagram. <img width="1956" height="1448" alt="image" src="https://github.com/user-attachments/assets/8de35e5b-9d3e-4dc2-bff0-47b49dbebc83" /> ### Key Features - ✅ **Auto-approve sub-agents** when main agent is approved - ✅ **Handle all scenarios**: new listings, existing versions, missing versions - ✅ **Transaction safety** with atomic operations via database transactions - ✅ **Parallel processing** using asyncio.gather for performance optimization - ✅ **Hidden from store** with isAvailable=false for all sub-agents ### Implementation Details - **Replaced** `_get_missing_sub_store_listing` with comprehensive `_handle_sub_agent_approvals` - **Added** `_approve_sub_agent` function with early returns for clean, readable code flow - **Used** `transaction()` context manager to ensure data consistency across operations - **Process sub-agents in parallel** while maintaining transaction integrity ### Business Logic Flow 1. **Check if sub-agent is already listed** in store 2. **If not listed**: create new store listing with `isAvailable=false` 3. **If listed but not approved**: approve the correct version 4. **If correct version not listed**: create store listing version and approve it 5. **If already approved**: no action needed (early return) All sub-agents remain **hidden from public store** while being internally approved for system use. ## Files Changed - `backend/server/v2/store/db.py` - Core implementation of sub-agent approval logic ## Test Plan - [ ] Verify main agent approval triggers sub-agent approvals - [ ] Test all sub-agent scenarios: new, existing unapproved, existing approved - [ ] Confirm sub-agents remain hidden (`isAvailable=false`) - [ ] Validate transaction rollback on failures - [ ] Check parallel processing works correctly 🤖 Generated with [Claude Code](https://claude.ai/code) |
||
|
Reinier van der Leer
|
e16e69ca55 |
feat(library, executor): Make "Run Again" work with credentials (#10821)
- Resolves [OPEN-2549: Make "Run again" work with credentials in `AgentRunDetailsView`](https://linear.app/autogpt/issue/OPEN-2549/make-run-again-work-with-credentials-in-agentrundetailsview) - Resolves #10237 ### Changes 🏗️ - feat(frontend/library): Make "Run Again" button work for runs with credentials - feat(backend/executor): Store passed-in credentials on `GraphExecution` ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - Go to `/library/agents/[id]` for an agent with credentials inputs - Run the agent manually - [x] -> runs successfully - [x] -> "Run again" shows among the action buttons on the newly created run - Click "Run again" - [x] -> runs successfully |
||
|
Nicholas Tindle
|
2bb8e91040 |
feat(backend): Add user timezone support to backend (#10707)
Co-authored-by: Swifty <craigswift13@gmail.com> resolve issue #10692 where scheduled time and actual run |
||
|
Nicholas Tindle
|
76090f0ba2 |
feat(backend,frontend): Send applicant email on review response (#10718)
### Changes 🏗️ This PR implements email notifications for agent creators when their agent submissions are approved or rejected by an admin in the marketplace. Specifically, the changes include: - Added `AGENT_APPROVED` and `AGENT_REJECTED` notification types to `schema.prisma`. - Created `AgentApprovalData` and `AgentRejectionData` Pydantic models for notification data. - Configured the notification system to use immediate queues and new Jinja2 templates for these types. - Designed two new email templates: `agent_approved.html.jinja2` and `agent_rejected.html.jinja2`, with dynamic content for agent details, reviewer feedback, and relevant action links. - Modified the `review_store_submission` function to: - Include `User` and `Reviewer` data in the database query. - Construct and queue the appropriate email notification based on the approval/rejection status. - Ensure email sending failures do not block the agent review process. ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] Approve an agent via the admin dashboard. - [x] Verify the agent creator receives an "Agent Approved" email with correct details and a link to the store. - [x] Reject an agent via the admin dashboard (providing a reason). - [x] Verify the agent creator receives an "Agent Rejected" email with correct details, the rejection reason, and a link to resubmit. - [x] Verify that if email sending fails (e.g., misconfigured SMTP), the agent approval/rejection process still completes successfully without error. <img width="664" height="975" alt="image" src="https://github.com/user-attachments/assets/d397f2dc-56eb-45ab-877e-b17f1fc234d1" /> <img width="664" height="975" alt="image" src="https://github.com/user-attachments/assets/25597752-f68c-46fe-8888-6c32f5dada01" /> --- Linear Issue: [SECRT-1168](https://linear.app/autogpt/issue/SECRT-1168) <a href="https://cursor.com/background-agent?bcId=bc-7394906c-0341-4bd0-8842-6d9d6f83c56c"> <picture> <source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/open-in-cursor-dark.svg"> <source media="(prefers-color-scheme: light)" srcset="https://cursor.com/open-in-cursor-light.svg"> <img alt="Open in Cursor" src="https://cursor.com/open-in-cursor.svg"> </picture> </a> <a href="https://cursor.com/agents?id=bc-7394906c-0341-4bd0-8842-6d9d6f83c56c"> <picture> <source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/open-in-web-dark.svg"> <source media="(prefers-color-scheme: light)" srcset="https://cursor.com/open-in-web-light.svg"> <img alt="Open in Web" src="https://cursor.com/open-in-web.svg"> </picture> </a> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> |
||
|
Abhimanyu Yadav
|
2610c4579f |
feat(platform/dashboard): Enable editing for agent submissions (#10545)
- resolves - https://github.com/Significant-Gravitas/AutoGPT/issues/10511 In this PR, I’ve added backend endpoints and a frontend UI for edit functionality on the Agent Dashboard. Now, users can update their store submission, if status is `PENDING` or `APPROVED`, but not for `REJECTED` and `DRAFT`. When users make changes to a pending status submission, the changes are made to the same version. However, when users make changes to an approved status submission, a new store listing version is created. Backend works something like this: <img width="866" height="832" alt="Screenshot 2025-08-15 at 9 39 02 AM" src="https://github.com/user-attachments/assets/209c60ac-8350-43c1-ba4c-7378d95ecba7" /> ### Changes - I’ve updated the `StoreSubmission` view to include `video_url` and `categories`. - I’ve added a new frontend UI for editing submissions. - I’ve created an endpoint for editing submissions. - I’ve added more end-to-end tests to ensure the edit submission functionality works as expected. ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] I have checked manually, everything is working perfectly. - [x] All e2e tests are also passing. --------- Co-authored-by: Zamil Majdy <zamil.majdy@agpt.co> Co-authored-by: neo <neo.dowithless@gmail.com> Co-authored-by: Reinier van der Leer <pwuts@agpt.co> Co-authored-by: Swifty <craigswift13@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Ubbe <hi@ubbe.dev> Co-authored-by: Lluis Agusti <hi@llu.lu> |
||
|
Zamil Majdy
|
f81d7e6a56 |
feat(backend): Add Missing FK indexes and remove unused & redundant indexes (#10412)
### Changes 🏗️ This PR optimizes database performance by adding missing foreign key indexes, removing unused/redundant indexes, cleaning up all legacy untracked indexes, and adding performance indexes for materialized views to achieve 100% optimized database indexing. **Foreign Key Indexes Added:** - `AgentGraph`: `[forkedFromId, forkedFromVersion]` - For fork relationship queries - `AgentGraphExecution`: `[agentPresetId]` - For preset-based execution filtering - `AgentNodeExecution`: `[agentNodeId]` - For node execution lookups - `AgentNodeExecutionInputOutput`: `[agentPresetId]` - For preset input/output queries - `AgentPreset`: `[agentGraphId, agentGraphVersion]` & `[webhookId]` - For graph and webhook lookups - `LibraryAgent`: `[agentGraphId, agentGraphVersion]` & `[creatorId]` - For agent and creator queries - `StoreListing`: `[agentGraphId, agentGraphVersion]` - For marketplace agent lookups - `StoreListingReview`: `[reviewByUserId]` - For user review queries **Unused/Redundant Indexes Removed:** - `User.email` - Unused index identified by linter - `AnalyticsMetrics.userId` - Unused index causing write overhead - `AnalyticsDetails.type` - Redundant (covered by composite `[userId, type]`) - `APIKey.key`, `APIKey.status` - Unused indexes - Named index `"analyticsDetails"` - Converted to standard composite index **All Legacy Untracked Indexes Removed:** - `idx_store_listing_version_status` - Redundant with Prisma composite index - `idx_slv_agent` - Redundant with Prisma-managed `[agentGraphId, agentGraphVersion]` - `idx_store_listing_version_approved_listing` - Redundant with unique constraint - `StoreListing_agentId_owningUserId_idx` - Legacy index superseded by current strategy - `StoreListing_isDeleted_isApproved_idx` - Replaced by optimized composite index - `StoreListing_isDeleted_idx` - Redundant with composite index - `StoreListingVersion_agentId_agentVersion_isDeleted_idx` - Legacy index replaced - `idx_store_listing_approved` - Redundant with existing `[owningUserId, slug]` unique constraint and `[isDeleted, hasApprovedVersion]` index - `idx_slv_categories_gin` - Specialized array search index removed (can be re-added if category filtering is implemented) - `idx_profile_user` - Duplicate of Prisma-managed `Profile_userId_idx` **Materialized View Performance Indexes Added:** - `idx_mv_review_stats_rating` on `mv_review_stats(avg_rating DESC)` - Optimizes sorting agents by rating - `idx_mv_review_stats_count` on `mv_review_stats(review_count DESC)` - Optimizes sorting agents by review count **Result: 100% Optimized Database Indexing** - All database indexes are now defined and managed through Prisma schema - No more untracked indexes requiring manual SQL maintenance - Added performance indexes for materialized views used by marketplace views - Improved query performance for agent sorting and filtering - Enhanced maintainability and consistency across environments **Schema Comments Updated:** - Removed all references to dropped untracked indexes - Simplified documentation to reflect Prisma-only approach for regular tables - Added comprehensive documentation for materialized view indexes and their purposes - Maintained documentation for materialized view refresh strategy ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] Schema changes compile successfully with Prisma - [x] Migration adds required FK indexes and materialized view performance indexes - [x] Migration drops all legacy indexes and redundant untracked indexes - [x] All pre-commit hooks pass (linting, formatting, type checking) - [x] No breaking changes to existing foreign key relationships - [x] Verified existing Prisma indexes cover all query patterns - [x] Schema comments comprehensively document all indexing strategy - [x] Materialized view performance indexes optimize marketplace sorting 🤖 Generated with [Claude Code](https://claude.ai/code) --------- Co-authored-by: Swifty <craigswift13@gmail.com> Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Swifty
|
6dba6ec3e6 |
feat(backend): Add database index for improved query performance (#10411)
This PR adds a database index to improve query performance based on Supabase query performance insights and index recommendations. These indexes target frequently queried columns and relationships to reduce query execution time. ### Changes 🏗️ - Added index on `AgentNodeExecutionInputOutput.agentPresetId` ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] Verified schema changes are valid Prisma syntax - [x] Confirmed indexes target frequently queried columns based on Supabase recommendations - [x] Ensured no duplicate indexes are created #### For configuration changes: - [x] `.env.example` is updated or already compatible with my changes - [x] `docker-compose.yml` is updated or already compatible with my changes - [x] I have included a list of my configuration changes in the PR description (under **Changes**) |
||
|
Swifty
|
7688a9701e |
perf(backend/db): Optimize StoreAgent and Creator views with database indexes and materialized views (#10084)
### Summary Performance optimization for the platform's store and creator functionality by adding targeted database indexes and implementing materialized views to reduce query execution time. ### Changes 🏗️ **Database Performance Optimizations:** - Added strategic database indexes for `StoreListing`, `StoreListingVersion`, `StoreListingReview`, `AgentGraphExecution`, and `Profile` tables - Implemented materialized views (`mv_agent_run_counts`, `mv_review_stats`) to cache expensive aggregation queries - Optimized `StoreAgent` and `Creator` views to use materialized views and improved query patterns - Added automated refresh function with 15-minute scheduling for materialized views (when pg_cron extension is available) **Key Performance Improvements:** - Filtered indexes on approved store listings to speed up marketplace queries - GIN index on categories for faster category-based searches - Composite indexes for common query patterns (e.g., listing + version lookups) - Pre-computed agent run counts and review statistics to eliminate expensive aggregations ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] Verified migration runs successfully without errors - [x] Confirmed materialized views are created and populated correctly - [x] Tested StoreAgent and Creator view queries return expected results - [x] Validated automatic refresh function works properly - [x] Confirmed rollback migration successfully removes all changes #### For configuration changes: - [x] `.env.example` is updated or already compatible with my changes - [x] `docker-compose.yml` is updated or already compatible with my changes - [x] I have included a list of my configuration changes in the PR description (under **Changes**) **Note:** No configuration changes were required as this is purely a database schema optimization. |
||
|
Zamil Majdy
|
095199bfa6 |
feat(backend): implement KV data storage blocks (#10294)
This PR introduces key-value storage blocks. ### Changes 🏗️ - **Database Schema**: Add AgentNodeExecutionKeyValueData table with composite primary key (userId, key) - **Persistence Blocks**: Create PersistInformationBlock and RetrieveInformationBlock in persistence.py - **Scope-based Storage**: Support for within_agent (per agent) vs across_agents (global user) persistence - **Key Structure**: Use formal # delimiter for storage keys: `agent#{graph_id}#{key}` and `global#{key}` ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] Run all 244 block tests - all passing ✅ - [x] Test PersistInformationBlock with mock data storage - [x] Test RetrieveInformationBlock with mock data retrieval - [x] Verify scope-based key generation (within_agent vs across_agents) - [x] Verify database function integration through all manager classes - [x] Run lint and type checking - all passing ✅ - [x] Verify database migration is included and valid #### For configuration changes: - [x] `.env.example` is updated or already compatible with my changes - [x] `docker-compose.yml` is updated or already compatible with my changes - [x] I have included a list of my configuration changes in the PR description (under **Changes**) Note: This change adds database schema and new blocks but doesn't require environment or docker-compose changes as it uses existing database infrastructure. --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Reinier van der Leer
|
efa4b6d2a0 |
feat(platform/library): Triggered-agent support (#10167)
This pull request adds support for setting up (webhook-)triggered agents in the Library. It contains changes throughout the entire stack to make everything work in the various phases of a triggered agent's lifecycle: setup, execution, updates, deletion. Setting up agents with webhook triggers was previously only possible in the Builder, limiting their use to the agent's creator only. To make it work in the Library, this change uses the previously introduced `AgentPreset` to store information on, instead of on the graph's nodes to which only a graph's creator has access. - Initial ticket: #10111 - Builds on #9786   ### Changes 🏗️ Frontend: - Amend the Library's `AgentRunDraftView` to handle creating and editing Presets - Add `hideIfSingleCredentialAvailable` parameter to `CredentialsInput` - Add multi-select support to `TypeBasedInput` - Add Presets section to `AgentRunsSelectorList` - Amend `AgentRunSummaryCard` for use for Presets - Add `AgentStatusChip` to display general agent status (for now: Active / Inactive / Error) - Add Preset loading logic and create/update/delete handlers logic to `AgentRunsPage` - Rename `IconClose` to `IconCross` API: - Add `LibraryAgent` properties `has_external_trigger`, `trigger_setup_info`, `credentials_input_schema` - Add `POST /library/agents/{library_agent_id}/setup_trigger` endpoint - Remove redundant parameters from `POST /library/presets/{preset_id}/execute` endpoint Backend: - Add `POST /library/agents/{library_agent_id}/setup_trigger` endpoint - Extract non-node-related logic from `on_node_activate` into `setup_webhook_for_block` - Add webhook-related logic to `update_preset` and `delete_preset` endpoints - Amend webhook infrastructure to work with AgentPresets - Add preset trigger support to webhook ingress endpoint - Amend executor stack to work with passed-in node input (`nodes_input_masks`, generalized from `node_credentials_input_map`) - Amend graph validation to work with passed-in node input - Add `AgentPreset`->`IntegrationWebhook` relation - Add `WebhookWithRelations` model - Change behavior of `BaseWebhooksManager.get_manual_webhook(..)` to avoid unnecessary changes of the webhook URL: ignore `events` to find matching webhook, and update `events` if necessary. - Fix & improve `AgentPreset` API, models, and DB logic - Add `isDeleted` filter to get/list queries - Add `user_id` attribute to `LibraryAgentPreset` model - Add separate `credentials` property to `LibraryAgentPreset` model - Fix `library_db.update_preset(..)` replacement of existing `InputPresets` - Make `library_db.update_preset(..)` more usage-friendly with separate parameters for updateable properties - Add `user_id` checks to various DB functions - Fix error handling in various endpoints - Fix cache race condition on `load_webhook_managers()` ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - Test existing functionality - [x] Auto-setup and -teardown of webhooks on save in the builder still works - [x] Running an agent normally from the Library still works - Test new functionality - [x] Setting up a trigger in the Library - [x] Updating a trigger in the Library - [x] Disabling and re-enabling a trigger in the Library - [x] Deleting a trigger in the Library - [x] Triggers set up in the Library result in a new run when the webhook receives a payload |
||
|
Zamil Majdy
|
3df6dcd26b |
feat(blocks): Improve SmartDecisionBlock & AIStructuredResponseGeneratorBlock (#10198)
Main issues: * `AIStructuredResponseGeneratorBlock` is not able to produce a list of objects. * `SmartDecisionBlock` is not able to call tools with some optional inputs. ### Changes 🏗️ * Allow persisting `null` / `None` value as execution output. * Provide `multiple_tool_calls` option for `SmartDecisionBlock`. * Provide `list_result` option for `AIStructuredResponseGeneratorBlock` ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: <!-- Put your test plan here: --> - [x] Run `SmartDecisionBlock` & `AIStructuredResponseGeneratorBlock` |
||
|
Krzysztof Czerwinski
|
9f2b9d08c9 |
feat(platform): Add Run 10 agents wallet task (#9937)
### Changes 🏗️ This PR adds `Run 10 agents` step to wallet tasks that can be done by running any agents 10 times either from Library or Builder (onboarding agent run also counts). - Merge `Finish onboarding` and `See results` steps into one in the wallet - User is redirected directly to onboarding agent runs in Library after congrats screen - Add `RUN_AGENTS` step and `agentRuns` integer to schema and related migration - Running agent from Library and Builder increments `agentRuns` - Open NPS survey popup when 10 agents are run - Fix resuming onboarding on login when unfinished - Remove no longer needed `get-results.mp4` tutorial video ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] Onboarding can be completed and proper reward is awarded - [x] `Run 10 agents` can be completed and reward is awarded - [x] When unning different agents and the same agent - [x] Running from library and builder counts - [x] Onboarding is resumed to last finished step on login |
||
|
Swifty
|
17e973a8cb |
fix(platform): Optimise Query Indexes (#10038)
# Query Optimization for AgentNodeExecution Tables
## Overview
This PR describes the database index optimizations applied to improve
the performance of slow queries in the AutoGPT platform backend.
## Problem Analysis
The following queries were identified as consuming significant database
resources:
### 1. Complex Filtering Query (19.3% of total time)
```sql
SELECT ... FROM "AgentNodeExecution"
WHERE "agentNodeId" = $1
AND "agentGraphExecutionId" = $2
AND "executionStatus" = $3
AND "id" NOT IN (
SELECT "referencedByInputExecId"
FROM "AgentNodeExecutionInputOutput"
WHERE "name" = $4 AND "referencedByInputExecId" IS NOT NULL
)
ORDER BY "addedTime" ASC
```
### 2. Multi-table JOIN Query (8.9% of total time)
```sql
SELECT ... FROM "AgentNodeExecution"
LEFT JOIN "AgentNode" ON ...
LEFT JOIN "AgentBlock" ON ...
WHERE "AgentBlock"."id" IN (...)
AND "executionStatus" != $11
AND "agentGraphExecutionId" IN (...)
ORDER BY "queuedTime" DESC, "addedTime" DESC
```
### 3. Bulk Graph Execution Queries (multiple variations, ~10% combined)
Multiple queries filtering by `agentGraphExecutionId` with various
ordering requirements.
## Optimization Strategy
### 1. Composite Indexes for AgentNodeExecution
Set the following composite indexes to the `AgentNodeExecution` model:
```prisma
@@index([agentGraphExecutionId, agentNodeId, executionStatus])
@@index([addedTime, queuedTime])
```
#### Benefits:
- **Index 1**: Covers the exact WHERE clause of the complex filtering
query, allowing index-only scans
- **Index 2**: Optimizes queries filtering by graph execution and status
- **Index 3**: Supports efficient sorting when filtering by graph
execution
- **Index 4**: Optimizes ORDER BY operations on time fields
### 2. Composite Index for AgentNodeExecutionInputOutput
Added the following composite index:
```prisma
// Input and Output pin names are unique for each AgentNodeExecution.
@@unique([referencedByInputExecId, referencedByOutputExecId, name])
@@index([referencedByOutputExecId])
// Composite index for `upsert_execution_input`.
@@index([name, time])
```
#### Benefits:
- Dramatically improves the NOT IN subquery performance in Query 1
- Allows the database to use an index scan instead of a full table scan
- Reduces the subquery execution time from O(n) to O(log n)
## Expected Performance Improvements
1. **Query 1 (19.3% of total time)**:
- Expected improvement: 80-90% reduction in execution time
- The composite index on `[agentNodeId, agentGraphExecutionId,
executionStatus]` will allow index-only scans
- The subquery will benefit from the new index on
`AgentNodeExecutionInputOutput`
2. **Query 2 (8.9% of total time)**:
- Expected improvement: 50-70% reduction in execution time
- The `[agentGraphExecutionId, executionStatus]` index will reduce the
initial filtering cost
3. **Bulk Queries (10% combined)**:
- Expected improvement: 60-80% reduction in execution time
- Composite indexes including time fields will optimize sorting
operations
## Migration Considerations
1. **Index Creation Time**: Creating these indexes on existing large
tables may take time
2. **Storage Impact**: Each index requires additional storage space
3. **Write Performance**: Slight decrease in INSERT/UPDATE performance
due to index maintenance
## Additional Optimizations
### NotificationEvent Table Index
Added index for notification batch queries:
```prisma
@@index([userNotificationBatchId])
```
This optimizes the query:
```sql
SELECT ... FROM "NotificationEvent"
WHERE "userNotificationBatchId" IN (...)
```
#### Benefits:
- Eliminates full table scans when filtering by batch ID
- Improves query performance from O(n) to O(log n)
- Particularly beneficial for users with many notification events
## Future Optimizations
Consider these additional optimizations if needed:
1. Partitioning `AgentNodeExecution` table by `createdAt` or
`agentGraphExecutionId`
2. Implementing materialized views for frequently accessed aggregate
data
3. Adding covering indexes for specific query patterns
4. Implementing query result caching at the application level
---------
Co-authored-by: Zamil Majdy <zamil.majdy@agpt.co>
|
||
|
Zamil Majdy
|
ac8ef9bdb2 |
feat(backend): Introduce late execution check scheduled job (#9914)
Introduce a late execution check scheduled job. The late threshold duration is configurable. This initial version only reports the error to Sentry. ### Changes 🏗️ * Added late execution check scheduled job * Move the registration weekly notification processing job out of API call and calling it directly from the scheduler service. ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: <!-- Put your test plan here: --> - [x] Manual firing of scheduled job through an exposed API |
||
|
Krzysztof Czerwinski
|
160a622ba4 |
feat(platform): Forking agent in Library (#9870)
This PR introduces copying agents feature in the Library. Users can copy and download their library agents but they can edit only the ones they own (included copied ones). ### Changes 🏗️ - DB migration: add relation in `AgentGraph`: `forked_from_id` and `forked_from_version` - Add `fork_graph` function that makes a hardcopy of agent graph and its nodes (all with new ids) - Add `fork_library_agent` that copies library agent and its graph for a user - Add endpoint `/library/agents/{libraryAgentId}/fork` - Add UI to `library/agents/[id]/page.tsx`: `Edit a copy` button with dialog confirmation ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] Agent can be copied, edited and runs |
||
|
Nicholas Tindle
|
3e0742f9c5 |
Spike/infra pooling (#9812)
<!-- Clearly explain the need for these changes: --> Swap to pooling supabase connections rather than depending on x number of max open connections ### Changes 🏗️ Adds direct connect URL to be used throughout the system <!-- Concisely describe all of the changes made in this pull request: --> ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: <!-- Put your test plan here: --> - [x] Test thoroughly all of the endpoints in the dev env with switched infra matching pr - [x] Follow the new release plan tests - [x] Follow the old release plan tests #### For configuration changes: - [x] `.env.example` is updated or already compatible with my changes - [x] `docker-compose.yml` is updated or already compatible with my changes - [x] I have included a list of my configuration changes in the PR description (under **Changes**) <details> <summary>configuration changes</summary> - Change how we connect to the database to use direct when configured and database URL when not - update prisma for this - have default matching database and default </details> |
||
|
Krzysztof Czerwinski
|
d791cdea76 |
feat(platform): Onboarding Phase 2 (#9736)
### Changes 🏗️ - Update onboarding to give user rewards for completing steps - Remove `canvas-confetti` lib and add `party-js` instead; the former didn't allow to play confetti from a component - Add onboarding videos in `frontend/public/onboarding/` - Remove Balance (`CreditsCard.tsx`) and add openable `Wallet.tsx` (and accompanying `WalletTaskGroup.tsx`) instead that displays grouped onboarding tasks with descriptions and short instructional videos - Further relevant updates to `useOnboarding`, `types.ts` - Implement onboarding rewards - Add `onboarding_reward` function in `credit.py` that is used to reward user for finished onboarding tasks safely - transaction key is deterministic, so the same user won't be rewarded twice for the same step. - Add `reward_user` in `onboarding.py` - Update `UserOnboarding` model and add a migration <img width="464" alt="Screenshot 2025-04-05 at 6 06 29 PM" src="https://github.com/user-attachments/assets/fca8d09e-0139-466b-b679-d24117ad01f0" /> ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] Onboarding works - [x] Tasks can be completed - [x] Rewards are added correctly for all completed tasks |
