mirror of https://github.com/Significant-Gravitas/AutoGPT.git synced 2026-01-09 07:08:09 -05:00

Files

Swifty 243400e128 feat(platform): Add Block Development SDK with auto-registration system (#10074 )

## Block Development SDK - Simplifying Block Creation

### Problem
Currently, creating a new block requires manual updates to **5+ files**
scattered across the codebase:
- `backend/data/block_cost_config.py` - Manually add block costs
- `backend/integrations/credentials_store.py` - Add default credentials
- `backend/integrations/providers.py` - Register new providers
- `backend/integrations/oauth/__init__.py` - Register OAuth handlers
- `backend/integrations/webhooks/__init__.py` - Register webhook
managers

This creates significant friction for developers, increases the chance
of configuration errors, and makes the platform difficult to scale.

### Solution
This PR introduces a **Block Development SDK** that provides:
- Single import for all block development needs: `from backend.sdk
import *`
- Automatic registration of all block configurations
- Zero external file modifications required
- Provider-based configuration with inheritance

### Changes 🏗️

#### 1. **New SDK Module** (`backend/sdk/`)
- **`__init__.py`**: Unified exports of 68+ block development components
- **`registry.py`**: Central auto-registration system for all block
configurations
- **`builder.py`**: `ProviderBuilder` class for fluent provider
configuration
- **`provider.py`**: Provider configuration management
- **`cost_integration.py`**: Automatic cost application system

#### 2. **Provider Builder Pattern**
```python
# Configure once, use everywhere
my_provider = (
    ProviderBuilder("my-service")
    .with_api_key("MY_SERVICE_API_KEY", "My Service API Key")
    .with_base_cost(5, BlockCostType.RUN)
    .build()
)
```

#### 3. **Automatic Cost System**
- Provider base costs automatically applied to all blocks using that
provider
- Override with `@cost` decorator for block-specific pricing
- Tiered pricing support with cost filters

#### 4. **Dynamic Provider Support**
- Modified `ProviderName` enum to accept any string via `_missing_`
method
- No more manual enum updates for new providers

#### 5. **Application Integration**
- Added `sync_all_provider_costs()` to `initialize_blocks()` for
automatic cost registration
- Maintains full backward compatibility with existing blocks

#### 6. **Comprehensive Examples** (`backend/blocks/examples/`)
- `simple_example_block.py` - Basic block structure
- `example_sdk_block.py` - Provider with credentials
- `cost_example_block.py` - Various cost patterns
- `advanced_provider_example.py` - Custom API clients
- `example_webhook_sdk_block.py` - Webhook configuration

#### 7. **Extensive Testing**
- 6 new test modules with 30+ test cases
- Integration tests for all SDK features
- Cost calculation verification
- Provider registration tests

### Before vs After

**Before SDK:**
```python
# 1. Multiple complex imports
from backend.data.block import Block, BlockCategory, BlockOutput
from backend.data.model import SchemaField, CredentialsField
# ... many more imports

# 2. Update block_cost_config.py
BLOCK_COSTS[MyBlock] = [BlockCost(...)]

# 3. Update credentials_store.py
DEFAULT_CREDENTIALS.append(...)

# 4. Update providers.py enum
# 5. Update oauth/__init__.py
# 6. Update webhooks/__init__.py
```

**After SDK:**
```python
from backend.sdk import *

# Everything configured in one place
my_provider = (
    ProviderBuilder("my-service")
    .with_api_key("MY_API_KEY", "My API Key")
    .with_base_cost(10, BlockCostType.RUN)
    .build()
)

class MyBlock(Block):
    class Input(BlockSchema):
        credentials: CredentialsMetaInput = my_provider.credentials_field()
        data: String = SchemaField(description="Input data")
    
    class Output(BlockSchema):
        result: String = SchemaField(description="Result")
    
    # That's it\! No external files to modify
```

### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
  - [x] Created new blocks using SDK pattern with provider configuration
  - [x] Verified automatic cost registration for provider-based blocks
  - [x] Tested cost override with @cost decorator
  - [x] Confirmed custom providers work without enum modifications
  - [x] Verified all example blocks execute correctly
  - [x] Tested backward compatibility with existing blocks
  - [x] Ran all SDK tests (30+ tests, all passing)
  - [x] Created blocks with credentials and verified authentication
  - [x] Tested webhook block configuration
  - [x] Verified application startup with auto-registration

#### For configuration changes:
- [x] `.env.example` is updated or already compatible with my changes
(no changes needed)
- [x] `docker-compose.yml` is updated or already compatible with my
changes (no changes needed)
- [x] I have included a list of my configuration changes in the PR
description (under **Changes**)

### Impact

- **Developer Experience**: Block creation time reduced from hours to
minutes
- **Maintainability**: All block configuration in one place
- **Scalability**: Support hundreds of blocks without enum updates
- **Type Safety**: Full IDE support with proper type hints
- **Testing**: Easier to test blocks in isolation

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Abhimanyu Yadav <122007096+Abhi1992002@users.noreply.github.com>

2025-07-10 16:17:55 +02:00

4.8 KiB

Raw Blame History

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Repository Overview

AutoGPT Platform is a monorepo containing:

Backend (/backend): Python FastAPI server with async support
Frontend (/frontend): Next.js React application
Shared Libraries (/autogpt_libs): Common Python utilities

Essential Commands

Backend Development

# Install dependencies
cd backend && poetry install

# Run database migrations
poetry run prisma migrate dev

# Start all services (database, redis, rabbitmq, clamav)
docker compose up -d

# Run the backend server
poetry run serve

# Run tests
poetry run test

# Run specific test
poetry run pytest path/to/test_file.py::test_function_name

# Lint and format
# prefer format if you want to just "fix" it and only get the errors that can't be autofixed
poetry run format  # Black + isort
poetry run lint    # ruff

More details can be found in TESTING.md

Creating/Updating Snapshots

When you first write a test or when the expected output changes:

poetry run pytest path/to/test.py --snapshot-update

⚠️ Important: Always review snapshot changes before committing! Use git diff to verify the changes are expected.

Frontend Development

# Install dependencies
cd frontend && npm install

# Start development server
npm run dev

# Run E2E tests
npm run test

# Run Storybook for component development
npm run storybook

# Build production
npm run build

# Type checking
npm run type-check

Architecture Overview

Backend Architecture

API Layer: FastAPI with REST and WebSocket endpoints
Database: PostgreSQL with Prisma ORM, includes pgvector for embeddings
Queue System: RabbitMQ for async task processing
Execution Engine: Separate executor service processes agent workflows
Authentication: JWT-based with Supabase integration
Security: Cache protection middleware prevents sensitive data caching in browsers/proxies

Frontend Architecture

Framework: Next.js App Router with React Server Components
State Management: React hooks + Supabase client for real-time updates
Workflow Builder: Visual graph editor using @xyflow/react
UI Components: Radix UI primitives with Tailwind CSS styling
Feature Flags: LaunchDarkly integration

Key Concepts

Agent Graphs: Workflow definitions stored as JSON, executed by the backend
Blocks: Reusable components in /backend/blocks/ that perform specific tasks
Integrations: OAuth and API connections stored per user
Store: Marketplace for sharing agent templates
Virus Scanning: ClamAV integration for file upload security

Testing Approach

Backend uses pytest with snapshot testing for API responses
Test files are colocated with source files (*_test.py)
Frontend uses Playwright for E2E tests
Component testing via Storybook

Database Schema

Key models (defined in /backend/schema.prisma):

User: Authentication and profile data
AgentGraph: Workflow definitions with version control
AgentGraphExecution: Execution history and results
AgentNode: Individual nodes in a workflow
StoreListing: Marketplace listings for sharing agents

Environment Configuration

Backend: .env file in /backend
Frontend: .env.local file in /frontend
Both require Supabase credentials and API keys for various services

Common Development Tasks

Adding a new block:

Create new file in /backend/backend/blocks/
Inherit from Block base class
Define input/output schemas
Implement run method
Register in block registry
Generate the block uuid using uuid.uuid4()

Modifying the API:

Update route in /backend/backend/server/routers/
Add/update Pydantic models in same directory
Write tests alongside the route file
Run poetry run test to verify

Frontend feature development:

Components go in /frontend/src/components/
Use existing UI components from /frontend/src/components/ui/
Add Storybook stories for new components
Test with Playwright if user-facing

Security Implementation

Cache Protection Middleware:

Located in /backend/backend/server/middleware/security.py
Default behavior: Disables caching for ALL endpoints with Cache-Control: no-store, no-cache, must-revalidate, private
Uses an allow list approach - only explicitly permitted paths can be cached
Cacheable paths include: static assets (/static/*, /_next/static/*), health checks, public store pages, documentation
Prevents sensitive data (auth tokens, API keys, user data) from being cached by browsers/proxies
To allow caching for a new endpoint, add it to CACHEABLE_PATHS in the middleware
Applied to both main API server and external API applications

4.8 KiB Raw Blame History