docs: Add comprehensive Block SDK guide (#10767)

## Summary
- Added comprehensive Block SDK guide documenting the new SDK pattern
for creating blocks
- Integrated the guide into the documentation structure
- Updated existing documentation to reference the new guide

## Changes
- Created `docs/content/platform/block-sdk-guide.md` with detailed
instructions for:
  - Provider configuration using `ProviderBuilder`
  - Block schema definition and implementation
  - Authentication methods (API keys, OAuth, webhooks)
  - Testing and validation
  - File organization and best practices

- Updated documentation structure:
  - Added guide to `mkdocs.yml` navigation
  - Added cross-references in `new_blocks.md`
  - Added links in `blocks/blocks.md` overview
  - Updated `CLAUDE.md` with reference to the new guide

## Test plan
- [ ] Documentation builds correctly with mkdocs
- [ ] All internal links resolve properly
- [ ] Guide examples are syntactically correct
- [ ] Navigation structure is logical and accessible

autogpt_platform/CLAUDE.md

@@ -149,12 +149,21 @@ Key models (defined in `/backend/schema.prisma`):
 
 **Adding a new block:**
 
+Follow the comprehensive [Block SDK Guide](../../../docs/content/platform/block-sdk-guide.md) which covers:
+- Provider configuration with `ProviderBuilder`
+- Block schema definition
+- Authentication (API keys, OAuth, webhooks)
+- Testing and validation
+- File organization
+
+Quick steps:
 1. Create new file in `/backend/backend/blocks/`
-2. Inherit from `Block` base class
+2. Configure provider using `ProviderBuilder` in `_config.py`
-3. Define input/output schemas
+3. Inherit from `Block` base class
-4. Implement `run` method
+4. Define input/output schemas using `BlockSchema`
-5. Register in block registry
+5. Implement async `run` method
-6. Generate the block uuid using `uuid.uuid4()`
+6. Generate unique block ID using `uuid.uuid4()`
+7. Test with `poetry run pytest backend/blocks/test/test_block.py`
 
 Note: when making many new blocks analyze the interfaces for each of these blocks and picture if they would go well together in a graph based editor or would they struggle to connect productively?
 ex: do the inputs and outputs tie well together?

docs/content/platform/block-sdk-guide.md

@@ -0,0 +1,307 @@
+# Block Creation with SDK
+
+This guide explains how to create new blocks for the AutoGPT Platform using the SDK pattern with advanced features.
+
+## Overview
+
+Blocks are reusable components that perform specific tasks in AutoGPT workflows. They can integrate with external services, process data, or perform any programmatic operation.
+
+## Basic Structure
+
+### 1. Create Provider Configuration
+
+First, create a `_config.py` file to configure your provider using the `ProviderBuilder`:
+
+```python
+from backend.sdk import BlockCostType, ProviderBuilder
+
+# Simple API key provider
+my_provider = (
+    ProviderBuilder("my_provider")
+    .with_api_key("MY_PROVIDER_API_KEY", "My Provider API Key")
+    .with_base_cost(1, BlockCostType.RUN)
+    .build()
+)
+```
+
+For OAuth providers:
+
+```python
+from backend.sdk import BlockCostType, ProviderBuilder
+from ._oauth import MyProviderOAuthHandler
+
+my_provider = (
+    ProviderBuilder("my_provider")
+    .with_oauth(
+        MyProviderOAuthHandler,
+        scopes=["read", "write"],
+        client_id_env_var="MY_PROVIDER_CLIENT_ID",
+        client_secret_env_var="MY_PROVIDER_CLIENT_SECRET",
+    )
+    .with_base_cost(1, BlockCostType.RUN)
+    .build()
+)
+```
+
+### 2. Create the Block Class
+
+Create your block file (e.g., `my_block.py`):
+
+```python
+import uuid
+from backend.sdk import (
+    APIKeyCredentials,
+    Block,
+    BlockCategory,
+    BlockOutput,
+    BlockSchema,
+    CredentialsMetaInput,
+    SchemaField,
+)
+from ._config import my_provider
+
+
+class MyBlock(Block):
+    class Input(BlockSchema):
+        # Define input fields
+        credentials: CredentialsMetaInput = my_provider.credentials_field(
+            description="API credentials for My Provider"
+        )
+        query: str = SchemaField(description="The query to process")
+        limit: int = SchemaField(
+            description="Number of results", 
+            default=10,
+            ge=1,  # Greater than or equal to 1
+            le=100  # Less than or equal to 100
+        )
+        advanced_option: str = SchemaField(
+            description="Advanced setting",
+            default="",
+            advanced=True  # Hidden by default in UI
+        )
+
+    class Output(BlockSchema):
+        # Define output fields
+        results: list = SchemaField(description="List of results")
+        count: int = SchemaField(description="Total count")
+        error: str = SchemaField(description="Error message if failed")
+
+    def __init__(self):
+        super().__init__(
+            id=str(uuid.uuid4()),  # Generate unique ID
+            description="Brief description of what this block does",
+            categories={BlockCategory.SEARCH},  # Choose appropriate categories
+            input_schema=self.Input,
+            output_schema=self.Output,
+        )
+
+    async def run(
+        self, 
+        input_data: Input, 
+        *, 
+        credentials: APIKeyCredentials,
+        **kwargs
+    ) -> BlockOutput:
+        try:
+            # Your block logic here
+            results = await self.process_data(
+                input_data.query,
+                input_data.limit,
+                credentials
+            )
+            
+            # Yield outputs
+            yield "results", results
+            yield "count", len(results)
+            
+        except Exception as e:
+            yield "error", str(e)
+
+    async def process_data(self, query, limit, credentials):
+        # Implement your logic
+        # Use credentials.api_key.get_secret_value() to access the API key
+        pass
+```
+
+## Key Components Explained
+
+### Provider Configuration
+
+The `ProviderBuilder` allows you to:
+- **`.with_api_key()`**: Add API key authentication
+- **`.with_oauth()`**: Add OAuth authentication  
+- **`.with_base_cost()`**: Set resource costs for the block
+- **`.with_webhook_manager()`**: Add webhook support
+- **`.with_user_password()`**: Add username/password auth
+
+### Block Schema
+
+- **Input/Output classes**: Define the data structure using `BlockSchema`
+- **SchemaField**: Define individual fields with validation
+- **CredentialsMetaInput**: Special field for handling credentials
+
+### Block Implementation
+
+1. **Unique ID**: Generate using `uuid.uuid4()`
+2. **Categories**: Choose from `BlockCategory` enum (e.g., SEARCH, AI, PRODUCTIVITY)
+3. **async run()**: Main execution method that yields outputs
+4. **Error handling**: Always include error output field
+
+## Advanced Features
+
+### Testing
+
+Add test configuration to your block:
+
+```python
+def __init__(self):
+    super().__init__(
+        # ... other config ...
+        test_input={
+            "query": "test query",
+            "limit": 5,
+            "credentials": {
+                "provider": "my_provider",
+                "id": str(uuid.uuid4()),
+                "type": "api_key"
+            }
+        },
+        test_output=[
+            ("results", ["result1", "result2"]),
+            ("count", 2)
+        ],
+        test_mock={
+            "process_data": lambda *args, **kwargs: ["result1", "result2"]
+        }
+    )
+```
+
+### OAuth Support
+
+Create an OAuth handler in `_oauth.py`:
+
+```python
+from backend.integrations.oauth.base import BaseOAuthHandler
+
+class MyProviderOAuthHandler(BaseOAuthHandler):
+    PROVIDER_NAME = "my_provider"
+    
+    def _get_authorization_url(self, scopes: list[str], state: str) -> str:
+        # Implementation
+        pass
+    
+    def _exchange_code_for_token(self, code: str, scopes: list[str]) -> dict:
+        # Implementation
+        pass
+```
+
+### Webhook Support
+
+Create a webhook manager in `_webhook.py`:
+
+```python
+from backend.integrations.webhooks._base import BaseWebhooksManager
+
+class MyProviderWebhookManager(BaseWebhooksManager):
+    PROVIDER_NAME = "my_provider"
+    
+    async def validate_event(self, event: dict) -> bool:
+        # Implementation
+        pass
+```
+
+## File Organization
+
+```
+backend/blocks/my_provider/
+├── __init__.py          # Export your blocks
+├── _config.py           # Provider configuration  
+├── _oauth.py           # OAuth handler (optional)
+├── _webhook.py         # Webhook manager (optional)
+├── _api.py             # API client wrapper (optional)
+├── models.py           # Data models (optional)
+└── my_block.py         # Block implementations
+```
+
+## Best Practices
+
+1. **Error Handling**: Always include an error field in outputs
+2. **Credentials**: Use the provider's `credentials_field()` method
+3. **Validation**: Use SchemaField constraints (ge, le, min_length, etc.)
+4. **Categories**: Choose appropriate categories for discoverability
+5. **Advanced Fields**: Mark complex options as `advanced=True`
+6. **Async Operations**: Use `async`/`await` for I/O operations
+7. **API Clients**: Use `Requests()` from SDK or external libraries
+8. **Testing**: Include test inputs/outputs for validation
+
+## Common Patterns
+
+### Making API Requests
+
+```python
+from backend.sdk import Requests
+
+async def run(self, input_data: Input, *, credentials: APIKeyCredentials, **kwargs):
+    headers = {
+        "Authorization": f"Bearer {credentials.api_key.get_secret_value()}",
+        "Content-Type": "application/json"
+    }
+    
+    response = await Requests().post(
+        "https://api.example.com/endpoint",
+        headers=headers,
+        json={"query": input_data.query}
+    )
+    
+    data = response.json()
+    yield "results", data.get("results", [])
+```
+
+### Multiple Auth Types
+
+```python
+async def run(
+    self, 
+    input_data: Input, 
+    *, 
+    credentials: OAuth2Credentials | APIKeyCredentials,
+    **kwargs
+):
+    if isinstance(credentials, OAuth2Credentials):
+        # Handle OAuth
+        token = credentials.access_token.get_secret_value()
+    else:
+        # Handle API key
+        token = credentials.api_key.get_secret_value()
+```
+
+## Testing Your Block
+
+```bash
+# Run all block tests
+poetry run pytest backend/blocks/test/test_block.py -xvs
+
+# Test specific block
+poetry run pytest 'backend/blocks/test/test_block.py::test_available_blocks[MyBlock]' -xvs
+```
+
+## Integration Checklist
+
+- [ ] Create provider configuration in `_config.py`
+- [ ] Implement block class with Input/Output schemas
+- [ ] Generate unique block ID with `uuid.uuid4()`
+- [ ] Choose appropriate block categories
+- [ ] Implement `async run()` method
+- [ ] Handle errors gracefully
+- [ ] Add test configuration
+- [ ] Export block in `__init__.py`
+- [ ] Test the block
+- [ ] Document any special requirements
+
+## Example Blocks for Reference
+
+- **Simple API**: `/backend/blocks/firecrawl/` - Basic API key authentication
+- **OAuth + API**: `/backend/blocks/linear/` - OAuth and API key support
+- **Webhooks**: `/backend/blocks/exa/` - Includes webhook manager
+
+Study these examples to understand different patterns and approaches for building blocks.

docs/content/platform/blocks/blocks.md

@@ -2,6 +2,12 @@
 
 AutoGPT uses a modular approach with various "blocks" to handle different tasks. These blocks are the building blocks of AutoGPT workflows, allowing users to create complex automations by combining simple, specialized components.
 
+!!! info "Creating Your Own Blocks"
+    Want to create your own custom blocks? Check out our guides:
+    
+    - [Build your own Blocks](../new_blocks.md) - Step-by-step tutorial with examples
+    - [Block SDK Guide](../block-sdk-guide.md) - Advanced SDK patterns with OAuth, webhooks, and provider configuration
+
 Below is a comprehensive list of all available blocks, categorized by their primary function. Click on any block name to view its detailed documentation.
 
 ## Basic Operations

docs/content/platform/new_blocks.md

@@ -2,6 +2,9 @@
 
 This guide will walk you through the process of creating and testing a new block for the AutoGPT Agent Server, using the WikipediaSummaryBlock as an example.
 
+!!! tip "New SDK-Based Approach"
+    For a more comprehensive guide using the new SDK pattern with ProviderBuilder and advanced features like OAuth and webhooks, see the [Block SDK Guide](block-sdk-guide.md).
+
 ## Understanding Blocks and Testing
 
 Blocks are reusable components that can be connected to form a graph representing an agent's behavior. Each block has inputs, outputs, and a specific function. Proper testing is crucial to ensure blocks work correctly and consistently.

docs/mkdocs.yml

@@ -18,6 +18,7 @@ nav:
     - Advanced Setup: platform/advanced_setup.md
     - Agent Blocks: platform/agent-blocks.md
     - Build your own Blocks: platform/new_blocks.md
+    - Block SDK Guide: platform/block-sdk-guide.md
     - Using Ollama: platform/ollama.md
     - Using AI/ML API: platform/aimlapi.md
     - Using D-ID: platform/d_id.md