AutoGPT

github/AutoGPT

Fork 0

mirror of https://github.com/Significant-Gravitas/AutoGPT.git synced 2026-04-30 03:00:41 -04:00

Commit Graph

Author	SHA1	Message	Date
Swifty	c955e9a4d7	feat(blocks): Add Airtable Integration (#10338 ) ## Overview This PR adds comprehensive Airtable integration to the AutoGPT platform, enabling users to seamlessly connect their Airtable bases with AutoGPT workflows for powerful no-code automation capabilities. ## Why Airtable Integration? Airtable is one of the most popular no-code databases used by teams for project management, CRMs, inventory tracking, and countless other use cases. This integration brings significant value: - Data Automation: Automate data entry, updates, and synchronization between Airtable and other services - Workflow Triggers: React to changes in Airtable bases with webhook-based triggers - Schema Management: Programmatically create and manage Airtable table structures - Bulk Operations: Efficiently process large amounts of data with batch create/update/delete operations ## Key Features ### 🔌 Webhook Trigger - AirtableWebhookTriggerBlock: Listens for changes in Airtable bases and triggers workflows - Supports filtering by table, view, and specific fields - Includes webhook signature validation for security ### 📊 Record Operations - AirtableCreateRecordsBlock: Create single or multiple records (up to 10 at once) - AirtableUpdateRecordsBlock: Update existing records with upsert support - AirtableDeleteRecordsBlock: Delete single or multiple records - AirtableGetRecordBlock: Retrieve specific record details - AirtableListRecordsBlock: Query records with filtering, sorting, and pagination ### 🏗️ Schema Management - AirtableCreateTableBlock: Create new tables with custom field definitions - AirtableUpdateTableBlock: Modify table properties - AirtableAddFieldBlock: Add new fields to existing tables - AirtableUpdateFieldBlock: Update field properties ## Technical Implementation Details ### Authentication - Supports both API Key and OAuth authentication methods - OAuth implementation includes proper token refresh handling - Credentials are securely managed through the platform's credential system ### Webhook Security - Added `credentials` parameter to WebhooksManager interface for proper signature validation - HMAC-SHA256 signature verification ensures webhook authenticity - Webhook cursor tracking prevents duplicate event processing ### API Integration - Comprehensive API client (`_api.py`) with full type safety - Proper error handling and response validation - Support for all Airtable field types and operations ## Changes 🏗️ ### Added Blocks: - AirtableWebhookTriggerBlock - AirtableCreateRecordsBlock - AirtableDeleteRecordsBlock - AirtableGetRecordBlock - AirtableListRecordsBlock - AirtableUpdateRecordsBlock - AirtableAddFieldBlock - AirtableCreateTableBlock - AirtableUpdateFieldBlock - AirtableUpdateTableBlock ### Modified Files: - Updated WebhooksManager interface to support credential-based validation - Modified all webhook handlers to support the new interface ## Test Plan 📋 ### Manual Testing Performed: 1. Authentication Testing - ✅ Verified API key authentication works correctly - ✅ Tested OAuth flow including token refresh - ✅ Confirmed credentials are properly encrypted and stored 2. Webhook Testing - ✅ Created webhook subscriptions for different table events - ✅ Verified signature validation prevents unauthorized requests - ✅ Tested cursor tracking to ensure no duplicate events - ✅ Confirmed webhook cleanup on block deletion 3. Record Operations Testing - ✅ Created single and batch records with various field types - ✅ Updated records with and without upsert functionality - ✅ Listed records with filtering, sorting, and pagination - ✅ Deleted single and multiple records - ✅ Retrieved individual record details 4. Schema Management Testing - ✅ Created tables with multiple field types - ✅ Added fields to existing tables - ✅ Updated table and field properties - ✅ Verified proper error handling for invalid field types 5. Error Handling Testing - ✅ Tested with invalid credentials - ✅ Verified proper error messages for API limits - ✅ Confirmed graceful handling of network errors ### Security Considerations 🔒 1. API Key Management - API keys are stored encrypted in the credential system - Keys are never logged or exposed in error messages - Credentials are passed securely through the execution context 2. Webhook Security - HMAC-SHA256 signature validation on all incoming webhooks - Webhook URLs use secure ingress endpoints - Proper cleanup of webhooks when blocks are deleted 3. OAuth Security - OAuth tokens are securely stored and refreshed - Scopes are limited to necessary permissions - Token refresh happens automatically before expiration ## Configuration Requirements No additional environment variables or configuration changes are required. The integration uses the existing credential management system. ## Checklist 📋 #### For code changes: - [x] I have read the [contributing instructions](https://github.com/Significant-Gravitas/AutoGPT/blob/master/.github/CONTRIBUTING.md) - [x] Confirmed that `make lint` passes - [x] Confirmed that `make test` passes - [x] Updated documentation where needed - [x] Added/updated tests for new functionality - [x] Manually tested all blocks with real Airtable bases - [x] Verified backwards compatibility of webhook interface changes #### Security: - [x] No hard-coded secrets or sensitive information - [x] Proper input validation on all user inputs - [x] Secure credential handling throughout	2025-07-25 13:26:23 +00:00
Swifty	243400e128	feat(platform): Add Block Development SDK with auto-registration system (#10074 ) ## Block Development SDK - Simplifying Block Creation ### Problem Currently, creating a new block requires manual updates to 5+ files scattered across the codebase: - `backend/data/block_cost_config.py` - Manually add block costs - `backend/integrations/credentials_store.py` - Add default credentials - `backend/integrations/providers.py` - Register new providers - `backend/integrations/oauth/__init__.py` - Register OAuth handlers - `backend/integrations/webhooks/__init__.py` - Register webhook managers This creates significant friction for developers, increases the chance of configuration errors, and makes the platform difficult to scale. ### Solution This PR introduces a Block Development SDK that provides: - Single import for all block development needs: `from backend.sdk import ` - Automatic registration of all block configurations - Zero external file modifications required - Provider-based configuration with inheritance ### Changes 🏗️ #### 1. New SDK Module* (`backend/sdk/`) - `__init__.py`: Unified exports of 68+ block development components - `registry.py`: Central auto-registration system for all block configurations - `builder.py`: `ProviderBuilder` class for fluent provider configuration - `provider.py`: Provider configuration management - `cost_integration.py`: Automatic cost application system #### 2. Provider Builder Pattern ```python # Configure once, use everywhere my_provider = ( ProviderBuilder("my-service") .with_api_key("MY_SERVICE_API_KEY", "My Service API Key") .with_base_cost(5, BlockCostType.RUN) .build() ) ``` #### 3. Automatic Cost System - Provider base costs automatically applied to all blocks using that provider - Override with `@cost` decorator for block-specific pricing - Tiered pricing support with cost filters #### 4. Dynamic Provider Support - Modified `ProviderName` enum to accept any string via `_missing_` method - No more manual enum updates for new providers #### 5. Application Integration - Added `sync_all_provider_costs()` to `initialize_blocks()` for automatic cost registration - Maintains full backward compatibility with existing blocks #### 6. Comprehensive Examples (`backend/blocks/examples/`) - `simple_example_block.py` - Basic block structure - `example_sdk_block.py` - Provider with credentials - `cost_example_block.py` - Various cost patterns - `advanced_provider_example.py` - Custom API clients - `example_webhook_sdk_block.py` - Webhook configuration #### 7. Extensive Testing - 6 new test modules with 30+ test cases - Integration tests for all SDK features - Cost calculation verification - Provider registration tests ### Before vs After Before SDK: ```python # 1. Multiple complex imports from backend.data.block import Block, BlockCategory, BlockOutput from backend.data.model import SchemaField, CredentialsField # ... many more imports # 2. Update block_cost_config.py BLOCK_COSTS[MyBlock] = [BlockCost(...)] # 3. Update credentials_store.py DEFAULT_CREDENTIALS.append(...) # 4. Update providers.py enum # 5. Update oauth/__init__.py # 6. Update webhooks/__init__.py ``` After SDK: ```python from backend.sdk import * # Everything configured in one place my_provider = ( ProviderBuilder("my-service") .with_api_key("MY_API_KEY", "My API Key") .with_base_cost(10, BlockCostType.RUN) .build() ) class MyBlock(Block): class Input(BlockSchema): credentials: CredentialsMetaInput = my_provider.credentials_field() data: String = SchemaField(description="Input data") class Output(BlockSchema): result: String = SchemaField(description="Result") # That's it\! No external files to modify ``` ### Checklist 📋 #### For code changes: - [x] I have clearly listed my changes in the PR description - [x] I have made a test plan - [x] I have tested my changes according to the test plan: - [x] Created new blocks using SDK pattern with provider configuration - [x] Verified automatic cost registration for provider-based blocks - [x] Tested cost override with @cost decorator - [x] Confirmed custom providers work without enum modifications - [x] Verified all example blocks execute correctly - [x] Tested backward compatibility with existing blocks - [x] Ran all SDK tests (30+ tests, all passing) - [x] Created blocks with credentials and verified authentication - [x] Tested webhook block configuration - [x] Verified application startup with auto-registration #### For configuration changes: - [x] `.env.example` is updated or already compatible with my changes (no changes needed) - [x] `docker-compose.yml` is updated or already compatible with my changes (no changes needed) - [x] I have included a list of my configuration changes in the PR description (under Changes) ### Impact - Developer Experience: Block creation time reduced from hours to minutes - Maintainability: All block configuration in one place - Scalability: Support hundreds of blocks without enum updates - Type Safety: Full IDE support with proper type hints - Testing: Easier to test blocks in isolation --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Abhimanyu Yadav <122007096+Abhi1992002@users.noreply.github.com>	2025-07-10 16:17:55 +02:00

Author

SHA1

Message

Date

Swifty

c955e9a4d7

feat(blocks): Add Airtable Integration (#10338 )

## Overview

This PR adds comprehensive Airtable integration to the AutoGPT platform,
enabling users to seamlessly connect their Airtable bases with AutoGPT
workflows for powerful no-code automation capabilities.

## Why Airtable Integration?

Airtable is one of the most popular no-code databases used by teams for
project management, CRMs, inventory tracking, and countless other use
cases. This integration brings significant value:

- **Data Automation**: Automate data entry, updates, and synchronization
between Airtable and other services
- **Workflow Triggers**: React to changes in Airtable bases with
webhook-based triggers
- **Schema Management**: Programmatically create and manage Airtable
table structures
- **Bulk Operations**: Efficiently process large amounts of data with
batch create/update/delete operations

## Key Features

### 🔌 Webhook Trigger
- **AirtableWebhookTriggerBlock**: Listens for changes in Airtable bases
and triggers workflows
- Supports filtering by table, view, and specific fields
- Includes webhook signature validation for security

### 📊 Record Operations  
- **AirtableCreateRecordsBlock**: Create single or multiple records (up
to 10 at once)
- **AirtableUpdateRecordsBlock**: Update existing records with upsert
support
- **AirtableDeleteRecordsBlock**: Delete single or multiple records
- **AirtableGetRecordBlock**: Retrieve specific record details
- **AirtableListRecordsBlock**: Query records with filtering, sorting,
and pagination

### 🏗️ Schema Management
- **AirtableCreateTableBlock**: Create new tables with custom field
definitions
- **AirtableUpdateTableBlock**: Modify table properties
- **AirtableAddFieldBlock**: Add new fields to existing tables
- **AirtableUpdateFieldBlock**: Update field properties

## Technical Implementation Details

### Authentication
- Supports both API Key and OAuth authentication methods
- OAuth implementation includes proper token refresh handling
- Credentials are securely managed through the platform's credential
system

### Webhook Security
- Added `credentials` parameter to WebhooksManager interface for proper
signature validation
- HMAC-SHA256 signature verification ensures webhook authenticity
- Webhook cursor tracking prevents duplicate event processing

### API Integration
- Comprehensive API client (`_api.py`) with full type safety
- Proper error handling and response validation
- Support for all Airtable field types and operations

## Changes 🏗️ 

### Added Blocks:
- AirtableWebhookTriggerBlock
- AirtableCreateRecordsBlock
- AirtableDeleteRecordsBlock
- AirtableGetRecordBlock
- AirtableListRecordsBlock
- AirtableUpdateRecordsBlock
- AirtableAddFieldBlock
- AirtableCreateTableBlock
- AirtableUpdateFieldBlock
- AirtableUpdateTableBlock

### Modified Files:
- Updated WebhooksManager interface to support credential-based
validation
- Modified all webhook handlers to support the new interface

## Test Plan 📋

### Manual Testing Performed:
1. **Authentication Testing**
   - ✅ Verified API key authentication works correctly
   - ✅ Tested OAuth flow including token refresh
   - ✅ Confirmed credentials are properly encrypted and stored

2. **Webhook Testing**
   - ✅ Created webhook subscriptions for different table events
   - ✅ Verified signature validation prevents unauthorized requests
   - ✅ Tested cursor tracking to ensure no duplicate events
   - ✅ Confirmed webhook cleanup on block deletion

3. **Record Operations Testing**
   - ✅ Created single and batch records with various field types
   - ✅ Updated records with and without upsert functionality
   - ✅ Listed records with filtering, sorting, and pagination
   - ✅ Deleted single and multiple records
   - ✅ Retrieved individual record details

4. **Schema Management Testing**
   - ✅ Created tables with multiple field types
   - ✅ Added fields to existing tables
   - ✅ Updated table and field properties
   - ✅ Verified proper error handling for invalid field types

5. **Error Handling Testing**
   - ✅ Tested with invalid credentials
   - ✅ Verified proper error messages for API limits
   - ✅ Confirmed graceful handling of network errors

### Security Considerations 🔒

1. **API Key Management**
   - API keys are stored encrypted in the credential system
   - Keys are never logged or exposed in error messages
   - Credentials are passed securely through the execution context

2. **Webhook Security**
   - HMAC-SHA256 signature validation on all incoming webhooks
   - Webhook URLs use secure ingress endpoints
   - Proper cleanup of webhooks when blocks are deleted

3. **OAuth Security**
   - OAuth tokens are securely stored and refreshed
   - Scopes are limited to necessary permissions
   - Token refresh happens automatically before expiration

## Configuration Requirements

No additional environment variables or configuration changes are
required. The integration uses the existing credential management
system.

## Checklist 📋

#### For code changes:
- [x] I have read the [contributing
instructions](https://github.com/Significant-Gravitas/AutoGPT/blob/master/.github/CONTRIBUTING.md)
- [x] Confirmed that `make lint` passes
- [x] Confirmed that `make test` passes  
- [x] Updated documentation where needed
- [x] Added/updated tests for new functionality
- [x] Manually tested all blocks with real Airtable bases
- [x] Verified backwards compatibility of webhook interface changes

#### Security:
- [x] No hard-coded secrets or sensitive information
- [x] Proper input validation on all user inputs
- [x] Secure credential handling throughout

2025-07-25 13:26:23 +00:00

Swifty

243400e128

feat(platform): Add Block Development SDK with auto-registration system (#10074 )

## Block Development SDK - Simplifying Block Creation

### Problem
Currently, creating a new block requires manual updates to **5+ files**
scattered across the codebase:
- `backend/data/block_cost_config.py` - Manually add block costs
- `backend/integrations/credentials_store.py` - Add default credentials
- `backend/integrations/providers.py` - Register new providers
- `backend/integrations/oauth/__init__.py` - Register OAuth handlers
- `backend/integrations/webhooks/__init__.py` - Register webhook
managers

This creates significant friction for developers, increases the chance
of configuration errors, and makes the platform difficult to scale.

### Solution
This PR introduces a **Block Development SDK** that provides:
- Single import for all block development needs: `from backend.sdk
import *`
- Automatic registration of all block configurations
- Zero external file modifications required
- Provider-based configuration with inheritance

### Changes 🏗️

#### 1. **New SDK Module** (`backend/sdk/`)
- **`__init__.py`**: Unified exports of 68+ block development components
- **`registry.py`**: Central auto-registration system for all block
configurations
- **`builder.py`**: `ProviderBuilder` class for fluent provider
configuration
- **`provider.py`**: Provider configuration management
- **`cost_integration.py`**: Automatic cost application system

#### 2. **Provider Builder Pattern**
```python
# Configure once, use everywhere
my_provider = (
    ProviderBuilder("my-service")
    .with_api_key("MY_SERVICE_API_KEY", "My Service API Key")
    .with_base_cost(5, BlockCostType.RUN)
    .build()
)
```

#### 3. **Automatic Cost System**
- Provider base costs automatically applied to all blocks using that
provider
- Override with `@cost` decorator for block-specific pricing
- Tiered pricing support with cost filters

#### 4. **Dynamic Provider Support**
- Modified `ProviderName` enum to accept any string via `_missing_`
method
- No more manual enum updates for new providers

#### 5. **Application Integration**
- Added `sync_all_provider_costs()` to `initialize_blocks()` for
automatic cost registration
- Maintains full backward compatibility with existing blocks

#### 6. **Comprehensive Examples** (`backend/blocks/examples/`)
- `simple_example_block.py` - Basic block structure
- `example_sdk_block.py` - Provider with credentials
- `cost_example_block.py` - Various cost patterns
- `advanced_provider_example.py` - Custom API clients
- `example_webhook_sdk_block.py` - Webhook configuration

#### 7. **Extensive Testing**
- 6 new test modules with 30+ test cases
- Integration tests for all SDK features
- Cost calculation verification
- Provider registration tests

### Before vs After

**Before SDK:**
```python
# 1. Multiple complex imports
from backend.data.block import Block, BlockCategory, BlockOutput
from backend.data.model import SchemaField, CredentialsField
# ... many more imports

# 2. Update block_cost_config.py
BLOCK_COSTS[MyBlock] = [BlockCost(...)]

# 3. Update credentials_store.py
DEFAULT_CREDENTIALS.append(...)

# 4. Update providers.py enum
# 5. Update oauth/__init__.py
# 6. Update webhooks/__init__.py
```

**After SDK:**
```python
from backend.sdk import *

# Everything configured in one place
my_provider = (
    ProviderBuilder("my-service")
    .with_api_key("MY_API_KEY", "My API Key")
    .with_base_cost(10, BlockCostType.RUN)
    .build()
)

class MyBlock(Block):
    class Input(BlockSchema):
        credentials: CredentialsMetaInput = my_provider.credentials_field()
        data: String = SchemaField(description="Input data")
    
    class Output(BlockSchema):
        result: String = SchemaField(description="Result")
    
    # That's it\! No external files to modify
```

### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
  - [x] Created new blocks using SDK pattern with provider configuration
  - [x] Verified automatic cost registration for provider-based blocks
  - [x] Tested cost override with @cost decorator
  - [x] Confirmed custom providers work without enum modifications
  - [x] Verified all example blocks execute correctly
  - [x] Tested backward compatibility with existing blocks
  - [x] Ran all SDK tests (30+ tests, all passing)
  - [x] Created blocks with credentials and verified authentication
  - [x] Tested webhook block configuration
  - [x] Verified application startup with auto-registration

#### For configuration changes:
- [x] `.env.example` is updated or already compatible with my changes
(no changes needed)
- [x] `docker-compose.yml` is updated or already compatible with my
changes (no changes needed)
- [x] I have included a list of my configuration changes in the PR
description (under **Changes**)

### Impact

- **Developer Experience**: Block creation time reduced from hours to
minutes
- **Maintainability**: All block configuration in one place
- **Scalability**: Support hundreds of blocks without enum updates
- **Type Safety**: Full IDE support with proper type hints
- **Testing**: Easier to test blocks in isolation

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Abhimanyu Yadav <122007096+Abhi1992002@users.noreply.github.com>

2025-07-10 16:17:55 +02:00

2 Commits