mirror of
https://github.com/Significant-Gravitas/AutoGPT.git
synced 2026-02-09 14:25:25 -05:00
## Overview This PR adds comprehensive Airtable integration to the AutoGPT platform, enabling users to seamlessly connect their Airtable bases with AutoGPT workflows for powerful no-code automation capabilities. ## Why Airtable Integration? Airtable is one of the most popular no-code databases used by teams for project management, CRMs, inventory tracking, and countless other use cases. This integration brings significant value: - **Data Automation**: Automate data entry, updates, and synchronization between Airtable and other services - **Workflow Triggers**: React to changes in Airtable bases with webhook-based triggers - **Schema Management**: Programmatically create and manage Airtable table structures - **Bulk Operations**: Efficiently process large amounts of data with batch create/update/delete operations ## Key Features ### 🔌 Webhook Trigger - **AirtableWebhookTriggerBlock**: Listens for changes in Airtable bases and triggers workflows - Supports filtering by table, view, and specific fields - Includes webhook signature validation for security ### 📊 Record Operations - **AirtableCreateRecordsBlock**: Create single or multiple records (up to 10 at once) - **AirtableUpdateRecordsBlock**: Update existing records with upsert support - **AirtableDeleteRecordsBlock**: Delete single or multiple records - **AirtableGetRecordBlock**: Retrieve specific record details - **AirtableListRecordsBlock**: Query records with filtering, sorting, and pagination ### 🏗️ Schema Management - **AirtableCreateTableBlock**: Create new tables with custom field definitions - **AirtableUpdateTableBlock**: Modify table properties - **AirtableAddFieldBlock**: Add new fields to existing tables - **AirtableUpdateFieldBlock**: Update field properties ## Technical Implementation Details ### Authentication - Supports both API Key and OAuth authentication methods - OAuth implementation includes proper token refresh handling - Credentials are securely managed through the platform's credential system ### Webhook Security - Added `credentials` parameter to WebhooksManager interface for proper signature validation - HMAC-SHA256 signature verification ensures webhook authenticity - Webhook cursor tracking prevents duplicate event processing ### API Integration - Comprehensive API client (`_api.py`) with full type safety - Proper error handling and response validation - Support for all Airtable field types and operations ## Changes 🏗️ ### Added Blocks: - AirtableWebhookTriggerBlock - AirtableCreateRecordsBlock - AirtableDeleteRecordsBlock - AirtableGetRecordBlock - AirtableListRecordsBlock - AirtableUpdateRecordsBlock - AirtableAddFieldBlock - AirtableCreateTableBlock - AirtableUpdateFieldBlock - AirtableUpdateTableBlock ### Modified Files: - Updated WebhooksManager interface to support credential-based validation - Modified all webhook handlers to support the new interface ## Test Plan 📋 ### Manual Testing Performed: 1. **Authentication Testing** - ✅ Verified API key authentication works correctly - ✅ Tested OAuth flow including token refresh - ✅ Confirmed credentials are properly encrypted and stored 2. **Webhook Testing** - ✅ Created webhook subscriptions for different table events - ✅ Verified signature validation prevents unauthorized requests - ✅ Tested cursor tracking to ensure no duplicate events - ✅ Confirmed webhook cleanup on block deletion 3. **Record Operations Testing** - ✅ Created single and batch records with various field types - ✅ Updated records with and without upsert functionality - ✅ Listed records with filtering, sorting, and pagination - ✅ Deleted single and multiple records - ✅ Retrieved individual record details 4. **Schema Management Testing** - ✅ Created tables with multiple field types - ✅ Added fields to existing tables - ✅ Updated table and field properties - ✅ Verified proper error handling for invalid field types 5. **Error Handling Testing** - ✅ Tested with invalid credentials - ✅ Verified proper error messages for API limits - ✅ Confirmed graceful handling of network errors ### Security Considerations 🔒 1. **API Key Management** - API keys are stored encrypted in the credential system - Keys are never logged or exposed in error messages - Credentials are passed securely through the execution context 2. **Webhook Security** - HMAC-SHA256 signature validation on all incoming webhooks - Webhook URLs use secure ingress endpoints - Proper cleanup of webhooks when blocks are deleted 3. **OAuth Security** - OAuth tokens are securely stored and refreshed - Scopes are limited to necessary permissions - Token refresh happens automatically before expiration ## Configuration Requirements No additional environment variables or configuration changes are required. The integration uses the existing credential management system. ## Checklist 📋 #### For code changes: - [x] I have read the [contributing instructions](https://github.com/Significant-Gravitas/AutoGPT/blob/master/.github/CONTRIBUTING.md) - [x] Confirmed that `make lint` passes - [x] Confirmed that `make test` passes - [x] Updated documentation where needed - [x] Added/updated tests for new functionality - [x] Manually tested all blocks with real Airtable bases - [x] Verified backwards compatibility of webhook interface changes #### Security: - [x] No hard-coded secrets or sensitive information - [x] Proper input validation on all user inputs - [x] Secure credential handling throughout
230 lines
7.3 KiB
Python
230 lines
7.3 KiB
Python
import logging
|
|
import secrets
|
|
from abc import ABC, abstractmethod
|
|
from typing import ClassVar, Generic, Optional, TypeVar
|
|
from uuid import uuid4
|
|
|
|
from fastapi import Request
|
|
from strenum import StrEnum
|
|
|
|
import backend.data.integrations as integrations
|
|
from backend.data.model import Credentials
|
|
from backend.integrations.providers import ProviderName
|
|
from backend.util.exceptions import MissingConfigError
|
|
from backend.util.settings import Config
|
|
|
|
from .utils import webhook_ingress_url
|
|
|
|
logger = logging.getLogger(__name__)
|
|
app_config = Config()
|
|
|
|
WT = TypeVar("WT", bound=StrEnum)
|
|
|
|
|
|
class BaseWebhooksManager(ABC, Generic[WT]):
|
|
# --8<-- [start:BaseWebhooksManager1]
|
|
PROVIDER_NAME: ClassVar[ProviderName]
|
|
# --8<-- [end:BaseWebhooksManager1]
|
|
|
|
WebhookType: WT
|
|
|
|
async def get_suitable_auto_webhook(
|
|
self,
|
|
user_id: str,
|
|
credentials: Credentials,
|
|
webhook_type: WT,
|
|
resource: str,
|
|
events: list[str],
|
|
) -> integrations.Webhook:
|
|
if not app_config.platform_base_url:
|
|
raise MissingConfigError(
|
|
"PLATFORM_BASE_URL must be set to use Webhook functionality"
|
|
)
|
|
|
|
if webhook := await integrations.find_webhook_by_credentials_and_props(
|
|
user_id=user_id,
|
|
credentials_id=credentials.id,
|
|
webhook_type=webhook_type,
|
|
resource=resource,
|
|
events=events,
|
|
):
|
|
return webhook
|
|
|
|
return await self._create_webhook(
|
|
user_id=user_id,
|
|
webhook_type=webhook_type,
|
|
events=events,
|
|
resource=resource,
|
|
credentials=credentials,
|
|
)
|
|
|
|
async def get_manual_webhook(
|
|
self,
|
|
user_id: str,
|
|
webhook_type: WT,
|
|
events: list[str],
|
|
graph_id: Optional[str] = None,
|
|
preset_id: Optional[str] = None,
|
|
) -> integrations.Webhook:
|
|
"""
|
|
Tries to find an existing webhook tied to `graph_id`/`preset_id`,
|
|
or creates a new webhook if none exists.
|
|
|
|
Existing webhooks are matched by `user_id`, `webhook_type`,
|
|
and `graph_id`/`preset_id`.
|
|
|
|
If an existing webhook is found, we check if the events match and update them
|
|
if necessary. We do this rather than creating a new webhook
|
|
to avoid changing the webhook URL for existing manual webhooks.
|
|
"""
|
|
if (graph_id or preset_id) and (
|
|
current_webhook := await integrations.find_webhook_by_graph_and_props(
|
|
user_id=user_id,
|
|
provider=self.PROVIDER_NAME.value,
|
|
webhook_type=webhook_type,
|
|
graph_id=graph_id,
|
|
preset_id=preset_id,
|
|
)
|
|
):
|
|
if set(current_webhook.events) != set(events):
|
|
current_webhook = await integrations.update_webhook(
|
|
current_webhook.id, events=events
|
|
)
|
|
return current_webhook
|
|
|
|
return await self._create_webhook(
|
|
user_id=user_id,
|
|
webhook_type=webhook_type,
|
|
events=events,
|
|
register=False,
|
|
)
|
|
|
|
async def prune_webhook_if_dangling(
|
|
self, user_id: str, webhook_id: str, credentials: Optional[Credentials]
|
|
) -> bool:
|
|
webhook = await integrations.get_webhook(webhook_id, include_relations=True)
|
|
if webhook.triggered_nodes or webhook.triggered_presets:
|
|
# Don't prune webhook if in use
|
|
return False
|
|
|
|
if credentials:
|
|
await self._deregister_webhook(webhook, credentials)
|
|
await integrations.delete_webhook(user_id, webhook.id)
|
|
return True
|
|
|
|
# --8<-- [start:BaseWebhooksManager3]
|
|
@classmethod
|
|
@abstractmethod
|
|
async def validate_payload(
|
|
cls,
|
|
webhook: integrations.Webhook,
|
|
request: Request,
|
|
credentials: Credentials | None,
|
|
) -> tuple[dict, str]:
|
|
"""
|
|
Validates an incoming webhook request and returns its payload and type.
|
|
|
|
Params:
|
|
webhook: Object representing the configured webhook and its properties in our system.
|
|
request: Incoming FastAPI `Request`
|
|
|
|
Returns:
|
|
dict: The validated payload
|
|
str: The event type associated with the payload
|
|
"""
|
|
|
|
# --8<-- [end:BaseWebhooksManager3]
|
|
|
|
# --8<-- [start:BaseWebhooksManager5]
|
|
async def trigger_ping(
|
|
self, webhook: integrations.Webhook, credentials: Credentials | None
|
|
) -> None:
|
|
"""
|
|
Triggers a ping to the given webhook.
|
|
|
|
Raises:
|
|
NotImplementedError: if the provider doesn't support pinging
|
|
"""
|
|
# --8<-- [end:BaseWebhooksManager5]
|
|
raise NotImplementedError(f"{self.__class__.__name__} doesn't support pinging")
|
|
|
|
# --8<-- [start:BaseWebhooksManager2]
|
|
@abstractmethod
|
|
async def _register_webhook(
|
|
self,
|
|
credentials: Credentials,
|
|
webhook_type: WT,
|
|
resource: str,
|
|
events: list[str],
|
|
ingress_url: str,
|
|
secret: str,
|
|
) -> tuple[str, dict]:
|
|
"""
|
|
Registers a new webhook with the provider.
|
|
|
|
Params:
|
|
credentials: The credentials with which to create the webhook
|
|
webhook_type: The provider-specific webhook type to create
|
|
resource: The resource to receive events for
|
|
events: The events to subscribe to
|
|
ingress_url: The ingress URL for webhook payloads
|
|
secret: Secret used to verify webhook payloads
|
|
|
|
Returns:
|
|
str: Webhook ID assigned by the provider
|
|
config: Provider-specific configuration for the webhook
|
|
"""
|
|
...
|
|
|
|
# --8<-- [end:BaseWebhooksManager2]
|
|
|
|
# --8<-- [start:BaseWebhooksManager4]
|
|
@abstractmethod
|
|
async def _deregister_webhook(
|
|
self, webhook: integrations.Webhook, credentials: Credentials
|
|
) -> None: ...
|
|
|
|
# --8<-- [end:BaseWebhooksManager4]
|
|
|
|
async def _create_webhook(
|
|
self,
|
|
user_id: str,
|
|
webhook_type: WT,
|
|
events: list[str],
|
|
resource: str = "",
|
|
credentials: Optional[Credentials] = None,
|
|
register: bool = True,
|
|
) -> integrations.Webhook:
|
|
if not app_config.platform_base_url:
|
|
raise MissingConfigError(
|
|
"PLATFORM_BASE_URL must be set to use Webhook functionality"
|
|
)
|
|
|
|
id = str(uuid4())
|
|
secret = secrets.token_hex(32)
|
|
provider_name: ProviderName = self.PROVIDER_NAME
|
|
ingress_url = webhook_ingress_url(provider_name=provider_name, webhook_id=id)
|
|
if register:
|
|
if not credentials:
|
|
raise TypeError("credentials are required if register = True")
|
|
provider_webhook_id, config = await self._register_webhook(
|
|
credentials, webhook_type, resource, events, ingress_url, secret
|
|
)
|
|
else:
|
|
provider_webhook_id, config = "", {}
|
|
|
|
return await integrations.create_webhook(
|
|
integrations.Webhook(
|
|
id=id,
|
|
user_id=user_id,
|
|
provider=provider_name,
|
|
credentials_id=credentials.id if credentials else "",
|
|
webhook_type=webhook_type,
|
|
resource=resource,
|
|
events=events,
|
|
provider_webhook_id=provider_webhook_id,
|
|
config=config,
|
|
secret=secret,
|
|
)
|
|
)
|