github/AutoGPT
1
1
Fork 0
mirror of https://github.com/Significant-Gravitas/AutoGPT.git synced 2026-04-30 03:00:41 -04:00
Code Issues Packages Projects Releases Wiki Activity
7,094 Commits 706 Branches 104 Tags
d4b5508ed14dc1742bf5f0bdb7762b5253a73708
Commit Graph

8 Commits

Author SHA1 Message Date
Nicholas Tindle
0116866199 feat(backend): add more discord blocks support (#10586) 
# Enhanced Discord Integration Blocks

Introduces new blocks for sending DMs, embeds, files, and replies in
Discord, as well as blocks for retrieving user and channel information.
Enhances existing message blocks with additional metadata fields and
server/channel identification. Improves test coverage and input/output
schemas for all Discord-related blocks.

Co-Authored-By: Claude <claude@users.noreply.github.com>

## Why These Changes Are Needed 🎯

The existing Discord integration was limited to basic message sending
and reading. Users needed more sophisticated Discord functionality to
build comprehensive automation workflows:

1. **Limited messaging options** - Could only send plain text to
channels, no DMs, embeds, or file attachments
2. **Poor graph connectivity** - Blocks didn't output IDs needed for
chaining operations (e.g., couldn't reply to a message after sending it)
3. **No user management** - Couldn't get user information or send direct
messages
4. **Type safety issues** - Discord.py's incomplete type hints caused
linting errors
5. **No channel resolution** - Had to manually find channel IDs instead
of using names

### Changes 🏗️

#### New Blocks Added
- **SendDiscordDMBlock** - Send direct messages to users via their
Discord ID
- **SendDiscordEmbedBlock** - Create rich embedded messages with images,
fields, and formatting
- **SendDiscordFileBlock** - Upload any file type (images, PDFs, videos,
etc.) using MediaFileType
- **ReplyToDiscordMessageBlock** - Reply to specific messages in threads
- **DiscordUserInfoBlock** - Retrieve user profile information
(username, avatar, creation date, etc.)
- **DiscordChannelInfoBlock** - Resolve channel names to IDs and get
channel metadata

#### Enhanced Existing Blocks
- **ReadDiscordMessagesBlock**:
- Now outputs: `message_id`, `channel_id`, `user_id` (previously missing
all IDs)
- Enables workflows like: read message → reply to it, or read message →
DM the author
  
- **SendDiscordMessageBlock**:
- Now outputs: `message_id`, `channel_id` (previously had no outputs
except status)
  - Enables tracking sent messages and replying to them later

#### Technical Improvements
- **MediaFileType Support**: SendDiscordFileBlock accepts data URIs,
URLs, or local paths
- **Defensive Programming**: Added runtime type checks for Discord.py's
incomplete typing
- **ID Passthrough**: DiscordUserInfoBlock passes through user_id for
chaining
- **Better Error Messages**: Clear feedback when operations fail (e.g.,
"Channel cannot receive messages")
- **Channel Flexibility**: Blocks accept both channel names and IDs

### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:

#### Test Plan 🧪
- [x] **Import and initialization**: All 8 Discord blocks import and
initialize without errors
- [x] **Type checking**: `poetry run format` passes with no type errors
- [x] **Interface connectivity**: Verified blocks can chain together:
- [x] ReadDiscordMessages → ReplyToDiscordMessage (via message_id,
channel_id)
  - [x] ReadDiscordMessages → SendDiscordDM (via user_id)
- [x] SendDiscordMessage → ReplyToDiscordMessage (via message_id,
channel_id)
  - [x] DiscordUserInfo → SendDiscordDM (via user_id passthrough)
  - [x] DiscordChannelInfo → SendDiscordEmbed/File (via channel_id)
- [x] **MediaFileType handling**: SendDiscordFileBlock correctly
processes:
  - [x] Data URIs (base64 encoded files)
  - [x] URLs (downloads from web)
  - [x] Local paths (from other blocks)
- [x] **Defensive checks**: Verified error handling for:
  - [x] Non-text channels (forums, categories)
  - [x] Private/DM channels without guilds
  - [x] Missing attributes on channel objects
- [x] **Mock test data**: All blocks have appropriate test
inputs/outputs defined

## Example Workflows Now Possible 🚀

1. **Auto-reply to mentions**: Read messages → Check if bot mentioned →
Reply in thread
2. **File distribution**: Generate report → Send as PDF to Discord
channel
3. **User notifications**: Get user info → Check if online → Send DM
with alert
4. **Cross-platform sync**: Receive email attachment → Forward to
Discord channel
5. **Rich notifications**: Create embed with thumbnail → Add fields →
Send to announcement channel

## Breaking Changes ⚠️

None - all changes are backward compatible. Existing workflows using
SendDiscordMessageBlock and ReadDiscordMessagesBlock will continue to
work, they just now have additional outputs available.

## Dependencies 📦

No new dependencies added. Uses existing:
- `discord.py` (already in project)
- `aiohttp` (already in project)
- Backend utilities: `MediaFileType`, `store_media_file` (already in
project)

---------

Co-authored-by: Claude <claude@users.noreply.github.com>
 2025-08-08 18:45:04 +00:00
Nicholas Tindle
178c91d6b9 ref(backend): time/date blocks to support ISO 8601 and custom formats (#10576) 
Introduces discriminated unions for time, date, and date-time format
selection, supporting both strftime and ISO 8601 (with timezone and
microsecond options). Updates schemas, test cases, and block logic to
handle the new format types, improving flexibility and standards
compliance for time and date outputs.

<!-- Clearly explain the need for these changes: -->

### Why these changes are needed

Users need to output timestamps in ISO 8601/RFC 3339 format for API
integrations and standardized data exchange. The previous implementation
only supported strftime formatting, which made it difficult to generate
properly formatted timestamps with timezone information. This change
enables:

- **Standards compliance**: ISO 8601 and RFC 3339 compliant timestamps
- **Timezone support**: 38 timezone options covering all UTC offsets
globally
- **API compatibility**: Many APIs require RFC 3339 timestamps (e.g.,
"2011-06-03T10:00:00-07:00")
- **Backward compatibility**: Existing workflows continue to work with
default strftime format

### Changes 🏗️

<!-- Concisely describe all of the changes made in this pull request:
-->

- **Added discriminated union format types** for all time/date blocks:
- `GetCurrentTimeBlock`: Now supports `TimeStrftimeFormat` and
`TimeISO8601Format`
- `GetCurrentDateBlock`: Now supports `DateStrftimeFormat` and
`DateISO8601Format`
- `GetCurrentDateAndTimeBlock`: Now supports `StrftimeFormat` and
`ISO8601Format`

- **Implemented shared timezone support**:
- Created `TimezoneLiteral` type with 38 timezone options (all UTC
offsets)
  - Supports fractional offsets (e.g., India UTC+05:30, Nepal UTC+05:45)
  - Deduplicated timezone lists across all format classes

- **Added ISO 8601 format features**:
  - Timezone-aware timestamps with proper offset formatting
  - Optional microseconds inclusion
  - RFC 3339 compliance (subset of ISO 8601 with mandatory timezone)

- **Updated test cases** for all three blocks to verify:
  - Default behavior unchanged (backward compatibility)
  - Custom strftime formats still work
  - ISO 8601 format produces correct output

### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
  <!-- Put your test plan here: -->
- [x] Verified backward compatibility - default strftime format
unchanged
  - [x] Tested ISO 8601 format with UTC timezone
- [x] Tested ISO 8601 format with various timezones (India, New York,
etc.)
  - [x] Tested microseconds option for ISO formats
  - [x] Verified all existing tests pass for GetCurrentTimeBlock
  - [x] Verified all existing tests pass for GetCurrentDateBlock
  - [x] Verified all existing tests pass for GetCurrentDateAndTimeBlock
  - [x] Manually tested each block with different format configurations
- [x] Confirmed RFC 3339 compliance for timestamps with mandatory
timezone

---------

Co-authored-by: Claude <claude@users.noreply.github.com>
 2025-08-07 22:34:31 +00:00
Zamil Majdy
b85e8204df refactor(platform): remove unused functions and imports (#10535) 
## Summary
- Removed unused metadata functions from user.py (get_user_metadata,
update_user_metadata)
- Removed unused execution and database functions from database.py and
related imports
- Added NodeExecutionStats validation in execution.py
- Updated CLAUDE.md with PR and commit conventions

## Changes Made
### `/backend/backend/data/user.py`
- Removed `get_user_metadata()` function (unused)
- Removed `update_user_metadata()` function (unused)
- Removed unused import `UserMetadataRaw`

### `/backend/backend/data/execution.py`
- Added `NodeExecutionStats` validation in `from_db()` method

### `/backend/backend/executor/database.py`
- Removed unused imports and function exposures
- Cleaned up DatabaseManagerClient to remove unused client methods

### `/CLAUDE.md`
- Added documentation for creating pull requests
- Added conventional commit types and scopes guide

## Testing
- Existing tests should pass as removed functions were not being used
- No new functionality added

## Checklist
- [x] Code follows the project's style guidelines
- [x] Self-review completed
- [x] Changes are backward compatible
- [x] No new warnings introduced

🤖 Generated with [Claude Code](https://claude.ai/code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2025-08-04 21:34:33 +07:00
Swifty
243400e128 feat(platform): Add Block Development SDK with auto-registration system (#10074) 
## Block Development SDK - Simplifying Block Creation

### Problem
Currently, creating a new block requires manual updates to **5+ files**
scattered across the codebase:
- `backend/data/block_cost_config.py` - Manually add block costs
- `backend/integrations/credentials_store.py` - Add default credentials
- `backend/integrations/providers.py` - Register new providers
- `backend/integrations/oauth/__init__.py` - Register OAuth handlers
- `backend/integrations/webhooks/__init__.py` - Register webhook
managers

This creates significant friction for developers, increases the chance
of configuration errors, and makes the platform difficult to scale.

### Solution
This PR introduces a **Block Development SDK** that provides:
- Single import for all block development needs: `from backend.sdk
import *`
- Automatic registration of all block configurations
- Zero external file modifications required
- Provider-based configuration with inheritance

### Changes 🏗️

#### 1. **New SDK Module** (`backend/sdk/`)
- **`__init__.py`**: Unified exports of 68+ block development components
- **`registry.py`**: Central auto-registration system for all block
configurations
- **`builder.py`**: `ProviderBuilder` class for fluent provider
configuration
- **`provider.py`**: Provider configuration management
- **`cost_integration.py`**: Automatic cost application system

#### 2. **Provider Builder Pattern**
```python
# Configure once, use everywhere
my_provider = (
    ProviderBuilder("my-service")
    .with_api_key("MY_SERVICE_API_KEY", "My Service API Key")
    .with_base_cost(5, BlockCostType.RUN)
    .build()
)
```

#### 3. **Automatic Cost System**
- Provider base costs automatically applied to all blocks using that
provider
- Override with `@cost` decorator for block-specific pricing
- Tiered pricing support with cost filters

#### 4. **Dynamic Provider Support**
- Modified `ProviderName` enum to accept any string via `_missing_`
method
- No more manual enum updates for new providers

#### 5. **Application Integration**
- Added `sync_all_provider_costs()` to `initialize_blocks()` for
automatic cost registration
- Maintains full backward compatibility with existing blocks

#### 6. **Comprehensive Examples** (`backend/blocks/examples/`)
- `simple_example_block.py` - Basic block structure
- `example_sdk_block.py` - Provider with credentials
- `cost_example_block.py` - Various cost patterns
- `advanced_provider_example.py` - Custom API clients
- `example_webhook_sdk_block.py` - Webhook configuration

#### 7. **Extensive Testing**
- 6 new test modules with 30+ test cases
- Integration tests for all SDK features
- Cost calculation verification
- Provider registration tests

### Before vs After

**Before SDK:**
```python
# 1. Multiple complex imports
from backend.data.block import Block, BlockCategory, BlockOutput
from backend.data.model import SchemaField, CredentialsField
# ... many more imports

# 2. Update block_cost_config.py
BLOCK_COSTS[MyBlock] = [BlockCost(...)]

# 3. Update credentials_store.py
DEFAULT_CREDENTIALS.append(...)

# 4. Update providers.py enum
# 5. Update oauth/__init__.py
# 6. Update webhooks/__init__.py
```

**After SDK:**
```python
from backend.sdk import *

# Everything configured in one place
my_provider = (
    ProviderBuilder("my-service")
    .with_api_key("MY_API_KEY", "My API Key")
    .with_base_cost(10, BlockCostType.RUN)
    .build()
)

class MyBlock(Block):
    class Input(BlockSchema):
        credentials: CredentialsMetaInput = my_provider.credentials_field()
        data: String = SchemaField(description="Input data")
    
    class Output(BlockSchema):
        result: String = SchemaField(description="Result")
    
    # That's it\! No external files to modify
```

### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
  - [x] Created new blocks using SDK pattern with provider configuration
  - [x] Verified automatic cost registration for provider-based blocks
  - [x] Tested cost override with @cost decorator
  - [x] Confirmed custom providers work without enum modifications
  - [x] Verified all example blocks execute correctly
  - [x] Tested backward compatibility with existing blocks
  - [x] Ran all SDK tests (30+ tests, all passing)
  - [x] Created blocks with credentials and verified authentication
  - [x] Tested webhook block configuration
  - [x] Verified application startup with auto-registration

#### For configuration changes:
- [x] `.env.example` is updated or already compatible with my changes
(no changes needed)
- [x] `docker-compose.yml` is updated or already compatible with my
changes (no changes needed)
- [x] I have included a list of my configuration changes in the PR
description (under **Changes**)

### Impact

- **Developer Experience**: Block creation time reduced from hours to
minutes
- **Maintainability**: All block configuration in one place
- **Scalability**: Support hundreds of blocks without enum updates
- **Type Safety**: Full IDE support with proper type hints
- **Testing**: Easier to test blocks in isolation

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Abhimanyu Yadav <122007096+Abhi1992002@users.noreply.github.com>
 2025-07-10 16:17:55 +02:00
Zamil Majdy
24b4ab9864 feat(block): Enhance Mem0 blocks filetering & add more GoogleSheets blocks (#10287) 
The block library was missing two key capabilities that keep coming up
in real-world agent flows:

1. **Granular Mem0 filtering.** Agents often work side-by-side for the
same user, so memories must be scoped to a specific run or agent to
avoid crosstalk.
2. **First-class Google Sheets support.** Many community projects (e.g.,
data-collection, lightweight dashboards, no-code workflows) rely on
Sheets, but we only had a brittle REST call block.

This PR adds fine-grained filters to every Mem0 retrieval block and
introduces a complete, OAuth-ready Google Sheets suite so agents can
create, read, write, format, and manage spreadsheets safely.
:contentReference[oaicite:0]{index=0}

---

### Changes 🏗️
#### 📚 Mem0 block enhancements  
* Added `categories_filter`, `metadata_filter`, `limit_memory_to_run`,
and `limit_memory_to_agent` inputs to **SearchMemoryBlock**,
**GetAllMemoriesBlock**, and **GetLatestMemoryBlock**.
* Added identical scoping logic to **AddMemoryBlock** so newly-created
memories can be tied to run/agent IDs.

#### 📊 New Google Sheets blocks (`backend/blocks/google/sheets.py`)  
| Block | Purpose |
|-------|---------|
| `GoogleSheetsReadBlock` | Read a range |
| `GoogleSheetsWriteBlock` | Overwrite a range |
| `GoogleSheetsAppendBlock` | Append rows |
| `GoogleSheetsClearBlock` | Clear a range |
| `GoogleSheetsMetadataBlock` | Fetch spreadsheet + sheet info |
| `GoogleSheetsManageSheetBlock` | Create / delete / copy a sheet |
| `GoogleSheetsBatchOperationsBlock` | Batch update / clear |
| `GoogleSheetsFindReplaceBlock` | Find & replace text |
| `GoogleSheetsFormatBlock` | Cell formatting (bg/text colour, bold,
italic, font size) |
| `GoogleSheetsCreateSpreadsheetBlock` | Spin up a new spreadsheet |

* Each block has typed input/output schemas, built-in test mocks, and is
disabled in prod unless Google OAuth is configured.
* Added helper enums (`SheetOperation`, `BatchOperationType`) and
updated **CLAUDE.md** contributor guide with a UUID-generation step.
:contentReference[oaicite:2]{index=2}

---

### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
  <!-- Put your test plan here: -->
- [x] Manual E2E run: agent writes chat summary to new spreadsheet,
reads it back, searches memory with run-scoped filter
- [x] Live Google API smoke-tests (read/write/append/clear/format) using
a disposable spreadsheet
 2025-07-03 18:01:30 +00:00
Zamil Majdy
68749a28d4 feat(backend): Add ClamAV support for anti-virus scan on file upload to the platform (#10232) 
An anti-virus file scan step is added to each file upload step on the
platform before the file is sent to cloud storage or local machine
storage.

### Changes 🏗️

* Added ClamAV service
* Added AV file scan on each upload step
* Added tests & documentation
* Make the step mandatory even on local development

### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
  <!-- Put your test plan here: -->
  - [x] Tried using FileUploadBlock & AgentFileInputBlock
 2025-06-25 15:29:04 +00:00
Nicholas Tindle
568f5a449e feat(backend): cache control headers (#10160) 
### Why? 🤔
  <!-- Clearly explain the need for these changes: -->
  We need to prevent sensitive data (authentication tokens, API
  keys, user credentials, personal information) from being cached by
   browsers and proxies. Following the principle of "secure by
  default", we're switching from a deny list to an allow list
  approach for cache control.

  ### Changes 🛠️
  <!-- Concisely describe all of the changes made in this pull
  request: -->
  - **Refactored cache control middleware from deny list to allow 
  list approach**
    - By default, ALL endpoints now have `Cache-Control: no-store,
  no-cache, must-revalidate, private` headers
    - Only explicitly allowed paths (static assets, health checks,
  public store pages) can be cached
    - This ensures new endpoints are automatically protected without
   developers having to remember to add them to a list

  - **Updated `SecurityHeadersMiddleware` in 
  `/backend/backend/server/middleware/security.py`**
    - Renamed `SENSITIVE_PATHS` to `CACHEABLE_PATHS`
    - Inverted the logic in `is_cacheable_path()` method
    - Cache control headers are now applied to all paths NOT in the
  allow list

  - **Updated test suite to match new behavior**
    - Tests now verify that most endpoints have cache control
  headers by default
    - Tests verify that only allowed paths (static assets, health
  endpoints, etc.) can be cached

  - **Updated documentation in `CLAUDE.md`**
    - Documented the new allow list approach
    - Added instructions for developers on how to allow caching for
  new endpoints


### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
  <!-- Put your test plan here: -->
  - [x] Test modified endpoints work still
  - [x] Test modified endpoints correctly have no cache rules

---------

Co-authored-by: Swifty <craigswift13@gmail.com>
 2025-06-20 14:19:52 +00:00
Swifty
a3d082a5fa feat(backend): snapshot test responses (#10039) 
This pull request introduces a comprehensive backend testing guide and
adds new tests for analytics logging and various API endpoints, focusing
on snapshot testing. It also includes corresponding snapshot files for
these tests. Below are the most significant changes:

### Documentation Updates:
* Added a detailed `TESTING.md` file to the backend, providing a guide
for running tests, snapshot testing, writing API route tests, and best
practices. It includes examples for mocking, fixtures, and CI/CD
integration.

### Analytics Logging Tests:
* Implemented tests for logging raw metrics and analytics in
`analytics_test.py`, covering success scenarios, various input values,
invalid requests, and complex nested data. These tests utilize snapshot
testing for response validation.
* Added snapshot files for analytics logging tests, including responses
for success cases, various metric values, and complex analytics data.
[[1]](diffhunk://#diff-654bc5aa1951008ec5c110a702279ef58709ee455ba049b9fa825fa60f7e3869R1-R3)
[[2]](diffhunk://#diff-e0a434b107abc71aeffb7d7989dbfd8f466b5e53f8dea25a87937ec1b885b122R1-R3)
[[3]](diffhunk://#diff-dd0bc0b72264de1a0c0d3bd0c54ad656061317f425e4de461018ca51a19171a0R1-R3)
[[4]](diffhunk://#diff-63af007073db553d04988544af46930458a768544cabd08412265e0818320d11R1-R30)

### Snapshot Files for API Endpoints:
* Added snapshot files for various API endpoint tests, such as:
- Graph-related operations (`graphs_get_single_response`,
`graphs_get_all_response`, `blocks_get_all_response`).
[[1]](diffhunk://#diff-b25dba271606530cfa428c00073d7e016184a7bb22166148ab1726b3e113dda8R1-R29)
[[2]](diffhunk://#diff-1054e58ec3094715660f55bfba1676d65b6833a81a91a08e90ad57922444d056R1-R31)
[[3]](diffhunk://#diff-cfd403ab6f3efc89188acaf993d85e6f792108d1740c7e7149eb05efb73d918dR1-R14)
- User-related operations (`auth_get_or_create_user_response`,
`auth_update_email_response`).
[[1]](diffhunk://#diff-49e65ab1eb6af4d0163a6c54ed10be621ce7336b2ab5d47d47679bfaefdb7059R1-R5)
[[2]](diffhunk://#diff-ac1216f96878bd4356454c317473654d5d5c7c180125663b80b0b45aa5ab52cbR1-R3)
- Credit-related operations (`credits_get_balance_response`,
`credits_get_auto_top_up_response`, `credits_top_up_request_response`).
[[1]](diffhunk://#diff-189488f8da5be74d80ac3fb7f84f1039a408573184293e9ba2e321d535c57cddR1-R3)
[[2]](diffhunk://#diff-ba3c4a6853793cbed24030cdccedf966d71913451ef8eb4b2c4f426ef18ed87aR1-R4)
[[3]](diffhunk://#diff-43d7daa0c82070a9b6aee88a774add8e87533e630bbccbac5a838b7a7ae56a75R1-R3)
- Graph execution and deletion (`blocks_execute_response`,
`graphs_delete_response`).
[[1]](diffhunk://#diff-a2ade7d646ad85a2801e7ff39799a925a612548a1cdd0ed99b44dd870d1465b5R1-R12)
[[2]](diffhunk://#diff-c0d1cd0a8499ee175ce3007c3a87ba5f3235ce02d38ce837560b36a44fdc4a22R1-R3)##
Summary
- add pytest-snapshot to backend dev requirements
- snapshot server route response JSONs
- mention how to update stored snapshots

## Testing
- `poetry run format`
- `poetry run test` 

### Checklist 📋

#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
  <!-- Put your test plan here: -->
  - [x] run poetry run test

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Nicholas Tindle <nicholas.tindle@agpt.co>
 2025-06-06 20:36:00 +00:00