We need stricter URL validation for the hostname we can request in the
block code.
### Changes 🏗️
* Canonicalization: Ensures \ are converted to /, adds http:// if
missing, and normalizes the input URL.
* Scheme Check: Only http or https are allowed.
* Hostname Validation:
- Ensures a hostname exists.
- Converts it to an IDNA ASCII form to prevent Unicode spoofing.
- Verifies that the hostname matches a safe DNS pattern.
* Trusted Origins Check: Allows certain hostnames explicitly if needed.
* IP Resolution and Blocking:
- Resolves the hostname to its IP addresses.
- Checks against a list of private/reserved IP networks to prevent SSRF
to internal services.
### Checklist 📋
#### For code changes:
- [ ] I have clearly listed my changes in the PR description
- [ ] I have made a test plan
- [ ] I have tested my changes according to the test plan:
<!-- Put your test plan here: -->
- [ ] ...
<details>
<summary>Example test plan</summary>
- [ ] Create from scratch and execute an agent with at least 3 blocks
- [ ] Import an agent from file upload, and confirm it executes
correctly
- [ ] Upload agent to marketplace
- [ ] Import an agent from marketplace and confirm it executes correctly
- [ ] Edit an agent from monitor, and confirm it executes correctly
</details>
#### For configuration changes:
- [ ] `.env.example` is updated or already compatible with my changes
- [ ] `docker-compose.yml` is updated or already compatible with my
changes
- [ ] I have included a list of my configuration changes in the PR
description (under **Changes**)
<details>
<summary>Examples of configuration changes</summary>
- Changing ports
- Adding new services that need to communicate with each other
- Secrets or environment variable changes
- New or infrastructure changes such as databases
</details>
- Resolves#8743
- Follow-up to #8358
### Demo
https://github.com/user-attachments/assets/f983dfa2-2dc2-4ab0-8373-e768ba17e6f7
### Changes 🏗️
- feat(frontend): Add webhook status indicator on `CustomNode`
- Add `webhookId` to frontend node data model
- fix(backend): Fix webhook ping endpoint
- Remove `provider` path parameter
- Fix return values and error handling
- Fix `WebhooksManager.trigger_ping(..)`
- Add `credentials` parameter
- Fix usage of credentials
- Fix `.data.integrations.wait_for_webhook_event(..)`
- Add `AsyncRedisEventBus.wait_for_event(..)`
- feat(frontend): Add `BackendAPIProvider` + `useBackendAPI`
- feat(frontend): Improve layout of node header
Before:
After:
- refactor(backend): Clean up `.data.integrations`
- refactor(backend): Fix naming in `.data.queue` for understandability
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
<!-- Put your test plan here: -->
- [x] Add webhook block, save -> gray indicator
- [x] Add necessary info to webhook block, save -> green indicator
- [x] Remove necessary info, save -> gray indicator
---------
Co-authored-by: Nicholas Tindle <nicholas.tindle@agpt.co>
<!-- Clearly explain the need for these changes: -->
I want to be able to have agents 3d print things and deliver them to my
house!
### Changes 🏗️
<!-- Concisely describe all of the changes made in this pull request:
-->
- Adds slant3d as a provider
- Adds slant3d order webhook (disabled on the cloud by default due to
how it notifies users)
- Adds several blocks to order from slant3d
- Diables Get Orders (for the same reason as webhook)
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan
<details>
<summary>Test Plan</summary>
- [ ] Add filament block and fill API key
- [ ] Run filament block
- [ ] Add slice block and use this value:
https://files.printables.com/media/prints/1081287/stls/8176524_a9edde2d-68c1-41de-a207-b584fcf42f30_f9127d5b-39ed-4ef8-b59f-d3a0bc874373/rod-holder.stl
- [ ] Run slice block
- [ ] Add estimate blocks (print and shipping) and use your address, and
the above file
- [ ] select petg and count 1
- [ ] run the blocks
- [ ] Create an order using same information
- [ ] Run the block and note the order number
- [ ] Delete the create order block so you don't keep ordering stuff
- [ ] Run get orders block
- [ ] Check your order exists
- [ ] Run the cancel order block with the order id
- [ ] run the get orders block and check the order no longer exists
</details>
---------
Co-authored-by: Reinier van der Leer <pwuts@agpt.co>
- resolve#8739
I don't think so that this is a frontend issue [might be wrong] ,
because if we are not classifying that a particular input is `advanced =
true/false`. Then we automatically get `advanced = True`.
<img width="1142" alt="Screenshot 2024-11-27 at 10 36 59 AM"
src="https://github.com/user-attachments/assets/e8d9c037-5b8b-45b2-b40b-8390bc63de99">
- Resolves#8859
- Follow-up to #8751
### Changes
- Add `autogpt_libs` to the backend CI path filter
- Add `ruff format` step for `autogpt_libs` to `linter.py` and
`pre-commit` config
- Run `poetry run format` with the new setup
<!-- Clearly explain the need for these changes: -->
On windows this file load kept crashing stuff on startup so I specified
the encoding
### Changes 🏗️
<!-- Concisely describe all of the changes made in this pull request:
-->
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] Run the app!
Replace Dict with dict
### Changes 🏗️
Replace Dict with dict
### Checklist 📋
#### For code changes:
- [ ] I have clearly listed my changes in the PR description
- [ ] I have made a test plan
- [ ] I have tested my changes according to the test plan:
<!-- Put your test plan here: -->
- [ ] ...
<details>
<summary>Example test plan</summary>
- [ ] Create from scratch and execute an agent with at least 3 blocks
- [ ] Import an agent from file upload, and confirm it executes
correctly
- [ ] Upload agent to marketplace
- [ ] Import an agent from marketplace and confirm it executes correctly
- [ ] Edit an agent from monitor, and confirm it executes correctly
</details>
#### For configuration changes:
- [ ] `.env.example` is updated or already compatible with my changes
- [ ] `docker-compose.yml` is updated or already compatible with my
changes
- [ ] I have included a list of my configuration changes in the PR
description (under **Changes**)
<details>
<summary>Examples of configuration changes</summary>
- Changing ports
- Adding new services that need to communicate with each other
- Secrets or environment variable changes
- New or infrastructure changes such as databases
</details>
linter.py, only applies in the `backend` module, not `autogpt_libs`.
The scope of this PR is to clear this out.
### Changes 🏗️
* Add a linting scope to both the `backend` & `autogpt_libs` modules,
and apply the linter.
### Checklist 📋
#### For code changes:
- [ ] I have clearly listed my changes in the PR description
- [ ] I have made a test plan
- [ ] I have tested my changes according to the test plan:
<!-- Put your test plan here: -->
- [ ] ...
<details>
<summary>Example test plan</summary>
- [ ] Create from scratch and execute an agent with at least 3 blocks
- [ ] Import an agent from file upload, and confirm it executes
correctly
- [ ] Upload agent to marketplace
- [ ] Import an agent from marketplace and confirm it executes correctly
- [ ] Edit an agent from monitor, and confirm it executes correctly
</details>
#### For configuration changes:
- [ ] `.env.example` is updated or already compatible with my changes
- [ ] `docker-compose.yml` is updated or already compatible with my
changes
- [ ] I have included a list of my configuration changes in the PR
description (under **Changes**)
<details>
<summary>Examples of configuration changes</summary>
- Changing ports
- Adding new services that need to communicate with each other
- Secrets or environment variable changes
- New or infrastructure changes such as databases
</details>
---------
Co-authored-by: Reinier van der Leer <pwuts@agpt.co>
When calculating the next month, we are not rolling the month number
causing an error on credits.
### Changes 🏗️
Add modulo while calculating next month.
### Checklist 📋
#### For code changes:
- [ ] I have clearly listed my changes in the PR description
- [ ] I have made a test plan
- [ ] I have tested my changes according to the test plan:
<!-- Put your test plan here: -->
- [ ] ...
<details>
<summary>Example test plan</summary>
- [ ] Create from scratch and execute an agent with at least 3 blocks
- [ ] Import an agent from file upload, and confirm it executes
correctly
- [ ] Upload agent to marketplace
- [ ] Import an agent from marketplace and confirm it executes correctly
- [ ] Edit an agent from monitor, and confirm it executes correctly
</details>
#### For configuration changes:
- [ ] `.env.example` is updated or already compatible with my changes
- [ ] `docker-compose.yml` is updated or already compatible with my
changes
- [ ] I have included a list of my configuration changes in the PR
description (under **Changes**)
<details>
<summary>Examples of configuration changes</summary>
- Changing ports
- Adding new services that need to communicate with each other
- Secrets or environment variable changes
- New or infrastructure changes such as databases
</details>
- Move `autogpt_libs.supabase_integration_credentials_store` into
`backend`
- `.store` -> `backend.integrations.credentials_store`
- `.types` -> added to `backend.data.model`
- Rename `SupabaseIntegrationCredentialsStore` to
`IntegrationCredentialsStore`
We wanted to get a few security things in quickly in #8403 and had to
make some compromises to do so. This picks those up and fixes them.
- Resolves#8540
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
<!-- Put your test plan here: -->
---------
Co-authored-by: Reinier van der Leer <pwuts@agpt.co>
Co-authored-by: Aarushi <50577581+aarushik93@users.noreply.github.com>
This fix is triggered by an error observed on db connection failure on
SupaBase:
```
2024-11-28 07:45:24,724 INFO [DatabaseManager] Starting...
2024-11-28 07:45:24,726 INFO [PID-18|DatabaseManager|Prisma-7f32369c-6432-4edb-8e71-ef820332b9e4] Acquiring connection started...
2024-11-28 07:45:24,726 INFO [PID-18|DatabaseManager|Prisma-7f32369c-6432-4edb-8e71-ef820332b9e4] Acquiring connection completed successfully.
{"is_panic":false,"message":"Can't reach database server at `...pooler.supabase.com:5432`\n\nPlease make sure your database server is running at `....pooler.supabase.com:5432`.","meta":{"database_host":"...pooler.supabase.com","database_port":5432},"error_code":"P1001"}
2024-11-28 07:45:35,153 INFO [PID-18|DatabaseManager|Prisma-7f32369c-6432-4edb-8e71-ef820332b9e4] Acquiring connection failed: Could not connect to the query engine. Retrying now...
2024-11-28 07:45:36,155 INFO [PID-18|DatabaseManager|Redis-e14a33de-2d81-4536-b48b-a8aa4b1f4766] Acquiring connection started...
2024-11-28 07:45:36,181 INFO [PID-18|DatabaseManager|Redis-e14a33de-2d81-4536-b48b-a8aa4b1f4766] Acquiring connection completed successfully.
2024-11-28 07:45:36,183 INFO [PID-18|DatabaseManager|Pyro-2722cd29-4dbd-4cf9-882f-73842658599d] Starting Pyro Service started...
2024-11-28 07:45:36,189 INFO [DatabaseManager] Connected to Pyro; URI = PYRO:DatabaseManager@0.0.0.0:8005
2024-11-28 07:46:28,241 ERROR Error in get_user_integrations: All connection attempts failed
```
Where even
```
2024-11-28 07:45:35,153 INFO [PID-18|DatabaseManager|Prisma-7f32369c-6432-4edb-8e71-ef820332b9e4] Acquiring connection failed: Could not connect to the query engine. Retrying now...
```
is present, the Redis connection is still proceeding without waiting for
the retry to complete. This was likely caused by Tenacity not fully
awaiting the DB connection acquisition command.
### Changes 🏗️
* Add special handling for the async function to explicitly await the
function execution result on each retry.
* Explicitly raise exceptions on `db.connect()` if the db is not
connected even after `prisma.connect()` command.
### Checklist 📋
#### For code changes:
- [ ] I have clearly listed my changes in the PR description
- [ ] I have made a test plan
- [ ] I have tested my changes according to the test plan:
<!-- Put your test plan here: -->
- [ ] ...
<details>
<summary>Example test plan</summary>
- [ ] Create from scratch and execute an agent with at least 3 blocks
- [ ] Import an agent from file upload, and confirm it executes
correctly
- [ ] Upload agent to marketplace
- [ ] Import an agent from marketplace and confirm it executes correctly
- [ ] Edit an agent from monitor, and confirm it executes correctly
</details>
#### For configuration changes:
- [ ] `.env.example` is updated or already compatible with my changes
- [ ] `docker-compose.yml` is updated or already compatible with my
changes
- [ ] I have included a list of my configuration changes in the PR
description (under **Changes**)
<details>
<summary>Examples of configuration changes</summary>
- Changing ports
- Adding new services that need to communicate with each other
- Secrets or environment variable changes
- New or infrastructure changes such as databases
</details>
This PR adds the first few Hubspot blocks so we can create _real_ sales
and marketing agents.
### Changes 🏗️
Added Hubspot blocks;
- Aded auth for hubspot
- Added Company block
- Added Contact block
- Added Engagement block
### Checklist 📋
#### For code changes:
- [ ] I have clearly listed my changes in the PR description
- [ ] I have made a test plan
- [ ] I have tested my changes according to the test plan:
<!-- Put your test plan here: -->
- [ ] ...
<details>
<summary>Example test plan</summary>
- [ ] Create from scratch and execute an agent with at least 3 blocks
- [ ] Import an agent from file upload, and confirm it executes
correctly
- [ ] Upload agent to marketplace
- [ ] Import an agent from marketplace and confirm it executes correctly
- [ ] Edit an agent from monitor, and confirm it executes correctly
</details>
#### For configuration changes:
- [ ] `.env.example` is updated or already compatible with my changes
- [ ] `docker-compose.yml` is updated or already compatible with my
changes
- [ ] I have included a list of my configuration changes in the PR
description (under **Changes**)
<details>
<summary>Examples of configuration changes</summary>
- Changing ports
- Adding new services that need to communicate with each other
- Secrets or environment variable changes
- New or infrastructure changes such as databases
</details>
We've started enabling cost based on the *partial value* of the
`credentials` field. And this logic has never been supported.
### Changes 🏗️
* Add partial object matching on the input data filter for evaluating
the block cost.
* Add missing credentials for `ExtractWebsiteContentBlock`
* Removed fallback cost on LLM blocks.
### Checklist 📋
#### For code changes:
- [ ] I have clearly listed my changes in the PR description
- [ ] I have made a test plan
- [ ] I have tested my changes according to the test plan:
<!-- Put your test plan here: -->
- [ ] ...
<details>
<summary>Example test plan</summary>
- [ ] Create from scratch and execute an agent with at least 3 blocks
- [ ] Import an agent from file upload, and confirm it executes
correctly
- [ ] Upload agent to marketplace
- [ ] Import an agent from marketplace and confirm it executes correctly
- [ ] Edit an agent from monitor, and confirm it executes correctly
</details>
#### For configuration changes:
- [ ] `.env.example` is updated or already compatible with my changes
- [ ] `docker-compose.yml` is updated or already compatible with my
changes
- [ ] I have included a list of my configuration changes in the PR
description (under **Changes**)
<details>
<summary>Examples of configuration changes</summary>
- Changing ports
- Adding new services that need to communicate with each other
- Secrets or environment variable changes
- New or infrastructure changes such as databases
</details>
- Add `/integrations/credentials` endpoint which lists all credentials for the authenticated user
- Amend credential fetching logic in front end to fetch all at once instead of per provider
- Resolves#8770
- Resolves (hopefully) #8613
- feat(blocks): Add GitHub Pull Request Trigger block
## feat(platform): Add support for Webhook-triggered blocks
- ⚠️ Add `PLATFORM_BASE_URL` setting
- Add webhook config option and `BlockType.WEBHOOK` to `Block`
- Add check to `Block.__init__` to enforce type and shape of webhook event filter
- Add check to `Block.__init__` to enforce `payload` input on webhook blocks
- Add check to `Block.__init__` to disable webhook blocks if `PLATFORM_BASE_URL` is not set
- Add `Webhook` model + CRUD functions in `backend.data.integrations` to represent webhooks created by our system
- Add `IntegrationWebhook` to DB schema + reference `AgentGraphNode.webhook_id`
- Add `set_node_webhook(..)` in `backend.data.graph`
- Add webhook-related endpoints:
- `POST /integrations/{provider}/webhooks/{webhook_id}/ingress` endpoint, to receive webhook payloads, and for all associated nodes create graph executions
- Add `Node.is_triggered_by_event_type(..)` helper method
- `POST /integrations/{provider}/webhooks/{webhook_id}/ping` endpoint, to allow testing a webhook
- Add `WebhookEvent` + pub/sub functions in `backend.data.integrations`
- Add `backend.integrations.webhooks` module, including:
- `graph_lifecycle_hooks`, e.g. `on_graph_activate(..)`, to handle corresponding webhook creation etc.
- Add calls to these hooks in the graph create/update endpoints
- `BaseWebhooksManager` + `GithubWebhooksManager` to handle creating + registering, removing + deregistering, and retrieving existing webhooks, and validating incoming payloads
## Other improvements
- fix(blocks): Allow having an input and output pin with the same name
- fix(blocks): Add tooltip with description in places where block inputs are rendered without `NodeHandle`
- feat(blocks): Allow hiding inputs (e.g. `payload`) with `SchemaField(hidden=True)`
- fix(frontend): Fix `MultiSelector` component styling
- feat(frontend): Add `AlertDialog` UI component
- feat(frontend): Add `NodeMultiSelectInput` component
- feat(backend/data): Add `NodeModel` with `graph_id`, `graph_version`; `GraphModel` with `user_id`
- Add `make_graph_model(..)` helper function in `backend.data.graph`
- refactor(backend/data): Make `RedisEventQueue` generic and move to `backend.data.execution`
- refactor(frontend): Deduplicate & clean up code for different block types in `generateInputHandles(..)` in `CustomNode`
- dx(backend): Add `MissingConfigError`, `NeedConfirmation` exception
---------
Co-authored-by: Zamil Majdy <zamil.majdy@agpt.co>
* feat: Add Open Router integration credentials
- Added support for Open Router integration credentials in the Supabase integration credentials store.
- Updated the LLM provider field to include "open_router" as a valid provider option.
- Added Open Router API key field to the backend settings.
- Updated the profile page to display the Open Router integration credentials.
- Updated the credentials input and provider components to include Open Router as a provider option.
- Updated the autogpt-server-api types to include "open_router" as a provider name.
- Updated the LLM provider schema to include "open_router" as a valid provider name.
- Added GEMINI_FLASH_1_5_8B as the first Open Router LLM
* Add type ignore to new llm prompt to match the rest of them.
* Update LlmModel with a selection of new OpenRouter models
* format
- Remove `secrets_dir` and other references to `get_secrets_path()`
- Remove unused `get_config_path()`
Follow-up to #8521, which removed the `secrets` dir but not the references to it.
In #8524, the "llm" credentials provider was replaced. There are still entries with `"provider": "llm"` in the system though, and those break if not migrated.
- SQL migration to fix the obvious ones where we know the provider from `credentials.id`
- Non-SQL migration to fix the rest
