Commit Graph

182 Commits

Author SHA1 Message Date
SwiftyOS
8d0bbc5ffe adding template for refactored rest service app 2024-09-17 11:51:46 +02:00
SwiftyOS
3d4aca9fcc Splitting rest services into routes 2024-09-16 16:56:09 +02:00
Zamil Majdy
c1f301ab8b feat(rnd): Add initial credit accounting system for block execution (#8047)
### Background

We need a way to set an execution quota per user for each block execution.

### Changes 🏗️

* Introduced a `UserBlockCredit`, a transaction table tracking the block usage along with it cost/quota.
* The tracking is toggled by `ENABLE_CREDIT` config, default = false.
* Introduced  `BLOCK_COSTS` | `GET /blocks/costs` as a source of information for the cost on each block depending on the input configuration.

Improvements:
* Refactor logging in manager.py to always print a prefix and pass the metadata.
* Make executionStatus on AgentNodeExecution prisma enum. And add executionStatus on AgentGraphExecution.
* Use executionStatus from AgentGraphExecution to improve waiting logic on test_manager.py.
2024-09-14 23:47:28 +07:00
Swifty
6fde030c37 feat(rnd): Reduce container size remove dep with forge and autogpt (#8040)
* Remove forge and autogpt

* update lock files

* Update build process to reduce image size

* Reduced built image size

* fixed docker compose watch

* Updated logging

* updated env.example

* formatting

* linting issue

* linting not working in github actions..

* trying to get around github action linting issue

* updated version

* sleep for prisma issues

* add exp backoff on connection issues

* updated config based on review comments

* Sorting alphabetical

* updated default config

* updated depends checks

* fixed missing prisma binaries

* remove dead layer

* remove try

* remove dead layer
2024-09-12 13:03:37 +02:00
Aarushi
bf1e01d423 tweak(rnd) Add prefix in logs (#8001)
* add prefix

* fix typos

* fix conflicts
2024-09-12 11:48:47 +01:00
Zamil Majdy
52c731abd6 fix(rnd): Fix decorator function type hint (#8043) 2024-09-12 05:35:33 +07:00
Swifty
6c001bd595 Create Input Node Custom UI Node (#8016) 2024-09-11 14:14:03 +02:00
Aarushi
1d9b01fc77 tweak(rnd): Use docker compose not docker-compose (#8034)
* use docker compose not docker-compose

* linting
2024-09-11 10:20:31 +01:00
Swifty
e992cdf8c2 Fixing docker setup for local testing (#8026)
* Fixing docker setup

* Updated docker compose setup

* update helm charts

* Corrected agent server host name
2024-09-10 15:46:22 +02:00
Aarushi
0b919522ae feat(rnd): Split Execution Manager (#8008)
* split execution manager and removed ns and use direct uri with k8s and docker specific dns

* formating

* split execution manager

* refactor(builder): Fix linting warning and errors (#8021)

* Fix lint errors

* Fix dependency loop

* address feedback

* docker compose

* remove ns entirely

* remove yarn lock changes

* update readme

* remove ref

* dockerfile and log

* update log

* debug

* rename to executor

* remove execution from rest

* exec.py

* linting

* udpate tests to use config

* fix test

---------

Co-authored-by: Krzysztof Czerwinski <34861343+kcze@users.noreply.github.com>
2024-09-10 10:05:31 +01:00
Reinier van der Leer
a60ed21404 feat(server): Add OAuth flow endpoints for integrations (#7872)
- feat(server): Initial draft of OAuth init and exchange endpoints
  - Add `supabase` dependency
  - Add Supabase credentials to `Secrets`
  - Add `get_supabase` utility to `.server.utils`
  - Add `.server.integrations` API segment with initial implementations for OAuth init and exchange endpoints
- Move integration OAuth handlers to `autogpt_server.integrations.oauth`
- Change constructor of `SupabaseIntegrationCredentialsStore` to take a Supabase client
- Fix type issues in `GoogleOAuthHandler`
2024-09-09 17:21:56 +02:00
Nicholas Tindle
60669903a0 fix: couple block categories improved (#8017) 2024-09-06 22:47:40 -05:00
Zamil Majdy
b1b31390a4 feat(rnd): Add creation & update time for AgentGraphExecutionSchedule, AgentGraphExecution, and AgentGraph (#8015) 2024-09-06 21:29:53 +00:00
Aarushi
126d070396 tweak(rnd,redis) Make redis logging more clear (#8014)
make redis logging more clear
2024-09-06 15:23:33 +00:00
Reinier van der Leer
090f22b05c fix(server): Improve logging consistency (#8012)
- Make process/service startup/shutdown messages consistent
- Configure `uvicorn` to use our logging config instead of its own
- Replace `print(..)` statements in ws_api.py with log statements
- Improve log statements in ws_api.py
2024-09-06 17:05:30 +02:00
Reinier van der Leer
1b9adf5434 fix(server): Always JSON-encode AgentNodeExecutionInputOutput data (#8010)
- Handle JSON-encoding inside `.data.execution.upsert_execution_output(..)` to ensure it is always encoded the same
- Amend `.executor.manager.execute_node(..)` to pass unencoded data into `upsert_execution_output(..)`
2024-09-06 16:58:04 +02:00
Reinier van der Leer
3bd8040d6a feat(server): Clean up resources when spinning down services/processes (#7938)
- Add SIGTERM handler and `cleanup()` hook to `AppProcess`
- Implement `cleanup()` on `AppService` to close DB and Redis connections
- Implement `cleanup()` on `ExecutionManager` to shut down worker pool
- Add `atexit` and SIGTERM handlers to node executor to close DB connection and shut down node workers
- Improve logging in `.executor.manager`
- Fix shutdown order of `.util.test:SpinTestServer`
2024-09-06 16:50:59 +02:00
Aarushi
b12dba13f4 docs(readme) Update readme to include non docker steps (#8013)
update readme
2024-09-06 15:33:17 +01:00
Swifty
2cae9ba8da feat(server): Updated Output block (#7997)
* Add Block UI Types and StickyNote Block

* Renamed StickyNote to Note

* Add comment

* Updated Input Block

* rename default_values to placeholder_values

* Update sample graph

* Update input block output to match previous change

* fixing test

* Updated the output block

* re-remove old output block
2024-09-06 15:56:41 +02:00
Swifty
fd54ad8666 update(server): update input nodes (#7996)
* Add Block UI Types and StickyNote Block

* Updated Input Block
2024-09-06 13:27:37 +01:00
Zamil Majdy
e645cc4b33 tweak(rnd): Remove duplicated call 2024-09-05 17:20:55 -05:00
Zamil Majdy
010a8ffaaf tweak(rnd): Stop publishing custom Sentry metrics (#8000) 2024-09-05 21:51:00 +00:00
Zamil Majdy
2df325d033 fix(rnd): Fix prisma connection acquisition intermittent error on linux (#7999) 2024-09-06 03:22:25 +07:00
Zamil Majdy
e5eb42d84a tweak(rnd): Post infra change cleanup - fix process creation lifecycle (#7981) 2024-09-06 01:41:24 +07:00
Reinier van der Leer
8fd22bcfd7 feat(server, builder): Implement "STOP" button for graph runs (#7892)
- feat(builder): Add "Stop Run" buttons to monitor and builder
  - Implement additional state management in `useAgentGraph` hook
    - Add "stop" request mechanism
    - Implement execution status tracking using WebSockets
    - Add `isSaving`, `isRunning`, `isStopping` outputs
    - Add `requestStopRun` method
      - Rename `requestSaveRun` to `requestSaveAndRun` for clarity
  - Add needed functionality for the above to `AutoGPTServerAPI` client
    - Add `stopGraphExecution` method
    - Add support for multiple handlers per WebSocket method
    - Fix parsing of timestamps in `execution_event` WebSocket messages
  - Add `IconSquare` from Lucide to `@/components/ui/icons`

- feat(server): Add `POST /graphs/{graph_id}/executions/{graph_exec_id}/stop` route
  - Add `stop_graph_run` method to `AgentServer`

- feat(server): Add `cancel_execution` method to `ExecutionManager`
  - Replace node executor `ProcessPoolExecutor` by `multiprocessing.Pool` (which has a `terminate()` method)
    - Remove now unnecessary `Executor.wait_future(..)` method
  - Add `get_graph_execution(..)` in `.data.execution`

- fix(server): Reduce number of node executors to 5 per graph executor
  This is necessary because `multiprocessing.Pool` spawns its workers on init, instead of based on demand like `ProcessPoolExecutor` does

- dx(server): Improve debug logging in `ExecutionManager`
- ci(server): Add debug logging mode to CI Pytest step

### Other improvements
Server:
- Improve output type of `ExecutionManager.add_execution(..)`
- Renamed a few things in `.server.rest_api` for consistency

Front end:
- Improved typing in `AutoGPTServerAPI` client
2024-09-05 14:42:28 +02:00
Zamil Majdy
70fab8711a fix(rnd): avoid duplicating name on input/output pin for blocks (#7979) 2024-09-05 10:54:02 +00:00
SwiftyOS
bc7d2f0f37 added test password as default for redis queue 2024-09-05 11:41:53 +02:00
Zamil Majdy
54694709bb tweak(rnd): Hide non required field on Blocks (#7977)
Add prompt_values for LLM block, and make it non-advanced.
Make all-field, advanced by default.
cleanup .env.example and .env.template
2024-09-05 05:00:14 +07:00
Zamil Majdy
b4b5a09b6b fix(rnd): Dockerfile Avoid full rebuild on each file change (#7971)
Co-authored-by: Aarushi <50577581+aarushik93@users.noreply.github.com>
2024-09-04 17:30:13 +00:00
Reinier van der Leer
82239dd129 fix(server): Fix node input concurrency deadlock (#7936)
In `autogpt_server.util.lock:KeyedMutex`:
- track number of pending requests for each lock
- only remove a lock from `self.locks` when the number of pending lock requests hits 0
2024-09-04 18:13:35 +02:00
Aarushi
5000aa7ee0 tweak(rnd,docker) Remove SQLite (#7966)
* move migrations, update networking and dockignore

* update docs

* remove sqlite from ci

* remove schema linting checks

* fix formatting

* remove schema linting

* add test script

* formatting and linting

* stop pg not down

* seperate test db

* diff port

* remove duplicate
2024-09-04 10:18:57 +01:00
Swifty
80df44a978 feat(server): Add endpoint to calculate required graph inputs (#7965) 2024-09-04 09:45:45 +02:00
Zamil Majdy
c2a79d2f10 feat(rnd): Add Node & Graph level execution stats instrumentation (#7957)
### **User description**
### Background

The scope of this change is collecting the required information that will be needed for the execution analytics.

### Changes 🏗️

* Add sentry integration.
* Refactor logging_metadata on manager.py.
* Collect graph-level & node-level instrumentation.
* Introduced `stats` column for `AgentNodeExecution` & `AgentGraphExecution`.
2024-09-04 02:45:19 +07:00
Krzysztof Czerwinski
09951fed4b feat(builder, server): Add advanced block inputs (#7934)
- Add `advanced` to `SchemaField` and pass it to `json_extra`
- Add `advanced` to `BlockIOSubSchemaMeta` type
- Update `CustomNode`, so that:
  - non-required advanced inputs are hidden
  - non-advanced and required inputs are always shown
2024-09-03 12:01:42 +01:00
Aarushi
6204d82d84 feat(server): Integrate forge.logging (#7915)
* feat(server): Integrate `forge.logging`

- Add `configure_logging()` in `.util.logging` - a wrapper for `forge.logging.configure_logging()` with project-specific extras
- Call `configure_logging()` in `.app.main()`, and in child process initializers (e.g. `AppProcess.execute_run_command(..)`, `ExecutorManager.on_graph_executor_start()`)
- Change some `logger.warning` statements to `logger.info` where appropriate

* fix warnings to info

* fix(rnd): Fix broken test and Input/Output block field renaming

* Rename

* fix(rnd): Fix flaky CI

* feat(server): Add OAuth handlers for GitHub, Notion, Google & amend store data structure (#7868)

- Add `BaseOAuthHandler` + 3 initial implementations
  - Add `GitHubOAuthHandler`
  - Add `NotionOAuthHandler`
  - Add `GoogleOAuthHandler`
- Amend `OAuth2Credentials` type
  - Add `metadata` attribute
  - Make `access_token_expires_at`, `refresh_token`, `refresh_token_expires_at` optional

* extend GCP Logger

* update manager & add flag

* linting

* use default logger behaviour

* update messages

* update another message

---------

Co-authored-by: Reinier van der Leer <pwuts@agpt.co>
Co-authored-by: Zamil Majdy <zamil.majdy@agpt.co>
2024-09-03 11:56:21 +01:00
Reinier van der Leer
71de1a6a5e fix(server): Fix type checking and propagation issues (#7941)
- fix type propagation by `AppService.run_and_wait(..)`
- fix type propagation by `@expose` and add note
- fix type propagation by `wait(..)` in `.executor.manager.execute_node(..)`
- fix type propagation by `wait(..)` in `.executor.manager._enqueue_next_nodes(..)`
- remove unnecessary null checks for `.data.graph.get_node(..)`
- fix type issue in `ExecutionScheduler`
- reduce use of `# type: ignore` in `.data.execution`
- reduce usage of `# type: ignore` in `.executor.manager`
- reduce usage of `# type: ignore` in `.server`
- reduce usage of `# type: ignore` in cli.py
- update `pyright` to v1.1.378
2024-09-02 14:13:56 +00:00
Aarushi
cfa0b6610c feat(rnd,infra): Pull out websockets away from server api (#7899)
* standalone websocket server

* add websocket url

* wip: talk to ws directly

* rename to api server

* dockerfile and queue

* fix paths

* update poetry lock

* helm charts for websockets

* create seperate deployments for websockets and rest server with redis queue for async comms

* delete duplicate queue

* add depends in ws_api

* singleton for conn manager

* update from review

* fix CI

* address feedback

* update readme

* update docker file and add migration step in readm

* ad watch

* add step to copy example env file

* put connect back in
2024-09-02 10:32:51 +02:00
Zamil Majdy
f6d09c74f5 feat(rnd): Rename Blocks (#7925)
### Background

Standardize block names with a focus of making them intuitive to a non-technical person.

### Changes 🏗️

Replace these names:
* TextLlmCallBlock -> AITextGeneratorBlock
* ObjectLlmCallBlock -> AIStructuredResponseBlock
* AdvancedLlmCallBlock -> AIConversationBlock
* CreateTalkingAvatarClipBlock -> CreateTalkingAvatarVideoBlock
* DiscordReaderBlock -> ReadDiscordMessagesBlock
* DiscordMessageSenderBlock -> SendDiscordMessageBlock
* YouTubeTranscriberBlock -> TranscribeYouTubeVideoBlock
* CreateMediumPostBlock -> PublishToMediumBlock
* ForEachBlock -> ListIteratorBlock
* MathsBlock -> CalculatorBlock
* CounterBlock -> CountItemsBlock
* ValueBlock -> StoreValueBlock
* PrintingBlock -> PrintToConsoleBlock
* DictionaryAddEntryBlock -> AddToDictionaryBlock
* ListAddEntryBlock -> AddToListBlock
* ObjectLookupBlock -> FindInDictionaryBlock
* RedditGetPostsBlock -> GetRedditPostsBlock
* RedditPostCommentBlock -> PostRedditCommentBlock
* WikipediaSummaryBlock -> GetWikipediaSummaryBlock
* WebSearchBlock -> SearchTheWebBlock
* WebScraperBlock -> ExtractWebsiteContentBlock
* GetOpenWeatherMapBlock -> GetWeatherInformationBlock
* HttpRequestBlock -> SendWebRequestBlock
* CurrentTimeBlock -> GetCurrentTimeBlock
* CurrentDateBlock -> GetCurrentDateBlock
* CurrentDateAndTimeBlock -> GetCurrentDateAndTimeBlock
* TimerBlock -> CountdownTimerBlock
* RSSReaderBlock -> ReadRSSFeedBlock
* TextMatcherBlock -> MatchTextPatternBlock
* TextParserBlock -> ExtractTextInformationBlock
* TextFormatterBlock -> FillTextTemplateBlock
* TextCombinerBlock -> CombineTextsBlock
2024-08-30 21:36:42 +07:00
Reinier van der Leer
087d3a3760 test(server): Add type to server fixture usages 2024-08-29 15:42:15 +02:00
Zamil Majdy
5da58aa284 fix(rnd): Fix jumping caret problem on builder input text field (#7917)
Issue 1:
Input text field cursor keeps moving to the end of the text.
Try to type "Hello World!" into the input text. Then try to type "some string" in the middle of the "Hello" and "World".

Issue 2:
History should only tracks on the input box onBlur/onLeave
Try to type a "longcharacters" and try to undo it, the undo is removing 1 character at a time, polluting the history, and make the undo pretty much unusable.

Issue 3:
KeyValue & ArrayInput is non-undoable.
Try to add key-value or add an entry to the list, it doesn't undo the value, but you need to click as many number of entries being added to make the undo work again
2024-08-29 13:12:10 +00:00
Reinier van der Leer
8f1c63a7ea feat(server): Add OAuth handlers for GitHub, Notion, Google & amend store data structure (#7868)
- Add `BaseOAuthHandler` + 3 initial implementations
  - Add `GitHubOAuthHandler`
  - Add `NotionOAuthHandler`
  - Add `GoogleOAuthHandler`
- Amend `OAuth2Credentials` type
  - Add `metadata` attribute
  - Make `access_token_expires_at`, `refresh_token`, `refresh_token_expires_at` optional
2024-08-29 09:11:37 +00:00
Zamil Majdy
6ec200f912 fix(rnd): Fix flaky CI 2024-08-28 15:59:51 -05:00
Zamil Majdy
b5db7f575e Rename 2024-08-28 15:56:02 -05:00
Zamil Majdy
98c909f99f fix(rnd): Fix broken test and Input/Output block field renaming 2024-08-28 15:54:48 -05:00
Zamil Majdy
ea6f37bf98 feat(rnd): Integrate Jinja2 into TextFormatter (#7891)
Add Jinja2 to TextFormatter while maintaining backward compatibility.
2024-08-27 07:17:57 +00:00
Aarushi
407cf858e7 fix(server): Ensure nameserver is started before other processes (#7788)
ensure nameserver is started before other processes
2024-08-24 20:54:11 +00:00
Zamil Majdy
f9b8b0a41a feat(rnd): Add dynamic input pin for input object construction (#7871)
### Background

Currently, there is no way to construct the output of nodes into a composite data structure (list/dict/object) using the builder UI.

The backend already supports this feature by connecting the output pin to the input pin using these format:
* <pin_name>_$_<list_index> for constructing list
* <pin_name>_#_<dict_key> for constructing dict
* <pin_name>_@_<field_name> for constructing object

The scope of this PR is implementing the UX for this in the builder UI.

### Changes 🏗️

<img width="765" alt="image" src="https://github.com/user-attachments/assets/8fc319a4-1350-410f-98cf-24f2aa2bc34b">

This allows you to add more pins in a key value & list input: `_$_` list constructor & `_#_` dict constructor.
2024-08-23 18:21:38 +00:00
Zamil Majdy
e59e138352 fix(rnd): Prevent boolean with no default value on AGPT-builder (#7884)
### Background

Boolean without default value is a UX problem. It's currently displayed as a toggle and it has no way to describe the `null` value.
So we need to prevent blocks from introducing a nullable boolean.

### Changes 🏗️

Add explicit check to prevent nullable boolean. Fix existing block field that has nullable boolean.
2024-08-23 18:03:21 +00:00
Aarushi
a95ee693dd fix(rnd): Add missing libs folder (#7887)
add missing libs folder
2024-08-23 16:42:24 +00:00
Aarushi
012bad72e8 feat(rnd,blocks): Add D-ID Block (#7798)
* talking head

* linting

* remove clip id, not needed

* add more descriptive name

* add min requirement to polling attempts and intervals

* add docs and link to docs

* remove extra space

* force new tab

* fix linting

* add did key to .env.template
2024-08-23 09:39:55 +01:00