Commit Graph

24 Commits

Author SHA1 Message Date
Reinier van der Leer
f107ff8cf0 Set up unified pre-commit + CI w/ linting + type checking & FIX EVERYTHING (#7171)
- **FIX ALL LINT/TYPE ERRORS IN AUTOGPT, FORGE, AND BENCHMARK**

### Linting
- Clean up linter configs for `autogpt`, `forge`, and `benchmark`
- Add type checking with Pyright
- Create unified pre-commit config
- Create unified linting and type checking CI workflow

### Testing
- Synchronize CI test setups for `autogpt`, `forge`, and `benchmark`
   - Add missing pytest-cov to benchmark dependencies
- Mark GCS tests as slow to speed up pre-commit test runs
- Repair `forge` test suite
  - Add `AgentDB.close()` method for test DB teardown in db_test.py
  - Use actual temporary dir instead of forge/test_workspace/
- Move left-behind dependencies for moved `forge`-code to from autogpt to forge

### Notable type changes
- Replace uses of `ChatModelProvider` by `MultiProvider`
- Removed unnecessary exports from various __init__.py
- Simplify `FileStorage.open_file` signature by removing `IOBase` from return type union
  - Implement `S3BinaryIOWrapper(BinaryIO)` type interposer for `S3FileStorage`

- Expand overloads of `GCSFileStorage.open_file` for improved typing of read and write modes

  Had to silence type checking for the extra overloads, because (I think) Pyright is reporting a false-positive:
  https://github.com/microsoft/pyright/issues/8007

- Change `count_tokens`, `get_tokenizer`, `count_message_tokens` methods on `ModelProvider`s from class methods to instance methods

- Move `CompletionModelFunction.schema` method -> helper function `format_function_def_for_openai` in `forge.llm.providers.openai`

- Rename `ModelProvider` -> `BaseModelProvider`
- Rename `ChatModelProvider` -> `BaseChatModelProvider`
- Add type `ChatModelProvider` which is a union of all subclasses of `BaseChatModelProvider`

### Removed rather than fixed
- Remove deprecated and broken autogpt/agbenchmark_config/benchmarks.py
- Various base classes and properties on base classes in `forge.llm.providers.schema` and `forge.models.providers`

### Fixes for other issues that came to light
- Clean up `forge.agent_protocol.api_router`, `forge.agent_protocol.database`, and `forge.agent.agent`

- Add fallback behavior to `ImageGeneratorComponent`
   - Remove test for deprecated failure behavior

- Fix `agbenchmark.challenges.builtin` challenge exclusion mechanism on Windows

- Fix `_tool_calls_compat_extract_calls` in `forge.llm.providers.openai`

- Add support for `any` (= no type specified) in `JSONSchema.typescript_type`
2024-05-28 05:04:21 +02:00
Krzysztof Czerwinski
4c325724ec refactor(autogpt, forge): Remove autogpts directory (#7163)
- Moved `autogpt` and `forge` to project root
- Removed `autogpts` directory
- Moved and renamed submodule `autogpts/autogpt/tests/vcr_cassettes` to `autogpt/tests/vcr_cassettes`
- When using CLI agents will be created in `agents` directory (instead of `autogpts`)
- Renamed relevant docs, code and config references from `autogpts/[forge|autogpt]` to `[forge|autogpt]` and from `*../../*` to `*../*`
- Updated `CODEOWNERS`, GitHub Actions and Docker `*.yml` configs
- Updated symbolic links in `docs`
2024-05-22 13:08:54 +01:00
Reinier van der Leer
d5ad719757 ci: Allow telemetry for non-push events, as long as it's on master
Also disable telemetry for AutoGPT's unit/integration tests.
2024-02-17 15:12:43 +01:00
Reinier van der Leer
1ca9b9fa93 ci: Fix setting/passing TELEMETRY_* environment variables 2024-02-17 14:26:03 +01:00
Reinier van der Leer
fa4bdef17c ci: Update actions to newest versions
- `actions/stale` -> `v9`
- `actions/cache` -> `v4`
- `actions/checkout` -> `v4`
- `actions/setup-node` -> `v4`
- `docker/login-action` -> `v3`
- `actions/setup-python` -> `v5`
- `codecov/codecov-action` -> `v4`
- `actions/upload-artifact` -> `v4`
- `subosito/flutter-action` -> `v2`
- `docker/build-push-action` -> `v5`
- `docker/setup-buildx-action` -> `v3`
2024-02-17 13:59:13 +01:00
Reinier van der Leer
6017eefb32 ci: Enable telemetry in CI runs on master 2024-02-14 12:03:54 +01:00
Reinier van der Leer
b106a61352 Clean up & fix GitHub workflows (#6313)
* ci: Mitigate security issues in autogpt-ci.yml

- Remove unnecessary pull_request_target paths and related variables and config
- Set permissions for contents to read only

* ci: Simplify steps in autogpt-ci.yml workflow using GitHub CLI

- Simplify step in 'autogpt-ci.yml' by using GitHub CLI instead of API for adding label and comment functionality
- Replace curl command with 'gh issue edit' to add "behaviour change" label to the pull request
- Replace gh api command with 'gh issue comment' to leave a comment about the changed behavior of AutoGPT in the pull request

* ci: Fix issues in workflows

- Move environment variable definition to top level in benchmark-ci.yml (because the other job also needs it)
- Removed invalid 'branches: [hackathon]' restriction in hackathon.yml workflow
- Removed redundant 'ref' and 'repository' fields in the 'checkout' step of both workflows.

* ci: Delete legacy benchmarks.yml workflow

* ci: Add triggers for CI workflows

- Add triggers to run CI workflows when they are edited.
- Update the paths for the CI workflows in the trigger configuration.

* fix: Fix benchmark lint error

- Removed unnecessary blank lines in report_types.py
- Fixed string quotes in challenge.py to maintain consistency

* fix: Update task description in password generator data.json

- Update task description in `data.json` file for the password generator challenge to clarify the input requirements and error handling.
- This change is made in an attempt to make the Benchmark CI pass.

* fix: Fix PasswordGenerator challenge in CI

- Fix the behavior of the reference password_generator.py to align with the task description
- Use default password length 8 instead of a random length in the generate_password function
- Retrieve the password length from the command line arguments if "--length" is provided, else set it to 8
2023-11-21 10:58:54 +01:00
Reinier van der Leer
16e266c65f Set up CI for development branch; deprecate stable branch 2023-10-21 15:46:53 -07:00
Reinier van der Leer
8cf246bb9e Unbreak benchmark-ci.yml 2023-10-17 11:48:40 -07:00
merwanehamadi
bcb24c1a58 Fix challenges (#5561)
Fix challenges and CI

Signed-off-by: Merwane Hamadi <merwanehamadi@gmail.com>
2023-10-05 10:59:50 -07:00
merwanehamadi
a30cbcc2ce Fix benchmark ci (#5478)
Fix benchmark CI

Signed-off-by: Merwane Hamadi <merwanehamadi@gmail.com>
2023-10-02 12:41:32 -07:00
merwanehamadi
37fbb52d19 Add more challenges + cleanup (#5368)
Signed-off-by: Merwane Hamadi <merwanehamadi@gmail.com>
2023-09-27 17:58:58 -07:00
merwanehamadi
f4e7b1c61c Add eval_id and sync Skill Tree with Frontend(#5287)
Add eval_id to skill tree

Signed-off-by: Merwane Hamadi <merwanehamadi@gmail.com>
2023-09-21 13:36:17 -07:00
merwanehamadi
ff4c76ba00 Make agbenchmark a proxy of the evaluated agent (#5279)
Make agbenchmark a Proxy of the evaluated agent

Signed-off-by: Merwane Hamadi <merwanehamadi@gmail.com>
2023-09-20 16:06:00 -07:00
merwanehamadi
c09a0e7afa Implement old polling mechanism (#5248)
Implement old polling mechanism

Signed-off-by: Merwane Hamadi <merwanehamadi@gmail.com>
2023-09-18 16:23:06 -07:00
merwanehamadi
f4d319cee4 Refactor benchmark (#5247)
Signed-off-by: Merwane Hamadi <merwanehamadi@gmail.com>
2023-09-17 06:55:20 -07:00
merwanehamadi
f76d45cd9e Remove start from agbenchmark (#5241)
Signed-off-by: Merwane Hamadi <merwanehamadi@gmail.com>
2023-09-16 17:22:49 -07:00
Reinier van der Leer
b21d68a8ab Migrate AutoGPT agent to poetry (#5219)
Inspired by #1102

* Migrate AutoGPT agent to poetry

  Co-authored-by: rickythefox <richard@ginzburg.se>

* Rewrite automatic dependency check (check_requirements.py) for poetry

* Sort dependencies

* Add instructions for poetry to README
2023-09-15 05:18:44 +02:00
Merwane Hamadi
cd4589d4d9 Add CI to the forge
Signed-off-by: Merwane Hamadi <merwanehamadi@gmail.com>
2023-09-11 16:12:44 -07:00
Merwane Hamadi
b512808653 Name agents like their github repos
Signed-off-by: Merwane Hamadi <merwanehamadi@gmail.com>
2023-09-07 17:25:50 -07:00
Merwane Hamadi
8ccd2fd367 Benchmark agents without submodule + ability to pin a specific commit.
Signed-off-by: Merwane Hamadi <merwanehamadi@gmail.com>
2023-09-07 16:58:22 -07:00
Merwane Hamadi
fa888bfafa Add back api mode
Signed-off-by: Merwane Hamadi <merwanehamadi@gmail.com>
2023-09-06 22:51:45 -07:00
Merwane Hamadi
d901d01be8 Benchmark all agents
Signed-off-by: Merwane Hamadi <merwanehamadi@gmail.com>
2023-09-06 22:00:03 -07:00
Merwane Hamadi
bc14028294 Add benchmark CI
Signed-off-by: Merwane Hamadi <merwanehamadi@gmail.com>
2023-09-06 19:50:24 -07:00