### Summary
Performance optimization for the platform's store and creator
functionality by adding targeted database indexes and implementing
materialized views to reduce query execution time.
### Changes 🏗️
**Database Performance Optimizations:**
- Added strategic database indexes for `StoreListing`,
`StoreListingVersion`, `StoreListingReview`, `AgentGraphExecution`, and
`Profile` tables
- Implemented materialized views (`mv_agent_run_counts`,
`mv_review_stats`) to cache expensive aggregation queries
- Optimized `StoreAgent` and `Creator` views to use materialized views
and improved query patterns
- Added automated refresh function with 15-minute scheduling for
materialized views (when pg_cron extension is available)
**Key Performance Improvements:**
- Filtered indexes on approved store listings to speed up marketplace
queries
- GIN index on categories for faster category-based searches
- Composite indexes for common query patterns (e.g., listing + version
lookups)
- Pre-computed agent run counts and review statistics to eliminate
expensive aggregations
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
- [x] Verified migration runs successfully without errors
- [x] Confirmed materialized views are created and populated correctly
- [x] Tested StoreAgent and Creator view queries return expected results
- [x] Validated automatic refresh function works properly
- [x] Confirmed rollback migration successfully removes all changes
#### For configuration changes:
- [x] `.env.example` is updated or already compatible with my changes
- [x] `docker-compose.yml` is updated or already compatible with my
changes
- [x] I have included a list of my configuration changes in the PR
description (under **Changes**)
**Note:** No configuration changes were required as this is purely a
database schema optimization.
This pull request introduces a comprehensive backend testing guide and
adds new tests for analytics logging and various API endpoints, focusing
on snapshot testing. It also includes corresponding snapshot files for
these tests. Below are the most significant changes:
### Documentation Updates:
* Added a detailed `TESTING.md` file to the backend, providing a guide
for running tests, snapshot testing, writing API route tests, and best
practices. It includes examples for mocking, fixtures, and CI/CD
integration.
### Analytics Logging Tests:
* Implemented tests for logging raw metrics and analytics in
`analytics_test.py`, covering success scenarios, various input values,
invalid requests, and complex nested data. These tests utilize snapshot
testing for response validation.
* Added snapshot files for analytics logging tests, including responses
for success cases, various metric values, and complex analytics data.
[[1]](diffhunk://#diff-654bc5aa1951008ec5c110a702279ef58709ee455ba049b9fa825fa60f7e3869R1-R3)
[[2]](diffhunk://#diff-e0a434b107abc71aeffb7d7989dbfd8f466b5e53f8dea25a87937ec1b885b122R1-R3)
[[3]](diffhunk://#diff-dd0bc0b72264de1a0c0d3bd0c54ad656061317f425e4de461018ca51a19171a0R1-R3)
[[4]](diffhunk://#diff-63af007073db553d04988544af46930458a768544cabd08412265e0818320d11R1-R30)
### Snapshot Files for API Endpoints:
* Added snapshot files for various API endpoint tests, such as:
- Graph-related operations (`graphs_get_single_response`,
`graphs_get_all_response`, `blocks_get_all_response`).
[[1]](diffhunk://#diff-b25dba271606530cfa428c00073d7e016184a7bb22166148ab1726b3e113dda8R1-R29)
[[2]](diffhunk://#diff-1054e58ec3094715660f55bfba1676d65b6833a81a91a08e90ad57922444d056R1-R31)
[[3]](diffhunk://#diff-cfd403ab6f3efc89188acaf993d85e6f792108d1740c7e7149eb05efb73d918dR1-R14)
- User-related operations (`auth_get_or_create_user_response`,
`auth_update_email_response`).
[[1]](diffhunk://#diff-49e65ab1eb6af4d0163a6c54ed10be621ce7336b2ab5d47d47679bfaefdb7059R1-R5)
[[2]](diffhunk://#diff-ac1216f96878bd4356454c317473654d5d5c7c180125663b80b0b45aa5ab52cbR1-R3)
- Credit-related operations (`credits_get_balance_response`,
`credits_get_auto_top_up_response`, `credits_top_up_request_response`).
[[1]](diffhunk://#diff-189488f8da5be74d80ac3fb7f84f1039a408573184293e9ba2e321d535c57cddR1-R3)
[[2]](diffhunk://#diff-ba3c4a6853793cbed24030cdccedf966d71913451ef8eb4b2c4f426ef18ed87aR1-R4)
[[3]](diffhunk://#diff-43d7daa0c82070a9b6aee88a774add8e87533e630bbccbac5a838b7a7ae56a75R1-R3)
- Graph execution and deletion (`blocks_execute_response`,
`graphs_delete_response`).
[[1]](diffhunk://#diff-a2ade7d646ad85a2801e7ff39799a925a612548a1cdd0ed99b44dd870d1465b5R1-R12)
[[2]](diffhunk://#diff-c0d1cd0a8499ee175ce3007c3a87ba5f3235ce02d38ce837560b36a44fdc4a22R1-R3)##
Summary
- add pytest-snapshot to backend dev requirements
- snapshot server route response JSONs
- mention how to update stored snapshots
## Testing
- `poetry run format`
- `poetry run test`
### Checklist 📋
#### For code changes:
- [x] I have clearly listed my changes in the PR description
- [x] I have made a test plan
- [x] I have tested my changes according to the test plan:
<!-- Put your test plan here: -->
- [x] run poetry run test
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Nicholas Tindle <nicholas.tindle@agpt.co>
This pull request includes several changes to improve the backend
functionality and configuration of the `autogpt_platform`. The most
important changes involve adding a RabbitMQ service for testing,
enhancing logging configuration, updating the linter script to handle
errors gracefully, and modifying test configurations.
Backend configuration improvements:
*
[`autogpt_platform/backend/docker-compose.test.yaml`](diffhunk://#diff-f6a211ff1c6d96d19adb5641ee287258a6af8d72a99e33dafb4a334094205a43R29-R43):
Added RabbitMQ service configuration for testing, including health
checks and environment variables.
*
[`autogpt_platform/backend/.env.example`](diffhunk://#diff-62020caf1b9a15e0e3b9b3b1b69d5f6464bf7643f62354cbbaabf755d57b6064R191-R192):
Added a section delimiter for optional API keys for use in finding the
optional keys end when auto generating integrations.
Error handling and logging enhancements:
*
[`autogpt_platform/backend/linter.py`](diffhunk://#diff-0787e3ef718ac9963df64d9ab1d8e7a3b35dc4ab0cb874c65da6c2901e1e4991R3):
Updated the `run` function to handle `subprocess.CalledProcessError`
exceptions and print error output to `stderr` and prevent raising a
stack trace when it should not.
[[1]](diffhunk://#diff-0787e3ef718ac9963df64d9ab1d8e7a3b35dc4ab0cb874c65da6c2901e1e4991R3)
[[2]](diffhunk://#diff-0787e3ef718ac9963df64d9ab1d8e7a3b35dc4ab0cb874c65da6c2901e1e4991L13-R23)
Testing configuration updates:
*
[`autogpt_platform/backend/pyproject.toml`](diffhunk://#diff-26ebebd91da791c6484f07d9d91484a66f52836708f5294b24365603438b880cR111):
Added `asyncio_default_fixture_loop_scope` to pytest configuration for
better control over asyncio fixtures.
*
[`autogpt_platform/backend/run_tests.py`](diffhunk://#diff-f09930577243a4ef5213bf6191a3c500a4b8d3dcfee2d4b452cf7ce66b3c494fL55):
Removed the `postgres-test` service from the test setup script as we
need all of docker services up for the tests to run.
# 🌎 Overview
AutoGPT Store Version 2 expands on the Pre-Store by enhancing agent
discovery, providing richer content presentation, and introducing new
user engagement features. The focus is on creating a visually appealing
and interactive marketplace that allows users to explore and evaluate
agents through images, videos, and detailed descriptions.
### Vision
To create a visually compelling and interactive open-source marketplace
for autonomous AI agents, where users can easily discover, evaluate, and
interact with agents through media-rich listings, ratings, and version
history.
### Objectives
📊 Incorporate visuals (icons, images, videos) into agent listings.
⭐ Introduce a rating system and agent run count.
🔄 Provide version history and update logs from creators.
🔍 Improve user experience with advanced search and filtering features.
### Changes 🏗️
<!-- Concisely describe all of the changes made in this pull request:
-->
### Checklist 📋
#### For code changes:
- [ ] I have clearly listed my changes in the PR description
- [ ] I have made a test plan
- [ ] I have tested my changes according to the test plan:
<!-- Put your test plan here: -->
- [ ] ...
<details>
<summary>Example test plan</summary>
- [ ] Create from scratch and execute an agent with at least 3 blocks
- [ ] Import an agent from file upload, and confirm it executes
correctly
- [ ] Upload agent to marketplace
- [ ] Import an agent from marketplace and confirm it executes correctly
- [ ] Edit an agent from monitor, and confirm it executes correctly
</details>
#### For configuration changes:
- [ ] `.env.example` is updated or already compatible with my changes
- [ ] `docker-compose.yml` is updated or already compatible with my
changes
- [ ] I have included a list of my configuration changes in the PR
description (under **Changes**)
<details>
<summary>Examples of configuration changes</summary>
- Changing ports
- Adding new services that need to communicate with each other
- Secrets or environment variable changes
- New or infrastructure changes such as databases
</details>
---------
Co-authored-by: Bently <tomnoon9@gmail.com>
Co-authored-by: Aarushi <aarushik93@gmail.com>
Restructuring the Repo to make it clear the difference between classic autogpt and the autogpt platform:
* Move the "classic" projects `autogpt`, `forge`, `frontend`, and `benchmark` into a `classic` folder
* Also rename `autogpt` to `original_autogpt` for absolute clarity
* Rename `rnd/` to `autogpt_platform/`
* `rnd/autogpt_builder` -> `autogpt_platform/frontend`
* `rnd/autogpt_server` -> `autogpt_platform/backend`
* Adjust any paths accordingly
