mirror of
https://github.com/Significant-Gravitas/AutoGPT.git
synced 2026-01-09 15:17:59 -05:00
af7d56612d
## Problem After applying the CloudLoggingHandler fix to use BackgroundThreadTransport (#10634), scheduler pods entered a new deadlock during startup when uvicorn reconfigures logging. ## Root Cause When uvicorn starts with a log_config parameter, it calls `logging.config.dictConfig()` which: 1. Calls `_clearExistingHandlers()` 2. Which calls `logging.shutdown()` 3. Which tries to `flush()` all handlers including CloudLoggingHandler 4. CloudLoggingHandler with BackgroundThreadTransport tries to flush its queue 5. The background worker thread tries to acquire the logging module lock to check log levels 6. **Deadlock**: shutdown holds lock waiting for flush to complete, worker thread needs lock to continue ## Thread Dump Evidence From py-spy analysis of the stuck pod: - **Thread 21 (FastAPI)**: Stuck in `flush()` waiting for background thread to drain queue - **Thread 13 (google.cloud.logging.Worker)**: Waiting for logging lock in `isEnabledFor()` - **Thread 1 (MainThread)**: Waiting for logging lock in `getLogger()` during SQLAlchemy import - **Threads 30, 31 (Sentry)**: Also waiting for logging lock ## Solution Set `log_config=None` for all uvicorn servers. This prevents uvicorn from calling `dictConfig()` and avoids the deadlock entirely. **Trade-off**: Uvicorn will use its default logging configuration which may produce duplicate log entries (one from uvicorn, one from the app), but the application will start successfully without deadlocks. ## Changes - Set `log_config=None` in all uvicorn.Config() calls - Remove unused `generate_uvicorn_config` imports ## Testing - [x] Verified scheduler pods can start and become healthy - [x] Health checks respond properly - [x] No deadlocks during startup - [x] Application logs still appear (though may be duplicated) ## Related Issues - Fixes the startup deadlock introduced after #10634
AutoGPT Libs
This is a new project to store shared functionality across different services in the AutoGPT Platform (e.g. authentication)