Compare commits

..

27 Commits

Author SHA1 Message Date
Zamil Majdy
9106f0b5ce poetry lock 2026-02-20 15:11:46 +07:00
Zamil Majdy
37355f7581 fix(copilot): non-cancelling heartbeat, incremental saves, frontend reconnection
- Replace asyncio.timeout() with asyncio.wait() for SDK message iteration
  to avoid corrupting the internal anyio stream on timeout (root cause of
  tool outputs getting stuck)
- Add CancelledError handling + pending task cleanup in finally block
- Fix _end_text_if_open([]) discarding StreamTextEnd events (Sentry bug)
- Save session to DB after each tool input/output for cross-device recovery
- Optimize incremental saves by passing existing_message_count to skip
  redundant DB count queries
- Frontend: invalidate session cache + reset resume ref on stream end
  so SSE reconnection works after drops
2026-02-20 15:07:18 +07:00
Zamil Majdy
e1e3b6094e poetry lock 2026-02-20 15:07:18 +07:00
Zamil Majdy
e18b3c561f Merge branch 'dev' into fix/messed-up-copilot 2026-02-20 11:51:49 +05:30
Zamil Majdy
d937c6839a fix(copilot): handle stream ending without text + PostToolUse logging
When the SDK CLI exits without sending a ResultMessage (parallel tool
execution), the frontend never gets StreamFinish and tools appear stuck.
Now detect StopAsyncIteration and emit StreamFinish as a fallback.

Also add INFO-level PostToolUse hook logging to trace whether the SDK
fires hooks and stashes output for built-in tools like WebSearch.
2026-02-20 13:12:05 +07:00
Zamil Majdy
8c2363ea88 fix(copilot): add safety-net flush and diagnostic logging for parallel tools
WebSearch/web_fetch parallel tool calls end with spinners resolving but no
output shown, then the session ends with no text response at all. Add:

- Safety-net flush after streaming loop for any unresolved tools
- INFO-level logging for every SDK message (type, unresolved count)
- UserMessage block detail logging to trace tool result delivery
- Flush-called-but-empty logging to detect already-resolved-elsewhere
2026-02-20 13:07:42 +07:00
Zamil Majdy
a408b45542 fix(copilot): don't flush parallel tool calls prematurely
The SDK sends parallel tool calls as separate AssistantMessages each
containing only ToolUseBlocks.  The flush logic treated each new
AssistantMessage as a new turn and prematurely emitted empty output for
prior tools, causing spinners to disappear and the stream to appear
stuck.

Skip flush and wait_for_stash when the incoming AssistantMessage is a
parallel continuation (contains only ToolUseBlocks).  Also prevent
duplicate StreamToolOutputAvailable for already-resolved tool calls.
2026-02-20 11:43:44 +07:00
Zamil Majdy
3a38b5e9bd fix(copilot): address review comments — wait_for_stash fast path, error marker, compat test
- Add fast path in wait_for_stash: check event.is_set() before clearing
  to avoid unnecessary 0.5s timeout when PostToolUse hook completes
  before the streaming loop calls wait_for_stash
- Tighten "failed" error marker to "failed to" in _is_tool_error_or_denial
  to avoid matching benign outputs like "3 tests failed"
- Add max_buffer_size to SDK compat test fields_we_use
2026-02-20 11:18:49 +07:00
Zamil Majdy
3491365b45 poetry lock 2026-02-20 11:15:03 +07:00
Zamil Majdy
d3299cfd7f fix(copilot): remove redundant resolveInProgressTools streaming→ready effect
The backend wait_for_stash() + _flush_unresolved_tool_calls() already
ensures all tool calls are resolved before StreamFinish. The useEffect
that called resolveInProgressTools on streaming→ready was a frontend
safety net for the same issue — no longer needed.

Keep the function itself for stop() (user cancellation).
2026-02-20 10:52:32 +07:00
Zamil Majdy
9161090944 Merge branch 'dev' of github.com:Significant-Gravitas/AutoGPT into fix/messed-up-copilot
Resolve conflict in service.py:
- Take dev's _build_query_message() refactor
- Restore _compress_conversation_history (dev's signature)
- Keep _is_tool_error_or_denial (tested in dev, harmless)
- Drop redundant inline query-building and approach logging
2026-02-20 10:48:20 +07:00
Zamil Majdy
372f9bff32 fix(copilot): address review comments — SDK compat test, output_len, error marker
- Add sdk_compat_test.py (17 tests) verifying the claude-agent-sdk public
  API surface we depend on, replacing the need for a tight version pin.
- Fix output_len logging: use len(str(...)) so dict outputs report
  serialized size, not key count.
- Tighten "failed" error marker to "failed to" to avoid false positives
  on benign tool output like "3 tests failed out of 10".
2026-02-20 10:30:03 +07:00
Zamil Majdy
17995596db poetry lock 2026-02-20 10:29:33 +07:00
Zamil Majdy
7acbbd0f05 poetry lock 2026-02-20 10:21:29 +07:00
Zamil Majdy
4be03fcc08 fix(copilot): remove redundant resolveInProgressTools frontend safety net
The backend already resolves all tool calls via wait_for_stash +
_flush_unresolved_tool_calls before StreamFinish, making the
streaming→ready transition cleanup unnecessary. The isComplete
hydration fix (for page refresh/crash recovery) is kept since it
covers a genuinely different failure mode.
2026-02-20 10:12:33 +07:00
Zamil Majdy
eb7bd6bdae fix(copilot): unify context-building logic for resume and non-resume paths
Consolidates the two separate context-injection paths (gap detection for
--resume, full compression for non-resume) into a single flow: determine
messages → compress → format → prepend. Renames _compress_conversation_history
to _compress_messages accepting a list directly.
2026-02-20 10:05:04 +07:00
Zamil Majdy
d81e7dd6c9 Merge branch 'dev' into fix/messed-up-copilot
Resolve service.py conflicts: take dev's file-based transcript approach
(CapturedTranscript.path + read_transcript_file) and public client API,
layer our fixes on top (wait_for_stash race-condition fix, session_id
logging, approach logging).
2026-02-20 09:54:07 +07:00
Zamil Majdy
78b52b956d fix(copilot): address PR review comments — runtime check, SDK version pin, event-based stash
- Replace bare `assert client._query` with proper RuntimeError check
- Add TECH DEBT comments on private SDK internal usage
- Pin claude-agent-sdk to ~0.1.35 (tighter constraint for private API access)
- Replace sleep(0.1) with event-based wait_for_stash() for race-condition fix
- Add wait_for_stash synchronisation tests
2026-02-20 09:46:19 +07:00
Zamil Majdy
e476185c3a fix(copilot): mitigate SDK hook race condition and improve diagnostic logging
- Add 100ms yield before flush when unresolved tool calls exist, giving
  PostToolUse hooks time to complete before the stash is checked. This
  mitigates the race condition in claude_agent_sdk where hooks are
  fire-and-forget (start_soon) while messages arrive immediately.
- Add has_unresolved_tool_calls property to SDKResponseAdapter
- Differentiate empty flush warnings to flag likely race conditions
- Add session_id to all SDK log messages ([SDK] [<session>] ...)
- Log session approach (resume/compression/single-turn) with context sizes
- Elevate session save log from debug to info
2026-02-20 09:32:30 +07:00
Zamil Majdy
b1c5000937 fix(copilot): improve tool flush logging, transcript capture, and stale spinner safety nets
- Elevate flush logging from debug to info/warning with structured messages
  showing tool names and IDs for production diagnostics
- Capture raw SDK output for transcript instead of relying on Stop hook
  file path (CLI doesn't write JSONL in SDK mode)
- Add _build_transcript() to reconstruct JSONL from captured entries
- Add isComplete option to hydration conversion — marks dangling tool calls
  as completed when session has no active stream (fixes stale spinners on
  page refresh)
- Add resolveInProgressTools safety net on streaming→ready transition
  (catches tool parts the backend didn't emit output for)
- Add 3 new tests for flush mechanism (ResultMessage, AssistantMessage,
  stashed output)
2026-02-20 08:56:56 +07:00
Zamil Majdy
7ee870ed70 fix(copilot): catch OSError in sandbox killpg to prevent zombie processes
Catch OSError broadly (not just ProcessLookupError) when calling
os.killpg so that EPERM or other errors don't skip the subsequent
await proc.communicate(), which would leave the subprocess un-reaped.
2026-02-20 02:49:51 +07:00
Zamil Majdy
240e403592 fix(copilot): fix transcript validation and resume test resilience
- Replace brittle line-count check (< 3) in read_transcript_file with
  proper validate_transcript() which checks for actual user/assistant
  entries — avoids rejecting valid short transcripts while still
  filtering metadata-only files
- Add debug logging for transcript source selection and fallback path
  to aid diagnosing resume issues in Docker
- Make test_sdk_resume_multi_turn skip gracefully when the CLI doesn't
  produce usable transcripts (environment-dependent: CLI version,
  platform) instead of hard-failing
2026-02-20 02:47:23 +07:00
Zamil Majdy
c3e94f7d9c fix(copilot): address review comments — counter order, error markers, tests
- Move task_spawn_count increment after limit check so denied spawns
  don't consume a slot (greptile feedback)
- Add "failed" marker to _is_tool_error_or_denial to catch internal
  tool execution failures from _mcp_error (coderabbit feedback)
- Add 17 unit tests for _is_tool_error_or_denial covering all markers,
  denial messages, and false-positive scenarios
2026-02-20 02:29:51 +07:00
Zamil Majdy
a0a040f102 fix(copilot): sandbox kill, tool event logging, and background task UX
- Fix sandbox process kill: use start_new_session + os.killpg to kill
  the entire bwrap process group on timeout (proc.kill alone only kills
  the parent, leaving children running until natural completion)
- Add StreamToolInputAvailable/StreamToolOutputAvailable to publish_chunk
  logging filter so tool events are visible in Docker logs
- Add system prompt instruction telling Claude not to use
  run_in_background on Task tool (gets denied by security hooks)
- Add tool event debug logging in SDK streaming loop for tracing
  tool execution visibility issues
2026-02-20 02:24:31 +07:00
Zamil Majdy
23225aa323 fix(copilot): address review comments — slot counting, heartbeat, error detection
- Move run_in_background check before task_spawn_count increment so
  denied background Tasks don't consume a subtask slot
- Replace asyncio.wait_for() with asyncio.timeout() for heartbeat loop
  to avoid leaving the async generator in a broken state
- Tighten _is_tool_error_or_denial: remove overly broad markers
  ("error", "failed", "not found") that cause false positives; add
  markers for actual denial messages ("not supported", "maximum")
2026-02-20 01:19:33 +07:00
Zamil Majdy
fed645cb79 Merge branch 'dev' of github.com:Significant-Gravitas/AutoGPT into fix/messed-up-copilot 2026-02-20 01:14:46 +07:00
Zamil Majdy
009753f2b3 fix(copilot): prevent background agent stalls and context hallucination
- Block Task tool's run_in_background param in security hooks — background
  agents stall the SSE stream and get killed when the main agent exits
- Add heartbeats (15s interval) to SDK streaming loop so proxies/LBs don't
  close idle SSE connections during long tool execution
- Fix summarization prompt that forced LLM to fabricate content for all 9
  mandatory sections; now sections are optional and hallucination is
  explicitly prohibited
- Include tool error/denial outcomes in conversation context formatting —
  previously all tool messages were dropped, so the agent couldn't see
  that security denials blocked its file writes and hallucinated success
2026-02-19 23:28:22 +07:00
24 changed files with 603 additions and 781 deletions

View File

@@ -1,572 +0,0 @@
2026-02-21 20:31:19,811 INFO Initializing LaunchDarkly Client 9.15.0
2026-02-21 20:31:19,812 INFO Starting event processor
2026-02-21 20:31:19,812 INFO Starting StreamingUpdateProcessor connecting to uri: https://stream.launchdarkly.com/all
2026-02-21 20:31:19,812 INFO Waiting up to 5 seconds for LaunchDarkly client to initialize...
2026-02-21 20:31:19,812 INFO Connecting to stream at https://stream.launchdarkly.com/all
2026-02-21 20:31:20,051 INFO StreamingUpdateProcessor initialized ok.
2026-02-21 20:31:20,051 INFO Started LaunchDarkly Client: OK
2026-02-21 20:31:20,051 INFO LaunchDarkly client initialized successfully
2026-02-21 20:31:21,578 WARNING Provider LINEAR implements OAuth but the required env vars LINEAR_CLIENT_ID and LINEAR_CLIENT_SECRET are not both set
2026-02-21 20:31:21,623 WARNING Authentication error: Langfuse client initialized without public_key. Client will be disabled. Provide a public_key parameter or set LANGFUSE_PUBLIC_KEY environment variable. 
2026-02-21 20:31:21,796 INFO Metrics endpoint exposed at /metrics for external-api
2026-02-21 20:31:21,800 INFO Metrics endpoint exposed at /metrics for rest-api
2026-02-21 20:31:21,881 INFO Metrics endpoint exposed at /metrics for websocket-server
2026-02-21 20:31:21,913 WARNING Postmark server API token not found, email sending disabled
2026-02-21 20:31:21,956 INFO [DatabaseManager] started with PID 6089
2026-02-21 20:31:21,958 INFO [Scheduler] started with PID 6090
2026-02-21 20:31:21,959 INFO [NotificationManager] started with PID 6091
2026-02-21 20:31:21,960 INFO [WebsocketServer] started with PID 6092
2026-02-21 20:31:21,961 INFO [AgentServer] started with PID 6093
2026-02-21 20:31:21,962 INFO [ExecutionManager] started with PID 6094
2026-02-21 20:31:21,963 INFO [CoPilotExecutor] Starting...
2026-02-21 20:31:21,963 INFO [CoPilotExecutor] Pod assigned executor_id: fb7d76b3-8dc3-40a4-947e-a93bfad207da
2026-02-21 20:31:21,963 INFO [CoPilotExecutor] Spawn max-5 workers...
2026-02-21 20:31:21,970 INFO [PID-6048|THREAD-77685505|CoPilotExecutor|RabbitMQ-124e33d7-4877-4745-9778-6b6b06de92d2] Acquiring connection started...
2026-02-21 20:31:21,971 INFO [PID-6048|THREAD-77685506|CoPilotExecutor|RabbitMQ-124e33d7-4877-4745-9778-6b6b06de92d2] Acquiring connection started...
2026-02-21 20:31:21,973 INFO Pika version 1.3.2 connecting to ('::1', 5672, 0, 0)
2026-02-21 20:31:21,973 INFO Pika version 1.3.2 connecting to ('::1', 5672, 0, 0)
2026-02-21 20:31:21,974 INFO Socket connected: <socket.socket fd=30, family=30, type=1, proto=6, laddr=('::1', 55999, 0, 0), raddr=('::1', 5672, 0, 0)>
2026-02-21 20:31:21,975 INFO Socket connected: <socket.socket fd=29, family=30, type=1, proto=6, laddr=('::1', 55998, 0, 0), raddr=('::1', 5672, 0, 0)>
2026-02-21 20:31:21,975 INFO Streaming transport linked up: (<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x120f5eba0>, _StreamingProtocolShim: <SelectConnection PROTOCOL transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x120f5eba0> params=<ConnectionParameters host=localhost port=5672 virtual_host=/ ssl=False>>).
2026-02-21 20:31:21,976 INFO Streaming transport linked up: (<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x120fa0410>, _StreamingProtocolShim: <SelectConnection PROTOCOL transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x120fa0410> params=<ConnectionParameters host=localhost port=5672 virtual_host=/ ssl=False>>).
2026-02-21 20:31:21,990 INFO AMQPConnector - reporting success: <SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x120fa0410> params=<ConnectionParameters host=localhost port=5672 virtual_host=/ ssl=False>>
2026-02-21 20:31:21,991 INFO AMQPConnectionWorkflow - reporting success: <SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x120fa0410> params=<ConnectionParameters host=localhost port=5672 virtual_host=/ ssl=False>>
2026-02-21 20:31:21,991 INFO AMQPConnector - reporting success: <SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x120f5eba0> params=<ConnectionParameters host=localhost port=5672 virtual_host=/ ssl=False>>
2026-02-21 20:31:21,991 INFO Connection workflow succeeded: <SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x120fa0410> params=<ConnectionParameters host=localhost port=5672 virtual_host=/ ssl=False>>
2026-02-21 20:31:21,991 INFO AMQPConnectionWorkflow - reporting success: <SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x120f5eba0> params=<ConnectionParameters host=localhost port=5672 virtual_host=/ ssl=False>>
2026-02-21 20:31:21,991 INFO Created channel=1
2026-02-21 20:31:21,992 INFO Connection workflow succeeded: <SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x120f5eba0> params=<ConnectionParameters host=localhost port=5672 virtual_host=/ ssl=False>>
2026-02-21 20:31:21,992 INFO Created channel=1
2026-02-21 20:31:22,005 INFO [PID-6048|THREAD-77685505|CoPilotExecutor|RabbitMQ-124e33d7-4877-4745-9778-6b6b06de92d2] Acquiring connection completed successfully.
2026-02-21 20:31:22,005 INFO [PID-6048|THREAD-77685506|CoPilotExecutor|RabbitMQ-124e33d7-4877-4745-9778-6b6b06de92d2] Acquiring connection completed successfully.
2026-02-21 20:31:22,007 INFO [CoPilotExecutor] Starting to consume cancel messages...
2026-02-21 20:31:22,008 INFO [CoPilotExecutor] Starting to consume run messages...
2026-02-21 20:31:23,199 INFO Initializing LaunchDarkly Client 9.15.0
2026-02-21 20:31:23,201 INFO Starting event processor
2026-02-21 20:31:23,202 INFO Starting StreamingUpdateProcessor connecting to uri: https://stream.launchdarkly.com/all
2026-02-21 20:31:23,202 INFO Waiting up to 5 seconds for LaunchDarkly client to initialize...
2026-02-21 20:31:23,202 INFO Connecting to stream at https://stream.launchdarkly.com/all
2026-02-21 20:31:23,331 INFO StreamingUpdateProcessor initialized ok.
2026-02-21 20:31:23,331 INFO Started LaunchDarkly Client: OK
2026-02-21 20:31:23,332 INFO LaunchDarkly client initialized successfully
2026-02-21 20:31:23,891 INFO Initializing LaunchDarkly Client 9.15.0
2026-02-21 20:31:23,892 INFO Starting event processor
2026-02-21 20:31:23,893 INFO Starting StreamingUpdateProcessor connecting to uri: https://stream.launchdarkly.com/all
2026-02-21 20:31:23,893 INFO Waiting up to 5 seconds for LaunchDarkly client to initialize...
2026-02-21 20:31:23,893 INFO Connecting to stream at https://stream.launchdarkly.com/all
2026-02-21 20:31:23,946 INFO Initializing LaunchDarkly Client 9.15.0
2026-02-21 20:31:23,947 INFO Starting event processor
2026-02-21 20:31:23,947 INFO Starting StreamingUpdateProcessor connecting to uri: https://stream.launchdarkly.com/all
2026-02-21 20:31:23,947 INFO Waiting up to 5 seconds for LaunchDarkly client to initialize...
2026-02-21 20:31:23,948 INFO Connecting to stream at https://stream.launchdarkly.com/all
2026-02-21 20:31:24,017 INFO StreamingUpdateProcessor initialized ok.
2026-02-21 20:31:24,017 INFO Started LaunchDarkly Client: OK
2026-02-21 20:31:24,017 INFO LaunchDarkly client initialized successfully
2026-02-21 20:31:24,065 INFO StreamingUpdateProcessor initialized ok.
2026-02-21 20:31:24,065 INFO Started LaunchDarkly Client: OK
2026-02-21 20:31:24,065 INFO LaunchDarkly client initialized successfully
2026-02-21 20:31:24,707 INFO [NotificationManager] Starting...
2026-02-21 20:31:24,750 INFO Metrics endpoint exposed at /metrics for NotificationManager
2026-02-21 20:31:24,754 INFO [PID-6091|THREAD-77685702|NotificationManager|FastAPI server-d17271ed-e3a2-4e93-900b-a0d3bd2b8100] Running FastAPI server started...
2026-02-21 20:31:24,755 INFO [NotificationManager] Starting RPC server at http://localhost:8007
2026-02-21 20:31:24,756 INFO [NotificationManager] [NotificationManager] ⏳ Configuring RabbitMQ...
2026-02-21 20:31:24,757 INFO [PID-6091|THREAD-77685703|NotificationManager|AsyncRabbitMQ-7963c91c-c443-4479-a55e-5e9a8d7d942d] Acquiring async connection started...
2026-02-21 20:31:24,775 INFO Started server process [6091]
2026-02-21 20:31:24,775 INFO Waiting for application startup.
2026-02-21 20:31:24,776 INFO Application startup complete.
2026-02-21 20:31:24,777 ERROR [Errno 48] error while attempting to bind on address ('::1', 8007, 0, 0): [errno 48] address already in use
2026-02-21 20:31:24,781 INFO Waiting for application shutdown.
2026-02-21 20:31:24,781 INFO [NotificationManager] ✅ FastAPI has finished
2026-02-21 20:31:24,782 INFO Application shutdown complete.
2026-02-21 20:31:24,783 INFO [NotificationManager] 🛑 Shared event loop stopped
2026-02-21 20:31:24,783 INFO [NotificationManager] 🧹 Running cleanup
2026-02-21 20:31:24,783 INFO [NotificationManager] ⏳ Disconnecting RabbitMQ...
Process NotificationManager:
Traceback (most recent call last):
File "/opt/homebrew/Cellar/python@3.13/3.13.1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/multiprocessing/process.py", line 313, in _bootstrap
self.run()
~~~~~~~~^^
File "/opt/homebrew/Cellar/python@3.13/3.13.1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/majdyz/Code/AutoGPT/autogpt_platform/backend/backend/util/process.py", line 83, in execute_run_command
self.cleanup()
~~~~~~~~~~~~^^
File "/Users/majdyz/Code/AutoGPT/autogpt_platform/backend/backend/notifications/notifications.py", line 1094, in cleanup
self.run_and_wait(self.rabbitmq_service.disconnect())
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/majdyz/Code/AutoGPT/autogpt_platform/backend/backend/util/service.py", line 136, in run_and_wait
return asyncio.run_coroutine_threadsafe(coro, self.shared_event_loop).result()
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/python@3.13/3.13.1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/asyncio/tasks.py", line 1003, in run_coroutine_threadsafe
loop.call_soon_threadsafe(callback)
~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "/opt/homebrew/Cellar/python@3.13/3.13.1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/asyncio/base_events.py", line 873, in call_soon_threadsafe
self._check_closed()
~~~~~~~~~~~~~~~~~~^^
File "/opt/homebrew/Cellar/python@3.13/3.13.1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/asyncio/base_events.py", line 551, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
/opt/homebrew/Cellar/python@3.13/3.13.1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/multiprocessing/process.py:327: RuntimeWarning: coroutine 'AsyncRabbitMQ.disconnect' was never awaited
traceback.print_exc()
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
2026-02-21 20:31:24,846 INFO Initializing LaunchDarkly Client 9.15.0
2026-02-21 20:31:24,848 INFO Starting event processor
2026-02-21 20:31:24,848 INFO Starting StreamingUpdateProcessor connecting to uri: https://stream.launchdarkly.com/all
2026-02-21 20:31:24,849 INFO Waiting up to 5 seconds for LaunchDarkly client to initialize...
2026-02-21 20:31:24,849 INFO Connecting to stream at https://stream.launchdarkly.com/all
2026-02-21 20:31:24,857 INFO Initializing LaunchDarkly Client 9.15.0
2026-02-21 20:31:24,858 INFO Starting event processor
2026-02-21 20:31:24,858 INFO Starting StreamingUpdateProcessor connecting to uri: https://stream.launchdarkly.com/all
2026-02-21 20:31:24,858 INFO Waiting up to 5 seconds for LaunchDarkly client to initialize...
2026-02-21 20:31:24,858 INFO Connecting to stream at https://stream.launchdarkly.com/all
2026-02-21 20:31:24,862 INFO Initializing LaunchDarkly Client 9.15.0
2026-02-21 20:31:24,863 INFO Starting event processor
2026-02-21 20:31:24,864 INFO Starting StreamingUpdateProcessor connecting to uri: https://stream.launchdarkly.com/all
2026-02-21 20:31:24,864 INFO Waiting up to 5 seconds for LaunchDarkly client to initialize...
2026-02-21 20:31:24,864 INFO Connecting to stream at https://stream.launchdarkly.com/all
2026-02-21 20:31:24,966 INFO StreamingUpdateProcessor initialized ok.
2026-02-21 20:31:24,967 INFO Started LaunchDarkly Client: OK
2026-02-21 20:31:24,967 INFO LaunchDarkly client initialized successfully
2026-02-21 20:31:24,976 INFO StreamingUpdateProcessor initialized ok.
2026-02-21 20:31:24,976 INFO Started LaunchDarkly Client: OK
2026-02-21 20:31:24,976 INFO LaunchDarkly client initialized successfully
2026-02-21 20:31:24,989 INFO StreamingUpdateProcessor initialized ok.
2026-02-21 20:31:24,989 INFO Started LaunchDarkly Client: OK
2026-02-21 20:31:24,989 INFO LaunchDarkly client initialized successfully
2026-02-21 20:31:25,035 INFO Metrics endpoint exposed at /metrics for websocket-server
2026-02-21 20:31:25,036 INFO [WebsocketServer] Starting...
2026-02-21 20:31:25,036 INFO CORS allow origins: ['http://localhost:3000', 'http://127.0.0.1:3000']
2026-02-21 20:31:25,076 INFO Started server process [6092]
2026-02-21 20:31:25,076 INFO Waiting for application startup.
2026-02-21 20:31:25,077 INFO Application startup complete.
2026-02-21 20:31:25,077 INFO [PID-6092|THREAD-77685501|WebsocketServer|AsyncRedis-b6fb3c5c-0070-4c5c-90eb-922d4f2152c2] Acquiring connection started...
2026-02-21 20:31:25,077 INFO [PID-6092|THREAD-77685501|WebsocketServer|AsyncRedis-b6fb3c5c-0070-4c5c-90eb-922d4f2152c2] Acquiring connection started...
2026-02-21 20:31:25,078 ERROR [Errno 48] error while attempting to bind on address ('0.0.0.0', 8001): address already in use
2026-02-21 20:31:25,080 INFO Waiting for application shutdown.
2026-02-21 20:31:25,080 INFO Application shutdown complete.
2026-02-21 20:31:25,080 INFO Event broadcaster stopped
2026-02-21 20:31:25,081 WARNING [WebsocketServer] 🛑 Terminating because of SystemExit: 1
2026-02-21 20:31:25,081 INFO [WebsocketServer] 🧹 Running cleanup
2026-02-21 20:31:25,081 INFO [WebsocketServer] ✅ Cleanup done
2026-02-21 20:31:25,081 INFO [WebsocketServer] 🛑 Terminated
2026-02-21 20:31:25,915 INFO [DatabaseManager] Starting...
2026-02-21 20:31:25,947 INFO Metrics endpoint exposed at /metrics for DatabaseManager
2026-02-21 20:31:25,970 INFO [ExecutionManager] Starting...
2026-02-21 20:31:25,970 INFO [GraphExecutor] [ExecutionManager] 🆔 Pod assigned executor_id: 90ff5962-bdc8-456d-a864-01c5f4f199bd
2026-02-21 20:31:25,971 INFO [GraphExecutor] [ExecutionManager] ⏳ Spawn max-10 workers...
2026-02-21 20:31:25,973 INFO [Scheduler] Starting...
2026-02-21 20:31:25,971 WARNING [ExecutionManager] 🛑 Terminating because of OSError: [Errno 48] Address already in use
Traceback (most recent call last):
File "/Users/majdyz/Code/AutoGPT/autogpt_platform/backend/backend/util/process.py", line 65, in execute_run_command
self.run()
~~~~~~~~^^
File "/Users/majdyz/Code/AutoGPT/autogpt_platform/backend/backend/executor/manager.py", line 1554, in run
start_http_server(settings.config.execution_manager_port)
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/majdyz/Code/AutoGPT/autogpt_platform/backend/.venv/lib/python3.13/site-packages/prometheus_client/exposition.py", line 251, in start_wsgi_server
httpd = make_server(addr, port, app, TmpServer, handler_class=_SilentHandler)
File "/opt/homebrew/Cellar/python@3.13/3.13.1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/wsgiref/simple_server.py", line 150, in make_server
server = server_class((host, port), handler_class)
File "/opt/homebrew/Cellar/python@3.13/3.13.1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/socketserver.py", line 457, in __init__
self.server_bind()
~~~~~~~~~~~~~~~~^^
File "/opt/homebrew/Cellar/python@3.13/3.13.1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/wsgiref/simple_server.py", line 50, in server_bind
HTTPServer.server_bind(self)
~~~~~~~~~~~~~~~~~~~~~~^^^^^^
File "/opt/homebrew/Cellar/python@3.13/3.13.1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/http/server.py", line 136, in server_bind
socketserver.TCPServer.server_bind(self)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
File "/opt/homebrew/Cellar/python@3.13/3.13.1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/socketserver.py", line 473, in server_bind
self.socket.bind(self.server_address)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 48] Address already in use
2026-02-21 20:31:25,978 INFO [ExecutionManager] 🧹 Running cleanup
2026-02-21 20:31:25,978 INFO [GraphExecutor] [ExecutionManager][on_graph_executor_stop 6094] 🧹 Starting graceful shutdown...
2026-02-21 20:31:25,978 INFO [PID-6094|THREAD-77685503|ExecutionManager|RabbitMQ-5b203f2b-8b80-46b1-8e47-481497e68a82] Acquiring connection started...
2026-02-21 20:31:25,980 INFO Pika version 1.3.2 connecting to ('::1', 5672, 0, 0)
2026-02-21 20:31:25,981 INFO Socket connected: <socket.socket fd=14, family=30, type=1, proto=6, laddr=('::1', 56040, 0, 0), raddr=('::1', 5672, 0, 0)>
2026-02-21 20:31:25,982 INFO Streaming transport linked up: (<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x1316cd550>, _StreamingProtocolShim: <SelectConnection PROTOCOL transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x1316cd550> params=<ConnectionParameters host=localhost port=5672 virtual_host=/ ssl=False>>).
2026-02-21 20:31:25,991 INFO AMQPConnector - reporting success: <SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x1316cd550> params=<ConnectionParameters host=localhost port=5672 virtual_host=/ ssl=False>>
2026-02-21 20:31:25,991 INFO AMQPConnectionWorkflow - reporting success: <SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x1316cd550> params=<ConnectionParameters host=localhost port=5672 virtual_host=/ ssl=False>>
2026-02-21 20:31:25,991 INFO Connection workflow succeeded: <SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x1316cd550> params=<ConnectionParameters host=localhost port=5672 virtual_host=/ ssl=False>>
2026-02-21 20:31:25,991 INFO Created channel=1
2026-02-21 20:31:26,001 INFO [PID-6094|THREAD-77685503|ExecutionManager|RabbitMQ-5b203f2b-8b80-46b1-8e47-481497e68a82] Acquiring connection completed successfully.
2026-02-21 20:31:26,001 INFO [GraphExecutor] [ExecutionManager][on_graph_executor_stop 6094] ✅ Exec consumer has been signaled to stop
2026-02-21 20:31:26,001 INFO [GraphExecutor] [ExecutionManager][on_graph_executor_stop 6094] ✅ Executor shutdown completed
2026-02-21 20:31:26,001 INFO [GraphExecutor] [ExecutionManager][on_graph_executor_stop 6094] ✅ Released execution locks
2026-02-21 20:31:26,001 ERROR [GraphExecutor] [ExecutionManager][on_graph_executor_stop 6094] [run-consumer] ⚠️ Error disconnecting run client: <class 'RuntimeError'> cannot join thread before it is started 
2026-02-21 20:31:26,003 INFO [PID-6094|THREAD-77685503|ExecutionManager|RabbitMQ-5b203f2b-8b80-46b1-8e47-481497e68a82] Acquiring connection started...
2026-02-21 20:31:26,005 INFO Pika version 1.3.2 connecting to ('::1', 5672, 0, 0)
2026-02-21 20:31:26,005 INFO Socket connected: <socket.socket fd=20, family=30, type=1, proto=6, laddr=('::1', 56043, 0, 0), raddr=('::1', 5672, 0, 0)>
2026-02-21 20:31:26,006 INFO Streaming transport linked up: (<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x1318e4cd0>, _StreamingProtocolShim: <SelectConnection PROTOCOL transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x1318e4cd0> params=<ConnectionParameters host=localhost port=5672 virtual_host=/ ssl=False>>).
2026-02-21 20:31:26,009 INFO Metrics endpoint exposed at /metrics for Scheduler
2026-02-21 20:31:26,010 INFO AMQPConnector - reporting success: <SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x1318e4cd0> params=<ConnectionParameters host=localhost port=5672 virtual_host=/ ssl=False>>
2026-02-21 20:31:26,010 INFO AMQPConnectionWorkflow - reporting success: <SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x1318e4cd0> params=<ConnectionParameters host=localhost port=5672 virtual_host=/ ssl=False>>
2026-02-21 20:31:26,010 INFO Connection workflow succeeded: <SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x1318e4cd0> params=<ConnectionParameters host=localhost port=5672 virtual_host=/ ssl=False>>
2026-02-21 20:31:26,011 INFO Created channel=1
2026-02-21 20:31:26,015 INFO [PID-6090|THREAD-77685897|Scheduler|FastAPI server-6caca9cc-c4c1-417f-8b83-d96f02472df9] Running FastAPI server started...
2026-02-21 20:31:26,016 INFO [Scheduler] Starting RPC server at http://localhost:8003
2026-02-21 20:31:26,016 INFO [PID-6094|THREAD-77685503|ExecutionManager|RabbitMQ-5b203f2b-8b80-46b1-8e47-481497e68a82] Acquiring connection completed successfully.
2026-02-21 20:31:26,016 ERROR [GraphExecutor] [ExecutionManager][on_graph_executor_stop 6094] [cancel-consumer] ⚠️ Error disconnecting run client: <class 'RuntimeError'> cannot join thread before it is started 
2026-02-21 20:31:26,019 INFO [GraphExecutor] [ExecutionManager][on_graph_executor_stop 6094] ✅ Finished GraphExec cleanup
2026-02-21 20:31:26,019 INFO [ExecutionManager] ✅ Cleanup done
2026-02-21 20:31:26,019 INFO [ExecutionManager] 🛑 Terminated
2026-02-21 20:31:26,188 INFO [PID-6089|THREAD-77685901|DatabaseManager|FastAPI server-7019e67b-30c1-4d08-a0ec-4f0175629d0e] Running FastAPI server started...
2026-02-21 20:31:26,189 INFO [DatabaseManager] Starting RPC server at http://localhost:8005
2026-02-21 20:31:26,197 INFO [DatabaseManager] ⏳ Connecting to Database...
2026-02-21 20:31:26,197 INFO [PID-6089|THREAD-77685902|DatabaseManager|Prisma-64fcde85-3de3-4783-b2c6-789775451cd0] Acquiring connection started...
2026-02-21 20:31:26,254 INFO [Scheduler] [APScheduler] Adding job tentatively -- it will be properly scheduled when the scheduler starts
2026-02-21 20:31:26,255 INFO [Scheduler] [APScheduler] Adding job tentatively -- it will be properly scheduled when the scheduler starts
2026-02-21 20:31:26,255 INFO [Scheduler] [APScheduler] Adding job tentatively -- it will be properly scheduled when the scheduler starts
2026-02-21 20:31:26,255 INFO [Scheduler] [APScheduler] Adding job tentatively -- it will be properly scheduled when the scheduler starts
2026-02-21 20:31:26,255 INFO [Scheduler] [APScheduler] Adding job tentatively -- it will be properly scheduled when the scheduler starts
2026-02-21 20:31:26,255 INFO [Scheduler] [APScheduler] Adding job tentatively -- it will be properly scheduled when the scheduler starts
2026-02-21 20:31:26,256 INFO [Scheduler] [APScheduler] Adding job tentatively -- it will be properly scheduled when the scheduler starts
2026-02-21 20:31:26,346 INFO [PID-6089|THREAD-77685902|DatabaseManager|Prisma-64fcde85-3de3-4783-b2c6-789775451cd0] Acquiring connection completed successfully.
2026-02-21 20:31:26,346 INFO [DatabaseManager] ✅ Ready
2026-02-21 20:31:26,347 ERROR [Errno 48] error while attempting to bind on address ('::1', 8005, 0, 0): [errno 48] address already in use
2026-02-21 20:31:26,349 INFO [DatabaseManager] ⏳ Disconnecting Database...
2026-02-21 20:31:26,349 INFO [PID-6089|THREAD-77685902|DatabaseManager|Prisma-2397ec31-7da6-4598-a012-6c48f17ea97f] Releasing connection started...
2026-02-21 20:31:26,350 INFO [PID-6089|THREAD-77685902|DatabaseManager|Prisma-2397ec31-7da6-4598-a012-6c48f17ea97f] Releasing connection completed successfully.
2026-02-21 20:31:26,351 INFO [DatabaseManager] ✅ FastAPI has finished
2026-02-21 20:31:26,351 INFO [DatabaseManager] 🛑 Shared event loop stopped
2026-02-21 20:31:26,351 INFO [DatabaseManager] 🧹 Running cleanup
Process DatabaseManager:
Traceback (most recent call last):
File "/opt/homebrew/Cellar/python@3.13/3.13.1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/multiprocessing/process.py", line 313, in _bootstrap
self.run()
~~~~~~~~^^
File "/opt/homebrew/Cellar/python@3.13/3.13.1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/majdyz/Code/AutoGPT/autogpt_platform/backend/backend/util/process.py", line 83, in execute_run_command
self.cleanup()
~~~~~~~~~~~~^^
File "/Users/majdyz/Code/AutoGPT/autogpt_platform/backend/backend/util/service.py", line 153, in cleanup
self.shared_event_loop.call_soon_threadsafe(self.shared_event_loop.stop)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/python@3.13/3.13.1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/asyncio/base_events.py", line 873, in call_soon_threadsafe
self._check_closed()
~~~~~~~~~~~~~~~~~~^^
File "/opt/homebrew/Cellar/python@3.13/3.13.1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/asyncio/base_events.py", line 551, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
2026-02-21 20:31:26,382 INFO [Scheduler] [APScheduler] Added job "process_weekly_summary" to job store "weekly_notifications"
2026-02-21 20:31:26,390 INFO [Scheduler] [APScheduler] Added job "report_late_executions" to job store "execution"
2026-02-21 20:31:26,392 INFO [Scheduler] [APScheduler] Added job "report_block_error_rates" to job store "execution"
2026-02-21 20:31:26,395 INFO [Scheduler] [APScheduler] Added job "cleanup_expired_files" to job store "execution"
2026-02-21 20:31:26,397 INFO [Scheduler] [APScheduler] Added job "cleanup_oauth_tokens" to job store "execution"
2026-02-21 20:31:26,399 INFO [Scheduler] [APScheduler] Added job "execution_accuracy_alerts" to job store "execution"
2026-02-21 20:31:26,401 INFO [Scheduler] [APScheduler] Added job "ensure_embeddings_coverage" to job store "execution"
2026-02-21 20:31:26,401 INFO [Scheduler] [APScheduler] Scheduler started
2026-02-21 20:31:26,402 INFO [Scheduler] Running embedding backfill on startup...
2026-02-21 20:31:26,440 WARNING Provider LINEAR implements OAuth but the required env vars LINEAR_CLIENT_ID and LINEAR_CLIENT_SECRET are not both set
2026-02-21 20:31:26,468 INFO [PID-6090|THREAD-77685499|Scheduler|AppService client-24942e64-d380-4d36-a245-5c41172e5293] Creating service client started...
2026-02-21 20:31:26,468 INFO [PID-6090|THREAD-77685499|Scheduler|AppService client-24942e64-d380-4d36-a245-5c41172e5293] Creating service client completed successfully.
2026-02-21 20:31:26,485 WARNING Authentication error: Langfuse client initialized without public_key. Client will be disabled. Provide a public_key parameter or set LANGFUSE_PUBLIC_KEY environment variable. 
2026-02-21 20:31:26,652 INFO Metrics endpoint exposed at /metrics for external-api
2026-02-21 20:31:26,655 INFO Metrics endpoint exposed at /metrics for rest-api
2026-02-21 20:31:26,735 INFO [AgentServer] Starting...
2026-02-21 20:31:26,745 INFO Started server process [6093]
2026-02-21 20:31:26,745 INFO Waiting for application startup.
2026-02-21 20:31:26,746 WARNING ⚠ JWT_SIGN_ALGORITHM is set to 'HS256', a symmetric shared-key signature algorithm. We highly recommend using an asymmetric algorithm such as ES256, because when leaked, a shared secret would allow anyone to forge valid tokens and impersonate users. More info: https://supabase.com/docs/guides/auth/signing-keys#choosing-the-right-signing-algorithm
2026-02-21 20:31:26,747 INFO [PID-6093|THREAD-77685502|AgentServer|Prisma-9d930243-0262-4697-b4af-e0bcbec281c4] Acquiring connection started...
2026-02-21 20:31:26,812 INFO [PID-6093|THREAD-77685502|AgentServer|Prisma-9d930243-0262-4697-b4af-e0bcbec281c4] Acquiring connection completed successfully.
2026-02-21 20:31:26,825 INFO Thread pool size set to 60 for sync endpoint/dependency performance
2026-02-21 20:31:26,825 INFO Successfully patched IntegrationCredentialsStore.get_all_creds
2026-02-21 20:31:26,825 INFO Syncing provider costs to blocks...
2026-02-21 20:31:27,576 WARNING Provider WORDPRESS implements OAuth but the required env vars WORDPRESS_CLIENT_ID and WORDPRESS_CLIENT_SECRET are not both set
2026-02-21 20:31:27,631 INFO Registered 1 custom costs for block FirecrawlExtractBlock
/Users/majdyz/Code/AutoGPT/autogpt_platform/backend/backend/blocks/exa/helpers.py:56: UserWarning: Field name "schema" in "SummarySettings" shadows an attribute in parent "BaseModel"
class SummarySettings(BaseModel):
2026-02-21 20:31:27,954 WARNING Provider AIRTABLE implements OAuth but the required env vars AIRTABLE_CLIENT_ID and AIRTABLE_CLIENT_SECRET are not both set
2026-02-21 20:31:29,238 INFO Successfully patched IntegrationCredentialsStore.get_all_creds
2026-02-21 20:31:29,397 WARNING Block WordPressCreatePostBlock credential input 'credentials' provider 'wordpress' has no authentication methods configured - Disabling
2026-02-21 20:31:29,397 WARNING Block WordPressGetAllPostsBlock credential input 'credentials' provider 'wordpress' has no authentication methods configured - Disabling
2026-02-21 20:31:29,465 INFO Synced 82 costs to 82 blocks
2026-02-21 20:31:29,466 WARNING Executing <Task pending name='Task-2' coro=<LifespanOn.main() running at /Users/majdyz/Code/AutoGPT/autogpt_platform/backend/.venv/lib/python3.13/site-packages/uvicorn/lifespan/on.py:86> created at /Users/majdyz/Code/AutoGPT/autogpt_platform/backend/.venv/lib/python3.13/site-packages/uvicorn/lifespan/on.py:51> took 2.654 seconds
2026-02-21 20:31:29,511 INFO [Scheduler] All content has embeddings, skipping backfill
2026-02-21 20:31:29,512 INFO [Scheduler] Running cleanup for orphaned embeddings (blocks/docs)...
2026-02-21 20:31:29,542 INFO [Scheduler] Cleanup completed: no orphaned embeddings found
2026-02-21 20:31:29,542 INFO [Scheduler] Startup embedding backfill complete: {'backfill': {'processed': 0, 'success': 0, 'failed': 0}, 'cleanup': {'deleted': 0}}
2026-02-21 20:31:29,553 INFO Started server process [6090]
2026-02-21 20:31:29,553 INFO Waiting for application startup.
2026-02-21 20:31:29,554 INFO Application startup complete.
2026-02-21 20:31:29,555 INFO Uvicorn running on http://localhost:8003 (Press CTRL+C to quit)
2026-02-21 20:31:31,074 INFO Migrating integration credentials for 0 users
2026-02-21 20:31:31,087 INFO Fixing LLM credential inputs on 0 nodes
2026-02-21 20:31:31,087 INFO Migrating LLM models
2026-02-21 20:31:31,107 INFO Migrated 0 node triggers to triggered presets
2026-02-21 20:31:31,107 INFO [PID-6093|THREAD-77685502|AgentServer|AsyncRedis-f8b888fc-8b03-4807-adfd-c93710c11c85] Acquiring connection started...
2026-02-21 20:31:31,114 INFO [PID-6093|THREAD-77685502|AgentServer|AsyncRedis-f8b888fc-8b03-4807-adfd-c93710c11c85] Acquiring connection completed successfully.
2026-02-21 20:31:31,115 INFO Created consumer group 'chat_consumers' on stream 'chat:completions'
2026-02-21 20:31:31,115 INFO Chat completion consumer started (consumer: consumer-2f92959a)
2026-02-21 20:31:31,116 INFO Application startup complete.
2026-02-21 20:31:31,117 INFO Uvicorn running on http://0.0.0.0:8006 (Press CTRL+C to quit)
2026-02-21 20:31:45,616 INFO 127.0.0.1:56174 - "GET /api/health HTTP/1.1" 404
2026-02-21 20:32:07,632 INFO 127.0.0.1:56317 - "GET /openapi.json HTTP/1.1" 200
2026-02-21 20:32:07,635 WARNING Executing <Task finished name='Task-7' coro=<RequestResponseCycle.run_asgi() done, defined at /Users/majdyz/Code/AutoGPT/autogpt_platform/backend/.venv/lib/python3.13/site-packages/uvicorn/protocols/http/httptools_impl.py:414> result=None created at /Users/majdyz/Code/AutoGPT/autogpt_platform/backend/.venv/lib/python3.13/site-packages/uvicorn/protocols/http/httptools_impl.py:295> took 0.346 seconds
2026-02-21 20:32:41,502 INFO 127.0.0.1:56681 - "POST /api/v2/chat/sessions HTTP/1.1" 404
2026-02-21 20:32:50,005 INFO 127.0.0.1:56736 - "GET /api/docs HTTP/1.1" 404
2026-02-21 20:33:10,267 INFO 127.0.0.1:56898 - "GET /openapi.json HTTP/1.1" 200
2026-02-21 20:33:28,399 INFO 127.0.0.1:56993 - "POST /api/chat/sessions HTTP/1.1" 401
2026-02-21 20:34:20,913 INFO 127.0.0.1:57313 - "GET /openapi.json HTTP/1.1" 200
2026-02-21 20:36:26,326 INFO Running job "report_late_executions (trigger: interval[0:05:00], next run at: 2026-02-21 13:36:26 UTC)" (scheduled at 2026-02-21 13:36:26.255260+00:00)
2026-02-21 20:36:26,333 INFO [PID-6090|THREAD-77695300|Scheduler|AppService client-24942e64-d380-4d36-a245-5c41172e5293] Creating service client started...
2026-02-21 20:36:26,336 INFO [PID-6090|THREAD-77695300|Scheduler|AppService client-24942e64-d380-4d36-a245-5c41172e5293] Creating service client completed successfully.
2026-02-21 20:36:26,336 INFO [PID-6090|THREAD-77695300|Scheduler|AppService client-24942e64-d380-4d36-a245-5c41172e5293] Creating service client started...
2026-02-21 20:36:26,340 INFO [PID-6090|THREAD-77695300|Scheduler|AppService client-24942e64-d380-4d36-a245-5c41172e5293] Creating service client completed successfully.
2026-02-21 20:36:26,439 WARNING Service communication: Retry attempt 1 for '_call_method_sync': HTTPServerError: HTTP 500: Server error '500 Internal Server Error' for url 'http://localhost:8005/get_graph_executions'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500
2026-02-21 20:36:27,802 WARNING Service communication: Retry attempt 2 for '_call_method_sync': HTTPServerError: HTTP 500: Server error '500 Internal Server Error' for url 'http://localhost:8005/get_graph_executions'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500
2026-02-21 20:36:30,362 WARNING Service communication: Retry attempt 3 for '_call_method_sync': HTTPServerError: HTTP 500: Server error '500 Internal Server Error' for url 'http://localhost:8005/get_graph_executions'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500
2026-02-21 20:36:34,885 WARNING Service communication: Retry attempt 4 for '_call_method_sync': HTTPServerError: HTTP 500: Server error '500 Internal Server Error' for url 'http://localhost:8005/get_graph_executions'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500
2026-02-21 20:36:43,438 WARNING Service communication: Retry attempt 5 for '_call_method_sync': HTTPServerError: HTTP 500: Server error '500 Internal Server Error' for url 'http://localhost:8005/get_graph_executions'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500
2026-02-21 20:36:59,905 WARNING Service communication: Retry attempt 6 for '_call_method_sync': HTTPServerError: HTTP 500: Server error '500 Internal Server Error' for url 'http://localhost:8005/get_graph_executions'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500
2026-02-21 20:37:12,581 WARNING Executing <Task pending name='Task-13' coro=<RequestResponseCycle.run_asgi() running at /Users/majdyz/Code/AutoGPT/autogpt_platform/backend/.venv/lib/python3.13/site-packages/uvicorn/protocols/http/httptools_impl.py:416> cb=[set.discard()] created at /Users/majdyz/Code/AutoGPT/autogpt_platform/backend/.venv/lib/python3.13/site-packages/uvicorn/protocols/http/httptools_impl.py:295> took 0.109 seconds
2026-02-21 20:37:12,767 INFO 127.0.0.1:58472 - "GET /api/store/profile HTTP/1.1" 404
2026-02-21 20:37:12,886 INFO 127.0.0.1:58469 - "GET /api/chat/sessions?limit=50 HTTP/1.1" 200
/Users/majdyz/Code/AutoGPT/autogpt_platform/backend/.venv/lib/python3.13/site-packages/pyiceberg/expressions/parser.py:72: PyparsingDeprecationWarning: 'enablePackrat' deprecated - use 'enable_packrat'
ParserElement.enablePackrat()
/Users/majdyz/Code/AutoGPT/autogpt_platform/backend/.venv/lib/python3.13/site-packages/pyiceberg/expressions/parser.py:85: PyparsingDeprecationWarning: 'escChar' argument is deprecated, use 'esc_char'
quoted_identifier = QuotedString('"', escChar="\\", unquoteResults=True)
/Users/majdyz/Code/AutoGPT/autogpt_platform/backend/.venv/lib/python3.13/site-packages/pyiceberg/expressions/parser.py:85: PyparsingDeprecationWarning: 'unquoteResults' argument is deprecated, use 'unquote_results'
quoted_identifier = QuotedString('"', escChar="\\", unquoteResults=True)
/Users/majdyz/Code/AutoGPT/autogpt_platform/backend/.venv/lib/python3.13/site-packages/pyiceberg/table/metadata.py:365: PydanticDeprecatedSince212: Using `@model_validator` with mode='after' on a classmethod is deprecated. Instead, use an instance method. See the documentation at https://docs.pydantic.dev/2.12/concepts/validators/#model-after-validator. Deprecated in Pydantic V2.12 to be removed in V3.0.
@model_validator(mode="after")
/Users/majdyz/Code/AutoGPT/autogpt_platform/backend/.venv/lib/python3.13/site-packages/pyiceberg/table/metadata.py:494: PydanticDeprecatedSince212: Using `@model_validator` with mode='after' on a classmethod is deprecated. Instead, use an instance method. See the documentation at https://docs.pydantic.dev/2.12/concepts/validators/#model-after-validator. Deprecated in Pydantic V2.12 to be removed in V3.0.
@model_validator(mode="after")
/Users/majdyz/Code/AutoGPT/autogpt_platform/backend/.venv/lib/python3.13/site-packages/pyiceberg/table/metadata.py:498: PydanticDeprecatedSince212: Using `@model_validator` with mode='after' on a classmethod is deprecated. Instead, use an instance method. See the documentation at https://docs.pydantic.dev/2.12/concepts/validators/#model-after-validator. Deprecated in Pydantic V2.12 to be removed in V3.0.
@model_validator(mode="after")
/Users/majdyz/Code/AutoGPT/autogpt_platform/backend/.venv/lib/python3.13/site-packages/pyiceberg/table/metadata.py:502: PydanticDeprecatedSince212: Using `@model_validator` with mode='after' on a classmethod is deprecated. Instead, use an instance method. See the documentation at https://docs.pydantic.dev/2.12/concepts/validators/#model-after-validator. Deprecated in Pydantic V2.12 to be removed in V3.0.
@model_validator(mode="after")
/Users/majdyz/Code/AutoGPT/autogpt_platform/backend/.venv/lib/python3.13/site-packages/pyiceberg/table/metadata.py:506: PydanticDeprecatedSince212: Using `@model_validator` with mode='after' on a classmethod is deprecated. Instead, use an instance method. See the documentation at https://docs.pydantic.dev/2.12/concepts/validators/#model-after-validator. Deprecated in Pydantic V2.12 to be removed in V3.0.
@model_validator(mode="after")
/Users/majdyz/Code/AutoGPT/autogpt_platform/backend/.venv/lib/python3.13/site-packages/pyiceberg/table/metadata.py:538: PydanticDeprecatedSince212: Using `@model_validator` with mode='after' on a classmethod is deprecated. Instead, use an instance method. See the documentation at https://docs.pydantic.dev/2.12/concepts/validators/#model-after-validator. Deprecated in Pydantic V2.12 to be removed in V3.0.
@model_validator(mode="after")
/Users/majdyz/Code/AutoGPT/autogpt_platform/backend/.venv/lib/python3.13/site-packages/pyiceberg/table/metadata.py:542: PydanticDeprecatedSince212: Using `@model_validator` with mode='after' on a classmethod is deprecated. Instead, use an instance method. See the documentation at https://docs.pydantic.dev/2.12/concepts/validators/#model-after-validator. Deprecated in Pydantic V2.12 to be removed in V3.0.
@model_validator(mode="after")
/Users/majdyz/Code/AutoGPT/autogpt_platform/backend/.venv/lib/python3.13/site-packages/pyiceberg/table/metadata.py:546: PydanticDeprecatedSince212: Using `@model_validator` with mode='after' on a classmethod is deprecated. Instead, use an instance method. See the documentation at https://docs.pydantic.dev/2.12/concepts/validators/#model-after-validator. Deprecated in Pydantic V2.12 to be removed in V3.0.
@model_validator(mode="after")
/Users/majdyz/Code/AutoGPT/autogpt_platform/backend/.venv/lib/python3.13/site-packages/pyiceberg/table/metadata.py:550: PydanticDeprecatedSince212: Using `@model_validator` with mode='after' on a classmethod is deprecated. Instead, use an instance method. See the documentation at https://docs.pydantic.dev/2.12/concepts/validators/#model-after-validator. Deprecated in Pydantic V2.12 to be removed in V3.0.
@model_validator(mode="after")
2026-02-21 20:37:14,074 INFO 127.0.0.1:58470 - "GET /api/executions HTTP/1.1" 200
2026-02-21 20:37:14,081 WARNING Executing <Task finished name='Task-14' coro=<RequestResponseCycle.run_asgi() done, defined at /Users/majdyz/Code/AutoGPT/autogpt_platform/backend/.venv/lib/python3.13/site-packages/uvicorn/protocols/http/httptools_impl.py:414> result=None created at /Users/majdyz/Code/AutoGPT/autogpt_platform/backend/.venv/lib/python3.13/site-packages/uvicorn/protocols/http/httptools_impl.py:295> took 1.169 seconds
2026-02-21 20:37:15,102 WARNING Executing <Task pending name='Task-1' coro=<Server.serve() running at /Users/majdyz/Code/AutoGPT/autogpt_platform/backend/.venv/lib/python3.13/site-packages/uvicorn/server.py:71> wait_for=<Future pending cb=[Task.task_wakeup()] created at /opt/homebrew/Cellar/python@3.13/3.13.1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/asyncio/tasks.py:713> cb=[run_until_complete.<locals>.done_cb()] created at /opt/homebrew/Cellar/python@3.13/3.13.1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/asyncio/runners.py:100> took 0.224 seconds
2026-02-21 20:37:17,085 INFO 127.0.0.1:58530 - "GET /api/store/profile HTTP/1.1" 404
2026-02-21 20:37:20,772 WARNING Executing <Task pending name='Task-1' coro=<Server.serve() running at /Users/majdyz/Code/AutoGPT/autogpt_platform/backend/.venv/lib/python3.13/site-packages/uvicorn/server.py:71> wait_for=<Future pending cb=[Task.task_wakeup()] created at /opt/homebrew/Cellar/python@3.13/3.13.1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/asyncio/tasks.py:713> cb=[run_until_complete.<locals>.done_cb()] created at /opt/homebrew/Cellar/python@3.13/3.13.1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/asyncio/runners.py:100> took 0.261 seconds
2026-02-21 20:37:21,276 INFO 127.0.0.1:58568 - "GET /api/integrations/providers/system HTTP/1.1" 200
2026-02-21 20:37:21,309 WARNING Executing <Task finished name='Task-23' coro=<RequestResponseCycle.run_asgi() done, defined at /Users/majdyz/Code/AutoGPT/autogpt_platform/backend/.venv/lib/python3.13/site-packages/uvicorn/protocols/http/httptools_impl.py:414> result=None created at /Users/majdyz/Code/AutoGPT/autogpt_platform/backend/.venv/lib/python3.13/site-packages/uvicorn/protocols/http/httptools_impl.py:295> took 0.158 seconds
2026-02-21 20:37:21,329 INFO 127.0.0.1:58570 - "GET /api/integrations/providers HTTP/1.1" 200
2026-02-21 20:37:21,421 WARNING Executing <Task finished name='Task-24' coro=<RequestResponseCycle.run_asgi() done, defined at /Users/majdyz/Code/AutoGPT/autogpt_platform/backend/.venv/lib/python3.13/site-packages/uvicorn/protocols/http/httptools_impl.py:414> result=None created at /Users/majdyz/Code/AutoGPT/autogpt_platform/backend/.venv/lib/python3.13/site-packages/uvicorn/protocols/http/httptools_impl.py:295> took 0.110 seconds
2026-02-21 20:37:22,406 INFO 127.0.0.1:58590 - "GET /api/store/profile HTTP/1.1" 404
2026-02-21 20:37:22,430 INFO 127.0.0.1:58588 - "GET /api/onboarding HTTP/1.1" 200
2026-02-21 20:37:22,453 INFO 127.0.0.1:58570 - "GET /api/executions HTTP/1.1" 200
2026-02-21 20:37:22,476 INFO Loaded session 322af5c3-70fc-4a06-9443-8c5df0aa0c9f from DB: has_messages=True, message_count=11, roles=['user', 'assistant', 'tool', 'assistant', 'tool', 'assistant', 'tool', 'tool', 'assistant', 'tool', 'tool']
2026-02-21 20:37:22,485 INFO Cached session 322af5c3-70fc-4a06-9443-8c5df0aa0c9f from database
2026-02-21 20:37:22,510 INFO 127.0.0.1:58568 - "GET /api/library/agents?page=1&page_size=100 HTTP/1.1" 200
2026-02-21 20:37:22,515 INFO [GET_SESSION] session=322af5c3-70fc-4a06-9443-8c5df0aa0c9f, active_task=False, msg_count=11, last_role=tool
2026-02-21 20:37:22,524 INFO 127.0.0.1:58599 - "GET /api/chat/sessions/322af5c3-70fc-4a06-9443-8c5df0aa0c9f HTTP/1.1" 200
2026-02-21 20:37:22,535 INFO 127.0.0.1:58607 - "GET /api/chat/sessions?limit=50 HTTP/1.1" 200
2026-02-21 20:37:22,608 INFO 127.0.0.1:58568 - "GET /api/integrations/credentials HTTP/1.1" 200
2026-02-21 20:37:23,531 INFO 127.0.0.1:58568 - "GET /api/store/profile HTTP/1.1" 404
2026-02-21 20:37:25,612 INFO 127.0.0.1:58568 - "GET /api/store/profile HTTP/1.1" 404
2026-02-21 20:37:29,708 INFO 127.0.0.1:58671 - "GET /api/store/profile HTTP/1.1" 404
2026-02-21 20:37:29,975 WARNING Service communication: Retry attempt 7 for '_call_method_sync': HTTPServerError: HTTP 500: Server error '500 Internal Server Error' for url 'http://localhost:8005/get_graph_executions'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500
2026-02-21 20:37:34,125 INFO [TIMING] stream_chat_post STARTED, session=322af5c3-70fc-4a06-9443-8c5df0aa0c9f, user=68383665-d3d9-41f3-b10c-fca0dc6080ed, message_len=36
2026-02-21 20:37:34,134 INFO Loading session 322af5c3-70fc-4a06-9443-8c5df0aa0c9f from cache: message_count=11, roles=['user', 'assistant', 'tool', 'assistant', 'tool', 'assistant', 'tool', 'tool', 'assistant', 'tool', 'tool']
2026-02-21 20:37:34,135 INFO [TIMING] session validated in 10.6ms
2026-02-21 20:37:34,136 INFO [STREAM] Saving user message to session 322af5c3-70fc-4a06-9443-8c5df0aa0c9f
2026-02-21 20:37:34,138 INFO Loading session 322af5c3-70fc-4a06-9443-8c5df0aa0c9f from cache: message_count=11, roles=['user', 'assistant', 'tool', 'assistant', 'tool', 'assistant', 'tool', 'tool', 'assistant', 'tool', 'tool']
2026-02-21 20:37:34,168 INFO Saving 1 new messages to DB for session 322af5c3-70fc-4a06-9443-8c5df0aa0c9f: roles=['user'], start_sequence=11
2026-02-21 20:37:34,201 INFO [STREAM] User message saved for session 322af5c3-70fc-4a06-9443-8c5df0aa0c9f
2026-02-21 20:37:34,202 INFO [TIMING] create_task STARTED, task=bba63941-8048-4f39-9329-8568e5ebe9cd, session=322af5c3-70fc-4a06-9443-8c5df0aa0c9f, user=68383665-d3d9-41f3-b10c-fca0dc6080ed
2026-02-21 20:37:34,202 INFO [TIMING] get_redis_async took 0.0ms
2026-02-21 20:37:34,205 INFO [TIMING] redis.hset took 2.9ms
2026-02-21 20:37:34,208 INFO [TIMING] create_task COMPLETED in 6.1ms; task=bba63941-8048-4f39-9329-8568e5ebe9cd, session=322af5c3-70fc-4a06-9443-8c5df0aa0c9f
2026-02-21 20:37:34,208 INFO [TIMING] create_task completed in 6.8ms
2026-02-21 20:37:34,210 INFO [PID-6093|THREAD-77685502|AgentServer|AsyncRabbitMQ-bbe1cabd-35fe-4944-89d1-fddd09c93923] Acquiring async connection started...
2026-02-21 20:37:34,296 INFO [PID-6093|THREAD-77685502|AgentServer|AsyncRabbitMQ-bbe1cabd-35fe-4944-89d1-fddd09c93923] Acquiring async connection completed successfully.
2026-02-21 20:37:34,305 INFO [TIMING] Task enqueued to RabbitMQ, setup=180.6ms
2026-02-21 20:37:34,307 INFO [TIMING] event_generator STARTED, task=bba63941-8048-4f39-9329-8568e5ebe9cd, session=322af5c3-70fc-4a06-9443-8c5df0aa0c9f, user=68383665-d3d9-41f3-b10c-fca0dc6080ed
2026-02-21 20:37:34,307 INFO [TIMING] subscribe_to_task STARTED, task=bba63941-8048-4f39-9329-8568e5ebe9cd, user=68383665-d3d9-41f3-b10c-fca0dc6080ed, last_msg=0-0
2026-02-21 20:37:34,309 INFO [TIMING] Redis hgetall took 2.1ms
2026-02-21 20:37:34,353 INFO [PID-6048|THREAD-77685506|CoPilotExecutor|Redis-943506d1-86e7-48a7-871b-9977fb0ace47] Acquiring connection started...
2026-02-21 20:37:34,435 INFO [PID-6048|THREAD-77685506|CoPilotExecutor|Redis-943506d1-86e7-48a7-871b-9977fb0ace47] Acquiring connection completed successfully.
2026-02-21 20:37:34,442 INFO [CoPilotExecutor] Acquired cluster lock for bba63941-8048-4f39-9329-8568e5ebe9cd, executor_id=fb7d76b3-8dc3-40a4-947e-a93bfad207da
2026-02-21 20:37:34,535 INFO [CoPilotExecutor] [CoPilotExecutor] Worker 13455405056 started
2026-02-21 20:37:34,536 INFO [CoPilotExecutor|task_id:bba63941-8048-4f39-9329-8568e5ebe9cd|session_id:322af5c3-70fc-4a06-9443-8c5df0aa0c9f|user_id:68383665-d3d9-41f3-b10c-fca0dc6080ed] Starting execution
2026-02-21 20:37:35,596 INFO [CoPilotExecutor|task_id:bba63941-8048-4f39-9329-8568e5ebe9cd|session_id:322af5c3-70fc-4a06-9443-8c5df0aa0c9f|user_id:68383665-d3d9-41f3-b10c-fca0dc6080ed] Using SDK service
2026-02-21 20:37:35,596 INFO [PID-6048|THREAD-77697399|CoPilotExecutor|AsyncRedis-2e10c980-0364-4c4b-9b2d-8186f23b1735] Acquiring connection started...
2026-02-21 20:37:35,600 INFO [PID-6048|THREAD-77697399|CoPilotExecutor|AsyncRedis-2e10c980-0364-4c4b-9b2d-8186f23b1735] Acquiring connection completed successfully.
2026-02-21 20:37:35,601 INFO Loading session 322af5c3-70fc-4a06-9443-8c5df0aa0c9f from cache: message_count=12, roles=['user', 'assistant', 'tool', 'assistant', 'tool', 'assistant', 'tool', 'tool', 'assistant', 'tool', 'tool', 'user']
2026-02-21 20:37:35,601 INFO [PID-6048|THREAD-77697399|CoPilotExecutor|AppService client-34797c8f-0201-4f99-bf73-3f3fb4697e6d] Creating service client started...
2026-02-21 20:37:35,601 INFO [PID-6048|THREAD-77697399|CoPilotExecutor|AppService client-34797c8f-0201-4f99-bf73-3f3fb4697e6d] Creating service client completed successfully.
2026-02-21 20:37:35,657 WARNING Service communication: Retry attempt 1 for '_call_method_async': HTTPServerError: HTTP 500: Server error '500 Internal Server Error' for url 'http://localhost:8005/get_chat_session_message_count'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500
2026-02-21 20:37:36,713 WARNING Service communication: Retry attempt 2 for '_call_method_async': HTTPServerError: HTTP 500: Server error '500 Internal Server Error' for url 'http://localhost:8005/get_chat_session_message_count'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500
2026-02-21 20:37:39,646 WARNING Service communication: Retry attempt 3 for '_call_method_async': HTTPServerError: HTTP 500: Server error '500 Internal Server Error' for url 'http://localhost:8005/get_chat_session_message_count'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500
2026-02-21 20:37:43,415 INFO 127.0.0.1:58782 - "GET /api/store/profile HTTP/1.1" 404
2026-02-21 20:37:44,423 WARNING Service communication: Retry attempt 4 for '_call_method_async': HTTPServerError: HTTP 500: Server error '500 Internal Server Error' for url 'http://localhost:8005/get_chat_session_message_count'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500
2026-02-21 20:37:44,486 INFO 127.0.0.1:58782 - "GET /api/store/profile HTTP/1.1" 404
2026-02-21 20:37:45,048 INFO Loading session 322af5c3-70fc-4a06-9443-8c5df0aa0c9f from cache: message_count=12, roles=['user', 'assistant', 'tool', 'assistant', 'tool', 'assistant', 'tool', 'tool', 'assistant', 'tool', 'tool', 'user']
2026-02-21 20:37:45,053 INFO [TASK_LOOKUP] Found running task bba63941... for session 322af5c3...
2026-02-21 20:37:45,063 INFO [CoPilotExecutor] Received cancel for bba63941-8048-4f39-9329-8568e5ebe9cd
2026-02-21 20:37:45,064 INFO [CANCEL] Published cancel for task ...e5ebe9cd session ...f0aa0c9f
2026-02-21 20:37:45,113 INFO Loading session 322af5c3-70fc-4a06-9443-8c5df0aa0c9f from cache: message_count=12, roles=['user', 'assistant', 'tool', 'assistant', 'tool', 'assistant', 'tool', 'tool', 'assistant', 'tool', 'tool', 'user']
2026-02-21 20:37:45,120 INFO [TASK_LOOKUP] Found running task bba63941... for session 322af5c3...
2026-02-21 20:37:45,121 INFO [GET_SESSION] session=322af5c3-70fc-4a06-9443-8c5df0aa0c9f, active_task=True, msg_count=12, last_role=user
2026-02-21 20:37:45,123 INFO 127.0.0.1:58802 - "GET /api/chat/sessions/322af5c3-70fc-4a06-9443-8c5df0aa0c9f HTTP/1.1" 200
2026-02-21 20:37:45,306 INFO [TASK_LOOKUP] Found running task bba63941... for session 322af5c3...
2026-02-21 20:37:45,307 INFO [TIMING] subscribe_to_task STARTED, task=bba63941-8048-4f39-9329-8568e5ebe9cd, user=68383665-d3d9-41f3-b10c-fca0dc6080ed, last_msg=0-0
2026-02-21 20:37:45,309 INFO [TIMING] Redis hgetall took 1.5ms
2026-02-21 20:37:45,604 INFO [CoPilotExecutor|task_id:bba63941-8048-4f39-9329-8568e5ebe9cd|session_id:322af5c3-70fc-4a06-9443-8c5df0aa0c9f|user_id:68383665-d3d9-41f3-b10c-fca0dc6080ed] Cancellation requested
2026-02-21 20:37:45,604 INFO [CoPilotExecutor|task_id:bba63941-8048-4f39-9329-8568e5ebe9cd|session_id:322af5c3-70fc-4a06-9443-8c5df0aa0c9f|user_id:68383665-d3d9-41f3-b10c-fca0dc6080ed] Execution completed in 11.07s
2026-02-21 20:37:45,604 INFO [CoPilotExecutor] Run completed for bba63941-8048-4f39-9329-8568e5ebe9cd
2026-02-21 20:37:45,604 INFO [CoPilotExecutor|task_id:bba63941-8048-4f39-9329-8568e5ebe9cd|session_id:322af5c3-70fc-4a06-9443-8c5df0aa0c9f|user_id:68383665-d3d9-41f3-b10c-fca0dc6080ed] Task cancelled
2026-02-21 20:37:45,605 INFO [CoPilotExecutor] Releasing cluster lock for bba63941-8048-4f39-9329-8568e5ebe9cd
2026-02-21 20:37:45,609 INFO [CoPilotExecutor] Cleaned up completed task bba63941-8048-4f39-9329-8568e5ebe9cd
2026-02-21 20:37:45,610 INFO [TIMING] Redis xread (replay) took 301.1ms, status=running
2026-02-21 20:37:45,610 INFO [TIMING] publish_chunk StreamFinish in 1.8ms (xadd=1.3ms)
2026-02-21 20:37:45,612 INFO [TIMING] Replayed 1 messages, last_id=1771681065606-0
2026-02-21 20:37:45,612 INFO [TIMING] Task still running, starting _stream_listener
2026-02-21 20:37:45,613 INFO [TIMING] subscribe_to_task COMPLETED in 305.8ms; task=bba63941-8048-4f39-9329-8568e5ebe9cd, n_messages_replayed=1
2026-02-21 20:37:45,614 INFO [TIMING] _stream_listener STARTED, task=bba63941-8048-4f39-9329-8568e5ebe9cd, last_id=1771681065606-0
2026-02-21 20:37:45,614 INFO Resume stream chunk
2026-02-21 20:37:45,615 INFO 127.0.0.1:58802 - "GET /api/chat/sessions/322af5c3-70fc-4a06-9443-8c5df0aa0c9f/stream HTTP/1.1" 200
2026-02-21 20:37:45,615 INFO [TIMING] Redis xread (replay) took 11305.8ms, status=running
2026-02-21 20:37:45,616 INFO [TIMING] Replayed 1 messages, last_id=1771681065606-0
2026-02-21 20:37:45,616 INFO [TIMING] Task still running, starting _stream_listener
2026-02-21 20:37:45,616 INFO [TIMING] subscribe_to_task COMPLETED in 11308.9ms; task=bba63941-8048-4f39-9329-8568e5ebe9cd, n_messages_replayed=1
2026-02-21 20:37:45,616 INFO [TIMING] Starting to read from subscriber_queue
2026-02-21 20:37:45,616 INFO [TIMING] FIRST CHUNK from queue at 11.31s, type=StreamFinish
2026-02-21 20:37:45,616 INFO 127.0.0.1:58710 - "POST /api/chat/sessions/322af5c3-70fc-4a06-9443-8c5df0aa0c9f/stream HTTP/1.1" 200
2026-02-21 20:37:45,617 INFO [TIMING] StreamFinish received in 11.31s; n_chunks=1
2026-02-21 20:37:45,617 INFO [TIMING] _stream_listener CANCELLED after 3.5ms, delivered=0
2026-02-21 20:37:45,617 INFO [TIMING] _stream_listener FINISHED in 0.0s; task=bba63941-8048-4f39-9329-8568e5ebe9cd, delivered=0, xread_count=1
2026-02-21 20:37:45,618 INFO Resume stream completed
2026-02-21 20:37:45,618 INFO [TIMING] event_generator FINISHED in 11.31s; task=bba63941-8048-4f39-9329-8568e5ebe9cd, session=322af5c3-70fc-4a06-9443-8c5df0aa0c9f, n_chunks=1
2026-02-21 20:37:45,691 INFO Loading session 322af5c3-70fc-4a06-9443-8c5df0aa0c9f from cache: message_count=12, roles=['user', 'assistant', 'tool', 'assistant', 'tool', 'assistant', 'tool', 'tool', 'assistant', 'tool', 'tool', 'user']
2026-02-21 20:37:45,694 INFO [GET_SESSION] session=322af5c3-70fc-4a06-9443-8c5df0aa0c9f, active_task=False, msg_count=12, last_role=user
2026-02-21 20:37:45,695 INFO 127.0.0.1:58710 - "GET /api/chat/sessions/322af5c3-70fc-4a06-9443-8c5df0aa0c9f HTTP/1.1" 200
2026-02-21 20:37:45,710 INFO 127.0.0.1:58802 - "GET /api/chat/sessions/322af5c3-70fc-4a06-9443-8c5df0aa0c9f/stream HTTP/1.1" 204
2026-02-21 20:37:45,771 INFO Loading session 322af5c3-70fc-4a06-9443-8c5df0aa0c9f from cache: message_count=12, roles=['user', 'assistant', 'tool', 'assistant', 'tool', 'assistant', 'tool', 'tool', 'assistant', 'tool', 'tool', 'user']
2026-02-21 20:37:45,775 INFO [GET_SESSION] session=322af5c3-70fc-4a06-9443-8c5df0aa0c9f, active_task=False, msg_count=12, last_role=user
2026-02-21 20:37:45,775 INFO 127.0.0.1:58710 - "GET /api/chat/sessions/322af5c3-70fc-4a06-9443-8c5df0aa0c9f HTTP/1.1" 200
2026-02-21 20:37:46,075 INFO [CANCEL] Task ...e5ebe9cd confirmed stopped (status=failed) after 1.0s
2026-02-21 20:37:46,076 INFO 127.0.0.1:58782 - "POST /api/chat/sessions/322af5c3-70fc-4a06-9443-8c5df0aa0c9f/cancel HTTP/1.1" 200
2026-02-21 20:37:46,573 INFO 127.0.0.1:58710 - "GET /api/store/profile HTTP/1.1" 404
2026-02-21 20:37:50,090 INFO 127.0.0.1:58710 - "GET /api/integrations/providers/system HTTP/1.1" 200
2026-02-21 20:37:50,103 INFO 127.0.0.1:58842 - "GET /api/integrations/providers HTTP/1.1" 200
2026-02-21 20:37:50,681 INFO Loading session 322af5c3-70fc-4a06-9443-8c5df0aa0c9f from cache: message_count=12, roles=['user', 'assistant', 'tool', 'assistant', 'tool', 'assistant', 'tool', 'tool', 'assistant', 'tool', 'tool', 'user']
2026-02-21 20:37:50,686 INFO 127.0.0.1:58710 - "GET /api/library/agents?page=1&page_size=100 HTTP/1.1" 200
2026-02-21 20:37:50,692 INFO 127.0.0.1:58850 - "GET /api/store/profile HTTP/1.1" 404
2026-02-21 20:37:50,702 INFO 127.0.0.1:58842 - "GET /api/integrations/credentials HTTP/1.1" 200
2026-02-21 20:37:50,710 INFO [GET_SESSION] session=322af5c3-70fc-4a06-9443-8c5df0aa0c9f, active_task=False, msg_count=12, last_role=user
2026-02-21 20:37:50,711 INFO 127.0.0.1:58862 - "GET /api/chat/sessions/322af5c3-70fc-4a06-9443-8c5df0aa0c9f HTTP/1.1" 200
2026-02-21 20:37:50,714 INFO 127.0.0.1:58852 - "GET /api/onboarding HTTP/1.1" 200
2026-02-21 20:37:50,720 INFO 127.0.0.1:58854 - "GET /api/executions HTTP/1.1" 200
2026-02-21 20:37:50,795 INFO 127.0.0.1:58710 - "GET /api/chat/sessions?limit=50 HTTP/1.1" 200
2026-02-21 20:37:51,955 INFO 127.0.0.1:58710 - "GET /api/store/profile HTTP/1.1" 404
2026-02-21 20:37:54,064 INFO 127.0.0.1:58710 - "GET /api/store/profile HTTP/1.1" 404
2026-02-21 20:37:54,157 INFO [TIMING] stream_chat_post STARTED, session=322af5c3-70fc-4a06-9443-8c5df0aa0c9f, user=68383665-d3d9-41f3-b10c-fca0dc6080ed, message_len=5
2026-02-21 20:37:54,169 INFO Loading session 322af5c3-70fc-4a06-9443-8c5df0aa0c9f from cache: message_count=12, roles=['user', 'assistant', 'tool', 'assistant', 'tool', 'assistant', 'tool', 'tool', 'assistant', 'tool', 'tool', 'user']
2026-02-21 20:37:54,170 INFO [TIMING] session validated in 13.0ms
2026-02-21 20:37:54,170 INFO [STREAM] Saving user message to session 322af5c3-70fc-4a06-9443-8c5df0aa0c9f
2026-02-21 20:37:54,172 INFO Loading session 322af5c3-70fc-4a06-9443-8c5df0aa0c9f from cache: message_count=12, roles=['user', 'assistant', 'tool', 'assistant', 'tool', 'assistant', 'tool', 'tool', 'assistant', 'tool', 'tool', 'user']
2026-02-21 20:37:54,212 INFO Saving 1 new messages to DB for session 322af5c3-70fc-4a06-9443-8c5df0aa0c9f: roles=['user'], start_sequence=12
2026-02-21 20:37:54,238 INFO [STREAM] User message saved for session 322af5c3-70fc-4a06-9443-8c5df0aa0c9f
2026-02-21 20:37:54,238 INFO [TIMING] create_task STARTED, task=6360d249-c803-47d3-8a08-d77275e4b2d8, session=322af5c3-70fc-4a06-9443-8c5df0aa0c9f, user=68383665-d3d9-41f3-b10c-fca0dc6080ed
2026-02-21 20:37:54,238 INFO [TIMING] get_redis_async took 0.0ms
2026-02-21 20:37:54,242 INFO [TIMING] redis.hset took 3.1ms
2026-02-21 20:37:54,250 INFO [TIMING] create_task COMPLETED in 11.6ms; task=6360d249-c803-47d3-8a08-d77275e4b2d8, session=322af5c3-70fc-4a06-9443-8c5df0aa0c9f
2026-02-21 20:37:54,251 INFO [TIMING] create_task completed in 12.9ms
2026-02-21 20:37:54,261 INFO [TIMING] Task enqueued to RabbitMQ, setup=103.8ms
2026-02-21 20:37:54,262 INFO [TIMING] event_generator STARTED, task=6360d249-c803-47d3-8a08-d77275e4b2d8, session=322af5c3-70fc-4a06-9443-8c5df0aa0c9f, user=68383665-d3d9-41f3-b10c-fca0dc6080ed
2026-02-21 20:37:54,263 INFO [TIMING] subscribe_to_task STARTED, task=6360d249-c803-47d3-8a08-d77275e4b2d8, user=68383665-d3d9-41f3-b10c-fca0dc6080ed, last_msg=0-0
2026-02-21 20:37:54,264 INFO [TIMING] Redis hgetall took 1.7ms
2026-02-21 20:37:54,265 INFO [CoPilotExecutor] Acquired cluster lock for 6360d249-c803-47d3-8a08-d77275e4b2d8, executor_id=fb7d76b3-8dc3-40a4-947e-a93bfad207da
2026-02-21 20:37:54,267 INFO [CoPilotExecutor|task_id:6360d249-c803-47d3-8a08-d77275e4b2d8|session_id:322af5c3-70fc-4a06-9443-8c5df0aa0c9f|user_id:68383665-d3d9-41f3-b10c-fca0dc6080ed] Starting execution
2026-02-21 20:37:54,286 INFO [CoPilotExecutor|task_id:6360d249-c803-47d3-8a08-d77275e4b2d8|session_id:322af5c3-70fc-4a06-9443-8c5df0aa0c9f|user_id:68383665-d3d9-41f3-b10c-fca0dc6080ed] Using SDK service
2026-02-21 20:37:54,290 INFO Loading session 322af5c3-70fc-4a06-9443-8c5df0aa0c9f from cache: message_count=13, roles=['user', 'assistant', 'tool', 'assistant', 'tool', 'assistant', 'tool', 'tool', 'assistant', 'tool', 'tool', 'user', 'user']
2026-02-21 20:37:54,357 WARNING Service communication: Retry attempt 1 for '_call_method_async': HTTPServerError: HTTP 500: Server error '500 Internal Server Error' for url 'http://localhost:8005/get_chat_session_message_count'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500
2026-02-21 20:37:56,312 WARNING Service communication: Retry attempt 2 for '_call_method_async': HTTPServerError: HTTP 500: Server error '500 Internal Server Error' for url 'http://localhost:8005/get_chat_session_message_count'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500
2026-02-21 20:37:58,224 INFO 127.0.0.1:58917 - "GET /api/store/profile HTTP/1.1" 404
2026-02-21 20:37:58,928 WARNING Service communication: Retry attempt 3 for '_call_method_async': HTTPServerError: HTTP 500: Server error '500 Internal Server Error' for url 'http://localhost:8005/get_chat_session_message_count'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500
2026-02-21 20:38:00,041 WARNING Service communication: Retry attempt 8 for '_call_method_sync': HTTPServerError: HTTP 500: Server error '500 Internal Server Error' for url 'http://localhost:8005/get_graph_executions'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500
2026-02-21 20:38:03,701 WARNING Service communication: Retry attempt 4 for '_call_method_async': HTTPServerError: HTTP 500: Server error '500 Internal Server Error' for url 'http://localhost:8005/get_chat_session_message_count'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500
2026-02-21 20:38:06,882 INFO Loading session 322af5c3-70fc-4a06-9443-8c5df0aa0c9f from cache: message_count=13, roles=['user', 'assistant', 'tool', 'assistant', 'tool', 'assistant', 'tool', 'tool', 'assistant', 'tool', 'tool', 'user', 'user']
2026-02-21 20:38:06,888 INFO [TASK_LOOKUP] Found running task 6360d249... for session 322af5c3...
2026-02-21 20:38:06,898 INFO [CoPilotExecutor] Received cancel for 6360d249-c803-47d3-8a08-d77275e4b2d8
2026-02-21 20:38:06,898 INFO [CANCEL] Published cancel for task ...75e4b2d8 session ...f0aa0c9f
2026-02-21 20:38:06,919 INFO Loading session 322af5c3-70fc-4a06-9443-8c5df0aa0c9f from cache: message_count=13, roles=['user', 'assistant', 'tool', 'assistant', 'tool', 'assistant', 'tool', 'tool', 'assistant', 'tool', 'tool', 'user', 'user']
2026-02-21 20:38:06,925 INFO [TASK_LOOKUP] Found running task 6360d249... for session 322af5c3...
2026-02-21 20:38:06,926 INFO [GET_SESSION] session=322af5c3-70fc-4a06-9443-8c5df0aa0c9f, active_task=True, msg_count=13, last_role=user
2026-02-21 20:38:06,927 INFO 127.0.0.1:58976 - "GET /api/chat/sessions/322af5c3-70fc-4a06-9443-8c5df0aa0c9f HTTP/1.1" 200
2026-02-21 20:38:07,136 INFO [TASK_LOOKUP] Found running task 6360d249... for session 322af5c3...
2026-02-21 20:38:07,138 INFO [TIMING] subscribe_to_task STARTED, task=6360d249-c803-47d3-8a08-d77275e4b2d8, user=68383665-d3d9-41f3-b10c-fca0dc6080ed, last_msg=0-0
2026-02-21 20:38:07,140 INFO [TIMING] Redis hgetall took 1.3ms
2026-02-21 20:38:07,359 INFO [CoPilotExecutor|task_id:6360d249-c803-47d3-8a08-d77275e4b2d8|session_id:322af5c3-70fc-4a06-9443-8c5df0aa0c9f|user_id:68383665-d3d9-41f3-b10c-fca0dc6080ed] Cancellation requested
2026-02-21 20:38:07,360 INFO [CoPilotExecutor|task_id:6360d249-c803-47d3-8a08-d77275e4b2d8|session_id:322af5c3-70fc-4a06-9443-8c5df0aa0c9f|user_id:68383665-d3d9-41f3-b10c-fca0dc6080ed] Execution completed in 13.09s
2026-02-21 20:38:07,360 INFO [CoPilotExecutor] Run completed for 6360d249-c803-47d3-8a08-d77275e4b2d8
2026-02-21 20:38:07,360 INFO [CoPilotExecutor|task_id:6360d249-c803-47d3-8a08-d77275e4b2d8|session_id:322af5c3-70fc-4a06-9443-8c5df0aa0c9f|user_id:68383665-d3d9-41f3-b10c-fca0dc6080ed] Task cancelled
2026-02-21 20:38:07,360 INFO [CoPilotExecutor] Releasing cluster lock for 6360d249-c803-47d3-8a08-d77275e4b2d8
2026-02-21 20:38:07,362 INFO [CoPilotExecutor] Cleaned up completed task 6360d249-c803-47d3-8a08-d77275e4b2d8
2026-02-21 20:38:07,364 INFO [TIMING] Redis xread (replay) took 224.1ms, status=running
2026-02-21 20:38:07,364 INFO [TIMING] Replayed 1 messages, last_id=1771681087362-0
2026-02-21 20:38:07,365 INFO [TIMING] Task still running, starting _stream_listener
2026-02-21 20:38:07,365 INFO [TIMING] publish_chunk StreamFinish in 2.1ms (xadd=1.2ms)
2026-02-21 20:38:07,365 INFO [TIMING] subscribe_to_task COMPLETED in 226.8ms; task=6360d249-c803-47d3-8a08-d77275e4b2d8, n_messages_replayed=1
2026-02-21 20:38:07,366 INFO [TIMING] _stream_listener STARTED, task=6360d249-c803-47d3-8a08-d77275e4b2d8, last_id=1771681087362-0
2026-02-21 20:38:07,366 INFO Resume stream chunk
2026-02-21 20:38:07,366 INFO 127.0.0.1:58976 - "GET /api/chat/sessions/322af5c3-70fc-4a06-9443-8c5df0aa0c9f/stream HTTP/1.1" 200
2026-02-21 20:38:07,367 INFO [TIMING] Redis xread (replay) took 13101.9ms, status=running
2026-02-21 20:38:07,367 INFO [TIMING] Replayed 1 messages, last_id=1771681087362-0
2026-02-21 20:38:07,367 INFO [TIMING] Task still running, starting _stream_listener
2026-02-21 20:38:07,367 INFO [TIMING] subscribe_to_task COMPLETED in 13104.6ms; task=6360d249-c803-47d3-8a08-d77275e4b2d8, n_messages_replayed=1
2026-02-21 20:38:07,367 INFO [TIMING] Starting to read from subscriber_queue
2026-02-21 20:38:07,368 INFO [TIMING] FIRST CHUNK from queue at 13.11s, type=StreamFinish
2026-02-21 20:38:07,368 INFO 127.0.0.1:58710 - "POST /api/chat/sessions/322af5c3-70fc-4a06-9443-8c5df0aa0c9f/stream HTTP/1.1" 200
2026-02-21 20:38:07,368 INFO [TIMING] StreamFinish received in 13.11s; n_chunks=1
2026-02-21 20:38:07,368 INFO [TIMING] _stream_listener CANCELLED after 2.7ms, delivered=0
2026-02-21 20:38:07,368 INFO [TIMING] _stream_listener FINISHED in 0.0s; task=6360d249-c803-47d3-8a08-d77275e4b2d8, delivered=0, xread_count=1
2026-02-21 20:38:07,369 INFO Resume stream completed
2026-02-21 20:38:07,369 INFO [TIMING] event_generator FINISHED in 13.11s; task=6360d249-c803-47d3-8a08-d77275e4b2d8, session=322af5c3-70fc-4a06-9443-8c5df0aa0c9f, n_chunks=1
2026-02-21 20:38:07,408 INFO [CANCEL] Task ...75e4b2d8 confirmed stopped (status=failed) after 0.5s
2026-02-21 20:38:07,409 INFO 127.0.0.1:58974 - "POST /api/chat/sessions/322af5c3-70fc-4a06-9443-8c5df0aa0c9f/cancel HTTP/1.1" 200
2026-02-21 20:38:07,447 INFO Loading session 322af5c3-70fc-4a06-9443-8c5df0aa0c9f from cache: message_count=13, roles=['user', 'assistant', 'tool', 'assistant', 'tool', 'assistant', 'tool', 'tool', 'assistant', 'tool', 'tool', 'user', 'user']
2026-02-21 20:38:07,451 INFO [GET_SESSION] session=322af5c3-70fc-4a06-9443-8c5df0aa0c9f, active_task=False, msg_count=13, last_role=user
2026-02-21 20:38:07,451 INFO 127.0.0.1:58710 - "GET /api/chat/sessions/322af5c3-70fc-4a06-9443-8c5df0aa0c9f HTTP/1.1" 200
2026-02-21 20:38:07,468 INFO 127.0.0.1:58710 - "GET /api/chat/sessions/322af5c3-70fc-4a06-9443-8c5df0aa0c9f/stream HTTP/1.1" 204
2026-02-21 20:38:07,521 INFO Loading session 322af5c3-70fc-4a06-9443-8c5df0aa0c9f from cache: message_count=13, roles=['user', 'assistant', 'tool', 'assistant', 'tool', 'assistant', 'tool', 'tool', 'assistant', 'tool', 'tool', 'user', 'user']
2026-02-21 20:38:07,527 INFO [GET_SESSION] session=322af5c3-70fc-4a06-9443-8c5df0aa0c9f, active_task=False, msg_count=13, last_role=user
2026-02-21 20:38:07,528 INFO 127.0.0.1:58710 - "GET /api/chat/sessions/322af5c3-70fc-4a06-9443-8c5df0aa0c9f HTTP/1.1" 200
2026-02-21 20:38:18,440 INFO 127.0.0.1:59077 - "GET /api/store/profile HTTP/1.1" 404
2026-02-21 20:38:19,553 INFO 127.0.0.1:59077 - "GET /api/store/profile HTTP/1.1" 404
2026-02-21 20:38:21,643 INFO 127.0.0.1:59077 - "GET /api/store/profile HTTP/1.1" 404
2026-02-21 20:38:30,090 WARNING Service communication: Retry attempt 9 for '_call_method_sync': HTTPServerError: HTTP 500: Server error '500 Internal Server Error' for url 'http://localhost:8005/get_graph_executions'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500
2026-02-21 20:39:00,123 WARNING Service communication: Retry attempt 10 for '_call_method_sync': HTTPServerError: HTTP 500: Server error '500 Internal Server Error' for url 'http://localhost:8005/get_graph_executions'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500
2026-02-21 20:39:13,881 INFO 127.0.0.1:59398 - "GET /api/chat/sessions?limit=50 HTTP/1.1" 200
2026-02-21 20:39:30,173 WARNING Service communication: Retry attempt 11 for '_call_method_sync': HTTPServerError: HTTP 500: Server error '500 Internal Server Error' for url 'http://localhost:8005/get_graph_executions'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500
2026-02-21 20:39:35,355 INFO 127.0.0.1:59522 - "GET /api/store/profile HTTP/1.1" 404
2026-02-21 20:39:35,685 INFO 127.0.0.1:59526 - "GET /api/executions HTTP/1.1" 200
2026-02-21 20:39:38,916 INFO 127.0.0.1:59522 - "GET /api/store/profile HTTP/1.1" 404
2026-02-21 20:39:40,019 INFO 127.0.0.1:59522 - "GET /api/store/profile HTTP/1.1" 404

View File

@@ -22,4 +22,3 @@ migrations/*/rollback*.sql
# Workspace files
workspaces/
sample.logs

View File

@@ -34,9 +34,6 @@ class ResponseType(str, Enum):
TOOL_INPUT_AVAILABLE = "tool-input-available"
TOOL_OUTPUT_AVAILABLE = "tool-output-available"
# Long-running tool notification (custom extension - uses AI SDK DataUIPart format)
LONG_RUNNING_START = "data-long-running-start"
# Other
ERROR = "error"
USAGE = "usage"
@@ -148,10 +145,6 @@ class StreamToolInputAvailable(StreamBaseResponse):
input: dict[str, Any] = Field(
default_factory=dict, description="Tool input arguments"
)
providerMetadata: dict[str, Any] | None = Field(
default=None,
description="Provider metadata - used to pass isLongRunning flag to frontend",
)
class StreamToolOutputAvailable(StreamBaseResponse):
@@ -180,20 +173,6 @@ class StreamToolOutputAvailable(StreamBaseResponse):
return f"data: {json.dumps(data)}\n\n"
class StreamLongRunningStart(StreamBaseResponse):
"""Notification that a long-running tool has started.
Custom extension using AI SDK DataUIPart format. Signals the frontend to show
UI feedback while the tool executes.
"""
type: ResponseType = ResponseType.LONG_RUNNING_START
data: dict[str, Any] = Field(
default_factory=dict,
description="Data for the long-running event containing toolCallId and toolName",
)
# ========== Other ==========

View File

@@ -34,7 +34,6 @@ from backend.copilot.response_model import (
StreamToolInputStart,
StreamToolOutputAvailable,
)
from backend.copilot.tools import get_tool
from .tool_adapter import MCP_TOOL_PREFIX, pop_pending_tool_output
@@ -112,15 +111,6 @@ class SDKResponseAdapter:
# instead of "mcp__copilot__find_block".
tool_name = block.name.removeprefix(MCP_TOOL_PREFIX)
# Check if this is a long-running tool to trigger UI feedback
tool = get_tool(tool_name)
is_long_running = tool.is_long_running if tool else False
logger.info(
f"[ADAPTER] Tool: {tool_name}, has_tool={tool is not None}, "
f"is_long_running={is_long_running}"
)
responses.append(
StreamToolInputStart(toolCallId=block.id, toolName=tool_name)
)
@@ -129,15 +119,8 @@ class SDKResponseAdapter:
toolCallId=block.id,
toolName=tool_name,
input=block.input,
providerMetadata=(
{"isLongRunning": True} if is_long_running else None
),
)
)
logger.info(
f"[ADAPTER] Created StreamToolInputAvailable with "
f"providerMetadata={{'isLongRunning': {is_long_running}}}"
)
self.current_tool_calls[block.id] = {"name": tool_name}
elif isinstance(sdk_message, UserMessage):

View File

@@ -11,6 +11,7 @@ from typing import Any
from backend.util.exceptions import NotFoundError
from .. import stream_registry
from ..config import ChatConfig
from ..model import (
ChatMessage,
@@ -30,7 +31,12 @@ from ..response_model import (
StreamToolInputAvailable,
StreamToolOutputAvailable,
)
from ..service import _build_system_prompt, _generate_session_title
from ..service import (
_build_system_prompt,
_execute_long_running_tool_with_streaming,
_generate_session_title,
)
from ..tools.models import OperationPendingResponse, OperationStartedResponse
from ..tools.sandbox import WORKSPACE_PREFIX, make_session_path
from ..tracking import track_user_message
from .response_adapter import SDKResponseAdapter
@@ -38,6 +44,7 @@ from .security_hooks import create_security_hooks
from .tool_adapter import (
COPILOT_TOOL_NAMES,
SDK_DISALLOWED_TOOLS,
LongRunningCallback,
create_copilot_mcp_server,
set_execution_context,
wait_for_stash,
@@ -116,9 +123,9 @@ When you create or modify important files (code, configs, outputs), you MUST:
are available from previous turns
### Long-running tools
Long-running tools (create_agent, edit_agent, etc.) run synchronously
with heartbeats to keep the connection alive. The frontend shows UI feedback
during execution based on stream events.
Long-running tools (create_agent, edit_agent, etc.) are handled
asynchronously. You will receive an immediate response; the actual result
is delivered to the user via a background stream.
### Sub-agent tasks
- When using the Task tool, NEVER set `run_in_background` to true.
@@ -126,6 +133,121 @@ during execution based on stream events.
"""
def _build_long_running_callback(user_id: str | None) -> LongRunningCallback:
"""Build a callback that delegates long-running tools to the non-SDK infrastructure.
Long-running tools (create_agent, edit_agent, etc.) are delegated to the
existing background infrastructure: stream_registry (Redis Streams),
database persistence, and SSE reconnection. This means results survive
page refreshes / pod restarts, and the frontend shows the proper loading
widget with progress updates.
The returned callback matches the ``LongRunningCallback`` signature:
``(tool_name, args, session) -> MCP response dict``.
"""
async def _callback(
tool_name: str, args: dict[str, Any], session: ChatSession
) -> dict[str, Any]:
operation_id = str(uuid.uuid4())
task_id = str(uuid.uuid4())
tool_call_id = f"sdk-{uuid.uuid4().hex[:12]}"
session_id = session.session_id
# --- Build user-friendly messages (matches non-SDK service) ---
if tool_name == "create_agent":
desc = args.get("description", "")
desc_preview = (desc[:100] + "...") if len(desc) > 100 else desc
pending_msg = (
f"Creating your agent: {desc_preview}"
if desc_preview
else "Creating agent... This may take a few minutes."
)
started_msg = (
"Agent creation started. You can close this tab - "
"check your library in a few minutes."
)
elif tool_name == "edit_agent":
changes = args.get("changes", "")
changes_preview = (changes[:100] + "...") if len(changes) > 100 else changes
pending_msg = (
f"Editing agent: {changes_preview}"
if changes_preview
else "Editing agent... This may take a few minutes."
)
started_msg = (
"Agent edit started. You can close this tab - "
"check your library in a few minutes."
)
else:
pending_msg = f"Running {tool_name}... This may take a few minutes."
started_msg = (
f"{tool_name} started. You can close this tab - "
"check back in a few minutes."
)
# --- Register task in Redis for SSE reconnection ---
await stream_registry.create_task(
task_id=task_id,
session_id=session_id,
user_id=user_id,
tool_call_id=tool_call_id,
tool_name=tool_name,
operation_id=operation_id,
)
# --- Save OperationPendingResponse to chat history ---
pending_message = ChatMessage(
role="tool",
content=OperationPendingResponse(
message=pending_msg,
operation_id=operation_id,
tool_name=tool_name,
).model_dump_json(),
tool_call_id=tool_call_id,
)
session.messages.append(pending_message)
await upsert_chat_session(session)
# --- Spawn background task (reuses non-SDK infrastructure) ---
bg_task = asyncio.create_task(
_execute_long_running_tool_with_streaming(
tool_name=tool_name,
parameters=args,
tool_call_id=tool_call_id,
operation_id=operation_id,
task_id=task_id,
session_id=session_id,
user_id=user_id,
)
)
_background_tasks.add(bg_task)
bg_task.add_done_callback(_background_tasks.discard)
await stream_registry.set_task_asyncio_task(task_id, bg_task)
logger.info(
f"[SDK] Long-running tool {tool_name} delegated to background "
f"(operation_id={operation_id}, task_id={task_id})"
)
# --- Return OperationStartedResponse as MCP tool result ---
# This flows through SDK → response adapter → frontend, triggering
# the loading widget with SSE reconnection support.
started_json = OperationStartedResponse(
message=started_msg,
operation_id=operation_id,
tool_name=tool_name,
task_id=task_id,
).model_dump_json()
return {
"content": [{"type": "text", "text": started_json}],
"isError": False,
}
return _callback
def _resolve_sdk_model() -> str | None:
"""Resolve the model name for the Claude Agent SDK CLI.
@@ -462,7 +584,7 @@ async def stream_chat_completion_sdk(
set_execution_context(
user_id,
session,
long_running_callback=None,
long_running_callback=_build_long_running_callback(user_id),
)
try:
from claude_agent_sdk import ClaudeAgentOptions, ClaudeSDKClient

View File

@@ -52,7 +52,6 @@ from .response_model import (
StreamFinish,
StreamFinishStep,
StreamHeartbeat,
StreamLongRunningStart,
StreamStart,
StreamStartStep,
StreamTextDelta,
@@ -64,7 +63,12 @@ from .response_model import (
StreamUsage,
)
from .tools import execute_tool, get_tool, tools
from .tools.models import ErrorResponse
from .tools.models import (
ErrorResponse,
OperationInProgressResponse,
OperationPendingResponse,
OperationStartedResponse,
)
from .tracking import track_user_message
logger = logging.getLogger(__name__)
@@ -1398,15 +1402,16 @@ async def _yield_tool_call(
"""
Yield a tool call and its execution result.
Executes tools synchronously and yields heartbeat events every 15 seconds to
keep the SSE connection alive during execution. The is_long_running property
is only used by the frontend to display UI feedback during long operations.
For tools marked with `is_long_running=True` (like agent generation), spawns a
background task so the operation survives SSE disconnections. For other tools,
yields heartbeat events every 15 seconds to keep the SSE connection alive.
Raises:
orjson.JSONDecodeError: If tool call arguments cannot be parsed as JSON
KeyError: If expected tool call fields are missing
TypeError: If tool call structure is invalid
"""
import uuid as uuid_module
tool_name = tool_calls[yield_idx]["function"]["name"]
tool_call_id = tool_calls[yield_idx]["id"]
@@ -1424,17 +1429,163 @@ async def _yield_tool_call(
input=arguments,
)
# Notify frontend if this is a long-running tool (e.g., agent generation)
# Check if this tool is long-running (survives SSE disconnection)
tool = get_tool(tool_name)
if tool and tool.is_long_running:
yield StreamLongRunningStart(
data={
"toolCallId": tool_call_id,
"toolName": tool_name,
}
)
# Atomic check-and-set: returns False if operation already running (lost race)
if not await _mark_operation_started(tool_call_id):
logger.info(
f"Tool call {tool_call_id} already in progress, returning status"
)
# Build dynamic message based on tool name
if tool_name == "create_agent":
in_progress_msg = "Agent creation already in progress. Please wait..."
elif tool_name == "edit_agent":
in_progress_msg = "Agent edit already in progress. Please wait..."
else:
in_progress_msg = f"{tool_name} already in progress. Please wait..."
# Run tool execution synchronously with heartbeats
yield StreamToolOutputAvailable(
toolCallId=tool_call_id,
toolName=tool_name,
output=OperationInProgressResponse(
message=in_progress_msg,
tool_call_id=tool_call_id,
).model_dump_json(),
success=True,
)
return
# Generate operation ID and task ID
operation_id = str(uuid_module.uuid4())
task_id = str(uuid_module.uuid4())
# Build a user-friendly message based on tool and arguments
if tool_name == "create_agent":
agent_desc = arguments.get("description", "")
# Truncate long descriptions for the message
desc_preview = (
(agent_desc[:100] + "...") if len(agent_desc) > 100 else agent_desc
)
pending_msg = (
f"Creating your agent: {desc_preview}"
if desc_preview
else "Creating agent... This may take a few minutes."
)
started_msg = (
"Agent creation started. You can close this tab - "
"check your library in a few minutes."
)
elif tool_name == "edit_agent":
changes = arguments.get("changes", "")
changes_preview = (changes[:100] + "...") if len(changes) > 100 else changes
pending_msg = (
f"Editing agent: {changes_preview}"
if changes_preview
else "Editing agent... This may take a few minutes."
)
started_msg = (
"Agent edit started. You can close this tab - "
"check your library in a few minutes."
)
else:
pending_msg = f"Running {tool_name}... This may take a few minutes."
started_msg = (
f"{tool_name} started. You can close this tab - "
"check back in a few minutes."
)
# Track appended message for rollback on failure
pending_message: ChatMessage | None = None
# Wrap session save and task creation in try-except to release lock on failure
try:
# Create task in stream registry for SSE reconnection support
await stream_registry.create_task(
task_id=task_id,
session_id=session.session_id,
user_id=session.user_id,
tool_call_id=tool_call_id,
tool_name=tool_name,
operation_id=operation_id,
)
# Attach tool_call and save pending result — lock serialises
# concurrent session mutations during parallel execution.
async def _save_pending() -> None:
nonlocal pending_message
session.add_tool_call_to_current_turn(tool_calls[yield_idx])
pending_message = ChatMessage(
role="tool",
content=OperationPendingResponse(
message=pending_msg,
operation_id=operation_id,
tool_name=tool_name,
).model_dump_json(),
tool_call_id=tool_call_id,
)
session.messages.append(pending_message)
await upsert_chat_session(session)
await _with_optional_lock(session_lock, _save_pending)
logger.info(
f"Saved pending operation {operation_id} (task_id={task_id}) "
f"for tool {tool_name} in session {session.session_id}"
)
# Store task reference in module-level set to prevent GC before completion
bg_task = asyncio.create_task(
_execute_long_running_tool_with_streaming(
tool_name=tool_name,
parameters=arguments,
tool_call_id=tool_call_id,
operation_id=operation_id,
task_id=task_id,
session_id=session.session_id,
user_id=session.user_id,
)
)
_background_tasks.add(bg_task)
bg_task.add_done_callback(_background_tasks.discard)
# Associate the asyncio task with the stream registry task
await stream_registry.set_task_asyncio_task(task_id, bg_task)
except Exception as e:
# Roll back appended messages — use identity-based removal so
# it works even when other parallel tools have appended after us.
async def _rollback() -> None:
if pending_message and pending_message in session.messages:
session.messages.remove(pending_message)
await _with_optional_lock(session_lock, _rollback)
# Release the Redis lock since the background task won't be spawned
await _mark_operation_completed(tool_call_id)
# Mark stream registry task as failed if it was created
try:
await stream_registry.mark_task_completed(task_id, status="failed")
except Exception as mark_err:
logger.warning(f"Failed to mark task {task_id} as failed: {mark_err}")
logger.error(
f"Failed to setup long-running tool {tool_name}: {e}", exc_info=True
)
raise
# Return immediately - don't wait for completion
yield StreamToolOutputAvailable(
toolCallId=tool_call_id,
toolName=tool_name,
output=OperationStartedResponse(
message=started_msg,
operation_id=operation_id,
tool_name=tool_name,
task_id=task_id, # Include task_id for SSE reconnection
).model_dump_json(),
success=True,
)
return
# Normal flow: Run tool execution in background task with heartbeats
tool_task = asyncio.create_task(
execute_tool(
tool_name=tool_name,

View File

@@ -540,15 +540,21 @@ async def decompose_goal(
async def generate_agent(
instructions: DecompositionResult | dict[str, Any],
library_agents: list[AgentSummary] | list[dict[str, Any]] | None = None,
operation_id: str | None = None,
task_id: str | None = None,
) -> dict[str, Any] | None:
"""Generate agent JSON from instructions.
Args:
instructions: Structured instructions from decompose_goal
library_agents: User's library agents available for sub-agent composition
operation_id: Operation ID for async processing (enables Redis Streams
completion notification)
task_id: Task ID for async processing (enables Redis Streams persistence
and SSE delivery)
Returns:
Agent JSON dict, error dict {"type": "error", ...}, or None on error
Agent JSON dict, {"status": "accepted"} for async, error dict {"type": "error", ...}, or None on error
Raises:
AgentGeneratorNotConfiguredError: If the external service is not configured.
@@ -556,9 +562,13 @@ async def generate_agent(
_check_service_configured()
logger.info("Calling external Agent Generator service for generate_agent")
result = await generate_agent_external(
dict(instructions), _to_dict_list(library_agents)
dict(instructions), _to_dict_list(library_agents), operation_id, task_id
)
# Don't modify async response
if result and result.get("status") == "accepted":
return result
if result:
if isinstance(result, dict) and result.get("type") == "error":
return result
@@ -749,6 +759,8 @@ async def generate_agent_patch(
update_request: str,
current_agent: dict[str, Any],
library_agents: list[AgentSummary] | None = None,
operation_id: str | None = None,
task_id: str | None = None,
) -> dict[str, Any] | None:
"""Update an existing agent using natural language.
@@ -761,10 +773,12 @@ async def generate_agent_patch(
update_request: Natural language description of changes
current_agent: Current agent JSON
library_agents: User's library agents available for sub-agent composition
operation_id: Operation ID for async processing (enables Redis Streams callback)
task_id: Task ID for async processing (enables Redis Streams callback)
Returns:
Updated agent JSON, clarifying questions dict {"type": "clarifying_questions", ...},
error dict {"type": "error", ...}, or None on error
{"status": "accepted"} for async, error dict {"type": "error", ...}, or None on error
Raises:
AgentGeneratorNotConfiguredError: If the external service is not configured.
@@ -775,6 +789,8 @@ async def generate_agent_patch(
update_request,
current_agent,
_to_dict_list(library_agents),
operation_id,
task_id,
)

View File

@@ -101,6 +101,8 @@ async def decompose_goal_dummy(
async def generate_agent_dummy(
instructions: dict[str, Any],
library_agents: list[dict[str, Any]] | None = None,
operation_id: str | None = None,
task_id: str | None = None,
) -> dict[str, Any]:
"""Return dummy agent JSON after a simulated delay."""
logger.info("Using dummy agent generator for generate_agent (30s delay)")
@@ -112,6 +114,8 @@ async def generate_agent_patch_dummy(
update_request: str,
current_agent: dict[str, Any],
library_agents: list[dict[str, Any]] | None = None,
operation_id: str | None = None,
task_id: str | None = None,
) -> dict[str, Any]:
"""Return dummy patched agent (returns the current agent with updated description)."""
logger.info("Using dummy agent generator for generate_agent_patch")

View File

@@ -242,18 +242,24 @@ async def decompose_goal_external(
async def generate_agent_external(
instructions: dict[str, Any],
library_agents: list[dict[str, Any]] | None = None,
operation_id: str | None = None,
task_id: str | None = None,
) -> dict[str, Any] | None:
"""Call the external service to generate an agent from instructions.
Args:
instructions: Structured instructions from decompose_goal
library_agents: User's library agents available for sub-agent composition
operation_id: Operation ID for async processing (enables Redis Streams callback)
task_id: Task ID for async processing (enables Redis Streams callback)
Returns:
Agent JSON dict or error dict {"type": "error", ...} on error
Agent JSON dict, {"status": "accepted"} for async, or error dict {"type": "error", ...} on error
"""
if _is_dummy_mode():
return await generate_agent_dummy(instructions, library_agents)
return await generate_agent_dummy(
instructions, library_agents, operation_id, task_id
)
client = _get_client()
@@ -261,9 +267,25 @@ async def generate_agent_external(
payload: dict[str, Any] = {"instructions": instructions}
if library_agents:
payload["library_agents"] = library_agents
if operation_id and task_id:
payload["operation_id"] = operation_id
payload["task_id"] = task_id
try:
response = await client.post("/api/generate-agent", json=payload)
# Handle 202 Accepted for async processing
if response.status_code == 202:
logger.info(
f"Agent Generator accepted async request "
f"(operation_id={operation_id}, task_id={task_id})"
)
return {
"status": "accepted",
"operation_id": operation_id,
"task_id": task_id,
}
response.raise_for_status()
data = response.json()
@@ -295,6 +317,8 @@ async def generate_agent_patch_external(
update_request: str,
current_agent: dict[str, Any],
library_agents: list[dict[str, Any]] | None = None,
operation_id: str | None = None,
task_id: str | None = None,
) -> dict[str, Any] | None:
"""Call the external service to generate a patch for an existing agent.
@@ -302,13 +326,15 @@ async def generate_agent_patch_external(
update_request: Natural language description of changes
current_agent: Current agent JSON
library_agents: User's library agents available for sub-agent composition
operation_id: Operation ID for async processing (enables Redis Streams callback)
task_id: Task ID for async processing (enables Redis Streams callback)
Returns:
Updated agent JSON, clarifying questions dict, or error dict on error
Updated agent JSON, clarifying questions dict, {"status": "accepted"} for async, or error dict on error
"""
if _is_dummy_mode():
return await generate_agent_patch_dummy(
update_request, current_agent, library_agents
update_request, current_agent, library_agents, operation_id, task_id
)
client = _get_client()
@@ -320,9 +346,25 @@ async def generate_agent_patch_external(
}
if library_agents:
payload["library_agents"] = library_agents
if operation_id and task_id:
payload["operation_id"] = operation_id
payload["task_id"] = task_id
try:
response = await client.post("/api/update-agent", json=payload)
# Handle 202 Accepted for async processing
if response.status_code == 202:
logger.info(
f"Agent Generator accepted async update request "
f"(operation_id={operation_id}, task_id={task_id})"
)
return {
"status": "accepted",
"operation_id": operation_id,
"task_id": task_id,
}
response.raise_for_status()
data = response.json()

View File

@@ -38,7 +38,12 @@ class BaseTool:
@property
def is_long_running(self) -> bool:
"""Whether this tool takes a long time to execute (triggers long-running UI)."""
"""Whether this tool is long-running and should execute in background.
Long-running tools (like agent generation) are executed via background
tasks to survive SSE disconnections. The result is persisted to chat
history and visible when the user refreshes.
"""
return False
def as_openai_tool(self) -> ChatCompletionToolParam:

View File

@@ -18,6 +18,7 @@ from .base import BaseTool
from .models import (
AgentPreviewResponse,
AgentSavedResponse,
AsyncProcessingResponse,
ClarificationNeededResponse,
ClarifyingQuestion,
ErrorResponse,
@@ -48,7 +49,6 @@ class CreateAgentTool(BaseTool):
@property
def is_long_running(self) -> bool:
"""Agent generation takes several minutes."""
return True
@property
@@ -100,6 +100,10 @@ class CreateAgentTool(BaseTool):
save = kwargs.get("save", True)
session_id = session.session_id if session else None
# Extract async processing params (passed by long-running tool handler)
operation_id = kwargs.get("_operation_id")
task_id = kwargs.get("_task_id")
if not description:
return ErrorResponse(
message="Please provide a description of what the agent should do.",
@@ -226,6 +230,8 @@ class CreateAgentTool(BaseTool):
agent_json = await generate_agent(
decomposition_result,
library_agents,
operation_id=operation_id,
task_id=task_id,
)
except AgentGeneratorNotConfiguredError:
return ErrorResponse(
@@ -270,6 +276,19 @@ class CreateAgentTool(BaseTool):
session_id=session_id,
)
# Check if Agent Generator accepted for async processing
if agent_json.get("status") == "accepted":
logger.info(
f"Agent generation delegated to async processing "
f"(operation_id={operation_id}, task_id={task_id})"
)
return AsyncProcessingResponse(
message="Agent generation started. You'll be notified when it's complete.",
operation_id=operation_id,
task_id=task_id,
session_id=session_id,
)
agent_name = agent_json.get("name", "Generated Agent")
agent_description = agent_json.get("description", "")
node_count = len(agent_json.get("nodes", []))

View File

@@ -48,7 +48,6 @@ class CustomizeAgentTool(BaseTool):
@property
def is_long_running(self) -> bool:
"""Agent customization takes several minutes."""
return True
@property

View File

@@ -17,6 +17,7 @@ from .base import BaseTool
from .models import (
AgentPreviewResponse,
AgentSavedResponse,
AsyncProcessingResponse,
ClarificationNeededResponse,
ClarifyingQuestion,
ErrorResponse,
@@ -46,7 +47,6 @@ class EditAgentTool(BaseTool):
@property
def is_long_running(self) -> bool:
"""Agent editing takes several minutes."""
return True
@property
@@ -105,6 +105,10 @@ class EditAgentTool(BaseTool):
save = kwargs.get("save", True)
session_id = session.session_id if session else None
# Extract async processing params (passed by long-running tool handler)
operation_id = kwargs.get("_operation_id")
task_id = kwargs.get("_task_id")
if not agent_id:
return ErrorResponse(
message="Please provide the agent ID to edit.",
@@ -153,6 +157,8 @@ class EditAgentTool(BaseTool):
update_request,
current_agent,
library_agents,
operation_id=operation_id,
task_id=task_id,
)
except AgentGeneratorNotConfiguredError:
return ErrorResponse(
@@ -172,6 +178,19 @@ class EditAgentTool(BaseTool):
session_id=session_id,
)
# Check if Agent Generator accepted for async processing
if result.get("status") == "accepted":
logger.info(
f"Agent edit delegated to async processing "
f"(operation_id={operation_id}, task_id={task_id})"
)
return AsyncProcessingResponse(
message="Agent edit started. You'll be notified when it's complete.",
operation_id=operation_id,
task_id=task_id,
session_id=session_id,
)
# Check if the result is an error from the external service
if isinstance(result, dict) and result.get("type") == "error":
error_msg = result.get("error", "Unknown error")

View File

@@ -459,6 +459,23 @@ class OperationInProgressResponse(ToolResponseBase):
tool_call_id: str
class AsyncProcessingResponse(ToolResponseBase):
"""Response when an operation has been delegated to async processing.
This is returned by tools when the external service accepts the request
for async processing (HTTP 202 Accepted). The Redis Streams completion
consumer will handle the result when the external service completes.
The status field is specifically "accepted" to allow the long-running tool
handler to detect this response and skip LLM continuation.
"""
type: ResponseType = ResponseType.OPERATION_STARTED
status: str = "accepted" # Must be "accepted" for detection
operation_id: str | None = None
task_id: str | None = None
class WebFetchResponse(ToolResponseBase):
"""Response for web_fetch tool."""

View File

@@ -109,7 +109,7 @@ class TestGenerateAgent:
instructions = {"type": "instructions", "steps": ["Step 1"]}
result = await core.generate_agent(instructions)
mock_external.assert_called_once_with(instructions, None)
mock_external.assert_called_once_with(instructions, None, None, None)
assert result is not None
assert result["name"] == "Test Agent"
assert "id" in result
@@ -173,7 +173,9 @@ class TestGenerateAgentPatch:
current_agent = {"nodes": [], "links": []}
result = await core.generate_agent_patch("Add a node", current_agent)
mock_external.assert_called_once_with("Add a node", current_agent, None)
mock_external.assert_called_once_with(
"Add a node", current_agent, None, None, None
)
assert result == expected_result
@pytest.mark.asyncio

View File

@@ -34,8 +34,7 @@ export const ChatContainer = ({
onStop,
headerSlot,
}: ChatContainerProps) => {
const isBusy =
status === "streaming" || status === "submitted" || !!isReconnecting;
const isBusy = status === "streaming" || !!isReconnecting;
const inputLayoutId = "copilot-2-chat-input";
return (

View File

@@ -13,7 +13,6 @@ import { LoadingSpinner } from "@/components/atoms/LoadingSpinner/LoadingSpinner
import { toast } from "@/components/molecules/Toast/use-toast";
import { ToolUIPart, UIDataTypes, UIMessage, UITools } from "ai";
import { useEffect, useRef, useState } from "react";
import { ToolWrapper } from "../ToolWrapper/ToolWrapper";
import { CreateAgentTool } from "../../tools/CreateAgent/CreateAgent";
import { EditAgentTool } from "../../tools/EditAgent/EditAgent";
import {
@@ -209,110 +208,86 @@ export const ChatMessagesContainer = ({
);
case "tool-find_block":
return (
<ToolWrapper
<FindBlocksTool
key={`${message.id}-${i}`}
part={part as ToolUIPart}
>
<FindBlocksTool part={part as ToolUIPart} />
</ToolWrapper>
/>
);
case "tool-find_agent":
case "tool-find_library_agent":
return (
<ToolWrapper
<FindAgentsTool
key={`${message.id}-${i}`}
part={part as ToolUIPart}
>
<FindAgentsTool part={part as ToolUIPart} />
</ToolWrapper>
/>
);
case "tool-search_docs":
case "tool-get_doc_page":
return (
<ToolWrapper
<SearchDocsTool
key={`${message.id}-${i}`}
part={part as ToolUIPart}
>
<SearchDocsTool part={part as ToolUIPart} />
</ToolWrapper>
/>
);
case "tool-run_block":
return (
<ToolWrapper
<RunBlockTool
key={`${message.id}-${i}`}
part={part as ToolUIPart}
>
<RunBlockTool part={part as ToolUIPart} />
</ToolWrapper>
/>
);
case "tool-run_agent":
case "tool-schedule_agent":
return (
<ToolWrapper
<RunAgentTool
key={`${message.id}-${i}`}
part={part as ToolUIPart}
>
<RunAgentTool part={part as ToolUIPart} />
</ToolWrapper>
/>
);
case "tool-create_agent":
return (
<ToolWrapper
<CreateAgentTool
key={`${message.id}-${i}`}
part={part as ToolUIPart}
>
<CreateAgentTool part={part as ToolUIPart} />
</ToolWrapper>
/>
);
case "tool-edit_agent":
return (
<ToolWrapper
<EditAgentTool
key={`${message.id}-${i}`}
part={part as ToolUIPart}
>
<EditAgentTool part={part as ToolUIPart} />
</ToolWrapper>
/>
);
case "tool-view_agent_output":
return (
<ToolWrapper
<ViewAgentOutputTool
key={`${message.id}-${i}`}
part={part as ToolUIPart}
>
<ViewAgentOutputTool part={part as ToolUIPart} />
</ToolWrapper>
/>
);
case "tool-search_feature_requests":
return (
<ToolWrapper
<SearchFeatureRequestsTool
key={`${message.id}-${i}`}
part={part as ToolUIPart}
>
<SearchFeatureRequestsTool
part={part as ToolUIPart}
/>
</ToolWrapper>
/>
);
case "tool-create_feature_request":
return (
<ToolWrapper
<CreateFeatureRequestTool
key={`${message.id}-${i}`}
part={part as ToolUIPart}
>
<CreateFeatureRequestTool part={part as ToolUIPart} />
</ToolWrapper>
/>
);
default:
// Render a generic tool indicator for SDK built-in
// tools (Read, Glob, Grep, etc.) or any unrecognized tool
if (part.type.startsWith("tool-")) {
return (
<ToolWrapper
<GenericTool
key={`${message.id}-${i}`}
part={part as ToolUIPart}
>
<GenericTool part={part as ToolUIPart} />
</ToolWrapper>
/>
);
}
return null;

View File

@@ -1,32 +0,0 @@
import { PlusCircleIcon } from "@phosphor-icons/react";
import { ContentGrid, ContentHint } from "../ToolAccordion/AccordionContent";
import { ToolAccordion } from "../ToolAccordion/ToolAccordion";
import { MiniGame } from "../../tools/CreateAgent/components/MiniGame/MiniGame";
interface Props {
/** Whether the tool is currently streaming/executing */
isStreaming: boolean;
}
/**
* Displays UI feedback while a long-running tool executes.
* Automatically shown for tools marked as is_long_running=True in the backend.
*/
export function LongRunningToolDisplay({ isStreaming }: Props) {
if (!isStreaming) return null;
return (
<ToolAccordion
icon={<PlusCircleIcon size={32} weight="light" />}
title="This may take a few minutes. Play while you wait."
defaultExpanded={true}
>
<ContentGrid>
<MiniGame />
<ContentHint>
This could take a few minutes play while you wait!
</ContentHint>
</ContentGrid>
</ToolAccordion>
);
}

View File

@@ -1,52 +0,0 @@
import type { ToolUIPart } from "ai";
import { LongRunningToolDisplay } from "../LongRunningToolDisplay/LongRunningToolDisplay";
interface Props {
part: ToolUIPart;
children: React.ReactNode;
}
/**
* Wrapper for all tool components. Automatically shows UI feedback
* for long-running tools by detecting the isLongRunning flag on the tool part.
*/
export function ToolWrapper({ part, children }: Props) {
const isStreaming =
part.state === "input-streaming" || part.state === "input-available";
// Extract tool name from type (format: "tool-{name}")
const toolName = part.type.startsWith("tool-")
? part.type.substring(5)
: "unknown";
// Check if this tool is marked as long-running via providerMetadata
const isLongRunning =
"providerMetadata" in part &&
part.providerMetadata &&
typeof part.providerMetadata === "object" &&
"isLongRunning" in part.providerMetadata &&
part.providerMetadata.isLongRunning === true;
// Debug logging
if (part.type.startsWith("tool-")) {
console.log("[ToolWrapper]", {
toolName,
type: part.type,
hasProviderMetadata: "providerMetadata" in part,
providerMetadata:
"providerMetadata" in part ? part.providerMetadata : undefined,
isLongRunning,
state: part.state,
isStreaming,
});
}
return (
<>
{/* Show UI feedback if tool is long-running and streaming */}
{isLongRunning && <LongRunningToolDisplay isStreaming={isStreaming} />}
{/* Render the actual tool component */}
{children}
</>
);
}

View File

@@ -16,6 +16,7 @@ import {
ContentCardDescription,
ContentCodeBlock,
ContentGrid,
ContentHint,
ContentMessage,
} from "../../components/ToolAccordion/AccordionContent";
import { ToolAccordion } from "../../components/ToolAccordion/ToolAccordion";
@@ -24,6 +25,7 @@ import {
ClarifyingQuestion,
} from "./components/ClarificationQuestionsCard";
import sparklesImg from "./components/MiniGame/assets/sparkles.png";
import { MiniGame } from "./components/MiniGame/MiniGame";
import { SuggestedGoalCard } from "./components/SuggestedGoalCard";
import {
AccordionIcon,
@@ -34,6 +36,9 @@ import {
isAgentSavedOutput,
isClarificationNeededOutput,
isErrorOutput,
isOperationInProgressOutput,
isOperationPendingOutput,
isOperationStartedOutput,
isSuggestedGoalOutput,
ToolIcon,
truncateText,
@@ -81,6 +86,18 @@ function getAccordionMeta(output: CreateAgentToolOutput) {
expanded: true,
};
}
if (
isOperationStartedOutput(output) ||
isOperationPendingOutput(output) ||
isOperationInProgressOutput(output)
) {
return {
icon,
title:
"Creating agent, this may take a few minutes. Play while you wait.",
expanded: true,
};
}
return {
icon: (
<WarningDiamondIcon size={32} weight="light" className="text-red-500" />
@@ -102,10 +119,19 @@ export function CreateAgentTool({ part }: Props) {
const isError =
part.state === "output-error" || (!!output && isErrorOutput(output));
const isOperating =
!!output &&
(isOperationStartedOutput(output) ||
isOperationPendingOutput(output) ||
isOperationInProgressOutput(output));
const hasExpandableContent =
part.state === "output-available" &&
!!output &&
(isAgentPreviewOutput(output) ||
(isOperationStartedOutput(output) ||
isOperationPendingOutput(output) ||
isOperationInProgressOutput(output) ||
isAgentPreviewOutput(output) ||
isAgentSavedOutput(output) ||
isClarificationNeededOutput(output) ||
isSuggestedGoalOutput(output) ||
@@ -145,6 +171,15 @@ export function CreateAgentTool({ part }: Props) {
{hasExpandableContent && output && (
<ToolAccordion {...getAccordionMeta(output)}>
{isOperating && (
<ContentGrid>
<MiniGame />
<ContentHint>
This could take a few minutes play while you wait!
</ContentHint>
</ContentGrid>
)}
{isAgentSavedOutput(output) && (
<div className="rounded-xl border border-border/60 bg-card p-4 shadow-sm">
<div className="flex items-baseline gap-2">

View File

@@ -2,6 +2,9 @@ import type { AgentPreviewResponse } from "@/app/api/__generated__/models/agentP
import type { AgentSavedResponse } from "@/app/api/__generated__/models/agentSavedResponse";
import type { ClarificationNeededResponse } from "@/app/api/__generated__/models/clarificationNeededResponse";
import type { ErrorResponse } from "@/app/api/__generated__/models/errorResponse";
import type { OperationInProgressResponse } from "@/app/api/__generated__/models/operationInProgressResponse";
import type { OperationPendingResponse } from "@/app/api/__generated__/models/operationPendingResponse";
import type { OperationStartedResponse } from "@/app/api/__generated__/models/operationStartedResponse";
import { ResponseType } from "@/app/api/__generated__/models/responseType";
import type { SuggestedGoalResponse } from "@/app/api/__generated__/models/suggestedGoalResponse";
import {
@@ -13,6 +16,9 @@ import type { ToolUIPart } from "ai";
import { OrbitLoader } from "../../components/OrbitLoader/OrbitLoader";
export type CreateAgentToolOutput =
| OperationStartedResponse
| OperationPendingResponse
| OperationInProgressResponse
| AgentPreviewResponse
| AgentSavedResponse
| ClarificationNeededResponse
@@ -33,6 +39,9 @@ function parseOutput(output: unknown): CreateAgentToolOutput | null {
if (typeof output === "object") {
const type = (output as { type?: unknown }).type;
if (
type === ResponseType.operation_started ||
type === ResponseType.operation_pending ||
type === ResponseType.operation_in_progress ||
type === ResponseType.agent_preview ||
type === ResponseType.agent_saved ||
type === ResponseType.clarification_needed ||
@@ -41,6 +50,9 @@ function parseOutput(output: unknown): CreateAgentToolOutput | null {
) {
return output as CreateAgentToolOutput;
}
if ("operation_id" in output && "tool_name" in output)
return output as OperationStartedResponse | OperationPendingResponse;
if ("tool_call_id" in output) return output as OperationInProgressResponse;
if ("agent_json" in output && "agent_name" in output)
return output as AgentPreviewResponse;
if ("agent_id" in output && "library_agent_id" in output)
@@ -60,6 +72,30 @@ export function getCreateAgentToolOutput(
return parseOutput((part as { output?: unknown }).output);
}
export function isOperationStartedOutput(
output: CreateAgentToolOutput,
): output is OperationStartedResponse {
return (
output.type === ResponseType.operation_started ||
("operation_id" in output && "tool_name" in output)
);
}
export function isOperationPendingOutput(
output: CreateAgentToolOutput,
): output is OperationPendingResponse {
return output.type === ResponseType.operation_pending;
}
export function isOperationInProgressOutput(
output: CreateAgentToolOutput,
): output is OperationInProgressResponse {
return (
output.type === ResponseType.operation_in_progress ||
"tool_call_id" in output
);
}
export function isAgentPreviewOutput(
output: CreateAgentToolOutput,
): output is AgentPreviewResponse {
@@ -108,6 +144,10 @@ export function getAnimationText(part: {
case "output-available": {
const output = parseOutput(part.output);
if (!output) return "Creating a new agent";
if (isOperationStartedOutput(output)) return "Agent creation started";
if (isOperationPendingOutput(output)) return "Agent creation in progress";
if (isOperationInProgressOutput(output))
return "Agent creation already in progress";
if (isAgentSavedOutput(output)) return `Saved ${output.agent_name}`;
if (isAgentPreviewOutput(output)) return `Preview "${output.agent_name}"`;
if (isClarificationNeededOutput(output)) return "Needs clarification";

View File

@@ -4,14 +4,17 @@ import { WarningDiamondIcon } from "@phosphor-icons/react";
import type { ToolUIPart } from "ai";
import { useCopilotChatActions } from "../../components/CopilotChatActionsProvider/useCopilotChatActions";
import { MorphingTextAnimation } from "../../components/MorphingTextAnimation/MorphingTextAnimation";
import { OrbitLoader } from "../../components/OrbitLoader/OrbitLoader";
import {
ContentCardDescription,
ContentCodeBlock,
ContentGrid,
ContentHint,
ContentLink,
ContentMessage,
} from "../../components/ToolAccordion/AccordionContent";
import { ToolAccordion } from "../../components/ToolAccordion/ToolAccordion";
import { MiniGame } from "../CreateAgent/components/MiniGame/MiniGame";
import {
ClarificationQuestionsCard,
ClarifyingQuestion,
@@ -25,6 +28,9 @@ import {
isAgentSavedOutput,
isClarificationNeededOutput,
isErrorOutput,
isOperationInProgressOutput,
isOperationPendingOutput,
isOperationStartedOutput,
ToolIcon,
truncateText,
type EditAgentToolOutput,
@@ -69,6 +75,17 @@ function getAccordionMeta(output: EditAgentToolOutput): {
description: `${questions.length} question${questions.length === 1 ? "" : "s"}`,
};
}
if (
isOperationStartedOutput(output) ||
isOperationPendingOutput(output) ||
isOperationInProgressOutput(output)
) {
return {
icon: <OrbitLoader size={32} />,
title: "Editing agent, this may take a few minutes. Play while you wait.",
expanded: true,
};
}
return {
icon: (
<WarningDiamondIcon size={32} weight="light" className="text-red-500" />
@@ -87,10 +104,18 @@ export function EditAgentTool({ part }: Props) {
const output = getEditAgentToolOutput(part);
const isError =
part.state === "output-error" || (!!output && isErrorOutput(output));
const isOperating =
!!output &&
(isOperationStartedOutput(output) ||
isOperationPendingOutput(output) ||
isOperationInProgressOutput(output));
const hasExpandableContent =
part.state === "output-available" &&
!!output &&
(isAgentPreviewOutput(output) ||
(isOperationStartedOutput(output) ||
isOperationPendingOutput(output) ||
isOperationInProgressOutput(output) ||
isAgentPreviewOutput(output) ||
isAgentSavedOutput(output) ||
isClarificationNeededOutput(output) ||
isErrorOutput(output));
@@ -125,6 +150,15 @@ export function EditAgentTool({ part }: Props) {
{hasExpandableContent && output && (
<ToolAccordion {...getAccordionMeta(output)}>
{isOperating && (
<ContentGrid>
<MiniGame />
<ContentHint>
This could take a few minutes play while you wait!
</ContentHint>
</ContentGrid>
)}
{isAgentSavedOutput(output) && (
<ContentGrid>
<ContentMessage>{output.message}</ContentMessage>

View File

@@ -2,6 +2,9 @@ import type { AgentPreviewResponse } from "@/app/api/__generated__/models/agentP
import type { AgentSavedResponse } from "@/app/api/__generated__/models/agentSavedResponse";
import type { ClarificationNeededResponse } from "@/app/api/__generated__/models/clarificationNeededResponse";
import type { ErrorResponse } from "@/app/api/__generated__/models/errorResponse";
import type { OperationInProgressResponse } from "@/app/api/__generated__/models/operationInProgressResponse";
import type { OperationPendingResponse } from "@/app/api/__generated__/models/operationPendingResponse";
import type { OperationStartedResponse } from "@/app/api/__generated__/models/operationStartedResponse";
import { ResponseType } from "@/app/api/__generated__/models/responseType";
import {
NotePencilIcon,
@@ -12,6 +15,9 @@ import type { ToolUIPart } from "ai";
import { OrbitLoader } from "../../components/OrbitLoader/OrbitLoader";
export type EditAgentToolOutput =
| OperationStartedResponse
| OperationPendingResponse
| OperationInProgressResponse
| AgentPreviewResponse
| AgentSavedResponse
| ClarificationNeededResponse
@@ -32,6 +38,8 @@ function parseOutput(output: unknown): EditAgentToolOutput | null {
const type = (output as { type?: unknown }).type;
if (
type === ResponseType.operation_started ||
type === ResponseType.operation_pending ||
type === ResponseType.operation_in_progress ||
type === ResponseType.agent_preview ||
type === ResponseType.agent_saved ||
type === ResponseType.clarification_needed ||
@@ -39,6 +47,9 @@ function parseOutput(output: unknown): EditAgentToolOutput | null {
) {
return output as EditAgentToolOutput;
}
if ("operation_id" in output && "tool_name" in output)
return output as OperationStartedResponse | OperationPendingResponse;
if ("tool_call_id" in output) return output as OperationInProgressResponse;
if ("agent_json" in output && "agent_name" in output)
return output as AgentPreviewResponse;
if ("agent_id" in output && "library_agent_id" in output)
@@ -57,6 +68,30 @@ export function getEditAgentToolOutput(
return parseOutput((part as { output?: unknown }).output);
}
export function isOperationStartedOutput(
output: EditAgentToolOutput,
): output is OperationStartedResponse {
return (
output.type === ResponseType.operation_started ||
("operation_id" in output && "tool_name" in output)
);
}
export function isOperationPendingOutput(
output: EditAgentToolOutput,
): output is OperationPendingResponse {
return output.type === ResponseType.operation_pending;
}
export function isOperationInProgressOutput(
output: EditAgentToolOutput,
): output is OperationInProgressResponse {
return (
output.type === ResponseType.operation_in_progress ||
"tool_call_id" in output
);
}
export function isAgentPreviewOutput(
output: EditAgentToolOutput,
): output is AgentPreviewResponse {
@@ -97,6 +132,10 @@ export function getAnimationText(part: {
case "output-available": {
const output = parseOutput(part.output);
if (!output) return "Editing the agent";
if (isOperationStartedOutput(output)) return "Agent update started";
if (isOperationPendingOutput(output)) return "Agent update in progress";
if (isOperationInProgressOutput(output))
return "Agent update already in progress";
if (isAgentSavedOutput(output)) return `Saved "${output.agent_name}"`;
if (isAgentPreviewOutput(output)) return `Preview "${output.agent_name}"`;
if (isClarificationNeededOutput(output)) return "Needs clarification";

View File

@@ -16,7 +16,7 @@ import { useCallback, useEffect, useMemo, useRef, useState } from "react";
import { useChatSession } from "./useChatSession";
import { useLongRunningToolPolling } from "./hooks/useLongRunningToolPolling";
const STREAM_START_TIMEOUT_MS = 30_000; // 30s to detect if backend is down/not responding
const STREAM_START_TIMEOUT_MS = 12_000;
/** Mark any in-progress tool parts as completed/errored so spinners stop. */
function resolveInProgressTools(
@@ -203,9 +203,8 @@ export function useCopilotPage() {
prevStatusRef.current = status;
const wasActive = prev === "streaming" || prev === "submitted";
const isReady = status === "ready";
// Only invalidate on successful completion, not on error to avoid infinite refetch loop
if (wasActive && isReady && sessionId) {
const isIdle = status === "ready" || status === "error";
if (wasActive && isIdle && sessionId) {
queryClient.invalidateQueries({
queryKey: getGetV2GetSessionQueryKey(sessionId),
});