Vidit Ostwal 1c4f44af80 Adding usage info in llm.py (#4172) ...

* Adding usage info everywhere

* Changing the check

* Changing the logic

* Adding tests

* Adding casellets

* Minor change

* Fixing testcase

* remove the duplicated test case, thanks to cursor

* Adding async test cases

* Updating test case

---------

Co-authored-by: Lorenze Jay <63378463+lorenzejay@users.noreply.github.com>

2026-01-07 10:42:27 -08:00

..

a2a

feat: add a2a update mechanisms (poll/stream/push) with handlers, config, and tests

2026-01-07 11:36:36 -05:00

agents

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

experimental/evaluation

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

hooks

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

knowledge

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

llms

feat: add streaming tool call events; fix provider id tracking; add tests and cassettes

2026-01-05 14:33:36 -05:00

memory

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

pipeline

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

telemetry

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

tools

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

tracing

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

utilities

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_after_crew_modification.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_after_kickoff_modification.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_agent_usage_metrics_are_captured_for_hierarchical_process.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_agents_do_not_get_delegation_tools_with_there_is_only_one_agent.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_anthropic_function_calling.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_anthropic_thinking_blocks_preserved_across_turns.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_anthropic_thinking.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_api_calls_throttling.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_before_crew_modification.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_before_crew_with_none_input.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_before_kickoff_callback.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_before_kickoff_modification.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_before_kickoff_with_none_input.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_before_kickoff_without_inputs.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_cache_hitting_between_agents.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_conditional_task_last_task_when_conditional_is_false.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_conditional_task_last_task_when_conditional_is_true.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_crew_creation.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_crew_does_not_interpolate_without_inputs.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_crew_function_calling_llm.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_crew_kickoff_streaming_usage_metrics.yaml

Release/v1.0.0 (#3618)

2025-10-20 14:10:19 -07:00

test_crew_kickoff_usage_metrics.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_crew_log_file_output.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_crew_output_file_end_to_end.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_crew_testing_function.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_crew_train_success.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_crew_verbose_output.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_crew_with_delegating_agents_should_not_override_agent_tools.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_crew_with_delegating_agents_should_not_override_task_tools.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_crew_with_delegating_agents.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_crew_with_failing_task_guardrails.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_crew_with_knowledge_sources_works_with_copy.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_custom_converter_cls.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_custom_llm_implementation.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_custom_llm_within_crew.yaml

Release/v1.0.0 (#3618)

2025-10-20 14:10:19 -07:00

test_deepseek_r1_with_open_router.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_delegation_is_not_enabled_if_there_are_only_one_agent.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_disabled_memory_using_contextual_memory.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_ensure_exchanged_messages_are_propagated_to_external_memory.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_gemini_models[gemini-gemini-2.0-flash-001].yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_gemini_models[gemini-gemini-2.0-flash-lite-001].yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_gemini_models[gemini-gemini-2.0-flash-thinking-exp-01-21].yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_gemini_models[gemini-gemini-3-pro-preview].yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_gemma3[gemini-gemma-3-27b-it].yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_gpt_4_1[gpt-4.1-mini-2025-04-14].yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_gpt_4_1[gpt-4.1-nano-2025-04-14].yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_gpt_4_1[gpt-4.1].yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_guardrail_emits_events.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_guardrail_when_an_error_occurs.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_handle_context_length_exceeds_limit.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_handle_streaming_tool_calls_no_available_functions.yaml

Release/v1.0.0 (#3618)

2025-10-20 14:10:19 -07:00

test_handle_streaming_tool_calls_no_tools.yaml

Release/v1.0.0 (#3618)

2025-10-20 14:10:19 -07:00

test_handle_streaming_tool_calls_with_error.yaml

Release/v1.0.0 (#3618)

2025-10-20 14:10:19 -07:00

test_handle_streaming_tool_calls.yaml

Release/v1.0.0 (#3618)

2025-10-20 14:10:19 -07:00

test_hierarchical_crew_creation_tasks_with_agents.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_hierarchical_crew_creation_tasks_with_async_execution.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_hierarchical_crew_creation_tasks_with_sync_last.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_hierarchical_process.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_hierarchical_verbose_false_manager_agent.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_hierarchical_verbose_manager_agent.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_increment_delegations_for_hierarchical_process.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_increment_delegations_for_sequential_process.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_increment_tool_errors.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_inject_date_custom_format.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_inject_date.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_json_property_without_output_json.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_kickoff_for_each_error_handling.yaml

Release/v1.0.0 (#3618)

2025-10-20 14:10:19 -07:00

test_kickoff_for_each_multiple_inputs.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_kickoff_for_each_single_input.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_litellm_auth_error_handling.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_llm_call_when_stop_is_unsupported_when_additional_drop_params_is_provided.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_llm_call_when_stop_is_unsupported.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_llm_call_with_error.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_llm_call_with_message_list.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_llm_call_with_string_input_and_callbacks.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_llm_call_with_string_input.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_llm_call_with_tool_and_message_list.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_llm_call_with_tool_and_string_input.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_llm_callback_replacement.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_llm_passes_additional_params.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_long_term_memory_with_memory_flag.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_manager_agent_delegating_to_all_agents.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_manager_agent_delegating_to_assigned_task_agent.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_manager_agent_with_tools_raises_exception.yaml

tests: cassettes, threading for flow tests

2025-10-21 13:48:21 -04:00

test_manager_agent.yaml

tests: cassettes, threading for flow tests

2025-10-21 13:48:21 -04:00

test_memory_events_are_emitted.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_multimodal_agent_describing_image_successfully.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_multimodal_agent_live_image_analysis.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_multiple_before_after_crew.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_multiple_before_after_kickoff.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_no_inject_date.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_o3_mini_reasoning_effort_high.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_o3_mini_reasoning_effort_low.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_o3_mini_reasoning_effort_medium.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_output_json_dict_hierarchical.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_output_json_dict_sequential.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_output_json_hierarchical.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_output_json_sequential.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_output_json_to_another_task.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_output_pydantic_hierarchical.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_output_pydantic_sequential.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_output_pydantic_to_another_task.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_replay_interpolates_inputs_properly.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_replay_setup_context.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_replay_with_context_set_to_nullable.yaml

tests: cassettes, threading for flow tests

2025-10-21 13:48:21 -04:00

test_replay_with_context.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_save_task_json_output.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_save_task_output.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_save_task_pydantic_output.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_sequential_async_task_execution_completion.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_sets_parent_flow_when_inside_flow.yaml

tests: cassettes, threading for flow tests

2025-10-21 13:48:21 -04:00

test_single_task_with_async_execution.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_task_execution_times.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_task_guardrail_process_output.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_task_interpolation_with_hyphens.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_task_output_includes_messages.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_task_tools_override_agent_tools.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_task_tools_preserve_code_execution_tools.yaml

tests: cassettes, threading for flow tests

2025-10-21 13:48:21 -04:00

test_task_with_max_execution_time_exceeded.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_task_with_max_execution_time.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_task_with_no_arguments.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_tools_with_custom_caching.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_usage_info_non_streaming_with_acall.yaml

Adding usage info in llm.py (#4172)

2026-01-07 10:42:27 -08:00

test_usage_info_non_streaming_with_call.yaml

Adding usage info in llm.py (#4172)

2026-01-07 10:42:27 -08:00

test_usage_info_streaming_with_acall.yaml

Adding usage info in llm.py (#4172)

2026-01-07 10:42:27 -08:00

test_usage_info_streaming_with_call.yaml

Adding usage info in llm.py (#4172)

2026-01-07 10:42:27 -08:00

test_using_contextual_memory_with_long_term_memory.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_using_contextual_memory_with_short_term_memory.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_using_contextual_memory.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

test_warning_long_term_memory_without_entity_memory.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

TestLLMHooksIntegration.test_direct_llm_call_hooks_integration.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

TestLLMHooksIntegration.test_lite_agent_hooks_integration_with_real_llm.yaml

fix: ensure token usage recording, validate response model on stream

2025-12-10 20:32:10 -05:00

TestStreamingCrewIntegration.test_async_streaming_from_docs.yaml

feat: add streaming result support to flows and crews

2025-11-24 15:43:48 -05:00

TestStreamingCrewIntegration.test_basic_crew_streaming_from_docs.yaml

feat: add streaming result support to flows and crews

2025-11-24 15:43:48 -05:00

TestStreamingCrewIntegration.test_kickoff_for_each_streaming_from_docs.yaml

feat: add streaming result support to flows and crews

2025-11-24 15:43:48 -05:00

TestStreamingCrewIntegration.test_streaming_properties_from_docs.yaml

feat: add streaming result support to flows and crews

2025-11-24 15:43:48 -05:00

TestStreamingCrewIntegration.test_streaming_with_chunk_context_from_docs.yaml

feat: add streaming result support to flows and crews

2025-11-24 15:43:48 -05:00

TestStreamingFlowIntegration.test_async_flow_streaming_from_docs.yaml

feat: add streaming result support to flows and crews

2025-11-24 15:43:48 -05:00

TestStreamingFlowIntegration.test_basic_flow_streaming_from_docs.yaml

feat: add streaming result support to flows and crews

2025-11-24 15:43:48 -05:00