Context survives async fallback
asyncio.to_thread() carries the scoped call context into async fallback work. Explicit contexts already travel with the callback, so both paths preserve the conversation identity and prompt-cache key.
diff --git a/.pr/design.html b/.pr/design.html new file mode 100644 index 00000000000..8f0437cb972 --- /dev/null +++ b/.pr/design.html @@ -0,0 +1,147 @@ + + +
+ + +A fallback changes the model serving a request. Its tokens, cost and telemetry still belong to the agent, condenser or ask-agent role that made the request.
+The session refactor already records a fallback response in a conversation ledger. Its old post-call merge records the same usage again, and a profile's usage_id="default" can route auxiliary calls into the agent ledger. This fix gives a temporary fallback LLM the caller's existing session references and removes the second accounting path for SDK calls.
Before: fallback_strategy.py:99 · After: fallback_strategy.py:106
+asyncio.to_thread() carries the scoped call context into async fallback work. Explicit contexts already travel with the callback, so both paths preserve the conversation identity and prompt-cache key.
Telemetry passes its current model into token, cost and latency entries. Session totals stay with the role while individual entries identify the fallback model.
telemetry.py:227 · on_response()The original positional arguments remain valid. The fallback reuses existing session references; serialized LLM and profile shapes are unchanged. fallback_strategy.py:117
+try_fallback(model, error, metrics, call_fn) +metrics.add_cost(value) +metrics.add_response_latency(value, response_id) +metrics.add_token_usage(...)
try_fallback(model, error, metrics, call_fn, + *, primary=None) +metrics.add_cost(value, *, model_name=None) +metrics.add_response_latency(..., model_name=None) +metrics.add_token_usage(..., model_name=None)
fallback_strategy.py:73 · metrics.py:189
+The original four-argument try_fallback() API keeps its metrics-diff merge. SDK calls pass primary=self, reuse the caller's sessions and callbacks, and skip that merge. The fallback example's accounting description is updated to match. fallback_strategy.py:117
The same script ran on PR #6690's 0a68ab94 and on this fix. Each operation sent one request that received HTTP 503, then one successful fallback request reporting 100 prompt tokens.
| Public operation | Before: token increase | After: token increase |
|---|---|---|
run(), first turn | default +200 | default +100 |
run(), second turn | default +200 | default +100 |
run(), third turn | default +200 | default +100 |
ask_agent() | default +100; ask-agent-llm +100 | ask-agent-llm +100 |
condense() | default +100; condenser +100 | condenser +100 |
Evidence: fallback-session-evidence.json · fallback-session-repro.py
+macOS · Python 3.13.11 · real settings construction, profile save/load, conversation lifecycle and HTTP transport. The provider runs on loopback and supplies deterministic responses. These observations establish token totals and role attribution; they do not measure callback counts, actual provider billing or remote-service behavior.
+OH_PERSISTENCE_DIR=/tmp/fallback-repro-state \ +OPENHANDS_SUPPRESS_BANNER=1 LITELLM_LOCAL_MODEL_COST_MAP=True \ +uv run python .pr/fallback-session-repro.py --assert-fixed
The final flag makes incorrect accounting fail with exit code 1. The script starts and shuts down its own local HTTP server and uses temporary profile and conversation directories.
1,791 tests passed; 2 expected failures. The LLM, conversation-session, model-switch and agent-server event-service checks passed. Two credential-path checks ran separately without the temporary persistence-directory override. The two fallback regression files account for 58 passing cases.
+The focused regression file checks sync and async Chat Completions and Responses calls, explicit and scoped contexts, standalone use, a colliding default role, and one LLM shared across registries. Two fallback calls followed by a recovered primary call verify cumulative tokens, cost, callback counts, returned metrics and actual-model labels. A separate case exercises the original direct try_fallback() API.
test_fallback_preserves_caller_accounting:41 · test_direct_fallback_preserves_supplied_metrics:230
+uv run pytest tests/sdk/llm/test_llm_fallback_sessions.py \ + tests/sdk/llm/test_llm_fallback.py tests/sdk/llm/test_llm_session.py -q
These regression tests replace provider responses at the LiteLLM boundary. The loopback reproduction above separately exercises real HTTP transport.