Context survives async fallback
asyncio.to_thread() carries the scoped call context into async fallback work. Explicit contexts already travel with the callback, so both paths preserve the conversation identity and prompt-cache key.
diff --git a/.pr/design.html b/.pr/design.html index 8f0437cb972..a5307e6a4a8 100644 --- a/.pr/design.html +++ b/.pr/design.html @@ -2,146 +2,61 @@
- -A fallback changes the model serving a request. Its tokens, cost and telemetry still belong to the agent, condenser or ask-agent role that made the request.
-The session refactor already records a fallback response in a conversation ledger. Its old post-call merge records the same usage again, and a profile's usage_id="default" can route auxiliary calls into the agent ledger. This fix gives a temporary fallback LLM the caller's existing session references and removes the second accounting path for SDK calls.
Before: fallback_strategy.py:99 · After: fallback_strategy.py:106
-asyncio.to_thread() carries the scoped call context into async fallback work. Explicit contexts already travel with the callback, so both paths preserve the conversation identity and prompt-cache key.
Telemetry passes its current model into token, cost and latency entries. Session totals stay with the role while individual entries identify the fallback model.
telemetry.py:227 · on_response()Fixes OpenHands#18888 and the agent-server half of
+OpenHands#15430. Install-time MCP OAuth
+(POST /api/mcp/oauth/start) always registered http://localhost:<random port>/callback
+as the redirect URI. That port lives on the agent-server host, so the browser can only reach it when the
+browser and the agent-server share one machine and one network namespace.
browser ── authorize ──▶ provider +provider ── 302 ──▶ http://localhost:47071/callback + (browser's own localhost: + ERR_CONNECTION_REFUSED) +probe waits 5 min ─▶ "Sign-in was not completed"
browser ── authorize ──▶ provider +provider ── 302 ──▶ https://canvas.example/api/mcp/oauth/redirect +agent-server: find job by OAuth state, hand code to probe +probe exchanges code ─▶ "Connected — N tool(s)"
The original positional arguments remain valid. The fallback reuses existing session references; serialized LLM and profile shapes are unchanged. fallback_strategy.py:117
-try_fallback(model, error, metrics, call_fn) -metrics.add_cost(value) -metrics.add_response_latency(value, response_id) -metrics.add_token_usage(...)
try_fallback(model, error, metrics, call_fn, - *, primary=None) -metrics.add_cost(value, *, model_name=None) -metrics.add_response_latency(..., model_name=None) -metrics.add_token_usage(..., model_name=None)
fallback_strategy.py:73 · metrics.py:189
-The original four-argument try_fallback() API keeps its metrics-diff merge. SDK calls pass primary=self, reuse the caller's sessions and callbacks, and skip that merge. The fallback example's accounting description is updated to match. fallback_strategy.py:117
The same script ran on PR #6690's 0a68ab94 and on this fix. Each operation sent one request that received HTTP 503, then one successful fallback request reporting 100 prompt tokens.
| Public operation | Before: token increase | After: token increase | |
|---|---|---|---|
run(), first turn | default +200 | default +100 | |
run(), second turn | default +200 | default +100 | |
run(), third turn | default +200 | default +100 | |
ask_agent() | default +100; ask-agent-llm +100 | ask-agent-llm +100 | |
condense() | default +100; condenser +100 | condenser +100 | |
| OAuth client | web_url / OH_WEB_URL set | only callback_base_url in the request | neither |
| dynamically registered (DCR) | {web_url}/api/mcp/oauth/redirect | {callback_base_url}/api/mcp/oauth/redirect | loopback (unchanged) |
pre-registered (client_id or metadata URL) | {web_url}/api/mcp/oauth/redirect | loopback (unchanged) | loopback (unchanged) |
Evidence: fallback-session-evidence.json · fallback-session-repro.py
-macOS · Python 3.13.11 · real settings construction, profile save/load, conversation lifecycle and HTTP transport. The provider runs on loopback and supplies deterministic responses. These observations establish token totals and role attribution; they do not measure callback counts, actual provider billing or remote-service behavior.
-OH_PERSISTENCE_DIR=/tmp/fallback-repro-state \ -OPENHANDS_SUPPRESS_BANNER=1 LITELLM_LOCAL_MODEL_COST_MAP=True \ -uv run python .pr/fallback-session-repro.py --assert-fixed
The final flag makes incorrect accounting fail with exit code 1. The script starts and shuts down its own local HTTP server and uses temporary profile and conversation directories.
1,791 tests passed; 2 expected failures. The LLM, conversation-session, model-switch and agent-server event-service checks passed. Two credential-path checks ran separately without the temporary persistence-directory override. The two fallback regression files account for 58 passing cases.
-The focused regression file checks sync and async Chat Completions and Responses calls, explicit and scoped contexts, standalone use, a colliding default role, and one LLM shared across registries. Two fallback calls followed by a recovered primary call verify cumulative tokens, cost, callback counts, returned metrics and actual-model labels. A separate case exercises the original direct try_fallback() API.
test_fallback_preserves_caller_accounting:41 · test_direct_fallback_preserves_supplied_metrics:230
-uv run pytest tests/sdk/llm/test_llm_fallback_sessions.py \ - tests/sdk/llm/test_llm_fallback.py tests/sdk/llm/test_llm_session.py -q
These regression tests replace provider responses at the LiteLLM boundary. The loopback reproduction above separately exercises real HTTP transport.
A pre-registered client's redirect URIs are fixed at the provider, so a browser-supplied base must not
+change them; only the operator's web_url does. A DCR client stored from an earlier
+loopback registration is dropped and registered again when its redirect URIs do not contain the new URI.
POST /api/mcp/oauth/start body: MCPTestRequest + callback_base_url?: string (plain http(s) base, no query/fragment/userinfo) +GET /api/mcp/oauth/redirect new, unauthenticated, not in OpenAPI; ?code&state[&iss] or ?error[&error_description]+
The redirect route is outside the session-key dependency because the provider sends the user's browser
+there. It accepts a request only for a registered, unexpired job that is currently waiting for a redirect and
+whose OAuth state matches (constant-time compare). The first redirect wins; a replay gets 409;
+a provider error fails the probe with the provider's description.
/api/mcp/oauth/callback/{job} relay).callback_base_url is optional; older clients keep working.callback_base_url = its backend origin (companion OpenHands PR).{html.escape(message)}
", + status_code=status_code, + ) + + +@mcp_oauth_redirect_router.get("/redirect", include_in_schema=False) +async def receive_mcp_oauth_redirect(request: Request) -> HTMLResponse: + """Receive the provider's OAuth redirect for a pending install probe. + + The probe job is found by the OAuth ``state`` it put in the authorization + URL, which only the provider and the browser that signed in have seen. + The probe then exchanges the code; the status route reports the outcome. + """ + params = request.query_params + state = params.get("state") + job = _find_oauth_job_by_state(state) if state else None + if state is None or job is None: + return _oauth_redirect_page( + "This sign-in is unknown or has expired. " + "Start the connection again in OpenHands.", + 400, + ) + error = params.get("error") + code = params.get("code") + if error is None and not code: + error = "the provider returned no authorization code" + if error is not None: + error = params.get("error_description") or error + if not job.accept_redirect( + code=code, state=state, iss=params.get("iss"), error=error + ): + return _oauth_redirect_page("This sign-in was already completed.", 409) + if error is not None: + logger.info("MCP OAuth job %s was not authorized: %s", job.id, error) + return _oauth_redirect_page(f"The provider reported: {error}", 400) + return _oauth_redirect_page( + "You can close this window and return to OpenHands.", 200 + ) + + def _validate_callback_url(callback_url: str, job: _MCPOAuthProbeJob) -> None: try: parsed = urlparse(callback_url) diff --git a/tests/agent_server/test_mcp_oauth_redirect.py b/tests/agent_server/test_mcp_oauth_redirect.py new file mode 100644 index 00000000000..dd1b78336e4 --- /dev/null +++ b/tests/agent_server/test_mcp_oauth_redirect.py @@ -0,0 +1,400 @@ +"""Install-time MCP OAuth through the agent-server's own redirect route. + +A real FastMCP server protected by FastMCP's in-memory OAuth provider performs +dynamic client registration, validates the redirect URI and exchanges the +code. The test plays the browser: it follows the authorization URL and +delivers the provider's redirect to wherever the provider sent it. +""" + +from __future__ import annotations + +import time +from collections.abc import Generator +from urllib.parse import parse_qs, urlparse + +import httpx +import pytest +from fastapi.testclient import TestClient +from fastmcp.server.auth.providers.in_memory import InMemoryOAuthProvider +from pydantic import SecretStr + +from openhands.agent_server.api import create_app +from openhands.agent_server.config import Config +from openhands.agent_server.mcp_router import ( + MCPTestRequest, + _BrowserCoordinatedOAuth, + _MCPOAuthProbeJob, + _oauth_probe_jobs, + _register_oauth_job, +) +from tests.agent_server.test_mcp_oauth_store import ( # noqa: F401 - fixtures + oauth_provider, + protected_oauth_mcp_server, +) + + +PUBLIC_REDIRECT = "https://canvas.example.test/api/mcp/oauth/redirect" + + +def _client(**config: object) -> TestClient: + return TestClient( + create_app( + Config( + session_api_keys=[], + secret_key=SecretStr("mcp-oauth-redirect-test-key"), + **config, # type: ignore[arg-type] + ) + ), + raise_server_exceptions=False, + ) + + +def _oauth_server(url: str, state: dict | None = None) -> dict: + auth: dict = { + "strategy": "oauth2", + "authentication": { + "type": "oauth", + "client_auth_method": "none", + "scopes": ["mail.read"], + }, + } + if state is not None: + auth["state"] = state + return {"transport": "http", "url": url, "auth": auth} + + +def _start(client: TestClient, body: dict) -> tuple[str, str]: + response = client.post("/api/mcp/oauth/start", json={"timeout": 10.0, **body}) + assert response.status_code == 200, response.text + started = response.json() + assert started["ok"] is True, started + job_id = started["job_id"] + for _ in range(100): + if client.get(f"/api/mcp/oauth/status/{job_id}").json()["callback_ready"]: + break + time.sleep(0.05) + return job_id, started["authorization_url"] + + +def _redirect_uri(authorization_url: str) -> str: + return parse_qs(urlparse(authorization_url).query)["redirect_uri"][0] + + +def _sign_in(authorization_url: str) -> str: + """Return where the provider redirects the browser after consent.""" + response = httpx.get(authorization_url, follow_redirects=False, timeout=10.0) + assert response.status_code in {302, 303, 307, 308}, response.text + return response.headers["location"] + + +def _wait_for_result(client: TestClient, job_id: str) -> dict: + status: dict = {} + for _ in range(200): + status = client.get(f"/api/mcp/oauth/status/{job_id}").json() + if status["status"] in {"succeeded", "failed"}: + return status + time.sleep(0.05) + raise AssertionError(f"OAuth job {job_id} did not finish: {status}") + + +@pytest.fixture +def jobs() -> Generator[None]: + yield + _oauth_probe_jobs.clear() + + +def test_browser_redirect_goes_to_callback_base_url_and_completes( + protected_oauth_mcp_server: str, # noqa: F811 + jobs: None, +): + client = _client() + job_id, authorization_url = _start( + client, + { + "server": _oauth_server(protected_oauth_mcp_server), + "callback_base_url": "https://canvas.example.test/", + }, + ) + + assert _redirect_uri(authorization_url) == PUBLIC_REDIRECT + location = _sign_in(authorization_url) + assert location.startswith(f"{PUBLIC_REDIRECT}?") + assert client.get(f"/api/mcp/oauth/status/{job_id}").json()["status"] == ( + "authorizing" + ) + + redirect = urlparse(location) + landing = client.get(f"{redirect.path}?{redirect.query}") + + assert landing.status_code == 200 + assert "Authorization complete" in landing.text + status = _wait_for_result(client, job_id) + assert status["status"] == "succeeded", status + assert status["tools"] == ["read_subject"] + assert status["oauth_state"]["tokens"]["access_token"].startswith("gAAAA") + + +def test_web_url_with_path_prefix_wins_over_callback_base_url( + protected_oauth_mcp_server: str, # noqa: F811 + jobs: None, +): + client = _client(web_url="https://agent.example.com/openhands/") + job_id, authorization_url = _start( + client, + { + "server": _oauth_server(protected_oauth_mcp_server), + "callback_base_url": "https://canvas.example.test", + }, + ) + + assert _redirect_uri(authorization_url) == ( + "https://agent.example.com/openhands/api/mcp/oauth/redirect" + ) + redirect = urlparse(_sign_in(authorization_url)) + # The reverse proxy strips the path prefix before the agent-server. + assert redirect.path == "/openhands/api/mcp/oauth/redirect" + landing = client.get(f"/api/mcp/oauth/redirect?{redirect.query}") + + assert landing.status_code == 200 + assert _wait_for_result(client, job_id)["status"] == "succeeded" + + +def test_without_a_public_url_the_loopback_callback_is_unchanged( + protected_oauth_mcp_server: str, # noqa: F811 + jobs: None, +): + client = _client() + job_id, authorization_url = _start( + client, {"server": _oauth_server(protected_oauth_mcp_server)} + ) + + redirect_uri = urlparse(_redirect_uri(authorization_url)) + assert (redirect_uri.scheme, redirect_uri.hostname, redirect_uri.path) == ( + "http", + "localhost", + "/callback", + ) + assert client.get(f"/api/mcp/oauth/status/{job_id}").json()["status"] == ( + "authorizing" + ) + # The loopback callback server inside the probe receives the redirect. + httpx.get(_sign_in(authorization_url), timeout=10.0) + + assert _wait_for_result(client, job_id)["status"] == "succeeded" + + +def test_a_client_registered_for_the_loopback_callback_is_registered_again( + protected_oauth_mcp_server: str, # noqa: F811 + jobs: None, +): + client = _client() + job_id, authorization_url = _start( + client, {"server": _oauth_server(protected_oauth_mcp_server)} + ) + httpx.get(_sign_in(authorization_url), timeout=10.0) + first = _wait_for_result(client, job_id) + stored_client = first["oauth_state"]["client_info"] + assert stored_client["redirect_uris"][0].startswith("http://localhost:") + + job_id, authorization_url = _start( + client, + { + "server": _oauth_server( + protected_oauth_mcp_server, {"client_info": stored_client} + ), + "callback_base_url": "https://canvas.example.test", + }, + ) + redirect = urlparse(_sign_in(authorization_url)) + client.get(f"{redirect.path}?{redirect.query}") + second = _wait_for_result(client, job_id) + + assert second["status"] == "succeeded", second + assert ( + second["oauth_state"]["client_info"]["client_id"] + != (stored_client["client_id"]) + ) + assert second["oauth_state"]["client_info"]["redirect_uris"] == [PUBLIC_REDIRECT] + + +def test_changed_callback_refreshes_stored_tokens_without_authorization( + protected_oauth_mcp_server: str, # noqa: F811 + oauth_provider: InMemoryOAuthProvider, # noqa: F811 + jobs: None, +): + client = _client() + job_id, authorization_url = _start( + client, {"server": _oauth_server(protected_oauth_mcp_server)} + ) + httpx.get(_sign_in(authorization_url), timeout=10.0) + first = _wait_for_result(client, job_id) + state = first["oauth_state"] + state["token_expires_at"] = time.time() - 60 + registered_clients = set(oauth_provider.clients) + refresh_tokens = set(oauth_provider.refresh_tokens) + + response = client.post( + "/api/mcp/oauth/start", + json={ + "timeout": 10.0, + "server": _oauth_server(protected_oauth_mcp_server, state), + "callback_base_url": "https://canvas.example.test", + }, + ) + + assert response.status_code == 200, response.text + started = response.json() + assert started["ok"] is True, started + assert "authorization_url" not in started, started + second = _wait_for_result(client, started["job_id"]) + assert second["status"] == "succeeded", second + assert second["tools"] == ["read_subject"] + assert second["oauth_state"]["client_info"] == state["client_info"] + assert second["oauth_state"]["token_expires_at"] > time.time() + assert set(oauth_provider.clients) == registered_clients + assert not refresh_tokens.intersection(oauth_provider.refresh_tokens) + + +def _job() -> _MCPOAuthProbeJob: + return _MCPOAuthProbeJob( + request=MCPTestRequest.model_validate( + {"server": _oauth_server("https://mcp.example.com/mcp")} + ), + cipher=None, + public_redirect_url=PUBLIC_REDIRECT, + ) + + +def _waiting_job(state: str) -> _MCPOAuthProbeJob: + job = _job() + job.oauth_state = state + job.set_callback_ready(PUBLIC_REDIRECT) + _register_oauth_job(job) + return job + + +def test_redirect_with_an_unknown_state_completes_no_job(jobs: None): + job = _waiting_job("expected-state") + + response = _client().get("/api/mcp/oauth/redirect?code=abc&state=other-state") + + assert response.status_code == 400 + assert "unknown or has expired" in response.text + assert not job.redirect_received.is_set() + + +def test_redirect_before_the_job_waits_for_it_is_rejected(jobs: None): + job = _job() + job.oauth_state = "early-state" + _register_oauth_job(job) + + response = _client().get("/api/mcp/oauth/redirect?code=abc&state=early-state") + + assert response.status_code == 400 + assert not job.redirect_received.is_set() + + +def test_redirect_for_an_expired_job_is_rejected( + jobs: None, monkeypatch: pytest.MonkeyPatch +): + job = _waiting_job("late-state") + monkeypatch.setattr(job, "created_at", time.monotonic() - 16 * 60) + + response = _client().get("/api/mcp/oauth/redirect?code=abc&state=late-state") + + assert response.status_code == 400 + assert job.id not in _oauth_probe_jobs + + +def test_provider_error_fails_the_job_and_a_replay_is_rejected(jobs: None): + job = _waiting_job("the-state") + client = _client() + + denied = client.get( + "/api/mcp/oauth/redirect?state=the-state&error=access_denied" + "&error_description=The+user+denied+access" + ) + replay = client.get("/api/mcp/oauth/redirect?state=the-state&code=abc") + + assert denied.status_code == 400 + assert "The user denied access" in denied.text + assert job.redirect_error == "The user denied access" + assert job.redirect_result is None + assert replay.status_code == 409 + + +def test_redirect_route_does_not_need_the_session_api_key(jobs: None): + job = _waiting_job("keyless-state") + client = TestClient( + create_app(Config(session_api_keys=["a-session-key"])), + raise_server_exceptions=False, + ) + + assert client.get("/api/mcp/oauth/status/x").status_code == 401 + response = client.get("/api/mcp/oauth/redirect?code=abc&state=keyless-state") + + assert response.status_code == 200 + assert job.redirect_result is not None + assert job.redirect_result.code == "abc" + + +@pytest.mark.parametrize( + ("client_id", "static_redirect", "expected"), + [ + (None, None, PUBLIC_REDIRECT), + ("static-client", None, "http://localhost:"), + ( + "static-client", + "https://agent.example.com/api/mcp/oauth/redirect", + "https://agent.example.com/api/mcp/oauth/redirect", + ), + ], + ids=["dynamic-client", "static-client-keeps-loopback", "static-client-web-url"], +) +def test_pre_registered_clients_only_follow_a_configured_web_url( + client_id: str | None, static_redirect: str | None, expected: str +): + job = _MCPOAuthProbeJob( + request=MCPTestRequest.model_validate( + {"server": _oauth_server("https://mcp.example.com/mcp")} + ), + cipher=None, + public_redirect_url=static_redirect or PUBLIC_REDIRECT, + static_client_redirect_url=static_redirect, + ) + oauth = _BrowserCoordinatedOAuth( + job=job, mcp_url="https://mcp.example.com/mcp", client_id=client_id + ) + + redirect_uris = [ + str(uri) for uri in oauth.context.client_metadata.redirect_uris or [] + ] + assert len(redirect_uris) == 1 + assert redirect_uris[0].startswith(expected) + if client_id is not None: + assert oauth._static_client_info is not None + assert [ + str(uri) for uri in oauth._static_client_info.redirect_uris or [] + ] == redirect_uris + + +@pytest.mark.parametrize( + "callback_base_url", + [ + "ftp://canvas.example.test", + "https://user:pass@canvas.example.test", + "https://canvas.example.test/?next=x", + "https://canvas.example.test/#x", + "not a url", + ], +) +def test_callback_base_url_must_be_a_plain_http_base(callback_base_url: str): + response = _client().post( + "/api/mcp/oauth/start", + json={ + "server": _oauth_server("https://mcp.example.com/mcp"), + "callback_base_url": callback_base_url, + }, + ) + + assert response.status_code == 422