[FieldMarshallObvious]

Description

This PR reduces the CPU cost of _watch_disconnect in stream_completion. Before, the watcher polled request.is_disconnected() every 100 ms for every open stream, so CPU grew with the number of open streams rather than with throughput. Under multiturn load, pods hit their CPU limit while completing very few requests per second. _watch_disconnect now awaits request.receive(), so the task sleeps until the ASGI server delivers http.disconnect when the client disconnects. When a stream finishes normally, stream cleanup cancels the waiting watcher, as before. Polling remains only as a fallback if receive() returns an unexpected message type. Additionally I disabled sentry middleware_spans as it added cost when switching between middleware, and it is unused.

Unit tests where failing under the new code, I changed these tests to more closely match the ASGI behavior expected in actual runtime behavior.

QA

To validate this change I ran some small load tests on my local machine measuring CPU performance with py-spy in a docker container which mirrors stage. Additionally I ran a small test harness with a mocked litellm, in a docker container, and measure CPU consupmtion of different parts of the completions.py.

In flame charts on a load test _watch_disconnect went from 32.25% to 0.98%

Flame Chart Performances

Full flamecharts are attached in a google drive link below, these flame charts show reducing impact of _watch_disconnect on observed CPU cycles.

Watch Disconnect Without Any Changes

Watch Disconnect Disabling Sentry

Watch Disconnect Waiting Instead of Polling

Mocked MLPA Harness

This test was configured with the following setup, and demonstrates the improvement of CPU load when simulating load on MLPA.

Setup:

• Built images from each commit's Dockerfile and ran the real FastAPI app, middleware stack and stream_completion.
• Stubbed auth, user lookup and the DB lifespan, so only the request and streaming path is measured.
• The mocked LiteLLM ran on the host
• Streamed 30 OpenAI-style SSE chunks at 1 chunk/s after a 200 ms TTFT.
• The load generator also ran on the host. It held N concurrent streams open.
• CPU was read from the container's cgroup cpu.stat over a 15 s window once all streams were open, and is reported as % of one core.
• HTTPX_MAX_CONNECTIONS was raised to 1000 so all 500 streams could reach the upstream.

Limit	Open streams	Before (302dee9)	Both changes (3cff8b6)
2 CPU / 500 MiB	200	55%	6%
2 CPU / 500 MiB	500	92%	18%

Related Tickets & Documents

AIPLAT-1599
22.10.2026 - Stage MLPA - multiturn - sim-gpt
MLPA Flame Charts - 05.11.2025

[FieldMarshallObvious]

/build-pr

[FieldMarshallObvious]

Performed small scale tests on exa. and mistral in dev to confirm build performs as expected. When doing mistral test experienced some errors on the litellm side, these where errors were not related to the change.

MLPA Errors:
"Client error '429 Too Many Requests' for url 'http://litellm:8000/v1/chat/completions' - Rate limiting coming from the litellm side
mlpa.core.utils:raise_and_log:451 - Failed to proxy request: ReadError: ReadError('') - Most likely a stale keep-alive connection. It's raised while waiting for response headers on a reused pooled connection that LiteLLM had already closed

Dev Load Tests:
mistral-small-2503
Exa Load Test