Repository navigation
fix(dsh-memory-plugin): never give up on a returning server - #7740
lztxwd6666 wants to merge 2 commits into
Conversation
The bridge's reconnect budget is terminal. Spend it and it unregisters every tool and schedules nothing further, leaving memory disconnected until the host restarts. Ten attempts cover about 150 s, which a cold OpenViking server can outlast on its own. A longer allowance only moves the cliff, since the server behind this transport is a local daemon that can be away for an unbounded stretch. Leave the count effectively unbounded and let the 30 s backoff ceiling bound the cost instead.
|
I checked With the endpoint initially unavailable:
One useful reproduction boundary: an HTTP-only restart after a healthy connection recovers on both heads with zero supervisor retries in this fixture; the stdio proxy stays alive. That control alone does not exercise the exhausted initialization budget. The initially-unavailable case above does.
AI-only verification on macOS arm64 / Node 26.13.1. The host effect/tool registry and reconnect timers were substituted; retry delays were advanced virtually. Actual SDK transport, proxy child processes and loopback HTTP requests ran. No model, deployed OpenViking, real startup-duration or native Windows validation. |
|
Thanks for the independent run — that's a more careful check than I managed myself, and the reproduction boundary you drew is the useful part. You're right that my end-to-end evidence was measuring the wrong path. I had stopped the HTTP server, held it down for 247 seconds, then restarted it and watched the bridge recover. But the stdio proxy doesn't exit when the endpoint goes away: it only leaves on stdin close or a signal, and its own I re-ran it the way you described — server down, then remount the plugin so the connection fails at initialization, which is the state the bridge can't leave on its own.
That also lines up with the incident that started all this: the server died while the host was restarting, so the bridge's very first attempt had nothing to talk to. Initialization failure is the case that actually bites; a later blip is survivable either way. I've corrected the claim in #7738, which had the same mistake. Worth noting for anyone reading the fixture: Two other things, in case they're useful to you or the maintainers:
The macOS arm64 / Node 26 caveat and the substituted host effect are fine by me — your fixture covers the supervisor arithmetic, which is the part I couldn't reach, and I've now got the real host path on Windows plus your virtualized timers. Between them the picture is better than either alone. CI is still waiting on maintainer approval for a first-time contributor, so there's nothing else for me to run until that clears. |
The reconnect allowance this plugin passes to
dsh-mcp-clientis currently DSH's default: ten attempts, about 150 seconds of backoff. Running out isn't patient, it's fatal. The bridge unregisters every tool and schedules nothing further, so memory stays disconnected until the host restarts.That collides with how this server behaves. A cold OpenViking takes roughly 70 seconds to bind its port — models warm up first — and any restart with real work in it takes longer than 150 seconds. The proxy starts instantly and fails immediately while the endpoint is down, so the budget is spent before the listener exists. One restart, and memory is gone for the rest of the session while
/healthkeeps answering{"status":"ok"}.Widening the allowance only pushes the cliff further out. The server on the other end is a local daemon: a reboot, a manual stop, or a crash loop can keep it away for an arbitrary stretch, and no guess about how long is generous enough. This drops the limit to effectively unbounded and lets the existing 30 s backoff ceiling bound the cost — after the ramp it settles into one probe every 30 s, which is nothing while the server is away and reconnects within half a minute of it returning.
Note what this does not change: the first attempt is still immediate, so a reachable server connects as fast as before, and the ceiling keeps a missing server cheap forever.
The test asserts the property rather than the number. With the default policy it reports
maxAttempts 10 still gives up eventually; against the change it also checks that backoff actually plateaus at the ceiling and stays modest there, so a future edit can't satisfy the assertion with a busy loop.Verification
Ran the plugin's own suite on Windows, after setting
core.autocrlf falseso the checked-out files stay LF:End to end on a real host, with the same policy applied to a local profile — and in the state that actually matters, which is a connection that fails at initialization. A later HTTP blip doesn't tear the stdio proxy down at all: it only exits on stdin close or a signal, and its own retry path reinitializes the HTTP session, so the supervisor never enters its reconnect branch. Stopping the server therefore proves nothing about this budget, and my first attempt at measuring it was measuring the proxy instead.
So: server down, then remount the plugin so the first connect has nothing to reach.
mainthey would have been unregistered by then.listandfindworking, no host restart and no reload.That is also the shape of the original incident — the server died while the host was restarting, so the very first attempt had nobody to talk to.
Fixes #7738.