Skip to content

fix(dsh-memory-plugin): never give up on a returning server - #7740

Open
lztxwd6666 wants to merge 2 commits into
volcengine:mainfrom
lztxwd6666:fix/dsh-mcp-reconnect-window
Open

lztxwd6666 wants to merge 2 commits into
volcengine:mainfrom
lztxwd6666:fix/dsh-mcp-reconnect-window

Conversation

@lztxwd6666

@lztxwd6666 lztxwd6666 commented Nov 16, 2026 •

Copy link
Copy Markdown

The reconnect allowance this plugin passes to dsh-mcp-client is currently DSH's default: ten attempts, about 150 seconds of backoff. Running out isn't patient, it's fatal. The bridge unregisters every tool and schedules nothing further, so memory stays disconnected until the host restarts.

That collides with how this server behaves. A cold OpenViking takes roughly 70 seconds to bind its port — models warm up first — and any restart with real work in it takes longer than 150 seconds. The proxy starts instantly and fails immediately while the endpoint is down, so the budget is spent before the listener exists. One restart, and memory is gone for the rest of the session while /health keeps answering {"status":"ok"}.

Widening the allowance only pushes the cliff further out. The server on the other end is a local daemon: a reboot, a manual stop, or a crash loop can keep it away for an arbitrary stretch, and no guess about how long is generous enough. This drops the limit to effectively unbounded and lets the existing 30 s backoff ceiling bound the cost — after the ramp it settles into one probe every 30 s, which is nothing while the server is away and reconnects within half a minute of it returning.

reconnect: {
  enabled: true,
  initialDelayMs: 1000,
  maxDelayMs: 30000,
  maxAttempts: Number.MAX_SAFE_INTEGER,
}

Note what this does not change: the first attempt is still immediate, so a reachable server connects as fast as before, and the ceiling keeps a missing server cheap forever.

The test asserts the property rather than the number. With the default policy it reports maxAttempts 10 still gives up eventually; against the change it also checks that backoff actually plateaus at the ceiling and stays modest there, so a future edit can't satisfy the assertion with a busy loop.

Verification

Ran the plugin's own suite on Windows, after setting core.autocrlf false so the checked-out files stay LF:

# tests 113
# pass 112
# fail 0

End to end on a real host, with the same policy applied to a local profile — and in the state that actually matters, which is a connection that fails at initialization. A later HTTP blip doesn't tear the stdio proxy down at all: it only exits on stdin close or a signal, and its own retry path reinitializes the HTTP session, so the supervisor never enters its reconnect branch. Stopping the server therefore proves nothing about this budget, and my first attempt at measuring it was measuring the proxy instead.

So: server down, then remount the plugin so the first connect has nothing to reach.

  • 214 s in: tools still registered and still reaching the transport, well past the 151,500 ms the default policy spends. On main they would have been unregistered by then.
  • Server restored after 233 s of downtime, listener bound 35 s later. The bridge reconnected by itself, list and find working, no host restart and no reload.

That is also the shape of the original incident — the server died while the host was restarting, so the very first attempt had nobody to talk to.

Fixes #7738.

The bridge's reconnect budget is terminal. Spend it and it unregisters
every tool and schedules nothing further, leaving memory disconnected
until the host restarts. Ten attempts cover about 150 s, which a cold
OpenViking server can outlast on its own.

A longer allowance only moves the cliff, since the server behind this
transport is a local daemon that can be away for an unbounded stretch.
Leave the count effectively unbounded and let the 30 s backoff ceiling
bound the cost instead.
@r266-tech

Copy link
Copy Markdown
Contributor

I checked b74f7a35c41bed3acf06282b4290340202b2c319 against main@5f53cbf0 through the real mountOpenVikingMcp() → pinned dsh-mcp-client@0.1.0-rc.6 → spawned stdio proxy → loopback HTTP MCP path.

With the endpoint initially unavailable:

  • Main gives up after 11 failed connections (10 scheduled retries, 151,500 ms of accumulated virtual delay), leaves no registered tools, and does not recover when the endpoint returns.
  • This head survives 41 failed connections, keeps its later retry delays at 30,000 ms, then reconnects, registers the tool and completes a tool call when the endpoint returns. Number.MAX_SAFE_INTEGER passes the actual bridge schema.
  • Disposing during backoff clears the pending retry, registrations and owned proxy process on both heads.

One useful reproduction boundary: an HTTP-only restart after a healthy connection recovers on both heads with zero supervisor retries in this fixture; the stdio proxy stays alive. That control alone does not exercise the exhausted initialization budget. The initially-unavailable case above does.

npm test: 112 passed, 1 skipped (the opt-in live-recall test); npm run check passes. Main has 111 passed / the same skip.

AI-only verification on macOS arm64 / Node 26.13.1. The host effect/tool registry and reconnect timers were substituted; retry delays were advanced virtually. Actual SDK transport, proxy child processes and loopback HTTP requests ran. No model, deployed OpenViking, real startup-duration or native Windows validation.

@lztxwd6666

Copy link
Copy Markdown
Author

Thanks for the independent run — that's a more careful check than I managed myself, and the reproduction boundary you drew is the useful part.

You're right that my end-to-end evidence was measuring the wrong path. I had stopped the HTTP server, held it down for 247 seconds, then restarted it and watched the bridge recover. But the stdio proxy doesn't exit when the endpoint goes away: it only leaves on stdin close or a signal, and its own sendWithRetry re-initializes on 400/404. So that proxy was never torn down, the supervisor never entered scheduleReconnect, and the recovery I saw came from the proxy's session handling, not from my change. The budget I claimed to have exercised was untouched. Your control case says exactly this, and it's correct.

I re-ran it the way you described — server down, then remount the plugin so the connection fails at initialization, which is the state the bridge can't leave on its own.

  • 214 s after the failed initial connect: tools still registered, calls still reaching the transport. That's past the 151,500 ms the default policy would have spent; on main the tools would already be gone by then.
  • Restored the server after 233 s of downtime, listener bound at +35 s. The bridge reconnected by itself — no host restart, no reload — and list and find both worked.

That also lines up with the incident that started all this: the server died while the host was restarting, so the bridge's very first attempt had nothing to talk to. Initialization failure is the case that actually bites; a later blip is survivable either way. I've corrected the claim in #7738, which had the same mistake.

Worth noting for anyone reading the fixture: Number.MAX_SAFE_INTEGER passing the schema doesn't mean the supervisor tolerates it gracefully at every point, only that the validator does. The AppExit-style arithmetic stays sane because the delay is min(maxDelayMs, initialDelayMs * 2 ** (n - 1)), which saturates long before the exponent gets interesting — but if you have a cheaper way to express "unbounded" that the maintainers would rather see, I'd take it over a sentinel.

Two other things, in case they're useful to you or the maintainers:

  • The unrelated timeout in [Bug]: dsh-memory-plugin stops reconnecting before the server finishes starting #7738 is real and easy to trip while testing this: on a SiliconFlow embedder a single find takes 8–20 s, the dsh harness default is 10 000 ms, and every search fails on a server that's perfectly healthy. Different failure, same face.
  • If you run the plugin suite on Windows, set core.autocrlf=false first. bundle.test.mjs matches \n, so CRLF checkouts fail three tests that have nothing to do with the change under test. Cost me an hour.

The macOS arm64 / Node 26 caveat and the substituted host effect are fine by me — your fixture covers the supervisor arithmetic, which is the part I couldn't reach, and I've now got the real host path on Windows plus your virtualized timers. Between them the picture is better than either alone. CI is still waiting on maintainer approval for a first-time contributor, so there's nothing else for me to run until that clears.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Backlog

Development

Successfully merging this pull request may close these issues.

[Bug]: dsh-memory-plugin stops reconnecting before the server finishes starting

2 participants