Skip to content

fix(vmcp): allow server/discover in peer health probes (PAIINFRA-399) - #28

Merged
carlos-lopez-vecino merged 1 commit into
dp-stablefrom
PAIINFRA-399/peer-probe-server-discover
Oct 26, 2026
Merged

carlos-lopez-vecino merged 1 commit into
dp-stablefrom
PAIINFRA-399/peer-probe-server-discover

Conversation

@carlos-lopez-vecino

Copy link
Copy Markdown

Jira Issue

Problem

After staging moved to v0.42.1 (DocPlanner/platform-apps#2047), the staging hubs lost the i0 and global backends. The tool count dropped from 698 to 154.

Root cause

Fork patch 43623c1 (peer_probe.go) lets health-probe requests (header X-ToolHive-Health-Check) use only initialize, ping and the */list methods.

v0.42.1 added the Modern (2026-07-28) protocol. Once a hub caches a backend as Modern, its health checker opens with server/discover. The backend vMCP refuses that with 403.

The hub maps the 403 to errModernAuth, then to ErrAuthenticationFailed. Backends with no outgoing auth (in-cluster i0 and global) are marked unauthenticated by authErrorStatus. The circuit breaker opens and their tools drop out of aggregation.

Evidence:

  • The i0-tools audit log shows 398 denied server/discover calls (0 ms, empty clientCapabilities, so the health checker).
  • hub-codex logs "failed to modern discover for backend i0: modern backend returned an auth status: HTTP 403" every 2 minutes.

Side effect

istanbul and security also get the 403. They have outgoing auth, so authErrorStatus counts the 403 as Healthy. Their health checks have been hollow since v0.42.1. This fix covers them too.

Upstream

Upstream is not affected. Neither v0.42.1 nor upstream main reads the probe header server-side. This is a fork-only bug. The v0.42.1 merge (#27) missed updating the allowlist.

Fix

Add server/discover to peerProbeMethods, with a comment.

There is no new exposure. server/discover is already always allowed without the probe header (pkg/authz/middleware.go).

Testing

  • Probe server/discover is passed and marked.
  • Probe subscriptions/listen stays refused.
  • The new case failed before the change.
  • go test ./pkg/vmcp/server/ passes (65 s). go vet is clean.

Rollout

  1. Build the images with toolhive-builder.
  2. Bump vmcpImage of toolhive-operator-stg in platform-apps app-of-apps/chart/values-platform.yaml. Bump the other images too, for consistency.
  3. This is a server-side fix, so all four vMCPs need the new image: i0-tools, global-tools, istanbul-tools, security-tools.

Production (PAIINFRA-242) must include this fix before it moves to v0.42.1.

🤖 Generated with Claude Code

Add server/discover to peerProbeMethods so Modern-protocol health
checks from a hub are not refused with 403.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@carlos-lopez-vecino
carlos-lopez-vecino merged commit ebc9d8e into dp-stable Oct 26, 2026
36 of 43 checks passed
@carlos-lopez-vecino
carlos-lopez-vecino deleted the PAIINFRA-399/peer-probe-server-discover branch October 26, 2026 09:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants