Skip to content

feat(admin): Monitoring page to view and set a cluster's page/notify policy - #1989

Merged
DavidCockerill merged 4 commits into
stagefrom
david/1986-admin-monitoring-policy
Oct 21, 2026
Merged

DavidCockerill merged 4 commits into
stagefrom
david/1986-admin-monitoring-policy

Conversation

@DavidCockerill

@DavidCockerill DavidCockerill commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

⊙ Problem

Staff can stop paging for a cluster by setting its monitoring policy to notify, but must hand-write a central manager request. Nothing shows which clusters are on notify, who set them, or why.

🤔 Is a Studio page warranted, rather than the API alone?

Yes: the fleet-wide list of notify clusters and their history need a page, and the main caller is support, who should not have to hand-write the request. The page is additive; removing it is one route.

💡 Solution

A Monitoring page in admin (/admin/monitoring) uses the central manager's /Admin/ClusterMonitoringPolicy (HarperFast/central-manager#984):

  • Lists the clusters on notify: reason, who set it and when, and whether their synthetic checks reflect the change. It polls while a change is pending.
  • Set policy finds a cluster by name or clu-… id, then sets page or notify with a required reason and an apply mode. The result shows what the save did, including a refused restart.
  • History shows each cluster's changes, newest first.
  • Deploy it after HarperFast/central-manager#984: until then the page's requests 404.

⚖️ Alternatives

  • List only, with writes left to the API: support is the main caller, and the search and confirm step is what makes the change safe to make quickly.
  • Central manager reports check status explicitly: Studio infers it from row timestamps instead; moving it server-side later is cheap here.

🤔 Offer the rolling restart from this dialog?

Yes, only to accounts that may restart the cluster, warned as undrained and off-peak. Without it, metric and log alerts follow only at each container's next recreate. Dropping the option is a one-line change.

🔧 Changes

Monitoring list

The notify list: each badge says whether that cluster's synthetic checks carry its latest change (seeded scratch data, 1440px).

Set policy dialog

Set policy after picking a cluster from search: the other policy is preselected and a reason is required.

Product and architecture tour

Product and architecture tour

Setting a policy

What does a support engineer do to take a cluster off paging, and what can stop them?

From the rail to the result

  1. Open Admin → Monitoring — every staff role that can read clusters sees the list; only monitoringPolicy:write sees Set policy and Change.
  2. Find the cluster — by name or clu-… id; each match shows its organization, status and current policy.
  3. Choose — the other policy is preselected, a reason is required, and the rolling restart is offered only to an account that may restart that cluster.
  4. Save — the dialog cannot be closed and no second save can start until the answer arrives.
  5. Read the result — instances updated, any failed instance, and a refused restart.

The page never claims more than the central manager confirmed: a failed list read says so, and "Rolling restart started" appears only when the answer carries a restart.

Did the checks take the change

How does the list know whether a cluster's synthetic checks carry its latest policy?

The central manager re-stamps a cluster's synthetic checks in the background and records the outcome on the row. The Checks badge reads it, counting only an attempt that finished after the change and projected it (a skip, which keeps the old projected change, counts by its finish time), because the row keeps the previous attempt's outcome until the new one finishes. While a change is pending the list polls every 30 seconds; after 20 minutes, past the central manager's own retries, the badge reads Not confirmed and polling stops.

Before After
Whether a change reached the checks meant reading checkProjection from a raw API response. Applied, Updating, Not stamped (with the gate), Errors (with the failed checks), or Not confirmed.

Every file

✅ Verification

  • pnpm test (all 571 files), pnpm lint, pnpm format, tsc -b: pass. One unrelated Monaco test (setValueHistory.test.ts) times out intermittently under full-suite load and passes alone.
  • Live: this branch against a scratch central manager running HarperFast/central-manager#984, signed in as a support account. Listed two notify clusters, searched "prod", set page with apply: rolling, and saw the refused restart in the result while the list refetched.
What the new tests prove

🤖 Generated by Claude Code (Claude Opus 5.5); posted via @DavidCockerill.

Refs #1986

Related PRs: #1975 overlaps (merged first; this branch is rebased on it and keeps both rail items, routes and visibility expectations)
Complexity: medium

Review-Coverage: authored=claude; ran=gemini,codex; adjudicated=domain; blocked=cursor-grok(lost); declined=cursor-composer,cursor-kimi,cursor-muse,muse; rounds=3; full=1 @ a8c55c7

Review-Attention: read ~8m (decisions: do-less-alternative, in-flight-guard-scope, check-status-inferred-client-side, monitoring-gated-on-cluster-read, rolling-restart-from-policy-dialog) @ a8c55c7

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Coverage Report

Status Category Percentage Covered / Total
🔵 Lines 80.17% 15133 / 18874
🔵 Statements 80.11% 16069 / 20057
🔵 Functions 75.67% 4073 / 5382
🔵 Branches 74.81% 11805 / 15779
File Coverage
File Stmts Branches Functions Lines Uncovered Lines
Changed Files
src/features/admin/routes.ts 69.23% 100% 55.55% 69.23% 33-34, 40-41, 47-48, 54-55
src/features/admin/components/AdminShell.tsx 29.41% 46.66% 60% 33.33% 43-65
src/features/admin/monitoring/SetPolicyDialog.tsx 94.59% 85.24% 87.5% 94.59% 69, 181
src/features/admin/monitoring/checkStatus.ts 100% 100% 100% 100%
src/features/admin/monitoring/index.tsx 79.41% 64% 80.95% 78.78% 82-85, 166, 174-188, 209
src/features/admin/monitoring/monitoringPolicyApi.ts 100% 100% 100% 100%
Generated in workflow #2409 for commit a8c55c7 by the Vitest Coverage Report Action

gemini-code-assist[bot]

This comment was marked as resolved.

@DavidCockerill
DavidCockerill marked this pull request as ready for review October 21, 2026 15:10
@DavidCockerill
DavidCockerill requested a review from a team as a code owner October 21, 2026 15:10

@dawsontoth dawsontoth left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We will want to integrate this more into the existing search and navigation among clusters with staff only features. Plus, we are getting too deep into Studio PRs that require CM PRs to merge. We need to start knocking those out.

@DavidCockerill
DavidCockerill added this pull request to the merge queue Oct 21, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Oct 21, 2026
DavidCockerill and others added 4 commits October 21, 2026 15:40
Lists clusters on notify with whether their synthetic checks took the change, finds a cluster by
name or id, and sets its policy with a reason and an apply mode (rolling only for accounts that
may restart the cluster). Built on central manager's /Admin/ClusterMonitoringPolicy endpoint.

Refs #1986

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Block every way out of the set dialog while a save is in flight, show a failed list as a failure,
poll while a change is pending so a stuck one flips, tie errors to the change they were for, read
the restart from the answer, and trim exports and comments.

Refs #1986

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…stop polling a stuck row

Refs #1986

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #1986

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@DavidCockerill
DavidCockerill force-pushed the david/1986-admin-monitoring-policy branch from 9bb2abe to a8c55c7 Compare October 21, 2026 19:41
@DavidCockerill
DavidCockerill added this pull request to the merge queue Oct 21, 2026
Merged via the queue into stage with commit 9098813 Oct 21, 2026
3 checks passed
@DavidCockerill
DavidCockerill deleted the david/1986-admin-monitoring-policy branch October 21, 2026 20:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants