Skip to content

docs: recompute runner script and review reports are untracked, rollout steps unreferenced #29

Open

Description

@mrgoonie

Summary

The bulk-recompute runbook tooling and two code-review reports exist only in the working tree. No commit contains them, and nothing under docs/ references the runner script, so the bulk recompute rollout cannot be reproduced from a clean checkout.

Location

  • Doc: docs/deployment-guide.md (runbook section, currently silent on bulk recompute) and docs/UPDATE_SUMMARY.md
  • Source of truth: apps/web/app/routes/api.admin.recompute.ts:1-12 (the endpoint's own usage header) and scripts/run-recompute.sh

Problem

git status --porcelain reports these paths as untracked:

?? docs/UPDATE_SUMMARY.md
?? plans/reports/code-review-260305-skill-references-scripts.md
?? plans/reports/code-reviewer-260306-1531-admin-recompute-endpoint.md
?? scripts/run-recompute.sh
  • scripts/run-recompute.sh is the operational driver for the recompute endpoint: it defaults API_URL to https://skillx.sh/api/admin/recompute, requires ADMIN_SECRET, takes a batch size argument, and supports resume by re-running.
  • grep -rn 'run-recompute' --include=*.md --include=*.json . excluding node_modules returns no matches, so the script is unreferenced and a maintainer has no documented way to find it.
  • docs/UPDATE_SUMMARY.md describes the Mar 5 documentation pass and overlaps docs/project-roadmap.md, which already records the same Phase 3.1-3.3 completions.

Proposed content

  • Commit scripts/run-recompute.sh and add a short runbook subsection (deployment guide) covering the required ADMIN_SECRET, the batch argument, resume behavior, and the all=true batch cap.
  • Decide the fate of docs/UPDATE_SUMMARY.md: keep it as a dated record or delete it as duplicated by docs/project-roadmap.md.
  • Commit the two review reports under plans/reports/, since the recompute hardening issues cite them as their evidence source.

Acceptance criteria

  • scripts/run-recompute.sh is tracked in git and referenced from a docs page
  • docs/UPDATE_SUMMARY.md is either committed or deleted
  • The two plans/reports/ review reports are committed or deleted
  • Documentation matches live behavior (verified against apps/web/app/routes/api.admin.recompute.ts)

Notes

  • Two further untracked entries, .claude/session-state/ and apps/web/.claude/session-state/, are session state and should stay ignored — they are out of scope here.
  • The review reports are the evidence base for the recompute hardening issues, so deleting them would remove the cited source unless it is summarised into the issues first.
  • Related: PR feat: skill references & scripts support #9 merged the recompute endpoint itself; this issue only closes the gap between the merged code and its operational instructions.

Activity

  1. clark-cant commented on Oct 31, 2026

    @clark-cant

    Verified on main at c95ab77 — 3 of the 4 acceptance criteria are already met; the runbook subsection is the only real work left

    What is done

    Criterion State
    scripts/run-recompute.sh tracked in git ✅ tracked — landed in acfb24f (2026-10-14, #83 "accept reviewed session-derived overlay proposals"), i.e. the overlay PR carried it, which is why this gap stayed invisible
    Referenced from a docs page ⚠️ partial — scripts/README.md:125-139 has a "Leaderboard Recompute (Operations)" section; docs/deployment-guide.md still has no recompute entry
    docs/UPDATE_SUMMARY.md committed or deleted ✅ resolved by deletion — absent from the working tree and from git log --all -- docs/UPDATE_SUMMARY.md (empty), so it never entered history and nothing dangles
    The two plans/reports/ reviews committed ✅ both tracked: code-review-260305-skill-references-scripts.md (9,020 B) and code-review-260306-1531-admin-recompute-endpoint.md (7,629 B)
    Docs match live behavior ❌ not yet — see below

    Where the docs still disagree with the endpoint (verified against apps/web/app/routes/api.admin.recompute.ts at c95ab77)

    1. The usage header is the stale one. Lines 1-12 document POST … — run one batch (default 100), batch=200, offset, resume=true and all=true, but never mention MAX_BATCHES_PER_REQUEST = 50 (line 23) or any upper bound on batch. The header is what a curl user reads first, and it is less complete than scripts/README.md, which does state the 50-batch cap. The two docs contradict each other in coverage.
    2. The batch cap is about to change. Open PR fix(admin): clamp the recompute batch size to 1..500 #169 swaps parseIntSafe for parseBatchSize, clamping batch to 1..500 with fallback 100 (apps/web/app/lib/admin/batch-limit.ts, constants RECOMPUTE_BATCH_LIMIT). After it lands, batch=0 and negatives silently become 100 and anything above 500 is clamped — undocumented today. Whatever numbers the runbook publishes should be taken from that module, not retyped.
    3. The script forwards the argument unvalidated. run-recompute.sh interpolates BATCH_SIZE="${1:-10}" straight into ?batch=. Today ./scripts/run-recompute.sh 0 reproduces the fix(admin): batch param is unclamped and batch=0 spins 50 no-op batches in all=true mode #28 spin (50 no-op batches, nextOffset never advances); after fix(admin): clamp the recompute batch size to 1..500 #169 the same command silently becomes 100. The README's "larger batches" example is fine, but it should state the 1..500 bound.
    4. Resume wording is optimistic. "Checkpoint lives in KV, so re-run the same command after an interruption" is true, but fix(leaderboard): bulk recompute has no concurrency guard, overlapping all=true runs race on the KV checkpoint #26 documents there is no in-flight guard, and the script hardcodes resume=true on its very first call — so two overlapping invocations silently share one checkpoint and can double-score rows. One line ("run a single instance at a time until fix(leaderboard): bulk recompute has no concurrency guard, overlapping all=true runs race on the KV checkpoint #26 lands") is the whole fix.
    5. all=true is a different code path. The script loops client-side over single batches and never sends all=true, so "a single all=true request is capped at 50 batches" describes the endpoint, not the script. Worth splitting into two sentences so the 50×batch ceiling is not misread as a ceiling on the script.

    Narrowed scope — one docs-only PR

    No work in flight: no open PR references #29; #169 and #170 touch the recompute path but neither adds documentation.

    Generated by the github-maintain skill at 2026-10-31T16:07Z

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentation

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions