Skip to content

fix(leaderboard): bulk recompute has no concurrency guard, overlapping all=true runs race on the KV checkpoint #26

Open

Description

@mrgoonie

Summary

POST /api/admin/recompute?all=true has no concurrency guard, so two simultaneous runs share one KV checkpoint. Both requests read and write the same state, offset can be advanced or reset by whichever write lands last, and every skill in the overlap window is scored twice or skipped. Both callers still get HTTP 200.

Current behavior

  • apps/web/app/routes/api.admin.recompute.ts:51 — action() verifies the admin secret and starts batching immediately. There is no in-flight check and no 409 response path anywhere in the file.
  • apps/web/app/lib/leaderboard/recompute-all-skills.ts:45-46 — KV state is read only when resume is true, and only to choose a starting offset:
    savedState = await loadState(kv);
    offset = savedState?.status === "running" ? savedState.offset : 0;
  • apps/web/app/lib/leaderboard/recompute-all-skills.ts:191-194 — every batch then overwrites or clears that same state, with no compare-and-set:
    if (isComplete) { await clearState(kv); } else { await saveState(kv, state); }
  • apps/web/app/routes/api.admin.recompute.ts:21 caps one request at MAX_BATCHES_PER_REQUEST = 50, so a single call can run for many seconds — long enough for a second call to overlap. batchCount is per-request only, so it never coordinates across requests.

Expected behavior

A second run is rejected while a run is active — for example 409 Conflict when status === "running" and lastBatchAt is recent — or the runs coordinate through an atomic lock so the checkpoint advances monotonically.

Reproduction

  1. Send two overlapping requests with the admin secret in the header:
    ADMIN_SECRET=... curl -sS -X POST "https://skillx.sh/api/admin/recompute?all=true&batch=100" &
    ADMIN_SECRET=... curl -sS -X POST "https://skillx.sh/api/admin/recompute?all=true&batch=100" &
    wait
  2. Watch the returned nextOffset and state from each response, then GET /api/admin/recompute to read the surviving checkpoint.
  3. Interleaved responses show both runs advancing the same checkpoint.

Conditions: requires ADMIN_SECRET plus the D1 and KV bindings; admin-only endpoint. The code paths above are read directly from main at 9d0f9e9 — no live concurrent run against production was performed for this report.

Flow

flowchart LR
  A[POST all=true] --> B[verifyAdmin]
  B --> C[recomputeBatch: load KV offset]
  D[POST all=true] --> E[verifyAdmin]
  E --> F[recomputeBatch: load KV offset]
  C --> G{{both write same KV checkpoint}}
  F --> G
  G --> H[offset / totalProcessed corrupted]
Loading

Evidence

  • Affected code: apps/web/app/routes/api.admin.recompute.ts:51, apps/web/app/routes/api.admin.recompute.ts:21, apps/web/app/lib/leaderboard/recompute-all-skills.ts:45, apps/web/app/lib/leaderboard/recompute-all-skills.ts:191
  • Introduced by: 88a5bff ("feat: add bulk recompute admin endpoint for leaderboard scores"), shipped on main via 9d0f9e9 (PR feat: skill references & scripts support #9)
  • Related: plans/reports/code-reviewer-260306-1531-admin-recompute-endpoint.md, section "4. No concurrency guard"

Open questions

  • Preferred mechanism: a KV read-check-write optimistic lock, a Durable Object, or simply rejecting when lastBatchAt is recent (under 5 minutes)?
  • Is this reachable in practice? scripts/run-recompute.sh sleeps DELAY seconds between batches but holds no lock, and it is currently untracked — see the runbook tooling issue.

Notes

  • The other findings from the same review are already fixed on main: the date/millisecond bug, the N+1 aggregation, the all=true batch guard, the secret-in-query-param auth, and the redundant ternary.
  • A KV read-check-write is not atomic on its own, so if that route is chosen the check must tolerate the small race window (accepting a rare double-reject) rather than assume a real lock.

Activity

  1. clark-cant commented on Oct 30, 2026

    @clark-cant

    Re-verified on main (c95ab77) — still present, with one correction and two extra defects

    Source read at c95ab77, no live concurrent run performed.

    Confirmed as filed

    • apps/web/app/routes/api.admin.recompute.ts — action() verifies the secret and batches immediately; no in-flight check, no 409 path anywhere in the file.
    • apps/web/app/lib/leaderboard/recompute-all-skills.ts — loadState is only called when resume is true, and the persist block at the end of recomputeBatch is a plain if (isComplete) clearState(kv) else saveState(kv, state) with no compare-and-set.
    • apps/web/app/lib/leaderboard/recompute-state.ts — one key, expirationTtl: 86400, plain kv.put / kv.delete. No lock primitive anywhere in apps/web/app/lib.

    Correction: "or skipped" is not reachable. nextOffset only ever advances by offset + batch.length, and each writer derives it from rows it actually processed. The realistic failure is duplicated scoring plus checkpoint regression, not skipped skills. Two things are worse than the report says:

    1. A completed run can be resurrected. If run A finishes and its clearState lands before run B's saveState, B recreates a status: "running" checkpoint at a stale offset that then lives for the full 24h TTL. GET reports a permanently in-progress run, and any later resume=true call re-scans from that stale offset.
    2. Progress fields reset on every batch. In the all=true loop, recomputeBatch is called with an explicit offset, so savedState stays null and both state.totalProcessed = batch.length and state.startedAt = now are rewritten per batch. A stuck run therefore reports roughly one batch of progress while offset is far ahead — which also weakens the lastBatchAt-freshness heuristic proposed below, since that field is refreshed by each batch of any run.

    Why a KV lock alone cannot fix this: KV reads are eventually consistent (up to ~60s stale in another colo), so a read-check-write on KV cannot serialize two Worker requests. The claimed-vs-unclaimed check needs a store with a conditional write.

    Suggested shape (both keep the 409 contract; { error: "recompute_in_progress", lastBatchAt, offset })

    • Cheapest, no schema change: reject with 409 when state?.status === "running" && now - Date.parse(state.lastBatchAt) < 120_000, and document the residual race. This fixes the realistic triggers — a double-clicked admin call and a CI retry overlapping a manual run — and the issue already accepts a rare double-reject.
    • Correct: one single-row D1 table (recompute_run: run_id, heartbeat_at, offset), claimed with UPDATE ... SET run_id = ?, heartbeat_at = ? WHERE run_id IS NULL OR heartbeat_at < ? and accepted only when meta.changes === 1. Thread run_id through the batch loop and fence every state write on it, so a stale writer cannot overwrite a newer run's checkpoint. KV then serves only GET progress — or is dropped from this feature entirely, which also removes the 24h TTL edge case.

    Tests: neither the route nor the library has a test file today (api.admin.recompute.test.ts does not exist). Minimum coverage: second caller gets 409 while a run holds the claim; completion releases it; a writer holding an old run_id cannot persist state.

    Severity: admin-only behind ADMIN_SECRET, so not urgent-exploitable — but it does defeat the resume story in the untracked run-recompute.sh (#29). recompute-all-skills.ts is 210 LOC against the 200 limit (#27); this fix is ~15 lines, so it does not need to wait for the split.

  2. clark-cant commented on Nov 4, 2026

    @clark-cant

    Re-verified at 2c1d2ba — plus the one hole in the cheapest fix, and how the guard sequences against #169/#170

    Status first: nothing in this path moved. git log c95ab77..2c1d2ba is only 51c3745 (docs: README, cli-reference, codebase-summary, personal-kits-api) and 2c1d2ba (CLI visibility). Neither touches apps/web/app/lib/leaderboard/** or apps/web/app/routes/api.admin.recompute.ts, so the 2026-10-30 findings stand line for line: no in-flight check, no 409 path, plain saveState/clearState with no compare-and-set, and a 50-batch all=true window.

    The KV-freshness option I suggested has a blind spot at t=0

    I recommended "reject when status === "running" and lastBatchAt is recent". Reading the persist block again, that check alone does not close the race:

    • State is written only after a batch completes (recompute-all-skills.ts:181-196). Nothing is written when a run starts.
    • In all=true the route passes an explicit offset, so savedState stays null and the state object is built fresh every iteration — still persisted only at the end of the batch.
    • So between action() accepting the request and the end of batch 1, KV says idle. A second POST that lands in that window passes the freshness check and starts its own run: exactly the overlap the issue describes.

    The window is one first-batch duration: three aggregate queries, the LIMIT/OFFSET select, then one UPDATE per skill. After #169 lands, batch is clamped to a max of 500 (RECOMPUTE_BATCH_LIMIT, apps/web/app/lib/admin/batch-limit.ts), so the worst case in that window is ~500 statements — noticeably longer than today's 100-row default. The clamp that fixes #28 therefore widens this hole unless the guard lands with it.

    Minimum shape that actually closes it, still KV-only:

    1. Claim before batch 1. After auth and parameter parsing, loadState; if status === "running" and lastBatchAt is fresh, return 409 { error: "recompute_in_progress", lastBatchAt, offset }. Otherwise saveState({ offset: resolvedStart, totalProcessed: 0, ..., status: "running", startedAt, lastBatchAt: now }) before calling recomputeBatch.
    2. Pick the stale threshold from one batch, not from the whole request. Every batch refreshes lastBatchAt, so the threshold only has to exceed the longest single-batch duration (plus one query round-trip), not MAX_BATCHES_PER_REQUEST × batch. 120 s stays reasonable; what was missing was the write, not the number.
    3. Add runId (crypto.randomUUID()) to RecomputeState and fence the persist: a batch may only saveState if the stored runId still matches. That is what makes a late write from a superseded run harmless instead of corrupting the checkpoint.

    Step 3 is the one that needs a store with a conditional write to be airtight (KV reads are eventually consistent across colos, as noted before). The D1 single-row claim with UPDATE ... WHERE heartbeat_at < ? and a meta.changes === 1 check remains the correct version; the KV claim above is the interim that fixes the realistic triggers — two overlapping run-recompute.sh invocations, a double-clicked admin call, and a CI retry.

    Sequencing against the two open PRs (both read from their diffs, both CLEAN)

    PR Touches the recompute path Conflict with a guard PR
    #169 (clamp, d2e7c44) api.admin.recompute.ts — adds the import and swaps parseIntSafe(batch) for parseBatchSize(batch); no 409 added Same function preamble, ~2 lines. Land the guard after #169 and the rebase is trivial.
    #170 (score builder, 370a7a8) recompute-all-skills.ts — replaces the computeBayesianRating/computeCompositeScore/computeTrendingScore block with buildScoreUpdate and deletes the local SkillAggregates interface None on the state code: savedState handling, the state object, saveState/clearState and the return are untouched, so the guard's anchors are stable under #170

    Neither PR adds a route test, and #169 already establishes the pattern (apps/web/app/lib/admin/batch-limit.test.ts). So the guard can be a single PR on top of #169: claim in the route, runId in recompute-state.ts, and — if you take the correct option — apps/web/drizzle/migrations/0023_recompute_run.sql as the next free migration number.

    Test shape that fits the existing harness

    Route tests are colocated and mock ~/lib/db with a chainable stub (apps/web/app/routes/api.cookie-mutation-origin.test.ts is the closest template). The 409 path needs no D1 at all: stub getDb with the same chain and context.cloudflare.env.KV with a get returning a status:"running" state, then assert the second action() returns 409 without calling recomputeBatch. Two more cases cover the rest: a fresh claim is written before the first batch, and a writer holding an old runId cannot persist.

    Still admin-only behind ADMIN_SECRET, so severity is unchanged — but it does defeat the resume story in scripts/run-recompute.sh, and the script hardcodes resume=true on its first call, which is exactly the overlapping case.

    Posted by the github-maintain skill at 2026-11-04T10:05Z

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions