Repository navigation
Description
Summary
POST /api/admin/recompute?all=true has no concurrency guard, so two simultaneous runs share one KV checkpoint. Both requests read and write the same state, offset can be advanced or reset by whichever write lands last, and every skill in the overlap window is scored twice or skipped. Both callers still get HTTP 200.
Current behavior
apps/web/app/routes/api.admin.recompute.ts:51—action()verifies the admin secret and starts batching immediately. There is no in-flight check and no 409 response path anywhere in the file.apps/web/app/lib/leaderboard/recompute-all-skills.ts:45-46— KV state is read only whenresumeis true, and only to choose a starting offset:savedState = await loadState(kv); offset = savedState?.status === "running" ? savedState.offset : 0;
apps/web/app/lib/leaderboard/recompute-all-skills.ts:191-194— every batch then overwrites or clears that same state, with no compare-and-set:if (isComplete) { await clearState(kv); } else { await saveState(kv, state); }
apps/web/app/routes/api.admin.recompute.ts:21caps one request atMAX_BATCHES_PER_REQUEST = 50, so a single call can run for many seconds — long enough for a second call to overlap.batchCountis per-request only, so it never coordinates across requests.
Expected behavior
A second run is rejected while a run is active — for example 409 Conflict when status === "running" and lastBatchAt is recent — or the runs coordinate through an atomic lock so the checkpoint advances monotonically.
Reproduction
- Send two overlapping requests with the admin secret in the header:
ADMIN_SECRET=... curl -sS -X POST "https://skillx.sh/api/admin/recompute?all=true&batch=100" & ADMIN_SECRET=... curl -sS -X POST "https://skillx.sh/api/admin/recompute?all=true&batch=100" & wait
- Watch the returned
nextOffsetandstatefrom each response, thenGET /api/admin/recomputeto read the surviving checkpoint. - Interleaved responses show both runs advancing the same checkpoint.
Conditions: requires ADMIN_SECRET plus the D1 and KV bindings; admin-only endpoint. The code paths above are read directly from main at 9d0f9e9 — no live concurrent run against production was performed for this report.
Flow
flowchart LR
A[POST all=true] --> B[verifyAdmin]
B --> C[recomputeBatch: load KV offset]
D[POST all=true] --> E[verifyAdmin]
E --> F[recomputeBatch: load KV offset]
C --> G{{both write same KV checkpoint}}
F --> G
G --> H[offset / totalProcessed corrupted]
Evidence
- Affected code:
apps/web/app/routes/api.admin.recompute.ts:51,apps/web/app/routes/api.admin.recompute.ts:21,apps/web/app/lib/leaderboard/recompute-all-skills.ts:45,apps/web/app/lib/leaderboard/recompute-all-skills.ts:191 - Introduced by:
88a5bff("feat: add bulk recompute admin endpoint for leaderboard scores"), shipped onmainvia9d0f9e9(PR feat: skill references & scripts support #9) - Related:
plans/reports/code-reviewer-260306-1531-admin-recompute-endpoint.md, section "4. No concurrency guard"
Open questions
- Preferred mechanism: a KV read-check-write optimistic lock, a Durable Object, or simply rejecting when
lastBatchAtis recent (under 5 minutes)? - Is this reachable in practice?
scripts/run-recompute.shsleepsDELAYseconds between batches but holds no lock, and it is currently untracked — see the runbook tooling issue.
Notes
- The other findings from the same review are already fixed on
main: the date/millisecond bug, the N+1 aggregation, theall=truebatch guard, the secret-in-query-param auth, and the redundant ternary. - A KV read-check-write is not atomic on its own, so if that route is chosen the check must tolerate the small race window (accepting a rare double-reject) rather than assume a real lock.
Activity
clark-cant commentedon Oct 30, 2026
Re-verified on
main(c95ab77) — still present, with one correction and two extra defectsSource read at
c95ab77, no live concurrent run performed.Confirmed as filed
apps/web/app/routes/api.admin.recompute.ts—action()verifies the secret and batches immediately; no in-flight check, no409path anywhere in the file.apps/web/app/lib/leaderboard/recompute-all-skills.ts—loadStateis only called whenresumeis true, and the persist block at the end ofrecomputeBatchis a plainif (isComplete) clearState(kv) else saveState(kv, state)with no compare-and-set.apps/web/app/lib/leaderboard/recompute-state.ts— one key,expirationTtl: 86400, plainkv.put/kv.delete. No lock primitive anywhere inapps/web/app/lib.Correction: "or skipped" is not reachable.
nextOffsetonly ever advances byoffset + batch.length, and each writer derives it from rows it actually processed. The realistic failure is duplicated scoring plus checkpoint regression, not skipped skills. Two things are worse than the report says:clearStatelands before run B'ssaveState, B recreates astatus: "running"checkpoint at a stale offset that then lives for the full 24h TTL.GETreports a permanently in-progress run, and any laterresume=truecall re-scans from that stale offset.all=trueloop,recomputeBatchis called with an explicitoffset, sosavedStatestays null and bothstate.totalProcessed = batch.lengthandstate.startedAt = noware rewritten per batch. A stuck run therefore reports roughly one batch of progress whileoffsetis far ahead — which also weakens thelastBatchAt-freshness heuristic proposed below, since that field is refreshed by each batch of any run.Why a KV lock alone cannot fix this: KV reads are eventually consistent (up to ~60s stale in another colo), so a read-check-write on KV cannot serialize two Worker requests. The claimed-vs-unclaimed check needs a store with a conditional write.
Suggested shape (both keep the
409contract;{ error: "recompute_in_progress", lastBatchAt, offset })state?.status === "running" && now - Date.parse(state.lastBatchAt) < 120_000, and document the residual race. This fixes the realistic triggers — a double-clicked admin call and a CI retry overlapping a manual run — and the issue already accepts a rare double-reject.recompute_run:run_id,heartbeat_at,offset), claimed withUPDATE ... SET run_id = ?, heartbeat_at = ? WHERE run_id IS NULL OR heartbeat_at < ?and accepted only whenmeta.changes === 1. Threadrun_idthrough the batch loop and fence every state write on it, so a stale writer cannot overwrite a newer run's checkpoint. KV then serves onlyGETprogress — or is dropped from this feature entirely, which also removes the 24h TTL edge case.Tests: neither the route nor the library has a test file today (
api.admin.recompute.test.tsdoes not exist). Minimum coverage: second caller gets 409 while a run holds the claim; completion releases it; a writer holding an oldrun_idcannot persist state.Severity: admin-only behind
ADMIN_SECRET, so not urgent-exploitable — but it does defeat the resume story in the untrackedrun-recompute.sh(#29).recompute-all-skills.tsis 210 LOC against the 200 limit (#27); this fix is ~15 lines, so it does not need to wait for the split.clark-cant commentedon Nov 4, 2026
Re-verified at
2c1d2ba— plus the one hole in the cheapest fix, and how the guard sequences against #169/#170Status first: nothing in this path moved.
git log c95ab77..2c1d2bais only51c3745(docs: README, cli-reference, codebase-summary, personal-kits-api) and2c1d2ba(CLI visibility). Neither touchesapps/web/app/lib/leaderboard/**orapps/web/app/routes/api.admin.recompute.ts, so the 2026-10-30 findings stand line for line: no in-flight check, no409path, plainsaveState/clearStatewith no compare-and-set, and a 50-batchall=truewindow.The KV-freshness option I suggested has a blind spot at t=0
I recommended "reject when
status === "running"andlastBatchAtis recent". Reading the persist block again, that check alone does not close the race:recompute-all-skills.ts:181-196). Nothing is written when a run starts.all=truethe route passes an explicitoffset, sosavedStatestaysnulland the state object is built fresh every iteration — still persisted only at the end of the batch.action()accepting the request and the end of batch 1, KV saysidle. A second POST that lands in that window passes the freshness check and starts its own run: exactly the overlap the issue describes.The window is one first-batch duration: three aggregate queries, the
LIMIT/OFFSETselect, then oneUPDATEper skill. After #169 lands,batchis clamped to a max of 500 (RECOMPUTE_BATCH_LIMIT,apps/web/app/lib/admin/batch-limit.ts), so the worst case in that window is ~500 statements — noticeably longer than today's 100-row default. The clamp that fixes #28 therefore widens this hole unless the guard lands with it.Minimum shape that actually closes it, still KV-only:
loadState; ifstatus === "running"andlastBatchAtis fresh, return409 { error: "recompute_in_progress", lastBatchAt, offset }. OtherwisesaveState({ offset: resolvedStart, totalProcessed: 0, ..., status: "running", startedAt, lastBatchAt: now })before callingrecomputeBatch.lastBatchAt, so the threshold only has to exceed the longest single-batch duration (plus one query round-trip), notMAX_BATCHES_PER_REQUEST × batch. 120 s stays reasonable; what was missing was the write, not the number.runId(crypto.randomUUID()) toRecomputeStateand fence the persist: a batch may onlysaveStateif the storedrunIdstill matches. That is what makes a late write from a superseded run harmless instead of corrupting the checkpoint.Step 3 is the one that needs a store with a conditional write to be airtight (KV reads are eventually consistent across colos, as noted before). The D1 single-row claim with
UPDATE ... WHERE heartbeat_at < ?and ameta.changes === 1check remains the correct version; the KV claim above is the interim that fixes the realistic triggers — two overlappingrun-recompute.shinvocations, a double-clicked admin call, and a CI retry.Sequencing against the two open PRs (both read from their diffs, both CLEAN)
d2e7c44)api.admin.recompute.ts— adds the import and swapsparseIntSafe(batch)forparseBatchSize(batch); no409added370a7a8)recompute-all-skills.ts— replaces thecomputeBayesianRating/computeCompositeScore/computeTrendingScoreblock withbuildScoreUpdateand deletes the localSkillAggregatesinterfacesavedStatehandling, thestateobject,saveState/clearStateand the return are untouched, so the guard's anchors are stable under #170Neither PR adds a route test, and #169 already establishes the pattern (
apps/web/app/lib/admin/batch-limit.test.ts). So the guard can be a single PR on top of #169: claim in the route,runIdinrecompute-state.ts, and — if you take the correct option —apps/web/drizzle/migrations/0023_recompute_run.sqlas the next free migration number.Test shape that fits the existing harness
Route tests are colocated and mock
~/lib/dbwith a chainable stub (apps/web/app/routes/api.cookie-mutation-origin.test.tsis the closest template). The409path needs no D1 at all: stubgetDbwith the same chain andcontext.cloudflare.env.KVwith agetreturning astatus:"running"state, then assert the secondaction()returns 409 without callingrecomputeBatch. Two more cases cover the rest: a fresh claim is written before the first batch, and a writer holding an oldrunIdcannot persist.Still admin-only behind
ADMIN_SECRET, so severity is unchanged — but it does defeat the resume story inscripts/run-recompute.sh, and the script hardcodesresume=trueon its first call, which is exactly the overlapping case.Posted by the github-maintain skill at 2026-11-04T10:05Z