Skip to content

Ask once whether a reused learning helped - #727

Open
anirudhkumar-nanonets wants to merge 12 commits into
mainfrom
claude/learning-helped-feedback-e9a0c2
Open

anirudhkumar-nanonets wants to merge 12 commits into
mainfrom
claude/learning-helped-feedback-e9a0c2

Conversation

@anirudhkumar-nanonets

@anirudhkumar-nanonets anirudhkumar-nanonets commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

The issue: Trail couldn't tell whether a learning it handed Claude actually helped, so the launch goal "every internal user loves it" had no number.
Why: The only sign of reuse is Claude's 📝 Started from …'s learning credit line, and nothing ever asked the person about it (the old learning_reused event was removed).
The fix: Once a session, in a reply that asks nothing else, Claude ends that credit line with "did it help? yes/no"; the next message becomes a learning_rated event (a yes or "it helped" is up, a short no or "didn't help" is down) with only the vote and the learning's id.

Moves: share of reused learnings rated helpful (learning_rated with vote = up over up + down, PostHog posthog_event_full_view, Metabase db 11). Today: 0 events (new event).

What changed

  • src/claude/learning-vote.ts (new): the ask text, reading the vote, marking it pending at Stop (strict when the turn asks something else too), settling it on the next prompt, closing it as none when the session closes unanswered.
  • src/claude/format.ts: the credit instruction in notesForPrompt / notesOnTouch gains "if it asks the user nothing else, end that line with ' · did it help? yes/no'" until the session has asked, and remembers the id of the top learning shown. This is the default full view. With trail summary one-line, learnings come as one line with no credit request, so nothing is asked.
  • src/claude/stop.ts: turnEnd marks the vote pending when last_assistant_message has a 📝 line carrying "did it help? yes/no". If that reply, or any later one before the answer, asks something else (a sentence ending in "?", a "(y/n)", or a tool question with answer buttons), the vote turns strict. Same single session write as before.
  • src/claude/tally.ts: the transcript read already done at Stop also notes whether the open turn called a tool with an ask (a question shown as buttons that answer with a new message).
  • src/claude/prompt-hook.ts: the next prompt settles it. "It helped" / "didn't help" anywhere decides it. Otherwise a leading yes word (yes, yeah, yep, sure, haan, sí, ja…), 👍 or +1 → up; 👎, -1, or a no word (no, nope, nah, not really, nahi…) in a reply of up to four words or one that mentions help → down; anything else → none. In strict mode only the help phrases count. The prompt goes on to Claude unchanged.
  • src/telemetry/sessions.ts: the idle sweep that closes a session for session_summary (untouched two hours, at the next session start) also sends a still-waiting vote as none.
  • src/telemetry/contract.ts + TELEMETRY.md: new learning_rated event with vote (closed set) and learning_id (checked by shape, l- + hex, so a title can't ride along).

Calls I made

Decided: yes/no in words, not thumbs

Nobody types a thumbs emoji in a terminal, so thumbs would mostly have measured how hard it is to answer. The question is now "did it help? yes/no", and a next message starting with the word yes or no is the vote. Thumbs and +1 / -1 still count. A clickable yes/no isn't possible here: hooks can only add text to Claude's reply, and Claude's own multiple-choice question would stop the turn until answered. The decision ships as a learning in .trail/learnings/.

Follow-ups (not in this PR)

  • Cursor and Copilot: their Stop payload has no last_assistant_message, so no vote is recorded there yet. Spotting the ask in the transcript read (src/claude/tally.ts) would cover them.
  • When two learnings are shown together and Claude credits the second, the vote lands on the first (the id is the top one of the last batch shown).
  • A Metabase card for the share, once votes come in.

Checks

  • After the stricter reading and merging main (share-session survives deeply nested sessions #844, share-session says what to type, folder numbers included #847) on 17 Nov: npm test with a scratch HOME, 4,548 tests, 4,534 pass, 5 fail, 9 skipped. All 5 fail the same way on a clean origin/main on this laptop: 3 in graft-to-trail, hosts-trail and upkeep-hooks, and 2 first-minute tests from init ends by showing Trail on your own last task (prototype) #757 that print "you" when the local git user wrote the learnings they list. None is in code this PR touches.
  • Typecheck clean. trail check: no conflicts.
  • test/learning-vote.test.ts: the ask once a session and only when the reply asks nothing else; the vote table (yes words, a short no, help phrases anywhere, idioms like "no worries", never a bare y/n); strict mode after another question, from the reply's text and from a question tool in the transcript; Stop → prompt → one event with no prompt text in it; a session closed unanswered sends none once, and a late yes after that counts nothing; the contract dropping a non-id learning_id.
  • Windows CI: main now has the 4× slower deadlineClock for the skills prompt tests. This PR gives skills-session-start's ask the same clock, because a shared Windows runner ran past the 400 ms deadline there and kept no block.

🤖 Generated with Claude Code

@trailhq-graft

trailhq-graft Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

🌱 graft blast radius

2 areas changed → 7 areas can be affected. 24 dependent symbols, depth 2.
Tests: Session Telemetry has tests the diff did not touch; 1 area updated its tests.
Tag: @shhdwi — 9 of 9 areas · @afeddersen — Session Telemetry · @Faeeth — Note Reading

flowchart TB
  A0(("Learning Tracking<br/>10 symbols"))
  A1(("Initialization<br/>4 symbols"))
  A2(("Note Reading<br/>3 symbols"))
  A3(("Telemetry Events<br/>3 symbols"))
  A4(("Cloud Nudging<br/>2 symbols"))
  AX(("2 smaller areas<br/>2 symbols"))
  classDef reached fill:#D9EDF3,stroke:#3AA7C9,stroke-width:1.5px,color:#0E313C;
  class A0,A1,A2,A3,A4 reached;
  classDef tail fill:#EEF2F3,stroke:#9AA4A9,stroke-width:1px,color:#3A4247;
  class AX tail;
Loading
Can be affected Symbols Nearest hop Reached from
Learning Tracking 10 src/claude/hooks.ts:L202-L232 learningsOnTouch — calls, depth 1 Prompt Voting, Session Telemetry
Initialization 4 src/init/first-minute.ts:L615-L645 sayEnding — calls, depth 2 Session Telemetry
Note Reading 3 src/notes/memory-notes.ts:L256-L284 finishMemoryRead — calls, depth 2 Session Telemetry
Telemetry Events 3 src/telemetry/track.ts:L109-L139 track — calls, depth 1 Session Telemetry
Cloud Nudging 2 src/cloud/nudge.ts:L164-L213 cloudLines — calls, depth 2 Session Telemetry
Session Persistence 1 src/learn/sessions.ts:L281-L309 finishSessionRead — calls, depth 2 Session Telemetry
1 smaller area 1 CLI Skills see below
Who knows this code — 4 people across 9 areas
Area Who knows it
Session Telemetry · changed @shhdwi — 1 commit, last 25d ago · @afeddersen — 1 commit, last 3mo ago
Prompt Voting · changed @shhdwi — 19 commits, last 21d ago
Learning Tracking · affected @shhdwi — 14 commits, last 21d ago
Initialization · affected @shhdwi — 9 commits, last 21d ago
Note Reading · affected @shhdwi — 5 commits, last 9d ago · @Faeeth — 1 commit, last 9d ago
Telemetry Events · affected @shhdwi — 1 commit, last 25d ago
Cloud Nudging · affected @shhdwi — 4 commits, last 21d ago · Rushabh Nagda — 1 commit, last 27d ago
Session Persistence · affected @shhdwi — 2 commits, last 21d ago
…1 further area

Ownership is git history over each area's own files, weighted towards recent work (120-day half-life). Merge commits and bots are dropped, and you are dropped from your own PR. A name with no @ has no GitHub handle in its commit email — tag them by hand, or add a .mailmap entry. A suggestion from history, not a CODEOWNERS rule.

All 24 dependent symbols, grouped by area

Learning Tracking — 10 symbols in 4 files

  • src/claude/hooks.ts:L202-L232 — learningsOnTouch (calls, depth 1)
    210: // In a Claude Desktop worktree session the project is the main checkout and the file is in
  • src/claude/hooks.ts:L537-L627 — main (calls, depth 1)
    546: // Registered in the repo and in ~/.claude, Claude Code runs both: one answers.
  • src/claude/session-metrics.ts:L163-L165 — SessionSummary (extends, depth 1)
    163: export interface SessionSummary extends SessionState {
  • src/claude/hooks.ts:L106-L120 — handlePostEdit (calls, depth 2)
    107: const file = editedFilePath(input, dir);
  • src/claude/hooks.ts:L240-L245 — handlePreEdit (calls, depth 2)
    243: const notice = hookNotice(dir, "Claude just got your team's learnings on the file it's changing", { session: payloadSessionId(input) }) ?? undefined;
  • src/claude/hooks.ts:L252-L257 — withTouchLearnings (calls, depth 2)
  • src/claude/hooks.ts:L355-L376 — trackLearningsCommitted (calls, depth 2)
    371: { repo: place.checkout },
  • src/claude/hooks.ts:L330-L346 — trackRepoSeen (calls, depth 2)
    334: const mark = join(cacheDir(place.checkout), 'trail-repo-seen.json');
  • src/claude/prompt-skills.ts:L73-L109 — promptSkills (calls, depth 2)
    85: track: deps.track ?? ((e, p) => track(e, p, { env: ctx.env, ...(ctx.checkout ? { repo: ctx.checkout } : {}) })),
  • src/claude/skills-wiring.ts:L33-L37 — send (calls, depth 2)

Initialization — 4 symbols in 3 files

  • src/init/first-minute.ts:L615-L645 — sayEnding (calls, depth 2)
    615: export async function sayEnding(e: Ending, repo: string, opts: {
  • src/init/keep.ts:L184-L252 — keepItems (calls, depth 2)
    186: const at = place ?? { dir: process.cwd(), repoKey: all[0]?.applies && all[0].applies !== 'every repo' ? all[0].applies : '' };
  • src/init/keep.ts:L100-L119 — keepSkill (calls, depth 2)
  • src/init/run.ts:L363-L369 — tele (calls, depth 2)
    368: }, { repo });

Note Reading — 3 symbols in 2 files

  • src/notes/memory-notes.ts:L256-L284 — finishMemoryRead (calls, depth 2)
    257: repo: string,
  • src/notes/pr-reviews.ts:L441-L456 — beginReviewRead (calls, depth 2)
    441: export function beginReviewRead(repo: string, env: NodeJS.ProcessEnv = process.env, gh = "gh", capMs?: number, onLearning?: (f: ReviewFinding) => void): ReviewRead | null {
  • src/notes/pr-reviews.ts:L549-L576 — finishReviewRead (calls, depth 2)
    550: repo: string,

Telemetry Events — 3 symbols in 1 file

  • src/telemetry/track.ts:L109-L139 — track (calls, depth 1)
    115: if (!telemetryOn(ctx.home, ctx.env)) return null;
  • src/telemetry/track.ts:L151-L158 — trackFirstRunIfNew (calls, depth 2)
    153: if (!telemetryOn(ctx.home, ctx.env)) return;
  • src/telemetry/track.ts:L171-L179 — trackInstallIfNew (calls, depth 2)
    173: if (!telemetryOn(ctx.home, ctx.env)) return;

Cloud Nudging — 2 symbols in 1 file

  • src/cloud/nudge.ts:L164-L213 — cloudLines (calls, depth 2)
    164: export function cloudLines(repo: string, opts: NudgeOptions): string[] {
  • src/cloud/nudge.ts:L405-L405 — settled (calls, depth 2)

Session Persistence — 1 symbol in 1 file

  • src/learn/sessions.ts:L281-L309 — finishSessionRead (calls, depth 2)
    282: repo: string,

CLI Skills — 1 symbol in 1 file

  • src/skills/cli-skills.ts:L82-L94 — io (calls, depth 2)
Test signal per changed area — 1 ✓ · 1 ⚠

Reached = a node under a test path has a resolved edge into the changed symbol. It undercounts anything called indirectly — through a CLI, a spawned process or a dynamic import — so read a low ratio as “look here”, never as a coverage gate.

  • ⚠ Session Telemetry — 2 of 2 reached · 6 test files reach it, none changed here
  • ✓ Prompt Voting — 8 of 13 reached · 2 test files changed here: test/learning-vote.test.ts, test/trail-learning-on-touch.test.ts
    • not reached: closeVote, voteClause, turnEnd, fold, asksPerson
29 test suites also reference this code

37 symbols, kept out of the diagram and the table so they cannot crowd out the areas a reviewer has to look at.

  • test/claude-hook-timeouts.test.ts
  • test/claude-hooks-1b.test.ts
  • test/claude-hooks-perf.test.ts
  • test/claude-hooks-skills.test.ts
  • test/claude-hooks.test.ts
  • test/claude-host-standdown.test.ts
  • test/claude-post-bash-walls.test.ts
  • test/claude-prompt-learnings.test.ts
  • test/claude-prompt-private.test.ts
  • test/claude-session-start-learnings.test.ts
  • test/claude-skills-journey-fixtures.ts
  • test/claude-start-skills-1b.test.ts
  • test/claude-stop-trace.test.ts
  • test/claude-stop.test.ts
  • test/claude-tally.test.ts
  • test/hook-savings.test.ts
  • test/summary-view.test.ts
  • test/telemetry-gate.test.ts
  • test/telemetry-sessions.test.ts
  • test/telemetry-skills-2.test.ts
  • …9 more

⚠️ 5 changed files not in the graph (.trail/learnings/2026-10-29-the-did-it-help-vote-rides-anirudhkumar.md, .trail/learnings/2026-10-30-trail-asks-whether-a-learning-helped-anirudhkumar.md, .trail/learnings/2026-11-02-skills-tests-that-check-what-a-anirudhkumar.md, .trail/learnings/2026-11-17-the-did-it-help-vote-counts-anirudhkumar.md, TELEMETRY.md) — no parser claims the extension, or the index predates the file.

graft blast · origin/main...HEAD · depth 2 · 17 changed files

Open the interactive graph → — click an area to see its dependent symbols at file:line.

github-actions Bot added a commit that referenced this pull request Oct 29, 2026
…d-feedback-e9a0c2

# Conflicts:
#	TELEMETRY.md
#	src/telemetry/contract.ts
github-actions Bot added a commit that referenced this pull request Oct 29, 2026
github-actions Bot added a commit that referenced this pull request Oct 30, 2026
github-actions Bot added a commit that referenced this pull request Nov 2, 2026
…d-feedback-e9a0c2

# Conflicts:
#	src/claude/format.ts
#	src/telemetry/contract.ts
#	test/telemetry-contract.test.ts
github-actions Bot added a commit that referenced this pull request Nov 16, 2026
github-actions Bot added a commit that referenced this pull request Nov 17, 2026
github-actions Bot added a commit that referenced this pull request Nov 17, 2026
github-actions Bot added a commit that referenced this pull request Nov 17, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant