Repository navigation
Ask once whether a reused learning helped - #727
anirudhkumar-nanonets wants to merge 12 commits into
Conversation
🌱 graft blast radius2 areas changed → 7 areas can be affected. 24 dependent symbols, depth 2. flowchart TB
A0(("Learning Tracking<br/>10 symbols"))
A1(("Initialization<br/>4 symbols"))
A2(("Note Reading<br/>3 symbols"))
A3(("Telemetry Events<br/>3 symbols"))
A4(("Cloud Nudging<br/>2 symbols"))
AX(("2 smaller areas<br/>2 symbols"))
classDef reached fill:#D9EDF3,stroke:#3AA7C9,stroke-width:1.5px,color:#0E313C;
class A0,A1,A2,A3,A4 reached;
classDef tail fill:#EEF2F3,stroke:#9AA4A9,stroke-width:1px,color:#3A4247;
class AX tail;
Who knows this code — 4 people across 9 areas
Ownership is git history over each area's own files, weighted towards recent work (120-day half-life). Merge commits and bots are dropped, and you are dropped from your own PR. A name with no All 24 dependent symbols, grouped by areaLearning Tracking — 10 symbols in 4 files
Initialization — 4 symbols in 3 files
Note Reading — 3 symbols in 2 files
Telemetry Events — 3 symbols in 1 file
Cloud Nudging — 2 symbols in 1 file
Session Persistence — 1 symbol in 1 file
CLI Skills — 1 symbol in 1 file
Test signal per changed area — 1 ✓ · 1 ⚠Reached = a node under a test path has a resolved edge into the changed symbol. It undercounts anything called indirectly — through a CLI, a spawned process or a dynamic import — so read a low ratio as “look here”, never as a coverage gate.
29 test suites also reference this code37 symbols, kept out of the diagram and the table so they cannot crowd out the areas a reviewer has to look at.
Open the interactive graph → — click an area to see its dependent symbols at file:line. |
…d-feedback-e9a0c2 # Conflicts: # TELEMETRY.md # src/telemetry/contract.ts
…d-feedback-e9a0c2 # Conflicts: # src/claude/format.ts # src/telemetry/contract.ts # test/telemetry-contract.test.ts
The issue: Trail couldn't tell whether a learning it handed Claude actually helped, so the launch goal "every internal user loves it" had no number.
Why: The only sign of reuse is Claude's
📝 Started from …'s learningcredit line, and nothing ever asked the person about it (the oldlearning_reusedevent was removed).The fix: Once a session, in a reply that asks nothing else, Claude ends that credit line with "did it help? yes/no"; the next message becomes a
learning_ratedevent (a yes or "it helped" is up, a short no or "didn't help" is down) with only the vote and the learning's id.Moves: share of reused learnings rated helpful (
learning_ratedwithvote = upoverup+down, PostHogposthog_event_full_view, Metabase db 11). Today: 0 events (new event).What changed
src/claude/learning-vote.ts(new): the ask text, reading the vote, marking it pending at Stop (strict when the turn asks something else too), settling it on the next prompt, closing it asnonewhen the session closes unanswered.src/claude/format.ts: the credit instruction innotesForPrompt/notesOnTouchgains "if it asks the user nothing else, end that line with ' · did it help? yes/no'" until the session has asked, and remembers the id of the top learning shown. This is the default full view. Withtrail summary one-line, learnings come as one line with no credit request, so nothing is asked.src/claude/stop.ts:turnEndmarks the vote pending whenlast_assistant_messagehas a 📝 line carrying "did it help? yes/no". If that reply, or any later one before the answer, asks something else (a sentence ending in "?", a "(y/n)", or a tool question with answer buttons), the vote turns strict. Same single session write as before.src/claude/tally.ts: the transcript read already done at Stop also notes whether the open turn called a tool with anask(a question shown as buttons that answer with a new message).src/claude/prompt-hook.ts: the next prompt settles it. "It helped" / "didn't help" anywhere decides it. Otherwise a leading yes word (yes, yeah, yep, sure, haan, sí, ja…), 👍 or+1→up; 👎,-1, or a no word (no, nope, nah, not really, nahi…) in a reply of up to four words or one that mentions help →down; anything else →none. In strict mode only the help phrases count. The prompt goes on to Claude unchanged.src/telemetry/sessions.ts: the idle sweep that closes a session forsession_summary(untouched two hours, at the next session start) also sends a still-waiting vote asnone.src/telemetry/contract.ts+TELEMETRY.md: newlearning_ratedevent withvote(closed set) andlearning_id(checked by shape,l-+ hex, so a title can't ride along).Calls I made
noneis sent when the next message isn't a vote, and for a session closed with the question unanswered, so we also get the answer rate.y/n: trail's own "Commit it with this change? (y/n)" can end a reply..trail/learnings/.format.ts, theVALUEStable incontract.ts, and the pinned event list intelemetry-contract.test.ts(now twenty-eight events, withlearning_ratednext to main'ssession_sharedandshare_bundle). Main's one-line view, stale marking and saved-token count stay as main has them.trail share-session). No conflicts: none of those touch a file this PR changes, and they add no telemetry event, so the event list stays at twenty-eight.stop.ts(Fix trail's save-a-learning prompts #739'saskAnswer,branchAddsLearningandtracknext to this PR'smarkVoteAsked), and the pinned event list, now twenty-nine with Fix trail's save-a-learning prompts #739'snote_askedand this PR'slearning_rated. Fix trail's save-a-learning prompts #739's Stop change reads the session again afterturnEndwrites it, so it keeps the pending vote.TELEMETRY.md's event table (this PR'slearning_ratedrow next to init ends by showing Trail on your own last task (prototype) #757's longercold_startrow). Both kept.trail share-session): no conflicts.Decided: yes/no in words, not thumbs
Nobody types a thumbs emoji in a terminal, so thumbs would mostly have measured how hard it is to answer. The question is now "did it help? yes/no", and a next message starting with the word yes or no is the vote. Thumbs and
+1/-1still count. A clickable yes/no isn't possible here: hooks can only add text to Claude's reply, and Claude's own multiple-choice question would stop the turn until answered. The decision ships as a learning in.trail/learnings/.Follow-ups (not in this PR)
last_assistant_message, so no vote is recorded there yet. Spotting the ask in the transcript read (src/claude/tally.ts) would cover them.Checks
npm testwith a scratch HOME, 4,548 tests, 4,534 pass, 5 fail, 9 skipped. All 5 fail the same way on a cleanorigin/mainon this laptop: 3 ingraft-to-trail,hosts-trailandupkeep-hooks, and 2 first-minute tests from init ends by showing Trail on your own last task (prototype) #757 that print "you" when the local git user wrote the learnings they list. None is in code this PR touches.trail check: no conflicts.test/learning-vote.test.ts: the ask once a session and only when the reply asks nothing else; the vote table (yes words, a short no, help phrases anywhere, idioms like "no worries", never a bare y/n); strict mode after another question, from the reply's text and from a question tool in the transcript; Stop → prompt → one event with no prompt text in it; a session closed unanswered sendsnoneonce, and a late yes after that counts nothing; the contract dropping a non-idlearning_id.deadlineClockfor the skills prompt tests. This PR givesskills-session-start'saskthe same clock, because a shared Windows runner ran past the 400 ms deadline there and kept no block.🤖 Generated with Claude Code