Skip to content

Fix trail's save-a-learning prompts - #739

Merged
anirudhkumar-nanonets merged 5 commits into
mainfrom
claude/fix-save-learning-prompts-84888d
Nov 17, 2026
Merged

anirudhkumar-nanonets merged 5 commits into
mainfrom
claude/fix-save-learning-prompts-84888d

Conversation

@anirudhkumar-nanonets

@anirudhkumar-nanonets anirudhkumar-nanonets commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

Issue: In Claude Desktop, a user's correction never led to a "save a learning" ask. After a long session was compacted, the Stop hook asked about a correction nobody made ("The user corrected you ("This session is being continued…")"). After a push, it said no learning went with the work when one had.
Why: The hook's typed() had its own filter. It dropped anything starting with <, which is every Desktop message (they open with a <system-reminder> block), and it let Claude Code's compaction summary and other sessions' messages through as typed. Its advice also sent business reasoning to the repo, and it counted writes outside the repo as code edits.
Fix: The hook now reads typed words through the shared transcript filter, knows more "keep it out of the repo" phrasings, says business reasoning goes in --personal, and ignores writes outside the repo or pushes whose branch already adds a learning.

Moves: share of user corrections that end in a saved learning. Measured by the new note_asked event in PostHog (Metabase db 11, posthog_event_full_view): kind = corrected with saved in (repo, personal) over all kind = corrected. Value today: none. Nothing measured it before this PR, and db 11 has no note_asked events (checked 2026-11-16).

What changed

  • typed() uses typedText (src/trace/claude.ts), the filter every other learning path already uses. It strips <system-reminder> and IDE wrappers and keeps the words after them. It never counts compaction summaries (isCompactSummary, or text starting "This session is being continued"), task notices or other sessions' messages (origin.kind other than human) as typed. A queued message typed while Claude was busy now counts too.
  • Artifact comments: ownText opens "[Artifact comment sent to Claude]" to its quoted comment between the BEGIN/END lines. Before, the whole harness wrapper counted as typed on every learning path.
  • Correction words: adds na, need not / needn't go|be, shouldn't go, keep it out of the repo, not in the repo (not when the sentence is a question). Tests cover look-alikes that must stay quiet: "is this not in the repo yet?", "you need not ask…", "nan values…", "naming is fine".
  • Repo vs personal advice: session start (format.ts), the trail skill (skill-template.ts) and the corrected and shipped asks now say business, growth, sign-up, pricing or competitor reasoning and internal numbers go in --personal, never the repo, "even if you also keep your own copy" (the answer to why Claude's own memory got the rule and trail didn't). The words test bans "memory" in agent-facing text, hence "your own copy".
  • False "no learning went with it": shellEdit returns no edit when a command starts with cd outside the repo, or when every full path it names is outside the repo (a scratch PR body via $P, Claude's memory index). A write it can't name from inside the repo still counts. The shipped sign is skipped when the branch's commits already add a .trail/learnings/ file (branchAddsLearning in src/cloud/diff.ts, next to branchDiff).
  • Telemetry: note_asked {kind, saved: repo/personal/no}, sent on the stop after an ask (the stop_hook_active stop). askAnswer only counts it when the last Stop hook feedback is trail's own ask, so another hook's block isn't counted. Listed in contract.ts, its allowed values, TELEMETRY.md and the pinned event list.

Choices made

  • A new event rather than a property: hook_ms (the Stop hook's only event today) is sent only when learned skills are enabled, and a property on note_saved alone gives saves without the asks they answer, so no share.
  • The incident's exact correction ("na, these learnings need not go to repo…", behind the Desktop status block) is a test case.
  • Conflicts with other open PRs: contract.ts, TELEMETRY.md and the pinned event list in test/telemetry-contract.test.ts also change in Ask once whether a reused learning helped #727 (adds learning_rated). Main now has twenty-seven events and this PR pins twenty-eight; whichever of the two merges second needs "twenty-nine" and both names.

trail check

Exits 1 with 6 hits, all word overlaps with unrelated learnings, not conflicts:

  • TELEMETRY.md:57 (the note_asked row) and note-signals.ts:152 (the typed() comment) vs hooks going quiet under a sandboxed claude -p (matched on "claude", "stop", "hook", "tool").
  • format.ts:417 (the session-start --personal sentence) vs the hook shim with a real HOME ("copy"), showing the person one line of hook output ("show", "person", "one"), and the old --no-statusline README advice ("keep", "line").
  • note-signals.ts:82 (FULL_PATH) vs replaces: holding ids ("file name").

Tests

  • test/trail-note-ask.test.ts, test/trace-typed.test.ts, test/telemetry-contract.test.ts, test/trail-words.test.ts, test/claude-stop.test.ts: 61 pass.
  • Full suite with a scratch HOME: 142 failing on this branch vs 143 on origin/main (dd0e6cb), the same parser/extraction set from the worktree resolving the main checkout's node_modules. Nothing new fails.
  • After merging main (a44da3d, Fix npm downloads showing 0 after the rename #723 and Install trail without sudo #734): no conflicts. skill-template.ts and the skill fixture keep both this PR's --personal lines and Install trail without sudo #734's no-sudo lines. Typecheck is clean, and the tests above plus cli-meta, node-version, claude-shim-resolve, trail-notes and skills-fixtures pass (130).
  • After merging main again on 2026-11-16 (b22055d): conflicts only in TELEMETRY.md, contract.ts and the pinned event list. Kept main's into on note_saved and its session_shared/share_bundle events, added note_asked beside them. Main hadn't changed note-signals.ts, stop.ts or trace/claude.ts since the last merge, and its typed() still has the old filter, so the fix applies unchanged. Typecheck is clean and the five files above pass (63). Full suite with a scratch HOME: 4,505 tests on this branch, 4,499 on main (b22055d). 3 fail on each, the same three on both: the graft-to-trail move tests, which depend on this machine having trail installed. Nothing new fails.
  • After merging main on 2026-11-17 (8b56ca0, the share-session commits Share sync re-sends only sessions that grew #828, share-session signs in by itself, and needs no flag #836, share-session's sync hook survives a trail upgrade #838, share-session's preview: four sections, one send question #840): no conflicts, and none of them touch the files this PR changes or add telemetry events. Typecheck is clean. Full suite with a scratch HOME: the same three graft-to-trail move tests fail as before; 11 more timed out with the machine's load average near 170 (2 MB linear-time checks, the prompt hook's 400 ms deadline, a 100 ms FIFO gate) and all pass when run on their own. trail check gives the same 6 hits listed above.
  • Fixture run on both heads with each version's noteSignals, a line followed by an edit. main asks "The user corrected you" on a flagged compaction summary, on the same summary without flags, and on an origin: peer message; this branch stays quiet on all three. Both ask on "no, use a channel instead…"; only this branch asks on "na, these learnings need not go to repo" behind the Desktop status block.

🤖 Generated with Claude Code

github-actions Bot added a commit that referenced this pull request Oct 29, 2026
@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

🌱 graft blast radius

4 areas changed → 11 areas can be affected. 42 dependent symbols, depth 2.
Tests: no test reaches Trace Management, branchAddsLearning; 1 area updated its tests.
Tag: Ercan Er — Note Handling, Hook Management · MINJUN KIM — Note Handling · @johnybradshaw — Hook Management

flowchart TB
  A0(("Hook Management<br/>16 symbols"))
  A1(("Initialization Process<br/>7 symbols"))
  A2(("Note Management<br/>4 symbols"))
  A3(("Cloud Session Sharing<br/>4 symbols"))
  A4(("Skill Offerings<br/>2 symbols"))
  AX(("6 smaller areas<br/>9 symbols"))
  classDef reached fill:#D9EDF3,stroke:#3AA7C9,stroke-width:1.5px,color:#0E313C;
  class A0,A1,A2,A3,A4 reached;
  classDef tail fill:#EEF2F3,stroke:#9AA4A9,stroke-width:1px,color:#3A4247;
  class AX tail;
Loading
Can be affected Symbols Nearest hop Reached from
Hook Management 16 src/claude/hooks.ts:L516-L526 orientationAtStart — calls, depth 1 Note Handling, Contract Management
Initialization Process 7 src/init/keep.ts:L1-L387 keep.ts — imports, depth 1 Contract Management, Trace Management
Note Management 4 src/notes/memory-notes.ts:L1-L285 memory-notes.ts — imports, depth 1 Contract Management, Trace Management
Cloud Session Sharing 4 src/cloud/nudge.ts:L1-L508 nudge.ts — imports, depth 1 Contract Management, Trace Management
Skill Offerings 2 src/skills/cli-skills.ts:L1-L376 cli-skills.ts — imports, depth 2 Contract Management, Trace Management
Host Registry 2 src/hosts/antigravity.ts:L43-L61 installAntigravitySkill — calls, depth 2 Note Handling
5 smaller areas 7 Session Management, Telemetry Tracking, Command Line Interface, … see below
Who knows this code — 4 people across 15 areas
Area Who knows it
Note Handling · changed Ercan Er — 1 commit, last 8d ago · MINJUN KIM — 1 commit, last 8d ago
Hook Management · affected Ercan Er — 1 commit, last 8d ago · @johnybradshaw — 1 commit, last 8d ago
Host Registry · affected ezadEzanee — 1 commit, last 8d ago
Contract Management · changed only you — nobody else has touched these files
Trace Management · changed only you — nobody else has touched these files
branchAddsLearning · changed only you — nobody else has touched these files
Initialization Process · affected only you — nobody else has touched these files
Note Management · affected only you — nobody else has touched these files
…7 further areas

Ownership is git history over each area's own files, weighted towards recent work (120-day half-life). Merge commits and bots are dropped, and you are dropped from your own PR. A name with no @ has no GitHub handle in its commit email — tag them by hand, or add a .mailmap entry. A suggestion from history, not a CODEOWNERS rule.

All 42 dependent symbols, grouped by area

Hook Management — 16 symbols in 13 files

  • src/claude/hooks.ts:L516-L526 — orientationAtStart (calls, depth 1)
  • src/claude/hooks.ts:L1-L628 — hooks.ts (imports, depth 1)
  • src/claude/session-metrics.ts:L1-L230 — session-metrics.ts (imports, depth 1)
  • src/claude/skills-wiring.ts:L1-L116 — skills-wiring.ts (imports, depth 1)
  • src/claude/state.ts:L1-L165 — state.ts (imports, depth 1)
  • src/claude/hook-io.ts:L1-L140 — hook-io.ts (imports, depth 2)
  • src/claude/hooks.ts:L537-L627 — main (calls, depth 2)
  • src/claude/init.ts:L136-L184 — runInit (calls, depth 2)
  • src/claude/init.ts:L197-L256 — runTrailInit (calls, depth 2)
  • src/claude/prompt-hook.ts:L1-L309 — prompt-hook.ts (imports, depth 2)
  • src/claude/prompt-skills.ts:L1-L327 — prompt-skills.ts (imports, depth 2)
  • src/claude/skill-use.ts:L1-L80 — skill-use.ts (imports, depth 2)
  • src/claude/stale-count-run.ts:L1-L104 — stale-count-run.ts (imports, depth 2)
  • src/claude/stats.ts:L1-L27 — stats.ts (imports, depth 2)
  • src/claude/statusline.ts:L1-L40 — statusline.ts (imports, depth 2)
  • src/claude/sync-run.ts:L1-L77 — sync-run.ts (imports, depth 2)

Initialization Process — 7 symbols in 6 files

  • src/init/keep.ts:L1-L387 — keep.ts (imports, depth 1)
  • src/init/run.ts:L1-L483 — run.ts (imports, depth 1)
  • src/init/keep-list.ts:L161-L203 — afterList (calls, depth 2)
  • src/init/keep-list.ts:L210-L224 — typedLine (calls, depth 2)
  • src/init/replay-runs.ts:L445-L478 — typedAt (calls, depth 2)
  • src/init/review-screen.ts:L1-L319 — review-screen.ts (imports, depth 2)
  • src/init/transcript.ts:L171-L223 — scanTranscript (calls, depth 2)

Note Management — 4 symbols in 4 files

  • src/notes/memory-notes.ts:L1-L285 — memory-notes.ts (imports, depth 1)
  • src/notes/pr-reviews.ts:L1-L577 — pr-reviews.ts (imports, depth 1)
  • src/notes/always-follow-gate.ts:L39-L63 — typedLines (calls, depth 2)
  • src/notes/claude-export.ts:L1-L128 — claude-export.ts (imports, depth 2)

Cloud Session Sharing — 4 symbols in 4 files

  • src/cloud/nudge.ts:L1-L508 — nudge.ts (imports, depth 1)
  • src/cloud/lookup-activity.ts:L1-L60 — lookup-activity.ts (imports, depth 2)
  • src/cloud/pair.ts:L1-L368 — pair.ts (imports, depth 2)
  • src/cloud/share-session.ts:L174-L185 — firstPrompt (calls, depth 2)

Skill Offerings — 2 symbols in 2 files

  • src/skills/cli-skills.ts:L1-L376 — cli-skills.ts (imports, depth 2)
  • src/skills/offer-lines.ts:L33-L50 — typedLineAt (calls, depth 2)

Host Registry — 2 symbols in 2 files

  • src/hosts/antigravity.ts:L43-L61 — installAntigravitySkill (calls, depth 2)
  • src/hosts/registry.ts:L1-L200 — registry.ts (references, depth 2)

Session Management — 2 symbols in 1 file

  • src/learn/sessions.ts:L1-L310 — sessions.ts (imports, depth 1)
  • src/learn/sessions.ts:L57-L110 — sessionLines (calls, depth 2)

Telemetry Tracking — 2 symbols in 2 files

  • src/telemetry/sessions.ts:L1-L113 — sessions.ts (imports, depth 1)
  • src/telemetry/track.ts:L1-L180 — track.ts (imports, depth 1)

Command Line Interface — 1 symbol in 1 file

  • src/cli.ts:L1-L3477 — cli.ts (imports, depth 2)

Tool Management — 1 symbol in 1 file

  • src/mcp/tools.ts:L1-L352 — tools.ts (imports, depth 2)

Dialect Tracking — 1 symbol in 1 file

  • src/trace/dialect.ts:L114-L121 — typedOnLine (calls, depth 2)
Test signal per changed area — 1 ✓ · 2 ✗ · 1 –

Reached = a node under a test path has a resolved edge into the changed symbol. It undercounts anything called indirectly — through a CLI, a spawned process or a dynamic import — so read a low ratio as “look here”, never as a coverage gate.

  • – Contract Management — 1 test file changed here: test/telemetry-contract.test.ts
  • ✗ Trace Management — 0 of 1 reached · no test file reaches it
    • not reached: ownText
  • ✓ Note Handling — 4 of 10 reached · 1 test file changed here: test/trail-note-ask.test.ts
    • not reached: home, typed, outside, shellEdit, trailSkill, askForNote
  • ✗ branchAddsLearning — 0 of 1 reached · no test file reaches it
    • not reached: branchAddsLearning
124 test suites also reference this code

125 symbols, kept out of the diagram and the table so they cannot crowd out the areas a reviewer has to look at.

  • test/agent-trust-merge-1b.test.ts
  • test/claude-copy-dashes.test.ts
  • test/claude-export.test.ts
  • test/claude-format.test.ts
  • test/claude-hooks-1b.test.ts
  • test/claude-hooks-perf.test.ts
  • test/claude-hooks-skills.test.ts
  • test/claude-hooks.test.ts
  • test/claude-host-standdown.test.ts
  • test/claude-orientation-brand.test.ts
  • test/claude-post-bash-walls.test.ts
  • test/claude-prompt-learnings.test.ts
  • test/claude-prompt-private.test.ts
  • test/claude-session-start-learnings.test.ts
  • test/claude-skill-template.test.ts
  • test/claude-skills-journey-fixtures.ts
  • test/claude-start-skills-1b.test.ts
  • test/claude-start-skills-2.test.ts
  • test/claude-state.test.ts
  • test/claude-statusline.test.ts
  • …104 more

⚠️ 4 changed files not in the graph (.trail/learnings/2026-10-29-the-stop-hook-reads-what-the-anirudhkumar.md, .trail/learnings/2026-11-17-when-it-s-unclear-who-wrote-anirudhkumar.md, TELEMETRY.md, test/fixtures/skills/trail/SKILL.md) — no parser claims the extension, or the index predates the file.

graft blast · refs/graft/base...HEAD · depth 2 · 14 changed files

Open the interactive graph → — click an area to see the code that changed, and the line that reaches it.

github-actions Bot added a commit that referenced this pull request Nov 16, 2026
github-actions Bot added a commit that referenced this pull request Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant