Skip to content

feat(observability): alert when a voice note is given up, stuck or undelivered - #250

Merged
anantham merged 2 commits into
mainfrom
claude/voice-note-alerts
Nov 17, 2026
Merged

anantham merged 2 commits into
mainfrom
claude/voice-note-alerts

Conversation

@anantham

Copy link
Copy Markdown
Owner

Why

Between 2026-10-17 and 2026-11-14, 21 live voice notes were given up without a transcript, and 27 transcript deliveries went to a deleted Telegram chat. Nobody noticed for a month: the transcription queue logged a warning and moved on. The operator's requirement is that every failure is visible, along with what it cost.

What

A new indrasnet-voice group, built on gauges the IndrasNet telemetry exporter now publishes (anantham/TemporalCoordination#411):

Alert Fires when Severity
IndrasVoiceNoteGivenUp indras_voice_given_up_24h > 0: a voice note's transcription was given up in the last 24 h critical
IndrasVoiceNoteWaitingTooLong the oldest voice note still waiting is over 2 h old, for 10 minutes warning
IndrasVoiceDeliveryFailing indras_voice_delivery_failed_24h > 0: a transcript delivery failed in the last 24 h critical
IndrasVoiceMetricsMissing the exporter can't read voice-note health for 15 minutes, so the alerts above would be blind warning

The 24-hour window means a new give-up alerts once, while the recovered historical ones don't page forever. Alerts reach the operator's Beeper Note to self through the fixed notifier (TC #404).

Verification

  • promtool check rules passes (36 rules).
  • 52 promtool test rules cases pass, 7 of them new: given up / none given up; waiting 3 h / 30 min; delivery failed; metrics missing / OK.
  • Loosening the 2-hour threshold, or the give-up threshold, makes its test fail.

Not live until

  • This merges and Prometheus is restarted.
  • The exporter change (#411) runs, since these metrics don't exist before it.

🤖 Generated with Claude Code

…delivered

New indrasnet-voice group on the telemetry exporter's live voice gauges
(TemporalCoordination#411):

- IndrasVoiceNoteGivenUp (critical): a live voice note's transcription gave up
  in the last 24 h. 21 notes were lost this way 2026-10-17..11-14 unnoticed.
- IndrasVoiceNoteWaitingTooLong (warning): oldest waiting note > 2 h for 10m.
- IndrasVoiceDeliveryFailing (critical): a transcript delivery failed in 24 h
  (until 11-14 every delivery to a deleted Telegram chat failed silently).
- IndrasVoiceMetricsMissing (warning): the exporter cannot read voice health.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
lct_app Ready Ready Preview Nov 17, 2026 2:34pm UTC

…bids

test_alert_rules_cover_loss_pressure_and_host_contention forbids
"transcript" (and authorization/cookie/prompt/response_body) anywhere in the
rules file, so no alert can be built around sensitive content. The voice
alerts only used the word descriptively, but the guard is a substring check
and it should stay strict: reworded to "voice-to-text" / "text". No rule
logic changed. promtool: 36 rules, 52 cases pass; the guard test passes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@anantham
anantham merged commit 8d65f59 into main Nov 17, 2026
8 checks passed

This branch was successfully deployed

1 active deployment
Preview — e5aeb087 Deployed Nov 17, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant