fix(session-recorder): prevent unbounded error loop from corrupted TextDecoder in emit by NickSolante · Pull Request #305 · hyperdxio/hyperdx-js

NickSolante · 2026-08-11T01:33:24Z

Fixes #303

Problem

On Safari, the session recorder's emit() could enter a permanent, unbounded error loop: WebKit's TextDecoder can permanently corrupt after high cumulative decode volume and then throw RangeError: Bad value on valid input (WebKit #286266, wasm-bindgen report). emit() reused one module-level decoder for the page lifetime with no error handling, so a single corrupted instance turned every rrweb mutation flush into an uncaught error (~40/sec, observed 64,606 errors over 26 minutes from one session) until page unload. Full telemetry in #303.

Changes

Single-chunk events (the overwhelmingly common case) skip the encode→decode round-trip entirely the JSON string is already in hand, so most events never touch TextDecoder. This also removes the cumulative decode volume that ages the decoder toward WebKit's corruption threshold.
Multi-chunk events use a fresh per-event TextDecoder with { stream: true }, which also fixes a latent all-browser bug: byte-slice boundaries that bisected a multi-byte UTF-8 character previously produced U+FFFD replacement characters at chunk seams, silently corrupting reassembled replay JSON.
A failed decode is retried once with a new decoder, restarting from chunk 0 so streaming state stays consistent.
Circuit breaker: emit() failures are caught (each dropped event is surfaced via console.error), and after 10 consecutive failures the recorder stops instead of erroring on every event forever. init() resets the breaker state so a later recording lifetime starts clean.
Chunks are all converted before any is sent, so a mid-loop failure can't export a partial, unreassemblable chunk set.

Tests

splitIntoChunks unit tests: single-chunk identity (no decode), lossless multi-chunk reassembly, a multi-byte character bisected at the chunk boundary (fails against the old non-streaming decode), and recovery via the fresh-decoder retry (first decoder instance always throws RangeError: Bad value).
emit() error-handling tests (mocked rrweb/exporter/tracer): failures don't throw into rrweb dispatch, the breaker trips on the 10th consecutive failure and stops recording, nothing is emitted after tripping, and recording works again after a re-init.

compile:tsc clean, jest 10/10 passing.

Notes

Includes a changeset (patch bump for @hyperdx/otel-web-session-recorder).
Pre-existing and untouched here: MutationRateLimiter's constructor starts a setInterval that is never cleared (it also survives deinit()); the emit tests use fake timers to work around it. Happy to fix in a follow-up.

changeset-bot · 2026-08-11T01:33:29Z

🦋 Changeset detected

Latest commit: f66c0e4

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages

Name	Type
@hyperdx/otel-web-session-recorder	Patch
@hyperdx/browser	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

greptile-apps · 2026-08-11T01:35:39Z

Confidence Score: 5/5

[Medium risk] Fixes error handling in session recording event emission.

The PR appears safe to merge.

Summary

The PR hardens session recording against persistent Safari TextDecoder failures and preserves UTF-8 data across chunk boundaries.

Avoids decoding for single-chunk events and uses fresh streaming decoders for multi-chunk events.
Adds retry and circuit-breaker handling around event emission.
Adds unit coverage and a patch changeset.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[rrweb event] --> B[Serialize and enforce message size]
  B --> C{Fits one chunk?}
  C -->|Yes| D[Use original string]
  C -->|No| E[Encode and stream-decode chunks]
  E --> F{Decode succeeds?}
  F -->|No| G[Retry with fresh decoder]
  F -->|Yes| H[Convert all chunks]
  G --> H
  D --> H
  H --> I[Queue logs]
  B -->|Failure| J[Increment consecutive failures]
  E -->|Retry fails| J
  H -->|Failure| J
  J --> K{10 failures?}
  K -->|No| A
  K -->|Yes| L[Stop recorder]

_{Reviews (7) · Last reviewed commit: "fix(session-recorder): prevent unbounded..." · Reviewed by Greptile}

…xtDecoder in emit WebKit's TextDecoder can permanently corrupt after high cumulative decode volume and then throw `RangeError: Bad value` on valid input (https://bugs.webkit.org/show_bug.cgi?id=286266). Because emit() reused one module-level decoder with no error handling, a single corrupted instance turned every rrweb flush into an uncaught error (~40/sec) until page unload. - Skip the encode/decode round-trip entirely for single-chunk events, with a length*3 fast path that avoids encoding on the hot path at all - Use a fresh per-event TextDecoder for multi-chunk events, with { stream: true } so chunk boundaries no longer bisect multi-byte characters into U+FFFD - Retry a failed decode once with a new decoder - Stop the recorder after 10 consecutive emit failures instead of erroring on every event forever Circuit-breaker hardening (from review): - init() resets paused/consecutiveEmitFailures so a re-init after a breaker trip records again with the full failure tolerance - Every emit failure is surfaced via diag.error, not just in debug - All chunks are converted before any is sent, so a mid-loop failure can't export a partial (unreassemblable) chunk set - The breaker's deinit() can't throw back into rrweb, deinit() clears inited before calling stop, and init() finishes teardown if the trip happened during rrweb's synchronous initial snapshot - emit tests are now self-contained per it-block Fixes hyperdxio#303

teeohhem · 2026-09-21T19:47:56Z

+    expect(mockOnLog).toHaveBeenCalledTimes(1);
+    expect(mockStopRecording).toHaveBeenCalledTimes(1);
+  });
+});


I don't think these tests are actually running anywhere in CI. Mind updating the session-recorder/package.json to add add "ci:unit": "jest --ci" ? That way they will get run with the main ci:unit job on the PR.

teeohhem · 2026-09-21T19:51:41Z

+            diag.error(
+              'OpenTelemetry Session Recorder: stopping recording after repeated errors',
+              e,
+            );


just a note here: diag.error only shows when otel-web's debug is true. Do we want these errors to be more visible to the user? If so, maybe we emit a span with this error instead.

ah, thanks for letting me know. Yea I think we would definitely want the users to know if somethings started breaking.

teeohhem · 2026-09-21T19:52:14Z

@NickSolante Just a few comments/questions. Sorry for the delay!

teeohhem · 2026-10-06T13:32:00Z

@NickSolante bump

NickSolante · 2026-10-07T07:57:22Z

@NickSolante Just a few comments/questions. Sorry for the delay!

All good, apologies as well will get in this by eow @teeohhem

…xtDecoder in emit WebKit's TextDecoder can permanently corrupt after high cumulative decode volume and then throw `RangeError: Bad value` on valid input (https://bugs.webkit.org/show_bug.cgi?id=286266). Because emit() reused one module-level decoder with no error handling, a single corrupted instance turned every rrweb flush into an uncaught error (~40/sec) until page unload. - Skip the encode/decode round-trip entirely for single-chunk events - Use a fresh per-event TextDecoder for multi-chunk events, with { stream: true } so chunk boundaries no longer bisect multi-byte characters into U+FFFD - Retry a failed decode once with a new decoder - Stop the recorder after 10 consecutive emit failures instead of erroring on every event forever

github-actions Bot added the external Opened by an external contributor label Aug 11, 2026

greptile-apps Bot reviewed Aug 11, 2026

View reviewed changes

Comment thread packages/session-recorder/src/index.ts

NickSolante force-pushed the fix/session-recorder-decoder-crash-loop branch from cb52d8d to c43ff08 Compare August 11, 2026 01:44

NickSolante force-pushed the fix/session-recorder-decoder-crash-loop branch from c43ff08 to ac43184 Compare August 11, 2026 01:46

Merge branch 'main' into fix/session-recorder-decoder-crash-loop

245f970

karl-power assigned teeohhem Aug 11, 2026

teeohhem removed their assignment Sep 2, 2026

Merge branch 'main' into fix/session-recorder-decoder-crash-loop

e42b015

teeohhem reviewed Sep 21, 2026

View reviewed changes

NickSolante force-pushed the fix/session-recorder-decoder-crash-loop branch from 52a798e to f66c0e4 Compare October 7, 2026 10:21

Navigation Menu

Repository navigation

fix(session-recorder): prevent unbounded error loop from corrupted TextDecoder in emit - #305

fix(session-recorder): prevent unbounded error loop from corrupted TextDecoder in emit#305
NickSolante wants to merge 4 commits into
hyperdxio:mainfrom
NickSolante:fix/session-recorder-decoder-crash-loop

NickSolante commented Aug 11, 2026 •

edited

Loading

Uh oh!

changeset-bot Bot commented Aug 11, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Aug 11, 2026 •

edited

Loading

Uh oh!

Uh oh!

teeohhem Sep 21, 2026

Uh oh!

teeohhem Sep 21, 2026

Uh oh!

NickSolante Oct 7, 2026

Uh oh!

teeohhem commented Sep 21, 2026

Uh oh!

teeohhem commented Oct 6, 2026

Uh oh!

NickSolante commented Oct 7, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

NickSolante commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Changes

Tests

Notes

Uh oh!

changeset-bot Bot commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

greptile-apps Bot commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

teeohhem Sep 21, 2026

Choose a reason for hiding this comment

Uh oh!

teeohhem Sep 21, 2026

Choose a reason for hiding this comment

Uh oh!

NickSolante Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

teeohhem commented Sep 21, 2026

Uh oh!

teeohhem commented Oct 6, 2026

Uh oh!

NickSolante commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

NickSolante commented Aug 11, 2026 •

edited

Loading

changeset-bot Bot commented Aug 11, 2026 •

edited

Loading

greptile-apps Bot commented Aug 11, 2026 •

edited

Loading

NickSolante commented Oct 7, 2026 •

edited

Loading