Repository navigation
Description
Found during the #649 review. Related: #651, #595.
Problem
#651 restores a 16 MiB cap on one inbound WebSocket message. It does not bound how much one connection can add to a document over many messages.
beforeHandleMessage in apps/hocuspocus.server/src/hocuspocus.server.ts checks messages before they apply. It refuses awareness frames over 64 KiB and client BroadcastStateless frames. onStateless drops relayed payloads over 64 KiB. Sync updates have no budget below the message cap.
A loaded room's memory grows with its document size, and every replica that serves the room holds a full copy (apps/hocuspocus.server/CLAUDE.md §Runtime limits). Nothing sets a per-connection bound on that growth today.
Fix
Add a per-connection budget for sync update bytes, checked in beforeHandleMessage, in the same shape as the awareness budget.
- Size it from real traffic first. Read
ydoc_update_bytes(observed inonChange,src/hocuspocus.server.ts:218) from production: its p99 and its highest filled bucket. It is a histogram, so it holds no exact maximum. The first sync message after a connect can carry a whole document, and the largest stored head was 5,357,733 B on 2026-09-05. The budget must stay well above both. - Count the bytes of each
MessageType.Syncmessage per connection, over a sliding window. Keep the count oncontext, like the other per-connection state, never in a module-levelMap. - Above the budget, refuse the message. Count it in a new counter, and seed it at 0 in
seedWsAlertSeries, as §Persistence requires. Log a warn withdocumentName,socketId, the bytes and the limit. Close that document connection, as the awareness budget does.
Verification
- Add a control and a check to section
[8]ofscripts/e2e-versions.ts:- Control: a connection that stays under the budget keeps syncing.
- A connection that goes over it is closed, and the new counter moves by 1.
- Run the check once without the budget. The close assertion must fail.
- After the deploy, the new counter must stay at 0 for a week. A real editor that trips it means the budget is too low.
Decisions
- Per connection or per document? Recommended: per connection. A per-document budget would let one connection lock every other editor out.
- Refuse and close, or switch the connection to read-only? Recommended: refuse and close, the same as the awareness budget. The provider reconnects, and the counter shows it.
- The window and the budget? Recommended: set both from the step 1 numbers, and post them on this issue before building.
Activity
HMarzban commentedon Nov 3, 2026
Status, 2026-11-03. Nothing for this issue is built yet. Sync updates still have no budget below the message cap, and the #651 cap is not pushed either. What is left:
ydoc_update_bytesp99 and its highest bucket from production, and post the window and budget here;[8]checks with a sabotage run;