Skip to content

Bound how much one connection can grow a collaboration document #659

Open

Description

@HMarzban

Found during the #649 review. Related: #651, #595.

Problem

#651 restores a 16 MiB cap on one inbound WebSocket message. It does not bound how much one connection can add to a document over many messages.

beforeHandleMessage in apps/hocuspocus.server/src/hocuspocus.server.ts checks messages before they apply. It refuses awareness frames over 64 KiB and client BroadcastStateless frames. onStateless drops relayed payloads over 64 KiB. Sync updates have no budget below the message cap.

A loaded room's memory grows with its document size, and every replica that serves the room holds a full copy (apps/hocuspocus.server/CLAUDE.md §Runtime limits). Nothing sets a per-connection bound on that growth today.

Fix

Add a per-connection budget for sync update bytes, checked in beforeHandleMessage, in the same shape as the awareness budget.

  1. Size it from real traffic first. Read ydoc_update_bytes (observed in onChange, src/hocuspocus.server.ts:218) from production: its p99 and its highest filled bucket. It is a histogram, so it holds no exact maximum. The first sync message after a connect can carry a whole document, and the largest stored head was 5,357,733 B on 2026-09-05. The budget must stay well above both.
  2. Count the bytes of each MessageType.Sync message per connection, over a sliding window. Keep the count on context, like the other per-connection state, never in a module-level Map.
  3. Above the budget, refuse the message. Count it in a new counter, and seed it at 0 in seedWsAlertSeries, as §Persistence requires. Log a warn with documentName, socketId, the bytes and the limit. Close that document connection, as the awareness budget does.

Verification

  • Add a control and a check to section [8] of scripts/e2e-versions.ts:
    • Control: a connection that stays under the budget keeps syncing.
    • A connection that goes over it is closed, and the new counter moves by 1.
  • Run the check once without the budget. The close assertion must fail.
  • After the deploy, the new counter must stay at 0 for a week. A real editor that trips it means the budget is too low.

Decisions

  1. Per connection or per document? Recommended: per connection. A per-document budget would let one connection lock every other editor out.
  2. Refuse and close, or switch the connection to read-only? Recommended: refuse and close, the same as the awareness budget. The provider reconnects, and the counter shows it.
  3. The window and the budget? Recommended: set both from the step 1 numbers, and post them on this issue before building.

Activity

  1. added this to the v2.3.0 milestone on Oct 31, 2026
  2. HMarzban commented on Nov 3, 2026

    @HMarzban
    CollaboratorAuthor

    Status, 2026-11-03. Nothing for this issue is built yet. Sync updates still have no budget below the message cap, and the #651 cap is not pushed either. What is left:

    • maintainer: read ydoc_update_bytes p99 and its highest bucket from production, and post the window and budget here;
    • then build the budget, the counter and the [8] checks with a sabotage run;
    • after deploy, the counter stays at 0 for a week.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    DevOpsSecuritySecurity, access control, and data exposure

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions