Skip to content

497dfac1: crm activity fails with workspace_source_too_large on prod - read org deals via a paged scoped scan - #321

Merged
alexeygrigorev merged 1 commit into
mainfrom
fix/crm-activity-paginate
Nov 4, 2026
Merged

alexeygrigorev merged 1 commit into
mainfrom
fix/crm-activity-paginate

Conversation

@alexeygrigorev

@alexeygrigorev alexeygrigorev commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

Task: 497dfac1 (DataOps CRM: crm activity fails with workspace_source_too_large on prod - paginate, read only what's needed)

Problem

dataops crm activity organization <id> (default types include deal-history) returned HTTP 500 workspace_source_too_large on prod-sized data.

Where the whole workspace source was requested in one shot:

  • backend/src/crmWorkspace/activity.ts:231 (on origin/main): `const deals = (await listCrmRecords(client, 'deal')).filter(...)` ran for every organization activity read.
  • `listCrmRecords` (`backend/src/db/sponsorCrm.ts:139`) -> `scanCrmPrefix` -> inside the request's `withCrmReadSnapshot` scope (every GET on `/api/sponsor-crm/*` is wrapped: `backend/src/router.ts:372`) it triggered `scanCrmTable(client, SNAPSHOT_PREFIXES)` (`backend/src/db/sponsorCrm.ts:135`) - a single full-workspace Scan over nine PK prefixes with `ConsistentRead`, accumulated in memory under `SNAPSHOT_STORAGE_LIMITS` (250k items / 128 MB / 30 s, `backend/src/db/boundedPagination.ts:8`).
  • On prod-sized sources the scan trips those bounds -> `StoragePaginationError` -> `workspaceErrorOf(error, 'activity')` (`activity.ts:311`, `reads.ts:23`) -> `sourceTooLarge` -> 500 `workspace_source_too_large` (`backend/src/crmWorkspace/reads.ts:15`).

Fix

  • New `listOrganizationDeals(client, organizationId)` in `backend/src/db/sponsorCrm.ts`: a `DEAL#`-prefix Scan filtered to `organizationId`, paginated with the existing `collectBoundedPages` pattern used by `scanCrmTable`. It never populates or reads the request snapshot, and only this organization's deals accumulate inside the page loop.
  • `readActivity` uses it for the `deal-history` source; per-deal history partitions keep their existing keyed, bounded queries.
  • The `reads.snapshot` disclosure now reports what is actually scanned: `['DEAL#']` when the org scan runs, `[]` otherwise (no whole-workspace snapshot scan happens anymore).
  • CLI needed no change: `dataops crm activity` already sends `--limit`/`--cursor` and prints `More: --cursor ...`.

Test

`backend/tests/crm-workspace-activity.test.ts`:

  • New test "assembles deal history from a paged organization-scoped scan, never the workspace snapshot": caps every Scan page at 25 evaluated items (simulating a prod-sized multi-page source), pages the whole timeline with `limit=2`, asserts no duplicate items, exact deal-history coverage of all 5 deals, that the scan took >10 storage pages, and that every observed Scan is the org-scoped one - the workspace snapshot scan (the too-large request) never fires. Verified it fails against the unfixed source.
  • Updated the existing reads-disclosure assertion to the truthful scoped scan.

Command and results (targeted file only; full suite runs in CI):

```
cd backend && NODE_ENV=test FRONTEND_ROOT=../frontend SKIP_AUTH=true node --import tsx --test --test-force-exit tests/crm-workspace-activity.test.ts

tests 13, pass 13, fail 0

```

Also green: `tests/crm-workspace-views-notes.test.ts`, `tests/crm-workspace-directory.test.ts`, `tests/sponsor-crm-read-snapshot.test.ts` (32 tests), and `npm run typecheck` in `backend/`.

Remaining gaps

  • `relatedWork` still lists all workspace open tasks (GSI-Status queries via `listTasksByStatus`). That read is keyed per status and does not produce `workspace_source_too_large`; removing the whole-open-task read needs a subject-keyed index (offline index change per `docs/dynamodb-index-changes.md`) and is out of scope here.
  • The scoped scan still physically reads the table's pages (DynamoDB applies filters after the 1 MB page boundary), matching the existing `DEAL#` scan pattern in `sponsorFinance/alerts.ts`; a org->deal adjacency or GSI would remove that cost but requires a write-path/backfill change.

…he workspace snapshot

crm activity organization <id> with default types fetched its deal history
through listCrmRecords, which inside the request's read snapshot scans the
whole workspace source in one shot (nine prefixes, ConsistentRead) under the
SNAPSHOT_STORAGE_LIMITS bounds. Prod-sized sources exceed those bounds, the
scan raises StoragePaginationError, and readActivity surfaces it as a 500
workspace_source_too_large instead of the record's activity.

Deal history now comes from listOrganizationDeals: a DEAL#-prefix scan
filtered to the organization, paginated with the existing collectBoundedPages
pattern. Only the org's deals accumulate, the snapshot is never populated or
read, and the reads disclosure reports the scan truthfully (['DEAL#'] when
the org scan runs, [] otherwise).

The new test caps every scan page at 25 evaluated items so the org's deals
arrive across many storage pages, pages the whole timeline with limit 2,
asserts no duplicates, exact deal-history coverage, and that every scan is
the org-scoped one - the whole-workspace snapshot scan (the too-large
request) never fires. It fails against the previous implementation.
@alexeygrigorev
alexeygrigorev merged commit 40d454c into main Nov 4, 2026
4 checks passed
@alexeygrigorev
alexeygrigorev deleted the fix/crm-activity-paginate branch November 5, 2026 15:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant