Repository navigation
497dfac1: crm activity fails with workspace_source_too_large on prod - read org deals via a paged scoped scan - #321
Merged
Conversation
…he workspace snapshot crm activity organization <id> with default types fetched its deal history through listCrmRecords, which inside the request's read snapshot scans the whole workspace source in one shot (nine prefixes, ConsistentRead) under the SNAPSHOT_STORAGE_LIMITS bounds. Prod-sized sources exceed those bounds, the scan raises StoragePaginationError, and readActivity surfaces it as a 500 workspace_source_too_large instead of the record's activity. Deal history now comes from listOrganizationDeals: a DEAL#-prefix scan filtered to the organization, paginated with the existing collectBoundedPages pattern. Only the org's deals accumulate, the snapshot is never populated or read, and the reads disclosure reports the scan truthfully (['DEAL#'] when the org scan runs, [] otherwise). The new test caps every scan page at 25 evaluated items so the org's deals arrive across many storage pages, pages the whole timeline with limit 2, asserts no duplicates, exact deal-history coverage, and that every scan is the org-scoped one - the whole-workspace snapshot scan (the too-large request) never fires. It fails against the previous implementation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Task: 497dfac1 (DataOps CRM:
crm activityfails with workspace_source_too_large on prod - paginate, read only what's needed)Problem
dataops crm activity organization <id>(default types includedeal-history) returned HTTP 500workspace_source_too_largeon prod-sized data.Where the whole workspace source was requested in one shot:
backend/src/crmWorkspace/activity.ts:231(onorigin/main): `const deals = (await listCrmRecords(client, 'deal')).filter(...)` ran for every organization activity read.Fix
Test
`backend/tests/crm-workspace-activity.test.ts`:
Command and results (targeted file only; full suite runs in CI):
```
cd backend && NODE_ENV=test FRONTEND_ROOT=../frontend SKIP_AUTH=true node --import tsx --test --test-force-exit tests/crm-workspace-activity.test.ts
tests 13, pass 13, fail 0
```
Also green: `tests/crm-workspace-views-notes.test.ts`, `tests/crm-workspace-directory.test.ts`, `tests/sponsor-crm-read-snapshot.test.ts` (32 tests), and `npm run typecheck` in `backend/`.
Remaining gaps