Skip to content

compress: separate dictionary IDs from error returns - #1

Open
trestoncuzzort wants to merge 4 commits into
devfrom
compress-high-dict-id
Open

trestoncuzzort wants to merge 4 commits into
devfrom
compress-high-dict-id

Conversation

@trestoncuzzort

@trestoncuzzort trestoncuzzort commented Oct 26, 2026 •

Copy link
Copy Markdown
Owner

Validates the fix for facebook/zstd#4580. High dictionary IDs allowed for private use overlap encoded size_t errors on 32-bit builds. Internal loaders now return status and write the ID separately.

The source patch is ac6e2475a827ad0e373bd3844f0e2df01c781b5d on fix-high-dict-id. This validation branch adds workflow changes only; all non-workflow files match the source patch.

Validation:

  • The new regression fails on the original 32-bit build. Fixed 32-bit and 64-bit basic suites and 100 seeded randomized iterations pass on each. Three public dictionary APIs, four IDs, frame IDs and byte round trips are checked.
  • All 104 original GitHub jobs report success at 93d04fe, but only 101 validate the patch. The three OSS-Fuzz jobs failed to fetch this fork's PR from upstream and continued on upstream code. Their green results are excluded. Exact-source replacement run 40256366732 has now passed. All 21 targets completed under each of address, undefined and memory sanitizers with the 600-second budget per job. Source commit and four hashes match before/after building; logs confirm the checkout was copied into the builder. Changed-file filtering failed, so all targets ran. Five public corpus downloads returned 403; those targets still completed, with warnings retained.
  • Eighteen jobs use Ubuntu 24.04 for available gold/qemu dependencies and historical build compatibility. MinGW Clang uses the official 21.1.8 package because Clang 23 reports five unchanged CLI globals; the same errors were reproduced on the base. PowerPC uses the multilib package matching its compiler. Original test commands, flags, durations and size limits are retained.
  • make staticAnalyze is not clean: Clang 23 gives the same 15 diagnostic/source signatures on base and fix; a separate Clang 18 comparison gives the same nine. Both versions fail on both revisions. Those results remain retained.
  • Binary-size comparison: all four ELF sizes are identical between base and fix. Both exceed the ordinary-build limit at 1,101,192 bytes with that toolchain; the other three limits pass. This does not establish instruction or performance equivalence.

Upstream draft: facebook#4864. The fork workflow changes are excluded from that source-only PR. Meta requires the contributor to sign its CLA before review and merge.

OpenAI Codex (GPT-6) assisted with investigation, implementation and tests. dawnr checked a mathematical status-encoding model; it does not prove the C implementation.

ZSTD_loadZstdDictionary() and ZSTD_compress_insertDictionary() returned
both dictionary IDs and errors through size_t. On 32-bit builds, valid
private dictionary IDs near UINT32_MAX overlap the encoded error range
and are rejected by their callers.

Return a status and write the ID through an output parameter instead.
Raw or absent dictionaries still produce ID zero, and failed loads still
return their error before the caller stores the ID.

Add round trips and frame-ID checks for low and high IDs through direct
dictionaries, prepared CDicts, and CCtx-loaded dictionaries.
Treston Cuzzort added 3 commits October 25, 2026 19:49
Pin jobs requiring gold or qemu-user-static to Ubuntu 24.04 and the MinGW Clang job to 21.1.8. Preserve all test targets and flags. Compare Clang 18 static analysis of the upstream base and exact source patch. Fork validation only; not part of the upstream change.
Use the distribution multilib compiler package instead of a hardcoded GCC 13 support library. The latest runner uses GCC 15 and cannot link the 64-bit target with the previous dependency. Fork validation only.
Pin the two legacy environment checks without changing their commands or size limits. Keep the already collected exact-base/exact-fix analyzer comparison manual so infrastructure-only changes do not repeat it. Both analyzer failures remain documented.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant