Skip to content

Preserve NaNs during BF16 stochastic rounding - #3414

Merged
bghira merged 1 commit into
mainfrom
fix/issue-3355
Nov 21, 2026
Merged

bghira merged 1 commit into
mainfrom
fix/issue-3355

Conversation

@bghira

@bghira bghira commented Nov 21, 2026

Copy link
Copy Markdown
Owner

Related to #3355.

Preserve NaNs when stochastically rounding FP32 optimizer updates to BF16. A carry from the random mantissa bits could turn a NaN bit pattern into a signed zero. Apply the NaN mask in the existing rounding buffer.

Validation

  • Replayed the captured step 22 LTX AdamWBF16 update on H100. The corrected rounding preserved all 65,536 NaN weight updates that previously became finite.
  • Seeded rounding matched the original output bit for bit on 65,536 finite CUDA values. NaNs and signed infinities were preserved.
  • Passed 22 targeted optimizer and LTX tests. The regression exercises the failing bit patterns on CPU and MPS.
  • The LTX backward path that produces the first non-finite gradient is still under investigation.

@bghira
bghira merged commit 1f487ad into main Nov 21, 2026
2 checks passed
@bghira
bghira deleted the fix/issue-3355 branch November 21, 2026 20:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant