Repository navigation
[WS2][CUDA][MiniMax-H3] ws2 tensor / sequence / context parallel and the hierarchical cross-node collective (RFC #420 stack 3/7) - #569
Open
fusheng-ji wants to merge 70 commits into
Conversation
fusheng-ji
requested review from
EthanZero2Hero,
Flink-ddd,
KJLdefeated,
bitborne,
inaniloquentee and
ryankert01
as code owners
October 22, 2026 12:18
|
Warning Review limit reachedThis review ran on the open-source allowance, not this organization's plan, because the pull request author doesn't have an assigned seat. Waiting won't change this — ask an organization admin to assign them a seat, or add seats in Billing if every seat is already assigned, then retry. Next included review available in 48 minutes. View limit details
Comment |
fusheng-ji
force-pushed
the
feat/h3-ws2-parallel
branch
from
October 22, 2026 14:37
900f962 to
d086b5c
Compare
fusheng-ji
added a commit
to fusheng-ji/RL-Kernel
that referenced
this pull request
Oct 22, 2026
… dual trajectory (RL-Align#571) on H100 at current branch heads
…fixed-order Triton GEMM Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com> (cherry picked from commit 650e2148c6f0e5dcdef136537e0e62f745020273) Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com> (cherry picked from commit d099c0b)
Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com> (cherry picked from commit b41f868)
…de collective, 2-node H100 evidence Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com> (cherry picked from commit 7f1296a970efb0cfcebd9300ff92551de08cf311) Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com> (cherry picked from commit d56c391)
…il-closed test instead of a skip Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com> (cherry picked from commit 74e5b83)
Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com> (cherry picked from commit 33cbcd6)
…l keys Review finding (RL-Align#568): jointly permuting x and g changed dW/dbias because the token axis was reduced in call order. All projection ops (audio in/out, text context, Q/K/V) now take an optional logical_keys [T,2|3] (negative first column = padding); rows are reduced in ascending key order with padding dropped, active keys must be unique. Key format follows RL-Align#562. Adds the reviewer's repro plus permutation/padding tests for every row and Q/K/V. Co-authored-by: Yitao Wang <75578057+walker-ai@users.noreply.github.com> Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com> (cherry picked from commit d2941f0)
Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com> (cherry picked from commit 5f4e34c)
…ready ordered rows Follows the canonical-row design of RL-Align#562: the Q/K/V node of the block graph supplies (batch row, position) keys so its dW is reduced in logical order, and rows that already arrive in canonical order are reduced in place (same bytes, no gather of x and g). Co-authored-by: Yitao Wang <75578057+walker-ai@users.noreply.github.com> Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com> (cherry picked from commit 0da0583)
Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com> (cherry picked from commit d086b5c)
…block 0 node-by-node and 50-block/generation gates Squashes the one-block branch (its row-by-row history: timestep sinusoid, RMSNorm, AdaLN rows, sequence/tensor parallel norm rows, final AdaLN, one block, 50-block qualification) onto the current test-h3 so the pull request carries a reviewable, signed history. RFC RL-Align#420. Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com>
…pull request Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com>
…hard leaves Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com>
fusheng-ji
force-pushed
the
feat/h3-ws2-parallel
branch
from
October 22, 2026 21:04
d086b5c to
5ae9f5e
Compare
fusheng-ji
added a commit
to fusheng-ji/RL-Kernel
that referenced
this pull request
Oct 23, 2026
… on one 8-GPU node 128 tests pass (8-GPU NCCL cases included) and all six layouts (TP2, TP4, TP4xCP2, TP4xCP2+SP, TP8 with the node-local tree, TP8 with the 2x4 hierarchical tree inside one node) are bitwise equal to the single-GPU block. Host names replaced by node-a. Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com>
Flink-ddd
added
platform: cuda
fusheng-ji
changed the title
- h3_block._backward: run the golden backward under strict_fp32_matmul (run_graph only covered the forward) - h3_block_layout: reject num_timesteps outside [1, seq_len] instead of silently returning fewer runs - h3_block_diagnostic / h3_evidence_integrity: hash files with a local digest; nothing here imports h3_full_model, which only exists in the next PR of the stack - h3_first_step_diagnostic: removed from this PR; it imports h3_full_model and h3_full_generation and belongs with the generation gate, where both exist - plot_h3_evidence: draw a placeholder when an SP report has no run with SP > 1 - rmsnorm backward_partials / backward_dx: require a dense 1-D rstd like h3_rmsnorm_backward (a strided or stride-0 view with M elements passed the old numel check and was read as a dense array) Regression tests for each; each fails on the previous code. Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com>
fusheng-ji
changed the title
Takes 7aa29b8 (review findings on the block-0 replay). h3_first_step_diagnostic.py is removed from this PR: it imports h3_full_generation, which only exists in the generation gate (RL-Align#588), where it stays.
Takes the RL-Align#586 review fixes through RL-Align#587. h3_first_step_diagnostic.py stays here (it needs h3_full_generation, which this PR adds) and its usage example now shows the required --contract.
…S2 layout move, SP row-index fix) into PR RL-Align#587
…S2 layout move, SP row-index fix) into PR RL-Align#588
…S2 layout move, SP row-index fix) into PR RL-Align#586
… PR RL-Align#586 RL-Align#494 fixed the same two review findings as 7aa29b8 (dense 1-D rstd in the SP backward entry points, SP plot without an SP > 1 run) with tests of its own. The conflicting plot hunks and the SP plot test take RL-Align#494's version, and the duplicate rstd test added here is dropped; the kernel text is identical on both sides.
…after taking RL-Align#494's version RL-Align#494's copy of the two textwrap.dedent blocks is in a newer black style than the pinned 24.4.2 that the pre-commit hook and the linting job use; this restores the pinned style (no code change). Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com>
- rmsnorm backward_partials: the 2N grid bound belongs to the modulated call (shift and scale partials side by side); a plain call launches over N and now gets the N bound (no longer rejected above ~8.3M columns) - gate_grad_partials: reject a zero-width input (it would launch dim3(tiles, 0)) with a clear message - gate_grad_partials: state that the tile coverage of the caller-built list is the caller's contract - adaln_row_gather backward: pin the contract that a table without rows is rejected (the RFC's gate asks empty cases to fail clearly) with a native test - layout map: restore tests/h3/test_h3_rmsnorm_native_validation.py, dropped by an earlier merge resolution - SP evidence plot: keep a run slower than WS1 inside the timing axis; draw the RL-Kernel bars of the naive-reduction panel from the recorded equality flags instead of fixed zeros - TP doc/test: the ownership test now also covers TP 12 and 14, which the document says it does Tests added for each. The three new plot tests fail on the previous plot code (checked); the native tests were run against the rebuilt extension only. Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com>
gather_rows padded every shard past the single-block fast-path limit. On 8 x B200 (2, 4 and 8 ranks, outputs 2 KiB to 512 KiB) the multi-block path costs a flat 55-65 us for 2 and 4 ranks (~90 us for 8) while the single-block path costs ~4 us/KiB, so below ~8 KiB of output the plain gather is up to ~25 us faster (2 ranks: 54 us plain against 59 us padded at 8 KiB, 63 against 59 at 10 KiB) and above it padding wins by up to 25x (256 KiB: 1.1-1.7 ms plain, 60-95 us padded). Outputs of at most 8 KiB are now gathered unpadded. A copy either way: the new test checks gather_rows against the plain all-gather bit for bit at sizes around the threshold. Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com>
…with its measured size Signed-off-by: Wenbo Ji <36562829+fusheng-ji@users.noreply.github.com>
|
🔍 OpenCodeReview found 2 issue(s) in this PR.
📄
|
|
@fusheng-ji The automated OCR review has completed. Please review and address all automated review bot findings, then push your updates. Maintainers will begin manual review after the bot findings have been addressed. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Contract
h3-parallel-v1: eight canonical reduction units, so tensor parallel 1 / 2 / 4 / 8 add the same partials in the same tree.dX: unit j is block j of every fused weight, in q,k,v / hidden,gate order.SeqShard; sequence parallel: all-gather / reduce-scatter;coshardslices metadata with the rows.HierarchicalCollective(hierarchical_ipc_nccl_tree): node-local deterministic tree, NCCL all-gather of the group partials, fixed balanced add. The input dtype is the accumulation dtype, so the result matches the single-node tree bit for bit.h3_parallel_run.pylaunches a layout on N processes and compares every rank with the single-GPU canonical block.Prior art & reuse decision
DeterministicCollectiveDecision: canonical-unit partial sums plus a fixed tree; NCCL only moves bytes between nodes.
Results (H100 / Hopper, 2 nodes x 4 GPUs)
Measured on NVIDIA H100 (93 GiB, sm90), driver 580, CUDA 13.0, torch 2.13.0+cu130, pinned
MiniMax-H3@42ed227weights, at head5ae9f5eb. Block 0 of the pinned checkpoint, 264 packed rows; forward and input gradient compared bitwise with the single-GPU canonical block for every layout, and in addition every weight-gradient shard (to_q/k/v, to_out, ff hidden/gate/down) for the layouts without CP (TP2, TP4, TP8).Numbers:
reports/experiments/h3-ws2-parallel-h100/(one report per layout; host names are replaced bynode-a,node-b).Results (B200 / Blackwell, 1 node x 8 GPUs)
Measured on NVIDIA B200 (179 GiB, sm100), driver 580.126, CUDA 13.0, torch 2.13.0+cu130, pinned
MiniMax-H3@42ed227weights, clean tree at5ae9f5eb. One node with 8 GPUs, one process per GPU; same block, 264 packed rows and comparisons as the H100 table above (forward and input gradient against the single-GPU canonical block for every layout, plus every weight-gradient shard for the layouts without CP).Numbers:
h3-ws2-parallel-b200/(one report per layout,tests.json; host names replaced bynode-a).--local-size 4, which selects the hierarchical IPC + NCCL tree and treats the 8 GPUs as two groups of four inside one physical node. It exercises that code path; it is not a measurement of the cross-node network, which was only measured on H100 (two nodes).test_h3_tp_adaln.py/test_h3_sp_norm_adaln.pywere not run on the B200 either.Status
aea92811. The branch merges the reworked [WS1][CUDA][MiniMax-H3] ws1_one_h3_block: block 0 node-by-node replay and diagnostics (RFC #420 stack 10a/10) #586 → [WS1][CUDA][MiniMax-H3] full-model qualification: thresholds, ensemble rule and reference manifest (RFC #420 stack 10b/10) #587 → [WS1][CUDA][MiniMax-H3] generation gate: 50-block generation compare and resume check (RFC #420 stack 10c/10) #588 base (carrying the review fixes of [WS1][CUDA][MiniMax-H3] timestep_sinusoid_h3: FP32 timestep features + shared H3 harness (RFC #420 stack 1/10) #483–[WS2][CUDA][MiniMax-H3] sp_norm_adaln: sequence-parallel norm/AdaLN/gated residual, byte-equal to WS1 (RFC #420 stack 9/10) #494 attest-h328894da1); this PR's own diff against [WS1][CUDA][MiniMax-H3] projection rows: audio in/out, text context and QKV on a fixed-order GEMM (RFC #420 stack 2/7) #568 is unchanged.test_h3_tp_adaln.pyandtest_h3_sp_norm_adaln.py("needs 8 GPUs"). If you have a single-node 8 x H100/H200 machine, please runpytest tests/models/minimax_h3/test_h3_parallel.py tests/models/minimax_h3/test_h3_tp_adaln.py tests/models/minimax_h3/test_h3_sp_norm_adaln.pyand the WS2 evidence runner there and post the result; I will gladly credit you and fold the numbers into this PR.All five layouts are bitwise equal to the single-GPU reference on real nodes; ranks are packed node-major with 1 / 2 / 4 / 8 nodes supported.
B200 status: all six layouts, including the 8-rank cases on one 8-GPU node, are bitwise equal to the single-GPU block (128 of 128 tests pass); the cross-node collective is not measured on B200.
History
feat/h3-projection-rows), which is synced to [WS2][CUDA][MiniMax-H3] sp_norm_adaln: sequence-parallel norm/AdaLN/gated residual, byte-equal to WS1 (RFC #420 stack 9/10) #494's head and totest-h3after [WS1][CUDA][MiniMax-H3] timestep_sinusoid_h3: FP32 timestep features + shared H3 harness (RFC #420 stack 1/10) #483–[WS1][CUDA][MiniMax-H3] h3_rmsnorm: RMSNorm with fused AdaLN modulation (RFC #420 stack 5/10) #487 merged. Merge commits only, no force-push; every commit is signed off. The set of lines this PR adds over its parent is identical to the previous head (checked line by line).end-of-file-fixerandblack(24.4.2, the pinned version) failed on two files inherited fromtest-h3(reports/experiments/h3-sigma-schedule/benchmark-4090d.json,rl_engine/_C.pyi); both are fixed once at the bottom of the stack and merged up.pr_assetscommits pinned in earlier versions of this body no longer exist (that branch's history was rewritten); the links now point at the current commit that holds the same files. Only the commit id changed; numbers and the commits they were measured at are unchanged.Tests
python -m pytest tests/models/minimax_h3/test_h3_parallel.py tests/models/minimax_h3/test_h3_projection_rows.py -q # 122 passed (4 x H100)The 8-GPU NCCL cases need eight GPUs in one node or two nodes: we only have 4-GPU Hopper nodes, so they are covered by the two-node runner above and not run on a single 8-GPU Hopper node (open for anyone with that resource, see the note at the top).
B200: the same two files on 8 x B200, 128 passed, 0 failed, 0 skipped (the 8-GPU NCCL cases ran on one node).
Measured on the synced branches (2 × NVIDIA RTX PRO 6000 Blackwell, sm120, pinned
MiniMax-H3@42ed227weights incl. block 0, each build compiled in place from the checked-out commit):672c1b36(#575, the top of this chain, contains this PR)pytest tests/models/minimax_h3672c1b36pytest tests/models/minimax_h3 tests/build tests/runtime/test_dispatch.py tests/validationSkips are environment only: no
matplotlib,diffusersorsglang; tests that need 4 or 8 GPUs (2 were visible); no full-transformer shard index; noVERL_OMNI_PATH. TheKERNEL_ALIGN_USE_FAST_MATH=1build was not run on these heads. The per-device results under Results were measured at the heads named there, before this sync.H100 (sm90) on the synced head
aea92811(evidence:tests.json, environment header and the raw pytest logs). H100-94GB, driver 580.173.02, CUDA 13.0, torch 2.13.0+cu130, triton 3.7.1, Python 3.12, pinnedMiniMax-H3@42ed227weights incl. block 0, 4 GPUs visible, extension built in place from this commit:pytest tests/models/minimax_h3 tests/build tests/runtime/test_dispatch.py tests/validationKERNEL_ALIGN_USE_FAST_MATH=1The one default-build failure is
tests/validation/operators/test_ws1_candidate_evidence.py::test_ws1_cuda_and_triton_candidate_runtime_provenance: it asks for the strict SM90 GEMM and this build was made withoutKERNEL_ALIGN_DET_GEMM_SM90=1("refusing naive fallback"). It fails the same way on the other test machine and is not an H3 test. The fast-math build has the same failure plusTestCuda::test_unsupported_device_is_refused_at_constructionintests/models/minimax_h3/test_h3_final_adaln_out.py(#489): with the H3 kernels excluded, the op constructor raises "rl_engine._Clacks the h3_det_linear_* / h3_rmsnorm_* ops" before the SM80 check the test expects. Default-build skips (6): the 8-GPU cases (4 GPUs here),sglangnot installed, ROCm-only, CI-only extension enforcement, and the SM90 log-prob backend not compiled; fast-math skips (611) are the H3 kernel tests the build leaves out.Scope / Known limitations
test_h3_tp_adaln.py::andtest_h3_sp_norm_adaln.py::8-GPU cases are skipped here. Help from anyone with a single-node 8 x Hopper machine is welcome.Strict mode is fail-closed: no TF32, no fast-math, no split-K; FP32 accumulation with one final rounding; unsupported shapes, dtypes and devices raise instead of falling back.
ws2_pipeline_parallelis not included. Not part of this PR: video projections (#562), O / gate_up / down GEMMs (#554, #535), full attention (#553), ROCm, SGLang adapter, pipeline parallel.