All pull requests

Pull requests

• Insert type error suppressions necessary for Pyrefly

  #2912·copybara-service[bot] opened Oct 15, 2026Bot·

  ·
  

• fix(deepswe): exclude newly added untracked agent test scripts before grading

  #2908·niting opened Oct 15, 2026Collaborator·

  ·
  3

  

• orchestrator: add 2-deep background stage_batch pipeline and TrainerWorker.stage_batch

  #2906·A9isha opened Oct 15, 2026Collaborator·

  ·
  12

  

• orchestrator: compact routed_experts wire packing with uint10/uint8 encoding and unrouted-token stripping

  #2905·A9isha opened Oct 15, 2026Collaborator·

  ·
  12

  

• Extract task pooling primitives from remote_execution into task_pool.

  #2904·copybara-service[bot] opened Oct 15, 2026Bot·

  ·
  

• deepswe: SANDBOX_RELOCATE_VENV moves /testbed/.venv out of the repo (on in mlperf recipes)

  #2903·s-noghabi opened Oct 15, 2026Collaborator·

  ·
  4

  

• Pipelined step boundary: run end-of-step bookkeeping off the event loop

  #2901·s-noghabi opened Oct 15, 2026Collaborator·

  ·
  4

  

• Stub xgrammar in the vLLM tests when it is not installed.

  #2900·copybara-service[bot] opened Oct 15, 2026Bot·

  ·
  1

  

• chore(recipes): remove dead, overwritten, and redundant knobs in 397B v7x recipes

  #2897·tianshub opened Oct 14, 2026Collaborator·

  ·
  1

  

• recipes(397b v7x): trainer perf flags (SparseCore offload thresholds, decoder-layer-input placement) and MaxText gather-once pin

  #2896·khatwanimohit opened Oct 14, 2026Collaborator·

  ·
  4

  

• Prepare gemma4_test and the RaidenConfig mock for the vLLM 5ab4e444 sync.

  #2894·copybara-service[bot] opened Oct 14, 2026Bot·

  ·
  1

  

• Add TrajectoryMetrics in tunix/experimental/trajectory/trajectory.py as a standalone Pydantic model (extra="forbid") with trajectory_id and an extensible extra: MetadataDict field for rollout performance and RL training metrics.

  #2891·copybara-service[bot] opened Oct 14, 2026Bot·

  ·
  1

  

• The multiprocess Raiden weight sync E2E tests drive trainer and rollout workers in spawned processes over multiprocessing pipes, and every wait in that harness could block forever. Replies were read with a blocking Connection.recv() and no timeout. The parent kept its copy of each child's pipe end, so a dead worker never produced EOF. The blocking reads ran inside the async proxies, so the coordinator's own phase deadlines could not fire either. Leftover workers were joined without a timeout at interpreter exit. A CPU CI job running these tests sat in one of them for almost six hours until the job limit killed it.

  #2890·copybara-service[bot] opened Oct 14, 2026Bot·

  ·
  1

  

• feat(orchestrator): optional in-flight rollout cap for dispatch (MAX_IN_FLIGHT_BATCHES)

  #2889·Rohan-Bierneni opened Oct 14, 2026·

  ·
  1

  

• The library side of router replay is on main: the rollout collector encodes the MoE experts each token was routed to, the orchestrator aligns them to the training batch, and the trainer passes them to MaxText as forced_routed_experts. Nothing in the distributed examples turns it on: no example sets GenerationArgs.return_routed_experts, the rollout node never enables capture in vLLM, and the encoding stays at auto with no expert count. For a model with 256 to 1023 experts per layer, such as Qwen3.5-35B-A3B with 256, auto without a count picks the tier from the highest expert id it sees: a batch whose ids all happen to be below 255 is encoded as uint8 instead of uint10, and the trainer step recompiles for the other shape.

  #2888·copybara-service[bot] opened Oct 14, 2026Bot·

  ·
  

• mlperf_gbs1024_submission / mlperf_397b_256_v7x: enable 128-chip trainer perf flags and pin MaxText #5721

  #2884·Rohan-Bierneni opened Oct 14, 2026·

  ·
  8

  

• Improve Gemma 4 vLLM weight synchronization, Qwix LoRA compatibility, and Orbax checkpoint loading:

  #2883·copybara-service[bot] opened Oct 14, 2026Bot·

  ·
  1

  

• Support multimodal image inputs in VllmSampler and TrainExample for PPO/GRPO.

  #2882·copybara-service[bot] opened Oct 14, 2026Bot·

  ·
  1

  

• In multi-turn agentic rollouts, Gemma4ChatTemplateParser._parse_tool emitted bare <|tool_response>...<tool_response|> blocks without enclosing <|turn>user\n...<turn|>\n turn delimiters, and bos_token="<bos>" caused duplicate BOS tokens when tokenizers already prepend BOS. Additionally, sanitize_control_tokens stripped authored <|tool_call> and <|thought|> delimiters when rendering stored multi-turn messages.

  #2881·copybara-service[bot] opened Oct 14, 2026Bot·

  ·
  1

  

• Copy reward_fns in DAPOLearner instead of appending to the caller's list

  #2880·vineethsaivs opened Oct 14, 2026·6 tasks done·

  ·
  3

  

• With --trainer_backend=maxtext, the trainer forwarded only the learning rate and warmup to MaxText. It silently dropped --optimizer_b1/b2/eps/weight_decay and the --optimizer_opt_chain_type=clip_by_global_norm chain, so MaxText trained with its base.yml defaults (adam_b2=0.95, adam_weight_decay=0.1, clipping at 1.0) whatever the flags said.

  #2869·copybara-service[bot] opened Oct 13, 2026Bot·

  ·
  

• Split rollout worker pools: pin batch b to pool b % (max_staleness+1), optional per-pool weight sync

  #2868·wenxindongwork opened Oct 13, 2026Collaborator·

  ·
  2

  

• orchestrator: hold router-replay routing bit-packed until packed (opt-in, 0.6x per held trajectory)

  #2866·parambole opened Oct 13, 2026Collaborator·

  ·
  1

  

• move splash attention out of experimental ops

  #2865·copybara-service[bot] opened Oct 13, 2026Bot·

  ·
  

• [Experimental] Add deadlock-free orchestrator memory backpressure for polling and dispatch

  #2864·tianshub opened Oct 13, 2026Collaborator·

  ·
  1