Skip to content

Prefill remote Webshart shards from the training cursor - #3410

Merged
bghira merged 1 commit into
mainfrom
feature/webshart-cache-prefill
Nov 21, 2026
Merged

bghira merged 1 commit into
mainfrom
feature/webshart-cache-prefill

Conversation

@bghira

@bghira bghira commented Nov 21, 2026

Copy link
Copy Markdown
Owner

Sequential Webshart training now starts cache prefill at the physical shard window the sampler opens, including on checkpoint resume. Consuming a window frees space for more upcoming shards. A local main process manages prefill and closes the cursor with the epoch. Older Webshart builds continue to support on-demand reads. The cache cursor is provided by Webshart PR #53.

Set SIMPLETUNER_WEBSHART_PREFILL_DOWNLOADS to tune the background reader independently of the saved dataset configuration. On the live job this allowed 16 background downloads with the original four-way foreground I/O and the original eight rank cursors.

Performance

On four L40S GPUs, two 160-step HDiT+ runs used identical inputs, separate cold 1 GiB shard caches, and 32 images per rank. From the end of step 16 through step 159, mean wall time fell from 0.483 to 0.313 seconds per step. Throughput rose from 265 to 408 images/s. Elapsed time from the first shard-window scan through step 160 fell from 100.6 to 78.6 seconds. Sample order, pixels, and captions matched for all 640 rank-batches.

The live eight-H200 run averaged 0.798 seconds per step over 400 steps with on-demand reads and 0.347 over 600 steps with a full prefill buffer. These were different sequential shard windows, excluding checkpoint and validation pauses. The buffer reached 484 shards within 15 MiB of its 32 GiB limit and continued refilling as training advanced.

Verification

The Webshart epoch and Parquet backend suites passed 51 SimpleTuner tests. Coverage includes the resumed first window, reproduced batch order, cursor cleanup, and independent background concurrency.

@bghira
bghira merged commit 128335a into main Nov 21, 2026
4 checks passed
@bghira
bghira deleted the feature/webshart-cache-prefill branch November 21, 2026 20:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant