Skip to content

Honor --delete_problematic_images during Webshart streaming training - #3416

Merged
bghira merged 1 commit into
mainfrom
fix/webshart-runtime-exclusions
Nov 21, 2026
Merged

bghira merged 1 commit into
mainfrom
fix/webshart-runtime-exclusions

Conversation

@bghira

@bghira bghira commented Nov 21, 2026

Copy link
Copy Markdown
Owner

Streaming Webshart Parquet training could stop on a corrupt image. The reader omitted the row, and the batch loader then failed while reconstructing the requested batch. Enabling --delete_problematic_images also forced a full catalogue scan before training.

With the flag enabled, the sampler now records unreadable rows in the metadata cache for the affected shard revision. It fills empty slots from the same aspect-ratio bucket before resolving captions and conditioning. If necessary, it repeats a readable row to keep each rank’s batch full. All ranks advance past a bucket with no readable rows. Pixel training and on-demand VAE encoding reuse the decoded images.

Verification

  • 101 tests passed across the Webshart Parquet backend, epoch scheduling, collation, and VAE cache traversal suites. The regression test covers an unreadable rank-local batch, a fully unreadable window, and checkpoint resume.
  • Replayed both rows that previously stopped training. Each produced a full replacement batch, and reopening the window skipped the excluded row.
  • Resumed an eight-H200 training job from step 23,160 and verified all eight sampler cursors in the rolling checkpoint at step 23,250.

@bghira
bghira merged commit db3f4d4 into main Nov 21, 2026
4 checks passed
@bghira
bghira deleted the fix/webshart-runtime-exclusions branch November 21, 2026 21:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant