Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions documentation/DATALOADER.es.md
Original file line number Diff line number Diff line change
Expand Up @@ -1445,6 +1445,7 @@ Los datasets Webshart cargan shards tar estilo WebDataset mediante el paquete `w
- `metadata` es opcional y puede apuntar a metadatos separados con captions. Para repositorios Hugging Face de metadata como `webshart/conceptual-captions-12m-webdataset-metadata`, pasa el repo id; Webshart sigue el layout de subcarpetas del source, como `data/`.
- `metadata_backend` debe ser `webshart`; `caption_strategy` debe ser `webshart` o `instanceprompt`.
- `webshart.cache_dir` almacena la metadata de SimpleTuner y las caches de Webshart. `shard_cache_gb` y `parallel_downloads` se pasan a la cache de shards de Webshart; define `shard_cache_gb` como `0` para desactivar la cache de shards completos y mantener lecturas por rango indexadas.
- En el entrenamiento secuencial por ventanas con Parquet remoto, las versiones de Webshart con `sequential_prefill` descargan los próximos shards hasta que el siguiente superaría `shard_cache_gb`. Un proceso por caché compartida usa hasta `parallel_downloads` descargas simultáneas. Al reanudar, empieza en la ventana restaurada y permite desalojar los shards ya consumidos. Al reanudar un entrenamiento, usa `SIMPLETUNER_WEBSHART_PREFILL_DOWNLOADS` para ajustar por separado las descargas simultáneas en segundo plano.
- `webshart.metadata_cache_dir` almacena el catálogo de shards y el índice del manifiesto de captura. Por defecto usa `metadata_cache` dentro de `webshart.cache_dir`.
- `webshart.shard_cache_dir` almacena los shards descargados. Por defecto usa `shard_cache` dentro de `webshart.cache_dir`.
- Con la caché de fragmentos completos habilitada para Parquet remoto, cada backend mantiene abierto un lector de fragmentos en caché entre lecturas. Reserva espacio adicional por encima de `shard_cache_gb` para las descargas simultáneas y los fragmentos que los lectores activos mantienen bloqueados. Webshart escribe cada descarga en un archivo temporal antes de descartar los fragmentos antiguos.
Expand Down
1 change: 1 addition & 0 deletions documentation/DATALOADER.hi.md
Original file line number Diff line number Diff line change
Expand Up @@ -1444,6 +1444,7 @@ Webshart datasets `webshart` package के जरिए WebDataset-style tar sh
- `metadata` optional है और captions वाले separate metadata location को point कर सकता है। `webshart/conceptual-captions-12m-webdataset-metadata` जैसे Hugging Face metadata repos के लिए repo id दें; Webshart source shard के `data/` जैसे subfolder layout को follow करता है।
- `metadata_backend` को `webshart` होना चाहिए; `caption_strategy` `webshart` या `instanceprompt` हो सकता है।
- `webshart.cache_dir` SimpleTuner metadata और Webshart caches store करता है। `shard_cache_gb` और `parallel_downloads` Webshart shard cache को pass किए जाते हैं; whole-shard caching disable करने और indexed range reads बनाए रखने के लिए `shard_cache_gb` को `0` सेट करें।
- रिमोट Parquet पर क्रमिक विंडो में प्रशिक्षण के दौरान, `sequential_prefill` वाले Webshart संस्करण आगे के शार्ड तब तक डाउनलोड करते हैं जब तक अगला शार्ड जोड़ने से `shard_cache_gb` की सीमा पार न हो। हर साझा कैश का एक वर्कर एक साथ अधिकतम `parallel_downloads` डाउनलोड करता है। प्रशिक्षण फिर शुरू करने पर यह सहेजी गई विंडो से चलता है और इस्तेमाल हो चुके शार्ड को कैश से हटाने देता है। प्रशिक्षण फिर शुरू करते समय पृष्ठभूमि में एक साथ चलने वाले डाउनलोड की संख्या अलग से तय करने के लिए `SIMPLETUNER_WEBSHART_PREFILL_DOWNLOADS` सेट करें।
- `webshart.metadata_cache_dir` में shard catalogue और capture manifest index रहते हैं। डिफ़ॉल्ट पथ `webshart.cache_dir` के भीतर `metadata_cache` है।
- `webshart.shard_cache_dir` में डाउनलोड किए गए shards रहते हैं। डिफ़ॉल्ट पथ `webshart.cache_dir` के भीतर `shard_cache` है।
- रिमोट Parquet के लिए पूरे शार्ड का कैश चालू होने पर हर बैकएंड दो रीड के बीच कैश किए गए शार्ड का एक रीडर खुला रखता है। एक साथ चल रहे डाउनलोड और सक्रिय रीडर द्वारा लॉक किए गए शार्ड के लिए `shard_cache_gb` से अधिक डिस्क स्थान रखें। Webshart पुराने शार्ड हटाने से पहले हर डाउनलोड को एक अस्थायी फ़ाइल में लिखता है।
Expand Down
1 change: 1 addition & 0 deletions documentation/DATALOADER.ja.md
Original file line number Diff line number Diff line change
Expand Up @@ -1444,6 +1444,7 @@ Webshart データセットは `webshart` パッケージで WebDataset 形式
- `metadata` は任意で、captions を含む別 metadata location を指定できます。`webshart/conceptual-captions-12m-webdataset-metadata` のような Hugging Face metadata repo では repo id だけを渡します。Webshart は source shard の `data/` などのサブフォルダ構成に従います。
- `metadata_backend` は `webshart`、`caption_strategy` は `webshart` または `instanceprompt` にします。
- `webshart.cache_dir` は SimpleTuner metadata と Webshart caches を保存します。`shard_cache_gb` と `parallel_downloads` は Webshart の shard cache に渡されます。`shard_cache_gb` を `0` にすると、shard 全体の cache を無効にし、index 付き range read を維持します。
- リモート Parquet をウィンドウ単位で順次学習する場合、`sequential_prefill` 対応の Webshart は、次のシャードを追加すると `shard_cache_gb` を超えるところまで先読みします。共有キャッシュごとに 1 つのワーカーが最大 `parallel_downloads` 件を並列でダウンロードします。学習再開時は復元したウィンドウから開始し、読み終えたシャードをキャッシュから削除できるようにします。 再開時にバックグラウンドの同時ダウンロード数だけを調整するには、`SIMPLETUNER_WEBSHART_PREFILL_DOWNLOADS` を設定します。
- `webshart.metadata_cache_dir` はシャードカタログとキャプチャマニフェストのインデックスを保存します。デフォルトは `webshart.cache_dir` 配下の `metadata_cache` です。
- `webshart.shard_cache_dir` はダウンロード済みのシャードを保存します。デフォルトは `webshart.cache_dir` 配下の `shard_cache` です。
- リモートの Parquet でシャード全体のキャッシュが有効な場合、各バックエンドは読み取りの合間にキャッシュ済みシャードのリーダーを 1 つ保持します。同時ダウンロードと、使用中のリーダーがロックしているシャードのために、`shard_cache_gb` を超える空き容量を確保してください。Webshart は古いシャードを削除する前に、ダウンロード内容を一時ファイルに書き込みます。
Expand Down
1 change: 1 addition & 0 deletions documentation/DATALOADER.md
Original file line number Diff line number Diff line change
Expand Up @@ -1500,6 +1500,7 @@ Webshart datasets load WebDataset-style tar shards through the `webshart` packag
- `metadata_backend` must be `webshart`; it reads dimensions and captions from Webshart metadata.
- `caption_strategy` should be `webshart` to train from metadata captions, or `instanceprompt` to ignore stored captions.
- `webshart.cache_dir` stores SimpleTuner metadata plus Webshart metadata and shard caches. `shard_cache_gb` and `parallel_downloads` are passed to Webshart's shard cache; set `shard_cache_gb` to `0` to disable whole-shard caching and retain indexed range reads.
- For sequential, windowed training on remote Parquet, Webshart builds with `sequential_prefill` download upcoming shards until the next shard would exceed `shard_cache_gb`. One worker per shared cache uses up to `parallel_downloads` concurrent downloads. It starts at the restored window on resume and releases consumed shards for eviction. Set `SIMPLETUNER_WEBSHART_PREFILL_DOWNLOADS` to tune background download concurrency independently when resuming an existing run.
- `webshart.metadata_cache_dir` stores the shard catalogue and capture manifest index. It defaults to `metadata_cache` under `webshart.cache_dir`.
- `webshart.shard_cache_dir` stores downloaded shards. It defaults to `shard_cache` under `webshart.cache_dir`.
- With whole-shard caching enabled for remote Parquet, each backend keeps one cached shard reader open between reads. Allow extra disk space beyond `shard_cache_gb` for concurrent downloads and for shards that active readers still have locked. Webshart writes each download to a temporary file before evicting old shards.
Expand Down
1 change: 1 addition & 0 deletions documentation/DATALOADER.pt-BR.md
Original file line number Diff line number Diff line change
Expand Up @@ -1445,6 +1445,7 @@ Datasets Webshart carregam shards tar no estilo WebDataset pelo pacote `webshart
- `metadata` é opcional e pode apontar para metadados separados com captions. Para repos Hugging Face de metadata como `webshart/conceptual-captions-12m-webdataset-metadata`, passe o repo id; o Webshart segue o layout de subpastas do source, como `data/`.
- `metadata_backend` deve ser `webshart`; `caption_strategy` deve ser `webshart` ou `instanceprompt`.
- `webshart.cache_dir` armazena os metadados do SimpleTuner e os caches do Webshart. `shard_cache_gb` e `parallel_downloads` são passados ao cache de shards do Webshart; defina `shard_cache_gb` como `0` para desativar o cache de shards completos e manter leituras por intervalo indexadas.
- No treinamento sequencial por janelas com Parquet remoto, as versões do Webshart com `sequential_prefill` baixam os próximos shards até que o próximo exceda `shard_cache_gb`. Um único processo por cache compartilhado faz até `parallel_downloads` downloads simultâneos. Ao retomar o treinamento, ele começa na janela restaurada e libera para remoção os shards já consumidos. Ao retomar o treinamento, use `SIMPLETUNER_WEBSHART_PREFILL_DOWNLOADS` para ajustar separadamente as transferências simultâneas em segundo plano.
- `webshart.metadata_cache_dir` armazena o catálogo de shards e o índice do manifesto de captura. O padrão é `metadata_cache` dentro de `webshart.cache_dir`.
- `webshart.shard_cache_dir` armazena os shards baixados. O padrão é `shard_cache` dentro de `webshart.cache_dir`.
- Com o cache de fragmentos inteiros ativado para Parquet remoto, cada backend mantém aberto um leitor de fragmentos em cache entre as leituras. Reserve espaço em disco além de `shard_cache_gb` para downloads simultâneos e fragmentos que ainda estão bloqueados por leitores ativos. O Webshart grava cada download em um arquivo temporário antes de remover os fragmentos antigos.
Expand Down
1 change: 1 addition & 0 deletions documentation/DATALOADER.zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -1442,6 +1442,7 @@ Webshart 数据集通过 `webshart` 包加载 WebDataset 风格的 tar shards。
- `metadata` 可选,可指向包含 captions 的独立 metadata location。对于 `webshart/conceptual-captions-12m-webdataset-metadata` 这样的 Hugging Face metadata repo,传 repo id 即可;Webshart 会跟随 source shard 的 `data/` 等子目录布局。
- `metadata_backend` 必须为 `webshart`;`caption_strategy` 应为 `webshart` 或 `instanceprompt`。
- `webshart.cache_dir` 存储 SimpleTuner metadata 与 Webshart caches。`shard_cache_gb` 和 `parallel_downloads` 会传给 Webshart 的 shard cache;将 `shard_cache_gb` 设为 `0` 可禁用整 shard cache,并保留基于索引的 range reads。
- 按窗口顺序训练远程 Parquet 数据时,支持 `sequential_prefill` 的 Webshart 会依次预取后续分片,直到再加入一个分片就会超过 `shard_cache_gb`。每个共享缓存由一个工作进程负责,最多同时执行 `parallel_downloads` 个下载。恢复训练时从已保存的窗口开始,已用完的分片可被缓存淘汰。 恢复训练时,可以用 `SIMPLETUNER_WEBSHART_PREFILL_DOWNLOADS` 单独调整后台并发下载数。
- `webshart.metadata_cache_dir` 存放分片目录和采集清单索引。默认位于 `webshart.cache_dir` 下的 `metadata_cache`。
- `webshart.shard_cache_dir` 存放已下载的分片。默认位于 `webshart.cache_dir` 下的 `shard_cache`。
- 为远程 Parquet 启用整分片缓存时,每个后端在两次读取之间保留一个已缓存分片的读取器。除 `shard_cache_gb` 外,还需为并发下载和仍被读取器锁定的分片预留磁盘空间。Webshart 先将下载内容写入临时文件,再清理旧分片。
Expand Down
15 changes: 15 additions & 0 deletions simpletuner/helpers/data_backend/webshart.py
Original file line number Diff line number Diff line change
Expand Up @@ -204,6 +204,21 @@ def _open_source(self):
)
return dataset

def sequential_prefill(self, window_shards):
# Webshart releases before the byte-bounded cursor use on-demand caching.
if not hasattr(self.dataset, "sequential_prefill") or self.shard_cache_dir is None or Path(self.source).is_dir():
return None
downloads = os.environ.get("SIMPLETUNER_WEBSHART_PREFILL_DOWNLOADS")
if downloads is None:
return self.dataset.sequential_prefill(active_shards=window_shards)
downloads = int(downloads)
if downloads < 1:
raise ValueError("SIMPLETUNER_WEBSHART_PREFILL_DOWNLOADS must be positive.")
# Keep foreground I/O and the checkpointed dataset configuration independent.
reader = self.open_refresh_source().dataset
reader.enable_shard_cache(self.shard_cache_dir, self.shard_cache_gb, downloads)
return reader.sequential_prefill(active_shards=window_shards)

def open_refresh_source(self):
"""An unpinned Parquet reader using the same transport and cache settings."""
fresh = copy(self)
Expand Down
11 changes: 10 additions & 1 deletion simpletuner/helpers/metadata/backends/webshart.py
Original file line number Diff line number Diff line change
Expand Up @@ -501,8 +501,17 @@ def _window_buckets(self, partition):
def open_epoch(self, spec):
if self._window_recipe is not None:
partitions = ceil(self.data_backend.num_shards() / self.window_shards)
prefill = None
if spec.sampling_order == "sequential" and getattr(self.accelerator, "is_local_main_process", True):
prefill = self.data_backend.sequential_prefill(self.window_shards)
return WindowedWebshartEpoch(
spec, self._source_description()["source_revision"], self._window_revision, partitions, self._window_buckets
spec,
self._source_description()["source_revision"],
self._window_revision,
partitions,
self._window_buckets,
window_shards=self.window_shards,
prefill=prefill,
)
source = self._open_membership()
try:
Expand Down
8 changes: 7 additions & 1 deletion simpletuner/helpers/metadata/backends/webshart_epoch.py
Original file line number Diff line number Diff line change
Expand Up @@ -317,11 +317,13 @@ class WindowedWebshartEpoch:

version = "shard-window-v1"

def __init__(self, spec, source, view, partition_count, open_partition):
def __init__(self, spec, source, view, partition_count, open_partition, *, window_shards=None, prefill=None):
self.spec = spec
self.source = source
self.view = view
self.partition_count = partition_count
self.prefill = prefill
self.window_shards = window_shards
self.open_partition = open_partition
self.order_key = _key(spec.seed, spec.epoch, spec.dataset_id, self.version)
self.global_batch = spec.batch_size * spec.dp_size
Expand All @@ -336,6 +338,8 @@ def _open_window(self):
self.partition = self.window
if self.spec.sampling_order == "shuffled":
self.partition = _permute(self.window, self.partition_count, self.order_key)
if self.prefill is not None:
self.prefill.advance(self.partition * self.window_shards)
self.revision, buckets = self.open_partition(self.partition)
self.buckets = sorted(buckets.items())
self.ends = []
Expand Down Expand Up @@ -416,4 +420,6 @@ def load_state_dict(self, state):
self.frontier = (window, batch, self.revision)

def close(self):
if self.prefill is not None:
self.prefill.close()
self.buckets = ()
28 changes: 26 additions & 2 deletions tests/test_webshart_epoch.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@
from itertools import groupby
from pathlib import Path
from tempfile import TemporaryDirectory
from unittest.mock import patch
from unittest.mock import Mock, patch

from simpletuner.helpers.metadata.backends.webshart_epoch import EpochSpec, WebshartEpoch
from simpletuner.helpers.metadata.backends.webshart_epoch import EpochSpec, WebshartEpoch, WindowedWebshartEpoch
from simpletuner.helpers.metadata.backends.webshart_exclusions import Exclusions, _publish
from simpletuner.helpers.metadata.backends.webshart_geometry import snapshot_path, stage_geometry
from simpletuner.helpers.metadata.backends.webshart_membership import (
Expand Down Expand Up @@ -60,6 +60,30 @@ def publish_geometry(path, buckets, shard_of, *, shards_per_partition=None):
return destination


class WindowPrefillTests(unittest.TestCase):
def test_prefill_resumes_at_first_shard_of_saved_window(self):
spec = EpochSpec("data", "shard_local", 42, 1, 1, 1, 0, 1, 0, False, (), sampling_order="sequential")

def window(index):
return str(index), {"bucket": [f"{index}/0", f"{index}/1"]}

with closing(WindowedWebshartEpoch(spec, "source", "view", 8, window)) as original:
for _ in range(7):
original.next_batch()
saved = original.state_dict()
expected = [original.next_batch() for _ in range(3)]

prefill = Mock()
with closing(WindowedWebshartEpoch(spec, "source", "view", 8, window, window_shards=4, prefill=prefill)) as resumed:
prefill.advance.assert_not_called()
resumed.load_state_dict(saved)
prefill.advance.assert_called_once_with(12)
self.assertEqual(resumed.state_dict(), saved)
self.assertEqual([resumed.next_batch() for _ in range(3)], expected)
self.assertEqual(prefill.advance.call_args.args, (16,))
prefill.close.assert_called_once()


class WebshartEpochTests(unittest.TestCase):
def setUp(self):
directory = TemporaryDirectory()
Expand Down
16 changes: 16 additions & 0 deletions tests/test_webshart_parquet_backend.py
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,22 @@ def setUp(self):
None, "parquet", str(source), format="parquet", cache_dir=str(self.root / "cache")
)

def test_prefill_concurrency_uses_a_separate_reader(self):
backend = self.backend
backend.source = "s3://capture"
foreground = Mock()
backend.dataset = foreground
background = Mock()
with (
patch.dict("os.environ", {"SIMPLETUNER_WEBSHART_PREFILL_DOWNLOADS": "16"}),
patch.object(backend, "open_refresh_source", return_value=SimpleNamespace(dataset=background)),
):
self.assertIs(backend.sequential_prefill(4), background.sequential_prefill.return_value)
background.enable_shard_cache.assert_called_once_with(backend.shard_cache_dir, backend.shard_cache_gb, 16)
background.sequential_prefill.assert_called_once_with(active_shards=4)
self.assertEqual(backend.parallel_downloads, 4)
foreground.enable_shard_cache.assert_not_called()

def test_bundled_pixart_plan_prepares_all_three_uncropped_stages(self):
from simpletuner.cli.examples import copy_example, list_examples
from simpletuner.helpers.data_backend.dataset_runtime import DatasetRuntime
Expand Down
Loading