Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion documentation/DATALOADER.es.md
Original file line number Diff line number Diff line change
Expand Up @@ -845,7 +845,7 @@ En este ejemplo, el dataset de 512px se usa para los pasos 1-300, luego el datas
- **Advertencia:** Esto es destructivo y no se puede deshacer. Úsalo con cuidado.
- **Default:** Usa el argumento `--delete_problematic_images` del trainer (default: `false`).

En Webshart nativo, esta opción registra las muestras ilegibles como exclusiones en la vista preparada. El entrenamiento en espacio de píxeles y la codificación VAE bajo demanda decodifican las imágenes aceptadas antes del entrenamiento. Con latentes precalculados, la preparación del VAE realiza la comprobación.
Webshart nativo conserva los fragmentos de origen. Durante el entrenamiento por streaming con Parquet, el muestreador registra las filas ilegibles en la caché de metadatos para esa revisión del fragmento y ocupa sus lugares con filas del mismo grupo de relación de aspecto. Si es necesario, repite una fila legible para completar el lote de cada proceso. Todos los procesos omiten un grupo sin filas legibles. Los conjuntos indexados registran las exclusiones durante la preparación.

### `delete_nsfw_images`

Expand Down
2 changes: 1 addition & 1 deletion documentation/DATALOADER.hi.md
Original file line number Diff line number Diff line change
Expand Up @@ -845,7 +845,7 @@ effective_batch_size = dataset train_batch_size × num_gpus × gradient_accumula
- **Warning:** यह destructive है और undo नहीं किया जा सकता। सावधानी से उपयोग करें।
- **Default:** trainer के `--delete_problematic_images` argument पर fallback करता है (डिफ़ॉल्ट: `false`)।

नेटिव Webshart में यह विकल्प न पढ़े जा सकने वाले नमूनों को तैयार व्यू की बहिष्करण सूची में दर्ज करता है। पिक्सेल-स्पेस प्रशिक्षण और ऑन-डिमांड VAE एन्कोडिंग में स्वीकृत इमेज प्रशिक्षण से पहले डिकोड की जाती हैं। पहले से गणना किए गए लैटेंट के लिए यह जाँच VAE की तैयारी के दौरान होती है।
नेटिव Webshart मूल शार्ड को सुरक्षित रखता है। Parquet स्ट्रीमिंग प्रशिक्षण में सैंपलर अपठनीय पंक्तियों को शार्ड के उस संस्करण के लिए मेटाडेटा कैश में दर्ज करता है और उसी आस्पेक्ट रेशियो बकेट से उनकी जगह भरता है। आवश्यकता पड़ने पर हर रैंक का बैच पूरा करने के लिए पढ़ने योग्य पंक्ति दोहराई जाती है। यदि पूरा बकेट अपठनीय है तो सभी रैंक उसे छोड़ देते हैं। इंडेक्स वाले डेटासेट में बहिष्करण तैयारी के समय दर्ज होते हैं।

### `delete_nsfw_images`

Expand Down
2 changes: 1 addition & 1 deletion documentation/DATALOADER.ja.md
Original file line number Diff line number Diff line change
Expand Up @@ -845,7 +845,7 @@ effective_batch_size = データセットの train_batch_size × num_gpus × gra
- **警告:** 破壊的で元に戻せません。注意して使用してください。
- **既定値:** トレーナーの `--delete_problematic_images` 引数(既定: `false`)にフォールバックします。

ネイティブ Webshart では、このオプションは読み取れないサンプルを準備済みビューの除外項目として記録します。ピクセル空間の学習とオンデマンド VAE エンコードでは、学習前に採用した画像をデコードします。潜在表現を事前計算する場合は VAE の前処理で確認します。
ネイティブ Webshart は元のシャードを保持します。Parquet のストリーミング学習では、サンプラーが読み取れない行をシャードのリビジョンごとにメタデータキャッシュへ記録し、同じアスペクト比のバケットから補充します。必要に応じて読み取れる行を再利用し、各ランクのバッチサイズを維持します。バケット全体が読み取れない場合は全ランクがそのバケットをスキップします。インデックス付きデータセットでは準備時に除外項目を記録します。

### `delete_nsfw_images`

Expand Down
2 changes: 1 addition & 1 deletion documentation/DATALOADER.md
Original file line number Diff line number Diff line change
Expand Up @@ -847,7 +847,7 @@ In this example, the 512px dataset is used for steps 1-300, then the 1024px data
- **Warning:** This is destructive and cannot be undone. Use with caution.
- **Default:** Falls back to the trainer's `--delete_problematic_images` argument (default: `false`).

For native Webshart, this option records unreadable samples as exclusions in the prepared view. Pixel-space training and on-demand VAE encoding decode the accepted images before training. With precomputed latents, VAE preparation handles the check.
Native Webshart keeps source shards intact. In streaming Parquet training, the sampler records unreadable rows in the metadata cache for that shard revision and fills their slots from the same aspect bucket. If needed, it repeats a readable row to keep each rank’s batch full. All ranks skip a bucket with no readable rows. Indexed datasets record exclusions during preparation.

### `delete_nsfw_images`

Expand Down
2 changes: 1 addition & 1 deletion documentation/DATALOADER.pt-BR.md
Original file line number Diff line number Diff line change
Expand Up @@ -845,7 +845,7 @@ Neste exemplo, o dataset de 512px é usado para steps 1-300, então o dataset de
- **Aviso:** Isso é destrutivo e não pode ser desfeito. Use com cuidado.
- **Padrão:** Usa o argumento do trainer `--delete_problematic_images` (padrão: `false`).

No Webshart nativo, esta opção registra amostras ilegíveis como exclusões na visualização preparada. O treinamento no espaço de pixels e a codificação sob demanda pelo VAE decodificam as imagens aceitas antes do treinamento. Com latentes pré-computados, a preparação do VAE faz essa verificação.
O Webshart nativo preserva os fragmentos de origem. No treinamento por streaming com Parquet, o amostrador registra as linhas ilegíveis no cache de metadados para a revisão do fragmento e preenche as vagas com linhas do mesmo grupo de proporção. Se necessário, repete uma linha legível para completar o lote de cada processo. Todos os processos pulam grupos sem linhas legíveis. Em conjuntos indexados, as exclusões são registradas durante a preparação.

### `delete_nsfw_images`

Expand Down
2 changes: 1 addition & 1 deletion documentation/DATALOADER.zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -844,7 +844,7 @@ effective_batch_size = 数据集 train_batch_size × num_gpus × gradient_accumu
- **警告:** 此操作不可恢复,请谨慎使用。
- **默认值:** 回退到训练器的 `--delete_problematic_images` 参数(默认:`false`)。

对于原生 Webshart,此选项将无法读取的样本记录为已准备视图中的排除项。像素空间训练和按需 VAE 编码会在训练前解码已接受的图像。预计算潜变量时,由 VAE 预处理执行检查。
原生 Webshart 保留源分片。流式 Parquet 训练时,采样器将无法解码的行按分片版本记入元数据缓存,并用同一宽高比分桶中的行补齐空位。必要时会重复使用可读行,以保证各进程的批次大小一致。如果整个分桶都不可读,所有进程都会跳过它。索引数据集在准备阶段记录排除项。

### `delete_nsfw_images`

Expand Down
13 changes: 8 additions & 5 deletions simpletuner/helpers/caching/vae.py
Original file line number Diff line number Diff line change
Expand Up @@ -1202,7 +1202,7 @@ def _sample_latent_distribution(self, distribution, filenames):
return distribution.sample(generator=generators)
return distribution.sample()

def _load_or_encode_latents(self, images, filepaths, cache_paths, *, view=None):
def _load_or_encode_latents(self, images, filepaths, cache_paths, *, view=None, decoded_images=None):
latents = list(self.cache_data_backend.read_cached_tensors(cache_paths))
missing = [index for index, (_, latent) in enumerate(zip(cache_paths, latents, strict=True)) if latent is None]
if not missing:
Expand All @@ -1211,8 +1211,11 @@ def _load_or_encode_latents(self, images, filepaths, cache_paths, *, view=None):
pixels = [(images[i], filepaths[i], None, False) for i in missing if images[i] is not None]
image_refs = [filepaths[i] for i in missing if images[i] is None]
if image_refs:
payloads = self._read_from_storage_concurrently(image_refs, hide_errors=True)
image_data = list(ordered_sample_payloads(self.id, image_refs, payloads))
if decoded_images is None:
payloads = self._read_from_storage_concurrently(image_refs, hide_errors=True)
image_data = list(ordered_sample_payloads(self.id, image_refs, payloads))
else:
image_data = [decoded_images[ref] for ref in image_refs]
pixels.extend(self._process_images_in_batch(image_refs, image_data, disable_queue=True, view=view))
encoded = self._encode_images_in_batch(image_pixel_values=pixels, disable_queue=True, view=view) or []
written = self._write_latents_in_batch(encoded)
Expand All @@ -1222,7 +1225,7 @@ def _load_or_encode_latents(self, images, filepaths, cache_paths, *, view=None):
latents[index] = latent
return latents

def encode_images(self, images, filepaths, load_from_cache=True, *, view=None):
def encode_images(self, images, filepaths, load_from_cache=True, *, view=None, decoded_images=None):
# images must be same dimension
# if load_from_cache=True and entry not found, throws exception
batch_size = len(images)
Expand All @@ -1231,7 +1234,7 @@ def encode_images(self, images, filepaths, load_from_cache=True, *, view=None):

full_filenames = [self.generate_vae_cache_filename(filepath, view=view)[0] for filepath in filepaths]
if load_from_cache and self.vae_cache_ondemand:
return self._load_or_encode_latents(images, filepaths, full_filenames, view=view)
return self._load_or_encode_latents(images, filepaths, full_filenames, view=view, decoded_images=decoded_images)

uncached_image_indices = [
i
Expand Down
2 changes: 1 addition & 1 deletion simpletuner/helpers/data_backend/webshart.py
Original file line number Diff line number Diff line change
Expand Up @@ -723,7 +723,7 @@ def read_image_batch(self, filepaths: list, delete_problematic_images: bool = Fa
**({"metadata_fields": ()} if self.supports_caption_projection else {}),
)
available_keys, output_images = [], []
for filepath, sample in zip(filepaths, samples):
for filepath, sample in zip(filepaths, samples, strict=True):
if sample.data is None:
continue
try:
Expand Down
60 changes: 57 additions & 3 deletions simpletuner/helpers/metadata/backends/webshart.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
from copy import copy, deepcopy
from functools import partial
from hashlib import file_digest, sha256
from itertools import batched, chain, islice
from itertools import batched, chain, cycle, islice
from math import ceil
from pathlib import Path
from queue import SimpleQueue
Expand All @@ -36,7 +36,9 @@
from simpletuner.helpers.metadata.backends.webshart_exclusions import (
collect_exclusions,
exclusion_path,
load_window_exclusions,
retain_source_exclusions,
window_exclusion_db,
)
from simpletuner.helpers.metadata.backends.webshart_geometry import (
_read_index,
Expand Down Expand Up @@ -91,6 +93,8 @@ class _TrainingWindow(NamedTuple):
buckets: dict[str, list[str]]
records: dict[str, dict]
captions: dict[str, CaptionValue]
pins: dict[int, str]
excluded: set[str]


class WebshartMetadataBackend(MetadataBackend):
Expand Down Expand Up @@ -431,7 +435,6 @@ def prepare_for_training(self, request):
and getattr(args, "data_backend_sampling", "uniform") != "auto-weighting"
and self.max_num_samples is None
and not (config.get("crop") and config.get("crop_style") == "face")
and not self.delete_problematic_images
and not self.delete_unwanted_images
):
return False
Expand Down Expand Up @@ -490,10 +493,61 @@ def _open_training_partition(self, partition):
captions.pop(ref, None)
self._window_transforms.flush()
revision = sha256(json.dumps([pins, buckets], sort_keys=True).encode()).hexdigest()
window = _TrainingWindow(partition, revision, buckets, records, captions)
shard_pins = dict(zip(indices, pins, strict=True))
excluded = (
load_window_exclusions(self.window_exclusions_path, shard_pins) if self.delete_problematic_images else set()
)
window = _TrainingWindow(partition, revision, buckets, records, captions, shard_pins, excluded)
self._window_cache = window
return window

@property
def window_exclusions_path(self):
return self.membership_path.with_suffix(".runtime-exclusions.sqlite")

def read_window_batch(self, partition, bucket, refs):
"""Replace undecodable occurrences before resolving their captions and conditioning."""
window = self._open_training_partition(partition)
images = {}

def read(candidates):
requested = [ref for ref in dict.fromkeys(candidates) if ref not in window.excluded]
if not requested:
return
paths, decoded = self.data_backend.read_image_batch(requested)
images.update(zip(paths, decoded, strict=True))
missing = [ref for ref in requested if ref not in images]
if missing:
with window_exclusion_db(self.window_exclusions_path) as db:
for ref in missing:
shard = self.data_backend.shard_of(ref)
db.execute(
"INSERT OR IGNORE INTO exclusions VALUES (?,?,?,'decode')", (shard, window.pins[shard], ref)
)
window.excluded.update(missing)
logger.warning("(id=%s) Excluded unreadable Webshart rows: %s", self.id, missing)

read(refs)
vacancies = sum(ref not in images for ref in refs)
if vacancies:
members = window.buckets[bucket]
start = members.index(refs[-1]) + 1
ordered = chain(islice(members, start, None), islice(members, start))
candidates = (ref for ref in ordered if ref not in window.excluded and ref not in images)
replacements = []
while len(replacements) < vacancies:
group = list(islice(candidates, vacancies - len(replacements)))
if not group:
break
read(group)
replacements.extend(ref for ref in group if ref in images)
if not images:
# Every rank has the same bucket and will advance past this occurrence.
return [], {}
pads = chain(replacements, cycle(images))
refs = [ref if ref in images else next(pads) for ref in refs]
return refs, images

def _window_buckets(self, partition):
window = self._open_training_partition(partition)
return window.revision, window.buckets
Expand Down
22 changes: 22 additions & 0 deletions simpletuner/helpers/metadata/backends/webshart_exclusions.py
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,28 @@ def read_exclusion_page(publication, header, reason, limit, offset):
return counts, rows.fetchall()


@contextmanager
def window_exclusion_db(path):
path.parent.mkdir(parents=True, exist_ok=True)
with closing(sqlite3.connect(path, timeout=30)) as db, db:
db.execute(
"CREATE TABLE IF NOT EXISTS exclusions("
"shard INTEGER, pin TEXT, ref TEXT, reason TEXT NOT NULL, PRIMARY KEY(shard, pin, ref)) WITHOUT ROWID"
)
yield db


def load_window_exclusions(path, pins):
if not path.exists():
return set()
with window_exclusion_db(path) as db:
return {
ref
for shard, pin in pins.items()
for (ref,) in db.execute("SELECT ref FROM exclusions WHERE shard=? AND pin=?", (shard, pin))
}


class Exclusions:
def __init__(self, db):
self.db = db
Expand Down
8 changes: 8 additions & 0 deletions simpletuner/helpers/multiaspect/sampler.py
Original file line number Diff line number Diff line change
Expand Up @@ -213,7 +213,15 @@ def _iterate_native(self):
spec = cursor.spec
rng = random.Random(f"{self.id}:{spec.seed}:{spec.epoch}:{spec.dp_rank}:{cursor.position}")
view = DatasetReadView(state["view"], spec.epoch, spec.seed, getattr(cursor, "partition", None))
images = None
if view.partition is not None and self.epoch_provider.delete_problematic_images:
refs, images = self.epoch_provider.read_window_batch(view.partition, bucket, refs)
if not refs:
continue
records = self._validate_and_yield_images_from_samples(refs, bucket, rng=rng, view=view)
if images is not None:
for record in records:
record["decoded_training_image"] = images[record["image_path"]]
records[0]["dataset_continuation"] = state
records[0]["dataset_read_view"] = view
records[0]["caption_dropout_seed"] = rng.getrandbits(64)
Expand Down
22 changes: 15 additions & 7 deletions simpletuner/helpers/training/collate.py
Original file line number Diff line number Diff line change
Expand Up @@ -183,13 +183,15 @@ def fetch_latent(fp, data_backend_id: str, vae_cache, view=None):
return latent


def pixel_batch(filepaths, data_backend_id: str, model, *, dataset=None, view=None):
def pixel_batch(filepaths, data_backend_id: str, model, *, dataset=None, view=None, decoded_images=None):
"""Stack normalised pixel tensors for models without an autoencoder."""
if dataset is None:
dataset = StateTracker.get_dataset(data_backend_id)
pixels = []
for record in dataset.read_samples(filepaths, fields=("image", "metadata"), view=view):
image = record.pop("image")
fields = ("image", "metadata") if decoded_images is None else ("metadata",)
records = dataset.read_samples(filepaths, fields=fields, view=view)
for record in records:
image = record.pop("image") if decoded_images is None else decoded_images[record["image_path"]]
sample = TrainingSample(
image=image,
image_path=record["image_path"],
Expand Down Expand Up @@ -258,19 +260,20 @@ def conditioning_pixels(pairs, *, dataset, model, view=None):
return torch.stack(pixels).to(memory_format=torch.contiguous_format).float()


def compute_latents(filepaths, data_backend_id: str, model, *, vae_cache=None, dataset=None, view=None):
def compute_latents(filepaths, data_backend_id: str, model, *, vae_cache=None, dataset=None, view=None, decoded_images=None):
# Use a thread pool to fetch latents concurrently
try:
extra = {"decoded_images": decoded_images} if decoded_images is not None else {}
if getattr(model, "AUTOENCODER_CLASS", None) is None:
return pixel_batch(filepaths, data_backend_id, model, dataset=dataset, view=view)
return pixel_batch(filepaths, data_backend_id, model, dataset=dataset, view=view, **extra)
if vae_cache is None:
if dataset is None:
dataset = StateTracker.get_dataset(data_backend_id)
vae_cache = dataset.vae_cache
if vae_cache is None:
raise ValueError(f"Dataset {data_backend_id} has no VAE cache for this batch.")
if getattr(vae_cache, "vae_cache_ondemand", False):
latents = vae_cache.encode_images([None] * len(filepaths), filepaths, view=view)
latents = vae_cache.encode_images([None] * len(filepaths), filepaths, view=view, **extra)
else:
with concurrent.futures.ThreadPoolExecutor() as executor:
latents = list(
Expand Down Expand Up @@ -590,7 +593,12 @@ def collate_fn(batch, *, dataset=None):

debug_log("Compute latents")
view = examples[0].get("dataset_read_view")
batch_data = compute_latents(filepaths, batch_backend_id, model, vae_cache=dataset.vae_cache, dataset=dataset, view=view)
inputs = {}
if "decoded_training_image" in examples[0]:
inputs["decoded_images"] = {record["image_path"]: record.pop("decoded_training_image") for record in examples}
batch_data = compute_latents(
filepaths, batch_backend_id, model, vae_cache=dataset.vae_cache, dataset=dataset, view=view, **inputs
)
latent_metadata = None
if isinstance(batch_data[0], dict):
latent_metadata = []
Expand Down
Loading
Loading