[Standard] | [Reader] | [World Snapshot Models] | [WebUI] | [Documentation] | [Organization]
World Snapshot Graphics (.wsg) is a 256-channel image format that records a world snapshot on one two-dimensional image plane. RGB, geometry, motion, human pose, object structure, physical properties, materials, semantics, audio, and other machine-usable information remain spatially aligned inside the same file.
World Snapshot Video (.wsv) is the multi-frame form of WSG. Together, WSG and WSV provide the data boundary used by World Snapshot Models: RGB is one part of a snapshot, rather than the endpoint of Computer Vision.
This repository contains the complete WSG/WSV Maker and WSG Reader / WSV Player. If you only need to read or visualize existing files, use the smaller wsg_reader repository.
| Format | Unit | Repository representation |
|---|---|---|
.wsg |
One world snapshot | NumPy-compatible archive containing an H × W × 256 array, normally under wsg_data |
.wsv |
An ordered snapshot sequence | NumPy/ZIP-compatible archive containing frames such as index_00000001, index_00000002, ... |
The channel axis stores defined variables rather than arbitrary feature dimensions. Channels 0–2 store RGB; the remaining channels store dense values, sparse structures, identifiers, encoded payloads, or reserved data according to the WSG standard. The value -1 denotes an unavailable or undefined location unless a channel contract specifies otherwise.
Raw WSG preserves injected values. Normalized WSG is used for model training and must carry reversible metadata: the modality definition, linear or logarithmic mode, and value range. main/normalized_wsg_loader.py uses a normalized file only when this information is available; otherwise it safely reads the raw WSG.
Non-RGB channels are not ordinary pseudo-RGB images. The reader applies each modality's official interpretation, palette, value range, and legend.
WSG construction has three direct stages:
- Extract: obtain world information from images, videos, meshes, annotations, or sensors.
- Inject: write the extracted values and their spatial relationships into defined WSG channels.
- Read: interpret, visualize, reconstruct, or export the requested information.
The scripts follow the same naming convention:
pipelineNN_extract_* source data → modality result
pipelineNN_inject_* modality result → WSG/WSV channels
pipelineNN_read_* WSG/WSV channels → values and official visualization
Single-frame WSG files can be merged into one WSV archive with pipeline32_merge_wsg_to_wsv.py. Large datasets can therefore be built frame by frame and deployed as compact sequential archives.
The reported WSM v1 data was built with the following extraction tools and source datasets. This section lists what entered the released training/evaluation system, rather than every backend explored during development.
- DWPose: 2D articulated human structure.
- SMPLer-X: 3D human mesh and body parameters.
- MediaPipe Pose: 3D skeletal human representation.
- MediaPipe Seg: human-part segmentation.
- PrimeDepth: relative depth.
- Depth Pro: metric depth.
- NormalCrafter: surface normals.
- The repository's curvature and edge extraction pipelines.
- PTLFlow with FlowNetSD: optical flow.
Object structure, physical properties, material values, PBR maps, semantic instances, and audio were injected from the source annotations or rendered assets below instead of being re-estimated by an additional model.
- AVIS: RGB/video, relative and metric depth, curvature, edge, optical flow, human representations, semantic instances, captions, and frame-aligned audio.
- Articulation-XL 2.0: object category, mesh, vertices, rig, UV, joints, and articulated structure.
- ShapeNetSem: object structure, category, physical properties, conventional material attributes, and optical attributes.
- MatSynth: base color, diffuse, normal, height, roughness, metallic, specular, opacity, and other PBR material maps.
- XPose: a separately prepared RGB/pose cache used for the released channels 122–124.
Almost all currently defined WSG information can be estimated from visual inputs. When reliable measurements or annotations already exist, injecting them directly avoids unnecessary re-estimation and preserves a cleaner training target.
The repository preserves several alternatives and prototypes that informed development but did not contribute to the WSM v1 results reported in the paper. Their presence in the repository does not mean that the released model was trained on them.
- Sapiens: retained extraction prototypes; the released human segmentation path uses MediaPipe Seg.
- Depth Any Camera and Metric3D v2: depth/normal alternatives considered before the final PrimeDepth, Depth Pro, and NormalCrafter paths.
- DPFlow: considered as an optical-flow alternative; the released extraction script uses PTLFlow FlowNetSD.
- VGGT and MonST3R: extract-only experimental pipelines for camera and temporal geometry; no corresponding released WSG injection path was used in WSM v1.
- ParticleSfM: camera and point-track reference.
- Tokenize Anything: text segmentation and visual-annotation reference.
- Neural LightRig: material/illumination reference.
- SF3D and Pixel3DMM: UV, geometry, normal, and material alternatives.
- Meerkat: audio-visual reference considered as an AVIS alternative.
- DynPose-100K, Panda-70M, and Koala-36M: camera, pose, video, and caption references.
- UniRig: articulated-rig alternative.
- DAM and RAM: mask-description and recognition references.
- Multiple Light Source Dataset: illumination reference.
- ShapeNetCore: considered alongside ShapeNetSem; the reported WSM v1 physical-property data uses ShapeNetSem.
- MMFi: WSG injection/reader support and a prepared cache exist, but the wireless channels were not included in the released WSM v1 task graph or reported evaluation.
- UKBOB: object/physical-property reference.
- image2mass, M3-VOS, and ImageNet: investigated as possible future mass, phase-transition, and general-image sources.
Clone the full maker and reader:
git clone https://github.com/World-Snapshot/wsg.git
cd wsg/mainThe extraction backends use separate environments because pose, depth, geometry, rendering, and sensor models have different dependencies. Environment definitions are stored in main/env/.
For the complete workflow, use:
pipeline_one_stop_extract.ipynb
pipeline_one_stop_inject.ipynb
pipeline_one_stop_read.ipynb
pipeline_one_stop_tell_you_how_to_process_dataset.ipynb
The normal sequence is:
- Place the source dataset under
input/, or set its external path in the selected pipeline. - Run
pipeline_one_stop_extract.ipynbto create the required environments and extract available modalities. - Run
pipeline_one_stop_inject.ipynbto initialize RGB and inject the extracted information. - Run
pipeline_one_stop_read.ipynbto verify values and official visualizations. - Merge validated frame files into
wsv/merge.wsvwhen a sequential archive is required.
Each stage can also be run through its individual Python script. Dataset paths and model checkpoints remain explicit because different source datasets expose different contracts.
To read WSG/WSV without installing every extraction backend:
git clone https://github.com/World-Snapshot/wsg_reader.gitThe current AVIS cache uses one merged wsv/merge.wsv per video. Semantic and audio injection preserve the complete frame-level relationship:
- channel 195 stores every valid AVIS category inside its own instance mask;
- channels 175–184 store frame-aligned mixed audio over the union of all valid instance masks;
- simultaneous instances are retained instead of being reduced to the largest object;
- frame-to-frame audio amplitude is preserved;
- incomplete audio payloads are never written; and
- each WSV is atomically replaced only after a successful rewrite.
Check paths and annotations without modifying data:
cd main
python pipeline29_inject_avis_semantic_instance_into_wsv.py \
--base-output-dir ../output \
--subsets train val test \
--dry-run
python pipeline18_inject_audio_into_wsg.py \
--base-output-dir ../output \
--subsets train val test \
--dry-runRemove --dry-run after the reported video, annotation, WAV, and WSV paths are correct. Both pipelines support deterministic sharding for large datasets.
The reader has three jobs:
- Interpret: recover the stored numerical, categorical, structural, or encoded value.
- Visualize: render it with the modality's official palette and global value definition.
- Save: export images, structured values, audio, meshes, or other modality-specific results.
Shared categorical palettes are stored in main/configs/channel_visualization_palettes.json. The readers under main/pipeline*_read_*.py are also used by WSM evaluation and ControlWorld, keeping data construction, evaluation, paper figures, and WebUI inference under the same visualization contract.
wsg/
├── main/
│ ├── env/ # Per-backend environments
│ ├── configs/ # Visualization and channel configuration
│ ├── tools/ # Dataset preprocessing tools
│ ├── pipeline00_* # Frame extraction and WSG initialization
│ ├── pipeline01-20_* # Modality extract/inject/read pipelines
│ ├── pipeline29-33_* # WSV injection, repair, merge, and release tools
│ └── pipeline_one_stop_*.ipynb # Complete workflows
├── common/ # Shared model and geometry utilities
├── data/ # Dataset adapters
├── input/ # Local input workspace
├── output/ # Generated result workspace
├── pretrained_models/ # Local extraction checkpoints
├── LICENSE
└── README.md
- WSMs trains task routes on WSG variables and produces requested WSG targets.
- ControlWorld selects a task, runs WSM inference, and displays the target through its WSG Reader.
- wsg_reader provides the portable read/visualize subset without the extraction toolchain.
In a historical pose-domain measurement, 12 GB of source data produced 382 GB of intermediate visualizations and at least 100 GB of valid extracted information; injecting the machine-usable values into WSG increased final storage by 66 GB. WSG retains substantially more information than RGB without requiring every intermediate visualization to be distributed.
In short: You are welcome to use, study, modify, and extend this project
for noncommercial purposes, including research, experimentation, development,
training, evaluation, education, and personal projects. Commercial use
requires separate permission from Rutgers; contact
compliance@research.rutgers.edu and reference Rutgers Docket #2027-020. The
LICENSE file contains the governing terms.
The original WSG/WSV format implementation, registry, normalization,
serialization, reader/writer code, and documentation in this repository are
released under the WSM Non-commercial Research License. Use is
limited to academic, research, experimental, personal, and other
noncommercial purposes. For commercial-use rights, contact Rutgers at
compliance@research.rutgers.edu and reference Rutgers Docket #2027-020.
Any Rutgers-owned model artifacts distributed with this repository, including
trained or fine-tuned weights, checkpoints, adapters, and .pth files, are
covered by the same license.
Third-party extraction tools, pretrained models, datasets, and adapted components retain their respective licenses and terms. The RU-NCRL does not relicense or claim ownership of those third-party materials.
WSG turns geometry, motion, pose, physics, materials, semantics, audio, and other information into addressable variables on the same image plane. Once information shares this boundary, it can be read, generated, transformed, and connected by one visual system.