README.md
| 1 | --- |
| 2 | license: other |
| 3 | tags: |
| 4 | - world-model |
| 5 | - robotics |
| 6 | - tactile |
| 7 | - cosmos |
| 8 | --- |
| 9 | |
| 10 | # ACTS world model: sensor-only / no-force ablation (code + data + checkpoints) |
| 11 | |
| 12 | This repo contains everything needed to run three action-conditioned visuo-tactile world-model runs on a new machine, and to train |
| 13 | the four follow-up experiments listed under **TODO**. **1116 TODO** (below): val-set inference of pushT / rope at epochs 14 and 15. It holds the code, the preprocessed data, the raw media and the trained |
| 14 | checkpoints. Only the two public base models are downloaded from their own repos. |
| 15 | |
| 16 | | run | task | what it is | ckpt in this repo | status | |
| 17 | |---|---|---|---|---| |
| 18 | | `nano_action_causal_umi_sensoronly_noforce_v1` | motherboard | **Slurm job 884599** (`rebuttal_2`): sensor-only (no view-action streams) + no force | `checkpoints/…/epoch_013.pt` | training was still running at upload (epoch 14/20) | |
| 19 | | `nano_action_causal_pusht_v1` | pushT | full model (sensor + view actions, with force) | `checkpoints/…/epoch_019.pt` (+ `epoch_014/015.pt` for the 1116 val TODO) | done | |
| 20 | | `nano_action_causal_rope_v1` | rope | full model (sensor + view actions, with force) | `checkpoints/…/epoch_019.pt` (+ `epoch_014/015.pt` for the 1116 val TODO) | done | |
| 21 | |
| 22 | ## TODO experiments |
| 23 | |
| 24 | The same ablation that was run on motherboard (sensor-only, with and without force), now on **pushT** and **rope**. |
| 25 | **pushT and rope are two separate tasks**: each is trained only on its own data, and each run is evaluated with **its own |
| 26 | inference script on its own test set**. Nothing is shared between the two tasks except the code. |
| 27 | |
| 28 | | | **pushT** | **rope** | |
| 29 | |---|---|---| |
| 30 | | experiments | 1a `nano_action_causal_pusht_sensoronly_v1` (with force)<br>1b `nano_action_causal_pusht_sensoronly_noforce_v1` (no force) | 2a `nano_action_causal_rope_sensoronly_v1` (with force)<br>2b `nano_action_causal_rope_sensoronly_noforce_v1` (no force) | |
| 31 | | configs | `configs/nano_action_causal_pusht_sensoronly{,_noforce}_v1.yaml` | `configs/nano_action_causal_rope_sensoronly{,_noforce}_v1.yaml` | |
| 32 | | training data | `data/mixed_pushT_{latents,actions}`, split `splits/mixed_pusht_wrist_holdout.json`: 56 train / 4 val episodes, 55,199 train windows | `data/mixed_rope_{latents,actions}`, split `splits/mixed_rope_wrist_holdout.json`: 53 train / 8 val episodes, 77,188 train windows | |
| 33 | | training time (4× H100, 20 epochs) | ≈ 2.3 h/epoch (6,900 steps) → ~2 days per experiment | ≈ 3.3 h/epoch (9,648 steps) → ~3 days per experiment | |
| 34 | | **inference script** | **`scripts/infer_pusht.sh`** | **`scripts/infer_rope.sh`** | |
| 35 | | **test set** | test split of the pushT split file: 8 episodes → `eval/cases/pusht_test_cases.json`, **30 cases** (28 with a long rollout; cases 5 and 29 are short-only, the segment is too short) | test split of the rope split file: 9 episodes → `eval/cases/rope_test_cases.json`, **34 cases** (all with a long rollout) | |
| 36 | | compared against | `nano_action_causal_pusht_v1` epoch 019 (`eval/cases/pusht_reference_metrics.json`) | `nano_action_causal_rope_v1` epoch 019 (`eval/cases/rope_reference_metrics.json`) | |
| 37 | |
| 38 | Each config differs from its parent (`nano_action_causal_pusht_v1` / `nano_action_causal_rope_v1`) only in `run_name`, |
| 39 | `model.active_streams` (no `view_action_*`), and for the no-force variants `data.action_file: sensor_actions_fps15_noforce.pt`. |
| 40 | |
| 41 | ### Requirements (all 4 experiments) |
| 42 | |
| 43 | * **Each experiment is its own training job on 4× H100 80 GB (one node)**: one DDP rank per GPU, batch 2 per GPU = global batch 8, |
| 44 | ~31 GiB per GPU. Do not change the GPU count or `training.batch_size`; the parent runs and job 884599 used exactly 4×2. |
| 45 | * **Train for the full 20 epochs** (`training.epochs: 20` → `epoch_000.pt` … `epoch_019.pt`). Do not stop early and do not use |
| 46 | `best.pt`. The training is finished only when `runs/<run_name>/epoch_019.pt` exists. |
| 47 | * If a Slurm job hits its time limit or is preempted, submit **the same command again**. It resumes from `runs/<run_name>/last.pt`; |
| 48 | do not change the run name. |
| 49 | * **Only after epoch 20**: run the inference script **of that task** on **that task's test set** (pushT run → `infer_pusht.sh`, |
| 50 | rope run → `infer_rope.sh`). It uploads the checkpoint and the test results to this HF repo, so it needs `HF_TOKEN` with |
| 51 | **write** access to `WitneyWW/acts-sensoronly-ablation`. |
| 52 | |
| 53 | ### Step 1: setup and data (once per machine) |
| 54 | |
| 55 | ```bash |
| 56 | bash scripts/setup_env.sh && source env.sh && export HF_TOKEN=hf_... # see section 1 |
| 57 | $PYTHON_BIN scripts/download.py --what data raw ckpts nano wan_vae --tasks pusht # pushT: train + val + test data |
| 58 | $PYTHON_BIN scripts/download.py --what data raw ckpts nano wan_vae --tasks rope # rope: train + val + test data |
| 59 | ``` |
| 60 | |
| 61 | ### pushT (experiments 1a, 1b) |
| 62 | |
| 63 | ```bash |
| 64 | source env.sh |
| 65 | # train: two separate jobs, 4x H100 each, 20 epochs |
| 66 | sbatch -N1 --ntasks-per-node=4 --gres=gpu:4 scripts/train.sh configs/nano_action_causal_pusht_sensoronly_v1.yaml # 1a |
| 67 | sbatch -N1 --ntasks-per-node=4 --gres=gpu:4 scripts/train.sh configs/nano_action_causal_pusht_sensoronly_noforce_v1.yaml # 1b |
| 68 | # (without Slurm, on a 4x H100 machine: NGPU=4 bash scripts/train.sh configs/<config>.yaml) |
| 69 | |
| 70 | # after runs/<run_name>/epoch_019.pt exists: pushT inference script on the pushT test set (30 cases) + upload, 4 GPUs, ~1 h |
| 71 | sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_pusht.sh configs/nano_action_causal_pusht_sensoronly_v1.yaml # 1a |
| 72 | sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_pusht.sh configs/nano_action_causal_pusht_sensoronly_noforce_v1.yaml # 1b |
| 73 | ``` |
| 74 | |
| 75 | ### rope (experiments 2a, 2b) |
| 76 | |
| 77 | ```bash |
| 78 | source env.sh |
| 79 | # train: two separate jobs, 4x H100 each, 20 epochs |
| 80 | sbatch -N1 --ntasks-per-node=4 --gres=gpu:4 scripts/train.sh configs/nano_action_causal_rope_sensoronly_v1.yaml # 2a |
| 81 | sbatch -N1 --ntasks-per-node=4 --gres=gpu:4 scripts/train.sh configs/nano_action_causal_rope_sensoronly_noforce_v1.yaml # 2b |
| 82 | |
| 83 | # after runs/<run_name>/epoch_019.pt exists: rope inference script on the rope test set (34 cases) + upload, 4 GPUs, ~1 h |
| 84 | sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_rope.sh configs/nano_action_causal_rope_sensoronly_v1.yaml # 2a |
| 85 | sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_rope.sh configs/nano_action_causal_rope_sensoronly_noforce_v1.yaml # 2b |
| 86 | ``` |
| 87 | |
| 88 | Sanity checks at the start of training (`runs/<run_name>/train.log`): `[data] train episodes=56 windows=55199` (pushT) or |
| 89 | `train episodes=53 windows=77188` (rope), and `trainable=45,301,952`. Progress: `ls runs/<run_name>/epoch_*.pt`. |
| 90 | |
| 91 | ### What the inference scripts do |
| 92 | |
| 93 | `scripts/infer_pusht.sh` / `scripts/infer_rope.sh` (both call `scripts/todo_eval_upload.py --task <task>`): |
| 94 | 1. refuse a config of the other task, and refuse to run unless `runs/<run_name>/epoch_019.pt` exists (20 epochs finished); |
| 95 | 2. check that the test cases come from the test split of the same split file the model was trained with; |
| 96 | 3. run `eval/run_test_case.py` with `epoch_019.pt` on **all test cases of that task** (each = short 8→8 prediction + 15-window long |
| 97 | rollout, UniPC 35 steps, seed 0), spread over the 4 GPUs. If interrupted, run the same command again: finished cases are skipped; |
| 98 | 4. check every case finished and write `summary.md` (PSNR/SSIM/LPIPS + paired Δ vs `pusht_v1` / `rope_v1` epoch 019); |
| 99 | 5. upload to this repo: |
| 100 | * `checkpoints/<run_name>/`: `epoch_019.pt`, `config.yaml`, `action_normalizer.json`, `train.log`, `metrics.jsonl`, … |
| 101 | * `results/<run_name>/test_epoch_019/`: every test case (metrics json, predicted latents, GT|PRED PNG strips, `short_gt_pred.mp4`, |
| 102 | `long_gt_pred.mp4`, ~5 GB per experiment) + `summary.md`. |
| 103 | |
| 104 | ### Checklist |
| 105 | |
| 106 | | # | task | trained 20 epochs (`epoch_019.pt`) | inference on own test set | uploaded `checkpoints/` + `results/` | |
| 107 | |---|---|---|---|---| |
| 108 | | 1a sensor-only | pushT | ☐ | ☐ `infer_pusht.sh` (30 cases) | ☐ | |
| 109 | | 1b sensor-only no-force | pushT | ☐ | ☐ `infer_pusht.sh` (30 cases) | ☐ | |
| 110 | | 2a sensor-only | rope | ☐ | ☐ `infer_rope.sh` (34 cases) | ☐ | |
| 111 | | 2b sensor-only no-force | rope | ☐ | ☐ `infer_rope.sh` (34 cases) | ☐ | |
| 112 | |
| 113 | ### What the configs change |
| 114 | |
| 115 | * **sensor-only** = the two per-hand view-action streams are not modelled (not tokenised; `view_action2llm9` frozen). The dataset |
| 116 | still reads `view_actions_fps15.pt`, so the window list and the action normaliser are identical to the parent run's. Trainable |
| 117 | params 45.30 M (vs 45.34 M); 1,310 tokens/sample (vs 1,342). |
| 118 | * **with force** = the sensor action is `sensor_actions_with_dforce_fps15.pt`: `dp = R_kᵀΔp + k_f·Δf·n`, so the change of normal |
| 119 | force enters as a displacement along the gel normal. |
| 120 | * **no force** = `sensor_actions_fps15_noforce.pt`: the same file with the `k_f·Δf·n` term removed (pure geometry); it differs from |
| 121 | the force file only in `dp_y`. |
| 122 | * Everything else = parent recipe: lr 5e-5 (100 warm-up steps), wd 0.01, bf16, LoRA r16/α32 on the GEN tower, Cosmos3-Nano init, seed 0. |
| 123 | |
| 124 | ## 1116 TODO: validation set (pushT / rope) at epochs 14 and 15 |
| 125 | |
| 126 | Added 2026-11-16. Run the **val** split of pushT and rope on **epoch 14 and epoch 15** (`epoch_014.pt`, `epoch_015.pt`) for |
| 127 | the parent runs and for the four TODO runs, so the sensor-only / no-force ablation can be compared on held-out val episodes |
| 128 | at matched epochs. This does not replace the epoch-19 test inference above; it is an extra check. |
| 129 | |
| 130 | ### The validation sets |
| 131 | |
| 132 | Each task's val split is the `val` list of its own split file (the same file the run trains with; val ≠ test, whole source |
| 133 | episodes held out). The latents, actions and raw media of these episodes are in this repo under the same `data/` trees as |
| 134 | train/test, so nothing extra has to be prepared. |
| 135 | |
| 136 | | task | split file | val episodes (segments) | val windows (stride 1, valid actions) | val cases | file | |
| 137 | |---|---|---|---:|---:|---| |
| 138 | | pushT | `splits/mixed_pusht_wrist_holdout.json` | 0910_ep002, 0911_ep003 (seg00, seg01), 0912_ep000 → 4 segments | 4,439 | **14** (all with a 15-window long rollout) | `eval/cases/pusht_val_cases.json` | |
| 139 | | rope | `splits/mixed_rope_wrist_holdout.json` | 0911_ep001 (2 segs), 0914_ep003 (4 segs), 0915_ep002 (2 segs) → 8 segments | 6,174 | **30** (all with a long rollout) | `eval/cases/rope_val_cases.json` | |
| 140 | |
| 141 | Val cases use the test-case protocol: one case per (val segment × contact class `none`/`onset`/`release`/`full`, from |
| 142 | `max(F_L, F_R) > 0.5 N` on the initial 16-frame window) where such a window exists, the window drawn with seed 0 among those |
| 143 | with room for the 15-window rollout. Built by `scripts/build_val_cases.py`, which also checks that the parent config and both |
| 144 | TODO configs of the task give the **identical** val window index, so the same case file serves all three runs. (pushT val |
| 145 | has no `none` window in 0911_ep003; rope 0911_ep001_seg01 and 0914_ep003_seg02 have no `none` window.) |
| 146 | |
| 147 | **Download** (only the val episodes of one task, + checkpoints incl. the parent `epoch_014.pt` / `epoch_015.pt`): |
| 148 | |
| 149 | ```bash |
| 150 | $PYTHON_BIN scripts/download.py --what data raw ckpts nano wan_vae --tasks pusht --splits val |
| 151 | $PYTHON_BIN scripts/download.py --what data raw ckpts nano wan_vae --tasks rope --splits val |
| 152 | ``` |
| 153 | |
| 154 | **Load in Python** (the dataset must be built exactly like this, otherwise the case `index` points at a different window): |
| 155 | |
| 156 | ```python |
| 157 | import json |
| 158 | from omegaconf import OmegaConf |
| 159 | from vtm_cosmos.data.dataset import VtmWindowDataset |
| 160 | from vtm_cosmos.data.splits import load_split |
| 161 | |
| 162 | cfg = OmegaConf.load("configs/nano_action_causal_rope_sensoronly_v1.yaml") # any pushT/rope config; needs `source env.sh` |
| 163 | episodes = load_split(cfg.data.split_file)["val"] # ['rope_0911_episode_001_seg00.pt', ...] |
| 164 | ds = VtmWindowDataset(cfg.data.latent_root, cfg.data.action_root, episodes, |
| 165 | action_file=cfg.data.action_file, window_stride=1, |
| 166 | include_view_actions=True, view_action_file=cfg.data.view_action_file, |
| 167 | require_valid_actions=True, # action_full drops invalid transitions |
| 168 | active_streams=tuple(cfg.model.active_streams) if cfg.model.get("active_streams") else None) |
| 169 | print(len(ds)) # rope 6174, pushT 4439 |
| 170 | item = ds[0] # latents.{video,wrist_left,wrist_right,tactile_left,tactile_right}: (48,4,16,16) = 16 frames @ 15 fps |
| 171 | # actions.action_{left,right}: (15,9); actions.view_action_{left,right}: (16,9); action_valid.*; stream_present |
| 172 | cases = json.load(open("eval/cases/rope_val_cases.json"))["cases"] |
| 173 | c = cases[0]; assert ds.window_meta(c["index"]) == (c["episode"], c["t_start"]) |
| 174 | ``` |
| 175 | |
| 176 | Raw media of a val episode: `data/raw/videos/<episode>/{view_middle,tactile_left,tactile_right,wrist_left,wrist_right}.mp4`, |
| 177 | `data/raw/parquet/<episode>.parquet`. During training the val split is only used for the per-epoch flow loss |
| 178 | (`data.val_window_stride: 50`). |
| 179 | |
| 180 | ### What to run (4× H100 per command, resumable, uploads to this repo) |
| 181 | |
| 182 | `scripts/infer_val.sh <config>` = for epoch 14 and 15: all val cases of that task on that epoch's checkpoint |
| 183 | (`todo_eval_upload.py --split val --epoch E`) → `runs/<run_name>/val_epoch_01{4,5}/summary.md` (paired Δ vs the parent run |
| 184 | at the same epoch, if the parent has already been run) → upload `results/<run_name>/val_epoch_01{4,5}/` and, for the TODO runs, |
| 185 | `checkpoints/<run_name>/epoch_01{4,5}.pt`. It can start as soon as `runs/<run_name>/epoch_014.pt` exists (training can keep |
| 186 | running on another node; epoch 15 needs `epoch_015.pt`). **Run the parent first** so the TODO summaries get the paired column. |
| 187 | |
| 188 | ```bash |
| 189 | source env.sh && export HF_TOKEN=hf_... # write access |
| 190 | # pushT (14 val cases; ~30 min per epoch on 4 GPUs) |
| 191 | sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_val.sh configs/nano_action_causal_pusht_v1.yaml # parent, ckpts shipped |
| 192 | sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_val.sh configs/nano_action_causal_pusht_sensoronly_v1.yaml # 1a |
| 193 | sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_val.sh configs/nano_action_causal_pusht_sensoronly_noforce_v1.yaml # 1b |
| 194 | # rope (30 val cases; ~1 h per epoch on 4 GPUs) |
| 195 | sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_val.sh configs/nano_action_causal_rope_v1.yaml # parent, ckpts shipped |
| 196 | sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_val.sh configs/nano_action_causal_rope_sensoronly_v1.yaml # 2a |
| 197 | sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_val.sh configs/nano_action_causal_rope_sensoronly_noforce_v1.yaml # 2b |
| 198 | # one epoch only / no upload: EPOCHS=14 bash scripts/infer_val.sh <config> --no-upload |
| 199 | # re-summarise later (e.g. after the parent finished): |
| 200 | $PYTHON_BIN eval/summarize.py --task rope --split val --root runs/nano_action_causal_rope_sensoronly_v1/val_epoch_014 \ |
| 201 | --ref-root runs/nano_action_causal_rope_v1/val_epoch_014 |
| 202 | ``` |
| 203 | |
| 204 | Single case by hand: `$PYTHON_BIN eval/run_test_case.py --task pusht --split val --case 3 --ckpt <epoch_014.pt>` |
| 205 | (output `runs/<run dir>/val_epoch_014/03_.../`). |
| 206 | |
| 207 | ### 1116 checklist |
| 208 | |
| 209 | | run | task | epoch 14 val | epoch 15 val | uploaded `results/<run>/val_epoch_01{4,5}/` | |
| 210 | |---|---|---|---|---| |
| 211 | | `nano_action_causal_pusht_v1` (parent) | pushT | ☐ 14 cases | ☐ 14 cases | ☐ | |
| 212 | | 1a `pusht_sensoronly_v1` | pushT | ☐ | ☐ | ☐ (+ ckpts) | |
| 213 | | 1b `pusht_sensoronly_noforce_v1` | pushT | ☐ | ☐ | ☐ (+ ckpts) | |
| 214 | | `nano_action_causal_rope_v1` (parent) | rope | ☐ 30 cases | ☐ 30 cases | ☐ | |
| 215 | | 2a `rope_sensoronly_v1` | rope | ☐ | ☐ | ☐ (+ ckpts) | |
| 216 | | 2b `rope_sensoronly_noforce_v1` | rope | ☐ | ☐ | ☐ (+ ckpts) | |
| 217 | |
| 218 | ## 1. Setup on a new machine |
| 219 | |
| 220 | Tested on Linux, NVIDIA H100 80 GB, a driver that supports CUDA 12.8, and conda. |
| 221 | |
| 222 | ```bash |
| 223 | export HF_TOKEN=hf_... # this repo is private |
| 224 | # code only (skip the ~150 GB of data/ and checkpoints/ for now) |
| 225 | hf download WitneyWW/acts-sensoronly-ablation --exclude "data/*" "checkpoints/*" --local-dir acts-sensoronly-ablation |
| 226 | cd acts-sensoronly-ablation |
| 227 | bash scripts/setup_env.sh # conda env `acts_wm` (override ENV_NAME=..., CONDA_ROOT=...): torch 2.10.0+cu128, |
| 228 | # pinned deps, and NVIDIA/cosmos-framework @ 3f94d25 cloned into third_party/ |
| 229 | source env.sh # paths + PYTHONPATH; PYTHON_BIN defaults to $CONDA_ROOT/envs/acts_wm/bin/python |
| 230 | $PYTHON_BIN scripts/download.py --what all # everything (~175 GB incl. Cosmos3-Nano 29 GB) |
| 231 | # or only what you need, e.g. evaluating pushT on its test split: |
| 232 | $PYTHON_BIN scripts/download.py --what data raw ckpts nano wan_vae --tasks pusht --splits test |
| 233 | ``` |
| 234 | |
| 235 | `download.py` places files in `./data` and `./checkpoints`, which are the `env.sh` defaults of `ACTS_DATA_ROOT` / `ACTS_CKPT_ROOT`. |
| 236 | `--tasks` ∈ `mb` (motherboard), `pusht`, `rope`. Training needs the train and val splits (the default fetches all three). |
| 237 | If the env is not at `~/miniconda3/envs/acts_wm`, export `PYTHON_BIN=/path/to/env/bin/python` before `source env.sh`. |
| 238 | |
| 239 | | variable (env.sh) | default | content | |
| 240 | |---|---|---| |
| 241 | | `ACTS_DATA_ROOT` | `./data` | `mixed_*_latents`, `mixed_*_actions`, `raw/` | |
| 242 | | `ACTS_CKPT_ROOT` | `./checkpoints` | `Cosmos3-Nano/`, `Wan2.2_VAE.pth`, `vae/lora_last.pt`, `nano_action_causal_*/epoch_XXX.pt` | |
| 243 | | `ACTS_RUNS_ROOT` | `./runs` | new training runs and eval outputs | |
| 244 | | `VTM_SOURCE_VIDEO_ROOT` | `$ACTS_DATA_ROOT/raw/videos` | mp4s used to rebuild the VAE anchor frame at decode time | |
| 245 | |
| 246 | ## 2. What is in this repo |
| 247 | |
| 248 | ``` |
| 249 | configs/ exact configs of the 3 runs above + umi_sensoronly_v1 (force arm of 884599) + the 4 TODO configs |
| 250 | splits/ train/val/test episode lists (mixed_{mb,pusht,rope}_wrist_holdout.json) |
| 251 | vtm_cosmos/ model / data / train / infer package (Cosmos3 Generator adapter, causal mask, dataset, rollout) |
| 252 | eval/ run_test_case.py, summarize.py, cases/<task>_test_cases.json (+ reference metrics), |
| 253 | cases/{pusht,rope}_val_cases.json (1116 TODO) |
| 254 | scripts/ setup_env.sh, download.py, train.sh, infer_pusht.sh / infer_rope.sh (TODO: per-task test inference + upload, |
| 255 | via todo_eval_upload.py), infer_val.sh (1116 TODO: val at epochs 14/15), build_val_cases.py, |
| 256 | select_contact_windows.py |
| 257 | results/ (filled by the TODO runs) <run_name>/test_epoch_019/: per-case test outputs + summary.md; |
| 258 | <run_name>/val_epoch_01{4,5}/ (1116 TODO) |
| 259 | preprocessing/ raw mp4 + parquet -> latents + actions (see section 5) |
| 260 | third_party/wan22_vae/ standalone Wan2.2 VAE + tactile LoRA + encoder + pad_latent.py |
| 261 | data/ |
| 262 | mixed_{mb,pushT,rope}_latents/{tac,visual,wrist}/<episode>.pt Wan2.2 VAE latents, (N_windows,48,4,16,16) fp16 per stream |
| 263 | mixed_{mb,pushT,rope}_actions/<episode>/ |
| 264 | sensor_actions_with_dforce_fps15.pt per-hand 9-D sensor action, force as k_f·Δf·n (pusht_v1, rope_v1, TODO 1a/2a) |
| 265 | sensor_actions_fps15_noforce.pt same without the force term (884599, TODO 1b/2b) |
| 266 | view_actions_fps15.pt per-hand view action (read by every run; modelled only by pusht_v1 / rope_v1) |
| 267 | raw/videos/<episode>/{view_middle,tactile_left,tactile_right,wrist_left,wrist_right}.mp4 30 fps source |
| 268 | raw/parquet/<episode>.parquet, raw/episodes.json poses + normal force |
| 269 | checkpoints/ |
| 270 | vae/lora_last.pt tactile (GelSight) LoRA of the Wan2.2 VAE |
| 271 | nano_action_causal_umi_sensoronly_noforce_v1/epoch_013.pt (+ config.yaml, action_normalizer.json, train.log, metrics.jsonl) |
| 272 | nano_action_causal_pusht_v1/epoch_{014,015,019}.pt (+ same files) |
| 273 | nano_action_causal_rope_v1/epoch_{014,015,019}.pt (+ same files) |
| 274 | ``` |
| 275 | |
| 276 | Data = exactly the episodes of the three split files (motherboard 21/2/5, pushT 56/4/8, rope 53/8/9 train/val/test = 166 episodes; |
| 277 | latents + actions 117 GB, raw 16 GB). Training windows (stride 1, after dropping windows with an invalid action transition): |
| 278 | motherboard 71,386, pushT 55,199, rope 77,188. |
| 279 | |
| 280 | ## 3. Inference / evaluation |
| 281 | |
| 282 | Test protocol: per task, a fixed set of contact-stratified cases (`none` / `onset` / `release` / `full`, from the GelSight normal force, |
| 283 | `max(F_L, F_R) > 0.5 N`). Each case = one short prediction (8 observed → 8 predicted frames) plus one autoregressive rollout of 15 |
| 284 | windows (120 frames) from the same start, conditioned on the recorded actions; seed 0, UniPC 35 steps per latent block. |
| 285 | |
| 286 | ```bash |
| 287 | source env.sh |
| 288 | $PYTHON_BIN eval/run_test_case.py --task rope --case 0 # one case (~7 min on H100; ~40 s with --short-only) |
| 289 | $PYTHON_BIN eval/run_test_case.py --task rope --case all # 34 cases, sequential, resumable |
| 290 | $PYTHON_BIN eval/summarize.py --task rope --reference # table + paired Δ vs the shipped reference |
| 291 | $PYTHON_BIN eval/run_test_case.py --task umi_sensoronly_noforce --case all # job 884599's checkpoint, 20 motherboard cases |
| 292 | ``` |
| 293 | |
| 294 | | `--task` | cases | default checkpoint | |
| 295 | |---|---:|---| |
| 296 | | `pusht` | 30 (28 with a long rollout) | `nano_action_causal_pusht_v1/epoch_019.pt` | |
| 297 | | `rope` | 34 | `nano_action_causal_rope_v1/epoch_019.pt` | |
| 298 | | `umi_sensoronly_noforce` | 20 | `nano_action_causal_umi_sensoronly_noforce_v1/epoch_013.pt` | |
| 299 | |
| 300 | Outputs: `$ACTS_RUNS_ROOT/<ckpt run dir>/test_<ckpt stem>/<NN_episode_class>/` with `window_eval.json` (short metrics), |
| 301 | `reference_rollout.json` (per-window long metrics), `*_pred.pt`, GT/PRED strip PNGs, `short_gt_pred.mp4`, `long_gt_pred.mp4`. |
| 302 | Cases are independent, so on a cluster run them as an array job, e.g. |
| 303 | `sbatch --array=0-33 --gres=gpu:1 --wrap "source env.sh; \$PYTHON_BIN eval/run_test_case.py --task rope --case \$SLURM_ARRAY_TASK_ID"`. |
| 304 | |
| 305 | Lower-level CLI for arbitrary windows (the checkpoint stores the original absolute paths, hence the overrides): |
| 306 | |
| 307 | ```bash |
| 308 | $PYTHON_BIN -m vtm_cosmos.infer.rollout --ckpt $ACTS_CKPT_ROOT/nano_action_causal_rope_v1/epoch_019.pt \ |
| 309 | --split test --window-stride 1 --windows 0 500 --rollout-windows 15 --steps 35 --solver unipc --seed 0 \ |
| 310 | --out-dir runs/my_eval --override data.latent_root=$ACTS_DATA_ROOT/mixed_rope_latents \ |
| 311 | data.action_root=$ACTS_DATA_ROOT/mixed_rope_actions data.split_file=splits/mixed_rope_wrist_holdout.json \ |
| 312 | model.nano_checkpoint=$ACTS_CKPT_ROOT/Cosmos3-Nano |
| 313 | ``` |
| 314 | |
| 315 | ### Reference results (test split, PSNR dB) |
| 316 | |
| 317 | | run | n | horizon | video | wrist_left | wrist_right | tactile_left | tactile_right | |
| 318 | |---|---:|---|---:|---:|---:|---:|---:| |
| 319 | | rope_v1 e019 | 34 | short | 28.92 | 22.30 | 23.70 | 46.08 | 46.51 | |
| 320 | | rope_v1 e019 | 34 | long | 23.68 | 16.40 | 18.22 | 38.26 | 40.09 | |
| 321 | | pusht_v1 e019 | 30 | short | 29.68 | 22.86 | 22.76 | 46.43 | 48.37 | |
| 322 | | pusht_v1 e019 | 28 | long | 23.28 | 14.77 | 15.50 | 39.21 | 39.65 | |
| 323 | | umi_sensoronly_noforce e013 | 20 | short | 27.54 | 17.56 | 20.49 | 41.56 | 45.94 | |
| 324 | | umi_sensoronly_noforce e013 | 20 | long | 21.02 | 13.53 | 14.35 | 34.53 | 35.51 | |
| 325 | |
| 326 | SSIM / LPIPS and per-case numbers: `eval/cases/<task>_reference_metrics.json`. |
| 327 | |
| 328 | ## 4. Training |
| 329 | |
| 330 | ```bash |
| 331 | source env.sh |
| 332 | bash scripts/train.sh configs/nano_action_causal_rope_v1.yaml # all local GPUs (torchrun) |
| 333 | NGPU=1 bash scripts/train.sh configs/nano_action_causal_rope_sensoronly_v1.yaml \ |
| 334 | run_name=smoke training.max_steps=20 # quick smoke test (~31 GiB) |
| 335 | sbatch -N1 --ntasks-per-node=4 --gres=gpu:4 scripts/train.sh configs/<config>.yaml # Slurm |
| 336 | ``` |
| 337 | |
| 338 | Any `key=value` after the config is an OmegaConf dot-list override. The run directory gets `config.yaml`, `train.log`, `metrics.jsonl`, |
| 339 | `action_normalizer.json` (quantile normaliser fitted on the train split), `epoch_XXX.pt`, `last.pt`, `best.pt` (lowest val flow loss; |
| 340 | only a monitor: report the last or a matched epoch). Re-launching with the same `run_name` resumes from `last.pt`. |
| 341 | |
| 342 | ## 5. Preprocessing (raw → latents + actions) |
| 343 | |
| 344 | Only needed to rebuild or extend the data; `data/` already holds its output. |
| 345 | |
| 346 | ```bash |
| 347 | bash preprocessing/run_preprocess.sh rope # -> $ACTS_DATA_ROOT/rebuilt/mixed_rope_{latents,actions} |
| 348 | ``` |
| 349 | |
| 350 | (`run_preprocess.sh` task names: `umi_all` = motherboard, `rope`, `pusht`.) Latents: 30 → 15 fps, 256² crop, sliding 16-frame windows, |
| 351 | Wan2.2 VAE (tactile with the GelSight LoRA), anchor latent dropped. Sensor actions: `build_force_actions.py --force-term delta` |
| 352 | (→ `..._with_dforce_fps15.pt`). View actions: `build_view_actions.py --stride 2`. |
| 353 | |
| 354 | ## Notes and pitfalls |
| 355 | |
| 356 | * Always `source env.sh` first. It sets `PYTHONNOUSERSITE=1` and puts `third_party/cosmos-framework` + `third_party/wan22_vae` on `PYTHONPATH`. |
| 357 | * `setup_env.sh` pins torch 2.10.0+cu128 and removes `nvidia-*-cu13` libraries that some PyPI deps pull in, because they break cuDNN |
| 358 | convolutions (`CUDNN_STATUS_NOT_INITIALIZED`). If you install extra packages, re-run its last block. |
| 359 | * `imageio-ffmpeg` is pinned to 0.6.0 (ffmpeg 7.0.2). Bit-exact decoding of the anchor frame depends on the ffmpeg build. |
| 360 | * The first training / inference step compiles the FlexAttention Triton kernels, which takes a few minutes. |
| 361 | * Code relative to the 11-08 release `WitneyWW/acts-world-model`: `streams.py`, `model/layout.py` and `model/generator_adapter.py` |
| 362 | allow `action_full` runs without the view-action pair (sensor-only), and the eval scripts take any checkpoint. |
| 363 | |
| 364 | Checks done before upload: `REPRO_CHECKS.md`. |
| 365 | |