README.md
24.7 KB · 365 lines · markdown Raw
1 ---
2 license: other
3 tags:
4 - world-model
5 - robotics
6 - tactile
7 - cosmos
8 ---
9
10 # ACTS world model: sensor-only / no-force ablation (code + data + checkpoints)
11
12 This repo contains everything needed to run three action-conditioned visuo-tactile world-model runs on a new machine, and to train
13 the four follow-up experiments listed under **TODO**. **1116 TODO** (below): val-set inference of pushT / rope at epochs 14 and 15. It holds the code, the preprocessed data, the raw media and the trained
14 checkpoints. Only the two public base models are downloaded from their own repos.
15
16 | run | task | what it is | ckpt in this repo | status |
17 |---|---|---|---|---|
18 | `nano_action_causal_umi_sensoronly_noforce_v1` | motherboard | **Slurm job 884599** (`rebuttal_2`): sensor-only (no view-action streams) + no force | `checkpoints/…/epoch_013.pt` | training was still running at upload (epoch 14/20) |
19 | `nano_action_causal_pusht_v1` | pushT | full model (sensor + view actions, with force) | `checkpoints/…/epoch_019.pt` (+ `epoch_014/015.pt` for the 1116 val TODO) | done |
20 | `nano_action_causal_rope_v1` | rope | full model (sensor + view actions, with force) | `checkpoints/…/epoch_019.pt` (+ `epoch_014/015.pt` for the 1116 val TODO) | done |
21
22 ## TODO experiments
23
24 The same ablation that was run on motherboard (sensor-only, with and without force), now on **pushT** and **rope**.
25 **pushT and rope are two separate tasks**: each is trained only on its own data, and each run is evaluated with **its own
26 inference script on its own test set**. Nothing is shared between the two tasks except the code.
27
28 | | **pushT** | **rope** |
29 |---|---|---|
30 | experiments | 1a `nano_action_causal_pusht_sensoronly_v1` (with force)<br>1b `nano_action_causal_pusht_sensoronly_noforce_v1` (no force) | 2a `nano_action_causal_rope_sensoronly_v1` (with force)<br>2b `nano_action_causal_rope_sensoronly_noforce_v1` (no force) |
31 | configs | `configs/nano_action_causal_pusht_sensoronly{,_noforce}_v1.yaml` | `configs/nano_action_causal_rope_sensoronly{,_noforce}_v1.yaml` |
32 | training data | `data/mixed_pushT_{latents,actions}`, split `splits/mixed_pusht_wrist_holdout.json`: 56 train / 4 val episodes, 55,199 train windows | `data/mixed_rope_{latents,actions}`, split `splits/mixed_rope_wrist_holdout.json`: 53 train / 8 val episodes, 77,188 train windows |
33 | training time (4× H100, 20 epochs) | ≈ 2.3 h/epoch (6,900 steps) → ~2 days per experiment | ≈ 3.3 h/epoch (9,648 steps) → ~3 days per experiment |
34 | **inference script** | **`scripts/infer_pusht.sh`** | **`scripts/infer_rope.sh`** |
35 | **test set** | test split of the pushT split file: 8 episodes → `eval/cases/pusht_test_cases.json`, **30 cases** (28 with a long rollout; cases 5 and 29 are short-only, the segment is too short) | test split of the rope split file: 9 episodes → `eval/cases/rope_test_cases.json`, **34 cases** (all with a long rollout) |
36 | compared against | `nano_action_causal_pusht_v1` epoch 019 (`eval/cases/pusht_reference_metrics.json`) | `nano_action_causal_rope_v1` epoch 019 (`eval/cases/rope_reference_metrics.json`) |
37
38 Each config differs from its parent (`nano_action_causal_pusht_v1` / `nano_action_causal_rope_v1`) only in `run_name`,
39 `model.active_streams` (no `view_action_*`), and for the no-force variants `data.action_file: sensor_actions_fps15_noforce.pt`.
40
41 ### Requirements (all 4 experiments)
42
43 * **Each experiment is its own training job on 4× H100 80 GB (one node)**: one DDP rank per GPU, batch 2 per GPU = global batch 8,
44 ~31 GiB per GPU. Do not change the GPU count or `training.batch_size`; the parent runs and job 884599 used exactly 4×2.
45 * **Train for the full 20 epochs** (`training.epochs: 20` → `epoch_000.pt` … `epoch_019.pt`). Do not stop early and do not use
46 `best.pt`. The training is finished only when `runs/<run_name>/epoch_019.pt` exists.
47 * If a Slurm job hits its time limit or is preempted, submit **the same command again**. It resumes from `runs/<run_name>/last.pt`;
48 do not change the run name.
49 * **Only after epoch 20**: run the inference script **of that task** on **that task's test set** (pushT run → `infer_pusht.sh`,
50 rope run → `infer_rope.sh`). It uploads the checkpoint and the test results to this HF repo, so it needs `HF_TOKEN` with
51 **write** access to `WitneyWW/acts-sensoronly-ablation`.
52
53 ### Step 1: setup and data (once per machine)
54
55 ```bash
56 bash scripts/setup_env.sh && source env.sh && export HF_TOKEN=hf_... # see section 1
57 $PYTHON_BIN scripts/download.py --what data raw ckpts nano wan_vae --tasks pusht # pushT: train + val + test data
58 $PYTHON_BIN scripts/download.py --what data raw ckpts nano wan_vae --tasks rope # rope: train + val + test data
59 ```
60
61 ### pushT (experiments 1a, 1b)
62
63 ```bash
64 source env.sh
65 # train: two separate jobs, 4x H100 each, 20 epochs
66 sbatch -N1 --ntasks-per-node=4 --gres=gpu:4 scripts/train.sh configs/nano_action_causal_pusht_sensoronly_v1.yaml # 1a
67 sbatch -N1 --ntasks-per-node=4 --gres=gpu:4 scripts/train.sh configs/nano_action_causal_pusht_sensoronly_noforce_v1.yaml # 1b
68 # (without Slurm, on a 4x H100 machine: NGPU=4 bash scripts/train.sh configs/<config>.yaml)
69
70 # after runs/<run_name>/epoch_019.pt exists: pushT inference script on the pushT test set (30 cases) + upload, 4 GPUs, ~1 h
71 sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_pusht.sh configs/nano_action_causal_pusht_sensoronly_v1.yaml # 1a
72 sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_pusht.sh configs/nano_action_causal_pusht_sensoronly_noforce_v1.yaml # 1b
73 ```
74
75 ### rope (experiments 2a, 2b)
76
77 ```bash
78 source env.sh
79 # train: two separate jobs, 4x H100 each, 20 epochs
80 sbatch -N1 --ntasks-per-node=4 --gres=gpu:4 scripts/train.sh configs/nano_action_causal_rope_sensoronly_v1.yaml # 2a
81 sbatch -N1 --ntasks-per-node=4 --gres=gpu:4 scripts/train.sh configs/nano_action_causal_rope_sensoronly_noforce_v1.yaml # 2b
82
83 # after runs/<run_name>/epoch_019.pt exists: rope inference script on the rope test set (34 cases) + upload, 4 GPUs, ~1 h
84 sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_rope.sh configs/nano_action_causal_rope_sensoronly_v1.yaml # 2a
85 sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_rope.sh configs/nano_action_causal_rope_sensoronly_noforce_v1.yaml # 2b
86 ```
87
88 Sanity checks at the start of training (`runs/<run_name>/train.log`): `[data] train episodes=56 windows=55199` (pushT) or
89 `train episodes=53 windows=77188` (rope), and `trainable=45,301,952`. Progress: `ls runs/<run_name>/epoch_*.pt`.
90
91 ### What the inference scripts do
92
93 `scripts/infer_pusht.sh` / `scripts/infer_rope.sh` (both call `scripts/todo_eval_upload.py --task <task>`):
94 1. refuse a config of the other task, and refuse to run unless `runs/<run_name>/epoch_019.pt` exists (20 epochs finished);
95 2. check that the test cases come from the test split of the same split file the model was trained with;
96 3. run `eval/run_test_case.py` with `epoch_019.pt` on **all test cases of that task** (each = short 8→8 prediction + 15-window long
97 rollout, UniPC 35 steps, seed 0), spread over the 4 GPUs. If interrupted, run the same command again: finished cases are skipped;
98 4. check every case finished and write `summary.md` (PSNR/SSIM/LPIPS + paired Δ vs `pusht_v1` / `rope_v1` epoch 019);
99 5. upload to this repo:
100 * `checkpoints/<run_name>/`: `epoch_019.pt`, `config.yaml`, `action_normalizer.json`, `train.log`, `metrics.jsonl`, …
101 * `results/<run_name>/test_epoch_019/`: every test case (metrics json, predicted latents, GT|PRED PNG strips, `short_gt_pred.mp4`,
102 `long_gt_pred.mp4`, ~5 GB per experiment) + `summary.md`.
103
104 ### Checklist
105
106 | # | task | trained 20 epochs (`epoch_019.pt`) | inference on own test set | uploaded `checkpoints/` + `results/` |
107 |---|---|---|---|---|
108 | 1a sensor-only | pushT | ☐ | ☐ `infer_pusht.sh` (30 cases) | ☐ |
109 | 1b sensor-only no-force | pushT | ☐ | ☐ `infer_pusht.sh` (30 cases) | ☐ |
110 | 2a sensor-only | rope | ☐ | ☐ `infer_rope.sh` (34 cases) | ☐ |
111 | 2b sensor-only no-force | rope | ☐ | ☐ `infer_rope.sh` (34 cases) | ☐ |
112
113 ### What the configs change
114
115 * **sensor-only** = the two per-hand view-action streams are not modelled (not tokenised; `view_action2llm9` frozen). The dataset
116 still reads `view_actions_fps15.pt`, so the window list and the action normaliser are identical to the parent run's. Trainable
117 params 45.30 M (vs 45.34 M); 1,310 tokens/sample (vs 1,342).
118 * **with force** = the sensor action is `sensor_actions_with_dforce_fps15.pt`: `dp = R_kᵀΔp + k_f·Δf·n`, so the change of normal
119 force enters as a displacement along the gel normal.
120 * **no force** = `sensor_actions_fps15_noforce.pt`: the same file with the `k_f·Δf·n` term removed (pure geometry); it differs from
121 the force file only in `dp_y`.
122 * Everything else = parent recipe: lr 5e-5 (100 warm-up steps), wd 0.01, bf16, LoRA r16/α32 on the GEN tower, Cosmos3-Nano init, seed 0.
123
124 ## 1116 TODO: validation set (pushT / rope) at epochs 14 and 15
125
126 Added 2026-11-16. Run the **val** split of pushT and rope on **epoch 14 and epoch 15** (`epoch_014.pt`, `epoch_015.pt`) for
127 the parent runs and for the four TODO runs, so the sensor-only / no-force ablation can be compared on held-out val episodes
128 at matched epochs. This does not replace the epoch-19 test inference above; it is an extra check.
129
130 ### The validation sets
131
132 Each task's val split is the `val` list of its own split file (the same file the run trains with; val ≠ test, whole source
133 episodes held out). The latents, actions and raw media of these episodes are in this repo under the same `data/` trees as
134 train/test, so nothing extra has to be prepared.
135
136 | task | split file | val episodes (segments) | val windows (stride 1, valid actions) | val cases | file |
137 |---|---|---|---:|---:|---|
138 | pushT | `splits/mixed_pusht_wrist_holdout.json` | 0910_ep002, 0911_ep003 (seg00, seg01), 0912_ep000 → 4 segments | 4,439 | **14** (all with a 15-window long rollout) | `eval/cases/pusht_val_cases.json` |
139 | rope | `splits/mixed_rope_wrist_holdout.json` | 0911_ep001 (2 segs), 0914_ep003 (4 segs), 0915_ep002 (2 segs) → 8 segments | 6,174 | **30** (all with a long rollout) | `eval/cases/rope_val_cases.json` |
140
141 Val cases use the test-case protocol: one case per (val segment × contact class `none`/`onset`/`release`/`full`, from
142 `max(F_L, F_R) > 0.5 N` on the initial 16-frame window) where such a window exists, the window drawn with seed 0 among those
143 with room for the 15-window rollout. Built by `scripts/build_val_cases.py`, which also checks that the parent config and both
144 TODO configs of the task give the **identical** val window index, so the same case file serves all three runs. (pushT val
145 has no `none` window in 0911_ep003; rope 0911_ep001_seg01 and 0914_ep003_seg02 have no `none` window.)
146
147 **Download** (only the val episodes of one task, + checkpoints incl. the parent `epoch_014.pt` / `epoch_015.pt`):
148
149 ```bash
150 $PYTHON_BIN scripts/download.py --what data raw ckpts nano wan_vae --tasks pusht --splits val
151 $PYTHON_BIN scripts/download.py --what data raw ckpts nano wan_vae --tasks rope --splits val
152 ```
153
154 **Load in Python** (the dataset must be built exactly like this, otherwise the case `index` points at a different window):
155
156 ```python
157 import json
158 from omegaconf import OmegaConf
159 from vtm_cosmos.data.dataset import VtmWindowDataset
160 from vtm_cosmos.data.splits import load_split
161
162 cfg = OmegaConf.load("configs/nano_action_causal_rope_sensoronly_v1.yaml") # any pushT/rope config; needs `source env.sh`
163 episodes = load_split(cfg.data.split_file)["val"] # ['rope_0911_episode_001_seg00.pt', ...]
164 ds = VtmWindowDataset(cfg.data.latent_root, cfg.data.action_root, episodes,
165 action_file=cfg.data.action_file, window_stride=1,
166 include_view_actions=True, view_action_file=cfg.data.view_action_file,
167 require_valid_actions=True, # action_full drops invalid transitions
168 active_streams=tuple(cfg.model.active_streams) if cfg.model.get("active_streams") else None)
169 print(len(ds)) # rope 6174, pushT 4439
170 item = ds[0] # latents.{video,wrist_left,wrist_right,tactile_left,tactile_right}: (48,4,16,16) = 16 frames @ 15 fps
171 # actions.action_{left,right}: (15,9); actions.view_action_{left,right}: (16,9); action_valid.*; stream_present
172 cases = json.load(open("eval/cases/rope_val_cases.json"))["cases"]
173 c = cases[0]; assert ds.window_meta(c["index"]) == (c["episode"], c["t_start"])
174 ```
175
176 Raw media of a val episode: `data/raw/videos/<episode>/{view_middle,tactile_left,tactile_right,wrist_left,wrist_right}.mp4`,
177 `data/raw/parquet/<episode>.parquet`. During training the val split is only used for the per-epoch flow loss
178 (`data.val_window_stride: 50`).
179
180 ### What to run (4× H100 per command, resumable, uploads to this repo)
181
182 `scripts/infer_val.sh <config>` = for epoch 14 and 15: all val cases of that task on that epoch's checkpoint
183 (`todo_eval_upload.py --split val --epoch E`) → `runs/<run_name>/val_epoch_01{4,5}/summary.md` (paired Δ vs the parent run
184 at the same epoch, if the parent has already been run) → upload `results/<run_name>/val_epoch_01{4,5}/` and, for the TODO runs,
185 `checkpoints/<run_name>/epoch_01{4,5}.pt`. It can start as soon as `runs/<run_name>/epoch_014.pt` exists (training can keep
186 running on another node; epoch 15 needs `epoch_015.pt`). **Run the parent first** so the TODO summaries get the paired column.
187
188 ```bash
189 source env.sh && export HF_TOKEN=hf_... # write access
190 # pushT (14 val cases; ~30 min per epoch on 4 GPUs)
191 sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_val.sh configs/nano_action_causal_pusht_v1.yaml # parent, ckpts shipped
192 sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_val.sh configs/nano_action_causal_pusht_sensoronly_v1.yaml # 1a
193 sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_val.sh configs/nano_action_causal_pusht_sensoronly_noforce_v1.yaml # 1b
194 # rope (30 val cases; ~1 h per epoch on 4 GPUs)
195 sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_val.sh configs/nano_action_causal_rope_v1.yaml # parent, ckpts shipped
196 sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_val.sh configs/nano_action_causal_rope_sensoronly_v1.yaml # 2a
197 sbatch -N1 --ntasks=1 --gres=gpu:4 scripts/infer_val.sh configs/nano_action_causal_rope_sensoronly_noforce_v1.yaml # 2b
198 # one epoch only / no upload: EPOCHS=14 bash scripts/infer_val.sh <config> --no-upload
199 # re-summarise later (e.g. after the parent finished):
200 $PYTHON_BIN eval/summarize.py --task rope --split val --root runs/nano_action_causal_rope_sensoronly_v1/val_epoch_014 \
201 --ref-root runs/nano_action_causal_rope_v1/val_epoch_014
202 ```
203
204 Single case by hand: `$PYTHON_BIN eval/run_test_case.py --task pusht --split val --case 3 --ckpt <epoch_014.pt>`
205 (output `runs/<run dir>/val_epoch_014/03_.../`).
206
207 ### 1116 checklist
208
209 | run | task | epoch 14 val | epoch 15 val | uploaded `results/<run>/val_epoch_01{4,5}/` |
210 |---|---|---|---|---|
211 | `nano_action_causal_pusht_v1` (parent) | pushT | ☐ 14 cases | ☐ 14 cases | ☐ |
212 | 1a `pusht_sensoronly_v1` | pushT | ☐ | ☐ | ☐ (+ ckpts) |
213 | 1b `pusht_sensoronly_noforce_v1` | pushT | ☐ | ☐ | ☐ (+ ckpts) |
214 | `nano_action_causal_rope_v1` (parent) | rope | ☐ 30 cases | ☐ 30 cases | ☐ |
215 | 2a `rope_sensoronly_v1` | rope | ☐ | ☐ | ☐ (+ ckpts) |
216 | 2b `rope_sensoronly_noforce_v1` | rope | ☐ | ☐ | ☐ (+ ckpts) |
217
218 ## 1. Setup on a new machine
219
220 Tested on Linux, NVIDIA H100 80 GB, a driver that supports CUDA 12.8, and conda.
221
222 ```bash
223 export HF_TOKEN=hf_... # this repo is private
224 # code only (skip the ~150 GB of data/ and checkpoints/ for now)
225 hf download WitneyWW/acts-sensoronly-ablation --exclude "data/*" "checkpoints/*" --local-dir acts-sensoronly-ablation
226 cd acts-sensoronly-ablation
227 bash scripts/setup_env.sh # conda env `acts_wm` (override ENV_NAME=..., CONDA_ROOT=...): torch 2.10.0+cu128,
228 # pinned deps, and NVIDIA/cosmos-framework @ 3f94d25 cloned into third_party/
229 source env.sh # paths + PYTHONPATH; PYTHON_BIN defaults to $CONDA_ROOT/envs/acts_wm/bin/python
230 $PYTHON_BIN scripts/download.py --what all # everything (~175 GB incl. Cosmos3-Nano 29 GB)
231 # or only what you need, e.g. evaluating pushT on its test split:
232 $PYTHON_BIN scripts/download.py --what data raw ckpts nano wan_vae --tasks pusht --splits test
233 ```
234
235 `download.py` places files in `./data` and `./checkpoints`, which are the `env.sh` defaults of `ACTS_DATA_ROOT` / `ACTS_CKPT_ROOT`.
236 `--tasks` ∈ `mb` (motherboard), `pusht`, `rope`. Training needs the train and val splits (the default fetches all three).
237 If the env is not at `~/miniconda3/envs/acts_wm`, export `PYTHON_BIN=/path/to/env/bin/python` before `source env.sh`.
238
239 | variable (env.sh) | default | content |
240 |---|---|---|
241 | `ACTS_DATA_ROOT` | `./data` | `mixed_*_latents`, `mixed_*_actions`, `raw/` |
242 | `ACTS_CKPT_ROOT` | `./checkpoints` | `Cosmos3-Nano/`, `Wan2.2_VAE.pth`, `vae/lora_last.pt`, `nano_action_causal_*/epoch_XXX.pt` |
243 | `ACTS_RUNS_ROOT` | `./runs` | new training runs and eval outputs |
244 | `VTM_SOURCE_VIDEO_ROOT` | `$ACTS_DATA_ROOT/raw/videos` | mp4s used to rebuild the VAE anchor frame at decode time |
245
246 ## 2. What is in this repo
247
248 ```
249 configs/ exact configs of the 3 runs above + umi_sensoronly_v1 (force arm of 884599) + the 4 TODO configs
250 splits/ train/val/test episode lists (mixed_{mb,pusht,rope}_wrist_holdout.json)
251 vtm_cosmos/ model / data / train / infer package (Cosmos3 Generator adapter, causal mask, dataset, rollout)
252 eval/ run_test_case.py, summarize.py, cases/<task>_test_cases.json (+ reference metrics),
253 cases/{pusht,rope}_val_cases.json (1116 TODO)
254 scripts/ setup_env.sh, download.py, train.sh, infer_pusht.sh / infer_rope.sh (TODO: per-task test inference + upload,
255 via todo_eval_upload.py), infer_val.sh (1116 TODO: val at epochs 14/15), build_val_cases.py,
256 select_contact_windows.py
257 results/ (filled by the TODO runs) <run_name>/test_epoch_019/: per-case test outputs + summary.md;
258 <run_name>/val_epoch_01{4,5}/ (1116 TODO)
259 preprocessing/ raw mp4 + parquet -> latents + actions (see section 5)
260 third_party/wan22_vae/ standalone Wan2.2 VAE + tactile LoRA + encoder + pad_latent.py
261 data/
262 mixed_{mb,pushT,rope}_latents/{tac,visual,wrist}/<episode>.pt Wan2.2 VAE latents, (N_windows,48,4,16,16) fp16 per stream
263 mixed_{mb,pushT,rope}_actions/<episode>/
264 sensor_actions_with_dforce_fps15.pt per-hand 9-D sensor action, force as k_f·Δf·n (pusht_v1, rope_v1, TODO 1a/2a)
265 sensor_actions_fps15_noforce.pt same without the force term (884599, TODO 1b/2b)
266 view_actions_fps15.pt per-hand view action (read by every run; modelled only by pusht_v1 / rope_v1)
267 raw/videos/<episode>/{view_middle,tactile_left,tactile_right,wrist_left,wrist_right}.mp4 30 fps source
268 raw/parquet/<episode>.parquet, raw/episodes.json poses + normal force
269 checkpoints/
270 vae/lora_last.pt tactile (GelSight) LoRA of the Wan2.2 VAE
271 nano_action_causal_umi_sensoronly_noforce_v1/epoch_013.pt (+ config.yaml, action_normalizer.json, train.log, metrics.jsonl)
272 nano_action_causal_pusht_v1/epoch_{014,015,019}.pt (+ same files)
273 nano_action_causal_rope_v1/epoch_{014,015,019}.pt (+ same files)
274 ```
275
276 Data = exactly the episodes of the three split files (motherboard 21/2/5, pushT 56/4/8, rope 53/8/9 train/val/test = 166 episodes;
277 latents + actions 117 GB, raw 16 GB). Training windows (stride 1, after dropping windows with an invalid action transition):
278 motherboard 71,386, pushT 55,199, rope 77,188.
279
280 ## 3. Inference / evaluation
281
282 Test protocol: per task, a fixed set of contact-stratified cases (`none` / `onset` / `release` / `full`, from the GelSight normal force,
283 `max(F_L, F_R) > 0.5 N`). Each case = one short prediction (8 observed → 8 predicted frames) plus one autoregressive rollout of 15
284 windows (120 frames) from the same start, conditioned on the recorded actions; seed 0, UniPC 35 steps per latent block.
285
286 ```bash
287 source env.sh
288 $PYTHON_BIN eval/run_test_case.py --task rope --case 0 # one case (~7 min on H100; ~40 s with --short-only)
289 $PYTHON_BIN eval/run_test_case.py --task rope --case all # 34 cases, sequential, resumable
290 $PYTHON_BIN eval/summarize.py --task rope --reference # table + paired Δ vs the shipped reference
291 $PYTHON_BIN eval/run_test_case.py --task umi_sensoronly_noforce --case all # job 884599's checkpoint, 20 motherboard cases
292 ```
293
294 | `--task` | cases | default checkpoint |
295 |---|---:|---|
296 | `pusht` | 30 (28 with a long rollout) | `nano_action_causal_pusht_v1/epoch_019.pt` |
297 | `rope` | 34 | `nano_action_causal_rope_v1/epoch_019.pt` |
298 | `umi_sensoronly_noforce` | 20 | `nano_action_causal_umi_sensoronly_noforce_v1/epoch_013.pt` |
299
300 Outputs: `$ACTS_RUNS_ROOT/<ckpt run dir>/test_<ckpt stem>/<NN_episode_class>/` with `window_eval.json` (short metrics),
301 `reference_rollout.json` (per-window long metrics), `*_pred.pt`, GT/PRED strip PNGs, `short_gt_pred.mp4`, `long_gt_pred.mp4`.
302 Cases are independent, so on a cluster run them as an array job, e.g.
303 `sbatch --array=0-33 --gres=gpu:1 --wrap "source env.sh; \$PYTHON_BIN eval/run_test_case.py --task rope --case \$SLURM_ARRAY_TASK_ID"`.
304
305 Lower-level CLI for arbitrary windows (the checkpoint stores the original absolute paths, hence the overrides):
306
307 ```bash
308 $PYTHON_BIN -m vtm_cosmos.infer.rollout --ckpt $ACTS_CKPT_ROOT/nano_action_causal_rope_v1/epoch_019.pt \
309 --split test --window-stride 1 --windows 0 500 --rollout-windows 15 --steps 35 --solver unipc --seed 0 \
310 --out-dir runs/my_eval --override data.latent_root=$ACTS_DATA_ROOT/mixed_rope_latents \
311 data.action_root=$ACTS_DATA_ROOT/mixed_rope_actions data.split_file=splits/mixed_rope_wrist_holdout.json \
312 model.nano_checkpoint=$ACTS_CKPT_ROOT/Cosmos3-Nano
313 ```
314
315 ### Reference results (test split, PSNR dB)
316
317 | run | n | horizon | video | wrist_left | wrist_right | tactile_left | tactile_right |
318 |---|---:|---|---:|---:|---:|---:|---:|
319 | rope_v1 e019 | 34 | short | 28.92 | 22.30 | 23.70 | 46.08 | 46.51 |
320 | rope_v1 e019 | 34 | long | 23.68 | 16.40 | 18.22 | 38.26 | 40.09 |
321 | pusht_v1 e019 | 30 | short | 29.68 | 22.86 | 22.76 | 46.43 | 48.37 |
322 | pusht_v1 e019 | 28 | long | 23.28 | 14.77 | 15.50 | 39.21 | 39.65 |
323 | umi_sensoronly_noforce e013 | 20 | short | 27.54 | 17.56 | 20.49 | 41.56 | 45.94 |
324 | umi_sensoronly_noforce e013 | 20 | long | 21.02 | 13.53 | 14.35 | 34.53 | 35.51 |
325
326 SSIM / LPIPS and per-case numbers: `eval/cases/<task>_reference_metrics.json`.
327
328 ## 4. Training
329
330 ```bash
331 source env.sh
332 bash scripts/train.sh configs/nano_action_causal_rope_v1.yaml # all local GPUs (torchrun)
333 NGPU=1 bash scripts/train.sh configs/nano_action_causal_rope_sensoronly_v1.yaml \
334 run_name=smoke training.max_steps=20 # quick smoke test (~31 GiB)
335 sbatch -N1 --ntasks-per-node=4 --gres=gpu:4 scripts/train.sh configs/<config>.yaml # Slurm
336 ```
337
338 Any `key=value` after the config is an OmegaConf dot-list override. The run directory gets `config.yaml`, `train.log`, `metrics.jsonl`,
339 `action_normalizer.json` (quantile normaliser fitted on the train split), `epoch_XXX.pt`, `last.pt`, `best.pt` (lowest val flow loss;
340 only a monitor: report the last or a matched epoch). Re-launching with the same `run_name` resumes from `last.pt`.
341
342 ## 5. Preprocessing (raw → latents + actions)
343
344 Only needed to rebuild or extend the data; `data/` already holds its output.
345
346 ```bash
347 bash preprocessing/run_preprocess.sh rope # -> $ACTS_DATA_ROOT/rebuilt/mixed_rope_{latents,actions}
348 ```
349
350 (`run_preprocess.sh` task names: `umi_all` = motherboard, `rope`, `pusht`.) Latents: 30 → 15 fps, 256² crop, sliding 16-frame windows,
351 Wan2.2 VAE (tactile with the GelSight LoRA), anchor latent dropped. Sensor actions: `build_force_actions.py --force-term delta`
352 (→ `..._with_dforce_fps15.pt`). View actions: `build_view_actions.py --stride 2`.
353
354 ## Notes and pitfalls
355
356 * Always `source env.sh` first. It sets `PYTHONNOUSERSITE=1` and puts `third_party/cosmos-framework` + `third_party/wan22_vae` on `PYTHONPATH`.
357 * `setup_env.sh` pins torch 2.10.0+cu128 and removes `nvidia-*-cu13` libraries that some PyPI deps pull in, because they break cuDNN
358 convolutions (`CUDNN_STATUS_NOT_INITIALIZED`). If you install extra packages, re-run its last block.
359 * `imageio-ffmpeg` is pinned to 0.6.0 (ffmpeg 7.0.2). Bit-exact decoding of the anchor frame depends on the ffmpeg build.
360 * The first training / inference step compiles the FlexAttention Triton kernels, which takes a few minutes.
361 * Code relative to the 11-08 release `WitneyWW/acts-world-model`: `streams.py`, `model/layout.py` and `model/generator_adapter.py`
362 allow `action_full` runs without the view-action pair (sensor-only), and the eval scripts take any checkpoint.
363
364 Checks done before upload: `REPRO_CHECKS.md`.
365