RESULTS_sensoronly_force_1117.md
6.6 KB · 61 lines · markdown Raw
1 # Sensor-only action ablation: without vs with force at epoch 10 and epoch 15 (1-indexed; files epoch_009 / epoch_014)
2
3 Generated 2026-11-17 from WitneyWW/acts-sensoronly-ablation open PRs #1–#13 (collaborator jhhj25) + local vtm_cosmos runs.
4 Metrics = mean over cases; short = 8 observed → 8 predicted frames; long = mean over 15 chained windows (120 frames).
5 wrist = mean of wrist_left/right, tactile = mean of tactile_left/right. Rows marked — have no result yet (not evaluated / not uploaded).
6
7 ## Short horizon
8
9 | dataset | split | variant | epoch | video PSNR | wrist PSNR | tactile PSNR | video SSIM | wrist SSIM | tactile SSIM | video LPIPS | wrist LPIPS | tactile LPIPS |
10 |---|---|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
11 | motherboard (umi) | test 20 cases | sensor-only, no force | ep 10 (epoch_009) | 27.60 | 18.99 | 43.50 | 0.877 | 0.604 | 0.983 | 0.048 | 0.216 | 0.032 |
12 | pushT | val 14 cases | sensor-only, no force | ep 10 (epoch_009) | 29.44 | 23.12 | 42.56 | 0.929 | 0.838 | 0.980 | 0.027 | 0.094 | 0.049 |
13 | pushT | test 30 cases | sensor-only, no force | ep 10 (epoch_009) | 29.18 | 22.28 | 46.60 | 0.920 | 0.828 | 0.988 | 0.041 | 0.115 | 0.022 |
14 | rope | val 30 cases | sensor-only, no force | ep 10 (epoch_009) | 28.51 | 22.26 | 45.33 | 0.907 | 0.775 | 0.986 | 0.048 | 0.148 | 0.030 |
15 | rope | test 34 cases | sensor-only, no force | ep 10 (epoch_009) | 28.77 | 22.89 | 46.16 | 0.906 | 0.783 | 0.989 | 0.046 | 0.133 | 0.021 |
16 | motherboard (umi) | test 20 cases | sensor-only + force | ep 10 (epoch_009) | 27.64 | 19.02 | 43.47 | 0.879 | 0.605 | 0.983 | 0.048 | 0.215 | 0.032 |
17 | pushT | val 14 cases | sensor-only + force | ep 10 (epoch_009) | 29.79 | 23.14 | 42.96 | 0.932 | 0.837 | 0.981 | 0.026 | 0.095 | 0.048 |
18 | pushT | test 30 cases | sensor-only + force | ep 10 (epoch_009) | 29.20 | 22.18 | 46.98 | 0.920 | 0.826 | 0.988 | 0.041 | 0.118 | 0.020 |
19 | rope | val 30 cases | sensor-only + force | ep 10 (epoch_009) | 28.71 | 22.35 | 45.35 | 0.909 | 0.776 | 0.986 | 0.045 | 0.147 | 0.030 |
20 | rope | test 34 cases | sensor-only + force | ep 10 (epoch_009) | 28.74 | 22.67 | 46.11 | 0.905 | 0.777 | 0.989 | 0.048 | 0.137 | 0.021 |
21 | motherboard (umi) | test 20 cases | sensor-only + force | ep 15 (epoch_014) | 27.62 | 19.04 | 43.43 | 0.878 | 0.605 | 0.983 | 0.049 | 0.214 | 0.034 |
22 | pushT | val 14 cases | sensor-only + force | ep 15 (epoch_014) | 29.88 | 23.77 | 43.76 | 0.933 | 0.843 | 0.982 | 0.027 | 0.086 | 0.037 |
23 | pushT | test 30 cases | sensor-only + force | ep 15 (epoch_014) | — | — | — | — | — | — | — | — | — |
24 | rope | val 30 cases | sensor-only + force | ep 15 (epoch_014) | 28.57 | 22.34 | 45.65 | 0.907 | 0.777 | 0.986 | 0.047 | 0.147 | 0.030 |
25 | rope | test 34 cases | sensor-only + force | ep 15 (epoch_014) | 28.88 | 22.84 | 46.31 | 0.905 | 0.781 | 0.989 | 0.048 | 0.132 | 0.019 |
26
27 ## Long horizon
28
29 | dataset | split | variant | epoch | video PSNR | wrist PSNR | tactile PSNR | video SSIM | wrist SSIM | tactile SSIM | video LPIPS | wrist LPIPS | tactile LPIPS |
30 |---|---|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
31 | motherboard (umi) | test 20 cases | sensor-only, no force | ep 10 (epoch_009) | 20.84 | 13.75 | 36.15 | 0.706 | 0.417 | 0.954 | 0.203 | 0.452 | 0.181 |
32 | pushT | val 14 cases | sensor-only, no force | ep 10 (epoch_009) | 22.69 | 15.74 | 35.63 | 0.795 | 0.660 | 0.957 | 0.150 | 0.320 | 0.153 |
33 | pushT | test 30 cases | sensor-only, no force | ep 10 (epoch_009) | 22.82 | 14.63 | 38.08 | 0.795 | 0.604 | 0.969 | 0.169 | 0.364 | 0.115 |
34 | rope | val 30 cases | sensor-only, no force | ep 10 (epoch_009) | 23.17 | 17.42 | 36.33 | 0.774 | 0.644 | 0.965 | 0.171 | 0.314 | 0.125 |
35 | rope | test 34 cases | sensor-only, no force | ep 10 (epoch_009) | 23.38 | 17.09 | 36.83 | 0.783 | 0.637 | 0.969 | 0.172 | 0.303 | 0.115 |
36 | motherboard (umi) | test 20 cases | sensor-only + force | ep 10 (epoch_009) | 21.13 | 13.84 | 35.99 | 0.713 | 0.419 | 0.955 | 0.191 | 0.452 | 0.179 |
37 | pushT | val 14 cases | sensor-only + force | ep 10 (epoch_009) | 22.55 | 15.80 | 35.55 | 0.793 | 0.658 | 0.959 | 0.155 | 0.318 | 0.146 |
38 | pushT | test 30 cases | sensor-only + force | ep 10 (epoch_009) | 22.62 | 14.22 | 37.51 | 0.791 | 0.591 | 0.971 | 0.172 | 0.380 | 0.107 |
39 | rope | val 30 cases | sensor-only + force | ep 10 (epoch_009) | 23.16 | 17.41 | 36.91 | 0.774 | 0.645 | 0.966 | 0.173 | 0.313 | 0.126 |
40 | rope | test 34 cases | sensor-only + force | ep 10 (epoch_009) | 23.32 | 17.11 | 37.27 | 0.782 | 0.636 | 0.969 | 0.174 | 0.299 | 0.118 |
41 | motherboard (umi) | test 20 cases | sensor-only + force | ep 15 (epoch_014) | 21.13 | 14.05 | 36.06 | 0.712 | 0.413 | 0.953 | 0.190 | 0.439 | 0.185 |
42 | pushT | val 14 cases | sensor-only + force | ep 15 (epoch_014) | 22.78 | 15.78 | 36.03 | 0.798 | 0.656 | 0.962 | 0.148 | 0.314 | 0.129 |
43 | pushT | test 30 cases | sensor-only + force | ep 15 (epoch_014) | — | — | — | — | — | — | — | — | — |
44 | rope | val 30 cases | sensor-only + force | ep 15 (epoch_014) | 23.02 | 17.56 | 38.03 | 0.779 | 0.645 | 0.967 | 0.168 | 0.303 | 0.121 |
45 | rope | test 34 cases | sensor-only + force | ep 15 (epoch_014) | 23.07 | 17.37 | 38.48 | 0.782 | 0.642 | 0.970 | 0.173 | 0.286 | 0.112 |
46
47 ## Epoch coverage of what has been uploaded / evaluated
48
49 | run | evaluated epochs available | source |
50 |---|---|---|
51 | nano_action_causal_pusht_sensoronly_noforce_v1 | test_epoch_009 (n=30, pr1), val_epoch_009 (n=14, pr5) | |
52 | nano_action_causal_pusht_sensoronly_v1 | test_epoch_009 (n=30, local-pr3eval), test_epoch_019 (n=30, pr3), val_epoch_009 (n=14, local-pr3eval), val_epoch_014 (n=14, pr14), val_epoch_015 (n=14, pr15), val_epoch_019 (n=14, pr6) | |
53 | nano_action_causal_pusht_v1 | val_epoch_014 (n=14, pr11), val_epoch_015 (n=14, pr12) | |
54 | nano_action_causal_rope_sensoronly_noforce_v1 | test_epoch_009 (n=34, pr2), val_epoch_009 (n=30, pr8) | |
55 | nano_action_causal_rope_sensoronly_v1 | test_epoch_009 (n=34, local-pr3eval), test_epoch_013 (n=34, pr9), test_epoch_014 (n=34, pr4), val_epoch_009 (n=30, local-pr3eval), val_epoch_013 (n=30, pr10), val_epoch_014 (n=30, pr7) | |
56 | nano_action_causal_rope_v1 | val_epoch_015 (n=30, pr13) | |
57 | nano_action_causal_umi_all_noforce_v1 | contact_epoch_009_test (n=20, local), contact_epoch_010_test (n=20, local) | |
58 | nano_action_causal_umi_all_v1 | contact_epoch_009_test (n=20, local), contact_epoch_010_test (n=20, local), contact_epoch_019_test (n=20, local) | |
59 | nano_action_causal_umi_sensoronly_noforce_v1 | contact_epoch_009_test (n=20, local), contact_epoch_013_test (n=20, local) | |
60 | nano_action_causal_umi_sensoronly_v1 | contact_epoch_009_test (n=20, local), contact_epoch_013_test (n=20, local), contact_epoch_014_test (n=20, local), contact_epoch_017_test (n=20, local) | |
61