reports/review/step1500.md
5.3 KB · 104 lines · markdown Raw
1 # Evaluation — reports/review/step1500_ckpt
2
3 base: `/root/models/Qwen3.5-9B` · temperature: yes · max_seq_len 1024
4
5 ## Acceptance (DESIGN §8.3, main = test_set_30k)
6
7 **all rows**
8
9 | metric | target | value | B0 | pass |
10 |---|---|---|---|---|
11 | noul_auroc | >= 0.95 | 0.9909 | 0.8236 | ✅ |
12 | noul_brier_soft | <= 0.1 | 0.0032 | 0.0960 | ✅ |
13 | choice_top1 | >= 0.9 | 0.8605 | 0.5323 | ❌ |
14 | score_mae_expected | <= 0.35 | 0.1506 | 1.1297 | ✅ |
15 | kl | <= 0.15 | 0.0404 | 0.5100 | ✅ |
16 | ece | <= 0.03 | 0.0025 | 0.0942 | ✅ |
17 | ood KL ratio (ood/test_set_30k) | <= 2.0 | 5.817 | — | ❌ |
18
19 **excluding yuri_v1 exact-uniform placeholders (D1)**
20
21 | metric | target | value | B0 | pass |
22 |---|---|---|---|---|
23 | noul_auroc | >= 0.95 | 0.9909 | nan | ✅ |
24 | noul_brier_soft | <= 0.1 | 0.0039 | nan | ✅ |
25 | choice_top1 | >= 0.9 | 0.8605 | nan | ❌ |
26 | score_mae_expected | <= 0.35 | 0.1506 | nan | ✅ |
27 | kl | <= 0.15 | 0.0437 | nan | ✅ |
28 | ece | <= 0.03 | 0.0021 | nan | ✅ |
29 | ood KL ratio (ood/test_set_30k) | <= 2.0 | 5.383 | — | ❌ |
30
31 ## test_set_30k
32
33 all rows: `n=29955 · kl=0.0404 · js=0.0104 · top1=0.8846 · ece=0.0025 · mce=0.0092 · noul_auroc=0.9909 · noul_brier_soft=0.0032 · noul_brier_hard=0.0807 · score_mae_expected=0.1506 · score_rps=0.0182 · choice_top1=0.8605 · choice_kl=0.0744`
34
35 excluding yuri_v1 placeholders (2260 rows): `n=27695 · kl=0.0437 · js=0.0112 · top1=0.8846 · ece=0.0021 · mce=0.0092 · noul_auroc=0.9909 · noul_brier_soft=0.0039 · noul_brier_hard=0.0807 · score_mae_expected=0.1506 · score_rps=0.0182 · choice_top1=0.8605 · choice_kl=0.0744`
36
37 throughput: 25883 tok/s · 242.7 rows/s
38
39 ### test_set_30k by kind
40
41 | kind | n | kl | top1 | ece | noul_auroc | noul_brier_soft | score_mae_expected | choice_top1 |
42 |---|---|---|---|---|---|---|---|---|
43 | choice | 9199 | 0.074 | 0.861 | 0.004 | — | — | — | 0.861 |
44 | noul | 12229 | 0.010 | 0.950 | 0.007 | 0.991 | 0.003 | — | — |
45 | score | 8527 | 0.048 | 0.835 | 0.009 | — | — | 0.151 | — |
46
47 ### test_set_30k by source × kind
48
49 | source | kind | n | kl | top1 | ece | noul_auroc | noul_brier_soft | score_mae_expected | choice_top1 |
50 |---|---|---|---|---|---|---|---|---|---|
51 | openjev_v2 | choice | 887 | 0.264 | 0.855 | 0.029 | — | — | — | 0.855 |
52 | openjev_v2 | noul | 1432 | 0.028 | 0.992 | 0.005 | 0.998 | 0.006 | — | — |
53 | yuri_v1 | noul | 2260 | 0.001 | — | 0.013 | — | 0.000 | — | — |
54 | yuri_v3 | choice | 8312 | 0.054 | 0.861 | 0.002 | — | — | — | 0.861 |
55 | yuri_v3 | noul | 8537 | 0.009 | 0.943 | 0.006 | 0.989 | 0.004 | — | — |
56 | yuri_v3 | score | 8527 | 0.048 | 0.835 | 0.009 | — | — | 0.151 | — |
57
58 ### test_set_30k by family
59
60 | family | n | kl | top1 | ece | noul_auroc | noul_brier_soft | score_mae_expected | choice_top1 |
61 |---|---|---|---|---|---|---|---|---|
62 | agent | 2583 | 0.035 | 0.885 | 0.005 | 0.991 | 0.003 | 0.156 | 0.861 |
63 | biology | 1654 | 0.038 | 0.881 | 0.008 | 0.996 | 0.003 | 0.178 | 0.870 |
64 | business | 2352 | 0.035 | 0.904 | 0.005 | 0.994 | 0.002 | 0.112 | 0.866 |
65 | chemistry | 1333 | 0.035 | 0.902 | 0.006 | 0.993 | 0.004 | 0.158 | 0.927 |
66 | genomics | 1914 | 0.037 | 0.880 | 0.007 | 0.990 | 0.004 | 0.146 | 0.853 |
67 | knowledge | 4807 | 0.023 | 0.861 | 0.008 | 0.978 | 0.002 | 0.165 | 0.841 |
68 | medical | 2120 | 0.035 | 0.880 | 0.004 | 0.992 | 0.003 | 0.120 | 0.831 |
69 | openjev | 2319 | 0.118 | 0.941 | 0.010 | 0.998 | 0.006 | — | 0.855 |
70 | physics | 1656 | 0.034 | 0.867 | 0.006 | 0.988 | 0.003 | 0.155 | 0.871 |
71 | science | 1667 | 0.040 | 0.874 | 0.006 | 0.989 | 0.003 | 0.171 | 0.884 |
72 | spatial | 1647 | 0.037 | 0.863 | 0.004 | 0.984 | 0.003 | 0.156 | 0.829 |
73 | structured | 1668 | 0.039 | 0.854 | 0.006 | 0.974 | 0.006 | 0.191 | 0.832 |
74 | technical | 2581 | 0.035 | 0.899 | 0.008 | 0.991 | 0.003 | 0.110 | 0.857 |
75 | theology | 1654 | 0.036 | 0.872 | 0.006 | 0.989 | 0.003 | 0.179 | 0.907 |
76
77 ## ood
78
79 all rows: `n=13058 · kl=0.2350 · js=0.0503 · top1=0.9148 · ece=0.0253 · mce=0.1149 · noul_auroc=0.9775 · noul_brier_soft=0.0443 · noul_brier_hard=0.0443 · score_mae_expected=0.7366 · score_rps=0.5194 · choice_top1=0.8388 · choice_kl=0.3785`
80
81 excluding yuri_v1 placeholders (0 rows): `n=13058 · kl=0.2350 · js=0.0503 · top1=0.9148 · ece=0.0253 · mce=0.1149 · noul_auroc=0.9775 · noul_brier_soft=0.0443 · noul_brier_hard=0.0443 · score_mae_expected=0.7366 · score_rps=0.5194 · choice_top1=0.8388 · choice_kl=0.3785`
82
83 throughput: 27269 tok/s · 102.9 rows/s
84
85 ### ood by kind
86
87 | kind | n | kl | top1 | ece | noul_auroc | noul_brier_soft | score_mae_expected | choice_top1 |
88 |---|---|---|---|---|---|---|---|---|
89 | choice | 3219 | 0.379 | 0.839 | 0.042 | — | — | — | 0.839 |
90 | noul | 9767 | 0.180 | 0.943 | 0.021 | 0.978 | 0.044 | — | — |
91 | score | 72 | 1.219 | 0.528 | 0.251 | — | — | 0.737 | — |
92
93 ### ood by source × kind
94
95 | source | kind | n | kl | top1 | ece | noul_auroc | noul_brier_soft | score_mae_expected | choice_top1 |
96 |---|---|---|---|---|---|---|---|---|---|
97 | openjev_v2 | choice | 3219 | 0.379 | 0.839 | 0.042 | — | — | — | 0.839 |
98 | openjev_v2 | noul | 9767 | 0.180 | 0.943 | 0.021 | 0.978 | 0.044 | — | — |
99 | openjev_v2 | score | 72 | 1.219 | 0.528 | 0.251 | — | — | 0.737 | — |
100
101 ## Choice permutation consistency
102
103 `{"n_rows": 500, "mean_max_abs_diff": 0.050133880227804184, "p90_max_abs_diff": 0.11231743544340134, "top1_flip_rate": 0.068}`
104