reports/review/step2000_27b.md
5.3 KB · 104 lines · markdown Raw
1 # Evaluation — reports/review/step2000_27b_ckpt
2
3 base: `/root/models/Qwen3.8-27B` · temperature: yes · max_seq_len 1024
4
5 ## Acceptance (DESIGN §8.3, main = test_set_30k)
6
7 **all rows**
8
9 | metric | target | value | B0 | pass |
10 |---|---|---|---|---|
11 | noul_auroc | >= 0.95 | 0.9934 | 0.8763 | ✅ |
12 | noul_brier_soft | <= 0.1 | 0.0021 | 0.0628 | ✅ |
13 | choice_top1 | >= 0.9 | 0.8723 | 0.5814 | ❌ |
14 | score_mae_expected | <= 0.35 | 0.1333 | 0.8420 | ✅ |
15 | kl | <= 0.15 | 0.0306 | 0.4296 | ✅ |
16 | ece | <= 0.03 | 0.0028 | 0.0501 | ✅ |
17 | ood KL ratio (ood/test_set_30k) | <= 2.0 | 3.880 | — | ❌ |
18
19 **excluding yuri_v1 exact-uniform placeholders (D1)**
20
21 | metric | target | value | B0 | pass |
22 |---|---|---|---|---|
23 | noul_auroc | >= 0.95 | 0.9934 | 0.8763 | ✅ |
24 | noul_brier_soft | <= 0.1 | 0.0026 | 0.0552 | ✅ |
25 | choice_top1 | >= 0.9 | 0.8723 | 0.5814 | ❌ |
26 | score_mae_expected | <= 0.35 | 0.1333 | 0.8420 | ✅ |
27 | kl | <= 0.15 | 0.0331 | 0.4411 | ✅ |
28 | ece | <= 0.03 | 0.0020 | 0.0307 | ✅ |
29 | ood KL ratio (ood/test_set_30k) | <= 2.0 | 3.590 | — | ❌ |
30
31 ## test_set_30k
32
33 all rows: `n=29955 · kl=0.0306 · js=0.0079 · top1=0.8954 · ece=0.0028 · mce=0.1858 · noul_auroc=0.9934 · noul_brier_soft=0.0021 · noul_brier_hard=0.0765 · score_mae_expected=0.1333 · score_rps=0.0140 · choice_top1=0.8723 · choice_kl=0.0592`
34
35 excluding yuri_v1 placeholders (2260 rows): `n=27695 · kl=0.0331 · js=0.0085 · top1=0.8954 · ece=0.0020 · mce=0.1858 · noul_auroc=0.9934 · noul_brier_soft=0.0026 · noul_brier_hard=0.0765 · score_mae_expected=0.1333 · score_rps=0.0140 · choice_top1=0.8723 · choice_kl=0.0592`
36
37 throughput: 9147 tok/s · 85.8 rows/s
38
39 ### test_set_30k by kind
40
41 | kind | n | kl | top1 | ece | noul_auroc | noul_brier_soft | score_mae_expected | choice_top1 |
42 |---|---|---|---|---|---|---|---|---|
43 | choice | 9199 | 0.059 | 0.872 | 0.005 | — | — | — | 0.872 |
44 | noul | 12229 | 0.006 | 0.957 | 0.004 | 0.993 | 0.002 | — | — |
45 | score | 8527 | 0.036 | 0.848 | 0.005 | — | — | 0.133 | — |
46
47 ### test_set_30k by source × kind
48
49 | source | kind | n | kl | top1 | ece | noul_auroc | noul_brier_soft | score_mae_expected | choice_top1 |
50 |---|---|---|---|---|---|---|---|---|---|
51 | openjev_v2 | choice | 887 | 0.213 | 0.865 | 0.023 | — | — | — | 0.865 |
52 | openjev_v2 | noul | 1432 | 0.007 | 0.997 | 0.002 | 1.000 | 0.002 | — | — |
53 | yuri_v1 | noul | 2260 | 0.000 | — | 0.013 | — | 0.000 | — | — |
54 | yuri_v3 | choice | 8312 | 0.043 | 0.873 | 0.005 | — | — | — | 0.873 |
55 | yuri_v3 | noul | 8537 | 0.007 | 0.951 | 0.002 | 0.992 | 0.003 | — | — |
56 | yuri_v3 | score | 8527 | 0.036 | 0.848 | 0.005 | — | — | 0.133 | — |
57
58 ### test_set_30k by family
59
60 | family | n | kl | top1 | ece | noul_auroc | noul_brier_soft | score_mae_expected | choice_top1 |
61 |---|---|---|---|---|---|---|---|---|
62 | agent | 2583 | 0.029 | 0.897 | 0.005 | 0.995 | 0.003 | 0.142 | 0.871 |
63 | biology | 1654 | 0.030 | 0.877 | 0.006 | 0.996 | 0.003 | 0.170 | 0.872 |
64 | business | 2352 | 0.026 | 0.921 | 0.004 | 0.997 | 0.002 | 0.099 | 0.881 |
65 | chemistry | 1333 | 0.023 | 0.934 | 0.008 | 0.997 | 0.002 | 0.118 | 0.927 |
66 | genomics | 1914 | 0.029 | 0.885 | 0.005 | 0.990 | 0.003 | 0.129 | 0.876 |
67 | knowledge | 4807 | 0.016 | 0.874 | 0.009 | 0.981 | 0.001 | 0.145 | 0.866 |
68 | medical | 2120 | 0.028 | 0.891 | 0.011 | 0.993 | 0.002 | 0.110 | 0.854 |
69 | openjev | 2319 | 0.086 | 0.948 | 0.009 | 1.000 | 0.002 | — | 0.865 |
70 | physics | 1656 | 0.027 | 0.866 | 0.004 | 0.990 | 0.002 | 0.125 | 0.863 |
71 | science | 1667 | 0.028 | 0.885 | 0.007 | 0.986 | 0.002 | 0.157 | 0.906 |
72 | spatial | 1647 | 0.028 | 0.883 | 0.007 | 0.987 | 0.002 | 0.142 | 0.848 |
73 | structured | 1668 | 0.028 | 0.877 | 0.007 | 0.984 | 0.004 | 0.165 | 0.841 |
74 | technical | 2581 | 0.031 | 0.900 | 0.005 | 0.994 | 0.002 | 0.105 | 0.864 |
75 | theology | 1654 | 0.028 | 0.885 | 0.005 | 0.992 | 0.003 | 0.149 | 0.903 |
76
77 ## ood
78
79 all rows: `n=13058 · kl=0.1188 · js=0.0307 · top1=0.9380 · ece=0.0094 · mce=0.2832 · noul_auroc=0.9962 · noul_brier_soft=0.0192 · noul_brier_hard=0.0192 · score_mae_expected=0.6131 · score_rps=0.4415 · choice_top1=0.8257 · choice_kl=0.2507`
80
81 excluding yuri_v1 placeholders (0 rows): `n=13058 · kl=0.1188 · js=0.0307 · top1=0.9380 · ece=0.0094 · mce=0.2832 · noul_auroc=0.9962 · noul_brier_soft=0.0192 · noul_brier_hard=0.0192 · score_mae_expected=0.6131 · score_rps=0.4415 · choice_top1=0.8257 · choice_kl=0.2507`
82
83 throughput: 9438 tok/s · 35.6 rows/s
84
85 ### ood by kind
86
87 | kind | n | kl | top1 | ece | noul_auroc | noul_brier_soft | score_mae_expected | choice_top1 |
88 |---|---|---|---|---|---|---|---|---|
89 | choice | 3219 | 0.251 | 0.826 | 0.036 | — | — | — | 0.826 |
90 | noul | 9767 | 0.069 | 0.978 | 0.013 | 0.996 | 0.019 | — | — |
91 | score | 72 | 1.009 | 0.569 | 0.131 | — | — | 0.613 | — |
92
93 ### ood by source × kind
94
95 | source | kind | n | kl | top1 | ece | noul_auroc | noul_brier_soft | score_mae_expected | choice_top1 |
96 |---|---|---|---|---|---|---|---|---|---|
97 | openjev_v2 | choice | 3219 | 0.251 | 0.826 | 0.036 | — | — | — | 0.826 |
98 | openjev_v2 | noul | 9767 | 0.069 | 0.978 | 0.013 | 0.996 | 0.019 | — | — |
99 | openjev_v2 | score | 72 | 1.009 | 0.569 | 0.131 | — | — | 0.613 | — |
100
101 ## Choice permutation consistency
102
103 `{"n_rows": 500, "mean_max_abs_diff": 0.03275444358587265, "p90_max_abs_diff": 0.0730958953499794, "top1_flip_rate": 0.0495}`
104