reports/review/step0500.md
| 1 | # Evaluation — reports/review/step0500_ckpt |
| 2 | |
| 3 | base: `/root/models/Qwen3.5-9B` · temperature: yes · max_seq_len 1024 |
| 4 | |
| 5 | ## Acceptance (DESIGN §8.3, main = test_set_30k) |
| 6 | |
| 7 | **all rows** |
| 8 | |
| 9 | | metric | target | value | B0 | pass | |
| 10 | |---|---|---|---|---| |
| 11 | | noul_auroc | >= 0.95 | 0.9772 | 0.8236 | ✅ | |
| 12 | | noul_brier_soft | <= 0.1 | 0.0174 | 0.0960 | ✅ | |
| 13 | | choice_top1 | >= 0.9 | 0.8168 | 0.5323 | ❌ | |
| 14 | | score_mae_expected | <= 0.35 | 0.2244 | 1.1297 | ✅ | |
| 15 | | kl | <= 0.15 | 0.0809 | 0.5100 | ✅ | |
| 16 | | ece | <= 0.03 | 0.0215 | 0.0942 | ✅ | |
| 17 | | ood KL ratio (ood/test_set_30k) | <= 2.0 | 3.658 | — | ❌ | |
| 18 | |
| 19 | **excluding yuri_v1 exact-uniform placeholders (D1)** |
| 20 | |
| 21 | | metric | target | value | B0 | pass | |
| 22 | |---|---|---|---|---| |
| 23 | | noul_auroc | >= 0.95 | 0.9772 | nan | ✅ | |
| 24 | | noul_brier_soft | <= 0.1 | 0.0126 | nan | ✅ | |
| 25 | | choice_top1 | >= 0.9 | 0.8168 | nan | ❌ | |
| 26 | | score_mae_expected | <= 0.35 | 0.2244 | nan | ✅ | |
| 27 | | kl | <= 0.15 | 0.0806 | nan | ✅ | |
| 28 | | ece | <= 0.03 | 0.0074 | nan | ✅ | |
| 29 | | ood KL ratio (ood/test_set_30k) | <= 2.0 | 3.670 | — | ❌ | |
| 30 | |
| 31 | ## test_set_30k |
| 32 | |
| 33 | all rows: `n=29955 · kl=0.0809 · js=0.0207 · top1=0.8240 · ece=0.0215 · mce=0.0762 · noul_auroc=0.9772 · noul_brier_soft=0.0174 · noul_brier_hard=0.0927 · score_mae_expected=0.2244 · score_rps=0.0369 · choice_top1=0.8168 · choice_kl=0.1185` |
| 34 | |
| 35 | excluding yuri_v1 placeholders (2260 rows): `n=27695 · kl=0.0806 · js=0.0208 · top1=0.8240 · ece=0.0074 · mce=0.0526 · noul_auroc=0.9772 · noul_brier_soft=0.0126 · noul_brier_hard=0.0927 · score_mae_expected=0.2244 · score_rps=0.0369 · choice_top1=0.8168 · choice_kl=0.1185` |
| 36 | |
| 37 | throughput: 25871 tok/s · 242.6 rows/s |
| 38 | |
| 39 | ### test_set_30k by kind |
| 40 | |
| 41 | | kind | n | kl | top1 | ece | noul_auroc | noul_brier_soft | score_mae_expected | choice_top1 | |
| 42 | |---|---|---|---|---|---|---|---|---| |
| 43 | | choice | 9199 | 0.118 | 0.817 | 0.004 | — | — | — | 0.817 | |
| 44 | | noul | 12229 | 0.046 | 0.903 | 0.042 | 0.977 | 0.017 | — | — | |
| 45 | | score | 8527 | 0.090 | 0.741 | 0.016 | — | — | 0.224 | — | |
| 46 | |
| 47 | ### test_set_30k by source × kind |
| 48 | |
| 49 | | source | kind | n | kl | top1 | ece | noul_auroc | noul_brier_soft | score_mae_expected | choice_top1 | |
| 50 | |---|---|---|---|---|---|---|---|---|---| |
| 51 | | openjev_v2 | choice | 887 | 0.405 | 0.758 | 0.044 | — | — | — | 0.758 | |
| 52 | | openjev_v2 | noul | 1432 | 0.128 | 0.964 | 0.024 | 0.986 | 0.034 | — | — | |
| 53 | | yuri_v1 | noul | 2260 | 0.084 | — | 0.193 | — | 0.038 | — | — | |
| 54 | | yuri_v3 | choice | 8312 | 0.088 | 0.823 | 0.004 | — | — | — | 0.823 | |
| 55 | | yuri_v3 | noul | 8537 | 0.022 | 0.892 | 0.008 | 0.978 | 0.009 | — | — | |
| 56 | | yuri_v3 | score | 8527 | 0.090 | 0.741 | 0.016 | — | — | 0.224 | — | |
| 57 | |
| 58 | ### test_set_30k by family |
| 59 | |
| 60 | | family | n | kl | top1 | ece | noul_auroc | noul_brier_soft | score_mae_expected | choice_top1 | |
| 61 | |---|---|---|---|---|---|---|---|---| |
| 62 | | agent | 2583 | 0.064 | 0.826 | 0.008 | 0.981 | 0.009 | 0.244 | 0.827 | |
| 63 | | biology | 1654 | 0.071 | 0.809 | 0.016 | 0.990 | 0.011 | 0.277 | 0.822 | |
| 64 | | business | 2352 | 0.062 | 0.855 | 0.009 | 0.987 | 0.007 | 0.168 | 0.854 | |
| 65 | | chemistry | 1333 | 0.056 | 0.873 | 0.017 | 0.990 | 0.006 | 0.223 | 0.915 | |
| 66 | | genomics | 1914 | 0.076 | 0.804 | 0.007 | 0.978 | 0.008 | 0.257 | 0.796 | |
| 67 | | knowledge | 4807 | 0.077 | 0.811 | 0.102 | 0.953 | 0.031 | 0.213 | 0.803 | |
| 68 | | medical | 2120 | 0.069 | 0.823 | 0.016 | 0.984 | 0.008 | 0.179 | 0.797 | |
| 69 | | openjev | 2319 | 0.234 | 0.886 | 0.020 | 0.986 | 0.034 | — | 0.758 | |
| 70 | | physics | 1656 | 0.062 | 0.802 | 0.009 | 0.976 | 0.009 | 0.214 | 0.841 | |
| 71 | | science | 1667 | 0.060 | 0.805 | 0.023 | 0.974 | 0.009 | 0.215 | 0.844 | |
| 72 | | spatial | 1647 | 0.069 | 0.822 | 0.012 | 0.972 | 0.007 | 0.245 | 0.782 | |
| 73 | | structured | 1668 | 0.068 | 0.785 | 0.015 | 0.961 | 0.013 | 0.278 | 0.770 | |
| 74 | | technical | 2581 | 0.069 | 0.818 | 0.009 | 0.985 | 0.009 | 0.186 | 0.808 | |
| 75 | | theology | 1654 | 0.065 | 0.804 | 0.010 | 0.983 | 0.009 | 0.265 | 0.874 | |
| 76 | |
| 77 | ## ood |
| 78 | |
| 79 | all rows: `n=13058 · kl=0.2958 · js=0.0782 · top1=0.8789 · ece=0.0053 · mce=0.2488 · noul_auroc=0.9489 · noul_brier_soft=0.0647 · noul_brier_hard=0.0647 · score_mae_expected=0.6340 · score_rps=0.4477 · choice_top1=0.7645 · choice_kl=0.4942` |
| 80 | |
| 81 | excluding yuri_v1 placeholders (0 rows): `n=13058 · kl=0.2958 · js=0.0782 · top1=0.8789 · ece=0.0053 · mce=0.2488 · noul_auroc=0.9489 · noul_brier_soft=0.0647 · noul_brier_hard=0.0647 · score_mae_expected=0.6340 · score_rps=0.4477 · choice_top1=0.7645 · choice_kl=0.4942` |
| 82 | |
| 83 | throughput: 27224 tok/s · 102.7 rows/s |
| 84 | |
| 85 | ### ood by kind |
| 86 | |
| 87 | | kind | n | kl | top1 | ece | noul_auroc | noul_brier_soft | score_mae_expected | choice_top1 | |
| 88 | |---|---|---|---|---|---|---|---|---| |
| 89 | | choice | 3219 | 0.494 | 0.765 | 0.047 | — | — | — | 0.765 | |
| 90 | | noul | 9767 | 0.224 | 0.919 | 0.013 | 0.949 | 0.065 | — | — | |
| 91 | | score | 72 | 1.151 | 0.611 | 0.179 | — | — | 0.634 | — | |
| 92 | |
| 93 | ### ood by source × kind |
| 94 | |
| 95 | | source | kind | n | kl | top1 | ece | noul_auroc | noul_brier_soft | score_mae_expected | choice_top1 | |
| 96 | |---|---|---|---|---|---|---|---|---|---| |
| 97 | | openjev_v2 | choice | 3219 | 0.494 | 0.765 | 0.047 | — | — | — | 0.765 | |
| 98 | | openjev_v2 | noul | 9767 | 0.224 | 0.919 | 0.013 | 0.949 | 0.065 | — | — | |
| 99 | | openjev_v2 | score | 72 | 1.151 | 0.611 | 0.179 | — | — | 0.634 | — | |
| 100 | |
| 101 | ## Choice permutation consistency |
| 102 | |
| 103 | `{"n_rows": 500, "mean_max_abs_diff": 0.06840353459119797, "p90_max_abs_diff": 0.14390206336975098, "top1_flip_rate": 0.1115}` |
| 104 | |