reports/b0_qwen38_27b.md
| 1 | # Evaluation — B0 /root/models/Qwen3.8-27B |
| 2 | |
| 3 | base: `/root/models/Qwen3.8-27B` · temperature: no · max_seq_len 1024 |
| 4 | |
| 5 | ## Acceptance (DESIGN §8.3, main = test_set_30k) |
| 6 | |
| 7 | **all rows** |
| 8 | |
| 9 | | metric | target | value | B0 | pass | |
| 10 | |---|---|---|---|---| |
| 11 | | noul_auroc | >= 0.95 | 0.8763 | nan | ❌ | |
| 12 | | noul_brier_soft | <= 0.1 | 0.0628 | nan | ✅ | |
| 13 | | choice_top1 | >= 0.9 | 0.5814 | nan | ❌ | |
| 14 | | score_mae_expected | <= 0.35 | 0.8420 | nan | ❌ | |
| 15 | | kl | <= 0.15 | 0.4296 | nan | ❌ | |
| 16 | | ece | <= 0.03 | 0.0501 | nan | ❌ | |
| 17 | | ood KL ratio (ood/test_set_30k) | <= 2.0 | 1.525 | — | ✅ | |
| 18 | |
| 19 | **excluding yuri_v1 exact-uniform placeholders (D1)** |
| 20 | |
| 21 | | metric | target | value | B0 | pass | |
| 22 | |---|---|---|---|---| |
| 23 | | noul_auroc | >= 0.95 | 0.8763 | nan | ❌ | |
| 24 | | noul_brier_soft | <= 0.1 | 0.0552 | nan | ✅ | |
| 25 | | choice_top1 | >= 0.9 | 0.5814 | nan | ❌ | |
| 26 | | score_mae_expected | <= 0.35 | 0.8420 | nan | ❌ | |
| 27 | | kl | <= 0.15 | 0.4411 | nan | ❌ | |
| 28 | | ece | <= 0.03 | 0.0307 | nan | ❌ | |
| 29 | | ood KL ratio (ood/test_set_30k) | <= 2.0 | 1.485 | — | ✅ | |
| 30 | |
| 31 | ## test_set_30k |
| 32 | |
| 33 | all rows: `n=29955 · kl=0.4296 · js=0.1105 · top1=0.5597 · ece=0.0501 · mce=0.2047 · noul_auroc=0.8763 · noul_brier_soft=0.0628 · noul_brier_hard=0.1580 · score_mae_expected=0.8420 · score_rps=0.3772 · choice_top1=0.5814 · choice_kl=0.4314` |
| 34 | |
| 35 | excluding yuri_v1 placeholders (2260 rows): `n=27695 · kl=0.4411 · js=0.1147 · top1=0.5597 · ece=0.0307 · mce=0.0532 · noul_auroc=0.8763 · noul_brier_soft=0.0552 · noul_brier_hard=0.1580 · score_mae_expected=0.8420 · score_rps=0.3772 · choice_top1=0.5814 · choice_kl=0.4314` |
| 36 | |
| 37 | throughput: 13918 tok/s · 130.5 rows/s |
| 38 | |
| 39 | ### test_set_30k by kind |
| 40 | |
| 41 | | kind | n | kl | top1 | ece | noul_auroc | noul_brier_soft | score_mae_expected | choice_top1 | |
| 42 | |---|---|---|---|---|---|---|---|---| |
| 43 | | choice | 9199 | 0.431 | 0.581 | 0.056 | — | — | — | 0.581 | |
| 44 | | noul | 12229 | 0.176 | 0.805 | 0.054 | 0.876 | 0.063 | — | — | |
| 45 | | score | 8527 | 0.791 | 0.253 | 0.051 | — | — | 0.842 | — | |
| 46 | |
| 47 | ### test_set_30k by source × kind |
| 48 | |
| 49 | | source | kind | n | kl | top1 | ece | noul_auroc | noul_brier_soft | score_mae_expected | choice_top1 | |
| 50 | |---|---|---|---|---|---|---|---|---|---| |
| 51 | | openjev_v2 | choice | 887 | 0.836 | 0.551 | 0.121 | — | — | — | 0.551 | |
| 52 | | openjev_v2 | noul | 1432 | 0.659 | 0.680 | 0.096 | 0.619 | 0.226 | — | — | |
| 53 | | yuri_v1 | noul | 2260 | 0.288 | — | 0.288 | — | 0.096 | — | — | |
| 54 | | yuri_v3 | choice | 8312 | 0.388 | 0.584 | 0.054 | — | — | — | 0.584 | |
| 55 | | yuri_v3 | noul | 8537 | 0.066 | 0.826 | 0.020 | 0.923 | 0.027 | — | — | |
| 56 | | yuri_v3 | score | 8527 | 0.791 | 0.253 | 0.051 | — | — | 0.842 | — | |
| 57 | |
| 58 | ### test_set_30k by family |
| 59 | |
| 60 | | family | n | kl | top1 | ece | noul_auroc | noul_brier_soft | score_mae_expected | choice_top1 | |
| 61 | |---|---|---|---|---|---|---|---|---| |
| 62 | | agent | 2583 | 0.393 | 0.543 | 0.035 | 0.921 | 0.026 | 0.937 | 0.570 | |
| 63 | | biology | 1654 | 0.384 | 0.562 | 0.048 | 0.967 | 0.023 | 0.780 | 0.624 | |
| 64 | | business | 2352 | 0.496 | 0.536 | 0.033 | 0.932 | 0.024 | 1.012 | 0.478 | |
| 65 | | chemistry | 1333 | 0.429 | 0.563 | 0.075 | 0.912 | 0.035 | 1.127 | 0.788 | |
| 66 | | genomics | 1914 | 0.420 | 0.559 | 0.057 | 0.913 | 0.027 | 0.746 | 0.635 | |
| 67 | | knowledge | 4807 | 0.342 | 0.538 | 0.157 | 0.897 | 0.076 | 0.674 | 0.514 | |
| 68 | | medical | 2120 | 0.442 | 0.586 | 0.024 | 0.954 | 0.030 | 0.917 | 0.566 | |
| 69 | | openjev | 2319 | 0.727 | 0.632 | 0.063 | 0.619 | 0.226 | — | 0.551 | |
| 70 | | physics | 1656 | 0.421 | 0.521 | 0.052 | 0.882 | 0.026 | 0.707 | 0.561 | |
| 71 | | science | 1667 | 0.338 | 0.596 | 0.019 | 0.909 | 0.023 | 0.657 | 0.632 | |
| 72 | | spatial | 1647 | 0.397 | 0.597 | 0.045 | 0.948 | 0.023 | 0.791 | 0.605 | |
| 73 | | structured | 1668 | 0.320 | 0.565 | 0.036 | 0.878 | 0.032 | 0.607 | 0.600 | |
| 74 | | technical | 2581 | 0.535 | 0.501 | 0.059 | 0.947 | 0.025 | 1.070 | 0.548 | |
| 75 | | theology | 1654 | 0.350 | 0.566 | 0.042 | 0.913 | 0.030 | 0.807 | 0.632 | |
| 76 | |
| 77 | ## ood |
| 78 | |
| 79 | all rows: `n=13058 · kl=0.6550 · js=0.1893 · top1=0.6902 · ece=0.0617 · mce=0.2023 · noul_auroc=0.6069 · noul_brier_soft=0.2071 · noul_brier_hard=0.2071 · score_mae_expected=0.6939 · score_rps=0.5406 · choice_top1=0.6356 · choice_kl=0.7858` |
| 80 | |
| 81 | excluding yuri_v1 placeholders (0 rows): `n=13058 · kl=0.6550 · js=0.1893 · top1=0.6902 · ece=0.0617 · mce=0.2023 · noul_auroc=0.6069 · noul_brier_soft=0.2071 · noul_brier_hard=0.2071 · score_mae_expected=0.6939 · score_rps=0.5406 · choice_top1=0.6356 · choice_kl=0.7858` |
| 82 | |
| 83 | throughput: 9876 tok/s · 37.3 rows/s |
| 84 | |
| 85 | ### ood by kind |
| 86 | |
| 87 | | kind | n | kl | top1 | ece | noul_auroc | noul_brier_soft | score_mae_expected | choice_top1 | |
| 88 | |---|---|---|---|---|---|---|---|---| |
| 89 | | choice | 3219 | 0.786 | 0.636 | 0.036 | — | — | — | 0.636 | |
| 90 | | noul | 9767 | 0.606 | 0.710 | 0.083 | 0.607 | 0.207 | — | — | |
| 91 | | score | 72 | 1.488 | 0.472 | 0.348 | — | — | 0.694 | — | |
| 92 | |
| 93 | ### ood by source × kind |
| 94 | |
| 95 | | source | kind | n | kl | top1 | ece | noul_auroc | noul_brier_soft | score_mae_expected | choice_top1 | |
| 96 | |---|---|---|---|---|---|---|---|---|---| |
| 97 | | openjev_v2 | choice | 3219 | 0.786 | 0.636 | 0.036 | — | — | — | 0.636 | |
| 98 | | openjev_v2 | noul | 9767 | 0.606 | 0.710 | 0.083 | 0.607 | 0.207 | — | — | |
| 99 | | openjev_v2 | score | 72 | 1.488 | 0.472 | 0.348 | — | — | 0.694 | — | |
| 100 | |
| 101 | ## Choice permutation consistency |
| 102 | |
| 103 | `{"n_rows": 500, "mean_max_abs_diff": 0.23575635254383087, "p90_max_abs_diff": 0.4639355540275574, "top1_flip_rate": 0.4095}` |
| 104 | |