Quality
Calibration report: dex-1.0.0
Calibration cal-20260925-1. Measured agreement and calibration error, with the reliability bins behind the curves.
Model dex-1.0.0, calibration cal-20260925-1. Measured 2026-09-25 on Dex's production path in the EU. Evaluation set: Dex decision eval v1, test split: 300 items, 921 questions, English and Dutch; reference = average of gpt-5.3-chat and claude-opus-5 (they agree on 96.3%); temperatures fitted on the 100-item dev split.
Measurements
| Measurement | Value |
|---|---|
| Agreement with reference labels, English and Dutch (921 questions) | 82.2% |
| Agreement, English (465 questions) | 84.7% |
| Agreement, Dutch (456 questions) | 79.6% |
| Agreement, pick (400 questions) | 83.2% |
| Agreement, check (353 questions) | 90.1% |
| Agreement, rate (beta) (168 questions) | 63.1% |
| Calibration error (ECE), all questions (921 questions) | 0.037 |
| Calibration error (ECE), pick (400 questions) | 0.055 |
| Calibration error (ECE), check (353 questions) | 0.062 |
| Calibration error (ECE), rate (beta) (168 questions) | 0.121 |
For rate, agreement counts the most probable level. Counted on rating rounded to the nearest level, it is 60.1%. The rating is off by 0.582 levels on average.
Percentages are rounded to one decimal and ECE to three, and a value exactly halfway is rounded down. rate questions are in beta: their agreement and calibration error miss our bars. See Quality bars.
Reliability
Each row is one bin of answers of a question type, with about the same number of answers per bin: the probability Dex gave on average, how often that answer was right, and how many answers the bin holds. A calibrated version is right about as often as it says.
| Type | Bin | Probability given | Right this often | Answers |
|---|---|---|---|---|
| pick | 1 | 0.4525 | 0.3462 | 26 |
| pick | 2 | 0.6016 | 0.5926 | 27 |
| pick | 3 | 0.7148 | 0.6667 | 27 |
| pick | 4 | 0.7941 | 0.7692 | 26 |
| pick | 5 | 0.8411 | 0.7037 | 27 |
| pick | 6 | 0.8740 | 0.8519 | 27 |
| pick | 7 | 0.8910 | 0.8077 | 26 |
| pick | 8 | 0.9041 | 1.0000 | 27 |
| pick | 9 | 0.9148 | 0.8519 | 27 |
| pick | 10 | 0.9247 | 1.0000 | 26 |
| pick | 11 | 0.9319 | 1.0000 | 27 |
| pick | 12 | 0.9384 | 0.9630 | 27 |
| pick | 13 | 0.9434 | 0.9615 | 26 |
| pick | 14 | 0.9485 | 1.0000 | 27 |
| pick | 15 | 0.9598 | 0.9630 | 27 |
| check | 1 | 0.0353 | 0.0000 | 23 |
| check | 2 | 0.0528 | 0.0000 | 25 |
| check | 3 | 0.0775 | 0.0000 | 22 |
| check | 4 | 0.1177 | 0.0800 | 25 |
| check | 5 | 0.1691 | 0.1364 | 22 |
| check | 6 | 0.2648 | 0.3333 | 24 |
| check | 7 | 0.4115 | 0.3200 | 25 |
| check | 8 | 0.5984 | 0.7727 | 22 |
| check | 9 | 0.7289 | 0.7826 | 23 |
| check | 10 | 0.8277 | 0.8800 | 25 |
| check | 11 | 0.8926 | 0.9545 | 22 |
| check | 12 | 0.9269 | 1.0000 | 25 |
| check | 13 | 0.9446 | 1.0000 | 22 |
| check | 14 | 0.9579 | 1.0000 | 25 |
| check | 15 | 0.9708 | 1.0000 | 23 |
| rate | 1 | 0.3713 | 0.2727 | 11 |
| rate | 2 | 0.4458 | 0.2727 | 11 |
| rate | 3 | 0.4949 | 0.6364 | 11 |
| rate | 4 | 0.5468 | 0.4545 | 11 |
| rate | 5 | 0.5976 | 0.5000 | 12 |
| rate | 6 | 0.6808 | 0.5455 | 11 |
| rate | 7 | 0.7232 | 0.3636 | 11 |
| rate | 8 | 0.7495 | 0.7273 | 11 |
| rate | 9 | 0.7799 | 0.8182 | 11 |
| rate | 10 | 0.8209 | 0.7500 | 12 |
| rate | 11 | 0.8578 | 0.4545 | 11 |
| rate | 12 | 0.8765 | 0.9091 | 11 |
| rate | 13 | 0.9053 | 0.8182 | 11 |
| rate | 14 | 0.9251 | 0.9091 | 11 |
| rate | 15 | 0.9472 | 1.0000 | 12 |
All versions and the current measurements are on Calibration reports. How the numbers are made: Calibration.