Calibration

Calibration is what makes a probability mean what it says. When Dex says 80%, it should be right about 80% of the time. Every model version is calibrated before it ships, and every response names the calibration version behind its numbers.

What calibration means#

Take every answer where Dex gave its top choice a probability near 0.8. If the probabilities are calibrated, about 80% of those choices are correct. The same should hold at 0.6, at 0.95, and everywhere else.

A model can rank options well and still be overconfident or underconfident. Calibration fixes the scale, so you can set thresholds on probabilities and reason about how often an answer above them is wrong.

How a version is calibrated#

Dex uses temperature scaling, with one temperature per question type: T_pick, T_rate and T_check for each exact model version.

• A temperature is one number that sharpens or softens a distribution. Raw scores are divided by it before they become probabilities (see Confidence).
• Temperature scaling never changes which answer is most likely. It changes only how sure the probabilities are.
• Each temperature is fitted by minimizing negative log-likelihood on the dev split of our data, searching over temperatures from 0.25 to 8.
• Fallback versions get their own three temperatures, fitted on the fallback model's own scores.

The set of temperatures for a version has an id of the form cal-YYYYMMDD-n. Every response names it in calibration, and GET /v1/models lists it per version.

Data and reference labels#

dex-1.0.1 was measured on our own decision set, Dex decision eval v1.

• The items. 400 items, 200 in English and 200 in Dutch, with 1,224 questions across pick, rate and check. They cover seven kinds of work: support routing, moderation, lead intent, product categorisation, document triage, sentiment and stance, and agent guardrails. 50 items are hard cases: negation, literal instructions, numbers and dates, long noisy text, and prompt injection.
• Where they come from. Every item was written for Dex: by AI models working from our briefs, with fictional people, companies and contact details. No third-party text or datasets were used.
• Reference labels. Two frontier models from two different vendors label every question with a probability per answer. The reference is the average of the two, and its most likely answer is the reference label.
• No customer data. Customer content is never used to train, tune or calibrate any model.
• Splits. 100 items form the dev split, used to fit the temperatures. The other 300 form the test split, which is frozen and used only for the reported numbers. The test split is never used for fitting or for choosing a model.

Known limits of this set. Say so before you rely on the numbers:

• The items are synthetic, written by AI models of one family, and cleaner than real traffic.
• The Dutch items have not yet been checked by a native speaker, and no person has settled the questions where the two reference models disagree.
• The set is small, so the numbers for smaller groups, such as the rate questions, carry several points of uncertainty.
• Most pick and check questions are clear-cut, so their calibration is measured mostly at high probabilities.

Metrics#

All metrics are reported on the test split: overall, per language and per question type.

Expected calibration error (ECE). Sort the answers by their predicted probability and split them into 15 bins with the same number of answers in each. In each bin, compare the average predicted probability with how often the answers were actually right. ECE is the average of those gaps across the bins. 0 is perfect.

Agreement. How often Dex matches the reference label:

For rate we also report agreement of rating rounded to the nearest level.

Quality bars#

Our bars: agreement of at least 68% overall and for each question type, at least 64% in each language, and ECE of at most 0.05 for each question type.

dex-1.0.1 meets the agreement bars overall (82.3%) and in both languages (English 84.9%, Dutch 79.6%). Its ECE over all questions is 0.044.

It misses two bars, and we publish the numbers as they are:

• ECE per type is above 0.05 for all three types: pick 0.048, check 0.052, rate 0.138.
• rate agreement is 63.7%, below 68%. That is why rate is labelled beta. See Questions.

What this means in practice: treat a probability as a good guide, not an exact rate, and for rate set your thresholds from your own data.

Publication#

Each calibration version gets a report on the calibration page. For dex-1.0.1 it shows agreement overall, per language and per question type, ECE overall and per question type, the number of questions behind each, and a reliability curve per question type. ECE and curves per language, bootstrap bands, Brier scores and the fitted temperatures are not published yet.

Because every response names its calibration, you can trace any answer to the numbers behind it.

Measured results#

Model dex-1.0.1, calibration cal-20260926-1. Measured 2026-09-26 on Dex's production path in the EU. Evaluation set: Dex decision eval v1, test split: 300 items, 921 questions, English and Dutch; reference = average of gpt-5.3-chat and claude-opus-5 (they agree on 96.3%); temperatures fitted on the 100-item dev split.

For rate, agreement counts the most probable level. Counted on rating rounded to the nearest level, it is 58.3%. The rating is off by 0.591 levels on average.

Percentages are rounded to one decimal and ECE to three, and a value exactly halfway is rounded down.