Confidence

Every answer has a confidence between 0 and 1. It says how concentrated the answer's probability distribution is: 1 when all probability sits on one answer, lower as it spreads out. You use it to decide when to trust an answer, usually through abstention.

From logits to probabilities#

For each question, the model produces one raw score (a logit) per possible answer: one per option for a pick, one per level for a rate, and a yes and a no score for a check. Dex divides these scores by a temperature T for the question type, then turns them into probabilities. The temperatures come from the model version's calibration, so there is one T each for pick, rate and check.

The formulas below use z for the logits and p for the probabilities.

pick#

The probabilities are a softmax over the options. The choice is the most probable option. Confidence is the gap between the two largest probabilities.

p_i        = softmax(z_i / T_pick)      over the n options
choice     = argmax p                    ties go to the earlier option
confidence = p_(1) - p_(2)              largest minus second largest
confidence = 1                           when there is only one option

In the Dutch support example in the contract, garantie has 0.6423 and reparatie has 0.3287, so confidence is 0.6423 minus 0.3287, which is 0.3136. Two options are close, and confidence is low.

rate#

The probabilities are a softmax over the levels. The rating is their probability-weighted index. Confidence measures how tightly the probability clusters around the rating, relative to the widest possible spread.

p_j        = softmax(z_j / T_rate)      over the k levels
rating     = sum_j j * p_j
sigma      = sqrt(sum_j p_j * (j - rating)^2)
confidence = 1 - sigma / ((k - 1) / 2)

Confidence is 1 when all probability sits on one level. It is 0 when the probability splits evenly between the lowest and the highest level, the widest spread possible.

check#

The probability of yes is a sigmoid of the gap between the yes and no scores. Confidence is the distance from 50/50, scaled to 0 to 1.

probability = sigmoid((z_yes - z_no) / T_check)
confidence  = |2 * probability - 1|

This is the same gap as a pick with two options. A probability of 0.5 has confidence 0, and a probability of 0.75 has confidence 0.5.

It helps to translate a threshold back into probabilities. A check reaches confidence 0.6 only when the probability is at least 0.8 or at most 0.2.

What confidence is not#

• Not a correctness guarantee. A confident answer can be wrong. Confidence only describes the shape of the distribution. Calibration is what makes the probabilities mean what they say.
• Not one scale across types. Each type has its own formula, so 0.5 on a rate does not mean the same as 0.5 on a pick. Choose min_confidence per question, not one number for all.
• Not a probability. For a check, probability is the probability of yes, and confidence is derived from it. Use probability when you need the chance that something holds.

Rounding#

Dex rounds numbers the same way every time, so the same inputs always give the same bytes.

• Probabilities are rounded to 4 decimals with the largest-remainder method, so each distribution sums to exactly 1.0000. Equal remainders go to the earlier option or level.
• rating, confidence and a check's probability are computed from the unrounded probabilities, then rounded half to even to 4 decimals.
• choice and abstained are decided on the unrounded values.
• Numbers are written in plain decimal notation, never with an exponent.

Because confidence comes from unrounded values, recomputing it from the rounded numbers can differ in the last digit. In the Dutch support example, the check has probability 0.8732 and confidence 0.7465, while two times 0.8732 minus 1 gives 0.7464.