Calibration report: dex-1.0.1

Model dex-1.0.1, calibration cal-20260926-1. The alias dex-1 answers with this version. Measured 2026-09-26 on Dex's production path in the EU. Evaluation set: Dex decision eval v1, test split: 300 items, 921 questions, English and Dutch; reference = average of gpt-5.3-chat and claude-opus-5 (they agree on 96.3%); temperatures fitted on the 100-item dev split.

Measurements#

For rate, agreement counts the most probable level. Counted on rating rounded to the nearest level, it is 58.3%. The rating is off by 0.591 levels on average.

Latency: 14,868 successful requests to https://api.thinqit.ai on 27 Sep 2026, sent by our own client with 4 to 12 requests in flight to one Dex inference node that also served other test traffic, timed from sending the request to the end of the answer. A dedicated run on the same GPU path on 26 Sep 2026 at 8 requests per second (1,448 requests) measured p50 127 ms and p95 259 ms.

Determinism: on 27 Sep 2026 each of 400 evaluation items was sent 3 times through the live API, and all 400 gave byte-identical answers every time. Each of the 4 recorded race workflows gave one answer set over 500 repeats.

Percentages are rounded to one decimal and ECE to three, and a value exactly halfway is rounded down. rate questions are in beta: their agreement and calibration error miss our bars. See Quality bars.

Reliability#

Each row is one bin of answers of a question type, with about the same number of answers per bin: the probability Dex gave on average, how often that answer was right, and how many answers the bin holds. A calibrated version is right about as often as it says.

All versions and the current measurements are on Calibration reports. How the numbers are made: Calibration.