Pattern: pin versions in CI

On the GPU path, identical request bytes sent to the same exact version return byte-identical answers, every time. That turns your Dex requests into ordinary test fixtures: record the answers once, pin the version, and let CI fail when anything changes. See Determinism.

The rules#

• Pin the exact version in every test request: "model": "dex-1.0.1", exactly as GET /v1/models lists it. An exact version never goes to the hosted fallback, so a test can never be answered by a path that is not deterministic.
• Send the same bytes. Keep each test request as a file and send the file's text as it is. Two requests that differ only in whitespace or key order are different requests.
• Compare answers, model, calibration and the token counts of usage. id and created differ between calls, and on a live key the split between allowance_tokens and paid_tokens depends on the week's allowance.
• Raw HTTP, on purpose. The tests send the file's bytes with plain HTTP, so the SDK's retries do not apply: rerun the job after a 429 or 503, or wrap the call in the retry loop of Retries, timeouts and fallbacks.
• Use a test key. Test calls are free up to 250,000 tokens a day per account and always run on the GPU path. Store the key as a CI secret.
• Watch the lifecycle. A replaced version stays available for at least 90 days and GET /v1/models shows its retires_at. Move the pin on purpose: record new answers on the new version and review the difference like any other change.

Record once#

Put each test request in tests/dex/<name>.request.json, with "model" pinned. The request of the support ticket recipe with a pin is a good first one. Then record the answers:

# record.py: python record.py  (writes tests/dex/<name>.expected.json next to each request)
import json, os, pathlib
import httpx

for req in sorted(pathlib.Path("tests/dex").glob("*.request.json")):
    body = req.read_bytes()  # the exact bytes, never re-serialised
    r = httpx.post(
        "https://api.thinqit.ai/v1/decide",
        content=body,
        headers={"authorization": f"Bearer {os.environ['DEX_API_KEY']}", "content-type": "application/json"},
        timeout=30,
    )
    r.raise_for_status()
    d = r.json()
    assert d["served_by"] == "gpu", "pinned requests are always served by the GPU path"
    keep = {k: d[k] for k in ("model", "calibration", "answers")}
    keep["usage"] = {k: d["usage"][k] for k in ("input_tokens", "state_tokens", "question_tokens")}
    req.with_name(req.name.replace(".request.json", ".expected.json")).write_text(json.dumps(keep, indent=2) + "\n")
    print(req.name, d["model"], d["usage"]["input_tokens"], "tokens")

Commit the .expected.json files.

Test on every change#

# tests/test_dex_pinned.py
import json, os, pathlib
from datetime import datetime, timedelta, timezone

import httpx
import pytest

API = "https://api.thinqit.ai"
HEADERS = {"authorization": f"Bearer {os.environ['DEX_API_KEY']}", "content-type": "application/json"}
CASES = sorted(pathlib.Path("tests/dex").glob("*.request.json"))


@pytest.mark.parametrize("req", CASES, ids=lambda p: p.name)
def test_pinned_answers_do_not_change(req: pathlib.Path) -> None:
    expected = json.loads(req.with_name(req.name.replace(".request.json", ".expected.json")).read_text())
    r = httpx.post(f"{API}/v1/decide", content=req.read_bytes(), headers=HEADERS, timeout=30)
    r.raise_for_status()
    got = r.json()
    got["usage"] = {k: got["usage"][k] for k in expected["usage"]}  # token counts only
    assert {k: got[k] for k in expected} == expected


def test_pinned_versions_are_not_about_to_retire() -> None:
    pins = {json.loads(p.read_text())["model"] for p in CASES}
    models = {m["id"]: m for m in httpx.get(f"{API}/v1/models", headers=HEADERS, timeout=30).json()["data"]}
    soon = datetime.now(timezone.utc) + timedelta(days=30)
    for pin in pins:
        assert pin in models, f"{pin} is not listed any more"
        m = models[pin]
        assert m["status"] == "active", f"{pin} is {m['status']}"
        if m["retires_at"]:
            assert datetime.fromisoformat(m["retires_at"].replace("Z", "+00:00")) > soon, f"{pin} retires {m['retires_at']}"

// tests/dex-pinned.test.mts: node --test "tests/*.test.mts"  (Node 22.18 or newer)
import { test } from "node:test";
import assert from "node:assert/strict";
import { readFile, readdir } from "node:fs/promises";

const API = "https://api.thinqit.ai";
const headers = { authorization: `Bearer ${process.env.DEX_API_KEY}`, "content-type": "application/json" };
const dir = "tests/dex";
const cases = (await readdir(dir)).filter((f) => f.endsWith(".request.json")).sort();

for (const name of cases) {
  test(`pinned answers do not change: ${name}`, async () => {
    const body = await readFile(`${dir}/${name}`, "utf8"); // sent as it is: never JSON.parse and stringify
    const expected = JSON.parse(await readFile(`${dir}/${name.replace(".request.json", ".expected.json")}`, "utf8"));
    const res = await fetch(`${API}/v1/decide`, { method: "POST", headers, body });
    assert.equal(res.status, 200, await res.clone().text());
    const got = await res.json();
    got.usage = Object.fromEntries(Object.keys(expected.usage).map((k) => [k, got.usage[k]])); // token counts only
    for (const k of Object.keys(expected)) assert.deepEqual(got[k], expected[k], k);
  });
}

test("pinned versions are not about to retire", async () => {
  const pins = new Set<string>();
  for (const name of cases) pins.add(JSON.parse(await readFile(`${dir}/${name}`, "utf8")).model);
  const models = (await (await fetch(`${API}/v1/models`, { headers })).json()).data as Array<{
    id: string;
    status: string;
    retires_at: string | null;
  }>;
  const soon = Date.now() + 30 * 24 * 3600 * 1000;
  for (const pin of pins) {
    const m = models.find((x) => x.id === pin);
    assert.ok(m, `${pin} is not listed any more`);
    assert.equal(m.status, "active", `${pin} is ${m.status}`);
    if (m.retires_at) assert.ok(Date.parse(m.retires_at) > soon, `${pin} retires ${m.retires_at}`);
  }
});

Comparing parsed JSON with deepEqual is exact here: the answers are compared value by value, and a pick's labels are keys of an object, so their order does not matter to the comparison.

Run it in CI#

Store a test key as a CI secret and run the tests with it in DEX_API_KEY. In GitHub Actions, these are the steps of a job that has checked out the repository and set up Python, with the key stored as the repository secret DEX_TEST_KEY:

      - run: pip install pytest httpx
      - run: pytest tests/test_dex_pinned.py
        env:
          DEX_API_KEY: ${{ secrets.DEX_TEST_KEY }}

The TypeScript version runs with node --test "tests/*.test.mts" and the same variable, on Node 22.18 or newer, which runs TypeScript files as they are. On Node 20, install tsx and run node --import tsx --test tests/dex-pinned.test.mts.

When a test fails#

• Answers differ on the same version: check that the request file changed only where you meant it to (whitespace and key order count). If it did not change, write to hello@thinqit.io with the request id: that would break the determinism guarantee.
• 404 model_retired or the retirement check fails: move the pin. Record the answers on the new version with record.py, review the difference, and commit the new files together.
• 503 no_capacity: the pinned version had no GPU capacity for a moment. Retry the job; do not switch a pinned test to the alias.

For production code, keep the alias dex-1 (or pin as well and move deliberately) and keep the answers your code acts on above a min_confidence: a new version can move borderline answers.