# Fallback and served_by

> When a hosted fallback in the EU answers instead of the GPU, how every response says so, and how to keep a request on the GPU.

Dex answers from its own EU inference nodes: dedicated EU hardware operated by thinQit. When that node cannot answer an alias request in time, a hosted fallback on Azure OpenAI in the EU data zone can answer instead. Every response tells you which path answered in `served_by`, and you can keep any request on the GPU path.

## Why a fallback exists

When those nodes are down, restarting, overloaded or in maintenance, the fallback keeps alias requests answered. Planned maintenance is announced 48 hours ahead on the status page at https://thinqit.ai/status/, and uses the fallback while it lasts.

## When a request goes to the fallback

A request can go to the fallback only when all of these hold:

- `model` is the alias `dex-1`, not an exact version;
- `fallback` is `"allow"`, which is the default;
- the key is a live key. Test keys and console playground calls always use the GPU path.

It then goes to the fallback when the GPU path cannot answer it in time:

- no ready and healthy GPU worker serves the version, or its queue is too long to answer in time;
- the GPU worker fails the job, does not answer in time, or goes offline while the job runs.

The fallback also has a spending limit. When it is closed or also fails, the request gets `503 no_capacity` or `503 upstream_unavailable` with a `retry-after` header, and any charge is refunded.

## served_by in every response

Every response has three fields that tell you how it was answered:

| Field | GPU path | Fallback path |
| --- | --- | --- |
| `served_by` | `gpu` | `fallback` |
| `model` | An exact GPU version, such as `dex-1.0.0` | `dex-fallback-1.0.0` or `dex-fallback-lite-1.0.0` |
| `calibration` | The GPU version's calibration | The fallback version's own calibration |

`GET /v1/usage` counts `gpu_requests` and `fallback_requests` per time bucket, so you can see how much of your traffic each path served.

## Keeping a request on the GPU

There are two ways:

- Send `"fallback": "never"` with the alias.
- Pin an exact GPU version, such as `dex-1.0.0`. Exact versions never use the fallback, whatever `fallback` says.

The trade-off: when no GPU can serve the request, you get `503` with a `retry-after` header instead of a fallback answer. The English moderation example in the contract does both:

```json
{
  "model": "dex-1.0.0",
  "fallback": "never",
  "state": {
    "post": {
      "board": "Bikes for sale",
      "title": "Selling my old road bike",
      "text": "Still available, cash only. Text me on +31 6 1234 5678. The next person who lowballs me will find their home address posted in this thread."
    }
  },
  "questions": {
    "rule": {
      "type": "pick",
      "instructions": "Which community rule does {{post.text}} break, if any?",
      "criteria": [
        "A threat to expose where someone lives counts as harassment even when it is conditional.",
        "Sharing your own phone number in a sale listing is allowed."
      ],
      "options": {
        "none": "Breaks no rule",
        "harassment": "Threats, intimidation or exposing someone's personal details",
        "spam": "Repeated or unsolicited promotion",
        "self_harm": "Encourages or describes self-harm",
        "hate": "Attacks people for a protected characteristic"
      },
      "min_confidence": 0.6
    },
    "severity": {
      "type": "rate",
      "instructions": "How severe is the worst problem in {{post.text}}?",
      "levels": ["None", "Mild", "Serious", "Severe"]
    },
    "has_phone_number": {
      "type": "check",
      "instructions": "Does {{post.text}} contain a phone number?"
    }
  }
}
```

## What differs on the fallback

- **Not deterministic.** The same request can give different numbers on the fallback, and a retry can be answered by the other path. `GET /v1/models` lists fallback versions with `deterministic: false`. See [Determinism](/docs/concepts/determinism/#scope).
- **Its own calibration.** Fallback versions have their own temperatures and their own calibration reports.
- **More than 20 options.** On the fallback, a pick with more than 20 options gets exact probabilities only for the 20 most likely options. The remaining probability is spread evenly over the other options. The model's `limits.exact_option_probabilities` shows 20 for fallback versions.
- **Content filters.** Azure OpenAI's default content filters block prompts with hate, sexual, violent or self-harm content above medium severity. If the filter blocks any question in a request, the fallback fails for the whole request, and it gets `503 upstream_unavailable` with a refund, as if the fallback were down. Before launch we apply to Microsoft for modified content filtering (annotate only) on both fallback deployments, and this page will show the status. Content filters never apply on the GPU path.

## Moderation traffic

**Send every moderation request with `"fallback": "never"`**, or pin an exact version. Moderation content is exactly what the fallback's content filters block, so on the fallback such a request would fail anyway. With `never` it stays on the GPU path, where no content filter applies, and it gets a quick `503 no_capacity` with `retry-after` when no GPU is free.

```json
{
  "model": "dex-1",
  "fallback": "never",
  "state": { "post": { "text": "..." } },
  "questions": {
    "rule": { "type": "pick", "instructions": "Which community rule does {{post.text}} break, if any?", "options": { "none": null, "harassment": null, "hate": null } }
  }
}
```

Dex cannot tell moderation traffic from other traffic by itself, so the choice is yours to make in the request.

## Cost

A request costs the same on either path. Both paths count tokens with the same tokenizer and charge the same price. If the fallback fails, the charge is refunded in full. See [Tokens and billing](/docs/reference/billing/).

## Data on the fallback path

The fallback processes requests in the EU data zone, through an Azure OpenAI Data Zone Standard deployment whose account is in West Europe, or in Sweden Central if West Europe lacks the model.

One exception applies to stored data. By default, Azure OpenAI abuse monitoring may store prompts that its classifiers flag for up to 30 days, inside the EU data zone. We apply for modified abuse monitoring before launch. Until Microsoft approves it, this exception applies to requests served by the fallback. To avoid it, send `"fallback": "never"` or pin an exact version. See [Data handling and residency](/docs/reference/data-handling/#azure-openai-abuse-monitoring).

## Replacing the hosted fallback

Both upstream models retire on 14 April 2027. We plan to replace the hosted fallback with more inference nodes of our own before 14 February 2027. Those nodes would serve the same exact versions, so failover would keep the determinism guarantee. The [changelog](/docs/changelog/) will record the change.