Rate limits

Each API key has three limits: requests per minute, billable tokens per minute and requests in flight at once. Test keys also share a daily token quota per account. When you hit a limit, you get a 429 that names the limit and tells you how long to wait.

Limits#

• Per key. A new live key starts at the defaults. In the console you can set a key's limits lower, at or below the account caps: 1 to 600 requests per minute, 1,000 to 600,000 tokens per minute, and 1 to 32 concurrent requests.
• Account cap. The ceiling for your account's live limits. To go above it, ask for a raise (see Raising limits).
• Test quota. The daily test quota counts the billable tokens of every test key and every console playground call of the account together, per UTC day.

How limits are counted#

• Requests per minute counts every call the key makes to a public operation: decide, listModels, getCurrentKey, getUsage and getBalance, each one request. Idempotent replays count too.
• Tokens per minute counts billable tokens, the same input_tokens you are billed for. Only decide uses it. Dex knows this count before it runs the request, so the limit is charged with the exact number.
• Concurrent requests counts decide requests that have been admitted and have not yet answered.

The per-minute limits work like a bucket that refills at a steady rate, not like a counter that resets on the minute. You can send a short burst up to the quota, and capacity comes back gradually after it. All three limits are checked together in one step for every request, so a request is either admitted under all of them or rejected.

The largest request always fits a default limit. A request bills at most 16,384 tokens, and the smallest default tokens-per-minute limit is the test key's 40,000. So a key with a full bucket admits any valid request. If you lower a key's limit below 16,384 (the minimum is 1,000), a request larger than that limit is admitted only when the bucket is full, and it empties the bucket.

Current values are in the ratelimit-policy and ratelimit headers of every authenticated response. See Headers.

When you hit a limit#

You get HTTP 429 with type rate_limited, a code that names the limit, and a retry-after header in whole seconds:

{
  "error": {
    "type": "rate_limited",
    "code": "tokens_per_minute",
    "message": "This key used its 300000 tokens for this minute. Retry in 7 seconds.",
    "request_id": "req_01M5CJXHG0M9S346Q3D25VT4F5"
  }
}

A 429 is never charged. The SDKs retry it for you after retry-after, up to 3 times, except test_daily_quota, which they raise at once.

A request that fails with 500 or 503 gives back the tokens it took from your tokens-per-minute limit and from the daily test quota. Requests per minute stay counted.

Because each question in a request shares one state, one request with five questions uses fewer requests and fewer tokens than five requests with one question each.

Degraded mode#

Limits are counted in a shared store. If that store is unreachable, each gateway replica enforces its share of your limit (the limit divided by the number of live replicas) on its own. Your limits still apply, counted per replica, so they hold approximately rather than exactly while the store is down.

Fair sharing#

The queue in front of the GPU is weighted-fair per account. One account at its cap cannot starve other accounts. Live keys go ahead of test keys and playground calls.

Raising limits#

Ask for higher limits in the console. We raise account caps in steps, and each step is no bigger than the spare capacity the GPU pool showed over the last 7 days.