# Agent integration guide

This page is for an AI coding agent that builds RouterPlus into a product's code. It covers the whole integration in order: the key, the SDK, model calls, provider pinning, sessions and prompt caching, errors, limits and cost. Each step links to the reference page that has every detail.

Every docs page is also raw markdown: add `.md` to its URL, for example `https://app.routerplus.com/docs/errors.md`. The index of all pages is `https://app.routerplus.com/llms.txt`. All pages in one file: `https://app.routerplus.com/llms-full.txt`.

> [!NOTE]
> **For agents.** Do not guess model ids, request fields or headers. Copy them from this page or from the live catalog. If an error message disagrees with your plan, trust the error message and read the page it names. Some steps need a person. They are listed in [Steps only a human can do](#steps-only-a-human-can-do). Stop and ask at those steps.

## The short version

1. Point the OpenAI SDK at `https://api.routerplus.com/v1`, or the Anthropic SDK at `https://api.routerplus.com`. Read the key from `TM_API_KEY`, on the server only.
2. Copy model ids exactly from the catalog. The gateway never substitutes a model: an unknown id is a 404.
3. Send `max_tokens` on every request, from 1 to the model's `max_output_tokens` (16,384 on GPT-4o; 32,768 when none is declared). Without it, the model gets its own maximum (128,000 tokens on the Claude 5 and GPT-6 models), and your balance hold counts at most 32,768 output tokens while the call runs.
4. When you need provider features such as `thinking` or server tools, call Claude models on `/v1/messages` and other models on `/v1/chat/completions`.
5. Keep the start of the prompt identical from turn to turn, and only append. This keeps prompt caches warm.
6. Pin a provider only when you must, with the `provider` object. Check the plan first with `POST /v1/route`.
7. Switch on `error_type`, never on message text. Retry `rate_limit` after `retry-after`. Stop on `insufficient_quota` and tell a human.
8. There is no idempotency key. A retry is a new request, and it is billed again if it reaches a provider.
9. Keep at most 8 requests in flight per organization, unless you raised that limit.
10. Log `x-request-id` and `usage.cost` for every call.

## At a glance

| Item | Value |
|---|---|
| OpenAI-compatible base URL | `https://api.routerplus.com/v1`, for `POST /v1/chat/completions`, and `POST /v1/responses` for OpenAI's models |
| Anthropic-compatible base URL | `https://api.routerplus.com`, for `POST /v1/messages` (the SDK adds `/v1`) |
| API key | `tm_vk_` followed by 48 hex characters |
| Auth header | `Authorization: Bearer <key>` or `x-api-key: <key>`, on every route |
| Model list | `GET https://api.routerplus.com/v1/models` with your key, or `https://app.routerplus.com/api/models.json` (public, with prices) |
| Cost of a call | `usage.cost`, in USD, in every billed response |
| Audit of a call | the `x-request-id` response header, then `GET https://api.routerplus.com/v1/generation?id=<id>` |
| Images and decisions | `POST /v1/images/generations` and `POST /v1/decisions` |

### What is not available

Do not build on these. They do not exist on the gateway today:

- Embeddings, audio, moderation, files and batches. These paths return 404 `not_found`. Use Chat Completions, Messages or Responses.
- Stored responses on the Responses API: `store` is always `false`, and `previous_response_id` is a 400. The Responses API serves OpenAI's models; Claude and the other models are a preview there. See [POST /v1/responses](api-responses.md).
- Server-side conversation memory and idempotency keys. (Session ids exist: see [Sessions and caching](#sessions-and-caching).)
- More than one answer per request (`n` above 1 is a 400).
- Model fallback lists. A `models` array and a `route` field are dropped. The only aliases are the `rp-` names of the Claude models, which exist for Cursor: in code, use the catalog ids.
- Sorting providers by price or speed, and price limits. `provider.sort` and `provider.max_price` are a 400.
- Provider keys or endpoints in a request. `api_key`, `base_url` and `connection_id` in the body are a 400.

## 1. Get a key and keep it safe

| Key | Where it comes from | Requests per minute | Use it for |
|---|---|---|---|
| First trial key | Shown after verified browser sign-in through Clerk | 20 | The first test calls |
| Console key | [https://app.routerplus.com/console/keys](https://app.routerplus.com/console/keys) | 300 | Production, staging and CI |
| Identity API key | `POST https://app.routerplus.com/api/identity/keys` | 300 | One key for each end user or service (step 7) |

To get a key, the user opens [Sign up](https://app.routerplus.com/signup), completes Clerk
authentication and email verification, and saves the first key shown after
sign-in. Add paid credits in [Billing](https://app.routerplus.com/console/billing); signup does not grant free credit. Existing customers sign in
at [https://app.routerplus.com/login](https://app.routerplus.com/login) using the same verified email and
create a key at [API keys](https://app.routerplus.com/console/keys). Each raw key is shown once.

We match your first $100 in credit purchases, dollar for dollar. The offer applies to eligible accounts; Billing shows your remaining match.

Browser signup is required; the former programmatic signup and local magic-link
submission routes have been removed. Once `TM_API_KEY` is available, the agent
can configure clients and make API calls normally.

Rules for the key:

1. Keep the key on the server. Read it from the `TM_API_KEY` environment variable or from your secret store. Never commit it, log it or put it in a URL.
2. Do not put the key in a browser or a mobile app. Anyone can copy it from there. The gateway accepts cross-origin requests, so a copied key works from any web page. If a browser-only internal tool must call the gateway, give it its own key with a low monthly spend cap.
3. Use one key for each service and each environment. Spend, caps and logs are kept per key, and you can disable one key without stopping the others.
4. Use a console key in production. A trial key allows only 20 requests per minute.
5. To rotate a key: create the new key, deploy it, then disable the old key in the console. A disabled key stops working within about 5 seconds. The console cannot enable it again. An owner or admin can, with the identity API.

Details: [Authentication](authentication.md).

## 2. Connect the SDK

Change two values in the client you already use: the base URL and the key. Set both in code. Do not depend on `OPENAI_BASE_URL` or `ANTHROPIC_BASE_URL` in the environment. When that variable is missing, the SDK sends the request, with its key, to the provider's own API instead of the gateway.

Python — the OpenAI SDK:

```python
import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.routerplus.com/v1",
    api_key=os.environ["TM_API_KEY"],
    # Optional: these name your app in usage analytics. They carry no content.
    default_headers={"HTTP-Referer": "https://your-app.example", "X-Title": "Your App"},
)
```

TypeScript — the OpenAI SDK:

```typescript
import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.routerplus.com/v1",
  apiKey: process.env.TM_API_KEY,
  defaultHeaders: { "HTTP-Referer": "https://your-app.example", "X-Title": "Your App" },
});
```

With the Anthropic SDK, the base URL has no `/v1`:

```python
import os
import anthropic

client = anthropic.Anthropic(base_url="https://api.routerplus.com", api_key=os.environ["TM_API_KEY"])
```

```typescript
import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic({ baseURL: "https://api.routerplus.com", apiKey: process.env.TM_API_KEY });
```

### Which surface to call

Every chat model answers on both surfaces, because the gateway translates between the two formats. But a request keeps all its provider features only when your surface matches the format of the route that serves it. Then the gateway forwards the request as you sent it. The table gives the native surface of each model family's first route:

| Model ids | Native surface | Features that need it |
|---|---|---|
| `claude-*` | `/v1/messages` (Anthropic SDK) | `thinking`, server tools such as web search, PDFs given by URL |
| `gpt-*` | `/v1/chat/completions` (OpenAI SDK) | `response_format`, `reasoning_effort`, `seed`, `logprobs`, custom tools |

On the other surface, the gateway translates. A field that the translation cannot carry is dropped and named in the `x-tm-dropped-params` response header. If dropping the field would change the answer, the request is a 400 that names the field instead. Before it refuses, the gateway tries another provider of the same model that speaks your format. The `x-tm-provider` header names the provider that served you. The catalog gives each model's native format in `providers[0].dialect`. If your code sends only text, images, PDFs and function tools, either surface is fine for every model. Prompt caching marks (`cache_control`) reach Claude from both surfaces.

Details: [Wire compatibility](compat.md).

## 3. Call models

### Rules for every request

1. **Model ids are exact.** Copy them from `GET /v1/models` or `/api/models.json`. Claude and GPT ids are bare: `claude-sonnet-5`, `gpt-4o-mini`. Decision models keep their author prefix: `typesafe/jev-1.13`. Variants such as `:free` do not exist here. A dated snapshot of a listed model, such as `claude-haiku-4-5-20251001`, also works: it routes and bills as its model.
2. **Keep model ids in configuration**, in one place. Check them against `GET /v1/models` when your service starts. Then a model change is a configuration change.
3. **Use the right route for the output.** Chat models answer on the two chat surfaces. Image models answer only on `POST /v1/images/generations`, and decision models only on `POST /v1/decisions`. The catalog field `output_modalities` (`text`, `image` or `decisions`) tells you which is which. A decision model also takes its own question format: the catalog field `decisions_wire` in `/api/models.json` names it (`systemone` for Jev, the Bespoke Nimble models, Clef, Clef-flash, Decider 2B, Kev 4B, Perplexity Decider v1 27B, GPT-6 Luna Decisions, Sage and ScaleDown Classify-1, and `gliner`), and [POST /v1/decisions](/docs/api-decisions) gives each one. Do not guess the format from the model's name.
4. **Send `max_tokens` every time**, from 1 to the model's declared `max_output_tokens` (16,384 on GPT-4o; 128,000 on the Claude 5 models). When none is declared, the maximum is 32,768. If you leave it out, the gateway writes the model's own maximum into the request, lowered when needed so that the input and the output fit the model's context window. A value above the model's maximum is a 400. The gateway holds balance for the output limit you send while the call runs (at most 32,768 tokens when you send none), so a realistic value helps on a small balance, and lets more calls run at once.
5. **Ask for one answer.** `n` above 1 is a 400. Make separate calls for several samples.
6. **Send the whole conversation on every turn.** The gateway keeps no conversation state between calls. See [Data policy](data-policy.md) for how request data is handled.
7. **Keep `temperature` at 1 or less** in code that can reach Claude models. On a translated request, a higher value is a 400.
8. **Refuse drops where they matter.** A field outside the forwarded set is dropped and named in `x-tm-dropped-params`. Send `"provider": {"require_parameters": true}` to get a 400 instead of a drop. The 400 comes from the first route that would drop a field: the gateway does not go on to a later route that could carry it. So combine it with `only` or `order`. For example, the `anthropic` route drops `seed`, so a Claude request on `/v1/chat/completions` that sends `seed` and `"require_parameters": true` is a 400.

### Streaming

Stream answers that a person watches, and long answers. The OpenAI surface sends, in this order: a role chunk, content chunks, a finish chunk, a usage chunk with an empty `choices` list and `usage.cost`, then `data: [DONE]`. The Anthropic surface sends the native event sequence, with usage and `cost` in the final `message_delta`.

- If the provider fails after the answer starts, the stream ends with one error event and nothing after it (no `[DONE]`). The SDKs raise an error. The partial answer is billed. The gateway never continues an answer on another provider. To try again, send the whole turn again.
- During a silence of 15 seconds or more, the gateway sends a keep-alive: an SSE comment on the OpenAI surface, a `ping` event on the Anthropic surface. The SDKs skip them. If you parse SSE yourself, skip them too.
- The first response headers must arrive within 20 seconds, or the gateway tries the next provider. A request can run for at most 450 seconds.
- To stop an answer, close the connection. The provider stops. You pay for the usage that the provider reported. If it reported no final usage, you pay for the input (the provider's count when it sent one, else the gateway's estimate) and for the output already streamed to you. If nothing streamed yet, you pay for the input only.

Python — the OpenAI SDK, keeping the cost and handling a failure in the middle of an answer:

```python
from openai import APIError

parts, usage = [], None
try:
    stream = client.chat.completions.create(
        model="claude-sonnet-5",
        max_tokens=2048,
        stream=True,
        messages=messages,
    )
    for chunk in stream:
        if chunk.choices and chunk.choices[0].delta.content:
            parts.append(chunk.choices[0].delta.content)
        if chunk.usage:  # the usage chunk; when usage is known, it also comes before an error
            usage = chunk.usage
except APIError:
    # The answer failed part way. What arrived is billed. Do not join a retry onto
    # `parts`: send the whole turn again, or show the error.
    raise
print("".join(parts), "cost:", usage.cost if usage else None)
```

Details: [Streaming](streaming.md).

### Calls that do not stream

Set the client timeout above 450 seconds. The SDK default of 10 minutes is fine. The Anthropic SDKs refuse a call that does not stream when `max_tokens` is above about 21,000, unless you pass an explicit `timeout`. Stream those calls instead.

If you close the connection of a call that does not stream, the gateway stops the provider. Nothing was streamed to you, so you pay for the input only. An image render is different: it does not stop. It finishes and is billed at the usage that the provider reports. A call that does not stream and sends no `max_tokens` can run into the 450-second budget on a long answer, and then it fails (a 504, billed $0): send `max_tokens`, or stream.

### Tools, reasoning and structured output

- Function tools work on both surfaces for every chat model. The gateway translates tool calls and tool results.
- Custom (free-form) tools, `type: "custom"`, work on `/v1/chat/completions` for the GPT-6 and GPT-5.6 models. Their calls come back as `tool_calls` entries of `type: "custom"`.
- Anthropic server tools (web search, computer use, bash, text editor) work only for Claude models on `/v1/messages`, while the `anthropic` route serves them. Web search costs $10 per 1,000 searches (Anthropic's price), on top of the tokens, inside `usage.cost`. The `anthropic` route refuses `mcp_servers` and `container`: a request with either one is a 400 that names the field.
- `thinking` works only for Claude models on `/v1/messages`, while the `anthropic` route serves them. Reasoning comes back in different fields. On `/v1/chat/completions`, Claude's thinking arrives as `reasoning_content`. On `/v1/messages`, an answer from a non-Claude model carries its reasoning as `thinking` blocks, streamed or not (a non-streamed answer starts with it). You can send `thinking` blocks back in the next turn.
- `response_format` needs an OpenAI-format provider: GPT ids. For JSON from a Claude model, give it one tool whose input schema is the JSON you want, and read the tool input. Claude Sonnet 4.5 and Haiku 4.5 take a forced `tool_choice`; the Claude 5 models refuse a forced tool choice (`required` or a named tool) with a 400, so use `auto` there.
- Do not end the message list with an assistant message when a Claude model can serve the request from the OpenAI surface. That is a 400, because Claude rejects a prefilled answer.

### Images in prompts

Images and PDFs in user messages work on both surfaces. The gateway translates JPEG, PNG, GIF and WebP images (as a data URL or an `http(s)` URL) and PDFs (as a data URL) between the OpenAI and Anthropic formats. When the gateway must translate, an audio or video part, a file given by id, a file other than a PDF, or an image in an assistant message or a tool result is a 400 that names the part. Send an image, document, audio or video part only to a model that takes it: `architecture.input_modalities` in `GET /v1/models` lists each model's inputs. A part the model does not take is a 400 that names the part, and nothing is billed. Each image or document part counts as 65,536 tokens against your limits and your balance hold while the call runs.

### Other routes

- **Images:** `POST /v1/images/generations` takes the OpenAI Images API body. The image comes back as base64 in the response (`response_format: "url"` is a 400). `n` is 1 unless the model allows more: see `supported_parameters.n` in `/api/models.json` (GPT Image models allow up to 4). There is no streaming. A failed render bills nothing. See [POST /v1/images/generations](api-images.md).
- **Decisions:** `POST /v1/decisions` sends a state and typed questions to a decision model: `typesafe/jev-1.13` (Jev, from TypeSafe), `bespokelabs/nimble-v4` and `bespokelabs/nimble-v4-max` (Bespoke Nimble v4 and v4 Max, from Bespoke Labs; `bespokelabs/nimble-latest` lets Bespoke pick the model), `cloudflare/clef` and `cloudflare/clef-flash` (Clef and Clef-flash, from Cloudflare), `routerplus/decider-2b` and `routerplus/kev-4b` (Decider 2B and Kev 4B, from RouterPlus), `perplexity/pplx-decider-v1-27b` (Perplexity Decider v1 27B, from Perplexity), `openai/gpt-6-luna-decisions` (GPT-6 Luna Decisions, from OpenAI; `gpt-6-luna` is OpenAI's chat model, not this one), `levanto/sage-1.2` (Sage, from Levanto), `scaledown/classify-1` (ScaleDown Classify-1, from ScaleDown) or `fastino/gliner-2.5-decide` (GLiNER-2.5-Decide, from Fastino). It returns one typed answer for each question, such as a choice, a score or a yes/no probability. The question format follows the model: System One questions (Jev, Nimble, Clef, Clef-flash, Decider 2B, Kev 4B, Perplexity Decider, GPT-6 Luna Decisions, Sage and ScaleDown Classify-1) have a `type`, GLiNER takes a `schema` of classifications and extractions in place of `questions`, and a question in another model's format is a 400. GLiNER's numbers are confidences, not calibrated probabilities. Perplexity Decider bills the state once per question, so on it ask only the questions you need. GPT-6 Luna Decisions adds about 150 input tokens for each question, and it can decline one question: that answer is `{"type": "refusal"}`, so check each answer's `type` before you read its value. The gateway never fails over from one model to another. Use it for routing, classification and moderation points in your code, in place of a chat model that you parse. See [POST /v1/decisions](api-decisions.md).
- **Token counting:** `POST /v1/messages/count_tokens` is free, for models that have an Anthropic-format provider. It still counts against your requests per minute.
- **Your deployed models:** an id such as `tm/<name>-v1` from [Optimize](model-search.md) works on `/v1/chat/completions` only, with keys of the organization that deployed it. With `stream: true` it returns the finished answer as one chunk.

## 4. Pin providers (only when you must)

How routing works without a pin:

- Each chat model has a fixed, ordered list of providers. Today Claude models go to `anthropic`, and GPT models go to `openai`.
- The order does not change from request to request. There is no random spread across providers.
- The gateway moves to the next provider only when the current one fails before it sends any output, or has no capacity left.
- The price of a model is the same whichever provider serves it.

Pin when a rule requires one provider, when a feature exists at one provider only, or when you compare providers in a test. A pin can cost you fallbacks: `only` and `"allow_fallbacks": false` remove the fallback provider. `order` keeps every fallback.

### The `provider` object

Send it in the request body on `/v1/chat/completions` or `/v1/messages`. The gateway reads it and never forwards it. An unknown key inside it is a 400.

| Field | Value | Effect |
|---|---|---|
| `order` | provider ids | Try these providers first, in this order. The others stay as fallbacks. |
| `only` | provider ids | Use only these providers. |
| `ignore` | provider ids | Never use these providers. |
| `allow_fallbacks` | boolean | `false` keeps only the first route. If that route is resting after failures, the request fails with 503 and `retry-after` instead of moving on. |
| `require_parameters` | boolean | `true`: when the first route that would serve you would drop one of your request fields, the whole request is a 400. The gateway does not try a later route. |
| `connections`, `connection_order`, `funding` | ids, `"byok"` or `"house"` | Choose among your own provider connections and who pays. See [Routing policies](routing-policies.md). |
| `regions`, `zdr`, `data_collection` | region list, boolean, `"allow"` or `"deny"` | Hard filters. A route passes only with operator evidence, and marketplace routes have none today. So on marketplace traffic, `regions`, `"zdr": true` or `"data_collection": "deny"` removes every route. |

`order`, `only` and `ignore` take route labels. The marketplace routes are `anthropic` and `openai`: the `providers[].id` values in `/api/models.json`. Your own connections use their profile: `openai`, `anthropic`, `azure` or `bedrock`.

- A label that matches a route label selects or orders that route, even when that route does not serve the model: `only: ["anthropic"]` means the Anthropic route, so on a GPT model it leaves no route.
- Any other label matches no route: it selects, orders or skips nothing. An `only` list without a route label leaves no route.

| Goal | `provider` |
|---|---|
| Claude only from Anthropic's own API | `{"only": ["anthropic"]}` |
| Fail instead of falling back to another provider | add `"allow_fallbacks": false` |

Check a pin before you ship it. `POST /v1/route` returns the ordered candidates and each excluded route with its reason. It calls no provider and costs nothing. If your controls exclude every route, a real request returns 404 `model_unavailable`, the same error as an unknown model. So when a pinned request gets a 404, run `/v1/route`.

```bash
curl -s https://api.routerplus.com/v1/route \
  -H "Authorization: Bearer $TM_API_KEY" \
  -H "content-type: application/json" \
  -d '{"model":"claude-sonnet-5","provider":{"only":["anthropic"]}}'
```

Python — the OpenAI SDK sends fields it does not know through `extra_body`:

```python
reply = client.chat.completions.create(
    model="claude-haiku-4-5",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Hello"}],
    extra_body={"provider": {"only": ["anthropic"]}},
)
```

TypeScript — pass a variable, so the field the SDK types do not know is kept:

```typescript
const request = {
  model: "claude-haiku-4-5",
  max_tokens: 1024,
  messages: [{ role: "user" as const, content: "Hello" }],
  provider: { only: ["anthropic"] }, // read by the gateway, never forwarded
};
const reply = await client.chat.completions.create(request);
```

Check each answer with its response headers: `x-tm-provider` names the provider, `x-tm-served-by` names the provider that ran the request when a route reports one, `x-tm-upstream-model` gives the id the provider received when it differs from yours, and `x-tm-attempts` counts provider attempts. `GET /v1/generation?id=<x-request-id>` shows the route of every attempt.

To pin for a whole organization, workspace or key instead of each request, save a routing policy with the policy API. It takes a console session, so a human sets it up. A request's `provider` object can narrow a saved policy, but it can never add routes. See [Routing policies](routing-policies.md).

### Your own provider keys (BYOK)

You can route through your own OpenAI, Anthropic or Azure key, or an AWS Bedrock role. A human adds it once in the console as a connection. Your code still calls the gateway with a marketplace key. The provider bills you directly, and `usage.cost` is 0. A key bound to a connection can call only that connection's exact model ids, and it does not fall back to marketplace supply unless a routing policy allows it. See [Bring your own key](byok.md).

## 5. Keep sessions on one route and caches warm

A prompt cache saves money and time only when the next request reaches the same provider with the same prompt start. This is how the gateway treats sessions today:

- The gateway keeps no session state. Your app stores the conversation and sends all of it on every turn.
- Routing is deterministic. For the same model, key and `provider` object, every request tries the same route first. So the turns of a session stay on one provider without a session id.
- A turn moves to the next route when the first route fails before it sends output (one failure is enough), when the route's capacity pool is full, or while its provider's 429 pause lasts (the provider's `retry-after`, 1 to 60 seconds). After two failures in a row, later turns skip the route for 30 seconds. A turn that moves can miss the cache once. On a follow-up turn, the gateway first waits up to 5 seconds for a full pool: see [Waiting for the first route](#waiting-for-the-first-route).
- Send a session id, and the gateway gives it to each provider in the provider's own field: see [Sessions and caching](#sessions-and-caching). Claude Code's `x-claude-code-session-id` header counts, with no change on your side.

Rules for a high cache hit rate:

1. Keep the start of the prompt byte-identical: the same system prompt, the same tool definitions in the same order, and earlier messages unchanged. Only append.
2. Put content that changes (the time, ids, documents retrieved for this turn) after the stable part, at the end.
3. Keep the model and the `provider` object the same for the whole session.
4. For Claude, put `cache_control` on the block that ends the stable part (a system, tool or message block), or send the top-level `cache_control` field for automatic caching. Both surfaces keep the marks for Anthropic: see [Cache marks](#cache-marks). On marketplace routes the cache lasts 5 minutes: `ttl: "1h"` is removed. If you send no marks at all, the gateway turns on automatic caching for a request with tools or earlier turns: see [Automatic caching on Claude](#automatic-caching-on-claude).
5. OpenAI caches long repeated prompt starts by itself. You set nothing for it.
6. Measure. Cache reads are `usage.prompt_tokens_details.cached_tokens` on the OpenAI surface and `usage.cache_read_input_tokens` on the Anthropic surface. Cache writes are `cache_write_tokens` and `cache_creation_input_tokens`. Cache reads are billed at the lower `cached_prompt` price. Claude cache writes are billed at the higher `cache_write` price. Prices are in `/api/models.json`.

Python — the Anthropic SDK, with a cache mark on a long system prompt that does not change:

```python
reply = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=1024,
    system=[{
        "type": "text",
        "text": STABLE_INSTRUCTIONS,              # the same bytes on every turn
        "cache_control": {"type": "ephemeral"},   # cache everything up to here
    }],
    messages=history,                             # append only
)
print(reply.usage)  # cache_read_input_tokens, cache_creation_input_tokens and cost
```

The provider caches a prompt start only above a minimum length, so a short system prompt shows no cache reads.

### Sessions and caching

#### Session ids

On `POST /v1/chat/completions` and `POST /v1/messages`, the session id is the first valid value among these inputs, in this order. Token counting and images do not use it.

| Order | Input | Kind |
|---|---|---|
| 1 | `session_id` | Top-level body field |
| 2 | `x-session-id` | Header |
| 3 | `x-session-affinity` | Header |
| 4 | `x-claude-code-session-id` | Header that Claude Code sends by itself |

- A valid value has 1 to 256 printable ASCII characters. `prompt_cache_key`, a top-level body field, follows the same rule.
- The gateway ignores an invalid value and uses the next input. An invalid value alone is never an error. The ignored input is named in `x-tm-dropped-params` as `session_id (invalid)`, `prompt_cache_key (invalid)`, `header:x-session-id (invalid)`, `header:x-session-affinity (invalid)` or `header:x-claude-code-session-id (invalid)`. With `"require_parameters": true`, that makes the request a 400, as any dropped field does.
- Valid session inputs are never named in `x-tm-dropped-params`, and they never trigger `require_parameters`.
- `prompt_cache_key` stays OpenAI's own field. It wins over the session id on OpenAI routes.
- When two valid inputs disagree, the order decides. There is no error.
- Codex's `session-id` header is not read. Codex calls `POST /v1/responses` and sends its conversation id as `prompt_cache_key` in the body, which reaches OpenAI unchanged.

What each route receives:

| Route | Field | Value |
|---|---|---|
| `openai` (marketplace), and `openai` and `azure` (your connections) | `prompt_cache_key` | Your `prompt_cache_key`, or else the session id. Nothing if you sent neither |
| `morphllm` | Header `x-session-id` | The session id, or else your `prompt_cache_key`. Nothing if you sent neither. Morph uses it to pick the region and the worker that hold your cache |
| `anthropic` and `bedrock` | Nothing | Anthropic has no session input. Its cache matches the prompt start |

- Values go to the provider unchanged. A value longer than the provider accepts is replaced by `tm-` followed by 43 base64url characters: a hash of the value.
- The gateway keeps no session table. The id goes to the provider, and the provider routes on it.
- `prompt_cache_retention` goes unchanged to `openai` and `azure` routes, both `"in_memory"` and `"24h"`. On other routes it is dropped and named.

#### End-user ids on marketplace routes

On marketplace routes, the provider gets a hash in place of your end user's id, and the raw values do not go. The id is the first string among the body fields `safety_identifier`, `user` and `metadata.user_id`. The hash is `tm-` followed by 43 base64url characters, made from your organization id and that id. Without an id, it is made from your organization id alone. Each marketplace route gets exactly one field:

| Marketplace route | Field that carries the hash | Fields removed |
|---|---|---|
| `openai`, `azure` | `safety_identifier` | `user` |
| `anthropic` | `metadata.user_id` | `user`, `safety_identifier` |
| every other route (OpenAI-compatible), dedicated capacity included | `user` | `safety_identifier`, `metadata.user_id` |
| `mistral-ai` | none: Mistral has no end-user field | `user`, `safety_identifier`, `metadata.user_id`, each named in `x-tm-dropped-params` when you sent it |

- A provider that blocks an id then blocks one user of one customer, not the whole marketplace account.
- On your own connections, `user` and `metadata` pass unchanged. `safety_identifier` passes unchanged to `openai` and `azure` connections, and it is dropped and named on `anthropic` and `bedrock` connections.
- Identity fields that a route uses are never named in `x-tm-dropped-params`. Mistral's route uses none, so it names each one you sent, and `require_parameters` refuses that route.

#### Cache marks

The gateway never adds a cache mark that you did not send. (Automatic caching, below, adds only the top-level field, and only when you sent no marks.) Block and part marks (`cache_control`) are handled this way:

| Request | Route | What happens to the marks |
|---|---|---|
| `/v1/chat/completions` | Anthropic direct or Bedrock | Each text part keeps its mark. A marked system part turns the system prompt into blocks. A tool message's last mark goes on its `tool_result` block. Other part keys are not forwarded |
| `/v1/chat/completions` | OpenAI direct or Azure | Part marks pass unchanged. OpenAI and Azure ignore them |
| `/v1/messages` | Anthropic direct or Bedrock | Block marks pass unchanged |
| `/v1/messages` | OpenAI direct or Azure | Marks are removed and named, for example `messages[i].content[j].cache_control` or `tools[i].cache_control` |

A top-level `cache_control` field (automatic caching), on either surface:

- Anthropic direct: sent unchanged.
- Bedrock: turned into a mark on the last block that can carry one, counted back from the end of the messages. If no block can carry it, it is dropped and named as `cache_control`.
- OpenAI direct and Azure: dropped and named as `cache_control`.
- Any route: a field that Anthropic would refuse is dropped and named as `cache_control`. Anthropic refuses a field other than `{"type": "ephemeral"}` with an optional `ttl` of `"5m"` or `"1h"`, a fifth mark (four marks exist already), a 1-hour field after a 5-minute mark, and a field whose TTL differs from the mark on the last block.

On marketplace routes, `ttl: "1h"` is removed from every mark, so the cache lasts 5 minutes. The drop is named as `<path>.cache_control.ttl`, for example `system[0].cache_control.ttl`, `messages[2].content[0].cache_control.ttl`, or `cache_control.ttl` for the top-level field. `ttl: "5m"` passes. Your own connections keep `ttl: "1h"`.

#### Automatic caching on Claude

Many agents send no cache marks. For them, the gateway adds the top-level `cache_control: {"type": "ephemeral"}` field (5 minutes) when all of these are true:

- Anthropic's own API serves the request: the `anthropic` route or your own Anthropic connection. Not Bedrock, and not a dedicated endpoint.
- The request has no `cache_control` anywhere.
- It has tools, or an assistant turn before its last message.
- The prompt is at least the model's minimum cacheable size: 512 tokens on the Claude 5 models except Sonnet 5, 1,024 on Claude Sonnet 5 and Sonnet 4.5, and 4,096 on Claude Haiku 4.5.
- The request does not carry the header `x-tm-auto-cache: off`.

The response header `x-tm-auto-cache` is `applied`, or `skipped:` with the reason, for example `skipped:single-turn` or `skipped:client-markers`. The first turn pays a cache write (1.25 times the input price) and later turns read the prompt at the cache price. Details: [Wire compatibility](compat.md#automatic-caching-on-claude).

#### Waiting for the first route

On a follow-up turn (an assistant or tool message after a user message), if the first route's capacity pool is full, the gateway waits for that pool for up to 5 seconds (an operator setting) before it moves the request to the next route. It never waits past the request's deadline. The `x-tm-affinity-wait-ms` response header gives the wait in milliseconds, only when the wait was more than 0.

## 6. Handle errors and retries

Every failure carries one stable class, `error_type`. Read it from `error.metadata.error_type` on the OpenAI surface, or `error.error_type` on the Anthropic surface. The `x-tm-error-code` header carries it too, but browsers cannot read that header. Never parse the message text. Log the `x-request-id` header with every failure.

| `error_type` | HTTP | Retry | What to do |
|---|---|---|---|
| `rate_limit` | 429 | Yes | Wait `retry-after` seconds, then retry; a capped wait can take a second try. If `x-tm-limit-kind` is `concurrency`, send fewer requests at once. |
| `email_not_verified` | 403 | No | Stop and tell a human to open the verify link in their email; the key starts working within seconds of the click. |
| `insufficient_quota` | 429 | No | Stop and tell a human: add credits or raise a spend cap. On a cap, `x-tm-cap-reset` gives the reset time. |
| `upstream_error` | 5xx | Yes, once, after about 2 seconds | The gateway already tried every other provider it could. If the retry fails too, report it with the `x-request-id`. |
| `upstream_error` | 4xx | No | The provider refused the request itself. The message carries the provider's own words. |
| `gateway_error` | 503 | Yes | Wait `retry-after`. The request did not reach a provider. |
| `gateway_error` | 504 | Yes, once | The deadline passed while the gateway tried providers. An earlier attempt may be billed: check `GET /v1/generation?id=<x-request-id>` first. |
| `gateway_error` | 400 | No | Send one output limit, from 1 to the maximum the message names (32,768 only when no maximum is declared). |
| `gateway_error` | 500 | No | Report it with the `x-request-id`. |
| `model_unavailable` | 404 | No | Fix the model id, or fix your `provider` object (check it with `POST /v1/route`). |
| `model_unavailable` | 502 | Yes, once | The connection to every provider failed. |
| `invalid_request` | 400 | No | Fix the field that the message names. |
| `context_overflow` | 400 | No | Shorten the input, or use a model with a larger context. The body has the standard shape that agents compact on: `error.code` `context_length_exceeded` on the OpenAI surface, `prompt is too long: N tokens > M maximum` on the Anthropic surface. |
| `content_policy` | 400 | No | Change the prompt. It is never sent to another provider. |
| `auth` | 401 | No | Check the header, the key, and whether the key was disabled. A 401, 402 or 403 relayed from every provider is the marketplace's own account problem: report it with the `x-request-id`. |
| `request_too_large` | 413 | No | Keep the body under 10 MB. |
| `not_found` | 404 | No | Use a route that exists. The list is in [Errors](errors.md). |

Rules:

1. Let the SDK retry. The official OpenAI and Anthropic SDKs retry 429 and 5xx responses twice by default, and they wait for `retry-after`. Keep that default. Do not add a second retry loop around it. The SDKs also retry an `insufficient_quota` response. That costs nothing, but your code must then stop.
2. There is no idempotency key. A retried request is a new request. If it reaches a provider, it is billed again. A refusal by the gateway's own limits (`x-tm-error-origin: gateway_admission`) cost nothing.
3. Never retry `insufficient_quota`, `invalid_request`, `context_overflow`, `content_policy` or `auth` in a loop. The request, or the account, must change first.
4. An error in the middle of a stream is final for that answer. Send the whole turn again. Never join two partial answers.

Python — the OpenAI SDK, reading the class:

```python
from openai import APIStatusError

try:
    reply = client.chat.completions.create(model=MODEL, max_tokens=1024, messages=messages)
except APIStatusError as e:
    error = e.response.json().get("error") or {}
    kind = (error.get("metadata") or {}).get("error_type") or e.response.headers.get("x-tm-error-code")
    request_id = e.response.headers.get("x-request-id")
    if kind == "insufficient_quota":
        notify_operator(request_id, error.get("message"))  # money or a cap: a retry cannot help
    raise
```

Details: [Errors](errors.md).

## 7. Limits, spend caps and your users

Default limits:

| Scope | Requests per minute | Tokens per minute | In flight at once |
|---|---|---|---|
| Trial key | 20 | 1,000,000 | 8 |
| Console or identity API key | 300 | 1,000,000 | 8 |
| Organization, all keys together | 300 | 1,000,000 | 8 |

Rules:

1. Limit how many requests your code sends at once. By default the organization allows 8 in flight. A ninth gets 429 `rate_limit` with `x-tm-limit-kind: concurrency`. Use a semaphore or a fixed pool of workers. To run more at once, a human first raises the organization and key limits in [/console/limits](https://app.routerplus.com/console/limits). A key's requests per minute cannot go above its ceiling (300 for a console key). Each marketplace provider pool also limits one organization to half of the pool's capacity. That refusal carries `x-tm-limit-scope: pool`.
2. Read `x-tm-remaining-rpm` and `x-tm-remaining-tpm` on each response, and slow down before they reach zero. `GET https://api.routerplus.com/v1/limits?model=<id>` returns the limits that apply to your key.
3. While a call runs, token limits count an estimate: one token for each four bytes of the request, rounded up, 65,536 for each image or document part, plus `max_tokens` (at most 32,768 of it). An image sent inline as base64 counts its 65,536 only, not its bytes. The real count replaces the estimate when the call ends. Small requests and a realistic `max_tokens` let more calls run at once.
4. Spend caps are monthly, for each key and for the organization, on the UTC calendar month. A human sets them in the console. A cap refuses with 429 `insufficient_quota` and the `x-tm-cap-reset` header. Give each feature or customer its own key and cap, so a bug or a loop cannot spend everything.

Your users:

- By default, one backend key serves all your users. The gateway does not know who your users are, so enforce per-user quotas in your app.
- `safety_identifier`, `user` and `metadata.user_id` identify your end user to the provider. On marketplace routes the gateway sends only a hash of that id with your organization; on your own connections they go unchanged (see [End-user ids on marketplace routes](#end-user-ids-on-marketplace-routes)). They do not identify, limit or bill anyone on the gateway. Do not put emails or names in them.
- For limits that the gateway enforces per user, use the identity API. Create an `end_user` principal for each user, issue a key for it, and set limits on the principal. Your backend keeps those keys. Never give a key to the user. The identity API takes a console session, not an API key, so a human with an owner or admin role sets it up. See [Workspaces and principals](identity.md).
- To name your app in analytics, send the `HTTP-Referer` and `X-Title` headers. They carry no content.

Details: [Rate limits and spend caps](limits.md) and [Limits and capacity](admission.md).

## 8. Track cost

- Every billed response has `usage.cost` in USD. It is the full charge for the request, including provider attempts that failed before the answer. Store it with the `x-request-id`, the model and your own ids, such as the user and the feature.
- In a stream, `cost` is in the usage chunk on the OpenAI surface, and in the last `message_delta` on the Anthropic surface.
- An Anthropic-surface stream that fails in the middle of the answer has no usage in it. Read its cost from `GET /v1/generation?id=<x-request-id>`.
- `GET /v1/usage` returns the balance and the last 20 attempts. For a key bound to its own workspace or principal, the balance fields are `null`. The console shows usage and logs for the whole organization.
- BYOK calls show a cost of 0, because your provider bills you directly.
- Anthropic's web search adds its fee per search to `usage.cost` (see [Pricing and billing](pricing.md#web-search)).
- To check a charge, multiply the token counts by the prices in `/api/models.json`. The gateway rounds the final charge down.

Details: [Pricing and billing](pricing.md) and [GET /v1/usage and /v1/generation](api-usage.md).

## 9. Verify the integration

One cheap call shows the answer, the cost and the headers that the gateway adds:

```bash
curl -sS -i https://api.routerplus.com/v1/chat/completions \
  -H "Authorization: Bearer $TM_API_KEY" \
  -H "content-type: application/json" \
  -d '{"model":"gpt-4o-mini","max_tokens":16,"messages":[{"role":"user","content":"Reply with OK"}]}'
```

Then confirm that the ledger saw it:

```bash
curl -s -H "Authorization: Bearer $TM_API_KEY" https://api.routerplus.com/v1/usage
```

The integration is done when all of these are true:

1. Every client uses the gateway base URL and reads the key from `TM_API_KEY` on the server.
2. The key is not in the repository, in a client bundle or in a log.
3. Every model id in the configuration appears in `GET /v1/models`.
4. Every request sends `max_tokens`.
5. One streamed and one non-streamed test call return `usage.cost`.
6. The `x-tm-dropped-params` header is absent, or it names only fields that you accept losing.
7. `x-tm-provider` shows the route that you expect.
8. The error handling switches on `error_type`. It stops on `insufficient_quota`, and it adds no retry loop on top of the SDK's own retries.
9. The number of requests in flight stays within the organization's limit.
10. The logs keep `x-request-id` and `usage.cost` for every call.
11. The test calls appear in `GET /v1/usage`.

## Leave instructions for the next agent

Add this block to the `AGENTS.md` or `CLAUDE.md` file of the repository that you integrated. Later coding sessions then keep the integration correct:

```markdown
## LLM calls: RouterPlus gateway

- Every LLM call goes through the RouterPlus gateway. Full guide: https://app.routerplus.com/docs/agent-integration.md
- OpenAI SDK base URL: https://api.routerplus.com/v1. Anthropic SDK base URL: https://api.routerplus.com. Key: the TM_API_KEY environment variable, on the server only.
- Model ids live in configuration. Each one must exist in https://app.routerplus.com/api/models.json. Never invent or alias an id.
- Send max_tokens on every request (1 to the model's max_output_tokens in models.json; 32768 only when none is declared). Never send n above 1.
- Claude models: /v1/messages, with cache_control on the stable start of the prompt. GPT models: /v1/chat/completions.
- Keep the start of the prompt byte-identical across turns. Only append.
- Send one id per conversation as the x-session-id header (or session_id in the body). Providers then keep its cache warm.
- Provider pins go in the request body "provider" object (only, ignore, order, allow_fallbacks, require_parameters). Check a pin with POST https://api.routerplus.com/v1/route.
- Errors: switch on error_type. rate_limit: wait retry-after, then retry; a capped wait can take a second try. insufficient_quota: stop and tell a human. Never retry invalid_request, context_overflow or content_policy.
- No idempotency key: a retry is a new, billed request. Add no retry loop on top of the SDK's own retries.
- At most 8 requests in flight per organization, unless the limit was raised.
- Log x-request-id and usage.cost for every call.
- Not available: embeddings, audio, files and batches. POST /v1/responses serves OpenAI models; Claude and the other models are a preview there. On it, store is always false and previous_response_id is a 400.
```

## Steps only a human can do

- Complete browser signup through Clerk, including email verification, then buy credits in Billing. Signup and card verification alone do not add credit.
- Fix billing when `insufficient_quota` appears: add credits, or raise a spend cap in the console.
- Sign in to the console to create keys, set spend caps and change limits.
- Add your own provider keys (BYOK), save routing policies, and set up workspaces and end-user principals.
- Approve large test runs. Every call is billed.

## Where next

- [Quickstart](/docs/quickstart) — the first streamed call, step by step
- [Errors](/docs/errors) — every error class, and what to do about it
- [Routing policies](/docs/routing-policies) — the provider object, saved policies and BYOK routes
- [Streaming](/docs/streaming) — frame order, keep-alives and failures in the middle of an answer
- [Rate limits & spend caps](/docs/limits) — the numbers behind every 429
- [Wire compatibility](/docs/compat) — which fields pass, which translate and which drop