Skip to main content
Every inference call in graph resolves through one system: a provider (who serves the model), a role (which job the call is doing), and optionally a named model (a specific entry selected at the point of use). Cost tuning is pure config — strong model where judgment lives, fast model where volume lives — and this page is the canonical map of where each call goes. The TOML syntax lives in the configuration reference.

​
Providers

A provider whose config can’t be honored — most commonly an unset ${VAR} behind its api_key, or the bedrock stub — is configured but not usable: the config still loads and every command that never resolves a model to it keeps working, while the first call that does errors naming the reason (for a missing variable, the variable and the config path that references it). Requests retry transient failures (429, 5xx, connection errors, timeouts) up to 3 attempts with backoff. Structured output uses each provider’s native mechanism — a forced tool on Anthropic, json_schema with a json_object fallback on OpenAI-compatible servers that reject strict schema mode — and a result that fails to parse or validate gets one repair-role fix-up pass before erroring. Anthropic requests retry once without temperature when a model rejects the parameter.

​
Roles

Each pipeline role resolves to a model, falling back to default:
The common cost setup: a strong model for chat/planner (they do the judgment), a fast model for solver/repair/judge (they do the volume). graph plan run on an authored plan touches only solver (or nothing) — the cost table maps invocations to calls.

​
Named models

Beyond the fixed roles, [models.named.<name>] entries are referenceable wherever a model name is accepted. That’s exactly three places:
  1. a prompt tool’s model field
  2. builtin__infer’s model input
  3. the model: override on an infer gate (exit/decide)
Role names resolve in those same places, with their usual fallback to default; names may not shadow role names (config load fails); an unknown name fails the call listing what is configured — never a silent fallback.
The description is a planner-facing routing signal: when named models exist, builtin__infer’s catalog schema advertises them with guidance to prefer the smallest adequate model, so planner-authored plans route small chunks of work — per-item map bodies especially — to cheap models on their own. Write descriptions for that audience.

​
Provider failover

Any model entry — a role or a named model — can carry ordered fallbacks for outages:
Each candidate names its own provider and model (model names rarely carry across providers); temperature optionally overrides — otherwise the primary’s effective temperature carries over. Every referenced provider must exist under [providers], checked at startup so a typo’d fallback surfaces immediately rather than mid-outage. Semantics:
  • A call moves to the next candidate only on outage-shaped errors — the transient class the retry layer recognizes — and only after the failing provider’s own retries are exhausted. Permanent errors (4xx, parse/schema failures) propagate immediately: a bad request would fail everywhere, and a fallback would only mask it.
  • Streaming fails over only while the stream is being established; once tokens flow, a mid-stream error surfaces as-is.
  • Fallbacks apply wherever the entry resolves — the agent loop, every pipeline role, structured output, and named-model calls. Each failover is logged as a warning on stderr.

Related topics

Execution modelConfigurationChangeloggraph as an MCP serverQuickstart