The guardrails do not depend on the model: they are code in the orchestrator and the tool servers. That is what makes the model layer swappable. The agents ask for a tier; the gateway decides which model serves it, in which region, at what price.
01The model gateway
What it does
- One endpoint and one credential per environment for every agent process; no agent talks to a provider directly.
- Maps tiers (small, standard, large) to concrete models, per environment, and can route by task (fast and cheap for summaries and jobs, capable for root cause and change planning).
- Applies budgets in tokens and currency, logs usage per origin, agent and model, and enforces data-residency rules (which provider, which region, which data may leave).
- Falls back between deployments of the same model, never silently to a different kind of model.
Behind it
- Hosted APIs: Anthropic, OpenAI, Azure OpenAI, Google.
- Self-hosted models: vLLM, Ollama or TGI, behind an OpenAI-compatible API, on your GPUs, for data that may not leave.
- Agent frameworks on the ODIN side: the Claude Agent SDK, the OpenAI Agents SDK, LangGraph or similar. The framework runs the loop; it never holds the guardrails.
02Choosing and swapping models
- Evaluate before you swap. Keep a behaviour evaluation set (read questions, refusals of destructive requests, delegation to the right specialist, prompt injection in attachments, protected-target refusals) and run it against any new model or tier mapping in a test environment; promote only if it passes.
- Safety does not move with the model. A weaker model can make worse plans, but it cannot bypass the gate, the cards, admission or the vault: those are code.
- Tool use matters most. The agents call tools in loops; pick models that follow tool schemas reliably at the tier you assign.
- Private hosting. For sensitive environments, map every tier to self-hosted models and block hosted providers at the gateway and the egress firewall.
03Budgets
Per origin (people's chats, scheduled jobs, alerts): tool calls per agent run and per message, tokens per message (over it, the turn is interrupted), tokens and currency per chat (new messages refused once spent). Token budgets count input, output and cache writes; cache reads are left out so a long conversation isn't exhausted just by being re-read, but they count toward the currency cap. A monthly ceiling belongs at the gateway or the provider account, outside ODIN.
04Prompt caching
- Most input is a re-read. Every tool round re-sends the context; with a provider-side prompt cache those tokens bill at a fraction of input.
- Keep the prefix stable. The system prompt and tool definitions should not change within a chat; starting automation chats with only the tool servers they need keeps the prefix small.
- Idle gaps cost. If an agent stays warm longer than the provider's cache lifetime, a reply after a pause pays cache writes again.
- Long chats get expensive per turn. Offer a fresh chat from a summary once the context is large.
05Cost metrics
Count tokens per model response by origin, agent, model and type (input, output, cache read, cache write), cost per origin and model, turns by outcome, and turns stopped by a budget; keep user names and chat ids out of metric labels. Record each turn's tokens and cost in the audit log too, so spend is attributable even if the metrics store restarts.
06Estimating a month
Rates per million tokens (examples, editable)
| Tier | Input | Cache write | Cache read | Output |
|---|---|---|---|---|
| Large tier (example: a hosted frontier model) | ||||
| Standard tier (example: a hosted mid-size model) | ||||
| Small tier (example: a small hosted model) |
If your provider has no prompt cache, set the cache rates equal to input.
Self-hosted (example: GPU cost per hour)
A fixed cost whatever the token volume, plus operations: compare it with the hosted total at your volume, and check that the self-hosted model passes your evaluation set first.
Ceiling inputs
A per-chat cap is checked before each new message, so a chat can end slightly above it; a per-message token cap interrupts a runaway turn.
07In an organisation
- One gateway credential per environment (dev, test, prod), each with its own spend limit at the provider or the gateway.
- Rotate provider keys in the vault; the gateway picks them up, the agents never see them.
- Allow egress only from the gateway to the providers you approved, and only from ODIN to the gateway.
- Charge back by origin and model from the cost metrics.