New playbook: when your own AI model pays. Read
LLM model routing is a gateway that sends each request to the cheapest model that meets the quality bar for that task.
It is the foundation layer of an AI-native company. This page covers the decisions a CEO owns here, the options, what drives cost, and what AT&T, Pinterest, Ramp, Meta and Travelers built in 2026.
Most work a company sends to AI does not need the most expensive model. A router classifies each request and picks a model by expected quality, cost and speed.
The router sits inside a gateway that every model call passes through. The gateway also holds budgets, fallbacks, data rules and the cost log. Grab’s engineering team wrote in July 2026: if moving between providers takes more than a few config changes, technical debt is already slowing you down.
Three choices set the economics of everything above this layer. The first is which workloads justify frontier prices. The second is which data may leave your infrastructure, which decides whether open models run on your own hardware. The third is the unit of work you track cost against.
Uber’s engineering team published its cost equation in August 2026: total spend is users times sessions per user times turns per session times requests per turn times tokens per request times price per token, and each managed agent is tracked by cost per merged change or per review next to a quality signal. Braintrust’s October 2026 version adds the failures: divide total spend on all attempts by the attempts that pass the quality gate.
Companies combine these four patterns. The table shows where each one fits.
| Option | How it works | Fits when |
|---|---|---|
| One frontier model | Every request goes to the top model | Launching a new workload and learning what good looks like |
| Rules first, model as fallback | Versioned rules resolve routine cases; a model handles the rest | Most cases repeat and errors are costly |
| Task-aware router | Each task goes to the cheapest model that meets the bar | Volume is high and tasks vary in difficulty |
| Own or open models behind the router | Post-trained open models take the high-volume tasks | Data must stay inside, or volume makes training pay back |
Databricks named four levers in August 2026: move defaults to newer efficient models, route each task to the cheapest capable model, show teams their spend with progressive friction in place of hard caps, and cut token overhead. All four are set in the gateway.
When agents run long tasks, caching affects cost. Databricks noted in August 2026 that routing for agents has to account for server-side caching, because a cold cache is costly on large contexts; task-level routing hands a whole task to one model.
AT&T runs an internal AI gateway that routes each prompt, turn by turn, to the cheapest adequate model, across an average of 45 billion tokens a day. AT&T says this is reducing AI costs “as much as 90%” and is “already saving millions” (July 2026).
Pinterest’s CEO told investors in August 2026 that open models post-trained on Pinterest data cost less than 8% per transaction of comparable closed models, on Pinterest’s own use cases. A central routing layer sends routine work to lighter models.
Ramp built one routing layer for more than 100 AI use cases, and its Router product shadow-tests candidate models on a sample of live requests. Ramp says the router cut its own LLM costs 30% (July 2026).
Meta reports that its privacy classifier resolves about 85% of traffic with versioned rules in milliseconds. The model handles the remaining 15% at roughly 400 times the compute cost, and a nightly loop turns reviewed model decisions into new rules (June 2026).
Travelers trained TravelersLLM on millions of company documents and routes each query to it or to a frontier model. Travelers says it beat commercial models on quality, cost and speed in internal testing on tens of thousands of insurance questions (June 2026).
Put one gateway in front of every model call this quarter and attribute cost to teams and workloads. Pick the three highest-volume workloads, replay a week of real requests against two cheaper models, and compare cost per successful task. Write the result into a routing policy with a named owner.
Other layers: Data and ontology · Agents, traceability and evals · The learning loop
Up to 90%
lower AI cost, AT&T says
An internal AI Gateway that, turn by turn, weighs speed, cost and expected quality and routes each prompt to the most cost-effective model, switching models mid-session, plus a library of post-trained open telco models (OTel 2.0) that run on-prem or in cloud (source).
<8%
of closed-model cost per transaction
A blended model portfolio (in-house compact models, open-source models post-trained on Pinterest data in its own secure cloud, closed models only in 'very limited cases') behind a centralized routing layer that sends routine work to lighter models; being extended to internal engineering and sales tools in H2 2026 (source).
30%
lower LLM cost on its own products
A routing layer that runs Ramp's production AI traffic across 100+ use cases: one API, shadow-testing candidate models on live traffic, flex-tier routing, benchmark routing, and per-request cost tracing (source).
85%
of cases resolved by versioned rules, LLM only for the rest
A privacy asset-classification system where versioned deterministic rules resolve routine assets and an LLM handles only novel or ambiguous cases; a nightly loop distils new rules from reviewed LLM decisions (source).
Own LLM
beat commercial models in internal tests, no figures published
TravelersLLM, a proprietary insurance LLM built by in-house engineers on millions of company documents with underwriting, claims and service experts; applications route each query to it or to a frontier model (source).
52%
lower cost per agent session since the June peak
A cost equation and weekly metric stack for its agentic 'software factory' (70%+ of PRs agent-attributed), with benchmark-driven Pareto model selection (incl. open-weight models), cheaper default subagent models, Medium reasoning effort default, 400K-token compaction cap and prompt-cache strategy (source).
30%+
lower average task cost from smart routing, internal coding work
An internal control layer between developers and models: moving defaults to newer cheaper models in one place, task-aware routing across models and harnesses, per-developer spend reports, and progressive friction (warning, approval, downshift to cheaper model) in place of hard caps (source).
53.6%
recall for a two-model reviewer vs 30.7% single-pass
A staged code-review agent (cheap scout model flags areas, stronger reviewer verifies) plus DashBench, which replays historical PRs to compare model mixes on recall, precision, cost and latency (source).
Routing holds quality when each task has a quality bar and a replay set. The router needs a quality bar per task and a replay set of real requests. DoorDash, for example, compares model mixes on historical code reviews for recall, precision, cost and latency before switching. Companies that publish results, such as Ramp and AT&T, report lower cost with quality held, on the workloads they measured.
Buy the gateway unless model traffic is part of your product. Gateways from cloud and data platforms already handle budgets, fallbacks and logging. What you own is the routing policy: which tasks go where, the quality bar, and the replay set. Those encode your workloads and stay with you when the vendor changes.
Open models fit when data must stay inside your infrastructure, or when one workload has enough volume to justify post-training. Pinterest reported open-model cost per transaction under 8% of comparable closed models in August 2026. Run them behind the same gateway so you can move each workload back if quality drops.
