When to own a model, and how to get there
Build your own AI model when one workload has high volume, a quality bar your own evals can measure, and data or expert judgment that general models lack.
Until then, use frontier models through an API and route between them. This playbook gives the stages from frontier launch to a continual learning loop, with cases from Shopify, Bridgewater, Intercom and Kasisto.
This playbook belongs to The learning loop and Models and routing.
Why now
Shopify replaced the frontier model behind one Sidekick sub-agent with a smaller open model it post-trains on its own production failures. Shopify estimates serving cost fell from about $27M to about $1M a year, with quality above the frontier baseline on its own judge (August 2026). By October its judge score had risen from 37.7 to 74.5 after six loops, against 73 for the frontier model, with median responses in 0.87 seconds against 6.50.
Financial firms report the same pattern. Bridgewater fine-tuned an open model on investor-labelled data for six daily information-filtering tasks. In the companies’ own evaluation it reached 84.7% accuracy against 78.2% for the best frontier model, at 13.8 times lower inference cost per task (June 2026). Kasisto’s FinRAG-12B, a grounded banking Q&A model deployed at 40+ financial institutions, cost about $1,800 to train (ACL paper, July 2026).
Satya Nadella framed the strategic point in June 2026: the advantage is a private learning loop built on private evals and private reinforcement learning environments trained on the company’s own traces, and a company should be able to swap the generalist model without losing its accumulated expertise.
96%
lower estimated serving cost on one Sidekick sub-agent (Shopify)
84.7%
accuracy against 78.2% for the best frontier model (Bridgewater Associates)
$1,800
approximate training cost of FinRAG-12B (Kasisto, deployed at 40+ financial institutions)
Launch on a frontier model
Frontier models are the fastest way to ship. Shopify’s engineers call them frozen and expensive at volume, so improvements go first into prompts and the agent harness (August 2026). Fireworks describes four stages: use a closed frontier model, push prompts and harness, mix in open models chosen on your own evals, then train (September 2026).
Mollick’s warning applies here: hand-built scaffolding keeps losing to general methods, so avoid building elaborate structure the next general model will make obsolete (October 2026). Train nothing until you have volume, logs and a measured gap.
The step produces the workload in production on a frontier model, with every request and outcome logged.
Route each task to the cheapest adequate model
Most tasks do not need the top model. AT&T built an internal gateway that sends each prompt to the cheapest adequate model and says it is reducing AI costs “as much as 90%” on selected workloads (July 2026, company figure with no company-wide average). Pinterest’s CEO said open models post-trained on its data run at less than 8% of the cost per transaction of comparable closed models (Q2 2026 earnings call, August 2026).
Travelers routes between its own insurance LLM and frontier models by task; its CTO’s rule is quoted below. Applied Compute trained a small router for Clay and says it cut Clay’s inference cost by up to 47% at retained quality (vendor case, October 2026).
The step produces a routing policy and cost per successful task for each model in use.
“…if you don't need to be using the most expensive frontier model to answer the same question, we shouldn't be,” said Travelers CTO, via CIO Dive, August 2026.
Build your own eval set and judge
You cannot beat a frontier model you cannot measure against. Shopify defines quality as a rubric and calibrates an LLM judge against past A/B results. Harvey found that grading with a smaller model and batching criteria cut grading cost up to 40 times against a single-criterion frontier grader (June 2026).
Expert preferences make good training data. Heidi Health trains on clinicians’ side-by-side choices. Its post-trained NVIDIA Nemotron 3 Ultra wins about 54% of live clinician comparisons against the frontier model in production, up from about 42% for the base model (Fireworks customer case, October 2026).
The step produces a held-out set of real cases and a judge calibrated against your experts.
Decide per workload with the worksheet
Score each candidate workload on volume, unit cost, the measured quality gap, data you own, latency and sovereignty needs. Own a model only where several of these line up.
Bridgewater found newer frontier models gained little per dollar on its judgment tasks: GPT-5.4 cost 43% more than GPT-5.2 for marginal gains (June 2026). Its conclusion is in the quote below. L3Harris says a fine-tuned open model beat a frontier model on one country-of-origin monitoring task at 95% lower model cost (September 2026). Red Hat reports that Atos raised a small fine-tuned model’s valid structured output from 1% to about 99%, above GPT-4.1’s 96%, training on three GPUs (October 2026).
The step produces a go or no-go decision per workload, signed by the CTO and the CFO.
“An explicit prompt can only convey the intuition an expert is able to put into words,” said Bridgewater AIA Labs, June 2026.
Post-train an open model on your data
Start from an open-weight base model, fine-tune it on your best examples, then train it with reinforcement learning against your judge. Shape the training data around the cases your business handles. Kasisto made 22% of its training examples unanswerable so the model learned to say “I don’t know”; it answers at about $0.001 per query, 20–50 times cheaper than GPT-4.1, an older baseline.
Intercom’s Apex 2.0, post-trained on NVIDIA Nemotron 3 Super, is Fin’s flagship answer-generation model. Intercom says that against Apex 1.0, hard resolutions rose 8.6% and hallucinations fell 16.5%, and the model is live for every Fin customer (October 2026). NVIDIA says a model trained on planner decision records reached 86.7% accuracy on allocation decisions in a back-tested development set, against 55.5% for its much larger Nemotron 3 Ultra (September 2026).
The step produces a candidate model that beats the incumbent on the held-out set and in a live comparison.
Run the continual learning loop
On a fixed cadence, the judge scores production traffic, failures are repaired, and the repaired examples are trained into the next version. Shopify ran six loops on about 61,000 repaired examples to reach its October result.
Dwarkesh Patel argues that when models learn from deployment, shipping earlier gives a model more time to learn, and enterprises may want that learning to stay on their side (August 2026). If the company owns the model weights, that learning stays inside the company.
The step produces a scheduled retraining cadence with promotion rules tied to the judge and live metrics.
“…because Shopify controls the post-trained weights, we can serve the model on our own GPUs and decide when it changes,” said Shopify Engineering, October 2026.
Keep the generalist model swappable
LangChain’s July 2026 guidance: buy generic infrastructure and own what improves with use, meaning model choice or open weights, orchestration, context, memory, evals, cost controls and the feedback loop. Keep the eval set, judge and training data independent of any one model vendor, so a better base model arrives as an upgrade.
The step produces a list of owned assets with an owner each: eval set, judge, training data, routing policy, weights.
Do this week
- List your five highest-volume AI workloads with monthly requests and cost.
- For the largest one, collect 200 real cases with expert-approved answers as the first eval set.
- Ask your platform team whether every model call goes through one gateway you control.
Measure
| Metric | How to measure | Target direction |
|---|---|---|
| Cost per successful task | All inference and review cost divided by tasks passing the quality gate | Down |
| Judge score against frontier | Owned model score minus frontier score on the held-out set | Up |
| Live preference rate | Share of expert side-by-side comparisons won by the owned model | Up |
| Latency | Median full response time in production | Down |
| Loop cadence | Days between promoted model versions | Down |
Questions
When should a company build its own AI model?
When one workload has enough volume that unit cost matters, a quality bar you can measure on your own held-out cases, and data or expert judgment general models lack. Shopify, Intercom and Kasisto met all three on a single high-volume task. Below that bar, routing between rented models delivers most of the savings with less effort.
How much does it cost to train a custom model?
The training runs in these cases are small. Kasisto reports about $1,800 to train FinRAG-12B, and Atos used at most three GPUs. The larger cost is the work around training: building the eval set, calibrating a judge with your experts, and running the loop with engineers who own it.
Should we fine-tune a model or keep using frontier APIs?
Keep frontier APIs for broad reasoning, research and new workloads. Fine-tune or post-train for narrow, repeated tasks where your evals show a gap or the cost is large. Travelers does both: applications call its own insurance model or a frontier model depending on the task.
Will the next frontier model make our own model obsolete?
A trained model is overtaken by newer base models within months. The eval set, judge, training data and decision records keep their value, so keep them independent of any one model, as Nadella and LangChain recommend. A stronger open base model is then a better starting point for your next training run.
