New playbook: when your own AI model pays. Read
Fine-tuning beats frontier API cost on high-volume, well-measured tasks where your own data and expert judgment are the edge.
It comes after corrections are logged as test cases and used for new versions. This page covers the CEO’s decisions, the options, cost drivers, and what Shopify, Bridgewater, Intercom, Kasisto and Heidi Health built.
The learning loop runs in four steps. People correct the agent’s work. Reviewed corrections are kept as eval cases, and every new version must pass them. Once volume justifies it, the same corrections train a model the company owns.
Satya Nadella described the advantage in June 2026 as a private learning loop: private evals that measure outcomes that matter to the business, and private reinforcement-learning environments built on real internal traces. He said a company should be able to swap models without losing that expertise.
The CEO decides which workloads justify an owned model, whether expert corrections, clinician choices and planner overrides are kept as training data, and who controls the model weights.
Shopify’s engineers stated the control point in October 2026: because Shopify controls the post-trained weights, it can serve the model on its own GPUs and decide when it changes.
Shopify’s August 2026 rule: launch on a frontier model, put early improvements into prompts and harness, and move them into weights once a judge is calibrated.
| Option | What you own | Fits when |
|---|---|---|
| Frontier API with prompts | Prompts and harness | The workload is new or low-volume |
| Open model chosen on your evals | Model choice and eval set | You need cost or data control without training |
| Post-trained open model | Weights trained on your data | Volume is high and a judge measures quality |
| Continual learning loop | Weights that improve every cycle | Production failures arrive daily and can be repaired |
Training compute is the smallest cost of post-training. Kasisto reports a training cost of about $1,800 for its banking model (July 2026). The larger costs are expert labels, a judge calibrated against real outcomes, and the engineers who run the loop. Serving volume decides payback: the more traffic, the faster a cheaper owned model repays the work.
Shopify runs a daily loop for one Sidekick sub-agent: a calibrated judge scores conversations, failures are repaired, and repaired examples train a smaller model. Shopify estimates that agent would cost about $27M a year to serve on a frontier model and closer to $1M on its own, with quality above the frontier model on its judge (August 2026).
Bridgewater fine-tuned an open model on investor-labelled data for six daily information-filtering tasks. Bridgewater says it reached 84.7% accuracy against 78.2% for the best frontier model, at 13.8 times lower inference cost per task (June 2026).
Intercom post-trained Apex 2.0 on NVIDIA Nemotron 3 Super and made it Fin’s flagship answering model, live for every Fin customer. Intercom reports hallucinations down 16.5% against Apex 1.0 (October 2026).
Kasisto fine-tuned a 12B banking model to cite sources and say “I don’t know”. Kasisto reports it runs at about $0.001 per query, 20 to 50 times cheaper than GPT-4.1, and scored 6.21 against 5.72 for GPT-4.1 on independent evaluation (July 2026).
Heidi Health post-trains open models on clinician side-by-side choices. Fireworks reported in July 2026 that scribe latency fell from 25 to 7 seconds, and in October 2026 that the post-trained model wins about 54% of live clinician comparisons against the frontier model.
Pick the one workload with the highest volume and a clear definition of correct. Build a judge and calibrate it against expert decisions. Log every failure and its fix for a quarter. Then train an open model on the repairs and compare it with the frontier model on a held-out set.
Other layers: Models and routing · Data and ontology · Agents, traceability and evals
96%
lower estimated serving cost for one agent
A daily continual-learning loop for Sidekick's GraphQL agent: an expert-calibrated LLM judge scores production conversations, a self-healing pipeline repairs failures, and the repaired trajectories are trained into a smaller model (SFT, then GRPO against the judge) that replaces the frontier model (source).
13.8x
lower cost per task, with higher accuracy than the best frontier model
A Qwen3-235B model fine-tuned on Tinker with investor-labelled data to do six daily information-filtering tasks (e.g. which news matters to a macro investor) (source).
16.5%
fewer hallucinations than its previous own model
Apex 2.0, Fin's own customer-service answering model, post-trained on NVIDIA Nemotron 3 Super with Fin's custom post-training system; it is now live for every Fin customer (source).
20-50x
lower cost per query than GPT-4.1
FinRAG-12B, a 12B model fine-tuned for grounded banking Q&A with citations and trained to say 'I don't know' (22% of training examples unanswerable), serving customer queries in production (source).
25 s → 7 s
scribe latency on a post-trained open model
Moved its clinical scribe from closed models to open models post-trained on Fireworks, and post-trained NVIDIA Nemotron 3 Ultra (SFT, then RL with a clinician-informed reward) for its 'Evidence' clinical Q&A product, now in final evaluation ahead of production rollout (source).
1% → 99%
valid structured output after fine-tuning a small model
A proof of value on Red Hat OpenShift AI: GPT-4.1 generated synthetic training data, then small models (Phi-4 Mini 3.8B, Qwen 2.5 1.5B/7B) were fine-tuned for multilingual structured extraction from social media, deployable locally (source).
~7x
faster responses from a post-trained open model
A post-trained NVIDIA Nemotron 3.5 Lightning (30B MoE, 3B active) 'task expert' that took over Sidekick's GraphQL sub-agent from a frontier model, served on Shopify's own GPUs (source).
Own LLM
beat commercial models in internal tests, no figures published
TravelersLLM, a proprietary insurance LLM built by in-house engineers on millions of company documents with underwriting, claims and service experts; applications route each query to it or to a frontier model (source).
When one workload has high volume, a stable definition of correct, and your own data or expert judgment as the edge. Shopify estimates one sub-agent would cost about $27M a year to serve on a frontier model and closer to $1M on its own post-trained model (August 2026). Low-volume or fast-changing workloads stay on frontier APIs.
Continual learning means the model improves from production on a regular cycle. A judge scores live output, failures are repaired, and repaired examples train the next version. Shopify ran six such loops on about 61,000 repaired examples; its judge score rose from 37.7 to 74.5, against 73 for the frontier model (October 2026).
The advantage is your data, judge, expert labels and the weights trained on them. Bridgewater found expert judgment that is hard to put into words is best moved into weights, where its model beat the best frontier model on its own tasks (June 2026).
