13.8x
lower cost per task, with higher accuracy than the best frontier model
A Qwen3-235B model fine-tuned on Tinker with investor-labelled data to do six daily information-filtering tasks (e.g. which news matters to a macro investor) (source).
Shopify: 96% lower estimated serving cost for one agent
Shopify (commerce software, ~8,000 employees) reported this on 5 Aug 2026 in company self-report (source). It belongs to the learning loop layer.
What they built
A daily continual-learning loop for Sidekick's GraphQL agent: an expert-calibrated LLM judge scores production conversations, a self-healing pipeline repairs failures, and the repaired trajectories are trained into a smaller model (SFT, then GRPO against the judge) that replaces the frontier model.
The result
Estimated serving cost fell from ~$27M/year on a frontier model to ~$1M/year (-96%); quality surpassed the frontier baseline on Shopify's judge; end-to-end latency -38% and time-to-first-token -19% at 350 req/min; ~14% fewer GPUs for the same traffic.
The number covers one Sidekick sub-agent (GraphQL data retrieval) for all merchants; cost figure is Shopify's own estimate from average token prices.
“How we compress production failures into model weights every day, beat frontier-model quality, and cut serving costs 96%,” Shopify said.
The layer: The learning loop
Reviewed corrections are kept as test cases and later used to train models.
- A log of every correction and who reviewed it
- Eval sets that grow from reviewed corrections
- New versions, then post-trained models, from that signal
Read the guide to the learning loop.
Playbooks that use this case
More cases on the learning loop
1% → 99%
valid structured output after fine-tuning a small model
A proof of value on Red Hat OpenShift AI: GPT-4.1 generated synthetic training data, then small models (Phi-4 Mini 3.8B, Qwen 2.5 1.5B/7B) were fine-tuned for multilingual structured extraction from social media, deployable locally (source).
Own LLM
beat commercial models in internal tests, no figures published
TravelersLLM, a proprietary insurance LLM built by in-house engineers on millions of company documents with underwriting, claims and service experts; applications route each query to it or to a frontier model (source).