Skip to the stories
Microsoft 365 & SharePoint

Small language models save on paper; the bill arrives after go-live

  • Stephanie Overby
  • 836 views

CIOs running small language models say the savings are real but conditional, with hidden costs in data quality, governance and troubleshooting after go-live.

Small language models save on paper; the bill arrives after go-live

The pitch for small language models has become almost reflexive. AI too expensive? Too slow? Too risky? Try a smaller model. Analyst forecasts back that instinct up. Gartner has predicted that by 2027 organizations will run specialized, task-specific models at roughly three times the volume of general-purpose large language models. In July, Gartner projected that spending on domain-specific and specialized AI models will grow 210% in 2026 alone, more than double the growth rate of foundation models.

Talk to the people actually running SLMs in production, though, and the picture gets messier. Efficiency is there, but not always at the level the business case assumed, and a set of costs tends to surface only after go-live. My read is that the savings are real, just conditional on things most budgets don't yet account for.

Tiered stacks, not wholesale replacement

Ameet Ubhayaker, director of enterprise data analytics and AI at Protiviti, says enterprise SLM adoption has moved from pilots toward mainstream architecture, and that growth is accelerating. He isn't seeing large models get ripped out. What he sees is a tiered stack: small models handling routing, classification and structured output, with frontier models kept for ambiguous, multistep work.

That split matches how Jason James, formerly CIO at Aptos and now CIO at Acumatica, approached the problem. At Aptos he ran three or four specialized models as prototypes on local hardware instead of in the cloud. The point was to measure whether the cost and effort of running them on-premises beat what the company was paying frontier models in token charges.

His test was blunt. "It is really a math exercise," James said. "What is the cost to implement? What is the cost to run? And what is the volume of use? If these are all significantly less to run locally, only then should it be considered. If it's only slightly less, then why bother?"

The numbers had been trending toward the small models, enough that Aptos kept the pilots going. But the SLMs weren't replacing the frontier tools teams already relied on. James described them as extra options, and a way to avoid what he calls "tokenization bankruptcy": the pattern in which poorly trained users burn through a year's token budget in weeks because of weak model selection and prompting.

Ubhayaker places SLMs where the work is narrow, repeatable and easy to verify: classification, routing or structured extraction, especially where the data has real boundaries. "This does not mean SLMs are replacing frontier models," he said. "The destination is a layered architecture, not wholesale migration."

A fine-tuned model that looked fine in testing

Eric Pace, who recently left an enterprise IT leadership role and now advises companies on AI deployment, learned the limits of this approach the hard way. His team deployed a small, fine-tuned Llama model to curate marketing messaging. It was trained on historical data labeled good or bad, testing looked strong, and the path to production was fast and inexpensive, unexpectedly so.

Then the data that had looked reliable in testing didn't hold up at production scale. Pace still considers SLMs a time-saver, with drift and versioning manageable within realistic budgets. But he now treats data quality as a materially bigger issue for fine-tuned SLMs than for general AI use.

"Not only is there a need to ensure that the data sets represent what you want as inputs to the model, but you have to find the line," he said. "Overfitting is real." His advice is to keep the task contained and specific, plan for back testing, and not underestimate the business knowledge and data stewardship that teams outside IT have to supply.

Ubhayaker has seen a related mistake repeat: companies evaluating SLMs against public benchmarks rather than their own production traffic and data. A model that scores well on a leaderboard can still stumble on the messy inputs your users actually send.

Cost per successful task, not cost per token

Ubhayaker's core advice on cost is to measure cost per successful task rather than cost per token. "A smaller model may appear cheaper, but retries, human review and escalation to larger models can quickly erase those savings," he said.

That framing changes the arithmetic. A cheap model that fails often and sends work back to a person or a frontier model can cost more than the expensive model it replaced. James's view is that model cost isn't total cost. Budget for governance, management overhead and training before you tally any savings.

Fleet overhead and drift

Running several specialized models adds a different kind of load. James points to troubleshooting: when one workflow passes through multiple specialized models, tracing a failure means working through several model owners to find where the problem started and who owns it. Ownership gets blurry fast.

Ubhayaker says every specialized model needs its own evaluation and monitoring to catch behavioral drift. A fleet adds complexity around routing, upgrades and lineage. Base model updates and downstream fine-tunes force retraining and re-benchmarking, which many budgets don't include.

The bigger risk, in his experience, is what he calls a "confidence-maturity gap." Leadership assumes monitoring policies are being followed, while telemetry shows they often aren't. That gap widens when teams deploy specialized models faster than the organization can catalogue and govern them. If you're running more than a handful, that's worth checking against what's actually happening in your logs, not what the policy document says.

Where that leaves the decision

None of this makes SLMs a bad bet. They just demand more upfront diligence than many teams expect. Ubhayaker frames the choice as a per-use-case call, with a few deciding questions: is the task bounded and verifiable, does the organization have the engineering capacity to maintain the model, and can a smaller model hit the required accuracy on your data? Which architecture is cheaper on paper matters far less than those answers.

The value of SLMs, in short, depends on the task, the data and who owns the model after it ships. For most teams, that last question is the one still missing from the business case.

Source: Stephanie Overby · www.informationweek.com

Neighbouring cells