Why Token-Based AI Pricing Breaks the Old Cloud Cost Model

The old cloud cost model worked because compute was priced by the hour, predictable, and easy to tie to a workload. AI breaks that. Token-based pricing means your costs are a function of how people actually use the system, not a fixed rate you set in advance. The fix is designing for cost per outcome before you ship, not patching a token dashboard after the bill arrives.
  
For most of the cloud era, cost planning was straightforward. You provisioned a server or a cluster, paid for the compute by the hour, and could tie that number directly to a workload. Run the job for two hours on a given instance type, and the cost was whatever the math said it would be, regardless of what the job actually did once it started running. Finance could forecast it. Engineering could budget for it. Nobody was surprised by the invoice.

That model assumed something important: cost tracked provisioning, not usage. Whether a request was simple or complex, the meter ran the same way. AI pricing doesn’t work like that.

With AI, the unit of cost isn’t compute time. It’s tokens, the chunks of text a model reads and generates, and the number of tokens a request consumes depends entirely on what the person on the other end does with it. That single shift changes the economics of the whole system.

Sharma Vedula, Head of Solutions at Data Society, lays out exactly why this trips companies up:

“The old cloud cost model was simple. You pay for the compute by the hour, you know the cost, and you tie it to a workload. With AI, that breaks down really fast, because AI is based on token-based pricing, which means your costs are a function of how people use the system. A user who asks a long, complex question costs more than one who asks something simple. A poorly written prompt that forces a long response costs more than a tight one. If a product has an open-ended input field and you haven’t really thought through it, you’re going to get surprised by your bill.”
Sharma Vedula, Head of Solutions, Data Society

What’s actually driving cost in that description is the behavior of the person typing into the box, not the provider’s rate card. A support chatbot that lets users write freely will see wildly different costs from one conversation to the next, depending on how verbose the user is, how many follow-up questions they ask, and how much context the model has to hold to answer well. None of that is under an engineering team’s direct control once the product ships, which is exactly why it’s so easy to get blindsided.

The pattern shows up in a few predictable places, and it’s worth naming them plainly, because each one is a design decision, not an accident.

Verbose users. A long, multi-part question costs more to process and more to answer than a short, specific one. Multiply that across thousands of users and the variance in a single day’s spend can be enormous, even if average usage looks stable on a monthly chart.

Sloppy prompts. A prompt template that doesn’t constrain the model’s response length invites long, rambling answers. The team that wrote the prompt sees it as a small implementation detail. Finance sees it as a token bill that doesn’t match the traffic numbers.

Open-ended input fields. A free-text box is the single biggest lever on AI cost that most product teams don’t realize they’re pulling. Every additional word a user is invited to type is a cost the product owner didn’t explicitly choose, and if nobody’s thought through what happens when someone pastes in three paragraphs, the bill will eventually explain it for them.
  
The instinctive response is to watch the token dashboard more closely and optimize after the fact: cap output length here, tune a prompt there, add a rate limit somewhere else. That helps at the margins, but it’s a retrofit, and retrofits are expensive in a way that’s easy to underestimate. By the time a cost problem is visible on a dashboard, it’s already been shipped to every user of the product, and unwinding it means changing behavior people have gotten used to.
  
The better move is the one Vedula recommends: treat cost per outcome as a design constraint from day one, not a metric you retrofit after launch.

“What companies need to measure instead is cost per outcome, not cost per token: cost per the thing the business actually cares about. Cost per resolved ticket, cost per processed document, or even cost per completed workflow. Once you have that number, you can have a real conversation about where the system is wasteful and where it’s delivering value. So what we’d recommend is building it into your product design from the start, rather than retrofitting it after you’ve shipped and finding that your costs have gone through the roof.”

That means asking, before a feature ships, what a single output-length change might cost the business, and where the input field itself is a cost decision. A product team that knows its target cost per resolved ticket can design prompt constraints, response formats, and escalation paths around that number from the beginning, instead of discovering the number is too high only after finance flags the spend.
  
A few concrete habits separate teams that get surprised by AI costs from teams that don’t.

Define the outcome unit before writing the first prompt. Decide what “resolved,” “processed,” or “completed” means for the workflow, and design toward a target cost for that unit, not toward a token budget in the abstract.

Put real constraints on open-ended inputs. That means being deliberate without stripping away flexibility users need: character limits where they make sense, prompts that push the model toward concise answers by default, and clear signals to users about what a good, specific question looks like.

Model the cost of your worst-case user, not your average one. A cost model built only on typical usage will always be broken by the outlier who writes paragraphs or asks the model to loop through a long back-and-forth. Design and price for that case explicitly.

Treat pricing changes from providers as a design trigger, not just a finance line item. When token pricing shifts, the workflows most exposed to verbose or unpredictable input are the ones worth revisiting first.

Teams that keep applying the old pay-for-provisioning playbook to AI get surprised by the bill. Data Society’s AI Advisory work helps product and engineering teams design for cost per outcome from the first prompt, and AI upskilling programs build that fluency across product, engineering, and finance.