A successful AI pilot and a scalable one are not the same thing, and most teams don’t find out which they built until the cost of running it for real users lands on someone’s desk. The clearest warning sign shows up before that, in how the team talks about the pilot: they can describe model accuracy in detail but can’t tell you what it costs to serve a single request. If nobody has asked what it costs to scale, the pilot isn’t ready, no matter how well it performs in the demo.
Every AI pilot gets built to answer one question: does it work. Teams run evaluations, tune prompts, compare model versions, and track accuracy improvements sprint over sprint. By the time a pilot is ready to show leadership, that question has usually been answered thoroughly, sometimes exhaustively.
The question that quietly goes unanswered is what it costs to run this for real, at the actual number of users and requests the business expects once it ships, not a demo with a handful of test cases. That question doesn’t show up naturally in the pilot process, because a pilot’s entire job is to prove the concept works, not to prove it’s affordable at scale. Nobody budgets time to ask it unless someone insists.
Sharma Vedula, Head of Solutions at Data Society, has learned to listen for this gap directly in how a team describes its own pilot:
“Most of the signals aren’t in the infrastructure. They’re in how the team talks about the pilot. The clearest one is when they can describe the model accuracy in detail but can’t tell you what it costs to serve a single request at production volume. They optimized for ‘does it work’ and never asked, ‘What does it cost to work for a million users? What does it cost to scale it up?’ The gap between those two questions is where scaling falls apart.”
Sharma Vedula, Head of Solutions, Data Society
That’s a striking diagnostic, because it doesn’t require an audit of the infrastructure or a deep technical review. It shows up in conversation. A team that’s ready to scale can answer both questions fluently: how well does it perform, and what will it cost when volume goes up by two or three orders of magnitude. A team that’s only ready to demo can usually answer just the first.
This gap is a natural consequence of how pilots get built, not a lack of rigor. A pilot succeeds by proving the model can do the task, so all the incentive during that phase points toward accuracy, quality, and getting a convincing demo in front of stakeholders. Cost at production volume rarely gets a pilot-phase review from most teams; it becomes a “we’ll figure that out later” line item, because the pilot is small enough that the actual dollar figure looks trivial.
That’s exactly what makes it dangerous. A pilot serving a hundred test requests a day can look inexpensive even with an inefficient architecture, verbose prompts, or an oversized model, because the absolute numbers are small. Multiply that same architecture by a million users, and the inefficiencies that were invisible at pilot scale become the entire cost structure. Nobody caught it, because nobody was looking at cost per request while the team was busy proving the concept worked.
A pilot that’s genuinely ready to move to production can answer a short list of questions that go beyond accuracy.
What does it cost to serve one request at expected production volume, not pilot volume. This is the number Vedula points to directly, and it has to be calculated at the target scale, not extrapolated loosely from pilot-era usage.
What happens to that cost as usage grows. A linear relationship between users and cost behaves very differently from one that spikes because of retrieval overhead, multi-step agent calls, or context that grows with each interaction. Knowing which curve you’re on matters more than knowing the number at any single point.
Who owns the cost conversation. If the answer is “nobody yet,” that’s itself a signal. Teams that are ready to scale usually have an engineering or finance stakeholder who can already speak to unit economics, not just model performance.
What tradeoffs were made for the demo that won’t hold at scale. A prompt tuned for a specific set of test cases, a model chosen for accuracy without regard to inference cost, or an architecture that assumed low concurrency are all common shortcuts that work fine in a pilot and become expensive liabilities in production.
The fix is to ask the scaling cost question earlier than teams typically do, running it in parallel with proving accuracy rather than adding heavyweight cost modeling before anyone knows if the idea works. A pilot doesn’t need a finished cost-per-outcome model on day one, but it does need someone asking, from the start, what a successful pilot would cost multiplied by real volume, and revisiting that estimate as the pilot evolves.
Teams that build this habit catch the expensive surprises while they’re still cheap to fix, a prompt that’s too verbose, a model that’s more powerful than the task requires, an architecture that doesn’t degrade gracefully under load. Teams that don’t find out the same things after they’ve already shipped to a million users.
A pilot proves an idea works. It doesn’t automatically prove the idea works affordably at the volume the business actually needs. Data Society’s AI Advisory work helps teams pressure-test a pilot’s scaling cost before it ships, and AI upskilling programs build the habit of asking both questions from the start.
