Most executives can tell you exactly how much their organization spent rolling out an AI tool last year. Far fewer can tell you whether it worked. Not “did people log in,” but did the tool actually save time, cut cost, or improve an outcome that matters to the business. That gap between spend and proof is one of the most persistent problems Donna Medeiros, VP of AI and Data Advisory at Data Society, sees across the enterprises she advises.
The root cause isn’t a lack of enthusiasm for AI. It’s a missing step that happens before the tool ever gets deployed: benchmarking. Without a “before” snapshot, there’s no honest “after” comparison, and every claim about AI’s impact becomes a guess dressed up as a metric.
TRAINING GAPS CREATE RISK BEFORE THEY CREATE ROI PROBLEMS
Before getting to measurement, it’s worth naming why training matters at all. Donna points out that most AI investment skews heavily toward the tools themselves, not the people using them.
“Training is key to AI fluency right now. We see for investment, most of it’s on the tool use. But if the staff is not trained, and contractors, on the tools and the policies, there’s going to be, of course, unauthorized use.”
That’s a compliance and governance risk hiding inside what looks like an adoption problem. When employees and contractors are handed a powerful AI tool with no structured training on what it’s for, what it isn’t for, and what the acceptable use policy actually says, they’ll use it anyway, just without the guardrails anyone intended. This is precisely why governance and training can’t be separated in practice, and it’s a theme that runs through Data Society’s broader work on building a genuine data governance culture rather than a policy that exists only in a compliance binder: https://datasociety.com/data-leadership-collaborative-creating-a-data-governance-culture/
WHY GENERIC, SELF-PACED TRAINING DOESN’T MOVE THE NEEDLE
There’s a specific failure pattern Donna calls out that a lot of executives will recognize: rolling out a tool, pointing employees to a generic course library, and assuming adoption will follow.
“You can’t sit there and release a tool and expect folks to just adopt it unless they’re convinced it’s going to help them in their job, probably help them in their career.”
That’s a statement about human motivation as much as it is about training design. Employees don’t adopt tools because IT deployed them. They adopt tools when they can see, concretely, how the tool changes their day-to-day work for the better, ideally in a way that also strengthens their standing and skills within the organization. Self-paced, one-size-fits-all training rarely gets there, because it isn’t built around any particular role’s actual workflow. It teaches generic tool mechanics disconnected from the specific tasks a marketing analyst, a call center rep, or a financial planner is actually trying to accomplish. That disconnect is a major reason live, instructor-led training tends to outperform self-paced learning for skill retention and real adoption, particularly when the training is anchored in capstone-style projects tied to the learner’s actual job: https://datasociety.com/live-instructor-led-training-vs-self-paced-learning-why-instructor-led-training-is-the-better-option/
THE BENCHMARKING FRAMEWORK MOST COMPANIES SKIP
Here’s the piece that Donna considers foundational, and the piece most organizations get backward.
“There was a framework developed at Gartner that before you implement your tools and literacy training, you benchmark. But if things have already been in use, that’s really hard to go back.”
Read that last sentence carefully, because it’s the part executives most need to hear. If your organization already rolled out an AI tool six months ago without capturing a baseline, you’ve lost the ability to cleanly measure its impact. You can still estimate, survey, and infer, but you can’t produce the kind of clean before-and-after comparison that actually proves value to a board or a CFO. The sequence matters: benchmark current performance on the specific tasks the AI tool is meant to improve, then implement the tool alongside role-specific training, then measure again against that same baseline. Skip step one, and steps two and three produce anecdotes instead of evidence.
This is also where role-specific key performance indicators come in. A benchmark isn’t one company-wide number. It’s a set of measures tied to the actual use case: time saved per task, cost per transaction, resolution speed for a call center team, turnaround time on a marketing report, customer satisfaction scores, adoption rates within a specific department. The KPI for a customer service AI deployment should look nothing like the KPI for a financial forecasting tool, and organizations that try to measure AI success with a single generic metric usually end up with numbers nobody trusts.
PROOF IT WORKS: THE TOYOTA BENCHMARKING EXAMPLE
Skeptical executives often ask whether benchmarking is worth the extra time before a rollout. Donna points to a real example that answers that directly.
“We had a case study at Toyota, but they benchmarked before they put a lot of governance platforms in place and said, this is what it’s taking now. And then they put some tracking mechanisms in place so that they could measure how AI affected things.”
Notice the sequence again: measure current state first, then implement governance and tooling, then track ongoing performance against that original baseline. That’s not a complicated framework. It’s a disciplined one. Most organizations don’t lack the analytical capability to do this. They lack the patience to pause a rollout long enough to capture the “before” picture, because the pressure to show AI progress fast tends to override the pressure to show AI progress accurately. Toyota’s approach demonstrates that the extra weeks spent benchmarking pay for themselves many times over once you actually need to justify the investment or decide where to expand it.
THE CAUTIONARY TALE: WHAT HAPPENS WITHOUT ROI DISCIPLINE
If Toyota shows what benchmarking discipline looks like, Uber’s recent experience shows what happens without it.
“We’ve seen this one in the news lately, Uber. They, as of April of this year, burned through their entire AI budget because of the processing time and usage, token usage costs.”
This is the scenario every executive sponsoring an AI initiative should treat as a warning. Token usage and processing costs can scale in ways that are difficult to predict without careful tracking from day one, and if there’s no benchmark or ongoing measurement discipline in place, budget overruns aren’t caught until the money is already gone. This isn’t an argument against investing in AI. It’s an argument for pairing every dollar of tool investment with a measurement plan robust enough to catch problems before they become budget crises. Data Society’s 2025 AI Readiness Report found that return-on-investment tracking remains one of the weakest links in enterprise AI strategy heading into 2026, which tracks closely with what Donna sees in her advisory work: https://datasociety.com/the-2025-ai-readiness-report-insights-to-build-your-2026-strategy/
WHAT THIS MEANS FOR YOUR NEXT AI ROLLOUT
If your organization is about to launch a new AI tool, resist the urge to skip straight to deployment. Before you write a single line of a training curriculum, define what success looks like for the specific roles that will use the tool, and measure where those roles stand today. Build training that’s tied to that role’s real workflow rather than a generic course, and lean toward live, instructor-led formats built around a genuine project or capstone, since that structure tends to produce far stronger adoption than passive, self-paced content. Then track the same metrics after rollout that you captured before it. That’s the entire discipline. It’s not complicated, but it does require sequencing the work correctly, which is exactly where most organizations go wrong.
FREQUENTLY ASKED QUESTIONS
Measuring AI training effectiveness requires establishing a performance benchmark before the tool and training are deployed, then tracking the same role-specific metrics, such as time saved, cost reduction, or adoption rate, after rollout. Without a pre-deployment baseline, before-and-after comparisons become unreliable estimates rather than verified results.
Effective KPIs are role-specific rather than company-wide, and typically include time saved per task, cost per transaction, adoption rates, resolution speed, and customer satisfaction scores. The right metric depends entirely on the specific use case the AI tool is meant to improve.
Generic self-paced training rarely connects to an employee’s actual daily workflow, so employees aren’t convinced the tool will meaningfully help their job or career. Live, instructor-led training built around role-specific projects tends to drive stronger adoption and retention.
Skipping structured training and governance around AI tools often leads to unauthorized or inconsistent use by staff and contractors, and it removes the measurement discipline needed to catch cost overruns, such as runaway token usage, before they become budget crises.
READY TO PUT REAL NUMBERS BEHIND YOUR AI TRAINING?
If your organization rolled out an AI tool without capturing a baseline, or you’re about to launch one and want to get the sequencing right this time, book time with Donna to build a benchmarking and training plan that proves AI training effectiveness instead of assuming it: https://meetings.hubspot.com/donna-medeiros/meet-with-data-societys-ai-and-data-advisor
