The 80/20 Problem With AI Agents, and Why Domain Experts Are the Fix

The AI agents that actually work are built by pairing engineers with the people who’ve done the underlying job for years, not by technical teams alone. Agents are consistently good at the 80% of a workflow that’s rote and standardized. What separates the ones that hold up in production from the ones that quietly fail is whether anyone thought through the other 20%, or 10%, or 5%, the fringe cases that don’t follow the pattern, and built in a way for the agent to recognize them and escalate rather than guess.

Ask most organizations who built their AI agent, and the answer is usually “the tech team.” That’s necessary, but it isn’t enough on its own.

That’s the distinction Merav Yuravlivker, Chief Learning Officer and Co-Founder of Data Society, draws from watching agent deployments across industries:
“The most effective agents that I’ve seen are created by people who’ve already been doing the work for years. These are people who are experts in their field, who understand all of the nuances that need to be in place in order to make a process and a workflow successful.”
Merav Yuravlivker, Chief Learning Officer and Co-Founder, Data Society

The nuance she’s pointing to is exactly what’s hardest to encode after the fact. A technical team can build an agent that follows a documented process. Only someone who’s actually done the job knows where that process quietly breaks down, the exception nobody wrote a policy for, the judgment call that depends on context a flowchart can’t capture.

This isn’t a hypothetical risk. MIT’s NANDA research, tracking 300 public generative AI deployments, found that 95% of generative AI pilots fail to deliver measurable business impact. And a common failure pattern in production agent deployments traces back to the agent’s tool layer, not the underlying model: the agent calls a tool incorrectly, loops on retries, or escalates a transient failure into a cascade. Those are exactly the kind of fringe-case failures that show up when nobody with deep process knowledge was involved in defining what “correct” looks like at the edges.

The reason this is easy to miss during a pilot is that agents perform well on exactly the cases you’d think to test. Yuravlivker names the pattern directly:
“One of the biggest pieces of friction I see when it comes to deploying these agents is that they’re good for the 80% of tasks that are rote and standardized, but you have 20% of cases, maybe it’s 5% or 10%, that sit outside of those lines.”

An agent that handles 80 or 90% of a workflow flawlessly looks like a success in a demo. It’s only in production, at volume, that the remaining slice starts costing real money and real trust. That gap matters more than the percentage suggests: customer-facing failures compound fast. 75% of consumers say they’ve had a fast AI response that still left them frustrated, per a December 2025 Glance survey, and 79% say they’d switch companies after just one bad experience, per a 2026 Verint survey.
  
Yuravlivker frames the fix as a set of specific questions that need answers before an agent ships, not after it breaks:
“How does an agent react when something comes in that’s outside the normal parameters? Does it have instructions to alert a particular person or a particular team? And how do you make sure you put those gates and those checks in place so you can handle those edge cases?”
Those questions are hard to answer from a technical spec alone, because they require knowing what “outside the normal parameters” actually looks like in a given workflow. That’s domain knowledge, not engineering knowledge. A subject matter expert working alongside the technical team is what turns “we’re not sure what happens on weird inputs” into a documented escalation path: which fringe cases exist, who gets alerted, and what the agent should do while waiting for a human to weigh in.

Teams with pre-written runbooks and kill-switches for agent failures catch problems fast and contain them before they spread. Teams without that preparation are improvising in the moment, and it shows in how long the same category of problem takes to resolve. Whether anyone thought through the exception path in advance is what separates the two, not the underlying technology.
  
Yuravlivker is direct about the cost of skipping this step:
“As you’re thinking about building and deploying, you want to make sure to have those pieces in place, because otherwise you’re still going to have to check the agent’s work much more closely than you would otherwise.”
That’s the quiet tax organizations pay for deploying agents without domain expertise built in: not necessarily a dramatic failure, but a permanent need for closer human oversight than the agent was supposed to eliminate in the first place. The efficiency gain that justified the project in the first place gets eaten by the review process it was supposed to replace.
  
In practice, pairing a subject matter expert with a technical team means:
Naming an owner before development starts. Someone who has done the actual job, not just a stakeholder who signs off at the end, should be in charge of or consulting on the agent from the start, not brought in to review it once it’s built.
Mapping the fringe cases explicitly. Before deployment, the expert and the technical team should walk through what falls outside the standardized 80%, and agree on what counts as an edge case worth flagging versus one the agent can handle.
Defining the escalation path for each one. For every identified edge case, there should be a clear answer to who gets alerted, what team owns it, and what the agent does in the meantime, not just “it will probably figure it out.”
Building in the gates and checks as part of the design, not a patch after launch. The checks Yuravlivker describes work best when they’re part of the agent’s original architecture, not bolted on after the first production failure exposes the gap.
  
The technology to build an AI agent is now widely available. The judgment to know where that agent’s confidence should end and a human’s should begin still comes from people who’ve done the work. Data Society’s AI Advisory work helps organizations pair domain experts with technical teams before deployment, and AI upskilling programs build that judgment across the organization.