The AI strategy conversation for small and midsize enterprises almost always arrives at the same impasse. The enterprise technology vendors want to sell a complete transformation. The implementation consultants talk about building modern data infrastructure from scratch. And the SME leaders who are actually responsible for keeping the business running look at the price tag and the timeline and decide that AI can wait.
The premise is wrong. SMEs do not need to rebuild their data infrastructure to start building AI capability. They need a strategy that works with what they have, builds incrementally, and does not require betting the operating budget on a single transformation initiative.
Sharma Vedula, Head of Solutions at Data Society, describes the approach directly:
“You don’t rip and replace, you build integration layers that let the new and the old talk to each other while you gradually migrate.”
That is not a workaround. It is a sound architectural principle, and it is specifically well-suited to organizations that have real operational constraints and cannot afford extended periods of instability.
Why The Rip-And-Replace Approach Fails SMEs
Large enterprises sometimes have the capital and the organizational capacity to absorb a wholesale infrastructure replacement. Most small and midsize enterprises do not. A full system replacement requires extended periods where legacy processes and new systems have to coexist awkwardly, staff who are managing both the transition and the daily operations, and significant tolerance for disruption.
More importantly, rip-and-replace assumes that the destination is clearly understood before the journey begins. For most SMEs, the AI use cases that will actually produce value only become clear after the organization has started working with the data and the tools. Building a comprehensive new infrastructure before those use cases are identified is a significant investment in a direction that may need to change.
The modular approach allows SMEs to start with the use cases that have the clearest value, build the infrastructure required to support those specific cases, and expand from there as the picture becomes clearer.
Related reading: Why 85% of AI Projects Fail: and How to Make Sure Yours Doesn’t
https://datasociety.com/why-85-of-ai-projects-fail-and-how-to-make-sure-yours-doesnt/
What Modular AI Architecture Looks Like In Practice
A modular AI strategy for SMEs starts with mapping what already exists. Not what the org chart says should exist, but what actually exists. Legacy systems are usually more tangled than anyone admits. Data flows live in spreadsheets, in people’s heads, and in decade-old databases that nobody wants to touch.
Vedula describes the first step in every advisory engagement:
“The first thing we do is to map what exists. Not what the org chart says it exists. Typically legacy systems are usually more tangled than anyone admits.”
Once the actual state of the data and systems is understood, the strategy becomes a sequencing question: what do you stabilize first, and what do you modernize in parallel? Integration layers, connectors that allow new AI systems to work with existing data sources without requiring those sources to be replaced or rebuilt, are the practical mechanism.
The infrastructure build is designed for flexibility from the start. As Vedula puts it:
“The infrastructure has to have change baked into it from the start. If it can only handle today’s use case, it’s already behind.”
This means that even when the initial use case is narrow, the way it is built should not create technical debt that makes the next use case harder to add. Choosing integration patterns that are extensible, data schemas that can evolve, and monitoring systems that scale with volume are the architectural decisions that make the modular approach viable over time.
Learn more: Data Society AI Advisory Services
https://datasociety.com/data-and-ai-advisory-services/
The City Of Dallas Model: Meeting Organizations Where They Are
Data Society’s work with the City of Dallas offers a practical illustration of modular AI strategy at organizational scale. The city needed to modernize how 42 city departments worked with data, across teams with very different systems, different maturity levels, and different user needs.
The answer was not to force everyone onto a single platform. It was to build toward a shared data vocabulary and capability baseline that could flex as each department evolved. Individual departments moved at the pace their infrastructure and workforce could support, while the overall architecture was designed to allow them to connect and share as they developed.
The principle for SMEs is the same, even at a different scale. Different parts of the business will have different readiness levels for AI. A modular strategy meets each part where it is rather than requiring the whole organization to be ready at the same time. Vedula describes the goal:
“Meet people where they are, build the connective tissue and design for the next two years, not just this quarter.”
That phrase, “design for the next two years, not just this quarter,” is the core of the modular approach. The decisions you make in the first phase of AI implementation create constraints or create options for what you can do next. Building modularly means building in a way that keeps options open.
Selecting Technology For An SME AI Strategy
One of the most consequential decisions in any AI strategy for small and midsize enterprises is technology selection: open-source versus proprietary APIs, cloud versus on-premises, build versus buy. Vedula has a consistent response to how organizations should approach this question:
“The right question is what does this organization actually need to own and what do they need to control?”
Proprietary APIs are fast to deploy and well-supported, but they introduce dependency risk, cost variability, and limited visibility into what is happening inside the model. For many SME workflows, that trade-off is acceptable. For others, it becomes a constraint that is difficult to reverse once the infrastructure is built around it.
Open-source models offer more control, but they require the team to manage them, the compute to run them, and the expertise to troubleshoot them. For SMEs without a dedicated data science team, this overhead can be prohibitive. The technology choice should follow the operational reality, not precede it.
Explore Data Society’s AI and data solutions: https://datasociety.com/solutions/
Frequently Asked Questions
An AI strategy for SMEs is a sequenced plan for building AI capability that is matched to the organization’s current infrastructure, budget, and workforce readiness. It prioritizes high-value use cases that can be implemented without wholesale system replacement, uses a modular architecture that can expand over time, and builds the workforce capability needed for adoption alongside the technical implementation.
Start by mapping what actually exists: data, systems, and workflows. Identify one or two use cases where AI could produce clear value with the data and infrastructure already in place. Build the integration layer for those specific cases and measure the results before expanding. The first phase does not need to be large to be valuable.
The answer depends on what the organization needs to own and control. Proprietary APIs are faster to deploy and require less internal expertise, but they introduce dependency risk and cost variability. Open-source models offer more control but require the team and compute to manage them. Most SMEs start with proprietary APIs for initial use cases and move to more control as their internal capability grows.
A modular AI architecture is built from components that can be added, removed, or replaced without requiring a complete system rebuild. It uses integration layers to connect new AI capabilities to existing data sources and workflows, rather than requiring legacy systems to be replaced before AI can be deployed. This approach allows organizations to start small, validate use cases, and expand incrementally.
The timeline depends on the complexity of the existing systems and the scope of the initial use cases. A focused first phase, building one or two AI capabilities on top of existing infrastructure with proper integration layers, can be completed in weeks to months. Building toward a more comprehensive architecture that can support multiple AI use cases typically takes one to two years of iterative development.
Build the Foundation That Makes Scale Possible
SMEs that are waiting for the right moment to start building AI capability are already falling behind the organizations that started with a modest, modular first step. You do not need to replace your infrastructure to start. You need a strategy built around what you actually have, designed to grow with you.
Data Society works with small and midsize enterprises to build AI strategies that are matched to real operational constraints. Talk to our team about where to start: https://datasociety.com/contact/
