Why Good Policies Fail When They Treat Evidence and Change as Separate Problems

The hidden mistake in most policymaking

What if the biggest reason policies fail is not that we lack data, but that we ask data to do the wrong job?

That sounds counterintuitive because policy conversations are usually framed as a contest between more evidence and less evidence. Yet the deeper problem is not scarcity. It is category error. We often collect data as if the world were a static object to be measured, then design theories of change as if the world were a living system to be nudged. Those are not the same task. One tells us what is happening. The other tells us how something might happen. When they are disconnected, even the most sophisticated program can become a beautifully documented guess.

The tension is simple to state and hard to resolve: evidence asks what is true, while theory of change asks what might become true. Policymaking sits in the narrow space between them. If we overtrust numbers, we mistake measurement for understanding. If we overtrust plans, we mistake aspiration for causation. Real judgment begins when we treat data and theory not as rivals, but as partners in a disciplined conversation.

---

Data does not speak for itself, and plans do not survive contact with reality

A policy team can have mountains of information and still misunderstand the situation. Surveys may show rising enrollment, but they cannot tell whether students are learning. Administrative records may show that services were delivered, but not whether they were received as intended. Interviews may reveal frustrations that never appear in dashboards. Each of these sources carries partial truth, but none is complete. That is why the question is never simply, “Do we have data?” The better question is, “What kind of reality does this data illuminate, and what does it leave in shadow?”

Theories of change can fall into the same trap. They often look elegant on paper: if we provide training, then skills improve; if skills improve, then outcomes improve; if outcomes improve, then lives improve. The logic is plausible, but plausibility is not proof. A theory of change is not a crystal ball. It is a structured hypothesis about how change might unfold under specific conditions. In that sense, it should be treated less like a mission statement and more like a scientific model, one that must be revised when the world refuses to behave as expected.

The real task of policymaking is not to choose between evidence and theory. It is to make them interrogate each other.

Consider a workforce program designed to help unemployed adults find stable work. The theory of change might say that job coaching leads to better applications, which leads to interviews, which leads to employment. But evaluation data could reveal a different bottleneck: participants do improve their applications, yet still fail because transportation is unreliable, childcare is unstable, or local employers filter out applicants by automated systems. Here, the theory did not fail because it was irrational. It failed because it was incomplete. The data did not merely measure outcomes. It exposed the missing mechanisms.

This is where the real value of triangulation appears. Not as a bureaucratic checklist, but as an epistemic discipline. Multiple kinds of data can catch one another’s blind spots. Quantitative data may show scale; qualitative data may show meaning; contextual data may show constraints; participatory data may show legitimacy. Together they do something no single source can do alone: they reduce the risk of being seduced by a narrow slice of reality.

---

The best theory of change is not a prediction, it is a map of uncertainty

Most people hear “theory of change” and imagine a polished pathway from inputs to outputs to outcomes. But the most useful version of a theory of change is not a neat arrow diagram. It is a map of assumptions, dependencies, and vulnerabilities. In other words, it tells you where the program is most likely to break.

That changes how we should use it. Instead of asking, “Does the plan look reasonable?” ask:

1. Which links in the chain are assumptions, not facts?
2. What evidence would prove each link weak or strong?
3. Which conditions must be present for the pathway to work at all?
4. What would we expect to see if the theory were wrong?

This makes theory of change more than a design tool. It becomes an evaluation tool, because it tells you what to look for. It also becomes a governance tool, because it clarifies where decision makers should pay attention when reality shifts.

Think of it like a bridge inspection. The point is not to admire the bridge’s blueprint. The point is to identify the load-bearing beams, inspect the joints, and ask what happens when weather, traffic, or wear changes the structure. Policies are no different. A program may look strong from a distance, but the crucial question is whether its weak points have been identified before the first shock arrives.

This is why good policymaking requires a certain humility. A theory of change should not be defended like a doctrine. It should be stress-tested like an engineering model. Data becomes the force that applies pressure to the model. When the model bends, the policy learns. When it snaps, the policy should not cling to elegance. It should adapt.

---

A better model: policies as hypotheses under revision

The most useful synthesis of evidence and theory is to treat every policy as a living hypothesis.

This framing has three advantages. First, it removes the illusion of finality. Policies are not sacred plans. They are provisional bets about how to move the world. Second, it makes evidence more actionable. Data is not gathered for reporting theater, but to test specific claims embedded in the intervention. Third, it encourages iteration. If a program is a hypothesis, then evaluation is not judgment day. It is learning.

This matters because many policy failures come from confusing implementation with validation. A program can be implemented faithfully and still fail because the theory was wrong. It can also appear to fail because the implementation was too weak to test the theory fairly. Without good evidence, these two cases are easy to confuse. That is one reason triangulation matters so much. It helps distinguish theory failure from delivery failure.

Imagine a public health campaign promoting vaccination. Administrative data might show low uptake. Survey data might show confusion about access. Focus groups might reveal fear of side effects. Community partner observations might reveal distrust toward institutions. Each source alone points to a different slice of the problem. But together they may reveal the true bottleneck: not persuasion, but legitimacy. The policy’s theory of change may have assumed that better information would drive behavior. The data shows that information is necessary, yet not sufficient, because trust is the real precondition.

That is the deeper lesson. Good policy is not just evidence based. It is evidence constrained by a realistic theory of human behavior and social context. People do not respond to interventions as isolated rational agents. They respond through institutions, relationships, incentives, histories, and identities. A theory of change that ignores this complexity becomes a tidy fiction. Data, when used well, keeps that fiction from hardening into policy.

---

From measurement to sensemaking: what organizations should actually do

The practical implication is that evaluation should look less like an audit and more like a conversation with reality. That means shifting from asking whether a program hit its targets to asking whether the target was derived from a credible causal story in the first place.

Here is a useful mental model: think of policymaking as operating on three layers.

1. The surface layer: indicators

These are the numbers and observations we track. Enrollment rates, completion rates, service coverage, satisfaction scores, time to delivery.

2. The mechanism layer: causal pathways

These are the links that explain why the indicators should move. Training increases confidence. Access reduces friction. Cash transfers change household tradeoffs. Community outreach changes trust.

3. The context layer: conditions of success

These are the external realities that can strengthen or undermine the pathway. Labor markets, norms, geography, institutional trust, digital access, political stability.

Most evaluation failures happen when organizations stay stuck on the first layer. They track indicators because they are visible and reportable, but they never seriously test the mechanism or context layers. Then, when a program underperforms, everyone debates the numbers without asking whether the causal story itself was too thin.

A stronger approach is to ask what data belongs at each layer.

• Use quantitative data to detect patterns, trends, and magnitude.
• Use qualitative data to understand mechanisms, meanings, and lived experience.
• Use contextual and administrative data to identify constraints and system effects.
• Use participatory data to check whether the intervention makes sense to the people it is meant to serve.

The point is not to create data abundance for its own sake. The point is to build epistemic redundancy, so that no single blind spot can dominate the policy narrative.

When the same conclusion survives different kinds of inquiry, confidence grows. When the sources disagree, the disagreement is often the most important finding.

That last point is easy to miss. In many organizations, inconsistent findings are treated as a problem to be smoothed over. In reality, inconsistency is often where the learning lives. If the survey says one thing and the field interviews say another, the answer may be that the program is experienced differently across groups. If the administrative data is clean but the community story is negative, the issue may be that compliance is rising while trust is collapsing. Triangulation is not about averaging away complexity. It is about identifying it.

---

Key Takeaways

1. Do not treat data as proof and theory as decoration. Data tests a causal story; theory gives data something meaningful to test.
2. Write theories of change as maps of assumptions, not as promotional diagrams. The most valuable part is identifying where the pathway could fail.
3. Use multiple data sources to expose different kinds of truth. Numbers, narratives, administrative records, and community feedback each reveal different layers of the problem.
4. Look for disagreement between sources. Contradictions are not noise to suppress, they are clues to hidden mechanisms or context effects.
5. Treat policy as a living hypothesis. The goal is not to defend the first plan, but to learn quickly enough to improve it.

---

The policy mindset we actually need

The deepest mistake in policymaking is not lack of intelligence. It is overconfidence in a single mode of knowing. We want data to certify our ideas and theories to preserve our hopes. But the world is not obligated to cooperate with our categories. A policy succeeds when it is designed for revision from the start.

That is why the union of triangulated evidence and theory of change is so powerful. It forces us to think in two directions at once: backward toward causation and forward toward action. It asks not only, “What happened?” but also, “Why did it happen, under what conditions, and what would have to be true for it to happen again?” Those are not technical questions alone. They are questions of judgment, humility, and institutional maturity.

In the end, the goal is not to make policies more certain. It is to make them more honest about uncertainty. A policy that admits what it does not know, tests its assumptions, and updates its beliefs is not weak. It is the only kind that can learn in a complicated world.

That is the real shift: from seeing evidence as a verdict to seeing it as a conversation, and from seeing theories of change as plans to seeing them as propositions about reality. Once you make that shift, policymaking stops being an exercise in defending certainty and starts becoming an engine for discovering what actually works.