Why Surrogates Fail Unless You Treat Features Like Causal Paths

May 15, 2026

11 min read

8 views

86%

The hidden mistake in prediction: confusing a proxy with a pathway

What if the real problem in forecasting long term outcomes is not that we lack enough data, but that we keep choosing the wrong variables to trust?

In many settings, the thing we ultimately care about arrives late. Earnings take years to observe. Health outcomes unfold slowly. Retention, learning, and behavior often reveal themselves only after the experiment is over. That delay tempts us into a familiar shortcut: use early indicators as stand ins for the final outcome. If early employment rates predict nine year earnings, why not just forecast from them? If a model can predict well, why not use its features as evidence of impact?

The danger is subtle. A variable can be a good predictor without being a good surrogate. It can compress information about the future while still failing to preserve the causal effect of a treatment. This is the central tension: prediction cares about correlation, causal inference cares about pathway coverage. If the features do not span the causal routes from intervention to outcome, they may improve accuracy while quietly distorting the effect you are trying to estimate.

That is why the most useful way to think about surrogates is not as shortcuts, but as a theory of what the treatment actually changes.

A surrogate is not just a feature, it is a claim about the world

A common instinct is to say: if a short term outcome is highly predictive of a long term outcome, then it should be safe to substitute one for the other. But that instinct misses the deeper requirement. A valid surrogate is not merely informative. It must be sufficient, in a causal sense, for the effect of treatment on the final outcome.

In plain language, the long term outcome should be independent of the treatment once you condition on the surrogate. That is a strong statement. It means the surrogate has absorbed the relevant channels through which the treatment works. If treatment raises earnings by improving employment stability, and your surrogate captures employment stability, then the surrogate may be enough. If treatment also changes social networks, confidence, or access to informal opportunities, and your surrogate misses those, then the estimate can be biased even if the surrogate predicts earnings beautifully.

This distinction matters because we often mistake a good summary for a complete summary.

A feature can be highly predictive and still be causally incomplete.

Think of a thermometer versus a weather model. Temperature may tell you a lot about tomorrow's weather, but it does not encode wind, pressure, humidity, or the dynamics that produce storms. If a policy affects the weather system through multiple channels, then a single temperature reading is not enough to capture the effect. Similarly, in causal work, the question is not whether a surrogate predicts the outcome. The question is whether it captures the entire causal mechanism relevant to the treatment.

This is where the language of surrogate indexes becomes powerful. Instead of relying on one indicator, you build a composite index from intermediate outcomes. In a linear setting, this looks like a weighted prediction. More generally, it is the conditional expectation of the primary outcome given the surrogates and pre treatment covariates. The index is not interesting because it is a clever forecast. It is interesting because it is supposed to be the causal shadow of the missing outcome.

Three assumptions, one fragile bridge

The bridge from short term signals to long term impacts rests on three conditions.

Unconfoundedness: treatment assignment is as good as random after adjusting for observed pre treatment variables.
Surrogacy: once you condition on the surrogates and covariates, treatment no longer has a direct effect on the final outcome.
Comparability: the relationship between surrogates and the primary outcome is the same in the experimental and observational samples.

These conditions are not just technicalities. They are three different ways the bridge can collapse.

Unconfoundedness says the treatment itself is cleanly identified. Surrogacy says the intermediate outcomes really do contain the full causal signal. Comparability says the mapping from surrogates to outcomes learned in one sample can be transported to another. If any one of these fails, the surrogate index can lead you astray.

This is why the two sample setup is so illuminating. One sample identifies how treatment changes the surrogates. Another sample identifies how surrogates relate to the final outcome. Put together, they can reveal the long term effect without waiting years. But only if the bridge between the two samples is stable and the surrogates truly span the pathway.

The deeper lesson is that causal estimation is often an exercise in translating between worlds. The experimental sample tells you what treatment does. The observational sample tells you what outcomes mean. The surrogate index is the translation layer. If the translation is lossy, your estimate is no longer the object you think it is.

Why predictive power and causal sufficiency are different engineering problems

This tension has a surprising cousin in machine learning feature engineering, especially in tree based models. In boosted trees, we often obsess over feature encoding, interactions, and transformations because the raw variable may not expose the signal in a usable form. A categorical variable can be embedded, one hot encoded, target encoded, or split in different ways. Interaction terms can reveal structure that single features miss. The point is not merely to improve fit. The point is to make the relevant structure visible to the learner.

That instinct is useful, but it can also mislead. In prediction, a feature is valuable if it helps reduce error. In causal inference, a feature is valuable only if it helps preserve the relevant mechanism. These are related but distinct engineering goals.

Here is a useful mental model:

Prediction asks: what variables help approximate the outcome?
Causal estimation asks: what variables preserve the effect of treatment on the outcome?

A boosted tree can exploit proxies aggressively. It is designed to find splits that maximize predictive gain. But a highly predictive split can be a terrible surrogate if it picks up downstream consequences rather than upstream causal channels. For example, a model may use early job search activity to predict later earnings. But if treatment affects earnings through a channel that does not show up in job search, the feature set is incomplete for causal purposes even if it is excellent for forecasting.

This is why surrogate selection should not be treated like ordinary feature selection. It is not enough to ask whether a variable is correlated with the outcome. You must ask whether it is correlated with the treatment and whether, collectively, the selected variables can reconstruct the causal mechanism. In other words, good surrogate design is feature engineering under a causal contract.

In prediction, missing a weak feature may cost accuracy. In causal inference, missing a pathway can create bias.

That is a much harsher standard.

Bias is not all or nothing, it is governed by what your surrogates miss

When surrogacy fails, the resulting bias is not random noise. It has structure. The size of the bias depends on two things: how much of the treatment effect runs through the observed surrogates, and how much residual variation the surrogates leave in the treatment and outcome.

This gives a practical intuition. If a surrogate almost perfectly predicts the treatment or almost perfectly predicts the outcome, then the room for bias shrinks. The more completely your surrogates explain the relevant variation, the less damage a violation can do. But if they only explain a thin slice of the causal story, then even small departures from surrogacy can matter a lot.

This is where bounds become more than a theoretical comfort. They are a way to say: even if the surrogate is imperfect, how wrong could we plausibly be? If the surrogate leaves a lot of unexplained variation in the primary outcome, the estimate can be highly sensitive. If the surrogate captures most of the variation, the estimate is more robust.

The important conceptual shift is that bias is not an abstract specter. It is the cost of omitted pathways.

You can think of this like a map. A map is useful only to the extent that it contains the roads you need. If it omits the highway, you can still navigate the neighborhood, but you cannot trust it for a cross country trip. The surrogate index is the map. The omitted causal pathways are the missing roads. The bias is the detour you were never warned about.

The most valuable surrogate is the one you can validate over time

One of the smartest ideas in this area is temporal validation. If the long term outcome is not yet observed, you can still ask whether an early surrogate index tracks experimental effects as time progresses. That turns surrogacy from a static assumption into a dynamic test.

Suppose the first six quarters of employment data are used to predict nine year employment. If the surrogate index built from those early quarters closely matches the eventual treatment effect, that is evidence, not proof, that the surrogate is capturing the right mechanism. If adding more quarters barely improves the estimate, that suggests early outcomes already span most of the causal pathway. If the surrogate index drifts away from the actual long term effect, you have learned something crucial: the early indicators are informative, but incomplete.

This is also where the idea of validation out of sample becomes philosophically important. In ordinary prediction, we validate by checking forecast accuracy. In causal surrogate work, we validate by checking transportability of the mapping from surrogates to outcomes. That is a much stricter test. The issue is not just whether the model predicts. It is whether the causal relationship survives transfer.

A good way to see this is with two questions:

Does the treatment move the surrogates in the experimental sample?
Does the relationship between surrogates and the final outcome look stable in the observational sample?

Only if both answers are yes do you have a credible bridge.

The practical synthesis: design features like a causal investigator, not a model optimizer

The most important lesson is not mathematical. It is methodological. When we use intermediate variables, we should stop asking only which ones maximize predictive power and start asking which ones capture the treatment's causal footprint.

That changes the feature selection problem in at least four ways.

First, you should prefer features that are directly tied to plausible mechanisms. If treatment affects wages through job stability, attendance, and hours worked, those are better candidates than a generic score that merely correlates with earnings.

Second, you should look for features that are strong in one of two ways: strongly linked to the primary outcome, or strongly linked to the treatment. A variable that is weak on both dimensions is unlikely to help.

Third, you should favor combinations that span pathways rather than single clever predictors. A surrogate index is often better than a lone proxy because it can aggregate multiple partial signals into something closer to causal sufficiency.

Fourth, you should check stability across samples and time. If the surrogate relationship changes when the environment changes, then the translation layer has broken.

This framework also explains why more features are not automatically better. In boosted tree systems, adding variables can improve fit but degrade interpretability. In surrogate work, adding variables can improve coverage of the causal pathways, but only if they are pre treatment or valid intermediate outcomes. The ideal set is not the largest set. It is the set that most completely and parsimoniously spans the mechanism.

A surrogate is not a shortcut around causality. It is a test of whether you have understood causality well enough to compress it.

That is a much higher standard than ordinary prediction, and that is exactly why it is valuable.

Key Takeaways

Do not confuse prediction with surrogacy. A variable can forecast the future without preserving the causal effect of treatment on that future.
Treat surrogate selection as pathway coverage. Ask whether your features collectively span the main causal routes, not just whether they correlate with the outcome.
Validate over time and across samples. A surrogate index is credible only if its relationship to the final outcome is stable and transportable.
Use bounds when surrogacy is imperfect. Bias depends on how much variation your surrogates explain and what they fail to capture.
Prefer mechanism aware feature engineering. In causal settings, the best features are the ones that reflect how treatment actually works.

Conclusion: the best proxy is the one that deserves to stand in

We usually think of proxies as second best solutions, something to use when the real thing is unavailable. But that view is too shallow. A good surrogate is not merely a stand in. It is an argument that the treatment effect has been sufficiently observed through the intermediate outcomes we can measure.

That is why this topic matters far beyond econometrics or program evaluation. It changes how we think about features, prediction, and evidence itself. The central question is not whether a variable is useful. The central question is whether it faithfully carries the causal signal we care about.

Once you see that distinction, you start noticing it everywhere. In machine learning, in policy evaluation, in product analytics, in medicine. The most seductive predictors are often the least trustworthy surrogates. The challenge is not to find the strongest signal. It is to find the signal that still means what you think it means when the outcome has not yet arrived.