The Real Job of Interpretation: Compressing Complexity Without Lying About It

What if importance is not a number, but a negotiation?

When a model predicts well, we are tempted to ask a second question that sounds simple but is actually dangerous: which features mattered most? The problem is that “mattered” can mean at least three different things. A feature can appear often, it can carry a lot of data, or it can change the prediction a great deal when removed. These are not the same thing, and treating them as interchangeable is how good models become misleading stories.

That tension is the hidden link between linear decomposition methods and feature attribution methods. One family of tools tries to compress data into the fewest meaningful directions. The other tries to assign credit to inputs without distorting the model’s behavior. Both are wrestling with the same question: how do you simplify a complex system without pretending the simplification is the whole truth?

Interpretation is not the art of finding the one true cause. It is the discipline of choosing a compression that preserves the structure you care about.

That framing changes everything. PCA is not just a trick for reducing dimensions. XGBoost feature importance is not just a ranking. Both are attempts to solve the same epistemic problem: how to turn tangled information into something legible while keeping the essential geometry intact.

---

Compression is not subtraction, it is geometry

The usual story about dimensionality reduction is too shallow. It says there are too many features, so we keep the most important ones and drop the rest. But that makes it sound like we are selecting items from a shopping list. In reality, methods like SVD and PCA do something more subtle: they rotate the coordinate system until the data becomes easier to describe.

This is why the singular value decomposition is so powerful. Any matrix can be factored into a sequence of transformations: a rotation, a scaling, and another rotation. In plain language, the data is first re-expressed in a new basis, then stretched along the most informative axes, then mapped back into the output space. PCA uses this to find directions of maximal variance, which means it does not ask which original variable is individually largest. It asks which combination of variables best explains the shape of the cloud.

Imagine you are looking at a map of a city with hundreds of roads. If you only ask which road is longest, you learn very little. But if you rotate the map until you can see the main corridors of traffic flow, the city suddenly becomes intelligible. PCA does exactly that. It finds the dominant corridors in data, the directions along which the information is most concentrated.

This matters because correlation is a geometric phenomenon. When variables are highly correlated, much of their apparent complexity is redundant. SVD identifies that redundancy by revealing which directions contain genuine variation and which are mostly noise or repetition. Dropping small singular values is not simply throwing away features. It is removing thin directions in the data geometry that contribute little to the total shape.

A useful mental model here is this: data complexity is often a matter of orientation, not quantity. Many datasets look high dimensional only because they are viewed from the wrong angle. PCA finds a better angle.

---

Feature importance has the same problem as PCA, but with credit instead of variance

If PCA asks, “What directions explain the data?”, feature importance asks, “What inputs explain the model?” The difference seems obvious until you try to answer it precisely. In a tree ensemble, one feature may split early and often, another may appear less frequently but produce larger loss reductions, and a third may matter only in interaction with others. Any single importance score can be true in one sense and false in another.

That is why the common measures in gradient boosted trees are so revealing. Weight counts how often a feature is used. Cover counts how many samples pass through those splits. Gain estimates how much the loss improves when the feature is used. Each one captures a different notion of influence, and each can be misleading if taken as the whole story.

The deeper issue is not just measurement, but attribution under dependence. When features are correlated, the model can use one as a proxy for another. Then the question “Which feature mattered?” becomes unstable. A feature may look unimportant simply because its correlated partner got there first. This is strikingly similar to PCA, where correlated variables do not deserve equal narrative weight because they are partly describing the same underlying direction.

Feature importance, then, is not about identifying isolated heroes. It is about distributing credit in a system where causes are entangled. A good attribution method should obey two constraints. First, consistency: if a model relies more on a feature, its importance should not go down. Second, accuracy: the total importance assigned across features should account for the total predictive power of the model. These are not cosmetic preferences. They are attempts to make explanation obey conservation laws.

Think of a model like a lighting rig on a stage. Weight tells you how many lamps are pointed at each actor. Cover tells you how many people can see each actor. Gain tells you how much brighter the scene gets when that lamp turns on. But none of those alone tells you which actor is actually illuminated enough to matter to the audience. To know that, you need a method that tracks the change in the whole scene, not just the hardware.

---

The hidden common problem: preserving structure under simplification

Here is the deeper connection between PCA and feature attribution: both are trying to answer a structural question, not a purely statistical one. They ask what remains invariant when we simplify.

PCA preserves the dominant variance structure of the data. Good attribution methods try to preserve the behavioral structure of the model. In both cases, the danger is false clarity. A bad compression makes the world look cleaner than it is. A bad attribution makes a model look more certain, more separable, and more causal than it really is.

This suggests a powerful general principle:

The best simplifications do not remove complexity. They relocate it into a smaller number of meaningful coordinates.

That is exactly what happens when PCA turns many correlated variables into a few principal components. The complexity does not disappear. It becomes visible as a small set of axes that explain most of the variation. Likewise, a rigorous attribution method does not eliminate interaction effects or dependence. It expresses them in a way that obeys consistency and accounting rules.

This is why “importance” is such a slippery word. Importance is never absolute. It is always relative to a representation. In the original feature space, a variable may look central. In the principal component space, that same variable may dissolve into a combination. In a tree model, a feature may appear decisive because it is used in many splits, while another may be more influential in the actual output shifts. The answer depends on whether you are measuring geometry, usage, or effect.

A strong analyst learns to ask a better first question: important for what transformation? Important for compression? Important for prediction? Important for explanation? The right metric depends on the job.

---

A practical framework: three kinds of importance

To make this concrete, it helps to separate importance into three levels.

1. Structural importance

This is the PCA question. Which directions carry the most variance, the most geometry, the most shape?

Example: Suppose you measure height and weight in a population. These are correlated. PCA may find that the first principal component is basically body size, a weighted combination of both. The second component might reflect a contrast between them, such as body composition. Structural importance tells you what the data is “about” in a compressed representation.

2. Predictive importance

This is the model question. Which features or combinations reduce error most?

Example: In an XGBoost model predicting loan default, income may be used many times, but debt-to-income ratio may produce larger gain when it appears because it separates risky from safe applicants more sharply. Predictive importance is about how much the model benefits from the signal, not how often the signal appears.

3. Counterfactual importance

This is the removal question. What changes if we take a feature away?

Example: If you remove weather data from a sales model, does prediction degrade slightly or collapse? If the output barely changes, the feature may be redundant. If the output changes a lot, the feature is causally central within the model, even if it appears only occasionally. This kind of importance is often the most honest because it asks not how a feature is represented, but what happens when it disappears.

These three levels often disagree, and that disagreement is informative. A feature can be structurally important but not predictive, predictive but not uniquely important, or counterfactually important only because it stands in for a family of correlated inputs. The point is not to pick one as universally correct. The point is to understand which lens matches the decision you are trying to make.

That is the same lesson PCA teaches. The first principal component is not “the most important variable.” It is the dominant direction of variation. Similarly, gain is not “true causal importance.” It is a measure of training improvement. Confusing those categories leads to overconfident narratives.

---

Why orthogonality matters more than we admit

One of the most elegant ideas in PCA is orthogonality. Principal components are uncorrelated, which means each component captures a distinct axis of variation. This is not just mathematically neat. It is psychologically revealing. Humans are drawn to stories with distinct causes, but real systems rarely separate so cleanly. Orthogonality is a way of forcing overlap to become legible.

In model interpretation, the same need appears under a different name: additivity. A good attribution method should allocate total importance across features so the pieces sum to the whole. That is a kind of accounting orthogonality. It says, if the model’s total behavior is fixed, our explanation should not invent or destroy importance along the way.

This is where the analogy gets interesting. PCA creates orthogonal directions in data space. Proper attribution creates additive directions in explanation space. Both are attempts to prevent double counting.

A correlated set of variables creates the illusion of multiple signals when there may be one underlying factor. A sloppy importance method can do the same: it can credit two correlated features for the same effect, inflating their apparent role. Orthogonal decomposition solves this by separating shared variance into clean axes. Attribution should aspire to the same discipline by assigning credit in a way that respects interactions and dependence.

In that sense, interpretation is a kind of basis selection. The question is not only what the model knows, but in what coordinates we choose to describe its knowledge. Choose poorly, and you get stories that are easy to tell but hard to trust. Choose well, and the model becomes less mysterious without becoming falsely simple.

---

Key Takeaways

1. Ask what kind of importance you mean. Distinguish structural importance, predictive importance, and counterfactual importance before making claims.

2. Treat compression as a geometric operation, not just feature dropping. PCA works by rotating into a basis where the data becomes easier to understand.

3. Be skeptical of single-number explanations. In boosted trees, weight, cover, and gain answer different questions. None is sufficient alone.

4. Prefer methods that obey consistency and accounting. A reliable attribution should not decrease when a model relies on a feature more, and the parts should sum to the whole.

5. Use correlation as a signal of shared structure, not duplicate evidence. Correlated variables often represent the same latent direction, so credit should be distributed carefully.

---

The real lesson: explanations should preserve the shape of uncertainty

The deepest connection between SVD, PCA, and feature attribution is not technical. It is epistemic. All three are attempts to preserve structure while reducing clutter. All three face the same temptation: to make a complex system seem cleaner than it is.

A good principal component does not pretend the original variables were independent. It reveals the opposite, that their entanglement can be summarized by a smaller set of directions. A good feature importance method does not pretend the model is built from isolated causes. It reveals how prediction is distributed across features while respecting the model’s actual dependence structure. In both cases, the goal is not to eliminate ambiguity. It is to compress it without losing the shape of the truth.

That is a more demanding standard than most people apply. We often ask whether a method is accurate, then whether it is interpretable, as if these were separate virtues. They are not. Interpretation is accurate only when it preserves the geometry of the system it explains. Otherwise, it is just a nicer story.

The next time a model looks impressive, do not ask first, “Which feature matters most?” Ask instead: What is the right basis for understanding this system? That question is harder. It is also the one that leads to explanations worth trusting.