When Explainability Is Just Another Hyperparameter

The hidden problem with “interpretable” models

What if the hardest part of explaining a model is not finding the explanation, but deciding whether the explanation is trustworthy enough to believe?

That question sits underneath a surprisingly common pattern in machine learning. We build models that promise clarity, then immediately discover that clarity itself has knobs, thresholds, and tradeoffs. The graph looks smooth or jagged. A feature appears important or not. An interaction term seems real or merely decorative. Suddenly, interpretation is no longer a neutral act of reading a model. It becomes a modeling choice.

This is the deeper tension connecting modern interpretable methods with feature attribution: explanations are not separate from training, they are part of the optimization problem. If that sounds uncomfortable, good. It should. The moment we accept that explanations can be tuned, weighted, and stabilized, we stop treating interpretability as a magic mirror and start treating it as engineering.

And engineering has consequences.

---

Why explanations wobble when models are too eager, or too timid

A useful way to think about this is through a familiar tradeoff: resolution versus stability. On one side, you want a model that can see fine-grained patterns, subtle thresholds, and meaningful interactions. On the other side, you want a model whose learned structure does not flinch when the data shifts slightly. The more resolution you ask for, the more likely you are to see noise dressed up as insight. The more stability you demand, the more likely you are to flatten real structure into bland averages.

This is why so many “interpretable” models expose tuning parameters that feel, at first glance, like ordinary performance settings but are in fact explanation settings in disguise. If you increase the number of bins, leaves, or interaction terms, you are not merely changing accuracy. You are changing the shape of the story the model tells.

Imagine a city map. A coarse map shows only highways and major districts. It is stable, easy to read, and useful for planning a road trip. A detailed map reveals alleyways, shortcuts, and neighborhood boundaries. It can also become cluttered, misleading, or overly sensitive to small drafting errors. Interpretability works the same way. The question is not whether more detail is better. The question is whether the detail reveals structure or invents it.

That is why a disciplined workflow matters. Start with defaults. Inspect the learned functions. Look for unusual spikes, jagged curves, unstable interactions, or suspiciously flat behavior. In other words, do not tune first and interpret later. Interpret first, tune second. The learned shape tells you which kind of failure you are dealing with: overfit, underfit, or missing structure.

This order matters because it changes the role of the human. Instead of blindly optimizing a score, you become a diagnostician. You are not asking only, “How accurate is the model?” You are asking, “What kind of evidence has the model actually learned?”

---

Feature importance is not a ranking, it is a theory of causality under constraints

Feature importance often gets presented as a leaderboard. Which variable matters most? Which feature should we keep? Which one should we drop? But that framing hides the real issue. Importance is not a pure measurement like weight on a scale. It is a definition of contribution.

Different attribution methods answer different questions. One can count how often a feature is used in splits. Another can weight those splits by how many data points flow through them. Another can ask how much a feature reduces training loss. These are not interchangeable views of the same object. They are different lenses on model behavior.

That distinction becomes especially important when we insist on two desirable properties of attribution.

First, consistency: if a model relies more on a feature, then the assigned importance should not go down. That sounds obvious, but many intuitive measures violate it. A feature can become more central in the model while looking less important under a brittle metric.

Second, accuracy of attribution: the parts should sum to the whole. If the total model performance is explained by features, the accounting should close. Otherwise, the attribution story leaves value on the table or invents value from nowhere.

Together, these properties reveal something profound: feature importance is an accounting system for predictive power. Good accounting must be coherent, additive, and resistant to manipulation by representation choices. If the system says a feature matters, the claim should survive changes in how the model encodes that feature's role.

A helpful analogy is a restaurant bill. You can split the total among dishes in many ways, but the receipts should still add up. If one method charges you twice for the appetizer and ignores the dessert, it is not just a different style of accounting, it is a broken one. Feature importance has the same problem. A ranking that does not reconcile with the model’s overall behavior may be visually satisfying but intellectually hollow.

This is also why simple split counts can mislead. A feature might appear frequently because it is easy to split on, not because it truly drives the model. Another feature might appear less often but carry more predictive weight when it does appear. Counting appearances is like judging a movie by how many scenes a character occupies, rather than by the role that character plays in the plot.

The deeper lesson is that interpretation requires a theory of aggregation. When you ask “how important is this feature?”, you are really asking how to distribute a complex outcome across many interacting causes. That is not a trivial descriptive task. It is a modeling choice with philosophical baggage.

---

The real tension: local clarity versus global truth

Once you put these two ideas together, a sharper question emerges: What kind of truth do we want from an interpretable model?

A highly tuned explainable model can produce elegant, smooth function plots for each feature. Feature attribution can produce clean rankings that seem to isolate the winners. But both can fail in the same way: they can make the model look more singular than it really is. Real predictive systems often depend on interaction, context, and threshold effects. One feature only matters when another is present. A variable appears important only within a narrow region of the data. A split that looks meaningful on its own may be merely a gatekeeper for a deeper rule.

This is why explanation is not just about transparency. It is about compression without distortion. The best interpretable model is not the one that tells the simplest story. It is the one that tells a simplified story that preserves the model’s real structure.

Think of it like summarizing a novel. A bad summary tells you the plot but destroys the motive. A good summary reduces length without erasing the logic of the characters. Interpretability faces the same test. A feature importance score or a smooth effect curve is a summary. The real question is whether it preserves the causal and predictive grammar of the model.

This is also where interaction terms become decisive. If many interactions rank highly, the model is telling you that the world it has learned is not separable into independent feature effects. In that case, the usual desire for a tidy feature ranking is almost a category error. You are asking for a solo performance in a symphony.

The best explanation is not always the cleanest one. It is the one that does the least violence to the model’s actual dependencies.

That reframing matters because it changes how we debug models. If the model looks overfit, the answer is not merely “regularize.” It may be “reduce the resolution of the story.” If the model looks underfit, the answer is not merely “increase capacity.” It may be “allow the story to admit more distinctions.” And if attribution looks unstable, the problem may not be the feature itself. The problem may be that the model’s structure is too entangled for the chosen attribution lens.

---

A practical framework: tune for narrative fidelity, not just metric gain

The most useful synthesis of these ideas is a shift in goal. Instead of tuning only for predictive performance, tune for narrative fidelity: how faithfully the learned explanation reflects the regularities in the data.

That sounds abstract, so here is a concrete framework you can use.

1. Start with the default model as a diagnostic instrument

Treat the first model as a probe, not a final answer. Look at the learned shapes, feature effects, and interaction structure before touching the knobs. Ask:

• Are the curves smooth in a way that makes sense, or suspiciously flat?
• Are there abrupt spikes that suggest overfitting?
• Do a few interactions dominate, or is the model missing obvious second-order structure?

The point is to let the model reveal its own failure mode.

2. Choose the knob that matches the failure mode

Different adjustments solve different interpretability problems.

• If the model is too jagged or unstable, reduce resolution or make stopping more aggressive.
• If the model is too conservative and misses important variation, increase resolution or allow more training.
• If interactions seem overloaded into a few terms, expand the interaction budget.

This is not just model tuning. It is explanation shaping. You are deciding how much complexity the model is allowed to use when it tells its story.

3. Demand attribution methods that satisfy coherence

When measuring feature importance, prefer methods that respect consistency and add up to the whole. If changing the model to rely more on a feature does not increase that feature’s importance, the attribution method is failing a basic sanity check. If the total importance does not reconcile with the model’s overall performance or output change, the explanation is incomplete.

4. Test explanations against counterfactuals

Ask what happens if a feature is removed, perturbed, or masked. If the ranking says a feature is crucial, the model should behave accordingly when that feature is absent. This is a powerful way to distinguish symbolic importance from operational importance.

5. Use interpretability to discover when simplicity is lying

Sometimes the most important result is not a neat rule. It is discovering that the world is more entangled than you hoped. If many features only matter in combination, the honest conclusion may be that a simple additive narrative is false. That is not a failure of interpretability. It is interpretability doing its job.

---

Key Takeaways

• Treat interpretability as a design choice, not a post-processing step. The shape of an explanation is influenced by training settings, not just by the model architecture.
• Use default settings as a diagnostic baseline. Inspect learned functions and interaction patterns before tuning. They often reveal whether the issue is overfitting, underfitting, or missing structure.
• Prefer attribution methods with coherence. Good feature importance should be consistent and additive, so the explanation matches the model’s real behavior.
• Do not mistake frequency for importance. A feature used often is not necessarily a feature that contributes most to predictive power.
• Tune for narrative fidelity. The goal is not merely higher accuracy, but a model whose explanation preserves the real structure of the data.

---

The model is not just predicting, it is arguing

The deepest lesson here is that every interpretable model makes an argument about the world. It says which variables matter, how they matter, when they matter, and whether they matter alone or only in combination. Tuning changes that argument. Attribution methods formalize that argument. Together, they determine not just what the model knows, but what kind of knowledge it is willing to claim.

That is why explainability should make us both more confident and more cautious. More confident, because we can inspect the shape of the reasoning instead of trusting a black box. More cautious, because once we can inspect it, we see how fragile the reasoning can be. A smooth curve may be a faithful simplification, or it may be a convenient illusion. A feature ranking may reveal the model’s structure, or it may merely reflect the quirks of the accounting method.

The real advance is not that machines can now explain themselves. The real advance is that we can finally ask a better question: what does it cost, in complexity and distortion, to make a model’s reasoning visible? Once you ask that, interpretation stops being a decorative layer on top of prediction. It becomes part of the science of modeling itself.

And that is the reframing worth remembering: the best explanation is not the one that looks simplest, but the one that remains true after you test how it was made.