The Feature Table Is a Prior: Why Representation Determines What Models Can See

What if changing the way you write down a variable changes what your model believes, even when the underlying information has not changed?

That sounds like a philosophical puzzle, but it is a daily engineering problem. A probability model can treat a parameter differently after a smooth change of coordinates. A machine learning model can treat a feature differently after a transformation, encoding choice, or interaction is introduced. In both cases, the data may be identical in substance while the model's behavior changes dramatically.

The deeper issue is not whether a representation is mathematically equivalent to another. It is this: what preferences does a representation smuggle into inference?

A useful answer emerges by connecting two ideas that are usually taught apart: invariant probability and feature engineering for boosted trees. Both concern the hidden geometry of a modeling problem. Both ask whether our conclusions depend on arbitrary coordinates. And both reveal that there is no such thing as a completely neutral representation once a model has limited capacity.

The fantasy of a neutral representation

Suppose a parameter p describes the probability of success. We might say that we have no preference among possible values of p, and assign a uniform distribution to the interval from 0 to 1. Now define a new parameter η, perhaps the log odds:

$$ η = \log\left(\frac{p}{1-p}\right) $$

The values of η range from negative infinity to positive infinity. If we transform the uniform distribution on p into a distribution on η, the result is not uniform. It places more density near p values close to 0 and 1 when viewed on the η scale.

But if we begin with η and apply the same verbal principle, namely “no preference,” we might choose a uniform distribution over η. Transforming that distribution back to p gives a different prior, one that concentrates more heavily around the middle of the probability interval.

Both procedures sound reasonable. They cannot both represent the same notion of neutrality.

This is not a minor technical inconvenience. It exposes a basic fact about statistical reasoning: uniformity is not an intrinsic property of a parameter. It is a property of a chosen geometry. The parameter space does not arrive with a universally correct ruler. We choose a coordinate system, and the ruler comes with it.

Jeffreys prior addresses this problem by using the information geometry of the statistical model. Rather than declaring a parameter uniform according to ordinary Euclidean distance, it weights regions according to how much the data can distinguish nearby parameter values. A small interval in one part of the parameter space may represent as much inferential difference as a large interval elsewhere.

The important lesson is broader than the formula. A transformation should not alter the underlying judgment merely because we renamed the parameter. When it does, the model is responding not only to the phenomenon but also to the coordinate system.

Now consider a boosted tree model. It seems, at first, to live in a completely different world. It does not begin with a probability density over a parameter. It repeatedly chooses feature thresholds, fits simple trees, and adds them together. Yet it faces the same question: which changes in representation should count as substantive, and which should count as mere changes of coordinates?

A tree does not see your feature. It sees a geometry

Imagine a feature called income. One dataset records income in dollars. Another records the same income in thousands of dollars. A standard decision tree generally sees these as equivalent because the ordering is unchanged. A split at 50,000 dollars corresponds to a split at 50 thousand dollars.

This gives trees a useful kind of invariance to monotone transformations. If every value is transformed by a strictly increasing function, the rank ordering remains the same, so the candidate partitions of the observations remain essentially unchanged. A logarithm, a square root, or a change of units may leave the tree's possible splits intact.

That is a real form of coordinate invariance. But it is easy to overstate it. Trees are not invariant to every representation choice. They are invariant to some transformations because their geometry is based largely on ordering. They remain sensitive to choices that alter ordering, ties, missing value handling, category structure, available interactions, or the number of opportunities to find a useful split.

Take the feature age. As a raw numeric variable, a tree can create regions such as:

• age less than 30,
• age between 30 and 47,
• age greater than 47.

If age is instead encoded into broad categories such as “young,” “middle aged,” and “older,” the model loses possible boundaries. If it is replaced by a collection of overlapping indicators such as “under 25,” “under 35,” and “under 50,” the model gains a larger menu of candidate partitions. The underlying information may be closely related, but the model's search space has changed.

This is the crucial connection to invariant priors: feature engineering is not merely preparing data for a model. It is selecting a geometry in which the model will search.

A boosted tree with finite depth, finite rounds, regularization, and a particular split algorithm cannot express every function that is theoretically compatible with the data. It prefers some explanations over others because some can be built with fewer, cleaner, or more profitable splits. Representation determines which explanations are cheap.

In a Bayesian model, a prior assigns relative plausibility before observing the data. In a boosted tree, the combination of feature representation and training constraints creates an algorithmic prior. It determines which patterns are easy to discover, which require many steps, and which are practically invisible.

A coordinate system is never just a label. It is a budget for what the model can notice.

The hidden prior inside feature engineering

Consider a credit model predicting default. Suppose we include annual income, debt, and the ratio of debt to income. The ratio is mathematically derived from the first two variables, so it may seem redundant. But it changes the model's effective prior.

Without the ratio, a shallow tree may need several splits to approximate the idea that debt becomes dangerous relative to income. It might first split on income, then split on debt within each income region. That pattern is possible, but expensive in depth and perhaps difficult for a greedy algorithm to find.

With the ratio, the same concept can be represented by one threshold: debt to income greater than 0.4. The feature does not add new information in the strict sense. It adds a cheap coordinate for an important relationship.

This is analogous to choosing a parameterization that aligns with the structure of the problem. If a scientific theory is naturally expressed in terms of odds rather than probabilities, or elasticity rather than raw change, the right coordinates can make the relevant regularities more visible. The transformation does not create evidence. It changes the path by which the model can use evidence.

The reverse can also happen. Suppose we provide dozens of closely related thresholds, ratios, bins, and interaction features. Each may be defensible individually. Together, they create a large search space in which accidental patterns become easier to find. The model now has many ways to explain noise, particularly when the dataset is small or the validation procedure is weak.

This resembles an improper prior in Bayesian analysis. An improper prior is not automatically forbidden if the posterior is proper for every possible observation. The practical criterion is whether the full inferential procedure produces a valid, normalized conclusion after data arrive.

The machine learning analogue is not that feature-rich models are inherently wrong. It is that a flexible representation is acceptable only when the complete training and evaluation pipeline controls the resulting complexity. More features, encodings, and interactions can be useful, but only if validation, regularization, and deployment checks establish that the learned function remains stable.

A feature set can therefore be judged in two ways:

1. Does it contain information relevant to the target?
2. Does it present that information in a geometry that encourages reliable learning rather than accidental discovery?

The second question is often neglected because it is harder to measure.

Invariance is a design principle, not a superstition

A powerful way to engineer features is to distinguish transformations that should leave predictions unchanged from transformations that should change the model's available meaning.

Changing income from dollars to euros should not change the prediction if the exchange rate is applied consistently. Replacing a variable with a strictly increasing transformation may not matter much to a tree if the transformation preserves ordering. These are cases where the representation change is semantically irrelevant, and invariance is desirable.

By contrast, converting a continuous measurement into coarse bins can remove distinctions that matter. Treating an ordinal variable as an unordered category can erase order. Replacing a missing value with zero can create a false statement that “not observed” means “none.” One hot encoding a high cardinality identifier can expose memorization routes that were not present in a carefully designed representation.

The goal is not maximal invariance. If a model were invariant to every transformation, it would be unable to learn from representation at all. The goal is invariance to nuisance transformations and sensitivity to meaningful ones.

This gives us a practical audit. For every feature transformation, ask:

• What changes in the real world does this representation preserve?
• What distinctions does it erase?
• What thresholds or interactions does it make cheap?
• What accidental distinctions does it introduce?
• Should predictions remain stable under this transformation?

For boosted trees, this audit should include the mechanics of splitting. A monotone transformation may preserve order, but it can still affect numerical precision, ties, quantile binning, missing value behavior, or implementation details. A categorical encoding can alter the number and arrangement of candidate partitions. An interaction feature can make a previously expensive relationship available in one split.

The right question is therefore not, “Are these two feature sets mathematically equivalent?” It is, “Are they equivalent under the model's actual search procedure and constraints?”

A two layer model of representation

It helps to separate two layers that are often conflated.

The first is semantic content: what facts about the world are encoded in the data. The second is computational accessibility: how easily the chosen algorithm can express and test those facts.

Two representations can have nearly identical semantic content but very different computational accessibility. Raw income and debt may contain enough information to determine debt to income, but the explicit ratio makes proportional burden easier for a shallow boosted tree to discover. Conversely, a collection of arbitrary interactions may preserve the original information while making spurious relationships easier to fit.

This two layer model also clarifies why feature engineering remains valuable even for powerful algorithms. A sufficiently large model might approximate a ratio, an interaction, or a nonlinear transformation from raw variables. But “can represent” is not the same as “will reliably discover under the available data, optimization path, and regularization.” Engineering changes the inductive bias toward or away from certain solutions.

A useful workflow follows from this:

1. State the invariances of the domain

Write down transformations that should not affect the prediction. Unit changes, currency conversion, harmless monotone rescaling, or reordered category labels may belong here. Test whether the pipeline respects them.

2. State the meaningful relations

Identify thresholds, ratios, saturation effects, recency patterns, and interactions that have a domain rationale. Add representations that make these relations inexpensive, but do not add them merely because they improve one validation split.

3. Measure representation sensitivity

Train models across defensible representations and compare not only average accuracy but also prediction agreement, subgroup behavior, calibration, and performance under distribution shift. Large changes reveal that the model is relying on a coordinate choice that may not be stable.

4. Charge a complexity cost for every new representation

A feature is not free because it is deterministic. Each binning scheme, interaction, encoding, and derived ratio expands the set of patterns the learner can search. Track this expansion as carefully as you track tree depth or number of boosting rounds.

5. Validate the whole posterior, not just the feature

In Bayesian language, the concern is whether the posterior is proper. In predictive modeling, the concern is whether the end to end procedure produces stable, honest predictions. Leakage controls, nested validation, temporal splits, calibration, and monitoring are part of feature engineering, not administrative afterthoughts.

Key Takeaways

• Treat representation as an inductive bias. A feature transformation changes which patterns are easy for a model to express, even when it adds no new raw information.
• Separate nuisance invariance from meaningful sensitivity. Predictions should survive irrelevant changes such as units, but respond to genuine changes such as altered ratios, thresholds, or temporal context.
• Remember that tree invariance is conditional. Monotone transformations often preserve split order, but encodings, ties, missing values, quantization, interactions, and search constraints can still change behavior.
• Audit derived features for both value and search cost. Ratios and interactions can expose important structure, while large feature menus can invite accidental fits.
• Validate representations as competing hypotheses. Compare stability, calibration, subgroup performance, and shift robustness, not only a single score.

The deepest lesson is that neutrality is not the absence of choices. It is the disciplined identification of which choices should not matter.

A parameterization that appears uniform may hide a geometry. A feature table that appears factual may hide a preference for certain thresholds, interactions, and explanations. In both statistics and machine learning, the model does not encounter the world directly. It encounters a constructed space in which some distances, boundaries, and patterns have been made more consequential than others.

So when a model changes its answer after a supposedly harmless transformation, do not ask only whether the algorithm is behaving badly. Ask what geometry the transformation revealed. Perhaps the change is a bug. Perhaps it exposes a genuine modeling assumption that was invisible before. Or perhaps it shows that the original representation was already making a choice and merely succeeded at hiding it.

The most reliable model is not the one with no prior preferences. That model does not exist. It is the one whose preferences are explicit, defensible, and stable under every transformation that should have been irrelevant.