The Hidden Geometry of Bayesian Inference: Why Your Coordinates Change the Journey, but Not the Destination

What if two analysts describe the same uncertainty in different coordinates, run different sampling algorithms, and yet one of them quietly invents information the other never intended to assume?

This is one of the most important dangers in Bayesian inference. A parameter can be expressed as a probability, an odds ratio, a log odds, a rate, or a transformed scientific quantity. Mathematically, these may all describe the same underlying reality. But probability distributions do not automatically preserve their meaning when the labels change. At the same time, computational methods such as Gibbs sampling move through a model one conditional distribution at a time, and those movements are also shaped by the coordinates chosen.

The deeper issue is not merely how to assign a prior or how to generate samples. It is this:

How can inference remain faithful to the underlying model when both our description of uncertainty and our method of exploring it depend on representation?

A useful answer emerges by separating three things that are often confused: the belief structure, the geometry of the parameter space, and the computational route used to explore the posterior. Jeffreys prior addresses the first two. Gibbs sampling addresses the third. Together, they reveal a general principle: good Bayesian practice requires invariance in what we mean, and flexibility in how we compute it.

The Same Parameter Can Smuggle in Different Beliefs

Consider a coin with an unknown probability (p) of landing heads. Suppose we have observed no data. What should our prior belief be?

A natural answer might be to assign equal density to every value of (p) between zero and one. That produces a uniform prior:

[ \pi(p) = 1, \quad 0 < p < 1. ]

At first glance, this seems like perfect neutrality. But now describe the same coin using the log odds:

[ \eta = \log\left(\frac{p}{1-p}\right). ]

The transformation is smooth and one to one. Nothing about the physical coin has changed. Yet a uniform distribution over (p) is not uniform over (\eta). The density in log odds becomes concentrated around zero and thins out toward both extremes. Conversely, a uniform density over (\eta) induces a density over (p) that gives more weight to probabilities near zero and one than to probabilities near one half.

So which choice represents no preference?

The question cannot be answered by appealing to ordinary flatness. Flatness depends on the ruler used. A line looks equally spaced in meters but not in a nonlinear scale. If uncertainty is represented by a distribution, the same problem appears in a more consequential form: uniformity in one coordinate system becomes a structured preference in another.

This is not a technical curiosity. In applied work, analysts routinely choose parameters for convenience. A variance may be replaced by its logarithm to enforce positivity. A probability may be represented by a logit. A rate may be expressed on a logarithmic scale. If a supposedly neutral prior is placed directly on whichever representation is convenient, the analyst may be adding assumptions without noticing.

Jeffreys prior offers a response based on the information supplied by the likelihood. Instead of asking which numerical coordinate should look flat, it asks how distinguishable nearby parameter values are according to the model. The resulting prior is proportional to the square root of the Fisher information:

[ \pi_J(\theta) \propto \sqrt{I(\theta)}. ]

The central virtue is invariance under smooth, monotone transformations. If the parameter is reexpressed, the prior transforms in a way that preserves the same underlying measure of uncertainty. It is not uniform according to ordinary Euclidean distance. It is uniform according to the information geometry induced by the statistical model.

That distinction is profound. It says that neutrality is not the absence of structure. It is the choice to let the model's own sensitivity define the relevant structure.

Information Geometry Is a Ruler Built from Distinguishability

Imagine walking across a landscape where the ground itself tells you how far apart two points are. In a flat field, equal physical distances feel equal. In a mountain range, a small horizontal movement may represent a dramatic change in elevation, while a longer movement across a plateau may represent little experiential difference.

Parameter spaces behave similarly. Two values of a parameter may be numerically close but produce sharply different data distributions. Other values may be numerically far apart yet make almost no practical difference to what we observe. Fisher information measures this local sensitivity.

For a Bernoulli parameter (p), the Fisher information is

[ I(p) = \frac{1}{p(1-p)}. ]

Therefore Jeffreys prior is

[ \pi_J(p) \propto \frac{1}{\sqrt{p(1-p)}}. ]

This prior assigns more density near zero and one than the uniform prior does. That is not because extreme probabilities are being declared more likely in some naive sense. Rather, the model is highly sensitive near the boundaries. A small change in (p) near zero can substantially alter the relative likelihood of rare events. The information geometry therefore stretches those regions.

Under the log odds parameter (\eta), the same prior takes a different algebraic form, but it represents the same distribution over the underlying statistical possibilities. The density changes because densities must account for the Jacobian of transformation. The prior measure does not change merely because the coordinates do.

This gives us a valuable diagnostic:

When a conclusion changes after a harmless reexpression of the parameter, ask whether the conclusion came from the model or from the ruler used to describe it.

There is an important qualification. Jeffreys priors can be improper, meaning their total integral is infinite. That does not automatically make them unusable. In Bayesian inference, an improper prior may be acceptable when it yields a proper posterior for every possible dataset under consideration. But this is a condition to verify, not a permission to ignore normalization. A prior that fails to produce a genuine posterior is not a statement of ignorance. It is a broken inferential procedure.

This distinction between prior measure and posterior validity will matter when we turn to computation. A sampler can only explore a legitimate target distribution. No algorithm can rescue a posterior that does not exist.

Gibbs Sampling Turns Global Uncertainty into Local Questions

Suppose a model has several unknown quantities, such as a regression coefficient, a variance, and a latent class assignment. The joint posterior may be difficult to sample from directly. Gibbs sampling takes a different route. It repeatedly samples each variable from its conditional distribution while holding the others fixed.

For two parameters, (\theta_1) and (\theta_2), one iteration might look like this:

1. Draw (\theta_1) from (p(\theta_1 \mid \theta_2, x)).
2. Draw (\theta_2) from (p(\theta_2 \mid \theta_1, x)).

These updates create a Markov chain. Under suitable conditions, the chain converges to the joint posterior (p(\theta_1, \theta_2 \mid x)). The method succeeds because compatible local views can reconstruct a global distribution.

This is where the connection to invariant priors becomes unexpectedly useful. Jeffreys prior asks whether our representation of uncertainty changes when we change coordinates. Gibbs sampling asks whether we can recover a complicated global distribution by examining a sequence of conditional perspectives. Both are concerned with a whole that is not directly visible from any single representation.

But there is a crucial asymmetry. The target posterior is a mathematical object that should not depend on arbitrary coordinate choices, provided the prior and likelihood are transformed correctly. The Gibbs chain, however, is a computational object whose behavior can depend dramatically on those choices.

Consider a posterior shaped like a long, narrow valley. In one parameterization, the coordinates may be strongly correlated. A Gibbs sampler that updates one coordinate at a time will move across the valley in small zigzags. It may require an enormous number of iterations to travel through the region of high probability. Transform the parameters into a more suitable coordinate system, and the same posterior may become closer to circular. The target has not changed, but the sampler can now explore it much more efficiently.

This produces a distinction that should guide all computational Bayesian work:

Invariance protects the meaning of the answer. Reparameterization can improve the speed and reliability of reaching it.

A poor coordinate system does not necessarily make the posterior wrong. It can make the computation misleading in practice because a chain that has not mixed well may appear to provide evidence for a narrow conclusion. The danger is not only inefficiency. It is false confidence caused by mistaking repeated local movement for genuine exploration.

The Two Kinds of Neutrality

The combined lesson can be organized around two kinds of neutrality.

The first is semantic neutrality. This asks whether the assumptions encoded in a prior survive a legitimate change of parameterization. Jeffreys prior is designed for this problem. It replaces arbitrary numerical flatness with a geometry based on the likelihood's information.

The second is procedural neutrality. This asks whether the computational procedure explores all relevant parts of the target rather than becoming trapped in a convenient local neighborhood. Gibbs sampling provides a valid route under appropriate conditions, but its effectiveness depends on conditional structure, dependence, blocking, and parameterization.

These are related but not identical. A model may have a principled, invariant prior and still be sampled badly. Conversely, a highly efficient Gibbs implementation may sample from a posterior that encodes an arbitrary and coordinate dependent prior.

A practical Bayesian workflow should therefore pass through three tests:

1. The meaning test

If the parameter is rewritten in an equivalent form, do the substantive prior beliefs remain the same? If not, identify the source of the difference. Sometimes the change is intentional. Often it is simply an artifact of assigning a flat distribution on a convenient scale.

2. The existence test

Does the chosen prior and likelihood produce a proper posterior for all relevant data configurations? This matters especially when using an improper prior. A formal density is not enough. The posterior must define a real probability distribution.

3. The exploration test

Has the computational chain actually explored the posterior? Inspect trace plots, effective sample sizes, autocorrelation, and results from multiple starting points. If the variables are strongly dependent, consider transformations or joint updates rather than assuming that many iterations automatically imply reliable inference.

This framework also clarifies why conditional sampling is both powerful and limited. Conditioning reduces a difficult multivariate problem into manageable pieces. But if the pieces are badly aligned with the geometry of the target, the chain can crawl. The local questions may be easy to answer while the global journey remains painfully slow.

A Worked Intuition: The Coin, the Log Odds, and the Chain

Return to the coin. Suppose we observe a small number of tosses, all tails. On the probability scale, the posterior may place substantial mass near (p=0). On the log odds scale, that same concentration appears as a long tail toward negative values. These are not competing conclusions. They are two coordinate descriptions of one posterior measure.

Now imagine embedding this coin model inside a hierarchical model with several related coins. Each coin has its own probability, but all probabilities share information through a population level distribution. The posterior can develop strong dependence between individual probabilities and the population parameters. Updating one quantity at a time may cause the Gibbs chain to move slowly: an individual coin probability changes, then the population parameter must adjust, then the other probabilities must compensate.

A transformation such as centering, rescaling, or using log odds may improve the geometry. A blocked Gibbs update may improve it further by sampling several strongly linked quantities together. The target remains the same if the transformations are handled correctly. What changes is the path by which the algorithm reaches the target.

This suggests a useful metaphor. The posterior is a city, parameterization is the map, and the sampler is the transportation system. A distorted map can make distances hard to interpret. A poor transportation network can leave important neighborhoods unexplored. But neither the map nor the roads should be confused with the city itself.

The most reliable analysis uses a map whose distances reflect statistical meaning and a transportation system adapted to the city's shape.

Key Takeaways

• Separate beliefs from coordinates. A flat prior on a transformed parameter is not generally equivalent to a flat prior on the original parameter. Ask what substantive belief the transformation is introducing.

• Use information geometry when neutrality matters. Jeffreys prior provides a principled way to construct a prior that is invariant under smooth, monotone reexpression of the parameter.

• Treat improper priors as conditional tools, not automatic defaults. Verify that the posterior is proper for every data configuration that matters to the analysis.

• Distinguish correctness from computational efficiency. A valid posterior can be explored very slowly. Reparameterization, rescaling, and blocked conditional updates can improve mixing without changing the inferential target.

• Diagnose the journey, not just the destination. Trace plots, autocorrelation, effective sample size, and multiple chains are evidence about whether the sampler has genuinely explored the posterior.

Conclusion: Inference Is a Problem of Meaning and Movement

Bayesian inference is often presented as a sequence of formulas: choose a prior, multiply by a likelihood, normalize, and sample. But beneath those operations lies a more subtle problem. We must decide what counts as the same uncertainty when descriptions change, and we must find a way to move through that uncertainty without confusing local repetition for global understanding.

Invariant priors address the first challenge by grounding neutrality in the information structure of the model rather than in arbitrary coordinates. Gibbs sampling addresses the second by assembling a global distribution from conditional views. Their intersection reveals a broader lesson: statistical reasoning has both a geometry and a dynamics.

Geometry determines which distinctions are meaningful. Dynamics determines whether our computational process can discover them.

The mature question is therefore not simply, What is the posterior? It is also: What measure defines it, what coordinates express it, and what evidence shows that our algorithm has truly explored it? When those questions are kept separate, reparameterization becomes a source of clarity rather than confusion. A change of coordinates can no longer quietly alter the meaning of ignorance, and a long Markov chain can no longer masquerade as certainty merely because it has taken many steps.