Bayesian Inference

Overview

Bayesian inference represents uncertainty about parameters or hypotheses with probability distributions. A prior distribution is combined with a likelihood describing how data would arise under a model. Bayes' theorem produces a posterior distribution that quantifies the remaining uncertainty after conditioning on the observations.

Technical foundations

Bayes' theorem states that posterior density is proportional to likelihood times prior density, with the marginal likelihood providing normalisation. In hierarchical models, population-level distributions partially pool information across groups and regularise noisy estimates. Posterior predictive distributions integrate parameter uncertainty when forecasting new data. Identifiability depends on whether distinct parameter values generate distinguishable observations; a concentrated posterior can still be misleading if the model excludes plausible mechanisms. Priors can encode previous measurements, physical constraints or weak regularisation and should be examined through prior-predictive simulation before data are fitted.

How it works

Analysts specify a generative model, derive or compute its likelihood and combine it with prior information. Posterior expectations, credible intervals and predictive distributions are then evaluated analytically or with numerical algorithms such as Markov chain Monte Carlo, sequential Monte Carlo or variational inference. Model checking compares simulated predictions with observed structure.

Measurement and research methods

Closed-form posteriors occur in conjugate models, but modern applications use computation. Markov chain Monte Carlo constructs dependent draws that converge toward the posterior; diagnostics include effective sample size, split-chain convergence statistics and checks for divergent trajectories. Variational inference optimises an approximating family and can scale well while underrepresenting uncertainty. Sequential Monte Carlo evolves weighted particles through a series of distributions. Model assessment uses posterior-predictive checks, calibration, residual structure and out-of-sample scoring. Sensitivity analyses vary priors, likelihoods and data exclusions to expose conclusions dependent on fragile assumptions.

Key ideas

• A posterior is conditional on the assumed model, data quality and prior, not a model-free truth statement.
• Credible intervals and frequentist confidence intervals answer different probability questions.
• Predictive performance and calibration should be evaluated on observations not used to fit flexible models.

Current research frontier

Active research improves inference for high-dimensional latent variables, simulation-based models with intractable likelihoods and adaptive experiments. Probabilistic programming automates differentiation and sampling but cannot determine whether a scientific model is meaningful. Bayesian decision theory combines posterior uncertainty with explicit utilities or losses, making value judgements inspectable. Causal applications require assumptions about interventions and confounding beyond probability updating. Robust methods model outliers and distributional misspecification, while safe workflows record all analysis choices. The central challenge is not merely computing a posterior but building, criticising and revising a model that predicts relevant features of reality.

Why it matters

Bayesian methods integrate heterogeneous evidence, propagate uncertainty through complex systems and support sequential decisions. They are widely used in astronomy, epidemiology, engineering, machine learning and experimental science.

Limits and open questions

Results may be sensitive to weakly identified parameters or implausible priors. Approximate algorithms can converge to the wrong distribution, and comparing many models on the same data can hide selection effects unless the complete workflow is reported.