Bayesian Scientific Computing

Description

The once esoteric idea of embedding scientific computing into a probabilistic framework, mostly along the lines of the Bayesian paradigm, has recently enjoyed wide popularity and found its way into numerous applications. This book provides an insider’s view of how to combine two mature fields, scientific computing and Bayesian inference, into a powerful language leveraging the capabilities of both components for computational efficiency, high resolution power and uncertainty quantification ability. The impact of Bayesian scientific computing has been particularly significant in the area of computational inverse problems where the data are often scarce or of low quality, but some characteristics of the unknown solution may be available a priori. The ability to combine the flexibility of the Bayesian probabilistic framework with efficient numerical methods has contributed to the popularity of Bayesian inversion, with the prior distribution being the counterpart of classical regularization. However, the interplay between Bayesian inference and numerical analysis is much richer than providing an alternative way to regularize inverse problems, as demonstrated by the discussion of time dependent problems, iterative methods, and sparsity promoting priors in this book. The quantification of uncertainty in computed solutions and model predictions is another area where Bayesian scientific computing plays a critical role. This book demonstrates that Bayesian inference and scientific computing have much more in common than what one may expect, and gradually builds a natural interface between these two areas. cover 1 Preface Preface to the 2007 Book Introduction to Bayesian Scientific Computing Contents 978-3-031-23824-6_1 1 Bayesian Scientific Computing and Inverse Problems 1.1 What Do We Talk About When We Talk About Random Variables? 1.2 Through the Formal Theory, Lightly 1.2.1 Elementary Probabilities 1.2.2 Probability Distributions and Densities 1.2.3 Expectation and Covariance 1.2.4 Change of Variables in Probability Densities 978-3-031-23824-6_2 2 Linear Algebra 2.1 Vectors and Matrices 2.1.1 The Singular Value Decomposition 2.1.2 The Four Fundamental Subspaces 2.2 Solving Linear Systems 2.2.1 What Is a Solution? 2.2.2 Direct Linear System Solvers 978-3-031-23824-6_3 3 Continuous and Discrete Multivariate Distributions 3.1 Covariance Matrices 3.2 Normal Distributions 3.3 How Normal is it to be Normal? 3.4 Discrete Distributions 3.4.1 Normal Approximation to the Poisson Distribution 978-3-031-23824-6_4 4 Introduction to Sampling 4.1 On Averaging 4.2 Whitening and P–P Plots 4.3 Quadratures and Law of Large Numbers 4.4 Drawing from Discrete Densities 4.5 Sampling from a One-Dimensional Continuous Density 4.6 Sampling from Gaussian Distributions 4.7 Some Useful Sampling Algorithms 4.7.1 Importance Sampling 4.7.2 Drawing from Mixtures: SIR and Weighted Bootstrap 4.8 Rejection Sampling: Prelude to Metropolis–Hastings 978-3-031-23824-6_5 5 The Praise of Ignorance: Randomness as Lack of Certainty 5.1 Construction of Likelihood 5.2 Noise Models 5.2.1 Additive Noise 5.2.2 Multiplicative Noise 5.2.3 Poisson Noise 5.2.4 Composite Noise Models 978-3-031-23824-6_6 6 Enter Subject: Construction of Priors 6.1 Smoothness Priors 6.1.1 Freeing the Boundary Values 6.2 Generalization to Higher Dimensions 6.3 Whittle–Matérn Priors 6.4 Smoothness Priors with Structure 6.5 Conditionally Gaussian Priors and Hierarchical Models 6.6 Sparsity-Promoting Priors 6.7 Kernel-Based Priors 6.8 Data-Driven Priors 978-3-031-23824-6_7 7 Posterior Densities, Ill-Conditioning, and Classical Regularization 7.1 Likelihood Densities and Ill-Posedness of Inverse Problems 7.2 Maximum a Posteriori Estimate and Regularization 978-3-031-23824-6_8 8 Conditional Gaussian Densities 8.1 Gaussian Conditional Densities 8.2 Linear Inverse Problems 8.3 Interpolation and Conditional Densities 8.4 Covariance or Precision? 8.5 Some Computed Examples 8.5.1 Bayesian Interpolation of Multi-Dimensional Data 8.5.2 Posterior Density by Low-Rank Updating 978-3-031-23824-6_9 9 Iterative Linear Solvers and Priorconditioners 9.1 Iterative Methods in Linear Algebra 9.2 Krylov Subspace Iterative Methods 9.2.1 Conjugate Gradient Algorithm 9.2.2 Conjugate Gradient Method for Least Squares 9.3 Ill-Conditioning and Errors in the Data 9.4 Iterative Solvers in the Bayesian Framework 9.4.1 Preconditioning and Tikhonov Regularization 9.4.2 Priorconditioners: Specially Chosen Preconditioners 9.4.3 Stopping Rule Revisited 978-3-031-23824-6_10 10 Hierarchical Models and Bayesian Sparsity 10.1 Posterior Densities with Conditionally Gaussian Priors 10.1.1 IAS, Sparsity, Sensitivity Weighting and Exchangeability 10.1.2 IAS with Priorconditioned CGLS 10.2 More General Sparse Representations 10.3 Some Examples 978-3-031-23824-6_11 11 Sampling: The Real Thing 11.1 Preliminaries: Markov Chains and Random Walks 11.1.1 An Introductory Example 11.1.2 Random Walks in mathbbRn 11.2 Metropolis–Hastings Algorithm 11.2.1 Balance and Detailed Balance Equations 11.2.2 Construction of the MH Transition 11.2.3 Metropolis–Hastings in Action 11.3 Gibbs Sampler 11.4 Preconditioned Crank–Nicholson 978-3-031-23824-6_12 12 Dynamic Methods and Learning from the Past 12.1 The Dog and the Hunter 12.2 Sampling Importance Resampling (SIR) 12.2.1 Survival of the Fittest 12.2.2 Estimation of Static Parameters 978-3-031-23824-6_13 13 Bayesian Filtering for Gaussian Densities 13.1 Kalman Filtering 13.2 The Best of Two Worlds: Ensemble Kalman Filtering 13.2.1 Adding Unknown Parameters 1 (1) Appendix References Index