What Are Bayesian Neural Network Posteriors Really Like?

The posterior over Bayesian neural network (BNN) parameters is extremely
high-dimensional and non-convex. For computational reasons, researchers
approximate this posterior using inexpensive mini-batch methods such as
mean-field variational inference or stochastic-gradient Markov chain Monte
Carlo (SGMCMC). To investigate foundational questions in Bayesian deep
learning, we instead use full-batch Hamiltonian Monte Carlo (HMC) on modern
architectures. We show that (1) BNNs can achieve significant performance gains
over standard training and deep ensembles; (2) a single long HMC chain can
provide a comparable representation of the posterior to multiple shorter
chains; (3) in contrast to recent studies, we find posterior tempering is not
needed for near-optimal performance, with little evidence for a "cold
posterior" effect, which we show is largely an artifact of data augmentation;
(4) BMA performance is robust to the choice of prior scale, and relatively
similar for diagonal Gaussian, mixture of Gaussian, and logistic priors; (5)
Bayesian neural networks show surprisingly poor generalization under domain
shift; (6) while cheaper alternatives such as deep ensembles and SGMCMC methods
can provide good generalization, they provide distinct predictive distributions
from HMC. Notably, deep ensemble predictive distributions are similarly close
to HMC as standard SGLD, and closer than standard variational inference.