We introduce the \textit{epistemic neural network} (ENN) as an interface for
uncertainty modeling in deep learning. All existing approaches to uncertainty
modeling can be expressed as ENNs, and any ENN can be identified with a
Bayesian neural network. However, this new perspective provides several
promising directions for future research. Where prior work has developed
probabilistic inference tools for neural networks; we ask instead, `which
neural networks are suitable as tools for probabilistic inference?'. We propose
a clear and simple metric for progress in ENNs: the KL-divergence with respect
to a target distribution. We develop a computational testbed based on inference
in a neural network Gaussian process and release our code as a benchmark at
\url{this https URL}. We evaluate several canonical approaches
to uncertainty modeling in deep learning, and find they vary greatly in their
performance. We provide insight to the sensitivity of these results and show
that our metric is highly correlated with performance in sequential decision
problems. Finally, we provide indications that new ENN architectures can
improve performance in both the statistical quality and computational cost.

Authors

Ian Osband, Zheng Wen, Mohammad Asghari, Morteza Ibrahimi, Xiyuan Lu, Benjamin Van Roy