The Kullback-Leibler Decomposition
A Bayesian Characterization of Relative Entropy
We give a new characterization of relative entropy, also known as the kullback-leibler divergence.Our proof is independent of all earlier characterizations, but inspired by the work of petz.We use a number of interesting categories relatedto probability theory, in particular, we consider a category finstat where an object is a finite set equipped with a probability distribution, while amorphism is a measure-preserving function together with a measure-preserving right inverse.We show that any convex linear, lower semicontinuous functor from finstat to the additive monoid which vanishes when is optimal must be a scalar multiple of this relative entropy.
