3D Scene Compression through Entropy Penalized Neural Representation Functions
Thomas Bird, Johannes Ballé, Saurabh Singh, Philip A. Chou
Some forms of novel visual media enable the viewer to explore a 3D scene from
arbitrary viewpoints, by interpolating between a discrete set of original
views. Compared to 2D imagery, these types of applications require much larger
amounts of storage space, which we seek to reduce. Existing approaches for
compressing 3D scenes are based on a separation of compression and rendering:
each of the original views is compressed using traditional 2D image formats;
the receiver decompresses the views and then performs the rendering. We unify
these steps by directly compressing an implicit representation of the scene, a
function that maps spatial coordinates to a radiance vector field, which can
then be queried to render arbitrary viewpoints. The function is implemented as
a neural network and jointly trained for reconstruction as well as
compressibility, in an end-to-end manner, with the use of an entropy penalty on
the parameters. Our method significantly outperforms a state-of-the-art
conventional approach for scene compression, achieving simultaneously higher
quality reconstructions and lower bitrates. Furthermore, we show that the
performance at lower bitrates can be improved by jointly representing multiple
scenes using a soft form of parameter sharing.