Interpolation and Extrapolation in High-Dimensional Datasets
Learning in High Dimension Always Amounts to Extrapolation
State-of-the-art algorithms work so well because of their ability to correctly interpolate training data.
We empirically and theoretically argue against those two points and demonstrate that on any high-dimensional dataset, interpolation almost surely never happens.
Those results challenge the validity of our current interpolation/extrapolation definition as an indicator of generalization performances.