Relative Predictive Coding

Home

Discover

Bookmarks

Keep Up With Latest Trending Papers. Computer Science, AI and Machine Learning and more.Subscribe

Self-supervised Representation Learning with Relative Predictive Coding

Relative predictive coding (rpc) is a new contrastiverepresentation learning objective that maintains a good balance among trainingstability, minibatch size sensitivity, and downstream task performance.

The keyto the success of rpc is two-fold.

First, rpc introduces the relativeparameters to regularize the objective for boundedness and low variance.

Second, rpc contains no logarithm and exponential score functions, which are the main cause of training instability in prior contrastive objectives.

We empirically verify the effectiveness of rpc on benchmark vision and speech self-supervised learning tasks.

Authors

Yao-Hung Hubert Tsai, Martin Q. Ma, Muqiao Yang, Han Zhao, Louis-Philippe Morency, Ruslan Salakhutdinov

◐ Recommended Follows

◐ Latest Activity