Sparse Updates for Gradient-Based Deep Neural Network Training

Home

Discover

Bookmarks

Keep Up With Latest Trending Papers. Computer Science, AI and Machine Learning and more.Subscribe

CC BY

Source arXiv

Machine Learning

Sparse Updates for Gradient-Based Deep Neural Network Training

Training Neural Networks with Fixed Sparse Masks

We show that it is possible to induce a fixed sparse mask on the model s parameters that selects a subset to update over many iterations.

Our method constructs the mask out of the parameters with the largest fisher information as a simple approximation as to which parameters are most importantfor the task at hand.

Experiments on parameter-efficient transfer learning and distributed training show that our approach matches or exceeds the performance of other methods for training with sparse updates while being moreefficient in terms of memory usage and communication costs.

Authors

Yi-Lin Sung, Varun Nair, Colin Raffel

◐ Recommended Follows

◐ Latest Activity