Trending

Bookmarks

Keep Up With Latest Trending Papers. Computer Science, AI and more.Subscribe

Added to collectionML

Machine Learning

Transfer Laws for Fine-tuning Neural Networks

Scaling Laws for Transfer

We study empirical scaling laws for transfer learning between distributions in an unsupervised, fine-tuning setting.

We find that the effective data transferred is described well in the low data regime by a power-law of parameter count and fine-tuning dataset size.

We believe the exponents in these power-laws correspond to measures of the generality of a model and proximity of distributions (in a directed rather than symmetric sense).

We find that pre-training effectively multiplies the fine-tuning dataset size.

Authors

Danny Hernandez, Jared Kaplan, Tom Henighan, Sam McCandlish

Related Topics

Neural networks

Read the Paper

Read Paper

Trending

Bookmarks

◐ Latest Activity

More

◐ Recommended Follows

Julien Chaumond

More

Transfer Laws for Fine-tuning Neural Networks - 42Papers

Read Related Papers

Criticality of Deep Neural Networks with Infinity Approximation

Data Selection Based Fine-tuning Pipeline for Image Classification

Softmax Gradient Tampering: Decoupling the Backward Pass for Improved Fitting

Learning to Refit for Convex Learning Problems

Scaling Up Vision-Language Pre-training for Image Captioning

Stylized Adversarial Defense

Disparity Between Batches as a Signal for Early Stopping

On the Provable Generalization of Recurrent Neural Networks

Adaptive Transfer Learning: a simple but effective transfer learning

Finding the Winning Ticket of BERT for Binary Text Classification via Adaptive Layer Truncation before Fine-tuning