A Power Law Functional Form for Neural Scaling Behavior

Home

Discover

Bookmarks

Keep Up With Latest Trending Papers. Computer Science, AI and Machine Learning and more.Subscribe

Added to collectionTop 100 Papers By Signal Impressions

CC BY

Source arXiv

Machine Learning

Artificial Intelligence

A Power Law Functional Form for Neural Scaling Behavior

Broken Neural Scaling Laws

A Broken Neural Scaling Law (BNSL) (dark black solid line) with 3 breaks (where purple dotted lines intersect with dark black solid line) decomposed into the individual power laws (dashed lines that are yellow, blue, red, and green) that it is composed of overlaid on top of it. The 1st and 2nd break are very smooth; the 3rd break is very sharp. See Section

We present a smoothly broken power law functional form that accurately models and extrapolates the scaling behaviors of deep neural networks (i.e.

How the evaluation metric of interest varies as the amount of compute used for training, number of model parameters, or training dataset size varies) for each task within a large and diverse set of upstream and downstream tasks, in zero-shot, prompted, and fine-tuned settings.

This set includes large-scale vision and unsupervised language tasks, diffusion generative modeling of images, arithmetic, and reinforcement learning.

When compared to other functional forms for neural scaling behavior, this functional form yields extrapolations of scaling behavior that are considerably more accurate (root mean squared log error of its extrapolations are 0.86 times that of previous state-of-the-art on average) on this set.

Moreover, this functional form accurately models and extrapolates scaling behavior that other functional forms are incapable of expressing such as the non-monotonic transitions present in the scaling behavior of phenomena such as double descent and the delayed, sharp inflection points present in the scaling behavior of tasks such as arithmetic.

Authors

Ethan Caballero, Kshitij Gupta, Irina Rish, David Krueger

Related Topics

Introduction

Neural scaling laws aim to predict the behavior of large-scale models from smaller, cheaper experiments, allowing to focus on the best-scaling architectures, algorithms, datasets, and so on.

However, no organization currently has direct access to these larger resources of the future, and it has been empirically verified many times that methods which perform best at smaller scales often are no longer the best performing methods at larger scales.

To work on, identify, and steer the methods that are most probable to stand the test-of-time as these larger resources come online, one needs a way to predict how all relevant performance evaluation metrics of artificial neural networks vary in all relevant settings as scale increases.

The functional forms of scaling laws evaluated in previous work are not up to this challenge.

One salient defect is that they can only represent monotonic functions.

Many also lack the expressive power to model inflection points (on a linear-linear plot), which can be observed empirically for many downstream tasks, and even some upstream tasks, such as our @xmath0 -digit arithmetic task, or the modular arithmetic task introduced by @cite3 in their work on grokking.

To overcome the above limitations, we present-a functional form that generalizes power laws (linear in log-log plot) to a smoothly broken power law, i.e.

A smoothly connected piecewise (approximately) linear function in a log-log plot.

An extensive empirical evaluation demonstrates that this functional form accurately model and extrapolates the scaling behaviors for each task within a large and diverse set of upstream and downstream tasks, in zero-shot, prompted, and fine-tuned settings.

Result

We evaluate how well various functional forms extrapolate performance on vision tasks as training dataset size increases.

In this large-scale vision subset of the scaling laws benchmark of @cite1, the tasks that are evaluated are error rate on each of various few-shot downstream image classification (ic) tasks.

The downstream tasks are: birds 200, caltech101, cifar-100, and imagenet.

We show that big-transfer residual neural networks (bit) yields extrapolations with the lowest root mean squared log error (root standard log error) for 63.89% of tasks of any of the functional forms, while the next best functional form performs the best on only 13.89%.

Read the Paper

Read Paper

Bookmarks

Bookmarks

Keep Up With Latest Trending Papers. Computer Science, AI and Machine Learning and more.Subscribe

Added to collectionTop 100 Papers By Signal Impressions

CC BY

Source arXiv

Machine Learning

Artificial Intelligence

A Power Law Functional Form for Neural Scaling Behavior

Broken Neural Scaling Laws

We present a smoothly broken power law functional form that accurately models and extrapolates the scaling behaviors of deep neural networks (i.e.

This set includes large-scale vision and unsupervised language tasks, diffusion generative modeling of images, arithmetic, and reinforcement learning.

Authors

Ethan Caballero, Kshitij Gupta, Irina Rish, David Krueger

◐ Recommended Follows

◐ Latest Activity