You need to enable JavaScript to run this app.
Sign Up
Home
Trending
Discover
Bookmarks
Keep Up With Latest Trending Papers. Computer Science, AI and more.
Subscribe
Liked by
Luislamb
CC BY
Source arXiv
Computers and Society
Machine Learning
Artificial Intelligence
The Rise and Fall of Training Compute in Machine Learning
Compute Trends Across Three Eras of Machine Learning
We study trends in the most readily quantified factor-compute.
We show that before 2010 training compute grew in line with moore s law, doubling roughly every 20months.
Since the advent of deep learning in the early 2010s, the scaling of training compute has accelerated, doubling approximately every 6 months.
In late 2015, a new trend emerged as firms developed large-scale models with 10to 100-fold larger requirements in training compute.
Overall, ourwork highlights the fast-growing compute requirements for training advanced machine learning systems.
Authors
Jaime Sevilla, Lennart Heim, Anson Ho, Tamay Besiroglu, Marius Hobbhahn, Pablo Villalobos
Related Topics
Deep learning
Large-scale ml models
Read the Paper
Read Paper
Sign Up
Home
Trending
Discover
Bookmarks
◐ Recommended Follows
Qz4650292950
Follow
Ani Bajirao
Follow
Julien Chaumond
Follow
More
◐ Latest Activity
More
The Rise and Fall of Training Compute in Machine Learning - 42Papers
Read Related Papers
Extreme Tensoring for Low-Memory Preconditioning
Criticality of Deep Neural Networks with Infinity Approximation
Evolving Normalization-Activation Layers
EH-DNAS: End-to-End Hardware-aware Differentiable Neural Architecture Search
Accelerating Deep Learning with Dynamic Data Pruning
8-bit approximation of softmax layer in multi-layer perceptron oral neural networks
Minibatch Estimation of Hyperparameters for the Process
Mesa: A Resource-efficient Training Framework for High-Performance Transformers
Transfer Learning with Gaussian Processes for Bayesian Optimization
Graph Conal Gated Recurrent Neural Network for Multi-step Traffic Volume Prediction