The Rise and Fall of Training Compute in Machine Learning

Bookmarks

Keep Up With Latest Trending Papers. Computer Science, AI and more.Subscribe

Compute Trends Across Three Eras of Machine Learning

We study trends in the most readily quantified factor-compute.

We show that before 2010 training compute grew in line with moore s law, doubling roughly every 20months.

Since the advent of deep learning in the early 2010s, the scaling of training compute has accelerated, doubling approximately every 6 months.

In late 2015, a new trend emerged as firms developed large-scale models with 10to 100-fold larger requirements in training compute.

Overall, ourwork highlights the fast-growing compute requirements for training advanced machine learning systems.

Authors

Jaime Sevilla, Lennart Heim, Anson Ho, Tamay Besiroglu, Marius Hobbhahn, Pablo Villalobos

◐ Recommended Follows