Quantization of Zero-Shot Performance in Large Language Models

Home

Discover

Bookmarks

Keep Up With Latest Trending Papers. Computer Science, AI and Machine Learning and more.Subscribe

Added to collectionTop 100 Papers By Signal Impressions

CC BY

Source arXiv

Machine Learning

Neural and Evolutionary Computing

Quantization of Zero-Shot Performance in Large Language Models

The case for 4-bit precision: k-bit Inference Scaling Laws

Bit-level scaling laws for mean zero-shot performance across four datasets for 125M to 66B parameter OPT models. Zero-shot performance increases steadily for a fixed number of model bits as we reduce the quantization precision from 16 to 4 bits. At 3-bits, this relationship reverses, making 4-bit precision optimal.

We develop inference scaling laws of zero-shot performance in large language models to determine the bit-precision and model size that maximizes zero-shot accuracy.

We run more than 35,000 zero-shot experiments with 16-bit inputs and k-bit parameters to examine which quantization methods improve scaling for 3 to 8-bit precision at scales of 19m to 66b parameters across the families bloom, opt, neox/pythia, and gpt-2.

We find that it is challenging to improve the bit-level scaling trade-off, with the only improvements being the use of a small block size (splitting the parameters into small independently quantized blocks) and the quantization data type being used (e.g., int vs float).

Overall, our findings show that 4-bit precision is almost universally optimal for total model bits and zero-shot precision.

Authors

Tim Dettmers, Luke Zettlemoyer

Related Topics

Introduction

We study bit-level inference scaling laws to determine the precision that maximizes zero-shot accuracy given a certain number of total bits in the model.

We study five different model families, opt, pythia/neox, gpt-2, bloom, and bloomz, with 19m to 66b parameters, and for 3 to 16-bit precision.

In addition, we run more than 35,000 zero-shot experiments to vary many of the recently studied advances in quantization precision, including the underlying data types, quantization block size, and distribution centering.

We find that reducing the precision steadily from 16 to 4 bits increases the zero-shot performance for a fixed number of model bits, while at 3-bit, the performance decreases.

At 3-bits, this relationship reverses, making 4-bit precision optimal.

Result

We run more than 35,000 zero-shot experiments to vary many of the recently studied advances in quantization precision, including the underlying data types, quantization block size, distribution centering, and outlier-dependent quantization.

We make the following observations : for a given zero-shot performance, 4-bit precision yields optimal bit-level scaling for almost all model families and model scales.

For 3 to 5-bit precision, we do see improvements in scaling by applying quantization methods.

However, scaling behavior can only be improved by using less than 6-bit precision rather than enhancing the quantization precision through other means.

Read the Paper

Read Paper

Bookmarks

Bookmarks

Keep Up With Latest Trending Papers. Computer Science, AI and Machine Learning and more.Subscribe

Added to collectionTop 100 Papers By Signal Impressions

CC BY

Source arXiv

Machine Learning

Neural and Evolutionary Computing

Quantization of Zero-Shot Performance in Large Language Models

The case for 4-bit precision: k-bit Inference Scaling Laws

We develop inference scaling laws of zero-shot performance in large language models to determine the bit-precision and model size that maximizes zero-shot accuracy.

Overall, our findings show that 4-bit precision is almost universally optimal for total model bits and zero-shot precision.

Authors

Tim Dettmers, Luke Zettlemoyer

Related Topics

Introduction

We study bit-level inference scaling laws to determine the precision that maximizes zero-shot accuracy given a certain number of total bits in the model.

We study five different model families, opt, pythia/neox, gpt-2, bloom, and bloomz, with 19m to 66b parameters, and for 3 to 16-bit precision.

We find that reducing the precision steadily from 16 to 4 bits increases the zero-shot performance for a fixed number of model bits, while at 3-bit, the performance decreases.

At 3-bits, this relationship reverses, making 4-bit precision optimal.

Result

We make the following observations : for a given zero-shot performance, 4-bit precision yields optimal bit-level scaling for almost all model families and model scales.

For 3 to 5-bit precision, we do see improvements in scaling by applying quantization methods.

However, scaling behavior can only be improved by using less than 6-bit precision rather than enhancing the quantization precision through other means.

Read the Paper

Read Paper

Home

Discover

Bookmarks

◐ Recommended Follows

◐ Latest Activity