Parameter-efficient Fine-tuning for Few-shot In-Context Learning

Home

Discover

Bookmarks

Keep Up With Latest Trending Papers. Computer Science, AI and Machine Learning and more.Subscribe

Added to collectionInteresting Papers in Quantum and AI

CC BY

Source arXiv

Machine Learning

Artificial Intelligence

Computation and Language

Parameter-efficient Fine-tuning for Few-shot In-Context Learning

Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning

Few-shot in-context learning (i.e.

Parameter-efficient fine-tuning (e.g.

Offers an alternative paradigm where a small set of parameters are trained to enable a model to perform the new task.

In this paper, we rigorously compare few-shot in-context learning and parameter-efficient fine-tuning and demonstrate that the latter offers better accuracy as well as dramatically lower computational costs.

We also propose a simple recipe basedon the t0 model called that can be applied to new tasks without task-specific tuning or modifications.

We validate the effectiveness of this recipe on completely unseen tasks by applying it to the raft benchmark, attaining super-human performance for the first time and outperforming the state-of-the-art by 6% absolute.

Authors

Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, Colin Raffel

◐ Recommended Follows

◐ Latest Activity