Learning to Reinforcement Learning with Causal Sequence Models

Home

Discover

Bookmarks

Keep Up With Latest Trending Papers. Computer Science, AI and more.Subscribe

CC BY

Source arXiv

Machine Learning

Artificial Intelligence

Learning to Reinforcement Learning with Causal Sequence Models

In-context Reinforcement Learning with Algorithm Distillation

We propose a method for distilling reinforcement learning (rl)algorithms into neural networks by modeling their training histories with a causal sequence model.

A dataset of learning histories is generated by a source rl algorithm, and then a causal transformer is trained by autoregressively predicting actions given their preceding learning histories as context.

We demonstrate that the method can reinforcement learn in-context in a variety of environments with sparse rewards, combinatorial task structure, and pixel-based observations, and find that it learns a more data-efficient rlalgorithm than the one that generated the source data.

Authors

Michael Laskin, Luyu Wang, Junhyuk Oh, Emilio Parisotto, Stephen Spencer, Richie Steigerwald, DJ Strouse, Steven Hansen, Angelos Filos, Ethan Brooks, Maxime Gazeau, Himanshu Sahni, Satinder Singh, Volodymyr Mnih

Related Topics

Introduction

Transformers have emerged as powerful neural network architectures for sequence modeling.

After pre-training on large offline datasets, large transformers have been shown to generalize to downstream tasks in text completion, language understanding, and image generation.

Recent work demonstrated that transformers can also learn policies from offline data by treating offline reinforcement learning (rl) as a sequential prediction problem.

We refer to the family of approaches that learns policies from offline rl data via imitation learning as offline policy distillation, or simply policy distillation (pd).

Despite its simplicity and scalability, a substantial drawback of pd is that the resulting policy does not improve incrementally from additional interaction with the environment.

We hypothesize that the reason policy distillation does not improve through trial an error is that it trains on data that does not show learning progress.

Our key observation is that the sequential nature of learning within rl algorithm training could, in principle, make it possible to model the process of reinforcement learning itself as a causal sequence prediction problem.

Specifically, if a transformer’s context is long enough to include policy improvement due to learning updates it should be able to represent not only a fixed policy but a policy improvement operator by attending to states, actions and rewards from previous episodes.

This opens the possibility that any rl algorithm can be distilled into a sufficiently powerful sequence model such as a transformer via imitation learning, converting it into an in-context rl algorithm.

Result

In this work, we demonstrate that algorithm distillation can distill an in-weights reinforcement learning (rl) algorithm into an in-context reinforcement learning (in-context rl) algorithm by modeling rl learning histories with a causal transformer.

We also show that the algorithm distillation process can learn more data-efficient algorithms than those that generated the source data.

Read the Paper

Read Paper

Bookmarks

Bookmarks

Keep Up With Latest Trending Papers. Computer Science, AI and more.Subscribe

CC BY

Source arXiv

Machine Learning

Artificial Intelligence

Learning to Reinforcement Learning with Causal Sequence Models

In-context Reinforcement Learning with Algorithm Distillation

We propose a method for distilling reinforcement learning (rl)algorithms into neural networks by modeling their training histories with a causal sequence model.

Authors

Related Topics

Introduction

Transformers have emerged as powerful neural network architectures for sequence modeling.

After pre-training on large offline datasets, large transformers have been shown to generalize to downstream tasks in text completion, language understanding, and image generation.

Recent work demonstrated that transformers can also learn policies from offline data by treating offline reinforcement learning (rl) as a sequential prediction problem.

We refer to the family of approaches that learns policies from offline rl data via imitation learning as offline policy distillation, or simply policy distillation (pd).

Despite its simplicity and scalability, a substantial drawback of pd is that the resulting policy does not improve incrementally from additional interaction with the environment.

We hypothesize that the reason policy distillation does not improve through trial an error is that it trains on data that does not show learning progress.

Result

We also show that the algorithm distillation process can learn more data-efficient algorithms than those that generated the source data.

Read the Paper

Read Paper

Home

Discover

Bookmarks

◐ Recommended Follows

◐ Latest Activity