A Bidirectional Gated State-Space Model for NLP Pretraining

Home

Discover

Bookmarks

Keep Up With Latest Trending Papers. Computer Science, AI and Machine Learning and more.Subscribe

CC BY

Source arXiv

Machine Learning

Computation and Language

A Bidirectional Gated State-Space Model for NLP Pretraining

Pretraining Without Attention

Model Variants. (

Recently developed routing layers based on state-space models (ssm) and modelarchitectures based on multiplicative gating have a large impact on pretraining accuracy in natural language processing (nlp).

The proposed bidirectional gated ssm (bigs) replicates benchmark pretraining results without attention and can be extended to long-form pretraining without approximation.

Authors

Junxiong Wang, Jing Nathan Yan, Albert Gu, Alexander M. Rush

Related Topics

Introduction

State-space models (ssms) are a competitive architecture for long-range sequence modeling.

They have proven central to next-to - leading order (nlp) tasks with their ability to learn effectively on unlabeled unlabeled datasets.

Other architectures have been used but require attention layers for high-accuracy.

This work proposes the bidirectional gated ssm (bigs) for pretraining without attention.

Experiments show that bigs is able to match the performance of a bert model when trained on the same data in a controlled setting.

Result

We present a state-space model for encoder-and encoder-supported experiments in the long-range language modeling benchmark.

Multiplicative gating is necessary to achieve this accuracy.

We show that the model can replicate the accuracy of a pretrained transformer model at both scale of tokens and scale of tokens.

These results are significantly better than other non-attention based pretrained models.

Read the Paper

Read Paper

Bookmarks

Bookmarks

Keep Up With Latest Trending Papers. Computer Science, AI and Machine Learning and more.Subscribe

CC BY

Source arXiv

Machine Learning

Computation and Language

A Bidirectional Gated State-Space Model for NLP Pretraining

Pretraining Without Attention

Model Variants. (

The proposed bidirectional gated ssm (bigs) replicates benchmark pretraining results without attention and can be extended to long-form pretraining without approximation.

Authors

Junxiong Wang, Jing Nathan Yan, Albert Gu, Alexander M. Rush

Related Topics

Introduction

State-space models (ssms) are a competitive architecture for long-range sequence modeling.

They have proven central to next-to - leading order (nlp) tasks with their ability to learn effectively on unlabeled unlabeled datasets.

Other architectures have been used but require attention layers for high-accuracy.

This work proposes the bidirectional gated ssm (bigs) for pretraining without attention.

Experiments show that bigs is able to match the performance of a bert model when trained on the same data in a controlled setting.

Result

We present a state-space model for encoder-and encoder-supported experiments in the long-range language modeling benchmark.

Multiplicative gating is necessary to achieve this accuracy.

We show that the model can replicate the accuracy of a pretrained transformer model at both scale of tokens and scale of tokens.

These results are significantly better than other non-attention based pretrained models.

Read the Paper

Read Paper

Home

Discover

Bookmarks

◐ Recommended Follows

◐ Latest Activity