Trending

Bookmarks

Keep Up With Latest Trending Papers. Computer Science, AI and Machine Learning and more.Subscribe

Added to collectionHow does it work?

Machine Learning

Computer Vision

Multi-Head Self-Attentions and Vision Transformers

How Do Vision Transformers Work?

The success of multi-head self-attentions (msas) for computer vision is now indisputable.

However, little is known about how they work.

We present fundamental explanations to help better understand the nature of msas.

In particular, we demonstrate the following properties of msas and visiontransformers (vits): (1) msas improve not only accuracy but also generalization by flattening the loss landscapes by flattening the loss landscapes.

Such improvement is primarily attributable to their data specificity, not long-range dependency.

On the other hand, vits suffer from non-convex losses.

Large datasets and loss landscape smoothingmethods alleviate this problem.

For example, msas are low-pass filters, but convs are high-pass filters.

Therefore, msas and convs are complementary.

In addition, msas at the end of a stage play a key role in prediction.

Based on these insights, we propose alternet, a model in which convolutional neural network (conv) blocks at the end of a stage are replaced with multi-head self-attentions (msa) blocks.

We show that alternet outperforms cnns not only in large data regimes but also in small data regimes.

Authors

Namuk Park, Songkuk Kim

Related Topics

Vision transformers

Multi-stage neural networks

Loss landscape smoothing methods

Multi-head self-attentions (msas)

Read the Paper

Read Paper

Trending

Bookmarks

◐ Recommended Follows

More

◐ Latest Activity

More

Multi-Head Self-Attentions and Vision Transformers - 42Papers

Read Related Papers

Multi-Scale Deep Learning for Estimating Horizontal Velocity Fields on the Solar Surface

Temporal Modeling via Video Transformer and Masked Visual-token Modeling

Multiset-Equivalence in Deep Set Prediction

A generative adversarial network based image transformation policy for visual self-supervision

Data Selection Based Fine-tuning Pipeline for Image Classification

Hidden-Fold Networks: Random Recurrent Residuals Using Sparse Supermasks

Softmax Gradient Tampering: Decoupling the Backward Pass for Improved Fitting

MorphMLP: A Self-Attention Free, MLP-Like Backbone for Image and Video

PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers

NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion