transformer architectures

Home

Discover

Bookmarks

Keep Up With Latest Trending Papers. Computer Science, AI and more.Subscribe

Machine Learning

Information Retrieval

Sound

Artificial Intelligence

Multimedia

Attention is All You Need? Good Embeddings with Statistics are enough: Audio Understanding WITHOUT Convolutions/Transformers/BERTs/Mixers/Attention/RNNs or ....

This paper presents a way of doing large scale audio understanding without traditional state of the art neural architectures. Ever since the introduction of deep learning for understanding audio signa

More...

CC BY

Machine Learning

Sound

Audio and Speech Processing

Artificial Intelligence

Multimedia

Causal Transformer generative models for raw waveform synthesis

A Generative Model for Raw Audio Using Transformer Architectures

This paper proposes a novel way of doing audio synthesis at the waveformlevel using generative transformer architectures.

We propose a deep neural network for generating waveforms, similar to wavenet \cite{oord2016wavenet}.

CC BY

Machine Learning

Computation and Language

Cryptography and Security

Mischief: A Lightweight, Human-readable, Adversarial Example for Language Models

Mischief: A Simple Black-Box Attack Against Transformer Architectures

We introduce a simple and lightweight method to produce a class of human-readable, realistic adversarial examples for language models.

We perform exhaustive experimentations of our algorithm on four transformer-based architectures, across a variety of downstream tasks, as well as under varying concentrations of said examples.

CC BY

Machine Learning

Computation and Language

On the Depth-to-Width of Transformer Architectures

Which transformer architecture fits my data? A vocabulary bottleneck in self-attention

We theoretically predict the existence of an embedding rank bottleneck that limits the contribution of self-attention width to the transformer expressivity.

We empirically demonstrate the existence of this rank bottleneck and its implications on the depth-to-width interplay of transformerarchitectures, linking the architecture variability across domains to the often glossed-over usage of different vocabulary sizes or embedding ranks in different domains.

Machine Learning

Artificial Intelligence

Spatio-Temporal Wind Speed Forecasting using Graph Networks and Novel Transformer Architectures

To improve the security and reliability of wind energy production, short-term forecasting has become of utmost importance. This study focuses on multi-step spatio-temporal wind speed forecasting for t

More...

Computer Vision

Machine Learning

Artificial Intelligence

Twins: Revisiting Spatial Attention Design in Vision Transformers

Very recently, a variety of vision transformer architectures for dense prediction tasks have been proposed and they show that the design of spatial attention is critical to their success in these task

More...

Computation and Language

HuggingFace's Transformers: State-of-the-art Natural Language Processing

Recent progress in natural language processing has been driven by advances in both model architecture and model pretraining. Transformer architectures have facilitated building higher-capacity models

More...

Machine Learning

Artificial Intelligence

Computation and Language

Language Models with Transformers

The Transformer architecture is superior to RNN-based models in computational efficiency. Recently, GPT and BERT demonstrate the efficacy of Transformer models on various NLP tasks using pre-trained l

More...

Computation and Language

Transition-based Parsing with Stack-Transformers

Modeling the parser state is key to good performance in transition-based parsing. Recurrent Neural Networks considerably improved the performance of transition-based systems by modelling the global st

More...

CC BY

Computer Vision

Machine Learning

Vision Transformer Architectures for 2D Body Possibilistic Estimation

PE-former: Pose Estimation Transformer

We investigate the use of a pure transformerarchitecture (i.e., one with no convolutional backbones for feature extraction) for the problem of 2d body pose estimation.

We evaluate two pure transformer architectures on the coco dataset.

Computation and Language

Relphormer: Relational Graph Transformer for Knowledge Graph Representation

Transformers have achieved remarkable performance in widespread fields, including natural language processing, computer vision and graph mining. However, in the knowledge graph representation, where t

More...

Machine Learning

Biomolecules

Stats Machine Learning

Modern Hopfield Networks and Attention for Immune Repertoire Classification

A central mechanism in machine learning is to identify, store, and recognize patterns. How to learn, access, and retrieve such patterns is crucial in Hopfield networks and the more recent transformer

More...

Machine Learning

Information Retrieval

Computation and Language

Stats Machine Learning

Towards Lithuanian grammatical error correction

Everyone wants to write beautiful and correct text, yet the lack of language skills, experience, or hasty typing can result in errors. By employing the recent advances in transformer architectures, we

More...

CC BY

Machine Learning

Neural and Evolutionary Computing

Artificial Intelligence

Transformer Architectures for Graphs

Attending to Graph Transformers

Transformer architectures for graphs emerged as an alternative to established techniques for machine learning with graphs, such as graph neural networks, often attributed to their ability to circumvent graph neural networks'shortcomings, such as over-smoothing and over-squashing.

Here, we derive a taxonomy of graph transformer architectures, bringing some order to this emerging field.

CC BY

Machine Learning

Neural and Evolutionary Computing

Artificial Intelligence

Computation and Language

Transformer Architectures and Algorithms

Formal Algorithms for Transformers

This document aims to be a self-contained, mathematically precise overview of transformers architectures and algorithms.

It covers what transformers are, how they are trained, what they are used for, their keyarchitectural components, and a preview of the most prominent models.

Computer Vision

Machine Learning

Translational Equivariance in Kernelizable Attention

While Transformer architectures have show remarkable success, they are bound to the computation of all pairwise interactions of input element and thus suffer from limited scalability. Recent work has

More...

CC BY

Computer Vision

Boosting Vision Transformer with k-NN Attention

KVT: k-NN Attention for Boosting Vision Transformers

We propose a sparse attention scheme, dubbed k-nn attention, for boosting vision transformers.

Instead of involving all the image patches (tokens) for attention matrix calculation, we only select the top-k similar tokens from the keys for each query to compute the attentionmap.

Computer Vision

Incorporating Convolution Designs into Visual Transformers

Motivated by the success of Transformers in natural language processing (NLP) tasks, there emerge some attempts (e.g., ViT and DeiT) to apply Transformers to the vision domain. However, pure Transform

More...

Home

Discover

Bookmarks

◐ Recommended Follows

◐ Latest Activity