AV-CAT: An Audio-Visual Context-Aware Transformer for Face Shaping

Home

Discover

Bookmarks

Keep Up With Latest Trending Papers. Computer Science, AI and Machine Learning and more.Subscribe

Artificial Intelligence

AV-CAT: An Audio-Visual Context-Aware Transformer for Face Shaping

Masked Lip-Sync Prediction by Audio-Visual Contextual Exploitation in Transformers

Qualitative results of our method. We focus on the task of high-fidelity person-agnostic lip-sync generation, which modifies the mouth shapes of any target template video according to the audio source. Here our lip-sync results should have the same mouth shape as the synced video to the audio source. The figures are selected from VoxCeleb

We propose the audio-visual context-aware transformer (av-cat) framework, which produces accurate lip-sync with photo-realistic quality by predicting the masked mouth shapes of the targetperson.

Our key insight is to exploit desired contextual information provided in audio and visual modalities thoroughly with delicately designed transformers.

Specifically, we propose a convolution-transformer backbone and design an attention-based fusion strategy for filling the unmasked parts.

Then the semantic audio information is involved in enhancing the self-attention computation.

Extensive experiments validate that our model can generate high-fidelity lip-synced results for arbitrary subjects given audio conditions.

Authors

Yasheng Sun, Hang Zhou, Kaisiyuan Wang, Qianyi Wu, Zhibin Hong, Jingtuo Liu, Errui Ding, Jingdong Wang, Ziwei Liu, Hideki Koike

Related Topics

Introduction

Driving human mouth movements with speech audio is in great need in the field of virtual human creation, entertainment, the film industry, and the detection of deep fakes.

Driving only the mouth movements of an arbitrary target video with audio input is an intriguing property that can be directly applied to scenarios like audio dubbing.

We identify that such a setting of modifying an arbitrary target image with audio is a particular type of conditional masked image prediction, which we refer to as.

Specifically, we mask most of the lower face with speech and a reference frame.

Two types of information, including the textural appearance information and the semantic mouth information, are needed for filling the masked areas.

They can be extracted from three perspectives : the visible part of the masked image itself which contains pose and environment information ; an appearance reference image ; and the semantic speech content information from audio.

In this work, we achieve masked lip-sync prediction for an arbitrary person through our and successfully produce high-quality results.

Result

In this work, we present the audio-visual context-aware transformer (av-cat), which produces high-quality lip-synced talking faces.

We formulate the problem in a masked lip-sync prediction manner, and propose to learn cross-frame and cross-modal context information with transformers.

Read the Paper

Read Paper

Bookmarks

Bookmarks

Keep Up With Latest Trending Papers. Computer Science, AI and Machine Learning and more.Subscribe

Artificial Intelligence

AV-CAT: An Audio-Visual Context-Aware Transformer for Face Shaping

Masked Lip-Sync Prediction by Audio-Visual Contextual Exploitation in Transformers

We propose the audio-visual context-aware transformer (av-cat) framework, which produces accurate lip-sync with photo-realistic quality by predicting the masked mouth shapes of the targetperson.

Our key insight is to exploit desired contextual information provided in audio and visual modalities thoroughly with delicately designed transformers.

Specifically, we propose a convolution-transformer backbone and design an attention-based fusion strategy for filling the unmasked parts.

Then the semantic audio information is involved in enhancing the self-attention computation.

Extensive experiments validate that our model can generate high-fidelity lip-synced results for arbitrary subjects given audio conditions.

Authors

Yasheng Sun, Hang Zhou, Kaisiyuan Wang, Qianyi Wu, Zhibin Hong, Jingtuo Liu, Errui Ding, Jingdong Wang, Ziwei Liu, Hideki Koike

Related Topics

Introduction

Driving human mouth movements with speech audio is in great need in the field of virtual human creation, entertainment, the film industry, and the detection of deep fakes.

Driving only the mouth movements of an arbitrary target video with audio input is an intriguing property that can be directly applied to scenarios like audio dubbing.

We identify that such a setting of modifying an arbitrary target image with audio is a particular type of conditional masked image prediction, which we refer to as.

Specifically, we mask most of the lower face with speech and a reference frame.

Two types of information, including the textural appearance information and the semantic mouth information, are needed for filling the masked areas.

In this work, we achieve masked lip-sync prediction for an arbitrary person through our and successfully produce high-quality results.

Result

In this work, we present the audio-visual context-aware transformer (av-cat), which produces high-quality lip-synced talking faces.

We formulate the problem in a masked lip-sync prediction manner, and propose to learn cross-frame and cross-modal context information with transformers.

Read the Paper

Read Paper

Home

Discover

Bookmarks

◐ Recommended Follows

◐ Latest Activity