Text-to-image Translation with Spatial Features

Home

Discover

Bookmarks

Keep Up With Latest Trending Papers. Computer Science, AI and Machine Learning and more.Subscribe

CC BY

Source arXiv

Computer Vision

Artificial Intelligence

Text-to-image Translation with Spatial Features

Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation

Text-to-image synthesis has been a revolutionary breakthrough in the evolution of generative artificial intelligence (generative ai), allowing us to synthesize diverse images that convey highly complex visual concepts.

However, a pivotal challenge in leveraging such models for real-world content creation tasks is providing users with control over the generated content.

High-quality results on versatile text-guided image translation tasks, including translating sketches, rough drawings and animations into realistic images, changing of the class and appearance of objects in a given image, and modifications of global qualities such as lighting and color.

Authors

Narek Tumanyan, Michal Geyer, Shai Bagon, Tali Dekel

Related Topics

Introduction

With the rise of text-to-image foundation models, it seems that we can translate our imagination into high-quality images through text @cite0.

While such foundation models unlock a new world of creative processes in content creation, their power and expressivity come at the expense of user controllability, which is largely restricted to guiding the generation solely through an input text.

In this paper, we focus on attaining control over the generated structure and semantic layout of the scene, an imperative component in various real-world content creation tasks, ranging from visual branding and marketing to digital art.

Specifically, we pose the fundamental question of how structure information is internally encoded in such a model, and devise a new framework that enables fine-grained control over the structure by applying simple manipulations to spatial features inside the model.

Specifically, spatial features and their self-attentions are extracted from the guidance image, and are directly injected into the text-guided generation process of the target image.

Result

We present a versatile text-guided image-to-image translation (i2i) method that can be applied to diverse guidance image domains, both real and generated ones.

Our method successfully handles both naturally looking as well as artistic and textureless guidance images.

We thoroughly evaluate our method both quantitatively and qualitatively on state-of-the-art baselines under three different noising levels, including: (i)sdedit, (ii)p2p, (iii)diffuseit, (iv)vqgan-clip, and (v)text2live.

Read the Paper

Read Paper

Bookmarks

Bookmarks

Keep Up With Latest Trending Papers. Computer Science, AI and Machine Learning and more.Subscribe

CC BY

Source arXiv

Computer Vision

Artificial Intelligence

Text-to-image Translation with Spatial Features

Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation

However, a pivotal challenge in leveraging such models for real-world content creation tasks is providing users with control over the generated content.

Authors

Narek Tumanyan, Michal Geyer, Shai Bagon, Tali Dekel

◐ Recommended Follows

◐ Latest Activity