M-VADER: a diffusion model for image generation using combinations of images and text

Home

Discover

Bookmarks

Keep Up With Latest Trending Papers. Computer Science, AI and Machine Learning and more.Subscribe

CC BY

Source arXiv

Computer Vision

M-VADER: a diffusion model for image generation using combinations of images and text

M-VADER: A Model for Diffusion with Multimodal Context

M-VADER: image synthesis with multimodal context. The guidance prompt, comprised of interleaved images and text, is embedded using a multimodal decoder, S-MAGMA. The output of S-MAGMA is used to condition the generation process of a fine-tuned version of Stable Diffusion via cross-attention. This allows the finetuned version of Stable Diffusion to convert the starting (random noise) image into the output image shown.

Diffusion models have been introduced that make it possible to specify the output image using a text prompt.

Inspired by the success of those models, and led by the notion that language was already developed to describe the elements of visual contexts that humans find most important, we introduce an embedding model closely related to a vision-language model.

Specifically, we introduce the embedding model s-magma: a 13 billionparameter multimodal decoder combining components from an autoregressive vision-language model magma and biases finetuned for semantic search.

Authors

Samuel Weinbach, Marco Bellagente, Constantin Eichenberg, Andrew Dai, Robert Baldock, Souradeep Nanda, Björn Deiseroth, Koen Oostermeijer, Hannah Teufel, Andres Felipe Cruz-Salinas

Related Topics

Introduction

Text-to-image synthesis has enabled the creation of artistic and photorealistic images with impressive generalization from text prompts.

In recent years, text guided diffusion models have been developed, which transform either random noise, or a starting image, into a new image, guided by a text contextual prompt.

However, there is no effective method to use multimodal context guidance in diffusion models : a context that combines one or more passages of text with one or more images.

This is the key contribution of this paper : a method for image generation with multimodal contextual guidance (of arbitrary prompt length).

To this end, we develop a novel embedding model for multimodal guidance prompts, and we finetune a version of the text guided diffusion model, which we call m-vader, to follow contextual guidance.

We additionally show how to control the relative importance of the components of the guidance prompt, by reweighting the attention scores to each component.

Result

In pure text conditioned diffusion models, style modification can be achieved by starting from a corrupted version of an image and using a natural language instruction as a condition to steer generation.

The level of image corruption has to be chosen carefully, since there is a tradeoff between being close to the original image and enforcing the desired style.

On the one hand, if only a little amount of noise is added to the image, the denoising process is very short and the conditional guidance has not enough steps to have a major effect.

On the other hand, degrading the image too much one loses all information about it and the result is a random image in the desired style.

We present a new paradigm for style change using a multimodal prompt.

We simply concatenate the style instruction with the image to form a single multimodal conditional prompt and then generate from pure noise.

This way we can use the full diffusion length to steer generation while maintaining information about the original image consistently.

Our model offers an alternate approach to the task of image variation by prompting with just a single image and includes examples of this in appendix .

Read the Paper

Read Paper

Bookmarks

Bookmarks

Keep Up With Latest Trending Papers. Computer Science, AI and Machine Learning and more.Subscribe

CC BY

Source arXiv

Computer Vision

M-VADER: a diffusion model for image generation using combinations of images and text

M-VADER: A Model for Diffusion with Multimodal Context

Diffusion models have been introduced that make it possible to specify the output image using a text prompt.

Authors

Samuel Weinbach, Marco Bellagente, Constantin Eichenberg, Andrew Dai, Robert Baldock, Souradeep Nanda, Björn Deiseroth, Koen Oostermeijer, Hannah Teufel, Andres Felipe Cruz-Salinas

Related Topics

Introduction

Text-to-image synthesis has enabled the creation of artistic and photorealistic images with impressive generalization from text prompts.

In recent years, text guided diffusion models have been developed, which transform either random noise, or a starting image, into a new image, guided by a text contextual prompt.

However, there is no effective method to use multimodal context guidance in diffusion models : a context that combines one or more passages of text with one or more images.

This is the key contribution of this paper : a method for image generation with multimodal contextual guidance (of arbitrary prompt length).

To this end, we develop a novel embedding model for multimodal guidance prompts, and we finetune a version of the text guided diffusion model, which we call m-vader, to follow contextual guidance.

We additionally show how to control the relative importance of the components of the guidance prompt, by reweighting the attention scores to each component.

Result

The level of image corruption has to be chosen carefully, since there is a tradeoff between being close to the original image and enforcing the desired style.

On the one hand, if only a little amount of noise is added to the image, the denoising process is very short and the conditional guidance has not enough steps to have a major effect.

On the other hand, degrading the image too much one loses all information about it and the result is a random image in the desired style.

We present a new paradigm for style change using a multimodal prompt.

We simply concatenate the style instruction with the image to form a single multimodal conditional prompt and then generate from pure noise.

This way we can use the full diffusion length to steer generation while maintaining information about the original image consistently.

Our model offers an alternate approach to the task of image variation by prompting with just a single image and includes examples of this in appendix .

Read the Paper

Read Paper

Home

Discover

Bookmarks

◐ Latest Activity

◐ Recommended Follows