eDiffi: Large-scale diffusion-based generative models for text-to-image synthesis

Home

Discover

Bookmarks

Keep Up With Latest Trending Papers. Computer Science, AI and Machine Learning and more.Subscribe

eDiffi: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers

Large-scale text-to-image diffusion models have led to breakthroughs in text-conditioned high-resolution image synthesis.

Such models gradually synthesize images in an iterative fashion while conditioning on text prompts.

However, we find that their synthesis behavior qualitatively changes throughout this process : early in sampling, generation strongly relies on the text prompt to generate text-aligned content, while later, the text conditioning is almost entirely ignored.

This suggests that sharing model parameters throughout the entire generation process may not be ideal.

Our ensemble of diffusion models, called ediffi, results in improved text alignment while maintaining the same inference computation cost and preserving high visual quality, outperforming previous large-scale text-to-image diffusion models on the standard benchmark.

Authors

Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, Ming-Yu Liu

Related Topics

Introduction

Text-to-image diffusion models that generate images through iterative denoising are revolutionizing the field of image generation.

These models have led to the development of numerous interactive tools and creative applications and turbocharged the democratization of content creation.

This success is attributed largely to the great scalability of diffusion models because the scalability provides a clear pathway for practitioners to translate larger model capacity, compute, and datasets into better image generation quality.

Motivated by this observation, we propose to further scaling diffusion models in terms of model capability by training an ensemble of, each specialized for a particular stage in the generation process.

We first note that simply increasing the capacity by using deeper or wider neural networks for each denoising step will negatively impact the test-time computational complexity of sampling, since sampling amounts to solving a reverse (generative) differential equation in which a denoising network is called many times.

To remedy this, we propose pre-training a shared diffusion model for all stages.

We then use this pre-trained model to initialize specialized models and finetune them for a smaller number of iterations.

Result

We propose, a state-of-the-art text-to-image diffusion model that consists of a base diffusion model and two super-resolution modules, producing @xmath0 high-definition outputs.

Utilizes an ensemble of expert denoisers to achieve superior performance compared to previous work.

Our different expert denoiser networks allow us to specialize the model for different behaviors during different intervals of the iterative synthesis process.

We found that the generation process qualitatively changes throughout synthesis : initially, the model focuses on generating globally coherent content aligned with the text prompt, while later in the process, the model largely ignores the text conditioning and its primary goal is to produce visually high-quality outputs.

Moreover, we showed that by conditioning on both t5 text, clip text, and clip image embeddings, not only enjoys improved performance but also enables rich controllability.

In particular, the t5 and clip text embeddings capture complementary aspects of the generated images, and the image embedding further can be used for stylization according to reference images.

Finally, we demonstrated expressive spatial control using the model s ``paint-with-words''capability.

Read the Paper

Read Paper

Bookmarks

Bookmarks

Keep Up With Latest Trending Papers. Computer Science, AI and Machine Learning and more.Subscribe

eDiffi: Large-scale diffusion-based generative models for text-to-image synthesis

eDiffi: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers

Large-scale text-to-image diffusion models have led to breakthroughs in text-conditioned high-resolution image synthesis.

Such models gradually synthesize images in an iterative fashion while conditioning on text prompts.

This suggests that sharing model parameters throughout the entire generation process may not be ideal.

Authors

Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, Ming-Yu Liu

Related Topics

Introduction

Text-to-image diffusion models that generate images through iterative denoising are revolutionizing the field of image generation.

These models have led to the development of numerous interactive tools and creative applications and turbocharged the democratization of content creation.

To remedy this, we propose pre-training a shared diffusion model for all stages.

We then use this pre-trained model to initialize specialized models and finetune them for a smaller number of iterations.

Result

We propose, a state-of-the-art text-to-image diffusion model that consists of a base diffusion model and two super-resolution modules, producing @xmath0 high-definition outputs.

Utilizes an ensemble of expert denoisers to achieve superior performance compared to previous work.

Our different expert denoiser networks allow us to specialize the model for different behaviors during different intervals of the iterative synthesis process.

Moreover, we showed that by conditioning on both t5 text, clip text, and clip image embeddings, not only enjoys improved performance but also enables rich controllability.

In particular, the t5 and clip text embeddings capture complementary aspects of the generated images, and the image embedding further can be used for stylization according to reference images.

Finally, we demonstrated expressive spatial control using the model s ``paint-with-words''capability.

Read the Paper

Read Paper

Home

Discover

Bookmarks

◐ Recommended Follows

◐ Latest Activity