Text-Guided 3D Diffusion Models

Home

Discover

Bookmarks

Keep Up With Latest Trending Papers. Computer Science, AI and Machine Learning and more.Subscribe

Bookmarked byEvan

CC BY

Source arXiv

Computer Vision

Text-Guided 3D Diffusion Models

3DDesigner: Towards Photorealistic 3D Object Generation and Editing with Text-guided Diffusion Models

Text-guided diffusion models have shown superior performance in image/video generation and editing.

While few explorations have been performed in 3dscenarios, we discuss three fundamental and interesting problems on this topic.

First, we equip text-guided models to achieve \textbf{3d-consistent generation}.

Specifically, we integrate a neural field to generate low-resolution coarse results for a given camera view.

Such results can provide 3d priors as condition information for the following diffusion process.

During denoising diffusion, we further enhance the 3dconsistency by modeling cross-view correspondences with a novel two-stream(corresponding to two different views) asynchronous diffusion process.

Second, we propose a two-step solution that can generate manipulated results by editing an object from a single view.

Step 1, we propose to perform 2d local editing by blending the predicted noises.

Step 2, we conduct a noise-to-text inversion process that maps 2dblended noises into the view-independent text embedding space.

Once the corresponding text embedding is obtained, images can be generated.

Last but not least, we extend our model to perform one-shot view synthesis by fine-tuning on a single image, firstly showing the potential of leveraging text guidance for novel view synthesis.

Authors

Gang Li, Heliang Zheng, Chaoyue Wang, Chang Li, Changwen Zheng, Dacheng Tao

Related Topics

Introduction

Text-guided diffusion models have revolutionized the field of image synthesis, showing excellent performance on high-fidelity, diverse and controllable generation.

Based on these models, various tools have been developed to empower humans to create rich and fantastical visual content.

However, few explorations have been performed in 3d scenarios.

Real-world objects are 3d, and it usually requires multi-view information to represent every detail of an object.

Meanwhile, creating a 3d asset requires very professional software and skills and may cost times of labor than creating its 2d counterpart.

Thus a tool capable of 3d object generation would be critical for real-world applications.

In this paper, we study text-guided diffusions for 3d novel object generation and manipulation.

Our proposed model consists of a condition module and a two-stream asynchronous diffusion module to further enhance the 3d consistency by modeling cross-view correspondences.

Result

In this paper, we introduce 3ddesigner, a text-guided diffusion model which is capable of generating and manipulating 3d objects.

Benefitting from the proposed novel diffusion training, sampling&blending strategies, and novel diffusion models, our 3ddesigner outperforms existing methods and achieves highly realistic, detailed, and controlled 3d object generation and manipulation.

Read the Paper

Read Paper