spiderman DreamScene360

Unconstrained Text-to-3D Scene Generation with Panoramic Gaussian Splatting

Shijie Zhou1*, Zhiwen Fan2*, Dejia Xu2*, Haoran Chang1, Pradyumna Chari1, Tejas Bharadwaj1, Suya You3, Zhangyang Wang2, Achuta Kadambi1
1University of California, Los Angeles 2University of Texas at Austin 3DEVCOM ARL
*Denotes Equal Contribution
ECCV 2024, Milan
Paper Video Code
teaser gif
We introduce a 3D scene generation pipeline that creates immersive, high-quality scenes with complete 360∘ coverage from text prompts of any level of specificity, leveraging GPT-4V with 2D diffusion model and panoramic Gaussian splatting to achieve exceptional content quality and rendering speed.

Abstract

The increasing demand for virtual reality applications has highlighted the significance of crafting immersive 3D assets. We present a text-to-3D 360∘ scene generation pipeline that facilitates the creation of comprehensive 360∘ scenes for in-the-wild environments in a matter of minutes. Our approach utilizes the generative power of a 2D diffusion model and prompt self-refinement to create a high-quality and globally coherent panoramic image. This image acts as a preliminary "flat" (2D) scene representation. Subsequently, it is lifted into 3D Gaussians, employing splatting techniques to enable real-time exploration. To produce consistent 3D geometry, our pipeline constructs a spatially coherent structure by aligning the 2D monocular depth into a globally optimized point cloud. This point cloud serves as the initial state for the centroids of 3D Gaussians. In order to address invisible issues inherent in single-view inputs, we impose semantic and geometric constraints on both synthesized and input camera views as regularizations. These guide the optimization of Gaussians, aiding in the reconstruction of unseen regions. In summary, our method offers a globally consistent 3D scene within a 360∘ perspective, providing an enhanced immersive experience over existing techniques.

Video

Approach

Pipeline Image

Beginning with a concise text prompt, we employ a diffusion model to generate a 360∘ panoramic image. A self-refinement process is employed to produce the optimal 2D candidate panorama. Subsequently, a 3D geometric field is utilized to initialize the Panoramic 3D Gaussians. Throughout this process, both semantic and geometric correspondences are employed as guiding principles for the optimization of the Gaussians, aiming to address and fill the gaps resulting from the single-view input.

Real Unconstrained Scene Generation from Any Text

Unlike previous works that generate 3D scenes through a time-consuming score distillation process or progressive inpainting, our work uses Panoramic Gaussian Splatting to achieve ultra-fast “one-click” 3D scene generation. We integrate GPT-4V to facilitate iterative user input refinement and quality evaluation during the generation process, a feature that was challenging to implement in previous methods due to the absence of global 2D image representations.

Specific Text Input

Vague Text Input

Real Immersive Scene with 360∘ Coverage

In each row, from left to right, displays novel views as the camera undergoes clockwise rotation in yaw, accompanied by slight random rotations in pitch and random translations. LucidDreamer hallucinates novel views from a conditioned image (indicated by a red bounding box) but lacks global semantic, stylized, and geometric consistency. In contract, our method provides complete 360∘ coverage without any blind spots (black areas in baseline results), and show globally consistent semantics.

Indoor Scene

indoor compare

Outdoor Scene

outdoor compare

Real 3D with Diverse Style and High-quality Geometry

Our generated 3D scenes are diverse in style, consistent in geometry, and highly matched with the simple text inputs.

BibTeX

@inproceedings{zhou2024dreamscene360,
      title={Dreamscene360: Unconstrained text-to-3d scene generation with panoramic gaussian splatting},
      author={Zhou, Shijie and Fan, Zhiwen and Xu, Dejia and Chang, Haoran and Chari, Pradyumna and Bharadwaj, Tejas and You, Suya and Wang, Zhangyang and Kadambi, Achuta},
      booktitle={European Conference on Computer Vision},
      pages={324--342},
      year={2024},
      organization={Springer}
}