3D for everyone? Nvidia’s Magic3D can generate 3D models from text

Magic3D is an artificial intelligence model that can generate 3D models. In about 40 minutes, Magic3D will generate a 3D mesh model, complete with colored texture, from the prompt "A blue poison-dart frog sitting on a water lily." Modifications can be used in video games.

The paper frames Magic3D as a response to DreamFusion. Magic3D uses a two-stage process that takes a coarse model generated in low resolution and improves it to higher resolution. The Magic3D method can generate 3D objects two times faster than DreamFusion.

Prompt-based editing of meshes is possible with Magic3D. It is possible to change the text in a low resolution model. The authors of Magic3D show how to apply the style of a 2D image to a 3D model.

The academic paper and Magic3D code were not released by the company.

The ability to generate 3D from text feels like a natural evolution in today's models, which use neural networks to create novel content after intense training. We've seen the emergence of text-to-image models such as DALL-E and Stable Diffusion and rudimentary text-to-video generators. People have adapted similar techniques to work with as an open source model based on Stable Diffusion since the debut of the text-to- 3D model DreamFusion.

The researchers behind Magic3D hope that it will allow anyone to create 3D models. The resulting technology could speed up video game development and possibly be used in special effects for film and TV. "We hope with Magic3D, we can open up everyone's creativity in 3D content creation," they wrote at the end of their paper.