How Does NVIDIA’s New AI Grow Virtual Objects Out of Images?

TL;DR

NVIDIA’s new AI turns one image or a set of images into a virtual object by reconstructing its geometry, materials, and lighting setup with high fidelity. A usable model emerges after as little as two minutes, while an hour produces an even better result. The reconstructed assets can be edited, relit, animated, and placed in physics or cloth simulations. Read on to see how the technique also handles entire scenes.

Transcript

Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér. Today we are going to see how NVIDIA’s new AI transfers real objects into a virtual world. So, what is going on here? Simple, in goes just one image, or a set of images of an object. And the result is that an AI really transfers this real object into a virtual world almo... Read More

Key Insights

• 🤔 The current AI technology is able to transfer real objects into a virtual world, but the results are still quite coarse in terms of geometry and materials.
• 😮 NVIDIA's new work promises to reconstruct everything - geometry, materials, and even the lighting setup - with high fidelity.
• 💡 The AI improves its results over time, and after just two minutes, it creates a usable model. ⏰ With more time, such as an hour, the AI can generate even better results.
• 📸 This technology can be applied to historical artifacts, allowing for virtual versions to be edited or used in physics simulations and cloth simulations.
• 🎨 Lighting can be changed around the virtual objects, giving users a better understanding of how they would look in different environments.
• 🍔 The AI can even handle virtual versions of everyday objects, such as hot dogs, allowing for fun interactions like throwing jelly boxes at them.
• 🌍 The AI is not limited to individual objects, as it can also reconstruct entire scenes with impressive accuracy, making it a powerful tool for creating virtual worlds.

Questions & Answers

Q: How does NVIDIA’s new AI transfer real objects into virtual worlds?

The AI takes one image or a set of images of an object and reconstructs a virtual version. Unlike earlier approaches that captured only geometry or produced coarse materials, NVIDIA’s work aims to reproduce geometry, materials, and the lighting setup with high fidelity.

Q: How quickly can NVIDIA’s AI create a usable virtual model?

The initial result improves progressively as the AI works. After as little as two minutes, it produces a usable model, and the result becomes even better after an hour.

Q: What does NVIDIA’s AI reconstruct from the source images?

The technique promises to reconstruct the object’s geometry, material models, and lighting setup. This combines the strengths of earlier work that either reconstructed geometry well or attempted a coarser reconstruction of everything.

Q: Why is NVIDIA’s approach an improvement over earlier reconstruction methods?

One earlier method reconstructed target geometry well but did not reconstruct materials. Another attempted both geometry and materials, but both remained quite coarse; NVIDIA’s new work tries to deliver all components with high fidelity.

Q: Can the reconstructed virtual objects be edited or simulated?

Yes. A reconstructed object can be placed in a physics simulation engine, where its material can be edited or the object can be embedded in a cloth simulation.

Q: Can NVIDIA’s AI reconstruct historical artifacts from photographs?

Yes, the demonstrated workflow starts with a collection of photos of a historical artifact and creates a virtual version. That version can then be edited, relit, or used in physics and cloth simulations.

Q: Can users change the lighting and materials of reconstructed objects?

Yes. Users can change the lighting around a virtual object to see how it appears under different illumination, and they can edit its material after placing it in a simulation engine. The technique can even reimagine an existing solid object as if it were made of jelly.

Q: Can NVIDIA’s AI reconstruct an entire scene instead of one object?

Yes. The technique reconstructs an entire reference scene and retains a close resemblance even when the result is animated. In the transcript’s comparison, the presenter swaps the labels to make the reconstruction harder to identify.

Summary & Key Takeaways

• NVIDIA's AI can transfer real objects into virtual worlds by reconstructing their geometry, materials, and lighting setup.

• Previous works only focused on reconstructing geometry, but NVIDIA's AI aims to achieve high fidelity in all aspects.

• The AI improves the results over time, even allowing for virtual objects to be edited and simulated in physics engines.