What Are NERFIES and How Do They Work?

TL;DR

NERFIES, or Deformable Neural Radiance Fields, allow users to create 360-degree rotating portraits from selfie videos, rendering realistic images of moving objects. This advancement builds on the original NERF technique and significantly improves handling of occlusions, variable lighting, and intricate details like hairstyles and glasses, making it suitable for both humans and dogs.

Transcript

Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér. Today we are going to get a taste of how insanely quick progress is in machine learning research. In March of 2020, a paper appeared that goes by the name Neural Radiance Fields, NERF in short. With this technique, we could take a bunch of input photos, get a neural netw... Read More

Key Insights

• 👨‍🔬 Neural Radiance Fields (NERF) research has made immense progress in just nine months, with significant improvements in handling occlusions and capturing realistic scenes.
• 🤝 NERF-W specializes in tourist attractions and deals with variable lighting conditions and crowded environments.
• 🔚 Deformable Neural Radiance Fields (D-NERF) enables the creation of 360-degree rotatable portraits with realistic rendering of moving objects.
• 😎 D-NERF successfully handles challenging cases with beards, various hairstyles, and glasses.
• 😘 The technique performs well on both humans and dogs, although lower resolution outputs are expected for the latter.
• 🪈 D-NERF can achieve the iconic dolly zoom effect and even allows for nerfception, recording oneself while recording.
• 👾 Machine learning research in the field of Neural Radiance Fields is progressing at an astonishing pace, demonstrating the rapid evolution of the technology.

Questions & Answers

Q: What is the main difference between NERF and NERF-W?

NERF focuses on synthesizing new views of scenes, while NERF-W specializes in handling occlusions in tourist attractions with various lighting conditions and lots of people.

Q: What is the goal of Deformable Neural Radiance Fields (D-NERF)?

D-NERF aims to transform selfie videos into realistic and freely rotatable portraits by extending the NERF technique with deformable capabilities.

Q: Does D-NERF perform well with moving objects?

Yes, D-NERF addresses the limitations of the original NERF technique by effectively rendering moving objects in portraits, eliminating artifacts seen in NERF-rendered images.

Q: What additional features does D-NERF offer?

D-NERF can accurately render reflections and refractions on glasses, capture full-body images, create dolly zoom effects, and even allow nerfception by recording oneself while recording.

Summary & Key Takeaways

• Neural Radiance Fields (NERF) technique can synthesize new views of entire scenes from a few input photos, including real-world scenes.

• NERF in the Wild (NERF-W) specializes in handling occlusions in tourist attractions with a large number of people.

• Deformable Neural Radiance Fields (D-NERF) extends the NERF technique to create 360-degree rotating "nerfies" with improved rendering of moving objects.