AI Learns 3D Face Reconstruction | Two Minute Papers #198

TL;DR

Convolutional Neural Networks can generate 3D face reconstructions from a single 2D photograph, achieving impressive results.

Transcript

Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér. Now that facial recognition is becoming more and more of a hot topic, let's talk a bit about 3D face reconstruction! This is a problem where we have a 2D input photograph, or a video of a person, and the goal is to create a piece of 3D geometry from it. To accomplish this, p... Read More

Key Insights

• 😀 3D face reconstruction can be achieved using only a single 2D photograph, eliminating the need for additional data or multiple photographs.
• 😀 Convolutional Neural Networks are effective in learning the mapping between 2D input photographs and 3D face geometry.
• 😀 The output representation of the 3D face reconstruction is a 3D voxel array, similar to Minecraft, allowing for accurate reconstruction of facial features.
• 😀 An online demo is available for users to try the 3D face reconstruction algorithm with their own photographs.
• 📼 The limitations of this technique include difficulty in detecting expressions outside the training set and minor differences in reconstructions for video input.
• 😒 Future improvements could involve exploring the use of Recurrent Neural Networks, such as Long Short Term Memory, to enhance temporal coherence in reconstructions.
• 👨‍💻 The source code for this algorithm is available for those interested in experimenting and tinkering with it.

Questions & Answers

Q: How does 3D face reconstruction differ from traditional methods?

Traditional methods require multiple photographs and extra data, while this new approach only needs a single 2D photograph.

Q: How was the algorithm trained to generate 3D geometry from 2D photographs?

The algorithm was trained on a dataset of 2D input images paired with their corresponding 3D output geometry using Convolutional Neural Networks.

Q: What is the output representation of the 3D face reconstruction?

The output is represented as a 3D voxel array, where the face is built from small, identical Lego-like pieces, allowing for fine resolution.

Q: Are there any limitations to this technique?

The technique struggles with detecting expressions far from those seen in the training set, and there are minor differences in reconstructions when applied to video input.

Summary & Key Takeaways

• Traditional 3D face reconstruction methods require multiple photographs and additional data, but this new approach only needs one 2D photograph.

• The algorithm was trained on a large dataset of 2D input image and 3D output geometry pairs using Convolutional Neural Networks.

• The output is represented as a 3D voxel array, similar to Minecraft, allowing for accurate reconstructions of arbitrary face positions, expressions, and occlusions.