Selfie to 3D Model - Computerphile

TL;DR

Researchers have developed a tool that can convert a 2D photograph into a 3D volumetric model of a face using Convolutional Neural Networks (CNNs).

Transcript

the idea is to take a single photograph of a face and as outputs we get a 3d model in volumetric form where do you start doing something like that this kind of thing is becoming more common now with cnns you can give them a problem and just let them try and solve it this is convolutional neural networks right yeah that's right so the idea is that u... Read More

Key Insights

• ❓ Convolutional Neural Networks (CNNs) can be used to convert 2D photographs into 3D volumetric models.
• 🍵 The stacked hourglass architecture handles the progressive reduction and up-sampling of the image to improve resolution and detail.
• 🖤 The system has good performance but lacks detailed texturing, focusing primarily on facial shape reconstruction.
• 🙂 The 3D volumes produced by the system are spatially aligned with the original face, including slight rotations.

Questions & Answers

Q: How does the system convert a 2D photograph into a 3D model?

The system uses a CNN architecture called the stacked hourglass, which progressively reduces the spatial resolution of the image until it reaches around 10x10 pixels. It then up-samples the image while combining the results from smaller CNNs to increase resolution and detail. The output is a 3D volumetric representation.

Q: How accurate is the 3D reconstruction compared to actual 3D scans?

Extensive testing and error calculations have been performed to evaluate the accuracy of the system. While the 3D volumes lack details like wrinkles or spots, they generally match the shape of the face accurately.

Q: What happens if the input photograph shows a person in a side pose?

The system currently doesn't work well with side poses because the face detector used doesn't recognize these faces. Instead of displaying poor results, the system is designed to prioritize visual appeal and only produce good-looking outputs.

Q: How does the system handle hair in the 3D model?

The training dataset used for the system only contains the 3D shape of the face without including hair. Incorporating the back of the head or hair would require additional time-consuming voxel calculations, so it's currently not implemented.

Summary & Key Takeaways

• The research project aims to create a user-friendly online demonstration for generating 3D models of faces from photographs.

• The system utilizes CNNs and filters to learn the values for the 3D reconstruction.

• The resulting 3D mesh can be downloaded and viewed, although the texturing is simple and can be distorted in some cases.