Waymo's AI Recreates San Francisco From 2.8 Million Photos! 🚘

TL;DR

Waymo's AI uses NERF-based technique to recreate a 3D model of San Francisco from 2.8 million photos, allowing for new paths, different viewpoints, and appearance modulation.

Transcript

Dear Fellow Scholars, this is Two Minute  Papers with Dr. Károly Zsolnai-Fehér. Today we are going to see if Waymo’s AI   can recreate a virtual copy of San  Francisco from 2.8 million photos. To be able to do that, they rely on a previous  AI-based technique that can take a collection   of photos like these, and magically, create a  video where we... Read More

Key Insights

• 😒 Waymo's AI uses a NERF-based technique to create a virtual copy of San Francisco from 2.8 million photos.
• 👻 The AI-generated 3D model allows for exploration of new paths, different viewpoints, and appearance modulation.
• 😨 Sim2real applications are relevant for self-driving cars, as virtual training in unlikely scenarios can enhance real-world performance.
• 👻 Waymo's technique showcases significant improvements in just a few years, allowing for the creation of entire city blocks in virtual worlds.
• 🏙️ The technique can be used for artistic vision and customization of virtual cities.
• ❓ Limitations include resolution and details, as well as challenges with dynamic moving objects.
• 🌍 Waymo's AI technique aligns with the trend of sim2real and training AI in simulated worlds before deploying in real-world scenarios.

Questions & Answers

Q: How does Waymo's AI recreate a virtual copy of San Francisco?

Waymo's AI uses a NERF-based technique to fill in the gaps between 2.8 million photos of San Francisco, creating a 3D model of the city. The AI generates high-quality synthetic data to complete the missing information.

Q: What are some advantages of Waymo's virtual copy of San Francisco?

With the virtual copy, it is possible to explore new paths that haven't been driven before, view buildings from different viewpoints, and even choose the time of day for the virtual city. This opens up possibilities for various applications and artistic vision.

Q: How does sim2real relate to Waymo's AI techniques?

Sim2real refers to training an AI in a simulated world before deploying it in the real world. Waymo's technique allows for the creation of virtual worlds where self-driving cars can be trained in unlikely and potentially unsafe scenarios, providing them with additional knowledge and training before hitting the road.

Q: What are the limitations of Waymo's AI technique for creating virtual cities?

While the technique is impressive, there are still improvements needed in resolution and details. Additionally, the technique is better suited for stationary scenes, and dynamic moving objects pose challenges. Further research is needed to enhance these aspects.

Summary & Key Takeaways

• Waymo's AI uses a NERF-based technique to create a video by filling in the gaps between a collection of photos, resulting in a virtual city model.

• The AI-generated 3D model of San Francisco is mostly synthetic information and can be explored from different viewpoints and different lighting conditions.

• The technique has potential applications in sim2real, allowing AI to be trained in simulated worlds before being deployed in the real world.