NVIDIA Vid2Vid: AI-Based Video-to-Video Synthesis!

TL;DR

A new algorithm takes the pix2pix concept to the next level by animating edge maps into realistic human faces, as well as generating animations from labeled maps and achieving temporal coherence.

Transcript

Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér. Do you remember the amazing pix2pix algorithm from last year? It was able to perform image translation, which means that it could take a daytime image and translate it into a nighttime image, create maps from satellite images, or create photorealistic shoes from a crude draw... Read More

Key Insights

• 🌚 The new algorithm builds upon the pix2pix algorithm and extends its capabilities to animate edge maps into human faces.
• 👻 It can also generate animations from labeled maps, allowing for easy changes in object classes.
• 💐 The algorithm achieves temporal coherence by using a flow map and remembering past images, resulting in smoother videos.
• 😒 The use of two discriminator networks ensures both the quality of individual images and the temporal coherence of the image sequence.
• ❓ The training process for the algorithm is progressive, starting with an easier version of the problem and gradually increasing the difficulty.
• 🎮 The algorithm supports up to 2k resolution and 30 seconds of video.
• 👨‍💻 The source code for the algorithm is available.

Questions & Answers

Q: What was the previous algorithm that the new one builds upon?

The new algorithm is an extension of the pix2pix algorithm, which was capable of performing image translation, turning daytime images into nighttime images, creating maps from satellite images, and generating photorealistic shoes from rough drawings.

Q: How does the algorithm transform edge maps into human faces?

The algorithm uses a generator neural network and two discriminator networks. One discriminator judges the quality of individual images, while the other ensures the temporal coherence of the image sequence. This results in minimal flickering and realistic animated human faces.

Q: Can the algorithm also generate animations from labeled maps?

Yes, the algorithm can generate animations by following the evolution of labeled maps in time. It allows for easy changes in object classes, transforming buildings into trees or vice versa, for example.

Q: How does the algorithm achieve temporal coherence and generate smoother videos?

The algorithm achieves temporal coherence by using a flow map that describes changes occurring since the previous frame. This allows the algorithm to remember past images and generate videos with minimal flickering, resulting in smoother animations.

Summary & Key Takeaways

• The new algorithm transforms edge maps into animated human faces, creating multiple options for different faces from the same edges.

• It can also generate animations from labeled maps, allowing for easy changes in object classes.

• The algorithm achieves temporal coherence, generating smoother videos by remembering past images and making minimal flickering in the output.