This Neural Network Restores Old Videos

TL;DR

Neural network-based approach can restore and colorize old black and white movies, fixing artifacts, contrast issues, and missing data.

Transcript

Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér. In this series, we often discuss a class of techniques by the name image inpainting. Image inpainting methods are capable of filling in missing details from a mostly intact image. You see the legendary PatchMatch algorithm at work here, which is more than 10 years old, and i... Read More

Key Insights

• ⚾ The PatchMatch algorithm, a non-neural network-based method, has been successful in image inpainting for over 10 years.
• ⚾ Neural network-based learning methods show promise in expanding image inpainting to movies, addressing more complex problems.
• 🥶 The approach includes restoration and colorization steps, significantly improving the quality of old movies.
• 🖼️ The neural network architecture involves spatial and temporal convolution layers to achieve effective colorization across frames.
• ⏮️ The technique compares favorably to previous restoration and colorization methods, with more stable colorization and fewer artifacts.
• ❓ Quantitative results show a notable improvement of 3-4 decibels, indicating a substantial difference in image quality.
• 🥶 The technique's results suggest the potential for reviving old movies and giving them a much-deserved facelift.

Questions & Answers

Q: How does the neural network-based approach fix artifacts and contrast issues in old movies?

The restoration step of the approach addresses artifacts and contrast issues, resulting in a significantly improved version of the movie. It provides a detailed report of the technique's actions, showcasing the improvements.

Q: How does the colorization process work in the neural network-based approach?

The approach uses only six colorized reference images provided by the user as art direction. It propagates the colorization to the rest of the frames, achieving impressive results. The technique even indicates which reference image it is using for colorizing specific frames.

Q: How does this technique compare to previously published restoration and colorization methods?

This technique compares favorably to previous methods. The colorization is more stable over time, and fewer artifacts make it to the final footage. Quantitative results show a significant improvement of 3-4 decibels, indicating a substantial difference.

Q: What is the potential impact of this technique for old movies?

The results of this technique are approaching a quality level where it may be possible to revive old black and white movies and give them a much-needed facelift. This could bring new life to classic films.

Summary & Key Takeaways

• The PatchMatch algorithm has been a successful image inpainting method in computer graphics for over 10 years.

• Neural network-based learning methods can now inpaint and colorize movies, addressing problems like missing data, flickering, blurriness, and contrast changes.

• The approach involves restoration to fix artifacts and contrast, and colorization using reference images for art direction.