An AI Learned To See Through Obstructions! 👀

TL;DR

This paper introduces a learning-based method that can remove obstructions from image sequences, revealing what is behind them, and also performs well with video outputs.

Transcript

Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér. Approximately two years ago, we covered a work where a learning-based algorithm was able to read the wifi signals in a room to not only locate a person in a building, but even estimate their pose. An additional property of this method was that, as you see here, it does n... Read More

Key Insights

• 🫵 The learning-based method can remove obstructions in image sequences, providing a clear view of the background.
• 👻 It is also effective in removing reflections, allowing viewers to focus on the actual background instead of distractions.
• 👋 The technique shows good temporal coherence in video outputs, minimizing flickering.
• 💨 The algorithm performs well even without online optimization, making it a fast and efficient option.
• ❓ The results of the technique surpass other existing methods, highlighting its effectiveness.
• 😎 The potential applications of this technology in AR glasses for glare, reflection, and obstruction removal are promising.
• ✊ The learning-based approach demonstrates the power of utilizing image sequences for data completion in various scenarios.

Questions & Answers

Q: How does the learning-based method remove obstructions in image sequences?

The method uses multiple viewpoints from an image sequence to reconstruct what is behind obstructions by leveraging the visibility of the scene from different angles.

Q: Can the algorithm also remove reflections from images?

Yes, the algorithm can decompose an image into two images – one with the reflections and one without, effectively removing distracting reflections from the background.

Q: Does the technique exhibit temporal coherence in video outputs?

While there is a tiny amount of flickering, the results are surprisingly consistent, indicating a good level of temporal coherence in the output.

Q: What is the difference between the quicker and slower versions of the technique?

The slower version includes an online optimization step that improves separation in the outputs. However, even without this step, the quicker version outperforms other methods in terms of speed and quality.

Summary & Key Takeaways

• The learning-based algorithm can remove obstructions in image sequences, such as fences, and show the background that is obscured.

• It is also capable of removing reflections from images, improving the clarity of the background.

• The technique performs consistently well in video outputs, with only minimal flickering observed.