Amazing Slow Motion Videos With Optical Flow | Two Minute Papers #119

TL;DR

Optical flow creates smoother slow-motion video by estimating motion between known frames and synthesizing new frames to replace duplicates. For example, placing 25 or 30 frames-per-second footage on a 60 frames-per-second timeline leaves roughly every other frame duplicated. Optical flow can outperform frame blending, but abrupt, high-frequency changes may produce artifacts. Read on to understand when each technique works best and how they differ.

Transcript

Dear Fellow Scholars, this is Two Minute Papers with Károly Zsolnai-Fehér. I am really excited to show this to you as I was looking to make this episode for quite a while. You'll see lots of beautiful slow-motion footage during the narration. And at first, it may seem disconnected from the narrative, but by the end of video, you'll understand why t... Read More

Key Insights

• 🐢 Optical flow enables interpolation and extrapolation in videos, improving the smoothness and visual quality of slow-motion footage.
• 🖼️ Frame interpolation fills in duplicated frames with new information, while frame blending averages the results without understanding the motion.
• 💐 Optical flow has applications beyond video processing, such as motion estimation for unmanned aerial vehicles.
• 😒 Individual judgement and trial and error are often required to determine the appropriate use of optical flow and frame blending.
• ✋ The support from the Patreon community has allowed for the enhancement of Two Minute Papers episodes by implementing frame blending and optical flow on higher-resolution and higher-frame-rate footage.
• ⌛ The ultimate goal is to make Two Minute Papers a full-time endeavor, producing even more elaborate episodes with the continued support of Fellow Scholars.

Questions & Answers

Q: How does optical flow create smooth slow-motion videos?

Optical flow estimates the motion occurring between known images and synthesizes intermediate frames containing new information. This can replace duplicated frames and produce smooth slow-motion footage that looks almost like it was captured with a slow-motion camera.

Q: What is the difference between interpolation and extrapolation in optical flow?

Interpolation estimates what happened between two known measurement points, such as a person’s location between four and five o’clock. Extrapolation predicts beyond the available samples, such as what happens after the final frame of a video.

Q: Why are frames duplicated when 25 or 30 fps footage is placed on a 60 fps timeline?

A 60 frames-per-second timeline requires more frames than 25 or 30 frames-per-second footage provides. As a result, roughly every other frame is duplicated and contains no new information.

Q: How does frame interpolation replace duplicated video frames?

Frame interpolation uses before-and-after image pairs to estimate what happened between them. An optical flow algorithm can then replace doubled frames with synthesized frames that carry new visual information.

Q: What are the limitations of optical flow for frame interpolation?

The basic optical flow equations are designed for tiny changes in position, so they are not expected to work reliably across long time intervals. Abrupt, high-frequency changes are especially difficult to predict and can create nasty visual artifacts.

Q: How is frame blending different from optical flow?

Frame blending simply computes an average of two frames without meaningfully estimating the motion between them. It is less likely to produce nasty artifacts, but its results are significantly more limited because it does not understand the motion in the footage.

Q: When should frame blending be used instead of optical flow?

Frame blending can be preferable when optical flow creates artifacts, particularly around abrupt, high-frequency changes. Until an algorithm can adaptively choose between the methods, the decision requires individual judgment and sometimes considerable trial and error.

Q: What other applications does optical flow have besides slow-motion video?

Optical flow can support motion estimation for an unmanned aerial vehicle. With an optical flow sensor, the vehicle can determine which direction to rotate to avoid a tree and assess whether it is stable at a given moment.

Summary & Key Takeaways

• Optical flow can interpolate frames in videos by estimating motion between known measurement points, providing new information and improving the viewing experience.

• Frame blending is a simpler technique that averages the results of frame interpolation, but it lacks an understanding of the motion in the footage.

• The use of optical flow and frame blending depends on individual judgement and the specific requirements of the video.