# Difference between revisions of "Unsupervised Learning of Optical Flow via Brightness Constancy and Motion Smoothness"

(→Optical Flow) |
(→Traditional Approaches) |
||

Line 40: | Line 40: | ||

== Traditional Approaches == | == Traditional Approaches == | ||

− | Traditional approaches to the optical flow problem consisted of many differential (gradient-based) methods. Horn and Schunck, 1981, being one of the first to create an approach for for optical flow estimation, | + | Traditional approaches to the optical flow problem consisted of many differential (gradient-based) methods. Horn and Schunck, 1981, being one of the first to create an approach for for optical flow estimation, is one of the classical examples for optical flow estimation. Without diving deep into the math, Horn and Schunk created constraints based on spatio-temporal derivatives of image brightness. Their estimation tries to solve the aperture problem by adding a smoothness condition where that the optical flow field varies smoothly through the entire image (a global motion smoothness). They assume that object motion in a sequence will be rigid and approximately constant, that the objects in a pixel’s neighborhood will have similar velocities, and therefore the object changes smoothly over space and time. The challenges with this approach is that on frames with rougher movements, the accuracy of the estimates dramatically decrease. |

− | + | Another classical method, Lucas-Kanade, approaches the problem by taking a local motion smoothness assumption. Lucas-Kanade addressed the sensitivity to rough movements in the Horn and Schunk approach by making a local motion smoothness assumption instead of a global motion smoothness. While the Lucas-Kanade estimation reduced sensitivity to rougher movements, it still has inaccuracy in rough frames as a differential method to optical flow estimation. | |

− | |||

− | |||

− | |||

− | |||

− | |||

− | has a | ||

− | |||

− | |||

− | |||

− | |||

− | |||

It wasn't until 2015 that FlowNet [Dosovitskiy et al., 2015] was proposed as the first approach to use a deep neural network for end-to-end optical flow estimation. | It wasn't until 2015 that FlowNet [Dosovitskiy et al., 2015] was proposed as the first approach to use a deep neural network for end-to-end optical flow estimation. | ||

− | |||

− | |||

== Problem & Motivation == | == Problem & Motivation == |

## Revision as of 16:38, 20 November 2018

## Presented by

- Hudson Ash
- Stephen Kingston
- Richard Zhang
- Alexandre Xiao
- Ziqiu Zhu

## Optical Flow

Optical flow is the apparent motion of image brightness patterns in objects, surfaces and edges in videos. In more laymen terms, it tracks the change in position of pixels between two frames caused by the movement of the object or the camera. Most optical flows are estimated on the basis of two assumptions:

1. Pixel intensities do not change rapidly between frames (brightness constancy).

2. Groups of pixels move together (motion smoothness).

Both of these assumptions are derived from real-world implications. Firstly, the time between two consecutive frames of a video are so minuscule, such that it becomes extremely improbable for the intensity of a pixel to completely change, even if its location has changed. Secondly, pixels do not teleport. The assumption that groups of pixels move together implies that there is spacial coherence and that the image motion of objects changes gradually over time, creating motion smoothness.

Given these assumptions, imagine a video frame (which is 2D image) with a pixel at position [math] (x,y) [/math] at some time t, and in later frame, the pixel is now in position [math](x + \Delta x, y + \Delta) [/math] at some time [math] t + \Delta t [/math].

Then by the first assumption, the intensity of the pixel at time t is the same as the intensity of the pixel at time [math] t + \Delta t [/math]:

- [math]I(x+\Delta x,y+\Delta y,t+\Delta t) = I(x,y,t)[/math]

Using Taylor series, we get:

- [math]I(x+\Delta x,y+\Delta y,t+\Delta t) = I(x,y,t) + \frac{\partial I}{\partial x}\Delta x+\frac{\partial I}{\partial y}\Delta y+\frac{\partial I}{\partial t}\Delta t[/math] ignoring the higher order terms.

From the two equations, it follows that:

- [math]\frac{\partial I}{\partial x}\Delta x+\frac{\partial I}{\partial y}\Delta y+\frac{\partial I}{\partial t}\Delta t = 0[/math]

which results in

- [math]\frac{\partial I}{\partial x}V_x+\frac{\partial I}{\partial y}V_y+\frac{\partial I}{\partial t} = 0[/math]

where [math]V_x,V_y[/math] are the [math]x[/math] and [math]y[/math] components of the velocity (displacement over time) or optical flow of [math]I(x,y,t)[/math] and [math]\tfrac{\partial I}{\partial x}[/math], [math]\tfrac{\partial I}{\partial y}[/math], and [math]\tfrac{\partial I}{\partial t}[/math] are the derivatives of the image at [math](x,y,t)[/math] in the corresponding directions.

This can be rewritten as:

- [math]I_xV_x+I_yV_y=-I_t[/math]

or

- [math]\nabla I^T\cdot\vec{V} = -I_t[/math]

Where [math] \nabla I^T [/math] is known as the spatial gradient

Since this results in one equation with two unknowns [math]V_x,V_y[/math], it results into what is known as the aperture problem of the optical flow algorithms. In order to solve the optical flow problem, another set of constraints are required, which is where assumption 2 can be applied.

## Traditional Approaches

Traditional approaches to the optical flow problem consisted of many differential (gradient-based) methods. Horn and Schunck, 1981, being one of the first to create an approach for for optical flow estimation, is one of the classical examples for optical flow estimation. Without diving deep into the math, Horn and Schunk created constraints based on spatio-temporal derivatives of image brightness. Their estimation tries to solve the aperture problem by adding a smoothness condition where that the optical flow field varies smoothly through the entire image (a global motion smoothness). They assume that object motion in a sequence will be rigid and approximately constant, that the objects in a pixel’s neighborhood will have similar velocities, and therefore the object changes smoothly over space and time. The challenges with this approach is that on frames with rougher movements, the accuracy of the estimates dramatically decrease.

Another classical method, Lucas-Kanade, approaches the problem by taking a local motion smoothness assumption. Lucas-Kanade addressed the sensitivity to rough movements in the Horn and Schunk approach by making a local motion smoothness assumption instead of a global motion smoothness. While the Lucas-Kanade estimation reduced sensitivity to rougher movements, it still has inaccuracy in rough frames as a differential method to optical flow estimation.

It wasn't until 2015 that FlowNet [Dosovitskiy et al., 2015] was proposed as the first approach to use a deep neural network for end-to-end optical flow estimation.

## Problem & Motivation

The approaches to solving optimal flow problems, albeit widely successful, has mostly been a result of supervised learning methods using convolutional neural networks (convnets). The inherent challenge with these supervised learning approaches lies in the groundtruth flow, the process of gathering provable data for the measure of the target variable for the training and testing datasets. Directly obtaining the motion field ground-truth is not possible but instead, segmentation ground-truthing is generally used. Segmentation ground-truth is the classification of all pixels in an image. Since the segmentation ground-truthing isn't always automated, it requires laborious labeling of items in the video, sometimes even requiring manually using a ground-truth labeling software. Then as the training and test datasets become larger in size, the more laborious the segmentation ground-truthing becomes.

In the case of the KITTI dataset, a collection of images captured from driving cars around a mid-sized city in Germany, accurate segmentation ground truth for the training and testing data is obtained using high-tech laser scanners, as well as a GPS localization device installed onto the top of the cars.

The paper "Back to Basics: Unsupervised Learning of Optical Flow via Brightness Constancy and Motion Smoothness" by Jason J. Yu, Adam W. Harley and Konstantinos G. Derpanis presents an unsupervised approach to address the supervised challenges of optical flow.