# Unsupervised Learning of Optical Flow via Brightness Constancy and Motion Smoothness

## Presented by

• Hudson Ash
• Stephen Kingston
• Richard Zhang
• Alexandre Xiao
• Ziqiu Zhu

## Optical Flow

Optical flow is the apparent motion of image brightness patterns in objects, surfaces and edges in videos. In more laymen terms, it tracks the change in position of pixels between two frames caused by the movement of the object or the camera, and it does this on the basis of two assumptions:

1. Pixel intensities do not change rapidly between frames (brightness constancy).

2. Groups of pixels move together (motion smoothness).

Both of these assumptions are derived from real-world implications. Firstly, the time between two consecutive frames of a video are so minuscule, such that it becomes extremely improbable for the intensity of a pixel to completely change, even if its location has changed. Secondly, pixels do not teleport. The assumption that groups of pixels move together implies that there is spacial coherence and that the image motion of objects changes gradually over time, creating motion smoothness.

Given these assumptions, imagine a video frame (which is 2D image) with a pixel at position $(x,y)$ at some time t, and in later frame, the pixel is now in position $(x + \Delta x, y + \Delta)$ at some time $t + \Delta t$.

Then by the first assumption, the intensity of the pixel at time t is the same as the intensity of the pixel at time $t + \Delta t$:

$I(x+\Delta x,y+\Delta y,t+\Delta t) = I(x,y,t)$

Using Taylor series, we get:

$I(x+\Delta x,y+\Delta y,t+\Delta t) = I(x,y,t) + \frac{\partial I}{\partial x}\Delta x+\frac{\partial I}{\partial y}\Delta y+\frac{\partial I}{\partial t}\Delta t$ ignoring the higher order terms.

From the two equations, it follows that:

$\frac{\partial I}{\partial x}\Delta x+\frac{\partial I}{\partial y}\Delta y+\frac{\partial I}{\partial t}\Delta t = 0$

which results in

$\frac{\partial I}{\partial x}V_x+\frac{\partial I}{\partial y}V_y+\frac{\partial I}{\partial t} = 0$

where $V_x,V_y$ are the $x$ and $y$ components of the velocity (displacement over time) or optical flow of $I(x,y,t)$ and $\tfrac{\partial I}{\partial x}$, $\tfrac{\partial I}{\partial y}$, and $\tfrac{\partial I}{\partial t}$ are the derivatives of the image at $(x,y,t)$ in the corresponding directions.

This can be rewritten as:

$I_xV_x+I_yV_y=-I_t$

or

$\nabla I^T\cdot\vec{V} = -I_t$

Since this results in one equation with two unknowns $V_x,V_y$, it results into what is known as the aperture problem of the optical flow algorithms. In order to solve the optical flow problem, another set of constraints are required.