Project 5: Tracking and Structure from Motion
CS 143: Introduction to Computer Vision

Overview

In this project, we estimate optical flow from short and simple video sequences, and construct the 3D structure following structure to motion algorithms.

Details

Key Point Detection

In this project we use the harris corner detector to find key points. These points are easy to detect and persist over frames. An example visualization of the key points is shown below.

A sample visualization of harris corner key point detection results.

Optical Flow

The next step is to track the key points using standard optical flow algorithm. In particular, we implement a basic KLT tracker to track the strongest 500 points throughout the sequence. To avoid noisy tracking results, I apply a median filtering step (5x5 window) to the flow field.

A sample visualization of optical flow field from the basic KLT tracker.

A sample visualization of 20 tracks from the video sequence showing the motion of the key points.

Some points will be occluded of go out of bounds in subsequent frames. We discard these points to avoid future problems. Below is a sample visualization of the points that go out of bounds, shown with the first frame:

Structure from Motion

Using the keypoint tracks computed above, we implement the structure from motion procedure described in Shape and Motion from Image Streams under Orthography: a Factorization Method (Tomasi and Kanade 1992). Affine ambiguity is eliminated using the computation outlined in A Sequential Factorization Method for Recovering Shape and Motion from Image Streams, Morita and Kanade, 1997.

Below are several visualization of the structure recovered using the above algorithm