CS 143 / Project 2 / Local Feature Matching

Project Description and Implementation Overview

The purpose of this project is to implement a basic image feature matching pipeline, which consists of three major components that I will briefly discuss before showcasing a few results of the full matching pipeline. These components are:

  1. Feature Extraction
  2. Local Feature Description
  3. Local Feature Matching

Feature Extraction

Feature extraction for this project is accomplished through an implementation of the Harris Corner Detector, which essentially ranks the "cornerness" of a point based on its gradient response to a sliding window around it. Once each point is ranked, the best ones in each region are found by supressing all non maxima in a region around each point. The output of such extraction looks like the image below (the original image is shown on the left.)

Original Image

Image of accented interest points

Feature Description

Features are described using a SIFT-like description of the normalized image gradients in 8 directions in 16 bins. Unfortunately, without a proper scaling pipeline or a strong resistance to affine transformations the descriptors suffer under large transformations of the image and under large changes in scale. The effectiveness of these features can be seen better in the matching examples below, as well as a comparison to a normalized pixel patch descriptor.

Feature Matching

Features are matched based on a nearest neighbors search, and each match is ranked according to the ratio of it's two nearest neighbor's distances, the smaller the ratio the stronger the match. When used on similar images, this matching scheme is fairly effective.

Examples

Feature Matching Using Normalized Patches (74 correct matches out of 100)

Feature Matching Using SIFT-like descriptors (93 correct matches out of 100)

As is evident, switching to the SIFT-like descriptor yields approximately a 25% increase in correct matches. However, even the SIFT-like descriptor has difficulty with large changes in scale, as evidenced by this next image pair.

Feature Matching Using SIFT-like descriptors on Images of Different scales (14 correct matches out of 40)

Finally, the results of feature matching on two images with a large transformation between them can be seen below, ground truth has not been collected, but it is clear to see that the descriptor does not do a great job of matching across such a large transformation.

Feature Matching Using SIFT-like descriptors on the Pantheon