face morph

Project 6: Automated Panorama Stitching

Due Date: 11:59pm on Saturday, March 17th, 2010

Brief

Requirements

You are required to implement the following:

Details

MATLAB stencil code is available in /course/cs195g/asgn/proj6/stencil/. You're free to do this project in whatever language you want, but the TAs are only offering support in MATLAB. The TAs have supplied a few files to get you started.

Recover Homographies

You will be recovering a projective transformation H. H is a 3x3 matrix with 8 degrees of freedom. It is possible to solve H by pairs of corresponding points since q=Hp. You will need to solve a system of at least 8 linear equations to solve for the 8 unknowns of H. These 8 linear equations are based on the 4 pairs of corresponding points. Here is a useful resource on projective mappings.

You are not allowed to use maketform except to check your results.

Warp Images

Once you have the homography, you will need to warp images. You should map the pixels in the warped image to pixels in the input image so that you don't end up with holes in your image. The matlab function interp2 will be useful for warping. Here is a useful resource on image warping. Note the following when reading this resources:

You are not allowed to use imtransform except to check your results.

Rectify Images

Rectifying images simply means you should be able to turn a rectangle that has been persepctively transformed into an axis-aligned rectangle. You should pick four corners of a square and calculate the transform of those points to [0 0; 0 1; 1 0; 1 1] or something similar. If you know the aspect ratio of the rectangular object you take a picture of, you can input those coordinates instead of [0 0; 0 1; 1 0; 1 1].

Composite Warped Images

You will need to composite two (or more) images to make a panorama. It is your choice on whether to incrementally warp one image to another or warp all the images to one image. It is also your choice on how to composite. You can simply do over, averaging, feathering (bwdist may help with that), GraphCut seams, Poisson image blending or any other technique you feel like. You will not be penalized for doing simple composite techniques, but your results may not look as good.

Detect and Match Features

Up to here, you have been using user selected corresponding points. You may have noticed that the transformation is very sensitive to slight errors in selecting correspondence points. Now you will fix that by automating that process. This part of the process is based on Brown, et al. It is strongly recommended that you read this paper.

  1. Harris Interest Point Detector (section 2)- done for you
  2. Implement adaptive non-maximal suppression (section 3)
  3. Implement feature descriptor extraction (section 4) - Ignore rotation invariance, sub-pixel accuracy, and the wavelet transform. Remember to standardize each of your feature vectors: subtract the mean and divide by the standard deviation - the paper calls this normalizing.
  4. Implement feature matching (section 5) - Look at pairs of features and compute the distance between them. This is why we give you dist2. For thresholding, use the Lowe approach of thresholding the ratio between the first and second nearest neighbors.

Robustly Recover Homographies

With the automatic feature extraction, you will have a overdetermined system of equations for H. Unfortunately, some of the points may be false positives: points that matched under the threshold but are not actually matches. You will need to remove these outliers as they will throw off your transformation matrix. A robust way to solve that is using the RANSAC algorithm. Instead of using one data point, you will need four points to create the transformation matrix. The TAs suggest 1000 iterations and a minimum consensus of 10 and an error of 0.5. Which equates to trying 1000 times to find a transform based on 4 points in which at least 10 other transformed points have at most an error of 0.5 (half a pixel) to their actual correspondence points.

Write up

For this project, just like all other projects, you must do a project report in HTML. In the report you will describe your algorithm and any decisions you made to write your algorithm a particular way. Then you will show and discuss the results of your algorithm. Also discuss any extra credit you did. Feel free to add any other information you feel is relevant.

globe panorama

Extra Credit

You are free to do whatever extra credit you can think of, but here are a few examples.

Graduate Credit

You are required to do at least 10 points of extra credit.

Handing in

This is very important as you will lose points if you do not follow instructions. Every time after the first that you do not follow instructions, you will lose 5 points. The folder you hand in must contain the following:

Then run: cs195g_handin proj6
If it is not in your path, you can run it directly: /course/cs195g/bin/cs195g_handin proj6

Rubric

Final Advice

Credits

Project derived from Alexei A. Efros' Computational Photography course, with permission.

Good Luck!