Project 6: Automated Panorama Stitching on the n900

Due Date: 11:59pm on Wednesday, April 20th, 2011


Envoirnment Updates

You will again need to update your vm if you wish to test your code on the desktop. To do this run: sudo apt-get install libqt4-dev This update is unecessary if you only with to do testing on the phone.

Building and Testing Your Code

You may use the file and QtCreator to build either the desktop or phone versions of the code. The code can also be built from the command line using:

You can use the following command to test your code on the desktop: ./Features panorama img1 img2 destimage
You should can test your panorama stitcher on these two Yosemite images. If you would like a larger set to test on, try out this larger set of Yosemite images.


You are required to implement the following:


Recover Homographies

You will be recovering a projective transformation H. H is a 3x3 matrix with 8 degrees of freedom. It is possible to solve H by pairs of corresponding points since q=Hp. You will need to solve a system of at least 8 linear equations to solve for the 8 unknowns of H. These 8 linear equations are based on the 4 pairs of corresponding points. Here is a useful resource on projective mappings. You should be able to test your code for recovering homographies independantly of your other code. It's important to make sure this part works since other parts build on top of it.


Warp Images

Once you have the homography, you will need to warp images. You should map the pixels in the warped image to pixels in the input image so that you don't end up with holes in your image. Here is a useful resource on image warping. Note the following when reading this resources:


Composite Warped Images

You will need to composite two (or more) images to make a panorama. It is your choice on whether to incrementally warp one image to another or warp all the images to one image. It is also your choice on how to composite. You can simply do over, averaging, feathering, GraphCut seams, Poisson image blending or any other technique you feel like. You will not be penalized for doing simple composite techniques, but your results may not look as good.


Detect Features

Feature Detection and Description has been implemented for you! We have provided code for SURF (Speeded Up Robust Features). You are encouraged to look through and understand the code. You may also wish to experiment with other feature detectors and descriptors. We have provided space in the stencil code for the scheme we discussed in class using harris corners, non maximal supression, and MOPS descriptors.

Match Feature

Basic Feature Matching using the sum of squared differences metric to find the nearest neighbor has been implemented for you for testing purposes. It will work reasonably well, but a bit slowly. You are required to implement a faster approximate version using a provided kd-tree library. You are also required to use the ratio between the first and second nearest neighbors in this function; however, this should require little extra work because the kd-tree will be doing the heavy lifting.


Robustly Recover Homographies

With the automatic feature extraction, you will have a overdetermined system of equations for H. Unfortunately, some of the points may be false positives: points that matched under the threshold but are not actually matches. You will need to remove these outliers as they will throw off your transformation matrix. A robust way to solve that is using the RANSAC algorithm.

Implementation Details

C++ stencil code is available in /course/cs129/asgn/proj6/stencil/. You're free to do this project in whatever language you want, but the TAs are only offering support in C++. The TAs have supplied several files to get you started. The only source file you should need to edit is Features.cpp in which you will need to complete the following functions:

If you're running into problems with your phone, check to see if the issue is addressed in the FAQ.

Allowed OpenCV Functions

You are allowed to use any of the basic math and input / output functionality provided by OpenCV or the c++ standard libraries. You are also allowed to use functions and classes related to the singular value decomposition and / or solving linear equations. You should not use any function that trivializes any portion of the project. Use your judgement. If you are unsure about whether or not you can use a function please email the staff.

Write up

Each group will produce a single hand in. Make sure you say who is in your group. Feel free to name your group, as well. Only one group member needs to run the handin script.

Describe your algorithm and any decisions you made to write your algorithm a particular way. Also discuss any extra credit you did. Feel free to add any other information you feel is relevant.

Because you are building a (hopefully) interactive system, it would be compelling to show videos of actual usage in different scenes.

globe panorama

Extra Credit

You are free to do whatever extra credit you can think of, but here are a few examples.

Graduate Credit

Groups contain mixtures of graduate and undergraduate students, so there are no distinct graduate requirements.

Handing in

This is very important as you will lose points if you do not follow instructions. Every time after the first that you do not follow instructions, you will lose 5 points. The folder you hand in must contain the following:

Then run: cs129_handin proj6
If it is not in your path, you can run it directly: /course/cs129/bin/cs129_handin proj6


Final Advice


Project based on Noah Snavely's Computer Vision course at Cornell University. Handout written by Travis Webb and James Hays.

Good Luck!