cs129 / panorama project

CS129 / Project 6 / Automatic Panorama Construction

The concept of automatic panorama construction is rather simple, take a bunch of pictures that overlap and make them into one large image. Unfortunately, it's not nearly as easy at it sounds. Here's the basic idea behind the algorithm.

The Algorithm

The first thing that you do when combining two images is you find their interest points. This has a few steps. The first is you look for distinctive parts of the image. Specifically, pieces of the image that change the most rapidly. Then you choose the strongest ones that are the most reasonably spaced out, so that you don't have all of your interest points right next to each other. After this you match the interest points across the two images. This is done by sampling small patches of the image, blurring them, and then very literally taking the difference of the two patches. Blurring the patches takes away the noise, so that the pixel differences are based more on the features of the image and less on the small pixel artifacts.

Example of found interest points, highlighted in red. Click the images to see larger versions.

Resulting Panorama.

The second half consists of looking through the pairs of interest points to find the best combination of the two images. This process considers a combination of rotation, perspective skewing, translation, and scaling the images. It also is the cause of the most problems as we will see shortly.

The Problems

Example normal panorama.

Example panorama with distortion issues.

There were really only two major problems that I found with this process. The first problem and the most annoying problem was image skewing. This was largely a problem when combining too many images in a row, the perspective transformation begins to make the images (usually on the left side) look extremely warped or stretched. The reason for this seems to be that the images are all being warped to match the right most images perspective plane. A nice fix for this issue would be to transform each of the images to match the middle images perspective plane, so that rather than having one overly stretched side and one normal side, both sides would look a little bit stretched. This also seems to only be a problem with images that are taken very near to their target, although not exclusively so.

The second problem was that the image overlap would occaisionally generate a little ghostly shadowing or blurriness of objects. This is obviously because the two images did not overlap perfectly, despite their transformations. The most obvious way to fix this would be to do a better job of blending the two images. A better blending algorithm would also fix the obviousness of the break between some of the images.