Project 4: Scene Completion

Diana Huang (dkh) - March 12, 2010

Description

For this algorithm, we had to try to find a matching image that would complete the scene underneath the mask. My algorithm has a few steps to it.

For each matching image:

  1. First I resize the matching image so that it is the same size as the source image. If the image does not resize perfectly, it is placed over the source image so that the aspect ratio is maintained.
  2. Then, I used Image Alignment to attempt to align the image. This uses the strategy of the image pyramid to try a larger range of alignments. I shrunk the size of the search window down from a 5x5 window to a 3x3 window to try to preserve the alignment of the original picture.
  3. Each alignment has a score which is a slightly modified version of SSD which dilates the mask and only compares the pixels that surround the original mask image.
  4. This alignment score is returned along with the best shift found, and this score is multiplied by 10,000 and added to the score given for that image. The multiplication is necessary to ensure that the scores are in the same range. The given score is given higher weight.

After each image is scored with a particular shift, the 20 best scoring images are chosen. The shift is applied to the overlaid image. The graphcut is limited to a 30 pixel range around the given mask, and the mask found by the graphcut is passed along to the Poisson solver to find the best merging of the two without transparency.

There are some problems with my implementation of this algorithm, however. Some of the shifts are very obvious in the final version, especially when the mask is near the edges. In addition, it seems that there are some images where graphcut does not seem to hold to the constraints placed on it, so the entire original image is the result, or there are offsets have a significant amount of the original image left behind. This is probably an error in the image resizing on my part, but I was not able to identify the exact reasons why this was occurring.

Extra Credit

I did not implement any extra credit.

Results Images

As there were many results images generated, I only selected a few of the best ones: