Scene Completion

Ben Swanson

In this project I implemented scene completion, in which I used the possion blending code from project 2 and the graphcut code from project 3 along with image alignment code using fft convolution. My implementation first dialates the mask over the area we want to fill, then subtracts the original. This gives a boundary region around the hole over which we will perform the alignment. For each MxN match candidate image, its alignment and score are determined in the following way.

  1. Create a MxN matrix where each pixel is the sum of its squared RGB values. Convolve this with a logical mask of the boundary region to get C1.
  2. Split the color channels and for each convolve that color channel with the same color channel of the source image, masked by the boundary region to get CR,CB,CG for red/blue/green respectively.
  3. Compute C = C1 - 2 * (CR + CB + CG)
  4. The best alignment score for this match is the minimum entry of C, and the offset at which this occurs is the index pair of this minimum entry.

My implementation selects for each source image the minimum metric (which is an constant value less than the SSD, where the constant depends on the source image in the boundary area). It also scales each match image to 1.4 its original size and repeats the process. The minimum of all these candidates is chosen as the best match. Using this match, a seam is found using graphcut in the region between the original mask and the area outside the dialated boundary. Finally, the image is poisson blended to remove artifacts.

The results were varied, performing best when the boundary area contained high frequency spatial artifacts such as grass, as the matched image would often also contain such artifacts and create a smooth transition. Using 1.4 as another optional scale often produced a better match by the metric, but the difference in resolution is often noticable and undesirable.

This image is clearly not correct, but I really liked how it turned out