Project 4: Scene Completion

Ben Cohen (btcohen)
March 13, 2010

Algorithm

Alignment

For this assignment, I modified my image pyramid code from Project 1. To align the images, I computed the SSD between the source image and the target image at various offsets, ignoring the area under the mask. To speed up the alignment, I limited the function to four recursive calls.

Initially I was having problems with the images being shifted too much, causing seams to be visible in the resulting images. To fix this, I bounded the amount that an image could be shifted so that no seam would appear underneath the mask. Although this greatly improved the results, there are still seams visible in some images since the graphcut expanded the masks past their original boundaries.

Graphcut

When the images are aligned, they are passed to the graphcut solver. Since there is only a mask for the source image, I had to generate a mask for the target as well. To do this, I expanded the source mask using imdilate and then inverted it; this ensured that there was no overlap between the two masks, while forcing some of each image to be present in the composite.

Ranking

To rank the images, I used both the flow computed by graphcut and the ssd computed by the image alignment. Since I wanted to pick images that have high flow, but a low ssd, I ranked images by flow/log(ssd).

When I initially ran my algorithm, I noticed that the images with the highest values of flow/log(ssd) weren't very good; every more puzzling was that the images with a very low score seemed to be some of the best. Although it seems counter-intuitive to me, I reversed the ranking so that images with the lowest scores are ranked the highest; I've found that with this ranking metric, the scenes ranked highest by the algorithm are most often the same images I would have chosen.

Poisson Blend

Once all the images are ranked, the top 5 images have a Poisson blend applied. Since blending is fairly slow, processing all the results would be very time consuming.