CS 129 Project 2 Writeup
Jeroen Chua (jchua)
September 28 2012
Outline
I briefly describe a method to seamlessly blend an area from a source image (eg, area corresponding to an object) into a target image. The method uses constraints on the discrete Laplacian of the source area and target image. I also explore methods for making the source area more "transparent" when being translplanted into the target image. The method is largely based on "Poission Image Editing", Perez et. al., SIGGRAPH 2003.
For extra credit, the mixing of gradients was explored, as mentioned in the assignment handout. I tried both taking, for each masked pixel, 1) the largest corresponding gradient in the source or target image, and 2) the average gradients in the source and target. Computation of the larget gradient, and average gradient are straightforward. Constraints are placed on the resulting discrete Laplacian-like quantity when solving for the blended image. Results are shown below.
Implementation and optimization details
The image blending problem is phrased as a least-squares problem, which requires solving an equation of the form Av = b where v is the blended image, and A and b take the meanings as defined in the assignment handout.
A is a potentially large matrix (N x N, where N is number of pixels in the image), and sparse. So, A is constructed using a single call to MATLAB's "sparse" function for efficiency. This was the only significant code optimization I carried out.
Result Images
The images, from left to right, are:
- Source: Source image
- Mask: Masked area in source image
- Target: Target image
- Naive: Naive method that copies pixels directly from source area to target image. This is the method implemented by default in the assignment code.
- Poission: Blended image obtained by putting constraints on the discrete Laplacian. This is the baseline method that had to be implemented for this assignment.
- Max: Blended image obtained by using the maximum gradient available at each pixel from the corresponding pixel in either source or target image. Contraints are place on the resulting discrete Laplacian-like quantity.
- Average: Blended image obtained by using the average gradient at each pixel from the corresponding pixel in the source and target image. Contraints are place on the resulting discrete Laplacian-like quantity.
The results are discussed after the displayed images.
Click an image to see it at its full size.
Source Mask Target Naive Poission Max Average
| 
| 
| 
| 
| 
| 
|
| 
| 
| 
| 
| 
| 
|
Source Mask Target Naive Poission Max Average
| 
| 
| 
| 
| 
| 
|
| 
| 
| 
| 
| 
| 
|
Source Mask Target Naive Poission Max Average
| 
| 
| 
| 
| 
| 
|
| 
| 
| 
| 
| 
| 
|
Discussion of results
Overall, all non-trivial image blending methods (Poission, Max, Average) produce more photorealisitc images than the Naive method. However, there are still numerous problems with the non-trivial methods. For example, in the first image (with the swimming bear), the blending procedure results in a blue bear, which looks unnatural. This is a side-effect of discrete Laplacian constraint and the large amount of blue at the border between the masked and non-masked pixels in the target image. The blue in the target image around the masked area has been "propagated" to the centre of the image, which creates the undesirable blue bear effect. There are also minor problems in other images. For example, for the "Poission method" and the image with the giant octopus, a smudged "halo" can be seen around the octopus's head. Although the "Max" method does not have as pronounced halo, the octopus's tentacle seems unnrealistically transparent, showing the aircraft carrier underneath.
Lastly, the "Average" method failed to reliably produce photorealistic images. In general, the images produced by this method contain "ghostly" objects (eg, blue bear, delorean in image 3, octopus), none of which look realistic.