CS129 / Project 2 / Image Blending

An example of a blended image constructed out of a source image and a target image. See table below for list of source, target and results.

In this project, I implemented gradient domain image blending on six test images, and several other pictures. This is a way to seamlessly "fuse" a source image onto a target image, specifying the areas that we want from the source by using a mask.

The Laplacian was used to try and preserve the gradient of the target image. This represents an improvement over simply copying the pixels we are interested in from the source and pasting them onto the target. In this report, I will be discussing my algorithm for gradient domain image blending, as well as showing a set of results that illustrates both success and failure cases.

Math

For unmasked pixels, we want the output pixel to simply be the target pixel. However, for masked pixels, we want the gradients at the output to match the gradients at the source. We use the Laplacian to help us define the gradient. The Laplacian is a 3x3 array that consists of a 4 in the middle, and -1 in each position adjacent to it. We can set up a system of equations relating the output gradients to the source gradients. Since we know what the source gradients are, we can find each output pixel by the equation x = A\b, where b is the source gradients, and A is a matrix of the size (rows*cols) x (rows*cols) which contains the coefficients of each output pixel x.

Algorithm

The imblend function takes in a source image, a mask and a target, each represented as a 3 dimensional array in Matlab. I processed each color channel independently by passing the corresponding 2D arrays into a separate function, called procedure, that takes in 5 parameters - the number of rows, the number of columns, the 2D source array, the 2D mask array and the 2D target array.

The procedure function populates a sparse array by first initializing four 1D arrays and a counter, and then iterating through a for loop m number of times, where m is the number of pixels in the image. Seee below for the pseudocode. The four 1D arrays are b (which represents b in our equation above), i (the x coordinate of non-zero entries in the sparse matrix), j (the y coordinate of non-zero entries in the sparse matrix) and s (the value of the non-zero entries in the sparse matrix) respectively, and the counter c keeps track of the index of the i, j and s arrays.


for x = 1 to m:
  if the value of the mask at x is above 0, and x is not on one of the four edges of the image:
    put x in i(c) to i(c+4)
	put x in j(c), x-1 in j(c+1), x+1 in j(c+2), x+rows in j(c+3), x-rows in j(c+4)
	put 4 in s(c), and -1 in s(c+1) to s(c+4)
	add 5 to c
	put 4*source(x)-source(x-1)-source(x+1)-source(x+rows)-source(x-rows) in b(x)
  else
    put target(x) in b(x)
	put x in i(c)
	put x in j(c)
	put 1 in s(c)
	add 1 to c

The sparse matrix is populated using i, j and s. This forms A as described in the equation above. x for each color channel can then be obtained by using the mldivide function in matlab, passing in A and b.

The output of the three procedure function calls are then reshaped to the desired rows x cols dimension of the target image and put together to form back a color image.

Results


Discussion

The first six cases in the table above can be considered 'successful'. They don't look that inconceivable, and it seems like the edges of the source, especially, integrate well with the target image. This is readily apparent when compared to the naively-blended result. However, the last four cases show the shortcomings of the algorithm. The seventh case (candle on a cake) did not work out because the edges of the candle did not seem to blend seamlessly into the cake picture, perhaps because of significant differences in lighting. The eighth case (snowman in a desert) was unexpected in that the snowman became brown in color, and looked like something else entirely (perhaps a tanned snowman?). Also, the light sources in the source image and the target image were shining from two opposite directions, and the snowman was illuminated the wrong way in the result image. The ninth case (octopus on a warship) was not perfect because the texture near the edges of the octopus did not match the wave-like texture of the sea although the color blended rather seamlessly. This is because the algorithm does not account for texture differences. Finally, the tenth case (rainbow on island paradise) was ugly, perhaps because the sharpness of the target image did not match the sharpness of the source image (the rainbow was much more blurry than the island)

by Kai Herng Loh