CS 195-G: Image Blending

Evan Wallace

The goal of this project was to implement the seamless image blending algorithm of Pérez, et al., which determines the final image using a discrete Poisson equation.

Algorithm

Basic Algorithm


The source image

The target image

This algorithm copies regions from a source image to a target image.  In this example it will use the face from the source image to replace the face of the destination image.  The face in the source image has been aligned manually to fit the target.  To select the regions from the source image that will be used, a third mask image is needed.  It is white in the region of interest and black elsewhere.


The mask image

The pixels in the final image are computed by solving a Poisson equation.  In code, each final pixel is defined as:

if mask(x,y) is 0:
    final(x,y) = target(x,y)
else:
    final(x,y) = 0
    for each neighbor (nx,ny) in (x-1,y), (x+1,y), (x,y-1), (x,y+1):
        final(x,y) += source(x,y) - source(nx,ny)
        if mask(nx,ny) is 0:
            final(x,y) += target(nx,ny)
        else:
            final(x,y) += final(nx,ny)

However, all masked pixels (where the mask is nonzero) depend on each other for their final value.  We have to compute everything simultaneously, which can be done by solving the matrix equation Ax = b.  The size of both vectors x and b is the number of pixels in the target image.  The vector x contains all pixels in the final image (what we are solving for).  The vector b is the guiding gradient plus the sum of all non-masked neighbor pixels in the target image.  The non-masked neighbor pixels define the value of the pixels at the boundary of the mask, which are blended across the mask area.  The guiding gradient defines the second derivative of the final pixels in the mask area, and helps to "shape" what is blended in the mask area.  The matrix A is a sparse matrix with a 4 for each pixel and a -1 for each neighbor pixel under the mask, and computes the gradient of the masked pixels in the final image.  To get the final image, simply solve the equation for x.  If the guiding gradient is zero, we are just solving a Laplace equation and the values at the mask boundary are blended smoothly across:


The final image with no guiding gradient

With the gradient of the source image as the guiding gradient, the mask area takes on the appearance of the source image, but is smoothly blended at the boundary:


The gradient of the source image

The final image

Mixing Gradients

It is sometimes better, especially for transparent objects, to retain some of the gradient of the target image in the mask area.  The two gradients are mixed by picking the gradient to each neighbor either from the source image or the target image, whichever gradient has a higher absolute value.


The gradient of the source image

The gradient of the target image

Both gradients mixed

The final image using gradient mixing

Results

Project Images

Regular Poisson blending didn't work that well for the jet image because the area around the jet ended up blurry.  This was almost completely fixed using gradient mixing, although the background can be seen slightly through the right of the jet.


The source image

The target image

Poisson blending

Poisson blending with gradient mixing

The pool image worked better with regular Poisson blending.  Gradient mixing let some of the water behind the bear show through.


The source image

The target image

Poisson blending

Poisson blending with gradient mixing

The rainbow image worked better with gradient mixing because rainbows are transparent.  Regular Poisson blending blurred out the area behind the rainbow.


The source image

The target image

Poisson blending

Poisson blending with gradient mixing

The whiteboard image also worked better with gradient mixing because the texture of the bricks needs to be preserved under the marker.  Regular Poisson blending took the texture from the flat whiteboard.


The source image

The target image

Poisson blending

Poisson blending with gradient mixing

The Mona Lisa image worked better with regular Poisson blending because Mona Lisa's actual face doesn't show through.


The source image

The target image

Poisson blending

Poisson blending with gradient mixing

My Images

In this image the Rubik's cube completely replaces the plate.  The differences in color temperature are mostly compensated for, but the shadow on the bottom is slightly blue.


The source image

The target image

Poisson blending

In this image the pepper shaker is placed on the snow, although it is slightly transparent because gradient mixing was used.


The source image

The target image

Poisson blending with gradient mixing

In this image the washer door is grafted onto the wall.  The boundary is perfectly seamless.


The source image

The target image

Poisson blending

In this image the apple laptop logo is applied to the side of the truck, with no visible boundary.


The source image

The target image

Poisson blending

In this image lettering from the Watson Institute is put on a Sunlab wrist rest.  The brushed metal behind the lettering is still slightly visible.


The source image

The target image

Poisson blending with gradient mixing