Overview

Image matting is used everywhere. It's how Superman flies, or how weathermen point to hurricanes on maps. The technique of blue screen matting has been around for ages. Blue screen matting allows for a foreground object (e.g. Superman), to be removed from his background and pasted onto another background. The method is relatively simple: you know the background color (blue, but increasingly green in movies), so anything that isn't the same as the background is part of the foreground. The problem with this technique, however, is that it's all or nothing: either you're part of the foreground, or not.

The Matting Equation

C = Co + (1-αo)Ck 
C = composite [R,G,B] values Co = foreground [R,G,B] values αo = foreground transparency value Ck = background [R,G,B] values

There's more trouble when we try to use general matting, however. What you're trying to obtain is the RGB and alpha (transparency) values for the foreground. But all you have is the single composite RGB values. That gives you four unknowns, but only three equations

There are a few ways around this. The first is to have the background be only blue (RGB = [0,0,x]), and the foreground contain no blue (RGB = [y,z,0]). The limitations here are very strict: no gray, no white, and no hues that contain any blue. This effectively reduces the the number of unknowns from 4 to 3, allowing for a single solution.

Another solution is to have the foreground be gray (R=G=B). This drops the number of unknowns from 4 to 2, making the system overconstrained (2 unknowns, 3 equations).

The triangulation method tries to increase the number of equations, rather than try to decrease the number of unknowns. Smith and Blinn's method requires more photographs, but provides and added bonus: partial transparencies. Another plus is that the backgrounds do not have to be uniform: they can be anything. One stipulation: the camera has to be perfectly still and all the images must be perfectly aligned. For their method, two different backgrounds and four images are required: one of each of the backgrounds, and one of the foreground object in front of each of the backgrounds. The algorithm for solving this system of equations is detailed below.

Algorithm

For every pixel in each set of images, triangulation matting gives you the following:

Backing color Ck1 = [Rk1, Gk1, Bk1]
Backing color Ck2 = [Rk2, Gk2, Bk2]
Composite color C1 = [R1, G1, B1]
Composite color C2 = [R2, G2, B2]

This allows for six equations with only 4 unknowns. We can solve a matrix system of equations to obtain the values of the foreground RGB and alpha.

Where CΔ = C - Ck

Results

Here's what the program produced!

Some images are pulled from other sources (Smith & Blinn, Miklos Bergou)