CS 143 / Project 1 / Image Filtering and Hybrid Images

Dog-cat hybrid image

Image Filtering

The algorithm of filtering, or convolution, is straightforward. For each output pixel, its value is the sum of each pixel of the filter times each pixel of the corresponding image patch. MATLAB is fast at vector/matrix operation and slow at looping, so we avoid loops as much as possible.

Considering the performance characteristics of MATLAB, first we can avoid looping through each color channels, because operations on all channels can be done in a single operation if we duplicate the 1-channel filter to 3 channels.

filters = repmat(filter, 1, 1, channels);

Furthur, the convoluted value can be obtained with operator .* and subsequently sum(sum()), instead of looping through each of the filter and summing. 

output(y, x, :) = sum(sum(padded_image(y:y+filter_height-1, x:x+filter_width-1, :) .* filters, 1), 2)

Note that one tricky part here is that when summing the result of .*operator you must specify the dimension of the sum operator, otherwise if the filter is an 1xN matrix, sum() will first collapse all color channels into one. Symmetric padding looks better and can be done easily with padarray(), while I didn't see it at the first time so the manual implementation of mirroring took some time.

With assert() I have verified the correctness of my_imfilter() comparing to imfilter(). The performance is greatly improved without looping through the filter.

Hybrid Images

Cat-dog hybrid (cutoff = 7)

Dog-cat hybrid (cutoff = 7)

The dog-cat hybrid looks very implausible. Mostly it is because the most distinctive feature of the dog face, its black nose, gets bleached by removing low frequency. And it is perceptually unlikely for a dog to have such vivid orange nose.

Dog-cat hybrid in Lab colorspace (cutoff = 7)

To improve the perceptual performance of the dog-cat hybrid, we need to fix its color. One approach is to operate in Lab colorspace by creating the hybrid with only lightness channels of the dog and cat, and then mixing it with the dog's color channels so that the color looks more plausible while the fine structures are preserved in the lightness channel.

The high frequency image in Lab colorspace is visualized by adding 50 to L, which is defined in [0, 100]. (Assuming all images are in sRGB.)

The color the of dog's nose is quite hard to fix because it is similar to the color of the dog's mouth, and by fixing that the dog's mouth will obfuscate the cat's nosal bridge even in lower frequency. It is okay for a cat to have a black nosal bridge, but it is not okay for a dog to have a white nose.

Bicycle-motorcycle hybrid (cutoff = 7)

Plane-bird hybrid (cutoff = 7)

Submarine-fish hybrid and Fish-submarine hybrid (cutoff = 5)

The perceptual problem with these hybrids are that the high frequencies have very sharp and regular edges, which are again not well aligned with the low frequencies. Then the high frequencies and low frequencies can be perceptually seperated and make viewers feel like those are forged images.