CS129 Project #4: Image Quilting for Texture Synthesis and Transfer
October 24, 2012
Background
Texture synthesis is the process of generating a region of texture based on the texture of a source image. Texture synthesis is achieved in this project with "image quilting," a technique that rearranges sample patches taken from the source image. This particular implementation of texture synthesis lends itself to a different application: texture transfer. Texture transfer entails "painting" the features of an input image with the texture of a different image.
Texture Synthesis Method
Image quilting is performed by taking a number of square "sample patches" from the source image and then fitting them together to form a new texture image. The naive implementation of this technique would be to fill the output texture image with randomly chosen samples. This creates obvious edge artifacts between the different patches, as well as a lack of continuity in content in adjacent patches.
A better method is to partially overlap the sample patches and check for a certain degree of correlation between the overlapping regions. The sample patch with the "best" overlap with its neighbors should be chosen. The algorithm implemented to perform this task fills the output image with patches in row-major order. Therefore, each additional patch need only check its overlap with its left and upper neighbors. The correlation between the overlapping regions of two neighbor patches is calculated by determining the pixel brightness difference between each overlapping pixel. These values are squared, as the algorithm is only concerned with the magnitude of the differences between pixels. The patch whose overlap has the smallest total pixel differences is considered the best match. This method produces a significantly better result than the previous implementation. Continuity between neighboring patches is much better, yet there remain noticeable edge artifacts.
Texture Synthesis Implementation Details
Simply choosing the patch with the "best" overlap with its neighbor could result in a repeating pattern. To circumvent this, a patch is randomly selected from patches within a certain error of the best patch's "overlap-matching score." The ideal patch may not be chosen every time, but this allows for variation.
Extra feature: The image sampling algorithm is implemented to accept a single argument specifying the number of samples to take. It then samples the source image by taking samples across rows and columns of the image so that the ratio of samples between each row and column is the same as that between the width and height of the source image. If the number of samples exceeds the size of the source image, the samples will evenly overlap eachother. This implementation ensures a fair, even, and thorough sampling of the input image that is easily manipulated with a single argument.
Extra feature: Sometimes a flawless seam does not exist between two neighbors. A local Gaussian blur on the overlaps provides a quick and relatively effective fix to reduce the appearance of these otherwise unavoidable artifacts. The blur is generally not noticeable nor does it detract from the image, except in a few cases where details (like text) highlight the blurring.
Texture Transfer Method
The texture transfer algorithm utilizes most of the texture synthesis algorithm; however, instead of choosing patches that have the best overlaps with their neighbor patches, the transfer algorithm chooses a patch based on its overlap and its correspondence with a target image. The algorithm synthesizes the texture depicting the target image over the course of several iterations. Each iteration, the patch size is reduced and the correspondence to neighboring texture patches becomes a more significant factor in patch choice, as opposed to the correspondence with the target image. On every iteration except the first, a potential patch is compared to the previously synthesized texture image as well as the target image. This helps to preserve the content contained in the previously synthesized image. Since earlier images take the target content into account more heavily, it is important for following iterations to preserve the previously rendered details. At a high level, the algorithm essentially lays out the rough image before focusing on rendering a believable texture.
Note that in the example pictures, the texturized version of Lincoln is shrunk to fit this page. The texture size matches that of the source texture.
Texture Transfer Implementation Details
The apparent patterns in the first three iterations of the Lincoln texture transfer are due to the algorithm's choosing of the absolute best-matching patch. Choosing patches within a small error of the best patch's match score tends to obfuscate the target content in the image and is not necessary, as it is important for only the final iteration to be free of repetition. Therefore, the algorithm only randomly chooses from a range of best-match patches for the final round of synthesis.
Results
While the texture synthesis algorithm produces some nice results, it is highly dependent on a number of parameters including sample size, number of samples, overlap size, source image size, and tolerance for less-than-best-match patches. To achieve acceptable results, these parameters must often be adjusted. There are several problems that can occur.-Sampling Discussion: First, the sample size may not "fit" the entire texture units. For instance, if a brick wall image is sampled, but the sample sizes does not include entire bricks, the algorithm will not be able to accurately reproduce brick sizes in the output texture. The quantity of sampling can prove to be another problem. If an image is oversampled and the tolerance of less-than-best-match patches is not low enough, undesired texture patterns can occur because the algorithm chooses patches that are only slightly shifted from the best-match patch's location in the source texture. Tolerance should generally be lowered if sampling is increased, which suggests that there may not be a good reason to take a huge number of samples. Then again, the number of samples is relative to the size of the image. Sometimes more samples are required to fully contain the content of the source texture. Why not always sample no more or less than the area of the entire image? Sometimes shifted patches of the same region are useful. Also, there are no bounds on texture image size.
-Texture Transfer Discussion As can be seen in the example texture-transferred images, the results are not astounding. This is partly because the texture transfer algorithm introduces additional parameters that should vary to accomodate different images. The number of iterations and the weighting of target correspondence versus texture correspondence are probably the most significant factors. There is an undeniable trade-off between the texture quality and the accuracy of the representation of the target image. To render the details of the target image, very small samples are required, which tend to detract from the quality of the texture. The pictures of the Statue of Liberty demonstrate this problem. To achieve a better depiction of the target image, the paint texture would have to be further degraded. As such, texture transfer appears to provide nothing more than a means of producing abstract art.
Texture synthesis provides a unique tool with possible hole-filling applications in images. Although results vary, image quilting provides a simple yet often convincing approach to creating texture regions.
