Jonathan Mace jcmace

CS129 / Project 4 / Texture synthesis and transfer with Image Quilting

Part 1: Texture Synthesis

The first part of this project was to implement texture synthesis. Texture synthesis is when a source texture is taken, and used as a seed for generating a much larger texture. Patches of the original texture are stitched together to minimise the boundary error between patches.

The following images illustrate the algorithm part way through execution:

For determining the minimum-error patch, there are a number of methods available. I implemented the following:

The naive approach is to do an exhaustive search over all possible textures from the source, calculating error using something like sum of squared differences (SSD).

Extra Credit: I implemented an image-pyramid based approach for selecting minimum error patches. It selects k starting seeds from the smallest level of the pyramid, then works its way up the pyramid maintaining the m best matches at each level. The results of this approach are much, much faster than an exhaustive search - a factor of about 20.

One downside of choosing the 'best' patch each time is that it can result in repetitious output textures. I also implemented an approach that selects the k- minimum error patches and then randomly picks one.

Extra Credit: Selecting the k- minimum error patches is not necessarily the best technique, as in some cases there may be fewer than k good candidates. I implemented a further approach that selects the minimum error patch, then picks all further patches whose errors are within some bound on the minimum error (ie. error less than minerror * 1.5). Then we randomly pick one of these

We also calculate a minimum-error cut for inserting the patch into the synthesized texture. To do this, I used my forwards-energy seam carving implementation. I created a new image based on the differences between the synthesized texture and new patch, then applied seam carving to select the lowest energy seam.

Extra Credit: I also utilised poisson blending from assignment 2 to blur the texture seams on inserted textures

The results using image pyramid + bounded error + random selection were mostly good. The following table illustrates the results:

Part 2: Texture Transfer

For the second part of the assignment, I implemented texture transfer. Texture transfer is an extension to texture synthesis, whereby the error for an incoming patch is not just derived according to the overlap with previously synthesized textures, but also takes into account some underlying source image that we are 'transferring' the texture onto.

For my implementation, I carried across my implementation of texture synthesis and incorporated SSD error to the underlying image as well in calculating the error.

The results of texture transfer are below in a table. For some reason, things look good on toast. Unfortunately, some of the texture transfer leaves things to be desired. In particular, it is obvious that ce certain parts of the source texture are repeated too much. One fix to this would be to adjust the error threshold on sampled textures to be more liberal, thereby giving a coarser approximation of the input image but with less repetition. Another alternative would be to use a larger source texture.

Also, some texture are not suitable for texture transfer. Textures such as toast are good, because they get their intensity variations from the burnt-ness of the toast. Other textures such as rice are not so good, because their intensity variations come from shadows, and therefore they can look silly after being transferred.