CS129 / Project 4 / Texture synthesis and transfer with Image Quilting
Texture Synthesis.
The first step in the algorithm is to find the best SSD patch. The method by which I find these lowest SSD patches is by generating a specific number of random patches, computing their SSD's with respect to the current area of the image that is to be filled (to_fill). The coordinates of these patches along with their SSD's are stored in a matrix. After the patches have all been generated, this matrix is sorted by SSD, and a random index is generated from the top 1% of num_iterations. The coordinates of this patch are then used to pull the correct patch from the texture. The point of this random index is to minimize the recurrence of similar patches across the image, as would likely happen in cases such as the toast image.
After the patch has been found, seam carving is then used, edge casing for when seam carving is required along the top or left edge of the current patch. The energy matrix must be masked with respect to the to_fill_mask, where values in the mask are set to infinity to keep paths from being generated where they shouldn't be.
Texture Transfer
Texture transfer progresses very similarly to texture synthesis. Here however we don't have to worry about duplicate patches (although this might occur). Thus, we can simply find the minimal error patch. This time, we need not only compare the patch's error to the space to be filled with respect to the overlap area, but it also needs to be compared to the image onto which the texture is to be transferred. The equation to calculate this, from the project specifications follows:
error = (α) * (overlap_error + previous_synthezised_error) + (1 - α) * correspondence_errorAfter the patch has been found, seam carving procedes just as before in the texture transfer.
Texture Synthesis Results - Random, SSD, and SSD + Mincut
|
|
|
|
|
|
|
|
|
|
|
|
|
Texture Transfer Results
|
|
|
|
|
Analysis
The results are fairly good. The texture synthesis tends to overuse certain patches despite randomizing them a bit. If the input texture is very small, this is bound to occur as in the case with the wood grain (an example where the algorithm fails). In some of the images, the seams are rough, but that is to be expected. The texture transfer is decent enough, but relies on very sharp edges to be defined, and an input texture with varying luminosity across the image.