CS129 / Project 4 / Texture synthesis and transfer with Image Quilting
Image quilting applied to texture synthesis (top row) and transfer (bottom row)
The purpose of this assignment is two parts: to synthesize a texture and transfer it to another source image, via a technique called image quilting. As shown in the image above, the first row of images demonstrate how to synthesize a texture from a given sample. The texture of the patch of grass is taken and expanded into larger patches of realistic grass texture. The second row demonstrates texture transfer. The toast texture is taken and synthesized in a way that is correspondent to the soruce image(Abraham Lincoln), until we eventually achieve a image of Lincoln in the texture of toast.
Algorithm - texture synthesis
The concept of image quilting is that we take patches of a set sample size from the texture image, and try to use those patches to generate new texture. We also determine the amount of overlap that we want in the generated patches. After the first patch is placed into the output image, the second patch now has an overlapping region with the first patch, and the algorithm goes through all the possible patches from the texture image and finds a best fitting (smallest sum-of-squared distance between texture patch and overlapping region) patch to place at that position. After the output image is filled with patches of synthesized texture, there may still be visible lines where the patches meet. This is remedied by finding a least-error boundary cut at the overlapping region between each of the patches, so that the patches can smoothly transition.
Results - texture synthesis
Below I will show the results of the synthesized images. From left to right, I will show 1) the original texture image, 2) synthesized texture from randomly selecting input patches, 3) synthesized texture from intelligently selecting texture patches by finding the minimum ssd between bordering patches, and finally 4) minimum ssd patches while applying minimum error boundary cuts between overlapping regions.
|
|
|
|
|
|
|
|
|
|
Albeit a little bit of artifacts, most textures were synthesized really well and realistically.
Algorithm - texture transfer
The idea behind texture transfer is very similar to that of texture synthesis, except that when generating each patch, the algorithm also needs to take into consideration some feature of the source image.
My algorithm employed two different feature comparisons: ssd between the texture patch and the source image, or overall intensity between the texture patch and the source image.
An alpha value is defined to determine the amount of emphasis to synthesizing a realistic texture and to representing the source image.
Also, several iterations of texture synthesis is performed over the image, each iteration with a smaller tilesize and overlapsize. This method improves texture transfer and allows the algorithm to first set the tone of the image with larger patches then gradually define it with more precise patches.
I used 3 iterations on most of my results.
The following calculations are done in calculating how much error a texture patch to be inserted has, with regards to the neighboring patches (overlap error), the source image (correspondence error) and with the previously synthesized images(existing error).
error = alpha * (overlap_error + existing_error) + (1-alpha) * correspondence_error;
Results - texture transfer
|
| Abraham Lincoln synthesized with texture of random.png, from left to right is results of 1st iteration, 2nd, and 3rd. |
|
| Abraham Lincoln synthesized with texture of toast, 4 iterations |
|
| Friend Wonmin synthesized with texture of toast, 3 iterations |
|
| Friend Wonmin synthesized with texture of rice, 3 iterations |
|
| Marilyn Monroe synthsized with brick texture, first result is using ssd, second using overall intensity |
|
| Picture of moon synthesized with grass texture. First result emphasizes resemblance to source image, second result emphasizes realistic texture. |
|
| Picture of moon synthesized with grass texture. First result emphasizes resemblance to source image, second result emphasizes realistic texture. |
As it can be seen the synthesized results are quite good in some textures, such as the toast and rice, but less so in other more structured ones, such as grass and brick. As it turns out some of the other textures such as text has too many white spaces to synthesize realistic images.