CS129 / Project 4 / Texture synthesis and transfer with Image Quilting
Texture Synthesis
The first half of this project was the Texture Synthesis. The goal of texture synthesis is to take any texture image and tile it convincingly. In order to do this we implemented three methods. The first was a completely naive method, that took random patches of a uniform size from the texture image and tiled them into a larger image.
The second looks for the best tile possible to be placed in the next position. It starts by drawing a random sample from the tile image. It uses the overlap between the tiles above and to the left to determine how well the tile will fit into the image. It then repeats this process 1000 times and uses the best one.
The last method is called the min-cut method, and starts by using the best possible tile method. It then looks at the overlap and finds the seam with the least difference between the new and old pixels. It uses that seam as the edge of the tile to avoid having obvious straight lines between tiles.
Below are examples of all three. Click the examples for higher quality images.
Example texture
Naive tiling
Best tile algorithm
Best tile and min cut algorithm
The reason that this algorithm is generally so successful is that with any pattern or texture, if you take a sample of that texture, it will likely be an accurate representation of that texture. The primary thing that makes a tiled texture look bad is the edges of each tile. So by intelligently choosing and cutting tiles with the least edge error, we minimize the chance of having obvious lines and artifacts.
Texture Transfer
Texture transfer is an extension on texture synthesis that also takes on a "source image." The goal of texture transfer is to represent the source image using tiles from the given texture. In order to implement this, I adapted the min-cut texture synthesis method to take into account the similarity between the texture tile intensity and the source tile intensity. This means the algorithm in addition to take tiles that matched their neighbors, also took tiles that were of a similar darkness or lightness to the source image.
Example texture.
Example source image.
Example transfered image.
Unfortunately, when choosing tiles that recreate some source image, you end up picking tiles that don't blend well together. So in order to make the image texture smoother, we essentially repeated this process using the previous result as the source image, so as not to lose the source image but also to allow the texture synthesis some room to make the resulting texture more believeable.
Synthesis Results
Results Key
| Texture Image | Naive Image |
| Best Tile | Best Tile and Min Cut |
Resulting Images
Click images for full size versions|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
For the most part the textures turned out well. There will always be some randomness and slight imperfections, but most of them look pretty believable. The textures that I would like to draw attention to as failures are the flowers, the random lines, and the toast.
The toast has the problem that there is one tile section that is the best match for itself to repeat over and over again. This causes an obvious problem and can be solved by implementing some sort of forced anti-reptition.
The flowers have the problem of not having a large enough tile size, resulting in a closer bunching of flowers than the original, and also some cut off flowers that look rather strange.
The random lines have the problem that it is nearly impossible for the algorithm to find the right tile to put next to any given other tile, making the texture look crazy and broken up.
Transfer Results
Results Key
| Texture Image | Source Image | Transfer Image |
Resulting Images
Click images for full size versions|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This task really challenged by ability to tell what was good and what wasn't, as I wasn't sure which image was more important. Making a texture look like another image on most of the textures came with the drawback of the image no longer looking like the texture. It also requires you to have a very high contrast source image with very distinctive features. Despite all of that, I did notice a large and rather obvious problem of repeating tiles used to cover the same intensity pixels over and over again. This, as with the texture synthesis, would be fixed well by somehow forcing tiles to not repeat, or maybe using a tolerance value instead of choosing the best of 1000.