Video Texture

Algorithm

The objective of doing video texture is that we can generate the any long videos as we like, and the textures of the video is extracted antomaticly. Generally, the video is should have some patterns that the textures can be extracted and I tried some method to generate the textures and combined them to seamless videos. The methods are seperated into 2 parts: 1. is to generate the motion-preserve and future cost preserved distances matrix, so that the probility can be obtained fairly. 2. is to do the sequencing the video textures and select and set of transitions wisely using the filtered cost information obtained before. After constructing the transition table, it is time to schedule the loops. Actually, the part 2 is harder to implement because part 2 deals a lot with loops and data structure. And to be honest, I still got bugs in the second part, so I use some trick to generate the videos.

Part 1. Extracting The Video Texture

The following steps is based on the paper (Video Texture). .

Calculating the distance matrix Dis for gray channel for every frame from source

This step is straight forward, just calculate the distances between the frames, using the squar distances will be OK. Here is the distance matrix below.

The rows and columns of the matrix are all corresponding to the frame numbers, and of course, as you can see, the distance between the same frame is always 0.

Filter to Get the Motion History

The motion history information must be preserved because the information can help people judge if the next frame transition is apporopriate. The paper uses the filter method to get the motion history information. And here, the paper notions a trick that the length of the filter kernel should be even so that the information will not be symmetric. And the motion of the distance is preserved. Here is the pictures of the motion preserved information in Distance and Probility.

Adding future cost

One problem encounted when doing the video texture is that there maybe some bad dead ends, and you really should not achieve such a dead ends. The solver to this problem is to add the future cost to the distance matrix so that as the dead ends loop further, more cost will be added to the current loop so that, for the next loop, the best choice for the next frame, will be a diffrent frame other than the dead end one. Here the method to do this is to use Q-Learning, a very effective method to learn the new state, the alpha is set to 0.995 so that the learning rate is high, and the converge rate will not be too low. Also, in the implementation, the author suggest a better formula, which ignors the probility and converges faster. The problem I did find that, the paper sometimes may let you confused about the cost matrix and distance matrix, which means that sometimes you have to be careful about the shift. Here is the Probility matrix after adding the future cost.

Part 2. Sequencing the video texture

This part can be devided into 3 parts: the random one, the video loop, select transition sets and schedul loops. This part is hard to implement in matlab because of a lot of loops and also, the using of a lot of diffrent data structures.

Random play

Random play is simple, my method is that choose the best several probilities in the probility matrix, form potential locations that the frame can randomly choose at each frame. My method is a slightly diffrent from the paper but the result is fine.

Video Loop and Select Transtions

In this part, we are dealing with dynamic program method to generate the transition table, with cost and range. The overrall method is that I tried using DY table to dynamily generate the table and update the range. A problem encoutered here is that( a problem that I encoutered when I implement this part ) the table does not preserve enough information for any length. For example, for some length, it is possible to generate some paths, but as the table just preserve the shortest cost information, that it will ignore the possibilities that we can achieve some length using transmission.

Schedule Loops

This part's algorithm is for recursing, and I implement it, but the cost of effiency and space is huge when the number of desired frames increase. So, I decided to choose random one which is faster. But the implementation can also be optimized in the future.

Part 3. Some results and Discussions

Clock

the orginal texture video. the video with bad threshold better with only 2 random options and enforce jump. The better is also in increasing the frame rate!