Skin and Bones: Multi-layer, locally affine, optical flow and regularization with transparency

(with Shanon X. Ju and Allan Jepson)

This work explores a new method for estimating optical flow that strikes a balance between the flexibility of local dense computations and the robustness and accuracy of global parameterized flow models. The approach assumes that image motion can be represented by an affine flow model within local image patchs. Since some image regions may not have sufficient information to estimate an affine motion model robustly, we define a spatial smoothness constraint on the affine flow parameters of neighboring patches. We refer to this as a ``Skin and Bones'' model in which the affine patches can be thought of as rigid bones connected by a flexible skin. Since local image patches may contain multiple motions we use a layered representation for the affine bones. With the possibility of mulitple motions at a given point, standard regularization schemes cannot be used to smooth the multiple sets of affine parameters. We therefore develop a new framework for regularization with transparency that can applied to produce a smoothed layered motion representation. The motion estimation problem, with layered locally affine bones and transparent regularization, is formulated as an objective function that is minimized using an EM-algorithm.

Related Publications

Ju, S. X., Black, M. J., and Jepson, A. D., Estimating image motion in layers: The "Skin and Bones" model, submitted: Int. Journal of Computer Vision. (postscript).

Ju, S., Black, M. J., and Jepson, A. D., Skin and Bones: Multi-layer, locally affine, optical flow and regularization with transparency, IEEE Conf. on Computer Vision and Pattern Recognition, CVPR'96, San Francisco, CA, June 1996, pp. 307-314. (postscript).

Jepson A. and Black, M., Mixture models for optical flow computation, in Partitioning Data Sets, DIMACS Workshop, April 1993 Eds. Ingemar Cox, Pierre Hansen, and Bela Julesz, AMS Pub., Providence, RI., pp. 271-286; also, Tech. Report, Res. in Biol. and Comp. Vision, Dept. of Comp. Sci., Univ. of Toronto, RBCV-TR-93-44, 1993.

Jepson A. and Black, M., Mixture models for optical flow computation, IEEE Conf. on Computer Vision and Pattern Recognition, CVPR-93, New York, NY, June, 1993, pp. 760-761; also, University of Toronto, Technical Report RBCV-TR-93-44, April 1993.