Single View Reconstruction is pretty well documented
(Horry et. al 1997,
Criminisi &
Zisserman 1999), but doing this in JavaScript was way more than I bargained for. I ended up with nearly
2000 lines of JavaScript, split roughly evenly between math, GUI code, and WebGL code. The pipeline goes
something like this:
- Given a photo, have the user select a rear wall and vanishing point
- Extract our five faces from the input (rear wall + four trapezoids, one on each side)
- Morph trapezoids to rectangles to construct a rectangular prism, with the image "painted" on the inside of it
- Find the view from an arbitrary position within this rectangular box, via WebGL
Difficult things:
- Determining edge cases (quite literally: how to pick vertices when the trapezoid goes
outside of the picture) became really ugly really quickly. It didn't help that the papers
(linked above) kind of elided over their math a bit, resulting in my fudging around with
geometry for several hours before I had something reasonable.
- Computing homographies got pretty ugly as well, since there is no library (at least, none
that I could find) that would solve systems of equations in JavaScript. I ended up having to
adapt some Python code to implement Gauss-Jordan elimination. On the other hand, I now have a
working in-place equation solver in JavaScript. Sweet!
- On the other hand, WebGL is really really nifty, and I wish more browsers had support for it.
FireFox 4.0+ and Chrome seem to work (or _should_ work, according to their docs). Opera and Safari
nightlies also have support, though I haven't been able to test them. IE is, as always, the problem
child, with absolutely no support for WebGL.
- WebGL does linear filtering if the texture is far away, but zooming up close things get really
ugly. After computing homographies, I generate textures via nearest-neighbor lookup, which of course
results in blocky, unrealistic textures when viewed up close. I tried to do my own bilinear sampling but
it made everything 4x slower, and even that didn't seem to improve quality much.