In-Browser View Reconstruction with WebGL

Your browser does not support HTML5 Canvas :(.

MORE EXAMPLES: Living Room Alley Monet Hallway

INSTRUCTIONS

• Click Load example first, to make sure it works
• Only works on browsers with WebGL support, which right now is just Firefox (4.0+) and some versions of Chrome.
• Firefox might warn you about security. Click 'allow'.
• Left canvas is 2d, right canvas is 3d
• Left canvas: click+drag to identify the rear plane, then click to place vanishing point
• Click 'Reset perspective' to retry if you mess up
• Right canvas: click+drag to view, click+scroll to zoom in/out

• You can use any image from the internet -- just click 'Pick image' and paste its URL into the box
• (Though of course this works best with one-point perspectives)
• WebGL likes to texture the outside of the viewing box as well as the inside. There's not much I can do about this, though it's pretty neat to 'invert' a perspective (view it from outside the box rather than inside).

INFORMATION

Single View Reconstruction is pretty well documented (Horry et. al 1997, Criminisi & Zisserman 1999), but doing this in JavaScript was way more than I bargained for. I ended up with nearly 2000 lines of JavaScript, split roughly evenly between math, GUI code, and WebGL code. The pipeline goes something like this:

1. Given a photo, have the user select a rear wall and vanishing point
2. Extract our five faces from the input (rear wall + four trapezoids, one on each side)
3. Morph trapezoids to rectangles to construct a rectangular prism, with the image "painted" on the inside of it
4. Find the view from an arbitrary position within this rectangular box, via WebGL

Difficult things:

• Determining edge cases (quite literally: how to pick vertices when the trapezoid goes outside of the picture) became really ugly really quickly. It didn't help that the papers (linked above) kind of elided over their math a bit, resulting in my fudging around with geometry for several hours before I had something reasonable.
• Computing homographies got pretty ugly as well, since there is no library (at least, none that I could find) that would solve systems of equations in JavaScript. I ended up having to adapt some Python code to implement Gauss-Jordan elimination. On the other hand, I now have a working in-place equation solver in JavaScript. Sweet!
• On the other hand, WebGL is really really nifty, and I wish more browsers had support for it. FireFox 4.0+ and Chrome seem to work (or _should_ work, according to their docs). Opera and Safari nightlies also have support, though I haven't been able to test them. IE is, as always, the problem child, with absolutely no support for WebGL.
• WebGL does linear filtering if the texture is far away, but zooming up close things get really ugly. After computing homographies, I generate textures via nearest-neighbor lookup, which of course results in blocky, unrealistic textures when viewed up close. I tried to do my own bilinear sampling but it made everything 4x slower, and even that didn't seem to improve quality much.