A (discrete) light field is a sampling of the plenoptic function, as we discussed in class. We can capture a light field with a camera array, a lens array, or for static scenes we could move a camera on a gantry or a linear stage.
Today we're going to see how to use images from a 3D ray sampling of a light field captured by a gantry to provide control over the focal plane and focus blur. For this, we will consider slices of the light field called epipolar images (EPIs). We can create an EPI image by slicing images in our linear-stage-captured light field along a scanline. This operation is like stacking all our 1D images into a rectangular prism, and slicing it horizontally:
Relate the EPI image to the horizontal position in the camera array (Fig 4. high res):
cv2.imshow("window",data)will work fine - just remember to
Train: 2D images x 1D linear stage with 500 views. ZIP with 100 images [start with this]. ZIP with all images [be careful; ~400MB; you might not be able to fit them all in memory]
Chess: 2D images x 2D linear stage with 289 views. ZIP with 1D - just 17 images [start with this]. ZIP with all images [be careful; ~400MB; you might not be able to fit them all in memory]
Lego: 2D images x 2D linear stage with 289 views. ZIP with 1D - just 17 images [start with this]. ZIP with all images [be careful; ~400MB; you might not be able to fit them all in memory]
From our study of projection, we know that perfect pinhole cameras do not blur the image of the world—everything is in focus. However, as we increase the aperture, objects start to blur. We can introduce a lens to focus some of these rays, but rays from objects that are not at the focal plane converge to a depth-dependent point spread function on the sensor.
From our light field datasets above, each one is (almost) all in focus; the aperture is small and the focal plane is set to infinity. Let's assume that they are pinhole cameras. Now, by shifting the position of the camera, we see rays that come from directions that were not visible with just a single view ('parallax'). This is very similar to what happens when we open the aperture, except now these rays are not averaged together—we have sampled them individually (or, at least, more individually!).
This individual ray sampling is powerful. Remember how we integrated rays over time with multiple short exposures to simulate a long exposure without noise in the night lab? Now we're going to integrate rays over space to simulate a larger aperture than we ever had before. Let's average across the vertical dimension of our EPI image—what will we see?Tasks:
Next, we're going to virtually change the focal plane. We can see that the slope of the line in the EPI relates to how far away the scene point is from the focal plane. In our case, the scenes are focused at infinity, so the slope is directly proportional to the depth: a larger slope is closer to the camera, because we see a greater disparity from camera view to camera view.
So, what happens if we change the slope of the line and then integrate? We know how to do this—skew matrices!
skimage.transform.AffineTransform. Use any test image to begin, and make sure you can skew correctly.
To estimate depth, we can use the knowledge we just gained: a point in focus will be at the focal plane, and the skew of the matrix determines the slope of the line (or the depth!!!) of that focal plane. Let's look at how the image content varies across skew values, and optimize for a skew value per pixel.
Light field cameras and camera arrays are powerful computational tools because they let us sample and integrate rays in new ways to simulate optical and lens effects in a principled way. Lytro is the most famous light field camera company (now defunct; bought by Google). Why do you think they went under? Imagine I had to build a light field camera. What is a practical way to build a portable camera? I only have access to the same image sensor technologies as every other camera manufacturer. What do I gain and what do I lose in which rays I sample, and how does this affect the final image quality? As a photographer, or as a consumer, what is most useful to me?
Talk to James to know more : )
Please upload your Python code, input/result images, and any notes of interest as a PDF to Gradescope. Please use writeup.tex for your submission.
This lab was developed by the 1290 course staff. Thanks to The (New) Stanford Light Field Archive, and the MERL Light Field Repository.