CS129 / Final Project / Separation of Global and Direct Illumination

Global component of a full beer bottle

In this project I separate the direct and global illumination components of objects by taking many photographs under specific high frequency lighting conditions. In the context of this project, direct illumination refers to color conveyed by photons traveling from the light source, bouncing off of an object and then being recorded by the camera, while global illumination is any illumination reaching the camera that has taken multiple bounces to get from the light source to the camera (i.e. reflections, refractions, subsurface scattering, etc.). This project is based on a paper by Nayar et. al. from Siggraph 2006.

Algorithm

I implemented the simplest version of the suggested algorithm in the paper. An interesting aspect of this algorithm is that most of the complexity is on the image capture side, and the computaion is fairly simple. The idea is to take images of the same scene lit by a checkerboard pattern alternating between pure bright white light and pure lack of light, i.e. blackness or darkness. The scene is half lit, so half of the total global illumination of the scene will register in the image, and every pixel will get that component of the light. The pixels for parts of the scene that are under the light part of the checkerboard also recieve a full component of direct light. In the simplest case you can take two images, each with complementary checkerboard patterns for the light. In these two images, you can see each pixel when it is fully lit (maximium brightness) and when it is not fully lit (minimum brightness). Using the logic above, we see that the maximum brightness pixel is the sum of the direct component and half of the global component, and the minimum brightness pixel is just half of the global component. Solving, we can see that the direct component is just the maximum brightness minus the minimum brightness, and the global component is twice the minimum brightness.

Implementing the imaging system described above will give a result, but intuitively it won't work perfectly if, for instance, there is global component stemming from very local effects between objects that are lit/unlit together. To rectify this problem, we take 25 images. Instead of simply inverting the checkerboard illumination pattern, we create 25 unique checkerboards to illuminate with. In this case, we used a pattern that was 8x8 pixels on the projector and shifted the pattern by 3 pixels 5 times in each of the X and Y directions. This pattern increases the likelihood of detecting correct global lighting due to highly local effects. To leverage the 25 images, for each pixel we take the pixel from the source with the maximum brightness and the minimum brightness for our values. This means we will probably use information from each of our 25 images. To find maxima and minima, we want to combine all three of the RGB channels into one brightness, so we create a grayscale image and use that as our basis for judgement. Once we know which pixels are brightest and darkest, we index into our color image to get color versions of the maxima and minima. See below for maximum and minimum images for the onion example.

RGB Color Maximum for Onion RGB Color Minimum for Onion

There is one more issue at play, which is idiosyncracies of the imaging process. My setup involved using a Viewsonic PJD6553w projector to project the checkerboard patterns as my illumination source. This projector is fairly decent quality, but it does not project black perfectly. In the original paper they amended the above explanation by saying that even what would be "unlit" parts of the scene do get some direct illumination due to the projector projecting some light when it should be projecting none. To test what that parameter was for this projector, I took an image of black fabric, which has little to no global component. I then found the minima as described above and looked for the maximum value of a pixel within that minima. The idea was that that was the brightest that "black" would be for this projector. The value ended up being around .3, which I factored into the equations for direct and global illumination. The inclusion of this parameter turned solving the for the direct and global illuminations into a linear system of equations since both direct and global contribute to the maximum and minimum at a given pixel.

Results

Direct Global

A good way to test the algorithm is to composite the calculated global and direct components and see if it matches an image taken in pure white light. Below you'll find a few examples of the composite images versus images of the same scene taken with full white light.

Composite of Direct and Global Single Photograph White Light

Discussion

This approach seems to work quite well, and the results are pretty striking in some cases. Some of my favorites are the onion, the beer, the rice, the candles and the assortment of food including the tamarind candies, the bagel and the graham cracker. All of these have strong global components due to various factors. I can't be certain about all of them, but the beer, for instance, is due primarily to the refractive nature of the glass and liquid inside. Some of the results, like the blueness on the direct component of the onion surprised me, but upon compositing the global and direct illumination, the image was essentially correct, so I believe my results are correct. One interesting side note: For the milk image, the cup on the left contains milk that had gone sour and had curds on the top layer of the glass. Under white light the color doesn't change much, but the difference between the solid and liquid parts of the milk are visible in the global component. Perhaps this technique could be used in some sort of diagnostic way to detect spoiled milk, although smelling it works just as well and much faster.

Issues

I didn't run into too many issues with this project. The main problems involved issues with the projector. By the nature of some of the scenes, the projector couldn't keep the checkerboard pattern completely in focus for every point on the object, so especially in scenes with rounded objects, it is easy to see some artifacts of the checkerboarding on the results. Similarly the pixel density of the projector left some black gaps where no light hit if the objects were small enough, which detracts from the analysis. It manifests itself as horizontal black stripes across the objects. It is especially noticeable in the direct component of the eggs. The pixels recording those black lines have no useful information, but since pixels are independent of one another during the analysis, it still produced decent results.

Special Thanks and Notes

I'd like to acknowledge some people who helped me out working on this project. First and foremost I want to thank Nick Sinnott-Armstrong. Nick let me use his very nice Nikon D300 camera to take all of my images. He also helped me set up the apparatus with the projector and did a lot of the camera parameters. He also had the brilliant idea of scripting the whole process of image capture, which sped things up immensely. Nick owns or donated many of the objects in the scenes as well. I would not have been able to do this project without his photographic expertise.

I'd like to thank John Ryan-Henry for being my hand model and for acquiring various crystalline minerals from the Geology Department for me to image. I'd also like to thank Emily Matteson and Audrey Lew for donating some objects (mostly food) for these scenes. Last I'd like to thank Technology House Inc. for letting me use the house projector for this project.

All of the images were taken on Nick's Nikon D300 with a 50mm f/1.8 lens. The images were at ISO 100 with a 1/6 second exposure and f22. The white balance was manually set to 6500K. The framing and focusing was all done manually for each scene. The projector is a Viewsonic PJD6553w with a 1280x800 native display. We displayed the checkerboard image at that resolution. Below you'll find two images of the setup. I moved the projector halfway through imaging because some of the objects were not getting enough light.