High Dynamic Range and Tone Mapping

By Yan Li (yanli)

 

           

    说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:garden:1_160.jpg说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:garden:1_320.jpg说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:garden:1_800.jpg说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:garden:1_1600.jpg说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:garden:1_3200.jpg说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:garden:1_4000.jpg          说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:garage:1_40.JPG说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:garage:1_160.JPG说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:garage:1_640.JPG

 

Introduction

The objective of the project is to render a HDR image from several images of different exposures. Modern cameras are unable to capture the full dynamic range of the world. The dynamic range of radiance, generally speaking, is from 1 to 1000, or even larger. However, as the images are often stored in 8-bit format, the maximum intensity that can be saved is 255. Some large radiance values are cropped or clamped to 255, and small radiance values are cropped to 0 as well. Because of that, one has to choose the range of radiance values of interest.

 

This project implements a method for recovering the high dynamic range image, using some global operators and tone mapping. Simple global operators often cannot fully achieve the goal of recovering HDR image, because it does not take local contrast into consideration. I use tone mapping to preserve such contrast and combine it with global operators, which can finally generate an excellent result.

 

My Works

·        Recovering HDR from images at various exposures

·        Results with global tone mapping

·        Bilateral filter

·        Local tone mapping

·        Optimization: fast bilateral filter (Extra credit)

·        Optimization: Image alignment (Extra credit)

·        Other method: Gradient based method (Part of it)(Extra credit)

 

Overview of Algorithm

·       Recovering HDR

The first step is to recover the response curve between pixels and real radiance. Several images of different exposures are already provided. With these images, I randomly sample many positions and get the corresponding pixel value in each image. The observed pixel value Zij for pixel i in image j is a function of Zij = f(Ei*tj), where Ei is the unknown radiance. I have known tj, since I have those different exposures images. I solve the inverse function of f in log domain, g(Zij). As I have sampled many different positions, I can build n equations where n is the number of samples. This is still not enough for solving the equation. I suppose the response curve is a smooth and monotonic curve, since a larger radiance won’t correspond to a smaller pixel value. Then I give the smooth part a lambda value, which is used to adjust the weight of the smooth part and I also give different weights for different g. Finally, I nail down a value to be zero and using x = A\b in Matlab to solve the equation system in least square sense.

 

·       Building radiance map

The second step is to build the radiance map, which may range from a very small value, e.g. 0.0005, to a quite large value, e.g. 1000. Because I have got all the possible g from the last step, the radiance map is easily obtained by simple substitution. I reuse the weighting function to give higher weight to exposures in which the pixel’s value is closer to the middle of the curve.

 

·       Tone mapping

The final step is tone mapping. I decompose the image into two layers: base layer and detail layer and only do reducing contrast for the base layer while preserve the detail layer. The first step is to separate the color with intensity, and I only do contrast reducing in log domain of the intensity. I use bilateral filter to get the base layer. Bilateral filter is a technique to blur the image while preserve the edges between regions of large contrast, which is totally different from simple Gaussian blur. After reducing the contrast of the base layer, I compose the base layer and detail layer together and get the new color channels from the intensity that has been reduced contrast. At last, I use gamma correction to make the image brighter. I don’t map the final color channels to 0.0~1.0 since I think the final result is already very good.

 

HDR Result

Here are all the results of the test cases provided by the project. I didn’t use my photos. In the following part I will show the radiance map, the detail layer, different exposure images and the final HDR image. Because I use image alignment, some boundary is cut so the size of the result image is a bit smaller than the original one. I will introduce alignment later. The radiance map should be three channels, but since each channel’s response curve is similar, I average them and show the one channel version.

       

    说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:house:1_320.JPG 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:house:1_640.JPG 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:house:1_1250.JPG

 

      

    说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:mug:1_2.jpg 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:mug:1_4.jpg 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:mug:1_8.jpg 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:mug:1_20.jpg 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:mug:1_40.jpg 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:mug:1_80.jpg 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:mug:1_160.jpg 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:mug:1_320.jpg

 

      

    说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:arch:17_1.jpg 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:arch:3_1.jpg 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:arch:1_25.jpg 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:arch:1_4.jpg

 

      

    说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:chapel:32_1.png 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:chapel:16_1.png 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:chapel:8_1.png 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:chapel:4_1.png 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:chapel:2_1.png 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:chapel:1_1.png 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:chapel:1_2.png 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:chapel:1_4.png 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:chapel:1_8.png 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:chapel:1_16.png 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:chapel:1_32.png 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:chapel:1_64.png 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:chapel:1_128.png 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:chapel:1_256.png 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:chapel:1_512.png 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:chapel:1_1024.png

 

      

    说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:garden:1_80.jpg 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:garden:1_160.jpg 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:garden:1_320.jpg 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:garden:1_800.jpg 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:garden:1_1600.jpg 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:garden:1_3200.jpg 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:garden:1_4000.jpg

 

      

    说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:garage:1_40.JPG 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:garage:1_160.JPG 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:garage:1_640.JPG

 

      

    说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:bonsai:1_2.jpg 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:bonsai:1_4.jpg 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:bonsai:1_10.jpg 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:bonsai:1_25.jpg 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:bonsai:1_320.jpg

 

      

    说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:window:15_1.jpg 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:window:4_1.jpg 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:window:1_1.jpg 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:window:1_4.jpg 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:window:1_15.jpg 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:window:1_250.jpg 说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:data:window:1_1000.jpg 

 

Image Alignment (Extra credit)

I did the image alignment like the project 1, using normalized correlation as the matching metric. I align those images with different exposures using their gray scale image. The process of alignment is very time-consuming, and I have to say that Matlab is doing alignment in my project rather than bilateral filter in most of the running time. The average cost of time is around 1 minute, depending on how many images are needed to be aligned. If I don’t do alignment, some of the results will be terrible since the samples are not accurate. I also notice, in some images the orientation of the camera is changed. In that situation, simple alignment may not be a good method.

Here I want to compare the results with alignment and without alignment. On the left side are the results without alignment, and the right side are the results with it. You can notice that “ghost” appears if not doing alignment. I marked them with red circles. The wrong sample data makes the final response curve wrong. The curve has sharp jaggy.

  

       说明: Macintosh HD:Users:yanli:Documents:delete_later:untitled2.jpg                        说明: Macintosh HD:Users:yanli:Documents:delete_later:untitled3.jpg

Global Operator

In this project, I also implement a simple global operator, using Lnew = L / (1+L), and then apply a gamma compression to make the image brighter. Different images may need different gamma, so the process is not automatic. I want to show some of the results. The results are not as good as those generated by combining local tone mapping and global operator. The left one is the result using global operator, and the right one is obtained by tone mapping.

说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:code 4:results:global_simple.png  说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:code 4:results:durand.png

 

说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:code 4:results:global_simple.png  说明: Macintosh HD:Users:yanli:Documents:ComputerPhotography:proj5:code 4:results:durand.png

 

Fast Bilateral Filter (Extra Credit)

I have implemented the fast bilateral filter as well. The naïve version of bilateral filter costs a huge amount of time. Generally speaking, it costs around 10 minutes for a 1600*1200 image. My fast bilateral filter uses five-step approximation of Gaussian. I just follow the instructions of what the guest speaker said. The result turns to be very fast (less than 1 second).

I firstly downsample the image and build a 3-D space grid. Then I compute the convolution between the new grid and the five-step Gaussian. At last I use interp3() in Matlab to tri-linearly interpolate the values and get the final blurry results. Because it’s just the enhancement in efficiency, and the method has been illustrated in that class, I won’t show any images or illustrate the algorithm here.

 

Gradient domain tone mapping (Extra Credit)

I have tried to implement the Gradient domain tone mapping. The method is from Raanan Fattal et al. However I think I haven’t successfully implemented the algorithm, only part of it. The map of gradient attenuation turns to be OK, but I think I might get something wrong in the last step, which is to solve the Poisson equation. As far as I can see, the boundary condition in my implementation should be problematic. Here I want show the map of gradient attenuation and the final results generated by this algorithm. The images are darker than those got by Durant’s method, and the boundary is even darker.  Because the time is limited, I haven’t found the reasons, and I want to improve that later. The left image is the result image and the right one is map of gradient attenuation.