System and methods for random parameter filtering

Abstract

The invention produces a higher quality image from a rendering system based on a relationship between the output of a rendering system and the parameters used to compute them. Specifically, noise is removed in rendering by estimating the functional dependency between sample features and the random inputs to the system. Mutual information is applied to a local neighborhood of samples in each part of the image. This dependency is then used to reduce the importance of certain scene features in a cross-bilateral filter, which preserves scene detail. The results produced by the invention are computed in a few minutes thereby making it reasonably robust for use in production environments.

Description

In one embodiment, a value of 30 instead of 3 is used when testing the world position since it varies much more than the other features. Also, this test is only done when σ_,k^f>0.1 because it is not desirable to throw all the samples away in cases where the variance is very small, such as constant-valued regions.

Before the statistical dependencies for a set of samples in a neighborhood is computed, the scene features are normalized by removing the mean and dividing by the standard deviation for each of the elements in the sample vector. The reason for this is that the features in f reside in very different coordinate systems. For example, world positions could be in the range of 0 to 1000, while the normal vector could have components in the range of 0 to 1. If this discrepancy is not corrected, a larger weight could inadvertently be given to certain features when calculating dependency that may not necessarily be more important. Vectors that have been normalized in this manner are represented with a bar, for example, f becomes f.

The core of the algorithm according to the invention is the computation of the color weights α and feature weights β as shown in FIG. 5. Mutual information is used to compute the statistical dependencies between a sample feature and the inputs to the Monte Carlo system.

Since it is difficult to derive an exact functional relationship between scene features and the inputs of the rendering system p_i and r_i for complex scenes, a statistical dependency is proposed based on the inputs providing information about the scene features. The invention uses mutual information, which is the exact measure of dependence between two random variables and indicates how much information one tells about another. In one embodiment, the mutual information between two random variables X and Y can be calculated as:

$\begin{array}{cc}\mu \left(X;Y\right)=\sum _{x\in X}\sum _{y\in Y}p\left(x,y\right)\log \frac{p\left(x,y\right)}{p\left(x\right)p\left(y\right)}& \left(\mathrm{Eq}.1\right)\end{array}$
where these probabilities are computed over the neighborhood of samples around a given pixel. To calculate the mutual information between two vectors x and y, specifically, f_,k and r_,l, respectively, the histogram for each is calculated. To compute the histograms, all values are made positive by subtracting the minimum element in the vector and quantizing the elements into integer bins by rounding their values. How many times the values of x fall inside each bin are counted and the probabilities are found by dividing by the length of x. A joint histogram is calculated in a similar way, except with pairs of values (x, y).

To estimate statistical dependencies on inputs, the dependency of the k^th scene feature on all random parameters (given by D_f,k^r) is calculated using mutual information. The heuristic approximates this by measuring the dependency on individual random parameters and adding them up. Therefore, the statistical dependency is first calculated between the k^th scene feature and the l^th random parameter by D_f,k^r,l=μ(f_,k; r_,l), and then the dependency of the k^th scene feature on all n random parameters is approximated as:

$\begin{array}{cc}{D}_{f,k}^r=\sum _{1\le l\le n}{D}_{f,k}^{r,l}=\sum _{1\le l\le n}\mu \left({\stackrel{\_}{f}}_{N,k};{\stackrel{\_}{r}}_{N,l}\right)& \left(\mathrm{Eq}.2\right)\end{array}$

The dependency of the k^th scene feature on screen position (D_f,k^p) and color dependencies D_c,k^r and D_c,k^p are similarly computed:

$\begin{array}{cc}{D}_{f,k}^p=\sum _{1\le l\le 2}{D}_{f,k}^{p,l}=\sum _{1\le l\le 2}\mu \left({\stackrel{\_}{f}}_{N,k};{\stackrel{\_}{p}}_{N,l}\right),& \left(\mathrm{Eq}.3\right)\\ D_{c,k}^r=\sum _{1\le l\le n}{D}_{c,k}^{r,l}=\sum _{1\le l\le n}\mu \left({\stackrel{\_}{c}}_{N,k};{\stackrel{\_}{r}}_{N,l}\right),& \left(\mathrm{Eq}.4\right)\\ D_{c,k}^p=\sum _{1\le l\le 2}{D}_{c,k}^{p,l}=\sum _{1\le l\le 2}\mu \left({\stackrel{\_}{c}}_{N,k};{\stackrel{\_}{p}}_{N,l}\right).& \left(\mathrm{Eq}.5\right)\end{array}$

The dependency of the k^th color channel on all the scene features is computed so that later the weight for features that do not contribute to the final color can be reduced:

$\begin{array}{cc}{D}_{c,k}^f=\sum _{1\le l\le m}{D}_{c,k}^{f,l}=\sum _{1\le l\le 2}\mu \left({\stackrel{\_}{c}}_{N,k};{\stackrel{\_}{f}}_{N,l}\right).& \left(\mathrm{Eq}.6\right)\end{array}$

In addition, a related term is computed pertaining to how all color channels are dependent on the k^th scene features:

$\begin{array}{cc}{D}_c^{f,k}=\sum _{1\le l\le 3}{D}_{c,l}^{f,k}=\sum _{1\le l\le 3}\mu \left({\stackrel{\_}{c}}_{N,l};{\stackrel{\_}{f}}_{N,k}\right).& \left(\mathrm{Eq}.7\right)\end{array}$

Finally, the terms D_c,k^r, D_c,k^p, D_c^f are calculated by summing over the color channels:

$\begin{array}{cc}{D}_c^r=\sum _{1\le k\le 3}{D}_{c,k}^r,D_c^p=\sum _{1\le k\le 3}{D}_{c,k}^p,D_c^f=\sum _{1\le k\le 3}{D}_{c,k}^f.& \left(\mathrm{Eq}.8\right)\end{array}$

As shown in FIG. 5, the error of the approximation is determined. Ideally, the statistical dependency of the k^th scene feature on all random parameters is calculated using the joint mutual information μ(r_,1, r_,2, . . . , r_,n;f_,k). Unfortunately, this joint mutual information can be difficult and expensive to compute as the number n gets larger, because the histogram grows to the power of n while the number of samples to do statistics grows linearly. This means that the ability to compute the histogram accurately degenerates quickly and it becomes much slower to do so. For this reason, the approximation is performed by measuring the dependency on individual random parameters and adding them up as described above.

Now the effect this has in the overall calculation of statistical dependency is examined. To keep things simple, it is assumed that two statistically independent random variables R₁ and R₂ are inputs to the system and produce the output feature Y. In order to measure μ(R₁, R₂; Y), it is approximated as μ(R₁;Y)+μ(R₂;Y). The following derivative shows that the statistical dependence is underestimated:
μ(R₁,R₂;Y)=H(R₁,R₂)−H(R₁,R₂|Y)
=H(R₁)+H(R₂|R₁)−H(R₁|Y)−H(R₂|R₁,Y)
=μ(R₁;Y)+H(R₂|R₁)−H(R₂|R₁,Y)

If R₁ are R₂ are independent, then H(R₂|R₁)=H(R₂) such that:
μ(R₁,R₂;Y)=μ(R₁;Y)+H(R₂)−H(R₂|R₁,Y)
=μ(R₁;Y)+μ(R₂;R₁,Y)
=μ(R₁;Y)+μ(Y,R₁;R₂)
=μ(R₁;Y)+μ(R₂;Y)+μ(R₁;R₂|Y)

Thus, the approximation that μ(R₁, R₂; Y)≈μ(R₁; Y)+μ(R₂; Y) effectively assumes that μ(R₁; R₂|Y)=0. This means that the information is ignored that the output feature tells about relationship between the inputs, which might not be zero even though the inputs are statistically independent. To understand why, the function ƒ( ) is set to act as an XOR gate of two inputs. If one of the inputs and the output is known, the other input can automatically be determined even though the two inputs may be statistically independent. Since μ(R₁; R₂|Y)≥0, the approximation is an underestimate of the true joint mutual information between the random parameters and the scene feature. However, the approximation works quite reasonably, even for intricate scenes with complex relationships between the random parameters and the scene features.

Since the sample features are only functions of the random parameters r_i and the screen position p_i, the heuristic computes the fractional contribution of the random parameters to the k^th scene feature with the following formula:

$\begin{array}{cc}{W}_{f,k}^r=\frac{D_{f,k}^r}{D_{f,k}^r+D_{f,k}^p+\varepsilon }& \left(\mathrm{Eq}.9\right)\end{array}$

The addition of the term ϵ term prevents degeneration when the dependencies D_f,k^r and D_f,k^p are both small. This expression communicates how much the k^th feature was affected by the random parameters as a fraction of the contributions from both sets of inputs, with the reasonable assumption that the position and random parameters are statistically independent. When the sample is only a function of the random parameters, this value will be close to 1, and when it is dependent only on the screen position it will be 0. In the common case where there is some contribution from both inputs, for example, a partially out-of-focus object is dependent on both screen position and u, v, the invention simply interpolates between the two.

The invention also includes a similar algorithm using the dependencies of the k^th sample color channel on the random parameters D_c,k^r and on the screen position D_c,k^p to compute the fractional contribution of the random parameters on the k^th color channel:

$\begin{array}{cc}{W}_{c,k}^r=\frac{D_{c,k}^r}{D_{c,k}^r+D_{c,k}^p+\varepsilon }& \left(\mathrm{Eq}.10\right)\end{array}$

The overall contribution of the random parameters on the color W_c^r for use in sizing the filter is obtained by averaging the W_c,1^r, W_c,2^r, W_c,3^r terms:

$\begin{array}{cc}{W}_c^r=\frac{1}{3}\left(W_{c,1}^r+W_{c,2}^r+W_{c,3}^r\right).& \left(\mathrm{Eq}.11\right)\end{array}$

Lastly, the W_c^f,k term is computed to communicate how much the color depends on a specific feature:

$\begin{array}{cc}{W}_c^{f,k}=\frac{D_c^{f,k}}{D_c^r+D_c^p+D_c^f}& \left(\mathrm{Eq}.12\right)\end{array}$

As shown in FIG. 5, the computation of the fractional contributions is determined. Specifically, once the statistical dependencies have been calculated, the normalized dependencies are computed to determine the α and β parameters. The α and β parameters are adjusted in each iteration as the block size decreases by giving more weight to the dependency on the random parameters. The idea behind this is that when the block sizes are large, there will be an increase in dependency on the spatial screen position because of the natural spatial variations in the image. However, the statistics are more corrupt because of the mixing of statistics that happens with large block sizes. Therefore, more weight is given to the normalized dependency on the random parameters as the block size goes down with each iteration. This adjustment is expressed as:
α_k=max(1−2(1+0.1t)W_c,k^r,0)  (Eq. 13)
β_k=W_c^f,k·max(1−(1+0.1t)W_f,k^r,0)  (Eq. 14)
where the t term is the iteration of the multi-pass approach, with the first pass t=0. The incorporation of the t term increases the weight of W_c,k^r and W_f,k^r upon each successive iteration, and the max( ) term is added to ensure that the value stays positive.

As shown in FIG. 6, the color samples are filtered. The invention filters the color of samples x_i using a weighted bilateral filter in which the importance of the color and scene features is adjusted to reflect their dependence on the random parameters:

$\begin{array}{cc}{w}_{\mathrm{ij}}=\exp \left[-\frac{1}{2{\sigma }_p^2}\sum _{1\le k\le 2}{\left({\stackrel{\_}{p}}_{i,k}-{\stackrel{\_}{p}}_{j,k}\right)}^2\right]\times \exp \left[-\frac{1}{2{\sigma }_c^2}\sum _{1\le k\le 3}{{\alpha }_k\left({\stackrel{\_}{c}}_{i,k}-{\stackrel{\_}{c}}_{j,k}\right)}^2\right]\times \exp \left[-\frac{1}{2{\sigma }_f^2}\sum _{1\le k\le m}{{\beta }_k\left({\stackrel{\_}{f}}_{i,k}-{\stackrel{\_}{f}}_{j,k}\right)}^2\right],& \left(\mathrm{Eq}.15\right)\end{array}$
where w_i,j is the contribution—or weight—of the j^th sample to the i^th sample during filtering. Because of the way the samples are selected in neighborhood randomly using a Gaussian distribution with standard deviation σ_p (where σ_p=b/4), the first term of this expression is dropped and becomes:

$\begin{array}{cc}{w}_{\mathrm{ij}}=\exp \left[-\frac{1}{2{\sigma }_c^2}\sum _{1\le k\le 3}{{\alpha }_k\left({\stackrel{\_}{c}}_{i,k}-{\stackrel{\_}{c}}_{j,k}\right)}^2\right]\times \exp \left[-\frac{1}{2{\sigma }_f^2}\sum _{1\le k\le m}{{\beta }_k\left({\stackrel{\_}{f}}_{i,k}-{\stackrel{\_}{f}}_{j,k}\right)}^2\right].& \left(\mathrm{Eq}.16\right)\end{array}$

The variances of the Gaussians for both the color and the feature are set to the same value:

$\begin{array}{cc}{\sigma }_c^2={\sigma }_f^2=\frac{{\sigma }^2}{{\left(1-W_c^r\right)}^2}& \left(\mathrm{Eq}.17\right)\end{array}$

The variances are divided by (1−W_c^r)² because, in the end, only the sample color is of importance and a large filter is desired wherever the color depends a lot on the random parameters, i.e., is very noisy. This term adjusts the size of the Gaussian based on the overall noise level, making it large when needed. The terms σ_c² and σ_f² are separated since they depend on all three color channels (because of the W_c^r term) as opposed to α_k (whose W_c,k^r term varies per color channel). Therefore, the terms σ_c² and σ_f² modulate the overall size of the Gaussian while α_k and β_k adjust it further based on dependencies with the random parameters.

Because the constant σ_s² is divided by the number of samples when computing the filter's variance σ², the invention is a biased but consistent estimator, meaning that the estimator converges to the value of the integral as the number of samples per pixel s goes to infinity. As s→∞, a weight of w_i,j=1 is produced only when i=j and zero is everywhere else. Therefore, the colors of the samples are not filtered at all, so the invention converges to standard Monte Carlo, which is a consistent estimator.

Once the filter weights w_i,j are obtained, these weights are used to blend in the color contributions from these samples:

$\begin{array}{cc}{c}_{i,k}^{″}=\frac{\sum _{j\in N}{w}_{\mathrm{ij}}{c}_{j,k}^{\prime }}{\sum _{j\in N}{w}_{\mathrm{ij}}}& \left(\mathrm{Eq}.18\right)\end{array}$
where the denominator is never zero because at least w_i,i=1 (a sample fully contributes to itself). Note that this process filters the colors of individual samples (not pixels), and is performed separately for every pixel in the image, since statistics change from pixel to pixel. After all samples in the image have been filtered, the process is repeated with a new iteration as shown in FIG. 3.

The invention can be applied to a variety of scenarios such as using only direct lighting to highlight the illumination contribution from path tracing. Many regions of the image may be completely dark when using only direct lighting because these regions are totally occluded from the sky light source. This means that the illumination in these regions that is visible is due exclusively to path-tracing. The invention is able to denoise these regions by examining the relationship between the sample values and the random parameters used to compute the bounces of each path.

Path-tracing is notoriously noisy, and when Monte Carlo samples are input to the algorithm, much of the detail in the textures in the scene is completely gone. This is more evident if the color channel is multiplied by 1000. Many of the pixels remain black, which indicates that these pixels have no useful color information. The cross-bilateral filter examines other sample features, such as world position, surface normal, and texture value, each weighted depending on their amount of functional dependency on the random parameters. In embodiments where the samples' colors are extremely noisy because the path tracing produces a lot of noise while computing the global illumination, the invention detects the connection between the sample color and the random parameters of the path tracer, and essentially ignores the color when bilateral filtering. The texture value, on the other hand, is found to have little dependence on the random parameters so it is weighted heavily by the cross-bilateral filter.

Therefore, to filter a sample its color is ignored while close attention is paid to its texture value. When blending in values from around the filter kernel, only samples with similar texture values are blended together. Therefore, if a sample hits a dark part of the texture, samples from other parts of the texture are blended in that are also dark. Essentially, the filter according to the invention combines many noisy samples of dark texture together to approximate a noise-free dark texture. Of course, some blurring of the texture detail occurs when a large filter kernel is used to help denoise a very noisy image. Ideally, a small filter is desired to help preserve detail. The invention is reasonably robust and of great importance in production environments.

Although color is treated as a special feature since the color channel is filtered, it is contemplated that only incident illumination may be filtered and used in a surface shader to get an improved result.

FIG. 7 illustrates an exemplary computer rendering system 700 that may be used to implement the methods according to the invention. One or more computer systems 700 may carry out the methods presented herein as computer code.

Computer system 700 includes an input/output display interface 702 connected to communication infrastructure 704—such as a bus—, which forwards data such as graphics, text, and information, from the communication infrastructure 704 or from a frame buffer (not shown) to other components of the computer system 700. The input/output display interface 702 may be, for example, a keyboard, touch screen, joystick, trackball, mouse, monitor, speaker, printer, any other computer peripheral device, or any combination thereof, capable of entering and/or viewing data.

Computer system 700 includes one or more processors 706, which may be a special purpose or a general-purpose digital signal processor that processes certain information. Computer system 700 also includes a main memory 708, for example random access memory, read-only memory, mass storage device, or any combination thereof. Computer system 700 may also include a secondary memory 710 such as a hard disk unit 712, a removable storage unit 714, or any combination thereof. Computer system 700 may also include a communication interface 716, for example, a modem, a network interface (such as an Ethernet card or Ethernet cable), a communication port, a PCMCIA slot and card, wired or wireless systems (such as Wi-Fi, Bluetooth, Infrared), local area networks, wide area networks, intranets, etc.

It is contemplated that the main memory 708, secondary memory 710, communication interface 716, or a combination thereof, function as a computer usable storage medium, otherwise referred to as a computer readable storage medium, to store and/or access computer software including computer instructions. For example, computer programs or other instructions may be loaded into the computer system 700 such as through a removable storage device, for example, a floppy disk, ZIP disks, magnetic tape, portable flash drive, optical disk such as a CD or DVD or Blu-ray, Micro-Electro-Mechanical Systems, nanotechnological apparatus. Specifically, computer software including computer instructions may be transferred from the removable storage unit 714 or hard disc unit 712 to the secondary memory 710 or through the communication infrastructure 704 to the main memory 708 of the computer system 700.

Communication interface 716 allows software, instructions and data to be transferred between the computer system 700 and external devices or external networks. Software, instructions, and/or data transferred by the communication interface 716 are typically in the form of signals that may be electronic, electromagnetic, optical or other signals capable of being sent and received by the communication interface 716. Signals may be sent and received using wire or cable, fiber optics, a phone line, a cellular phone link, a Radio Frequency link, wireless link, or other communication channels.

Computer programs, when executed, enable the computer system 700, particularly the processor 706, to implement the methods of the invention according to computer software including instructions.

The computer system 700 described herein may perform any one of, or any combination of, the steps of any of the methods presented herein. It is also contemplated that the methods according to the invention may be performed automatically, or may be invoked by some form of manual intervention.

The computer system 700 of FIG. 7 is provided only for purposes of illustration, such that the invention is not limited to this specific embodiment. It is appreciated that a person skilled in the relevant art knows how to program and implement the invention using any computer system.

The computer system 700 may be a handheld device and include any small-sized computer device including, for example, a personal digital assistant, smart hand-held computing device, cellular telephone, or a laptop or netbook computer, hand held console or MP3 player, tablet, or similar hand held computer device, such as an iPad®, iPad Touch® or iPhone®.

While the invention has been described with reference to particular embodiments, those skilled in the art will recognize that many changes may be made thereto without departing from the scope of the invention. Each of these embodiments and variants thereof is contemplated as falling with the scope of the claimed invention, as set forth in the following claims.

Claims

1. A method for performing a random parameter filter Monte Carlo denoising, comprising the steps of:

rendering one or more image samples or one or more pixels at a given sampling rate;

storing a vector of a plurality of scene features and one or more rendering system inputs for each image sample or each pixel;

saving one or more random parameters for each image sample used to calculate the image sample by a rendering system;

choosing the one or more image samples or the one or more pixels to process;

performing pre-processing processing on the one or more image samples or the one or more pixels using one or more stored features and filter parameters that are automatically adjusted for each image sample or each pixel to compute filter weights;

calculating a dependency of a color and a feature of the one or more random parameters and one or more rendering system inputs to obtain a calculated dependency;

using the calculated dependency to determine a weight for each scene feature to obtain dependency information;

modifying the one or more image samples using the dependency information to obtain a final modified sample;

filtering the final modified sample to produce one or more pixels one or more image samples or the one or more pixels using the computed filter weights to compute a filtered value; and

outputting a final image, wherein the final image includes the computed filtered value of each image sample or each pixel.

2. The method for performing a random parameter filter Monte Carlo denoising according to claim 1, wherein the choosing step further comprises the step of conducting iterations of using a block around a pixel of the image sample the one or more pixels from a large block size to a small block size.

3. The method for performing a random parameter filter Monte Carlo denoising according to claim 1, wherein the choosing step further comprises the step of selecting a random subset of image samples within each a block around the one or more pixels.

4. The method for performing a random parameter filter Monte Carlo denoising according to claim 1, wherein the performing step further comprises the step of clustering the one or more image samples or the one or more pixels into one or more groups.

5. The method for performing a random parameter filter Monte Carlo denoising according to claim 4, wherein, wherein the clustering step includes the step of calculating the a standard deviation of the a mean for the one or more pixels of the image sample.

6. The method for performing a random parameter filter Monte Carlo denoising according to claim 1, wherein the performing step further comprises the step of manipulating the vector by removing the a mean and dividing by the a standard deviation for each scene feature of the plurality of scene features, wherein each feature is a scene feature for each image sample.

7. The method for performing a random parameter filter Monte Carlo denoising according to claim 1 13, wherein the a dependency is a statistical dependency.

8. The method for performing random parameter filter Monte Carlo denoising according to claim 1, wherein the one or more scene features each feature is at least one selected from the group comprising consisting of: world position, surface normal, color, texture value, texture coordinate, and shader value.

9. The method for performing random parameter filter Monte Carlo denoising according to claim 1, wherein the one or more rendering system inputs is at least one selected from the group comprising consisting of: screen position and random parameter.

10. The method for performing a random parameter filter Monte Carlo denoising according to claim 1, wherein the filtering step further comprises the steps of:

classifying the weight filter weights above a certain value to mean that the scene a feature of the plurality has little or no dependency on the one or more random parameters and the weight filter weights below a certain value to mean that the scene a feature of the plurality has a significant dependency on the one or more random parameters.

11. The method for performing Monte Carlo denoising according to claim 1 further comprising the step of reconstructing the computed filtered value of each image sample or each pixel to obtain the final image.

12. The method for performing Monte Carlo denoising according to claim 1, further comprising the step of saving by a rendering system one or more random parameters for each image sample or each pixel used to calculate the image sample.

13. The method for performing Monte Carlo denoising according to claim 12, further comprising the steps of:

calculating a dependency of a color and a feature of the one or more random parameters and the one or more rendering system inputs to obtain a calculated dependency;

using the calculated dependency to determine a weight for each feature to obtain dependency information; and

modifying the one or more image samples or the one or more pixels using the dependency information to obtain a final modified sample.