Method and apparatus for reduction mosquito noise in decoded images

Abstract

A method and an apparatus of reducing the mosquito noise in a decoded image by using a noise reduction algorism that does not depend on the compression coding algorism applied to the image to be improved. The decoded compression-coded digital image is divided into blocks. The standard deviation or the “variance coefficient” of luminance values is calculated for each block of the decoded compression-coded digital image. An ε matrix for the compression-coded digital image is composed so that all components of each block of the ε matrix is filled with the standard deviations or the variance coefficients for each block. An ε filter is composed so that its ε values are expressed by the ε matrix. The decoded compression-coded digital image is applied to the ε filter to reduce the mosquito noise that arises during decoding process.

Description

BACKGROUND OF THE INVENTION

The invention relates to an algorithm of reducing noises that arise during decoding process for a compression-coded digital image, especially to a method and an apparatus of reducing the mosquito noise using said algorithm, applied to the JPEG compression coding scheme.

DESCRIPTION OF THE PRIOR ART

The JPEG (Joint Photographic Experts Group: a cooperative working body between ISO and ITU-TS for image compression standardization) compression coding scheme is generally used for compression of a digital image. Although both reversible and irreversible coding methods are provided in the JPEG compression coding scheme, the JPEG baseline algorithm that is a kind of irreversible coding methods is generally used because of its high compression efficiency. A JPEG compressed image can be obtained by using DCT (Discrete Cosine Transform), a quantization table and the Haffman coding scheme. Although an irreversible coding method can realize a drastically high degree of compression of an image, it often induces degradation or noises in the image in return.

The JPEG2000 compression coding scheme was standardized in January 2001 as a new standard that was extended from the JPEG compression coding scheme. Presently, however, the JPEG compression coding scheme is used widely, since there are not so many software programs or software/hardware plug-ins available that use the JPEG2000 format, as compared to those which use the JPEG format.

In the JPEG2000 compression coding scheme, an image is divided into two portions horizontally and then divided into two portions vertically, resulting in four image elements each of which has the half horizontal size and the half vertical size of the original image. Each of these four elements is characterized as a combination of a low frequency component which is visually significant and a high frequency component which is visually less-significant, such as “the low frequency component and the low frequency component”, “the low frequency component and the high frequency component”, “the high frequency component and the low frequency component” and “the high frequency component and the high frequency component”. A hierarchical set of elements is generated by recurrently applying the above-mentioned process to low frequency components which are visually important. During the process, data can be compressed by quantizing high frequency components coarsely as compared to low frequency components. Processing flows of JPEG2000 coding and decoding are shown in FIG. 4.

The block noise in JPEG2000 coding is less conspicuous as compared to the one in the JPEG coding where DCT is applied to each of 8×8 blocks, since DWT (Discrete Wavelet Transform) is applied for JPEG2000 coding. JPEG2000 coding also enables reduction of the mosquito noise. On the other hand, however, degradation of image quality increases as the degree of compression increases in JPEG2000 coding; Unsightly degradation often appears at an edge portion in the image, since the edge portion where the change in luminance is sharp tends to become dull after JPEG2000 coding and decoding are applied there. FIG. 5(b) shows the result after JPEG2000 coding and decoding are applied to the image of FIG. 5(a), as an example of typical degradation. FIGS. 6(a) and 6(b) show luminance value changes for pixels at each of the same horizontal positions in FIGS. 5(a) and 5(b). Degradation of image quality at the edge portion can be observed clearly from these figures that show luminance value changes.

Major noises arising from the JPEG compression coding scheme are the block noise and the mosquito noise, both of which are unsightly for human eyes (the false contour noise may arise in the JPEG2000 compression coding scheme, although the block noise does not exist in the JPEG2000 scheme because of its algorism characteristics). The inventor has already proposed an algorithm that can reduce the block noise and the false contour noise among major noises mentioned above. This invention relates to an algorithm of reducing especially the mosquito noise, among noises arising during such high-degree compression coding such as the JPEG and the JPEG2000 schemes.

It is known that the mosquito noise arises in an image having a sharp contour, an isolated point or a slant line. It is also known that application of an ε-separation nonlinear digital filter (an ε filter, hereafter) is effective for reducing the mosquito noise while maintaining the contour. The inventor has already proposed a variable ε filter algorism where the value of ε is changed for each region of the image (this variable ε filter algorism will be called “the conventional algorism” hereafter). The ε filter can eliminate noises of which values are smaller than a prescribed ε value, while maintaining the sharp change in the image. The original image is kept unchanged when the ε value is set to 0. When the ε value becomes infinity, the ε filter works as a simple low-pass filter. Conceptual image of an ε filter is shown in FIG. 12.

The conventional algorism is explained in the following as Step 1 and as Step 2.

• (a) Step 1: Detection of a contour in an image by using the variance of the luminance values
  • Step 1.1: In order to detect a sharp contour in an image that induces degradation, calculate the variance of luminance values of N×N pixels around the pixel of interest and iterate this process for each pixel of the image, as shown by 10 in FIG. 8. Luminance values for the YCbCr color format are used as luminance values for the calculation, if the algorism is to be applied to a JPEG/JPEG2000 decoded color image. A luminance value is expressed by 256 steps (8 bits) for each of three primary colors. After calculating variance values for all pixels of the image, a matrix of valiance values for the image (the “variance matrix”), can be obtained as shown in FIG. 8.
  • Step 1.2: Considering that noises arise around a sharp contour, compare variance values obtained by Step 1 for each pixel within an M×M block of the image, and select the largest variance value in the region as the representative variance value for the pixel point at the center of the M×M block, as shown by 11 in FIG. 9. After calculating representative variance values for all pixels of the image,
    • an appropriate adjusting coefficient is multiplied to each of the representative variance values in order to obtain a matrix of ε values for the image (the “ε matrix” or the “threshold matrix”) as shown in FIG. 9.
• (b) Step 2: Execution of noise reduction filtering (variable ε filtering) Execute ε filtering for each pixel in the image by using ε values obtained for each block by Step 1 as shown in FIG. 10.

Through the above process, the ε filter that was effective in the JPEG compression coding scheme for reducing noises generated around a contour in an image is basically applied also to this JPEG/JPEG2000 compression coding schemes, in order to reduce noises around a contour in an image while maintaining the contour as it is. In FIG. 7(a) small variations (that are recognized as the mosquito noise) can be seen at the edge portion of the luminance value change, but they are eliminated clearly after ε filtering is applied as shown in FIG. 7(b). Since the ε filter works to set the output signal level zero if the input signal level lies outside of predetermined threshold values, it is important to decide appropriate threshold levels. For example, if the threshold level is set larger than the difference of signal levels at points across the border of the contour in the original image, not only variations caused by degradation but also meaningful information in the original image will be filtered, causing degradation in image quality. In Step 1 and 2 aforementioned, such loss of meaningful information can be avoided by adjusting the ε value of the filter according to the contour information in the image.

• (c) Characteristics of the ε Filter:

The transfer function of the ε filter is expressed by equations (1) and (2).

$\begin{array}{cc}{y}_{n,m}=\frac{\sum _{i=-\frac{M-1}{2}}^{\frac{M-1}{2}}\sum _{j=-\frac{M-1}{2}}^{\frac{M-1}{2}}T\left[x_{n+i,m+j}-x_{n,m}\right]·x_{n,m}}{\sum _{i=-\frac{M-1}{2}}^{\frac{M-1}{2}}\sum _{j=-\frac{M-1}{2}}^{\frac{M-1}{2}}T\left[x_{n+i,m+j}-x_{n,m}\right]},& \left(1\right)\\ T\left[x\right]=\{\begin{array}{cc}1& \left(-\varepsilon \le x\le \varepsilon \right)\\ 0& \left(x<-\varepsilon ,\varepsilon <x\right).\end{array}& \left(2\right)\end{array}$

Since the output luminance value y_n,m of this filter for the pixel of interest having an original (input) luminance value of x_n,m is the average of luminance values in the region considered (the M×M region around the pixel of interest that has the luminance value of x_n,m) only for pixels in the region that have a luminance value within the luminance value of x_n,m±ε, the filter can remove small noises within ±ε while maintaining a big change that occurs across the contour.

The filter has characteristics such that it passes through all input signals as they are when ε is 0 and that it works as a low-pass filter when ε is infinity. The inventor studied possible noise value distribution for several typical contours in order to obtain an appropriate value for N that is necessary for determining the variance matrix of the image, and an appropriate value for M that is necessary for determining the ε matrix of the image. As the result of the study, the inventor found that almost all of the noise value distributions could be covered when N=7 and M=9. Here the value of ε is set to a fixed value, 0.01 times the maximum value of the 9×9 variance matrix. By these settings, almost all noises arising around typical contours can be filtered.

In the conventional algorism explained above, variation degree of luminance values in a prescribed filtering window (or a “block”) for an image is expressed by the variance of luminance values of all pixel points within the block, by using each luminance value for each pixel point within the block (here, luminance values for the YCbCr color format are used as luminance values for the calculation, if the algorism is to be applied to a JPEG/JPEG2000 decoded color image). It is necessary in the conventional algorism, however, to set an appropriate coefficient for correcting the variance value in order to obtain proper ε values, and strictly speaking, the optimum coefficient value will change if the image changes (in the above explanation, the coefficient is set to a constant value of 0.01). FIG. 11 shows an example of a block having the size of 5×5.

The purpose of the invention is to revise the conventional ε filter algorism explained above so that the parameter of the noise reduction filter (ε value) can be automatically determined, in order to provide a method and an apparatus of reducing the mosquito noise in a decoded image by using the noise reduction algorism that does not depend on the compression coding algorism applied to the image.

SUMMARY OF THE INVENTION

The method or the apparatus of this invention for reducing the mosquito noise in a decoded image that arises during decoding process is characterized in that it uses an algorism wherein: (1) a decoded compression-coded digital image is divided into blocks, (2) the standard deviation or the “variance coefficient” of luminance values is calculated for each block of said decoded compression-coded digital image (in order to numerically express the degree of variance of luminance values in a block, the standard deviation or the “variance coefficient” of luminance values of pixels in the block is used for the invention, where the “variance coefficient” is the standard deviation of luminance values of pixels in the block divided by the average value of luminance values of pixels in the block), (3) an ε matrix for said compression-coded digital image is composed so that all components of each block of said ε matrix is filled with said standard deviations or said variance coefficients for each block, (4) an ε filter is composed so that its ε values are expressed by said ε matrix and (5) said decoded compression-coded digital image is applied to said ε filter to reduce the mosquito noise that arises during decoding process. The invention is especially effective when said compression-coded digital image is an image compressed by the JPEG compression coding scheme.

The algorism of the invention is practical and effective when it is applied to an image that needs picture quality improvement, since the algorism does not require calculation of coefficients for correcting variance values that are necessary for the conventional algorism, and consequently, the calculation is simpler and takes less time as compared to the conventional algorism.

DESCRIPTION OF DRAWINS

FIG. 1 explains how an ε matrix is generated.

FIG. 2 is a figure to explain the process of ε filtering.

FIG. 3 is a figure to explain the process of the JPEG2000 compression coding.

FIG. 4 is a block diagram to show flows of JPEG2000 coding and decoding.

FIG. 5 shows an example of JPEG2000 coding and decoding.

FIG. 6 shows luminance value changes at the edge portion of an original image (a) and at the edge portion of decoded image using the JPEG2000 coding scheme (b).

FIG. 7 shows characteristics of an ε filter.

FIG. 8 is a figure to explain the variance value calculation method in the conventional algorism.

FIG. 9 is a figure to explain the representative variance value calculation method in FIG. 8.

FIG. 10 is a figure to explain the process of conventional ε filtering.

FIG. 11 shows a 5×5 filtering window.

FIG. 12 is a figure to explain the effect of ε filtering.

FIG. 13 shows an original JPEG2000 decoded image used as a reference for comparing the mosquito noise reduction algorism of the invention with the conventional algorism.

FIG. 14 shows the image after applying ε filtering to the image of FIG. 13.

FIG. 15 shows the image after applying the conventional algorism to the image of FIG. 13.

FIG. 16 shows the image after applying the algorism of the invention to the image of FIG. 13.

FIG. 17 shows the original decoded image of a JPEG2000 compression-coded image that includes the mosquito noise.

FIG. 18 shows the image after applying the algorism of the invention to the image of FIG. 17.

FIG. 19 is a figure to explain the process of mirroring that is applied to a block for pixels that exist at the bottom/top left or the top/bottom right corners, or to a block for pixels that exist on one of the four sides of the image.

DESCRIPTION OF THE PREFERRED EMBODIMENT

The preferred embodiment of the invention is explained in detail in the following, referring to figures.

The algorism of this invention consists of two phases, “Phase I” for collecting image information and “Phase II” for applying ε filtering, as are explained in the following:

Phase I

In order to find out the region in an original decoded image (2 in FIG. 1) where a sharp contour exists, the image is divided into blocks (an example of the block size is 8×8 pixel points is used for the embodiment explained in the following). The standard deviation or the “variance coefficient” of luminance values (1 in FIG. 1) is calculated for each block of the decoded image. Here, the “variance coefficient” is the standard deviation of luminance values of pixels in the block divided by the average value of luminance values of pixels in the block. By filling each block with the standard deviation or the variance coefficient calculated for each respective block, a matrix that has components of which values are the same within each block is composed, resulting in a mosaic pattern of component values by blocks. The composed matrix is called the “ε matrix” (3 in FIG. 3). Each standard deviation or variance coefficient obtained by this procedure becomes the ε value of the ε filter for each block.

Phase II

ε filtering is then applied to each pixel in the original decoded image by using the ε matrix obtained in Phase I (5 in FIG. 2). As the result of the filtering, the average of luminance values of pixels within each block of an arbitrarily predetermined size is obtained for each pixel of interest in the original decoded image, as the output of ε filter (4 in FIG. 2) for each pixel of interest in the original decoded image. Here, “luminance values of pixels” can be obtained as pixel data of the original decoded image, and the luminance value average is taken only for pixels in the block satisfying the condition that luminance values of them are within the luminance value ±ε of the pixel of interest in the block. The averaged luminance value for each pixel is used as the noise-removed decoded luminance value for each pixel of the image. The same procedure of Phase I and Phase II is carried out for each color component when the image is a colored picture.

Further detailed procedure of Phase II is described in the following:

The output value of the ε filter corresponding to each pixel in the compression decoded image is calculated by the filter by using information in the ε matrix obtained by Step I. Although the block size in the calculation can be arbitral, the size of 5×5, 7×7 or 9×9 is recommended for use. FIG. 2 shows a schematic explanation for the procedure. The averaged output luminance value of the filter y_i,j corresponding to the input luminance value x_i,j for the pixel of interest in the decoded image is calculated as follows: By using each ε value for corresponding x_i,j (the value of the ε matrix component at the i-th row and the j-th column), values of x_i,j±ε can be determined for each x_i,j. Next, all luminance values within x_i,j±ε in the block that includes the pixel of interest are extracted and then the average of them is calculated as each output y_i,j corresponding to each input x_i,j. FIG. 19 shows how this procedure is carried out with respect to the x_i,j of pixel of interest, in case of the block size of 7×7. The calculation formula for the output y_i,j in response to the input of interest x_i,j is expressed by Equation (3) in the following.

$\begin{array}{cc}{y}_{\mathrm{ij}}=\frac{\sum _{k=-3}^3\sum _{l=-3}^{3}T\left[x_{\mathrm{ij}}-x_{i-k,j-l}\right]·x_{i-k,j-l}}{\sum _{k=-3}^3\sum _{l=-3}^{3}T\left[x_{\mathrm{ij}}-x_{i-k,j-l}\right]}& \left(3\right)\end{array}$
Here, T[x] is=1 for −ε<x<ε, and 0 for any other x value.

Above procedure is iterated for all pixels in the image. Blocks for the pixel of interest at the bottom/top left or the bottom/top right corner of the image or on one of four sides of the image can have the same size as other blocks by using mirroring as shown in FIG. 19. Here, mirroring means filling the un-existing portion of a block by luminance values of pixels that are the same values as those of pixels in the existing portion of that block.

Experimental Results

Experimental results of using the algorism proposed by the invention are shown in FIGS. 13 to 16. The block size of the ε filter used in the experiments is 5×5.

FIG. 13 shows an original JPEG2000 decoded image, FIG. 14 shows the image after applying ε filtering to the image of FIG. 13, FIG. 15 shows the image after applying the conventional algorism to the image of FIG. 13, and FIG. 16 shows the image after applying the algorism of the invention (the “new algorism”, hereafter) to the image of FIG. 13.

FIG. 18 shows the result of applying the new algorism of noise reduction to another image example shown in FIG. 17 that is a decoded image using the JPEG2000 compression coding scheme. Intensive mosquito noises are observed in the image in FIG. 17.

All images obtained by using the new algorism acquired better subjective ratings of picture quality as compared to images obtained by using the conventional algorism. Although the average used for calculation of standard deviations or variance coefficients in Step I is the arithmetic average, there are several other average such as geometric average or the harmonic average. However, the arithmetic average produced the best results in the experiments. The variance coefficient is used in calculating the image of FIG. 16.

As already explained, after calculating each luminance value variance for each pixel of interest and then composing the ε matrix, properly-defined coefficients for correcting ε values are required in the conventional algorism. In the new algorism, however, not only the procedure of calculating ε matrix becomes simpler, but also automatic correction of ε values becomes possible by using only the luminance information of pixels of interest in the image. This makes the calculation method simpler, reduces the total calculation volume and enables efficient program coding. Reduction of the total calculation volume means that the effect of calculation cost reduction becomes more significant when the image size becomes larger.

In terms of picture quality improvement, the new algorism can reduce the mosquito noise better than the conventional algorism while maintaining the texture of the image, as shown clearly by the experimental results.

The method and the apparatus of the invention for reducing the mosquito noise in a decoded compression-coded image with the bit map format thus enable to reduce efficiently the mosquito noise in an image by applying the automatically-variable ε filter, regardless the type of compression coding schemes used.

The method and the apparatus of the invention are especially effective in improving quality of compression coded images transmitted through a transmission medium that has a limited bandwidth. Actual applications such as moving image exchange between mobile telephone subscribers, video distribution to end users and application to built-in type video capture cards for personal users can be expected, since the algorism requires less calculation volume and is capable of frame-by-frame calculation for moving images coded by video coding schemes such as Motion-JPEG and Motion JPEG2000 coding schemes.

Claims

1. A method for reducing the mosquito noise in a decoded image that arises during decoding process by using a mosquito noise reducing algorism wherein: (1) the decoded compression-coded digital image is divided into blocks, (2) the standard deviation or the variance coefficient of luminance values is calculated for each block of said decoded compression-coded digital image, (3) an ε matrix for said compression-coded digital image is composed so that all components of each block of said ε matrix is filled with said standard deviations or said variance coefficients for each block, (4) an ε filter is composed so that its ε values are expressed by said ε matrix and (5) said decoded compression-coded digital image is applied to said ε filter to reduce the mosquito noise that arises during decoding process.

2. A method for reducing the mosquito noise in a decoded image that arises during decoding process claimed in claim 1 wherein said compression-coded digital image is a compressed image obtained by the JPEG or JPEG2000 compression coding scheme.

3. An apparatus for reducing the mosquito noise in a decoded image that arises during decoding process by using a mosquito noise reducing algorism wherein: (1) the decoded compression-coded digital image is divided into blocks, (2) the standard deviation or the variance coefficient of luminance values is calculated for each block of said decoded compression-coded digital image, (3) an ε matrix for said compression-coded digital image is composed so that all components of each block of said ε matrix is filled with said standard deviations or said variance coefficients for each block, (4) an ε filter is composed so that its ε values are expressed by said ε matrix and (5) said decoded compression-coded digital image is applied to said ε filter to reduce the mosquito noise that arises during decoding process.

4. An apparatus for reducing the mosquito noise in a decoded image that arises during decoding process claimed in claim 3 wherein said compression-coded digital image is a compressed image obtained by the JPEG or JPEG2000 compression coding scheme.