A pixel section outputs r, g and b signals which are obtained by photoelectrically converting light incident on r, g and b pixels. An adding section determines a prescribed area in which a certain pixel is set as a central pixel, and adds the r, g and b signals from the central pixel and peripheral pixels arranged on the periphery of the central pixel in the prescribed area in order to produce an addition signal. A ratio calculating section calculates an average value of each of the r, g and b signals, and a ratio coefficient of the average value of each of the r, g and b signals to a total value of the average values. An rgb generating section generates a new r signal, g signal and b signal by using the addition signal and the ratio coefficients calculated by the ratio calculating section.
|
1. A solid-state image pickup device comprising:
a pixel section in which an r pixel, a g pixel and a b pixel having photoelectric conversion elements provided with respective color filters of red (r), green (g) and blue (b) are arranged two-dimensionally in a matrix form, the pixel section outputting an r signal, a g signal and a b signal which are obtained by photoelectrically converting light incident on the r, g and b pixels;
an adding section which determines a prescribed area with respect to a certain pixel set as a central pixel, and which adds the r, g and b signals from the central pixel and peripheral pixels arranged on the periphery of the central pixel in the prescribed area to produce an addition signal;
a ratio calculating section which calculates an average value of each of the r, g and b signals, and a ratio coefficient of the average value of each of the r, g and b signals to a total value of the average values; and
an rgb generating section which multiplies the addition signal by the ratio coefficients to generate a new r signal, a new g signal and a new b signal, respectively, which correspond to the central pixel and replace the r signal, g signal and the b signal which are obtained by photoelectrically converting light incident on the r, g and b pixels.
2. The solid-state image pickup device according to
the adding section adds up signals from four pixels of a 2×2 pixel arrangement, from nine pixels of a 3×3 pixel arrangement, or from twenty-five pixels of a 5×5 pixel arrangement, the four pixels, the nine pixels or the twenty-five pixels being arranged in a matrix form.
3. The solid-state image pickup device according to
4. The solid-state image pickup device according to
5. The solid-state image pickup device according to
6. The solid-state image pickup device according to
the edge detecting section divides the prescribed area into four blocks each of which is constituted of a plurality pixels including the central pixel, and is provided with a difference determining section which compares a difference between a signal including the central pixel and a signal of the other pixels of the same color and a predetermined value with each other in order to determine which of the difference and the predetermined value is larger/smaller than the other.
7. The solid-state image pickup device according to
8. The solid-state image pickup device according to
calculates the average value of each of the r signal, the g signal and the b signal from said each 5×5 pixel arrangement arranged on the periphery of the central pixel.
9. The solid-state image pickup device according to
in the filter operation, a signal of a central pixel of the 3×3 pixel arrangement is multiplied by four, signals of pixels arranged in the row direction and in the column direction with respect to the central pixel are multiplied by two, signals of pixels arranged at corners are multiplied by one, and a signal obtained by totaling these signals is divided by four.
10. The solid-state image pickup device according to
11. The solid-state image pickup device according to
12. The solid-state image pickup device according to
13. The solid-state image pickup device according to
14. The solid-state image pickup device according to
|
This application is based upon and claims the benefit of priority from prior Japanese Patent Application No. 2007-016893, filed Jan. 26, 2007, the entire contents of which are incorporated herein by reference.
1. Field of the Invention
The present invention relates to a solid-state image pickup device such as a charge-coupled device (CCD) image sensor and a CMOS image sensor, and the solid-state image pickup device is used in, for example, a mobile-phone having an image sensor, a digital camera, and a video camera.
2. Description of the Related Art
As for the image sensor, in recent years, miniaturization of pixels has advanced, pixels of a size from 2.0 to 2.9 μm have already been put into practical use, and development of 1.7- and 1.4-μm pixels is now underway. In a minute pixel having a size of 2 μm or less, an amount of incident light largely decreases, and hence the signal-to-noise ratio is decreased. Further, heretofore, in a color camera, there has been a problem that an image quality is deteriorated due to a color artifact or color noise. Regarding suppression of a color artifact and noise reduction, various methods have been proposed (see, for example, Jpn. Pat. Appln. KOKAI Publication No. 4-235472, Jpn. Pat. Appln. KOKAI Publication No. 2002-10108, Jpn. Pat. Appln. KOKAI Publication No. 2005-303731, Jpn. Pat. Appln. KOKAI Publication No. 2001-245307, and Jpn. Pat. Appln. KOKAI Publication No. 5-168029). However, radical measures are not proposed.
According to an aspect of the present invention, there is provided a solid-state image pickup device comprising: a pixel section in which an R pixel, G pixel and B pixel each having a photoelectric conversion element provided with a color filter of red (R), green (G) or blue (B) are arranged two-dimensionally in a matrix form, the pixel section outputting an R signal, G signal and B signal which are obtained by photoelectrically converting light incident on the R, G and B pixels; an adding section which determines a prescribed area in which a certain pixel is set as a central pixel, and which adds the R, G and B signals from the central pixel and peripheral pixels arranged on the periphery of the central pixel in the prescribed area to produce an addition signal; a ratio calculating section which calculates an average value of each of the R, G and B signals, and a ratio coefficient of the average value of each of the R, G and B signals to a total value of the average values; and an RGB generating section which generates a new R signal, G signal and B signal by using the addition signal and the ratio coefficients calculated by the ratio calculating section.
A solid-state image pickup device of the embodiment of the present invention will be described below with reference to the accompanying drawings. Here, a CMOS image sensor is exemplified as a solid-state image pickup device. In the description, common parts are denoted by common reference symbols throughout all the drawings.
First, a solid-state image pickup device including a CMOS image sensor of a first embodiment of the present invention will be described below.
A pixel section 111 and a column-type analog-to-digital converter (ADC) 112 are arranged in the sensor section 11. Pixels (cells) are arranged in a two-dimensional form of rows and columns in the pixel section 111 on a semiconductor substrate. Each pixel is constituted of a photoelectric converting means (for example, a photodiode) and a color filter, and color filters of three colors red (R), green (G), and blue (B) are arranged on the photodiodes. The color filter arrangement is the Bayer arrangement of the RGB three primary colors.
In the sensor section 11, a light signal condensed by a lens 18 is separated into RGB light signals by color filters of RGB three colors, and the RGB light signals are converted into signal charge by a photodiode array arranged in a two-dimensional form by photoelectric conversion. The signal charge is converted into digital signals (R, G and B signals) by the column-type analog-to-digital converter (ADC) 112. The converted digital signals are output to the line memory 12, and the digital signals for five vertical lines are stored in memories 1 to 5 in the line memory 12. The digital signals stored in the memories 1 to 5 are respectively input in parallel to the color correlation RGB generating circuit 13.
In the color correlation RGB generating circuit 13, the R, G and B signals input thereto from the line memory 12 are added up by an adding section 131 and an addition signal S is generated. Further, a ratio calculating section 132 calculates the respective ratio coefficients of the average values Rave, Gave and Bave of the R, G and B signals to the total value Save of the average values. Further, an RGB generating section 133 newly generates signals Rs, Gs and Bs from the addition signal S and the calculated ratio coefficients as signals of the same positions as the pixel arrangement. This processing serves as a replacement for the conventional color separation interpolation circuit.
Thereafter, the signals Rs, Gs and Bs processed by the RGB generating section 133 are input to the signal processing circuit 14 in the subsequent stage. The signals input to the signal processing circuit 14 are processed by a white balance circuit, contour emphasizing circuit, gamma correction circuit, RGB matrix circuit, and the like, in order to be turned into YUV signals and RGB signals and output as digital signals DOUT0 to DOUT7. Further, operations of the above-mentioned sensor section 11, line memory 12, color correlation RGB generating circuit 13, and signal processing circuit 14 are performed on the basis of a clock signal output from the system timing generating circuit (SG) 15. Further, a command can be controlled by data DATA input from the outside. The data DATA is input to the command decoder 16 through the serial interface 17, and a decoded signal is input to each circuit.
Next, a processing method of the color correlation RGB generating circuit 13 in the first embodiment will be described below.
Further, in the 5×5 pixel signals constituted of the RGB Bayer arrangement shown in
Rs=S0*(Rave/Save)
Gs=S0*(Gave/Save)
Bs=S0*(Bave/Save)
Here, by generating the addition signal S0, random noise is reduced from the addition signal S0. Further, by generating the signals Rs, Gs and Bs from the addition signal S0 as the pixel signals of the same position, the false color due to the conventional edge can be suppressed. Further, although color noise has been caused by single random noise of the R, G and B signals, by generating the Rs, Gs and Bs signals from the addition signal S0, random noise components of the Rs, Gs and Bs signals become the same, and hence color noise does not occur. That is, only brightness noise occurs (noise is not colored).
Another processing method employed in the color correlation RGB generating circuit 13 will now be described.
Subsequently, in the 5×5 pixel signals constituted of the RGB Bayer arrangement shown in
Rs=S0*(Rave/Save)
Gs=S0*(Gave/Save)
Bs=S0*(Bave/Save)
As described above, the addition signal S0 reduces the random noise of the addition signal S0. Conventional false color due to edges can be suppressed by generating, from the addition signal S0, signals Rs, Gs and Bs as same-position pixel signals. Furthermore, since signals Rs, Gs and Bs are generated from the addition signal S0, they have the same random noise component, and hence contain no color noise. Namely, brightness noise does not occur. In the filter operation, since the ratio of the level of the G signal, which is greater in level than each of the R and B signals, to the level of each of the R and B signals is set to 2, an addition signal S0 improved in SNR by about 3 dB can be generated.
As described above, according to the first embodiment, by generating addition signals obtained by adding up pixel signals, random noise can be reduced and the signal-to-noise ratio can be increased. Further, by newly and simultaneously generating the R, G and B signals from the addition signal, a false color suppressing circuit for suppressing a false color resulting from demosaicking is made unnecessary, and an adverse influence of random noise resulting from color shifts of the R, G and B signals and an RGB single pixel can be eliminated. Hence, color noise of a single color can be suppressed. Further, even when the R, G and B signals are newly generated by a color matrix operation for improving color reproducibility, the signals RS, Gs and Bs are generated from one addition signal, and hence random noise components included in the respective signals are in phase with each other. Accordingly, even when a signal is subjected to subtraction processing in the color matrix operation, noise is not increased.
Next, a solid-state image pickup device including a CMOS image sensor of a second embodiment of the present invention will be described below. The second embodiment is that to which a configuration for improving resolution of an edge of an image is added. The other configuration and advantage are similar to those of the first embodiment, the same parts as those of the first embodiment are denoted by the same reference symbols, and a description of them are omitted.
Then, as shown in
Further, in the edge detecting circuit 134, when the central pixel of the 5×5 pixel arrangement is a B signal, the B signal of the central pixel is made to be B0, and the 5×5 pixel signals are separated into four blocks AZ, BZ, CZ and DZ, in each of which the B0 signal of the central pixel is located at a corner as shown in
Further, in the edge detecting circuit 134, when the central pixel of the 5×5 pixel arrangement is a Gr signal, the 5×5 pixel signals are separated into four blocks AZ, BZ, CZ and DZ, in each of which the Gr signal of the central pixel is located at a corner as shown in
First, a two-pixel difference determining method will be described below. As shown in
Then, a three-pixel stripe difference determining method will be described below. As shown in
Besides, in a three-pixel L-shaped difference determining method, two groups each constituted of three pixels arranged in an L-shape which are symmetrical with respect to the D5 pixel as a center are selected as shown in
Then, in accordance with a result of the edge determination for each of the four blocks, a block selected by a block selecting section 135 shown in
Rs=S0*(Rave/Save)
Gs=S0*(Gave/Save)
Bs=S0*(Bave/Save)
Likewise, as for the upper right, lower left, and lower right blocks BZ, CZ and DZ, calculation can also be performed. In the determination results shown in
Rs=S*(Rave/Save)
Gs=S*(Gave/Save)
Bs=S*(Bave/Save)
Likewise, as for a case of a block of some other L-shape, calculation can also be performed.
Rs=S*(Rave/Save)
Gs=S*(Gave/Save)
Bs=S*(Bave/Save)
Likewise, as for a case of the other two blocks, calculation can also be performed.
Rs=S*(Rave/Save)
Gs=S*(Gave/Save)
Bs=S*(Bave/Save)
Likewise, as for a case of the other two blocks which are obliquely arranged in the opposite way, calculation can also be performed.
Rs=S*(Rave/Save)
Gs=S*(Gave/Save)
Bs=S*(Bave/Save)
The processing method shown in
In the above-mentioned embodiments, although the description has been given on the basis of the 5×5 pixel arrangement, if the pixel arrangement is changed to a 7×7 pixel arrangement, and production of the addition signal in the block, and calculation of ratio coefficients are performed by using 4×4 pixels, a further higher signal-to-noise ratio and a higher image quality can be realized.
Further, when the number of blocks to be selected is two to four, and when the number of blocks to be selected is zero, the processing method is changed according to the number of blocks to be selected. However, in order to reduce the number of circuits, the respective signals Rs, Gs and Bs that can be obtained from the processing for one block shown in
Further, as a factor in the deterioration of the signal-to-noise ratio of a color camera, there is the RGB matrix circuit. This circuit performs an RGB matrix operation to improve the RGB color reproducibility. An example of the operational expression used at this time is shown as the formula (1).
In this processing, the other two colors are subtracted from the own color. That is, by reducing an amount of the other two colors mixed into the own color, the purity of the own color is increased and the color reproducibility is improved. Factors in the color mixture are spectral characteristics of the color filter itself, optical crosstalk occurring at stages up to the photodiode of the image sensor, diffusion of a signal in the silicon substrate, and the like. As a result of the subtraction processing, the noise amount has been increased because the noise in the R, G and B signals has been randomly caused in the prior art technique. On the other hand, according to this system of the present invention, each of the random noise components of the Rs, Gs and Bs signals is of the same component, and hence a random noise reduction effect can be obtained by the subtraction processing. For example, when the R signal is produced, if the Rs signal is increased by the random noise, the signals Gs and Bs are also increased. In the matrix operation, the R signal subtracts Gs and Bs components from the Rs signal, and hence a larger amount of the signal corresponding to the increased amount of the noise is subtracted. Thus, the R signal subtracts larger amounts of the signals from the Rs signal. Conversely, if the Rs signal is decreased by the random noise, the signals Gs and Bs are also decreased. In the matrix operation, the R signal subtracts Gs and Bs components from the Rs signal, and hence a smaller amount of the signal corresponding to the decreased amount of the noise is subtracted. Accordingly, the reduction in the signal amount is small in the R signal. As a result of this, the R signal has an effect of reducing the random noise. Likewise, the random noise of each of the G and B signals is also reduced.
Further, as a problem of the color camera to be solved, there is a problem that a false color is attached to an edge of an image due to the chromatic aberration of the optical lens. This chromatic aberration occurs due to a difference in the refractive index of the RGB light caused by the optical lens. By performing a determination based on the difference signal of only the G signal as the block determination in the embodiments of the present invention, it is possible to make the R and B signals at an edge negligible, and suppress an artifact at an edge by producing Rs, Gs and Bs signals by the calculation of the ratio coefficients of the peripheral pixels.
Furthermore, as a problem of the color camera, moire which causes coloring at a low frequency when a fine pattern of a high frequency is imaged occurs. The cause of this phenomenon is that the pixel pitch of the RGB pixels and the signal pitch of the subject are out of phase, and hence the amount of the signal incident on the pixel varies, and a beat-like false color signal is generated. However, in the formation of addition signal in the embodiments of the present invention, high-frequency components are reduced, and the Rs, Gs and Bs signals are produced from the addition signal, and hence a signal that enhances only the R and B signals (false color) is not generated, and the Rs, Gs and Bs signals vary simultaneously, whereby only the brightness signal is varied. Thus, the false coloring caused by moire that has been the problem of the prior art is suppressed.
Further, in the embodiments of the present invention, even when an image sensor having a different color filter arrangement is used, after producing the R, G and B signals, new Rs, Gs and Bs signals are produced on the basis of formation of the similar addition signal and calculation of the ratio coefficients of the R, G and B signals, whereby the same effect can be obtained. Further, the embodiments of the present invention are not limited to the CMOS image sensor, and other CCD image sensors and stacked image sensors formed of stacked photoelectric transfer films can also be applied. Further, a dedicated image signal processor (ISP) can be used for processing.
An outline of the color correlation noise reduction method used in the embodiments of the present invention will be described below.
Further, the threshold level LevN setting method used in the threshold level setting circuit 19 in the embodiment described previously will be described below.
Y=0.59G+0.3R+0.11B
Thus, if the threshold level LevN is set large when noise suppression processing of R and B having a small contribution effect on the brightness signal is performed, the random noise suppression effect for R and B can be enhanced.
Further, it is more effective to set the threshold level LevN according to each of the RGB signal amounts and in accordance with the gain ratio of the white balance. Further, when shading correction is performed for some reason of the optical characteristic of a lens, the farther the area on the screen is from the center of the screen, i.e., the closer the area on the screen is to the uppermost, lowermost, rightmost, leftmost positions, and the corners, the higher the digital gain for amplifying the signal is made. For this reason, the random noise is increased at the upper, lower, right and left parts, and at the corners on the screen. Thus, if the threshold level LevN is made large at the upper, lower, right and left parts, and at the corners of the screen in accordance with the gain, the noise suppression effect can be enhanced, and the image quality can be increased. As described above, by appropriately changing the threshold level LevN in accordance with the signal amount, screen position, color information, gain information, and the like, a higher image quality can be realized.
As has been described above, according to the embodiments of the present invention, it is possible to provide a solid-state image pickup device capable of suppressing the color artifact of the R, G and B signals, and reducing noise. More specifically, in the embodiments of the present invention, the addition signal is produced by adding the R, G and B signals. Further, a color ratio is calculated from the color information on the peripheral pixels of the addition pixel, and R, G and B signals are newly produced from the addition signal, whereby demosaicking processing can be executed simultaneously. The addition signal is produced by adding up the pixel signals as described above, whereby the signal-to-noise ratio can be increased. Further, by simultaneously and newly producing the R, G and B signals from the addition signal, it is possible to provide an image pickup device of a high image quality in which color shift and color noise of the R, G and B signals are suppressed.
According to the embodiments of the present invention, it is possible to provide a solid-state image pickup device capable of suppressing the color artifact of the R, G and B signals, and reducing noise.
Furthermore, the embodiments described previously can not only be implemented singly, but also can be implemented by appropriately combining with each other. Moreover, inventions of various stages are included in the embodiments described previously, and by appropriately combining a plurality of constituent elements disclosed in the embodiments with each other, inventions of various stages can be extracted.
Additional advantages and modifications will readily occur to those skilled in the art. Therefore, the invention in its broader aspects is not limited to the specific details and representative embodiments shown and described herein. Accordingly, various modifications may be made without departing from the spirit or scope of the general inventive concept as defined by the appended claims and their equivalents.
Patent | Priority | Assignee | Title |
8712161, | Jan 12 2010 | CLOUD NETWORK TECHNOLOGY SINGAPORE PTE LTD | Image manipulating system and method |
8792746, | Mar 11 2011 | Sony Corporation | Image processing apparatus, image processing method, and program |
9179083, | Oct 24 2013 | Ricoh Company, Limited | Photoelectric conversion element, image reading device, image forming apparatus, image reading method, and image forming method |
Patent | Priority | Assignee | Title |
6091862, | Nov 26 1996 | Minolta Co., Ltd. | Pixel interpolation device and pixel interpolation method |
6856708, | Mar 04 1999 | Ricoh Co., Limited | Method and system for composing universally focused image from multiple images |
6940556, | Apr 16 1998 | Nikon Corporation | Electronic still camera and information recording appartus |
7196724, | Dec 08 2000 | Nikon Corporation | Image signal processing device, digital camera and computer program product that perform white balance adjustment using pixel outputs from extracted areas |
20030016205, | |||
20050134713, | |||
20050200733, | |||
20060114526, | |||
20080143844, | |||
JP11220751, | |||
JP2001245307, | |||
JP200210108, | |||
JP2003244715, | |||
JP2005303731, | |||
JP4235472, | |||
JP5168029, |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Jan 24 2008 | Kabushiki Kaisha Toshiba | (assignment on the face of the patent) | / | |||
Feb 07 2008 | EGAWA, YOSHITAKA | Kabushiki Kaisha Toshiba | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 020681 | /0194 |
Date | Maintenance Fee Events |
Mar 04 2016 | REM: Maintenance Fee Reminder Mailed. |
Jul 24 2016 | EXP: Patent Expired for Failure to Pay Maintenance Fees. |
Date | Maintenance Schedule |
Jul 24 2015 | 4 years fee payment window open |
Jan 24 2016 | 6 months grace period start (w surcharge) |
Jul 24 2016 | patent expiry (for year 4) |
Jul 24 2018 | 2 years to revive unintentionally abandoned end. (for year 4) |
Jul 24 2019 | 8 years fee payment window open |
Jan 24 2020 | 6 months grace period start (w surcharge) |
Jul 24 2020 | patent expiry (for year 8) |
Jul 24 2022 | 2 years to revive unintentionally abandoned end. (for year 8) |
Jul 24 2023 | 12 years fee payment window open |
Jan 24 2024 | 6 months grace period start (w surcharge) |
Jul 24 2024 | patent expiry (for year 12) |
Jul 24 2026 | 2 years to revive unintentionally abandoned end. (for year 12) |