A method, apparatus, and system that use a white balance operation. A selecting process is applied to each pixel selected and considered for automatic white balance statistics to determine the distance from the selected pixel to a white curve defined in a white area corresponding to an image sensor.
|
1. An imaging device, comprising:
a pixel array for capturing an image; and
an image processing circuit for performing a white balancing operation; the image processing circuit comprising:
a first circuit portion for:
calculating first respective distances between a node from a selected pixel from the captured image and each node on a white curve defining a predetermined white area of the image sensor, where the nodes are plotted in a two-dimension space;
selecting two of the white curve nodes closest to the selected pixel node;
dividing an interval between the two closest white curve nodes into multiple intervals having a set of sub-nodes;
calculating second respective distances between the selected pixel node and each sub-node;
selecting a distance determined to be shortest distance among the second respective distances; and
saving pixel data for a selected pixel if said selected distance is less than a predetermined value; and
a second circuit portion for performing an automatic white balance operation using the saved pixel data.
5. A camera, comprising:
a lens;
an image sensor for receiving an image from the lens and outputting an image signal that includes pixel data for each pixel of the image;
a memory for storing a white curve associated with said image sensor, the white curves being defined by multiple node coordinates in a coordinate color space; and
an image processor for processing the image signal, the image processor being operable on each of at least some pixels of the image, to:
select the pixel from the image, the pixel having a pixel node in the color space;
obtain coordinates for the stored white curve;
calculate first respective distances from the selected pixel node to each white curve node;
select two of the white curve nodes determined to be closest to the selected pixel node;
divide an interval between the two white curve nodes into four equal sub-intervals, the sub-intervals being defined in the color space by sub-nodes;
define the sub-intervals by a set of sub-nodes;
calculate second respective distances from the selected pixel node to each sub-node; and
save a pixel which has a shortest distance among the second respective distances, which is less than a threshold defining a boundary of a white area in the color space.
4. An imaging system, comprising:
a processor; and
an imaging device coupled to the processor, the imaging device comprising:
an image sensor for receiving an image and outputting an image signal that includes pixel data for each pixel of the image; and
an image processor in communication with the image sensor for performing a white balancing operation on the image, the image processor comprising:
means for defining a white curve as a piece-wise linear curve in a two-dimension space, wherein the white curve is located in the white area, and the piece-wise linear curve is defined by a set of white curve nodes;
means for defining the pixel by a pixel node in the two-dimension space;
means for calculating first respective distances between the selected pixel node and each white curve node;
means for selecting two white curve nodes determined to be closest to the selected pixel node;
means for defining an interval between the selected two white curve nodes by multiple sub-nodes;
means for calculating second respective distances between the selected pixel node and each sub-node;
means for selecting a shortest distance among the second respective distances;
means for determining if the shortest distance is less than a predefined threshold distance;
means for saving only pixels having a shortest distance which is less than the predefined distance to obtain automatic white balance statistics; and
means for performing white balance on the image by applying the obtained automatic white balance statistics.
2. The imaging device of
3. The imaging device of
6. The camera of
|
Embodiments of the invention relate generally to selecting pixels for automatic white balance processing.
One of the most challenging problems in color image processing is adjusting the color gains of a system to compensate for variations in illumination spectra incident on an image sensor. This process is typically known as white balance processing. The human eye and brain are capable of “white balancing.” If a human observer takes a white card and exposes it to different kinds of illumination, it will look white even though the white card is reflecting different colors of the spectrum. If a person takes a white card outside, it looks white to the person's eye. If a person takes a white card inside and views it under fluorescent lights, it still looks white. When viewed under an incandescent light bulb, the card still looks white to the human eye. Moreover, even when placed under a yellow light bulb, within a few minutes, the card will look white. With each of these light sources, the white card is reflecting a different color spectrum, but the human brain is smart enough to know that the card looks white.
Obtaining the same result with a camera or other image-capturing device having an image sensor is much harder. When the white card moves from light source to light source, an image sensor “sees” different colors under the different lights. Consequently, when an image-capturing device, e.g., a digital camera, is moved from outdoors (sunlight) to indoor fluorescent or incandescent light conditions, the color in the image shifts. If the white card looks white when indoors, for example, it might look bluish outside. Alternatively, if the card looks white under fluorescent light, it might look yellowish under an incandescent lamp.
The white balance problem stems from the fact that spectral emission curves of common sources of illumination are significantly different from each other. For example, in accordance with Plank's law, the spectral energy curve of the sun is shifted towards the shorter wavelengths relative to the spectral energy curve of an incandescent light source. Therefore, the sun can be considered to be a “blue-rich” illuminator while an incandescent bulb can be considered to be a “red-rich” illuminator. As a result, if the image processing settings are not adjusted, scenes illuminated by sunlight produce “bluish” imagery, while scenes illuminated by an incandescent source appear “reddish”.
After a digital camera (i.e., an image-capturing device) captures an image, the circuitry within the camera performs image processing to compensate for changes in illumination spectra. To compensate for changes in illumination spectra, the gains of the color channels of, e.g. R,G,B, of image processing systems and/or image sensors are adjusted. This adjustment is usually performed by the image processing systems to preserve the overall luminance (brightness) of the image. As a result of proper adjustment, gray/white areas of the image appear gray/white on the image-capturing device (hence the term “white balance”).
In the absence of specific knowledge of the spectra of the illumination source, this adjustment can be performed based on automatic white balance (AWB) statistics. Automatic white balance statistics are based on a statistical analysis of the pixels in the image itself to obtain information about the luminance of colors in the image. The statistical analysis selects a sample of pixels in the image by applying one or more criteria. The values of the pixels that meet the criteria are then used to obtain the color balance statistical information. The image-capturing device can initiate a white balance operation and perform color correction on the image based on the automatic white balance statistics. That is, the collected statistics are compared to expected values and the results of the comparison are used to correct the white balance in the image.
For obtaining automatic white balance statistics, pixels must be selected. One approach to selecting pixels is a white point estimation method. White point estimation can be determined by applying a known gray world model. The gray world model is premised on having the entire image balancing out to gray, i.e., the average color of the image balances out to gray, where gray comprises equivalent amounts of red, green, and blue components. In applying the gray world model for white point estimation, the white point chromaticity corresponds to the average image chromaticity. Since gray is a neutral tone, any variations from the neutral tone in the illumination spectra would be adjusted accordingly.
Several selecting criteria for the selecting pixels are used to obtain automatic white balance statistics. For white point estimation, one selecting criterion requires that only pixels within a white area of the image are selected for the automatic white balance statistics. One method of applying the selecting criterion includes determining a white area of the image sensor, which can be specified during manufacturing by calibrating white curves within the white area. One approach to calibrate a white curve is to take pictures of GretagMacBeth Color Rendition Chart (or similar chart) at different light sources (i.e., different color temperatures) and plot coordinates for gray patches (i.e., applying the gray world model). Since it is known what the colors are supposed to look like (from the chart), raw color data is determined for the image sensor. By applying the gray world model, the coordinates for the gray patches identify a white area for the image sensor; accordingly, a white curve within the white area can be specified for the image sensor.
Additionally, a threshold distance, shown in
A white curve can also be defined by other coordinates, such as, B/G vs. R/G; Log 2(B/G) vs. Log 2(R/B); Y vs. X; (R−B)/Y vs. (R+B−2G)/Y. Each pixel considered for automatic white balance statistics has to be tested to determine if it is within the white area of the image.
A method and system for selecting pixels for automatic white balancing that do not require a large amount of processing resources for selecting are desirable.
In the following detailed description, reference is made to various specific embodiments in which the invention may be practiced. These embodiments are described with sufficient detail to enable those skilled in the art to practice them. It is to be understood that other embodiments may be employed, and that structural and logical changes may be made.
The term “pixel” refers to a picture element in an image. Digital data defining an image may, for example, include one or more color values for each pixel. For a color image, each pixel's values may include a value for each of a plurality of colors, such as red, green, and blue.
The image sensor 404 is a sensor that may be any type of solid state images including CMOS, CCD and others. The image sensor 404 receives image information in the form of photons, and converts that information to pixel analog electrical signals, which are subsequently provided to down stream processing circuits. In the imaging apparatus 400, the image sensor 404 provides electrical signals to the image processing circuit 408.
The image processing circuit 408 performs image processing on the digital signals received from analog-to-digital (A/D) converter 406. The image processing circuit can be implemented using logic circuits in hardware, or using a programmed processor, or by a combination of both. The image processing circuit 408 may include other circuits that perform pixel defect correction, demosaicing, image resizing, aperture correction, and correction for other effects or defects.
In an embodiment, the image processing circuit 408 outputs are pixels having RGB channels representational of the image data from red, blue and green pixels of image sensor 404. The image processing circuit 408 also converts the RGB image signals into a second set of data structures representing image pixel data in YUV format that are also representational of the RGB data. YUV stores image data in Y—luminance (“luma”), U—blue chrominance (“blue chroma” or “Cb”) and V—red chrominance (“red chroma” or “Cr”).
The auto-white balance circuit 414 receives image data, in the form of the YUV data structures and computes the correction values, if required, to perform white balancing. The auto-white balance circuit 414 provides the results of its computation to the image processing circuit 408. The white balance computation in the auto-white balance circuit 414 is performed using the YUV data structure because of the inherent properties of the YUV data structure. The YUV data structure breaks down an image into the Y, the luminance values, and UV, which is essentially a two-dimensional representation of color, where the two color components are U and V (i.e., Cb and Cr). Any color can be expressed in terms of the two color components, which can be plotted in an x and y coordinate space.
The image processing circuit 408 receives information from the auto-white balance circuit 414 and provides gain information to the component of the YUV data structure and thus makes appropriate adjustments to an image.
At step 104, a first pixel considered for automatic white balance statistics is selected from the captured image. Typically, the pixels considered for automatic white balance statistics are selected from the captured image in same order, row by row and pixel by pixel within a row. Next, at step 112, it is determined if the pixel is in the white area (described below in more detail). The acceptance of the pixel for automatic white balance statistics, depicted as a “Yes” response in Step 112, depends on the selecting criterion (described below in more detail) used at step 112.
At step 120, the data for a selected pixel that meets the selecting criterion is saved, and the method 100 continues at step 124. On the other hand, if the selected pixel does not meet the selecting criterion at step 112, then the method 100 continues at step 124 without saving the pixel data. At step 124, a determination is made to decide if the selected pixel is the last pixel to be tested. If it is determined that the pixel is not the last pixel, the method 100 continues at step 128. Otherwise, the method 100 continues at step 136.
Steps 112-128 are repeated until it is determined at step 124 that the last pixel from the captured image has been selected. Once it is determined at step 124 that a selected pixel is the last pixel considered for automatic white balance statistics; at step 136, saved pixel data is used to obtain the automatic white balance statistics. At step 140, the automatic white balance statistics are then used by the auto-white balance circuit 408 to perform a white balancing operation.
If it is determined at step 124 that the selected pixel is not the last pixel to be considered for automatic white balance statistics, then at step 128, the next pixel is selected, and the method 100 continues at step 112.
Determining if a pixel meets the selecting criterion used at step 112 includes estimating the distance from the selected pixel to a white curve, e.g.,
Once the first pixel has been selected at step 102 of
Referring to
Dis1=DIST(P0,P1);
Dis2=DIST(P0,P2);
Dis3=DIST(P0,P3); and
Dis4=DIST(P0,P4);
where DIST(A, B) estimates the distance between node A and B. If node A has coordinate [Xa,Ya] and node B has coordinate [Xb,Yb], the equation defining the distance between these two nodes is DIST(A, B)=√{square root over ((Xa−Xb)2+(Ya−Yb)2)}{square root over ((Xa−Xb)2+(Ya−Yb)2)}. Implementation of this equation might be difficult and costly; alternatively, the following estimation can be easily implemented. The following estimation provides an acceptable, less than 10% error, in distance detection.
Applying the estimation to the illustrated example in
(1) Performing an initial determination:
dX=|X P0−X P1|; dY=|Y P0−Y P1|,
D=dX+dY;
(2) Calculating the Min and Max:
IF (dX > dY) then
Min = dY;
Max1 = dX;
Max2 = dX / 2;
Max4 = dX / 4;
Max8 = dX / 8;
ELSE
Min = dX;
Max1 = dY;
Max2 = dY / 2;
Max4 = dY / 4;
Max8 = dY / 8; and
(3) Setting the distance based on these relationships:
IF (Min <= Max8)
DIST = D;
IF (Max8 < Min <= Max4)
DIST = D/2 + D/4 + D/8;
IF (Min > Max4)
DIST = D/2 + D/4;
The DIST(A, B) procedure discussed above is repeated to calculate the distance between selected pixel node P0, and each white curve node P1, P2, P3, P4. Accordingly, DIST(A, B) procedure continues, where Dis2=DIST(P0, P2); Dis3=DIST(P0, P3); and Dis4=DIST(P0, P4).
At step 208, the two nodes on the white curve closest to the selected pixel node P0 defines an interval on the white curve, which is selected for further evaluation. In the example shown in
X5=X1+deltaX/4;
X6=X1+deltaX/2;
X7=X1+deltaX/2+deltaX/4;
Y5=Y2+deltaY/2+deltaY/4;
Y6=Y2+deltaY/2; and
Y7=Y2+deltaY/4,
where deltaX=X2−X1 and deltaY=Y1−Y2.
Next at step 212, the distance between the selected pixel node P0 and each sub-node P1, P5, P6, P7, and P2 is calculated.
d1=DIST(P0,P1);
d5,=DIST(P0,P5);
d6,=DIST(P0,P6);
d7=DIST(P0,P7); and
d2=DIST(P0,P2),
where the procedure DIST(A,B) calculates the distances d1, d5, d6, d7 and d2, applying the same estimation, as discussed above.
It should be appreciated to one skilled in the art, that steps 208 through 212 can be repeatedly performed to further narrow the intervals on the white curve and define the sub-node closest to the pixel; thereby determining the shortest distance possible from the pixel to the white curve.
At step 214, the shortest distance among the distances d1, d5, d6, d7 and d2 is selected for the next evaluation. At step 216, the threshold distance TH, shown in
The selecting process described above is independent of the type of coordinates used to define the white curve. Furthermore, the selecting process does not require a large amount of resources for implementation due to the estimations used.
System 700, for example, a camera system includes a lens 402 for focusing an image on a pixel array of the image sensor 404, central processing unit (CPU) 705, such as a microprocessor, which controls camera operation, and which communicates with one or more input/output (I/O) devices 710 over a bus 715. Imaging apparatus 400 also communicates with the CPU 705 over bus 715. The processor system 700 also includes random access memory (RAM) 720, and can include removable memory 725, such as flash memory, which also communicate with CPU 705 over the bus 715. Imaging apparatus 400 may be combined with the CPU, with or without memory storage on a single integrated circuit or on a different chip than the CPU.
The above description and drawings illustrate embodiments of the invention. Although certain advantages and embodiments have been described above, those skilled in the art will recognize that substitutions, additions, deletions, modifications and/or other changes may be made. Accordingly, the embodiments are not limited by the foregoing description but are only limited by the appended claims.
Patent | Priority | Assignee | Title |
8031236, | Nov 21 2006 | Samsung Electronics Co., Ltd. | White detecting method and apparatus using the same |
8885062, | Dec 10 2010 | Kabushiki Kaisha Toshiba | Auto white balance adjustment system, auto white balance adjustment method, and camera module |
Patent | Priority | Assignee | Title |
6023264, | Apr 24 1998 | Adobe Systems Incorporated | Method to estimate the white point on a display device |
6670987, | Jul 24 1997 | Sony Corporation | Chrominance signal processing circuit, chrominance signal processing method and camera using them |
7177467, | Nov 12 2002 | Novatek Microelectronics Corp. | Method for estimating white point shift for auto white balance |
7453500, | Aug 09 2001 | Canon Kabushiki Kaisha | Image sensing apparatus, white balance adjusting method, and operation processing program therefor |
7593043, | May 18 2005 | Monument Peak Ventures, LLC | Image processing device and white balance adjustment device |
20040090536, | |||
20060092171, |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Aug 21 2006 | SUBBOTIN, IGOR | Micron Technology, Inc | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 018215 | /0224 | |
Aug 24 2006 | Micron Technology, Inc. | (assignment on the face of the patent) | / | |||
Apr 26 2016 | Micron Technology, Inc | MORGAN STANLEY SENIOR FUNDING, INC , AS COLLATERAL AGENT | PATENT SECURITY AGREEMENT | 038954 | /0001 | |
Apr 26 2016 | Micron Technology, Inc | U S BANK NATIONAL ASSOCIATION, AS COLLATERAL AGENT | CORRECTIVE ASSIGNMENT TO CORRECT THE REPLACE ERRONEOUSLY FILED PATENT #7358718 WITH THE CORRECT PATENT #7358178 PREVIOUSLY RECORDED ON REEL 038669 FRAME 0001 ASSIGNOR S HEREBY CONFIRMS THE SECURITY INTEREST | 043079 | /0001 | |
Apr 26 2016 | Micron Technology, Inc | U S BANK NATIONAL ASSOCIATION, AS COLLATERAL AGENT | SECURITY INTEREST SEE DOCUMENT FOR DETAILS | 038669 | /0001 | |
Jun 29 2018 | U S BANK NATIONAL ASSOCIATION, AS COLLATERAL AGENT | Micron Technology, Inc | RELEASE BY SECURED PARTY SEE DOCUMENT FOR DETAILS | 047243 | /0001 | |
Jul 03 2018 | MICRON SEMICONDUCTOR PRODUCTS, INC | JPMORGAN CHASE BANK, N A , AS COLLATERAL AGENT | SECURITY INTEREST SEE DOCUMENT FOR DETAILS | 047540 | /0001 | |
Jul 03 2018 | Micron Technology, Inc | JPMORGAN CHASE BANK, N A , AS COLLATERAL AGENT | SECURITY INTEREST SEE DOCUMENT FOR DETAILS | 047540 | /0001 | |
Jul 31 2019 | JPMORGAN CHASE BANK, N A , AS COLLATERAL AGENT | Micron Technology, Inc | RELEASE BY SECURED PARTY SEE DOCUMENT FOR DETAILS | 051028 | /0001 | |
Jul 31 2019 | JPMORGAN CHASE BANK, N A , AS COLLATERAL AGENT | MICRON SEMICONDUCTOR PRODUCTS, INC | RELEASE BY SECURED PARTY SEE DOCUMENT FOR DETAILS | 051028 | /0001 | |
Jul 31 2019 | MORGAN STANLEY SENIOR FUNDING, INC , AS COLLATERAL AGENT | Micron Technology, Inc | RELEASE BY SECURED PARTY SEE DOCUMENT FOR DETAILS | 050937 | /0001 |
Date | Maintenance Fee Events |
Sep 16 2010 | ASPN: Payor Number Assigned. |
Feb 26 2014 | M1551: Payment of Maintenance Fee, 4th Year, Large Entity. |
Mar 15 2018 | M1552: Payment of Maintenance Fee, 8th Year, Large Entity. |
Mar 15 2022 | M1553: Payment of Maintenance Fee, 12th Year, Large Entity. |
Date | Maintenance Schedule |
Sep 28 2013 | 4 years fee payment window open |
Mar 28 2014 | 6 months grace period start (w surcharge) |
Sep 28 2014 | patent expiry (for year 4) |
Sep 28 2016 | 2 years to revive unintentionally abandoned end. (for year 4) |
Sep 28 2017 | 8 years fee payment window open |
Mar 28 2018 | 6 months grace period start (w surcharge) |
Sep 28 2018 | patent expiry (for year 8) |
Sep 28 2020 | 2 years to revive unintentionally abandoned end. (for year 8) |
Sep 28 2021 | 12 years fee payment window open |
Mar 28 2022 | 6 months grace period start (w surcharge) |
Sep 28 2022 | patent expiry (for year 12) |
Sep 28 2024 | 2 years to revive unintentionally abandoned end. (for year 12) |