A method for determining whether a image pixel of an image comprising a plurality of image pixels generated from a pixel array of an image sensor, each having an image pixel value formed from a respective reset level, has suffered a darkening resulting from a drop in its reset level prior to sampling due to a high intensity illumination. Where a first image pixel has its reset level detected to have crossed a threshold and a second image pixel is saturated, a third image pixel between the first and second image pixels is determined to have suffered such darkening if it is not saturated and if no intervening image pixel between the first and the third image pixels either is saturated or is generated from a reset level detected to have crossed a threshold. The crossing of the reset level may be signaled by a reserved codeword.
|
5. A method for generating an image from a pixel array in an image sensor where a reset signal that undergoes a sampling to generate an image pixel value for an image pixel of the image changes prior to the sampling due to a high intensity illumination, comprising:
setting the image pixel value to a digital reserved codeword if the reset signal is detected to have crossed a predetermined threshold, the digital reserved codeword being otherwise avoided on the image pixel value.
1. A method for determining whether a third image pixel that is part of an image generated from a pixel array of an image sensor and comprising a plurality of image pixels, each having an image pixel value formed from a respective reset level generated in said pixel array, has suffered a darkening resulting from a drop in its reset level prior to sampling due to a high intensity illumination, the method comprising:
finding, within said image sensor or within a processor coupled to said image sensor, a first image pixel among the plurality of image pixels whose reset level is detected to have crossed a threshold;
finding a second image pixel among the plurality of image pixels that has a saturated image pixel value, the third image pixel being located between the first and second image pixels; and,
determining that the third image pixel has suffered the darkening where its image pixel value is not saturated and where there is not an intervening image pixel among the plurality of image pixels that is located between the first and the third image pixels and that either has its image pixel value saturated or has its reset level detected to have crossed a threshold.
2. The method of
3. The method of
4. The method of
|
This application claims priority to Application No. 60/967,657 filed on Sep. 5, 2007, and Application No. 60/967,651 filed on Sep. 5, 2007.
1. Field of the Invention
The subject matter disclosed generally relates to the field of semiconductor image sensors.
2. Background Information
Photographic equipment such as digital cameras and digital camcorders contain electronic image sensors that capture light for processing into a still or video image, respectively. There are two primary types of electronic image sensors, charge coupled devices (CCDs) and complimentary metal oxide semiconductor (CMOS) sensors. CCD image sensors have relatively high signal to noise ratios (SNR) that provide quality images. Additionally, CCDs can be fabricated to have pixel arrays that are relatively small while conforming with most camera and video resolution requirements. A pixel is the smallest discrete element of an image. For these reasons, CCDs are used in most commercially available cameras and camcorders.
CMOS sensors are faster and consume less power than CCD devices. Additionally, CMOS fabrication processes are used to make many types of integrated circuits. Consequently, there is a greater abundance of manufacturing capacity for CMOS sensors than CCD sensors.
The image sensor is typically connected to an external processor and external memory. The external memory stores data from the image sensor. The processor processes the stored data. It is desirable to provide a low noise, high speed, high resolution image sensor that can utilize external memory and provide data to the processor in an efficient manner.
An image sensor with a pixel array that includes at least one pixel. The sensor may also include a circuit that is connected to the pixel and provides a final image pixel value that is a function of a sampled reset output signal subtracted from a sampled light response output signal that are generated from the pixel. The final image pixel value is set to a maximum value if the sampled reset output signal exceeds a threshold. The final image may be a function of first, second and/or third images and a field that provides information on whether the final image includes a first exposure rate, a second exposure rate and/or a third exposure rate.
Disclosed is an image sensor that has one or more pixels within a pixel array. The pixel array may be coupled to a control circuit and a subtraction circuits. The control circuit may cause each pixel to provide a first reference output signal and a reset output signal. The control circuit may then cause each pixel to provide a light response output signal and a second reference output signal. The light response output signal corresponds to the image that is to be captured by the sensor.
The subtraction circuit may provide a difference between the reset output signal and the first reference output signal to create a noise signal that is stored in an external memory. The subtraction circuit may also provide a difference between the light response output signal and the second reference output signal to create a normalized light response output signal. The noise signal is retrieved from memory and combined with the normalized light response output signal to generate the output data of the sensor. The output data may be set to a maximum value if the reset signal exceeds a threshold, indicative of being exposed to sunlight or reflection from a mirror. The final image may be a function of first, second, third and fourth images. The image data may be transferred to a processor with a field that provides information on the exposure rate of the image data.
Referring to the drawings more particularly by reference numbers,
The pixel array 12 is coupled to a light reader circuit 16 by a bus 18 and to a row decoder 20 by control lines 22. The row decoder 20 can select an individual row of the pixel array 12. The light reader 16 can then read specific discrete columns within the selected row. Together, the row decoder 20 and light reader 16 allow for the reading of an individual pixel 14 in the array 12.
The light reader 16 may be coupled to an analog to digital converter 24 (ADC) by output line(s) 26. The ADC 24 generates a digital bit string that corresponds to the amplitude of the signal provided by the light reader 16 and the selected pixels 14.
The ADC 24 is coupled to a pair of first image buffers 28 and 30, and a pair of second image buffers 32 and 34 by lines 36 and switches 38, 40 and 42. The first image buffers 28 and 30 are coupled to a memory controller 44 by lines 46 and a switch 48. The memory controller 44 can more generally be referred to as a data interface. The second image buffers 32 and 34 are coupled to a data combiner 50 by lines 52 and a switch 54. The memory controller 44 and data combiner 50 are connected to a read back buffer 56 by lines 58 and 60, respectively. The output of the read back buffer 56 is connected to the controller 44 by line 62. The data combiner 50 is connected to the memory controller 44 by line 64. Additionally, the controller 44 is connected to the ADC 24 by line 66.
The memory controller 44 is coupled to an external bus 68 by a controller bus 70. The external bus 68 is coupled to an external processor 72 and external memory 74. The bus 70, processor 72 and memory 74 are typically found in existing digital cameras, cameras and cell phones. The processor can perform various computations typically associated with processing images. For example, the processor can perform white balancing or coloring compensation, or image data compression such as compression under the JPEG or MPEG compression standards.
To capture a still picture image, the light reader 16 retrieves a first image of the picture from the pixel array 12 line by line. The switch 38 is in a state that connects the ADC 24 to the first image buffers 28 and 30. Switches 40 and 48 are set so that data is entering one buffer 28 or 30 and being retrieved from the other buffer 30 or 28 by the memory controller 44. For example, the second line of the pixel may be stored in buffer 30 while the first line of pixel data is being retrieved from buffer 28 by the memory controller 44 and stored in the external memory 74.
When the first line of the second image of the picture is available the switch 38 is selected to alternately store first image data and second image data in the first 28 and 30, and second 32 and 34 image buffers, respectively. Switches 48 and 54 may be selected to alternatively store first and second image data into the external memory 74 in an interleaving manner. This process is depicted in
There are multiple methods for retrieving and combining the first and second image data. As shown in
In the event the processor data rate is the same as the memory data rate the processor 72 may directly retrieve the pixel data rate from the external memory 74 in either an interleaving or concatenating manner as shown in
To capture a video picture, the lines of pixel data of the first image of the picture may be stored in the external memory 74. When the first line of the second image of the picture is available, the first line of the first image is retrieved from memory 74 at the memory data rate and combined in the data combiner 50 as shown in
For video capture the buffers 28, 30, 32 and 34 may perform a resolution conversion of the incoming pixel data. There are two common video standards NTSC and PAL. NTSC requires 480 horizontal lines. PAL requires 590 horizontal lines. To provide high still image resolution the pixel array 12 may contain up to 1500 horizontal lines. The image sensor converts the output data into a standard format. Converting on board the image sensor reduces the overhead on the processor 72.
R=¼*(R1+R2+R3+R4) (1)
B=¼*(B1+B2+B3+B4) (2)
GB=½*(G1+G2) (3)
GR=½*(G3+G4) (4)
The net effect is a 75% reduction in the data rate, arranged in a Bayer pattern.
R=¼*(R1+R2+R3+R4) (5)
B=¼*(B1+B2+B3+B4) (6)
GB=½*(G1+G2) (7)
GR=½*(G3+G4) (8)
GBB=½*(G5+G6) (9)
GRR=½*(G7+G8) (10)
The net effect is a 62.5% reduction in the data rate.
G12=½*(G1+G2) (11)
G34=½*(G3+G4) (12)
G56=½*(G5+G6) (13)
G78=½*(G7+G8) (14)
R12=½*(R1+R2) (15)
R34=½*(R3+R4) (16)
B12=½*(B1+B2) (17)
B34=½*(B3+B4) (18)
The net effect is a 50% reduction in the data rate.
To conserve energy the memory controller 44 may power down the external memory 74 when memory is not receiving or transmitting data. To achieve this function the controller 44 may have a power control pin 76 connected to the CKE pin of a SDRAM (see
The gate of reset transistor 112 may be connected to a RST line 118. The drain node of the transistor 112 may be connected to IN line 120. The gate of select transistor 114 may be connected to a SEL line 122. The source node of transistor 114 may be connected to an OUT line 124. The RST 118 and SEL lines 122 may be common for an entire row of pixels in the pixel array 12. Likewise, the IN 120 and OUT 124 lines may be common for an entire column of pixels in the pixel array 12. The RST line 118 and SEL line 122 are connected to the row decoder 20 and are part of the control lines 22.
The double sampling circuits 150 are connected to an operational amplifier 180 by a plurality of first switches 182 and a plurality of second switches 184. The amplifier 180 has a negative terminal − coupled to the first capacitors 152 by the first switches 182 and a positive terminal + coupled to the second capacitors 154 by the second switches 184. The operational amplifier 180 has a positive output + connected to an output line OP 188 and a negative output − connected to an output line OM 186. The output lines 186 and 188 are connected to the ADC 24 (see
The operational amplifier 180 provides an amplified signal that is the difference between the voltage stored in the first capacitor 152 and the voltage stored in the second capacitor 154 of a sampling circuit 150 connected to the amplifier 180. The gain of the amplifier 180 can be varied by adjusting the variable capacitors 190. The variable capacitors 190 may be discharged by closing a pair of switches 192. The switches 192 may be connected to a corresponding control line (not shown). Although a single amplifier is shown and described, it is to be understood that more than one amplifier can be used in the light reader circuit 16.
The RST line 118 may be connected to a tri-state buffer (not shown) that is switched to a tri-state when the IN line 120 is switched to a high state. This allows the gate voltage to float to a value that is higher than the voltage on the IN line 120. This causes the transistor 112 to enter the triode region. In the triode region the voltage across the photodiode 100 is approximately the same as the voltage on the IN line 120. Generating a higher gate voltage allows the photodetector to be reset at a level close to Vdd. CMOS sensors of the prior art reset the photodetector to a level of Vdd-Vgs, where Vgs can be up to 1 V.
The SEL line 122 is also switched to a high voltage level which turns on transistor 114. The voltage of the photodiode 100 is provided to the OUT line 124 through level shifter transistor 116 and select transistor 114. The SAM1 control line 166 of the light reader 16 (see
Referring to
The SAM2 line 168 is driven high, the SEL line 122 is driven low and then high again, so that a level shifted voltage of the photodiode 100 is stored as a reset output signal in the second capacitor 154 of the light reader circuit 16. Process blocks 300 and 302 are repeated for each pixel 14 in the array 12.
Referring to
Referring to
Referring to
Referring to
Referring to
The process described is performed in a sequence across the various rows of the pixels in the pixel array 12. As shown in
Referring to
To prevent such a scenario, the reset level may be compared to a threshold. By way of example, the combiner 50 shown in
The combiner 50 may output a special reserved code, for example 11 0000 0000 (“MAX signal”), to represent this maximum illumination value. For normal processing, i.e. the reset level does not cross the threshold, the combiner 50 outputs all possible codes except this special reserved code. For example, if the normal processing would produce a value equal to this special reserved code, the combiner 50 may skip to the next higher value code, in this example 11 0000 0001.
In this manner, the processor 72 can unambiguously detect that a pixel value designates a reset level crossing threshold due to excessive illumination on the pixel when the pixel value is equal to the MAX signal. The processor 72 may proceed to image processing on the picture received from the image sensor 10 to eliminate the picture artifact of a darkened ring as follows.
As shown in
The various control signals RST, SEL, IN, SAM1, SAM2 and SUB can be generated in the circuit generally referred to as the row decoder 20.
The comparators 350 are connected to plurality of AND gates 356 and OR gates 358. The OR gates 358 are connected to latches 360. The latches 360 provide the corresponding IN, SEL, SAM1, SAM2 and RST signals. The AND gates 356 are also connected to a mode line 364. To operate in accordance with the timing diagram shown in
The latches 360 switch between a logic 0 and a logic 1 in accordance with the logic established by the AND gates 356, OR gates 358, comparators 350 and the present count of the counter 352. For example, the hardwired signals for the comparator coupled to the IN latch may contain a count values of 6 and a count value of 24. If the count from the counter is greater or equal to 6 but less than 24 the comparator 350 will provide a logic 1 that will cause the IN latch 360 to output a logic 1. The lower and upper count values establish the sequence and duration of the pulses shown in
The sensor 10 may have a plurality of reset RST(n) drivers 370, each driver 370 being connected to a row of pixels.
In block 402 a short exposure output signal is generated in the selected pixel and stored in the second capacitor 154 of the light reader circuit 16.
In block 404 the selected pixel is then reset. The level shifted reset voltage of the photodiode 100 is stored in the first capacitor 152 of the light reader circuit 16 as a reset output signal. The short exposure output signal is subtracted from the reset output signal in the light reader circuit 16. The difference between the short exposure signal and the reset signal is converted into a binary bit string by ADC 24 and stored into the external memory 74 in accordance with one of the techniques shown in
In block 406 the light reader circuit 16 stores a long exposure output signal from the pixel in the second capacitor 154. In block 408 the pixel is reset and the light reader circuit 16 stores the reset output signal in the first capacitor 152. The long exposure output signal is subtracted from the reset output signal, amplified and converted into a binary bit string by ADC 24 as long exposure data.
Referring to
By way of example, the image may be initially set to all zeros. The processor 72 then analyzes the long exposure data. If the long exposure data does not exceed a threshold then N least significant bits (LSB) of the image is replaced with all N bits of the long exposure data. If the long exposure data does exceed the threshold then N most significant bits (MSB) of the image are replaced by all N bits of the short exposure data. This technique increases the dynamic range by M bits, where M is the exponential in an exposure duration ratio of long and short exposures that is defined by the equation l=2M. The replaced image may undergo a logarithmic mapping to a final picture of N bits in accordance with the mapping equation Y=2N log2(X)/(N+M).
The memory controller 44 begins to retrieve short exposure data for the pixels in row (n−k−l) at the same time as the (n−k−l)-th pixel array is completing the long exposure period. At the beginning of a line period, the light reader circuit 16 retrieves the short exposure output signals from the (n−k)-th row of the pixel array 12 as shown by the enablement of signals SAM1, SAM2, SEL(n−k) and RST(n−k). The light reader circuit 16 then retrieves the long exposure data of the (n−k−l)-th row.
The dual modes of the image sensor 10 can compensate for varying brightness in the image. When the image brightness is low the output signals from the pixels are relatively low. This would normally reduce the SNR of the resultant data provided by the sensor, assuming the average noise is relatively constant. The noise compensation scheme shown in
The exposure durations from the first image to the last image may change from longer to shorter, such that the exposure rate of the first image is longer than the exposure rate of the fourth image. Each exposure may be made a power-of-two times as long as the short exposure. For example, if there are 4 exposures, and the shortest exposure lasts 3 line periods, the next longer exposure may last 3 times 2, i.e. 6 line periods, the next longer may last 6 times 4, i.e. 24 line periods, and the longest 24 times 4, i.e. 96 line periods.
The process begin in
The image C is then read from memory and combined with an image D that is read from the pixel array to create an image E. In case of video, image D may have been processed through a resolution conversion circuit. Image D's readout row pixel data is combine with image C's combined row pixel data read-back for the same row. The combined image E is stored into memory in a manner that may overwrite the C image in memory. The image E is read from memory and combined with an image F read from the pixel array to create a final image G. In case of video, the image F may be processed through a resolution conversion circuit. The combined image G is written to the processor.
Referring to
Source label h is one number for each pixel in image Ik−1 and is previously created by the combiner 50 and written to memory during the creation of Ik−1, except in the case of I0 wherein source label h is zero. Combiner output 64 {j, Ik} is such that, for each pixel, source label j's value is either h's or k's depending on the output 640 of comparator 630.
The comparator 630 and multiplexor 610 select the shortest exposure pixel value unless it is too low (i.e. dim). It can do this by comparing the pixel value with a threshold. This decision avoids using over-exposed pixel values. If comparator 630 may provide an output that causes multiplexor 610 to select the prior combined image Ik−1's pixel value over raw image pixel Hk's value, Ik's associated source label j at this pixel is assigned the source label value of h, i.e. j=h; otherwise j is assigned the value of k, i.e. j=k. For example, among the raw image sequence I0 H1 H2 H3, a j=3 in {j, I3} for a particular pixel means the corresponding pixel value is copied from raw image H3. For each pixel, the comparator 630 compares Hk with a given threshold and instructs the multiplexor 610 to output Hk and source label k if Hk≧threshold, otherwise the multiplexor provides an output Ik−1 and source label h. In other words, if Hk≧threshold, j=k and Ik=Hk, otherwise j=h and Ik=Ik−1. By way of example the threshold value may be 50 out of a maximum of 255 if the pixel value is 8 bits the and ratio of successive exposure durations is 4. The choice of threshold is preferably such that the threshold value multiplied with the ratio is less than the maximum of pixel value range.
Another method to select label j is to choose h without considering the output of the comparator 630 if the source label h of the combiner input 60 is less than k−1 for images I2 and up higher. This is so because an h<k−1 indicates a prior decision by comparator 630 that raw image Hk−1 has a pixel value less than the threshold value, and hence raw image Hk also has pixel value less than the threshold value at this pixel since raw image Hk has even less exposure duration than raw image Hk−1.
The final combined image has, for each pixel, the pixel value and its associated source label, which informs the processor of the exposure ratio relative to the longest first image exposure associated with the pixel value. In the final step, combiner 50 generates {j, Ik} for the last combined image from penultimate combined image Ik−1 and the last raw image Hk. The last combined image and its source labels {j, Ik} may be output to the external processor 72 on data bus 68, or processed within combiner 50, to generate a high dynamic range linear image.
To form a high dynamic range linear image from the final combined image {j, Ik}, the pixel values are initially linearized to removed distortions introduced into the light-to-digital conversion process of received light causing digital pixel values. Such sources include PN-junction capacitance variation with bias voltage at the sensing node, threshold voltage variation at the source-follower transistor in the pixel due to body effect, and changes in other analog circuit characteristics due to pixel output voltage change. These variations as a function of pixel output voltage can be characterized and measured either in the factory on by an on-chip self-calibration circuit as is common practice in analog integrated circuit design practice. The result of such calibration can be a linearizing lookup table. Combiner 50 can include one such lookup table. To linearize a pixel value, the combiner 50 inputs this value into the lookup table and receives an output which is the linearized pixel value with distortions removed. The linearized pixel value is directly proportional to exposure duration times light intensity impinging on the pixel array. Linearized pixel values are then scaled inversely proportional to how much their corresponding raw images' exposure durations are scaled with respect to the first, longest exposure image. For example, if a pixel's source label is 2, and the ratio of exposure duration is 1-to-2 for 3rd raw image to 2nd raw image, and 1-to-3 for 2nd raw image to first raw image, then the ratio is 1-to-6 for 3rd raw image to 1st raw image, and thus the linearized pixel value is to be multiplied by 6 to produce high dynamic range linear pixel value.
While certain exemplary embodiments have been described and shown in the accompanying drawings, it is to be understood that such embodiments are merely illustrative of and not restrictive on the broad invention, and that this invention not be limited to the specific constructions and arrangements shown and described, since various other modifications may occur to those ordinarily skilled in the art.
For example, although interleaving techniques involving entire lines of an image are shown and described, it is to be understood that the data may be interleaved in a manner that involves less than a full line, or more than one line. By way of example, one-half of the first line of image A may be transferred, followed by one-half of the first line of image B, followed by the second-half of the first line of image A, and so forth and so on. Likewise, the first two lines of image A may be transferred, followed by the first two lines of image B, followed by the third and fourth lines of image A, and so forth and so on.
Additionally, the memory 74 may be on the same integrated circuit (on board) as the image sensor 14.
Patent | Priority | Assignee | Title |
Patent | Priority | Assignee | Title |
6831689, | Oct 07 1999 | ORLY YADID-PECHT, AN ISRAELI CITIZEN | Optical imager using a method for adaptive real-time expanding of the dynamic range |
7339621, | Dec 13 2001 | Harusaki Technologies, LLC | Imager output signal processing |
7397505, | Jan 17 2002 | Qualcomm Incorporated | CMOS sensor with over-saturation abatement |
7432967, | Apr 02 1997 | Gentex Corporation | Control circuit for image array sensors |
7702179, | Apr 28 2006 | Novatek Microelectronics Corp. | Method and related compensation circuit for compensating black cores of an image sensor |
7872682, | May 10 2005 | U S BANK NATIONAL ASSOCIATION, AS COLLATERAL AGENT | Eclipse elimination by monitoring the pixel signal level |
7880779, | Sep 30 2005 | STMICROELECTRONICS RESEARCH AND DEVELOPMENT LIMITED | Image sensor artifact elimination |
20060119717, | |||
20060262211, | |||
20060278809, | |||
20070146539, |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Date | Maintenance Fee Events |
Dec 18 2015 | REM: Maintenance Fee Reminder Mailed. |
May 08 2016 | EXP: Patent Expired for Failure to Pay Maintenance Fees. |
Date | Maintenance Schedule |
May 08 2015 | 4 years fee payment window open |
Nov 08 2015 | 6 months grace period start (w surcharge) |
May 08 2016 | patent expiry (for year 4) |
May 08 2018 | 2 years to revive unintentionally abandoned end. (for year 4) |
May 08 2019 | 8 years fee payment window open |
Nov 08 2019 | 6 months grace period start (w surcharge) |
May 08 2020 | patent expiry (for year 8) |
May 08 2022 | 2 years to revive unintentionally abandoned end. (for year 8) |
May 08 2023 | 12 years fee payment window open |
Nov 08 2023 | 6 months grace period start (w surcharge) |
May 08 2024 | patent expiry (for year 12) |
May 08 2026 | 2 years to revive unintentionally abandoned end. (for year 12) |