A method and system is disclosed for reducing motion blur using ccd charge shifting. In one embodiment, photodiode wells are exposed for a set of successive exposures with each exposure duration being a fraction of a total exposure time. After each successive exposure, the photodiode wells integrate signal charges and shift them to corresponding storage lines. The shifted signal charges are then shifted along the storage lines for a specified number of storage units. At the same time, the ccd is moved in the direction of a leading edge of the ccd. The photodiode wells are then exposed for another exposure to produce another set of signal charges, which are shifted to the storage lines. Signal charges from the successive exposures are accumulated at the storage lines. After all successive exposures have been taken, the accumulated signal charges are shifted to a serial shift register and output to form an image.
|
1. A method of reducing motion blur in a digital image produced by a moving charge-coupled device (ccd), comprising:
integrating signal charges at photodiode wells from successive exposures on the ccd, each photodiode well corresponding to a pixel of the digital image, by exposing the ccd to a subject, wherein the duration of each successive exposure is chosen to be a fraction of a total exposure duration that is proportional to a speed at which the ccd physically moves;
shifting the signal charges for each successive exposure in a direction opposite to a direction of ccd motion as the ccd physically moves, wherein a distance of the shift is proportional to a speed at which the ccd physically moves;
accumulating the signal charges from the successive exposures in storage lines of the ccd to enhance the light intensity of each pixel, wherein the integrating and shifting cause the signal charges to be accumulated with other signal charges that correspond to a same portion of the subject;
shifting the accumulated signal charges from the storage lines to a serial shift register; and
outputting the accumulated signal charges from the serial shift register to produce a digital image.
15. A non-transitory computer readable storage medium having instructions stored thereon that, when executed by a processor, cause the processor to perform a method for reducing motion blur in a digital image produced by a moving charge-coupled device (ccd), the method comprising:
integrating signal charges at photodiode wells from successive exposures on the ccd, each photodiode well corresponding to a pixel of the digital image, by exposing the ccd to a subject, wherein the duration of each successive exposure is chosen to be a fraction of a total exposure duration that is proportional to a speed at which the ccd physically moves;
shifting the signal charges for each successive exposure in a direction opposite to a direction of ccd motion as the ccd physically moves, wherein a distance of the shift is proportional to a speed at which the ccd physically moves;
accumulating the signal charges from the successive exposures in storage lines of the ccd to enhance the light intensity of each pixel, wherein the integrating and shifting cause the signal charges to be accumulated with other signal charges that correspond to a same portion of the subject;
shifting the accumulated signal charges from the storage lines to a serial shift register; and
outputting the accumulated signal charges from the serial shift register to produce a digital image.
8. A system for reducing motion blur in a digital image produced by a moving charge-coupled device (ccd), comprising:
a ccd comprising photodiode wells, storage lines, storage control lines, shifting control lines, and a serial shift register;
a shutter-control unit configured to expose the ccd to a subject, wherein the duration of each successive exposure is chosen to be a fraction of a total exposure duration that is proportional to a speed at which the ccd physically moves;
a delay-and-integration control unit configured to:
integrate signal charges at the photodiode wells from the successive exposures on the ccd, each photodiode well corresponding to a pixel of the digital image;
shift the signal charges for each successive exposure in a direction opposite to a direction of ccd motion as the ccd physically moves, wherein a distance of the shift is proportional to a speed at which the ccd physically moves;
accumulate the signal charges from the successive exposures in the storage lines of the ccd to enhance the light intensity of each pixel, wherein the integrating and shifting cause the signal charges to be accumulated with other signal charges that correspond to a same portion of the subject; and
shift the accumulated signal charges from the storage lines to the serial shift register; and
a signal-charge-output unit configured to output the accumulated signal charges from the serial shift register to produce a digital image.
2. The method of
3. The method of
4. The method of
6. The method of
7. The method of
9. The system of
10. The system of
11. The system of
13. The system of
14. The system of
16. The non-transitory computer readable storage medium of
17. The non-transitory computer readable storage medium of
18. The non-transitory computer readable storage medium of
19. The non-transitory computer readable storage medium of
20. The non-transitory computer readable storage medium of
|
This application is a continuation of U.S. patent application Ser. No. 12/344,341, filed Dec. 26, 2008, which claims the benefit of U.S. Provisional Patent Application No. 61/006,128, filed on Dec. 26, 2007, entitled “System and Method for Reducing Motion Blur Using CCD Charge Shifting” by lain Richard Tyrone McClatchie, which is incorporated herein by reference in its entirety.
The present invention relates to digital imaging using charge-coupled devices (CCDs), and in particular to reducing motion blur caused by CCD movement.
When taking an image with a camera moving during a period of exposure, an object in the image that is not moving at the same rate as the camera may appear blurred or smeared along the direction of relative motion. This effect is often called motion blur. The longer the exposure time, more the imaged object moves with respect to the camera and the more the picture is blurred. One simple solution to avoid motion blur is fixing the camera position with respect to the objects to be captured in the image. There are also techniques such as image stabilization, vibration reduction, and anti-shake techniques to counteract the motion of a camera to reduce motion blur. However, such techniques only apply when the motion blur is caused by a slight motion of the camera, such as shaking, vibrating, etc. These techniques can not help in situations where the camera motion is significant.
Because motion blur is related to the image exposure time, reducing the exposure time can reduce the amount of motion blur. However, a shorter exposure time also results in a lower amount of light entering the camera sensor, which in turn, affects the quality of the picture. That is, a picture produced with a shorter exposure time appears to be darker than a picture produced with a longer exposure time.
With the introduction of digital cameras and advances of digital imaging, there are also efforts to reduce motion blur using software-based image processing techniques to process digitalized pictures (or digital images) after they are produced. However, because digital images alone do not provide information about the camera movement during exposures, results from software-based techniques are often not as good as solutions applied on the camera itself.
A method and system is disclosed for reducing motion blur using charge-coupled device (CCD) charge shifting. In one embodiment, photodiode wells and storage lines on a CCD are reset for an exposure. The photodiode wells are then exposed for a first exposure duration as a fraction of a total exposure time. The photodiode wells integrate signal charges during the exposure and shift the signal charges to corresponding storage lines. The signal charges are shifted along the storage lines for a specified number of storage units. At the same time, the CCD is moved in the direction of a leading edge of the CCD for the same specified number of storage units. Then the photodiode wells are exposed for a second exposure duration as a fraction of a total exposure time. The photodiode wells integrate a second set of signal charges and shift the second set of signal charges to the corresponding storage lines. The shifted second set of signal charges are accumulated with the existing signal charges at the storage lines. An additional number of successive exposures can be produced following the above procedure if necessary. Then the accumulated signal charges from all the successive exposures are shifted to a serial shift register and output from the serial shift register to form an image.
Further embodiments, features, and advantages of the invention, as well as the structure and operation of the various embodiments of the invention are described in detail below with reference to accompanying drawings.
Embodiments of the invention are described with reference to the accompanying drawings. In the drawings, like reference numbers may indicate identical or functionally similar elements. The drawings in which an element first appears is generally indicated by the left-most digit in the corresponding reference number.
The present invention relates to reducing motion blur in digital imaging caused by movements of a charge-coupled device (CCD). In the detailed description of the invention herein, references to “one embodiment”, “an embodiment”, “an example embodiment”, etc., indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the art to effect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.
A CCD-based digital camera uses a CCD image sensor to sense the light intensities to which it is exposed in order to produce digital images.
In one embodiment of the invention, because of the charge-shifting feature of a CCD, motion blur caused by movement of a CCD-based digital camera can be reduced by integrating and shifting signal charges from successive exposures. For example, a total exposure time required to produce an image can be divided into a set of successive exposure durations. In this manner, a set of successive exposures are produced within the total exposure time instead of having just one exposure during the total exposure time. Each of the successive exposures has a corresponding exposure duration in the set of successive exposure durations. For example, if a total exposure time is T seconds and two successive exposures are desired, each successive exposure duration in the set of successive exposure durations may be one-half of T seconds.
Signal charges for each successive exposure may be integrated and shifted while moving the CCD along a direction of the CCD at a specified speed. In one embodiment, the direction of the CCD is the direction of a leading edge of the CCD. A leading edge of the CCD, such as leading edge 160 in
Delay-and-integration-control unit 230 includes a charge-reset module 231, an integration-and-shifting module 232, a storage-shifting-control module 233, a delay-shifting-control module 234, an accumulating-control module 235, and a serial-shifting-control module 236. Charge-reset module 231 controls resetting photodiode wells 110 and storage lines 120 for an exposure. Integration-and-shifting module 232 controls integrating signal charges at photodiode wells 110 during an exposure and shifting the integrated signal charges to storage lines 120. Storage-shifting-control module 233 controls shifting signal charges along storage lines 120. Delay-shifting-control module 234 controls delaying the shifting of signal charges from storage lines 120 to serial shift register 150 for a pre-defined duration. Accumulating-control module 235 controls accumulating the signal charges shifted from photodiode wells 110 with existing signal charges at storage lines 120. Serial-shifting-control module 236 controls shifting signal charges from storage lines 120 to serial shift register 150.
In a CCD-based camera mounted on a moving vehicle and used to take street view pictures, motion blur may occur because of the camera movement. In general, when taking a picture, a shorter exposure time may reduce the motion blur caused by the camera movement. However, this also causes the photodiode wells of the CCD to receive a lower amount of light compared with a longer exposure. This produces low quality images (e.g., dark images). According to one embodiment of the invention, accumulating signal charges from successive, short-duration exposures enhances the signal charges for most pixels in the produced image and improves the image quality.
In step 710, signal charges from a plurality of successive exposures are integrated and shifted on a CCD. In one embodiment of the invention, a total exposure time for producing an image is divided into a set of durations for successive exposures. That is, signal charges from a first exposure are integrated and shifted from photodiode wells, such as photodiode wells 110, to storage lines, such as storage lines 120. Before storage lines 120 are reset, photodiode wells 110 are exposed with a second exposure. Signal charges from the second exposure are then integrated and shifted from photodiode wells 110 to storage lines 120.
In step 720, the signal charges from successive exposures are accumulated in the storage lines. For example, the signal charges may be combined or further integrated in storage lines 120.
In step 730, the accumulated signal charges are shifted from the storage lines, such as storage lines 120, to the serial shift register, such as serial shift register 150.
In step 740, the signal charges in the serial shift register, such as serial shift register 150, are output to form an image.
In step 810, photodiode wells 110 and storage lines 120 of CCD 100 are reset to clear any existing signal charges.
In step 820, photodiode wells 110 are exposed to light. The amount of light received by photodiode wells 110 is controlled by shutter unit 210. Shutter unit 210 is in turn controlled by shutter-control unit 220. Exposure time (or duration of an exposure) may be controlled by shutter unit 210 or shutter-control unit 220 also control the exposure time.
In step 830, photodiode wells 110 integrate a set of signal charges due to the exposure. The integrated signal charges are then shifted to storage lines 120. For example, as illustrated in
In step 840, when signal charges 310 are shifted to storage lines 120, signal charges 310 accumulate with existing signal charges in storage lines 120. If signals charges 310 correspond to a first exposure, there are no existing signal charges in storage lines 120 and the accumulation only includes the shifted signal charges from photodiode wells 110.
After the signal charges have been accumulated in the storage lines, routine 800 proceeds to step 850. In step 850, system 200 tests whether another exposure is needed. In one embodiment, system 200 can set a total number of successive exposures to produce an image. System 200 may compare the number of produced successive exposures for the image to be produced with the total number of successive exposures. If the number of produced successive exposures is smaller than the total number, another exposure is needed. Routine 800 proceeds to step 852.
In step 852, the accumulated signal charges are shifted along storage lines 120 for a specified number of storage units. As mentioned above, each storage unit may correspond to a pixel in the produced image. For example, as illustrated in
The specified number of storage units for the shifting of signal charges along storage lines 120 is determined by several factors, including the type of the CCD (e.g. Bayer mosaic CCD or panchromatic CCD, etc.), the speed of the CCD movement, the number of successive exposures, and image quality desired by a user, etc. Different types of CCDs include, for example and without limitation, Bayer mosaic CCDs, RGBE CCDs, and panchromatic CCDs.
One type of CCD used in many digital cameras is a Bayer mosaic CCD. A Bayer mosaic CCD uses a Bayer filter over the CCD that allows the photodiode wells to integrate signal charges for individual RGB colors (i.e. red, green, and blue).
In a Bayer mosaic CCD, every other photodiode well in an orthogonal direction of the CCD has a same color arrangement. Therefore, if a Bayer mosaic CCD is used in one embodiment, the specified number of storage units for the shifting of the signal charges along the storage lines and the CCD movement will be a multiple of two.
A RGBE CCD is similar to a Bayer mosaic CCD with a different color filter (i.e. RGBE filter) over the CCD. The RGBE filter has a color distribution of red, green, blue and emerald (similar to the color cyan).
Another type of CCD is a panchromatic CCD. A panchromatic CCD uses photodiode wells that are all sensitive to the same wavelength(s) of light and collect a larger amount of light when exposed. Because each photodiode well captures the same wavelength(s), a panchromatic CCD does not need a color arrangement such as the RGB color distribution in the Bayer mosaic CCD. Thus, if a panchromatic CCD is used in system 800, the specified number of storage units for the shifting of the signal charges along the storage lines and the CCD movement can be any natural number depending on image quality requirements.
Returning to routine 800 in
After CCD 100 is moved in step 852, routine 800 returns to step 820, where photodiode wells 110 are exposed for a successive exposure to integrate corresponding signal charges.
When routine 800 returns to step 840, signal charges already exist in storage lines 120 as a result of the first pass through steps 820 through 854. For example, as illustrated in
In step 850 of routine 800, if it is determined that no further exposures are needed, routine 800 proceeds to step 860. In step 860, accumulated signal charges are shifted to a serial shift register. For example, as illustrated in
In step 860 of routine 800, accumulated signal charges in the serial shift register are output to form an image. For example, accumulated signal charges 610 in serial shift register 150 are output by signal-charge-output unit 240 to form an image.
In this manner, multiple short-exposure-time signals are combined to create a single image with reduced motion blur.
The systems and methods of the present invention can be applied to any CCD-based camera, including a Bayer mosaic CCD camera a panchromatic CCD camera, etc. The present invention is not limited to reducing motion blur caused by camera movement. By moving the CCD in the direction of movement of an object for a number of storage units related to the speed of the object, motion blur caused by the movement of the object can also be reduced.
It is to be appreciated that the Detailed Description section, and not the Summary and Abstract sections, is intended to be used to interpret the claims. The Summary and Abstract sections may set forth one or more but not all exemplary embodiments of the present invention as contemplated by the inventor, and thus, are not intended to limit the present invention and the appended claims in any way.
The present invention has been described above with the aid of functional building blocks illustrating the implementation of specified functions and relationships thereof. The boundaries of these functional building blocks have been arbitrarily defined herein for the convenience of the description. Alternate boundaries can be defined so long as the specified functions and relationships thereof are appropriately performed.
The foregoing description of the specific embodiments will so fully reveal the general nature of the invention that others can, by applying knowledge within the skill of the art, readily modify and/or adapt for various applications such specific embodiments, without undue experimentation, without departing from the general concept of the present invention. Therefore, such adaptations and modifications are intended to be within the meaning and range of equivalents of the disclosed embodiments, based on the teaching and guidance presented herein. It is to be understood that the phraseology or terminology herein is for the purpose of description and not of limitation, such that the terminology or phraseology of the present specification is to be interpreted by the skilled artisan in light of the teachings and guidance.
The breadth and scope of the present invention should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.
McClatchie, Iain Richard Tyrone
Patent | Priority | Assignee | Title |
10079975, | Feb 21 2013 | MOBILEYE VISION TECHNOLOGIES LTD | Image distortion correction of a camera with a rolling shutter |
10834324, | Feb 21 2013 | Mobileye Vision Technologies Ltd. | Image distortion correction of a camera with a rolling shutter |
9277132, | Feb 21 2013 | MOBILEYE VISION TECHNOLOGIES LTD | Image distortion correction of a camera with a rolling shutter |
Patent | Priority | Assignee | Title |
5572256, | Jul 20 1993 | Kabushiki Kaisha Toshiba | Solid-state imaging apparatus |
5675623, | Dec 04 1995 | THE CHASE MANHATTAN BANK, AS COLLATERAL AGENT | Two dimensional image processing having linear filter within a CCD array |
5990952, | Jun 02 1995 | Driving method of solid-state image pickup device | |
6670986, | Sep 10 1998 | KODAK I L, LTD | Apparatus for orthogonal movement of a CCD sensor and a method of light sampling therewith |
6784935, | Apr 11 1998 | FUJIFILM Corporation | Image pickup apparatus capable of wide dynamic range by double exposure, and control thereof |
6930724, | Sep 17 1999 | RAKUTEN GROUP, INC | Solid-state image sensing apparatus with a relative positional light shift |
20050248676, | |||
20060023099, | |||
20060098107, | |||
20060209204, | |||
20070075218, | |||
20100157126, | |||
WO9712483, |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Mar 12 2009 | MCCLATCHIE, IAIN RICHARD TYRONE | Google Inc | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 028107 | /0872 | |
Apr 25 2012 | Google Inc. | (assignment on the face of the patent) | / | |||
Sep 29 2017 | Google Inc | GOOGLE LLC | CHANGE OF NAME SEE DOCUMENT FOR DETAILS | 044101 | /0299 |
Date | Maintenance Fee Events |
Aug 04 2017 | M1551: Payment of Maintenance Fee, 4th Year, Large Entity. |
Sep 27 2021 | REM: Maintenance Fee Reminder Mailed. |
Mar 14 2022 | EXP: Patent Expired for Failure to Pay Maintenance Fees. |
Date | Maintenance Schedule |
Feb 04 2017 | 4 years fee payment window open |
Aug 04 2017 | 6 months grace period start (w surcharge) |
Feb 04 2018 | patent expiry (for year 4) |
Feb 04 2020 | 2 years to revive unintentionally abandoned end. (for year 4) |
Feb 04 2021 | 8 years fee payment window open |
Aug 04 2021 | 6 months grace period start (w surcharge) |
Feb 04 2022 | patent expiry (for year 8) |
Feb 04 2024 | 2 years to revive unintentionally abandoned end. (for year 8) |
Feb 04 2025 | 12 years fee payment window open |
Aug 04 2025 | 6 months grace period start (w surcharge) |
Feb 04 2026 | patent expiry (for year 12) |
Feb 04 2028 | 2 years to revive unintentionally abandoned end. (for year 12) |