A high-resolution image device that is used to receive images in a variable depth of field environment may include a color image sensor, a panchromatic image sensor, and a measuring device. The color image sensor may be configured to receive a color image of an object. Similarly, a synchronous panchromatic image may be received by the panchromatic image sensor. To generate a fine spatial color image with higher resolution than the image received by the color image sensor, the image device may be configured to replace a luminance component from the original color image with a luminance component from the panchromatic image. However, due to a spatial difference between apertures of the color and panchromatic image sensors, parallax error may occur due to a perceived shift of the imaged object against a reference background in the color and panchromatic images. The measuring device may be configured to measure a distance between the image sensors and the object, such that the image device may be configured to use the distance to account for the effect of parallax.
|
20. An apparatus, comprising:
a means for synchronously imaging an object to generate a high-resolution image of the object in a wide depth of field environment;
a means for measuring a distance from image sensors to the object; and
a means for reducing the effect of parallax due to a spatial distance between apertures of the image sensors.
1. An apparatus for generating a high-resolution image in a wide depth of field environment, comprising:
a color image sensor;
a panchromatic image sensor coupled to the color image sensor; and
a measuring device coupled to the color and panchromatic image sensors, wherein the measuring device is configured to measure a distance from the color and panchromatic image sensors to the object.
6. A method for generating a high-resolution image in a wide depth of field environment, comprising:
synchronously imaging an object with color and panchromatic image sensors;
measuring a distance from the color and panchromatic image sensors to the object; and p1 adjusting an image of the object based on the measured distance to account for the effect of parallax due to a spatial difference between apertures of the color and panchromatic image sensors.
13. A system for generating a high-resolution image in a wide depth of field environment, comprising:
a processor;
a memory in communication with the processor, the memory for storing a plurality of processing instructions for directing the processor to:
synchronously image an object with color and panchromatic image sensors;
measure a distance from the color and panchromatic image sensors to the object; and
adjust an image of the object based on the measured distance to account for the effect of parallax due to spatial difference between apertures of the color and panchromatic image sensors.
2. The apparatus of
a device configured to filter light from an image received by the color image sensor into a multi-spectral output with a luminance component; and
a device configured to replace the luminance component from the color image with a luminance component from a panchromatic image synchronously received by the panchromatic image sensor.
3. The apparatus of
4. The apparatus of
a device configured to detect a luminance component from the image taken by the panchromatic image sensor; and
a device configured to spatially shift the luminance component from the panchromatic image to a position correlating to a synchronous image received by the color image sensor.
5. The apparatus of
7. The method of
8. The method of
9. The method of
10. The method of
11. The method of
12. The method of
14. The system of
15. The system of
16. The system of
17. The system of
18. The system of
19. The system of
|
This application claims the benefit of U.S. Provisional Application No. 61/016,950, filed Dec. 27, 2007, titled “High-Resolution, Variable Depth of Field Image Device,” which is incorporated herein by reference in its entirety.
1. Field
Embodiments of the present invention generally relate to the field of high-resolution imagery. More specifically, the present invention refers to the generation of high-resolution images in a variable depth of field environment.
2. Background
Many satellite sensors provide high-resolution panchromatic (black and white) imagery along with lower resolution multi-spectral (color) imagery. Image fusion techniques, such as pan sharpening, are used to combine these multiple images into a composite product. More specifically, pan sharpening methods process the multi-spectral image with the coarse spatial resolution into a fine spatial color image by fusing the multi-spectral image with the high-resolution panchromatic image. As a result, the high-resolution color image preserves the original color fidelity and allows for better visualization and interpretation.
Satellite imagery typically captures images that have a narrow depth of field. Depth of field refers to the range in front of and beyond an object that appears to be in focus. In imagery applications with a narrow depth of field, pan sharpening techniques provide a reliable method to generate high-resolution color images. However, for images with a wide depth of field, the resolution of color images generated from pan sharpening techniques may be degraded. It is desirable to implement techniques to account for a wide depth of field, such that high-resolution images may be generated using pan sharpening methods.
Additionally, resolution may be degraded when near-field images are captured using pan sharpening techniques due to parallax. Parallax refers to a perceived shift of an imaged object against a background caused by the distance between apertures of the panchromatic and multi-spectral image sensors. Existing satellite and other airborne sensors do not have to account for parallax since the focal length between the sensors and the object being imaged (e.g., the ground) is large enough such that the distance between apertures of the panchromatic and multi-spectral image sensors is negligible. When the focal length is decreased, such as in near-field imaging, loss of resolution due to parallax becomes a greater issue.
In order to obtain high-resolution color images of near- and variable-field objects, the effect of parallax caused by a distance between two separate imaging devices must be reduced. In an embodiment of the present invention, an apparatus for generating a high-resolution image in a wide depth of field environment may include a color image sensor, a panchromatic image sensor, and a measuring device. The measuring device may be configured to measure a distance from the color and panchromatic image sensors to an object, where the distance may be used in accounting for the effect of parallax.
In another embodiment, a method for generating a high-resolution image in a wide depth of field environment may include the following steps: synchronously imaging an object with color and panchromatic image sensors; measuring a distance from the image sensors to the object; and, adjusting an image of the object. In adjusting the image, pixels from an image received by the panchromatic image sensor may be spatially shifted in order to form a resulting panchromatic image with a spatial relationship uniform to a synchronously received color image.
In another embodiment, a system for generating a high-resolution image in a wide depth of field environment may include a processor and a memory. The memory is in communication with the processor and may store processing instructions. These processing instructions may include directing the processor to perform the following functions: synchronously image an object with color and panchromatic image sensors; measure a distance from the color and panchromatic image sensors to the object; and, adjust an image of the object based on the measured distance to account for the effect of parallax. Parallax error may occur when imaging the object due to a spatial difference between apertures of the color and panchromatic image sensors. This embodiment of a high-resolution image system may account for the effect of parallax.
In another embodiment, an apparatus for generating a high-resolution image may include a means for synchronously imaging an object to generate a high-resolution image of the object in a wide depth of field environment. The apparatus may also include a means for measuring a distance from image sensors to the object. Further, the apparatus may include a means for reducing the effect of parallax due to a spatial distance between apertures of the image sensors.
Further embodiments, features, and advantages of the present invention, as well as the structure and operation of the various embodiments are described in detail below with reference to accompanying drawings.
The present invention is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which:
In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be evident, however, to one skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known circuits, structures, and techniques are not shown in detail, but rather in a block diagram in order to avoid unnecessarily obscuring an understanding of this description.
Reference in the description to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the invention. The phrase “in one embodiment” located in various places in this description does not necessarily refer to the same embodiment.
In one embodiment, the apparatus and methods described herein may be used with various moving vehicles such as automobiles, boats, and airplanes. Alternatively, the apparatus and methods herein may be used with other types of moving vehicles.
Image fusion device 210 processes images received by a color image sensor 311 and a panchromatic image sensor 313 to produce a high-resolution color image of an object at output 300 of image device 200. In one embodiment, image fusion device 210 includes color image sensor 311, panchromatic image sensor 313, and a luminance merging device 312. Color image sensor 311 filters light from an image received by color image sensor 311 into a multi-spectral output having a luminance component and two chrominance components (e.g., YCbCr). The luminance component (Y) provides the brightness of objects in the image, while the chrominance components (CbCr) provide color information. An example of color image sensor 311 is a Bayer mosaic charge-coupled device (CCD), which is composed of a Bayer filter that passes red, green, or blue light to selected pixels. Alternatively, other color image sensors may be used, such as a Fovean X3 sensor (array of photosites containing stacked photodiodes) or a three-CCD sensor (three separate charge-coupled devices that measure red, green, and blue light separately).
Panchromatic image sensor 313 detects a luminance component from an image received by panchromatic image sensor 313. An example of panchromatic image sensor 313 is a black and white image sensor. Alternatively, other panchromatic image sensors may be used such as a grayscale image sensor. An output 321 of panchromatic image sensor 321 is coupled to an output 325 of image fusion device 210.
As illustrated in
Returning to
Scanner 332 measures the distance from image device 200 to an imaged object. An example of scanner 332 is a laser scanner. Alternatively, other types of scanners may be used such as a sonic detection and ranging device (which uses sound waves for measurement) and a radar device (which uses radio waves for measurement). The distance measured by scanner 332 may be used by region-specific shifting device 331 to account for the effect of parallax due to a spatial difference between apertures of color and panchromatic image sensors 311 and 313, respectively.
For example,
As illustrated in
In step 630, the image is adjusted based on the measurement in step 620. More specifically, the distance may be used to adjust the images to account for the effect of parallax, where feature matching or a shift and variant feature transform may be used in the adjustment. The effect of parallax is similar to that described with respect to
In step 640, the image is transformed to a fine spatial color image of the object using a luminance component from the panchromatic image. The luminance component from the panchromatic image may be used to transform the color and panchromatic images to a fine spatial image of the object, as described with respect to
Further, the signal processing functions described herein could be embodied by computer program instructions that are executed by a computer processor or any one of the hardware devices listed above. The computer program instructions cause the processor to perform the signal processing functions described herein. The computer program instructions (e.g., software) can be stored in a computer usable medium, computer program medium, or any storage medium that can be accessed by a computer or processor. Such media include a memory device such as a RAM or ROM, or other type of computer storage medium such as a computer disk or CD ROM, or the equivalent. Accordingly, any computer storage medium having computer program code that cause a processor to perform the signal processing functions described herein are within the scope and spirit of the present invention.
In reference to
In one embodiment, the processing instruction for directing the processor to adjust the image based on the measured distance may include spatially shifting a luminance component from the image generated by the panchromatic image sensor to a position correlating to the image received by the color image sensor. Alternatively, the processing direction for directing the processor to adjust the image of the object may also include spatially shifting a luminance component of the image generated by the color image sensor to a position correlating to the image received by the panchromatic image sensor. In shifting the luminance component of either the color or panchromatic image, an angular position between the image sensors and the object may also be calculated as part of the processing instruction. The distance between the image sensors and the object and a distance between apertures of the image sensors may be used in the calculation of the angular position. As described with respect to
In one embodiment, the process instruction for directing the processor to transform the image to a fine spatial color image of the object may include replacing a luminance component from the image generated by the color image sensor with the spatially-shifted luminance component from the panchromatic image. Alternatively, the process instruction for directing the processor to transform the image to a fine spatial color image of the object may also include replacing the spatially-shifted luminance component from the color image with a luminance component from the image generated by the panchromatic image sensor. It is advantageous to replace the color image luminance component with the panchromatic image luminance component since the panchromatic luminance component has a finer spatial resolution than the color luminance component and, therefore, a high-resolution color image may be generated.
The foregoing description of the specific embodiments will so fully reveal the general nature of the invention that others can, by applying knowledge within the skill of the art, readily modify and/or adapt for various applications such specific embodiments, without undue experimentation, without departing from the general concept of the present invention. Therefore, such adaptations and modifications are intended to be within the meaning and range of equivalents of the disclosed embodiments, based on the teaching and guidance presented herein. It is to be understood that the phraseology or terminology herein is for the purpose of description and not of limitation, such that the terminology or phraseology of the present specification is to be interpreted by the skilled artisan in light of the teachings and guidance. Accordingly, the breadth and scope of the present invention should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.
McClatchie, Iain Richard Tyrone
Patent | Priority | Assignee | Title |
8538132, | Sep 24 2010 | Apple Inc. | Component concentricity |
8610726, | Sep 26 2008 | Apple Inc. | Computer systems and methods with projected display |
8761596, | Sep 26 2008 | Apple Inc. | Dichroic aperture for electronic imaging device |
9225889, | Aug 18 2014 | ENTROPIX, INC.; ENTROPIX, INC | Photographic image acquisition device and method |
9356061, | Aug 05 2013 | Apple Inc. | Image sensor with buried light shield and vertical gate |
9792668, | Aug 18 2014 | ENTROPIX, INC. | Photographic image acquistion device and method |
9842875, | Aug 05 2013 | Apple Inc. | Image sensor with buried light shield and vertical gate |
9973706, | Jul 17 2014 | Nokia Technologies Oy | Method and apparatus for detecting imaging conditions |
Patent | Priority | Assignee | Title |
6097835, | Jul 23 1997 | Lockheed Martin Corporation | Projective pan sharpening methods and apparatus |
6694064, | Nov 19 1999 | CORVEQ LLC IMAGING | Digital aerial image mosaic method and apparatus |
20040012670, | |||
20040027347, | |||
20040264796, | |||
20070188601, | |||
20070206242, | |||
CN1556374, |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Dec 29 2008 | Google Inc. | (assignment on the face of the patent) | / | |||
Mar 03 2009 | MCCLATCHIE, IAIN RICHARD TYRONE | Google Inc | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 022359 | /0293 | |
Sep 29 2017 | Google Inc | GOOGLE LLC | CHANGE OF NAME SEE DOCUMENT FOR DETAILS | 044101 | /0405 |
Date | Maintenance Fee Events |
May 27 2016 | M1551: Payment of Maintenance Fee, 4th Year, Large Entity. |
Jul 20 2020 | REM: Maintenance Fee Reminder Mailed. |
Jan 04 2021 | EXP: Patent Expired for Failure to Pay Maintenance Fees. |
Date | Maintenance Schedule |
Nov 27 2015 | 4 years fee payment window open |
May 27 2016 | 6 months grace period start (w surcharge) |
Nov 27 2016 | patent expiry (for year 4) |
Nov 27 2018 | 2 years to revive unintentionally abandoned end. (for year 4) |
Nov 27 2019 | 8 years fee payment window open |
May 27 2020 | 6 months grace period start (w surcharge) |
Nov 27 2020 | patent expiry (for year 8) |
Nov 27 2022 | 2 years to revive unintentionally abandoned end. (for year 8) |
Nov 27 2023 | 12 years fee payment window open |
May 27 2024 | 6 months grace period start (w surcharge) |
Nov 27 2024 | patent expiry (for year 12) |
Nov 27 2026 | 2 years to revive unintentionally abandoned end. (for year 12) |