In a method for generating color image signals derived from an array of color image pixels, the array of color pixels are subsampled in order to generate an array of subsampled color pixels. An array of new, evenly spaced color pixels are interpolated from the subsampled color pixels using surrounding pixels of the same color, and the color image signals derived from the array of new color pixels are subsequently processed, e.g., in a color filter array interpolation. By providing this correction in a preprocessing step, the rest of the normal image processing hardware or software may be used on the resultant subsampled CFA image without developing the characteristic blocky and wavy artifacts caused by an irregular subsampling pattern.
|
1. A method for generating color image signals derived from an array of color image pixels, said method comprising the steps of:
subsampling the array of color pixels in order to generate an array of subsampled color pixels, the pixels of the array of subsampled color pixels being unevenly spaced in a row direction and a column direction; interpolating an array of new, evenly spaced color pixels from the subsampled color pixels, the pixels of the array of new color pixels being evenly spaced in the row direction and the column direction, a given pixel of the array of new color pixels being interpolated using surrounding pixels, of the same color as the given pixel, in the array of subsampled color pixels; and processing the color image signals derived from the array of new color pixels.
3. A method for processing color image signals derived from an array of color image pixels arranged at a predetermined pixel spacing, said method comprising the steps of:
subsampling the array of color pixels in 2×2 blocks of color pixels at half resolution in order to generate an array of subsampled color pixels, the pixels of the array of subsampled color pixels being unevenly spaced in a row direction and a column direction; interpolating an array of new color pixels at half pixel spacing from the subsampled color pixels, the pixels of the array of new color pixels being evenly spaced in the row direction and the column direction, a given pixel of the array of new color pixels being interpolated using surrounding pixels, of the same color as the given pixel, in the array of subsampled color pixels; and processing the color image signals derived from the array of new color pixels.
6. A method for processing color image signals derived from an array of color image pixels arranged at a predetermined pixel spacing, said method comprising the steps of:
capturing a color image through a color filter array to generate a corresponding array of color pixels; subsampling the array of color pixels in 2×2 blocks of color pixels at half resolution in order to generate an array of subsampled color pixels, the pixels of the array of subsampled color pixels being unevenly spaced in a row direction and a column direction; interpolating an array of new color pixels at half pixel spacing from the subsampled color pixels, the pixels of the array of new color pixels being evenly spaced in the row direction and the column direction, a given pixel of the array of new color pixels being interpolated using surrounding pixels, of the same color as the given pixel, in the array of subsampled color pixels; and processing the color image signals derived from the array of new color pixels.
2. A method as claimed in
4. A method as claimed in
5. A method as claimed in
7. A method as claimed in
8. A method as claimed in
10. A method as claimed in
11. A method as claimed in
|
The invention relates generally to the field of electronic photography, and in particular to a single sensor camera of the type that utilizes a color filter array.
A typical digital camera uses a single color image sensor (e.g., a charge coupled device (CCD)) to capture a representation of a scene, including a representation of the color in the scene. These sensor devices typically comprise a two-dimensional array of photosite pixels that measure the light at each point on the array. To allow these sensors to capture a color image, a color filter array (CFA) is placed above the pixels so that each pixel measures the light of the color of its associated filter. One common CFA pattern is known as the Bayer RGB pattern. As shown in
In some digital cameras, it is necessary to capture images at a resolution that is lower than the normal resolution of the CCD sensor. One reason for this is the need to take a series of pictures quickly, such as a short low resolution video burst, with minimal memory and image processing time requirements. To accomplish this, the pixels in the CCD sensor are often subsampled. For example, U.S. Pat. No. 5,493,335 describes a variety of subsampling patterns, including one pattern based on 2×2 blocks of pixels. The CFA pattern in
However, the problem with this approach is that the resulting image quality is low because of the irregularly sampled pixels. It is common for images to appear slightly blocky and for edges to appear wavy because of the sampling pattern. It is desirable to find a method of correcting for this type of sampling as a preprocessing step while still using the rest of the normal image processing hardware or software.
The present invention is directed to overcoming one or more of the problems set forth above. Briefly summarized, according to one aspect of the present invention, a method for generating color image signals derived from an array of color image pixels comprises the steps of: subsampling the array of color pixels in order to generate an array of subsampled color pixels; interpolating an array of new, evenly spaced color pixels from the subsampled color pixels using surrounding pixels of the same color; and processing the color image signals derived from the array of new color pixels.
The advantage of this invention is that it corrects for the blocky and wavy artifacts caused by an irregular sampling pattern. Moreover, these artifacts can be corrected in a preprocessing step while still using the rest of the normal image processing hardware or software on the resultant subsampled CFA image.
These and other aspects, objects, features and advantages of the present invention will be more clearly understood and appreciated from a review of the following detailed description of the preferred embodiments and appended claims, and by reference to the accompanying drawings.
Because digital cameras employing electronic sensors, and electronic processing and storage of the resultant image data, are well known, the present description will be directed in particular to elements forming part of, or cooperating more directly with, apparatus in accordance with the present invention. Elements not specifically shown or described herein may be selected from those known in the art. Certain aspects of the embodiments to be described may be provided in software. Given the system as described in the following materials, all such software implementation needed to practice the invention is conventional and within the ordinary skill in such arts.
Referring to
The resultant digital data is applied to a digital signal processor 46, which may compress the image data using, e.g., any one of a number of compression techniques including JPEG compression or a numerically lossless or visually lossless compression technique employing DPCM coding, and otherwise process the image data for storage. More particularly, the digital signal processor 46 applies a CFA correction algorithm stored in a correction ROM 47 to the image data in certain situations. For instance, a user interface section 49 can be employed to instruct the camera to capture a burst of reduced resolution images, and to apply the CFA correction to these images. (In this embodiment, a removable memory 50 serves as a buffer for storing digital CFA data prior to processing, although a separate buffer could be provided.) The processed digital CFA data is applied to the removable memory 50 via an output interface 48. In operation, the CFA image data represents an image of a fixed resolution, usually an image substantially corresponding to the actual resolution of the image sensor 36. Consequently, the memory 50 stores the digital CFA image data from a fixed number of picture elements corresponding to this fixed resolution. However, in the aforementioned case of burst images, the digital CFA image data stored in the memory 50 may be a lower resolution rendition of the fixed resolution provided by the image sensor.
The output interface 48 is a memory card interface adapted to a conventional card interface standard, such as the PCMCIA card interface standard, such as described in the PC Card Standard, published by The Personal Computer Memory Card International Association, Sunnyvale, Calif., March 1997. The removable memory 50 accordingly is a non-volatile PCMCIA memory card containing either solid state memory, such as Flash EPROM memory, or a small hard drive (categorized as a PCMCIA-ATA Type III Hard Drive under the PCMCIA card interface standard). Another card interface standard is the CompactFlash ATA with DOS file format. Alternatively, other non-volatile storage devices may be used, such as a floppy disk magnetic medium or optical storage (in such cases, a suitable interface and a conventional read/write apparatus would be provided in the camera 30, e.g., a magnetic or optical read/write head).
In addition, the camera includes a host computer interface driver 52 for directly connecting the camera 30 to the host computer 32, for example, to download the digital CFA data corresponding to the captured images. (In this process, the removable memory 50 may serve as a buffer memory or a separate buffer memory (not shown) may be provided.) The camera 30 further includes a control processor 54 for controlling (a) the clock generator and driver circuit 42, (b) the digital signal processing chain comprising the ASP and A/D section 44, the digital signal processor 46, the output interface 48, and the user interface 49 and (c) the interface driver 52. The interface driver 52 provides a conventional computer interface, such as a SCSI or IEEE-1394 interface. Consequently, the digital CFA data from the camera 30 interfaces with the host computer 32 either through the interface driver 52, or through a card reader 56 that receives the removable memory 50.
In order to produce a processed image, the host computer 32 includes application programs for processing the captured images and producing a soft copy on a display 58 or a hard copy on a printer 60 (or, e.g., a film writer or the like). For example, the application programs include an algorithm section 62, which applies CFA interpolation 64 to the CFA data, and a user interface 67. The application programs interpolate full color data for each picture element from the CFA data and produce an interpolated output image. The CFA interpolation algorithm 64 may include any of a number of known interpolation techniques. For instance the interpolation techniques in the following patents may be used: U.S. Pat. No. 5,373,322, entitled "Apparatus and method for adaptively interpolating a full color image utilizing chroma gradients"; U.S. Pat. No. 5,382,976, entitled "Apparatus and method for adaptively interpolating a full color image utilizing image gradients"; and U.S. Pat. Nos. 5,506,619 and 5,629,734, both entitled "Adaptive color plan interpolation in a single sensor color electronic camera". Each of these patents is incorporated herein by reference. The processing algorithms may also apply noise reduction, white balance correction, color correction, edge enhancement, and the like. Alternatively, the camera 30 may include the CFA interpolation algorithm for interpolating the digital CFA data (as well as one or more of the processing algorithms that apply noise reduction, white balance correction, color correction, or edge enhancement) and, in that case, the camera 30 provides fully interpolated digital CFA data, making subsequent interpolation in the computer 32 unnecessary.
As mentioned before, the problem with use of the data that is subsampled as in
One method of correcting for this type of sampling artifact is to use surrounding pixels to interpolate a pixel at a location half a pixel diagonally away from the original pixels, as shown in the diagram in FIG. 4. In this diagram, the (large case) R, G and B pixels are the original sampled pixels, while the (small case) r, g, and b pixels are the interpolated pixels. It is important to note that the newly interpolated pixels are centered in a normal rectangular CFA type of pattern that produces a lower resolution version of the original CFA pattern. In particular, the newly interpolated pixels are evenly spaced across the image plane, that is, as shown in
The new pixels can be generated by doing bilinear interpolation on the four pixels of the same color surrounding the new pixel. For example, new pixels r21, g11, b12 and g22 can be generated at half pixel spacing from the old pixels R21, G11, B12, and G22. More specifically, new pixel r21 is bilinearly interpolated for the half pixel separation from the original data by using the following equation:
To simplify the required computation, the interpolation can be approximated with the following equation:
Similarly, the g11, b12, and g22 pixels can be interpolated using the following equations:
The other new pixels (small case r, g, b) in
The overall steps of the CFA correction process are shown in FIG. 5. In the beginning, either the user or the imaging system selects a smaller image size (step 70). For example, the user may select a burst mode of operation, which triggers the system to select a smaller image size. Then, a full resolution CFA image is captured by the image sensor 36 (step 72). The full resolution CFA image data is then subsampled to generate a smaller size, subsampled CFA image (step 74). It is well known in the art to perform this subsampling, e.g., by reading only half of the pixel rows from the sensor 36. Since the other half of the pixel rows are not read from the sensor 36, this results in a significant speed increase while reading the sensor. (Alternatively, the CCD and processing hardware can be arranged so that the full resolution image is read from the sensor 36 and then stored in a buffer memory, which is then sampled according to the subsampling pattern. In that case, however, the speed advantage would be lost in readout of the sensor.) An example of a subsampled CFA image is shown in FIG. 2. Next, a bilinear interpolation is executed on the subsampled CFA image to generate a new subsampled CFA image (step 76). The new subsampled CFA image is shown by the small case r, g, b values in FIG. 4. Finally, processing is performed on the new CFA subsampled image (step 78). This processing could involve a single processing step, such as CFA interpolation, or an entire imaging chain including such steps as noise reduction, white balance correction, CFA interpolation, color correction, edge enhancement, and image compression.
It is important to note that all of the new pixels shown above were generated using the same type of equation. This means, after CFA interpolation of the separate colors, that the smoothing caused by the interpolation is the same in all three color planes. Because of this, the frequency characteristics of each of the three color planes is the same, which means that the subsequent image processing algorithms will not have extra color aliasing.
In some CFA sensors, there is a row-to-row mismatch of green pixels in the CFA pattern. This typically has to do with the readout electronics, where the green/red line has one set of gains, and the green/blue line has another set of gains, and the greens are therefore not completely balanced line to line. In other words, the green pixels in the green/red rows are slightly different from the green pixels in the green/blue rows. This is caused by the aforementioned interaction between consecutive pixels being read from the sensor. On a green/red row, the red pixels affect the green pixels in one way, while on a green/blue row, the blue pixels affect the green pixels in a different way. The resulting image can be much more noisy than it should be.
The algorithm described above can be modified to hide this row-to-row mismatch problem by using different pixels to interpolate the green pixels. If each original green pixel is averaged with the average of two other green pixels from the opposite row-type (i.e., odd row mixed with even row, and vice versa) the line-to-line offset is automatically canceled out, while at the same time creating a new interpolated green pixel at the desired location. The following equations show how this can be done.
It should be noted that while this eliminates the row-to-row mismatch problem without the need of an additional, much more complicated algorithm, it does mean that the interpolation blur in the green channel is slightly different than the interpolation blur in the red and blue channels. As a result, a very small amount of color aliasing becomes visible at certain frequencies in the resulting final image. In some resource constrained systems, this tradeoff is acceptable.
The invention has been described with reference to a preferred embodiment. However, it will be appreciated that variations and modifications can be effected by a person of ordinary skill in the art without departing from the scope of the invention.
30 camera
32 host computer
34 optical section
36 image sensor
38 lens
40 shutter-aperture device
42 clock generator and driver circuit
44 ASP and A/D section
46 DSP
47 CFA correction ROM
48 output interface
49 user interface
50 removable memory
52 interface driver
54 control processor
56 card reader
58 display
60 hard copy printer
62 algorithm section
64 CFA interpolation algorithm
66 CFA correction algorithm
67 user interface
70-78 steps
Smith, Craig M., Adams, Jr., James E.
Patent | Priority | Assignee | Title |
6686961, | Nov 07 1997 | MINOLTA CO , LTD | Image pickup apparatus |
6900838, | Oct 14 1999 | Hitachi Denshi Kabushiki Kaisha | Method of processing image signal from solid-state imaging device, image signal processing apparatus, image signal generating apparatus and computer program product for image signal processing method |
6977683, | Dec 03 1997 | MINOLTA CO , LTD | Digital camera |
7009639, | May 31 1999 | Sony Corporation | Color imaging by independently controlling gains of each of R, Gr, Gb, and B signals |
7019391, | Apr 06 2004 | TRAN, BAO | NANO IC packaging |
7256828, | Jan 16 2003 | Gula Consulting Limited Liability Company | Weighted gradient based and color corrected interpolation |
7330369, | Apr 06 2004 | TRAN, BAO | NANO-electronic memory array |
7393699, | Jun 12 2006 | TRAN, BAO | NANO-electronics |
7456881, | Jan 12 2006 | Aptina Imaging Corporation | Method and apparatus for producing Bayer color mosaic interpolation for imagers |
7474337, | Oct 24 2000 | Sony Corporation; Sony Electronics Inc.; Sony Electronics INC | Method and apparatus to provide edge enhancements as part of a demosaicing process |
7589772, | Sep 21 2005 | Yale University | Systems, methods and devices for multispectral imaging and non-linear filtering of vector valued data |
7652701, | Mar 14 2000 | FUJIFILM Corporation | Solid-state honeycomb type image pickup apparatus using a complementary color filter and signal processing method therefor |
7671398, | Feb 23 2005 | TRAN, BAO | Nano memory, light, energy, antenna and strand-based systems and methods |
7769229, | Nov 30 2006 | OmniVision Technologies, Inc | Processing images having color and panchromatic pixels |
7769230, | Nov 30 2006 | OmniVision Technologies, Inc | Producing low resolution images |
7796814, | Apr 14 2006 | Sony Semiconductor Solutions Corporation | Imaging device |
7862624, | Feb 23 2005 | TRAN, BAO | Nano-particles on fabric or textile |
7864560, | Apr 06 2004 | TRAN, BAO | Nano-electronic array |
7973831, | Dec 12 2007 | ACUTELOGIC CORPORATION | Solid-state image sensor and imaging apparatus using the same |
7999858, | Feb 23 2000 | The Trustees of Columbia University in the City of New York; Sony Corporation | Method and apparatus for obtaining high dynamic range images |
8115840, | Nov 10 2005 | DigitalOptics Corporation | Image enhancement in the mosaic domain |
8139130, | Jul 28 2005 | OmniVision Technologies, Inc | Image sensor with improved light sensitivity |
8154636, | Dec 21 2005 | DigitalOptics Corporation International | Image enhancement using hardware-based deconvolution |
8194296, | May 22 2006 | OmniVision Technologies, Inc | Image sensor with improved light sensitivity |
8274715, | Jul 28 2005 | OmniVision Technologies, Inc | Processing color and panchromatic pixels |
8330839, | Jul 28 2005 | OmniVision Technologies, Inc. | Image sensor with improved light sensitivity |
8350936, | Jul 21 2010 | Canon Kabushiki Kaisha | Recording successive frames of raw sensor data depicting a moving scene |
8416339, | Oct 04 2006 | Omni Vision Technologies, Inc. | Providing multiple video signals from single sensor |
8452082, | Sep 27 2007 | Monument Peak Ventures, LLC | Pattern conversion for interpolation |
8610789, | Feb 23 2000 | The Trustees of Columbia University in the City of New York; Sony Corporation | Method and apparatus for obtaining high dynamic range images |
8711452, | Jul 28 2005 | OmniVision Technologies, Inc. | Processing color and panchromatic pixels |
8797438, | Feb 16 2009 | Nikon Corporation | Digital camera and pixel adding method of same |
8818085, | Sep 27 2007 | Monument Peak Ventures, LLC | Pattern conversion for interpolation |
RE47458, | Sep 27 2007 | Monument Peak Ventures, LLC | Pattern conversion for interpolation |
Patent | Priority | Assignee | Title |
3971065, | Mar 05 1975 | Eastman Kodak Company | Color imaging array |
5019903, | May 04 1989 | Sony Corporation | Spatial interpolation between lines of a supersampled digital video signal in accordance with a gradient vector selected for maximum matching of blocks of samples which are offset in opposite directions |
5053861, | Jul 24 1989 | Eastman Kodak Company | Compression method and apparatus for single-sensor color imaging systems |
5373322, | Jun 30 1993 | Intellectual Ventures Fund 83 LLC | Apparatus and method for adaptively interpolating a full color image utilizing chrominance gradients |
5382976, | Jun 30 1993 | Intellectual Ventures Fund 83 LLC | Apparatus and method for adaptively interpolating a full color image utilizing luminance gradients |
5493335, | Jun 30 1993 | Apple Inc | Single sensor color camera with user selectable image record size |
5506619, | Mar 17 1995 | Intellectual Ventures Fund 83 LLC | Adaptive color plan interpolation in single sensor color electronic camera |
5596367, | Feb 23 1996 | Intellectual Ventures Fund 83 LLC | Averaging green values for green photosites in electronic cameras |
5629734, | Mar 17 1995 | Intellectual Ventures Fund 83 LLC | Adaptive color plan interpolation in single sensor color electronic camera |
5652621, | Feb 23 1996 | Intellectual Ventures Fund 83 LLC | Adaptive color plane interpolation in single sensor color electronic camera |
5852468, | Feb 27 1995 | XACTI CORPORATION | 1-chip color video camera for generating inter-pixel color signal component by interpolating primary color signals from neighboring pixels |
6091851, | Nov 03 1997 | Intel Corporation | Efficient algorithm for color recovery from 8-bit to 24-bit color pixels |
Date | Maintenance Fee Events |
Aug 11 2003 | ASPN: Payor Number Assigned. |
Sep 27 2005 | M1551: Payment of Maintenance Fee, 4th Year, Large Entity. |
Sep 22 2009 | M1552: Payment of Maintenance Fee, 8th Year, Large Entity. |
Sep 25 2013 | M1553: Payment of Maintenance Fee, 12th Year, Large Entity. |
Oct 04 2013 | ASPN: Payor Number Assigned. |
Oct 04 2013 | RMPN: Payer Number De-assigned. |
Date | Maintenance Schedule |
Apr 02 2005 | 4 years fee payment window open |
Oct 02 2005 | 6 months grace period start (w surcharge) |
Apr 02 2006 | patent expiry (for year 4) |
Apr 02 2008 | 2 years to revive unintentionally abandoned end. (for year 4) |
Apr 02 2009 | 8 years fee payment window open |
Oct 02 2009 | 6 months grace period start (w surcharge) |
Apr 02 2010 | patent expiry (for year 8) |
Apr 02 2012 | 2 years to revive unintentionally abandoned end. (for year 8) |
Apr 02 2013 | 12 years fee payment window open |
Oct 02 2013 | 6 months grace period start (w surcharge) |
Apr 02 2014 | patent expiry (for year 12) |
Apr 02 2016 | 2 years to revive unintentionally abandoned end. (for year 12) |