Pattern conversion for interpolation

Abstract

A method of processing a digital image having a predetermined color pattern, includes converting the predetermined color pattern of the digital image into a converted digital image having a different desired color pattern; and using algorithms adapted for use with the desired color pattern for processing the converted digital image.

Description

CROSS-REFERENCE TO RELATED PATENT shown in FIG. 18green
PI1=(P2+P5)/2
PI2=(P2+P7)/2
PI3=(P5+P8)/2
The high-resolution panchromatic image can be enhanced 512. This enhancement can include sigma filtering noise reduction, taking care not to infringe on the image detail. The resulting enhanced high-resolution
RI2=(R3+R6)/2
BI6=(B9+B12)/2
GI8=(G11+G14)/2
The low-resolution Bayer color pattern image 530 is interpolated using an algorithm
PT9=(P5+P13)/2
PT11=(P7+P15)/2
PT14=(P13+P15)/2
To find a pan interpolated value for the position held by G10 the pair of pixels with the smallest difference is used to interpolate P10.
gradBackslash=abs(P5−P15)
gradSlash=abs(P7−P13)
gradVert=abs(PT6−PT14)
gradHoriz=abs(PT9−PT11)
These four classifiers are tested and the minimum value is found. The predictor complementing the minimum classifier is used to predict P10.
predBackslash=(P5+P15)/2
predSlash=(P7+P13)/2
predVert=(PT6+PT14)/2
predHoriz=(PT9+PT11)/2
If the absolute value of gradBackslash−gradSlash is below a threshold then P5 can be the average of P5, P7, P13, and P15.

The result of the interpolation step 615 produces the panchromatic image with checkerboard pattern 620, illustrated in FIG. 7D.

The next step 630 is interpolation of the panchromatic image with checkerboard pattern 620 to a full panchromaticimage 640, illustrated in FIG. 7E. Note that step 630 is essentially identical to the problem of interpolating the green channel in the standard Bayer pattern image, allowing use of known algorithms techniques for this processing step.

Returning to the top of FIG. 6, step 650 interpolates green pixel values for pan pixel locations in the starting color image 600, filling in the green checkerboard of a standard Bayer color image 660.

The preferred calculations for step 650 are shown below, referring to FIGS. 7A and 7F. These calculations adaptively interpolate a pixel value G5 for the location occupied by pixel P5. These calculations use pixels G0, G2, G8, and G10 and four additional temporary pixel values.
GT1=(G0+G2)/2
GT9=(G8+G10)/2
GT4=(G0+G8)/2
GT6=(G2+G10)/2
To find a green interpolated value for the position held by P5 the pair of pixels with the smallest difference is used to interpolate G5
gradBackslash=abs(G0−G10)
gradSlash=abs(G8−G2)
gradVert=abs(GT1−GT8)
gradHoriz=abs(GT4−GT6)
These four classifiers are tested and the minimum value is found. The predictor complementing the minimum classifier is used to predict G5.
predBackslash=(G0+G10)/2
predSlash=(G2+G8)/2
predVert=(GT1+GT9)/2
predHoriz=(GT4+GT6)
If the absolute value of gradBackslash−gradSlash is below a threshold then G5 can be the average of G0, G2, G8, and G10. The result of this interpolation step is an RGB Bayer pattern, illustrated in FIG. 7G. At this point, a known technique for conventional Bayer pattern interpolation 665 is used to produce a full resolution color image 670, illustrated in FIG. 7H. This process gives us the full resolution color image 670 and the full resolution panchromatic image 640. The full resolution panchromatic image 640 is used to enhance the full resolution color image 670 in processing step 680, such as using the panchromatic image to guide noise reduction and sharpening of the full color image, producing an enhanced full resolution color image 690.

A third preferred embodiment for this invention is shown in FIG. 8. This processing flow starts with an image having a predefined color pattern 600, which is illustrated in FIG. 7A and shown below:

	G	R	G	R
	B	P	B	P
	G	R	G	R
	B	P	B	P

The preferred processing for this pattern begins with the interpolation of panchromatic pixels 705 from the starting image to create a Bayer pattern RPB image 710, illustrated in FIG. 9. The preferred calculations for step 705 are the same as those shown in the second embodiment for interpolating a pan checkerboard. The next processing step is the use of a known technique for Bayer pattern interpolation 715 to create a full resolution RPB image 720. This image is a full three color image, although the effective spectral sensitivity of the panchromatic channel is not a good match to a color matching function.

Returning to the top of FIG. 8, step 750 interpolates green pixel values for pan pixel locations in the starting color image 600, filling in the green checkerboard of a standard Bayer color image 760. The preferred calculations for step 750 are the same as those for step 650 in the second embodiment. At this point, a known technique for conventional Bayer pattern interpolation 765 is used to produce a full color RGB image 770. In step 780, the full resolution RPB image 720 is used to enhance the full color RGB image 770, such as using the panchromatic image to guide noise reduction and sharpening of the full color image, producing an enhanced full resolution color image 790.

A fourth preferred embodiment for this invention is shown in FIG. 10. This processing flow starts with an image having a predefined color pattern 500, which is illustrated in FIG. 11A and shown below:

	P	G	P	R
	G	P	R	P
	P	B	P	G
	B	P	G	P

The processing for this embodiment begins with the interpolation of color pixels 825 from the starting image to create a low resolution Bayer pattern RGB image 830, illustrated in FIG. 11C. A preferred method for doing this is bilinear interpolation, illustrated by the diagonal lines connecting pairs of circled pixels in FIG. 11B. Specifically, the pixels shown in FIG. 11D are computed as follows:
GI0=(G1+G4)/2
RI2=(R3+R6)/2
BI6=(B12+B9)/2
GI8=(G11+G14)/2
Step 825 can also include panchromatic pixels in the interpolation, as illustrated in FIG. 11C. Specifically, several of the pixels shown in FIG. 11D can be computed as follows:
GI0=(G1+G4+P0+P5)/4
RI2=(R3+R6+P2+P7)/4
BI6=(B12+B9+P8+P13)/4
GI8−(G11+G14+P10+P15)/4
In this case, the Bayer color pattern image produced has different effective spectral sensitivities than the original color and panchromatic pixels. By mixing panchromatic and color pixels, the color pixels produced have a spectral sensitivity that is a combination of the original spectral sensitivities. Other processing steps, such as color balance and color correction, can be adjusted accordingly.

After converting the image to a Bayer color pattern 830, a known technique for conventional Bayer pattern interpolation 835 is used to produce a full color RGB image 840, illustrated in FIG. 11E.

After obtaining the full color RGB image 840, another processing step 850 can be included. In step 850, the full color RGB image 840 is interpolated, such as with bilinear or bicubic interpolation, to produce a high-resolution color image 890, illustrated in FIG. 11F. This additional processing step would be included if it is important to provide a color image with the approximately the same dimensions as the starting color pattern image 500. If the computing resources are not available or the dimensions of the full color image are not critical, then processing can stop once the full color RGB image 840 is provided.

These embodiments have illustrated processing from color patterns that include panchromatic pixels. Color patterns that do not include panchromatic pixels can be used to provide a derived panchromatic image by combining pixels in the initial color pattern. The pixels in the derived panchromatic image must have controlled centers to match the interpolation of other color channels in order to avoid introducing spatial artifacts. For example, assuming you start with a traditional Bayer pattern, then contiguous red green and blue pixels can be combined to provide the equivalent of a panchromatic pixel. In this way, a panchromatic image can be readily derived and used to enhance the color image as described above.

These embodiments have specifically described processing from CFA patterns using R, G, B and P pixels. The same processing paths can also be used with C, M, Y, and P pixels with only minor alterations (such as different color correction and noise reduction parameters). For clarity, these embodiments describe color filter pattern and other images without specific reference to the details of memory management. This invention can be practiced with a variety of memory and buffer management approaches.

The invention has been described in detail with particular reference to certain preferred embodiments thereof, but it will be understood that variations and modifications are effected within the spirit and scope of the invention.

PARTS LIST


10  light from subject scene
11  imaging stage
12  lens
13  neutral density filter
14  iris
16  brightness sensor
18  shutter
20  image sensor
22  analog signal processor
24  analog to digital (AID) converter
26  timing generator
28  image sensor stage
30  digital signal processor (DSP) bus
32  digital signal processor (DSP) memory
36  digital signal processor (DSP)
38  processing stage
40  exposure controller
50  system controller
52  system controller bus
54  program memory
56  system memory
57  host interface
60  memory card interface
62  memory card socket
64  memory card
70  viewfinder display
72  exposure display
74  user inputs
76  status display
80  video encoder
82  display controller
88  image display
100 minimal repeating unit for Bayer pattern
102 repeating unit for Bayer pattern that is not minimal
110 spectral transmission curve of infrared blocking filter
112 unfiltered spectral photoresponse curve of sensor
114 red photoresponse curve of sensor
116 green photoresponse curve of sensor
118 blue photoresponse curve of sensor
500 image with predefined color pattern
505 processing step
510 high resolution panchromatic image
512 processing step
515 enhanced high resolution panchromatic image
517 processing step
520 low resolution panchromatic image
525 processing step
530 low resolution Bayer color pattern image
535 processing step
540 interpolated low resolution color image
545 processing step
550 low resolution color difference image
560 processing step
570 enhanced low resolution color difference image
575 processing step
580 high resolution color difference image
585 processing step
590 high resolution color image
600 image with predefined color pattern
605 processing step
610 distinct sparse panchromatic image
615 processing step
620 panchromatic image with checker board pattern
630 processing step
640 full resolution panchromatic image
650 processing step
660 standard Bayer color image
665 processing step
670 full resolution color image
680 processing step
690 enhanced full resolution color image
705 processing step
710 Bayer pattern RGB image
715 processing step
720 full resolution RGB image
750 processing step
760 standard Bayer color image
765 processing step
770 full color RGB image
780 processing step
790 enhanced full resolution color image
825 processing step
830 Bayer color pattern
835 Bayer pattern interpolation
840 full color RGB image
850 processing step
890 high resolution color image

Claims

1. A method of processing a digital image having a predetermined color pattern, the method comprising:

creating, by a processing device, a panchromatic image from the digital image, wherein the creating the panchromatic image comprises interpolating a partial panchromatic image to create a full panchromatic image:

creating, by the processing device, a color image from the digital image; and

enhancing, by the processing device, the color image using the panchromatic image.

2. The method of claim 1, wherein the creating the panchromatic image further comprises: A method of processing a digital image having a predetermined color pattern, the method comprising:

creating, by a processing device of a digital camera system, a panchromatic image from the digital image, wherein the creating the panchromatic image comprises extracting panchromatic pixels from the digital image; and, interpolating alternating panchromatic pixels from the extracted panchromatic pixels to create the a partial panchromatic image, and interpolating the partial panchromatic image to create a full panchromatic image;

creating, by the processing device of the digital camera system, a color image from the digital image; and

enhancing, by the processing device of the digital camera system, the color image using the panchromatic image.

3. The method of claim 2, wherein the creating the color image comprises:

interpolating color pixel values from the digital image to create a partial color image; and

interpolating the partial color image to create a full color image.

4. The method of claim 3, wherein the color pixel values comprise green pixel values.

5. The method of claim 3, wherein the partial color image comprises a Bayer color image.

6. The method of claim 5, wherein the interpolating the partial color image comprises performing Bayer pattern interpolation.

7. The method of claim 3, wherein the enhancing the color image comprises using the full panchromatic image to enhance the full color image and to produce an enhanced full resolution color image.

8. The method of claim 7, wherein the using the full panchromatic image to enhance the full color image comprises using the full panchromatic image to reduce noise and sharpen resolution of the full color image.

9. The method of claim 1, wherein the A method of processing a digital image having a predetermined color pattern, the method comprising:

creating, by a processing device of a digital camera system, a panchromatic image from the digital image, wherein the panchromatic image has a checkerboard pattern and creating the panchromatic image comprises interpolating a partial panchromatic image to create a full panchromatic image;

creating, by the processing device of the digital camera system, a color image from the digital image; and

enhancing, by the processing device of the digital camera system, the color image using the panchromatic image.

10. The method of claim 1 2, wherein the color image comprises pixels having three different effective spectral sensitivities.

11. The method of claim 1, wherein A method of processing a digital image having a predetermined color pattern, the method comprising:

creating, by a processing device of a digital camera system, a panchromatic image from the digital image, wherein the creating the panchromatic image comprises interpolating a partial panchromatic image to create a full panchromatic image and the digital image comprises a predetermined color pattern having a minimal repeating unit of:

12. The method of claim 1, wherein A method of processing a digital image having a predetermined color pattern, the method comprising:

creating, by a processing device of a digital camera system, a panchromatic image from the digital image, wherein the creating the panchromatic image comprises interpolating a partial panchromatic image to create a full panchromatic image and the digital image comprises a predetermined color pattern having a minimal repeating unit of:

13. A non-transitory computer-readable medium having instructions stored thereon that, upon execution, are configured to cause a processing device to perform operations comprising:

creating a panchromatic image from the digital image, wherein the creating the panchromatic image comprises interpolating a partial panchromatic image to create a full panchromatic image;

creating a color image from the digital image; and

enhancing the color image using the panchromatic image.

14. The non-transitory computer-readable medium of claim 13 A non-transitory computer-readable medium having instructions stored thereon that, upon execution, are configured to cause a processing device to perform operations comprising:

creating a panchromatic image from a digital image, wherein the creating the panchromatic image comprises interpolating a partial panchromatic image to create a full panchromatic image;

creating a color image from the digital image; and

enhancing the color image using the panchromatic image, wherein the creating the panchromatic image further comprises:

extracting panchromatic pixels from the digital image; and

interpolating alternating panchromatic pixels from the extracted panchromatic pixels to create the partial panchromatic image.

15. The non-transitory computer-readable medium of claim 14, wherein the creating the color image comprises:

interpolating color pixel values from the digital image to create a partial color image; and

interpolating the partial color image to create a full color image.

16. A digital camera system comprising:

an image sensor configured to generate a digital image; and

a processor configured to:

create a panchromatic image from the digital image by interpolating a partial panchromatic image to create a full panchromatic image;

create a color image from the digital image; and

enhance the color image using the panchromatic image.

17. The digital camera system of claim 16, wherein the processor is further configured to: A digital camera system comprising:

an image sensor configured to generate a digital image; and

a processor and a memory storing instructions that, when executed by the processor, cause the processor to:

create a panchromatic image from the digital image by interpolating a partial panchromatic image to create a full panchromatic image;

create a color image from the digital image;

enhance the color image using the panchromatic image;

extract panchromatic pixels from the digital image;

interpolate alternating panchromatic pixels from the extracted panchromatic pixels to create the partial panchromatic image;

interpolate color pixel values from the digital image to create a partial color image;

interpolate the partial color image to create a full color image; and

use the full panchromatic image to enhance the full color image and to produce an enhanced full resolution color image.

18. A method of processing a digital image, the method comprising:

creating, by a processing device of a digital camera system, a panchromatic image from the digital image, wherein creating the panchromatic image comprises extracting panchromatic pixels from the digital image, interpolating alternating panchromatic pixels from the extracted panchromatic pixels to create a partial panchromatic image, and interpolating the partial panchromatic image to create a full panchromatic image; and

enhancing, by the processing device, a color image using the panchromatic image.

19. The method of claim 18, wherein the enhancing the color image comprises:

interpolating color pixel values from the digital image to create a partial color image; and

interpolating the partial color image to create a full color image.

20. The method of claim 19, wherein the interpolating the partial color image comprises performing Bayer pattern interpolation.

21. The method of claim 19, wherein the enhancing the color image comprises using the full panchromatic image to enhance the full color image and to produce an enhanced full resolution color image.

22. The method of claim 21, wherein the using the full panchromatic image to enhance the full color image comprises using the full panchromatic image to reduce noise and sharpen resolution of the full color image.

23. The method of claim 18, further comprising capturing, by the processing device, the digital image.

24. A method of processing a digital image, the method comprising: creating, by a processing device of a digital camera system, an enhanced image from the digital image, wherein creating the enhanced image comprises extracting panchromatic pixels from the digital image, interpolating alternating panchromatic pixels from the extracted panchromatic pixels to create a partial panchromatic image, and interpolating the partial panchromatic image to create a full panchromatic image.

25. The method of claim 24, wherein the creating the enhanced image comprises using the full panchromatic image to produce an enhanced full resolution color image.

26. The method of claim 24, further comprising receiving, by the processing device, the digital image from an image capture system.

27. The method of claim 24, further comprising capturing, by the processing device, the digital image.

28. A non-transitory computer-readable medium having instructions stored thereon that, upon execution, are configured to cause a processing device to perform operations comprising:

creating a panchromatic image from a digital image, wherein the creating the panchromatic image comprises interpolating a partial panchromatic image to create a full panchromatic image; and

enhancing a color image using the panchromatic image, wherein creating the panchromatic image further comprises extracting panchromatic pixels from the digital image and interpolating alternating panchromatic pixels from the extracted panchromatic pixels to create the partial panchromatic image.

29. The non-transitory computer-readable medium of claim 28, wherein the operations further comprise receiving the digital image from an image capture system.