The present invention is directed at method of designing a color filter Arrays (CFA) for CMOS image sensors under the effects of crosstalk for optimal color reproduction. Instead of a focus on lowering crosstalk, a novel method of designing color filter spectral responses to compensate for the effect of crosstalk at the color imaging system level is proposed. As part of this method, a color reproduction model for CMOS and CCD image sensor under the effect of crosstalk is also proposed.
|
0. 37. A method of designing a color filter array for use in an image sensor system, the method comprising:
using crosstalk between pixels of an image sensor to derive a crosstalk characteristic function;
generating a color reproduction model using the crosstalk characteristic function and a first color filter array design having a first spectral response; and
determining whether the color reproduction model has an acceptable color reproduction quality, wherein the color reproduction model is regenerated using a second color filter array design having a second spectral response when the color reproduction model does not have the acceptable color reproduction quality.
0. 30. A method for designing a color filter array for an image sensor comprising:
determining a crosstalk characteristic function for one or more pixels of a combined color filter array and image sensor;
using the crosstalk characteristic function and a spectral response curve of the color filter array to provide a color reproduction model;
generating a further spectral response curve for said color filter array, wherein the spectral response curve has a shape selected to optimize color reproduction accuracy of the color reproduction model;
iteratively executing the determining, using, and generating operations until a predetermined metric threshold for the color reproduction accuracy of the color reproduction model is met or exceeded; and
applying a resultant spectral response curve to manufacture the color filter array.
0. 19. A method for designing a color filter array for use with an image sensor, the method comprising:
a) determining a crosstalk characteristic function of pixels of a color filter array and image sensor combination using cumulative effects of crosstalk between the pixels;
b) using the crosstalk characteristic function and one or more spectral response curves of the color filter array to generate a color reproduction model;
c) revising shapes of the spectral response curves for as many as all color channels extant on the image sensor to optimize color reproduction accuracy of the color reproduction model; and
d) repeating operation a), operation b), and operation c) using the revised shapes of the spectral response curves determined at operation c) until a predetermined metric threshold is reached or exceeded by a predetermined amount; and
e) applying resultant spectral response curves to manufacture the color filter array.
1. A method of designing a image sensing system including a color filter array and an image sensor, having wherein the color filter array has spectral response curves for all color channels, for digital imaging sensors under the effect of crosstalk comprising:
determining a crosstalk characteristic function using cumulative effects of up to all pixels on a the image sensor;
modeling color reproduction based on said crosstalk characteristic function and said spectral response curves;
optimizing said spectral response curves of said color filter array wherein the shape shapes of the spectral response curves for as many as all color channels extant on the image sensor are selected to optimize the accuracy of color reproduction; and,
inputting the optimized spectral response curves of said color filter array back to the determining crosstalk characteristic function and the generating a color reproduction model steps until a predetermined metric threshold is reached or exceeded by some predetermined amount.
2. The method of
3. The method of
4. The method of
5. The method of
6. The method of
7. The method of
8. The method of
9. The method of
10. The method of
11. The method of
constructing a process flow including all major mask steps and thermal cycles using a process simulator;
calibrating said process flow against Secondary Ion Mass Spectrometry data; and,
entering said calibrated process flow into a device simulator; and
constructing the crosstalk characteristic function using said device simulator.
12. The method of
13. The method of
where the λl and λn represent the extremes of the a range of wavelengths, and CTmin and CTmax are the minimum and maximum values of crosstalk corresponding to λl and λn, respectively.
14. The method of
where the λlb λbl and λnb λbn represent the extremes of the a range of wavelengths of the a bth band of the an overall wavelength range of the crosstalk characteristic function of the bth band, and CTmin b and CTmab x are the minimum and maximum values of crosstalk correpsonding to λlb λbl and λnb λbn , respectively.
15. The method of
16. The method of
17. The method of
18. The method of
0. 20. The method of claim 19, wherein determining the crosstalk characteristic function comprises directly measuring spectral responses of individual pixels as a function of wavelength using a monochromator to step through a range of wavelengths to illuminate the pixels one wavelength band at a time.
0. 21. The method of claim 19, wherein revising the shapes of the spectral response curves comprises reducing a difference between a predetermined threshold value of a quality metric of the goodness of spectral sensitivities of an image sensor and the value of a quality metric of the goodness of spectral sensitivities of the image sensor used to generate the color reproduction model.
0. 22. The method of claim 21, wherein the quality metric of the goodness is selected from a group consisting of: q-factor, μ-factor, Qst and Qsf, Figure of Merit, and Unified Measure of Goodness.
0. 23. The method of claim 19, wherein the determining the crosstalk characteristic function comprises simulating the cumulative effects of up to all pixels on the image sensor.
0. 24. The method of claim 19, wherein the crosstalk characteristic function is approximated as a linear function CT(λ)=a+b ·λ, wherein
where the λl and λn represent extremes of a range of wavelengths, and CTmin and CTmax are minimum and maximum values of crosstalk corresponding to λl and λn, respectively.
0. 25. The method of claim 19, wherein said crosstalk characteristic function is approximated as a set of piece wise linear functions, wherein each linear function is CTb(λ)=ab+bb·λ, wherein
where the λbl and λbn represent extremes of a range of wavelengths of a bth band of an overall wavelength range of the crosstalk characteristic function of the bth band, and CTmin b and CTmab x are minimum and maximum values of crosstalk corresponding to λbl and λbn , respectively.
0. 26. The method of claim 19, wherein the color reproduction model comprises: Rct(λ)=Rrr(λ)+Ggr(λ)+Bbr(λ), Gct(λ)=Rrg(λ)+Ggg(λ)+Bbr(λ), and Bct(λ)=Rrb(λ)+Ggb(λ)+Bbb(λ), where Rcr(λ), Gcr(λ), Bcr(λ) are respective composite spectral sensitivities for red, green and blue pixels in a Bayer color filter array, Rrr(λ), Ggg(λ), Bbb(λ) are primary spectral sensitivities components of red, green and bluish pixels respectively, Rrg(λ) is a spectral sensitivity contribution of red pixels to green pixels due to crosstalk, Rrb(λ) is a spectral sensitivity contribution of red pixels to blue pixels due to crosstalk, Ggr(λ) is a spectral sensitivity contribution of green pixels to red pixels due to crosstalk, Ggb(λ) is a spectral sensitivity contribution of green pixels to blue pixels due to crosstalk, Bbr(λ) is a spectral sensitivity contribution of blue pixels to red pixels due to crosstalk, and Bbg(λ) is a spectral sensitivity contribution of blue pixels to green pixels due to crosstalk.
0. 27. The method of claim 30, wherein the crosstalk characteristic function is approximated as a linear function CT(λ)=a+b·λ, wherein
where the λl and λn represent extremes of a range of wavelengths, and CTmin and CTmax are minimum and maximum values of crosstalk corresponding to λl and λn, respectively.
0. 28. The method of claim 30, wherein said crosstalk characteristic function is approximated as a set of piece wise linear functions, wherein each linear function is CTb(λ)=ab+bb·λ, wherein
where λbl and λbn represent extremes of a range of wavelengths of a bth band of an overall wavelength range of the crosstalk characteristic function of the bth band, and CTmin b and CTmab x are minimum and maximum values of crosstalk corresponding to λbl and λbn, respectively.
0. 29. The method of claim 30, wherein the color reproduction model comprises: Rct(λ)=Rrr(λ)+Ggr(λ)+Bbr(λ), Gct(λ)=Rrg(λ)+Ggg(λ)+Bbr(λ), and Bct(λ)=Rrb(λ)+Ggb(λ)+Bbb(λ), where Rcr(λ), Gcr(λ), Bcr(λ) are respective composite spectral sensitivities for red, green and blue pixels in a Bayer color filter array, Rrr(λ), Ggg(λ), Bbb(λ) are primary spectral sensitivities components of red, green and bluish pixels respectively, Rrg(λ) is a spectral sensitivity contribution of red pixels to green pixels due to crosstalk, Rrb(λ) is a spectral sensitivity contribution of red pixels to blue pixels due to crosstalk, Ggr(λ) is a spectral sensitivity contribution of green pixels to red pixels due to crosstalk, Ggb(λ) is a spectral sensitivity contribution of green pixels to blue pixels due to crosstalk, Bbr(λ) is a spectral sensitivity contribution of blue pixels to red pixels due to crosstalk, and Bbg(λ) is a spectral sensitivity contribution of blue pixels to green pixels due to crosstalk.
0. 31. The method of claim 30, wherein the crosstalk characteristic function is determined using cumulative effects of up to all pixels on the image sensor.
0. 32. The method of claim 30, wherein the color reproduction model is provided using the crosstalk characteristic function and a plurality of spectral response curves, wherein the plurality of spectral response curves correspond to all color channels of the image sensor.
0. 33. The method of claim 30, wherein determining the crosstalk characteristic function comprises directly measuring spectral responses of individual pixels as a function of wavelength using a monochromator to step through a range of wavelengths to illuminate the pixels one wavelength band at a time.
0. 34. The method of claim 30, wherein revising the shapes of the spectral response curves comprises reducing a difference between a predetermined threshold value of a quality metric of the goodness of spectral sensitivities of an image sensor and the value of a quality metric of the goodness of spectral sensitivities of an image sensor used to generate the color reproduction model.
0. 35. The method of claim 34, wherein the quality metric of the goodness is selected from a group consisting of: q-factor, μ-factor, Qst and Qsf, Figure of Merit, and Unified Measure of Goodness.
0. 36. The method of claim 30, wherein the determining the crosstalk characteristic function comprises simulating cumulative effects of up to all pixels on the image sensor.
0. 38. The method of claim 37, further comprising determining the crosstalk characteristic function by directly measuring spectral responses of individual pixels of the image sensor as a function of wavelength using a monochromator to step through a range of wavelengths to illuminate the pixels one wavelength band at a time.
0. 39. The method of claim 37, wherein the color reproduction quality is selected from a group of metrics consisting of: q-factor, μ-factor, Qst and Qsf, Figure of Merit, and Unified Measure of Goodness.
|
1. Field
This invention relates to the field of digital image processing, in particular the design and optimization of Color Filter Arrays for CMOS image sensors.
2. Related Art
To the average consumer of digital imaging products, the pixel count, or the number of ‘mega pixels’, is synonymous with image quality. Consequently, there is a trend amongst consumers to buy the digital imaging device that advertises the largest pixel count in their price range. As a result of this positive feedback, sensor manufactures are fiercely competing to roll out larger and larger format sensors with ever shrinking pixel size. Shrinking sensor pixel size, however, poses significant challenges to sensor design.
As pixels size decreases, the pixel crosstalk inevitably increases. Crosstalk describes the phenomena in which photons falling on one pixel are “falsely” sensed by surrounding pixels. For example, crosstalk occurs when highly focused light is directed to hit only a red colored pixel, yet the surrounding green and blue colored pixels show a response as demonstrated in
Hence there is a need for a method in color filter design to compensate for the crosstalk in color imaging at the system level. Part of this method is a color reproduction model for CMOS and CCD image sensor under the effect of pixel crosstalk.
Embodiments of the present invention include methods for designing Color Filter Arrays (CFA's) for CMOS image sensors under the effects of crosstalk. In one embodiment, the present invention includes illuminating a single pixel, determining the spectral sensitivity of that pixel and the pixels surrounding it as a function wavelength, compiling the cumulative spectral sensitivity of some number of pixel into a crosstalk characteristic function, using that crosstalk characteristic function to generate a color reproduction model and then, in an iterative fashion, optimizing the spectral response curves of the color filter array to produce the most accurate color reproduction.
In one embodiment, the primary wavelength of the incident light is controlled using a monochromator so that the full range of the spectrum of interest can be incrementally stepped through in narrowband steps.
In one embodiment, the incident light is projected onto only one pixel at a time using a highly focus beam of light. In such an embodiment, only the pixel under measurement is illuminated while all other pixels on the image sensor remain dark.
In one embodiment, a metal mask layer on the image sensor itself is used to block light from hitting all but the one pixel under measurement.
In one embodiment, the crosstalk characteristic function can be generated by simulation of the spectral sensitivity of individual pixels using a process simulator and a device simulator.
In one embodiment, the crosstalk characteristic function can be approximated as super-linear function that takes into consideration the wavelength spectrum of interest and the maximum and minimum spectral sensitivity responses of the all pixels measured.
In one embodiment, the optimization of the CFA is based on comparing a merit function of the color reproduction model against a predetermined threshold value the same merit function.
In one embodiment, the color reproduction model generated takes into account the spectral sensitivity under the effects of crosstalk for each color channel as the sum of the spectral sensitivity component of the primary color of a particular color channels and the contribution of the crosstalk signal of all other surrounding pixels within some area.
Described herein are techniques for optimizing Color Filter Arrays (CFA's) for imaging sensors under the effects of crosstalk. In the following description, for purposes of explanation, numerous examples and specific details are set forth in order to provide a thorough understanding of the present invention. It will be evident, however, to one skilled in the art that the present invention as defined by the claims may include some or all of the features in these examples alone or in combination with other features described below, and may further include obvious modifications and equivalents of the features and concepts described herein.
In one embodiment, one iteration of the process comprises four steps. In the first step, the relationship between crosstalk and photon wavelength in the image sensors is characterized with a crosstalk characteristic function. This function is typically expressed as the percentage of crosstalk signal as a function of wavelength. In the second step, based on the crosstalk characteristic function and a novel color reproduction model proposed in this invention, the spectral sensitivity of the whole image sensor is obtained. Then the spectral sensitivity of the image sensor system can be optimized by optimizing the selection of color filters used in color filter array (CFA) in the third step. In practice, however, due to the manufactory limitations on the shape of the spectral response curves of the color filters, the whole optimization and selection can be performed in an iterative way to obtain the best selection of color filters, hence achieve the best color reproduction for the image sensor system. Therefore, the fourth step is iterative feedback step. Optimized color filters obtained in the third step are fed back into the 1st and 2nd steps to re-characterize crosstalk and re-perform the color reproduction.
Crosstalk Characterization
Traditionally, crosstalk of an image sensor is measured using while light, a certain lighting illuminant containing a whole range of wavelength components. The crosstalk measured using this method in an accumulated result for a whole range of wavelengths limited by the sensor and the illuminant. Recent research, however, has indicated that the pixel crosstalk in silicon image sensors is strongly wavelength dependent. Generally speaking, the pixel crosstalk increases monotonically with wavelength in the visible range. For accurate crosstalk characteristics, characterization should therefore be performed with a wavelength controlled light source. Such characterization can be obtained using either real measurement or simulation.
In one embodiment, actual measurement of the crosstalk characteristics is achieved using a spectral monochromator system. The spectral monochromator system provides calibrated lighting with controlled narrow band light over the whole range of the visible spectrum or any other range of interest.
In one embodiment, an imaging system is used to provide illumination on a single pixel only. The single pixel could be one of any of the available colored pixels, such as a red, green or blue when using a Bayer color filter array.
According to the present invention, two techniques can be used for isolating light onto a single pixel in crosstalk characterization measurement. In one embodiment, highly focused lighting is directed so that only a single pixel will be illuminated. Optionally, the light can be near perfectly collimated so as to avoid crosstalk between pixels due to a diverging incoming beam after as the light is absorbed in the silicon.
In another embodiment, a metal mask inside silicon sensor is used to block light from hitting all but a single pixel. Using either of the foregoing selective pixel illumination techniques in conjunction with a selectable wavelength light source, the crosstalk characteristics function for a particular color pixel relative to its neighboring pixels can be measured.
In one embodiment, the crosstalk characteristics function used for the following color reproduction estimation is an accumulated result considering simultaneous effect from all pixels of the whole image sensor.
In another embodiment, a reliable approximation of the crosstalk characteristics function may be achieved by using small blocks of pixels, such as blocks of 3×3, 5×5 and 7×7 pixels.
In yet another embodiment, some average operations may also be required to performed to achieve a reliable crosstalk characteristics function.
Optionally, measurement of the crosstalk characteristics function can be also performed for pixels of a ‘raw’ image sensor without using a CFA or micro-lenses.
In one embodiment, the crosstalk characteristics function is obtained by using one of many mature and complex commercial process simulator tools. For example, commercial process simulator Athena and device simulator Atlas can be applied to simulate the crosstalk in CMOS image sensors.
In one embodiment, the simulation comprises the following steps: First, a brief process flow including all major mask steps and thermal cycles is constructed using a process simulator. The process flow is then calibrated against SIMS (Secondary Ion Mass Spectrometry) data. The process simulation results are then fed into a device model comprising a row of pixels of interest. Finally, device simulation is performed to study the optoelectronic behavior of the structure and generate the crosstalk characteristics function. An example of crosstalk characteristics function is shown in
In one embodiment, a good approximation of crosstalk characteristics function can be represented by a super-linear model as shown in
where
Where
Gct(λ)=Rrg(λ)+Ggg(λ)+Bbg(λ) (2)
Bct(λ)=Rrb(λ)+Ggh(λ)+Bbb(λ)
where Rct(λ), Gct(λ), Bct(λ) are the composite spectral sensitivities for red, green and blue pixels in a Bayer CFA, respectively. Rrr(λ), Ggg(λ), Bbb(λ) are the primary spectral sensitivities components of a pixel of a particular color. For example, Rrr(λ) is the red signal contribution when measuring a red pixel. Ggr(λ) is the spectral sensitivity factor contributed from green pixels to red pixels due to the pixel crosstalk and so on and so forth. Other variables on the right side of Equation 2 have similar definitions as Ggr(λ). It is worth noting that Equation 2 averages the difference between two types of green pixels. If necessary, in practice, Eq. 2 is flexible to be extended for two different types of green pixels.
In one embodiment, Equation 2 can be rewritten in different formats for different application considerations. For example, the crosstalk characteristics function can be measured without the CFA and defined as a function of wavelength, CT(λ). Assuming the pixel crosstalk is independent of CFA, Equation 2 can be written as:
Rct(λ)=kr(λ)·[Wrr(λ)·R(λ)+Wgr(λ)·G(λ)+Wbr(λ)·B(λ)]
Gct(λ)=kg(λ)·[Wrg(λ)·R(λ)+Wgg(λ)·G(λ)+Wbg(λ)·B(λ)] (3)
Bct(λ)=kb(λ)·[Wrb(λ)·R(λ)+Wgb(λ)·G(λ)+Wbb(λ)·B(λ)]
where kr(λ), kg(λ), kb(λ) are wavelength dependent factors related to micro-lens and black-white spectral sensitivity of the image sensor for red, green and blue pixels respectively, and R(λ), G(λ), B(λ) are the spectral sensitivities for the color filters of CFA, Wij(λ) are crosstalk factors related to CT(λ) contributed from i color pixels of j color pixels when using Bayer CFA. Optionally, kr(λ), kg(λ), kb(λ) can be treated as the same for different color pixels.
In one embodiment, only the crosstalk in a block of 3×3 pixels is considered for each center test pixel. Based on experimental results, the decrease of crosstalk between two pixels can be approximated as a simple function of 1/d2, where d is the distance from the illuminated pixel to its neighboring pixels, instead of as an exponential function. For example, using Bayer CFA, if an overall 20% crosstalk is measured in the center pixel, then the crosstalk for its neighboring pixels on the diagonal will be 20%/12, and the crosstalk for its neighboring pixels on the vertical and horizontal will be 20%/6. Equation. 3 can then be represented in detail as Equation 4.
Rct(λ)=kr(λ)·[(1−CT(λ))·R(λ)+4(CT(λ)/6)·G(λ)+4(CT(λ)/12)·B(r)]
Gct(λ)=kg(λ)·[2(CT(λ)/6)·R(λ)+(1−CT(λ))·G(λ)+2(CT(λ)/6)·B(λ)] (4)
Bct(λ)=kb(λ)·[4(CT(λ)/12)·R(λ)+4(CT(λ)/6)·G(λ)+(1−CT(λ))·B(λ)]
It is worth noting that Equation 4 represents the average effect and ignores the difference between two green channels of the Bayer pattern in CMOS image sensor. It is also worth noting that the energy is not conserved in Equation 4 since the block using 3×3 pixels is not a closed system.
In one embodiment, in which crosstalk is measured with a CFA, the Equation 2 can be rewritten as Equation 5:
Rct(λ)=kr(λ)·[(1−CTrr(λ))·R(λ)+CTgr(λ)·G(λ+CTbr(λ)·B(λ)]
Gct(λ)=kg(λ)·[CTrg(λ)·R(λ)+(1−CTgg(λ))·G(λ)+CThg(λ)·B(λ)] (5)
Bct(λ)=kh(λ)·[CTrh(λ)·R(λ)+CTgh(λ)·G(λ)+(1−CTbb(λ))·B(λ)]
where R(λ), G(λ), B(λ) are the spectral sensitivities for the color filters of CFA, CTrr(λ), CTgg(λ), CTbb(λ) are sensitivity loses due to crosstalk from red, green and blue pixels to their neighboring pixels, respectively. In Equation 5, CTgr(λ) is the crosstalk factor contributed from green pixels to red pixels, CTbr(λ)is the crosstalk factor contributed from blue pixels to red pixels and so on and so forth. The rest of the variables in the right side of Equation 5 have the similar definitions such that CTxy(λ) is the crosstalk factor contributed from x color pixels to y color pixels.
Optimization for Color Filters
Colorimetrically, the accuracy of color reproduction is based on how closely the spectral responses of the image sensor match the human vision responses. Mathematically, this can be described as how close the spectral sensitivities of the image sensor match the color matching functions or their linear combinations (when noise is not involved). Thus, the design goal should be to optimize the spectral responses of the whole image sensor system, Rct(λ), Gct(λ), Bct(λ) as shown in the Equations 2 to 5, to achieve the best color reproduction. The optimal color filters here will provide us the best overall color reproduction for the image sensor system.
There are many quality metrics to measure the goodness of spectral sensitivities of an image sensor, such as q-factor, μ-factor, Qst and Qsf, Figure of Merit and Unified Measure of Goodness. Noise effects can be also considered in some of these metrics. In one embodiment of the present invention, it is possible to choose one or more of the aforementioned metrics to perform the optimization for the spectral sensitivities of the image sensor and thus obtain the optimal color filters required.
Feedback Loop in Design Chain
As discussed for
In one embodiment, due to manufacturing limitations, the color filters manufactured may differ to the optimal ones. Therefore, the crosstalk characteristics might be changed and need to be re-characterized or simulated. The spectral sensitivities of an image sensor using the color filters may also change and need to be re-modeled. By performing this iterative feedback process, an optimal set of color filters can be obtained.
The foregoing description illustrates various embodiments of the present invention along with examples of how aspects of the present invention may be implemented. The above examples and embodiments should not be deemed to be the only embodiments, and are presented to illustrate the flexibility and advantages of the present invention as defined by the following claims. For example, color filter optimization and design methods according to the present invention may include some or all of the innovative features described above. Based on the above disclosure and the following claims, other arrangements, embodiments, implementations and equivalents will be evident to those skilled in the art and may be employed without departing from the spirit and scope of the invention as defined by the claims.
Tian, Hui, Sun, Qun, Feng, Chen, Li, Jim
Patent | Priority | Assignee | Title |
11640670, | Jun 04 2020 | Samsung Electronics Co., Ltd. | Device and method for compensating crosstalk of image sensor having multi-color filter array |
Patent | Priority | Assignee | Title |
6594388, | May 25 2000 | Monument Peak Ventures, LLC | Color image reproduction of scenes with preferential color mapping and scene-dependent tone scaling |
6791716, | Feb 18 2000 | Monument Peak Ventures, LLC | Color image reproduction of scenes with preferential color mapping |
7305141, | Feb 06 2001 | Pendragon Wireless LLC | Contour filter for image sensor output signals |
7425933, | Apr 14 2004 | TRANSCHIP ISRAEL LTD | Systems and methods for correcting green disparity in imager sensors |
7561194, | Oct 17 2003 | OmniVision Technologies, Inc | Charge diffusion crosstalk reduction for image sensors |
20060098868, |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Nov 24 2010 | Intellectual Ventures II LLC | (assignment on the face of the patent) | / | |||
Jul 18 2011 | Crosstek Capital, LLC | Intellectual Ventures II LLC | MERGER SEE DOCUMENT FOR DETAILS | 026637 | /0632 |
Date | Maintenance Fee Events |
Apr 25 2016 | M1552: Payment of Maintenance Fee, 8th Year, Large Entity. |
Apr 15 2020 | M1553: Payment of Maintenance Fee, 12th Year, Large Entity. |
Date | Maintenance Schedule |
Feb 12 2016 | 4 years fee payment window open |
Aug 12 2016 | 6 months grace period start (w surcharge) |
Feb 12 2017 | patent expiry (for year 4) |
Feb 12 2019 | 2 years to revive unintentionally abandoned end. (for year 4) |
Feb 12 2020 | 8 years fee payment window open |
Aug 12 2020 | 6 months grace period start (w surcharge) |
Feb 12 2021 | patent expiry (for year 8) |
Feb 12 2023 | 2 years to revive unintentionally abandoned end. (for year 8) |
Feb 12 2024 | 12 years fee payment window open |
Aug 12 2024 | 6 months grace period start (w surcharge) |
Feb 12 2025 | patent expiry (for year 12) |
Feb 12 2027 | 2 years to revive unintentionally abandoned end. (for year 12) |