Methods to generate, and devices to use, an adaptive tone curve generation operation are disclosed. tone curves so generated have been found to produce stable tone mapping in a wide range of operational environments, including during multi-exposure fused images (e.g., during high dynamic range image fusion). In general, techniques are disclosed that use a content-load histogram in combination with image capture device noise information to adaptively bound a tone curve's slope and, therefore, the resulting tone curve. As used here, a content-load histogram is a histogram that captures spatial, color or other information carried by an image's pixels at various gray levels.
|
27. A non-transitory program storage device, readable by a processor and comprising instructions stored thereon to cause the processor to:
obtain, from an image capture device, a first image comprising a first plurality of pixels, wherein each pixel has a value;
for each of the first plurality of pixels—
select a pixel, wherein the pixel identifies a bin in a histogram, and
update the identified bin based, at least in part, on the value of the pixel;
obtain noise characteristics of the image capture device;
constrain maximum and minimum values of the histogram based, at least in part, on the noise characteristics;
normalize the histogram following application of the constraints based on the image capture device's noise characteristics;
generate a tone curve based, at least in part, on the constrained and normalized histogram; and
generate an output image based, at least in part, on the tone curve.
16. A method to generate an image using a noise-constrained tone curve, comprising:
obtaining, from an image capture device, a first image comprising pixels wherein each pixel has a value;
identifying a bin within a content-load histogram based, at least in part, on the value of the selected pixel, wherein a content-load histogram comprises a histogram in which each bin identifier corresponds to a value each pixel in the first image may assume, and whose bin values represent an amount of a specified structural information carried by the first image's pixels at a bin's corresponding bin identifier,
determining a content-load value corresponding to the selected pixel,
determining a weight factor for the selected pixel based, at least in part, on the selected pixel and the selected anchor pixel,
updating the identified content-load histogram bin based, at least in part, on the content-load value and the weight factor;
adjusting the content-load histogram based, at least in part, on noise characteristics of the image capture device;
generating a tone curve based, at least in part, on the adjusted content-load histogram; and
generating an output image based, at least in part, on the tone curve.
1. A non-transitory program storage device, readable by a processor and comprising instructions stored thereon to cause the processor to:
obtain, from an image capture device, a first image comprising pixels wherein each pixel has a value;
select one or more anchor pixels;
select, for each of the one or more anchor pixels, every pixel in the first image one at a time, and with each selected pixel—
identify a bin within a content-load histogram based, at least in part, on the value of the selected pixel, wherein a content-load histogram comprises a histogram in which each bin identifier corresponds to a value each pixel in the first image may assume, and whose bin values represent an amount of a specified structural information carried by the first image's pixels at a bin's corresponding bin identifier,
determine a content-load value corresponding to the selected pixel,
determine a weight factor for the selected pixel based, at least in part, on the selected pixel and the selected anchor pixel,
update the identified content-load histogram bin based, at least in part, on the content-load value and the weight factor;
adjust the content-load histogram based, at least in part, on noise characteristics of the image capture device;
generate a tone curve based, at least in part, on the adjusted content load histogram; and
generate an output image based, at least in part, on the tone curve.
30. An electronic device, comprising:
an image capture sensor;
a memory communicatively coupled to the image capture sensor and having stored therein computer program code;
a display element communicatively coupled to the memory; and
one or more processors communicatively coupled to the image capture sensor, the memory, and the display element, the one or more processors adapted to execute the computer program code to:
obtain, from the memory, a first image comprising pixels wherein each pixel has a value,
select one or more anchor pixels;
select, for each of the one or more anchor pixels, every pixel in the first image, one at a time, and with each selected pixel—
identify a bin within a content-load histogram based, at least in part, on the value of the selected pixel, wherein a content-load histogram comprises a histogram in which each bin identifier corresponds to a value each pixel in the first image may assume, and whose bin values represent an amount of a specified structural information carried by the first image's pixels at a bin's corresponding bin identifier,
determine a content-load value corresponding to the selected pixel,
determine a weight factor for the selected pixel base, at least in part, on the selected pixel and the selected anchor pixel, update the identified content-load histogram bin based, at least in part, on the content-load value and the weight factor;
adjust the content-load histogram based, at least in part, on noise characteristics of the image capture sensor;
generate a tone curve based, at least in part, on the adjusted content load histogram; and
store in the memory an output image based, at least in part, on the tone curve.
2. The non-transitory program storage device of
3. The non-transitory program storage device of
4. The non-transitory program storage device of
5. The non-transitory program storage device of
6. The non-transitory program storage device of
identify a second image comprising the same number of pixels as the first image, wherein each pixel in the second image has a value and corresponds to one pixel in the first image; and
select that pixel in the second image corresponding to the selected pixel from the first image, wherein the value of the pixel selected from the second image comprises the content-load value.
7. The non-transitory program storage device of
determine a distance measure value between the selected pixel and the selected anchor pixel; and
determine a weight factor based, at least in part, on the distance measure value.
8. The non-transitory program storage device of
9. The non-transitory program storage device of
10. The non-transitory program storage device of
11. The non-transitory program storage device of
12. The non-transitory program storage device of
13. The non-transitory program storage device of
obtain noise characteristics for the image capture device;
determine one or more of a maximum value and a minimum value for the content-load histogram based, at least in part, on the noise characteristics;
limit values in the content-load histogram in accordance with the one or more of a maximum value and a minimum value to generate a second content-load histogram; and
normalize the second content-load histogram to generate an adjusted content-load histogram.
14. The non-transitory program storage device of
15. The non-transitory program storage device of
17. The method of
18. The method of
19. The method of
identifying a second image comprising the same number of pixels as the first image, wherein each pixel in the second image has a value and corresponds to one pixel in the first image; and
selecting that pixel in the second image corresponding to the selected pixel from the first image, wherein the value of the pixel selected from the second image comprises the content-load value.
20. The method of
determining a distance measure value between the selected pixel and the selected anchor pixel; and
determining a weight factor based, at least in part, on the distance measure value.
21. The method of
22. The method of
23. The method of
24. The method of
25. The method of
obtaining noise characteristics for the image capture device;
determining one or more of a maximum value and a minimum value for the content-load histogram based, at least in part, on the noise characteristics;
limiting values in the content-load histogram in accordance with the one or more of a maximum value and a minimum value to generate a second content-load histogram; and
normalizing the second content-load histogram to generate an adjusted content-load histogram.
26. The method of
28. The non-transitory program storage device of
29. The non-transitory program storage device of
31. The electronic device of
32. The electronic device of
33. The electronic device of
34. The electronic device of
35. The electronic device of
identify a second image comprising the same number of pixels as the first image, wherein each pixel in the second image has a value and corresponds to one pixel in the first image; and
select that pixel in the second image corresponding to the selected pixel from the first image, wherein the value of the pixel selected from the second image comprises the content-load value.
36. The electronic device of
determine a distance measure value between the selected pixel and the selected anchor pixel; and
determine a weight factor based, at least in part, on the distance measure value.
37. The electronic device of
38. The electronic device of
39. The electronic device of
40. The electronic device of
41. The electronic device of
42. The electronic device of
obtain noise characteristics for the image capture sensor;
determine one or more of a maximum value and a minimum value for the content-load histogram based, at least in part, on the noise characteristics;
limit values in the content-load histogram in accordance with the one or more of a maximum value and a minimum value to generate a second content-load histogram; and
normalize the second content-load histogram to generate an adjusted content-load histogram.
43. The electronic device of
44. The electronic device of
|
This disclosure relates generally to the field of image processing and, more particularly, to various techniques to generate noise-constrained tone curves for use in real-time image processing.
Tone mapping is the process of remapping gray levels from a first or input image to different levels in a second or output image. Tone mapping is generally used to adjust the input image to more closely match the visual appearance anticipated by the human eye on a particular output device, and/or to better preserve image information when mapping an input image to a smaller output dynamic range. Global tone mapping refers to the situation where there is a single tone curve that maps input gray levels to output gray levels. Local tone mapping refers to the case where a single gray level in the input image maps to multiple gray levels in the output image depending on the spatial location and configuration of the input image. Tone mapping is used in almost all image processing pipelines, but is of particular importance when rendering high dynamic range images for a limited dynamic range output device (e.g., created for example by fusing multiple exposures). Illustrative devices having a limited output dynamic range include most consumer desktop and portable displays.
In general, the inventive concept provides methods, systems and computer readable media to generate noise-constrained tone curves. In one embodiment, each pixel in an image obtained from an image capture device may be evaluated to determine its contribution to a content-load histogram (e.g., a content-load measure). For example, each pixel may be used to identify a bin in the content-load histogram, where after the pixel (or pixel neighborhood) may be evaluated to identify spatial information corresponding to the pixel. In one embodiment, the spatial information may be generated in real-time as part of the pixel evaluation procedure. In another embodiment, the spatial information may be obtained from a separate image. However obtained, the information may be used to update the identified content-load histogram's bin. Once an image's pixels have been evaluated, the resulting content-load histogram may be adjusted based on the image capture device's noise characteristics. In particular, upper and lower bounds for the content-load histogram may be established. The adjusted content-load histogram may then be use to generate a noise-constrained tone curve which may be used to map an input image to an output image.
In another embodiment, each content-load measure may be adjusted or weighted to account for the presence of one or more anchor pixels. In one such embodiment, a first anchor pixel may be selected and each pixel in the input image evaluated in terms of that anchor pixel. For example, each pixel's content-load measure may be adjusted in accordance with a specified functional relationship between itself and the current anchor pixel. In one embodiment, the specified functional relationship may be based on the distance between the two pixels. At one extreme, every input image pixel may be selected as an anchor pixel. Alternatively, a specified subset of image pixels may be selected to be anchor pixels. Once a content-load histogram has been generated for each anchor pixel, the histograms may be adjusted to account for the image capture devices noise characteristics. Once adjusted, each content-load histogram may be converted to a noise-constrained tone curve and used to generate an output image.
This disclosure pertains to systems, methods, and computer readable media to generate noise-constrained tone curves. The disclosed tone curves have been found to produce stable tone mapping for high dynamic range images produced through multi-exposure fusion (e.g., during high dynamic range image fusion). In general, techniques are disclosed that use a content-load histogram in combination with an image capture device's noise information to adaptively bound the content-load histogram's slope and, therefore, the resulting tone curve. As used herein, a content-load histogram is a histogram where each bin corresponds to a value, or range of values, that pixels in an input image may assume (e.g., luma or color intensity), and where each bin's value represents the amount of spatial, color or other information carried by the input image's pixels at the bin's corresponding value or range of values (e.g., luma or color intensity).
In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the inventive concept. As part of this description, some of this disclosure's drawings represent structures and devices in block diagram form in order to avoid obscuring the invention. In the interest of clarity, not all features of an actual implementation are described in this specification. Moreover, the language used in this disclosure has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter, resort to the claims being necessary to determine such inventive subject matter. Reference in this disclosure to “one embodiment” or to “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the invention, and multiple references to “one embodiment” or “an embodiment” should not be understood as necessarily all referring to the same embodiment.
It will be appreciated that in the development of any actual implementation (as in any development project), numerous decisions must be made to achieve the developers' specific goals (e.g., compliance with system- and business-related constraints), and that these goals may vary from one implementation to another. It will also be appreciated that such development efforts might be complex and time-consuming, but would nevertheless be a routine undertaking for those of ordinary skill in the design an implementation of image processing systems having the benefit of this disclosure.
Referring to
Together the luma values and content-load values at each pixel in the image may be used to generate a content-load histogram (block 115). A content-load histogram may be thought of as a histogram in which each bin corresponds to a gray level in a luma image, and each bin's value represents the amount of spatial or color information carried by the pixels with the corresponding gray level. As such, it will be understood that the content-load histogram generally has a number of bins equal to the number of gray levels in the luma image. Sometimes, however, the content-load histogram's bin size can be different from one digital count, resulting in a histogram having a different size.
One illustrative method to generate a content-load histogram is shown in
H[i]=Y[i]S[i], EQ. 1
where H[i] represents the i-th bin of the content-load histogram, Y[i] represents the i-th element/pixel of the input luma image, S[i] represents the i-th element/pixel of the spatial image, and ‘’ represents a generic composition operator. In one embodiment, for example, the composition operator may represent a sum operator in which each spatial image value is added to that content-load histogram bin at the corresponding luma image's gray level (i.e., value). In another embodiment, the composition operator may yield a mean or standard deviation value. Substantially any function (linear or nonlinear) may be used that captures or characterizes the amount of information carried by the selected spatial image at each gray level present in the input luma image. Once the first luma pixel has been processed, a check can be made to determine if additional luma pixels remain (block 315). If the luma image has not been completely processed in accordance with blocks 305-310 (the “NO” prong of block 315), the next pixel from the luma image may be selected (block 320), where after operation 115 continues at block 305. If all pixels in the input luma image have been processed (the “YES” prong of block 315), operations continue at block 120 of
While spatial content image 225 may comprise a separate buffered image as suggested by
As an aside, the process outlined in
Returning to
where h[i] represents the i-th bin of the normalized and moderated content-load histogram, H[i] represents the i-th bin of the content-load histogram in accordance with block 115, ‘a’ represents a dampening factor, ‘n’ represents the number of bins in the content-load histogram, and the summation value represents a normalizing parameter. The factor a may be chosen to provide the desired stability and may, in general, be determined empirically.
Both system noise (block 125) and quantization noise (block 130) may be obtained and represented by a table or array with one element for each gray level or each gray level range. System or camera noise (represented as Ns) is typically a function of the type of image sensor and the particular image capture parameters used during image capture (e.g., exposure and gain settings). System noise may includes dark noise and shot noise. Quantization noise (represented as Ng) is typically a function of the image processing pipeline that takes the sensor's output and delivers the image or images being processed (e.g., luma image 200). When a single image is being processed, quantization noise Nq may be a constant (for that image). When multiple frames having different exposures are being combined or fused, such as during High Dynamic Range (HDR) or wide area-of-view image generation, the quantization error for each gray level must be estimated from both the original quantization error and the fusing weights for each exposure.
With normalized content-load histogram h, system noise table Ns, and quantization noise Nq tables known, the signal-to-noise ratio for each gray level may be expressed as follows:
where SNR[i] represents the signal-to-noise ratio at the i-th gray level, max( ) represents the algebraic maximum function, and ‘i’ runs from 0 to (n−1). See discussion above.
From EQ. 3, an upper-bound to the tone curve's slope at output gray level ‘i’ may be expressed as follows (block 135):
where TSNR represents the minimum allowed target signal-to-noise ratio (typically a tuning parameter that may be adjusted to suit the needs of the developer's particular implementation). In general, the larger the TSNR value, the less tolerant the resulting tone curve will be to noise. It may be recognized that in conventional histogram equalization, the slope at the i-th gray level is proportional to the luma histogram count at that level. In contrast, operation 100 (and, more specifically, block 135) constrains the maximum slope at any given gray level by incorporating the effect of noise.
When an input image has no, or very few, pixels at some gray levels (e.g., a snow scene without any dark objects), it is usually desirable to not allocate as many output gray levels to these input gray levels. It is also not desirable, however, to eliminate these gray levels altogether from the output image. Doing so could cause gradient areas in the output image to become unnatural, or overly exaggerate contrast at the dominant gray levels. It has been found that to avoid eliminating an input image's zero or low use gray levels during the tone mapping process, a lower-bound to the tone curve's slope may be set (block 140). While this lower-bound may be set arbitrarily, there is usually no need for it to be higher than one quantization level for a single exposure image or, in the case of a multi-exposure fused HDR image, higher than the estimated quantization step size. In another embodiment, lower-bound slope may be a function of the gray level.
Referring to
h[i]=max(SMIN,min(SMAX[i],h[i])) EQ. 5
to clip, and
to renormalize. Here h[i], SMAX[i], SMIN, and max( ) are as discussed above while min ( ) represents the algebraic minimum function.
As shown in
where T[i] represents the value of the i-th tone curve element/value.
Tone curve T[ ] may be used in a number of ways to generate an output image. Referring to
where G[i] represents the gain factor at the i-th pixel position, Yin[i] represents the input luminance image 535 value at the i-th pixel position, and Yout[i] represents the output luminance image 540 value at the i-th pixel position.
While the above describes how to create and use an adaptive global tone curve, the approach set-forth in operation 100 may also be used to provide adaptive local tone curves. In one embodiment, a content-load histogram may be generated for every pixel, with each histogram used to create a corresponding tone curve. In another embodiment, a content-load histogram may be generated for a subset of the image pixels and a tone curve generated for each histogram. In all embodiments disclosed herein, and unlike prior art region-based tones curves, every pixel in an image may contribute to every tone curve. Allowing each pixel to have an influence on each content-load histogram/tone curve has provided unexpectedly good results vis a vis output image quality.
Referring to
Next, a content-load histogram for the selected anchor pixel may be initialized (block 610), a “first” pixel from the image selected (block 615), and a content-load measure determined as described above (block 620). A weight factor may then be determined based on a specified relationship between the current anchor pixel and the selected image pixel (block 625). In general, this specified relationship may be linear or non-linear. In one embodiment, the specified relationship may be a Gaussian distribution of the distance between the anchor pixel and the currently selected image pixel. (For a particular example, see the discussion below in connection with
By way of example, consider the special-case embodiment illustrated in
In general, functional relationship 810 may be whatever a particular implementation needs and, as noted above, may be linear or non-linear. In one embodiment, functional relationship 810 may be a Gaussian distribution of the distance between an anchor pixel (e.g., pixel 705) and a pixel being evaluated (e.g., pixel 715). For example, this relationship may be expressed as follows:
where (xa, ya) represents the x and y coordinates of a first anchor pixel, (x1, y1) represents the x and y coordinates of a pixel from the input image being evaluated, f( ) represents a functional relationship between the pixels at (xa, ya) and (x1, y1), exp( ) represents the exponential function, ‘d’ represents a distance measure between the pixels at (xa, ya) and (x1, y1), and ‘σ’ represents the spread of the Gaussian distribution about the anchor pixel at (xa, ya). In general, distance measure d may represent any metric that makes sense for the particular implementation; not just distance. Because images are composed of pixels that are generally arranged in a rectangular array however, it is convenient to think of d as a distance metric. One example distance measure is a Euclidean distance that may be represented as:
d=(xa−x1)2+(ya−y1)2, EQ. 10
where (xa, ya) and (x1, y1) are as described above.
For the example distribution of EQ. 9, σ represents the spread of the Gaussian distribution about the anchor pixel located at (xa, ya). It can be seen that as σ increases, the spread of the modeled distribution increases which, in turn, gives pixels distal to the anchor pixel at (xa, ya) more influence on the generated values (compared to a smaller σ). Likewise, as σ decreases, the spread of the modeled distribution decreases which, in turn, gives pixels distal to the anchor pixel less influence on the generated values (compared to a larger σ). In one embodiment, a threshold may be applied to the output of the selected functional relationship so that any value below a certain threshold may return a zero and thus, pixels greater than a certain distance from the anchor pixel have no influence on it's histogram. In another embodiment, a floor and/or ceiling function may be applied to the output of the selected functional relationship. Once a anchor pixel's content-load histogram has been determined (e.g., in accordance with
Output image generation in accordance with block 660 in
where pout(x1, x1) represents the value for the pixel located at (x1, x1) in the output image, ω represents weighting factors for each local tone curve, pin (x1, x1) represents the value for the pixel located at (x1, x1) in the input image, T represents the local tone curve operator such that Ti(x1, x1) represents output from the i-th local tone curve corresponding to the value of the input image's pixel located at (x1, x1), and ‘M’ represents the number of local tone curves. It will be recognized that the summation in the denominator of EQ. 11 represents a normalization factor.
With respect to weighting factor ωi, values may be found by looking at some functional relationship (e.g., distance) between pout(x1, x1) and each of the pixels for which tone curves have been generated (two in
Referring to
In one embodiment histogram generation unit 950 produces a content-load histogram in accordance with
Referring to
Processor 1005 may execute instructions necessary to carry out or control the operation of many functions performed by device 1000 (e.g., such as the generation and/or processing of images in accordance with operations 100 and 600). Processor 1005 may, for instance, drive display 1010 and receive user input from user interface 1015. User interface 1015 can take a variety of forms, such as a button, keypad, dial, a click wheel, keyboard, display screen and/or a touch screen. Processor 1005 may be a system-on-chip such as those found in mobile devices and include a dedicated graphics processing unit (GPU). Processor 1005 may be based on reduced instruction-set computer (RISC) or complex instruction-set computer (CISC) architectures or any other suitable architecture and may include one or more processing cores. Graphics hardware 1020 may be special purpose computational hardware for processing graphics and/or assisting processor 1005 process graphics information. In one embodiment, graphics hardware 1020 may include a programmable graphics processing unit (GPU).
Sensor and camera circuitry 1050 may capture still and video images that may be processed to generate images in accordance with this disclosure and may, for example, incorporate raw image processing pipeline 940. Output from camera circuitry 1050 may be processed, at least in part, by video codec(s) 1055 and/or processor 1005 and/or graphics hardware 1020, and/or a dedicated image processing unit incorporated within circuitry 1050. Images so captured may be stored in memory 1060 and/or storage 1065. Memory 1060 may include one or more different types of media used by processor 1005, graphics hardware 1020, and image capture circuitry 1050 to perform device functions. For example, memory 1060 may include memory cache, read-only memory (ROM), and/or random access memory (RAM). Storage 1065 may store media (e.g., audio, image and video files), computer program instructions or software, preference information, device profile information, and any other suitable data. Storage 1065 may include one more non-transitory storage mediums including, for example, magnetic disks (fixed, floppy, and removable) and tape, optical media such as CD-ROMs and digital video disks (DVDs), and semiconductor memory devices such as Electrically Programmable Read-Only Memory (EPROM), and Electrically Erasable Programmable Read-Only Memory (EEPROM). Memory 1060 and storage 1065 may be used to retain computer program instructions or code organized into one or more modules and written in any desired computer programming language. When executed by, for example, processor 1005 such computer program code may implement one or more of the methods described herein.
It is to be understood that the above description is intended to be illustrative, and not restrictive. The material has been presented to enable any person skilled in the art to make and use the invention as claimed and is provided in the context of particular embodiments, variations of which will be readily apparent to those skilled in the art (e.g., some of the disclosed embodiments may be used in combination with each other). The scope of the invention therefore should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. In the appended claims, the terms “including” and “in which” are used as the plain-English equivalents of the respective terms “comprising” and “wherein.”
Zhang, Xuemei, Hordley, Steven D., Finlayson, Graham
Patent | Priority | Assignee | Title |
11386534, | May 01 2014 | Imagination Technologies Limited | Noise enhanced histograms |
11790503, | May 01 2014 | Imagination Technologies Limited | Noise enhanced histograms |
9418311, | Sep 04 2014 | Apple Inc. | Multi-scale tone mapping |
Patent | Priority | Assignee | Title |
4731671, | May 06 1985 | Eastman Kodak Company | Contrast adjustment in digital image processing method employing histogram normalization |
6717698, | Feb 02 2000 | Monument Peak Ventures, LLC | Tone scale processing based on image modulation activity |
7506135, | Jun 03 2002 | Histogram generation with vector operations in SIMD and VLIW processor by consolidating LUTs storing parallel update incremented count values for vector data elements | |
20030021488, | |||
20040057632, | |||
20060182361, | |||
20070014470, | |||
20070252834, | |||
20090100339, | |||
20100157078, | |||
20100226547, | |||
20120170842, | |||
20120188247, | |||
20130329093, | |||
20140247870, |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Jun 05 2012 | ZHANG, XUEMEI | Apple Inc | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 028331 | /0136 | |
Jun 05 2012 | HORDLEY, STEVEN D | Apple Inc | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 028331 | /0136 | |
Jun 06 2012 | Apple Inc. | (assignment on the face of the patent) | / | |||
Jun 06 2012 | FINLAYSON, GRAHAM | Apple Inc | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 028331 | /0136 |
Date | Maintenance Fee Events |
Jul 13 2015 | ASPN: Payor Number Assigned. |
Feb 07 2019 | M1551: Payment of Maintenance Fee, 4th Year, Large Entity. |
Apr 10 2023 | REM: Maintenance Fee Reminder Mailed. |
Sep 25 2023 | EXP: Patent Expired for Failure to Pay Maintenance Fees. |
Date | Maintenance Schedule |
Aug 18 2018 | 4 years fee payment window open |
Feb 18 2019 | 6 months grace period start (w surcharge) |
Aug 18 2019 | patent expiry (for year 4) |
Aug 18 2021 | 2 years to revive unintentionally abandoned end. (for year 4) |
Aug 18 2022 | 8 years fee payment window open |
Feb 18 2023 | 6 months grace period start (w surcharge) |
Aug 18 2023 | patent expiry (for year 8) |
Aug 18 2025 | 2 years to revive unintentionally abandoned end. (for year 8) |
Aug 18 2026 | 12 years fee payment window open |
Feb 18 2027 | 6 months grace period start (w surcharge) |
Aug 18 2027 | patent expiry (for year 12) |
Aug 18 2029 | 2 years to revive unintentionally abandoned end. (for year 12) |