A chemical characteristic of a material is measured using a calibration unit and container for holding a sample of the material. The calibration unit has a front portion and an imaging background positioned behind the sample and viewable through the container and the sample of the material, wherein a portion of the sample of the material is positioned in front of the imaging background so as to form a sample imaging region. A calibration strip extends along the front portion, which includes a location marker and a plurality of calibration indicia of differing colors. A computing device captures an image of the calibration strip and the sample imaging region. A calibration step is performed using the calibration strip to generate a calibration model, which is applied to the sample color vector to generate a calibrated sample color vector.
|
35. An apparatus comprising a calibration unit for use in measuring a chemical characteristic of a sample of a material in a container, the calibration unit comprising:
a front portion and a rear portion configured to hold the container, and
an imaging background extending along the rear portion so as to be positioned behind the container when mated with the calibration unit and viewable through the container and the sample of the material contained therein, wherein a portion of the sample of the material is positioned in front of the imaging background so as to form a sample imaging region, and
a calibration strip extending along the front portion, the calibration strip comprising a location marker and a plurality of calibration indicia juxtaposed in relation to the location marker, each of the plurality of calibration indicia having a different color.
1. A method for measuring a chemical characteristic of a material comprising:
providing a chemical measurement apparatus comprising, in combination, a calibration unit and a container comprising a sample of the material, the calibration unit comprising
a front portion and an imaging background positioned behind the container and viewable through the container and the sample of the material contained therein, wherein a portion of the sample of the material is positioned in front of the imaging background so as to form a sample imaging region, and
a calibration strip extending along the front portion, the calibration strip comprising a location marker and a plurality of calibration indicia juxtaposed in relation to the location marker, each of the plurality of calibration indicia having a different color;
capturing, with a computing device, an image comprising the location marker, the plurality of calibration indicia, and the sample imaging region;
performing a calibration step comprising:
analyzing the image to determine the location of the location marker within the image,
calculating, by referencing the location of the location marker, the location within the image of each of the plurality of calibration indicia,
extracting a non-standardized color value from each of the plurality of the calibration indicia,
comparing each of the extracted non-standardized color values to previously stored standardized color values for a corresponding calibration indicia, and
generating a calibration model as a function of comparing the extracted non-standardized color values to the previously stored standardized color values;
analyzing the sample imaging region from the image to extract a sample color vector;
applying the calibration model to the sample color vector to generate a calibrated sample color vector; and
analyzing, with respect to a chemical characteristic model, the calibrated sample color vector to determine the measurement of the chemical characteristic of the material.
18. A chemical measurement apparatus for measuring a chemical characteristic of a sample of a material in a container, the apparatus comprising:
a calibration unit comprising:
a front portion and a rear portion configured to hold the container, and
an imaging background extending along the rear portion so as to be positioned behind the container when mated with the calibration unit and viewable through the container and the sample of the material contained therein, wherein a portion of the sample of the material is positioned in front of the imaging background so as to form a sample imaging region, and
a calibration strip extending along the front portion, the calibration strip comprising a location marker and a plurality of calibration indicia juxtaposed in relation to the location marker, each of the plurality of calibration indicia having a different color; and
a computing device comprising
image capture means for capturing an image comprising the location marker, the plurality of calibration indicia, and the sample imaging region,
memory means for storing the image, and
processing means programmed to
perform a calibration step comprising
analyzing the image to determine the location of the location marker within the image,
calculating, by referencing the location of the location marker, the location within the image of each of the plurality of calibration indicia,
extracting a non-standardized color value from each of the plurality of the calibration indicia,
comparing each of the extracted non-standardized color values to previously stored standardized color values for a corresponding calibration indicia, and
generating a calibration model as a function of comparing the extracted non-standardized color values to the previously stored color standard;
analyze the sample imaging region from the image to extract a sample color vector;
apply the calibration model to the sample color vector to generate a calibrated sample color vector; and
analyze, with respect to a chemical characteristic model, the calibrated sample color vector to determine the measurement of the chemical characteristic of the material.
2. The method of
3. The method of
4. The method of
5. The method of
6. The method of
9. The method of
calculating, by referencing the location of the location marker, the location within the image of the coded region,
decoding the coded region to provide the printer identification,
retrieving a printer calibration model associated with the printer identification, and
adjusting the calibration model as a function of the printer calibration model.
11. The method of
12. The method of
13. The method of
the computing device receiving a manual input that identifies the chemical characteristic being measured, and
retrieving the chemical characteristic model as a function of the identification of the chemical characteristic being measured.
14. The method of
calculating, by referencing the location of the location marker, the location within the image of the coded region,
decoding the coded region to provide the chemical identification, and
retrieving the chemical characteristic model as a function of the identification of the chemical characteristic being measured.
16. The method of
17. The method of
19. The chemical measurement apparatus of
20. The chemical measurement apparatus of
21. The chemical measurement apparatus of
22. The chemical measurement apparatus of
23. The chemical measurement apparatus of
24. The chemical measurement apparatus of
26. The chemical measurement apparatus of
calculate, by referencing the location of the location marker, the location within the image of the coded region,
decode the coded region to provide the printer identification,
retrieve a printer calibration model associated with the printer identification, and
adjust the calibration model as a function of the printer calibration model.
27. The chemical measurement apparatus of
28. The chemical measurement apparatus of
29. The chemical measurement apparatus of
30. The chemical measurement apparatus of
receive a manual input that identifies the chemical characteristic being measured, and
retrieve the chemical characteristic model as a function of the identification of the chemical characteristic being measured.
31. The chemical measurement apparatus of
calculate, by referencing the location of the location marker, the location within the image of the coded region,
decode the coded region to provide the chemical identification, and
retrieve the chemical characteristic model as a function of the identification of the chemical characteristic being measured.
32. The chemical measurement apparatus of
33. The chemical measurement apparatus of
34. The chemical measurement apparatus of
36. The apparatus of
37. The apparatus of
38. The apparatus of
39. The apparatus of
40. The apparatus of
42. The apparatus of
43. The apparatus of
44. The apparatus of
46. The apparatus of
47. The apparatus of
|
The method and apparatus disclosed and claimed herein relate to an improved manner of measuring a colorimetric chemical characteristic of a material, and in particular to measuring a colorimetric chemical characteristic of a sample of the material such as water from a fish tank or a pool using a computing device such as a smartphone that captures an image of the material sample and calibrates the colors in the sampled image to account for variations in lighting and the image sensor in the smartphone, and then calculates the chemical characteristic of the calibrated image for display to a user as well as further processing and analysis.
It is often required to measure one or more chemical characteristics of a sample of a material in order to ensure that the material complies with certain standards and take appropriate steps when the material is non-compliant. For example, it is desired to test the PH and other characteristics of water in a fish tank or pool, and then take corrective action if the PH is not within tolerance. Many types of chemistry test kits are available commercially that enable a user to test the water sample, such as test strips or mixing kits in which a reagent is added to the water sample that will turn the sample a certain color based on the PH of the water. Prior art test kits require a user to compare the color of the mixed water sample to a pre-printed chart by eye, which leads to errors in interpretation. Other prior art systems have been proposed in which a smartphone or other similar handheld computing device will capture an image of the water sample and then automatically compare the color(s) of the captured image to a predefined standard. Problems arise in this situation since the color(s) of the captured image will vary based on various parameters including but not limited to lighting conditions, optical qualities of the camera lens, and the sensitivity of the camera sensor. To attempt to compensate for these variations, calibration methodologies have been proposed whereby a calibration profile is pre-computed for a specific device and applied at the time of computation. This manual or infrequent calibration introduces measurement error as it is highly likely that the profile will change from the time of computation. Additionally, these proposed prior art methods are insufficient for various reasons, such as requiring the user hold the camera a fixed distance from the water sample. Other deficiencies in the prior art exist in the spatial region identification, wherein prior art methods do not clearly address finding the location of the sample in space which is required for future processing.
The method and apparatus disclosed and claimed herein overcome the deficiencies in the prior art for measuring a chemical characteristic of a material, and in particular in measuring a chemical characteristic of a sample of the material such as water from a fish tank or a pool, by using a computing device such as a smartphone that captures an image of the material sample and calibrates the image to account for variations in lighting and the image sensor in the smartphone, and then calculates the chemical characteristic of the calibrated image for display to a user as well as further processing and analysis.
Thus, provided is a system and method for measuring a chemical characteristic of a material, for example measuring the PH of a sample of water obtained from a fish tank or a swimming pool. The present method utilizes a chemical measurement apparatus that includes the combination of a calibration unit and a container for holding the sample of the material being tested. For example, the container may be a transparent container such as a standard test tube that is provided with most commercially available fish tank water test kits, in which case the calibration unit and the test tube are provided as separate units, and the test tube is inserted into a channel in the calibration unit. Alternatively, the container and calibration unit may be provided as a single integrated unit. In either case, the calibration unit has a front portion and an imaging background that is positioned behind the sample of the material in the container and viewable through the container as well as the sample. As such, a portion of the sample of the material is positioned in front of the imaging background so as to form a sample imaging region. The imaging background provides a standard reference color to prevent colored light from the environment from changing the color of the imaging region. Additionally, the imaging background may block ballistic light from the environment to the sides of the sample. A calibration strip extends along the front portion of the calibration unit. The calibration strip includes a location marker and a plurality of calibration indicia that are each juxtaposed in relation to the location marker, wherein each of the plurality of calibration indicia is printed in a different color. For example, an RGB color space of red, green and blue may be used. The location marker includes any regions or marks which allow for spatial localization of the calibration unit. All other markers and imaging indicia are then located by some relation to the location marker.
After the sample of the material is placed into the container, a computing device such as a smartphone is used to capture an image of the calibration strip (the location marker and the plurality of calibration indicia) and the sample imaging region. Then, the smartphone executes a calibration step in order to adjust for variations in the captured image from what would otherwise be considered to be a true or standardized value (or calibrated color space), such as variations due to lighting as well as imperfections in the camera lens and/or sensor in the smartphone. The calibration step is performed by first analyzing the captured image to determine the location of the location marker within the image. This will provide a spatial reference which is used to calculate the location within the image of each of the plurality of calibration indicia. Next, a non-standardized (i.e. raw or uncalibrated) color value is extracted from each of the plurality of the calibration indicia, and then each non-standardized color value is compared to a previously stored standardized color value for a corresponding indicia. The offset between the non-standardized (raw) color value and a standardized color value for each indicia is then used to compute a calibration model, which may for example be a table of RGB values or a mathematical algorithm (e.g. regression). Thus, this step maps the entire non-standardized color space to a calibrated color space.
The next step is to locate in the captured image the imaging region(s) relative to the location marker. The color information in the imaging region is then adjusted from a non-standardized color space to a calibrated color space by computation with the calibration model constructed in the prior step. A second mapping is then computed from the calibrated color space to a chemical characteristic value via a chemical characteristic model. This model may take the form of a pre-computed regression model or stored table values and may have been obtained through theory or empirical derivation. For example, once the calibrated sample color value is computed in a water PH test, that color value can be analyzed to determine the PH of the water, if that PH is within a certain range, etc.
In an exemplary embodiment, the location marker on the calibration strip may include a pair of location targets, i.e. a first location target and a second location target. In this embodiment, the plurality of calibration indicia are located in a region between the first location target and the second location target. Thus, as long as the location of the calibration indicia with respect to the location marker is known, then the processing software is able to determine where in the image the calibration indicia may be found once the location of the location marker is determined.
The location targets preferably are printed as a black target on a white background, so that appropriate techniques such as edge detection routines may be used to ascertain the location of the targets in the captured image. In this case, the color value(s) of the location targets may also be included in the set of non-standardized color values that are extracted from the calibration indicia. For example, when the calibration indicia include a green calibration indicia, a blue calibration indicia, and a red calibration indicia, the captured location target(s) and the background color would then contribute the color black and white respectively to that set, which further increases the accuracy of the calibration step.
Optionally, a coded region may be juxtaposed in relation to the location marker and captured in the image by the smartphone. The coded region may be encoded with a printer identification and/or a chemical identification. The printer identification will indicate which printer was used to print the calibration strip during the manufacturing phase of the calibration unit, and will assist the calibration step in accounting for variations that may have existed during that printing step. The chemical identification will act to inform the processing software which chemical is being measured (e.g. PH), so that the appropriate chemical characteristic model may be used in the measurement step. Thus, when the coded region is implemented, the location of the coded region within the image is calculated by referencing the location of the location marker (in the same manner as calculating the location of the calibration indicia). The coded region is then decoded to provide the printer identification and/or the chemical identification. When the printer identification is decoded, then a printer calibration model associated with the printer identification is retrieved (e.g. from local memory or a resource located externally on the internet), and the calibration model is adjusted as a function of the printer calibration model. When the chemical identification is decoded, then that is used to retrieve the appropriate chemical characteristic model. In the event that the chemical identification is not encoded into the coded region, then the user would manually input the chemical identification into the smartphone such as by selection from a menu on the display, etc.
The coded region may be located on the calibration strip near the location marker and/or the calibration indicia. In an alternative embodiment, a cap may be provided that is suitable to cover the test tube container, and the coded region may be put on the cap such that the image will include the cap. This will allow the use of a single calibration unit and a different cap for each chemical that is being measured. In the embodiment using a cap, one of the location targets may also be located on the cap.
In an optional embodiment, the sample imaging background is not required. In this embodiment, a chemical measurement apparatus has a calibration unit with a calibration strip extending along a front portion of the calibration unit. The calibration strip includes a location marker and a plurality of calibration indicia juxtaposed in relation to the location marker, wherein each of the plurality of calibration indicia has a different color. A sample of the material is placed in close proximity to the calibration strip so as to form a sample imaging region. A computing device captures an image including the location marker, the plurality of calibration indicia, and the sample imaging region. The calibration steps are then performed, including analyzing the image to determine the location of the location marker within the image; calculating, by referencing the location of the location marker, the location within the image of each of the plurality of calibration indicia; extracting a non-standardized color value from each of the plurality of the calibration indicia; comparing the extracted non-standardized color values to a previously stored color standard; and generating a calibration model as a function of comparing the extracted non-standardized color values to the previously stored color standard. Next, the sample imaging region is analyzed from the image to extract a sample color vector (which may include or or more sample color values), and the calibration model is applied to the sample color vector to generate a calibrated sample color vector. Finally, the calibrated sample color vector is analyzed with respect to a chemical characteristic model in order to determine the measurement of the chemical characteristic of the material.
Referring to
In the illustrated embodiment, a channel is formed by the front portion 102 and the rear portion 106 so that the container (test tube) 110 may be placed therein as shown by the arrow. A sample 112 of the material being tested, in this case water from a fish tank, is poured into the container 110 so that the assembled device appears as in
A calibration strip 202 extends along the front portion 102 of the calibration unit 100. As shown in further detail in
The location targets 604, 606 preferably are printed as a black target on a white background, so that proper edge detection routines may be used to ascertain the location of the targets in the captured image. If desired, the color value(s) of the location targets 604, 606 may also be included in the set of non-standardized color values that are extracted from the calibration indicia 612, 614, 616. Thus, in this embodiment, the captured location targets 604, 606 would then contribute the colors black and white to the set of calibration indicia that include the colors red, green and blue, which further increases the accuracy of the calibration step. Of course, other color spaces may be used if desired.
The calibration strip 202 may be a label that is printed and affixed onto the front portion 102 of the calibration unit 100 during a manufacturing process, as well known in the art. In the alternative, the location targets and calibration indicia may be printed directly onto the front portion 102 without using a separate label or other substrate, if desired.
Also shown in
The coded region on calibration strip 202 may also include a text area 610 that indicates the identification of the chemical or property/parameter that is being tested. Thus, as shown in
Additionally, the chemical identification may be encoded into the code 608 in addition to being printed in the text area 610. In this embodiment, the code 608 is located within the image and decoded to provide the chemical identification, instead of requiring the user to manually input the chemical identification as described above.
Thus, when this coded region is implemented, the location of the code 608 in the coded region within the image is calculated by referencing the location of the location marker (in the same manner as calculating the location of the calibration indicia). The code 608 is then decoded to provide the printer identification and/or the chemical identification. When the printer identification is decoded, then a printer calibration model associated with the printer identification is retrieved, and the calibration model is adjusted as a function of the printer calibration model. When the chemical identification is decoded, then that is used to retrieve the appropriate chemical characteristic model instead of requiring the user to manually input the chemical identification.
Of course, virtually any type of chemical characteristic may be measured as described herein, as may be desired. By way of a non-limiting example,
In an alternative embodiment as shown in
Also shown in
In addition to the primary embodiment in which the calibration unit 100 is used as a base into which the container is inserted as shown in
The main components of the smartphone 404 that are utilized in this embodiment are shown in
With reference to the flowcharts of
After the image 406 is captured by the smartphone, a calibration process is executed by the smartphone at step 810. With further reference to
Once the locations of the location targets are established, then the processing software is able to calculate the locations of the various calibration indicia at step 904. That is, the spatial relationship between the location targets 604, 606 and the calibration indicia 612, 614, 616 is stored in memory and used in this step. At step 906, a non-standardized color value is extracted for each of the calibration indicia 612, 614, 616 (e.g. green, blue and red). Then, at step 908, these extracted non-standardized color values are each compared to a previously-stored set of standardized color values. The offset between the non-standardized color taken from the imaged calibration indicia and a standardized value for each color is then used to compute a calibration model at step 910 (for example a table of RGB values or a mathematical algorithm (e.g. regression)). This step maps the entire non-standardized color space to a calibrated color space. The calibration model represents a shift of the non-standardized color values that will take place in order to adjust for the variations described above.
Referring back to
The specific processing algorithms implemented in executing the calibration and standardization routines are well known in the art and need not be repeated here. For example, reference is made to Yetisen et al., A Smartphone algorithm with inter-phone repeatability for the analysis of colorimetric tests, from Sensors and Actuators B: Chemical, pp 156-160, 2014, http://www.elsevier.com/locate/snb, which is incorporated by reference herein. Other algorithms for performing the calibration and standardization routines may also be used if desired.
At step 824, a chemical characteristic model for the material being measured is retrieved from memory. For example, if PH is being tested, then the chemical characteristic model for PH is retrieved. With reference to
As explained above, the coded region on calibration strip 202 may also include a text area 610 that indicates the identification of the chemical characteristic that is being tested. As shown in
In another embodiment, in order to avoid the manual entry of the characteristic identification, the chemical identification may be encoded into the code 608. In this embodiment, the code 608 is located within the image and decoded to provide the chemical identification, instead of requiring the user to manually input the chemical identification as described above.
Thus, when this coded region is implemented, at step 1104 the location of the code 608 in the coded region within the image is calculated by referencing the location of the location marker in the same manner as calculating the location of the calibration indicia. The code 608 is then decoded at step 1106 to provide the chemical identification, and at step 1108 the associated chemical characteristic model is retrieved from memory at step 1108 instead of requiring the user to manually input the chemical identification.
Once the chemical characteristic model has been retrieved from memory, the analysis of step 816 occurs. There, the calibrated sample color vector is analyzed with respect to the retrieved chemical characteristic model. For example, the analysis may determine that the calibrated color vector of the sample indicates that the PH of the sample is exceedingly low. At step 820 those analysis results may be displayed on the screen for the user to view. Optionally, at step 822, the analysis results may be stored in memory on the smartphone or transmitted wirelessly to an external store. The entire process may be repeated, for additional chemical characteristics other than PH, as shown at step 818.
Other features may be provided in alternate embodiments. For example, an illumination source, such as an LED light(s), may be integrated with the calibration unit 100 such that it provides illumination to the sample imaging region 204. Optionally, the imaging described above could then be done in a dark environment, so that the only light incident on the sample imaging region comes from the LED. This provides a means for controlling the light source and obtaining consistent results.
In another embodiment, the container 110 may be integrated with the calibration unit 100 instead of providing separate units as shown in
As mentioned above, in addition to the primary embodiment in which the calibration unit 100 is used as a base into which the container is inserted as shown in
In another embodiment, it may not be necessary to use an imaging background or a container for the sample being measured. For example, a material other than a liquid may be measured that does not require a container, such as a sample of soil or a solid object. In that case, the sample would only need to rest on a platform in the vicinity of the calibration strip, such as on a platform provided instead of the container. In this embodiment, it may not be necessary to utilize the imaging background. Thus, a chemical measurement apparatus may be provided with a calibration unit having a calibration strip extending along a front portion of the calibration unit. The calibration strip includes a location marker and a plurality of calibration indicia juxtaposed in relation to the location marker, wherein each of the plurality of calibration indicia has a different color. A sample of the material is placed in close proximity to the calibration strip so as to form a sample imaging region. A computing device captures an image including the location marker, the plurality of calibration indicia, and the sample imaging region. The processing steps are then performed as described above in order to calibrate the non-standardized color values and generate the calibration model, and then apply the calibration model to the sample color vector to generate a calibrated sample color vector and analyze that calibrated sample color vector with respect to a chemical characteristic model to determine the measurement of the chemical characteristic of the material.
In a further embodiment, a test tube cap may be adapted to provide a stem that extends into the water sample, with a small imaging background attached to the end of the stem such that the imaging background is submerged within the water sample. With this submerged imaging background juxtaposed in the vicinity of the calibration strip as described above, the imaging, calibration, and measurement steps disclosed herein may also be performed.
While the above is a complete description of selected embodiments, it is possible to use various alternatives, modifications, combinations and equivalents. In general, in the following claims, the terms used should be construed to include all possible embodiments along with the full scope of equivalents to which such claims are entitled. Accordingly, the claims are not limited by the disclosure.
Patent | Priority | Assignee | Title |
11035733, | Sep 28 2018 | Imam Abdulrahman Bin Faisal University | Method for chromatic calibration of an embedded camera of a mobile wireless device |
11988596, | Oct 23 2020 | Becton, Dickinson and Company | Systems and methods for imaging and image-based analysis of test devices |
11996183, | Sep 17 2020 | SCANWELL HEALTH, INC | Methods of analyzing diagnostic test kits |
D970033, | Oct 23 2020 | Becton, Dickinson and Company | Cartridge imaging background device |
Patent | Priority | Assignee | Title |
4682895, | Aug 06 1985 | MCKINLEY, BRUCE D ; BUTLER, BRUCE D | Fiber optic probe for quantification of colorimetric reactions |
5201755, | Sep 11 1990 | Datascope Investment Corp | Method and apparatus for early detection of leakage and failure of a balloon membrane of a balloon catheter |
5583666, | Feb 12 1993 | Intellectual Ventures Fund 83 LLC | Method for cross-device color calibration and enhancement using explicit constraints |
6016161, | Jan 25 1996 | MICROSCAN SYSTEMS, INC | Method and system for automatically calibrating a color-based machine vision system |
6249593, | Jun 05 1995 | E-Y LABORATORIES, INC | Optical specimen analysis system and method |
8848988, | Nov 14 2008 | CHEMBIO DIAGNOSTICS HOLDINGS GERMANY GMBH | Appliance and method for evaluation and assessment of a test strip |
8877140, | Nov 03 2010 | SCANWELL HEALTH, INC | Color-based reaction testing of biological materials |
8911679, | Nov 03 2010 | SCANWELL HEALTH, INC | Color-based reaction testing of biological materials |
9129350, | Jul 20 2010 | CORTEVA AGRISCIENCE LLC | Systems and methods to analyze an immunoassay test strip comb member |
9285323, | Aug 08 2012 | HEALTHY IO LTD | Quantifying color changes of chemical test pads induced concentrations of biological analytes under different lighting conditions |
9311520, | Aug 08 2012 | HEALTHY IO LTD | Method and apparatus for performing and quantifying color changes induced by specific concentrations of biological analytes in an automatically calibrated environment |
9445749, | Jan 21 2013 | Cornell University | Smartphone-based apparatus and method for obtaining repeatable, quantitative colorimetric measurement |
9528941, | Aug 08 2012 | HEALTHY IO LTD | Method and apparatus for determining analyte concentration by quantifying and interpreting color information captured in a continuous or periodic manner |
9607380, | Aug 08 2012 | HEALTHY IO LTD | Methods and apparatus for quantifying color changes induced by specific concentrations of biological analytes |
20060257285, | |||
20070161103, | |||
20150055134, | |||
20150359458, | |||
20160080548, | |||
20160266157, |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Apr 20 2016 | Chroma Fish Corp. | (assignment on the face of the patent) | / | |||
Apr 22 2016 | RUTTNER, GABRIEL T | CHROMA FISH CORP | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 038369 | /0140 |
Date | Maintenance Fee Events |
Feb 15 2021 | REM: Maintenance Fee Reminder Mailed. |
Aug 02 2021 | EXP: Patent Expired for Failure to Pay Maintenance Fees. |
Date | Maintenance Schedule |
Jun 27 2020 | 4 years fee payment window open |
Dec 27 2020 | 6 months grace period start (w surcharge) |
Jun 27 2021 | patent expiry (for year 4) |
Jun 27 2023 | 2 years to revive unintentionally abandoned end. (for year 4) |
Jun 27 2024 | 8 years fee payment window open |
Dec 27 2024 | 6 months grace period start (w surcharge) |
Jun 27 2025 | patent expiry (for year 8) |
Jun 27 2027 | 2 years to revive unintentionally abandoned end. (for year 8) |
Jun 27 2028 | 12 years fee payment window open |
Dec 27 2028 | 6 months grace period start (w surcharge) |
Jun 27 2029 | patent expiry (for year 12) |
Jun 27 2031 | 2 years to revive unintentionally abandoned end. (for year 12) |