A vision measuring device includes: an imaging device which images a workpiece; an illumination device which irradiates the workpiece with light; a position control system which controls an in-focus position of the imaging device and outputs the in-focus position as information representing a position in an in-focus axis direction; and a control device which, when the position control system controls the in-focus position, controls the frame rate of the imaging device, and controls at least one of the imaging device and the illumination device such that an amount of light to be received by the imaging device becomes substantially constant.
|
11. A vision measuring device, comprising:
an imaging device which images a workpiece, and of which frame rate is variable;
an illumination device which irradiates the workpiece with light;
a position control system which controls an in-focus position of the imaging device and outputs the in-focus position as information representing a position in an in-focus axis direction; and
a control device which, when the position control system controls the in-focus position, controls the frame rate of the imaging device and adjusts a light volume of the illumination device in accordance with the frame rate of the imaging device.
1. A vision measuring device, comprising:
an imaging device which images a workpiece;
an illumination device which irradiates the workpiece with light;
a position control system which controls an in-focus position of the imaging device and outputs the in-focus position as information representing a position in an in-focus axis direction; and
a control device which, when the position control system controls the in-focus position, controls a frame rate of the imaging device, and also controls at least one of the imaging device and the illumination device such that an amount of light to be received by the imaging device becomes substantially constant.
3. A vision measuring device, comprising:
an imaging device which imaging a workpiece, and of which exposure time is variable;
an illumination device which is controlled to emit light at a certain emission frequency and irradiate the workpiece with illumination light;
a position control system which controls an in-focus position of the imaging device and outputs the in-focus position as information representing a position in an in-focus axis direction; and
a control device which, when the position control system controls the in-focus position, sets the exposure time of the imaging device to an integral multiple of a lighting period of an interior lamp based on which of a first frequency and a second frequency a lighting frequency of the interior lamp is,
wherein the emission frequency of the illumination device is set to a common multiple of the first frequency and the second frequency.
2. The vision measuring device according to
wherein the control device controls at least one of the imaging device and the illumination device in response to a signal from the imaging device.
4. The vision measuring device according to
wherein the control device detects the lighting frequency of the interior lamp when the illumination device is turned off.
5. The vision measuring device according to
wherein the control device controls the position control system both in a case when the exposure time of the imaging device is set to an integral multiple of a lighting period corresponding to the first frequency and a case when the exposure time of the imaging device is set to an integral multiple of a lighting period corresponding to the second frequency, such that the imaging device performs imaging at a plurality of different in-focus positions corresponding to these cases,
the control device calculates a fluctuation rate of average brightness of acquired images, and
the control device sets the exposure time of the imaging device to an integral multiple of the lighting period corresponding to either the first frequency or the second frequency that corresponds to a lower fluctuation rate of average brightness.
6. The vision measuring device according to
wherein the first frequency is 100 Hz, and the second frequency is 120 Hz.
7. The vision measuring device according to
wherein the emission frequency of the illumination device is an integral multiple of 600 Hz.
8. The vision measuring device according to
wherein the control device changes the in-focus position of the imaging device after three or more images of the workpiece are acquired.
9. The vision measuring device according to
wherein the illumination device receives a vertical synchronization signal from the imaging device, and outputs a certain number of pulses within the exposure time of the imaging device.
10. The vision measuring device according to
wherein the control device sets a long exposure time when the workpiece is irradiated with external diffused light other than the interior lamp.
12. The vision measuring device according to
wherein when controlling the in-focus position of the imaging device, the control device receives a portion of an image of an imaging range of the imaging device, increases the frame rate of the imaging device, and increases the light volume of the illumination device.
13. The vision measuring device according to
wherein the control device adjusts the light volume of the illumination device in response to a vertical synchronization signal output by the imaging device.
14. The vision measuring device according to
wherein when controlling the in-focus position of the imaging device, the control device adjusts the light volume of the illumination device in accordance with a first frame rate calculated from an interval of the vertical synchronization signal output by the imaging device and a second frame rate used in a normal measurement.
15. The vision measuring device according to
wherein the control device adjusts a gain of the imaging device in accordance with the frame rate of the imaging device.
16. The vision measuring device according to
wherein the control device obtains the in-focus position in response to the vertical synchronization signal.
17. The vision measuring device according to
wherein the control device adjusts the light volume of the illumination device by using illuminance defined by a color, a reflectivity, etc. of a surface of the workpiece.
18. The vision measuring device according to
a frame rate detecting device which detects the frame rate of the imaging device; and
a standard frame rate retaining unit which retains the frame rate of the imaging device during vision measurement.
19. The vision measuring device according to
wherein the frame rate detecting device and the frame rate retaining unit are implemented by a program such as firmware.
20. The vision measuring device according to
wherein the control device adjusts the gain of the imaging device when an output of the illumination device exceeds a certain value.
|
This application is based on and claims the benefit of priority from prior Japanese Patent Application No. 2011-9513, filed on Jan. 20, 2011, and from prior Japanese Patent Application No. 2011-14899, filed on Jan. 27, 2011, the entire contents of which are incorporated herein by reference.
1. Field of the Invention
The present invention relates to a vision measuring device which measures a measuring target by imaging the measuring target.
2. Description of the Related Art
A vision measuring device having an auto-focusing function sequentially acquires images of a measuring target by moving an imaging device such as a camera or its optical system in the light axis direction, and defines the position in the light axis direction at which an image with the highest contrast is acquired as the in-focus position for the measuring target (JP2009-168607A).
The auto-focusing accuracy and speed of such a vision measuring device having a contrast-type auto-focusing function are more or less influenced by the frame rate, etc. in the auto-focusing operation.
For example, such a vision measuring device might acquire images with uneven brightness due to flickers which might be caused depending on the combination of the frame rate of the imaging device and the emission frequency of an illumination unit that illuminates the imaging range. In this case, the contrast of an image with poor brightness becomes low regardless of whether the image is acquired at an in-focus position or not because such an image has a small brightness difference between the pixels, whereas the contrast of an image with affluent brightness becomes high. Hence, there is a problem that it is difficult to judge the in-focus position based on the contrast, which reduces the reliability of auto-focusing.
In order to overcome this problem, a vision measuring device commonly has an illumination unit that does not cause flickers depending on its combination with the imaging device. However, also in this case, there may be many occasions when it is difficult to eliminate the influence of external diffused light depending on the configuration of the vision measuring device. A fluorescent tube (hereinafter, referred to as interior lamp) is an example of external diffused light that is difficult to eliminate and tends to be problematic. An interior lamp repeats blinking at 100 Hz in a district where the power source frequency is 50 Hz and at 120 Hz in a district where the power source frequency is 60 Hz, and gives an adverse effect to a measurement result and an auto-focusing result. There is also a case when an illumination unit itself flickers.
Moreover, although such a vision measuring device can be easily realized by a camera and software, there is a problem that the device takes a long time for the auto-focusing process. In order to solve this problem, it is conceivable to move the camera and the optical system at a high speed. However, this makes the image acquiring pitch during auto-focusing rough because the frame rate is constant, and makes it difficult to obtain a correct in-focus position.
The present invention was made in view of these matters, and an object of the present invention is to provide a vision measuring device capable of an auto-focusing process that is highly accurate and high-speed.
A vision measuring device according to one embodiment of the present invention includes: an imaging device which images a workpiece; an illumination device which irradiates the workpiece with light; a position control system which controls an in-focus position of the imaging device and outputs the in-focus position as information representing a position in an in-focus axis direction; and a control device which, when the position control system controls the in-focus position, controls a frame rate of the imaging device, and also controls at least one of the imaging device and the illumination device such that an amount of light to be received by the imaging device becomes substantially constant.
In a vision measuring device according to another embodiment of the present invention, the control device may control at least one of the imaging device and the illumination device in response to a signal from the imaging device.
A vision measuring device according to another embodiment of the present invention includes: an imaging device which imaging a workpiece, and of which exposure time is variable; an illumination device which is controlled to emit light at a certain emission frequency and irradiate the workpiece with illumination light; a position control system which controls an in-focus position of the imaging device and outputs the in-focus position as information representing a position in an in-focus axis direction; and a control device which, when the position control system controls the in-focus position, sets the exposure time of the imaging device to an integral multiple of a lighting period of an interior lamp based on which of a first frequency and a second frequency a lighting frequency of the interior lamp is, wherein the emission frequency of the illumination device is set to a common multiple of the first frequency and the second frequency.
Such a configuration enables a highly accurate auto-focusing process by eliminating the influence of external diffused light that repeats blinking at a certain period and acquiring images with even brightness. Furthermore, such a configuration can be used commonly in districts with different power source frequencies.
In a vision measuring device according to another embodiment of the present invention, the control device may detect the lighting frequency of the interior lamp when the illumination device is turned off.
In a vision measuring device according to another embodiment of the present invention, the control device may control the position control system both in a case when the exposure time of the imaging device is set to an integral multiple of a lighting period corresponding to the first frequency and a case when the exposure time of the imaging device is set to an integral multiple of a lighting period corresponding to the second frequency, such that the imaging device performs imaging at a plurality of different in-focus positions corresponding to these cases, may calculate a fluctuation rate of average brightness of acquired images, and may set the exposure time of the imaging device to an integral multiple of the lighting period corresponding to either the first frequency or the second frequency that corresponds to a lower fluctuation rate of average brightness.
Such a configuration makes it possible to set the frame rate of the imaging device and the emission frequency of the illumination device without requiring a complicated operation, enabling a highly accurate auto-focusing process to be performed easily in any district.
In a vision measuring device according to another embodiment of the present invention, the first frequency may be 100 Hz, and the second frequency may be 120 Hz.
In a vision measuring device according to another embodiment of the present invention, the emission frequency of the illumination device may be an integral multiple of 600 Hz.
In a vision measuring device according to another embodiment of the present invention, the control device may change the in-focus position of the imaging device after three or more images of the workpiece are acquired.
In a vision measuring device according to another embodiment of the present invention, the illumination device may receive a vertical synchronization signal from the imaging device, and output a certain number of pulses within the exposure time of the imaging device.
In a vision measuring device according to another embodiment of the present invention, the control device may set a long exposure time when the workpiece is irradiated with external diffused light other than the interior lamp.
A vision measuring device according to another embodiment of the present invention includes: an imaging device which images a workpiece, and of which frame rate is variable; an illumination device which irradiates the workplace with light; a position control system which controls an in-focus position of the imaging device and outputs the in-focus position as information representing a position in an in-focus axis direction; and a control device which, when the position control system controls the in-focus position, controls the frame rate of the imaging device and adjusts a light volume of the illumination device in accordance with the frame rate of the imaging device.
In a vision measuring device according to another embodiment of the present invention, when controlling the in-focus position of the imaging device, the control device may receive a portion of an image of an imaging range of the imaging device, increase the frame rate of the imaging device, and increase the light volume of the illumination device.
In a vision measuring device according to another embodiment of the present invention, the control device may adjust the light volume of the illumination device in response to a vertical synchronization signal output by the imaging device.
In a vision measuring device according to another embodiment of the present invention, when controlling the in-focus position of the imaging device, the control device may adjust the light volume of the illumination device in accordance with a first frame rate calculated from an interval of the vertical synchronization signal output by the imaging device and a second frame rate used in a normal measurement.
In a vision measuring device according to another embodiment of the present invention, the control device may adjust a gain of the imaging device in accordance with the frame rate of the imaging device.
In a vision measuring device according to another embodiment of the present invention, the control device may obtain the in-focus position in response to the vertical synchronization signal.
In a vision measuring device according to another embodiment of the present invention, the control device may adjust the light volume of the illumination device by using illuminance defined by a color, a reflectivity, etc. of a surface of the workpiece.
A vision measuring device according to another embodiment of the present invention may further include: a frame rate detecting device which detects the frame rate of the imaging device; and a standard frame rate retaining unit which retains the frame rate of the imaging device during vision measurement.
In a vision measuring device according to another embodiment of the present invention, the frame rate detecting device and the frame rate retaining unit may be implemented by a program.
In a vision measuring device according to another embodiment of the present invention, the control device may adjust the gain of the imaging device when an output of the illumination device exceeds a certain value.
Next, the configuration of a vision measuring device according to the first embodiment of the present invention will be explained in detail with reference to the drawings.
The vision measuring machine 1 is configured as follows. Namely, a sample holder 12 is placed on a sample moving unit 11 such that the top surface of the sample holder 12, which is a base surface, becomes a horizontal plane. An X-axis guide 13c is supported on the top ends of arm supports 13a and 13b that stand on both side ends of the sample moving means 11. The sample holder 12 is driven in a Y-axis direction by the sample moving means 11. An imaging unit 14 is supported on the X-axis guide 13c such that it can be driven in the X-axis direction. A camera 141 is mounted on the lower end of the imaging unit 14.
Though the present embodiment employs a manner of imaging the workpiece 3 that is positioned on the sample holder 12, needless to say, it may employ any other manner, and for example, may employ a manner of imaging a workpiece placed on the floor from a lateral side. The camera 141 may be a camera of various types such as a CCD, a CMOS, etc.
Next, the configuration of the imaging unit 14 of the vision measuring device according to the present embodiment will be explained.
The position control system 151 outputs a Z-axis drive instruction to the power unit 16. The power unit 16 feeds a driving power to the Z-axis motor 145, and the Z-axis motor 145 drives the camera 141 by means of the camera driving mechanism 144. The camera 141 performs imaging at an arbitrary frame rate, and sends image information to the PC 2 through the USB interface. At this time, the camera 141 may output a vertical synchronization signal to the position control system 151 as a trigger signal. In this case, the position control system 151 receives the vertical synchronization signal, and in response to this, acquires the Z-coordinate of the camera 141 from the linear encoder 143. The acquired Z-coordinate is retained in the Z-value latch buffer, and the latch counter is updated. The retained Z-value is sent to the PC 2 in response to a readout instruction. In the present embodiment, the camera 141 is driven in the Z-axis direction, but a similar operation is available also by adjusting the optical system provided in the camera 141 such as a lens, etc. In the present embodiment, a USB interface is used as a digital serial communication means, but needless to say, any other means such as Gig-E, FireWire, etc. may also be used.
Next, an auto-focusing method of the vision measuring device according to the present embodiment will be explained.
In the auto-focusing by the vision measuring device according to the present embodiment, imaging is performed at a plurality of Z-coordinates, contrasts are calculated from the images acquired at the respective coordinate positions, and the Z-coordinate at which an image with the highest value among the plurality of calculated contrasts is acquired is judged as the in-focus position. In the example of
Next, the problems of the conventional contrast-type auto-focusing will be explained. As described above, as regards such a vision measuring device, there is a case when the workplace 3 is irradiated not only by the light from the illumination device 142 but also by external diffused light. In many cases, such a vision measuring device is used indoors, and the external diffused light is light from an interior lamp. The interior lamp repeats blinking at a constant period, and might generate a flicker in the images to be acquired depending on the relationship between the blinking period and the frame rate of the camera 141.
The problem of the flicker will be explained with reference to
As shown in the upper row of
As described above, when the power source frequency is 60 Hz, the lighting frequency of an interior lamp is 120 Hz. Accordingly, when the exposure time is set to 1/60=16.7 ms, the amount of light to be received by the camera 141 from the start of the light reception till the end thereof corresponds to two periods of the interior lamp. Hence, even if the lighting period of the interior lamp and the frame period go disagreeing, the images to be picked up by the camera 141 will all have the same brightness.
As compared with this, when the power source frequency is 50 Hz, i.e., when the lighting frequency of the interior lamp is 100 Hz, the interior lamp repeats blinking at a period of 10 msec. Therefore, as shown in the lower row of
As described above, the contrast-type auto-focusing calculates a contrast from an image acquired by the camera 141. Hence, when a flicker is generated in an image acquired by the camera 141, it is difficult to calculate the contrast correctly.
To solve this problem, the present invention employs adjusting the exposure time of the camera 141 according to the lighting frequency of the interior lamp. This adjusting method is shown in
Next, a method of automatically detecting the lighting frequency of the interior lamp by using software will be explained.
Next, the operation of the illumination device 142 of the vision measuring device according to the present embodiment will be explained. In the present embodiment, a PWM-controlled LED is used as the illumination device 142, and the PWM frequency is set to an integral multiple of 600 Hz which is the least common multiple of 120 and 100. The reason of this will be explained below.
When the PWM frequency is 480 Hz and the frame rate of the camera 141 is 60 fps as shown in the upper row of
As compared with this, the vision measuring device according to the present embodiment sets the emission frequency of the illumination device 142 to a multiple of 600 Hz.
With such a configuration, it becomes possible to acquire appropriate images with no flicker even when the frame rate of the camera 141 changes. When the illumination is turned on by a switching-type power circuit, ripples occur due to the switching, in principle. The illumination also flickers due to the ripples, and a flicker might be generated in the input images. However, this problem can be overcome by the configuration of the present embodiment.
Since the frame rate of the camera is selected from between 100 fps and 120 fps in the present embodiment, the emission frequency of the illumination device 142 is set to 600 Hz. However, it is only necessary that the emission frequency be a common multiple of the plurality of frame rates to select from. For example, when selecting the frame rate of the camera from between 50 fps and 60 fps, it is possible to set the emission frequency to 300 Hz which is the least common multiple of these frame rates. When setting the frame rate of the camera 141 to any other values than 50 fps and 60 fps, it is possible to adjust the emission frequency of the illumination device 142 automatically to an integral multiple of the frame rates of the camera 141. Moreover, it is also possible to make the illumination device 142 receive a vertical synchronization signal from the camera 141 and emit light by only a fixed number of pulses within the exposure time.
Furthermore, for example, when controlling a minute current or voltage by a switching-type power circuit, there is a method for, by thinning the switching, improving the efficiency or making it seem like a more minute current or voltage than can be controlled by the actual ability of the switching control element is controlled. Also in such a case, setting a constant thinning period and setting the thinning frequency to a common multiple of the frame rates makes it possible to ensure that no flicker is generated in the input images.
Next, the configuration of the vision measuring device according to the second embodiment of the present invention will be explained in detail with reference to the drawings.
The vision measuring device according to the second embodiment of the present invention is the same as the first embodiment except the following point. As shown in
The contrast-type auto-focusing employed by the vision measuring device according to the present embodiment can figure out the correct in-focus position by increasing image output positions. However, if the image output positions are increased, the amount of data to be sent from the camera 141 to the PC 2 is increased. In the present embodiment, because the camera 141 and the PC 2 are connected through a USB interface, the image data transfer speed is limited to approximately 400 Mbps, which increases the time taken for auto-focusing. Hence, the vision measuring device according to the present embodiment reduces the amount of data to be sent from the camera 141 to the PC 2 by sending only a partial image that is cut out from the image of the imaging range during auto-focusing, thereby raising the frame rate.
This auto-focusing process will be explained with reference to
In the live display state before auto-focusing is started, the image data representing the whole imaging range is sent from the camera 141 to the PC 2 as shown in the lower left of
At the timing S2, an instruction designating an image output range is sent from the PC 2 to the camera 141. In this way, the range of the image to be sent from the camera 141 to the PC 2 is restricted as shown in the picture in the lower center of
At the timing S4 at which auto-focusing ends, an instruction to the effect that image output should be stopped is sent from the PC 2 to the camera 141. Then, at the timing 85, a signal to the effect that the settings of the camera 141 during the auto-focusing (the designation of the image output range and output of a vertical synchronization signal) should be canceled is sent. A Z-move stop instruction, a latch stop instruction, and a latch number readout instruction are sent from the PC 2 to the position control system 151. The position control system 151 stops the camera 141 from moving, stops the operation of the latch counter and Z-value latch buffer, and sends the latch number to the PC 2. Then, a latch data readout instruction is output from the PC 2 to the position control system 151, and the data (Z-coordinate and time stamp) in the Z-value latch buffer is sent from the position control system 151 to the PC 2. The PC 2 associates the image data with the Z-coordinate based on their time stamps, and figures out the relationship between the contrast calculated from the image data and the Z-value. After this, judging that a Z-value at which the image with the highest contrast was acquired as the in-focus position, the PC 2 causes the camera 141 to be moved such that its Z-coordinate becomes the calculated in-focus position.
Finally, when an instruction to the effect that image output by live display should be resumed is output at the timing S6, the auto-focusing operation is completed, and transfer of normal image data representing the whole imaging range is resumed. At this time, the image to be sent from the camera 141 to the PC 2 has the same size as before the auto-focusing was started, as shown in the lower right of
According to this method, the size of the image to be sent from the camera 141 to the PC 2 is reduced, and the frame rate of the camera 141 can be increased significantly regardless of the transfer rate of the USB interface. However, this method has to reduce the exposure time per frame, which results in reduction of the amount of light in the acquired image. Hence, the contrast and the auto-focusing accuracy become lower.
To solve this problem, the present embodiment turns up the light volume of the illumination device 142 during auto-focusing. This illumination control will be explained with reference to
The illumination control device 152 and the camera 141 are connected in parallel to the PC 2, and the illumination device 142 is connected to the illumination control device 152. When auto-focusing is started, an instruction to the effect that image output should be stopped is output at the timing S1 of
By increasing the light volume of the illumination device 142 in accordance with an increase in the frame rate of the camera 141, it becomes possible to compensate for the reduction in the amount of light due to the reduction of the exposure time, and to prevent the contrast from being lowered. Hence, it becomes possible to increase the number of frames of image information to be acquired per unit time significantly while maintaining the accuracy of contrast, which enables highly accurate and high-speed auto-focusing. It is possible to realize such a configuration easily and at a low cost by software only, by diverting existing PC and vision measuring machine. It is also possible to make the PC 2 calculate the average values of brightness, contrast, etc. of the acquired images and control the illumination control device 152 based on these values.
Next, the vision measuring device according to the third embodiment of the present invention will be explained. The vision measuring device according to the present embodiment is the same as the vision measuring device according to the second embodiment except the following point. The vision measuring device according to the present embodiment controls the light volume of the illumination device 142 based on a vertical synchronization signal output from the camera 141.
In the first embodiment, the frame rate of the camera 141 and the light volume of the illumination device 142 are controlled separately by the PC 2. Therefore, there is a risk that any delay in the calculation process of the PC 2, etc. might lag the timing to give an instruction to the camera 141 and the timing to give an instruction to the illumination device 142 one behind another. Therefore, there is a possibility that a phenomenon that is not desirable for the measurement operation might occur, such as instantaneous darkening or conversely instantaneous excessive brightening of the image that is displayed on the PC 2 when the normal measurement and auto-focusing are switched. However, in the present embodiment, the frame rate of the camera 141 and the light volume of the illumination device 142 are directly synchronized by the vertical synchronization signal. Therefore, it is possible to perform imaging at an appropriate light volume at all times even when normal measurement and auto-focusing are switched, and to prevent any troubles such as fluctuation of the brightness in the screen of the PC 2, etc.
There are many cameras in the market that have a vertical synchronization signal output function, and also the frame rate detecting device 153 and the standard frame rate retaining unit 154 can be manufactured easily. Moreover, when using such one of existing controllers 15 that is mounted with a microcomputer or the like, it is possible to implement these additional circuits by means of a program such as firmware. Hence, such a configuration as described above can be realized at a very low cost.
Next, a vision measuring device according to the fourth embodiment of the present invention will be explained. The vision measuring device according to the present embodiment is substantially the same as the vision measuring device according to the third embodiment, but different in adjusting the gain of the camera 141 under a certain condition described later.
The vision measuring device according to the present embodiment increases the light volume of the illumination device 142 during auto-focusing. However, in such a case, there is a possibility that a light volume higher than the maximum output of the illumination device 142 might be required. The present embodiment compensates for any shortage in the light volume of the illumination device 142 by increasing the gain of the camera 141. Such a configuration can realize highly accurate and high-speed auto-focusing with excellent operability even when the light volume of the illumination device 142 is insufficient.
Yoshida, Hiroyuki, Shirai, Naoki
Patent | Priority | Assignee | Title |
Patent | Priority | Assignee | Title |
6268918, | Jun 18 1998 | MINOLTA CO , LTD | Three-dimensional input device |
20010019361, | |||
20040016814, | |||
20040246367, | |||
20070177860, | |||
20080062297, | |||
20100157136, | |||
20100171875, | |||
EP1486758, | |||
JP2008224808, | |||
JP2009168607, | |||
WO2010131444, |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Jan 13 2012 | Mitutoyo Corporation | (assignment on the face of the patent) | / |
Date | Maintenance Fee Events |
Date | Maintenance Schedule |
Sep 09 2017 | 4 years fee payment window open |
Mar 09 2018 | 6 months grace period start (w surcharge) |
Sep 09 2018 | patent expiry (for year 4) |
Sep 09 2020 | 2 years to revive unintentionally abandoned end. (for year 4) |
Sep 09 2021 | 8 years fee payment window open |
Mar 09 2022 | 6 months grace period start (w surcharge) |
Sep 09 2022 | patent expiry (for year 8) |
Sep 09 2024 | 2 years to revive unintentionally abandoned end. (for year 8) |
Sep 09 2025 | 12 years fee payment window open |
Mar 09 2026 | 6 months grace period start (w surcharge) |
Sep 09 2026 | patent expiry (for year 12) |
Sep 09 2028 | 2 years to revive unintentionally abandoned end. (for year 12) |