An imaging unit is configured to capture an image of a subject in the water. A lighting unit is configured to illuminate the subject. A histogram generator is configured to divide a range from the minimum to the maximum luminance values of a luminance signal, included in an image signal generated by capturing the image of the subject into a plurality of luminance groups, and to generate histogram data representing the distribution of frequencies of the plurality of luminance groups. Based on the generated histogram data, a histogram analyzer is configured to analyze the distribution of frequencies of the plurality of luminance groups. According to the result of the analysis for the histogram data by the histogram analyzer, a distance controller is configured to control to adjust the distance between the subject and the lighting unit.
|
6. An underwater imaging apparatus control program stored in a non-transitory storage medium, the program causing a computer to execute:
a step of dividing a range from the minimum to the maximum luminance values of a luminance signal included in an image signal generated by capturing an image of a subject in the water into a plurality of luminance groups, and generating a histogram representing the distribution of frequencies of the plurality of luminance groups;
a step of, based on the generated histogram, analyzing how the luminance values are distributed in a low-luminance range, a high-luminance range, and a middle-luminance range, the low-luminance range being located on a low-luminance side of the histogram the high-luminance range being located on a high-luminance side of the histogram and the middle-luminance range being located between the low and high-luminance ranges of the histogram;
a step of calculating a first sum of the frequencies in the low-luminance range, a second sum of the frequencies in the middle-luminance range, a third sum of the frequencies in the high-luminance range, and a total sum of the frequencies in the histogram;
a step of calculating a low-luminance concentration rate, a middle-luminance concentration rate, and a high-luminance concentration rate, the low-luminance concentration rate being a ratio of the first sum of the frequencies to the total sum of the frequencies, the middle-luminance concentration rate being a ratio of the second sum of the frequencies to the total sum of the frequencies, and the high-luminance concentration rate being a ratio of the third sum of the frequencies to the total sum of the frequencies; and
a step of controlling the lighting unit so as to move away from the subject when the histogram analyzer analyzes that the low-luminance concentration rate is greater than the middle-luminance concentration rate, and that the high-luminance concentration rate is greater than the middle-luminance concentration rate.
5. An underwater imaging apparatus control method, comprising:
by an imaging unit, capturing an image of a subject in the water;
by a histogram generator, dividing a range from the minimum to the maximum luminance values of a luminance signal included in an image signal generated by capturing the image of the subject, into a plurality of luminance groups, and generating histogram representing the distribution of frequencies of the plurality of luminance groups;
by a histogram analyzer, based on the generated histogram, analyzing how the luminance values are distributed in a low-luminance range, a high-luminance range, and a middle-luminance range, the low-luminance range being located on a low-luminance side of the histogram, the high-luminance range being located on a high-luminance side of the histogram, and the middle-luminance range being located between the low and high-luminance ranges of the histogram;
by the histogram analyzer, calculating a first sum of the frequencies in the low-luminance range, a second sum of the frequencies in the middle-luminance range, a third sum of the frequencies in the high-luminance range, and a total sum of the frequencies in the histogram;
by the histogram analyzer, calculating a low-luminance concentration rate, a middle-luminance concentration rate, and a high-luminance concentration rate, the low-luminance concentration rate being a ratio of the first sum of the frequencies to the total sum of the frequencies the middle-luminance concentration rate being a ratio of the second sum of the frequencies to the total sum of the frequencies and the high-luminance concentration rate being a ratio of the third sum of the frequencies to the total sum of the frequencies; and
by a distance controller, controlling the lighting unit so as to move away from the subject, when the histogram analyzer analyzes that the low-luminance concentration rate is grater than the middle-luminance concentration rate, and that the high-luminance concentration rate is greater than the middle-luminance concentration rate.
1. An underwater imaging apparatus, comprising:
an imaging unit configured to capture an image of a subject in the water;
a lighting unit configured to illuminate the subject;
a histogram generator configured to divide a range from the minimum to the maximum luminance values of a luminance signal included in an image signal generated by capturing the image of the subject into a plurality of luminance groups, and to generate a histogram representing the distribution of frequencies of the plurality of luminance groups;
a histogram analyzer configured, based on the generated histogram, to analyze how the luminance values are distributed in a low-luminance range, a high-luminance range, and a middle-luminance range, the low-luminance rang being located on a low-luminance side of the histogram, the high-luminance range being located on a high-luminance side of the histogram, and the middle-luminance range being located between the low and high-luminance ranges of the histogram; and
a distance controller configured to control to adjust the distance between the subject and the lighting unit according to the result of the analysis for the histogram-data by the histogram analyzer,
wherein the histogram analyzer calculates a first sum of the frequencies in the low-luminance range, a second sum of the frequencies in the middle-luminance range, a third sum of the frequencies in the high-luminance range, and a total sum of the frequencies in the histogram,
wherein the histogram analyzer calculates a low-luminance concentration rate, a middle-luminance concentration rate, and a high-luminance concentration rate, the low-luminance concentration rate being a ratio of the first sum of the frequencies to the total sum of the frequencies, the middle-luminance concentration rate being a ratio of the second sum of the frequencies to the total sum of the frequencies, and the high-luminance concentration rate being a ratio of the third sum of the frequencies to the total sum of the frequencies,
and wherein the distance controller controls the lighting unit so as to move away from the subject, when the histogram analyzer analyzes that the low-luminance concentration rate is greater than the middle-luminance concentration rate, and that the high-luminance concentration rate is greater than the middle-luminance concentration rate.
2. The underwater imaging apparatus according to
the distance controller controls the lighting unit so as to move toward the subject, when the histogram analyzer analyzes that the low-luminance concentration rate or the middle-luminance concentration rate is greater than a first value.
3. The underwater imaging apparatus according to
the lighting controller controls the lighting unit so as to reduce the brightness of illumination by the lighting unit and the distance controller controls the lighting unit so as to move toward the subject, when the histogram analyzer analyzes that the high-luminance concentration rate is greater than a second value.
4. The underwater imaging apparatus according to
|
This application is a Continuation of PCT Application No. PCT/JP2016/063142, filed on Apr. 27, 2016, and claims the priority of Japanese Patent Application No. 2015-164419, filed on Aug. 24, 2015, the entire contents of both of which are incorporated herein by reference.
The present disclosure relates to an underwater imaging apparatus which captures an image of a subject underwater, a method for controlling the underwater imaging apparatus, and a program for controlling the underwater imaging apparatus.
Wall surfaces of dams storing water are an example of underwater structures. To inspect whether wall surfaces in the water are deteriorated, underwater imaging apparatuses are used in some cases. Such an underwater imaging apparatus is immersed in the water to capture an image of the wall surface. Based on the captured image, the inspector checks whether the wall surface is deteriorated.
Sunlight does not sufficiently reach deep places in the water. Underwater imaging apparatuses therefore include lighting apparatuses and capture images of wall surfaces while projecting light onto the wall surfaces. However, some of the images captured in such a manner are too dark or too light, depending on lighting conditions. The inspector is not able to accurately determine whether the wall surface is deteriorated based on the captured images.
A first aspect of the embodiments provides an underwater imaging apparatus, including: an imaging unit configured to capture an image of a subject in the water; a lighting unit configured to illuminate the subject; a histogram generator configured to divide a range from the minimum to the maximum luminance values of a luminance signal included in an image signal generated by capturing the image of the subject into a plurality of luminance groups, and to generate histogram data representing the distribution of frequencies of the plurality of luminance groups; a histogram analyzer configured, based on the generated histogram data, to analyze the distribution of frequencies of the plurality of luminance groups; and a distance controller configured to control to adjust the distance between the subject and the lighting unit according to the result of the analysis for the histogram data by the histogram analyzer.
A second aspect of the embodiments provides an underwater imaging apparatus control method, including: by an imaging unit, capturing an image of a subject in the water; by a histogram generator, dividing a range from the minimum to the maximum luminance values of a luminance signal included in an image signal generated by capturing the image of the subject, into a plurality of luminance groups, and generating histogram data representing the distribution of frequencies of the plurality of luminance groups; by a histogram analyzer, based on the generated histogram data, analyzing the distribution of frequencies of the plurality of luminance groups; and by a distance controller, according to the result of the analysis for the histogram data by the histogram analyzer, adjusting the distance between the subject and a lighting unit which illuminates the subject.
A third aspect of the embodiments provides an underwater imaging apparatus control program stored in a non-transitory storage medium, the program causing a computer to execute: a step of dividing a range from the minimum to the maximum luminance values of a luminance signal included in an image signal generated by capturing an image of a subject in the water into a plurality of luminance groups, and generating histogram data representing the distribution of frequencies of the plurality of luminance groups; a step of, based on the generated histogram data, analyzing the distribution of frequencies of the plurality of luminance groups; and a step of, according to the result of the analysis for the histogram data by the histogram analyzer, adjusting the distance between the subject and a lighting unit which illuminates the subject.
Hereinafter, a description is given of an underwater imaging apparatus, a method for controlling an underwater imaging apparatus, and a program for controlling an underwater imaging apparatus, according to each embodiment, with reference to the accompanying drawings.
In
In the first embodiment, the body 10 is hung by wire, for example, from above the water. The body 10 is configured to move toward and away from the wall surface 101.
The body 10 includes: an imaging unit 1 configured to capture an image of the wall surface 101; an image signal processor 2 configured to process an image signal outputted from the imaging unit 1; and a lighting unit 3 configured to illuminate the wall surface 101. The wall surface 101 is a subject. The imaging unit 1 includes a CCD or CMOS. The lighting unit 3 can be composed of any light source.
The image signal processor 2 includes: a histogram generator 21, a histogram analyzer 22; and an image quality corrector 23. The histogram generator 21 generates histogram data based on a luminance signal included in the image signal. The histogram analyzer 22 analyzes the histogram data. The image quality corrector 23 corrects the image signal for image quality correction. The image signal processor 2 may be composed of a hardware circuit, or all or a part thereof may be composed of software (a computer program).
The concrete way to generate the histogram data by the histogram generator 21 and the concrete way to analyze the histogram data by the histogram analyzer 22 are described later.
The underwater imaging apparatus further includes a controller 51, a driver 52, and a display 53, which are situated on the water or on the ground. The controller 51 includes a lighting controller 511 and a distance controller 512. The lighting controller 511 controls lighting by the lighting unit 3. The distance controller 512 controls the distance between the body 10 and the wall surface 101.
The controller 51 can be composed of a personal computer, a microcomputer, a microprocessor, or the like. The driver 52 can be composed of any driving mechanism. The display 53 can be composed of any display such as a liquid crystal panel.
The controller 51 is supplied with data representing the result of analyzing the histogram data by the histogram analysis unit 22.
The driver 52 drives and moves the body 10, which is hung by wire, in the horizontal direction. In addition to the horizontal direction, the driver 52 drives and moves the body 10 in the vertical direction. The display 53 displays the image signal outputted from the image signal processor 2. The inspector watches the image displayed on the display 53 to check whether the wall surface 101 is deteriorated.
Using the flowchart illustrated in
The histogram generator 21 divides all the luminance levels of the luminance signal into a plurality of luminance groups. The histogram generator 21 then counts the number of pixels included in each luminance group to generate histogram data. The image signal has 256 levels of luminance, for example. The histogram generator 21 divides all the luminance levels of the luminance signal into 16 luminance groups, for example.
In step S3, the histogram analyzer 22 analyzes the distribution of luminance values of the luminance signal in the low-luminance range Rlow, middle-luminance range Rmid, and high luminance range Rhigh.
The histogram analyzer 22 calculates a total sum Hist_sum in the histogram based on Equation (1). Hist [i] is a frequency of each of the luminance groups 0 to 15.
The histogram analyzer 22 calculates: a low-luminance concentration rate Low_rate based on Equation (2); a middle-luminance concentration rate Mid_rate based on Equation (3); and a high-luminance concentration rate High_rate based on Equation (4).
Low_rate=(Hist[0]+Hist[1]+Hist[2]+Hist[3]+Hist[4])×100/Hist_sum (2)
Mid_rate=(Hist[5]+Hist[6]+Hist[7]+Hist[8]+Hist[9]+Hist[10])×100/Hist_sum (3)
High_rate=(Hist[11]+Hist[12]+Hist[13]+Hist[14]+Hist[15])×100/Hist_sum. (4)
Equations (2) to (4) to calculate the low, middle, and high-luminance concentration rates Low_rate, Mid_rate, and High_rate are just an example. The formula to calculate the low, middle, and high-luminance concentration rates Low_rate, Mid_rate, and High_rate is not limited to Equations (2) to (4). The histogram analyzer 22 can calculate the rate of concentration of frequencies in each of the low, middle, and high-luminance ranges Rlow, Rmid, and Rhigh using any calculation formula.
In step S4, the histogram analyzer 22 determines whether luminance values of the luminance signal are concentrated in the low-luminance range Rlow. The histogram analyzer 22 determines that luminance values are concentrated in the low-luminance range Rlow when the analysis of the histogram shows that luminance values are concentrated in the low-luminance range Rlow at or above a predetermined concentration rate. For instance, the histogram analyzer 22 determines that luminance values of the luminance signal are concentrated in the low-luminance range Rlow when the low-luminance concentration rate Low_rate is equal to or greater than 60%.
When the body 10 is distant from the wall surface 101, light from the lighting unit 3 does not sufficiently reach the wall surface 101 in some cases, and the captured image is dark. In the luminance histogram obtained at this time, luminance values are concentrated in the low-luminance range Rlow as illustrated in
When it is determined in step S4 that luminance values are concentrated in the low-luminance range Rlow (YES), the distance controller 512 controls the driver 52 so that the lighting unit 3 (that is, the body 10) moves toward the wall surface 101 in step S9.
When the lighting unit 3 moves toward the wall surface 101, the image becomes brighter as illustrated in
In step S9, the distance controller 512 preferably moves the lighting unit 3 by a travel distance corresponding to the low-luminance concentration rate Low_rate as illustrated in
The distance controller 512 linearly increases the travel distance as the low-luminance concentration rate Low_rate increases from a predetermined value to 100%. The symbol “−” of the travel distance in
After step S9, the process goes to step S11. When it is not determined in step S4 that luminance values are concentrated in the low-luminance range Rlow (NO), the process goes to step S5.
In step S5, the histogram analyzer 22 determines whether luminance values are concentrated in the middle-luminance range Rmid. The histogram analyzer 22 determines that luminance values are concentrated in the middle-luminance range Rmid when the analysis of the histogram shows that luminance values are concentrated in the middle-luminance range Rmid at or above a predetermined concentration rate. Specifically, the histogram analyzer 22 determines that luminance values of the luminance signal are concentrated in the middle-luminance range Rmid when the middle-luminance concentration rate Mid_rate is equal to or greater than 60%.
When it is determined in step S5 that luminance values are concentrated in the middle-luminance range Rmid (YES), the distance controller 512 controls the driver 52 in step S9 so that the lighting unit 3 moves toward the wall surface 101.
When the body 10 moves toward the wall surface 101, the image becomes brighter as illustrated in
In step S9, the distance controller 512 preferably moves the body 10 so that the travel distance linearly increases as the middle-luminance concentration rate Mid_rate increases from a predetermined value to 100% as illustrated in
When it is not determined in step S5 that luminance values are concentrated in the middle-luminance range Rmid (NO), the process goes to step S6.
In step S6, the histogram analyzer 22 determines whether luminance values are concentrated in the high-luminance range Rhigh. The histogram analyzer 22 determines that luminance values are concentrated in the high-luminance section Rhigh when the analysis of the histogram shows that luminance values are concentrated in the high-luminance range Rhigh at or above a predetermined concentration rate. Specifically, the histogram analyzer 22 determines that luminance values are concentrated in the high-luminance range Rhigh when the high-luminance concentration rate High_rate is equal to or greater than 60%.
When it is determined in step S6 that luminance values are concentrated in the high-luminance range Rhigh (YES), the lighting controller 511 controls the lighting unit 3 so that the brightness of lighting is reduced in step S8. The image is then in the state illustrated in
The distance controller 512 subsequently controls the driver 52 so that the lighting unit 3 moves toward the wall surface 101 in step S9. The image is then in the state illustrated in
In step S9, the distance controller 512 preferably moves the lighting unit 3 so that the travel distance linearly increases as the high-luminance concentration rate High_rate increases from a predetermined value to 100% as illustrated in
When it is not determined in step S6 that luminance values are concentrated in the high-luminance range Rhigh (No), the process goes to step S7.
The histogram analyzer 22 determines in step S7 whether luminance values are concentrated both in the low and high-luminance ranges Rlow and Rhigh.
If the body 10 is too close to the wall surface 101, the wall surface 101 is locally illuminated. An area of the image is extremely bright while other areas corresponding to not-illuminated part are dark, as illustrated in
The histogram analysis unit 22 determines whether the following conditions are satisfied in which: the low-luminance concentration rate Low_rate is higher than the middle-luminance concentration rate Mid_rate and the high-luminance concentration rate High_rate is higher than the middle-luminance concentration rate Mid-rate. When this condition is satisfied, the histogram analyzer 22 determines that the image is in the state illustrated in
When it is determined in step S7 that luminance values are concentrated in the low and high-luminance ranges Rlow and Rhigh (YES), in step S10, the distance controller 512 controls the driver 52 so that the lighting unit 3 moves away from the wall surface 101.
The image is then in the state illustrated in
After step S10, the process goes to step S11. When it is not determined in step S7 that luminance values are concentrated in the low and high-luminance ranges Rlow and Rhigh (NO), the process goes to step S11.
In step S11, the image quality corrector 23 performs an image quality correction process for the image signal. In step S12, the image signal processor 2 outputs the image signal subjected to the image quality correction process. After step S12, the process returns to step S1, and steps S1 to S12 are repeated.
As described above, when luminance values in the image signal generated by capturing an image of a subject are concentrated in the low or middle-luminance range Rlow or Rmid, the lighting unit 3 moves toward the subject. The image thereby has appropriate brightness as illustrated in
When luminance values in the image signal are concentrated in the high-luminance range Rhigh, the brightness of the lighting by the lighting unit 3 is reduced, and the lighting unit 3 moves toward the subject. The obtained image thereby has appropriate brightness as illustrated in
When luminance values in the image signal are concentrated in the low and high-luminance ranges Rlow and Rhigh, the lighting unit 3 is moved away from the subject. The obtained image thereby has an appropriate brightness illustrated in
The determination “NO” in steps S4 to S7 means that the generated image signal represents an image that originally has appropriate brightness and provides a luminance histogram in which luminance values are distributed across a wide range.
Herein, a description is given of a concrete configuration example of the image quality corrector 23 and the image quality correction process executed by the concrete configuration example, by using
As illustrated in
Herein, the luminance signal included in the image signal inputted to the image signal processor 2 is a luminance signal Y0. The luminance signal Y0 is inputted to the delay device 231 and the Gaussian filter 232. The Gaussian filter is a low-pass filter that extracts extremely low-frequency signals using a Gaussian function. As illustrated in
The Gaussian filter 232 may be replaced with a low-pass filter having an extremely low cutoff frequency and including a large number of taps.
The delay device 231 delays the luminance signal Y0 by the time needed for processing at the Gaussian filter 232, and outputs the resultant signal as a luminance signal Y1. The Gaussian filter 232 extracts a low-frequency component signal Y2 from the luminance signal Y0.
As illustrated in (a) of
The subtractor 233 subtracts the low-frequency component signal Y2 from the luminance signal Y1 to output the resultant signal as a high-frequency component signal Y3 illustrated in (b) of
The high-frequency component signal Y3 outputted from the subtractor 233 is inputted to the low-pass filter 235. The low-pass filter 235 has a frequency characteristic FL illustrated in
An outputted signal Y4 from the low-pass filter 235 is a signal obtained by extracting a signal having lower frequencies, from the high-frequency component signal Y3 through the frequency characteristic FL. The outputted signal Y4 has a waveform illustrated in (c) of
The multiplier 236 multiplies the lower-frequency side high-frequency component signal Y4 by a gain G1 to produce a correction component signal Y5 illustrated in (d) of
The delay device 234 further delays the luminance signal Y1 by the time needed for processing at the low-pass filter 235 and multiplier 236, and outputs the resultant signal as a luminance signal Y6. The adder 237 adds up the luminance signal Y6 and the correction component signal Y5 to output the resultant signal as a corrected luminance signal Y7 illustrated in (e) of
As described above, the image quality corrector 23 preferably treats the luminance signal Y0 with the image quality correction process that emphasizes the contrast at a part corresponding to edge signals in the luminance signal Y0.
Specifically, for an edge signal in which the luminance rises from a first level to a second level, luminance values are reduced at the corner of the edge where the luminance starts to rise from the first level. Moreover, the luminance values are increased at the corner of the edge where the rising luminance reaches the second level.
For an edge signal in which the luminance decreases from the second level to the first level, the luminance values are increased at the corner of the edge where the luminance starts to decrease from the second level. Moreover, the luminance values are reduced at the corner of the edge where the falling luminance reaches the first level.
In
At least one of the lighting controller 511 and the distance controller 512, which are provided for the controller 51 in the aforementioned configuration, may be provided for the controller within the body 10.
The functions of the histogram generator 21, histogram analyzer 22, lighting controller 511, and distance controller 512 may be implemented by a computer program (an underwater imaging apparatus control program). The underwater imaging apparatus control program may be recorded in a non-transitory storage medium.
Specifically, the underwater imaging apparatus control program causes a computer to execute the following steps. First, the control program causes the computer to execute the step of dividing a range from the minimum to the maximum values of the luminance signal included in an image signal outputted by the imaging unit, which captures an image of a subject in the water, into plural luminance groups and generating histogram data representing the distribution of frequencies of the plural luminance groups.
Next, the control program causes the computer to execute the step based on the generated histogram data, analyzing the distribution of frequencies of the plural luminance groups. The control program causes the computer to execute a step of adjusting the distance between the subject and the lighting unit that illuminates the subject, in accordance with the result of the analysis for the histogram data.
In an underwater imaging apparatus according to the second embodiment illustrated in
As illustrated in
The underwater imaging apparatus according to the second embodiment also operates as illustrated in
In an underwater imaging apparatus according to the third embodiment illustrated in
In the underwater imaging apparatus according to the third embodiment, which is illustrated in
The distance controller 512 controls the lighting unit extension mechanism 4 so that the lighting unit 3 moves toward or away from the wall surface 101. The distance controller 512 may control the propulsion mechanism 54 so that the body 30 moves toward or away from the wall surface 101 while controlling the lighting unit extension mechanism 4 so that the lighting unit 3 moves toward or away from the wall surface 101.
The lighting unit extension mechanism 4, which moves only the lighting unit 3 toward or away from the wall surface 101, may be provided for the body 10, which is illustrated in
As described above, in accordance with the underwater imaging apparatus, the method for controlling the underwater imaging apparatus, and the program for controlling the underwater imaging apparatus according to each embodiment, it is possible to provide an image that enables a subject in the water to be precisely identified.
The present invention is not limited to each embodiment described above and can be variously changed without departing from, the scope of the invention.
Patent | Priority | Assignee | Title |
Patent | Priority | Assignee | Title |
4769776, | Aug 30 1985 | Hitachi, LTD | Apparatus for measuring the concentration of filamentous microorganisms in a mixture including microorganisms |
5185671, | Jun 21 1991 | Northrop Grumman Systems Corporation | Adaptive control of an electronic imaging camera |
6192145, | Feb 12 1996 | Sarnoff Corporation | Method and apparatus for three-dimensional scene processing using parallax geometry of pairs of points |
9258489, | Sep 30 2013 | Samsung Electronics Co., Ltd.; SAMSUNG ELECTRONICS CO , LTD | Image acquisition method and apparatus |
20070237505, | |||
20080069433, | |||
20080151057, | |||
20150312487, | |||
20170053384, | |||
20170293217, | |||
JP107084, | |||
JP1168169, | |||
JP2009300871, | |||
JP201028178, | |||
JP2012137706, | |||
JP2013220254, | |||
JP7201712, | |||
JP7220058, | |||
WO2013038550, |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Jan 09 2018 | KIDA, SHINGO | JVC Kenwood Corporation | CORRECTIVE ASSIGNMENT TO CORRECT THE COUNTRY OF THE ASSIGNEE FROM CHINA TO JAPAN PREVIOUSLY RECORDED ON REEL 044794 FRAME 0527 ASSIGNOR S HEREBY CONFIRMS THE ASSIGNMENT OF ASSIGNOR S INTEREST | 051208 | /0668 | |
Jan 09 2018 | KIDA, SHINGO | JVC Kenwood Corporation | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 044794 | /0527 | |
Jan 31 2018 | JVC Kenwood Corporation | (assignment on the face of the patent) | / |
Date | Maintenance Fee Events |
Jan 31 2018 | BIG: Entity status set to Undiscounted (note the period is included in the code). |
Aug 30 2023 | M1551: Payment of Maintenance Fee, 4th Year, Large Entity. |
Date | Maintenance Schedule |
Mar 17 2023 | 4 years fee payment window open |
Sep 17 2023 | 6 months grace period start (w surcharge) |
Mar 17 2024 | patent expiry (for year 4) |
Mar 17 2026 | 2 years to revive unintentionally abandoned end. (for year 4) |
Mar 17 2027 | 8 years fee payment window open |
Sep 17 2027 | 6 months grace period start (w surcharge) |
Mar 17 2028 | patent expiry (for year 8) |
Mar 17 2030 | 2 years to revive unintentionally abandoned end. (for year 8) |
Mar 17 2031 | 12 years fee payment window open |
Sep 17 2031 | 6 months grace period start (w surcharge) |
Mar 17 2032 | patent expiry (for year 12) |
Mar 17 2034 | 2 years to revive unintentionally abandoned end. (for year 12) |