sequential video data frames are encoded using cores including a first core and a second core. A first beginning frame is divided into slices. The first core is assigned to process a first slice. The second core is assigned to process a second slice. The first beginning frame is processed using the cores which results in a first ending frame in which the first slice was partitioned into a third slice and a fourth slice. The third slice was processed by the first core. The fourth slice and the second slice were processed by the second core. A second beginning frame, which immediately follows the first ending frame, is divided into a second plurality of slices. The first core is assigned to the third slice. The second core is assigned to a fifth slice which has a size equal to a sum of the second and fourth slices.
|
1. A method of encoding sequential video data frames using a plurality of cores comprising a first core and a second core, the method comprising:
dividing a beginning portion of a first frame into a first plurality of slices among the plurality of cores in which the first core is assigned to process a first slice of the plurality of slices and the second core is assigned to process a second slice of the plurality of slices;
processing the first frame using the plurality of cores;
partitioning the first slice into a third slice and a fourth slice, wherein the third slice is processed by the first core and the fourth slice and the second slice are processed by the second core; and
dividing a second frame, which immediately follows the first ending frame, into a second plurality of slices among the plurality of cores in which the first core is assigned to a fifth slice equivalent in size to the third slice and the second core is assigned to a sixth slice which has a size equal to a sum of the second and fourth slices.
18. An encoder, comprising:
a plurality of slave cores, wherein each slave core is configured to process one or more slices of a frame;
a master core that directs operation of the slave cores and is configured to process one or more slices of a frame, wherein:
the master core is configured to assign itself and each slave core to a single slice of a first plurality of slices of a beginning portion of a first frame, wherein a number of slices of the first plurality of slices is equal to a number of slave cores plus one for the master core;
the master core is configured to partition the slices of the first plurality of slices to result in a second plurality of slices greater in number than the first plurality of slices according to a protocol based on relative processing complexity among the first plurality of slices;
the master core is configured to selectively assign itself and each slave core of the plurality of slave cores to process the second plurality of slices at an ending portion of the first frame, wherein a size of slice processed by each core is in accordance with the protocol; and
the master core is configured to assign itself and each slave core to a single slice of a third plurality of slices of a beginning portion of a second frame, wherein a number of slices of the third plurality of slices is equal to a number of slices of the first plurality of slices and less than a number of slices of the second plurality of slices; wherein each slave core is assigned to a slice having a size equal to the size processed by it during the ending portion of the first frame.
10. A method of processing frames, comprising:
providing a processing unit comprising a plurality of cores of a first number;
dividing a first frame into a first plurality of slices of the first number, wherein each slice of the first plurality of slices has a size;
assigning each core of the plurality of cores to a slice of the first plurality of slices, wherein each slice of the first plurality of slices has a core of the plurality of cores assigned to it;
initially processing a beginning portion of the first frame with the plurality of cores processing the first plurality of slices as assigned according to the step of assigning each core of the plurality of cores to a slice of the first plurality of slices;
processing an ending portion of the first frame, wherein the ending portion of the first frame has a second plurality of slices of a second number, wherein the second number is greater than the first number, wherein each core has processed at least one slice of the second plurality of slices and at least one core has processed more than slice of the second plurality of slices, wherein each core has processed an amount of the first ending frame equal to a sum of the slices processed by it;
dividing a beginning portion of a second frame into a third plurality of slices of the first number;
assigning each core of the plurality of cores to a slice of the third plurality of slices, wherein each slice of the third plurality of slices has a core of the plurality of cores assigned to it, wherein each core is assigned to an amount of the frame equal to the amount it processed on the ending portion of the first frame; and
initially processing the beginning portion of the second frame with the plurality of cores processing the third plurality of slices as assigned according to the step of assigning each core of the plurality of cores to a slice of the third plurality of slices.
2. The method of
3. The method of
4. The method of
5. The method of
processing the second frame using the plurality of cores to produce a modified second frame.
6. The method of
7. The method of
8. The method of
11. The method of
finally processing the second frame to produce a modified second frame.
12. The method of
13. The method of
14. The method of
15. The method of
16. The method of
17. The method of
19. The encoder of
20. The encoder of
|
1. Field
This disclosure relates generally to video processing, and more specifically, to a system and method for load balancing a video signal in a multi-core processor.
2. Related Art
The Advanced Video Coding (AVC) standard, Part 10 of MPEG4 (Motion Picture Experts Group), otherwise known as H.264, was developed to enable transmission or storing of video signals with improved quality at a constant bit rate (CBR), or variable bit rate (VBR). A present day encoder compliant with the H-264 standard can be designed to run on a single-core processor. However, a commonly available single-core processor is not capable of running fast enough to reach high-definition resolutions (e.g. 720p, 1080i, and 1080p). In order to reach high-definition resolutions in near real-time, the H.264 encoder can be designed to run in parallel on a multi-core processor.
To effectively encode a video signal in a multi-core processor, video data should be relatively evenly divided between the cores for parallel processing. This load balancing is generally achieved using either data partitioning or function partitioning. In data partitioning, a frame of data is divided into multiple portions and provided to the cores in parallel. This approach is relatively easily implemented. However, it is difficult to insure proper load balancing between processor cores because some of the data will have more complex processing requirements. For example, one slice of a frame may include movement, or finer details than another slice of the frame. This can lead to processing bottlenecks causing some cores to be idle while others are busy.
In load balancing using function partitioning each core is assigned a different processing function, such as motion estimating or encoding. The data is then processed in a pipelined fashion. For example, one processor may apply a first function to video data and then pass the modified data to a second processor for a second function. However, it can be difficult to load balance by function because algorithms cannot easily be divided into an arbitrary number of blocks to match the number of processor cores.
Therefore, what is needed is a system and method that solves the above problems.
The present invention is illustrated by way of example and is not limited by the accompanying figures, in which like references indicate similar elements. Elements in the figures are illustrated for simplicity and clarity and have not necessarily been drawn to scale.
As used herein, the term “bus” is used to refer to a plurality of signals or conductors which may be used to transfer one or more various types of information, such as data, addresses, control, or status. The conductors as discussed herein may be illustrated or described in reference to being a single conductor, a plurality of conductors, unidirectional conductors, or bidirectional conductors. However, different embodiments may vary the implementation of the conductors. For example, separate unidirectional conductors may be used rather than bidirectional conductors and vice versa. Also, plurality of conductors may be replaced with a single conductor that transfers multiple signals serially or in a time multiplexed manner. Likewise, single conductors carrying multiple signals may be separated out into various different conductors carrying subsets of these signals. Therefore, many options exist for transferring signals.
Generally, there is provided, a system and method for load balancing the encoding of a video signal in a multi-core processor that includes designating one core as a master and the other cores as slaves. A first frame is divided into slices and each slice is assigned to a core. The master core determines if the load is balanced between the cores while processing the first frame. If the load is unbalanced, and a core is doing more work than the other cores, the slice assigned to the overloaded core is divided into two or more smaller slices and the smaller slices are assigned to one or more other cores by the master core to reduce the work of the overloaded core. The redistribution is maintained until processing of the first frame is completed. Prior to beginning a second frame, the slices for each of the cores are resized based on the reassigned slice sizes at the end of the first frame. By reassigning slices to the cores during a second frame based on the load redistribution of a first frame, the work load of the cores can be more equally balanced throughout the encoding of a video signal.
In one aspect, there is provided, a method of encoding sequential video data frames using a plurality of cores comprising a first core and a second core. The method comprising: dividing a first beginning frame into a first plurality of slices among the plurality of cores in which the first core is assigned to process a first slice of the plurality of slices and the second core is assigned to process a second slice of the plurality of slices; processing the first beginning frame using the plurality of cores which results in a first ending frame in which the first slice was partitioned into a third slice and a fourth slice, wherein the third slice was processed by the first core and the fourth slice and the second slice were processed by the second core; and dividing a second beginning frame, which immediately follows the first ending frame, into a second plurality of slices among the plurality of cores in which the first core is assigned to the third slice and the second core is assigned to a fifth slice which has a size equal to a sum of the second and fourth slices. The plurality of cores may further comprise a master core and a plurality of slave cores. The step of processing the first beginning frame may be further characterized by the master core partitioning the slices according to a protocol. The step of dividing the second beginning frame may be further characterized by each of the plurality of cores being assigned to a slice having a plurality of macroblocks equal in number to the number of macroblocks processed on the first ending frame. The method may further comprise processing the second beginning frame using the plurality of cores to produce a second ending frame. The step of processing the second beginning frame may be further characterized as partitioning the second plurality of slices into a third plurality of slices greater in number than the second plurality of slices. The step of processing the second beginning frame may be further characterized by each slice of the third plurality of slices being processed by one of the plurality of cores, wherein each core processes a size equal to a sum of the sizes of the slices that it processes. The step of dividing the second beginning frame may be further characterized by the second plurality of slices being of a number of slices equal to a number of slices of the first plurality of slices. The second slice and the fourth slice may be non-adjacent.
In another aspect, there is provided, a method of processing frames. The method comprising: providing a processing unit comprising a plurality of cores of a first number; dividing a first beginning frame into a first plurality of slices of the first number, wherein each slice of the first plurality of slices has a size; assigning each core of the plurality of cores to a slice of the first plurality of slices, wherein each slice of the first plurality of slices has a core of the plurality of cores assigned to it; initially processing the first beginning frame with the plurality of cores processing the first plurality of slices as assigned according to the step of assigning each core of the plurality of cores to a slice of the first plurality of slices; finally processing the first beginning frame to produce a first ending frame, wherein the first ending frame has a second plurality of slices of a second number, wherein the second number is greater than the first number, wherein each core has processed at least one slice of the second plurality of slices and at least one core has processed more than slice of the second plurality of slices, wherein each core has processed an amount of the first ending frame equal to a sum of the slices processed by it; dividing a second beginning frame into a third plurality of slices of the first number; assigning each core of the plurality of cores to a slice of the third plurality of slices, wherein each slice of the third plurality of slices has a core of the plurality of cores assigned to it, wherein each core is assigned to an amount of the frame equal to the amount it processed on the first ending frame; and initially processing the second beginning frame with the plurality of cores processing the third plurality of slices as assigned according to the step of assigning each core of the plurality of cores to a slice of the third plurality of slices. The method may further comprise finally processing the second beginning frame to produce a second ending frame. The step of finally processing the second beginning frame to produce a second ending frame may be further characterized by the second ending frame comprising a fourth plurality of slices greater in number than the third plurality of slices. The step of finally processing the first beginning frame may be further characterized by a master core of the plurality of cores controlling the plurality of cores according to a protocol. The method may operate in compliance with H.264. The first beginning frame and the second beginning frame may comprise a plurality of macroblocks. A size equal to a sum of the first plurality of slices is equal to a size equal to a sum of the plurality of macroblocks. The method may further comprising initially processing a third beginning frame with the cores processing sizes equal to the sizes processed on the second ending frame.
In yet another aspect, there is provided, an encoder, comprising: a plurality of slave cores, wherein each slave core is capable of processing one or more slices of a frame; a master core that directs operation of the slave cores and is capable of processing one or more slices of a frame, wherein: the master core assigns itself and each slave core to a single slice of a first plurality of slices of a first beginning frame, wherein a number of slices of the first plurality of slices is equal to a number of slave cores plus one for the master core; the master core partitions the slices of the first plurality of slices to result in a second plurality of slices greater in number than the first plurality of slices according to a protocol based on relative complexity among the first plurality of slices; the master core selectively assigns itself and each slave core of the plurality of slave cores to process the second plurality of slices to a achieve a first ending frame, wherein a size processed by each core is in accordance with the protocol; and the master core assigns itself and each slave core to a single slice of a third plurality of slices of a second beginning frame, wherein a number of slices of the third plurality of slices is equal to a number of slices of the first plurality of slices and less than a number of slices of the second plurality of slices; wherein each slave core is assigned to a slice having a size equal to the size processed by it in achieving the first ending frame. The encoder may further comprise a memory shared among the master core and the slave cores. The first beginning frame may comprise a plurality of macroblocks, wherein a size equal to a sum of the first plurality of slices is equal to a size equal to a sum of the plurality of macroblocks which is equal to a size of a sum of the second plurality of slices.
The plurality of cores 26 can include any number of data processing cores. In one example, cores 26 are digital signal processors (DSPs). In another example, cores 26 are general purpose processors. Bus 22 comprises a plurality of conductors and is coupled to memory 18, encoder 16, a video capture device 12, and display 14. In one embodiment, video data is provided to memory 18 via bus 22 by the video capture device 12. Video capture device 12 may be, for example, a video camera. The video data is retrieved from memory 18 by encoder 16 for processing. The processed data is then provided to decoder/display 14 to be decoded and then displayed, or to communication channels to be transmitted, or to storages to be recorded (not shown). The display of decoder/display 14 may be, for example, a personal computer or a video monitor. Although not illustrated in
A frame of video data is partitioned into slices. For higher quality, frames should be partitioned into as few slices as possible. In one embodiment, the number of slices is equal to the number of cores. For example, in the four core system of
At beginning portion f1b of FRAME 1, each core is given the same size slice as was assigned at ending portion f0e as can be seen in
Because the apparatus implementing the present invention is, for the most part, composed of electronic components and circuits known to those skilled in the art, circuit details will not be explained in any greater extent than that considered necessary as illustrated above, for the understanding and appreciation of the underlying concepts of the present invention and in order not to obfuscate or distract from the teachings of the present invention.
Moreover, the terms “front,” “back,” “top,” “bottom,” “over,” “under” and the like in the description and in the claims, if any, are used for descriptive purposes and not necessarily for describing permanent relative positions. It is understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments of the invention described herein are, for example, capable of operation in other orientations than those illustrated or otherwise described herein.
Some of the above embodiments, as applicable, may be implemented using a variety of different information processing systems. For example, although
Thus, it is to be understood that the architectures depicted herein are merely exemplary, and that in fact many other architectures can be implemented which achieve the same functionality. In an abstract, but still definite sense, any arrangement of components to achieve the same functionality is effectively “associated” such that the desired functionality is achieved. Hence, any two components herein combined to achieve a particular functionality can be seen as “associated with” each other such that the desired functionality is achieved, irrespective of architectures or intermedial components. Likewise, any two components so associated can also be viewed as being “operably connected,” or “operably coupled,” to each other to achieve the desired functionality.
Also for example, in one embodiment, the illustrated elements of codec 10 are circuitry located on multiple die in a single integrated circuit package. Alternatively, codec 10 may be included on a single integrated circuit. Also for example, codec 10 or portions thereof may be soft or code representations of physical circuitry or of logical representations convertible into physical circuitry. As such, codec 10 may be embodied in a hardware description language of any appropriate type.
Furthermore, those skilled in the art will recognize that boundaries between the functionality of the above described operations merely illustrative. The functionality of multiple operations may be combined into a single operation, and/or the functionality of a single operation may be distributed in additional operations. Moreover, alternative embodiments may include multiple instances of a particular operation, and the order of operations may be altered in various other embodiments.
Although the invention is described herein with reference to specific embodiments, various modifications and changes can be made without departing from the scope of the present invention as set forth in the claims below. Accordingly, the specification and figures are to be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of the present invention. Any benefits, advantages, or solutions to problems that are described herein with regard to specific embodiments are not intended to be construed as a critical, required, or essential feature or element of any or all the claims.
The term “coupled,” as used herein, is not intended to be limited to a direct coupling or a mechanical coupling.
Furthermore, the terms “a” or “an,” as used herein, are defined as one or more than one. Also, the use of introductory phrases such as “at least one” and “one or more” in the claims should not be construed to imply that the introduction of another claim element by the indefinite articles “a” or “an” limits any particular claim containing such introduced claim element to inventions containing only one such element, even when the same claim includes the introductory phrases “one or more” or “at least one” and indefinite articles such as “a” or “an.” The same holds true for the use of definite articles.
Unless stated otherwise, terms such as “first” and “second” are used to arbitrarily distinguish between the elements such terms describe. Thus, these terms are not necessarily intended to indicate temporal or other prioritization of such elements.
Patent | Priority | Assignee | Title |
10045032, | Jan 24 2013 | Intel Corporation | Efficient region of interest detection |
9319702, | Dec 03 2012 | Intel Corporation | Dynamic slice resizing while encoding video |
Patent | Priority | Assignee | Title |
6870883, | Jul 15 1998 | Sony Corporation | Parallel encoding and decoding processor system and method |
20070086528, | |||
20080151997, |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Jun 27 2008 | Freescale Semiconductor, Inc. | (assignment on the face of the patent) | / | |||
Jun 27 2008 | YAN, YONG | Freescale Semiconductor, Inc | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 021160 | /0977 | |
Jul 28 2008 | Freescale Semiconductor, Inc | CITIBANK, N A | SECURITY AGREEMENT | 021570 | /0449 | |
Feb 19 2010 | Freescale Semiconductor, Inc | CITIBANK, N A | SECURITY AGREEMENT | 024085 | /0001 | |
Apr 13 2010 | Freescale Semiconductor, Inc | CITIBANK, N A , AS COLLATERAL AGENT | SECURITY AGREEMENT | 024397 | /0001 | |
May 21 2013 | Freescale Semiconductor, Inc | CITIBANK, N A , AS NOTES COLLATERAL AGENT | SECURITY AGREEMENT | 030633 | /0424 | |
Nov 01 2013 | Freescale Semiconductor, Inc | CITIBANK, N A , AS NOTES COLLATERAL AGENT | SECURITY AGREEMENT | 031591 | /0266 | |
Oct 02 2015 | Freescale Semiconductor, Inc | NORTH STAR INNOVATIONS INC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 037694 | /0264 | |
Dec 07 2015 | CITIBANK, N A | MORGAN STANLEY SENIOR FUNDING, INC | CORRECTIVE ASSIGNMENT TO CORRECT THE REMOVE APPLICATION 11759915 AND REPLACE IT WITH APPLICATION 11759935 PREVIOUSLY RECORDED ON REEL 037486 FRAME 0517 ASSIGNOR S HEREBY CONFIRMS THE ASSIGNMENT AND ASSUMPTION OF SECURITY INTEREST IN PATENTS | 053547 | /0421 | |
Dec 07 2015 | CITIBANK, N A | MORGAN STANLEY SENIOR FUNDING, INC | CORRECTIVE ASSIGNMENT TO CORRECT THE REMOVE APPLICATION 11759915 AND REPLACE IT WITH APPLICATION 11759935 PREVIOUSLY RECORDED ON REEL 037486 FRAME 0517 ASSIGNOR S HEREBY CONFIRMS THE ASSIGNMENT AND ASSUMPTION OF SECURITY INTEREST IN PATENTS | 053547 | /0421 | |
Dec 07 2015 | CITIBANK, N A , AS COLLATERAL AGENT | Freescale Semiconductor, Inc | PATENT RELEASE | 037354 | /0719 | |
Dec 07 2015 | CITIBANK, N A | MORGAN STANLEY SENIOR FUNDING, INC | ASSIGNMENT AND ASSUMPTION OF SECURITY INTEREST IN PATENTS | 037486 | /0517 | |
Dec 07 2015 | CITIBANK, N A | MORGAN STANLEY SENIOR FUNDING, INC | CORRECTIVE ASSIGNMENT TO CORRECT THE REMOVE PATENTS 8108266 AND 8062324 AND REPLACE THEM WITH 6108266 AND 8060324 PREVIOUSLY RECORDED ON REEL 037518 FRAME 0292 ASSIGNOR S HEREBY CONFIRMS THE ASSIGNMENT AND ASSUMPTION OF SECURITY INTEREST IN PATENTS | 041703 | /0536 | |
Jun 22 2016 | MORGAN STANLEY SENIOR FUNDING, INC | NXP B V | RELEASE BY SECURED PARTY SEE DOCUMENT FOR DETAILS | 040928 | /0001 | |
Jun 22 2016 | MORGAN STANLEY SENIOR FUNDING, INC | NXP B V | CORRECTIVE ASSIGNMENT TO CORRECT THE REMOVE APPLICATION 11759915 AND REPLACE IT WITH APPLICATION 11759935 PREVIOUSLY RECORDED ON REEL 040928 FRAME 0001 ASSIGNOR S HEREBY CONFIRMS THE RELEASE OF SECURITY INTEREST | 052915 | /0001 | |
Jun 22 2016 | MORGAN STANLEY SENIOR FUNDING, INC | NXP B V | CORRECTIVE ASSIGNMENT TO CORRECT THE REMOVE APPLICATION 11759915 AND REPLACE IT WITH APPLICATION 11759935 PREVIOUSLY RECORDED ON REEL 040928 FRAME 0001 ASSIGNOR S HEREBY CONFIRMS THE RELEASE OF SECURITY INTEREST | 052915 | /0001 | |
Sep 12 2016 | MORGAN STANLEY SENIOR FUNDING, INC | NXP, B V F K A FREESCALE SEMICONDUCTOR, INC | CORRECTIVE ASSIGNMENT TO CORRECT THE REMOVE APPLICATION 11759915 AND REPLACE IT WITH APPLICATION 11759935 PREVIOUSLY RECORDED ON REEL 040925 FRAME 0001 ASSIGNOR S HEREBY CONFIRMS THE RELEASE OF SECURITY INTEREST | 052917 | /0001 | |
Sep 12 2016 | MORGAN STANLEY SENIOR FUNDING, INC | NXP, B V F K A FREESCALE SEMICONDUCTOR, INC | CORRECTIVE ASSIGNMENT TO CORRECT THE REMOVE APPLICATION 11759915 AND REPLACE IT WITH APPLICATION 11759935 PREVIOUSLY RECORDED ON REEL 040925 FRAME 0001 ASSIGNOR S HEREBY CONFIRMS THE RELEASE OF SECURITY INTEREST | 052917 | /0001 | |
Sep 12 2016 | MORGAN STANLEY SENIOR FUNDING, INC | NXP, B V , F K A FREESCALE SEMICONDUCTOR, INC | RELEASE BY SECURED PARTY SEE DOCUMENT FOR DETAILS | 040925 | /0001 | |
Oct 06 2016 | NORTH STAR INNOVATIONS INC | TAIWAN SEMICONDUCTOR MANUFACTURING CO , LTD | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 041717 | /0736 | |
Feb 17 2019 | MORGAN STANLEY SENIOR FUNDING, INC | SHENZHEN XINGUODU TECHNOLOGY CO , LTD | CORRECTIVE ASSIGNMENT TO CORRECT THE TO CORRECT THE APPLICATION NO FROM 13,883,290 TO 13,833,290 PREVIOUSLY RECORDED ON REEL 041703 FRAME 0536 ASSIGNOR S HEREBY CONFIRMS THE THE ASSIGNMENT AND ASSUMPTION OF SECURITY INTEREST IN PATENTS | 048734 | /0001 | |
Feb 17 2019 | MORGAN STANLEY SENIOR FUNDING, INC | SHENZHEN XINGUODU TECHNOLOGY CO , LTD | CORRECTIVE ASSIGNMENT TO CORRECT THE TO CORRECT THE APPLICATION NO FROM 13,883,290 TO 13,833,290 PREVIOUSLY RECORDED ON REEL 041703 FRAME 0536 ASSIGNOR S HEREBY CONFIRMS THE THE ASSIGNMENT AND ASSUMPTION OF SECURITY INTEREST IN PATENTS | 048734 | /0001 | |
Sep 24 2019 | TAIWAN SEMICONDUCTOR MANUFACTURING CO , LTD | TAIWAN SEMICONDUCTOR MANUFACTURING CO , LTD | 323 01 C ASSIGNMENT OR CHANGE OF NAME IMPROPERLY FILED AND RECORDED BY ANOTHER PERSON AGAINST OWNER S PATENT | 052459 | /0656 |
Date | Maintenance Fee Events |
Apr 20 2015 | M1551: Payment of Maintenance Fee, 4th Year, Large Entity. |
Jun 10 2019 | REM: Maintenance Fee Reminder Mailed. |
Nov 25 2019 | EXP: Patent Expired for Failure to Pay Maintenance Fees. |
Date | Maintenance Schedule |
Oct 18 2014 | 4 years fee payment window open |
Apr 18 2015 | 6 months grace period start (w surcharge) |
Oct 18 2015 | patent expiry (for year 4) |
Oct 18 2017 | 2 years to revive unintentionally abandoned end. (for year 4) |
Oct 18 2018 | 8 years fee payment window open |
Apr 18 2019 | 6 months grace period start (w surcharge) |
Oct 18 2019 | patent expiry (for year 8) |
Oct 18 2021 | 2 years to revive unintentionally abandoned end. (for year 8) |
Oct 18 2022 | 12 years fee payment window open |
Apr 18 2023 | 6 months grace period start (w surcharge) |
Oct 18 2023 | patent expiry (for year 12) |
Oct 18 2025 | 2 years to revive unintentionally abandoned end. (for year 12) |