An adaptive predictive encoding apparatus for encoding a video signal by utilizing correlation between frames in both the forward and reverse directions of the time axis. A prediction signal for use in deriving prediction error values to be encoded for a frame is selected by an adaptive prediction section, in units of blocks, from a plurality of mutually differently derived prediction signals, in accordance with the degree of correlation of the block with corresponding ones of a specific preceding independently encoded frame and a specific succeeding independently encoded frame. A complementary adaptive decoding apparatus receives the encoded information and reconstructs the video signal in accordance with information supplied to the adaptive decoding apparatus by the encoding signal.

Patent
   RE35158
Priority
Apr 27 1989
Filed
Dec 28 1992
Issued
Feb 20 1996
Expiry
Feb 20 2013
Assg.orig
Entity
Large
95
2
all paid
1. An adaptive encoding apparatus for encoding an input video signal, said video signal comprising a sequence of frames each comprising successive pixel data the apparatus comprising:
encoder means for encoding successive blocks of a frame of said video signal, each of said blocks comprising a fixed-size array of said pixel data values;
means for selecting one in every N of said frames to be transferred directly to said encoder means as a reference frame, to be encoded by intra-frame encoding, where N is a fixed integer of value greater than one; and
adaptive prediction means for executing adaptive prediction processing, as a dependent frame, of each frame occurring between a preceding one and a succeeding one of said reference frames in said frame sequence, by deriving for the data values of each block of a dependent frame respective prediction error values based upon an optimum prediction signal selected from a plurality of prediction signals derived using a plurality of combinations of said preceding and succeeding reference frames.
2. An adaptive predictive encoding apparatus according to claim 1, in which said adaptive prediction means comprises:
means for deriving a first prediction signal based on a combination of pixel data of said preceding and succeeding reference frames, a second prediction signal derived only from said preceding reference frame, a third prediction signal derived only from said succeeding, and a non-prediction signal derived only from said dependent frame; and
predictive mode selection means for selecting, for each of said blocks, one out of four prediction modes in which said first, second and third prediction signals and said non-prediction signal are respectively used in deriving predictive error values for respective pixel data of said block, to be sent to said encoder means and encoded thereby, said selection being based upon judgement of said errors, said predictive mode selection means further supplying to said encoding means, to be encoded thereby, predictive mode data indicating predictive modes which have been selected for respective ones of the blocks.
3. An adaptive predictive encoding apparatus according to claim 2, in which said adaptive prediction means further comprises means for varying, in accordance with respective time axis positions of said frames in said video signal, respective weighting values assigned to said preceding and succeeding reference frames for establishing said combination.
4. An adaptive predictive encoding apparatus according to claim 1, and further comprising decoding means for decoding said reference frames after encoding by said encoding means, and for supplying resultant decoded reference frames to said adaptive prediction means for use in producing said prediction signals.
5. An adaptive predictive encoding apparatus according to claim 1, and further comprising an (N-1) frame memory for temporarily storing each dependent frame of said video signal and outputting said each dependent frame to said adaptive prediction means after a fixed delay time, and first and second 1-frame memories for respectively holding pixel data of said preceding and succeeding reference frames and supplying pixel data of said preceding and succeeding reference frames to said adaptive prediction means during adaptive prediction processing of successive ones of said dependent frames.
6. An adaptive decoding apparatus for decoding a video signal encoded by the apparatus of claim 1, comprising:
decoding means for receiving and decoding the encoded reference frames, the prediction error values and a prediction mode signal that identifies which of said plurality of combinations of said preceding and succeeding reference frames were used during encoding to obtain said optimum prediction signal;
prediction signal generating means responsive to the decoded prediction mode signal for reconstructing said optimum prediction signal;
means for combining said prediction error values and said optimum prediction signal for each dependent frame to generate display information corresponding thereto; and
means for outputting the decoded reference frames and the display information for each dependent frame in proper sequence to produce a video signal. 7. A decoding system for decoding video signals that have been encoded in an encoder by arranging said video signals into spaced-apart reference frames and dependent frames located therebetween; said reference frames being output from said encoder in encoded form and used therein to implement one of a plurality of prediction modes for adaptively predicting the display information in each of said dependent frames based upon the degree of correlation between each dependent frame and the reference frames which immediately precede and follow said dependent frame; said encoder thereby generating and outputting therefrom an encoded frame signal for each dependent frame and a prediction mode signal for identifying the prediction mode used to generate said frame signal, said decoding system comprising:
decoding means for receiving and decoding each of the encoded reference frames, and the frame signals and the prediction mode signals associated with each dependent frame;
processing means responsive to said prediction mode signal for reconstructing the display information for each dependent frame from its respective decoded frame signal and the reference frames which precede and follow said dependent frame; and
means for outputting the decoded reference frames and the reconstructed display information generated for each of said dependent frames in proper sequence to produce a video signal. 8. The decoding system in accordance with claim 7, wherein said processing means includes
memory means for storing the two decoded reference frames which respectively precede and follow each dependent frame; and
means responsive to the prediction mode signal for generating and combining weighted values of said display information from said two reference frames to reconstruct the predicted display information for said dependent frame.
9. The decoding system in accordance with claim 8, including means for combining said predicted display information with said decoded frame signal to produce an output representing the display information for said dependent frame. 10. The decoding system in accordance with claim 8, wherein said weighted values are generated by multiplying said display information in the preceding reference frame by a first weighting coefficient and said display information in the following reference frame by a second weighting coefficient. 11. The decoding system in accordance with claim 10, wherein, in response to a first prediction mode signal, the first and second weighting coefficients are non-zero. 12. The decoding system in accordance with claim 11 wherein the weighting coefficients are selected such that the reference frame temporally closer to the dependent frame is given a greater weight than the other reference frame. 13. The decoding system in accordance with claim 10, wherein, in response to a second prediction mode signal, the second weighting coefficient is effectively zero. 14. The decoding system in accordance with claim 10, wherein, in response to a third prediction mode signal, the first weighting coefficient is
effectively zero. 15. The decoding system in accordance with claim 7, wherein, in response to a fourth prediction mode signal, the decoded frame signal is output to represent the display information of said dependent frame. 16. A method for decoding and generating a video signal from video signals that have been encoded in an encoder by arranging said video signals into spaced-apart reference frames and dependent frames located therebetween, said reference frames being output from said encoder in encoded form and used therein to implement one of a plurality of prediction modes for adaptively predicting the display information in each of said dependent frames based upon the degree of correlation between each of said dependent frames and the reference frames which immediately precede and follow each of said dependent frames, said encoder generating and outputting therefrom an encoded frame signal for each of said dependent frames and a prediction mode signal for identifying the prediction mode used to generate said encoded frame signal, said method comprising the steps of:
(a) receiving and decoding each of said encoded reference frames, said encoded frame signals and said prediction mode signal associated with each dependent frame;
(b) reconstructing the display information for each dependent frame from the corresponding decoded frame signal and the decoded reference frames which precede and follow said dependent frame in accordance with said associated prediction mode signal; and
(c) outputting the decoded reference frames and the reconstructed display information generated for each of said dependent frames in proper sequence
to generate a video signal. 17. The method of claim 16 wherein step (b) includes the steps of storing the display information of the two decoded reference frames which respectively precede and follow said dependent frame and combining weighted values of said display information with said decoded frame signal to reconstruct the display information for said dependent frame. 18. The method in accordance with claim 17, wherein said weighted values are generated by multiplying said display information of the preceding reference frame by a first weighting coefficient and said display information in the following reference frame by a second weighting coefficient. 19. The method in accordance with claim 18, wherein, in response to a first prediction mode signal, the first and second weighting coefficients are non-zero. 20. The method in accordance with claim 19, wherein the weighting coefficients are selected such that the reference frame temporally closer to the dependent frame is given a greater weight
than the other reference frame. 21. The method in accordance with claim 18, wherein, in response to a second prediction mode signal, the second weighting coefficient is effectively zero. 22. The method in accordance with claim 18, wherein, in response to a third prediction mode signal, the first weighting coefficient is effectively zero. 23. The method in accordance with claim 16, wherein, in response to a fourth prediction mode signal, the decoded frame signal is added to a fixed data value to reconstruct the display information of said dependent frame.

The present application is a FIG. 6 is a block diagram of a decoding apparatus for decoding signals in accordance with the present invention.

FIG. 7 is a schematic diagram of the adaptive prediction section of the decoding apparatus of FIG. 6. (not shown in the drawings) shown in FIG. 6, the various quantization errors etc. that are present in the final decoded data will also be present in the data that are used in denying the prediction error values. This ensures a greater accuracy of prediction than would be the case if the independent frames of the input video signal data were to be written directly into the memories 12 and then 13.

With this embodiment, evaluation for determining the prediction mode is based upon error-squared values of prediction error values that are obtained directly from the input video signal. Greater accuracy of evaluation would be obtained by using the video signal data of the dependent frames after all of the encoding processing (including transform processing, and quantization) has been executed. However this would require additional circuits for executing the inverse of such encoding, i.e. for the inverse transform processing etc., increasing the circuit scale substantially and making the apparatus more difficult to realize in practical form.

As stated above, the DC component of the current frame signal is subtracted from the current frame signal in the subtractor 39, to thereby prevent an excessively high output value being produced by the cumulative adder 31. However if not compensated for, this will tend to produce an excessively high probability that mode 4 will be selected by the minimum value selector circuit 32, i.e. the output from the cumulative adder 31 will tend to have too low a value. For that reason, a compensating offset value B is added to the output from the cumulative adder 31 in the adder 37.

On the other hand, in cases where there are only small differences between the respective values of prediction error that are being produced from the prediction signal subtractor 21, subtractor 20 and subtractor 22, it is preferable to prevent unnecessary switching between the modes 1, 2 and 3. For that reason, a slight amount of bias is given towards the selection of mode 1 (2-dimensional linear prediction) by the minimum value selector circuit 32. This is done by subtracting an offset value A from the output of the cumulative adder 29, in the subtractor 36. This has the advantage of increasing the rate of selection of mode 1, and so enabling a reduction in the amount of encoded data that are produced by encoding the prediction mode data from the minimum value selector circuit 32, if entropy encoding using for example the Huffman code is employed in the variable-length encoder section 8.

The As shown in FIG. 6, the decoding apparatus for decoding the encoded data on line 48 that are transmitted from such an adaptive predictive encoded apparatus can be implemented very simply, by using the mode prediction data that are contained in the encoded output data. After the inverse of the variable-length encoding executed by the variable-length encoder section 8 has been performed, in variable length decoder 50, followed by dequantizing in dequantizer 51 and inverse transform processing in inverse orthogonal transform section 52, each independent frame is transferred successively to a first (succeeding) frame memory 54 and then to a second frame memory for use in processing (preceding) frame memory 55 for use in processing the dependent frames, corresponding to the memories 12 and 13 of FIG. 1 3, and are outputted on line 70 without further processing. The operation of switches 61 and 62 correspond to the operation of switches 2 and 5 in the encoder of FIG. 3. During processing of the independent frames, their contacts are in the respective positions "b" shown in FIG. 6 and during processing of the dependent frames, their contacts are switched to their respective "a" positions. Each block of a dependent frame is processed by performing the inverse of the operations performed by the adaptive prediction section 4 of FIG. 3 during the encoding process in adaptive prediction section 53 of the decoder, shown schematically in FIG. 7, depending upon the position of the movable contact of switch 64 which is controlled by the associated decoded prediction mode data for that block, as follows:

(1) If the prediction mode data on line 57 indicates that the block has been encoded in mode 1, then the pixel data values of the corresponding blocks of the corresponding preceding and succeeding independent frame (read out on lines 58 and 59 from the aforementioned two frame memories 54 and 55) are respectively multiplied by the weighting values W and (1-W), the results added in adder 63, and the resultant value added in adder 65 to the current frame signal supplied on line 56 to generate the output of the decoder on line 60.

(2) If the prediction mode data on line 57 indicates that the block has been encoded in mode 2, then the pixel data values of the corresponding blocks of the corresponding preceding independent frame on line 58 are added in adder 65 to the current frame signal on line 56 to generate the output of the decoder on line 60.

(3) If the prediction mode data on line 57 indicates that the block has been encoded in mode 3, then the pixel data values of the corresponding blocks of the corresponding succeeding independent frame supplied on line 59 are added in adder 65 to the current frame signal on line 56 to generate the output of the decoder on line 60.

(It will be apparent that a single circuit can be used to implement all of the functions (1), (2) and (3) above, by appropriately setting the weighting value W to either 1 or 0 for functions (2) and (3)).

(4) If the prediction mode data on line 57 indicates that the block has been encoded in mode 4, then the fixed value on line 66 (subtracted in the subtractor 23 of FIG. 2 of the encoder apparatus) is added in adder 65 to the current frame signal on line 56 to generate the output of the decoder on line 60.

It will be apparent that the decoder apparatus shown in FIGS. 6 and 7 for receiving an encoded output signal produced by an adaptive predictive encoder apparatus according to the present invention can have a simple configuration, and can for example by implemented by slightly modifying an encoder apparatus that is described in the aforementioned related U.S. application by the assignee of the present invention.

Sugiyama, Kenji

Patent Priority Assignee Title
10063863, Jul 18 2003 Microsoft Technology Licensing, LLC DC coefficient signaling at small quantization step sizes
10116959, Jun 03 2002 Microsoft Technology Licesning, LLC Spatiotemporal prediction for bidirectionally predictive (B) pictures and motion vector prediction for multi-picture reference motion compensation
10284843, Jan 25 2002 Microsoft Technology Licensing, LLC Video coding
10368065, Dec 17 2001 Microsoft Technology Licensing, LLC Skip macroblock coding
10382750, Nov 18 2005 Apple Inc. Region-based processing of predicted pixels
10554985, Jul 18 2003 Microsoft Technology Licensing, LLC DC coefficient signaling at small quantization step sizes
10659793, Jul 18 2003 Microsoft Technology Licensing, LLC DC coefficient signaling at small quantization step sizes
6317459, Mar 14 1997 Microsoft Technology Licensing, LLC Digital video signal encoder and encoding method
6351545, Dec 14 1999 NIO INC Motion picture enhancing system
6408029, Apr 02 1998 Intel Corporation Method and apparatus for simplifying real-time data encoding
6449352, Jun 20 1995 MATSUSHITA ELECTRIC INDUSTRIAL CO , LTD Packet generating method, data multiplexing method using the same, and apparatus for coding and decoding of the transmission data
6574278, Apr 02 1998 Intel Corporation Method and apparatus for performing real-time data encoding
6584226, Mar 14 1997 Microsoft Technology Licensing, LLC Method and apparatus for implementing motion estimation in video compression
6647425, Jul 03 1997 Microsoft Technology Licensing, LLC System and method for selecting the transmission bandwidth of a data stream sent to a client based on personal attributes of the client's user
6707852, Mar 14 1997 Microsoft Technology Licensing, LLC Digital video signal encoder and encoding method
6904174, Dec 11 1998 Intel Corporation Simplified predictive video encoder
6937657, Mar 14 1997 Microsoft Technology Licensing, LLC Motion video signal encoder and encoding method
7046734, Apr 02 1998 Intel Corporation Method and apparatus for performing real-time data encoding
7054494, Nov 30 1998 Microsoft Technology Licensing, LLC Coded block pattern decoding with spatial prediction
7072396, Mar 14 1997 Microsoft Technology Licensing, LLC Motion video signal encoder and encoding method
7092576, Sep 07 2003 Microsoft Technology Licensing, LLC Bitplane coding for macroblock field/frame coding type information
7099515, Sep 07 2003 Microsoft Technology Licensing, LLC Bitplane coding and decoding for AC prediction status information
7127114, Nov 30 1998 Microsoft Technology Licensing, LLC Coded block pattern encoding with spatial prediction
7139313, Mar 14 1997 Microsoft Technology Licensing, LLC Digital video signal encoder and encoding method
7154951, Mar 14 1997 Microsoft Technology Licensing, LLC Motion video signal encoder and encoding method
7200275, Dec 17 2001 Microsoft Technology Licensing, LLC Skip macroblock coding
7215384, Apr 02 1998 Intel Corporation Method and apparatus for simplifying field prediction motion estimation
7224731, Jun 28 2002 ZHIGU HOLDINGS LIMITED Motion estimation/compensation for screen capture video
7231091, Sep 21 1998 Intel Corporation Simplified predictive video encoder
7263127, Apr 02 1998 Intel Corporation Method and apparatus for simplifying frame-based motion estimation
7280700, Jul 05 2002 Microsoft Technology Licensing, LLC Optimization techniques for data compression
7289673, Nov 30 1998 Microsoft Technology Licensing, LLC Decoding macroblock type and coded block pattern information
7317839, Sep 07 2003 Microsoft Technology Licensing, LLC Chroma motion vector derivation for interlaced forward-predicted fields
7352905, Sep 07 2003 Microsoft Technology Licensing, LLC Chroma motion vector derivation
7379607, Dec 17 2001 Microsoft Technology Licensing, LLC Skip macroblock coding
7408990, Nov 30 1998 Microsoft Technology Licensing, LLC Efficient motion vector coding for video compression
7426308, Jul 18 2003 Microsoft Technology Licensing, LLC Intraframe and interframe interlace coding and decoding
7499495, Jul 18 2003 Microsoft Technology Licensing, LLC Extended range motion vectors
7529302, Sep 07 2003 Microsoft Technology Licensing, LLC Four motion vector coding and decoding in bi-directionally predicted interlaced pictures
7555167, Dec 17 2001 Microsoft Technology Licensing, LLC Skip macroblock coding
7567617, Sep 07 2003 Microsoft Technology Licensing, LLC Predicting motion vectors for fields of forward-predicted interlaced video frames
7577200, Sep 07 2003 Microsoft Technology Licensing, LLC Extended range variable length coding/decoding of differential motion vector information
7590179, Sep 07 2003 Microsoft Technology Licensing, LLC Bitplane coding of prediction mode information in bi-directionally predicted interlaced pictures
7599438, Sep 07 2003 Microsoft Technology Licensing, LLC Motion vector block pattern coding and decoding
7606308, Sep 07 2003 Microsoft Technology Licensing, LLC Signaling macroblock mode information for macroblocks of interlaced forward-predicted fields
7606311, Sep 07 2003 Microsoft Technology Licensing, LLC Macroblock information signaling for interlaced frames
7609763, Jul 18 2003 Microsoft Technology Licensing, LLC Advanced bi-directional predictive coding of video frames
7616692, Sep 07 2003 Microsoft Technology Licensing, LLC Hybrid motion vector prediction for interlaced forward-predicted fields
7620106, Sep 07 2003 Microsoft Technology Licensing, LLC Joint coding and decoding of a reference field selection and differential motion vector information
7623574, Sep 07 2003 Microsoft Technology Licensing, LLC Selecting between dominant and non-dominant motion vector predictor polarities
7630438, Sep 07 2003 Microsoft Technology Licensing, LLC Direct mode motion vectors for Bi-directionally predicted interlaced pictures
7646810, Jan 25 2002 Microsoft Technology Licensing, LLC Video coding
7664177, Sep 07 2003 Microsoft Technology Licensing, LLC Intra-coded fields for bi-directional frames
7664871, Mar 14 1997 Microsoft Technology Licensing, LLC Methods and apparatus for communication media commands and data using the HTTP protocol
7680185, Sep 07 2003 Microsoft Technology Licensing, LLC Self-referencing bi-directionally predicted frames
7738554, Jul 18 2003 Microsoft Technology Licensing, LLC DC coefficient signaling at small quantization step sizes
7761585, Mar 14 1997 Microsoft Technology Licensing, LLC Techniques for automatically detecting protocols in a computer network
7826530, Jan 14 2005 AVAGO TECHNOLOGIES GENERAL IP SINGAPORE PTE LTD Use of out of order encoding to improve video quality
7852936, Sep 07 2003 Microsoft Technology Licensing, LLC Motion vector prediction in bi-directionally predicted interlaced field-coded pictures
7924920, Sep 07 2003 Microsoft Technology Licensing, LLC Motion vector coding and decoding in interlaced frame coded pictures
7925774, May 30 2008 ZHIGU HOLDINGS LIMITED Media streaming using an index file
7949775, May 30 2008 ZHIGU HOLDINGS LIMITED Stream selection for enhanced media streaming
8031777, Nov 18 2005 Apple Inc Multipass video encoding and rate control using subsampling of frames
8064520, Sep 07 2003 Microsoft Technology Licensing, LLC Advanced bi-directional predictive coding of interlaced video
8189666, Feb 02 2009 Microsoft Technology Licensing, LLC Local picture identifier and computation of co-located information
8233535, Nov 18 2005 Apple Inc Region-based processing of predicted pixels
8254455, Jun 30 2007 Microsoft Technology Licensing, LLC Computing collocated macroblock information for direct mode macroblocks
8290288, Nov 30 1998 Microsoft Technology Licensing, LLC Encoding macroblock type and coded block pattern information
8295343, Nov 18 2005 Apple Inc Video bit rate control method
8370887, May 30 2008 ZHIGU HOLDINGS LIMITED Media streaming with enhanced seek operation
8374245, Jun 03 2002 Microsoft Technology Licensing, LLC Spatiotemporal prediction for bidirectionally predictive(B) pictures and motion vector prediction for multi-picture reference motion compensation
8379722, Jul 19 2002 Microsoft Technology Licensing, LLC Timestamp-independent motion vector prediction for predictive (P) and bidirectionally predictive (B) pictures
8406300, Jan 25 2002 Microsoft Technology Licensing, LLC Video coding
8428374, Dec 17 2001 Microsoft Technology Licensing, LLC Skip macroblock coding
8582903, Nov 30 1998 Microsoft Technology Licensing, LLC Efficient macroblock header coding for video compression
8625669, Sep 07 2003 Microsoft Technology Licensing, LLC Predicting motion vectors for fields of forward-predicted interlaced video frames
8638853, Jan 25 2002 Microsoft Technology Licensing, LLC Video coding
8687697, Jul 18 2003 Microsoft Technology Licensing, LLC Coding of motion vector information
8774280, Jul 19 2002 Microsoft Technology Licensing, LLC Timestamp-independent motion vector prediction for predictive (P) and bidirectionally predictive (B) pictures
8780997, Nov 18 2005 Apple Inc Regulation of decode-side processing based on perceptual masking
8781240, Dec 17 2001 Microsoft Technology Licensing, LLC Skip macroblock coding
8819754, May 30 2008 ZHIGU HOLDINGS LIMITED Media streaming with enhanced seek operation
8873630, Jun 03 2002 Microsoft Technology Licensing, LLC Spatiotemporal prediction for bidirectionally predictive (B) pictures and motion vector prediction for multi-picture reference motion compensation
8917768, Jul 18 2003 Microsoft Technology Licensing, LLC Coding of motion vector information
9049451, Nov 18 2005 Apple Inc. Region-based processing of predicted pixels
9077960, Aug 12 2005 Microsoft Technology Licensing, LLC Non-zero coefficient block pattern coding
9088785, Dec 17 2001 Microsoft Technology Licensing, LLC Skip macroblock coding
9148668, Jul 18 2003 Microsoft Technology Licensing, LLC Coding of motion vector information
9185427, Jun 03 2002 Microsoft Technology Licensing, LLC Spatiotemporal prediction for bidirectionally predictive (B) pictures and motion vector prediction for multi-picture reference motion compensation
9313509, Jul 18 2003 Microsoft Technology Licensing, LLC DC coefficient signaling at small quantization step sizes
9538189, Dec 17 2001 Microsoft Technology Licensing, LLC Skip macroblock coding
9571854, Jun 03 2002 Microsoft Technology Licensing, LLC Spatiotemporal prediction for bidirectionally predictive (B) pictures and motion vector prediction for multi-picture reference motion compensation
9706201, Nov 18 2005 Apple Inc. Region-based processing of predicted pixels
9774852, Dec 17 2001 Microsoft Technology Licensing, LLC Skip macroblock coding
9888237, Jan 25 2002 Microsoft Technology Licensing, LLC Video coding
Patent Priority Assignee Title
4383272, Apr 13 1981 MULTIMEDIA PATENT TRUST C O Video signal interpolation using motion estimation
4651207, Mar 05 1984 ANT Nachrichtentechnik GmbH Motion adaptive interpolation of television image sequences
/
Executed onAssignorAssigneeConveyanceFrameReelDoc
Dec 28 1992Victor Company of Japan Limited(assignment on the face of the patent)
Date Maintenance Fee Events
Jun 12 1996ASPN: Payor Number Assigned.
Jun 22 1998M184: Payment of Maintenance Fee, 8th Year, Large Entity.
Jun 07 2002M185: Payment of Maintenance Fee, 12th Year, Large Entity.


Date Maintenance Schedule
Feb 20 19994 years fee payment window open
Aug 20 19996 months grace period start (w surcharge)
Feb 20 2000patent expiry (for year 4)
Feb 20 20022 years to revive unintentionally abandoned end. (for year 4)
Feb 20 20038 years fee payment window open
Aug 20 20036 months grace period start (w surcharge)
Feb 20 2004patent expiry (for year 8)
Feb 20 20062 years to revive unintentionally abandoned end. (for year 8)
Feb 20 200712 years fee payment window open
Aug 20 20076 months grace period start (w surcharge)
Feb 20 2008patent expiry (for year 12)
Feb 20 20102 years to revive unintentionally abandoned end. (for year 12)