A new predictive coding is used to increase the temporal frame rate and coding efficiency without introducing excessive delay. Currently the motion vector for the blocks in the bi-directionally predicted frame is derived from the motion vector of the corresponding block in the forward predicted frame using a linear motion model. This however is not effective when the motion in the image sequence is not linear. The efficiency of this method can be further improved if a non-linear motion model is used. In this model a delta motion vector is added to or subtracted from the derived forward and backward motion vector, respectively. The encoder performs an additional search to determine if there is a need for the delta motion vector. The presence of this delta motion vector in the transmitted bitstream is signalled to the decoder which then takes the appropriate action to make use of the delta motion vector to derive the effective forward and backward motion vectors for the bi-directionally predicted block.
|
0. 13. A method for decoding a compressed video image sequence of a group of pictures including an i-frame followed by a plurality of p-frames and b-frames, comprising the steps of:
decoding a block in the i-frame independently from any other frames in the group of pictures;
predictively decoding a block in a p-frame based on the previous i-frame or a previous p-frame;
bi-directionally predictively decoding a block in a b-frame based on the previous i-frame or a previous p-frame and a block in a p-frame positioned after the b-frame;
deriving a scaled forward motion vector and a scaled backward motion vector for the block in the b-frame by scaling a motion vector of the block in the p-frame positioned after the b-frame;
obtaining a final forward motion vector for the block in the b-frame by adding a delta motion vector to the scaled forward motion vector; and
obtaining a final backward motion vector for the block in the b-frame by adding the delta motion vector to the scaled backward motion vector.
0. 1. A method for encoding a sequence of video image frames comprising the steps of:
dividing a source sequence into a set of group of pictures, each group of pictures comprising a first frame (i-frame) followed by a plurality of pairs of predictively encoded frames (PB-frame pairs), each PB-frame pair having a corresponding p-block;
dividing each i-frame or PB-frame pair into a plurality of spatially non-overlapping blocks of pixel data;
encoding the blocks from the i-frame (i-blocks) independently from any other frames in the group of pictures;
predictively encoding the blocks from the second frame of the PB-frame pair (p-blocks), based on the i-blocks in the previous i-frame or the p-blocks in the previous PB-frame pair;
bi-directionally predictively encoding the blocks from the first frame of the PB-frame pair (b-blocks), based on the i-blocks in the previous i-frame or the p-blocks in the previous PB-frame pair and the corresponding p-block in the current PB-frame pair;
deriving a sealed forward motion vector and a sealed backward motion vector of the b-block by sealing the motion vector of the corresponding p-block in the current PB-frame pair;
obtaining a final forward motion vector for the b-block by adding a delta motion vector on the sealed forward motion vector; and
obtaining a final backward motion vector for the b-block by subtracting the delta motion vector from the sealed backward motion vector.
0. 2. A method for encoding a sequence of video image frames according to
the sealing of the motion vector is based on a temporal reference of the first and second frames of the PB-frame pair.
0. 3. A method for encoding a sequence of video image frames according to
temporal reference information for the first and second frames of the PB-frame pairs;
motion vector information for the p-blocks;
quantized residual error information for the p-blocks;
delta motion vector information for the b-blocks; and
quantized residual error information for the b-blocks.
0. 4. A method for encoding a sequence of video image frames according to
the output bitstream contains additional information to indicate the presence of at least one of:
the delta motion vector information for the b-blocks; and
the quantized residual error information for the b-blocks.
0. 5. A method for decoding a sequence of video image frames comprising the steps of:
decoding the compressed video image sequence as a set of group of pictures, each group of pictures comprising an i-frame followed by a plurality of PB-frame pairs, each PB-frame pair having a corresponding p-block;
decoding each i-frame or PB-frame pair into a plurality of spatially non-overlapping blocks of pixel data;
decoding the i-blocks from the i-frame independently from any other frames in the group of pictures;
predictively decoding the p-block from the second frame of the PB-frame pair based on the i-blocks in the previous i-frame or the p-blocks in the previous PB-frame pair;
bi-directionally predictively decoding the b-blocks from the first frame of the PB-frame pair based on the i-blocks in the previous i-frame or the p-blocks in the previous PB-frame pair and the corresponding p-block in the current PB-frame pair;
driving a sealed forward motion vector and a sealed backward motion vector for the b-block by sealing the motion vector of the corresponding p-block in the current PB-frame pair;
obtaining a final forward motion vector for the b-block by adding a delta motion vector to the sealed forward motion vector; and
obtaining a final backward motion vector for the b-block by subtracting the delta motion vector from the sealed backward motion vector.
0. 6. A method for decoding a sequence of video image frames according to
temporal reference information for the first and second frames of the PB-frame pairs;
motion vector information for the p-blocks;
quantized residual error information for the p-blocks;
the delta motion vector information for the b-blocks; and
quantized residual error information for the b-blocks.
0. 7. A method for decoding a sequence of video image frames according to
the bitstream contains additional information to indicate the presence of at least one of:
the delta motion vector information for the b-blocks; and
the quantized residual error information for the b-block.
0. 8. A method of decoding a sequence of video image frames according to
the sealing is based on a temporal reference of the first and second frames of the PB-frame pair.
0. 9. An apparatus for encoding a sequence of video image frames comprising:
means for encoding each frame in a sequence of video image frames into a set of group of pictures, each group of pictures comprising an i-frame followed by a plurality of PB-frame pairs;
means for dividing the i-frame and the PB-frame pair into a plurality of spatially non-overlapping blocks of pixel data;
means for encoding and decoding the i-blocks of the i-frame independently from any other frames in the group of pictures;
means for storing the decoded i-blocks to predictively encode subsequent frames;
means for predictively encoding and decoding the p-blocks of the second frame of the PB-frame pair based on the i-blocks in the previous i-frame or the p-blocks in the previous PB-frame pair;
means for storing the decoded p-block to predictively encode subsequent frames;
means of deriving a sealed forward motion vector and a sealed backward motion vector for a b-block by sealing the motion vector of the corresponding p-block in the current PB-frame pair, the b-block being the first frame of the PB-frame pair;
means for obtaining a final forward motion vector for the b-block by adding a delta motion vector to the sealed forward motion vector;
means for obtaining a final backward motion vector for the b-block by subtracting the same delta motion vector from the sealed backward motion vector; and
means for encoding the b-blocks of the first frame of the PB-frame pairs based on the i-blocks in the previous i-frame or the p-blocks in the previous PB-frame pair and the corresponding p-block in the current PB-frame pair using the final forward motion vector and the final backward motion vector.
0. 10. An apparatus for decoding a sequence of video image frames comprising:
means for decoding each frame in a sequence of video image frames into a set of group of pictures, each group of pictures composing an i-frame followed by a plurality of PB-frame pairs;
means for decoding the i-blocks of the i-frame independently of any other frames in the group of pictures;
means for storing the decoded i-blocks to predictively decode subsequent frames;
means for decoding the p-blocks of the second frame of the PB-frame pair based on the i-blocks in the previous i-frame or the p-blocks in the previous PB-frame pair;
means for storing the decoded p-blocks to predictively decode subsequent frames;
means for deriving a sealed forward motion vector and a sealed backward motion vector for a b-block by sealing the motion vector of the corresponding p-block in the current PB-frame pair, the b-block being the first frame of the PB-frame pair;
means for obtaining final forward motion vector for the b-block by adding a delta motion vector to the sealed forward motion vector;
means for obtaining a final backward motion vector for the b-block by subtracting the delta motion vector to the sealed backward motion vector; and
means for decoding the b-blocks of the first frame of the PB-frame pairs based on the i-blocks in the previous i-frame of the p-blocks in the previous PB-frame pair and the corresponding p-block in the current PB-frame pair using the final forward motion vector and the final backward motion vector.
0. 11. A method for encoding a sequence of video image frames comprising the steps of:
dividing a source sequence into a plurality of groups of pictures, each group of pictures comprising a first frame (i-frame) followed by a plurality of pairs of predictively encoded frames (PB-frame pairs);
dividing each i-frame or PB-frame pair into a plurality of blocks;
encoding the blocks from the i-frame;
predictively encoding the blocks from the second frame of the PB-frame pair;
bi-directionally predictively encoding the blocks from the first frame of a PB-frame pair (b-blocks);
deriving a sealed forward motion vector and a sealed backward motion vector for the b-block;
obtaining a final forward motion vector for the b-block by adding a delta motion vector to the sealed forward motion vector; and
obtaining a final backward motion vector for the b-block by subtracting the delta motion vector from the sealed backward motion vector.
0. 12. An apparatus for encoding a sequence of video image frames comprising:
means for dividing a source sequence into a plurality of groups of pictures, each group of pictures comprising a first frame (i-frame) followed by a plurality of pairs of predictively encoded frames (PB-frame pairs);
means for dividing each i-frame or PB-frame pair into a plurality of blocks;
means for encoding the blocks from the i-frame;
means for predictively encoding the blocks from the second frame of the PB-frame pair;
means for bi-directionally predictively encoding the blocks from the first frame of a PB-frame pair (b-blocks);
means for deriving a sealed forward motion vector and a sealed backward motion vector for the b-block;
means for obtaining a final forward motion vector for the b-block by adding a delta motion vector to the sealed forward motion vector; and
means for obtaining a final backward motion vector for the b-block by subtracting the delta motion vector from the sealed backward motion vector.
0. 14. A method of decoding a sequence of video image frames according to
scaling of the forward and backward motion vectors is based on a temporal reference of the b-frame and the p-frame.
0. 15. A method for decoding a sequence of video image frames according to
temporal reference information for the b-frame and the p-frame;
motion vector information for the block in the p-frame;
quantized residual error information for the block in the p-frame;
the delta motion vector information for the block in the b-frame; and
quantized residual error information for the block in the b-frame.
0. 16. A method for decoding a sequence of video image frames according to
the bitstream contains additional information indicating a presence of at least one of
the delta motion vector information for the block in the b-frame; and
the quantized residual error information for the block in the b-frame.
|
MVB=((TRB−TRP)×MV)/TRP (2)
where
Currently the method used in the prior art assumes a linear motion model. However this assumption is not valid in a normal scene where the motion is typically not linear. This is especially true when the camera shakes and when objects are not moving at constant velocities.
A second problem involves the quantization and transmission of the residual of the predication error in the B-block Currently the coefficients from the P-block and the B-block are interleaved in some scanning order which requires the B-block efficients to be transmitted even when they are all zero. This is not very efficient as it is quite often that there are no residual coefficients to transmit (all coefficients are zero).
In order to solve the first problem, the current invention employs a delta motion vector to compensate for the non-linear motion. Thus it becomes necessary for the encoder to perform an additional motion search to obtain the optimum delta motion vector that when added to the derived motion vectors would result in the best match in the prediction. This delta motion vectors are transmitted to the decoder at the block level only when necessary. A flag is used to indicate to the decoder if there are delta motion vectors present for the B-block.
For the second problem, this invention also uses a flag to indicate if there are coefficients for the B-block to be decoded.
The operation of the Invention is described as follows.
In the current invention the problem is solved by adding a small delta motion vector to the derived motion vector to compensate for the difference between the derived and true motion vector. Therefore the equation in (1) and (2) are now replaced by equations (3) and (4), respectively.
MVF′=(TRB×MV)/TRP+MVDelta (3)
MVB′=((TRB=TRP)×MV)/TRP−MVDelta (4)
where
The preferred embodiment of the current invention is described here.
The encoding functionality block diagram depicts an encoder using a motion estimation and compensation for reducing the temporal redundancy in the sequence to be coded. The input sequences is organized into a first frame and pairs of subsequent frames. The first frame, hereafter referred to as the I-frame, is coded independent of all other frames. The pair of subsequent frames, hereafter referred to as PB-frame, consist of a B-frame followed by a P-frame. The P-frame is forward predicted based on the previously reconstructed I-frame or P-frame and the B-frame is bi-directionally predicted based on the previously reconstructed I-frame or P-frame and the information in the current P-frame.
The input frame image sequence, 1, is placed in the Frame Memory 2. If the frame is classified as an I-frame or a P-frame it is passed through line 14 to the Reference Memory 3, for use as the reference frame in the motion estimation of the next PB-frame to be predictively encoded. The signal is then passed through line 13 to the Block Sampling module 4, where it is partitioned into spatially non-overlapping blocks of pixel data for further processing.
If the frame is classified as an I-frame, the sampled blocks are passed through line 16 to the DCT module 7. If the frame is classified as a PB-frame, the sampled blocks are passed through line 17 to the Motion Estimation module 5. The Motion Estimation module 5 uses information from the Reference Frame Memory 3 and the current block 17 to obtain the motion vector for that provides the best match for the P-block. The motion vector and the local reconstructed frame, 12, are passed through line 19 and 20, respectively, to the Motion Compensation module 6. The difference image is formed by subtracting the motion compensated decoded frame, 21, from the current P-block 15. This signal is then passed through line 22 to the DCT module 7.
In the DCT module 7, each block is transformed into the DCT domain coefficients. The transform coefficienta are passed through line 23 to Quantization module 8, where they are quantizied. The quantizied coefficients are then passed though line 24 to the Run-length & Variable Length Coding module 9. Here the coefficients are entropy coded to form the Output Bit Stream 25.
If the current block is an I-block or a P-block, the quantized coefficients are also passed through line 26 to the Inverse Quantization module 10. The output of the Inverse Quantization 10, is then passed through line 27 to the Inverse DCT module 11. If the current block is an I-block then the reconstructed block is placed, via line 28, in the Local Decoded Frame Memory 12. If the current block is a P-block then the output of the Inverse DCT 29 is added to the motion compensated output 21, to from the reconstructed block 30. The reconstructed block 30, is then placed in the Local Decoded Frame Memory 12, for the motion compensation of the subsequent frames.
After the P-block have been locally reconstructed, the information is passed again to the Motion Compensation Module 6, where the prediction of the B-block is formed.
The reduction difference block is then passed through line 22 to the DCT module 7. The DCT coefficients are then passed through line 23 to the Quantization module 8. The result of the Quantization module 8, is passed through line 24 to the Run-length & Variable Length Coding 9. In this module the presence of the delta motion vector and the quantized residual error in the Output Bitstream 25, is indicated a variable length code. NOB which is the acronym for No B-block. This flag is generated in Run-length & Variable Length Coding module 9 based on whether there are residual error in the Quantization module 8 and delta motion vectors found in the Delta Motion Search module 54 is not zero. Table 1 provides the preferred embodiment of the variable length code for the NOB flag. The variable length code of the NOB flag is inserted in the Output Bitstream, 25, prior to the delta motion vector an quantized residual error codes.
TABLE 1
(Variable length code for the NOB flag)
Quantized Residual
Delta Motion
NOB
Error Coded
Vectors Coded
0
No
No
10
No
Yes
110
Yes
No
111
Yes
Yes
If the current frame is an I-frame then the output of Inverse DCT 34, is passed through line 39 and stored in the Frame Memory 42.
If the current frame is a PB-frame, the side information containing the motion vector are passe through line 45 to the Motion compensation module 36. The motion Compensation module 36, uses this information and the information in the Local Decoded Memory, 35, to from the motion compensated signal, 44. This signal is then added to the output of the Inverse DCT module 34, to form the reconstruction of the P-block.
The Motion Compensation module 36, then uses the additional information obtained in the reconstructed P-block to obtain the bi-directional prediction for the B-block. The B-block is then reconstructed and placed in the Frame Memory, 42, together with the P-block.
By implementing this invention, the temporal frame rate of the decoded sequences can be effectively doubled at a fraction of the expected cost in bit rate. The delay is similar to that of the same sequence decoded at half the frame rate.
As described above in the present invention a new predictive coding is used to increase the temporal frame rate and coding efficiency without introducing excessive delay. Currently the motion vector for the blocks in the bi-directionally predicted frame is derived from the motion vector of the corresponding block in the forward predicted frame using a linear motion model. This however is not effective when the motion in the image sequence is not linear. According to this invention, the efficiency of this method can be further improved if a non-linear motion model is used. In this model a delta motion vector is added to or subtracted from the derived forward and backward motion vector, respectively. The encoder performs an additional search to determine if there is a need for the delta motion vector. The presence of this delta motion vector in the transmitted bitstream is sinaglled to the decoder which then takes the appropriate action to make use of the delta motion vector to derive the effective forward and backward motion vectors for the bi-directionally predicted block.
Patent | Priority | Assignee | Title |
10244254, | Mar 07 2008 | SK PLANET CO , LTD | Encoding system using motion estimation and encoding method using motion estimation |
10334271, | Mar 07 2008 | SK Planet Co., Ltd. | Encoding system using motion estimation and encoding method using motion estimation |
10341679, | Mar 07 2008 | SK Planet Co., Ltd. | Encoding system using motion estimation and encoding method using motion estimation |
10412409, | Mar 07 2008 | SK Planet Co., Ltd. | Encoding system using motion estimation and encoding method using motion estimation |
10476928, | May 05 2014 | Huawei Technologies Co., Ltd. | Network video playback method and apparatus |
8107533, | Nov 06 2001 | Panasonic Intellectual Property Corporation of America | Moving picture coding method, and moving picture decoding method |
8126056, | Nov 06 2001 | Panasonic Intellectual Property Corporation of America | Moving picture coding method, and moving picture decoding method |
8126057, | Nov 06 2001 | Panasonic Intellectual Property Corporation of America | Moving picture coding method, and moving picture decoding method |
8194747, | Nov 06 2001 | Panasonic Intellectual Property Corporation of America | Moving picture coding method, and moving picture decoding method |
8213517, | Nov 06 2001 | Panasonic Intellectual Property Corporation of America | Moving picture coding method, and moving picture decoding method |
8265153, | Nov 06 2001 | Panasonic Intellectual Property Corporation of America | Moving picture coding method, and moving picture decoding method |
8964839, | Nov 06 2001 | Panasonic Intellectual Property Corporation of America | Moving picture coding method, and moving picture decoding method |
9078003, | Nov 06 2001 | Panasonic Intellectual Property Corporation of America | Moving picture coding method, and moving picture decoding method |
9241161, | Nov 06 2001 | Panasonic Intellectual Property Corporation of America | Moving picture coding method, and moving picture decoding method |
9241162, | Nov 06 2001 | Panasonic Intellectual Property Corporation of America | Moving picture coding method, and moving picture decoding method |
9338448, | Nov 06 2001 | Panasonic Intellectual Property Corporation of America | Moving picture coding method, and moving picture decoding method |
9344714, | Nov 06 2001 | Panasonic Intellectual Property Corporation of America | Moving picture coding method, and moving picture decoding method |
9462267, | Nov 06 2001 | Panasonic Intellectual Property Corporation of America | Moving picture coding method, and moving picture decoding method |
9578323, | Nov 06 2001 | Panasonic Intellectual Property Corporation of America | Moving picture coding method, and moving picture decoding method |
Patent | Priority | Assignee | Title |
5136378, | Sep 07 1990 | Matsushita Electric Industrial Co., Ltd. | Moving picture coding apparatus |
5144426, | Oct 13 1989 | MATSUSHITA ELECTRIC INDUSTRIAL CO , LTD , | Motion compensated prediction interframe coding system |
5150432, | Mar 26 1990 | Kabushiki Kaisha Toshiba | Apparatus for encoding/decoding video signals to improve quality of a specific region |
5155593, | Sep 27 1989 | Sony Corporation | Video signal coding method |
5267334, | May 24 1991 | Apple Inc | Encoding/decoding moving images with forward and backward keyframes for forward and reverse display |
5293229, | Mar 27 1992 | Panasonic Corporation of North America | Apparatus and method for processing groups of fields in a video data compression system |
5315326, | Apr 26 1991 | JVC Kenwood Corporation | Efficient coding/decoding apparatuses for processing digital image signal |
5361105, | Mar 05 1993 | Panasonic Corporation of North America | Noise reduction system using multi-frame motion estimation, outlier rejection and trajectory correction |
5386234, | Nov 13 1991 | Sony Corporation | Interframe motion predicting method and picture signal coding/decoding apparatus |
5412428, | Dec 28 1992 | Sony Corporation | Encoding method and decoding method of color signal component of picture signal having plurality resolutions |
5436665, | Mar 03 1992 | Kabushiki Kaisha Toshiba | Motion picture coding apparatus |
5467136, | May 31 1991 | Kabushiki Kaisha Toshiba | Video decoder for determining a motion vector from a scaled vector and a difference vector |
5481310, | Aug 29 1991 | Sharp Kabushiki Kaisha | Image encoding apparatus |
5905534, | Jul 12 1993 | Sony Corporation | Picture decoding and encoding method and apparatus for controlling processing speeds |
6104753, | Feb 03 1996 | LG Electronics Inc | Device and method for decoding HDTV video |
6184935, | Mar 12 1997 | Vertex Pacific Limited | Upsampling filter and half-pixel generator for an HDTV downconversion system |
6219383, | Jun 30 1997 | QUARTERHILL INC ; WI-LAN INC | Method and apparatus for selectively detecting motion vectors of a wavelet transformed video signal |
EP651574, |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Oct 18 2000 | Matsushita Electric Industrial Co., Ltd. | (assignment on the face of the patent) | / |
Date | Maintenance Fee Events |
Sep 03 2008 | ASPN: Payor Number Assigned. |
Apr 14 2010 | M1553: Payment of Maintenance Fee, 12th Year, Large Entity. |
Date | Maintenance Schedule |
Feb 19 2011 | 4 years fee payment window open |
Aug 19 2011 | 6 months grace period start (w surcharge) |
Feb 19 2012 | patent expiry (for year 4) |
Feb 19 2014 | 2 years to revive unintentionally abandoned end. (for year 4) |
Feb 19 2015 | 8 years fee payment window open |
Aug 19 2015 | 6 months grace period start (w surcharge) |
Feb 19 2016 | patent expiry (for year 8) |
Feb 19 2018 | 2 years to revive unintentionally abandoned end. (for year 8) |
Feb 19 2019 | 12 years fee payment window open |
Aug 19 2019 | 6 months grace period start (w surcharge) |
Feb 19 2020 | patent expiry (for year 12) |
Feb 19 2022 | 2 years to revive unintentionally abandoned end. (for year 12) |