A method for predicting motion compensation of determining of an input image based on a motion vector of the input image from this input image to a reference image which has been sampled at a first set time, and the method includes calculating a motion vector of the input image based on a move, at a second set time, of a block unit which is a part of the input image and consists of a plurality of pixels, and calculating a motion vector of the reference image based on a move, at the first set time, of a block unit which is a part of the reference image and consists of a plurality of pixels. Move compensation of the input image is calculated both from the motion vector of the input image and from the motion vector of the reference image, to thereby realize a method for determining motion compensation with high precision.
|
0. 5. A method of obtaining a motion-compensated image, said method comprising the steps of:
(a) obtaining a first motion vector MV1 between the motion-compensated image and one reference image R1 of a plurality of reference images at a second set time interval T2 between the motion-compensated image and said one reference image R1;
(b) calculating a second motion vector MV2 between the motion-compensated image and another reference image R2 of said plurality of reference images at a first set time interval T1 between the motion-compensated image and said another reference image R2, said second motion vector MV2 being parallel to said first motion vector MV1 and having a magnitude satisfying the relation MV2=MV1·(T1/T2);
(c) calculating first pixel values from pixels which are neighbors of positions corresponding to said first motion vector MV1 and calculating second pixel values from pixels which are neighbors of positions corresponding to said second motion vector MV2, wherein said reference images R1 and R2 are previous to said motion-compensated image in a time sequence; and
(d) calculating motion-compensated pixel values of said motion-compensated image from said first and second pixel values calculated in step (c) to obtain said motion-compensated image.
0. 4. A method of obtaining a motion-compensated image, said method comprising the steps of:
(a) obtaining a first motion vector MV1 between the motion-compensated image and one reference image R1 of a plurality of reference images at a second set time interval T2 between the motion-compensated image and said one reference image R1;
(b) calculating a second motion vector MV2 between the motion-compensated image and another reference image R2 of said plurality of reference images at a first set time interval T1 between the motion-compensated image and said another reference image R2, said second motion vector MV2 being parallel to said first motion vector MV1 and having a magnitude satisfying the relation MV2=MV1·(T1/T2);
(c) calculating first pixel values from pixels which are neighbors of positions corresponding to said first motion vector MV1 and calculating second pixel values from pixels which are neighbors of positions corresponding to said second motion vector MV2, wherein said reference images R1 and R2 are such that a motion vector MV3 between said reference images R1 and R2 has a mathematical relationship with said first and second motion vectors MV1 and MV2 in which said motion vector MV3 is parallel to and different in value from each of said first and second motion vectors MV1 and MV2; and
(d) calculating motion-compensated pixel values of said motion-compensated image from said first and second pixel values calculated in step (c) to obtain said motion-compensated image.
0. 1. A method of determining motion compensation for an input image from motion vectors between the input image and a plurality of reference images, said method comprising the steps of:
(a) calculating a motion vector mv1 between the input image and one reference image of said plurality of reference images from a motion of at least one block unit at a second set time interval T2 between the input image and said one reference image, said at least one block unit being a part of said input image and comprising a plurality of pixels;
(b) providing a motion vector mv2 between at least two reference images of the plurality of reference images at a first set time interval T1, which is parallel to the motion vector mv1 at the second set time interval T2 and different in magnitude from the motion vector mv1 at the second set time interval T2 by a value determined by MV1·T1/T2; and
(c) calculating the motion compensation of the input image from both of (i) the motion vector mv1 between the input image and said one reference image and (ii) the motion vector mv2 between the at least two reference images of the plurality of reference images.
0. 2. A method of determining motion compensation for an input image from a motion vector between the input image and a plurality of reference images, said method comprising the steps of:
(a) detecting a motion vector mv1 between the input image and one reference image R1 of said plurality of reference images at a second set time interval T2;
(b) providing a motion vector mv3 between the reference image R1 and another reference image R2 of said plurality of reference images at a first set time interval T1, said motion vector mv3 being parallel to the motion vector mv1 and different in magnitude from the motion vector mv1 by a value determined by MV1·T1/T2;
(c) obtaining a motion vector mv2 between the input image and the another reference image R2 at a third set time interval T3 from a sum of the motion vector mv1 and the motion vector mv3, and calculating respective pixels corresponding to the motion vector mv1 and the motion vector mv2 from pixels of the reference image R1 and the reference image R2 corresponding to the motion vector mv1 and the motion vector mv2 or from pixels positioned peripherally of the pixels of the reference image R1 and the reference image R2; and
(d) calculating motion-compensated pixel values from the calculated pixels of the reference images.
0. 3. A method of obtaining a motion-compensated image from a motion vector between the motion-compensated image and a plurality of reference images, said method comprising the steps of:
(a) obtaining a motion vector mv1 between the motion-compensated image and one reference image R1 of said plurality of reference images at a second set time interval T2;
(b) providing a motion vector mv3 between the reference image R1 and another reference image R2 of said plurality of reference images at a first set time interval T1, which is parallel to the motion vector mv1 and different in magnitude from the motion vector mv1 by a value determined by MV1·T1/T2;
(c) obtaining a motion vector mv2 between the motion-compensated image and said another reference image R2 at a third set time interval T3 from a sum of the motion vector mv1 and the motion vector mv3, and calculating respective pixels corresponding to the motion vector mv1 and the motion vector mv2 from pixels of the reference image R1 and the reference image R2 corresponding to the motion vector mv1 and the motion vector mv2 or from pixels positioned peripherally of the pixels of the reference image R1 and the reference image R2; and
(d) calculating motion-compensated pixel values from the calculated pixels of the reference images to obtain the motion-compensated image.
0. 6. A method in accordance with
|
|||||||||||||||||
This is a
Therefore, when the vertical component of the MV is 1, the vertical component of the MV adj becomes −0.5. When this is applied to the positional correction for correcting the positions of the (M−1)-th field to the positions of the (M−2)-th field, a (M−1, 2) in the (M−1)-th field is corrected to the position of (M−2, 2.5) in the (M−2)-th field, as shown in FIG. 1. After the above operation, the position of (M−2, 2) which is the determined value of motion compensation of the a (M, 1) is obtained from pixel values of the (M−2)-th field and the (M−1)-th field which is the result of a time positional correction. In this case, when a necessary pixel value is to be obtained by a weighted average, which is inversely proportional to a distance, from near-by pixel values, for example, the pixel value at the position of the (M−2, 2), that is the determined value of motion compensation of the a (M, 1), is obtained by the following expression:
a(M−2, 1)/3+2*a(M−1, 2)/3
Although the above explains a determination of motion compensation taking account of only vertical components, a similar operation is also applied to the case for determining motion compensation for an image having both vertical and horizontal components.
As described above, according to the first embodiment of the present invention, a time position of a reference image is corrected by using a motion vector as required so that a plurality of pieces of reference images sampled at different times become an image at a time which is positioned within a certain time interval from the position of the input image, according to detected motion of the above time interval of a block unit including at least one pixel. Therefore, it is possible to obtain a plurality of pieces of images at positions which are separated from the input image by the above time intervals. A reference image with high pixel density is obtained by combining these pieces of images, and a pixel value at a position which compensates the detected motion is calculated by using the reference image of high pixel density, and the calculated pixel value is used as a determined value. As a result, there is an effect that it is possible to carry out a determination of motion compensation of an image at a very high precision level.
The assumptions used in the above description of the first embodiment are flexible and the following alternative assumptions can also be accepted. Instead of assuming two images for a reference image, any plurality of number of pieces of images can also be used. Instead of assuming the positions of the plurality of reference images as the previous image and its previous image, these positions can also be set flexibly. Instead of assuming constant motion of an image for calculating the MV adj from the MV, any assumption of motion can also be used according to a certain rule for calculating the MV adj from the MV. Instead of using a weighted average which is inverse proportional to a distance from near-by pixel values to obtain a pixel value at a necessary position, a coefficient of a low-pass filter, for example, can also be used to obtain the pixel value at a necessary position. Instead of using interpolation, for obtaining a necessary pixel value, extrapolation, for example, can also be used to obtain the necessary pixel value.
Further, although the vector M adj for correcting positions of a plurality of pieces of reference images is calculated from a motion vector MV which is detected in a block unit in the above-described first embodiment of the present invention, the MV can also be detected independently between the (M−1)-th field and the (M−2)-th field. In this case, by an independent detection of the MV, the time correction can be effected based on a more accurate motion of the image. Thus, this method has an effect that a determination of motion compensation of an image at a higher level of precision can be ensured.
Further, although an interlace signal is used as an input signal and a field image is assumed as each image in the above-described first embodiment of the present invention, a non-interlace image can also be used, with an effect that a determination of motion compensation of an image can be ensured at a higher level of precision, for the same reason as explained above.
Next, a second embodiment of the present invention for determining motion compensation of an image by using an interlace signal as an input signal and using a frame as an input, will be explained.
In
a(M−2,1)/7+6*a(M−1,2)/7
Similarly, in the same manner as that of the determination of a pixel within the M-th field, a pixel within the (M+1)-th field is determined from the two fields of the reference frame, that is, the (M−1)-th and (M−2)-th fields. The method of determination in this case is the same as the method for determining a pixel within the M-th field, except that the (M−2)-th field needs to be corrected to the position of the (M−1)-th field. A vector for correcting this time position is set as MV adj (M+1). In calculating the MV adj (M+1) from the MV(M+1), when the motion from the (M−2)-th field to the (M+1)-th field, for example, are constant, the following relationship can be obtained easily:
MV adj (M+1)=MV(M+1)/2
Therefore when the vertical component of the MV(M+1) is 1, the vertical component of the MV adj (M+1) becomes 0.5. As shown in
a(M−1,4)/3+2*a(M−2,3)/3
Although the above explains the determination of motion compensation of an image for only the vertical component, a similar operation is also applied to the case for determining motion compensation of an image having both vertical and horizontal components.
As described above, according to the second embodiment of the present invention, the above-described determination of motion compensation can be applied to a frame image, by using an interlace signal as an input signal and by setting a reference image in two fields of a certain frame. As a result, there is an effect that it is possible to determining motion compensation of an image at a high level of precision by using a frame as a base.
The above-described second embodiment also has a flexibility in the assumptions used, in the same manner as that of the first embodiment. For example, the number of reference frames, the positions of the reference frames, the assumptions for obtaining the MV adj (M) or MV adj (M+1) from either the MV(M) or the MV(M+1), the calculation method for obtaining a pixel value at a necessary position, and either interpolation or extrapolation, can also be selected freely. Further, although it is assumed in the present embodiment that a move vector for determining motion compensation can be obtained within the input image and reference frame and between fields of the same phase as that of the input image, it is also possible to obtain the motion vector between fields of opposite phases, in the same manner of operation, with the similar effect. Further, when a position correction vector is obtained independent of a detection motion vector, in the same manner as that of the first embodiment, there is an effect that it is possible to determining motion compensation at a higher level of precision.
Next, as a third embodiment of the present invention, another method for determining motion compensation for an interlace input signal based on a frame unit will be explained. FIG. 3 and
In other words, in the case of the third embodiment, motion vectors to be used for pixels within a block generated from the above frame take the same value regardless of whether the pixels belong to the M-th field or the (M+1)-th field. The other operations become the same as those of the second embodiment.
As described above, according to the third embodiment of the present invention, the same motion vector is used for pixels in the two input fields positioned within a predetermined spacial area such as a block generated by the frame. Accordingly, it is not necessary to detect motion vectors for each field according to this method, which also has an effect that it is possible to determine motion compensation at a high level of precision.
The above-described third embodiment also has a flexibility in the assumptions used, similar to the case of the second embodiment. For example, th number of reference frames, the positions of the reference frames, the assumption for obtaining the MV adj from the MV, the calculation method for obtaining a pixel value at a necessary position, and whether interpolation or extrapolation is to be used, can all be selected freely. Although description has been made of the case for determining motion compensation based on a frame as a unit in the present embodiment, it is needless to mention that there is no change in the effect of determination if the determination is carried out based on a field as shown in the first embodiment or if the determination is carried out based on a noninterlace image. Further, when a block to be used for having the same value of the motion vector is selected from among blocks of a plurality of pieces of input images in such a way that the block selected is a block of each input image of which part or whole of the spatial position superposes with those of the other blocks selected, there is no change in the effect of prediction. Further, similar to the case of the second embodiment, by obtaining a position correction vector independent of a detection motion vector, there is an effect that it is possible to determine motion compensation at a higher level of precision.
Now consider the case of obtaining a determined value of a(M, 1). When it is assumed that the vertical component of the detected MV is 3, the determined value of move compensation of the a(M, 1) becomes the pixel value at the position of (M−2, 4). First, this pixel value is obtained from the pixel value within the (M−2)-th field. When the pixel value is to be obtained based on a weighted average, inversely proportional to a distance, from near-by pixel values, for example, the pixel value at the position of the (M−2, 4) is obtained by the following expression:
a(M−2, 3)/2+a(M−2,5)/2
Next, based on the above MV, motion of the input image from the (M−1)-th field to the M-th field is calculated. The time difference between the M-th field and the (M−1)-th field is ½ of the time difference between the M-th field and the (M−2)-th field. Accordingly, this motion vector can be considered to be MV/2. Since the vertical component of the MV is now 3, the vertical component of MV/2 becomes 1.5. Accordingly, when a determined value of motion compensation of the a(M, 1) is obtained from the image in the (M−1)-th field, this becomes the pixel value at the position of (M−1, 2.5). This pixel value is obtained from a pixel value within the (M−1)-th field. When the pixel value is to be obtained by a weighted average, inversely proportional to a distance, from near-by pixel values, for example, the pixel value at the position of (M−1, 2.5) can be obtained by the following expression:
3*a(M−1,2)/4+a(M−1,4)/4
Based on the two determined values obtained above, a mean of the two determined values is obtained and the result is used as the determined value of the a(M, 1).
Although the above explains the case for determining motion compensation of an image for only the vertical component, a similar operation is also carried out for the case of determining motion compensation of an image having both vertical and horizontal components.
As described above, according to the fourth embodiment of the present invention, motion of an input image from a plurality of pieces of reference images sampled at different times according to detected motion at certain time intervals of a block unit including at least one pixel is calculated based on the above detected motion, and a pixel value at a position which has been compensated by the calculated motion portion for each reference image is calculated, so that it is possible to obtain a plurality determined values of motion compensation from the plurality of pieces of reference images. Since a determined value of the input image is calculated from the plurality of determined values, noise can be eliminated if noise is included in the determined values, thus ensuring a determination at a high precision level, of motion compensation.
In the manner similar to the case of the first embodiment, it is also possible in the fourth embodiment to freely select the number of pieces of reference images, the positions of the reference images, the calculation method for obtaining a pixel value at a necessary position within each reference image, and either interpolation or extrapolation. For calculating determined values from a plurality of pixel values obtained from the respective reference images, there are other alternative methods than a simple averaging method such as a weighted average method and a method for calculating by using a coefficient of a low-pass filter. Although description has been made of the case for determining motion compensation based on a field of an interlace signed in the present embodiment, it is needless to mention that the effect of the determination does not change if a frame is used as a base or a noninterlace image is used as a base as shown in the second and third embodiments respectively.
According to the present invention, as is clear from the above-described embodiments, a time position of a reference image is corrected by using a motion vector as required so that a plurality of pieces of reference images sampled at different times according to detected motion at certain time intervals of a block unit including at least one pixel become images at times separated from the input image by the above time intervals, so that it is possible to obtain a plurality of pieces of images at positions separated by the above time intervals from the input image. By combining the plurality of pieces of images together, a reference image of high pixel density can be obtained and a pixel value at a position which has been compensated by the detected motion is calculated by using the reference image of the high pixel density, so that the calculated pixel value is used as a determined value. Thus, there is an effect that it is possible to determine motion compensation of an image at a very high level of precision.
Further, according to the present invention, a vector for correcting a time position of the above reference image can be calculated based on motion detected at a certain time interval, which does not require a detection again of a motion vector for correcting the time position, so that this has an effect that motion compensation at a high precision level can be ensured. Further, since an interlace signal can be used as an input signal and a reference image can be in two fields of a certain frame, the above determination of motion compensation can be applied to a frame image, thus ensuring a determination, at a high precision level, of motion compensation based on a frame.
Further, since the same value can be used for a block of each input image among blocks of a plurality of pieces of input images, each block having its whole or part of spatial position superposed with that of the other blocks, as a move detected at a certain time interval of a block unit including at least one pixel, it is not necessary to carry out a plurality of detections of moves of many block in a plurality of input images so that there is an effect that a determination of motion compensation at a high precision level can be ensured.
Inoue, Shuji, Yukitake, Takeshi
| Patent | Priority | Assignee | Title |
| Patent | Priority | Assignee | Title |
| 4691230, | Mar 04 1985 | Kokusai Denshin Denwa Co., Ltd. | Motion vector detection system of a moving object on a screen |
| 4862266, | Jun 09 1987 | Sony Corporation | Television standards converters |
| 4864394, | Jun 09 1987 | Sony Corp. | Motion vector selection in television images |
| 4982285, | Apr 27 1989 | Victor Company of Japan, LTD | Apparatus for adaptive inter-frame predictive encoding of video signal |
| 4989089, | Nov 08 1988 | U S PHILIPS CORPORATION | Television picture coding device and corresponding decoding device, a television picture transmission system incorporating such devices, and transmission and receiving stages of such a system |
| 4998168, | Apr 27 1989 | Sony Corporation | Motion dependent video signal processing |
| 5005077, | Apr 27 1989 | Sony Corporation | Motion dependent video signal processing |
| 5021881, | Apr 27 1989 | Sony Corporation | Motion dependent video signal processing |
| 5027205, | Apr 27 1989 | Sony Corporation | Motion dependent video signal processing |
| 5036393, | Apr 27 1989 | Sony Corporation | Motion dependent video signal processing |
| 5049991, | Feb 20 1989 | Victor Company of Japan, LTD | Movement compensation predictive coding/decoding method |
| 5072293, | Aug 29 1989 | NXP B V | Method of estimating motion in a picture signal |
| 5093720, | Aug 20 1990 | CIF LICENSING, LLC | Motion compensation for interlaced digital television signals |
| 5105271, | Sep 29 1989 | Victor Company of Japan, LTD | Motion picture data coding/decoding system having motion vector coding unit and decoding unit |
| 5132792, | Oct 14 1989 | Sony Corporation | Video signal transmitting system |
| 5138446, | Aug 29 1989 | U S PHILIPS CORPORATION, A CORP OF DE | Method and device for processing of pictures with improved movement estimation |
| 5142361, | Jun 21 1990 | Graphics Communication Technologies, Ltd. | Motion vector detecting apparatus for video telephone/teleconference systems |
| 5144427, | Jun 20 1990 | Victor Company of Japan, LTD | Television receiver decoder apparatus for bandwidth-compressed high definition television signal |
| 5157742, | Feb 28 1990 | Victor Company of Japan, Ltd. | Motion image data compression system |
| 5162907, | Sep 28 1990 | Sony United Kingdom Limited | Motion dependent video signal processing |
| 5175618, | Oct 31 1990 | Victor Company of Japan, LTD | Compression method for interlace moving image signals |
| 5191414, | Mar 27 1990 | Victor Company of Japan, LTD | Interfield predictive encoder and decoder for reproducing a signal subjected to predictive encoding by encoder into an image signal |
| 5200820, | Apr 26 1991 | SHINGO LIMITED LIABILITY COMPANY | Block-matching motion estimator for video coder |
| 5210605, | Jun 11 1991 | Trustees of Princeton University | Method and apparatus for determining motion vectors for image sequences |
| 5424779, | May 31 1991 | Kabushiki Kaisha Toshiba | Video coding apparatus |
| 5436674, | May 23 1991 | Nippon Hoso Kyokai | Method of detecting motion vector, apparatus therefor, and picture signal processing system utilizing the apparatus |
| 6016162, | May 31 1991 | Kabushiki Kaisha Toshiba | Video coding apparatus |
| EP395271, | |||
| EP395440, | |||
| EP447068, | |||
| EP458249, | |||
| EP484140, | |||
| WO92210061, |
| Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
| Apr 13 2001 | Matsushita Electric Industrial Co., Ltd. | (assignment on the face of the patent) | / |
| Date | Maintenance Fee Events |
| Sep 19 2007 | ASPN: Payor Number Assigned. |
| Oct 02 2009 | R1553: Refund - Payment of Maintenance Fee, 12th Year, Large Entity. |
| Date | Maintenance Schedule |
| Sep 12 2009 | 4 years fee payment window open |
| Mar 12 2010 | 6 months grace period start (w surcharge) |
| Sep 12 2010 | patent expiry (for year 4) |
| Sep 12 2012 | 2 years to revive unintentionally abandoned end. (for year 4) |
| Sep 12 2013 | 8 years fee payment window open |
| Mar 12 2014 | 6 months grace period start (w surcharge) |
| Sep 12 2014 | patent expiry (for year 8) |
| Sep 12 2016 | 2 years to revive unintentionally abandoned end. (for year 8) |
| Sep 12 2017 | 12 years fee payment window open |
| Mar 12 2018 | 6 months grace period start (w surcharge) |
| Sep 12 2018 | patent expiry (for year 12) |
| Sep 12 2020 | 2 years to revive unintentionally abandoned end. (for year 12) |