Methods for motion estimation with adaptive motion accuracy

Abstract

methods for motion estimation with adaptive motion accuracy of the present invention include several techniques for computing motion vectors of high pixel accuracy with a minor increase in computation. One technique uses fast-search strategies in sub-pixel space that smartly searches for the best motion vectors. An alternate technique estimates high-accurate motion vectors using different interpolation filters at different stages in order to reduce computational complexity. Yet another technique uses rate-distortion criteria that adapts according to the different motion accuracies to determine both the best motion vectors and the best motion accuracies. Still another technique uses a VLC table that is interpreted differently at different coding units, according to the associated motion vector accuracy.

Description

anencoder estimates the best motion vector in two steps shown in FIG. 2. First, the Telenor encoder searches for the best integer-pel vector V₁ (FIG. 1) 100. Second, the Telenor encoder searches for the best ⅓-pixel accurate vector V_1/3 (FIG. 1) near V₁ 102. This second step is shown graphically in FIG. 1 where a total of eight blocks (each having an array of 16×16 pixels) in the 3×3 interpolated reference frame are checked to find the best match. The motion vectors for these eight blocks are represented by the eight solid dots in the grid centered on V₁. In FIG. 1 the best match is the block associated to the motion vector V_1/3=(V_x, V_y)=(1+⅓, 1).

The technology of the present invention allows the encoder to choose between any set of motion accuracies (for example, ½, ⅓, and ⅙-pel accurate motion vectors) using either a full search strategy or a fast search strategy.

Full-Search AMA Search Strategy

As shown in FIGS. 3 and 4, in the full-search adaptive motion accuracy (“AMA”) search strategy the encoder searches all the motion vector candidates in a grid of ⅙-pixel resolution and a “square radius” (defined herein as a square block defined by a number of pixels up, a number of pixels down, and a number of pixels to both sides) of five pixels as shown in FIG. 3. FIG. 4 shows that the first step of the full-search AMA is to search for the best integer-pel vector V₁ (FIG. 1) 104. In the second step of the full-search AMA, the encoder searches for the best ⅙-pixel accurate vector V_1/6 (FIG. 3) near V₁ 106. In other words, the full-search AMA modifies the second step of the Telenor's process so that the encoder also searches for motion vector candidates in other sub-pixel locations in the velocity space. The objective is to find the best motion vector in the grid, i.e., the vector that points to the block (in the interpolated reference frame) that best matches the current macroblock. Although the full-search strategy is computationally complex since it searches 120 sub-pixel candidates, it shows the full potential of this preferred method of the present invention.

A critical issue in the motion vector search is the choice of a measure or criterion for establishing which block is the best match for the given macroblock. In practice, most methods use either the mean squared error (“MSE”) or mean absolute difference (“MAD”) criteria. The MSE between two blocks consists of subtracting the pixel values of the two blocks, squaring the pixel differences, and then taking the average. The MAD difference between two blocks is a similar distortion measure, except that the absolute value of the pixel differences is computed instead of the squares. If two image blocks are similar to each other, the MSE and MAD values will be small. If, however, the image blocks are dissimilar, these values will be large. Hence, typical video coders find the best match for a macroblock by selecting the motion vector that produces either the smallest MSE or the smallest MAD. In other words, the block associated to the best motion vector is the one closest to the given macroblock in an MSE or MAD sense.

Unfortunately, the MSE and MAD distortion measures do not take into account the cost in bits of actually encoding the vector. For example, a given motion vector may minimize the MSE, but it may be very costly to encode with bits, so it may not be the best choice from a coding standpoint.

To deal with this, advanced encoders such as those described by Telenor use rate-distortion (“RD”) criteria of the type “distortion+L*Bits” to select the best motion vector. The value of “distortion” is typically the MSE or MAD, “L” is a constant that depends on the compression level (i.e., the quantization step size), and “Bits” is the number of bits required to code the motion vector. In general, any RD criteria of this type would work with the present invention. However, in the present invention “Bits” include the bits needed for encoding the vector and those for encoding the accuracy of the vector. In fact, some candidates can have several “Bits” values, because they can have several accuracy modes. For example, the candidate at location (½, −½) can be thought of having ½ or ⅓ pixel 1/6-pixel accuracy.

Fast-Search AMA Search Strategy

As shown in FIGS. 5 and 6, in the fast-search adaptive motion accuracy (“AMA”) search strategy the encoder checks only a small set of the motion vector candidates. In the first step of the fast-search AMA, the encoder checks the eight motion vector candidates in a grid of ½-pixel resolution of square radius 1, which is centered on V₁ 108. V₂ is then set to denote the candidate that has the smallest RD cost (i.e., the best of the eight previous vectors and V₁) 110. Next, the encoder checks the eight motion vector locations in a grid of ⅙-pixel resolution of square radius 1 that is now centered on V₂ 112. If V₂ has the smallest RD cost 114, the encoder stops its search and selects V₂ as the motion vector for the block. Otherwise, V₃ is set to denote the best motion vector of the eight 116. The encoder then searches for a new motion vector candidate in the grid of ⅙-pixel resolution of square radius 1 that is centered on V₃ 118. It should be noted that some of the candidates in this grid have already been tested and can be skipped. The candidate with the smallest RD cost in this last step is selected as the motion vector for the block 120.

Experimental data has shown that, on average, this simple fast search strategy typically checks the RD cost of about eighteen locations in sub-pixel space (ten more than Telenor's search strategy), and hence the overall computational complexity is only moderately increased.

The experimental data discussed below in connection with FIGS. 8-18 show that there is practically no loss in compression performance from using this fast-search version of AMA. This is because the fast-search AMA search strategy exploits the convexity of the “distortion+L*Bits” curve (c.f., “distortion” is known to be convex), by creating a path that smartly follows the RD cost from higher to lower levels.

Alternate embodiments of the invention replace one or more of the steps 108-120. These embodiments have also been effective and have further reduced the number of motion vector candidates to check in the sub-pixel velocity space.

FIG. 7, for example, checks candidates of ⅓-pel accuracy. In this embodiment step 112 is replaced by one of three possible scenarios. First, if the best motion vector candidate from step 110 is at the center of V₁ (the “integer-pel vector”) 130, then the encoder checks three candidates of ⅓-pel accuracy between the center vector and the ½-pel location with the next lowest RD cost 132. Second, if the best motion vector candidate from step 110 is a corner vector 134, then, the encoder checks the four vector candidates of ⅓-pel accuracy that are closest to such corner 136. Third, if the best motion vector candidate from step 110 is between two corners 138, then, the encoder determines which of these two corners has lower RD cost and checks the four vector candidates of ⅓-pel accuracy that are closest to the line between such corner and the best candidate from step 110 140. It should be noted that in implementing this process step 138 may be unnecessary because if V2 V₂ is neither at the center or a corner vector, then it would necessarily be between two corners. If the encoder is set to find motion vectors with ⅓-pixel accuracy, FIG. 7 could be modified to end rather than continuing with step 114.

Computation And Memory Savings

Because step 108 checks only motion vector candidates of ½-pixel accuracy, the computation and memory requirements for the hardware or software implementation are significantly reduced. To be specific, in a smart implementation embodiment of this fast-search the reference frame is interpolated by 2×2 in order to obtain the RD costs for the ½-pel vector candidates. A significant amount of fast (or cache) memory for a hardware or software encoder is saved as compared to Telenor's approach that needed to interpolate the reference frame by 3×3. In comparison to the Telenor encoder, this is a cache memory savings of 9/4or 9/4, or a factor of 2.25. The few additional interpolations can be done later on a block-by-block basis.

Additionally, since the interpolations in step 108 are used to direct the search towards the lower values of the RD cost function, a complex filter is not needed for these interpolations. Accordingly, computation power may be saved by using a simple bilinear filter for step 108.

Also, other key coding decisions such as selecting the mode of a macroblock (e.g., 16×16, four-8×8, etc.) can be done using the ½-pel vectors because such decisions do not benefit significantly from using higher accuracies. Then, the encoder can use a more complex cubic filter to interpolate the required sub-pixel values for the few additional vector candidates to check in the remaining steps. Since the macroblock mode has already been chosen, these final interpolations only need to be done for the chosen mode.

Use of multiple-filters obtained computation savings of over twenty percent in running time on a Sparc Ultra 10 Workstation in comparison to Telenor's approach, which uses a cubic interpolation all the time. Additionally, the fast-memory requirements were reduced by nearly half. Also, there was little or no loss in compression performance. Comparing one preferred embodiment of the fast-search, Benzler's technique requires about 70 interpolations per pixel in the Telenor encoder and the present invention requires only about 7 interpolations per pixel.

Coding The Motion Vector And Accuracies With Bits

Once the best motion vector and accuracy are determined, the encoder encodes both the motion vector and accuracy is values with bits. One approach is to encode the motion vector with a given accuracy (e.g., half-pixel accuracy) and then add some extra bits for refining the vector to the higher motion accuracy. This is the strategy suggested by B. Girod, but it is sub-optimal in a rate-distortion sense.

In one preferred embodiment of the present invention, the accuracy of the motion vector for a macroblock is first encoded using a simple code such as the one given in Table 1. Any other table with code lengths {1, 2, 2} could be used as well. The bit rate could be further reduced using a typical DPCM approach.

TABLE 1

VLC table to indicate the accuracy mode for a given macroblock.
     Motion
Code Accuracy

1    ½-pel
01   ⅓-pel
11   ⅙-pel

Next, the value of the vector/s in the respective accuracy space is encoded. These bits can be obtained from entries of a single VLC table such as the one used in the H26L codec. The key idea is that these bits are interpreted differently depending on the motion accuracy for the macroblock. For example, if the motion accuracy is ⅓ and the code bits for the X component of the difference motion vector are 00001¹ 00001 (observe that this code is the fourth entry (code number 3) of H26L's VLC table in [6]), the X component of the vector is Vx=⅔. If the accuracy is ½, such code corresponds to Vx=1.

Compared to the Benzler method for encoding the motion vectors with a variable length code (“VLC”) table that could be used for encoding ½and ¼pixel accurate vectors, the method of the present invention can be used for encoding vectors of any motion accuracy and the table can be interpreted differently at each frame and macroblock. Further, the general method of the present invention can be used for any motion accuracy, not necessarily those that are multiples of each other or those that are of the type 1/n (with n an integer). The number of increments in the given sub-pixel space is simply counted and the bits in the associated entry of the table is used as the code.

From the decoder's viewpoint, once the motion accuracy is decoded, the motion vector can also be easily decoded. After that, the associated block in the previous frame is reconstructed using a typical 4-tap cubic interpolator. There is a different 4-tap filter for each motion accuracy.

The AMA does not increase decoding complexity, because the number of operations needed to reconstruct the predicted block are the same, regardless of the motion accuracy.

Experimental Results

FIGS. 8-18 show test results of the Telenor encoder codec with and without AMA in a variety of video sequences, resolutions, and frame rates, as described in Table 2. These figures show rate-distortion (“RD”) plots for each case. The “Anchor” curve shows RD points from optimized H.263+ (FIGS. 8 and 9 only). The “Telenor ½+b” curve shows Telenor with ½-pel vectors and bilinear interpolation (the “classical case”). The “Telenor ⅓” curve shows the current Telenor proposal (the “Telenor encoder”). The “Telenor+AMA+c” curve shows the Telenor encoder with the full-search strategy of the present invention. The “Telenor+FSAMA+c”, as shown in FIGS. 15-17, shows the current Telenor encoder with the fast-search strategy. (Unless otherwise specified, the full-search version of AMA was the encoder strategy used in the experiments.) All of the test results were cross-checked at the encoder and decoder. These results show that with AMA the gains in peak signal-to-noise ratio (“PSNR”) can be as high as 1 dB over H26L, and even higher over the classical case.

TABLE 2
Description of the Experiments
	Video sequence	FIG. #	Resolution	Frame rate
	Container	FIG. 8	QCIF	10
	News	FIG. 9	QCIF	10
	Mobile	FIG. 10	QCIF	10
		FIG. 11	SIF	15
	Garden	FIG. 12	QCIF	15
	Tempete	FIG. 13	SIF	15
		FIG. 14	QCIF	15
	Paris Shaked	FIG. 15	QCIF	10

The video sequences are commonly used by the video coding community, except for “Paris Shaked.” The latter is a synthetic sequence obtained by shifting the well-known sequence “Paris” by a motion vector whose X and Y components take a random value within [−1,1]. This synthetic sequence simulates small movements caused by a hand-held camera in a typical video phone scene.

Comparison Of Full-Search And Fast-Search AMA

The experimental results shown in FIGS. 16 and 17 demonstrate that the encoder performance with fast-search (“Telenor FSAMA+c”) and full-search (“Telenor AMA+c”) strategies for AMA is practically the same. This is true because the fast-search strategies exploit the convexity of the RD cost curve in the sub-pixel velocity space. In other words, since the shape of the RD cost follows a smooth convex curve, its minimum should be easy to find with some smart fast-search schemes that descend down the curve.

Combining AMA And Multiple Reference Frames

In the plot shown in FIG. 18, the curves labeled “1r” used only one reference frame for the motion compensation, so these curves are the same as those presented in FIG. 10. The curves labeled “5r” used five reference frames.

The experiments show that the gains with AMA add to those obtained using multiple reference frames. The gain from AMA in the one-reference case can be measured by comparing the curve labeled with a “+” (Telenor AMA+c+1r) with the curve labeled with an “x” (Telenor ⅓+1r), and the gain in the five-reference case can be measured between the curve labeled with a “diamond” (Telenor AMA+c+5r) with the curve labeled with a “*” (Telenor ⅓+5r).

It should be noted that the present invention may be implemented at the frame level so that different frames could use different motion accuracies, but within a frame all motion vectors would use the same accuracy. Preferably in this embodiment the motion vector accuracy would then be signaled only once at the frame layer. Experiments have shown that using the best, fixed motion accuracy for the whole frame should also produce compression gains as those presented here for the macroblock-adaptive case.

In another frame-based embodiment the encoder could do motion compensation on the entire frame with the different vector accuracies and then select the best accuracy according to the RD criteria. This approach is not suitable for pipeline, one-pass encoders, but it could be appropriate for software-based or more complex encoders. Still In still another fame-based embodiment, the encoder could use previous statistics and/or formulas to predict what will be the best accuracy for a given frame (e.g., the formulas in set forth in the Ribas work or a variation thereof can be used). This approach would be well-suited for one-pass encoders, although the performance gains would depend on the precision of the formulas used for the prediction.

The terms and expressions which have been employed in the foregoing specification are used therein as terms of description and not of limitation, and there is no intention, in the use of such terms and expressions, of excluding equivalents of the features shown and described or portions thereof, it being recognized that the scope of the invention is defined and limited only by the claims that follow.

Claims

1. A fast-search adaptive motion accuracy search method for estimating motion vectors in motion-compensated video coding by finding a best motion vector for a macroblock, said method comprising the steps of:

(a) searching a first set of motion vector candidates in a grid of sub-pixel resolution of a predetermined square radius centered on V₁, to find a best motion vector V₂ using a first criteria;

(b) searching a second set of motion vector candidates in a grid of sub-pixel resolution of a predetermined square radius centered on V₂ to find a best motion vector V₃ using a second criteria;

(c) searching a third set of motion vector candidates in a grid of sub-pixel resolution of a predetermined square radius centered on V₃ to find said best motion vector of said macroblock using a third criteria, and

(d) wherein at least one of said first criteria, said second criteria, and said third criteria is a rate-distortion criteria.

2. The method of claim 1, said step of searching a first set of motion vector candidates in a grid of sub-pixel resolution of a predetermined square radius centered on V₁, to find a best motion vector V₂ further comprising the step of searching a first set of eight motion vector candidates in a grid of ½-pixel resolution of square radius 1 centered on V₁ to find a best motion vector V₂.

3. The method of claim 1, said step of searching a second set of motion vector candidates in a grid of sub-pixel resolution of a predetermined square radius centered on V₂ to find a best motion vector V₃ further comprising the step of searching a second set of eight motion vector candidates in a grid of ⅙-pixel resolution of square radius 1 centered on V₂ to find a best motion vector V₃.

4. The method of claim 1 further comprising the steps of using V₂ as the motion vector for the macroblock if V₂ has the smallest rate-distortion cost and skipping step (c) of claim 1.

5. The method of claim 1, said step of searching a third set of motion vector candidates in a grid of sub-pixel resolution of a predetermined square radius centered on V₃ to find said best motion vector of said macroblock further comprising the step of searching a third set of eight motion vector candidates in a grid of ⅙-pixel resolution of square radius 1 centered on V₃ to find said best motion vector of said macroblock.

6. The method of claim 1, said step of searching a third set of motion vector candidates in a grid of sub-pixel resolution of a predetermined square radius centered on V₃ to find said best motion vector of said macroblock further comprising the step of skipping motion vector candidates of said third set of motion vector candidates that have already been tested.

7. The method of claim 1 further wherein said step of searching said first set of motion vector candidates further comprises the step of searching said first set of motion vector candidates using a first filter to do a first interpolation, said step of searching said second set of motion vector candidates further comprises the step of searching said second set of motion vector candidates using a second filter to do a second interpolation, and said step of searching said third set of motion vector candidates further comprises the step of searching said third set of motion vector candidates using a third filter to do a third interpolation.

8. The method of claim 1, said step of searching a second set of motion vector candidates in a grid of sub-pixel resolution of a predetermined square radius centered on V₂ to find a best motion vector V₃ further comprising the steps of:

(a) searching three candidates of ⅓-pel accuracy V₂ and a ½-pel location with the next lowest rate-distortion cost if V₂ is at the center;

(b) searching four vector candidates of ⅓-pel accuracy that are closest to V₂ if V₂ is a corner vector; and

(c) determining which of two corners has lower rate-distortion cost and searching four vector candidates of ⅓-pel accuracy that are closest to a line between said corner with lower rate-distortion cost, if V₂ is between two corners vectors.

9. An adaptive motion accuracy search method for estimating motion vectors in motion-compensated video coding by finding a best motion vector for a macroblock, said method comprising the steps of:

(a) searching a first set of motion vector candidates in a grid centered on V₁ using a first criteria to find a best motion vector V₂ using a first filter to do a first interpolation;

(b) searching a second set of motion vector candidates in a grid centered on V₂ using a second criteria to find a best motion vector V₃ using a second filter to do a second interpolation; and

(c) searching a third set of motion vector candidates in a grid centered on V₃ using a third criteria to find said best motion vector of said macroblock using a third filter to do a third interpolation;

(d) wherein at least one of said first criteria, said second criteria, and said third criteria is a rate-distortion criteria.

10. The method of claim 9 wherein said step of searching using a first filter to do a first interpolation further comprises using a simple filter to do a coarse interpolation.

11. The method of claim 9 wherein said step of searching using a first filter to do a first interpolation further comprises using a simple filter to do a coarse interpolation and said step of searching using a second filter to do a second interpolation further comprises using a complex filter to do a fine interpolation.

12. The method of claim 11 wherein said step of searching using a third filter to do a third interpolation further comprises using a complex filter to do a fine interpolation.

13. The method of claim 9 wherein said step of searching using a first filter to do a first interpolation further comprises using a bilinear filter to interpolate the reference frame by 2×2.

14. The method of claim 9 wherein said step of searching using a first filter to do a first interpolation further comprises to using a bilinear filter to interpolate the reference frame by 2×2 and said step of searching using a second filter to do a second interpolation further comprises using a cubic filter to do a fine interpolation.

15. The method of claim 14 wherein said step of searching using a third filter to do a third interpolation further comprises using a cubic filter to do a fine interpolation.

16. An adaptive motion accuracy search method for estimating motion vectors in motion-compensated video coding by finding a best motion vector for a macroblock, said method comprising the steps of:

(a) searching at a first motion accuracy for a first best motion vector of said macroblock;

(b) encoding said first best motion vector and said first motion accuracy;

(c) searching for at least one second best motion vector of said macroblock at an at least one second motion accuracy;

(d) encoding said at least one second best motion vector and said at least one second motion accuracy; and

(e) selecting the best motion vector of said first and at least one second best motion vectors using rate-distortion criteria.

17. The method of claim 16 wherein said step of selecting the best motion vector using rate-distortion criteria further comprises the step of said rate-distortion criteria adapting according to the different motion accuracies to determine both the best motion vectors and the best motion accuracies.

18. The method of claim 16, said step of searching for at least one second best motion vector at an at least one second motion accuracy further comprising the step of searching for at least one second best motion vector of said macroblock at an at least one second motion accuracy that is finer than said first motion accuracy.

19. The method of claim 16 wherein said step of selecting the best motion vector using rate-distortion criteria further comprises the step of using rate-distortion criteria of the type “distortion+L*Bits” to select the best motion vector.

20. An adaptive motion accuracy search method for estimating motion vectors in motion-compensated video coding by finding a best motion vector for a macroblock, said method comprising the steps of:

(a) searching at a motion accuracy for a best motion vector of said macroblock using rate-distortion criteria;

(b) encoding said motion accuracy using a code from a VLC table that is interpreted differently at different coding units according to the associated motion vector accuracy; and

(c) encoding said best motion vector in the respective accuracy space.

21. A system for estimating motion vectors in motion-compensated video coding by finding a best motion vector for a macroblock, said system comprising:

(a) a first encoder for searching a first set of motion vector candidates in a grid of sub-pixel resolution of a predetermined square radius centered on V₁ using a first criteria to find a best motion vector V₂;

(b) a second encoder for searching a second set of motion vector candidates in a grid of sub-pixel resolution of a predetermined square radius centered on V₂ using a second criteria to find a best motion vector V₃; and (c) a third encoder for searching a third set of motion vector candidates in a grid of sub-pixel resolution of a predetermined square radius centered on V₃ using a third criteria to find said best motion vector of said macroblock;

(d) wherein at least one of said first criteria, said second criteria, and said third criteria is a rate-distortion criteria.

22. The system of claim 21 wherein said first, second, and third encoders are a single encoder.

23. A fast-search adaptive motion accuracy search method for estimating motion vectors in motion-compensated video coding by finding a best motion vector for a macroblock, said method comprising the steps of:

(a) searching a first set of motion vector candidates in a grid of sub-pixel resolution of a predetermined square radius centered on V₁ to find a best motion vector V₂;

(b) searching a second set of motion vector candidates in a grid of sub-pixel resolution of a predetermined square radius centered on V₂ to find a best motion vector V₃;

(c) searching a third set of motion vector candidates in a grid of sub-pixel resolution of a predetermined square radius centered on V₃ to find said best motion vector of said macroblock, and

(d) using V₂ as the motion vector for the macroblock if V₂ has the smallest rate-distortion cost and skipping step (c).

24. The method of claim 1, wherein said first criteria, said second criteria, and said third criteria are all rate-distortion criteria.

25. The method of claim 9, wherein said first criteria, said second criteria, and said third criteria are all rate-distortion criteria.

26. The system of claim 21, wherein said first criteria, said second criteria, and said third criteria are all rate-distortion criteria.

27. A motion compensated video encoding apparatus comprising:

a motion compensator that compensates a motion using a motion vector having a fractional accuracy level; and

an encoder that encodes the motion vector and a fractional accuracy level which indicates two or more levels of a fractional accuracy expressed by 1/N pel (N is an arbitrary integer) of the motion vector, wherein

the motion compensation is performed by interpolation with a filter corresponding to the fractional accuracy level,

the fractional accuracy level is set frame-by-frame so that different frames could use different motion accuracies and is sent frame-by-frame,

the encoder encodes a variable length fractional accuracy level which indicates the fractional accuracy level, separately from encoding the motion vector, and

the encoder encodes the motion vector for each block in a block by block manner.

28. A motion compensated video encoding method comprising:

performing a motion compensation using a motion vector having a fractional accuracy level; and

encoding the motion vector and a fractional accuracy level which indicates two or more levels of a fractional accuracy expressed by 1/N pel (N is an arbitrary integer) of the motion vector, wherein

the motion compensation is performed by interpolation with a filter corresponding to the fractional accuracy level,

the fractional accuracy level is set frame-by-frame so that different frames could use different motion accuracies and is sent frame-by-frame,

encoding a variable length fractional accuracy level which indicates the fractional accuracy level, separately from encoding the motion vector, and

encoding the motion vector for each block in a block by block manner.

29. A video processing method comprising:

performing a motion compensation using a motion vector having a fractional accuracy level; and

computing the motion vector and a fractional accuracy level which indicates two or more levels of a fractional accuracy expressed by 1/N pel (N is an arbitrary integer) of the motion vector, wherein

the motion compensation is performed by interpolation with a filter corresponding to the fractional accuracy level,

the fractional accuracy level is set frame-by-frame so that different frames could use different motion accuracies and is computed frame-by-frame,

computing the fractional accuracy level separately from the motion vector by using a variable length code, and

computing the motion vector for each block in a block by block manner.