A novel Lucas-Kanade sub-pixel motion estimation method is provided. The motion estimation algorithm enables the estimating of a motion vector with reduced computation cost while maintaining high sub-pixel accuracy. The novel algorithm consists of two processing stages. In the first stage, a conventional motion estimation method is applied to obtain the motion vector at integer-pixel level. In the second stage, the Lucas-Kanade algorithm is applied to improve the motion vector to sub-pixel accuracy based on gradient information. Experimental result shows that the proposed method reaches comparable PSNR performance as conventional ⅛-pel algorithm but with significant saving on computation cost.
|
18. A device for sub-pixel motion estimation, the device comprising:
a processor; and
means for estimating a first motion vector and a sub-pixel motion vector ΔP of a current image with respect to a reference image, wherein the sub-pixel motion vector ΔP is estimated based on image gradient information of the reference image, wherein the sub-pixel motion vector ΔP improves from integer-pixel accuracy to sub-pixel accuracy with no more than one iterative search step in applying Lucas-Kanade algorithm, and wherein a second motion vector is determined by adding the sub-pixel motion vector ΔP to the first motion vector.
1. A method of sub-pixel motion estimation, the method comprising:
a) estimating a first motion vector to integer-pixel accuracy, wherein the first motion vector maps a reference image to a current image;
b) computing image gradient information of the reference image; and
c) determining a second motion vector to sub-pixel accuracy by adding a sub-pixel motion vector ΔP to the first motion vector, wherein the sub-pixel motion vector ΔP is estimated based on the image gradient information, and wherein the sub-pixel motion vector ΔP improves from integer-pixel accuracy to sub-pixel accuracy with no more than one iterative search step in applying Lucas-Kanade algorithm.
10. A non-transitory computer-readable medium having processor-executable instructions for performing the steps of:
a) estimating a first motion vector to integer-pixel accuracy, wherein the first motion vector maps a reference image to a current image;
b) computing image gradient information of the reference image; and
c) determining a second motion vector to sub-pixel accuracy by adding a sub-pixel motion vector ΔP to the first motion vector, wherein the sub-pixel motion vector ΔP is estimated based on the image gradient information, and wherein the sub-pixel motion vector ΔP improves from integer-pixel accuracy to sub-pixel accuracy with no more than one iterative search step in applying Lucas-Kanade algorithm.
2. The method of
3. The method of
4. The method of
5. The method of
6. The method of
7. The method of
8. The method of
9. The method of
11. The computer-readable medium of
12. The computer-readable medium of
13. The computer-readable medium of
14. The computer-readable medium of
15. The computer-readable medium of
16. The computer-readable medium of
17. The computer-readable medium of
19. The device of
20. The device of
21. The device of
22. The device of
24. The device of
|
The present invention relates generally to motion estimation, and, more particularly, to a method of sub-pixel accuracy motion vector estimation.
Motion estimation is the process of determining motion vectors that describe the transformation from a first frame to a second frame in a video sequence. Motion compensation is the process of applying the motion vectors to the first frame to synthesize the transformation to the second frame. The combination of motion estimation and motion compensation forms a critical component of video compression as used by MPEG as well as many other video codes. Each frame in a typical video sequence is made up of some changed regions of another frame. By exploiting strong interframe correlation along the temporal dimensions, motion estimation thus provides means for reducing temporal redundancy and achieving video compression.
Motion vectors may relate to the whole image or specific parts, such as rectangular blocks, arbitrary shaped patches or even per pixel. Motion vectors may be represented by a translational model or many other models that approximate the motion of a real video camera, such as rotation, translation, or zoom. There are various methods for finding motion vectors. One of the popular methods is a block-matching algorithm (BMA), which finds a matching block from one frame in another frame. Different searching strategies such as cross search, full search, spiral search, or three-step search may be utilized in BMA to evaluate possible candidate motion vectors over a predetermined neighborhood search window to find the optimum motion vector.
Theoretical and experimental analyses have established that sub-pixel accuracy has a significant impact on the performance of motion compensation. Sub-pixel accuracy mainly can be achieved through interpolation. Various methods of performing interpolative up sampling at spatial domain or frequency domain have been proposed over the years. One major concern of implementing interpolative sub-pixel methods, however, is the computation cost. For example, to achieve one-eighth pixel accuracy, an image-processing system needs to handle the storage and manipulation of data arrays that are 64 times larger than integer-pixel motion estimation.
Motion estimation is also commonly used in image registration process, which finds a variety of applications in computer vision such as image matching, pattern recognition, and motion analysis. The Lucas-Kanade algorithm has been proven to be a highly accurate image registration method and has been used in computer vision and medical imaging industry for years. One major concern of applying the Lucas-Kanade algorithm to block-based motion estimation, however, is its high computational complexity. In addition, the Lucas-Kanade algorithm suffers matching deficiency when the starting point of the search is far away from the optimum. It is therefore desirable to have a motion estimation method that reduces the computation cost and complexity while maintaining high sub-pixel accuracy.
A novel Lucas-Kanade sub-pixel motion estimation method is provided. The motion estimation algorithm enables the estimating of a motion vector with reduced computation cost while maintaining high sub-pixel accuracy. The motion vector maps a reference image block in a reference frame to a current image block in a current frame. The motion estimation algorithm consists of two processing stages. In the first stage, a conventional motion estimation method is applied to obtain the motion vector at integer-pixel level. Optionally, the motion vector is modified based on motion information from neighboring pixels. In the second stage, the Lucas-Kanade algorithm is applied to improve the motion vector to sub-pixel accuracy based on gradient information. To find the proper estimate for sub-pixel motion vector, the Lucas-Kanade algorithm performs gradient descent on the Sum of Square Difference evaluation metric (ESSD) between the current image block and the reference image block and iteratively finds the minimum matching error of ESSD. In one example, the Levenberg-Marquardt algorithm is applied to obtain the least square solution while avoiding the instability problem of matrix inversion.
In one embodiment, the proposed motion estimation algorithm takes advantage of the sub-pixel registration accuracy from the Lucas-Kanade algorithm and alleviates both the computation complexity and the deficiency of large displacement. To simplify the computational complexity, the number of iterative searches is limited to as few as only one step. The one-step constraint of the Lucas-Kanade search step is based on three assumptions. First, the motion model is two-dimension (2D) translation. Second, the integer-pixel motion estimation is accurate, so the sub-pixel motion vector update is less than one pixel grid. Third, a smaller damping factor λ is preferred since it makes Levenberg-Marquardt behaves more like Gauss-Newton method, which works better than steepest descent when local minimum is near. In other words, the objective of the Lucas-Kanade search is changed from iteratively finding minimum matching error ESSD, to finding the smallest damping factor λ from a pre-defined set such that the length of the resulting sub-pixel motion vector does not exceed the integer grid. Experimental result shows that the proposed method reaches comparable PSNR performance as conventional ⅛-pel algorithm but with significant saving on computation cost.
Other embodiments and advantages are described in the detailed description below. This summary does not purport to define the invention. The invention is defined by the claims.
The accompanying drawings, where like numerals indicate like components, illustrate embodiments of the invention.
Reference will now be made in detail to some embodiments of the invention, examples of which are illustrated in the accompanying drawings.
For motion vectors with integer-pixel accuracy, the unit of displacement is one whole pixel. For instance, a motion vector may equal to (4, 1)T, which finds the matching block by moving four pixels to the right and one pixel upwards with respect to the position of the search block. For motion vectors with sub-pixel accuracy, the unit of displacement may be a half pixel, a quarter pixel, an eighth pixel, or any fraction of a whole pixel. For instance, a motion vector may equal to (4.75, 1.5)T, which finds the matching block by moving four and three quarters pixels to the right and one and half pixels upwards with respect to the position of the search block. Because the true frame-to-frame displacements of the image contents are, of course, completely unrelated to the sampling grid, the performance of motion compensation with sub-pixel accuracy is much improved as compared to the performance of motion compensation with integer-pixel accuracy. More accurate sub-pixel motion compensation, however, typically requires more complicate motion estimation techniques and higher computation cost.
In accordance with one novel aspect, motion estimation routine 62 in
The Lucas-Kanade algorithm is a two-frame differential method for optical flow estimation. Given two image I1(x) and I0(x), where x=(x, y)T is the pixel coordinate, the goal of the Lucas-Kanade algorithm is to find a warping function H(x; p) that minimizes the sum of square difference (SSD) between the two images:
where ESSD is a type of evaluation metrics defining the error between the two images. The warping function H is the mathematical relationship that maps pixel coordinates from one image to another, and p=(p1, p2, . . . pn)T is the parameter vector of H.
A variety of such parametric warping models are possible, from 2D transformations, to planar perspective models, 3D camera rotations, lens distortions, mapping to non-planar (e.g. cylindrical) surfaces, and non-rigid body adaptive meshes. Among these models, the most commonly used one, and the simplest, is the translation model:
where tx is the x directional displacement between the two images, and ty is the y directional displacement between the two images, and where p=(tx, ty)T is called the motion vector in video and imaging industry. To find the proper estimate for motion vector p, the Lucas-Kanade algorithm performs gradient descent on the SSD energy function in Equation (1) using the first-order Taylor expansion:
where
is the image gradient, and
is the Jacobian of H(x; p).
The least square solution of Equation (2) can be written in matrix form as follows:
Δp=(JT(∇IT∇I)J)−1JT∇ITe (3)
where e is the column vector formed by pixel intensity difference between two images. Several numerical methods can be applied to obtain the solution while avoiding the instability problem of matrix inversion. One of such methods is the Levenberg-Marquardt algorithm, an iterative approach to make matrix inversion numerically solvable. The Levenberg-Marquardt algorithm combines the benefits from the Gauss-Newton algorithm, which works better if local minimum is close, and the steepest descent algorithm, which moves faster when the starting point is far away. The algorithm is an iterative procedure. It adds a positive “damping factor” λ onto the matrix diagonal terms in order to stabilize the inversion:
Δp=(JT(∇IT∇I)J+λI)−1JT∇ITe (4)
where the italic I is the identity matrix.
The damping factor λ is adjusted at each iteration step. A smaller brings the algorithm closer to the Gauss-Newton method, whereas a large λ brings the update closer to the gradient descent direction. Marquardt recommended starting with a value λ0 and a factor v>1. Compute the evaluation metrics ESSD after one step from the starting point with the damping factor of λ=λ0 and secondly with λ/v. If both steps are worse than the initial point, then the damping factor is increased by successive multiplication by v until a better point is found with a new damping factor λvk for some k. If λ/v results in a reduction of ESSD, then λ/v is taken as the new value of λ (and the new location is taken as that obtained with this damping factor) and process continues; on the other hand, if λ/v results in an increase of ESSD, then λ is left unchanged.
One major concern of applying the Lucas-Kanade algorithm to block-based motion estimation, however, is its high computational complexity. As shown in Equation (4), it requires matrix inversion to obtain the motion update Δp at each Levenberg-Marquardt iteration step, and the validity of λ needs to be confirmed via the calculation of block-matching error ESSD. In addition to the concern of the computation cost, the Lucas-Kanade algorithm also suffers the matching deficiency when the starting point of the search is far away from the optimum.
In accordance with one novel aspect, the novel motion estimation algorithm takes advantage of the sub-pixel registration accuracy from the Lucas-Kanade algorithm and alleviates both the computation complexity and the deficiency of large displacement. To simplify the computational complexity, the number of iterative searches is limited to as few as only one step. The one-step constraint of the Lucas-Kanade search step is based on three assumptions. First, the motion model is 2D translation, so the Jacobian J becomes a 2×2 identity matrix. Equation (4) is therefore simplified to
Second, the integer-pixel motion estimation is accurate, so the sub-pixel motion vector update Δp has the property of |Δp|<1. Third, a smaller damping factor λ is preferred since it makes Levenberg-Marquardt behaves more like Gauss-Newton method, which works better than steepest descent when local minimum is near. In other words, the objective of the Lucas-Kanade search is changed from iteratively finding minimum matching error ESSD, to finding the smallest damping factor λ from a pre-defined set such that the length of the resulting sub-pixel motion vector does not exceed the integer grid.
In step 103, image gradient information ∇I of the reference image block I1 is computed. Other relevant information is also computed in this step. For example, the damping factor of λ is defined from a finite set M={λ: λ=0 or λ0vk, k=−m, . . . , −1, 0, 1, . . . , n}. In addition, the following information is calculated:
In step 104, the sub-pixel motion estimation using Lucas-Kanade algorithm is performed. This step involves the finding of the smallest damping factor from a pre-defined set such that the length of the resulting sub-pixel motion vector Δp does not exceed the integer grid. In other words, it involves the finding of λ such that |det*|>max(|D*X|,|D*Y|,0), where det*=det(∇IT∇I)+λ·(λ+a00+a11), D*X=DX−λ·b0, D*Y=DY−λ·b1. Because Δp=[Δx/Δy], Δx=DX*/det*, and Δy=DY*/det*, the damping factor λ is chosen to guarantee that the length of both |Δx| and |Δy| is less than one. This ensures that Δp has the property of |Δp|<1. If no λ is found, then return the original integer-pixel accuracy motion vector p. If λ is found, then the λ is feed into Equation (5) and sub-pixel motion vector Δp is calculated accordingly.
Once a sub-pixel motion vector Δp is found, the original integer-accuracy motion vector P is then iteratively updated to the final motion vector p′←p+Δp (step 105). The updated motion vector p′ maps a new reference image block I1′ to the current image block I0. In step 105, an error metric E2 between the new reference image block I1′ and the current image block I0 is then calculated, (E2=Σ|prefi′−pnewi|). E2 is compared with the original error metric E1. If E2 is smaller than E1, then the update motion vector p′ is returned as the final desired motion vector. Otherwise, the original motion vector p is returned.
In order to estimate sub-pixel motion vector in only one step, the proposed method requires the calculation of first-order image gradient on x and y directions. The calculation is on the integer grid of the reference image. Several discrete approximations of the gradient for optical flow estimation have been proposed in literature. Simple difference operator, for example [−1, 0, 1], is sufficient if image noise is not large. To obtain the normal matrix ∇IT∇I, four multiply-and-add operations per pixel are used. Another per-pixel operation is the ∇ITe, which takes two multiply-and-add. By combining the two operations, the number of the pixel operations in the proposed method, excluding the calculation of errors E1 and E2, is about the same as that of one SAD block-matching function at sub-pixel precision using bilinear interpolation. The other calculations in this algorithm are per-block operations, whose number of operations is small comparing to the per-pixel calculation.
As illustrated in
Although the present invention has been described in connection with certain specific embodiments for instructional purposes, the present invention is not limited thereto. The motion model may not be a 2D translational model, instead, it may be varies from 2D transformations, to planar perspective models, 3D camera rotations, lens distortions, mapping to non-planar (e.g. cylindrical) surfaces, and non-rigid body adaptive meshes. The error metric may not be SSD or SAD, instead, it may be Mean Square Error (MSE), Mean Absolute Difference (MAD), Sum of Absolute Transformed Differences (SATD), and other evaluation metrics. Accordingly, various modifications, adaptations, and combinations of various features of the described embodiments can be practiced without departing from the scope of the invention as set forth in the claims.
Patent | Priority | Assignee | Title |
10282831, | Dec 28 2015 | Novatek Microelectronics Corp. | Method and apparatus for motion compensated noise reduction |
10609314, | May 05 2016 | Pixart Imaging Inc. | High accuracy displacement detection system with offset pixel array |
11272203, | Jul 23 2019 | TENCENT AMERICA LLC | Method and apparatus for video coding |
11341750, | Dec 04 2015 | Texas Instruments Incorporated | Quasi-parametric optical flow estimation |
11800133, | Jul 23 2019 | TENCENT AMERICA LLC | Method and apparatus for video coding |
8705626, | Apr 02 2009 | MEDIATEK INC | Motion estimation method |
8837595, | Apr 02 2009 | MEDIATEK INC | Motion estimation method |
8947449, | Feb 21 2012 | GOOGLE LLC | Color space conversion between semi-planar YUV and planar YUV formats |
9420284, | Nov 29 2010 | ECOLE DE TECHNOLOGIE SUPERIEURE | Method and system for selectively performing multiple video transcoding operations |
9438910, | Mar 11 2014 | GOOGLE LLC | Affine motion prediction in video coding |
9819841, | Apr 17 2015 | Altera Corporation | Integrated circuits with optical flow computation circuitry |
Patent | Priority | Assignee | Title |
6996180, | Sep 05 2001 | Intel Corporation | Fast half-pixel motion estimation using steepest descent |
7227896, | Mar 08 2002 | Sharp Kabushiki Kaisha | Method and apparatus for global motion estimation |
7280594, | Oct 29 2001 | PARTHUSCEVA LTD | Method and apparatus for motion estimation in a sequence of digital images |
20040076233, | |||
20070041445, | |||
EP1104197, |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Jan 08 2009 | LIN, SHANG-HUNG | Mediatek USA Inc | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 022179 | /0014 | |
Jan 12 2009 | MEDIATEK SINGAPORE PTE LTD. | (assignment on the face of the patent) | / | |||
Nov 18 2009 | Mediatek USA Inc | Mediatek Singapore Pte Ltd | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 023546 | /0868 | |
Dec 23 2020 | Mediatek Singapore Pte Ltd | XUESHAN TECHNOLOGIES INC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 055283 | /0048 |
Date | Maintenance Fee Events |
Mar 04 2016 | M1551: Payment of Maintenance Fee, 4th Year, Large Entity. |
Mar 04 2020 | M1552: Payment of Maintenance Fee, 8th Year, Large Entity. |
Apr 22 2024 | REM: Maintenance Fee Reminder Mailed. |
Oct 07 2024 | EXP: Patent Expired for Failure to Pay Maintenance Fees. |
Date | Maintenance Schedule |
Sep 04 2015 | 4 years fee payment window open |
Mar 04 2016 | 6 months grace period start (w surcharge) |
Sep 04 2016 | patent expiry (for year 4) |
Sep 04 2018 | 2 years to revive unintentionally abandoned end. (for year 4) |
Sep 04 2019 | 8 years fee payment window open |
Mar 04 2020 | 6 months grace period start (w surcharge) |
Sep 04 2020 | patent expiry (for year 8) |
Sep 04 2022 | 2 years to revive unintentionally abandoned end. (for year 8) |
Sep 04 2023 | 12 years fee payment window open |
Mar 04 2024 | 6 months grace period start (w surcharge) |
Sep 04 2024 | patent expiry (for year 12) |
Sep 04 2026 | 2 years to revive unintentionally abandoned end. (for year 12) |