The present invention is embodied in a system and method for estimating epipolar geometry, in terms of a fundamental matrix, between multiple images of an object for stereo vision processing. The fundamental matrix embodies the epipolar geometry between the images. In general, the present invention includes a method for estimating epipolar geometry between multiple images of an original space given an initial estimate of the fundamental matrix found using a standard linear estimation technique. Namely, the system and method of the present invention estimates the fundamental matrix by transforming image points of multiple images into projective space. After this transformation is performed, nonlinear optimization is used with one parameterization of the fundamental matrix. The images are then inverse transformed back to the original space with a final estimate of the fundamental matrix. In addition, the original noise information is preserved during the optimization in the projective space.
|
1. A method for estimating epipolar geometry between at least two images of an original space given an initial estimate of a fundamental matrix, comprising:
transforming image points of the two images into a projective space; performing nonlinear optimization on the fundamental matrix with at least one predefined parameter characterizing the fundamental matrix; and performing an inverse transform of the fundamental matrix back to the original space.
10. A method for estimating epipolar geometry between two images of an original space given an initial estimate of a fundamental matrix, comprising:
identifying an element of the fundamental matrix with the largest absolute value; ensuring that the initial element of the fundamental matrix in the transformed space has the largest value; ensuring that the epipoles are not at infinity; estimating the fundamental matrix in the permutated space; and performing an inverse permutation to the estimated fundamental matrix to obtain the fundamental matrix of the original image space.
16. A method for estimating epipolar geometry between two images of an original space given an initial estimate of a fundamental matrix, comprising:
identifying an element of the fundamental matrix with the largest absolute value; constructing two appropriate permutation matrices of image points such that the element with the largest absolute value of the fundamental matrix is at an initial location of the matrix; permuting the coordinates of each sampled image point accordingly; estimating the fundamental matrix in the permutated space; and performing an inverse permutation to the estimated fundamental matrix to obtain the fundamental matrix of the original image space.
2. The method of
3. The method of
4. The method of
5. The method of
6. The method of
7. The method of
8. The method of
9. The method of
11. The method of
12. The method of
14. The method of
15. The method of
17. The method of
18. The method of
19. The method of
20. The method of
21. The method of
23. The method of
|
1. Field of the Invention
The present invention relates in general to object modeling, and in particular to a system and method for estimating the epipolar geometry of images representing an object that are used to model the object with image processing systems that require the epipolar geometry for modeling the object.
2. Related Art
Many stereo and computer vision applications that model three dimensional (3D) objects base the 3D modeling on several two dimensional (2D) images and a fundamental matrix that embodies the epipolar geometry between the images. The fundamental matrix is a crucial determination in object modeling.
For example, typically, in these systems, it is assumed that at least a pair of two dimensional images of the three dimensional object or environment are taken from two distinct viewpoints. In operation, these system usually first obtain two-dimensional images of a three-dimensional object and then obtain a fundamental matrix between the two images. In other words, the epipolar geometry of the two images is usually determined first and corresponding points between the two images typically need to satisfy an epipolar constraint before performing additional steps for modeling the object.
Current systems that perform estimation of the fundamental matrix typically require consideration of 36 distinct parameterizations. These distinct 36 parameterizations are considered because an epipole may be at infinity and an element of the epipolar transformation may be equal to 0, thus, requiring 36 maps to parameterize the fundamental matrix. However, these systems are time consuming and lead to cumbersome implementation of optimization procedures.
Therefore, what is needed is a system and method for estimating the fundamental matrix without using 36 maps to parameterize the fundamental matrix. What is also needed is a system and method for transforming image points of the image in projective space, so that nonlinear optimization is performed with one parameterization of the fundamental matrix. What is further needed is a system and method for estimating the fundamental matrix that preserves the characteristics of the data noise model from the original image space.
To overcome the limitations in the prior art described above, and to overcome other limitations that will become apparent upon reading and understanding the present specification, the present invention is embodied in a system and method for estimating epipolar geometry, in terms of a fundamental matrix, between multiple images for stereo vision processing. The fundamental matrix embodies the epipolar geometry between the images.
In general, the present invention includes a method for estimating epipolar geometry between multiple images of an original space given an initial estimate of the fundamental matrix found using a standard linear estimation technique. Namely, the system and method of the present invention estimates the fundamental matrix by transforming image points of multiple images into projective space. After this transformation is performed, nonlinear optimization is used with one parameterization of the fundamental matrix, rather than considering 36 distinct parameterizations, as in previous methods. The images are then inverse transformed back to the original space with a final estimate of the fundamental matrix.
Specifically, one method of the present invention involves first identifying an element of the fundamental matrix with the largest absolute value. Second, two appropriate permutation matrices (projective transformations) of the image points are constructed such that the element with the largest absolute value of the fundamental matrix is at location (0,0) of the matrix. Third, the coordinates of each image point are permuted accordingly. Fourth, the fundamental matrix in the permutated space is estimated using a non-linear estimation technique. Last, an inverse permutation is performed to the estimated fundamental matrix of the original space. In addition, the present invention includes a method for preserving the characteristics of a data noise model from the original image space.
The present invention as well as a more complete understanding thereof will be made apparent from a study of the following detailed description of the invention in connection with the accompanying drawings and appended claims.
Referring now to the drawings in which like reference numbers represent corresponding parts throughout:
In the following description of the invention, reference is made to the accompanying drawings, which form a part hereof, and in which is shown by way of illustration a specific example in which the invention may be practiced. It is to be understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the present invention.
FIG. 1 and the following discussion are intended to provide a brief, general description of a suitable computing environment in which the invention may be implemented. Although not required, the invention will be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Moreover, those skilled in the art will appreciate that the invention may be practiced with a variety of computer system configurations, including personal computers, server computers, hand-held devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, and the like. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located on both local and remote computer storage media including memory storage devices.
With reference to
A number of program modules may be stored on the hard disk, magnetic disk 120, optical disk 124, ROM 110 or RAM 112, including an operating system 132, one or more application programs 134, other program modules 136, and program data 138. A user may enter commands and information into the computer 100 through input devices such as a keyboard 140 and pointing device 142. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, or the like. These and other input devices are often connected to the processing unit 102 through a serial port interface 144 that is coupled to the system bus 106, but may be connected by other interfaces, such as a parallel port, game port or a universal serial bus (USB). A monitor 146 or other type of display device is also connected to the system bus 106 via an interface, such as a video adapter 148. In addition to the monitor 146, computers may also include other peripheral output devices (not shown), such as speakers and printers.
The computer 100 may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 150. The remote computer 150 may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the personal computer 100, although only a memory storage device 152 has been illustrated in FIG. 1. The logical connections depicted in
When used in a LAN networking environment, the computer 100 is connected to the local network 154 through a network interface or adapter 158. When used in a WAN networking environment, the computer 100 typically includes a modem 160 or other means for establishing communications over the wide area network 156, such as the Internet. The modem 160, which may be internal or external, is connected to the system bus 106 via the serial port interface 144. In a networked environment, program modules depicted relative to the computer 100, or portions thereof, may be stored in the remote memory storage device. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used.
In general, the first view 214 and the Nth view 216 of the respective images represent the object 210 in original space 218. An initial estimate 220 of the fundamental matrix between the images is predefined, assumed or obtained by any suitable means. Respective image points of the images 214, 216 are then transformed into projective space 222. After this transform, parameters of the fundamental matrix representing the images are analyzed 224 for estimating the fundamental matrix in the projective space 222.
Suitable analysis in accordance with the present invention is performed, such as nonlinear optimization. Only one parameterization of the fundamental matrix needs to be considered, rather than consideration of 36 distinct parameterizations, as in previous systems discussed above. Further, the present invention preserves the characteristics of a data noise model from the original image space. Last, the images 214, 216 are inverse transformed back to the original space and a final estimate of the fundamental matrix 226 is produced in the original space 218.
Next, parameters of the fundamental matrix representing the images are analyzed 224 for estimating the fundamental matrix in the projective space 222, as shown in FIG. 2. This can be accomplished, for example, by performing nonlinear optimization on the fundamental matrix with one predefined parameter characterizing the fundamental matrix (step 314), unlike other methods, which require all 36 parameters. Fifth, an inverse transform of the fundamental matrix is performed for transforming the images 214, 216 back to the original space 218 (step 316). From these steps of
A. Parameterization of the Fundamental Matrix
The discussion below of this section and sections B. and C. is for providing a better understanding of the parameterization of the fundamental matrix. The fundamental matrix embodies the epipolar geometry between the images 214, 216. Referring to the method described above and shown in
For example, the epipolar geometry between a pair of images can be defined by setting C and C' as a pair of pinhole cameras in 3D space and letting m and m' be the projections through cameras C and C' of a 3D point M in images I and I' respectively, as shown in FIG. 4. For point m, in the first image 214, the correspondence in the Nth or second image 216, m'i, lies on an epipolar line in the second image. This epipolar line is the projection of the semi-line Cm onto the second image. This is commonly referred to as the epipolar constraint.
Algebraically, in order for mi and m'i to be matched, the following equation must be satisfied:
where F, is referred to as the fundamental matrix. The fundamental matrix is a 3×3 matrix of rank 2 (i.e., det(F)=0), defined up to a scale factor, where {tilde over (m)}i and {tilde over (m)}'i are homogeneous coordinates of points mi and m'i, i.e., {tilde over (m)}i=[miT,1]T and {tilde over (m)}'i=[m'iT, 1]T. The fundamental matrix F maps points in I to lines in I', and points in I' to lines in I'. That is, if m is a point in I, then Fm=1' is an epipolar line in I'. In fact, any point m' that corresponds with m lies on the epipolar line. For a fundamental matrix F, there exists a pair of unique points e∈I and e'∈I'.
The points e and e' are the epipoles of images I and I' respectively. The epipoles have the property that all epipolar lines in I pass through e. Similarly all epipolar lines in I' pass through e'. In 3D space, e and e' are the intersections of the line CC' with the planes containing image I and I'. The set of planes containing the line CC' are the epipolar planes. Any 3D point M not on line CC' will define an epipolar plane, the intersection of this epipolar plane with the plane containing I or I' will result in an epipolar line, as shown in FIG. 4. Additional epipolar geometrical formulations are described in "Determining the Epipolar Geometry and its Uncertainty: A Review," by Z. Zhang, International Journal of Computer Vision, 27(2):161-1195, 1998, which is incorporated herein by reference.
B. Parameterization Based on the Epipolar Transformation
There are several possible parameterizations for the fundamental matrix. Namely, one row (or column) of the fundamental matrix can be expressed as the linear combination of the other two rows (or columns). One possibility is the following:
with F33=(ax+by)x'+(cx+dy)y'.
In this case, x and y are the coordinates of the first epipole, x' and y' are the coordinates of the second epipole, and a,b,c and d, defined up to a scale factor, parameterize the epipolar transformation mapping an epipolar line in the first image 214 to its corresponding epipolar line in the second image 216.
However, this parameterization does not work if an epipole is at infinity. This is because at least one of x,y,x' and y' has value of infinity. In order to overcome this problem, methods have been proposed to use in total 36 maps to parameterize the fundamental matrix, as summarized below.
C. 36 Maps to Parameterize the Fundamental Matrix
For definitional purposes, the columns of F are denoted by the vectors c1, c2 and c3. The rank-2 constraint on F is equivalent to the following two conditions:
for j0, j1, j2∈{1, 2, 3}, where λ1, λ2 and λ are scalars. Condition (4), as a non-existence condition, cannot be expressed by parameterization. As such, condition (3) is kept and the parameterized set is extended to all the 3×3 matrices of rank strictly less than 3. Hence, the rank-2 matrices of, for example, the following forms:
do not have any parameterization if j0=1 is taken. A parameterization of F is then given by (cj1, cj2, λ1, λ2). This parameterization implies to divide the parameterized set among three maps, corresponding to j0=1, j0=2 and j0=3.
If 3-vector is constructed such that λ1 and λ2 are the j1th and j2th coordinates and 1 is the j0th coordinate, it follows that this vector is the eigenvector of F, and is thus, the epipole in the case of the fundamental matrix. Using such parameterization implies to compute directly the epipole which is often a useful quantity, instead of the matrix itself.
To make the above symmetrical and since the epipole in the other image 216 is desirable, the same decomposition used for the columns is used for the rows, which now divides the parameterized set into 9 maps, corresponding to the choice of column and row as linear combinations of the two columns and two rows left. A parameterization of the matrix is then formed by the two coordinates x and y of the first epipole, the two coordinates x' and y' of the second epipole and the four elements a, b, c and d left by ci
Finally, to take into account the fact that the fundamental matrix is defined only up to a scale factor, the matrix is normalized by dividing the four elements (a, b, c, d) by the largest in absolute value. Therefore, a total of 36 maps to parameterize the fundamental matrix exists.
D. Detailed Operation for Estimating the Fundamental Matrix
The present invention is embodied in a procedure for estimating the fundamental matrix for multiple images taken from distinct viewpoints of a 3D scene without considering all 36 distinct parameterizations. As a result, the present invention models the three-dimensional (3D) object 210 with two-dimensional (2D) images 214, 216, without the burden of implementing a system that considers all 36 parameterizations.
Next, the coordinates of each image point are permuted accordingly (step 514). This ensures that the epipoles are not at infinity. Fourth, the fundamental matrix in the permutated space is estimated using a non-linear estimation technique (step 516). Last, an inverse permutation is performed to the estimated fundamental matrix of the original space (518) to obtain the final estimate of the fundamental matrix (step 520).
More specifically, the present invention transforms the image points of the images 214, 216 of the object 210 of
Namely, the present invention finds the projective transformation in each image, denoted by P and P', such that the first element in the transformed image space of the fundamental matrix has the largest value and the epipoles are not at infinity. For instance, let the image points in the transformed space be
then the fundamental matrix in the transformed space is given by
Given an initial estimate of matrix F0 obtained for example with Hartley's normalized 8-point algorithm, as described in: "In defence of the 8-point algorithm," by R. I. Hartley, In Proceedings of the 5th International Conference on Computer Vision, pages 1064-070, Boston, Mass., June 1995, IEEE Computer Society Press, and incorporated herein by reference, the epipoles {tilde over (e)}0 and {tilde over (e)}'0 are computed. The matrices P and P' are 3×3 permutation matrices determined as follows: First, P and P' are initialized to be identity matrices; second, the position of the largest element of F0, is found, denoted by (i0,j0) (Index of vector or a matrix starts with 0); third, if j0≠0, rows 0 and j0 of matrix P are permuted and elements 0 and j0 of epipole {tilde over (e)}0 are permuted; fourth if i0≠0, rows 0 and i0 of matrix P' are permuted and elements 0 and i0 of epipole {tilde over (e)}'0 are permuted; fifth if |{tilde over (e)}0[1]|>|{tilde over (e)}0[2], elements 1 and 2 of epipole {tilde over (e)}0 are permuted and rows 1 and 2 of matrix P are permuted; and last, if |{tilde over (e)}'0[1]|>|{tilde over (e)}'0[2], then elements 1 and 2 of epipole {tilde over (e)}'0 are permuted and rows 1 and 2 of matrix P' are permuted.
The third and fourth steps ensure that the first element of the fundamental matrix in the transformed space has the largest value, while the fifth and sixth steps ensure that the epipoles are not at infinity. It should be noted that P-1=PT and P'-1=P'T because they are permutation matrices. A detailed explanation for efficiently representing permutation matrix with an integer vector is described in "Matrix Computations," by G. H. Golub and C. F. van Loan, The John Hopkins University Press, Baltimore, Md., 3 edition, 1996, which is incorporated herein by reference. The parameterization of equation (2) can now be used to estimate {circumflex over (F)} from the transformed image points {circumflex over ({tilde over (m)})}i and {circumflex over ({tilde over (m)})}'i. The fundamental matrix in the original image space is given by F=P'T{circumflex over (F)}P.
E. Preserving Information of the Original Noise Distribution
In addition, the present invention includes a method for preserving the characteristics of a data noise model from the original image space. In the present invention, points are detected in the original image space. All nonlinear optimization criteria are derived from the noise distribution of these points. One reasonable assumption is that the image points are corrupted by independent and identically distributed Gaussian noise with mean zero and covariance matrices given by
where σ is the noise level, which is usually unknown. This assumption is clearly no longer reasonable in the transformed image space. In the following description, it is shown how to estimate the transformed fundamental matrix while preserving information of the original noise distribution, which will be illustrated using the gradient-weighted criterion.
From equations (1), (5) and (6), it can be seen that the transformed fundamental matrix and the original image points are related by
fi≡{tilde over (m)}'iTP'T{circumflex over (F)}P{tilde over (m)}i=0. (8)
The least-squares technique produces an optimal solution if each term has the same variance. Therefore, the transformed fundamental matrix can be estimated by minimizing the following weighted sum of squares (the chi-square χ2):
where σf
From equation (7), the covariances of the homogeneous coordinates of the image points are given by
Under first order approximation, the variance of fi is then given by
where {circumflex over (1)}'i={circumflex over (F)}{circumflex over ({tilde over (m)})}i and {circumflex over (1)}i={circumflex over (F)}T{circumflex over ({tilde over (m)})}'i are epipolar lines in the transformed image space. Since multiplying each term by a constant does not affect the minimization, the expression in (9) becomes
The denominator is simply the gradient of fi. The minimization can be conducted by any suitable technique, such as the Levenberg-Marquardt technique.
In conclusion, the present invention estimates the fundamental matrix of multiple images of an original space given an initial estimate of the fundamental matrix found using a standard linear estimation technique. Namely, the fundamental matrix is estimated by transforming image points of multiple images into projective space. After this transformation is performed, nonlinear optimization is used with one parameterization of the fundamental matrix, rather than considering 36 distinct parameterizations, as in previous methods. The images are then inverse transformed back to the original space with a final estimate of the fundamental matrix. In addition, the present invention includes a method for preserving the characteristics of a data noise model from the original image space.
The foregoing description of the invention has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. It is intended that the scope of the invention be limited not by this detailed description, but rather by the claims appended hereto.
Loop, Charles T., Zhang, Zhengyou
Patent | Priority | Assignee | Title |
10354407, | Mar 15 2013 | LABLANS, PETER, MR | Camera for locating hidden objects |
10388031, | Aug 31 2016 | National Chiao Tung University | Method and system for estimating epipolar geometry |
10585344, | May 19 2008 | LABLANS, PETER, MR | Camera system with a plurality of image sensors |
10896327, | Mar 15 2013 | LABLANS, PETER, MR | Device with a camera for locating hidden object |
11119396, | May 19 2008 | LABLANS, PETER, MR | Camera system with a plurality of image sensors |
7031497, | Nov 05 2001 | UNILOC 2017 LLC | Method for computing optical flow under the epipolar constraint |
7612832, | Mar 29 2005 | Microsoft Technology Licensing, LLC | Method and system for video clip compression |
7751613, | Sep 13 2005 | Industrial Technology Research Institute | Method for rapidly establishing image space relation using plane filtering constraint |
8179448, | Aug 26 2008 | NATIONAL TAIWAN UNIVERSITY | Auto depth field capturing system and method thereof |
8238612, | May 06 2008 | Honeywell International Inc. | Method and apparatus for vision based motion determination |
8525879, | Dec 15 2009 | Industrial Technology Research Institute | Depth detection method and system using thereof |
9171221, | Jul 18 2010 | Spatial Cam LLC | Camera to track an object |
9736368, | Mar 15 2013 | LABLANS, PETER, MR | Camera in a headframe for object tracking |
Patent | Priority | Assignee | Title |
5821943, | Apr 25 1995 | COGNITENS LTD | Apparatus and method for recreating and manipulating a 3D object based on a 2D projection thereof |
6516099, | Aug 05 1997 | Canon Kabushiki Kaisha | Image processing apparatus |
6614429, | May 05 1999 | Microsoft Technology Licensing, LLC | System and method for determining structure and motion from two-dimensional images for multi-resolution object modeling |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Jun 09 2000 | ZHANG, ZHENGYOU | Microsoft Corporation | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 010934 | /0414 | |
Jun 09 2000 | LOOP, CHARLES T | Microsoft Corporation | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 010934 | /0414 | |
Jun 16 2000 | Microsoft Corporation | (assignment on the face of the patent) | / | |||
Oct 14 2014 | Microsoft Corporation | Microsoft Technology Licensing, LLC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 034541 | /0001 |
Date | Maintenance Fee Events |
Jan 11 2008 | M1551: Payment of Maintenance Fee, 4th Year, Large Entity. |
Sep 21 2011 | M1552: Payment of Maintenance Fee, 8th Year, Large Entity. |
Jan 20 2016 | M1553: Payment of Maintenance Fee, 12th Year, Large Entity. |
Date | Maintenance Schedule |
Aug 03 2007 | 4 years fee payment window open |
Feb 03 2008 | 6 months grace period start (w surcharge) |
Aug 03 2008 | patent expiry (for year 4) |
Aug 03 2010 | 2 years to revive unintentionally abandoned end. (for year 4) |
Aug 03 2011 | 8 years fee payment window open |
Feb 03 2012 | 6 months grace period start (w surcharge) |
Aug 03 2012 | patent expiry (for year 8) |
Aug 03 2014 | 2 years to revive unintentionally abandoned end. (for year 8) |
Aug 03 2015 | 12 years fee payment window open |
Feb 03 2016 | 6 months grace period start (w surcharge) |
Aug 03 2016 | patent expiry (for year 12) |
Aug 03 2018 | 2 years to revive unintentionally abandoned end. (for year 12) |