Wide-angle dewarping method and apparatus

Abstract

A method and apparatus for transforming wide angle video into perspective corrected viewing zones which either a single user or multiple users may select, orient and magnify. The present invention first captures a wide angle digital video input by any suitable means. The captured image is then stored in a suitable memory means so portions of the image may be selected at a later time. When a portion of the stored video is selected for viewing, a plurality of discrete viewing vectors in three dimensional space are chosen on the video input and transformed to a plurality of control points in a two dimensional plane or any other suitable surface. The area between these points which is still warped from the original wide angle image capture is then transformed to a perspective corrected field of view. The perspective corrected field of view is then displayed on a suitable displaying apparatus, such as a monitor or head mounted display.

Description

This is a
V=WB  (5)
To discover a_ij and b_ij according to the method of the present invention, a pseudo-inverse technique is used. However, one skilled in the art will appreciate that there are methods to solve equations (5) other than by a pseudo inverse technique, i.e. a least squares technique. The pseudo-inverse solutions for A and B in the above equation (5) are:
A=(W^TW)⁻¹W^TU
B=(W^TW)⁻¹W^TV  (6)
Therefore, for a target display Cot a given pixel resolution N×M, W and its pseudo-inverse (W^TW)⁻¹W^T can be calculated a priori. The values for a_ij and b_ij are then found by mapping the points in the U-V plane for the 3×3 grid of control points using the above equations (6). The biquadratic polynomial transformations of the equations (3) are then used to transform the area between the control points. In this embodiment, the determination of the coordinates of each pixel in the U-V plane takes a total of thirteen multiplication and ten addition operations. Additionally, three of the required multiplication operations per pixel may be obviated by storing a table of xy, x² and y² values for each xy coordinate pair in the dewarped destination image. In another embodiment, the “x” values which do not vary as “y” changes (i.e. a₁*x+a₄*x² and b₁*x+b₄*x²) may also be precomputed and stored. Likewise, the “y” values which do not vary as “x” changes may be precomputed and stored. These further optimizations reduce the operations needed to determine the coordinates of each pixel in the U-V plane to two multiplication and four addition operations.

It will be appreciated by one skilled in the art that the accuracy of the dewarping transformation will increase as the number of transformed viewing vectors increases, i.e. a 4×4 grid of control points will produce a more accurate transformation than a 3×3 grid of control points. However, the amount of increase in accuracy quickly draws near an asymptote as the number of control points is increased. One skilled in the art will recognize, therefore, that there is little reason in increasing the number of viewing vectors to more than half of the total number of pixels in the displayed region.

It will be further appreciated by one skilled in the art that the selection of a rectangular shape of the video input could be changed to another shape and still be within the scope of the invention. Further, the number of control points could be increased or decreased to correspondingly increase or decrease the accuracy of the transformation. Further still, an image filtering stage could be applied during the inverse texture mapping without deviating from the present invention.

FIG. 9 shows a functional flow chart of the major elements of one embodiment of the present invention. First, the fixed warped image parameters (step 400) are defined, such as the size of the input image, the input image radius, and the input image center in U-V coordinates typically measured in pixels. The next step 410 is to initialize the variable dewarped image parameters, such as the size of the dewarped image area, the horizontal and vertical field of views (generally shown in degrees), the creation of an untransformed view cone centered in this embodiment on the +Z axis and the initialization of the layout and number control points used therewith. Typically, the next step is to load the precomputed inner-loop matrix values as well as the “xy” product terms, as shown in step 420, to ensure that the transformation is accomplished as quickly and efficiently as possible. In step 430, the video signal is input to the system in any suitable form, i.e. live or pre-recorded real-time digitized video or computer synthesized video environments. The system then allows the user to select the viewing vector (step 440) which in turn determines the portion of video which is to be transformed. The control points are next transformed from the selected viewing vectors (step 450) and the region defined by the control points is dewarped (step 460). The signal is then sent to the video buffer and to an appropriate viewing apparatus (step 470). There is a recursive loop from step 470 to step 430 to allow the video signal to be refreshed, as is needed with live motion video. The loop also allows the user to make on-the-fly selections of alternate portions of the incoming video.

The foregoing description describes an inverse texture mapping technique whereby the biquadratic output (X-Y) is mapped to the input (U-V). In the case where a forward texture mapping ASIC is used, the coordinates for the destination control points in X-Y must be supplied so that the rectilinear source texture region can be mapped from the U-V plane, as provided by the inverse texture mapping software solution above, to the X-Y plane. An example of a forward texture mapping ASIC is the NV-1 chip sold by N-Vidia Corporation. FIG. 6 gives another example of an ASIC chip 230 which accepts control pixel coordinates at a forward mapping solution stage 240. Four pixel coordinates will be accepted for a bilinear mapping, nine in the case of quadratic mapping, 16 in the case of cubic mapping, etc. These control pixels are produced in the host CPU according to the equations (7), below. As shown in FIG. 7, the coordinates of a rectangular bounding box are determined to enclose an exemplary 3×3 grid of control points u_iv_i in the U-V plane. The corners of the bounding box are found from the u_iv_i extrema. Using the same technique described in equations (3), a_ij and b_ij for N=2 can be solved with the equations (7) for enclosing the region to be warped to the corners of the display screen.

$\begin{array}{cc}x=\underset{i=0}{\sum ^N}\underset{j=0}{\sum ^{N-i}}{a}_{\mathrm{ij}}{u}^i{v}^j& \left(7\right)\\ y=\underset{i=0}{\sum ^N}\underset{j=0}{\sum ^{N-i}}{b}_{\mathrm{ij}}{u}^i{v}^j& \end{array}$

Thus, the same control points for the U-V plane map to the corners of the display screen in the X-Y plane. The warped regions outside the bounding box may be clipped by hardware or software so that they are not visible on the display screen.

The source pixel coordinates, which are fed from the host CPU, are converted to a_ij and b_ij coordinates for forward mapping in the forward mapping solution stage 240, again using techniques mathematically equivalent to those of the equations (7). A series of instructions is further sent from the host CPU to the chip 230 and received by a control unit 260. The control unit 260 sequences and controls the operation of the other functional stages within the chip 230. The host CPU also directs a linear sequence of source pixels, which are to be warped, to an interpolation sampler stage 250 within chip 230. Optionally, these can be subject to a low-pass spatial prefiltering stage 270 prior to transmission to the chip, to reduce sampling error during the warping process. Thus, within the chip 230, the source pixels and the a_ij and b_ij coordinates are both fed to the interpolation sampler 250. For each input pixel, one or more destination pixels together with their corresponding X-Y destination coordinates are produced. These warped pixels are then fed into the video frame buffer 280, located outside of the ASIC chip 230. Optionally, anti-aliasing circuitry 290 within the chip performs interpolation on output pixel values, such as bilinear interpolation between adjacent pixel samples, to minimize the effects of output spatial quantization error. One skilled in the art will recognize that the preceding hardware solution is merely exemplary and that there are many such solutions which could be employed and still fall within the scope of the present invention.

The techniques described herein may also be applied to synthetic images. Such images may be created entirely within a computer environment and may be composed of three dimensional geometrical descriptions of objects which can be produced by computer graphics rendering techniques generally known to those skilled in the art. Typically, synthetic images are produced by linear perspective projection, emulating the physical process of imaging onto planar film with a lens having a narrow field of view and producing a view of the synthetic environment as seen through a cone or truncated three dimensional pyramid. The color, intensity shading and other simulated physical properties of each pixel on the planar image grid can also be readily determined. For a synthetic environment, the viewing vectors in X-Y-Z space are rewritten in terms of the warped control points coordinates in the U-V plane

$\begin{array}{cc}r=\sqrt{u^2+v^2}& \left(8\right)\\ s=\frac{r}{\mathrm{RADIUS}}& \\ x=\frac{\mathrm{us}}{r}& \\ y=\frac{\mathrm{vs}}{r}& \\ z=\sqrt{1-s^2}& \end{array}$

A direction vector in X-Y-Z space can thus be generated for each pixel in the U-V plane in the synthetic wide angle image which is created. For a perfect hemispherical fisheye, the generated vectors point in all directions within the created hemisphere, spaced to the limits of the resolution of the U-V image. This simulates a non-planar image grid, such as the projection of the synthetic environment onto a surface of a spherical image substrate. In this way, a fisheye or other wide angle image of a synthetic three dimensional environment can be produced. This technique can be used for the production of three dimensional modeled cartoons or interactive home gaming applications, among others.

One skilled in the art will appreciate that the present invention may be applied to a sequence of wide angle images changing in time, either live or recorded to an analog or digital storage media. The image substrate for recordation may be an electronic two dimensional image sensor, such as a CCD chip, or photographic film capable of chemically recording the image for subsequent transfer into digital form.

One skilled in the art will also appreciate that the present invention is not limited to transforming wide angle video onto a planar (U-V) surface, but that it is within the scope of the invention to transform wide angle video onto any suitable surface for displaying the video for the user.

Further, two real world, wide angle lenses can be positioned opposite each other to permit near 360 degrees of total coverage of the environment. If seamless omnidirectional coverage of an environment is required, this could be achieved with six wide angle lenses positioned along each direction of a three dimensional axis, as shown in FIG. 8. This arrangement can be coupled with a video switching mechanism for choosing which signal is to be dewarped for the selected view and orientation of the video input.

Further still, the same video signal may be simultaneously transmitted to an arbitrarily large number of viewers all having the ability to simultaneously dewarp the same or different portions of the video input, as in the case of interactive cable TV viewing or multiple player online interactive video game playing.

Having fully described the preferred embodiment of the present invention, it will be apparent to those of ordinary skill in the art that numerous alternatives and equivalents exist which do not depart from the invention set forth above. It is therefore to be understood that the present invention is not to be limited by the foregoing description, but only by the appended claims.

Claims

1. A method for providing perspective corrected images from at least one distorted image, the method comprising steps of:

receiving said distorted image;

storing a portion of said distorted image;

transforming a set of control vectors to a set of control points that defines an area that associates said portion of said distorted image with a portion of a perspective corrected image;

transforming said portion of said distorted image associated with said area to said portion of said perspective corrected image using a global bivariate polynomial transformation;

displaying said portion of said perspective corrected image;

sensing inputted information; and

controlling the transformation and display of said perspective corrected image through said inputted information.

2. The method of claim 1 wherein the step of transforming said portion of said distorted image is accomplished using:

3. The method of claim 2 wherein N is 2 or 3.

4. The method of claim 1 wherein said set of control points contains a number of control points, said number having a lower limit of five and an upper limit of one-half of the number of pixels in said portion of said perspective corrected image.

5. The method of claim 1 wherein said distorted image is received from a computer storage device.

6. The method of claim 1 wherein said distorted image is received from a network.

7. The method of claim 6 wherein said network is a computer network.

8. The method of claim 6 wherein said network is in communication with the Internet.

9. The method of claim 1 wherein said distorted image is a result of imaging an environment through at least one wide-angle lens.

10. The method of claim 1 wherein said distorted image is a result of imaging an environment through at least one fisheye lens.

11. The method of claim 1 wherein said distorted image includes an image of at least one sixth of an environment.

12. An apparatus for providing perspective corrected images from at least one distorted image, the apparatus comprising:

an input configured to receive said distorted image;

a memory, coupled to the input, configured to store a portion of said distorted image;

a processor, coupled to the memory, configured to transform a set of control vectors to a set of control points that defines an area that associates said portion of said distorted image with a portion of a perspective corrected image, the processor further configured to transform said portion of said distorted image associated with said area to said portion of said perspective corrected image using a global bivariate polynomial transformation;

a presentation mechanism, coupled to the memory, configured to present said portion of said perspective corrected image; and

a selection mechanism, coupled to the processor, configured to specify said set of control vectors.

13. The apparatus of claim 12 wherein the transformation of said portion of said distorted image is accomplished using:

14. The method of claim 13 wherein N is 2 or 3.

15. The apparatus of claim 12 wherein said set of control points contains a number of control points, said number having a lower limit of five and an upper limit of one-half of the number of pixels in said portion of said perspective corrected image.

16. The apparatus of claim 12 wherein said distorted image is received from a computer storage device.

17. The apparatus of claim 12 wherein said distorted image is received from a network.

18. The apparatus of claim 17 wherein said network is a computer network.

19. The apparatus of claim 17 wherein said network is in communication with the Internet.

20. The apparatus of claim 12 wherein said distorted image is a result of imaging an environment through at least one wide-angle lens.

21. The apparatus of claim 12 wherein said distorted image is a result of imaging an environment through at least one fisheye lens.

22. The apparatus of claim 12 wherein said plurality of distorted images includes an image of at least one sixth of an environment.

23. A method of providing perspective corrected views of live, prerecorded or simulated environments from wide angle video signals, the method comprising:

receiving video input data at a computing device;

transforming, by the computing device, a plurality of viewing vectors defining a portion of the video input data to a plurality of control points;

transforming, by the computing device, pixel data in an area between the plurality of control points to define perspective corrected pixel data; and

sending the perspective corrected pixel data to a display for display of the perspective corrected pixel data.

24. The method of claim 23 further comprising determining the plurality of viewing vectors at the computing device based on position and orientation information of a device and a horizontal field of view and a vertical field of view.

25. The method of claim 23 further comprising receiving a selection of the plurality of viewing vectors at the computing device.

26. The method of claim 23 wherein the plurality of viewing vectors define an area of interest.

27. The method of claim 23 wherein a global bivariate polynomial transformation is used to define the perspective corrected pixel data.

28. The method of claim 27 wherein the global bivariate polynomial transformation is a biquadratic polynomial transformation.

29. A system comprising:

a computer-readable memory configured to store video input data; and

a processor operably coupled to the computer-readable memory to receive the stored video input data, the processor configured to perform operations comprising

transforming a plurality of viewing vectors defining a portion of the stored video input data to a plurality of control points;

transforming pixel data in an area between the plurality of control points to define perspective corrected pixel data; and

sending the perspective corrected pixel data to a display.

30. The system of claim 29, wherein the processor is an ASIC.

31. The system of claim 29, wherein the computer-readable memory has computer-executable instructions stored thereon execution of which by the processor causes the processor to perform the operations.

32. The system of claim 29, wherein the operations further comprise determining the plurality of viewing vectors based on position and orientation information of a device, a horizontal field of view, and a vertical field of view.

33. The system of claim 29, wherein the operations further comprise receiving a selection of the plurality of viewing vectors.

34. The system of claim 29, wherein the plurality of viewing vectors define an area of interest.

35. The system of claim 29, wherein a global bivariate polynomial transformation is used to define the perspective corrected pixel data.

36. The system of claim 35, wherein the global bivariate polynomial transformation is a biquadratic polynomial transformation.

37. A method of performing perspective correction, the method comprising:

transforming, by a computing device, a set of control vectors into a set of control points using a function that models a wide angle lens;

generating, by the computing device, a polynomial transform function that maps the control points into rectangular points;

transforming, by the computing device, an area of image data proximate the set of control points using the polynomial transform function; and

sending the transformed area of image data to a display for display of the transformed area of image data.

38. The method of claim 37, wherein the polynomial transformation function comprises a global bivariate polynomial.

39. The method of claim 37, wherein the polynomial transformation comprises a biquadratic polynomial.

40. The method of claim 37, wherein the set of control vectors comprise a principal viewing vector and eight surrounding vectors that form a rectangular view.

41. The method of claim 37, wherein the wide-angle lens comprises a fish eye lens.

42. A system comprising:

a computer-readable memory configured to store captured image data; and

a processor operably coupled to the computer-readable memory to receive the stored captured image data, the processor configured to perform operations comprising

transforming a set of control vectors into a set of control points using a function that models a wide angle lens;

generating a polynomial transform function that maps the control points into rectangular points;

transforming an area of the captured image data proximate the set of control points using the polynomial transform function; and

sending the transformed area of image data to a display.