Method and multi-camera portable device for producing stereo images

Abstract

A method and a multi-camera portable device for producing stereo images. The device has at least two image sensors, at least one of which is arranged on a first face of the portable device and a second one of which is arranged on a second face of the portable device opposite to the first face thereof. The method involves substantially simultaneously recording a first image containing a picture of an object and a second image containing a picture of a mirrored view of the object, obtaining the direction and the length of a mirroring vector from the distance ratio of a first distance in reality and a corresponding second distance in the second image, obtaining a capturing focal length of the second image sensor, and transforming the coordinate system of the first image sensor into the coordinate system of the virtual second image sensor to generate a stereo image pair.

Description

This application
U_1,y−U_2,y=(R_1,y−R_2,y)H₂/R_z
Since
d_u1,u2=√{square root over ((R_1,x−R_2,x)²+(R_1,y−R_2,y)²)}
therefore

$d_{U_1,U_2}=d_{u_1,u_2}\frac{H_2}{R_z}$
That means

$R_z=H_2\frac{d_{u_1,u_2}}{d_{U_1,U_2}}$

Since the coordinates R₁ and R₂ are located on the portable device 100, it makes sense to assume that R_z is equal to the length of m. (If one of the selected points u₁, u₂ on the portable device 100 was the second camera's center point, then R_z would coincide with the end point of vector m.)

As a further step, the pictures of the two selected points u₁, u₂ of the portable device 100 in the second initial image I2 are to be found. To this end, one of the possible techniques is to find the pictures of the corners of the portable device 100, or for example, the distance d between the picture of flash and picture of the second camera's lens in the second initial image I2, which can be performed by any appropriate edge-detection technique in the second initial image I2.

Upon obtaining a set of hypothetical positions for the picture C2 of the mirrored view of the second camera 104 within the second initial image I2, a filtering is carried out for all of the possible positions of the picture C2 of the mirrored view of the second camera 104 in the search area 150 of the second initial image I2 (this search region is defined by the output of step S206) to find a limited set of likely positions and distance ratios thereof. This can be done, for example, by averaging edge-strength values in various regions of the search area that are specific to the particular portable device 100. For speeding up this processing, a pre-stored picture of an edge map of the portable device may be used.

In FIG. 5 a typical edge map of a portable device is shown, wherein the edge map contains a picture D2 of the mirrored view of the portable device with marked side edge boxes 140, 141, a frame box 142 of the second camera lens and a box 143 surrounding the flash lamp of the portable device. As a result, specific camera positions (center and size of picture C2) in the previously defined search region 150 in the second initial image I2 can be located.

Next, the estimated position of the picture C2 of the mirrored view of the second camera 104 in the second initial image I2 is determined by searching for the specific rounded square contour of the frame of the second camera 104. This may be done using a standard generalized Hough transform, resulting in the recognition of the picture of said specific portion of the portable device 100 in the second initial image I2.

It should be noted that the above described steps of finding the second camera's picture C2 (or the picture of the frame around the camera lens) within the second initial image I2 is only one of the various possible algorithms to determine the camera's center (c_x,c_y) and the distance ratio

$\left(\frac{d_{u_1,u_2}}{d_{U_1,U_2}}\right)$
that are obvious for those skilled in the art, and therefore it no way means any limitation of the present invention to such a specific search technique. The length of the vector m can then be calculated using the above equations.

3) Finally, after having obtained the direction and the length of the vector m, only the focal length (or the corresponding relative focal length H₂) of the second camera 104 should be determined in step S210 for the mirror transformation matrix M.

In some cases the actual relative focal length H₂ of the second camera 104 may be obtained from a controller circuit of the second camera 104, and therefore it may be regarded known. However, when the relative focal length H₂ is not available, a good estimation for the focal length thereof may be given by various estimation techniques.

The focal length f₂ of the second camera 104 may be estimated in the following way. In general, it is carried out by creating several fundamental matrices F, with different values of the relative focal length H₂ (assuming that the pixel size s of the second camera is known), generating a series of candidate stereo image pairs, in which the first initial image I1 is mapped into the coordinate system of the virtual second camera 105 with the various fundamental matrices F_i, followed by choosing the stereo image pair that corresponds to the best stereo result. The capturing focal length of the second camera 104 is then determined from the relative focal length belonging to the best fundamental matrix F_i of the best matching stereo image pair.

One possible technique to find the best estimate for the relative focal length H₂ of the second camera 104 uses the fact that the stereo disparity has a very low resolution near the epipole. Since the position of the epipole (i.e. the image of the focal point of the first camera 102) in the second initial image I2 is known when the picture C2 of the mirrored view of the second camera 104 has been found within the second initial image I2, the corresponding region of the first initial image I1 that best matches to the picture near the epipole where the portable device 100 does not occlude the object 120. Although the picture of the portable device 100 (and therefore the picture of the first camera 102) does not appear on the first initial image I1, the corresponding region can be still found, for example, by using various fundamental matrices F_i and checking the correlation between associated sub-windows of the initial image pair.

These search regions 162, 163 are shown in FIG. 6, wherein the search regions 162 are the regions in the second image I2 for which the best matching search regions 163 in the first image I1 was previously round. Regardless of scale (defined by the fundamental matrix) the search region 163 of image I1 is expected to appear quite exactly (the color image correlation between the windows 162 and 163 are expected to be high) in the first initial image I1. These regions 163 are searched in the first image I1 with fundamental matrices defined by different values of the focal length of the second camera 104.

The focal length that refers to the best match, i.e. the highest correlation between the associated sub-windows, via the corresponding calibration matrix F_i, is the estimate for the actual focal length. This algorithm results in a stereo calibration matrix F that can be visualized by the epipolar lines shown in FIG. 7, in which the lines with the same reference sign (e.g. a, a′; or b, b′, etc.) are mutually corresponding epipolar lines, i.e. any point on one of the initial images I1, I2 should belong to a real 3D point that has its picture on the other one of the images I1, I2 along the corresponding epipolar line (assuming that the mentioned point is not occluded).

Upon obtaining the best estimation for the relative focal length H₂ of the second camera 104, the mirror transformation matrix M is given, therefore the fundamental matrix F can be calculated using the formula F=M*K, and therefore in step S212, the coordinate system of the first camera 102 can be mapped into the coordinate system of the virtual second camera 105 to produce a stereo image pair for the object 120 from the initial images I1 and I2.

Once the stereo system used in the method of the invention is calibrated through the above steps, a depth estimation for a captured object may be performed to generate a depth image of the object. In particularly, when the object arranged in front of the portable device (i.e. in front of its first camera) is a human face, several characterizing parameters of the face can be estimated using the depth image thereof, including facial expression, position of the nose, the mouth, the eyes, lips, etc.

Once the 3-dimensional image of a human face is estimated, for example virtual glasses, make-ups, jewelleries, or the like, may be displayed over the captured images resulting in augmented reality. Using this augmented reality, for example, a software application running on the portable device may be used to take synchronized photos or video sequences by means of the front and rear cameras of a multi-camera portable device with using a mirror, and additional virtual objects may be displayed by the software over those images or video sequences, such as virtual glasses over the face of a user.

In a second aspect, the present invention relates to a multi-camera portable device for producing stereo images with using an external mirror. As shown in FIG. 8, the multi-camera portable device has at least two cameras 302, 304 at opposite faces thereof. Preferably, the multi-camera portable device 300 has at least one front camera 302 on its front side and at least one rear camera 304 on its backside. The multi-camera portable device 300 is adapted for performing the above described method of producing stereo images when an object to be captured is arranged in front of its front-side camera 302, and a mirror is arranged in front of its back-side camera 304. The multi-camera portable device 300 farther comprises all of those common components that are needed for the normal operation of such a device, including at least a processor unit 306, a memory unit 308, a non-transitory data storage medium 309, a display unit 310 and an input unit 312, wherein the multi-camera portable device 300 is configured to be capable of carrying out the steps of the methods according to the present invention. Preferably, the device 300 comprises a set of computer readable instructions either in its memory unit 308 or its non-transitory data storage medium 309, said instructions being adapted, when executed by the processor unit 306, to carry out the portable device-related steps of the method of the present invention.

In a third aspect, the present invention relates to a non-transitory data storage medium comprising a set of processor-readable instructions, said instructions being adapted, when executed by a processor of a multi-camera portable device, to carry out the steps of:

• • instructing a first image sensor of the device to record a first initial image containing a picture of an object arranged within the field of view of the first image sensor, and a second image sensor of the device to substantially simultaneously record a second initial image containing (i) a picture of a mirrored view of the object appearing in a mirror and (ii) a picture of at least a portion of a mirrored view of the multi-camera portable device itself appearing in the mirror, thereby to produce an initial pair of images, wherein said mirrored view of the second image sensor and said portion of the mirrored view of the device are arranged within the field of view of the second image sensor,
  • finding the center of the picture of the mirrored view of the second image sensor within the second initial image,
  • determining a distance ratio of a first distance between two points of the device in reality and a second distance of the respective points in the second initial image for obtaining a direction and a length of a mirroring vector, which points from the center of the second image sensor to the center of the virtual second image sensor, obtaining a capturing focal length of the second image sensor, and
  • by using said mirroring vector and the capturing focal length of the second image sensor, transforming the coordinate system of the first image sensor into the coordinate system of the virtual second image sensor to generate a stereo image pair from the first initial image and the second initial image for the object.

Claims

1. A method for producing a stereo image of an object with using a multi-camera portable device having at least two image sensors, at least a first one of said image sensors being arranged on a first face of the portable device and a second one of said image sensors being arranged on a second face of the portable device opposite to the first face thereof, the method comprising the steps of:

arranging the object in front of the first image sensor so that the object falls within the field of view of the first image sensor,

arranging an external mirror in front of the second image sensor so that a mirrored view of the object and a mirrored view of the portable device fall within the field of view of the second image sensor,

substantially simultaneously recording, by the first image sensor, a first initial image containing a picture of the object, and by the second image sensor, a second initial image containing (i) a picture of the mirrored view of the object appearing in the mirror and (ii) a picture of at least a portion of the mirrored view of the portable device appearing in the mirror, thereby producing an initial pair of images,

finding the center of the picture of the mirrored view of the second image sensor within the second initial image,

determining the distance ratio of a first distance between two points of the portable device in reality and a second distance of the respective points in the second initial image for obtaining the direction and the length of a mirroring vector, which points from the center of the second image sensor to the center of the virtual second image sensor,

obtaining a capturing focal length of the second image sensor, and

by using said mirroring vector and the capturing focal length of the second image sensor, transforming the coordinate system of the first image sensor into the coordinate system of the virtual second image sensor to generate a stereo image pair from the first initial image and the second initial image for the object.

2. The method of claim 1, wherein the multi-camera portable device is any one of a mobile phone, a smart phone, a phablet, a tablet computer, a notebook.

3. The method of claim 1, wherein finding the center of the picture of the mirrored view of the second image sensor in the second initial image comprises the steps of:

a. finding a sub-image within the second initial image, wherein said sub-image best matches to the first initial image,

b. determining a plurality of candidate positions and sizes of the picture of the mirrored view of the second image sensor in the second initial image by using various selected values of the relative focal length of the second image sensor and various selected distance values between the second image sensor and the object to find the center point of the picture of the mirrored view of the second image sensor in said plurality of candidate positions in the second initial image,

c. selecting a best estimate for the position and the size of the picture of the mirrored view of the second image sensor in the second initial image by using a standard generalized Hough transform.

4. The method of claim 1, wherein the capturing focal length of the second image sensor is obtained from a controller of the second image sensor.

5. The method of claim 1, wherein the capturing focal length of the second image sensor is obtained by performing the steps of:

a. creating several coordinate transformation matrices with different values of a relative focal length of the second image sensor for mapping the coordinate system of the first image sensor into the coordinate system of the virtual second image sensor, said relative focal length being defined as the ratio of the focal length and the pixel size of the second image sensor,

b. generating a series of candidate stereo image pairs, wherein in each of the candidate stereo image pairs, the first initial image is transformed into the coordinate system of the virtual second image sensor with a different coordinate transformation matrix, and

c. selecting the best matching stereo image pair,

d. determining the capturing focal length of the second image sensor from the relative focal length belonging to the coordinate transformation matrix of the best matching stereo image pair.

6. The method of claim 1, wherein the method further comprises the step of producing a depth map of the object using the stereo image pair containing the object.

7. A non-transitory data storage medium comprising a set of processor-readable instructions, said instructions being adapted, when executed by a processor of a multi-camera portable device, to carry out the steps of:

instructing a first image sensor of the device to record a first initial image containing a picture of an object arranged within the field of view of the first image sensor, and a second image sensor of the device to substantially simultaneously record a second initial image containing (i) a picture of a mirrored view of the object appearing in a mirror and (ii) a picture of at least a portion of a mirrored view of the multi-camera portable device itself appearing in the mirror, thereby to produce an initial pair of images, wherein said mirrored view of the second image sensor and said portion of the mirrored view of the device are arranged within the field of view of the second image sensor,

finding the center of the picture of the mirrored view of the second image sensor within the second initial image,

determining a distance ratio of a first distance between two points of the device in reality and a second distance of the respective points in the second initial image for obtaining a direction and a length of a mirroring vector, which points from the center of the second image sensor to the center of the virtual second image sensor, obtaining a capturing focal length of the second image sensor, and

by using said mirroring vector and the capturing focal length of the second image sensor, transforming the coordinate system of the first image sensor into the coordinate system of the virtual second image sensor to generate a stereo image pair from the first initial image and the second initial image for the object.

8. A method of producing a stereo image of an object with using a multi-camera device having at least two image sensors, the method comprising the steps of:

arranging a first image sensor on one side of a portable device and another image sensor on the opposite side of the portable device and an external mirror in front of the first image sensor in which the first image sensor captures a picture of an object and substantially simultaneously second image sensor captures picture of the mirrored image of the object and mirrored image of at least a portion of the portable device;

determining a distance ratio of a first distance between two points of the portable device in reality and a second distance of the respective points in the second initial image for obtaining the direction and the length of a mirroring vector, which points from the center of the second image sensor to the center of the virtual second image sensor; obtaining a capturing focal length of the second image sensor, and by using said mirroring vector and the capturing focal length of the second image sensor, transforming the coordinate system of the first image sensor into the coordinate system of the virtual second image sensor to generate a stereo image pair from the first initial image and the second initial image for the object.

9. A method comprising:

obtaining a first image, the first image comprising a first picture of an object;

obtaining a second image comprising a second picture and a third picture, the second picture comprising a mirror view of the object, and the third picture comprising a mirror view of a camera;

identifying a center of the third picture comprising the mirror view of the camera;

determining a relative focal length defined for the camera;

determining a distance ratio based at least on the relative focal length for the camera and the center of the third picture comprising the mirror view of the camera;

determining a mirroring vector based on the distance ratio and the center of the third picture comprising the mirror view of the camera;

determining a coordinate transformation from a first coordinate system of the first image and a second coordinate system of the second image based at least on the distance ratio, the relative focal length of the camera, and the mirroring vector; and

generating a stereo image pair of the object based at least on the coordinate transformation from the first coordinate system of the first image and the second coordinate system of the second image.

10. The method of claim 9, further comprising measuring a distance associated with the object based on the stereo image pair of the object.

11. The method of claim 9, further comprising generating a depth image of the object based on the stereo image pair of the object.

12. The method of claim 9, further comprising generating a three dimensional (3D) rendering of the object based on the stereo image pair of the object.

13. The method of claim 9, wherein determining the coordinate transformation from the first coordinate system of the first image and the second coordinate system of the second image is based on a coordinate transformation associated with a coordinate system from a perspective of a camera taking the first image and a coordinate system from a perspective of a camera taking the second image.

14. The method of claim 9, wherein the second coordinate system of the second image comprises a coordinate system corresponding to the second picture comprising a mirror view of the object in the second image.

15. A non-transitory data storage medium comprising a set of processor-readable instructions that when executed by a processor cause the processor to execute a method comprising:

obtaining a first image, the first image comprising a first picture of an object;

obtaining a second image comprising a second picture and a third picture, the second picture comprising a mirror view of the object, and the third picture comprising a mirror view of a camera;

identifying a center of the third picture comprising the mirror view of the camera;

determining a relative focal length defined for the camera;

determining a distance ratio based at least on the relative focal length for the camera and the center of the third picture comprising the mirror view of the camera;

determining a mirroring vector based on the distance ratio and the center of the third picture comprising the mirror view of the camera;

determining a coordinate transformation from a first coordinate system of the first image and a second coordinate system of the second image based at least on the distance ratio, the relative focal length of the camera, and the mirroring vector; and

generating a stereo image pair of the object based at least on the coordinate transformation from the first coordinate system of the first image and the second coordinate system of the second image.

16. The non-transitory data storage medium of claim 15, wherein the method executed by the processor further comprises measuring a distance associated with the object based on the stereo image pair of the object.

17. The non-transitory data storage medium of claim 15, wherein the method executed by the processor further comprises generating a depth image of the object based on the stereo image pair of the object.

18. The non-transitory data storage medium of claim 15, wherein the method executed by the processor further comprises generating a three dimensional (3D) rendering of the object based on the stereo image pair of the object.

19. The non-transitory data storage medium of claim 15, wherein determining the coordinate transformation from the first coordinate system of the first image and the second coordinate system of the second image is based on a coordinate transformation associated with a coordinate system from a perspective of a camera taking the first image and a coordinate system from a perspective of a camera taking the second image.

20. The non-transitory data storage medium of claim 15, wherein the second coordinate system of the second image comprises a coordinate system corresponding to the second picture comprising a mirror view of the object in the second image.