A method and combined video display and camera system are disclosed. In one embodiment, the system comprises a first sheet and a second sheet oriented parallel to the first sheet, the second sheet including a light diffuser. A light source is placed along an edge of the second sheet, wherein the second sheet diffuses light generated by the light source. One or more cameras are placed behind the second sheet to capture an image through the second sheet and the first sheet.
|
1. A method for reducing defects in an image captured by a camera, the defects caused by a flat panel screen of a display, comprising:
providing a partially transparent display pixel during a time period when the camera is exposed;
estimating an average transparency of the partially transparent display pixel;
calculating values, as a function of the average transparency, for insertion in an occlusion map;
inserting the values in the occlusion map; and
estimating a corrected image using the occlusion map and the image captured by the camera,
wherein estimating the corrected image comprises numerically dividing one or more pixel values of the image captured by the camera by the values inserted in the occlusion map.
2. The method of
3. The method of
4. The method of
5. The method of
6. The method of
7. The method of
10. The method of
11. The method of
12. The method of
|
|||||||||||||||||||||||||||||||||||||||||||||||||
The present application is a continuation of patent application No. 11/549,851 entitled “CAMERA PLACED BEHIND A DISPLAY WITH A TRANSPARENT BACKLIGHT” filed on Oct. 16, 2006 at the USPTO, granted on Sep. 20, 2011 as U.S. Pat. No. 8,022,977, which in turn claimed the benefit of and priority to Indian Provisional Patent Application No. 1300/MUM/2005 entitled “A Combined Video Display and Camera System” and filed on Oct. 17, 2005.
The present invention relates to electronic devices. More particularly, the invention relates to a combined video display and camera system.
In video conferencing, two people communicate audio-visually. Each person is near a video conferencing terminal having a camera and a display. The camera captures the image of the person, which is transmitted to the distant person. The image of the distant person, is displayed on the display. Each person in the video conference is looking at his or her display to view the image of the other person. The camera is usually placed somewhere near the display. Since the person is looking at the display, the image captured by the camera is of the person looking away from the camera at the display. Each person is, thus unable to maintain eye contact. Absence of eye contact during a conversation greatly reduces the effectiveness of communication.
Many prior art systems use a two-way mirror, also called a half silvered mirror or beam splitter. A two-way mirror simultaneously reflects some light and passes some light.
Another prior art video conferencing system uses a terminal equipped with a beamsplitter for reflecting an image generated by a video display so that only the reflection and not a direct view of the display is seen by the conferee. The camera is positioned behind the viewing side of the beamsplitter to capture the conferee's image through the beamsplitter. The direct view of the display is blocked by an image blocking film applied between the beamsplitter and the display. Blocking the direct view of the video display greatly improves teleconferencing by eliminating the distraction of simultaneously viewing both the video display and the reflection of the display.
Prior art systems are quite bulky, especially when compared to modern display systems or modern teleconferencing systems. These systems waste a lot of energy, since a large amount of the energy radiated by the display is wasted since it goes through the two-way mirror.
Many prior art systems compute a three-dimensional model of the conferee. Then the model is used to render an image of the conferee as if a camera were placed just behind the screen. The three-dimensional model is computed from multiple views of the conferee captured by cameras near the display, or by illuminating the conferee using light of a particular known pattern, and using the data pertaining to the illumination caused by the light.
In another prior art system, the three-dimensional model is not computed, but the final virtual view from the direction of the display is estimated by visual flow interpolation techniques. All these methods are computationally expensive. Furthermore, they do not perfectly capture the required image, but just estimate it. Also, the closer the viewer is to the display, the larger the disparity between the images of the conferee captured by the various cameras, and the harder it is to compute an accurate three-dimensional model of the conferee. Also, such approximation models falter under improper lighting conditions and improper viewing conditions such as the presence of particulate matter or obstructions.
A prior art method for achieving eye-contact in a video conferencing situation uses a camera placed directly in the line of sight between the conferee and the display. Though a correct image of the user may be captured this way, the visual obstruction of the camera is not comfortable to the conferee.
An attachment mechanism removably secures the camera to a screen portion of a display screen such that the camera is disposed between the display screen and a person engaged in videoconferencing. The attachment mechanism can be a suction cup, strips of double-sided tape, or magnets. Magnetic force between the first and second magnets removably secures the camera to a screen portion of the flat panel display.
Other prior art systems use projection systems and are bulky in nature. Furthermore, these systems do not offer complete isolation of the camera sensor from the light due to the display, causing unwanted glare. Also, in many situations flat panel displays are preferred to projection systems due to image quality reasons.
A method and combined video display and camera system are disclosed. In one embodiment, the system comprises a first sheet and a second sheet oriented parallel to the first sheet, the second sheet including a light diffuser. A light source is placed along an edge of the second sheet, wherein the second sheet diffuses light generated by the light source. One or more cameras are placed behind the second sheet to capture an image through the second sheet and the first sheet.
The above and other preferred features, including various details of implementation and combination of elements are more particularly described with reference to the accompanying drawings and pointed out in the claims. It will be understood that the particular methods and systems described herein are shown by way of illustration only and not as limitations. As will be understood by those skilled in the art, the principles and features described herein may be employed in various and numerous embodiments without departing from the scope of the invention.
The accompanying drawings, which are included as part of the present specification, illustrate the presently preferred embodiment and together with the general description given above and the detailed description of the preferred embodiment given below serve to explain and teach the principles of the present invention.
A method and combined video display and camera system are disclosed. In one embodiment, the system comprises a first sheet and a second sheet oriented parallel to the first sheet, the second sheet including a light diffuser. A light source is placed along an edge of the second sheet, wherein the second sheet diffuses light generated by the light source. One or more cameras are placed behind the second sheet to capture an image through the second sheet and the first sheet.
According to one embodiment, the present system and method captures images with a camera from almost the same location as a display is displaying another image. The system uses optical, electrical, electronic and optionally computational elements.
One embodiment of the invention comprises a flat-panel screen which displays images picture by adjusting the optical transparency of its individual pixels. The display is illuminated using a backlight which diffuses light from a light source such that the light passes through the flat panel screen and the user sees the images. A camera captures an image of the user and his/her surroundings through the flat panel screen and backlight. An image correction system and method is employed on the captured image to reduce or eliminate the effects of displaying an image on a screen, backlight and illumination on the captured image. This corrected image is available at output 214.
Almost all computers have displays, and many have web cams or other image capture mechanisms. The present system works as both of these, in a more compact form. Many present day mobile phones have both a display and a camera. Many mobile phones have a camera directed towards the viewer of the display of the mobile phone. The present system allows mobile phones to be more compact, and enable eye-to-eye teleconferencing in a mobile phone.
In one embodiment, the system may be used for reading text while appearing to look straight at the camera. The present system allows energy conservation in such applications, saves studio space in traditional uses, and allows alternate uses such as web casting news or other programs from a personal computer, or reading text from the camera panel while shooting at an outdoor location.
The present system may be used to conceal a camera behind a display or a patch of illumination, for surveillance purposes. For example, this could be used to place a camera inside the display of an automatic teller machine.
The present system may also act as a mirror, by presenting the captured image on the display. The image may be flipped left-to-right to emulate a normal mirror, or it may not be flipped.
The backlight 206 behind screen 204 scatters light into the camera 210 and interferes with its operation. The light scattered into the camera 210 is predictable and may be corrected with image correction system 212. A camera with high sensitivity and accuracy is used because the scattered light, may be more powerful than the light that is intended to be captured in the camera 210. The scattered light (being noise) decreases the signal-to-noise ratio of the captured image.
System 299 displays images and captures images in separate periods of time, where the image display period and image capture period occurs in a rapidly alternating sequence. The frequency of alternating periods is so fast that the eye observing the screen does not perceive a flicker. The human eye cannot perceive flickers which occur at a rate faster than about 40 Hz. This principle is used in movies and cathode ray tube televisions.
The light source 208 cycles between off and on (or between light output and no light). This light source 208 may be a fluorescent light tube. Fluorescent lights have the ability to alternate quickly between outputting light and not outputting light. Fluorescent lights are used to light flat panel display backlights. These fluorescent lights cycle between the light and dark states at a rate as high as 1000 Hz. It is possible to vary the duty cycle of the light and dark states, (i.e. it is possible to electronically control what fraction of the total time the light source is in the light state, sourcing light into the backlight 206). In an alternate embodiment, the light source 208 is one or more LED light sources. The cycling between light and dark states is achieved by turning the LED light sources on and off in rapid succession. Incandescent lights are also a very good light source, but incandescent lights cannot switch between light and dark states very fast. Illumination from an incandescent light sources may be caused to alternate between light and dark by placing an obstruction between light source 208 and backlight 206. The obstruction switches at a high speed between transparent and opaque to cause the backlight 206 to glow and not glow. Such an obstruction may be a mechanical shutter, implemented as a rapidly opening and closing aperture or as a spinning disc with holes. Another system causing a periodic obstruction is an LCD panel which switches at a fast speed between opaque and transparent. Continuous light sources such as incandescent light are sometimes preferred in lighting systems because they produce a much more natural and even light spectrum. Other continuous systems such as arc lights produce a high power output. A light source and its driving electronic circuitry are arranged such that the fraction of time for which the light source is on is as small as possible. This allows for maximum time to be allotted to capturing the image with the camera 210, and thus increases the sensitivity of the camera 210. LED light sources exist which source a very high amount of light energy for a very small duty cycle.
The camera 210 continuously switches between image recording mode and a mode wherein the light falling on the input aperture of the camera 210 has no effect on the camera 210, hereinafter referred to as non-recording mode. In one embodiment, this is achieved by using an obstruction near the input aperture of the camera 210 which alternates at a high frequency between transparent and opaque. If the camera 210 is an electronic camera, including a CCD camera or other such electronic device, another way of achieving an alternating obstruction is to use an electronic shutter. When electric potential is applied to the pixels of the image recording plane of the electronic camera 210, photons captured by the camera 210 are converted to electrons which are accumulated in physical spaces corresponding to each pixel of the picture. When this electric potential is removed, light falling on the recording plane has no effect. High frequency switching between recording and non-recording mode for an electronic camera 210 may be achieved by alternately applying and removing said electric potential.
The time between application and removal of the electric potential is an exposure. After the electric potential is applied and removed, the amount of charge accumulated in each pixel position is recorded that serves as an electronic description of the image. In one embodiment, after each application and removal of the electric potential, the charge accumulated in each pixel is recorded, i.e. an image is captured per exposure. For very short exposures it may not be possible to accumulate enough photons in a single exposure to accurately estimate the intensities at each point. In such a case, a single image is recorded after many exposures. Thus, electric potential is applied and removed many times, causing the camera to switch between recording and non-recording mode many times. After a certain fixed number of exposures, the amount of charge accumulated at each pixel position is measured and recorded. Choosing the number of exposures after which an image is recorded achieves exposure control, i.e. controlling the total amount of time for which the pixels are exposed for the capturing of a single image. In an alternate embodiment, choosing the exposure time of each exposure achieves exposure control. Exposure control is a feature in cameras to achieve more control over the image recorded. Automatic exposure control is achieved by choosing the exposure time based upon the intensity of the previous image captured, or other light measurements. Cameras capture images at a fixed image rate. After the requisite number of exposures for the capture of an image occurs, the camera shutter is kept in non-recording mode until the next image is recorded. During this extended non-recording mode, the charge accumulated at each pixel position is recorded. Non-CCD cameras, non-electronic cameras and film based cameras may also be used. In the case of a film based camera, to achieve multiple exposures per image capture, the film frame is advanced after switching multiple times between recording and non-recording mode. This may require, similar to the case of the electronic camera, fast switching between the recording and non-recording mode, followed by an extended non-recording mode during which the film is advanced. Two serially placed apertures may be used—one for keeping the light of the backlight from entering the camera, and one to allow frame advance.
The switching between light and dark of the light source 208 and the switching between recording and non-recording mode of the camera 210 are synchronized such that the camera 210 is in the recording mode only when the light source 208 is dark. Both the camera and light source are controlled by a single electronic oscillator and control circuit. In the case that the alternating apertures for the light source and camera are discs with holes, both the discs are driven by a single mechanical shaft. Alternatively, a single disc with holes is provided, and positions of the light source and camera are arranged such that either the light source is on or the camera is recording.
A fluorescent or LED light source may be used together with an electronic camera and driving both of these devices using a single electronic circuit having a single oscillator controlling both these devices. According to one embodiment, a single electronic circuit having a single oscillator controls both the light source (either, fluorescent or LED) and the electronic camera.
To further reduce costs, one may substitute a light source which can alternate between light and dark, but whose dark period is not a small fraction of the total time, but a large fraction of it. In this case, the synchronization mentioned above is adjusted such that a minimum possible light from the light source 208 enters the camera 210.
In another embodiment, if the aperture of the camera 210 is very small, the illumination light entering the camera 210 may be of a tolerable limit, and its effects are nullified by the image correction system 212.
Backlight 206 of the display 300 allows light incident at a particular angle to disperse in all directions. Light incident in other angles, though, is almost completely passed through the backlight without any change.
It is beneficial to use a non-uniform concentration of dispersive particles such that a uniform illumination is achieved. In the case of a single light source, the concentration of dispersive particles is lesser near the light source, and more at the other end of the middle sheet 504 of the backlight 206. The camera is placed behind a part of the backlight 206 which has lesser concentration of dispersive particles. Using non-uniform concentration, all the light due to the light source can be dispersed by the backlight.
Image correction system 212 corrects defects in the image captured by camera 210 through the flat panel display 204 and backlight 206. The camera 210 is located very close to the flat panel display 204. A very few pixels of the flat panel display 204 affect the image captured by camera 210. Furthermore, the pixels of the flat panel display 204 are out of focus at the image capturing plane of the camera 210. The object to be captured by the camera 210, being further away, is in focus at the image capturing plane of the camera 210. The effect of the occlusion of the flat panel display 204 on any particular pixel of the image captured by camera 210 is a multiplicative change in intensity. The actual intensity at the particular pixel is multiplied by a function of the occluding pixels. Furthermore, since the occluding pixels are out of focus, more than one occluding pixel will contribute to the occlusion of each captured image pixel. The amount of occlusion suffered by each pixel of the captured image, henceforth called the occlusion map, has a specific relation to the occluding pixels on the flat panel display 204. In an embodiment, this relation is approximated as a linear relation that is evaluated by performing experiments whereby a single pixel on the screen 204 is opaque and all other pixels are transparent. The effect of a single screen pixel on the occlusion map is recorded. Knowing the occlusion map for each screen pixel, the occlusion map for any setting of values for the relevant screen pixels is evaluated. In an alternate embodiment, such linear relation is further characterized as a shift-invariant relation. Thus various fast convolution methods such as the methods based on the Fast Fourier Transform may be used to calculate the occlusion map from the values of the relevant screen pixels. Once the occlusion map is known, the occlusion suffered by each captured pixel is known, and the original value of the screen pixel is determined.
There are many flat panel displays which have statically opaque elements, such as pixel boundaries and transistors. Statically opaque elements have a static additive effect on the occlusion map. The effect is estimated experimentally, by presenting the camera 210 with a flat intensity of light, and keeping all the screen pixels transparent. The known additive effect of statically opaque elements are added into the occlusion map to generate a composite occlusion map. The map reflects changes due to static as well as dynamic elements. The effect of the occlusion may then be nullified by dividing by the occlusion map. The defects introduced into the image captured by the camera 210 by the flat panel screen 204 are thus removed.
There are two kinds of defects introduced by the backlight 206 into the image captured by the camera 210. The first of these defects is a slight dispersion of the light rays before the image is captured. In an alternative embodiment, to get a clear picture, the defect may be removed by standard deconvolution or linear system inversion. As a simple approximation, all the pixel values may be added to estimate the total dispersed light. The amount of this light, approximating the total light entering the camera 210 is further estimated to uniformly affect all pixels of the picture, and thus, a single uniform value is subtracted from each pixel. This improves the contrast of the captured picture by reducing the slight haze introduced by backlight 206.
The second defect introduced by backlight 206 is dispersion of light from light source 208 into camera 210. Image correction system 212 corrects defects by subtracting the intensity at each pixel that is introduced by the scattering of the backlight. This is estimated per picture as a function of the time overlap between the display phase and capture phase.
While the camera 210 is capturing the image, the image is occluded by the transparency setting of various screen pixels which are in front of the camera aperture. This occlusion causes darkness at the camera 210. Though the darkness can be overcome by amplification, the amplification causes an equivalent amplification in the noise including quantization noise, the stochastic nature of the photons falling on the pixel area, the stochastic nature of the photoelectric effect, thermal and electrical noise, and other similar causes of sensor noise. This decreases the accuracy and sensitivity of the captured image.
In one embodiment of the invention, some or all of the screen pixels in the path of light to the camera 210 are made transparent during the time period when the light source 208 is in ‘off’ mode, i.e when the display is not lighted. The original transparency values of the pixels are restored before the light source 208 starts sourcing light again. If a new image is to be displayed on the screen 204, the pixels are now set to the new transparency values. Thus, in the time periods when an image is being displayed on the screen, the screen pixels are set to transparency values that result in the correct image being displayed, and in the time periods when an image is being captured by the camera, the screen pixels are set to a transparency value that cause more light to fall on the image plane of the camera 210.
If screen 204 is a liquid crystal display (LCD), the changes in transparency are made by changing the charges stored at those pixel values. LCD pixels are slow to respond to changes in electric potential. Hence, the timing of these changes is accurately adjusted such that the maximum average transparency during the capture phase is possible. The frequency of switching between display and capture periods is kept slow enough to allow for the LCD pixels to switch to transparent and switch back to their original setting. Also, setting the duty cycle of the light source 208 such that a small fraction of the time is spent in the display period leaves more time for switching. Even if complete transparency is not achieved, the average transparency value achieved for a particular pixel during the capture period increases, thus causing an increase in the sensitivity of the camera. If complete transparency is achieved, the image correction system 212 has to compensate for static occlusion, which is a very simple algorithm. If complete transparency is not achieved, there is some effect of the screen pixel values on the occlusion map. This effect not only depends upon the screen pixel values themselves but also upon the time of exposure of the camera 210, and the average transparency of the particular pixel during the exposure. This is calculated by the image correction system 212 and the occlusion map is calculated from these changed pixel values.
There are many methods of making a select set of pixels transparent on a flat-panel display, especially if the display is an LCD. For example, in matrix addressing, pixel values for a whole row of pixels are set at a time. An individual row is selected by a row enable wire that runs across the display. Multiple rows are made transparent in one operation by selecting multiple row enable wires. In systems using both row and column enables, a square area is selected to be made transparent in one operation by choosing the corresponding rows and columns to be enabled. Many LCD systems use a storage capacitor per pixel to store the charge which applies a given voltage across the liquid crystal pixel. Extra switching elements such as transistors may be used in the pixel to couple and decouple the liquid crystal from the charge stored in the capacitor. When the liquid crystal is decoupled from the charge, the pixel becomes transparent. Furthermore, coupling the liquid crystal to the capacitor causes the original transparency value to be restored. Usually, the voltage on one terminal of the storage capacitor or one end of the liquid crystal is varied, while the other end is kept constant. The voltage at the other end of the liquid crystal or storage capacitor is varied for the specified pixels or all pixels to turn the corresponding pixels transparent in a single operation without losing the voltages stored on the first end. When the voltage at the other end of the liquid crystal or storage capacitor is brought back to its original value, the original transparency values are restored.
In another embodiment of the invention, whenever the display and camera are used together, as during a teleconference, the brightness of a particular pixel on the display is not allowed to drop below a particular transparency level. This is achieved with the driving circuit of the flat panel display, or in software by the computer attached to the display. This guarantees a minimum sensitivity for the captured picture.
In yet another embodiment, the exposure time for obtaining a single image is varied to compensate for the loss of light due to occlusion. Such alteration of exposure time is usually controlled by the user, or by an automatic algorithm which adapts to the brightness level of the captured image. In one embodiment, the expected exposure time is multiplied by a factor derived from the advance knowledge of the average occlusion value that the captured image is going to suffer. The image correction system 212 corrects not only the defects due to the occlusion, but also due to the altered exposure time, the latter achieved by dividing by the same factor by which the actual exposure time is more than the expected exposure time, the division causing a reduction in the noise present in the image.
If the mirror 602 is a complete mirror, light will not pass through it into the camera 210. One way to overcome this problem is to use a partially silvered mirror. Though the energy efficiency of the backlight is improved, the image received by the camera 210 is attenuated, thus adversely affecting the sensitivity of the camera 210.
The mirror 602 may produce highly specular reflections of objects on the other side of the display, which may make the display uncomfortable to view. This problem is overcome by making the mirror 602 a diffusing mirror, such that it has a glazed or brushed finish. In one embodiment, this is achieved by abrading the surface of transparent sheet 506 and depositing a metal on its surface. Alternatively, a light scattering sheet is used between mirror 602 and transparent sheet 506.
It is frequently the case that the objects whose image is to be captured by the camera 210 are themselves illuminated by a regularly alternating external light source such as fluorescent light. Further, there is also the possibility of other displays within the field of view, these displays having their own frequencies of light emanation (both CRT and LCD displays exhibit this behavior). Such flickering light may cause a frequency aliasing effect when sampled through the high frequency capture periods of the camera 210. The effect of this aliasing on the captured images would be a low frequency oscillation in the illumination due to that particular light source.
In one embodiment of the present invention, switching between capture and display periods is synchronized with the illumination due to the alternating light source, preferably such that the present system is in the capture period at the same time that the alternating external light source is illuminating the object whose image is to be captured. This eliminates the aliasing effect, and has the added benefit of a large amount of the illumination reaching the camera 210. Synchronization may be achieved by synching with the alternating mains power, which is usually driving the switching of the external light source, or by using a phase locked loop to lock into the frequency of the light source. The phase locked loop may use a photo-sensor or feedback from the camera 210 to achieve phase lock. In another embodiment, the switching frequency is a multiple of the frequency of switching of the external light source, or vice versa.
In another embodiment, the aliasing effect is minimized by alternating between the capture and display periods at a very fast frequency compared to the frequency of switching the external light source off and on. Because of the higher sampling rate, the effect of aliasing is reduced. The residual aliasing effect causes a low frequency periodic change in the illumination of the captured image. Such periodic change is detected and cancelled out by the image correction system 212. The period of the frequency change is predicted by detecting the mains power frequency and subtracting the whole fraction of the switching frequency between the capture and display periods that are closest to it.
To minimize flicker from high frequency light sources, the sequence of alternating between capture and display periods is randomized. The amount of time the system remains in each of the two phases of displaying an image and capturing an image is decided randomly based on some random or pseudorandom sequence of numbers. In many cases, the amount of time the light source 208 can remain on is limited. In such cases the randomization relates to the period between two flashes of light from light source 208. The random sequence is arranged such that the average illumination of the display is the chosen average illumination for the display, such that no flicker is perceived by the human eye considering the principle of persistence of vision, and each image being captured is exposed for the requisite time. The randomization in the sequence breaks up the symmetry of sampling and greatly reduces the effects of aliasing.
The camera 210 is placed in a position closest to the expected location of the image of the remote conferee's eyes. Since the eyes of a human are located in the upper half of the head, and since usually head-and-shoulder images are most commonly transferred while video conferencing, the camera 210 is placed behind the upper part of the screen 204.
In another embodiment, multiple cameras may be placed behind the screen. These may generate a stereoscopic or multiple-view image of the subject and surroundings. Alternately, views from various cameras could be transmitted to various conferees in a multi-party video conference. The image captured by the camera closest to the image of a particular remote conferee will be transferred to that particular conferee. This will simulate eye contact for the remote conferee only when the present conferee actually looks at that particular remote conferee.
Since the camera 210 is an integral part of the video display unit, turning the camera in a different direction than the display is difficult. It may also be difficult to keep the display connected and yet disconnect the camera. To minimize the possibility of turning the camera on and capturing images unbeknownst to the user of the present invention, a simple electronic indication such as an LED indicator is provided on the front panel of the display whenever the camera starts capturing images.
Many image display mechanisms and image capture mechanisms have a non-linear response. This causes the relation between occluding screen pixels and the occlusion mask to be non-linear, as well as the occlusion mask itself to be more complicated than a simple multiplier per pixel of the captured image. Though these relations are linear as far as the transparency values of the screen to the intensities at the camera pixels are concerned, the non-linear complications arise because the transparency of a screen pixel is not a linear function of the input pixel value, and because the pixel value recorded by the camera is not a linear function of the intensity incident at that pixel. To reduce or remove the non-linear effects, the image correction system 212 first operates on the captured image with the inverse of the non-linear function for the camera, to get a record pixel values which are linearly related to the intensities of light incident at those pixels. Also, before the occlusion map is calculated, the relevant pixels of the image to be displayed upon the screen are operated upon (computationally) by the same non-linear function as the non-linear function of the screen, to get an estimate of the transparency of each occluding pixel.
The timing of the system is arranged such that the occluding screen pixels are not changing their transparency values during a particular image capture. This is achieved by synchronizing the display refresh cycle and the camera frame capture cycle. In the case that the transparency values of the occluding pixels change during the capture of a single image, the occlusion map calculated by the image correction system 212 is calculated as some function (a linear-combination), of the occlusion maps due to the first and the second setting of pixels on the screen.
To display color images, color filters are used on individual pixels of the screen 204. To capture color images, color filters are applied to individual pixels of the camera 210. When viewed through a camera pixel of a particular color, pixels of different color than the color of the camera pixel seem much darker than pixels of the same color. This effect on perceived transparency of same and different colors is used in the calculation of the occlusion map by image correction system 212.
A method and combined video display and camera system are disclosed. It is understood that the embodiments described herein are for the purpose of elucidation and should not be considered limiting the subject matter of the present patent. Various modifications, uses, substitutions, recombinations, improvements, methods of productions without departing from the scope or spirit of the present invention would be evident to a person skilled in the art.
Kanade, Udayan, Ganapathy, Balaji
| Patent | Priority | Assignee | Title |
| 10331164, | Jan 26 2017 | GUANGDONG OPPO MOBILE TELECOMMUNICATIONS CORP., LTD. | Display screen assembly, electronic device and image acquisition method |
| 10911656, | Nov 21 2017 | Microsoft Technology Licensing, LLC | Optical isolation systems for displays |
| 11272106, | Aug 31 2020 | Samsung Electronics Co., Ltd. | Method and apparatus for image acquisition including image sensor |
| 11792515, | Aug 31 2020 | Samsung Electronics Co., Ltd. | Method and apparatus for image acquisition including image sensor |
| 11864452, | Aug 24 2021 | Apple Inc. | Black masking layer in displays having transparent openings |
| 9392219, | Jul 31 2013 | Howlett-Packard Development Company, L.P.; HEWLETT-PACKARD DEVELOPMENT COMPANY, L P | Display-camera system |
| Patent | Priority | Assignee | Title |
| 4400725, | Jan 25 1980 | Picture display-image pickup apparatus | |
| 5025314, | Jul 30 1990 | XEROX CORPORATION, A CORP OF NY | Apparatus allowing remote interactive use of a plurality of writing surfaces |
| 5159445, | Dec 31 1990 | AT&T Bell Laboratories; AMERICAN TELEPHONE AND TELEGRAPH COMPANY, A CORP OF NY | Teleconferencing video display system for improving eye contact |
| 5187611, | Aug 27 1991 | Robotic Vision Systems, Inc | Diffuse on-axis light source |
| 5243413, | Sep 02 1992 | AT&T Bell Laboratories | Color parallax-free camera and display |
| 5539485, | Oct 31 1994 | MICROSCAN SYSTEMS, INC | Illumination device for providing continuous diffuse light on and off an observing axis |
| 5650844, | Jul 14 1994 | Advantest Corporation | LCD panel image quality inspection system and LCD image presampling method |
| 5850256, | Dec 15 1995 | Portable test pattern illuminator | |
| 5856842, | Aug 26 1997 | Kaiser Optical Systems Corporation | Apparatus facilitating eye-contact video communications |
| 6219443, | Aug 11 1998 | AVAGO TECHNOLOGIES GENERAL IP SINGAPORE PTE LTD ; AVAGO TECHNOLOGIES GENERAL IP PTE LTD | Method and apparatus for inspecting a display using a relatively low-resolution camera |
| 6269565, | Nov 28 1994 | Smartlight Ltd. | Display device |
| 6450662, | Sep 14 2000 | LIGHT VISION SYSTEMS, INC | Solid state traffic light apparatus having homogenous light source |
| 6831995, | Mar 23 1999 | Hitachi, LTD | Method for detecting a defect in a pixel of an electrical display unit and a method for manufacturing an electrical display unit |
| 6919892, | Aug 14 2002 | AVAWORKS, INCORPROATED | Photo realistic talking head creation system and method |
| 7027054, | Aug 14 2002 | AvaWorks, Incorporated | Do-it-yourself photo realistic talking head creation system and method |
| 7084937, | Nov 29 2002 | Hon Hai Precision Ind. Co., Ltd. | Liquid crystal display and backlight module thereof with reflective polarizer and quarter wave plate |
| 7196728, | Mar 27 2002 | Ericsson AB | Method and apparatus for displaying images in combination with taking images |
| 7257272, | Apr 16 2004 | Microsoft Technology Licensing, LLC | Virtual image generation |
| 7697053, | Nov 02 2006 | Eastman Kodak Company | Integrated display having multiple capture devices |
| 7714923, | Nov 02 2006 | Eastman Kodak Company | Integrated display and capture apparatus |
| 7808540, | Jan 09 2007 | Monument Peak Ventures, LLC | Image capture and integrated display apparatus |
| 20020157033, | |||
| 20030071932, | |||
| 20030215129, | |||
| 20040165060, | |||
| 20040213449, | |||
| 20040257473, | |||
| 20050046768, | |||
| 20050052402, | |||
| 20050078866, | |||
| 20050167620, | |||
| 20050232476, | |||
| 20050232509, | |||
| 20050232510, | |||
| 20060049768, | |||
| 20080106591, |
| Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
| Aug 12 2011 | I2iC Corporation | (assignment on the face of the patent) | / |
| Date | Maintenance Fee Events |
| Aug 12 2016 | REM: Maintenance Fee Reminder Mailed. |
| Jan 01 2017 | EXP: Patent Expired for Failure to Pay Maintenance Fees. |
| Date | Maintenance Schedule |
| Jan 01 2016 | 4 years fee payment window open |
| Jul 01 2016 | 6 months grace period start (w surcharge) |
| Jan 01 2017 | patent expiry (for year 4) |
| Jan 01 2019 | 2 years to revive unintentionally abandoned end. (for year 4) |
| Jan 01 2020 | 8 years fee payment window open |
| Jul 01 2020 | 6 months grace period start (w surcharge) |
| Jan 01 2021 | patent expiry (for year 8) |
| Jan 01 2023 | 2 years to revive unintentionally abandoned end. (for year 8) |
| Jan 01 2024 | 12 years fee payment window open |
| Jul 01 2024 | 6 months grace period start (w surcharge) |
| Jan 01 2025 | patent expiry (for year 12) |
| Jan 01 2027 | 2 years to revive unintentionally abandoned end. (for year 12) |