Near-eye parallax barrier displays

Abstract

In embodiments of the invention, an apparatus may include a display comprising a plurality of pixels and a computer system coupled with the display and operable to instruct the display to display images. The apparatus may further include an slm array located adjacent to the display and comprising a plurality of slms, wherein the slm array is operable to produce a light field by altering light emitted by the display to simulate an object that is in focus to an observer while the display and the slm array are located within a near-eye range of the observer.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority from U.S. Provisional Application No. 61/667,362, filed Jul. 2, 2012, the entire disclosure of which is incorporated herein by reference. This application claims priority from U.S. Provisional Application No. 61/668,953, filed Jul. 6, 2012, the entire disclosure of which is incorporated herein by reference. The following copending U.S. Patent Application are incorporated herein by reference for all purposes: U.S. Patent Application Ser. No. 13/720,809, “NEAR-EYE MICROLENS ARRAY DISPLAYS,”David Luebke, filed Dec.

FIG. 12B depicts images before and after deconvolution, according to embodiments of the present invention. FIG. 12B includes the same dot 1204 as in FIG. 12A. In order to cancel, reverse, or counter the blurring effect caused by the eye, a deconvolved or pre-filtered image may be produced. For example, a deconvolved dot 1212 of the dot 1204 may be produced by performing a deconvolution operation on the dot 1204. The result of the deconvolution operation, e.g. the deconvolved dot 1212, may be depicted by two concentric rings. The two concentric rings may have differing intensities.

More specifically, if the dot 1204 described by the function i(x, y) is convoluted with the inverse of the second function If h⁻¹(x, y), the resulting function describing the deconvolved dot 1212 may be ĩ(x, y). The inverse of the second function may be, for example, the inverse of the PSF.

Accordingly, the opposite or inverse of the natural blurring effect caused by the eye may be described by a 35 deconvolution operation. The following mathematical equation may describe the relationship between the dot 1204 and the deconvolved dot 1212:
i(x, y)*h⁻¹(x, y)=ĩ(x, y)

The deconvolution operation may reduce in negative values, which may not be synthesized by the display or values outside the dynamic range of the display. The deconvolved image ĩ(x, y) may be filtered to transform the deconvolution output to be within the dynamic range of the display device.

FIG. 12C depicts a deconvolved image before and after convolution, according to embodiments of the present invention. When a convolution operation is performed on a function describing a deconvolved image, the resulting function may describe the original image. For example, when the deconvolved dot 1212 described by ĩ(x, y) under-goes a convolution operation with the second function h(x, y), the result may be the function i(x, y) describing the original dot 1204. The second function may be, for example, the PSF.

The following mathematical equation may describe the relationship between the deconvolved dot 1212 and the dot 1204:
ĩ(x, y)*h(x, y)=i(x, y)

Accordingly, an eye may perceive an image completely or at least approximately similar to the original dot 1204 in focus when viewing a deconvolved version 1212 of the dot in a near-eye range (nearer to the eye than the near plane of the eye) because the eye's convolution effect may translate the deconvolved version of the dot completely or at least approximately similar to the original dot 1204. This approximation may have reduced contrast or other artifacts, but may still improve the legibility or recognizability of the image, as compared to a conventional display without pre-filtering or deconvolution applied.

It should be appreciated that the function i(x, y) may describe multiple points or pixels on a surface that together form an image. Accordingly, the deconvolved function ĩ(x, y) may correspond to multiple points or pixels that together form a deconvolved version of the image. As a result, when the deconvolved version of the image described by the deconvolved function ĩ(x, y) is viewed in near-eye ranges, the original image described by the function i(x, y) may be perceived by an observer.

Returning to FIG. 11, a deconvolved image may be displayed by the display 1124. Since the display 1124 is within the near-eye range, the observer may perceive a convoluted version of the deconvolved image. As discussed above, a convolution of an image deconvolved by the inverse of the convolution function will result in substantially the original image. Accordingly, the observer will perceive an in focus image since the blurring effect of the eye will have been countered by the display of the deconvolved image. Therefore, an image may be recognizable by an observer in near-eye ranges.

It should be appreciated that embodiments of the present invention allow for pre-filtering processes other than deconvolution. For example, other operations besides deconvolution may be used to create a pre-filtered image that when viewed at near-eye distances, provides a recognizable image to an observer after undergoing the eye's convolution effect.

It should be appreciated that multiple displays may be used. It should further be appreciated that the displays 1124 and 1125 may be semi-transparent. As a result, the eye 204 may be able to view images displayed by the display 1124 through the display 1125. The eye 204 may also be able to view the surrounding environment through both the displays 1124 and 1125. Multiple layers of displays may also decrease or eliminate artifact ringing and improve contrast.

It should also be appreciated that optical deconvolution displays may block the light from the surrounding environment to provide VR applications. For example, a display may block a portion of an observer's view while providing a deconvolved image in another portion. Or, for example, a first display in a multilayer deconvolution display may block light while a second display provides a deconvolved image.

Alternatively, such displays may generally allow the light from the surrounding environment and block only portions of the incoming light and/or augment portions with light produced by the display to provide AR applications.

It should also be appreciated that the displays 1124 and 1125 may display an image that is recognizable or in focus only when viewed while located closer than the near plane 216. For example, the image may seem blurry or out of focus when viewed in the accommodation range. The displays 1124 and 1125 may display a pre-filtered image, corresponding to a target image to be ultimately projected, that is unrecognizable when viewed within the accommodation range. When the pre-filtered image is viewed within the accommodation range, the target image may be recognizable. A computer system or graphics processing system may generate the pre-filtered image corresponding to the target image.

Additional Embodiments

It should be appreciated that embodiments of the invention provide for combining layers of near-eye light field displays, near-eye parallax barrier displays, and/or near-eye optical deconvolution displays. Light field displays and optical deconvolution displays may present different performance trade-offs. Light field displays may require high-resolution underlying displays to achieve sharp imagery, but otherwise preserve image contrast. In contrast, optical deconvolution displays may preserve image resolution, but reduce contrast.

The light field displays and optical deconvolution displays may be combined in order to benefit from the performance of each display and to support a continuous trade-off between resolution and contrast. For example, embodiments of the invention support performing optical deconvolution in the light field domain, rather than applied independently to each display layer.

Near-eye light field displays, near-eye parallax barrier displays, and/or near-eye optical deconvolution displays may be combined because such displays may implement semi-transparent displays. For example, such displays may implement a combination of light-attenuating (e.g., LCD) or light-emitting (e.g., OLED) displays.

It should be appreciated that embodiments of the invention allow for the use of multiple displays tiled together to form one effective display. For example, the display 324, display 624, display 824, or display 1124 and 1125 may comprise multiple sub-displays. Sub-displays may be tiled, e.g. side by side, to synthesize a form display. Unlike multiple monitor workstations, any gaps between displays may not introduce artifacts because the pre-filtered images may be modified to display on each tile to accommodate for the gaps between them.

Embodiments of the invention provide for both virtual reality (VR) and augmented reality (AR) applications. For example, near-eye light field displays, near-eye parallax barrier displays, and/or near-eye optical deconvolution displays may block the light from the surrounding environment to provide VR applications. Alternatively, such displays may generally allow the light from the surrounding environment and block only portions of the incoming light and/or augment portions with light produced by the display to provide AR applications.

In various embodiments, light from the surrounding environment may function as a backlight, with the display layers attenuating the incident light field. In some embodiments, at least one display layer may contain light-emitting elements (e.g., an OLED panel). In embodiments of the invention, a combination of light-attenuating and light-emitting layers can be employed. It should be appreciated that more than one layer may emit light. For example, in FIG. 9, in addition to display 824, SLM arrays 830, 832, and 834 may also emit light.

In one or more embodiments, each display layer may include either a light-attenuating display or a light-emitting display, or a combination of both (each pixel may attenuate and/or emit rays of light). Further embodiments may include multi-layer devices, for example, OLED and LCD, LCD and LCD, or and so on.

For near-eye light field displays for VR applications, a 2D display may be covered with either a parallax barrier or microlens array to support comfortable accommodation. Furthermore, multiple light-attenuating layers may be used to increase brightness, resolution, and depth of field.

Further embodiments of the invention may include holographic display elements. For example, as the resolution increases, the pitch may become small enough such that diffractive effects may be accounted for. Image formation models and optimization methods may be employed to account for diffraction, encompassing the use of computer-generated holograms for near-eye displays in a manner akin to light field displays. Embodiments of the present invention provide for applying optical deconvolution to holographic systems, thereby eliminating the contrast loss observed with incoherent displays.

Embodiments of the present invention provide for light-weight “sunglasses-like” form factors with a wide field of view using near-eye displays as discussed above. Such displays can be practically constructed at high volumes and at low cost. Such displays may have a viable commercial potential as information displays, for example, depicting basic status messages, the time of day, and augmenting the directly perceived physical world.

Embodiments of the present invention provide for adjusting produced images to account for aberrations or defects of an observer's eyes. The aberrations may include, for example, myopia, hyperopia, astigmatism, and/or presbyopia. For example, a near-eye light field display, near-eye parallax display, or near-eye optical deconvolution display may produce images to counteract the effects of the observer's aberrations based on the observer's optical prescription. As a result, an observer may be able to view images in focus without corrective eyewear like eyeglasses or contact lenses. It should be appreciated that embodiments of the invention may also automatically calibrate the vision correction adjustments with the use of a feedback system that may determine the defects of an eye.

Embodiments of the invention may also adjust the provided image based on information from an eye-track adjustment system that may determine the direction of gaze and/or the distance of the eye from the display(s). Accordingly, the display(s) may adjust the image displayed to optimize the recognizability of the image for different directions of gaze, distances of the eye from the display, and/or aberrations of the eye.

Embodiments of the invention may also adjust the provided image based on information from one or more sensors. For example, embodiments may include an environmental motion-tracking component that may include a camera. The environmental motion-tracking component may track movement or changes in the surrounding environment (e.g., movement of objects or changes in lighting). In a further example, the movement of a user's body may be tracked and related information may be provided. As a result, embodiments of the invention may adjust the provided image based on the environment of a user, motions of a user, or movement of a user.

In another example, embodiments of the invention may include an internal motion-tracking component that may include a gyroscopic sensor, accelerometer sensor, an electronic compass sensor, or the like. The internal motion-tracking component may track movement of the user and provide information associated with the tracked movement. As a result, embodiments of the invention may adjust the provided image based on the motion. In other examples, sensors may determine and provide the location of a user (e.g., GPS), a head position or orientation of a user, the velocity and acceleration of the viewer's head position and orientation, environmental humidity, environmental temperature, altitude, and so on.

Information related to the sensor determinations may be expressed in either a relative or absolute frame of reference. For example, GPS may have an absolute frame of reference to the Earth's longitude and latitude. Alternatively, inertial sensors may have a relative frame of reference while measuring velocity and acceleration relative to an initial state (e.g., the phone is currently moving a 2 mm per second vs. the phone is at a given latitude/longitude).

Near-eye light field displays, near-eye parallax barrier displays, and/or near-eye optical deconvolution displays may be included in eyeglasses. For example, such displays may replace conventional lenses in a pair of eyeglasses.

FIG. 13 depicts a flowchart 1300 of an exemplary process of displaying a near-eye image, according to an embodiment of the present invention. In a block 1302, a pre-filtered image to be displayed is determined, wherein the pre-filtered image corresponds to a target image. For example, a computer system may determine a pre-filtered image that may be blurry when viewed by itself in an accommodation range but in focus when viewed through a filter or light field generating element.

In a block 1304, the pre-filtered image is displayed on a display. For example, in FIGS. 3B, 6, and 8, a pre-filtered image is displayed on the display 324, 624, and 826, respectively.

In a block 1306, a near-eye light field is produced after the pre-filtered image travels through a light field generating element adjacent to the display, wherein the near-eye light field is operable to simulate a light field corresponding to the target image. For example, in FIG. 3A, a light field corresponding to a target image is produced after the pre-filtered image passes through the microlens array 328. Similarly, in FIGS. 6 and 8, a light field corresponding to a target image is produced after the pre-filtered image passes through the SLM array 626 and multiple SLM arrays 826, respectively.

FIG. 14 depicts a flowchart 1400 of an exemplary process of displaying a near-eye image, according to an embodiment of the present invention. In a block 1402, a target image is received. For example, a computer system may receive a target image from a graphics processing system

In a block 1404, a deconvolved image corresponding to a target image is determined, wherein when the deconvolved image is displayed within a near-eye range of an observer, the target image may be perceived in focus by the observer. For example, in FIG. 12B, a deconvolved version of a target image is determined As in FIG. 12C, when the deconvolved version of the target image undergoes a convolution operation of the eye, the target image is perceived in focus by an observer.

In a block 1406, the deconvolved image is displayed on a display. For example, in FIG. 11, a deconvolved image may be displayed on a display 1124 or 1125.

It should be appreciated that while embodiments of the present invention have been discussed and illustrated with various displays located within the near-plane but a distance from the eye, for example in FIGS. 3B, 6, 8, 11, embodiments of the present invention also provide for displays adjacent to the eye. For example, one or more layers of displays may be operable to adjoin an eye, similar to a contact lens. Because such displays may have a semi-spherical shape, the displays may account for affects of the shape to provide a sharp and recognizable image to the eye.

While the foregoing disclosure sets forth various embodiments using specific block diagrams, flowcharts, and examples, each block diagram component, flowchart step, operation, and/or component described and/or illustrated herein may be implemented, individually and/or collectively, using a wide range of hardware, software, or firmware (or any combination thereof) configurations. In addition, any disclosure of components contained within other components should be considered as examples because many other architectures can be implemented to achieve the same functionality.

The process parameters and sequence of steps described and/or illustrated herein are given by way of example only. For example, while the steps illustrated and/or described herein may be shown or discussed in a particular order, these steps do not necessarily need to be performed in the order illustrated or discussed. The various example methods described and/or illustrated herein may also omit one or more of the steps described or illustrated herein or include additional steps in addition to those disclosed.

While various embodiments have been described and/or illustrated herein in the context of fully functional computing systems, one or more of these example embodiments may be distributed as a program product in a variety of forms, regardless of the particular type of computer-readable media used to actually carry out the distribution. The embodiments disclosed herein may also be implemented using software modules that perform certain tasks. These software modules may include script, batch, or other executable files that may be stored on a computer-readable storage medium or in a computing system. These software modules may configure a computing system to perform one or more of the example embodiments disclosed herein. One or more of the software modules disclosed herein may be implemented in a cloud computing environment. Cloud computing environments may provide various services and applications via the Internet. These cloud-based services (e.g., software as a service, platform as a service, infrastructure as a service, etc.) may be accessible through a Web browser or other remote interface. Various functions described herein may be provided through a remote desktop environment or any other cloud-based computing environment.

The foregoing description, for purpose of explanation, has been described with reference to specific embodiments. However, the illustrative discussions above are not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings. The embodiments were chosen and described in order to best explain the principles of the invention and its practical applications, to thereby enable others skilled in the art to best utilize the invention and various embodiments with various modifications as may be suited to the particular use contemplated.

Embodiments according to the invention are thus described. While the present disclosure has been described in particular embodiments, it should be appreciated that the invention should not be construed as limited by such embodiments, but rather construed according to the below claims.

Claims

1. An apparatus comprising:

a display comprising a plurality of pixels;

a computer system coupled with said display and operable to instruct said display to display images including a pre-filtered image associated with a target image, wherein said computer system is operable to determine said pre-filtered image that counteracts aberrations of an observer's eye, said aberrations including at least one of myopia, hyperopia, astigmatism, or presbyopia; and

a slm array located plurality of spatial light modulator (slm) arrays arranged in a layered configuration adjacent to said display and each comprising a plurality of slms,

wherein when said pre-filtered image passes through said plurality of slm arrays, each of said slm array arrays is operable to produce a light field by altering an intensity of received light emitted by said display to simulate at varying degrees, wherein an output of said plurality of slms simulates an object of said pre-filtered image that is in focus to an and within an accommodation range of said observer while said display and said plurality of slm array arrays are located within a near-eye range of said observer,

wherein each of said slm array arrays modulates said intensity of said received light emitted by said display without changing direction of said received light emitted by said display,

wherein each of said slm arrays refreshes at a rate that is faster than a human flicker fusion threshold to increase at least one of a resolution or a brightness of said target image observed by said observer.

2. The apparatus of claim 1, wherein each slm of said plurality of slms is operable to modulate said received light produced by said display by partially attenuating said received light emitted by said display.

3. The apparatus of claim 2, wherein said display is semi-transparent, and further wherein each slm of said plurality of slms is operable to modulate such that said received light includes in part light passing through said display.

4. The apparatus of claim 1, wherein said plurality of slm array arrays is operable to project anisotropic light by altering isotropic light produced by said display.

5. The apparatus of claim 1, wherein said light field is operable to simulate a 3D object located beyond said near-eye range of said observer.

6. The apparatus of claim 1, wherein said computer system is operable to determine an image for display that counteracts aberrations of said observer's eye.

7. The apparatus of claim 1, further comprising a feedback system operable to make measurements of said aberrations of said observer's eye; and

wherein said computer system is further operable to determine an said pre-filtered image for display that counteracts said aberrations based on said measurements.

8. The apparatus of claim 1, further comprising a sensor operable to provide information related to a surrounding environment; and

wherein said computer system is further operable to determine an said pre-filtered image for display that counteracts said aberrations based on said information.

9. The apparatus of claim 1, further comprising an eye-track adjustment system operable to track a gaze of an eye, wherein said eye-track adjustment system is operable to communicate information related to a gaze of an eye to said computer system for determination of an said pre-filtered image for display by said computer system based on said information.

10. The apparatus of claim 1, wherein said display comprises a plurality of sub-displays disposed side by side to one another.

11. An apparatus comprising:

a display operable to produce an a pre-filtered image associated with a target image based upon receipt of corresponding image related signals; and

a computer system coupled with said display and operable to instruct said display to display said pre-filtered image, wherein said computer system is operable to determine said pre-filtered image that counteracts aberrations of an observer's eye, said aberrations including at least one of myopia, hyperopia, astigmatism, or presbyopia;

a first slm array located plurality of spatial light modulator (slm) arrays arranged in a layered configuration adjacent to said display,

wherein when said pre-filtered image passes through said plurality of slm arrays, each of said first slm array arrays together with said display is operable to produce a light field simulating a 3D object that is recognizable to an said observer while said display and said first slm array are located within a near-eye range of said observer by altering an intensity of received light at varying degrees,

wherein each of said slm array arrays modulates said intensity of said received light emitted by said display by partially attenuating said intensity of said received light emitted by said display and without changing direction of said received light emitted by said display,

wherein each of said slm arrays refreshes at a rate that is faster than a human flicker fusion threshold to increase at least one of a resolution or a brightness of said target image observed by said observer.

12. The apparatus of claim 11, further comprising at least one additional slm array located adjacent to said first slm array.

13. The apparatus of claim 11, wherein said first slm array is operable to display different versions, of an image produced by said display, in different directions.

14. The apparatus of claim 11, wherein said pre-filtered image is out of focus if viewed without said first plurality of slm array arrays and said pre-filtered image is in focus if viewed through said first plurality of slm array arrays.

15. The apparatus of claim 11, wherein said display is semi-transparent.

16. The apparatus of claim 11, wherein said first plurality of slm array and said display are arrays is operable to alter light from a surrounding environment and light from said display to provide an augmented reality experience.

17. The apparatus of claim 11, wherein said first plurality of slm array arrays and said display are operable to provide a virtual reality experience.

18. A method comprising:

determining a pre-filtered image to be displayed, wherein said pre-filtered image corresponds to a target image;

using a computer system to instruct a display to display said pre-filtered image, wherein said computer system is operable to determine said pre-filtered image that counteracts aberrations of an observer's eye, said aberrations including at least one of myopia, hyperopia, astigmatism, or presbyopia;

displaying said pre-filtered image on a said display in accordance with received image presentation information; and

producing a near-eye light field after said pre-filtered image travels through a slm array plurality of spatial light modulator (slm) arrays arranged in a layered configuration adjacent to said display,

wherein said near-eye light field is operable to simulate a light field corresponding to said target image,

wherein each of said slm array arrays modulates an intensity of received light emitted by said display at varying degrees by partially attenuating said intensity of said received light emitted by said display without changing direction of said received light emitted by said display,

wherein each of said slm arrays refreshes at a rate that is faster than a human flicker fusion threshold to increase at least one of a resolution or a brightness of said target image.

19. The method of claim 18, wherein said target image simulated by said near-eye light field is in focus to an said observer's eye when said display and said plurality of slm array arrays are in a near-eye range of said observer's eye.

20. The method of claim 18, wherein said plurality of slm array arrays is operable to project anisotropic light by altering isotropic light produced by said display.

21. The method of claim 18, wherein said determining is based on said aberrations of said observer's eye, a gaze of said observer's eye, and a distance between said observer's eye and said plurality of slm array arrays.

22. The method of claim 18, wherein said slm array is operable to display different versions of said target image in different directions.

23. The method of claim 18, wherein said slm array comprises a plurality of slm array layers.

24. The apparatus of claim 1, wherein a plurality of light rays emitted by said pixels in a plurality of different directions are passed through said plurality of slm arrays without changing direction of said plurality of light rays.

25. The apparatus of claim 24, wherein said plurality of slm arrays refines, layer by layer, which light rays emitted by said pixels of said display are passed through for observation by said observer, wherein a first light ray passes through a first slm array in the layered configuration but is blocked or attenuated by a subsequent slm array in the layered configuration.

26. The apparatus of claim 1, wherein a near-eye light field output by said plurality of slm arrays corresponds to said target image.