An exemplary imaging device includes an optical sensor having an optical axis, a lens positioned to focus light from a scene onto the optical sensor, a matrix of variable-phase optical elements that are dimensioned to introduce at least two different phase delays into a wavefront of a light signal received from the scene through the lens, a housing that secures the matrix of variable-phase optical elements between the optical sensor and the lens, and a processing subsystem programmed to determine a phase difference associated with the light signal based on the at least two different phase delays. Related systems and methods are also provided.
|
12. A depth imaging system comprising:
an imaging device comprising:
an optical sensor having an optical axis and an array of photosensitive pixels;
a lens positioned to focus light from a scene onto the optical sensor;
a diffractive optical element (doe) having features that are dimensioned to introduce at least two different phase delays into a wavefront of a light signal received from the scene through the lens to at least partially optically demodulate the light signal, a width of at least one of the features of the doe being substantially the same as a width of a first pixel of the array of photosensitive pixels;
a housing that secures the doe between the optical sensor and the lens; and
a processing subsystem programmed to determine a phase difference associated with the light signal based on at least partially optically demodulated light received by the optical sensor from the doe.
18. An imaging device comprising:
an optical sensor having an optical axis and an array of photosensitive pixels;
a lens positioned to focus light from a scene onto the optical sensor;
a diffractive optical element (doe) having features that are dimensioned to introduce at least two different phase delays into a wavefront of a light signal received from the scene through the lens to at least partially optically demodulate the light signal;
a housing that secures the doe between the optical sensor and the lens; and
a processing subsystem programmed to:
perform a calibration of the delays of the at least two different phase delays; and
determine a phase difference associated with the light signal based on the calibration of the delays of the at least two different phase delays and based on the at least partially optically demodulated light signal received by the optical sensor from the doe.
1. An imaging device comprising:
an optical sensor having an optical axis;
a lens positioned to focus light from a scene onto the optical sensor;
a matrix of variable-phase optical elements that are dimensioned to introduce at least two different phase delays into a light signal received from the scene through the lens, the matrix of variable-phase optical elements:
directing, when in a first position, a portion of the light signal having a first phase delay of the at least two different phase delays to a first pixel of the optical sensor; and
directing, when at least one optical component of the imaging device is shifted laterally relative to another optical component of the imaging device, a portion of the light signal having a second phase delay of the at least two different phase delays to the first pixel of the optical sensor;
a housing that secures the matrix of variable-phase optical elements between the optical sensor and the lens; and
a processing subsystem programmed to determine a phase difference associated with the light signal based on the at least two different phase delays.
2. The imaging device of
3. The imaging device of
4. The imaging device of
5. The imaging device of
a third portion of the light signal having a third phase delay of the at least two different phase delays to a third pixel of the optical sensor; and
a fourth portion of the light signal having a fourth phase delay of the at least two different phase delays to a fourth pixel of the optical sensor, the optical component comprising at least one of the lens, the optical sensor, or the matrix of variable-phase optical elements.
6. The imaging device of
the first phase delay of the at least two different phase delays is 90° out of phase from the second phase delay of the at least two different phase delays;
the second phase delay of the at least two different phase delays is 90° out of phase from the third phase delay of the at least two different phase delays; and
the third phase delay of the at least two different phase delays is 90° out of phase from the fourth phase delay of the at least two different phase delays, the first, second, third, and fourth phase delays producing signals that permit optical quadrature demodulation.
7. The imaging device of
a first diffractive optical element (doe) disposed between the lens and the optical sensor; and
a second doe disposed between the lens and the first doe, the first and second does producing the at least two different phase delays.
8. The imaging device of
the first doe comprises a first substrate having a first pattern of protruding features; and
the second doe comprises a second substrate having a second pattern of protruding features, the first and second patterns of protruding features having different periodicities.
9. The imaging device of
10. The imaging device of
11. The imaging device of
the processing subsystem is programmed to perform a calibration of the delays of the at least two different phase delays; and
the processing subsystem is programmed to determine the phase difference associated with the light signal based on the calibration of the delays of the at least two different phase delays.
13. The depth imaging system of
the processing subsystem is programmed to perform a calibration of the delays of the at least two different phase delays; and
the processing subsystem determines the phase difference associated with the light signal based on the calibration of the delays of the at least two different phase delays.
14. The depth imaging system of
directs a first portion of the light signal having a first phase delay of the at least two different phase delays to the first pixel of the array of photosensitive pixels; and
directs a second portion of the light signal having a second phase delay of the at least two different phase delays to a second pixel of the array of photosensitive pixels.
15. The depth imaging system of
directs a third portion of the light signal having a third phase delay of the at least two different phase delays to a third pixel of the array of photosensitive pixels; and
directs a fourth portion of the light signal having a fourth phase delay of the at least two different phase delays to a fourth pixel of the array of photosensitive pixels.
16. The depth imaging system of
the optical sensor, the lens, and the doe comprise optical components of the imaging device; and
when at least one of the optical components of the imaging device is shifted laterally relative to another of the optical components of the imaging device, the doe:
directs a portion of the light signal having a first phase delay of the at least two different phase delays to the first pixel of the array of photosensitive pixels; and
directs a portion of the light signal having a second phase delay of the at least two different phase delays to a second pixel of the array of photosensitive pixels.
17. The depth imaging system of
19. The imaging device of
|
Imaging systems are used in an increasing number of applications, including in machine vision. Such systems permit devices, such as a head-mounted display, a produce-picking machine, a vehicle, etc., to develop a picture of the immediate operating environment. This may permit many other actions to be performed based on the relationship between the device and its environment, such as the device's position and orientation relative to one or more objects in the scene. Among depth sensing systems for machine vision, conventional phase-based time-of-flight (ToF) sensors may have lower resolution than other image sensors due, at least in part, to the circuitry that may be required to demodulate a reflected signal to determine the phase difference relative to an emitted signal. The phase difference can then be used to calculate depth values within a scene. What is needed, therefore, are improved imaging devices, systems, and methods for phase-based ToF sensors.
As will be described in greater detail below, the instant disclosure describes systems that methods that enable optical demodulation of signals used in phase-based ToF systems. By at least partially demodulating signals in the optical domain, rather than relying only on circuitry, more area on a ToF depth sensor may be used for photosensing and less can be used on circuitry.
In one example, an imaging device may include an optical sensor having an optical axis, a lens positioned to focus light from a scene onto the optical sensor, a matrix of variable-phase optical elements that are dimensioned to introduce at least two different phase delays into a wavefront of a light signal received from the scene through the lens, a housing that secures the matrix of variable-phase optical elements between the optical sensor and the lens, and a processing subsystem programmed to determine a phase difference associated with the light signal based on the at least two different phase delays.
In some implementations, the matrix of variable-phase optical elements, when in a first position, may direct a portion of the light signal having a first phase delay of the at least two different phase delays to a first pixel of the optical sensor. When at least one optical component of the optical device is shifted laterally relative to another optical component of the optical device, the matrix of variable-phase optical elements may direct a portion of the light signal having a second phase delay of the at least two different phase delays to the first pixel of the optical sensor. The matrix of variable-phase optical elements may include a diffractive optical element that directs a portion of the light signal having a first phase delay of the at least two different phase delays to a first pixel of the optical sensor and directs a portion of the light signal having a second phase delay of the at least two different phase delays to a second pixel of the optical sensor. The second phase delay of the at least two different phase delays may be separated from the first phase delay of the at least two different phase delays by a predetermined fraction of a wavelength of the light carrying the light signal. In some implementations, when at least one optical component of the optical device is shifted laterally relative to another optical component of the optical device, the matrix of variable-phase optical elements may direct a portion of the light signal having a third phase delay to the first pixel of the optical sensor and may direct a portion of the light signal having a fourth phase delay to a second pixel of the optical sensor. An optical component of the imaging device may direct a first portion of the light signal having a first phase delay of the at least two different phase delays to a first pixel of the optical sensor, a second portion of the light signal having a second phase delay of the at least two different phase delays to a second pixel of the optical sensor, a third portion of the light signal having a third phase delay of the at least two different phase delays to a third pixel of the optical sensor, and a fourth portion of the light signal having a fourth phase delay of the at least two different phase delays to a fourth pixel of the optical sensor. The optical component may include at least one of the lens, the optical sensor, or the matrix of variable-phase optical elements.
In some implementations, the first phase delay of the at least two different phase delays may be 90° out of phase from the second phase delay of the at least two different phase delays. The second phase delay of the at least two different phase delays may be 90° out of phase from the third phase delay of the at least two different phase delays. The third phase delay of the at least two different phase delays may be 90° out of phase from the fourth phase delay of the at least two different phase delays, the first, second, third, and fourth phase delays producing signals that permit optical quadrature demodulation. The optical sensor may include an array of individual photosensitive regions, with each of the individual photosensitive regions having an area that be less than approximately 2 microns by approximately 2 microns.
In some implementations, the matrix of variable-phase optical elements may include a first diffractive optical element (DOE) disposed between the lens and the optical sensor and a second DOE disposed between the lens and the first DOE, the first and second DOEs producing the at least two different phase delays. The first DOE may include a first substrate having a first pattern of protruding features and the second DOE may include a second substrate having a second pattern of protruding features, with the first and second patterns of protruding features having different periodicities. The imaging device may further include a positioning system that couples the first DOE and the second DOE to the housing, wherein the positioning system independently positions the first and second DOEs to alter a phase delay associated with a first pixel of the optical sensor. The imaging device may include a light projector that projects the light signal as a pulsed light signal onto the scene to be imaged, the pulse light signal being reflected from objects in the scene and directed by the lens toward the optical sensor. The pulsed light signal may include light in a wavelength range from approximately 800 nm to approximately 1000 nm. The pulsed light may be modulated by a continuous-wave, the continuous wave being at least one of a sinusoid or a square wave.
In another example, an imaging device may include an optical sensor having an optical axis and an array of photosensitive pixels, a lens positioned to focus light from a scene onto the optical sensor, a diffractive optical element (DOE) having features that are dimensioned to introduce at least two different phase delays into a wavefront of a light signal received from the scene through the lens to at least partially optically demodulate the light signal, a housing that secures the DOE between the optical sensor and the lens, and a processing subsystem programmed to determine a phase difference associated with the light signal based on at least partially optically demodulated light received by the optical sensor from the DOE.
In some implementations, a width of at least one of the features of the DOE is substantially the same as a width of a first pixel of the array of photosensitive pixels. The processing subsystem may be programmed to perform a calibration of the delays of the at least two phase delays and the processing subsystem may determine the phase difference associated with the light signal based on the calibration of the delays of the at least two phase delays.
In another example, a method for generating a three-dimensional image of a scene may include receiving a first electronic signal from a first pixel of an optical sensor, the first electronic signal characterizing a first portion of a reflected light signal having a first phase delay, receiving a second electronic signal from a second pixel of the optical sensor, the second electronic signal characterizing a second portion of the reflected light signal having a second phase delay that is different than the first phase delay, determining phase characteristics of the reflected light signal based on the first electronic signal and the second electronic signal, determining a distance between the optical sensor and a surface reflecting the reflected light signal based on the determined phase characteristics, and generating a three-dimensional image of a scene based the determined phase characteristics and the received first and second electronic signals from the first and second pixels of the optical sensor.
In some implementations, the method may include receiving a third electronic signal from a third pixel of the optical sensor, the third electronic signal characterizing a third portion of the reflected light signal having a third phase delay, and receiving a fourth electronic signal from a fourth pixel of the optical sensor, the fourth electronic signal characterizing a fourth portion of the reflected light signal having a fourth phase delay, wherein the first, second, third, and fourth phase delays are different. The first portion, second portion, third portion, and fourth portion of reflected light may be received substantially simultaneously by the optical sensor.
In some implementations, the method may further include activating a positioning system to move, into an altered position, a matrix of variable-phase optical elements that are dimensioned to introduce phase delays into a wavefront of reflect light that may include the reflected light signal, while the matrix is in the altered position, receiving a third electronic signal from the first pixel of the optical sensor, the third electronic signal characterizing a third portion of the reflected light signal having a third phase delay, and while the matrix is in the altered position, receiving a fourth electronic signal from the second pixel of the optical sensor, the fourth electronic signal characterizing a fourth portion of the reflected light signal having a fourth phase delay, wherein the first, second, third, and fourth phase delays are different. The method may further include determining a phase difference between the reflected light signal and a previously emitted light signal based on the first, second, third, and fourth electronic signals and activating emission of a pulsed light signal into a scene, the pulsed light signal being reflected off objects in the scene as the reflected light signal. The activation of the positioning system to provide different perspectives may cause dithering of the matrix of variable-phase optical elements between the different perspectives.
Features from any of the above-mentioned embodiments may be used in combination with one another in accordance with the general principles described herein. These and other embodiments, features, and advantages will be more fully understood upon reading the following detailed description in conjunction with the accompanying drawings and claims.
The accompanying drawings illustrate several exemplary embodiments and are a part of the specification. Together with the following description, these drawings demonstrate and explain various principles of the instant disclosure.
Throughout the drawings, identical reference characters and descriptions indicate similar, but not necessarily identical, elements. While the exemplary embodiments described herein are susceptible to various modifications and alternative forms, specific embodiments have been shown by way of example in the drawings and will be described in detail herein. However, the exemplary embodiments described herein are not intended to be limited to the particular forms disclosed. Rather, the instant disclosure covers all modifications, equivalents, and alternatives falling within the scope of the appended claims.
The present disclosure is generally directed to systems, devices, and methods that use a matrix of variable-phase optical elements (e.g., diffractive optical elements (DOES)) to introduce phase delays into a wavefront of light received through a lens, thereby enhancing the performance of optical sensors and devices that capture aspects of a scene. These enhancements may be a function of how light passes through the variable-phase optical elements and, in some embodiments, may also be a function of shifting the matrix or another optical component (e.g., a sensor or lens) of an optical device. For example, the phase delays introduced by the matrix of variable-phase optical elements may enable the optical sensor to capture at least two different perspectives of a scene, and the systems and devices presented herein may use the different perspectives to provide or increase resolution (e.g., in an angular, depth, and/or spectral dimension) of output images or frames of an optical device.
Devices disclosed herein may use variable-phase optical elements to capture different perspectives of a scene in a variety of different manners and for numerous different purposes. For example, a DOE may be configured to, while in an initial position, disperse light from a scene as an interference pattern on an optical sensor, which may capture the interference pattern as a first perspective of the scene. The DOE may be shifted laterally to a second position such that the resulting interference pattern represents another perspective of the scene, which may also be captured by the optical sensor. These two perspectives may be processed to increase angular resolution (e.g., via oversampling) or to provide depth sensing (e.g., via triangulation and/or phase discrimination). For example, depth values of a scene may be obtained using triangulation between two perspectives, by using a DOE to provide the two perspectives to a single optical sensor. As another example, each element within a layer or matrix of variable-phase optical elements may be configured to deterministically phase-shift and focus light onto particular pixels (or sets of pixels) of an optical sensor. These phase-shifted wavefronts, which may represent different perspectives of a scene, may be captured, mixed, and compared against a reference signal to detect depth within a scene.
Embodiments of the instant disclosure may also be implemented within various types of systems (e.g., traditional CMOS sensor systems, time-of-flight (ToF) systems, hyperspectral imaging systems, etc.) having diverse configurations (e.g., configurations with static or movable optical components). As an example of an implementation with movable optical components, an imaging device may include a matrix of variable-phase optical elements positioned over individual pixels of an image sensor and an actuator configured to move a component of the imaging device (e.g., the matrix, the sensor, a lens, etc.) to obtain two different images representing two different instantaneous fields of view (iFOVs) per pixel. The system may then analyze these images to obtain or deduce additional spatial information for the imaged scene. In some examples with a ToF sensor, a scene may be captured in greater spatial resolution by using a conventional large pixel ToF sensor system and translating the component to oversample the portion of the image plane or scene. In examples with a non-ToF sensor (e.g., a traditional CMOS sensor), the system may perform a triangulation operation and/or a phase-discrimination operation on the different iFOVs to calculate a depth map of the scene. The system may also, for both non-ToF and ToF sensors, interpolate between the phase-shifted iFOVs to improve angular resolution of images captured by the sensors.
The oversampling process may also be used to increase spatial resolution in various hyperspectral imaging systems (e.g., snapshot hyperspectral imaging systems). Traditional hyperspectral imaging may use hyperspectral filters (e.g., tiled filters, mosaic filters, etc.) disposed directly on a sensor to sample broadband light in the spectral domain, which may increase spectral resolution at the expense of spatial resolution. In contrast, the proposed hyperspectral imaging system may decouple the hyperspectral filters from the sensor and position the variable-phase optical elements between the filters and the sensor to facilitate spatial oversampling and improved spatial resolution. For example, a scene may be captured in a hyperspectral image in greater spatial resolution by translating the variable-phase optical elements to oversample portions of the image plane or scene through the individual windows of the hyperspectral filter.
In addition to being used to improve resolution through triangulation, phase discrimination, and oversampling, the optical elements disclosed herein may be configured to replace at least one electrical phase-shift component of a demodulation system. For example, an optical device may include variable-phase optical elements positioned over a sensor to introduce deterministic phase shifts into an incident wavefront. The system may then capture the phase-shifted images at a sensor and send them to a demodulation circuit that (1) uses the images to determine a phase shift of the incident wavefront relative to a transmitted carrier signal and (2) uses the phase shift to identify depth within a scene. In some examples, the system may provide relatively low phase resolution by comparing two phase-shifted signals or may provide higher phase resolution by comparing several (e.g., three or more) phase-shifted signals. A time of flight measurement can be performed using the phase-shifted signals in a ToF depth sensor. Additionally or alternatively, the system may include a single layer of variable-phase optical elements or stacked layers of variable-phase optical elements configured to introduce phase shifts that are traditionally handled by electrical components. Examples of such stacked or layered configurations are included in
In such a system, each optical component may be fixed in a single position and/or movable among two or more positions in a plane perpendicular to the optical axis. For example, a system with fixed optical components may introduce two or more different phase shifts in an incident wavefront. These phase-shifted signals may then be mixed and compared with a reference signal. As another example, a global shutter system may include optical elements that create two phase-shifted optical paths that are captured and stored by a sensor while the optical elements are in a first position. The system may then shift the optical elements to a second position to create two additional phase-shifted optical paths, which may also be captured by the sensor. As a result, the sensor may simultaneously provide four phase-shifted signals to an electrical quadrature demodulation component, where they may be mixed and compared to a reference signal to create a depth map of a scene. Embodiments of the instant disclosure may also provide various other configurations, features, and advantages over traditional optical sensor systems, as discussed in greater detail with reference to the
The following will provide, with reference to
While
Some embodiments of the optical sensor device 105 may include an imaging device 120, an electronic display 125, an optical assembly 130 (also referred to as an optics block 130), one or more position sensors 135, and an inertial measurement unit (IMU) 140. Some embodiments of the optical sensor device 105 may have different components than those described in conjunction with
The imaging device 120 may capture data characterizing a scene or local area surrounding some or all of the optical sensor device 105. In some embodiments, the imaging device 120 may include a traditional image sensor, such that the signals captured by the imaging device 120 include only two-dimensional image data (e.g., data having no depth information). In some embodiments, the imaging device 120 may operate as a depth imaging system that computes depth information for a scene using collected data (e.g., based on captured light according to one or more computer-vision schemes or algorithms, by processing a portion of a structured light pattern, by time-of-flight (ToF) imaging, by simultaneous localization and mapping (SLAM), etc.), or the imaging device 120 may transmit corresponding data to another device, such as the processing subsystem 110, which may determine or generate the depth information using the data from the imaging device 120. To enable or augment such computer-vision schemes, the imaging device 120 may include a projector device, in some embodiments.
In some embodiments, the imaging device 120 may be a hyperspectral imaging device that can represent a scene as multiple spectra of light such that different features or objects within the scene, which may be best visualized utilizing light of specific wavelengths, may be better understood, analyzed, and/or visually or quantitatively described.
In embodiments including the electronic display 125, the electronic display 125 may display two-dimensional or three-dimensional images to the user in accordance with data received from the processing subsystem 110. In embodiments including the electronic display 125, the optical assembly 130 may magnify image light received from the electronic display 125, correct optical errors associated with the image light, and/or present the corrected image light to a user of the optical sensor device 105.
The I/O interface 115 in
The processing subsystem 110 may receive data from the optical sensor device 105 for processing to extract information or to combine data sets. In some embodiments, the processing subsystem 110 may provide content to the optical sensor device 105 for processing in accordance with information received from one or more of the imaging device 120, the optical sensor device 105, and the I/O interface 115. In the example shown in
The application store 150 may store one or more applications or instruction sets for execution by the processing subsystem 110 or by the optical sensor device 105. An application may, in some examples, represent a group of instructions that, when executed by a processor, generates content for presentation to the user. Content generated by an application may be generated in response to inputs received from the user via movement of the optical sensor device 105 or the I/O interface 115. Examples of applications include gaming applications, conferencing applications, video playback applications, or other suitable applications. The application store 150 may be a non-transitory memory store that also stores data obtained from the imaging device 120 or from other sources included in the optical sensor device 105 or received from the processing subsystem 110. Some exemplary applications in the application store 150 may include instructions for performing the methods described herein.
The tracking module 155 may calibrate the optical sensor system 100 using one or more calibration parameters and may adjust the calibration parameters to reduce error in determination of the position of the optical sensor device 105 or of the I/O interface 115. Additionally, the tracking module 155 may track movements of the optical sensor device 105 or of the I/O interface 115 using information from the imaging device 120, the one or more position sensors 135, the IMU 140, or some combination thereof.
The engine 160 may generate a three-dimensional depth mapping or multiple three-dimensional depth mappings of the area (e.g., the “scene” or the “local area”) surrounding some or all of the optical sensor device 105 based on information received from the optical sensor device 105 or from components thereof, such as the imaging device 120. In some embodiments, the engine 160 may generate depth information for the three-dimensional mapping of the scene based on two-dimensional information or three-dimensional information received from the imaging device 120 that is relevant for techniques used in computing depth maps. The depth maps may include depth dimension values for each of the pixels in the depth map, which may represent multiple different portions of a scene. The engine 160 may calculate depth information using one or more techniques in computing depth from structured light or unstructured light. In various embodiments, the engine 160 may use the depth information to, for example, generate or update a model of the local area, and may generate content based in part on the model. For example, the engine 160 may identify a first delay component or phase difference ϕ1 based on an emitted signal and a received signal in a ToF depth sensor system. The phase difference ϕ1 may be determined by the engine 160 by subtracting a known or deterministic second delay component ϕ2 from a measured phase difference ϕ, as is described herein in further detail.
Additionally, the optical sensor system 100 may include a communication bus 165 that may transmit information between individual components of the optical sensor device 105, the processing subsystem 110, and/or the I/O interface 115 to permit the individual components to cooperate according to embodiments described herein. The I/O interface 115 may permit the optical sensor system 100 to interact, via a wired or wireless channel, with external devices and/or system accessories, such as additional standalone-sensor systems, hand-held controllers, etc.
As described herein, the imaging device 210 may be used to permit a conventional image capture system to provide depth information in addition to two-dimensional image information, to oversample light reflected from a scene to increase resolution of depth images, to enable optical demodulation to detect phase differences in reflected and reference light signals, and/or to increase resolution of hyperspectral images beyond the limits imposed by hyperspectral filtering.
In some embodiments, the optical sensor 312 may be configured to capture light primarily in the visible wavelength range. For example, the optical sensor 312 may include an optical layer 318 disposed directly thereon or thereover. The optical layer 318 may include an infrared filter and/or an antireflective coating, in some embodiments. In other embodiments, the optical layer 318 may be omitted or may include an antireflective coating without an infrared filter or other color filter. Additionally, some embodiments of the optical layer 318 may include a visible wavelength filter that blocks or inhibits light in the visible spectrum while permitting other light, such as infrared light of a predetermined wavelength range, to be received by the optical sensor 312. In some embodiments, the optical sensor 312 may be another type of sensor, such as a ToF sensor that detects the time delay or phase difference between direct and reflected transmissions of an emitted light wave or light signal, such as light emitted by the light projector device 250 of
The imaging device 310A may further include an optical matrix 320, disposed along the optical axis 314 and between the lens 316 (which may represent multiple lenses) and the optical sensor 312. The optical matrix 320 may be a set or matrix of filters, lenses, lenslets, prisms, refractive arrays, and/or other optical elements that can alter light directed by the lens 316 to the optical sensor 312 by altering a direction of the light, focusing the light on a particular region of the optical sensor 312, and/or introducing a phase delay into the light. Unlike a single lens, some embodiments of the optical matrix 320 may have a discontinuous impact on the light passing therethrough, such that the effect of the optical matrix 320 may not be described by a continuous function along the surface of the optical matrix 320. Rather, the optical matrix 320 may generate a desired interference pattern. The optical matrix 320 may have a matrix of variable-phase optical elements present on at least one surface. As shown in
DOEs may operate by using interference and diffraction to produce a desired change in the light passing through. For example, based on the pattern of optical elements on a particular DOE, the DOE can operate as a beam shaper to produce a desired pattern in the transmitted light. The optical matrix 320 may include a matrix of optical elements that cause light to be directed in a desired pattern to individual pixels or sets of pixels in the optical sensor 312. Accordingly, DOEs may be used in some embodiments of the optical matrix 320 to direct light and/or to introduce desired phase delays into light that is directed to specific pixels in the optical sensor 312. Some examples of patterns that may be induced by a DOE are included in
The optical matrix 320 may be coupled to the lens 316 and the optical sensor 312 in a variety of ways. For example, an edge or edges of the optical matrix 320 may be mechanically secured between the lens 316 and the optical sensor 312 by a housing 322 (e.g., housing 322 may include corresponding recesses or channels formed that match external dimensions of the lens 316). The lens 316 may also be secured to the housing 322 by chemical means, such as an adhesive. The housing 322 may be similarly coupled to the optical matrix 320. For example, the optical matrix 320 may be coupled to the housing 322 in a fixed relationship, such as by an adhesive or secure press-fit relationship, or in a movable relationship, such that the optical matrix 320 may be moved relative to the housing 322 in at least one dimension and thereby moved relative to the optical sensor 312 at least one dimension. For example, the optical element matrix 320 may include portions positioned within one or more channels 324 formed in an interior wall of the housing 322 that constrains the optical matrix 320 to movement in two dimensions substantially parallel to the optical sensor 312.
Additionally or alternatively, the optical matrix 320 may be secured to the housing 322 by one or more components of a positioning system 326. As shown in
By operation of circuitry included on the optical sensor 312 or an external processing device, the optical matrix 320 may be positioned in a controlled manner in a plane substantially parallel to the optical sensor 312 itself and orthogonal to the optical axis 314. In some embodiments, the optical matrix 320 may further be movable along the optical axis 314 by the positioning system 326. When the optical matrix 320 is moved parallel to the optical sensor 312, light directed to an individual element of the optical matrix 320 may be redirected from a first pixel or first set of pixels of the optical sensor 312 to a second pixel or second set of pixels of the optical sensor 312. Accordingly, movement of the optical matrix 320 may result in a corresponding movement in the FOV of any given pixel (e.g., the iFOV of a pixel) in the optical matrix 320. In some embodiments, the light directed to the first pixel or first set of pixels may have a different phase delay after the optical matrix 320 is moved than before such movement.
While some embodiments of the DOE 330 may include patterned features forming a matrix of variable-phase optical elements on one side (shown in
As shown in
While the DOE 400A in
Returning to
In some embodiments, the optical matrix 320 and the optical matrix 340 may be considered as layers or components in a single stacked optical matrix, such that the optical matrix has additional dimensions. For example, the optical matrix 320 of
The imaging device 310D may further include a hyperspectral filter 350. The hyperspectral filter 350 may include a plurality of filter windows, with each window or type of window passing a specific wavelength of light. In some embodiments, the windows may be formed by depositing material layers on a substrate that is transparent to a broad range of wavelengths. In some embodiments, the windows are formed such that they extend in lines across a main substrate of the filter 350. In other embodiments, the windows may be tiled or mosaiced windows, such that each pixel has a corresponding window or that sets of pixels (e.g., 4 pixels by 4 pixels, 4 by 8 pixels, 10 pixels by 10 pixels, etc.) are associated with a particular window. The tiled or mosaiced windows in the hyperspectral filter 350 may be arranged in a repeating pattern across the surface of the hyperspectral filter 350. In embodiments including the hyperspectral filter 350, the optical matrix 320 and/or the optical sensor 312 may omit a color-filter array deposited thereon. Additionally, in some embodiments of the imaging device 310D, the hyperspectral filter 350 may be secured to the housing 322 by fixed or movable positioning components, like those described herein for securing optical matrices to housings.
Embodiments of the instant disclosure may enable spatial oversampling and/or spectral oversampling within hyperspectral imaging systems. Movement of an optical component of the imaging device 310D may provide for spectral oversampling by shifting an iFOV of a scene such that the iFOV is captured through a filter of an initial wavelength spectrum in a first position and through a filter of a different wavelength spectrum in a second position. For example, either or both of the optical matrix 320 and/or the optical matrix 340 may be moved, causing an iFOV of a particular pixel to shift to be captured via a different spectral filter. By capturing an iFOV of a scene via multiple different filters, a higher resolution spectral image may be created.
To provide spatial oversampling, the imaging device 310D may be used to capture a first perspective of a scene when the optical component, e.g. the optical matrix 320, is in a first position and is directing filtered light through a first filter window to the first pixel and to capture a second perspective of the scene when the optical component is in a second position and is directing filtered light through the first filter window to the first pixel. In other words, the optical matrix 320, when in a first position, may enable a pixel (or set of pixels) to capture a first iFOV through a particular spectral filter. When moved to the second position, optical matrix 320 may enable the pixel (or set of pixels) to capture a second iFOV through the same spectral filter. To produce an enhanced hyperspectral image, information collected from the pixels at different times, i.e., when the moveable optical matrices 320 and/or 340 are in different positional configurations, may be combined by the processing subsystem 110 of
As shown in
By operating as shown in
When an optical matrix 510 is included in the imaging device, more detailed information may be obtained from the scene 508. In some embodiments, multiple optical components or a single optical component may be included in an imaging system and may be moveable in one or more directions while remaining parallel to the optical sensor of an imaging device. By including an optical component, such as an optical matrix, in the optical path of an imaging device, the iFOVs of the pixels of the imaging device may be decreased to provide increased angular resolution, as shown by the default position sample 602 of
To perform oversampling, multiple images may be combined into one image or frame. For example, a first image may be captured by the optical sensor 312 while the optical matrix 510 is in the default position, corresponding to a default perspective on the scene 508. When the optical matrix 510 is moved, a second image may be captured by the optical sensor 312, the second image corresponding to a different perspective on the scene 508. Accordingly, to capture the three samples shown in the oversampling pattern 608A or 608B, three perspectives may be captured. Similarly, for the oversampling pattern 608C, four perspectives may be captured, and for the oversampling pattern 608D, nine perspectives may be captured. These additional perspectives may be captured in an image. Such images, which may be intended to be combined into a single output image or output frame, may be referred to as intermediate images. By combining the information from the intermediate images according to information characterizing the corresponding positions, the images may be properly combined into a final output image or frame.
Intermediate images may be combined in any suitable manner using any suitable process. For example, intermediate images may be combined in an interpolation process that interpolates data points between pixel data of two different intermediate images (e.g., two different perspectives or iFOVs). For example, the values defining the altered position sample 606 and the default position sample 602 may be combined, such as by averaging, to estimate an intermediate position sample. Such interpolation may increase resolution of the scene in one or more dimensions. For example, a processing subsystem may combine a first intermediate image, a second intermediate image, and interpolated pixel data into an increased-resolution output image.
As shown in
In some embodiments, one or more of the optical matrices 620G and 640G may be fixed. As shown, the optical matrix 620G may be moved from a default position in a first direction substantially parallel to the optical sensor 312, while the optical matrix 640G may be moved in a second direction that may also be substantially parallel to the optical sensor 312. As shown, the second direction may be orthogonal to the first direction. The combined movements of the optical matrices 620G and 640G may produce a shift or movement of the induced pattern 630G in directions opposite to the movement of the optical matrices 620G and 640G, such that the pattern 630G is shifted diagonally with respect to the optical sensor 312. For example, the matrices 620G and 640G may induce a phase delay 2 in the light incident on the pixel 612 while the matrices 620G and 640G are in a default position.
While
When the wavelength of the carrier light is known in advance, the relative dimensions of the optical matrices 701A and 701B may be derived from the wavelength. As shown in
Phase differences introduced by optical elements may be capture, stored, and compared in a variety of ways. For example, each pixel in the optical sensor 706 may capture a first iFOV when the optical matrix 701A is in a first position and a second, phase-shifted iFOV when the optical matrix 701A is in a second position. Two or more of these phase-shifted signals may, in addition to being used to increase resolution of an output image, be compared against a reference signal to determine depth with within a scene. Additionally or alternatively, phase-shifted signals captured by different pixels may be compared against the reference signal to determine depth within the scene. For example, the signals produced by the pixels 708A and 708B of optical sensor 706 may be compared to determine a phase difference. Such phase differences may be used to identify the time it took for the signal 700 to be emitted (the time of emission may be known) and to be reflected back to the optical sensor 706. From this time, the distance from the pixels 708A and 708B to whatever object in the scene corresponds to the FOV of these pixels can be determined. Using such pixel-to-object distances, a three-dimensional depth map of the scene may be reconstructed.
Conventional ToF depth sensors may have large pixels in part due to the circuitry required to perform circuit-based demodulation to recover the phase information accurately. The embodiments described herein may permit such demodulation to be performed, at least partially, in the optical domain, without the need for at least a portion of the traditional demodulation circuitry. In such embodiments, the overall pixel size of the optical sensor 312 may be decreased, which may enable fabrication of higher-resolution ToF sensors capable of creating higher-resolution depth maps than may be obtained with conventional ToF sensors. This may permit CMOS image sensors, the type of sensors used to capture conventional two-dimensional images such as a cellphone camera sensor, to be employed as ToF depth sensors with correspondingly smaller pixel sizes, such as approximately 2 microns by 2 microns or less in certain examples. In some embodiments, the phase difference between the emitted and reflected signals may be determined using a combination of optical demodulation and circuit-based demodulation.
The emitted signal 802 and the reflected signal 804 may be separated by a phase shift or phase difference φ, as shown in
An image processor may perform the operations of equation (1) to determine a depth from the phase difference φ1. For example, the image processing engine 160 of the processing subsystem 110 of
In some embodiments, a calibration process may be performed before collecting data so that the delays may be determined precisely, and any undesired deviations from the intended dimensions of the optical matrix 701A may be compensated for by resulting calibration factors. The phase delays θ2, θ3, θ4, and θ5 may be selected and embodied in the substrate and protruding features and/or recessed features of an optical matrix or other optical component such that the phase of the reflected signal 804 may be sampled in all four quadrants (I, II, III, and IV) of the unit circle 806 shown by sample points 808A, 808B, 808C, and 808D. The signals may be generated by the accumulation of charges by pixels receiving the portions of light with the different known phase delays, for which the signals may be compensated. These signals generated by pixels 706A-D of
In this way, the phase difference φ1 may be determined based on optically delayed signals rather than electronically delayed signals as in some conventional phase-based ToF depth sensors.
Some embodiments may also leverage existing global-shutter and pixel-level-storage technologies to use two pixels to create four phase-shifted signals without sacrificing the additional resolution that is lost when using four different pixels to create the signals shown in
In some embodiments, the optical matrices 701A and 701B, the DOEs 400A-400D, and some other optical components described herein may be produced using manufacturing techniques similar to those used in semiconductor device manufacturing and semiconductor mask fabrication. For example, the substrate 702 may be a portion of a wafer, such as a silicon wafer, a silicon-on-insulator (SOI) wafer, or another wafer of a material having a suitable refractive index that provides for a change of refraction and/or diffraction that can introduce a phase delay or alter the direction of propagation of a light signal in an interference pattern.
The substrate 702 may have a thickness between approximately 0.5 microns and approximately tens or hundreds of microns, in some examples. The features 704 and 712 may be formed by additive processes and/or subtractive processes. For example, the material of the features 704 may be deposited over the substrate 702 by physical vapor deposition (PVD), chemical vapor deposition (CVD), ion beam deposition, vapor-phase epitaxy, atomic layer deposition, etc. Some embodiments may include an etching process or another material removal process that removes substrate material from a material layer to formed the patterned features 704 and 712. Accordingly, the material of the substrate 702 may be different from the material of the features 704/712. In some implementations, the various steps of the feature 712 may be produced as a result of several patterning and growth processes and/or a result of patterning and etching process. In some embodiments, such as in embodiments using epitaxial growth, the features 704 may be grown on a patterned surface. For example, the features may be grown to the desired height H3 on portions of the substrate 702 exposed by windows formed in a photoresist layer.
The height H3 may introduce a desired phase delay (e.g., 45°, 90°, 180°, 270° of phase delay) based on the refractive index of the material of feature 704 and its dimensions. In some embodiments, the height H3 may be approximately 5 nm to approximately 50 nm to produce the desired phase delay. In other embodiments, the height H3 may be greater such that it introduces a greater phase delay that is equivalent to 360° plus the actual desired phase delay of 90°, 180°, 270°, etc. This greater height H3 and associated greater phase delay may provide a phase-equivalent delay to the desired delay of the lower height while improving manufacturability.
As illustrated in
At step 904, one or more of the systems described herein may capture, with the optical sensor, at least two different perspectives of the scene. For example, the optical sensor 502 of
At step 906, one or more of the systems described herein may process the two different perspectives of the scene to create an output frame or output image with a higher resolution than either of the captured two different perspectives of the scene. For example, the image processing circuitry of the optical sensor 212, 312, or 502, or an external processor, may combine the different perspectives to generate an enhanced representation of the scene 508 having samples as shown in
Additionally or alternatively, operations of the method 900A may include embodiments in which capturing the at least two perspectives of the scene may include capturing a first portion of light with a first pixel or set of pixels of an optical sensor. The “portion of light” may refer to the light from a scene, like the scene 508 of
Embodiments of the method 900B may begin at step 912 in which any of the systems described herein may position a matrix of variable-phase optical elements, disposed between an optical sensor and a lens positioned to focus light from a scene onto the optical sensor, in a first position at least substantially perpendicular to an optical axis of the optical sensor. For example, the DOE 330 of
At step 914, one or more of the disclosed systems may capture, with the optical sensor, at least two different perspectives of a scene. For example, the optical sensor 312 of
At step 916, one or more of the disclosed systems may determine depth characteristics of the scene based on the captured at least two perspectives of the scene. For example, the depth characteristics may be distances between a particular pixel or set of pixels and an object or feature of the scene being observed. In some embodiments, the depth characteristics may be obtained by performing a triangulation algorithm based on the at least two perspectives of the scene obtained at step 914. For example, the positioning system used to move the optical sensor between different positions corresponding to the at least two perspectives may be calibrated so that the absolute distance between the positions is known based on actuation signals used to control the positioning system. Because the distance between the two perspectives may be known, the difference in perspectives of various objects or features in the captured perspectives may be used to determine angles in each of the captured perspectives. For example, using the angle to a feature in a first captured perspective, the angle to the same feature in a second captured perspective, and the known distance between the two captured perspectives, an estimate of the distance from the centroid of the captured perspectives, which will be at some position on the optical sensor 312, may be determined using trigonometric relationships. The algorithm may be applied for a plurality of the pixels in an image to generate a depth map with depth values for many, most, or all of the pixels of the optical sensor 312. Any optical component of the imaging system may be moved in an oscillating or dithered manner to provide for many different measurements or captures, which may be combined or averaged in order to improve the accuracy of the triangulation. For example, an optical matrix may be dithered between two positions to obtain two baseline perspectives of a scene.
At step 918, one of more of the described systems may generate an output depth image of the scene. When rendered or similarly processed, the output depth image may provide a visual or mathematical representation of the depth characteristics of the scene. The scene may be representationally recreated by rendering the output depth image. In some embodiments, the depth image or depth map may have the same x- and y-direction resolution as the intermediate images captured to represent the at least two perspectives. The depth map further includes depth values, such as z-direction values extending along the optical axis of the optical sensor, like the optical axis 314 of
Some embodiments of the method 900B may further include a step of emitting a light signal into the scene. Determining depth characteristics of the scene based on the captured at least two perspectives of the scene may include determining phase characteristics of the light signal. For example, the light signal 700 of
In some embodiments of the method 900B, capturing the at least two different perspective may include capturing an image from each perspective, including at least a first image and a second image. The first image and the second image each include depth information, in some embodiments, while not including depth information in other embodiments.
At a step 922, one or more systems described herein may receive a first electronic signal from a first pixel of an optical sensor. This first electronic signal may represent a first portion of a reflected light signal having a first phase delay. For example, the pixel 708A of the optical sensor 706 shown in
At a step 924, one or more systems described herein may receive a second electronic signal from a second pixel of the optical sensor. The second electronic signal may represent a second portion of the reflected light signal having a second phase delay that is different than the first phase delay. For example, the pixel 708B of the optical sensor 706 shown in
At a step 926, one or more systems described herein may determine phase characteristics of the reflected light signal based on the first electronic signal and the second electronic signal. For example, the phase of the light signal 700 and the time-of-flight of the light signal may be recovered based on the known phase difference between the signals received at both the pixel 708A and the pixel 708B. In some embodiments of the method 900C, additional electronic signals may be received from additional pixels of the optical sensor. These additional signals may also include phase delays that are different than the first and second phase delays.
For example, embodiments of the method 900C may include steps of receiving a third electronic signal from a third pixel of the optical sensor and receiving a fourth electronic signal from a fourth pixel of the optical sensor. The third electronic signal may characterize a third portion of the reflected light signal having a third phase delay and the fourth electronic signal may characterize a fourth portion of the reflected light signal having a fourth phase delay. As shown in
At a step 928, one or more systems described herein may determine a distance between the optical sensor and a surface reflecting the reflected light signal based on the determined phase characteristics. For example, the optical sensor 706 may include circuitry to determine a distance between individual pixels of the optical sensor 706 and features of the scene, based on the time-of-flight of the reflected signal 700. In determining the distance, a phase difference between the reflected signal and the previously emitted light signal that was reflected may be used. Embodiments of the method 900C may include activating emission of a light signal into a scene, so that the reflections from objects and features in the scene may be captured using the optical sensor 706. For example, the light source 252 of the projector device 250 of
One or more of the steps of the methods 900A, 900B, and/or 900C, or other operations described herein may be performed by a processing subsystem. Such a processing subsystem may be a combination of discrete components, firmware, or software, and any of which can be located on a common circuit board, such as PCB 202, to which an optical sensor is attached, within another local component of an exemplary imaging device, or in a remote component, like the processing subsystem 110 of
The front rigid body 1005 may include one or more electronic display elements, one or more integrated eye tracking systems, an IMU 1030, one or more position sensors 1035, and one or more reference points 1015. In the embodiment shown by
In some embodiments, the imaging system 1102 may determine depth and/or surface information for objects within the scene 1101 in a variety of ways. For example, the imaging system 1102 may be utilized in a SLAM tracking system to identify and/or map features of the scene 1101 and/or to identify a location, orientation, and/or movement of the HMD 1000 and/or other objects (e.g., hand-held controllers, users, etc.) in the scene 1101. In some examples, the projector device 1130 may emit light 1131 as a structured light pattern (e.g., a symmetric and/or quasi-random dot pattern, a grid pattern, horizontal bars, etc.) into the scene 1101. The emitted light 1131 may have a wavelength range of 400 nm to about 1100 nm. In some embodiments, the emitted light 1131 may have a narrower wavelength range, such as 800 nm to about 980 nm.
In these examples, the imaging system 1102 may determine the depth and/or surface information based on triangulation or perceived deformation of the emitted pattern. Additionally or alternatively, the imaging system 1102 may capture ToF information describing the time required for light emitted from the illumination source of the projector device 1130 to be reflected from one or more objects in the scene 1101 back to the imaging device 1120, which collects reflected light 1121. In this embodiment, the imaging system 1102 may determine a distance between the imaging system 1102 and the objects in the scene 1101 based on the ToF information.
In some examples, information collected by the imaging system 1102 may be used as part of an image and/or video (e.g., an artificial reality image and/or video) displayed to a user wearing the HMD 1000. In one example, shown in
Accordingly, embodiments of the instant disclosure may include or be implemented in conjunction with an artificial reality system. Artificial reality is a form of reality that has been adjusted in some manner before presentation to a user, which may include, e.g., a virtual reality (VR), an augmented reality (AR), a mixed reality (MR), a hybrid reality, or some combination and/or derivatives thereof. Artificial reality content may include completely generated content or generated content combined with captured (e.g., real-world) content. The artificial reality content may include video, audio, haptic feedback, or some combination thereof, any of which may be presented in a single channel or in multiple channels (such as stereo video that produces a three-dimensional effect to the viewer). Additionally, in some embodiments, artificial reality may also be associated with applications, products, accessories, services, or some combination thereof, that are used to, e.g., create content in an artificial reality and/or are otherwise used in (e.g., perform activities in) an artificial reality. The artificial reality system that provides the artificial reality content may be implemented on various platforms, including an HMD connected to a host computer system, a standalone HMD, a mobile device or computing system, or any other hardware platform capable of providing artificial reality content to one or more viewers.
The process parameters and sequence of the steps described and/or illustrated herein are given by way of example only and can be varied as desired. For example, while the steps illustrated and/or described herein may be shown or discussed in a particular order, these steps do not necessarily need to be performed in the order illustrated or discussed. The various exemplary methods described and/or illustrated herein may also omit one or more of the steps described or illustrated herein or include additional steps in addition to those disclosed.
The preceding description has been provided to enable others skilled in the art to best utilize various aspects of the exemplary embodiments disclosed herein. This exemplary description is not intended to be exhaustive or to be limited to any precise form disclosed. Many modifications and variations are possible without departing from the spirit and scope of the instant disclosure. The embodiments disclosed herein should be considered in all respects illustrative and not restrictive. Reference should be made to the appended claims and their equivalents in determining the scope of the instant disclosure.
Unless otherwise noted, the terms “connected to” and “coupled to” (and their derivatives), as used in the specification and claims, are to be construed as permitting both direct and indirect (i.e., via other elements or components) connection. In addition, the terms “a” or “an,” as used in the specification and claims, are to be construed as meaning “at least one of.” Finally, for ease of use, the terms “including” and “having” (and their derivatives), as used in the specification and claims, are interchangeable with and have the same meaning as the word “comprising.”
Bikumandla, Manoj, Bardagjy, Andrew Matthew, Hazegh, Cina
Patent | Priority | Assignee | Title |
Patent | Priority | Assignee | Title |
5227915, | Feb 13 1990 | Holo-or Ltd. | Diffractive optical element |
5742433, | Oct 18 1993 | Matsushita Electric Industrial Co., Ltd. | Diffractive optical device including grating elements with different grating periods and duty ratios |
6327086, | Jul 09 1996 | Canon Kabushiki Kaisha | Optical diffraction device and exposure apparatus |
9102055, | Mar 15 2013 | GOOGLE LLC | Detection and reconstruction of an environment to facilitate robotic interaction with the environment |
20020135825, | |||
20030223618, | |||
20070105445, | |||
20070132857, | |||
20070146531, | |||
20080297752, | |||
20090127429, | |||
20120105823, | |||
20120120030, | |||
20130002903, | |||
20130075607, | |||
20130088621, | |||
20130226344, | |||
20140111620, | |||
20140145281, | |||
20140153783, | |||
20150035946, | |||
20160026253, | |||
20160148349, | |||
20160260205, | |||
20160288330, | |||
20160309133, | |||
20160364015, | |||
20170059299, | |||
20170220875, | |||
20180063390, | |||
20180096468, | |||
20180124371, | |||
20180217260, | |||
20180218509, | |||
20180247424, | |||
20180304824, | |||
20190004535, | |||
20190037136, | |||
20190049232, | |||
20190204599, | |||
20190208183, | |||
20190356895, |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Jan 24 2018 | Facebook Technologies, LLC | (assignment on the face of the patent) | / | |||
Jan 25 2018 | BIKUMANDLA, MANOJ | OCULUS VR, LLC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 044826 | /0494 | |
Jan 25 2018 | BARDAGJY, ANDREW MATTHEW | OCULUS VR, LLC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 044826 | /0494 | |
May 21 2018 | BIKUMANDLA, MANOJ | OCULUS VR, LLC, | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 049610 | /0316 | |
May 21 2018 | BARDAGJY, ANDREW MATTHEW | OCULUS VR, LLC, | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 049610 | /0316 | |
Jul 16 2018 | HAZEGH, CINA | OCULUS VR, LLC, | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 049610 | /0316 | |
Sep 04 2018 | OCULUS VR, LLC | Facebook Technologies, LLC | CHANGE OF NAME SEE DOCUMENT FOR DETAILS | 047112 | /0836 | |
Mar 18 2022 | Facebook Technologies, LLC | META PLATFORMS TECHNOLOGIES, LLC | CHANGE OF NAME SEE DOCUMENT FOR DETAILS | 060199 | /0876 |
Date | Maintenance Fee Events |
Jan 24 2018 | BIG: Entity status set to Undiscounted (note the period is included in the code). |
Mar 26 2024 | M1551: Payment of Maintenance Fee, 4th Year, Large Entity. |
Date | Maintenance Schedule |
Oct 13 2023 | 4 years fee payment window open |
Apr 13 2024 | 6 months grace period start (w surcharge) |
Oct 13 2024 | patent expiry (for year 4) |
Oct 13 2026 | 2 years to revive unintentionally abandoned end. (for year 4) |
Oct 13 2027 | 8 years fee payment window open |
Apr 13 2028 | 6 months grace period start (w surcharge) |
Oct 13 2028 | patent expiry (for year 8) |
Oct 13 2030 | 2 years to revive unintentionally abandoned end. (for year 8) |
Oct 13 2031 | 12 years fee payment window open |
Apr 13 2032 | 6 months grace period start (w surcharge) |
Oct 13 2032 | patent expiry (for year 12) |
Oct 13 2034 | 2 years to revive unintentionally abandoned end. (for year 12) |