System and method for monitoring eye movement

Abstract

Apparatus for monitoring movement of a person's eye, e.g., to monitor drowsiness. The system includes a frame that is worn on a person's head, an array of emitters on the frame for directing light towards the person's eye, and an array of sensors on the frame for detecting light from the array of emitters. The sensors detect light that is reflected off of respective portions of the eye or its eyelid, thereby producing output signals indicating when the respective portions of the eye is covered by the eyelid. The emitters project a reference frame towards the eye, and a camera on the frame monitors movement of the eye relative to the reference frame. This movement may be correlated with the signals from the array of sensors and/or with signals from other sensors on the frame to monitor the person's level of drowsiness.

Description

This application is 810 812. Preferably, the camera 830 is mounted on or adjacent the bridge piece 814 such that the camera 830 is oriented towards the region surrounding one of the user's eyes while minimizing interference with the user's vision. The camera 830 preferably includes a bundle of fiberoptic cables 832 that terminate in a lens 834, as shown in FIG. 14, on a first end mounted adjacent the bridge piece 814 and a second end 837 that is connected to a detector 838, e.g., a CCD or CMOS sensor, such as those used in endoscopes, that may convert an image into a digital video signal. The camera 830 is configured to detect the frequency of light emitted by the emitters 820, e.g., infrared light. The camera 830 may rely on the light projected by the emitters 820, or the fiberoptic cables 832 may include emitters 836 for projecting light, e.g., infrared light, onto the user's eyes and/or face. In addition, the system 810 may include a second camera 840 oriented away from the user's head, e.g., to monitor the user's surroundings.

One of the ear supports 816 may include a panel 818 for mounting a controller or other processor 842, a transmitter 844, an antenna 845, and a battery 846. Preferably, the processor 840 842 is coupled to the emitters 820, the sensors 822, and/or the camera 830 for controlling their operation. The transmitter 844 may be coupled to the processor 842 for receiving the output signals from the sensors 822 and/or the video signals from the camera 830, e.g., to transmit the signals to a remote location, as described below. Alternatively, the transmitter 844 may be coupled directly to output leads from the sensors 822 and the camera 830. The frame 812 may also include manual controls (not shown), e.g., on the ear support 816, for example, to turn the power off and on, or to adjust the intensity and/or threshold of the emitters 820, the sensors 822, and/or the camera 830.

If desired, the system 810 may also include one or more additional sensors on the frame 812. The sensors may be coupled to the processor 842 and/or to the transmitter 844 so that the signals from the sensors may be monitored, recorded, and/or transmitted to a remote location. For example, one or more position sensors 852a, 852b may be provided, e.g., for determining the spatial orientation of the frame 812, and consequently the user's head. For example, actigraphic sensors may be provided to measure tilt or movement of the head, e.g., to monitor whether the user's head is drooping forward or tilting to the side. Acoustic sensors, e.g., a microphone 854 may be provided for detecting environmental noise or sounds produced by the user.

In addition or alternatively, the frame 812 may include one or more sensors for measuring one or more physical characteristics of the user. For example, EEG electrodes 856 may be provided on the ear support 816, above or below the nasion, and/or other region that may contact the patient's skin to measure brain activity, e.g., waking, drowsy, or other sleep-related brain activity. An EKG electrode (not shown) may be provided that is capable of measuring cardiac activity through a skin contact site. A pulse sensor (not shown) may be used to measure cardiovascular pulsations, or an oximetry sensor 858 may be used to measure oxygen saturation levels. A thermistor or other sensor may measure of respiratory air flow, e.g., through the user's nose. A thermister, thermocouple, or other temperature sensor (not shown) may be provided for measuring the user's skin temperature. A sweat detector (not shown) may be provided for measuring moisture on the user's skin.

In addition, the system 810 may include one or more feedback devices on the frame 812. These devices may provide feedback to the user, e.g., to alert and/or wake the user, when a predetermined condition is detected, e.g., a state of drowsiness or lack of consciousness. The feedback devices may be coupled to the processor 842, which may control their activation. For example, a mechanical vibrator device 860 may be provided at a location that may contact the user, e.g., on the ear support 816, that may provide tactile vibrating stimuli through skin contact. An electrode (not shown) may be provided that may produce relatively low power electrical stimuli. A light emitter, such as one or more LED's may provided at desired locations, e.g., above the bridge piece 814. Alternatively, audio devices 862, such as a buzzer or other alarm, may be provided, similar to the previous embodiments. In a further alternative, aroma-emitters may be provided on the frame 810 812, e.g., on or adjacent to the bridge piece 814.

Alternatively, the feedback devices may be provided separate from the frame, but located in a manner capable of providing a feedback response to the user. For example, audio, visual, tactile (e.g., vibrating seat), or olfactory emitters may be provided in the proximity of the user, such as any of the devices described above. In a further alternative, heat or cold generating devices may be provided that are capable of producing thermal stimuli to the user, e.g., a remotely controlled fan or air conditioning unit.

The system 810 may also include components that are remote from the frame 812, similar to the embodiments described above. For example, the system 810 may include a receiver, a processor, and/or a display (not shown) at a remote location from the frame 812, e.g., in the same room, at a nearby monitoring station, or at a more distant location. The receiver may receive signals transmitted by the transmitter 842, including output signals from the sensors 822 or any of the other sensors provided on the frame 812 and/or the video signals from the camera 830.

A processor may be coupled to the receiver for analyzing signals from the components on the frame 812, e.g., to prepare the signals for graphical display. For example, the processor may prepare the video signals from the camera 830 for display on a monitor, thereby allowing personal monitoring of the user. Simultaneously, other parameters may be displayed, either on a single monitor or on separate displays. For example, FIG. 15a FIGS. 15A-15I shows signals indicating the output of various sensors that may be on the frame 812, which may be displayed along a common time axis or otherwise correlated, e.g., to movement of the user's eye and/or level of drowsiness. The processor may superimpose or otherwise simultaneously display the video signal in conjunction with the other sensed parameters to allow a physician or other individual to monitor and personally correlate these parameters to the user's behavior.

In a further alternative, the processor may automatically process the signals to monitor or study the user's behavior. For example, the processor may use the output signals to monitor various parameters related to eye movement, such as eye blink duration (EBD), eye blink frequency, eye blink velocity, eye blink acceleration, interblink duration (IBD), PERCLOS, PEROP (percentage eyelid is open), and the like.

The video signals from the camera 830 may be processed to monitor various eye parameters, such as pupillary size, location, e.g., within the four quadrant defined by the crossed bands 850, eye tracking movement, eye gaze distance, and the like. For example, because the camera 830 is capable of detecting the light emitted by the emitters 822, the camera 830 may detect a reference frame projected onto the region of the user's eye by the emitters. FIG. 16 shows an exemplary video output from a camera included in a system having twenty emitters disposed in a vertical arrangement. The camera may detect twenty discrete regions of light arranged as a vertical band. The camera may also detect a “glint” point, G, and/or a moving bright pupil, P. Thus, the movement of the pupil may be monitored in relation to the glint point, G, and/or in relation to the vertical band 1-20.

Because the emitters 822 are fixed to the frame 812, the reference frame 850 remains substantially stationary. Thus, the processor may determine the location of the pupil in terms of orthogonal coordinates (e.g., x-y or angle-radius) relative to the reference frame 850. Alternatively, if the reference frame is eliminated, the location of the pupil may be determined relative to any stationary “glint” point on the user's eye. For example, the camera 830 itself may project a point of light onto the eye that may be reflected and detected by the camera. This “glint” point remains substantially stationary since the camera 830 is fixed to the frame 812.

In addition, the video signals from a remote camera that may view the user's face from a distance may be used to monitor various facial measures, such as facial expression, yawning frequency, and the like, in addition to or alternatively, the project instead of the projected light reference frame from the emitters. In addition or alternatively, the parameters from other sensors may be processed and correlated, such as head orientation, tilt, body movement, physiological parameters, and the like. Preferably, the processor may correlate these parameters to generate a composite fatigue index (CFI) that is a function of two or more of these parameters. When a predetermined CFI is detected, the system 810 may activate an alarm or other notice to the user and/or to another party at a remote location. Thus, the system 810 may provide a more effective way to monitor the user's fatigue, drowsiness, alertness, mental state, and the like. In a further alternative, the system 810 may be used to generate predetermined outputs, e.g., to activate or deactivate equipment, such as a vehicle being operated by the user when a predetermined condition, e.g., CFI value, is determined by the system 810.

Alternatively, the processor may be provided on the frame 812, e.g. as part of processor 842, for monitoring the parameters for a predetermined event, such as a predetermined CFI value, to occur. Although only a single lens and set of emitters, sensors, and cameras are shown, it will be appreciated that another set may be provided for the other eye of the user of the system 810. In a further alternative, the eye tracking parameters described above may be monitored by a remote camera, e.g., in a fixed position in front of the user, such as the dashboard of a vehicle and the like. The remote camera may be coupled to the processor, either directly or via its own transmitter, as will be appreciated by those skilled in the art.

Thus, a system in accordance with the present invention may monitor or detect one or more parameters, such as those listed below in Table 1.

TABLE 1

Potential Biometric Measures


EYELID MEASURES            EYEGLAZE MEASURES
Percentage of time (t) and Eye Tracking Movements (ETM)
the amount palpebral       including Directional Nystagmus
fissure is opened          Eye Gaze Distance (EGD) and
(PEROP-t, -d, -dt),        Direction
or closed                  Eye Movement Distance
(PERCLOS-t, -d, -dt),      Eye Movement Velocity (EMV)
lid droop                  Eye Movement Acceleration (EMA)
Eye Blink Duration (EBD)   and Deceleration (EMD)
Eye Blink Frequency (EBF)  Eye Movement Frequency (EMF)
Eye Blink Velocity (EBV)   Phoria/eye Drift Measures (PDM)
Eye Blink Acceleration     HEAD ORIENTATION MEASURES
(EBAc)                     Head Direction or Orientation
Decceleration (EBDc)       (HDir)
Interblink duration (IBD)  HEAD MOVEMENT MEASURES
Eye blink flurries         Head Nodding Frequency (HNF)
PUPIL MEASURES             Head Tilt (HT)
Pupillary Appearance or    OTHER NON-VIDEO SENSOR METRICS
Disappearance (with eyelid EEG, EKG, pulse, oxygen
movement)                  saturation, respiration rate,
Pupillary Size Measurement body temp, skin conductance,
(PSM)                      actigraphic movements, head
Presence and quality of    tilt sensors
Pupillary
Dilation or Construction
(including Hippus)

While the invention is susceptible to various modifications, and alternative forms, specific examples thereof have been shown in the drawings and are herein described in detail. It should be understood, however, that the invention is not to be limited to the particular forms or methods disclosed, but to the contrary, the invention is to cover all modifications, equivalents and alternatives falling within the spirit and scope of the appended claims.

Claims

1. A system for monitoring movement of a person's eye, comprising:

a device configured to be worn on a person's head;

an array of emitters on the device for directing light towards an eye of the person when the device is worn, the array of emitters configured for projecting a reference frame towards the eye ; and

a camera oriented towards the eye for monitoring movement of the eye relative to the reference frame mounted on the device; and

one or more sensors on the device for detecting light from the array of emitters that is reflected off of the eye or its eyelid, the one of more sensors producing an output signal indicating when the eye is open or closed.

2. The system of claim 1, wherein the one or more sensors comprise an array of sensors in a predetermined relationship with the array of emitters for detecting light from the array of emitters that is reflected off of respective portions of the eye or its eyelid, each sensor producing an output signal indicating when the respective portion of the eye is covered or not covered by the eyelid.

3. The system of claim 1, wherein the array of emitters and the one or more sensors are disposed separately and substantially laterally from one another.

4. The system of claim 1, wherein the array of emitters and the one or more sensors comprise solid state devices capable of operating both as an emitter and as a sensor.

5. The system of claim 1, wherein the camera is configured for producing a video signal, and wherein the system further comprises a processor for correlating the output signal from the one or more sensors with the video signal from the camera for determining the person's level of drowsiness.

6. The system of claim 5, further comprising a warning indicator on the device, the warning indicator being activated when the processor determines a predetermined level of drowsiness has occurred.

7. The system of claim 1, wherein the array of emitters comprises a plurality of emitters disposed in a substantially vertical arrangement on the device.

8. The system of claim 7, wherein the array of emitters further comprises a plurality of emitters disposed in a substantially horizontal arrangement on the device.

9. The system of claim 1, wherein the array of emitters is configured for projecting a set of crossed bands towards the eye for dividing a region including the eye into four quadrants.

10. The system of claim 1, further comprising a transmitter on the device for wireless transmission of video output signals from the camera to a remote location.

11. The system of claim 1, wherein the array of emitters comprise infrared emitters configured to emit pulses of infrared light.

12. The system of claim 11, wherein the camera comprises an infrared camera.

13. The system of claim 1, wherein the camera is mounted on the device.

14. The system of claim 13 1, wherein the camera comprises a fiberoptic assembly.

15. The system of claim 13 1, wherein the camera comprises at least one of a CCD and CMOS detector.

16. The system of claim 1, further comprising a sensor on the device for detecting one or more physiological characteristics of the person.

17. The system of claim 16, wherein the sensor comprises at least one of an EEG electrode, an EKG electrode, an oximetry sensor, a pulse sensor, an airflow senior, and a temperature sensor.

18. The system of claim 1, further comprising at least one of an orientation sensor for detecting the spatial orientation of the device and an actigraphic sensor.

19. The system of claim 1, wherein the device comprises at least one of an eyeglass frame, a hat, a helmet, a visor, and a mask.

20. A system for monitoring movement of a person's eye, comprising:

a frame configured to be worn on a person's head;

an array of emitters on the frame for directing light towards an eye of the person when the frame is worn, the array of emitters configured to project a reference frame towards the eye;

an array of sensors on the frame in a predetermined relationship with the array of emitters for detecting light from the array of emitters that is reflected off of respective portions of the eye or its eyelid, each sensor producing an output signal indicating when the respective portion of the eye is covered or not coveted by the eyelid;

a camera on the frame for monitoring movement of the eye relative to the reference frame, the camera configured for producing a video signal of a region of the eye and the reference frame; and

a transmitter coupled to the sensor for wireless transmission of the output signal and the video signal to a remote location.

21. The system of claim 20, further comprising a processor for correlating the output signal and the video signal to determine the person's level of drowsiness.

22. The system of claim 21, further comprising a display for providing a graphical output of the output signal simultaneous with the video signal.

23. A method for monitoring movement of a person's eye using a detection device including an array of emitters that are directed towards an eye of the person when the detection device is worn, and a camera oriented towards the eye, the method comprising:

emitting light from the array of emitters towards the eye to project a reference frame onto the eye;

monitoring movement of the eye relative to the reference frame with the camera; and

generating a graphical output of the movement monitored by the camera relative to the reference frame;

wherein the detection device further comprises one or more sensors, and wherein the method further comprises detecting light from the array of emitters reflected off of the eye with the one or more sensors, the one or more sensors producing a light intensity signal indicating when the eye is open or closed.

24. The method of claim 23, wherein the one or more sensors comprise an array of sensors is disposed in a predetermined relationship with the array of emitters for detecting light from the array of emitters that is reflected off of respective portions of the eye or its eyelid, each sensor producing an output signal indicating when the respective portion of the eye is covered or not covered by the eyelid.

25. The method of claim 24, further comprising correlating the output signal from the one or more sensors with video signals produced by the camera monitoring movement of the eye relative to the reference frame, thereby determining the person's level of alertness.

26. The method of claim 23, wherein the monitoring step comprises measuring movement of the eye's pupil relative to the reference frame.

27. The method of claim 26, further comprising graphically displaying the movement of the eye's pupil relative to the reference frame.

28. The method of claim 25, further comprising providing a warning to the person when the determined level of alertness falls below a predetermined level.

29. The system of claim 1, wherein the camera is oriented towards the eye for monitoring movement of the eye.

30. The system of claim 1, wherein the camera is oriented away from the person to monitor the person's surroundings.

31. A system for monitoring movement of a person's eye, comprising:

a device configured to be worn on a person's head;

one or more emitters on the device for directing light towards an eye of the person when the device is worn;

a sensor on the device comprising a bundle of fiberoptic cables terminating in a lens directed towards the eye of the person when the device is worn, the sensor configured for converting images of the eye into output signals; and

a processor coupled to the sensor for monitoring one or more eye movement parameters of the eye based upon the output signals.

32. The system of claim 31, further comprising a display coupled to the processor for displaying the images of the eye together with the one or more eye movement parameters of the eye.

33. The system of claim 32, further comprising one or more sensors on the device for monitoring one or more physiological parameters, wherein the processor is coupled to the one or more sensors for displaying the one or more physiological parameters on the display.

34. A system for monitoring movement of a person's eye, comprising:

a device configured to be worn on a person's head;

one or more emitters on the device for directing light towards an eye of the person when the device is worn;

a bundle of fiberoptic cables comprising a first end mounted to the device for detecting light that is reflected off of the eye or its eyelid; and

a sensor coupled to a second end of the bundle for converting images of the eye from the first end of the bundle of fiberoptic cables into output signals.

35. The system of claim 34, wherein the one or more emitters are carried by the fiberoptic bundle.

36. The system of claim 34, further comprising a camera on the device that is oriented away from the person to monitor the person's surroundings.

37. The system of claim 34, further comprising a processor coupled to the sensor, the processor configured for monitoring one or more eye movement parameters of the person's eye based upon the one or more output signals.

38. The system of claim 37, further comprising one or more sensors on the device for detecting one or more physiological characteristics of the person, the processor further configured for correlating the one or more physiological characteristics with the output signals to generate a composite fatigue index of the person wearing the device.