System and method for monitoring eye movement

Abstract

Apparatus for monitoring movement of a person's eye, e.g., to monitor drowsiness. The system includes a frame that is worn on a person's head, an array of emitters on the frame for directing light towards the person's eye, and an array of sensors on the frame for detecting light from the array of emitters. The sensors detect light that is reflected off of respective portions of the eye or its eyelid, thereby producing output signals indicating when the respective portions of the eye is covered by the eyelid. The emitters project a reference frame towards the eye, and a camera on the frame monitors movement of the eye relative to the reference frame. This movement may be correlated with the signals from the array of sensors and/or with signals from other sensors on the frame to monitor the person's level of drowsiness.

Description

This application is f e. The user 10 may blink to create a transmitted stream of data 553 that includes commands to turn off and on, or otherwise control, selected appliances using the control unit 550 and control modules 552a-552f e, such as, a radio 554, a television 556, a light 558a. , a light 562 controlled by a wall switch 560, a fan 566 plugged into a wall socket 564, and the like.

Alternatively, as shown in FIG. 11B, the receiver 554 may be coupled to other systems, such as a computer 570 and printer 572, a vehicle integration system 574, a lifeline unit 576, a GPS or other satellite transmitter 578, and the like. The transmitted stream of data 553 may be processed alone or along with additional data, such as other vehicle sensor information 573, to further enhance monitoring a user, such as a long-distance truck driver.

Turning to FIG. 13, yet another embodiment of a system 810 for monitoring eye movement is shown. Generally, the system 810 includes a frame 812 that may include a bridge piece 814 and a pair of car supports 816. The frame 812 may include a pair of lenses (not shown), such as prescription, shaded, or protective lenses, although they are not necessary for operation of the invention. Alternatively, the system may be provided on other devices that may be worn on a user's head, such as a pilot's oxygen mask, protective eye gear, a patient's ventilator, a scuba or swimming mask, a helmet, a hat, a head band, a head visor, and the like (not shown). The components of the system may be provided at a variety of locations on the device that generally minimize interference with the user's vision and/or normal use of the device.

An array of emitters 820 are provided on the frame 812, preferably in a vertical array 820a and a horizontal array 820b. In a preferred embodiment, the emitters 820 are infrared emitters configured to emit pulses at a predetermined frequency, similar to the embodiments described above. The emitters 820 are arranged on the frame such that they project a reference frame 850 onto the region of the user's eye. In a preferred embodiment, the reference frame includes a pair of crossed bands 850a, 850b dividing the region into four quadrants. The intersection of the crossed bands is preferably disposed at a location corresponding substantially to the eye's pupil during primary gaze, i.e., when the user is looking generally straight forward. Alternatively, other reference frames may be provided, generally including a vertical component and a horizontal component.

An array of sensors 822 are also provided on the frame 812 for detecting light from the emitters 820 that is reflected off of the user's eyelid. The sensors 822 preferably generate output signals having an intensity identifying whether the eyelid is closed or open, similar to the embodiments described above. Preferably, the sensors 822 are disposed adjacent to respective emitters 820 for detecting light reflected off of respective portions of the eyelid. Alternatively, sensors 822 may only be provided in a vertical array, e.g., along the bridge piece 814, for monitoring the amount of eyelid closure, similar to the embodiments described above. In a further alternative, the emitters 820 and sensors 822 may be solid state biosensors (not shown) that provide both the emitting and sensing functions in a single device.

Circuitry may be provided for measuring PERCLOS or other parameters using the signals generated by the array of sensors. For example, FIG. 17 shows an exemplary schematic that may be used for processing signals from a five element array, e.g., to obtain PERCLOS measurements or other alertness parameters.

Returning to FIG. 13, the system 810 also includes a camera 830 provided on the frame 810 812. Preferably, the camera 830 is mounted on or adjacent the bridge piece 814 such that the camera 830 is oriented towards the region surrounding one of the user's eyes while minimizing interference with the user's vision. This camera 830 preferably includes a bundle of fiberoptic cables 832 that terminate in a lens 834, as shown in FIG. 14, on a first end mounted adjacent the bridge piece 814 and a second end 837 that is connected to a detector 838, e.g., a CCD or CMOS sensor, such as those used in endoscopes, that may convert an image into a digital video signal. The camera 830 is configured to detect the frequency of light emitted by the emitters 820, e.g., infrared light. The camera 830 may rely on the light projected by the emitters 820, or the fiberoptic cables 832 may include emitters 836 for projecting light, e.g., infrared light, onto the user's eyes and/or face. In addition, the system 810 may include a second camera 840 oriented away from the user's head, e.g., to monitor the user's surroundings.

One of the ear supports 816 may include a panel 818 for mounting a controller or other processor 842, a transmitter 844, an antenna 845, and a battery 846. Preferably, the processor 840 842 is coupled to the emitters 820, the sensors 822, and/or the camera 830 for controlling their operation. The transmitter 844 may be coupled to the processor 842 for receiving the output signals from the sensors 822 and/or the video signals from the camera 830, e.g., to transmit the signals to a remote location, as described below. Alternatively, the transmitter 844 may be coupled directly to output leads from the sensors 822 and the camera 830. The frame 812 may also include manual controls (not shown), e.g., on the ear support 816, for example, to turn the power off and on, or to adjust the intensity and/or threshold of the emitters 820, the sensors 822, and/or the camera 830.

If desired, the system 810 may also include one or more additional sensors on the frame 812. The sensors may be coupled to the processor 842 and/or to the transmitter 844 so that the signals from the sensors may be monitored, recorded, and/or transmitted to a remote location. For example, one or more position sensors 852a, 852b may be provided, e.g., for determining the spatial orientation of the frame 812, and consequently the user's head. For example, actigraphic sensors may be provided to measure tilt or movement of the head, e.g., to monitor whether the user's head is drooping forward or tilting to the side. Acoustic sensors, e.g., a microphone 854 may be provided for detecting environmental noise or sounds produced by the user.

In addition or alternatively, the frame 812 may include one or more sensors for measuring one or more physical characteristics of the user. For example, EEG electrodes 856 may be provided on the ear support 816, above or below the nasion, and/or other region that may contact the patient's skin to measure brain activity, e.g., waking, drowsy, or other sleep-related brain activity. An EKG electrode (not shown) may be provided that is capable of measuring cardiac activity through a skin contact site. A pulse sensor (not shown) may be used to measure cardiovascular pulsations, or an oximetry sensor 858 may be used to measure oxygen saturation levels. A thermistor or other sensor may measure of respiratory air flow, e.g., through the user's nose. A thermistor, thermocouple, or other temperature sensor (not shown) may be provided for measuring the user's skin temperature. A sweat detector (not shown) may be provided for measuring moisture on the user's skin.

In addition, the system 810 may include one or more feedback devices on the frame 812. These devices may provide feedback to the user, e.g., to alert and/or wake the user, when a predetermined condition is detected, e.g., a state of drowsiness or lack of consciousness. The feedback devices may be coupled to the processor 842, which may control their activation. For example, a mechanical vibrator device 860 may be provided at a location that may contact the user, e.g., on the ear support 816, that may provide tactile vibrating stimuli through skin contact. An electrode (not shown) may be provided that may produce relatively low power electrical stimuli. A light emitter, such as one or more LED's may provided at desired locations, e.g., above the bridge piece 814. Alternatively, audio devices 862, such as a buzzer or other alarm, may be provided, similar to the previous embodiments. In a further alternative, aroma-emitters may be provided on the frame 810 812, e.g., on or adjacent in the bridge piece 814.

Alternatively, the feedback devices may be provided separate from the frame, but located in a manner capable of providing a feedback response to the user. For example, audio, visual, tactile (e.g., vibrating seat), or olfactory emitters may be provided in the proximity of the user, such as any of the devices described above. In a further alternative, heat or cold generating devices may be provided that are capable of producing thermal stimuli to the user, e.g., a remotely controlled fan or air conditioning unit.

The system 810 may also include components that are remote from the frame 812, similar to the embodiments described above. For example, the system 810 may include a receiver, a processor, and/or a display (not shown) at a remote location from the frame 812, e.g., in the same room, at a nearby monitoring station, or at a more distant location. The receiver may receive signals transmitted by the transmitter 842, including output signals from the sensors 822 or any of the other sensors provided on the frame 812 and/or the video signals from the camera 830.

A processor may be coupled to the receiver for analyzing signals from the components on the frame 812, e.g., to prepare the signals for graphical display. For example, the processor may prepare the video signals from the camera 830 for display on a monitor, thereby allowing personal monitoring of the user. Simultaneously, other parameters may be displayed, either on a single monitor or on separate displays. For example, FIG. 15a FIGS. 15A-15I shows signals indicating the output of various sensors that may be on the frame 812, which may be displayed along a common time axis or otherwise correlated, e.g., to movement of the user's eye and/or level of drowsiness. The processor may superimpose or otherwise simultaneously display the video signal in conjunction with the other sensed parameters to allow a physician or other individual to monitor and personally correlate these parameters to the user's behavior.

In a further alternative, the processor may automatically process the signals to monitor or study the user's behavior. For example, the processor may use the output signals to monitor various parameters related to eye movement, such as eye blink duration (EBD), eye blink frequency, eye blink velocity, eye blink acceleration, interblink duration (IBD), PERCLOS, PEROP (percentage eyelid is open), and the like.

The video signals from the camera 830 may be processed to monitor various eye parameters, such as pupillary size, location, e.g., within the four quadrant defined by the crossed bands 850, eye tracking movement, eye gaze distance, and the like. For example, because the camera 830 is capable of detecting the light emitted by the emitters 822, the camera 830 may detect a reference frame projected onto the region of the user's eye by the emitters. FIG. 16 shows an exemplary video output from a camera included in a system having twenty emitters disposed in a vertical arrangement. The camera may detect twenty discrete regions of light arranged as a vertical band. The camera may also detect a “glint” point, G, and/or a moving bright pupil, P. Thus, the movement of the pupil may be monitored in relation to the glint point, G, and/or in relation to the vertical band 1-20.

Because the emitters 822 are fixed to the frame 812, the reference frame 850 remains substantially stationary. Thus, the processor may determine the location of the pupil in terms of orthogonal coordinates (e.g., x-y or angle-radius) relative to the reference frame 850. Alternatively, if the reference frame is eliminated, the location of the pupil may be determined relative to any stationary “glint” point on the user's eye. For example, the camera 830 itself may project a point of light onto the eye that may be reflected and detected by the camera. This “glint” point remains substantially stationary since the camera 830 is fixed to the frame 812.

In addition, the video signals from a remote camera that may view the user's face from a distance may be used to monitor various facial measures, such as facial expression, yawning frequency, and the like, in addition to or alternatively, the project instead of the projected light reference frame from the emitters. In addition or alternatively, the parameters from other sensors may be processed and correlated, such as head orientation, tilt, body movement, physiological parameters, and the like. Preferably, the processor may correlate these parameters to generate a composite fatigue index (CFI) that is a function of two or more of these parameters. When a predetermined CFI is detected, the system 810 may activate an alarm or other notice to the user and/or to another party at a remote location. Thus, the system 810 may provide a more effective way to monitor the user's fatigue, drowsiness, alertness, mental state, and the like. In a further alternative, the system 810 may be used to generate predetermined outputs, e.g., to activate or deactivate equipment, such as a vehicle being operated by the user when a predetermined condition, e.g., CFI value, is determined by the system 810.

Alternatively, the processor may be provided on the frame 812, e.g. as part of processor 842, for monitoring the parameters for a predetermined event, such as a predetermined CFI value, to occur. Although only a single lens and set of emitters, sensors, and cameras are shown, it will be appreciated that another set may be provided for the other eye of the user of the system 810. In a further alternative, the eye tracking parameters described above may be monitored by a remote camera, e.g., in a fixed position in front of the user, such as the dashboard of a vehicle and the like. The remote camera may be coupled to the processor, either directly or via its own transmitter, as will be appreciated by those skilled in the art.

Thus, a system in accordance with the present invention may monitor or detect one or more parameters, such as those listed below in Table 1.

TABLE 1

Potential Biometric Measures


EYELID MEASURES
Percentage of time (t) and the amount palpebral
fissure is opened (PEROP-t, -d, -dt), or closed
(PERCLOS-t, -d, -dt),
lid droop
Eye Blink Duration (EBD)
Eye Blink Frequency (EBF)
Eye Blink Velocity (EBV)
Eye Blink Accelaration (EBAc)
Deceleration (EBDc)
Interblink duration (IBD)
Eye blink flurries
PUPIL MEASURES
Pupillary Appearance or Disappearance (with
eyelid movement)
Pupillary Size Measurement (PSM)
Presence and quality of Pupillary
Dilation or Construction
(including Hippus)
EYEGLAZE MEASURES
Eye Tracking Movements (ETM)
including Directional Nystagmus
Eye Gaze Distance (EGD) and Direction
Eye Movement Distance
Eye Movement Velocity (EMV)
Eye Movement Acceleration (EMA)
and Deceleration (EMD)
Eye Movement Frequency (EMF)
Phoria/eye Drill Measures (PDM)
HEAD ORIENTATION MEASURES
Head Direction or Orientation (HDir)
HEAD MOVEMENT MEASURES
Head Nodding Frequency (HNF)
Head Tilt (HT)
OTHER NON-VIDEO SENSOR METRICS
EEG, EKG, pulse, oxygen saturation,
respiration rate, body temp, skin
conductance, actigraphic movements, head
tilt sensors

While the invention is susceptible to various modifications, and alternative forms, specific examples thereof have been shown in the drawings and are herein described in detail. It should be understood, however, that the invention is not to be limited to the particular forms or methods disclosed, but to the contrary, the invention is to cover all modifications, equivalents and alternatives falling within the spirit and scope of the appended claims.

Claims

1. A system for monitoring movement of a person's eye, comprising:

a device configured to be worn on a person's head;

an array of one or more emitters on the device for directing light towards an eye of the person when the device is worn, the array of emitters configured for projecting a reference frame towards the eye ; and

a camera oriented towards the eye for monitoring movement of the eye relative to the reference frame mounted on the device; and

one or more sensors on the device for detecting light from the array of one or more emitters that is reflected off of the eye or its eyelid, the one or more sensors producing an output signal indicating when the eye is open or closed.

2. The system of claim 1, wherein the one or more sensors comprise an array of sensors in a predetermined relationship with the array of one or more emitters for detecting light from the array of one or more emitters that is reflected off of respective portions of the eye or its eyelid, each sensor producing an output signal indicating when the respective portion of the eye is covered or not covered by the eyelid.

3. The system of claim 1, wherein the array of one or more emitters and the one or more sensors are disposed separately and substantially laterally from one another.

4. The system of claim 1, wherein the array of one or more emitters and the one or more sensors comprise solid state devices capable of operating both as an emitter and as a sensor.

5. The system of claim 1, wherein the camera is configured for producing a video signal, and wherein the system further comprises a processor for correlating the output signal from the one or more sensors with the video signal from the camera for determining the person's level of drowsiness.

6. The system of claim 5, further comprising a warning indicator on the device, the warning indicator being activated when the processor determines a predetermined level of drowsiness has occurred.

7. The system of claim 1, wherein the array of one or more emitters comprises a plurality of emitters disposed in a substantially vertical arrangement on the device.

8. The system of claim 7, wherein the array of one or more emitters further comprises a plurality of emitters disposed in a substantially horizontal arrangement on the device.

9. The system of claim 1, wherein the array of one or more emitters is comprise an array of emitters configured for projecting a set of crossed bands towards the eye for dividing a region including the eye into four quadrants.

10. The system of claim 1, further comprising a transmitter on the device for wireless transmission of video output signals from the camera to a remote location.

11. The system of claim 1, wherein the array of one or more emitters comprise infrared emitters configured to emit pulses of infrared light.

12. The system of claim 11, wherein the camera comprises an infrared camera.

13. The system of claim 1, wherein the camera is mounted on the device.

14. The system of claim 13 1, wherein the camera comprises a fiberoptic assembly.

15. The system of claim 13 1, wherein the camera comprises at least one of a CCD and CMOS detector.

16. The system of claim 1, further comprising a sensor on the device for detecting one or more physiological characteristics of the person.

17. The system of claim 16, wherein the sensor comprises at least one of a EEG electrode, an EKG electrode, an oximetry sensor, a pulse sensor, an airflow sensor, and a temperature sensor.

18. The system of claim 1, further comprising at least one of an orientation sensor for detecting the spatial orientation of the device and an actigraphic sensor.

19. The system of claim 1, wherein the device comprises at least one of an eyeglass frame, a hat, a helmet, a visor, and a mask.

20. A system for monitoring movement of a person's eye, comprising:

a frame configured to be worn on a person's head;

an array of emitters on the frame for directing light towards an eye of the person when the device is worn, the array emitters configured to project a reference frame towards the eye;

an array of sensors on the frame in a predetermined relationship with the array of emitters for detecting light from the array of emitters that is reflected off of respective portions of the eye or its eyelid, each sensor producing an output signal indicating when the respective portion of the eye is covered or not covered by the eyelid; and

a camera on the frame for monitoring movement of the eye relative to the reference frame, the camera configured for producing a video signal of a region of the eye and the reference frame; and

a transmitter coupled to the sensor for wireless transmission of the output signal and the video signal to a remote location .

21. The system of claim 20, further comprising a processor for correlating the output signal and the video signal to determine the person's level of drowsiness.

22. The system of claim 21, further comprising a display for providing a graphical output of the output signal simultaneous with the video signal.

23. A method for monitoring movement of a person's eye using a detection device including an array of emitters that are directed towards an eye of the person when the detection device is worn, and a camera oriented towards the eye, the method comprising:

emitting light from the array of emitters towards the eye to project a reference frame onto the eye;

monitoring movement of the eye relative to the reference frame with the camera; and

generating a graphical output of the movement monitored by the camera relative to the reference frame;

wherein the detection device further comprises one or more sensors, and wherein the method further comprises detecting light from the array of emitters reflected off of the eye with the one or more sensors, the one or more sensors producing a light intensity signal indicating when the eye is open or closed.

24. The method of claim 23, wherein the array of sensors is disposed in a predetermined relationship with the array of emitters for detecting light from the array of emitters that is reflected off of respective portions of the eye or its eyelid, each sensor producing an output signal indicating when the respective portion of the eye is covered or not covered by the eyelid.

25. The method of claim 24, further comprising correlating the output signal from the one or more sensors with video signals produced by the camera monitoring movement of the eye relative to the reference frame, thereby determining the person's level of alertness.

26. The method of claim 23, wherein the monitoring step comprises measuring movement of the eye's pupil relative to the reference frame.

27. The method of claim 26, further comprising graphically displaying the movement of the eye's pupil relative to the reference frame.

28. The method of claim 25, further comprising providing a warning to the person when the determined level of alertness falls below a predetermined level.

29. The system of claim 1, wherein the camera is oriented towards the eye for monitoring movement of the eye.

30. The system of claim 1, wherein the camera is oriented away from the person to monitor the person's surroundings.

31. A system for monitoring movement of a person's eye, comprising:

a device configured to be worn on a person's head;

one or more emitters on the device for directing light towards an eye of the person when the device is worn;

an imaging sensor on the device comprising a lens directed towards the eye of the person when the device is worn, the sensor configured for converting images of the eye into output signals;

a processor coupled to the sensor for monitoring one or more eye movement parameters of the eye based upon the output signals; and

one or more physiological sensors on the device for monitoring one or more physiological parameters.

32. The system of claim 31, further comprising a display coupled to the imaging sensor and the one or more physiological sensors for displaying the one or more eye movement parameters of the eye and the one or more physiological parameters.

33. The system of claim 31, wherein the imaging sensor further comprises a bundle of fiberoptic cables comprising a first end terminating in the lens.

34. A system for monitoring movement of a person's eye, comprising:

a device configured to be worn on a person's head;

one or more emitters on the device for directing light towards an eye of the person when the device is worn;

a two-dimensional array of sensors on the device directed towards the eye of the person when the device is worn, the sensors configured for converting images of the eye into output signals;

a processor coupled to the sensors for monitoring one or more eye movement parameters of the eye based upon the output signals; and

one or more sensors on the device for detecting one or more physiological characteristics of the person, the processor further configured for correlating the one or more physiological characteristics with the output signals to generate a composite fatigue index of the person wearing the device.

35. The system of claim 34, further comprising a camera on the device that is oriented away from the person to monitor the person's surroundings.

36. A system for monitoring movement of a person's eye, comprising:

a device configured to be worn on a person's head;

one or more emitters on the device for directing light towards an eye of the person when the device is worn;

one or more sensors on the device for detecting light from the one or more emitters that is reflected off of the eye or its eyelid, the sensors configured for converting images of the eye into output signals;

one or more position sensors for determining a spatial orientation of the device when the device is worn; and

a processor coupled to the one or more sensors for monitoring one or more eye movement parameters of the eye based upon the output signals and coupled to the one or more position sensors, the processor correlating at least the spatial orientation determined using the one or more position sensors with the output signals to generate a composite fatigue index of the person wearing the device.

37. The system of claim 36, wherein the one or more position sensors comprise one or more actigraphic sensors to measure tilt or movement of the person's head when the device is worn.

38. A method for monitoring movement of a person's eye using a detection device including one or more emitters and an array of sensors that are directed towards an eye of the person when the detection device is worn, the method comprising:

emitting light from one or more emitters towards the eye;

detecting light from the one or more emitters reflected off of the eye with the array of sensors, the array of sensors producing light intensity signals indicating when the eye is open or closed; and

monitoring one or more eye movement parameters of the eye based upon the light intensity signals,

wherein the one or more eye movement parameters are monitored to monitor a psychological condition of the person.

39. The method of claim 38, wherein the one or more eye movement parameters are monitored to detect whether the person is lying.

40. A method for monitoring movement of a person's eye using a detection device including one or more emitters and an array of sensors that are directed towards an eye of the person when the detection device is worn, the method comprising:

emitting light from one or more emitters towards the eye;

detecting light from the one or more emitters reflected off of the eye with the array of sensors, the array of sensors producing light intensity signals indicating when the eye is open or closed; and

monitoring one or more eye movement parameters of the eye based upon the light intensity signals,

wherein the one or more eye movement parameters are monitored to monitor a medical condition of the person.

41. The method of claim 40, wherein the one or more eye movement parameters are monitored to monitor a neurological condition of the person.

42. A method for monitoring movement of a person's eye using a detection device including one or more emitters and an array of sensors that are directed towards an eye of the person when the detection device is worn, the method comprising:

emitting light from one or more emitters towards the eye;

detecting light from the one or more emitters reflected off of the eye with the array of sensors, the array of sensors producing light intensity signals indicating when the eye is open or closed; and

monitoring one or more eye movement parameters of the eye based upon the light intensity signals,

wherein the one or more eye movement parameters are monitored to monitor biofeedback from the person.

43. A method for monitoring movement of a person's eye using a detection device including one or more emitters and an array of sensors that are directed towards an eye of the person when the detection device is worn, the method comprising:

emitting light from one or more emitters towards the eye;

detecting light from the one or more emitters reflected off of the eye with the array of sensors, the array of sensors producing light intensity signals indicating when the eye is open or closed; and

transmitting data signals to communicate with a remote location, the data signals comprising the light intensity signals and identification information transmitted with the light intensity signals to distinguish the detection device from a number of detection devices.

44. The method of claim 43, further comprising monitoring the person from the remote location using the light intensity signals.

45. The method of claim 44, further comprising activating or deactivating equipment when a predetermined condition is detected using the light intensity signals.

46. A method for monitoring movement of a person's eye using a detection device including one or more emitters and an array of sensors that are directed towards an eye of the person when the detection device is worn, the method comprising:

emitting light from one or more emitters towards the eye;

detecting light from the one or more emitters reflected off of the eye with the array of sensors, the array of sensors producing light intensity signals indicating when the eye is open or closed; and

transmitting the light intensity signals to communicate with a remote location;

monitoring the person from the remote location using the light intensity signals; and

activating a warning indicator when a predetermined condition is detected using the light intensity signals.