System and method for monitoring eye movement

Abstract

Apparatus for monitoring movement of a person's eye, e.g., to monitor drowsiness. The system includes a frame that is worn on a person's head, an array of emitters on the frame for directing light towards the person's eye, and an array of sensors on the frame for detecting light from the array of emitters. The sensors detect light that is reflected off of respective portions of the eye or its eyelid, thereby producing output signals indicating when the respective portions of the eye is covered by the eyelid. The emitters project a reference frame towards the eye, and a camera on the frame monitors movement of the eye relative to the reference frame. This movement may be correlated with the signals from the array of sensors and/or with signals from other sensors on the frame to monitor the person's level of drowsiness.

Description

fe. The user 10 may blink to create a transmitted stream of data 553 that includes commands to turn off and on, or otherwise control, selected appliances using the control unit 550 and control modules 552a-552fe, such as, a radio 554, a television 556, a light 558a., a light 562 controlled by a wall switch 560, a fan 566 plugged into a wall socket 564, and the like.

Alternatively, as shown in FIG. 11B, the receiver 554 may be coupled to other systems, such as a computer 570 and printer 572, a vehicle integration system 574, a lifeline unit 576, a GPS or other satellite transmitter 578, and the like. The transmitted stream of data 553 may be processed alone or along with additional data, such as other vehicle sensor information 573, to further enhance monitoring a user, such as a long-distance truck driver.

Turning to FIG. 13, yet another embodiment of a system 810 for monitoring eye movement is shown. Generally, the system 810 includes a frame 812 that may include a bridge piece 814 and a pair of ear supports 816. The frame 812 may include a pair of lenses (not shown), such as prescription, shaded, or protective lenses, although they are not necessary for operation of the invention. Alternatively, the system may be provided on other devices that may be worn on a user's head, such as a pilot's oxygen mask, protective eye gear, a patient's ventilator, a scuba or swimming mask, a helmet, a hat, a head band, a head visor, and the like (not shown). The components of the system may be provided at a variety of locations on the device that generally minimize interference with the user's vision and/or normal use of the device.

An array of emitters 820 are provided on the frame 812, preferably in a vertical array 820a and a horizontal array 820b. In a preferred embodiment, the emitters 820 are infrared emitters configured to emit pulses at a predetermined frequency, similar to the embodiments described above. The emitters 820 are arranged on the frame such that they project a reference frame 850 onto the region of the user's eye 300. In a preferred embodiment, the reference frame includes a pair of crossed bands 850a, 850b dividing the region into four quadrants. The intersection of the crossed bands is preferably disposed at a location corresponding substantially to the eye's pupil during primary gaze, i.e., when the user is looking generally straight forward along axis 310 extending directly ahead of the user's eye 300. Alternatively, other reference frames may be provided, generally including a vertical component and a horizontal component.

An array of sensors 822 are also provided on the frame 812 for detecting light from the emitters 820 that is reflected off of the user's eyelid. The sensors 822 preferably generate output signals having an intensity identifying whether the eyelid is closed or open, similar to the embodiments described above. Preferably, the sensors 822 are disposed adjacent to respective emitters 820 for detecting light reflected off of respective portions of the eyelid. Alternatively, sensors 822 may only be provided in a vertical array, e.g., along the bridge piece 814, for monitoring the amount of eyelid closure, similar to the embodiments described above. In a further alternative, the emitters 820 and sensors 822 may be solid state biosensors (not shown) that provide both the emitting and sensing functions in a single device.

Circuitry may be provided for measuring PERCLOS or other parameters using the signals generated by the array of sensors. For example, FIG. 17 shows an exemplary schematic that may be used for processing signals from a five element array, e.g., to obtain PERCLOS measurements or other alertness parameters.

Returning to FIG. 13, the system 810 also includes a camera 830 provided on the frame 810. Preferably, the camera 830 is mounted on or adjacent the bridge piece 814 offset from the axis 310 such that the camera 830 is oriented towards the region surrounding one of the user's eyes 300 while minimizing interference with the user's vision. The camera 830 preferably includes a bundle of fiberoptic cables 832 that terminate in a lens 834, as shown in FIG. 14, on a first end mounted adjacent the bridge piece 814 and a second end 837 that is connected to a detector 838, e.g., a CCD or CMOS sensor, such as those used in endoscopes, that may convert an image into a digital video signal. The camera 830 is configured to detect the frequency of light emitted by the emitters 820, e.g., infrared light. The camera 830 may rely on the light projected by the emitters 820, or the fiberoptic cables 832 may include emitters 836 for projecting light, e.g., infrared light, onto the user's eyes and/or face. In addition, the system 810 may include a second camera 840 oriented away from the user's head, e.g., to monitor the user's surroundings.

One of the ear supports 816 may include a panel 818 for mounting a controller or other processor 842, a transmitter 844, an antenna 845, and a battery 846. Preferably, the processor 840 842 is coupled to the emitters 820, the sensors 822, and/or the camera 830 for controlling their operation. The transmitter 844 may be coupled to the processor 842 for receiving the output signals from the sensors 822 and/or the video signals from the camera 830, e.g., to transmit the signals to a remote location, as described below. Alternatively, the transmitter 844 may be coupled directly to output leads from the sensors 822 and the camera 830. The frame 812 may also include manual controls (not shown), e.g., on the ear support 816, for example, to turn the power off and on, or to adjust the intensity and/or threshold of the emitters 820, the sensors 822, and/or the camera 830.

If desired, the system 810 may also include one or more additional sensors on the frame 812. The sensors may be coupled to the processor 842 and/or to the transmitter 844 so that the signals from the sensors may be monitored, recorded, and/or transmitted to a remote location. For example, one or more position sensors 852a, 852b may be provided, e.g., for determining the spatial orientation of the frame 812, and consequently the user's head. For example, actigraphic sensors may be provided to measure tilt or movement of the head, e.g., to monitor whether the user's head is drooping forward or tilting to the side. Acoustic sensors, e.g., a microphone 854 may be provided for detecting environmental noise or sounds produced by the user.

In addition or alternatively, the frame 812 may include one or more sensors for measuring one or more physical characteristics of the user. For example, EEG electrodes 856 may be provided on the ear support 816, above or below the nasion, and/or other region that may contact the patient's skin to measure brain activity, e.g., waking, drowsy, or other sleep-related brain activity. An EKG electrode (not shown) may be provided that is capable of measuring cardiac activity through a skin contact site. A pulse sensor (not shown) may be used to measure cardiovascular pulsations, or an oximetry sensor 858 may be used to measure oxygen saturation levels. A thermistor or other sensor may measure of respiratory air flow, e.g., through the user's nose. A thermister, thermocouple, or other temperature sensor (not shown) may be provided for measuring the user's skin temperature. A sweat detector (not shown) may be provided for measuring moisture on the user's skin.

In addition, the system 810 may include one or more feedback devices on the frame 812. These devices may provide feedback to the user, e.g., to alert and/or wake the user, when a predetermined condition is detected, e.g., a state of drowsiness or lack of consciousness. The feedback devices may be coupled to the processor 842, which may control their activation. For example, a mechanical vibrator device 860 may be provided at a location that may contact the user, e.g., on the ear support 816, that may provide tactile vibrating stimuli through skin contact. An electrode (not shown) may be provided that may produce relatively low power electrical stimuli. A light emitter, such as one or more LED's may provided at desired locations, e.g., above the bridge piece 814. Alternatively, audio devices 862, such as a buzzer or other alarm, may be provided, similar to the previous embodiments. In a further alternative, aroma-emitters may be provided on the frame 810 812, e.g., on or adjacent to the bridge piece 814.

Alternatively, the feedback devices may be provided separate from the frame, but located in a manner capable of providing a feedback response to the user. For example, audio, visual, tactile (e.g., vibrating seat), or olfactory emitters may be provided in the proximity of the user, such as any of the devices described above. In a further alternative, heat or cold generating devices may be provided that are capable of producing thermal stimuli to the user, e.g., a remotely controlled fan or air conditioning unit.

The system 810 may also include components that are remote from the frame 812, similar to the embodiments described above. For example, the system 810 may include a receiver, a processor, and/or a display (not shown) at a remote location from the frame 812, e.g., in the same room, at a nearby monitoring station, or at a more distant location. The receiver may receive signals transmitted by the transmitter 842, including output signals from the sensors 822 or any of the other sensors provided on the frame 812 and/or the video signals from the camera 830.

A processor may be coupled to the receiver for analyzing signals from the components on the frame 812, e.g., to prepare the signals for graphical display. For example, the processor may prepare the video signals from the camera 830 for display on a monitor, thereby allowing personal monitoring of the user. Simultaneously, other parameters may be displayed, either on a single monitor or on separate displays. For example, FIG. 15a FIGS. 15A-15I shows signals indicating the output of various sensors that may be on the frame 812, which may be displayed along a common time axis or otherwise correlated, e.g., to movement of the user's eye and/or level of drowsiness. The processor may superimpose or otherwise simultaneously display the video signal in conjunction with the other sensed parameters to allow a physician or other individual to monitor and personally correlate these parameters to the user's behavior.

In a further alternative, the processor may automatically process the signals to monitor or study the user's behavior. For example, the processor may use the output signals to monitor various parameters related to eye movement, such as eye blink duration (EBD), eye blink frequency, eye blink velocity, eye blink acceleration, interblink duration (IBD), PERCLOS, PEROP (percentage eyelid is open), and the like.

The video signals from the camera 830 may be processed to monitor various eye parameters, such as pupillary size, location, e.g., within the four quadrant defined by the crossed bands 850, eye tracking movement, eye gaze distance, and the like. For example, because the camera 830 is capable of detecting the light emitted by the emitters 822, the camera 830 may detect a reference frame projected onto the region of the user's eye by the emitters. FIG. 16 shows an exemplary video output from a camera included in a system having twenty emitters disposed in a vertical arrangement. The camera may detect twenty discrete regions of light arranged as a vertical band. The camera may also detect a “glint” point, G, and/or a moving bright pupil, P. Thus, the movement of the pupil may be monitored in relation to the glint point, G, and/or in relation to the vertical band 1-20.

Because the emitters 822 are fixed to the frame 812, the reference frame 850 remains substantially stationary. Thus, the processor may determine the location of the pupil in terms of orthogonal coordinates (e.g., x-y or angle-radius) relative to the reference frame 850. Alternatively, if the reference frame is eliminated, the location of the pupil may be determined relative to any stationary “glint” point on the user's eye. For example, the camera 830 itself may project a point of light onto the eye that may be reflected and detected by the camera. This “glint” point remains substantially stationary since the camera 830 is fixed to the frame 812.

In addition, the video signals from a remote camera that may view the user's face from a distance may be used to monitor various facial measures, such as facial expression, yawning frequency, and the like, in addition to or alternatively, the project instead of the projected light reference frame from the emitters. In addition or alternatively, the parameters from other sensors may be processed and correlated, such as head orientation, tilt, body movement, physiological parameters, and the like. Preferably, the processor may correlate these parameters to generate a composite fatigue index (CFI) that is a function of two or more of these parameters. When a predetermined CFI is detected, the system 810 may activate an alarm or other notice to the user and/or to another party at a remote location. Thus, the system 810 may provide a more effective way to monitor the user's fatigue, drowsiness, alertness, mental state, and the like. In a further alternative, the system 810 may be used to generate predetermined outputs, e.g., to activate or deactivate equipment, such as a vehicle being operated by the user when a predetermined condition, e.g., CFI value, is determined by the system 810.

Alternatively, the processor may be provided on the frame 812, e.g. as part of processor 842, for monitoring the parameters for a predetermined event, such as a predetermined CFI value, to occur. Although only a single lens and set of emitters, sensors, and cameras are shown, it will be appreciated that another set may be provided for the other eye of the user of the system 810. In a further alternative, the eye tracking parameters described above may be monitored by a remote camera, e.g., in a fixed position in front of the user, such as the dashboard of a vehicle and the like. The remote camera may be coupled to the processor, either directly or via its own transmitter, as will be appreciated by those skilled in the art.

Thus, a system in accordance with the present invention may monitor or detect one or more parameters, such as those listed below in Table 1.

TABLE 1

Potential Biometric Measures


EYELID MEASURES            EYEGLAZE MEASURES
Percentage of time (t) and Eye Tracking Movements (ETM)
the amount palpebral       including Directional Nystagmus
fissure is opened          Eye Gaze Distance (EGD) and
(PEROP-t, -d, -dt),        Direction
or closed                  Eye Movement Distance
(PERCLOS-t, -d, -dt),      Eye Movement Velocity (EMV)
lid droop                  Eye Movement Acceleration (EMA)
Eye Blink Duration (EBD)   and Deceleration (EMD)
Eye Blink Frequency (EBF)  Eye Movement Frequency (EMF)
Eye Blink Velocity (EBV)   Phoria/eye Drift Measures (PDM)
Eye Blink Acceleration     HEAD ORIENTATION MEASURES
(EBAc)                     Head Direction or Orientation
Decceleration (EBDc)       (HDir)
Interblink duration (IBD)  HEAD MOVEMENT MEASURES
Eye blink flurries         Head Nodding Frequency (HNF)
PUPIL MEASURES             Head Tilt (HT)
Pupillary Appearance or    OTHER NON-
Disappearance (with eyelid VIDEO SENSOR METRICS
movement)                  EEG, EKG, pulse, oxygen
Pupillary Size Measurement saturation, respiration rate,
(PSM)                      body temp, skin conductance,
Presence and quality of    actigraphic movements, head
Pupillary                  lilt sensors
Dilation or Construction
(including Hippus)

While the invention is susceptible to various modifications, and alternative forms, specific examples thereof have been shown in the drawings and are herein described in detail. It should be understood, however, that the invention is not to be limited to the particular forms or methods disclosed, but to the contrary, the invention is to cover all modifications, equivalents and alternatives falling within the spirit and scope of the appended claims.

Claims

1. A system for monitoring movement of a person's eye, comprising:

a device configured to be worn on a person's head;

an array of emitters on the device for directing light towards an eye of the person when the device is worn, the array of emitters configured for projecting a reference frame towards the eye; and

a camera oriented towards the eye for monitoring movement of the eye relative to the reference frame; and

one or more sensors on the device for detecting light from the array of emitters that is reflected off of the eye or its eyelid, the one or more sensors producing an output signal indicating when the eye is open or closed.

2. The system of claim 1, wherein the one or more sensors comprise an array of sensors in a predetermined relationship with the array of emitters for detecting light from the array of emitters that is reflected off of respective portions of the eye or its eyelid, each sensor producing an output signal indicating when the respective portion of the eye is covered or not covered by the eyelid.

3. The system of claim 1, wherein the array of emitters and the one or more sensors are disposed separately and substantially laterally from one another.

4. The system of claim 1, wherein the array of emitters and the one or more sensors comprise solid state devices capable of operating both as an emitter and as a sensor.

5. The system of claim 1, wherein the camera is configured for producing a video signal, and wherein the system further comprises a processor for correlating the output signal from the one or more sensors with the video signal from the camera for determining the person's level of drowsiness.

6. The system of claim 5, further comprising a warning indicator on the device, the warning indicator being activated when the processor determines a predetermined level of drowsiness has occurred.

7. The system of claim 1, wherein the array of emitters comprises a plurality of emitters disposed in a substantially vertical arrangement on the device.

8. The system of claim 7, wherein the array of emitters further comprises a plurality of emitters disposed in a substantially horizontal arrangement on the device.

9. The system of claim 1, wherein the array of emitters is configured for projecting a set of crossed bands towards the eye for dividing a region including the eye into four quadrants.

10. The system of claim 1, further comprising a transmitter on the device for wireless transmission of video output signals from the camera to a remote location.

11. The system of claim 1, wherein the array of emitters comprise infrared emitters configured to emit pulses of infrared light.

12. The system of claim 11, wherein the camera comprises an infrared camera.

13. The system of claim 1, wherein the camera is mounted on the device.

14. The system of claim 13, wherein the camera comprises a fiberoptic assembly.

15. The system of claim 13, wherein the camera comprises at least one of a CCD and CMOS detector.

16. The system of claim 1, further comprising a sensor on the device for detecting one or more physiological characteristics of the person.

17. The system of claim 16, wherein the sensor comprises at least one of an EEG electrode, an EKG electrode, an oximetry sensor, a pulse sensor, an airflow sensor, and a temperature sensor.

18. The system of claim 1, further comprising at least one of an orientation sensor for detecting the spatial orientation of the device and an actigraphic sensor.

19. The system of claim 1, wherein the device comprises at least one of an eyeglass frame, a hat, a helmet, a visor, and a mask.

20. A system for monitoring movement of a person's eye, comprising:

a frame configured to be worn on a person's head;

an array of emitters on the frame for directing light towards an eye of the person when the frame is worn, the array emitters configured to project a reference frame towards the eye;

an array of sensors on the frame in a predetermined relationship with the array of emitters for detecting light from the array of emitters that is reflected off of respective portions of the eye or its eyelid, each sensor producing an output signal indicating when the respective portion of the eye is covered or not covered by the eyelid;

a camera on the frame for monitoring movement of the eye relative to the reference frame, the camera configured for producing a video signal of a region of the eye and the reference frame; and

a transmitter coupled to the sensor for wireless transmission of the output signal and the video signal to a remote location.

21. The system of claim 20, further comprising a processor for correlating the output signal and the video signal to determine the person's level of drowsiness.

22. The system of claim 21, further comprising a display for providing a graphical output of the output signal simultaneous with the video signal.

23. A method for monitoring movement of a person's eye using a detection device including an array of emitters that are directed towards an eye of the person when the detection device is worn, and a camera oriented towards the eye away from the person when the detection device is worn, the method comprising:

emitting light from the array of one or more emitters towards the eye to project a reference frame onto the eye;

monitoring movement of the eye relative to the reference frame the person's surroundings with the camera; and

generating a graphical output of the movement monitored by the camera relative to the reference frame;

wherein the detection device further comprises one or more sensors, and wherein the method further comprises detecting light from the array of one or more emitters reflected off of the eye with the one or more sensors, the one or more sensors producing a light intensity signal indicating when the eye is open or closed.

24. The method of claim 23, wherein the array of sensors is disposed in a predetermined relationship with the array of one or more emitters for detecting light from the array of one or more emitters that is reflected off of respective portions of the eye or its eyelid, each sensor producing an output signal indicating when the respective portion of the eye is covered or not covered by the eyelid.

25. The method of claim 24, further comprising correlating the output signal from the one or more sensors with video signals produced by the camera monitoring movement of the eye relative to the reference frame, thereby determining the person's level of alertness.

26. The method of claim 23, wherein the monitoring step comprises measuring movement of the eye's pupil relative to the reference frame.

27. The method of claim 26, further comprising graphically displaying the movement of the eye's pupil relative to the reference frame.

28. The method of claim 25, further comprising providing a warning to the person when the determined level of alertness falls below a predetermined level.

29. A system for monitoring movement of a person's eye, comprising:

a device configured to be worn on a person's head such that the device does not interfere substantially with the person's vision along an axis extending directly ahead of a first eye of the person;

one or more emitters on the device for directing light towards the first eye when the device is worn;

an array of sensors on the device directed towards the first eye when the device is worn, the sensors configured for converting images of the first eye into output signals; and

a processor coupled to the sensors for interpreting the output signals to control one or more devices,

wherein the one or more emitters and the sensors are provided on the device at locations offset from the axis to generally minimize interference with the person's vision and such that the array of sensors are oriented directly towards the region surrounding the first eye.

30. The system of claim 29, further comprising a control system communicating with the processor, the processor configured for directing the control system to control one or more devices based upon the output signals.

31. The system of claim 29, further comprising a computer communicating with the processor, the processor configured for controlling the computer based upon the output signals.

32. The system of claim 31, wherein the processor is configured to interpret the output signals to operate the computer as an eye-activated mouse.

33. The system of claim 29, wherein the one or more emitters comprise a plurality of infrared emitters configured to emit pulses at a predetermined frequency.

34. The system of claim 29, wherein the device comprises a frame comprising a bridge piece extending between a pair of ear supports, the bridge piece including a nose bridge configured to be placed on the person's nose, and wherein the array of sensors are provided on or adjacent the nose bridge lateral from the first eye when the frame is worn to generally minimize interference with the person's vision along the axis.

35. The system of claim 34, wherein the one or more emitters comprise an array of emitters provided on the frame.

36. The system of claim 29, wherein the device comprises a nose bridge configured to be placed on the person's nose, and wherein the array of sensors are provided on or adjacent the nose bridge lateral from the first eye when the device is worn to generally minimize interference with the person's vision along the axis.

37. The system of claim 36, wherein the device comprises a mask.

38. The system of claim 36, wherein the device comprises an eyeglass frame.

39. A method for monitoring movement of a person's eye using a detection device including one or more emitters and an array of sensors that are directed towards a first eye of the person when the detection device is worn, the method comprising:

placing the detection device on a person's head such that the detection device does not interfere substantially with the person's vision along an axis extending directly ahead of the first eye and the array of sensors are offset from the axis and oriented directly towards a region surrounding the first eye;

emitting light from one or more emitters towards the first eye;

detecting light from the one or more emitters reflected off of the first eye with the array of sensors, the array of sensors producing light intensity signals indicating when the first eye is open or closed; and

interpreting the light intensity signals to control one or more devices.

40. The method of claim 39, wherein the array of sensors comprises a two-dimensional array of sensors.

41. The method of claim 39, wherein the light intensity signals are used to control a computer.

42. A system for controlling a computing device, comprising:

a frame configured to be worn on a person's head such that the frame does not interfere substantially with the person's vision along an axis extending directly ahead of a first eye of the person;

a sensor on the frame comprising a lens offset from the axis and oriented directly towards the region surrounding the first eye when the frame is worn, the sensor generating output signals representing video images of the first eye; and

a processor coupled to the sensor for processing the output signals to monitor movement of the first eye relative to a reference frame, the processor communicating with an electronic device remote from the frame and interpreting the output signals to control the electronic device.

43. The system of claim 42, wherein the sensor comprises a detector coupled to the lens by a fiberoptic cable.

44. The system of claim 42, wherein the electronic device comprises a computer, and wherein the processor is configured to interpret the output signals to operate the computer as an eye-activated mouse.

45. The system of claim 42, wherein the frame comprises a bridge piece extending between a pair of ear supports, the bridge piece including a nose bridge configured to be placed on the person's nose, the sensor provided on or adjacent the nose bridge at a location lateral from the first eye when the frame is worn to generally minimize interference with the person's vision along the axis.

46. The system of claim 42, further comprising a plurality of infrared emitters on the frame for directing light towards the first eye of the person when the frame is worn, the emitters configured to emit pulses at a predetermined frequency.

47. The system of claim 46, wherein the emitters are configured for projecting the reference frame onto the first eye of the person wearing the frame.

48. A system for controlling a computing device, comprising:

a frame configured to be worn on a person's head such that the device does not interfere substantially with the person's vision;

a sensor on the frame comprising a lens directed towards the eye of the person when the frame is worn, the sensor generating output signals representing video images of the eye; and

a processor coupled to the sensor for processing the output signals to monitor movement of the eye relative to a reference frame, the processor communicating with an electronic device remote from the frame and interpreting the output signals to control the electronic device,

wherein the processor is configured for monitoring the output signals to detect video images indicating that the person wearing the device has blinked in a predetermined sequence, the processor configured for executing the command on the electronic device based upon the predetermined sequence.

49. A system for monitoring movement of a person's eye, comprising:

an eyeglass frame comprising a nose bridge configured to be placed on the person's nose, and a pair of ear supports configured to be placed over the person's ears such that the frame, when worn on the person's head, does not interfere substantially with the person's vision along an axis extending directly ahead of a first eye of the person;

one or more emitters on the frame for directing light towards the first eye when the frame is worn, the one or more emitters provided on the frame at one or more locations that generally minimize interference with the person's vision along the axis;

a sensor on or adjacent the nose bridge such that the sensor is offset from the axis to generally minimize interference with the person's vision along the axis and oriented directly towards the region surrounding the first eye, the sensor configured for converting images of the first eye into output signals; and

a processor coupled to the sensor for interpreting the output signals to control one or more devices.

50. The system of claim 49, wherein the sensor comprises a lens oriented directly towards the region surrounding the first eye when the frame is worn, and a detector coupled to the lens for converting the images of the first eye into digital video signals comprising the output signals.

51. The system of claim 49, wherein the one or more devices comprise a computer.

52. The system of claim 49, wherein the one or more devices comprise a wheelchair.