Embodiments of the present invention relate to systems and methods for minimizing motion clutter in image-generation devices. Temporally-interleaved image-subtraction reduces the magnitude of motion clutter and has no adverse effect on the desired ambient-light cancellation of static images. Embodiments of image-generation devices employing temporally-interleaved image-subtraction include single, double, triple, and series accumulator configurations. All four embodiments allow synchronization with scene illuminators and may be implemented on a single electronic chip. Temporally-interleaved image-subtraction is particularly well suited for use in video eyetracking applications where ambient light and scene motion can cause significant problems.
|
15. An apparatus for generating eye images including an image-generation apparatus comprising;
an image sensor that generates an output from a physical wave input;
an image accumulator that forms a composite image;
an accumulator-control switch that controls whether the image-sensor output is added to or subtracted from the image accumulator; and
a controller for the accumulator-control switch that toggles between addition and subtraction over multiple sub-periods within an overall frame-capture period, wherein the sub-periods of the constituent images are temporally interleaved.
32. A method for generating eye images including a method for suppressing motion clutter comprising:
capturing a first image during a first sub-period;
capturing a second image during a second sub-period;
calculating a first sub-composite image from the difference between the first image and second image;
capturing a third image during a third sub-period;
calculating a second sub-composite image from the difference between the third image and the first sub-composite image;
capturing a fourth image during a fourth sub-period; and
calculating a composite image from the difference between the fourth image and the second sub-composite image.
1. An apparatus for generating eye images including an image-generation apparatus comprising:
an image sensor that generates an output from a physical wave input; two constituent-image accumulators;
an accumulator-control switch that directs the image-sensor output to either one of the two constituent-image accumulators;
a controller for the accumulator-control switch that toggles the image sensor output between the two constituent-image accumulators over multiple sub-periods within an overall frame-capture period, wherein the sub-periods of the constituent images are temporally interleaved; and
a processor that subtracts the constituent images from the two constituent-image accumulators to form a composite image.
29. A method for generating eye images including a method for suppressing motion clutter comprising:
accumulating a first image during a first sub-period;
accumulating a second image during a second sub-period;
accumulating a third image during a third sub-period;
accumulating a fourth image during a fourth sub-period;
adding the first image to the third image to derive the first sub-composite image;
adding the second image to the fourth image to derive the second sub-composite image; and
subtracting the second sub-composite image from the first sub-composite image to form a composite image,
wherein the second sub-period is between the first and third sub-periods, and the third sub-period is between the second and fourth sub-periods.
22. An apparatus for generating eye images including an image-generation apparatus comprising:
an image sensor that generates an output from a physical wave input;
a sub-period accumulator that accumulates the image-sensor output within individual sub-frame periods;
a composite-image accumulator that accumulates a composite image comprised of added and subtracted sets of sub-images;
an accumulator-control switch that directs the output from the sub-period accumulator to be added to or subtracted from the composite-image accumulator; and
a controller that initializes the sub-period accumulator before each sub-period and generates an accumulator-control switch signal that toggles between addition and subtraction over multiple sub-periods within an overall frame-capture period, wherein the sub-periods of the constituent images are temporally interleaved.
8. An apparatus for generating eve images including an image-generation apparatus comprising:
an image sensor that generates an output from a physical wave input;
a sub-period accumulator that accumulates the image-sensor output within individual sub-frame periods;
two constituent-image accumulators that accumulate constituent images comprised of different sets of sub-images from the sub-period accumulator;
an accumulator-control switch that directs the output from the sub-period accumulator to be added to either one of the constituent-image accumulators;
a controller, wherein the controller initializes the sub-period accumulator before each sub-period and generates an accumulator-control switch signal that toggles the output of the sub-period accumulator between the two constituent-image accumulators over multiple sub-periods within an overall frame-capture period, wherein the sub-periods of the constituent images are temporally interleaved; and
a processor that subtracts the constituent images from the two constituent-image accumulators to form a composite image.
2. The apparatus according to
3. The apparatus according to
4. The apparatus according to
5. The apparatus according to
6. The apparatus according to
7. The apparatus according to
9. The apparatus according to
10. The apparatus according to
11. The apparatus according to
12. The apparatus according to
13. The apparatus according to
14. The apparatus according to
16. The apparatus according to
17. The apparatus according to
18. The apparatus according to
19. The apparatus according to
20. The apparatus according to
21. The apparatus according to
23. The apparatus according to
24. The apparatus according to
25. The apparatus according to
26. The apparatus according to
27. The apparatus according to
28. The apparatus according to
30. The method according to
31. The method according to
33. The method according to
34. The method according to
|
This application claims the benefit of U.S. Provisional Patent Application Ser. No. 60/403,389, filed Aug. 15, 2002, which is herein incorporated by reference in its entirety.
1. Field of the Invention
Embodiments of the present invention relate to image processing. More particularly, embodiments of the present invention relate to systems and methods for image processing that subtract successive images to provide a resultant composite image, for purposes such as ambient light canceling, motion clutter suppression, or motion detection.
2. Background Information
Within the field of photography and image processing, a class of operations involves the subtraction of successive images captured from a camera to obtain a desired effect in the resultant composite image. For example, image subtraction is used to capture and emphasize lighting differences arising from alternate forms of scene illuminations, or to detect moving objects within a scene.
One example of image subtraction is ambient-light-canceling (ALC) photography. A camera captures two successive images of a scene, one with a camera lamp illuminating the scene and the other with the camera lamp off, and the two images are subtracted to form a synthetic image. Assuming that neither the camera nor the scene has moved between the two image-capture periods, and that the ambient light, i.e., the light coming from sources other than the camera lamp, has remained relatively constant between the two image captures, the subtracted image represents what the scene would look like if it were illuminated by the camera lamp only, without the ambient light. The subtraction of the two successive images cancels the effect of the ambient light, but leaves the effect of the camera lamp.
The process of eliminating ambient light from a camera image by subtracting two successive images, one image taken with the camera illuminator on and the other camera image taken with the camera illuminator off, is described in U.S. Pat. No. 4,274,735 by Tamura et al., U.S. Pat. No. 4,315,159 by Niwa et al., U.S. Pat. No. 4,490,037 by Anagnostopoulos et al., U.S. Pat. No. 5,287,183 by Thomas et al., U.S. Pat. No. 6,021,210 by Camus et al., and U.S. Pat. No. 6,256,067 by Yamada.
Image subtraction, however, is highly sensitive to scene motion, either to motion of the camera or to motion of objects within the scene. Indeed, because image-subtraction is sensitive to image motion, motion detection is a second key application of image subtraction. If there is motion between the successive images, all or portions of the two images become spatially displaced with respect to each other, and the image subtraction results in residuals around the edges of the moving objects.
While it is desired to emphasize the effects of motion in motion-detection applications, it is often desired to minimize motion effects in other applications of image subtraction. In applications where motion effects of image-subtraction are undesirable, these motion effects are often called ‘motion clutter.’ Motion clutter, for example, is generally highly undesirable in ambient-light-cancellation applications.
While the term “camera” typically implies the generation of images from light, one skilled in the art will appreciate that the term “camera” is contemplated to include all forms of imagers that generate images from any form of physical wave, including vibrational waves and/or particle wavefronts. Waves include, for example, visible and non-visible light, electromagnetic, radar, x-rays, gamma rays, electron, magnetic, magnetic-resonance, sonic, ultrasonic, seismic, surface, body, compression, and longitudinal waves. Particle wavefronts include nuclear particles (such as electrons, protons, positrons, neutrinos, etc.) atomic particles, molecular particles, and other material particles. The concepts discussed here apply to all wavelengths, visible or invisible, audible or inaudible. Additionally, the concepts apply to both single-frame and video modes of camera operation. Finally, though images are typically considered to be two-dimensional, the concepts discussed here apply to any number of image dimensions, including 1, 2, 3, and more.
Ambient-light-cancelling Photography
Camera images formed with multiple light sources on a scene are conceptually equivalent to the superposition of multiple images, each formed with individual light sources. Once an aggregate image is formed however, it is generally not possible to separate its constituent images. If a specialized camera properly captures an appropriate set of constituent images ahead of time, however, the desired synthetic images can be generated computationally, typically by an additive or subtractive operation on the constituent images on pixel-by-pixel basis.
It is known that the effects of ambient light may be largely reduced or eliminated from a camera's image by taking two consecutive images, one with the camera's controlled illuminator on and the other with the camera illuminator off, and then subtracting the two images to form the composite ambient-light-cancelled image.
Assuming that the ambient illumination remains approximately constant during the two image periods, the subtraction of the off-cycle image from the on-cycle image results in a net-zero contribution from the ambient light source, while leaving the full result of the camera's illumination in the resultant image.
The first camera image A, 100 shown in
The final output image C, 300 shown in
The person in the foreground now stands out significantly from the background, making objects in the foreground far easier to detect with respect to objects in the background. Note that the subtracted image C, 300 in
Motion Detectors
Image-subtraction photography is also used to detect objects that are moving within a scene. When using image subtraction to perform motion detection, typically both the first and second images are illuminated equally. When the two images are subtracted, all the stationary objects are ‘cancelled out’ in the composite image, leaving only a uniform “neutral-gray” response. If an object moves, however, the images of the object are spatially separated between the first and second constituent images, so the subtraction does not result in pure cancellation of the object image; rather, the subtraction results in a residual ‘ghost’ image around the edges of the moving object. The existence of the ghosted image within the otherwise neutral-gray background allows the moving object to be detected easily within the subtracted image.
As an object moves, the ghosting effect occurs around the leading and trailing edges of the object. Edges of an object are typically characterized by a change, i.e., a gradient, in the image intensity. As an object moves between two successive images, the differences in the leading and trailing edge locations between the two successive images produce a difference in intensity that shows up as non-neutral-gray in the difference image. It is the emergence of the edges of the moving objects that produces the ghosting effect.
Since the ghosting appears along the leading and trailing edges of moving objects, the ghost image emphasizes the object's direction of motion. If the surrounding background of an object is brighter than the object, the leading edge of the ghost image appears brighter than neutral gray. In the first image, the area just ahead of the moving object is the brighter background intensity. In the second image, that area is replaced by the darker intensity of the object that has moved into the space. When the two images are subtracted, the lower (darker) intensity of the second image is subtracted from the brighter (higher) intensity of the first image, resulting in a net brighter-than-neutral-gray in the difference image.
Conversely, if the background is darker than the object, the leading edge of the object appears darker than neutral gray. Similarly, the trailing edges of dark objects are dark, and the trailing edges of bright objects are bright.
Note the ghosting effect on the highlighted moving objects, i.e., the ghosting on the car and the plane. Since the surrounding backgrounds of both the car and plane are brighter than the vehicles themselves, the leading edges of these objects are bright and the trailing edges are dark. It can thus be deduced from the ghost images that both vehicles are moving leftward.
Also note that in image-subtraction, the subtraction process results in positive intensities in areas where the ‘added’ image is brighter than the ‘subtracted’ image, and results in negative intensities where the ‘subtracted’ image is brighter than the ‘added’ image. To accommodate negative intensities when displaying difference images, the difference image is customarily drawn with a “neutral gray” value so that all display intensities are positive and can be interpreted visually. In areas of the two constituent images where both intensities are equal, the subtracted output takes the value of the neutral-gray offset. Brighter-than-neutral-gray regions indicate that the ‘added’ image is brighter, and darker-than-neutral-gray regions indicate that the ‘subtracted’ image is brighter.
Image Subtraction Timing
As shown in
Motion Clutter
In many image-subtraction applications, such as ambient light cancellation, the ghosting effects of object motion are undesired. When the ghosting is unwanted, it is referred to as “motion clutter.” Physically, the desired ghosting effects in motion detectors and the undesired motion clutter effects in ambient-light-cancellation applications are identical. The difference lies only in the desirability of the effect when images are subtracted.
Motion clutter, like ghosting, is measured in units of image brightness, i.e., image intensity. In motion detection applications, the reference level for “zero ghosting” is neutral gray. At all positions in the image where there is no moving object, the subtracted image is neutral gray, and therefore no ghosting exists. As the intensity of the subtracted image moves away from neutral gray, either positively or negatively, the magnitude of the ghosting increases. In ambient-light-cancellation applications, the reference level for “zero motion clutter” is black. The magnitude of motion clutter is measured as an increase in brightness from black.
Note that motion clutter is a different phenomenon than motion blur, though both result from motion in the camera scene. Blur results from scene motion during the overall frame period, whether or not image subtraction is performed. Motion blur can be seen on the moving car in
In image subtraction, the spatial displacement Δx of a moving object in two successive camera images is proportional to both the velocity V of the objects in the scene and the average temporal offset Δt between the shutter periods of the two images:
Δx=V×Δt (Eq 1)
This quantity Δx, i.e., the average spatial displacement between images, is also called the ‘motion clutter width,’ which refers to the width of the ghosting edges that appear in the motion clutter. It is assumed in the above equation that there is no appreciable “dead time” between the two image shutter periods. Dead time between shutter periods further increases image motion clutter.
If the shutter periods ΔtA and ΔtB are different, the average time difference Δt between two successive shutter periods is equal to the average of ΔtA and ΔtB:
Δt=(ΔtA+ΔtB)/2 (Eq 2)
An example of undesired motion clutter is illustrated in
Embodiments of the present invention relate to systems and methods for suppressing motion clutter while providing ambient light cancellation in dynamic camera scenes. In contrast to conventional image-subtraction frame capture techniques, in which the full-scaled intensity profile of one captured image is subtracted from the full-scale intensity profile of the second to form a final image, the present invention temporally interleaves sub-periods of the constituent images, and can incorporate more than two images in the formation of the final composite image. Specifically, the present invention temporally interleaves the collection of A and B images within an overall frame exposure period.
Referring to the equations discussed above, to minimize the magnitude of motion clutter in dynamic camera scenes, it is necessary to minimize the motion clutter width Δx, which, as a consequence of Eq 1, means that it is necessary to minimize the inter-image offset time Δt. If a single pair of images is used in the image subtraction, however, reducing the inter-image offset time results in an equal reduction of image shutter time Δt, which in turn reduces the total light available to generate quality images. To maintain long net frame illumination periods while simultaneously decreasing Δt, and thus minimizing motion clutter, the present invention temporally interleaves the collection of the A and B images within the overall frame exposure period.
One embodiment of the present invention provides a method for suppressing motion clutter that includes accumulating constituent images over multiple sub-periods and temporally interleaving the sub-periods of the different constituent images. A composite image is formed from the subtraction of the accumulated constituent images. The number of sub-periods equals the factor by which the magnitude of motion clutter is reduced. A further embodiment involves controlling one or more camera-scene illuminators that are synchronized with the capture of sub-periods of the constituent images.
Another embodiment of the present invention provides an apparatus for suppressing motion clutter when forming a composite camera image from the subtraction of constituent images. The apparatus includes a containing means for accumulating each constituent image over multiple sub-periods where the sub-periods of the different constituent images are temporally interleaved. In a further embodiment, the apparatus also includes a control means for one or more camera-scene illuminators that are synchronized to the capture sub-periods of the constituent images.
Four additional embodiments of an image-generation device that use temporally-interleaved constituent images to achieve motion clutter minimization are provided. All four embodiments have an image sensor that is capable of generating an image-sensor output to a physical wave input. In a first embodiment, the image-sensor output is directed to one of two constituent-image accumulators by an accumulator-control switch. This is called the double accumulator configuration. A controller toggles the accumulator-control switch to direct the image sensor output to either one of the two constituent-image accumulators multiple times within an overall frame-capture period. Finally, the constituent images from the two constituent-image accumulators are subtracted to form a composite image.
In a second embodiment, the image-sensor output is first sent to a sub-period accumulator that accumulates the image-sensor output within individual sub-frame periods. Output from the sub-period accumulator is directed to one of two constituent-image accumulators by an accumulator-control switch. Since this embodiment has three accumulators it is called the triple accumulator configuration. A controller initializes the sub-period accumulator before each sub-period and generates an accumulator-control switch signal that toggles the output of the sub-period accumulator between the two constituent-image accumulators multiple times within an overall frame-capture period. The constituent images from the two constituent-image accumulators are then subtracted to form a composite image.
In a third embodiment, the image-sensor output is directed to a single image accumulator that forms the composite image. This embodiment is called the single accumulator configuration. An accumulator-control switch controls whether or not the image-sensor output is added to or subtracted from the image accumulator. A controller generates an accumulator-control switch signal that toggles between addition and subtraction multiple times within an overall frame-capture period.
Similarly, in a fourth embodiment, the image-sensor output is sent first to a sub-period accumulator that accumulates the image-sensor output within individual sub-frame periods. The output of the sub-period accumulator is then directed to a single composite-image accumulator that accumulates a composite image comprised of added and subtracted sets of sub-images. Since the sub-period accumulator and composite-image accumulator operate in series, this embodiment is called the series accumulator configuration. An accumulator-control switch directs the output from the sub-period accumulator to be added to or subtracted from the composite-image accumulator. A controller initializes the sub-period accumulator before each sub-period and generates an accumulator-control switch signal that toggles between addition and subtraction multiple times within an overall frame-capture period.
All four embodiments may further include at least one scene illuminator that is connected to its power source through a scene illuminator-control switch. The scene illuminator-control switch may be synchronized with the accumulator-control switch. This may be accomplished by having the controller generate an illuminator-control signal in response to the accumulator-control switch signal. This illuminator-control signal may or may not be the same as the accumulator-control switch signal. Alternatively, this may be accomplished by having the controller generate an accumulator-control switch signal in response to an externally generated illuminator-control signal.
In electronic cameras, the accumulators are implemented on the same electronic circuit chip that contains the photo sensor. Consequently, embodiments of the present invention may be implemented on a single electronic chip.
Finally, the effects of head motion and ambient light cause significant problems in video eyetracking applications. As a result, embodiments of the present invention may be advantageously used in eyetracking applications.
Before one or more embodiments of the invention are described in detail, one skilled in the art will appreciate that the invention is not limited in its application to the details of construction, the arrangements of components, and the arrangement of steps set forth in the following detailed description or illustrated in the drawings. The invention is capable of other embodiments and of being practiced or being carried out in various ways. Also, it is to be understood that the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting.
According to an embodiment of the present invention, a system to minimize the magnitude of motion clutter in dynamic camera scenes is described. In order to minimize this motion clutter, it is necessary to minimize the motion clutter width Δx, which, as a consequence of Eq 1, means that is it is necessary to minimize the inter-image offset time Δt. If a single pair of images is used in the image subtraction, however, reducing the inter-image offset time results in an equal reduction of image shutter time Δt, which in turn reduces the total light available to generate quality images. To maintain long net frame illumination periods while simultaneously decreasing Δt, and thus minimizing motion clutter, the collection of A and B images are interleaved temporally within the overall frame exposure period.
Referring back to
The key purpose of temporally interleaving the collection of the A and B images in image-subtracting cameras is to maximally overlap the aggregate shutter periods of the two constituent images in time. This temporal overlap reduces the spatial separation of moving objects in the scene between the two images, thus reducing the motion-clutter effect. When using temporally-interleaved image-subtraction, the magnitude of the motion clutter suppression is reduced by a factor equal to the number Nsub-period-pairs of A/B sub-pairs. Thus, increasing the number of sub-pairs increases the effectiveness of motion clutter suppression proportionately. With an ever-increasing number of A/B sub-period pairs, the sub-period intervals Δt may theoretically be reduced indefinitely, providing an arbitrarily small motion clutter width Δx, while simultaneously providing a sufficiently long overall frame exposure period 73 to obtain a high quality image. (Note, however, that while temporally-interleaved-frame-capture minimizes motion clutter, longer overall frame exposure periods are more susceptible to motion blur.)
Because the car is in the near background and receives some illumination from the camera flash, the basic image of the car remains, although its ghosting has been significantly suppressed. The image of the airplane in the far background, which received no light from the flash, has almost disappeared, despite its motion.
To further illustrate the motion-clutter minimization effect of temporally interleaving the collection of the A and B images prior to image subtraction,
The key results of the image subtraction process are illustrated in the intensity profiles of the subtracted, composite image C, 153, displayed in the bottom panels of
Note that the image intensity profile of the airplane, however, still stands out, despite receiving negligible light from the camera's flash attachment. Though the plane's intensity profile has no body (because it received no light from the flash), it does exhibit motion clutter at its leading edge, 154, and trailing edge, 155. Ideally, given the ambient-light-canceling camera's objective of highlighting only near-field objects, there would be no image of the plane at all.
The images of both the car and plane, because they are moving, are distorted by motion clutter at their leading edge, 154, and trailing edge, 155. This motion clutter results from the interacting dynamics of the moving vehicles, the image capture, and the image subtraction. The purpose of temporal interleaving is to reduce the magnitude of this motion clutter. As can be seen by comparing
The intensity curves in the subtracted images (i.e., image C=image A−image B) go negative at the trailing edges, 155, of the car and plane. These negative values indicate regions where image A, which is taken with the flash, is actually darker than image B, which is taken without the flash. These negative values result from motion of the car and plane. In ambient light cancellation photography (in contrast to motion detection photography), negative intensities are typically displayed as black, i.e., they are treated as having zero intensity. In
The advantage of temporally interleaving the constituent images A and B is illustrated by comparing the magnitude of the motion clutter in profile 153 in
The comparison of the temporally-interleaved and non-temporally-interleaved intensity profiles in
One, the intensity profile of the temporally-interleaved image is different from the non-temporally-interleaved image only at the leading edge, 154, and trailing edge, 155, of the moving vehicles. Thus the temporal interleaving affects only motion clutter; it has no effect (in particular, no adverse effect) on the desired ambient-light cancellation on static images.
Two, and more importantly, the temporal interleaving significantly reduces the magnitude of the motion clutter, as indicated by the gray regions, 172, in
In another embodiment, if a larger number of A/B sub-period pairs is used, the magnitude of the motion clutter decreases proportionately. As the number of pulse pairs is increased, the number of teeth in the motion clutter's saw-tooth pattern increases, and the heights of the individual teeth decrease proportionately. In the limiting case with a very large number of sub-period pairs, the saw-tooth pattern eventually approaches a straight line, completely eliminating motion-clutter.
By comparison to the large, single peak that results from conventional, non-temporally-interleaved image subtraction, the motion-clutter suppression achievable with temporal interleaving is theoretically limitless. In the case of the airplane example, the saw-tooth clutter pattern would approach a horizontal straight line with zero height, and the image of the plane would disappear completely. In the case of the car, the leading and trailing edges of the object would become sloped lines, representing conventional motion blur. The images of the leading and trailing edge would contain no residual motion clutter.
When using a conventional electronic camera sensor and a computer to perform image subtraction, the process involves the serial transfer and digitization of the raw constituent images and the subsequent pixel-by-pixel subtraction of the constituent images to form the composite image. These processes may be excessively time consuming and may severely limit the speed at which composite images can be formed. If multiple, temporally-interleaved sub-period images are generated to minimize motion clutter, the processing loads of the digitization and subtraction processes are further multiplied by the number of sub-period A/B pairs used.
One embodiment of the present invention uses dedicated, special purpose hardware, wherein the accumulations for each pixel within the image are performed in parallel. This eliminates the requirement for the camera sensor to (serially) output multiple sub-period images per frame, and eliminates the requirement for the computer to (serially) perform all the additions and subtractions (accumulations) of the sub-period images. In preferred embodiments, the accumulators are implemented on the same platform that contains the image sensor. In electronic cameras, the accumulators are implemented on the same electronic circuit chip that contains the photo sensor, for example. Thus, the composite image is fully generated within the camera sensor chip, and it is available for direct output at the end of the overall frame exposure period.
Following are four alternative embodiments for image-subtracting cameras, or more generally image-generation devices, that use temporally-interleaved constituent images to achieve motion-clutter minimization. Each embodiment begins with a sensor device that is capable of generating an image-sensor output in response to a physical wave input. Typically, the sensor output is a two-dimensional, time-varying image representation of the wavefront's energy distribution as the waves impinge on the sensor surface. The concepts apply, however, to any number of image dimensions.
Double Accumulator Configuration
At the end of the overall frame exposure period, accumulator A, 1801, and accumulator B, 1802, each contain a temporally-interleaved constituent image. The camera forms the final, composite output image, 1806, by subtracting the two constituent images with processor 1807. Controller 1805 also controls the image-generation device's scene illuminator 1808 by controlling switch 1809 between scene illuminator 1808 and scene illuminator power 1810. In one embodiment, the control signal to scene illuminator-control switch 1809 is identical to the control signal to accumulator accumulator-control switch 1804. In another embodiment, the control signal to scene illuminator-control switch 1809 is not identical to the control signal to accumulator accumulator-control switch 1804. In both embodiments, however, controller 1805 drives scene illuminator-control switch 1809 in synchronization with accumulator accumulator-control switch 1804.
Triple Accumulator Configuration
During the overall frame exposure period, controller 1805 toggles between A and B states at a high rate. At the end of each A sub-period, the sub-period accumulator's charge is transferred to accumulator A, 1801, and at the end of each B sub-period, the sub-period accumulator's charge is transferred to accumulator B, 1802. At the end of the overall frame exposure period, accumulators A and B each contain a temporally-interleaved constituent image. The image-generation device forms the final, composite output image, 1806, by subtracting the two constituent images with processor 1807.
Controller 1805 also controls the image-generation device's scene illuminator 1808 by controlling switch 1809 between scene illuminator 1808 and scene illuminator power 1810. In one embodiment, the control signal to scene illuminator-control switch 1809 is identical to the control signal to accumulator accumulator-control switch 1804. In another embodiment, the control signal to scene illuminator-control switch 1809 is not identical to the control signal to accumulator accumulator-control switch 1804. In both embodiments, however, controller 1805 drives scene illuminator-control switch 1809 in synchronization with accumulator accumulator-control switch 1804.
Single Accumulator Configuration
During the overall frame exposure period, controller 1805 toggles accumulator-control switch 1804 between A and B states at a high rate, and switches the input state of accumulator 2001 between addition and subtraction accordingly. At the end of the overall frame exposure period, composite image accumulator 2001 contains the composite image 1806. Controller 1805 also controls the image-generation device's scene illuminator 1808 by controlling switch 1809 between scene illuminator 1808 and scene illuminator power 1810. In one embodiment, the control signal to scene illuminator-control switch 1809 is identical to the control signal to accumulator accumulator-control switch 1804. In another embodiment, the control signal to scene illuminator-control switch 1809 is not identical to the control signal to accumulator accumulator-control switch 1804. In both embodiments, however, controller 1805 drives scene illuminator-control switch 1809 in synchronization with accumulator accumulator-control switch 1804.
Series Accumulator Configuration
Prior to the overall frame exposure period, composite image accumulator 2001 is reset to an initial value, which may be zero or a reference gray level. The initialization of composite image accumulator 2001 to a reference gray level may be used to prevent portions of the image from going below zero when the sub-period image is subtracted from the composite image. During the overall frame exposure period, a controller 1805 toggles accumulator-control switch 1804 between A and B states at a high rate. At the end of each A sub-period, the sub-period accumulator's charge is added to composite image accumulator 2001, and at the end of each B sub-period, the sub-period accumulator's image is subtracted from composite image accumulator 2001. At the end of the overall frame exposure period, composite image accumulator 2001 contains composite image 1806.
Controller 1805 also controls the image-generation device's scene illuminator 1808 by controlling switch 1809 between scene illuminator 1808 and scene illuminator power 1810. In one embodiment, the control signal to scene illuminator-control switch 1809 is identical to the control signal to accumulator accumulator-control switch 1804. In another embodiment, the control signal to scene illuminator-control switch 1809 is not identical to the control signal to accumulator accumulator-control switch 1804. In both embodiments, however, controller 1805 drives scene illuminator-control switch 1809 in synchronization with accumulator accumulator-control switch 1804.
Synchronizing to External Illumination Signals
In the above embodiments of image-generation device configurations, the controller is described (and shown) as generating the signal that controls the camera's illuminator. In other embodiments, however, the illuminator may be controlled by a source outside the camera. In this case, the controller may be configured to take an externally-generated illuminator-synchronization signal as an input, and to generate the accumulator-switch-control signal in response to it (as opposed to generating the illuminator-control signal in response to the internally generated accumulator-control switch signal).
Access to Constituent Images
If it is desired to recover the constituent A and/or B images from the image-generation device, either the Double-Accumulator or Triple-Accumulator provide access to the constituent A and B images.
Considerations of Image Dynamic Range
In image-subtraction applications, the intensity range (i.e., dynamic range) of a subtracted image is often significantly less than the intensity range of either of the two constituent images. In ambient-light-cancellation applications, for example, the ambient light may be significantly stronger than the image-generation device's own controlled light (as illustrated in the above examples), and the difference image (i.e. the ambient-light-cancelled image) may be relatively weak with respect to the two raw ambient-lighted images. Using conventional non-temporally-interleaved collection of the two images to be subtracted (or using the Double-Accumulator or Triple-Accumulator embodiments of the temporally-interleaved image-generation devices described above), the accumulators used in collecting the constituent raw images must contain proportionately more dynamic range than is required in the composite output image.
When using temporally-interleaved image-subtraction, however, the amount of light received during each sub-period interval is reduced by a factor of the number of A/B sub-pairs, and the required dynamic range of each sub-period image is proportionately reduced. In the above-described single-accumulator or serial-accumulator embodiments of a temporally-interleaved image-subtraction image-generation device, neither of the two constituent images A and B is integrated separately and explicitly, so high-dynamic-range image accumulators are not required. For image-subtraction applications where the dynamic range of the constituent images being subtracted significantly exceeds the required dynamic range of the composite subtracted image, the single-accumulator and serial-double-accumulator embodiments are preferred over the double-accumulator and triple-accumulator embodiments, in that they do not require high-dynamic-range accumulators for the A and B sub-images.
An embodiment of the series accumulator configuration (
The circuit shown in
At the end of accumulation period A, switch 2211 is closed and the image charge is moved from image sensor 2221 to sub-period accumulator 2201. Switch 2211 is then opened and switch 2212 is closed, and the charge is added to the charge in composite image accumulator 2203.
At the end of accumulation period B, switch 2211 is closed and the image charge is moved from image sensor to the aub-period accumulator 2201. Switch 2211 is opened. Switches 2213 and 2214 are closed to their upper contacts. The charge moves from the sub-period accumulator 2201 to capacitor 2202. After the charge has moved, switch 2213 and 2214 are closed to their lower contacts. The charge on 2202 drains from capacitor 2202. This pulls an equivalent amount of charge from the composite image accumulator 2203.
At the end of the frame period, switch 2215 is closed and the composite image accumulator is read out.
This is an over simplification of the circuit since the charge must be moved, rather than simply shared across the capacitors. However, it illustrates how a charge pump, similar to a charge pump used for a voltage inverter, can be used to implement the image subtraction circuit.
Often, in image-subtraction photography such as the ambient-light-cancellation application, the camera has a single illuminator, the illuminator is on during the collection of one of the constituent images and off during the other (e.g., on during phase A and off during phase B), and the exposure times ΔtA and ΔtB are equal. In the more general case of image-subtraction, however, the camera may employ multiple illuminators, alternative illuminators may be activated during either or both of the A and B phases, the illumination periods need not be timed to correspond exactly with the A and B phases, and the shutter periods for the two phases need not be equal in length.
Multiple illuminators, for example, may be used to enhance color contrast in a scene. In this case, the camera may illuminate the scene with one color (or more of one color) during one phase, and illuminate the scene with another color (or more of that color) during the second phase.
The proportions of the alternative lighting contained in the final composite image may also be controlled by varying the relative durations of the two exposure periods ΔtA and ΔtB. When controlling ambient light content in a composite image, for example, it may be desirable to suppress only a portion of the ambient light. In this case, the relative durations of exposure periods may be adjusted to provide a composite image that represents the ambient image alone, the camera-illuminated image alone, or any combination in between. In fact, if the exposure period for the ambient-alone constituent image is longer than the ambient-plus-camera-illuminator constituent image, the effect of ambient light in the composite image may even be reversed.
In addition to subtracting constituent images, it may be useful to combine images in other ways. For example, an image taken with the scene illuminated by a single color may be added to an image taken with normal illumination to enhance that color in the scene. Temporally-interleaved images may then be added as well as subtracted.
One important use of ambient light canceling (ALC) photography (i.e., subtracting two successive images, where one image is taken with controlled illumination and the other is taken without the controlled illumination) is in the field of video eyetracking. Typically, a video eyetracker uses an illuminator such as a light-emitting diode (LED) to illuminate the eye, and it calculates the orientation of the eye by measuring the location(s) of the pupil and/or the corneal reflection of the illuminator off the corneal surface of the eye. From the point of view of the eyetracker's logic that analyzes the image of the eye, light from other (ambient) sources adds ‘noise’ to the camera's idealized image of the eye. For example, the camera might ‘see’ the reflection of a window or a lamp on the surface of the cornea. With reflections of this type superimposed on the camera's image of the eye, the image-processing functions often have difficulty detecting and measuring the relevant features of the eye. In this case, ambient-light-cancellation can be used to preserve the desired, well-controlled image of the eye created by the camera's illuminator, despite the existence of considerable ambient light.
In addition, it is desirable to have an eyetracker function in a similar manner both indoors and outdoors. The ambient light varies tremendously between these environments. It is not possible to overcome the ambient infrared light by increasing the illumination due to eye safety limitations. Therefore, it is necessary to use ambient light cancellation methods or redesign the system. Because of the dynamic range requirements, the preferred embodiment of the system should be either the single accumulator configuration or the series accumulator configuration.
The problem of eye motion compounds the ambient light problem in video eyetracking. When the user moves his head, the eye moves within the camera's field of view, and the resulting motion clutter from the ALC's image subtraction severely degrades the camera's image of the eye. The use of temporally-interleaved constituent images significantly reduces motion clutter in the subtracted image, thus permitting more accurate and robust eyetracking, even in the presence of both ambient light and head motion.
Systems and methods in accordance with an embodiment of the present invention disclosed herein can advantageously minimize the magnitude of motion clutter in dynamic camera scenes. Systems employing temporally-interleaved image-subtraction significantly reduce the magnitude of motion clutter and have no adverse effect on the desired ambient-light cancellation of static images. Embodiments containing either double accumulator or triple accumulator configurations allow constituent images to be recovered. Embodiments containing either single accumulator or series accumulator configurations do not require high-dynamic-range accumulators for the collection of sub-images. Temporally-interleaved image-subtraction is particularly well suited for reducing the effects of ambient light and head motion in video eyetracking applications.
The foregoing disclosure of the preferred embodiments of the present invention has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many variations and modifications of the embodiments described herein will be apparent to one of ordinary skill in the art in light of the above disclosure. The scope of the invention is to be defined only by the claims, and by their equivalents.
Further, in describing representative embodiments of the present invention, the specification may have presented the method and/or process of the present invention as a particular sequence of steps. However, to the extent that the method or process does not rely on the particular order of steps set forth herein, the method or process should not be limited to the particular sequence of steps described. As one of ordinary skill in the art would appreciate, other sequences of steps may be possible. Therefore, the particular order of the steps set forth in the specification should not be construed as limitations on the claims. In addition, the claims directed to the method and/or process of the present invention should not be limited to the performance of their steps in the order written, and one skilled in the art can readily appreciate that the sequences may be varied and still remain within the spirit and scope of the present invention.
Cleveland, Dixon, Joyce, III, Arthur W.
Patent | Priority | Assignee | Title |
10182190, | Nov 10 2014 | Nikon Corporation | Light detecting apparatus, image capturing apparatus and image sensor |
7627225, | Jun 10 2005 | ADVANCED INTERCONNECT SYSTEMS LIMITED | Image capturing device and image capturing method |
7649536, | Jun 16 2006 | Nvidia Corporation | System, method, and computer program product for utilizing natural motions of a user to display intuitively correlated reactions |
7773118, | Mar 25 2007 | TOBII TECHNOLOGY LIMITED | Handheld article with movement discrimination |
7924316, | Mar 14 2007 | Aptina Imaging Corporation | Image feature identification and motion compensation apparatus, systems, and methods |
7965317, | Sep 14 2005 | Nokia Technologies Oy | System and method for implementation motion-driven multi-shot image stabilization |
8005301, | Aug 25 2006 | Q5 INNOVATIONS INC | Method of difference sensing through optical coherent change detection |
8169486, | Jun 05 2006 | FotoNation Limited | Image acquisition method and apparatus |
8212882, | Mar 25 2007 | TOBII TECHNOLOGY LIMITED | Handheld article with movement discrimination |
8243167, | Mar 20 2009 | Industrial Technology Research Institute | Image sensor having output of integral image |
8244053, | Nov 10 2004 | Adeia Imaging LLC | Method and apparatus for initiating subsequent exposures based on determination of motion blurring artifacts |
8264576, | Mar 05 2007 | FotoNation Limited | RGBW sensor array |
8270751, | Nov 10 2004 | FotoNation Limited | Method of notifying users regarding motion artifacts based on image analysis |
8285067, | Nov 10 2004 | FotoNation Limited | Method of notifying users regarding motion artifacts based on image analysis |
8320615, | Feb 27 2008 | Honeywell International Inc. | Systems and methods for recognizing a target from a moving platform |
8330810, | Feb 27 2008 | Honeywell International Inc. | Systems and method for dynamic stabilization of target data detected from a moving platform |
8417055, | Mar 05 2007 | FotoNation Limited | Image processing method and apparatus |
8494300, | Nov 10 2004 | FotoNation Limited | Method of notifying users regarding motion artifacts based on image analysis |
8520082, | Jun 05 2006 | Adeia Imaging LLC | Image acquisition method and apparatus |
8878967, | Mar 05 2007 | FotoNation Limited | RGBW sensor array |
8885882, | Jul 14 2011 | The Research Foundation for The State University of New York; The Research Foundation of State University of New York | Real time eye tracking for human computer interaction |
8989516, | Sep 18 2007 | TOBII TECHNOLOGY LIMITED | Image processing method and apparatus |
9100582, | Jan 25 2013 | Fujitsu Limited | Image capturing method and image capturing apparatus for illuminating subject |
9160897, | Jun 14 2007 | Adeia Imaging LLC | Fast motion estimation method |
Patent | Priority | Assignee | Title |
4158492, | May 30 1977 | Minolta Camera Kabushiki Kaisha | Light responsive circuit |
4272787, | Sep 07 1978 | QUANTEL LIMITED, 37 VICTORIA AVENUE, SOUTHEND ON SEA, ESSEX | T.V. Picture freeze system |
4274735, | May 25 1978 | Canon Kabushiki Kaisha | Distance measuring device |
4315159, | May 25 1978 | NORTHROP CORPORATION, A DEL CORP | Optical sensor device with reduction of ambient light effects |
4490037, | Aug 18 1982 | Eastman Kodak Company | Image sensor and rangefinder device having background subtraction with bridge network |
4636850, | Sep 07 1984 | KONINKLIJKE PHILIPS ELECTRONICS N V | Apparatus and method for enhancement of video images |
4678323, | Jul 20 1984 | Canon Kabushiki Kaisha | Distance measuring devices and light integrators therefor |
4755874, | Aug 31 1987 | EDO CORPORATION, BARNES DIVISION | Emission microscopy system |
4843565, | Jul 30 1987 | ZIMMERMAN ASSOCIATES, INC | Range determination method and apparatus |
4851689, | Apr 15 1987 | ALPS Electric Co., Ltd. | Photodetecting circuit |
4878116, | Jun 02 1988 | ARMY, THE UNITED STATES OF AMERICA, AS REPRESENTED BY THE DEPARTMENT OF THE | Vector lock-in imaging system |
4888644, | Nov 17 1987 | Ferranti International PLC | Television camera system having differentiated illumination between fields |
4985774, | Jan 20 1988 | MINOLTA CAMERA KABUSHIKI KAISHA, C O OSAKA KOKUSAI BUILDING, | Image sensing device having direct drainage of unwanted charges |
5115321, | Jan 06 1987 | Minolta Camera Kabushiki Kaisha | Image sensing system |
5162914, | Jun 09 1987 | Canon Kabushiki Kaisha | Image sensing device with diverse storage fumes used in picture composition |
5287183, | Mar 05 1990 | Wayne State University | Synchronous imaging system |
5376783, | Sep 16 1992 | Ophir Optronics Ltd. | Power meter with background subtraction |
5617141, | Apr 28 1992 | HITACHI CONSUMER ELECTRONICS CO , LTD | Image pickup devices having an image quality control function and methods of controlling an image quality in image pickup devices |
5631976, | Apr 29 1994 | International Business Machines Corporation | Object imaging system |
5657402, | Nov 01 1991 | MASSACHUSETTS INSTITUTE OF TECHNOLOGY, THE | Method of creating a high resolution still image using a plurality of images and apparatus for practice of the method |
5705807, | Oct 24 1994 | NISSAN MOTOR CO , LTD | Photo detecting apparatus for detecting reflected light from an object and excluding an external light componet from the reflected light |
5754682, | Sep 27 1995 | Sony Corporation | Picture processing method and apparatus |
5760415, | Apr 18 1996 | Krupp Fordertechnik GmbH | Photogrammetric process for the three-dimensional monitoring of a moving object |
5818954, | Jul 14 1988 | ATR Communication Systems Research Laboratories | Method of detecting eye fixation using image processing |
5982951, | May 28 1996 | Canon Kabushiki Kaisha | Apparatus and method for combining a plurality of images |
6021210, | Dec 01 1997 | SENSAR, INC | Image subtraction to remove ambient illumination |
6128108, | Sep 03 1997 | HANGER SOLUTIONS, LLC | Method and system for compositing images |
6157751, | Dec 30 1997 | Cognex Corporation | Method and apparatus for interleaving a parallel image processing memory |
6256067, | Aug 07 1996 | Agilent Technologies Inc | Electronic camera for selectively photographing a subject illuminated by an artificial light source |
6278490, | May 23 1996 | Olympus Optical Co., Ltd. | Exposure control for an image pickup apparatus that uses an electronic flash |
7012635, | Jan 31 1997 | Kabushiki Kaisha Toshiba | Solid state image sensor and video system using the same |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Nov 24 2003 | CLEVELAND, DIXON | LC TECHNOLOGIES, INC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 014735 | /0862 | |
Nov 24 2003 | JOYCE III, ARTHUR W | LC TECHNOLOGIES, INC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 014735 | /0862 | |
Sep 27 2019 | LC TECHNOLOGIES, INC | EYEGAZE INC | CHANGE OF NAME SEE DOCUMENT FOR DETAILS | 057087 | /0534 |
Date | Maintenance Fee Events |
Aug 08 2011 | REM: Maintenance Fee Reminder Mailed. |
Aug 17 2011 | M2551: Payment of Maintenance Fee, 4th Yr, Small Entity. |
Aug 17 2011 | M2554: Surcharge for late Payment, Small Entity. |
Apr 27 2015 | M2552: Payment of Maintenance Fee, 8th Yr, Small Entity. |
Jun 04 2019 | M2553: Payment of Maintenance Fee, 12th Yr, Small Entity. |
Date | Maintenance Schedule |
Jan 01 2011 | 4 years fee payment window open |
Jul 01 2011 | 6 months grace period start (w surcharge) |
Jan 01 2012 | patent expiry (for year 4) |
Jan 01 2014 | 2 years to revive unintentionally abandoned end. (for year 4) |
Jan 01 2015 | 8 years fee payment window open |
Jul 01 2015 | 6 months grace period start (w surcharge) |
Jan 01 2016 | patent expiry (for year 8) |
Jan 01 2018 | 2 years to revive unintentionally abandoned end. (for year 8) |
Jan 01 2019 | 12 years fee payment window open |
Jul 01 2019 | 6 months grace period start (w surcharge) |
Jan 01 2020 | patent expiry (for year 12) |
Jan 01 2022 | 2 years to revive unintentionally abandoned end. (for year 12) |