object detection and tracking operations on images that may be performed independently are presented. A detection module receives images, extracts edges in horizontal and vertical directions, and generates an edge map where object-regions are ranked by their immediacy. filters remove attached edges and ensure regions have proper proportions/size. The regions are tested using a geometric constraint to ensure proper shape, and are fit with best-fit rectangles, which are merged or deleted depending on their relationships. Remaining rectangles are objects. A tracking module receives images in which objects are detected and uses Euclidean distance/edge density criterion to match objects. If objects are matched, clustering determines whether the object is new; if not, a sum-of-squared-difference in intensity test locates matching objects. If this succeeds, clustering is performed and rectangles are applied to regions and merged or deleted depending on their relationships and are considered objects; if not the regions are rejected.
|
32. A method for the surveillance of objects in an image comprising a step of detecting objects in the image, with the step of detecting objects comprising sub-steps of:
receiving an image including edge regions;
independently extracting horizontal edges and vertical edges from the image;
removing horizontal edges and vertical edges that exceed a particular threshold;
combining the remaining horizontal edges and vertical edges into edge regions to generate an integrated edge map; and
applying a compactness filter to the edge regions detect objects likely to be of interest.
1. A method for the surveillance of objects in an image, the method comprising a step of detecting objects comprising sub-steps of:
receiving an image including edge regions;
independently extracting horizontal edges and vertical edges from the image;
eliminating horizontal edges and vertical edges that exceed at least one length-based threshold;
combining the remaining horizontal edges and vertical edges into edge regions to generate an integrated edge map; and
applying a ranking filter to rank relative immediacies of objects represented by the edge regions in the integrated edge map.
112. A computer program product for the surveillance of objects in an image, the computer program product comprising a computer-readable medium further comprising computer-readable means for detecting objects including means for:
receiving an image including edge regions;
independently extracting horizontal edges and vertical edges from the image;
removing horizontal edges and vertical edges that exceed a particular threshold;
combining the remaining horizontal edges and vertical edges into edge regions to generate an integrated edge map; and
applying a compactness filter to the edge regions detect objects likely to be of interest.
81. A computer program product for the surveillance of objects in an image, the computer program product comprising a computer-readable medium further comprising computer-readable means for detecting objects including means for:
receiving an image including edge regions;
independently extracting horizontal edges and vertical edges from the image;
eliminating horizontal edges and vertical edges that exceed at least one length-based threshold;
combining the remaining horizontal edges and vertical edges into edge regions to generate an integrated edge map; and
applying a ranking filter to rank relative immediacies of objects represented by the edge regions in the integrated edge map.
72. An apparatus for the surveillance of objects in an image, the apparatus comprising a computer system including a processor, a memory coupled with the processor, an input coupled with the processor for receiving images from an image source, and an output coupled with the processor for outputting information regarding objects in the image, wherein the computer system further comprises means, residing in its processor and memory, for detecting objects in the image, including means for:
receiving an image including edge regions;
independently extracting horizontal edges and vertical edges from the image;
removing horizontal edges and vertical edges that exceed a particular threshold;
combining the remaining horizontal edges and vertical edges into edge regions to generate an integrated edge map; and
applying a compactness filter to the edge regions detect objects likely to be of interest.
41. An apparatus for the surveillance of objects in an image, the apparatus comprising a computer system including a processor, a memory coupled with the processor, an input coupled with the processor for receiving images from an image source, and an output coupled with the processor for outputting information regarding objects in the image, wherein the computer system further comprises means, residing in its processor and memory, for detecting objects including means for:
receiving an image including edge regions;
independently extracting horizontal edges and vertical edges from the image;
eliminating horizontal edges and vertical edges that exceed at least one length-based threshold;
combining the remaining horizontal edges and vertical edges into edge regions to generate an integrated edge map; and
applying a ranking filter to rank relative immediacies of objects represented by the edge regions in the integrated edge map.
2. A method for the surveillance of objects in an image as set forth in
3. A method for the surveillance of objects in an image as set forth in
4. A method for the surveillance of objects in an image as set forth in
5. A method for the surveillance of objects in an image as set forth in
6. A method for the surveillance of objects in an image as set forth in
7. A method for the surveillance of objects in an image as set forth in
receiving an image including edge regions representing potential objects;
determining a geometric shape to apply as a compactness filter, with the geometric shape including a reference point;
determining an edge region reference on each edge region in the image, and aligning the reference point on each edge region with the reference point of the geometric shape;
extending search lines from the edge region reference in at least one direction; and
determining intersections of the search lines with the geometric shape, and if a predetermined number of intersections exists, classifying the edge region as a compact object.
8. A method for the surveillance of objects in an image as set forth in
9. A method for the surveillance of objects in an image as set forth in
10. A method for the surveillance of objects in an image as set forth in
11. A method for the surveillance of objects in an image as set forth in
12. A method for the surveillance of objects in an image as set forth in
13. A method for the surveillance of objects in an image as set forth in
applying an attached line edge filter to the modified image to remove remaining oriented lines from edge regions in the modified image;
applying an aspect ratio and size filter to remove edge regions from the image that do not fit a pre-determined aspect ratio or size criterion;
fitting each of the remaining edge regions within a minimum enclosing rectangle; and
removing rectangles that are fully enclosed in a larger rectangle;
removing rectangles that are vertically stacked above another rectangle;
merging rectangles that overlap in excess of a pre-determined threshold; and
labeling each remaining rectangle as an object of interest.
14. A method for surveillance of objects in an image as set forth in
15. A method for surveillance of objects in an image as set forth in
16. A method for the surveillance objects in an image as set forth in
17. A method for the surveillance of objects in an image as set forth in
receiving an image including edge regions representing potential objects;
determining a geometric shape to apply as a compactness filter, with the geometric shape including a reference point;
determining an edge region reference on each edge region in the image, and aligning the reference point on each edge region with the reference point of the geometric shape;
extending search lines from the edge region reference in at least one direction; and
determining intersections of the search lines with the geometric shape, and if a predetermined number of intersections exists, classifying the edge region as a compact object.
18. A method for the surveillance of objects in an image as set forth in
19. A method for the surveillance of objects in an image as set forth in
20. A method for the surveillance of objects in an image as set forth in
21. A method for the surveillance of objects in an image as set forth in
22. A method for the surveillance of objects in an image as set forth in
23. A method for the surveillance of objects in an image as set forth in
receiving an image including edge regions;
determining a ranking reference in the image with respect to which to rank the edge regions;
determining an edge region reference on each edge region;
determining an offset of the edge region reference with respect to the ranking reference;
normalizing the offset for each edge region to determine a set of normalized offsets;
ranking the set of normalized offsets, whereby the ranked normalized offsets provide a ranking of the immediacy of objects represented by the edge regions.
24. A method for the surveillance of objects in an image as set forth in
25. A method for the surveillance of objects in an image as set forth in
26. A method for the surveillance of objects in an image as set forth in
27. A method for the surveillance of objects in an image as set forth in
28. A method for the surveillance of objects in an image as set forth in
29. A method for the surveillance of objects in an image as set forth in
30. A method for the surveillance of objects in an image as set forth in
31. A method for the surveillance of objects in an image as set forth in
receiving an image including edge regions;
determining a ranking reference in the image with respect to which to rank the edge regions;
determining an edge region reference on each edge region;
determining an offset of the edge region reference with respect to the ranking reference;
normalizing the offset for each edge region to determine a set of normalized offsets;
ranking the set of normalized offsets, whereby the ranked normalized offsets provide a ranking of the immediacy of objects represented by the edge regions.
33. A method for the surveillance of objects in an image as set forth in
34. A method for the surveillance of objects in an image as set forth in
after combining the remaining horizontal edges and vertical edges to generate an integrated edge map and prior to applying the compactness filter:
applying an attached line edge filter to the image to remove remaining oriented lines from edge regions in the image;
applying an aspect ratio and size filter to remove edge regions from the image that do not fit a pre-determined aspect ratio or size criterion;
fitting each of the remaining edge regions within a minimum enclosing rectangle; and
removing rectangles that are fully enclosed in a larger rectangle;
removing rectangles that are vertically stacked above another rectangle;
merging rectangles that overlap in excess of a pre-determined threshold; and
labeling each remaining rectangle as an object of interest.
35. A method for the surveillance of objects in an image as set forth in
receiving an image including edge regions representing potential objects;
determining a geometric shape to apply as a compactness filter, with the geometric shape including a reference point;
determining an edge region reference on each edge region in the image, and aligning the reference point on each edge region with the reference point of the geometric shape;
extending search lines from the edge region reference in at least one direction; and
determining intersections of the search lines with the geometric shape, and if a predetermined number of intersections exists, classifying the edge region as a compact object.
36. A method for the surveillance of objects in an image as set forth in
37. A method for the surveillance of objects in an image as set forth in
38. A method for the surveillance of objects in an image as set forth in
39. A method for the surveillance of objects in an image as set forth in
40. A method for the surveillance of objects in an image as set forth in
42. An apparatus for the surveillance of objects in an image as set forth in
43. An apparatus for the surveillance of objects in an image as set forth in
44. An apparatus for the surveillance of objects in an image as set forth in
45. An apparatus for the surveillance of objects in an image as set forth in
46. An apparatus for the surveillance of objects in an image as set forth in
47. An apparatus for the surveillance of objects in an image as set forth in
receiving an image including edge regions representing potential objects;
determining a geometric shape to apply as a compactness filter, with the geometric shape including a reference point;
determining an edge region reference on each edge region in the image, and aligning the reference point on each edge region with the reference point of the geometric shape;
extending search lines from the edge region reference in at least one direction; and
determining intersections of the search lines with the geometric shape, and if a predetermined number of intersections exists, classifying the edge region as a compact object.
48. An apparatus for the surveillance of objects in an image as set forth in
49. An apparatus for the surveillance of objects in an image as set forth in
50. An apparatus for the surveillance of objects in an image as set forth in
51. An apparatus for the surveillance of objects in an image as set forth in
52. An apparatus for the surveillance of objects in an image as set forth in
53. An apparatus for the surveillance of objects in an image as set forth in
applying an attached line edge filter to the modified image to remove remaining oriented lines from edge regions in the modified image;
applying an aspect ratio and size filter to remove edge regions from the image that do not fit a pre-determined aspect ratio or size criterion;
fitting each of the remaining edge regions within a minimum enclosing rectangle; and
removing rectangles that are fully enclosed in a larger rectangle;
removing rectangles that are vertically stacked above another rectangle;
merging rectangles that overlap in excess of a pre-determined threshold; and
labeling each remaining rectangle as an object of interest.
54. An apparatus for surveillance of objects in an image as set forth in
55. An apparatus for surveillance of objects in an image as set forth in
56. An apparatus for the surveillance objects in an image as set forth in
57. An apparatus for the surveillance of objects in an image as set forth in
receiving an image including edge regions representing potential objects;
determining a geometric shape to apply as a compactness filter, with the geometric shape including a reference point;
determining an edge region reference on each edge region in the image, and aligning the reference point on each edge region with the reference point of the geometric shape;
extending search lines from the edge region reference in at least one direction; and
determining intersections of the search lines with the geometric shape, and if a predetermined number of intersections exists, classifying the edge region as a compact object.
58. An apparatus for the surveillance of objects in an image as set forth in
59. An apparatus for the surveillance of objects in an image as set forth in
60. An apparatus for the surveillance of objects in an image as set forth in
61. An apparatus for the surveillance of objects in an image as set forth in
62. An apparatus for the surveillance of objects in an image as set forth in
63. An apparatus for the surveillance of objects in an image as set forth in
receiving an image including edge regions;
determining a ranking reference in the image with respect to which to rank the edge regions;
determining an edge region reference on each edge region;
determining an offset of the edge region reference with respect to the ranking reference;
normalizing the offset for each edge region to determine a set of normalized offsets;
ranking the set of normalized offsets, whereby the ranked normalized offsets provide a ranking of the immediacy of objects represented by the edge regions.
64. An apparatus for the surveillance of objects in an image as set forth in
65. An apparatus for the surveillance of objects in an image as set forth in
66. An apparatus for the surveillance of objects in an image as set forth in
67. An apparatus for the surveillance of objects in an image as set forth in
68. An apparatus for the surveillance of objects in an image as set forth in
69. An apparatus for the surveillance of objects in an image as set forth in
70. An apparatus for the surveillance of objects in an image as set forth in
71. An apparatus for the surveillance of objects in an image as set forth in
receiving an image including edge regions;
determining a ranking reference in the image with respect to which to rank the edge regions;
determining an edge region reference on each edge region;
determining an offset of the edge region reference with respect to the ranking reference;
normalizing the offset for each edge region to determine a set of normalized offsets;
ranking the set of normalized offsets, whereby the ranked normalized offsets provide a ranking of the immediacy of objects represented by the edge regions.
73. An apparatus for the surveillance of objects in an image as set forth in
74. An apparatus for the surveillance of objects in an image as set forth in
generating an integrated edge map after combining the remaining and prior to applying the compactness filter:
applying an attached line edge filter to the image to remove remaining oriented lines from edge regions in the image;
applying an aspect ratio and size filter to remove edge regions from the image that do not fit a pre-determined aspect ratio or size criterion;
fitting each of the remaining edge regions within a minimum enclosing rectangle; and
removing rectangles that are fully enclosed in a larger rectangle;
removing rectangles that are vertically stacked above another rectangle;
merging rectangles that overlap in excess of a pre-determined threshold; and
labeling each remaining rectangle as an object of interest.
75. An apparatus for the surveillance of objects in an image as set forth in
receiving an image including edge regions representing potential objects;
determining a geometric shape to apply as a compactness filter, with the geometric shape including a reference point;
determining an edge region reference on each edge region in the image, and aligning the reference point on each edge region with the reference point of the geometric shape;
extending search lines from the edge region reference in at least one direction; and
determining intersections of the search lines with the geometric shape, and if a predetermined number of intersections exists, classifying the edge region as a compact object.
76. An apparatus for the surveillance of objects in an image as set forth in
77. An apparatus for the surveillance of objects in an image as set forth in
78. An apparatus for the surveillance of objects in an image as set forth in
79. An apparatus for the surveillance of objects in an image as set forth in
80. An apparatus for the surveillance of objects in an image as set forth in
82. A computer program product for the surveillance of objects in an image as set forth in
83. A computer program product for the surveillance of objects in an image as set forth in
84. A computer program product for the surveillance of objects in an image as set forth in
85. A computer program product for the surveillance of objects in an image as set forth in
86. A computer program product for the surveillance of objects in an image as set forth in
87. A computer program product for the surveillance of objects in an image as set forth in
receiving an image including edge regions representing potential objects;
determining a geometric shape to apply as a compactness filter, with the geometric shape including a reference point;
determining an edge region reference on each edge region in the image, and aligning the reference point on each edge region with the reference point of the geometric shape;
extending search lines from the edge region reference in at least one direction; and
determining intersections of the search lines with the geometric shape, and if a predetermined number of intersections exists, classifying the edge region as a compact object.
88. A computer program product for the surveillance of objects in an image as set forth in
89. A computer program product for the surveillance of objects in an image as set forth in
90. A computer program product for the surveillance of objects in an image as set forth in
91. A computer program product for the surveillance of objects in an image as set forth in
92. A computer program product for the surveillance of objects in an image as set forth in
93. A computer program product for the surveillance of objects in an image as set forth in
applying an attached line edge filter to the modified image to remove remaining oriented lines from edge regions in the modified image;
applying an aspect ratio and size filter to remove edge regions from the image that do not fit a pre-determined aspect ratio or size criterion;
fitting each of the remaining edge regions within a minimum enclosing rectangle; and
removing rectangles that are fully enclosed in a larger rectangle;
removing rectangles that are vertically stacked above another rectangle;
merging rectangles that overlap in excess of a pre-determined threshold; and
labeling each remaining rectangle as an object of interest.
94. A computer program product for surveillance of objects in an image as set forth in
95. A computer program product for surveillance of objects in an image as set forth in
96. A computer program product for the surveillance objects in an image as set forth in
97. A computer program product for the surveillance of objects in an image as set forth in
receiving an image including edge regions representing potential objects;
determining a geometric shape to apply as a compactness filter, with the geometric shape including a reference point;
determining an edge region reference on each edge region in the image, and aligning the reference point on each edge region with the reference point of the geometric shape;
extending search lines from the edge region reference in at least one direction; and
determining intersections of the search lines with the geometric shape, and
if a predetermined number of intersections exists, classifying the edge region as a compact object.
98. A computer program product for the surveillance of objects in an image as set forth in
99. A computer program product for the surveillance of objects in an image as set forth in
100. A computer program product for the surveillance of objects in an image as set forth in
101. A computer program product for the surveillance of objects in an image as set forth in
102. A computer program product for the surveillance of objects in an image as set forth in
103. A computer program product for the surveillance of objects in an image as set forth in
receiving an image including edge regions;
determining a ranking reference in the image with respect to which to rank the edge regions;
determining an edge region reference on each edge region;
determining an offset of the edge region reference with respect to the ranking reference;
normalizing the offset for each edge region to determine a set of normalized offsets;
ranking the set of normalized offsets, whereby the ranked normalized offsets provide a ranking of the immediacy of objects represented by the edge regions.
104. A computer program product for the surveillance of objects in an image as set forth in
105. A computer program product for the surveillance of objects in an image as set forth in
106. A computer program product for the surveillance of objects in an image as set forth in
107. A computer program product for the surveillance of objects in an image as set forth in
108. A computer program product for the surveillance of objects in an image as set forth in
109. A computer program product for the surveillance of objects in an image as set forth in
110. A computer program product for the surveillance of objects in an image as set forth in
111. A computer program product for the surveillance of objects in an image as set forth in
receiving an image including edge regions;
determining a ranking reference in the image with respect to which to rank the edge regions;
determining an edge region reference on each edge region;
determining an offset of the edge region reference with respect to the ranking reference;
normalizing the offset for each edge region to determine a set of normalized offsets;
ranking the set of normalized offsets, whereby the ranked normalized offsets provide a ranking of the immediacy of objects represented by the edge regions.
113. A computer program product for the surveillance of objects in an image as set forth in
114. A computer program product for the surveillance of objects in an image as set forth in
generating an integrated edge map after combining the remaining and prior to applying the compactness filter:
applying an attached line edge filter to the image to remove remaining oriented lines from edge regions in the image;
applying an aspect ratio and size filter to remove edge regions from the image that do not fit a pre-determined aspect ratio or size criterion;
fitting each of the remaining edge regions within a minimum enclosing rectangle; and
removing rectangles that are fully enclosed in a larger rectangle;
removing rectangles that are vertically stacked above another rectangle;
merging rectangles that overlap in excess of a pre-determined threshold; and
labeling each remaining rectangle as an object of interest.
115. A computer program product for the surveillance of objects in an image as set forth in
receiving an image including edge regions representing potential objects;
determining a geometric shape to apply as a compactness filter, with the geometric shape including a reference point;
determining an edge region reference on each edge region in the image, and aligning the reference point on each edge region with the reference point of the geometric shape;
extending search lines from the edge region reference in at least one direction; and
determining intersections of the search lines with the geometric shape, and if a predetermined number of intersections exists, classifying the edge region as a compact object.
116. A computer program product for the surveillance of objects in an image as set forth in
117. A computer program product for the surveillance of objects in an image as set forth in
118. A computer program product for the surveillance of objects in an image as set forth in
119. A computer program product for the surveillance of objects in an image as set forth in
120. A computer program product for the surveillance of objects in an image as set forth in
|
This application is related to U.S. Patent Application 60/391,294, titled “METHOD AND APPARATUS FOR THE SURVEILLANCE OF OBJECTS IN IMAGES”, filed with the U.S. Patent and Trademark Office on Jun. 20, 2002, which is incorporated by reference herein in its entirety.
(1) Technical Field
The present invention relates to techniques for the surveillance of objects in images, and more specifically to systems that automatically detect and track vehicles for use in automotive safety.
(2) Discussion
Current computer vision techniques for vehicle detection and tracking require explicit parameter assumptions (such as lane locations, road curvature, lane widths, road color, etc.) and/or use computationally expensive approaches such as optical flow and feature-based clustering to identify and track vehicles. Systems that use explicit parameter assumptions suffer from excessive rigidity, as changes in scenery can severely degrade system performance. On the other hand, computationally expensive approaches are typically difficult to implement and expensive to deploy.
To overcome these limitations, a need exists in the art for a system that does not make any explicit assumptions regarding lane and/or road models and that does not need expensive computation.
The following references are provided as additional general information regarding the field of the invention.
1. Zielke, T., Brauckmann, M., and von Seelen, W., “Intensity and edge-based symmetry detection with an application to car-following,” CVGIP: Image Understanding, vol. 58, pp. 177-190, 1993.
2. Thomanek, F., Dickmanns, E. D., and Dickmanns, D., “Multiple object recognition and scene interpretation for autonomous road vehicle guidance,” Intelligent Vehicles, pp. 231-236, 1994.
3. Rojas, J. C., and Crisman, J. D., “Vehicle detection in color images,” in Proc. of IEEE Intelligent Transportationm Systems, Boston, Mass., 1995.
4. Betke, M., and Nguyen, H., “Highway scene analysis from a moving vehicle under reduced visibility conditions,” in Proc. of IEEE International Conference on Intelligent Vehicles, pp. 131-13, 1998.
5. Kaminiski, L., Allen, J., Masaki, I., and Lemus, G., “A sub-pixel stereo vision system for cost-effective intelligent vehicle applications,” Intelligent Vehicles, pp. 7-12, 1995.
6. Luong, Q., Weber, J., Koller, D., and Malik, J., “An integrated stereo-based approach to automatic vehicle guidance,” in International Conference on Computer Vision, pp. 52-57, 1995.
7. Willersinn, D., and Enkelmann, W., “Robust Obstacle Detection and Tracking by Motion Analysis,” in Proc. of IEEE International Conference on Intelligent Vehicles, 1998.
8. Smith, S. M., and Brady, J. M., “A Scene Segmenter: Visual Tracking of Moving Vehicles,” Engineering Applications of Artificial Intelligence, vol. 7, no. 2, pp. 191-204, 1994.
9. Smith, S. M., and Brady, J. M., “ASSET-2: Real Time Motion Segmentation and Shape Tracking,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 17, no. 8, August 1995.
10. Kalinke, T., Tzomakas, C., and Seelen, v. W., “A Texture-based object detection and an adaptive model-based classification,” Proc. of IEEE International Conference on Intelligent Vehicles, 1998.
11. Gavrila, D. M., and Philomin, P., “Real-Time Object Detection Using Distance Transform,” in Proc. of IEEE International Conference on Intelligent Vehicles, pp. 274-279, 1998.
12. Barron, J. L., Fleet, D. J., and Beauchemin, S. S., “Performance of Optic Flow Techniques,” International Journal of Computer Vision, vol. 12, no. 1, pp. 43-77, 1994.
14. Jain, R., Kasturi, R., and Schunck, B. G., Machine Vision, pp. 140-180, co-published by MIT Press and McGraw-Hill Inc, 1995.
15. Kolodzy, P. “Multidimensional machine vision using neural networks,” Proc. of IEEE International Conference on Neural Networks, pp. 747-758, 1987.
16. Grossberg, S., and Mingolla, E., “Neural Dynamics of surface perception:
boundary webs, illuminants and shape from shading,” Computer Vision, Graphics and Image Processing, vol. 37, pp. 116-165, 1987.
17. Ballard, D. H., “Generalizing Hough Transform to Detect Arbitrary Shapes,” Pattern Recognition, vol. 13, no. 2, pp. 111-122, 1981.
18. Daugmann, J., “Complete discrete 2D Gabor Transforms by Neural Networks for Image Analysis and Compression,” IEEE Trans. On Acoustics, Speech and Signal Processing, vol. 36, pp. 1169-1179, 1988.
19. Hashimoto, M., and Sklansky, J., “Multiple-Order Derivatives for Detecting Local Image Characteristics,” Computer Vision, Graphics and Image Processing, vol. 39, pp. 28-55, 1987.
20. Freeman, W. T., and Adelson, E. H., “The Design and Use of Steerable Filters,” IEEE Trans. On Pattern Analysis and Machine Intelligence, vol. 13, no. 9. September 1991.
21. Yao, Y., and Chellapa, R., “Tracking a Dynamic Set of Feature Points,” IEEE Transactions on Image Processing, vol. 4, no. 10, pp. 1382-1395, October 1995.
22. Lucas, B., and Kanade, T., “An iterative image registration technique with an application to stereo vision,” DARPA Image Understanding Workshop, pp. 121-130, 1981.
23. Wang, Y. A., and Adelson, E. H., “Saptio-Temporal Segmentation of Video Data,” M.I.T. Media Laboratory Vision and Modeling Group, Technical Report, no. 262, Febraury, 1994.
The present invention relates to techniques for the surveillance of objects in images, and more specifically to systems that automatically detect and track vehicles for use in automotive safety. The invention includes detection and tracking modules that may be used independently or together depending on a particular application. In addition, the invention also includes filtering mechanisms for assisting in object detection.
In a one aspect, the present invention provides a method for ranking edge regions representing objects in an image based on their immediacy with respect to a reference. In this aspect, an image is received, including edge regions. A ranking reference is determined within the image; and the edge regions are ranked with respect to it. Next, an edge region reference is determined for each edge region, and the offset from the edge region references are determined with respect to the ranking reference. The results are then normalized and ranked such that the ranked normalized offsets provide a ranking of the immediacy of objects represented by the edge regions. The edge region reference may be a centroid of the area of the edge region to which it corresponds.
In another aspect, the image comprises a plurality of pixels, and the edge regions each comprise areas represented by a number of pixels. Each offset is represented by a distance measured as a number of pixels, and each normalized offset is determined by dividing the number of pixels representing the offset for an edge region by the number of pixels representing the area of the edge region.
In another aspect, the objects represented by edge regions are automobiles and the ranking reference is the hood of an automobile from which the image was taken, and in a still further aspect, the offset includes only a vertical component measured as a normal line from a point on the hood of the automobile to the centroid of the area of the edge region. In another aspect, the immediacy is indicative of a risk of collision between the automobile from which the image was taken and the object represented by the edge region.
In another aspect, the present invention further comprises a step of selecting a fixed-size set of edge regions based on their ranking and discarding all other edge regions, whereby only the edge regions representing objects having the greatest immediacy are kept. In yet another aspect, the present invention comprises a further step of providing the rankings of the edge regions representing objects kept to an automotive safety system.
In another aspect, the present invention provides a method for detecting objects in an image including a step of applying a compactness filter to the image. The compactness filter comprises a first step of receiving an image including edge regions representing potential objects. Next, a geometric shape is determined for application as a compactness filter, with the geometric shape including a reference point. Then, an edge region reference is determined on each edge region in the image, and the reference point on each edge region is aligned with the reference point of the geometric shape. Next, search lines are extended from the edge region reference in at least one direction. Then, intersections of the search lines with the geometric shape are determined, and if a predetermined number of intersections exists, classifying the edge region as a compact object.
In another aspect, the objects are cars, and the geometric shape is substantially “U”-shaped.
In a still further aspect, the search lines are extended in eight directions with one search line residing on a horizontal axis, with a 45 degree angle between each search line and each other search line. Also, at least four intersections between search lines and the geometric shape, including an intersection in a bottom vertical direction, must occur for the edge region to be classified as a compact object. Then all edge regions not classified as compact objects may be eliminated from the image. After the compact objects have been classified, information regarding them may be provided to an automotive safety system.
In another aspect, the present invention provides a method for the surveillance of objects in an image, with the method comprising a step of detecting objects. First, an image is received including edge regions. Next horizontal edges and vertical edges are independently extracted from the image. Then, horizontal edges and vertical edges that exceed at least one length-based threshold are eliminated. Next, the remaining horizontal edges and vertical edges are combined into edge regions to generate an integrated edge map. Then, a ranking filter is applied to rank relative immediacies of objects represented by the edge regions in the integrated edge map.
In another aspect, the edge extraction operations are performed by use of a Sobel operator. In another aspect, the horizontal and vertical edges are grouped into connected edge regions based on proximity. A fixed-size set of objects may be selected based on their ranking and other objects may be discarded to generate a modified image. A compactness filter, as previously described, may be applied to the modified image to detect edge regions representing objects likely to be of interest.
In another aspect, an attached line edge filter is applied to the modified image to remove remaining oriented lines from edge regions in the modified image. Next, an aspect ratio and size filter is applied to remove edge regions from the image that do not fit a pre-determined aspect ratio or size criterion. Then, each of the remaining edge regions is fitted within a minimum enclosing rectangle. Subsequently, rectangles that are fully enclosed in a larger rectangle are removed along with rectangles that are vertically stacked above another rectangle. Rectangles that overlap in excess of a pre-determined threshold are merged. Each remaining rectangle is labeled as an object of interest.
In another aspect, the attached line edge filter uses a Hough Transform to eliminate oriented edge segments that match patterns for expected undesired edge segments.
In another aspect, the aspect ratio has a threshold of 3.5 and the size criterion has a threshold of about 1/30 of the pixels in the image, and where edge regions that do not match both of these criterions are excluded from the image.
Between the application of the aspect ratio and size filter and the fitting of each of the remaining edge regions within a minimum enclosing rectangle, a compactness filter, as described above may be applied to remove those edge regions from the image that do not match the predetermined geometrical constraint.
Note that the compactness filter and the ranking filter may be used together or independently in the object detection process.
In another aspect, the present invention provides a method for the surveillance of objects in an image comprising a step of tracking objects from a first frame to a next frame in a series of images. Each image includes at least one region. In this method, first objects within the first frame are detected. Note that although labeled as the “first” frame, this label is intended to convey simply that the first frame was received before the second frame, and not that the first frame is necessarily the first frame in any absolute sense with respect to a sequence of frames. After the objects are detected in the first frame, they are also detected in the next frame. “Next” as used here is simply intended to convey that this frame was received after the first frame, and is not intended to imply that it must be the immediately next one in a sequence of images. After the objects have been detected in the first and next frames, the objects of the first frame and the next frame are matched using at least one geometric criterion selected from a group consisting of a Euclidean distance criterion and an edge density criterion, in order to find clusters of objects.
In another aspect, in the process of matching regions of the first frame and the next frame, the Euclidean distance criterion is used to determine a centroid of the area of an object in the first frame and a centroid of the area of the same object in the next frame. When the position of the centroid in the first frame is within a predetermined distance of the centroid in the next frame, the object in the first frame is considered matched with the object in the next frame. For objects with centroids that are matched, the edge density criterion is applied to determine a density of edges in regions of the objects and a size of the objects. If a size difference between the matched objects is below a predetermined threshold, then the object with the greater density of edges is considered to be the object and an operation of determining object clusters is performed to determine if the object is a new object or part of an existing cluster of objects. When the position of the centroid in the first frame is not within a predetermined distance of the centroid in the next frame, or when the size difference between the matched objects is above a predetermined threshold, a sum-of-square-of-difference in intensity criterion is applied to determine whether to consider the object in the first image or the object in the next image as the object. If the sum-of-square-of-difference in intensity criterion fails, the object is eliminated from the image. If the sum-of square-of-difference in intensity criterion passes, an operation of determining object clusters is performed to determine if the object is a new object or part of an existing cluster of objects, whereby objects may be excluded, may be considered as new objects, or may be considered as an object in an existing cluster of objects.
In another aspect, the operation of determining object clusters begins by clustering objects in the first frame into at least one cluster. Next, a centroid of the area of the at least one cluster is determined. Then, a standard deviation in distance of each object in the second image from a point in the next frame is determined, corresponding to the centroid of the area of the at least one cluster. When the distance between an individual object and the point corresponding to the centroid exceeds a threshold, the individual object is considered a new object. When the distance between an individual object and the point corresponding to the centroid is less than the threshold, the individual object is considered part of the cluster.
In a further aspect, the image includes a plurality of pixels, with each pixel including a Cartesian vertical “Y” component. The predetermined geometric constraint is whether a pixel representing a centroid of the individual object has a greater “Y” component than the centroid of the cluster. When the individual object has a greater “Y” component, the individual object is excluded from further consideration.
In a still further aspect, the method for the surveillance of objects in an image further comprises operations of merging best-fit rectangles for regions that are fully enclosed, are vertically stacked, or are overlapping; and labeling each remaining region as a region of interest.
In another aspect, the present invention includes operation of a sum-of-square-of-difference in intensity of pixels over an area of the next frame associated with an area of an object in the first frame over all positions allowed by a search area constraint. Next, the position which yields the least sum-of-square-of-difference in intensity of pixels is determined. If the least sum-of-square-of-difference in intensity is beneath a predetermined threshold, the area of the next frame is shifted into alignment based on the determined position, whereby errors in positions of objects are minimized by aligning them properly in the next frame.
Each of the operations of the method discussed above typically corresponds to a software module or means for performing the function on a computer or a piece of dedicated hardware with instructions “hard-coded” therein in circuitry. In other aspects, the present invention provides an apparatus comprising a computer system including a processor, a memory coupled with the processor, an input coupled with the processor for receiving images from at least one image source, and an output coupled with the processor for outputting an output selected from a group consisting of a recommendation, a decision, and a classification based on the information collected. The computer system may be a stand-alone system or it may be a distributed computer system. The computer system further comprises means, residing in its processor and memory for performing the steps mentioned above in the discussion of the method. In another aspect, the apparatus may also include the image sources from which images are gathered (e.g., video cameras, radar imaging systems, etc.).
In other aspects, the means or modules (steps) may be incorporated onto a computer readable medium to provide a computer program product.
The objects, features and advantages of the present invention will be apparent from the following detailed descriptions of the various aspects of the invention in conjunction with reference to the following drawings.
The present invention relates to techniques for the surveillance of objects in images, and more specifically to systems that automatically detect and track vehicles for use in automotive safety. The following description, taken in conjunction with the referenced drawings, is presented to enable one of ordinary skill in the art to make and use the invention and to incorporate it in the context of particular applications. Various modifications, as well as a variety of uses in different applications, will be readily apparent to those skilled in the art, and the general principles defined herein, may be applied to a wide range of aspects. Thus, the present invention is not intended to be limited to the aspects presented, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein. Furthermore, it should be noted that, unless explicitly stated otherwise, the figures included herein are illustrated diagrammatically and without any specific scale, as they are provided as qualitative illustrations of the concept of the present invention.
In order to provide a working frame of reference, first a glossary of terms used in the description and claims is given as a central resource for the reader. Next, a discussion of various physical aspects of the present invention is provided. Finally, a discussion is provided to give an understanding of the specific details.
Before describing the specific details of the present invention, a centralized location is provided in which various terms used herein and in the claims are defined. The glossary provided is intended to provide the reader with a general understanding of the intended meaning of the terms, but is not intended to convey the entire scope of each term. Rather, the glossary is intended to supplement the rest of the specification in more accurately explaining the terms used.
Immediacy—This term refers to the (possibly weighted) location of objects detected/tracked with respect to a reference point. For example, in a vehicle detecting and tracking system, where the bottom of the image represents the hood of a car, immediacy is measured by determining the location of other vehicles in the image with respect to the bottom of the image. The location may be normalized/weighted, for example, by dividing the distance of a detected vehicle from the hood of the car (vertically measured in pixels) by the area of an edge-based representation of the vehicle. The resulting number could be considered the immediacy of the vehicle with respect to the hood of the car. This measure of immediacy may then be provided to warning systems and used as a measure of the level of threat. The specification of what constitutes “immediacy” depends on the goal of a particular application, and could depend on a variety of factors, non-limiting examples of which could include object size, object location within the image, object shape, and overall density of objects in the image, or a combination of many factors.
Means—The term “means” as used with respect to this invention generally indicates a set of operations to be performed on, or in relation to, a computer. Non-limiting examples of “means” include computer program code (source or object code) and “hard-coded” electronics. The “means” may be stored in the memory of a computer or on a computer readable medium, whether in the computer or in a location remote from the computer.
Object—This term refers to any item or items in an image for which detection is desired. For example, in a case where the system is deployed for home security, objects would generally be human beings. On the other hand, when automobile detection and tracking is desired, cars, buses, motorcycles, and trucks are examples of objects to be tracked. Although other items generally exist in an image, such as lane markers, signs, buildings, and trees in an automobile detection and tracking case, they are not considered to be objects, but rather extraneous features in the image. Also, because this Description is presented in the context of a vehicle detecting and tracking system, the term “object” and “vehicle” are often used interchangeably.
Surveillance—As used herein, this term is intended to indicate an act of observing objects, which can include detecting operations or tracking operations, or both. The results of the surveillance can be utilized by other systems in order to take action. For example, if specified criterion are met in a situation involving home security, a command may be sent to an alarm system to trigger an alarm or, perhaps, to call for the aid of law enforcement.
The present invention has three principal “physical” aspects. The first is a system surveillance of objects in images, and is typically in the form of a computer system operating software or in the form of a “hard-coded” instruction set. The computer system may be augmented by other systems or by imaging equipment. The second physical aspect is a method, typically in the form of software, operated using a data processing system (computer). The third principal physical aspect is a computer program product. The computer program product generally represents computer readable code stored on a computer readable medium such as an optical storage device, e.g., a compact disc (CD) or digital versatile disc (DVD), or a magnetic storage device such as a floppy disk or magnetic tape. Other, non-limiting examples of computer readable media include hard disks, read only memory (ROM), and flash-type memories.
A block diagram depicting the components of a general or specific purpose computer system used in an aspect the present invention is provided in
An illustrative diagram of a computer program product embodying the present invention is depicted in
The present invention includes method, apparatus, and computer program product aspects that assist in object surveillance (e.g. object detection and tracking). A few non-limiting examples of applications in which the present invention may be employed include vehicle detection and tracking in automotive safety systems, intruder detection and tracking in building safety systems, and limb detection and tracking in gesture recognition systems. The Discussion, below, is presented in the context of an automobile safety system. However, it should be understood that it is contemplated that the invention may be adapted to detect and track a wide variety of other objects in a wide variety of other situations.
A simple flow chart presenting the major operations of a system incorporating the present invention is depicted in
The Discussion presented next expands on the framework presented in
a. The Object Detecting Module
The purpose of the object detecting module is to detect objects of interest in images in a computationally simple, yet robust, manner without the need for explicit assumptions regarding extraneous features (items not of interest) in the images. The object detecting module provides an edge-based constraint technique that uses each image in a sequence of images without forming any associations between the objects detected in the current image with those detected in previous or subsequent images. The final result of the process, in a case where the objects detected are vehicles on a road, is the detection of in-path and nearby vehicles in front of a host vehicle. The vehicle detection process is then followed by the vehicle tracking process to ensure the stability of the vehicle shapes and sizes as well as to make associations between vehicles in consecutive image frames. This step ensures that the vehicles in front of the host vehicle are not just detected, but also tracked. This capability is useful for making predictions regarding the position and velocity of a vehicle so as to enable preventative action such as collision avoidance or collision warning.
A flow diagram providing an overview of the operations performed by the object detecting module is presented in
After the image snapping operation 402 has been performed, edges are extracted from the image. The edges may be extracted in a single step. However, it is advantageous to independently extract horizontal and vertical edges. Most edge-extraction operations involve a one-step process using gradient operators such as the Sobel or the Canny edge detectors, both of which are well-known in the art. However, unless the object to be detected is already well-separated from the background, it is very difficult to directly segment the object from the background using these edges alone. This difficulty is referred to as “figure-ground separation”. To more clearly illustrate this issue,
After the horizontal edge extraction operation 404 and the vertical edge extraction operation 406 have been performed, it is desirable to further filter for edges that are unlikely to belong to objects sought. In the case of automobiles, the horizontal and vertical edges generally have known length limits (for example, the horizontal edges representing an automobile are generally shorter than the distance between two neighboring lanes on a road). Thus, operations are performed for eliminating horizontal edges and vertical edges that exceed a length-based threshold, and are represented by horizontal and vertical edge size filter blocks 408 and 410, respectively. The results of these operations are merged into an integrated (combined) edge map in an operation of computing an integrated edge map 412. The results of the horizontal and vertical edge extraction operations, 404 and 406, as well as of the horizontal and vertical edge size filter operations, 408 and 410, and the operation of computing an integrated edge map 412 are shown illustratively in
After the operation of computing an integrated edge map 412 has been performed, each surviving 8-connected edge is treated as an edge region. These edge regions are further filtered using a rank filter 414. The rank filter 414 identifies edge regions that are closer to a reference point (in the case of automotive systems, the host vehicle) and retains them while filtering those that are farther away and thus less immediate. In the case of a vehicle, this has the effect of removing vehicles that constitute a negligible potential threat to the host vehicle. In the operation of the rank filter 414 for vehicle detection, first the host vehicle is hypothesized to be at the bottom center of the image. This is a reasonable hypothesis about the relative position of the host vehicle with respect to the other vehicles in the scene, given that there is a single image and that the camera is mounted facing forward on the vehicle (a likely location is in on top of the rear-view mirror facing forward). In this automotive scenario, where the bottom of the image is used as the reference, in order to identify the edge regions that are closest to the vehicle, the vertical offset from the image bottom (the host vehicle) to the centroid of each edge region is computed. The result of this operation is shown in the image depicted in
The edge regions remaining after the rank filter 414 has been applied are then provided to an attached line edge filter 416 for further processing. When horizontal and vertical edge responses are separately teased out and grouped as described above, one remaining difficulty, particularly in the case of vehicle detection where vertical and horizontal edges are used for object detection, is that oriented lines receive almost equal responses from both the horizontal and vertical edge extraction operations, 404 and 406, respectively. Thus, some of these oriented line edges can survive and remain attached to the vehicle edge regions. Predominantly, these oriented edges are from lanes and road edges. The attached line edge filter 416 is applied in order to remove these unwanted attachments. In this filtering process, each edge region from the output of the rank filter is processed by a Hough Transform filter. Hough Transform filters are well-known for matching and recognizing line elements in images. So, a priori knowledge regarding the likely orientation of these unwanted edge segments can be used to search for oriented edge segments within each edge region that match the orientation of potential lanes and other noisy edges that normally appear in edge maps of roads. Before deleting any matches, the significance of the match is determined (i.e., whether the significance is greater exceeds threshold for that orientation alone). Two images are depicted in
Referring again to
Referring back to
The edge regions remaining after the application of the compactness filter 420 are then each fitted with a minimum enclosing rectangle in a rectangle-fitting operation 422.
The result of application of the object detection module to several images is shown in
Next, the object tracking module will be discussed. The object tracking module complements and further stabilizes the output of the object detection module.
b. The Object Tracking Module
The object tracking module is used in conjunction with the object detection module to provide more robust results. The object tracking module provides a robust tracking approach based on a multi-component matching metric, which produces good tracking results. The use of only edge information in both the object detection module and the object tracking module make the present invention robust to illumination changes and also very amenable to implementation using simple CMOS-type cameras that are both ubiquitous and inexpensive. The object tracking module uses a metric for matching object regions between frames based on Euclidean distance, a sum-of-square-of-difference (SSD), and edge-density of vehicle regions. It also uses vehicle clusters in order to help detect new vehicles. The details of each operation performed by the object tracking module will be discussed in further detail below, with reference to the flow diagram shown in
After the object tracking operations begin 1400, a series of images is received 1402. Two paths are shown exiting the image receiving operation 1402, representing operations using a first frame and operations using a next frame. Note that the terms “first frame” and “next frame”, as used herein are intended simply to indicate two different frames in a series of frames, without limitation as to their positions in the series. In many cases, such as in vehicle detection, it is desirable that the first frame and the next frame be a pair of sequential images at some point in a sequence (series) of images. In other words, the first frame is received “first” relative to the next frame; the term “first” is not intended to be construed in an absolute sense with respect to the series of images. In the case of
In order to overcome this difficulty, a metric is employed for matching the intensity of the pixels within the region. An example of such a metric is the sum-of-square-of-difference (SSD) in intensity, which is relatively insensitive to scale changes. In order to generate a SSD in intensity metric 1410 for a region of interest, a matching template is defined, with approximately the same area as the region of interest. The matching template may be of a pre-defined shape and area or it may be adapted for use with a particular pair of objects to be matched. A search area constraint is used to limit the search area over which various possible object positions are checked for a match. In order to more clearly illustrate the SSD in intensity metric 1410,
If a region from the current frame goes unmatched using both the Euclidean distance criterion/edge density criterion 1408 and the SSD intensity metric 1410, then the region must be rejected. However, if a region from the current frame satisfies either one of the Euclidean distance criterion/edge density criterion 1408 and the SSD intensity metric 1410, then, two possible conclusions may be drawn. The first conclusion is that the unmatched region is a newly appearing object (vehicle) that must be retained. The second conclusion is that the new region is a noisy region that managed to slip by all the edge constraints in the object detection module, and must be rejected. In order to ensure that the proper conclusion is made, a new cluster-based constraint for region addition is presented, and is depicted in
If there are more regions to process, the procedure is repeated beginning with the Euclidean distance criterion/edge density criterion 1408 until all of the remaining regions have been processed, as indicated by the decision block 1416. If all of the regions have been processed, the same operations that were performed by the object detection module on the rectangle configurations in the image (represented by operation block 424 in
Summarizing with respect to object surveillance, the present invention provides a technique for automatically detecting and tracking objects in front of a host that is equipped with a camera or other imaging device. The object detection module uses a set of edge-based constraint filters that help in segmenting objects from background clutter. These constraints are computationally simple and are flexible to tolerate the wide variations, for example, those that occur in detection and tracking of vehicles such as cars, trucks and vans. Furthermore, the constraints aid in eliminating noisy edges. After object detection has been performed, an object tracking module can be applied for tracking the segmented vehicles over multiple frames. The tracking module utilizes a three-component metric for matching object regions between frames based on Euclidean distance, sum-of-square-of-difference (SSD), and edge-density of vehicle regions. Additionally, if the three-component metric is only partially satisfied, then object clusters are used in order to detect new vehicles.
c. Further Applications
Although the present invention has been discussed chiefly in the context of a vehicle surveillance system that can provide information to automotive safety systems such as collision warning systems and adaptive airbag deployment systems, it has a wide variety of other uses. By adjusting the filters and operations described, other types of objects may be detected and tracked.
Additionally, various operations of the present invention may be performed independently as well as in conjunction with the others mentioned. For example, the ranking filter may be applied to images, using the proper reference point, along with proper distance measures in order to rank any type of object detected by its immediacy. The compactness filter, with the proper geometric shape and properly chosen tests in relation to the geometric shape, may also be used independently in order to match objects to a desired pattern. Finally, though discussed together as a system, the object detection and object tracking modules may be used independently for a variety of purposes.
Patent | Priority | Assignee | Title |
10002435, | Jan 29 2016 | GOOGLE LLC | Detecting motion in images |
10081308, | Jul 08 2011 | Bendix Commercial Vehicle Systems LLC | Image-based vehicle detection and distance measuring method and apparatus |
10083366, | Feb 14 2014 | NANT HOLDINGS IP, LLC | Edge-based recognition, systems and methods |
10127441, | Mar 13 2013 | KOFAX, INC. | Systems and methods for classifying objects in digital images captured using mobile devices |
10146795, | Jan 12 2012 | KOFAX, INC | Systems and methods for mobile image capture and processing |
10146803, | Apr 23 2013 | KOFAX, INC | Smart mobile application development platform |
10207409, | Oct 16 2014 | Canon Kabushiki Kaisha | Image processing method, image processing device, and robot system |
10242285, | Jul 20 2015 | KOFAX, INC.; KOFAX, INC | Iterative recognition-guided thresholding and data extraction |
10657600, | Jan 12 2012 | KOFAX, INC. | Systems and methods for mobile image capture and processing |
10664919, | Jan 12 2012 | KOFAX, INC. | Systems and methods for mobile image capture and processing |
10803350, | Nov 30 2017 | KOFAX, INC.; KOFAX, INC | Object detection and image cropping using a multi-detector approach |
10957054, | Jan 29 2016 | GOOGLE LLC | Detecting motion in images |
11062176, | Nov 30 2017 | KOFAX, INC. | Object detection and image cropping using a multi-detector approach |
11132530, | Jul 19 2018 | Conti Temic Microelectronic GmbH | Method for three-dimensional graphic reconstruction of a vehicle |
11176406, | Feb 14 2014 | NANT HOLDINGS IP, LLC | Edge-based recognition, systems and methods |
11625840, | Jan 29 2016 | GOOGLE LLC | Detecting motion in images |
7639841, | Dec 21 2004 | Siemens Corporation | System and method for on-road detection of a vehicle using knowledge fusion |
7750840, | Dec 04 2007 | Raytheon Company | Method and apparatus for assessing contact clusters |
7787703, | May 11 2005 | Xerox Corporation | Method and system for extending binary image data to contone image data |
7859569, | Aug 23 2004 | Intergraph Software Technologies Company | Real-time image stabilization |
7869662, | Apr 22 2005 | Keysight Technologies, Inc | System architecture for sensing an absolute position using a target pattern |
8385647, | Jan 25 2006 | KOFAX, INC | Method of image analysis using sparse Hough transform |
8462218, | Aug 23 2004 | Intergraph Software Technologies Company | Real-time image stabilization |
8731815, | Sep 18 2009 | Holistic cybernetic vehicle control | |
8879120, | Jan 12 2012 | KOFAX, INC.; KOFAX, INC | Systems and methods for mobile image capture and processing |
8885229, | May 03 2013 | KOFAX, INC.; KOFAX, INC | Systems and methods for detecting and classifying objects in video captured using mobile devices |
8958605, | Feb 10 2009 | KOFAX, INC | Systems, methods and computer program products for determining document validity |
8971587, | Jan 12 2012 | KOFAX, INC. | Systems and methods for mobile image capture and processing |
8989515, | Jan 12 2012 | KOFAX, INC.; KOFAX, INC | Systems and methods for mobile image capture and processing |
9058515, | Jan 12 2012 | KOFAX, INC. | Systems and methods for identification document processing and business workflow integration |
9058580, | Jan 12 2012 | KOFAX, INC. | Systems and methods for identification document processing and business workflow integration |
9070023, | Sep 23 2013 | Toyota Jidosha Kabushiki Kaisha | System and method of alerting a driver that visual perception of pedestrian may be difficult |
9137417, | Mar 24 2005 | KOFAX, INC | Systems and methods for processing video data |
9141926, | Apr 23 2013 | KOFAX, INC | Smart mobile application development platform |
9158967, | Jan 12 2012 | KOFAX, INC.; KOFAX, INC | Systems and methods for mobile image capture and processing |
9165187, | Jan 12 2012 | KOFAX, INC.; KOFAX, INC | Systems and methods for mobile image capture and processing |
9165188, | Jan 12 2012 | KOFAX, INC.; KOFAX, INC | Systems and methods for mobile image capture and processing |
9208536, | Sep 27 2013 | KOFAX, INC | Systems and methods for three dimensional geometric reconstruction of captured image data |
9253349, | May 03 2013 | KOFAX, INC. | Systems and methods for detecting and classifying objects in video captured using mobile devices |
9311531, | Mar 13 2013 | KOFAX, INC | Systems and methods for classifying objects in digital images captured using mobile devices |
9342742, | Jan 12 2012 | KOFAX, INC.; KOFAX, INC | Systems and methods for mobile image capture and processing |
9355312, | Mar 13 2013 | KOFAX, INC | Systems and methods for classifying objects in digital images captured using mobile devices |
9386235, | Nov 15 2013 | KOFAX, INC. | Systems and methods for generating composite images of long documents using mobile video data |
9396388, | Feb 10 2009 | KOFAX, INC. | Systems, methods and computer program products for determining document validity |
9483794, | Jan 12 2012 | KOFAX, INC. | Systems and methods for identification document processing and business workflow integration |
9514357, | Jan 12 2012 | KOFAX, INC.; KOFAX, INC | Systems and methods for mobile image capture and processing |
9524444, | Oct 16 2013 | Canon Kabushiki Kaisha | Method, system and apparatus for determining a contour segment for an object in an image captured by a camera |
9576272, | Feb 10 2009 | KOFAX, INC | Systems, methods and computer program products for determining document validity |
9576370, | Feb 17 2014 | Canon Kabushiki Kaisha | Distance measurement apparatus, imaging apparatus, distance measurement method and program |
9584729, | May 03 2013 | KOFAX, INC. | Systems and methods for improving video captured using mobile devices |
9747269, | Feb 10 2009 | KOFAX, INC. | Smart optical input/output (I/O) extension for context-dependent workflows |
9747504, | Nov 15 2013 | KOFAX, INC. | Systems and methods for generating composite images of long documents using mobile video data |
9754164, | Mar 13 2013 | KOFAX, INC. | Systems and methods for classifying objects in digital images captured using mobile devices |
9760788, | Oct 30 2014 | KOFAX, INC.; KOFAX, INC | Mobile document detection and orientation based on reference object characteristics |
9767354, | Feb 10 2009 | KOFAX, INC. | Global geographic information retrieval, validation, and normalization |
9769354, | Mar 24 2005 | KOFAX, INC | Systems and methods of processing scanned data |
9779296, | Apr 01 2016 | KOFAX, INC. | Content-based detection and three dimensional geometric reconstruction of objects in image and video data |
9946954, | Sep 27 2013 | KOFAX, INC. | Determining distance between an object and a capture device based on captured image data |
9996741, | Mar 13 2013 | KOFAX, INC. | Systems and methods for classifying objects in digital images captured using mobile devices |
Patent | Priority | Assignee | Title |
6418242, | Nov 18 1998 | PROJECT GIANTS, LLC | Efficient detection of error blocks in a DCT-based compressed video sequence |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Dec 16 2002 | SRINIVASA, NARAYAN | HRL Laboratories, LLC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 013617 | /0425 | |
Dec 23 2002 | HRL Laboratories, LLC | (assignment on the face of the patent) | / |
Date | Maintenance Fee Events |
Sep 23 2011 | M1551: Payment of Maintenance Fee, 4th Year, Large Entity. |
Jan 29 2016 | M1552: Payment of Maintenance Fee, 8th Year, Large Entity. |
Mar 23 2020 | REM: Maintenance Fee Reminder Mailed. |
Sep 07 2020 | EXP: Patent Expired for Failure to Pay Maintenance Fees. |
Date | Maintenance Schedule |
Aug 05 2011 | 4 years fee payment window open |
Feb 05 2012 | 6 months grace period start (w surcharge) |
Aug 05 2012 | patent expiry (for year 4) |
Aug 05 2014 | 2 years to revive unintentionally abandoned end. (for year 4) |
Aug 05 2015 | 8 years fee payment window open |
Feb 05 2016 | 6 months grace period start (w surcharge) |
Aug 05 2016 | patent expiry (for year 8) |
Aug 05 2018 | 2 years to revive unintentionally abandoned end. (for year 8) |
Aug 05 2019 | 12 years fee payment window open |
Feb 05 2020 | 6 months grace period start (w surcharge) |
Aug 05 2020 | patent expiry (for year 12) |
Aug 05 2022 | 2 years to revive unintentionally abandoned end. (for year 12) |