Various embodiments of the present invention relate to a method, system and computer program product for detecting and recognizing text in the images captured by cameras and scanners. First, a series of image-processing techniques is applied to detect text regions in the image. Subsequently, the detected text regions pass through different processing stages that reduce blurring and the negative effects of variable lighting. This results in the creation of multiple images that are versions of the same text region. Some of these multiple versions are sent to a character-recognition system. The resulting texts from each of the versions of the image sent to the character-recognition system are then combined to a single result, wherein the single result is detected text.
|
8. A system for detecting and recognizing text in an image, the system comprising:
one or more processors; and
memory, including instructions that, when collectively executed by the one or more processors, cause the system to at least:
generate first output by using an optical character recognizer to recognize the text in at least one binary chip formed at least in part by processing an input image;
generate second output by using the optical character recognizer to separately and independently recognize the text in an output image formed at least in part by processing the input image; and
analyze at least the generated first output and the generated second output to form consensus output.
15. A non-transitory computer program product for use with a computer, the computer program product comprising a computer usable medium having computer-readable program code embodied therein for detecting and recognizing text in an image, the computer program product performing:
obtain an output image formed at least in part by processing an input image, the output region comprising at least one detected text region;
obtain at least one binary chip formed at least in part by separately processing the input image;
obtain first output by at least recognizing the text in each of the at least one binary chip;
obtain second output by at least separately and independently recognizing the text from the text regions of the output image; and
analyze at least the generated first output and the generated second output to form consensus output.
1. A computer-implemented method for detecting and recognizing text in an image, the method comprising:
under the control of one or more computer systems configured with executable instructions,
obtaining an output image that differs from an input image in at least one visual aspect, the output image comprising one or more text regions;
separately processing the input image to create at least one binary chip, each binary chip corresponding to a text region of the output image;
generating first output by at least recognizing the text in each binary chip from the text region corresponding to the binary chip using an optical character recognizer;
generating second output by at least separately and independently recognizing the text from the one or more text regions of the output image using the optical character recognizer; and
analyzing at least the generated first output and the generated second output to form consensus output.
2. The computer-implemented method of
3. The computer-implemented method of
4. The computer-implemented method of
detecting the one or more text regions by at least filtering and segmenting the input image and intersecting the filtered and segmented input image with a mask created from a plurality of bounding boxes, each bounding box enclosing a connected component, each connected component including a plurality of pixels comprising the image and connected on the basis of a predetermined pixel intensity and predefined distance between the pixels.
5. The computer-implemented method of
6. The computer-implemented method of
7. The computer-implemented method of
9. The system of
10. The system of
forming a gray-level chip from the input image; and
processing the formed gray-level chip to form the binary chip.
11. The system of
12. The system of
13. The system of
14. The system of
16. The non-transitory computer program product of
17. The non-transitory computer program product of
18. The non-transitory computer program product of
19. The non-transitory computer program product of
20. The non-transitory computer program product of
|
This application is a continuation of allowed U.S. application Ser. No. 12/284,283, entitled “Method and System for Detecting and Recognizing Text in Images,” filed Sep. 19, 2008; which is a continuation of abandoned U.S. application Ser. No. 12/009,913, entitled “Method and System for Detecting and Recognizing Text in Images,” filed Jan. 23, 2008 of which the full disclosure of these applications is incorporated herein by reference for all purposes.
The present invention involves a method and system for the detection and recognition of text in images. The quality of these images varies due to improper focus, motion blur, lighting variations or noise.
Printed text can be detected and recognized by using an optical character-recognition engine (OCR). The OCR technology that is currently used runs on images of clear text with modern fonts printed against clean backgrounds. In addition, the images are assumed to be created by a scanner at high resolution. The scanning process produces high quality sharp images of text under uniform illumination. This is also true when high resolution cameras are used for scanning under uniform illumination. However, commercial OCR and conventional OCR technologies do not work well when the fonts are unusual or when the text is printed against a non-uniform image background. Also, commercial OCR technology does not work well when the images are taken with hand-held cameras whose viewpoint is no longer fronto-parallel to the text and lighting changes or ambient illumination may affect the results. In a fronto-parallel view, a rectangle is imaged as a rectangle and the world and image rectangle have the same aspect ratio.
Images produced by cameras on mobile computational devices such as cell phones and personal digital assistants (PDAs) are often of poor quality because they have inexpensive optics, small apertures, slow shutters and in many cases fixed-focus lenses. Such cameras often show blur (both motion and focus blur) and noise.
Moreover, in situations where the lenses are close to the object, the depth of field is poor and the blur problem grows worse with the introduction of varying amounts of blur through the images produced by different lenses.
Illumination variations are an additional problem and cannot be rectified easily using the flash on cell phone cameras, since the flash on these devices is usually not strong enough and tends to create illumination variations. The OCR technology used currently often works poorly on the text in such images.
Some efforts have been made to detect text against the general image in the background and then extract and clean the text to create black text against a white background, which is then passed on to an OCR for text-detection. Examples of such efforts can be seen in Wu et al (V. Wu, R. Manmatha, and E. M. Riseman “TextFinder: An Automatic System to Detect and Recognize Text In Images,” IEEE PAMI, vol. 21, no. 11, pp. 1224-1229, November 1999) and more recently in Chen and Yuille (X. Chen and A. Yuille, “Detecting and Reading Text in Natural Scenes”, CVPR 2004, vol. 2, pp. 366-373). The effort by Wu et al was designed mainly for scanned images, while the more recent work of Chen and Yuille was designed for street signs taken by high quality cameras. Neither of these is designed to rectify images of poor quality with problems like blur.
There is, therefore, a need for a method and system for detecting and extracting text in the images of varying quality produced by mobile computational devices such as cell phones and PDAs.
An object of the present invention is to provide a method, system and computer program product for detecting and recognizing text in an image with lighting variations.
Another object of the present invention is to provide a method, system and computer program product for detecting and recognizing text in blurred images. These blurred images can be both focus and motion blurred.
Yet another object of the present invention is to provide a method, system and computer program product for detecting and recognizing text not only in images captured by camera phones but also in images from cameras and scanners.
Various embodiments of the present invention relate to a method, system and computer program product for detecting and recognizing text in the images captured by cameras and scanners. To begin with, a series of image-processing techniques is applied to detect text regions in the image captured by cameras and scanners. These images are blurred, noisy, or contain lighting variations. Subsequently, the detected text regions undergo different processing stages that reduce blurring and the negative effects of lighting variations. This results in the creation of multiple images that are versions of the same text region. Some of these multiple versions are sent to a character-recognition system. The resulting text from all of these images sent to the character-recognition system is then combined to a single text result.
The embodiments of the present invention will hereinafter be described in conjunction with the appended drawings, provided to illustrate and not to limit the invention, wherein like designations denote like elements, and in which:
Various embodiments of the present invention relate to a method, system and computer program product for detecting and recognizing text in images captured by camera phones and scanners. First, a series of image-processing techniques is applied to detect text regions in the image. Subsequently, the detected text regions pass through different processing stages to reduce blurring and other negative effects of variable lighting. Thus, multiple images including the processed image and the original image are obtained. Such images are used to create multiple versions of the same text region. Some of these multiple versions are sent to an optical character-recognition system. The resulting text from all of these images sent to the character-recognition system is then combined into a single text result.
A word-finding module 210 removes the potential text regions from the Image C that are not consistent with words to create an Image D. In particular, word-finding module 210 performs an image-closing operation on the Image C. The image closing operation is derived from operations of erosion and dilation. For example a gray-level closing operation consists of gray-level dilation of the image followed by gray-level erosion. The image closing operation is performed with the help of a structuring element. The structuring element consists of a pattern specified as the coordinates of a number of discrete points relative to an origin. For example, if Cartesian coordinates are used, the structuring element can be represented as an image on a rectangular grid. The result of the image closing operation is determined by the structuring element used. The image closing operation preserves background regions that have a shape similar to the structuring element, while eliminating all other regions of background pixels. The background pixels can be regions in the image that are not of interest or are not being considered as prime object in the image. The background pixels can be pixels with an intensity value of 0. The word finding module 210 then checks to verify whether the properties of connected components created after the image-closing operation are consistent with the properties associated with the words by applying a series of predefined word constraints. For example, it checks whether the aspect ratio of the connected component falls within a certain range. Components that are not consistent with at least one of these predefined word constraints are removed to create the Image D. Output text-image-creation module 212 finds the connected components in the Image D and creates a bounding box for each connected component. The connected components computing algorithm finds regions of connected pixels on the basis of intensity value of the pixels and predefined special adjacency. A bounding box is the smallest rectangle enclosing a given connected component. The bounding boxes can be of different sizes depending on the size of the connected component or the text. The output text-image-creation module then intersects the Image A with a mask created from the union of all bounding boxes to create an OutputImage2, the, OutputImage2 comprising detected text regions. Any pixel in the Image A that is not within the bounding boxes is zeroed out while retaining the regions of the Image A within the bounding boxes to create OutputImage2. This module also extracts gray level image areas corresponding to the bounding boxes to form gray-level image chips. Each of the gray-level image chips is a new gray-level image comprising a region from the gray-level image corresponding to a single bounding box. Hence, the gray-level image chips are sub-parts of the gray-level image, wherein each of the gray-level image chips corresponds to a bounding box in Image D.
In accordance with an embodiment of the present invention, filtering module 302 uses a Laplacian-of-Gaussian (LOG) filter, which responds strongly to line-like and oar-like entities. The response of the LOG filter depends on the relative contrast across the line-like and bar-like regions and hence is not susceptible to illumination variations. This is because illumination variations involve gradual changes in light intensity which are removed by the smoothening action of the Gaussian filter of the LOG filter. In the smoothening action, sharp edges of the image are blurred. Blur causes lines and bars to be diffused. The LOG filter has a local support region, which is a function of the standard deviation of the LOG filter. An appropriate adjustment of the standard deviation makes the local support region large enough to take into account the averaged response over the diffused bar. Small noise regions are also created by the LOG filter. However, their local support is usually very small, and hence the response for most noise regions is small. Thresholding module 304 then eliminates many of these noisy regions but tends to preserve strong bar-like regions. Examples of a thresholding algorithm include, but are not limited to, an adaptive thresholding algorithm and a constant thresholding algorithm.
Therefore, in accordance with an embodiment of the present invention, a line constraint-detection module 404 computes the ratio of the perimeter to the maximum distance transformed for a connected component. The maximum distance transformed for the connected component can be the maximum dimension for the bounding box of the connected component. The maximum distance transformed finds the thickness of the connected component. If the ratio is larger than a threshold, the connected component is removed by line-removing processor 406. The threshold is determined empirically. The remaining connected components form the Image B.
In accordance with an embodiment of the present invention, it is assumed that characters must be close to and similar to other nearby characters with which they form a word. In one such embodiment of the present invention, character constraint-detection module 504 decides, on the basis of predefined character constraints, if the connected component is a part of the character or not. This is performed by checking whether the centroids of adjacent connected components are approximately along the same line, whether the height and width of the adjacent characters fall within a specified range of ratios among them, and whether the spacing between characters is almost similar. The parameters used for making these decisions are determined empirically. Additionally, to allow for the breaking up of a character into two or more connected components, an additional check is made to determine the multiple components that can be treated as a single unit to satisfy the above constraints. Components that do not satisfy at least one of the character constraints are removed by non-text removing processor 506 from potential text regions in the Image B, to create the Image C.
In accordance with an embodiment of the present invention, these predefined word constraints may include the aspect ratio, height and width of the bounding box of the connected component in a prescribed range, and the number of pixels in the connected component divided by the width and height of the bounding box of the connected component within a prescribed range. The bounding box for a connected component is the smallest rectangle enclosing the connected component. The bounding boxes can be of different sizes depending on the size of the connected component or the text. The prescribed ranges are determined empirically. Any connected components that do not satisfy at least one of these predefined word constraints are removed from the Image C to create the Image D. Therefore, the Image D contains a set of connected components that are likely to correspond to words.
Each gray-level image chip contains text as illustrated by 902 in
At step 1206, line-removal module 206 removes long line structures to create an Image B. The long line structures comprise long straight lines and rectangles. This can be based on a predefined ratio of the perimeter to the maximum distance transformed for a connected component. For example, line-removal module 206 first computes connected components in the Image A using connected component module 402. A connected components computing algorithm finds regions of connected pixels on the basis of intensity value of the pixels and predefined special adjacency. The line removal module 206 then uses line constraint-detection module 404 to check whether the connected components satisfy certain constraints. If the constraints show that the components are more likely to be long straight lines or rectangles, then they are removed by line-removing processor 406 to create an Image B. The long straight lines and rectangles are assumed not to correspond to text. At step 1208, character-finding module 208 removes connected components in the image B that are not likely to be characters to identify potential text regions in the Image B. It first finds connected components in the Image B using connected component module 502, then checks whether the connected components satisfy at least one character constraint, based on the properties of the neighboring text, and whether they are likely to be characters using character constraint-detection module 504. It is assumed that the neighboring text characters share certain properties; the spacing between such characters is roughly constant, and they are approximately aligned vertically. Components that do not satisfy at least one of the constraints are removed from the Image B by non-text removing processor 506 to create an Image C.
At step 1210, word-finding module 210 performs an image-closing operation on the Image C using image-closing operation module 602. At step 1212, word-finding module 210 finds connected components using connected component module 604 and then checks whether the resulting components satisfy certain predefined word constraints using text word constraint-detection module 606. Components that do not satisfy at least one of the predefined word constraints are removed by non-text words removing processor 608 to create an Image D. Thus, non-text removing processor 608 removes the potential text region not consistent with words from the Image C to create the Image D.
At step 1214, output text-image-creation module 212 finds the bounding boxes for the connected components in the Image D, using bounding box module 704. Bounding boxes are the smallest rectangles enclosing a connected component. At step 1216, output text-image-creation module 212 uses output image module 708 to compute the intersection of the Image A and the union of the bounding boxes from the Image D to create an OutputImage2, the OutputImage2 comprising detected text regions. Any pixel in the Image A that is not within the bounding boxes is zeroed out while retaining the regions of the Image A within the bounding boxes to create OutputImage2. At step 1218, gray-level image chips are also created by extracting, for each bounding box, the corresponding image area from, the gray-level image to create gray-level image chips. At step 1220, each of the gray-level image chips is processed to create binary chips. In the step of processing, each of the gray-level image chips is first histogram-equalized using histogram-equalization module 802 and then thresholded using thresholding module 804 to create binary chips.
At step 1222, the text in the OutputImage2 and the binary chips is recognized. This is done by transforming the OutputImage2 and the binary chips into the electronic representation of their text by optical character-recognizes module 108. At step 1224, the electronic representations of text for the binary chips and the OutputImage2 are combined into a single output, wherein the single output is detected text. Possible ways of combination include either taking a majority vote or taking the logical OR of the text outputs.
Various embodiments of the present invention provide a method, system and computer program product for detecting and recognizing a text in an image with lighting variations. The image is captured by a computational device with a camera, such that the camera is not fronto-parallel to the text. This results in lighting variations in the image, which makes detection of the text in the image difficult and reduces the accuracy of the text. This method detects text in such images by improving the quality of the images by using various image-processing techniques.
Further, various embodiments of the present invention provide a method, system and computer program product for detecting and recognizing text in highly blurred images. This process is performed by decreasing the blurriness of these images by performing various image-processing techniques.
Furthermore, various embodiments of the present invention provide a method, system and computer program product for recognizing text, not only in camera phone images, but also from other cameras and scanners.
The system, as described in the present invention or any of its components, may be embodied in the form of a computer system. Typical examples of a computer system include a general-purpose computer, a programmed microprocessor, a micro-controller, a peripheral integrated circuit element, and other devices or arrangements of devices that are capable of implementing the steps constituting the method of the present invention.
The computer system comprises a computer, an input device, a display unit and the Internet. The computer comprises a microprocessor, which is connected to a communication bus. The computer also includes a memory, which may include Random Access Memory (RAM) and Read Only Memory (ROM). Further, the computer system comprises a storage device, which can be a hard disk drive or a removable storage drive such as a floppy disk drive, an optical disk drive, and so forth. The storage device can also be other similar means for loading computer programs or other instructions on the computer system.
The computer system executes a set of instructions that are stored in at least one storage elements, to process input data. The storage elements may also hold data or other information, as desired, and may be an information source or physical memory element present in the processing machine.
The set of instructions may include various commands that instruct the processing machine to execute specific tasks such as the steps constituting the method of the present invention. The set of instructions may be in the form of a software program. The software may be in various forms such as system software or application software. Further, the software may be in the form of a collection of separate programs, a program module with a larger program, or a portion of a program module. The software may also include modular programming in the form of object-oriented programming. Processing of input data by the processing machine may be in response to user commands, to the results of previous processing, or to a request made by another processing machine.
While the embodiments of the invention have been discussed and described, the invention is not limited to these embodiments only. A number of changes and modifications can be thought of, without moving away from the scope of the invention, as discussed in the claims.
Manmatha, Raghavan, Ruzon, Mark A
Patent | Priority | Assignee | Title |
10402590, | Sep 11 2015 | Infosys Limited | Method and system for masking of cheque image based on user access privileges |
8594424, | Dec 08 2008 | Samsung Electronics Co., Ltd | Apparatus and method for recognizing characters using a camera |
8818098, | Dec 08 2008 | Samsung Electronics Co., Ltd | Apparatus and method for recognizing characters using a camera |
8977072, | Jan 23 2008 | A9 COM, INC | Method and system for detecting and recognizing text in images |
9047528, | Feb 19 2013 | Amazon Technologies, Inc | Identifying characters in grid-based text |
9430704, | Jan 16 2015 | Sony Corporation | Image processing system with layout analysis and method of operation thereof |
9530069, | Jan 23 2008 | A9.COM, INC. | Method and system for detecting and recognizing text in images |
9569692, | Oct 31 2014 | CITIBANK, N A | Context-based image recognition for consumer market research |
9582727, | Jan 16 2015 | Sony Corporation | Text recognition system with feature recognition and method of operation thereof |
9646202, | Jan 16 2015 | Sony Corporation | Image processing system for cluttered scenes and method of operation thereof |
9710723, | Oct 31 2014 | CITIBANK, N A | Context-based image recognition for consumer market research |
Patent | Priority | Assignee | Title |
5065437, | Dec 08 1989 | Technology Licensing Corporation | Identification and segmentation of finely textured and solid regions of binary images |
5134668, | Feb 08 1990 | International Business Machines Corporation | Masked combinations of video slices for computer display |
5201011, | Nov 19 1991 | Xerox Corporation | Method and apparatus for image hand markup detection using morphological techniques |
5790696, | Mar 29 1995 | International Business Machines Corporation | Image segmentation apparatus and a character recognition apparatus |
5809167, | Apr 15 1994 | Canon Kabushiki Kaisha | Page segmentation and character recognition system |
6026177, | Aug 29 1995 | ASIA VISION TECHNOLOGY LIMITED | Method for identifying a sequence of alphanumeric characters |
6438265, | May 28 1998 | International Business Machines Corporation | Method of binarization in an optical character recognition system |
6865290, | Apr 25 2000 | Ricoh Company, LTD | Method and apparatus for recognizing document image by use of color information |
6888957, | Mar 29 2001 | Ciena Corporation | System and method for detecting blobs on an end surface of an optical fiber |
6920247, | Jun 27 2000 | HEWLETT-PACKARD DEVELOPMENT COMPANY, L P | Method for optical recognition of a multi-language set of letters with diacritics |
7200266, | Aug 27 2002 | Princeton University | Method and apparatus for automated video activity analysis |
7646921, | Sep 11 2006 | GOOGLE LLC | High resolution replication of document based on shape clustering |
7983483, | Nov 10 2005 | PFU Limited | Automatic character recognition system and computer product |
20060013511, | |||
20070058856, | |||
20080101726, | |||
20090148043, |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Aug 03 2011 | A9.COM, INC. | (assignment on the face of the patent) | / |
Date | Maintenance Fee Events |
Jun 20 2016 | M1551: Payment of Maintenance Fee, 4th Year, Large Entity. |
Jun 18 2020 | M1552: Payment of Maintenance Fee, 8th Year, Large Entity. |
Jun 18 2024 | M1553: Payment of Maintenance Fee, 12th Year, Large Entity. |
Date | Maintenance Schedule |
Dec 18 2015 | 4 years fee payment window open |
Jun 18 2016 | 6 months grace period start (w surcharge) |
Dec 18 2016 | patent expiry (for year 4) |
Dec 18 2018 | 2 years to revive unintentionally abandoned end. (for year 4) |
Dec 18 2019 | 8 years fee payment window open |
Jun 18 2020 | 6 months grace period start (w surcharge) |
Dec 18 2020 | patent expiry (for year 8) |
Dec 18 2022 | 2 years to revive unintentionally abandoned end. (for year 8) |
Dec 18 2023 | 12 years fee payment window open |
Jun 18 2024 | 6 months grace period start (w surcharge) |
Dec 18 2024 | patent expiry (for year 12) |
Dec 18 2026 | 2 years to revive unintentionally abandoned end. (for year 12) |