Described is a technique for optimization an image for facial detection. More specifically, described is a process of predicting the location of a face within an image and adjusting image settings based on at least a portion of the predicted location of the face. An image may be adjusted based on the characteristics of a metering region, which may be selected prior to performing facial detection. For example, the metering region may be a specified shape with dimensions equal to a certain percentage of the input image and placed at a specified location. The result of using such a metering region is that the image adjustments may be based on a portion of the face, and therefore, may be optimized for facial detection.
|
11. A method comprising:
receiving, by a computing device, an image captured by a camera;
receiving, by the computing device, an indication to perform facial recognition on the image;
selecting, by the computing device, in response to the indication, a metering region within a portion of the captured image based on a set of coordinates, wherein the metering region is selected prior to locating a position of a face within the image, wherein the selected metering region is smaller in size than a first metering region, the first metering region configured as part of a default capture setting of the camera;
adjusting, by the computing device, the image based on the selected metering region; and
detecting, by the computing device, the face within the adjusted image.
1. A method comprising:
receiving, by a computing device, a live image captured by a camera, wherein the live image continuously captures an image within a field of view of the camera;
receiving, by the computing device, an indication to perform facial recognition on a frame of the live image;
selecting, by the computing device, in response to the indication, a metering region within a portion of the frame prior to attempting to locate a position of a face within any portion of the frame, wherein the selected metering region is smaller in size than a first metering region, the first metering region configured as part of a default capture setting of the camera;
adjusting, by the computing device, the frame based on the selected metering region; and
detecting, by the computing device, the face within the adjusted frame.
19. A method comprising:
receiving, by a computing device, a live image captured by a camera, wherein the live image continuously captures an image within a field of view of the camera;
receiving, by the computing device, an indication to perform facial recognition on a frame of the live image;
during a facial detection step of the facial recognition, selecting, by the computing device, in response to the indication, a metering region within a portion of the frame prior to attempting to locate a position of a face within any portion of the frame, wherein the selected metering region is smaller in size than a first metering region, the first metering region configured as part of a default capture setting of the camera;
adjusting, by the computing device, the frame based on the selected metering region; and
detecting, by the computing device, the face within the adjusted frame.
2. The method of
3. The method of
5. The method of
6. The method of
7. The method of
displaying a preview of the captured live image; and
receiving a user input selecting a portion of the preview and wherein the selected metering region is based on the received user input.
8. The method of
9. The method of
10. The method of
comparing facial features of the face to features in a stored reference template; and
determining, based on the comparison, a recognition confidence score.
14. The method of
15. The method of
16. The method of
17. The method of
comparing facial features of the face to features in a stored reference template; and
determining, based on the comparison, a recognition confidence score.
18. The method of
making a determination that the recognition confidence score is above a threshold value; and
in response to determination, identifying a user associated with the facial features.
|
This application is a continuation of co-pending U.S. patent application Ser. No. 15/014,222, filed Feb. 3, 2016, which is a divisional of U.S. patent application Ser. No. 13/655,817, filed Oct. 19, 2012, both of which are incorporated by reference herein in their entirety and for all purposes.
Devices with a built-in camera may have the ability to authenticate a user by performing facial recognition. Facial recognition methods often process an image in two steps. The first step, which may be described as detection, locates a face within an image. The second step, which may be described as recognition, analyzes the detected face to determine a similarity between the detected face and a stored face or template. To achieve accurate results, the facial recognition processes may depend upon the quality of detected images. When capturing images, a camera may automatically adjust capture settings based on image characteristics. These settings may include automatic adjustments to exposure and/or gain. For example, when an image condition includes a face with strong backlighting, the automatic adjustment may reduce exposure in the image to account for the backlighting. These image settings, however, may be optimized for capturing the most visually appealing pictures.
In an implementation, described is a method of optimizing a captured image for facial detection. In response to receiving an indication to perform facial detection, a metering region within a portion of the captured image may be selected. The metering region may be based on an expected location of a face and may be selected prior to detecting a face within the image. The image may be adjusted based on the metering region and a face may be detected within the adjusted image. The image may be optimized by adjusting an exposure, and/or another setting including an automatic gain control (AGC), contrast, saturation, and the like.
In an implementation, a captured image may be optimized based on a predefined and static metering region. The metering region may be described based on a size. The size of the metering region may be described as being within a percentage range of the image. The metering region may also be described based on a relative location within the image. For example, the metering region may be centered, within a specified portion of the image, or include at least two quadrants of the image.
In an implementation, described is a facial detection method including executing a facial detection algorithm on a captured image a first time and determining that a face is not detected within the image. A metering region within a portion of the image may be selected and the image may be adjusted based on the metering region. The facial detection algorithm may execute on the adjusted image a second time and may detect a face. The first execution may be based on a metering region of the entire image and the second execution may be based on a metering region optimized for facial detection.
In an implementation, described is a device for optimizing an image for facial detection. The device may include a camera for capturing the image, a display previewing the captured image, and a processor for adjusting the captured image based on a metering region within a portion of the image. The processor may adjust the camera based on a metering region and a face may be detected based on the adjusted image. A preview of the captured image may be displayed and the metering region may be based on a positioning guide displayed on the previewed image. The display may be a touch sensitive display and the metering region may be based on a user input touching a portion of the previewed image.
The accompanying drawings, which are included to provide a further understanding of the disclosed subject matter, are incorporated in and constitute a part of this specification. The drawings also illustrate implementations of the disclosed subject matter and together with the detailed description serve to explain the principles of implementations of the disclosed subject matter. No attempt is made to show structural details in more detail than may be necessary for a fundamental understanding of the disclosed subject matter and various ways in which it may be practiced.
Described is a process of improving facial detection in an image by adjusting image settings. Implementations describe adjusting the image settings based on a metering region that optimizes the image for detecting a face. The metering region may use a relevant subset of the image and may be based on an expected location of at least a portion of a face within the image. By positioning the metering region in an expected location of a face, the metering region may be selected prior to actually detecting a face.
The device 50 may include or be part of a variety of types of devices, such as a mobile phone (including a “smartphone”), tablet computer, netbook, laptop, desktop, personal digital assistant (“PDA”), set-top box, television, and/or watch, among others. The processor 52 may be any suitable programmable control device and may control the operation of one or more processes, such as the facial detection process as discussed herein, as well as other processes performed by the device 50.
The memory 54 may include one or more different types of memory which may be used for performing device functions. For example, the memory 54 may include cache, ROM, and/or RAM. The bus 51 may provide a data transfer path for transferring between components of the device 50. The CODEC 55 may be included to process digital and analog audio and/or video signals to produce sound, voice, and multimedia. The display 58 may provide visual output and may include a touch-sensitive screen.
The storage 57 may store multimedia (e.g. video, music, and photos, etc.), software (e.g., for implementing various functions on device 50), and any other suitable data. The storage 57 may include a storage medium, such as a hard-drive, solid state drive, flash drive, and the like. The storage 57 may be integral with the device 50 or may be separate and accessed through an interface to receive a memory card, USB drive, optical disk, and the like.
The user input 56 may allow a user to interact with the device 50. For example, the user input 56 may include buttons, a keypad, a touch screen, and the like. The communications circuitry 53 may include circuitry for wireless communications for short-range and/or long range communication. For example, the wireless communication circuitry may include Wi-Fi enabling circuitry for one of the 802.11 standards, and circuitry for other wireless network protocols including Bluetooth, the Global System for Mobile Communications (GSM), and code division multiple access (CDMA) based wireless protocols. Communications circuitry 53 may also include circuitry that enables the device 50 to be electrically coupled to another device (e.g. a computer or an accessory device) and communicate with that other device.
The camera 59 enables the device 50 to capture still images, video, or both. The camera 59 may capture an image or series of images continuously, periodically, at select times, and/or under select conditions. The camera 59 may be part of or coupled to one or both a front-facing camera or rear-facing camera of the device 50.
The device 50 may be battery-operated and portable so as to allow a user to communicate with others, listen to music, play games or video, record video or take pictures, or control other devices. The device 50 may be relatively compact which enables a user to easily manipulate the device's position, orientation, and movement. Accordingly, the device 50 may provide techniques of sensing such changes in position, orientation, and movement to enable a user to interface with or control the device 50 by affecting such changes. Further, the device 50 may include a vibration source, under the control of processor 52, for example, to facilitate sending motion, vibration, and/or movement information to a user related to an operation of the device 50. Other devices or components may be part of or connected to the device 50, and conversely, all of the components shown in
In step 23, the device 50 may receive an indication to perform facial detection. The indication for facial detection may be explicit or implicit. An explicit request to perform facial detection may include a user selecting a facial detection setting on the device 50 for image capture. This selection may be made in real-time, for example, while capturing a live image, or prior to capturing an image such as setting an image capture preference (e.g. facial detection mode). The explicit indication may also include a request to perform an image adjustment for facial detection and/or recognition on a stored image (e.g. a picture). An implicit indication may include initiating or requesting a process that includes facial detection as a sub-process or step. For example, a facial recognition process may include performing an initial or concurrent step of facial detection. The device 50 may have an access control feature that authenticates a user of the device by performing facial recognition. Accordingly, the indication may include various forms of a request for user authentication. Facial recognition as a form of authentication may be common for many applications in handheld devices. Facial recognition is particularly convenient for devices with a front-facing camera because they may efficiently perform facial recognition when a user is located in a position to view a display 58. The request for authentication may be automatic or may be initiated manually by the user. For example, the authentication request may occur upon a request to “unlock” the device from a “locked” state. This may occur automatically, for example, when the device 50 “awakens” from a powered-down or “sleep” state, or it may occur manually, for example, by direct user input. The device 50 may include controlling user access at an application level. Accordingly, another example of direct user input may include launching an access controlled application to initiate an authentication request.
In step 24, a metering region may be selected. The selection may be in response to the indication to perform facial recognition. The response may include an immediate response, or it may include a deferred response that may occur during the detection process. For example, upon receiving an indication that a facial detection process has been initiated, the device 50 may first activate the camera 59. Accordingly, the selection of the metering region may be deferred until the camera 59 is activated and initiates image capture. The selection of a metering region may occur without, or prior to, detecting a face within an image. Conventional facial recognition programs typically perform facial detection first, and then optimize the image based on image characteristics of the detected face. Facial detection methods may include detecting a face based on a shape, skin color, relative positioning of facial features, etc. In certain circumstances, however, the facial detection may not be possible. For example, when capturing images, a camera may automatically adjust capture settings based on characteristics of the entire image. For example, these settings may include automatic adjustments to exposure or other images settings. These image settings may be optimized for capturing the most visually appearing pictures. At this stage, these adjustments may not be optimized for facial detection because a face has not been detected yet. For example, when an image condition includes a face with strong backlighting, the automatic adjustment may reduce exposure in the image to account for the backlighting. This adjustment may alter the image to a point where there is insufficient contrast in the region of the face. As a result, the device 50 may not be able to perform facial detection.
Implementations of the disclosed subject matter may optimize an image without the requirement to perform facial detection. More specifically, implementations may select a metering region by predicting an expected location of a face. Accordingly, the metering region may provide characteristics of a portion face and not the entire image. For example, implementations may predefine the metering region to include a central region of a captured image. A user capturing an image of their face is likely to place their face within the center or close to the center of a field of view of a camera 59. Accordingly, there may be a high likelihood that the metering region may include at least a portion of the user's face, and as a result, the image may be adjusted according to a more relevant subset of the image for facial detection. Thus, the image may be optimized without, or prior to, performing facial detection.
The metering region may be described relative to the captured image. The captured image may be described in various manners including a coordinate space, quadrants, pixels values based on a resolution, zones, and the like. For example, using a coordinate space, the lower left most point of the image may have a value of −1000, −1000, and the upper right most point may have a value of 1000, 1000. Accordingly, a center point of the image would be 0, 0. A coordinate space may also be based on resolution. For example, a 1280×800 resolution screen may have 1, 1 (or 0, 0) as the lower left most point and 1280, 800 as the upper right most point. A metering region may be defined by its size, position, behavior and other characteristics. The metering region may be described in terms of a percentage size of the image and a relative position. For example, the metering region may be centered within the image and have a size of 20% of the image. In some cases, the metering region may be described as being within a portion of the image. A portion of the image is defined as an area less than the entire image. For example, a metering region may be within a central portion of the image. A central portion is defined as a rectangular shape centered at the center point of the image and may have a specified size. For example, using a coordinate space as described above, a central portion with a size of 50% of the image may have a coordinate definition of −500, −500 as the lower left most point and 500, 500 as the upper right most point of the rectangle. A metering region within this central portion may include the entire central portion or may have a smaller area. A metering region within a portion of an image may not necessarily be rectangular in shape and may not necessarily be centered at the center point of the portion of the image.
The position of the metering region may also be described with respect to quadrants. For example, if a metering region is centered within the image, it may include a portion of all four quadrants. If the metering region is centered only horizontally or vertically, it may only include a portion of a minimum of two quadrants.
Typically, a metering region may be defined as a set of coordinates at a programming level. It should be noted that the above methods of describing the location of the metering region and describing the organization of the image may not be explicitly defined at a programming level. A person of ordinary skill in the art, however, would understand such descriptions and select a set of coordinates accordingly. For example, a description of a metering region with a size of 20% of the image and within a central portion with a 50% size of the image, would enable a person of ordinary skill in the art to understand that the region may be defined with a set of coordinates that need only be within the −500,−500 and 500, 500 rectangular boundary in the coordinate space as described above.
The metering region may also be described by its behavior. For example, the metering region may be static or dynamic. When the metering region is static, the region does not necessarily track particular objects within the image. For example, with live images, objects may move within the image and these may or may not remain within the metering region. For instance, if a coordinate system is used, the coordinates of the metering region may remain static despite movement of objects within the image. In contrast, when the metering region is dynamic, the metering region may track a particular object within the live image and the coordinates of the metering region may adjust accordingly.
The metering region may also be predefined. For example, a set of coordinates defining the metering region may be set prior to capturing an image. Accordingly, the metering region may be positioned in a particular region regardless of the characteristics of the captured image. As described above, the metering region may be based on an expected location of a face. Typically, the expected location of a face corresponds to a central portion of the image. This positioning may be further refined by additional factors. These factors may include characteristics derived from preferences, usage history, probability, etc. For example, a usage history may determine that a user typically places the face within a lower portion of the image and not precisely in a central portion. Accordingly, the selection of the metering region may be repositioned accordingly.
In implementations, the captured image may be displayed as a preview on a display 58 of the device 50. In these situations, a positioning guide may be used to direct a user to position their face within a certain region of a field of view of the camera 59. For example, a guide may be displayed on the previewed image and a user may alter the position of the device 50 and/or their face to position their face according to the guide. Accordingly, a metering region corresponding to the displayed guide may be used to adjust the image without actually detecting the face. In this instance, it is likely that a user cooperates and positions their face according to the guide. In some cases, the positioning guide may be dynamic and the metering region may dynamically track the positioning guide. The positioning guide may include a boundary, cross-hairs, one or more boundary edges, a shape (e.g. circle or rectangle), shading, or any other form of displaying a guide on the previewed image.
The metering region may also be based on receiving a user input that selects a portion of the captured image. The user input may include an input from an input device or a touch on a on touch sensitive display. For example, a user may touch the display 58 at a location corresponding to the location of a face within a captured image. The touch may be registered by the device 50 and a metering region may be selected based on the location of the touch. This may be an effective technique when performing facial recognition on an image including multiple faces in a live image or a stored image.
The metering region may also be based on characteristics of the image. As described, a face may not necessarily need to be detected to select a metering region. The metering region may be based on a contrast level, saturation level, color, and other image characteristics. For example, a face in front of a strongly lit background may have a low contrast level. The low contrast level may be identified as a probable location of a face, and accordingly, the metering region may be within this low contrast region. Similarly, a low saturation may be identified for a metering region. Other image characteristics may also be used to distinguish between regions or identify a region that may include a face. More than one metering region may be selected and they may overlap. The metering region may also be defined with a particular weighting of pixels within the image. For example, the weights of overlapping metering regions may be added. Unlike typical facial detection processes, it may not be necessary to show an outline or boundary region of the metering region on a previewed image. Instead, the metering region and adjustment may be done transparently to the user.
In step 25, the captured image may be adjusted based on characteristics of the metering region. The adjustment may adjust a live image in real-time or may adjust a picture that may be stored. For example, an adjustment to a live image may involve dynamically adjusting the captured image in response to real-time changes in the field of view of the camera 59. The adjusting may adjust capture settings of the camera 59, or may adjust the characteristics of a frame, or a stored image (e.g. snapshot) after a picture is taken. The metering region may typically correspond to at least a portion of a face within the image even when the face is not detected. Accordingly, the metering region may be optimized based on the characteristics of a portion of a face. The image adjustments may include optimizing one or more image settings. These settings may include exposure, gain, color balance (including white-balance), focus, contrast, (which may be expressed as a ratio or dynamic range), saturation, brightness, gamma, hue, sharpness, and the like. In should be noted that these settings may be adjusted in an image and would be readily understood by a person of ordinary skill in the art.
Typically, an exposure setting may be adjusted. An exposure setting may generally relate to camera exposure parameters, such as exposure time (e.g. shutter speed), gain, and/or f-number (e.g. relative aperture). By adjusting the exposure setting, an optimum brightness may be achieved for a face within the captured image. As described, the exposure may be optimized based on a metering region which may be a subset of the image. Conventional metering may be optimized for capturing the most visually appearing pictures, and therefore, are based on metering an entire image. Instead of metering an entire image, implementations may select the most relevant subset of the image for facial detection (e.g. at least a portion of a face). Accordingly, these adjustments may not produce the images most suitable for pictures as they may be optimized for facial detection. For example, the adjustment may adjust exposure to enhance contrast in a region of the image including a face (e.g. foreground), and as a result, it may oversaturate a background of the image to a point where objects in the background may not be visible. As mentioned above, other image settings may also be adjusted to optimize facial recognition.
Returning to
In step 33, facial detection may be executed a first time. In implementations, the facial detection in step 33 may occur without adjusting the image in steps 31 and 32. In step 34, it may be determined that a face is not detected when executing facial detection the first time as the image adjustment (if performed in steps 31 and 32) may not have necessarily been optimized for facial detection. After it is determined that a face was not detected, a metering region may be selected in step 35 and the captured image may be adjusted in step 36 in a similar manner as described in steps 24 and 25 of
In implementations, once facial detection is performed, the device 50 may perform facial recognition on the detected face. The recognition process may include comparing the detected face to a stored template. For example, features of the detected face template may be compared against the features of the stored reference template and a recognition confidence score may be determined. Upon a successful facial recognition (e.g. confidence score above a specified value), the device 50 may identify the face of the user. For example, upon the completion of facial recognition, a user's face may be authenticated and the device may be accessed or “unlocked” in response to the authentication request.
Implementations described herein may be included in an article of manufacture (e.g., one or more computer program products) having, for instance, computer usable media. The media has embodied therein, for instance, computer readable program code for providing and facilitating the capabilities described herein. The article of manufacture may be included as a part of a device or sold separately.
The flow diagrams described herein are just examples. There may be variations to these diagrams or the steps (or operations) described therein without departing from the implementations described. For instance, the steps may be performed in a differing order, or steps may be added, deleted or modified. Described herein, references to “one implementation,” “an implementation,” “an example implementation,” etc., indicate that the implementation described may include a particular feature, structure, or characteristic, but every implementation may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same implementation. Further, when a particular feature, structure, or characteristic is described in connection with an implementation, such feature, structure, or characteristic may be included in other implementations whether or not explicitly described.
The foregoing description, for purpose of explanation, has been described with reference to specific implementations. However, the illustrative discussions above are not intended to be exhaustive or to limit implementations of the disclosed subject matter to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings. The implementations were chosen and described in order to explain the principles of implementations of the disclosed subject matter and their practical applications, to thereby enable others skilled in the art to utilize those implementations as well as various implementations with various modifications as may be suited to the particular use contemplated.
Patent | Priority | Assignee | Title |
11288894, | Oct 19 2012 | GOOGLE LLC | Image optimization during facial recognition |
11741749, | Oct 19 2012 | GOOGLE LLC | Image optimization during facial recognition |
Patent | Priority | Assignee | Title |
10169643, | Oct 19 2012 | GOOGLE LLC | Image optimization during facial recognition |
4561752, | Jul 26 1983 | Nippon Kogaku K.K. | Metering device of a camera |
4912495, | Jun 06 1985 | Canon Kabushiki Kaisha | Light measuring device |
4929824, | Feb 26 1988 | FUJIFILM Corporation | light metering device with detector matrix and mean value detection |
4937611, | Mar 07 1988 | FUJIFILM Corporation | Light metering device |
5023649, | Apr 20 1988 | MINOLTA CAMERA KABUSHIKI KAISHA, A CORP OF JAPAN | Exposure controlling apparatus of a camera having a plurality of areas for measuring brightness |
5146258, | Dec 24 1990 | Eastman Kodak Company | Multiple photodiode array for light metering |
6081669, | Nov 15 1996 | Asahi Kogaku Kogyo Kabushiki Kaisha | Photometric operation system of camera |
7315631, | Aug 11 2006 | FotoNation Limited | Real-time face tracking in a digital image acquisition device |
7551223, | Dec 26 2002 | Sony Corporation | Apparatus, method, and computer program for imaging and automatic focusing |
7646405, | Jul 11 2002 | 138 EAST LCD ADVANCEMENTS LIMITED | Output image adjustment of image data |
7663476, | May 17 2006 | Alpine Electronics, Inc. | Surrounding image generating apparatus and method of adjusting metering for image pickup device |
8213737, | Jun 21 2007 | FotoNation Limited | Digital image enhancement with reference images |
8233789, | Apr 07 2010 | Apple Inc. | Dynamic exposure metering based on face detection |
8325890, | Jun 06 2010 | Apple Inc. | Auto exposure techniques for variable lighting conditions |
9286509, | Oct 19 2012 | GOOGLE LLC | Image optimization during facial recognition |
20020008771, | |||
20020111773, | |||
20030071908, | |||
20040227826, | |||
20050275747, | |||
20060077264, | |||
20060262211, | |||
20070073275, | |||
20070086761, | |||
20070263090, | |||
20080225136, | |||
20080309788, | |||
20090080715, | |||
20090231468, | |||
20100002093, | |||
20100020220, | |||
20100302393, | |||
20110249961, | |||
20110298886, | |||
20130002884, | |||
20130250106, |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Oct 18 2012 | RODRIGUEZ, URIEL GERARDO | Google Inc | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 047694 | /0426 | |
Sep 29 2017 | Google Inc | GOOGLE LLC | CHANGE OF NAME SEE DOCUMENT FOR DETAILS | 047734 | /0698 | |
Nov 28 2018 | GOOGLE LLC | (assignment on the face of the patent) | / |
Date | Maintenance Fee Events |
Nov 28 2018 | BIG: Entity status set to Undiscounted (note the period is included in the code). |
Mar 01 2024 | M1551: Payment of Maintenance Fee, 4th Year, Large Entity. |
Date | Maintenance Schedule |
Sep 01 2023 | 4 years fee payment window open |
Mar 01 2024 | 6 months grace period start (w surcharge) |
Sep 01 2024 | patent expiry (for year 4) |
Sep 01 2026 | 2 years to revive unintentionally abandoned end. (for year 4) |
Sep 01 2027 | 8 years fee payment window open |
Mar 01 2028 | 6 months grace period start (w surcharge) |
Sep 01 2028 | patent expiry (for year 8) |
Sep 01 2030 | 2 years to revive unintentionally abandoned end. (for year 8) |
Sep 01 2031 | 12 years fee payment window open |
Mar 01 2032 | 6 months grace period start (w surcharge) |
Sep 01 2032 | patent expiry (for year 12) |
Sep 01 2034 | 2 years to revive unintentionally abandoned end. (for year 12) |