Detecting a consistency of image components is disclosed. A first constraint associated with a possible convergence area of a first component of an image is received. A second constraint associated with a possible convergence area of a second component of the image is received. The first constraint and the second constraint are used to provide a result associated with whether the first component and the second component are consistent.
|
1. A method of detecting a consistency of image components, comprising:
receiving a first constraint that defines a possible convergence area of a first component of an image;
receiving a second constraint that defines a possible convergence area of a second component of the image; and
using a processor that uses the first constraint and the second constraint to provide a result associated with whether the first component and the second component are consistent with a common source.
22. A system for detecting a consistency of image components, comprising:
a processor configured to:
receive a first constraint that defines a possible convergence area of a first component of an image;
receive a second constraint that defines a possible convergence area of a second component of the image; and
use the first constraint and the second constraint to provide a result associated with whether the first component and the second component are consistent with a common source; and
a memory coupled to the processor and configured to provide the processor with instructions.
23. A computer program product for detecting a consistency of image components, the computer program product being embodied in a tangible non-transitory computer readable storage medium and comprising computer instructions for:
receiving a first constraint that defines a possible convergence area of a first component of an image;
receiving a second constraint that defines a possible convergence area of a second component of the image; and
using the first constraint and the second constraint to provide a result associated with whether the first component and the second component are consistent with a common source.
2. The method of
3. The method of
6. The method of
9. The method of
11. The method of
12. The method of
13. The method of
14. The method of
15. The method of
16. The method of
17. The method of
18. The method of
19. The method of
20. The method of
21. The method of
24. The system of
27. The system of
28. The system of
|
This application claims priority to U.S. Provisional Patent Application No. 61/594,935 entitled DETECTING IMAGE INCONSISTENCIES filed Feb. 3, 2012 which is incorporated herein by reference for all purposes.
Recent advances in computational photography, computer vision, and computer graphics has allowed for the creation of visually compelling digitally altered “fake” photographs. The resulting undermining of trust in photographs impacts law enforcement, national security, the media, advertising, e-commerce, and more. The field of photo forensics has emerged to assist in identifying tamper and manipulation of images. A technique utilized in photo forensics operates on the assumption that most forms of tampering will disturb some statistical or geometric property of an image. To the extent that these manipulations can be quantified and detected, they may be used to objectively invalidate an image as altered.
Various embodiments of the invention are disclosed in the following detailed description and the accompanying drawings.
The invention can be implemented in numerous ways, including as a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer readable storage medium; and/or a processor, such as a processor configured to execute instructions stored on and/or provided by a memory coupled to the processor. In this specification, these implementations, or any other form that the invention may take, may be referred to as techniques. In general, the order of the steps of disclosed processes may be altered within the scope of the invention. Unless stated otherwise, a component such as a processor or a memory described as being configured to perform a task may be implemented as a general component that is temporarily configured to perform the task at a given time or a specific component that is manufactured to perform the task. As used herein, the term ‘processor’ refers to one or more devices, circuits, and/or processing cores configured to process data, such as computer program instructions.
A detailed description of one or more embodiments of the invention is provided below along with accompanying figures that illustrate the principles of the invention. The invention is described in connection with such embodiments, but the invention is not limited to any embodiment. The scope of the invention is limited only by the claims and the invention encompasses numerous alternatives, modifications and equivalents. Numerous specific details are set forth in the following description in order to provide a thorough understanding of the invention. These details are provided for the purpose of example and the invention may be practiced according to the claims without some or all of these specific details. For the purpose of clarity, technical material that is known in the technical fields related to the invention has not been described in detail so that the invention is not unnecessarily obscured.
Detecting whether an image is consistent is disclosed. In some embodiments, components of an image (e.g., a photograph, a painting, a drawing or a computer rendered image) are analyzed to determine whether the components of the image are consistent. For example shadows contained in the image are analyzed to determine whether the shadows are consistent with a consistent light source. Other examples of image components that may be analyzed include perspective lines in the image that should be consistent with a vanishing point and a reflection in the image that should be consistent with a reflection vanishing point. In some embodiments, by analyzing whether the components of the image are consistent, a user may be able to determine whether a photograph has been altered since the photograph was captured.
In some embodiments, analyzing the components of the image for consistency includes identifying two-dimensional areas where a convergence point may be included for each of the analyzed image components. For example, a convergence point, such as a lighting source, must be consistent in an unaltered photograph. If an image has a single light source, all shadows included in the image must be consistent with the single light source. For one or more analyzed shadows of the image, a two dimensional area where the single light source may exist to cast the shadow is identified. The areas of possible light source locations for different shadows must at least partially overlap in a single area where the single light source must exist. If areas of possible light sources for different shadows do not overlap, the image has been likely altered and at least one of the shadows or an object associated with shadow has been likely altered in the image. These techniques may be used even in the absence of any digital watermark or signature or any other assumed information embedded or extracted from a photograph at the time of recording.
In some embodiments, techniques described in this specification are used for images with a single dominant light source such as outdoor scenes lit by the sun or indoor scenes photographed with a flash. The light source does not have to be within the image. In some embodiments, techniques described in this specification are used for images with linear perspective projection. If an image has been distorted due to lens distortion, the lens distortion may be estimated and removed using a number of standard techniques. In some embodiments, analyzing the components of the image for consistency includes determining whether the image includes an object that was not captured (e.g., photograph), drawn (e.g., a painting), or rendered (e.g., computer generated image) when the image was generated. In some embodiments, the components of the image are analyzed for consistency to allow insertion or removal of image components in a manner that is consistent with the other image components.
If the location of light source 102 was unknown or located outside the image, the location of the light source could be located using line 114 and line 120. Because any line traced through a point on a shadow and a corresponding point on an associated object must intersect at a single light source point, an alteration of the image containing the shadows can be detected by detecting any line trace that does not intersect another line trace at a single point associated with the light source. For example, if another object was digitally inserted into the image of
In the example shown in
In some embodiments, a user may identify a wedge area by selecting a point on a shadow and identifying the portion of the associated object to be included in the wedge. In some embodiments, the portion of the associated object is automatically determined based on the selection of the point on the shadow. A user may be able to modify the automatically determined portion. In some embodiments, the entire wedge is automatically determined and the user may be able to modify the automatically determined wedge area.
Convergence area 202 of wedge area 214 and wedge area 220 should include the location of a light source. Any additional consistent wedge areas identified in the image must intersect at least a portion of area 202. Any inauthentic alternation of an image may be detected by detecting any wedge area (or trace line or other identified constraint areas as described in the specification) that does not intersect another constraint area in a consistent location with all other wedge areas (or trace line or other constraint areas as described in the specification). For example, if another object was digitally inserted into the image of
Similar to the example shown in
In some embodiments, a user may identify a half plane area by selecting a point on an attached shadow and identifying a line (e.g., on a shadow terminator) that passes through the point and divides an image into an indicated area where a light source may exist and another area where the light source may not exist. For example, an arrow normal to the line and pointing to the direction of the light source side may be specified. In some embodiments, the line and area associated with the light source is automatically determined based on a selected point on the attached shadow. A user may be able to modify the automatically determined portion. In some embodiments, the entire half plane area is automatically determined and the user may be able to modify the automatically determined half plane area.
In some embodiments, in an image that has not been inconsistently altered, the convergence area associated with the light source should be consistent with other areas identified as potential areas where a light source may be located. For example, at least a portion of an overlapping area (e.g., convergence area 202 of
Image 402 is image 400 with a plurality of image components analyzed for consistency. All of the analyzed components are consistent with convergence area 410 (i.e., overlaps at this common location) except wedge area 404. The canvas of image 402 has been enlarged to visually show convergence area 410. Wedge area 404 corresponds to a point on a cast shadow on the ground of a child playing basketball. This point corresponds with the head of the child and wedge 404 includes points on the head of the child. Because wedge area 404 is not consistent with other determined areas that overlap at area 410, it may be determined that objects/components (e.g., child's associated shadow) associated with wedge area 404 have been altered and/or image 404 has been altered. This may be visually indicated to a user by indicating wedge area 404 and/or visually showing area 410 that does not overlap with area 404.
As shown in
As shown in
In some embodiments, the received constraint includes a wedge area similar to one or more of the wedge areas discussed previously in the specification. In some embodiments, a user may visually identify a wedge area by selecting a point on an image component and identifying the portion of the associated object to be included in the wedge area. In some embodiments, the portion of the associated object is automatically determined based on the selection of the point. A user may be able to modify the automatically determined portion. In some embodiments, the entire wedge area is automatically determined and the user may be able to modify the automatically determined wedge area.
In some embodiments, the constraint includes a half plane area similar to one or more of the half plane areas discussed previously in the specification. In some embodiments, a user may visually identify a half plane area by selecting a point on an attached shadow and identifying a line (e.g., on a shadow terminator) that passes through the point and divides an image into an indicated area associated with a light source and another area not associated with the light source. For example, an arrow normal to the line and pointing to the direction of the light source side may be specified. In some embodiments, the line and area associated with the light source are automatically determined based on the point on the shadow. A user may be able to modify the automatically determined portion. In some embodiments, the entire half plane area is automatically determined and the user may be able to modify the automatically determined half plane area.
At 804, it is determined whether an additional constraint for the same image exists. If the additional constraint exists, the process returns to 802 where the next constraint is received. If the additional constraint does not exist, the process proceeds to 806. At 806, the received constraint(s) are used to determine whether the associated with the image component(s) are consistent. In some embodiments, determining whether the image components are consistent includes determining whether received constraints are consistent. In some embodiments, determining whether image components are consistent includes determining the location and/or existence of the convergence area and/or displaying/indicating the convergence area.
In some embodiments, determining whether the image component(s) are consistent includes determining whether the received constraints at least in part overlap. For example, the received constraints are visually displayed over the image including the image components and a user is able to determine whether the constraints visually overlap because the areas where the constraints overlap are shaded darker as more constraints overlap the same area. Visually displaying the constraints may include displaying the constraint in an area that extends beyond an area of the image and the size of the extension may be automatically determined or manually specified to enable a determination as to whether the image components are consistent. In some embodiments, an alternative visual representation of a constraint is displayed. For example, as shown in
At 808, a result based on the determination is provided. In some embodiments, the result includes an indication that the received constraints are consistent. In some embodiments, the result includes an indication that the received constraints are not consistent and may include an indication of which constraint(s) are not consistent with which constraint(s). In some embodiments, the result includes a visual indication of which one or more of the constraints overlap in a convergence area and/or a visual indication of which one or more of the constraints do not overlap in a convergence area. In some embodiments, the result includes an indication of how to alter the image containing the image components such that the alteration will be consistent with the received constraints. For example, the process of
ni·x−ni·pi≧0, (1)
ni1·x−ni1·pi≧0 and ni2·x−ni2·pi≧0. (2)
At 904, it is determined whether the mathematical representations of the constraints are consistent. In some embodiments, determining the whether the mathematical representations are consistent includes solving a system of linear inequalities. For example, a collection of half-plane and wedge constraints can be combined into a single system of m inequalities:
Nx−P≧0. (4)
Given error-free constraints from a consistent scene, a solution to this system of inequalities should exist. Errors or inconsistency may be accounted by introducing a set of m slack variables si where si is the ith component of the m-vector s:
Nx−P≧−s (5)
s≧0, (6)
If the constraints are fully satisfiable then a solution will exist where all slack variables are zero. Solutions with a non-zero slack variable si mean that the solution is not consistent with constraint i. The previous inequalities can be combined into a single system as follows where I is an m×m identity matrix:
A solution that minimizes the amount of slack required to satisfy this system is determined. As such, subject to the above equation, the linear program consists of minimizing the L1 norm of the vector of slack variables:
In other words, a solution that minimizes the slack variables, while satisfying all of the cast and attached shadow constraints, is determined. If the slack variables for the optimal solution are all zero, then there exists a convergence area that satisfies all of the specified constraints. Otherwise, one or more of the constraints are inconsistent.
As described earlier, a special case exists if a light source is behind the center of projection. In this case, there exists an inherent sign ambiguity in specifying these constraints and its projected location in the image plane is inverted by negating the normals of the constraints. These constraints would take the form:
−Nx+P≧−s. (9)
For example as shown in
In some embodiments, both linear programs (e.g., with constraints Equation (5) and (9)) select the solution with the minimal L1 norm, Equation (8). If either the regular or inverted system has a solution with zero slack, then it is determined that the constraints are mutually consistent. Otherwise, the constraints are inconsistent if it is determined that at least a portion/component of the image has been altered.
In some embodiments, if it is determined that there exists inconsistent violating constraints, it is then desirable to identify which constraints are in conflict with others. This information can be useful to an analyst in determining what parts of an image have been manipulated. In some embodiments, by iteratively solving a system of inequalities for various combinations of received constraints, a determination is made as to which one or more of the constraints is inconsistent or consistent with which one or more of the other constraints. This information may be used to determine which portion and or component of an image has been altered and/or is inconsistent. For example, two constraints are selected at random. If these constraints are not satisfiable, then they form a minimal set of inconsistent constraints. If they are satisfiable, then a randomly selected constraint is added to the set and the linear program is solved. Constraints are added in this way until the system is no longer satisfiable. This entire process is repeated with different random starting conditions. The smallest set of violating constraints, which may or may not be unique, identifies the inconsistent constraints.
Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative and not restrictive.
Farid, Hany, O'Brien, James, Kee, Eric
Patent | Priority | Assignee | Title |
11094135, | Mar 05 2021 | LEXISNEXIS RISK SOLUTIONS FL INC | Automated measurement of interior spaces through guided modeling of dimensions |
11682174, | Mar 05 2021 | LEXISNEXIS RISK SOLUTIONS FL INC | Automated measurement of interior spaces through guided modeling of dimensions |
9430715, | May 01 2015 | Adobe Inc | Identifying and modifying cast shadows in an image |
Patent | Priority | Assignee | Title |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
May 16 2012 | Fourandsix Technologies, Inc. | (assignment on the face of the patent) | / | |||
Jul 05 2012 | KEE, ERIC | FOURANDSIX TECHNOLOGIES, INC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 028898 | /0882 | |
Jul 09 2012 | FARID, HANY | FOURANDSIX TECHNOLOGIES, INC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 028898 | /0882 | |
Jul 09 2012 | O BRIEN, JAMES | FOURANDSIX TECHNOLOGIES, INC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 028898 | /0882 | |
Nov 28 2018 | FOURANDSIX TECHNOLOGIES, INC | TRUEPIC ANALYZE LLC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 047907 | /0954 | |
May 30 2023 | TRUEPIC ANALYZE LLC | TRUEPIC INC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 063803 | /0339 | |
Jul 10 2024 | TRUEPIC INC | WTI FUND X, INC | SECURITY INTEREST SEE DOCUMENT FOR DETAILS | 067970 | /0287 | |
Jul 10 2024 | TRUEPIC INC | WTI FUND XI, INC | SECURITY INTEREST SEE DOCUMENT FOR DETAILS | 067970 | /0287 |
Date | Maintenance Fee Events |
Oct 15 2018 | REM: Maintenance Fee Reminder Mailed. |
Oct 25 2018 | M1551: Payment of Maintenance Fee, 4th Year, Large Entity. |
Oct 25 2018 | M1554: Surcharge for Late Payment, Large Entity. |
Aug 10 2022 | M1552: Payment of Maintenance Fee, 8th Year, Large Entity. |
Date | Maintenance Schedule |
Feb 24 2018 | 4 years fee payment window open |
Aug 24 2018 | 6 months grace period start (w surcharge) |
Feb 24 2019 | patent expiry (for year 4) |
Feb 24 2021 | 2 years to revive unintentionally abandoned end. (for year 4) |
Feb 24 2022 | 8 years fee payment window open |
Aug 24 2022 | 6 months grace period start (w surcharge) |
Feb 24 2023 | patent expiry (for year 8) |
Feb 24 2025 | 2 years to revive unintentionally abandoned end. (for year 8) |
Feb 24 2026 | 12 years fee payment window open |
Aug 24 2026 | 6 months grace period start (w surcharge) |
Feb 24 2027 | patent expiry (for year 12) |
Feb 24 2029 | 2 years to revive unintentionally abandoned end. (for year 12) |