A system determines ranking scores for objects based on “virtual” links defined for the objects. A link-based ranking score may then be calculated for the objects based on the virtual links. In one implementation, the virtual links are determined based on a metric of content-based similarity between the objects.
|
22. A system comprising:
one or more processors to:
modify transitional probabilities between objects, in a collection of objects, based on an analysis of documents associated with the collection of objects,
a transitional probability being a likelihood that a user, having accessed a first object, in the collection of objects, selects a second object in the collection of objects,
where the transitional probabilities are based on a conversion operation that relates a particular measure of similarity between the objects to a particular transitional probability value,
obtain scores for the objects in the collection of objects,
the scores being based on the modified transitional probabilities between the objects in the collection of objects, and
provide the objects or links to the objects, to the user, in an order that is determined based on the scores for the objects.
1. A method comprising:
determining, by one or more processors, transitional probabilities between images in a collection of images,
a transitional probability being a likelihood that a user, accessing a first image, selects a second image,
where the transitional probabilities are based on a conversion operation that relates a particular measure of similarity between the images to a particular transitional probability value;
modifying, by the one or more processors, the transitional probabilities to form modified transitional probabilities based on an analysis of documents associated with the collection of images;
calculating, by the one or more processors, scores that define a measure of quality for the images,
the scores being calculated using the modified transitional probabilities; and
providing, by the one or more processors, the images to users in an order based on the scores.
11. A device comprising:
a memory to store instructions; and
one or more processors to execute the instructions to:
determine a transitional probability for a first object in a collection of objects,
the transitional probability representing a likelihood that a user, interacting with the first object, selects a second object in the collection of objects,
the objects including images, audio, video, books, or products,
where the transitional probability is based on a conversion operation that relates a particular measure of similarity between the first object and the second object to a particular transitional probability value;
modify the transitional probability, based on an analysis of documents associated with the collection of objects, to form a modified transitional probability;
calculate a score for the first object,
the score defining a measure of quality for the first object, and
the score being calculated using the transitional probability; and
present the first object to the user based on the score.
29. A method comprising:
calculating, by one or more processors, a similarity value between a first object, from a collection of objects, and a second object from the collection of objects,
the objects including images, audio, video, books, or products;
converting, by the one or more processors, the similarity value into a transitional probability,
the transitional probability being a likelihood that a user, interacting with the first object, selects the second object,
where the transitional probability is based on a conversion operation that relates a particular measure of similarity between the first object and the second object to a particular transitional probability value;
calculating, by the one or more processors, a score for the first object based on the transitional probability; and
presenting, by the one or more processors, at least some objects, of the collection of objects, to a user with the first object,
where the first object is presented, among the at least some objects, in an order based on the score.
15. A computer-readable medium storing instructions, the instructions comprising:
one or more instructions which, when executed by one or more processors, cause the one or more processors to generate a transitional probability between a first object, in a collection of objects, and a second object in the collection of objects,
the objects including images, audio, video, books, or products,
the transitional probability being a likelihood that a user, interacting with the first object, selects the second object,
where the transitional probability is based on a conversion operation that relates a particular measure of similarity between the first object and the second object to a particular transitional probability value;
one or more instructions which, when executed by the one or more processors, cause the one or more processors to modify the transitional probability, based on an analysis of documents associated with the collection of objects, to form a modified transitional probability;
one or more instructions which, when executed by the one or more processors, cause the one or more processors to calculate a score for the first object in the collection of objects based on the modified transitional probability; and
one or more instructions which, when executed by the one or more processors, cause the one or more processors to present the first object to the user based on the score calculated for the first object.
3. The method of
4. The method of
5. The method of
modifying the transitional probabilities based on a URL (uniform resource locator) analysis of the documents associated with the collection of images.
6. The method of
where the method further includes:
modifying the transitional probabilities further based on a query-independent measure of quality of the images.
7. The method of
8. The method of
calculating the scores as a recursive function of the scores and the transitional probabilities.
9. The method of
determining the transitional probabilities based on click-data gathered from users.
10. The method of
13. The device of
14. The device of
determine the transitional probabilities based on data obtained from a plurality of users,
where the data includes occurrences of selections of the first object followed by selections of the second object.
16. The computer-readable medium of
one or more instructions to present the first object and a plurality of other objects, in the collection of objects, to the user,
the first object and the plurality of other objects being presented in an order determined based on the score of the first object and scores of the plurality of other objects.
17. The computer-readable medium of
one or more instructions to determine the order, to present the first object and the plurality of other objects, based on a relevancy value of the first object to a search query.
18. The computer-readable medium of
where the particular measure of similarity between the first object and the second object is based on a comparison of features of the images.
19. The computer-readable medium of
20. The computer-readable medium of
21. The computer-readable medium of
one or more instructions to receive a search query from the user, and
where the one or more instructions to calculate the score for the first object include one or more instructions to calculate the score based on the search query and at least one category defined for the collection of objects.
24. The system of
obtain the scores for the objects using a technique to calculate the scores in which links used by the technique, are defined by the transitional probabilities.
25. The system of
modify the transitional probabilities based on a URL (uniform resource locator) analysis of the documents associated with the collection of objects.
26. The system of
modify the transitional probabilities further based on a query-independent measure of quality of the objects.
27. The system of
28. The system of
determine the transitional probabilities based on click-data gathered from users.
30. The method of
31. The method of
determining the order further based on a relevancy of the objects to a search query.
|
The instant application claims priority from provisional application No. 60/910,757, filed Apr. 9, 2007, the disclosure of which is incorporated by reference herein in its entirety.
The World Wide Web (“web”) contains a vast amount of information. Search engines assist users in locating desired portions of this information by cataloging web pages. Typically, in response to a user's request, the search engine returns references to documents relevant to the request.
One type of search engine is an image search engine. An image search engine, such as a web-based image search engine, catalogs images from the web. Typically, the image search engine may associate text, such as text that occurs near a particular image, with the image. The text associated with the images may then be searched using conventional key word-based search queries to locate images relevant for the search.
When a user enters a search query for images in an image search engine, the search engine may return hundreds of results (e.g., images or links to images). The large number of potentially relevant results can be daunting for the user. Additionally, because the relevancy of the results may be primarily determined based on text associated with the image, such as text near the image in a web page, the image itself may not match the search or may be of relatively low quality. For example, a search for “Eiffel Tower” may result in some images that include the Eiffel Tower while others may not. Of the images that contain the Eiffel Tower, some may be images taken at odd poses or odd croppings of the Eiffel Tower, or the Eiffel Tower may not be the main focus of the image (for example, the image may primarily be a picture of a person standing at the base of the Eiffel Tower, with the Eiffel Tower only minimally visible in the background).
It is a goal of an image search engine to return relevant and high quality images to the user.
One aspect is directed to a method that may include receiving a set of images for which ranking scores are to be calculated and determining transitional probabilities between images in the set of images, the transitional probabilities each being an estimate of a likelihood that a user viewing a first image is likely to select a second image. The method may further include calculating link-based ranking scores that each define a measure of quality for the images, the link-based ranking scores being calculated using the transitional probabilities as links between images in the set of images; and providing the images to users in an order influenced by the calculated link-based ranking scores.
Yet another aspect is directed to a method that may include receiving a set of objects and generating virtual links between pairs of objects in the set of objects. The method may further include calculating a link-based ranking score for a first object in the set of objects based on the generated virtual links and presenting the first object to a user based on the link-based ranking score of the first object.
Yet another aspect is directed to a system that comprises a server connected to a network to receive a search query from a user via the network. The server includes a database to store descriptive text relating to objects and a processor. The processor may generate an initial set of objects relevant to the search query based on a matching of terms in the search query to the descriptive text stored in the database and obtains ranking scores for the objects in the generated set of objects, the ranking scores being based on transitional probabilities that each represent a likelihood that a user having accessed a first object in the set of objects is likely to access a second object in the set of objects. The processor may additionally return a result set of objects or links to the objects to the user, the order of the objects in the result set being determined based on the ranking scores for the objects.
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate one or more embodiments of the invention and, together with the description, explain these embodiments. In the drawings:
The following detailed description refers to the accompanying drawings. The detailed description does not limit the invention.
Implementations described herein relate to assigning ranking scores to objects, such as images. The ranking scores may define an objective measure of quality of the objects and may be calculated based on “virtual links” between different objects. The virtual links do not necessarily need to be explicitly defined by the objects. Instead, the virtual links may be generated using content-based similarity information determined for the objects.
The objects will be particularly described herein as images, although other types of objects, such as audio or video objects, could also be used.
For each pair of images shown in
In the example of
The content-based similarity metrics may be used as, or used as the basis for, “virtual links” between pairs of images 110, 120, 130, and 140. The virtual links can be thought of as analogous to explicit links between objects such as web documents. In contrast to web documents, however, where the links are explicitly defined within the web documents, the virtual links described herein are not explicitly defined and are instead estimated from information such as the content-based similarity metric values.
The virtual links may used to calculate a final ranking score in a manner similar to the way explicit links between web documents are used to calculate ranking scores for documents. PageRank™, for example, is link-based quality score generation technique and is described in “The Anatomy of a Large-Scale Hypertextual Search Engine” by S. Brin and L. Page, 7th International World Wide Web Conference, Brisbane, Australia and U.S. Pat. No. 6,285,999. A technique such as PageRank™ or another link-based technique may be used to generate the ranking scores for images 110, 120, 130, and 140. The ranking score for each of images 110, 120, 130, and 140 are shown in
As shown in
A client 210 may include a device such as a wireless telephone, a personal computer, a personal digital assistant (PDA), a lap top, or another type of computation or communication device, a thread or process running on one of these devices, and/or an object executable by one of these devices. Server 220 may include a server device that processes, searches, and/or maintains documents and images. Clients 210 and server 220 may connect to network 240 via wired or wireless connections.
Server 220 may include an image search engine 225 usable by clients 210. In general, in response to a client request, image search engine 225 may return images to the client that are relevant to the client request.
Processor 320 may include a processor(s), a microprocessor(s), or processing logic that interprets and executes instructions. Main memory 330 may include a random access memory (RAM) or another type of dynamic storage device that stores information and instructions for execution by processor 320. ROM 340 may include a conventional ROM device or another type of static storage device that stores static information and instructions for use by processor 320. Storage device 350 may include a magnetic and/or optical recording medium and its corresponding drive.
Input device 360 may include one or more mechanisms that permit a user to input information to client/server 210/220, such as a keyboard, a mouse, a pen, voice recognition and/or biometric mechanisms, etc. Output device 370 may include one or more mechanisms that output information to the user, including a display, a printer, a speaker, etc. Communication interface 380 may include any transceiver-like mechanism that enables client/server 210/220 to communicate with other devices and/or systems. For example, communication interface 380 may include mechanisms for communicating with another device or system via a network, such as network 240.
As will be described in detail below, server 220 may implement image search engine 225. Image search engine 225 may be stored in a computer-readable medium, such as memory 330. A computer-readable medium may be defined as one or more physical or logical memory devices.
The software instructions defining image search engine 225 may be read into memory 330 from another computer-readable medium, such as data storage device 350, or from another device via communication interface 380. The software instructions contained in memory 330 may cause processor 320 to perform processes that will be described later. Alternatively, hardwired circuitry or other logic may be used in place of, or in combination with, software instructions to implement processes described herein. Thus, embodiments described are not limited to any specific combination of hardware circuitry and software.
Database 430 generally stores a collection of data. Database 430 may be implemented as, for example, a relational or non-relational database capable of storing and accessing data. Database 430 may be implemented on a single computing device or distributed across many computing devices and/or storage devices.
Search component 410 may receive user search queries, such as from clients 210, search database 430 for results based on the search queries, and generate relevant results (i.e., images or links to images) based on a comparison of the search query to the descriptive text for the images that are stored in database 430. In other words, search component 410 may generally match terms in the search query to the descriptive text associated with the images. Additionally, when returning results to the user, search component may take into account the image ranking score determined for the images that match the search query.
The ranking scores may be generated by ranking score generator 415. In some implementations, ranking score generator 415 may dynamically generate the ranking scores for images in a set of images that initially match the user's query. In another implementation, ranking score generator 415 may generate the ranking scores “offline” (i.e., not in response to a user search query) based on, for example, all of the indexed images or based on subsets of the indexed images. The generated ranking scores may be stored in database 430. Although shown as a separate element in
The calculation of the ranking scores by ranking score generator 415 will be described in additional detail below.
In general, the ranking score for an image can be though as a probability distribution that represents the likelihood that a hypothetical user selects a particular image given the following:
(1) The user randomly selects an image, called image A, from a collection of images.
(2) Given image A, the hypothetical user requests to view similar images and is shown the top N resultant images sorted according to the similarity between image A and the N images.
(3) The user selects another image, called image B, based on a transitional probability of selecting image B given image A.
(4) Image A is set to image B and (1) through (3) are repeated. To summarize (1) through (4), if the user is not satisfied with the initial results, the user can select one image among the search results and browse through a new set of “similar” images. The user may then select another image and continue this process until the user finishes the current search session.
The mechanism described in the preceding paragraph may create probabilistic hyperlinks between images, where the user jumps from one image to another following a certain probability distribution (called the transitional probability, P(A→B), herein). Given N images with interconnected probability hyperlinks, a Markov network with images as nodes and similarity measurements as transitional links is essentially created. The image ranking score for each image may therefore be viewed as the stationary probabilistic distribution for the Markov network.
It can be appreciated that the accuracy of the image ranking score for an image depends on the accurate estimation of the transitional probability, P(A→B), given two images A and B.
Ranking score generator 415 may calculate the transitional probabilities, P(A→B), between each image A and each other image B (act 510). As previously mentioned, this represents the estimated probability that a user viewing image A will select image B. The transitional probabilities may be estimated based on content-based similarity metric(s). Calculation of the transitional probabilities is described in more detail below with reference to
The transitional probabilities may be used to determine the ranking scores (act 520). Generation of the ranking scores from the transitional probabilities is described in more detail below.
Results may be output or transmitted to users based on the ranking scores (act 530). For example, in an image search, images returned to the user may be sorted in an order determined by or influenced by the ranking scores.
A number of different image features may be used to compare the images. The particular image features to use may be decided ahead of time or offline by an administrator. Ranking score generator 415 may receive the selection of which features to use (act 601). Examples of image features that may be used include image features based on, for example, intensity, color, edges, texture, wavelet based techniques, or other aspects of the image. For example, regarding intensity, each image may be divided into small patches (e.g., rectangles, circles, etc.) and an intensity histogram computed for each patch. Each intensity histogram may be considered to be a feature for the image. Similarly, as an example of a color-based feature, a color histogram may be computed for each patch (or for different patches) within each image. A color histogram can be similarly computed to obtain a possible color-based histogram. The color histogram may be calculated using any known color space, such as the RGB (red, green, blue) color space, YIQ (luma (Y) and chrominance (IQ)), or another color space.
Histograms can also be used to represent edge and texture information. For example, histograms can be computed based on patches of edge information or texture information in an image. For wavelet based techniques, a wavelet transform may be computed for each patch and used as an image feature.
The features discussed above represent an exemplary list of possible image features that may be used. Other image features, such as features identified using the known Scale-Invariant Feature Transform (SIFT) may be used.
In some implementations, to improve computation efficiency, features may be computed only for certain areas within images. For example, “objects of interest” within an image may be determined and image features may only be computed for the objects of interest. For example, if the image feature being used is a color histogram, a histogram may be computed for each patch in the image that includes an object of interest. Objects of interest within an image can be determined in a number of ways. For example, for color, objects of interest may be defined as points where there is high variation in color (i.e., areas where color changes significantly). In general, objects of interest can be determined mathematically in a variety of ways and are frequently based on determining discontinuities or differences from surrounding points. The SIFT algorithm is an example of one technique for locating objects of interest.
Additionally, in some implementations, the various features described above may be computed using different image scales. For example, an image can be examined and features computed in its original scale and then features may be successively examined at smaller scales. Additionally or alternatively, features may be selected as features that are scale invariant or invariant to affine transformations. The SIFT techniques, for example, can be used to extract distinctive invariant objects from images. The extracted objects are invariant to image scale and rotation.
For each feature that is to be used, a comparison function may be selected. A number of different comparison functions may be used to compare images. The particular comparison function to use may be decided ahead of time or offline by an administrator. Ranking score generator 415 may receive the selection of the comparison functions (act 602).
In general, a comparison function may operate to generate a value defining a similarity between a particular feature computed for two images. As an example of a possible comparison function, consider a simple histogram comparer function, which is described in pseudo-code in Table I, below. As shown in Table I, the histogram comparer function returns a value that is the sum of the absolute values of the differences between corresponding bins in the input histograms. Smaller values returned from this function indicate greater similarity between the input histograms.
TABLE I
Compare_histogram_type_features(histogram1,histogram2)
Difference = 0;
For all bins, b, in histogram:
Difference = Difference + |histogram1[b] −
histogram2[b]|
Return(Difference).
The histogram comparer function is exemplary. It can be appreciated that other comparison functions can be used to compare histograms. For example, squared differences may be used rather than absolute differences, bin correlations may be taken into account instead of absolute differences, or percent differences may be used instead of absolute differences. Additionally, for image features other than those based on histograms, different comparison functions may, of course, be used.
The selection of the image features to use and the comparison functions to use (acts 601 and 602) may be performed offline or in non-runtime operation. For example, an administrator may initially design or configure ranking score generator 415 to use one or more image features and one or more comparison functions. After these initial acts, ranking score generator 415 may function in a runtime mode (acts 604-610).
In one possible implementation, the selection of image features to use and the comparison functions to use (acts 601 and 602) may define a set of possible image features and comparison functions. During runtime operation, the set of possible image features and comparison functions may be dynamically narrowed. For example, the number of matches for a feature across the images for that feature may be used as a measure as to whether the feature is interesting. Assume, for instance, that the user query “black and white pictures of the london tower” initially generated a number of black and white images. With this set of images, using a color histogram as a feature is not likely to be very helpful. However, other features, such as edge-based features, may play a more important part as they may be more distinguishing. A number of techniques may be used to determine whether a feature is interesting for a set of images, such as techniques based on the variation, distribution, or entropy in the responses across images for a feature.
Ranking score generator 415 may set the first received image as the active image (act 604). The active image may then be compared to each other image in the set of images (act 605). The comparison may be based on the image features from act 501 and may be performed using the comparison functions from act 502.
An exemplary implementation of act 605 is shown in pseudo-code in Table II, below. In Table II, the final feature similarity metric is tabulated in the variable “image_similarity_score.”
TABLE II
image_similarity_score = 0
For each feature Fi of active image i:
For each image j (where j is not i):
For each feature Fj of image j that is of type Fi
feature_similarity_score = compare_features(Fi, Fj)
image_similarity_score = image_similarity_score +
feature_similarity_score
In the implementation of act 605 shown in Table II, each feature Fi of the active image is compared with every feature Fj of image j that is the same feature type. In other words, if Fi is a color histogram then the comparison is performed with the color histogram features of image j, if Fi is an edge histogram then the edge histogram features of image j are compared, etc.
Ranking score generator 415 may assign a similarity metric value to the active image based on the comparison of act 605 (act 606). For example, the “image_similarity_score” variable (see Table II) indicates how well the features in image i appear in other images and, in this implementation, represents the similarity metric assigned to the active image. In this implementation, image_similarity_score is simply the sum of the feature similarity scores for an image. In other implementations, the similarity metric may be calculated in other ways, such as by only adding the value of the variable “feature_similarity_score” to “image_similarity_score” when “feature_similarity_score” is above some threshold. In general, the similarity metric may be designed so that more similar image features will contribute more significantly to the similarity metric.
Acts 605 and 606 may be repeated for each image in the set of images (acts 607 and 608) to thus obtain a similarity metric for each image in the set of images.
Acts 605-608 can be relatively computationally expensive, as it requires N2 comparisons among the N images, and for each comparison, Mi*Mj feature comparisons, for Mi and Mj local features in each image. Techniques are known that may potentially accelerate this type of operation. For example, one such technique is described in the publication “The Pyramid Match Kernel: Discriminative Classification with Sets of Image Features,” K. Grauman et al., Proceedings of the IEEE International Conference on Computer Vision, Beijing, China, October 2005.
Additionally, it can be appreciated that to further increase run-time processing performance, the features used for each of the images may be pre-computed ahead of time and stored, for instance, in database 430.
The similarity metrics generated in acts 605 through 608 may be converted to transitional probabilities (act 609). The transitional probability that a hypothetical user viewing an image A selects an image B, P(A→B), may be conceptualized based on the following two intuitions: (1) the user is more likely to select an image that is similar to image A than an image that is not; and (2) the user is likely to ignore images that are too similar to image A. Intuition (2) reflects the notion that images that are too similar to the currently selected image are essentially redundant to the user.
In one implementation, intuition (1) and (2) may be incorporated into a transitional probability based on a beta distribution over the similarity metric values calculated in act 606.
It can be appreciated that many other functions could be used to express the probability of selecting an image relative to similarity. Further, in some implementations, the function may be an “editable” function. For example, the user may be able to use a settable slider or other graphical tool to set the type of images the user is looking for (e.g., very similar images or less similar images).
Referring back to
One possible way to enhance the effectiveness of the transitional probabilities is based on query independent metrics of images.
A query independent measure of quality, called Quality(A), may be determined for image A (act 801). Quality(A) may, for example, measure how much or how well an image is in focus, how many colors are in the image, or the saturation of the image. Techniques are known for measuring the “focus” of an image. Such techniques may be based on the sharpness of the image and will not be described further herein. Similarly, techniques for determining the number of colors in an image and determining a measure of the saturation of an image are also known and will not be described further herein.
Another possible factor for improving transitional probabilities is based on a URL (uniform resource locator) analysis. Two images may be considered to be related when the corresponding web pages the images are related. For example, the user may enjoy finding images that are from the same blog, same website, or from the same social network. Techniques are known for estimating “similarity” based on relations between objects, such as hyperlinks between web pages. One such technique is described in the publication “SimRank: A Measure of Structural-Context Similarity,” G. Jeh and J. Widom, Proceedings of the eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2002.
A value measuring similarity of image A to image B based on the URL analysis may be determined (act 802). This “hyperlink-induced” similarity between image A and image B will be referred to a LINKSIM(A, B) herein. LINKSIM(A, B) may be determined based on the web pages containing images A and B. For example, the SimRank technique, or another technique capable of estimating similarity based on relationships between objects, may be used to calculate LINKSIM(A,B).
The initial transitional probability, P(A→B), determined in act 609 may be modified or calculated based on Quality(A) and LINKSIM(A,B) (act 803). For example, P(A→B) may be recalculated as
Quality(A)·LINKSIM(A,B)·SIMILARITY(A,B),
where SIMILARITY(A,B) may refer to a distribution defined over the raw similarity metric values.
In some implementations, P(A→B), as determined in act 803, may also be normalized to obtain a desired distribution of the transitional probabilities (act 804). The normalization may be performed, for example, after a number of or after all of the transitional probabilities are calculated.
It can be appreciated that the post processing performed in act 610 may include additional or fewer factors than those shown in
Referring back to
Alternatively, it is noted that the ranking scores are also the entries of the dominant eigenvector of the transitional probability matrix. Stated formally:
where RankingScores is the solution for
where each column sums to one.
In some implementations, a damping factor may be used. The damping factor may model the idea that the user is likely to stop selecting images at some point. For example, a damping factor d may be defined to have a value between zero and one. With the damping factor, equation (1) becomes:
The ranking scores described above provide a link-based measure of quality for objects (e.g., images) that do not necessarily include explicit links between objects. The ranking scores were instead calculated based on “virtual” links determined for the objects based on similarity information (i.e., the transitional probabilities) of the objects. In one possible alternative implementation, the transitional probabilities, instead of being based on similarity information, may be based on user click data.
Referring to
The transitional probabilities may be calculated based on the gathered click-data (act 910). In one implementation, the click data may be used to directly calculate the transitional probabilities. For example, assume that the click-data includes 1000 samples of users viewing a first image A and then selecting a second image B. Assume that of these 1000 samples, 100 times the second image selected by the users was image B. The transitional probability, P(A→B), may be directly calculated as 0.10.
In one implementation, the click-data described above may be used by itself to generate the transitional probabilities. In an alternative implementation, the click-data may be used in addition to the similarity metric values to generate the transitional probabilities. In still another implementation, the click-data may be used to modify the similarity metric values or vice-versa.
The ranking scores described above were generally described as being generated over a set of images (act 500,
For a query-dependent calculation of the ranking scores, the initial set of images determined in act 500 may be the set of images returned by search engine 225 in response to the initial user search query. For example, if the user submits the search query “Lincoln Memorial,” search engine 225 may retrieve a number of result images (e.g., hundreds or thousands) that are associated with text that matches the search query.
Other techniques for performing query-dependent calculation of the ranking scores are possible.
As shown in
Each category of images defines a set of images. For each set, a ranking score may be calculated as previously discussed with respect to
The ranking score used by search engine 225 when returning images to the user (the “final” ranking score for the image) may be calculated as a query-dependent combination of ranking scores for the image in the categories (act 1030). In one implementation, the query-dependent final ranking score may be calculated as a linear combination of the category ranking scores. More specifically, the final ranking score for an image A, under a query Q, may be calculated as:
I(A,Q)=I(A,Q,C1)·W1+I(A,Q,C2)·W2+ . . . +I(A,Q,CN)·WN,
where I(A,Q) refers to the ranking score of image A under query Q, I(A,Q,Ci) refers to the ranking score of image A when calculated over the set of images that match the Query, Q, and are in category Ci and Wi refers to a weight value by which I (A,Ci) is multiplied. In one implementation, the weights, Wi, may be calculated based on the semantic distance between the query, Q, and the image category. For example, if Q equals “beach” then the category may be most likely to be “outdoor” and the corresponding weight should be significantly larger than the weight for a category such as “indoor.”
As previously mentioned with respect to
The ranking scores may be employed in contexts other than a general image search engine. In general, any application in which images are to be ranked or in which a subset of an initial set of images are to be selected may be an application that is appropriate for application of ranking score generator 415. For example, in an image search performed by a user using a mobile device, in which the display area is limited, it may be desirable to prune the number of images returned to the user to some small number (e.g., 5).
Ranking score generator 415 may additionally be used in situations in which it is desirable to find a single most representative image from a set of images. For example, in a product search engine or a news search engine, it may be desirable to display a single image next to a product listing or next to a news article. For a product search engine, for instance, the name of the product may be used as a search query to return an initial set of possible product images. Ranking score generator 415 may be used as described above to rank the initial set of product images. The single highest ranking product image may be chosen to display next to the product listing. Similarly, for a news article search engine, selected text from the news article may be used as a search query to return an initial set of possible images. Ranking score generator 415 may be used as described above to rank the initial set of images. The single highest ranking image may be chosen to display next to the news listing.
Further, the concepts described herein may be applied to applications other than image-based applications. For example, a ranking score calculated as described above may be generated for other sets of objects for which transitional probabilities can be generated. For example, ranking scores can be calculated for audio media objects (e.g., songs, audio books, audio clips, etc.) or video media objects (e.g., movies, short video clips, etc.). Other domains in which ranking scores can be calculated include book search, stock search, product search, resume search, or other search tasks where the user may find a “see more items like this one” option useful. In general, the above-described ranking scores can be calculated for any set of objects in which it is possible to create virtual links based on similarity of the objects or other information, such as user click behavior.
As described above, a set of objects are processed to rank the objects using link-based ranking techniques even when the objects do not include explicit links between them. Instead, virtual links may be generated based on, for example, similarity between objects or click-data.
The foregoing description of exemplary embodiments provides illustration and description, but is not intended to be exhaustive or to limit the embodiments to the precise form disclosed. Modifications and variations are possible in light of the above teachings or may be acquired from practice of the invention.
Moreover, while a series of acts have been described with regard to
It will also be apparent that aspects, as described above, may be implemented in many different forms of software, firmware, and hardware in the implementations illustrated in the figures. The actual software code or specialized control hardware used to implement these aspects is not limiting of the invention. Thus, the operation and behavior of the aspects of the invention were described without reference to the specific software code—it being understood that one of ordinary skill in the art would be able to design software and control hardware to implement the aspects based on the description herein.
Further, certain portions described herein may be implemented as “components” or “models” that performs one or more functions. These components/models may be implemented as hardware, such as an application specific integrated circuit or a field programmable gate array, software, or a combination of hardware and software.
No element, act, or instruction used in the description of the invention should be construed as critical or essential to the invention unless explicitly described as such. Also, as used herein, the article “a” is intended to include one or more items. Where only one item is intended, the term “one” or similar language is used. Further, the phrase “based on” is intended to mean “based, at least in part, on” unless explicitly stated otherwise.
Jing, Yushi, Rowley, Henry A., Baluja, Shumeet
Patent | Priority | Assignee | Title |
10089762, | Jul 04 2014 | Meta Platforms, Inc | Methods for navigating through a set of images |
10282055, | Mar 06 2012 | Apple Inc. | Ordered processing of edits for a media editing application |
10545631, | Mar 06 2012 | Apple Inc. | Fanning user interface controls for a media editing application |
10552016, | Mar 06 2012 | Apple Inc. | User interface tools for cropping and straightening image |
10936173, | Mar 06 2012 | Apple Inc. | Unified slider control for modifying multiple image properties |
10942634, | Mar 06 2012 | Apple Inc. | User interface tools for cropping and straightening image |
11030235, | Jan 04 2016 | Meta Platforms, Inc | Method for navigating through a set of images |
11119635, | Mar 03 2012 | Apple Inc. | Fanning user interface controls for a media editing application |
11481097, | Mar 06 2012 | Apple Inc. | User interface tools for cropping and straightening image |
9367756, | Aug 31 2010 | GOOGLE LLC | Selection of representative images |
9384241, | Nov 24 2011 | Microsoft Technology Licensing, LLC | Reranking using confident image samples |
9977816, | Apr 09 2007 | GOOGLE LLC | Link-based ranking of objects that do not include explicitly defined links |
Patent | Priority | Assignee | Title |
6285999, | Jan 10 1997 | GOOGLE LLC | Method for node ranking in a linked database |
7773800, | Jun 06 2001 | Ying, Liu | Attrasoft image retrieval |
8090222, | Nov 15 2006 | GOOGLE LLC | Selection of an image or images most representative of a set of images |
20020057343, | |||
20040162827, | |||
20040189691, | |||
20050223031, |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Feb 01 2008 | JING, YUSHI | Google Inc | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 020782 | /0844 | |
Feb 01 2008 | ROWLEY, HENRY A | Google Inc | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 020782 | /0844 | |
Feb 05 2008 | BALUJA, SHUMEET | Google Inc | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 020782 | /0844 | |
Apr 08 2008 | Google Inc. | (assignment on the face of the patent) | / | |||
Sep 29 2017 | Google Inc | GOOGLE LLC | CHANGE OF NAME SEE DOCUMENT FOR DETAILS | 044277 | /0001 |
Date | Maintenance Fee Events |
Mar 25 2014 | ASPN: Payor Number Assigned. |
Nov 20 2017 | M1551: Payment of Maintenance Fee, 4th Year, Large Entity. |
Nov 22 2021 | M1552: Payment of Maintenance Fee, 8th Year, Large Entity. |
Date | Maintenance Schedule |
May 20 2017 | 4 years fee payment window open |
Nov 20 2017 | 6 months grace period start (w surcharge) |
May 20 2018 | patent expiry (for year 4) |
May 20 2020 | 2 years to revive unintentionally abandoned end. (for year 4) |
May 20 2021 | 8 years fee payment window open |
Nov 20 2021 | 6 months grace period start (w surcharge) |
May 20 2022 | patent expiry (for year 8) |
May 20 2024 | 2 years to revive unintentionally abandoned end. (for year 8) |
May 20 2025 | 12 years fee payment window open |
Nov 20 2025 | 6 months grace period start (w surcharge) |
May 20 2026 | patent expiry (for year 12) |
May 20 2028 | 2 years to revive unintentionally abandoned end. (for year 12) |