A method and system for generating clusters of images for a search result of an image query is provided. When an original image query is received, the search system identifies text associated with the original image query by submitting the original image query to a search engine. The search system identifies phrases from the text of the web page containing the search result. The search system uses each of the identified phrases as an image query and submits the image queries to an image search engine. The search system considers the image search result for each image query to represent a cluster of related images. The search system then presents the clusters of images as the images of the image search result of the original image query.
|
4. A computer-readable medium storing computer-executable instructions for controlling a computing device to generate clusters of images, by a method comprising:
receiving an image query having search terms;
submitting the image query to a web-based search engine and receives search results that include links to web pages and text associated with each link, the text being a snippet describing content of the linked-to web page, the web-based search engine being a search engine that maintains an index of keywords to web pages and when a query is submitted, the web-based search engine identifies web pages for search results using the index of keywords to identify web pages that match the submitted query and returns as search results links to the identified web pages and associated snippets;
identifying phrases associated with the image query by identifying phrases from the snippets of the search results;
identifying images associated with each identified phrase by submitting each identified phrase to an image search engine and receiving identifications of images as search results for each submitted phrase wherein the identified images associated with each submitted phrase represent a cluster of images related to that submitted phrase, the image search engine being a search engine that maintains an index of keywords to images and when a query is submitted, the image search engine identifies images for the search results using the index of keywords to identify images that match the submitted query and returns as search results links to the identified images, the image search engine being different from the web-based search engine; and
displaying an indication of each cluster of images as search results for the received image query.
9. A method performed by a computing device for generating clusters of images, by a method comprising:
receiving an image query having search terms;
submitting by the computing device the image query to a web-based search engine and receives search results that include links to web pages and text associated with each link, the text being a snippet describing content of the linked-to web page, the web-based search engine being a search engine that maintains an index of keywords to web pages and when a query is submitted, the web-based search engine identifies web pages for search results using the index of keywords to identify web pages that match the submitted query and returns as search results links to the identified web pages and associated snippets;
identifying by the computing device phrases associated with the image query by identifying phrases from the snippets of the search results;
identifying by the computing device images associated with each identified phrase by submitting each identified phrase to an image search engine and receiving identifications of images as search results for each submitted phrase wherein the identified images associated with each submitted phrase represent a cluster of images related to that submitted phrase, the image search engine being a search engine that maintains an index of keywords to images and when a query is submitted, the image search engine identifies images for the search results using the index of keywords to identify images that match the submitted query and returns as search results links to the identified images, the image search engine being different from the web-based search engine; and
displaying an indication of each cluster of images as search results for the received image query.
1. A computing device for generating clusters of images, comprising:
a memory containing computer-executable instructions comprising:
a component that receives an image query having search terms;
a component that submits the image query to a web-based search engine and receives search results that include links to web pages and text associated with each link, the text being a snippet describing content of the linked-to web page, the web-based search engine being a search engine that maintains an index of keywords to web pages and when a query is submitted, the web-based search engine identifies web pages for search results using the index of keywords to identify web pages that match the submitted query and returns as search results links to the identified web pages and associated snippets;
a component that identifies phrases associated with the image query by identifying phrases from the snippets of the search results;
a component that identifies images associated with each identified phrase by submitting each identified phrase to an image search engine and receiving identifications of images as search results for each submitted phrase wherein the identified images associated with each submitted phrase represent a cluster of images related to that submitted phrase, the image search engine being a search engine that maintains an index of keywords to images and when a query is submitted, the image search engine identifies images for the search results using the index of keywords to identify images that match the submitted query and returns as search results links to the identified images, the image search engine being different from the web-based search engine; and
a component that displays an indication of each cluster of images as search results for the received image query; and
a processor for executing the computer-executable instructions stored in the memory.
2. The computing device of
3. The computing device of
5. The computer-readable medium of
6. The computer-readable medium of
7. The computer-readable medium of
8. The computer-readable medium of
11. The method of
13. The method of
|
Many search engine services, such as Google and Overture, provide for searching for information that is accessible via the Internet. These search engine services allow users to search for display pages, such as web pages, that may be of interest to users. After a user submits a search request (also referred to as a “query”) that includes search terms, the search engine service identifies web pages that may be related to those search terms. To quickly identify related web pages, the search engine services may maintain a mapping of keywords to web pages. This mapping may be generated by “crawling” the web (i.e., the World Wide Web) to identify the keywords of each web page. To crawl the web, a search engine service may use a list of base web pages to identify all web pages that are accessible through those base web pages. The keywords of any particular web page can be identified using various well-known information retrieval techniques, such as identifying the words of a headline, the words supplied in the metadata of the web page, the words that are highlighted, and so on. The search engine service may generate a relevance score to indicate how related the information of the web page may be to the search request. The search engine service then displays to the user links to those web pages in an order that is based on their relevance.
Several search engine services also provide for searching for images that are available on the Internet. These image search engines typically generate a mapping of keywords to images by crawling the web in much the same way as described above for mapping keywords to web pages. An image search engine service can identify keywords based on text of the web pages that contain the images. An image search engine may also gather keywords from metadata associated with images of web-based image forums, which are an increasingly popular mechanism for people to publish their photographs and other images. An image forum allows users to upload their photographs and requires the users to provide associated metadata such as title, camera setting, category, and description. The image forums typically allow reviewers to rate each of the uploaded images and thus have ratings on the quality of the images. Regardless of how the mappings are generated, an image search engine service inputs an image query and uses the mapping to find images that are related to the image query. An image search engine service may identify thousands of images that are related to an image query and presents thumbnails of the related images. To help a user view the images, an image search engine service may order the thumbnails based on relevance of the images to the image query. An image search engine service may also limit the number of images that are provided to a few hundred of the most relevant images so as not to overwhelm the viewer.
Unfortunately, the relevance determination may not be particularly accurate because image queries may be ambiguous (e.g., “tiger” may represent the animal or the golfer), the keywords derived from web pages may not be very related to an image of the web page (e.g., a web page can contain many unrelated images), and so on. To help a user view the thousands of images, an image search engine service could cluster a search result based on the content of the images and present the clusters, rather than individual images, to the user. Such clustering techniques include content-based techniques and link-based techniques. The content-based techniques use low-level visual information to identify related images. There are, however, disadvantages to content-based clustering. Content-based clustering is computationally expensive and cannot be practically performed in real time when an image search result contains thousands of images. Moreover, if the clustering is limited to a few hundred of what are thought to be the most relevant images, some very relevant images may be missed because of the difficulties in assessing relevance. The link-based search techniques typically assume that images on the same web page are likely to be related and that images on web pages that are each linked to by the same web page are related. Since this assumption is, however, often not true, unrelated images are often clustered together. It is also difficult for either cluster technique to automatically identify meaningful names for a cluster of images. As a result, a user may not be able to effectively identify relevant clusters.
A method and system for generating clusters of images for a search result of an image query is provided. An image search system receives an original image query, identifies phrases related to the original image query, submits to an image search engine each of the identified phrases as an image query, and provides the images of each image search result as a cluster of images for the image search result of the original image query. When an original image query is received, the search system identifies text associated with the original image query by submitting the original image query to a search engine. The search system identifies phrases from the text of the web page containing the search result. The search system uses each of the identified phrases as an image query and submits the image queries to an image search engine. The search system considers the image search result for each image query to represent a cluster of related images. The search system then presents the clusters of images as the images of the image search result of the original image query.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
A method and system for generating clusters of images for a search result of an image query is provided. In one embodiment, an image search system receives an original image query, identifies phrases related to the original image query, submits to an image search engine each of the identified phrases as an image query, and provides the images of each image search result as a cluster of images for the image search result of the original image query. When an image query is received, the search system identifies text associated with the image query. The search system may identify the text by submitting the original image query to a conventional web page search engine. The conventional search engine searches for web pages that are related to the original image query and returns a web page that contains links to the related web pages possibly along with a snippet of text from those web pages. Upon receiving the web page containing the search result, the search system identifies phrases from the text of the web page containing the search result. For example, if the original image query is “tiger,” then the search result web page (as part of the links or in the snippets) may contain the phrases “white tiger,” “Tiger Woods,” “crouching tiger,” and so on. The search system uses each of the identified phrases as an image query and submits the image queries to an image search engine. The search system considers the image search result for each image query to represent a cluster of related images. For example, the search system submits to an image search engine the phrase “white tiger” as an image query and receives the image search result. The search system considers images of the image search result to be a cluster of images relating to “white tiger.” The search system also submits to an image search engine the phrase “Tiger Woods” as an image query and receives the image search result. The search system considers the images of the image search result to be a cluster of images related to “Tiger Woods.” The search system may also submit the original image query to the image search engine to generate an “others” cluster. The search system may remove from the “others” cluster those images that are already in another cluster. For example, the search system submits “tiger” to the image search engine, removes from the image search result the images that are also in the “white tiger,” “Tiger Woods,” and “crouching tiger” image search results and forms an “others” cluster of the remaining images. The search system then presents to a user the clusters of images as the images of the image search result of the original image query, such as “tiger.” In addition, the search system may use the phrase of the image query as the name of the cluster. For example, one cluster may be named “white tiger,” and another cluster may be named “Tiger Woods.” In this way, the search system can generate an image search result that has clusters of images without the computational expense of identifying clusters from a set of images and can provide meaningful names to the clusters.
In one embodiment, the search system may identify phrases from query logs. A query log contains the queries that have been submitted to a search engine. The search system may identify the most popular queries that contain the original image query and use those as phrases. For example, the most popular queries that contain “tiger” may be “Tiger Woods” and “Detroit Tigers.” The search system then submits each query to an image search engine and names the cluster of image search result with the text of the query.
In one embodiment, the search system may present the images within each cluster in an order based on relevance as determined by the image search engine service to which the identified phrase was submitted. The search system may also generate sub-clusters of images within each cluster. The search system may identify sub-clusters based on the visual similarities between the images of a cluster. Because of the computational expense of identifying visually similar images, the search system may maintain a mapping of images (e.g., identified by URL) to a feature set of visual features. The search system can then generate the sub-clusters based on the feature sets of the images without having to generate the feature sets at the time of the query. The search system may use various well-known clustering techniques to identify clusters of visually similar images from the feature sets.
In one embodiment, the search system may identify phrases from the text identified from an image search result provided by an image search engine, such as one adapted to search image forums. As described above, an image forum may contain metadata associated with the images that have been uploaded by users of the forum. The search system may submit the original image query to the image search engine for an image forum. The search result may contain the thumbnails of images along with metadata. The search system identifies the text of the metadata for use in identifying phrases for use in generating clusters of images for the image search result. The search system may identify text from search results provided by various web page-based search engines and image search engines.
The search system may identify phrases using various techniques for identifying key phrases. For example, the search system may extract all phrases of certain lengths (e.g., n-grams) from the text of the search result and identify the various properties of the phrases such as phrase frequency, the document frequency, phrase length, and so on. The search system may identify the key phrases based on the properties of the phrases. The search system may also filter the key phrases based on those that are unlikely to generate meaningful results (e.g., too many noise words), likely to generate redundant results (e.g., “tiger photo” and “tiger photograph”), and so on. After the search system submits a key phrase and receives the image search result for the cluster associated with that key phrase, the search system may merge, discard, or otherwise modify the clusters. For example, the search system may discard clusters that have too many or too few images or that have too many images in common with other clusters. The search system may also merge clusters with too few images that appear to have many images in common. Techniques for identifying phrases from a search result that can be used for clustering the search results are described in Zeng, H., He, Q., Chen, Z., Ma, W., and Ma, J., “Learning to Cluster Web Search Results,” SIGIR 2004, July 25-29, Sheffield, South Yorkshire, U.K., and U.S. patent application Ser. No. 10/889,841, entitled “Query-Based Snippet Clustering for Search Result Grouping” and filed on Jul. 13, 2004, which are hereby both incorporated by reference. One technique described in these references trains a linear regression model to learn the scores of phrases from a feature set of the phrases.
The computing devices on which the search system may be implemented may include a central processing unit, memory, input devices (e.g., keyboard and pointing devices), output devices (e.g., display devices), and storage devices (e.g., disk drives). The memory and storage devices are computer-readable media that may contain instructions that implement the search system. In addition, the data structures and message structures may be stored or transmitted via a data transmission medium, such as a signal on a communications link. Various communications links may be used, such as the Internet, a local area network, a wide area network, or a point-to-point dial-up connection.
The search system may receive original image queries from various computing systems or devices including personal computers, server computers, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The search system may also provide its services to various computing systems such as personal computers, cell phones, personal digital assistants, consumer electronics, home automation devices, and so on.
The search system may be described in the general context of computer-executable instructions, such as program modules, executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, and so on that perform particular tasks or implement particular abstract data types. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments. For example, the search system may include a page search engine and/or an image search engine and may not use the services of an external search engine.
Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims. The search system may be used to cluster content of various types, such as photographs, drawings, artwork, videos, music, and so on. The search system may also be used an alternate way to cluster web pages of a conventional web page search. The search system may identify phrases for clusters as described above from text derived from the search result by submitting the original page query to a web page search engine. The search can then submit each phrase to a web page search engine to identify the web pages for the cluster and return the clusters as the search result for the original page query. The search system may also submit variations of the original image query, rather than the original image query itself, to identify text. For example, if the original image query is long or does not produce enough results, the search system may divide the original image query into sub-queries and submit each sub-query to a search engine to identify text. Accordingly, the invention is not limited except as by the appended claims.
Zhang, Lei, Ma, Wei-Ying, Jing, Feng, Li, Mingjing, Wang, Chang-Hu
Patent | Priority | Assignee | Title |
10007946, | Mar 07 2013 | VAST COM; VAST COM, INC | Systems, methods, and devices for measuring similarity of and generating recommendations for unique items |
10109001, | Mar 13 2013 | Vast.com, Inc. | Systems, methods, and devices for determining and displaying market relative position of unique items |
10115074, | Dec 12 2007 | Vast.com, Inc. | Predictive conversion systems and methods |
10120883, | Jan 23 2006 | Microsoft Technology Licensing, LLC | User interface for viewing clusters of images |
10127596, | Dec 10 2013 | VAST COM, INC | Systems, methods, and devices for generating recommendations of unique items |
10157231, | Mar 07 2013 | Vast.com, Inc. | Systems, methods, and devices for identifying and presenting identifications of significant attributes of unique items |
10185769, | Jun 08 2011 | Meta Platforms, Inc | Presenting images as search results |
10268704, | Oct 12 2017 | Vast.com, Inc. | Partitioned distributed database systems, devices, and methods |
10296158, | Dec 20 2011 | Verizon Patent and Licensing Inc | Systems and methods involving features of creation/viewing/utilization of information modules such as mixed-media modules |
10303723, | Jun 12 2012 | R2 SOLUTIONS LLC | Systems and methods involving search enhancement features associated with media modules |
10387503, | Dec 15 2011 | R2 SOLUTIONS LLC | Systems and methods involving features of search and/or search integration |
10394882, | Feb 19 2014 | International Business Machines Corporation | Multi-image input and sequenced output based image search |
10417289, | Jun 12 2012 | YAHOO ASSETS LLC | Systems and methods involving integration/creation of search results media modules |
10504555, | Dec 20 2011 | Verizon Patent and Licensing Inc | Systems and methods involving features of creation/viewing/utilization of information modules such as mixed-media modules |
10521692, | Feb 08 2010 | Microsoft Technology Licensing, LLC | Intelligent image search results summarization and browsing |
10572555, | Mar 07 2013 | Vast.com, Inc. | Systems, methods, and devices for identifying and presenting identifications of significant attributes of unique items |
10643265, | Mar 07 2013 | Vast.com, Inc. | Systems, methods, and devices for measuring similarity of and generating recommendations for unique items |
10839442, | Mar 13 2013 | Vast.com, Inc. | Systems, methods, and devices for determining and displaying market relative position of unique items |
10942976, | Mar 07 2013 | Vast.com, Inc. | Systems, methods, and devices for identifying and presenting identifications of significant attributes of unique items |
10963942, | Dec 10 2013 | Vast.com, Inc. | Systems, methods, and devices for generating recommendations of unique items |
11048734, | Aug 20 2018 | Pinterest, Inc. | Auto-completion based on content similarities |
11099714, | Feb 28 2012 | Verizon Patent and Licensing Inc | Systems and methods involving creation/display/utilization of information modules, such as mixed-media and multimedia modules |
11127067, | Mar 07 2013 | Vast.com, Inc. | Systems, methods, and devices for measuring similarity of and generating recommendations for unique items |
11204957, | Feb 19 2014 | International Business Machines Corporation | Multi-image input and sequenced output based image search |
11210318, | Oct 12 2017 | Vast.com, Inc. | Partitioned distributed database systems, devices, and methods |
11270252, | Dec 12 2007 | Vast.com, Inc. | Predictive conversion systems and methods |
11423100, | Mar 07 2013 | Vast.com, Inc. | Systems, methods, and devices for identifying and presenting identifications of significant attributes of unique items |
11651411, | Mar 13 2013 | Vast.com, Inc. | Systems, methods, and devices for determining and displaying market relative position of unique items |
11755598, | Dec 12 2007 | Vast.com, Inc. | Predictive conversion systems and methods |
11886518, | Mar 07 2013 | Vast.com, Inc. | Systems, methods, and devices for identifying and presenting identifications of significant attributes of unique items |
11947577, | Aug 20 2018 | Pinterest, Inc. | Auto-completion based on content similarities |
12061642, | Mar 29 2016 | A9 COM, INC | Methods to present search keywords for image-based queries |
12106358, | Jul 24 2020 | VAST COM, INC | Systems, methods, and devices for unified e-commerce platforms for unique items |
12141854, | Mar 07 2013 | Vast.com, Inc. | Systems, methods, and devices for measuring similarity of and generating recommendations for unique items |
8171043, | Oct 24 2008 | R2 SOLUTIONS LLC | Methods for improving the diversity of image search results |
8326845, | Dec 12 2007 | Vast.com, Inc. | Predictive conversion systems and methods |
8375037, | Dec 12 2007 | Vast.com, Inc. | Predictive conversion systems and methods |
8774526, | Feb 08 2010 | Microsoft Technology Licensing, LLC | Intelligent image search results summarization and browsing |
8868572, | Dec 12 2007 | Vast.com, Inc. | Predictive conversion systems and methods |
8943074, | Feb 05 2013 | International Business Machines Corporation | Generating and displaying an image tour in response to a user search query |
9092458, | Mar 08 2005 | iRobot Corporation | System and method for managing search results including graphics |
9104718, | Mar 07 2013 | VAST COM, INC | Systems, methods, and devices for measuring similarity of and generating recommendations for unique items |
9324104, | Mar 07 2013 | Vast.com, Inc. | Systems, methods, and devices for measuring similarity of and generating recommendations for unique items |
9465873, | Mar 07 2013 | VAST COM, INC | Systems, methods, and devices for identifying and presenting identifications of significant attributes of unique items |
9600919, | Oct 20 2009 | YAHOO ASSETS LLC | Systems and methods for assembling and/or displaying multimedia objects, modules or presentations |
9672282, | Apr 14 2010 | NAVER CORPORATION | Method and system for providing query using an image |
9690857, | Mar 07 2013 | Vast.com, Inc. | Systems, methods, and devices for identifying and presenting identifications of significant attributes of unique items |
9710843, | Mar 07 2013 | Vast.com, Inc. | Systems, methods, and devices for measuring similarity of and generating recommendations for unique items |
9799000, | Dec 12 2007 | Vast.com, Inc. | Predictive conversion systems and methods |
9830635, | Mar 13 2013 | VAST COM, INC | Systems, methods, and devices for determining and displaying market relative position of unique items |
9843823, | May 23 2012 | Verizon Patent and Licensing Inc | Systems and methods involving creation of information modules, including server, media searching, user interface and/or other features |
ER2396, |
Patent | Priority | Assignee | Title |
6119124, | Mar 26 1998 | R2 SOLUTIONS LLC | Method for clustering closely resembling data objects |
6349296, | Mar 26 1998 | R2 SOLUTIONS LLC | Method for clustering closely resembling data objects |
6643641, | Apr 27 2000 | Mineral Lassen LLC | Web search engine with graphic snapshots |
6728752, | Jan 26 1999 | GOOGLE LLC | System and method for information browsing using multi-modal features |
6823335, | Oct 05 1998 | Canon Kabushiki Kaisha | Information search apparatus and method, and storage medium |
6944612, | Nov 13 2002 | Xerox Corporation | Structured contextual clustering method and system in a federated search engine |
7010751, | Feb 18 2000 | MIND FUSION, LLC | Methods for the electronic annotation, retrieval, and use of electronic images |
7051019, | Aug 17 1999 | BEN GROUP, INC | Method and system for obtaining images from a database having images that are relevant to indicated text |
7065520, | Oct 03 2000 | Method of locating web-pages by utilising visual images | |
7099860, | Oct 30 2000 | Microsoft Technology Licensing, LLC | Image retrieval systems and methods with semantic and feature based relevance feedback |
7644373, | Jan 23 2006 | Microsoft Technology Licensing, LLC | User interface for viewing clusters of images |
20040249774, | |||
20060026152, | |||
20060242178, | |||
20070174865, | |||
20070174872, | |||
20070209025, | |||
20080086468, | |||
20080086686, |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Jan 23 2006 | Microsoft Corporation | (assignment on the face of the patent) | / | |||
Mar 13 2006 | JING, FENG | Microsoft Corporation | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 017479 | /0373 | |
Mar 13 2006 | ZHANG, LEI | Microsoft Corporation | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 017479 | /0373 | |
Mar 13 2006 | MA, WEI-YING | Microsoft Corporation | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 017479 | /0373 | |
Mar 14 2006 | WANG, CHANG-HU | Microsoft Corporation | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 017479 | /0373 | |
Apr 17 2006 | LI, MINGJING | Microsoft Corporation | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 017479 | /0373 | |
Oct 14 2014 | Microsoft Corporation | Microsoft Technology Licensing, LLC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 034543 | /0001 |
Date | Maintenance Fee Events |
Jan 03 2014 | REM: Maintenance Fee Reminder Mailed. |
May 25 2014 | EXP: Patent Expired for Failure to Pay Maintenance Fees. |
Date | Maintenance Schedule |
May 25 2013 | 4 years fee payment window open |
Nov 25 2013 | 6 months grace period start (w surcharge) |
May 25 2014 | patent expiry (for year 4) |
May 25 2016 | 2 years to revive unintentionally abandoned end. (for year 4) |
May 25 2017 | 8 years fee payment window open |
Nov 25 2017 | 6 months grace period start (w surcharge) |
May 25 2018 | patent expiry (for year 8) |
May 25 2020 | 2 years to revive unintentionally abandoned end. (for year 8) |
May 25 2021 | 12 years fee payment window open |
Nov 25 2021 | 6 months grace period start (w surcharge) |
May 25 2022 | patent expiry (for year 12) |
May 25 2024 | 2 years to revive unintentionally abandoned end. (for year 12) |