An architecture for a multimedia search system is described. To perform similarity matching of multimedia query frames against reference content, reference database comprising of a cluster index using cluster keys to perform similarity matching and a multimedia index to perform sequence matching is built. Methods to update and maintain the reference database that enables addition and removal of the multimedia contents, including portions of multimedia content, from the reference database in a running system are described. Hierarchical multi-level partitioning methods to organize the reference database are presented. Smart partitioning of the reference multimedia content according to the nature of the multimedia content, and according to the popularity among the social media, that supports scalable fast multimedia identification is also presented. A caching mechanism for multimedia search queries in a centralized or in a decentralized distributed system and a client based local multimedia search system enabling multimedia tracking are described.
|
0. 69. A method comprising:
receiving, by a server device, from a client device, a first query signature that was generated by the client device and that represents a portion of a first content unit, wherein the client device presented the portion of the first content unit;
based at least in part on the received first query signature, selecting a set of reference signatures representing additional portions of the first content unit; and
sending, by the server device, to the client device, the selected set of reference signatures to facilitate the client device:
(i) comparing a second query signature generated by the client device and that represents a portion of a second content unit that is different from the first content unit with the set of reference signatures;
(ii) determining that the comparing resulted in no match being found, and
(iii) responsive to determining that the comparing resulted in no match being found, performing an action.
1. A method of tracking multimedia content, the method comprising:
searching in a remote database in response to a query sent from a user device for multimedia content of a multimedia program that is currently playing on the user device;
finding matching signatures for the multimedia content in a reference program stored in the remote database;
retrieving the matching signatures and reference signatures for succeeding segments of the reference program beginning from the point in the reference program the multimedia content was found in the remote database;
storing the matching signatures and reference signatures in a local database on the user device;
tracking the multimedia program in the local database to find matching content for the succeeding segments of the reference program;
determining the multimedia content has changed to different multimedia content than that stored in the succeeding segments of the reference program; and
replacing the determined different multimedia content to a preselected multimedia content.
0. 85. A non-transitory computer-readable storage medium having stored thereon program instructions that, when executed by one or more processors of a server device, cause the server device to perform operations comprising:
receiving, from a client device, a first query signature that was generated by the client device and that represents a portion of a first content unit, wherein the client device presented the portion of the first content unit;
based at least in part on the received first query signature, selecting a set of reference signatures representing additional portions of the first content unit; and
sending, to the client device, the selected set of reference signatures to facilitate the client device:
(i) comparing a second query signature generated by the client device and that represents a portion of a second content unit that is different from the first content unit with the set of reference signatures;
(ii) determining that the comparing resulted in no match being found, and
(iii) responsive to determining that the comparing resulted in no match being found, performing an action.
0. 21. A method comprising:
generating, by a client device, a first query signature representing a portion of a first content unit, wherein the portion of the first content unit is presented by the client device;
sending, by the client device, to a server device, the generated first query signature;
responsive to sending the generated first query signature, receiving, by the client device, from the server device, a set of reference signatures representing additional portions of the first content unit, wherein the set of reference signatures was selected by the server device based at least in part on the generated first query signature;
generating, by the client device, a second query signature representing a portion of a second content unit that is different from the first content unit;
comparing, by the client device, the generated second query signature with the received set of reference signatures;
determining, by the client device, that the comparing resulted in no match being found; and
responsive to determining that the comparing resulted in no match being found, performing, by the client device, an action.
15. A computer readable non-transitory medium storing a computer program which causes a computer system to perform a method of tracking multimedia content, the method comprising:
searching in a remote database in response to a query sent from a user device for multimedia content of a multimedia program that is currently playing on the user device;
finding matching signatures for the multimedia content in a reference program stored in the remote database;
retrieving the matching signatures and reference signatures for succeeding segments of the reference program beginning from the point in the reference program the multimedia content was found in the remote database;
storing the matching signatures and reference signatures in a local database on the user device;
tracking the multimedia program in the local database to find matching content for the succeeding segments of the reference program;
determining the multimedia content has changed to different multimedia content than that stored in the succeeding segments of the reference program; and
replacing the determined different multimedia content to a preselected multimedia content.
0. 101. A server device comprising one or more processors and a non-transitory computer-readable storage medium having stored thereon program instructions that, when executed by the one or more processors, cause the server device to perform operations comprising:
receiving, from a client device, a first query signature that was generated by the client device and that represents a portion of a first content unit, wherein the client device presented the portion of the first content unit;
based at least in part on the received first query signature, selecting a set of reference signatures representing additional portions of the first content unit; and
sending, to the client device, the selected set of reference signatures to facilitate the client device:
(i) comparing a second query signature generated by the client device and that represents a portion of a second content unit that is different from the first content unit with the set of reference signatures;
(ii) determining that the comparing resulted in no match being found, and
(iii) responsive to determining that the comparing resulted in no match being found, performing an action.
0. 37. A non-transitory computer-readable storage medium having stored thereon program instructions that, when executed by one or more processors of a client device, cause the client device to perform operations comprising:
generating a first query signature representing a portion of a first content unit, wherein the portion of the first content unit is presented by the client device;
sending, to a server device, the generated first query signature;
responsive to sending the generated first query signature, receiving, from the server device, a set of reference signatures representing additional portions of the first content unit, wherein the set of reference signatures was selected by the server device based at least in part on the generated first query signature;
generating a second query signature representing a portion of a second content unit that is different from the first content unit;
comparing the generated second query signature with the received set of reference signatures;
determining that the comparing resulted in no match being found; and
responsive to determining that the comparing resulted in no match being found, performing an action.
0. 53. A client device comprising one or more processors and a non-transitory computer-readable storage medium having stored thereon program instructions that, when executed by the one or more processors, cause the client device to perform operations comprising:
generating a first query signature representing a portion of a first content unit, wherein the portion of the first content unit is presented by the client device;
sending, to a server device, the generated first query signature;
responsive to sending the generated first query signature, receiving, from the server device, a set of reference signatures representing additional portions of the first content unit, wherein the set of reference signatures was selected by the server device based at least in part on the generated first query signature;
generating a second query signature representing a portion of a second content unit that is different from the first content unit;
comparing the generated second query signature with the received set of reference signatures;
determining that the comparing resulted in no match being found; and
responsive to determining that the comparing resulted in no match being found, performing an action.
10. An apparatus for tracking multimedia content, the apparatus comprising:
a user processing device configured for sending a query to a remote database to search for multimedia content of a multimedia program identified by a user, wherein matching signatures are found for the multimedia content in a reference program stored in the remote database, and the matching signatures and reference signatures for succeeding segments of the reference program beginning from the point in the reference program the multimedia content was found are retrieved from the remote database; and
a local database coupled to the user processing device and configured for storing the matching signatures and reference signatures retrieved from the remote database,
wherein the user processing device is further configured to track the multimedia program in the local database to find matching content for the succeeding segments of the reference program,
the user processing device is further configured to determine the multimedia content has changed to different multimedia content than that stored in the succeeding segments of the reference program, and
wherein the user processing device is further configured to replace the determined different multimedia content to a preselected multimedia content.
2. The method of
playing the preselected multimedia content for a prespecified duration; and
returning to play the multimedia program upon detecting matching content for subsequent segments of the reference program.
3. The method of
4. The method of
5. The method of
6. The method of
7. The method of
blocking the playing of the preselected multimedia content; and
returning to display the multimedia program upon detecting matching content for subsequent segments of the reference program.
8. The method of
11. The apparatus of
12. The apparatus of
13. The apparatus of
14. The apparatus of
16. The computer readable non-transitory medium method of
playing the preselected multimedia content for a prespecified duration; and
returning to play the multimedia program upon detecting matching content for subsequent segments of the reference program.
17. The computer readable non-transitory medium method of
18. The computer readable non-transitory medium method of
blocking the playing of the preselected multimedia content; and
returning to display the multimedia program upon detecting matching content for subsequent segments of the reference program.
19. The computer readable non-transitory medium method of
20. The computer readable non-transitory medium method of
0. 22. The method of claim 21, wherein the first content unit comprises video content.
0. 23. The method of claim 22, wherein the first content unit further comprises audio content.
0. 24. The method of claim 21, wherein the client device is a television set and wherein presented by the client device comprises displayed by the client device.
0. 25. The method of claim 21, wherein the portion of the second content unit is received but not presented by the client device.
0. 26. The method of claim 21, wherein the first content unit is a television show and the second content unit is an advertisement.
0. 27. The method of claim 21, wherein comparing the generated second query signature with the received set of reference signatures comprises using a track search technique to compare the generated second query signature with the received set of reference signatures.
0. 28. The method of claim 21, wherein comparing the generated second query signature with the received set of reference signatures comprises using a full search query technique to compare the generated second query signature with the received set of reference signatures.
0. 29. The method of claim 21, wherein comparing the generated second query signature with the received set of reference signatures comprises using both a track search technique and a full search query technique to compare the generated second query signature with the received set of reference signatures.
0. 30. The method of claim 21, wherein performing the action comprises:
using user-preference data associated with a user of the client device to identify a user-preferred action; and
performing the identified user-preferred action.
0. 31. The method of claim 21, wherein performing the action comprises at least temporarily blocking presentation of the second content unit.
0. 32. The method of claim 21, wherein performing the action comprises blocking presentation of the second content unit for a duration of the second content unit.
0. 33. The method of claim 21, wherein performing the action comprises switching to a different channel on which to receive and present content.
0. 34. The method of claim 21, wherein performing the action comprises switching to presenting different content.
0. 35. The method of claim 21, wherein the client device is a television set and wherein performing the action comprises initiating a picture-in-picture mode of the television set.
0. 36. The method of claim 21, wherein performing the action comprises reducing an audio volume level of the client device.
0. 38. The non-transitory computer-readable storage medium of claim 37, wherein the first content unit comprises video content.
0. 39. The non-transitory computer-readable storage medium of claim 38, wherein the first content unit further comprises audio content.
0. 40. The non-transitory computer-readable storage medium of claim 37, wherein the client device is a television set and wherein presented by the client device comprises displayed by the client device.
0. 41. The non-transitory computer-readable storage medium of claim 37, wherein the portion of the second content unit is received but not presented by the client device.
0. 42. The non-transitory computer-readable storage medium of claim 37, wherein the first content unit is a television show and the second content unit is an advertisement.
0. 43. The non-transitory computer-readable storage medium of claim 37, wherein comparing the generated second query signature with the received set of reference signatures comprises using a track search technique to compare the generated second query signature with the received set of reference signatures.
0. 44. The non-transitory computer-readable storage medium of claim 37, wherein comparing the generated second query signature with the received set of reference signatures comprises using a full search query technique to compare the generated second query signature with the received set of reference signatures.
0. 45. The non-transitory computer-readable storage medium of claim 37, wherein comparing the generated second query signature with the received set of reference signatures comprises using both a track search technique and a full search query technique to compare the generated second query signature with the received set of reference signatures.
0. 46. The non-transitory computer-readable storage medium of claim 37, wherein performing the action comprises:
using user-preference data associated with a user of the client device to identify a user-preferred action; and
performing the identified user-preferred action.
0. 47. The non-transitory computer-readable storage medium of claim 37, wherein performing the action comprises at least temporarily blocking presentation of the second content unit.
0. 48. The non-transitory computer-readable storage medium of claim 37, wherein performing the action comprises blocking presentation of the second content unit for a duration of the second content unit.
0. 49. The non-transitory computer-readable storage medium of claim 37, wherein performing the action comprises switching to a different channel on which to receive and present content.
0. 50. The non-transitory computer-readable storage medium of claim 37, wherein performing the action comprises switching to presenting different content.
0. 51. The non-transitory computer-readable storage medium of claim 37, wherein the client device is a television set and wherein performing the action comprises initiating a picture-in-picture mode of the television set.
0. 52. The non-transitory computer-readable storage medium of claim 37, wherein performing the action comprises reducing an audio volume level of the client device.
0. 54. The client device of claim 53, wherein the first content unit comprises video content.
0. 55. The client device of claim 54, wherein the first content unit further comprises audio content.
0. 56. The client device of claim 53, wherein the client device is a television set and wherein presented by the client device comprises displayed by the client device.
0. 57. The client device of claim 53, wherein the portion of the second content unit is received but not presented by the client device.
0. 58. The client device of claim 53, wherein the first content unit is a television show and the second content unit is an advertisement.
0. 59. The client device of claim 53, wherein comparing the generated second query signature with the received set of reference signatures comprises using a track search technique to compare the generated second query signature with the received set of reference signatures.
0. 60. The client device of claim 53, wherein comparing the generated second query signature with the received set of reference signatures comprises using a full search query technique to compare the generated second query signature with the received set of reference signatures.
0. 61. The client device of claim 53, wherein comparing the generated second query signature with the received set of reference signatures comprises using both a track search technique and a full search query technique to compare the generated second query signature with the received set of reference signatures.
0. 62. The client device of claim 53, wherein performing the action comprises:
using user-preference data associated with a user of the client device to identify a user-preferred action; and
performing the identified user-preferred action.
0. 63. The client device of claim 53, wherein performing the action comprises at least temporarily blocking presentation of the second content unit.
0. 64. The client device of claim 53, wherein performing the action comprises blocking presentation of the second content unit for a duration of the second content unit.
0. 65. The client device of claim 53, wherein performing the action comprises switching to a different channel on which to receive and present content.
0. 66. The client device of claim 53, wherein performing the action comprises switching to presenting different content.
0. 67. The client device of claim 53, wherein the client device is a television set and wherein performing the action comprises initiating a picture-in-picture mode of the television set.
0. 68. The client device of claim 53, wherein performing the action comprises reducing an audio volume level of the client device.
0. 70. The method of claim 69, wherein the first content unit comprises video content.
0. 71. The method of claim 70, wherein the first content unit further comprises audio content.
0. 72. The method of claim 69, wherein the client device is a television set and wherein presented the portion of the first content unit comprises displayed the portion of the first content unit.
0. 73. The method of claim 69, wherein the portion of the second content unit is received but not presented by the client device.
0. 74. The method of claim 69, wherein the first content unit is a television show and the second content unit is an advertisement.
0. 75. The method of claim 69, wherein comparing the second query signature with the set of reference signatures comprises using a track search technique to compare the second query signature with the set of reference signatures.
0. 76. The method of claim 69, wherein comparing the second query signature with the set of reference signatures comprises using a full search query technique to compare the second query signature with the set of reference signatures.
0. 77. The method of claim 69, wherein comparing the second query signature with the set of reference signatures comprises using both a track search technique and a full search query technique to compare the second query signature with the set of reference signatures.
0. 78. The method of claim 69, wherein performing the action comprises:
using user-preference data associated with a user of the client device to identify a user-preferred action; and
performing the identified user-preferred action.
0. 79. The method of claim 69, wherein performing the action comprises at least temporarily blocking presentation of the second content unit.
0. 80. The method of claim 69, wherein performing the action comprises blocking presentation of the second content unit for a duration of the second content unit.
0. 81. The method of claim 69, wherein performing the action comprises switching to a different channel on which to receive and present content.
0. 82. The method of claim 69, wherein performing the action comprises switching to presenting different content.
0. 83. The method of claim 69, wherein the client device is a television set and wherein performing the action comprises initiating a picture-in-picture mode of the television set.
0. 84. The method of claim 69, wherein performing the action comprises reducing an audio volume level of the client device.
0. 86. The non-transitory computer-readable storage medium of claim 85, wherein the first content unit comprises video content.
0. 87. The non-transitory computer-readable storage medium of claim 86, wherein the first content unit further comprises audio content.
0. 88. The non-transitory computer-readable storage medium of claim 85, wherein the client device is a television set and wherein presented the portion of the first content unit comprises displayed the portion of the first content unit.
0. 89. The non-transitory computer-readable storage medium of claim 85, wherein the portion of the second content unit is received but not presented by the client device.
0. 90. The non-transitory computer-readable storage medium of claim 85, wherein the first content unit is a television show and the second content unit is an advertisement.
0. 91. The non-transitory computer-readable storage medium of claim 85, wherein comparing the second query signature with the set of reference signatures comprises using a track search technique to compare the second query signature with the set of reference signatures.
0. 92. The non-transitory computer-readable storage medium of claim 85, wherein comparing the second query signature with the set of reference signatures comprises using a full search query technique to compare the second query signature with the set of reference signatures.
0. 93. The non-transitory computer-readable storage medium of claim 85, wherein comparing the second query signature with the set of reference signatures comprises using both a track search technique and a full search query technique to compare the second query signature with the set of reference signatures.
0. 94. The non-transitory computer-readable storage medium of claim 85, wherein performing the action comprises:
using user-preference data associated with a user of the client device to identify a user-preferred action; and
performing the identified user-preferred action.
0. 95. The non-transitory computer-readable storage medium of claim 85, wherein performing the action comprises at least temporarily blocking presentation of the second content unit.
0. 96. The non-transitory computer-readable storage medium of claim 85, wherein performing the action comprises blocking presentation of the second content unit for a duration of the second content unit.
0. 97. The non-transitory computer-readable storage medium of claim 85, wherein performing the action comprises switching to a different channel on which to receive and present content.
0. 98. The non-transitory computer-readable storage medium of claim 85, wherein performing the action comprises switching to presenting different content.
0. 99. The non-transitory computer-readable storage medium of claim 85, wherein the client device is a television set and wherein performing the action comprises initiating a picture-in-picture mode of the television set.
0. 100. The non-transitory computer-readable storage medium of claim 85, wherein performing the action comprises reducing an audio volume level of the client device.
0. 102. The server device of claim 101, wherein the first content unit comprises video content.
0. 103. The server device of claim 102, wherein the first content unit further comprises audio content.
0. 104. The server device of claim 101, wherein the client device is a television set and wherein presented the portion of the first content unit comprises displayed the portion of the first content unit.
0. 105. The server device of claim 101, wherein the portion of the second content unit is received but not presented by the client device.
0. 106. The server device of claim 101, wherein the first content unit is a television show and the second content unit is an advertisement.
0. 107. The server device of claim 101, wherein comparing the second query signature with the set of reference signatures comprises using a track search technique to compare the second query signature with the set of reference signatures.
0. 108. The server device of claim 101, wherein comparing the second query signature with the set of reference signatures comprises using a full search query technique to compare the second query signature with the set of reference signatures.
0. 109. The server device of claim 101, wherein comparing the second query signature with the set of reference signatures comprises using both a track search technique and a full search query technique to compare the second query signature with the set of reference signatures.
0. 110. The server device of claim 101, wherein performing the action comprises:
using user-preference data associated with a user of the client device to identify a user-preferred action; and
performing the identified user-preferred action.
0. 111. The server device of claim 101, wherein performing the action comprises at least temporarily blocking presentation of the second content unit.
0. 112. The server device of claim 101, wherein performing the action comprises blocking presentation of the second content unit for a duration of the second content unit.
0. 113. The server device of claim 101, wherein performing the action comprises switching to a different channel on which to receive and present content.
0. 114. The server device of claim 101, wherein performing the action comprises switching to presenting different content.
0. 115. The server device of claim 101, wherein the client device is a television set and wherein performing the action comprises initiating a picture-in-picture mode of the television set.
0. 116. The server device of claim 101, wherein performing the action comprises reducing an audio volume level of the client device.
|
|||||||||||||||||||||||||||
This application is a continuation of U.S. patent application Ser. No. 14/151,335 filed Jan. 9, 2014 and U.S. patent application Ser. No. 14/151,294 filed Jan. 9, 2014 and issued as U.S. Pat. No. 8,965,863 which are both divisionals of U.S. patent application Ser. No. 13/102,479 filed May 6, 2011 issued as U.S. Pat. No. 8,655,878 which claims the benefit of U.S. Provisional Patent Application Ser. No. 61/331,965 entitled “Scalable, Adaptable, and Manageable System for Multimedia Identification” filed May 6, 2010 all of which are hereby incorporated by reference in their entirety.
U.S. patent application Ser. No. 14/151,294 filed Jan. 9, 2014 entitled “A Scalable, Adaptable, and Manageable System for Multimedia Identification”, U.S. patent application Ser. No. 12/141,163 filed Jun. 18, 2008 entitled “Methods and Apparatus for Providing a Scalable Identification of Digital Video Sequences”, U.S. patent application Ser. No. 12/141,337 filed on Jun. 18, 2008, “Methods and Apparatus for Multi-Dimensional Content Search and Video Identification”, U.S. patent application Ser. No. 12/491,896 filed Jun. 25, 2009 entitled “Digital Video Fingerprinting Based on Resultant Weighted Gradient Orientation Computation”, U.S. patent application Ser. No. 12/612,729 filed Nov. 5, 2009 entitled “Digital Video Content Fingerprinting Based on Scale Invariant Interest Region Detection with an Array of Anisotropic Filters”, U.S. patent application Ser. No. 12/772,566 filed May 3, 2010 entitled “Media Fingerprinting and Identification System”, U.S. patent application Ser. No. 12/788,796 filed May 27, 2010 entitled “Multi-Media Content Identification Using Multi-Level Content Signature Correlation and Fast Similarity Search”, U.S. patent application Ser. No. 12/955,416 filed Nov. 29, 2010 entitled “Digital Video Content Fingerprinting Using Image Pixel Intensity and Color Information”, and U.S. patent application Ser. No. 13/076,628 filed Mar. 31, 2011 entitled “Scale/Affine Invariant Interest Region Detection with an Array of Anisotropic Filters for Video Fingerprinting” have the same assignee as the present application, are related applications, and are hereby incorporated by reference in their entirety.
Media applications which include video and audio database management, database browsing and identification are undergoing explosive growth and are expected to continue to grow. To address this growth, there is a need for a comprehensive solution related to the problem of creating a multimedia sequence database and identifying, within such a database, a particular multimedia sequence or sequences that are tolerant of media content distortions. Multiple applications include video database mining, copyright content detection for video hosting web-sites, contextual advertising placement, and broadcast monitoring of video programming and advertisements.
Multimedia fingerprinting refers to the ability to generate associated identifying data, referred to as a fingerprint, from the multimedia image, audio and video content. A fingerprint ideally has several properties. First, the fingerprint should be much smaller than the original data. Second, the fingerprint should be designed such that it can be searched for in a large database of fingerprints. Third, the original multimedia content should not be able to be reconstructed from the fingerprint. Fourth, for multimedia content that is a distorted version of another multimedia content, fingerprints of the original and distorted versions should be similar. Examples of some common multimedia distortions include, selecting a clip of video content temporally, cropping the image data, re-encoding the image or audio data to a lower bit rate, changing a frame rate of the video or audio content, re-recording the multimedia data via some analog medium such as a camcorder in a movie theatre, and changing the aspect ratio of the image content. A fingerprint with the fourth property is deemed to be robust against such distortions. Such a system of fingerprinting and search is preferable to other methods of content identification. For example, multimedia watermarking changes the multimedia content by inserting watermark data. Unlike multimedia watermarking, fingerprinting does not change the content.
Fingerprinting is a very challenging problem. So also developing a scalable system that can easily be managed, changed, replicated is a challenging system problem.
Increasing demand for such fingerprinting and search solutions, which include standard definition (SD) and high definition (HD) formats of video, three dimensional (3D) videos, virtual reality media content, requires increasing sophistication, flexibility, and performance in the supporting algorithms and hardware. The sophistication, flexibility, and performance that are desired exceed the capabilities of current generations of software based solutions, in many cases, by an order of magnitude.
In one or more of its several aspects, the present invention recognizes and addresses problems such as those described above. To such ends, an embodiment of the invention addresses a method for creating a distributed reference multimedia database. A reference multimedia database is split into a first identifiable portion and a second identifiable portion. The first identifiable portion is stored at a first search server and the second identifiable portion is stored at a second search server. A first query is assigned to the first search server based on the first identifiable portion stored therein and a second query is assigned to the second search server based on the second identifiable portion stored therein. The first query is searched for at the first search server in parallel with the second query searched for at the second search server to find a first content stored in the first identifiable portion and a second content stored in the second identifiable portion that have a close match to the associated first query and to the associated second query.
Another embodiment of the invention addresses a method for creating a tiered multimedia reference database. A reference multimedia database is split into a first identifiable portion having multimedia content representing the most sought content and a second identifiable portion representing the remaining content. The first identifiable portion is stored at a first search server and the second identifiable portion is stored at a second search server coupled to the first search server. A first query is assigned to the first search server. The first query is searched for at the second search server if a search for the first query at the first search server is not successful to find multimedia content stored in the first search server.
Another embodiment of the invention addresses a method of tracking units of multimedia content. Reference signatures retrieved from a remote search server for currently displayed units of multimedia content are stored in a client device. Matches in query results are detected for succeeding displayed units of multimedia content using a track search approach to track current displayed units of multimedia content. Display of succeeding units of multimedia content are blocked upon a no match indication detected in the query results.
Another embodiment of the invention addresses a method for query caching. Signatures of a video sequence are generated at a client. A cache key is generated from the signatures at the client. A search of a remote reference database is requested using the cache key. The remote reference database is searched for a match with the cache key. Results linked with a matching cache key are sent to the client, wherein the results were generated from a previous full search of the reference database.
A further embodiment of the invention addresses a method of signature database organization by a cluster index. A first set of first signature records for units of multimedia content are stored in a cluster index data structure, wherein the first signature records are grouped by a cluster key. A second set of second signature records for units of multimedia content are stored in a multimedia signature index data structure, wherein the second signature records are grouped by a multimedia identification. Signatures are shared in the single server search system between the cluster index data structure and multimedia signature index data structure.
These and other features, aspects, techniques and advantages of the present invention will be apparent to those skilled in the art from the following detailed description, taken together with the accompanying drawings and claims.
The present invention will now be described more fully with reference to the accompanying drawings, in which several embodiments of the invention are shown. This invention may, however, be embodied in various forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.
It will be appreciated that the present disclosure may be embodied as methods, systems, or computer program products. Accordingly, the present inventive concepts disclosed herein may take the form of a hardware embodiment, a software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present inventive concepts disclosed herein may take the form of a computer program product on a computer readable storage medium having non-transitory computer usable program code embodied in the medium. Any suitable computer readable medium may be utilized including hard disks, CD-ROMs, optical storage devices, flash memories, or magnetic storage devices.
Computer program code or software programs that are operated upon or for carrying out operations according to the teachings of the invention may be written in a high level programming language such as C, C++, JAVA′, Smalltalk, JavaScript®, Visual Basic®, TSQL, Python, Ruby, Perl, use of .NET™ Framework, Visual Studio® or in various other programming languages. Software programs may also be written directly in a native assembler language for a target processor. A native assembler program uses instruction mnemonic representations of machine level binary instructions. Program code or computer readable medium as used herein refers to code whose format is understandable by a processor. Software embodiments of the disclosure do not depend upon their implementation with a particular programming language.
The methods described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. A computer-readable storage medium may be coupled to the processor through local connections such that the processor can read information from, and write information to, the storage medium or through network connections such that the processor can download information from or upload information to the storage medium. In the alternative, the storage medium may be integral to the processor.
Systems and methods are described that are highly scalable to very large multimedia databases. A reference multimedia database can be modified by adding a unit of multimedia content or removing an existing unit of multimedia content while it is being used for multimedia identification. A unit of multimedia content may be a frame or a sequence of frames of a video, an audio clip, other multimedia formatted data, such as frames from a movie. A unit of multimedia content may also be a television show without advertisements, an advertisement, a song, or similar unit of communication. The search system can be tuned to the desired speed of multimedia matching by centralized and distributed systems, by replication of individual search machines or search machine clusters, by use of a hierarchical tier of search machines and reference databases, by partitioning of reference databases, by multimedia query caching, by local client search methods, by client tracking, or by combinations of the previously mentioned arrangements. As an example, the search system can be implemented in a centralized client server model, or as a distributed system, or by a combination of such approaches. Also, a distributed search system may be operable on a variety of distributed networks, such as a peer to peer (P2P) system. In addition, search functions or a complete search operation may be operable at the client.
The following nomenclature is used in describing the present invention. For example, multimedia content represents any video, audio or audio-visual content. Multimedia content may also represent a series of photographs or pictures, a series of audio files, or other associated data, such as 3D video content or 4D content in which sensory feedbacks such as touch feedback sensations are presented simultaneously with visual and audio content.
The terms is signature and fingerprint both denote the same structure of a sequence of bits and may be used interchangbly. A fingerprint is generated to represent a unit of multimedia content using a fingerprinting method that operates on the unit of multimedia content. A cluster key is a type of hash key. A cluster index is a data structure that holds all of the signatures that have the same cluster key. A multimedia signature index is a data structure that is used to hold signatures associated with a unit of multimedia content.
A number of exemplary goals of a multimedia identification system include an ability to handle large capacity multimedia databases and high density media files. The multimedia identification system is to provide high performance and respond with accurate media identification when queried. Also, the overall design should be scalable to efficiently handle increasing capacity of the multimedia databases and an arbitrary length of a query sequence.
To provide for such needs,
The user site 102 may comprise, for example, a personal computer, a laptop computer, a tablet computer, or the like equipped with programs and interfaces to support data input and output and video fingerprinting and search monitoring that may be implemented both automatically and manually. The user site 102, for example, may store programs, such as the video fingerprinting and search process 112 which is an implementation of a content based video identification process of the present invention. The user site 102 may also have access to such programs through electronic media, such as may be downloaded over the Internet from an external server, accessed through a universal serial bus (USB) port from flash memory, accessed from disk media of various types, or the like. The fingerprinting and search system 100 may also suitably include more servers and user sites than shown in
User sites 102 and 103 may generate user video content which is uploaded over the Internet 104 to a server 106 for storage in the video database 108. The user sites 102 and 103, for example, may also operate a video fingerprinting and video identification process 112 to generate fingerprints and search for video content in the video database 108. The video fingerprinting and video identification process 112 in
The video database 108 may store video archives, as well as data related to video content stored in the video database 108. The video database 108 also may store a plurality of video fingerprints that have been adapted for use as described herein and in accordance with the present invention. It is noted that depending on the size of an installation, the functions of the video fingerprinting and search process 112 and the management of the video database 108 may be combined in a single processor system, such as user site 102 or server 106, and may operate as directed by separate program threads for each function.
The fingerprinting and search system 100 for both media fingerprinting and identification is readily scalable to very large multimedia databases, has high accuracy in finding a correct clip, has a low probability of misidentifying a wrong clip, and is robust to many types of distortion. The fingerprinting and search system 100 uses one or more fingerprints for a unit of multimedia content that are composed of a number of compact signatures, including cluster keys and associated metadata. The compact signatures and cluster keys are constructed to be easily searchable when scaling to a large database of multimedia fingerprints. The multimedia content is also represented by many signatures that relate to various aspects of the multimedia content that are relatively independent from each other. Such an approach allows the system to be robust to distortion of the multimedia content even when only small portions of the multimedia content are available.
Multimedia, specifically audio and video content, may undergo several different types of distortions. For instance, audio distortions may include re-encoding to different sample rates, rerecording to a different audio quality, introduction of noise and filtering of specific audio frequencies or the like. Video distortions may include cropping, stretching, re-encoding to a lower quality, using image overlays, or the like. While these distortions change the digital representation, the multimedia is perceptually similar to undistorted content to a human listener or viewer. Robustness to these distortions refers to a property that content that is perceptually similar will generate fingerprints that have a small distance according to some distance metric, such as Hamming distance for bit based signatures. Also content that is perceptually distinct from one another will generate fingerprints that have a large distance, according to the same distance metric. A search for perceptually similar content, hence, is transformed to a problem of searching for fingerprints that are a small distance away from the desired fingerprints.
It is advantageous for search operations that the signature records for a particular cluster key be stored contiguously. The set of signatures belonging to a cluster key is called a cluster. For example, 100 signatures and corresponding cluster keys may be generated having 50 signatures with a cluster key A, 30 signatures with a cluster key B and 20 signatures with a cluster key C. Thus, the 100 signatures are organized into three clusters, cluster A with 50 signatures, cluster B with 30 signatures and cluster C with 20 signatures that are stored in memory.
For each cluster key, the number of signatures and a pointer to the location where the corresponding signature records begin is stored for processing. Since the space of cluster keys may be relatively small, for example, a 16-bit cluster key implies a maximum 65,536 entries, other possible cluster keys, such as 24-bit or 32-bit cluster keys, can be used as indexes to locate signature clusters in an array. Alternative arrangements of signatures, such as organizations in the form of a binary tree or in the form of a B-tree or similar data structures may also be used. However, aspects of using the cluster keys as indexes in an array are discussed further below.
As shown in
The exemplary signature database 2000 and the exemplary signature database 2050 may be stored either in a local computer's main memory, such as RAM, or on a hard disk drive. One embodiment is to store one or both of the video database structures in main memory as access speeds are significantly faster. A performance versus capacity tradeoff may be made concerning the remaining capacity of main memory versus the remaining capacity of the hard drive once the video database structures are stored.
An exemplary embodiment of signature formation, also referred to as fingerprinting, and database formation is described in U.S. patent application Ser. No. 12/141,163 filed Jun. 18, 2008, FIGS. 11-16 and page 25, line 3 to page 28, line 18. Another exemplary embodiment of fingerprinting and database formation is described in U.S. patent application Ser. No. 12/612,729 filed Nov. 5, 2009, FIGS. 12-14 and page 28, line 20 to page 31, line 13. Another exemplary embodiment of fingerprinting and database fin nation is described in U.S. patent application Ser. No. 12/491,896 filed Jun. 25, 2009, FIGS. 8-10 and page 20, line 8 to page 24, line 22. Another exemplary embodiment of fingerprinting and database formation is described in U.S. patent application Ser. No. 12/772,566 filed May 3, 2010, FIGS. 4-9B and page 23, line 6 to page 40, line 6. A further exemplary embodiment of fingerprinting and database formation is described in U.S. patent application Ser. No. 12/955,416 filed Nov. 29, 2010, FIGS. 6-12, and page 16, line 8 to page 29, line 15. An exemplary embodiment of a system and database formation process is described in U.S. patent application Ser. No. 12/141,337 filed Jun. 18, 2008, FIGS. 1A, 1B, 1C, and 4-7, and page 6, line 15 to page 14, line 18 and page 21, line 11 to page 24, line 21. Another exemplary embodiment of a system and database formation process is described in U.S. patent application Ser. No. 12/772,566 filed May 3, 2010, FIGS. 1-3, and page 10, line 10 to page 23, line 5. A further exemplary embodiment of a system and database formation process is described in U.S. patent application Ser. No. 12/788,796 filed May 27, 2010, FIGS. 1, 2A and 2B, and page 6, line 14 to page 13, line 2. An exemplary embodiment of query search is described in U.S. patent application Ser. No. 12/141,163 filed Jun. 18, 2008, FIG. 17, and page 28, line 19 to page 29, line 6. Another exemplary embodiment of query search is described in U.S. patent application Ser. No. 12/141,337 filed Jun. 18, 2008, FIGS. 2A-3B and 8, and page 14, line 19 to page 21, line 10 and page 24, line 22 to page 26, line 10. Another exemplary embodiment of query search is described in U.S. patent application Ser. No. 12/612,729 filed Nov. 5, 2009, FIG. 15, and page 31, line 14 to page 32, line 15. Another exemplary embodiment of query search is described in U.S. patent application Ser. No. 12/772,566 filed May 3, 2010, FIGS. 10-13, and page 40, line 7 to page 45, line 14. A further exemplary embodiment of query search is described in U.S. patent application Ser. No. 12/788,796 filed May 27, 2010, FIGS. 3-10, and page 13, line 3 to page 43, line 2. Modifications of the above illustrative approaches or other approaches may be employed consistent with the teachings of the present invention.
The following discussion now focuses on further details of the search system. First a search system composed of a single server is described. Later a multi-server search system is described.
A single search server has two main databases that store video fingerprints of reference videos. Each fingerprint is a string of bits of specified length and associated metadata such as the frame number, video name, type of the signature and the like. The fingerprint may also contain the compact cluster key. A cluster key is a string of bits of smaller length than that of the main fingerprint. A cluster key can be generated at the search server using a hashing algorithm. A single search server may use a predefined hashing algorithm to generate a cluster key for a given reference signature.
An example cluster key generation process 300 is presented in
The generated cluster key 304 can be modified using other information such as metadata associated with this fingerprint 302.
The single search server system continues with the creation of the cluster index in reference to
Briefly, fingerprints having the same cluster key are grouped and stored together in an array. The beginning of this array is then indexed using the associated cluster key. Metadata of each signature along with the actual signatures are also stored in this array. Signatures from a single video are stored preferably consecutively in one chunk though alternative methods of storing are not precluded.
A cluster index is an associated array that maps a cluster key to the array of signatures associated with that cluster key. Each entry in the cluster index may also include other metadata information such as the number of signatures associated with this hash entry, list of the videos that have some signatures associated with this hash entry and the like.
It is not necessary to have only a single cluster index.
The single server search system continues with the creation of the multimedia signature index in reference to
In the single server search system, signature sharing is illustrated between cluster indexes and multimedia signature indexes in
In a first approach, shown in
In
At decision block 632, if enough space is not available to store the new signatures the process 625 proceeds to the block 634. At block 634, a new array, sufficient to hold the original number of signatures in that particular cluster as well as the new additional signatures to be added into this cluster is obtained from the system. Also at block 634, the original signatures from the old array are copied into the newly obtained array. At block 636, an exclusive write lock on the newly established cluster index is obtained. At block 638, a pointer from the cluster index array for this cluster entry is modified to point to the new array. The old array is de-allocated and returned to operating system. At block 640, the exclusive write lock on the original array is released. At block 642, the new signatures are added to the newly established array since it has enough space to store additional signatures. The process 625 returns to the decision block 632. At decision block 644, it is determined whether each cluster and for each signature in the cluster has been processed. If more processing is to be done, the loop returns to the decision block 632.
A suitable method to update the multimedia signatures index is similar to the process described in regard to
While the search server is actively running multimedia identification operations, the reference database can be updated by removing a multimedia content from it. Updating the reference database by deleting fingerprints has two main components. The first component is to update a cluster index by removing signatures associated with the associated video. The second component is to update the multimedia signature index by removing the signatures associated with the associated video.
In process 720 of
At block 724, each cluster in this list that needs modification is determined. At block 726, an exclusive read-write lock is obtained for each cluster. Note that the exclusive write lock can also be acquired for plurality of clusters that need the modification at once. At block 728, a calculation is made for start and end positions of all signatures blocks associated with the unit of multimedia content. Thus, in the state diagram 700, for the cluster associated with key 3 701, the start and end positions of the signature block to be deleted is determined to be entry 2 703 up to and including entry 11 705. At block 730, the signatures to be deleted are removed and all of the signatures that follow this block are shifted according to the space of the signatures that are removed so as to coalesce the signatures before and after the block to be deleted. Thus, after the removal is completed, all active signatures are stored consecutively. In the state diagram 700, 10 signatures are removed for the cluster associated with key 3 701 resulting in a reduced signature array 712. The signatures that are deleted in this way can be moved to the end of signature array and marked as deleted. This approach will enable the deleted content to be rapidly readded into reference database if ever needed. At block 730, metadata associated with the cluster is also updated to reflect the new number of signatures and multimedia contest present in this cluster.
As a result of operations at block 730, at the block 732, the signature array-size that holds the signature is optionally and dynamically adjusted to free up some of the memory. In the state diagram 700, the cluster array 712 with key 3 701 now has only three active signatures after deleting 10 signatures associated with the unit of multimedia content that was deleted. The total size of array 704 before deletion was 14 out of which only 3 signatures remain after deletion in the array 712. Hence the array size is dynamically readjusted by reallocating the array or by returning the extra space for 10 signatures. After this operation, the size of array 712 becomes 4. At block 734, the exclusive write is released 734. If further clusters need modification, the process 720 returns to block 726.
Some applications need the ability to add a unit of multimedia content that is being currently produced, broadcasted, streamed or displayed. For example, if a video is long then these applications may require an ability to update a reference database with parts of the videos before the end of the video. Updates that are carried out in this fashion are called live event updates. As an example, a 60 minute video may be added into the reference database in three separate update chunks or units each of 20 minutes. Note that multiple devices may request updating the same video with different or overlapping parts of a video. The methods presented here do not depend on a device or a number of devices requesting such an update.
In a first method for live event updates, each update part of the multimedia content is treated as a separate unit of multimedia content with a different video id. This case is similar to normal multimedia content update described with regards to
In a second method for live event updates, each new update can be processed by deleting previously added live event updates for this multimedia content and then re-adding a combined update of all of the previous live event update parts and the new live event update part. In this case, deletion of previous live event update parts is performed as described with regard to
In a third method, each update is treated as a part of the same unit of multimedia content. The cluster index and the multimedia signature index are updated accordingly as described with regard to
Note that the process of updating the cluster index and multimedia signature index due to a new live event update request does not depend on the source of the update request. Thus, a first update chunk request of a live event may come from a one source, a second update chunk request of a live event may come from another source and a third update chunk request may come from even different source. Thus, the search server is oblivious to the source of these live event update requests.
A further aspect of live event updating concerns deleting video from a multimedia signature index. If different live event update chunks of multimedia content are added using different content ids then deleting a content from the multimedia signature index is the same as deleting corresponding multiple contents, each deletion using the procedure described with regard to
When a reference video database becomes too large to fit on one search server, it can be divided across multiple search servers 10021, 10022, . . . , 1002N. A video identification request, formatted as a search query, is dispatched from the search query dispatcher 1003 to all of the search servers 10021, 10022, . . . , 1002N to search across the distributed reference database. For example, a similarity search, as described with regard to
The distributed reference database can be arranged in many ways. For example, the search system 1000 is configurable to assign the selected portion of the reference database according to a random splitting of the reference database. In this arrangement, selected multimedia content of a plurality of multimedia contents can be randomly assigned to one of the search servers in the search cluster. Then, the signatures associated with the selected portion of the reference database are loaded in a local cluster index on the assigned search server as well as in the multimedia signature index of that particular search server.
In another method of arranging the distributed reference database, the plurality of multimedia contents are split into specified categories. Common examples of categories for movies are—romance, thriller, animation, family, comedy, and other identifiable categories. Multimedia content is categorized through examination of metadata that may accompany the content. A movie category may be interpreted by a multimedia content analysis, for example. Also, an unknown category can be used when a specific category cannot be determined. Each search server holds a reference database for multimedia contents from a single or multiple categories. This arrangement of the reference database is termed a categorical splitting of the reference database.
An opposite method of the categorical splitting of the reference database is a method for reverse categorical splitting. In the reverse categorical splitting method, multimedia contents within the same category are purposefully distributed across as many different search servers as possible. The categorical splitting minimizes the information entropy of the distributed reference database while the reverse categorical splitting maximizes it.
The choice of the method to arrange a distributed reference database may be based on a single server's search algorithm. Some search algorithms may perform better within a database composed of similar content while other search algorithms may perform better within a database composed of non similar contents. Further, for some search algorithms composition of the reference database might not be a factor.
All normal updates or live event updates to a reference database are assigned to a specific server as per the distributed reference database splitting policy.
The multi-cluster multimedia identification system 1100 shows three search clusters 1102, 1104, and 1106 with a multimedia search query distributor 1120 and a final result unit 1124. Each cluster has its own result combiner 1110, 1112, 1114, respectively. These result combiners combine results from each individual search machine inside each cluster and then provide each cluster's final result that may be combined as a system final result in the final result unit 1124 for consumption by the query client. Every new multimedia search request 1120 is distributed by the multimedia search query distributor to one of the clusters, such as cluster 1106 while a reference database modification request 1122 is forwarded to all of the clusters 1102, 1104, and 1106. With three search clusters, three searches may operate in parallel. Also, with multi-threaded search servers, the search servers are operable to work separately on a plurality of search query requests if so assigned.
Replicating the search clusters increases the number of search queries a multi-cluster multimedia identification system can handle within a given amount of time by a factor of N where N is the number of replicated search clusters.
In many applications of a video identification system, most of the video identification queries would match a very small set of reference videos. For example, most people are interested in watching current TV programs, recent movie releases, current sport matches. The complete reference database may consist of tens of thousands of videos, however the most sought video content might be just a fraction of it. In such scenarios, a hierarchical search system can be configured as illustrated in
The most sought reference content is marked in the reference database contents and is loaded into a first tier, the tier-1 system 1204 of a multi-tier system. The most sought content can be determined in many ways. For example, if the reference database consists of recent movies, then a movie ranking report based on the weekly popularity of currently showing movies could be used to mark the most popular movie content in the reference database. The system can start without any external information about the popularity of the reference multimedia content and then build a popularity index for its reference database contents by analyzing matched results for recent search queries.
Multimedia identification search requests are dispatched by the query dispatcher 1202 and enter this tier-1 system 1204 in which the multimedia identification query is matched against only tier-1 content. If a match is not found, or if certain criteria is not met, for example, confidence in the detected match is less than a predetermined threshold or the length of the match found is less than another predetermined threshold, this multimedia identification query is sent on to the next tier, the tier-2 system 1210, of the three tier hierarchical system 1200. This search system organization is generalized to multiple tiers in a hierarchical organization where the most popular content is organized in the tier-1 system 1204, some of the next most popular content is organized into the tier-2 system 1210, some of the less popular content is organized into the tier-3 system 1216 and so on.
In
The tier structure can also be considered as a way of performing reference database caching. Similar to an L1 cache in computer system, the tier 1 contains the references that are most often accessed or matched to search queries.
The tier organization may lead to a smaller system cost due to a number of advantages. For example, overall computing load of the system may be reduced thus enabling fast content identification. The tiered structure of the multi-cluster search system delivers a lower number of queries to the second tier-2 system 1210 and an even lower number of queries of to the last tier-3 system 1216. The multi-media identification speed on a single search server is proportional to the reference database size that it holds. As the tiered structure has reduced the load on the second and third tiers compared to the first tier, the reference database size in the second tier-2 system 1210 and third tier-3 system 1216 can be increased and still maintain an adequate search speed required to answer every incoming query to the overall multi-tier identification system. Hence the tiered structure can also help in increasing the distributed reference database size that can support a particular search speed.
The most sought content can be determined in multiple ways. In a first exemplary approach to determine the most sought content, the number of search queries that match against a prespecified reference multimedia content over a predefined time interval are counted as a match number by keeping track of all search queries answered by the search system. Then all the reference multimedia contents having a match number greater than a prespecified threshold are termed as the most sought videos. In another exemplary approach, the probability of a random search query matching to set of reference multimedia content is prespecified as a probability threshold. Then, a set of reference content that achieves or exceeds this probability threshold of matching a random query is determined. This set of reference multimedia content is then defined as the most sought content. The set of most sought reference content can be determined using a cumulative probability distribution function determined by statistics maintained by the search system of the search results, for example.
If the reference content is a set of movies, then movie popularity ratings can be used to determine the top N popular movies or top x % of popular movies. These top movies then can be termed as the most sought content.
If the reference database is the TV shows, then similarly TV show ratings could be used to determine the top N popular shows or top x % of popular shows. Using this information the most sought TV shows can be determined.
Further, different tiers can have different number of clusters, each with a different number of search controllers, to handle correspondingly different query search loads and distributed reference database sizes. Thus, tier organization and clusters of search servers are two independent concepts and combinations of these two system configurations can result in an even more flexible and advantageous search system. For example, if a tier-1 system is able to respond to most of the search queries, then the following tiers would have much smaller load. In such a case, to utilize the tiers efficiently, two tier-1 clusters may share a single tier-2 system as illustrated in
When a new search query 1301 is received in the search system 1300, the query distributor 1302 dispatches the search query to the tier 1 search system 1304. The query distributor 1302 selects one of the search clusters, such as search cluster 1306, out of the two search clusters 1305 and 1306 in tier 1 and dispatches the search request 1301. The selection of the search cluster may be done in a pure random fashion, or in a round robin way, or according to any other scheduling policy such as least loaded cluster selection, or physically closest cluster selection. The result combiner 1308 combines the results of various search controllers in the search cluster 1306 and forwards the results to another query distributer 1310. Query distributor 1310 and query distributor 1302 may be implemented as a part of the same process or may be separate processes.
Query distributor 1310 checks the results for the match. If a match is found, then the results are forwarded to the result monitor 1318 which in turn forwards the result to the search client as final results 1320. If the query distributor 1310 does not find a match, it dispatches the search query to tier 2 1312. Tier 2 in this example consists of only one search cluster hence the query distributor 1310 does not need to choose a search cluster for this query in tier-2 1312. Also, in this example, a search query is dispatched to all the search servers in the tier 2. The result combiner 1315 collects the match information from all the search servers and forwards the result to the result monitor 1318 which in turn forwards the results to the query client as final results 1320. The result monitor 1318 stores the match results from tier 1 and tier 2, and sends the combined processed results as final results 1320 to the query client.
Note that the reference databases in each tier need not be static. Rather, they can be dynamic. Depending on a current profile of queries, or current time, or some other parameter or parameters, the set of reference content in the different tiers can be changed or moved around. For example, some of reference content from tier 1 that is not so frequently accessed and matched may get transferred to tier-2. To enable this functionality for system 1300, the result monitor 1318 also maintains statistics about the matches and which reference multimedia contents are matched most often across large sets of queries. Even though, it has not been shown explicitly in
In some applications, an objective might be to first find a matching reference video to a queried multimedia content and then track the queried multimedia content as it progresses against a matched set of multimedia contents with minimum overhead and reduced latency. For example, consider an application for the purpose of blocking advertisements on a television set. A reference database is instantiated in the television set and contains one or more television programs without advertisements. Whenever an advertisement is displayed on the television set, a multimedia identification query issued in response to a user's selection would return a “no match”. By having the television set configured to track the program being watched, whenever it detects deviation from the reference program being tracked it can conclude an advertisement is present. The television set then can take various actions as per the user preference. For example, the television set can simply block the advertisement till the program resumes, which the television set would know by tracking the displayed content on the television. Some other possible actions that the television set can take include switching the channel, going into picture-in-picture mode with another channel, and reducing the audio volume.
With reference to the step 1456, as the client has found a matching reference video, it then downloads signatures for the matching reference video or signatures for the part of the matching reference video around the match. The decision to download full or part of the matching reference multimedia signatures can be taken by the client or by the remote server. Various factors can affect this decision, including but not limited to the size of the matched reference content and the probability of client continuing to display the same content without changing the displayed program.
After downloading, the process 1450 builds a local search database at step 1458 with the downloaded matching reference multimedia signatures in the reference database. At step 1460, the process 1450 then enters into a track mode. At step 1462, the process 1450 makes new track search requests to the local search database to track the video. For example, the television set searches for the content being displayed for the past five seconds. The track search does not use the similarity search phase of the normal search and instead uses a correlation phase of the normal search. Hence, track search consumes less computational resources and can be performed faster.
At step 1464, a determination is made whether the results returned match the query. If the track search at the local search system return a match to the same reference, then the process goes back to step 1462 where it can make a new track search query. Instead, if at decision step 1464, a match is not found, then at step 1466, a new full search query is made to the local reference database. This full local search is more expensive than the track search query in terms of the computational resources needed. However, the full local search is faster than a remote search query because the full search is performed against a much smaller database. When the local full search results are returned, at decision step 1468, a determination is made whether a match to the same reference video is found. If indeed a match has been found, then the tracking operation is continued by returning to step 1462. Otherwise, it is concluded that the local content has diverged from the reference content and the local track method is abandoned at step 1470 and the process 1450 returns at the beginning step 1452.
The next search query 1410 is first made as a local tack search query and, for example, results in a no match finding. Hence, since the local track search filed then the same query is also made as a local full search query 1410. The local full search query 1410 is made with the same signatures and, for example matching results are found 1411. The same process with similar results 1413 is repeated for a next query 1412. For the next two queries 1414 and 1416, local track search operations return positive results 1415, 1417, so the track mode continues till that point in the time. However, for the next search query 1418, neither the local track search query nor the local full search query returns a positive match 1419. Hence, it is concluded that the content being played locally has diverged from the reference content. Further, the track mode is abandoned and a remote search query 1418 is performed. In this case, the remote system also does not find a match 1419. The next query 1420 is made directly to the remote search system, which returns a no match indication 1421 and the process is repeated for query 1422 with no match result 1423. However, for the query 1424, the remote search system finds the match 1425 and hence the client downloads the signatures, updates the local search database for this reference and enters into the track mode. The track mode successfully continues for the next three queries 1426, 1428, 1430 with successful results 1427, 1429 and 1431.
Now consider a scenario where a video that is aired on a television channel, a TV or any other device that can receive the TV channel such as a set top box may fingerprint the content aired on the television channel and may query the search system to find a matching reference. However, if two such querying devices are tuned to the same channel, there is a considerable chance that both clients may generate the same signatures and thus would generate exactly the same query. In such case, if one query is searched by the search system after another, and if somehow the search system can notice that exact same query has been searched before, it can reduce the computational overhead of the second search by remembering the results of the first search query and returning those results for the second query. This operation is termed as query caching in an embodiment of the present invention.
In one aspect, the search system can cache the completed query results. The results can be cached using a caching key storage unit, such as a look-up table which may be organized as a hash map data structure, for example. The keys for this look-up table are generated from query signatures and search results are stored as values of the look-up table. The search system can employ a separate caching server or servers or the caching functionality can be integrated at every search server.
Note that instead of generating a single hash key for the look-up table, multiple hash keys can be generated using different set of query signatures. For example, a hash key can be generated using signatures from a 5 second query. Then three hash keys could be generated for a query that is 15 seconds long by using non-overlapping signatures blocks each representing a duration of 5 seconds. While doing a search even if a single hash key out of the three hash keys is found to be in look-up table, a result associated with that key is declared as a matching result. The look-up table is searched and updated for all hash keys associated with every query. Even more hash keys can be generated using overlapping signature blocks and generating a hash key for each block of signatures. This method is termed a generation of a sequence of hash keys.
Also, note that the caching need not be performed only at the servers. The method of query caching involves generating a unique hash key or sequences of hash keys from the signatures. This can be performed at the querying client or any other device. Only this unique key or sequences of keys need to be queried to find cached query results. No actual search needs to be performed if a cached result is found. For this reason, this caching based querying for matching content can be performed in various forms such as querying to a peer to peer (P2P) network that is made of individual query clients, querying to a content delivery network (CDN) or querying to a separate caching server that only maintains such cached results.
If the search has been forwarded to the centralized cache server at step 1506, the centralized cache server checks for the results using the hash keys. If the results are found, the results are returned back to the client 1502. However, if the results are not found at the centralized cache server, then the cache server forwards the search request to the centralized search system 1508 that performs the actual search using query signatures. The main search system 1508 then informs the query client 1502 of the actual search result. The main search system 1508 also informs the cache server 1506 about the results which in turn updates its look-up table by associating the search results to the hash keys linked with this query signature.
If the search at step 1504 is forwarded to a distributed system to find the cached results, the distributed system returns with a result.
If the client 1502 decides to be a part of distributed P2P caching network, it then stores the obtained results into a lookup table with the keys being the generated hash keys at step 1504. At a later time, a different client may produce a hash key that is the same as one of the hash keys produced by this client 1502. If that new client queries using that hash key to a distributed network in which the client 1502 has joined, potentially the client 1502 can reply with the matching result. Thus the P2P caching network can share some of the search load of the main search system.
Those of skill in the art will appreciate from the present disclosure additional, alternative systems and methods for actionable television event generation, based on television program audio and video content fingerprinting, in accordance with the disclosed principles of the present invention. Thus, while particular embodiments and applications of the present invention have been illustrated and described, it is to be understood that the invention is not limited to the precise construction and components disclosed herein and that various modifications, changes and variations which will be apparent to those of ordinary skill in the art may be made in the arrangement, operation and details of the method and apparatus of the present invention disclosed herein without departing from the spirit and scope of the invention.
Stojancic, Mihailo M., Merchant, Shashank, Kulkarni, Sunil Suresh, Gajjar, Pradipkumar Dineshbhai, Pereira, Jose Pio, Ramanathan, Prashant
| Patent | Priority | Assignee | Title |
| Patent | Priority | Assignee | Title |
| 6728881, | Oct 01 1999 | The United States of America as represented by the Secretary of the Army | Fingerprint and signature identification and authorization card and pen |
| 6763148, | Nov 13 2000 | SYTE - VISUAL CONCEPTION LTD | Image recognition methods |
| 6772196, | Jul 27 2000 | Proofpoint, Inc | Electronic mail filtering system and methods |
| 7155676, | Dec 19 2001 | Resource Consortium Limited | System and method for multimedia authoring and playback |
| 7552196, | Apr 15 1999 | Silicon Valley Bank | Detecting corrupted data before transmission to a client |
| 7693813, | Mar 30 2007 | GOOGLE LLC | Index server architecture using tiered and sharded phrase posting lists |
| 20020112180, | |||
| 20040003132, | |||
| 20040049780, | |||
| 20040098671, | |||
| 20040133927, | |||
| 20050021972, | |||
| 20080052739, | |||
| 20090077580, | |||
| 20090164213, | |||
| 20100005488, | |||
| 20100034206, | |||
| 20100177642, | |||
| 20110246471, | |||
| 20120056911, | |||
| 20150163545, |
| Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
| May 06 2011 | PEREIRA, JOSE PIO | Zeitera, LLC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 054158 | /0377 | |
| May 06 2011 | STOJANCIC, MIHAILO | Zeitera, LLC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 054158 | /0377 | |
| May 06 2011 | RAMANATHAN, PRASHANT | Zeitera, LLC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 054158 | /0377 | |
| May 06 2011 | MERCHANT, SHASHANK | Zeitera, LLC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 054158 | /0377 | |
| May 06 2011 | GAJJAR, PRADIPKUMAR DINESHBHAI | Zeitera, LLC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 054158 | /0377 | |
| May 06 2011 | KULKARNI, SUNIL SURESH | Zeitera, LLC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 054158 | /0377 | |
| Jul 01 2015 | Zeitera, LLC | GRACENOTE, INC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 054158 | /0386 | |
| Oct 09 2019 | ROKU, INC. | (assignment on the face of the patent) | / | |||
| Jun 04 2020 | NIELSEN CONSUMER INSIGHTS, INC | CITIBANK, N A | CORRECTIVE ASSIGNMENT TO CORRECT THE PATENTS LISTED ON SCHEDULE 1 RECORDED ON 6-9-2020 PREVIOUSLY RECORDED ON REEL 053473 FRAME 0001 ASSIGNOR S HEREBY CONFIRMS THE SUPPLEMENTAL IP SECURITY AGREEMENT | 054066 | /0064 | |
| Jun 04 2020 | NIELSEN AUDIO, INC | CITIBANK, N A | CORRECTIVE ASSIGNMENT TO CORRECT THE PATENTS LISTED ON SCHEDULE 1 RECORDED ON 6-9-2020 PREVIOUSLY RECORDED ON REEL 053473 FRAME 0001 ASSIGNOR S HEREBY CONFIRMS THE SUPPLEMENTAL IP SECURITY AGREEMENT | 054066 | /0064 | |
| Jun 04 2020 | NETRATINGS, LLC | CITIBANK, N A | CORRECTIVE ASSIGNMENT TO CORRECT THE PATENTS LISTED ON SCHEDULE 1 RECORDED ON 6-9-2020 PREVIOUSLY RECORDED ON REEL 053473 FRAME 0001 ASSIGNOR S HEREBY CONFIRMS THE SUPPLEMENTAL IP SECURITY AGREEMENT | 054066 | /0064 | |
| Jun 04 2020 | GRACENOTE MEDIA SERVICES, LLC | CITIBANK, N A | CORRECTIVE ASSIGNMENT TO CORRECT THE PATENTS LISTED ON SCHEDULE 1 RECORDED ON 6-9-2020 PREVIOUSLY RECORDED ON REEL 053473 FRAME 0001 ASSIGNOR S HEREBY CONFIRMS THE SUPPLEMENTAL IP SECURITY AGREEMENT | 054066 | /0064 | |
| Jun 04 2020 | GRACENOTE DIGITAL VENTURES, LLC | CITIBANK, N A | CORRECTIVE ASSIGNMENT TO CORRECT THE PATENTS LISTED ON SCHEDULE 1 RECORDED ON 6-9-2020 PREVIOUSLY RECORDED ON REEL 053473 FRAME 0001 ASSIGNOR S HEREBY CONFIRMS THE SUPPLEMENTAL IP SECURITY AGREEMENT | 054066 | /0064 | |
| Jun 04 2020 | GRACENOTE, INC | CITIBANK, N A | CORRECTIVE ASSIGNMENT TO CORRECT THE PATENTS LISTED ON SCHEDULE 1 RECORDED ON 6-9-2020 PREVIOUSLY RECORDED ON REEL 053473 FRAME 0001 ASSIGNOR S HEREBY CONFIRMS THE SUPPLEMENTAL IP SECURITY AGREEMENT | 054066 | /0064 | |
| Jun 04 2020 | EXELATE, INC | CITIBANK, N A | CORRECTIVE ASSIGNMENT TO CORRECT THE PATENTS LISTED ON SCHEDULE 1 RECORDED ON 6-9-2020 PREVIOUSLY RECORDED ON REEL 053473 FRAME 0001 ASSIGNOR S HEREBY CONFIRMS THE SUPPLEMENTAL IP SECURITY AGREEMENT | 054066 | /0064 | |
| Jun 04 2020 | CZT ACN TRADEMARKS, L L C | CITIBANK, N A | CORRECTIVE ASSIGNMENT TO CORRECT THE PATENTS LISTED ON SCHEDULE 1 RECORDED ON 6-9-2020 PREVIOUSLY RECORDED ON REEL 053473 FRAME 0001 ASSIGNOR S HEREBY CONFIRMS THE SUPPLEMENTAL IP SECURITY AGREEMENT | 054066 | /0064 | |
| Jun 04 2020 | ATHENIAN LEASING CORPORATION | CITIBANK, N A | CORRECTIVE ASSIGNMENT TO CORRECT THE PATENTS LISTED ON SCHEDULE 1 RECORDED ON 6-9-2020 PREVIOUSLY RECORDED ON REEL 053473 FRAME 0001 ASSIGNOR S HEREBY CONFIRMS THE SUPPLEMENTAL IP SECURITY AGREEMENT | 054066 | /0064 | |
| Jun 04 2020 | ART HOLDING, L L C | CITIBANK, N A | CORRECTIVE ASSIGNMENT TO CORRECT THE PATENTS LISTED ON SCHEDULE 1 RECORDED ON 6-9-2020 PREVIOUSLY RECORDED ON REEL 053473 FRAME 0001 ASSIGNOR S HEREBY CONFIRMS THE SUPPLEMENTAL IP SECURITY AGREEMENT | 054066 | /0064 | |
| Jun 04 2020 | AFFINNOVA, INC | CITIBANK, N A | CORRECTIVE ASSIGNMENT TO CORRECT THE PATENTS LISTED ON SCHEDULE 1 RECORDED ON 6-9-2020 PREVIOUSLY RECORDED ON REEL 053473 FRAME 0001 ASSIGNOR S HEREBY CONFIRMS THE SUPPLEMENTAL IP SECURITY AGREEMENT | 054066 | /0064 | |
| Jun 04 2020 | NIELSEN CONSUMER NEUROSCIENCE, INC | CITIBANK, N A | CORRECTIVE ASSIGNMENT TO CORRECT THE PATENTS LISTED ON SCHEDULE 1 RECORDED ON 6-9-2020 PREVIOUSLY RECORDED ON REEL 053473 FRAME 0001 ASSIGNOR S HEREBY CONFIRMS THE SUPPLEMENTAL IP SECURITY AGREEMENT | 054066 | /0064 | |
| Jun 04 2020 | NIELSEN FINANCE CO | CITIBANK, N A | CORRECTIVE ASSIGNMENT TO CORRECT THE PATENTS LISTED ON SCHEDULE 1 RECORDED ON 6-9-2020 PREVIOUSLY RECORDED ON REEL 053473 FRAME 0001 ASSIGNOR S HEREBY CONFIRMS THE SUPPLEMENTAL IP SECURITY AGREEMENT | 054066 | /0064 | |
| Jun 04 2020 | NIELSEN INTERNATIONAL HOLDINGS, INC | CITIBANK, N A | CORRECTIVE ASSIGNMENT TO CORRECT THE PATENTS LISTED ON SCHEDULE 1 RECORDED ON 6-9-2020 PREVIOUSLY RECORDED ON REEL 053473 FRAME 0001 ASSIGNOR S HEREBY CONFIRMS THE SUPPLEMENTAL IP SECURITY AGREEMENT | 054066 | /0064 | |
| Jun 04 2020 | VNU INTERNATIONAL B V | CITIBANK, N A | CORRECTIVE ASSIGNMENT TO CORRECT THE PATENTS LISTED ON SCHEDULE 1 RECORDED ON 6-9-2020 PREVIOUSLY RECORDED ON REEL 053473 FRAME 0001 ASSIGNOR S HEREBY CONFIRMS THE SUPPLEMENTAL IP SECURITY AGREEMENT | 054066 | /0064 | |
| Jun 04 2020 | THE NIELSEN COMPANY B V | CITIBANK, N A | CORRECTIVE ASSIGNMENT TO CORRECT THE PATENTS LISTED ON SCHEDULE 1 RECORDED ON 6-9-2020 PREVIOUSLY RECORDED ON REEL 053473 FRAME 0001 ASSIGNOR S HEREBY CONFIRMS THE SUPPLEMENTAL IP SECURITY AGREEMENT | 054066 | /0064 | |
| Jun 04 2020 | NIELSEN HOLDING AND FINANCE B V | CITIBANK, N A | CORRECTIVE ASSIGNMENT TO CORRECT THE PATENTS LISTED ON SCHEDULE 1 RECORDED ON 6-9-2020 PREVIOUSLY RECORDED ON REEL 053473 FRAME 0001 ASSIGNOR S HEREBY CONFIRMS THE SUPPLEMENTAL IP SECURITY AGREEMENT | 054066 | /0064 | |
| Jun 04 2020 | NMR LICENSING ASSOCIATES, L P | CITIBANK, N A | CORRECTIVE ASSIGNMENT TO CORRECT THE PATENTS LISTED ON SCHEDULE 1 RECORDED ON 6-9-2020 PREVIOUSLY RECORDED ON REEL 053473 FRAME 0001 ASSIGNOR S HEREBY CONFIRMS THE SUPPLEMENTAL IP SECURITY AGREEMENT | 054066 | /0064 | |
| Jun 04 2020 | VNU MARKETING INFORMATION, INC | CITIBANK, N A | CORRECTIVE ASSIGNMENT TO CORRECT THE PATENTS LISTED ON SCHEDULE 1 RECORDED ON 6-9-2020 PREVIOUSLY RECORDED ON REEL 053473 FRAME 0001 ASSIGNOR S HEREBY CONFIRMS THE SUPPLEMENTAL IP SECURITY AGREEMENT | 054066 | /0064 | |
| Jun 04 2020 | VIZU CORPORATION | CITIBANK, N A | CORRECTIVE ASSIGNMENT TO CORRECT THE PATENTS LISTED ON SCHEDULE 1 RECORDED ON 6-9-2020 PREVIOUSLY RECORDED ON REEL 053473 FRAME 0001 ASSIGNOR S HEREBY CONFIRMS THE SUPPLEMENTAL IP SECURITY AGREEMENT | 054066 | /0064 | |
| Jun 04 2020 | THE NIELSEN COMPANY US , LLC | CITIBANK, N A | CORRECTIVE ASSIGNMENT TO CORRECT THE PATENTS LISTED ON SCHEDULE 1 RECORDED ON 6-9-2020 PREVIOUSLY RECORDED ON REEL 053473 FRAME 0001 ASSIGNOR S HEREBY CONFIRMS THE SUPPLEMENTAL IP SECURITY AGREEMENT | 054066 | /0064 | |
| Jun 04 2020 | TNC US HOLDINGS, INC | CITIBANK, N A | CORRECTIVE ASSIGNMENT TO CORRECT THE PATENTS LISTED ON SCHEDULE 1 RECORDED ON 6-9-2020 PREVIOUSLY RECORDED ON REEL 053473 FRAME 0001 ASSIGNOR S HEREBY CONFIRMS THE SUPPLEMENTAL IP SECURITY AGREEMENT | 054066 | /0064 | |
| Jun 04 2020 | TCG DIVESTITURE INC | CITIBANK, N A | CORRECTIVE ASSIGNMENT TO CORRECT THE PATENTS LISTED ON SCHEDULE 1 RECORDED ON 6-9-2020 PREVIOUSLY RECORDED ON REEL 053473 FRAME 0001 ASSIGNOR S HEREBY CONFIRMS THE SUPPLEMENTAL IP SECURITY AGREEMENT | 054066 | /0064 | |
| Jun 04 2020 | NMR INVESTING I, INC | CITIBANK, N A | CORRECTIVE ASSIGNMENT TO CORRECT THE PATENTS LISTED ON SCHEDULE 1 RECORDED ON 6-9-2020 PREVIOUSLY RECORDED ON REEL 053473 FRAME 0001 ASSIGNOR S HEREBY CONFIRMS THE SUPPLEMENTAL IP SECURITY AGREEMENT | 054066 | /0064 | |
| Jun 04 2020 | NIELSEN MOBILE, LLC | CITIBANK, N A | CORRECTIVE ASSIGNMENT TO CORRECT THE PATENTS LISTED ON SCHEDULE 1 RECORDED ON 6-9-2020 PREVIOUSLY RECORDED ON REEL 053473 FRAME 0001 ASSIGNOR S HEREBY CONFIRMS THE SUPPLEMENTAL IP SECURITY AGREEMENT | 054066 | /0064 | |
| Jun 04 2020 | ACNIELSEN ERATINGS COM | CITIBANK, N A | CORRECTIVE ASSIGNMENT TO CORRECT THE PATENTS LISTED ON SCHEDULE 1 RECORDED ON 6-9-2020 PREVIOUSLY RECORDED ON REEL 053473 FRAME 0001 ASSIGNOR S HEREBY CONFIRMS THE SUPPLEMENTAL IP SECURITY AGREEMENT | 054066 | /0064 | |
| Jun 04 2020 | ACNIELSEN CORPORATION | CITIBANK, N A | CORRECTIVE ASSIGNMENT TO CORRECT THE PATENTS LISTED ON SCHEDULE 1 RECORDED ON 6-9-2020 PREVIOUSLY RECORDED ON REEL 053473 FRAME 0001 ASSIGNOR S HEREBY CONFIRMS THE SUPPLEMENTAL IP SECURITY AGREEMENT | 054066 | /0064 | |
| Jun 04 2020 | ACN HOLDINGS INC | CITIBANK, N A | CORRECTIVE ASSIGNMENT TO CORRECT THE PATENTS LISTED ON SCHEDULE 1 RECORDED ON 6-9-2020 PREVIOUSLY RECORDED ON REEL 053473 FRAME 0001 ASSIGNOR S HEREBY CONFIRMS THE SUPPLEMENTAL IP SECURITY AGREEMENT | 054066 | /0064 | |
| Jun 04 2020 | NIELSEN CONSUMER INSIGHTS, INC | CITIBANK, N A | SUPPLEMENTAL SECURITY AGREEMENT | 053473 | /0001 | |
| Jun 04 2020 | NIELSEN AUDIO, INC | CITIBANK, N A | SUPPLEMENTAL SECURITY AGREEMENT | 053473 | /0001 | |
| Jun 04 2020 | NETRATINGS, LLC | CITIBANK, N A | SUPPLEMENTAL SECURITY AGREEMENT | 053473 | /0001 | |
| Jun 04 2020 | GRACENOTE MEDIA SERVICES, LLC | CITIBANK, N A | SUPPLEMENTAL SECURITY AGREEMENT | 053473 | /0001 | |
| Jun 04 2020 | GRACENOTE DIGITAL VENTURES, LLC | CITIBANK, N A | SUPPLEMENTAL SECURITY AGREEMENT | 053473 | /0001 | |
| Jun 04 2020 | GRACENOTE, INC | CITIBANK, N A | SUPPLEMENTAL SECURITY AGREEMENT | 053473 | /0001 | |
| Jun 04 2020 | EXELATE, INC | CITIBANK, N A | SUPPLEMENTAL SECURITY AGREEMENT | 053473 | /0001 | |
| Jun 04 2020 | CZT ACN TRADEMARKS, L L C | CITIBANK, N A | SUPPLEMENTAL SECURITY AGREEMENT | 053473 | /0001 | |
| Jun 04 2020 | A C NIELSEN COMPANY, LLC | CITIBANK, N A | SUPPLEMENTAL SECURITY AGREEMENT | 053473 | /0001 | |
| Jun 04 2020 | ACN HOLDINGS INC | CITIBANK, N A | SUPPLEMENTAL SECURITY AGREEMENT | 053473 | /0001 | |
| Jun 04 2020 | ACNIELSEN CORPORATION | CITIBANK, N A | SUPPLEMENTAL SECURITY AGREEMENT | 053473 | /0001 | |
| Jun 04 2020 | ACNIELSEN ERATINGS COM | CITIBANK, N A | SUPPLEMENTAL SECURITY AGREEMENT | 053473 | /0001 | |
| Jun 04 2020 | AFFINNOVA, INC | CITIBANK, N A | SUPPLEMENTAL SECURITY AGREEMENT | 053473 | /0001 | |
| Jun 04 2020 | ATHENIAN LEASING CORPORATION | CITIBANK, N A | SUPPLEMENTAL SECURITY AGREEMENT | 053473 | /0001 | |
| Jun 04 2020 | NIELSEN CONSUMER NEUROSCIENCE, INC | CITIBANK, N A | SUPPLEMENTAL SECURITY AGREEMENT | 053473 | /0001 | |
| Jun 04 2020 | NIELSEN FINANCE CO | CITIBANK, N A | SUPPLEMENTAL SECURITY AGREEMENT | 053473 | /0001 | |
| Jun 04 2020 | NIELSEN INTERNATIONAL HOLDINGS, INC | CITIBANK, N A | SUPPLEMENTAL SECURITY AGREEMENT | 053473 | /0001 | |
| Jun 04 2020 | A C NIELSEN COMPANY, LLC | CITIBANK, N A | CORRECTIVE ASSIGNMENT TO CORRECT THE PATENTS LISTED ON SCHEDULE 1 RECORDED ON 6-9-2020 PREVIOUSLY RECORDED ON REEL 053473 FRAME 0001 ASSIGNOR S HEREBY CONFIRMS THE SUPPLEMENTAL IP SECURITY AGREEMENT | 054066 | /0064 | |
| Jun 04 2020 | A C NIELSEN ARGENTINA S A | CITIBANK, N A | CORRECTIVE ASSIGNMENT TO CORRECT THE PATENTS LISTED ON SCHEDULE 1 RECORDED ON 6-9-2020 PREVIOUSLY RECORDED ON REEL 053473 FRAME 0001 ASSIGNOR S HEREBY CONFIRMS THE SUPPLEMENTAL IP SECURITY AGREEMENT | 054066 | /0064 | |
| Jun 04 2020 | VNU INTERNATIONAL B V | CITIBANK, N A | SUPPLEMENTAL SECURITY AGREEMENT | 053473 | /0001 | |
| Jun 04 2020 | THE NIELSEN COMPANY B V | CITIBANK, N A | SUPPLEMENTAL SECURITY AGREEMENT | 053473 | /0001 | |
| Jun 04 2020 | NIELSEN HOLDING AND FINANCE B V | CITIBANK, N A | SUPPLEMENTAL SECURITY AGREEMENT | 053473 | /0001 | |
| Jun 04 2020 | NMR LICENSING ASSOCIATES, L P | CITIBANK, N A | SUPPLEMENTAL SECURITY AGREEMENT | 053473 | /0001 | |
| Jun 04 2020 | VNU MARKETING INFORMATION, INC | CITIBANK, N A | SUPPLEMENTAL SECURITY AGREEMENT | 053473 | /0001 | |
| Jun 04 2020 | VIZU CORPORATION | CITIBANK, N A | SUPPLEMENTAL SECURITY AGREEMENT | 053473 | /0001 | |
| Jun 04 2020 | NIELSEN MOBILE, LLC | CITIBANK, N A | SUPPLEMENTAL SECURITY AGREEMENT | 053473 | /0001 | |
| Jun 04 2020 | NIELSEN UK FINANCE I, LLC | CITIBANK, N A | SUPPLEMENTAL SECURITY AGREEMENT | 053473 | /0001 | |
| Jun 04 2020 | NMR INVESTING I, INC | CITIBANK, N A | SUPPLEMENTAL SECURITY AGREEMENT | 053473 | /0001 | |
| Jun 04 2020 | TCG DIVESTITURE INC | CITIBANK, N A | SUPPLEMENTAL SECURITY AGREEMENT | 053473 | /0001 | |
| Jun 04 2020 | TNC US HOLDINGS, INC | CITIBANK, N A | SUPPLEMENTAL SECURITY AGREEMENT | 053473 | /0001 | |
| Jun 04 2020 | THE NIELSEN COMPANY US , LLC | CITIBANK, N A | SUPPLEMENTAL SECURITY AGREEMENT | 053473 | /0001 | |
| Jun 04 2020 | ART HOLDING, L L C | CITIBANK, N A | SUPPLEMENTAL SECURITY AGREEMENT | 053473 | /0001 | |
| Apr 15 2021 | CITIBANK, N A | GRACENOTE, INC | PARTIAL RELEASE OF SECURITY INTEREST | 056973 | /0280 | |
| Apr 15 2021 | CITIBANK, N A | THE NIELSEN COMPANY US , LLC | PARTIAL RELEASE OF SECURITY INTEREST | 056973 | /0280 | |
| Apr 15 2021 | GRACENOTE, INC | ROKU, INC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 056103 | /0786 | |
| Jun 22 2021 | ROKU, INC | MORGAN STANLEY SENIOR FUNDING, INC | PATENT SECURITY AGREEMENT SUPPLEMENT | 056982 | /0194 | |
| Oct 11 2022 | CITIBANK, N A | A C NIELSEN COMPANY, LLC | RELEASE REEL 054066 FRAME 0064 | 063605 | /0001 | |
| Oct 11 2022 | CITIBANK, N A | A C NIELSEN COMPANY, LLC | RELEASE REEL 053473 FRAME 0001 | 063603 | /0001 | |
| Oct 11 2022 | CITIBANK, N A | EXELATE, INC | RELEASE REEL 053473 FRAME 0001 | 063603 | /0001 | |
| Oct 11 2022 | CITIBANK, N A | GRACENOTE, INC | RELEASE REEL 053473 FRAME 0001 | 063603 | /0001 | |
| Oct 11 2022 | CITIBANK, N A | GRACENOTE MEDIA SERVICES, LLC | RELEASE REEL 053473 FRAME 0001 | 063603 | /0001 | |
| Oct 11 2022 | CITIBANK, N A | GRACENOTE MEDIA SERVICES, LLC | RELEASE REEL 054066 FRAME 0064 | 063605 | /0001 | |
| Oct 11 2022 | CITIBANK, N A | THE NIELSEN COMPANY US , LLC | RELEASE REEL 053473 FRAME 0001 | 063603 | /0001 | |
| Oct 11 2022 | CITIBANK, N A | NETRATINGS, LLC | RELEASE REEL 054066 FRAME 0064 | 063605 | /0001 | |
| Oct 11 2022 | CITIBANK, N A | THE NIELSEN COMPANY US , LLC | RELEASE REEL 054066 FRAME 0064 | 063605 | /0001 | |
| Oct 11 2022 | CITIBANK, N A | NETRATINGS, LLC | RELEASE REEL 053473 FRAME 0001 | 063603 | /0001 | |
| Oct 11 2022 | CITIBANK, N A | GRACENOTE, INC | RELEASE REEL 054066 FRAME 0064 | 063605 | /0001 | |
| Oct 11 2022 | CITIBANK, N A | EXELATE, INC | RELEASE REEL 054066 FRAME 0064 | 063605 | /0001 | |
| Feb 21 2023 | MORGAN STANLEY SENIOR FUNDING, INC | ROKU DX HOLDINGS, INC | TERMINATION AND RELEASE OF INTELLECTUAL PROPERTY SECURITY AGREEMENT REEL FRAME 056982 0194 | 062826 | /0664 | |
| Feb 21 2023 | MORGAN STANLEY SENIOR FUNDING, INC | ROKU, INC | TERMINATION AND RELEASE OF INTELLECTUAL PROPERTY SECURITY AGREEMENT REEL FRAME 056982 0194 | 062826 | /0664 | |
| Sep 16 2024 | ROKU, INC | CITIBANK, N A | SECURITY INTEREST SEE DOCUMENT FOR DETAILS | 068982 | /0377 |
| Date | Maintenance Fee Events |
| Oct 09 2019 | BIG: Entity status set to Undiscounted (note the period is included in the code). |
| Date | Maintenance Schedule |
| Oct 26 2024 | 4 years fee payment window open |
| Apr 26 2025 | 6 months grace period start (w surcharge) |
| Oct 26 2025 | patent expiry (for year 4) |
| Oct 26 2027 | 2 years to revive unintentionally abandoned end. (for year 4) |
| Oct 26 2028 | 8 years fee payment window open |
| Apr 26 2029 | 6 months grace period start (w surcharge) |
| Oct 26 2029 | patent expiry (for year 8) |
| Oct 26 2031 | 2 years to revive unintentionally abandoned end. (for year 8) |
| Oct 26 2032 | 12 years fee payment window open |
| Apr 26 2033 | 6 months grace period start (w surcharge) |
| Oct 26 2033 | patent expiry (for year 12) |
| Oct 26 2035 | 2 years to revive unintentionally abandoned end. (for year 12) |