A method and apparatus which reduces the start-up delay that may occur when switching programs in audio and/or video streaming applications while maintaining high quality steady-state performance thereof. A program source (e.g., an audio and/or video data stream) is encoded and transmitted as two or more separate bit streams (e.g., sequences of data packets), the transmission of one of these bit streams being delayed by a given amount of time relative to the transmission of the other bit stream(s). At the receiving end of the transmission channel, the two or more bit streams are buffered by receive buffers having different sizes (thereby resulting in different time delays when the contents thereof are decoded), wherein the time delay difference corresponds (inversely) to the relative delay times prior to transmission. Advantageously, either a multiple descriptive source coding scheme or an embedded coding scheme may be employed, in which at least one of the individual bit streams is sufficient to obtain a satisfactory decoded signal, but wherein the addition of the other bit stream(s) will improve the quality of the decoded signal. Alternatively, the data streams may comprise multiple encodings of the program source having different bit rates, wherein the lower bit rate encodings are transmitted with the correspondingly larger delays.
|
1. A method of processing two or more sequences of data packets received from a network, each of said sequences of data packets representing a common program source, the method comprising the steps of:
buffering data packets comprised in a first one of said sequences in a first receive buffer having a first buffering delay;
buffering data packets comprised in a second one of said sequences in a second receive buffer having a second buffering delay, wherein said second buffering delay is smaller than said first buffering delay;
decoding the data packets buffered in said second receive buffer after said second buffering delay has elapsed;
decoding the data packets buffered in said first receive buffer after said first buffering delay has elapsed;
generating a reproduction of said program source based at least upon the decoding of the data packets buffered in said second receive buffer before said first buffering delay has elapsed, and based at least upon the decoding of the data packets buffered in said first receive buffer after said first buffering delay has elapsed.
11. An apparatus for processing two or more sequences of data packets received from a network, each of said sequences of data packets representing a common program source, the apparatus comprising:
a first receive buffer having a first buffer delay for buffering data packets comprised in a first one of said sequences;
a second receive buffer having a second buffering delay for buffering data packets comprised in a second one of said sequences, wherein said second buffering delay is smaller than said first buffering delay;
a decoder for decoding the data packets buffered in said second receive buffer after said second buffering delay has elapsed and for decoding the data packets buffered in said first receive buffer after said first buffering delay has elapsed;
a signal generator for generating a reproduction of said program source based at least upon the decoding of the data packets buffered in said second receive buffer before said first buffering delay has elapsed, and based at least upon the decoding of the data packets buffered in said first receive buffer after said first buffering delay has elapsed.
5. The method of
6. The method of
7. The method of
8. The method of
9. The method of
10. The method of
15. The apparatus of
16. The apparatus of
17. The apparatus of
18. The apparatus of
19. The apparatus of
20. The apparatus of
|
The present invention relates generally to audio and video streaming applications employed in data packet-based networks such as, for example, the Internet, and more particularly to the buffering of received data packets which is typically performed in such applications.
Internet applications that employ audio and video streaming are becoming increasingly prevalent. (When used herein, the term “audio” will be intended to include speech as one example of an audio signal.) As a natural consequence of transmitting and receiving data over a data packet-based network such as the Internet, when network traffic is large the network gives rise to relatively large packet delays. In particular, packet delays usually vary considerably depending on the momentary level of network congestion. Moreover, data packets are sometimes even lost completely by the network. Since applications which employ audio and video streaming are typically used in non-interactive environments, however, the end-to-end delay is usually not critical.
For these reasons, and as is totally familiar to those of ordinary skill in the art, data packets from such streaming applications are usually buffered at the receiving end over a time period which may typically be several seconds in duration. This buffering helps to reduce the detrimental effects of the relatively large and variable packet delays which result from the varying levels of network congestion. Packet losses in the network are typically addressed by using a forward error correction code across the packets, as is also fully familiar to those skilled in the art. The error correction capability of such an error correcting code typically improves with the size of the data packets.
Clearly then, a large receive buffer is highly desirable to provide a better quality signal, because it increases the probability that most of the transmitted packets representing data within the given (i.e. the buffered) period of time will have been successfully accumulated in the buffer before it is necessary to decode them for “playback.” However, since the receive buffer usually needs to be initially filled before the signal can be decoded, a large buffer necessarily gives rise to a correspondingly large buffering delay, and, in particular, a large start-up delay. Start-up delays of a few seconds can be quite annoying, especially when a channel switch is made in an Internet broadcast environment. Such an environment typically involves an Internet backbone which broadcasts many independent programs, and a number of users which receive their individually selected program via a server connected to the backbone. A large start-up delay could thus be quite bothersome when a user changes the selected broadcast program. It would be highly desirable, therefore, to provide a source coding and receive data buffering scheme which results in more acceptable start-up delays without sacrificing the benefits of using a large receive buffer. In this manner, relatively painless channel switches may be effectuated while still maintaining high quality steady-state performance.
In accordance with the principles of the present invention, a novel technique is disclosed whereby the start-up delay that may occur when initiating or switching received programs in audio (including speech) or video streaming applications is advantageously reduced while maintaining high quality steady-state performance thereof. Specifically, the instant invention comprises a method and apparatus for processing two or more sequences of data packets received from a network, each of the sequences of data packets representing a common program source, in which a reproduction of the program source is generated by (a) buffering the data packets comprised in a first one of said sequences in a first receive buffer having a first buffering delay; (b) buffering the data packets comprised in a second one of said sequences in a second receive buffer having a second buffering delay, wherein the second buffering delay is smaller than the first buffering delay; (c) decoding the data packets buffered in the second receive buffer after the second buffering delay has elapsed; (d) decoding the data packets buffered in the first receive buffer after the first buffering delay has elapsed; and (e) generating the reproduction of the program source based at least upon the decoding of the data packets buffered in the second receive buffer before the first buffering delay has elapsed, and based at least upon the decoding of the data packets buffered in the first receive buffer after the first buffering delay has elapsed.
In accordance with an illustrative embodiment of the present invention, a program source (e.g., an audio and/or video data stream) is encoded and transmitted as two or more separate bit streams (e.g., sequences of data packets), wherein the transmission of one of these bit streams has been delayed prior to transmission by a given amount of time relative to the transmission of the other bit stream(s). At the receiving end, these two or more bit streams are buffered by receive buffers having different buffering delays (e.g., as a result of having different buffer sizes), wherein the time delay difference corresponds to the relative delay times effectuated between the bit streams prior to transmission.
In accordance with one illustrative embodiment of the present invention, it may be advantageous to employ a multiple descriptive source coding scheme, familiar to those skilled in the art. As is well known, in such a scheme, each of two or more individual bit streams are coded, each bit stream being sufficient by itself so that when it is decoded, a reproduction of the original program source having a satisfactory signal quality is obtained. However, when a combination of two or more of these individual bit streams is decoded, a reproduced signal of improved quality will result. (See, e.g., Michael T. Orchard et al., “Redundancy Rate-Distortion Analysis of Multiple Description Coding Using Pairwise Correlating Transforms,” Proc. IEEE International Conference on Image Processing, October, 1997.) In this case, it is useful to transmit one of the individual bit streams as the “second” sequence of data packets, and either another one of the individual bit streams, or preferably, a combination of two or more of the individual bit streams as the “first” sequence of data packets. (Note that it will be obvious to those skilled in the art that for the efficient use of channel bandwidth, in any embodiment of the present invention for which it is desirable that the same data be included in multiple bit streams, that data can be advantageously transmitted as a part of only one of these bit streams, and then re-combined with the other(s) at the receive end of the channel, either before or after the corresponding decoding of the encoded bit stream data is performed.)
In accordance with another illustrative embodiment of the invention, it may be advantageous to employ an embedded coder, also familiar to those skilled in the art, in which a core layer and one or more enhancement layers are separately coded. (See, e.g., Jurgen Herre et al., “The Integrated Filterbank Based Scalable MPEG-4 Audio Coder,” 105th Audio Engineering Society Convention, San Francisco, September, 1998.) In this case, it is useful to transmit the core layer bit stream as the “second” sequence of data packets, and a combination of the core layer bit stream and the enhancement layer(s) bit stream(s) as the “first” sequence of data packets. Again, the channel bandwidth efficiency “technique” described above can be advantageously employed. And also, in accordance with still another illustrative embodiment of the invention, the data streams may advantageously comprise multiple encodings of the program source each having a different bit rate, wherein lower bit rate encodings are transmitted with correspondingly larger delays (and thus buffered at the receiver with receive buffers having correspondingly shorter buffer delays).
Advantageously, the encodings performed by these two source coders are such that a reproduction of the original program source signal should be obtainable by decoding a2(n) only, and, preferably, such decoding should produce a signal of a reasonably acceptable quality. Moreover, a decoding of either a1(n) individually, or, alternatively, a combination of a decoding of each of a1(n) and a2(n), advantageously results in a reproduction of the original program source signal having a superior quality to that which is obtainable by decoding a2(n) only.
Specifically, in accordance with certain illustrative embodiments of the present invention, source coders 11 and 12 may advantageously implement one of a number of various coding schemes, each of which is familiar to those of ordinary skill in the art, designed to provide these advantageous characteristics. For example, it may be desirable to employ a multiple descriptive source coding scheme, in which each of two (or more) individual bit streams are coded and are sufficient by themselves to obtain a decoded signal having a satisfactory quality, but wherein the decoding of a combination of the two (or more) of these individual bit streams will result in a signal having improved quality as compared thereto. In this case, one of theses individual bit streams may be generated by source coder 12 and transmitted as signal a2(n), while the other one of these individual bit streams (or one of the other bit streams if there are more than two such individual bit streams) is generated by source coder 11 and transmitted as signal a1(n). At the receiving end of the transmission channel (see the discussion below), a combination of these two (or more) individual bit streams (the combination being effectuated either before or after the decoding process) can be used to produce the higher quality signal.
In accordance with another illustrative embodiment of the invention, it may be advantageous to employ an embedded coder, also familiar to those skilled in the art, in which a core layer and one (or more) enhancement layers are separately coded. In this case, the core layer is coded by source coder 12 and transmitted as signal a2(n), while the enhancement layer (or one of the enhancement layers if there is more than one such layer) is coded by source coder 11 and transmitted as signal a1(n). As above, at the receiving end of the transmission channel (see below), a combination of these two (or more) individual bit streams (combined either before or after decoding) can be used to produce the higher quality signal.
And also, in accordance with still another illustrative embodiment of the invention, the data streams may advantageously comprise multiple encodings of the program source each having a different bit rate. In this case, source coder 11 produces the encoding having the larger of the bit rates and transmits the resultant data stream as signal a1(n), whereas source coder 12 produces the encoding having the smaller of the bit rates and transmits the resultant data stream as signal a2(n).
In any case, and in accordance with the illustrative environment of
In accordance with the illustrative embodiment of
Receive buffer 17 and receive buffer 18 are used to provide the input to decoder 21 and decoder 22, respectively. These two decoders correspond to source coders 11 and 12, and are used to generate corresponding reproductions of the original source program from the two data streams a1(n) and a2(n), denoted in the figure as b1(n) and b2(n), respectively. In particular, decoders 21 and 22 implement conventional decoding of packet-based bit streams, and, in particular, use decoding algorithms which correspond to the encoding algorithms that were used by source coders 11 and 12. (That is, the decoding algorithm implemented by decoder 21 comprises a conventional decoding method for decoding data streams produced by the conventional encoding algorithm implemented by coder 11, and the decoding algorithm implemented by decoder 22 comprises a conventional decoding method for decoding data streams produced by the conventional encoding algorithm implemented by coder 12.) In order to provide accurate and robust decoding of the incoming bit streams, decoders 21 and 22 advantageously delay for a time period equal to n1 and n2, respectively, both after an initial receiver start-up (e.g., when power is applied) and after a receiver “channel” change is effectuated (i.e., the selection of a different broadcast source program), before the corresponding decoding process begins. (As described above, n1 and n2 are the buffer delays associated with buffers 17 and 18, respectively.)
Finally, in accordance with the principles of the present invention as embodied in the illustrative environment of
As discussed above, in accordance with certain embodiments of the present invention in which a coding scheme is used in which a higher quality signal is advantageously obtained based upon a combination of the two data streams a1(n) and a2(n), decoder 21 may use the outputs of both receive buffer 17 and receive buffer 18 to produce its reproduction of the original source program after delay n1 has elapsed. Alternatively, embodiments employing such coding schemes may allow for the individual decoding of the two data streams a1(n) and a2(n) by decoder 21 and decoder 22, respectively, and then combine (e.g., within selection processor 24) these decodings in a conventional manner to produce the higher quality reproduction. Note that in this latter case, selection processor 24 advantageously outputs signal b2(n) after the delay n2 has elapsed but before the delay n1 has elapsed (recall that n1>n2), and then outputs an appropriate combination of signal b1(n) and signal b2(n) (i.e., after the delay n1 has elapsed) as its final reproduction of the original source program, B(n).
Specifically, the illustrative environment of
Specifically, signal a2(n), as generated by source coder 12, and signal a3(n), as generated by source coder 13, are each delayed prior to transmission (relative to the transmission of signal a1(n) as generated by source coder 11 as well as relative to each other) by conventional delay elements 14 and 15, respectively. The amount of delay which is applied by delay element 14, nd1, is advantageously approximately equal to the difference in the delays incurred by the receive buffers used at the receiver for the decoding of the two data streams, a1(n) and a2(n), while the amount of delay which is applied by delay element 15, nd2, is advantageously approximately equal to the difference in the delays incurred by the receive buffers used at the receiver for the decoding of the two data streams, a1(n) and a3(n). (See the discussion below.) Thus, delay element 14 produces signal a2(n-nd1), namely, signal a2(n) delayed by an amount of time nd1, while delay element 15 produces signal a3(n-nd2), namely, signal a3(n) delayed by an amount of time nd2. Note that advantageously, nd1<nd2.
In accordance with the illustrative embodiment of
Receive buffers 17, 18, and 19 are used to provide the input to decoders 21, 22, and 23, respectively. These three decoders correspond to source coders 11, 12, and 13, and are used to generate corresponding reproductions of the original source program from the three data streams a1(n), a2(n), and a3(n), denoted in the figure as b1(n), b2(n), and b3(n), respectively. In particular, decoders 21, 22, and 23 implement conventional decoding of packet-based bit streams, and, in particular, use decoding algorithms which correspond to the encoding algorithms that were used by source coders 11, 12, and 13.
Finally, in accordance with the principles of the present invention as embodied in the illustrative environment of
The preceding merely illustrates the principles of the invention. It will thus be appreciated that those skilled in the art will be able to devise various arrangements which, although not explicitly described or shown herein, embody the principles of the invention and are included within its spirit and scope. Furthermore, all examples and conditional language recited herein are principally intended expressly to be only for pedagogical purposes to aid the reader in understanding the principles of the invention and the concepts contributed by the inventors to furthering the art, and are to be construed as being without limitation to such specifically recited examples and conditions. Moreover, all statements herein reciting principles, aspects, and embodiments of the invention, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. Additionally, it is intended that such equivalents include both currently known equivalents as well as equivalents developed in the future, i.e. any elements developed that perform the same function, regardless of structure.
Thus, for example, it will be appreciated by those skilled in the art that the block diagrams herein represent conceptual views of illustrative circuitry embodying the principles of the invention. Similarly, it will be appreciated that any flow charts, flow diagrams, state transition diagrams, pseudocode, and the like represent various processes which may be substantially represented in a computer readable medium and so executed by a computer or processor, whether or not such computer or processor is explicitly shown.
The functions of the various elements shown in the figures, including functional blocks labeled as “processors” may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. When provided by a processor, the functions may be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which may be shared. Moreover, explicit use of the term “processor” or “controller” should not be construed to refer exclusively to hardware capable of executing software, and may implicitly include, without limitation, digital signal processor (DSP) hardware, read-only memory (ROM) for storing software, random access memory (RAM), and non-volatile storage. Other hardware, conventional and/or custom, may also be included. Similarly, any switches shown in the figures are conceptual only. Their function may be carried out through the operation of program logic, through dedicated logic, through the interaction of program control and dedicated logic, or even manually, the particular technique being selectable by the implementor as more specifically understood from the context.
In the claims hereof any element expressed as a means for performing a specified function is intended to encompass any way of performing that function including, for example, a) a combination of circuit elements which performs that function, or b) software in any form, including, therefore, firmware, microcode or the like, combined with appropriate circuitry for executing that software to perform the function. The invention as defined by such claims resides in the fact that the functionalities provided by the various recited means are combined and brought together in the manner which thc claims call for. Applicant thus regards any means which can provide those functionalities as equivalent as those shown herein.
Lou, Hui-Ling, Schuller, Gerald Dietrich Thomas, Weerackody, Vijitha
Patent | Priority | Assignee | Title |
10356450, | Sep 12 2013 | Sony Corporation | Encoding device, transmission device, and reception device |
10499112, | Dec 17 2012 | INTERDIGITAL CE PATENT HOLDINGS | Robust digital channels |
10715877, | Jun 27 2008 | Microsoft Technology Licensing, LLC | System for adaptive video switching for variable network conditions |
7286601, | Aug 27 2002 | MATSUSHITA ELECTRIC INDUSTRIAL CO , LTD | Digital broadcast system having transmission apparatus and receiving apparatus |
7430222, | Feb 27 2004 | Rovi Technologies Corporation | Media stream splicer |
7443791, | Oct 10 2003 | Rovi Technologies Corporation | Priority mechanism for distributed sending of media data |
7444419, | Oct 10 2003 | Microsoft Technology Licensing, LLC | Media stream scheduling for hiccup-free fast-channel-change in the presence of network chokepoints |
7464397, | Feb 05 2003 | Canon Kabushiki Kaisha | Streaming content receiving apparatus and playback apparatus with stopping of reception of second streaming data during period in which first streaming program is selected |
7477653, | Dec 10 2004 | Microsoft Technology Licensing, LLC | Accelerated channel change in rate-limited environments |
7516232, | Oct 10 2003 | Rovi Product Corporation | Media organization for distributed sending of media data |
7523482, | Aug 13 2002 | Microsoft Technology Licensing, LLC | Seamless digital channel changing |
7545812, | Oct 10 2003 | Rovi Product Corporation | Scheduling scheme for distributed sending of media data |
7562375, | Oct 10 2003 | Microsoft Technology Licensing, LLC | Fast channel change |
7587737, | Jun 13 2003 | Microsoft Technology Licensing, LLC | Fast start-up for digital video streams |
7603689, | Jun 13 2003 | Microsoft Technology Licensing, LLC | Fast start-up for digital video streams |
7614071, | Oct 10 2003 | Rovi Technologies Corporation | Architecture for distributed sending of media data |
7636934, | Jun 13 2003 | Microsoft Technology Licensing, LLC | Fast start-up for digital video streams |
7640352, | Sep 24 2004 | Microsoft Technology Licensing, LLC | Methods and systems for presentation of media obtained from a media stream |
7944863, | Dec 10 2004 | Microsoft Technology Licensing, LLC | Accelerated channel change in rate-limited environments |
8037200, | Oct 10 2003 | Rovi Product Corporation | Media organization for distributed sending of media data |
8135040, | Nov 30 2005 | Microsoft Technology Licensing, LLC | Accelerated channel change |
8156534, | Aug 13 2002 | Microsoft Technology Licensing, LLC | Seamless digital channel changing |
8352996, | Jun 27 2008 | Microsoft Technology Licensing, LLC | Adaptive video switching for variable network conditions |
8397269, | Aug 13 2002 | Microsoft Technology Licensing, LLC | Fast digital channel changing |
8514891, | Feb 27 2004 | Rovi Technologies Corporation | Media stream splicer |
8606951, | Oct 10 2003 | Microsoft Technology Licensing, LLC | Media stream scheduling for hiccup-free fast-channel-change in the presence of network chokepoints |
8724714, | Jan 22 2007 | Fraunhofer-Gesellschaft zur Foerderung der Angewandten Forschung E V | Device and method for generating and decoding a side channel signal transmitted with a main channel signal |
9363302, | Oct 10 2003 | Microsoft Technology Licensing, LLC | Media stream scheduling for hiccup-free fast-channel-change in the presence of network chokepoints |
9420347, | Jun 27 2008 | Microsoft Technology Licensing, LLC | Adaptive video switching for variable network conditions |
9813477, | Jan 26 2015 | T-Mobile USA, Inc. | Adjusting quality level of media streaming |
9936226, | Dec 17 2012 | InterDigital Madison Patent Holdings, SAS | Robust digital channels |
Patent | Priority | Assignee | Title |
4920534, | Feb 28 1986 | AT&T Bell Laboratories | System for controllably eliminating bits from packet information field based on indicator in header and amount of data in packet buffer |
6108626, | Oct 27 1995 | Nuance Communications, Inc | Object oriented audio coding |
6345125, | Feb 25 1998 | WSOU Investments, LLC | Multiple description transform coding using optimal transforms of arbitrary dimension |
6349152, | Mar 28 1996 | Microsoft Technology Licensing, LLC | Table-based compression with embedded coding |
6351474, | Jan 14 1998 | SKYSTREAM NETWORKS INC | Network distributed remultiplexer for video program bearing transport streams |
6351730, | Mar 30 1998 | Alcatel-Lucent USA Inc | Low-complexity, low-delay, scalable and embedded speech and audio coding with adaptive frame loss concealment |
6356665, | Dec 09 1998 | Sharp Kabushiki Kaisha | Quad-tree embedded image compression and decompression method and apparatus |
6366704, | Dec 01 1997 | Sharp Laboratories of America, Inc. | Method and apparatus for a delay-adaptive rate control scheme for the frame layer |
EP695094, |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Apr 08 1999 | Lucent Technologies Inc. | (assignment on the face of the patent) | / | |||
Apr 08 1999 | WEERACKODY, VIJITHA | Lucent Technologies Inc | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 009888 | /0001 | |
Apr 08 1999 | SCHULLER, GERALD DIETRICH THOMAS | Lucent Technologies Inc | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 009888 | /0001 | |
Apr 08 1999 | LOU, HUI-LING | Lucent Technologies Inc | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 009888 | /0001 | |
Nov 01 2008 | Lucent Technologies Inc | Alcatel-Lucent USA Inc | CHANGE OF NAME SEE DOCUMENT FOR DETAILS | 049887 | /0613 | |
Jan 30 2013 | Alcatel-Lucent USA Inc | CREDIT SUISSE AG | SECURITY INTEREST SEE DOCUMENT FOR DETAILS | 030510 | /0627 | |
Aug 19 2014 | CREDIT SUISSE AG | Alcatel-Lucent USA Inc | RELEASE BY SECURED PARTY SEE DOCUMENT FOR DETAILS | 033950 | /0261 | |
Sep 12 2017 | NOKIA SOLUTIONS AND NETWORKS BV | Provenance Asset Group LLC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 043877 | /0001 | |
Sep 12 2017 | ALCATEL LUCENT SAS | Provenance Asset Group LLC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 043877 | /0001 | |
Sep 12 2017 | Nokia Technologies Oy | Provenance Asset Group LLC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 043877 | /0001 | |
Sep 13 2017 | PROVENANCE ASSET GROUP HOLDINGS, LLC | NOKIA USA INC | SECURITY INTEREST SEE DOCUMENT FOR DETAILS | 043879 | /0001 | |
Sep 13 2017 | PROVENANCE ASSET GROUP, LLC | CORTLAND CAPITAL MARKET SERVICES, LLC | SECURITY INTEREST SEE DOCUMENT FOR DETAILS | 043967 | /0001 | |
Sep 13 2017 | PROVENANCE ASSET GROUP HOLDINGS, LLC | CORTLAND CAPITAL MARKET SERVICES, LLC | SECURITY INTEREST SEE DOCUMENT FOR DETAILS | 043967 | /0001 | |
Sep 13 2017 | Provenance Asset Group LLC | NOKIA USA INC | SECURITY INTEREST SEE DOCUMENT FOR DETAILS | 043879 | /0001 | |
Dec 20 2018 | NOKIA USA INC | NOKIA US HOLDINGS INC | ASSIGNMENT AND ASSUMPTION AGREEMENT | 048370 | /0682 | |
Nov 01 2021 | CORTLAND CAPITAL MARKETS SERVICES LLC | PROVENANCE ASSET GROUP HOLDINGS LLC | RELEASE BY SECURED PARTY SEE DOCUMENT FOR DETAILS | 058983 | /0104 | |
Nov 01 2021 | CORTLAND CAPITAL MARKETS SERVICES LLC | Provenance Asset Group LLC | RELEASE BY SECURED PARTY SEE DOCUMENT FOR DETAILS | 058983 | /0104 | |
Nov 29 2021 | NOKIA US HOLDINGS INC | PROVENANCE ASSET GROUP HOLDINGS LLC | RELEASE BY SECURED PARTY SEE DOCUMENT FOR DETAILS | 058363 | /0723 | |
Nov 29 2021 | Provenance Asset Group LLC | RPX Corporation | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 059352 | /0001 | |
Nov 29 2021 | NOKIA US HOLDINGS INC | Provenance Asset Group LLC | RELEASE BY SECURED PARTY SEE DOCUMENT FOR DETAILS | 058363 | /0723 |
Date | Maintenance Fee Events |
Dec 10 2004 | ASPN: Payor Number Assigned. |
Jun 15 2007 | RMPN: Payer Number De-assigned. |
Jul 08 2008 | M1551: Payment of Maintenance Fee, 4th Year, Large Entity. |
Jul 06 2012 | M1552: Payment of Maintenance Fee, 8th Year, Large Entity. |
Aug 19 2016 | REM: Maintenance Fee Reminder Mailed. |
Dec 14 2016 | M1553: Payment of Maintenance Fee, 12th Year, Large Entity. |
Dec 14 2016 | M1556: 11.5 yr surcharge- late pmt w/in 6 mo, Large Entity. |
Date | Maintenance Schedule |
Jan 11 2008 | 4 years fee payment window open |
Jul 11 2008 | 6 months grace period start (w surcharge) |
Jan 11 2009 | patent expiry (for year 4) |
Jan 11 2011 | 2 years to revive unintentionally abandoned end. (for year 4) |
Jan 11 2012 | 8 years fee payment window open |
Jul 11 2012 | 6 months grace period start (w surcharge) |
Jan 11 2013 | patent expiry (for year 8) |
Jan 11 2015 | 2 years to revive unintentionally abandoned end. (for year 8) |
Jan 11 2016 | 12 years fee payment window open |
Jul 11 2016 | 6 months grace period start (w surcharge) |
Jan 11 2017 | patent expiry (for year 12) |
Jan 11 2019 | 2 years to revive unintentionally abandoned end. (for year 12) |