A method and apparatus for achieving low-latency, and rapidly attained high-resolution, access reception for transmitted and received video data involving the pre-transmission dividing of a source video stream into two downstream-deliverable data streams that differ by, on the one hand, low-latency, low-resolution characteristics for one stream, and on the other hand, higher-latency, higher-resolution characteristics for the other stream. latency in these streams is determined by the frame spacing created between stream-inserted marker I-frames. The divided streams are multiplexed and transmitted. At the receiving end, monitoring, selecting and video output-signal switching take place under rules whereby the first-encountered marker frame in either stream directs that stream to provide the first content for the video output signal. If the first-encountered marker resides in the higher-resolution stream, the process ends. If it is the lower-resolution stream, a switch to the higher-resolution stream takes place on detection of the next-encountered marker frame in the higher-resolution stream.
|
1. A method associated with minimizing random-access latency to a compressed source video data stream which is characterized with one decoder access latency and one spatial resolution, said method comprising:
engaging such a source video data stream;
deriving, from that engaged data stream, two, broadcastable, downstream-deliverable video data streams that are characterized by differing, respective decoder access latencies and spatial resolutions, one of which broadcastable, downstream-deliverable video data streams is characterized, relatively speaking, by a low decoder access latency and a low spatial resolution, and the other of which is characterized, in comparison, by a higher decoder access latency and a higher spatial resolution, wherein, relatively speaking, said low decoder access latency is associated with more closely spaced I-frames in said one broadcastable, downstream-deliverable video data stream in comparison to more widely separated I-frames in said other broadcastable, downstream-deliverable video data stream; and
transmitting said two, broadcastable, downstream-deliverable video data streams using a first communication channel, wherein said transmitting comprises multiplexing said two, broadcastable, downstream-deliverable video data streams with a time-offset relative to each other, wherein said one broadcastable, downstream-deliverable video stream is delayed relative to said other broadcastable, downstream-deliverable video stream.
4. Apparatus associated with minimizing random-access latency to a compressed source video data stream which is characterized with one decoder access latency and one spatial resolution said apparatus comprising:
engaging structure for engaging a source video data stream;
deriving structure operatively connected to said engaging structure, operable to derive two, broadcastable, downstream-deliverable video data streams from such an engaged source data stream, wherein said two, downstream-deliverable video data streams are characterized by differing, respective decoder access latencies and spatial resolutions, one of which broadcastable, downstream-deliverable video data streams is characterized, relatively speaking, by a low decoder access latency and a low spatial resolution, and the other of which is characterized, by comparison, by a higher decoder access latency and a higher spatial resolution, wherein, relatively speaking, said low decoder access latency is associated with more closely spaced I-frames in said one broadcastable, downstream-deliverable video data stream in comparison to more widely separated I-frames in said other broadcastable, downstream-deliverable video data stream; and
transmitting structure for transmitting said two, broadcastable, downstream-deliverable video data streams over a first communication channel, wherein said transmitting structure comprises a multiplexer for multiplexing said two, broadcastable, downstream-deliverable video data streams with a time-offset relative to each other, wherein said one broadcastable, downstream-deliverable video stream is delayed relative to said other broadcastable, downstream-deliverable video stream.
3. A method, practicable at a video-data reception location, associated with minimizing random-access latency at that location to received compressed video data which is characterized by a pair of prior-derived, broadcastable, video data streams, one of which is further characterized by one decoder access latency and one spatial resolution, and the other of which is further characterized by another decoder access latency which is larger than the mentioned one decoder access latency, and another spatial resolution which is larger than the mentioned one spatial resolution, and where such decoder access latencies are differentiated by different time spacings that exist between designated video I-frames placed in the data streams, with larger spacings between such I-frames relating to larger decoder access latencies, and with smaller spacings between such I-frames relating to smaller decoder access latencies, said method comprising:
seeking access to the received, two-video-data-stream characterized video data, wherein the one video data stream is delayed relative to the other video data stream in the received, two-video-data-stream characterized video data,
in relation to said seeking, monitoring the two, associated video data streams to detect the first occurrence in either stream of an I-frame,
on detecting such an occurrence, selecting the associated data stream to be the source for a viewable output stream, and
(a) if the first detected occurrence involves an I-frame in the mentioned other video data stream, ending the monitoring process and the selecting process, but
(b) if the first detected occurrence involves an I-frame in the mentioned one video data stream, continuing to monitor the other video data stream to detect therein the first next occurrence of an I-frame, and on that detection taking place, switching to and selecting that other video data stream to be the source for a viewable output stream, and then ending the monitoring process and the selecting process.
5. Apparatus which is operable to practice a method implementable at a video-data reception location, for use in association with minimizing random-access latency, at that location, to received, compressed video data which is characterized by a pair of prior-derived, broadcastable, video data streams, one of which is further characterized by one decoder access latency and one spatial resolution, and the other of which is further characterized by another decoder access latency which is larger than the mentioned one decoder access latency, and another spatial resolution which is larger than the mentioned one spatial resolution, and where such decoder access latencies are differentiated by different time spacings that exist between designated video I-frames which are placed in the data streams, with larger spacings between such I-frames relating to larger decoder access latencies, and with smaller spacings between such I-frames relating to smaller decoder access latencies, said apparatus comprising:
seeking structure operable at the mentioned location to access such received video data, wherein the one video data stream is delayed relative to the other video data stream in the received video data,
monitoring structure operatively connected to said seeking structure for monitoring the two video data streams associated with such accessed video data for the purpose of detecting the first occurrence in either stream of an I-frame,
and selecting structure operatively connected to said monitoring structure, operable on the detection of such an I-frame occurrence to select the associated data stream to be the source for a viewable video-data output stream, with said selecting structure specifically operating whereby (a) if the first detected occurrence of an I-frame relates to the mentioned other video data stream, the selecting structure effects an ending of the monitoring process and the selecting process, but (b) if the first detected I-frame relates to the mentioned one video data stream, the selecting structure effects the continuation of monitoring by the monitoring structure of the other video data stream to detect therein the first next occurrence of I-frame, and on such a detection in the other video data stream taking place, effects a switching to and selecting of that other video data stream to be the source for a viewable output video data stream, and with the selecting structure also then effecting an ending of the operations of said monitoring and selecting structures.
|
This invention relates to random access to a compressed video data stream (bit stream), and in particular to methodology and a system (or apparatus) for furnishing and enabling low-latency random access to such a stream.
In both the systemic and methodologic realms, the invention involves, among other things, both the preparing of a conventional video data stream for low-latency, and quickly achieved high-quality (resolution) random access, and the ultimate reception and utilization (i.e. viewing) of that prepared data stream. For the purpose of illustration herein, a preferred and best-mode embodiment of, and manner of implementing and practicing, the invention are disclosed and illustrated in the context of television channel surfing-a representative setting wherein the invention has been found to offer special utility. This context is aptly representative generally of the invention's useful applicability in the areas of video broadcasting and video streaming.
By way of general background regarding conventional understanding, due to temporal prediction, a video decoder cannot begin decoding a video data stream at a frame that is predicted from previous frames. Many applications require a user to be able enter a bit stream at any time, such as during the activity known as channel surfing between broadcast streams, and during reverse, or fast-forward, “trick” video modes. A typical technique for providing random access to a compressed bit stream involves the insertion of so-called I-frames, or intra-frames. Such I-frames are pictured and employed herein, and are also referred to as video marker frames. I-frames are typically inserted into a so-called Group Of Pictures (GOP) structure, and are coded without any prediction from other frames. They are inserted at an interval, or rate, (R). A decoder is able to access an associated video data stream at any such inserted I-frame. The latency involved in random access in such a GOP is inversely proportional to R, while the compression performance decreases as R increases. As an example, I-frames are typically inserted periodically in MPEG-2 format to provide random access.
Another concept in the prior art involves the so-called concept of “dirty random access”, regarding which an access point does not begin with a clean I-frame. Rather, in this approach, a decoder begins decoding and displaying predicted frames without having access to a complete reference frame. Blocks of the frames are intra-coded so that, after some time, a clean picture is attained.
Other approaches to dealing with random access latency have been proposed in the prior art and are generally familiar to those generally skilled in the art.
The present invention departs from prior art approaches by proposing the creation, from an “engaged” source video data stream, of a compressed and either slightly time-offset (preferable), or time-synchronized, pair of divided video data streams (called downstream-deliverable streams) that are drawn directly from the source stream, and that are characterized by possessing respective, different access latencies and resolutions. In particular, and according to a preferred manner of practicing and implementing the invention, such a source data stream is split into two data streams which are slightly time-offset with respect to one another, with one of these streams being specifically characterized with a relatively low access latency (the stream which is slightly time-delayed relative to the other stream) and a relatively low image resolution, and with the other stream being characterized by a larger access latency, and a significantly larger, or greater, image resolution.
These two streams are preferably multiplexed and transmitted so as to be received at a receiving site where they can, in terms of certain fundamental steps practiced by a portion of the present invention, be (a) sought, (b) monitored, (c) and selected in a manner, which uniquely introduces a definitive low-latency access, followed by a rapidly achieved, high-resolution image display.
These, and various other features and advantages, which are offered and attained by the system, apparatus and methodology of the present invention will be more fully appreciated in light of the detailed description which shortly follows, when read in conjunction with the accompanying drawings.
As mentioned above, the present invention is described herein in relation to its utility in the contexts both of video broadcasting and video streaming. The description which now follows should be read with the understanding that it is presented in a manner intended to highlight the utility of the invention in these two particular areas of video data transmission and reception. Additionally,
In
Data stream 12, in accordance with practice of the present invention, is fed to a splitter (or divider, or dividing structure) 14, whose input side (its left side in
It is in the region of what is shown in
Preferably, data streams 16, 18 are appropriately compressed prior to transmission, and any appropriate, conventional compression technique, or techniques, may be employed for this purpose. Preferably also, the combined bandwidth resource called for by data streams 16, 18 for transmission is about the same as that which would be required to transmit a more conventional, single video data stream which is characterized by relatively rapid access and relatively high resolution. Obviously, the particular selected resolutions and I-frame placements (spacing) chosen for data streams 16, 18 will determine this combined data-stream bandwidth requirement. These are matters of user choice, and are not specifically critical to practice of the invention
While, as the case is here, the specific approaches that are employed regarding source data stream 12 to effect splitting and compression, and to assure establishment of the preferred bandwidth-utilization characteristic just mentioned, may be entirely conventional, and thus are not discussed in any further or greater detail herein, the act of splitting per se to create the two, mentioned, latency and resolution-differentiated, downstream-deliverable video data streams is unique, and forms an important part and contribution of the present invention. This preparation from a source data stream preferably takes place, as has been generally expressed with respect to the description given so far for
Continuing with
Disposed intermediate blocks 22, 26 in
Operations of the initiation, seeking, monitoring, selecting and switching functions results in a furnishing to the user's television receiver at site L of an appropriate uncompressed output video presentation (or output video data stream) which, in
Turning attention now to
The individual I-frames and P-frames in data stream 16 are represented in
Data stream 18, by way of contrast, includes both I-frames and P-frames organized in a fashion wherein a significant number (which is not specifically illustrated in
In
According to practice of the invention, when a user seeks access to the image information contained in data streams 16, 18, he or she does this by implementing the seek or start function represented by block 24, whereupon block 22 begins to monitor and examine received data streams 16, 18 for the purpose of detecting the very first I-frame in either of the two data streams which effectively passes a time datum mark, such as dash-dot line TD in
On this detection of an I-frame occurring, block 26 effectively directs into output signal 30, for presentation on the viewer's reception screen, the low-latency, low-resolution imagery data represented by, and contained within, data stream 16. Inasmuch as the first I-frame encountered has occurred in the low-latency data stream, the searching, monitoring and switching functions implementable by blocks 22, 26 remain active, with block 22 continuing now to search for the next-occurring I-frame which appears in the higher-latency, higher-resolution data stream 18. This “next-occurring” frame, which will be, in accordance with what is pictured in
Had the first-encountered I-frame been such a frame in data stream 18, output signal 30 would have immediately been derived from this higher-resolution stream, and the activities involving seeking, monitoring and selecting would have been immediately terminated.
Thus, and reviewing now, as a follow-up to the discussion above, various considerations relating (in the context of conventional practice) to the implementation of this invention, a video bit stream composed of intra frames (I-frames) and predicted frames such as P-frames can be accessed at an I-frame only. To control access latency, I-frames are periodically inserted. The I-frame period determines the access latency. If T is the time between I-frames, the access latency corresponds to a random variable uniformly distributed on the interval [O,T). The statistical values which describe this situation are presented in the table shown in
An I-frame of 1-second gives a maximum access latency of 1-second, and an average access latency of 500-ms. An I-frame period of 0.1-seconds reduces these numbers by a factor of ten. The compromise is bitrate. Decreasing the I-frame period increases the required bitrate for the same output visual quality. In practice an I-frame period is preferably chosen to be near 1-second.
Regarding the structures of the herein proposed two, downstream-deliverable video data streams, such two video streams are used effectively to reduce access latency. Four parameters are used in the encoding of each stream—namely (a) resolution, (b) frame rate, (c) signal-to-noise ratio (bitrate), and (d) access latency. The low-latency stream is coded in a fashion which compromises resolution, frame rate, and signal-to-noise ratio in exchange for greatly improved (shortened) access latency. The parameters selected for the higher-resolution stream are chosen such that that stream is “accessed”, delivered video-signal quality is excellent. Sample values of appropriate “dual-stream” parameters, and of related, representative access latencies, are shown in the tables presented in
The invention thus proposes an efficient, effective, and relatively simple method and apparatus for improving, by minimizing, access latency to a high-resolution video data stream. Utilizing the approach of splitting a source data stream into two latency- and resolution-differentiated data streams for transmission, the method and apparatus of the invention offer low-latency access time wherein the maximum delay (latency) at a viewer's site is never greater than the time distance between I-frames in the low-latency, low-resolution stream. A high-resolution image is presented to a viewer, in all cases, just as soon (after the viewer requests access) as the higher-latency, higher-resolution stream next presents an I-frame. Transmission of the two, proposed, divided video data streams can be accomplished without taxing available bandwidth resources, and specifically by constructing these two data streams in such a manner that, collectively, they require only about the same bandwidth as that required by a typical high-resolution, modest access-latency, single data stream.
Those generally skilled in the art will understand that, while a preferred and best-mode embodiment of the invention has been described and illustrated herein, and a modification mentioned regarding transmission of the proposed, two, divided streams in a synchronized manner, other variations and modifications are possible that come within the scope of the invention.
Patent | Priority | Assignee | Title |
10419513, | Feb 11 2014 | Kiswe Mobile Inc. | Methods and apparatus for reducing latency shift in switching between distinct content streams |
9900362, | Feb 11 2014 | KISWE MOBILE INC | Methods and apparatus for reducing latency shift in switching between distinct content streams |
Patent | Priority | Assignee | Title |
5543925, | Sep 19 1990 | U S PHILIPS CORPORATION | Playback apparatus with selective user preset control of picture presentation |
5995518, | May 01 1997 | Hughes Electronics Corporation | System and method for communication of information using channels of different latency |
6370274, | Oct 14 1993 | U.S. Philips Corporation | METHOD OF STORING IN CODED FORM IMAGE INFORMATION OF MOVING IMAGES, DIAGNOSTIC SYSTEM USING THE METHOD, AND IMAGE CODING AND RECORDING UNIT AND IMAGE-INFORMATION RETRIEVAL UNIT FOR USE IN SUCH A SYSTEM |
6543053, | Nov 27 1996 | MEDIATEK INC | Interactive video-on-demand system |
6665872, | Jan 06 1999 | AMTRAN TECHNOLOGY CO , LTD | Latency-based statistical multiplexing |
7610605, | Jun 14 2000 | Sony Corporation | Method and apparatus for conversion and distribution of data utilizing trick-play requests and meta-data information |
20020095681, | |||
20020146236, | |||
20030088877, | |||
20040025184, | |||
20060140276, | |||
20060156374, | |||
20070098079, | |||
JP2003037818, | |||
WO2004114667, | |||
WO2004114668, |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Nov 25 2003 | KEROFSKY, LOUIS J | Sharp Laboratories of America, Inc | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 014758 | /0218 | |
Dec 01 2003 | Sharp Laboratories of America, Inc. | (assignment on the face of the patent) | / | |||
Feb 14 2013 | Sharp Laboratories of America, Inc | Sharp Kabushiki Kaisha | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 030348 | /0706 |
Date | Maintenance Fee Events |
Oct 27 2014 | ASPN: Payor Number Assigned. |
Oct 27 2014 | RMPN: Payer Number De-assigned. |
May 25 2016 | M1551: Payment of Maintenance Fee, 4th Year, Large Entity. |
Jul 27 2020 | REM: Maintenance Fee Reminder Mailed. |
Jan 11 2021 | EXP: Patent Expired for Failure to Pay Maintenance Fees. |
Date | Maintenance Schedule |
Dec 04 2015 | 4 years fee payment window open |
Jun 04 2016 | 6 months grace period start (w surcharge) |
Dec 04 2016 | patent expiry (for year 4) |
Dec 04 2018 | 2 years to revive unintentionally abandoned end. (for year 4) |
Dec 04 2019 | 8 years fee payment window open |
Jun 04 2020 | 6 months grace period start (w surcharge) |
Dec 04 2020 | patent expiry (for year 8) |
Dec 04 2022 | 2 years to revive unintentionally abandoned end. (for year 8) |
Dec 04 2023 | 12 years fee payment window open |
Jun 04 2024 | 6 months grace period start (w surcharge) |
Dec 04 2024 | patent expiry (for year 12) |
Dec 04 2026 | 2 years to revive unintentionally abandoned end. (for year 12) |