A method of encoding a local time base embedded in the compressed data is disclosed. The local time base is encoded in two parts. The first part has a modulo time base that indicates the specific interval in the reference time base and the second part has a time base increment relative to the reference time. Two forms of time base increment is used to allow for the possibility of different encoding order and displaying order. A mechanism for the synchronization of multiple compressed streams with local time base is also described. A time base offset mechanism is also used to allow finer granularity synchronization of the multiple compressed streams.
|
0. 9. A method for determining a local time base of video object planes (VOPs) in compressed data that includes a time code, a modulo time base and a time base increment, wherein the time code is stored in a group of pictures (GOP) header inserted at a beginning of the compressed data, the modulo time base indicating a predetermined time interval to be combined with one of the time code and a reference time base, the time case increment representing a time increment from a reference time base marked by the modulo time base, the method comprising:
decoding the time code;
decoding the modulo time base of a video object plane;
decoding the time base increment of the video object plane; and
determining the local time base of the video object plane by one of;
combining the decoded modulo time base and the decoded time base increment with the decoded time code when the video object plane is a first plane in a display order after the group of pictures (GOP) header; and
combining the decoded modulo time base and the decoded time base increment with the reference time base marked by the decoded modulo time base of a preceding video object plane when the video object plane is a plane subsequent to the first plane.
0. 1. A method of encoding a local time base of an audio visual sequence in the compressed data comprising the steps of:
obtaining instances of the audio visual sequence by temporal sampling;
determining the local time base of the said instances to be encoded into compressed data;
encoding the said local time base in two parts, comprising of a modulo time base that marks the occurrence of a set of time reference spaced evenly at a specific interval on the local time base and a time base increment relative to the said evenly spaced time reference;
inserting the modulo time base into the compressed data whenever the specific interval have elapsed; and
inserting the time base increment within the compressed data of the said instances of the audio visual sequence.
0. 2. A method of
decoding the time base offset of the individual compressed bitstream;
examining the multiplexed bitstream for the next compressed instance and placing the said instance into the appropriate compressed bitstream until a modulo time base is encountered in the multiplexed bitstream;
inserting a modulo time base to each of the compressed bitstreams; and
repeating the last two steps until the multiplexed bitstream is exhausted.
0. 3. A method of decoding a local time base of an audio visual sequence from the time base of the compressed data encoded according to
initializing the reference time base taking into account the time base offset;
incrementing the reference time base by a specific interval for each modulo time base decoded;
decoding the time base increment of the compressed instance; and
determining the decoding time base of the said instance by adding the said decoded time base increment value to the reference time base.
0. 4. A method of encoding the time base according to
0. 5. A method of
synchronizing the local time base of the individual compressed bitstream to a common time base by encoding and inserting the time base offset in the multiplexed bitstream;
examining each of the compressed bitstream for the next compressed instance to be placed into the multiplexed bitstream until all the compressed bitstreams have arrived at the modulo time base;
inserting a common modulo time base into the multiplex bitstream and skipping the modulo time base of the compressed bitstreams; and
repeating the last two steps until all compressed bitstreams are exhausted.
0. 6. A method of multiplexing a plurality of compressed bitstreams as in
0. 7. A method of encoding a local time base of an audio visual sequence in the compressed data comprising the steps of:
obtaining instances of the audio visual sequence by temporal sampling;
determining the local time base of the said instances to be encoded into compressed data;
encoding the said instances in one of two methods comprising of a first method of compression without reference to any future instances and a second method of compression which refers to the future reconstructed instance;
encoding the said local time base in two parts, comprising of a modulo time base that marks the occurrence of a set of time reference spaced evenly at a specific interval on the local time base and a time base increment;
encoding the time base increment for the instance compressed using the first compression method as an absolute value relative to the said evenly spaced time reference;
encoding the time base increment for the instance compressed using the second compression method as a relative value to the local time base of the previously compressed instance using the said first method;
inserting the modulo time base into the compressed data whenever the specific interval have elapsed; and
inserting the time base increment within the compressed data of the said instances of the audio visual sequence.
0. 8. A method of decoding a local time base of an audio visual sequence from the time base of the compressed data encoded according to
initializing the reference time base taking into account of the time base offset;
incrementing the reference time base by a specific interval for each modulo time base decoded;
decoding the time base increment of the compressed instance; and
determining the time base increment to be one of two types, absolute or relative, depending on the first or second compression method used in the encoding of the instance, respectively;
determining the decoding time base of the said instance by adding the said decoded time base increment value to the reference time base if the time base is of the first type; and
determining the decoding time base of the said instance by adding the said decoded time base increment value to the decoding time base of the previous instance encoded using the first compression method, if the time base increment is of the second type.
0. 10. The method of
0. 11. The method of
|
|||||||||||||||||
tAVT1=tTB11P−tGTBn (2)
tRVT1=tETB−tETB11P (3)
where
Next, multiplexing more than one video object planes is described.
When more than one video object planes are multiplexed together, the multiplexer examines the multiple video object plane bitstreams to determine the order of multiplexing as well as the synchronization. The operations involved is depicted in FIG. 8. The VOP time offsets for each of the video object planes to be multiplexed are inserted into the bitstream in step 11. All the bitstreams of the video object planes to be multiplexed are then examined in step 12 to determine if all of the planes are at their respective modulo time base. If they are then the processing is passes to step 13 where a common modulo time base is inserted into the multiplexed bitstream. The processing then proceeds to step 14 ,where the next encoded video object is inserted into the multiplexed bitstream. In step 15 the bitstreams of the video object planes to be multiplexed are again examined to see if there are more video objects to be multiplexed. If there is, then the control is passed to step 12 again. Otherwise the process ends.
De-multiplexing the bitstream containing more than one video object planes is described.
The de-multiplexing of the bitstream containing multiple video object planes is depicted in FIG. 9. The process begins in step 21 where the VOP time offsets are decoded and passed to the decoders for synchronization. Then the multiplexed bitstream is examined in step 22 to see if the modulo time base is found. If the modulo time base is found then the process is passed to step 23 where the modulo time base is inserted into all the video object bitstreams. The processing then continues with step 24 where the next video object is examined and inserted into the appropriate video object bitstream. Finally the multiplexed bitstream is examined again to see if there is any more video object to de-multiplex. If there is, then the process is passed to step 22 again. Otherwise the process ends.
Recovering the time base is described.
The embodiment of the time base recovery is shown in FIG. 10. In recovering the local time base, the process begins in step 31 where the local time based is initialized taking into account of the VOP time offset decoded by the de-multiplexer. Then the process is passed to step 32 which checks the bitstream to determine if the modulo time base is decoded. If the modulo time base is decoded then the process is passed to step 33 where the local time base is incremented by the modulo time base increment. The process is then passed to step 37. If the modulo time base is not decoded then the process is passed to step 34 where the video object is examine to determine if it is a B-video object or not. If it is then the process is passed to step 35 where the B-video object decoding time base is calculated based on equation (6). The process is then passed to step 37. If the result of step 34 is not a B-video object then the process is passed to step 36 where the decoding time base is calculated based on equation (5). The process then proceeds to step 37. In step 37, the bitstream is examine to see if there is more video objects to decode. If there is then the process is passed to step 32 again. Otherwise the processing ends.
The following formula are used to determine the presentation time stamp of the video objects:
tGTBn=nXtGTB1+tGTB0(n=0, 1, 2, 3, . . . ) (4)
tDTB11P=tAVT1+tGTBn (5)
tDTBB=tRVT1+tDTB11P (6)
where
Implementation of the Bitstream Encoder is described.
The process begins in the initializer, step 41 where the bitstream encoder starts by initializing the local time base register to the initial value of the time code. The same time code value is encoded into the bitstream. At the start of the encoding of the next I-VOP the time code comparator, step 42, compares the presentation time of the I-VOP against the local time base register. The result is passed to the modulo time base encoder, step 43. The modulo time base encoder will insert into the bitstream the required number of “1” equivalent to the number of modulo time base increment that has elapsed. This is then followed by the symbol “O” to signal the end of the modulo time base code. The local time base register is updated to the current modulo time base. The processing then proceed to the VOP time base increment encoder, step 44, where the remainder of the presentation time code of the I-VOP is encoded.
The process then repeats with the next encoded video object plane which is the P-VOP. The time code comparator, step 42, compares the presentation time of the P-VOP against the local time base register. The result is passed to the modulo time base encoder, step 43. The modulo time base encoder will insert the required number of “1” equivalent to the number of modulo time base increment that has elapsed. This is then followed by the symbol “O” to signal the end of the modulo time base code. The B-VOP time base register is set to the value of the local time base register and the local time base register is updated to the current modulo time base. The processing then proceeds to the VOP time base increment encoder, step 44, where the remainder of the presentation time code of the P-VOP is encoded.
The process then repeats with the next encoded video object plane which is the B-VOP. The time code comparator, step 42, compares the presentation time of the B-VOP against the B-VOP time base register. The result is passed to the modulo time base encoder, step 43. The modulo time base encoder will insert the required number of “1” equivalent to the number of modulo time base increment that has elapsed. This is then followed by the symbol “O” to signal the end of the modulo time base code. Both the B-VOP time base register and the local time base register are not changed after the processing of the B-VOP. The processing then proceed to the VOP time base increment encoder, step 44, where the remainder of the presentation time code of the B-VOP is encoded.
The local time base register is reset at the next I-VOP which marks the beginning of the next group of VOP.
Implementation of the Bitstream Decoder is described.
The processing begins in the initializer, step 51, where the local time base register is set to the value of the time code decoded from the bitstream. The processing then proceeds the modulo time base decoder, step 52, where the modulo time base increment is decoded. The total number of modulo time base increment decoded is given by the number of “1” decoded before the symbol “0” Next the VOP time base increment is decoded in the VOP time base increment, step 53. In the time base calculator, step 54, the presentation time of the I-VOP is recovered. The total decoded modulo time base increment value is added to the local time base register. The VOP time base increment is then added to the local time base register to obtain the presentation time of the I-VOP. The process then goes to the video object decoder where the video object is decoded.
For the P-VOP, the process repeats in the modulo time base decoder, step 52, where the modulo time base increment is decoded. The total number of modulo time base increment decoded is given by the number of “1” decoded before the symbol “O”. Next the VOP time base increment is decoded in the VOP time base increment, step 53. In the time base calculator, step 54, the presentation time of the P-VOP is recovered. The B-VOP modulo time base register is set to the value in the local time base register. The total decoded modulo time base increment value is added to the local time base register. The VOP time base increment is then added to the local time base register to obtain the presentation time of the P-VOP. The process then goes to the video object decoder where the video object is decoded.
For the B-VOP, the process repeats in the modulo time base decoder, step 52, where the modulo time base increment is decoded. The total number of modulo time base increment decoded is given by the number of “1” decoded before the symbol “O”. Next the VOP time base increment is decoded in the VOP time base increment, step 53. In the time base calculator, step 54, the presentation time of the B-VOP is recovered. The total decoded modulo time base increment value and the VOP time base increment is added to the B-VOP time base register to obtain the presentation time of the B-VOP. Both the B-VOP time base register and the local time base register are left unchanged. The process then goes to the video object decoder where the video object is decoded.
The local time base register is reset at the next I-VOP which marks the beginning of the next group of VOP.
Specific example is described.
Referring to
For encoding the VOPs, it is necessary to insert the display time data to each VOP. The insertion of the full time data including hour, minute, second and millisecond occupies a certain data space in the header portion of each VOP. The aim of the present invention is to reduce such a data space and to simplify the time data to be inserted in each VOP.
Each of the VOPs shown in the first row in
As shown in the second row in
As shown in the third row in
Then, at time T2, the first VOP (I1-VOP) is taken up. The time code comparator compares the time (hour, minute, second) stored in the local time base register with the time (hour, minute, second) temporarily stored in the first VOP (I1-VOP). According to the example, the result of the comparison is the same. Thus, the comparator produces “0” indicating that the first VOP (I1-VOP) occurs in the same second as the second being held in the local time base register. The result “0” as produced from the comparator is added to the first VOP (I1-VOP) at the modulo time base area. At the same time, the hour, minute and second data temporarily stored in the first VOP (I1-VOP) is eliminated. Thus, before time T3, the bitstream data for the first VOP (I1-VOP) is obtained with “0” inserted in the modulo time base area and “350” inserted in the VOP time increment area.
Then, at time T3, the second VOP (P1-VOP) is taken up. The time code comparator compares the time (hour, minute, second) stored in the local time base register with the time (hour, minute, second) temporarily stored in the second VOP (P1-VOP). According to the example, the result of the comparison is such that the time temporarily stored in the second VOP (P1-VOP) is one second greater than the time stored in local time base register. Thus, the comparator produces “10” indicating that the second VOP (P1-VOP) occurs in the next one second to the second being held in the local time base register. If the second VOP (P1-VOP) occurs in the next—next second to the second being held in the local time base register, the comparator will produce “110”.
After time T3, the B-VOP time base register is set to the time equal to the time carried in the local time base register immediately before time T3. In the example, the B-VOP time base register is set to 1:23:45. Also, after time T3, the local time base register is incremented to the time equal to the time temporarily stored in the second VOP (P1-VOP). Thus, in the example, the local time base register is incremented to 1:23:46.
The result “10” as produced from the comparator is added to the second VOP (P1-VOP) at the modulo time base area. At the same time, the hour, minute and second data temporarily stored in the second VOP (P1-VOP) is eliminated. Thus, before time T4, the bitstream data for the second VOP (P1-VOP) is obtained with “10” inserted in the modulo time base area and “550” inserted in the VOP time increment area.
Then, at time T4, the third VOP (B1-VOP) is taken up. The time code comparator compares the time (hour, minute, second) stored in the B-VOP time base register with the time (hour, minute, second) temporarily stored in the third VOP (B1-VOP). According to the example, the result of the comparison is the same. Thus, the comparator produces “0” indicating that the third VOP (B1-VOP) occurs in the same second as the second being held in the B-VOP time base register. The result “0” as produced from the comparator is added to the third VOP (B1-VOP) at the modulo time base area. At the same time, the hour, minute and second data temporarily stored in the third VOP (I1-VOP) is eliminated. Thus, before time T5, the bitstream data for the third VOP (B1-VOP) is obtained with “0” inserted in the modulo time base area and “750” inserted in the VOP time increment area.
Then, at time T5, the fourth VOP (B2-VOP) is taken up. The time code comparator compares the time (hour, minute, second) stored in the B-VOP time base register with the time (hour, minute, second) temporarily stored in the fourth VOP (B2-VOP). According to the example, the result of the comparison is such that the time temporarily stored in the fourth VOP (B2-VOP) is one second greater than the time stored in B-VOP time base register. Thus, the comparator produces “10” indicating that the fourth VOP (B2-VOP) occurs in the next one second to the second being held in the B-VOP time base register.
During the processing of the B type VOP, neither the local time base register nor the B-VOP time base register is incremented regardless of whatever result the comparator produces.
The result “10” as produced from the comparator is added to the fourth VOP (B2-VOP) at the module time base area. At the same time, the hour, minute and second data temporarily stored in the fourth VOP (B2-VOP) is eliminated. Thus, before time T6, the bitstream data for the fourth VOP (B2-VOP) is obtained with “10” inserted in the modulo time base area and “150” inserted in the VOP time increment area.
Then, at time T6, the fifth VOP (P2-VOP) is taken up. The time code comparator compares the time (hour, minute, second) stored in the local time base register with the time (hour, minute, second) temporarily stored in the fifth VOP (P2-VOP). According to the example, the result of the comparison is such that the time temporarily stored in the fifth VOP (P2-VOP) is one second greater than the time stored in local time base register. Thus, the comparator produces “10” indicating that the fifth VOP (P2-VOP) occurs in the next one second to the second being held in the local time base register.
After time T6, the B-VOP time base register is incremented to the time equal to the time carried in the local time base register immediately before time T6. In the example, the B-VOP time base register is incremented to 1:23:46. Also, after time T6, the local time base register is incremented to the time equal to the time temporarily stored in the fifth VOP (P2-VOP). Thus, in the example, the local time base register is incremented to 1:23:47.
The result “10” as produced from the comparator is added to the fifth VOP (P2-VOP) at the modulo time base area. At the same time, the hour, minute and second data temporarily stored in the fifth VOP (P2-VOP) is eliminated. Thus, before time T4, the bitstream data for the fifth VOP (P2-VOP) is obtained with “10” inserted in the modulo time base area and “350” inserted in the VOP time increment area.
Thereafter, a similar operation is carried out for forming bitstream data for the following VOPs.
For decoding the bitstream data, an operation opposite to the above is carried out. First, the time (hour, minute, second) carried in the GOP header is read. The read time is stored in the local time base register.
Upon receipt of the I type or P type VOP, i.e., other then the B type VOP, the data stored in the modulo time base area is read. If the read data is “0”, i.e., without any is in front of 0, no change is made in the local time base register, and also no change is made in the B-VOP time base register. If the read data is “10”, the time stored in the local time base register is incremented by one second. If the read data is “110”, the time stored in the local time base register is incremented by two seconds. In this manner, the number of seconds which should be incremented is determined by the number of is inserted in front of 0. Also, when the read data is “10” or “110”, the B-VOP time base register, which is a memory, copies the time the local time base register had carried just before the most recent increment. Then, the time (hour, minute, second) carried in the local time base register is combined with the time (millisecond) carried in the VOP time increment area to establish a specific time at which the I type or P type VOP should occur.
Upon receipt of the B type VOP, the data stored in the module time base area is read. If the read data is “0”, the time (hour, minute, second) carried in the B-VOP VOP time base register is combined with the time (millisecond) carried in the VOP time increment area to establish a specific time at which the B type VOP should occur. If the read data is “10”, the time (hour, minute, second) carried in the B-VOP time base register is added one second, and then the added result time is combined with the time (millisecond) carried in the VOP time increment area to establish a specific time at which the B type VOP should occur. If the read data is “110”, the time (hour, minute, second) carried in the B-VOP time base register is added two seconds, and then the added result time is combined with the time (milliseconds) carried in the VOP time increment area to establish a specific time at which the B type VOP should occur.
The effects of this invention is to enable video object planes encoded by different encoders to be multiplexed. It will also facilitate object based bitstream manipulation of compressed data that come from different sources to create new bitstreams. It provides a method of synchronization for audio visual objects.
The invention being thus described, it will be obvious that the same may be varied in many ways. Such variations are not to be regarded as a departure from the spirit and scope of the invention, and all such modifications as would be obvious to one skilled in the art are intended to be included within the scope of the following claims.
Tan, Thiow Keng, Shen, Sheng Mei, Lee, Chak Joo
| Patent | Priority | Assignee | Title |
| Patent | Priority | Assignee | Title |
| 5430485, | Sep 30 1993 | Thomson Consumer Electronics, Inc. | Audio/video synchronization in a digital transmission system |
| 5467139, | Sep 30 1993 | THOMSON CONSUMER ELECTRONICS, INC 10330 NORTH MERIDIAN STREET | Muting apparatus for a compressed audio/video signal receiver |
| 5537409, | Jul 16 1993 | Pioneer Electronic Corporation | Synchronizing system for time-divided video and audio signals |
| 5784422, | Aug 05 1996 | Transcrypt International, Inc. | Apparatus and method for accurate synchronization with inbound data packets at relatively low sampling rates |
| 5793431, | Dec 02 1994 | U.S. Philips Corporation | Audio/video discrepancy management |
| 5953073, | Jul 29 1996 | Time Warner Cable Enterprises LLC | Method for relating indexing information associated with at least two indexing schemes to facilitate the play-back of user-specified digital video data and a video client incorporating the same |
| EP20634870, | |||
| EP20648056, | |||
| EP30634870, | |||
| EP30648056, |
| Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
| Jul 03 1997 | Matsushita Electric Industrial Co., Ltd. | (assignment on the face of the patent) | / | |||
| May 27 2014 | Panasonic Corporation | Panasonic Intellectual Property Corporation of America | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 033033 | /0163 |
| Date | Maintenance Fee Events |
| Date | Maintenance Schedule |
| Oct 17 2009 | 4 years fee payment window open |
| Apr 17 2010 | 6 months grace period start (w surcharge) |
| Oct 17 2010 | patent expiry (for year 4) |
| Oct 17 2012 | 2 years to revive unintentionally abandoned end. (for year 4) |
| Oct 17 2013 | 8 years fee payment window open |
| Apr 17 2014 | 6 months grace period start (w surcharge) |
| Oct 17 2014 | patent expiry (for year 8) |
| Oct 17 2016 | 2 years to revive unintentionally abandoned end. (for year 8) |
| Oct 17 2017 | 12 years fee payment window open |
| Apr 17 2018 | 6 months grace period start (w surcharge) |
| Oct 17 2018 | patent expiry (for year 12) |
| Oct 17 2020 | 2 years to revive unintentionally abandoned end. (for year 12) |