Method for display time stamping and synchronization of multiple video object planes

Abstract

A method of encoding a local time base embedded in the compressed data is disclosed. The local time base is encoded in two parts. The first part has a modulo time base that indicates the specific interval in the reference time base and the second part has a time base increment relative to the reference time. Two forms of time base increment is used to allow for the possibility of different encoding order and displaying order. A mechanism for the synchronization of multiple compressed streams with local time base is also described. A time base offset mechanism is also used to allow finer granularity synchronization of the multiple compressed streams.

Description

t_AVTI=t_TBI/P−t_GTBn  (2)
t_RVTI=t_ETB−t_ETBI/P  (3)
where

• t_GTBn is the encoder time base marked by the nth encoded modulo time base,
• t_GTBI is the pre-determined modulo time base interval,
• t_GTB0 is the encoder time base start time,
• t_AVTI is the absolute VOP time increment for the I or P-video object planes, and
• t_ETBI/P is the encoder time base at the start of the encoding of the I or P video object planes,
• t_RVTI is the relative VOP time increment for the B-video object planes, and
• t_ETBB is the encoder time base at the start of the encoding of the B video object plane.

Next, multiplexing more than one video object planes is described.

When more than one video object planes are multiplexed together, the multiplexer examines the multiple video object plane bitstreams to determine the order of multiplexing as well as the synchronization. The operations involved is depicted in FIG. 8. The VOP time offsets for each of the video object planes to be multiplexed are inserted into the bitstream in step 11. All the bitstreams of the video object planes to be multiplexed are then examined in step 12 to determine if all of the planes are at their respective modulo time base. If they are then the processing is passes to step 13 where a common modulo time base is inserted into the multiplexed bitstream. The processing then proceeds to step 14 where the next encoded video object is inserted into the multiplexed bitstream. In step 15 the bitstreams of the video object planes to be multiplexed are again examined to see if there are more video objects to be multiplexed. If there is, then the control is passed to step 12 again. Otherwise the process ends.

De-multiplexing the bitstream containing more than one video object planes is described.

The de-multiplexing of the bitstream containing multiple video object planes is depicted in FIG. 9. The process begins in step 21 where the VOP time offsets are decoded and passed to the decoders for synchronization. Then the multiplexed bitstream is examined in step 22 to see if the modulo time base is found. If the modulo time base is found then the process is passed to step 23 where the modulo time base is inserted into all the video object bitstreams. The processing then continues with step 24 where the next video object is examined and inserted into the appropriate video object bitstream. Finally the multiplexed bitstream is examined again to see if there is any more video object to de-multiplex. If there is, then the process is passed to step 22 again. Otherwise the process ends.

Recovering the time base is described.

The embodiment of the time base recovery is shown in FIG. 10. In recovering the local time base, the process begins in step 31 where the local time based is initialized taking into account of the VOP time offset decoded by the de-multiplexer. Then the process is passed to step 32 which checks the bitstream to determine if the modulo time base is decoded. If the modulo time base is decoded then the process is passed to step 33 where the local time base is incremented by the modulo time base increment. The process is then passed to step 37. If the modulo time base is not decoded then the process is passed to step 34 where the video object is examine to determine if it is a B-video object or not. If it is then the process is passed to step 35 where the B-video object decoding time base is calculated based on equation (6). The process is then passed to step 37. If the result of step 34 is not a B-video object then the process is passed to step 36 where the decoding time base is calculated based on equation (5). The process then proceeds to step 37. In step 37, the bitstream is examine to see if there is more video objects to decode. If there is then the process is passed to step 32 again. Otherwise the processing ends.

The following formula are used to determine the presentation time stamp of the video objects:
t_GTBn=nxt_GTBI+t_GTB0(n=1, 2, 3, . . . )  (4)
t_DTBI/P=t_ABTI+t_GTBn  (5)
t_DTBB=t_RVYI+t_DTBI/P  (6)
where

• t_GTBn is the decoding time base marked by the nth decoded modulo time base,
• t_GTBI is the pre-determined modulo time base interval,
• t_GTB0 is the decoding time base start time,
• t_DTBI/P is the decoding time base at the start of the decoding of the I or P video object planes,
• t_AVTI is the decoded absolute VOP time increment for the I or P-video object planes,
• t_DTBB is the decoding time base at the start of the decoding of the B video object plane, and
• t_RVTI is the decoded relative VOP time increment for the B-video object planes.

Implementation of the Bitstream Encoder is described.

FIG. 11 shows the block diagram for an implementation of the bitstream encoder for encoding the modulo time base and the VOP time increment. For the purpose of this illustration the example in FIG. 3B is used. Since bi-directional prediction is used, the encoding order is different from the presentation order shown in FIG. 3B. The encoding order begins with the I-VOP followed by the P-VOP before the B-VOP This is illustrated in the three paragraphs below.

The process begins in the initializer, step 41 where the bitstream encoder starts by initializing the local time base register to the initial value of the time code. The same time code value is encoded into the bitstream. At the start of the encoding of the next I-VOP the time code comparator, step 42, compares the presentation time of the I-VOP against the local time base register. The result is passed to the modulo time base encoder, step 43. The modulo time base encoder will insert into the bitstream the required number of “1” equivalent to the number of modulo time base increment that has elapsed. This is then followed by the symbol “O” to signal the end of the modulo time base code. The local time base register is updated to the current modulo time base. The processing then proceed to the VOP time base increment encoder, step 44, where the remainder of the presentation time code of the I-VOP is encoded.

The process then repeats with the next encoded video object plane which is the P-VOP The time code comparator, step 42, compares the presentation time of the P-VOP against the local time base register. The result is passed to the modulo time base encoder, step 43. The modulo time base encoder will insert the required number of “1” equivalent to the number of modulo time base increment that has elapsed. This is then followed by the symbol “O” to signal the end of the modulo time base code. The B-VOP time base register is set to the value of the local time base register and the local time base register is updated to the current modulo time base. The processing then proceeds to the VOP time base increment encoder, step 44, where the remainder of the presentation time code of the P-VOP is encoded.

The process then repeats with the next encoded video object plane which is the B-VOP The time code comparator, step 42, compares the presentation time of the B-VOP against the B-VOP time base register. The result is passed to the modulo time base encoder, step 43. The modulo time base encoder will insert the required number of “1” equivalent to the number of modulo time base increment that has elapsed. This is then followed by the symbol “O” to signal the end of the modulo time base code. Both the B-VOP time base register and the local time base register are not changed after the processing of the B-VOP The processing then proceed to the VOP time base increment encoder, step 44, where the remainder of the presentation time code of the B-VOP is encoded.

The local time base register is reset at the next I-VOP which marks the beginning of the next group of VOP.

Implementation of the Bitstream Decoder is described.

FIG. 12 illustrates the block diagram for the implementation of the decoder for the modulo time base and VOP time increment to recover the presentation time stamp. As in the implementation of the encoder the example in FIG. 3B is used. The decoding order is the same as the encoding order where the I-VOP is decoded followed by the P-VOP before the B-VOP This is explained in the paragraphs below.

The processing begins in the initializer, step 51, where the local time base register is set to the value of the time code decoded from the bitstream. The processing then proceeds the modulo time base decoder, step 52, where the modulo time base increment is decoded. The total number of modulo time base increment decoded is given by the number of “1” decoded before the symbol “0” Next the VOP time base increment is decoded in the VOP time base increment, step 53. In the time base calculator, step 54, the presentation time of the I-VOP is recovered. The total decoded modulo time base increment value is added to the local time base register. The VOP time base increment is then added to the local time base register to obtain the presentation time of the I-VOP The process then goes to the video object decoder where the video object is decoded.

For the P-VOP, the process repeats in the modulo time base decoder, step 52, where the modulo time base increment is decoded. The total number of modulo time base increment decoded is given by the number of “1” decoded before the symbol “O ”. Next the VOP time base increment is decoded in the VOP time base increment, step 53. In the time base calculator, step 54, the presentation time of the P-VOP is recovered. The B-VOP modulo time base register is set to the value in the local time base register. The total decoded modulo time base increment value is added to the local time base register. The VOP time base increment is then added to the local time base register to obtain the presentation time of the P-VOP. The process then goes to the video object decoder where the video object is decoded.

For the B-VOP, the process repeats in the modulo time base decoder, step 52, where the modulo time base increment is decoded. The total number of modulo time base increment decoded is given by the number of “1” decoded before the symbol “O”. Next the VOP time base increment is decoded in the VOP time base increment, step 53. In the time base calculator, step 54, the presentation time of the B-VOP is recovered. The total decoded modulo time base increment value and the VOP time base increment is added to the B-VOP time base register to obtain the presentation time of the B-VOP. Both the B-VOP time base register and the local time base register are left unchanged The process then goes to the video object decoder where the video object is decoded.

The local time base register is reset at the next I-VOP which marks the beginning of the next group of VOP.

Specific example is described.

Referring to FIG. 13, an example of the steps for encoding the compressed data into bitstream data is shown. As shown in the top row in FIG. 13, the compressed video data VOPs are aligned in the display order, I1 B1, B2, P1, B3, P3 with a GOP (group of pictures) header inserted at the beginning of group of VOPs. While being displayed, the local time, at which the display is effected, is measured for each VOP using the local time clock. For example, the first VOP (I1-VOP) is displayed at 1 hour 23 minute 45 second 350 millisecond (1:23:45:350) counted from the very beginning of the video data; the second VOP (B1-VOP) is displayed at 1:23:45:750; the third VOP (B2-VOP) is displayed at 1:23:46:150; and so on.

For encoding the VOPs, it is necessary to insert the display time data to each VOP The insertion of the full time data including hour, minute, second and millisecond occupies a certain data space in the header portion of each VOP. The aim of the present invention is to reduce such a data space and to simplify the time data to be inserted in each VOP.

Each of the VOPs shown in the first row in FIG. 13 is stored with the display time data of millisecond at the VOP time increment area. Each of the VOPs in the first row is also stored temporarily with the display time data of hour, minute and second. The GOP header is stored with the display data of hour, minute and second for the first VOP (I1-VOP).

As shown in the second row in FIG. 13, the VOPs are delayed by a predetermined time using a buffer (not shown). When the VOPs are produced from the buffer, the order of the VOPs are changed according to the bi-directional prediction rule such that the bi-directional VOP, i.e., B-VOP, should be positioned after the P-VOP to which the B-VOP references. Thus, the VOPs are aligned in the order of I1, P1, B1, B2, P2, B3.

As shown in the third row in FIG. 13, at time T1, i.e., when the GOP header is being encoded, the local time base register stores the hour, minute, second data as stored in the GOP header. In the example shown in FIG. 13, the local time base register stores 1:23:45. Then, before time T2, the bitstream data for the GOP header is obtained with the hour, minute, second data being inserted as shown at the bottom row in FIG. 13.

Then, at time T2, the first VOP (I1-VOP) is taken up. The time code comparator compares the time (hour, minute, second) stored in the local time base register with the time (hour, minute, second) temporarily stored in the first VOP (I1-VOP). According to the example, the result of the comparison is the same. Thus, the comparator produces “0” indicating that the first VOP (I1-VOP) occurs in the same second as the second being held in the local time base register. The result “0” as produced from the comparator is added to the first VOP (I1-VOP) at the modulo time base area. At the same time, the hour, minute and second data temporarily stored in the first VOP (I1-VOP) is eliminated. Thus, before time T3, the bitstream data for the first VOP (I1-VOP) is obtained with “0” inserted in the modulo time base area and “350” inserted in the VOP time increment area.

Then, at time T3, the second VOP (P1-VOP) is taken up. The time code comparator compares the time (hour, minute, second) stored in the local time base register with the time (hour, minute, second) temporarily stored in the second VOP (PI-VOP). According to the example, the result of the comparison is such that the time temporarily stored in the second VOP (P1-VOP) is one second greater than the time stored in local time base register. Thus, the comparator produces “10” indicating that the second VOP (P1-VOP) occurs in the next one second to the second being held in the local time base register. If the second VOP (P1-VOP) occurs in the next—next second to the second being held in the local time base register, the comparator will produce “110”.

After time T3, the B-VOP time base register is set to the time equal to the time carried in the local time base register immediately before time T3. In the example, the B-VOP time base register is set to 1:23:45. Also, after time T3, the local time base register is incremented to the time equal to the time temporarily stored in the second VOP (P1-VOP). Thus, in the example, the local time base register is incremented to 1:23:46.

The result “10” as produced from the comparator is added to the second VOP (P1-VOP) at the modulo time base area. At the same time, the hour, minute and second data temporarily stored in the second VOP (P1-VOP) is eliminated. Thus, before time T4, the bitstream data for the second VOP (P1-VOP) is obtained with “10” inserted in the modulo time base area and “550” inserted in the VOP time increment area.

Then, at time T4, the third VOP (B1-VOP) is taken up. The time code comparator compares the time (hour, minute, second) stored in the B-VOP time base register with the time (hour, minute, second) temporarily stored in the third VOP (B1-VOP). According to the example, the result of the comparison is the same. Thus, the comparator produces “0” indicating that the third VOP (B1-VOP) occurs in the same second as the second being held in the B-VOP time base register. The result “0” as produced from the comparator is added to the third VOP (B1-VOP) at the modulo time base area. At the same time, the hour, minute and second data temporarily stored in the third VOP (I1-VOP) is eliminated. Thus, before time T5, the bitstream data for the third VOP (B1-VOP) is obtained with “0” inserted in the modulo time base area and “750” inserted in the VOP time increment area.

Then, at time T5, the fourth VOP (B2-VOP) is taken up. The time code comparator compares the time (hour, minute, second) stored in the B-VOP time base register with the time (hour, minute, second) temporarily stored in the fourth VOP (B2-VOP). According to the example, the result of the comparison is such that the time temporarily stored in the fourth VOP (B2-VOP) is one second greater than the time stored in B-VOP time base register. Thus, the comparator produces “10” indicating that the fourth VOP (B2-VOP) occurs in the next one second to the second being held in the B-VOP time base register.

During the processing of the B type VOP, neither the local time base register nor the B-VOP time base register is incremented regardless of whatever result the comparator produces.

The result “10” as produced from the comparator is added to the fourth VOP (B2-VOP) at the modulo time base area. At the same time, the hour, minute and second data temporarily stored in the fourth VOP (B2-VOP) is eliminated. Thus, before time T6, the bitstream data for the fourth VOP (B2-VOP) is obtained with “10” inserted in the modulo time base area and “150” inserted in the VOP time increment area.

Then, at time T6, the fifth VOP (P2-VOP) is taken up. The time code comparator compares the time (hour, minute, second) stored in The local time base register with the time (hour, minute, second) temporarily stored in the fifth VOP (P2-VOP). According to the example, the result of the comparison is such that the time temporarily stored in the fifth VOP (P2-VOP) is one second greater than the time stored in local time base register. Thus, the comparator produces “10” indicating that the fifth VOP (P2-VOP) occurs in the next one second to the second being held in the local time base register.

After time T6, the B-VOP time base register is incremented to the time equal to the time carried in the local time base register immediately before time T6. In the example, the B-VOP time base register is incremented to 1:23:46. Also, after time T6, the local time base register is incremented to the time equal to the time temporarily stored in the fifth VOP (P2-VOP). Thus, in the example, the local time base register is incremented to 1:23:47.

The result “10” as produced from the comparator is added to the fifth VOP (P2-VOP) at the modulo time base area. At the same time, the hour, minute and second data temporarily stored in the fifth VOP (P2-VOP) is eliminated. Thus, before time T4, the bitstream data for the fifth VOP (P2-VOP) is obtained with “10” inserted in the modulo time base area and “350” inserted in the VOP time increment area.

Thereafter, a similar operation is carried out for forming bitstream data for the following VOPs.

For decoding the bitstream data, an operation opposite to the above is carried out. First, the time (hour, minute, second) carried in the GOP header is read. The read time is stored in the local time base register.

Upon receipt of the I type or P type VOP, i.e., other then the B type VOP, the data stored in the modulo time base area is read. If the read data is “0”, i.e., without any is in front of 0, no change is made in the local time base register, and also no change is made in the B-VOP time base register. If the read data is “10”, the time stored in the local time base register is incremented by one second. If the read data is “110”, the time stored in the local time base register is incremented by two seconds. In this manner, the number of seconds which should be incremented is determined by the number of is inserted in front of 0. Also, when the read data is “10” or “110”, the B-VOP time base register, which is a memory, copies the time the local time base register had carried just before the most recent increment. Then, the time (hour, minute, second) carried in the local time base register is combined with the time (millisecond) carried in the VOP time increment area to establish a specific time at which the I type or P type VOP should occur.

Upon receipt of the B type VOP, the data stored in the modulo time base area is read. If the read data is “0”, the time (hour, minute, second) carried in the B-VOP VOP time base register is combined with the time (millisecond) carried in the VOP time increment area to establish a specific time at which the B type VOP should occur. If the read data is “10”, the time (hour, minute, second) carried in the B-VOP time base register is added one second, and then the added result time is combined with the time (millisecond) carried in the VOP time increment area to establish a specific time at which the B type VOP should occur. If the read data is “110”, the time (hour, minute, second) carried in the B-VOP time base register is added two seconds, and then the added result time is combined with the time (millisecond) carried in the VOP time increment area to establish a specific time at which the B type VOP should occur.

The effects of this invention is to enable video object planes encoded by different encoders to be multiplexed. It will also facilitate object based bitstream manipulation of compressed data that come from different sources to create new bitstreams. It provides a method of synchronization for audio visual objects.

The invention being thus described, it will be obvious that the same may be varied in many ways. Such variations are not to be regarded as a departure from the spirit and scope of the invention, and all such modifications as would be obvious to one skilled in the art are intended to be included within the scope of the following claims.

Claims

1. A method of encoding a local time base of an audio visual sequence in the compressed data comprising the steps of:

obtaining instances of the audio visual sequence by temporal sampling;

determining the local time base of the said instances to be encoded into compressed data;

encoding the said local time base in two parts, comprising of a modulo time base that marks the occurrence of a set of time reference spaced evenly at a specific interval on the local time base and a time base increment relative to the said evenly spaced time reference;

inserting the modulo time base into the compressed data whenever the specific interval have elapsed; and

inserting the time base increment within the compressed data of the said instances of the audio visual sequence.

2. A method of claim 1, wherein a plurality of compressed bitstreams containing local time base information encoded therein are multiplexed and de-multiplexed, for de-multiplexing further comprising the steps of:

decoding the time base offset of the individual compressed bitstream;

examining the multiplexed bitstream for the next compressed instance and placing the said instance into the appropriate compressed bitstream until a modulo time base is encountered in the multiplexed bitstream;

inserting a modulo time base to each of the compressed bitstreams; and

repeating the last two steps until the multiplexed bitstream is exhausted.

3. A method of decoding a local time base of an audio visual sequence from the time base of the compressed data encoded according to claim 1 comprising the steps of:

initializing the reference time base taking into account the time base offset;

incrementing the reference time base by a specific interval for each modulo time base decoded;

decoding the time base increment of the compressed instance; and

determining the decoding time base of the said instance by adding the said decoded time base increment value to the reference time base.

4. A method of encoding the time base according to claim 1 where the local time base, time base increment and time base offsets are in units of milliseconds and the specific interval has a duration of 1000 milliseconds.

5. A method of claim 1, wherein a plurality of compressed bitstreams containing local time base information encoded therein are multiplexed, for multiplexing further comprising the steps of:

synchronizing the local time base of the individual compressed bitstream to a common time base by encoding and inserting the time base offset in the multiplexed bitstream;

examining each of the compressed bitstream for the next compressed instance to be placed into the multiplexed bitstream until all the compressed bitstreams have arrived at the modulo time base;

inserting a common modulo time base into the multiplex bitstream and skipping the modulo time base of the compressed bitstreams; and

repeating the last two steps until all compressed bit-streams are exhausted.

6. A method of multiplexing a plurality of compressed bitstreams as in claim 5 where the individual compressed bitstreams are themselves multiplexed bitstreams.

7. A method of encoding a local time base of an audio visual sequence in the compressed data comprising the steps of:

obtaining instances of the audio visual sequence by temporal sampling;

determining the local time base of the said instances to be encoded into compressed data;

encoding the said instances in one of two methods comprising of a first method of compression without reference to any future instances and a second method of compression which refers to the future reconstructed instance;

encoding the said local time base in two parts, comprising of a modulo time base that marks the occurrence of a set of time reference spaced evenly at a specific interval on the local time base and a time base increment;

encoding the time base increment for the instance compressed using the first compression method as an absolute value relative to the said evenly spaced time reference;

encoding the time base increment for the instance compressed using the second compression method as a relative value to the local time base of the previously compressed instance using the said first method;

inserting the modulo time base into the compressed data whenever the specific interval have elapsed; and

inserting the time base increment within the compressed data of the said instances of the audio visual sequence.

8. A method of decoding a local time base of an audio visual sequence from the time base of the compressed data encoded according to claim 7 comprising the steps of initializing the reference time base taking into account of the time base offset;

incrementing the reference time base by a specific interval for each modulo time base decoded;

decoding the time base increment of the compressed instance; and

determining the time base increment to be one of two types, absolute or relative, depending on the first or second compression method used in the encoding of the instance, respectively;

determining the decoding time base of the said instance by adding the said decoded time base increment value to the reference time base if the time base is of the first type; and

determining the decoding time base of the said instance by adding the said decoded time base increment value to the decoding time base of the previous instance encoded using the first compression method, if the time base increment is of the second type.

9. A method for decoding a local time base of a bidirectional-predictive-coded video object plane from compressed data, comprising:

decoding a modulo time base of the bidirectional-predictive-coded video object plane that is encoded by referring to one of an intra-coded video object plane and a predictive-coded video object plane after the bidirectional-predictive-coded video object plane in a display order;

obtaining a reference time base of the bidirectional-predictive-coded video object plane by combining a reference time base for one of an intra-coded video object plane and a predictive-coded video object plane before the bidirectional-predictive-coded video object plane in a display order and the decoded modulo time base of the bidirectional-predictive-coded video object plane;

decoding a time base increment of the bidirectional-predictive-coded video object plane; and

combining the decoded time base increment and the reference time base of the bidirectional-predictive-coded video object plane to determine the local time base of the bidirectional-predictive video object plane.

10. A method for decoding a local time base of a predetermined object plane from compressed data, comprising:

decoding a modulo time base of the predetermined object plane that is encoded by referring to a selected object plane after the predetermined object plane in an arrangement order;

obtaining a reference time base of the predetermined object plane by combining a reference time base for a selected object plane before the predetermined object plane in an arrangement order and the decoded modulo time base of the predetermined object plane;

decoding a time base increment of the predetermined object plane; and

combining the decoded time base increment and the reference time base of the predetermined object plane to determine the local time base of the predetermined object plane.

11. The method of claim 10, wherein the predetermined object plane comprises a video object plane.

12. The method of claim 10, wherein the predetermined object plane comprises a bidirectional-predictive-coded video object plane, and the selected object plane comprises at least one of an intra-coded video object plane and a predictive-coded video object plane.

13. A method for determining a local time base from compressed data obtained by encoding video object planes, the local time base being encoded in two parts, a first part being a modulo time base that indicates whether a reference time base is to be added by a predetermined time interval, and a second part being a time base increment, the method comprising:

obtaining the reference time base of a video object plane which is one of a last intra-coded video object plane and a last predictive coded video object plane, prior to a bidirectional-predictive-coded video object plane in a display order;

decoding the modulo time base of the bidirectional predictive video object plane;

decoding the time base increment of the bidirectional-predictive-coded video object plane; and

determining the local time base of the bidirectional-predictive-coded video object plane by combining the decoded modulo time base, the decoded time base increment, and the reference time base.