A plurality of bit streams are seamlessly spliced. Separate decoders decode each bit stream. A controller selects the decoded pictures according to a re-encoding range in the vicinity of a splicing point of the bit streams. pictures presenting a reordering of the streams are excluded in the selection of the decoded pictures. An encoder re-encodes the pictures within the re-encoding range. When it is determined that crossover motion compensation exists between pictures of different streams, the controller changes the motion prediction direction of the problematic picture. The controller changes a motion prediction picture type of a picture which is improperly motion predicted with reference to another stream. A quantization characteristic or motion vectors for the new picture type are generated by the controller. The controller effects the encoding in accordance with a target amount of bits to prevent a breakdown of a buffer and a discontinuation of an amount of data occupancy thereof. A multiplexer multiplexes the original streams with the re-encoded stream to produce a seamless bit stream.

Patent
   6567471
Priority
Jul 25 1997
Filed
Mar 25 1999
Issued
May 20 2003
Expiry
Jul 27 2018
Assg.orig
Entity
Large
47
9
all paid
8. A coding method for coding source video data, comprising the steps of:
encoding said source video data to output a first encoded stream;
decoding said first encoded stream encoded in a first encoding step to generate decoded video data;
encoding said decoded video data to generate a re-encoded stream in accordance with a value of a coding parameter of said first encoded stream;
controlling second-mentioned encoding step with respect to said coding parameter so as to prevent overflow and underflow of a vbv buffer resulting from encoding said decoded video data,
wherein said encoding control step calculates a target amount of bits for re-encoding said decoded video data in a second encoding step so as to prevent a discontinuity of a locus of a data occupancy of said vbv buffer, and said second encoding step encodes said decoded video data in accordance with said target amount of bits supplied from said encoding control step.
7. coding apparatus for coding source video data, comprising:
first encoding means for encoding said source video data to generate a first encoded stream;
decoding means for decoding said first encoded stream encoded by said first encoding means to generate decoded video data;
second encoding means for encoding said decoded video data to generate a re-encoded stream in accordance with a value of a coding parameter of said first encoded stream;
encoding control means for controlling said second encoding means with respect to said coding parameter so as to prevent overflow and underflow of a vbv buffer resulting from encoding said decoded video data,
wherein said encoding control means calculates a target amount of bits for re-encoding said decoded video data by said second encoding means so as to prevent a discontinuity of a locus of a data occupancy of said vbv buffer, and said second encoding means encodes said decoded video data in accordance with said target amount of bits supplied from said encoding control means.
9. coding apparatus for coding source video data,
wherein a decoding-side for decoding a spliced stream includes a video buffer verifier (vbv) comprising:
a first encoder for encoding said source video data to generate a first encoded stream;
a decoder for decoding said first encoded stream encoded by said first encoder to generate decoded video data;
a second encoder for encoding said decoded video data to generate a re-encoded stream in accordance with a value of a coding parameter of said first encoded stream;
an encoding controller for controlling said second encoder with respect to said coding parameter so as to prevent overflow and underflow of said vbv buffer resulting from encoding said decoded video data,
wherein said encoding controller calculates a target amount o f bits for re-encoding said decoded video data by said second encoder so as to prevent a discontinuity of a locus of a data occupancy of said vbv buffer, and said second encoder encode s said decoded video data in accordance with said target amount of bits supplied from said encoding controller.
4. A splicing method for splicing a plurality of source encoded streams to produce spliced streams, comprising the steps of:
setting splicing points for said plurality of source encoded streams;
decoding pictures in a region of said splicing points of said plurality of source encoded streams and generating the decoded video data,
wherein a decoding-side which decodes spliced streams and includes a video buffer verifier (vbv) characterized by a data occupancy;
re-encoding said decoded video data to generate re-encoded streams;
performing switching between said source encoded streams and said re-encoded streams to effect output so as to produce said spliced streams; and
controlling said re-encoding step and said spliced streams producing step so as to prevent a discontinuity of said spliced streams when said spliced streams are decoded,
wherein said splice control step controls said re-encoding step so as to approximate a locus of said data occupancy of said vbv buffer corresponding to said re-encoded streams to said locus of said data occupancy of said vbv buffer which is an original locus of said source coded streams.
6. A splicing apparatus for splicing a plurality of source encoded streams, comprising:
a splicing-pointer for setting a splicing point for said plurality of source encoded streams;
a decoder for decoding pictures included in said plurality of source encoded streams in a region of said splicing point to generate decoded video data;
wherein a decoding-side decodes spliced streams and includes a video buffer verifier (vbv) characterized by a data occupancy;
a re-encoder for re-encoding said decoded video data to generate re-encoded streams;
a spliced-stream switcher for switching between said source encoded streams and said re-encoded streams to produce spliced streams; and
a splice controller for controlling said re-encoder and said spliced-stream switcher so as to prevent a discontinuity of said spliced streams when said spliced streams are decoded,
wherein said splice controller controls said re-encoder so as to approximate a locus of said data occupancy of said vbv buffer corresponding to said re-encoded streams to said locus of said data occupancy in said vbv buffer which is an original locus of said source encoded streams.
2. A splicing apparatus for splicing a plurality of source encoded streams, comprising:
splicing-point setting means for setting a splicing point for said plurality of source encoded streams;
decoding means for decoding pictures included in said plurality of source encoded streams in a region of said splicing point to generate decoded video data,
wherein a decoding-side decodes spliced streams and includes a video buffer verifier (vbv) characterized by a data occupancy;
re-encoding means for re-encoding said decoded video data to generate re-encoded streams;
spliced-stream producing means for switching between said source encoded streams and said re-encoded streams to produce said spliced streams; and
splice control means for controlling said re-encoding means and said spliced-stream producing means so as to prevent a discontinuity of said spliced streams when said spliced streams are decoded,
wherein said splice control means controls said re-encoding means so as to approximate a locus of said data occupancy of said vbv buffer corresponding to said re-encoded streams to said locus of said data occupancy in said vbv buffer which is an original locus of said source encoded streams.
5. A splicing apparatus for splicing a plurality of source encoded streams, comprising:
a splicing-pointer for setting a splicing point for said plurality of source encoded streams;
a decoder for decoding pictures included in said plurality of source encoded streams in a region of said splicing point to generate decoded video data;
wherein a decoding-side decodes spliced streams and includes a video buffer verifier (vbv) characterized by a data occupancy;
a re-encoder for re-encoding said decoded video data to generate re-encoded streams;
a spliced-stream switcher for switching between said source encoded streams and said re-encoded streams to produce spliced streams; and
a splice controller for controlling said re-encoder and said spliced-stream switcher so as to prevent a discontinuity of said spliced streams when said spliced streams are decoded,
wherein said splice controller calculates a target amount of bits for re-encoding said decoded video data in re-encoding by said re-encoder so as to prevent a discontinuity of a locus of said data occupancy of said vbv buffer at a switching point between said source encoded streams and said re-encoded streams or said splicing points of said re-encoded streams, and said re-encoder re-encodes said decoded video data in accordance with said target amount of bits supplied from said splice controller.
3. A splicing method for splicing a plurality of source encoded streams to produce spliced streams, comprising the steps of:
setting splicing points for said plurality of source encoded streams;
decoding pictures in a region of said splicing points of said plurality of source encoded streams and generating decoded video data,
wherein a decoding-side which decodes spliced streams and includes a video buffer verifier (vbv) characterized by a data occupancy;
re-encoding said decoded video data to generate re-encoded streams;
performing switching between said source encoded streams and said re-encoded streams to effect output so as to produce said spliced streams; and
controlling said re-encoding and said spliced streams producing so as to prevent a discontinuity of said spliced streams when said spliced streams are decoded,
wherein said splice control step calculates a target amount of bits for re-encoding said decoded video data in re-encoding which is performed in said re-encoding step so as to prevent a discontinuity of a locus of said data occupancy of said vbv buffer at a switching point between said source encoded streams and said re-encoded streams or the splicing point of said re-encoded streams, and said re-encoding step encodes said source encoded streams in accordance with said target amount of bits generated in said splice control step.
1. A splicing apparatus for splicing a plurality of source encoded streams, comprising:
splicing-point setting means for setting a splicing point for said plurality of source encoded streams;
decoding means for decoding pictures included in said plurality of source encoded streams in a region of said splicing point to generate decoded video data,
wherein a decoding-side decodes spliced streams and includes a video buffer verifier (vbv) characterized by a data occupancy;
re-encoding means for re-encoding said decoded video data to generate re-encoded streams;
spliced-stream producing means for switching between said source encoded streams and said re-encoded streams to produce said spliced streams; and
splice control means for controlling said re-encoding means and said spliced-stream producing means so as to prevent a discontinuity of said spliced streams when said spliced streams are decoded,
wherein said splice control means calculates a target amount of bits for re-encoding said decoded video data in re-encoding by said re-encoding means so as to prevent a discontinuity of a locus of said data occupancy of said vbv buffer at a switching point between said source encoded streams and said re-encoded streams or the splicing points of said re-encoded streams, and said re-encoding means encodes said decoded video data in accordance with said target amount of bits supplied from said splice control means.

This is a continuation of co-pending International Application PCT/JP98/03332 having an international filing date of Jul. 27, 1998.

1. Field of the Invention

The present invention relates to an editing system, method and apparatus for editing images and, more particularly, an editing system, method and apparatus for seamlessly splicing a plurality of bit streams of video data.

2. Related Art

Recording/reproducing systems have recently been introduced which record/reproduce high quality audio/video data utilizing compression schemes. High quality recording/reproducing systems compression-encode/decode the audio/video data utilizing the MPEG (Moving Picture Experts Group) standard. One example of such a system is the DVD (Digital Versatile Disk or Digital Video Disk), which provides a powerful means by which unprecedented quantities of high quality audio/video are compressed on an optical disk.

FIG. 1 illustrates the general recording/reproducing system. The video encoder 111 of the encoding-side apparatus 110 encodes input video data Dv in accordance with the MPEG standard to thereby produce a video elementary stream (video ES). The packetizer 112 packetizes the video elementary stream into a video packetized elementary stream (video PES) comprising access units; each access unit representing a picture in a group of pictures making up a portion of the video program. The audio encoder 113 of the encoding-side apparatus encodes input audio data DA to thereby produce an audio elementary stream (audio ES) The packetizer 114 formats the audio elementary stream into an audio packetized elementary stream (audio PES) comprising access PATENT units; each access unit represent decodable segment of an audio bit stream. The transport stream multiplexer 115 multiplexes the audio and video packetized elementary streams to thereby produce a transport stream packet. A Video Buffer Verifier (VBV) buffer (not shown) stores/retrieves the multiplexed streams at a variable target rate which is controlled in accordance with the number of bits to be encoded and the capacity of the VBV buffer. An illustration of the Video Buffer Verifier is provided with reference to FIG. 2.

The decoding-side apparatus 120 of FIG. 1 stores in a decoding-side Video Buffer Verifier (VBV) buffer (not shown) the received transport stream which is transmitted via the transmission medium 116. The transport stream demultiplexer 121 demultiplexes the received transport stream fetched from the decoding buffer at a timing determined by a decoding time stamp (DTS) to thereby reproduce the video packetized elementary stream (video PES) and the audio packetized elementary stream (audio PES). The video packetized elementary stream is depacketized by depacketizer 122 and decoded by video decoder 123 thereby reproducing the video data Dv. The audio packetized elementary stream is depacketized by depacketizer 124 and decoded by audio decoder 125 thereby reproducing the audio data DA. For DVD applications, the transport stream multiplexer 115 and the transport stream demultiplexer 121 are respectively replaced with a program stream multiplexer and demultiplexer which DVD format/unformat the encoded bit streams.

In the recording/reproducing system of FIG. 1, it is desirable to seamlessly splice a plurality of bit streams by concentrating at the transport level two or more different elementary streams representing the merger of different video programs. In digital broadcasting, for example, editors at a broadcasting station splice a plurality of bit streams from different video sources such as, for example, live video feeds received from local stations for generating a spliced broadcast video program. In DVD applications, the director splices movie scenes to be recorded on the DVD optical disk. In another DVD application, the DVD decoder splices multiple bit streams reproduced from the DVD optical disk in response to user-entered actions which is particularly useful for generating alternate scenes for interactive movies and video games.

There are, however, unforeseen difficulties to splicing a plurality of bit streams using the MPEG compression standard. In order to illuminate the problem, a closer look at MPEG is warranted. In summary, the MPEG standard implements a compression process which includes motion-compensated predictive coding in conjunction with adaptive Discrete Cosine Transform (DCT) quantization. The motion-compensated predictive coding predicts motion in each image frame/field using both unidirectional and bidirectional motion prediction. The DCT quantization adaptively compresses each frame/field in accordance with the motion-compensated prediction. The term "frames" hereinafter refers to-pictures in general including frames as well as fields.

As illustrated in FIG. 3(a), motion-compensated prediction of the MPEG compression standard classifies the frames into one of three types: intracoded-frames (I-frames), predictively coded frames (P-frames) and bi-directionally coded frames (B-frames). MPEG establishes the I-frames as the reference by which the B- and P-frames are encoded and, thus, preserves the I-frames as complete frames. The I-frames are considered "intra-coded" since they proceed as complete frames, having bypassed the motion-compensated prediction, to the DCT quantization whereupon each I-frame is compression encoded with reference only to itself. P-frames, which rely on forward temporal prediction, are coded using the previous I- or P-frame. B-frames are coded using bi-directional (forward and/or backward) motion compensated predictive encoding using the two adjacent I- and/or P-frames. B- and P-frames are considered "inter-coded" since they are motion-prediction encoded with reference to other frames. FIG. 7 illustrates an example of the direction of prediction for each I, B and P-frame in a group of pictures (GOP) as indicated by the arrows in the figure.

In accordance with the MPEG standard, frames are arranged-in ordered groups of pictures (GOP), each group of pictures comprising a closed set of I-, B- and P-frames which are encoded with reference to only those frames within that group. FIG. 3(a) illustrates the natural presentation order (1 to 15) of the GOP in which the pictures are naturally presented to the viewer. Since the B- and P-frames within the GOP are encoded with reference to other frames, the MPEG standard dictates that the natural presentation order shown in FIG. 3(a) be rearranged into the decoding order shown in FIG. 3(b) in which the frames are to be decoded and transmitted in the coded order shown in FIG. 3(c). With this arrangement, the frames necessary for decoding other frames are-first.decoded to provide the basis upon which the following inter-coded frames are decoded. For example, an I-frame which forms the reference by which the following frames in the GOP are motion-compensation predicted is positioned first in the decoding order. Once decoded, the pictures are rearranged in their natural presentation order for display to the viewer.

Motion-compensated predictive coding divides each I-, B- and P-frame into 8×8 pel macroblocks. The motion vectors for a present-frame are motion-compensation predicted with reference to the motion vectors of another frame which is selected in accordance with the direction of prediction of the type of frame (e.g., I-, B- or P-frame). For example, P-frame macroblocks are motion-predicted with reference to the macroblocks in a previous I or P-frame; B-frame macroblocks are motion-predicted with reference to the previous/successive I- and/or P-frames. The I-frames, which are not inter-coded, bypass motion compensation and are directly DCT quantized.

The process for motion-predicting a current picture in a GOP is illustrated in FIGS. 4(a)-(e). The GOP are input in the natural presentation order shown in FIG. 4(a), rearranged in accordance with the decoding order shown in FIG. 4(b), motion-predicted utilizing two frame memories (FM1, FM2) as shown in FIGS. 4(c) and (d) and output in the form of the encoding stream (ES) shown in FIG. 4(e). For example, the I-frame (I3) of FIG. 4(b) is intra-coded and, therefore, output directly to the encoding stream (ES); the B-frame (B1) of FIG. 4(b) is motion predicted with reference to the I-frame (I3) stored in the first frame memory (FM1) of FIG. 4(c) and the P-frame (P) stored in the second frame memory (FM2) of FIG. 4(d); the P-frame (P6) of FIG. 4(b) is motion predicted with reference to the I-frame (I3) stored in the first frame memory (FM1) of FIG. 4(c). From the foregoing illustration, it is apparent that a minimum of two frame memories are needed for bi-directional motion prediction.

After the motion vectors are calculated, each macroblock is Discrete Cosine Transform (DCT) encoded. More particularly, the macroblocks are transformed from pixel domain to the DCT coefficient domain. Next, adaptive quantization is performed on each block of DCT coefficients in-accordance with a variable quantization step size. After adaptive quantization is applied to the DCT coefficients, the coefficients undergo further compression involving such techniques as differential coding, run-length coding or variable length coding. The encoded data is stored/retrieved to/from the Video Buffer Verifier (VBV) buffer at a controlled target bit rate in the form of a serial bit stream.

FIG. 2 illustrates a locus of the data occupancy of the VBV buffer wherein the bits (oordinate) of the I-, B- and P-frames are stored in the VBV buffer along a time axis (presentation time Tp-abscissa) at a transmission bit rate (inclination 131) and output from the VBV buffer as indicted by the vertical lines. The VBV buffer is considered a "virtual" buffer because it emulates the buffer on the decoding side. By controlling the amount of bits 132 of the VBV buffer on the encoding side, it can be assured that the appropriate amount of bits per decoding time stamp (DTS), i.e. target bit rate, is transmitted to the decoding side. This is important in MPEG where the number of bits for a particular frame varies depending upon the motion-prediction type. The I-frames in FIG. 2, for example, require four times the amount of storage time (VBV buffer delay) as the P-frames and twice the B-frames. For that matter, care must be taken that the varied amount of bits in a GOP does not cause an overflow when the number of bits exceeds the buffer capacity (upper-hatched line) or an underflow when the number of bits drops below a predetermined minimum number (lower-hatched line) which will sustain an efficient encoding/decoding process.

Referring to FIGS. 5A-C, the decoding process for decoding the transmitted group of pictures (GOP) is explained. The coded order shown in FIG. 5(a) is received by the decoding side apparatus 120 (FIG. 1) and stored in the decoding-side VBV buffer. The transport stream demultiplexer 121 demultiplexes the stream into the packetized elementary stream illustrated in FIG. 5(b). The GOP are decoded by fetching the compressed picture data from the decoding-side buffer at a timing determined by the decoding time stamp (DTS), de-compressing the fetched picture data and reconstructing each I-, B- and P-frame from the decompressed picture data. It will be appreciated that the I-frames are complete upon decompression. The B- and P-frames are reconstructed by motion estimating the previously decoded frames based on the decompressed motion vectors of the current B- or P-frame. Afterwards, the decoded frames are rearranged in their original presentation order for display as shown in FIG. 5(c).

When it is considered that the decoding-side apparatus requires relatively less hardware complexity than the encoding-side, the wisdom of the MPEG encoding/decoding scheme will be immediately recognized. To explain, the complex hardware necessary to perform motion prediction is not a part of the decoding-side apparatus since the decoder need only apply the motion vectors to the encoded-pictures. The high quality audio/video is, thus, generated by a high-end encoder for distribution enmasse to numerous, considerably less-complex (and less-expensive) decoders.

The motion decoding process is illustrated in FIGS. 6(a)-(d) wherein FIG. 6(a) shows the coded video elementary stream (ES) which is supplied to the decoder. A first frame memory (FM1) as illustrated in FIG. 6(b) stores a first previously-decoded picture for decoding the current picture. A second frame memory (FM2) as illustrated in FIG. 6(c) stores a second previously-decoded picture for decoding the current picture. For example, the decoded I-frame (I3) (first picture in the ES of FIG. 6(a)) is stored in the first frame memory (FM1) and the P-frame (previous ES) is stored in the second frame memory (FM2). In this example, the B-frame (B1) is decoded by motion estimating the frames in the frame memories (FM1, FM2) based on the motion vectors of B1. The decoded GOP are output in the presentation order illustrated in FIG. 6(d).

With the rudiments of the MPEG standard explained, the difficulties confronted when splicing coded streams will be better appreciated. In the conventional editing system for splicing bit streams, it is recognized that the bit streams must be decoded. This is because the prediction direction of the first stream may be inconsistent with that of the second. To explain, the selected direction of prediction (forward/backward) for the B-frames mutually effects the prediction direction of other B-frames and, for that matter, defines which frames are selected for the motion prediction throughout the GOP. When two coded bit streams are spliced arbitrarily, for example, the prediction direction for a frame in the first coded bit stream may be decoded with reference to a frame with an inconsistent prediction direction in the second coded bit stream. For this reason, motion estimation upon decoding in the area of the splicing point will result in reconstructing an incorrect picture. The error, referred to as a discontinuity, migrates to other frames in motion estimation, consequently effecting the motion estimation decoding of the GOP as a whole. This discontinuity manifests as visible macroblocks on the display when, for example, the channel of a digital television is changed.

In order to prevent discontinuity, it is suggested to decode the bit streams before splicing. When the bit streams are decoded, the frames thereof are not motion predicted, i.e., not encoded with reference to other frames and thus are not subject to the discontinuity of the foregoing method. However, the spliced bit stream must be re-encoded. Since MPEG coding is not a 100% reversible process, the signal quality is deteriorated when re-encoding is performed. The problem is compounded because the re-encoding process encodes a decoded signal, i.e., a degraded version of the original audio/video signal.

A splicing technique which addresses signal deterioration selectively decodes the bit streams at a splicing point. However, such a splicing technique produces unsatisfactory results. The first problem arises in the presentation order of the spliced stream which may be understood with reference to FIGS. 8(a)-(d) to 9(a)-(d). FIGS. 8(a)-(d) illustrate the ideal case where no problems arise in the presentation order of the spliced stream STsp. In this case, stream STA of FIG. 8(a) is spliced at the splicing point SPA with stream STB of FIG. 8(b) at the splicing point SPB. Thus, the spliced bit stream STsp of FIG. 8(c) presents the pictures of stream STA followed by the pictures of stream STB without problem.

FIGS. 9(a) to (d) illustrate the problem where the decoder rearranges the presentation order of the spliced bit stream. Stream STA of FIG. 9(a) is bit-spliced with stream STB of FIG. 9(b) at respective splicing positions (SPA, SPB). Unlike the ideal case, the decoder on the decoding-side rearranges the order of presentation of the frames of the spliced bit stream STsp, (FIG. 9(c)) such that, in this example, the last frame (P-frame) in bit stream STA is inserted at the third-picture position of stream STB. This appears visually as an arbitrary picture inserted in the video program.

The second problem, hereinafter termed "crossover", arises in motion estimation upon decoding of the spliced bit stream. In the ideal case illustrated in FIGS. 10(a), (b) the motion estimation reconstructs the pictures of stream STA of the spliced bit stream STsp, of FIG. 10(a) with reference to only those frames from that stream. This is indicated by the arrows in FIG. 10(b) which represent the motion estimation direction.

Likewise, stream STB is motion estimated with reference to only those pictures in that stream.

FIGS. 11(a) and (b) illustrate the problem of crossover motion estimation. For example, the P-frame in stream STA is based on frames in stream STB as illustrated by the hatched arrows labeled "NG" in FIG. 11(b). Thus, the P-frame in stream STB, is reconstructed from the wrong picture which appears visually as a distorted image. This problem is propagated through the GOP as shown in FIGS. 12(a), (b) when the incorrectly-estimated P-frame of stream STB is utilized by the decoder to motion estimate other frames. This results in a number of distorted pictures which are quite noticeable.

FIGS. 13(a) to 18(b) illustrate the third problem of underflow/overflow related to splicing bit streams. The ideal case is illustrated in FIGS. 13(a)-(d) wherein three streams (STA, STB, STC) are spliced at splicing points SPV and a buffer occupancy VOC. FIG. 13(a) illustrates the locus of the data occupancy of the video buffer verifier (VBV) buffer on the decoding side wherein I-, B- and P-frames are stored in the VBV buffer. FIG. 13(b) illustrates the spliced stream STsp, FIG. 13(c) the timing at which each of the pictures is generated after rearrangement and FIG. 13(c) the order of the pictures after the decoding operation. As will be appreciated from FIG. 13(a), the instant case does not present a problem of overflow (upper-hatched line) or underflow (lower-hatched line).

The problematic case is illustrated in FIGS. 14(a)-16(b). By themselves, bit streams STA, STB do not pose an overflow/underflow problem as will be appreciated from FIGS. 14(a), 15(a). However, when the bit streams STA, STB, are spliced as illustrated in FIGS. 16(a), (b) at a splicing point SPv an overflow/underflow condition occurs. The overflow condition which is illustrated in FIGS. 17(a), (b) occurs when bit stream STB continues to fill the VBV buffer to a point where the VBV buffer overflows as indicated at 141 in FIG. 17(a). The underflow case which is illustrated in FIGS. 18(a) and (b) occurs when stream STB does not thereafter fill the VBV buffer by a sufficient amount thereby resulting in an underflow 142 shown in FIG. 18(b). In the decoding-side apparatus (IRD), either an overflow or underflow of the VBV buffer consequently results in a failure in decoding pictures on the decoding-side. It is not atypical to see the effects of overflow/underflow manifesting as the skipping, freezing or interruption of the images.

Heretofore, there has been no solution for providing a seamlessly-spliced bit stream from a plurality of bit streams without the serious defects illustrated in the foregoing examples.

It is therefore an object of the present invention to provide a system for splicing bit streams;

It is another object of the present invention to provide a system for seamlessly splicing bit streams;

It is another object of the present invention to prevent signal deterioration in a system for splicing bit streams;

It is another object of the present invention to prevent degradation of image quality due to improper reordering of the pictures in the spliced bit stream;

It is another object of the present invention to prevent picture distortion due to improper motion-estimation and the propagation thereof;

It is another object of the present invention to prevent overflow/underflow in the video verifier buffer (VBV) buffer;

It is another object of the present invention to provide an editing system to-generate seamless bit streams on the fly from video feeds of various sources for broadcast by a broadcasting station;

It is another object of the present invention to provide a system for splicing bit streams in a DVD system;

It is another object of the present invention to provide a system for generating interactive movies by splicing a plurality of bit streams representing various portions of the movie;

It is another object of the present invention to provide a video game system for generating interactive video game scenes selected in accordance with user commands by splicing a plurality of bit streams representing alternative user-directed scenes of a video game; and

It is another object of the present invention to provide a system for encoding/decoding audio/video feeds spliced from a plurality of bit streams for on-line transmission.

According to the present invention, there is provided a system, method and apparatus for splicing a plurality of bit streams. The present invention inhibits a picture in the spliced bit stream which, upon decoding, would be out of sequence. In this manner, the present invention prevents an improper reordering of the spliced bit stream pictures on the decoding side.

In order to prevent deterioration in the image quality of the spliced bit stream, the present invention selectively reuses motion vector information fetched from the source coded streams for use in the re-encoding process. The new motion vectors are supplied to the motion compensation portion of the re-encoder in place of the original motion vectors. In order to prevent the improper prediction of a picture from an incorrect bit stream source, the present invention sets the direction of prediction to a picture which is positioned adjacent the splicing point thereby preventing degradation in image quality. In addition, the present invention has a capability of changing the picture type of a picture in the vicinity of the splicing point in order to prevent erroneous motion prediction from pictures from another bit stream source.

It is recognized in the present invention that the overflow/underflow condition occurs owing to an improper selection of the target bit rate for the spliced bit stream. So as to prevent overflow/underflow of the video buffer verifier (VBV) buffer, the target amount of bits is calculated anew for the spliced bit stream. The target amount of bits is calculated by reference to a quantizing characteristic produced in a previous coding process which may be retrieved from the source coded streams. In the alternative, the target amount is approximated. The plural bit streams are decoded in the region of the splicing point(s) and re-encoded in accordance with the new target bit rate.

With the present invention, seamlessly-spliced bit streams are provided without signal deterioration arising from improper reordering of the frames, picture distortion due to improper motion estimation or a breakdown in the video verifier (VBV) buffer due to improper selection of the target bit rate. It will be appreciated that the present invention is applicable to a wide range of applications including, for example, an editing system for generating seamless bit streams on the fly from video-feeds of various sources for broadcast by a broadcasting station, a DVD system, a system for providing interactive movies, a video game system for generating alternative user-directed scenes of a video game or a system for encoding/decoding audio/video feeds for on-line transmission.

FIG. 1 is a block diagram of a recording/reproducing system;

FIG. 2 illustrates the operation of a VBV buffer;

FIGS. 3(a)-(c) illustrate the operation of an encoder;

FIGS. 4(a)-(e) illustrate the operation of the frame memories of the encoder;

FIGS. 5(a)-(c) illustrate the operation of a decoder;

FIGS. 6(a)-(d) illustrate the operation of the frame memories of the decoder;

FIG. 7 illustrates the prediction direction for encoding/decoding;

FIGS. 8(a)-(d) illustrate the bit splicing operation;

FIGS. 9(a)-(d) illustrate the reordering of the spliced bit stream;

FIGS. 10(a), (b) illustrate motion estimation of the spliced bit stream;

FIGS. 11(a)-12(b) illustrate motion compensation crossover in the spliced bit stream;

FIGS. 13(a)-(d) illustrate the operation of the video buffer verifier;

FIGS. 14(a), (b) illustrate the operation of the video buffer verifier storing stream STA;

FIGS. 15(a), (b) illustrate the operation of the video buffer verifier storing stream STB;

FIGS. 16(a), (b) illustrate the video buffer verifier storing the spliced bit stream;

FIGS. 17(a), (b) illustrate an overflow of the video buffer verifier;

FIGS. 18(a), (b) illustrate an underflow of the video buffer verifier;

FIG. 19 illustrates the present invention;

FIG. 20 illustrates the block diagram of the encoder and decoder of FIG. 19;

FIGS. 21(a), (b) illustrate the re-encoding operation of the present invention;

FIGS. 22(a)-(d) illustrate the operation of decoding the bit streams according to the present invention;

FIGS. 23(a), (b) illustrate the splicing operation of the present invention;

FIGS. 24(a), (b) illustrate streams STA, STB for splicing in accordance with the present invention;

FIGS. 25(a)-(d) illustrate the decoding operation in accordance with the present invention;

FIGS. 26(a), (b) illustrate the spliced bit stream in accordance with the present invention;

FIGS. 27(a), (b) illustrate an underflow of the video buffer verifier;

FIGS. 28(a), (b) illustrate the prevention of underflow in accordance with the present invention;

FIGS. 29(a), (b) illustrate an overflow of the video buffer verifier;

FIGS. 30(a), (b) illustrate the prevention of overflow in accordance with the present invention;

FIG. 31 presents a flow diagram of the present invention;

FIG. 32 illustrates a continuation of the flow diagram of FIG. 31.

FIG. 19 illustrates the present invention. It will be appreciated from the figure that the present invention receives a plurality of bit streams, in this case streams A and B (STA and STB), which are selectively spliced in accordance with the bit splicing technique hereinafter described. The present invention is applicable to both the encoding and decoding sides and, as such, may optionally include encoders 1A and 1B for respectively encoding video data VDA and VDB which produce the bit streams STA and STB. In any case, it is preferable that the output spliced bit stream STsp of the present invention complies with the MPEG standard and is of course acceptable for any MPEG encoding/decoding system. It will be appreciated that the present invention is transparent to the end-viewer and, for that reason, is marketably attractive since the decoder may not need to be upgraded to receive the spliced bit streams of the present invention. The present invention is not limited to splicing one particular type of bit stream, but may of course be applied to any type of bit stream including, for example, the elementary stream, the packetized elementary stream and the transport stream.

In more detail, FIG. 19 illustrates that the streams STA and STB are input to a buffer memory 10, a stream counter 11 and a stream analyzing portion 12. The stream counter 11 counts the number of bits in each of the streams STA and STB whilst the stream analyzing portion 12 analyzes the syntax of each of the streams. A splice controller 13 controls the bit splicing operation of the present invention as will be described in more detail. MPEG decoders 14A and 14B decode the streams STA and STB retrieved from the buffer memory 10 which output respective base-band video data to a switch 15. At the control of the splice controller 13, the switch 15 outputs either stream STA or STB to an MPEG encoder 16. The MPEG encoder 16, at the control of the splice controller 13, encodes the video base-band data selected by the switch 15 to thereby output a re-encoded bit stream STRE. A switch 17, as controlled by the splice controller 13, selectively outputs either the bit streams STA, STB retrieved from the buffer memory 10 or the re-encoded bit stream STRE to thereby output the spliced bit stream STsp.

The operation of the present invention shown in FIG. 19 will now be described. The stream counter 11 counts the number of bits of each of the received streams STA and STB and supplies the count value to the splice controller 13. The number of bits of the streams is counted because the locus of the data occupancy of the video buffer verifier needs to be controlled to prevent overflow/underflow. The stream analyzing portion 12 analyzes the syntax of each of the streams to fetch appropriate information from the layers of the bit streams including the sequence layer, the GOP layer, the picture layer and the macroblock layer. For example, encoded information such as the picture type (I, B or P), motion vectors, quantizing steps and quantizing matrices are retrieved by the stream analyzing portion.

The splice controller 13, based on the count value from the stream counter 11 and the information from the stream analyzing portion 12, sets a re-encoding range for each bit stream in accordance with the range parameters n0 and m0. Likewise, the splicing point(s) are set in accordance with the splice point parameter p0(s) The splice controller 13 controls the timing of the switch 15 to select the appropriate bit stream STA or STB to be sent to the MPEG encoder 16 in accordance with the splicing point parameter p0 and the range parameters n0, m0. The phase and timing of the bit streams are controlled by the splice controller to coincide at the predetermined splicing point(s). The splice controller 13 controls the switch 17 to select the bit streams STA and STB normally. The re-encoded bit stream STRE produced by the MPEG encoder 16 is selected during the re-encoding range in accordance with the parameters n0, m0 and p0.

FIG. 20 illustrates in more detail the MPEG decoding/encoding section of-the.present invention wherein reference numeral 14 generally indicates the MPEG decoders 14A, B and reference numeral 16 generally indicates the MPEG encoder 16 shown in the previous figure. The decoding section 14 of the figure essentially performs MPEG decoding utilizing a decompression section for decompressing the input stream ST including a variable-length decoding circuit (VLD) 21, an inverse quantization circuit (IQ) 22 and an inverse discrete cosine transform circuit (IDCT) 23. The motion estimation section of the decoding section 14 includes an addition circuit 24 for adding the decompressed bit-stream motion prediction coefficients to the motion estimation coefficients produced in the motion estimation section of the decoder. A switch 25 alternates between selecting the decompressed data corresponding to the I-frames, which bypass motion estimation, and the motion estimated data output from the addition circuit 24. The motion estimation section performs motion estimation utilizing frame memories (FM1, FM2) 26, 27 and a motion compensation section (MC) in accordance with the operation described with reference to FIGS. 6(a)-(d).

The encoding section 16 shown in FIG. 20 encodes the decoded video data output from the MPEG decoders 14A and 14B in accordance with the operations of the splice controller 13. An encoder's previous processing circuit 30 preprocesses the decoded video data by rearranging the pictures of the decoded video date in accordance with the bidirectional predictive coding process, forms pixel macroblocks and calculates the difficulty in coding each picture. In the preferred embodiment, the encoder's previous processing circuit 30 forms 16×16 pixel macroblocks. The encoding section 16 further incorporates a subtraction circuit 31 for subtracting a motion prediction error from the input decoded video data, a switch 32 for bypassing the motion-compensated prediction process in the case of I-frames and a compression/motion-compensated prediction section. The compression portion includes a discrete cosine transform circuit (DCT) 33, a quantizing circuit (Q) 34 and a variable-length coding circuit (VLC) 35. The quantizing circuit 34 of the present invention is controlled by the splice controller 13.

The motion-compensated prediction portion predicts the motion within the B- and P-frames of the input decoded video data. In more detail, the compressed bit stream is decompressed by application to an inverse quantizing circuit (IQ) 36 followed by an inverse discrete cosine transform circuit (IDCT) 37. The decompressed bit stream is added by the addition circuit 38 to the motion-compensated version of the picture in order to reconstitute the current frame. The frame memories FM1, FM2 (39, 40) store the appropriate reconstructed frames at the control of the motion detection circuit 42 in accordance with the type of predictive coding (B- or P-frame encoding). The motion compensation circuit 41 performs motion compensation in accordance with the frame(s) stored in the frame memories (FM1, FM2) 39, 40 based on the motion vectors provided by the motion detection circuit 42. The motion compensated picture, which is essentially a prediction of the current frame, is subtracted from the actual current frame by the subtraction circuit 31. It will be appreciated that the output of the subtraction circuit 31 is essentially an error result representing the difference between the actual frame and the prediction. An encode controller 43 provides substitute motion vectors and controls a switch 44 in order to select between the motion vectors determined by the motion detection circuit 42 and the substitute motion vectors.

The operation of the decoding/encoding section shown in FIG. 20 will now be described. The decoding section 14 decodes the input stream ST preferably in accordance with the MPEG standard. The encoder's previous processing circuit 30 rearranges the pictures for encoding in accordance with the picture type information extracted by the stream analyzing circuit 12 and forms.picture data into macroblocks. The rearranged pictures are forwarded to the encoding section 16 of the figure for encoding.

The splice controller 13 forwards the encoded information, more particularly the motion vectors, which are extracted by the stream analyzing circuit to the encode controller 43. When it is determined to reuse the substitute motion vectors, the encode controller 43 causes the switch 44 to select the motion vectors supplied thereto. At other times, the encode controller 43 causes the switch 44 to select the motion vectors produced by the motion detection circuit 42. The encode controller 43 controls the frame memories (FM1, FM2) 39, 40 to store the appropriate pictures required to produce the predictive image data based on the substitute motion vectors and in accordance with the picture type of the current picture to be encoded. In addition, the encode controller 43 controls the quantization step size of the quantizing circuit 34 and the inverse quantization circuit 36 to accommodate the motion vectors in accordance with the target bit rate supplied by the splice controller 13.

The encode controller 43, moreover, controls the variable-length coding circuit 35. When it is determined that an amount of generated bits of the variable-length coding circuit 35 is insufficiently large with respect to the target amount of bits supplied by the splice controller 13, which forewarns of an underflow in the VBV buffer, the encode controller 43 adds dummy data to the variable-length coding-circuit 35 in order to account for the shortage with respect to the target amount of bits. Conversely, the encode controller 43 performs a skipped macroblock process (ISO/IEC 13818-27.6.6) which interrupts the coding process in terms of macroblock units when it is determined that the variable-length coding circuit 35 generates an amount of bits that is relatively larger than the target amount of bits which warns of an overflow.

An example of the control of the decoding/encoding section according to the present invention will now be described with reference to FIGS. 21(a) to 26(b). FIGS. 21(a), (b) illustrate the process of selecting the video data to be re-encoded (also referred to as "presentation video data") representing those portions of the bit streams STA (FIG. 21(a)) and STB (FIG. 21(b)) decoded respectively by the decoders 14A and 14B. In summary, when the splicing point as determined by the parameter p0 is set, the pictures comprising the presentation video data are selected to include those pictures within the re-encoding ranges as defined by the parameters n0 and m0.

A picture at the splicing point corresponding to stream STA is expressed as An-P0, wherein n is an integer and p0 is the splicing point parameter. Following this convention, pictures which are future to the picture at the splicing point are expressed as A(n-P0)+1, A(n-P0)+2, A(n-P0)+3, A(n-P0)+4 . . . A(n-P0)+n0, wherein n0 is the range parameter defining the range of the presentation video data corresponding to bit stream STA. Conversely, pictures more previous than the picture An-P0 at the splicing point are expressed as A(n-P0)-1, A(n-P0)-2, A(n-P0)-3, A(n-P0)-4 and so on. Likewise, the presentation video data corresponding to the stream STB at the splicing point is expressed as B(m-P) and the pictures in the re-encoding range defined by the parameter m0 are expressed as B(m-P0)+1, B(m-P0)+2, B(m-P0)+3, B(m-P0)+4 . . . B(m-P0)-1, B(m-P0)-2, B(m-P0)-3, B(m-P0)-4 . . . B(m-P0)-m0. As illustrated in FIGS. 21(a) and (b), the range of pictures in each respective bit stream STA, STB are indicated by the ranges for re-encoding (n0, m0). In other words, the re-encoding ranges include the pictures from picture A(n-P0)+n0 to picture A(n-P0) and pictures from picture B(m-P0) to picture B(m-P0)-m0.

With the present invention, the problem that the decoder on the decoding-side presents the pictures in the improper order is prevented. Each decoder 14A, B respectively decodes stream STA, STB thereby providing the decoded pictures A and B shown in FIGS. 22(b), (c). The splice controller 13 selects the re-encoding pictures REPA, REPB from pictures A, B by operation of switch 15. Since the streams STA, STB are decoded by two separate decoders, each set of pictures A, B are not cross-referenced and, therefore, not reordered upon decoding. In other words, the pictures which are incorrectly inserted into the wrong stream upon decoding are excluded by the splice controller 13. As shown in FIGS. 25(c), (d), for example, the B-pictures B(m-P0)+2, B(m-P0)+1 are excluded from the decoded pictures. Thus, the present invention provides a seamlessly-spliced stream which, upon decoding by the decoding-side decoder, arranges the pictures in the correct order of presentation as shown in FIG. 23(a).

FIGS. 23(a), (b) illustrate the solution to the problem of crossover motion compensation. In accordance with the present invention, the splice controller 13 controls the encoder 16 to change a direction of prediction of those pictures which improperly reference pictures in another stream. This occurs, as discussed with reference to FIGS. 11(a), (b), because a B-picture in stream STB, for example, is originally encoded by the encoder 1B with reference to the P-picture in the same stream. Since the P-picture occurs before the splicing point, however, the B-picture of stream STB now improperly refers to a picture in stream STA. In order to resolve this problem, the splice controller 13 according to the present invention changes the prediction direction.

FIGS. 24(a)-(b) illustrate an example of changing the picture type in accordance with the present invention to prevent an incorrect motion estimation of a particular picture in the region of the splicing point. FIG. 24(a) shows the presentation video data corresponding to stream STA and FIG. 24(b) shows the presentation video data corresponding to stream STB. The encoded stream STAis decoded by the decoder 14A. (FIG. 19) resulting in the decoded pictures shown in FIG. 25(b). Similarly, the encoded stream STB of FIG. 25(d) is decoded by the decoder 14B (FIG. 19) resulting in the decoded pictures of Fig. 25(c). In this example, FIG. 25(b) shows that the B-picture at the splicing point is motion predicted with reference to the following B-frame which occurs after the re-encoding region REPA. If this situation is left uncorrected, the B-picture at the splicing point will be motion estimated upon decoding with reference to the B-picture in the wrong bit stream, i.e., bit stream STB. Similarly, the P-picture at the splicing point of the bit stream STB shown in FIG. 25(c) is motion encoded on the basis of a P-picture occurring outside the re-encoding region REPB. This motion estimation error causes the macroblocks in the frame to be seen and, when compounded by propagation of the error throughout the group of pictures, becomes quite noticeable.

The splice controller 13 in accordance with the present invention changes the picture type of the problematic pictures of the foregoing example. As illustrated in FIGS. 26(a), (b), the B-picture of stream STA at the splicing point An-P0 is changed to a P-picture which is motion estimated on the basis of the previous P-picture which is within the re-encoding range of that stream STA. The P-frame of the stream STB is changed to an I-picture which is not motion estimated. It will be appreciated that the B-pictures (B(m-P0)+2 and B(m-P0)+1) are discarded as shown in FIG. 25(c) when the P-picture is changed to an I-picture and thus do not exist in the picture stream after the re-encoding process is performed. The new picture type may require new prediction direction data and motion vectors. In at least one embodiment, the splice controller 13 provides the encode controller 43 with the encoding information such as the prediction direction and the motion vectors of a previously-coded picture. However, the present invention may also provide new motion prediction data using other techniques such as reconstructing the new picture entirely.

Referring to FIGS. 27(a) to 30(b), a method for calculating a new target amount of bits for image data in a re-encoding range to prevent underflow/overflow in the VBV buffer according to the present invention will now be described. In the figures, TRE represents the re-encode control time, OSTA represents the original stream A and STRE' represents the stream which is re-encoded resulting in an underflow condition. OSTB represents original stream B, SPVBV represents a splicing point in the VBV buffer and SP represents a splicing point of the streams.

FIGS. 27(a), (b), illustrate the underflow condition. As shown in FIG. 27(a), the locus of the VBV buffer for the stream STRE' before the splicing point SP corresponds to stream A (STA). After the splicing point SP, the locus corresponds to stream B (STB). Since the level of data occupancy of the VBV buffer for stream STA at the splicing point is different from the level of data occupancy of the VBV buffer for stream STB, the data occupancy of the VBV buffer at the splicing point is discontinuous. In actuality, since the streams of STA, STB are seamlessly spliced, the VBV buffer continuously stores the streams without discontinuity. As a result, the VBV buffer occupancy is lower at the splicing point SPVBV by VBV_gap than in the case where the stream STB is stored by itself. Because of this artificially-low occupancy level, the VBV buffer suffers an underflow VBV_under when the following I-frame, which typically occupies four times the VBV buffer space as the B- or P-frames, is retrieved from the VBV buffer.

The problem of overflow of the VBV buffer for the spliced streams will now be described with reference to FIGS. 29(a), (b). FIG. 29(a) is a diagram showing a locus of data occupancy in the VBV buffer for the spliced stream STsp, shown in FIG. 29(b). In this case, the level of the data occupancy at the splicing point is artificially-higher as compared with an original locus of the data occupancy in the VBV buffer for stream STB. As a result, the VBV buffer suffers an overflow when an I-frame is stored in the VBV buffer as shown in the figure.

Overflow occurs because the target bit rate for each picture is too small for the spliced bit stream. The reason for this is that the target bit rate is set for the smaller bit stream STB including VBVOST--B which, as will be seen from FIGS. 27(a), 29(a), is not included-in the spliced-bit stream STRE'. The underflow condition is the opposite case where the target bit rate is too large for the spliced bit stream STB. To compound the problem, the locus of the data occupancy of the VBV buffer becomes discontinuous at a point where the stream STRE' to be re-encoded is switched back to the original stream OSTB which presents an additional overflow/underflow situation.

It is possible to resolve the overflow/underflow problem by controlling the locus VBVOST--B of the amount of data occupancy of the VBV buffer corresponding to the original stream OSTB. However, VBVOST--B is an optimum locus determined to prevent overflow or underflow of the original stream OSTB. If the level of the optimum locus is controlled, there is a possibility that overflow or underflow occurs.

The splice controller 13 operation for setting the new target bit rate will be discussed with reference to FIGS. 19, 28(a), (b) and 30(a), (b). Initially, with reference to FIG. 19, in accordance with a bit count value of stream STA and a bit count value of stream STB supplied from the stream counter 11, the splice controller 13 calculates a locus of the data occupancy of the VBV buffer for the original stream OSTA, a locus of a data occupancy of the VBV buffer for the original stream OSTB and a locus of a data occupancy of the VBV buffer for the stream STRE' to be re-encoded in a case where stream STA and stream STB are spliced. The locus of the data occupancy of the VBV buffer in each case can be calculated by subtracting an amount of bits output from the VBV buffer corresponding to the presentation times from the bit count value supplied from the stream counter 11. Therefore, the splice controller 13 is able to virtually recognize the locus of the data occupancy of the VBV buffer for the original stream OSTA, the locus of the data occupancy of the VBV buffer for the original stream OSTB and the locus of the data occupancy of the VBV buffer for the stream STRE' to be re-encoded in a case where stream STA and stream STB are spliced.

The splice controller 13 references the locus of the data occupancy of stream STRE', to calculate an amount of overflow/underflow (vbv_over)/(vbv_under) of the stream STRE' to be re-encoded. Moreover, the splice controller 13 makes reference to the data occupancy of the stream STRE' and the locus (VBVOST--B) of the data occupancy of the original stream OSTB in the VBV buffer. The splice controller 13 calculates the gap value (vbv_gap) in the VBV buffer at the switching point between the stream STRE' to be re-encoded and the original stream OSTB. The splice controller 13 calculates an offset amount vbv_off of a target amount of codes in accordance with the following Equations (1) and (2):

vbv--off=-(vbv--under-vbv--gap) (1)

vbv--off=+(vbv--over-vbv--gap) (2)

If the VBV buffer underflows as in the case shown in FIG. 27(a), Equation (1) is used to calculate the offset amount vbv_off. If the VBV buffer overflows as in the case shown in FIG. 29(a), Equation (2) is used to calculate the offset amount vbv_off.

Then, the splice controller 13 uses the offset amount vbv_off obtained in accordance with Equations (1) or (2) to calculate a target amount of codes (a target amount of bits) TBP0 in accordance with the following Equation (3): TB P0 = ∑ i = 0 n0 ⁢ ⁢ GB_A ( n - P0 ) + i ⁢ ∑ i = 0 n0 ⁢ ⁢ GB_B ( m - P0 ) - i + vbv_off ( 3 )

The target amount of bits TBP0 is a value assigned to the picture which is subjected to the re-encoding process. In Equation (3), GB_A is a value indicating an amount of generated bits of a picture which is any one of pictures A to An-P0 to A(n-P0)+n0 in stream STA and Σ GB13 A(n-P0)+i is a sum of the amount of generated bits of the pictures An-P0 to A(n-P0)+n0. Similarly, GB13 B is a value indicating an amount of generated bits of a picture which is any one of pictures Bm-P0 to B(m-P0)-m0 in stream STB and Σ GB_B(m-P0)+i is a sum of the amount of generated bits of the pictures Bm-P0 to B(m-P0)-m0.

That is, the target amount of bits TBP0 expressed by Equation (3) is a value obtained by adding the offset amount vbv_off of the VBV buffer to the total amount of generated bits of the pictures A(n-P0)+n0 to B(m-P0)-m0. The offset amount vbv_off is added to correct the target amount of bits TBP0 such that the gap of the locus of the data occupancy at the switching point between the stream STsp, which is to be re-encoded, and the original stream OSTB is minimized (preferably zero). With the present invention, seamless splicing is realized.

The splice controller 13 assigns the target amount of bits TBP0 obtained in accordance with Equation (3) to the pictures A(n-P0)+n0 to B(m-P0)-m0. Usually, the quantizing characteristic of each picture is determined in such a manner that the target amount of bits TBP0 is distributed at a ratio of I picture:P picture:B picture=4:2:1. The splicing apparatus according to at least one embodiment of the present invention is not so rigid but makes reference to the quantizing characteristics including the previous quantizing steps and the quantizing matrices of the pictures A(n-P0)+n0 to B(m-P0)-m0 so as to determine a new quantizing characteristic. Specifically, the encode controller 43 makes reference to the quantizing steps and the quantizing matrices included in streams STA and STB. To prevent an excessive deviation from the quantizing characteristic realized in the previous encoder process of the encoders 1A, 1B the encode controller 43 determines the quantizing characteristic when the re-encoding process is performed.

The present invention in accordance with the foregoing prevents underflow/overflow in the VBV buffer. FIGS. 28(a), (b) illustrate a data occupancy of the VBV buffer when a re-encoding process is performed using the target amount of bits TBP0 calculated by the splice controller 13 which resolves the problem of underflow described with reference to FIGS. 27(a), (b). FIGS. 30(a), (b) similarly illustrate a data occupancy of the VBV buffer when a re-encoding process is performed using the target amount of bits TBP0 calculated by the splice controller 13 which resolves the problem of overflow described with reference to FIGS. 29(a), (b).

The operations of the splicing and editing process according to the present invention will be described with reference to FIGS. 31 and 32. The present invention preferably meets regulations of Annex C of ISO13818-2 and ISO11172-2 and Annex L of ISO13818-1 and of course may conform to their encoding/decoding standards.

In step S10, the splice controller 13 receives the splicing point parameter p0 for splicing the streams STA and STB and re-encoding ranges n0 and m0. It is possible that an operator inputs these parameters. The re-encoding ranges n0 and m0 may be automatically set in accordance with the configuration of the GOP of the stream or the like. In step S11, the splice controller 13 temporarily stores the streams STA and STB in the buffer memory 10. The phases of the splicing point of each of the streams STA and STB are synchronized with reference to the presentation time by controlling a reading operation of the buffer memory 10.

In step S12, the splice controller 13 selects a picture to be output for re-encoding while inhibiting a picture in stream STA appearing after the picture An-P0. Moreover, the splice controller 13 selects a picture to be output for re-encoding while inhibiting a picture appearing before the picture Bm-P0 of stream STB at the splicing point. FIGS. 25(a), (b) illustrate the situation where a P picture A(n-P0)-2 of stream STA appears after the picture An-P0 at the splicing point. In an order of presentation, picture A(n-P0)-2 is a picture in the future as compared with picture An-P0. Therefore, the P picture A(n-P0)-2 is not output in the present invention. Similarly, as shown in FIGS. 25(c) and (d), the B pictures B(m-P0)+2 and B(m-P0)+1 are before the picture Bm-P0 at the splicing point. In an order of presentation, pictures B(m-P0)+2 and B(m-P0)+1 are previous to picture Bm-P0. Therefore, the B pictures B(m-P0)+2 and B(m-P0)+1 are not output in the present invention. As described, pictures to be transmitted are selected with reference to the order of presentation, thereby preventing the problem of the presentation order described with reference to FIGS. 9(a)-(d).

In step S13, the splice controller 13 initiates a process for setting the coding parameters required to reconstruct the pictures for re-encoding in accordance with steps S14 to S30. The parameters which are set in this process include the picture type, a direction of prediction and the motion vectors for example.

In step S14, the splice.controller 13 determines whether the picture to be subjected to the picture reconstruction process is the picture An-P0 at the splicing point. If so, the operation proceeds to step S15. Otherwise, the operation proceeds to step S20.

In step S15, the splice controller 13 determines whether the picture to be subjected to the picture reconstruction is a B picture, a P picture or an I picture. If the picture to be subjected to the picture reconstruction is a B picture, the operation proceeds to step S17. If the picture to be subjected to the picture reconstruction is a P picture or an I picture, the operation proceeds to step S18.

In step 16, the splice controller 13 determines whether two or more B pictures exist in front of picture An-P0 in the spliced stream STsp. For example, and as shown is FIG. 26(b), if two B pictures (A(n-P0)+2, A(n-P0)+3) exist in front of picture An-P0, the operation proceeds to step S18. Otherwise, the operation proceeds to step S17. In step S17, the splice controller 13 determines that the change of the picture type of the picture An-P0 is unnecessary. At this time, the splice controller 13 sets a picture type for use in the process for re-encoding the picture An-P0 to the same picture type (the B picture) used previously by the encoder 1A. Therefore, in the re-encoding process in this case the picture An-P0 is re-encoded as the B picture.

In step S18, the splice controller 13 changes the picture type of the picture An-P0 from the B picture to the P picture. To explain, when two B pictures (A(n-P0)+2, A(n-P0)+3) exist in front of the B picture (An-P0), there are three B pictures to be re-encoded which are arranged sequentially in the stream STRE'. Since a typical MPEG decoder has only two frame memories for temporarily storing predicted pictures, the third B picture cannot be decoded. Therefore, the present invention changes the picture An-P0 type from the B picture to the P picture type as described with reference to FIGS. 26(a), (b). Thus, the picture An-P0 is reliably decoded as a P picture.

In step S19, the splice controller 13 determines that the change in the picture type of the picture An-P0 is unnecessary. At this time, the splice controller 13 sets the picture type for use when the picture An-P0 is re-encoded to the picture type (the I picture or the P picture) set previously by the encoder 1A.

In step S20, the splice controller 13 determines that the change in the picture type of the picture An-P0 is unnecessary. At this time, the splice controller 13 sets the picture type for use when the picture An-P0 is re-encoded to the picture type (the I picture, the P picture or the B picture) set previously by the encoder 1A.

In step S21, the splice controller 13 sets a direction of prediction and the motion vectors for each picture. In the example shown in FIGS. 25(a)-(d) and 26(a), (b), the picture An-P0 to be subjected to the picture reconstruction process is a B picture in the original stream OSTA. In this case, the B picture An-P0 is bi-directionally predicted from the P pictures A(n-P0)+1 and A(n-P0)-2. According to step S12, the P picture A(n-P0)-2 is inhibited from being output as the spliced stream and, thus, is prevented from becoming an inversely predicted picture of the picture An-P0 specified in the picture reconstruction process. Therefore, when the picture An-P0 is a B picture, its picture type is unchanged in step S17 and, as such, is subjected to a forward and one-sided prediction in which only the P picture of A(n-P0)+1 is employed for prediction. This is similar to the case in step S18 where the B picture is changed to the P picture such that the one-sided prediction parameter for predicting the picture An-P0 is based only on the P picture A(n-P0)+1.

The-direction of prediction.when the picture An-P0 is a P picture in step S19 is unchanged. That is, the splice controller 13 sets a forward and one-sided prediction for the picture An-P0 as in the previous encode process performed by the encoder 1A.

A change in the direction of prediction of the pictures A(n-P0)+n0 to A(n-P0)+1 as determined in step S20 is unnecessary. That is, the splice controller 13 sets a direction of prediction for the pictures A(n-P0)+n0 to A(n-P0)+1 as set previously by the encoder 1A. If the two pictures A(n-P0)+1 and An-P0 are B pictures predicted from two directions from the forward-directional P picture or I picture and the inverse-directional I picture or the P picture, the prediction for the picture A(n-P0)+1 as well as the picture An-P0 must be changed to one-sided prediction such that prediction is performed from only the forward-directional picture.

In step S21, the splice controller 13 determines whether the motion vectors for each picture in the previous encode process performed by the encoder 1A is reused when the re-encoding process is performed in accordance with the newly set direction of prediction. As described above, the motion vectors used in a previous encode process performed by the encoder 1A are the same as in the re-encoding process,i.e., employed for the P picture and the B picture when the direction of prediction of each has not changed. In the examples shown in FIGS. 23(a), (b) and 26(a), (b), the motion vectors used in the previous encode process performed by the encoder 1A are reused when the pictures A(n-P0)+n0 to A(n-P0)+1 are re-encoded. When the picture A(n-P0)+1 and the picture An-P0 are B pictures predicted from both directions from a P picture or an I picture in the forward direction and an I picture or a P picture in the reverse direction, the prediction is changed to one-sided prediction in which prediction is performed in only a forward-directional picture. Therefore, only motion vectors corresponding to the forward-directional picture are used. That is, when the picture A(n-P0)+1 and the picture An-P0 are B pictures, the splice controller 13 sets the prediction direction such that the motion vector for the forward-directional picture is used and the inverse-directional motion vector is not used in step S21.

If the picture An-P0 is a picture predicted in one direction, e.g., the inverse direction from only a future picture such as A(n-P0)-2, the motion vectors produced in the previous encoder process performed by the encoder 1A are not used. In this case, new motion vectors corresponding to A(n-P0)+1 are produced. That is, the splice controller 13 sets the direction of prediction in step S21 such that any previous motion vectors are not used.

In step S22, the splice controller 13 determines whether all parameters of the picture type, the direction of prediction and previous motion vectors of the pictures from pictures A(n-P0)+n0 to An-P0 are set. If so, control proceeds to step S23.

In step S23, the splice controller 13 determines whether the picture to be subjected to the picture reconstruction process is a picture Bm-P0 at the splicing point. If so, the operation proceeds to step S24. Otherwise, if the picture to be subjected to the picture reconstruction is any one of pictures B(m-P0)-1 to B(m-P0)+m0, the operation proceeds to step S28. In step S24, the splice controller 13 determines whether the picture to be subjected to the picture reconstruction process is a B picture, a P picture or an I picture. If the picture to be subjected to the picture reconstruction process is a B picture, the operation proceeds to step S25. If the picture to be subjected to the picture reconstruction process is a P picture, the operation proceeds to step S26. If the picture to be subjected to the picture reconstruction process is an I picture, the operation proceeds to step S27.

In step S25, the splice controller 13 determines that a change in the picture type of the picture Bm-P0 in the re-encoding process is unnecessary as in the example shown in FIGS. 22(a)-(d) and 23(a), (b). Thus, the splice controller 13 sets the picture type for use in a re-encoding process of the picture Bm-P0 to the same picture type (the B picture) as set previously by the encoder 1B.

In step S26, the splice controller 13 changes the picture type of the picture Bm-P0 from the P picture to the I picture as in the examples shown in FIGS. 25(a)-(d) and 26(a), (b). The reason will now be described. Since the P picture is a one-sided prediction picture which is predicted from the forward-directional I- or P-picture, the P picture is always positioned behind the pictures used for prediction on the stream. If the first picture Bm-P0 at the splicing point in the stream STB is a P picture, prediction must be performed from a forward-directional picture of the stream STA which exists in front of the picture Bm-P0. Since the streams STA, STB are different, it is apparent that the quality of the image obtained by a decoding process deteriorates considerably if the picture type of the first picture Bm-P0 is set to the P picture. In this case, the splice controller 13 changes the picture type of the Bm-P0 picture to the I picture.

In step S27, the splice controller 13 determines that a change in the picture type of the picture Bm-P0 is unnecessary. Thus, the splice controller 13 sets the picture for use in the re-encoding process of the picture Bm-P0 to the same picture type (I picture) set previously by the encoder 1B.

In step S28, the splice controller 13 determines that a change in the picture type of the pictures B(m-P0)-1 to B(m-P0)-m0 is unnecessary. The splice controller 13 sets the picture for use in the re-encoding process of each of the foregoing pictures to the same picture type (the I picture, the P picture or the B picture) set previously by the encoder 1B.

In step S29, the splice controller 13 sets a direction of prediction and motion vectors for each picture. If the picture Bm-P0 to be subjected to the picture reconstruction process is, in the original stream OSTB, a B picture as in the example shown in FIGS. 22(a)-(d) and 23(a), (b), the picture Bm-P0 is a picture predicted from two directions, i.e., from the P picture B(m-P0)+1 and the I picture B(m-P0)-2. As described in step S12, the P picture of B(m-P0)+1 is not output as a splicing stream and, therefore, is not specified as a forward-directional prediction picture for the picture Bm-P0 to be subjected to the picture reconstruction process. Therefore, the picture Bm-P0 which is set such that a change in its picture type is unnecessary in step S25 must be set to perform an inverse and one-sided prediction such that only the I picture B(m-P0)-2 is predicted. Therefore, the splice controller 13 sets a direction of prediction for the picture Bm-P0 to perform the inverse and one-side prediction such that only the I picture B(m-P0)-2 is used in the prediction.

A change in the direction of prediction of the pictures B(m-P0)+m0 to B(m-P0)+1 in step S28 is deemed unnecessary. In this case, the splice controller 13 sets a direction of prediction for the pictures B(m-P0)+m0 to B(m-P0)+1 to the same picture previously set by the encoder 1B. If the picture B(m-P0)-1 is a B picture, a direction of prediction for the B(m-P0)-1 is set such that inverse and one-sided prediction is performed so that only the I picture of B(m-P0)-2 is predicted. This is similar to the foregoing case in which the picture Bm-P0 is predicted.

In accordance with the newly set direction of prediction, the splice controller 13 determines in step S29 whether the motion vectors set previously are reused for each picture when the re-encoding process is performed. As described above, the re-encoding process is performed such that the motion vectors used in a previous encode process of the encoder 1B are reused for the P pictures and the B pictures when the prediction direction has not been changed. For example, in FIGS. 22(a)-(d) and 23(a), (b), the motion vectors used in a previous encode process are used for the pictures from the I picture B(m-P0)-2 to the P picture B(m-P0)-m0. The direction of prediction for each of the pictures Bm-P0 and B(m-P0)-1 predicted from the two directions, e.g., from the P picture B(m-P0)+1 and the I picture of B(m-P0)-2, in a previous encoder process performed by-the encoder 1B is changed to one-sided prediction such that only the I picture B(m-P0)-2 is used for prediction. Therefore, the motion vectors corresponding to the picture B(m-P0)+1 are not used. That is, in step S29, the splice controller 13 reuses the previous motion vectors for only one direction for the pictures Bm-P0 and B(m-P0)-1. That is, the motion vectors for the inverse direction are not used.

Next, in step S30, the splice controller 13 determines whether the parameters relating to the picture type, the direction of prediction and the motion vectors for all of the pictures from the picture Bm-P0 to the picture B(m-P0)-m0 are set. If so, the splice controller 13 in step S31 calculates a target amount of bits (TBP0) to be generated in the re-encoding period in accordance with Equation (3). Specifically, the splice controller 13 initially calculates a locus of the data occupancy of the VBV buffer for the original stream OSTA, a locus of the data occupancy of the VBV buffer for the original stream OSTB and a locus of the data occupancy of the VBV buffer for the stream STRE' to be encoded in a case where streams STA, STB are spliced in accordance with a bit count value of stream STA and the bit count value of stream STB supplied from the stream counter 11.

Then, the splice controller 13 analyzes the virtually-obtained locus of the data occupancy of the VBV buffer for the stream STRE' to be re-encoded.

Thus, the splice controller 13 calculates an amount of underflow (vbv_under) or an amount of overflow (vbv_over) of the stream STRE' to be re-encoded. Moreover, the splice controller 13 compares the virtually-obtained locus of the data occupancy of the VBV buffer for stream STRE' to be re-encoded and a locus (VBVOSTB) of the data occupancy in the VBV buffer for the original stream OSTB. Thus, the splice controller 13 calculates a gap value (vbv_gap) of the VBV buffer at a switching point between stream STRE' to be re-encoded and the original stream OSTB. Then, the splice controller 13 calculates.an offset amount vbv_off of the target amount of codes in accordance with Equations (1) and (2). Then, the splice controller 13 uses the offset amount vbv_off calculated in accordance with Equation (1) or (2) to calculate a target amount of codes .(target amount of bits) TBP0 in accordance with Equation (3).

In step S32, the splice controller 13 determines a quantizing characteristic to be set for each picture. The quantizing characteristic is determined in accordance with an assignment to the pictures A(n-P0)+n0 to B(m-P0)-m0 of the target amount of bits TBP0 calculated in accordance with Equation (3). The splicing apparatus according to the present invention makes reference to quantizing characteristics including the previous quantizing steps and the quantizing matrices of each of the pictures A(n-P0)+n0 to B(m-P0)-m0 used by the encoders 1A and 1B so as to determine new quantizing characteristics. Specifically, the splice controller 13 initially receives from the stream analyzing portion 12 information about the coding parameters, quantizing steps and quantizing matrices produced in a previous coding process performed by the encoders 1A and 1B and included in the streams STA, STB.

Further, the splice controller 13 makes reference to the amounts of codes (bits) assigned to the target amount of bits TBP0 and information of the previous coding parameters. The splice controller 13 determines the quantizing characteristics when the re-encoding process is performed so as to prevent excessive deviation from the quantizing characteristics in the encoding processes performed by the encoders 1A and 1B. As described in steps S18 and S26, the quantizing characteristics of the pictures, the picture type of each of which has been changed by the picture reconstruction process, are newly calculated when the re-encoding process is performed without reference to the information of the quantizing steps and the quantizing matrices.

In step S33, the splice controller 13 decodes the pictures A(n-P0)+n0 to B(m-P0)-m0 included in the re-encoding range. In step S34, the splice controller 13 uses the quantizing characteristics set to pictures A(n-P0)+n0 to B(m-P0)-m0 while controlling the amount of generated bits. If the splice controller 13 reuses the previous motion vectors, the encode controller 43, at the control of the splice controller 13, causes switch 44 to channel the previous motion vectors to the motion compensation portion 41. When the previous motion vectors are not used, the encode controller 43 controls the switch 44 to channel the motion vectors newly produced by the motion detection circuit 42 to the motion compensation portion 41. At this time, the encode controller 43 controls the frame memories 39 and 40 in accordance with information about the picture type supplied from the splice controller 13 to store the pictures required to produce predicted image data. The encode controller 43 sets, to the quantizing circuit 34 and the inverse quantization circuit 36, the quantizing characteristics in the re-encoding range supplied from the splice controller 13.

In step S35, the splice controller 13 controls the switch 17 to selectively output stream STA from the buffer 10, stream STB from the buffer 10 or the re-encoded stream STRE from the MPEG encoder 16. Thus, the splice controller 13 seamlessly-splices stream STA which appears before the re-encoding range, re-encoded stream STRE in the re-encoding range and stream STB which appears after the re-encoding range to provide seamlessly-spliced bit stream STSP.

Although preferred embodiments of the present invention and modifications thereof have been described in detail herein, it is to be understood that this invention is not limited to those precise embodiments and modifications, and that other modifications and variations may be affected by one skilled in the art without departing from the spirit and scope of the invention as defined by the appended claims.

Yoshinari, Hiromi

Patent Priority Assignee Title
10977747, Jun 18 2010 AKAMAI TECHNOLOGIES, INC Extending a content delivery network (CDN) into a mobile or wireline network
6763390, Jan 24 2000 AVAGO TECHNOLOGIES INTERNATIONAL SALES PTE LIMITED Method and system for receiving and framing packetized data
6778533, Jan 24 2000 Broadcom Corporation Method and system for accessing packetized elementary stream data
6785336, Jan 24 2000 AVAGO TECHNOLOGIES GENERAL IP SINGAPORE PTE LTD Method and system for retrieving adaptation field data associated with a transport packet
6819712, Feb 28 2000 Kabushiki Kaisha Toshiba Video encoding apparatus and video encoding method
6831949, Jul 18 1997 British Broadcasting Corporation Switching compressed video bitstreams
6885680, Jan 24 2000 AVAGO TECHNOLOGIES INTERNATIONAL SALES PTE LIMITED Method for synchronizing to a data stream
6965597, Oct 05 2001 Verizon Patent and Licensing Inc Systems and methods for automatic evaluation of subjective quality of packetized telecommunication signals while varying implementation parameters
6983015, Aug 26 1999 Sony United Kingdom Limited Signal processor
6988238, Jan 24 2000 AVAGO TECHNOLOGIES INTERNATIONAL SALES PTE LIMITED Method and system for handling errors and a system for receiving packet stream data
7010168, Dec 26 2001 MTEKVISION CO , LTD Apparatus for prediction coding or decoding image signal and method therefor
7305040, Jan 19 1998 Sony Corporation Edit system, edit control device, and edit control method
7349675, Jul 03 2001 Microsoft Technology Licensing, LLC Low latency data stream encoding and transmission
7366961, Jan 24 2000 AVAGO TECHNOLOGIES INTERNATIONAL SALES PTE LIMITED Method and system for handling errors
7376692, Jan 24 2000 AVAGO TECHNOLOGIES INTERNATIONAL SALES PTE LIMITED Method and system for receiving and framing packetized data
7394851, Jun 01 1999 Sony Corporation Encoding apparatus and method, and multiplexing apparatus and method
7426306, Oct 24 2002 Altera Corporation Efficient use of keyframes in video compression
7558465, Aug 25 2004 Sony Corporation Information processing apparatus and information processing method, recording medium, and program
7561897, Jul 03 2001 Microsoft Technology Licensing, LLC System and apparatus for performing broadcast and localcast communications
7577407, Jul 03 2001 Microsoft Technology Licensing, LLC System and apparatus for performing broadcast and localcast communications
7577457, Jul 03 2001 Microsoft Technology Licensing, LLC System and apparatus for performing broadcast and localcast communications
7590242, Mar 21 2002 SmarDTV SA Selective multimedia data encryption
7609762, Sep 07 2003 Microsoft Technology Licensing, LLC Signaling for entry point frames with predicted first field
7653363, Jul 03 2001 Microsoft Technology Licensing, LLC System and apparatus for performing broadcast and localcast communications
7839930, Nov 13 2003 Microsoft Technology Licensing, LLC Signaling valid entry points in a video stream
7852919, Sep 07 2003 Microsoft Technology Licensing, LLC Field start code for entry point frames with predicted first field
7924921, Sep 07 2003 Microsoft Technology Licensing, LLC Signaling coding and display options in entry point headers
8170120, Sep 05 2006 Sony Corporation Information processing apparatus and information processing method
8194735, Aug 12 2005 Kabushiki Kaisha Toshiba Video encoding apparatus and video encoding method
8213779, Sep 07 2003 Microsoft Technology Licensing, LLC Trick mode elementary stream and receiver system
8260109, Jul 25 2006 ATI Technologies ULC System for digital time shifting and method thereof
8284845, Jan 24 2000 ATI Technologies ULC Method and system for handling data
8295347, Aug 25 2004 Sony Corporation Information processing apparatus and information processing method, recording medium, and program
8311104, Aug 22 2005 Sony Corporation Information processing apparatus and method, recording medium, and program
8311105, Apr 07 2006 Sony Corporation Information-processing apparatus, information-processsing method, recording medium and program
8320744, Jul 13 2006 Sony Corporation Video signal processing device, video signal processing method, video signal encoding device, video signal encoding method, and program
8335262, Jan 16 2008 VVNET, LLC Dynamic rate adjustment to splice compressed video streams
8411759, Jun 02 2004 Panasonic Corporation Multiplexing apparatus and demultiplexing apparatus
8543667, Jan 14 2008 AKAMAI TECHNOLOGIES, INC Policy-based content insertion
8650602, Feb 27 2009 AKAMAI TECHNOLOGIES, INC Input queued content switching using a playlist
8699856, Sep 20 2007 Sony Corporation Editing apparatus, editing method, program, and recording medium
8743906, Jan 23 2009 AKAMAI TECHNOLOGIES, INC Scalable seamless digital video stream splicing
8817887, Sep 05 2006 Sony Corporation Apparatus and method for splicing encoded streams
9544183, Jan 14 2008 AKAMAI TECHNOLOGIES, INC Methods and apparatus for providing content delivery instructions to a content server
9565397, Feb 26 2009 AKAMAI TECHNOLOGIES, INC Deterministically skewing transmission of content streams
9906757, Feb 26 2009 AKAMAI TECHNOLOGIES, INC Deterministically skewing synchronized events for content streams
RE47054, Nov 06 2000 ATI Technologies ULC System for digital time shifting and method thereof
Patent Priority Assignee Title
5982436, Mar 28 1997 Pendragon Wireless LLC Method for seamless splicing in a video encoder
6025878, Oct 11 1994 Hitachi America Ltd. Method and apparatus for decoding both high and standard definition video signals using a single video decoder
6137834, May 29 1996 INVIDI Technologies Corporation Method and apparatus for splicing compressed information streams
EP694921,
JP10112840,
JP6253331,
JP8037640,
JP8149408,
WO9708898,
//
Executed onAssignorAssigneeConveyanceFrameReelDoc
Mar 25 1999Sony Corporation(assignment on the face of the patent)
Apr 01 1999YOSHINARI, HIROMISony CorporationASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS 0099360658 pdf
Date Maintenance Fee Events
Nov 20 2006M1551: Payment of Maintenance Fee, 4th Year, Large Entity.
Nov 18 2009RMPN: Payer Number De-assigned.
Nov 19 2009ASPN: Payor Number Assigned.
Nov 12 2010M1552: Payment of Maintenance Fee, 8th Year, Large Entity.
Nov 13 2014M1553: Payment of Maintenance Fee, 12th Year, Large Entity.


Date Maintenance Schedule
May 20 20064 years fee payment window open
Nov 20 20066 months grace period start (w surcharge)
May 20 2007patent expiry (for year 4)
May 20 20092 years to revive unintentionally abandoned end. (for year 4)
May 20 20108 years fee payment window open
Nov 20 20106 months grace period start (w surcharge)
May 20 2011patent expiry (for year 8)
May 20 20132 years to revive unintentionally abandoned end. (for year 8)
May 20 201412 years fee payment window open
Nov 20 20146 months grace period start (w surcharge)
May 20 2015patent expiry (for year 12)
May 20 20172 years to revive unintentionally abandoned end. (for year 12)