The invention is a data pipeline system which synchronizes the display of digitized audio and video data regardless of the speed at which the data was recorded on its linear medium. To do this, the video data is played at a constant speed, synchronized by the audio speed. Further, the invention uses a method of “staging” data in storage buffers, i.e., ring buffers, which encourages efficient use of the viewer module resources by not permitting the viewer to read excessive amounts of data at any one time, i.e., to read only enough data into any one ring buffer so that the amount of data in the ring buffer is roughly equivalent to the amount of data in the other ring buffers and yet permitting the viewer to read large enough chunks of data to promote efficient use of the fill system.
|
0. 27. A process for transferring media data for a plurality of streams from a computer readable medium during playback of the media data to one or more output devices for playing back each stream, and wherein each stream has a buffer for storing media data before playback, the process comprising:
determining an actual amount of data in the buffer for each stream;
selecting one of the streams, and selecting a desired amount of data to be read for the stream;
for the selected stream, reading the desired amount of data from the computer readable medium into the buffer for the output device associated with the selected stream for playback by the output device; and
repeating steps of measuring, selecting and reading during playback of the media data of the plurality of streams.
0. 37. A process for transferring video data from one or more data files stored in a file system of an operating system of a general purpose computer to a video playback device and audio data from the one or more data files stored in the file system to an audio playback device, during playback of the audio and video, wherein a first buffer is associated with the video playback device to storing video data before playback and a second buffer is associated with the audio playback device to store audio data before playback, the process comprising:
determining an actual amount of data in the first buffer;
determining an actual amount of data in the second buffer;
selecting one of the video data and the audio data according to the measured amount of data in the first buffer and the second buffer;
for the selected data, reading a desired amount of data from the one or more data files in the file system; and
repeating steps of measuring, selecting, and reading while playing back the video data in the first buffer and the audio data in the second buffer.
0. 17. A process for transferring video data from one or more video data files stored in a file system of an operating system of a general purpose computer to a video playback device and audio data from one or more audio data files stored in the file system to an audio playback device, during synchronized playback of the audio and video, wherein a first buffer associated with the video playback device stores video data before playback and a second buffer associated with the audio playback device stores audio data before playback, the process comprising:
determining an actual amount of data in the first buffer;
determining an actual amount of data in the second buffer;
selecting one of the video data and the audio data according to the measured amount of data in the first buffer and the second buffer;
reading a desired amount of data from the data file for the selected data;
playing back the video data in the first buffer and the audio data in the second buffer in synchronism; and
repeating steps of measuring, selecting and reading while playing back the video data and audio data in synchronism.
0. 1. A media pipeline system for displaying digitized audio and video, comprising:
a media file database for accepting and making available a plurality of media files, at least one of said media files having digitized audio media data and at least another one of said media files having video media data, each of said media files being divided into a plurality of frames;
a plurality of ring buffers;
a viewer module comprising procedures to synchronize the transfer of media data from the media files into said plurality of ring buffers, said ring buffers storing the media data before it is displayed;
a video accelerator connected to receive an output of at least one of said ring buffers for generating a video display output; and
a sound accelerator connected to receive an output of at least one of said ring buffers for generating an audio output.
0. 2. The system of
0. 3. The system of
0. 4. The system of
0. 5. The system of
0. 6. The system of
0. 7. The system of
0. 8. The system of
0. 9. The system of
0. 10. The system of
0. 11. A method of displaying digitized media data on a monitor, comprising the steps of:
storing the media data in a plurality of media files and a plurality of ring buffers, at least one of which contains video media data and at least one of which contains audio media data;
determining which of the ring buffers contains the least media data;
reading media data from one of the media files into the ring buffer having the least media data;
signaling a sound accelerator card which plays a part of the media data found one of the audio ring buffers;
waiting for an interrupt from the sound accelerator card;
signaling a video accelerator card which plays the a part of the media data found in one of the video ring buffers.
0. 12. The method of
0. 13. The method of
0. 14. The system of
0. 15. The method of
0. 16. The system of
0. 18. The process of
0. 19. The process of
0. 20. The process of
0. 21. The process of
0. 22. The process of
signaling one of the playback devices to play a part of the data in the buffer for the playback device, the playback device producing an output signal after playing said part; and
signaling the other of the playback devices to play a part of the data in the buffer for the other playback device.
0. 23. The process of
for each buffer, determining a state according to the determined amount of data in the buffer;
selecting an action according to the state of each buffer.
0. 24. The process of
0. 25. The process of
0. 26. The process of
0. 28. The process of
0. 29. The process of
0. 30. The process of
0. 31. The process of
0. 32. The process of
signaling one of the playback devices to play a part of the data in the buffer for the one of the playback devices, the one of the playback devices producing an output signal after playing said part; and
signaling another of the playback devices to play a part of the data in the buffer for the another of the playback devices.
0. 33. The process of
for each buffer, determining a state according to the determined amount of data in the buffer;
selecting an action according to the state of each buffer.
0. 34. The process of
0. 35. The process of
0. 36. The process of
0. 38. The process of
0. 39. The process of
0. 40. The process of
0. 41. The process of
0. 42. The process of
signaling a one of the playback devices to play a part of the data in the buffer for the playback device;
the playback device producing an output signal after playing said part; and
signaling the other of the playback devices to play a part of the data in the buffer for the other playback device.
0. 43. The process of
for each buffer, determining a state according to the determined amount of data in the buffer;
selecting an action according to the state of each buffer.
0. 44. The process of
0. 45. The process of
0. 46. The process of
|
This application is a continuation reissue application of reissue application Ser. No. 08/418,862, filed on Apr. 7, 1995, which is a continuation of reissue application Ser. No. 08/116,905, filed on Sep. 3, 1993 for reissue of U.S. Pat. No. 5,045,940, which is incorporated herein by reference.
The application includes a microfiche appendix pursuant to 37 CFR 1.96(b) containing 1 microfiche with 44 frames.
The invention relates to displaying non-linear media data, i.e., digitized audio and video data.
Non-linear media data is audio and video data recorded on a linear medium, e.g., a VHS videotape cassette, and stored in digitized form on a computer storage device, e.g., a hard disk drive. Typically, linear audio data is recorded with a “pulse” equal to the speed at which the linear video data is recorded. That is, if the video data is recorded at 30 frames per second (fps), the accompanying audio data is likewise recorded at 30 fps. This is obviously the case where the audio and video data are recorded simultaneously on a single medium, e.g., a single videotape cassette. However, the recording of the audio data can be distinct from the recording of the video data, e.g. a soundtrack can be recorded in London and a film clip can be shot on location in the Sahara Desert, in which case the speed of recording the video and the audio may or may not be equal. In addition, the standard speeds for filming video and recording audio vary from country to country. For example, the standard speed for recording video in the United States is 30 fps, while the standard speed in Europe is 24 fps. Likewise, the audio sampling rate standard is 22 kHz, but 44 kHz is also used. Thus, in cases where the speeds are different, the two recordings (often referred to as media “channels”) must be effectively combined and displayed so that there are no visible or audible gaps or overlaps.
As noted above, the speed at which video data is recorded and displayed can vary, e.g., from 30 fps to 24 fps. Video data from different countries, however, can be successfully edited together and played at a single speed because the human eye cannot detect subtle variations in the number of frames per second. The human ear, however, can detect even subtle variations in the speed at which audio data is played. These variations appear as audible clicks, silences, or other distortions. Over time, differences in the speed at which the video data is displayed and the audio data are played results in visibility and audibly mismatched video and audio data.
The invention is a data pipeline system which synchronizes the display of digitized audio and video data regardless of the speed at which the data was recorded on its linear medium. To do this, the video data is played at a constant speed, synchronized by the audio speed. The system includes a media file database (MFD) that contains a number of media files, each of which contains either digitized audio or digitized video media data. The system also includes a viewer module which synchronizes operations to transfer the media data from the media files into a number of ring buffers (preferably software). Each ring buffer is associated with a media channel being displayed. For example, there is a ring buffer which contains media data for the video channel, a ring buffer which contains media data for a first audio channel, and a ring buffer which contains media data for a second audio channel. The two audio channels are for stereo. The viewer module also synchronizes calls to a video accelerator card and a sound accelerator card so that the video data and audio data, which were recorded on a linear medium at differing pulse rates, are displayed at a constant rate without visible or audible gaps or overlaps.
Further, the invention uses a method of “staging” data in the storage buffers, particularly ring buffers, to coordinate buffer loading to encourage efficient use of the viewer module resources by not permitting the viewer to read excessive amounts of data at any one time, i.e., to read only enough data into any one ring buffer so that the amount of data in the ring buffer is approximately equivalent to the amount of data in the other ring buffers.
Other advantages and features will become apparent from the following description, and from the claims.
Referring to
The pipeline system 10 also includes a viewer module 16 which synchronizes the transfer of media data from the media files 14 into three ring buffers 18a-18c which store the data or pointers to the data before it is displayed. For convenience, the terms “view” and “display” are used herein with respect to audio as well as video and should be understood to mean “play” when referring to audio. Each ring buffer is associated with a media channel. That is, ring buffer 18a contains media data for the video channel, ring buffer 18b contains media data for audio channel 1, and ring buffer 18c contains media data for audio channel 2. Upon a signal from the viewer module, a pointer to a frame of video data in the ring buffer 18a is transferred to a conventional video accelerator card 20 preferably TRUEVISION ® model NuVista which displays the video data on a monitor 22. Likewise, upon a signal from the viewer module 16, a pointer to a frame of audio data in the ring buffers 18b and 18c is transferred to a conventional sound accelerator card 24 preferably DIGIDESIGN ® model SOUND ACCELERATOR which plays the audio data through the monitor 22. The operation of the system and synchronization of the displaying of video data and playing of audio data is described below in connection with FIG. 2.
Referring to
In general, “staging” the data in the ring buffers 18 encourages efficient use of the viewer module resources by not permitting the viewer to read excessive amounts of data at any one time, i.e., to read only enough data into any one ring buffer so that the amount of data in the ring buffer is roughly equivalent to the amount of data in the other ring buffers (see steps 254, 260 and 266 in FIG. 3). The staging process also encourages efficient use of the file system (disks where media is stored) by permitting large efficient reads from the disk when there is time available.
Referring to
Before PLAY_AV is called the system preloads all the ring buffers with data. This considerably improves efficiency since this can be done before the time-critical operations occur.
Fist, PLAY_AV performs an error check, i.e., determines if the number of frames to be played is zero (step 200). If the number of frames to be displayed is not zero, then PLAY_AV performs further error checks, e.g., determines that the audio capability of the monitor 22 is initialized (by checking the value of the Boolean variable AUDIO_INIT_DONE) and that the audio data is to be used to synchronize the video data (by checking the value of the Boolean variable SOUND_SYNC) (step 202). Next, if there are no errors (step 202), then PLAY_AV determines which audio channels are in use (step 204), i.e., from which of the ring buffers 18 audio data will be read. Note that the embodiment described here uses two audio data channels, i.e., audio channel 1 and audio channel 2. Additional channels and hence additional buffers are of course possible. Fewer channels (i.e., 1) are also permitted.
Having determined which channels are in use, PLAY_AV next initializes variables for the frames of audio and video data to be displayed (step 206), i.e., assigns a value of 0 to the variables “nextAudio1”, “nextAudio2”, and “nextVideo”. In addition, PLAY_AV initializes the end of file marker for the video data (“videoEOF”) to FALSE (step 208) and also initializes the end of file markers for the audio data (“audio1EOF” and “audio2EOF”) (step 220). Specifically, if audio channel 1 is being used (step 204) and soundsync is TRUE (step 202), then audio1EOF equals FALSE. Otherwise, audio1EOF equals TRUE. Likewise, if audio channel 2 is being used (step 204) and soundsync is TRUE (step 202), then audio2EOF equals FALSE. Otherwise, audio2EOF equals TRUE.
Once PLAY_AV has determined from which channels it will read data (steps 204-210), it begins an infinite (while TRUE) loop to read, transfer, and display media data. PLAY_AV does not exit the loop until the clip is exhausted, i.e., there are no more frames to display. At the beginning of each pass through the loop, PLAY_AV initializes several variables, including the maximum number of bytes it will read (“max_read”) and the number of audio channel 1 and audio channel 2 bytes yet to be played (“AbytesUnplayed1” and “AbytesUnplayed2”) (step 212). In addition, PLAY_AV initializes several variables that indicate whether it should “wait”, execute a “critical” read, or execute an “efficient” read (step 214) (each of which is described below in more detail), and also initializes a variable “fewest_buffers” to MAX_LONG (step 216), i.e., a number far larger than the number of bytes in a ring buffer.
Having initialized the loop variables, PLAY_AV next determines which of the ring buffers has the least amount of data in it, i.e., which ring buffer has fewer bytes free. PLAY_AV begins by checking ring buffer 18b (audio channel 1) as described below.
To determine what the state of ring buffer 18b (audio channel 1) is, PLAY_AV determines if audio1EOF is FALSE and ring buffer 18b has at least 735 bytes free (step 218). If so, PLAY_AV goes on to determine if the number of bytes free in ring buffer 18b is less than fewest buffers (step 220) (which is always true initially since fewest_buffers was assigned MAX_LONG above). The action variable is then assigned a value of “read_AUDIO1” (step 222). The critical variable is assigned a value of TRUE if fewest_buffers is less than a predefined number (AUDIO_ALERT_BUFS), and is assigned a value of FALSE otherwise (step 224). And the efficient variable is assigned a value of TRUE if the number of bytes free in ring buffer 18b is greater than or equal to a predefined number (EFFICIENT_AUDIO_BYTES) and if the size of the next audio frame times the typical audio frame size is greater than or equal to EFFICIENT_AUDIO_BYTES (step 226). Otherwise the variable efficient is assigned a value of FALSE.
To determine what the state of ring buffer 18c (audio channel 2) is, PLAY_AV determines if audio2EOF is FALSE and ring buffer 18c has at least 735 bytes free (step 228). If so, PLAY_AV goes on to determine if the number of bytes free in ring buffer 18c is less than fewest buffers (step 230) (i.e., whether ring buffer 18c has fewer bytes than ring buffer 18b as determined above). If ring buffer 18c indeed contains fewer bytes, the action variable is assigned a value of “read_AUDIO2” (step 232). The critical variable is assigned a value of TRUE if fewest_buffers is less than a predefined number (AUDIO_ALERT_BUFS), and is assigned a value of FALSE otherwise (step 234). And the efficient variable is assigned a value of TRUE if the number of bytes free in ring buffer 18c is greater than or equal to a predefined number (EFFICIENT_AUDIO_BYTES) and if the size of the next audio frame times the typical audio frame size is greater than or equal to EFFICIENT_AUDIO_BYTES (step 236). Otherwise the efficient variable is assigned a value of FALSE.
Finally, to determine what the state of ring buffer 18a (video channel) is, PLAY_AV determines if videoEOF is FALSE and ring buffer 18c has at least 1 byte free (step 238). If so, PLAY_AV goes on to determine if the number of bytes free in ring buffer 18a is less than fewest_buffers (step 240) (i.e., whether ring buffer 18a has fewer bytes than ring buffer 18c as determined above). If ring buffer 18a indeed contains fewer bytes, the action variable is assigned a value of “read_VIDEO” (step 242). The critical variable is assigned a value of TRUE if fewest_buffers is less than a predefined number (VIDEO_ALERT_BUFS), and is assigned a value of FALSE otherwise (step 244). And the efficient variable is assigned a value of TRUE if the number of bytes free in ring buffer 18a is greater than or equal to a predefined number (EFFICIENT_VIDEO_BUFS) and if the size of the next video frame is greater than or equal to EFFICIENT_VIDEO_BUFS (step 246). Otherwise, the efficient variable is assigned a value of FALSE.
Having determined, in steps 218-246, which channel and hence ring buffer has the fewest bytes and therefore should be filled, PLAY_AV executes either a critical read operation or an efficient read operation depending on the values assigned to the critical and efficient variables. In addition, the execution of the efficient read operation further depends on two factors: 1) whether there is an upcoming transition between clips, i.e., the end of the current clip is near and the viewer 16 will soon need to retrieve data from a different media file 14; and 2) whether the viewer is coming to an end of the ring buffer from which it is reading. If either of these factors is true, the efficient variable is also true. Thus, if the critical and efficient variables are both FALSE (step 248), PLAY_AV assigns the value of “wait” to the action variable and checks several other conditions to determine if some other value should be assigned to the action variable (step 250). (The conditions are reproduced below in Boolean notation below for ease of understanding).
if (!critical && !efficient)
{
action = wait;
if (!videoEOF && vbufsFree >= VID_MIN_READ &&
( (vbufsFree >= EFFICIENT_VIDEO—
BUFS)
|| ( (nextVideoTA < EFFICIENT—
VIDEO_BUFS) && (nextVideoTA > 0))
)
)
{
action = read_VIDEO;
}
if (action == wait && !audio1EOF
&& (aring1.abytesFree >= 735)
&& ( (aring1.abytesFree >= EFFICIENT—
AUDIO_BYTES) ||
(
(nextAudio1TA*TYPICAL—
AUDIO_BUFFER_SIZE <
EFFICIENT_AUDIO—
BYTES)
&& (nextAudio1TA > 0)
)
)
)
{
action = read_AUDIO1:
}
if(action != read_VIDEO && !audio2EOF
&& (aring2.abytesFree > 735)
&& ( (aring2.abytesFree >= EFFICIENT—
AUDIO_BYTES) ||
(
(nextAudio2TA*TYPICAL—
AUDIO_BUFFER_SIZE <
EFFICIENT_AUDIO—
BYTES)
&& (nextAudio2TA > 0)
)
)
)
{
if (action == wait)
action = read_AUDIO2:
else /* action is read_AUDIO1 */
{
/*
* Could do either A1 or A2
stuff.
* Do the one with the most
empty ring buffer.
*/
if aring2.abytesFree > aring1.
abytesFree)
action = read_AUDIO2:
/* if not, then action is already
read_AUDIO1. */
}
}
} /* end analysis for non-critical. non-efficient reads */
Depending on the outcome of the analysis above, the action variable has one of three values: read_VIDEO, read_AUDIO1, or read_AUDIO2. In the case of read_VIDEO, PLAY_AV assigns to the variable “vid-Trigger” a number of bytes to read from the media file 14 (step 252). However, if that number exceeds the number necessary to match the number of bytes contained in the audio channels, PLAY_AV adjusts the number downward (step 254) so that viewer resources are not tied up reading an excessive amount of video data. (See the discussion of staging above.) Finally, PLAY_AV retrieves the video bytes from the media file and transfers them to the ring buffer 18a (step 256).
In the case of read_AUDIO1, PLAY_AV assigns to the variable max_read a number of bytes to read from the media file 14 (258). However, if that number exceeds the number of bytes contained in audio channel 2, PLAY_AV adjusts the number downward (step 260) so that viewer resources are not tied up reading an excessive amount of audio data. Finally, PLAY_AV retrieves the audio bytes from the media file 14 and transfers them to the ring buffer 18b (step 262).
In the case of read_AUDIO2, PLAY_AV assigns to the variable max_read a number of bytes to read from the media file 14 (step 264). However, if that number exceeds the number of bytes contained in audio channel I, PLAY_AV adjusts the number downward (step 266) so that viewer resources are not tied up reading an excessive amount of audio data. Finally, PLAY_AV retrieves the audio bytes from the media file 14 and transfer them to the ring buffer 18c (step 268).
Having determined into which ring buffer to read data and done so (steps 218-268), PLAY_AV next checks several conditions which might cause the display to stop (step 270), e.g., the viewer reached the end of file for the video data, the viewer reached the end of file for the audio1 or audio2 data, or the user interrupted the display. Finally, PLAY_AV selects the current frame from one of the ring_buffers (step 272), and sends a pointer to the frame to the appropriate hardware (step 274), i.e., the video accelerator card 22 or the sound accelerator card 24 depending on whether the frame is video or audio. The hardware plays the frame (step 276) and then interrupts the software (step 278), i.e., PLAY_AV, which then repeats the above described process.
In order to ensure that the audio and video stay in synch, it is essential that the system read the correct number of audio bytes of data corresponding to the video frame being played. This is especially important where the audio track was digitized independently of the video track. To ensure synchronization, when any audio is digitized, the system stores away in the audio media file the number of video frames associated with that audio. Then, later, when a request for a certain number of frames of audio is made, the system can form a proportion against the original number of video frames and audio bytes to find the correct number of audio bytes needed to agree with the number of video frames in the current request.
To ensure efficiency when playing video that has been captured at less than 30 frames per second, the system stores a capture mask with any video media file that has been captured at this lower rate. The capture mask consists of a sequence of 0's and 1's. There are m one-bits and a total of n bits all together, to indicate that only m video frames are present out of every n. When playing this video, the system successively rotates this capture mask one bit to the left. If the high order bit is a 1, this means this is a new frame and we play it. If the bit is a 0, this means this is a repeated frame and we need not play it. The capture mask always ends with a 1, so when it shifts into a word of all 0's, we reload the capture mask.
The attached microfiche appendix (incorporated herein by reference) embodies the viewer module 16 of FIG. 1. The programming language and compiler used are THINK C version 3.01 by Symantec Corporation, and the computer used is the Macintosh II running under Mac OS version 6.0.2.
Portions of the disclosure of this patent document, including the appendix, contain material which is subject to copyright protection and as to which copyright is claimed. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document as it appears in the Patent and Trademark Office files, but otherwise reserves all copyright rights whatsoever, for example, including but not restricted to the right to load the software on a computer system.
Other embodiments are within the following claims.
Peters, Eric C., Rabinowitz, Stanley
Patent | Priority | Assignee | Title |
10362550, | Oct 31 2013 | AT&T Intellectual Property I, L.P. | Synchronizing media presentation at multiple devices |
10805894, | Oct 31 2013 | AT&T Intellectual Property I, L.P. | Synchronizing media presentation at multiple devices |
Patent | Priority | Assignee | Title |
3609227, | |||
4536836, | Nov 27 1981 | Storage Technology Corporation | Detection of sequential data stream |
4667286, | Dec 20 1984 | Advanced Micro Devices, Inc. | Method and apparatus for transferring data between a disk and a central processing unit |
4698664, | Mar 04 1985 | Apert-Herzog Corporation | Audio-visual monitoring system |
4882743, | Aug 01 1988 | American Telephone and Telegraph; AT&T Bell Laboratories | Multi-location video conference system |
4949169, | Oct 27 1989 | International Business Machines Corporation | Audio-video data interface for a high speed communication link in a video-graphics display window environment |
4956768, | Feb 25 1987 | Etat Francais, Centre National d'Etudes des Telecommunications | Wideband server, in particular for transmitting music or images |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Oct 04 2001 | Avid Technology, Inc. | (assignment on the face of the patent) | / |
Date | Maintenance Fee Events |
Date | Maintenance Schedule |
Feb 26 2011 | 4 years fee payment window open |
Aug 26 2011 | 6 months grace period start (w surcharge) |
Feb 26 2012 | patent expiry (for year 4) |
Feb 26 2014 | 2 years to revive unintentionally abandoned end. (for year 4) |
Feb 26 2015 | 8 years fee payment window open |
Aug 26 2015 | 6 months grace period start (w surcharge) |
Feb 26 2016 | patent expiry (for year 8) |
Feb 26 2018 | 2 years to revive unintentionally abandoned end. (for year 8) |
Feb 26 2019 | 12 years fee payment window open |
Aug 26 2019 | 6 months grace period start (w surcharge) |
Feb 26 2020 | patent expiry (for year 12) |
Feb 26 2022 | 2 years to revive unintentionally abandoned end. (for year 12) |