movement through multi-dimensional space is simulated using a series of panoramic images which are projected or displayed in sequence. The user's direction of view, that is the selected view window, is maintained as the series of images is projected or displayed. Motion in directions other than forward or reverse is simulated by utilizing "branch" points in the sequence. Each path from a branch point simulates motion in a different direction. Branch points are generally indicated to a viewer by visual indicators called "hot spots"; however, branch points may also be hidden and activated in response to the viewer's selected direction of view. If a branch point is indicated by a visual indicator, a user can select motion in a desired direction by clicking on a "hot spot". In order to conserve storage space, the image representing each panorama can be stored in a compressed format. Only the portion of the panorama necessary to create a "view window" that is, the portion of the image displayed in response to the user's direction of view, is decompressed at view time. An index methodology is used to store the panoramic images. Use of the indexing methodology allows the images to be retrieved in both the forward and backward direction to simulate movement in either direction. Sound is provided in a special format, so that special effects can be provided based on the user's point of view and dependent upon the direction of motion selected by the user.
|
1. A system for simulating movement through multidimensional space comprising in combination,
a multi-lens camera for simultaneously capturing a plurality of digital images that cover the entire spherical view field, compression units for individually compressing said images into compressed images, means for transferring said images to a computer, a program operated by said computer which seams said images into panoramas and which links said panoramas into a fixed sequence of panoramas, a viewer which selectively automatically displays at a frame rate that creates a simulation of motion a portion of each of said linked panoramas in said fixed sequence whereby movement is simulated.
8. A method of simulating movement through multidimensional space comprising the steps of:
capturing a series of sets of individual images, each set of images covering at least a portion of a spherical view, individually compressing said images, transferring said images to a computer, decompressing said images, seaming the images in each set of images into a panorama, linking said panoramas into a fixed sequence of panoramas, compressing said panoramas without using any inter-frame compression, de-compressing at least a portion of each panorama which corresponds to a view window, and automatically displaying at a frame rate of at least 15 frames per second, said view windows in said fixed sequence of panoramas, whereby movement is simulated.
14. A three dimensional (3-D) panorama movie for enabling a user interactively to view movement through a three-dimensional space along a path through a series of viewpoints and to view in any viewing direction in the three-dimensional space, the 3-D panorama movie comprising:
a computer storage medium for storage of machine readable image data; a file of machine readable image data stored on the storage medium, the image data including a plurality of panorama frames forming a fixed sequence of panoramic images of the three dimensional space in which each panoramic image in the sequence has a spatially different viewpoint; a panframe directory stored on the storage medium in association with the file of machine readable image data, containing a set of frame indexes, each frame index identifying a location in the file of one of the panorama frames, and automatically displaying in sequence at a frame rate of at least 15 frames per second, a portion of each panoramic image is said sequence whereby motion is simulated.
25. A panorama movie method for simulating movement through multidimensional space comprising the steps of:
capturing a plurality of images consisting of imagery taken from a plurality of spatial positions in multidimensional space; seaming said images into key-frame panoramas, storing said key-frame panoramas in a file in a storage medium; indexing said key-frame panoramas within a key-frame directory according to a position of each key-frame within the file and storing said key-frame directory on said storage medium; displaying a portion of a first key-frame panorama according to a user position and viewing direction within the multidimensional space; accessing said key-frame directory to determine a next key-frame image to be displayed according to a user travel speed and travel direction; and automatically displaying at a frame rate that is fast enough to simulate motion a portion of a second key-frame panorama subsequent to the display of said first key-frame panorama according to a user position and viewing direction whereby motion is simulated.
2. The system recited in
3. The system recited in
4. The system recited in
5. The system recited in
6. The system recited in
7. The system recited in
9. The method recited in
10. The method recited in
11. The method recited in
12. The method recited in
15. A panorama movie according to
16. A panorama movie according to
17. A panorama movie according to
18. A panorama movie according to
19. A panorama movie according to
20. A panorama movie according to
21. A panorama movie according to
22. A panorama movie according to
23. A panorama movie according to
24. A panorama movie according to
26. The method of
27. The method of
28. The method of
29. The method of
30. The method of
31. The method of
33. The method of
34. The method of
35. The method of
|
This application is a non-provisional application of provisional application No. 60/085319 filed May 13, 1998.
This application includes a three part microfiche appendix which gives code for computer programs: There are a total of 36 microfiche with a total of 2451 frames. Appendix D1 has 12 microfiche which contain 787 frames. Appendix D2 has 12 microfiche which contain 829 frames. Appendix D3 has 12 microfiche which contain 835 frames.
The present invention relates to photography, digital image processing and to computer graphics. More particularly the present invention relates to a method and system for providing a viewer with a multidimensional view which simulates movement through space or time.
A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.
Motion is usually simulated by means of single view movies. Single view movies consist of a series of single photographs which are sequentially projected on a screen. At any time, a single photograph is entirely projected on the screen. Some movie theaters have screens which partially surround the viewer, so that a viewer can turn and look at a different portion of the screen as the movie progresses However, irrespective of where the viewer's attention is focused, in fact the entire image is being projected on the screen. With some equipment single view movies can be stopped and reversed; however, again at a particular time a selected frame is entirely displayed. In summary, traditional movies do not allow a viewer to control the portion of the image which is projected on the screen. Stated differently, with traditional movies a viewer can not control the "view window" through which each image is viewed.
It should be noted that as used herein the term "screen" refers to either a traditional screen onto which an image is projected or an electronic display which projects or displays an image in such a manner that the image can be seen by a viewer.
The technology for producing panoramic images and photographs is well known. Panoramic images are images which represent the visual surroundings from a single location (or view point) of a particular 3D environment. Panoramic images can be photographic, computer-generated (CG), or a composite of photo and CG imagery. Equipment is available which seams together a series of two dimensional conventional images to form a panoramic image. Panoramic images may consist of any field of view such as a full sphere full cylinder, semi-sphere, etc. however, full sphere views are generally preferred. Panoramic images may be in any of the known projection formats, such as equi-rectangular, Mercator, Peters, fisheye, cube, or hemicube, etc. If the field of view is wide enough to warrant, perspective correction may be applied to the portion of a panoramic image displayed in order to remove projection distortions from the given view of the user. Computer programs and systems which allow one to view a selected portion of, to pan, to rotate, etc a panoramic image or photograph in response to the movement of a computer mouse, joystick, keyboard, etc. are commercially available.
Panoramic image (or photographic image) viewing systems are available which provide a number of panoramas (for example views of adjacent rooms in a building) and which allow a user who is viewing one of the rooms to "click" on a door and to thus bring up the panorama of the next room, thereby, in some sense simulating movement into the next room. However, each of the panoramic views in such systems is static and some explicit action on the part of the viewer is required to move on to a different panorama.
U.S. Pat. Nos. 5,023,725 and 5,703,604 describe, and Dodeca L.L.C. located in Portland Oregon commercially markets, a system for capturing images using a multi lens camera, for seaming the images into panoramas, and for viewing selected portions of the panoramic images.
Other panoramic image or photographic viewing systems are available which initiate a conventional single view motion picture when a user "clicks" on a spot such as a door in a first panoramic view. The single view movie simulates movement into a different location and at the end of the movie, the user is presented with a second panoramic view.
A company named "Warp" located in the Kingdom of Tonga has demonstrated a system wherein a video sequence is captured using a video camera with a fisheye lens which is pointed in the vertical or "up" direction (see, VTV Software Development Kit Reference Manual 2.01 Win95 1996). This approach provides a hemispheric movie in which the user may "pan around" while the movie is playing. In the system demonstrated by Warp, the user views the movie "in sequence", meaning that each frame of the movie is played according to the temporal sequence in which the hemispheric movie was captured. The system demonstrated by Warp was limited to sub-spherical panorama dimensions and the camera was located at a fixed position.
Realistic simulation of movement from one location to another can be provided by three dimensional computer modeling systems, such as those used in some flight simulators. However, such systems are very computationally intensive
The present invention simulates movement through multi-dimensional space using a series of panoramic images which are projected or displayed in sequence. The user's direction of view, that is the selected view window, is maintained as the series of images is projected or displayed. Motion in directions other than forward or reverse is simulated by utilizing "branch" points in the sequence. Each path from a branch point can simulate motion in a different direction. Branch points are generally indicated to a viewer by visual indicators called "hot spots"; however, branch points may also be hidden and activated in response to the viewer's selected direction of view. If a branch point is indicated by a visual indicator, a user can select motion in a desired direction by "clicking" on a "hot spot". In order to conserve storage space, the image representing each panorama can be stored in a compressed format. If the images are stored in compressed format, in order to conserve time and processing power, when an image is displayed, only the portion of the panorama necessary to create a "view window" that is, the portion of the image displayed in response to the user's direction of view, is decompressed. Furthermore, the images are stored in a format that does not utilize inter-image compression (such as that used by the MPEG format). Since the images are stored in a format that does not utilize inter-image compression, it is possible to simulate motion in both the forward and backward direction without operating on a series of decompressed images.
An index methodology is used to store the panoramic images. Use of the indexing methodology allows the images to be retrieved in both the forward and backward direction to simulate movement in either direction.
Sound is provided in a special format, so that special effects can be provided based on the user's point of view and dependent upon the direction of motion selected by the user.
Appendix A is a print out of computer code for retrieving images and correcting the perspective of images in a pan movie.
Appendix B is a sample of link control file for a pan movie.
Appendix C is a print out of computer pseudocode for linking sequences of images to form a pan movie.
Appendix D (on microfiche) is a printout of computer code for the following three computer programs:"
1) A program which seams images and builds pan movies. This program takes six single view images and seams then into a panorama, and then connects a series of panoramic images into a pan movie.
2) A program which inserts hot spots into panoramas.
3) A program which displays a series of panoramic images as a pan movie.
In order to simulate movement through multi-dimensional space, one must first capture a series of panoramic images, the panoramic images must be stored as frames and then the appropriate view window from selected frames must be displayed in an appropriate sequence.
A panoramic image provides data concerning what is visible in any direction from a particular point in space. At any particular time a viewer or user can only look in one direction. The direction or point of view of a viewer or user determines the "view window", that is, the part of a panoramic image which is projected on a screen at a particular time.
The operations performed by the units in
The stored images are manually transferred to off line computer 21 which is programmed to perform the operations shown in FIG. 3C. First the images are decompresses as indicated by block 20a so that they can be manipulated. Next the single view images are seamed into a panorama and transformed to equirectangular format as indicated by block 21b. Hot spots which indicate break points in a sequence of images and sound tracks are added next as indicated by block 21c. Finally the images are compressed as indicated by block 21d and stored with an index file as indicated by block 21e. Each panorama is termed a "key frame". A series of key frames (or more precisely a sequence of view windows) projected in sequence is a pan movie.
A viewer program in viewer computer 22 is used to view the pan movies. The viewer 22 displays in sequence a series of images, that is, a series of key frames. For each key frame displayed the viewer 22 determines an appropriate view window as indicated by block 22a. The portion of the key frame which corresponds to the view window is then de-compressed and displayed as indicated by block 22b. As indicated by block 22c, sound is played if appropriate.
It is noted that the operations indicated by blocks 20a, 20b, 21a to 21e, 22a, 22b, and 22c are implemented by means of computer programs which perform the functions shown. Computer programs are given in appendices A, B, C, and D.
The sequence of images begins at the point or at the key frame indicated by the letter A and the sequence proceeds to the point or key frame indicated by the letter B. At this point the viewer can select to either go toward point C or toward point D. The selection may be made by "clicking" on a designated "hot spot" in the panorama designated B or it may be made depending on some other criteria or action by the user. An important point is that at the branch point B, the direction of view (indicated by the direction of the arrows) remains the same irrespective of which path of travel is chosen. The view from the first frame after the branch point will be almost identical in both paths. As time progresses and the viewer moves further from the branch point, the view will gradually change. This is the effect that a person experiences when one arrives at a dividing point in a path. When a person takes the first step on a branching path, the persons field of is view remains practically identical.
It is noted that at branch point B, the arrows are not pointing in the direction of the path leading to point D. Normally, a viewer would be looking in the direction of a branch point when the viewer selects to travel in the direction of the branch point. Thus, a viewer looking in the direction of the arrows shown in
Sequences of key frames can either be joined at branch points such as branch point B or alternatively a branch point may be located at the end of a sequence of key frames. That is, a branch point may be located at the terminal frame of a sequence of key frames. Such a branch point may have two alternative sequences, one of which can be selected by a user by clicking on one of two hot spots. Alternatively at the end of a sequence of key frames, there can be an implicit branch point. At such an implicit branch point a new sequence of frames would be selected by the system without any action by the user.
There is a one to one ratio of key frames to possible user positions. Hence, there exists a correlation between frame rate and user motion speed. If the user is moving through the environment, every frame displayed is a new key frame. The faster the frame rate for a given frame spacing, the faster the user travels. Given a fixed frame rate, the user's travel speed may be dictated by the relative spacing of key frames. The closer the key frames are, the slower the user will travel. For example, for a travel speed of approximately 5 mph and a playback frame rate of 15 fps, individual panoramic frames should be captured at about 6 inch increments. The math is as follows: (5 miles/hour*63,360 inches/mile)/(3600 sec/hour*15 frames/sec)=6 inches per frame. When the movie is being displayed, speed of travel can be increased by skipping some of the frames (for example if every other frame is skipped the speed of travel is doubled). Skipping frames reduces the rate at which frames need be sent to the viewer and thus reduces the bandwidth required.
In addition to the spacing of key frames to achieve different travel speeds, the orientation of individual key frames may be adjusted in order to achieve a desired motion effect, such as gate, slumber, waddle, crawl, skip, etc. The orientation of a key frame is defined to be the default view (or point of focus) of the user within the panoramic image if no other point of view is specifically selected.
Sound can accompany the visual effect provided by pan movies.
In the preferred embodiment the connection from camera unit 10 to computer 20 and from computer 20 to real time viewer 30 is a "HOTlink" serial bus. Such connections are commercially available from suppliers such a Cypress Semiconductor Corp. or from Dataforth Corporation which is a division of Burr-Brow Company. Alternatively other types of high speed connections could be used. For example the connection could be a standard SCSI connection.
Each of the lenses 41b to 41f has a set of associated components similar to the components associated with lens 41a. The following will discuss the components associated with lens 41a. It should be understood that the other lenses 41b to 41f have a similar sets of components.
The image from lens 41a is focused on a CCD (Charge Coupled Device) array 43a. CCD array 43a captures the image from lens 41a and send this image to embedded computer 45. CCD arrays 43a is controlled and operated by embedded computer 45. By resetting and reading the CCD array 43a in a particular time period, the embedded computer 45 in effect controls or provides an electronic shutter for lens 41a. The electronic shutters associated with each of the lenses 40a to 40f open and close simultaneously. Each CCD array 43a to 43f captures 30 images per second under normal operation.
The output of CCD array 43a is fed into a JPEG data compression chip 44a. Chip 44a compresses the image from lens 41a so that the image can be more easily transmitted to computer 40. The output from compression chip 41a fed to an embedded controller 45 which transmits signals to computer 40 on a serial time slice basis.
The lenses 41a nd the CCD arrays 43, and are similar to the components found in commercially available digital cameras. JPEG compression chips 44 and embedded computer 45 are also commercially available components. For example such components are available from suppliers such as Zoran Corporation or Atmel Corporation.
The electronic shutters associated with lenses 41 operate at 30 cycles per second and hence computer 21 receives six images (one from each lens) each {fraction (1/30)}th of a second. The six images received each {fraction (1/30)}th of a second must be seamed and transformed to equirectangular format to form one panorama as indicated by step 21b in FIG. 2.
While the specific embodiment of the invention shown herein utilizes a digital camera to take the initial single view images which are compressed and sent to computer 20 for storage, it should be understood that one could use a variety of other types of cameras to take these initial images. For example the images simultaneously taken from a number of lenses could be recorded on tape for later processing by off line computer 21.
The seaming operation is done by the program in computer 21. In general the seaming operation connects the individual images into a panoramic image by finding the best possible fit between the various individual images. The process of seaming images into a panoramic image is known. For example U.S. Pat. No. 5,694,531 describes seaming polygons into a panorama which has a low root-mean-square error. A computer program which can seam the six images from lenses 41a to 41f of camera 20 into a panorama is given in Appendix D.
After the seaming operation is complete each seamed image is a panoramic image (called a panorama) and each panorama is a frame of a pan movie. Prior to storage the seamed images are compressed so as that the file size will be manageable. A commercially available compression program known as "Indeo" is used to compress the images. The Indeo program was developed by and is marketed by the Intel Corporation. The Indeo compression program provides a mode of operation which does not utilize any inter-frame compression. The no inter-frame compression mode of the Indeo program is used with the present embodiment of the invention. Since there is no inter frame compression, the key frames can be accessed and viewed in either the forward or the reverse direction. Furthermore, only the portion of a panorama required for a particular view window is decompressed, thereby saving time and computational resources.
The compressed panoramic images are stored in files on computer disks, tape or compact discs (CDs). Each file includes a header and an index as shown in FIG. 7. The header includes information such as the following:
File Type Tag:
File Size: (total bytes used by the file)
Index Size: (Number of entries in frame Index)
Max Frame Size: (total bytes used by largest compressed frame)
Codec: (Codec used to compress frames.
After the file header, a frame index is provided (see FIG. 7). Each frame index points to the location of the associated frame as indicated by the arrows in FIG. 7. Thus, individual frames can be read in any order by obtaining their location from the frame index.
The indexing mechanism would not be necessary if the key frames were always going to be used in frame order. However, in the present embodiment, the system can play the key frames which comprise the pan movie in either forward or backward direction. Hence the system must be able to locate individual frames quickly in any order. Furthermore, it is desirable that the system be able to locate a key frame with only a single disk access. Consider the situation were the user is moving "backward" (in the opposite direction of the key frame disk storage) at a fast travel speed (to increase speed of movement some key-frames are skipped). Without a key frame directory, the disk would have to be searched in a "reverse-linear" manner in order to find and load the next appropriate key frame. With a key frame directory, the next key frame location is located immediately, and loaded with a single disk access (given the directory itself is stored in RAM memory).
As indicated in
It is noted that not all visually apparent alternate paths visible in any panorama are actually available as a pan movie branch. For example, at a street intersection, branches may not be provided to all visible streets. Care must be taken to insure that a viewer is given an indication of the branch points that are actually available to the viewer.
At a playback rate of 30 frames per second a user would have to be very "fast" (i.e. it would in fact be practically impossible) for a viewer to see and click on a hot spot that appears on a single frame. Without advanced notice, the viewer would have great difficulty actually taking a specific action to activate a branch during a specific single frame since in normal operation a particular frame is only displayed for about {fraction (1/30)}th of a second. In order to be effective and user friendly a user must be given an early indication of a upcoming branch opportunity that requires user action. A hot spot in a pan movie must be visible by a viewer in a relatively large number of key frames. For example a hot spot might be visible in the thirty key frames that precede (or follow for reverse operation) a branch point.
Hot spots are inserted into a pan movie in the manner illustrated in FIG. 8. The hot spots are inserted into the key frames by computer 21b before the frames are compressed as indicated by blocks 21c and 21d in FIG. 3C. It is noted that hot spots may be inserted into a pan movie by altering the original panoramic image so that it includes the hot spot or alternately by providing a overlay image which contains the hot spot image. If an overlay is used, the overlay image needs be projected at the same time as the original image. As indicated by block 87a one must first determine how much in advance one wants to warn the user. If a hot spot is to have a particular size at the time action is needed, when viewed in advance (i.e. from a distance) the hot spot will be much smaller. As indicated by block 87b, in order to insert hot spots in a pan movie, one must select the region where the hot spot is to be located. In general this will be in a view looking toward the direction where the branch will take place. The hot spot is then inserted into the panorama by modifying the images. The hot spot may be indicated by a light colored outline superimposed over the region. The area within the outline may be slightly darkened or lightened. The object is to highlight the region without obscuring the image itself. Various other alternative indications can also be used.
The process repeats as indicated by blocks 87d and 87e until the key frame at the branch point is reached. Finally the process is repeated from the opposite direction from the branch point so that the branch point will be visible if the pan movie is shown in the reverse direction.
The changes to the individual key frames may be made manually with a conventional image editor, or the process can be automated by a program designed just for this purpose. Computer code which implements a hot spot editor by performing the steps shown in
In order to avoid unnecessary user intervention, "hidden" hot spots may be added to connect multiple pan movies. A hidden hotspot is one that does not need to be manually selected by the user. With a hidden hot spot, if the user "travels" into a particular key frame which has a hidden hot spot, and the user is "looking" in the hot spot's general direction, then the system will react based upon the user's implicit selection of the hotspot and the user will be sent along the path directed by the hot spot.
As a user travels, the next required key frame is determined by the current user position and direction of travel. The location of this key frame within the file of images is determined via the file index directory; and the key frame is loaded into RAM memory, decompressed, and displayed. To increase performance, only the view window (depending on current user view) portions of the key frame need be loaded into RAM. If for ease of programming the entire key frame is loaded into memory, only view window portions of the key frame need be decompressed. If the entire key frame is compressed as a whole, then a de-compressor supporting "local decompression" is more efficient, e.g., Intel Indeo. To determine the portion of the panorama needed to display a particular view, each of the corner coordinates of the perspective view plane (display window) is converted to panorama coordinates. The resulting panorama coordinates do not necessarily represent a rectangle, therefore the bounding rectangle of these panorama data is needed to derive a perspective view at a given view orientation.
Once the corners of the desired bounding rectangle are determined the Indeo de compression program is instructed to decompress only that portion of the key frame needed for the particular view window. In order to do this, the program must call the Video For Windows function ICSetState prior to decompressing the frame. The C code to accomplish this follows.
#include "windows.h" |
#include "vfw.h" |
#include "vfw_spec.h" |
extern HIC hic; // Opened CODEC (IV41); |
extern RECT *viewRect; // Determined elsewhere |
static R4_DEC_FRAME_DATA StateInfo; |
void SetRectState |
( |
HIC hic; // Opened CODEC (IV41); |
RECT *viewRect; // Local Rectangle of interest |
) |
{ |
R4_DEC_FRAME_DATA StateInfo; |
memset(&StateInfo,0,sizeof(R4_DEC_FRAME_DATA)); |
StateInfo.dwSize = sizeof(R4_DEC_FRAME_DATA); |
StateInfo.dwFourCC = mmioStringToFOURCC("IV41",0); |
// Intel Video 4.1 |
StateInfo.dwVersion = SPECIFIC_INTERFACE_VERSION; |
StateInfo.mtType = MT_DECODE_FRAME_VALUE; |
StateInfo.oeEnvironment = OE_32 |
StateInfo.dwFlags = DECFRAME_VALID | DECFRAME-- |
DECODE_RECT |
StateInfo.rDecodeRect.dwX = min(viewRect->left,viewRect->right); |
StateInfo.rDecodeRect.dwY = min(viewRect->top, viewRect-> |
bottom); |
StateInfo.rDecodeRect.dwWidth = abs((viewRect->right-viewRect-> |
left)+1); |
StateInfo.rDecodeRect.dwHeight = abs((viewRect->bottom- |
viewRect->top)+1); |
ICSetState(hic,&StateInfo,sizeof(R4_DEC_FRAME_DATA)); |
} |
If the projection used to store the pan-frame is such that there exists a discontinuity in pixels with respect to the spherical coordinates they represent, then the local region required may be the combination of multiple continuous regions. For a full cylinder/sphere equirectangular projection (centered about 0 degrees), the left pixel edge represents -180 degrees and the right pixel edge represents 180 degrees. In spherical coordinates, -180 degrees is the same as 180 degrees. Therefore, the discontinuous left/right pixels represent a continuous "wrap-around" in spherical coordinates.
The math to determine the portion of the source key-frame panorama needed for a particular view window depends on the projection used to store the panorama. Optionally, the viewer may predict the next key-frame to be loaded (depending on user travel direction and speed), and pre-load it in order to increase performance. For an equirectangular projection of a full sphere panorama frame, the equations for determining the required portion are as follows: where:
Scaler variables are lower case, vectors are bold lower case, and matrices are bold upper case.
Panorama point (s,t) is derived from any perspective plane point (u.v).
The perspective plane has a focal length I from the center of projection.
In addition, the perspective plane can be arbitrarily rotated through a given view orientation, namely heading, pitch, and bank (h,p,b).
Any point in the perspective plane is specified by the 3D vector:
w=<u,v,I>
The rotations are applied by using a standard matrix-vector product. The three matrices accounting for Heading, Pitch and Bank are as follows:
H = | |cos(h) | 0 | sin(h) | | | |
|0 | 1 | 0 | | | ||
|-sin(h)0 | cos(h) | | | |||
P = | |1 0 | 0 | | | ||
|0 cos(p) | -sin(p) | | | |||
|0 sin(p) | cos(p) | | | |||
B = | |cos(b) | sin(b) 0 | | | ||
|-sin(b) | cos (b) 0 | | | |||
|0 0 | 1 | | | |||
The vector w is rotated using the above matrices to attain w' like such"
w'=H*P*B*w
The final step is converting from rectangular to spherical coordinates. Denoting the 3 components of the vector w' as x, y, z, then the conversion is:
s=atan2(x,z)
t=atan2(y, sqrt(x*x+z*z))
Note: atan2(a, b) is a standard C-function very similar to atan(a/b), but atan2 correctly handles the different cases that arise if a or b is negative or if b is 0.
Optionally, the viewer may predict the next key-frame to be loaded (depending on user travel direction and speed), and pre-load this key frame in order to increase performance.
Due to the one to one ratio of key frames to possible user positions, there exists an exact correlation between frame rate and user motion speed. If the user is currently moving through the environment, every frame displayed is a new key frame, thus the faster the frame rate, the faster the user travels. For this reason, the frame rate is "capped" during user travel to eliminate the problem of excessive user travel speed. In order to retain smooth motion, the frame rate is not decreased to below standard video frame rates (15 frames/sec.) The frame rate is not increased in order to keep the relative spacing of key frames to a manageable distance; the faster the frame rate, the closer the key frames must be to achieve the same user travel speed. The viewer may optionally skip key-frames in order to increase the user's travel speed through the environment. The more key-frames skipped, the faster the user will travel; if no key-frames are skipped, the user will travel at the slowest possible rate (given a constant frame rate.)
The system can link pan movie segments so as to permit branching and thereby follow a path selected by a user. Multiple linear (one dimensional) pan movies may be linked together to create a "graph" of pan movies (see appendix B). For each pan movie, the end of one segment may be associated with the start of a "next" pan movie. This association (in conjunction with the length of the individual pan movies) is the basis for the graph shape. In order to achieve smooth transitions, the "last" frame in the "first" pan movie must be the same as (or one frame off from) the "first" frame of the "next" pan movie. In addition to positional correctness, the relative view orientations of the joining frames must be known. For example, if the "last" frame of the "first" pan movie faces "north", and the "first" frame of the "next" Pan Movie faces "east", then the viewing software must be alerted to this orientation change. Without this information, there would be a 90 degree "snap" in the transition between the two Pan Movies. All this graph information may be stored in a separate file (text or binary form.)
The audio information associated with each frame of a pan movie must take into account the fact that a viewer of a pan movie has a great deal of control over what is presented on the screen. In addition to the ability to select branch points a user may choose to change the direction of view or to stop and backup. The audio information associated with each key frame must accommodate this flexibility.
As illustrated in
The control information that is recorded with each frame controls certain special effects. For example the control information on one frame can tell the program to continue playing the audio tracks from the following frame even if the user has stopped the movie at one particular frame. As the sound track on each frame is played, the control information on that frame is interrogated to determine what to do next. What sound is played at any particular time is determined by a combination of the control information on the particular frame being viewed and the action being taken by the viewer at that time. From a programming point of view, the commands associated with each rack are de-compressed and read when the view window for the associated frame is de-compressed and read. As a particular view window is being displayed (or slightly before) the commands stored in the control field are read and executed so that the appropriate sound can be de-compressed and played when the view window is displayed.
For example the control information could provide the following types of commands:
Stop this audio track if user stops pan movie here (typical setting). If this is not set the audio will continue playing in same direction until audio for this track ends
Start or continue to play this audio track if user is viewing pan movie in forward direction (typical setting)
Start or continue to play this audio track backwards if user if viewing pan move in a backwards direction. (note if the same audio information is played is reverse it may be distorted)
Start this audio track when image frames are in motion and being played in a reverse direction. This allows high quality audio to be played while reverse viewing
Continue audio track from / on other file structure (branch most likely has occurred) modify volume This is used to fade out an audio track that may have played ahead earlier
Stop all audio tracks
Stop this audio track if user slows pan movie playback
Start audio file X: where X is a conventional audio file that is separate from the pan movie.
A wide variety of other commands may be implements as desired by the designer of a particular movie.
The audio information can be recorded with a normal recorder when the initial images are recorded or it can be recorded separately. The audio data is merged with the key frames by computer 21. This can be done manually on a frame by frame basis or the process can be automated. When the sound is merged with the key frames the appropriate control information is added.
The attached appendices provide computer programs which implement various aspects of the present invention. These programs are designed to run under a conventional operating system such as the "Windows" operating system marketed by the Microsoft Corporation.
The program given in Appendix A will retrieve frames for a move, correct the perspective in accordance with known equations and then display the images of the movie in sequence.
Appendix B is an example of a link control file for the frames of a pan movie. Appendix C is pseudocode showing how sequences of images are linked to form a pan movie.
Appendix D (on microfiche) is a printout of computer code for the following previously described three computer programs:
1) A program which seams images and builds pan movies. This program takes six single view images and seams then into a panorama, and then connects a series of panoramic images into a pan movie.
2) A program which inserts hot spots into panoramas.
3) A program which displays a series of panoramic images as a pan movie.
It is noted that in a pan movie the frames do not all have to have the same resolution. Some frames may be of a higher resolution. For example, at the most interesting places in the Pan Movie may have a higher resolution.
Many alternative embodiments of the invention are possible. For example, the initial capture process could record the images on video tape rather than recording the images digitally. Electronic cameras could be used which include image capture devices other than CCD arrays to capture images. Branching can provide three or more optional paths rather than just two pats as shown in
It is noted that in alternative embodiments, compression schemes or techniques other than Intel Indio can be used. Furthermore, alternative embodiments could use no compression at all if enough storage and bandwidth were available.
While in the embodiment shown the images files are manually transferred between some of the units in the embodiment shown, in alternative embodiments these files could be transferred between units by electronic connections between the units.
While in the embodiment described above the camera has six lenses which record all six sides of the cube, in alternative embodiments the camera could record less than an entire sphere. For example the lens pointing down could be eliminated in some embodiments. Still other alternative embodiments could use lenses with wider or narrower fields of view. For example less lenses each with a wider field of view could be used. Furthermore, while the embodiment described above utilizes spherical panorama, other types of panoramas could be used. Various types of projections such as cubic could be used instead of equi-rectangular.
The embodiment shown includes a number of sound tracks with each key frame and control information which indicates which sound track should be played when the key frame is displayed depending on whether or not certain special conditions exist. Alternatively, there could be a single sound track associated with each frame. In such an embodiment the single sound track on each key frame could be the sounds recorded when the images is the particular frame were recorded. In other alternatively embodiments, there could be no sound tracks and in such case the images would be displayed without accompanying sound.
Having described and illustrated the principles of the invention in various embodiments thereof, it should be apparent that the invention can be modified in arrangement and detail without departing from the principles of the invention. We claim all modifications and variation coming within the spirit and scope of the following claims.
APPENDIX A: FRAME RETRIEVAL CODE | |||
#include "windows.h" | |||
#include "mmsystem.h" | |||
#include "vfw.h" | |||
#include "vfw_spec.h" | |||
#define S_BMIH sizeof(BITMAPINFOHEADER) | |||
#// Externally declared (and allocated) variables | |||
extern UINT currentFrameNumber; // Current Pan Movie file frame number | |||
(user position) | |||
extern HANDLE hFile; // Open file handle of Pan Movie file | |||
extern HIC hic; // Open IC handle (installed compressor) | |||
extern DWORD *Index; // Pan Movie Frame lndex (read from file at load | |||
time) | |||
extern LPBITMAPINFOHEADER viewFrame; // Buffer large enough to hold | |||
image the size of the display window | |||
extern LPBITMAPINFOHEADER panFrame; // Buffer large enough to hold | |||
largest uncompressed frame | |||
extern LPBITMAPINFOHEADER compressedFrame; // Buffer large enough to | |||
hoid largest compressed frame | |||
// Function prototypes | |||
extern void ViewToPan(int viewWidth,int viewHeight,int panWidth,int | |||
panHeight,float heading,float pitch,float bank,float zoom,POINT *point); | |||
static LPBITMAPINFOHEADER RetrievePanFrame(int frameNumber, RECT | |||
*viewRect); | |||
// | |||
// This function generates a perspectively correct bitmap image given a | |||
user view orientation and travel speed | |||
// | |||
static LPBITMAPINFOHEADER RetrieveViewFrame(float userHeading,float | |||
userpitch,float userBank,float userzoom, int userTravelspeed) | |||
{ | |||
// Determine Decode BoundingBox | |||
POINT point; | |||
RECT localDecompressionRect; | |||
// Upper left corner of viewFrame | |||
point.x = 0; point.y =0; | |||
ViewToPan(viewFrame->biwidth,viewFrame->biHeight, panFrame->biwidth, panFrame->biHeight, | |||
userHeading, userPitch, userBank, userzoom,&point); | |||
localDecompressionRect.top | = point.y; | ||
localDecompressionRect.left | = point.x; | ||
// Upper right corner of viewFrame | |||
point.x = viewFrame->biWidth-1; point.y = 0; | |||
ViewToPan(voewFrame->biWidth, viewFrame->biHeight, panFrame->biWidth,panFrame->biHeight, | |||
userHeading, userPitch, userBank, userzoom,&point); | |||
localDecompressionRect.top | = min(localDecompressionRect.top, point.y); | ||
localDecompressionRect.right | = point.x; | ||
// Lower left corner of viewFrame | |||
point.x = 0; point.y = viewFrame->biHeight-1; | |||
ViewToPan(viewFrame->biwidth,viewFrame->biHeight, panFrame->biWidth, panFrame->biHeight, | |||
userHeading, userPitch, userBank,userzoom,&point); | |||
localDecompressionRect.bottom = point.y; | |||
localDecompressionRect.left | = | ||
min(localDecompressionRect.left,point.x); | |||
// Lower right corner of viewFrame | |||
point.x = viewFrame->biWidth-1; point.y = viewFrame->biHeight-1; | |||
ViewToPan(viewFrame->biwidth,viewFrame->biHeight, panFrame->biwidth, panFrame->biHeight, | |||
userHeading, userPitch, userBank, userzoom,&point); | |||
local Decompression Rect. bottom= | |||
max(localDecompression Rect.bottom,point.y); | |||
localDecompressionRect.right = | |||
max(localDecompressionRect.right,point.x); | |||
// Get Pan Frame (or "userDecompressionRect" portion thereof) | |||
currentFrameNumber += userTravelSpeed; // userTravelSpeed is negative | |||
if traveling backwards | |||
LPBITMAPINFOHEADER pFrame = | |||
RetrievePanFrame(currentFrameNumber,&localDecompressionRect); | |||
if(pFrame == NULL) { | |||
currentFrameNumber -= userTravelSpeed; | |||
return NULL; | |||
} | |||
// A very slow warping routine (assumes 24-bit pixels) | |||
LPBYTE | srcPixels | = ((LPBYTE)pFrame) + S_BMIH; | |
LPBYTE | dstPixels | = ((LPBYTE)viewFrame) + S_BMIH; | |
for(int y = 0; y < viewFrame->biHeight; y++) { | |||
for(int x = 0; x < viewFrame->biHeight; x++) { | |||
point.y = y; point.x = x; | |||
ViewToPan(viewFrame->biwidth,viewFrame->biHeight, pFrame->biwidth,pFrame->biHeight, user | |||
Heading,userPitch,userBank,userzoom,&point); | |||
memcpy(&dstPixels[3*(x + y*viewFrame->biWidth)],&srcPixels[3*(point.x | |||
+ point.y*pFrame->biWidth)],3); // supports 24-Bit Pixels only | |||
} | |||
} | |||
return viewFrame; | |||
} | |||
// This function reads and decompresses a Pan Frame bitmap image from a | |||
Pan Movie file | |||
// | |||
static LPBITMAPINFOHEADER RetrievePanFrame(int frameNumber, RECT | |||
*viewRect) | |||
{ | |||
DWORD d | |||
UINT frameSize = index[frameNumber+ 1]-Index[frameNumber]; | |||
// Set the file pointer to the start of the requested frame and read in | |||
the bitmap header | |||
SetFilePointer(hFile,index[frameNumber],NULL,FILE_BEGIN); | |||
ReadFile(hFile,panFrame S_BMIH,&d,NULL); | |||
if(panFrame->biCompression == 0) { // Uncompressed frame (read rest of | |||
frame and return) | |||
ReadFile(hFile,((BYTE*)panFrame)+S_BMIH,frameSize-S_BMIH &d,NULL); | |||
return panFrame; | |||
} | |||
// Read the remainder of the compressed frame | |||
*compressedFrame | = *panFrame; | ||
ReadFile(hFile,((BYTE*)compressedFrame)+S_BMIH,frameSize-S_BMIH,&d,NULL); | |||
// Set up decompressed bitmap header | |||
panFrame->biCompression | = 0; | ||
panFrame->biSizeImage | = 0; | ||
panFrame->biBitCount = 24; | |||
panFrame->biClrUsed = 0; | |||
LPBITMAPINFOHEADER biSrc = compressedFrame; | |||
LPBITMAPINFOHEADER biDst = panFrame; | |||
LPBYTE srcPixels | = (BYTE*)biSrc + S_BMIH | ||
LPBYTE dstPixels | = (BYTE*)biDst + S_BMIH | ||
// If the frame is compressed with Intel Indeo 4 and a local rect was | |||
requested, then perform local decompression | |||
if(viewRect && biSrc->biCompression == mmioFOURCC(`i`,`v`,`4`,`1`)) { | |||
// Intel Indeo 4.1 | |||
R4_DEC_FRAME_DATA | Stateinfo; | ||
memset(&StateInfo,0,sizeof(R4_DEC_FRAME_DATA)); | |||
StateInfo.dwSize | = sizeof(R4_DEC_FRAME_DATA); | ||
StateInfo.dwFourCC | = biSrc->biCompression; | ||
StateInfo.dwVersion | = SPECIFIC_INTERFACE_VERSION; | ||
StateInfo.mtType | = MT_DECODE_FRAME_VALUE; | ||
StateInfo.oeEnvironment | = 0E_32 | ||
StateInfo.dwFlags | = DECFRAME_VALID | | ||
DECFRAME_DECODE_RECT; | |||
StateInfo.rDecodeRect.dwX | = min (viewRect->left,viewRect->right); | ||
StateInfo.rDecodeRect.dwY | = min (viewRect->top,viewRect->bottom); | ||
StateInfo.rDecodeRect.dwWidth = | |||
abs((viewRect->right-viewRect->left))+1; | |||
StateInfo.rDecodeRect.dwHeight= | |||
abs((viewRect->bottom-viewRect->top))+1; | |||
ICSetState(hic,&StateInfo,sizeof(R4_DEC_FRAME_DATA)); | |||
if(ICDecompressEx(hic,0,biSrc,srcPixels,0,0,biSrc->biWidth,biSrc->biHeight,biDst,dstPixels,0,0,bi | |||
Dst->biwidth,biDst->biHeight) | |||
!=ICERR_OK) | |||
return NULL | |||
} | |||
else { // Decompress entire frame | |||
if(ICDecompressEx(hic,0,biSrc,srcPixels,0,0,biSrc->biWidth,biSrc->biHeight,biDst,dstPixels,0,0,bi | |||
Dst->biwidth,biDst->biHeight) | |||
!=ICERR_OK) | |||
return NULL; | |||
} | |||
return panFrame; | |||
} | |||
APPENDIX B: SAMPLE PAN MOVIE LINK CONTROL FILE | |||
<-----------------.----------------> | |||
<-C | | | B -> | |
| | |||
| | |||
| | |||
| | |||
|A | |||
| | |||
| | |||
[Segment-A (start)] | |||
File= | "A.pan" | ||
North= | 0 | ||
[Segment-A (end)] | |||
File= | "A.pan" | ||
North= | 0 | ||
Link 90= | "Segment-B (start)" | ||
Link 270= | "Segment-C (start)" | ||
[Segment-B (start)] | |||
File= | "B.pan" | ||
North= | 90 | ||
Link 90= | "Segment-A (end)" | ||
Link 180= | "Segment-C (start)" | ||
[Segment-B (end)] | |||
File= | "B.pan" | ||
North= | 90 | ||
[Segment-C (start)] | |||
File= | "C.pan" | ||
North= | 270 | ||
Link 270= | "Segment-A (end)" | ||
Link 180= | "Segment-B (start)" | ||
[Segment-C (end)] | |||
File= | "C.pan" | ||
North= | 270 | ||
APPENDIX C PSEUDOCODE FOR | |||
LINKED PAN MOVIES (VIA CONTROL FILE) | |||
GLOBAL FILE controlFile // Control file | |||
GLOBAL STRING currentsegment // The name of the current pan movie | |||
segment | |||
GLOBAL INTEGER currentFrameNumber // The current frame number of the | |||
current Pan Movie | |||
GLOBAL INTEGER currentHeading // The current user view horizontal pan | |||
orientation | |||
// This function will read the control file and determine which linked | |||
segment is closest | |||
// to the current user heading orientation | |||
// It will also determine the new frame number of the new segment | |||
// | |||
BOOLEAN | RetrieveLink() | ||
{ | |||
INTEGER minAngle | |||
STRING nextSegment | |||
if currentFrameNumber == 0 | |||
currentSegment = currentSegment + (start) | |||
else | |||
currentSegment = currentSegment + (end) | |||
if no links in section currentSegment of controlFile | |||
return FALSE | |||
minAngle | = link angle closest to currentHeading | ||
nextSegment | = GetString(minAngle) | ||
if AngleDifference(currentHeading,MinAngle) > 45 degrees | |||
return FALSE; | |||
INTEGER nextNorth = GetNorth(nextSegment) | |||
INTEGER currentNorth = GetNorth(currentSegment) | |||
currentHeading = currentHeading + (nextNorth - currentNorth) | |||
currentSegment = nextSegment | |||
if stringFind(currentSegment,"(end)") | |||
currentFrameNumber = -1 | |||
else | |||
currentFrameNumber = 0 | |||
return TRUE | |||
} | |||
Gilbert, Scott, Park, Michael C., Ripley, G. David, Kaiman, David J.
Patent | Priority | Assignee | Title |
10032306, | Nov 12 2004 | SMARTER SYSTEMS, INC | Method for inter-scene transitions |
10032479, | Jan 31 2014 | NBCUniversal Media, LLC | Fingerprint-defined segment-based content delivery |
10074013, | Jul 23 2014 | GoPro, Inc. | Scene and activity identification in video summary generation |
10083537, | Feb 04 2016 | GoPro, Inc. | Systems and methods for adding a moving visual element to a video |
10083718, | Mar 24 2017 | GoPro, Inc. | Systems and methods for editing videos based on motion |
10084961, | Mar 04 2014 | GoPro, Inc. | Automatic generation of video from spherical content using audio/visual analysis |
10096341, | Jan 05 2015 | GoPro, Inc. | Media identifier generation for camera-captured media |
10109319, | Jan 08 2016 | GOPRO, INC | Digital media editing |
10120440, | Mar 30 2006 | Virtual navigation system for virtual and real spaces | |
10127943, | Mar 02 2017 | GOPRO, INC | Systems and methods for modifying videos based on music |
10185891, | Jul 08 2016 | GoPro, Inc.; GOPRO, INC | Systems and methods for compact convolutional neural networks |
10185895, | Mar 23 2017 | JPMORGAN CHASE BANK, N A , AS ADMINISTRATIVE AGENT | Systems and methods for classifying activities captured within images |
10186012, | May 20 2015 | GoPro, Inc.; GOPRO, INC | Virtual lens simulation for video and photo cropping |
10186298, | Oct 20 2015 | GoPro, Inc. | System and method of generating video from video clips based on moments of interest within the video clips |
10187690, | Apr 24 2017 | GoPro, Inc. | Systems and methods to detect and correlate user responses to media content |
10192585, | Aug 20 2014 | GoPro, Inc. | Scene and activity identification in video summary generation based on motion detected in a video |
10204273, | Oct 20 2015 | GOPRO, INC | System and method of providing recommendations of moments of interest within video clips post capture |
10204658, | Jul 14 2014 | SONY INTERACTIVE ENTERTAINMENT INC | System and method for use in playing back panorama video content |
10242714, | Dec 19 2016 | Microsoft Technology Licensing, LLC | Interface for application-specified playback of panoramic video |
10244200, | Nov 29 2016 | Microsoft Technology Licensing, LLC | View-dependent operations during playback of panoramic video |
10244215, | Nov 29 2016 | Microsoft Technology Licensing, LLC | Re-projecting flat projections of pictures of panoramic video for rendering by application |
10262639, | Nov 08 2016 | JPMORGAN CHASE BANK, N A , AS ADMINISTRATIVE AGENT | Systems and methods for detecting musical features in audio content |
10262695, | Aug 20 2014 | GoPro, Inc. | Scene and activity identification in video summary generation |
10284809, | Nov 07 2016 | JPMORGAN CHASE BANK, N A , AS ADMINISTRATIVE AGENT | Systems and methods for intelligently synchronizing events in visual content with musical features in audio content |
10304233, | Aug 30 2005 | SMARTER SYSTEMS, INC | Method for inter-scene transitions |
10339975, | Jul 23 2014 | GoPro, Inc. | Voice-based video tagging |
10341712, | Apr 07 2016 | GoPro, Inc. | Systems and methods for audio track selection in video editing |
10360945, | Aug 09 2011 | JPMORGAN CHASE BANK, N A , AS ADMINISTRATIVE AGENT | User interface for editing digital media objects |
10388007, | Mar 31 2017 | SIGMASTAR TECHNOLOGY LTD | Image processing method and image processing device |
10395338, | May 20 2015 | GoPro, Inc. | Virtual lens simulation for video and photo cropping |
10424102, | Feb 04 2016 | GOPRO, INC | Digital media editing |
10468066, | Sep 23 2015 | Nokia Technologies Oy | Video content selection |
10529051, | May 20 2015 | GoPro, Inc. | Virtual lens simulation for video and photo cropping |
10529052, | May 20 2015 | GoPro, Inc. | Virtual lens simulation for video and photo cropping |
10530997, | Jul 13 2017 | MFTB HOLDCO, INC | Connecting and using building interior data acquired from mobile devices |
10534966, | Feb 02 2017 | JPMORGAN CHASE BANK, N A , AS ADMINISTRATIVE AGENT | Systems and methods for identifying activities and/or events represented in a video |
10535115, | May 20 2015 | GoPro, Inc. | Virtual lens simulation for video and photo cropping |
10546566, | Nov 08 2016 | GoPro, Inc. | Systems and methods for detecting musical features in audio content |
10559324, | Jan 05 2015 | GoPro, Inc. | Media identifier generation for camera-captured media |
10560657, | Nov 07 2016 | GoPro, Inc. | Systems and methods for intelligently synchronizing events in visual content with musical features in audio content |
10565769, | Feb 04 2016 | GoPro, Inc. | Systems and methods for adding visual elements to video content |
10607651, | Jan 08 2016 | GoPro, Inc. | Digital media editing |
10643386, | Apr 11 2018 | MFTB HOLDCO, INC | Presenting image transition sequences between viewing locations |
10643663, | Aug 20 2014 | GoPro, Inc. | Scene and activity identification in video summary generation based on motion detected in a video |
10657693, | Nov 13 2006 | SMARTER SYSTEMS, INC | Method for scripting inter-scene transitions |
10666863, | May 25 2018 | Microsoft Technology Licensing, LLC | Adaptive panoramic video streaming using overlapping partitioned sections |
10679323, | May 20 2015 | GoPro, Inc. | Virtual lens simulation for video and photo cropping |
10679670, | Mar 02 2017 | GOPRO, INC | Systems and methods for modifying videos based on music |
10681335, | Sep 23 2015 | Nokia Technologies Oy | Video recording method and apparatus |
10708507, | Oct 11 2018 | MFTB HOLDCO, INC | Automated control of image acquisition via use of acquisition device sensors |
10748577, | Oct 20 2015 | GoPro, Inc. | System and method of generating video from video clips based on moments of interest within the video clips |
10764494, | May 25 2018 | Microsoft Technology Licensing, LLC | Adaptive panoramic video streaming using composite pictures |
10769834, | Feb 04 2016 | GoPro, Inc. | Digital media editing |
10776629, | Jul 23 2014 | GoPro, Inc. | Scene and activity identification in video summary generation |
10789478, | Oct 20 2015 | GoPro, Inc. | System and method of providing recommendations of moments of interest within video clips post capture |
10789985, | Mar 24 2017 | GoPro, Inc. | Systems and methods for editing videos based on motion |
10805592, | Jun 30 2016 | SONY INTERACTIVE ENTERTAINMENT INC | Apparatus and method for gaze tracking |
10809066, | Oct 11 2018 | MFTB HOLDCO, INC | Automated mapping information generation from inter-connected images |
10817977, | May 20 2015 | GoPro, Inc. | Virtual lens simulation for video and photo cropping |
10825247, | Nov 12 2019 | MFTB HOLDCO, INC | Presenting integrated building information using three-dimensional building models |
10834317, | Jul 13 2017 | MFTB HOLDCO, INC | Connecting and using building data acquired from mobile devices |
10991396, | Mar 02 2017 | GoPro, Inc. | Systems and methods for modifying videos based on music |
11049522, | Jan 08 2016 | GoPro, Inc. | Digital media editing |
11050991, | Sep 29 2016 | KONINKLIJKE PHILIPS N V | Image processing using a plurality of images for a three dimension scene, having a different viewing positions and/or directions |
11057561, | Jul 13 2017 | MFTB HOLDCO, INC | Capture, analysis and use of building data from mobile devices |
11069380, | Jul 23 2014 | GoPro, Inc. | Scene and activity identification in video summary generation |
11089280, | Jun 30 2016 | SONY INTERACTIVE ENTERTAINMENT INC | Apparatus and method for capturing and displaying segmented content |
11120837, | Jul 14 2014 | SONY INTERACTIVE ENTERTAINMENT INC. | System and method for use in playing back panorama video content |
11164282, | May 20 2015 | GoPro, Inc. | Virtual lens simulation for video and photo cropping |
11164361, | Oct 28 2019 | MFTB HOLDCO, INC | Generating floor maps for buildings from automated analysis of visual data of the buildings' interiors |
11164368, | Oct 07 2019 | MFTB HOLDCO, INC | Providing simulated lighting information for three-dimensional building models |
11165959, | Jul 13 2017 | MFTB HOLDCO, INC | Connecting and using building data acquired from mobile devices |
11217019, | Apr 11 2018 | MFTB HOLDCO, INC | Presenting image transition sequences between viewing locations |
11238635, | Feb 04 2016 | GoPro, Inc. | Digital media editing |
11238652, | Nov 12 2019 | MFTB HOLDCO, INC | Presenting integrated building information using building models |
11243656, | Aug 28 2019 | MFTB HOLDCO, INC | Automated tools for generating mapping information for buildings |
11252329, | Jan 08 2021 | MFTB HOLDCO, INC | Automated determination of image acquisition locations in building interiors using multiple data capture devices |
11282544, | Mar 24 2017 | GoPro, Inc. | Systems and methods for editing videos based on motion |
11284006, | Oct 11 2018 | MFTB HOLDCO, INC | Automated control of image acquisition via acquisition location determination |
11405549, | Jun 05 2020 | MFTB HOLDCO, INC | Automated generation on mobile devices of panorama images for building locations and subsequent use |
11405558, | Oct 11 2018 | MFTB HOLDCO, INC | Automated control of image acquisition via use of hardware sensors and camera content |
11408738, | Oct 11 2018 | MFTB HOLDCO, INC | Automated mapping information generation from inter-connected images |
11443771, | Mar 02 2017 | GoPro, Inc. | Systems and methods for modifying videos based on music |
11455709, | Nov 17 2017 | ZHEJIANG DAHUA TECHNOLOGY CO., LTD. | Methods and systems for image processing |
11468914, | Oct 20 2015 | GoPro, Inc. | System and method of generating video from video clips based on moments of interest within the video clips |
11480433, | Oct 11 2018 | MFTB HOLDCO, INC | Use of automated mapping information from inter-connected images |
11481925, | Nov 23 2020 | MFTB HOLDCO, INC | Automated determination of image acquisition locations in building interiors using determined room shapes |
11494973, | Oct 28 2019 | MFTB HOLDCO, INC | Generating floor maps for buildings from automated analysis of visual data of the buildings' interiors |
11501492, | Jul 27 2021 | MFTB HOLDCO, INC | Automated room shape determination using visual data of multiple captured in-room images |
11514674, | Sep 04 2020 | MFTB HOLDCO, INC | Automated analysis of image contents to determine the acquisition location of the image |
11592969, | Oct 13 2020 | MFTB HOLDCO, INC | Automated tools for generating building mapping information |
11627387, | Oct 11 2018 | MFTB HOLDCO, INC | Automated control of image acquisition via use of mobile device interface |
11632516, | Jul 13 2017 | MFTB HOLDCO, INC | Capture, analysis and use of building data from mobile devices |
11632602, | Jan 08 2021 | MFTB HOLDCO, INC | Automated determination of image acquisition locations in building interiors using multiple data capture devices |
11638069, | Oct 11 2018 | MFTB HOLDCO, INC | Automated control of image acquisition via use of mobile device user interface |
11645781, | Nov 23 2020 | MFTB HOLDCO, INC | Automated determination of acquisition locations of acquired building images based on determined surrounding room data |
11676344, | Nov 12 2019 | MFTB HOLDCO, INC | Presenting building information using building models |
11688034, | May 20 2015 | GoPro, Inc. | Virtual lens simulation for video and photo cropping |
11776579, | Jul 23 2014 | GoPro, Inc. | Scene and activity identification in video summary generation |
11790648, | Feb 25 2021 | MFTB HOLDCO, INC | Automated usability assessment of buildings using visual data of captured in-room images |
11797159, | Oct 13 2020 | MFTB Holdco, Inc. | Automated tools for generating building mapping information |
11823325, | Oct 07 2019 | MFTB HOLDCO, INC | Providing simulated lighting information for building models |
11830135, | Jul 13 2022 | MFTB HOLDCO, INC | Automated building identification using floor plans and acquired building images |
11836973, | Feb 25 2021 | MFTB HOLDCO, INC | Automated direction of capturing in-room information for use in usability assessment of buildings |
11842464, | Sep 22 2021 | MFTB HOLDCO, INC | Automated exchange and use of attribute information between building images of multiple types |
6616701, | May 23 1998 | EOLAS TECHNOLOGIES, INC | Method and apparatus for identifying features of multidimensional image data in hypermedia systems |
6654019, | May 13 1998 | IMMERSIVE LICENSING, INC | Panoramic movie which utilizes a series of captured panoramic images to display movement as observed by a viewer looking in a selected direction |
6690374, | May 12 1999 | IMMERSIVE LICENSING, INC | Security camera system for tracking moving objects in both forward and reverse directions |
6693959, | Mar 03 2000 | ATI Technologies ULC | Method and apparatus for indexing and locating key frames in streaming and variable-frame-length data |
6738073, | May 12 1999 | IMMERSIVE LICENSING, INC | Camera system with both a wide angle view and a high resolution view |
6810152, | Jan 11 2001 | Canon Kabushiki Kaisha | Image processing apparatus, method of processing images, and storage medium |
6859557, | Jul 07 2000 | Microsoft Technology Licensing, LLC | System and method for selective decoding and decompression |
6947059, | Aug 10 2001 | DD MICOY, INC | Stereoscopic panoramic image capture device |
7012637, | Jul 27 2001 | AI-CORE TECHNOLOGIES, LLC | Capture structure for alignment of multi-camera capture systems |
7050085, | Oct 26 2000 | IMMERSIVE LICENSING, INC | System and method for camera calibration |
7106335, | Nov 29 2000 | INTERDIGITAL CE PATENT HOLDINGS | Method for displaying an object in a panorama window |
7126630, | Feb 09 2001 | Kujin, Lee | Method and apparatus for omni-directional image and 3-dimensional data acquisition with data annotation and dynamic range extension method |
7183549, | Sep 09 2004 | Flir Systems, Inc.; FLIR SYSTEMS, INC | Multiple camera systems and methods |
7222021, | Sep 07 2001 | Kabushiki Kaisha Topcon | Operator guiding system |
7346451, | Dec 03 2003 | Denso Corporation | Electronic device and program for displaying map |
7347555, | Dec 22 2004 | DD MICOY, INC | Multi-dimensional imaging apparatus, systems, and methods |
7381952, | Sep 09 2004 | Teledyne FLIR, LLC | Multiple camera systems and methods |
7429997, | Nov 29 2000 | DD MICOY, INC | System and method for spherical stereoscopic photographing |
7553023, | Dec 26 2003 | DD MICOY, INC | Multi-dimensional imaging apparatus, methods, and systems |
7620909, | May 12 1999 | IMMERSIVE LICENSING, INC | Interactive image seamer for panoramic images |
7663662, | Feb 09 2005 | Flir Systems, Inc. | High and low resolution camera systems and methods |
7734413, | Dec 03 2003 | Denso Corporation | Electronic device and program for displaying map |
7746376, | Jun 16 2004 | True Voyager, LLC | Method and apparatus for accessing multi-dimensional mapping and information |
7834910, | Mar 01 2006 | Garmin Switzerland GmbH | Method and apparatus for panoramic imaging |
7916168, | Aug 14 2001 | KONINKLIJKE PHILIPS ELECTRONIC N V | Display of an arrangement of a panoramic video by applying navigation commands to said panoramic video |
7952608, | Nov 07 2002 | WQS LTD | Surveillance device |
7965314, | Feb 09 2005 | Teledyne FLIR, LLC | Foveal camera systems and methods |
8334895, | May 13 2005 | DD MICOY, INC | Image capture and processing using converging rays |
8447136, | Jan 12 2010 | Microsoft Technology Licensing, LLC | Viewing media in the context of street-level images |
8508577, | Aug 14 2001 | Koninklijke Philips N.V. | Display of an arrangement of a panoramic video by applying navigation commands to said panoramic video |
8627236, | Nov 03 2009 | LG Electronics Inc. | Terminal and control method thereof |
8885024, | May 13 2005 | DD MICOY, INC | Stereo imagers and projectors, and method |
8890940, | May 13 2005 | DD MICOY, INC | Stereo image capture and processing |
8896608, | Aug 10 2005 | Movinpics AS | Method for providing an animation from a prerecorded series of still pictures |
8977113, | Oct 25 2013 | ZHOOB, INC | Mobile device video decision tree |
9196072, | Nov 13 2006 | SMARTER SYSTEMS, INC | Method for scripting inter-scene transitions |
9449372, | Aug 22 2013 | BAE Systems Information and Electronic Systems Integration Inc. | Dust removal technology for driver vision leverage |
9473758, | Dec 06 2015 | SLIVER VR TECHNOLOGIES, INC | Methods and systems for game video recording and virtual reality replay |
9652667, | Mar 04 2014 | GOPRO, INC | Automatic generation of video from spherical content using audio/visual analysis |
9754159, | Mar 04 2014 | GOPRO, INC | Automatic generation of video from spherical content using location-based metadata |
9760768, | Mar 04 2014 | GOPRO, INC | Generation of video from spherical content using edit maps |
9794632, | Apr 07 2016 | GoPro, Inc. | Systems and methods for synchronization based on audio track changes in video editing |
9807319, | Jun 03 2009 | Teledyne FLIR, LLC | Wearable imaging devices, systems, and methods |
9812175, | Feb 04 2016 | GOPRO, INC | Systems and methods for annotating a video |
9836853, | Sep 06 2016 | GoPro, Inc. | Three-dimensional convolutional neural networks for video highlight detection |
9838731, | Apr 07 2016 | GoPro, Inc. | Systems and methods for audio track selection in video editing with audio mixing option |
9966108, | Jan 29 2015 | JPMORGAN CHASE BANK, N A , AS ADMINISTRATIVE AGENT | Variable playback speed template for video editing application |
9973692, | Oct 03 2013 | Teledyne FLIR, LLC | Situational awareness by compressed display of panoramic views |
9984293, | Jul 23 2014 | GoPro, Inc. | Video scene classification by activity |
Patent | Priority | Assignee | Title |
3909121, | |||
4853764, | Sep 16 1988 | Pedalo, Inc. | Method and apparatus for screenless panoramic stereo TV system |
5023725, | Oct 23 1989 | IMMERSIVE LICENSING, INC | Method and apparatus for dodecahedral imaging system |
5541773, | Mar 26 1993 | Olympus Optical Co., Ltd. | Two-unit zoom lens system |
5563649, | Jun 16 1993 | PORTE, MICHAEL | System and method for transmitting video material |
5703604, | May 22 1995 | IMMERSIVE LICENSING, INC | Immersive dodecaherdral video viewing system |
5708469, | May 03 1996 | International Business Machines Corporation | Multiple view telepresence camera system using a wire cage which surroundss a plurality of movable cameras and identifies fields of view |
5729471, | Mar 31 1995 | The Regents of the University of California | Machine dynamic selection of one video camera/image of a scene from multiple video cameras/images of the scene in accordance with a particular perspective on the scene, an object in the scene, or an event in the scene |
5748121, | Dec 06 1995 | Intel Corporation | Generation of huffman tables for signal encoding |
5872575, | Feb 14 1996 | Digital Media Interactive; DIGITAL MEDIA, INC | Method and system for the creation of and navigation through a multidimensional space using encoded digital video |
6058397, | Apr 08 1997 | Mitsubishi Electric Research Laboratories, Inc | 3D virtual environment creation management and delivery system |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
May 07 1999 | GILBERT, SCOTT | INFINITE PICTURES INC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 009965 | /0347 | |
May 07 1999 | KAIMAN, DAVID J | INFINITE PICTURES INC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 009965 | /0347 | |
May 07 1999 | PARK, MICHAEL C | INFINITE PICTURES INC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 009965 | /0347 | |
May 07 1999 | RIPLEY, G DAVID | INFINITE PICTURES INC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 009965 | /0347 | |
May 12 1999 | iMove Inc. | (assignment on the face of the patent) | / | |||
May 25 2000 | IMOVE, INC | IMPERIAL BANK | SECURITY INTEREST SEE DOCUMENT FOR DETAILS | 012092 | /0552 | |
Oct 02 2002 | IMOVE, INC | Silicon Valley Bank | SECURITY AGREEMENT | 013475 | /0988 | |
Nov 01 2006 | IMOVE, INC | Silicon Valley Bank | SECURITY INTEREST SEE DOCUMENT FOR DETAILS | 018635 | /0186 | |
Jan 25 2007 | COMERICA BANK | IMOVE INC | RELEASE BY SECURED PARTY SEE DOCUMENT FOR DETAILS | 018825 | /0121 | |
May 08 2008 | Silicon Valley Bank | IMOVE, INC | RELEASE BY SECURED PARTY SEE DOCUMENT FOR DETAILS | 020963 | /0884 | |
May 08 2008 | Silicon Valley Bank | IMOVE, INC | RELEASE | 020963 | /0902 | |
Dec 31 2013 | IMOVE INC | IMMERSIVE LICENSING, INC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 034498 | /0822 |
Date | Maintenance Fee Events |
Jun 16 2003 | ASPN: Payor Number Assigned. |
Jul 08 2005 | M2551: Payment of Maintenance Fee, 4th Yr, Small Entity. |
Jul 08 2009 | M2552: Payment of Maintenance Fee, 8th Yr, Small Entity. |
Jul 08 2013 | M2553: Payment of Maintenance Fee, 12th Yr, Small Entity. |
Date | Maintenance Schedule |
Jan 08 2005 | 4 years fee payment window open |
Jul 08 2005 | 6 months grace period start (w surcharge) |
Jan 08 2006 | patent expiry (for year 4) |
Jan 08 2008 | 2 years to revive unintentionally abandoned end. (for year 4) |
Jan 08 2009 | 8 years fee payment window open |
Jul 08 2009 | 6 months grace period start (w surcharge) |
Jan 08 2010 | patent expiry (for year 8) |
Jan 08 2012 | 2 years to revive unintentionally abandoned end. (for year 8) |
Jan 08 2013 | 12 years fee payment window open |
Jul 08 2013 | 6 months grace period start (w surcharge) |
Jan 08 2014 | patent expiry (for year 12) |
Jan 08 2016 | 2 years to revive unintentionally abandoned end. (for year 12) |