A volumetric dynamic virtual camera system employs a radial basis function (RBF) component that can utilize non-uniform training datasets that are blended to provide interpolated camera parameters during application runtime based on a player's position within virtual volumes in a 3D space. During application development, an artist or developer can interactively author cameras that are finely tuned to appear just right and which provide the training data for runtime. The RBF component blends the training data during runtime as the player's position within the volume changes to produce camera parameters that the camera system uses to capture scenes for rendering on a display device. The result is an overall camera system that lets authors very quickly develop film-quality cameras that appear and behave more like fully dynamic cameras having significant intelligence. The cameras are volumetric because they can exist in all the virtual spaces exposed by the application.
|
14. A method of operating a computing device that executes an application configured to render scenes occurring within a virtual three-dimensional (3D) application space, comprising:
receiving a training dataset that includes camera parameters corresponding to respective training virtual player positions in the virtual 3D application space;
implementing a radial basis function (RBF) algorithm that interpolates the training dataset to produce camera parameters for virtual player positions in the virtual 3D application space other than the training virtual player positions;
receiving inputs from a computing device user that control positions of a virtual player within the virtual 3D application space during application runtime;
using the camera parameters to operate a virtual camera system to capture respective scenes of the virtual player at controlled positions within the virtual 3D application space; and
rendering the captured scenes for display by the computing device.
7. A computing device configured to support an application configured to generate virtual scenes including a virtual player, comprising:
one or more processors; and
one or more hardware-based computer-readable memory devices storing instructions which, when executed by the one or more processors, cause the computing device to:
receive training data including a plurality of data pairs, each data pair including a trained virtual player position within a portion of a virtual 3D application space and an associated set of one or more trained camera parameters;
receive a current position of the virtual player in the virtual 3D application space;
use a radial basis function (RBF) to interpolate the training data in view of the current position to generate runtime camera parameters; and
output the runtime camera parameters to a virtual camera system which captures, in accordance with the camera parameters, a scene of a portion of the virtual 3D application space that includes the virtual player in the current position during application runtime.
1. One or more hardware-based computer-readable memory devices storing instructions which, when executed by one or more processors disposed in an electronic device, implement a camera system used in a virtual three-dimensional (3D) game space and further cause the electronic device to:
receive a training dataset that maps respective virtual player positions in a convex hull in the virtual 3D space to corresponding sets of camera parameters, the convex hull being fitted to sample points in a camera sample dataset;
receive a current position of a virtual player in the convex hull during game runtime;
interpolate data in the training dataset to generate interpolated camera parameters for the current virtual player position;
receive a current position of the virtual player in the virtual 3D space that is external to the convex hull;
extrapolate the data in the training dataset to generate extrapolated camera parameters for the current virtual player position;
send either the interpolated camera parameters or the extrapolated camera parameters to a camera system so that the camera system can capture scenes in the virtual 3D game space in accordance with either the interpolated camera parameters or the extrapolated camera parameters.
2. The one or more computer-readable memory devices of
3. The one or more computer-readable memory devices of
4. The one or more computer-readable memory devices of
5. The one or more computer-readable memory devices of
6. The one or more computer-readable memory devices of
8. The computing device of
9. The computing device of
11. The computing device of
12. The computing device of
13. The computing device of
15. The method of
16. The method of
17. The method of
18. The method of
19. The method of
|
This application is a divisional of U.S. application Ser. No. 14/261,173, field Apr. 24, 2014, entitled, “ARTIST-DIRECTED VOLUMETRIC DYNAMIC VIRTUAL CAMERAS” which is incorporated herein by reference in its entirety.
Typical real-time cameras used in virtual three dimensional (3D) spaces used in gaming and other computer-based applications are either fully animated or fully simulated. Fully animated cameras provide comprehensive directability for artists and cinematographers, but suffer from being non-interactive for game players. Fully simulated cameras allow for player interaction, but offer little ability for artists to provide direction. Some advanced camera solutions for simulated cameras allow for artists and engineers to provide limited direction by adding constraints such as splines or look-at targets. These solutions tend to be highly technical and tricky to use. They often require complex level scripting to dynamically change camera constraints.
This Background is provided to introduce a brief context for the Summary and Detailed Description that follow. This Background is not intended to be an aid in determining the scope of the claimed subject matter nor be viewed as limiting the claimed subject matter to implementations that solve any or all of the disadvantages or problems presented above.
A volumetric dynamic virtual camera system employs a radial basis function (RBF) component that can utilize non-uniform training datasets that are blended to provide interpolated camera parameters during application runtime based on a player's position within virtual volumes in a 3D space. During application development, an artist or developer can interactively author cameras that are finely tuned to appear just right and which provide the training data for runtime. The RBF component blends the training data during runtime as the player's position within the volume changes to produce camera parameters that the camera system uses to capture scenes for rendering on a display device. The result is an overall camera system that lets authors very quickly develop film-quality cameras that appear and behave more like fully dynamic cameras having significant intelligence. The cameras are volumetric because they can exist in all the virtual spaces exposed by the application.
The training data can be non-uniform and comprise both sparse and dense sets that are created interactively by authors or received from an external application such as a DCC (digital content creation) tool. The training data used by the RBF component can be fine tuned by manually selecting and positioning keyframes from a set of camera samples that define the virtual camera. The training data can also be extrapolated by the RBF component in some cases as the player is moved to volumes that are outside a given defined convex hull. The present volumetric dynamic virtual camera system can be utilized along with traditional camera techniques, for example, those that deal with obstacle and collision avoidance and those techniques implemented by other camera types (e.g., user-controlled, orbital, free, etc.). The system can also drive conventional interactive camera parameters such as distance/offset from the player, field of view, and camera look-direction bias.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this disclosure.
Like reference numerals indicate like elements in the drawings. Elements are not drawn to scale unless otherwise indicated.
Local content 120, including apps, games, and/or media content may also be utilized and/or consumed in order to provide a particular user experience in the environment 100. As shown in
The user 112 can typically interact with the multimedia console 114 using a variety of different interface devices including a camera system 128 that can be used to sense visual commands, motions, and gestures, and a headset 130 or other type of microphone or audio capture device. In some cases a microphone and camera can be combined into a single device. The user may also utilize a controller 132 to interact with the multimedia console 114. The controller 132 may include a variety of physical controls including joysticks, a directional pad (“D-pad”), and various buttons. One or more triggers and/or bumpers (not shown) may also be incorporated into the controller 132. The user 112 can also typically interact with a user interface 134 that is shown on a display device 136 such as a television, projector, or monitor.
As shown in
The application layer 305 in this illustrative example supports various applications 320 as well as the game application 122. The applications 305 and 122 are often implemented using locally executing code. However in some cases, the applications may rely on services and/or remote code execution provided by remote servers or other computing platforms such as those supported by an external service provider such as the service provider 118 and entertainment service provider 102 shown in
The game 122 typically utilizes one or more virtual game spaces in which the gameplay provided by the application takes place.
As the player 410 continues up the stairway 405, the virtual camera produces a wider shot of the player and staircase from the side at point “D” on the camera path 505, as shown in screen shot 900 in
Using conventional camera systems, authoring the cameras used along the path 505 (
The developer training data 1110 describes how the developer wants the camera to look and operate at various positions in the game space. For example, as shown in
As shown in
In some cases the camera samples and/or keyframes can be generated interactively by the developer while playing the game in a testing or development mode. The developer can use a hotkey or similar tool to indicate when keyframes are set as the developer moves the player's location within the game space. Once the keyframes are set, the developer can go back in, if desired, and edit the cameras by manually selecting and position keyframes from the set of camera samples in order to fine tune the parameters so that the camera system behaves during runtime just like the developer wishes. As noted above, the cameras can be positioned anywhere in the volume and be changed on the fly during gameplay. Thus, the developer can readily utilize any number of cinematic camera shots that can take any desired point of view.
The parameters associated with each keyframe constitute the training data for the camera system for the camera sample dataset 1200. Therefore, the training dataset will typically include multiple data pairs where each pair includes a player position in the space and associated set of camera parameters for that position. The number of keyframes utilized for any given camera sample dataset can vary by implementation. Advantageously, the RBF component is typically configured to perform well even when the training dataset is sparse.
The RBF component 1105 typically uses the player's position as an input (indicated by reference numeral 1305) and generates interpolated camera parameters 1310 for that player position as an output. The RBF component 1105 functions to smoothly blend the training data associated with each of the keyframes 1210 (
In addition to player position as an input to the RBF component 1105, other dimensionalities 1315 may also be used in some implementations.
Various interpolated camera parameters can be generated by the RBF component 1105 depending on the needs of a particular implementation.
In addition to generating interpolated camera parameters, the RBF component 1105 can also extrapolate training data from a given camera sample dataset to an external volume in which no training data has been generated by the developer. In some cases, the extrapolating functionality can be included with the RBF component 1105 while in other cases, such functionality is separately instantiated in another component that operates independently from the RBF component.
An example extrapolation scenario is shown in
In some camera system implementations, additional constraints may be imposed to control which training data is allowed to be interpolated or extrapolated or how camera parameters are utilized by the camera system. For example, rotation parameters are not expected to extrapolate well. To deal with this situation, a camera heading may be broken into a pitch yaw vector (that interpolates and extrapolates well using the RBF functionality) along with a roll parameter that is separately handled. Camera position and rotation parameters can be additionally subjected to constraints during runtime so that they are implemented in a relative relationship to the player to reduce the impact of interpolation errors. Similarly, the composition of the screen space may be constrained in some cases so that the camera is constrained to the player on the screen. Look-at constraints and other path constraints may also be advantageously applied in some cases, for example, as a camera traverses a spline.
In step 1705, the developer selects a camera sample dataset that is used in a virtual game space. In step 1710, the developer selects a player location anywhere in the scene and positions a camera to a desired placement (where the convex hull expands to contain the new sample point). Training data comprising the parameters associated with the placed camera are then captured in step 1715 and added to the training dataset in step 1720. The steps 1710, 1715, and 1720 are iterated for each keyframe that is utilized for the camera sample dataset and for any other camera sample dataset of interest. As discussed above, these steps can also be performed interactively in some cases and then the resulting parameters can be edited or tweaked as needed (not shown in
In response to the input, the RBF component returns interpolated camera parameters (or extrapolated parameters when the player is an external volume) to the camera system 325 (
A graphics processing unit (GPU) 1908 and a video encoder/video codec (coder/decoder) 1914 form a video processing pipeline for high speed and high resolution graphics processing. Data is carried from the GPU 1908 to the video encoder/video codec 1914 via a bus. The video processing pipeline outputs data to an A/V (audio/video) port 1940 for transmission to a television or other display. A memory controller 1910 is connected to the GPU 1908 to facilitate processor access to various types of memory 1912, such as, but not limited to, a RAM.
The multimedia console 114 includes an I/O controller 1920, a system management controller 1922, an audio processing unit 1923, a network interface controller 1924, a first USB (Universal Serial Bus) host controller 1926, a second USB controller 1928, and a front panel I/O subassembly 1930 that are preferably implemented on a module 1918. The USB controllers 1926 and 1928 serve as hosts for peripheral controllers 1942(1) and 1942(2), a wireless adapter 1948, and an external memory device 1946 (e.g., Flash memory, external CD/DVD ROM drive, removable media, etc.). The network interface controller 1924 and/or wireless adapter 1948 provide access to a network (e.g., the Internet, home network, etc.) and may be any of a wide variety of various wired or wireless adapter components including an Ethernet card, a modem, a Bluetooth module, a cable modem, or the like.
System memory 1943 is provided to store application data that is loaded during the boot process. A media drive 1944 is provided and may comprise a DVD/CD drive, hard drive, or other removable media drive, etc. The media drive 1944 may be internal or external to the multimedia console 114. Application data may be accessed via the media drive 1944 for execution, playback, etc. by the multimedia console 114. The media drive 1944 is connected to the I/O controller 1920 via a bus, such as a Serial ATA bus or other high speed connection (e.g., IEEE 1394).
The system management controller 1922 provides a variety of service functions related to assuring availability of the multimedia console 114. The audio processing unit 1923 and an audio codec 1932 form a corresponding audio processing pipeline with high fidelity and stereo processing. Audio data is carried between the audio processing unit 1923 and the audio codec 1932 via a communication link. The audio processing pipeline outputs data to the A/V port 1940 for reproduction by an external audio player or device having audio capabilities.
The front panel I/O subassembly 1930 supports the functionality of the power button 1950 and the eject button 1952, as well as any LEDs (light emitting diodes) or other indicators exposed on the outer surface of the multimedia console 114. A system power supply module 1936 provides power to the components of the multimedia console 114. A fan 1938 cools the circuitry within the multimedia console 114.
The CPU 1901, GPU 1908, memory controller 1910, and various other components within the multimedia console 114 are interconnected via one or more buses, including serial and parallel buses, a memory bus, a peripheral bus, and a processor or local bus using any of a variety of bus architectures. By way of example, such architectures can include a Peripheral Component Interconnects (PCI) bus, PCI-Express bus, etc.
When the multimedia console 114 is powered ON, application data may be loaded from the system memory 1943 into memory 1912 and/or caches 1902 and 1904 and executed on the CPU 1901. The application may present a graphical user interface that provides a consistent user experience when navigating to different media types available on the multimedia console 114. In operation, applications and/or other media contained within the media drive 1944 may be launched or played from the media drive 1944 to provide additional functionalities to the multimedia console 114.
The multimedia console 114 may be operated as a standalone system by simply connecting the system to a television or other display. In this standalone mode, the multimedia console 114 allows one or more users to interact with the system, watch movies, or listen to music. However, with the integration of broadband connectivity made available through the network interface controller 1924 or the wireless adapter 1948, the multimedia console 114 may further be operated as a participant in a larger network community.
When the multimedia console 114 is powered ON, a set amount of hardware resources are reserved for system use by the multimedia console operating system. These resources may include a reservation of memory (e.g., 16 MB), CPU and GPU cycles (e.g., 5%), networking bandwidth (e.g., 8 kbps), etc. Because these resources are reserved at system boot time, the reserved resources do not exist from the application's view.
In particular, the memory reservation preferably is large enough to contain the launch kernel, concurrent system applications, and drivers. The CPU reservation is preferably constant such that if the reserved CPU usage is not used by the system applications, an idle thread will consume any unused cycles.
With regard to the GPU reservation, lightweight messages generated by the system applications (e.g., pop-ups) are displayed by using a GPU interrupt to schedule code to render pop-ups into an overlay. The amount of memory needed for an overlay depends on the overlay area size and the overlay preferably scales with screen resolution. Where a full user interface is used by the concurrent system application, it is preferable to use a resolution independent of application resolution. A scaler may be used to set this resolution such that the need to change frequency and cause a TV re-sync is eliminated.
After the multimedia console 114 boots and system resources are reserved, concurrent system applications execute to provide system functionalities. The system functionalities are encapsulated in a set of system applications that execute within the reserved system resources described above. The operating system kernel identifies threads that are system application threads versus gaming application threads. The system applications are preferably scheduled to run on the CPU 1901 at predetermined times and intervals in order to provide a consistent system resource view to the application. The scheduling is to minimize cache disruption for the gaming application running on the console.
When a concurrent system application requires audio, audio processing is scheduled asynchronously to the gaming application due to time sensitivity. A multimedia console application manager (described below) controls the gaming application audio level (e.g., mute, attenuate) when system applications are active.
Input devices (e.g., controllers 1942(1) and 1942(2)) are shared by gaming applications and system applications. The input devices are not reserved resources, but are to be switched between system applications and the gaming application such that each will have a focus of the device. The application manager preferably controls the switching of input stream, without knowledge of the gaming application's knowledge and a driver maintains state information regarding focus switches.
A number of program modules may be stored on the hard disk, magnetic disk 2033, optical disk 2043, ROM 2017, or RAM 2021, including an operating system 2055, one or more application programs 2057, other program modules 2060, and program data 2063. A user may enter commands and information into the computer system 2000 through input devices such as a keyboard 2066 and pointing device 2068 such as a mouse. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, trackball, touchpad, touch screen, touch-sensitive module or device, gesture-recognition module or device, voice recognition module or device, voice command module or device, or the like. These and other input devices are often connected to the processing unit 2005 through a serial port interface 2071 that is coupled to the system bus 2014, but may be connected by other interfaces, such as a parallel port, game port, or USB. A monitor 2073 or other type of display device is also connected to the system bus 2014 via an interface, such as a video adapter 2075. In addition to the monitor 2073, personal computers typically include other peripheral output devices (not shown), such as speakers and printers. The illustrative example shown in
The computer system 2000 is operable in a networked environment using logical connections to one or more remote computers, such as a remote computer 2088. The remote computer 2088 may be selected as another personal computer, a server, a router, a network PC, a peer device, or other common network node, and typically includes many or all of the elements described above relative to the computer system 2000, although only a single representative remote memory/storage device 2090 is shown in
When used in a LAN networking environment, the computer system 2000 is connected to the local area network 2093 through a network interface or adapter 2096. When used in a WAN networking environment, the computer system 2000 typically includes a broadband modem 2098, network gateway, or other means for establishing communications over the wide area network 2095, such as the Internet. The broadband modem 2098, which may be internal or external, is connected to the system bus 2014 via a serial port interface 2071. In a networked environment, program modules related to the computer system 2000, or portions thereof, may be stored in the remote memory storage device 2090. It is noted that the network connections shown in
The architecture 2100 illustrated in
The mass storage device 2112 is connected to the CPU 2102 through a mass storage controller (not shown) connected to the bus 2110. The mass storage device 2112 and its associated computer-readable storage media provide non-volatile storage for the architecture 2100. Although the description of computer-readable storage media contained herein refers to a mass storage device, such as a hard disk or CD-ROM drive, it should be appreciated by those skilled in the art that computer-readable media can be any available computer storage media that can be accessed by the architecture 2100.
By way of example, and not limitation, computer-readable storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. For example, computer-readable media includes, but is not limited to, RAM, ROM, EPROM (erasable programmable read only memory), EEPROM (electrically erasable programmable read only memory), Flash memory or other solid state memory technology, CD-ROM, DVDs, HD-DVD (High Definition DVD), Blu-ray, or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by the architecture 2100.
According to various embodiments, the architecture 2100 may operate in a networked environment using logical connections to remote computers through a network. The architecture 2100 may connect to the network through a network interface unit 2116 connected to the bus 2110. It should be appreciated that the network interface unit 2116 also may be utilized to connect to other types of networks and remote computer systems. The architecture 2100 also may include an input/output controller 2118 for receiving and processing input from a number of other devices, including a keyboard, mouse, or electronic stylus (not shown in
It should be appreciated that the software components described herein may, when loaded into the CPU 2102 and executed, transform the CPU 2102 and the overall architecture 2100 from a general-purpose computing system into a special-purpose computing system customized to facilitate the functionality presented herein. The CPU 2102 may be constructed from any number of transistors or other discrete circuit elements, which may individually or collectively assume any number of states. More specifically, the CPU 2102 may operate as a finite-state machine, in response to executable instructions contained within the software modules disclosed herein. These computer-executable instructions may transform the CPU 2102 by specifying how the CPU 2102 transitions between states, thereby transforming the transistors or other discrete hardware elements constituting the CPU 2102.
Encoding the software modules presented herein also may transform the physical structure of the computer-readable storage media presented herein. The specific transformation of physical structure may depend on various factors, in different implementations of this description. Examples of such factors may include, but are not limited to, the technology used to implement the computer-readable storage media, whether the computer-readable storage media is characterized as primary or secondary storage, and the like. For example, if the computer-readable storage media is implemented as semiconductor-based memory, the software disclosed herein may be encoded on the computer-readable storage media by transforming the physical state of the semiconductor memory. For example, the software may transform the state of transistors, capacitors, or other discrete circuit elements constituting the semiconductor memory. The software also may transform the physical state of such components in order to store data thereupon.
As another example, the computer-readable storage media disclosed herein may be implemented using magnetic or optical technology. In such implementations, the software presented herein may transform the physical state of magnetic or optical media, when the software is encoded therein. These transformations may include altering the magnetic characteristics of particular locations within given magnetic media. These transformations also may include altering the physical features or characteristics of particular locations within given optical media to change the optical characteristics of those locations. Other transformations of physical media are possible without departing from the scope and spirit of the present description, with the foregoing examples provided only to facilitate this discussion.
In light of the above, it should be appreciated that many types of physical transformations take place in the architecture 2100 in order to store and execute the software components presented herein. It also should be appreciated that the architecture 2100 may include other types of computing devices, including hand-held computers, embedded computer systems, smartphones, PDAs, and other types of computing devices known to those skilled in the art. It is also contemplated that the architecture 2100 may not include all of the components shown in
Based on the foregoing, it should be appreciated that technologies for volumetric dynamic virtual cameras have been disclosed herein. Although the subject matter presented herein has been described in language specific to computer structural features, methodological and transformative acts, specific computing machinery, and computer-readable storage media, it is to be understood that the invention defined in the appended claims is not necessarily limited to the specific features, acts, or media described herein. Rather, the specific features, acts, and mediums are disclosed as example forms of implementing the claims.
The subject matter described above is provided by way of illustration only and should not be construed as limiting. Various modifications and changes may be made to the subject matter described herein without following the example embodiments and applications illustrated and described, and without departing from the true spirit and scope of the present invention, which is set forth in the following claims.
Patent | Priority | Assignee | Title |
Patent | Priority | Assignee | Title |
6856319, | Jun 13 2002 | Microsoft Technology Licensing, LLC | Interpolation using radial basis functions with application to inverse kinematics |
7024279, | Jun 13 2002 | Microsoft Technology Licensing, LLC | Interpolation using radial basis functions with application to inverse kinematics |
7753785, | May 06 2003 | Nintendo Co., Ltd. | Game apparatus, storing medium that stores control program of virtual camera, and control method of virtual camera |
8014985, | Mar 26 1999 | Sony Corporation; Sony Electronics Inc. | Setting and visualizing a virtual camera and lens system in a computer graphic modeling environment |
8414393, | Oct 28 2010 | KONAMI DIGITAL ENTERTAINMENT CO., LTD. | Game device, control method for a game device, and a non-transitory information storage medium |
20040219980, | |||
20100111370, | |||
20110242326, | |||
20140085293, | |||
20140195988, |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Apr 24 2014 | EDSALL, GERALD | Microsoft Corporation | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 041422 | /0203 | |
Oct 14 2014 | Microsoft Corporation | Microsoft Technology Licensing, LLC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 041422 | /0232 | |
Mar 01 2017 | Microsoft Technology Licensing, LLC | (assignment on the face of the patent) | / |
Date | Maintenance Fee Events |
Jan 26 2022 | M1551: Payment of Maintenance Fee, 4th Year, Large Entity. |
Date | Maintenance Schedule |
Aug 07 2021 | 4 years fee payment window open |
Feb 07 2022 | 6 months grace period start (w surcharge) |
Aug 07 2022 | patent expiry (for year 4) |
Aug 07 2024 | 2 years to revive unintentionally abandoned end. (for year 4) |
Aug 07 2025 | 8 years fee payment window open |
Feb 07 2026 | 6 months grace period start (w surcharge) |
Aug 07 2026 | patent expiry (for year 8) |
Aug 07 2028 | 2 years to revive unintentionally abandoned end. (for year 8) |
Aug 07 2029 | 12 years fee payment window open |
Feb 07 2030 | 6 months grace period start (w surcharge) |
Aug 07 2030 | patent expiry (for year 12) |
Aug 07 2032 | 2 years to revive unintentionally abandoned end. (for year 12) |