Moving scheduling of processor time for virtual processors (vps) out of a virtualization hypervisor. A host operating system schedules vp (virtual processor) processor time. The host operating system creates vp backing threads, one for each vp of each virtual machine. There is a one-to-one mapping between each vp thread in the host operating system and each vp in the hypervisor. When a vp thread is dispatched for a slice of processor time, the host operating system calls into the hypervisor to have the hypervisor start executing the vp, and the hypervisor may perform a processor context switch for the vp. Of note is the security separation between vp scheduling and vp context switching. The hypervisor manages vp context switching in kernel mode while vp scheduling is performed in user mode. There is a security/interface boundary between the unit that schedules vp processor time and the hypervisor.
|
8. A computing device comprising:
a central processing unit (CPU) comprising physical processors (pps);
storage hardware storing a hypervisor comprising a hypervisor kernel and a guest operating system (os) comprising a guest os kernel, the guest os kernel configured to run in a first virtual machine (vm) executed by a corresponding first virtual processor (vp) executing on a pp;
the hypervisor kernel configured to maintain representations of respective second virtual processors (vps) for second vms that do not include the first vm, wherein the first vp is not one of the second vps;
the guest os kernel configured to (i) make scheduling decisions for scheduling execution of the second vps and to (ii) identify a specific vp of the second vps to the hypervisor kernel when the guest os kernel decides to schedule the second BP, wherein operations to implement the scheduling decisions are performed by the hypervisor kernel; and
the hypervisor kernel further configured to, based on the guest os kernel identifying the specific vp to the hypervisor kernel, select a set of register values corresponding to the identified vp and swap the set of register values into a pp corresponding to the identified specific vp to begin executing the identified specific vp on the corresponding pp.
14. Computer storage hardware storing information configured to, when executed by processing hardware of a computing device, cause the computing device to perform a process, the process comprising:
executing a virtualization operating system that performs virtualization services for first virtual machines (vms) executing on the computing device, the virtualization services including virtualizing access to hardware of the computing device for the first vms, the virtualization operating system determining when the first vms will have execution time on a central processing unit (CPU) of the computing device, the CPU comprising physical processors (pps), the virtualization operating system executing in a privileged vm which is not one of the first vms, wherein when the virtualization operating system determines that a specific first vm is to have execution time, the privileged vm identifies the specific first vm to a hypervisor kernel; and
executing the hypervisor kernel on the CPU, the hypervisor kernel executing the privileged vm and the first vms on the pps of the CPU, wherein, based on the identifying of the specific first vm by the privileged vm, the hypervisor kernel executes the identified specific first vm by selecting a corresponding execution context comprising register values, and by swapping the register values into a pp to enable the specific first vm to begin executing on the pp.
1. A method performed by a computing device comprising a central processing unit (CPU) comprising physical processors (pps), the method comprising:
executing a hypervisor comprising a first kernel, and executing an operating system comprising a second kernel, wherein the operating system and the second kernel execute in a first virtual machine (vm) comprising a corresponding first vp;
managing, by the first kernel, second vms comprising respective second virtual processors (vps), the managing comprising maintaining physical processor states of the respective second vps and swapping the physical processor states into, and out of, the pps that correspond to the second vps, the physical processor states comprising respective sets of register values, wherein the second vps do not include the first vp; and
scheduling the second vps to execute on the pps that correspond to the second vps, wherein decisions to schedule the second vps for execution are made by the first vp executing the second kernel in the first vm, wherein the scheduling by the second kernel comprises:
determining, by the second kernel, to execute a specific vp of the second vps,
identifying, by the second kernel, to the hypervisor, the specific vp of the second vps to execute, and based on the identifying the hypervisor performs a processor context switch on a pp corresponding to the specific vp, the processor context switch performed using the physical processor state corresponding to the specific vp.
2. A method according to
3. A method according to
4. A method according to
5. A method according to
receiving a call from guest code executing in a second vm having a second vp scheduled by the second kernel; and
scheduling, by the second kernel, processing time for the second vp of the second vm or another second vm based on the call from the guest code.
6. A method according to
7. A method according to
9. A computing device according to
10. A computing device according to
11. A computing device according to
12. A computing device according to
13. A computing device according to
15. Computer storage hardware according to
16. Computer storage hardware according to
17. Computer storage hardware according to
18. Computer storage hardware according to
19. Computer storage hardware according to
20. A method according to
maintaining a mapping between identifiers of the vps and the respective physical processor states; and
accessing the physical processor state corresponding to the identified specific vp by referencing the identified specific vp in the mapping.
|
Among the many forms of computer virtualization, machine or system virtualization has become common due to many known advantages. System virtualization involves abstracting the hardware resources of a computer and presenting the computer as virtual machines. A layer of software referred to as a hypervisor or virtual machine monitor (VMM) runs directly on the hardware of a computer. The hypervisor manages access to the hardware of the computer by virtual machines (VMs), which are also known as partitions, domains, or guests. Each VM is a software environment or construct capable of hosting its own guest operating system. The hypervisor manages sharing of the computer's hardware, in particular processing hardware, by the VMs. The presence of a layer of software—the hypervisor—between the guest operating system and the computer hardware is mostly transparent to the guest operating system.
One type of hypervisor is the type I hypervisor, also referred to as a stand-alone, native, bare metal, or embedded hypervisor. Examples of type I hypervisors are Microsoft Hyper-V™ from Microsoft Corp., KVM™ (Kernel-based Virtual Machine) from Red Hat Corp., VMware ESXi™ from VMware Inc., and Xen™, which is available from various sources. Traditionally, in virtualization architectures that use a type I hypervisor (“hypervisor”, hereafter), the hypervisor is responsible for managing the virtualization of processors. Specifically, the hypervisor applies resource policies to virtual processors and schedules execution of virtual processors on physical processors. A hypervisor is usually accompanied by a privileged VM that hosts a special guest operating system that handles other virtualization-related tasks that need not reside in the hypervisor. This special operating system will be referred to as the host operating system or the host kernel. The hypervisor and host operating system together will be referred to as a virtualization layer. The virtualization tasks handled by the host operating system varies among hypervisor implementations. Some may handle aspects of device emulation/virtualization, I/O virtualization, inter-VM communication, etc. Some may provide a user interface to enable a user to configure the virtualization layer, VMs, etc. The host operating system may also have administrative tools or other software that is not necessary for virtualization.
Although which virtualization responsibilities are handled by the hypervisor and which are handled by the host operating system can vary, all type I hypervisors (including kernel-based hypervisors such as the KVM hypervisor) handle processor virtualization, including scheduling of processor time for virtual processors, managing the kernel objects (e.g. threads or processes) that back the virtual processors, etc. This approach has shortcomings.
One shortcoming of having the hypervisor handle processor scheduling is scheduler redundancy. Operating systems that are employed as VM guests or as host operating systems are usually designed to be capable of running directly on hardware. They have their own kernel scheduler designed for scheduling their own threads on processors (or on virtual processors, as the case may be). These operating systems' schedulers are usually heavily engineered for resource control and for supporting advanced power management policies. To date, hypervisors have also had their own scheduler implementation that is independent of the scheduler in the host/operating systems. Because a hypervisor has its own scheduler, some of the sophisticated scheduler logic often found in host/guest operating systems must either be omitted or else duplicated in the hypervisor scheduler. For example, supposing a host/guest kernel's scheduler is enhanced with a new feature to make its scheduling more power efficient, if the benefit is to be had on a computer where the host kernel executes in a VM on top of a hypervisor (which actually controls scheduling), this new feature must be duplicated in the hypervisor's scheduler for the host computer to obtain the same benefit.
The stacked layers of scheduling also make it difficult to realize certain performance and management improvements. Regarding performance, it has been observed that the nature of VM workloads often vary. For instance, some VMs might be I/O bound, others might be processor bound. A hypervisor scheduler cannot self-adjust to suit VM workload conditions because those conditions may not be visible to the hypervisor. For example, if some I/O virtualization occurs at the host operating system, actionable information about I/O conditions may not be available in the hypervisor. Regarding management, when the hypervisor handles processor scheduling, CPU resource reporting for virtual machines is not naturally accounted for in the host operating system. Because the host operating system does not have direct visibility to how a VM's virtual processors are being scheduled, the host operating system cannot determine how physical CPU resources are actually being consumed. CPU resource reporting tools in the host operating system may report an incorrect view of CPU resource consumption, since such tools are in fact reporting CPU resource consumption of the host operating system (and not the virtual machines). Accordingly, system tools must be retooled if system-wide CPU usage is to be monitored accurately in a virtualized environment. Moreover, hypervisor-based scheduling can make execution tracing and analysis difficult because of the need to correlate between events from two schedulers.
Implementing scheduling in the hypervisor also carries security risks. Because each VM's execution is controlled in the same security domain (the hypervisor), there is a risk that a hypervisor breach of one VM could lead to compromise of the hypervisor and thus compromise of the other VMs.
Some of the problems discussed above may be solved by scheduling virtual processors outside of the hypervisor, as described below.
The following summary is included only to introduce some concepts discussed in the Detailed Description below. This summary is not comprehensive and is not intended to delineate the scope of the claimed subject matter, which is set forth by the claims presented at the end.
Embodiments relate to moving scheduling of processor time for virtual processors (VPs) out of a virtualization hypervisor. In one embodiment, a host operating system schedules VP processor time. The host operating system creates VP backing threads, one for each VP of each VM. There is a one-to-one mapping between each VP thread in the host operating system and each VP in the hypervisor. When a VP thread is dispatched for a slice of processor time, the host operating system calls into the hypervisor to have the hypervisor start executing the VP, and the hypervisor may perform a processor context switch for the VP. Of note is the security separation between VP scheduling and VP context switching. In one embodiment, the hypervisor manages VP contexts and controls VP context switching while VP scheduling is performed in kernel mode in the context of the host operating system's VM. There is a security/interface boundary between the unit that schedules VP processor time and the hypervisor.
Many of the attendant features will be explained below with reference to the following detailed description considered in connection with the accompanying drawings.
The present description will be better understood from the following detailed description read in light of the accompanying drawings, wherein like reference numerals are used to designate like parts in the accompanying description.
Embodiments described below relate to shifting scheduling of processor time for virtual processors (VPs) from a hypervisor to a host operating system. In one embodiment, the host operating system schedules a VP by making a blocking call into the hypervisor to run a VP. The host operating system creates VP threads, one for each VP of each VM. There is a one-to-one mapping between each VP thread in the host operating system and each VP in the hypervisor. When a VP thread is dispatched for a slice of processor time, the host operating system calls into the hypervisor to have the hypervisor start executing the VP. When an interrupt occurs, execution of the VP stops, and the hypervisor returns control back to the host operating system. This allows the host operating system to then schedule a different thread, which can be a VP thread or just a regular software thread that has no relation to a virtual processor.
When the host operating system scheduler 138 determines that a VM is to be started, the host operating system creates VP backing threads for the respective VPs of the starting VM. Each VP of each VM has a VP backing thread. Each VP backing thread runs a dispatch loop until, for whatever reason, the corresponding VP is stopped. The scheduler 138 in the host VM schedules VP backing threads as regular threads subject to additional VM/VP-specific scheduling policies and possibly enlightenments (information provided by guests). The hypervisor 100 is configured with an interface 150 that exposes one or more hypercalls allowing the host operating system to make calls into the hypervisor.
To elaborate, the VP backing/dispatch threads are thread objects in the host operating system that function like any other thread object. The host operating system scheduler 138 schedules all of its threads including the VP backing threads. In one embodiment, each VP backing thread knows the identifier of the corresponding VP, and when a VP backing thread is scheduled to run, it calls into the hypervisor to dispatch the VP on the current processor. Each VP backing thread runs a dispatch loop. When a VP backing thread receives time to run, it makes a hypercall intercepted by the hypervisor that instructs the hypervisor to “run this VP”. The host operating system might need to tell the hypervisor, through hypercall parameters, which VP and/or VM to start running. The hypervisor only needs to know which VP (or VM) to run. The processor state for the VP need not be visible to the host VM and preferably the host VM is not permitted or able to access same directly.
From the perspective of the host VM, a VP is just another thread and execution of the VP by the hypervisor is transparent. When a VP backing thread is scheduled and starts executing, it is running on a PP (the one currently executing a VP of the host VM when the VP backing thread is started). The VP backing thread executes and invokes the hypercall to switch into the hypervisor (a privileged hypervisor operation). When entering hypervisor mode, the VP backing thread is initially still executing on the same PP with the host VM's context. As noted, the hypervisor maintains processor states of all VPs in the state table 149 and knows which VP the VP backing thread corresponds to. After saving the current PP state 152 (corresponding to the host VM's VP), the hypervisor swaps into the current PP the previously-stored processor state 154 of the VP that issued the hypercall, effectively beginning execution of the relevant VP. At this point, the VP is executing guest code on the PP. The hypervisor lets the VP run until either the VP blocks voluntarily, the VP generates an intercept for the root, or an interrupt has arrived for the root VP. On any of these three events the hypervisor context-switches back to the host's VP and the latter completes the hypercall and returns to the host VM. In sum, the hypervisor merely executes the VPs it is told to execute, and the host operating system controls the scheduling of VP execution.
Because the hypervisor is informed, through a hypercall parameter, which VP is executing, the hypervisor knows which processor state to swap in from the state table 149. Note that in one embodiment the host operating system sees each root/host VP as a logical/physical processor and does not need to schedule its own processing time in the same way that it schedules other VM processing time. When a VP is executing and there is an interrupt or other event that needs to be taken care of by host operating system, the hypervisor just switches context to the corresponding root/host VP (there is a one-to-one mapping between each root/host VP and PP). When the root VP enters (returns to) the host VM, it is running in the context of whatever thread made the hypercall, and the host kernel decides what to do next.
To summarize, there is no scheduler in the hypervisor. There may be one stack per PP in the hypervisor. The host VM owns all PP scheduling and idling decisions. Each VP has VP backing thread or other unit of execution. The host VM schedules the VP backing threads. Each VP backing thread: runs a dispatch loop, makes a hypercall to switch from the host VP to the corresponding target VP (e.g., VP context switch hypercall), and optionally provides host targeted work requests on return from the VP context switch hypercall.
Although a host operating system is convenient, other techniques for separating VP scheduling from the hypervisor may be used. In one embodiment, the privileged extra-hypervisor functionality of the host operating system need not be executed in a VM or container. Any module that interfaces with the hypervisor can perform the VP scheduling. For example, a module with an execution context permanently running on a PP can avoid context switches between host VPs and hypervisor idle threads. This may also reduce code path lengths and latencies for entering leaving idle states. In addition, as mentioned above, enlightenments may be implemented to allow guest VMs to provide the host VM with information relevant to scheduling, such as workload profiles, scheduling priorities, I/O statistics, etc.
Removing VP scheduling from the hypervisor can provide other advantages. Newer CPUs may have different types of cores on the same chip (so-called “big.LITTLE” designs). This type of CPU can be leveraged by exposing the CPU topology to guest VMs (e.g., big VPs and LITTLE VPs). The kernel scheduler in a big.LITTLE guest VM makes its own scheduling decisions. The host operating system is aware of which VP backing threads are big and which ones are LITTLE. The host operating system's scheduler decides how to schedule big and LITTLE VP backing threads according to the host operating system's power management policies and constraints.
Removing VP scheduling from the hypervisor can have other advantages. The host operating system may have tools for analyzing system performance. If host threads are used to back VPs, tools for analyzing thread performance can provide direct insights into VP performance. (e.g., CPU usage, context switches, dispatch latencies, etc.). That is, because each VP is backed by a host thread, the host kernel can maintain the counters inside the host VM such as time spent in the hypervisor, time spent inside of its partition, time spent in the host kernel, etc.
In some embodiments, the VM guest operating systems and the host operating system might have the same kernel; i.e., each might have an instance of the same basic operating system kernel, albeit within different VMs and for different purposes. In such a case, enlightenments can allow for better integration of lightweight VM containers that can host applications. This technique provides improved isolation and sandboxing of applications (i.e., better security but with acceptable performance). As noted above, other guest enlightenments might be supported. For example, a guest might invoke a hypercall to share information about its workload, processing priorities or characteristics, etc. The host operating system may take this into account when making scheduling decisions.
The computing device 102 may have one or more displays 222, a network interface 224 (or several), as well as storage hardware 226 and processing hardware 228, which may be a combination of any one or more: central processing units, graphics processing units, analog-to-digital converters, bus chips, FPGAs, ASICs, Application-specific Standard Products (ASSPs), or Complex Programmable Logic Devices (CPLDs), etc. The storage hardware 226 may be any combination of magnetic storage, static memory, volatile memory, non-volatile memory, optically or magnetically readable matter, etc. The meaning of the term “storage”, as used herein does not refer to signals or energy per se, but rather refers to physical apparatuses and states of matter. The hardware elements of the computing device 102 may cooperate in ways well understood in the art of machine computing. In addition, input devices may be integrated with or in communication with the computing device 102. The computing device 102 may have any form-factor or may be used in any type of encompassing device. The computing device 102 may be in the form of a handheld device such as a smartphone, a tablet computer, a gaming device, a server, a rack-mounted or backplaned computer-on-a-board, a system-on-a-chip, or others.
Embodiments and features discussed above can be realized in the form of information stored in volatile or non-volatile computer or device readable storage hardware. This is deemed to include at least hardware such as optical storage (e.g., compact-disk read-only memory (CD-ROM)), magnetic media, flash read-only memory (ROM), or any means of storing digital information in to be readily available for the processing hardware 228. The stored information can be in the form of machine executable instructions (e.g., compiled executable binary code), source code, bytecode, or any other information that can be used to enable or configure computing devices to perform the various embodiments discussed above. This is also considered to include at least volatile memory such as random-access memory (RAM) and/or virtual memory storing information such as central processing unit (CPU) instructions during execution of a program carrying out an embodiment, as well as non-volatile media storing information that allows a program or executable to be loaded and executed. The embodiments and features can be performed on any type of computing device, including portable devices, workstations, servers, mobile wireless devices, and so on.
Patent | Priority | Assignee | Title |
11556371, | Mar 24 2020 | Red Hat, Inc. | Hypervisor task execution management for virtual machines |
Patent | Priority | Assignee | Title |
10176095, | May 05 2014 | Microsoft Technology Licensing, LLC | Secure management of operations on protected virtual machines |
10191861, | Sep 06 2016 | FireEye Security Holdings US LLC | Technique for implementing memory views using a layered virtualization architecture |
10430226, | May 03 2013 | VMware LLC | Dynamic virtual machine sizing |
10447728, | Dec 10 2015 | FireEye Security Holdings US LLC | Technique for protecting guest processes using a layered virtualization architecture |
10831556, | Dec 23 2015 | Intel Corporation | Virtual CPU consolidation to avoid physical CPU contention between virtual machines |
6269391, | Feb 24 1997 | Oracle International Corporation | Multi-processor scheduling kernel |
7765428, | Sep 30 2005 | International Business Machines Corporation | Replacing a failing physical processor |
7797512, | Jul 23 2007 | Sun Microsystems, Inc; Sun Microsystems Technology LTD | Virtual core management |
8219988, | Aug 02 2007 | International Business Machines Corporation | Partition adjunct for data processing system |
8225315, | Jul 23 2007 | Sun Microsystems, Inc; Sun Microsystems Technology LTD | Virtual core management |
8464258, | Jun 02 2010 | Hitachi, Ltd. | Virtual computer and CPU allocation method |
8539499, | Feb 18 2008 | Parallels International GmbH | Symmetric multiprocessing with virtual CPU and VSMP technology |
8543790, | Apr 27 2006 | VMware LLC | System and method for cooperative virtual machine memory scheduling |
8695007, | Mar 09 2010 | Hitachi, LTD | Computer system and method of scheduling a virtual processor to run on physical processors based on the number of possessing cycles of each virtual computer |
9304803, | Nov 12 2012 | VMware LLC | Cooperative application workload scheduling for a consolidated virtual environment |
9417927, | Apr 01 2014 | International Business Machines Corporation | Runtime capacity planning in a simultaneous multithreading (SMT) environment |
9778943, | May 28 2013 | Red Hat Israel, Ltd.; Red Hat Israel, Ltd | Timer based virtual processor scheduling and suspension on physical processor for use of physical processor by other processing |
20060200663, | |||
20060259818, | |||
20080184227, | |||
20090037911, | |||
20090106754, | |||
20100125843, | |||
20110246171, | |||
20120096206, | |||
20120227042, | |||
20120291027, | |||
20130185720, | |||
20140137104, | |||
20140149633, | |||
20140282507, | |||
20140304414, | |||
20140331222, | |||
20150007170, | |||
20150033228, | |||
20150199532, | |||
20150334126, | |||
20150370592, | |||
20160147549, | |||
20160162336, | |||
20160179560, | |||
20160253196, | |||
20160371110, | |||
20160371118, | |||
20160371127, | |||
20170244557, | |||
20180060103, | |||
20180101486, | |||
20180349143, |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Mar 31 2017 | Microsoft Technology Licensing, LLC | (assignment on the face of the patent) | / | |||
Apr 14 2017 | OKS, ARTEM | Microsoft Technology Licensing, LLC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 042029 | /0358 | |
Apr 17 2017 | HEPKIN, DAVID | Microsoft Technology Licensing, LLC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 042029 | /0358 |
Date | Maintenance Fee Events |
Aug 20 2024 | M1551: Payment of Maintenance Fee, 4th Year, Large Entity. |
Date | Maintenance Schedule |
Mar 23 2024 | 4 years fee payment window open |
Sep 23 2024 | 6 months grace period start (w surcharge) |
Mar 23 2025 | patent expiry (for year 4) |
Mar 23 2027 | 2 years to revive unintentionally abandoned end. (for year 4) |
Mar 23 2028 | 8 years fee payment window open |
Sep 23 2028 | 6 months grace period start (w surcharge) |
Mar 23 2029 | patent expiry (for year 8) |
Mar 23 2031 | 2 years to revive unintentionally abandoned end. (for year 8) |
Mar 23 2032 | 12 years fee payment window open |
Sep 23 2032 | 6 months grace period start (w surcharge) |
Mar 23 2033 | patent expiry (for year 12) |
Mar 23 2035 | 2 years to revive unintentionally abandoned end. (for year 12) |