Methods, systems, and apparatus, including computer program products, for assuring application performance by matching the supply of resources (e.g., application resources, VM resources, or physical resources) with the fluctuating demand placed on the application. For example, the systems and methods disclosed herein can be used to ensure that the application is allocated sufficient resources when it is initially deployed to handle anticipated demand; dynamically alter the resources allocated to the application during operation by matching the resource requirements to the actual measured application demand; and predict future resource requirements based on planning assumptions related to future application demand.
|
1. A computer-implemented method for dynamic allocation of abstracted computer resources in a virtualization system, comprising:
determining, by an application manager running on a data processor in the virtualization system, a computer resource bundle to be purchased for a computer application running on a virtual machine in the virtualization system, using the computer application's available budget of virtual currency units;
automatically allocating the computer resource bundle from a computer server in the virtualization system for usage by the application;
determining, following an increase in the purchase price for the allocated computer resource bundle based at least in part on increased utilization of at least part of the bundle, that there is not sufficient budget of remaining virtual currency units available to pay for the application's continued usage of the entire bundle;
initiating a credit check process by the application manager, wherein the credit check process results in an increase in the application's budget of available virtual currency units; and
using the increased budget of available virtual currency units to purchase, for the application, continued usage of a determined amount of the bundle able to be purchased using the increased budget,
wherein the virtual machine is a software implementation of a computer that runs at least one guest operating system and executes one or more applications on the guest operating system.
17. A system for regulating dynamic allocation of abstracted computer resources, comprising:
a computer-based virtualization system that runs at least one guest operating system; and
instructions stored on a non-transitory computer readable medium in the virtualization system and executable by a data processing apparatus to cause the data processing apparatus to perform operations comprising:
determining, by an application manager running on a data processor in the virtualization system, a computer resource bundle to be purchased for a computer application running on a virtual machine in the virtualization system, using the computer application's available budget of virtual currency units;
automatically allocating the computer resource bundle from a computer server in the virtualization system for usage by the application;
determining, following an increase in the purchase price for the allocated computer resource bundle based at least in part on increased utilization of at least part of the bundle, that there is not sufficient budget of remaining virtual currency units available to pay for the application's continued usage of the entire allocated bundle;
initiating a credit check process by the application manager, wherein the credit check process results in an increase in the application's budget of available virtual currency units; and
using the increased budget of available virtual currency units to purchase, for the application, continued usage of a determined amount of the bundle able to be purchased using the increased budget,
wherein the virtual machine is a software implementation of a computer that runs at least one guest operating system and executes one or more programs or applications on the guest operating system.
2. The computer-implemented method of
receiving, from a proxy manager of a second virtualization system offering the computer resource bundle, a purchase price for the determined computer resource bundle in virtual currency units, wherein the second virtualization system also runs at least one guest operating system;
determining, following initiation of the credit check process, whether the determined computer resource bundle is to be purchased from a second computer server in the second virtualization system based at least in part on the purchase price received from the proxy manager of the second virtualization system; and
allocating the computer resource bundle from the second computer server in the second virtualization system to the application.
3. The computer-implemented method of
maintaining a budget, in virtual currency units, for the application;
determining whether the budget of virtual currency units for the application is sufficient to pay the purchase price for the computer resource bundle;
allocating the computer resource bundle to the application only if the budget of the application has been determined to be sufficient to pay the purchase price for the computer resource bundle; and
transferring virtual currency units from the budget for the application to a budget for the computer server based on the purchase price for the computer resource bundle determined for the computer server.
4. The computer-implemented method of
determining a performance metric level of the application; and
adjusting the allocation of at least one of the one or more computer resources to the application based on a comparison of the determined performance metric level to a predetermined threshold performance metric level.
5. The computer-implemented method of
6. The computer-implemented method of
determining a performance metric level of the application; and
adjusting a budget of virtual currency units for the application based on a comparison of the determined performance metric level to a predetermined threshold performance metric level.
7. The computer-implemented method of
8. The computer-implemented method of
9. The computer-implemented method of
10. The computer-implemented method of
11. The computer-implemented method of
12. The computer-implemented method of
13. The computer-implemented method of
14. The computer-implemented method of
15. The method of
16. The method of
18. The system of
receiving, from a proxy manager of a second virtualization system offering the computer resource bundle, a purchase price for the determined computer resource bundle in virtual currency units, wherein the second virtualization system also runs at least one guest operating system;
determining, following initiation of the credit check process, whether the determined computer resource bundle is to be purchased from a second computer server in the second virtualization system based at least in part on the purchase price received from the proxy manager of the second virtualization system; and
allocating the computer resource bundle from the second computer server in the second virtualization system to the application.
19. The system of
maintaining a budget, in virtual currency units, for the application;
determining whether the budget of virtual currency units for the application is sufficient to pay the purchase price for the computer resource bundle;
allocating the computer resource bundle to the application only if the budget of the application has been determined to be sufficient to pay the purchase price for the computer resource bundle; and
transferring virtual currency units from the budget for the application to a budget for the computer server based on the purchase price for the computer resource bundle determined for the computer server.
20. The system of
determining a performance metric level of the application; and
adjusting a budget of virtual currency units for the application based on a comparison of the determined performance metric level to a predetermined threshold performance metric level.
21. The system of
22. The system of
23. The system of
|
This
where
C=capacity of the resource;
U=amount of resource used; and
x=new demand.
Such a pricing function is proportional to costs, penalizing high utilization. When the utilization u=(U+x)/C approaches its limit of one, prices increase rapidly, preventing all but the highest budget applications from accessing the resource. For example, suppose containers require, on average, 2% of the CPU capacity of servers, but 20% of their storage I/O capacity. In this scenario, a container wanting to deploy with a server supporting three containers will see the following CPU and storage I/O prices:
priceCPU[0.02C]=costCPU/(1−0.08C/C)4=costCPU/0.924=1.4*costCPU
priceI/O[0.2C]=costI/O/(1−0.8C/C)4=costI/O/0.24=625*costI/O.
Thus, in the above-described scenario, CPU is priced at a relatively small multiplier of the cost base of CPU, while the storage I/O is priced at a relatively large multiplier of the cost base of I/O. Although specific pricing considerations and mechanisms have been described, a large variety of pricing functions may be used according to other embodiments to best reflect specific use considerations.
Composite service objects, which are objects that include more than one service object and which relate to the provision of multiple types of services, may take the following form according to various embodiments:
<service-identifier, service-1, service-2 . . . , service-n>,
where service-k is either a simple or composite service object and is referred to as a component of the composite service. In some embodiments, the “duration” attributes of all components of a composite service are identical, and their common value is called the duration of the composite service. For example, a hardware server may be described by the following composite service object:
<<server-1, Server, LS41>, CPU4, Memory-2, NIC-3, NIC-4, HBA-2>
where Memory-2, NIC-3, NIC-4 and HBA-2 indicate respective simple service objects associated with respective memory services, LAN-interface services provided by two NICs, and SAN I/O services provided by HBA-2. The HBA-2 may itself be described by a simple service object as follows:
<<HBA-2, FC-HBA, Emulex, LP11000-M4>, 0.1 Gbps, 1.1 Gbps, 2.9 Gbps, 1 hr, price(x)>.
This service object indicates that the duration of the composite service is one hour, as the durations of all components of a composite service are identical.
In some embodiments, the price of a composite service is defined as the sum of the prices of all its components. For example, the price of a server object is the sum of the prices of the units of CPU, memory, network I/O and storage I/O required by a consumer.
The supply chain model databases 246 are maintained by element managers (such as element managers 234, 236, 238, 240, 242, and 244 shown in
As mentioned above, the supply chain economy matches consumers and providers of resources or services by using pricing and budgeting. According to various embodiments, demand for services is matched to supply through a shopping model. A consumer element manager (such as one of element managers 234, 236, 238, 240, 242, and 244 shown in
Query: Server, CPU.units=50 Mhz, Memory.units=4 GB, StorageIO.units=200 Mbps, NetworkIO.units=100 Mbps.
Such a query may retrieve records of composite service objects of the servers 214 offering the respective CPU, memory, storage I/O and network I/O capacity at the lowest price. Once the consumer element manager acquires these records of lowest-priced service objects, it can proceed to extract the identities of the element managers posting these service offerings. The consumer element manager may then pursue direct interactions and contract with one or more respective provider element managers to acquire and pay for the desired services. There exists the possibility that multiple consumers may query the supply chain model databases 246 simultaneously for similar services, and thus potentially interfere with each other's shopping processes. Such interference may be avoided, for example, by providing standard locking mechanisms to maintain atomicity of the query and purchase transactions.
Moreover, various embodiments may use an auction, or bidding model, rather than a shopping model, to match demand and supply. For example, consumer element managers may post respective bids for services in a bidding database, which a provider element manager may then query for the highest bid price offered for its services and contract to serve it. The shopping model is generally preferred to bidding in situations where consumers' demands arrive asynchronously and unpredictably. In such cases, an arriving consumer can find the low-cost provider by searching the supply chain model databases 246. In contrast, a bidding process requires providers to poll, whether constantly or at intervals, the bidding database to detect arrivals of new bids, while bidding consumers may be required to wait until enough providers have polled the bidding database and accepted the bids, and thus contract with providers based at least in part on chance. There are various situations where bidding may offer benefits over shopping, and those situations may be handled using the principles described herein.
As shown in
The provider element manager determines whether the consumer budget is sufficient to pay the price for the requested provider services (decision block 506). If it is determined that there is sufficient budget, the provider element manager deploys the consumer at the provider, which proceeds to process its workload (step 508). For example, CPU and memory resources that have been purchased may be allocated to a container by the underlying scheduler of the container system, which may include the use of a traditional operating systems scheduling algorithm. The server element manager configures the scheduler parameters to accomplish fairly accurate allocation of the CPU and memory. Memory may be allocated by specifying an amount of memory to be provided. The container system can allocate physical memory, based on these specifications, or support virtual memory mechanisms that permit over 100% utilization of physical memory. Additionally, the CPU may be allocated by configuring reservations and shares parameters of the scheduler. For example, reservations may be used to allocate a reserved CPU slice, using a time-shared round-robin scheduler, while shares allocate the remaining CPU bandwidth through a Weighted Fair Queuing scheduler. CPU reservations and shares may be viewed as separate services, and may be individually priced according to supply and demand. For example, a low-priority application may be unable to buy reservations, and may thus need to settle for shares, which may be priced lower. A high-priority, mission-critical application, on the other hand, may have sufficient budget to afford sufficient reservations to support its needs.
Otherwise, if it is determined that there is not sufficient budget, the consumer element manager initiates a credit check process to decide whether the consumer can increase its budget or sufficiently lower its service demands, and thus continue to run (decision block 510). For example, suppose the consumer is a container whose budget is short of paying the cost of a provider server. In that case, the container may use credit it has accumulated to pay for the service, obtain additional budget from the applications it serves, or reduce its demand for services and the corresponding price to the point where it can afford to pay. If one or more of these scenarios is possible, the consumer uses credit, increases its budget and/or lowers its service demands (step 512), and the provider element manager thus deploys the consumer at the provider as described above. Otherwise, if none of these options is available, the consumer is suspended and then will either terminate or re-launch when adequate budget becomes available to it (step 514), as described in greater detail below.
After the provider element manager deploys the consumer at the provider, the provider element manager or the consumer element manager monitors consumer resource usage and adjusts allocation of resources to optimize or improve the use of the consumer's budget (step 516). For example, the provider element manager may find that the consumer is using only 20% of one service it bought, while using 90% of another service it bought. In that case, the provider element manager may reduce the allocation of the first service and use the corresponding released budget to increase the allocation of the second resource.
Upon completion or termination of the consumer service period, the provider element manager notifies the consumer element manager (step 518), which may proceed to shop for a new provider offering lowest cost services to meet the consumer's needs (step 520). The consumer element manager determines whether the price of the new provider found is lower than the price of the old provider (where the consumer resides at the time), or according to some embodiments, whether it is lower by a threshold amount (decision block 522). Assuming it is, the consumer element manager moves the consumer to the new provider, in which case it may also adjust the budget to reflect the price of moving, if any (step 524). Namely, according to various embodiments, a price of moving may be factored into the decision making process for whether the consumer should be moved to the new provider, and such price may be subtracted or deducted from the available budget. Otherwise, if the consumer element manager decides to keep the consumer with the old provider, it does not adjust the budget to reflect the price of moving. In either case, the provider element manager (of the new or old provider) checks to see if the consumer budget is sufficient to pay for the provider as described above.
According to various embodiments, the process of shopping for a new provider 520 may depend on specific characteristics of the consumer, the resource, and/or the provider. For example, the containers 120 and 124 shown in
As an example, in the step 520 described above and shown in
According to various embodiments, at step 524, the budget of the consumer element (e.g., container 120) may also be adjusted (e.g., increased or decreased) based at least in part in such change in pricing. As indicated above, in an alternative embodiment, the pricing of resources (e.g., associated with the I/O pathway) may be increased to account for performance improvement that would result from movement of a consumer element to another server and the resulting localization.
According to other embodiments, the process of shopping for a new provider 520 may depend on functional characteristics of the consumer or provider. For example, the server 102 may be used to support development of containerized applications. The server 104—the provider, for example—may be used for testing the containerized application 130—the consumer, in this example. The process 500 may be used to select a new provider (the server 104), from among a group of servers providing rests of containerized applications, to run tests (consumer) of the containerized application 130. Similarly, the server 104 may be a production system running containerized applications and the process 500 may be used to dispatch the containerized application 130, and its container 120, from the development server 102 to the production server 104.
The order of steps in the example process 500 described above is illustrative only, and can be done in different orders. Moreover, it is contemplated that modifications and extensions of the process 500 will be used according to various embodiments. For example, a consumer may need to contract with two or more providers to be deployed, as in the case of a container that needs to acquire a bundle of resources offered by a server as well as SAN switch bandwidth and storage space at a storage array. In such scenarios, deployment of the consumer can be supported by extending step 502 to shop for multiple providers and then repeating the remaining steps for each of these providers. Additionally, for example, as explained below with respect to
According to various embodiments, the above-described supply chain economic principles may also be used to manage software licenses, such as temporary (time-limited) software licenses. For example, regardless of type (such as authorizations of software use per user, per CPU, per server, or per container), licenses may be modeled as resources to be purchased by an application manager 234, much like other resources that it may purchase from the container 212. License element managers (while not shown, may be included as part of the platform layer 230) may be used to set the prices of the licenses based on costs and demands In this manner, license management may be greatly simplified and unified with the allocation of other types of resources. For example, an application that is unable to acquire a needed license may suspend its operations and release its resources, as explained below, thus increasing the overall efficiency of the system. Additionally, licenses may be more efficiently used, since in situations where the licenses are highly utilized, they will be allocated to high priority tasks, while lower priority tasks may be suspended until they can afford the licenses. As soon as a license is no longer needed, it may be released and available for other tasks. Additionally, an administrator may consider the ROI of licenses, as with other resources, to plan the expansion, or contraction, of licenses capacity. For example, if a license's ROI is above a certain threshold, it may be desirable to acquire more licenses to increase the supply to meet demand.
Following the initial monitoring of resource utilization and optimizing of the container's budget (step 516), it is determined whether the consumer service period has terminated (decision block 602), in which case the provider element manager notifies the container element manager (step 518) as described above. Otherwise, the container element manager monitors and obtains the value of the SLA parameter of interest, such as the average transaction rate of an application, the average transaction delay of an application, the average communications latency of the application, or the number of transactions performed within a predetermined prior time period by an application (step 604). For example, an application element manager may monitor the value of the SLA parameter, through respective instrumentation, and inform the container element manager of the SLA parameter. The application may define its SLA goal as 100 transactions per second, in which case the SLA parameter of interest is transaction-rate. In general, because SLA parameters can be assumed to increase monotonically with the amount of resources allocated to an application, the management of SLAs may be accomplished as described herein by finding a budget and a respective resource allocation that will accomplish the target SLA value.
The container element manager determines whether the SLA parameter of interest is below a desired target (decision block 606), in which case, for example, the application's payments to the container (e.g., of virtual currency units) are increased such that the container's budget is increased, and it is able to purchase more resources to increase the SLA parameter of the application (step 608). After such an increase, the container's budget use is again monitored and optimized or improved as described above.
If the container manager determines that the SLA parameter is at or above the desired target, it is determined whether the SLA parameter exceeds the desired target by more than an acceptable threshold (decision block 610), in which case the payments are reduced, thus reducing the container's budget and the resources it buys, saving on applications costs, and keeping the SLA performance within a desired tolerance range (step 612). After such a reduction, the container's budget use is again monitored and optimized or improved as described above. If the SLA parameter is within the acceptable range, however, a reduction is not applied, and the process is repeated until it is determined that the consumer service period has been completed or terminated.
According to various embodiments, the process 600 for delivering service level agreement targets through resource allocation in a container system may be modified, adapted, and/or simplified for certain resources and SLA metrics. For example, in the case of allocation of I/O pathways to reduce or minimize latency, the process 600 may be modified as follows. The SLA parameter may be selected as the latency-hop-count, e.g., the number of physical switches traversed by an I/O pathway. For example, I/O pathways between elements located, or resident, at the same server (e.g., the containers 120 and 122 in
According to various embodiments, the latency-hop-count associated SLA value may be described with respect to the ordinal preference {Class-0, Class-1, Class-2, . . . Class-n}, where Class-0 is preferred to Class-1, Class-1 is preferred to Class-2, and so on to the extent additional Classes are defined. With respect to the process 600, a comparison can be made between a Target Latency Class and an Actual Latency Class (e.g., Target=Class-0, Actual=Class-1) at step 606. If the Actual Latency Class does not meet the Target Latency Class, payments to the consumer (e.g., the container) may be increased at step 608, and, following return to step 516, an I/O pathway can be acquired that can deliver the Target SLA Value (e.g., Class-0). For example, the process 600 described with respect to
It will be understood that the SLA-delivery process 600 described above may be flexibly adapted to achieve various goals, such as improving its handling of stochastic fluctuations of an SLA parameter. For example, the steps of increasing (step 608) and decreasing (step 612) payments by the application to the container may use standard mechanisms of Stochastic Approximation theory, including the Robbins-Monro or Kiefer-Wolfowitz algorithms, to regulate the changes in payments to assure convergence. Such a design may be implemented, for example, to achieve more desirable results in connection with non-monotonic SLA parameters. For example, an embodiment using a Robbins-Monro procedure may replace steps 606-612 with the following iteration:
R(n+1)←R(n)+a(n)[SLATarget−SLAParameter(R(n))]
where n is a counter of the iterations, R(n) is a vector describing the resource bundle allocated after n iterations, SLATarget is the desired value of the SLAParameter, and SLAParameter(R(n)) is the observed value of the SLAParameter after n iterations. The vector a(n) represents the increase/decrease of resources through the n-th step of the iteration; typically a(n)=a/n, where a is a fixed bundle.
Although the SLA-delivery process 600 described above uses an economic model and virtual currency units to control SLA levels, other manners of controlling SLA levels may be used according to various embodiments. For example, the allocation of resources to a container, or to an application, may be independent of any economic budget or transfer of virtual currency units, and may instead be based on other measures of an application's or container's importance.
The process 500 described above may also be modified or extended according to various other embodiments. For example, since current container systems are not readily adaptable to handling the management of storage I/O through HBA or storage systems schedulers, as an alternative to an arbitrary first-come-first-serve process, the process 500 described above may be modified or extended as shown in
Following the deployment of the container at a server (step 508), the server element manager monitors storage or network I/O usage by one or more containers, such as by collecting data from one or more of the container system, the HBAs (step 702), or the NIC. According to various embodiments, the server element manager may be configured to prevent congestion along storage I/O pathways, as might occur in cases of usage levels approaching the capacity limits. For example, the server element manager may prevent congestion by using pricing functions as described below that increase prices dramatically when utilization approaches 50% of the capacity.
The server element manager optimizes or improves the resources allocated to containers, as described above (step 516), such that containers acquire a share of the storage I/O resources that is commensurate with and optimally reflects their budget. The server element manager then periodically estimates both the average storage I/O capacity used and the average available I/O capacity, and updates the respective attributes of the storage I/O objects in the above-described supply chain model databases 246 with this usage data (step 704). It is noted that the usage data reported to the supply chain model databases 246 will impact price computations, with excessive utilization of storage I/O capacity resulting in respective price increases, and higher prices in turn deflecting demand by new or existing containers to servers with lower utilization (and prices) of storage I/O. For example, price competition over using storage I/O resources may result in migration of low budget containers from overloaded servers to other servers where storage I/O resources are more highly available, and are thus priced lower. Higher priority containers, on the other hand, may use their higher budgets or credit to obtain a preferential share of storage I/O resources.
The server element manager also computes the actual (versus projected) costs expended by each container, and applies these prices to handle its current commitments to containers (step 706). For example, higher usage of storage I/O results in higher prices and immediate costs assigned to containers, such that containers of lower priority and high storage use requirements may quickly exhaust their budget or credit and be suspended or terminated, as described below. In this manner, the low priority containers relinquish storage I/O capacity to containers having a higher priority and, thus, a higher budget.
Based on the computed costs, the server element manager evaluates whether the container's budget is sufficient to pay the cost (decision block 708). If it is, the service period of the container continues until it ends, and the server element manager notifies the container element manager of the completion of the service period (step 518).
Otherwise, if the container's budget is not sufficient, the server element manager evaluates whether the container's credit (costs minus budget) exceeds an acceptable credit threshold (decision block 710). According to various embodiments, high priority containers may have higher budgets and credits and can thus afford to overpay the server element manager to guarantee that they do not run out of storage I/O resources. If it is determined that the container's credit exceeds the threshold, the container element manager initiates a credit check process to decide whether the container can increase its budget or sufficiently lower its service demands, and thus continue to run (decision block 712). If possible, the container makes any necessary adjustments (such as a budget increase in the case of high priority containers, or reduced service demands) and continues to run (step 714), until the service period has ended and the server element manager has notified the container manager of the termination of the service period as described above. Otherwise, the server element manager suspends or terminates the container execution and notifies the container element manager, which becomes responsible for addressing the suspension or termination (step 716).
Upon termination of the service period and notification to the container element manager, the server element manager reports usage data to the container element manager and settles any credit, overpayments or underpayments with the container element manager (step 718). The container element manager may then proceed to shop for a new server offering lowest cost services to meet the container's needs (step 520), as explained above.
The economic-based scheduling process 700 described above may be used effectively to de-correlate peaks of competing, bursty I/O flows. For example, consider the scenario of four containers sharing a common server and a 4 Mbps Fiber Channel HBA, where the containers generate average storage I/O flows of 250 Mbps, 250 Mbps, 200 Mbps and 300 Mbps, respectively. The aggregate demand average of 1 Gbps consumes only 25% of the HBA capacity. A resource scheduler may limit its consideration to only the average demand which, in this case, would be manageable by the HBA and SAN. However, consider an alternate scenario where the I/O traffic streams are bursty, with a peak/average ratio of five for each container. If the four I/O streams associated with the containers are uncorrelated, their peaks will be likely dispersed and the peak of the aggregate stream will generally be less than 2 Gbps, which can be handled by the HBA and SAN with negligible or relatively few queuing delays. However, if the I/O streams are correlated, their peaks may be compounded to generate, for example, up to 5 Gbps peaks, utilizing 125% of the capacity and generating sustainable congestion, delays, and losses. The scheduling process 700 described above reduces the likelihood of compounded peaks, since they result in peak prices and a corresponding depletion of budgets and credits of low budget containers, leading to suspension, termination, or migration of such containers to servers with lower storage I/O prices until they find servers where their peaks are sufficiently de-correlated from other containers.
Thus, the allocation of containers to common servers according to the scheduling process 700 may result in substantially de-correlated peaks and substantially reduce the peak/average ratio seen by servers. For example, consider the example of four containers above. If their peaks are uncorrelated, the peaks of the aggregate stream will generally require at most 1.5 Gbps (the peak of the largest component stream), while their average traffic is 1 Gbps. The burstiness ratio (peak/average) of the aggregate stream 1.5/1=1.5 therefore represents only 30% of the burstiness of the individual streams (1.5 divided by 5). The economic-based scheduling process 700 described above substantially reduces interference not only between traffic averages, but it also reduces the interference between correlated traffic peaks. This results in smoother, less bursty, aggregate workloads, which may permit more efficient processing.
It will be understood that, according to various embodiments, the process 700 described above to manage storage I/O flows may applied to other forms of I/O, such as network I/O. For example, the above description should be understood to include alternative processes whereby references to “storage” are replaced by references to “network.” It will similarly be understood that storage I/O flows typically utilize network-I/O flows, such as Ethernet (e.g., Fibre Channel over Ethernet (FCoE)), Transmission Control Protocol/Internet Protocol (TCP/IP) (e.g., Network File System (NFS)), and SAN (e.g., Fibre Channel (FC), Internet Small Computer System Interface (iSCSI)), to transfer information such as storage access commands. The scheduling process 700 is therefore independent of the specific underlying network, and of the specific access commands carried by the described flows. Accordingly, the process 700 may be applied to schedule network I/O flows and thereby provide similar or identical benefits to those associated with storage I/O flows, such as smoothing the peaks of bursty traffic and/or supporting priority services.
The order of steps described above with respect to scheduling process 700 is illustrative only, and can be done in different orders. Moreover, the aforementioned beneficial effects are true not only for I/O streams, but for workloads sharing other resources as well.
The contracting of services between a consumer and a provider, as described in the example processes above, may include the use of a standard request-response protocol (such as SOAP) to submit a purchase order to the provider and transfer a respective payment. In response, the provider may deploy the service requested by the consumer and respond with a service confirmation.
As described below, various embodiments may also be used to address the problems of container sprawling and energy consumption in container systems using supply chain economics. Regarding sprawling, as explained in greater detail below, these embodiments may be used to suspend or terminate containers that are no longer needed or productive. These embodiments may also be used to terminate containers, or to disallow their re-activation if in a standby state, that are determined to be inconsistent with the current versions of their container system and applications. Regarding energy consumption, these embodiments may be used to consolidate and shift containers into fewer servers, for example, while still providing desired SLA performance, and switching other unused or non-productive servers OFF or into standby mode to reduce energy use. The supply chain software model and processes described above provide mechanisms and metrics to quantify how productive or non-productive a service element is.
The following description details an example process 900, shown in
A container is first initialized, for example, through the use of an initialize signal generated by a management station (step 902) or an automated action of a container manager. Similarly, for example, an application element may interpret events generated by a launch as an initialize signal.
After being initialized, the container attempts to obtain an initial budget to acquire resources for its operations (step 904). It is next determined whether the container was successful in obtaining an initial budget (decision block 906), in which case the container tries to acquire the resources needed to launch a respective service component (step 908). Otherwise, it begins the termination procedure by releasing any resources allocated to it (step 910).
If the container is successful at acquiring resources (decision block 912), it is provisioned, deployed, and remains in an active state (step 914) until it receives a signal to switch the service element OFF to an idle or standby state (step 916). After the terminate signal has been received, the container begins the termination procedure by releasing resources allocated to it, as described above.
On the other hand, if the container is not successful at acquiring resources, the container will wait an amount of time for sufficient resources to become available before attempting to acquire resources again (step 918). For example, during this waiting period, the container may use an exponential “backoff” mechanism, whereby the container repeats its attempts to acquire resources, but doubles the waiting period between repetitions with every failure. If it is determined that the container should continue to try to acquire resources (decision block 920), it will do so as described above in step 908. Otherwise, for example, if failures persist beyond some timeout period, the container abandons attempts to launch and begins to terminate.
Once resources have been released, it is determined whether the container should remain in a standby state (decision block 922), in which case the execution of the container stops, but it remains in a suspended or standby state and retains sufficient state data, for example, by using storage services to retain state data in image form, and for which the container may be required to pay (step 924). Otherwise, the container terminates execution and may be deleted (step 926).
According to various embodiments, the applications being executed by the container are first terminated, and then the container is terminated. Such a graceful termination may be pursued through a recursive termination of the supply chain elements supported by the container. For example, a container element manager may issue a terminate signal to a corresponding operating system manager, which propagates the signal to an application manager, which in turn signals termination to is application. The application may then begin the termination steps as described above with respect to the process 900, after which a termination complete signal to the application manager, and is forwarded to the operating system manager, which in turn sends a terminate signal and receives a termination complete signal back from the operating system. Finally, the operating system's termination complete signal may be forwarded to the container manage, which can signal the container to terminate. It will be understood that terminating (or even suspending) a container operations may result in damages if conducted improperly or at an inappropriate time. Thus, according to various embodiments, a notification procedure may be invoked to notify administrators of pending terminations or suspensions, such that termination or suspension may only be completed once administrator permission has been received.
For a container in standby state, it is determined whether termination should follow (such as by receipt of a terminate signal) (decision block 928), in which case the container terminates execution as described above. Otherwise, for example, if it is determined that the container should re-activate, the container seeks to obtain a budget to acquire resources for its operations as described above, for example, upon receiving an initialize signal. It will be understood that the specific actions described above in connection with process 900 may be modified for non-container system elements, and that the order of steps in process 900 are also illustrative only.
According to various embodiments, a process such as process 900 described above may be used to control container sprawling by suspending or terminating non-productive system elements, such as containers. For example, consider the ROI of a container, which measures the relationship between the payments it collects from applications and the prices it pays for underlying server and I/O resources. If the container's ROI is greater than one, the container is earning more than it expends, and the container may be classified as being productive in creating applications value that exceeds the costs of the infrastructures it uses. However, if the container's ROI is less than one, this means that the container produces less value than the cost of resources it consumes, and the container may thus be classified as non-productive. In this manner, ROI is one example of a metric of productivity that may be used in determining whether a system element should be suspended or terminated, or whether it should remain active.
A process such as process 900 described above may be used to assure, for example, that applications' budgets are sufficient to keep one or more containers' ROI greater than one, and to notify applications' administrators (element managers) as needed when budgets are low. It the ROI of one or more containers remains less than one for more than a threshold period, for example, it may indicate that an application's budget is too low to sustain productive operation, and that the corresponding, non-productive container should be suspended or terminated. For example, a container may receive a terminate signal to switch it OFF to an idle or standby state (per step 916 of process 900 described above) as soon as the container's productivity level or score (for example, measured by its ROI) has been determined to be less than one for a predetermined time period. Additionally, for example, the length of time that the container's ROI has been less than one may be a factor in deciding whether the container should be terminated, or only suspended for the time being.
Similarly to dealing with the sprawling issue, the process 900 described above and similar processes may also be used for energy management. For example, such processes may be used to suspend or terminate (switch OFF) servers that are classified as being non-productive, as in the case where a server's ROI is less than one for a sufficiently long period of time. In this case, the server element manager, much like the case of the container manager described above, can monitor the ROI and detect termination or suspension conditions. The server manager may then pursue a termination process, similar to the recursive termination process described above, where all containers on the server are first terminated, or moved to another server, before the server manager suspends the server into Standby state (so as to consume less energy and cooling resources, for example) or switches the server OFF.
According to various embodiments, process 900 and similar processes may also be used to assure consistency of a suspended container with changes in applications. For example, the container manager may prevent such inconsistencies by sending a terminate signal, as described above, to all containers whenever their respective operating system or applications software has changed, thus causing the applicable containers to transition from standby to terminate state, at which point it may be deleted.
Although the above descriptions consider a single-domain container environment, it will be understood that the principles described herein may also be applied to multi-domain environments, e.g., a multi-cloud environment. For example,
As shown, the software system 1002 operating in the first domain includes a user interface subsystem 1006 and one or more functional managers 1008 and 1010. Together, these elements make up a functional management layer 1012 of software system 1002, and provide specific management applications as described above in connection with
Software system 1002 also includes one or more element managers 1014 and 1016, which monitor and control one or more respective container stack elements 1018 and 1020. The software system 1002 also includes one or more databases 1022 (such as the supply chain databases 246 and operations databases 248 described with reference to
The software system 1004 operates in the second domain, includes similar elements as the software system 1002, and also includes a proxy manager 1030. According to various embodiments, the domain software system 1004 exports one or more resources or services to the domain software system 1002 by using the proxy manager 1030. The proxy manager 1030 exports instrumentation to monitor and control these provided resources to one or more of the element managers 1014 and 1016, such as container element managers, of the first domain software system 1002. The first domain software system 1002 may view the second domain software system 1004 as a service element integral with its supply chain model.
According to various embodiments, the second domain software system 1004 is in complete control of the resources (or services) and capabilities exported to the first domain software system 1002. For example, the software system 1004 may be an external cloud provider exporting raw server services to the software system 1002. In this case, the software system 1002 can access these services, using its local element managers 1014 and 1016, to allocate, for example, CPU, memory, and storage resources at the second domain software system 1004 and then monitor and control their use and operations.
Moreover, according to various embodiments, software systems 1002 and 1004 are separately owned and/or managed. For example, software system 1002 may be owned and operated by a small business that experiences steady computing needs except for two hours in each day, during which time its computing needs are consistently elevated. In this case, rather than purchasing permanent computing resources to handle the two hours of elevated needs per day, for example, software system 1002 may lease or purchase additional computing resources from software system 1004 (e.g., owned by Amazon.com, Inc.) on an as-needed basis and transfer excess workloads to software system 1004 (“bursting”). For example, computing resources from software system 1004 may be leased or purchased to facilitate the execution of a multi-tier web service by a cluster of containers (or applications). In that example, the software system 1002 may lease or sell resources from software system 1004 to execute this cluster of containers (or applications) and then migrate the container cluster (or application cluster). For example, the migration may take place from a private cloud of a small business to the public cloud of another business (e.g., of Amazon, Inc.). It is noted that, according to various embodiments, even if needed computing resources are available from within software system 1002, such resources may be purchased from software system 1004 based on relative price offerings.
The asymmetric relationship between software systems 1002 and 1004 shown in
The supply-chain principles discussed herein may be used to scale containers up/down, by adding or removing resources to existing container components, or to scale containers out, by adding more container components or suspending containers. The decisions to scale up/down and out are based on a supply chain 1100 outlined in
Turning to
In some embodiments, the market may suggest an increase of (scaling up) the Memory Allocation 1101 of the Container 1120, or it may suggest the creation of another instance (scaling out) of the Container 1120. According to various embodiments, decisions to scale up/down will apply to Resources only, while decisions to scale out will apply to SEs only.
For example, in
With reference now to a supply chain 1200 shown in
Container MemAllocation size may increase as a result of over provisioned resources at the Virtual Machine level. Decreased utilization of Virtual Machine CPUAllocation due to a high capacity will lead to decreased CPUAllocation price, which in turn decreases expenses for CPUAllocation on the Container 1120. If the Container 1120 has high revenues for CPUAllocation this would lead to a decision to increase the capacity of CPUAllocation on the Container 1120.
Decisions for both resources and SEs are based on revenues and expenses of these resources. Similarly, expenses and revenues can be set to a predetermined value as desired. For example, the price of MemAllocation can be set to a minimum value to force higher expenses if attempting to maintain the size of the MemAllocation of the Container at or below some value. This advantageously avoids unnecessary resizing only for the purpose of having additional MemAllocation. Accordingly to other embodiments, the price of MemAllocation can be set to a maximum value.
Advantageously, the values of X and Y provide a mechanism to tune the responsiveness of the system to increases or decreases in demand. The value of revenues/expenses captures the profitability of the resource allocation (or the SE). If the ratio is >1, the resource is profitable. If it is <1, it is losing money. In process 1300, X is typically (but not necessarily) >=1 and Y is typically (but not necessarily) <1. Stated in another way, an increase in capacity typically is suggested when the resource is profitable, and a decrease when it is operating at a loss.
As an additional advantage, decisions capture the entire state of the system, and can optimize the system as a whole. Increased utilization of a resource will lead to increased price for the resource, which in turn increases expenses for the resource. In some embodiments, the ideal price for scaling the resources provides 70% utilization.
In some embodiments, revenues and expenses can refer to the accumulated revenues and expenses over a period of time. Different periods of time can be used to adjust the decision-making behavior (e.g., aggressive versus conservative behavior). Short time frames lead to aggressive decisions, where the system responds very quickly to changes in the supply and demand anywhere along the supply chain. This can be used, for example, to respond quickly to congestion for resources and guarantee the quality of service offered to the entities in the system. Long time frames dampen the effects of short-term changes in the supply and demand, and reflect accurately the longer-term trends of the demand and supply.
A similar decision tree to the one shown in
As discussed above, in addition to managing container resources, the supply-chain principles discussed herein also may be used to manage application performance in other virtualization systems. For example, an application server requires a certain amount of memory and CPU resources. A database will also require a certain amount of storage. In order for the application to perform adequately, the application must be allocated a sufficient amount of resource. In order for the infrastructure to be utilized efficiently, the application should only consume what it requires at any given point in time.
Accordingly, with respect to application performance, the supply-chain principles discussed in
Turning to
As discussed above, the resources and SEs have expenses and revenues. For example, the revenues of a virtual central processing unit (VCPU) 1504 sold by the Virtual Machine 1120 are generated from the Application Server 1530 buying this resource. Expenses of the VCPU 1504 come from paying to acquire a necessary resource, such as CPU 1505, from the underlying Physical Machine 1210 hosting the Virtual Machine 1120.
Similarly, a SE has revenues which can be the sum of the revenues of the resources it sells, while its expenses can be the sum of the expenses of the resources it buys. As another example, the revenues of the Virtual Machine 1120 can be the sum of the revenues of the VCPU 1504 and the VMem 1502 that it sells to the Application Server 1530 in
Revenues and expenses can depend on the prices of resources, which in turn can be a function of supply, e.g., attributes of the resource such as its capacity, as well as the demand—how much of the capacity is currently utilized by resources or SEs consuming this resource. In one embodiment, price is a function of the utilization (U) of the resource, and depends on it through the formula:
For example, an application server requires java heap in order to process transactions. This java heap is allocated from the underlying virtual machine's virtual memory allocation. In the event that the demand for java heap is very high (e.g., generating revenue for the application server), and the price of virtual memory from the virtual server (e.g., determined by the combination of supply and demand) is sufficiently low, then the application server will be able to buy more virtual memory from the virtual server and allocate additional java heap. In the event that the demand for java heap is low and the price of virtual memory is high then the application server will decrease its allocation of java heap and return virtual memory to the virtual machine to be used by other applications.
In some embodiments, the buyer can be assigned a budget for purchasing the resources.
Decisions for both resources and SEs are based on the revenues and expenses of these resources. Similarly, expenses and revenues can be set to a predetermined value as desired. For example, the price of VMem can be set to a minimum value to force higher expenses if attempting to maintain the size of the Heap at or below some value. This advantageously avoids unnecessary resizing only for the purpose of having additional VMem. Accordingly to other embodiments, the price of VMem can be set to a maximum value.
In some embodiments, the market may suggest to increase (scale up) the Heap size of an Application Server, or it may suggest to create another instance (scale out) of the Application Server. These decisions can be based on the process 1300 for resizing resources and process 1400 for scaling SEs as discussed above.
As discussed above, revenues and expenses can refer to the accumulated revenues and expenses over a period of time and different periods of time can be used to adjust the decision-making behavior (e.g., aggressive versus conservative behavior). For example, longer periods of time can be used to anticipate future needs for extra application servers based on steadily increasing revenues that reflect an increase in demand. Conversely, a longer term decrease in revenues indicates that the steady state operation of a system may not require a particular SE.
The use of supply chain economic principles and other principles explained above serve several purposes and provide several potential benefits, both expressly numerated and otherwise. For example, these principles can be used to provide a common software framework and abstractions to unify and automate the management of container systems. More specifically, they can be used to optimize or improve the allocation of IT resources (such as I/O resources or software licenses) to best process applications workloads according to their business value. The principles of supply chain economics can also be used to balance workloads to minimize disruptive operating conditions, such as I/O congestion, and to reduce resource waste by terminating or switching-off underutilized resources. These principles can also be used to empower business units to monitor and control the delivery of SLAs to their applications, as well as the ROI of individual elements and the overall container system. In addition, for example, these principles can be used to handle the management of virtual resources in a multi-cloud (or multi-domain) system.
Additionally and/or alternatively, the management of sources in container systems and conventional virtualization systems can include not only supply-chain based methods, but also access regulation to the resources.
In some embodiments, there is one type of permit per provider 1630. According to various embodiments, the AM 1620 can sell multiple types of action permits, regulating access to a number of resources. Each permit can be associated with a predetermined price. Additionally and alternatively, this price can be dynamically adjusted taking into consideration the availability of permits possessed by the AM 1620.
Permits sold by the AM 1620 can create both revenues and expenses for the AM 1620. The revenues come from the price the consumer 1610 has to pay to the AM 1620 to buy the permit. The expenses come from the price the AM 1620 has to pay to the resource provider 1630 for the right to sell these permits. For example, the AM 1620 may need to pay for Input/output Operations Per Second (IOPS) offered by a storage controller in order to allow access to the consumer 1610.
In some embodiments, the price that the AM 1620 pays for the right to sell these permits is determined by the provider 1630 based on one or more of the following parameters: the capacity and the percentage the provider 1630 wishes to make available to the consumers 1610; the current load of the provider 1630; and the rate at which the provider 1630 wishes its resources to be accessed.
The AM 1620 dynamically can adjust the number of permits it possesses at any time, depending on its revenues and its expenses. For example, if the AM 1620 is profitable (e.g., the charges based on price it is selling the permits to the consumer 1610 is higher than the charges based on price it pays to the provider 1630 for the right to sell these permits), the AM 1620 can consider increasing the number of permits it sells. Alternatively, if the AM 1620 is losing money, the AM 1620 can consider decreasing the number of permits it is selling.
Advantageously, the AM 1620 can be used to avoid I/O congestion in storage controllers when several VMs request to execute heavy-storage applications (e.g., VM Reboots, Antivirus database updates, OS Updates, and so on) at the same time. In one embodiment, the AM 1620 limits the number of concurrent consumers that can access the provider 1630. It may limit access across types of applications or within each type of application. For example, permits can be priced and provided for all anti-virus, OS updates, etc. separately, or all of them may be constrained by the same permits. In this example, the provider 1630 is the storage controller, while the consumer 1610 is the application performing the heavy-storage task. For instance, the application can be performing an anti-virus update on the virtual machine.
Turning to
In an alternative embodiment, the number of concurrent accesses to a resource may vary. For example, the AM 1620 adjusts the number of permits it is selling, to reflect the ability of the provider 1630 to satisfy concurrent requests by consumers 1610. For example, when the AM 1620 pays the provider 1630 for the use of the permit, the AM 1620 adjusts the number of the permits it sells based on how profitable it is. If demand for permits for a specific provider 1630 is high, the AM 1620 raises the prices for this permit, advantageously increasing revenues.
To become even more profitable, the AM 1620 can request the right to sell more permits from the provider 1630. If the provider 1630 agrees, the provider 1630 raises the price the AM 1620 has to pay for these rights. As the demand increases, the provider 1630 continues to increase the price it charges the AM 1620. At a threshold price, the AM 1620 can no longer make a profit, and the AM 1620 does not request any further increase in the number of rights it can sell. Similarly, the number of permits sold by the AM 1620 can decrease as a result of reduced demand by consumers 1610, or increased prices by the provider 1630.
In yet another embodiment, the AM 1620 controls rate of concurrent accesses to a particular resource. For example, the AM 1620 limits the rate at which the applications are accessing the storage controller to perform the heavy-storage tasks. In this case, once the application releases the permit, and until the predetermined period of time has elapsed, the AM 1620 cannot resell this permit. The storage controller can charge the AM 1620 a very small amount for the right to sell a first predetermined number of permits within a period of time, and then increase the price to infinity for permits beyond the first predetermined number in this period.
In yet another embodiment, the consumer request to access one or more permits is made directly to the resource or service provider.
In yet another embodiment, the AM 1620 controls the total number and/or the rate at which a group of consumers accesses a group of resources.
Another aspect discussed above formulates and evaluates the option to move the consumer to a new provider. “Formulating” includes the attributes taken into account when considering the option to move to the new provider. The cost of moving can be part of the comparison between two different alternatives (e.g., keeping a VM in an existing infrastructure or moving the VM to an external cloud provider). Cost can be expressed in actual currency or any unit suitable for the comparison. For example, moving time can be expressed in a real value that quantifies the cost of the VM downtime. In contrast, if there is a strict limit on acceptable downtime, the cost of moving the VM can be expressed in terms of time.
“Evaluating” includes making the decision (e.g., initiating an action based on the decision) and determining the right time to take the action. Compared to other economics-based decision-making systems, one embodiment described herein postpones the decision for the future, advantageously waiting for a sufficient amount of time until the decision-maker is convinced that the decision is the right one.
For example, a virtualization system is considering taking an action A with the cost of taking this action represented as C(A). If the action is taken, the savings over time is S(t). The decision to take the action at the time to when the savings would have exceeded the cost of the action is represented by the following Equation:
S(tA)>=C(A)
In one embodiment, with reference to
For example, the VM 1810 is evaluating a move to one or more service providers 1820, such as storage providers SP1, SP2, . . . SPN. Although storage providers 1820 are used herein as an example, it will be understood that the concepts disclosed herein can be applied to other types of service or resource providers.
In some embodiments, the cost C(Ai) of moving to provider i is set to a value that is proportional to the size of the data to be moved from the current SP to SPi, multiplied by a factor Pi that captures the ‘proximity’ of the current SP to SPi. For example, if the current and the future SPs are in the same datacenter 1830, Pi could be set to 1, whereas if they are in different datacenters 1830, it could be set to 10, to capture that it is more expensive to move across datacenters 1830 as opposed to moving within the same datacenter 1830.
The consumer periodically checks the prices at the current and each provider i, calculates the saving for this period and adds them to the savings from the previous periods. The price of the new provider for the current period may be higher than that of the current provider, and as a result the savings for this period will be negative and will decrease the total savings from previous rounds. The moment the savings up to now exceed the cost C(Ai) the VM 1810 decides to move SPi.
In an alternative embodiment, when the consumer considers moving to a new provider, the current provider gives the consumer some credit (e.g., equal to C(A)) to convince the consumer to stay. The consumer accepts the credit, and periodically checks the price of the new provider. If the price is cheaper, the consumer can use this credit to subsidize any loss of not having moved there. If it is more expensive, the consumer adds her gain to the credit. If the consumer runs out of credit, then the consumer can decide to move.
Advantageously, the system accounts for the fact that a decision that looks good now may not be good in the future. For example, a consumer that buys bandwidth from a network provider may see a cheaper price offered right now by a new provider. However, the new provider may change the price an hour later, and this new price may be higher than the price of the current provider an hour later.
Additionally, the system accounts for the actual behavior of other users. Assume a VM is interested in the latency of accessing data stored on a disk, and a decision is made to move its data from the current to a new disk that currently has lower latency. For large amounts of data, the move could take hours to complete. While the move takes place, other consumers who also see a slightly reduced latency move to the same new provider—effectively increasing the latency for everyone, and making it a bad decision.
Furthermore, the amount of time it takes to determine that the decision may be good is related to the cost of performing the action. Therefore, expensive decisions are carefully validated over longer periods than cheaper decisions, ensuring that undertaking the cost of the action will pay off in the future.
Advantageously, the systems and methods above minimize bad decisions and decisions that would frequently alternate between the current and the new provider.
Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a tangible program carrier for execution by, or to control the operation of, data processing apparatus. The tangible program carrier can be computer readable medium, such as a machine-readable storage device, a machine-readable storage substrate, a memory device, or a combination of one or more of them.
The terms “data processing apparatus” “data processor”, or “processing device” encompasses all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.
A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.
The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).
Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, to name just a few.
Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.
To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.
Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described is this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a LAN and a wide area network (“WAN”), e.g., the Internet.
While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.
Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.
Particular embodiments of the subject matter described in this specification have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.
Dailianas, Apostolos, Kliger, Shmuel, Crouchman, Charles, Borodiansky, Michael
Patent | Priority | Assignee | Title |
11403131, | Feb 05 2020 | International Business Machines Corporation | Data analysis for predictive scaling of container(s) based on prior user transaction(s) |
11411825, | Aug 25 2015 | VMware, Inc. | In intelligent autoscale of services |
11740921, | Nov 23 2020 | GOOGLE LLC | Coordinated container scheduling for improved resource allocation in virtual computing environment |
Patent | Priority | Assignee | Title |
6055571, | Nov 20 1997 | NEC Corporation | Computer network with microeconomic flow control |
6480861, | Feb 26 1999 | Bank of America Corporation | Distributed adaptive computing |
6735553, | Jul 13 2000 | NetPredict, Inc. | Use of model calibration to achieve high accuracy in analysis of computer networks |
7013296, | Jun 08 1999 | The Trustees of Columbia University in the City of New York | Using electronic security value units to control access to a resource |
7140039, | Jun 08 1999 | TRUSTEES OF COLUMBIA UNIVERISTY IN THE CITY OF NEW YORK, THE | Identification of an attacker in an electronic system |
7272855, | Jun 08 1999 | TRUSTEES OF COLUMBIA UNIVERSITY IN THE CITY OF NEW YORK, THE | Unified monitoring and detection of intrusion attacks in an electronic system |
7441023, | Sep 25 2003 | EMC IP HOLDING COMPANY LLC | Method and apparatus for modeling and analyzing MPLS and virtual private networks |
7464132, | Jun 12 2002 | EMC IP HOLDING COMPANY LLC | Method and apparatus for reference model change generation in managed systems |
7496655, | May 01 2002 | TECH MAHINDRA LTD | System and method for static and dynamic load analyses of communication network |
7590601, | Mar 17 2006 | GAMIGO INC | Licensing media consumption using digital currency |
7685270, | Mar 31 2005 | Amazon Technologies, Inc | Method and apparatus for measuring latency in web services |
7711789, | Dec 07 2007 | Intellectual Ventures II LLC | Quality of service in virtual computing environments |
7774457, | Mar 25 2005 | VALTRUS INNOVATIONS LIMITED | Resource evaluation for a batch job and an interactive session concurrently executed in a grid computing environment |
7840517, | Dec 21 2006 | Hitachi, LTD | Performance evaluating apparatus, method, and computer-readable medium |
7877754, | Aug 21 2003 | International Business Machines Corporation | Methods, systems, and media to expand resources available to a logical partition |
8051017, | Jan 17 2008 | International Business Machines Corporation | Practical model for high speed file delivery services supporting guaranteed delivery times and differentiated service levels |
8060875, | May 26 2006 | VMware, Inc.; VMWARE, INC | System and method for multiple virtual teams |
8181175, | Jan 28 2008 | THE CONUNDRUM IP LLC | Accounting for resource usage time by a virtual machine |
8271345, | Dec 22 2008 | AUCTIONOMICS INC | Systems and method for incorporating bidder budgets in multi-item auctions |
8286174, | Apr 17 2006 | VMware, Inc.; VMWARE, INC | Executing a multicomponent software application on a virtualized computer platform |
8347302, | Oct 09 2008 | Amazon Technologies, Inc | System-aware resource scheduling |
8370898, | Jun 18 2004 | III Holdings 12, LLC | System and method for providing threshold-based access to compute resources |
8396807, | Jun 26 2009 | International Business Machines Corporation | Managing resources in virtualization systems |
8429097, | Aug 12 2009 | Amazon Technologies, Inc. | Resource isolation using reinforcement learning and domain-specific constraints |
8433801, | Jun 26 2009 | International Business Machines Corporation | Managing resources in virtualization systems |
8478878, | Mar 11 2010 | International Business Machines Corporation | Placement of virtual machines based on server cost and network cost |
8650296, | Oct 31 2006 | VALTRUS INNOVATIONS LIMITED | Workload reallocation involving inter-server transfer of software license rights and intra-server transfer of hardware resources |
8661131, | Jun 26 2009 | International Business Machines Corporation | Managing resources in virtualization systems |
8762531, | Jun 26 2009 | International Business Machines Corporation | Managing resources in virtualization systems |
8914511, | Jun 26 2009 | International Business Machines Corporation | Managing resources in virtualization systems |
8959249, | Sep 15 2010 | EMC IP HOLDING COMPANY LLC | Cooperative cloud I/O scheduler |
9589495, | May 28 2013 | Innolux Corporation | Liquid crystal display and display method thereof |
9589501, | Dec 15 2006 | Lapis Semiconductor Co., Ltd. | Display drive circuit including an output terminal |
20020147611, | |||
20030154123, | |||
20050005271, | |||
20050132363, | |||
20050256683, | |||
20060020939, | |||
20060045100, | |||
20060090163, | |||
20060143617, | |||
20060167984, | |||
20060168584, | |||
20060184937, | |||
20060188011, | |||
20060190482, | |||
20060212332, | |||
20060244607, | |||
20060265470, | |||
20070105630, | |||
20080052387, | |||
20080080552, | |||
20080083031, | |||
20080104608, | |||
20080109241, | |||
20080127348, | |||
20080154837, | |||
20080155169, | |||
20080163194, | |||
20080201409, | |||
20080244607, | |||
20080250143, | |||
20080301027, | |||
20080319926, | |||
20080320474, | |||
20090007126, | |||
20090052333, | |||
20090089406, | |||
20090164356, | |||
20090172666, | |||
20090182565, | |||
20090240628, | |||
20090249488, | |||
20090276271, | |||
20090276771, | |||
20100005173, | |||
20100057625, | |||
20100077449, | |||
20100088205, | |||
20100106820, | |||
20100142522, | |||
20100153945, | |||
20100180275, | |||
20100318454, | |||
20100319004, | |||
20110131335, | |||
20110225277, | |||
20110276951, | |||
20120072762, | |||
20120131591, | |||
20120226796, | |||
20120317578, | |||
20140081766, | |||
20140376368, | |||
20140380324, | |||
WO2009091580, | |||
WO2010036731, |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Nov 11 2015 | DAILIANAS, APOSTOLOS | VMTURBO, INC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 056368 | /0653 | |
Nov 12 2015 | CROUCHMAN, CHARLES | VMTURBO, INC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 056368 | /0653 | |
Nov 12 2015 | BORODIANSKY, MICHAEL | VMTURBO, INC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 056368 | /0653 | |
Nov 12 2015 | KLIGER, SHMUEL | VMTURBO, INC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 056368 | /0653 | |
Aug 15 2016 | VMTURBO, INC | TURBONOMIC, INC | CHANGE OF NAME SEE DOCUMENT FOR DETAILS | 056408 | /0116 | |
Aug 15 2016 | VMTURBO, INC | TURBONOMIC, INC | CORRECTIVE ASSIGNMENT TO CORRECT THE APPLICATION NUMBER PREVIOUSLY RECORDED AT REEL: 056408 FRAME: 0116 ASSIGNOR S HEREBY CONFIRMS THE CHANGE OF NAME | 057603 | /0146 | |
Jul 28 2020 | TURBONOMIC, INC. | (assignment on the face of the patent) | / | |||
Dec 21 2022 | TURBONOMIC, INC | International Business Machines Corporation | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 062202 | /0030 |
Date | Maintenance Fee Events |
Jul 28 2020 | BIG: Entity status set to Undiscounted (note the period is included in the code). |
Jan 14 2021 | PTGR: Petition Related to Maintenance Fees Granted. |
Date | Maintenance Schedule |
Aug 31 2024 | 4 years fee payment window open |
Mar 03 2025 | 6 months grace period start (w surcharge) |
Aug 31 2025 | patent expiry (for year 4) |
Aug 31 2027 | 2 years to revive unintentionally abandoned end. (for year 4) |
Aug 31 2028 | 8 years fee payment window open |
Mar 03 2029 | 6 months grace period start (w surcharge) |
Aug 31 2029 | patent expiry (for year 8) |
Aug 31 2031 | 2 years to revive unintentionally abandoned end. (for year 8) |
Aug 31 2032 | 12 years fee payment window open |
Mar 03 2033 | 6 months grace period start (w surcharge) |
Aug 31 2033 | patent expiry (for year 12) |
Aug 31 2035 | 2 years to revive unintentionally abandoned end. (for year 12) |