A computing device of a data zone includes a persistent storage and a processor. The persistent storage includes a locked data batch of the data zone. The processor obtains an upstream computation request; instantiates a computation framework to process the locked data batch based on a global data batch specified in the upstream computation request; instantiates a downstream computation manager to manage a downstream computation; and instantiates, by the downstream computation manager, a second computation framework in a second computing device of a second data zone to process a second locked data batch of the second data zone.
|
1. A computing device of a first data zone, comprising:
a persistent storage comprising a first locked data batch of the first data zone; and
a processor programmed to:
obtain an upstream computation request;
instantiate a first computation framework to process the first locked data batch based on a global data batch specified in the upstream computation request;
instantiate a downstream computation manager to manage a downstream computation;
instantiate, by the downstream computation manager, a second computation framework in a second computing device of a second data zone to process a second locked data batch of the second data zone;
obtain a first computation result from the second computation framework; and
generate a second computation result using:
the first computation framework,
the first locked data batch, and
the first computation result,
wherein the first locked data batch and the first computation result match a same grouping criteria.
18. A method of operating a computing device of a first data zone, comprising:
obtaining, by the computing device, an upstream computation request;
instantiating, by the computing device, a first computation framework to process a first locked data batch based on a global data batch specified in the upstream computation request, wherein the first locked data batch is stored on a persistent storage of the computing device;
instantiating, by the computing device, a downstream computation manager on the computing device to manage a downstream computation;
instantiating, by the downstream computation manager of the computing device, a second computation framework in a second computing device of a second data zone to process a second locked data batch of the second data zone;
obtaining a first computation result from the second computation framework; and
generating a second computation result using:
the first computation framework,
the first locked data batch, and
the first computation result,
wherein the first locked data batch and the first computation result match a same grouping criteria.
19. A non-transitory computer readable medium comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a method for operating a computing device of a first data zone, the method comprising:
obtaining, by the computing device, an upstream computation request;
instantiating, by the computing device, a first computation framework to process a first locked data batch based on a global data batch specified in the upstream computation request, wherein the first locked data batch is stored on a persistent storage of the computing device;
instantiating, by the computing device, a downstream computation manager on the computing device to manage a downstream computation;
instantiating, by the downstream computation manager of the computing device, a second computation framework in a second computing device of a second data zone to process a second locked data batch of the second data zone;
obtaining a first computation result from the second computation framework; and
generating a second computation result using:
the first computation framework,
the first locked data batch, and
the first computation result,
wherein the first locked data batch and the first computation result match a same grouping criteria.
2. The computing device of
3. The computing device of
4. The computing device of
5. The computing device of
a local computation that generates a first computation result stored on the persistent storage,
wherein the local computation is based on the global data batch of the upstream computation request.
6. The computing device of
the first locked data batch, and
the second locked data batch.
7. The computing device of
8. The computing device of
before instantiating the second computation framework:
determine, by the downstream computation manager, a first computing resource cost of processing the second locked data batch using the second computing device;
determine, by the downstream computation manager, a second computing resource cost of processing the second locked data batch using a third computing device of a third data zone; and
select, by the downstream computation manager, to instantiate the second computing framework in the second computing device based on the first computing resource cost and the second computing resource cost.
9. The computing device of
generating a downstream computation request; and
sending the generated downstream computation request to the second computing device.
10. The computing device of
11. The computing device of
12. The computing device of
13. The computing device of
determining a format of the first locked data batch;
determining a computational cost of each of a plurality of computation types for the format of the first locked data batch;
determining a computation type of the plurality of computation types that has a relatively low computation cost of the computational costs of each of the plurality of computation types; and
instantiating a local computation of the determined computation type using the locked data batch.
14. The computing device of
15. The computing device of
16. The computing device of
17. The computing device of
generating a third computation result based on the first locked data batch;
generating the second computation result using the third computation result and the first computation result; and
marking the second computation result as being contemporaneous to the first computation result.
20. The non-transitory computer readable medium of
|
|||||||||||||||||||||||||||||
The present application is a continuation-in-part of U.S. patent application Ser. No. 14/982,341, filed Dec. 29, 2015 and entitled “Multi-Cluster Distributed Data Processing Platform,” now U.S. Pat. No. 10,015,106, which is incorporated by reference herein in its entirety, and which claims priority to U.S. Provisional Patent Application Ser. No. 62/143,404, entitled “World Wide Hadoop Platform,” and U.S. Provisional Patent Application Ser. No. 62/143,685, entitled “Bioinformatics,” both filed Apr. 6, 2015, and incorporated by reference herein in their entirety.
Computing devices may generate new data based on stored data. For example, a computing device may store a database that includes sales data for a variety of products over a period of time. The computing device may generate new data by calculating an average sale price of each sale.
In some cases, a database or other type of data source may be distributed across a number of computing devices. For example, a first portion of a database that stores sales at a first store location may be stored on a local storage of a first computing device and a second portion of the database that stores sales at a second store location may be stored on a local storage of a second computing device. To generate new data, the second portion of the database may be sent to the first computing device and stored on the local storage of the first computing device. The first computing device may, calculate the average sale price of each sale across the database using the first portion and second portion of the database stored on the local storage.
In one aspect, a computing device of a data zone in accordance with one or more embodiments of the invention includes a persistent storage and a processor. The persistent storage includes a locked data batch of the data zone. The processor obtains an upstream computation request; instantiates a computation framework to process the locked data batch based on a global data batch specified in the upstream computation request; instantiates a downstream computation manager to manage a downstream computation; and instantiates, by the downstream computation manager, a second computation framework in a second computing device of a second data zone to process a second locked data batch of the second data zone.
In one aspect, a method of operating a computing device of a data zone in accordance with one or more embodiments of the invention includes obtaining, by the computing device, an upstream computation request; instantiating, by the computing device, a computation framework to process a locked data batch based on a global data batch specified in the upstream computation request, wherein the locked data batch is stored on a persistent storage of the computing device; instantiating, by the computing device, a downstream computation manager on the computing device to manage a downstream computation; and instantiating, by the downstream computation manager of the computing device, a second computation framework in a second computing device of a second data zone to process a second locked data batch of the second data zone.
In one aspect, a non-transitory computer readable medium in accordance with one or more embodiments of the invention includes computer readable program code, which when executed by a computer processor enables the computer processor to perform a method for operating a computing device of a data zone, the method includes obtaining, by the computing device, an upstream computation request; instantiating, by the computing device, a computation framework to process a locked data batch based on a global data batch specified in the upstream computation request, wherein the locked data batch is stored on a persistent storage of the computing device; instantiating, by the computing device, a downstream computation manager on the computing device to manage a downstream computation; and instantiating, by the downstream computation manager of the computing device, a second computation framework in a second computing device of a second data zone to process a second locked data batch of the second data zone.
Certain embodiments of the invention will be described with reference to the accompanying drawings. However, the accompanying drawings illustrate only certain aspects or implementations of the invention by way of example and are not meant to limit the scope of the claims.
Specific embodiments will now be described with reference to the accompanying figures. In the following description, numerous details are set forth as examples of the invention. It will be understood by those skilled in the art that one or more embodiments of the present invention may be practiced without these specific details and that numerous variations or modifications may be possible without departing from the scope of the invention. Certain details known to those of ordinary skill in the art are omitted to avoid obscuring the description.
In the following description of the figures, any component described with regard to a figure, in various embodiments of the invention, may be equivalent to one or more like-named components described with regard to any other figure. For brevity, descriptions of these components will not be repeated with regard to each figure. Thus, each and every embodiment of the components of each figure is incorporated by reference and assumed to be optionally present within every other figure having one or more like-named components. Additionally, in accordance with various embodiments of the invention, any description of the components of a figure is to be interpreted as an optional embodiment, which may be implemented in addition to, in conjunction with, or in place of the embodiments described with regard to a corresponding like-named component in any other figure.
In general, embodiments of the invention relate to systems, devices, and methods for performing computations. More specifically, the systems, devices, and methods may enable computations to be performed across locked data batches distributed across any number of data zones. In one or more embodiments of the invention, the location and computation type performed in each data zone may be dynamically selected to reduce computing resource usage. Dynamically selecting the location and/or computation type performed in each data zone may enable computations to be formed without a centralized controller that orchestrates all computations across the data zones. Rather, requests sent to any data zone trigger a data zone wide computation to be performed.
As used herein, a data zone is any collection of computing and/or storage devices that are logically demarcated from all other computing devices. For example, a data zone may be a cloud computing environment. The cloud computing environment may utilize the computing resources of a number of computing devices. A system in accordance with embodiments of the invention may include multiple data zones.
As used herein, a locked data batch refers to any quantity of data in any format that is logically restricted to a corresponding data zone. For example, a cloud computing environment may host a medical record on a non-transitory storage of a computing device of the cloud computing environment. Access restrictions associated with medical records may lock the data to the cloud computing environment and prevent the medical record from being sent to a computing device of a different cloud computing environment.
In one or more embodiments of the invention, a worldwide computation may be performed by recursively instantiating computations in multiple data zones. For example, instantiating a computation in a first data zone may require, as input, a computation result from a second data zone. To obtain the computation result from the second data zone, a second computation may be instantiated in the second data zone. The second computation may use, as input, a third computation result from a third data zone. Thus, computations may be recursively instantiated across any number of data zones to service the first instantiated computation. The aforementioned process of recursively instantiating computations may enable a worldwide computation to be triggered by instantiating a single computation in a data zone. The aforementioned recursively process may be decentralized and thereby enable any number of computations to be performed. As computations are recursively instantiated, the location of instantiated computations may be dynamically selected to reduce the computing resource of performing the data zone wide computation.
The clients (100) may be computing devices. The computing devices may be, for example, mobile phones, tablet computers, laptop computers, desktop computers, servers, computing clusters, or cloud computing systems. The computing devices may include one or more processors, memory (e.g., random access memory), and persistent storage (e.g., disk drives, solid state drives, etc.). The persistent storage may store computer instructions, e.g., computer code, that when executed by the processor(s) of the computing device cause the computing device to perform the functions of the clients (100) described in this application. The clients (100) may be other types of computing devices without departing from the invention.
In one or more embodiments of the invention, the clients (100) may issue global computation requests to the data zones (110). A global computation request may, request that a computation result for data locked in any number of data zones (110) be provided to the requesting entity. A global computation request may specify: (i) a type of computation to be performed, (ii) the data on which the computation is to be performed, and/or (iii) an identifier of the client so that the computation result may be provided to the request issuing client. The global computation request may specify, different or additional information without departing from the invention. For additional details regarding global computation request, See
In one or more embodiments of the invention, the clients (100) may have access to a data map (not shown) that provides the clients (100) with information regarding the data stored in the data zones and/or the topology of the network of data zones. In one or more embodiments of the invention, the data map specifies: (i) the data stored in the data zones (110), (ii) the data zone of the data zones (110) in which the stored data resides, and (iii) the computing resources of the data zones (110). In one or more embodiments of the invention, the data map may be a data structure that specifies the aforementioned information. The data map may be stored on a non-transitory computer readable storage medium of any of the clients (100) or another computing device operably connected to the clients (100).
In one or more embodiments of the invention, the clients (100) utilize the data map to generate global computation requests. For example, the clients (100) may specify data stored in the data zones (110) on which to perform a calculation in the generated global computation requests using the data map.
As discuss above, the clients (100) may send global computation requests to data zones (110). The data zones (110) may collaboratively perform computations to obtain computation results requested by clients (100). In one or more embodiments of the invention, the data zones (110) may collaborate by using a uniform system for batching data and/or computation results in each data zone. The uniform system for batching may enable data/result having a similar grouping criteria to be identified. The grouping criteria may be, for example, a time stamp of the data/result. The computation results may include results of computations performed by multiple data zones that have matching grouping criteria.
In one or more embodiments of the invention each data zone may be a logical grouping of computing resources that stores data locked to the computing resources. Each of the computing resources of a data zone may be organized to complete computations specified in global computation requests from clients.
In one or more embodiments of the invention, the computing resources of the data zones (110) are computing devices. The computing devices may be, for example, mobile phones, tablet computers, laptop computers, desktop computers, servers, computing clusters, or cloud computing systems. The computing devices may include one or more processors, memory (e.g., random access memory), and persistent storage (e.g., disk drives, solid state drives, etc). The persistent storage may store computer instructions, e.g., computer code, that when executed by the processor(s) of the computing device cause the computing device to perform the functions described in this application and/or the methods illustrated in
In one or more embodiments of the invention, the computing resources of a first data zone are geographically separated from the computing resources of a second data zone. For example, a first data zone may be located in the US and the second data zone may be located in Canada.
In one or more embodiments of the invention, the computing resources of a first data zone are located adjacent to the computing resources of a second data zone. For example, the first and second data zone may include computing resources of a single computing cluster that are logically, rather than physically, separated.
In one or more embodiments of the invention, each data zone may store data that is locked to the data zone. As used herein, data that is locked to a data zone refers to data that may not be transmitted to computing resources that are not part of the logical grouping of computing resources defined by the data zone. Data may be locked to a data zone for any reason. For example, data may be locked to a data zone due to privacy concerns. In another example, data may be locked to a data zone due to the size of the data. In a further example, data may be locked to a data zone due a restriction imposed on the data by an owner of the data. The data may be locked to a data zone due to other restrictions/reasons without departing from the invention.
In one or more embodiments of the invention, the data zones (110) may be organized as a logical network. In other words, each of the data zones may be a node of the logical network. To perform computations, computation requests from clients may be distributed via the logical network. The logical network of data zones may be supported by any arrangement of operable connections.
In one or more embodiments of the invention, each data zone may include a map of the logical network of data zones. The map may specify: (i) the topology of the network, (ii) the computing resources available to each data zone, and (iii) the data stored by each data zone. The map may include more, different, and/or less information without departing from the invention.
In one or more embodiments of the invention, the data zones may send downstream computation requests to other data zones to service computation requests from clients. As used herein, a downstream computation request refers to a request generate by a data zone to service a computation request from a client. As noted above, each data zone may store locked data. A computation request from a client may require performing computations on locked data stored in different data zones. To service a client computation request, the data zones may analyze the client request and send appropriate downstream computation requests to other clients.
To further clarify the data zones (110),
In one or more embodiments of the invention, the data zone (120) is a logical computing device that utilizes the physical computing resources of one or more computing devices to provide the functionality of the data zone (120) described throughout this application and/or to perform the methods illustrated in
In one or more embodiments of the invention, the data zone (120) includes computing resources that provide processing (e.g., computations provided by a processor), memory (e.g., transitory storage provided by RAM), and persistent storage (e.g., non-transitory storage provided by a hard disk drive) by utilizing the physical computing resources of the computing devices of the data zone (120). In one or more embodiments of the invention, the data zone (120) may include instructions stored on a persistent storage of a computing device of the data zone that when executed by a processor of the data zone provides the functionality of the data zone (120) described throughout this application and/or the methods illustrated in
In one or more embodiments of the invention, the computing devices utilized by the data zone (120) are operably connected to each other and/or operably connected to computing devices of other data zones. For example, each of the computing devices of the data zone (120) may include a network interface that enables packets to be sent via a network to other computing devices of the data zone (120) or other data zones.
To provide the aforementioned functionality of the data zone (120), the data zone (120) may include a framework manager (122) that instantiates/manages instances of computation frameworks (124) including 124A, 124N executing using computing resources of the data zone (120), locked data batches (128) stored using computing resources of the data zone (120), computing results stored in cache(s) (130) including 130A, 130N implemented using computing resources of the data zone (120), and metadata (132) including 132A, 132N associated with the computation results (130) or locked batches (128) stored using computing resources of the data zone (120). Each component of the data zone (120) is discussed below.
In one or more embodiments of the invention, the framework manager (122) responds to upstream computation requests. The framework manager (122) may respond to the upstream computation requests by instantiating computing frameworks (124). The computation frameworks (124) may generate computation results (130) specified by the upstream computation requests.
As used herein, an upstream computation request refers to any computation request receiving from another data zone or client. In one or more embodiments of the invention, an upstream computation request is a global computation request sent from a client. In one or more embodiments of the invention, the upstream computation request is a downstream computation request generated by a computing device of another data zone. Thus, embodiments of the invention enable the recursive instantiation of any number of computations in any number of data zones. For example, receipt of an upstream computation request by a data zone may trigger a downstream computation request to be sent to service a computation instantiated in the data zone in response to the upstream computation request. In this manner, a worldwide computation may be initiated without the need for a centralized controller or other system wide orchestrating entity.
As used herein, a downstream computation request refers to a computation request generated by a data zone. The downstream computation requests may be generated by computation frameworks, as will be discussed in greater detail with respect to
As used herein, instantiating a computation framework means to start one or more processes that perform the functionality of a computation framework as will be discussed in greater details with respect to
In one or more embodiments of the invention, the framework manager (122) is implemented as one or more processes executing using computing resources of the data zone (120) based on computer instructions stored on a non-transitory computer readable media. The computing instructions, when executed using processing computing resources of the data zone (120) cause computing device(s) of the data zone (120) to perform the functions of the framework manager (122) and/or all or a portion of the methods illustrated in
In one or more embodiments of the invention, the computation frameworks (124) may service upstream computation requests. The computation frameworks (124) may service the upstream results by generating computation results (130) and/or providing generated computation results (130) to the requesting entity. In one or more embodiments of the invention, the computation results (130) may be stored in a cache of the data zone (120). For additional details regarding the computation frameworks (124), See
The locked data batches (128), computation results (130), and metadata (132) may be data stored using computing resources of the data zone (120). The data zone (120) may store additional, different types, and/or less data without departing from the invention. Each type of the aforementioned data is discussed below.
In one or more embodiments of the invention, each locked data batch of the locked data batches (128) is data stored in the data zone (120) that may not be transmitted to computing device that are not a part of the data zone (120). The data of the locked data batches (128) may be generated via any method without departing from the invention. As discussed above, the locked data batches (128) may not be transmitted to computing devices that are not a part of the data zone (120) for any reason without departing from the invention. For example, a locked data batch may, include private data that is restricted from being transmitted outside of the data zone (120). The locked data batches (128) may be used, in part, by computation frameworks (124) to generate computation results (130).
In one or more embodiments of the invention, the locked data batches (128) have varying formats. For example, a first locked data batch may be in a database format while a second locked data batch may be in a table format. Some of the locked data batches may have the same format without departing from the invention.
In one or more embodiments of the invention, the locked data batches (128) may be dynamic generated, modifies, and/or deleted. In other words, the content of each locked data batch may be changing over time. For example, a locked data batch may include data from a sensor being streamed to a computing device of the data zone.
In one or more embodiments of the invention, each locked data batch may include a batch identifier (not shown). The batch identifier may associate the locked data batch with one or more locked data batches stored in the data zone (120) and/or other data zones. The batch identifier may be, for example, a time stamp, a data source, an identifier of a data zone in which the locked batch is stored, a data format, a data type, a size of the locked batch, or another characteristic. In one or more embodiments of the invention, the batch identifier of a locked data batch may be stored as metadata (132) associated with the locked data batch.
In one or more embodiments of the invention, the computation results (130) may be results of computations performed by the computation frameworks (124). In one or more embodiments of the invention, the computation results (130) may be able to be transmitting to computing devices of other data zones, in contrast to the locked data batches (128) cannot be transmitted to computing devices of other data zones.
In one or more embodiments of the invention, each computation result may include a result identifier (not shown). The result identifier, much like a batch identifier, may associate the computation result with one or more computation results stored in the data zone (120) and/or other data zones. In one or more embodiments of the invention, the result identifier may associate the computation result with one or more locked data batches stored in the data zone (120) and/or other data zones. The result identifier may associate the computation result with any number of locked data batches and/or computation results stored in the data zone (120) and/or other data zones without departing from the invention. The result identifier may be, for example, a time stamp, a data source from which the result was generated, an identifier of a data zone in which the computation result is stored, a data format of the computation result, a data type of the computation result, a size of the computation result, or another characteristic of the computation result. In one or more embodiments of the invention, the result identifier of a computation result may be stored as metadata (132) associated with the computation result.
In one or more embodiments of the invention, the time stamp may specify the time at which the computation result was generated, the time at which the computation result was stored in a cache, of the time at which a computation that generated the computation result was instantiated.
In one or more embodiments of the invention, the metadata (132) may be data that specifies characteristics or other information associated with the locked data batches (128) and/or computation results (130). The metadata (132) may be stored in any format and/or include any type of data without departing from the invention. As will be discussed in greater details with respect to
As discussed above, the computation frameworks (124) may generate computation results (130) using metadata (132) and locked data batches (128).
In one or more embodiments of the invention, the computation manager (152) instantiates: (i) aggregate computation(s) (154), (ii) local computation managers (156), and/or (iii) downstream computation managers (160). The aforementioned computations and/or managers may be instantiated by the computation manager (152) to service an upstream computation request which triggered the instantiation of the computation framework (150). Instantiating the framework (150) may include instantiating the computation manager (152).
In one or more embodiments of the invention, the computation manager (152) may instantiate an aggregate computation based on a requested computation result. As used herein, an aggregate computation is a computation that uses, as input, the results of one or more other computations. For example, two computations may be performed on two locked data batches. The results of the two computations may be used as input to the aggregate computation. The aggregate computation may be any type of computation, use any number/type/quantity of input, and produce any type/quantity of results without departing from the invention. While the aggregate computation has been described as performing a computation, embodiments of the invention are not limited to the performance of computations. The aggregate computation may perform any type of analytical function, processing function, operate on any type of data/result, and produce any type/quantity/of output without departing from the invention.
In one or more embodiments of the invention, the aggregate computation (154) may be selected from several different types of aggregate computations (154) included in a template on which the computation framework (150) is based. The type of aggregate computation (154) may be selected based on: (i) the computation result specified by the request, (ii) the location of the data zone hosting the computation framework (150) within the network of data zones, (iii) the locked data batches of the data zone hosting the computation framework (150) that are implicated by the requested computation, and (iv) the locked data batches of other data zones that are not hosting the computation framework (150) that are implicated by the requested computation. As used herein, an implicated data batch is one on which a computation must be performed to service the computation request. The computation manager (152) may instantiate aggregate computations using additional, different, or fewer factors without departing from the invention.
In one or more embodiments of the invention, the aggregate computation (154) may generate an aggregate computation result using: (i) local computation results generated by the local computations (154) and/or (ii) downstream computation results generated by local and/or aggregate computations performed by data zones in response to downstream computation requests generated by the downstream computation managers (160). In one or more embodiments of the invention, the aggregate computation result may be stored as a computation result after being generated. In one or more embodiments of the invention, the aggregate computation result may be sent to a requesting entity.
In one or more embodiments of the invention, the local computation managers (156) may be instantiated by the computation manager (152). The local computation managers (156) may instantiate local computations (158) including 158A, 158N to generate local computation results used by the aggregate computation to form an aggregate computation result.
In one or more embodiments of the invention, the local computation managers (156) may instantiate local computations (158) based on: (i) a characteristic of a locked data batch result which the local computation will use to generate a result and (ii) the availability of computing resources to perform the local computation. The local computation (158) may be instantiated based on additional, different, and/or fewer factors without departing from the invention, in one or more embodiments of the invention, instantiating a local computation may include selecting a computation type for the local computation.
As used herein, a computation type refers to a method of implementing and processing a data set. The data set may be, for example, a locked data batch. Computation types include a map/reduce computation, a split-apply-combine computation, and a partially parallel computation. The partially parallel computation may be a spark computation. The map/reduce computation may be a parallel computation.
In one or more embodiments of the invention, the characteristic of the locked data batch is a data type of the locked data batch or a data format of the locked data batch. For example, a first computation type may more efficiently generate a local computation result then a second computation type may generate a result for a format of data. In one or more embodiments of the invention, the computation type generated by the local computations (158) may be selected to minimize the computation cost of generating a local computation result among a number of different types of local computations that could be performed.
In one or more embodiments of the invention, the availability of computing resources to perform the local computation is determined by querying a scheduler that performs a schedule of a type of computation to identify when the aforementioned computation may be performed/completed. For example, a data zone may include a scheduler/allocator used to schedule the performance of local computations (158). Different types of computations may have different scheduling availabilities. For example, a first type of computation may be able to be scheduled for execution before a second type of computation is available to be schedule for execution. In one or more embodiments of the invention, the local computations managers (156) may select a computation type that will be instantiated based on the aforementioned scheduling availability.
As noted above, in one or more embodiments of the invention, the local computations (158) may generate computation results using locked data batches. The local computations (158) may generate computation results continuously, periodically, at predetermined point in time, or may be triggered. In one or more embodiments of the invention, the local computations may generate computation results continuously, by generating computation results in response to changes in a locked data batch used by the local computation to generate the result.
In one or more embodiments of the invention, the downstream computation managers (160) may instantiate computation frameworks in other data zones. In one or more embodiments of the invention, the frameworks in other data zones may be instantiated based on: (i) a characteristic of a locked data batch stored in the other data zones and (ii) the location within the network of data zones of the other data zones. The location, i.e., data zone in which the framework is instantiated, may be selected to minimize computing resources used to generate a computation result and/or a time window in which the computation result is to be generated.
In one or more embodiments of the invention, the computation manager (152), aggregate computation (154), local computation managers (156), local computations (158), and downstream computation managers (160) are implemented as computer instructions, e.g., computer code, stored on a non-transitory storage that is executed using processing resources of the data zone (120,
To further clarify aspect of the invention,
The client identifier (202) may be an identifier of the client to which a result of the computation specified by the global computation request (200) is to be returned. In one or more embodiments of the invention, the client identifier (202) is an identifier of the client that generated the global computation request (200). In one or more embodiments of the invention, the client identifier (202) is a media access control address of the client that generated the global computation request (200). The client identifier (202) may be a media access control address of a client that did not generate the global computation request (200) without departing from the invention.
The computation description (204) may be a description of the computation result desired by the requesting entity. For example, the computation description (204) may indicate that an average of a number of values stored in various data zones is being requested by the requesting entity. The computation description (204) may indicate any type of computation without departing from the invention.
The global data batch (206) may indicate data stored in the data zones on which a computation specified by the computation description (204) is to be performed. The global data batch (206) may indicate the stored data at varying levels of granularity and/or in different formats without departing from the invention.
While
As discussed above, metadata may be used by clients and/or data zones to generate global/downstream computation requests. Metadata may include any type of data associated with stored data, e.g., locked data batches/computation results, or data that describes a topology of the network, e.g., topology of the computing devices of data zone(s), computing resources of data zone(s), topology of the network including clients and/or data zones.
The metadata (250) also includes a map (258). In one or more embodiments of the invention, the map (258) indicates the topology of the network of clients/data zones, or a portion thereof. In one or more embodiments of the invention, the topology of the network indicates the computing resources of the clients/data zones, or a portion thereof. In one or more embodiments of the invention, the topology of the network indicates the locked data batches of the data zones, or a portion thereof. The map (258) may include additional, different, or less information without departing from the invention.
As discussed above, the data zones (110,
While illustrated as separate methods, each of the methods illustrated in
In Step 300, a data generation request is obtained.
In one or more embodiments of the invention, the data generation request may be obtained from an application executing on the client. In one or more embodiments of the invention, the data generation request is obtained from a second client. In one or more embodiments of the invention, the data generation request is obtained from a data zone.
In Step 302, a global computation request is generated based on the obtained data generation request.
In one or more embodiments of the invention, the generated global computation request specifies the requesting entity, a computation to be performed, and the data on which the computation is to be performed.
In Step 304, the generated global computation request is sent to a data zone.
In Step 306, the requested data is obtained from a data zone. The data zone of Step 306 may be the same or different from the data zone in Step 304.
The method may end following Step 306.
In Step 400, an upstream computation request is obtained.
In one or more embodiments of the invention, the upstream computation request is a global computation request obtained from a client.
In one or more embodiments of the invention, the upstream computation request is a downstream computation request obtained from another data zone.
As discussed above, in one or more embodiments of the invention, the system of
In Step 402, a computation framework is instantiated in response to obtaining the upstream computation request.
In one or more embodiments of the invention, instantiating the computation framework includes generating a computation manager (e.g., 152,
In Step 404, an aggregate computation of the instantiated computation framework is instantiated based on the obtained upstream computation request.
In one or more embodiments of the invention, the aggregate computation is instantiated by the computation manager. To instantiate the aggregate computation, the computation manager may identify a computation type specified by the obtained upstream computation request and instantiate the aggregate computation based on the identified computation type.
In one or more embodiments of the invention, the aggregate computation may be instantiated using the method illustrated in
In Step 406, local computation managers and local computations are instantiated based on the obtained upstream computation request.
In one or more embodiments of the invention, the computation manager may instantiate the local computation managers. In turn, each of the local computation managers may instantiate a corresponding local computation.
In one or more embodiments of the invention, the local computation managers instantiate corresponding local computations based on a data type or data format of the data on which the respectively local computation will operate. In one or more embodiments of the invention, the type of local computation is selected to minimize the computing resource cost of performing the local computation. In one or more embodiments of the invention, the type of local computation is selected to improve the efficiency of performing the local computation.
In one or more embodiments of the invention, the local computation managers instantiate corresponding local computations based on an availability of computing resources to perform different types of local computations. In one or more embodiments of the invention, the type of local computation is selected to meeting a scheduling requirement of the result to be generated by the local computation.
In one or more embodiments of the invention, the local computation managers and local computations may be instantiated using the method illustrated in
In Step 408, downstream computation managers are instantiated and computation framework(s) on other data zones are instantiated based on the obtained upstream computation request.
In one or more embodiments of the invention, the computation manager may instantiate the downstream computation managers. As noted above, the computation manager may instantiate the aggregate computation and thereby be aware of the data on which the aggregate computation operates. To provide some of the data, the computation manager may instantiate downstream computation managers to obtain computation results from other data zones to use as input to the aggregate computation.
In one or more embodiments of the invention, the downstream computation managers may instantiate corresponding computation frameworks on other data zones to obtain data needed by its aggregate computation to generate a computation result.
In one or more embodiments of the invention, the downstream computation managers and computing frameworks on other data zones may be instantiated using the method illustrated in
The method may end following Step 408.
Thus, as illustrated in
In Step 410, a computation result type and data sources are identified based on the obtained upstream computation request.
In one or more embodiments of the invention, the computation result type and/or data sources may be specified in the obtained upstream computation request. For example, the upstream computation request may be a global computation request or a downstream computation request.
In Step 412, space in a cache may be obtained to store the results generated by the aggregate computation.
In Step 414, executable code may be generated based on the identified computation result type, the identified data sources, and the obtained space in the cache. In one or more embodiments of the invention, the executable code may be computing instructions stored on a persistent storage of the data zone.
In one or more embodiments of the invention, a template may be selected based on the identified computation result type. The template may include prototype executable code and place holders for input and output. The prototype executable may be modified to replace the place holders with the identified data sources and space obtained in the cache as the input and output, respectively.
In one or more embodiments of the invention, the generated executable code is executed by processing resources of the data zone after the executable code is generated and stored on a persistent storage of the data zone.
The method may end following Step 414.
In Step 420, locked data batches are identified based on a computation description.
In one or more embodiments of the invention, the computation description is obtained from the obtained upstream computation request. For example, the upstream computation request may be a global computation request or a downstream computation request.
In Step 422, a local computation manager is instantiated corresponding to each identified locked data batch.
In Step 424, a computation type is selected for each local computation manager based on the characteristics of the locked data batch associated with each local computation manager.
In one or more embodiments of the invention, the characteristics of the locked data batch are a type of the data and a format of the data. In one or more embodiments of the invention, the computation type for each local computation manager is one of a map/reduce computation, a split-apply-combine computation, and a partially parallel computation. The computation type may be other computation types without departing from the invention.
In one or more embodiments of the invention, the computation type for each local computation manager may also be selected based on a computing resource availability for each computation type. For example, some computation types may not be available to be performed within a predetermined time set by a request even though the aforementioned computation may require fewer computing resources than other types of computations. In one or more embodiments of the invention, a less computing resource efficient computation type is selected when a more efficient computation type cannot be performed with the predetermined time.
In Step 426, the local computations are instantiated for each locked data batch by the corresponding local computation manager based on the corresponding selected computation type for the corresponding local computation manager.
In one or more embodiments of the invention, instantiating the local computation includes scheduling execution of the local computations and executing the local computations after scheduling.
In one or more embodiments of the invention, each local computation may be generated using a template corresponding to the selected computation type. The template may include prototype executable code and place holders for input and output. The prototype executable may be modified to replace the place holders with an identifier of the corresponding locked data batch on which the computation will be performed and an identifier of storage space for storing the result of the local computation.
In one or more embodiments of the invention, the generated executable code is executed by processing resources of the data zone after the executable code is generated and stored on a persistent storage of the data zone.
The method may end following Step 426.
In Step 430, a downstream computation manager of a computation framework is instantiated and the computation result of a computation framework to be generated on another data zone is designated as a data source for the aggregate computation of the computation framework.
In one or more embodiments of the invention, the downstream computation manager is instantiated by a computation manager of the framework. As discussed above, the computation framework may instantiate an aggregate computation and thereby is aware of data required by the instantiated aggregate computation to generate a computation result. The computation manager of the framework may provide the downstream computation manager with a type of the computation to be performed and a data source on which the computation is to be performed.
In one or more embodiments of the invention, designating the computation result of the to be generated computation framework on another data zone as a data source includes notifying the instantiated downstream computation manager that the computation result is to be provided to the aggregate computation.
In Step 432, a downstream computation request is generated based on the obtained upstream computation request.
In one or more embodiments of the invention, the downstream computation request may be similar to that shown in
In one or more embodiments of the invention, the downstream computation manager may select a data zone in which to instantiate a computation framework. In one or more embodiments of the invention, the selection is made using a map. In one or more embodiments of the invention, the selection is made to minimize a computing resource cost of performing a computation. In one or more embodiments of the invention, the selection is made to minimize a network bandwidth cost of performing the computation. In one or more embodiments of the invention, the selection is made based on a computing resource availability of the data zones. The selection may be made based on additional, different, or fewer factors without departing from the invention.
In Step 434, the computation framework of Step 430 is instantiated on another data zone determined in Step 432.
In one or more embodiments of the invention, the computation framework may be instantiated by sending the generated downstream computation request to another data zone. The other data zone may instantiate the computation framework in response to the request.
The method may end following Step 434.
In Step 500, a downstream computation result is obtained from a downstream data zone.
In one or more embodiments of the invention, a downstream data zone is a data zone in which a downstream computation was instantiated by the data zone performing the method illustrated in
In Step 502, a batch identifier of the obtained downstream computation result is obtained. The batch identifier may be obtained from the metadata associated with the downstream computation result.
In Step 504, a local computation result having the obtained batch identifier is obtained.
In Step 506, an upstream computation result is generated using the obtained local computation result and the obtained downstream computation result.
In one or more embodiments of the invention, the upstream computation result is generated as the output of an aggregate computation of a computing framework.
In Step 508, the obtained upstream computation result is sent to an upstream data zone that requested the obtained upstream computation result.
The method may end following Step 508.
To further clarify aspects of the invention, a non-limiting example is shown in
Consider a system, as illustrated in
Consider a scenario where the client (600) sends a global computation request to data zone A (610) requesting a five minute average of all of the sensor data. Averaging the data over time may anonymize the data sufficiently to allow it to be freely transmitted without violating the privacy concerns of the raw data.
In response to the request, the data zone A (610) instantiates a framework manager (not shown) associated with the request. The framework manager, in turn, instantiates an aggregate 5 minute sensor average A (612). Since none of the sensor data specified in the request is present on the data zone A (610), no local computation managers or local computations are instantiated by the framework manager. To obtain the input data for the aggregate 5 minute sensor average A (612) computation, the framework manager instantiates two downstream computations mangers (614) for data zones B and C. In turn, the respective downstream computation managers (614) instantiate computation frameworks, and corresponding framework managers, in data zone B (630) and data zone C (620), to obtain 5 minute sensor data averages of the respective zones to be used as inputs for the aggregate 5 minute sensor average A (612).
In data zone B (630), the framework manager instantiates an aggregate 5 minute sensor average B (632) computation. Since locked sensor data is stored in data zone B (630), the framework manager instantiates a 5 minute average of local sensor B (634) computation to generate a local computation result as an input to the aggregate 5 minute sensor average B (632) computation. To perform the local computation, a partially, parallel (636) computation is selected. Since a sensor in data zone D (640) is generating data, the framework manager instantiates a downstream computation manager for data zone D (638), The downstream computation manager for data zone D (638) instantiates a computation framework, and associated framework manager, to obtain the five minute average of the sensor data in zone D to be used as a second input from the aggregate 5 minute sensor average B (632).
In data zone D (640), the framework manager instantiates an aggregate 5 minute sensor average D (642) computation. Since locked sensor data is stored in data zone D (640), the framework manager instantiates a 5 minute average of local sensor D (644) computation to generate a local computation result as an input to the aggregate 5 minute sensor average D (642) computation. To perform the local computation, a partially parallel (646) computation is selected. Since no other sensor data is used as an input for the aggregate 5 minute sensor average D (642), no other local computations or downstream computation managers are instantiated.
Returning to data zone C (620), the framework manager instantiates an aggregate 5 minute sensor average C (622) computation. Since locked sensor data is stored in data zone C (620), the framework manager instantiates a 5 minute average of local sensor C1 (624) computation to generate a local computation result for the first sensor C1 and instantiates a 5 minute average of local sensor C2 (626) computation to generate a second local computation result as inputs to the aggregate 5 minute sensor average C2 (626) computation. To perform the local computations, a map/reduce (628) computation type for each of the local computations is selected. Since no other sensor data is used as an input for the aggregate 5 minute sensor average C (622), no other local computations or downstream computation managers are instantiated.
In the above discussed scenario, aggregate 5 minute sensor average C (622) computation uses the 5 minute average of local sensor C1 (624) computation and the 5 minute average of local sensor C2 (626) computation as inputs. Similarly, the aggregate 5 minute sensor average D (642) computation uses the 5 minute average of local sensor D (644) as an input.
In contrast, the aggregate 5 minute sensor average B (632) calculation uses the 5 minute average of local sensor B (634) computation and the aggregate 5 minute sensor average D (642) computation results as inputs.
The aggregate 5 minute sensor average A (612) uses the aggregate 5 minute sensor average C (622) computation and aggregate 5 minute sensor average (B) computation results as input.
To facilitate generating a global computation result, each of the aforementioned computations are keyed to only use results of similar batches as input. In this case, the key is the timestamp of each sensor data batch because the client global computation request specified a five minute average of the sensors.
The example ends.
The above example illustrates a recursive computation enabled by embodiments of the invention. More specifically, since the instantiation of a computation framework is capable of triggering the instantiation of computation frameworks in other data zones, a global framework that orchestrates performance of computations throughout the network of data zones is not required to perform data zone wide computations. Rather, each data zone of the system may be capable of instantiating computation frameworks that manage the computations in the respective data zone and trigger the instantiation of computation frameworks in other data zones.
Additionally, embodiments of the invention may enable eventually synchronous computational results to be obtained. As used herein, eventually synchronous computational results mean a computation result that would occur if all computations in a number of data zones had been performed at the same time on streaming or dynamic data sources at the same point in time but the computations were, in fact, performed at different points in time, Embodiments of the invention enable batches of data to be marked using metadata to establish groupings based on grouping criteria such as, for example, a time stamp associated with all or a portion of the data of a locked data batch. By performing computations on all of the locked data batches matching the grouping criteria, an eventually synchronous result may be obtained by recursively establishing computing frameworks that perform successive computations on locked data batches and/or computation results matching the grouping criteria.
Further embodiments of the invention may improve the performance of computations in a network environment by decentralizing the control of the computations performed in the network environment. In one or more embodiments of the invention, the computations may be decentralized by delegating: (i) selection of where computations will be performed and (ii) selection of the computation type used to obtain the computation result. By decentralizing control of the computations performed across data zones, embodiments of the invention may reduce the computing resource cost of performing the computations by allowing each computation framework to make the corresponding selections when instantiating computations. For example, computation frameworks may select computation types based on a type of the data or a format of the data that minimizes the computing resource cost of obtaining the computation result.
Still further, embodiments of the invention address the problem of computational resource cost scaling in a network environment by delegating the implementation of computations to data zones implementing the computations. Computing devices in a network have limited available information regarding the structure of the network, the format of data stored on the network, the type of data stored on the network, and the content of the data stored on the network. Delegating the implementation of specific computations across computing devices of data zones of the network of data zones in a dynamic manner using a set of rules for establishing computational frameworks reduces the impact of computation resource cost scaling in the network. In one or more embodiments of the invention, the set of rules includes: an order of instantiating component of a computation framework and criteria on which the instantiation of each component is based. In one or more embodiments of the invention, instantiating different components of a computational framework are based on different sets of rules. For example, instantiating local computations may be based on a different set of rules than establishing downstream computation managers.
While the above discussion highlighted features and/or uses of the invention, embodiments of the invention are not limited to similar uses and are not required to include similar features without departing from the invention. For example, some embodiments of the invention may have different, fewer, or more uses without departing from the invention.
Throughout this application, elements of figures may be labeled as A to N. As used herein, the aforementioned labeling means that the element may include any number of items and does not require that the element include the same number of elements as any other item labeled as A to N. For example, a data structure may include a first element labeled as A and a second element labeled as N. This number convention means that the data structure may include any of the elements. A second data structure, also labeled as A to N, may also include any number of elements. The number of elements of the first data structure and the number of elements of the second data structure may be the same or different.
One or more embodiments of the invention may be implemented using instructions executed by one or more processors of the data management device. Further, such instructions may correspond to computer readable instructions that are stored on one or more non-transitory computer readable mediums.
While the invention has been described above with respect to a limited number of embodiments, those skilled in the art, having the benefit of this disclosure, will appreciate that other embodiments can be devised which do not depart from the scope of the invention as disclosed herein. Accordingly, the scope of the invention should be limited only by the attached claims.
Florissi, Patricia Gomes Soares, Masad, Ofri
| Patent | Priority | Assignee | Title |
| 10656861, | Dec 29 2015 | EMC IP HOLDING COMPANY LLC | Scalable distributed in-memory computation |
| 10706970, | Apr 06 2015 | EMC IP HOLDING COMPANY LLC | Distributed data analytics |
| 10776404, | Apr 06 2015 | EMC IP HOLDING COMPANY LLC | Scalable distributed computations utilizing multiple distinct computational frameworks |
| 10791063, | Apr 06 2015 | EMC IP HOLDING COMPANY LLC | Scalable edge computing using devices with limited resources |
| 10860622, | Apr 06 2015 | EMC IP HOLDING COMPANY LLC | Scalable recursive computation for pattern identification across distributed data processing nodes |
| 10944688, | Apr 06 2015 | EMC IP HOLDING COMPANY LLC | Distributed catalog service for data processing platform |
| 10984889, | Apr 06 2015 | EMC IP HOLDING COMPANY LLC | Method and apparatus for providing global view information to a client |
| 10986168, | Apr 06 2015 | EMC IP HOLDING COMPANY LLC | Distributed catalog service for multi-cluster data processing platform |
| 10999353, | Apr 06 2015 | EMC IP HOLDING COMPANY LLC | Beacon-based distributed data processing platform |
| 11749412, | Apr 06 2015 | EMC IP HOLDING COMPANY LLC | Distributed data analytics |
| 11854707, | Apr 06 2015 | EMC IP HOLDING COMPANY LLC | Distributed data analytics |
| Patent | Priority | Assignee | Title |
| 10015106, | Apr 06 2015 | EMC IP HOLDING COMPANY LLC | Multi-cluster distributed data processing platform |
| 10111492, | Aug 26 2014 | NIKE, Inc | Article of footwear with dynamic edge cavity midsole |
| 10114923, | Apr 06 2015 | EMC IP HOLDING COMPANY LLC | Metagenomics-based biological surveillance system using big data profiles |
| 10122806, | Apr 06 2015 | EMC IP HOLDING COMPANY LLC | Distributed analytics platform |
| 10127352, | Apr 06 2015 | EMC IP HOLDING COMPANY LLC | Distributed data processing platform for metagenomic monitoring and characterization |
| 10270707, | Apr 06 2015 | EMC IP HOLDING COMPANY LLC | Distributed catalog service for multi-cluster data processing platform |
| 10277668, | Apr 06 2015 | EMC IP HOLDING COMPANY LLC | Beacon-based distributed data processing platform |
| 10311363, | Apr 06 2015 | EMC IP HOLDING COMPANY LLC | Reasoning on data model for disease monitoring, characterization and investigation |
| 10331380, | Apr 06 2015 | EMC IP HOLDING COMPANY LLC | Scalable distributed in-memory computation utilizing batch mode extensions |
| 10348810, | Apr 06 2015 | EMC IP HOLDING COMPANY LLC | Scalable distributed computations utilizing multiple distinct clouds |
| 6112225, | Mar 30 1998 | International Business Machines Corporation | Task distribution processing system and the method for subscribing computers to perform computing tasks during idle time |
| 6516350, | Jun 17 1999 | International Business Machines Corporation | Self-regulated resource management of distributed computer resources |
| 7657537, | Apr 29 2005 | NetApp, Inc | System and method for specifying batch execution ordering of requests in a storage system cluster |
| 7934018, | Sep 30 2004 | EMC IP HOLDING COMPANY LLC | Methods and apparatus for synchronizing configuration data |
| 7934248, | Jun 27 2007 | EMC IP HOLDING COMPANY LLC | Network policy enforcement dashboard views |
| 8224825, | May 31 2010 | Microsoft Technology Licensing, LLC | Graph-processing techniques for a MapReduce engine |
| 8392564, | Jun 20 2005 | Oracle America, Inc | Cluster-wide resource usage monitoring |
| 8499331, | Jun 27 2007 | EMC IP HOLDING COMPANY LLC | Policy based network compliance |
| 8732118, | Jan 13 2012 | Amazon Technologies, Inc. | Distributed performance of data aggregation operations |
| 8806061, | Sep 14 2004 | VALTRUS INNOVATIONS LIMITED | System, method, and computer program product for automated categorization of data processing services and components |
| 8938416, | Jan 13 2012 | Amazon Technologies, Inc. | Distributed storage of aggregated data |
| 9020802, | Mar 30 2012 | EMC IP HOLDING COMPANY LLC | Worldwide distributed architecture model and management |
| 9031992, | Sep 30 2011 | EMC IP HOLDING COMPANY LLC | Analyzing big data |
| 9130832, | Oct 09 2014 | SPLUNK, Inc. | Creating entity definition from a file |
| 9158843, | Mar 30 2012 | EMC IP HOLDING COMPANY LLC | Addressing mechanism for data at world wide scale |
| 9229952, | Nov 05 2014 | WELLS FARGO BANK, N A | History preserving data pipeline system and method |
| 9235446, | Jun 22 2012 | Microsoft Technology Licensing, LLC | Parallel computing execution plan optimization |
| 9239711, | Dec 31 2013 | GOOGLE LLC | Run benchmark or analysis tools against massive repository of archived webpages on machines in the cloud for continuous builds or AD-HOC requests |
| 9280381, | Mar 30 2012 | EMC IP HOLDING COMPANY LLC | Execution framework for a distributed file system |
| 9338218, | Dec 21 2011 | EMC IP HOLDING COMPANY LLC | Distributed platform as a service |
| 9361263, | Dec 21 2011 | EMC IP HOLDING COMPANY LLC | Co-located clouds, vertically integrated clouds, and federated clouds |
| 9418085, | Mar 13 2013 | Amazon Technologies, Inc | Automatic table schema generation |
| 9489233, | Mar 30 2012 | EMC IP HOLDING COMPANY LLC | Parallel modeling and execution framework for distributed computation and file system access |
| 9678497, | Aug 30 2013 | Regents of the University of Minnesota | Parallel processing with cooperative multitasking |
| 9697262, | Dec 17 2013 | Microsoft Technology Licensing, LLC | Analytical data processing engine |
| 9747127, | Mar 30 2012 | EMC IP HOLDING COMPANY LLC | Worldwide distributed job and tasks computational model |
| 9747128, | Dec 21 2011 | EMC IP HOLDING COMPANY LLC | Worldwide distributed file system model |
| 9767149, | Oct 10 2014 | International Business Machines Corporation | Joining data across a parallel database and a distributed processing system |
| 9832068, | Dec 17 2012 | Microsoft Technology Licensing, LLC | Reachability-based coordination for cyclic dataflow |
| 9838410, | Aug 31 2015 | SPLUNK Inc. | Identity resolution in data intake stage of machine data processing platform |
| 9848041, | May 01 2015 | Amazon Technologies, Inc | Automatic scaling of resource instance groups within compute clusters |
| 9996662, | Apr 06 2015 | EMC IP HOLDING COMPANY LLC | Metagenomics-based characterization using genomic and epidemiological comparisons |
| 20020073167, | |||
| 20030212741, | |||
| 20040247198, | |||
| 20050010712, | |||
| 20050102354, | |||
| 20050114476, | |||
| 20050132297, | |||
| 20050153686, | |||
| 20050165925, | |||
| 20050266420, | |||
| 20060002383, | |||
| 20060122927, | |||
| 20060126865, | |||
| 20070026426, | |||
| 20070076703, | |||
| 20070088703, | |||
| 20080027954, | |||
| 20080028086, | |||
| 20080077607, | |||
| 20080155100, | |||
| 20080260119, | |||
| 20080279167, | |||
| 20090062623, | |||
| 20090076651, | |||
| 20090150084, | |||
| 20090198389, | |||
| 20090310485, | |||
| 20090319188, | |||
| 20100005077, | |||
| 20100042809, | |||
| 20100076856, | |||
| 20100122065, | |||
| 20100131639, | |||
| 20100184093, | |||
| 20100229178, | |||
| 20100250646, | |||
| 20100290468, | |||
| 20100293334, | |||
| 20100299437, | |||
| 20110020785, | |||
| 20110029999, | |||
| 20110103364, | |||
| 20110145828, | |||
| 20110314002, | |||
| 20120030599, | |||
| 20130035956, | |||
| 20130044925, | |||
| 20130054670, | |||
| 20130194928, | |||
| 20130246460, | |||
| 20130282897, | |||
| 20130290249, | |||
| 20130291118, | |||
| 20130318257, | |||
| 20130346229, | |||
| 20130346988, | |||
| 20140012843, | |||
| 20140025393, | |||
| 20140075161, | |||
| 20140081984, | |||
| 20140082178, | |||
| 20140143251, | |||
| 20140173331, | |||
| 20140173618, | |||
| 20140280298, | |||
| 20140280363, | |||
| 20140280604, | |||
| 20140280990, | |||
| 20140310258, | |||
| 20140310718, | |||
| 20140320497, | |||
| 20140325041, | |||
| 20140358999, | |||
| 20140365518, | |||
| 20140372611, | |||
| 20140379722, | |||
| 20150006619, | |||
| 20150019710, | |||
| 20150039586, | |||
| 20150039667, | |||
| 20150066646, | |||
| 20150081877, | |||
| 20150088786, | |||
| 20150120791, | |||
| 20150121371, | |||
| 20150169683, | |||
| 20150178052, | |||
| 20150254344, | |||
| 20150254558, | |||
| 20150269230, | |||
| 20150277791, | |||
| 20150278513, | |||
| 20150295781, | |||
| 20150302075, | |||
| 20150339210, | |||
| 20150355946, | |||
| 20150369618, | |||
| 20160004827, | |||
| 20160063191, | |||
| 20160072726, | |||
| 20160087909, | |||
| 20160098021, | |||
| 20160098472, | |||
| 20160098662, | |||
| 20160112531, | |||
| 20160132576, | |||
| 20160170882, | |||
| 20160179642, | |||
| 20160179979, | |||
| 20160182305, | |||
| 20160182327, | |||
| 20160188594, | |||
| 20160196324, | |||
| 20160205106, | |||
| 20160241893, | |||
| 20160246981, | |||
| 20160260023, | |||
| 20160261727, | |||
| 20160267132, | |||
| 20160269228, | |||
| 20160283551, | |||
| 20160323377, | |||
| 20160328661, | |||
| 20160337473, | |||
| 20160350157, | |||
| 20170006135, | |||
| 20170032263, | |||
| 20170083573, | |||
| 20170109299, | |||
| 20170116289, | |||
| 20170187785, | |||
| 20170220646, | |||
| 20170272458, | |||
| 20170323028, | |||
| 20180054355, | |||
| 20180101583, | |||
| 20180189296, | |||
| 20180308585, |
| Date | Maintenance Fee Events |
| Oct 31 2017 | BIG: Entity status set to Undiscounted (note the period is included in the code). |
| May 23 2023 | M1551: Payment of Maintenance Fee, 4th Year, Large Entity. |
| Date | Maintenance Schedule |
| Dec 10 2022 | 4 years fee payment window open |
| Jun 10 2023 | 6 months grace period start (w surcharge) |
| Dec 10 2023 | patent expiry (for year 4) |
| Dec 10 2025 | 2 years to revive unintentionally abandoned end. (for year 4) |
| Dec 10 2026 | 8 years fee payment window open |
| Jun 10 2027 | 6 months grace period start (w surcharge) |
| Dec 10 2027 | patent expiry (for year 8) |
| Dec 10 2029 | 2 years to revive unintentionally abandoned end. (for year 8) |
| Dec 10 2030 | 12 years fee payment window open |
| Jun 10 2031 | 6 months grace period start (w surcharge) |
| Dec 10 2031 | patent expiry (for year 12) |
| Dec 10 2033 | 2 years to revive unintentionally abandoned end. (for year 12) |