Methods and systems for managing queues. The disclosed methods involve receiving, at an interface, a request to store a message in a queue, storing the message as a row in a key-value store database, extracting at least one attribute from the message; and selecting the message for processing based on the at least one extracted attribute.
|
11. A system comprising:
one or more computers systems that a queue management system, configured to:
receive, at an interface, a request to create a queue to store messages from a client in a sort order;
create the queue in a data store, wherein the data store stores a plurality of queues managed by the queue management system, comprising a first queue that stores first messages in a first-in-first-out (FIFO) order based on a creation time attribute of the first messages and a second queue that stores second messages in a last-in-first-out (LIFO) order based on a creation time attribute of the second messages, and wherein the queues are stored on different partitions of the data store determined based on a queue name of individual ones of the queues;
monitor one or more statistics of the queue in the data store;
determine, based on the one or more statistics, that the queue is to be processed next among the plurality of queues; and
select a message from the queue for processing based on the sort order of the queue, wherein the message is processed to detect malicious or unusual activity in connection with the client.
1. A method comprising:
performing, by a queue management system implemented by one or more computers:
receiving, at an interface, a request to create a queue to store messages from a client in a sort order;
creating the queue in a data store, wherein the data store stores a plurality of queues managed by the queue management system, comprising a first queue that stores first messages in a first-in-first-out (FIFO) order based on a creation time attribute of the first messages and a second queue that stores second messages in a last-in-first-out (LIFO) order based on a creation time attribute of the second messages, and wherein the queues are stored on different partitions of the data store determined based on a queue name of individual ones of the queues;
monitoring one or more statistics of the queue in the data store;
determining, based on the one or more statistics, that the queue is to be processed next among the plurality of queues; and
selecting a message from the queue for processing based on the sort order of the queue, wherein the message is processed to detect malicious or unusual activity in connection with the client.
19. One or more non-transitory computer-readable media storing program instructions that when executed on or across one or more processors implement a queue management system and cause the queue management system to:
receive, at an interface, a request to create a queue to store messages from a client in a sort order;
create the queue in a data store, wherein the data store stores a plurality of queues managed by the queue management system, comprising a first queue that stores first messages in a first-in-first-out (FIFO) order based on a creation time attribute of the first messages and a second queue that stores second messages in a last-in-first-out (LIFO) order based on a creation time attribute of the second messages, and wherein the queues are stored on different partitions of the data store determined based on a queue name of individual ones of the queue;
monitor one or more statistics of the queue in the data store;
determine, based on the one or more statistics, that the queue is to be processed next among the plurality of queues; and
select a message from the queue for processing based on the sort order of the queue, wherein the message is processed to detect malicious or unusual activity in connection with the client.
2. The method of
elevating the queue for processing based on the one or more statistics; and
deprioritizing the queue for processing based on the one or more statistics.
3. The method of
a size of the queue;
an age of the queue; and
a last update time of the queue.
4. The method of
5. The method of
6. The method of
7. The method of
8. The method of
9. The method of
10. The method of
a size of the message;
an age of the message;
a priority of the message; and
a sensitivity of the message.
12. The system of
elevate the queue for processing based on the one or more statistics; and
deprioritize the queue for processing based on the one or more statistics.
13. The system of
a size of the queue;
an age of the queue; and
a last update time of the queue.
14. The system of
15. The system of
extract one or more attributes from the message; and
select the message from the queue based on the one or more attributes.
16. The system of
a size of the message;
an age of the message;
a priority of the message; and
a sensitivity of the message.
17. The system of
20. The one or more non-transitory computer-readable media of
|
|||||||||||||||||||||||||||||
The present application claims the benefit (and is a Continuation) of pending U.S. patent application Ser. No. 16/685,633 titled “Dynamic Queue Management” and filed on Nov. 15, 2019,” the entire disclosure of which is incorporated by reference herein.
Embodiments described herein generally relate to methods and systems for data storage and retrieval and, more particularly but not exclusively, to methods and systems implementing queues for data storage and retrieval.
Product vendors or other types of service providers often receive large amounts of data from one or multiple customers. These service providers may execute event systems that are tasked with analyzing the received data in accordance with one or more criteria, initiating some type of workflow in response to the analysis, and providing some type of report or summary regarding the received data. Event systems that have downstream batch processing workflows often have to group the received data by various attributes. These attributes can be organization, application, customer, data type, etc.
The act of grouping essentially represents the placement of related events in a queue. However, most available queue systems have a cost associated with the creation of each queue that can hinder the processing of large quantities of data. For queue systems that are dynamic, they often do not have durable persistence. Both types of queue systems are also typically limited in the number of queues and in the number of messages per queue that can be supported.
For example, existing queueing technologies include redis-backed queues that use libraries such as redisson. However, these types of technologies are completely memory-based. This makes these technologies susceptible to, e.g., power failures, which can result in the loss of messages or other types of data.
RabbitMQ® has dynamic queue capabilities as well. However, it is generally difficult to create dynamic queues with RabbitMQ®, as users have to predefine attributes that define the dynamic portion(s) of the queue.
Other existing techniques involve a user creating queues for processing. However, these processes are generally time consuming and labor intensive.
A need exists, therefore, for systems and methods that overcome the disadvantages of existing queueing techniques.
This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description section. This summary is not intended to identify or exclude key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
In one aspect, embodiments relate to a method of managing queues. The method includes receiving, at an interface, a request to store a message in a queue; storing the message as a row in a key-value store database, wherein the queue is associated with a partition in the database; extracting, using a processor executing instructions stored on memory, at least one attribute from the message; and selecting, using the processor, the message for processing based on the at least one extracted attribute.
In some embodiments, the database includes a plurality of partitions that are each associated with a different queue.
In some embodiments, the at least one attribute includes at least one of age of the message and sensitivity of the message.
In some embodiments, the partition includes a plurality of messages, and the method further includes: performing, using the processor, an attribute analysis on each of the plurality of messages; and elevating for processing a message within the partition based on the attribute analysis.
In some embodiments, the queue is associated with a sort key, and the method further includes associating a unique identifier of the message to the sort key of the queue.
In some embodiments, the method further associating the message with a key unique to the queue. In some embodiments, the key is defined by an organization field and a name field.
In some embodiments, the database includes a plurality of partitions that are each associated with a queue, and the method further includes elevating at least one of the queues for analysis based on a number of messages within the queue and an existence duration of the queue. In some embodiments, the method further includes deprioritizing at least a second queue based on a number of messages within the second queue and an existence duration of the second queue.
According to another aspect, embodiments relate to a system for managing queues.
The system includes an interface for at least receiving a request to store a message in a queue; a key-value store database for storing the message, wherein the queue is associated with a partition in the database; and a processor executing instructions stored in memory configured to extract at least one attribute from the message and select the message for processing based on the at least one extracted attribute.
In some embodiments, the database includes a plurality of partitions that are each associated with a different queue.
In some embodiments, the at least one attribute includes at least one of age of the message and sensitivity of the message.
In some embodiments, the partition includes a plurality of messages and the processor is further configured to perform an attribute analysis on each of the plurality of messages and elevate for processing a message within the partition based on the attribute analysis.
In some embodiments, the queue is associated with a sort key and the method further includes associating a unique identifier of the message to the sort key of the queue.
In some embodiments the processor is further configured to associate the message with a key unique to the queue. In some embodiments, the key is defined by an organization field and a name field.
In some embodiments, the database includes a plurality of partitions that are each associated with a queue, and the processor is further configured to elevate at least one of the queues for analysis based on a number of messages within the queue and an existence duration of the queue. In some embodiments, the processor is further configured to deprioritize at least a second queue based on a number of messages within the second queue and an existence duration of the second queue.
Non-limiting and non-exhaustive embodiments of this disclosure are described with reference to the following figures, wherein like reference numerals refer to like parts throughout the various views unless otherwise specified.
As discussed previously, existing queueing technologies generally are not dynamic in nature and require significant amounts of time to process incoming data. Oftentimes, the order in which the incoming data is processed is static in nature and incoming data is not processed in ways that are necessarily logical or expeditious.
Existing queueing technologies that are dynamic are generally memory-based. This makes these technologies susceptible to power loss or memory failure, which can result in system failure and the loss of messages or other data.
To address these shortcomings, the systems and methods described herein provide a dynamic queueing system that may be based on DynamoDB or another type of key-value store style database. Each row within DynamoDB consists of a partition key and a sort key, followed by any number of key-value attributes that a user may define or that are automatically generated.
In order to present a queue-style interface for a user, a queue-style application programming interface (API) is supplied for interactions with the database. The systems and methods described herein therefore adapt the tabular style format of these key-value databases to present a familiar and easy-to-understand queue style interface for data storage and retrieval.
The first object of the queue interface of the embodiments described herein is the “QueueName” class which may contain two fields: organization and name. Combined, these two fields represent a unique key. This key represents a queue instance that has the ability to hold one or more messages. “Message” is the other object of the queue interface and contains the actual information or data about the event and is associated with a queue via the QueueName object until it is removed from the queue.
All messages may contain a universally unique identifier (for simplicity, “UUID”) that maps to the sort key of DynamoDB or other types of key-value pair database structures. The message may also contain or otherwise be associated with any number of attributes to provide additional information about the event to support higher level constructs. These attributes may include, but are not limited to, message priority, message age, or any user-defined content that users wish to provide. The flexibility to associate different types of information with messages allows the systems and methods described herein to implement queue style features such as FIFO (first in, first out), LIFO (last in, first out), and priority-based queues.
Below is a table that shows how messages are adapted for DynamoDB or for another key-value style database for a particular queue:
TABLE 1
Queue storage configuration
Partition Key
Sort Key
AttrA
AttrB
AttrC
QueueName
Message UUID
. . .
. . .
. . .
As seen in Table 1, a queue in DynamoDB may be defined by a partition key. Sort keys are attached to a message's UUID so the association between message and queue is established. The number of attributes and their content are dynamic and can be any number of any type and at any time.
Accordingly, the embodiments described herein allow users to implement a dynamic queue system that can span thousands of potential queues without the need to implement any pre-provisioning of queues. Then, with these various queues, the embodiments described herein may be able to perform differential analysis among queues. As discussed above, the embodiments described herein may be based on DynamoDB or other types of key-value data structures to provide durability, scale, and convenience. This has proven to be a successful use case for batch-style processing systems where data or messages must be grouped by some key before being processed. Messages may also be schema-less, and may be analyzed in conjunction with different queue types such as queue sets, priority-based queues, FIFO queues, LIFO queues, etc.
Additionally, the embodiments described herein offer a number of advantages over existing techniques for managing queues. For example, there is no limit on the number of queues or the number of messages within each queue that may be supported. These embodiments also offer the ability to scale up/down read or write operations via the AWS® Dynamo API, and offer customizable sorting for queues based on statistics such as fairness (discussed below), size, age, or other attributes that may help determine which queue should be processed next. The embodiments described herein also provide for much more durability than existing techniques that are memory-based (e.g., Redisson).
The user device 102 may be any hardware device capable of executing the user interface 104. The user device 102 may be configured as a laptop, PC, tablet, mobile device, or the like. The exact configuration of the user device 102 may vary as long as it can execute and present the user interface 104 to the user 106. The user interface 104 may allow the user 106 to specify and view certain queues or messages for analysis, as well as attributes associated with messages.
The user 106 may be an employee of a product vendor or service provider interested in monitoring queues and messages or other data from various clients. The user 106 may rely on the user interface 104 to define or view various attributes associated with received message data. The system 100 reduces the need for the user 106 to manually create queues. Rather, these queues may be automatically generated as the messages are received.
The processor 108 may execute instructions stored on memory 110 to provide various modules to accomplish the objectives of the various embodiments described herein. Specifically, the processor 108 may execute or otherwise include an interface 112, an attribute extraction module 114, an analysis module 116, and a message selection module 118.
The interface 112 may receive messages from one or more clients 120, 122, and 124 over one or more networks 126. The clients 120, 122, and 124 may include servers, devices, storages, or the like, that are associated with clients or customers of a vendor or service provider. For example, the clients 120, 122, and 124 may have hired a vendor associated with the processor 108 to perform some type of analysis on the clients' message data. Each message may be associated with some identification label (ID), such that messages associated with the same client can be grouped together.
The interface 112 may receive messages in batches or otherwise at predetermined time intervals. For example, messages may be communicated to the interface 112 once every hour. The frequency at which messages are sent over the network(s) 126 to the interface 112 may vary and may be determined or set by the user 106 or personnel associated with the clients 120, 122, and 124.
Once received, and before processing, the messages and any associated data may be stored in one or more databases 128. As discussed previously, the database(s) 128 may store the received data in a key-value style format, such as that presented in Table 1 above.
Specifically, each row in the database(s) 128 may represent a message. For example,
For example, and as seen in
The analysis module 116 may compute statistics on the queues and their contents. Since potentially there can be thousands of queues, a user such as the user 106 may want to know which queues have the most messages as well as which queue(s) should have their messages processed next. Accordingly, users 106 and clients can be assured that messages are processed in an appropriate manner and order.
For example,
The statistics may contain information about a queue, including when it was created, how many messages it contains, the last update time, etc. This allows a user such as the user 106 to find the largest queues by message size or most important queues by fairness. In the context of the present application, the term “fairness” may be based on factors such as the number of messages in a queue and their existence duration. A higher fairness value indicates that the associated queue should be processed next. This allows users to guarantee that smaller queues will not remain unprocessed in favor of, for example, larger and highly-used queues.
The message selection module 118 may also provide various types of queues such as FIFO (first in, first out), LIFO (last in, first out), and priority-based queues. As discussed above, this may be done based on the attributes assigned to every message. For priority-based queues, a user such as the user 106 may provide a value to represent a “priority” attribute. Similarly, attributes reflecting the time of creation of the message may be used to implement LIFO/FIFO functionality.
The queues and their accompanying statistics in the one or more databases 128 may be accessed through an API library 310. A user 312 such as the user 106 of
The user 312 may choose certain queues or messages for processing in accordance with any number of criteria. For example, the user may configure the message selection module 118 to select messages in accordance with specified criterion.
In some embodiments, the message selection module 118 may select the queues with the oldest messages for processing first (i.e., FIFO). As another example, the message selection module 118 may select the queue that has the most messages therein for processing first. In this case, and with reference to
Step 402 involves receiving, at an interface, a request to store a message in a queue. The message may be received over one or more networks from one or more clients. In this case, a vendor or other type of service provider may be tasked with receiving incoming messages, processing the incoming messages, and then performing some further action. For example, the vendor may be tasked with informing their client that the former detected malicious or otherwise unusual activity in connection with the received message(s).
Step 404 involves storing the message as a row in a key-value store database, wherein the queue is associated with a partition in the database. The incoming message(s) may be stored in a key-value store database until the vendor is able to process the message(s). The database may be divided into a plurality of partitions that each contain one or more messages. This division avoids having too many messages being associated with the same key, and therefore avoids a particular partition or queue becoming “hot” and failing.
Step 406 involves extracting, using a processor executing instructions stored on memory, at least one attribute from the message. An attribute extraction module such as the attribute extraction module 114 may gather certain data related to one or more messages of one or more queues. These attributes may be related to, for example, the age of the message, the size of the message, a priority level of the message, whether the message includes sensitive information, or the like.
Step 408 involves selecting, using the processor, the message for processing based on the at least one extracted attribute. As discussed above, a message may be selected based on the message possessing one or more various attributes. For example, the oldest message in a queue may be selected for processing based solely on the fact that it is the oldest message in the queue. As another example, larger messages may be selected for processing before smaller messages are selected.
Step 410 involves deprioritizing at least a second queue based on a number of messages within the second queue and the existence duration of the second queue. Similar to Step 408, the message selection module 118 may opt to not process a message based on one or more extracted attributes. For example, if the message selection module 118 was configured to process older messages first, the message selection module 118 may specifically deprioritize more recently-received messages so that they are not processed until older messages are processed first.
Accordingly, the systems and methods described herein provide a dynamic queueing system that can support potentially thousands of queues at any time and are able to compare and contrast among queues to make decisions regarding which queue(s) should be processed and in what order.
| Patent | Priority | Assignee | Title |
| Patent | Priority | Assignee | Title |
| 6094694, | Oct 04 1997 | International Business Machines Corporation | System for storing new messages in both full-length and abbreviated versions of message expiration data, and eliminating old, expired messages when a message is retrieved |
| 6182120, | Sep 30 1997 | International Business Machines Corporation | Method and system for scheduling queued messages based on queue delay and queue priority |
| 6501733, | Oct 01 1999 | Lucent Technologies Inc. | Method for controlling data flow associated with a communications node |
| 6934950, | Jun 06 2000 | International Business Machines Corporation | Thread dispatcher for multi-threaded communication library |
| 7035852, | Jul 21 2000 | International Business Machines Corporation | Implementing a message queuing interface (MQI) indexed queue support that adds a key to the index on put commit |
| 7127507, | Sep 27 2001 | T-MOBILE INNOVATIONS LLC | Method and apparatus for network-level monitoring of queue-based messaging systems |
| 7543181, | Apr 10 2003 | Lenovo PC International | Recovery from failures within data processing systems |
| 7680793, | Oct 07 2005 | Oracle International Corporation | Commit-time ordered message queue supporting arbitrary read and dequeue patterns from multiple subscribers |
| 7849468, | May 20 2004 | International Business Machines Corporation | Enhanced browsing of messages in a message queue |
| 8627333, | Aug 03 2011 | International Business Machines Corporation | Message queuing with flexible consistency options |
| 20180013714, | |||
| 20200120541, | |||
| 20210089336, |
| Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
| Nov 30 2020 | LEE, AUSTIN | Rapid7, Inc | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 069049 | /0789 | |
| Jun 01 2021 | Rapid7, Inc. | (assignment on the face of the patent) | / |
| Date | Maintenance Fee Events |
| Jun 01 2021 | BIG: Entity status set to Undiscounted (note the period is included in the code). |
| Date | Maintenance Schedule |
| Aug 29 2026 | 4 years fee payment window open |
| Mar 01 2027 | 6 months grace period start (w surcharge) |
| Aug 29 2027 | patent expiry (for year 4) |
| Aug 29 2029 | 2 years to revive unintentionally abandoned end. (for year 4) |
| Aug 29 2030 | 8 years fee payment window open |
| Mar 01 2031 | 6 months grace period start (w surcharge) |
| Aug 29 2031 | patent expiry (for year 8) |
| Aug 29 2033 | 2 years to revive unintentionally abandoned end. (for year 8) |
| Aug 29 2034 | 12 years fee payment window open |
| Mar 01 2035 | 6 months grace period start (w surcharge) |
| Aug 29 2035 | patent expiry (for year 12) |
| Aug 29 2037 | 2 years to revive unintentionally abandoned end. (for year 12) |