A generic fault tolerant "match-and-set" locking mechanism and method for controlling access to resources shared among a plurality of users n. The match-and-set lock has a locked operating state and an unlocked operating state controlled by the value c of its content such that the lock is in its locked operating state when C≠0 and in its unlocked operating state where C=0. The lock returns a value R, equal to the lock's current content c, to an inquiring user seeking access to the resource. A return value R=0 denotes that the resource is free, and a return value R≠0 denotes that the resource is locked by another user. The lock is responsive to a command in the form (A, B), such that the lock substitutes B for c if A=C. Thus, the lock may be locked by issuing the command (A, B) where A=C and B≠0; and the lock may be released by issuing the command (A, B) where A=C and B=0. A deadlock condition may be avoided by setting the lock to the value B=P+T*(n+1), where P<(n+1) and identifies the user issuing this command (A, B), and T is the current global time stamp when the user issues this command. When the return value R≠0, R identifies the user currently locking the resource and the time when that user locked the resource: the locking user id is simply R%(n+1), and the locking time is R/(n+1). An inquiring user can then determine if the user currently locking the resource has failed, or restarted, since locking the resource. If so, the inquiring user can rescue the lock from the control of the failed user. The mechanism of "match-and-set" ensures that exactly one such user can succeed in rescuing that lock. Crashed users can come back online and participate in new locking operations: the locking semantics is always maintained and this system suffers no deadlocks. The implementation requirements are easy to satisfy: that there is a global reference clock, that any user maintains a time stamp (off the global clock) of its latest reboot time, and the match-and-set access is an indivisible operation. The last can be implemented in hardware in a way similar to that of the traditional test-and-set lock. The plurality of users may be processors in a multiprocessor computer system, and the processors may form a telecommunications system.
|
1. A match-and-set lock for controlling access to a resource that is shared among a plurality of users n, said lock having a locked operating state and an unlocked operating state controlled by a value c such that the lock is in its locked operating state when C≠0 and in its unlocked operating state where C=0, said lock being responsive to a command in the form (A, B), said lock substituting B for c if A=C, whereupon:
the lock may be released by a user issuing the command (A, B) where A=C and B=0; the lock may be set by a user issuing the command (A, B) where A=C and B≠0; and a deadlock condition may be avoided by always setting the lock to the value B=P+T*(n+1) where P<(n+1) and identifies the user issuing the command (A, B) and T is the current time stamp.
11. A method of controlling access to a shared resource from a plurality of processors n, said method employing a match-and-set lock that has a current content c and that is responsive to a match-and-set command (A, B) that writes B into the current content if A=C, said lock committing the shared resource to the control of that processor that most recently set the current content to a non-zero value, said method comprising the steps of:
(a) initializing the lock by setting the current content to zero; (b) issuing from one of the processors a command (M, S) where M equals an expected current content of the lock and S=P+T*(n+1) where P<(n+1) and identifies the processor issuing the command and T is a current time stamp; (c) if a non-zero value R is returned from the time lock, identifying the processor that set the lock as P'=R% (n+1) where R% (n+1) is the remainder of R, mod (n+1), and identifying the approximate locking time as T'=R/(n+1); (d) if processor P' is still operating and was last started before time T', repeating step (b); (e) if processor P' is not still operating or was last started after time T', issuing from processor P the command (M, S) where M=R.
8. A software program implemented in a multiprocessor computer system, said software program controlling access to a shared resource from a plurality of processors n, by employing a match-and-set lock that has a current content c and that is responsive to a match-and-set command (A, B) that writes B into the current content if A=C, said lock committing the shared resource to the control of that processor that most recently set the current content to a non-zero value, said software program allowing the multiprocessor system to:
(a) initialize the lock by setting the current content to zero; (b) issue from one of the processors a command (M, S) where M equals an expected current content of the lock and S=P+T*(n+1) where P<(n+1) and identifies the processor issuing the command and T is a current time stamp; (c) if a non-zero value R is returned from the lock, identify the processor that set the lock as P'=R% (n+1) where R% (n+1) is the remainder of R, mod (n+1), and identify the approximate locking time as T'=R/(n+1); (d) if processor P' is still operating and was last started before time T', repeat step (b); (e) if processor P' is not still operating or was last started after time T', issue from processor P the command (M, S) where M=R.
9. A method of controlling access to a shared resource from a plurality of processors n, comprising the steps of:
(a) providing the resource with a locking storage function, wherein the function stores either an initial value denoting that the resource is not currently held by a processor or a value denoting a holding processor currently holding the resource and an initial holding time when such holding began; (b) initializing the function by storing an initial value denoting that the resource is not currently held by a processor; (c) querying the function to determine if the resource is currently held by a processor; (d) if the resource is not currently held by a processor, providing to an inquiring processor access to the resource and resetting the value of the function to denote the inquiring processor's identity and a current time; (e) if the resource is currently held by a processor, determining if the holding processor has failed or been restarted since taking hold of the resource; (f) if the holding processor has failed or been restarted since taking hold of the resource, providing to the inquiring processor access to the resource and resetting the value of the function to denote the inquiring processor's identity and a current time; (g) if the holding processor has not failed or been restarted since taking hold of the resource, repeating steps (b) through (d); and (h) upon release of the resource, resetting the value stored in the function to an initial value denoting that the resource is not currently held by a processor.
2. The match-and-set lock of
3. The match-and-set lock of
4. The match-and-set lock of
5. The match-and-set lock of
6. The match-and-set lock of
|
The present invention relates generally to a fault tolerant locking mechanism and to the use of such a mechanism in preventing deadlocks in multiprocessor computer systems.
As used herein, the term computer includes any device or machine capable of accepting data, applying prescribed processes to the data, and supplying the results of the processes. A multiprocessing computer system has multiple processes executing on the system. Each process performs a particular task, and the processes, taken as a whole, perform some larger task, typically called an application. These processes may be executing on a single central computer or they may be running on separate computers which are connected to each other via some type of communications link, i.e., a distributed or networked computer system.
In multiprocessing systems, resources are often shared among the executing processes. Such resources may include, for example, disk drives, printers, shared memory and databases. During processing, a process may require exclusive access to a resource, such that another process may not use that resource until the first process is finished with it. Thus, several processes may compete for a finite number of resources. This is commonly known as mutually exclusive sharing of resources.
A problem with mutually exclusive access to resources is the possibility that the computer system will enter a deadlock state. A deadlock state is a state in the computer system in which, because of a resource allocation pattern, the computer system cannot progress past a processing point. For example, consider a computer system with two resources R1 and R2, and two processors P1 and P2, where P1 and P2 both need simultaneous exclusive access to R1 and R2 at some point in order to successfully complete their processing. If P1 gains exclusive access to R1 and R2, then P2 must wait until P1 releases the resources. In this situation, P2 is described as being in a "wait" state. Such a situation does not present a problem, because, it is presumed, that P1 will eventually release the resources, at which time P2 may gain access to the resources. However, consider the situation in which P1 gains access to and holds R1, and P2 gains access to and holds R2, as shown in FIG. 1. If this occurs, then P1 cannot finish its task until it gains access to R2, and P2 cannot finish its task until it gains access to R1, i.e., both P1 and P2 will enter a wait state. However, P1 is holding R1 and will not release R1 until it gains access to R2, and P2 is holding R2 and will not release R2 until it gains access to R1. At this point, the system is in a deadlock state.
Systems and mechanisms for deadlock avoidance and recovery are known in the art and described in U.S. Pat. No. 5,664,088 to Romanovsky et al. and U.S. Pat. No. 5,913,060 to Discavage.
Generally, in the prior art, once a deadlock is detected, one of the processes involved in the deadlock is terminated, so that it releases the resource it held, and the resource can be reclaimed by the system. The reclaimed resource may then be used by a waiting process. If the waiting process can finish processing using the reclaimed resource, then the system can progress past the deadlock state. The terminated process, called the victim, is generally selected on a random basis, or based on a static priority assigned to the processes.
In addition to the "circular" deadlock situation shown in
In any distributed or parallel computer system, access to shared resources is controlled by some form of a locking mechanism or scheme, whereby a shared resource is committed to, or "locked" by, a single holding processor until that processor releases the resource (i.e., releases the "lock"). To make such a system fault-tolerant, when a lock holding processor (i.e., a processor that has locked a resource) unexpectedly crashes, a rescuing method is needed to prevent system deadlock. The rescue operation will inherently be determined by the locking scheme. Problems may arise when two or more processors try to rescue the same locked resource. The correct behavior is that exactly one of them would succeed. Thus, there is a need for a generic fault-tolerant locking mechanism or scheme that can avoid deadlocks caused by failing processors in a multiprocessor system, especially mission-critical parallel systems where high availability is absolutely necessary. Such a mechanism will enable waiting processors to identify a locked resource held by a failed processor, and "rescue" the resource from the hold of its failed processor, by changing, or "re-setting," a lock associated with the resource. A single waiting processor will then rescue a resource to prevent the potential deadlock from locking up all other processors waiting for the same resource.
In a preferred embodiment, the present invention is a match-and-set lock for controlling access to a resource that is shared among a plurality of users N. The lock has a locked operating state and an unlocked operating state controlled by a value C such that the lock is in its locked operating state when C≠0 and in its unlocked operating state where C=0. The lock returns a value R, equal to the lock's current content C, to an inquiring user seeking access to the resource. A return value R=0 usually denotes that the resource is free, and a return value R≠0 denotes that the resource is locked by another user. The lock is responsive to an atomic command in the form (A, B), such that the lock substitutes B for C if A=C. Thus, the lock may be obtained/locked by issuing the command (A, B) where A=C and B≠0; and the lock may be released by issuing the command (A, B) where A=C and B=0.
In accordance with the invention, a deadlock condition may be avoided by setting the lock to the value B=P+T*(N+1). Here N is the total number of users, P is an integer within [1, N] that identifies the current user issuing the command (A, B), and T is the current global time stamp. When the lock is set in this fashion, a return value of R≠0, identifies the user currently locking the resource (via P'=R mod (N+1)) and the time when that user locked the resource (via T'=R/(N+1)). If the inquiring user determines that the user currently locking the resource has failed or restarted since locking the resource, this inquiring user can reset the lock by issuing the command (R, B). Here B=P+T*(N+1), P identifies this new user issuing the command, and T is the current time stamp; in so doing, the inquiring user gains access to the resource. If there are multiple users trying to issue this type of command (R, B) (of course, with different values of B for each inquiring user), then exactly one of them will succeed. This is because the very first of them that has succeeded will have changed the content of the lock to something different from R' causing all others to fail. When the inquiring user is finished with the resource, the user can reset the current content of the lock to C=0, to signal to other users that the resource is again free. The plurality of users may be processors in a multiprocessor computer system, and the processors may form a telecommunications system.
In another preferred embodiment, the invention is a software program implemented in a multiprocessor computer system, the software program controlling access to a shared resource from a plurality of processors N, by employing a match-and-set lock that has a current content C and that is responsive to a match-and-set command (A, B) that writes B into the current content if A=C. The lock commits the shared resource to the control of that processor that most recently set the current content to a non-zero value. The software program enables the multiprocessor system to: (a) initialize the lock by setting the current content to zero; (b) issue from one of the processors a command (M, S) where M equals an expected current content of the lock and S=P+T*(N+1) where P<(N+1) and identifies the processor issuing the command and T is a current time stamp; (c) if a non-zero value R is returned from the lock, identify the processor that set the lock as P'=R% (N+1) where R% (N+1) is the remainder of R, mod (N+1), and identify the approximate locking time as T'=R/(N+1); (d) if processor P' is still operating and was last started before time T', repeat step (b); in the alternative, (e) if processor P' is not still operating or was last started after time T', issue from processor P the command (M, S) where M=R. The users may be processors in a multiprocessor computer system, and the processors may form a telecommunications system.
In still another preferred embodiment, the invention is a method for controlling access to a shared resource from a plurality of processors N, comprising the steps of: (a) providing the resource with a locking storage function, wherein the function stores either an initial value denoting that the resource is not currently held by a processor or a value denoting a holding processor, currently holding the resource, if any, and an initial holding time when such holding began; (b) initializing the function by storing an initial value denoting that the resource is not currently held by a processor; (c) querying the function to determine if the resource is currently held by a processor; (d) if the resource is not currently held by a processor, providing to an inquiring processor access to the resource and resetting the value of the function to denote the inquiring processor's identity and a current time; in the alternative, (e) if the resource is currently held by a processor, determining if the holding processor has failed or been restarted since taking hold of the resource; (f) if the holding processor has failed or been restarted since taking hold of the resource, providing to the inquiring processor access to the resource and resetting the value of the function to denote the inquiring processor's identity and a current time; in the alternative, (g) if the holding processor has not failed or been restarted since taking hold of the resource, repeating steps (b) through (d); and (h) upon release of the resource, resetting the value stored in the function to an initial value denoting that the resource is not currently held by a processor. The plurality of processors N may form a telecommunications system.
In still another preferred embodiment, the invention is a method of controlling access to a shared resource from a plurality of processors N, the method employing a match-and-set lock that has a current content C and that is responsive to a match-and-set command (A, B) that writes B into the current content if A=C. The lock commits the shared resource to the control of that processor that most recently set the current content to a non-zero value. The method comprises the steps of: (a) initializing the lock by setting the current content to zero; (b) issuing from one of the processors a command (M, S) where M equals an expected current content of the lock and S=P+T*(N+1) where P<(N+1) and identifies the processor issuing the command and T is a current time stamp; (c) if a non-zero value R is returned from the time lock, identifying the processor that set the lock as P'=R% (N+1) where R% (N+1) is the remainder of R, mod (N+1), and identifying the approximate locking time as T'=R/(N+1); (d) if processor P' is still operating and was last started before time T', repeating step (b); in the alternative, (e) if processor P' is not still operating or was last started after time T', issuing from processor P the command (M, S) where M=R. The plurality of processors N may form a telecommunications system.
Using pseudo algorithmic language, the invention of fault-tolerant locking mechanism can be summarized as follows: here time( ) returns the current global time, N is the total number of users, my_id( ) returns the calling user's id (between 1 and N), alive(p) returns TRUE is user p is still running, and start_time(p) returns the latest reboot time of user p.
Lock: | |
M = 0; | |
S = my_id( ) + time( ) * N + 1); | |
While ((R = match-and-set(M, S)) != M) { | |
Who = R%(N + 1); | |
When = R/(N + 1); | |
If(!alive(who) ∥ start_time(who) > when) | |
M = R; | |
Else | |
M = ( ); | |
S = my_id( ) + time( ) * (N + 1); | |
Unlock: | |
match-and-set(0, S). | |
The present invention will be understood and appreciated more fully from the following detailed description, taken in conjunction with the drawings in which:
Reference is now made to
In
As shown in
The value S, which denotes the identity of processor 36 and the time when processor 36 took control of resource 30, is now stored in lock 32, as shown in FIG. 6. At this time, an inquiring processor can determine the identity (i.e., a processor number or code assigned to processor 36) of holding processor 36 from the value S, the identity being equal to S%(N+1), which is the remainder of S, mod(N+1), and the approximate time when processor 36 took control of resource 30, which is equal to S/(N+1). When processor 36 has completed its use of resource 30, it resets the value of lock 32 to zero, and resource 30 will be available for use by other potential inquiring processors 38, as shown in FIG. 7. Thus, the match-and-set lock of the present invention allows processors to rescue a resource from a "dead" processor. Moreover, because more than one processor can try to rescue the resource at the same time, it is important to note that the present invention ensures that only one such processor will succeed. The present invention is also extremely valuable in mission-critical parallel systems, such as fault tolerant telecommunications systems, where high-availability is absolutely necessary, and the failure of individual processors should not bring down the entire system.
Reference is now made to
Returning to step 86, if R≠0, the resource is currently held by another processor (i.e., another processor has already set the lock to a non-zero value). In step 88, the inquiring processor then determines the identity of the processor that set the lock, P'=R%(N+1) where R%(N+1) is the remainder of R, i.e. mod(N+1), and the time, T'=R/(N+1), when the lock was set by processor P'. In step 90, a test is made to determine if the holding processor P' has crashed or been restarted since time T' (the time when P' locked the resource). If so, the inquiring processor rescues the resource in step 92 by issuing the match-and-set command (M, S), with M=R. In step 98, the match-and-set command returns a new value R' in its response. If there is only one such processor trying to rescue this lock, then since the current content of the lock is equal to R' the match-and-set will succeed: namely R'=M in step 99. In this case, the lock is set to the value S, and the inquiring processor gains control of the resource, in step 94. Again, in step 96, when the holding processor has completed its use of the resource, it releases the resource and resets the lock content equal to zero, signaling to other inquiring processors that the resource is free. If there are more than one such processors trying to rescue this lock at the same time, only one of them will succeed: the others will all fail the test in step 99, which in turn go back to step 82 and try again from the beginning.
Returning to the test in step 90, if the holding processor P' has not crashed, or been restarted, since time T', the inquiring processor returns to step 82, and repeats the subsequent steps, as necessary, until the resource is released by its holding processor, or the inquiring processor determines that the holding processor has crashed. In the present method, a resource locked by a crashed processor in a multiprocessor system is rescued by the first processor to reset the lock with its processor ID and time stamp information, thereby eliminating the problems, and potential deadlocks, caused by multiple processors attempting to rescue the same resource at the same time.
An equivalent embodiment of the present invention is depicted in the flowchart of FIG. 9. In step 100, a "match and set" lock that controls access to a shared resource is initialized by setting the current content M of the lock equal to zero in step 102. The resource is shared by a plurality of N processors. The lock is responsive to a match-and-set command (A, B), issued by a processor, that writes B into the current content of the lock if A=C, and commits the shared resource to the control of that processor that most recently set the current content of the lock to a non-zero value. In step 102, a processor seeking access to the shared resource sets M=0 which is the expected content of a free lock. In step 103 it issues the command (M, S), and S=P+T*(N+1) where P<(N+1) and identifies the processor issuing the command and T is a current time stamp. In step 104, the lock returns a value R in response to the inquiring processor's query. In step 106, if R=M, the resource has been free, and the inquiring processor succeeds in locking the lock and gaining control over the resource in step 116. Also in step 116, the lock content has been set to a value S, which denotes the identity of the processor currently holding the lock (i.e., an identification number or code assigned to the processor in question), and the time that the holding processor gained control, as described above. In step 118, when the holding processor has completed its use of the resource, it releases the resource and resets the lock content equal to zero, signaling to other inquiring processors that the resource is free.
Returning to step 106, if R≠M, the resource is currently held by another processor (i.e., another processor has already set the lock to a non-zero value). In step 108, the inquiring processor then determines the identity of the processor that set the lock, P'=R%(N+1) where R%(N+1) is the remainder of R, i.e. mod(N+1), and the time, T'=R/(N+1), when the lock was set by processor P'. In step 110, a test is made to determine if the holding processor P' has crashed or been restarted since time T' (the time when P' locked the resource). If so, the inquiring processor sets M=R in step 114, and goes back to step 103 to try to rescue the lock, by issuing the match-and-set command (M, S), with M=R.
Returning to the test in step 110: if the holding processor P' has not crashed, or been restarted, since time T', the inquiring processor sets M=0, the expected content of a free lock, and returns to step 103 and repeats the subsequent steps, as necessary, until the resource is released by its holding processor, or the inquiring processor determines that the holding processor has crashed.
While the present invention has been described with reference to the preferred embodiments, those skilled in the art will recognize that numerous variations and modifications may be made without departing from the scope of the present invention. Accordingly, it should be clearly understood that the embodiments of the invention described above are not intended as limitations on the scope of the invention, which is defined only by the following claims.
Patent | Priority | Assignee | Title |
7571354, | May 09 2003 | Oracle America, Inc | System and method for request routing |
7573810, | Dec 14 2005 | International Business Machines Corporation | Avoiding deadlocks in performing failovers in communications environments |
8190857, | Sep 26 2007 | Hewlett Packard Enterprise Development LP | Deleting a shared resource node after reserving its identifier in delete pending queue until deletion condition is met to allow continued access for currently accessing processor |
9348659, | Dec 04 2009 | Microsoft Technology Licensing, LLC | Lock resolution for distributed durable instances |
9459931, | Jan 06 2014 | International Business Machines Corporation | Administering a lock for resources in a distributed computing environment |
9459932, | Jan 06 2014 | International Business Machines Corporation | Administering a lock for resources in a distributed computing environment |
Patent | Priority | Assignee | Title |
5623670, | Feb 17 1995 | THE CHASE MANHATTAN BANK, AS COLLATERAL AGENT | Method and apparatus for crash safe enforcement of mutually exclusive access to shared resources in a multitasking computer system |
5664088, | Sep 12 1995 | THE CHASE MANHATTAN BANK, AS COLLATERAL AGENT | Method for deadlock recovery using consistent global checkpoints |
5742785, | Dec 18 1992 | International Business Machines Corporation | Posting multiple reservations with a conditional store atomic operations in a multiprocessing environment |
5872981, | May 30 1997 | Oracle International Corporation | Method for managing termination of a lock-holding process using a waiting lock |
6301676, | Jan 22 1999 | Oracle America, Inc | Robust and recoverable interprocess locks |
6374362, | Jan 14 1998 | NEC Corporation | Device and method for shared process control |
6539446, | May 07 1999 | Oracle International Corporation | Resource locking approach |
6553512, | Feb 16 2000 | Hewlett Packard Enterprise Development LP | Method and apparatus for resolving CPU deadlocks |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Oct 23 2000 | WU, YUGUANG | AT&T Corp | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 011281 | /0107 | |
Nov 07 2000 | AT&T Corp. | (assignment on the face of the patent) | / | |||
May 22 2009 | AT&T Corp | AT&T Properties, LLC | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 022856 | /0780 | |
May 22 2009 | AT&T Properties, LLC | AT&T INTELLECTUAL PROPERTY II, L P | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 022928 | /0276 |
Date | Maintenance Fee Events |
Sep 14 2007 | M1551: Payment of Maintenance Fee, 4th Year, Large Entity. |
Jan 09 2012 | REM: Maintenance Fee Reminder Mailed. |
May 25 2012 | EXP: Patent Expired for Failure to Pay Maintenance Fees. |
Date | Maintenance Schedule |
May 25 2007 | 4 years fee payment window open |
Nov 25 2007 | 6 months grace period start (w surcharge) |
May 25 2008 | patent expiry (for year 4) |
May 25 2010 | 2 years to revive unintentionally abandoned end. (for year 4) |
May 25 2011 | 8 years fee payment window open |
Nov 25 2011 | 6 months grace period start (w surcharge) |
May 25 2012 | patent expiry (for year 8) |
May 25 2014 | 2 years to revive unintentionally abandoned end. (for year 8) |
May 25 2015 | 12 years fee payment window open |
Nov 25 2015 | 6 months grace period start (w surcharge) |
May 25 2016 | patent expiry (for year 12) |
May 25 2018 | 2 years to revive unintentionally abandoned end. (for year 12) |