A dual host system and method with back to back non-transparent bridges and a proxy packet generating mechanism. The proxy packet generating mechanism enables the hosts to send interrupt generating packets to each other.
|
10. A method comprising:
setting an assert change in state in a shadow register of a first non-transparent bridge;
creating a packet including a payload of a posted memory write; and
sending the packet to a second non-transparent bridge,
wherein at least a portion of the second non-transparent bridge is inaccessible to a first host including the first non-transparent bridge and at least a portion of the first non-transparent bridge is inaccessible to a second host including the second non-transparent bridge.
13. A method comprising:
receiving a packet including a payload of a posted memory write at a first non-transparent bridge comprising a shadow register from a second non-transparent bridge;
decoding the posted memory write; and
updating the shadow register,
wherein at least a portion of the second non-transparent bridge is inaccessible to a first host including the first non-transparent bridge and at least a portion of the first non-transparent bridge is inaccessible to a second host including the second non-transparent bridge.
1. A device comprising:
a first non-transparent bridge comprising a shadow register, wherein the first non-transparent bridge is adapted to create a packet including a payload of a posted memory write and send the packet to a second non-transparent bridge based on an assert change in state in the shadow register, wherein at least a portion of the second non-transparent bridge is inaccessible to the device and at least a portion of the first non-transparent bridge is inaccessible to a second device including the second non-transparent bridge.
6. A device comprising:
a first non-transparent bridge comprising a shadow register, wherein the first non-transparent bridge is adapted to receive a packet including a payload of a posted memory write from a second non-transparent bridge, decode the posted memory write as its own, and update the shadow register, wherein at least a portion of the second non-transparent bridge is inaccessible to the device and at least a portion of the first non-transparent bridge is inaccessible to a second device including the second non-transparent bridge.
15. A system comprising:
a first host comprising a first non-transparent bridge, wherein the first non-transparent bridge is adapted to create a packet including a payload of a posted memory write based on an assert change in state in a first shadow register; and
a second host comprising a second non-transparent bridge,
wherein at least a portion of the second non-transparent bridge is inaccessible to the first host and at least a portion of the first non-transparent bridge is inaccessible to the second host,
wherein the first non-transparent bridge is further configured to send the packet to the second non-transparent bridge, and
wherein the second non-transparent bridge is adapted to decode the posted memory write as its own, update a second shadow register, detect a change in state of the second shadow register, and generate an upstream interrupt.
2. The device of
3. The device of
4. The device of
5. The device of
7. The device of
8. The device of
9. The device of
11. The method of
12. The method of
14. The method of
16. The system of
17. The system of
|
This application is a continuation of U.S. application Ser. No. 13/570,889, filed Aug. 9, 2012, which is a continuation of U.S. application Ser. No. 12/238,427, filed Sep. 25, 2008, now U.S. Pat. No. 8,244,951, the entire contents of all of the foregoing applications are incorporated herein by reference in their entireties.
Embodiments of the invention are generally related to distributed multi-processor architecture and in particular to dual host systems with back-to-back non-transparent bridges.
Peripheral Component Interconnect (PCI) is a second generation parallel bus architecture developed in 1992 as a replacement for the Industry Standard Architecture (ISA) bus. In the PCI standard, all the devices share the same bidirectional, 32-bit (or 64-bit), parallel signal path. The PCI bus brought a number of advantages over the ISA bus, including processor independence, buffered isolation, bus mastering, and true plug-and-play operation. PCI Express (PCIe) is a third generation general-purpose serial I/O interconnect designed to replace the PCI bus. Rather than being a bus, PCI Express is structured around point-to-point serial links called lanes.
The point-to-point serial link architecture of PCI Express is well suited for distributed processing via a distributed multiprocessor architecture model. Distributed processors are generally optimized to implement data packet processing functions. Unlike general-purpose central processing units (CPUs) that rely heavily on caching for improving performance, distributed processors have a lack of locality in packet processing and need for high-performance I/O that has pushed designers to come up with innovative architectures to reduce processing latency while still processing packets at high data rates.
The invention may best be understood by referring to the following description and accompanying drawings that are used to illustrate embodiments of the invention.
As used in the specification and claims, the singular forms “a”, “an” and “the” include plural references unless the context clearly dictates otherwise. A “protocol” is a set of rules governing the format, syntax and order, of messages that are exchanged between at least two NTB subsystems. A “root complex” (“RC”) is a PCI Express device that connects a processor (or central processing unit (CPU)) and memory to the PCI Express switch fabric. The root complex generates transaction requests on behalf of the processor. A “doorbell register” is a generic, typically 16 bit register that can be used by the controlling software of two NTB subsystems to define a type of protocol. The doorbell serves as an interrupt generating mechanism to alert a one of the hosts of a dual host system to the actions of the opposite host. “Scratchpad Registers” are a generic set of typically 32 bit registers that can be used for cross-system communication and the storage of data related to those communications.
A PCI Express non-transparent bridge (NIB) facilitates expansion of the distributed multiprocessor architecture model by functioning as a gateway between dual host systems.
The primary host system 100A and a backup host system 100B typically communicate with each other via a protocol through a doorbell register (not shown) and a block of scratchpad registers (not shown) in MMI/O space 110 on the single non-transparent bridge 104. The primary and secondary host systems 100A, 100B use software, usually a device driver, to send messages to each other. One type of message could be a heartbeat message, which indicates the well-being of the software running on the host systems 100A, 100B. That is, the failure to receive a heartbeat in one of the host systems 100A, 100B would indicate catastrophic failure on the opposite host system 100A, 100B. Typically, a heartbeat message is delivered by a device driver that writes the heartbeat protocol message to the scratchpad register(s) and then writes to the doorbell register causing an interrupt to be delivered to the other host. That is, the interrupt notifies the host system 100A, 100B of the availability of the heartbeat message. In addition to simple heartbeat messages, the protocol may include other types of messages such as: implementing failover, moving windows, loading software, etc.
The configurations of the non-transparent bridges 104A, 104B of the dual host system of
The back-to-back configuration of the first and second non-transparent bridges 104A, 104B eliminates bus enumeration problems that occur when the primary and backup systems 100A, 100B do not boot at the same time. Complexity, however, may arise for software when the non-transparent bridges 104A, 104B are placed back to back. Each host system's 100A, 100B enumeration software stops at the non-transparent bridge endpoint, resulting in an inaccessible secondary space created between the secondary sides 108A, 108B of the two host systems 100A, 100B. Therefore, the primary host system 100A has no knowledge of the secondary host system's 100B non-transparent bridge doorbell 114B and scratchpad 116B. The reverse is also true. While data can be moved between the primary and secondary host systems 100A, 100B through the PCIe inaccessible secondary spaces, interrupts cannot travel from one system to the other.
Embodiments the invention solve this problem by creating a proxy packet generating mechanism 120 between the secondary sides 108A, 108B of the two host systems 100A, 100B. The proxy packet generating mechanism 120 enables one host system 100A, 100B to send a PCIe interrupt generating packet through the MMI/O− inaccessible area into the opposite non-transparent bridge 104A, 104B, effectively giving each host system 100A, 100B a tunnel into the other systems MMI/O spaces 100A, 100B. The interrupt can be generated by accessing and writing the scratchpad registers 116A, 116B and the interrupt generating doorbell registers 114A, 114B.
One method according to one embodiment of the invention can be explained with the help of
In this embodiment of the invention, a heartbeat protocol is communicated from the primary host system 100A to the backup host system 100B. A typical heartbeat protocol allows primary host system 100A to inform backup host system B that it is “alive and well,” In other words, a healthy link exists between them. Although the following example illustrates a heartbeat from the primary host system 100A to the backup host system 100B, the heartbeat protocol can be bi-directional. That is, the heartbeat protocol can be configured to send and receive heartbeat messages to and from the primary host system 100A and the backup host system 100B.
In the first step of the method, the primary host system 100A sets one or more agreed upon bit(s) in the first doorbell register 114A. When the hardware on the primary host 100A senses that the first doorbell 114A has been set, it creates a posted memory write (an interrupt proxy packet). The first non-transparent bridge 104A then sends the posted memory write across the inaccessible secondary space to the second non-transparent bridge 104B on backup host system 100B. The transaction is received by the secondary side 106B of the second non-transparent bridge 104B through the SB01BASE window of the second non-transparent bridge 104B in host system 100B.
The hardware in the second non-transparent bridge 104B in the backup host system 100B then decodes the posted memory write as its own and sets the equivalent bits in the primary doorbell register 114B in the second non-transparent bridge 104B in the backup host system 100B. On seeing the primary doorbell register 114B being set, the second non-transparent bridge 104B generates an upstream interrupt based on whether INTx or MSI or MS1-X is enabled and not masked. This interrupt could be set, for example, at a predetermined periodic rate such as every 1 second.
Offload Protocol
In an offload protocol, the primary host system 100A wishes to off load some packet processing to the backup host system 100B. Broadly, the offload protocol comprises three main steps which each typically comprise multiple substeps. The main steps comprise (1) sending the packets to be worked on from the primary host system 100A to the backup host system 100B, (2) sending a message which includes various information about the packets such as where the packets have been loaded into memory, length, type of work to be done, etc. from the primary host system 100A to the backup host system 100B, and (3) sending from the primary host system 100A to the backup host system 100B an interrupt proxy packet to tell the backup system host 100B that there is work to be done.
Send Packets to Work on
First, the Primary host system 100A writes the off-load packets to the primary BAR ⅔ window of the first non-transparent bridge in primary host system 100A. The first non-transparent bridge 104A then translates the packets to an agreed upon base address PBAR2XLAT and sends the packets. The second non-transparent bridge 104B in the backup host system 100B receives the packets at the same base address SB23BASE (PBAR2XLAT=SB23BASE). The packets are then translated into the backup host system's 100B domain using SBAR2XLAT and sent to system memory.
Send Message Detailing where Transactions have been Loaded into Memory, Length, Type of Work to be Done, Etc.
The primary host system 100A writes an agreed upon message into the first scratchpad registers 116A in the first non-transparent bridge 104A in the primary host system 100A. The hardware in the non-transparent bridge 104A, upon sensing a write to the scratchpad registers 116A, schedules a posted memory write targeting the scratchpad register 116B in backup host system 100B, via back to back BAR0XLAT. The message is received on the secondary side of the second non-transparent bridge 104B on the opposite side of the inaccessible secondary space through the SB01BASE window of the second non-transparent bridge 104B in backup host system 100B. Hardware in the backup host system's 100B non-transparent bridge 104B decodes the posted memory write as its own and sets the desired scratchpad register bit(s) 116B in the non-transparent bridge 104B in the backup host system 100B.
Send Interrupt Proxy Packet to Tell Backup Host System 100B that there is Work to be Done
The primary host system 100A sets a selected bit in the back to back doorbell register 116A. Hardware on primary host system 100A senses that the back to back doorbell has been set and creates a posted memory write and sends it across the inaccessible secondary space to the second non-transparent bridge 104B on backup host system 100B. The posted memory write is received by the secondary side 108B of the non-transparent bridge 104B through the SB01BASE window of the non-transparent bridge 104B in the backup host system 100B. The hardware in the second non-transparent bridge 104B decodes the posted memory write as its own and sets the equivalent bits in the primary doorbell register 114B, The hardware in the second non-transparent bridge 104B, upon seeing the bit(s) the primary doorbell 104B being set, generates an upstream interrupt based on whether INTx or MSI or MSI-X is enabled and not masked. The interrupt service routine in the backup host system 100BB decodes the interrupt and reads the message in the scratchpad 116B in the non-transparent bridge 104B on backup host system 100B. This message contains information such as the location and length of the transactions to work on and any may include any other relevant information needed to handle the request. Optionally, an acknowledge packet could be sent back to primary host system 100A when this is completed.
The various embodiments of the invention described herein simplify many problems for software in dual host systems having back to back non-transparent bridges. Absent a proxy packet generating mechanism 120, the solution around the tunnel problem would involve a highly proprietary software solution duplicating the original functionality and intent of doorbell 114A, 114B and scratchpad register 116A, 116B sets in each system's memory. A complex protocol would be required for driver to driver communications. The result would be increased complexity, asynchronous communication problems, reduced software reusability and scalability, and longer debug cycles. With the proxy packet generating mechanism 120, however, the device driver developer is able to more easily design a protocol by which heartbeat messages and scratchpad data can be communicated across the back to back non transparent bridges 104A, 104B. Example protocol messages for which this is advantageous include, but are not limited to, host fail-over applications, journaling, checkpoint data, and offload computations such as cryptography.
While the invention has been described in tem1s of several embodiments of the invention, those of ordinary skill in the art will recognize that the invention is not limited to the embodiments of the invention described, but can be practiced with modification and alteration within the spirit and scope of the appended claims. The description is thus to be regarded as illustrative instead of limiting.
Mitchell, James, Sullivan, Mark, Davis, Kimberly, Themins, Patrick
Patent | Priority | Assignee | Title |
10817447, | Nov 14 2016 | Intel Corporation | Input/output translation lookaside buffer (IOTLB) quality of service (QoS) |
Patent | Priority | Assignee | Title |
4954988, | Oct 28 1988 | Rockwell International Corporation | Memory device wherein a shadow register corresponds to each memory cell |
6006301, | Sep 30 1997 | Intel Corporation | Multi-delivery scheme interrupt router |
6175888, | Apr 10 1996 | International Business Machines Corporation | Dual host bridge with peer to peer support |
6621293, | May 31 2001 | TELEFONAKTIEBOLAGET L M ERICSSON PUBL | Integrated circuit arrangement with feature control |
6622191, | Nov 24 1999 | Kabushiki Kaisha Toshiba | Computer system |
6728818, | Jun 27 2001 | GOOGLE LLC | Dual storage adapters utilizing clustered adapters supporting fast write caches |
7234004, | Dec 19 2003 | International Business Machines Corporation | Method, apparatus and program product for low latency I/O adapter queuing in a computer system |
7293129, | Apr 22 2005 | Sun Microsystems, Inc. | Flexible routing and addressing |
7295543, | Jul 28 2003 | Sony Corporation | Network interconnection apparatus, network interconnection method, name resolution apparatus and computer program |
7657663, | Dec 19 2006 | International Business Machines Corporation | Migrating stateless virtual functions from one virtual plane to another |
7676625, | Aug 23 2006 | Oracle America, Inc | Cross-coupled peripheral component interconnect express switch |
7694025, | Mar 31 2006 | MICROSEMI STORAGE SOLUTIONS US , INC | Method and device for base address sorting and entry into base address registers |
20030126348, | |||
20080225871, |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Sep 10 2008 | DAVIS, KIMBERLY | Intel Corporation | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 036296 | /0204 | |
Sep 10 2008 | SULLIVAN, MARK | Intel Corporation | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 036296 | /0204 | |
Sep 11 2008 | MITCHELL, JAMES | Intel Corporation | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 036296 | /0204 | |
Sep 11 2008 | THEMINS, PATRICK | Intel Corporation | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 036296 | /0204 | |
Mar 27 2015 | Intel Corporation | (assignment on the face of the patent) | / |
Date | Maintenance Fee Events |
Sep 03 2020 | M1551: Payment of Maintenance Fee, 4th Year, Large Entity. |
Date | Maintenance Schedule |
Mar 14 2020 | 4 years fee payment window open |
Sep 14 2020 | 6 months grace period start (w surcharge) |
Mar 14 2021 | patent expiry (for year 4) |
Mar 14 2023 | 2 years to revive unintentionally abandoned end. (for year 4) |
Mar 14 2024 | 8 years fee payment window open |
Sep 14 2024 | 6 months grace period start (w surcharge) |
Mar 14 2025 | patent expiry (for year 8) |
Mar 14 2027 | 2 years to revive unintentionally abandoned end. (for year 8) |
Mar 14 2028 | 12 years fee payment window open |
Sep 14 2028 | 6 months grace period start (w surcharge) |
Mar 14 2029 | patent expiry (for year 12) |
Mar 14 2031 | 2 years to revive unintentionally abandoned end. (for year 12) |