A system and method for adaptive bus configuration operable to respond to hardware changes and other configuration changes is disclosed. In an embodiment, the computing system includes a circuit assembly having at least one processing resource coupled to a respective set of bus traces, at least one peripheral device socket coupled to a respective set of bus traces, and a bus switch coupled to the bus traces of the processing resource and the bus traces of the peripheral device. The bus switch implements a set of connections between the bus traces of the processing resource and the bus traces of the peripheral device sockets according to an instruction. The instruction may specify an allocation of peripheral device sockets to processing resources based on the number of installed processing resources so that no peripheral device is connected to a bus without an attached processor.
|
7. A method comprising:
determining, by a computing device, a first number of one or more installed processors of the computing device by polling one or more processor sockets of the one or more installed processors and identifying a set of peripheral device sockets associated with the one or more installed processors;
identifying, by the computing device, a maximum number of buses supported by the computing device based on the determined first number of the one or more installed processors;
determining, by the computing device, one or more bus performance attributes of the computing device based on the identified maximum number of buses;
allocating, by the computing device, a set of peripheral device sockets of one or more peripheral devices to the one or more installed processors based on the determined one or more bus performance attributes such that each socket of the set of peripheral device sockets of the one or more peripheral devices is communicatively coupled to at least one of the one or more installed processors and each of the one or more peripheral devices is allocated at least a minimum amount of an available communication resource;
generating, by the computing device, a switch instruction for implementing the allocation; and
providing, by the computing device, the switch instruction to a switch of the computing system for coupling the set of peripheral device sockets of the one or more peripheral devices to the one or more installed processors according to the allocation.
1. A non-transitory machine readable medium having stored thereon instructions for performing a method comprising machine executable code which when executed by at least one machine, causes the machine to:
determine a first number of one or more installed processors of a computing device by polling one or more processor sockets of the one or more installed processors and identify a set of peripheral device sockets associated with the one or more installed processors;
identify a maximum number of buses supported by the computing device based on the determined first number of the one or more installed processors;
determine one or more bus performance attributes of the computing device based on the identified maximum number of buses;
allocate a set of peripheral device sockets of one or more peripheral devices to the one or more installed processors based on the determined one or more bus performance attributes such that each socket of the set of peripheral device sockets of the one or more peripheral devices is communicatively coupled to at least one of the one or more installed processors and each of the one or more peripheral devices is allocated at least a minimum amount of an available communication resource;
generate a switch instruction for implementing the allocation; and
provide the switch instruction to a switch of the computing system for coupling the set of peripheral device sockets of the one or more peripheral devices to the one or more installed processors according to the allocation.
13. A computing device comprising:
a memory containing a machine readable medium comprising machine executable code having stored thereon instructions for performing a method of providing a configuration file that configures a switch to implement device allocation; and
a processor coupled to the memory, the processor configured to execute the machine executable code to cause the processor to:
determine a first number of one or more installed processors of a computing device by polling one or more processor sockets of the one or more installed processors and identify a set of peripheral device sockets associated with the one or more installed processors;
identify a maximum number of buses supported by the computing device based on the determined first number of the one or more installed processors;
determine one or more bus performance attributes of the computing device based on the identified maximum number of buses;
allocate a set of peripheral device sockets of one or more peripheral devices to the one or more installed processors based on the determined one or more bus performance attributes such that each socket of the set of peripheral device sockets of the one or more peripheral devices is communicatively coupled to at least one of the one or more installed processors and each of the one or more peripheral devices is allocated at least a minimum amount of an available communication resource;
generate a switch instruction for implementing the allocation; and
provide the switch instruction to a switch of the computing system for coupling the set of peripheral device sockets of the one or more peripheral devices to the one or more installed processors according to the allocation.
2. The non-transitory machine readable medium as set forth in
identify one or more peripheral device types of the same type for one or more peripheral devices installed in the computing system; and
allocate the set of peripheral device sockets of the one or more peripheral devices equally further based on the one or more peripheral devices to be identified as having the same device type, wherein the one or more peripheral device types comprises graphics processing unit, networking controller, peripheral controller or storage interface controller and wherein the one or more bus performance attributes comprises bandwidth, latency, communication medium, number of conductors or overhead.
3. The non-transitory machine readable medium as set forth in
generate a configuration file based on the first number of one or more installed processors, wherein the configuration file includes the switch instruction for implementing the allocation.
4. The non-transitory machine readable medium as set forth in
provide the configuration file to the switch of the computing system and coupling the set of peripheral device sockets of the one or more peripheral devices to the one or more installed processors according to the allocation.
5. The non-transitory machine readable medium as set forth in
6. The non-transitory machine readable medium as set forth in
provide the switch instruction to the switch for coupling the set of peripheral device sockets of the one or more peripheral devices to the one or more installed processors further based on a minimum number of PCI Express lanes.
8. The method of
identifying one or more peripheral device types of the same type for one or more peripheral devices installed in the computing system; and
allocating based on the one or more peripheral devices to be identified as having the same device type, wherein the one or more peripheral device types comprises graphics processing unit, networking controller, peripheral controller or storage interface controller and wherein the one or more bus performance attributes comprises bandwidth, latency, communication medium, number of conductors or overhead.
9. The method of
10. The method of
11. The method of
12. The method of
14. The device of
identify one or more peripheral device types of the same type for one or more peripheral devices installed in the computing system; and
allocate the set of peripheral device sockets of the one or more peripheral devices equally further based on the one or more peripheral devices to be identified as having the same device type, wherein the one or more peripheral device types comprises graphics processing unit, networking controller, peripheral controller or storage interface controller and wherein the one or more bus performance attributes comprises bandwidth, latency, communication medium, number of conductors or overhead.
15. The device of
16. The device of
17. The device of
18. The device of
|
The present disclosure relates generally to computing systems and, more particularly, to a printed circuit board for use in a computing system that incorporates programmable bus switches in order to support different configurations of installed devices.
While strong demand for storage and processing has translated into steady sales of ever-more-powerful computing systems, customers have also become increasingly cost sensitive. Customers won't pay for what they don't need, and vendors have responded accordingly. As merely one example, enterprise-class storage systems are offered in a wide array of storage, processing, and networking configurations. Many customers also request systems that are upgradable over time in order to preserve some of the substantial investment. However, tension arises when customers seek customized hardware and software solutions as a cost-saving measure, in part because custom solutions are rarely cheaper to provide. Both vendors and customers carefully balance flexibility against system complexity, which tends to increase support costs and reduce system reliability.
In particular, vendors have sought opportunities to use a single component (be it a code fragment, processor, controller, storage device, etc.) in a wide array of products. This, reusability may allow the vendor to leverage economy of scale. In the simplest cases, a component can be used in a variety of situations with little or no modification. However, this is not often the case for printed circuit boards (PCBs) and other circuit assemblies. PCBs contain a large number of devices (resistors, capacitors, power circuitry, etc.) and device sockets connected by conductive traces crossing multiple insulator layers. In order to provide optimum performance, the devices and the traces are carefully laid out based on the installed components. Thus, reusability may take a backseat to reducing trace length and noise. Compounding the problem, many common protocols used to communicate at the PCB level require direct point-to-point connections rather than more flexible topologies. Accordingly, for these reasons and others, it would be beneficial for PCB designs to have the flexibility to support a wider array of hardware configurations in order to provide more cost-effective solutions.
In the following description, specific details are set forth describing some embodiments consistent with the present disclosure. It will be apparent, however, to one skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are meant to be illustrative but not limiting. One skilled in the art may realize other elements that, although not specifically described here, are within the scope and the spirit of this disclosure. In addition, to avoid unnecessary repetition, one or more features shown and described in association with one embodiment may be incorporated into other embodiments unless specifically described otherwise or if the one or more features would make an embodiment non-functional.
Various embodiments of the present disclosure provide a circuit assembly that can be easily reconfigured based on the installed hardware. In one example, the circuit assembly includes a number of reconfigurable Peripheral Component Interconnect Express (PCI Express) bus switches that can be programmed to connect PCI Express peripherals to one of various processors based on which of the processors are installed. Because PCI Express is a point-to-point protocol, connecting a PCI Express peripheral to an empty processor socket may leave the peripheral inaccessible. To avoid this, an exemplary PCI Express switch is programmed to connect peripherals only to those processors that are installed. Because of the switching capability, the same circuit assembly can be used with varying numbers of processors.
The connections made by the switches can also be tailored to the peripheral devices installed in the system. In one example, the switch or switches are programmed to ensure that redundant peripheral devices are not connected to common hardware such as a common bus or a common processor to avoid a single point of failure disabling all of the redundant devices. In some examples, the switches ensure that each connected peripheral device has some minimum amount of a communication resource (e.g., a minimum number of PCI Express lanes). After the minimum has been met, any leftover amount of the communication resource may be allocated according to priorities assigned to the particular peripherals. The switches may also be programmed to account for certain connections that are fixed and should not be changed. The flexibility provided by the programmable switches allows the circuit assembly to be used throughout multiple product lines without substantial changes to the underlying hardware.
Referring first to
The computing system 100 will now be described in more detail and includes one or more processing resources 104, of which two (104A and 104B) are shown. Processing resources 104A and 104B may each include one or more microcontrollers, Central Processing Units (CPUs), Application-Specific Integrated Circuits (ASICs), Programmable Logic Devices (PLDs), and/or other suitable processing resources operable to perform programmed computing instructions. In some embodiments, processing resources 104A and 104B share a single package, such as discrete cores of the same processor. In some embodiments, processing resources 104A and 104B are discrete devices. For example, processing resource 104 may be a first processor chip and processing resource 104 may be a second processor chip. In further examples, processing resources 104A and 104B each include multiple devices and may be multi-processor clusters. The processing resources 104 may be coupled to the remainder of the computing system 100 via one or more processor sockets 106. Sockets 106 include any type of slot, connector, or connector array that provides a connection between an installed component and the system 100 and may include a mechanism for easily coupling and uncoupling the installed component. The computing system 100 may include empty sockets 106 and may function without every socket 106 populated.
In the illustrated embodiment, the processing resources 104A and 104B communicate with each other via an inter-processor bus 108 such as a Quick Path Interconnect (QPI) bus from Intel Corporation, of Santa Clara, Calif. To support this bus 108, each processing resource 104 may include an inter-processor bus controller 110 operable to send and receive data transactions across the bus 108. Processing resources 104A and 104B may also include a memory controller 112 coupled to one or more pools of memory 114, such as random access memory (RAM). A pool of memory 114 may be dedicated to a particular processing resource 104 or shared between them.
In addition to the processing resources 104, the exemplary computing system 100 also includes a number of peripheral devices 116 (including peripheral device 116A) in communication with the processing resources 104. Similar to the processing resources 104, the peripheral devices 116 may be coupled to the remainder of the computing system 100 via one or more peripheral sockets 118. Sockets 118 include any type of slot, connector, or connector array that provides a connection between an installed component and the system 100 and may include a mechanism for easily coupling and uncoupling the installed component. The computing system 100 may include empty sockets 118 and may function without every socket 118 populated.
The peripheral devices 116 may include any type of computing element. For example, a peripheral device 116 may include a graphics processing unit (GPU) or other co-processor. In some examples, a peripheral device 116 includes a networking controller such as an Ethernet controller, a wireless (IEEE 802.11 or other suitable standard) controller, or any other suitable wired or wireless communication controller. In some examples, a peripheral device 116 includes a storage interface controller such as a Serial Attached SCSI (SAS), SATA, iSCSI, Fibre Channel, Fibre Channel over Ethernet (FCoE), eSATA, and/or PATA controller. In some examples, the peripheral devices 116 include peripheral controllers such as USB controllers and/or FireWire controllers. Some storage devices (e.g., solid-state drives (SSDs) and/or hard disk drives (HDDs)) connect directly to a bus and are suitable peripheral devices 116.
In the illustrated embodiment, the peripheral devices 116 are coupled to the processing resources 104A and 104B by two different buses, bus 102A and bus 102B, although the computing system 100 may include many more buses. Accordingly, processing resources 104A and 104B may each include a peripheral bus controller 114 for use in communicating with the respective bus. In embodiments where buses 102A and 102B are PCI Express buses, the respective peripheral bus controller 114 may be referred to as a root complex.
Bus 102A and bus 102B are separate and independent such that the peripheral devices 116 and processing resource 104 coupled to bus 102A do not directly communicate with those of bus 102B. In one typical example, bus 102A and bus 102B are PCI Express buses, although further embodiments incorporate other bus types. PCI Express is a serial point-to-point interconnect that connects two devices via a link. PCI Express links are asymmetrical in that an upstream device, sometimes referred to as a root device or root complex (e.g., processing resource 104A or 104B) initiates communication with a downstream device, referred to as an endpoint (e.g., peripheral devices 116). While a root complex may be coupled to multiple downstream devices, PCI Express may not support a downstream device coupled to more than one root complex. Accordingly, in the illustrated embodiment, buses 102A and 102B are separate and independent with one bus per root complex (and in turn one bus per processing resource 104). Of course, it is understood that this is only one of many examples of independent buses within a storage system 100.
Because of the separate and independent nature of the buses, if the root device (e.g., a processing resource 104) fails or is not installed, the peripheral devices 116 on the bus may be unreachable. For example, if processing resource 104B were not installed, the associated peripheral devices 116, including peripheral device 116A, may not be accessible to the remaining processing resource 104A. Thus, it would be beneficial if the connections of bus 102A and bus 102B could be changed based on the presence of the processing resource 104B so that the same components could be used in both system configurations.
The computing system 200 also includes one or more bus switches 202. Switch 202 is operable to connect the processing resources 104A and 104B to the peripheral devices 116 and includes upstream ports connected to the processing resources 104 and downstream ports connected to the peripheral devices 116. The number of ports has been reduced for clarity, although the switch 202 may include any combination of upstream ports and downstream ports. In some examples, switch 202 is a packet-based switch where a field of a data packet determines its destination. One advantage of packet-based switching is that many types of packet switches can route data from any source to any destination regardless of which bus or buses the devices are on. However, some communication protocols (e.g., PCI Express) do not support packet-switching, opting to trade flexibility for improved data throughput. In these cases, the switch 202 may create one or more separate and independent point-to-point buses coupling peripheral devices 116 to a processing resource 104. The assignment of a peripheral device 116 to a bus and thereby assigning it to a processing resource 104 may be carried out by programming the switch 202 to connect the various elements. The selected assignment may be based on which processing resources 104 are installed and active, which peripheral devices 116 are installed, performance considerations, peripheral device priorities, and/or other system considerations.
An example of this allocation process is described with reference to
Referring to block 302 of
The computing system 200 may determine for itself how many bus-limiting devices are installed. However, there are also advantages to assessing the installed bus-limiting devices before the computing system 200 is operational or even before the system 200 is fully assembled. Thus, in various embodiments, the determination is made in part by a computing system other than computing system 200, such as a manufacturing system 204 used in the design and/or assembly of the computing system 200. Employing a separate system such as the manufacturing system 204 may avoid the need to add components to the computing system 200 to support a partial boot of the system 200. Similarly, in some embodiments, the determination is made in part by a technician. This may be useful when components (e.g., the switch 202) cannot be polled electronically.
Referring to block 304 of
Referring to block 306 of
Referring to block 308 of
An exemplary intermediate bus configuration file 400 is shown in
Referring to block 310 of
Referring to block 312 of
Various exemplary embodiments will now be described. While any of the considerations described in the various exemplary embodiments may be combined with any other considerations, in the interest of brevity, only a limited number of combinations will be described. In one exemplary embodiment, the allocation is determined so that each peripheral device 116 is connected to at least one processing resource 104 by at least one bus. In one exemplary embodiment, the allocation is determined so that each peripheral device 116 is connected to only one processing resource 104 by only one bus. In one exemplary embodiment, the allocation is determined so that peripheral devices 116 of similar types are divided equally between the processing resources 104. In one exemplary embodiment, the allocation is determined so that redundant peripheral devices 116 are not assigned to a common bus, switch 202, and/or processing resource 104. In one exemplary embodiment, the allocation is determined so that each processing resource 104 is coupled to at least one peripheral device 116 of a given type (e.g., GPUs, storage controllers, storage devices, etc.). In one exemplary embodiment, the allocation is determined so that each peripheral device 116 is allocated at least a minimum amount of an available communication resource (e.g., at least a x1 link, at least a x2 link, at least a x4 link, etc.). In one exemplary embodiment, the allocation is determined so that high priority peripheral devices 116 (e.g., GPUs, storage controllers, storage devices, etc.) are allocated the remainder of an available communication resource after a minimum has been satisfied among the peripheral devices 116. In one exemplary embodiment, the allocation is determined so that a particular peripheral device 116 is the only peripheral device 116 coupled to a particular bus. In one exemplary embodiment, the allocation is determined to account for certain connections, buses, and/or links that are fixed and cannot be changed. In one exemplary embodiment, the allocation is determined so that communication resources are directed away peripheral sockets 118 that do not have an attached peripheral device 116 and redistributed to populated sockets 118.
Referring to block 314 of
An exemplary implementation of the computing system 200 will now be described. Whereas
The circuit board assembly 600 includes one or more processing resources 104 (e.g., processing resources 104A and 104B) coupled to the circuit board assembly 600. In some embodiments, the processing resources 104 are inserted into a processor socket 106 that is soldered or otherwise affixed to the circuit board assembly 600. In alternate embodiments, the processing resources are soldered or otherwise affixed to the circuit board assembly 600 directly. The circuit board assembly 600 may also include memory and/or memory sockets disposed thereupon and may include traces 606 coupling the memory to one or more of the processing resources.
Circuit board assembly 600 may also include one or more peripheral devices 116 and/or peripheral sockets 118 for coupling peripheral devices 116. Referring to
Embodiments of the present disclosure can take the form of a computer program product accessible from a tangible computer-usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system. For the purposes of this description, a tangible computer-usable or computer-readable medium can be any apparatus that can store the program for use by or in connection with the instruction execution system, apparatus, or device. The medium can be an electronic, magnetic, optical, electromagnetic, infrared, or a semiconductor system (or apparatus or device). In some embodiments, one or more processors 110 of the storage system 102 execute code to implement the actions described above.
Accordingly, a system and method for adaptive bus configuration that accounts for installed hardware is provided. In some exemplary embodiments, the computing system comprises a circuit assembly having coupled to and disposed thereupon: at least one processing resource, at least one peripheral device socket, and a bus switch. Each of the at least one processing resource is coupled to a respective set of bus traces, and each of the at least one peripheral device socket is coupled to a respective set of bus traces. Accordingly, the bus switch is coupled to each set of bus traces of the at least one processing resource and to each set of bus traces of the at least one peripheral device socket. The bus switch is operable to receive an instruction and, based on the instruction, to implement a set of connections between each set of bus traces of the at least one processing resource and each set of bus traces of the at least one peripheral device socket. In one such embodiment, the bus switch is operable to communicatively couple the at least one peripheral device socket to the at least one processing resource such that each peripheral device of the computing system is communication with at least one of the at least one processing resource. In one such embodiment, the computing system further includes at least one root complex associated with the at least one processing resource and the bus switch includes a Peripheral Component Interconnect Express (PCI Express) bus switch. The bus switch is operable to communicatively couple the at least one peripheral device socket to the at least one processing resource such that each peripheral device of the computing system is coupled to at least one of the at least one root complex via a PCI Express bus.
In further exemplary embodiments, the method of configuring a computing system includes determining a count of installed processing resources of the computing system. Based on the count of installed processing resources, a configuration file is generated that specifies a device assignment of a peripheral device of the computing system to a bus of the computing system associated with an installed processing resource. The configuration file is provided to a switch in communication with the peripheral device and the bus. The configuration file configures the switch to implement the device assignment and to communicatively couple the peripheral device to the installed processing resource via the bus. In one such embodiment, the generating of the configuration file includes, based on the count of installed processing resources, determining the device assignment such that each peripheral device of the computing system is coupled to at least one of the installed processing resources. In one such embodiment, the generating of the configuration file also includes determining the device assignment such that each peripheral device of the computing system is allocated a minimum communication resource.
In yet further exemplary embodiments, the apparatus comprises a non-transitory, tangible computer readable storage medium storing a computer program, wherein the computer program has instructions. When executed by a computer processor, the instructions carry out: determining a processor count of a computing system having one or more installed processors; allocating a set of peripheral device sockets to the one or more installed processors based on the determined processor count such that each socket of the set of peripheral device sockets is communicatively coupled to at least one of the one or more installed processors; generating a switch instruction for implementing the allocation; and providing the switch instruction to a switch of the computing system and thereby coupling the set of peripheral device sockets to the one or more installed processors. In some such embodiments, the apparatus comprises further instructions that carry out allocating the set of peripheral device sockets further based on a peripheral device priority.
Although illustrative embodiments have been shown and described, a wide range of modification, change and substitution is contemplated in the foregoing disclosure and in some instances, some features of the embodiments may be employed without a corresponding use of other features. One of ordinary skill in the art would recognize many variations, alternatives, and modifications. Thus, the scope of the invention should be limited only by the following claims, and it is appropriate that the claims be construed broadly and in a manner consistent with the scope of the embodiments disclosed herein.
Walker, Robert, Kolor, Daniel John, Rollins, William Leo
Patent | Priority | Assignee | Title |
Patent | Priority | Assignee | Title |
7711886, | Dec 13 2007 | International Business Machines Corporation | Dynamically allocating communication lanes for a plurality of input/output (‘I/O’) adapter sockets in a point-to-point, serial I/O expansion subsystem of a computing system |
8103810, | May 05 2008 | International Business Machines Corporation | Native and non-native I/O virtualization in a single adapter |
8359415, | May 05 2008 | International Business Machines Corporation | Multi-root I/O virtualization using separate management facilities of multiple logical partitions |
8484399, | Jul 08 2005 | Dell Products L.P. | System and method for configuring expansion bus links to generate a double-bandwidth link slot |
8521915, | Aug 18 2009 | SanDisk Technologies LLC | Communicating between host computers and peripheral resources in an input/output (I/O) virtualization system |
8625615, | May 16 2008 | NEC Corporation | PCI express switch, PCI express system, and network control method |
9086919, | Aug 23 2012 | Dell Products, LP | Fabric independent PCIe cluster manager |
9203699, | Feb 11 2014 | LENOVO INTERNATIONAL LIMITED | Constructing and verifying switch fabric cabling schemes |
9235252, | Dec 21 2012 | Intel Corporation | Dynamic balancing of power across a plurality of processor domains according to power policy control bias |
9304799, | Dec 27 2013 | International Business Machines Corporation | Placement of input / output adapter cards in a server |
9424223, | Jun 08 2012 | NEC Corporation | Tightly coupled multiprocessor system |
9459978, | Jan 24 2013 | DEUTSCHE BANK AG NEW YORK BRANCH, AS COLLATERAL AGENT | Automated test platform utilizing segmented data sequencers to provide time controlled test sequences to device under test |
20130151750, | |||
20150254202, | |||
20150324312, | |||
20150347345, | |||
20160070661, |
Executed on | Assignor | Assignee | Conveyance | Frame | Reel | Doc |
Jul 25 2014 | NetApp, Inc. | (assignment on the face of the patent) | / | |||
Aug 06 2014 | ROLLINS, LEO | NetApp, Inc | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 033749 | /0256 | |
Aug 06 2014 | KOLOR, DAN | NetApp, Inc | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 033749 | /0256 | |
Aug 06 2014 | WALKER, ROBERT | NetApp, Inc | ASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS | 033749 | /0256 |
Date | Maintenance Fee Events |
Jun 07 2021 | M1551: Payment of Maintenance Fee, 4th Year, Large Entity. |
Date | Maintenance Schedule |
Dec 05 2020 | 4 years fee payment window open |
Jun 05 2021 | 6 months grace period start (w surcharge) |
Dec 05 2021 | patent expiry (for year 4) |
Dec 05 2023 | 2 years to revive unintentionally abandoned end. (for year 4) |
Dec 05 2024 | 8 years fee payment window open |
Jun 05 2025 | 6 months grace period start (w surcharge) |
Dec 05 2025 | patent expiry (for year 8) |
Dec 05 2027 | 2 years to revive unintentionally abandoned end. (for year 8) |
Dec 05 2028 | 12 years fee payment window open |
Jun 05 2029 | 6 months grace period start (w surcharge) |
Dec 05 2029 | patent expiry (for year 12) |
Dec 05 2031 | 2 years to revive unintentionally abandoned end. (for year 12) |