This disclosure relates to methods and systems for dynamically partitioning of pcie disk arrays based on software configuration/policy distribution. In one embodiment, at least one pcie switch has an input port operatively connected to a respective cpu and at least one output port. A multiplexer is connected between the output port(s) of the at least one pcie switch and a pcie disk array, for example an NVMe SSD, and is configured to connect the pcie disk array in a first configuration to a single pcie switch in either one-x4 port or two x2 port mode, or in a second configuration to two pcie switches in x2 port mode. The multiplexer can dynamically switch between the first configuration and the second configuration on the fly. Switching can occur, for example, in response to a hot-swap of an NVMe SSD or a policy change.

Patent
   10713203
Priority
Feb 28 2017
Filed
Feb 28 2017
Issued
Jul 14 2020
Expiry
Feb 26 2038
Extension
363 days
Assg.orig
Entity
Large
1
588
currently ok
10. A method for partitioning a peripheral component Interconnect Express (pcie) disk array, the method comprising:
connecting at least one multiplexer between the pcie disk array and at least one output port of at least one pcie switch, the at least one multiplexer configured to alternate a connection of the pcie disk array between a first configuration and a second configuration in response to a change in a configuration policy or a surprise addition/removal without notification, the first configuration being the at least one multiplexer connected to a single pcie switch in one x4 port mode or in two x2 port mode, the second configuration being the at least one multiplexer connected to two pcie switches in x2 port mode, and
dynamically switching with the at least one multiplexer between the first configuration and the second configuration in response to the change.
17. A computer program embodied in a non-transitory computer-readable medium and comprising program instructions which, when loaded into a computer memory and executed by a processor, causes the processor to:
switch at least one multiplexer connected between a peripheral component Interconnect Express (pcie) disk array and one or more output ports of one or more pcie switches between a first configuration and a second configuration, the at least one multiplexer connecting the pcie disk array in the first configuration to an output port of a single pcie switch in one x4 port mode or in two x2 port mode, or in the second configuration to output ports of two pcie switches in x2 port mode,
wherein the at least one multiplexer is dynamically switched between the first configuration and the second configuration in response to a change in a configuration policy, a surprise addition without notification, or a surprise removal without notification.
1. An apparatus comprising:
at least one peripheral component Interconnect Express (pcie) switch having an input port operatively connected to a respective cpu and at least one output port; and
at least one multiplexer connected between the at least one output port of the at least one pcie switch and a pcie disk array, the at least one multiplexer configured to alternate a connection of the pcie disk array between a first configuration and a second configuration in response to a change in a configuration policy, a surprise addition without notification, or a surprise removal without notification, the first configuration being the at least one multiplexer connected to a single pcie switch in one x4 port mode or two x2 port mode, the second configuration being the at least one multiplexer connected to two pcie switches in x2 port mode, the at least one multiplexer configured to dynamically affect a switch-over between the first configuration and the second configuration.
2. The apparatus of claim 1,
wherein,
the switch-over occurs in response to the change in the configuration policy,
and
the configuration policy is of the pcie disk array.
3. The apparatus of claim 1,
wherein,
the switch-over occurs in response to the surprise addition without the notification, and
the surprise addition is of a pcie disk array (hot-add).
4. The apparatus of claim 1,
wherein,
the switch-over occurs in response to the surprise removal without the notification, and
the surprise removal is of a pcie disk array (hot-remove).
5. The apparatus of claim 1, wherein when data traffic congestion is detected at a first pcie switch, the at least one multiplexer is configured to redirect a portion of data traffic from the first pcie switch to a second pcie switch for transmission to/from the pcie disk array.
6. The apparatus of claim 1, wherein when operating in x4 port mode, the at least one multiplexer is configured to migrate all data traffic from a first pcie switch to a second pcie switch upon detection of a failure of the first pcie switch for transmission to/from the pcie disk array.
7. The apparatus of claim 1, wherein the pcie disk array comprises a Non-Volatile memory Express Solid-State Drive (NVMe SSD).
8. The apparatus of claim 1, further comprising:
a control logic device operatively connected to at least one pcie switch and the at least one multiplexer and configured to monitor pcie disk array information and pcie disk configuration policy execution.
9. The apparatus of claim 8, wherein the control logic device is connected for communication to a management platform, with the management platform managing policy for the control logic device.
11. The method of claim 10, wherein the pcie disk array comprises a Non-Volatile memory Express Solid-State Drive (NVMe SSD).
12. The method of claim 10,
wherein,
the dynamically switching between the first configuration and the second configuration takes place responsive to the surprise addition without the notification and/or the surprise removal without the notification,
the surprise addition is of a pcie disk array (hot-add), and
the surprise removal is of a pcie disk array (hot-remove).
13. The method of claim 10,
wherein,
the dynamically switching between the first configuration and the second configuration takes place responsive to the change in the configuration policy, and
the change in the configuration policy is of the pcie disk array.
14. The method of claim 10, further comprising:
when data traffic congestion is detected at a first pcie switch, redirecting a portion of the data traffic from the first pcie switch to a second pcie switch for transmission to/from the pcie disk array.
15. The method of claim 10, further comprising:
when operating in x4 port mode, migrating all data traffic to a second pcie switch upon detection of a failure of a first pcie switch for transmission to/from the pcie disk array.
16. The method of claim 10, wherein the dynamically switching includes managing, with a control logic device operatively connected to at least one pcie switch and the at least one multiplexer, information of the pcie disk array and status information of the at least one multiplexer.
18. The computer program of claim 17,
wherein,
dynamically switching takes place responsive to the surprise addition without the notification or the surprise removal without the notification,
the surprise addition being of a pcie disk array (hot-add), and
the surprise removal being of a pcie disk array (hot-remove).
19. The computer program of claim 17, wherein the computer program causes a control logic device to communicate status information of the pcie disk array and the at least one multiplexer.

This disclosure relates in general to the field of communications and, more particularly, to methods and systems for dynamically partitioning of Peripheral Component Interconnect Express (PCIe) disk arrays based on software configuration/policy distribution.

Increased use of Solid-State-Drives (SSD) in PCIe disk arrays in data center storage solutions has not only significantly reduced input/output (I/O) latency, but has also been able to alleviate I/O bottlenecks by providing a much higher I/O performance without the need for a large number of disk enclosures. However, given the extreme advances in performance, the performance of network controllers and other network devices has thus far lagged behind in handling the vastly increased I/O traffic.

To handle the controller bottleneck, an NVMe (Non-Volatile Memory Express) interface has been developed, which gains great advantages over the traditional SAS)/SATA interface (SAS=Serial Attached SCSI; SATA=Serial Advanced Technology Attachment) due to the direct PCIe connection between central processing unit CPU and the SSD, which leads to linearly scalable bandwidth and considerable reduction in latency. However, on the other hand, current NVMe implementations are too tightly coupled to CPUs such that failover of SSDs may cause problems. To loosen the prior art tight coupling between CPU and SSD, PCIe switches have been involved in the storage solution, which, not only natively extend the limited CPU PCIe bandwidth, but also isolate the SSD failure issue, at the cost of increased CPU to SSD latency.

To handle the network bottleneck for I/O performance, hyper convergence has become the new trend for data center infrastructure. Comparing to the traditional centralized storage, such as SAN (Storage Area Network), where the storage pool as a whole is accessed by hosts via a network, hyper convergence is basically a distributed storage platform, implemented by natively integrating the compute and storage components together as a node, and aggregating nodes into cluster. Hyper convergence fully utilizes the localized I/O operation within a node, which significantly offloads the network traffic. However, there remains heavy traffic on the node-to-node network due to the distributed storage requirement, such as data replica and synchronization.

To provide a more complete understanding of the present disclosure and features and advantages thereof, reference is made to the following description, taken in conjunction with the accompanying figures, wherein like reference numerals represent like parts, in which:

FIGS. 1(a) through 1(e) show simplified schematic diagrams illustrating basic topologies for connecting CPU and NVMe SSDs by way of PCIe switches, according to some embodiments of the present disclosure;

FIG. 2 is a simplified schematic diagram illustrating another basic topologies for connecting multiple CPUs and NVMe SSDs by way of PCIe switches with CPU level and switch level redundancy, according to some embodiments of the present disclosure;

FIG. 3 illustrates a dynamically configurable NVMe SSD interface architecture according to some embodiments of the present disclosure;

FIG. 4 shows a simplified structure of the dynamically configurable NVMe SSD interface architecture of FIG. 3;

FIG. 5 illustrates schematically possible work state transitions for a NVMe SSD listed in Table 1;

FIG. 6 shows schematically a process flow for hot-adding a NVMe SSD;

FIG. 7 shows schematically a process flow for hot-remove of a NVMe SSD; and

FIG. 8 shows schematically a process flow for dynamic modification of a SSD configuration policy.

Various embodiments of the present disclosure relate to dynamically partitioning of PCIe disk arrays based on software configuration/policy distribution. The basic idea is to provide redundant PCIe links in the form of multiplexers connected between the downstream ports of switches SW0, SW1 and the Non-Volatile Memory Express (NVMe) Solid-State Drives (SSD), which can be switched dynamically according to a detected switch or SSD failure status and/or application policy/requirement. As used herein, the term “on the fly” and “dynamically” in the context of this disclosure describes activities or events that develop or occur, for example, while the process that the switch-over affects is ongoing, rather than as the result of something that is statically predefined. More particularly, the terms “hot swapping”, “hot-add” and “hot-remove” refer to “on-the-fly” replacement of computer hardware, such as the described NVMe SSDs.

In one aspect of the present disclosure, an apparatus is proposed with at least one PCIe switch having an input port operatively connected to a respective CPU and at least one output port; and at least one multiplexer connected between the output port of the at least one PCIe switch and a PCIe disk array and configured connect the PCIe disk array in a first configuration to a single PCIe switch in either one-x4 port or two x2 port mode, or in a second configuration to two PCIe switches in x2 port mode, and to dynamically switch over between the first configuration and the second configuration on the fly. The designation “x2” indicates a 2-lane configuration (2 physical ports-x2 port mode), wherein “x4” indicates a 4-lane configuration (4 physical ports=x4 port mode). The ports may be implemented with optical fibers.

In an embodiment, the switch-over between the first configuration and the second configuration may take place responsive to a change in a configuration policy of the PCIe disk array.

In another embodiment, the switch-over between the first configuration and the second configuration may take place responsive to a surprise addition of a PCIe disk array (hot-add).

In yet another embodiment, the switch-over between the first configuration and the second configuration may take place responsive to a surprise remove of an NVMe SSD (hot-remove).

In another embodiment, the switch-over between the first configuration and the second configuration may take place sequentially based on predetermined timing requirements.

In yet another embodiment, when data traffic congestion is detected at a first PCIe switch, the at least one multiplexer is configured to redirect a portion of the data traffic from the first PCIe switch to a second PCIe switch for transmission to the PCIe disk array.

In yet another embodiment, when operating in x4 port mode, the at least one multiplexer is configured to migrate all data traffic from the first PCIe switch to a second PCIe switch upon detection of a failure of a first PCIe switch for transmission to the PCIe disk array.

In yet another embodiment, the PCIe disk array may be an NVMe SSD.

In yet another embodiment, the apparatus may include a control logic device operatively connected to at least one PCIe switch and the at least one multiplexer and configured to monitor PCIe disk array information and PCIe disk configuration policy execution.

In yet another embodiment, the control logic device may be connected for communication to a management platform, with the management platform managing policy for the control logic device.

In another aspect of the present disclosure, a method is proposed for dynamically switching over between configurations that connect a CPU to an NVMe SSD. The method includes connecting the PCIe disk array in a first configuration to an output port of a single PCIe switch in either one-x4 port or two x2 port mode, or in a second configuration to output ports of two PCIe switches in x2 port mode, and dynamically switching over between the first configuration and the second configuration on the fly.

In one embodiment, the PCIe disk array may be dynamically switched over by way of at least one multiplexer interposed between the PCIe disk array and the PCIe disk array.

In another embodiment, the PCIe disk array may be dynamically switched over responsive to a change in a configuration policy of the PCIe disk array, a surprise addition of a PCIe disk array (hot-add), a surprise remove of an NVMe SSD (hot-remove), or a combination thereof.

In yet another embodiment, when data traffic congestion is detected at a first PCIe switch, a portion of the data traffic may be redirected from the first PCIe switch to a second PCIe switch for transmission to the PCIe disk array

In yet another embodiment, when operating in x4 port mode, all data traffic may be migrated to a second PCIe switch upon detection of a failure of a first PCIe switch for transmission to the PCIe disk array.

In yet another embodiment, dynamically switching over may include managing, with a control logic device operatively connected to at least one PCIe switch and the at least one multiplexer configuration, information of the PCIe disk array and status information of the at least one multiplexer.

As will be appreciated by one skilled in the art, aspects of the present disclosure, in particular the functionality related to various aspects of dynamic partitioning of PCIe disk arrays, may be controlled by computer programs. Accordingly, other aspects of the present disclosure relate to systems, computer programs, mechanisms, and means for carrying out the methods according to various embodiments described herein. Such systems, computer programs, mechanisms, and means could be included within various network devices, such as e.g. the management platform and the hypervisor. A computer program may, for example, be downloaded (updated) to the existing network devices and systems (e.g. to the existing routers, switches, various control nodes, etc.) or be stored upon manufacturing of these devices and systems.

In yet another aspect, the present application relates to one or more non-transitory computer readable storage media encoded with software comprising computer executable instructions and, when executed by a processor of a computer, operable to carry out the method according to various embodiments described herein.

In yet another aspect, the present application relates to a system comprising a plurality of computers, each computer connected to a first switching means in one-to-one correspondence, second switching means having each inputs connected to outputs of each of the first switching means and outputs connected to storage means, wherein the second switching means are configured to dynamically connect the storage means to the outputs of the first switching means on the fly.

Additional features and advantages of the disclosure will be set forth in the description which follows, and in part will be obvious from the description, or can be learned by practice of the herein disclosed principles. The features and advantages of the disclosure can be realized and obtained by means of the instruments and combinations particularly pointed out in the appended claims. These and other features of the disclosure will become more fully apparent from the following description and appended claims, or can be easily learned by the practice of the principles set forth herein.

Overview of the Peripheral Component Interconnect Express (PCIe) Architecture

One interconnect fabric architecture includes the Peripheral Component Interconnect Express (PCIe) architecture. A primary goal of PCIe is to enable components and devices from different vendors to inter-operate in an open architecture, spanning multiple market segments; Clients (Desktops and Mobile), Servers (Standard and Enterprise), and Embedded and Communication devices. PCIe is a high performance, general purpose I/O interconnects defined for a wide variety of future computing and communication platforms. Some PCIe attributes, such as its usage model, load-store architecture, and software interfaces, have been maintained through its revisions, whereas previous parallel bus implementations have been replaced by a highly scalable, fully serial interface. The more recent versions of PCIe take advantage of advances in point-to-point interconnects, Switch-based technology, and packetized protocol to deliver new levels of performance and features. Power Management, Quality of Service (QoS), Hot-Plug/Hot-Swap support, Data Integrity, and Error Handling are among some of the advanced features supported by PCI Express.

Disadvantageously, however, much of the conventional switch-based technology with single x4 NMVe SSD does not have switch level redundancy that is required in high reliable storage products. To overcome this, the dual x2 port NVMe SSDs are designed which could be connected to 2 switches to gain redundancy. The connections between switch and SSDs are fixed in all the conventional methodology, and do not allow switch-over between dual-port mode and single-port mode on-the fly based on policy. The design proposed and illustrated in various embodiments of the present disclosure attempts to alleviate these shortcomings.

FIGS. 1 (a) through 1(c) illustrate basic topologies for connecting a CPU and NVMe SSDs via interposed PCIe switches. FIG. 1(a) shows a topology with only a single CPU and a single switch SW0. For a larger number of SSDs, the topologies may be extended in either a parallel arrangement, as illustrated in FIG. 1(b), with two switches SW0, SW1 having their respective input connected to a single CPU, or in a cascaded arrangement with two levels of switches, wherein switch SW0 in an upper level has an input connected to a single CPU and lower level switches SW1, SW2, . . . SWn connected between switch SW0 and the NVMe SSDs, as illustrated in FIG. 1(c). In all these three topologies of FIGS. 1(a) through (c), the SSDs operate in x4 PCIe mode and the connections between the CPU and the SSDs are fixed or hard-wired, which deprives these topologies of flexibility and reduces the reliability of this type of storage solution.

Redundancy at the CPU level (CPU redundancy) could be achieved by combining MR-IOV (Multiple Root I/O Virtualization) feature of the exemplary switches SW0 and SW1 and the NVMe SSD's dual-port feature, wherein two CPUs would be connected to the upstream (US) ports of switch SW0 of the topology in FIG. 1(a), allowing each CPU to be switched to any of the NVMe SSDs. This topology is illustrated in FIG. 1(d). Meanwhile, switch level redundancy could be implemented as shown in FIG. 1(e), wherein the downstream (DS) ports of each switch SW0 and SW1 could be connected to any of the NVMe SSDs. This will be referred to as switch redundancy. The two switches SW0 and SW1 could work in either active/active or active/passive mode, depending on application and CPU's I/O bandwidth requirement. CPU redundancy and switch redundancy may advantageously be combined to provide double-redundancy, both at the CPU level and at the switch level, which is illustrated with CPU0 and CPU1, and switches SW0 and SW1 in FIG. 2.

As mentioned above, the aforementioned topologies having either CPU redundancy or switch redundancy, or both, without forcing NMVe SSDs to work in dual 2x port mode to facilitate the redundant connection. There are also circumstances where flexibility is required to swap NVMe SSDs between dual-port and single-port mode on-the fly. The proposed design attempts to achieve providing redundancy without limiting the SSDs type that could be used in the storage system.

Example Embodiments of a Switchable NVMe SSD Interface Architecture

The top level design of a proposed NVMe SSD interface architecture 300 is illustrated in FIG. 3. Considering the tradeoff between redundancy and switch port resources, a two-switch design may be an optimal choice. The basic idea is to provide redundant PCIe links between switches SW0, SW1 and the NVMe SSDs, which could be switched according to a detected switch or SSD failure status and/or application policy/requirement. The design has a total of four distinct components, namely switches; a hardware control logic device shown here exemplary as a FPGA (field-programmable gate array); NVMe SSDs connected at an 8639 drive backplane connector; and the upper level management platform, such as the Cisco UCSM (Unified Computing System Manager) in the present example.

The 8639 drive backplane connector is a 68-pin connector and designed to support PCI Express as well as hot-plug and hot-remove, both with and without prior system notification (surprise hot-remove).

The PCIe signal in the data plane (double arrows) and its reference clock in the clock plane (dashed lines), side band signals (includes Dual-Port Enable DUALPORTEN #/Interface Detect IFDET #/Hot-Plug Detect PRSNT #/PCIe device reset PERST #/Power Enable PWREN) in the control plane are necessary to facilitate all the SSD operation timing control and are assigned to specific pins on the 8639 connector. The control plane signals are connected, either directly or indirectly on the FPGA to the Expander_SW1 and Expander_SW2. The signal EN is an enable signal.

The present disclosure addresses only the traffic in the data plane between the downstream ports of switches SW0, SW1 and the NVMe SSDs. For the connection between the upstream ports of the switches SW0, SW1 and one or more CPUs, any of the topologies illustrated in FIG. 1 and the redundant topology illustrated in FIG. 2 could be used. In other words, the connectivity of the downstream ports to the NVMe SSDs is not affected by the topology of the upstream ports. It will be understood that the number of illustrated CPUs, switches and NVMe SSDs are merely examples and any number of such devices can be used that can be interconnected and addressed in the manner described.

The hardware control logic device, herein also referred to simply as FPGA, deals with all the control logic such as SSD information collection and reporting, policy execution and optional indicator light (LED) management. The management platform interfaces between hypervisor and FPGA and collects SSD information and delivers an application requirement in the form of policy.

The virtual channel connecting the FPGA to the management platform could be implemented via multiple ways, for example Ethernet. To support both single port and dual port mode for the NVMe, the system must be able to modify the bifurcation of the downstream ports of the switches SW0, SW1 on the fly. The term “on the fly” in the context of this disclosure describes activities or events that develop or occur dynamically, i.e. while the process that the switch-over affects is ongoing, rather than as the result of something that is statically predefined. More particularly, the term “hot swapping” refers to “on-the-fly” replacement of computer hardware, such as the described NVMe SSDs. This is accomplished by sending a reconfiguration command to switch the ASIC (application-specific integrated circuit) itself (MRPC=multicast remote procedure call command in PMC PCIe switch). The connection of the switches SW0, SW1 to a 2:1 PCIe multiplexer (PCIe MUX) is implemented downstream of PCIe port switching to gain redundancy regardless of whether single x4 SSD or dual x2 SSD are attached. This on the fly switching is based on policy or software configuration.

All valid SSD configurations for a two-switch redundant situation are summarized in Table 1.

TABLE 1
IO IO
Exp. Exp.
Work SW0 SW1
State Dual_Porten_N SW_SEL_0 SW_SEL_1 PE3 Ref_CLK_P0 Ref_CLK_P1 EN EN PE_RST_P0 PE_RST_P1
W1 0 0 0 Two x2 SW0 SW0 1 0 SW0 SW0, not
ports both used
from SW0
W2 0 1 1 Two x2 SW1 SW1 0 1 SW1 SW1, not
ports both used
from SW1
W3 0 0 1 Port 1 SW0 SW1 1 1 SW0 SW1
from SW0,
port 2
from SW1
W4 0 1 0 Port 1 SW1 SW0 1 1 SW1 SW0
from SW1,
port 2
from SW0
W5 1 0 0 X4 from SW0 SW0, not 1 0 SW0 SW0, not
SW0 used used
W6 1 1 1 X4 from SW1 SW1, not 0 1 SW1 SW1, not
SW1 used used
1 0 1 For- X X X X X X
bidden
1 1 0 For- X X X X X X
bidden

In top design diagram of the NVMe SSD interface architecture 300 shown in FIG. 3, two CPUs CPU0 and CPU1 are each connected to respective upstream (US) ports of switches SW0, SW1. Note that there is no connection between CPU0 and SW1 and between CPU1 and SW0. Downstream (DS) ports of each switch are connected for data transfer to respective input ports of each of two 2:1 PCIe multiplexers (MUX) 310, 312, i.e. each CPU can transfer data to and receive data from each of the MUXs 310, 312. DUALPORTEN # is disabled (low=0) so that the NVMe SSD operates in x2 mode. In FIG. 3, the input ports of each of the MUXs 310, 312 are connected in x2 mode to a respective PCIe terminal, namely PCIe [1:0] and PCIe [3:2], on the 8639 connector and from there to unillustrated NVMe SSDs. The possible data links depend on the switch selections and the corresponding work states in x2 mode are listed in Table 1 as W1-W4.

Note that not all components of the FPGA are shown in FIG. 3, and reference is made to FIG. 4 for any component not shown in FIG. 3

For an easier understanding of the top design diagram of the NVMe SSD interface architecture 300 in FIG. 3, reference is now made to the exemplary system 400 illustrated in FIG. 4 which uses only a single multiplexer 410. It will be assumed that DUALPORTEN # is enabled (high=1) so that the unillustrated NVMe SSD connected to terminals PCIe[3:0] on the 8639 connector operates in x4 mode (x4 PCIe). The traffic in the data plane could then only be swapped between Work State W5 and Work State W6 in Table 1. Only one MUX 410 is used in this example. Both switch select signals SW_SEL_0 and SW_SEL_1 are selectively applied to MUX 410. The two LEDs (red LED; blue LED) shown in FIG. 4 are status indicators and have no bearing on the operation of the system 400.

The I2C GPIO Expanders shown in FIG. 4 are general-purpose input/output (GPIO) expanders that can be used via the I2C interface (I2C=Inter-Integrated Circuit) and allow more than one device to connect to a single port on a computer.

In a first Work State W5, the REF_CLK_P0 is enabled (=1) at the pinout of the 8639 connector, whereas the REF_CLK_P1 is disabled (not used). The I2C GPIO Expander SW0 on the FPGA is high=1, whereas the I2C GPIO Expander SW1 on the FPGA is low=0. With both the SW_SEL_0 and the SW_SEL_1 low (=0) in Work State W5, the 4-port (x4) PCIe [3:0] on the 8639 connector is connected to the NVMe SSD by way of switch SW0. PE_RST_P0 is signaling SW0, whereas PE_RST_P1 is not connected in FIG. 4 because the second 2:1 RST MUX is not used in this configuration. Likewise, the second 2:1 PCIe MUX 312 and the second 2:1 CLK MUX of FIG. 3 are also not used in FIG. 4 since SW1 is not addressed in Work State W5. The aforedescribed situation for the x4 Work State W5 is substantially equivalent to the Work State W1, where SW_SEL_0=SW_SEL_1=0 and DUALPORTEN # is disabled.

In a second Work State W6, the REF_CLK_P0 is enabled at the pinout of the 8639 connector, whereas the REF_CLK_P1 is disabled (not used). The I2C GPIO Expander SW0 on the FPGA is high=1, whereas the I2C GPIO Expander SW1 on the FPGA is low=0. With both the SW_SEL_0 and the SW_SEL_1 high (=1) in Work State W6, the 4-port (x4) PCIe [3:0] on the 8639 connector is connected to the NVMe SSD by way of switch SW1. PE_RST_P0 is signaling SW1, whereas PE_RST_P1 is not connected in FIG. 4 because as above, the second 2:1 RST Mux is not used in this configuration. Likewise, the second 2:1 PCIe Mux and the second 2:1 CLK Mux of FIG. 3 are also not used in FIG. 4 since SW0 is not addressed in Work State W6. The aforedescribed situation for the x4 Work State W5 is substantially equivalent to the Work State W1, where SW_SEL_0=SW_SEL_1=0 and DUALPORTEN # is disabled. To summarize, the x4 PCIe bus from SW0 and the x4 PCIe bus from SW1 are selectively, but never simultaneously, connected via the 2:1 PCI2 Mux to the x4 PCIe bus terminating at the connector 8639 at the PCIe[3:0] pin, and ultimately connected to a NVMe SSD (not shown).

As mentioned above with reference to FIG. 3, the other four valid Work States W1-W4 all employ two x2 ports instead of a single x4 port. The Work States W1 and W2 have been discussed above due to their similarity with the Work States W5 and W6. For the Work States W3 and W4, both Ref_CLK_P0 and Ref_CLK_P1 are high and either SW_SEL_0 or SW_SEL_1 is high, while the other is low. This selects port 1 from SW0 and port 2 from SW1 for Work State W5 and port 2 from SW0 and port 1 from SW1 for Work State W5. Switching between these Work States W1-W6 is schematically depicted in FIG. 5.

Exemplary Work State transitions

Among all the possible work state transitions illustrated in FIG. 5, the following three examples are intended to demonstrate some of the benefits of this design.

Take into consideration the variant application I/O operation styles from high-bandwidth single-stream to lower bandwidth multiple-stream, storage virtualization platform needs to provide the flexibility to swap between them, and the transition from Work State_W5 to Work State_W1 facilitates the virtualization stack to make the best use of SSD's high capability for parallel I/O operation.

This transition could be utilized to implement switch level traffic offload. In one scenario where the upstream port of SW0 is suffering a traffic jam, half of its downstream SSDs could then be re-directed to SW1 to thus significantly improve the overall I/O data flow speed. This transition provides the possibility for a balanced storage load.

The most common application scenario for this transition is a switch failover. Assume, for example, that SW0 has failed. Data loss could then be avoided by migrating all traffic to and from the downstream port of SW0 to SW1. Since this redundancy is implemented in x4 single-port mode, no I/O performance loss occurs due to dual-port operation.

Another application could apply to a switch arrangement where each switch in FIG. 2 is connected to a single CPU, i.e. connections between CPU0 and SW1 and between CPU1 and SW0 would not exist. In case that large amount of cold data, i.e. data that are rarely accessed, needs to be transported from CPU0 to CPU1, e.g. when a new node is added into the current distributed storage network, copies of user data need to be delivered from the current nodes to this new node. The traditional solution would involve a communication network between the two CPUs CPU0, CPU1, resulting in a heavy traffic load. With aforedescribed exemplary design illustrated for example in FIG. 3, CPU0 could first clone of all the required data onto one NVMe SSD and thereafter migrate this NVMe SSD to CPU1 as a whole. In other words, the inner-CPU bandwidth (making a clone SSD within one node) is utilized, thus saving inter-CPU network bandwidth cost.

The operation during hot-add of a NVMe SSD will now be described with reference to a process flow chart shown in FIG. 6.

FPGA and Switch Initialization

Before any hot-swap, the system must be powered on and the firmware loaded. The FPGA will disable all I2C GPIO expander interfaces between switches SW0, SW1 and the NVMe SSDs before the NVMe SSDs can be successfully detected and configured.

The switches SW0, SW1 need to enable the dynamic bifurcation and dynamic partition feature during initialization. A switch should also reserve a sufficient number of logic P2P (PCIe-to-PCIe) bridges for the physical downstream (DS) ports. For each stack, all the DS ports would then be configured in x2 granularity. A 2 x2 port could be combined to work as a 1 x4 port through a configuration command as the policy is distributed downwards. During enumeration, each logic P2P bridge will be assigned its own primary/secondary/subordinate bus number. This is task of the root complex which connects the processor and memory subsystem to the PCI Express switch fabric composed of one or more switch devices.

For a cascaded switch similar to the arrangement of Topology (c) in FIG. 1, the top level SW's execute enumeration. In the context of this disclosure, the disclosed process relates only to the bottom switches that directly interface with the NVMe SSDs.

Surprise SSD Hot-Add

FIG. 6 shows schematically a process 600 for hot adding an NVMe SSD. The process 600 starts at 610. The FPGA is first to sense the toggle of IFDET # indicating a new inserted NVMe SSD, at 612. Triggered by this toggle, the FPGA will then poll the SSD E2PROM content and report the content to the Management Platform.

Typically, users will assign one specific application policy for this SSD via the Management Platform, with the Management Platform delivering the latest policy to FPGA. This may be a default configuration policy or a configuration policy updated for example by a user in response to an application update.

The FPGA needs to configure all the MUX devices in FIGS. 3 and 4 according to the policy. The I2C GPIO expander interface will only be enabled after all the configurations have been successfully executed.

At 614, the switch reads the asserted IFDET # and the DUALPORTEN # of the SSD. The switch will then configure the bifurcation as x2, as in the example of FIG. 3, or as x4, as in the example of FIG. 4, commensurate with the DUALPORTEN # signal status, at step 616. Thereafter, at 618, the switch powers the SSD on, de-asserts PERST #, and initiates the PCIe link training procedure to enter the normal operating mode.

Surprise SSD Hot-Remove

FIG. 7 illustrates a process 800 for a surprise SSD hot remove. When SSD remove is initiated, at step 714, the loss of the de-asserted IFDET # signal is reported to the FPGA, at 716, which will automatically disable the I2C GPIO expander interface to the switch, at 718. Detection of the link loss, at 720, will clear the SSD configuration space, at 722, whereafter the complete hot-remove procedure specified by the vendor of the SSD can be performed, at 724. Hot remove may also be triggered by the loss of a PCIe link. When the hot-remove process is concluded, the switch sends a hot-remove-done flag to the FPGA, at 726.

Dynamic Modification of a SSD Configuration Policy

The SSD configuration policy may be modified on-the-fly according to a changed application requirement. The associated process 800 will now be described with reference to the flow diagram in FIG. 8.

At 812, the configuration policy for the SSD is updated based on an application update. Specifically, this is done by performing at least some of the steps of the hot-remove process described in FIG. 7 (see process 700 in FIG. 7), although the NVMe SSD will actually not have to be removed physically, meaning that the PCIe link may still be active. The switch then informs the FPGA when the hot remove process is finished, at 816, and sets a flag. In one example, the switch may activate a blinking light (not shown) during the hot-remove process and then stop blinking when the hot-remove process is terminated. Details of the hot-remove process are not part of this disclosure and are typically specified by the vendor of the switch. The FPGA will configure the updated DUALPORTEN # and the SW_SEL_0/1 signals according to the new policy (see Table 1) after receiving the hot-remove-done signal, at 818. When the reconfiguration is done, FPGA will assert a FPGA_CONFG_DONE signal. The switch can then re-bifurcate its downstream ports and re-train the PCIe link, at 820. The process 800 does not need to be performed when a SSD is hot-added (see process 600 in FIG. 6).

The disclosed embodiments represent a consolidated solution to flexibly and dynamically modify the connection between NVMe SSD and PCIe switch in x2-port granularity on-the-fly. According to some embodiments of the disclosure, one NVMe SSD may be connected to either a single switch in one x4-port mode/two x2-port mode, or to two switches in x2-port mode. These different connections scenarios may then be swapped dynamically without requiring administrator intervention.

This flexibility is implemented by swapping NVMe SSD work mode (between dual-port and single-port) and adapting the switches' downstream partition policy sequentially based on specific timing requirement.

With this flexibility, the virtualization stack now obtains one more degree of freedom to schedule its I/O traffic in a more balanced and efficient way. Besides that, the switch level redundancy is natively realized in single-port mode without the cost of I/O performance loss due to dual-port operation.

In summary, these exemplary embodiments may provide at least the following advantages:

The embodiments of methods, hardware, software, firmware or code set forth above may be implemented via instructions or code stored on a machine-accessible, machine readable, computer accessible, or computer readable medium which are executable by a processing element. A non-transitory machine-accessible/readable medium includes any mechanism that provides (i.e., stores and/or transmits) information in a form readable by a machine, such as a computer or electronic system. For example, a non-transitory machine-accessible medium includes random-access memory (RAM), such as static RAM (SRAM) or dynamic RAM (DRAM); ROM; magnetic or optical storage medium; flash memory devices; electrical storage devices; optical storage devices; acoustical storage devices; other form of storage devices for holding information received from transitory (propagated) signals (e.g., carrier waves, infrared signals, digital signals); etc., which are to be distinguished from the non-transitory mediums that may receive information there from.

Instructions used to program logic to perform embodiments of the invention may be stored within a memory in the system, such as DRAM, cache, flash memory, or other storage. Furthermore, the instructions can be distributed via a network or by way of other computer readable media. Thus a machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), but is not limited to, floppy diskettes, optical disks, Compact Disc, Read-Only Memory (CD-ROMs), and magneto-optical disks, Read-Only Memory (ROMs), Random Access Memory (RAM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), magnetic or optical cards, flash memory, or a tangible, machine-readable storage used in the transmission of information over the Internet via electrical, optical, acoustical or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.). Accordingly, the computer-readable medium includes any type of tangible machine-readable medium suitable for storing or transmitting electronic instructions or information in a form readable by a machine (e.g., a computer)

Reference throughout this specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the appearances of the phrases “in one embodiment” or “in an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

In the foregoing specification, a detailed description has been given with reference to specific exemplary embodiments. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention as set forth in the appended claims. The specification and drawings are, accordingly, to be regarded in an illustrative sense rather than a restrictive sense. Furthermore, the foregoing use of embodiment and other exemplarily language does not necessarily refer to the same embodiment or the same example, but may refer to different and distinct embodiments, as well as potentially the same embodiment.

It should also be emphasized that although certain features are listed individually in the appended dependent claims, the features may be combined with one another in a single embodiment, unless specifically excluded or logically impossible.

Wang, Min, Sun, Yang, Bai, Yun, Li, Mengshi, Chen, Yepeng

Patent Priority Assignee Title
11762572, Oct 26 2020 Samsung Electronics Co., Ltd. Method of operating storage device and method of operating storage system using the same
Patent Priority Assignee Title
4688695, Oct 23 1984 Nifco Inc. Mechanism for opening and closing a lid
5263003, Nov 12 1991 Allen-Bradley Company, Inc. Flash memory circuit and method of operation
5339445, Nov 16 1992 Harris Corporation Method of autonomously reducing power consumption in a computer sytem by compiling a history of power consumption
5430859, Jul 26 1991 SanDisk Technologies Inc Solid state memory system including plural memory chips and a serialized bus
5457746, Sep 14 1993 SPEX TECHNOLOGIES, INC System and method for access control for portable data storage media
5535336, Sep 19 1990 Intel Corporation Apparatus and method for enabling a network interface to dynamically assign an address to a connected computer and to establish a virtual circuit with another network interface
5588012, Feb 10 1992 Fujitsu Limited Apparatus and method for ensuring data in external storage system
5617421, Jun 17 1994 Cisco Technology, Inc Extended domain computer network using standard links
5680579, Nov 10 1994 Kaman Aerospace Corporation Redundant array of solid state memory devices
5690194, Oct 30 1995 Illinois Tool Works Inc. One-way pivoting gear damper
5740171, Mar 28 1996 Cisco Technology, Inc Address translation mechanism for a high-performance network switch
5742604, Mar 28 1996 Cisco Technology, Inc Interswitch link mechanism for connecting high-performance network switches
5764636, Mar 28 1996 Cisco Technology, Inc Color blocking logic mechanism for a high-performance network switch
5809285, Dec 21 1995 Hewlett Packard Enterprise Development LP Computer system having a virtual drive array controller
5812814, Feb 26 1993 Kabushiki Kaisha Toshiba Alternative flash EEPROM semiconductor memory system
5812950, Nov 27 1995 Unwired Planet, LLC Cellular telephone system having prioritized greetings for predefined services to a subscriber
5838970, Oct 04 1994 BANCTEC, INC Object-oriented computer environment and related method
5999930, Aug 02 1996 Hewlett Packard Enterprise Development LP Method and apparatus for distributed control of a shared storage volume
6035105, Jan 02 1996 Cisco Systems, Inc Multiple VLAN architecture system
6043777, Jun 10 1997 Raytheon Company Method and apparatus for global positioning system based cooperative location system
6101497, May 31 1996 EMC Corporation Method and apparatus for independent and simultaneous access to a common data set
6148414, Sep 24 1998 EMC IP HOLDING COMPANY LLC Methods and systems for implementing shared disk array management functions
6185203, Feb 18 1997 AVAGO TECHNOLOGIES GENERAL IP SINGAPORE PTE LTD Fibre channel switching fabric
6188694, Dec 23 1997 Cisco Technology, Inc Shared spanning tree protocol
6202135, Dec 23 1996 EMC IP HOLDING COMPANY LLC System and method for reconstructing data associated with protected storage volume stored in multiple modules of back-up mass data storage facility
6208649, Mar 11 1998 Cisco Technology, Inc. Derived VLAN mapping technique
6209059, Sep 25 1997 EMC IP HOLDING COMPANY LLC Method and apparatus for the on-line reconfiguration of the logical volumes of a data storage system
6219699, Jan 02 1996 Cisco Technologies, Inc. Multiple VLAN Architecture system
6219753, Jun 04 1999 International Business Machines Corporation Fiber channel topological structure and method including structure and method for raid devices and controllers
6223250, Jun 22 1995 Fujitsu Limited Card array apparatus for mounting in a personal computer
6226771, Dec 14 1998 Cisco Technology, Inc. Method and apparatus for generating error detection data for encapsulated frames
6260120, Jun 29 1998 EMC IP HOLDING COMPANY LLC Storage mapping and partitioning among multiple host processors in the presence of login state changes and host controller replacement
6266705, Sep 29 1998 Cisco Systems, Inc. Look up mechanism and associated hash table for a network switch
6269381, Jun 30 1998 EMC IP HOLDING COMPANY LLC Method and apparatus for backing up data before updating the data and for restoring from the backups
6269431, Aug 13 1998 EMC IP HOLDING COMPANY LLC Virtual storage and block level direct access of secondary storage for recovery of backup data
6295575, Jun 29 1998 EMC IP HOLDING COMPANY LLC Configuring vectors of logical storage units for data storage partitioning and sharing
6400730, Mar 10 1999 AVAGO TECHNOLOGIES INTERNATIONAL SALES PTE LIMITED Method and apparatus for transferring data between IP network devices and SCSI and fibre channel devices over an IP network
6408406, Aug 31 1999 Western Digital Corporation Hard disk drive infant mortality test
6542909, Jun 30 1998 EMC IP HOLDING COMPANY LLC System for determining mapping of logical objects in a computer system
6542961, Dec 22 1998 Hitachi, Ltd. Disk storage system including a switch
6553390, Nov 14 2000 GLOBALFOUNDRIES U S INC Method and apparatus for simultaneous online access of volume-managed data storage
6564252, Mar 11 1999 Microsoft Technology Licensing, LLC Scalable storage system with unique client assignment to storage server partitions
6647474, Apr 23 1993 EMC Corporation Remote data mirroring system using local and remote write pending indicators
6675258, Jun 30 2000 AVAGO TECHNOLOGIES INTERNATIONAL SALES PTE LIMITED Methods and apparatus for seamless firmware update and propagation in a dual raid controller system
6683883, Apr 09 2002 CALLAHAN CELLULAR L L C ISCSI-FCP gateway
6694413, Apr 19 2000 Hitachi, Ltd. Computer system and snapshot data management method thereof
6708227, Apr 24 2000 ZHIGU HOLDINGS LIMITED Method and system for providing common coordination and administration of multiple snapshot providers
6715007, Jul 13 2000 GENERAL DYNAMICS C4 SYSTEMS, INC Method of regulating a flow of data in a communication system and apparatus therefor
6728791, Jan 16 2002 PMC-SIERRA, INC RAID 1 read mirroring method for host adapters
6772231, Jun 02 2000 Hewlett Packard Enterprise Development LP Structure and process for distributing SCSI LUN semantics across parallel distributed components
6820099, Apr 13 2001 NetApp, Inc Instantaneous data updating using snapshot volumes
6847647, Sep 26 2000 QNAP SYSTEMS, INC Method and apparatus for distributing traffic over multiple switched fiber channel routes
6848759, Apr 30 2002 Illinois Tool Works Inc.; Illinois Tool Works Inc Self-closing slide mechanism with damping
6850955, Nov 22 2001 Hitachi, Ltd. Storage system and control method
6876656, Jun 15 2001 AVAGO TECHNOLOGIES GENERAL IP SINGAPORE PTE LTD Switch assisted frame aliasing for storage virtualization
6880062, Feb 13 2001 NetApp, Inc Data mover mechanism to achieve SAN RAID at wire speed
6898670, Apr 18 2000 NetApp, Inc Storage virtualization in a storage area network
6907419, Feb 27 2002 Oracle America, Inc Method, system, and product for maintaining within a virtualization system a historical performance database for physical devices
6912668, Sep 24 1998 EMC IP HOLDING COMPANY LLC Methods and systems for implementing shared disk array management functions
6952734, Aug 21 2000 QNAP SYSTEMS, INC Method for recovery of paths between storage area network nodes with probationary period and desperation repair
6976090, Apr 20 2000 Cisco Technology, Inc Differentiated content and application delivery via internet
6978300, Oct 19 2000 International Business Machines Corporation Method and apparatus to perform fabric management
6983303, Jan 31 2002 Hewlett Packard Enterprise Development LP Storage aggregator for enhancing virtualization in data storage networks
6986015, Dec 10 2001 International Business Machines Corporation Fast path caching
6986069, Jul 01 2002 SANMINA CORPORATION Methods and apparatus for static and dynamic power management of computer systems
7051056, Sep 13 2000 Veritas Technologies LLC Conservative garbage collectors that can be used with general memory allocators
7069465, Jul 26 2002 International Business Machines Corporation Method and apparatus for reliable failover involving incomplete raid disk writes in a clustering system
7073017, Feb 25 2004 GOOGLE LLC Efficient update of firmware in a disk-type storage device
7108339, Mar 26 2002 Alfit AG Self-closing drawer guides with integrated damping
7149858, Oct 31 2003 Veritas Technologies LLC Synchronous replication for system and data security
7171514, Nov 20 2003 International Business Machines Corporation Apparatus and method to control access to logical volumes using parallel access volumes
7171668, Dec 17 2001 ServiceNow, Inc Automatic data interpretation and implementation using performance capacity management framework over many servers
7174354, Jul 31 2002 Oracle International Corporation System and method for garbage collection in a computer system, which uses reinforcement learning to adjust the allocation of memory space, calculate a reward, and use the reward to determine further actions to be taken on the memory space
7200144, Oct 18 2001 CAVIUM INTERNATIONAL; MARVELL ASIA PTE, LTD Router and methods using network addresses for virtualization
7222255, Feb 28 2001 VALTRUS INNOVATIONS LIMITED System and method for network performance testing
7237045, Jun 28 2002 AVAGO TECHNOLOGIES INTERNATIONAL SALES PTE LIMITED Apparatus and method for storage processing through scalable port processors
7240188, Jul 15 2004 RAKUTEN GROUP, INC Storage system
7246260, Sep 24 1998 EMC IP HOLDING COMPANY LLC Methods and systems for implementing shared disk array management functions
7266718, Feb 24 2004 Hitachi, LTD Computer system for recovering data based on priority of the data
7269168, Jul 31 2002 Brocade Communications Systems, Inc Host bus adaptor-based virtualization switch
7277431, Oct 31 2002 AVAGO TECHNOLOGIES INTERNATIONAL SALES PTE LIMITED Method and apparatus for encryption or compression devices inside a storage area network fabric
7277948, Jan 31 2000 Fujitsu Limited Network system with dynamic service profile updating functions
7305658, May 07 2001 Microsoft Technology Licensing, LLC Method and system for application partitions
7328434, Jan 24 2002 Alcatel Canada Inc. System and method for managing configurable elements of devices in a network element and a network
7340555, Sep 28 2001 Dot Hill Systems Corporation RAID system for performing efficient mirrored posted-write operations
7346751, Apr 30 2004 Commvault Systems, Inc Systems and methods for generating a storage-related metric
7352706, Sep 12 2003 Viavi Solutions Inc Network analysis scalable analysis tool for multiple protocols
7353305, Jun 28 2002 AVAGO TECHNOLOGIES INTERNATIONAL SALES PTE LIMITED Apparatus and method for data virtualization in a storage processing device
7359321, Jan 17 2002 Juniper Networks, Inc Systems and methods for selectively performing explicit congestion notification
7363417, Dec 02 2004 DIODES INCORPORATED Optimized topographies for dynamic allocation of PCI express lanes using differential muxes to additional lanes to a host
7383381, Feb 28 2003 Oracle America, Inc Systems and methods for configuring a storage virtualization environment
7403987, Jun 29 2001 Veritas Technologies LLC Transactional SAN management
7433326, Nov 27 2002 Cisco Systems, Inc; Cisco Technology, Inc Methods and devices for exchanging peer parameters between network devices
7433948, Jan 23 2002 Cisco Systems, Inc; Cisco Technology, Inc Methods and apparatus for implementing virtualization of storage within a storage area network
7434105, Nov 07 2005 Veritas Technologies LLC Selective self-healing of memory errors using allocation location information
7441154, Sep 12 2003 Viavi Solutions Inc Network analysis tool
7447839, Sep 13 2005 GOOGLE LLC System for a distributed column chunk data store
7487321, Apr 19 2004 Cisco Technology, Inc. Method and system for memory leak detection
7500053, Nov 05 2004 Commvault Systems, Inc Method and system for grouping storage system components
7512744, Jan 31 2005 Network Appliance, Inc. Technique for enabling multiple virtual filers on a single filer to participate in multiple address spaces with overlapping network addresses
7542681, Jun 29 2004 Viavi Solutions Inc Network tap with interface for connecting to pluggable optoelectronic module
7558872, Jan 31 2002 DELL MARKETING CORPORATION Point-to-point protocol flow control extension
7587570, May 31 2006 International Business Machines Corporation System and method for providing automated storage provisioning
7631023, Nov 24 2004 Veritas Technologies LLC Performance-adjusted data allocation in a multi-device file system
7643505, Nov 30 2006 CAVIUM INTERNATIONAL; MARVELL ASIA PTE, LTD Method and system for real time compression and decompression
7654625, Jun 05 2003 Grass GmbH Pull-out guide for drawers
7657796, Sep 30 2005 Veritas Technologies LLC System and method for distributed storage verification
7668981, Mar 28 2007 Veritas Technologies LLC Storage paths
7669071, May 05 2006 Dell Products L.P. Power allocation management in an information handling system
7689384, Mar 30 2007 UNITED SERVICES AUTOMOBILE ASSOCIATION USAA Managing the performance of an electronic device
7694092, Sep 06 2006 RAKUTEN GROUP, INC Storage subsystem and back-up/recovery method
7697554, Dec 27 2005 EMC IP HOLDING COMPANY LLC On-line data migration of a logical/virtual storage array by replacing virtual names
7706303, Jun 26 2006 Cisco Technology, Inc.; Cisco Technology, Inc Port pooling
7707481, May 16 2006 Pitney Bowes Inc.; Pitney Bowes Inc System and method for efficient uncorrectable error detection in flash memory
7716648, Aug 02 2005 Oracle America, Inc Method and apparatus for detecting memory leaks in computer systems
7752360, May 16 2007 Cisco Technology, Inc Method and system to map virtual PCIe I/O devices and resources to a standard I/O bus
7757059, Jun 29 2006 EMC IP HOLDING COMPANY LLC Virtual array non-disruptive management data migration
7774329, Dec 22 2006 Amazon Technologies, Inc Cross-region data access in partitioned framework
7774839, Nov 03 2003 RIVERBED TECHNOLOGY LLC Feedback mechanism to minimize false assertions of a network intrusion
7793138, Dec 21 2005 Cisco Technology, Inc. Anomaly detection for storage traffic in a data center
7840730, Jun 27 2008 Microsoft Technology Licensing, LLC Cluster shared volumes
7843906, Feb 13 2004 Oracle International Corporation Storage gateway initiator for fabric-backplane enterprise servers
7895428, Sep 28 2007 International Business Machines Corporation Applying firmware updates to servers in a data center
7904599, Mar 28 2003 Cisco Technology, Inc.; CISCO TECHNOLGY, INC Synchronization and auditing of zone configuration data in storage-area networks
7930494, Sep 10 2007 Cisco Technology, Inc. Storage area network (SAN) switch multi-pass erase of data on target devices
7975175, Jul 09 2008 Oracle America, Inc Risk indices for enhanced throughput in computing systems
7979670, Jan 24 2008 Quantum Corporation Methods and systems for vectored data de-duplication
7984259, Dec 17 2007 NetApp, Inc Reducing load imbalance in a storage system
8031703, Aug 14 2008 Dell Products, LP System and method for dynamic maintenance of fabric subsets in a network
8032621, Jan 03 2006 EMC IP HOLDING COMPANY LLC Methods and apparatus providing root cause analysis on alerts
8051197, Mar 29 2002 AVAGO TECHNOLOGIES INTERNATIONAL SALES PTE LIMITED Network congestion management systems and methods
8086755, Nov 29 2004 EGENERA, INC Distributed multicast system and method in a network
8161134, Sep 20 2005 Cisco Technology, Inc. Smart zoning to enforce interoperability matrix in a storage area network
8196018, Nov 19 2004 International Business Machines Corporation Enhanced error identification with disk array parity checking
8205951, Nov 04 2009 Knape & Vogt Manufacturing Company Closing device for drawers
8218538, Feb 13 2004 Oracle International Corporation Storage gateway configuring and traffic processing
8230066, Nov 04 2002 International Business Machines Corporation Location independent backup of data from mobile and stationary computers in wide regions regarding network and server activities
8234377, Jul 22 2009 Amazon Technologies, Inc.; Amazon Technologies, Inc Dynamically migrating computer networks
8266238, Dec 27 2006 Intel Corporation Memory mapped network access
8272104, Feb 25 2011 Lianhong Art Co., Ltd. Hinge-slide cover mounting structure using a sheet metal bracket mechanism
8274993, Jun 16 2006 Cisco Technology, Inc.; Cisco Technology, Inc Fibre channel dynamic zoning
8290919, Aug 27 2010 Disney Enterprises, Inc.; DISNEY ENTERPRISES, INC System and method for distributing and accessing files in a distributed storage system
8297722, Jun 03 2008 Rev-A-Shelf Company, LLC Soft close drawer assembly and bracket
8301746, Jan 26 2010 International Business Machines Corporation Method and system for abstracting non-functional requirements based deployment of virtual machines
8335231, Apr 08 2005 Cisco Technology, Inc. Hardware based zoning in fibre channel networks
8341121, Sep 28 2007 EMC IP HOLDING COMPANY LLC Imminent failure prioritized backup
8345692, Apr 27 2010 Cisco Technology, Inc. Virtual switching overlay for cloud computing
8352941, Jun 29 2009 EMC IP HOLDING COMPANY LLC Scalable and secure high-level storage access for cloud computing platforms
8392760, Oct 14 2009 Microsoft Technology Licensing, LLC Diagnosing abnormalities without application-specific knowledge
8442059, Sep 30 2008 VIOLIN SYSTEMS LLC Storage proxy with virtual ports configuration
8479211, Jun 29 2010 Amazon Technologies, Inc Dynamic resource commitment management
8495356, Dec 31 2010 International Business Machines Corporation System for securing virtual machine disks on a remote shared storage subsystem
8514868, Jun 19 2008 VIDEOLABS, INC Cloud computing gateway, cloud computing hypervisor, and methods for implementing same
8532108, Sep 30 2009 Alcatel Lucent Layer 2 seamless site extension of enterprises in cloud computing
8560663, Sep 30 2011 TELEFONAKTIEBOLAGET L M ERICSSON PUBL Using MPLS for virtual private cloud network isolation in openflow-enabled cloud computing
8619599, Sep 17 2010 CAVIUM INTERNATIONAL; MARVELL ASIA PTE, LTD Packet processor verification methods and systems
8626891, Nov 03 2010 ServiceNow, Inc Configured management-as-a-service connect process based on tenant requirements
8630983, Aug 27 2011 Accenture Global Services Limited Backup of data across network of devices
8660129, Feb 02 2012 Cisco Technology, Inc Fully distributed routing over a user-configured on-demand virtual network for infrastructure-as-a-service (IaaS) on hybrid cloud networks
8661299, May 31 2013 Microsoft Technology Licensing, LLC Detecting abnormalities in time-series data from an online professional network
8677485, Jul 13 2012 Hewlett Packard Enterprise Development LP Detecting network anomaly
8683296, Dec 30 2011 STREAMSCALE, INC Accelerated erasure coding system and method
8706772, Dec 30 2010 SAP SE Strict tenant isolation in multi-tenant enabled systems
8719804, May 05 2010 Microsoft Corporation Managing runtime execution of applications on cloud computing systems
8725854, Jan 23 2002 Cisco Technology, Inc. Methods and apparatus for implementing virtualization of storage within a storage area network
8768981, Aug 27 2010 Disney Enterprises, Inc.; DISNEY ENTERPRISES, INC System and method for distributing and accessing files in a distributed storage system
8775773, Aug 26 2011 VMware LLC Object storage system
8793372, Feb 03 2012 International Business Machines Corporation Allocation and balancing of storage resources
8805918, Sep 11 2002 Cisco Systems, Inc; Cisco Technology, Inc Methods and apparatus for implementing exchange management for virtualization of storage within a storage area network
8805951, Feb 08 2011 EMC IP HOLDING COMPANY LLC Virtual machines and cloud storage caching for cloud computing applications
8832330, May 23 2013 Hewlett Packard Enterprise Development LP Analysis of storage system latency by correlating activity of storage system components with latency measurements
8855116, May 15 2012 Cisco Technology, Inc. Virtual local area network state processing in a layer 2 ethernet switch
8856339, Apr 04 2012 Cisco Technology, Inc. Automatically scaled network overlay with heuristic monitoring in a hybrid cloud environment
8868474, Aug 01 2012 Empire Technology Development LLC Anomaly detection for cloud monitoring
8887286, Nov 06 2009 Adobe Inc Continuous anomaly detection based on behavior modeling and heterogeneous information analysis
8898385, Sep 09 2011 AVAGO TECHNOLOGIES GENERAL IP SINGAPORE PTE LTD Methods and structure for load balancing of background tasks between storage controllers in a clustered storage environment
8909928, Jun 02 2010 VMware LLC Securing customer virtual machines in a multi-tenant cloud
8918510, Apr 27 2012 Hewlett Packard Enterprise Development LP Evaluation of cloud computing services
8918586, Sep 28 2012 EMC IP HOLDING COMPANY LLC Policy-based storage of object fragments in a multi-tiered storage system
8924720, Sep 27 2012 Intel Corporation Method and system to securely migrate and provision virtual machine images and content
8930747, Mar 30 2012 11:11 SYSTEMS, INC Private cloud replication and recovery
8935500, Sep 24 2009 VMware LLC Distributed storage resource scheduler and load balancer
8949677, May 23 2012 Amazon Technologies, Inc Detecting anomalies in time series data
8996837, Mar 15 2013 EMC IP HOLDING COMPANY LLC Providing multi-tenancy within a data storage apparatus
9003086, Oct 27 2012 TWITTER, INC Dynamic distribution of replicated data
9007922, May 23 2013 Juniper Networks, Inc. Systems and methods for testing and analyzing controller-based networks
9009427, Dec 26 2001 Cisco Technology, Inc Mirroring mechanisms for storage area networks and network based virtualization
9009704, Sep 07 2012 Red Hat, Inc. Application partitioning in a multi-tenant platform-as-a-service environment in a cloud computing system
9075638, Aug 14 2012 Atlassian Pty Ltd Efficient hosting of virtualized containers using read-only operating systems
9141554, Jan 18 2013 Cisco Technology, Inc Methods and apparatus for data processing using data compression, linked lists and de-duplication techniques
9141785, Aug 03 2011 CloudByte, Inc. Techniques for providing tenant based storage security and service level assurance in cloud storage environment
9164795, Mar 30 2012 Amazon Technologies, Inc. Secure tunnel infrastructure between hosts in a hybrid network environment
9176677, Sep 28 2010 EMC IP HOLDING COMPANY LLC Virtual provisioning space reservation
9201704, Apr 05 2012 Cisco Technology, Inc. System and method for migrating application virtual machines in a network environment
9203784, Apr 24 2012 Cisco Technology, Inc.; Cisco Technology, Inc Distributed virtual switch architecture for a hybrid cloud
9207882, Dec 18 2012 Cisco Technology, Inc. System and method for in-band LUN provisioning in a data center network environment
9207929, Aug 26 2013 Hitachi, LTD Integrated system and firmware update method
9213612, Sep 29 2008 Cisco Technology, Inc Method and system for a storage area network
9223564, Jan 26 2012 AVAGO TECHNOLOGIES INTERNATIONAL SALES PTE LIMITED Update systems responsive to ongoing processing at a storage system
9223634, May 02 2012 Cisco Technology, Inc. System and method for simulating virtual machine migration in a network environment
9244761, Jun 25 2013 Microsoft Technology Licensing, LLC Erasure coding across multiple zones and sub-zones
9250969, Aug 30 2011 The Governing Council of the University of Toronto Tagging a copy of memory of a virtual machine with information for fetching of relevant portions of the memory
9264494, Oct 21 2013 International Business Machines Corporation Automated data recovery from remote data object replicas
9270754, Jun 06 2013 Cisco Technology, Inc. Software defined networking for storage area networks
9280487, Jan 18 2013 Cisco Technology, Inc. Methods and apparatus for data processing using data compression, linked lists and de-duplication techniques
9292460, Jun 20 2012 International Business Machines Corporation Versatile lane configuration using a PCIe PIE-8 interface
9292465, Dec 21 2011 Intel Corporation Dynamic link width adjustment
9304815, Jun 13 2013 Amazon Technologies, Inc Dynamic replica failure detection and healing
9313048, Apr 04 2012 Cisco Technology, Inc. Location aware virtual service provisioning in a hybrid cloud environment
9374270, Jun 06 2012 Juniper Networks, Inc Multicast service in virtual networks
9378060, Aug 28 2012 Oracle International Corporation Runtime co-location of executing logic and frequently-accessed application data
9396251, Jan 07 2016 ServiceNow, Inc Detecting and tracking virtual containers
9448877, Mar 15 2013 Cisco Technology, Inc Methods and apparatus for error detection and correction in data storage systems using hash value comparisons
9471348, May 31 2010 Microsoft Technology Licensing, LLC Applying policies to schedule network bandwidth among virtual machines
9501473, Dec 21 2004 Veritas Technologies LLC Workflow process with temporary storage resource reservation
9503523, Sep 20 2013 Cisco Technology, Inc. Hybrid fibre channel storage with end-to-end storage provisioning and external connectivity in a storage area network environment
9565110, Mar 29 2002 AVAGO TECHNOLOGIES INTERNATIONAL SALES PTE LIMITED Network congestion management systems and methods
9575828, Jul 08 2015 Cisco Technology, Inc. Correctly identifying potential anomalies in a distributed storage system
9582366, Nov 21 2014 International Business Machines Corporation Detecting and sparing of optical PCIE cable channel attached IO drawer
9582377, Mar 19 2015 Amazon Technologies, Inc Dynamic sizing of storage capacity for a remirror buffer
9614763, Mar 26 2015 Cisco Technology, Inc. Scalable handling of BGP route information in VXLAN with EVPN control plane
9658868, Jun 19 2008 VIDEOLABS, INC Cloud computing gateway, cloud computing hypervisor, and methods for implementing same
9658876, Apr 04 2012 Cisco Technology, Inc. Location-aware virtual service provisioning in a hybrid cloud environment
9733868, Sep 11 2002 Cisco Technology, Inc. Methods and apparatus for implementing exchange management for virtualization of storage within a storage area network
9763518, Aug 29 2014 Cisco Technology, Inc. Systems and methods for damping a storage system
9830240, May 14 2015 Cisco Technology, Inc.; Cisco Technology, Inc Smart storage recovery in a distributed storage system
9842075, Sep 12 2014 Amazon Technologies, Inc Presenting multiple endpoints from an enhanced PCI express endpoint device
9853873, Jan 10 2015 Cisco Technology, Inc. Diagnosis and throughput measurement of fibre channel ports in a storage area network environment
20020049980,
20020053009,
20020073276,
20020083120,
20020095547,
20020103889,
20020103943,
20020112113,
20020120741,
20020138675,
20020156971,
20030023885,
20030026267,
20030055933,
20030056126,
20030065986,
20030084359,
20030118053,
20030131105,
20030131165,
20030131182,
20030140134,
20030140210,
20030149763,
20030154271,
20030159058,
20030174725,
20030189395,
20030210686,
20040024961,
20040030857,
20040039939,
20040054776,
20040057389,
20040059807,
20040088574,
20040117438,
20040123029,
20040128470,
20040128540,
20040153863,
20040190901,
20040215749,
20040230848,
20040250034,
20050033936,
20050036499,
20050050211,
20050050270,
20050053073,
20050055428,
20050060574,
20050060598,
20050071851,
20050076113,
20050091426,
20050114611,
20050114615,
20050117522,
20050117562,
20050138287,
20050169188,
20050185597,
20050188170,
20050198523,
20050235072,
20050283658,
20060015861,
20060015928,
20060034302,
20060045021,
20060075191,
20060098672,
20060117099,
20060136684,
20060184287,
20060198319,
20060215297,
20060230227,
20060242332,
20060251111,
20070005297,
20070067593,
20070079068,
20070091903,
20070094465,
20070101202,
20070121519,
20070136541,
20070162969,
20070211640,
20070214316,
20070250838,
20070258380,
20070263545,
20070276884,
20070283059,
20080016412,
20080034149,
20080052459,
20080059698,
20080114933,
20080126509,
20080126734,
20080168304,
20080201616,
20080244184,
20080256082,
20080267217,
20080288661,
20080294888,
20090006708,
20090063766,
20090083484,
20090089567,
20090094380,
20090094664,
20090125694,
20090193223,
20090201926,
20090222733,
20090240873,
20090282471,
20090323706,
20100011365,
20100030995,
20100046378,
20100083055,
20100174968,
20100318609,
20100318837,
20110010394,
20110022691,
20110029824,
20110035494,
20110075667,
20110087848,
20110119556,
20110142053,
20110161496,
20110173303,
20110228679,
20110231899,
20110239039,
20110252274,
20110255540,
20110276584,
20110276675,
20110276951,
20110299539,
20110307450,
20110313973,
20120023319,
20120030401,
20120054367,
20120072578,
20120072985,
20120075999,
20120084445,
20120084782,
20120096134,
20120130874,
20120131174,
20120134672,
20120144014,
20120159112,
20120167094,
20120173581,
20120173589,
20120177039,
20120177041,
20120177042,
20120177043,
20120177044,
20120177045,
20120177370,
20120179909,
20120201138,
20120210041,
20120254440,
20120257501,
20120265976,
20120281706,
20120297088,
20120303618,
20120311106,
20120311568,
20120320788,
20120324114,
20120331119,
20130003737,
20130013664,
20130028135,
20130036212,
20130036213,
20130036449,
20130054888,
20130061089,
20130067162,
20130080823,
20130080825,
20130086340,
20130100858,
20130111540,
20130138816,
20130138836,
20130139138,
20130144933,
20130152076,
20130152175,
20130163426,
20130163606,
20130179941,
20130182712,
20130185433,
20130191106,
20130198730,
20130208888,
20130212130,
20130223236,
20130238641,
20130266307,
20130268922,
20130275470,
20130297655,
20130297769,
20130318134,
20130318288,
20140006708,
20140016493,
20140019684,
20140025770,
20140029441,
20140029442,
20140039683,
20140040473,
20140040883,
20140047201,
20140053264,
20140059187,
20140059266,
20140086253,
20140089273,
20140095556,
20140096249,
20140105009,
20140108474,
20140109071,
20140112122,
20140123207,
20140156557,
20140164666,
20140164866,
20140172371,
20140173060,
20140173195,
20140173579,
20140189278,
20140198794,
20140211661,
20140215265,
20140215590,
20140219086,
20140222953,
20140229790,
20140244585,
20140244897,
20140245435,
20140269390,
20140281700,
20140297941,
20140307578,
20140317206,
20140324862,
20140325208,
20140331276,
20140348166,
20140351654,
20140355450,
20140366155,
20140376550,
20150003450,
20150003458,
20150003463,
20150010001,
20150016461,
20150030024,
20150046123,
20150063353,
20150067001,
20150067226,
20150082432,
20150092824,
20150120907,
20150121131,
20150127979,
20150142840,
20150169313,
20150180672,
20150205974,
20150207763,
20150212755,
20150222444,
20150229546,
20150248366,
20150248418,
20150254003,
20150254088,
20150261446,
20150263993,
20150269048,
20150277804,
20150278040,
20150281067,
20150303949,
20150339246,
20150341237,
20150341239,
20150358136,
20150379150,
20160004611,
20160011936,
20160011942,
20160054922,
20160062820,
20160070652,
20160087885,
20160088083,
20160119159,
20160119421,
20160139820,
20160149639,
20160205189,
20160210161,
20160231928,
20160274926,
20160285760,
20160292359,
20160294983,
20160306768,
20160334998,
20160366094,
20160378624,
20160380694,
20170010874,
20170010930,
20170019475,
20170068630,
20170168970,
20170177860,
20170212858,
20170273019,
20170277655,
20170337097,
20170340113,
20170371558,
20180032462,
20180097707,
TW1566104,
WO2016003408,
//////
Executed onAssignorAssigneeConveyanceFrameReelDoc
Feb 06 2017SUN, YANGCisco Technology, IncASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS 0414030958 pdf
Feb 07 2017BAI, YUN Cisco Technology, IncASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS 0414030958 pdf
Feb 07 2017LI, MENGSHICisco Technology, IncASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS 0414030958 pdf
Feb 07 2017WANG, MINCisco Technology, IncASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS 0414030958 pdf
Feb 13 2017CHEN, YEPENGCisco Technology, IncASSIGNMENT OF ASSIGNORS INTEREST SEE DOCUMENT FOR DETAILS 0414030958 pdf
Feb 28 2017Cisco Technology, Inc.(assignment on the face of the patent)
Date Maintenance Fee Events
Jan 12 2024M1551: Payment of Maintenance Fee, 4th Year, Large Entity.


Date Maintenance Schedule
Jul 14 20234 years fee payment window open
Jan 14 20246 months grace period start (w surcharge)
Jul 14 2024patent expiry (for year 4)
Jul 14 20262 years to revive unintentionally abandoned end. (for year 4)
Jul 14 20278 years fee payment window open
Jan 14 20286 months grace period start (w surcharge)
Jul 14 2028patent expiry (for year 8)
Jul 14 20302 years to revive unintentionally abandoned end. (for year 8)
Jul 14 203112 years fee payment window open
Jan 14 20326 months grace period start (w surcharge)
Jul 14 2032patent expiry (for year 12)
Jul 14 20342 years to revive unintentionally abandoned end. (for year 12)