WO2003088594A1 - A method for providing redundancy for channel adapter failure - Google Patents

Abstract

The invention relates to a method for providing improved reliability of any node attaching to an InfiniBand fabric, the method comprising the steps of: a) providing a first and a second physical Channel Adapter having a first and a second number of ports, b) providing program means for registering the first and second physical Channel Adapters as one logical Channel Adapter having a number of first and second ports, c) providing first and second caching means for storing first and second control information for the first and second Channel Adapter, d) providing system memory means for storing first and second control information, and e) providing means for copying the first control information from the system memory to the second caching means in case of a failure of the first Channel Adapter and for initiating an Automatic Path Migration from the first number of ports to the second number of ports.

Description

D E S C R I P T I O N

A method for providing redundancy for Channel Adapter failure

Field of the invention

The present invention relates generally to digital network communication, and specifically to provide improved reliability of a computer system or any other node attaching to an InfiniBand subnet or fabric .

Background and prior art

The computer industry is moving towards fast, packetized, serial input/output (I/O) interconnect architectures, in which computing hosts and peripherals are linked by a switching network, commonly referred to as a switching fabric. A number of architectures of this type have been proposed, finally leading to the InfiniBand (IB) architecture, which has been promoted by a consortium led by a group of industry leaders (including Intel, Sun Microsystems, Hewlett Packard, IBM, Compaq, Dell and Microsoft) . The IB architecture is described in detail in the InfiniBand Architecture Specification, Release 1.0. a, which is available from the InfiniBand Trade Association at www.infinibandta.org and is incorporated herein by reference.

InfiniBand technology works by connecting Host Channel Adapters (HCAs) to other HCAs or to Target Channel Adapters (TCAs) . The HCAs tend to be located near the servers' CPUs and memory, while the TCAs tend to be located near the systems' disk storage and other peripherals. Switches or Routers may be located between HCAs and TCAs, directing data packets to the correct TCA destination based on information that is contained in the data packets themselves .

The connection between HCAs and TCAs (or other HCAs) is either an InfiniBand point-to-point link or a switch or router, which allows to create an uniform InfiniBand subnet or fabric environment, respectively. One of the key points of this switch is that it allows packets of information (or data) to be managed based on variables, such as service level (SL) and a destination identifier (DLID/DGID) .

Instead of the traditional memory mapped I/O interface bus, the InfiniBand architecture is developed with a serial, switched fabric approach. This switched nature allows for low- latency, high-bandwidth characteristics of the InfiniBand architecture. Clustered systems and networks require a connectivity standard that allows for fault tolerant interconnects .

This requirement is met by the InfiniBand architecture which incorporates advanced fault detection and correction mechanisms. One example of an InfiniBand compliant product is the IBM PCI-X to InfiniBand Host Channel Adapter which allows connectivity between a host's PCI-X bus and an InfiniBand network. The dual InfiniBand ports provide the capability to support Automatic Path Migration and single or multiple subnet connections with a single HCA device.

Automatic Path Migration (APM) is a means to continue processing in the case of a Host Channel Adapter (HCA) or Target Channel Adapter (TCA) port failure or a failure in a subnet or fabric. In other words, APM provides a redundancy mechanism in case of a port failure of a HCA or TCA or a link, switch, or router failure in a subnet or fabric. However, InfiniBand does only define a redundancy mechanism in case only one or more ports of an HCA fail but not in case the entire HCA fails. Summary of the invention

The invention provides for a redundancy mechanism for a Channel Adapter (CA) , such as a Host Channel Adapter (HCA) or a Target Channel Adapter (TCA) , in case of a complete Channel Adapter failure. It is a particular advantage of the invention that the redundancy mechanism fits seamlessly into the InfiniBand architecture and relies on the fault detection and correction methods which are specified in the InfiniBand architecture.

It is a particular advantage of the invention that a device which is designed in accordance with the principles of the invention can be fully compliant with the InfiniBand architecture and still offers a redundancy mechanism for complete Channel AdapterChannel Adapter failure.

In accordance with a preferred embodiment of the invention at least two physical Host Channel Adapters are provided. The two physical Host Channel Adapters are registered as one logical Host Channel Adapter in terms of the InfiniBand architecture. Both Host Channel Adapters have dedicated caching means which cooperate with the system memory for storaging Queue Pair (QP) control information in terms of Queue Pair Control Blocks (QPCBs) . In case of a complete failure of one of the physical Host Channel Adapters a copy of the respective QPCBs is provided for the remaining still operating physical Host Channel Adapter.

In accordance with a further preferred embodiment of the invention, write-through caches are utilized. In this case the QPCBs stored in system memory are an exact copy of the dedicated caches of each physical Host Channel Adapters.

In accordance with a further preferred embodiment of the invention, write-back caches are used for Host Channel Adapters. In this case the system memory is synchronized with the caches at certain times and does not always reflect the actual contents of the caches at any given point in time.

In case of a complete physical Host Channel Adapter failure a content of the cache belonging to the failing Host Channel Adapter is also lost. The system memory copy of the QPCBs is provided to the cache of the remaining physical Host Channel Adapter.

This copy may contain stale data. In order to re-synchronize the communication and to bring the QPCB information up-to- date, the fault detection and correction mechanisms provided by the InfiniBand architecture are utilized.

Although a preferred embodiment as described here refers to a Host Channel Adapter (HCA) , the invention encompasses Channel Adapters (CAs) in general which includes HCAs and TCAs according to the InfiniBand architecture.

Brief description of the drawings

In the following preferred embodiments of the invention will be explained in greater detail by making reference to the drawings, in which:

Figure 1 shows a block diagram illustrating the operation of a single Host Channel Adapter with a dedicated cache memory,

Figure 2 shows a block diagram of a computer system having a redundant logical Host Channel Adapter for the case of a write-through cache,

Figure 3 shows the block diagram of figure 2 after the replacement of the failing Host Channel Adapter by the redundancy mechanism, Figure 4 illustrates the discrepancy which can occur between the state of a cache and system memory for a write-back cache,

Figures 5 to 7 illustrate the utilization of the fault detection and correction methods provided by the InfiniBand architecture for implementing the redundancy mechanism of the invention in case of the use of write-back caches .

Detailed description

Figure 1 shows a computer system having a Host Channel Adapter 1 comprising a cache 2 and a cache directory 3. Further the computer system has system memory 4.

By means of the system memory 4, cache directory 3 and cache 2 the address space for the Queue Pair Control Blocks (QPCBs) is virtualized. In case there exist more than one Host Channel Adapter 1 the Queue Pair (QP) numbers between different Host Channel Adapters must be disjoint.

All Queue Pair Control Blocks reside in system memory 4 and are loaded (unloaded) into the Host Channel Adapter cache 2 when used (no longer used) . A failure of the Host Channel Adapter 1 does not prevent to access this data from a physically different Host Channel Adapter.

Figure 2 shows a block diagram of a preferred embodiment of the invention which illustrates the redundancy mechanism. Like elements of the computer system of figure 2 and the computer system of figure 1 are designated by the same reference numerals .

The computer system has a physical Host Channel Adapter 1 with one or more ports 6 and a physical Host Channel Adapter 7 with one or more ports 8. The ports 6 and 8 are connected to an InfiniBand subnet or fabric 9.

The two physical Host Channel Adapters 1 and 7 are recognized as one single Host Channel Adapter according to the InfiniBand architecture. Thereby a logical Host Channel Adapter 10 is constituted. The logical Host Channel Adapter 10 has the ports 6 and 8 of the physical Host Channel Adapters 1 and 7.

The physical Host Channel Adapter 1 has the cache 2 and the physical Host Channel Adapter 7 has the cache 11. Both caches 2 and 11 are organized as write-through caches.

Further the computer system has system memory 4 for storage of Queue Pair control block data for the physical Host Channel Adapters 1 and 7. The Queue Pair numbers of the different physical Host Channel Adapters 1 and 7 are disjoint.

There is no further restriction on the Queue Pair numbers . For ease of explanation it is assumed in the following that the physical Host Channel Adapter 1 has a block 12 of Queue Pair Control Blocks QPCB_2 to QPCB_m and that the physical Host Channel Adapter 7 has a block 13 of Queue Pair Control Blocks QPCB_m+l to QPCB_n. QPCB_0 and QPCB_1 are used for subnet management purposes and are not further considered here.

As the caches 2 and 11 are write-through caches, the QPCB data in the system memory 4 is identical to the data in the caches 2 and 11.

Figure 3 illustrates the redundancy mechanism for dealing with a complete failure of the physical Host Channel Adapter 1 of figure 2.

First there is a complete hardware failure of the physical Host Channel Adapter 1 including ports 6. This hardware failure invokes the Automatic Path Migration defined by the InfiniBand architecture. This way one or more communication paths involving ports 6 of Host Channel Adapter 1 are migrated to the ports 8 of the remaining physical Host Channel Adapter 7.

This procedure fully relies on the Automatic Path Migration (APM) mechanism provided by InfiniBand because in terms of the InfiniBand architecture the Host Channel Adapters 1 and 7 are not present as two separate (physical) Host Channel Adapters but only as a single (logical) Host Channel Adapter 10 providing ports 6 and 8.

A copy of a QPCB in block 12 is made into the cache 11 as needed. As block 12 contains an exact copy of the contents of cache 2, no further recovery mechanisms are required.

Figure 4 illustrates the situation in the case of write-back caches. If a write-back cache 14 is used rather than a write- through cache, the QPCBs stored in system memory 4 do not always reflect the up-to-date state of the QPCB data in cache 14. This is the reason why in case of using a write-back cache an additional fault detection and correction method of the InfiniBand architecture needs to be invoked.

Figure 5 illustrates the situation before failover of one of the physical Host Channel Adapters.

At the sender's side a sequence 15 of outstanding packet sequence numbers (PSNs) is stored in the system memory 4. One of the outstanding PSNs having the sequence number Sm is the next packet to be transmitted according to the information stored in the system memory 4.

Further a sequence 16 of outstanding PSNs is stored in the local cache memory, which is a write-back cache. This sequence 16 represents the up-to-date sequence of transmitted packets. Hence the sequence number Sn is up-to-date in this sequence

16

At the receiver's side there is a sequence 17 of PSNs. The next packet expected by the receiver is the packet with the sequence number Rn. After failover of one of the physical Host Channel Adapters the sequence 15 remains unaffected as it is stored in system memory 4.

A copy of the sequence 15 is provided to the remaining still operating physical Host Channel Adapter. This way the sequence 16 of the cache of the failing Host Channel Adapter is replaced by the sequence 15 in the cache of the remaining still operating physical Host Channel Adapter.

This is why the next packet which is sent from the Host Channel Adapter is the packet with the stale sequence number Sm which had been sent before failover. The receiver returns an acknowledgement (ACK) to the sending Host Channel Adapter and discards the packet.

In response the Host Channel Adapter sends the next packet identified in the sequence 15. This way the sequence 15 is processed until it reaches the original state of the sequence 16 before failover. After this state is reached the normal system operation continues normally.

Figure 6 shows a situation where the packet with the next sequence number Sn of the sequence 16 has been sent from the Host Channel Adapter. After sending this packet there is a hardware failure of the Host Channel Adapter. Still the receiver receives the expected packet with the sequence number Rn = Sn.

In response the receiver sends an acknowledgement for having received the packet with the sequence number Sn to the logical Host Channel Adapter. The logical Host Channel Adapter, i.e. the remaining still operating physical Host Channel Adapter, interprets this acknowledgement as a ghost acknowledgement and ignores it. Then the sender sends the packet with the sequence number Sm of the sequence 15 as in the scenario shown in figure 5.

Figure 7 shows a scenario where the Host Channel Adapter acts as a receiver. A sequence 18 of PSNs is stored in the system memory and an up-to-date sequence 19 in cache memory. Further there is a sequence 20 of outstanding PSNs to be sent by the sender. This is the situation before failover.

After failover the sequence 19 is replaced by the sequence 18, i.e. a copy of the sequence 18 is provided from system memory to the cache of the remaining still operating physical Host Channel Adapter part of the logical Host Channel Adapter. The sequence 20 remains unchanged.

When the Host Channel Adapter receives the packet with the next sequence number Sn of the sequence 20 from the sender this does not match the expected sequence number Rm of the sequence 18. In response the Host Channel Adapter returns a negative acknowledgement (NAK) to the sender. This indicates to the sender of the packet that packets have been lost in the subnet or fabric and that the sender has to resend those packets .

The negative acknowledgement response has a parameter which indicates which packet was the last one successfully received. This way the sequence 20 is set back to the sequence number Sn = Rm, where Rm is the expected sequence number of sequence 18. L I S T O F R E F E R E N C E N U M E R A L S

physical Host Channel Adapter 1 1

HCA 1 cache 2

HCA 1 cache directory 3 system memory 4

HCA 1 ports 6 physical Host Channel Adapter 2 7

HCA 2 ports 8

InfiniBand fabric 9 logical Host Channel Adapter 10

HCA 2 cache 11

QPCB block originally assigned to HCA 1 12

QPCB block originally assigned to HCA 2 13

HCA 1 or 2 cache 14

PSN sequence 15

PSN sequence 16

PSN sequence 17

PSN sequence 18

PSN sequence 19

PSN sequence 20

Claims

C L A I M S

1. A method for providing redundancy for a Channel Adapter failure, the method comprising the steps of:

providing a first physical Channel Adapter having a first number of ports and a second physical Channel Adapter having a second number of ports,

providing program means for registering the first and second physical Channel Adapters as one logical Channel Adapter having a number of first and second number of ports,

providing first caching means for storing first control information for the first Channel Adapter and second caching means for storing second control information for the second Channe1 Adapter,

providing system memory means for storing first and second control information, and

providing means for copying the first control information from the system memory to the second caching means in case of a failure of the first Channel Adapter and for initiating an Automatic Path Migration from the first number of ports to one or more of the second number of ports.

2. The method of claim 1, whereby the first and second caching means are operated as write-through caches.

3. The method of claim 1, whereby the first and second caching means are operated as store back caches.

4. The method of claim 3, further comprising providing means for using an InfiniBand type fault detection and correction method to re-synchronize a communication between one of the second ports and another InfinBandChannel Adapter.

5. A computer program product for performing a method in accordance with anyone of the preceding claims 1 to 4.

6. A computer system comprising:

a first physical Channel Adapter (1) having a first number of ports (6) and a second physical Channel Adapter (7) having a second number of ports (8),

means for registering the first and second physical Channel Adapters as one logical Channel Adapter (10) according to the InfiniBand type architecture, the logical Channel Adapter having a number of first and second ports,

first caching means (2) for storing of first control information for the first Channel Adapter and second caching means (11) for storing of second control information for the second Channel Adapter,

system memory means (4) for storing of first control information (12) and second control information (13), and

means for copying the first control information from the system memory means (12) to the second caching means (11) in case of a failure of the first Channel Adapter and for initiating an InfiniBand type Automatic Path Migration from the first number of ports (6) to one or more of the second number of ports (8) .

7. The computer system of claim 6, the first and second caching means being adapted to be operated as write-through caches .

8. The computer system of claim 6, the first and second caching means being adapted to be operated as store-back caches .

9. The computer system of claims 6, 7, or 8 further comprising means for using an InfiniBand type fault detection and correction method to re-synchronize a communication between one of the second number of ports and another InfiniBand Channel Adapter.