US20040148460A1 - Integrated-circuit implementation of a storage-shelf router and a path controller card for combined use in high-availability mass-storage-device shelves that may be incorporated within disk arrays, and a storage-shelf-interface tunneling method and system - Google Patents

Abstract

An integrated circuit implementing a storage-shelf router, used in combination with path controller cards and optionally with other storage-shelf routers, to interconnect SATA disks within a storage shelf or disk array to a high-bandwidth communications medium, such as an FC arbitrated loop. Various embodiments of the present invention provide a tunneling mechanism through the storage-shelf interface provided by one or more storage-shelf routers within a storage shelf to enable external processing entities to directly access various components within the storage shelf. In one embodiment of the present invention, a WRITE-BUFFER command and a READ-BUFFER command are added to the command interface supported by storage-shelf router. These commands are exchanged via the FCP protocol over the fiber channel in the same manner that SCSI commands are packaged within the FCP protocol. In certain cases, the information packaged within the WRITE-BUFFER and READ-BUFFER commands is directly exchanged with internal disk drives. In other cases, the information is extracted and written to various data structures maintained within internal components of the storage shelf. The WRITE-BUFFER and READ-BUFFER commands essential provide a breach or tunnel in the high-availability storage-shelf virtual interface provided by one or more storage-shelf routers within a storage shelf, allowing an external processing entity to exchange information through the tunnel without intervention by, but facilitated and supported by, the one or more storage-shelf routers within the high-availability storage shelf.

Description

CROSS-REFERENCE TO RELATED APPLICATION
• This application is a continuation-in-part of application Ser. No. 10/602,509, filed Jun. 23, 2003, which is a continuation-in-part of application Ser. No. 10/341,835, filed Jan. 13, 2003.[0001]
TECHNICAL FIELD
• The present invention relates to disk arrays and other mass-storage-devices composed of numerous individual mass-storage-devices and, in particular, to an integrated-circuit implementation of a storage-shelf router and to path controller cards that may be together employed, along with one or more additional storage-shelf routers, and to a tunneling method and system by which an external processing entity, such as a disk-array controller, can directly access disk drives within a storage shelf and various data structures stored within the disk drives and within the storage-shelf routers and path-controller cards.[0002]
BACKGROUND OF THE INVENTION
• The fibre channel (“FC”) is an architecture and protocol for a data communication network that interconnects a number of different combinations of computers and peripheral devices. The FC supports a variety of upper-level protocols, including the small computer systems interface (“SCSI”) protocol. A computer or peripheral device is linked to the network through an FC port and copper wires or optical fibers. An FC port includes a transceiver and an interface controller, and the computer peripheral device in which the FC port is contained is called a “host.” The FC port exchanges data with the host via a local data bus, such as a peripheral computer interface (“PCI”) bus. The interface controller conducts lower-level protocol exchanges between the fibre channel and the computer or peripheral device in which the FC port resides.[0003]
• A popular paradigm for accessing remote data in computer networks is the client/server architecture. According to this architecture, a client computer sends a request to read or write data to a server computer. The server computer processes the request by checking that the client server has authorization and permission to read or write the data, by mapping the requested read or write operation to a particular mass storage device, and by serving as an intermediary in the transfer of data from the client computer to the mass storage device, in case of a write operation, or from the mass storage device to the client, in case of a read operation.[0004]
• In common, currently-available and previously-available communication network architectures, the server computer communicates with the client computer through a local area network (“LAN”) and the server computer communicates with a number of mass storage devices over a local bus, such as a SCSI bus. In such systems, the server is required to store and forward the data transferred as a result of the read or write operation because the server represents a bridge between two dissimilar communications media. With the advent of the FC, client computers, server computers, and mass storage devices may all be symmetrically interconnected by a single communications medium. The traditional client/server architecture is commonly ported to the FC using the same type of client/server protocols as are used in the LAN and SCSI networks discussed above.[0005]
• SCSI-bus-compatible mass-storage devices, including high capacity disk drives, are widely available, and widely used, particularly in mid-sized and large-sized computer systems, and many FC-based systems employ FC-compatible disk drives, each including one or more FC ports and logic needed for the disk drives to function as FC responders. In smaller systems, including personal computers (“PCs”), a different family of disk drives, referred to as Integrated Drive Electronics (“IDE”) or Advanced Technology Attachment (“ATA”) disk drives is widely employed. A serial ATA disk (“SATA”) generally interconnects with a system via an Industry Standard Architecture (“ISA”) bus.[0006]
• The present invention is related to FC, SCSI, and IDE/ATA technologies. Each will be discussed, in turn, in three separate subsections, below. Those familiar with any or all of these technologies may wish to skip ahead to the final subsection of this section, describing FC-based disk arrays, and to the Summary of the Invention section that immediately follows that subsection.[0007]
Fibre Channel
• The Fibre Channel (“FC”) is defined by, and described in, a number of ANSI Standards documents, including the standards documents listed below in Table 1:[0008]

  TABLE 1
  Acronym	Title	Publication
  10 Bit Interface TR	10-bit Interface Technical Report	X3.TR-18:1997
  10GFC	Fibre Channel - 10 Gigabit	Project 1413-D
  AE-2 Study	AE-2 Study Group	Internal Study
  FC-10KCR	Fibre Channel - 10 km Cost-Reduced Physical variant	NCITS 326:1999
  FC-AE	Fibre Channel Avionics Environment	INCITS TR-31-2002
  FC-AL	FC Arbitrated Loop	ANSI X3.272:1996
  FC-AL-2	Fibre Channel 2ndGeneration Arbitrated Loop	NCITS 332:1999
  FC-AV	Fibre Channel - Audio-Visual	ANSI/INCITS 356:2001
  FC-BB	Fibre Channel - Backbone	ANSI NCITS 342
  FC-BB-2	Fibre Channel - Backbone - 2	Project 1466-D
  FC-CU	Fibre Channel Copper Interface Implementation Practice Guide	Project 1135-DT
  FC-DA	Fibre Channel - Device Attach	Project 1513-DT
  FC-FG	FC Fabric Generic Requirements	ANSI X3.289:1996
  FC-FLA	Fibre Channel - Fabric Loop Attachment	NCITS TR-20:1998
  FC-FP	FC - Mapping to HIPPI-FP	ANSI X3.254:1994
  FC-FS	Fibre Channel Framing and Signaling Interface	Project 1331-D
  FC-GS	FC Generic Services	ANSI X3.288:1996
  FC-GS-2	Fibre Channel 2ndGeneration Generic Services	ANSI NCITS 288
  FC-GS-3	Fibre Channel - Generic Services 3	NCITS 348-2000
  FC-GS-4	Fibre Channel Generic Services 4	Project 1505-D
  FC-HBA	Fibre Channel - HBA API	Project 1568-D
  FC-HSPI	Fibre Channel High Speed Parallel Interface (FC-HSPI)	NCITS TR-26:2000
  FC-LE	FC Link Encapsulation	ANSI X3.287:1996
  FC-MI	Fibre Channel- Methodologies for Interconnects Technical	INCITS TR-30-2002
  	Report
  FC-MI-2	Fibre Channel - Methodologies for Interconnects - 2	Project 1599-DT
  FC-MJS	Methodology of Jitter Specification	NCITS TR-25:1999
  FC-MJSQ	Fibre Channel - Methodologies for Jitter and Signal Quality	Project 1316-DT
  	Specification
  FC-PH	Fibre Channel Physical and Signaling Interface	ANSI X3.230:1994
  FC-PH-2	Fibre Channel 2ndGeneration Physical Interface	ANSI X3.297:1997
  FC-PH-3	Fibre Channel 3rdGeneration Physical Interface	ANSI X3.303:1998
  FC-PH:AM 1	FC-PH Amendment #1	ANSI
  		X3.230:1994/AM1:1996
  FC-PH:DAM 2	FC-PH Amendment #2	ANSI X3.230/AM2-1999
  FC-PI	Fibre Channel - Physical Interface	Project 1306-D
  FC-PI-2	Fibre Channel - Physical Interfaces - 2	Project
  FC-PLDA	Fibre Channel Private Loop Direct Attach	NCITS TR-19:1998
  FC-SB	FC Mapping of Single Byte Command Code Sets	ANSI X3.271:1996
  FC-SB-2	Fibre Channel - SB 2	NCITS 349-2000
  FC-SB-3	Fibre Channel - Single Byte Command Set-3	Project 1569-D
  FC-SP	Fibre Channel - Security Protocols	Project 1570-D
  FC-SW	FC Switch Fabric and Switch Control Requirements	NCITS 321:1998
  FC-SW-2	Fibre Channel - Switch Fabric - 2	ANSI/NCITS 355-2001
  FC-SW-3	Fibre Channel - Switch Fabric - 3	Project 1508-D
  FC-SWAPI	Fibre Channel Switch Application Programming Interface	Project 1600-D
  FC-Tape	Fibre Channel - Tape Technical Report	NCITS TR-24:1999
  FC-VI	Fibre Channel - Virtual Interface Architecture Mapping	ANSI/NCITS 357-2001
  FCSM	Fibre Channel Signal Modeling	Project 1507-DT
  MIB-FA	Fibre Channel Management Information Base	Project 1571-DT
  SM-LL-V	FC - Very Long Length Optical Interface	ANSI/NCITS 339-2000

• The documents listed in Table 1, and additional information about the fibre channel, may be found at the World Wide Web pages having the following addresses: “http://www.t11.org/index.htm” and “http://www.fibrechannel.com.”[0009]
• The following description of the FC is meant to introduce and summarize certain of the information contained in these documents in order to facilitate discussion of the present invention. If a more detailed discussion of any of the topics introduced in the following description is desired, the above-mentioned documents may be consulted.[0010]
• The FC is an architecture and protocol for data communications between FC nodes, generally computers, workstations, peripheral devices, and arrays or collections of peripheral devices, such as disk arrays, interconnected by one or more communications media. Communications media include shielded twisted pair connections, coaxial cable, and optical fibers. An FC node is connected to a communications medium via at least one FC port and FC link. An FC port is an FC host adapter or FC controller that shares a register and memory interface with the processing components of the FC node, and that implements, in hardware and firmware, the lower levels of the FC protocol. The FC node generally exchanges data and control information with the FC port using shared data structures in shared memory and using control registers in the FC port. The FC port includes serial transmitter and receiver components coupled to a communications medium via a link that comprises electrical wires or optical strands.[0011]
• In the following discussion, “FC” is used as an adjective to refer to the general Fibre Channel architecture and protocol, and is used as a noun to refer to an instance of a Fibre Channel communications medium. Thus, an FC (architecture and protocol) port may receive an FC (architecture and protocol) sequence from the FC (communications medium).[0012]
• The FC architecture and protocol support three different types of interconnection topologies, shown in FIGS.[0013]1A-1C. FIG. 1A shows the simplest of the three interconnected topologies, called the “point-to-point topology.” In the point-to-point topology shown in FIG. 1A, afirst node101 is directly connected to asecond node102 by directly coupling thetransmitter103 of theFC port104 of thefirst node101 to thereceiver105 of theFC port106 of thesecond node102, and by directly connecting thetransmitter107 of theFC port106 of thesecond node102 to thereceiver108 of theFC port104 of thefirst node101. Theports104 and106 used in the point-to-point topology are called N_Ports.
• FIG. 1B shows a somewhat more complex topology called the “FC arbitrated loop topology.” FIG. 1B shows four nodes[0014]110-113 interconnected within an arbitrated loop. Signals, consisting of electrical or optical binary data, are transferred from one node to the next node around the loop in a circular fashion. The transmitter of one node, such astransmitter114 associated withnode111, is directly connected to the receiver of the next node in the loop, in the case oftransmitter114, with thereceiver115 associated withnode112. Two types of FC ports may be used to interconnect FC nodes within an arbitrated loop. The most common type of port used in arbitrated loops is called the “NL_Port.” A special type of port, called the “FL_Port,” may be used to interconnect an FC arbitrated loop with an FC fabric topology, to be described below. Only one FL_Port may be actively incorporated into an arbitrated loop topology. An FC arbitrated loop topology may include up to 127 active FC ports, and may include additional non-participating FC ports.
• In the FC arbitrated loop topology, nodes contend for, or arbitrate for, control of the arbitrated loop. In general, the node with the lowest port address obtains control in the case that more than one node is contending for control. A fairness algorithm may be implemented by nodes to ensure that all nodes eventually receive control within a reasonable amount of time. When a node has acquired control of the loop, the node can open a channel to any other node within the arbitrated loop. In a half duplex channel, one node transmits and the other node receives data. In a full duplex channel, data may be transmitted by a first node and received by a second node at the same time that data is transmitted by the second node and received by the first node. For example, if, in the arbitrated loop of FIG. 1B,[0015]node111 opens a full duplex channel withnode113, then data transmitted through that channel fromnode111 tonode113 passes through NL_Port116 ofnode112, and data transmitted bynode113 tonode111 passes through NL_Port117 ofnode110.
• FIG. 1C shows the most general and most complex FC topology, called an “FC fabric.” The FC fabric is represented in FIG. 1C by the irregularly shaped[0016]central object118 to which four FC nodes119-122 are connected. The N_Ports123-126 within the FC nodes119-122 are connected to F_Ports127-130 within thefabric118. The fabric is a switched or cross-point switch topology similar in function to a telephone system. Data is routed by the fabric between F_Ports through switches or exchanges called “fabric elements.” There may be many possible routes through the fabric between one F_Port and another F_Port. The routing of data and the addressing of nodes within the fabric associated with F_Ports are handled by the FC fabric, rather than by FC nodes or N_Ports.
• The FC is a serial communications medium. Data is transferred one bit at a time at extremely high transfer rates. FIG. 2 illustrates a very simple hierarchy by which data is organized, in time, for transfer through an FC network. At the lowest conceptual level, the data can be considered to be a stream of[0017]data bits200. The smallest unit of data, or grouping of data bits, supported by an FC network is a 10-bit character that is decoded by FC port as an 8-bit character. FC primitives are composed of 10-bit characters or bytes. Certain FC primitives are employed to carry control information exchanged between FC ports. The next level of data organization, a fundamental level with regard to the FC protocol, is a frame. Seven frames202-208 are shown in FIG. 2. A frame may be composed of between 36 and 2,148 bytes, including delimiters, headers, and optional headers, and between 0 and 2048 bytes of data. The first FC frame, for example, corresponds to the data bits of the stream ofdata bits200 encompassed by thehorizontal bracket201. The FC protocol specifies a next higher organizational level called the sequence. Afirst sequence210 and a portion of asecond sequence212 are displayed in FIG. 2. Thefirst sequence210 is composed of frames one through four202-205. Thesecond sequence212 is composed of frames five through seven206-208 and additional frames that are not shown. The FC protocol specifies a third organizational level called the exchange. A portion of anexchange214 is shown in FIG. 2. Thisexchange214 is composed of at least thefirst sequence210 and thesecond sequence212 shown in FIG. 2. This exchange can alternatively be viewed as being composed of frames one through seven202-208, and any additional frames contained in thesecond sequence212 and in any additional sequences that compose theexchange214.
• The FC is a full duplex data transmission medium. Frames and sequences can be simultaneously passed in both directions between an originator, or initiator, and a responder, or target. An exchange comprises all sequences, and frames within the sequences, exchanged between an originator and a responder during a single I/O transaction, such as a read I/O transaction or a write I/O transaction. The FC protocol is designed to transfer data according to any number of higher-level data exchange protocols, including the Internet protocol (“IP”), the Small Computer Systems Interface (“SCSI”) protocol, the High Performance Parallel Interface (“HIPPI”), and the Intelligent Peripheral Interface (“IPI”). The SCSI bus architecture will be discussed in the following subsection, and much of the subsequent discussion in this and remaining subsections will focus on the SCSI protocol embedded within the FC protocol. The standard adaptation of SCSI protocol to fibre channel is subsequently referred to in this document as “FCP.” Thus, the FC can support a master-slave type communications paradigm that is characteristic of the SCSI bus and other peripheral interconnection buses, as well as the relatively open and unstructured communication protocols such as those used to implement the Internet. The SCSI bus architecture concepts of an initiator and target are carried forward in the FCP, designed, as noted above, to encapsulate SCSI commands and data exchanges for transport through the FC.[0018]
• FIG. 3 shows the contents of a standard FC frame. The[0019]FC frame302 comprises fivehigh level sections304,306,308,310 and312. The first high level section, called the start-of-frame deliminator304, comprises 4 bytes that mark the beginning of the frame. The next high level section, calledframe header306, comprises 24 bytes that contain addressing information, sequence information, exchange information, and various control flags. A more detailed view of theframe header314 is shown expanded from theFC frame302 in FIG. 3. The destination identifier (“D_ID”), orDESTINATION_ID316, is a 24-bit FC address indicating the destination FC port for the frame. The source identifier (“S_ID”), orSOURCE_ID318, is a 24-bit address that indicates the FC port that transmitted the frame. The originator ID, orOX_ID320, and theresponder ID322, or RX_ID, together compose a 32-bit exchange ID that identifies the exchange to which the frame belongs with respect to the originator, or initiator, and responder, or target, FC ports. The sequence ID, or SEQ_ID,324 identifies the sequence to which the frame belongs.
• The next[0020]high level section308, called the data payload, contains the actual data packaged within the FC frame. The data payload contains data and encapsulating protocol information that is being transferred according to a higher-level protocol, such as IP and SCSI. FIG. 3 shows four basic types of data payload layouts326-329 used for data transfer according to the SCSI protocol. The first of theseformats326, called the FCP_CMND, is used to send a SCSI command from an initiator to a target. TheFCP_LUN field330 comprises an 8-byte address that may, in certain implementations, specify a particular SCSI-bus adapter, a target device associated with that SCSI-bus adapter, and a logical unit number (“LUN”) corresponding to a logical device associated with the specified target SCSI device that together represent the target for the FCP_CMND. In other implementations, theFCP_LUN field330 contains an index or reference number that can be used by the target FC host adapter to determine the SCSI-bus adapter, a target device associated with that SCSI-bus adapter, and a LUN corresponding to a logical device associated with the specified target SCSI device. An actual SCSI command, such as a SCSI read or write I/O command, is contained within the 16-byte field FCP_CDB332.
• The second type of[0021]data payload format327 shown in FIG. 3 is called the FCP_XFER_RDY layout. This data payload format is used to as an indication by the target to the initiator that the target is prepared to begin receiving or sending data. The third type ofdata payload format328 shown in FIG. 3 is the FCP_DATA format. The FCP_DATA format is used for transferring the actual data that is being read from, or written to, a SCSI data storage device as a result of execution of a SCSI I/O transaction. The finaldata payload format329 shown in FIG. 3 is called the FCP_RSP layout, used to transfer a SCSI status byte334, as well as other FCP status information, from the target back to the initiator upon completion of the I/O transaction.
The SCSI Bus Architecture
• A computer bus is a set of electrical signal lines through which computer commands and data are transmitted between processing, storage, and input/output (“I/O”) components of a computer system. The SCSI I/O bus is the most widespread and popular computer bus for interconnecting mass storage devices, such as hard disks and CD-ROM drives, with the memory and processing components of computer systems. The SCSI bus architecture is defined in three major standards: SCSI-1, SCSI-2 and SCSI-3. The SCSI-1 and SCSI-2 standards are published in the American National Standards Institute (“ANSI”) standards documents “X3.131-1986,” and “X3.131-1994,” respectively. The SCSI-3 standard is currently being developed by an ANSI committee. An overview of the SCSI bus architecture is provided by “The SCSI Bus and IDE Interface,” Freidhelm Schmidt, Addison-Wesley Publishing Company, ISBN 0-201-17514-2, 1997 (“Schmidt”).[0022]
• FIG. 4 is a block diagram of a common personal computer (“PC”) architecture including a SCSI bus. The[0023]PC400 includes a central processing unit, or processor (“CPU”)402, linked to asystem controller404 by a high-speed CPU bus406. The system controller is, in turn, linked to asystem memory component408 via amemory bus410. Thesystem controller404 is, in addition, linked to various peripheral devices via a peripheral component interconnect (“PCI”)bus412 that is interconnected with a slower industry standard architecture (“ISA”)bus414 and aSCSI bus416. The architecture of the PCI bus is described in “PCI System Architecture,” Shanley & Anderson, Mine Share, Inc., Addison-Wesley Publishing Company, ISBN 0-201-40993-3, 1995. Theinterconnected CPU bus406,memory bus410,PCI bus412, andISA bus414 allow the CPU to exchange data and commands with the various processing and memory components and I/O devices included in the computer system. Generally, very high-speed and high bandwidth I/O devices, such as avideo display device418, are directly connected to the PCI bus. Slow I/O devices420, such as akeyboard420 and a pointing device (not shown), are connected directly to theISA bus414. The ISA bus is interconnected with the PCI bus through abus bridge component422. Mass storage devices, such as hard disks, floppy disk drives, CD-ROM drives, and tape drives424-426 are connected to theSCSI bus416. The SCSI bus is interconnected with thePCI bus412 via a SCSI-bus adapter430. The SCSI-bus adapter430 includes a processor component, such as a processor selected from the Symbios family of 53C8xx SCSI processors, and interfaces to thePCI bus412 using standard PCI bus protocols. The SCSI-bus adapter430 interfaces to theSCSI bus416 using the SCSI bus protocol that will be described, in part, below. The SCSI-bus adapter430 exchanges commands and data with SCSI controllers (not shown) that are generally embedded within each mass storage device424-426, or SCSI device, connected to the SCSI bus. The SCSI controller is a hardware/firmware component that interprets and responds to SCSI commands received from a SCSI adapter via the SCSI bus and that implements the SCSI commands by interfacing with, and controlling, logical devices. A logical device may correspond to one or more physical devices, or to portions of one or more physical devices. Physical devices include data storage devices such as disk, tape and CD-ROM drives.
• Two important types of commands, called I/O commands, direct the SCSI device to read data from a logical device and write data to a logical device. An I/O transaction is the exchange of data between two components of the computer system, generally initiated by a processing component, such as the[0024]CPU402, that is implemented, in part, by a read I/O command or by a write I/O command. Thus, I/O transactions include read I/O transactions and write I/O transactions.
• The[0025]SCSI bus416 is a parallel bus that can simultaneously transport a number of data bits. The number of data bits that can be simultaneously transported by the SCSI bus is referred to as the width of the bus. Different types of SCSI buses have widths of 8, 16 and 32 bits. The 16 and 32-bit SCSI buses are referred to as wide SCSI buses.
• As with all computer buses and processors, the SCSI bus is controlled by a clock that determines the speed of operations and data transfer on the bus. SCSI buses vary in clock speed. The combination of the width of a SCSI bus and the clock rate at which the SCSI bus operates determines the number of bytes that can be transported through the SCSI bus per second, or bandwidth of the SCSI bus. Different types of SCSI buses have bandwidths ranging from less than 2 megabytes (“Mbytes”) per second up to 40 Mbytes per second, with increases to 80 Mbytes per second and possibly 160 Mbytes per second planned for the future. The increasing bandwidths may be accompanied by increasing limitations in the physical length of the SCSI bus.[0026]
• FIG. 5 illustrates the SCSI bus topology. A[0027]computer system502, or other hardware system, may include one or more SCSI-bus adapters504 and506. The SCSI-bus adapter, the SCSI bus which the SCSI-bus adapter controls, and any peripheral devices attached to that SCSI bus together comprise a domain. SCSI-bus adapter504 in FIG. 5 is associated with afirst domain508 and SCSI-bus adapter506 is associated with asecond domain510. The most current SCSI-2 bus implementation allows fifteen different SCSI devices513-515 and516-517 to be attached to a single SCSI bus. In FIG. 5, SCSI devices513-515 are attached toSCSI bus518 controlled by SCSI-bus adapter506, and SCSI devices516-517 are attached toSCSI bus520 controlled by SCSI-bus adapter504. Each SCSI-bus adapter and SCSI device has a SCSI identification number, or SCSI_ID, that uniquely identifies the device or adapter in a particular SCSI bus. By convention, the SCSI-bus adapter hasSCSI_ID7, and the SCSI devices attached to the SCSI bus have SCSI_IDs ranging from 0 to 6 and from 8 to 15. A SCSI device, such asSCSI device513, may interface with a number of logical devices, each logical device comprising portions of one or more physical devices. Each logical device is identified by a logical unit number (“LUN”) that uniquely identifies the logical device with respect to the SCSI device that controls the logical device. For example,SCSI device513 controls logical devices522-524 havingLUNs 0, 1, and 2, respectively. According to SCSI terminology, a device that initiates an I/O command on the SCSI bus is called an initiator, and a SCSI device that receives an I/O command over the SCSI bus that directs the SCSI device to execute an I/O operation is called a target.
• In general, a SCSI-bus adapter, such as SCSI-[0028]bus adapters504 and506, initiates I/O operations by sending commands to target devices. The target devices513-515 and516-517 receive the I/O commands from the SCSI bus. The target devices513-515 and516-517 then implement the commands by interfacing with one or more logical devices that they control to either read data from the logical devices and return the data through the SCSI bus to the initiator or to write data received through the SCSI bus from the initiator to the logical devices. Finally, the target devices513-515 and516-517 respond to the initiator through the SCSI bus with status messages that indicate the success or failure of implementation of the commands.
• FIGS.[0029]6A-6C illustrate the SCSI protocol involved in the initiation and implementation of read and write I/O operations. Read and write I/O operations compose the bulk of I/O operations performed by SCSI devices. Efforts to maximize the efficiency of operation of a system of mass storage devices interconnected by a SCSI bus are most commonly directed toward maximizing the efficiency at which read and write I/O operations are performed. Thus, in the discussions to follow, the architectural features of various hardware devices will be discussed in terms of read and write operations.
• FIG. 6A shows the sending of a read or write I/O command by a SCSI initiator, most commonly a SCSI-bus adapter, to a SCSI target, most commonly a SCSI controller embedded in a SCSI device associated with one or more logical devices. The sending of a read or write I/O command is called the command phase of a SCSI I/O operation. FIG. 6A is divided into[0030]initiator602 and target604 sections by a centralvertical line606. Both the initiator and the target sections include columns entitled “state”606 and608 that describe the state of the SCSI bus and columns entitled “events”610 and612 that describe the SCSI bus events associated with the initiator and the target, respectively. The bus states and bus events involved in the sending of the I/O command are ordered in time, descending from the top of FIG. 6A to the bottom of FIG. 6A. FIGS.6B-6C also adhere to this above-described format.
• The sending of an I/O command from an initiator SCSI-bus adapter to a target SCSI device, illustrated in FIG. 6A, initiates a read or write I/O operation by the target SCSI device. Referring to FIG. 4, the SCSI-[0031]bus adapter430 initiates the I/O operation as part of an I/O transaction. Generally, the SCSI-bus adapter430 receives a read or write command via thePCI bus412,system controller404, andCPU bus406, from theCPU402 directing the SCSI-bus adapter to perform either a read operation or a write operation. In a read operation, theCPU402 directs the SCSI-bus adapter430 to read data from a mass storage device424-426 and transfer that data via theSCSI bus416,PCI bus412,system controller404, andmemory bus410 to a location within thesystem memory408. In a write operation, theCPU402 directs thesystem controller404 to transfer data from thesystem memory408 via thememory bus410,system controller404, andPCI bus412 to the SCSI-bus adapter430, and directs the SCSI-bus adapter430 to send the data via theSCSI bus416 to a mass storage device424-426 on which the data is written.
• FIG. 6A starts with the SCSI bus in the BUS[0032]FREE state614, indicating that there are no commands or data currently being transported on the SCSI device. The initiator, or SCSI-bus adapter, asserts the BSY, D7 and SEL signal lines of the SCSI bus in order to cause the bus to enter theARBITRATION state616. In this state, the initiator announces to all of the devices an intent to transmit a command on the SCSI bus. Arbitration is necessary because only one device may control operation of the SCSI bus at any instant in time. Assuming that the initiator gains control of the SCSI bus, the initiator then asserts the ATN signal line and the DX signal line corresponding to the target SCSI_ID in order to cause the SCSI bus to enter theSELECTION state618. The initiator or target asserts and drops various SCSI signal lines in a particular sequence in order to effect a SCSI bus state change, such as the change of state from theARBITRATION state616 to theSELECTION state618, described above. These sequences can be found in Schmidt and in the ANSI standards, and will therefore not be further described below.
• When the target senses that the target has been selected by the initiator, the target assumes[0033]control620 of the SCSI bus in order to complete the command phase of the I/O operation. The target then controls the SCSI signal lines in order to enter the MESSAGE OUTstate622. In a first event that occurs in the MESSAGE OUT state, the target receives from the initiator anIDENTIFY message623. TheIDENTIFY message623 contains aLUN field624 that identifies the LUN to which the command message that will follow is addressed. TheIDENTIFY message623 also contains aflag625 that is generally set to indicate to the target that the target is authorized to disconnect from the SCSI bus during the target's implementation of the I/O command that will follow. The target then receives aQUEUE TAG message626 that indicates to the target how the I/O command that will follow should be queued, as well as providing the target with aqueue tag627. The queue tag is a byte that identifies the I/O command. A SCSI-bus adapter can therefore concurrently manage256 different I/O commands per LUN. The combination of the SCSI_ID of the initiator SCSI-bus adapter, the SCSI_ID of the target SCSI device, the target LUN, and the queue tag together comprise an I_T_L_Q nexus reference number that uniquely identifies the I/O operation corresponding to the I/O command that will follow within the SCSI bus. Next, the target device controls the SCSI bus signal lines in order to enter theCOMMAND state628. In the COMMAND state, the target solicits and receives from the initiator the I/O command630. The I/O command630 includes anopcode632 that identifies the particular command to be executed, in this case a read command or a write command, alogical block number636 that identifies the logical block of the logical device that will be the beginning point of the read or write operation specified by the command, and adata length638 that specifies the number of blocks that will be read or written during execution of the command.
• When the target has received and processed the I/O command, the target device controls the SCSI bus signal lines in order to enter the MESSAGE IN[0034]state640 in which the target device generally sends adisconnect message642 back to the initiator device. The target disconnects from the SCSI bus because, in general, the target will begin to interact with the logical device in order to prepare the logical device for the read or write operation specified by the command. The target may need to prepare buffers for receiving data, and, in the case of disk drives or CD-ROM drives, the target device may direct the logical device to seek to the appropriate block specified as the starting point for the read or write command. By disconnecting, the target device frees up the SCSI bus for transportation of additional messages, commands, or data between the SCSI-bus adapter and the target devices. In this way, a large number of different I/O operations can be concurrently multiplexed over the SCSI bus. Finally, the target device drops the BSY signal line in order to return the SCSI bus to the BUSFREE state644.
• The target device then prepares the logical device for the read or write operation. When the logical device is ready for reading or writing data, the data phase for the I/O operation ensues. FIG. 6B illustrates the data phase of a SCSI I/O operation. The SCSI bus is initially in the BUS FREE state[0035]646. The target device, now ready to either return data in response to a read I/O command or accept data in response to a write I/O command, controls the SCSI bus signal lines in order to enter theARBITRATION state648. Assuming that the target device is successful in arbitrating for control of the SCSI bus, the target device controls the SCSI bus signal lines in order to enter theRESELECTION state650. The RESELECTION state is similar to the SELECTION state, described in the above discussion of FIG. 6A, except that it is the target device that is making the selection of a SCSI-bus adapter with which to communicate in the RESELECTION state, rather than the SCSI-bus adapter selecting a target device in the SELECTION state.
• Once the target device has selected the SCSI-bus adapter, the target device manipulates the SCSI bus signal lines in order to cause the SCSI bus to enter the MESSAGE IN[0036]state652. In the MESSAGE IN state, the target device sends both anIDENTIFY message654 and aQUEUE TAG message656 to the SCSI-bus adapter. These messages are identical to the IDENTITY and QUEUE TAG messages sent by the initiator to the target device during transmission of the I/O command from the initiator to the target, illustrated in FIG. 6A. The initiator may use the I_T_L_Q nexus reference number, a combination of the SCSI_IDs of the initiator and target device, the target LUN, and the queue tag contained in the QUEUE TAG message, to identify the I/O transaction for which data will be subsequently sent from the target to the initiator, in the case of a read operation, or to which data will be subsequently transmitted by the initiator, in the case of a write operation. The I_T_L_Q nexus reference number is thus an I/O operation handle that can be used by the SCSI-bus adapter as an index into a table of outstanding I/O commands in order to locate the appropriate buffer for receiving data from the target device, in case of a read, or for transmitting data to the target device, in case of a write.
• After sending the IDENTIFY and QUEUE TAG messages, the target device controls the SCSI signal lines in order to transition to a[0037]DATA state658. In the case of a read I/O operation, the SCSI bus will transition to the DATA IN state. In the case of a write I/O operation, the SCSI bus will transition to a DATA OUT state. During the time that the SCSI bus is in the DATA state, the target device will transmit, during each SCSI bus clock cycle, a data unit having a size, in bits, equal to the width of the particular SCSI bus on which the data is being transmitted. In general, there is a SCSI bus signal line handshake involving the signal lines ACK and REQ as part of the transfer of each unit of data. In the case of a read I/O command, for example, the target device places the next data unit on the SCSI bus and asserts the REQ signal line. The initiator senses assertion of the REQ signal line, retrieves the transmitted data from the SCSI bus, and asserts the ACK signal line to acknowledge receipt of the data. This type of data transfer is called asynchronous transfer. The SCSI bus protocol also allows for the target device to transfer a certain number of data units prior to receiving the first acknowledgment from the initiator. In this transfer mode, called synchronous transfer, the latency between the sending of the first data unit and receipt of acknowledgment for that transmission is avoided. During data transmission, the target device can interrupt the data transmission by sending a SAVE POINTERS message followed by a DISCONNECT message to the initiator and then controlling the SCSI bus signal lines to enter the BUS FREE state. This allows the target device to pause in order to interact with the logical devices which the target device controls before receiving or transmitting further data. After disconnecting from the SCSI bus, the target device may then later again arbitrate for control of the SCSI bus and send additional IDENTIFY and QUEUE TAG messages to the initiator so that the initiator can resume data reception or transfer at the point that the initiator was interrupted. An example of disconnect and reconnect660 are shown in FIG. 3B interrupting theDATA state658. Finally, when all the data for the I/O operation has been transmitted, the target device controls the SCSI signal lines in order to enter the MESSAGE INstate662, in which the target device sends a DISCONNECT message to the initiator, optionally preceded by a SAVE POINTERS message. After sending the DISCONNECT message, the target device drops the BSY signal line so the SCSI bus transitions to the BUS FREE state664.
• Following the transmission of the data for the I/O operation, as illustrated in FIG. 6B, the target device returns a status to the initiator during the status phase of the I/O operation. FIG. 6C illustrates the status phase of the I/O operation. As in FIGS.[0038]6A-6B, the SCSI bus transitions from the BUS FREE state666 to theARBITRATION state668,RESELECTION state670, and MESSAGE INstate672, as in FIG. 3B. Following transmission of an IDENTIFY message674 andQUEUE TAG message676 by the target to the initiator during the MESSAGE INstate672, the target device controls the SCSI bus signal lines in order to enter theSTATUS state678. In theSTATUS state678, the target device sends asingle status byte684 to the initiator to indicate whether or not the I/O command was successfully completed. In FIG. 6C, the status byte680 corresponding to a successful completion, indicated by a status code of 0, is shown being sent from the target device to the initiator. Following transmission of the status byte, the target device then controls the SCSI bus signal lines in order to enter the MESSAGE INstate682, in which the target device sends a COMMANDCOMPLETE message684 to the initiator. At this point, the I/O operation has been completed. The target device then drops the BSY signal line so that the SCSI bus returns to the BUS FREE state686. The SCSI-bus adapter can now finish its portion of the I/O command, free up any internal resources that were allocated in order to execute the command, and return a completion message or status back to the CPU via the PCI bus.
Mapping the SCSI Protocol onto FCP
• FIGS. 7A and 7B illustrate a mapping of FCP sequences exchanged between an initiator and target and the SCSI bus phases and states described in FIGS.[0039]6A-6C. In FIGS.7A-7B, the target SCSI adapter is assumed to be packaged together with a FCP host adapter, so that the target SCSI adapter can communicate with the initiator via the FC and with a target SCSI device via the SCSI bus. FIG. 7A shows a mapping between FCP sequences and SCSI phases and states for a read I/O transaction. The transaction is initiated when the initiator sends a single-frame FCP sequence containing aFCP_CMND702 data payload through the FC to a target SCSI adapter. When the target SCSI-bus adapter receives the FCP_CMND frame, the target SCSI-bus adapter proceeds through the SCSI states of thecommand phase704 illustrated in FIG. 6A, including ARBITRATION, RESELECTION, MESSAGE OUT, COMMAND, and MESSAGE IN. At the conclusion of the command phase, as illustrated in FIG. 6A, the SCSI device that is the target of the I/O transaction disconnects from the SCSI bus in order to free up the SCSI bus while the target SCSI device prepares to execute the transaction. Later, the target SCSI device re-arbitrates for SCSI bus control and begins the data phase of the I/O transaction706. At this point, the SCSI-bus adapter may send a FCP_XFER_RDY single-frame sequence708 back to the initiator to indicate that data transmission can now proceed. In the case of a read I/O transaction, the FCP_XFER_RDY single-frame sequence is optional. As the data phase continues, the target SCSI device begins to read data from a logical device and transmit that data over the SCSI bus to the target SCSI-bus adapter. The target SCSI-bus adapter then packages the data received from the target SCSI device into a number of FCP_DATA frames that together compose the third sequence of the exchange corresponding to the I/O read transaction, and transmits those FCP_DATA frames back to the initiator through the FC. When all the data has been transmitted, and the target SCSI device has given up control of the SCSI bus, the target SCSI device then again arbitrates for control of the SCSI bus to initiate the status phase of the I/O transaction714. In this phase, the SCSI bus transitions from the BUS FREE state through the ARBITRATION, RESELECTION, MESSAGE IN, STATUS, MESSAGE IN and BUS FREE states, as illustrated in FIG. 3C, in order to send a SCSI status byte from the target SCSI device to the target SCSI-bus adapter. Upon receiving the status byte, the target SCSI-bus adapter packages the status byte into an FCP_RSP single-frame sequence716 and transmits the FCP_RSP single-frame sequence back to the initiator through the FC. This completes the read I/O transaction.
• In many computer systems, there may be additional internal computer buses, such as a PCI bus, between the target FC host adapter and the target SCSI-bus adapter. In other words, the FC host adapter and SCSI adapter may not be packaged together in a single target component. In the interest of simplicity, that additional interconnection is not shown in FIGS.[0040]7A-B.
• FIG. 7B shows, in similar fashion to FIG. 7A, a mapping between FCP sequences and SCSI bus phases and states during a write I/O transaction indicated by a[0041]FCP_CMND frame718. FIG. 7B differs from FIG. 7A only in the fact that, during a write transaction, the FCP_DATA frames722-725 are transmitted from the initiator to the target over the FC and the FCP_XFER_RDY single-frame sequence720 sent from the target to theinitiator720 is not optional, as in the case of the read I/O transaction, but is instead mandatory. As in FIG. 7A, the write I/O transaction includes when the target returns an FCP_RSP single-frame sequence726 to the initiator.
IDE/ATA Disk Drives
• IDE/ATA drives were developed in order to integrate a disk logic controller and a hard disk together as a single module. IDE/ATA drives were specifically designed for easy integration, via an ISA bus, into PC systems. Originally, IDE/ATA drives were designed with parallel, 16-bit interconnections to permit the exchange of two bytes of data between the IDE/ATA drives and the system at discrete intervals of time controlled by a system or bus clock. Unfortunately, the parallel bus interconnection is reaching a performance limit, with current data rates of 100 to 133 MB/sec., and the 40 or 80-pin ribbon cable connection is no longer compatible with the cramped, high-density packaging of internal components within modern computer systems. For these reasons, a SATA (“SATA”) standard has been developed, and SATA disk drives are currently being produced, in which the 80-pin ribbon cable connection is replaced with a four-conductor serial cable. The initial data rate for SATA disks is 150 MB/sec, expected to soon increase to 300 MB/sec and then to 600 MB/sec.[0042]Standard 8B/10B encoding is used for serializing the data for transfer between the ATA serial disk drive and a peripheral component interconnect (“PCI”)-based controller. Ultimately, south-bridge controllers that integrate various I/O controllers, that provide interfaces to peripheral devices and buses, and that transfer data to and from a second bridge that links one or more CPUs and memory, may be designed to fully incorporate SATA technology to offer direct interconnection of SATA devices.
• The ATA interface, in particular the ATA-5 and ATA-6 standard interfaces, support a variety of commands that allow an external processor or logic controller to direct the logic controller within the ATA disk drive to carry out basic data transfer commands, seeking, cache management, and other management and diagnostics-related tasks. Table 2, below, relates a protocol number, such as protocol “1,” with a general type of ATA command. The types of commands include (“PIO”), non-data commands, and direct-memory-access[0043]

  TABLE 2
  protocol	type ofcommand
  1	PIO DATA-IN COMMAND
  2	PIODATA OUT COMMAND
  3	NON-DATA COMMAND
  4	DMA COMMAND
  5	DMA COMMAND

• Table 3, provided below, lists a number of ATA commands, along with a Corresponding protocol indicating the command type to which the command belongs, as defined above in Table 2:[0044]

  TABLE 3
  protocol	ATA Command
  3	CHECK POWER MODE
  2	DOWNLOAD MICROCODE
  3	EXECUTIVE DEVICE
  	DIAGNOSTICS
  3	FLUSH CACHE
  3	FLUSH CACHE EXTENDED
  1	IDENTIFY DEVICE
  3	IDLE IMMEDIATE
  4	READ DMA
  4	READ DMA EXTENDED
  3	READ VERIFYSECTORS
  3	READ VERIFY SECTORS
  	EXTENDED
  3	SEEK
  3	SET FEATURES
  3	SLEEP
  4	WRITE DMA
  4	WRITE DMA EXTENDED

• The CHECK POWER MODE command allows a host to determine the current power mode of an ATA device. The DOWNLOAD MICROCODE command allows a host to alter an ATA device's microcode. The EXECUTIVE DEVICE DIAGNOSTICS command allows a host to invoke diagnostic tests implemented by an ATA device. The FLUSH CACHE command allows a host to request that an ATA device flush its write cache. Two versions of this command are included in the table, with the extended version representing a 48-bit addressing feature available on devices supporting the ATA-6 standard interface. Additional extended versions of commands shown in Table 3 will not be discussed separately below. The IDENTIFY DEVICE command allows a host to query an ATA device for parameter information, including the number of logical sectors, cylinders, and heads provided by the device, the commands supported by the device, features supported by the device, and other such parameters. The READ DMA command allows a host to read data from the device using a DMA data transfer protocol, generally much more efficient for large amounts of data. The READ VERIFY SECTORS command allows a host to direct an ATA device to read a portion of the data stored within the host and determine whether or not any error conditions occur without transferring the data read from the device to the host. The SEEK command allows a host to inform an ATA device that the host may access one or more particular logical blocks in a subsequent command, to allow the device to optimize head positioning in order to execute the subsequent access to the specified one or more logical blocks. The SET FEATURES command allows the host to modify various parameters within an ATA device to turn on and off features provided by the device. The SLEEP command allows a host to direct an ATA device to spin down and wait for a subsequent reset command. The WRITE DMA command allows a host to write data to an ATA device using DMA data transfer that is generally more efficient for larger amounts of data.[0045]
FC-Based Disk Arrays
• In mid-sized and large computer systems, data storage requirements generally far exceed the capacities of embedded mass storage devices, including embedded disk drives. In such systems, it has become common to employ high-end, large-capacity devices, such as redundant arrays of inexpensive disks (“RAID”), that include internal processors that are linked to mid-sized and high-end computer systems through local area networks, fibre-optic networks, and other high-bandwidth communications media. To facilitate design and manufacture of disk arrays, disk manufacturers provide disk drives that include FC ports in order to directly interconnect disk drives within a disk array to a disk-array controller. Generally, the FC arbitrated loop topology is employed within disk arrays to interconnect individual FC disk drives to the disk-array controller.[0046]
• FIGS.[0047]8A-D illustrate several problems related to the use of FC disks in disk arrays. FIG. 8A shows a relatively abstract rendering of the internal components of a disk array. FIGS.8B-D and FIG. 9, discussed below, employ the same illustration conventions. In FIG. 8A, the disk-array controller802 is interconnected to remote computer systems and other remote entities via a high-bandwidth communications medium804. The disk-array controller includes one or more processors, one or more generally relatively large electronic memories, and other such components that allow disk-array-control firmware and software to be stored and executed within the disk-array controller in order to provide, to remote computer systems, a relatively high level, logical-unit and logical-block interface to the disk drives within the disk array. As shown in FIG. 8A, the disk-array includes the disk-array controller802 and a number of FC disk drives806-813. The FC disk drives are interconnected with the disk-array controller802 via an FC arbitratedloop814. An FC-based disk array, such as that abstractly illustrated in FIG. 8A, is relatively easily designed and manufactured, using standard and readily available FC disks as a storage medium, an FC arbitrated loop for interconnection, and standard FC controllers within the disk-array controller. Because the FC is a high-speed, serial communications medium, the FC arbitratedloop814 provides a generous bandwidth for data transfer between the FC disks806-813 and the disk-array controller802.
• However, at each FC node within the FC arbitrated loop, such as an FC disk drive, there is a significant node delay as data is processed and transferred through the FC ports of the node. Node delays are illustrated in FIG. 8A with short arrows labeled with subscripted, lower case letters “t.” The node delays are cumulative within an FC arbitrated loop, leading to significant accumulated node delays proportional to the number of FC nodes within the FC arbitrated loop.[0048]
• A second problem with the disk-array implementation illustrated in FIG. 8A is that the FC arbitrated loop represents a potential single point of failure. Generally, FC disks may be augmented with port bypass circuits to isolate nonfunctional FC disks from the arbitrated loop, but there are a number of different modes of failure that cannot be prevented by port bypass circuits alone.[0049]
• A third problem arises when an FC port that links a node to the arbitrated loop fails. In such cases, complex, and unreliable techniques must be employed to try to identify and isolate the failed FC port. In general, a failed FC port disrupts the loop topology, and the disk-array controller must sequentially attempt to activate port bypass circuits to bypass each node, in order to isolate the failed node. However, this technique may fail to identify the failed node, under various failure modes. Thus, node failure is a serious problem with arbitrated loop topologies.[0050]
• FIG. 8B illustrates a solution to the potential single-point failure problem. As shown in FIG. 8B, the disk-[0051]array controller802 is interconnected with the FC disks806-813 via two separate, independent FC arbitratedloops814 and816. Using two separate FC arbitrated loops largely removes the single-point failure problem. However, the node-delay problem is not ameliorated by using two FC arbitrated loops. Moreover, because each FC disk must include two separate FC ports, the individual FC disks are rather more complex and more expensive. Finally, the failed port identification and isolation problem is only partly addressed, because, in the case of a node failure that disrupts one of the two arbitrated loops, the other arbitrated loop continues to function, but there is no longer a two-fold redundancy in communications media. In order to restore the two-fold redundancy, the disk-array controller still needs to attempt to identify and isolate the failed node, and, as noted above, many failure modes are resistant to identification and isolation.
• FIG. 8C illustrates yet an additional problem with the FC-based implementation of disk arrays. In general, greater and greater amounts of available storage space are required from disk arrays, resulting in the addition of a greater number of individual FC disks. However, the inclusion of additional disks exacerbates the node-delay problem, and, as discussed above, a single FC arbitrated loop may include up to a maximum of only 127 nodes. In order to solve this maximum-node problem, additional independent FC arbitrated loops are added to the disk array. FIG. 8D illustrates a higher capacity disk array in which a first set of[0052]FC disks818 is interconnected with theFC controller802 via two separate FC arbitratedloops814 and816, and a second set ofFC disks820 is interconnected with the disk-array controller802 via a second pair of FC arbitratedloops822 and824. Each of the sets ofFC disks818 and820 are referred to as shelves, and are generally included in separate enclosures with redundant power systems, redundant control paths, and other features that contribute to the overall fault tolerance and high-availability of the disk array. However, the addition of each shelf increases the number of FC controllers and FC ports within the disk-array controller802. Note also that each separate FC arbitrated loop experiences cumulative node delay of the FC nodes included within the FC arbitrated loop. Designers, manufacturers, and users of disk arrays have thus recognized the need for a more flexible, more cost effective, and more efficient method for interconnecting disk-array controllers and FC disks within FC-based disk arrays. In addition, designers, manufacturers, and users of disk arrays have recognized the need for a method for interconnecting disk-array controllers and FC disks within FC-based disk arrays that allows for easier and more reliable identification of port failures and other communications and component failures.
• Although a storage-shelf router, or combination of storage-shelf routers, within a high-availability storage shelf essentially virtualizes the internal components of the high-availability storage shelf for external processing entities, such as disk-array controllers, there are situations that arise during manufacturing, configuration, and operation of disk arrays in which a disk-array controller or other external processing entity needs to directly access disk drives within a high-availability storage shelf as well as data structures stored in various internal components of a high-availability storage shelf, including a storage-shelf router, path-controller card, and the disk drives themselves. Manufacturers, designers, and users of high-availability storage shelves and disk arrays that include high-availability storage shelves have therefore recognized a need to facilitate direct access, by external processing entities, to internal components of a high-availability storage shelf which storage-shelf routers currently do not provide.[0053]
SUMMARY OF THE INVENTION
• One embodiment of the present invention is an integrated circuit implementing a storage-shelf router, used in combination with path controller cards and optionally with other storage-shelf routers, to interconnect SATA disks within a storage shelf or disk array to a high-bandwidth communications medium, such as an FC arbitrated loop. In one embodiment, each storage-shelf router is an integrated circuit that includes processing elements, multiple memories, a layered logic architecture, two complete FC ports for interconnecting the storage-shelf router to high-bandwidth communications media, and a number of SATA ports that each provides an interface to an SATA disk. FC ports link all storage-shelf routers within the storage shelf together to an internal fabric space within the storage-shelf router. The interconnected storage-shelf routers are linked to two separate, partially external fabric spaces, such as partially external FC arbitrated loops, that interconnect the storage shelf with a disk-array controller or other controlling component.[0054]
• Various embodiments of the present invention provide a tunneling mechanism through the storage-shelf interface provided by one or more storage-shelf routers within a storage shelf to enable external processing entities to directly access various components within the storage shelf. For example, various embodiments of the present invention provide a means for an external processing entity to write data into various data structures contained in flash memories and internal disk drives, and to directly interact with internal disk drives using internal-disk-drive protocols without translation or intervention by the storage-shelf router. In one embodiment of the present invention, a WRITE-BUFFER command and a READ-BUFFER command are added to the command interface supported by storage-shelf router. These commands are exchanged via the FCP protocol over the fiber channel in the same manner that SCSI commands are packaged within the FCP protocol. The storage-shelf router interprets WRITE-BUFFER and READ-BUFFER commands in various ways, depending on information included in the command frames. In certain cases, the information packaged within the WRITE-BUFFER and READ-BUFFER commands is directly exchanged with internal disk drives. In other cases, the information is extracted and written to various data structures maintained within internal components of the storage shelf. The WRITE-BUFFER and READ-BUFFER commands essential provide a breach or tunnel in the high-availability storage-shelf virtual interface provided by one or more storage-shelf routers within a storage shelf, allowing an external processing entity to exchange information through the tunnel without intervention by, but facilitated and supported by, the one or more storage-shelf routers within the high-availability storage shelf.[0055]
BRIEF DESCRIPTION OF THE DRAWINGS
• FIGS.[0056]1A-C show the three different types of FC interconnection topologies.
• FIG. 2 illustrates a very simple hierarchy by which data is organized, in time, for transfer through an FC network.[0057]
• FIG. 3 shows the contents of a standard FC frame.[0058]
• FIG. 4 is a block diagram of a common personal computer architecture including a SCSI bus.[0059]
• FIG. 5 illustrates the SCSI bus topology.[0060]
• FIGS.[0061]6A-C illustrate the SCSI protocol involved in the initiation and implementation of read and write I/O operations.
• FIGS.[0062]7A-B illustrate a mapping of the FC Protocol to SCSI sequences exchanged between an initiator and target and the SCSI bus phases and states described in FIGS.6A-C.
• FIGS.[0063]8A-D illustrate several problems related to the use of FC disks in disk arrays.
• FIG. 9 abstractly illustrates a storage-shelf router, representing one embodiment of the present invention, using the illustration convention employed for FIGS.[0064]8A-D.
• FIG. 10 illustrates the position, within a hierarchically interconnected system of computers and a disk array, occupied by a storage-shelf router that represents one embodiment of the present invention.[0065]
• FIGS.[0066]11-12 show a perspective view of the components of a storage shelf implemented using the storage-shelf routers that represent one embodiment of the present invention.
• FIGS.[0067]13A-C illustrate three different implementations of storage shelves using the storage-shelf router that represents one embodiment of the present invention.
• FIGS.[0068]14A-B illustrate two implementations of a path controller card suitable for interconnecting an ATA disk drive with two storage-shelf routers.
• FIG. 15 is a high-level block diagram illustrating the major functional components of a storage-shelf router.[0069]
• FIGS.[0070]16A-G illustrate a number of different logical interfaces provided by a high-availability storage shelf incorporating one or more storage-shelf routers that represent one embodiment of the present invention.
• FIGS.[0071]17A-F illustrate the flow of data and control information through the storage-shelf router that represents one embodiment of the present invention.
• FIG. 18 is a detailed block-diagram representation of the logical components of a storage-shelf router that represents one embodiment of the present invention.[0072]
• FIG. 19 is a detailed diagram of the FC-port layer.[0073]
• FIG. 20 is a detailed block-diagram representation of the routing layer.[0074]
• FIG. 21 is a detailed block-diagram representation of the FCP layer.[0075]
• FIG. 22 is a detailed block-diagram representation of the SATA-port layer.[0076]
• FIG. 23 is a detailed, block-diagram representation of an SATA port.[0077]
• FIG. 24 shows an abstract representation of the routing topology within a four-storage-shelf-router-availability storage shelf.[0078]
• FIG. 25 shows an abstract representation of the X and Y FC arbitrated loop interconnections within a two-storage-shelf-router, two-storage-shelf implementation of a disk array.[0079]
• FIGS.[0080]26A-E illustrate the data fields within an FC-frame header that are used for routing FC frames to particular storage-shelf routers or to remote entities via particular FC ports within the storage shelf that represents one embodiment of the present invention.
• FIG. 27 illustrates seven main routing tables maintained within the storage-shelf router to facilitate routing of FC frames by the routing layer.[0081]
• FIG. 28 provides a simplified routing topology and routing-destination nomenclature used in the flow-control diagrams.[0082]
• FIGS.[0083]29-35 are a hierarchical series of flow-control diagrams describing the routing layer logic.
• FIG. 36A illustrates the conceptual interface levels within a disk array including a storage-shelf router that represents an embodiment of the present invention.[0084]
• FIG. 36B illustrates a storage-shelf-router-interface tunneling method and system that represents an embodiment of the present invention.[0085]
• FIG. 37 illustrates various data structures included in various components of a high-availability storage shelf that represents one embodiment of the present invention.[0086]
• FIGS.[0087]38A-F show the FCP_CMND and FCP_CMND CDB field layouts for the WRITE_BUFFER and READ_BUFFER commands, as well as an example command header and response.
• FIGS.[0088]39A-B illustrate the WRITE-BUFFER protocol, in the context of APT tunneling, when a transfer of data from an external processing entity to the storage shelf is involved.
• FIGS.[0089]40A-B show a control-flow diagram, similar to the control-flow diagram of FIGS.39A-B, describing the WRITE-BUFFER and READ-BUFFER protocols with respect to an APT operation that transfers data from the storage shelf back to the external processing entity.
• FIG. 41 is a control-flow diagram for the WRITE-BUFFER and READ-BUFFER command protocols in an APT context associated with a control-type command that does not transfer data between the external processing entity and the storage shelf, other than the command specification and return status.[0090]
DETAILED DESCRIPTION OF THE INVENTION
• One embodiment of the present invention is an integrated-circuit implementation of a storage-shelf router that may be employed, alone or in combination, within a storage shelf of a disk array or other large, separately controlled mass storage device, to interconnect disk drives within the storage shelf to a high-bandwidth communications medium that, in turn, interconnects the storage shelf with a disk-array controller, or controller of a similar high capacity mass storage device. The described embodiment also includes path controller cards that provide redundant communications links between disk drives and one or more storage-shelf routers. As discussed above, with reference to FIGS.[0091]8A-D, disk arrays may currently employ FC-compatible disk drives within storage shelves, each FC-compatible disk drive acting as an FC node on one or two FC arbitrated loops, or other FC fabric topologies, that interconnect the FC compatible disk drives with a disk-array controller. By contrast, the storage-shelf router that represents, in part, one embodiment of the present invention serves as an intermediary communications hub, directly connected by point-to-point serial communications media to each disk drive within the storage shelf, and interconnected with the disk-array controller via one or more high-bandwidth communications media, such as fibre channel arbitrated loops.
• One embodiment of the present invention is described, in detail, in following subsections including: (1) Overview; (2) Storage-Shelf Internal Topologies; (3) Path Controller Card Overview; (4) Storage-Shelf-Router Overview; (5) Storage-Shelf Interfaces; (6) Storage-Shelf Router Implementation; and (7) Storage-Shelf-Router Routing Layer. In addition, a computer program appendix is provided on two compact disks, and is hereby incorporated by reference. The computer program appendix includes RTL code specifying routing layer logic, a C-language model of a storage-shelf-router implementation, and five chapters of a reference manual for a storage-shelf router implementation. The files included on each of the two compact disks are specified below:[0092]

  RTL Routing-Layer Specification

  File Name	File Size	Date/Time Created

  rl	45	KB	Jan. 3, 2003	11:25	AM
  rl_router	145	KB	Dec. 4, 2002	10:48	AM
  rl_rt
  	20	KB	Jan. 3, 2003	11:25	AM
  rl_rt_ert	86	KB	Oct. 8, 2002	6:21	PM
  rl_rt_irt	168	KB	Dec. 3, 2002	6:31	PM
  rl_rt_itt
  	12	KB	Jan. 3, 2003	11:26	AM
  rl_rt_lpt
  	16	KB	Dec. 3, 2002	6:31	PM
  C-LanguageModel
  sim1216.c
  	55	KB	Sep. 5, 2002	1:43	AM
  Reference Manual
  chaps1_5	1,881	KB	Jan. 13, 2003	11:36	AM

Overview
• FIG. 9 abstractly illustrates a storage-shelf router, representing one embodiment of the present invention, using the illustration conventions employed for FIGS.[0093]8A-D. In FIG. 9, disk-array controller902 is linked, via a LAN or fiber-optic communications medium904, to one or more remote computer systems. The disk-array controller902 is interconnected with a storage-shelf router906 via an FC arbitratedloop908. The storage-shelf router906 is directly interconnected with each of the disk drives within a storage shelf910-917 via separate point-to-point interconnects, such asinterconnect918. Comparing the implementation abstractly illustrated in FIG. 9 with the implementations illustrated in FIGS.8A-D, it is readily apparent that problems identified with the implementation shown in FIGS.8A-D are addressed by the storage-shelf-router-based implementation. First, the only node delay within the FC arbitrated loop of the implementation shown in FIG. 9 is that introduced by the storage-shelf router, acting as a single FC arbitrated loop node. By contrast, as shown in FIG. 8A, each FC-compatible disk drive introduces a separate node delay, and the cumulative node delay on the FC arbitratedloop814 is proportional to the number of FC-compatible disk drives interconnected by the FC arbitrated loop. The storage-shelf router is designed to facilitate highly parallel and efficient data transfer between FC ports and the internal serial interconnects linking the storage-shelf router to individual disk drives. Therefore, there is no substantial delay, and no cumulative delay, introduced by the storage-shelf router other than the inevitable node delay introduced by on board FC controllers that interconnect the storage-shelf router to the FC arbitratedloop908.
• Second, a much larger number of disk drives may be interconnected with a disk-array controller using the storage-shelf router. The FC arbitrated[0094]loop908 employed in the implementation shown in FIG. 9 contains only two nodes, the disk-array controller and the storage-shelf router. Assuming that each storage-shelf router can interconnect eight disk drives with the FC arbitrated loop, a single FC arbitrated loop can be used to interconnect125 storage-shelf routers to a disk-array controller, or126 storage-shelf routers if an address normally reserved for the FC fabric (public loop) is used by a storage-shelf router, thereby interconnecting8,000 or more individual disk drives with the disk-array controller via a single FC arbitrated loop. As noted above, when high availability is not needed, 16,000 or more individual disk drives may be interconnected with the disk-array controller via a single FC arbitrated loop. By contrast, as illustrated in FIG. 8C, when individual FC-compatible disk drives each function as a separate FC node, only 125 disk drives may be interconnected with the disk-array controller via a single FC arbitrated loop, or126 disk drives if an address normally reserved for the FC fabric (public loop) is used for a disk drive.
• Third, it is far easier to identify and isolate failures within a storage shelf employing the storage-shelf router. In a traditional, one-disk-drive-per-FC-arbitrated-loop-node implementation, a failed FC port or FC link may be extremely difficult to identify, and may be difficult to isolate, under certain failure modes, while, by contrast, problems associated with the disk drives interconnected through point-to-point links to a storage-shelf router are relatively easily identified, and failed ports or links are easily isolated.[0095]
• The disk drives are connected to the storage-[0096]shelf router906 via any of a number of currently available internal interconnection technologies. In one embodiment, SATA-compatible interconnects are used to interconnect SATA disk drives with the storage-shelf router. A storage-shelf router includes logic that translates each FCP command received from the disk-array controller into one or more equivalent ATA-interface commands that the storage-shelf router then transmits to an appropriate SATA disk drive. The storage-shelf router shown in FIG. 9 is interconnected with the disk-array controller via a single FC arbitratedloop908, but, as discussed below, a storage-shelf router is more commonly interconnected with the disk-array controller through two FC arbitrated loops or other FC fabric topologies.
• FIG. 10 illustrates the position, within a hierarchically interconnected system of computers and a disk array, occupied by a storage-shelf router that represents, in part, one embodiment of the present invention. In FIG. 10, two[0097]server computers1001 and1004 are interconnected with each other, and with a disk-array controller1006, via a high-bandwidth communications medium1008, such as any of various FC fabric topologies. The disk-array controller1006 is interconnected with astorage shelf1010 via two separate FC arbitrated loops. The first FC arbitratedloop1012 directly interconnects the disk-array controller1006 with a first storage-shelf router1014. The second FC arbitratedloop1016 directly interconnects the disk-array controller1006 with a second storage-shelf router1018. The two storage-shelf routers1014 and1018 are interconnected with an internal point-to-point FC interconnection1020 that carries FC frames from the first storage-shelf router1014 to the second storage-shelf router1018 as part of the first FC arbitratedloop1012, and carries FC frames between the second storage-shelf router1018 and first storage-shelf router1014 as part of the second FC arbitratedloop1016. In addition, theinternal FC link1020 may carry FC frames used for internal management and communications that are internally generated and internally consumed within thestorage shelf1010. As discussed below, it is common to refer to the two FC arbitrated loops interconnecting the disk-array with the storage shelf as the “X loop” or “X fabric” and the “Y loop” or “Y fabric,” and to refer to the exchange of internally generated and internally consumed management FC frames on theinternal FC1020 as the S fabric. Thestorage shelf1010 includes 16 SATA disk drives represented in FIG. 10 by the four disk drives1022-1025 and theellipsis1026 indicating12 disk drives not explicitly shown. Each storage-shelf router1014 and1018 is interconnected with each SATA disk drive via point-to-point serial links, such asserial link1028.
• As shown in FIG. 10, there is at least two-fold redundancy in each of the intercommunications pathways within the disk array containing the disk-[0098]array controller1006 and thestorage shelf1010. Moreover, there is a two-fold redundancy in storage-shelf routers. If any single link, or one storage-shelf router, fails, the remaining links and remaining storage-shelf router can assume the workload previously assumed by the failed link or failed storage-shelf router to maintain full connectivity between the disk-array controller1006 and each of the sixteen SATA disk drives within thestorage shelf1010. The disk-array controller may additionally implement any of a number of different high-availability data-storage schemes, such as the various levels of RAID storage technologies, to enable recovery and full operation despite the failure of one or more of the SATA disk drives. The RAID technologies may, for example, separately and fully redundantly restore two or more complete copies of stored data on two or more disk drives. The servers intercommunicate with the disk-array comprising the disk-array controller1006 and one or more storage shelves, such asstorage shelf1010, through a communications medium, such as an FC fabric, with built-in redundancy and failover. The disk-array controller presents a logical unit (“LUN”) and logical block address (“LBA”) interface that allows theserver computers1002 and1004 to store and retrieve files and other data objects from the disk array without regard for the actual location of the data within and among the disk drives in the storage shelf, and without regard to redundant copying of data and other functionalities and features provided by the disk-array controller1006. The disk-array controller1006, in turn, interfaces to thestorage shelf1010 through an interface provided by the storage-shelf routers1014 and1018. The disk-array controller1006 transmits FC exchanges to, and receives FC exchanges from, what appear to be discrete FC-compatible disk drives via the FCP protocol. However, transparently to the disk-array controller, the disk-shelf routers1014 and1018 translate FC commands into ATA commands in order to exchange commands and data with the SATA disk drives.
• FIGS.[0099]11-12 show a perspective view of the components of a storage shelf implemented using the storage-shelf routers that represent one embodiment of the present invention. In FIG. 11, two storage-shelf routers1102 and1104 mounted on router cards interconnect, via apassive midplane1106, with 16 SATA disk drives, such asSATA disk drive1108. Each SATA disk drive carrier contains an SATA disk drive and apath controller card1110 that interconnects the SATA disk drive with two separate serial links that run through the passive midplane to each of the two storage-shelf routers1102 and1104. Normally, a SATA disk drive supports only a single serial connection to an external system. In order to provide fully redundant interconnections within the storage shelf, thepath controller card1110 is needed. Thestorage shelf1100 additionally includesredundant fans1112 and1114 andredundant power supplies1116 and1118. FIG. 12 shows a storage-shelf implementation, similar to that shown in FIG. 11, with dual SATA disk drive carriers that each includes two path controller cards and two SATA disk drives. The increased number of disk drives necessitates a corresponding doubling of storage-shelf routers, in order to provide the two-fold redundancy needed for a high-availability application.
Storage Shelf Internal Topologies
• FIGS.[0100]13A-C illustrate three different implementations of storage shelves using the storage-shelf router that represents, in part, one embodiment of the present invention. In FIG. 13A, a single storage-shelf router1302 interconnects16 SATA disk drives1304-1319 with a disk-array controller via an FC arbitratedloop1320. In one embodiment, the storage-shelf router provides a maximum of 16 serial links, and can support interconnection of up to 16 SATA disk drives. The storage shelf shown in FIG. 13A is not highly available, because it contains neither a redundant storage-shelf router nor redundant serial links between one or more routers and each SATA disk drive.
• By contrast, the storage-shelf implementation shown in FIG. 13B is highly available. In this storage shelf, two storage-[0101]shelf routers1322 and1324 are linked via point-to-point serial links to each of the 16 SATA disk drives1326-1341. During normal operation, storage-shelf router1322 interconnects half of the SATA disk drives1326-1333 to the disk-array controller, while storage-shelf router1324 interconnects the other half of the SATA disk drives1334-1341 to the disk-array controller. The internal point-to-point serial links employed during normal operation are shown in bold in FIG. 13B, such asserial link1342, and are referred to as “primary links.” Those internal serial links not used during normal operation, such as interiorserial link1344, are referred to as “secondary links.” If a primary link fails during operation, then the failed primary link, and all other primary links connected to a storage-shelf router, may be failed over from the storage-shelf router to which the failed primary link is connected to the other storage-shelf router, to enable the failed primary link to be repaired or replaced, including replacing the storage-shelf router to which the failed primary link is connected. As discussed above, each of the two storage-shelf routers serves as the FC node for one of two FC arbitrated loops that interconnect the storage shelf with a disk-array controller. Should one FC arbitrated loop fail, data transfer that would normally pass through the failed FC arbitrated loop is failed over to the remaining, operable FC arbitrated loop. Similarly, should a storage-shelf router fail, the other storage-shelf router can assume the full operational control of the storage shelf. In alternative embodiments, a primary path failure may be individually failed over, without failing over the entire storage-shelf router. In certain embodiments and situations, a primary-path failover may be carried out within the storage-shelf router, while in other embodiments and situations, the primary-path failover may involve failing the primary path over to a second storage-shelf router.
• FIG. 13C illustrates implementation of a 32-ATA-disk high availability storage shelf. As shown in FIG. 13C, the 32-ATA-disk storage shelf includes four storage-[0102]shelf routers1350,1352,1354, and1356. Each storage-shelf router, during normal operation, interconnects eight SATA disks with the two FC arbitrated loops that interconnect the storage shelf with a disk-array controller. Each storage-shelf router is interconnected via secondary links to eight additional SATA disk drives so that, should failover be necessary, a storage-shelf router can interconnect a total of 16 SATA disk drives with the two FC arbitrated loops. Note that, in the four-storage-shelf-router configuration, storage-shelf router1350 serves as the FC node for all four storage-shelf routers with respect to one FC arbitrated loop, and storage-shelf router1356 serves as the FC node for all four storage-shelf routers with respect to the second FC arbitrated loop. As shown in FIG. 13C, the first FC arbitrated loop for which storage-shelf router1350 serves as FC node is considered the X loop or X fabric, and the other FC arbitrated loop, for which storage-shelf router1356 serves as the FC node, is considered the Y fabric or Y loop. FC frames transmitted from the disk-array controller via the X loop to an SATA disk within the storage shelf are first received by storage-shelf router1350. The FC frames are either directed to an SATA disk interconnected with storage-shelf router1350 via primary links, in the case of normal operation, or are directed via theinternal FC link1358 to storage-shelf router1352 which, in turn, either transforms the FC frames into one or more ATA commands that are transmitted through a primary link to an SATA disk, or forwards the FC frame downstream to storage-shelf router1354. If a response FC frame is transmitted by storage-shelf router1356 via the X fabric, then it must be forwarded throughinternal FC links1360,1362, and1358 via storage-shelf routers1354 and1352 to storage-shelf router1350, from which the response frame can be transmitted to the external X fabric. In the described embodiment, a high availability storage shelf needs to contain at least two storage-shelf routers, and needs to contain a storage-shelf router for each set of eight SATA disks within the storage shelf.
Path Controller Card Overview
• As discussed above, two components facilitate construction of a high availability storage shelf employing SATA disks, or other inexpensive disk drives, that can be interconnected with an FC arbitrated loop or other high-bandwidth communications medium using only a single slot or node on the FC arbitrated loop. One component is the storage-shelf router and the other component is the path controller card that provides redundant interconnection of an ATA drive to two storage-shelf routers. FIGS.[0103]14A-B illustrate two implementations of a path control card suitable for interconnecting an ATA disk drive with two storage-shelf routers. The implementation shown in FIG. 14A provides a parallel connector to a parallel ATA disk drive, and the implementation shown in FIG. 14B provides a serial connection to a SATA disk drive. Because, as discussed above, SATA disk drives provide higher data transfer rates, the implementation shown in FIG. 14B is preferred, and is the implementation that is discussed below.
• The path controller card provides an SCA-2[0104]connector1402 for external connection of a primaryserial link1404 and amanagement link1406 to a first storage-shelf router, and a secondaryserial link1408 andsecond management link1410 to a second storage-shelf router. The primary link and secondary link are multiplexed by a 2:1 multiplexer that is interconnected via aserial link1414 to aSATA disk drive1416. The management links1406 and1410 are input to amicrocontroller1418 that runs management services routines, such as routines that monitor the temperature of the disk drive environment, control operation of a fan within the disk drive carrier, and activate various light emitting diode (“LED”) signal lights visible from the exterior of the disk drive enclosure. In essence, under normal operation, ATA commands and data are received by the path controller card via the primary link, and are transferred via the 2:1 multiplexer to theserial link1414 input to theSATA disk drive1416. If a failover occurs within the storage shelf that deactivates the default storage-shelf router connected via the primary link to the path controller card, a second storage-shelf router assumes transfer of ATA commands and data via the secondary link which are, in turn, passed through the 2:1 multiplexer to theserial link1414 that is directly input to theSATA disk drive1416.
• The path controller card provides redundant interconnection to two separate storage-shelf routers, and is thus needed in order to provide the two-fold redundancy needed in a high availability storage shelf. The storage-shelf router provides interconnection between different types of communications medium and translation of commands and data packets between the different types of communications media. In addition, the storage-shelf router includes fail-over logic for automatic detection of internal component failures and execution of appropriate fail-over plans to restore full interconnection of disk drives with the disk-array controller using redundant links and non-failed components.[0105]
Storage-Shelf Router Overview
• FIG. 15 is a high-level block diagram illustrating the major functional components of a storage-shelf router. The storage-[0106]shelf router1500 includes twoFC ports1502 and1504, arouting layer1506, anFCP layer1508, a global sharedmemory switch1510,16 SATA ports1512-1518, aCPU complex1520, and anexternal flash memory1514. Depending on the logical position of the storage-shelf router within the set of storage-shelf routers interconnected within a storage shelf, one or both of the FC ports may be connected to an external FC arbitrated loop or other FC fabric, and one or both of the FC ports may be connected to internal point-to-point FC links. In general, one of the FC ports, regardless of the logical and physical positions of the storage-shelf router within a set of storage-shelf routers, may be considered to link the storage-shelf router either directly or indirectly with a first FC arbitrated loop, and the other FC port can be considered to directly or indirectly interconnect the storage-shelf router with a second FC arbitrated loop.
• The[0107]routing layer1506 comprises a number of routing tables stored in a memory, discussed below, and routing logic that determines where to forward incoming FC frames from both FC ports. TheFCP layer1508 comprises control logic for processing various types of incoming and outgoing FC frames, an interface to a global shared memory switch, described below, and an interface to theCPU complex1512 to allow firmware routines executing on the CPU complex to process FCP_CMND frames in order to set up FC exchange contexts in memory to facilitate the exchange of FC frames that together compose an FCP exchange.
• The global shared[0108]memory switch1510 is an extremely high-speed, time-multiplexed data-exchange facility for passing data between FCP-layer and the SATA ports1512-1518. The global shared memory switch (“GSMS”)1510 employs a virtual queue mechanism to allow allocation of a virtual queue to facilitate the transfer of data between the FCP layer and a particular SATA port. The GSMS is essentially a very high-bandwidth, high-speed bidirectional multiplexer that facilitates highly parallel data flow between the FCP layer and the 16 SATA ports, and that includes synchronization mechanisms to facilitate traversal of the synchronization boundary between the FCP layer and the SATA ports.
• The[0109]CPU complex1512 runs various firmware routines that process FCP commands in order to initialize and maintain context information for FC exchanges and translate FCP commands into ATA-equivalent commands, and that also monitor operation of the SATA disk drives and internal components of the storage-shelf router and carry out sophisticated fail-over strategies when problems are detected. In order to carry out the fail-over strategies, the CPU complex is interconnected with the other logical components of the storage-shelf router. Theexternal flash memory1514 stores configuration parameters and firmware routines. Note that the storage-shelf router is interconnected to external components via the twoFC ports1502 and1504, the 16 SATA ports1512-1518,16serial management links1520, an I²C BUS1522, and a link to aconsole1524.
Storage-Shelf Interfaces
• As discussed above, storage-shelf-router-based storage-shelf implementations provide greater flexibility, in many ways, than do current, FC-node-per-disk-drive implementations. The storage-shelf router can provide any of many different logical interfaces to the disk-array controller to which it is connected. FIGS.[0110]16A-G illustrate a number of different logical interfaces provided by a high-availability storage shelf incorporating one or more storage-shelf routers that, in part, represent one embodiment of the present invention. FIG. 16A shows the interface provided by current FC-compatible disk drive implementations of storage shelves, as described above with reference to FIGS.8A-D. FIG. 16A uses an abstract illustration convention used throughout FIGS.16A-H. In FIG. 16A, each disk drive1602-1605 is logically represented as a series of data blocks numbered 0 through 19. Of course, an actual disk drive contains hundreds of thousands to millions of logical blocks, but the 20 logical blocks shown for each disk in FIG. 16A are sufficient to illustrate various different types of interfaces. In FIG. 16A, each separate disk drive1602-1605 is a discrete node on an FC arbitrated loop, and therefore each disk drive is associated with a separate FC node address, or arbitrated-loop physical address (“AL_PA”), represented in FIG. 16A as “AL_PA1,” “AL_PA2,” “AL_PA3,” and “AL_PA4,” respectively. Note, however, that unlike in current, FC-arbitrated-loop disk-array implementations, such as those discussed with reference to FIGS.8A-D, there is no cumulative node delay associated with the nodes, because each node is interconnected with the complementary SATA port of the storage-shelf router via a point-to-point connection, as shown in FIG. 9. Thus, a disk-array controller may access a particular logical block within a particular disk drive via an FC address associated with the disk drives. A given disk drive may, in certain cases, provide a logical unit (“LUN”) interface in which the logical-block-address space is partitioned into separate logical-block-address spaces, each associated with a different LUN. However, for the purposes of the current discussion, that level of complexity need not be addressed.
• FIG. 16B shows a first possible interface for a storage shelf including the four disk drives shown in FIG. 16A interconnected to the FC arbitrated loop via a storage-shelf router. In this first interface, each disk drive remains associated with a separate FC node address. Each disk drive is considered to be a single logical unit containing a single logical-block-address space. This interface is referred to, below, as “transparent mode” operation of a storage shelf containing one or more storage-shelf routers that represent, in part, one embodiment of the present invention.[0111]
• A second possible interface provided by a storage shelf is shown in FIG. 16C. In this case, all four disk drives are associated with a single FC-arbitrated-loop-node address “AL_PA1.” Each disk drive is considered to be a different logical unit, with[0112]disk drive1602 considered logical unit zero,disk drive1603 considered logical unit one,disk drive1604 considered logical unit two, anddisk drive1605 considered logical unit three. Thus, a disk-array controller can access a logical block within any of the four disk drives in the storage shelf via a single FC-node address, a logical unit number, and a logical block address within the logical unit.
• An alternative interface to the four disk drives within the hypothetical storage shelf is shown in FIG. 16D. In this case, all four disk drives are considered to be included within a single logical unit. Each logical block within the four disk drives is assigned a unique logical block address. Thus, logical blocks 0-19 in[0113]disk drive1602 continue to be associated with logical block addresses 0-19, while logical blocks 0-19 indisk drive1603 are now associated with logical block address20-39. This interface is referred to, below, as a pure logical-block-address interface, as opposed to the pure LUN-based interface shown in FIG. 16C.
• FIG. 16E shows yet another possible logical interface provided by a hypothetical storage shelf containing four disk drives. In this case, the first set of two[0114]disk drives1602 and1603 is associated with a first FC node address “AL_PA1,” and the twodisk drives1602 and1603 are associated with two different LUN numbers,LUN 0 andLUN 1, respectively. Similarly, the second pair ofdisk drives1604 and1605 are together associated with a second FC node address “AL_PA2,” and each of the second pair of disk drives is associated with a different LUN number.
• FIG. 16F shows yet another possible interface. In this case, the first two[0115]disk drives1602 and1603 are associated with a first FC node address, and the second twodisk drives1604 and1605 are associated with a second FC node address. However, in this case, the two disk drives in each group are considered to both belong to a single logical unit, and the logical blocks within the two disk drives are associated with logical block addresses that constitute a single logical-block-address space.
• A final interface is shown in FIG. 16G. In this case, as in the previous two interfaces, and each pair of disk drives associated with a single FC node address are considered to constitute a single LUN with a single logical-block-address space. However, at this interface, the logical block addresses alternate between the two disk drives. For example, in the case of the pair of[0116]disk drives1602 and1603,logical block address 0 is associated with the firstlogical block1610 and thefirst disk drive1602, andlogical block address 1 is associated with thefirst block1612 in thesecond disk drive1603.
• FIGS.[0117]16A-G are meant simply to illustrate certain of the many possible interfaces provided to a disk-array controller by storage-shelf routers that represent, in part, one embodiment of the present invention. Almost any mapping of LUNs and logical block addresses to disk drives and physical blocks within disk drives that can be algorithmically described can be implemented by the storage-shelf routers within a storage shelf. In general, these many different types of logical interfaces may be partitioned into the following four general types of interfaces: (1) transparent mode, in which each disk drive is associated with a separate and locally unique FC node address; (2) pure LUN mode, in which each disk drive is associated with a different LUN number, and all disk drives are accessed through a single FC node address; (3) pure logical-block-addressing mode, in which all disk drives are associated with a single FC node address and with a single logical unit number; and (4) mixed LUN and logical-block-addressing modes that employ various different combinations of LUN and logical-block-address-space partitionings.
Storage-Shelf Router Implementation
• FIG. 17A is a high-level overview of the command-and-data flow within the storage-shelf router that represents one embodiment of the present invention. The storage-shelf router exchanges serial streams of data and commands with other storage-shelf routers and with a disk-array controller via one or more FC arbitrated loops or other FC fabrics[0118]1702-1703. The serial streams of data enterFC port layer1704, where they are processed at lower-level FC protocol levels. FC frames extracted from the data streams are input into first-in-first-out buffers (“FIFOs”)1706-1707. As the initial portions of FC frames become available, they are processed by therouting layer1708 and FCP-layer1710, even as latter portions of the FC frames are input into the FIFOs. Thus, the FC frames are processed with great time and computing efficiency, without needing to be fully assembled in buffers and copied from internal memory buffer to internal memory buffer.
• The[0119]routing layer1708 is responsible for determining, from FC frame headers, whether the FC frames are directed to the storage router, or to remote storage routers or other entities interconnected with the storage router by the FC arbitrated loops or other FC fabrics. Those frames directed to remote entities are directed by the routing layer to output FIFOs1712-1713 within the FC-port layer for transmission via the FC arbitrated loops or other FC fabrics to the remote entities. Frames directed to the storage router are directed by the routing layer to the FCP-layer, where state machines control their disposition within the storage-shelf router.
• FCP-DATA frames associated with currently active FC exchanges, for which contexts have been established by the storage-shelf router, are processed in a highly streamlined and efficient manner. The data from these frames is directed by the FCP-layer to virtual queues[0120]1714-1716 within theGSMS1718, from which the data is transferred to aninput buffer1720 within the SATA-port layer1722. From the SATA-port layer, the data is transmitted in SATA packets, or DATA FRAME INFORMATION STRUCTURES (“DATA FISes”), via one ofmany SATA links1724 to one of the number ofSATA disk drives1726 interconnected with the storage-shelf router.
• FCP-CMND frames are processed by the FCP-layer in a different fashion. These frames are transferred by the FCP-layer to a[0121]memory1728 shared between the FCP-layer and the CPUs within the storage-shelf router. The CPUs access the frames in order to process the commands contained within them. For example, when an incoming WRITE command is received, a storage-shelf-router CPU, under control of firmware routines, needs to determine to which SATA drive the command is directed and establish a context, stored in shared memory, for the WRITE operation. The CPU needs to prepare the SATA drive to receive the data, and direct transmission of an FCP_XFER_RDY frame back to the initiator, generally the disk-array controller. The context prepared by the CPU and stored in shared memory allows the FCP-layer to process subsequent incoming FCP_DATA messages without CPU intervention, streamlining execution of the WRITE operation.
• The various logical layers within the storage-shelf router function generally symmetrically in the reverse direction. Responses to ATA commands are received by the SATA-[0122]port layer1722 from SATA disk drives via the SATA links. The SATA-port layer then generates the appropriate signals and messages, to enable, when necessary, a CPU, under firmware control, to carry out the appropriate actions and responses. When data is transferred from an SATA disk to a remote entity, in response to a READ command, a CPU generates an appropriate queue entry that is stored in shared memory for processing by the FCP-layer. State machines within the FCP layer obtain, from shared memory, an FC frame header template, arrange for data transfer from the virtual queue1732-1733 associated with the READ command, prepare an FC frame header, and coordinate transfer of the FC frame header and data received from the SATA port layer through the virtual queue tooutput FIFOs1712 and1713 of the FC-port layer for transmission to the requesting remote entity, generally a disk-array controller.
• FIG. 17A is intended to provide a simplified overview of data and control flow within the storage-shelf router. It is not intended to accurately portray the internal components of the storage-shelf router, but rather to show the interrelationships between logical layers with respect to receiving and processing FCP_CMND and FCP_DATA frames. For example, a number of virtual queues are shown in FIG. 17A within the GSMS layer. However, virtual queues are generally not static entities, but are dynamically allocated as needed, depending on the current state of the storage-shelf router. FIG. 17A shows only a single SATA[0123]serial connection1724 andSATA disk drive1726, but, as discussed above, each storage router may be connected to 16 different SATA disk drives, in one embodiment.
• FIGS.[0124]17B-F provide greater detail about the flow of data and control information through the storage-shelf router that represents one embodiment of the present invention. In describing FIGS.17B-F, specific reference to both components of various pairs of identical components is not made, in the interest of brevity. The figures are intended to show how data and control information moves through various components of the storage-shelf router, rather than as a complete illustrated list of components. Moreover, the numbers of various components may vary, depending on various different implementations of the storage-shelf router. FIG. 17B shows the initial flow of FCP_DATA frames within the storage-shelf router. An FCP_DATA frame is first received by anFC port1736 and written to aninput FIFO1737, from which it may start being processed by therouter logic1738 as soon as sufficient header information is available in the input FIFO, even while the remainder of the FCP_DATA frame is still being written to the input FIFO. The FC port signals arrival of a new frame to the router logic to enable the router logic to begin processing the frame. Therouter logic1738 employs routing tables1739 to determine whether or not the frame is directed to the storage-shelf router, or whether the frame is directed to a remote entity. If the FCP-DATA frame is directed to a remote entity, the frame is directed by the router logic to an FC port for transmission to the remote entity. The router also interfaces withcontext logic1740 to determine whether or not a context has been created and stored in shared memory by a CPU for the FC exchange to which the FCP-DATA frame belongs. If a context for the frame can be found, then the router logic directs the frame to the FCP Inbound Sequence Manager (“FISM”) state machinel741. If a context is not found, the frame is directed to shared memory, from which it is subsequently extracted and processed as an erroneously received frame by a CPU under firmware control.
• The[0125]DISM1741 requests a GSMS channel from an FCP data mover logic module (“FDM”)1742, which, in turn, accesses a virtual queue (“VQ”)1743 within theGSMS1744, receiving parameters characterizing the VQ from the context logic via the FISM. The FDM then writes the data contained within the frame to the VQ, from which it is pulled by the SATA port that shares access to the VQ with the FDM for transmission to an SATA disk drive. Note that the SATA port may begin to transfer data to the disk drive as soon as the data begins to be written to the VQ. THe SATA port need not wait for all of the data to be written before beginning to transmit the data to the disk drive. Once all of the data associated with a data transfer operation is written to the VQ, the FDM signals the context manager that the data has been transferred, and the context manager, in turn, requests that a completion queue manager (“CQM”)1745 queues a completion message (“CMSG”) to acompletion queue1746 within sharedmemory1747. The CQM, in turn, requests that a CPU data mover (“CPUDM”)1748 write the CMSG into shared memory.
• FIG. 17C shows flow of FC_CMND frames, and frames associated with errors, within the storage shelf router. As discussed above, frames are received by an[0126]FC port1736 and directed byrouter logic1738, with reference to routing tables1739, to various target components within the storage-shelf router. FCP_CMND frames and FC frames received in error are routed to sharedmemory1747 for extraction and processing by a CPU. Therouting logic1738 issues a request for a frame buffer queue manager (“FBQM”)1760 to write the frame to sharedmemory1747. The FBQM receives a buffer pointer, stored in sharedmemory1750, from theCPUDM1748, and writes the frame to aframe buffer1749 within sharedmemory1747. Note that, in certain cases in which frames are transferred for error handling to shared memory, more than one frame buffer may be needed to accommodate larger frames, or to accommodate multiple frames. Finally, the router requests theCQM1745 to write a CMSG to theCQ1746. A CPU eventually processes the CMSG, using information contained within the CMSG to access the frame stored in aframe buffer1749.
• FIG. 17D shows the flow of FC frames from one FC port to another. In the case that the[0127]router logic1736 determines that a frame received via aninput FIFO1737 within afirst FC port1736 is not directed to the storage router, but is instead directed to a remote entity, the router logic writes the frame to anoutput FIFO1751 within asecond FC port1752 to transmit the frame to the remote entity.
• FIG. 17E shows flow of data and control information from a CPU within the storage-shelf router to the S fabric, an FC arbitrated loop, or another FC fabric. A CPU, under firmware control, stores an entry within a shared-memory queue “SRQ” within shared[0128]memory1747 and updates an SRQ producer index associated with the SRQ to indicate the presence of an SRQ entry (“SRE”) describing a sequence or frame that the CPU has created for transmission to an FC arbitrated loop or other FC fabric, along with pointers to headers and buffers. An SRQ manager module (“SRQM”)1755 detects the update of the SRQ producer index, and fetches a next SRE from sharedmemory1747 via theCPUDM1748. The SRQM passes the fetched SRE to an SRQ arbitration module (“SRQ_ARB”)1756, which implements an arbitration scheme, such as a round-robin scheme, to ensure processing of SREs generated by multiple CPUs and stored in multiple SRQs. The SRQ_ARB selects an SRQM from which to receive a next SRE, and passes the SRE to a FCP outbound sequence manager (“FOSM”)state machine1757. The FOSM processes the SRE to fetch an FC header template and frame payload from sharedmemory1747 via theCPUDM1748. The FOSM constructs an FC frame using the FC header template and a frame payload via the CPUDM from shared memory and writes it to anoutput FIFO1758 in anFC port1736, from which it is transmitted to an FC arbitrated loop or other FC fabric. When the frame has been transferred to the FC port, the FOSM directs theCQM1745 to write a CMSG to shared memory.
• FIG. 17F shows the flow of data and control information from the GSMS and shared memory to an FC arbitrated loop or other FC fabric. Many of the steps in this process are similar to those described with reference to FIG. 17E, and will not be again described, in the interest of brevity. In general, the control portion of an FCP_DATA frame, stored within the FC-frame header, is generated in similar fashion to generation of any other type of frame, described with reference to FIG. 17E. However, in the case of an FCP_DATA frame, the process needs to be staged in order to combine the control information with data obtained through the GSMS from an SATA port. When the[0129]FOSM1757 receives the SRE describing the FCP_DATA frame, the FOSM must construct the FCP_DATA-frame header, and request the data that is incorporated into the frame via a GSMS channel through theFDM1742, which, in turn, obtains the data via aVQ1759 within theGSMS1744. Once the data and control information are combined by the FOSM into an FCP_DATA frame, the frame is then passed to an FC port, and, when the FCP_DATA frame represents the completion of a data transfer operation, a CMSG message queued to the CQ, as described previously.
• FIG. 18 shows a more detailed block-diagram view of the logical components of a storage-shelf router that represents one embodiment of the present invention. The logical components include two[0130]FC ports1802 and1804, therouting layer1806, theFCP layer1808, theGSMS1810, the SATA-port layer1812, and the CPU complex, including twoCPUs1814 and1816, described above, with respect to FIGS. 16 and 17. The communications paths and links shown in FIG. 18 with bold arrows, such asbold arrow1818, represent the performance-critical communications pathways within the storage-shelf router. The performance-critical pathways are those pathways concerned with receiving and outputting FC frames, processing received frames in order to generate appropriate ATA commands for transmission by SATA ports to SATA disk drives, funneling data from received FCP-DATA frames through the GSMS to SATA ports, generation of FC frames for transmission through FC ports to an FC arbitrated loop or other FC fabric, and incorporating data obtained from SATA ports through the GSMS into outgoing FCP-DATA frames. Non-performance-critical pathways include various programmed I/O interfaces that interconnect theCPUs1814 and1816 directly with the various logical components of the storage-shelf router. For example, there are PIO interfaces between acentral arbitration switch1820 and the GSMS, SL-port layer, and aninternal BUS bridge1822 in turn interconnected with 17UART ports1824, an1²C BUS interface1826, a general PIO interface (“GPIO”)1828, atimer component1830, and several interruptcontrollers1832. These PIO interfaces are shown in FIG. 18 as non-bolded, double-headed arrows1834-1836. In addition, there is aPIO interface1838 between theCPUs1814 and1816 and a flash-memory controller1840 that, in turn, interfaces to anexternal flash memory1842. The external flash memory is used to store specialized configuration management information and firmware images. The CPUs are connected through a memory-access interface1844 to aninternal SRAM controller1846 that, in turn interfaces anSRAM memory1848 that stores non-performance path code and data, including firmware routines for directing fail-over within and between storage-shelf routers. TheCPUs1814 and1816 are interconnected with theFCP layer1808 and the SATA-port layer1812 via shared memory queues contained in two data-tightly-coupledmemories1850 and1852, also used for processor data space. Each CPU is also interconnected with a separate memory that storesfirmware instructions1854 and1856. Finally, both CPUs are connected via asingle PIO channel1858 to bothFC ports1802 and1804, therouting layer1806, and theFCP layer1808.
• FIG. 19 shows a more detailed diagram of the FC-port layer. The FC-port layer comprises two[0131]FC ports1902 and1904, each of which includes aninput FIFO1906 and1908 and two output FIFOs1910-1911 and1912-1913. The FC ports include physical and link layer logic1914-1917 that together transform incoming serial data from an FC arbitrated loop or other FC fabric into FC frames passed to the input FIFOs and that transform outgoing FC frames written to output FIFOs into serial data transmitted to the FC arbitrated loop or other FC fabric.
• FIG. 20 is a more detailed block-diagram representation of the routing layer. As shown in FIG. 20, the[0132]routing layer2002 includesseparate routing logic2004 and2006 for handling each of the FC ports. The routing layer also includes routing tables2008 stored in registers, essentially memory with extremely fast access, to facilitate the routing decisions needed to route incoming FC frames to appropriate queues. Note that FC data frames can be relatively directly routed by the routers to the GSMS layer2015 under control of theFISMs2010 and2012 via the FDM2011, as described above. Frames requiring firmware processing are routed by the routing layer to input queues under control of theFBQMs2014 and2016 via theCPUDMs2017 and2018.
• FIG. 21 is a more detailed block-diagram representation of the FCP layer. Many of these internal components shown in FIG. 21 have been described previously, or are described in more detail in subsequent sections. Note that there are, in general, duplicate sets of components arranged to handle, on one hand, the two[0133]FC ports1902 and1904, and, on the other hand, the twoCPUs2102 and2104. Information needed to generate outgoing frames is generated by the CPUs, under firmware control, and stored in sharedmemories2106 and2108, each associated primarily with a single CPU. The stored information within each memory is then processed by separate sets ofSRQMs2110 and2112,FOSMs2114 and2116, SRQ_ARBS2118 and2120,CPUDMs2122 and2124, and other components in order to generate FC frames that are passed to the twoFC ports1902 and1904 for transmission. Incoming frames at each FC port are processed byseparate router modules2004 and2006,FISMs2010 and2012, and other components.
• FIG. 22 shows a more detailed block-diagram representation of the SATA-port layer. The primary purpose of the SATA-port layer is for virtual queue management, a task shared between the SATA-port layer, the GSMS, and the FCP layer, and for exchange of data with the FCP layer through the GSMS and individual SATA ports.[0134]
• FIG. 23 is a more detailed, block-diagram representation of an SATA port. The SATA port includes a[0135]physical layer2302, alink layer2304, and atransport layer2306 that together implement an SATA interface. The transport layer includes aninput buffer2308 and anoutput buffer2310 that store portions of data transfers and ATA message information arriving from an interconnected SATA disk, and that store portions of data transfers from the GSMS layer and ATA commands passed from interfaces to CPUs and shared memory, respectively. Additional details regarding the SATA port are discussed in other sections.
Storage-Shelf-Router Routing Layer
• FIG. 24 shows an abstract representation of the routing topology within a four-storage-shelf-router-high-availability storage shelf. This abstract representation is a useful model and template for discussions that follow. As shown on FIG. 24, each storage-shelf router[0136]2402-2405 is connected via primary links to n disk drives, such asdisk drive2406. As discussed above, each storage-shelf router is connected via secondary links to a neighboring set of n disk drives, although the secondary links are not shown in FIG. 24 for the sake of simplicity. One storage-shelf router2402 serves as the end point or FC-node connection point for the entire set of storage-shelf routers with respect to a first FC arbitrated loop or other FC fabric, referred to asFabric X2408. A different storage-shelf router2405 serves as the end point, or FC node connection to a second FC arbitrated loop orother FC fabric2410 referred to as Fabric Y. Each storage-shelf router includes two FC ports, an X port and a Y port, as, for example,X port2412 andY port2414 in storage-shelf router2402. The four storage-shelf routers are interconnected with internal point-to-point FC links2416,2418, and2420. For any particular storage-shelf router, as for example, storage-shelf router2404, FC frames incoming from Fabric X are received on theX port2422 and FC frames output by storage-shelf router2404 to Fabric X are output via theX port2422. Similarly, incoming FC frames and outgoing FC frames are received from, and directed to, the Y Fabric, respectively, are input and output over theFC port2424. It should be noted that the assignments of particular FC ports to the X and Y fabrics are configurable, and, although, in following illustrative examples and discussions referencing the example,FC port 0 is assumed to be the X fabric port andFC port 1 is assumed to be the Y port, an opposite assignment may be configured.
• S-fabric management frames, identified as such by a two-bit reserved subfield within the DF_CTL field of an FC-frame header that is used within the S fabric and that is referred to as the “S bits,” are directed between storage-shelf routers via either X ports or Y ports and the point-to-point, internal FC links. Each storage-shelf router is assigned a router number that is unique within the storage shelf, and that, in management frames, forms part of the FC-frame-header D_ID field. The storage-shelf routers are numbered in strictly increasing order, with respect to one of the X and Y fabrics, and strictly decreasing order with respect to the other of the X and Y fabrics. For example, in FIG. 24, storage-[0137]shelf routers2402,2403,2404, and2405 may be assignedrouter numbers 1, 2, 3, and 4, respectively, and thus may be strictly increasing, or ascending, with respect to the X fabric and strictly decreasing, or descending, with respect to the Y fabric. This ordering is assumed in the detailed flow-control diagrams, discussed below, but, as noted above, the relationship between fabrics and ports, and ascending and descending orders to ports, is configurable.
• FIG. 25 shows an abstract representation of the X and Y FC arbitrated loop interconnections within a two-storage-shelf-router, two-storage-shelf implementation of a disk array. In FIG. 25, the disk-[0138]array controller2502 is linked by FC arbitrated loop X2504 to eachstorage shelf2506 and2508, and is linked by FC arbitratedloop Y2510 to bothstorage shelves2506 and2508. In FIG. 25, storage-shelf router2512 serves as the X-fabric endpoint forstorage shelf2506, and storage-shelf router2514 serves as the X-fabric endpoint forstorage shelf2508. Similarly, storage-shelf router2516 serves as the Y-fabric endpoint forstorage shelf2506 and storage-shelf router2518 serves as the Y-fabric endpoint forstorage shelf2508. Each individual disk drive, such asdisk drive2518, is accessible to the disk-array controller2502 via both the X and the Y arbitrated loops. In both storage shelves, the storage-shelf routers are internally interconnected via a single point-to-point FC link2520 and2522, and the interconnection may carry, in addition to X and Y fabric frames, internally generated and internally consumed management frames, or S-fabric frames. The internal point-to-point FC link withinstorage shelf2506 is referred to as the S₁fabric, and the internal point-to-point FC link within storage-shelf router2508 is referred as the S₂fabric. In essence, the internal point-to-point FC links carry FC frames for the X fabric, Y fabric, and internal management frames, but once X-fabric and Y-fabric frames enter the storage-shelf router through an endpoint storage-shelf router, they are considered S-fabric frames until they are consumed or exported back to the X fabric or Y fabric via an FC port of an endpoint storage-shelf router.
• FIGS.[0139]26A-E illustrate the data fields within an FC-frame header that are used for routing FC frames to particular storage-shelf routers or to remote entities via particular FC ports within the storage shelf that represents one embodiment of the present invention. The FC-frame header is discussed, above, with reference to FIG. 3. Of course, the FC header is designed for directing frames to FC nodes, rather than to disk drives interconnected with storage-shelf routers which together interface to an FC arbitrated loop or other FC fabric through a single FC node. Therefore, a mapping of FC-frame-header fields onto the storage-shelf router and SATA disk drive configuration within a storage shelf is needed for proper direction of FC frames. The three-byte D_ID field in an FC-frame header represents the node address of an FC node. In the case of FC arbitrated loops, the highest-order two bytes of the D_ID generally have the value “0,” for non-public loops, and the lowest-order byte contains an arbitrated-loop physical address (“AL_PA”) specifying one of 127 nodes. Generally, at least one node address is used for the disk-array controller, and another node address is reserved for a fabric arbitrated-loop (public loop) address. The three-byte S_ID field contains the node address of the node at which a frame was originated. In general, the S_ID field is the node address of a disk-array controller, although a storage-shelf may be interconnected directly to an FC fabric, in which case the S_ID may be a full 24-bit FC fabric address of any of a large number of remote entities that may access the storage-shelf.
• As shown in FIG. 26A, two[0140]reserved bits2602 within theDF_CTL field2604 of theFC frame header2606 are employed as a sort of direction indication, orcompass2608, for frames stored and transmitted within a storage shelf or, in other words, within the S fabric. Table 4, below, shows the encoding of this directional indicator:

  TABLE 4
  DF_CTL 19:18	Address Space
  00	Reserved
  01	X
  10	Y
  11	S

• Bit pattern “01” indicates that the frame entered the S-fabric as an X-fabric frame, bit pattern “10” indicates that the frame entered the S-fabric as a Y-fabric frame, and bit pattern “11” indicates that the frame is an S-fabric management frame. This directional indicator, or internal compass, represented by bits 18:19 of the DF_CTL field is needed because both S-fabric and external-fabric frames may be received by the storage-shelf router through a single FC port. As noted above, bits 18:19 of the DF_CTL field are collectively referred to as the “S bits.” The S bits are set upon receipt of an X-fabric or a Y-fabric frame by an endpoint storage-shelf router, and are cleared prior to export of an FC frame from an endpoint storage-shelf router back to the X fabric or the Y fabric. The R_CTL field of the FC frame header contains an indication of the type of FC frame, including indications of FCP_CMND, FCP_DATA, FCP_XFER_RDY, and FCP_RSP frames. This field is used by the routing logic to quickly determine how the frame is to be processed.[0141]
• FIG. 26B illustrates FC-frame-header fields involved with the routing of an FCP-CMND frame. The[0142]D_ID field2610 directs the FC frame to a particular FC node, but, as discussed above, a storage shelf, when operating in transparent mode, may contain a number of FC nodes, and when not operating in transparent mode, may contain a large number of data-storage devices to which FC frames all containing a single D_ID need to be dispersed. The routing logic of the storage-shelf router is essentially devoted to handling the various mappings between D_IDs, storage-shelves, storage-shelf routers, and, ultimately, disk drives.
• The routing logic cannot determine from the value of D_ID field, alone, whether or not the FC frame is directed to the storage-shelf router. In order to determine whether the D_ID directs an incoming FC_CMND frame to the storage-shelf router, the routing logic needs to consult an internal routing table[0143]2612 and several registers, discussed below, to determine whether the D_ID represents the address of a disk drive managed by the storage-shelf router. Thus, as shown in FIG. 26B, the D_ID field, as interpreted with respect to the internal routing table2612, specifies a particular storage-shelf router within astorage shelf2616 and a particular disk interconnected to the storage-shelf router2618. In addition, the routing logic consults additional internal tables2614, discussed below, to determine whether the source of the FC frame, specified by theS_ID field2611, is a remote entity currently logged in with the storage-shelf router, and whether the remote entity is identified as interconnected with the addressed disk drive. Thus, the S_ID field, as interpreted with respect to various internal tables2614, acts as anauthorization switch2620 that determines whether or not the command represented by the FC_CMND frame should be carried out.
• FIG. 26C illustrates FC-frame-header fields involved with the routing of an FCP-DATA frame. The D_ID and[0144]S_ID fields2610 and2611 and internal tables2612 and2614 are used, as with routing of FCP-CMND frames, to specify a particular storage-shelf router within astorage shelf2616 and a particular disk interconnected to the storage-shelf router, and to authorize2620 transfer of the data to a disk. However, because FCP_DATA frames may be part of multi-FCP_DATA-frame WRITE sequence, additional fields of the FC-frame header2606 are employed to direct the FCP_DATA frame within the storage-shelf router, once the routing logic has determined that the FC_DATA frame is directed to a disk local to the storage-shelf router. As shown in FIG. 26C, theRX_ID field2622 contains a value originally generated by the storage-shelf router, during processing of the FCP_CMND frame that specified the WRITE command associated with the FCP_DATA frame, that specifies acontext2624 for the WRITE command, in turn specifying avirtual queue2626 by which the data can be transferred from the FCP layer to the SATA-port layer via the GSMS. In addition, theparameter field2628 of the FC-frame header2606 contains a relative offset for the data, indicating theposition2630 of the data contained in the FCP_DATA frame within the total sequential length ofdata2632 transferred by the WRITE command. Thecontext2624 stores an expected relative offset for the next FCP_DATA frame, which can be used to check the FCP_DATA frame for proper sequencing. If the stored, expected relative offset does match the values of the parameter field, then the FCP_DATA frame has been received out-of-order, and error handling needs to be invoked.
• FIG. 26D illustrates FC-frame-header fields involved with the routing of an internally generated management frame. The S bits ([0145]2602 in FIG. 26A) indicate that the frame is a management frame. In the case of a management frame, the lowest-order byte of theD_ID field2610 contains a router number specifying a particular storage-shelf router within a storage shelf. The router number contained in the D_ID field is compared with a local-router number contained in aregister2634, to be discussed below, to determine whether the management frame is directed to the storage-shelf router, for example storage-shelf router2636, or whether the management frame is directed to another storage-shelf router within the storage shelf, accessible through the X-fabric-associatedFC port2638 or the Y-fabric-associatedFC port2640.
• Finally, FIG. 26E illustrates FC-frame-header fields involved with the routing of an received FCP_XFER_RDY and FCP_RSP frames. In the case of FCP_XFER_RDY and FCP_RSP frames, the routing logic immediately recognizes the frame as directed to a remote entity, typically a disk-array controller, by another storage-shelf router. Thus, the routing logic needs only to inspect the[0146]R_CTL field2642 of the FC-frame header to determine that the frame must be transmitted back to the X fabric or the Y fabric.
• FIG. 27 illustrates the seven main routing tables maintained within the storage-shelf router to facilitate routing of FC frames by the routing logic. These tables include the internal routing table (“IRT”)[0147]2702, X-fabric and Y-fabric external routing tables (“ERT_X”) and (“ERT_Y”)2704 and2706, respectively, X-fabric and Y-fabric initiator/target tables (“ITT_X”) and (“ITT_Y”)2708 and2710, and X-fabric and Y-fabric login pair tables (“LPT_X”) and (“LPT_Y”)2712 and2714, respectively. Each of these seven routing tables is associated with an index and a data register, such as index and data registers (“IRT_INDEX”) and (“IRT_DATA”)2716 and2718. The contents of the tables can be accessed by a CPU by writing a value indicating a particular field in the table into the index register, and reading the contents of the field from, or writing new contents for the field into, the data register. In addition, there are three registers “SFAR”2720, “XFAR”2722, and “YFAR”2724 that are used to store the router number and the high two bytes of the D_ID corresponding to the storage-shelf router address with respect to the X, and Y fabrics, respectively. This allows for more compact IRT, ERT_X and ERT_Y tables, which need only to store the low-order byte of the D_IDs.
• The IRT table[0148]2702 includes a row for each disk drive connected to the storage-shelf router or, in other words, for each local disk drive. The row includes the AL_PA assigned to the disk drive, contained in the low-order byte of the D_ID field of a frame directed to the disk drive, the LUN number for the disk drive, the range of logical block addresses contained within the disk drive, a CPU field indicating which of the two CPUs manages I/O directed the disk drive, and a valid bit indicating whether or not the row represents a valid entry in the table. The valid bit is convenient when less than the maximum possible number of disk drives is connected to the storage-shelf router.
• The ERT_X and ERT_Y tables[0149]2704 and2706 contain the lower byte of valid D_IDs that address disk drives not local to the storage-shelf router, but local to the storage shelf. These tables can be used to short-circuit needless internal FC frame forwarding, as discussed below.
• The X-fabric and Y-fabric ITT tables[0150]2708 and2710 include the full S_ID corresponding to remote FC originators currently logged in with the storage-shelf router and able to initiate FC exchanges with the storage-shelf router, and with disk drives interconnected to the storage-shelf router. The login-pair tables2712 and2714 are essentially sparse matrices with bit values turned on in cells corresponding to remote-originator and local-disk-drive pairs that are currently logged in for FCP exchanges. The login tables2712 and2714 thus provide indications of valid logins representing an ongoing interconnection between a remote entity, such as a disk-array controller, and a local disk drive interconnected to the storage-shelf router.
• Next, the routing logic that constitutes the routing layer of a storage-shelf router is described with reference to a series of detailed flow-control diagrams. FIG. 28 provides a simplified routing topology and routing-destination nomenclature used in the flow-control diagrams. FIGS.[0151]29-35 are a hierarchical series of flow-control diagrams describing the routing layer logic.
• As shown on FIG. 28, the[0152]routing layer2802 is concerned with forwarding incoming FC frames from theFC ports2804 and2806 either directly back to an FC port, to theFCP layer2810 for processing by FCP logic and firmware executing on a CPU, or relatively directly to the GSMS layer, in the case of data frames for which contexts have been established. The routing layer receives incoming FC frames frominput FIFOs2812 and2814 within the FC ports, designated “From_FP0” “From_FP1,” respectively. The routing layer may direct an FC frame back to an FC port, generally the port opposite from which the frame was received, by writing the FC frame to one of theFIFOs2816 and2818, designated “To_FP0” and “To_FP1,” respectively. The routing layer may forward an FCP_DATA frame relatively directly to the GSMS layer via a virtual queue, a process referred to as “To_GSMS,” and may forward an FC frame to theFCP layer2810 for processing, referred to as “To_FCP.” The designations “From_FP0,” “From_FP1,” “To_FP0,” “To_FP1,” “To_GSMS,” and “To_FCP are employed in the flow-control diagrams as shorthand notation for the processes of reading from, and writing to FIFOs, data transfer through the GSMS virtual queue mechanism, and state-machine-mediated transfer through a shared-memory interface to CPUs.
• FIG. 29 is the first, and highest level, flow-control diagram representing the routing layer logic. The routing layer logic is described as set of decisions made in order to direct an incoming FC frame to its appropriate destination. In a functioning storage router, the routing logic described with respect to FIGS.[0153]29-35 is invoked as an incoming FC frame is processed. The routing logic resides within state machines and logic circuits of a storage-shelf router. The storage-shelf router is designed to avoid, as much as possible, store-and-forward, data-copying types of internal data transfer, and is instead streamlined so that frames can be routed, using information in the frame headers, even as they are being input into the FIFOs of the FC ports. In other words, the routing logic may be invoked as soon as the frame header is available for reading from the FIFO, and the frame may be routed, and initial data contained in the frame forwarded to its destination, in parallel with reception of the remaining data by the FC port. The storage-shelf router includes arbitration logic to ensure fair handling of the two different input FIFOs of the two FC ports, so that FC frames incoming from both the X fabric and Y fabric are handled in timely fashion, and neither the X fabric nor the Y fabric experiences unnecessary FC-frame handling delays, or starvation. The routing logic is invoked by signals generated by FC ports indicating the availability of a newly arrived frame header in a FIFO.
• In[0154]step2902, the routing layer logic (“RLL”) reads the next incoming FC frame header from one of the input FIFOs of the FC ports, designated “From_FP0” and “From_FP1,” respectively. Instep2904, the routing layer logic determines whether or not the FC frame including the frame header is a class-3 FC frame. Only class-3 FC frames are supported by the described embodiment of the storage-shelf router. If the FC frame is not a class-3 FC frame, then the FC frame is directed to the FCP layer, To_FCP, for error processing, instep2906. Note that, in this and subsequent flow-control diagrams, a lower-case “e” associated with a flow arrow indicates that the flow represented by the flow arrow occurs in order to handle an error condition. If the FC frame is a class-3 FC frame, as determined instep2904, the RLL next determines, instep2908, whether the FC port from which the FC frame was received is an S-fabric endpoint, or, in other words, an X-fabric or Y-fabric node. A storage-shelf router can determine whether or not specific ports are endpoints with respect to the S fabric, or are, in other words, X-fabric or Y-fabric nodes from configurable settings. The FC-frame header contains the port address of the source port, as discussed above.
• If the source port of the FC frame is an S-fabric endpoint, indicating that the FC frame has been received from an entity external to the local S fabric, then the RLL determines, in[0155]step2910, whether any of the S bits are set within the DF_CTL field of FC frame header. If so, then an error has occurred, and the FC frame is directed to the FCP layer, To_FCP, for error processing instep2906. If not, then appropriate S bits are set to indicate whether the FC frame belongs to the X fabric, or X space, or to the Y fabric, or Y space instep2912. Note that one of the two FC ports corresponds to the X fabric, and other of the two FC ports corresponds to the Y fabric, regardless of the position of the storage-shelf router within the set of interconnected storage-shelf routers within a storage shelf. As noted above, the association between FC ports and the X and Y fabrics is configurable. Next, the RLL determines, instep2914, whether the S bits are set to indicate that the frame is an S-fabric frame. If so, then the sublogic “Management Destination” is invoked, instep2916, to determine the destination for the frame, after which the sublogic “Route To Destination” is called, instep2918, to actually route the FC frame to the destination determined instep2916. If the FC frame is not an S-fabric management frame, as determined instep2914, then, in step2920, the RLL determines whether or not the RLL is currently operating in transparent mode, described above as a mode in which each disk drive has its own FC node address. If the storage-shelf router is operating in transparent mode, then the sublogic “Transparent Destination” is called, instep2922, in order to determination the destination for the frame, and then the sublogic “Route To Destination” is called instep2918 to actually route the frame to its destination. Otherwise the sublogic “Destination” is called, instep2924, to determination the destination for the frame, after which it is routed to its destination via a call to the sublogic “Route To Destination” instep2918.
• FIG. 30 is a flow-control diagram representation of the sublogic “Management Destination,” called from[0156]step2916 of FIG. 29. Instep3002, the RLL determines whether the storage-shelf router number stored in the D_ID in the header of the FC frame is equal to that of the storage-shelf router. This determination can be made using the router number assigned to the storage-shelf router within the storage shelf, and stored in the SFAR register. If the router number contained in the D_ID matches the router number in the SFAR register, as determined instep3002, then a variable “destination” is set to the value “To_FCP” instep3004, indicating that the frame should be sent to the FCP layer. If the router numbers do not match, then, instep3006, the RLL determines whether the router number in the D_ID of the FC frame is greater than the storage-shelf router's router number. If the router number in the D_ID of the FC frame is greater than that of the storage-shelf router stored in the SFAR register, then control flows to step3008. Otherwise control flows to step3010. In bothsteps3008 and3010, the RRL determines if the frame has reached an S-fabric endpoint within the storage shelf. If so, then the management frame was either incorrectly addressed or mistakenly not fielded by the appropriate destination, and so, in both cases, the destination is set to “To_FCP,” instep3004, so that the frame will be processed by the CPU as an erroneously received frame. However, in bothsteps3008 and3010, if the current storage-shelf router is not an S-fabric endpoint, then the destination is set to “To_FP0,” instep3012, in the case that the router number in the D_ID is less than the current router's router number, and the destination is set to “To_FP1” instep3014, if the router number in the D_ID is greater than that of the current storage-shelf router. It should be noted again that the numeric identification of storage-routers within a storage shelf is ascending, with respect the X fabric, and descending, with respect to the Y fabric.
• FIG. 31 is a flow-control-diagram representation of the sublogic “Destination,” called from[0157]step2924 in FIG. 29. This sublogic determines the destination for an FC frame when the storage-shelf router is not operating in transparent mode or, in other words, when the storage-shelf router is mapping multiple disk drives to an AL_PA. Instep3102, the RLL determines if the frame is an XFER_RDY or RSP frame. These frames need to be sent back to the disk-array controller. If so, then, instep3102, the RLL determines whether the frame belongs to the X fabric. If the frame does belong to the X fabric, then the variable “destination” is set to the value “To_FP0,” instep3104, to direct the frame to the X FC port. If the frame is a Y-fabric frame, as determined instep3102, then the variable “destination” is set to “To_FP1,” instep3106, in order to direct the frame to the Y FC port. If the frame is not an FCP_XFER_RDY or FCP_RSP frame, as determined instep3102, then, instep3108, the RLL determines whether the frame is an FCP_CMND frame. If so, then the variable “destination” is set to “To_FCP,” instep3110, indicating that the frame is an FCP_CMND frame directed a LUN local to the storage-shelf router, and that the frame needs to be directed to the FCP layer for firmware processing in order to establish a context for the FCP command. If the frame is not an FCP_CMND frame, as determined instep3108, then, instep3112, the RLL determines whether or not the frame is an FCP_DATA frame. If the frame is not a data frame, then a variable “destination” is set to “To_FCP,” instep3114, to invoke error handling by which the firmware determines what type of frame has been received and how the frame should be handled. If the frame is an FCP_DATA frame, as determined instep3112, then, instep3116, the RLL determines whether or not the frame was sent by a responder or by an originator. If the frame was sent by an originator, then the variable “destination” is set “To_FCP,” instep3110, to direct the frame to FCP-layer processing. If a data frame was sent by a responder, then, instep3118, the RLL determines whether the frame was received initially from outside the S fabric or if the S-bit-encoded fabric indication within the frame header is inconsistent with the port opposite from which the frame was received. If either condition is true, then the frame has been received in error, and the variable “destination” is set to “To_FCP,” instep3114, to direct the frame to the CPU for error processing. Otherwise, control flows to step3103, for direction to either the X port or the Y port.
• FIG. 32 is a flow-control-diagram representation of the sublogic “Transparent Destination,” called from[0158]step2922 in FIG. 29. This sublogic determines destinations for FC frames when the storage-shelf router is operating in transparent mode, in which each disk drive has its own AL_PA. Instep3202, the RLL determines whether or not the high two bytes of the D_ID field of the header in the FC frame are equivalent to the contents of the XFAR or YFAR register corresponding to the source port in which the frame was received, and whether the low byte of the D_ID field contains an AL_PA contained in the IRT table indicating that the AL_PA has been assigned to a local disk drive. If so, then the FC frame was directed to the current storage-shelf router. Otherwise, the FC frame is directed to another storage shelf or storage-shelf router. In the case that the FC frame is directed to the current storage-shelf router, then, instep3204, the RLL determines whether the originator of the FC frame is a remote entity identified as an external FC originator currently capable of initiating FC exchanges with disk drives interconnected with the storage-shelf router, by checking to see if the S_ID corresponds to an S_ID contained in the appropriate IIT table, and, if the S_ID is found in the appropriate ITT table, the RLL further checks the appropriate LPT table to see if the remote entity associated with the S_ID contained in FC-frame header is currently logged in with respect to the disk to which the frame is directed. If the S_ID represents a remote entity currently logged in, and capable of undertaking FC exchanges with the disk drive, interconnected with the storage-shelf router, to which the frame is directed, as determined instep3204, then, instep3206, the variable “destination” is set to “To_FCP,” in order to direct the frame to the FCP layer for processing. If, by contrast, either the S_ID is not in the appropriate IIT table or the source and disk drive to which the FC frame is directed is not currently logged in, as indicated by the appropriate LPT table, then the variable “destination” is set to “To_FCP” instep3208 in order direct the frame to the FCP layer for error handling.
• If the D_ID field does not match the contents of the appropriate FAR registers, as determined in[0159]step3202, then, instep3210, the RLL determines whether or not the frame is an X-fabric frame. If so, then, instep3212, the RLL determines whether or not the frame is directed to another storage-shelf router within the storage shelf. If not, then the variable “destination” is set to “To_FP0” to return the frame to the external X fabric for forwarding to another storage shelf instep3214. If the ERT_X table contains an entry indicating that the destination of the frame is a disk drive attached to another storage-shelf router within the storage shelf, as determined instep3212, then, instep3216, the RLL determines whether or not the current storage-shelf router represents the Y-fabric endpoint. If so, then the frame was not correctly processed, and cannot be sent into the Y fabric, and therefore the variable “destination” is set to the value “To_FCP,” instep3208, so that the frame can be directed to the FCP layer for error handling. Otherwise, the variable destination is set to “To_FP1,” instep3218, to forward the frame on to subsequent storage-shelf routers within the storage shelf via the S fabric. If the received frame is not an X-fabric frame, as determined instep3210, then, instep3220, the RLL determines whether or not the received frame is a Y-fabric frame. If so, then the frame is processed symmetrically and equivalently to processing for X-fabric frames, beginning instep3222. Otherwise, the variable “destination” is set to “To_FCP,” instep3208, to direct the frame to the FCP layer for error handling.
• FIG. 33 is a flow-control-diagram representation of the sublogic “Route To Destination”-called from[0160]step2918 in FIG. 29. This sublogic directs received FC frames to the destinations determined in previously invoked logic. Instep3302, the RLL determines whether the value of the variable “destination” is “To_FP0” or “To_FP1.” If so, in the same step, the RLL determines whether the destination is associated with the port opposite the port on which the FC frame was received. If so, then, instep3304, the RLL determines whether the destination indicated by the contents of the variable “destination” is a queue associated with a port representing an S-fabric endpoint. If so, then instep3306, any S bits set within the DF_CTL field of the FC-frame header are cleared prior to transmitting the frame out of the local S fabric. Instep3308, the RLL determines to which of the X fabric or Y fabric the frame belongs, and queues to frame to the appropriate output queue insteps3310 or3312. If the contents of the variable “destination” either do not indicate the FP0 or FP1 ports, or the destination is not opposite from the port on which the FC frame was received, as determined instep3302, then, instep3314, the RLL determines whether or not the contents of the variable “destination” indicate that the frame should be directed to one of the FC ports. If the frame should be directed to one of the FC ports, then the frame is directed to the FCP layer instep3316, for error processing by the FCP layer. If the contents of the variable “destination” indicate that the frame is directed to the FCP layer, “To_FCP,” as determined by the RLL instep3318, then the frame is directed to the FCP layer instep3316. Otherwise, the RLL checks, instep3320, whether the R_CTL field of the FC-frame header indicates that the frame is an FCP frame. If not, then the frame is directed to the FCP layer instep3316, for error handling. Otherwise, instep3322, the RLL determines whether or not the frame is an FCP_CMND frame. If so, then the sublogic “Map Destination” is called, instep3324, after which the RLL determines whether or not the contents of the variable “destination” remain equal to “To_FCP” instep3326. If so, then the frame is directed to the FCP layer, instep3316. Otherwise, if the contents of the variable “destination” now indicate forwarding of the frame to one of the two FC ports and the FC port destination is the same FC port on which the frame was received, as determined instep3328, the frame is directed to the FCP layer, instep3316, for error handling. Otherwise, control flows to step3304, for queuing the frame to one of the two FCP ports. If the frame is not an FCP_CMND frame, as determined instep3322, then the sublogic “Other Routing” is called instep3330.
• FIG. 34 is a flow-control-diagram representation of the sublogic “Map Destination,” called in[0161]step3324. The RLL first determines, instep3402, whether LUN, LBA, or a combination of LUN and LBA mapping is currently being carried out by the storage-shelf router. If not, then the RLL determines, instep3404, whether the storage-shelf router is currently operating in transparent mode. If so, then the value of the variable “destination” is set to “To_FCP” instep3406. If the storage-shelf router is not operating in transparent mode, as determined instep3404, then the RLL determines, instep3408, whether the appropriate LPT table indicates that the source of the frame is logged in for exchanging data with the destination of the frame. If so, then the variable “destination” is set to “To_FCP” instep3406. Otherwise, the destination is also set to “To_FCP,” instep3406, in order to direct the frame to the CPU for error processing. If LUN, LBA, or a combination of LUN and LBA mapping is being carried out by the storage-shelf router, then the RLL determines, instep3410, whether the designated destination disk has an associated entry in the IRT table. If so, then control flows to step3404. Otherwise, instep3412, the RLL determines whether or not range checking has been disabled. If range checking is disabled, then, instep3414, the RLL determines if the frame was received on the FP0 port. If so, then the variable “destination” is set to “To_FP1” instep3416. Otherwise, the contents of the variable “destination” is set to “To_FP0” instep3418. If range checking is enabled, then, instep3420, the RLL determines whether the designated destination disk is accessible via the FP0 port. If so, then control flows to step3418. Otherwise, instep3422, the RLL determines whether the designated destination disk is accessible via the FC port FP1. If so, then controlflows step3416. Otherwise, the variable “destination” is set to “To_FCP” instep3406 for error handling purposes. In a final step, for frames mapped to one of the two FC ports in eithersteps3416 or3418, the RLL, instep3424, determines whether the port to which the frame is currently directed is an S-space endpoint. If so, then the value of the variable “destination” is set to “To_FCP” instep3406 in order to direct the frame to the FCP for error processing.
• FIG. 35 is a flow-control-diagram representation of the sublogic “Other Routing,” in[0162]step3330 of FIG. 33. Instep3502, the RLL determines whether the RX_ID field of the frame indicates that the current storage-shelf router, or a disk drive connected to it, is the FC responder for the frame. If so, then instep3504, the RLL determines whether or not the frame is an FCP_DATA frame. If so, then instep3506, the RLL determines whether or not there is a valid context for the frame. If so, then the frame is directed to the GSMS, “To_GSMS,” instep3508, for transfer of the data to an SATA port, as discussed above. Otherwise, the frame is directed, instep3510, to the FCP layer for error processing. If the RX_ID field of the FC-frame header does not indicate this storage-shelf router as the FC responder for the frame, as determined instep3502, then, instep3512, the RLL determines whether the storage-shelf router identified by the RX_ID field within the FC-frame header is accessible via the port opposite from the port on which the frame was received. If not, then the frame is queued to the queue “To_FCP” for error processing by the FCP layer. Otherwise in the case that the RX_ID identifies a storage-shelf router accessible from the port opposite from the port on which the frame was received, the RLL, instep3514, determines whether that port is an S-fabric endpoint. If so, then instep3516, the RLL removes any S-space bits set in the DF_CTL field of the FC frame header. Instep3518, the RLL determines to which of the X fabric and Y fabric the frame belongs and, in eitherstep3520 or3522, writes the frame to the FIFO of the port appropriate for the fabric to which the frame belongs.
SCSI Command/ATA Command Translation
• As discussed above, a the storage-shelf router that represents one embodiment of the present invention fields FCP_CMND frames, directed by the disk-array control to the storage-shelf router as if the FCP_CMND frames were directed to FC disk drives, and translates the SCSI commands within the FCP_CMND frames into one or more ATA commands than can transmitted to an SATA disk drive to carry out the SCSI command. Table 5, below, indicates the correspondence between SCSI commands received by the storage-shelf router and the ATA commands used to carry out the SCSI commands:[0163]

  TABLE 5
  	ATA Command to which
  SCSI Command	SCSI Command is Mapped
  TEST UNIT READY	CHECK POWER MODE
  FORMAT UNIT	PMA WRITE
  INQUIRY	IDENTIFY DEVICE
  MODE SELECT	SET FEATURES
  MODE SENSE	IDENTIFY DEVICE
  START UNIT	IDLE IMMEDIATE
  STOP UNIT	SLEEP
  SEND DIAGNOSTIC	EXECUTE DEVICE DIAGNOSTICS
  READ CAPACITY	IDENTIFY DEVICE
  READ	DMA READ
  WRITE	DMA WRITE
  SEEK	SEEK
  WRITE AND VERIFY	DMA WRITE/READ VERIFY SECTORS
  VERIFY	READ VERIFY SECTORS
  WRITE BUFFER	DOWNLOAD MIRCOCOPE
  WRITE SAME	DMA WRITE

Storage-Shelf-Router-Interface Tunneling
• FIG. 36A illustrates the conceptual interface levels within a disk array including a storage-shelf router that represents an embodiment of the present invention. In FIG. 36A, a disk-[0164]array controller3602 interfaces to a storage shelf via a storage-shelf router3604. The storage-shelf router, in turn, interfaces to the internal mass-storage devices, such as internal disk drives3606-3613, of a high-availability storage shelf. The storage-shelf router3606 presents a virtual interface to the disk-array controller3602. For example, the storage-shelf router may present an interface in which the disk-array controller3602 views the entire storage capacity of the storage shelf as four virtual logical units3614-3617. The storage-shelf router translates access operations directed by the disk-array controller3602 to the virtual logical units3614-3617 into corresponding access operations directed by the storage-shelf router3604 to the internal disk drives3606-3613. The disk-array controller3602 is generally unaware, when using the virtual interface, of the storage-shelf router3606 or the actual characteristics and capacities of the internal drives3606-3613, but instead is aware only of the virtual logical units3614-3617 of the virtual interface provided by the storage-shelf router3606.
• As discussed above, in previous subsections, the virtual interface and mechanisms supporting the virtual interface may be quite complex, and the virtual interface provides great benefit to the disk-array controller by hiding many complexities and details. However, there may be situations in which an external processing entity, such as the disk-[0165]array controller3602, may need to directly access the internal disk drives3606-3613. For example, the disk-array controller may need to download micro-code to internal disk drives, may need to test the disk drives in order to evaluate failed disk drives or to predict failure modes, or may need to carry out various testing protocols. The virtual interface commonly provided by storage-shelf routers, described in previous subsections, does not provide for such direct access by external processing entities to the internal disk drives.
• A storage router may be designed to provide a direct disk interface to external processing entities, such as disk-array controllers. However, the storage-shelf router that represents an embodiment of the present invention not only provides a virtual interface, as described in previous subsections, but also allows non-FC-compliant disk drives to be used in the storage shelf and accessed via the FC protocol. Although many device-specific commands, such as many ATA-disk-drive commands, have reasonably direct counterpart commands in the FCP protocol, there are a number of device-specific and vendor-specific commands that cannot be easily translated to the FCP protocol. Furthermore, the storage-shelf router cannot easily provide an interface to the many different device-specific and vendor-specific internal drive interfaces, since there are a great many different such commands sets and interfaces. In certain cases, external processing entities may need to direct specialized commands to internal disk drives that are not provided by disk-drive manufacturers to storage-shelf-router manufacturers. Thus, as shown in FIG. 36A, the disk-[0166]array controller3602 may be limited to accessing internal disk drives either through the virtual interface, such as logical units361403617, or through FC-disk-drive commands directed to virtual FC disk drives3620-3627 that the external entity assumes to reside in the storage shelf. In the case that the internal disk drives are ATA disk drives3606-3613, the disk-array controller has no way to direct disk-drive and vendor-specific commands to the internal disk drives.
• FIG. 36B illustrates a storage-shelf-router-interface tunneling method and system that represents an embodiment of the present invention. As shown in FIG. 36B, the storage-shelf writer provides a direct conduit, or tunnel, by which the disk-[0167]array controller3602 can direct commands to, and receive responses from, the internal disk drives, completely bypassing the virtual storage-shelf3614-3617 and virtual disk drives3620-3627 generally provided to the disk-array controller by the storage-shelf router. The storage-shelf router does not translate or interpret the commands sent and received through the storage-shelf-router-interface tunnel, but simply unpackages the device-specific and vendor-specific commands from FC frames and forwards the commands, unaltered and untranslated, to the internal disk drives, and packages responses from the internal disk drives into FC frames, unaltered and untranslated, and forwards the FC frames to the disk-array controller.
• In addition to storage-shelf-router-interface tunneling, the mechanism used to implement the storage-shelf-router-interface tunnel can also be used by the storage-shelf router to provide access by external processing entities, such as disk-array controllers, to various data structures and information stored in flash memories within the storage-shelf router, path controller cards, and internal disks which cannot be accessed by the normal data-access operations directed to the data stored within the internal mass-storage devices. FIG. 37 illustrates various data structures included in various components of a high-availability storage shelf that represents one embodiment of the present invention. In one embodiment of a high-availability storage shelf, a[0168]flash memory3702 within the storage-shelf router3704 includes a customer-specified behavior3706 data structure and a persistentlog data structure3708. These data structures may contain customer-specific parameters and error information logged by the storage-shelf router in a circular buffer. In the same embodiment, flash memories, such asflash memory3710, within path-controller cards, such aspath controller card3712, include a disk drive behavior data structure. Storage-shelf-router and path-controller-card firmware may also be stored in flash memory. The data structures and firmware may be written and read from the storage shelf via the mechanism employed for storage-shelf-router-interface tunneling. Finally, certain special sectors within each internal disk drive, such assectors3714 and3716 ininternal disk drive3618, may be initialized and maintained by the storage shelf router, but may be accessed, in part, by external processing entities, such as disk-array controllers, through mechanisms that implement the storage-shelf-router-interface tunnel.
• As discussed in previous subsections, an external processing entity, such as a disk-array controller, accesses the data stored within a storage shelf through a view of the storage shelf, or virtual storage-shelf interface, constructed and maintained by the storage-shelf router. That storage-shelf interface generally appears to the external processing entity as one or more fiber channel addresses, each associated with an FC disk or a SCSI device providing one or more logical units. The external processing entity interacts with SCSI-compliant mass-storage devices within the storage shelf using SCSI protocol commands packaged within the FCP protocol, as discussed above. However, the mass-storage devices actually resident within the storage shelf may not be FC/SCSI compliant, as discussed above.[0169]
• In order to provide direct access by an external processing entity, such as a disk-array controller, both to the native internal disk-drive interfaces, such as ATA disk-drive interfaces, and to the various data structures discussed with reference to FIGS.[0170]37-40, the storage-shelf router employs the FCP commands: (1) WRITE-BUFFER; and (2) READ-BUFFER. These commands are packaged within FCP_CMND frames sent by external processing entities to the storage-shelf router. The WRITE-BUFFER and READ-BUFFER are somewhat archaic, with respect to current FC-compliant devices, and are thus represent an available protocol vehicle for a tunneling mechanism.
• FIGS.[0171]38A-F show the FCP_CMND and FCP_CMND CDB field layouts for the WRITE_BUFFER and READ_BUFFER commands, as well as an example command header and response. FIG. 38A shows the fields within an FCP_CMND frame. The FCP_CMND fields include: (1) aLUN field3802 containing the address of the destination logical unit; (2) a task attributesfield3804; (3) task-management flags3806 that specify various reset and abort operations; (4) a command descriptor block (“CDB”)3808; and (5) a data length field that specifies the greatest number of bytes to be transferred. When used for storage-shelf-router-interface tunneling, the LUN field may contain an indication of the particular ATA disk drive to which a command is directed, although, in the case that a separate AL_PA is used for each internal disk drive, the LUN field may be 0, and the WRITE_BUFFER and READ_BUFFER commands directed to the disk drive via the FC frame header. The ATA-disk address obtained by an external processing entity via management tools through which the detailed configuration of a storage shelf may be accessed. No task-management flag may be set in the case of a WRITE_BUFFER or READ_BUFFER command.
• FIG. 38B shows the layout of a generic FCP_CMND CDB field. The CDB includes: (1) an operation-[0172]code field3810 that specifies the command represented by the CDB and the length of the CDB; (2) command-specific parameters3812 for the specified command; and (3) acontrol byte3814 that specifies check-condition handling and multiple-command linking. In the case of the WRITE_BUFFER and READ_BUFFER commands, the operation-code field contains the values “00111011b” and “00111100b,” respectively, and the control byte is “0,” in one embodiment of the present invention.
• FIG. 38C shows the CDB for a WRITE_BUFFER command. The CDB of a WRITE_BUFFER command includes the following command-specific parameters: (1) a[0173]mode field3815, with the value “01h,” indicating that a vendor specific mode is used, with the type of WRITE_BUFFER command specified by the following buffer ID field; (2) abuffer ID3816 that essentially specifies the operation requested via the WRITE_BUFFER command, where the requested operations may include: (a) “00h,” download and saving of storage-shelf-router and optional path-controller-card firmware; (b) “01h,” transfer and writing of a digital drive signature to special sectors within a disk drive; (c) “02h,” transfer of a customer-specified behavior data structure to storage-shelf-router flash memory; (d) “03h,” transfer and saving of a disk-drive behavior data structure to a path-controller flash memory; (e) “06h,” an ATA-pass-through (“APT”), or tunneled ATA command; and (f) “08h,” path-controller-card firmware. In essence, the buffer ID is a WRITE_BUFFER type field that specifies the type of requested operation represented by the WRITE_BUFFER command; (3) a buffer offsetfield3818 that is ignored; and (4) a transfer length flied3820 that species the command header and/or data payload size for the FCP_DATA frame to follow, with the transfer length either of a known, fixed length, in the case of data structures such as customer-specified-behavior and disk-drive-behavior data structures, or of variable length, in the case of APT commands and firmware downloads.
• FIG. 38D shows the layout of the command header for an ATA command sent in a subsequent FCP_DATA frame following an FCP_CMND WRITE_BUFFER frame as part of the APT, storage-router-interface tunneling mechanism. The command header includes: (1) a time-out[0174]value field3821 that specifies the timeout value for the ATA command to be used by the storage-shelf router; (2) a number of flags including: (a) aDCV flag3822 that whether or not the device control register within the device should be written with a value in the device-control field; (b) aPOF flag3824 that specifies removing power supplies from the disk drive; (c) aPUP field3826 that specifies that power supplies should be restored to the disk; (d) aHRST field3828 that specifies that a hard reset should be issued to the disk drive; (e) aPIO field3830 that specifies whether or not programmed I/O is used for command transfer; (f) adata flag3832 that specifies whether or not the command involves data transfer; and (g) an RNW flag that specifies whether the command is a read command or a write command; (3) apayload length3836 that indicates the number of bytes to be transferred (provided that the data flag is set); and (4) a number of ATA-command-specific fields with which an ATA command is specified3838.
• FIG. 38E shows the CDB for a READ_BUFFER command. The READ_BUFFER CDB is similar to the WRITE_BUFFER CDB, described above, with the exception that the operation-code field contains a different value, transfer-[0175]length field3840 specifies the length of data to be returned to the requesting external processing entity, and thebuffer ID field3842 contains one of the following READ_BUFFER type identifiers: (a) “02h,” indicating that a CSB is to be returned; (b) “03h,” indicating a DDB is to be returned; (c) “06h,” indicating an APT response header is to be returned; (d) “07h,” indicating APT response payload, such as data from a disk READ operation, is to be returned; (e) “08h,” indicating that path-controller firmware status is to be returned; (f) “09h,” indicating that persistent log data is to be returned; and (g) “0Ah,” indicating that console log data is to be returned.
• FIG. 38F shows the format for an ATA-command response returned by the storage-shelf router in an FCP_DATA frame following receipt of a READ_BUFFER command. The fields in the ATA-command response are ATA-specific, and are neither translated nor processed by the storage-shelf router. An ATA response is returned as a variable number of bytes in an FCP_DATA frame.[0176]
• Each different WRITE_BUFFER type, specified by the buffer ID field, employs a different format for the FCP_DATA payload that follows the FCP_CMND containing the WRITE_BUFFER CDB. A different data payload is used, for example, for firmware writes, data structure writes, and a digital disk signature write.[0177]
• FIGS.[0178]39A-41 are control-flow diagrams that describe the WRITE-BUFFER and READ-BUFFER protocols used for the APT, storage-shelf-router-interface tunneling that represents an embodiment of the present invention, and the actions of the external processing entity and the storage-shelf router in implementing the protocols. FIGS.39A-B illustrate the WRITE-BUFFER protocol, in the context of APT tunneling, when a transfer of data from an external processing entity to the storage shelf is involved. In FIGS.39A-B, and in FIGS.40A-41 to follow, the actual commands transferred appear, in order, in a vertical column at the center of the diagrams, with external-processing-entity actions illustrated on the left-hand portion of the diagram and storage-shelf router activities indicated on the right-hand portion of the diagram. First, instep3902, the external processing entity prepares a WRITE-BUFFER CDB within an FCP_CMND frame, and sends theFCP_CMND frame3904 to the storage-shelf router. Note that the FCP_CMND command is directed to a particular fiber channel arbitrated loop address recognized by the storage-shelf router. Various management tools allow an external processing entity and/or human user to determine the identities of all internal disk drives, storage-shelf routers, path-controller cards, and other internal storage-shelf components. This information allows an external processing entity to direct various commands via the WRITE-BUFFER protocol to the particular components within a storage shelf.
• The storage-shelf router, in[0179]step3906, receives the FCP command and returns anFCP_XFR_RDY frame3908 to the external processing entity to indicate that the storage-shelf router is prepared to receive data associated with the WRITE-BUFFER command. Upon receiving the FCP_XFR_RDY frame, instep3910, the external processing entity packages a command header into an FCP_DATA frame and sends theFCP_DATA frame3912 back to the storage-shelf router.
• When the storage-shelf router receives the packaged command header, in[0180]step3914, the storage-shelf router processes the command header, sends the included ATA command to the target ATA disk drive, and waits for an ATA-protocol X-RDY packet from the disk drive. In transferring the command to the ATA disk drive, the storage-shelf router allocates various data-transfer resources within the storage-shelf router, initializes the appropriate SATA port, locks the port, and otherwise prepares for the subsequent data transfer. The storage-shelf router then sends anotherFCP_XFR_RDY frame3916 to the external processing entity to indicate that the storage-shelf router is ready to receive data. Upon receiving the second FCP_XFR_RDY frame, instep3918, the external processing entity packages the data payload for the command into an FCP_DATA frame and sends theFCP_DATA frame3920 to the storage-shelf router. If all the data is transferred in the first FCP_DATA frame, then steps3922,2924,2926, and3928 are skipped, andstep3930 is carried out, as described below. Otherwise, the storage-shelf router, instep3922, receives the data in the FCP_DATA frame and employs the internal data-transfer resources to transfer the data to its target. The storage-shelf router returns anotherFCP_XFR_RDY frame3924, if necessary, to solicit an additional FCP_DATA frame. The controller packages additional data, instep3926, into a subsequent FCP_DATA frame and sends theFCP_DATA frame3928 to the storage-shelf router. The data transfer continues in this fashion until all data has been transferred.
• At this point, the storage-shelf router, in[0181]step3930, receives the final data frame, transfers the data, waits for the target ATA disk drive to return a status packet reporting the status for the executed command, deallocates the internal data-transfer resources, and sends anFCP_STATUS frame3932 to the external processing entity containing status for the WRITE_BUFFER command. Upon receiving the FCP_STATUS frame, instep3934, the external processing entity sends a READ-BUFFER command within anFCP_CMND frame3936 to the storage-shelf router in order to receive status information resulting from execution of the ATA command encapsulated within the original write buffer. The storage-shelf router then returns the status was already received by the storage shelf router and that was stored locally instep3930, and, instep3938, packages the status in a response format, as shown in FIG. 38F, within anFCP_DATA frame3940, which is returned to the external processing entity. The storage-shelf router then, instep3942, immediately sends anFCP_STATUS3944 frame to the external processing entity to indicate completion of the READ-BUFFER command. Instep3946, the external processing entity receives the FCP status frame and the response header, signifying the end of thedata transfer operation3948.
• FIGS.[0182]40A-B show a control-flow diagram, similar to the control-flow diagram of FIGS.39A-B, describing the WRITE-BUFFER and READ-BUFFER protocols with respect to an APT operation that transfers data from the storage shelf back to the external processing entity. Many of these steps are similar to the steps already described in FIGS.39A-B, and, in the interest of brevity and clarity, FIGS.40A-B are therefore described with less detail. The external processing entity sends anFCP_CMND frame4004 including a WRITE_BUFFER CDB to the storage-shelf router in order to direct the storage-shelf router to prepare to execute an operation that returns data to the external processing entity. The storage-shelf router responds with anFCP_XFR_RDY frame4008, allowing the external processing entity to package a command header within an FCP_DATA frame and send theFCP_DATA frame4012 back to the storage-shelf router. The storage-shelf router, instep4014, sets up a data transfer context for communication with the target ATA disk drive, sends an FCP_STATUS frame back to the external processing entity to report status for the WRITE_BUFFER command, but defers triggering execution of the operation until the storage-shelf router subsequently receives a READ-BUFFER command packaged within anFCP_CMND frame4020 back from an external processing entity. At that point, the storage-shelf router directs the target to execute the command, and transfers the data received from the target back to the external processing entity in one or more FCP_DATA4024-4025 frames. Finally, the storage-shelf router transmits anFCP_STATUS frame4028 to the external processing entity to signal completion of the command, returning status for the READ_BUFFER command but holding onto the status returned by the target ATA disk drive. The external processing entity then sends asecond READ_BUFFER command4030 to the storage router in order to retrieve the ATA status for the command, which the storage-shelf router returns in anFCP_DATA frame4032, and then returns, in athird FCP_STATUS frame4034 the status for the second READ_BUFFER command. Upon receiving the third FCP_STATUS frame in step4036, the external processing entity determines the original APT READ command to have successfully completed.
• FIG. 41 is a control-flow diagram for the WRITE-BUFFER and READ-BUFFER command protocols in an APT context associated with a control-type command that does not transfer data between the external processing entity and the storage shelf, other than the command specification and return status. The external processing entity sends a WRITE-BUFFER command in an[0183]FCP_CMND frame4104 to the storage-shelf router in order to initiate the control-type command. After receiving, in return, anFCP_XFR_RDY frame4108, the external processing entity transfers a command header in anFCP_DATA frame4112 to the storage-shelf router which processes, instep4114, the received command header, initiates execution of the command by the target, and waits for the command to complete. The storage-shelf router waits for completion of the requested operation and, in certain cases, for the ATA disk-drive target to return a completion status, and, in other cases, such as power down, the storage-router itself determines a command status. Finally, the storage-shelf router returns anFCP_STATUS frame4116 signaling completion of the WRITE-BUFFER phase of the operation. Upon receiving the FCP_STATUS frame, the external processing entity sends a READ-BUFFER command within anFCP_CMND frame4120 to the storage-shelf router in order for the storage-shelf router to return status information for the command included in the WRITE_BUFFER command in anFCP_DATA frame4124. The storage-shelf router finally returns anFCP_STATUS frame4128 for completion of the READ_BUFFER command, which the external processing entity receives in step4130 and determines execution of the control operation to be completed.
• In the above control-flow-based description of an embodiment of storage-shelf-router-interface tunneling, error-processing steps are omitted in the interest of clarity and brevity. Errors may be detected at various stages of execution, and may be handled in many different, implementation-specific ways.[0184]
• Although the present invention has been described in terms of a particular embodiment, it is not intended that the invention be limited to this embodiment. Modifications within the spirit of the invention will be apparent to those skilled in the art. For example, the above discussion describes the storage-shelf-interface tunneling technique that represents one embodiment of the present invention at a reasonably high common conceptual level. The details of the internal routing and transfer of data and the operations initiated by WRITE-BUFFER commands are described, in part, in previous subsections, and are otherwise implementation and device specific. An almost limitless number of different hardware, firmware, and software implementations are possible for the WRITE-BUFFER and READ-BUFFER tunneling mechanism that represents one embodiment of the present invention. In alternative embodiments, the WRITE-BUFFER and READ-BUFFER phases of such operations may be combined in order to minimize communications overhead. The types of operations specified within a WRITE-BUFFER command may be expanded, as necessary, in order to provide access to additional data structures and internal storage-shelf router components, and to allow tunneling to exploit alternative types of interfaces, including non-ATA disk-drive interfaces and interfaces to other types of mass-storage devices.[0185]
• The foregoing description, for purposes of explanation, used specific nomenclature to provide a thorough understanding of the invention. However, it will be apparent to one skilled in the art that the specific details are not required in order to practice the invention. In other instances, well-known circuits and devices are shown in block diagram form in order to avoid unnecessary distraction from the underlying invention. Thus, the foregoing descriptions of specific embodiments of the present invention are presented for purposes of illustration and description; they are not intended to be exhaustive or to limit the invention to the precise forms disclosed, obviously many modifications and variations are possible in view of the above teachings. The embodiments were chosen and described in order to best explain the principles of the invention and its practical applications and to thereby enable others skilled in the art to best utilize the invention and various embodiments with various modifications as are suited to the particular use contemplated. It is intended that the scope of the invention be defined by the following claims and their equivalents:[0186]

Claims (16)

1. A storage-shelf-router-interface tunnel comprising:

an external processing entity interconnected with a storage shelf by a first communications medium using commands of a first protocol;

mass-storage devices within the storage shelf that may be accessed by a mass-storage-device protocol through a second communications medium;

a storage-shelf router that provides a virtual interface to mass-storage devices within the storage shelf to external processing entities, translating first-protocol commands received from the first communications medium to mass-storage-device protocol commands sent to mass-storage devices through the second communications medium, and that also provides for direct access by the external processing entity to the mass-storage devices by unpackaging mass-storage-device-protocol commands from a particular type of first-protocol command frame received through the first communications medium and directing the unpackaged mass-storage-device-protocol commands through the second communications medium, without translation, to the mass-storage devices.

2. The storage-shelf-router-interface tunnel ofclaim 1 wherein the storage router receives responses to the unpackaged mass-storage-device-protocol commands directed to the mass-storage devices from the mass-storage devices through the second communications medium, packages the responses into a particular type of first-protocol frame, and returns the packaged responses through the first communications medium to the external processing entity.

3. The storage-shelf router ofclaim 1 further including providing, to the external processing entity, access to storage shelf data structures and storage-router firmware through a particular type of first-protocol command frame and response frame.

4. The storage-shelf router ofclaim 1 wherein the first communications medium is a fibre channel communications link and wherein the second communications medium is a serial or parallel communications medium interconnecting the storage-shelf router to a path controller card interfaced to an ATA disk drive.

5. The storage-shelf router ofclaim 1 wherein the first communications medium is a fibre channel communications link and wherein the second communications medium is a serial or parallel communications medium interconnecting the storage-shelf router to an ATA disk drive.

6. The storage-shelf router ofclaim 1 wherein the particular type of first-protocol command frame received through the first communications medium is one of a WRITE_BUFFER command or a READ_BUFFER command packaged within an FCP_CMND frame.

7. The storage-shelf router ofclaim 1 wherein, in addition to unpackaging mass-storage-device-protocol commands from a particular type of first-protocol command frame received through the first communications medium and directing the unpackaged mass-storage-device-protocol commands through the second communications medium, without translation, to the mass-storage devices, the storage-shelf router unpackages control directives from the particular type of first-protocol command frame received through the first communications medium and directs corresponding control operations to storage-shelf components, including mass-storage devices.

8. The storage-shelf router ofclaim 8 wherein the control directives include power-on, power-off, and reset control directives.

9. The storage-shelf router ofclaim 1 wherein, in addition to unpackaging mass-storage-device-protocol commands from a particular type of first-protocol command frame received through the first communications medium and directing the unpackaged mass-storage-device-protocol commands through the second communications medium, without translation, to the mass-storage devices, the storage-shelf router unpackages access directives from the particular type of first-protocol command frame received through the first communications medium and accesses specified storage-shelf components, including mass-storage devices.

10. The storage-shelf router ofclaim 8 wherein the access directives include:

writing all or a portion of the contents of a data structure stored in storage-shelf-router flash memory;

writing all or a portion of the contents of a data structure stored in path-controller-card flash memory;

writing all or a portion of the contents of a data structure stored in special sectors within a mass-storage device;

writing all or a portion of the storage-shelf firmware stored in storage-shelf-router flash memory;

writing all or a portion of the path-controller-card firmware stored in path-controller-card flash memory;

reading all or a portion of the contents of a data structure stored in storage-shelf-router flash memory;

reading all or a portion of the contents of a data structure stored in path-controller-card flash memory;

reading all or a portion of the contents of a data structure stored in special sectors within a mass-storage device;

reading all or a portion of the storage-shelf firmware stored in storage-shelf-router flash memory; and

reading all or a portion of the path-controller-card firmware stored in path-controller-card flash memory.

11. A method for providing direct access by an external processing entity to mass-storage devices within a storage shelf that are virtualized by a storage-shelf router within the storage shelf through a storage-shelf-router interface, the method comprising:

in a first communications protocol by which the external processing entity accesses virtual mass-storage devices through the storage-shelf-router interface, identifying a command frame that can be used to supplement the first-communications protocol with tunneling commands;

inserting a native mass-storage-device command into the identified tunneling command frame and passing the native mass-storage-device command through the tunneling command to the storage-shelf router;

receiving the tunneling command by the storage-router;

extracting the native mass-storage-device command from the tunneling command and forwarding the tunneling command to a mass-storage device within the storage shelf by a second communications medium.

12. The method ofclaim 11 further including:

receiving, by the storage-shelf router, a response to the forwarded native mass-storage-device command through the second communications medium, inserting the response into a first-protocol response frame, and returning the response to the external processing entity.

13. The method ofclaim 11 wherein the first communications medium is a fibre channel communications link and wherein the second communications medium is a serial or parallel communications medium interconnecting the storage-shelf router to a path controller card interfaced to an ATA disk drive.

14. The method ofclaim 11 wherein the identified first-protocol command frame is one of a WRITE_BUFFER command or a READ_BUFFER command packaged within an FCP_CMND frame.

15. A storage-shelf router comprising:

logic for receiving first-protocol commands through a first communications medium directed to virtual mass-storage devices;

logic for translating the received commands into native mass-storage-device commands;

logic for transmitting the translated, native mass storage commands to mass-storage devices through a second communications medium; and

logic for receiving native mass-storage-device commands packaged within selected first-protocol commands and forwarding the native mass-storage-device commands, untranslated, to mass-storage devices.

16. The storage-shelf router ofclaim 15 further including:

logic for receiving responses to translated mass-storage commands from mass-storage devices through the second communications medium, translating the responses into first-protocol responses, and forwarding the first-protocol responses to the first communications medium; and

logic for receiving responses to untranslated mass-storage commands from mass-storage devices through the second communications medium, packaging the responses, untranslated, into first-protocol responses, and forwarding the untranslated responses to untranslated mass-storage commands packaged within first-protocol responses to the first communications medium.

Images