US20050207134A1 - Cell board interconnection architecture - Google Patents

Abstract

According to at least one embodiment, a cell board interconnection architecture comprises an interconnection structure for interconnecting a plurality of cell boards, the interconnection structure configured to allow air to pass therethrough in a direction in which the cell boards couple therewith.

Description

RELATED APPLICATIONS
• This application claims priority to U.S. Provisional Patent Application Ser. No. 60/553,386 entitled “Cell Board Interconnection Architecture”, filed Mar. 16, 2004, the disclosure of which is hereby incorporated herein by reference.
BACKGROUND
• Cell boards are the building blocks for multi-processor computer systems. Cell boards may include such components as processor(s), memory, application specific integrated circuits (ASICs), and/or input/output (I/O) components. For instance, processor boards, memory boards, and I/O boards may be arranged in a system to form a desired configuration. Further, a single cell board may include a plurality of different types of components. For example, a cell board may include one or more processors, ASIC(s), memory subsystem, and in some cases a power subsystem.
• The most common method of interfacing cell boards in a computer system is to provide each cell board with a bus connector and to plug each cell board's bus connector into a matching socket or “slot” mounted to a backplane or motherboard. In general, a backplane provides a communicative interconnection for a plurality of cell boards that are coupled to the backplane. The backplane itself is typically a circuit card that contains sockets to which other cell boards (or “circuit cards”) can be connected. Backplanes may be either active or passive. Active backplanes typically contain, in addition to the sockets, logical circuitry that performs computing functions. In contrast, passive backplanes contain almost no computing circuitry. When multiple cell boards are connected to a single backplane, the resulting arrangement is often referred to as a cabinet (or “card cage”). In higher-end computer systems of this type, cell boards may be removed and replaced in the cabinet without powering down the backplane or any of the slots except the one corresponding to the cell board being replaced. Thus, such cabinets are often implemented for so-called high-availability systems. An example of cell boards and their arrangement in a cabinet is disclosed in U.S. Pat. No. 6,452,789 titled “PACKAGING ARCHITECTURE FOR32 PROCESSOR SERVER,” the disclosure of which is hereby incorporated herein by reference.
• Traditionally, backplanes are implemented as solid structures. For instance, backplanes are typically solid structures that are relatively densely populated with traces and cabling for interconnecting the cell boards coupled thereto. For example, traditional backplanes are generally arranged as a two-dimensional (“2D”) plane (e.g., commonly sized approximately 30 inches by 20 inches) to which cell boards couple, and the 2D plane of the backplane interconnects the cell boards coupled thereto. Traditional backplane designs may have several (e.g., 10) routing layers inside the board, wherein each routing layer comprises traces for interconnecting the cell boards that are coupled to the backplane.
• In high-end computing systems, a relatively large number of cell boards may be interconnected within cabinet(s). For example, the Superdome™ server available from Hewlett-Packard Company (“HP”) is available as a 16-way, 32-way, or 64-way server. The 16-way implementation may comprise four cell boards interconnected via a backplane within a cabinet, wherein each cell board may include four central processing units (“CPUs”) for a total of 16 CPUs, and the cell boards may comprise memory (e.g., dual in-line memory modules (“DIMMs”)) implemented thereon for a total of 64 gigabytes (“GB”) of memory available in the 16-way implementation. The 32-way implementation may comprise eight cell boards interconnected via a backplane within a cabinet, wherein each cell board may include four CPUs for a total of 32 CPUs, and the cell boards may comprise memory (e.g., DIMMs) implemented thereon for a total of 128 GB of memory available in the 32-way implementation. The 64-way implementation may comprise sixteen cell boards interconnected via a backplane within a cabinet, wherein each cell board may include four CPUs for a total of 64 CPUs, and the cell boards may comprise memory (e.g., DIMMs) implemented thereon for a total of 256 GB of memory available in the 32-way implementation. Further, as a greater number of cell boards is desired, multiple cabinets that each comprise multiple cell boards may be coupled together to form a high-end server.
• Competing design considerations are often encountered when developing such multi-processor computer systems. One design consideration commonly encountered involves cooling the components within the cabinet(s). Because of the heat generated by the components, some type of cooling system is typically included for cooling the components to prevent overheating and resulting improper or failed operation. Because traditional backplanes are solid structures, as described above, cooling systems typically generate air flow in a direction parallel to the backplane (e.g., bottom-to-top air flow). One technique for implementing bottom-to-top air flow is described in U.S. Pat. No. 6,452,789 titled “PACKAGING ARCHITECTURE FOR 32 PROCESSOR SERVER.” Traditional implementations of bottom-to-top air flow (or “front-to-top” air flow, as air may be ingested through the front of the cabinet and re-directed via blowers toward the top of the cabinet) is not optimal for several reasons. First, blowers are typically required for directing the air flow upward, which consume a relatively large amount of space in the cabinets (thus diminishing the space-efficiency of the architecture). Further, as the air moves upward through the cabinet, the air is heated by each cell board that it encounters, thus diminishing the affect of the air in cooling the upper cell board(s). To ensure proper cooling of the upper cell boards, increased air flow is needed, which means that the size of the blowers implemented for generating such increased air flow is undesirably large (and may be undesirably noisy in some architectures).
• Another design consideration often encountered in multi-processor computer systems is a desire for an architecture that enables cell boards to be accessed for service (e.g., by a technician). For instance, a cell board may be removable (e.g., hot swappable) from a cabinet for replacing or repairing the cell board. Service access has traditionally been in a direction orthogonal to the system's backplane. For instance, a cell board generally connects orthogonally to a backplane, and such cell board may be connected or removed from the front of a cabinet by moving the cell board in a direction orthogonal to the backplane. Thus, the service access and the air flow are orthogonal to each other in traditional multi-processor computer systems.
SUMMARY
• According to at least one embodiment, a cell board interconnection architecture comprises an interconnection structure for interconnecting a plurality of cell boards, the interconnection structure configured to allow air to pass therethrough in a direction in which the cell boards couple therewith.
BRIEF DESCRIPTION OF THE DRAWINGS
• FIG. 1 shows an example of one embodiment of a cell board interconnection architecture;
• FIG. 2A shows an example configuration of a cell board that may be used in a first example embodiment of a cell board interconnection architecture;
• FIG. 2B shows another example configuration of a cell board that may be used in the first example embodiment of a cell board interconnection architecture;
• FIG. 3A shows an example implementation of an interconnection card that may be used in the first example embodiment of a cell board interconnection architecture;
• FIG. 3B shows an example implementation of a switch card that may be used in the first example embodiment of a cell board interconnection architecture;
• FIG. 4 shows an example unit that includes cell boards ofFIG. 2B interconnected with the interconnection card ofFIG. 3A and switch card ofFIG. 3B in accordance with the first example embodiment of a cell board interconnection architecture;
• FIGS. 5A-5B show a plurality of the units ofFIG. 4 interconnected to form a cabinet in accordance with the first example embodiment of a cell board interconnection architecture;
• FIG. 6 shows an example of a plurality of interconnection cards ofFIG. 3A being interconnected in accordance with the first example embodiment of a cell board interconnection architecture;
• FIG. 7 shows an example configuration of a cell board that may be used in a second example embodiment of a cell board interconnection architecture;
• FIGS. 8A-8B show an example implementation of an interconnection structure that may be used in the second example embodiment of a cell board interconnection architecture;
• FIG. 9 shows an example implementation of a switch card that may be used in the second example embodiment of a cell board interconnection architecture;
• FIG. 10A shows a cell board ofFIG. 7 coupled to the interconnection structure ofFIGS. 8A-8B in accordance with the second example embodiment of a cell board interconnection architecture;
• FIG. 10B shows an example unit that is formed by combining a plurality of the cell boards ofFIG. 7 interconnected via a plurality of the interconnection structures ofFIGS. 8A-8B and a plurality of the switch cards ofFIG. 9 in accordance with the second example embodiment of a cell board interconnection architecture;
• FIG. 10C shows the backside of the example unit ofFIG. 10B;
• FIG. 10D shows the example unit ofFIG. 10B arranged within a cabinet;
• FIG. 11 shows a third example embodiment of a cell board interconnection architecture in which a first set of cell boards are coupled to a first interconnection structure (ofFIGS. 8A-8B) that is coupled to a first side of switch cards (ofFIG. 9) and a second set of cell boards are coupled to a second interconnection structure (ofFIGS. 8A-8B) that is coupled to an opposite side of the switch cards;
• FIGS. 12A-12B show an example implementation of an interconnection structure that may be used in a fourth example embodiment of a cell board interconnection architecture;
• FIG. 13 shows an example configuration of a cell board that may be used in the fourth example embodiment of a cell board interconnection architecture;
• FIGS. 14A-14C show an example unit that is formed by combining a plurality of the cell boards ofFIG. 13 interconnected via an interconnection structure ofFIGS. 12A-12B in accordance with the fourth example embodiment of a cell board interconnection architecture;
• FIG. 15 shows an example unit that is formed by interconnecting a plurality of the units ofFIGS. 14A-14C via switch cards in accordance with the fourth example embodiment of a cell board interconnection architecture;
• FIG. 16 shows an example cabinet that is formed by interconnecting a plurality of the units ofFIG. 15 in accordance with the fourth example embodiment of a cell board interconnection architecture;
• FIG. 17 shows an example configuration of a cell board that may be used in a fifth example embodiment of a cell board interconnection architecture;
• FIGS. 18A-18B show an example implementation of an interconnection structure that may be used in the fifth example embodiment of a cell board interconnection architecture;
• FIGS. 19A-19C show an example implementation of a switch card that may be used in the fifth example embodiment of a cell board interconnection architecture;
• FIG. 20 shows an example unit that is formed by combining a plurality of the cell boards ofFIG. 17 interconnected via a plurality of the interconnection structures ofFIGS. 18A-18B and a plurality of the switch cards ofFIGS. 19A-19C in accordance with the fifth example embodiment of a cell board interconnection architecture;
• FIG. 21A shows an example 3D interconnection architecture in accordance with certain embodiments;
• FIG. 21B shows another example 3D interconnection architecture in accordance with certain embodiments;
• FIG. 21C shows another example 3D interconnection architecture in accordance with certain embodiments;
• FIG. 22 shows an example of utilizing the example architecture ofFIG. 21 A for interconnecting a plurality of cell boards;
• FIG. 23 shows an example of utilizing the example architecture ofFIG. 21B for interconnecting a plurality of cell boards;
• FIG. 24 shows an example of utilizing the example architecture ofFIG. 21C for interconnecting a plurality of cell boards;
• FIG. 25 shows an example cabinet having a plurality of cell boards communicatively interconnected with 3D interconnection architectures wherein each cell board is communicatively coupled to a plurality of different switch cards; and
• FIG. 26 shows another example cabinet having a plurality of cell boards communicatively interconnected with 3D interconnection architectures wherein each cell board is communicatively coupled to a plurality of different switch cards.
DETAILED DESCRIPTION
• Various embodiments of a cell board interconnection architecture are now described with reference to the above figures, wherein like reference numerals represent like parts throughout the several views. As described further below, such embodiments provide an interconnection architecture for interconnecting a plurality of cell boards. As opposed to the planar interconnection structure of traditional backplanes, certain embodiments described herein provide a three-dimensional (“3D”) interconnection structure or interconnection “volume.” This advantageously allows for greater routing opportunity than the 2D backplanes traditionally used for interconnecting cell boards. For example, in certain embodiments the interconnection structure for communicatively interconnecting a plurality of cell boards has a first plane for routing information in at least a first dimension. The interconnection structure further has a second plane that is orthogonal to the first plane for routing information in at least a second dimension that is different from the first dimension. For example, in certain embodiments, a first plane is defined by porous structure or a partial backplane or partial midplane, and a second plane is defined by one or more switches that are coupled to the first plane. In some embodiments the first plane is merely a pass-through plane for passing information to the second plane (e.g., switches). For instance, the first plane may pass information along one dimension from a cell board to a second plane (e.g., switch), and the second plane may pass the information along another dimension to another cell board. Such a 3D interconnection structure may advantageously provide much routing opportunity without sacrificing efficiency and/or compactness of the structure. Various examples of such 3D interconnection structures are described further below.
• As also described below, certain embodiments provide an interconnection structure that advantageously enables air to flow through such structure. In some embodiments, for instance, front-to-back air flow may be used for cooling the components of the cell boards, whereby the interconnection structure does not prohibit such front-to-back air flow. Thus, a mechanism, such as a fan or blower, may be implemented to generate a flow of air directed toward the interconnection structure (e.g., front-to-back air flow), and the interconnection structure permits the generated air flow to pass through it. Service access may also be front-to-back, and thus the air flow and service access may be parallel to each other.
• In certain embodiments, an interconnection structure is formed by a plurality of interconnection cards, and an architecture is provided in which each of a plurality of cell boards is coupled to multiple ones of the interconnection cards. In some embodiments, the cell boards and interconnection cards may be arranged in a grid (or matrix)-like manner with periodic apertures available through such grid for air to flow through for cooling the cell boards' components. Other features of embodiments of a cell board interconnection architecture are described further below.
• In designing a cell board cabinet architecture, various conflicting ergonomic considerations are encountered. For instance, it is generally desirable for the cabinet architecture to provide at least the following features: 1) front access to cell boards (e.g., for ease of access to the cell boards for service), 2) appropriate air flow for cooling the cell boards, and 3) optimum utilization of space by providing a densely populated arrangement of cell boards in a space-efficient architecture. Front access to cell boards is becoming a feature commonly desired in the industry. Such front access to cell boards enables cabinets to be arranged side-by-side, thus allowing for a space-efficient, compact arrangement of the cabinets, while also allowing the cell boards to be easily serviced by a technician by accessing the cell boards from the front (e.g., by coupling and decoupling the cell boards from the front of the cabinet).
• Air flow is a problematic design consideration in traditional architectures. Industry standards are developing that dictate that air flow should be front-to-back or front-to-top. For example, standards of the American Society for Heating and Refrigeration Air Conditioning Engineering are emerging that dictate that computers are to have front-to-back or front-to-top cooling. These standards are emerging in an attempt to provide a common air flow for computers so that they can be arranged in a manner such that the computers do not ingest each others' exhaust. That is, by specifying where the exhaust (exiting air flow) is to be on computers, users can decide on an arrangement of their computers such that they do not ingest each others' exhaust. For instance, with front-to-back or front-to-top air flow, computers (or cabinets) may be arranged side-by-side without one computer ingesting the exhaust of another computer. Front-to-back air flow has not been an option in traditional architectures because, as described above, a solid backplane is typically implemented at the back of the cabinet for interconnecting the cell boards, which prevents the flow of air through the back of the cabinet.
• Embodiments provided herein enable an architecture in which a plurality of cell boards are interconnected without requiring a solid backplane. Rather, in certain embodiments, a porous backplane is implemented such that front-to-back air flow may be utilized within the architecture. Various architectures are provided that enable interconnection of a plurality of cell boards such that the interconnection does not prohibit (e.g., is transparent to) front-to-back air flow through the architecture. More specifically, in certain embodiments an interconnection structure is provided for interconnecting a plurality of cell boards, wherein air flow is generated in a direction toward the interconnection structure and is permitted to pass through the interconnection structure.
• Certain embodiments provide an architecture in which the cell boards and interconnection structure are arranged such that they each provide the least resistance to front-to-back air flow. For instance, they are arranged such that they have the smallest amount of surface area facing the front of the architecture to minimize the amount of surface area that produces resistance to front-to-back air flow. For example, traditional backplanes are oriented such that they have a large surface exposed to the front of the architecture, wherein cell boards connect into connectors arranged on the front-facing surface of the backplane. Generally, the width and height of a backplane provides a plane having a much larger surface area than the plane formed by the thickness and height (or the plane formed by the thickness and width). Thus, if a traditional backplane were rotated by 90 degrees and enabled the cell boards to connect to it along the plane formed by its thickness and height (or its thickness and width), the backplane would present much less resistance to front-to-back air flow because it would have a smaller surface area facing the front of the architecture.
• In certain embodiments, a plurality of cell boards are arranged in a first orientation and a plurality of interconnect cards are arranged in a second orientation that is orthogonal to the orientation of the cell boards, and each cell board couples to multiple ones of the plurality of interconnect cards. Such arrangement may be implemented to provide a 3D interconnection architecture that allows for greater routing opportunity for the total size of the architecture. Again, the cell boards and the interconnect cards may each be arranged to allow for front-to-back air flow. For instance, the plane of each cell board's surface having the smallest surface area for blocking front-to-back air flow (e.g., typically the plane formed by the cell board's thickness and height or its thickness and width) is arranged facing the front of the cabinet. Likewise, the plane of each interconnect card's surface having the smallest surface area for blocking front-to-back air flow (e.g., typically the plane formed by the interconnect card's thickness and height or its thickness and width) is arranged facing the front of the cabinet. As described further below, certain embodiments also allow for access to the cell boards (e.g., for servicing, such as removing and/or replacing the cell boards) via the front of the cabinet. Thus, air flow and access to the cell boards may be performed in a common direction (i.e., from the front of the cabinet) in certain embodiments.
• As described further below, certain embodiments implement some of the routing responsibility that is traditionally included on backplanes to other structures (e.g., to the cell boards and/or to switch cards), thus enabling the overall size of an interconnection structure to be reduced to allow for porous areas through which air can flow through the interconnection structure and/or enabling short routing distances of signals for improving signal integrity. For instance, in some embodiments, routing of information along one dimension (e.g., horizontal routing) is provided by the interconnection structure, and routing of information along another dimension (e.g., vertical routing) is provided by switch cards coupled to the interconnection structure. In other embodiments, routing of information along one dimension (e.g., vertical routing) is provided by the interconnection structure, and routing of information along another dimension (e.g., horizontal routing) is provided by the cell boards coupled to the interconnection structure. In still another example embodiment, routing of information along one dimension (e.g., vertical routing) is provided by switch cards coupled to the interconnection structure, and routing of information along another dimension (e.g., horizontal routing) is provided by the cell boards coupled to the interconnection structure.
• Further, certain embodiments provide an architecture that is modular. That is, the architecture can be readily expanded by combining separate units together. For instance, a “unit” (which may be formed via one or more interconnected cell boards) may comprise 4 processors, and to create a mid-range server that has 8 processors two of the units may be coupled (e.g., stacked) together. A 16-way or 32-way server may be similarly created by continuing to add additional units onto the overall structure. Thus, the architecture enables a manufacturer to readily utilize the architecture in its development of larger-scale systems, rather than requiring a separate architecture for each system. Accordingly, time, effort, and cost associated with producing larger-scale systems may be reduced.
• FIG. 1 shows an example of one embodiment of a 3D cell board interconnection architecture. As shown,architecture100 comprises a plurality ofcell boards102A,102B,102C, and102D that are communicatively interconnected via an interconnection structure that comprises a plurality ofinterconnection boards101A,101B,101C, and101D. In this example,cell boards102A-102D are arranged horizontally being parallel with the plane formed by the X and Z axes shown. Theinterconnection boards101A-101D are arranged orthogonal tocell boards102A-102D. That is,interconnection boards101A-101D are arranged vertically being parallel with the plane formed by the Y and Z axes shown. It should be recognized that this embodiment enables front-to-back air flow (along the Z axis), as shown by the arrows inFIG. 1. That is,architecture100 provides a porous interconnection structure, rather than a solid backplane.
• As mentioned above, in certain embodiments the horizontal routing of information may be performed along thecell boards102A-102D, and vertical routing may be performed along theinterconnection cards101A-101D. Thus, the amount of routing provided by the interconnection cards may be less than traditionally provided by a backplane, thereby enabling reduction in the size and amount of complexity required on the interconnection cards. For example, a cell board, such ascell board102A, may comprise a plurality of processors and other components, such as memory, ASICs, etc., and such horizontal routing between components on a cell board may be performed, in certain embodiments, by the cell board itself. The vertical routing from one cell board to another cell board may be performed by an interconnection card.
• WhileFIG. 1 shows one example embodiment, various other architectures may be implemented to enable front-to-back air flow and other desirable features, such as front access, compact design, etc. For instance, one embodiment is described further below in conjunction withFIGS. 2A-6, another embodiment is described further below in conjunction withFIGS. 7-10D, another embodiment is described further below in conjunction withFIG. 11, another embodiment is described further below in conjunction withFIGS. 12A-16, and another embodiment is described further below in conjunction withFIGS. 17-20. Further example embodiments are described below in conjunction withFIGS. 21A-26. It should be recognized that while the example embodiments are described independently below, various features of each embodiment may be implemented as described in a different embodiment (e.g., a cell board or interconnection structure of one embodiment may be implemented in another embodiment). That is, various features described with each embodiment may be interchanged to result in many other embodiments. Further, the scope of the present invention is not intended to be limited to the example embodiments shown and described herein, but rather the embodiments are intended solely as examples that render the disclosure enabling for many other implementations of the invention defined by the claims appended hereto.
• Turning toFIGS. 2A-6, an example embodiment of a cell board interconnection architecture is shown.FIG. 2A shows anexample cell board201 that comprisescomponents203A,203B,203C,204A,204B,205, and206 implemented thereon. More specifically, in this exampleconfiguration cell board201 comprisesprocessor206 and memory (e.g., DIMM)203A-203C. While oneprocessor206 is shown in this example, a plurality of such processors may be included oncell board201 in other configurations.Cell board201 also includesheat sinks205 in this example.Cell board201 further includesASICs204A-204B (which are shown as being implemented with heat sinks thereon).Such ASICs204A-204B may, for example, include controller chips for managing communications between components oncell board201.Cell board201 further includesconnectors202A,202B,202C, and202D, which in this example configuration are well-known orthogonal connectors, such as the “X-Vector HS High Speed Midplane for Cross-Connection” connector available from Japan Aviation Electronics Industry, Limited (“JAE”).
• FIG. 2B shows an alternativeexample cell board221 that may be implemented, which comprisescomponents223A,223B,223C,224A,224B,225, and226 implemented thereon. More specifically, in this exampleconfiguration cell board221 comprisesprocessor226 and memory (e.g., DIMM)223A-223C. While oneprocessor226 is shown in this example, a plurality of such processors may be included oncell board221 in other configurations.Cell board221 also includesheat sinks225 in this example.Cell board221 further includesASICs224A-224B (which are shown as being implemented with heat sinks thereon).Such ASICs224A-224B may, for example, include controller chips for managing communications between components oncell board221.
• Thus,cell board221 comprises the same components as described above withcell board201 ofFIG. 2, but such components are arranged differently.Cell board221 ofFIG. 2B also comprisesconnectors222A,222B,222C, and222D, which correspond toconnectors202A,202B,202C, and202D ofcell board201 described above withFIG. 2A. The components are arranged differently in the example configuration ofFIG. 2A than their arrangement on theexample cell board201 ofFIG. 2A, but it should be recognized that either arrangement of components permits front-to-back air flow in the manner described more fully below. For instance, in the example implementations ofFIGS. 2A and 2B, each component is arranged such that it provides the least amount of surface area in the path of the front-to-back air flow (e.g., in the path of air flow directed toward the interconnection structure described below). Thus, the components are arranged to minimize the amount of resistance that they provide to front-to-back air flow.
• FIG. 3A shows an example implementation of aninterconnection card301.Interconnection card301 comprisesconnectors302A-302H that are each capable of coupling with a connector of a cell board, such as one ofconnectors202A-202D ofcell board201 ofFIG. 2A or one ofconnectors222A-222D ofcircuit card221 ofFIG. 2B.Interconnection card301 also comprisesconnectors303A-303F that enable interconnection with a plurality of other interconnection cards301 (not shown) within a cabinet. Thus,connectors303A-303F are fabric connectors for a cabinet, as described below withFIG. 6.Interconnection card301 also comprisesconnector304 for coupling to a switch, such asswitch card351 ofFIG. 3B.
• In certainimplementations interconnection cards301 may be fixed within a unit or cabinet, and cell boards (such as those ofFIGS. 2A and 2B) and switch cards (such as that ofFIG. 3B described below) may be removably coupled thereto. Thus, for instance, cell boards and/or switch cards may be removed for servicing/repair. In this example embodiment,interconnection card301 is responsible for performing one-dimensional (“1D”) routing. More particularly, interconnection card301 (andswitch card351 ofFIG. 3B) performs vertical (e.g., along the Y axis ofFIG. 1) routing of information (e.g., routing of information from one of itsconnectors302A-302H to another ofsuch connectors302A-302H and/or routing information from one ofsuch interconnection cards301 to another interconnection card as described withFIG. 6 below). The cell boards are implemented to include the capability of performing horizontal routing (e.g., routing along the X and Z axes ofFIG. 1). Thus, in this example embodiment, 3D routing is achieved, but 1D is performed by the interconnection cards (and switch cards) and 2D is performed by the cell boards, rather than being limited to 2D routing that is performed entirely by an interconnection structure (such as with traditional backplanes).
• FIG. 3B shows anexample switch card351, which comprisesconnector352 for coupling withconnector304 ofinterconnection card301 ofFIG. 3A.Switch card351 also comprisescomponents353A and353B, which are ASICs or “cross-bar” chips (shown with heat sinks implemented thereon) for managing switching between the various cell boards coupled to interconnection card301 (i.e., for managing vertical routing within a cabinet). And,switch card351 comprises cabinet-to-cabinet fabric connectors354A-354H to enable a plurality of cabinets to be interconnected.Switch card351 controls the communication between the cell boards coupled tointerconnection card301. That is,switch card351 arbitrates the routing of information between the cell boards. Whileinterconnection card301 andswitch card351 are shown as separate cards in this example, which may improve the serviceability of the architecture, in alternative implementations the functionality of those two cards may be implemented as a single card.
• Turning toFIG. 4, anexample unit400 is shown. In thisexample architecture400, a plurality ofcell boards221 ofFIG. 2B are implemented, shown ascell boards221A-221H. As shown, each cell board is coupled to a plurality ofinterconnection boards301 ofFIG. 3A, shown asinterconnection boards301A-301D. For instance, as can be seen forcircuit card221A, itsfirst connector222A, is coupled to afirst interconnection board301A; itssecond connector222B, is coupled to asecond interconnection board301B; itsthird connector222C, is coupled to athird interconnection board301C; and itsfourth connector222D, is coupled to afourth interconnection board301D. Thus, each cell board is coupled to a plurality ofdifferent interconnection boards301A-301D. Also, aswitch card351 of FIG.3B is coupled to each interconnection board, wherein such switch cards are shown asswitch cards351A-351D. Alternating current (“AC”) to direct current (“DC”) power supplies (“front-end power supplies”)401 are also included. Such AC toDC power supplies401 may, for example, convert208 AC to48 DC. Of course, any other desired power conversion may be performed in alternative implementations.
• Thus, the example unit ofFIG. 4 comprises a plurality of cell boards (8 in this implementation) that are communicatively interconnected. Further, it should be recognized that the above architecture enables front-to-back air flow, such as indicated by the arrows shown inFIG. 4. Thus, a mechanism (not shown), such as a fan or blower, may be implemented in the example unit ofFIG. 4 to generate a flow of air directed toward the interconnection structure (e.g., front-to-back air flow), and the interconnection structure (e.g.,interconnection boards301A-301D andswitch cards351A-351D) permits the generated air flow to pass through it. It should also be recognized that the architecture ofFIG. 4 provides a dense arrangement of cell boards, thus providing a space-efficient architecture, while also allowing the cell boards to be accessed from the front of the architecture (which eases servicing the cell boards).
• Additionally, theexample architecture400 ofFIG. 4 is readily expandable. For instance, as shown inFIGS. 5A-5B, a plurality of the units may be interconnected (e.g., in a stacked arrangement) to form a larger overall system.FIGS. 5A-5B show an example in which 4 units ofFIG. 4, shown asunits400A-400D, are interconnected to formcabinet500 comprising a total of 32 cell boards.FIG. 5A shows an isometric view of the example arrangement from the front showing the front, right, and top sides thereof.FIG. 5B shows an isometric view of the example arrangement from the back showing the back, left, and top sides thereof. Theunits400A-400D are interconnected, thus enabling all of the cell boards ofsystem500 to be communicatively interconnected.
• More particularly, as described above, the interconnection cards ofFIG. 3A enable interconnection of a plurality of units within a cabinet (viaconnectors303A-303F shown inFIG. 3A).FIG. 6 shows an example of a plurality of interconnection cards being interconnected (e.g., as in the example cabinet ofFIG. 5B). More specifically,interconnection card301₁havingswitch card351₁coupled thereto is implemented within afirst unit400A;interconnection card301₂havingswitch card351₂coupled thereto is implemented within asecond unit400B;interconnection card301₃havingswitch card351₃coupled thereto is implemented within athird unit400C; andinterconnection card301₄havingswitch card351₄coupled thereto is implemented within afourth unit400D. Each interconnection card is communicatively coupled to each of the other interconnection cards. For instance, in this example, fiber optic cables are used to couple the interconnection cards (of course, other coupling techniques, such as copper wires or flex connectors may be used in alternative configurations). For example,interconnection card301₁has a fiber optic cable coupling from itsconnector303D (seeFIG. 3A) toconnector303D ofinterconnection card301₂;interconnection card301₁has a fiber optic cable coupling from itsconnector303E (seeFIG. 3A) toconnector303E ofinterconnection card301₃; andinterconnection card301₁has a fiber optic cable coupling from itsconnector303F (seeFIG. 3A) toconnector303F ofinterconnection card301₄. The other interconnection cards are likewise coupled to each of the interconnection cards in a column of the cabinet's architecture in this example. The interconnection betweencards301₃and301₄are shown enlarged in the inset portion ofFIG. 6. Accordingly, vertical routing of information may be performed (under the management of switch cards351) by an interconnected column ofinterconnection cards301, and horizontal routing of information may be performed by a cell board itself.
• Turning now toFIGS. 7-10C, another example embodiment of a 3D cell board interconnection architecture is shown.FIG. 7 shows anexample cell board701 that comprisescomponents703A,703B,703C,704A,704B,705, and706 implemented thereon, which correspond, for example, tocomponents203A,203B,203C,204A,204B,205, and206 ofcell board201 ofFIG. 2A described above. As withFIG. 2A, the components ofcell board701 are arranged to enable optimal air flow in the front-to-back direction.Cell board701 also comprisesconnectors702A,702B,702C, and702D for coupling to an interconnection structure as described further below. As opposed to the orthogonal connectors used in the example cell board configurations ofFIGS. 2A-2B,connectors702A-702D are connectors as are traditionally used for coupling to a backplane, such as the HMZD connector available from Tyco Electronics.
• FIGS. 8A and 8B show anexample interconnection structure800 that may be utilized for interconnecting a plurality of cell boards, such ascell board701 ofFIG. 7.FIG. 8A shows the front side ofstructure800 andFIG. 8B shows its back side.Example interconnection structure800 effectively provides a porous backplane for interconnecting a plurality of cell boards, which allows for front-to-back air flow. In this example,interconnection structure800 comprisesvertical columns801A,801B,801C, and801D of connectors. The fourcolumns801A-801D are structurally coupled together in this example implementation via horizontal cross members, such ascross member802, to form a matrix structure. Of course, in other implementations, thecolumns801A-801D of connectors may be separate columns that are not structurally coupled together, and such columns may be arranged together to provide an interconnection structure in the manner described below.
• As shown inFIG. 8A, the front-facing side ofcolumns801A-801D comprises connectors for coupling to cell boards, such ascell board701 ofFIG. 7. For instance,column801A comprises eight connectors in this example, includingconnector803 for coupling to a connector of a cell board. The back-facing side ofcolumns801A-801D (FIG. 8B) comprises connectors for coupling to switch cards, such as the switch card described below inFIG. 9. For instance, the back side ofcolumn801A comprises four connectors in this example, includingconnector806 for coupling to a connector of a switch card.
• In this example implementation,interconnection structure800 essentially provides an interface for cell boards and switch cards to be coupled thereto. That is,interconnection structure800 passes information received from a connector to its front side to a connector on its back side (and vice-versa). For instance,interconnection structure800 passes information between its connector803 (which is coupled to a cell board connector) and connector806 (which is coupled to a switch card connector, such as a connector of the switch card ofFIG. 9 described below).
• As shown inFIG. 8A,columns801A-801D and the horizontal cross members connecting such columns are arranged to allow pores (or apertures) through which air may flow. For instance,apertures805A and805B are specifically labeled inFIG. 8A, and permit front-to-back airflow therethrough, as indicated by the arrows inFIG. 8A. In this example implementation, projections, such asprojection804, are included on each column to aid in reducing the resistance to the front-to-back air flow by directing the air to the apertures. This interconnection structure provides a reference plane to reduce tolerancing issues. This interconnection structure may be produced very inexpensively.Such interconnection structure800 provides the 90 degree rotation in this example, rather than performing that rotation in an orthogonal connector (such as in the example cell board configurations ofFIGS. 2A-2B).
• Turning toFIG. 9, anexample switch card900 that may be utilized in this example embodiment is shown.Switch card900 comprisesconnectors901A,901B,901C, and901D for coupling to the connectors on the back-side of a column ofinterconnection structure800 described above, such asconnector806.Switch card900 also comprisescomponents902A and902B, which are ASICs or “cross-bar” chips (shown with heat sinks implemented thereon) for managing switching between the various cell boards coupled tointerconnection structure800. Thus, as withswitch card351 ofFIG. 3B,switch card900 controls the communication between thecell boards701 coupled tointerconnection structure800. That is,switch card900 arbitrates the routing of information between the cell boards. And,switch card351 comprises cabinet-to-cabinet fabric connectors903A-903L to enable a plurality of cabinets to be interconnected. In certain implementations, some ofsuch connectors903A-903L may be used for I/O connections. Further,switch card900 includes connector(s)904 for couplingsuch switch card900 to another switch card within a cabinet, as shown below in the example ofFIG. 10B. Thus, in this example embodiment, horizontal routing is performed by the cell boards701 (i.e., routing from one component on acell board701 to another component on such cell board701), and the vertical routing (i.e., routing from one cell board to another cell board) is performed byswitch900.Interconnection structure800 provides a pass-through structure for interconnecting thecell boards701 and theswitch cards900.Such interconnection structure800 provides a reference plane for connecting thecell boards701 andswitch cards900 to minimize tolerancing issues.
• Turning now toFIG. 10A, an example of a cell board701 (ofFIG. 7) being coupled to interconnection structure800 (ofFIGS. 8A-8B) is shown. As shown,cell board701 connects to a plurality ofdifferent columns801A-801D ofinterconnection structure800. More specifically,connector702A ofcell board701 connects to a connector ofcolumn801D;connector702B ofcell board701 connects to a connector ofcolumn801 C;connector702C ofcell board701 connects to a connector ofcolumn801B; andconnector702D ofcell board701 connects to a connector ofcolumn801A. As shown by the arrows inFIG. 10A, front-to-back air flow is permitted by this arrangement.
• FIG. 10B shows anexample unit1000 that is formed by combining a plurality of thecell boards701 ofFIG. 7 interconnected via a plurality of theinterconnection structures800 ofFIGS. 8A-8B and a plurality of theswitch cards900 ofFIG. 9. In thisexample architecture1000, a plurality of interconnection structures ofFIGS. 8A-8B are implemented, shown asinterconnection structures800A,800B,800C, and800D (not clearly seen inFIG. 10B).
• FIG. 10C shows the backside ofunit1000 ofFIG. 10B, which illustrates thatconnectors904E ofswitch card900E andconnectors904F ofswitch card900F are coupled toconnectors1051 and1052, respectively, of switchcard connector card1050. Such switchcard connector card1050 enables routing of information fromswitch card900E to switchcard900F and vice-versa. Of course, rather than being implemented as a separate card, in certain implementations the connectors ofswitch card connector1050 may be included oninterconnection structure800. That is,interconnection structure800 may be implemented as includingstructures800B and800C, as well asswitch card interconnector1050 ofFIG. 10C.
• In this example, each ofinterconnection structures800A-800D is capable of receiving eight (8)cell boards701, thus enabling a total of 32 cell boards to be included inunit1000. For instance, eightcell boards701 labeled1021 are coupled tointerconnection structure800A; eightcell boards701 labeled1022 are coupled tointerconnection structure800B; eightcell boards701 labeled1023 are coupled tointerconnection structure800C; and eightcell boards701 labeled1024 are coupled tointerconnection structure800D (not clearly shown inFIG. 10B). Coupled to the back-side of the interconnection structures are switch cards ofFIG. 9, such asswitch cards900A,900B, and900C (additional switch cards may be coupled to the back-side of the interconnection structures, but cannot be clearly seen inFIG. 10B).
• Also, in various implementations, a coupling betweencell boards1021 and1022 may be provided either with cables that interconnect theswitch cards900 or with one monolithic panel across the top, or with flex connectors betweeninterconnection structures800A and800B, as examples. Alternatively,interconnection structures800A and800B may be combined as a single interconnect structure, orinterconnection structures800A-800D may all be combined into a single interconnect structure in certain embodiments.
• FIG. 10D shows theunit1000 ofFIG. 10B arranged in acabinet1001. It should be recognized that this architecture allows for the cell boards to be accessed from the front ofcabinet1001, while permitting front-to-back air flow (as shown by the arrows). Thus, a plurality ofsuch cabinets1001 may be arranged side-by-side without the exhaust from one cabinet being ingested by another cabinet (because the air can flow front-to-back in each cabinet). It should also be understood that a plurality ofunits1000 may be coupled together within a cabinet, e.g., in a stacked arrangement, such as inFIGS. 5A-5B of the previous embodiment.
• FIG. 11 shows an example embodiment in which cell boards are arranged on opposing sides of an interconnection structure. More specifically, this example implementation showscell boards1022 that are coupled tointerconnection structure800B in the manner shown above inFIG. 10B. Further,switch cards900A-900D are coupled to the back-side ofsuch interconnection structure800B, as described above. In this example, asecond interconnection structure800E is coupled to switchcards900A-900D on a side opposite thefirst interconnection structure800B, andcell boards1025 are coupled to suchsecond interconnection structure800E.
• Switch cards900A-900D are all redundant, but serviceability of the switch cards may be more difficult in this architecture. Thus, in certain implementations,switch cards900A-900D may be implemented as passive cards and the cross-bar ASICs (if desired) may be included on the cell boards. In certain implementations,cards900A-900D may be implemented as a simple interface and the routing logic may be implemented on switch cards that are coupled tosuch cards900A-900D (e.g., such switch cards may be coupled to the raisededges1101 and/or1102 ofcards900A-900D). The arrangement ofFIG. 11 enables front-to-back air flow, while allowingcell boards1022 to be accessed from the front of the architecture and allowingcell boards1025 to be accessed from the back of the architecture.
• Turning now toFIGS. 12A-16, another example embodiment of a 3D cell board interconnection architecture is shown.FIGS. 12A-12B show an example interconnection structure1200 (e.g., a partial backplane structure).FIG. 12A shows the front side ofinterconnection structure1200, andFIG. 12B shows the back side thereof. As shown inFIG. 12A, this example implementation ofinterconnection structure1200 comprisesconnectors1201A-1201H arranged on its front side for receiving cell boards coupled thereto, as described further below. As shown inFIG. 12B,interconnection structure1200 comprisesconnectors1202A-1202D arranged on its back side for receiving switch cards coupled thereto, as also described further below.
• FIG. 13 shows an example implementation of acell board1300 that may be coupled tointerconnection structure1200 ofFIGS. 12A-12B. This example implementation ofcell board1300 comprisesconnectors1301A and1301B for coupling with connectors on the front-side ofinterconnection structure1200. For example,connector1301A may couple toconnector1201A ofinterconnection structure1200, andconnector1301B may couple toconnector1201B ofinterconnection structure1200.Cell board1300 comprisescomponents1303, which may comprise components such ascomponents203A,203B,203C,204A,204B,205, and206 ofcell board201 ofFIG. 2A described above, for example. As with the example cell board implementations ofFIGS. 2A, 2B, and7, the components ofcell board1300 are arranged to enable optimal air flow in the front-to-back direction.Cell board1300 also comprises power supplies (AC to DC power converters)1304 and coolingfans1305, which may generate a flow of air from front-to-back, as indicated by the arrows. In this example implementation,cell board1300 comprises porous back-cover1302 arranged aroundconnectors1301A and1301B, wherein such porous back-cover1302 permits the front-to-back air flow to exit therethrough.
• Turning toFIGS. 14A-14C, anexample unit1400 is shown.FIG. 14A shows an isometric view ofarchitecture1400 from the front thereof, showing its front, top, and right sides.FIG. 14B shows an isometric view ofarchitecture1400 from the back thereof, showing its back, top, and left sides.FIG. 14C shows a planar view ofarchitecture1400 from its back, without the back-covers of the cell boards (shown as back-covers1302A-1302D inFIG. 14B) being included.
• In thisexample architecture1400, a plurality ofcell boards1300 ofFIG. 13 are implemented, shown ascell boards1300A-1300D (seeFIG. 14A). As shown inFIG. 14B, each cell board is coupled tointerconnection structure1200. For instance, with reference toFIGS. 12A and 14A,cell board1300A is coupled toconnectors1201A and1201B;cell board1300B is coupled toconnectors1201C and1201D;cell board1300C is coupled toconnectors1201E and1201F; andcell board1300D is coupled toconnectors1201G and1201H. As shown inFIG. 14B, each cell board comprises a porous back-cover that permits the front-to-back air flow to exit therethrough. More specifically,cell board1300A comprises porous back-cover1302A;cell board1300B comprises porous back-cover1302B;cell board1300C comprises porous back-cover1302C; andcell board1300D comprises porous back-cover1302D.
• As shown more clearly inFIG. 14C, wherein the architecture is shown without the back-covers on the cell boards, theupper cell boards1300A and1300B are arranged upright, and thelower cell boards1300C and1300D have an opposite orientation when connected tointerconnection structure1200, in this example implementation. In this manner, theupper cell boards1300A and1300B are arranged such that theirrespective components1303A and1303B protrude upward from the cell board, and thelower cell boards1300C and1300D are arranged such that theirrespective components1303C and1303D protrude downward from the cell board. This arrangement aids in minimizing the resistance to the front-to-back air flow presented by the components.
• Theexample architecture1400 ofFIGS. 14A-14C is readily expandable. For instance, as shown inFIGS. 15-16, a plurality of the units may be interconnected (e.g., in a stacked arrangement) to form a larger overall system.FIG. 15 shows an example in which two units ofFIGS. 14A-14C, labeled1400A and1400B, are interconnected to form alarger unit1500 comprising a total of 8 cell boards.FIG. 15 shows an isometric view of the example arrangement from the back, showing its back, top, and left sides. As shown, two interconnection structures ofFIGS. 12A-12B are included, shown asinterconnection structures1200A and1200B. Four cellboards comprising group1400A are connected tointerconnection structure1200A, and four cellboards comprising group1400B are connected tointerconnection structure1200B.
• Also included inunit1500 areswitch cards1501A-1501D. In this example implementation, horizontal routing (e.g., between any ofconnectors1201A-1201H) ofinterconnection structure1200 is performed byinterconnection structure1200. Vertical routing (e.g., routing between a cell board coupled tointerconnection structure1200A and a cell board coupled tointerconnection structure1200B ofFIG. 15), on the other hand, is performed byswitch cards1501A-1501D.
• FIG. 16 shows an example in which 4 of theunits1500 ofFIG. 15, shown asunits1500A-1500D, are interconnected to formcabinet1600 comprising a total of 32 cell boards.FIG. 16 shows an isometric view of the example arrangement from the front, showing the cabinet's front, top, and right sides. Theunits1500A-1500D are interconnected, thus enabling all of the cell boards ofcabinet1600 to be communicatively interconnected.
• FIGS. 17-20 show another example embodiment of a 3D cell board interconnection architecture.FIG. 17 shows anexample cell board1701 that comprisescomponents1703A,1703B,1703C,1704A,1704B,1705, and1706 implemented thereon, which correspond, for example, tocomponents703A,703B,703C,704A,704B,705, and706 ofcell board701 ofFIG. 7 described above. As withFIG. 7, the components ofcell board1701 are arranged to enable optimal air flow in the front-to-back direction.Cell board1701 also comprisesconnectors1702A-1702G for coupling to an interconnection structure as described further below. As with the connectors ofFIG. 7,connectors1702A-1702G are connectors as are traditionally used for coupling to a backplane, such as the HMZD connector available from Tyco Electronics.
• FIGS. 18A and 18B show anexample interconnection structure1800 that may be utilized for interconnecting a plurality of cell boards, such ascell board1701 ofFIG. 17. As shown inFIG. 18A,example interconnection structure1800 includesedge connectors1801A-1801G (referred to collectively as connectors1801) for coupling to acell board1701. That is,edge connectors1801A-1801G are complementary connectors for coupling withconnectors1702A-1702G of afirst cell board1701. Further,interconnection structure1800 includes three additional sets of edge connectors, shown asconnectors1802A-1802G (referred to collectively as connectors1802),1803A-1803G (referred to collectively as connectors1803), and1804A-1804G (referred to collectively as connectors1804), that are each for similarly receiving acell board1701. Accordingly, as discussed further below in connection withFIG. 20, afirst cell board1701 may be coupled toconnectors1801, asecond cell board1701 may be coupled toconnectors1802, athird cell board1701 may be coupled toconnectors1803, and afourth cell board1701 may be coupled toconnectors1804, thereby resulting in a horizontal plane of interconnected cell boards.
• As further shown inFIG. 18A,interconnection structure1800 includes edge connectors for coupling with switch cards, such as theswitch card1900 discussed hereafter in connection withFIGS. 19A-19C. More specifically,interconnection structure1800 includesconnectors1805A-1805D for coupling with a first switch card,connectors1806A-1806D for coupling to a second switch card,connectors1807A-1807D for coupling to a third switch card, andconnectors1808A-1808D for coupling to a fourth switch card.
• FIG. 18B shows an example of the routing provided byinterconnection structure1800. More specifically,FIG. 18B shows an example of the routing provided bystructure1800 for afirst cell board1701 that is coupled tostructure1800 viaconnectors1801. As shown,structure1800 is capable of routing data from acell board1701 to any one of theswitch cards1900 that are coupled tostructure1800. That is,interconnection structure1800 is capable of routing data between a cell board coupled toconnectors1801 and any one of the switch-card interfaces (or connectors)1805,1806,1807, and1808.
• Turning toFIGS. 19A-19C, anexample switch card1900 that may be utilized in this example embodiment is shown.FIG. 19A shows one side ofswitch card1900 andFIG. 19B shows an opposite side ofswitch card1900, whileFIG. 19C shows an example of the routing provided by thisexample switch card1900. As shown inFIG. 19A,switch card1900 comprisesconnectors1901A-1901H that are each capable of coupling to a set of switch-card connectors ofinterconnection structure1800 described above, such asconnectors1805A-1805D. Thus, for example, switch-card connectors1805A-1805D of structure1800 (FIG. 18A) may be coupled toconnector1901A ofswitch card1900.
• As shown inFIG. 19B,switch card1900 also comprisesconnectors1902A-1902H, which are cabinet-to-cabinet fabric connectors, such as theconnectors903A-903L in theswitch card900 ofFIG. 9 to enable a plurality of cabinets to be interconnected.Connectors1902A-1902H may be implemented as copper wires or as optical cables, as examples. In certain implementations, some ofsuch connectors1902A-1902H may be used for I/O connections.Switch card1900 also comprisescomponents1903A-1903H, which are ASICs or “cross-bar” chips (shown with heat sinks implemented thereon) for managing switching between thevarious cell boards1701 coupled to interconnection structure(s)1800 that are coupled to switchcard1900. Thus, as withswitch card900 ofFIG. 9,switch card1900 controls the communication between thecell boards1701 coupled tointerconnection structure1800. That is,switch card1900 arbitrates the routing of information between thecell boards1701.
• FIG. 19C shows an example of the routing provided byswitch card1900. More specifically,FIG. 19C shows an example of the routing provided byswitch card1900 for afirst interconnection structure1800 that is coupled to switch1900 via connector1901. In this example,connectors1805A-1805D of interconnection structure1800 (FIG. 18A) are coupled toconnectors1901A ofswitch card1900. As shown,switch card1900 is capable of routing data from afirst interconnection structure1800 to anotherinterconnection structure1800 that is coupled to switchcard1900. For instance,ASICs1903A and1903B are operable to route data fromconnectors1805A-1805D of afirst interconnection structure1800 that are coupled toconnectors1901A to asecond interconnection structure1800 that is coupled toconnectors1901B ofswitch card1900. Further,ASIC1903A is capable of routing data toASIC1903C,1903E, and1903G, which are capable of routing such data toother interconnection structures1800 that are coupled toconnectors1901C-1901H.
• Thus, in this example embodiment, horizontal routing is performed by the cell boards1701 (e.g., routing from one component on acell board1701 to another component on such cell board1701), and the vertical routing (i.e., routing from one cell board to another cell board) is performed byswitch1900. Additionally, horizontal routing between different cell boards on a horizontal plane (e.g., a plane formed bymultiple cell boards1701 that are connected to acommon interconnection structure1800 is provided viasuch interconnection structure1800, while routing between different horizontal planes is provided by switch card(s)1900, as described further below in connection withFIG. 20.
• While the horizontal routing between cell boards on a common horizontal plane (i.e., coupled to a common interconnection structure), such as between afirst cell board1701 coupled toconnectors1801 and asecond cell board1701 coupled toconnectors1802, is performed byinterconnection structure1800 in this example, in certain implementations this horizontal routing may be supported by switch card(s)1900. For instance, in certain implementations, rather thaninterconnection structure1800 providing routing between different cell boards coupled thereto, it may route all communication to aswitch card1900, which then routes the communication to the to the proper cell board (even if the cell boards are on a common horizontal plane). For example,interconnection structure1800 may provide communication paths from each set of cell board connectors1801-1804 to switch card connectors1805-1808, such as shown in the example ofFIG. 18B forconnectors1801. Suppose data is received (from a cell board1701) atcell board connectors1801 and is destined to another cell board connector of the same interconnection structure, such asconnectors1802; in an example implementation in which routing between different cell boards of a common horizontal plane is performed through the switch cards, the received data is routed fromconnectors1801 to a switch card (e.g., via a switch card connector, such asconnectors1807, which in turn routes the data tocell board connectors1802 via the communication path between the switch card connector (1807) and suchcell board connectors1802. Of course, in certain implementations communication paths between each of cell board connectors1801-1804 may be included oninterconnection structure1800 such thatinterconnection structure1800 is capable of routing data between any of the cell boards coupled thereto (e.g., between any cell boards of this horizontal plane) without requiring routing of the data to the switch cards.
• Turning now toFIG. 20, anexample unit2000 that is formed by combining a plurality of thecell boards1701 ofFIG. 17 interconnected via a plurality of theinterconnection structures1800 ofFIGS. 18A-18B and a plurality of theswitch cards1900 ofFIGS. 19A-19C is shown. In thisexample architecture2000, eight (8)separate interconnection structures1800 ofFIGS. 18A-18B are implemented, a first of which labeled as1800_Acan be seen. Each of theinterconnection structures1800 are coupled to switch cards1900_A-1900_D(wherein1900_Cand1900_Dare not seen inFIG. 20). Further, four cell boards are coupled to each of theinterconnection structures1800, each forming a horizontal plane of interconnected cell boards (for a total of eight horizontal planes in this example). For instance, cell boards1701_A-1701_Dare coupled to afirst interconnection structure1800_A, forming a first horizontal plane of cell boards. Similarly, cell boards1701_E-1701_Gand another cell board (not seen inFIG. 20) are coupled to a second interconnection structure1800 (not seen inFIG. 20), forming a second horizontal plane of cell boards. In total, thisexample unit2000 provides interconnection of a 4 by 8 arrangement of cell boards, thus allowing interconnection of 32cell boards1701.
• Of course, whileinterconnection structure1800 in this example allows for up to 4 cell boards to be connected thereto, in other implementationssuch interconnection structure1800 may be implemented to permit any number of cell boards to be coupled thereto. For instance, while this example implementation provides for two cell boards to be coupled to opposing sides ofinterconnection structure1800, in other implementations a different number of cell boards (e.g., greater than or less than two) may be allowed for on the opposing sides ofinterconnection structure1800. For example, in certain implementations, cell board connectors may be included oninterconnection structure1800 for coupling four cell boards thereto on each opposing side, thus allowing for a horizontal plane of eight (8) interconnected cell boards. Further, while theexample unit2000 ofFIG. 20 has eight (8) horizontal planes of cell boards, in other implementations such a unit may be implemented to have any number of horizontal planes (andswitch cards1900 may be adapted to account for any such number of horizontal planes).
• Further, as with the example embodiment ofFIGS. 10A-10D, a plurality ofsuch units2000 may be communicatively interconnected (e.g., within a cabinet) viafabric connectors1902A-1902H (FIG. 19B) ofswitch cards1900. Alternatively, in certain embodiments,switch cards1900 may be implemented to span a plurality ofunits2000, andsuch switch cards1900 thereby interconnect the plurality of units. For instance, a first switch card may be available for use in connecting up to eight horizontal planes of cell boards together, and a second, larger, switch card may be available for use in place of the first switch card to enable two units (e.g.,16 horizontal planes of cell boards) to be interconnected when so desired. Additionally, a plurality of cabinets may be communicatively interconnected with each other viafabric connectors1902A-1902H. Thus, this provides a modular architecture that can be readily expanded to implement larger-scale systems as desired. Additionally, this example architecture also permits front-to-back air flow (as shown by the arrows inFIG. 20).
• In the example interconnection architecture ofFIG. 20, a plurality of horizontal planes of interconnected cell boards are provided, wherein each horizontal plane includes a plurality of cell boards interconnected via aninterconnection structure1800. Eachinterconnection structure1800 supports the horizontal routing within its respective horizontal plane (e.g., routing along axes X and Z ofFIG. 1) to enable cell boards within a common horizontal plane to communicate with each other. Additionally, a plurality of different interconnection structures are coupled to one or more switch cards1900 (e.g., switch cards1900_A-1900_Din the example ofFIG. 20). Theswitch cards1900 span a plurality of different horizontal planes, thereby communicatively interconnecting different horizontal planes. That is,switch cards1900 provide the vertical routing (along axis Y ofFIG. 1) for the architecture.
• FIGS. 21A-26 provide various other example 3D interconnection architectures that may be implemented for interconnecting cell boards for forming a desired computer system.FIG. 21A shows an example3D interconnection structure2100 that includesswitch board2101 to which a plurality of cellboard interconnect structures2102 and2103 are coupled. Each interconnect structure is capable of coupling to at least one cell board. For instance,cell board connectors2104 are shown for one interconnection structure andcell board connectors2105 are shown for another interconnection structure ofFIG. 21A. In various alternative implementations, the switching logic (such aslogic1903A-1903G in the example ofFIG. 19C) may be included on eitherswitch card2101 or oninterconnection structures2102 and2103. That is, in certain implementations,structure2101 may be implemented as a passive interconnect board, whilestructures2102 and2103 are implemented as switch cards. To minimize the number of connections and routing complexity,structure2101 is preferably implemented as a switch card whilestructures2102 and2103 are implemented as cell board interconnect structures, whereinsuch switch card2101 is operable to route data between different ones of theinterconnect structures2102 and2103.
• FIG. 21B shows another example3D interconnection structure2120 that includesswitch board2121 to which a plurality of cellboard interconnect structures2122 are coupled. Each interconnect structure is capable of coupling to at least one cell board. For instance,cell board connectors2123 are shown for one interconnection structure ofFIG. 21B. The example architecture ofFIG. 21B is similar to the architecture ofFIG. 21A, whereinswitch board2121 is analogous to switchboard2101 andinterconnection structures2122 are analogous tointerconnection structures2103.
• FIG. 21C shows another example3D interconnection structure2130 that includesswitch boards2131A and2131B to which a plurality of cell board interconnect structures22132 are coupled. Each interconnect structure is capable of coupling to at least one cell board. For instance,cell board connectors2133 and2134 are shown for coupling two cell boards to one interconnection structure ofFIG. 21A. In various alternative implementations, the switching logic (such aslogic1903A-1903G in the example ofFIG. 19C) may be included on eitherswitch cards2131A and2131B or oninterconnection structures2132. Preferably, in the example architecture ofFIG. 21C,switch cards2131A and2131B include the appropriate switching logic for routing data between different ones of theinterconnect structures2132.
• FIG. 22 shows an example of utilizing thearchitecture2100 ofFIG. 21A for interconnecting a plurality ofcell boards2201. As shown, afirst cell board2201_Ais coupled to interconnectstructures2102_Aand2103_A. More specifically,connectors2203 ofcell board2201_Acouple toconnectors2104 ofinterconnect structure2102_A, andconnectors2202 ofcell board2201_Acouple toconnectors2105 ofinterconnect structure2103_A. In this example,switch card2101 includes switching logic for routing data between any of theinterconnect structures2102 and2103, thereby communicatively interconnecting the plurality ofcell boards2201. As shown, this example architecture permits front-to-back air flow, and the cell boards may be accessed (for service) in a common direction with the air flow (i.e., front-to-back).
• FIG. 23 shows an example of utilizing thearchitecture2120 ofFIG. 21B for interconnecting a plurality ofcell boards2301. In this example, two of the 3D interconnection architectures are utilized, shown asarchitectures2120A and2120B. As shown, each cell board is coupled to bothinterconnection architectures2120A and2120B. For instance, afirst cell board2301_Ais coupled tointerconnect structure2122A ofarchitecture2120A and to interconnectstructure2122B ofarchitecture2120B. More specifically,connectors2303 ofcell board2301_Acouple toconnectors2123A ofinterconnect structure2122_A, andconnectors2302 ofcell board2301_Acouple toconnectors2123B ofinterconnect structure2122_B. In this example,switch cards2121A and2121B each include switching logic for routing data between any of the plurality ofcell boards2301. As with the example ofFIG. 22, this example architecture permits front-to-back air flow, and the cell boards may be accessed (for service) in a common direction with the air flow (i.e., front-to-back). Further, this example provides redundancy in that if one ofswitch cards2121A and2120B fails, the interconnection ofcell boards2301 is maintained. For instance,architecture2120A may be serviced whilearchitecture2120B maintains communicative interconnection of thecell boards2301. Accordingly,architecture2120A may be serviced without requiring that the system be shut down.
• FIG. 24 shows an example of utilizing thearchitecture2130 ofFIG. 21C for interconnecting a plurality ofcell boards2401. As shown, each cell board is coupled to an interconnection structure. For instance, afirst cell board2401_Ais coupled to afirst interconnect structure2132_A, which is coupled to bothswitch cards2131A and2131B. More specifically,connectors2403 ofcell board2401_Acouple toconnectors2134 ofinterconnect structure2132_A, andconnectors2402 ofcell board2401_Acouple toconnectors2133 ofinterconnect structure2132_A. In this example,switch cards2131A and2131B include switching logic for routing data between any of theinterconnect structures2132, thereby communicatively interconnecting the plurality ofcell boards2401. As shown, this example architecture permits front-to-back air flow, and the cell boards may be accessed (for service) in a common direction with the air flow (i.e., front-to-back). Additionally, this example provides redundancy in that if one ofswitch cards2131A and2131B fails, the interconnection ofcell boards2401 is maintained. While the example ofFIG. 24 shows one cell board coupled to each interconnect structure, such as cell board2401A connected to interconnect structure2132A, in other implementations a plurality of cell boards may be coupled to each interconnect structure, such as with theexample interconnect structure1800 ofFIGS. 18A-18B discussed above.
• Turning toFIG. 25, anexample cabinet2500 that may be formed utilizing the interconnection architectures ofFIGS. 21A and 21B is shown. In this example,interconnection architectures2100 ofFIG. 21A are used to straddle between different cell boards. That is, a first set of cell boards connect to theinterconnection structures2102 and a second set of cell boards connect to theinterconnection structures2103 ofarchitecture2100.
• More specifically, in the example ofFIG. 25, afirst interconnection structure2120A (ofFIG. 21B) connects to a first set ofcell boards2501. That is, each cell board ofset2501 connects to one ofinterconnection structures2122A, which are each coupled to switchcard2121A. For instance,cell board2501A is coupled to a first one ofinterconnection structures2122A via coupling of the cell board'sconnectors2506 with theconnectors2123A.
• The first set ofcell boards2501 also couple tointerconnection structures2102A ofarchitecture2100A (ofFIG. 21A). That is, each cell board ofset2501 connects to one ofinterconnection structures2102A, which are each coupled to switchcard2101A. For instance,cell board2501A is coupled to a first one ofinterconnection structures2102A via coupling of the cell board'sconnectors2507 with theconnectors2104A. A second set ofcell boards2502 couple tointerconnection structures2103A ofarchitecture2100A. That is, each cell board ofset2502 connects to one ofinterconnection structures2103A, which are each coupled to switchcard2101A. For instance,cell board2502A is coupled to a first one ofinterconnection structures2103A via coupling of the cell board'sconnectors2508 with theconnectors2105A. Thus,interconnection architecture2100A straddles the first set ofcell boards2501 and the second set ofcell boards2502, which enables interconnection of such first and second sets of cell boards in a manner that minimizes cabling withincabinet2500.
• The second set ofcell boards2502 also couple tointerconnection structures2102B ofarchitecture2100B. That is, each cell board ofset2502 connects to one ofinterconnection structures2102B, which are each coupled to switchcard2101B. For instance,cell board2502A is coupled to a first one ofinterconnection structures2102B via coupling of the cell board'sconnectors2509 with theconnectors2104B. A third set ofcell boards2503 couple tointerconnection structures2103B ofarchitecture2100B. That is, each cell board ofset2503 connects to one ofinterconnection structures2103B, which are each coupled to switchcard2101B. For instance,cell board2503A is coupled to a first one ofinterconnection structures2103B via coupling of the cell board'sconnectors2510 with theconnectors2105B. Thus,interconnection architecture2100B straddles the second set ofcell boards2502 and the third set ofcell boards2503, which enables interconnection of such second and third sets of cell boards in a manner that minimizes cabling withincabinet2500.
• The third set ofcell boards2503 also couple tointerconnection structures2102C ofarchitecture2100C. That is, each cell board ofset2503 connects to one ofinterconnection structures2102C, which are each coupled to switchcard2101C. For instance,cell board2503A is coupled to a first one ofinterconnection structures2102C via coupling of the cell board'sconnectors2511 with theconnectors2104C. A fourth set ofcell boards2504 couple tointerconnection structures2103C ofarchitecture2100C. That is, each cell board ofset2504 connects to one ofinterconnection structures2103C, which are each coupled to switchcard2101C. For instance,cell board2504A is coupled to a first one ofinterconnection structures2103C via coupling of the cell board'sconnectors2512 with theconnectors2105C. Thus,interconnection architecture2100C straddles the third set ofcell boards2503 and the fourth set ofcell boards2504, which enables interconnection of such third and fourth sets of cell boards in a manner that minimizes cabling withincabinet2500.
• In this example, aninterconnection architecture2120B (ofFIG. 21B) connects to the fourth set ofcell boards2504. That is, each cell board ofset2504 connects to one ofinterconnection structures2122B, which are each coupled to switchcard2121B. For instance,cell board2504A is coupled to a first one ofinterconnection structures2122B via coupling of the cell board'sconnectors2513 with theconnectors2123B. Further, in this example,interconnection structure2120A and2120B are coupled together via coupling2505 (e.g., cabling, such as a copper or fiber optic wire), which provides redundancy. That is, by having thetop interconnection structure2120A and thebottom interconnection structure2120B communicatively connected, an alternative route is provided for routing data between the cell boards when one of the middle structures has failed. For instance, suppose thatinterconnection structure2100A has failed; in this case, data may be routed between the cell boards ofset2501 and any of the other sets of cell boards via coupling2505 (and in some instances, depending on the other cell board with which set2501 is communicating, one or more of thestructures2100B and2100C). Thus, any one ofstructures2120A,2120B,2100A,2100B, and2100C may be serviced/replaced without shutting down the system in this example, as an alternative route exists around each of the structures. Further, this example architecture permits front-to-back air flow, and the cell boards may be accessed (for service) in a common direction with the air flow (i.e., front-to-back).
• As described above,FIG. 25 provides an example in which switch cards are arranged straddled between different cell boards. For instance,switch card2101A is arranged to communicatively straddle between afirst set2501 of cell boards and asecond set2502 of cell boards, which are coupled tointerconnection cards2102A and2103A, respectively.FIG. 26 provides anexample cabinet2600 in which switch cards are arranged staggered relative to each other.
• More specifically, the example implementation ofFIG. 26 communicatively interconnects32 cell boards (shown as cell boards2602₁-2602₃₂). Of course, in other implementations any number of cell boards may be interconnected in this manner. As shown, each cell board connects to two interconnection structures, which in turn each connect to a separate switch card. For instance,cell board2602₁connects tointerconnection structure2122₁, which is connected to switchcard2121A, andcell board2602₁also connects tointerconnection structure2121₃₃, which is connected to switchcard2121F. As can be seen, this example utilizes the example structures ofFIG. 21B, wherein each cell board connects to two of such structures, an upper and a lower structure. The switch cards of the upper and lower structures are staggered, as discussed further below.
• The upper structures in the example ofFIG. 26 includeswitch cards2121A-2121E. Cell boards2602₁-2602₃₂are each communicatively coupled to switchcards2121A-2121E via interconnection structures2122₁-2122₃₂. The lower structures in the example ofFIG. 26 includeswitch cards2121F-2121I. Cell boards2602₁-2602₃₂are each communicatively coupled tosuch switch cards2121F-2121I via interconnection structures2122₃₃-2122₆₄. Again, the switch cards of the upper and lower structures are arranged staggered relative to each other in this example.
• For instance, in this example implementation,switch card2121A has five interconnection structures,2122₁-2122₅, coupled thereto, for communicatively coupling cell boards2602₁-2602₅tosuch switch card2121A.Switch card2121B has eight interconnection structures,2122₆-2122₁₃, coupled thereto, for communicatively coupling cell boards2602₆-2602₁₃tosuch switch card2121B. Similarly,switch card2121C has eight interconnection structures,2122₁₄-2122₂₁, coupled thereto, for communicatively coupling cell boards2602₁₄-2602₂₁tosuch switch card2121C, andswitch card2121D has eight interconnection structures,2122₂₂-2122₂₉, coupled thereto, for communicatively coupling cell boards2602₂₂-2602₂₉tosuch switch card2121D.Switch card2121E has three interconnection structures,2122₃₀-2122₃₂, coupled thereto, for communicatively coupling cell boards2602₃₀-2602₃₂tosuch switch card2121E.
• The switch cards of the lower structure are arranged staggered relative to the switch cards of the upper structure. For instance,switch cards2121F-2121I each have eight interconnection structures coupled thereto, for communicatively coupling cell boards to them. That is,switch card2121F has eight interconnection structures,2122₃₃-2122₄₀, coupled thereto, for communicatively coupling cell boards2602₁-2602₈tosuch switch card2121F;switch card2121G has eight interconnection structures,2122₄₁-2122₄₈, coupled thereto, for communicatively coupling cell boards2602₉-2602₁₆tosuch switch card2121G;switch card2121H has eight interconnection structures,2122₄₉-2122₅₆, coupled thereto, for communicatively coupling cell boards2602₁₇-2602₂₄tosuch switch card2121H; and switch card2121I has eight interconnection structures,2122₅₇-2122₆₄, coupled thereto, for communicatively coupling cell boards2602₂₅-2602₃₂to such switch card2121I.
• Thus,switch card2121F overlaps (or is staggered) withswitch cards2121A and2121B. That is,switch card2121F is communicatively coupled to cell boards2602₁-2602₈, while cell boards2602₁-2602₅also couple to switchcard2121A and cell boards2602₆-2602₈also couple to switchcard2121B. Further, in this example,switch cards2121A and2121E are coupled together via coupling2601 (e.g., cabling, such as a copper or fiber optic wire), which provides redundancy. That is, by havingswitch cards2121A and2121E communicatively connected, an alternative route is provided for routing data between the cell boards2602₁-2602₃₂when one of theswitch cards2121A-2121I has failed. Thus, redundancy is provided for enabling any one of cell boards2602₁-2602₃₂to communicate with any other of cell boards2602₁-2602₃₂with any one of theswitch cards2121A-2121I having failed. Thus, any one ofswitch cards2121A-2121I may be serviced/replaced without shutting down the system in this example, as an alternative route exists around each of the switch cards. Further, this example architecture permits front-to-back air flow, and the cell boards may be accessed (for service) in a common direction with the air flow (i.e., front-to-back).

Claims (39)

1. A cell board interconnection architecture comprising:

an interconnection structure for interconnecting a plurality of cell boards, said interconnection structure configured to allow air to pass therethrough in the direction in which the cell boards couple therewith.

2. The architecture ofclaim 1 wherein the cell boards couple to the interconnection structure via a front-to-back interface, and wherein air flows through the interconnection structure in the front-to-back direction.

3. The architecture ofclaim 1 wherein the interconnection structure comprises a plurality of interconnect cards, and wherein the plurality of cell boards each couple to multiple ones of the plurality of interconnect cards.

4. The architecture ofclaim 1 wherein the cell boards and the interconnect cards are arranged orthogonal to each other.

5. The architecture ofclaim 1 wherein the interconnection structure comprises a matrix structure.

6. The architecture ofclaim 1 further comprising a plurality of switch cards coupled to the interconnection structure, wherein the switch cards are operable to route information between the plurality of cell boards.

7. A cell board interconnection architecture comprising:

a plurality of cell boards;

an interconnection structure to which said plurality of cell boards are coupled; and

a mechanism for generating a flow of air, wherein the interconnection structure is configured such that the generated air flow is permitted to flow through the interconnection structure.

8. The architecture ofclaim 7 wherein each of the plurality of cell boards includes:

at least one processor; and

a memory subsystem for storing data.

9. The architecture ofclaim 8 wherein each of the plurality of cell boards includes a plurality of processors.

10. The architecture ofclaim 7 wherein each of the plurality of cell boards includes:

a plurality of components; and

logic for routing data between the plurality of components.

11. The architecture ofclaim 7 wherein said interconnection structure comprises a plurality of interconnection boards and wherein each cell board connects to multiple ones of said plurality of interconnection boards.

12. A system comprising:

a plurality of cell boards arranged in a cabinet, wherein the plurality of cell boards are communicatively interconnected via an interconnection structure that permits front-to-back airflow and wherein the plurality of cell boards are accessible via the front of the cabinet.

13. The system ofclaim 12 wherein each of the plurality of cell boards includes:

a plurality of components and logic for managing routing of data between the plurality of components.

14. The system ofclaim 13 wherein the interconnection structure includes logic for managing routing of data between the plurality of cell boards.

15. The system ofclaim 14 wherein the interconnection structure includes at least one switch card.

16. The system ofclaim 12 wherein each of the plurality of cell boards includes a plurality of processors and a memory subsystem.

17. The system ofclaim 12 wherein each of the plurality of cell boards couple to the interconnection structure via front-to-back movement of the cell boards relative to the cabinet.

18. A computer system comprising:

a cell board interconnection architecture for communicatively interconnecting a plurality of cell boards, wherein said architecture is configured to minimize resistance to air flow in a direction.

19. The computer system ofclaim 18 wherein the cell board interconnection architecture comprises at least one interconnection card for communicatively interconnecting said plurality of cell boards.

20. The computer system ofclaim 19 wherein each of said at least one interconnection card is arranged to have its smallest surface area facing said direction.

21. The computer system ofclaim 20 wherein each of said plurality of cell boards are arranged to have their smallest surface area facing said direction.

22. A cell board interconnection architecture comprising:

a plurality of cell boards; and

an interconnection structure that communicatively interconnects the plurality of cell boards and allows for air flow in a direction parallel to service access of the plurality of cell boards.

23. The cell board interconnection architecture ofclaim 22 wherein said interconnection structure enables front-to-back access for both air flow and service.

24. A computer system comprising:

a plurality of cell boards; and

a porous interconnection structure for interconnecting the plurality of cell boards such that air can flow through the interconnection structure.

25. The computer system ofclaim 24 wherein the air can flow through the porous interconnection structure in a direction parallel to a direction in which the cell boards couple to the porous interconnection structure.

26. The computer system ofclaim 24 wherein the porous interconnection structure comprises a plurality of interconnect cards, and wherein the plurality of cell boards each couple to multiple ones of the plurality of interconnect cards.

27. The computer system ofclaim 24 further comprising a plurality of switch cards coupled to the porous interconnection structure, wherein the switch cards are operable to route information between the plurality of cell boards.

28. A multi-processor computer system comprising:

a plurality of cell boards that each include at least one processor and logic for managing routing of data between components of the cell board; and

an interconnection structure having a first side coupled to said plurality of cell boards and an opposite side coupled to at least one switch card, wherein at least one of said interconnection structure and said at least one switch card includes logic for managing routing of data between the plurality of cell boards.

29. The multi-processor computer system ofclaim 28 wherein said plurality of cell boards are arranged orthogonal to said interconnection structure.

30. The multi-processor computer system ofclaim 28 wherein said interconnection structure is configured to enable air to flow therethrough.

31. The multi-processor computer system ofclaim 28 wherein the interconnection structure comprises a plurality of interconnect cards, and wherein each of the plurality of cell boards couples to multiple ones of the plurality of interconnect cards.

32. A computer system comprising:

a plurality of cell boards that each include a plurality of components and logic for managing routing of data between the plurality of components; and

an interconnection structure comprising a plurality of interconnect cards, wherein each of the plurality of cell boards connects to multiple ones of the plurality of interconnect cards thereby communicatively interconnecting the plurality of cell boards.

33. The computer system ofclaim 32 wherein said interconnection structure includes logic for managing routing of data between the plurality of cell boards

34. The computer system ofclaim 32 wherein the interconnection structure comprises at least one switch card.

35. The computer system ofclaim 32 wherein the interconnection structure is adapted to allow air to flow therethrough.

36. A system comprising:

a plurality of cell boards;

means for communicatively interconnecting the plurality of cell boards, wherein the interconnecting means comprises a first plane for routing information in at least a first dimension and a second plane that is orthogonal to the first plane for routing information in at least a second dimension that is different from said first dimension.

37. The system ofclaim 36 wherein each of said plurality of cell boards includes at least one processor.

38. The system ofclaim 36 wherein the interconnecting means is configured to allow air to flow therethrough.

39. The system ofclaim 38 wherein the interconnecting means is configured to allow said air to flow therethrough in a direction parallel to a direction in which the plurality of cell boards couple to the interconnecting means.

Images