Summary of the invention
The present invention provides a kind of hardware reconstruction system and methods, to solve in prior art hardware reconstruction to all partsThe unbalanced problem of utilization rate.
On the one hand, the present invention provides a kind of hardware reconstruction systems includes:
Between each resource pool and the resource pool closed on, the multichannel interface connection based on remote PCI e crossbar switch, resourceMultichannel interface connection in pond based on PCIe crossbar switch, remote PCI e crossbar switch and the PCIe crossbar switch in resource pool,It is connected together by shared interface, forms the double-deck crossbar configuration based on PCIe crossbar switch and remote PCI e crossbar switchLink；
Wherein, the resource pool includes computing resource pool and/or memory resource pool, and each processor passes through resource in resource poolPCIe crossbar switch inside pond is interconnected, and when being attached configuration, is configured inside resource pool by configuration informationPCIe crossbar switch connection relationship is calculating and storage resource inside the specified PCIe bus assignment resource pool of CPU, realizes meterIt calculates and is directly connected between resource pool, memory resource pool inner treater, support computing resource and the distribution of storage resource dynamic on-demand；
The remote PCI e crossbar switch is interconnected by shared interface and each resource pool, realizes the resource based on PCIe protocolPoint-to-point peer switch between pond configures the connection of remote PCI e crossbar switch by configuration information when being attached configurationRelationship is the processor resource in the specified PCIe bus assignment Adjacent resource pond of CPU, carries out calculating and storage money between resource poolSource dynamically distributes.
Preferably, the shared interface supports two kinds of interface modes of transparent bridge and non-transparent bridge, can be configured to main sideMouthful or from port, interconnected between realizing port by PCIe shared interface cable.
Preferably, the PCIe crossbar switch in the resource pool includes with the realization of PCIe switching backplane, PCIe switching backplaneMultichannel PCIe bus slot and PCIe exchange chip.
Preferably, the remote PCI e crossbar switch is realized with PCIe switch, realizes the point-to-point exchange between resource poolMode, including one of following or a variety of: packet switch, circuit switching and virtual channel exchange.
Preferably, virtual channel exchange multiple shares the same physical channel for being made up of virtual channelVirtual circuit exchange connection, and virtual circuit exchange connection can connect with packet switch connection and circuit switching cooperate to transmitFlow data.
It preferably, further include compiler；The compiler is used for according to known internuclear stream communication situation, for everyKnow stream communication distribution path, and determine that virtual circuit exchange connection, circuit switching connection are connected with packet switch, at runtime basisAs a result, establishing each exchange connection by the link information prestored, the communication between different resource pond is respectively being corresponded to for communication compilingExchange connection on transmitted.
Preferably, the remote PCI e crossbar switch includes multiple input units, and a crossbar switch crosses over unit, pathComputing unit, virtual channel allocation unit, crossbar switch allocation unit and circuit configuration unit；
The input unit, for the virtual channel number after the PCIe device configuration output arbitration for connection, each inputUnit includes n output virtual channel VC1-VCn, and a bypass channel, a PS state stores and a VCS state storage,The input unit exchanges VCS signal with the input virtual channel of PCIe device and port input signal is connected, the storage of PS stateThe virtual channel state of corresponding basic packet switch router, VCS state store corresponding virtual circuit and exchange connection status, work as inputWhen for VCS signal, then the corresponding output virtual channel of input virtual channel is directly found by the information in the storage of VCS stateNumber, VCS signal is directly output to the corresponding PCIe device of output virtual channel number, and each input unit increases a bypassUnit is crossed over to allow packet switched data piece to be directly inputted to the crossbar switch in channel；
The path-calculating element exists for establishing the port ID- routed path or address: Port routed path for transmissionThe data slice of packet switch connection provides path arbitration result；
The virtual channel allocation unit, for the path arbitration result according to the path-calculating element or according to PS shapeState stores information, for input data piece distribution output virtual channel number；
The circuit configuration unit is matched for providing the crossbar switch prestored in the data slice that circuit switching connects for transmissionConfidence breath；
The crossbar switch allocation unit stores information for storing preset circuit configuration by circuit configuration, directlyCrossbar switch is configured, the data slice that circuit switching is connected directly enters the intersection by by-pass switch and opensIt closes and crosses over unit；
The crossbar switch exports data packet for the arbitration scheduling result according to the input unit across unitTo the corresponding PCIe device of output virtual channel number.
Preferably, the computing resource pool includes one of following or a variety of: CPU, GPU, FPGA, IPU.
Preferably, the memory resource pool includes one of following or a variety of: SSD, HDD.
On the other hand, the present invention provides a kind of hardware reconstruction methods, and this method is using hardware described in any of the above-described kindReconfiguration system, comprising:
Step 1: PCIe bus being scanned by management node, acquisition is connected to PCIe crossbar switch in PCIe multibus and pondOn PCIe device port information, wherein the management node be preset cpu node；
Step 2: the remote PCI e crossbar switch that PCIe crossbar switch shared interface connects in pond where management node obtainsPCIe device port information；
Step 3: CPU, GPU, FPGA, IPU, SSD, HDD for specifying PCIe bus to connect according to user requirements analysis etc. is hardThe type and quantity of part equipment；
Step 4: the type and quantity of the hardware device according to needed for step 3 judge local computing and memory resource poolWhether interior resource is enough, and step 5 is transferred to if enough, not enough, is transferred to step 7；
Step 5: management node configures the configuration space of the PCIe crossbar switch in resource pool, according to depth-firstSearching algorithm is that specified PCIe bus configuration distributes required processor, PCIe crossbar switch in configured pool, and is connected in pondPrincipal and subordinate's bus of device port, the information such as command register are managed everywhere in PCIe crossbar switch；
Step 6: passing through PCIe data needed for the packet format of PCIeDMA or the PCIe link layer protocol of the standard building present inventionPacket accesses the equipment for being connected to specified PCIe bus, terminates the configuration of PCIe hardware reconstruction；
Step 7: management node configures the configuration space of the PCIe crossbar switch in resource pool, total for specified PCIeLine distributes available calculating and storage resource；
Step 8: management node configures remote PCI e crossbar switch, carries out empty friendship according to remote PCI e crossbar switchChannel, packet switch channel, circuit-switched channels configuration are changed, two kinds of connecting interfaces of transparent bridge and non-transparent bridge are supported, needed for connectionAdjacent calculating and memory resource pool；
Step 9: management node is needed for specified PCIe bus assignment local resource pond according to Depth Priority AlgorithmProcessor, processor needed for selected adjacent resource pool, in configured pool/remote PCI e crossbar switch, and it is connected to PCIe intersectionPrincipal and subordinate's bus of device port, the information such as command register are managed everywhere in switch, and are transferred to step 6.
The present invention has the beneficial effect that:
The present invention GPU, FPGA, GPU, IPU, SSD etc. can be calculated by PCIe multibus and crossbar switch technology andStorage resource is decoupled from server and is come out, and is realized and the layers such as interconnected in the pond of Heterogeneous Computing and memory resource pool between interconnection, pondThe interconnection of secondaryization.The connection re-configurability of configuration offer standard PCIe computing resource based on PCIe crossbar switch is provided.Pass throughThe introducing of PCIe switch can create the high highly available system of cost performance, make full use of expanding by PCIe communicationMalleability, performance advantage, can support most of heterogeneous device, and versatility is stronger；PCIe configuration feature is provided, it can be according to data centerThe performance of server needs switching at runtime to calculate, storage and network, can be the calculate node of application distribution correct amount, each to countCalculating storage resource can be managed collectively, flexible allocation, support connection reconstruct；Data transfer delay, system complex can be reducedProperty and power consumption, Save power consumption rate is up to 50% in data-intensive environment.
The above description is only an overview of the technical scheme of the present invention, in order to better understand the technical means of the present invention,And it can be implemented in accordance with the contents of the specification, and in order to allow above and other objects of the present invention, feature and advantage canIt is clearer and more comprehensible, the followings are specific embodiments of the present invention.
Specific embodiment
Exemplary embodiments of the present disclosure are described in more detail below with reference to accompanying drawings.Although showing the disclosure in attached drawingExemplary embodiment, it being understood, however, that may be realized in various forms the disclosure without should be by embodiments set forth hereIt is limited.On the contrary, these embodiments are provided to facilitate a more thoroughly understanding of the present invention, and can be by the scope of the present disclosureIt is fully disclosed to those skilled in the art.
First embodiment of the invention provides a kind of hardware reconstruction system, referring to Fig. 1, comprising:
Between each resource pool and the resource pool closed on, the multichannel interface connection based on remote PCI e crossbar switch, resourceMultichannel interface connection in pond based on PCIe crossbar switch, remote PCI e crossbar switch and the PCIe crossbar switch in resource pool,It is connected together by shared interface, forms the double-deck crossbar configuration based on PCIe crossbar switch and remote PCI e crossbar switchLink；
Wherein, the resource pool includes computing resource pool and/or memory resource pool, and each processor passes through resource in resource poolPCIe crossbar switch inside pond is interconnected, and when being attached configuration, is configured inside resource pool by configuration informationPCIe crossbar switch connection relationship is calculating and storage resource inside the specified PCIe bus assignment resource pool of CPU, realizes meterIt calculates and is directly connected between resource pool, memory resource pool processor, support computing resource and the distribution of storage resource dynamic on-demand；
The remote PCI e crossbar switch is interconnected by shared interface and each resource pool, realizes the resource based on PCIe protocolPoint-to-point peer switch between pond configures the connection of remote PCI e crossbar switch by configuration information when being attached configurationRelationship is the processor resource in the specified PCIe bus assignment Adjacent resource pond of CPU, carries out calculating and storage money between resource poolSource dynamically distributes.
The present invention GPU, FPGA, GPU, IPU, SSD etc. can be calculated by PCIe multibus and crossbar switch technology andStorage resource is decoupled from server and is come out, and is realized and the layers such as interconnected in the pond of Heterogeneous Computing and memory resource pool between interconnection, pondThe interconnection of secondaryization.The connection re-configurability of configuration offer standard PCIe computing resource based on PCIe crossbar switch is provided.Pass throughThe introducing of PCIe switch makes full use of the scalability and performance advantage communicated by PCIe, can create cost performance heightHighly available system, can support most of heterogeneous device, versatility is stronger；PCIe configuration feature is provided, it can be according in dataThe performance of central server needs switching at runtime to calculate, storage and network, can be the calculate node of application distribution correct amount, respectivelyCalculating storage resource can be managed collectively, flexible allocation, support connection reconstruct；It is multiple that data transfer delay, system can be reducedPolygamy and power consumption, Save power consumption rate is up to 50% in data-intensive environment.
It should be noted that shared interface described in the embodiment of the present invention supports two kinds of interface moulds of transparent bridge and non-transparent bridgeFormula can be configured to master port or from port, interconnect between realizing port by PCIe shared interface cable.
In the embodiment of the present invention, the PCIe crossbar switch in the resource pool is with the realization of PCIe switching backplane, PCIe exchangeBottom plate includes multichannel PCIe bus slot and PCIe exchange chip.
In the embodiment of the present invention, the remote PCI e crossbar switch is realized with PCIe switch, realizes the point between resource poolTo an exchanged form, including one of following or a variety of: packet switch, circuit switching and virtual channel exchange.
The exchange of virtual channel described in the embodiment of the present invention multiple shares the same physics for being made up of virtual channelThe virtual circuit in channel exchanges connection, and virtual circuit exchange connection can connect cooperation with packet switch connection and circuit switching and comeTransmit flow data.
System described in the embodiment of the present invention further includes compiler；The compiler, for according to known internuclear circulationBelieve situation, be every known flow allocation of communications path, and determines that virtual circuit exchange connection, circuit switching connection and packet switch connectIt connects, is connected at runtime according to communication compiling as a result, establishing each exchange by the link information prestored, between different resource pondCommunication is transmitted in corresponding exchange connection.
In the embodiment of the present invention, the remote PCI e crossbar switch includes multiple input units, and a crossbar switch is crossed overUnit, path-calculating element, virtual channel allocation unit, crossbar switch allocation unit and circuit configuration unit；
The input unit, for the virtual channel number after the PCIe device configuration output arbitration for connection, each inputUnit includes n output virtual channel VC1-VCn, and a bypass channel, a PS state stores and a VCS state storage,The input unit exchanges VCS signal with the input virtual channel of PCIe device and port input signal is connected, the storage of PS stateThe virtual channel state of corresponding basic packet switch router, VCS state store corresponding virtual circuit and exchange connection status, work as inputWhen for VCS signal, then the corresponding output virtual channel of input virtual channel is directly found by the information in the storage of VCS stateNumber, VCS signal is directly output to the corresponding PCIe device of output virtual channel number, and each input unit increases a bypassUnit is crossed over to allow packet switched data piece to be directly inputted to the crossbar switch in channel；
The path-calculating element exists for establishing the port ID- routed path or address: Port routed path for transmissionThe data slice of packet switch connection provides path arbitration result；
The virtual channel allocation unit, for the path arbitration result according to the path-calculating element or according to PS shapeState stores information, for input data piece distribution output virtual channel number；
The circuit configuration unit is matched for providing the crossbar switch prestored in the data slice that circuit switching connects for transmissionConfidence breath；
The crossbar switch allocation unit stores information for storing preset circuit configuration by circuit configuration, directlyCrossbar switch is configured, the data slice that circuit switching is connected directly enters the intersection by by-pass switch and opensIt closes and crosses over unit；
The crossbar switch exports data packet for the arbitration scheduling result according to the input unit across unitTo the corresponding PCIe device of output virtual channel number.
It should be noted that computing resource pool described in the embodiment of the present invention includes one of following or a variety of: CPU,GPU, FPGA, IPU, etc..The memory resource pool includes one of following or a variety of: SSD, HDD, etc..Specific abilityThe technical staff in domain can be configured according to actual needs, and the present invention is not especially limited this.
Detailed explanation and illustration is carried out below in conjunction with Fig. 1-6 system described in the embodiment of the present invention:
As shown in Figure 1, the present invention constructs a series of resource pools first, and between each resource pool and the resource pool closed on, baseIt is connected in the multichannel interface of remote PCI e crossbar switch, the multichannel interface connection in resource pool based on PCIe crossbar switch, remotelyPCIe crossbar switch in PCIe crossbar switch and resource pool, is connected together by shared interface, formed based on crossbar switch andThe double-deck crossbar configuration link of remote PCI e crossbar switch supports star topology, network topology, tree-shaped between each resource poolA variety of connection relationships such as topology and full connection topology；
Wherein, resource pool includes computing resource pool and/or memory resource pool, and each processor passes through in resource pool in resource poolThe crossbar switch in portion is interconnected.Computing resource pool includes the Heterogeneous Computings resource such as CPU, GPU, FPGA, IPU, memory resource poolThe storage resources such as including SSD, HDD.Each calculating memory resource pool contains the pond CPU, the pond GPU, the pond FPGA, the pond IPU, storage poolDeng at least one resource pool.Wherein: the pond CPU contains muti-piece CPU processor, and the pond GPU contains muti-piece GPU processor, FPGAPond contains muti-piece FPGA processor, and the pond IPU contains the IPU processor such as muti-piece Cambrian, storage pool contain muti-piece SSD,The hard-disc storages resource such as HDD, each calculating and storage resource externally pass through PCIe bus and are attached with crossbar switch.ConnectedWhen connecing configuration, the crossbar switch connection relationship inside resource pool is configured by configuration information, is the specified PCIe bus assignment of CPUCalculating and storage resource inside resource pool are realized and are directly connected between computing resource pool, memory resource pool inner treater, are propped upHold computing resource and the distribution of storage resource dynamic on-demand.
Crossbar switch in resource pool includes multiple shared interfaces, and shared interface support transparent bridge and two kinds of non-transparent bridge connectMouth mold formula can be configured to master port or from port, interconnect, can connect between realizing port by PCIe shared interface cableThe PCIe device of arbitrary standards PCIe3.0 interface.Shared interface crossbar switch by multichannel PCIe bus and the pond CPU, the pond GPU,The pond FPGA, the pond IPU, in storage pool everywhere in reason device be directly connected to, configuration information can be passed through and control specified input interface and outputThe direct-connected relationship of interface can configure crossbar switch and specify PCIe bus assignment by the way that reconfiguration information is inputted crossbar switchIt calculates, storage resource, is calculated for CPU, FPGA, GPU, IPU, the memory etc. of multichannel PCIe bus assignment cascade on the busAnd storage resource.
Remote PCI e crossbar switch realizes point-to-point peer switch between the resource pool based on PCIe protocol.It is long-range to intersectSwitch is attached by shared interface and the crossbar switch in resource pool, realizes the communication interconnection between resource pool.ConnectedWhen connecing configuration, long-range crossbar switch connection relationship is configured by configuration information, is the specified PCIe bus assignment Adjacent resource of CPUThe processor resource in pond supports calculating and storage resource between resource pool to dynamically distribute.
By input reconstruct configuration information to crossbar switch, crossbar switch occupies feelings according to user demand and current systemCondition changes the connection relationship of the virtual PCI Bridge in crossbar switch by configuration information, is connected to it for specified PCIe bus assignmentIn calculating and storage resource.Such as a moment specifies the system 1 in PCIe bus for 4 pieces of GPU, 4 pieces of CPU, 4 pieces of FPGA, 8Block SSD；Subsequent time configures crossbar switch by reconfiguration information, and the computing system connected in the PCIe bus becomes system 2,Including 2 pieces of CPU, 1 piece of GPU, 1 piece of FPGA, 2 pieces of SSD；Subsequent time again configures crossbar switch by reconfiguration information, and the PCIe is totalThe computing system connected on line becomes system 3, including 1 piece of CPU, 1 piece of GPU, 1 piece of FPGA, 2 pieces of SSD.Pass through change to be formedThe connection relationship between storage resource is calculated, so that hardware system supports computing capability and concatenation ability reconstruct, to be formedIt calculates and the dynamic on-demand of storage resource distributes.
As shown in Fig. 2, the embodiment of the present invention includes 6 calculating and memory resource pool, 1 remote PCI e crossbar switch and 1A main controlled node.Wherein: including 1 piece of PCIe crossbar switch backboard in each resource pool, it includes multichannel PCIe that PCIe, which intersects backboard,Bus slot and 1 PEX8796 exchange chip, the calculating such as CPU, GPU, FPGA, IPU, SSD and storage resource insertion PCIe intersectIn the slot for switching backboard, the interconnection reconstruct in resource pool is realized.By PCIe shared interface connecting line and far between resource poolJourney crossbar switch is connected, and the MiniSAS interface cable of the shared interface connection line options PCIex4 of present case is realized.It is long-range to intersectSwitch realizes that the PCIe switch of the implementation case includes 12 ports in the form of PCIe switch, can support 12 metersIt calculates and the point-to-point peer switch of memory resource pool, 12 ports can be configured to transparent bridge mode and non-transparent bridge mode,Transparent bridge mode for connect calculating/memory resource pool without CPU, non-transparent bridge mode be used to connect include host calculating/Memory resource pool.Main controlled node is connected by remote PCI e crossbar switch with multiple calculating and memory resource pool, for reconstructingConfiguration and Data Transmission Controlling.Main controlled node is by PCIe crossbar switch in remote PCI e crossbar switch and resource pool to connectionHardware device is scanned, by PCIe crossbar switch in resource pool, remote PCI e crossbar switch according to reconstruct configuration informationIt is configured, distributes different calculating and storage resource for CPU, realize hardware system reconstruct.
As shown in figure 3, the reconfiguration system based on crossbar switch in the resource pool of another specific implementation case of the invention,Wherein, crossbar switch is realized in the form of PCIe switching backplane in resource pool, and PCIe switching backplane includes that multichannel PCIe bus is insertedSlot and PCIe exchange chip.It will be in crossbar switch backboard slot in different calculating and storage resource insertion pond.The resource poolHardware structure dynamic construction uses the method based on PCIe multiple bus and crossbar switch, and configuration information controls the defeated of crossbar switchThe interconnecting relation of incoming interface and output interface.The hardware resources such as difference CPU, GPU, FPGA, IPU, SSD, HDD are connected in figureThe different PCIe bus slot of crossbar switch, CPU, which can choose, to be connected to the hardware device group of specified bus slot and builds up differenceComputing system.By configuring the crossbar switch in resource pool, the hardware device connected in PCIe bus can change.ExampleSuch as: for subsequent time by changing crossbar configuration, the hardware resource of the PCIe buses such as PCIe bus 1, PCIe bus 2 connection canTo be exchanged, additions and deletions etc. change, to form new computing system.
For the hardware resources such as CPU, GPU, FPGA, IPU, SSD, HDD in different calculating and memory resource poolReconstruction Mechanism is as shown in Figure 4.It is interconnected between Adjacent resource pond by remote PCI e crossbar switch, is handled between realization resource poolThe communication of device.Remote PCI e crossbar switch is realized in the form of PCIe switch, including multiple PCIe shared ports, different resourceThe hardware structure dynamic construction in pond is realized based on the control of remote PCI e crossbar switch.
Such as: the hardware resource of resource pool 1 and resource pool 2 is carried out by remote PCI e crossbar switch shown in Fig. 4 in figureConnection allows resource pool 1 and the hardware resource of resource 2 directly can realize data interaction by the transmission of PCIe.Remote PCI e intersectsSwitch is connected together with the crossbar switch in resource pool by shared interface.It is formed and is opened based on crossbar switch and remote PCI e intersectionThe double-deck crossbar configuration link of pass.When being attached configuration, calculating and storage resource are configured by configuration information firstCrossbar switch connection relationship inside pond calculates with the specified PCIe bus assignment inside memory resource pool and stores money to calculateSource；Then the remote PCI e crossbar switch between resource pool is configured, realizes that resource pool is directly connected to resource pool.As shown in Figure 4Resource pool between hardware reconstruction mechanism, can be made by the configuration to crossbar switch and remote PCI e crossbar switch in pondObtain may be implemented to be in communication with each other between SSD, CPU in CPU, FPGA and mainboard 2 in mainboard 1, without influencing other hardware moneyThe use in source.
Fig. 5 is the composed structure schematic diagram of long-range crossbar switch, as shown in figure 5, the embodiment of the present invention is supported in followingOne or more exchanged forms: packet switch PS, circuit switching (CS) and virtual channel exchange VCS.Virtual channel exchange, for passing throughVirtual channel forms multiple virtual circuits exchange connections for sharing the same physical channel, and virtual circuit exchange connection can be withCooperation is connect with packet switch connection and circuit switching to transmit flow data.According to known internuclear stream communication situation, pass through compilingDevice is every known flow allocation of communications path, and determines that virtual circuit exchange connection, circuit switching connection are connected with packet switch,It is established respectively according to communication compiling as a result, link information is stored in each PCIe exchanger by the link information prestored when operationExchange connects, and the communication between different motherboard resources ponds is transmitted in corresponding exchange connection.
Compared with basic packet switcher, the exchanger of mixed strategy of the present invention needs some additional hardware: including sidePaths, circuit configuration storage and the storage of VCS state.Remote PCI e crossbar switch includes multiple input units, and an intersection is openedIt closes and crosses over, path computing, the submodules such as virtual channel distribution, crossbar switch distribution, circuit configuration.
Wherein: input unit is used for the virtual channel number after the PCIe device configuration output arbitration for connection, each inputUnit includes n output virtual channel VC1-VCn, a bypass channel, a PS state storage and a VCS state storage.The input unit exchanges VCS signal and port input signal with the input virtual channel of PCIe device and is connected, VCS signal and defeatedThe port numbers of the number and PCIe device that enter signal correspond.Virtual the leading to of the corresponding basic packet switch router of PS state storageRoad state.VCS state stores corresponding virtual circuit and exchanges connection status.When input is VCS signal, then directly pass through VCS shapeInformation in state storage finds the corresponding output virtual channel number of input virtual channel, and it is empty that VCS signal is directly output to outputThe quasi- corresponding PCIe device of channel number；PS state and VCS state have all respectively contained n domain, this corresponding n virtual channel.AndAnd n virtual channel is connected with packet switch by virtual circuit exchange connection and is shared.And each input unit increases a bypassChannel allows data slice not need through virtual channel, is directly inputted to crossbar switch across submodule.
Path calculation module is attached with input unit, for establishing the port ID- routed path or address: Port routingPath provides path arbitration result in the data slice that packet switch connects for transmission.Wherein ID- port routing table provides PCIe deviceMap paths relationship number between port numbers；Address: Port routing table is provided between PCIe device physical address and port numbersMap paths relationship.Communication path arbitration is carried out according to the ID in PCIe transaction packet, address when operation.
Virtual channel allocation unit stores letter for the path arbitration result according to path-calculating element or according to PS stateBreath, for input data piece distribution output virtual channel number；
Circuit configuration unit, for providing the crossbar configuration prestored letter in the data slice that circuit switching connects to transmitBreath, storage are used to connect the link information of storage physical channel to circuit switching, distribute with crossbar switch distribution and virtual channelModule is attached, and circuit switching connection is established by setting circuit configuration information.
Crossbar switch allocation unit stores information for storing preset circuit configuration by circuit configuration, directly to friendshipFork puts capable configuration into, and the data slice that circuit switching is connected directly enters crossbar switch across son by by-pass switchModule；
Data packet is output to output for the arbitration scheduling result according to input unit across submodule by crossbar switchThe corresponding PCIe device of virtual channel number.
For the mixed strategy for realizing support circuit-switched, packet switch and virtual channel exchange, this patent has modified PCIe thingIt is engaged in the PCIe transaction layer packet header wrapped, exchange flag bit being added in reserved field, as shown in Figure 6.Carrying out PCIe crossbar switchWhen hardware reconstruction configures, according to the exchange flag bit in PCIe transaction layer packet header, corresponding exchanging policy is selected.
If the exchange flag bit in input signal PCIe transaction layer packet header is input virtual channel VCS signal, directly logicalVCS status information is crossed, finds the corresponding output virtual channel number of input virtual channel, it is empty that VCS signal is directly output to outputThe quasi- corresponding PCIe device of channel number.
If input signal is PCIe port input signal, needs to be inputted in input unit according to PCIe port first and believeNumber carry out arbitration scheduling.It is that transmission is handed in circuit if the exchange flag bit in PCIe transaction layer packet header is circuit-switched signalsThe data slice for changing connection reaches input unit, and crossbar switch is stood across submodule according to the link information prestored in circuit configurationIt configures, then the data slice directly passes through bypass channel and enters crossbar switch across submodule, and crossbar switch crosses over submoduleData packet is directly output to corresponding PCIe device according to configuration result.
If input signal is PCIe port input signal, needs to be inputted in input unit according to PCIe port first and believeNumber carry out arbitration scheduling.It is that transmission connects in packet switch if the exchange flag bit in PCIe transaction layer packet header is packet switch signalThe data slice that connects reaches input unit, then first according to the path arbitration result of path-calculating element or according to PS status information,Output virtual channel number is distributed for input data piece, output virtual port number is then sent to crossbar switch and crosses over submodule,Data packet, according to arbitration scheduling result, is output to the corresponding PCIe of output virtual channel number and set by crossbar switch across submoduleIt is standby.
Second embodiment of the invention provides a kind of hardware reconstruction method, and referring to Fig. 7, this method application present invention first is realApply any hardware reconstruction system in example, comprising:
Step 1: PCIe bus being scanned by management node, acquisition is connected to PCIe crossbar switch in PCIe multibus and pondOn PCIe device port information, wherein the management node be preset cpu node；
Step 2: the remote PCI e crossbar switch that PCIe crossbar switch shared interface connects in pond where management node obtainsPCIe device port information；
Step 3: CPU, GPU, FPGA, IPU, SSD, HDD for specifying PCIe bus to connect according to user requirements analysis etc. is hardThe type and quantity of part equipment；
Step 4: the type and quantity of the hardware device according to needed for step 3 judge local computing and memory resource poolWhether interior resource is enough, and step 5 is transferred to if enough, not enough, is transferred to step 7；
Step 5: management node configures the configuration space of the PCIe crossbar switch in resource pool, according to depth-firstSearching algorithm is that specified PCIe bus configuration distributes required processor, PCIe crossbar switch in configured pool, and is connected in pondPrincipal and subordinate's bus of device port, the information such as command register are managed everywhere in PCIe crossbar switch；
Step 6: passing through PCIe data needed for the packet format of PCIeDMA or the PCIe link layer protocol of the standard building present inventionPacket accesses the equipment for being connected to specified PCIe bus, terminates the configuration of PCIe hardware reconstruction；
Step 7: management node configures the configuration space of the PCIe crossbar switch in resource pool, total for specified PCIeLine distributes available calculating and storage resource；
Step 8: management node configures remote PCI e crossbar switch, carries out empty friendship according to remote PCI e crossbar switchChannel, packet switch channel, circuit-switched channels configuration are changed, two kinds of connecting interfaces of transparent bridge and non-transparent bridge are supported, needed for connectionAdjacent calculating and memory resource pool；
Step 9: management node is needed for specified PCIe bus assignment local resource pond according to Depth Priority AlgorithmProcessor, processor needed for selected adjacent resource pool, in configured pool/remote PCI e crossbar switch, and it is connected to PCIe intersectionPrincipal and subordinate's bus of device port, the information such as command register are managed everywhere in switch, and are transferred to step 6.
The embodiment of the present invention is realized by the way that configuration is reconstructed to crossbar switch, remote PCI e crossbar switch in resource poolComputing capability reconstruct, concatenation ability reconstruct.For details, reference can be made to first embodiment of the invention, the embodiment of the present invention is not made this in detailIt discusses.
The embodiment of the present invention can at least bring it is below the utility model has the advantages that
1. the embodiment of the present invention supports the Heterogeneous Computings resources such as CPU, GPU, FPGA, IPU, the storage resources such as SSD, HDD pointCloth is interconnected between each memory resource pool by remote PCI e crossbar switch, storage resource in each calculating and memory resource poolEach CPU, GPU in pond, FPGA, IPU, it is interconnected by PCIe multibus and crossbar switch between storage resource, so that hardware resourceCan be decoupling, improve the dynamic configurability of network；
2. the present invention provides the stratification interconnection scheme of PCIe crossbar switch and remote PCI e crossbar switch in a kind of pond, lead toIt crosses and is configured in pond with remote PCI e crossbar switch, can be needed according to the performance of data center server for application distribution justWhen the calculate node and memory node of quantity, each storage resource that calculates can be managed collectively, and flexible allocation is each hard by reconstructingConnection between part resource can effectively promote hardware frame so that hardware structure supports computing capability and concatenation ability to construct on demandThe performance and efficiency of structure；
3. the present invention provides a kind of remote PCI e crossbar configuration scheme of hybrid switching mechanism, while packet being supported to hand overIt changes, circuit switching and virtual channel three kinds of exchanged forms of exchange, wherein virtual channel exchange can directly pass through crossbar switch, configureIt is more easy, while supporting to use packet switch or circuit switching in link conflict, guarantee efficiently to be based on PCIe multiple busWith the communication link of crossbar switch.
It should be noted that PCIe crossbar switch described in the embodiment of the present invention can be realized by PCIeswitch chip,Crossbar switch function can also be realized by FPGA.
Algorithm and display are not inherently related to any particular computer, virtual system, or other device provided herein.Various general-purpose systems can also be used together with teachings based herein.As described above, it constructs required by this kind of systemStructure be obvious.In addition, the present invention is also not directed to any particular programming language.It should be understood that can use variousProgramming language realizes summary of the invention described herein, and the description done above to language-specific is to disclose this hairBright preferred forms.
In the instructions provided here, numerous specific details are set forth.It is to be appreciated, however, that implementation of the inventionExample can be practiced without these specific details.In some instances, well known method, structure is not been shown in detailAnd technology, so as not to obscure the understanding of this specification.
Similarly, it should be understood that in order to simplify the disclosure and help to understand one or more of the various inventive aspects,Above in the description of exemplary embodiment of the present invention, each feature of the invention is grouped together into single implementation sometimesIn example, figure or descriptions thereof.However, the disclosed method should not be interpreted as reflecting the following intention: i.e. required to protectShield the present invention claims features more more than feature expressly recited in each claim.More precisely, as followingClaims reflect as, inventive aspect is all features less than single embodiment disclosed above.Therefore,Thus the claims for following specific embodiment are expressly incorporated in the specific embodiment, wherein each claim itselfAll as a separate embodiment of the present invention.
Those skilled in the art will understand that can be carried out adaptively to the module in the equipment in embodimentChange and they are arranged in one or more devices different from this embodiment.It can be the module or list in embodimentMember or component are combined into a module or unit or component, and furthermore they can be divided into multiple submodule or subelement orSub-component.Other than such feature and/or at least some of process or unit exclude each other, it can use anyCombination is to all features disclosed in this specification (including adjoint claim, abstract and attached drawing) and so disclosedAll process or units of what method or apparatus are combined.Unless expressly stated otherwise, this specification is (including adjoint powerBenefit require, abstract and attached drawing) disclosed in each feature can carry out generation with an alternative feature that provides the same, equivalent, or similar purposeIt replaces.
In addition, it will be appreciated by those of skill in the art that although some embodiments described herein include other embodimentsIn included certain features rather than other feature, but the combination of the feature of different embodiments mean it is of the inventionWithin the scope of and form different embodiments.For example, in the following claims, embodiment claimed is appointedMeaning one of can in any combination mode come using.
Various component embodiments of the invention can be implemented in hardware, or to run on one or more processorsSoftware module realize, or be implemented in a combination thereof.It will be understood by those of skill in the art that can be used in practiceMicroprocessor or digital signal processor (DSP) realize that Distributed File System Data according to an embodiment of the present invention importsThe some or all functions of some or all components in device.The present invention is also implemented as being retouched here for executingThe some or all device or device programs (for example, computer program and computer program product) for the method stated.It is such to realize that program of the invention can store on a computer-readable medium, or can have one or more signalForm.Such signal can be downloaded from an internet website to obtain, be perhaps provided on the carrier signal or with it is any itsHe provides form.
It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and abilityField technique personnel can be designed alternative embodiment without departing from the scope of the appended claims.In the claims,Any reference symbol between parentheses should not be configured to limitations on claims.Word "comprising" does not exclude the presence of notElement or step listed in the claims.Word "a" or "an" located in front of the element does not exclude the presence of multiple suchElement.The present invention can be by means of including the hardware of several different elements and being come by means of properly programmed computer realIt is existing.In the unit claims listing several devices, several in these devices can be through the same hardware branchTo embody.The use of word first, second, and third does not indicate any sequence.These words can be explained and be run after fameClaim.