相关申请的交叉引用CROSS-REFERENCE TO RELATED APPLICATIONS
本申请要求于2023年04月14日提交中国专利局，申请号为2023104033427，申请名称为“一种服务器、异构设备及其数据处理装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。This application claims priority to a Chinese patent application filed with the China Patent Office on April 14, 2023, with application number 2023104033427 and application name “A server, heterogeneous device and data processing device thereof”, the entire contents of which are incorporated by reference in this application.
本申请涉及数据处理技术领域，特别是涉及一种服务器、异构设备及其数据处理装置。The present application relates to the field of data processing technology, and in particular to a server, a heterogeneous device and a data processing device thereof.
目前，随着云计算的不断发展，数据中心系统的CPU(Central Processing Unit，中央处理器)、GPU(Graphics Processing Unit，图形处理器)、FPGA(Field Programmable Gate Array，现场可编程门阵列)等硬件资源越来越多，提高数据在数据中心的传输效率、数据处理任务的合理卸载，显得尤为重要。FPGA是一种具有高可编程性能的多元异构芯片，内部具有丰富的硬件资源，可以使用这些资源实现各种数据处理引擎、复杂总线协议和网络协议等。GPU则是一种专用的图形处理芯片，目前已经广泛应用于人工智能计算等领域，是一种重要的计算芯片。FPGA卡和GPU卡均属于PCIe(Peripheral Component Interconnect express，高速串行计算机扩展总线标准)卡，也可以称为扩展卡。目前，可以在主机的PCIe插槽中插入FPGA卡、GPU卡等PCIe卡实现数据处理，PCIe卡之间的通信，需要主机的CPU、内存等部件的参与，CPU开销较大，通信延迟也较高。At present, with the continuous development of cloud computing, the data center system has more and more hardware resources such as CPU (Central Processing Unit), GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), etc. It is particularly important to improve the data transmission efficiency in the data center and the reasonable unloading of data processing tasks. FPGA is a multi-heterogeneous chip with high programmable performance. It has rich hardware resources inside, and these resources can be used to implement various data processing engines, complex bus protocols and network protocols. GPU is a dedicated graphics processing chip, which has been widely used in fields such as artificial intelligence computing and is an important computing chip. FPGA cards and GPU cards are both PCIe (Peripheral Component Interconnect express, a high-speed serial computer expansion bus standard) cards, which can also be called expansion cards. At present, PCIe cards such as FPGA cards and GPU cards can be inserted into the PCIe slot of the host to realize data processing. The communication between PCIe cards requires the participation of the host's CPU, memory and other components, and the CPU overhead is large, and the communication delay is also high.
综上所述，如何有效地利用扩展卡实现数据处理，降低与CPU的耦合度，降低对CPU的资源占用，是目前本领域技术人员急需解决的技术问题。In summary, how to effectively utilize the expansion card to realize data processing, reduce the coupling degree with the CPU, and reduce the resource occupation of the CPU is a technical problem that technicians in this field urgently need to solve.
发明内容Summary of the invention
本申请的目的是提供一种服务器、异构设备及其数据处理装置，以有效地利用扩展卡实现数据处理，降低与CPU的耦合度，降低对CPU的资源占用。The purpose of the present application is to provide a server, a heterogeneous device and a data processing device thereof, so as to effectively utilize an expansion card to realize data processing, reduce the coupling degree with a CPU, and reduce the resource occupation of the CPU.
为解决上述技术问题，本申请提供如下技术方案：In order to solve the above technical problems, this application provides the following technical solutions:
一种数据处理装置，包括：被设置为向各个扩展卡插槽供电的第一主板，被设置为进行数据处理的第一扩展卡和第二扩展卡；A data processing device comprises: a first mainboard configured to supply power to each expansion card slot, a first expansion card and a second expansion card configured to perform data processing;
第一主板中设置有至少1组插槽对，每组插槽对中包括2个扩展卡插槽，且每组插槽对中的2个扩展卡插槽的收发差分线通过第一主板的电路进行连接；The first mainboard is provided with at least one slot pair, each slot pair includes two expansion card slots, and the transceiver differential lines of the two expansion card slots in each slot pair are connected through the circuit of the first mainboard;
第一扩展卡与第一主板的第一扩展卡插槽连接，第二扩展卡与第一主板的第二扩展卡插槽连接，以通过第一主板的电路完成第一扩展卡与第二扩展卡之间的通信；The first expansion card is connected to the first expansion card slot of the first mainboard, and the second expansion card is connected to the second expansion card slot of the first mainboard, so as to complete the communication between the first expansion card and the second expansion card through the circuit of the first mainboard;
其中，第一主板的第一扩展卡插槽和第二扩展卡插槽为第一主板中的1组插槽对。The first expansion card slot and the second expansion card slot of the first mainboard are a slot pair in the first mainboard.
可选的，还包括：被设置为进行数据处理的第三扩展卡，并且第三扩展卡和第二扩展卡均至少具有2个扩展卡接口；Optionally, the system further includes: a third expansion card configured to perform data processing, and both the third expansion card and the second expansion card have at least two expansion card interfaces;
第三扩展卡的第一扩展卡接口与第一主板的第三扩展卡插槽连接，第二扩展卡的第一扩展卡接口与第一主板的第二扩展卡插槽连接，第三扩展卡的第二扩展卡接口通过外接线缆与第二扩展卡的第二扩展卡接口通信连接。The first expansion card interface of the third expansion card is connected to the third expansion card slot of the first mainboard, the first expansion card interface of the second expansion card is connected to the second expansion card slot of the first mainboard, and the second expansion card interface of the third expansion card is communicatively connected to the second expansion card interface of the second expansion card via an external cable.
可选的，还包括：被设置为进行数据处理的第四扩展卡，第四扩展卡与第一主板的第四扩展卡插槽连接；Optionally, it further includes: a fourth expansion card configured to perform data processing, the fourth expansion card being connected to a fourth expansion card slot of the first mainboard;
其中，第一主板的第三扩展卡插槽和第四扩展卡插槽为第一主板中的1组插槽对。The third expansion card slot and the fourth expansion card slot of the first mainboard form a slot pair in the first mainboard.
可选的，第一扩展卡，第二扩展卡，第三扩展卡以及第四扩展卡均为现场可编程门阵列卡。Optionally, the first expansion card, the second expansion card, the third expansion card and the fourth expansion card are all field programmable gate array cards.
可选的，还包括：与第一主板连接，被设置为进行数据处理的K张现场可编程门阵列卡，并且通过第一主板的电路和外接线缆，K张现场可编程门阵列卡依次通信连接，K为正整数。Optionally, it also includes: K field programmable gate array cards connected to the first mainboard and configured to perform data processing, and the K field programmable gate array cards are communicatively connected in sequence through the circuit of the first mainboard and external cables, and K is a positive integer.
可选的，K张现场可编程门阵列卡中的一张现场可编程门阵列卡通过外接线缆与第四扩展卡通信连接，K张现场可编程门阵列卡中的一张现场可编程门阵列卡通过外接线缆与第一扩展卡通信连接，以构成第一主板中的各张现场可编程门阵列卡的环形通信。Optionally, one of the K field programmable gate array cards is communicatively connected to the fourth expansion card via an external cable, and one of the K field programmable gate array cards is communicatively connected to the first expansion card via an external cable to form a ring communication of the field programmable gate array cards in the first mainboard.
可选的，第一扩展卡，第二扩展卡，第三扩展卡以及第四扩展卡均通过自身的光模块接口与交换机连接，以与远程管理平台进行通信。Optionally, the first expansion card, the second expansion card, the third expansion card and the fourth expansion card are all connected to the switch through their own optical module interfaces to communicate with the remote management platform.
可选的，第一扩展卡和第四扩展卡均为图形处理器卡；第二扩展卡和第三扩展卡均为现场可编程门阵列卡。Optionally, the first expansion card and the fourth expansion card are both graphics processor cards; and the second expansion card and the third expansion card are both field programmable gate array cards.
可选的，第二扩展卡在数据处理过程中，通过自身的直接数据存取模块进行自身的不同接口之间的数据传输；Optionally, during the data processing, the second expansion card transmits data between different interfaces of the second expansion card through its own direct data access module;
第三扩展卡在数据处理过程中，通过自身的直接数据存取模块进行自身的不同接口之间的数据传输。During the data processing, the third expansion card transmits data between different interfaces of the third expansion card through its own direct data access module.
可选的，还包括：被设置为进行数据处理的第五扩展卡和第六扩展卡；Optionally, it further includes: a fifth expansion card and a sixth expansion card configured to perform data processing;
第五扩展卡与第一主板的第五扩展卡插槽连接，第六扩展卡与第一主板的第六扩展卡插槽连接，以通过第一主板的电路完成第五扩展卡与第六扩展卡之间的通信；The fifth expansion card is connected to the fifth expansion card slot of the first mainboard, and the sixth expansion card is connected to the sixth expansion card slot of the first mainboard, so as to complete the communication between the fifth expansion card and the sixth expansion card through the circuit of the first mainboard;
其中，第五扩展卡为图形处理器卡，第六扩展卡为现场可编程门阵列卡；第一主板的第五扩展卡插槽和第六扩展卡插槽为第一主板中的1组插槽对。Among them, the fifth expansion card is a graphics processor card, and the sixth expansion card is a field programmable gate array card; the fifth expansion card slot and the sixth expansion card slot of the first mainboard are a group of slot pairs in the first mainboard.
可选的，还包括：被设置为进行数据处理的第七扩展卡和第八扩展卡；Optionally, it further includes: a seventh expansion card and an eighth expansion card configured to perform data processing;
第七扩展卡与第一主板的第七扩展卡插槽连接，第八扩展卡与第一主板的第八扩展卡插槽连接，以通过第一主板的电路完成第七扩展卡与第八扩展卡之间的通信；The seventh expansion card is connected to the seventh expansion card slot of the first mainboard, and the eighth expansion card is connected to the eighth expansion card slot of the first mainboard, so as to complete the communication between the seventh expansion card and the eighth expansion card through the circuit of the first mainboard;
其中，第七扩展卡为图形处理器卡，第八扩展卡为现场可编程门阵列卡；第一主板的第七扩展卡插槽和第八扩展卡插槽为第一主板中的1组插槽对。Among them, the seventh expansion card is a graphics processor card, and the eighth expansion card is a field programmable gate array card; the seventh expansion card slot and the eighth expansion card slot of the first mainboard are a group of slot pairs in the first mainboard.
可选的，第六扩展卡和第八扩展卡均至少具有2个扩展卡接口；Optionally, the sixth expansion card and the eighth expansion card each have at least two expansion card interfaces;
第六扩展卡的第一扩展卡接口与第一主板的第六扩展卡插槽连接，第八扩展卡的第一扩展卡接口与第一主板的第八扩展卡插槽连接，第六扩展卡的第二扩展卡接口通过外接线缆与第八扩展卡的第二扩展卡接口通信连接。The first expansion card interface of the sixth expansion card is connected to the sixth expansion card slot of the first mainboard, the first expansion card interface of the eighth expansion card is connected to the eighth expansion card slot of the first mainboard, and the second expansion card interface of the sixth expansion card is communicatively connected to the second expansion card interface of the eighth expansion card via an external cable.
可选的，第八扩展卡和第二扩展卡均至少具有3个扩展卡接口；第八扩展卡的第三扩展卡接口通过外接线缆与第二扩展卡的第三扩展卡接口通信连接。Optionally, the eighth expansion card and the second expansion card each have at least three expansion card interfaces; and the third expansion card interface of the eighth expansion card is communicatively connected to the third expansion card interface of the second expansion card via an external cable.
可选的，还包括第二主板，第九扩展卡以及第十扩展卡；Optionally, it also includes a second mainboard, a ninth expansion card and a tenth expansion card;
第二主板中设置有至少1组插槽对，每组插槽对中包括2个扩展卡插槽，且每组插槽对中的2个扩展卡插槽的收发差分线通过第二主板的电路进行连接；The second mainboard is provided with at least one slot pair, each slot pair includes two expansion card slots, and the transceiver differential lines of the two expansion card slots in each slot pair are connected through the circuit of the second mainboard;
第九扩展卡与第二主板的第一扩展卡插槽连接，第十扩展卡与第二主板的第二扩展卡插槽连接，以通过第二主板的电路完成第九扩展卡与第十扩展卡之间的通信；第二主板的第一扩展卡插槽和第二扩展卡插槽为第二主板中的1组插槽对；The ninth expansion card is connected to the first expansion card slot of the second mainboard, and the tenth expansion card is connected to the second expansion card slot of the second mainboard, so as to complete the communication between the ninth expansion card and the tenth expansion card through the circuit of the second mainboard; the first expansion card slot and the second expansion card slot of the second mainboard are a pair of slots in the second mainboard;
第三扩展卡至少具有3个扩展卡接口，第九扩展卡至少具有2个扩展卡接口；The third expansion card has at least three expansion card interfaces, and the ninth expansion card has at least two expansion card interfaces;
第九扩展卡的第一扩展卡接口与第一主板的第九扩展卡插槽连接，第三扩展卡的第三扩展卡接口通过外接线缆与第九扩展卡的第二扩展卡接口通信连接。The first expansion card interface of the ninth expansion card is connected to the ninth expansion card slot of the first mainboard, and the third expansion card interface of the third expansion card is communicatively connected to the second expansion card interface of the ninth expansion card through an external cable.
可选的，第九扩展卡为现场可编程门阵列卡，第十扩展卡为图形处理器卡。Optionally, the ninth expansion card is a field programmable gate array card, and the tenth expansion card is a graphics processor card.
可选的，还包括：Optionally, also include:
设置在第一主板上，被设置为进行第一主板的散热的散热装置。A heat dissipation device is arranged on the first mainboard and is configured to dissipate heat of the first mainboard.
可选的，还包括：Optionally, also include:
设置在第一主板上的监控管理装置，被设置为监控与第一主板连接的各张扩展卡的状态。The monitoring management device arranged on the first mainboard is arranged to monitor the status of each expansion card connected to the first mainboard.
可选的，监控管理装置还被设置为：Optionally, the monitoring management device is further configured to:
当监测到任意一张扩展卡的状态异常时，向基板管理控制器输出日志信息。When the status of any expansion card is detected to be abnormal, log information is output to the baseboard management controller.
可选的，针对任意1张扩展卡，当该扩展卡为现场可编程门阵列卡时，在远程管理平台的控制下，该扩展卡通过初始化操作以及地址映射操作，建立与自身相连的扩展卡的通信连接。Optionally, for any expansion card, when the expansion card is a field programmable gate array card, under the control of the remote management platform, the expansion card establishes a communication connection with the expansion card connected to itself through an initialization operation and an address mapping operation.
一种异构设备，包括如上述的数据处理装置。A heterogeneous device comprises the data processing apparatus as described above.
一种服务器，包括如上述的异构设备。A server includes the heterogeneous device as described above.
应用本申请实施例所提供的技术方案，第一扩展卡和第二扩展卡均被设置为进行数据处理，且第一主板中设置有至少1组插槽对，每组插槽对中包括2个扩展卡插槽，第一扩展卡与第一主板的第一扩展卡插槽连接，第二扩展卡与第一主板的第二扩展卡插槽连接，即第一扩展卡和第二扩展卡设置在主板的1组插槽对当中。本申请的方案中，在第一主板的每组插槽对中，将每组插槽对中的2个扩展卡插槽的收发差分线通过第一主板的电路进行连接，因此，第一扩展卡和第二扩展卡可以直接通过第一主板的电路完成相互之间的通信，换句话说，本申请的方案中，第一扩展卡和第二扩展卡之间的通信不需要CPU的参数，使得本申请的方案实现了扩展卡与CPU的解耦。本申请的方案中第一主板上也无需设置CPU，不会出现占用CPU资源的情况。Applying the technical solution provided by the embodiment of the present application, the first expansion card and the second expansion card are both configured to perform data processing, and at least one slot pair is provided in the first mainboard, each slot pair includes two expansion card slots, the first expansion card is connected to the first expansion card slot of the first mainboard, and the second expansion card is connected to the second expansion card slot of the first mainboard, that is, the first expansion card and the second expansion card are arranged in one slot pair of the mainboard. In the solution of the present application, in each slot pair of the first mainboard, the transceiver differential lines of the two expansion card slots in each slot pair are connected through the circuit of the first mainboard, so the first expansion card and the second expansion card can directly communicate with each other through the circuit of the first mainboard. In other words, in the solution of the present application, the communication between the first expansion card and the second expansion card does not require the parameters of the CPU, so that the solution of the present application realizes the decoupling of the expansion card and the CPU. In the solution of the present application, there is no need to set the CPU on the first mainboard, and there will be no situation of occupying CPU resources.
综上，本申请的方案可以有效地利用扩展卡实现数据处理，且实现了与CPU的解耦，不需要占用CPU的资源。In summary, the solution of the present application can effectively utilize the expansion card to realize data processing, and realizes decoupling from the CPU without occupying CPU resources.
为了更清楚地说明本申请实施例或相关技术中的技术方案，下面将对实施例或相关技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
图1为本申请中数据处理装置的第一结构示意图；FIG1 is a first structural diagram of a data processing device in the present application;
图2为本申请中数据处理装置的第二结构示意图；FIG2 is a second structural diagram of a data processing device in the present application;
图3为本申请中数据处理装置的第三结构示意图；FIG3 is a third structural diagram of a data processing device in the present application;
图4为本申请中数据处理装置的第四结构示意图；FIG4 is a fourth structural diagram of a data processing device in the present application;
图5为本申请中数据处理装置的第五结构示意图；FIG5 is a fifth structural diagram of a data processing device in the present application;
图6为本申请中数据处理装置的第六结构示意图；FIG6 is a sixth structural diagram of a data processing device in the present application;
图7为本申请中一种可选实施方式中的FPGA卡的结构示意图；FIG7 is a schematic diagram of the structure of an FPGA card in an optional implementation manner of the present application;
图8为本申请中数据处理装置的第七结构示意图；FIG8 is a seventh structural diagram of a data processing device in the present application;
图9为本申请中数据处理装置的第八结构示意图；FIG9 is an eighth structural diagram of a data processing device in the present application;
图10为本申请中数据处理装置的第九结构示意图；FIG10 is a ninth structural diagram of a data processing device in the present application;
图11为本申请中数据处理装置的第十结构示意图；FIG11 is a tenth structural diagram of a data processing device in the present application;
图12为本申请中数据处理装置的第十一结构示意图；FIG12 is a schematic diagram of the eleventh structure of the data processing device in the present application;
图13为本申请中数据处理装置的第十二结构示意图。FIG. 13 is a twelfth structural diagram of the data processing device in the present application.
本申请的核心是提供一种数据处理装置，可以有效地利用扩展卡实现数据处理，且实现了与CPU的解耦，不需要占用CPU的资源。The core of this application is to provide a data processing device that can effectively utilize expansion cards to realize data processing, and realize decoupling from the CPU without occupying CPU resources.
为了使本技术领域的人员更好地理解本申请方案，下面结合附图和可选实施方式对本申请作进一步的详细说明。显然，所描述的实施例仅仅是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and optional implementation methods. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are protected by the present application. scope.
请参考图1，图1为本申请中一种数据处理装置的结构示意图，该数据处理装置包括：被设置为向各个扩展卡插槽供电的第一主板20，被设置为进行数据处理的第一扩展卡11和第二扩展卡12；Please refer to FIG. 1 , which is a schematic diagram of the structure of a data processing device in the present application, the data processing device comprising: a first mainboard 20 configured to supply power to each expansion card slot, a first expansion card 11 and a second expansion card 12 configured to perform data processing;
第一主板20中设置有至少1组插槽对，每组插槽对中包括2个扩展卡插槽，且每组插槽对中的2个扩展卡插槽的收发差分线通过第一主板20的电路进行连接；The first mainboard 20 is provided with at least one slot pair, each slot pair includes two expansion card slots, and the transceiver differential lines of the two expansion card slots in each slot pair are connected through the circuit of the first mainboard 20;
第一扩展卡11与第一主板20的第一扩展卡插槽连接，第二扩展卡12与第一主板20的第二扩展卡插槽连接，以通过第一主板20的电路完成第一扩展卡11与第二扩展卡12之间的通信；The first expansion card 11 is connected to the first expansion card slot of the first mainboard 20, and the second expansion card 12 is connected to the second expansion card slot of the first mainboard 20, so as to complete the communication between the first expansion card 11 and the second expansion card 12 through the circuit of the first mainboard 20;
其中，第一主板20的第一扩展卡插槽和第二扩展卡插槽为第一主板20中的1组插槽对。The first expansion card slot and the second expansion card slot of the first mainboard 20 are a slot pair in the first mainboard 20 .
可选的，传统的PCIe卡之间的通信，需要主机的CPU、内存等部件的参与，CPU开销较大，通信延迟也较高。以GPU卡为例，传统的一种方案中，GPU通过DSM(Direct Shared Memory，直接共享存储)的方式将数据从GPU显存拷贝到主机内存，再将数据从主机内存拷贝到另一GPU显存，该过程需要占用CPU资源，且数据通路延时较高，传输效率较低。还有的传统方案是在主机的同一PCIe域下的两个GPU设备之间，通过主机的PCIe Chip Set(PCIe芯片组)实现直接互访GPU显存，从而无需将数据拷贝到主机内存里，但是由于PCIe芯片组通常集成在CPU中，导致这样的方案仍然会占用CPU资源，即GPU卡与服务器的CPU紧耦合，且通信也仅限于单节点内的GPU卡。GPU Direct RDMA(Remote Direct Memory Access，远程直接数据存取)是GPU之间远程直接内存访问的技术，需要利用物理网卡和传输网络实现GPU之间显存数据的交互，该过程仍然需要器CPU的帮助，会占用CPU资源，GPU与服务器的CPU之间仍是通过PCIe连接的紧耦合关系。Alternatively, the communication between traditional PCIe cards requires the participation of the host's CPU, memory and other components, resulting in high CPU overhead and high communication latency. Taking the GPU card as an example, in a traditional solution, the GPU copies data from the GPU memory to the host memory through DSM (Direct Shared Memory), and then copies the data from the host memory to another GPU memory. This process requires CPU resources, and the data path latency is high, and the transmission efficiency is low. Another traditional solution is to achieve direct mutual access to the GPU memory between two GPU devices in the same PCIe domain of the host through the host's PCIe Chip Set (PCIe chipset), so there is no need to copy the data to the host memory. However, since the PCIe chipset is usually integrated in the CPU, such a solution will still occupy CPU resources, that is, the GPU card is tightly coupled with the server's CPU, and the communication is limited to the GPU card in a single node. GPU Direct RDMA (Remote Direct Memory Access) is a technology for remote direct memory access between GPUs. It requires the use of a physical network card and a transmission network to realize the interaction of video memory data between GPUs. This process still requires the help of the server CPU and will occupy CPU resources. The GPU and the server's CPU are still tightly coupled through a PCIe connection.
而本申请的各个实施例中都不需要占用CPU的资源。However, each embodiment of the present application does not need to occupy CPU resources.
本申请的第一主板20可以向第一主板20中的各个扩展卡插槽供电，本申请描述的扩展卡可以均为PCIe扩展卡，因此本申请后文中，均以PCIe为例进行说明，也就是说，后文各个实施方式中描述的扩展卡插槽，可以均为PCIe插槽。后文中描述的第一PCIe卡，便是第一扩展卡11，相应的，后文描述的第二PCIe卡，便是第二扩展卡12。同样的，后文实施方式中描述的第三PCIe卡至第十PCIe卡，依次指的是第三扩展卡13至第十扩展卡10。The first mainboard 20 of the present application can supply power to each expansion card slot in the first mainboard 20. The expansion cards described in the present application can all be PCIe expansion cards. Therefore, PCIe is used as an example for explanation in the following text. That is to say, the expansion card slots described in each implementation method in the following text can all be PCIe slots. The first PCIe card described in the following text is the first expansion card 11, and correspondingly, the second PCIe card described in the following text is the second expansion card 12. Similarly, the third PCIe card to the tenth PCIe card described in the implementation methods in the following text refer to the third expansion card 13 to the tenth expansion card 10, respectively.
PCIe是一种高速串行计算机扩展总线标准，属于高速串行点对点双通道高带宽传输。需要说明的是，本申请的扩展卡均为PCIe扩展卡，指的是硬件链路为PCIe总线标准，至于软件层面的通信协议，可以是PCI e协议或者是其他协议。PCIe is a high-speed serial computer expansion bus standard, which belongs to high-speed serial point-to-point dual-channel high-bandwidth transmission. It should be noted that the expansion cards in this application are all PCIe expansion cards, which means that the hardware link is the PCIe bus standard. As for the communication protocol at the software level, it can be PCI e protocol or other protocols.
PCIe作为一种高速串行计算机扩展总线标准，属于高速串行点对点双通道高带宽传输。PCIe总线使用端到端的连接方式，在一条PCIe链路的两端各连接一个设备，所连接的这两个设备互为数据发送端和数据接收端。PCIe总线除了总线链路外，还具有多个层次，发送端发送数据时将通过这些层次，而接收端接收数据时也使用这些层次，PCIe总线使用的层次结构与网络协议栈较为类似。PCIe is a high-speed serial computer expansion bus standard that belongs to high-speed serial point-to-point dual-channel high-bandwidth transmission. The PCIe bus uses an end-to-end connection method, with a device connected at each end of a PCIe link. The two connected devices are the data transmitter and receiver of each other. In addition to the bus link, the PCIe bus also has multiple layers. The transmitter will pass through these layers when sending data, and the receiver will also use these layers when receiving data. The hierarchical structure used by the PCIe bus is similar to the network protocol stack.
第一主板20可以向第一主板20中的各个PCIe插槽供电，实际应用中，第一主板20可以通过供电模块来实现供电，例如供电模块可以接收外部电能，并进行降压，滤波等操作，从而输出所需要的电平等级至第一主板20自身的各个PCIe插槽。图1的实施方式中，便在第一主板20中示出了电源模块。The first mainboard 20 can supply power to each PCIe slot in the first mainboard 20. In practical applications, the first mainboard 20 can be powered by a power supply module. For example, the power supply module can receive external power and perform operations such as voltage reduction and filtering, thereby outputting the required level to each PCIe slot of the first mainboard 20. In the embodiment of FIG1 , a power supply module is shown in the first mainboard 20.
本申请的方案中，第一主板20上不需要设置CPU、内存等部件，便可以实现第一扩展卡11和第二扩展卡12之间的数据交互，同样的，后文的实施方式中，也仍旧不需要CPU、内存等部件，便可以实现相应扩展卡之间直接或者间接的数据交互，因此，本申请的各个附图中，均未示出CPU、内存，但可以理解的是，如果第一主板20上原本就设置有CPU、内存，并不会影响本申请的实施。In the solution of the present application, the first mainboard 20 does not need to be provided with components such as a CPU and a memory, so that data interaction between the first expansion card 11 and the second expansion card 12 can be achieved. Similarly, in the implementation methods described below, components such as a CPU and a memory are still not required to achieve direct or indirect data interaction between the corresponding expansion cards. Therefore, the CPU and the memory are not shown in the various drawings of the present application, but it can be understood that if the first mainboard 20 is originally provided with a CPU and a memory, it will not affect the implementation of the present application.
在图1中，第一主板20中设置了1组插槽对，这1组插槽对由第一主板20的第一PCIe插槽和第一主板20的第二PCIe插槽构成，其他实施方式中可以设置有更多组的插槽对，例如图2中，第一主板20中设置了4组PCIe插槽对，当然，图2中并未示出任何与第一主板20连接的PCIe卡。In Figure 1, a group of slot pairs is set in the first motherboard 20, and the group of slot pairs is composed of a first PCIe slot of the first motherboard 20 and a second PCIe slot of the first motherboard 20. In other implementations, more groups of slot pairs may be set. For example, in Figure 2, four groups of PCIe slot pairs are set in the first motherboard 20. Of course, any PCIe card connected to the first motherboard 20 is not shown in Figure 2.
第一主板20中，每组插槽对由2个PCIe插槽构成，并且在每组插槽对中，这2个PCIe插槽的收发差分线通过第一主板20的电路进行连接，即这2个PCIe插槽的收发差分线通过第一主板20实现了直连，例如图1中第一PCIe卡和第二PCIe卡均为X16的PCIe卡，第一PCIe插槽的X16的收发差分线，与第二PCIe插槽的X16的收发差分线直连。可以理解的是，进行2个PCIe插槽的收发差分线的连接，是将一边的输入/输出连接另一边的输出/输入。In the first motherboard 20, each slot pair is composed of two PCIe slots, and in each slot pair, the transceiver differential lines of the two PCIe slots are connected through the circuit of the first motherboard 20, that is, the transceiver differential lines of the two PCIe slots are directly connected through the first motherboard 20, for example, in FIG. The first PCIe card and the second PCIe card are both X16 PCIe cards, and the X16 transceiver differential line of the first PCIe slot is directly connected to the X16 transceiver differential line of the second PCIe slot. It can be understood that the transceiver differential lines of the two PCIe slots are connected by connecting the input/output of one side to the output/input of the other side.
由于第一PCIe卡与第一主板20的第一PCIe插槽连接，第二PCIe卡与第一主板20的第二PCIe插槽连接，第一PCIe插槽和第二PCIe插槽是第一主板20中的1组插槽对，因此，通过第一主板20的电路，便实现了第一PCIe卡与第二PCIe卡之间的通信，即在不需要CPU、内存的参与下，实现了第一PCIe卡与第二PCIe卡之间的数据交互。Since the first PCIe card is connected to the first PCIe slot of the first motherboard 20, and the second PCIe card is connected to the second PCIe slot of the first motherboard 20, the first PCIe slot and the second PCIe slot are a pair of slots in the first motherboard 20. Therefore, communication between the first PCIe card and the second PCIe card is realized through the circuit of the first motherboard 20, that is, data interaction between the first PCIe card and the second PCIe card is realized without the participation of the CPU and memory.
第一PCIe卡和第二PCIe卡均可以进行数据处理，当然，数据处理流程，可以根据实际算法来设定，并且可以在远程管理平台中进行调整。例如某个场合中，是由第一PCIe卡通过网络接收远程管理平台的输入数据，第一PCIe卡进行步骤一的计算之后，将计算结果通过第一主板20的电路，即通过收发差分线传输至第二PCIe卡，由第二PCIe卡进行步骤二的计算，并将结果反馈回远程管理平台。当然，该例子仅是简单的举例，实际应用中，第一PCIe卡和第二PCIe卡可以根据需要设计更多轮的数据交互，第一PCIe卡和第二PCIe卡均可以接收远程管理平台的数据并受远程管理平台的控制。Both the first PCIe card and the second PCIe card can perform data processing. Of course, the data processing process can be set according to the actual algorithm and can be adjusted in the remote management platform. For example, in a certain situation, the first PCIe card receives the input data of the remote management platform through the network. After the first PCIe card performs the calculation of step one, the calculation result is transmitted to the second PCIe card through the circuit of the first motherboard 20, that is, through the transceiver differential line. The second PCIe card performs the calculation of step two and feeds the result back to the remote management platform. Of course, this example is only a simple example. In actual applications, the first PCIe card and the second PCIe card can design more rounds of data interaction as needed. The first PCIe card and the second PCIe card can both receive data from the remote management platform and be controlled by the remote management platform.
在本申请的一种可选实施方式中，可参阅图3，还可以包括：被设置为进行数据处理的第三扩展卡13，并且第三扩展卡13和第二扩展卡12均至少具有2个扩展卡接口；In an optional implementation of the present application, referring to FIG. 3 , it may further include: a third expansion card 13 configured to perform data processing, and both the third expansion card 13 and the second expansion card 12 have at least two expansion card interfaces;
第三扩展卡13的第一扩展卡接口与第一主板20的第三扩展卡插槽连接，第二扩展卡12的第一扩展卡接口与第一主板20的第二扩展卡插槽连接，第三扩展卡13的第二扩展卡接口通过外接线缆与第二扩展卡12的第二扩展卡接口通信连接。The first expansion card interface of the third expansion card 13 is connected to the third expansion card slot of the first mainboard 20, the first expansion card interface of the second expansion card 12 is connected to the second expansion card slot of the first mainboard 20, and the second expansion card interface of the third expansion card 13 is communicatively connected to the second expansion card interface of the second expansion card 12 via an external cable.
该种实施方式中，在第一主板20的第三扩展卡插槽中插入了第三扩展卡13，第三扩展卡13也可以进行数据处理。并且，第三扩展卡13还需要与第二扩展卡12通信连接，因此，第三扩展卡13和第二扩展卡12均至少具有2个扩展卡接口。In this embodiment, the third expansion card 13 is inserted into the third expansion card slot of the first mainboard 20, and the third expansion card 13 can also perform data processing. In addition, the third expansion card 13 also needs to communicate with the second expansion card 12, so the third expansion card 13 and the second expansion card 12 both have at least two expansion card interfaces.
图3中，用1条曲线表示连接第三扩展卡13和第二扩展卡12的线缆，仍以PCIe为例，即第三PCIe卡和第二PCIe卡通过外接线缆实现通信连接。此外需要说明的是，两张PCIe卡之间进行通信时，需要划分二者的主从，因此，该种实施方式中，例如可以将第三PCIe卡的第二PCIe接口设置为Root Complex模式，使得第三PCIe卡的第二PCIe接口作为主设备接口，而将第二PCIe卡的第二PCIe接口设置为Endpoint模式，使得第二PCIe卡的第二PCIe接口作为从设备接口，实现第三PCIe卡和第二PCIe卡之间的主从通信。In FIG3 , a curve is used to represent the cable connecting the third expansion card 13 and the second expansion card 12. PCIe is still used as an example, that is, the third PCIe card and the second PCIe card are connected to each other through an external cable. In addition, it should be noted that when two PCIe cards communicate with each other, it is necessary to divide the two into master and slave. Therefore, in this implementation, for example, the second PCIe interface of the third PCIe card can be set to Root Complex mode, so that the second PCIe interface of the third PCIe card is used as the master device interface, and the second PCIe interface of the second PCIe card is set to Endpoint mode, so that the second PCIe interface of the second PCIe card is used as the slave device interface, so as to realize the master-slave communication between the third PCIe card and the second PCIe card.
第二PCIe卡的第一PCIe接口例如可以设置Root Complex模式，使得第二PCIe卡的第一PCIe接口作为主设备接口，第一PCIe卡被设置为连接第一PCIe插槽的接口则可以设置Endpoint模式，实现第二PCIe卡和第一PCIe卡之间的主从通信。For example, the first PCIe interface of the second PCIe card can be set to Root Complex mode, so that the first PCIe interface of the second PCIe card serves as the master device interface. The first PCIe card is set to be the interface connected to the first PCIe slot and can be set to Endpoint mode to achieve master-slave communication between the second PCIe card and the first PCIe card.
可以理解的是，该种实施方式以及后文实施方式描述的PCIe接口也即扩展卡接口，以第三扩展卡13为例，第三扩展卡13的第一扩展卡接口，也就是第三PCIe卡的第一PCIe接口。It can be understood that the PCIe interface described in this implementation and the implementations below is also the expansion card interface. Taking the third expansion card 13 as an example, the first expansion card interface of the third expansion card 13 is also the first PCIe interface of the third PCIe card.
该种实施方式中，通过外接线缆实现了第三PCIe卡和第二PCIe卡之间的数据交互，同样不需要CPU、内存的参与，可以看出，该种实施方式中，实现了第一主板20上不同的插槽对之间的PCIe卡的数据交互。In this implementation, data interaction between the third PCIe card and the second PCIe card is achieved through an external cable, and the CPU and memory do not need to be involved. It can be seen that in this implementation, data interaction between PCIe cards between different slot pairs on the first motherboard 20 is achieved.
需要说明的是，本申请的实施方式中存在2种通信方式，第1种是扩展卡之间采用的PCIE的硬件链路，该种实施方式中连接2个扩展卡的线缆，上文中描述的第一主板20的各个插槽对之间，以及后文实施例中的第二主板30的各个插槽对之间，均是基于PCIE硬件链路实现的通信。第2种通信方式，是扩展卡与远程管理平台之间的网络通信，例如后文一种实施方式中介绍的通过扩展卡的光模块接口，基于网络实现的与远程管理平台之间的通信。It should be noted that there are two communication modes in the implementation of the present application. The first is the PCIE hardware link used between the expansion cards. In this implementation, the cable connecting the two expansion cards, the various slot pairs of the first mainboard 20 described above, and the various slot pairs of the second mainboard 30 in the embodiments below are all based on the communication realized by the PCIE hardware link. The second communication mode is the network communication between the expansion card and the remote management platform, such as the communication with the remote management platform based on the network through the optical module interface of the expansion card introduced in one implementation mode below.
进一步的，可参与图4，还可以包括：被设置为进行数据处理的第四扩展卡14，第四扩展卡14与第一主板20的第四扩展卡插槽连接；Further, referring to FIG. 4 , the system may further include: a fourth expansion card 14 configured to perform data processing, the fourth expansion card 14 being connected to a fourth expansion card slot of the first mainboard 20;
其中，第一主板20的第三扩展卡插槽和第四扩展卡插槽为第一主板20中的1组插槽对。The third expansion card slot and the fourth expansion card slot of the first mainboard 20 form a slot pair in the first mainboard 20 .
该种实施方式中，第四扩展卡14也可以进行数据处理，仍以PCIe为例，插入第四PCIe卡至第一主板20的第四PCIe插槽之后，便实现了4张PCIe卡插入第一主板20的21组插槽对的情况，并且可以看出，在第一PCIe卡、第二PCIe卡、第三PCIe卡以及第四PCIe卡之间，通过第一主板20的电路以及外接线缆，可以实现直接或者间接的通信连接。In this embodiment, the fourth expansion card 14 can also perform data processing. Taking PCIe as an example, after inserting the fourth PCIe card into the fourth PCIe slot of the first motherboard 20, the situation of 4 PCIe cards being inserted into 21 groups of slot pairs of the first motherboard 20 is realized. It can be seen that direct or indirect communication connection can be achieved between the first PCIe card, the second PCIe card, the third PCIe card and the fourth PCIe card through the circuit of the first motherboard 20 and the external cable.
在本申请的一种可选实施方式中，第一PCIe卡，第二PCIe卡，第三PCIe卡以及第四PCIe卡均为现场可编程门阵列卡，即均为FPGA卡，从而实现了全部使用FPGA卡插入第一主板20的实施方式，即实现了全部使用FPGA卡来完成数据处理。In an optional implementation of the present application, the first PCIe card, the second PCIe card, the third PCIe card and the fourth PCIe card are all field programmable gate array cards, that is, they are all FPGA cards, thereby realizing an implementation method in which all FPGA cards are inserted into the first mainboard 20, that is, all FPGA cards are used to complete data processing.
此外可以理解的是，该种实施方式中，对于第三PCIe卡和第二PCIe卡而言，由于既需要插入相应的PCIe插槽，又需要连接外接线缆，因此如上文的描述，第三PCIe卡和第二PCIe卡均至少具有2个PCIe接口。而对于该种实施方式的第一PCIe卡和第四PCIe卡而言，由于只需要插入相应的PCIe插槽，因此具有1个PCIe接口即可。当然，例如另一种实施方式中，还需要通过外接线缆，将第一PCIe卡和第四PCIe卡连接，构成这四张PCIe卡的环形通信，以提高这四张PCIe卡之间的通信效率，则第一PCIe卡和第四PCIe卡也均至少具有2个PCIe接口，并且可以理解的是，在进行接口模式设置时，如上文的描述，需要符合相连接的2个接口一主一从的原则。In addition, it can be understood that, in this embodiment, for the third PCIe card and the second PCIe card, since they need to be inserted into the corresponding PCIe slots and connected to the external cables, as described above, the third PCIe card and the second PCIe card each have at least 2 PCIe interfaces. For the first PCIe card and the fourth PCIe card in this embodiment, since they only need to be inserted into the corresponding PCIe slots, they only need to have 1 PCIe interface. Of course, in another embodiment, for example, it is also necessary to connect the first PCIe card and the fourth PCIe card through an external cable to form a ring communication of the four PCIe cards to improve the communication efficiency between the four PCIe cards, then the first PCIe card and the fourth PCIe card also have at least 2 PCIe interfaces, and it can be understood that when setting the interface mode, as described above, it is necessary to comply with the principle of one master and one slave of the two connected interfaces.
进一步的，在本申请的一种可选实施方式中，可参阅图5，还可以包括：与第一主板20连接，被设置为进行数据处理的K张现场可编程门阵列卡，即K张FPGA卡，并且通过第一主板20的电路和外接线缆，K张现场可编程门阵列卡依次通信连接。Furthermore, in an optional implementation of the present application, referring to Figure 5, it can also include: K field programmable gate array cards, i.e., K FPGA cards, connected to the first mainboard 20 and configured to perform data processing, and the K field programmable gate array cards are communicatively connected in sequence through the circuits of the first mainboard 20 and external cables.
可以理解的是，该种实施方式中还设置了与第一主板20连接的K张现场可编程门阵列卡，即K张FPGA卡，因此第一主板20上至少需要具有足够数量的PCIe插槽来容纳这些FPGA卡。It is understandable that in this embodiment, K field programmable gate array cards, namely K FPGA cards, connected to the first mainboard 20 are also provided, so the first mainboard 20 needs to have at least a sufficient number of PCIe slots to accommodate these FPGA cards.
K为正整数，例如在图5的实施方式中K＝4，由于第一PCIe卡，第二PCIe卡，第三PCIe卡以及第四PCIe卡均为FPGA卡，则图5中一共有8张FPGA卡。K is a positive integer. For example, in the embodiment of FIG. 5 , K=4. Since the first PCIe card, the second PCIe card, the third PCIe card and the fourth PCIe card are all FPGA cards, there are a total of 8 FPGA cards in FIG. 5 .
K张FPGA卡均可以进行数据处理，并且通过第一主板20的电路和外接线缆，实现了K张FPGA卡依次通信连接，图5中将K张FPGA卡依次标记为FPGA卡5至FPGA卡8，则可以看出，FPGA卡5与FPGA卡6之间通过第一主板20的电路实现通信连接，FPGA卡6与FPGA卡7之间则通过外接线缆实现通信连接，FPGA卡7与FPGA卡8之间则通过第一主板20的电路实现通信连接。K FPGA cards can all perform data processing, and through the circuits of the first mainboard 20 and external cables, the K FPGA cards are sequentially connected in communication. In Figure 5, the K FPGA cards are marked as FPGA card 5 to FPGA card 8 in sequence. It can be seen that FPGA card 5 and FPGA card 6 are communicated through the circuits of the first mainboard 20, FPGA card 6 and FPGA card 7 are communicated through external cables, and FPGA card 7 and FPGA card 8 are communicated through the circuits of the first mainboard 20.
K张FPGA卡中，任意1张FPGA卡均可以通过网络连接至远程管理平台。在实际应用中，例如在远程管理平台的控制下，可以使用图5的FPGA卡5至FPGA卡8，实现远程管理平台的项目1的数据处理，而使用图5的第一PCIe卡，第二PCIe卡，第三PCIe卡以及第四PCIe卡，实现远程管理平台的项目2的数据处理。Among the K FPGA cards, any one FPGA card can be connected to the remote management platform through the network. In practical applications, for example, under the control of the remote management platform, FPGA cards 5 to FPGA cards 8 in FIG. 5 can be used to implement data processing of project 1 of the remote management platform, and the first PCIe card, the second PCIe card, the third PCIe card, and the fourth PCIe card in FIG. 5 can be used to implement data processing of project 2 of the remote management platform.
进一步地，可参阅图6，K张现场可编程门阵列卡中的一张现场可编程门阵列卡通过外接线缆与第四扩展卡14通信连接，K张现场可编程门阵列卡中的一张现场可编程门阵列卡通过外接线缆与第一扩展卡11通信连接，以构成第一主板中的各张现场可编程门阵列卡的环形通信。Further, referring to Figure 6, one of the K field programmable gate array cards is communicatively connected to the fourth expansion card 14 via an external cable, and one of the K field programmable gate array cards is communicatively connected to the first expansion card 11 via an external cable to form a ring communication of the field programmable gate array cards in the first mainboard.
也就是说，图6中，K张FPGA卡中的一张FPGA卡通过外接线缆与第四PCIe卡通信连接，K张FPGA卡中的一张FPGA卡通过外接线缆与第一PCIe卡通信连接，以构成第一主板20中的各张FPGA卡的环形通信。That is to say, in Figure 6, one FPGA card among the K FPGA cards is connected to the fourth PCIe card through an external cable, and one FPGA card among the K FPGA cards is connected to the first PCIe card through an external cable to form a ring communication of the FPGA cards in the first motherboard 20.
上文的实施方式中，插入的K张FPGA卡，与第一PCIe卡至第四PCIe卡无数据交互，该种实施方式则考虑到，在部分实施方式中，可能需要K张FPGA卡以及第一PCIe卡至第四PCIe卡共同完成数据处理的工作，即需要实现K张FPGA与第一PCIe卡至第四PCIe卡之间的数据交互，因此，该种实施方式中，选取出K张FPGA卡中的其中一张FPGA卡通过外接线缆与第四PCIe卡通信连接，选取出K张FPGA卡中的其中一张FPGA卡通过外接线缆与第一PCIe卡通信连接，选取方式可以有多种，只要能够实现第一主板20中的各张FPGA卡的环形通信即可。In the above implementation, the inserted K FPGA cards have no data interaction with the first PCIe card to the fourth PCIe card. This implementation takes into account that in some implementations, K FPGA cards and the first PCIe card to the fourth PCIe card may be required to jointly complete the data processing task, that is, it is necessary to realize data interaction between the K FPGA cards and the first PCIe card to the fourth PCIe card. Therefore, in this implementation, one of the K FPGA cards is selected to communicate with the fourth PCIe card through an external cable, and one of the K FPGA cards is selected to communicate with the first PCIe card through an external cable. There can be multiple selection methods, as long as the ring communication of each FPGA card in the first motherboard 20 can be realized.
例如图6中，将FPGA卡5通过外接线缆与第四扩展卡14通信连接，将FPGA卡8通过外接线缆与第一扩展卡11通信连接，图6中的各张扩展卡均为FPGA卡，从而实现了第一主板20中的8张FPGA卡的环形通信。相较于链式线路通信，环形通信的效率更高，当然，例如图6这样的环形通信方式，要求图6中的8张FPGA卡均具有2个PCIe接口。For example, in FIG6, the FPGA card 5 is connected to the fourth expansion card 14 through an external cable, and the FPGA card 8 is connected to the first expansion card through an external cable. 11 communication connection, each expansion card in FIG6 is an FPGA card, thereby realizing the ring communication of the 8 FPGA cards in the first mainboard 20. Compared with the chain line communication, the ring communication is more efficient. Of course, the ring communication mode such as FIG6 requires that the 8 FPGA cards in FIG6 have 2 PCIe interfaces.
在本申请的一种可选实施方式中，第一扩展卡11，第二扩展卡12，第三扩展卡13以及第四扩展卡14均通过自身的光模块接口与交换机连接，以与远程管理平台进行通信。In an optional implementation of the present application, the first expansion card 11, the second expansion card 12, the third expansion card 13 and the fourth expansion card 14 are all connected to the switch through their own optical module interfaces to communicate with the remote management platform.
以第一PCIe卡为例，第一PCIe卡可能有1个或者多个光模块接口，需要将第一PCIe卡的至少1个光模块连接至交换机，从而实现第一PCIe卡与远程管理平台的通信。例如图7的实施方式中，该PCIe卡便具有2个100G光模块接口。图8则示出了第一主板20中的8块PCIe卡均与100G TOR(Top of Rack，架顶式)交换机连接，且图8中，插入第一主板20中插槽的PCIe卡均为FPGA卡，依次标记为FPGA卡1至FPGA卡8。Taking the first PCIe card as an example, the first PCIe card may have one or more optical module interfaces, and at least one optical module of the first PCIe card needs to be connected to the switch to achieve communication between the first PCIe card and the remote management platform. For example, in the implementation of Figure 7, the PCIe card has two 100G optical module interfaces. Figure 8 shows that the eight PCIe cards in the first motherboard 20 are all connected to a 100G TOR (Top of Rack) switch, and in Figure 8, the PCIe cards inserted into the slots in the first motherboard 20 are all FPGA cards, which are marked as FPGA card 1 to FPGA card 8 in sequence.
此外可以理解的是，在其他的实施方式中，对于存在光模块接口的PCIe卡，也均可以通过交换机连接至远程管理平台。Furthermore, it can be understood that, in other implementations, PCIe cards with optical module interfaces can also be connected to the remote management platform via a switch.
在本申请的一种可选实施方式中，第一扩展卡11和第四扩展卡14均为图形处理器卡，第二扩展卡12和第三扩展卡13均为现场可编程门阵列卡。即，第一PCIe卡和第四PCIe卡均为GPU卡；第二PCIe卡和第三PCIe卡均为FPGA卡。In an optional implementation of the present application, the first expansion card 11 and the fourth expansion card 14 are both graphics processor cards, and the second expansion card 12 and the third expansion card 13 are both field programmable gate array cards. That is, the first PCIe card and the fourth PCIe card are both GPU cards; the second PCIe card and the third PCIe card are both FPGA cards.
在前文的实施方式中，所使用的PCIe卡为FPGA卡，该种实施方式中，插入第一主板20的PCIe卡既有FPGA卡又有GPU卡，实现了FPGA卡和GPU卡组成的异构系统，满足了部分实施方式中的使用需求，特别是部分场合中，数据处理量较大，而GPU卡能够实现大量计算，FPGA卡则是作为GPU卡的辅助，例如在数据处理过程中，FPGA卡实现一些中间环节的数据预处理工作，例如，FPGA卡向GPU卡发送数据前，以及从GPU卡接收到数据之后，可以对数据进行乘累加运算等预处理工作。In the foregoing implementation, the PCIe card used is an FPGA card. In this implementation, the PCIe card inserted into the first mainboard 20 includes both an FPGA card and a GPU card, realizing a heterogeneous system composed of FPGA cards and GPU cards, and meeting the usage requirements of some implementations. In particular, in some occasions, the data processing volume is large, and the GPU card can perform a large amount of calculations, and the FPGA card serves as an auxiliary to the GPU card. For example, in the data processing process, the FPGA card implements data preprocessing work in some intermediate links. For example, before the FPGA card sends data to the GPU card, and after receiving data from the GPU card, it can perform preprocessing work such as multiplication and accumulation operations on the data.
可参阅图9，将第一PCIe卡标记为GPU卡1，将第四PCIe标记为GPU卡2。将第二PCIe卡标记为FPGA卡1，将第三PCIe卡标记为FPGA卡2。9 , the first PCIe card is marked as GPU card 1, the fourth PCIe card is marked as GPU card 2, the second PCIe card is marked as FPGA card 1, and the third PCIe card is marked as FPGA card 2.
在本申请的一种可选实施方式中，第二扩展卡12在数据处理过程中，通过自身的直接数据存取模块进行自身的不同接口之间的数据传输；In an optional implementation of the present application, during data processing, the second expansion card 12 performs data transmission between different interfaces of the second expansion card 12 through its own direct data access module;
第三扩展卡13在数据处理过程中，通过自身的直接数据存取模块进行自身的不同接口之间的数据传输。During the data processing, the third expansion card 13 performs data transmission between different interfaces of itself through its own direct data access module.
本申请的方案中，需要进行不同PCIe卡之间的数据交互，特别是在进行FPGA卡与GPU卡之间的数据交互时，需要实现高速的数据交互以保障本申请方案的数据处理效率，因此，该种实施方式中，第二PCIe卡和第三PCIe卡在数据处理过程中，都是通过自身的直接数据存取模块进行自身的不同接口之间的数据传输，例如，可以通过自身的MFDMA(Multi-Function DMA，多功能直接数据存取)模块进行自身的不同接口之间的高速的数据传输。In the scheme of the present application, data interaction is required between different PCIe cards, especially when data interaction is performed between FPGA cards and GPU cards, high-speed data interaction needs to be achieved to ensure the data processing efficiency of the scheme of the present application. Therefore, in this implementation, the second PCIe card and the third PCIe card perform data transmission between their different interfaces through their own direct data access modules during data processing. For example, high-speed data transmission between their different interfaces can be performed through their own MFDMA (Multi-Function DMA, multi-function direct data access) modules.
例如图7的实施方式中，FPGA卡中设置了3个对外接口，分别称为PCIe接口1，PCIe接口2以及PCIe接口3，通过MFDMA模块可以实现这3个接口之间的数据传输。此外MFDMA模块可以基于RoCE协议栈连接光模块接口，例如图7的MFDMA模块可以基于RoCE网络站，该RoCE网络站例如可以支持RoCEv2(RDMA over Converged Ethernet，第二代基于融合以太网的RDMA)的网络协议，通过2个100G光模块接口，实现与远程管理平台之间的数据传输。For example, in the implementation of FIG. 7 , three external interfaces are provided in the FPGA card, namely, PCIe interface 1, PCIe interface 2, and PCIe interface 3, and data transmission between the three interfaces can be realized through the MFDMA module. In addition, the MFDMA module can be connected to the optical module interface based on the RoCE protocol stack. For example, the MFDMA module of FIG. 7 can be based on the RoCE network station, which can support the RoCEv2 (RDMA over Converged Ethernet, the second generation of RDMA based on converged Ethernet) network protocol, and realize data transmission between the remote management platform through two 100G optical module interfaces.
在本申请的一种可选实施方式中，可参阅图9，还可以包括：被设置为进行数据处理的第五扩展卡15和第六扩展卡16；In an optional implementation of the present application, referring to FIG. 9 , the system may further include: a fifth expansion card 15 and a sixth expansion card 16 configured to perform data processing;
第五扩展卡15与第一主板20的第五扩展卡插槽连接，第六扩展卡16与第一主板20的第六扩展卡插槽连接，以通过第一主板20的电路完成第五扩展卡15与第六扩展卡16之间的通信；The fifth expansion card 15 is connected to the fifth expansion card slot of the first mainboard 20, and the sixth expansion card 16 is connected to the sixth expansion card slot of the first mainboard 20, so as to complete the communication between the fifth expansion card 15 and the sixth expansion card 16 through the circuit of the first mainboard 20;
其中，第五扩展卡15为图形处理器卡，即第五扩展卡15为GPU卡，第六扩展卡16为现场可编程门阵列卡，即第六扩展卡16为FPGA卡；第一主板20的第五扩展卡插槽和第六扩展卡插槽为第一主板20中的1组插槽对。Among them, the fifth expansion card 15 is a graphics processor card, that is, the fifth expansion card 15 is a GPU card, and the sixth expansion card 16 is a field programmable gate array card, that is, the sixth expansion card 16 is an FPGA card; the fifth expansion card slot and the sixth expansion card slot of the first motherboard 20 are a group of slot pairs in the first motherboard 20.
该种实施方式中，还设置了进行数据处理的第五PCIe卡和第六PCIe卡；并且第五PCIe卡为GPU卡，图9中标记为GPU卡3，第六PCIe卡为FPGA卡，图9中标记为FPGA卡3。第一主板20的第五PCIe插槽和第一主板20的第六PCIe插槽为第一主板20中的1组插槽对。In this embodiment, a fifth PCIe card and a sixth PCIe card for data processing are also provided; and the fifth PCIe card is a GPU card, which is marked as GPU card 3 in FIG9 , and the sixth PCIe card is an FPGA card, which is marked as FPGA card 3 in FIG9 . The fifth PCIe slot of the first mainboard 20 and the sixth PCIe slot of the first mainboard 20 are a pair of slots in the first mainboard 20 .
该种实施方式中，可以利用不同的GPU卡进行不同数据处理任务，例如一种场合中，在远程管理平台的控制下，可以使用图9的右边的4张PCIe卡实现远程管理平台的项目1的数据处理，而使用图9的第五PCIe卡和第六PCIe卡实现远程管理平台的项目2的数据处理。In this implementation, different GPU cards can be used to perform different data processing tasks. For example, in one scenario, under the control of the remote management platform, the four PCIe cards on the right side of Figure 9 can be used to implement data processing of Project 1 of the remote management platform, while the fifth PCIe card and the sixth PCIe card of Figure 9 can be used to implement data processing of Project 2 of the remote management platform.
在本申请的一种可选实施方式中，还可以包括：被设置为进行数据处理的第七扩展卡17和第八扩展卡18；In an optional implementation manner of the present application, the system may further include: a seventh expansion card 17 and an eighth expansion card 18 configured to perform data processing;
第七扩展卡17与第一主板20的第七扩展卡插槽连接，第八扩展卡18与第一主板20的第八扩展卡插槽连接，以通过第一主板20的电路完成第七扩展卡17与第八扩展卡18之间的通信；The seventh expansion card 17 is connected to the seventh expansion card slot of the first mainboard 20, and the eighth expansion card 18 is connected to the eighth expansion card slot of the first mainboard 20, so as to complete the communication between the seventh expansion card 17 and the eighth expansion card 18 through the circuit of the first mainboard 20;
其中，第七扩展卡17为GPU卡，第八扩展卡18为FPGA卡；第一主板20的第七扩展卡插槽和第八扩展卡插槽为第一主板20中的1组插槽对。Among them, the seventh expansion card 17 is a GPU card, and the eighth expansion card 18 is an FPGA card; the seventh expansion card slot and the eighth expansion card slot of the first mainboard 20 are a group of slot pairs in the first mainboard 20.
该种实施方式中，考虑到在部分场合中，可能会存在更多的数据处理任务，因此进一步地设置了用于进行数据处理的第七PCIe卡和第八PCIe卡，以进一步地提高本申请方案的灵活性，满足不同场合中的使用需求。In this implementation, considering that in some situations, there may be more data processing tasks, a seventh PCIe card and an eighth PCIe card for data processing are further provided to further improve the flexibility of the present application scheme and meet the usage requirements in different situations.
并且第七PCIe卡为GPU卡，图9中标记为GPU卡4，第八PCIe卡为FPGA卡，图9中标记为FPGA卡4。第一主板20的第七PCIe插槽和第一主板20的第八PCIe插槽是第一主板20中的1组插槽对。The seventh PCIe card is a GPU card, which is marked as GPU card 4 in FIG9 , and the eighth PCIe card is an FPGA card, which is marked as FPGA card 4 in FIG9 . The seventh PCIe slot of the first motherboard 20 and the eighth PCIe slot of the first motherboard 20 are a pair of slots in the first motherboard 20 .
进一步的，在本申请的一种可选实施方式中，可参阅图，第六扩展卡16和第八扩展卡18均至少具有2个扩展卡接口；Further, in an optional implementation of the present application, referring to the figure, the sixth expansion card 16 and the eighth expansion card 18 each have at least two expansion card interfaces;
第六扩展卡16的第一扩展卡接口与第一主板20的第六扩展卡插槽连接，第八扩展卡18的第一扩展卡接口与第一主板20的第八扩展卡插槽连接，第六扩展卡16的第二扩展卡接口通过外接线缆与第八扩展卡18的第二扩展卡接口通信连接。The first expansion card interface of the sixth expansion card 16 is connected to the sixth expansion card slot of the first mainboard 20, the first expansion card interface of the eighth expansion card 18 is connected to the eighth expansion card slot of the first mainboard 20, and the second expansion card interface of the sixth expansion card 16 is communicatively connected to the second expansion card interface of the eighth expansion card 18 via an external cable.
该种实施方式中，仍以PCIe为例进行说明，第六PCIe卡的第二PCIe接口通过外接线缆与第八PCIe卡的第二PCIe接口通信连接，可以看出，此时在图10的实施方式中，左侧的4张PCIe卡之间实现了通信连接，右侧的4张PCIe卡之间实现了通信连接，也就是说，本申请的方案中，可以支持在第一板卡内有多个数据处理分区，不同的数据处理分区可以负责不同的数据处理的任务，每个数据处理分区中可以包括1张或者多张GPU卡以满足该数据处理分区的资源需求。In this implementation, PCIe is still used as an example for explanation. The second PCIe interface of the sixth PCIe card is communicatively connected to the second PCIe interface of the eighth PCIe card through an external cable. It can be seen that at this time in the implementation of Figure 10, communication connection is achieved between the four PCIe cards on the left, and communication connection is achieved between the four PCIe cards on the right. That is to say, in the solution of the present application, multiple data processing partitions can be supported in the first board card, and different data processing partitions can be responsible for different data processing tasks. Each data processing partition can include one or more GPU cards to meet the resource requirements of the data processing partition.
进一步的，可参阅图11，第八扩展卡18和第二扩展卡12均至少具有3个扩展卡接口；第八扩展卡18的第三扩展卡接口通过外接线缆与第二扩展卡12的第三扩展卡接口通信连接。Further, referring to FIG. 11 , the eighth expansion card 18 and the second expansion card 12 both have at least three expansion card interfaces; the third expansion card interface of the eighth expansion card 18 is communicatively connected to the third expansion card interface of the second expansion card 12 via an external cable.
上文的实施方式中，是考虑到在部分场合中，可能会存在2项或者更多项目的数据处理任务，而该种实施方式则考虑到，在部分实施方式中，可能需要第一主板20中的各张FPGA卡以及GPU卡共同完成一项数据处理的任务，因此，该种实施方式中，将第八PCIe卡的第三PCIe接口通过外接线缆与第二PCIe卡的第三PCIe接口通信连接。In the above implementation, it is taken into account that in some cases, there may be two or more data processing tasks, and this implementation takes into account that in some implementations, the FPGA cards and GPU cards in the first motherboard 20 may be required to jointly complete a data processing task. Therefore, in this implementation, the third PCIe interface of the eighth PCIe card is communicatively connected to the third PCIe interface of the second PCIe card via an external cable.
可以看出，该种实施方式中，第八PCIe卡不仅需要插入第一主板20的第八PCIe插槽，还需要通过外接线缆连接第二PCIe卡以及第六PCIe卡，因此第八PCIe卡需要至少具有3个PCIe接口。同样的，第二PCIe卡不仅需要插入第一主板20的第二PCIe插槽，还需要通过外接线缆连接第三PCIe卡以及第八PCIe卡，因此第二PCIe卡需要至少具有3个PCIe接口。It can be seen that in this implementation, the eighth PCIe card not only needs to be inserted into the eighth PCIe slot of the first mainboard 20, but also needs to be connected to the second PCIe card and the sixth PCIe card through an external cable, so the eighth PCIe card needs to have at least 3 PCIe interfaces. Similarly, the second PCIe card not only needs to be inserted into the second PCIe slot of the first mainboard 20, but also needs to be connected to the third PCIe card and the eighth PCIe card through an external cable, so the second PCIe card needs to have at least 3 PCIe interfaces.
并且可以理解的是，如上文的描述，两张PCIe卡之间进行通信时，需要划分主从设备，以第二PCIe卡为例，例如可以将第三PCIe卡的第二PCIe接口设置为Root Complex模式，使得第三PCIe卡的第二PCIe接口作为主设备接口，而将第二PCIe卡的第二PCIe接口设置为Endpoint模式，使得第二PCIe卡的第二PCIe接口作为从设备接口，实现第三PCIe卡和第二PCIe卡之间的主从通信。And it is understandable that, as described above, when two PCIe cards communicate with each other, it is necessary to divide the master and slave devices. Taking the second PCIe card as an example, the second PCIe interface of the third PCIe card can be set to Root Complex mode, so that the second PCIe interface of the third PCIe card serves as the master device interface, and the second PCIe interface of the second PCIe card can be set to Endpoint mode, so that the second PCIe interface of the second PCIe card serves as the slave device interface, thereby realizing master-slave communication between the third PCIe card and the second PCIe card.
第二PCIe卡的第一PCIe接口例如可以设置Root Complex模式，使得第二PCIe卡的第一PCIe接口作为主设备接口，第一PCIe卡为GPU卡1，GPU卡1被设置为连接第一PCIe插槽的接口则可以设置Endpoint模式，实现第二PCIe卡和第一PCIe卡之间的主从通信。For example, the first PCIe interface of the second PCIe card can be set to Root Complex mode, so that the first PCIe interface of the second PCIe card serves as the master device interface. The first PCIe card is GPU card 1, and GPU card 1 is set to be the interface connected to the first PCIe slot. Endpoint mode can be set to achieve master-slave communication between the second PCIe card and the first PCIe card.
例如可以将第二PCIe卡的第三PCIe接口设置为Root Complex模式，使得第二PCIe卡的第三PCIe接口作为主设备接口，而将第八PCIe卡的第三PCIe接口设置为Endpoint模式，使得第八PCIe卡的第三PCIe接口作为从设备接口，实现第二PCIe卡和第八PCIe卡之间的主从通信。For example, the third PCIe interface of the second PCIe card can be set to Root Complex mode, so that the third PCIe interface of the second PCIe card serves as the master device interface, and the third PCIe interface of the eighth PCIe card can be set to Endpoint mode, so that the third PCIe interface of the eighth PCIe card serves as the slave device interface, thereby realizing master-slave communication between the second PCIe card and the eighth PCIe card.
可以看出，该种实施方式中，第二PCIe卡具有2个Root Complex模式的接口，1个Endpoint模式的接口。此外需要说明的是，本申请的实际应用中，通常可以采用该种实施方式，而GPU卡由于不便于进行接口调整，因此GPU卡可以设置为仅有1个接口，且为Endpoint模式，因此不同的GPU卡之间需要通过FPGA卡实现数据交互，本申请的图9至图13的实施方式也均是如此，而对于FPGA卡而言，则可以设置为3个接口以满足使用需求。在实际应用中，通过设置3个PCIe HIP(Hard IP，硬核IP)，便可以使得FPGA卡具有3个PCIe接口，并且可以在初始化时设定各个PCIe接口的模式。It can be seen that in this implementation, the second PCIe card has 2 Root Complex mode interfaces and 1 Endpoint mode interface. In addition, it should be noted that in the actual application of this application, this implementation can usually be adopted, and since it is not convenient for the GPU card to adjust the interface, the GPU card can be set to have only 1 interface, and it is in Endpoint mode. Therefore, different GPU cards need to realize data interaction through the FPGA card. The implementation of Figures 9 to 13 of this application is also the case. For the FPGA card, it can be set to 3 interfaces to meet the use requirements. In actual applications, by setting 3 PCIe HIPs (Hard IPs), the FPGA card can have 3 PCIe interfaces, and the mode of each PCIe interface can be set during initialization.
在本申请的一种可选实施方式中，可参阅图12，还可以包括第二主板30，第九扩展卡19以及第十扩展卡10；In an optional implementation of the present application, referring to FIG. 12 , a second mainboard 30 , a ninth expansion card 19 and a tenth expansion card 10 may also be included;
第二主板30中设置有至少1组插槽对，每组插槽对中包括2个扩展卡插槽，且每组插槽对中的2个扩展卡插槽的收发差分线通过第二主板30的电路进行连接；The second mainboard 30 is provided with at least one slot pair, each slot pair includes two expansion card slots, and the transceiver differential lines of the two expansion card slots in each slot pair are connected through the circuit of the second mainboard 30;
第九扩展卡19与第二主板30的第一扩展卡插槽连接，第十扩展卡10与第二主板30的第二扩展卡插槽连接，以通过第二主板30的电路完成第九扩展卡19与第十扩展卡10之间的通信；The ninth expansion card 19 is connected to the first expansion card slot of the second mainboard 30, and the tenth expansion card 10 is connected to the second expansion card slot of the second mainboard 30, so as to complete the communication between the ninth expansion card 19 and the tenth expansion card 10 through the circuit of the second mainboard 30;
第二主板30的第一扩展卡插槽和第二扩展卡插槽为第二主板30中的1组插槽对；第三扩展卡13至少具有3个扩展卡接口，第九扩展卡19至少具有2个扩展卡接口；The first expansion card slot and the second expansion card slot of the second mainboard 30 are a pair of slots in the second mainboard 30; the third expansion card 13 has at least three expansion card interfaces, and the ninth expansion card 19 has at least two expansion card interfaces;
第九扩展卡19的第一扩展卡接口与第一主板20的第九扩展卡插槽连接，第三扩展卡13的第三扩展卡接口通过外接线缆与第九扩展卡19的第二扩展卡接口通信连接。The first expansion card interface of the ninth expansion card 19 is connected to the ninth expansion card slot of the first mainboard 20 , and the third expansion card interface of the third expansion card 13 is communicatively connected to the second expansion card interface of the ninth expansion card 19 via an external cable.
该种实施方式中，实现了不同节点之间的PCIe卡的数据交互，即可以实现第一主板20中的PCIe卡，与第二主板30中的PCIe卡的数据交互，仍然无需CPU、内存的参与，即本申请的方案既可以使用在单节点的场合中，又可以使用在多个节点的场合中，灵活性很高。In this implementation, data interaction of PCIe cards between different nodes is realized, that is, data interaction between the PCIe card in the first motherboard 20 and the PCIe card in the second motherboard 30 can be realized, and the participation of the CPU and memory is still not required. That is, the solution of the present application can be used in single-node scenarios as well as in multiple-node scenarios, and is very flexible.
可选的，第二主板30与第一主板20类似，也是设置有至少1组插槽对，第二主板30中，每组插槽对中包括2个PCIe插槽，且第二主板30中的每组插槽对中的2个PCIe插槽的收发差分线通过第二主板30的电路进行连接，由于与上文同理便不再重复说明。Optionally, the second motherboard 30 is similar to the first motherboard 20 and is also provided with at least one slot pair. In the second motherboard 30, each slot pair includes 2 PCIe slots, and the transmit and receive differential lines of the 2 PCIe slots in each slot pair in the second motherboard 30 are connected through the circuit of the second motherboard 30. Since it is the same as above, it will not be repeated.
第九PCIe卡以及第十PCIe卡分别连接至第二主板30的第一PCIe插槽以及第二主板30的第二PCIe插槽，因此可以通过第二主板30的电路完成第九PCIe卡与第十PCIe卡之间的通信。The ninth PCIe card and the tenth PCIe card are connected to the first PCIe slot of the second motherboard 30 and the second PCIe slot of the second motherboard 30 respectively, so the communication between the ninth PCIe card and the tenth PCIe card can be completed through the circuit of the second motherboard 30.
由于第三PCIe卡的第三PCIe接口通过外接线缆与第九PCIe卡的第二PCIe接口通信连接，因此便实现了第一主板20中的PCIe卡，与第二主板30中的PCIe卡的数据交互。Since the third PCIe interface of the third PCIe card is communicatively connected to the second PCIe interface of the ninth PCIe card via an external cable, data interaction between the PCIe card in the first mainboard 20 and the PCIe card in the second mainboard 30 is implemented.
进一步的，在实际应用中，需要进行主板之间的数据交互时，通常是需要进行GPU卡之间的数据交互，因此在本申请的一种实施方式中，第九扩展卡19为现场可编程门阵列卡，第十扩展卡10为图形处理器卡，即第九PCIe卡可以为FPGA卡，第十PCIe卡可以为GPU卡。Furthermore, in actual applications, when data interaction between mainboards is required, data interaction between GPU cards is usually required. Therefore, in one embodiment of the present application, the ninth expansion card 19 is a field programmable gate array card, and the tenth expansion card 10 is a graphics processor card, that is, the ninth PCIe card can be an FPGA card, and the tenth PCIe card can be a GPU card.
当然，其他实施方式中，第二主板30中可以根据需要设置有更多数量的GPU卡和FPGA卡，例如图13的实施方式中，第一主板20的每一组插槽对中均设置了一张FPGA卡和一张GPU卡，第二主板30的每一组插槽对中也均设置了一张FPGA卡和一张GPU卡，通过这2块主板中的FPGA卡实现环形通信连接。Of course, in other implementations, the second motherboard 30 can be provided with a larger number of GPU cards and FPGA cards as needed. For example, in the implementation of FIG. 13 , one FPGA card and one GPU card are provided in each slot pair of the first motherboard 20, and one FPGA card and one GPU card are provided in each slot pair of the second motherboard 30. An FPGA card and a GPU card are also provided, and a ring communication connection is realized through the FPGA cards in the two main boards.
在本申请的一种可选实施方式中，还可以包括：In an optional implementation of the present application, it may also include:
设置在第一主板20上，被设置为进行第一主板20的散热的散热装置40。考虑到本申请的第一主板20中，插入的PCIe卡数量较多，需要进行大量的数据运算，因此，该种实施方式中在第一主板20上设置了进行第一主板20的散热的散热装置40，例如为风扇散热装置。图2的实施方式中，便示出了散热装置40。A heat sink 40 is provided on the first mainboard 20 and is configured to dissipate heat from the first mainboard 20. Considering that a large number of PCIe cards are inserted into the first mainboard 20 of the present application and a large amount of data calculations need to be performed, a heat sink 40, such as a fan heat sink, is provided on the first mainboard 20 in this embodiment to dissipate heat from the first mainboard 20. The heat sink 40 is shown in the embodiment of FIG. 2 .
在本申请的一种可选实施方式中，还可以包括：In an optional implementation of the present application, it may also include:
设置在第一主板20上的监控管理装置50，被设置为监控与第一主板20连接的各张扩展卡的状态。The monitoring management device 50 disposed on the first mainboard 20 is configured to monitor the status of each expansion card connected to the first mainboard 20 .
该种实施方式中，考虑到第一主板20上虽然不需要设置CPU、内存来实现PCIe卡之间的数据交互，但是，可以通过PCIe的日志，实现各张PCIe卡的状态监控，以便工作人员可以方便地掌控各张PCIe卡的状态。图2的实施方式中，便示出了监控管理装置50。In this implementation, although the first motherboard 20 does not need to be provided with a CPU or memory to implement data interaction between PCIe cards, the status of each PCIe card can be monitored through the PCIe log, so that the staff can easily control the status of each PCIe card. In the implementation of FIG. 2 , a monitoring management device 50 is shown.
进一步的，监控管理装置50还可以被设置为：Furthermore, the monitoring management device 50 may also be configured as follows:
当监测到任意一张PCIe卡的状态异常时，向BMC(Board Management Controller，基板管理控制器)输出日志信息。When the status of any PCIe card is detected to be abnormal, log information is output to the BMC (Board Management Controller).
考虑到部分服务器中会设置有BMC，因此，监控管理装置50监测到任意一张PCIe卡的状态时，可以向BMC输出日志信息，已通过BMC实现异常PCIe卡的处理。Considering that some servers are provided with a BMC, when the monitoring management device 50 detects the status of any PCIe card, it can output log information to the BMC, and handle the abnormal PCIe card through the BMC.
在本申请的一种可选实施方式中，针对任意1张扩展卡，当该PCIe卡为现场可编程门阵列卡时，即为FPGA卡时，在远程管理平台的控制下，该扩展卡通过初始化操作以及地址映射操作，建立与自身相连的扩展卡的通信连接。In an optional implementation of the present application, for any expansion card, when the PCIe card is a field programmable gate array card, that is, an FPGA card, under the control of a remote management platform, the expansion card establishes a communication connection with the expansion card connected to itself through initialization operations and address mapping operations.
在上文的不同实施方式中，与某一张PCIe卡直接连接的可能只有一张PCIe卡，也可能有2张或者3张PCIe卡，因此，在上文的不同实施方式中，某一张PCIe卡可能需要不同数量的接口，并且如上文的描述，GPU卡由于不便于进行接口调整，因此GPU卡可以设置为仅有1个接口，FPGA卡则可以根据需要设置有1个或者多个接口。In the different implementations above, there may be only one PCIe card directly connected to a certain PCIe card, or there may be two or three PCIe cards. Therefore, in the different implementations above, a certain PCIe card may require a different number of interfaces. As described above, since it is not convenient to adjust the interface of the GPU card, the GPU card can be set to have only one interface, and the FPGA card can be set with one or more interfaces as needed.
FPGA卡设置有多个PCIe接口，以通过不同的PCIe接口对外连接时，FPGA卡可能是作为主设备使用，也可能是作为从设备使用，因此，该种实施方式中，可以在上电启动之后进行初始化的设置。The FPGA card is provided with multiple PCIe interfaces. When the FPGA card is connected to the outside through different PCIe interfaces, the FPGA card may be used as a master device or a slave device. Therefore, in this implementation mode, the initialization settings may be performed after power-on.
可选的，对于任意1张PCIe卡而言，该PCIe卡可以通过初始化操作以及地址映射操作，建立与自身相连的PCIe卡的通信连接，这一过程通常可以在上电之后进行，并且由于本申请的主板中无需设置CPU，因此该过程可以由远程管理平台进行控制。Optionally, for any PCIe card, the PCIe card can establish a communication connection with the PCIe card connected to itself through initialization operations and address mapping operations. This process can usually be performed after power-on, and since there is no need to set up a CPU in the motherboard of the present application, the process can be controlled by a remote management platform.
例如，PCIe卡可以在远程管理平台的控制下，通过内部的初始化模块来执行初始化操作，例如图7的实施方式中便示出了FPGA卡中的初始化模块，在进行初始化时，需要按照初始化流程实现PCIe枚举初始化，按照顺序进行各个相关寄存器的配置，并且还需进行地址映射操作，即对该PCIe卡本地存储域的地址，与该PCIe卡自身相连接的其他PCIe卡的地址之间进行转换映射，从而使得该PCIe卡可以建立与自身相连的PCIe卡的通信连接。图7中，地址映射操作可以由内存重映射模块执行，内存重映射模块与3个PCIe接口设置在图7的PCIe桥中。For example, under the control of the remote management platform, the PCIe card can perform initialization operations through the internal initialization module. For example, the implementation of FIG7 shows the initialization module in the FPGA card. When initializing, it is necessary to implement PCIe enumeration initialization according to the initialization process, configure each related register in sequence, and perform address mapping operations, that is, convert and map the address of the local storage domain of the PCIe card to the address of other PCIe cards connected to the PCIe card itself, so that the PCIe card can establish a communication connection with the PCIe card connected to itself. In FIG7, the address mapping operation can be performed by the memory remapping module, and the memory remapping module and the three PCIe interfaces are set in the PCIe bridge of FIG7.
在初始化过程中，进行相关寄存器配置时，实现方式有多种，例如首先，远程管理平台可以通过数据中心网络，将配置信息发送给各个FPGA卡的初始化模块。FPGA卡的初始化模块接收了配置信息后，该初始化模块内部的Microblaze内核可以使用setpci命令配置PWR_LIMIT寄存器，配置PWR_LIMIT寄存器的作用是设定设备允许运行的功耗上限。然后配置BDF寄存器，即配置设备BUS、Device、Function三种ID的寄存器。之后配置写基地址寄存器，即，向Base Address Register寄存器写入全FF数值，再读取，然后根据该寄存器格式描述获取BAR上所挂载内存类型、大小等信息。之后可以分配PCIe域基地址给各BAR寄存器。然后，可以配置中断相关寄存器，配置连接控制寄存器，包括控制采用同步时钟还是异步时钟等。最后，配置命令寄存器，包括10空间使能、内存空间使能、总线主控使能等控制位，至此，便完成了该FPGA扩展卡的初始化操作。During the initialization process, when configuring the relevant registers, there are many ways to implement it. For example, first, the remote management platform can send the configuration information to the initialization module of each FPGA card through the data center network. After the initialization module of the FPGA card receives the configuration information, the Microblaze kernel inside the initialization module can use the setpci command to configure the PWR_LIMIT register. The function of configuring the PWR_LIMIT register is to set the upper limit of the power consumption allowed for the device to run. Then configure the BDF register, that is, configure the registers of the three IDs of the device BUS, Device, and Function. Then configure the write base address register, that is, write the full FF value to the Base Address Register register, then read it, and then obtain the type and size of the memory mounted on the BAR according to the register format description. Then, the PCIe domain base address can be assigned to each BAR register. Then, the interrupt-related registers can be configured, and the connection control register can be configured, including the control of whether to use a synchronous clock or an asynchronous clock. Finally, configure the command register, including control bits such as 10 space enable, memory space enable, and bus master enable. At this point, the initialization operation of the FPGA expansion card is completed.
应用本申请实施例所提供的技术方案，第一扩展卡11和第二扩展卡12均被设置为进行数据处理，且第一主板20中设置有至少1组插槽对，每组插槽对中包括2个扩展卡插槽，第一扩展卡11与第一主板20的第一扩展卡插槽连接，第二扩展卡12与第一主板20的第二扩展卡插槽连接，即第一扩展卡11和第二扩展卡12设置在主板的1组插槽对当中。本申请的方案中，在第一主板20的每组插槽对中，将每组插槽对中的2个扩展卡插槽的收发差分线通过第一主板20的电路进行连接，因此，第一扩展卡11和第二扩展卡12可以直接通过第一主板20的电路完成相互之间的通信，换句话说，本申请的方案中，第一扩展卡11和第二扩展卡12之间的通信不需要CPU的参数，使得本申请的方案实现了扩展卡与CPU的解耦。本申请的方案中第一主板20上也无需设置CPU，不会出现占用CPU资源的情况。Applying the technical solution provided by the embodiment of the present application, the first expansion card 11 and the second expansion card 12 are both configured to perform data processing, and at least one slot pair is provided in the first mainboard 20, each slot pair includes two expansion card slots, the first expansion card 11 is connected to the first expansion card slot of the first mainboard 20, and the second expansion card 12 is connected to the second expansion card slot of the first mainboard 20, that is, the first expansion card 11 and the second expansion card 12 are arranged in one slot pair of the mainboard. In the solution of the present application, in each slot pair of the first mainboard 20, the transceiver differential lines of the two expansion card slots in each slot pair are connected through the circuit of the first mainboard 20, so that the first expansion card 11 and the second expansion card 12 can directly communicate with each other through the circuit of the first mainboard 20. In other words, in the solution of the present application, the communication between the first expansion card 11 and the second expansion card 12 does not require the parameters of the CPU, so that the solution of the present application realizes the decoupling of the expansion card and the CPU. In the solution of the present application, there is no need to set a CPU on the first mainboard 20, and there will be no situation of occupying CPU resources.
综上，本申请的方案可以有效地利用扩展卡实现数据处理，且实现了与CPU的解耦，不需要占用CPU的资源。In summary, the solution of the present application can effectively utilize the expansion card to realize data processing, and realizes decoupling from the CPU without occupying CPU resources.
相应于上面的数据处理装置的实施例，本申请实施例还提供了一种异构设备以及一种服务器。该异构设备可以包括如上述任一实施例中的数据处理装置，该服务器可以包括上述的异构设备，可与上文相互对应参照，此处不再重复说明。Corresponding to the above data processing device embodiment, the present application embodiment further provides a heterogeneous device and a server. The heterogeneous device may include a data processing device as in any of the above embodiments, and the server may include the above heterogeneous device, which may correspond to the above and will not be repeated here.
还需要说明的是，在本文中，诸如第一和第二等之类的关系术语仅仅用来将一个实体或者操作与另一个实体或操作区分开来，而不一定要求或者暗示这些实体或操作之间存在任何这种实际的关系或者顺序。而且，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者设备所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括要素的过程、方法、物品或者设备中还存在另外的相同要素。It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
专业人员还可以进一步意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、计算机软件或者二者的结合来实现，为了清楚地说明硬件和软件的可互换性，在上述说明中已经按照功能一般性地描述了各示例的组成及步骤。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
本文中应用了个例对本申请的原理及实施方式进行了阐述，以上实施例的说明只是用于帮助理解本申请的技术方案及其核心思想。应当指出，对于本技术领域的普通技术人员来说，在不脱离本申请原理的前提下，还可以对本申请进行若干改进和修饰，这些改进和修饰也落入本申请的保护范围内。Individual examples are used in this article to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the technical solution and core ideas of this application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to this application, and these improvements and modifications also fall within the scope of protection of this application.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310403342.7ACN116401065A (en) | 2023-04-14 | 2023-04-14 | Server, heterogeneous equipment and data processing device thereof |
| CN202310403342.7 | 2023-04-14 |
| Publication Number | Publication Date |
|---|---|
| WO2024212744A1true WO2024212744A1 (en) | 2024-10-17 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/CN2024/080587PendingWO2024212744A1 (en) | 2023-04-14 | 2024-03-07 | Server, heterogeneous device, and data processing apparatus thereof |
| Country | Link |
|---|---|
| CN (1) | CN116401065A (en) |
| WO (1) | WO2024212744A1 (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN120067024A (en)* | 2025-04-27 | 2025-05-30 | 瀚博半导体(上海)有限公司 | Peripheral component quick interconnection additional clamping board and combination method and device thereof |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116401065A (en)* | 2023-04-14 | 2023-07-07 | 广东浪潮智慧计算技术有限公司 | Server, heterogeneous equipment and data processing device thereof |
| CN117493259B (en)* | 2023-12-28 | 2024-04-05 | 苏州元脑智能科技有限公司 | Data storage system, method and server |
| CN120371774A (en)* | 2025-06-26 | 2025-07-25 | 苏州元脑智能科技有限公司 | Boards and interconnection methods |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080065805A1 (en)* | 2006-09-11 | 2008-03-13 | Cameo Communications, Inc. | PCI-Express multimode expansion card and communication device having the same |
| US10296478B1 (en)* | 2015-09-11 | 2019-05-21 | Amazon Technologies, Inc. | Expansion card configuration of motherboard |
| CN114780472A (en)* | 2022-04-29 | 2022-07-22 | 苏州浪潮智能科技有限公司 | Host and PCIe device decoupling method, device, equipment and storage medium |
| CN217113223U (en)* | 2022-04-29 | 2022-08-02 | 苏州浪潮智能科技有限公司 | Power supply device of single slot position expansion card and server |
| CN116401065A (en)* | 2023-04-14 | 2023-07-07 | 广东浪潮智慧计算技术有限公司 | Server, heterogeneous equipment and data processing device thereof |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080065805A1 (en)* | 2006-09-11 | 2008-03-13 | Cameo Communications, Inc. | PCI-Express multimode expansion card and communication device having the same |
| US10296478B1 (en)* | 2015-09-11 | 2019-05-21 | Amazon Technologies, Inc. | Expansion card configuration of motherboard |
| CN114780472A (en)* | 2022-04-29 | 2022-07-22 | 苏州浪潮智能科技有限公司 | Host and PCIe device decoupling method, device, equipment and storage medium |
| CN217113223U (en)* | 2022-04-29 | 2022-08-02 | 苏州浪潮智能科技有限公司 | Power supply device of single slot position expansion card and server |
| CN116401065A (en)* | 2023-04-14 | 2023-07-07 | 广东浪潮智慧计算技术有限公司 | Server, heterogeneous equipment and data processing device thereof |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN120067024A (en)* | 2025-04-27 | 2025-05-30 | 瀚博半导体(上海)有限公司 | Peripheral component quick interconnection additional clamping board and combination method and device thereof |
| Publication number | Publication date |
|---|---|
| CN116401065A (en) | 2023-07-07 |
| Publication | Publication Date | Title |
|---|---|---|
| CN113872796B (en) | Method, device, equipment and medium for acquiring information of server and node equipment thereof | |
| WO2024212744A1 (en) | Server, heterogeneous device, and data processing apparatus thereof | |
| KR101686360B1 (en) | Control messaging in multislot link layer flit | |
| US11372787B2 (en) | Unified address space for multiple links | |
| CN107278299B (en) | Method, apparatus and system for implementing secondary bus functionality via a reconfigurable virtual switch | |
| CN117651940A (en) | Die-to-die interconnect | |
| CN100483373C (en) | PVDM (packet voice data module) generic bus protocol | |
| WO2025001344A1 (en) | Cxl data transmission board and data transmission control method | |
| CN117971740B (en) | Memory expansion board card and memory expansion method | |
| CN114445260A (en) | Distributed GPU communication method and device based on FPGA | |
| CN117421268A (en) | Interconnection system, equipment and network | |
| CN116483753A (en) | A server monitoring chip and system | |
| CN102780598A (en) | Bus communication method, bus communication unit and bus communication system | |
| CN107391332A (en) | A kind of storage system and debugging system | |
| CN207833500U (en) | A kind of 40G rate network interface cards for supporting non-standard interface | |
| WO2024152586A1 (en) | Signal transmission circuit and computing device | |
| CN216352292U (en) | Server mainboard and server | |
| CN104699655A (en) | Network chip and cloud server system | |
| CN206686217U (en) | A kind of multiserver network share framework | |
| CN116126761A (en) | A Master Control System with Multiple Controllers Sharing Serial Ports | |
| CN210572737U (en) | Secondary radar signal processing device | |
| CN206021155U (en) | A kind of fusion architecture server | |
| Klilou et al. | Performance optimization of high-speed Interconnect Serial RapidIO for onboard processing | |
| CN209265223U (en) | A serial/parallel conversion interface processing board supporting secondary development | |
| CN222167589U (en) | Computer system, system on chip and electronic device |
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application | Ref document number:24787850 Country of ref document:EP Kind code of ref document:A1 |