CN109088766B - Interconnection network fault detection and positioning method based on pairing test - Google Patents

Abstract

The invention discloses a method for detecting and positioning interconnection network faults based on pairing test, which is used for detecting and positioning functional faults based on pairing and two-time screening; performance and routing fault detection and location is performed based on layering and pairing. The invention relates to a large-scale interconnection network fault detection and positioning method based on pairing and multi-pass screening, which can quickly detect and position faults in a network, shorten the time for troubleshooting the faults and improve the availability and the usability of the large-scale network.

Description

Interconnection network fault detection and positioning method based on pairing test

Technical Field

The invention relates to the field of interconnection networks, in particular to an interconnection network fault detection and positioning method based on pairing test.

Background

High-speed interconnection networks including Infiniband are increasingly used in the fields of high-performance computing, big data, cloud computing, artificial intelligence and the like. The Infiniband is a high-performance network transmission solution, has the characteristics of high bandwidth and low delay, and is a preferred choice for constructing a super-large-scale low-delay high-bandwidth data transmission network at present. A large-scale interconnected network system may contain hundreds of thousands of nodes and ports; during the operation of the system, various network failures may occur, and the types of the failures are various, for example, a hardware failure includes a link Down, a link rate degradation, a link packet loss, and the like, and a software failure includes a routing configuration error, LID repetition, and the like. Some faults are easy to locate, such as PCIE link reset, HCA port Down, etc.; some errors are difficult to be found, for example, memory access errors, packet loss of the IB port is serious, and PCIE interfaces are unstable, and when such errors occur, the error phenomenon is that an operation subject is hung or directly exits, and it is difficult to locate a specific failure reason. The increase in the network size leads to a large increase in the time to perform a failure detection. Common diagnostic tools such as ibnetdiscover, ibibqueryerorrs, ibportstate, etc. have problems of large time overhead, difficulty in accurately positioning faults, etc. in a large-scale interconnection network.

Disclosure of Invention

The present invention aims to solve the problems mentioned in the background section above by a method for detecting and locating faults of an interconnection network based on a pairing test.

In order to achieve the purpose, the invention adopts the following technical scheme:

a method for detecting and positioning faults of an interconnection network based on pairing test comprises the following steps:

functional fault detection and location: pairing all network cards in the system, and running a function verification test question on each pair of network cards; after the test of all the network card pairs is finished, scanning the test result, dividing the network cards into two subsets according to the test result, putting the network card pairs with correct results into a set S, and putting the network card pairs with wrong results into a set E; thirdly, for each network card in the set E, optionally selecting one network card from the set S for pairing, and rerunning the function verification test question; after the test is finished, scanning the test result, if the network card test result in the set E is normal, indicating that the network card has no error, and moving the network card into the set S; through two-time screening, the network cards in the set E are all wrong;

performance and routing fault detection and location: firstly, performance detection in a switch chip: pairing all the network cards in pairs, so that each network card is paired with another network card in the switch core where the network card is located; secondly, detecting the performance in the super node: after the network card has no problem, checking whether the network card has performance problem; firstly, pairing all network cards in pairs, so that each network card is paired with another network card in a supernode where the network card is located; thirdly, detecting the performance among the super nodes: when processors in two super nodes communicate with each other, data packets pass through a top-level switching network; when detecting the routing balance between two super nodes, firstly selecting the two super nodes, respectively selecting processors corresponding to physical numbers in the super nodes, and pairing the processors with the same physical numbers pairwise; after the pairing is completed, a bidirectional bandwidth test program is operated on each pair of processors; when each pair of processors simultaneously communicate, if the routing distribution is uniform, no congestion is generated; after the test is finished, the average bandwidth of all the processors is sequenced from small to large, and the processors are printed and output, so that a tester can conveniently judge whether the routing distribution among the super nodes is uniform or not according to the test result.

Specifically, running a function verification test question on each pair of network cards specifically includes: the functional verification test questions adopt a ping-pong mode to send data, one party fills a sending buffer zone according to a data layout algorithm, and then the data in the buffer zone is sent to the other party; the other side checks the data after receiving the data, then fills a sending buffer area according to a data layout algorithm, and sends the data back; repeatedly and iteratively executing the process; and when the data receiving and sending between the two network cards have errors, recording the data in a corresponding result file.

In particular, the functional fault detection and localization further comprises: summarizing the network card lists in the set, and automatically or manually checking the message address, the message content and the error data of the data error according to the result recording file, thereby judging the specific reason of the message error.

In particular, the performance test within the switch chip: pairing all the network cards in pairs, so that each network card is paired with another network card in the switch core where the network card is located, specifically comprising: every two adjacent network cards in the same switch core are paired, and after pairing is completed, a bidirectional bandwidth test program is operated on each pair of network cards; network cards in the same switch chip are paired and communicated at the same time, so that congestion cannot be generated; after the test is finished, sorting the average bandwidth of all network card pairs from small to large, and printing the network card pairs with low performance; in the test, if the performance of a certain pair of network cards is lower than the set value when the network cards communicate with each other, the failure cause is as follows: the network card has problems, and a single switch chip has problems.

In particular, in the performance detection in the supernode, pairwise pairing is performed on all network cards, so that each network card is paired with another network card in the supernode where the network card is located, and the method specifically includes: if the physical number of the network card in the super node is n, the physical number of the network card matched with the super node is (n + 128)% 256; after the pairing is completed, a bidirectional bandwidth test program is operated on each pair of network cards; when 256 network cards in a super node are paired and communicated with each other at the same time, if the routing distribution is uniform, congestion cannot be generated, and the difference value between the bandwidth and the bandwidth when two network cards in a single switch are in butt joint is lower than a set value; after the test is finished, sorting the average bandwidth of all network card pairs from small to large, and printing the network card pairs with low performance; in this step of test, if the performance of a certain pair of network cards during communication is lower than a set value, the failure cause may be that the network cards have performance problems.

In particular, the performance detection between the supernodes specifically includes: when processors in two super nodes communicate with each other, data packets pass through a top-level switching network; when detecting the routing balance between two super nodes, firstly selecting the two super nodes, respectively selecting processors with physical numbers of 1-64 in the super nodes, and pairing the processors with the same physical numbers pairwise to form 64 pairs of processors; after the pairing is completed, a bidirectional bandwidth test program is operated on each pair of processors; 64 when the processors communicate simultaneously, if the routing distribution is uniform, no congestion is generated, and the difference value between the bandwidth and the bandwidth when two network cards in a single switch are connected is lower than a set value; after the test is finished, the average bandwidth of all the processors is sequenced from small to large, and the processors are printed and output, so that a tester can conveniently judge whether the routing distribution among the super nodes is uniform or not according to the test result.

The interconnection network fault detection and positioning method based on pairing test provided by the invention is used for carrying out functional fault detection and positioning based on pairing and two-time screening; performance and routing fault detection and location is performed based on layering and pairing. The invention relates to a large-scale interconnection network fault detection and positioning method based on pairing and multi-pass screening, which can quickly detect and position faults in a network, shorten the time for troubleshooting the faults and improve the availability and the usability of the large-scale network.

Drawings

Fig. 1 is a flowchart of a method for detecting and positioning a fault of an interconnection network based on a pairing test according to an embodiment of the present invention.

Detailed Description

The invention is further illustrated by the following figures and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It is also to be noted that, for the convenience of description, only a part of the contents, not all of the contents, which are related to the present invention, are shown in the drawings, and unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

Referring to fig. 1, fig. 1 is a flowchart of a method for detecting and locating a fault of an interconnection network based on a pairing test according to an embodiment of the present invention.

Themethod 100 for detecting and positioning faults of an interconnection network based on pairing test in this embodiment includes:

functional fault detection and localization 101:

firstly, pairing all network cards in the system, and running a function verification test question on each pair of network cards. The functional verification test questions adopt a ping-pong mode to send data, one party fills a sending buffer zone according to a data layout algorithm, and then the data in the buffer zone is sent to the other party; the other side checks the data after receiving the data, then fills a sending buffer area according to a data layout algorithm, and sends the data back; repeatedly and iteratively executing the process; and when the data receiving and sending between the two network cards have errors, recording the data in a corresponding result file. In this embodiment, the following data layout methods are adopted: the method for distributing the numbers by sequential filling and circular right shift comprises the following steps: the numbers 0, 1, 2.., 255 are filled into the transmit buffer in sequence, with the data position shifted to the right by one bit per iteration. A fixed numerical value distribution method: in any iteration, each buffer location is filled with a fixed value, such as 0xf, 0xff, etc. And thirdly, a jumping filling and circular right-shifting distribution algorithm: the numbers 0, 1, 2.., 255 are filled into the transmit buffer at step intervals, with the data location shifted to the right by one bit per iteration.

And secondly, scanning the test result after the test of all the network card pairs is finished, dividing the network cards into two subsets according to the test result, putting the network card pairs with correct results into a set S, and putting the network card pairs with wrong results into a set E. The network cards in the set E may have errors and need to be further screened.

Thirdly, for each network card in the set E, optionally selecting one network card from the set S for pairing, and rerunning the function verification test question; after the test is finished, scanning the test result, if the network card test result in the set E is normal, indicating that the network card has no error, and moving the network card into the set S; after two screening passes, the network cards in the set E are all wrong.

After the whole test is finished, the network card lists in the set are summarized, and then the message address, the message content and the error data of the data error are automatically or manually checked according to the result recording file, so that the specific reason of the message error is judged.

The routing tables are stored on the switches and each switch needs to have one copy. As nodes in the network are added and deleted, network routes may change, and thus, the problem of uneven distribution of network routes may occur. In addition, some nodes may have low message performance during operation. Possible causes for this phenomenon include: firstly, errors are more on optical fibers, so that the performance is low when communication between two network cards between super nodes is caused; secondly, errors are more on a network plug-in board, so that the performance of communication between two network cards in the super node is low; the network card has errors, which causes the low performance of the network card when communicating with other network cards, and the errors comprise: the PCIE link is degraded frequently, the access speed of the network card is slow, the signal of the HCA port is unstable, and the like. The super node refers to a set of processors and network cards, and the internal network cards have a full-connection interconnection structure.

Performance and routing failure detection and location 102:

firstly, performance detection in a switch chip: and pairing all the network cards in pairs, so that each network card is paired with the other network card in the switch chip where the network card is located. The specific pairing algorithm is as follows: pairing all the network cards in pairs, so that each network card is paired with another network card in the switch core where the network card is located, specifically comprising: every two adjacent network cards in the same switch core are paired, and after pairing is completed, a bidirectional bandwidth test program is operated on each pair of network cards; network cards in the same switch chip are paired and communicated at the same time, so that congestion cannot be generated; after the test is finished, sorting the average bandwidth of all network card pairs from small to large, and printing the network card pairs with low performance; in the test, if the performance of a certain pair of network cards is lower than the set value when the network cards communicate with each other, the failure cause is as follows: the network card has problems, and a single switch chip has problems. It should be noted that the performance of a certain pair of network cards lower than the set value in the set value when communicating with each other is determined according to the actual application.

Secondly, detecting the performance in the super node: after the above tests, after the network card is judged to have no problem, the next step needs to check whether the network card has a performance problem; firstly, pairing all the network cards in pairs, so that each network card is paired with the other network card in the supernode where the network card is located. The specific pairing algorithm is as follows: if the physical number of the network card in the super node is n, the physical number of the network card matched with the super node is (n + 128)% 256; after the pairing is completed, a bidirectional bandwidth test program is operated on each pair of network cards; when 256 network cards in a super node are paired and communicate simultaneously, if the routing distribution is uniform, congestion cannot be generated, the difference value between the bandwidth and the bandwidth when two network cards in a single switch are in butt joint is lower than a set value, it should be noted that the difference value between the bandwidth and the bandwidth when two network cards in a single switch are in butt joint is lower than the set value means that the difference value is very small, and the size of the set value is determined according to the actual application condition; after the test is finished, sorting the average bandwidth of all network card pairs from small to large, and printing the network card pairs with low performance; in the step of test, if the performance of a certain pair of network cards during mutual communication is lower than the set value, the failure cause may be that the network card has a performance problem, and it should be noted that the size of the set value in the performance of the certain pair of network cards during mutual communication is lower than the set value is determined according to the actual application situation.

Thirdly, detecting the performance among the super nodes: when processors in two super nodes communicate with each other, data packets pass through a top-level switching network; when detecting the routing balance between two super nodes, firstly selecting the two super nodes, respectively selecting processors corresponding to physical numbers in the super nodes, and pairing the processors with the same physical numbers pairwise; after the pairing is completed, a bidirectional bandwidth test program is operated on each pair of processors; when each pair of processors simultaneously communicate, if the routing distribution is uniform, no congestion is generated; after the test is finished, the average bandwidth of all the processors is sequenced from small to large, and the processors are printed and output, so that a tester can conveniently judge whether the routing distribution among the super nodes is uniform or not according to the test result. The processor numbers may be 1-64 in a specific application, and thus, the performance detection process among the supernodes is as follows: when processors in two super nodes communicate with each other, data packets pass through a top-level switching network; when detecting the routing balance between two super nodes, firstly selecting the two super nodes, respectively selecting processors with physical numbers of 1-64 in the super nodes, and pairing the processors with the same physical numbers pairwise to form 64 pairs of processors; after the pairing is completed, a bidirectional bandwidth test program is operated on each pair of processors; 64 when the processors communicate simultaneously, if the routing distribution is uniform, no congestion occurs, the difference between the bandwidth and the bandwidth when two network cards in a single switch are connected is lower than a set value, it should be noted that the difference between the bandwidth and the bandwidth when two network cards in a single switch are connected is lower than the set value, which means that the difference is very small, and the size of the set value is determined according to the actual application condition; after the test is finished, the average bandwidth of all the processors is sequenced from small to large, and the processors are printed and output, so that a tester can conveniently judge whether the routing distribution among the super nodes is uniform or not according to the test result.

The invention utilizes the large-scale network error detection technology, the route balance detection technology and the network performance detection technology to rapidly detect and position the fault in the network, shortens the time for troubleshooting the fault and improves the availability and the usability of the large-scale network.

Those skilled in the art will appreciate that all of the above embodiments can be implemented by a computer program, which can be stored in a computer readable storage medium, and the program can include the procedures of the embodiments of the methods described above when executed. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like.

It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims (6)

1. A method for detecting and positioning faults of an interconnection network based on pairing test is characterized by comprising the following steps:

functional fault detection and location: pairing all network cards in the system, and running a function verification test question on each pair of network cards; after the test of all the network card pairs is finished, scanning the test result, dividing the network cards into two subsets according to the test result, putting the network card pairs with correct results into a set S, and putting the network card pairs with wrong results into a set E; thirdly, for each network card in the set E, optionally selecting one network card from the set S for pairing, and rerunning the function verification test question; after the test is finished, scanning the test result, if the network card test result in the set E is normal, indicating that the network card has no error, and moving the network card into the set S; through two-time screening, the network cards in the set E are all wrong;

performance and routing fault detection and location: firstly, performance detection in a switch chip: pairing all the network cards in pairs, so that each network card is paired with another network card in the switch core where the network card is located; secondly, detecting the performance in the super node: after the network card has no problem, checking whether the network card has performance problem; firstly, pairing all network cards in pairs, so that each network card is paired with another network card in a supernode where the network card is located; thirdly, detecting the performance among the super nodes: when processors in two super nodes communicate with each other, data packets pass through a top-level switching network; when detecting the routing balance between two super nodes, firstly selecting the two super nodes, respectively selecting processors corresponding to physical numbers in the super nodes, and pairing the processors with the same physical numbers pairwise; after the pairing is completed, a bidirectional bandwidth test program is operated on each pair of processors; when each pair of processors simultaneously communicate, if the routing distribution is uniform, no congestion is generated; after the test is finished, the average bandwidth of all the processors is sequenced from small to large, and the processors are printed and output, so that a tester can conveniently judge whether the routing distribution among the super nodes is uniform or not according to the test result.

2. The interconnection network fault detection and location method based on pairing test as claimed in claim 1, wherein running functional verification test questions on each pair of network cards specifically includes: the functional verification test questions adopt a ping-pong mode to send data, one party fills a sending buffer zone according to a data layout algorithm, and then the data in the buffer zone is sent to the other party; the other side checks the data after receiving the data, then fills a sending buffer area according to a data layout algorithm, and sends the data back; repeatedly and iteratively executing the process; and when the data receiving and sending between the two network cards have errors, recording the data in a corresponding result file.

3. The pairing test-based interconnect network fault detection and location method of claim 1, wherein the functional fault detection and location further comprises: and summarizing the network card list in the set E, and automatically or manually checking the message address, the message content and the error data of the data error according to the result recording file, thereby judging the specific reason of the message error.

4. The interconnection network fault detection and location method based on pairing test as claimed in claim 1, wherein the performance detection in the switch core specifically comprises: pairing all the network cards in pairs, so that each network card is paired with another network card in the switch core where the network card is located, specifically comprising: every two adjacent network cards in the same switch core are paired, and after pairing is completed, a bidirectional bandwidth test program is operated on each pair of network cards; network cards in the same switch chip are paired and communicated at the same time, so that congestion cannot be generated; after the test is finished, sorting the average bandwidth of all network card pairs from small to large, and printing the network card pairs with low performance; in the test, if the performance of a certain pair of network cards is lower than the set value when the network cards communicate with each other, the failure cause is as follows: the network card has problems, and a single switch chip has problems.

5. The interconnection network fault detection and positioning method based on pairing test as claimed in claim 1, wherein in the performance detection in the supernode, all the network cards are paired pairwise, so that each network card is paired with another network card in the supernode where the network card is located, specifically comprising: if the physical number of the network card in the super node is n, the physical number of the network card matched with the super node is (n + 128)% 256; after the pairing is completed, a bidirectional bandwidth test program is operated on each pair of network cards; when 256 network cards in a super node are paired and communicated with each other at the same time, if the routing distribution is uniform, congestion cannot be generated, and the difference value between the bandwidth and the bandwidth when two network cards in a single switch are in butt joint is lower than a set value; after the test is finished, sorting the average bandwidth of all network card pairs from small to large, and printing the network card pairs with low performance; in the test, if the performance of a certain pair of network cards is lower than the set value when the network cards communicate with each other, the failure cause is that the network cards have performance problems.

6. The interconnection network fault detection and location method based on pairing test as claimed in one of claims 1 to 5, wherein the performance detection between the supernodes specifically includes: when processors in two super nodes communicate with each other, data packets pass through a top-level switching network; when detecting the routing balance between two super nodes, firstly selecting the two super nodes, respectively selecting processors with physical numbers of 1-64 in the super nodes, and pairing the processors with the same physical numbers pairwise to form 64 pairs of processors; after the pairing is completed, a bidirectional bandwidth test program is operated on each pair of processors; 64 when the processors communicate simultaneously, if the routing distribution is uniform, no congestion is generated, and the difference value between the bandwidth and the bandwidth when two network cards in a single switch are connected is lower than a set value; after the test is finished, the average bandwidth of all the processors is sequenced from small to large, and the processors are printed and output, so that a tester can conveniently judge whether the routing distribution among the super nodes is uniform or not according to the test result.

Images