Summary of the invention
The embodiment provides a kind of distributed data transport method and system based on load balancing, to improve the efficiency of the transfer of data of distributed data transport system.
The invention provides following scheme:
A kind of distributed data transport method based on load balancing, each transmission node in distributed data transport system is divided into many levels, multiple transmission node is comprised at each level, in distributed data transport system, arrange a load balancing management node, described method specifically comprises:
Described load balancing management node and each transmission node carry out data communication, receive the node status information that each transmission node sends over;
Described load balancing management node, according to the node status information of each transmission node received, controls the data transfer throughput between each transmission node according to the load balancing of setting.
Described load balancing management node and each transmission node carry out data communication, receive the node status information that each transmission node sends over, comprising:
In each transmission node in distributed data transport system, heartbeat module is set, the node status information of each transmission node Real-time Obtaining transmission node self;
Each transmission node adopts remote procedure call protocol to send the heartbeat message carrying described node status information to described load balancing management node according to the time interval of setting by described heartbeat module, described load balancing management node receives by designated port the heartbeat message that each transmission node sends over, and obtains the node status information of carrying in described heartbeat message.
Described node status information comprises meshed network flow rate, node maximum bandwidth, cpu load, internal memory use amount.
Described load balancing management node, according to the node status information of each transmission node received, controls the data transfer throughput between each transmission node according to the load balancing of setting, comprising:
Described load balancing management node does not receive the heartbeat message that certain transmission node sends within a certain period of time, and described load balancing management node confirms that the state of certain transmission node described was for losing efficacy;
Described load balancing management node judges whether that existence and certain transmission node described are positioned at same level, same position unit, and state is other transmission node effective, if, then be positioned at same level, same position unit described in described load balancing management node selection, and state is the standby transport node of other transmission node effective as certain transmission node described, is directed to described standby transport node by sending to the data traffic of certain transmission node described; Otherwise, perform following treatment step;
Described load balancing management node judges whether that existence and certain transmission node described are positioned at same level, state is other transmission node effective, if, then be positioned at same level described in described load balancing management node selection, state is the standby transport node of other transmission node effective as certain transmission node described, is directed to described standby transport node by sending to the data traffic of certain transmission node described.
Described load balancing management node, according to the node status information of each transmission node received, controls the data transfer throughput between each transmission node according to the load balancing of setting, comprising:
After described load balancing management node receives the heartbeat message that certain transmission node sends over, obtain the node status information of carrying in this heartbeat message, according to this node status information judge the CPU use amount of certain transmission node more than 80%, internal memory use amount more than 80%, data cached size account for whole spatial cache size more than 90% and/or network bandwidth use amount more than 90%, then described load balancing management node confirms that the state of certain transmission node described is for overload;
Described load balancing management node judges whether that existence and certain transmission node described are positioned at same level, same position unit, and state is other transmission node effective, if, then be positioned at same level, same position unit described in described load balancing management node selection, and state is the standby transport node of other transmission node effective as certain transmission node described, is directed to described standby transport node by sending to all or part of data traffic of certain transmission node described; Otherwise, perform following treatment step;
Described load balancing management node judges whether that existence and certain transmission node described are positioned at same level, state is other transmission node effective, if, then be positioned at same level described in described load balancing management node selection, state is the standby transport node of other transmission node effective as certain transmission node described, is directed to described standby transport node by sending to all or part of data traffic of certain transmission node described.
Described load balancing management node, according to the node status information of each transmission node received, controls the data transfer throughput between each transmission node according to the load balancing of setting, comprising:
After described load balancing management node receives the follow-up heartbeat message sended over of certain transmission node, obtain the node status information of carrying in this heartbeat message, the state judging certain transmission node described according to this node status information be normal, then described load balancing management node the data traffic of certain transmission node described in sending to is led again from described standby transport node described in certain transmission node.
Based on a distributed data transport system for load balancing, comprising: multiple transmission node and a load balancing management node, each transmission node described is divided into many levels, comprises multiple transmission node at each level;
Described load balancing management node, for carrying out data communication with each transmission node, receive the node status information that each transmission node sends over, according to the node status information of each transmission node received, according to the load balancing of setting, the data transfer throughput between each transmission node is controlled.
Described transmission node, for arranging heartbeat module, the node status information of Real-time Obtaining transmission node self; Remote procedure call protocol is adopted to send the heartbeat message carrying described node status information to described load balancing management node according to the time interval of setting by described heartbeat module;
Described load balancing management node, for being received the heartbeat message that each transmission node sends over by designated port, obtains the node status information of carrying in described heartbeat message.
Described load balancing management node, specifically for not receiving the heartbeat message that certain transmission node sends within a certain period of time, confirms that the state of certain transmission node described was for losing efficacy;
Judge whether that existence and certain transmission node described are positioned at same level, same position unit, and state is other transmission node effective, if, same level, same position unit is positioned at described in then selecting, and state is the standby transport node of other transmission node effective as certain transmission node described, is directed to described standby transport node by sending to the data traffic of certain transmission node described; Otherwise, perform following treatment step;
Judge whether that existence and certain transmission node described are positioned at same level, state is other transmission node effective, if, be positioned at same level described in then selecting, state is the standby transport node of other transmission node effective as certain transmission node described, is directed to described standby transport node by sending to the data traffic of certain transmission node described.
Described load balancing management node, after receiving heartbeat message that certain transmission node sends over, obtain the node status information of carrying in this heartbeat message, according to this node status information judge the CPU use amount of certain transmission node more than 80%, internal memory use amount more than 80%, data cached size account for whole spatial cache size more than 90% and/or network bandwidth use amount more than 90%, then described load balancing management node confirms that the state of certain transmission node described is for overload;
Judge whether that existence and certain transmission node described are positioned at same level, same position unit, and state is other transmission node effective, if, same level, same position unit is positioned at described in then selecting, and state is the standby transport node of other transmission node effective as certain transmission node described, is directed to described standby transport node by sending to all or part of data traffic of certain transmission node described; Otherwise, perform following treatment step;
Judge whether that existence and certain transmission node described are positioned at same level, state is other transmission node effective, if, be positioned at same level described in then selecting, state is the standby transport node of other transmission node effective as certain transmission node described, is directed to described standby transport node by sending to all or part of data traffic of certain transmission node described.
Described load balancing management node, after receiving the follow-up heartbeat message sended over of certain transmission node, obtain the node status information of carrying in this heartbeat message, the state judging certain transmission node described according to this node status information be normal, then certain transmission node described in the data traffic of certain transmission node described in sending to again being led from described standby transport node.
The technical scheme provided as can be seen from the embodiment of the invention described above, the embodiment of the present invention by arranging load balancing management node in distributed data transport system, this load balancing management node is by communicating with each transmission node, obtain the node status information of each transmission node, thus achieve and whole data transmission procedure effectively detected and manages, can control the data transfer throughput between each transmission node according to the load balancing of the node status information of each transmission node according to setting, the network architecture of real-time adjustment distributed data transport system, thus improve the efficiency of the transfer of data of distributed data transport system.
Embodiment
For ease of the understanding to the embodiment of the present invention, be further explained explanation below in conjunction with accompanying drawing for several specific embodiment, and each embodiment does not form the restriction to the embodiment of the present invention.
Embodiment one
For above-mentioned analysis, the embodiment of the present invention proposes a kind of distributed data transport method with load-balancing mechanism, monitor by introducing the transmitting procedure of a load balancing management node to data, and according to the factor such as message transmission rate and node state, transfer of data flow process is adjusted in real time, ensure the transmission security of data, improve the fault-tolerance of transfer of data.
The network topology structure schematic diagram of a kind of distributed data transport system based on load-balancing mechanism that this embodiment provides as shown in Figure 1.In the distributed data transport system shown in Fig. 1, each transmission node is divided into many levels, comprises multiple transmission node at each level.A load balancing management node is also set in distributed data transport system, this load balancing management node communicates with each transmission node, detect in real time according to the transfer of data flow process of result of communication to whole distributed data transport system, and accomplish to adjust the network architecture in real time, reach the object ensureing data transmission security.
In embodiments of the present invention, arrange heartbeat module in each transmission node in distributed transmission process, described load balancing management node and each transmission node carry out data communication, receive the node status information that each transmission node sends over.Then, described load balancing management node, according to the node status information of each transmission node received, controls the data transfer throughput between each transmission node according to the load balancing of setting.
The processing procedure that above-mentioned load balancing management node is monitored each transmission node in distributed data transport system is as follows:
1, load balancing management node is to the monitoring of each node in distributed transmission process.
In each transmission node in distributed data transport system, heartbeat module is set, the node status information of each transmission node Real-time Obtaining transmission node self.Each transmission node adopts RPC(Remote Procedure Call Protocol according to the time interval of setting by described heartbeat module, remote procedure call protocol) heartbeat message carrying described node status information is sent to described load balancing management node, this node status information comprises meshed network flow rate, node maximum bandwidth, cpu load, internal memory use amount.The heartbeat message that in load balancing management node, open designated port sends for receiving each transmission node simultaneously.In order to improve node communication efficiency, each transmission node and management node directly adopt RPC to communicate.
In embodiments of the present invention, the state of transmission node can be divided into inefficacy, overload or effectively.When described load balancing management node does not receive the heartbeat message that certain transmission node sends within a certain period of time, described load balancing management node confirms that the state of certain transmission node described was for losing efficacy.Then, above-mentioned load balancing management node judges whether that existence and certain transmission node described are positioned at same level, same position unit, and state is other transmission node effective, if, then be positioned at same level, same position unit described in described load balancing management node selection, and state is the standby transport node of other transmission node effective as certain transmission node described, is directed to described standby transport node by sending to the data traffic of certain transmission node described; Otherwise, perform following treatment step;
Described load balancing management node judges whether that existence and certain transmission node described are positioned at same level, state is other transmission node effective, if, then be positioned at same level described in described load balancing management node selection, state is the standby transport node of other transmission node effective as certain transmission node described, is directed to described standby transport node by sending to the data traffic of certain transmission node described.
When the quantity of other transmission node above-mentioned is multiple, then a transmission node in load balancing management node Stochastic choice other transmission node multiple is as secondary node.In above-mentioned processing procedure, the transmission node of prioritizing selection and inefficacy is positioned at the standby transport node of same position unit, and above-mentioned same position unit can be same machine room etc.
2, the real-time adjustment of the distributed data transport network architecture.
A kind of load balancing management node that this embodiment provides carries out the principle schematic of data traffic loads equilibrium treatment as shown in Figure 2 to transmission node.Described load balancing management node obtains the node status information of carrying in this heartbeat message after receiving the heartbeat message that each transmission node sends over, and the node status information of each transmission node is stored in a database.
When described load balancing management node according to the node status information of certain transmission node stored in above-mentioned database judge the CPU use amount of certain transmission node more than 80%, internal memory use amount more than 80%, data cached size account for whole spatial cache size more than 90% and/or network bandwidth use amount more than 90%, then described load balancing management node confirms that the state of certain transmission node described is for overload;
Described load balancing management node judges whether that existence and certain transmission node described are positioned at same level, same position unit, and state is other transmission node effective, if, then be positioned at same level, same position unit described in described load balancing management node selection, and state is the standby transport node of other transmission node effective as certain transmission node described, to all or part of data traffic of certain transmission node described be sent to be directed to described standby transport node, thus alleviate the data traffic pressure of certain transmission node above-mentioned; Otherwise, perform following treatment step;
Described load balancing management node judges whether that existence and certain transmission node described are positioned at same level, state is other transmission node effective, if, then be positioned at same level described in described load balancing management node selection, state is the standby transport node of other transmission node effective as certain transmission node described, to all or part of data traffic of certain transmission node described be sent to be directed to described standby transport node, thus alleviate the data traffic pressure of certain transmission node above-mentioned.
Afterwards, after described load balancing management node receives the follow-up heartbeat message sended over of certain transmission node, obtain the node status information of carrying in this heartbeat message, the state judging certain transmission node described according to this node status information be normal, then described load balancing management node the data traffic of certain transmission node described in sending to is led again from described standby transport node described in certain transmission node.
When the quantity of other transmission node above-mentioned is multiple, then a transmission node in load balancing management node Stochastic choice other transmission node multiple is as secondary node.In above-mentioned processing procedure, the transmission node of prioritizing selection and inefficacy is positioned at the standby transport node of same position unit, and above-mentioned same position unit can be same machine room etc.
Embodiment two
The process that a kind of load balancing management node that this embodiment provides carries out node failure issue handling to transmission node is as follows, comprises following treatment step:
1, each transmission node to send to management node with the timing of RPC communication mode and carries the heartbeat message of node status information, and above-mentioned node status information comprises meshed network flow rate, node maximum bandwidth, cpu load, internal memory use amount, nodal cache size of data account for whole spatial cache size;
2, load balancing management node obtains the node status information of carrying in the heartbeat message that each transmission node sends over, and node status information is stored in a database;
3, when transmission node A does not send heartbeat message to load balancing management node within a period of time, then load balancing management node adjudicates this transmission node A is failure node;
4, whether the inquiry of load balancing management node exists and transmission node A other transmission nodes in Same Physical position such as at same machine room;
If 5 exist and the transmission node B of transmission node A in Same Physical position, then load balancing management node notifies that all nodes transmitting data to transmission node A transfer data to transmission node B; If do not exist, stochastic searching and transmission node A are at other transmission nodes C of same level;
6, load balancing management node notifies that all nodes transmitting data to transmission node A transfer data to transmission node C.Thus complete the processing procedure of transmission node A Problem of Failure.
The process that a kind of load balancing management node that this embodiment provides carries out node overload issue handling to transmission node is as follows, comprises following treatment step:
1, each transmission node to send to management node with the timing of RPC communication mode and carries the heartbeat message of node status information, and above-mentioned node status information comprises meshed network flow rate, node maximum bandwidth, cpu load, internal memory use amount, nodal cache size of data account for whole spatial cache size;
2, load balancing management node obtains the node status information of carrying in the heartbeat message that each transmission node sends over, and node status information is stored in a database;
3, load balancing management node is according to the node status information of each transmission node stored in database, uses following four conditions to detect each transmission node: a) the CPU use amount of this node is more than 80%; B) the internal memory use amount of this node is more than 80%; C) nodal cache size of data accounts for whole spatial cache size more than 90%, and namely mass data is not also transferred to next node; D) network bandwidth use amount of this node is more than 90%, namely toward the data traffic of this node-node transmission close to this node maximum bandwidth.
4, when transmission node A meets any one condition above-mentioned, the shunting of load balancing management node is transferred to the data traffic of this transmission node A, to alleviate the data traffic pressure of transmission node A;
5, whether the inquiry of load balancing management node exists and transmission node A other nodes in Same Physical position such as at same machine room;
If 6 exist and the transmission node B of transmission node A in Same Physical position, then load balancing management node random selecting sends the transmission node C of data toward this transmission node, and traffic steering transmission node C being transferred to transmission node A is to on the transmission node B of the same level of transmission node A; If do not exist, stochastic searching and node A are at other transmission nodes D of same level;
7, load balancing management node random selecting sends the transmission node C of data toward this transmission node, and traffic steering transmission node C being transferred to transmission node A is to on the transmission node D of the same level of transmission node A.Thus, complete the processing procedure of transmission node A overload problem.
Embodiment three
This embodiment offers a kind of distributed data transport system based on load balancing, its specific implementation structure as shown in Figure 3, specifically can comprise: multiple transmission node and a load balancing management node, each transmission node described is divided into many levels, comprises multiple transmission node at each level;
Described load balancing management node, for carrying out data communication with each transmission node, receives the node status information that each transmission node sends over; According to the node status information of each transmission node received, according to the load balancing of setting, the data transfer throughput between each transmission node is controlled.
Concrete, described transmission node, for arranging heartbeat module, the node status information of Real-time Obtaining transmission node self; Remote procedure call protocol is adopted to send the heartbeat message carrying described node status information to described load balancing management node according to the time interval of setting by described heartbeat module;
Described load balancing management node, for being received the heartbeat message that each transmission node sends over by designated port, obtains the node status information of carrying in described heartbeat message.
Concrete, described load balancing management node, specifically for not receiving the heartbeat message that certain transmission node sends within a certain period of time, confirms that the state of certain transmission node described was for losing efficacy;
Judge whether that existence and certain transmission node described are positioned at same level, same position unit, and state is other transmission node effective, if, same level, same position unit is positioned at described in then selecting, and state is the standby transport node of other transmission node effective as certain transmission node described, is directed to described standby transport node by sending to the data traffic of certain transmission node described; Otherwise, perform following treatment step;
Judge whether that existence and certain transmission node described are positioned at same level, state is other transmission node effective, if, be positioned at same level described in then selecting, state is the standby transport node of other transmission node effective as certain transmission node described, is directed to described standby transport node by sending to the data traffic of certain transmission node described.
Concrete, described load balancing management node, after receiving heartbeat message that certain transmission node sends over, obtain the node status information of carrying in this heartbeat message, according to this node status information judge the CPU use amount of certain transmission node more than 80%, internal memory use amount more than 80%, data cached size account for whole spatial cache size more than 90% and/or network bandwidth use amount more than 90%, then described load balancing management node confirms that the state of certain transmission node described is for overload;
Judge whether that existence and certain transmission node described are positioned at same level, same position unit, and state is other transmission node effective, if, same level, same position unit is positioned at described in then selecting, and state is the standby transport node of other transmission node effective as certain transmission node described, is directed to described standby transport node by sending to all or part of data traffic of certain transmission node described; Otherwise, perform following treatment step;
Judge whether that existence and certain transmission node described are positioned at same level, state is other transmission node effective, if, be positioned at same level described in then selecting, state is the standby transport node of other transmission node effective as certain transmission node described, is directed to described standby transport node by sending to all or part of data traffic of certain transmission node described.
Concrete, described load balancing management node, after receiving the follow-up heartbeat message sended over of certain transmission node, obtain the node status information of carrying in this heartbeat message, the state judging certain transmission node described according to this node status information be normal, then certain transmission node described in the data traffic of certain transmission node described in sending to again being led from described standby transport node.
With the system of the embodiment of the present invention carry out based on the detailed process of the distributed data transport of load balancing and preceding method embodiment similar, repeat no more herein.
In sum, the embodiment of the present invention by arranging load balancing management node in distributed data transport system, this load balancing management node is by communicating with each transmission node, obtain the node status information of each transmission node, thus achieve and whole data transmission procedure effectively detected and manages, can control the data transfer throughput between each transmission node according to the load balancing of the node status information of each transmission node according to setting, the network architecture of real-time adjustment distributed data transport system, thus improve the efficiency of the transfer of data of distributed data transport system.
Load balancing management node in the embodiment of the present invention can according to the node status information of transmission node, Timeliness coverage failed transmission node and overload transmission node, and in time the total data flow of failed transmission node is shunted, ensure that the data traffic flowing to failed transmission node can not be lost.In time all or part of data traffic of overload transmission node is shunted, alleviate the data traffic pressure of overload transmission node, ensure that overload transmission node can process the quantity flow received.
One of ordinary skill in the art will appreciate that: accompanying drawing is the schematic diagram of an embodiment, the module in accompanying drawing or flow process might not be that enforcement the present invention is necessary.
As seen through the above description of the embodiments, those skilled in the art can be well understood to the mode that the present invention can add required general hardware platform by software and realizes.Based on such understanding, technical scheme of the present invention can embody with the form of software product the part that prior art contributes in essence in other words, this computer software product can be stored in storage medium, as ROM/RAM, magnetic disc, CD etc., comprising some instructions in order to make a computer equipment (can be personal computer, server, or the network equipment etc.) perform the method described in some part of each embodiment of the present invention or embodiment.
Each embodiment in this specification all adopts the mode of going forward one by one to describe, between each embodiment identical similar part mutually see, what each embodiment stressed is the difference with other embodiments.Especially, for system or system embodiment, because it is substantially similar to embodiment of the method, so describe fairly simple, relevant part illustrates see the part of embodiment of the method.System described above and system embodiment are only schematic, the wherein said unit illustrated as separating component or can may not be and physically separates, parts as unit display can be or may not be physical location, namely can be positioned at a place, or also can be distributed in multiple network element.Some or all of module wherein can be selected according to the actual needs to realize the object of the present embodiment scheme.Those of ordinary skill in the art, when not paying creative work, are namely appreciated that and implement.
The above; be only the present invention's preferably embodiment, but protection scope of the present invention is not limited thereto, is anyly familiar with those skilled in the art in the technical scope that the present invention discloses; the change that can expect easily or replacement, all should be encompassed within protection scope of the present invention.Therefore, protection scope of the present invention should be as the criterion with the protection range of claim.