CN111061557A - Method and device for balancing distributed memory database load - Google Patents
Method and device for balancing distributed memory database loadDownload PDFInfo
- Publication number
- CN111061557A CN111061557ACN201811204395.1ACN201811204395ACN111061557ACN 111061557 ACN111061557 ACN 111061557ACN 201811204395 ACN201811204395 ACN 201811204395ACN 111061557 ACN111061557 ACN 111061557A
- Authority
- CN
- China
- Prior art keywords
- node
- fragments
- fragment
- processed
- time length
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/505—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the load
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Description
Technical Field
The present application relates to the field of big data technologies, and in particular, to a method and an apparatus for balancing distributed memory database loads.
Background
With the development of computer technology and network technology, distributed systems become more and more extensive in the field of big data, a distributed memory database is taken as a main database in the distributed systems, data is mainly put in a memory to provide an operation database, and compared with disk storage, the storage based on the distributed memory database has higher applicability because the reading and writing speed of the distributed memory database is far higher than that of a disk.
In a distributed memory database comprising a plurality of nodes, each node can process a fragment (a fragment is a logical unit consisting of data with a fixed number of rows), but during the operation of the distributed memory database, for some reasons (such as capacity expansion of a distributed memory database, no fragment of a newly added node, missing of new data distribution due to network delay of a certain node, restart and recovery of offline of a certain node, elimination of partial expired data in the previously stored fragments, missing of new data distribution during offline, and the like), the CPU occupancy rates of the nodes are different (i.e., load imbalance), the CPU occupancy of some of the nodes is high, the CPU occupancy of other nodes is low, the capacity of processing fragments of the nodes with high CPU occupancy is low, and the waiting time is long, so that the fragment processing efficiency is low.
Disclosure of Invention
In order to solve the problems in the prior art, embodiments of the present invention provide a method and an apparatus for balancing distributed memory database loads. The technical scheme is as follows:
in a first aspect, a method for balancing distributed in-memory database load is provided, where the method includes:
acquiring the number of current fragments of each node in a target distributed memory database and an identifier of the current fragments, and acquiring a first time length and a second time length, wherein the first time length is the average time length of the single fragment processed on a local node, and the second time length is the average time length of the single fragment processed on a non-local node in a cross-network mode;
according to the number of the current fragments of each node, the identifier of the current fragments of each node, the first time length and the second time length, determining indication information of the fragments processed by each node in load balancing, wherein the indication information is used for indicating the fragments from each node;
and for each node, sending the indication information of the fragment of the node to the node so that the node acquires the fragment according to the indication information of the fragment of the node and processes the fragment.
Optionally, the obtaining the first duration and the second duration includes:
counting the time lengths of the first preset number of fragments which are respectively processed on the local node, and counting the time lengths of the second preset number of fragments which are respectively processed on the cross-network non-local node;
and determining that the counted average value of the time lengths of the first preset number of fragments respectively processed on the local nodes is a first time length, and determining that the counted average value of the time lengths of the second preset number of fragments respectively processed on the cross-network non-local nodes is a second time length.
Thus, the determined first duration and the second duration can be more accurate.
Optionally, the determining, according to the number of current fragments of each node, the identifier of the current fragment of each node, the first time length, and the second time length, indication information of a fragment processed by each node in load balancing, where the indication information is used to indicate a fragment from each node, includes:
determining the number of fragments from each node in the fragments processed by each node during load balancing according to the number of current fragments of each node, the first time length and the second time length;
and determining the indication information of the fragments processed by each node according to the number of the fragments from each node in the fragments processed by each node and the identification of the current fragment of each node.
Optionally, the indication information includes:
starting fragment identification and ending fragment identification in fragments from each node; or,
the number of fragments from each node and the starting fragment identifier; or,
the number of fragments from each node and the end fragment identification.
Optionally, the determining, according to the number of current fragments of each node, the first duration and the second duration, the number of fragments from each node in the fragments processed by each node in load balancing includes:
determining the number of fragments from each node in the fragments processed by each node in load balancing by using the principle that the TotalCost value is minimum, wherein all the nodes in the target distributed memory databaseThe sum of the number of the current fragments of the point is equal to the sum of the number of the fragments processed on all the nodes during load balancing, the TotalCost is the sum of the variances of the total time length of the fragments processed by each node in the target distributed memory database,
Costktotal duration of fragment processing for node k, CostkIncluding processing the duration X of the processing of the fragment on the node k by the node kkkT and the time length of the node k for processing the fragments of other nodes
t is a first duration, ct is a second duration, n is the number of nodes in the target distributed memory database, XkkProcessing the number of fragments stored on the node k for the node k, XikAnd processing the number of fragments stored on the node i for the node k, wherein the node k is any node in the target distributed memory database, and k is greater than or equal to 1.
Optionally, the determining, in the principle of minimizing the TotalCost value, the number of fragments from each node in the fragments processed by each node in load balancing includes:
converting the TotalCost into a matrix form TotalCost ═ XTHX, wherein X ═ X (X)11,X12,...,X1n,...Xij,...,Xn1,Xn2,...,Xnn)T，XijProcessing the number of fragments on the ith node for the jth node, wherein i, j and n are positive integers, H is a symmetric matrix, and the sum of elements in X is less than or equal to the sum of the current number of fragments of all nodes in the target distributed memory database;
with XTAnd determining the number of fragments from each node in the fragments processed by each node in load balancing by taking the minimum value of HX as a principle.
Optionally, the obtaining the number of current fragments of each node in the target distributed memory database includes:
when a data query request is received, acquiring the number of current fragments of each node in a target distributed memory database;
the method further comprises the following steps:
receiving the result of processing the fragments of each node in the target distributed memory database;
summarizing the results of processing the fragments of each node to obtain a total result;
and feeding back the total result.
In a second aspect, an apparatus for balancing distributed in-memory database load is provided, the apparatus comprising:
the system comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring the number of current fragments of each node in a target distributed memory database and the identifier of the current fragments, and acquiring a first time length and a second time length, the first time length is the average time length of the single fragment processed on a local node, and the second time length is the average time length of the single fragment processed on a non-local node in a cross-network mode;
a determining module, configured to determine, according to the number of current fragments of each node, an identifier of the current fragment of each node, the first time length, and the second time length, indication information of a fragment processed by each node in load balancing, where the indication information is used to indicate the fragment from each node;
and the sending module is used for sending the indication information of the fragments of the nodes to the nodes so that the nodes acquire the fragments according to the indication information of the fragments of the nodes for processing.
Optionally, the obtaining module is configured to:
counting the time lengths of the first preset number of fragments which are respectively processed on the local node, and counting the time lengths of the second preset number of fragments which are respectively processed on the cross-network non-local node;
and determining that the counted average value of the time lengths of the first preset number of fragments respectively processed on the local nodes is a first time length, and determining that the counted average value of the time lengths of the second preset number of fragments respectively processed on the cross-network non-local nodes is a second time length.
Optionally, the determining module is configured to:
determining the number of fragments from each node in the fragments processed by each node during load balancing according to the number of current fragments of each node, the first time length and the second time length;
and determining the indication information of the fragments processed by each node according to the number of the fragments from each node in the fragments processed by each node and the identification of the current fragment of each node.
Optionally, the indication information includes:
starting fragment identification and ending fragment identification in fragments from each node; or,
the number of fragments from each node and the starting fragment identifier; or,
the number of fragments from each node and the end fragment identification.
Optionally, the determining module is configured to:
determining the number of fragments from each node in the fragments processed by each node in load balancing on the basis of the minimum value of TotalCost, wherein the sum of the current numbers of the fragments of all nodes in the target distributed memory database is equal to the sum of the numbers of the fragments processed on all nodes in load balancing, TotalCost is the sum of the variances of the total time lengths of the fragments processed by each node in the target distributed memory database,
Costktotal duration of fragment processing for node k, CostkIncluding processing the duration X of the processing of the fragment on the node k by the node kkkT and the node kManaging the duration of fragmentation of other nodes
t is a first duration, ct is a second duration, n is the number of nodes in the target distributed memory database, XkkProcessing the number of fragments stored on the node k for the node k, XikAnd processing the number of fragments stored on the node i for the node k, wherein the node k is any node in the target distributed memory database, and k is greater than or equal to 1.
Optionally, the determining module is configured to:
converting the TotalCost into a matrix form TotalCost ═ XTHX, wherein X ═ X (X)11,X12,...,X1n,...Xij,...,Xn1,Xn2,...,Xnn)T，XijProcessing the number of fragments on the ith node for the jth node, wherein i, j and n are positive integers, H is a symmetric matrix, and the sum of elements in X is less than or equal to the sum of the current number of fragments of all nodes in the target distributed memory database;
with XTAnd determining the number of fragments from each node in the fragments processed by each node in load balancing by taking the minimum value of HX as a principle.
Optionally, the obtaining module is configured to:
when a data query request is received, acquiring the number of current fragments of each node in a target distributed memory database;
the device further comprises:
the receiving module is used for receiving the result of processing the fragments of each node in the target distributed memory database; summarizing the results of processing the fragments of each node to obtain a total result;
the sending module is further configured to feed back the total result.
In a third aspect, a computer-readable storage medium is provided, in which a computer program is stored which, when being executed by a processor, carries out the method steps of the first aspect described above.
In a fourth aspect, a management device is provided, comprising a processor and a memory, wherein the memory is configured to store a computer program; the processor is configured to execute the program stored in the memory, so as to implement the method steps of the first aspect.
The technical scheme provided by the embodiment of the invention has the beneficial effects that at least:
in the embodiment of the present invention, when balancing the load of the distributed memory database, the indication information of the fragment processed by each node in load balancing may be determined based on the number of the current fragments of each node, the average duration of the processing of the single fragment on the local node, and the average duration of the processing of the single fragment on the non-local node in a cross-network manner, where the indication information is used to indicate the fragment from each node and notify each node, and each subsequent node may acquire the fragment for processing based on the indication information. In this way, the fragments to be processed calculated during load balancing are determined firstly, and then the fragments to be processed calculated during load balancing are processed by each node, so that load balancing can be ensured as much as possible, and the fragment processing efficiency is improved.
Drawings
Fig. 1 is a system diagram for balancing loads of a distributed memory database according to an embodiment of the present invention;
fig. 2 is a flowchart of a method for balancing distributed internal memory database loads according to an embodiment of the present invention;
fig. 3 is a schematic diagram of sending indication information according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of balancing distributed memory database loads according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of balancing distributed memory database loads according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of a management device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, embodiments of the present invention will be described in detail with reference to the accompanying drawings.
To facilitate understanding of the embodiments of the present invention, a system architecture related to the embodiments of the present invention and concepts related to the terms are first described below.
The embodiment of the invention can be applied to a distributed system, wherein the distributed system is a set of software system established on a network and mainly comprises a distributed memory database, as shown in fig. 1, the distributed memory database comprises a plurality of nodes, and the nodes can be servers.
The distributed memory database stores data in a memory for operation, and has higher reading and writing speed compared with a magnetic disk.
And the local node fragments the stored nodes.
Spark, itself a distributed computing framework, is a top level item of the Apache foundation.
SQL (Structured Query Language), a database Query and programming Language, is used to access data and Query, update, and manage relational database systems.
The shard is a data storage mode in a database, and is a logic unit formed by data with fixed line number. Can be region in HBase, rowgroup in partial/RCFile, etc. When Spark calculation is used, one slice may correspond to one partition of Spark.
A quadratic polynomial, where the number of unknowns is n and the power of each term is 2, is called quadratic.
The load balancing mainly refers to the CPU occupancy rate balancing in the embodiment of the invention, namely the CPU occupancy rates of all the nodes are basically the same.
The embodiment of the invention provides a method for balancing distributed memory database load, wherein an execution main body of the method can be management equipment, and the management equipment can be an independent node or a certain node in the distributed memory database.
The management device may be provided with a processor, a storage and a transceiver, the processor may be configured to perform processing for balancing the load of the distributed memory database, the storage may be configured to perform data required and generated in the process of balancing the load of the distributed memory database, and the transceiver may be configured to receive and transmit data.
Before implementation, it is assumed that the processing capacity of each node is the same, the network capacity of each node is consistent, and the size of the fragments is the same. The implementation of the present invention is described by taking Spark SQL as a query processing engine of a distributed memory database.
An embodiment of the present invention provides a method for balancing a load of a distributed memory database, and as shown in fig. 2, an execution flow of the method may be as follows:
The target distributed memory database is any distributed memory database, a first time length and a second time length can be stored in the management device in advance, the first time length is the average time length of a single fragment processed on a local node, namely the average time length of the single fragment processed on a node for storing the fragment, the second time length is the average time length of the single fragment processed on a non-local node in a cross-network mode, namely the average time length of the single fragment not processed on the node for storing the fragment, the second time length comprises two parts, the first part is the time length for transmitting the fragment, and the second part is the time length for processing the fragment.
In implementation, each node in the distributed memory database may periodically report its own fragmentation number and fragmentation identifier to the management device, and the management device may store the received fragmentation number, fragmentation identifier, and node identifier correspondingly. Subsequently, the management device wants to balance the load of each node in the in-memory database, may obtain the number of stored current fragments of each node and the identifier of the fragments, and may obtain the first duration and the second duration that are stored in advance.
Or, the management device may send a fragment acquisition request to each node, and each node may send its own fragment number and fragment identifier to the management device, so that the management device may acquire the current fragment number and fragment identifier of each node, in order to balance the load of each node in the in-memory database. And the management apparatus may acquire the first time period and the second time period stored in advance.
It should be noted that, for a certain node, the current shard refers to the shard currently stored by the node.
Optionally, the process of obtaining the first duration and the second duration may be as follows:
counting the time lengths of the first preset number of fragments which are respectively processed on the local node, and counting the time lengths of the second preset number of fragments which are respectively processed on the cross-network non-local node; and determining that the average value of the counted time lengths of the first preset number of fragments respectively processed on the local node is a first time length, and determining that the average value of the counted time lengths of the second preset number of fragments respectively processed on the cross-network non-local node is a second time length.
The first preset number and the second preset number may be preset and stored in the management device, for example, the first preset number is 1000, the second preset number is 1500, and the first preset number and the second preset number may be the same, for example, the first preset number and the second preset number are both 1000.
In implementation, in the target distributed memory database, each node may count the time length for which the stored fragments are processed on the node itself, and report the time length to the management device, and after receiving the time length for which the first preset number of fragments are processed on the local node, the management device may calculate an average value of the first preset number of time lengths, and determine the average value as the first time length.
In the target distributed memory database, each node may count the time length for processing the fragments stored in other nodes (including the time length for obtaining the fragments from other nodes and the time length for obtaining post-processing), and report the counted time length to the management device, and after receiving the time length for which the processing of a second preset number of fragments on the cross-network non-local node is completed, the management device may calculate an average value of the second preset number of time lengths, and determine the average value as the second time length.
Optionally, load balancing considered in the embodiment of the present invention is CPU balancing, so that when a data query request is received, the processing instep 201 may be started to be executed:
and when a data query request is received, acquiring the current fragmentation number of each node in the target distributed memory database.
In implementation, a user may input a content to be queried in a search input box, click a search key, trigger a sending of a search request to a management device, and the management device may obtain the number of current segments of each node in a target distributed memory database (the obtaining manner is described in detail above, and is not described here again). The query processing engine for this process may be Spark SQL.
Therefore, when the fragmentation processing is carried out, the load imbalance of the distributed memory database is increased, so that the load balancing processing can be started when a data query request is received, the processing resources can be saved, and the query performance is improved.
In implementation, after obtaining the number of current fragments of each node, the identifier of the current fragment of each node, the first time length, and the second time length, the management device may determine, according to the number of current fragments of each node, the identifier of the current fragment of each node, the first time length, and the second time length, indication information of fragments processed by each node in load balancing, where the indication information is used to indicate the fragments from each node.
Optionally, the number of fragments from each node may be determined first, and then the indication information is determined, and the corresponding processing instep 202 may be as follows:
determining the number of fragments from each node in the fragments processed by each node during load balancing according to the number of current fragments of each node, the first time length and the second time length; and determining the indication information of the fragments processed by each node according to the number of the fragments from each node in the fragments processed by each node and the identification of the current fragment of each node.
In implementation, after obtaining the number of current fragments of each node, the identifier of the current fragment of each node, the first time length and the second time length, the management device may determine the number of fragments from each node in the fragments processed by each node in load balancing. For the current fragment of the target node (any one of all nodes), the management device may determine all nodes processing the current fragment of the target node and the number of each of all nodes processing the current fragment, then arrange the sequence numbers of all the nodes in a descending order, the sequence numbers of all the nodes are sequentially N1, N2, N3 and N4 … from the descending order, the number of the current fragment of the target node is respectively N1, N2, N3 and N4 …, the node with the sequence number of N1 determines the identifiers of the first N1 fragments from the current fragment of the target node, then the node with the sequence number of N2 determines the identifiers of the N2 fragments adjacent to the first N1 fragments from the current fragment of the target node, then the node with the sequence number of N3 determines the identifiers of the N3 fragments adjacent to the N1+ N2 fragments from the current fragment of the target node, according to the mode, the identifiers of the fragments in the current fragments of the target nodes to be processed by all the nodes are sequentially determined, then the indication information is determined based on the identifiers of the fragments, and according to the mode, the indication information of the fragments processed by each node in the distributed memory database can be determined.
For example, there are three nodes, node a, node b, and node c, where the number of fragments stored by node a is 10, the number of fragments stored by node b is 3, the number of fragments stored by node c is 2, and to implement load balancing, the number of fragments stored by node a in processing itself is 5, the number of fragments stored by node a in processing node b in processing node a is 0, the number of fragments stored by node a in processing node c in processing node a is 0, the number of fragments stored by node b in processing node b is 3, the number of fragments stored by node b in processing node a in processing node b is 2, the number of fragments stored by node b in processing node c in processing node b is 0, the number of fragments stored by node c in processing node a is 2, the number of fragments stored by node c in processing node a is 3, and the number of fragments stored by node c in processing node b is 0. The sequence numbers of the nodes are sequentially a node a, a node b and a node c from small to large, for the node a, the node a only processes fragments stored by the node a, the first 5 fragments can be obtained, and the indication information of the fragments processed by the node a is only used for indicating the first 5 fragments. For the node b, the node a does not process the fragment of the node b, the fragment of the node b processing the node a may be the 6 th fragment and the 7 th fragment, the node b processes the 3 own fragments, and the indication information of the fragment processed by the node b is only used for indicating the 6 th fragment and the 7 th fragment of the node a and the fragment stored by the node b. For the node c, the node a and the node b do not process the fragment of the node c, the fragment of the node a processed by the node c may be from the 8 th fragment to the 10 th fragment, the node c processes two fragments stored by itself, and the indication information of the fragment processed by the node c is only used for indicating the fragments from the 8 th fragment to the 10 th fragment of the node a and the fragments stored by itself.
In addition, for the current segment of the target node (any one of all nodes), the management device may determine all nodes that process the current segment of the target node and the number of each of all nodes that process the current segment, then arrange the sequence numbers of all nodes in a descending order, where the sequence numbers of all nodes are N1, N2, N3, and N4 … in that order, the number of the current segments that process the target node is N1, N2, N3, and N4 … in that order, the target node is N3, the node with the sequence number N3 first determines the identities of the first N3 segments from the current segment of the target node, then the node with the sequence number N1 determines the identities of the N1 segments that are adjacent after the first N3 segments from the current segment of the target node, then the node with the sequence number N48 determines the identities of the N2 segments that are adjacent after the N1+ N3 segments from the current segment of the target node, according to the mode, the identifiers of the fragments in the current fragments of the target nodes to be processed by all the nodes are sequentially determined, then the indication information is determined based on the identifiers of the fragments, and according to the mode, the indication information of the fragments processed by each node in the distributed memory database can be determined.
For example, there are four nodes in total, node a, node b, node c, and node d, and the sequence numbers of the nodes are, in order from small to large, node a, node b, node c, and node d. The number of fragments stored by the node a is 10, the number of fragments stored by the node b is 3, the number of fragments stored by the node c is 2, and the number of fragments stored by the node d is 9. To realize load balancing, the node a only processes fragments stored by itself, the number of the fragments is 6, and the number of the fragments stored on the node d processed by the node a is 0. The number of the fragments stored in the node b processing node b is 3, the number of the fragments stored in the node b processing node a is 2, the number of the fragments stored in the node b processing node c is 0, and the number of the fragments stored in the node b processing node d is 1. The number of the fragments stored by the node c for processing the node c is 2, the number of the fragments stored by the node c for processing the node a is 2, the number of the fragments stored by the node c for processing the node b is 0, the number of the fragments stored by the node c for processing the node d is 2, and the number of the fragments stored by the node d for processing the node c is only 6.
For the node a, the node a only processes the fragments stored by the node a, and can acquire the first 6 fragments, and the indication information of the fragments processed by the node a is only used for indicating the first 6 fragments.
For the node d, the node d only processes the fragments stored by itself, and may obtain the first 6 fragments, and the indication information of the fragments processed by the node d is only used to indicate the first 6 fragments.
For the node b, the fragment of the node b processing the node a may be the 7 th fragment and the 8 th fragment, the fragment of the node b processing the node d may be the 7 th fragment, the node b processes the 3 own fragments, and the indication information of the fragment processed by the node b is only used for indicating the 7 th fragment of the node a, the fragment stored by the node b, and the 7 th fragment of the node d.
For the node c, the fragments of the node a processed by the node c may be the 9 th fragment and the 10 th fragment, the node c processes the 8 th fragment and the 9 th fragment of the node d, the node c processes two fragments stored by itself, and the indication information of the fragment processed by the node c is only used for indicating the 9 th fragment and the 10 th fragment of the node a, the 8 th fragment and the 9 th fragment of the node d, and the fragment stored by itself.
Optionally, the content of the indication information is various, and three possible contents are given as follows:
A. and the starting fragment identifier and the ending fragment identifier in the fragments from each node.
In an implementation, instep 202, after determining that the fragment of another node is to be processed, for each node, an identifier of a first fragment (i.e., a starting fragment identifier) and an identifier of a last fragment (i.e., an ending fragment identifier) of the fragment to be processed in another node may be obtained. The subsequent indication information includes the start fragment identifier and the end fragment identifier, that is, the start fragment identifier and the end fragment identifier can be used to find all the fragments to be processed.
B. The number of fragments from each node and the starting fragment identification.
In an implementation, instep 202, for each node, after determining that the fragment of another node is to be processed, an identifier of a first fragment of the fragments to be processed in another node (i.e., a starting fragment identifier) and a number of fragments of another node to be processed may be obtained. The subsequent indication information includes the starting fragment identifier and the number of fragments, that is, all fragments to be processed can be found by using the starting fragment identifier and the number of fragments.
C. The number of fragments from each node and the end fragment identification.
In an implementation, instep 202, for each node, after determining the fragments to be processed in another node, an identification of a last fragment of the fragments to be processed in another node (i.e., a starting fragment identification) and a number of fragments of another node to be processed may be obtained. The subsequent indication information includes the ending fragment identifier and the number of the fragments, that is, the ending fragment identifier and the number of the fragments can be used to find all the fragments to be processed.
Optionally, the number of the fragments from each node in the fragments processed by each node may be determined based on a principle that the processing time is the smallest, and the corresponding processing may be as follows:
determining the number of fragments from each node in the fragments processed by each node in load balancing by using the principle that the value of TotalCost is minimum, wherein the sum of the current fragments of all nodes in the target distributed memory database is equal to the sum of the fragments processed on all nodes in load balancing, TotalCost is the sum of the variances of the total time length of the fragments processed by each node in the target distributed memory database,
Costktotal duration of fragment processing for node k, CostkIncluding processing node k processing the duration X of a fragment on node kkkDuration of processing fragments of other nodes by t and node k
t is a first duration, ct is a second duration, n is the number of nodes in the target distributed memory database, XkkProcessing the number of fragments stored on node k for node k, XikAnd processing the number of fragments stored on the node i for a node k, wherein the node k is any node in the target distributed memory database, and k is greater than or equal to 1.
In the implementation, assume that there are n nodes, taking Node k (kth Node, Node _ k) as an example, during load balancing, it is assumed that all nodes in the distributed memory database are to process fragments on Node _ k, and after scheduling, each Node processes the number of fragments on Node k, and we mark X ask1,Xk2,…,Xkn. Thus, we can get a matrix of n × n:
in the n × n matrix, each column corresponds to a node, the kth column corresponds to a node k, and the number of fragments to be processed by the node k in load balancing can be obtained by adding elements in the kth column, and the number of fragments stored in the node k can be obtained by adding elements in the kth row and corresponding to a fragment stored in a node (i.e., a current fragment in a node) in each row.
Assuming that the first duration is represented by t and the second duration is represented by ct, in load balancing, the duration required for processing all the fragments on the node k is:
in the formula (2), XkkT is the time duration for processing node k to process its own fragment,
the length of time to process the shards of other nodes for node k.
The sum of the variances of the total durations of the nodes in the target distributed memory database may then be determined:
in the formula (3), the reaction mixture is,
in equation (4), meanCost represents an average value of the processing time of the fragments by each node in the target distributed memory database during load balancing.
Under the allocation of a certain fragment, the TotalCost is minimized, that is, the processing capability of each node in the target distributed memory database is considered to be exerted in a balanced manner, and load balancing is also achieved.
Optionally, on the principle of TotalCost minimum, the manner of calculating the number of fragments from each node in the fragments processed by each node may be as follows:
converting the TotalCost into a matrix form TotalCost ═ XTHX, wherein X ═ X (X)11,X12,...,X1n,...Xij,...,Xn1,Xn2,...,Xnn)T，XijProcessing the number of fragments on the ith node for the jth node, wherein i, j and n are positive integers, H is a symmetric matrix, and the sum of elements in X is less than or equal to the sum of the current fragments of all nodes in the target distributed memory database; with XTAnd determining the number of fragments from each node in the fragments processed by each node in load balancing by taking the minimum value of HX as a principle.
In practice, as can be seen from formula (4), X11,X12,...,X1n,...Xij,...,Xn1,Xn2,...,XnnAll are unknowns, so TotalCost is a polynomial with n x n unknowns, and each monomial of the polynomial is a quadratic, so TotalCost is a quadratic polynomial with n x n unknowns, based on the properties of the quadratic polynomial, there must be a symmetric matrix such that:
TotalCost＝XTHX (5)
in formula (5), X ═ X11,X12,...,X1n,...Xij,...,Xn1,Xn2,...,Xnn)T。
Thus, based on equation (5), we can convert the load balancing problem into a quadratic programming problem with constraints, that is:
taking the minimum value of TotalCost: MinXTHX, and the requirement that HX is less than or equal to b is met.
The matrix H is:
in the formula (6), the reaction mixture is,
in formula (7), YnIndicates the number of current shards (i.e., stored shards) on the nth node during load balancing, i.e., the number of current shards
In formula (7), Xk＝(Xk1,Xk2,...,Xkn)T,Yk＝(Nk,0,...,0)T。
Note that T represents transposition.
X in equation (7) can be determined by solving a quadratic optimal solution. In this way, the number of fragments from each node in the fragments processed by each node is determined.
It should be noted that the method for solving the quadratic programming optimal solution is already a conventional method in the prior art, and a lagrangian multiplier method may be used in the calculation process, which is not described herein again.
In implementation, as shown in fig. 3, for each node, the management device may send the indication information of the fragment of the node to the node, and after the node receives the indication information, the node may enable the received indication information to obtain the fragment indicated by the indication information for processing.
Optionally, when the indication information is a start fragment identifier and an end fragment identifier in the fragments from each node, for a certain node, if the node is to acquire a fragment from another node for processing, the start fragment identifier and the end fragment identifier of the fragment from the another node may be determined based on the indication information, and then the fragments between the start fragment identifier and the end fragment identifier (the fragments including the fragment of the start fragment identifier and the fragment of the end fragment identifier) are acquired from a storage area of the another node for processing.
Optionally, when the indication information is the number of fragments from each node and the start fragment identifier, for a certain node, if the node is to acquire a fragment from another node for processing, the number of fragments from the another node and the start fragment identifier may be determined based on the indication information, and then the number of fragments identified by the start fragment and the number of fragments adjacent to the fragments identified by the start fragment identifier are acquired from the storage area of the another node and are reduced by one fragment for processing.
Optionally, when the indication information is the number of fragments from each node and the end fragment identifier, for a node, if the node is to acquire a fragment from another node for processing, the number of fragments from the another node and the end fragment identifier may be determined based on the indication information, and then the number of fragments before the fragment with the end fragment identifier and the fragment with the end fragment identifier is acquired from the storage area of the another node, minus one fragment, for processing.
It should be noted that, when acquiring the fragment of another node, the above-mentioned acquisition may be performed based on a metadata table, and a storage area of each node and a storage location of the fragment of each node are recorded in the metadata table.
It should be further noted that, in the distributed memory database, the storage area of each node is a logical storage area.
It should be noted that, in the above-mentioned acquiring the fragment of the other node for processing, only the data of the fragment is read from the other node for processing, and is not written into the other node, so that compared with the fragment migration in the prior art, the time for writing the data of the fragment can be saved. And when the time length for completing the processing of the fragments is longer than the time length for transmitting the data (namely the time length for transmitting the fragments to another node), the CPU occupancy rate of the local node can be reduced by rapidly completing the processing of the fragments by other idle nodes, so that the load balancing capability for the scene is stronger.
Based on thestep 201 being executed when the data query request is received instep 201, and thus the query result needs to be fed back to the user afterstep 203, the corresponding processing may be as follows:
receiving the result of processing the fragments of each node in the target distributed memory database; summarizing the results of processing the fragments of each node to obtain a total result; and feeding back the total result.
In implementation, after obtaining the fragments and performing query processing, each node may respectively feed back the query results to the management device, and after receiving the query results, the management device may collect the received query results to obtain a total result (i.e., a query result), and then feed back the collected total result to the user, so that the user can check the total result.
In the embodiment of the present invention, when determining load balancing of the distributed memory database, indication information of a fragment processed by each node during load balancing may be determined based on the number of current fragments of each node, an average duration of processing of a single fragment performed on a local node, and an average duration of processing of a single fragment performed on a non-local node in a cross-network manner, where the indication information is used to indicate a fragment from each node and notify each node, and each subsequent node may acquire a fragment for processing based on the indication information. In this way, the fragments to be processed calculated during load balancing are determined firstly, and then the fragments to be processed calculated during load balancing are processed by each node, so that load balancing can be ensured as much as possible, and the fragment processing efficiency is improved.
Based on the same technical concept, an embodiment of the present invention further provides a device for balancing a load of a distributed memory database, as shown in fig. 4, where the device includes:
an obtainingmodule 410, configured to obtain the number of current fragments of each node in a target distributed memory database and an identifier of the current fragments, and obtain a first duration and a second duration, where the first duration is an average duration for a single fragment to be processed on a local node, and the second duration is an average duration for a single fragment to be processed on a non-local node in a cross-network manner;
a determiningmodule 420, configured to determine, according to the number of current fragments of each node, an identifier of the current fragment of each node, the first time length, and the second time length, indication information of a fragment processed by each node in load balancing, where the indication information is used to indicate a fragment from each node;
a sendingmodule 430, configured to send, to each node, the indication information of the node fragment to the node, so that the node acquires the fragment according to the indication information of the node fragment and processes the fragment.
Optionally, the obtainingmodule 410 is configured to:
counting the time lengths of the first preset number of fragments which are respectively processed on the local node, and counting the time lengths of the second preset number of fragments which are respectively processed on the cross-network non-local node;
and determining that the counted average value of the time lengths of the first preset number of fragments respectively processed on the local nodes is a first time length, and determining that the counted average value of the time lengths of the second preset number of fragments respectively processed on the cross-network non-local nodes is a second time length.
Optionally, the determiningmodule 420 is configured to:
determining the number of fragments from each node in the fragments processed by each node during load balancing according to the number of current fragments of each node, the first time length and the second time length;
and determining the indication information of the fragments processed by each node according to the number of the fragments from each node in the fragments processed by each node and the identification of the current fragment of each node.
Optionally, the indication information includes:
starting fragment identification and ending fragment identification in fragments from each node; or,
the number of fragments from each node and the starting fragment identifier; or,
the number of fragments from each node and the end fragment identification.
Optionally, the determiningmodule 420 is configured to:
determining the number of fragments from each node in the fragments processed by each node in load balancing on the basis of the minimum value of TotalCost, wherein the sum of the current numbers of the fragments of all nodes in the target distributed memory database is equal to the sum of the numbers of the fragments processed on all nodes in load balancing, TotalCost is the sum of the variances of the total time lengths of the fragments processed by each node in the target distributed memory database,
Costktotal duration of fragment processing for node k, CostkIncluding processing the duration X of the processing of the fragment on the node k by the node kkkT and the time length of the node k for processing the fragments of other nodes
t is a first duration, ct is a second duration, n is the number of nodes in the target distributed memory database, XkkProcessing the number of fragments stored on the node k for the node k, XikAnd processing the number of fragments stored on the node i for the node k, wherein the node k is any node in the target distributed memory database, and k is greater than or equal to 1.
Optionally, the determiningmodule 420 is configured to:
converting the TotalCost into a matrix form TotalCost ═ XTHX, wherein X ═ X (X)11,X12,...,X1n,...Xij,...,Xn1,Xn2,...,Xnn)T，XijProcessing the number of fragments on the ith node for the jth node, wherein i, j and n are positive integers, H is a symmetric matrix, and the sum of elements in X is less than or equal to the target fragmentThe sum of the current fragmentation numbers of all nodes in the distributed memory database;
with XTAnd determining the number of fragments from each node in the fragments processed by each node in load balancing by taking the minimum value of HX as a principle.
Optionally, the obtainingmodule 410 is configured to:
when a data query request is received, acquiring the number of current fragments of each node in a target distributed memory database;
as shown in fig. 5, the apparatus further includes:
areceiving module 440, configured to receive a result of processing the fragment by each node in the target distributed memory database; summarizing the results of processing the fragments of each node to obtain a total result;
the sendingmodule 430 is further configured to feed back the total result.
In the embodiment of the present invention, when balancing the load of the distributed memory database, the indication information of the fragment processed by each node in load balancing may be determined based on the number of the current fragments of each node, the average duration of the processing of the single fragment on the local node, and the average duration of the processing of the single fragment on the non-local node in a cross-network manner, where the indication information is used to indicate the fragment from each node and notify each node, and each subsequent node may acquire the fragment for processing based on the indication information. In this way, the fragments to be processed calculated during load balancing are determined firstly, and then the fragments to be processed calculated during load balancing are processed by each node, so that load balancing can be ensured as much as possible, and the fragment processing efficiency is improved.
It should be noted that: in the apparatus for balancing the load of the distributed memory database provided in the foregoing embodiment, when balancing the load of the distributed memory database, only the division of each functional module is used for illustration, and in practical applications, the function distribution may be completed by different functional modules according to needs, that is, the internal structure of the apparatus is divided into different functional modules, so as to complete all or part of the functions described above. In addition, the apparatus for balancing the load of the distributed memory database and the method embodiment for balancing the load of the distributed memory database provided in the above embodiments belong to the same concept, and specific implementation processes thereof are detailed in the method embodiment and are not described herein again.
Fig. 6 is a schematic structural diagram of a management device according to an embodiment of the present invention, where themanagement device 600 may generate a relatively large difference due to different configurations or performances, and may include one or more processors (CPUs) 601 and one ormore memories 602, where thememory 602 stores at least one instruction, and the at least one instruction is loaded and executed by theprocessor 601 to implement the method for balancing the load of the distributed memory database.
The embodiment of the invention also provides a computer readable storage medium, wherein a computer program is stored in the storage medium, and when being executed by a processor, the computer program realizes the method for balancing the load of the distributed memory database.
The embodiment of the invention also provides a management device, which comprises a processor and a memory, wherein the memory is used for storing the computer program; the processor is used for executing the program stored in the memory and realizing the method for balancing the load of the distributed memory database.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (16)
1. A method for balancing distributed memory database load, the method comprising:
acquiring the number of current fragments of each node in a target distributed memory database and an identifier of the current fragments, and acquiring a first time length and a second time length, wherein the first time length is the average time length of the single fragment processed on a local node, and the second time length is the average time length of the single fragment processed on a non-local node in a cross-network mode;
according to the number of the current fragments of each node, the identifier of the current fragments of each node, the first time length and the second time length, determining indication information of the fragments processed by each node in load balancing, wherein the indication information is used for indicating the fragments from each node;
and for each node, sending the indication information of the fragment of the node to the node so that the node acquires the fragment according to the indication information of the fragment of the node and processes the fragment.
2. The method of claim 1, wherein obtaining the first duration and the second duration comprises:
counting the time lengths of the first preset number of fragments which are respectively processed on the local node, and counting the time lengths of the second preset number of fragments which are respectively processed on the cross-network non-local node;
and determining that the counted average value of the time lengths of the first preset number of fragments respectively processed on the local nodes is a first time length, and determining that the counted average value of the time lengths of the second preset number of fragments respectively processed on the cross-network non-local nodes is a second time length.
3. The method according to claim 1, wherein said determining, according to the number of current fragments of each node, the identifier of current fragments of each node, the first duration and the second duration, the indication information of the fragments processed by each node in load balancing comprises:
determining the number of fragments from each node in the fragments processed by each node during load balancing according to the number of current fragments of each node, the first time length and the second time length;
and determining the indication information of the fragments processed by each node according to the number of the fragments from each node in the fragments processed by each node and the identification of the current fragment of each node.
4. The method according to any one of claims 1 to 3, wherein the indication information comprises:
starting fragment identification and ending fragment identification in fragments from each node; or,
the number of fragments from each node and the starting fragment identifier; or,
the number of fragments from each node and the end fragment identification.
5. The method according to claim 3, wherein the determining the number of the fragments from each node among the fragments processed by each node in the load balancing according to the number of the current fragments of each node, the first time duration and the second time duration comprises:
determining the number of fragments from each node in the fragments processed by each node in load balancing on the basis of the minimum value of TotalCost, wherein the sum of the current numbers of the fragments of all nodes in the target distributed memory database is equal to the sum of the numbers of the fragments processed on all nodes in load balancing, TotalCost is the sum of the variances of the total time lengths of the fragments processed by each node in the target distributed memory database,
Costktotal duration of fragment processing for node k, CostkIncluding processing the duration X of the processing of the fragment on the node k by the node kkkT and the time length of the node k for processing the fragments of other nodes
t is a first duration, ct is a second duration, n is the number of nodes in the target distributed memory database, XkkProcessing the number of fragments stored on the node k for the node k, XikAnd processing the number of fragments stored on the node i for the node k, wherein the node k is any node in the target distributed memory database, and k is greater than or equal to 1.
6. The method according to claim 5, wherein the determining the number of the fragments from each node in the fragments processed by each node in load balancing based on the principle that the TotalCost value is minimum comprises:
converting the TotalCost into a matrix form TotalCost ═ XTHX, wherein X ═ X (X)11,X12,...,X1n,...Xij,...,Xn1,Xn2,...,Xnn)T，XijProcessing the number of fragments on the ith node for the jth node, wherein i, j and n are positive integers, H is a symmetric matrix, and the sum of elements in X is less than or equal to the sum of the current number of fragments of all nodes in the target distributed memory database;
with XTAnd determining the number of fragments from each node in the fragments processed by each node in load balancing by taking the minimum value of HX as a principle.
7. The method of claim 1, wherein the obtaining the current number of shards per node in the target distributed in-memory database comprises:
when a data query request is received, acquiring the number of current fragments of each node in a target distributed memory database;
the method further comprises the following steps:
receiving the result of processing the fragments of each node in the target distributed memory database;
summarizing the results of processing the fragments of each node to obtain a total result;
and feeding back the total result.
8. An apparatus for balancing distributed in-memory database load, the apparatus comprising:
the system comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring the number of current fragments of each node in a target distributed memory database and the identifier of the current fragments, and acquiring a first time length and a second time length, the first time length is the average time length of the single fragment processed on a local node, and the second time length is the average time length of the single fragment processed on a non-local node in a cross-network mode;
a determining module, configured to determine, according to the number of current fragments of each node, an identifier of the current fragment of each node, the first time length, and the second time length, indication information of a fragment processed by each node in load balancing, where the indication information is used to indicate the fragment from each node;
and the sending module is used for sending the indication information of the fragments of the nodes to the nodes so that the nodes acquire the fragments according to the indication information of the fragments of the nodes for processing.
9. The apparatus of claim 8, wherein the obtaining module is configured to:
counting the time lengths of the first preset number of fragments which are respectively processed on the local node, and counting the time lengths of the second preset number of fragments which are respectively processed on the cross-network non-local node;
and determining that the counted average value of the time lengths of the first preset number of fragments respectively processed on the local nodes is a first time length, and determining that the counted average value of the time lengths of the second preset number of fragments respectively processed on the cross-network non-local nodes is a second time length.
10. The apparatus of claim 8, wherein the determining module is configured to:
determining the number of fragments from each node in the fragments processed by each node during load balancing according to the number of current fragments of each node, the first time length and the second time length;
and determining the indication information of the fragments processed by each node according to the number of the fragments from each node in the fragments processed by each node and the identification of the current fragment of each node.
11. The apparatus according to any one of claims 8 to 10, wherein the indication information comprises:
starting fragment identification and ending fragment identification in fragments from each node; or,
the number of fragments from each node and the starting fragment identifier; or,
the number of fragments from each node and the end fragment identification.
12. The apparatus of claim 10, wherein the determining module is configured to:
determining the number of fragments from each node in the fragments processed by each node in load balancing on the basis of the minimum value of TotalCost, wherein the sum of the current numbers of the fragments of all nodes in the target distributed memory database is equal to the sum of the numbers of the fragments processed on all nodes in load balancing, TotalCost is the sum of the variances of the total time lengths of the fragments processed by each node in the target distributed memory database,
Costktotal duration of fragment processing for node k, CostkIncluding processing the duration X of the processing of the fragment on the node k by the node kkkT and the time length of the node k for processing the fragments of other nodes
t is a first duration, ct is a second duration, n is the number of nodes in the target distributed memory database, XkkProcessing the number of fragments stored on the node k for the node k, XikAnd processing the number of fragments stored on the node i for the node k, wherein the node k is any node in the target distributed memory database, and k is greater than or equal to 1.
13. The apparatus of claim 12, wherein the determining module is configured to:
converting the TotalCost into a matrix form TotalCost ═ XTHX, wherein X ═ X (X)11,X12,...,X1n,...Xij,...,Xn1,Xn2,...,Xnn)T，XijProcessing the number of fragments on the ith node for the jth node, wherein i, j and n are positive integers, H is a symmetric matrix, and the sum of elements in X is less than or equal to the sum of the current number of fragments of all nodes in the target distributed memory database;
with XTAnd determining the number of fragments from each node in the fragments processed by each node in load balancing by taking the minimum value of HX as a principle.
14. The apparatus of claim 8, wherein the obtaining module is configured to:
when a data query request is received, acquiring the number of current fragments of each node in a target distributed memory database;
the device further comprises:
the receiving module is used for receiving the result of processing the fragments of each node in the target distributed memory database; summarizing the results of processing the fragments of each node to obtain a total result;
the sending module is further configured to feed back the total result.
15. A computer-readable storage medium, characterized in that a computer program is stored in the storage medium, which computer program, when being executed by a processor, carries out the method steps of any one of the claims 1-7.
16. A management device comprising a processor and a memory, wherein the memory is configured to store a computer program; the processor, configured to execute the program stored in the memory, implements the method steps of any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811204395.1ACN111061557B (en) | 2018-10-16 | 2018-10-16 | Method and device for balancing distributed memory database load |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811204395.1ACN111061557B (en) | 2018-10-16 | 2018-10-16 | Method and device for balancing distributed memory database load |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111061557Atrue CN111061557A (en) | 2020-04-24 |
| CN111061557B CN111061557B (en) | 2023-04-14 |
Family
ID=70296403
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811204395.1AActiveCN111061557B (en) | 2018-10-16 | 2018-10-16 | Method and device for balancing distributed memory database load |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111061557B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112527767A (en)* | 2020-12-03 | 2021-03-19 | 许继集团有限公司 | Method and system for completely repairing multiple region tables after restart of distributed database |
| CN115033390A (en)* | 2022-08-09 | 2022-09-09 | 阿里巴巴（中国）有限公司 | Load balancing method and device |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5701482A (en)* | 1993-09-03 | 1997-12-23 | Hughes Aircraft Company | Modular array processor architecture having a plurality of interconnected load-balanced parallel processing nodes |
| CN103095806A (en)* | 2012-12-20 | 2013-05-08 | 中国电力科学研究院 | Load balancing management system of large-power-network real-time database system |
| CN105915630A (en)* | 2016-06-01 | 2016-08-31 | 东软集团股份有限公司 | Cross-network data transmission system and load balancing scheduling method |
| CN105975345A (en)* | 2016-05-20 | 2016-09-28 | 江苏得得空间信息科技有限公司 | Video frame data dynamic equilibrium memory management method based on distributed memory |
| CN107480254A (en)* | 2017-08-14 | 2017-12-15 | 上海交通大学 | Suitable for the online load-balancing method of distributed memory database |
| CN107844593A (en)* | 2017-11-17 | 2018-03-27 | 北京邮电大学 | Video data placement method and device in a kind of Distributed Computing Platform |
| US20180157729A1 (en)* | 2016-12-06 | 2018-06-07 | Electronics And Telecommunications Research Institute | Distributed in-memory database system and method for managing database thereof |
- 2018
- 2018-10-16CNCN201811204395.1Apatent/CN111061557B/enactiveActive
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5701482A (en)* | 1993-09-03 | 1997-12-23 | Hughes Aircraft Company | Modular array processor architecture having a plurality of interconnected load-balanced parallel processing nodes |
| CN103095806A (en)* | 2012-12-20 | 2013-05-08 | 中国电力科学研究院 | Load balancing management system of large-power-network real-time database system |
| CN105975345A (en)* | 2016-05-20 | 2016-09-28 | 江苏得得空间信息科技有限公司 | Video frame data dynamic equilibrium memory management method based on distributed memory |
| CN105915630A (en)* | 2016-06-01 | 2016-08-31 | 东软集团股份有限公司 | Cross-network data transmission system and load balancing scheduling method |
| US20180157729A1 (en)* | 2016-12-06 | 2018-06-07 | Electronics And Telecommunications Research Institute | Distributed in-memory database system and method for managing database thereof |
| CN107480254A (en)* | 2017-08-14 | 2017-12-15 | 上海交通大学 | Suitable for the online load-balancing method of distributed memory database |
| CN107844593A (en)* | 2017-11-17 | 2018-03-27 | 北京邮电大学 | Video data placement method and device in a kind of Distributed Computing Platform |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112527767A (en)* | 2020-12-03 | 2021-03-19 | 许继集团有限公司 | Method and system for completely repairing multiple region tables after restart of distributed database |
| CN112527767B (en)* | 2020-12-03 | 2024-05-10 | 许继集团有限公司 | Method and system for completely repairing multiple region tables after restarting distributed database |
| CN115033390A (en)* | 2022-08-09 | 2022-09-09 | 阿里巴巴（中国）有限公司 | Load balancing method and device |
| CN115033390B (en)* | 2022-08-09 | 2022-11-25 | 阿里巴巴（中国）有限公司 | Load balancing method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111061557B (en) | 2023-04-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10114682B2 (en) | Method and system for operating a data center by reducing an amount of data to be processed | |
| US9171044B2 (en) | Method and system for parallelizing database requests | |
| CN111897638B (en) | Distributed task scheduling method and system | |
| CN110427386B (en) | Data processing method, device and computer storage medium | |
| CN113485962B (en) | Log file storage method, device, equipment and storage medium | |
| CN108241539B (en) | Interactive big data query method and device based on distributed system, storage medium and terminal equipment | |
| WO2013078583A1 (en) | Method and apparatus for optimizing data access, method and apparatus for optimizing data storage | |
| JP5730386B2 (en) | Computer system and parallel distributed processing method | |
| CN109766318B (en) | File reading method and device | |
| US20230063541A1 (en) | Determining computer resource usage at multiple levels of a container orchestration system hierarchy | |
| CN106682167B (en) | Statistical device and method for user behavior data | |
| WO2013010414A1 (en) | Index constructing method, search method, device and system | |
| CN105094981B (en) | A method and device for data processing | |
| CN104077188A (en) | Method and device for scheduling tasks | |
| CN111752945A (en) | Time sequence database data interaction method and system based on container and hierarchical model | |
| CN103731498A (en) | Big data real-time enquiry system load balancing method based on copy selection | |
| CN107562803B (en) | Data supply system and method and terminal | |
| CN111061557B (en) | Method and device for balancing distributed memory database load | |
| CN111694793A (en) | Log storage method and device and log query method and device | |
| CN112799820A (en) | Data processing method, data processing apparatus, electronic device, storage medium, and program product | |
| CN110798492B (en) | Data storage method and device and data processing system | |
| CN111881086A (en) | Big data storage method, big data query method, electronic device and storage medium | |
| US20100017429A1 (en) | Method and apparatus of distributing data in partioned databases operating on a shared-nothing architecture | |
| US11416517B2 (en) | Partitioning data in a clustered database environment | |
| Yan et al. | Efficient processing of very large graphs in a small cluster |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |