Quick restarting method of distributed memory big data processing systemTechnical Field
The invention relates to a quick restarting method of a distributed memory big data processing system, belonging to the technical field of big data processing.
Background
The patent 201810605777.9 discloses a distributed memory big data processing system and a data processing method thereof, wherein the storage nodes of the system store data in a memory, and along with the increase of time, the accumulated amount of insertion and inquiry requests is increased, and a large amount of memory application and release are accompanied, which means that the generation and accumulation of memory fragments are also caused, and the problem of the memory fragments can be thoroughly solved only by restarting the system.
Disclosure of Invention
The invention provides a quick restarting method of a distributed memory big data processing system, which solves the problem of long time of the existing restarting method.
In order to solve the technical problems, the invention adopts the following technical scheme:
a fast restarting method for the big data processing system of distributed memory includes such steps as providing a data processing system,
responding to a restart command received by a storage process of a storage node, copying self data to a shared memory by the storage process, and exiting the storage process after copying is finished;
in response to the storage process restarting, the storage process recovers data from the shared memory.
If the shared memory is not available or the data is recovered from the shared memory abnormally, the storage process recovers the data from the disk.
And in response to the storage process receiving the restarting command, the storage process finishes executing the insertion request and the query request which are currently being executed, refuses the new insertion request and the query request, deletes the deletion request in the request queue, and finally copies all data in the storage process to the shared memory.
The storage process sequences the data and copies the data to the shared memory.
Data serialization is the conversion of all data stored in a storage process that has a spatial hierarchy into a linear sequence.
Dividing all storage nodes into a plurality of parts, restarting one of the storage nodes, restarting the other storage node after the restarting of the storage node is completed, and traversing all the storage nodes until the restarting of all the storage nodes is completed.
Each storage node accounts for 2% of all storage nodes.
The invention has the beneficial effects that: 1. the invention copies the data to the shared memory, and when restarting, the data is read from the shared memory, so that the restarting is faster; 2. the invention adopts progressive restarting, and ensures that a part of storage nodes can be normally used during restarting, so that the system can be normally used under the restarting condition.
Drawings
FIG. 1 is a flow chart of the present invention;
FIG. 2 is a data serialization storage mode;
FIG. 3 is a data table state at the time of data copying;
fig. 4 is a diagram of a progressive restart procedure.
Detailed Description
The invention is further described below with reference to the accompanying drawings. The following examples are only for more clearly illustrating the technical aspects of the present invention, and are not intended to limit the scope of the present invention.
As shown in FIG. 1, a method for quickly restarting a distributed memory big data processing system includes the following steps:
step 1, in response to a storage process of a storage node receiving a restarting command, the storage process copies own data to a shared memory, and the storage process exits after the copying is finished.
The storage process copies the data after serialization to the shared memory, the data serialization is to convert all the data with space hierarchical structure stored in the storage process into a linear sequence, so as to directly copy to the shared memory or restore from the shared memory, the specific data serialization is shown in fig. 2, the current version number is recorded first, the byte length after serialization is followed by the information of each data table. The data table includes header information of each data table, the number of line blocks, the header information of each line block, the data table (name-data type) corresponding to the line block, the header information stored for each line block in blocks for each column, and the corresponding data.
Shared memory: shared memory refers to a large amount of memory that can be accessed by different Central Processing Units (CPUs) in a multiprocessor computer system.
As shown in fig. 3, from the state of the data table, in response to receiving a restart command by the storage process, the storage process rejects new insertion requests and query requests after the execution of the insertion requests and query requests currently being executed, deletes the deletion requests in the request queue, writes the log into the disk at the same time, and finally copies all data in the storage process to the shared memory.
And step 2, in response to restarting the storage process, the storage process firstly recovers data from the shared memory, and if the shared memory is unavailable or the recovery of the data from the shared memory is abnormal, the storage process recovers the data from the disk.
The method copies the data to the shared memory, and when restarting, reads the data from the shared memory, and restarts faster.
Because the data is stored in the memory, the traditional restarting is to restart all the storage nodes at one time, and operations such as data insertion, query and the like cannot be performed in the restarting process, so that the normal operation of the system can be influenced; therefore, progressive restarting is adopted, and when restarting, a part of storage nodes can be ensured to be normally used, so that the system can be normally used under the restarting condition, specifically, as shown in fig. 4, all the storage nodes are divided into a plurality of parts, one part of the storage nodes is restarted, the other storage nodes can be normally used, when the part of the storage nodes are restarted, the other part of the storage nodes are restarted, and all the storage nodes are traversed until all the storage nodes are restarted. And by combining long-term practical experience, optimally, each storage node occupies 2% of all storage nodes, namely 98% of the storage nodes are ensured to be normally used at all times, and the normal operation of the system is satisfied.
The foregoing is merely a preferred embodiment of the present invention, and it should be noted that modifications and variations could be made by those skilled in the art without departing from the technical principles of the present invention, and such modifications and variations should also be regarded as being within the scope of the invention.