Deep learning training operation resource placement system and method based on reinforcement learningTechnical Field
The invention relates to the technical field of computing resource scheduling, in particular to a deep learning training job resource placement system and method based on reinforcement learning.
Background
Deep learning training (DEEP LEARNING TRAINING, DLT) jobs are typically computationally intensive tasks requiring powerful and expensive computing resources, such as GPU equipment, and in order to process training data of ever-increasing size, most currently mainstream IT companies or enterprises run DLT jobs through a cluster of GPU servers, perform Distributed deep learning (Distributed DEEP LEARNING, DDL) training to utilize multiple GPUs in parallel, thereby reducing the load on a single GPU and speeding up the training rate of the model.
The multi-machine multi-card training mode is a main characteristic of large-scale distributed DLT operation, and the operation error probability can rise in a same way along with the increase of the complexity of the system. Moreover, DLT operation training time is generally longer, and long-time operation can also improve the probability of operation errors. In addition, frequent submissions in a multi-tenant multi-job scenario often also bring about an increase in job error probability. While DLT job error is one of the important reasons for reducing the utilization rate of system resources, the time cost caused by job error is not negligible, the more the number of errors is, the greater the job restarting cost and the resource recovery cost caused by error are, and the lower the utilization rate of resources is.
In order to better place DLT operation reasonably under the situation of cluster resource error, a method based on cluster capacity sensing and a method based on load interference sensing are proposed in the prior art. The method based on cluster capacity sensing does not consider the error characteristics of different GPUs in the cluster, for example, when the GPU device with lower error probability is in a relatively high-load state for a long time, the scheduling policy has a high possibility of frequently placing the multi-card DLT large job on the GPU device with higher error probability, which can restart the job for multiple times and reduce the resource utilization rate. Although the method based on load interference sensing largely avoids the degradation of training performance and the reduction of resource utilization rate caused by interference among DLT jobs, the error characteristics of each GPU device in the cluster are not considered yet, for example, if the distribution of the GPUs with higher error probability in the cluster is dispersed, the distributed multi-card DLT job with higher interference degree is likely to be placed on the GPU with higher error probability when being placed separately, so that the operation frequency is restarted, and the more serious degradation of training performance and resource utilization rate is brought.
Reinforcement learning (Reinforcement Learning, RL) is also a self-learning method similar to the traditional deep learning method, but deep learning is a static learning algorithm by learning features in existing data to make predictions of unknown data. And the RL is a process of establishing a decision model and learning to obtain an optimal strategy through continuous exploration of an unknown environment, and is a dynamic learning algorithm. Thus, to some extent, the RL is more consistent with human thinking and learning processes, and in particular, the RL incorporating deep learning techniques, namely deep reinforcement learning (Deep Reinforcement Learning, DRL), is recognized as one paradigm closest to true artificial intelligence.
Therefore, how to apply the DRL algorithm to the decision problem of resource scheduling, namely the placement position of the job, and reasonably place the DLT job under the scene of cluster resource errors so as to maximize the resource utilization rate as much as possible and improve the service quality of the user, thereby becoming the problem to be solved.
Disclosure of Invention
The invention aims to provide a deep learning training job resource placement method based on reinforcement learning, which can carry out self-adaptive placement on DLT jobs under a resource error scene.
In order to solve the technical problems, the application provides the following technical scheme:
the deep learning training job resource placement method based on reinforcement learning comprises the following steps:
the initialization step is to randomly initialize the parameters of the DRL neural network model;
Generating state vectors of batch operation;
The reasoning step, sending the state vector into a DRL neural network model to infer and obtain placement position information of batch operation, and carrying out operation placement according to the placement position information to obtain the maximum completion time of batch operation to be recorded as T_RL;
A Random generation step, namely randomly generating a plurality of pieces of placement position information, carrying out operation placement according to the randomly generated placement position information, obtaining a plurality of maximum completion times of the batch operation, and obtaining a minimum maximum completion time which is recorded as T_random;
A reward calculation step of calculating a reward based on the maximum completion time T_RL and the maximum completion time T_random;
And a parameter updating step, namely updating the parameters of the DRL neural network model by reverse gradient.
The basic scheme principle and the beneficial effects are as follows:
In the scheme, the DRL neural network model is trained to perform reasoning of the operation placement position, and compared with a traditional heuristic algorithm, the DRL neural network model can automatically analyze and extract more effective and more accurate characteristics of cluster faults and DLT operation without manually selecting certain parameters as characteristics, so that the influence caused by error of manually selecting the characteristics is reduced.
The rewards of the DRL neural network model are calculated by taking the minimum value T_random of the completion time of the batch operation of multiple random scheduling as a reference, and a larger rewarding range is obtained by utilizing randomness, so that the learning capacity of the DRL neural network model can be improved.
The training process of the scheme can utilize the simulator to perform pre-training or complete training so as to save time and economic cost, can also use the historical data of the real cluster system to perform training so as to obtain a scheduling strategy more suitable for the system, and can also directly perform online training on the prototype system so as to obtain a more accurate scheduling strategy.
In summary, the method aims at the problem of placement position decision of DLT operation under the situation of cluster error, trains a DRL neural network model to perform self-adaptive placement under the situation of DLT operation resource error, reduces the maximum completion time of large-scale distributed DLT operation in batches, and improves the resource utilization rate.
Further, the method also comprises an experience playback step of sampling the tetrad sample generated in the training process of the DRL neural network model for experience playback.
Through experience playback, on one hand, the correlation among samples can be eliminated to meet the basic requirement of neural network training, and on the other hand, the dynamic experience playback can maximize the playback range, so that the effectiveness of experience playback is ensured.
Further, in the state vector generation step, a state vector is generated based on the DLT job information and the cluster information, and is written asThe method comprises the steps of determining a calculation unit number required by a current job, wherein N is the calculation unit number required by the current job, T is the estimated running time of the current job under the condition of no error, and S is the use state of each calculation unit in the current cluster.
In the preferred scheme, DLT operation information and cluster information are acquired, a state vector is generated after processing and is used as a characteristic to be input into a DRL neural network model for training, a random dispatching optimal scheme is combined with the maximum completion time of batch operation obtained by a dispatching scheme which adopts the reasoning of the current DRL neural network model, and the method is used as an evaluation criterion to guide the neural network to adaptively make a placement decision of the DLT operation, so that the maximum completion time of batch large-scale distributed DLT operation is reduced, and the resource utilization rate is improved.
Further, the reasoning step specifically includes:
A1, inputting a state vector into a value network of a DRL neural network model to obtain a long-term measurement index V;
a2, inputting the state vector into a strategy network of the DRL neural network model to obtain selection probabilities Pi of N computing units, wherein i=1, 2, N;
a3, setting the probability Pj corresponding to the occupied computing unit and the fault computing unit to zero to obtain P'i;
A4, selecting the kth computing unit as one of computing units to be placed in the operation, wherein Pk＝max(P′i is as follows;
a5, if the number of the computing units to be placed by the job is equal to the number of the computing units required by the job, completing the inference of the placement position information of the job, and inferring the position information of the next job, otherwise, jumping to the step A1.
Further, the experience playback step specifically includes:
B1, creating a playback buffer Chi Zhan;
B2, pushing the four-element sample generated in the training process to enter a return visit buffer Chi Zhan;
b3, overflowing the four-element sample which is the earliest to be stacked if the return visit buffer pool stack is full;
And B4, selecting X four-tuple samples as a batch for next training, wherein the number of X is the total number of the four-tuple samples in the current return visit buffer pool stack.
In the DRL neural network model, after a series of reasoning actions are made, a plurality of four-tuple samples are generated, the samples have strong correlation, the requirement of the deep neural network on independent and uniform distribution of training samples is not met, the sample sequence generated in one interval cannot represent global experience, and the forgetting characteristic of the neural network can enable the neural network to be easily trapped into local optimum in the training process. The preferred solution is solved by means of empirical playback. In view of the strong uncertainty of the resource error time, the difference between the number of samples generated in different scheduling intervals can be large, so that the dynamic batch is adopted for sampling. The size of the random sampling batch is equal to the number of samples obtained by reasoning in the current scheduling period when the DRL neural network model is trained each time, and the batch is input into the DRL neural network model for training, so that the effect of experience playback can be fully exerted, and the correlation among samples is reduced to a great extent.
Further, in the step of calculating the rewards, a calculation formula of the rewards is:
the smaller the desired t_rl, the better, but the t_rl has a relativity, i.e. is related to the actual working time, which is long, and the t_rl cannot be too small. In this scheme, t_range is compared with t_rl as a relative value.
Further, the training judgment step is further included, namely judging whether the DRL neural network model is trained, if not, returning to the state vector generation step, otherwise, finishing the training.
Further, the method also comprises the step of using the trained DRL neural network model to infer and obtain the placement position of each job in the batch job.
Further, the using step specifically includes:
C1, acquiring job information and cluster information of batch jobs;
c2, generating a state vector based on the information collected in the step C1;
c3, inputting the state vector in the step C2 into a strategy network of the DRL neural network model to obtain placement position information output by the strategy network;
C4, repeating the step C3 if the number of the calculation units inferred at present is smaller than the number of calculation units required by the current operation, otherwise, jumping to the step C5;
And C5, placing the corresponding operation according to the placement position information inferred in the step C3.
The second object of the invention is to provide a deep learning training job resource placement system based on reinforcement learning, which comprises a DRL neural network model and a job scheduling module, wherein the job scheduling module trains the DRL neural network model by using the steps of the method. And acquiring the placement position information from the trained DRL neural network model, and placing corresponding operation according to the placement position information.
According to the scheme, a DRL neural network model is adopted to schedule a calculation unit, cluster information and currently submitted job information are periodically acquired, the cluster information and the currently submitted job information are processed and then are used as characteristics to be input into the DRL neural network model for training, a random scheduling optimal mode is combined with batch job maximum completion time obtained by a scheduling mode adopting current DRL neural network model reasoning, and the DRL neural network model is guided to adaptively make a placement decision of DLT jobs as an evaluation criterion, so that the maximum completion time of batch large-scale distributed DLT jobs is reduced, namely the resource utilization rate is improved.
Drawings
FIG. 1 is a schematic diagram of a cluster operation lifecycle;
FIG. 2 is a flow chart of DRL neural network model training;
FIG. 3 is a schematic diagram of a structural design of a DRL neural network model;
FIG. 4 is a schematic diagram of an empirical playback;
Fig. 5 is a schematic diagram of a DRL neural network model reasoning process.
Detailed Description
The following is a further detailed description of the embodiments:
examples
As shown in fig. 1, the method of the present embodiment is applied to a job scheduling process of a cluster, and aims to give what nodes and what computing resources the job should be placed on in the cluster. In this embodiment, taking a GPU as a common computing unit, a deep learning training job resource placement method based on reinforcement learning is introduced, including the following steps:
Training neural network model parameters by reinforcement learning, as shown in fig. 2, specifically includes:
and initializing the parameters of the DRL neural network model randomly.
And generating workloads (batch job) state vectors based on the DLT job information and the cluster information. The state vector is noted asThe concrete representation is as follows:
and N, the number of GPUs required by the current operation.
And T, estimating the running time of the current operation under the condition of normal error.
S, the using state of each GPU in the current cluster. For example, a total of 4 GPU devices in the current cluster, the first two GPUs are available, the last two GPUs are not available due to errors or being occupied, s= [0, 1].
In this embodiment, DLT job information and cluster information are periodically acquired.
Reasoning step, state vectorAnd sending the information into the DRL neural network model to obtain the placement position information of workloads by inference, and carrying out operation placement according to the placement position information to obtain the maximum completion time T_RL of workloads operation.
The DRL neural network model of the reasoning step is shown in fig. 3, and specifically includes:
A1 vector of stateAnd (3) inputting a Value Network (Value Network) of the DRL neural Network model, and obtaining a long-term measurement index V through full-connection layers with the number of 5 layers of neurons of 256, 196, 128 and 1 respectively.
A2 vector of stateThe Policy Network (Policy Network) of the DRL neural Network model is input, and the selection probabilities Pi, i=1, 2, i=2, N of the N GPUs are obtained through the full connection layer with the number of 5 layers of neurons being 256, 196, 128, N, respectively, and then through the softmax layer.
A3, setting the probability Pj corresponding to the occupied GPU and the failed GPU to zero to obtain P'i.
A4, selecting the kth GPU as one of the GPUs to be placed in the operation, wherein Pk＝max(P′i).
A5, if the quantity of the GPUs to be placed by the job is equal to the quantity of the GPUs required by the job, completing the placement position information reasoning of the job, and reasoning the position information of the next job, otherwise, jumping to the step A1.
And randomly generating a series of placement position information, performing operation placement according to the placement position information to obtain a series of maximum completion time of workloads, and finally obtaining the minimum maximum completion time to be marked as T_random.
And calculating rewards (reward) based on the maximum completion time T_RL and the maximum completion time T_random. The calculation formula is as follows:
And experience playback, namely sampling the four-element samples (s, a, r, s') generated in the DRL training process to apply experience playback. In the quadruple samples (s, a, r, s '), s is an environmental state, a is an Actor, namely, one action selected based on the current strategy, s' is the next environmental state transferred after the action a is executed in the environmental state s, and r is an environmental feedback reward, namely, a report.
The experience playback steps are shown in fig. 4, and specifically include:
B1, creating a Replay Buffer stack.
And B2, stacking the four-element group samples (s, a, r, s') generated in the training process and entering a return visit buffer pool stack.
B3, overflowing the earliest data which is pushed into the stack if the buffer pool stack is full.
And B4, selecting x four-element group samples as a batch to wait for the next training. Wherein the number of X is the total number of four-tuple samples in the current return visit buffer pool stack.
And a parameter updating step, namely updating the parameters of the DRL neural network model by reverse gradient. In this embodiment, the reverse gradient update is performed by using reward, which is the prior art and will not be described here again.
And a training judging step, wherein if the training is not completed, the step is returned to the step S12 to continue execution, otherwise, the training is ended. In this embodiment, the planned training frequency is preset, and when the actual training frequency is equal to the planned training frequency, the training is considered to be completed.
The application step comprises the steps of using a DRL neural network model after training to infer and obtain the placement positions of each operation in workloads, wherein the method specifically comprises the following steps:
And C1, acquiring workloads job information and cluster GPU information.
C2, generating a state vector by using the information collected in the step C1
And C3, inputting the state vector in the step C2 into a strategy network to obtain the placement strategy output by the strategy network, namely the placement position information.
And C4, repeating the step C3 if the number of the GPU inferred at present is smaller than the number of the GPU required by the current operation.
And C5, placing the corresponding operation according to the placement position information inferred in the step C3.
In this embodiment, the GPU is a minimum scheduling unit, that is, one GPU cannot be allocated to multiple jobs for use. In other embodiments, the calculation unit may also employ TPU (tensor processing unit), MLU (machine learning processor), or the like.
According to the deep learning training job resource placement method based on reinforcement learning, the embodiment also provides a deep learning training job resource placement system based on reinforcement learning, which comprises a DRL neural network model and a job scheduling module. The job scheduling module trains the DRL neural network model by using the steps of the method. The job scheduling module also uses the steps of the method to obtain the placement position information from the trained DRL neural network model, and places the corresponding job according to the placement position information.
The foregoing is merely an embodiment of the present application, the present application is not limited to the field of this embodiment, and the specific structures and features well known in the schemes are not described in any way herein, so that those skilled in the art will know all the prior art in the field before the application date or priority date of the present application, and will have the capability of applying the conventional experimental means before the date, and those skilled in the art may, in light of the present application, complete and implement the present scheme in combination with their own capabilities, and some typical known structures or known methods should not be an obstacle for those skilled in the art to practice the present application. It should be noted that modifications and improvements can be made by those skilled in the art without departing from the structure of the present application, and these should also be considered as the scope of the present application, which does not affect the effect of the implementation of the present application and the utility of the patent. The protection scope of the present application is subject to the content of the claims, and the description of the specific embodiments and the like in the specification can be used for explaining the content of the claims.