Movatterモバイル変換


[0]ホーム

URL:


CN120104276A - Intelligent large-scale task scheduling method and system based on computing host business characteristics - Google Patents

Intelligent large-scale task scheduling method and system based on computing host business characteristics
Download PDF

Info

Publication number
CN120104276A
CN120104276ACN202510144306.2ACN202510144306ACN120104276ACN 120104276 ACN120104276 ACN 120104276ACN 202510144306 ACN202510144306 ACN 202510144306ACN 120104276 ACN120104276 ACN 120104276A
Authority
CN
China
Prior art keywords
task
computing power
dynamic
computing
business characteristics
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202510144306.2A
Other languages
Chinese (zh)
Inventor
李青壮
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Inspur Communication Technology Co Ltd
Original Assignee
Inspur Communication Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Inspur Communication Technology Co LtdfiledCriticalInspur Communication Technology Co Ltd
Priority to CN202510144306.2ApriorityCriticalpatent/CN120104276A/en
Publication of CN120104276ApublicationCriticalpatent/CN120104276A/en
Pendinglegal-statusCriticalCurrent

Links

Classifications

Landscapes

Abstract

Translated fromChinese

本发明公开了基于算力主机业务特性的智能化大规模任务调度方法及系统,属于云计算及边缘计算技术领域,该方法的实现包括以下步骤:任务和算力主机节点的数据采集与预处理;对任务业务特性进行多维度量化分析;算力主机节点性能状态的动态监控;动态算力匹配模型构建与优化;基于智能预测机制辅助优化任务调度;最终任务分配与执行。本发明解决现有算力主机在处理大规模并发任务时,由于调度策略缺乏对不同业务特性的精准识别和适配,导致的资源浪费、任务堆积以及响应延迟的问题;能够提高算力资源的利用率,降低任务响应延迟,提升用户体验和系统整体性能。

The present invention discloses an intelligent large-scale task scheduling method and system based on the business characteristics of computing power hosts, which belongs to the field of cloud computing and edge computing technology. The implementation of this method includes the following steps: data collection and preprocessing of tasks and computing power host nodes; multi-dimensional quantitative analysis of task business characteristics; dynamic monitoring of the performance status of computing power host nodes; dynamic computing power matching model construction and optimization; auxiliary optimization of task scheduling based on intelligent prediction mechanism; and final task allocation and execution. The present invention solves the problems of resource waste, task accumulation and response delay caused by the lack of accurate identification and adaptation of different business characteristics of the scheduling strategy when the existing computing power host processes large-scale concurrent tasks; it can improve the utilization rate of computing power resources, reduce task response delays, and improve user experience and overall system performance.

Description

Intelligent large-scale task scheduling method and system based on business characteristics of power host
Technical Field
The invention relates to the technical field of cloud computing and edge computing, in particular to an intelligent large-scale task scheduling method and system based on business characteristics of a computing host.
Background
With the rapid development of cloud computing, big data and artificial intelligence technology, the efficiency problem of a computing host computer in processing large-scale concurrent tasks is more and more remarkable. However, when facing multiple service characteristics, the existing task scheduling strategies generally adopt a static or single optimization target mode, and it is difficult to meet the diversified demands of different tasks on resources. The lack of adaptability scheduling strategy often causes problems of computational resource waste, task accumulation, response delay and the like, and severely restricts the performance and user experience of the system.
Disclosure of Invention
The technical task of the invention aims at the defects, and provides the intelligent large-scale task scheduling method and system based on the business characteristics of the power host, so that the utilization rate of the power resource can be improved, the task response delay can be reduced, and the user experience and the overall performance of the system can be improved.
The technical scheme adopted for solving the technical problems is as follows:
The intelligent large-scale task scheduling method based on the business characteristics of the power host comprises the following steps:
1) Data acquisition and preprocessing of the task and the computing host node;
2) Carrying out multidimensional quantitative analysis on task business characteristics;
3) Dynamic monitoring of the performance state of the node of the power host;
4) Constructing and optimizing a dynamic calculation force matching model;
5) Auxiliary optimization task scheduling based on an intelligent prediction mechanism;
6) And finally, distributing and executing the tasks.
By introducing a multi-objective optimization algorithm, a dynamic weight distribution strategy and an intelligent prediction mechanism, the utilization rate of computing power resources is improved, the response delay of tasks is reduced, and the user experience and the overall performance of a system are improved.
Further, the data acquisition and preprocessing includes:
1.1 Task data acquisition, wherein the scheduling system receives and stores the key information of the task in real time, and the method comprises the following steps:
determining the time urgency of a task and providing a basis for real-time demand quantification;
the concurrent execution record of the task is extracted from the database and is used for concurrent demand prediction;
Analyzing the resource calling frequency and algorithm complexity of tasks, and determining the computation density degree of different tasks;
service Level Agreements (SLAs) define task priorities according to an agreement, e.g., VIP service tasks have higher weights;
1.2 Data acquisition of the power host nodes, namely, acquiring performance data of each power host node in real time through a distributed monitoring system, wherein the data acquisition comprises the following steps:
CPU occupancy rate, which represents the use condition of computing resources and reflects the load capacity of nodes;
the memory utilization rate is that the allocation state of the node memory is displayed, and the suitability of task allocation is evaluated;
network delay, namely describing communication time among nodes and influencing scheduling of cross-node tasks;
The energy consumption is that the energy consumption of the node in operation is monitored and is used for optimizing the energy efficiency ratio;
1.3 Data preprocessing:
in order to enable the task characteristics and the node performance to be directly used for model calculation, normalization processing is needed, and all indexes are uniformly mapped to the [0,1] interval through the following formula:
the normalized data avoid the influence of dimension difference on the subsequent model, and consistent input is provided for scheduling optimization.
Further, the service characteristics comprise real-time requirements, concurrency requirements, computational complexity and service level agreements;
Carrying out multidimensional quantitative analysis on task business characteristics to determine the scheduling requirements:
2.1 Real-time demand quantification:
in order to measure the time urgency of the task, the resource priority allocation is guided, the scheduling is realized by calculating the ratio of the current time to the task deadline, and the formula is as follows:
if RTD approaches 1, the higher the task priority is, the task needs to be immediately scheduled;
If RTD approaches 0, indicating that the task is not urgent, and is suitable for later scheduling;
2.2 Concurrent demand prediction:
and optimizing a resource allocation strategy for perceiving possible concurrency pressure of tasks in advance. Predicting future concurrency demands by combining historical maximum concurrency with average values:
CD=α·Hmax+(1-α)·Emean
The user adjustable factor is used for adjusting the importance of the maximum concurrency and average concurrency of the history;
the high concurrency task allocates more nodes to meet the demand;
2.3 Calculation complexity evaluation):
To obtain the level of occupancy of computational resources by the quantization tasks. By analyzing the subtask complexity after task decomposition, the comprehensive complexity is calculated:
Wherein, Ci is the complexity index (such as calculated amount, I/O operation, etc.) of the I-th part task;
Wi, weight, represent the importance degree of subtasks;
2.4 Service level agreement priority calculation:
weights are directly assigned based on user requirements and terms of service (e.g., high priority tasks require quick response).
Further, the dynamic monitoring of the performance state of the node of the power host includes:
3.1 Real-time monitoring index:
Continuously collecting indexes comprising CPU occupancy rate and memory utilization rate through monitoring probes deployed on all the computing host nodes to form a node performance database with high frequency dynamic update;
3.2 Standardization and weight allocation):
Normalizing the node performance index according to a formula to enable the node performance index to have comparability;
Dynamic weights are assigned to different metrics (e.g., CPU occupancy may be more important than memory usage), and the weight values may be adjusted based on actual load.
Further, the dynamic computing power matching model optimizes task allocation rules based on a multi-objective optimization algorithm and a dynamic weight allocation strategy;
the dynamic computing force matching model construction and optimization specifically comprises the following steps:
4.1 Objective function definition:
the dynamic computing force matching model is combined with the following two main targets to optimize the resource utilization efficiency and response time:
the resource waste is minimized, namely, the distributed resources are ensured to be close to the actual demands, and idle is avoided;
minimizing response time by approximating task completion time to user expectations;
4.2 Dynamic weight policy):
According to the real-time system load, the weights of two targets are dynamically adjusted, and the formula is as follows:
Wfinal=λ1·WRW2·WRD
Wherein, lambda1、λ2 is a weight factor which can be automatically adjusted by a threshold condition;
4.3 Genetic algorithm optimization:
generating initial populations, wherein each population represents an allocation scheme of tasks and nodes;
Calculating fitness of each scheme based on the objective function;
crossover and mutation, namely introducing mutation factors into the population, improving the diversity of the scheme and avoiding sinking into local optimum;
iterative optimization, namely circularly updating the population until the objective function converges.
Furthermore, the intelligent prediction mechanism adopts a time sequence model to analyze historical task data and perceives the load change trend in advance;
the specific implementation of auxiliary optimization task scheduling based on the intelligent prediction mechanism comprises the following steps:
5.1 A load prediction model is constructed, task load change is predicted through a time sequence, and the formula is as follows:
Rfuture=β·Rcurrent+(1-β)·Rhistorical
The weight factor is used for carrying out weighted average between the current value Rcurrent and the historical value Rhistorical, and the influence of the current trend and the historical data on future prediction is balanced by adjusting the value of the beta;
When β approaches 1, the predicted value Rfuture depends more on the current value Rcurrent, i.e., the current trend is considered to have a greater impact on the future;
When β approaches 0, the predicted value Rfuture depends more on the history value Rhistorical, i.e., the history data is considered to have a greater impact on the future;
When β is equal to 0.5, the predicted value Rfuture is a simple average of the current and historical values, i.e., the effect of both is equal.
Optimizing a resource allocation strategy in advance by predicting a future high load period;
5.2 Resource warm-up and task migration):
for predicted high-priority task loads, resources are allocated in advance to reduce starting delay;
and dynamically migrating the non-critical tasks to the low-load nodes so as to reduce resource occupation conflict.
Further, the task allocation and execution includes:
Prioritizing tasks according to real-time Requirements (RTD), concurrency requirements (CD), and SLA priorities;
Performing resource allocation according to the output of the dynamic calculation matching model, and continuously monitoring the effect;
And (3) adaptively adjusting, namely recalculating task priority and resource allocation strategies according to system load changes.
The invention also claims an intelligent large-scale task scheduling system based on the business characteristics of the power host, which comprises:
The data acquisition and preprocessing module is used for realizing data acquisition and preprocessing of the task and the computing host node;
The task business characteristic quantitative analysis module is used for carrying out multidimensional quantitative analysis on the task business characteristics;
The power host node performance state monitoring module is used for dynamically monitoring the power host node performance state;
The dynamic computing force matching model construction and optimization module is used for realizing the construction and optimization of the dynamic computing force matching model;
the intelligent prediction mechanism auxiliary optimization module is used for auxiliary optimization task scheduling based on the intelligent prediction mechanism;
the task allocation and execution module is used for realizing final task allocation and execution;
The system can realize the method.
The invention also claims an intelligent large-scale task scheduling realization device based on the business characteristics of the power host, which comprises at least one memory and at least one processor;
the at least one memory for storing a machine readable program;
the at least one processor is configured to invoke the machine-readable program to implement the method described above.
The invention also claims a computer readable medium having stored thereon computer instructions which, when executed by a processor, are capable of carrying out the above-described method.
Compared with the prior art, the intelligent large-scale task scheduling method and system based on the business characteristics of the power host have the following beneficial effects:
1. and the dispatching accuracy is improved:
Through multidimensional quantitative analysis of the service characteristics (such as real time performance, concurrency requirements, computational complexity and service level agreements) of tasks and the performance states (such as CPU occupancy rate, memory utilization rate, network delay and energy consumption) of nodes, accurate matching of the tasks and resources of a computing host is achieved, and resource waste and task accumulation are effectively reduced.
2. System adaptability is enhanced:
the dynamic calculation force matching model and the multi-objective optimization algorithm are introduced, and the task allocation rule is adjusted by combining the real-time load and the service priority, so that the system can flexibly cope with complex task demands and load changes, and the robustness and the reliability of the system are improved.
3. Reducing response delay:
the intelligent prediction mechanism senses the load change trend in advance by analyzing the historical task data, performs resource preheating and task migration optimization, remarkably shortens the response time of the task and improves the user experience.
4. Optimizing the resource utilization rate:
The dynamic weight distribution strategy balances the resource utilization rate and the task execution efficiency in multi-objective optimization, maximizes the use efficiency of the nodes of the power host, and is particularly suitable for large-scale concurrent task scenes.
Drawings
Fig. 1 is a flowchart of an intelligent large-scale task scheduling method based on the business characteristics of a power host, which is provided by the embodiment of the invention.
Detailed Description
The invention will be further illustrated with reference to specific examples.
The embodiment of the invention provides an intelligent large-scale task scheduling method based on the business characteristics of a power host, which comprises the steps of carrying out multidimensional quantitative analysis on the business characteristics; the method comprises the steps of dynamically monitoring the performance state of a computing host node, constructing a dynamic computing matching model, and optimizing task scheduling based on an intelligent prediction mechanism. The business characteristics include real-time requirements, concurrency requirements, computational complexity, and service level agreements. The dynamic computing power matching model optimizes task allocation rules based on a multi-objective optimization algorithm and a dynamic weight allocation strategy. The intelligent prediction mechanism adopts a time sequence model to analyze historical task data, and perceives the load change trend in advance. Task scheduling optimization includes resource warm-up and task migration policies. The method comprises the following steps:
s1, acquiring data of a task and a power host node;
S2, quantitatively analyzing task business characteristics;
s3, monitoring the performance state of the node of the power host;
S4, constructing and optimizing a dynamic calculation force matching model;
s5, auxiliary optimization of an intelligent prediction mechanism;
S6, final task allocation and execution.
The above implementation steps are described in detail below.
And the first step is data acquisition and preprocessing.
1. The task data is collected and the data is stored,
The scheduling system receives and stores key information of tasks in real time:
determining the time urgency of a task and providing a basis for real-time demand quantification;
the concurrent execution record of the task is extracted from the database and is used for concurrent demand prediction;
Analyzing the resource calling frequency and algorithm complexity of tasks, and determining the computation density degree of different tasks;
Service Level Agreements (SLAs) define task priorities according to an agreement, e.g. VIP service tasks have a higher weight.
2. The data of the nodes of the power main machine are collected,
Through distributed monitored control system, gather every power host computer node's performance data in real time, include:
CPU occupancy rate, which represents the use condition of computing resources and reflects the load capacity of nodes;
the memory utilization rate is that the allocation state of the node memory is displayed, and the suitability of task allocation is evaluated;
network delay, namely describing communication time among nodes and influencing scheduling of cross-node tasks;
And the energy consumption is that the energy consumption of the node in operation is monitored and is used for optimizing the energy efficiency ratio.
3. The data is pre-processed and the data is pre-processed,
In order to enable the task characteristics and the node performance to be directly used for model calculation, normalization processing is needed, and all indexes are uniformly mapped to the [0,1] interval through the following formula:
the normalized data avoid the influence of dimension difference on the subsequent model, and consistent input is provided for scheduling optimization.
And secondly, quantitatively analyzing task business characteristics.
Carrying out multidimensional quantitative analysis on task business characteristics to determine the scheduling requirements:
1. The real-time demand is quantified and the real-time demand is quantified,
In order to measure the time urgency of the task, the resource priority allocation is guided, the scheduling is realized by calculating the ratio of the current time to the task deadline, and the formula is as follows:
if RTD approaches 1, the higher the task priority is, the task needs to be immediately scheduled;
if RTD approaches 0, indicating that the task is not urgent, it is suitable for later scheduling.
2. The concurrent demand forecast is based on the result of the demand forecast,
To sense possible concurrency pressure of tasks in advance, optimizing a resource allocation strategy, and predicting future concurrency demands by combining the historical maximum concurrency quantity and the average value:
CD=α·Hmax+(1-α)·Emean
And alpha is a user adjustable factor for adjusting the importance of the maximum concurrency and average concurrency of the history.
The high concurrency task allocates more nodes to meet the demand.
3. The evaluation of the computational complexity is carried out,
In order to obtain the occupation level of the quantized task on the computational power resource, the comprehensive complexity is calculated by analyzing the subtask complexity after task decomposition:
Wherein, Ci is the complexity index (such as calculated amount, I/O operation, etc.) of the I-th part task;
wi, weight, represents the importance of subtasks.
4. Service level agreement priority calculation,
Weights are directly assigned based on user requirements and terms of service (e.g., high priority tasks require quick response).
And thirdly, monitoring the performance state of the node of the power host.
1. The index is monitored in real time,
And continuously collecting indexes comprising CPU occupancy rate and memory utilization rate through monitoring probes deployed on all the computing host nodes to form a node performance database with high frequency dynamic update.
2. The standardization and the weight distribution are carried out,
And normalizing the node performance index according to a formula to enable the node performance index to have comparability.
Dynamic weights are assigned to different metrics (e.g., CPU occupancy may be more important than memory usage), and the weight values may be adjusted based on actual load.
And fourthly, constructing and optimizing a dynamic calculation force matching model.
1. The objective function definition:
the dynamic computational power matching model aims at optimizing the resource use efficiency and response time, and combines the following two main targets:
the resource waste is minimized, namely, the distributed resources are ensured to be close to the actual demands, and idle is avoided;
minimizing response time by approximating task completion time to user expectations;
2. dynamic weight policy:
According to the real-time system load, the weights of two targets are dynamically adjusted, and the formula is as follows:
Wfinal=λ1·WRW2·WRD
Wherein, lambda1、λ2 is a weight factor which can be automatically adjusted by a threshold condition.
3. Genetic algorithm optimization:
generating initial populations, wherein each population represents an allocation scheme of tasks and nodes;
Calculating fitness of each scheme based on the objective function;
crossover and mutation, namely introducing mutation factors into the population, improving the diversity of the scheme and avoiding sinking into local optimum;
iterative optimization, namely circularly updating the population until the objective function converges.
And fifthly, the intelligent prediction mechanism assists in optimizing.
1. A load prediction model is constructed and a load prediction model is constructed,
Predicting task load change through time sequence, wherein the formula is as follows:
Rfuture=β·Rcurrent+(1-β)·Rhistorical
Wherein β is a weight factor for weighted averaging between the current value Rcurrent and the historical value Rhistorical, and the influence of the current trend and the historical data on the future predictions can be balanced by adjusting the value of β, specifically as follows:
When β approaches 1, the predicted value Rfuture depends more on the current value Rcurrent, i.e., the current trend is considered to have a greater impact on the future;
When β approaches 0, the predicted value Rfuture depends more on the history value Rhistorical, i.e., the history data is considered to have a greater impact on the future;
When β is equal to 0.5, the predicted value Rfuture is a simple average of the current and historical values, i.e., the effect of both is equal.
By predicting the future high load period, the resource allocation strategy is optimized in advance.
2. The resource preheating and the task migration are carried out,
For predicted high-priority task loads, resources are allocated in advance to reduce starting delay;
and dynamically migrating the non-critical tasks to the low-load nodes so as to reduce resource occupation conflict.
And sixthly, task allocation and execution.
Prioritizing tasks according to real-time Requirements (RTD), concurrency requirements (CD), and SLA priorities;
Performing resource allocation according to the output of the dynamic calculation matching model, and continuously monitoring the effect;
And (3) adaptively adjusting, namely recalculating task priority and resource allocation strategies according to system load changes.
The method solves the problems of resource waste, task accumulation and response delay caused by lack of accurate identification and adaptation of different service characteristics in a scheduling strategy when the existing power-computing host processes large-scale concurrent tasks. And (3) analyzing service characteristics through multi-dimension quantification, constructing a dynamic computing power matching model by combining the performance states of the computing power host nodes, and intelligently adjusting task allocation rules by adopting a multi-objective optimization algorithm and a dynamic weight allocation strategy. Meanwhile, an intelligent prediction mechanism is introduced, the load change trend is perceived in advance, and resource preheating and task migration optimization are realized. The method improves the accuracy and adaptability of task scheduling and has the technical advantages of high efficiency, flexibility and reliability.
The embodiment of the invention also provides an intelligent large-scale task scheduling system based on the business characteristics of the power host, which comprises the following steps:
The data acquisition and preprocessing module is used for realizing data acquisition and preprocessing of the task and the computing host node;
The task business characteristic quantitative analysis module is used for carrying out multidimensional quantitative analysis on the task business characteristics;
The power host node performance state monitoring module is used for dynamically monitoring the power host node performance state;
The dynamic computing force matching model construction and optimization module is used for realizing the construction and optimization of the dynamic computing force matching model;
the intelligent prediction mechanism auxiliary optimization module is used for auxiliary optimization task scheduling based on the intelligent prediction mechanism;
the task allocation and execution module is used for realizing final task allocation and execution;
the system can realize the intelligent large-scale task scheduling method based on the business characteristics of the power host machine, which is described in the embodiment. The specific implementation is as follows:
1. the data acquisition and preprocessing module comprises:
1. The task data is collected and the data is stored,
The scheduling system receives and stores key information of tasks in real time:
determining the time urgency of a task and providing a basis for real-time demand quantification;
the concurrent execution record of the task is extracted from the database and is used for concurrent demand prediction;
Analyzing the resource calling frequency and algorithm complexity of tasks, and determining the computation density degree of different tasks;
Service Level Agreements (SLAs) define task priorities according to an agreement, e.g. VIP service tasks have a higher weight.
2. The data of the nodes of the power main machine are collected,
Through distributed monitored control system, gather every power host computer node's performance data in real time, include:
CPU occupancy rate, which represents the use condition of computing resources and reflects the load capacity of nodes;
the memory utilization rate is that the allocation state of the node memory is displayed, and the suitability of task allocation is evaluated;
network delay, namely describing communication time among nodes and influencing scheduling of cross-node tasks;
And the energy consumption is that the energy consumption of the node in operation is monitored and is used for optimizing the energy efficiency ratio.
3. The data is pre-processed and the data is pre-processed,
In order to enable the task characteristics and the node performance to be directly used for model calculation, normalization processing is needed, and all indexes are uniformly mapped to the [0,1] interval through the following formula:
the normalized data avoid the influence of dimension difference on the subsequent model, and consistent input is provided for scheduling optimization.
2. The task business characteristic quantitative analysis module is used for analyzing the task business characteristics,
Carrying out multidimensional quantitative analysis on task business characteristics to determine the scheduling requirements:
1. The real-time demand is quantified and the real-time demand is quantified,
In order to measure the time urgency of the task, the resource priority allocation is guided, the scheduling is realized by calculating the ratio of the current time to the task deadline, and the formula is as follows:
if RTD approaches 1, the higher the task priority is, the task needs to be immediately scheduled;
if RTD approaches 0, indicating that the task is not urgent, it is suitable for later scheduling.
2. The concurrent demand forecast is based on the result of the demand forecast,
To sense possible concurrency pressure of tasks in advance, optimizing a resource allocation strategy, and predicting future concurrency demands by combining the historical maximum concurrency quantity and the average value:
CD=α·Hmax+(1-α)·Emean
And alpha is a user adjustable factor for adjusting the importance of the maximum concurrency and average concurrency of the history.
The high concurrency task allocates more nodes to meet the demand.
3. The evaluation of the computational complexity is carried out,
In order to obtain the occupation level of the quantized task on the computational power resource, the comprehensive complexity is calculated by analyzing the subtask complexity after task decomposition:
Wherein, Ci is the complexity index (such as calculated amount, I/O operation, etc.) of the I-th part task;
wi, weight, represents the importance of subtasks.
4. Service level agreement priority calculation,
Weights are directly assigned based on user requirements and terms of service (e.g., high priority tasks require quick response).
3. The power host node performance state monitoring module comprises:
1. The index is monitored in real time,
And continuously collecting indexes comprising CPU occupancy rate and memory utilization rate through monitoring probes deployed on all the computing host nodes to form a node performance database with high frequency dynamic update.
2. The standardization and the weight distribution are carried out,
And normalizing the node performance index according to a formula to enable the node performance index to have comparability.
Dynamic weights are assigned to different metrics (e.g., CPU occupancy may be more important than memory usage), and the weight values may be adjusted based on actual load.
4. The dynamic computing force matching model construction and optimization module comprises:
1. The objective function definition:
the dynamic computational power matching model aims at optimizing the resource use efficiency and response time, and combines the following two main targets:
the resource waste is minimized, namely, the distributed resources are ensured to be close to the actual demands, and idle is avoided;
minimizing response time by approximating task completion time to user expectations;
2. dynamic weight policy:
According to the real-time system load, the weights of two targets are dynamically adjusted, and the formula is as follows:
Wfinal=λ1·WRW2·WRD
Wherein, lambda1、λ2 is a weight factor which can be automatically adjusted by a threshold condition.
3. Genetic algorithm optimization:
generating initial populations, wherein each population represents an allocation scheme of tasks and nodes;
Calculating fitness of each scheme based on the objective function;
crossover and mutation, namely introducing mutation factors into the population, improving the diversity of the scheme and avoiding sinking into local optimum;
iterative optimization, namely circularly updating the population until the objective function converges.
5. The intelligent prediction mechanism assists the optimization module, including:
1. a load prediction model is constructed and a load prediction model is constructed,
Predicting task load change through time sequence, wherein the formula is as follows:
Rfuture=β·Rcurrent+(1-β)·Rhistorical
Wherein β is a weight factor for weighted averaging between the current value Rcurrent and the historical value Rhistorical, and the influence of the current trend and the historical data on the future predictions can be balanced by adjusting the value of β, specifically as follows:
When β approaches 1, the predicted value Rfuture depends more on the current value Rcurrent, i.e., the current trend is considered to have a greater impact on the future;
When β approaches 0, the predicted value Rfuture depends more on the history value Rhistorical, i.e., the history data is considered to have a greater impact on the future;
When β is equal to 0.5, the predicted value Rfuture is a simple average of the current and historical values, i.e., the effect of both is equal.
By predicting the future high load period, the resource allocation strategy is optimized in advance.
By predicting the future high load period, the resource allocation strategy is optimized in advance.
2. The resource preheating and the task migration are carried out,
For predicted high-priority task loads, resources are allocated in advance to reduce starting delay;
and dynamically migrating the non-critical tasks to the low-load nodes so as to reduce resource occupation conflict.
6. A task allocation and execution module comprising:
Prioritizing tasks according to real-time Requirements (RTD), concurrency requirements (CD), and SLA priorities;
Performing resource allocation according to the output of the dynamic calculation matching model, and continuously monitoring the effect;
And (3) adaptively adjusting, namely recalculating task priority and resource allocation strategies according to system load changes.
The embodiment of the invention also provides an intelligent large-scale task scheduling realization device based on the business characteristics of the computing host, which comprises at least one memory and at least one processor;
the at least one memory for storing a machine readable program;
The at least one processor is configured to invoke the machine-readable program to implement the intelligent large-scale task scheduling method based on the business characteristics of the computing host described in the foregoing embodiments.
The embodiment of the invention also provides a computer readable medium, wherein the computer readable medium is stored with computer instructions, and the computer instructions realize the intelligent large-scale task scheduling method based on the business characteristics of the power host described in the embodiment when being executed by a processor. Specifically, a system or apparatus provided with a storage medium on which a software program code realizing the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or apparatus may be caused to read out and execute the program code stored in the storage medium.
In this case, the program code itself read from the storage medium may realize the functions of any of the above-described embodiments, and thus the program code and the storage medium storing the program code form part of the present invention.
Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (e.g., CD-ROMs, CD-R, CD-RWs, DVD-ROMs, DVD-RAMs, DVD-RWs, DVD+RWs), magnetic tapes, nonvolatile memory cards, and ROMs. Alternatively, the program code may be downloaded from a server computer by a communication network.
Further, it should be apparent that the functions of any of the above-described embodiments may be implemented not only by executing the program code read out by the computer, but also by causing an operating system or the like operating on the computer to perform part or all of the actual operations based on the instructions of the program code.
Further, it is understood that the program code read out by the storage medium is written into a memory provided in an expansion board inserted into a computer or into a memory provided in an expansion unit connected to the computer, and then a CPU or the like mounted on the expansion board or the expansion unit is caused to perform part and all of actual operations based on instructions of the program code, thereby realizing the functions of any of the above embodiments.
While the invention has been illustrated and described in detail in the drawings and in the preferred embodiments, the invention is not limited to the disclosed embodiments, and it will be appreciated by those skilled in the art that the code audits of the various embodiments described above may be combined to produce further embodiments of the invention, which are also within the scope of the invention.

Claims (10)

Translated fromChinese
1.基于算力主机业务特性的智能化大规模任务调度方法,其特征在于,该方法的实现包括以下步骤:1. An intelligent large-scale task scheduling method based on the business characteristics of the computing power host, characterized in that the implementation of the method includes the following steps:1)任务和算力主机节点的数据采集与预处理;1) Data collection and preprocessing of tasks and computing host nodes;2)对任务业务特性进行多维度量化分析;2) Conduct multi-dimensional quantitative analysis of mission business characteristics;3)算力主机节点性能状态的动态监控;3) Dynamic monitoring of the performance status of computing host nodes;4)动态算力匹配模型构建与优化;4) Construction and optimization of dynamic computing power matching model;5)基于智能预测机制辅助优化任务调度;5) Assist in optimizing task scheduling based on intelligent prediction mechanism;6)最终任务分配与执行。6) Final task allocation and execution.2.根据权利要求1所述的基于算力主机业务特性的智能化大规模任务调度方法,其特征在于,所述数据采集与预处理,包括:2. According to claim 1, the intelligent large-scale task scheduling method based on the business characteristics of the computing power host is characterized in that the data collection and preprocessing include:1.1)任务数据采集:调度系统实时接收并存储任务的关键信息,包括:1.1) Task data collection: The scheduling system receives and stores key task information in real time, including:任务截止时间:确定任务的时间紧迫性,为实时性需求量化提供依据;Task deadline: Determine the time urgency of the task and provide a basis for quantifying real-time requirements;任务历史并发量:从数据库中提取任务的并发执行记录,用于并发需求预测;Historical concurrent task volume: extract concurrent execution records of tasks from the database for concurrent demand prediction;计算复杂度:分析任务的资源调用频率与算法复杂度,确定不同任务的计算密集程度;Computational complexity: Analyze the resource call frequency and algorithm complexity of the task to determine the computational intensity of different tasks;服务级别协议:根据协议定义任务优先级;Service Level Agreement: Define task priorities according to the agreement;1.2)算力主机节点数据采集:通过分布式监控系统,实时采集每个算力主机节点的性能数据,包括:1.2) Data collection of computing host nodes: Through the distributed monitoring system, the performance data of each computing host node is collected in real time, including:CPU占用率;CPU usage;内存利用率;Memory utilization;网络延迟;Network latency;能耗;Energy consumption;1.3)数据预处理:1.3) Data preprocessing:进行归一化处理,通过以下公式将所有指标统一映射到[0,1]区间:Normalize and map all indicators to the [0,1] interval using the following formula:3.根据权利要求1所述的基于算力主机业务特性的智能化大规模任务调度方法,其特征在于,所述业务特性包括实时性需求、并发需求、计算复杂度和服务级别协议;3. The intelligent large-scale task scheduling method based on the business characteristics of the computing power host according to claim 1 is characterized in that the business characteristics include real-time requirements, concurrency requirements, computational complexity and service level agreement;对任务业务特性进行多维度量化分析,明确其调度需求:Conduct multi-dimensional quantitative analysis of task business characteristics to clarify their scheduling requirements:2.1)实时性需求量化:2.1) Quantification of real-time requirements:计算当前时间到任务截止时间的比值,公式如下:Calculate the ratio of the current time to the task deadline. The formula is as follows:如果RTD趋近1,表示任务优先级高,则任务需立即调度;If RTD approaches 1, it means that the task priority is high and the task needs to be scheduled immediately;如果RTD接近0,表示任务不急,适合稍后调度;If the RTD is close to 0, it means that the task is not urgent and is suitable for later scheduling;2.2)并发需求预测:2.2) Concurrent demand forecasting:结合历史最大并发量与平均值预测未来并发需求:Combining the historical maximum concurrency and average value to predict future concurrency requirements:CD=α·Hmax+(1-α)·EmeanCD=α·Hmax +(1-α)·Emean其中,α:用户可调因子,用于调节历史最大并发和平均并发的重要性;Where, α is a user-adjustable factor used to adjust the importance of historical maximum concurrency and average concurrency;高并发任务分配更多节点以满足需求;High-concurrency tasks allocate more nodes to meet demand;2.3)计算复杂度评估:2.3) Computational complexity evaluation:分析任务分解后的子任务复杂度,计算综合复杂度:Analyze the complexity of subtasks after task decomposition and calculate the overall complexity:其中,Ci:第i部分任务的复杂度指标;Where, Ci : complexity index of the i-th task;Wi:权重,表示子任务的重要程度;Wi : weight, indicating the importance of the subtask;2.4)服务级别协议优先级计算:2.4) Service Level Agreement Priority Calculation:基于用户需求和服务条款,直接赋予权值。Weights are directly assigned based on user needs and terms of service.4.根据权利要求1所述的基于算力主机业务特性的智能化大规模任务调度方法,其特征在于,所述算力主机节点性能状态的动态监控,其实现包括:4. The intelligent large-scale task scheduling method based on the business characteristics of the computing power host according to claim 1 is characterized in that the dynamic monitoring of the performance status of the computing power host node includes:3.1)实时监控指标:3.1) Real-time monitoring indicators:通过部署在各算力主机节点的监控探针,持续采集包括CPU占用率、内存使用率的指标,形成高频率动态更新的节点性能数据库;By deploying monitoring probes on each computing host node, we continuously collect indicators including CPU occupancy and memory usage, forming a node performance database that is dynamically updated at a high frequency.3.2)标准化和权重分配:3.2) Standardization and weight distribution:将节点性能指标按公式归一化,使之具备可比性;Normalize the node performance indicators according to the formula to make them comparable;为不同指标分配动态权重,权重值可根据实际负载调整。Dynamic weights are assigned to different indicators, and the weight values can be adjusted according to the actual load.5.根据权利要求1所述的基于算力主机业务特性的智能化大规模任务调度方法,其特征在于,所述动态算力匹配模型基于多目标优化算法和动态权重分配策略,优化任务分配规则;5. The intelligent large-scale task scheduling method based on the business characteristics of computing power hosts according to claim 1 is characterized in that the dynamic computing power matching model optimizes the task allocation rules based on a multi-objective optimization algorithm and a dynamic weight allocation strategy;动态算力匹配模型构建与优化具体包括:The construction and optimization of the dynamic computing power matching model specifically include:4.1)目标函数定义:4.1) Objective function definition:动态算力匹配模型结合以下两大目标优化资源使用效率和响应时间:The dynamic computing power matching model optimizes resource utilization efficiency and response time by combining the following two goals:资源浪费最小化:Minimize resource waste:响应时间最小化:Minimize response time:4.2)动态权重策略:4.2) Dynamic Weight Strategy:根据实时系统负载,动态调整两大目标的权重,公式如下:According to the real-time system load, the weights of the two major goals are dynamically adjusted. The formula is as follows:Wfinal=λ1·WRW2·WRDWfinal1 ·WRW2 ·WRD其中,λ1、λ2:权重因子,可通过阈值条件自动调整;Among them, λ1 , λ2 : weight factors, which can be automatically adjusted by threshold conditions;4.3)遗传算法优化:4.3) Genetic Algorithm Optimization:初始种群生成:每个种群表示任务与节点的分配方案;Initial population generation: Each population represents the allocation scheme of tasks and nodes;适应度计算:基于目标函数计算每个方案的适应度;Fitness calculation: Calculate the fitness of each solution based on the objective function;交叉与变异:在种群中引入变异因子;Crossover and mutation: introducing mutation factors into the population;迭代优化:循环更新种群,直到目标函数收敛。Iterative optimization: The population is updated repeatedly until the objective function converges.6.根据权利要求1所述的基于算力主机业务特性的智能化大规模任务调度方法,其特征在于,所述智能预测机制采用时间序列模型对历史任务数据进行分析,提前感知负载变化趋势;所述任务调度优化包括资源预热和任务迁移策略;6. According to claim 1, the intelligent large-scale task scheduling method based on the business characteristics of the computing power host is characterized in that the intelligent prediction mechanism uses a time series model to analyze historical task data and perceive the load change trend in advance; the task scheduling optimization includes resource preheating and task migration strategies;基于智能预测机制辅助优化任务调度的具体实现包括:The specific implementation of assisted optimization of task scheduling based on intelligent prediction mechanism includes:5.1)构建负载预测模型,通过时间序列预测任务负载变化,公式为:5.1) Build a load prediction model to predict task load changes through time series. The formula is:Rfuture=β·Rcurrent+(1-β)·RhistoricalRfuture =β·Rcurrent +(1-β)·Rhistorical其中,β:权重因子,用于在当前值和历史值之间进行加权平均,通过调整β的值来平衡当前趋势和历史数据对未来预测的影响;Among them, β is a weight factor, which is used to perform a weighted average between the current value and the historical value. By adjusting the value of β, the impact of the current trend and historical data on future predictions can be balanced;通过预测未来高负载时段,提前优化资源分配策略;Optimize resource allocation strategies in advance by predicting future high-load periods;5.2)资源预热与任务迁移:5.2) Resource preheating and task migration:针对预测的高优先级任务负载,提前分配资源以降低启动延迟;Allocate resources in advance to reduce startup delays for predicted high-priority task loads;将非关键任务动态迁移到低负载节点,以减少资源占用冲突。Dynamically migrate non-critical tasks to low-load nodes to reduce resource occupation conflicts.7.根据权利要求1所述的基于算力主机业务特性的智能化大规模任务调度方法,其特征在于,所述任务分配与执行,包括:7. The intelligent large-scale task scheduling method based on computing host service characteristics according to claim 1 is characterized in that the task allocation and execution includes:优先级排序:根据实时性需求、并发需求和SLA优先级排序任务;Prioritization: prioritize tasks based on real-time requirements, concurrency requirements, and SLA priorities;分配与调整:按动态算力匹配模型的输出执行资源分配,并持续监控效果;Allocation and adjustment: Execute resource allocation according to the output of the dynamic computing power matching model and continuously monitor the results;自适应调整:根据系统负载变化重新计算任务优先级和资源分配策略。Adaptive adjustment: Recalculate task priorities and resource allocation strategies based on changes in system load.8.基于算力主机业务特性的智能化大规模任务调度系统,其特征在于,包括:8. An intelligent large-scale task scheduling system based on the business characteristics of the computing power host, characterized by including:数据采集与预处理模块,用于实现任务和算力主机节点的数据采集与预处理;Data collection and preprocessing module, used to realize data collection and preprocessing of tasks and computing host nodes;任务业务特性量化分析模块,用于对任务业务特性进行多维度量化分析;Mission business characteristics quantitative analysis module, used to conduct multi-dimensional quantitative analysis of mission business characteristics;算力主机节点性能状态监控模块,用于算力主机节点性能状态的动态监控;The computing power host node performance status monitoring module is used for dynamic monitoring of the performance status of the computing power host node;动态算力匹配模型构建与优化模块,用于实现动态算力匹配模型的构建与优化;Dynamic computing power matching model construction and optimization module, used to realize the construction and optimization of dynamic computing power matching model;智能预测机制辅助优化模块,用于基于智能预测机制辅助优化任务调度;Intelligent prediction mechanism assisted optimization module, used to assist in optimizing task scheduling based on intelligent prediction mechanism;任务分配与执行模块,用于实现最终任务分配与执行;Task allocation and execution module, used to realize final task allocation and execution;该系统能够实现权利要求1至7任一所述的方法。The system can implement the method described in any one of claims 1 to 7.9.基于算力主机业务特性的智能化大规模任务调度实现装置,其特征在于,包括:至少一个存储器和至少一个处理器;9. An intelligent large-scale task scheduling implementation device based on the business characteristics of a computing host, characterized in that it includes: at least one memory and at least one processor;所述至少一个存储器,用于存储机器可读程序;The at least one memory is used to store a machine-readable program;所述至少一个处理器,用于调用所述机器可读程序,实现权利要求1至7任一所述的方法。The at least one processor is used to call the machine-readable program to implement the method described in any one of claims 1 to 7.10.计算机可读介质,其特征在于,所述计算机可读介质上存储有计算机指令,所述计算机指令在被处理器执行时,能够实现权利要求1至7任一项所述的方法。10. A computer-readable medium, characterized in that computer instructions are stored on the computer-readable medium, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 7 can be implemented.
CN202510144306.2A2025-02-102025-02-10 Intelligent large-scale task scheduling method and system based on computing host business characteristicsPendingCN120104276A (en)

Priority Applications (1)

Application NumberPriority DateFiling DateTitle
CN202510144306.2ACN120104276A (en)2025-02-102025-02-10 Intelligent large-scale task scheduling method and system based on computing host business characteristics

Applications Claiming Priority (1)

Application NumberPriority DateFiling DateTitle
CN202510144306.2ACN120104276A (en)2025-02-102025-02-10 Intelligent large-scale task scheduling method and system based on computing host business characteristics

Publications (1)

Publication NumberPublication Date
CN120104276Atrue CN120104276A (en)2025-06-06

Family

ID=95872857

Family Applications (1)

Application NumberTitlePriority DateFiling Date
CN202510144306.2APendingCN120104276A (en)2025-02-102025-02-10 Intelligent large-scale task scheduling method and system based on computing host business characteristics

Country Status (1)

CountryLink
CN (1)CN120104276A (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication numberPriority datePublication dateAssigneeTitle
CN120336033A (en)*2025-06-172025-07-18青岛国实科技集团有限公司 Dynamic allocation method of heterogeneous computing resources for marine supercomputing environment
CN120508404A (en)*2025-07-222025-08-19八维通科技有限公司AIGC-based distributed computing power scheduling method and system
CN120540819A (en)*2025-07-172025-08-26苏州元脑智能科技有限公司Data processing method, device, equipment, medium and product for edge calculation
CN120540819B (en)*2025-07-172025-10-17苏州元脑智能科技有限公司Data processing method, device, equipment, medium and product for edge calculation

Cited By (4)

* Cited by examiner, † Cited by third party
Publication numberPriority datePublication dateAssigneeTitle
CN120336033A (en)*2025-06-172025-07-18青岛国实科技集团有限公司 Dynamic allocation method of heterogeneous computing resources for marine supercomputing environment
CN120540819A (en)*2025-07-172025-08-26苏州元脑智能科技有限公司Data processing method, device, equipment, medium and product for edge calculation
CN120540819B (en)*2025-07-172025-10-17苏州元脑智能科技有限公司Data processing method, device, equipment, medium and product for edge calculation
CN120508404A (en)*2025-07-222025-08-19八维通科技有限公司AIGC-based distributed computing power scheduling method and system

Similar Documents

PublicationPublication DateTitle
CN110096349B (en)Job scheduling method based on cluster node load state prediction
CN109960573B (en)Cross-domain computing task scheduling method and system based on intelligent perception
CN120104276A (en) Intelligent large-scale task scheduling method and system based on computing host business characteristics
CN118467186B (en)Multi-tenant server-free platform resource management method and system
CN119322684B (en) Dynamic scheduling method for chip platform resources
Petrovska et al.Sequential Series-Based Prediction Model in Adaptive Cloud Resource Allocation for Data Processing and Security
CN120066720A (en)Intelligent scheduling system and method based on calculation force demand prediction
CN119557067B (en) Method, device and storage medium for collaborative scheduling of computing resources in multi-cloud scenarios
CN118094209A (en) Cloud resource automation management and optimization system
CN120017608B (en) Disaster recovery data recovery method and system for disaster recovery system
CN107566535A (en)Adaptive load balancing strategy based on user concurrent access timing planning in a kind of web map service
CN119225933A (en) A process scheduling method, device, equipment, medium and computer program product
CN117993579A (en)Project declaration flow self-adaptive optimization method, device, equipment and storage medium
CN119739535A (en)Computing power resource scheduling method, system and storage medium
Choppara et al.Efficient task scheduling and load balancing in fog computing for crucial healthcare through deep reinforcement learning
CN120416250A (en) Adaptive load balancing and resource scheduling optimization method based on cloud computing
CN120104352A (en) Computing power node load balancing method, device, equipment and storage medium
CN119781996A (en) Multi-tenant environment intelligent management method, device, equipment and storage medium
CN120256087A (en) A method and system for optimizing computing resources for analytical tasks
CN119883640A (en)Dynamic resource scheduling system and method based on decoupling of CPU and memory
Milocco et al.Evaluating the upper bound of energy cost saving by proactive data center management
CN119292771A (en) Scheduling method, device, equipment and storage medium
CN117221328B (en)Service scheduling method and system based on situation awareness
CN118092817A (en)Intelligent management method and system for space of tablet personal computer
CN117112213A (en)Resource scheduling method and device and electronic equipment

Legal Events

DateCodeTitleDescription
PB01Publication
PB01Publication
SE01Entry into force of request for substantive examination
SE01Entry into force of request for substantive examination

[8]ページ先頭

©2009-2025 Movatter.jp