




技术领域technical field
本发明涉及计算机技术领域,尤其涉及一种任务处理方法、装置、调度服务器及介质。The present invention relates to the field of computer technology, and in particular, to a task processing method, device, scheduling server and medium.
背景技术Background technique
随着网络技术的发展以及互联网用户数量的不断增长,应用的后台服务器往往会接收到用户发送的批量任务,采用单个服务器对批量任务进行处理会导致处理效率低下,为了提升针对批量任务的处理效率,当前多采用服务器集群对批量任务进行处理,其中,服务器集群中包括多个执行服务器,每个执行服务器都可以对任务进行处理。With the development of network technology and the continuous growth of the number of Internet users, the background server of the application often receives batch tasks sent by users. Using a single server to process batch tasks will lead to low processing efficiency. In order to improve the processing efficiency of batch tasks Currently, a server cluster is mostly used to process batch tasks, wherein the server cluster includes multiple execution servers, and each execution server can process the tasks.
现有技术中,采用服务器集群对任务进行处理时,往往是用于任务分配的调度服务器在接收到一个任务后,立即将该任务发送至适配的执行服务器,使得执行服务器对任务进行处理,然而,采用上述方式,针对N个任务,则调度服务器需要建立N个线程发送该N个任务,对调度服务器的消耗较高,且发送N次也需占用大量的发送时长,执行服务器一次也只能接收到一个待处理任务,而剩余大量的任务处理空间,导致单个执行服务器的性能无法得到充分发挥,任务处理效率低。In the prior art, when a server cluster is used to process a task, the scheduling server for task allocation often sends the task to the adapted execution server immediately after receiving the task, so that the execution server processes the task, However, using the above method, for N tasks, the scheduling server needs to establish N threads to send the N tasks, which consumes a lot of the scheduling server, and N times of sending also takes up a lot of sending time, and the execution server only needs to be executed once. A task to be processed can be received, but a large amount of task processing space is left, resulting in that the performance of a single execution server cannot be fully utilized, and the task processing efficiency is low.
发明内容SUMMARY OF THE INVENTION
本发明实施例提供了一种任务处理方法、装置、调度服务器及介质,可以对基于执行服务器的运行状态对任务进行合并处理后发送至执行服务器,提升针对任务处理的效率。Embodiments of the present invention provide a task processing method, device, scheduling server and medium, which can combine tasks based on the running state of the execution server and send them to the execution server, thereby improving the efficiency of task processing.
第一方面,本发明实施例提供了一种任务处理方法,所述方法包括:In a first aspect, an embodiment of the present invention provides a task processing method, the method comprising:
接收终端发送的N个待处理任务,并确定用于任务处理的服务器集群中每个执行服务器的状态信息以及每个待处理任务对应的任务量,所述状态信息包括处理效率和剩余负载量,所述N为正整数;Receive N tasks to be processed sent by the terminal, and determine the status information of each execution server in the server cluster used for task processing and the task amount corresponding to each task to be processed, and the status information includes processing efficiency and remaining load amount, The N is a positive integer;
根据所述每个执行服务器的状态信息从所述服务器集群中筛选出第一执行服务器,并确定所述第一执行服务器的第一剩余负载量;Screen out the first execution server from the server cluster according to the status information of each execution server, and determine the first remaining load of the first execution server;
根据所述第一剩余负载量从所述N个待处理任务中筛选出M个第一待处理任务,所述M个第一待处理任务的任务量的和值与所述第一剩余负载量之间的差值小于预设阈值,所述M为小于或等于N的正整数;M first to-be-processed tasks are screened from the N to-be-processed tasks according to the first remaining load amount, and the sum of the task amounts of the M first to-be-processed tasks and the first remaining load amount The difference between them is less than a preset threshold, and the M is a positive integer less than or equal to N;
对所述M个第一待处理任务进行聚合处理,得到第一待处理任务集合;performing aggregation processing on the M first to-be-processed tasks to obtain a first to-be-processed task set;
将所述第一待处理任务集合发送至所述第一执行服务器,以使得所述第一执行服务器对所述第一待处理任务集合中的M个第一待处理任务进行处理。Sending the first set of tasks to be processed to the first execution server, so that the first execution server processes the M first tasks to be processed in the first set of tasks to be processed.
第二方面,本发明实施例提供了一种任务处理装置,所述装置包括:In a second aspect, an embodiment of the present invention provides a task processing device, the device comprising:
接收模块,用于接收终端发送的N个待处理任务;a receiving module for receiving N pending tasks sent by the terminal;
确定模块,用于确定用于任务处理的服务器集群中每个执行服务器的状态信息以及每个待处理任务对应的任务量,所述状态信息包括处理效率和剩余负载量,所述N为正整数;A determination module, used for determining the status information of each execution server in the server cluster used for task processing and the task amount corresponding to each task to be processed, the status information includes processing efficiency and remaining load, and N is a positive integer ;
筛选模块,用于根据所述每个执行服务器的状态信息从所述服务器集群中筛选出第一执行服务器;a screening module, configured to screen out the first execution server from the server cluster according to the state information of each execution server;
所述确定模块,还用于确定所述第一执行服务器的第一剩余负载量;The determining module is further configured to determine the first remaining load of the first execution server;
所述筛选模块,还用于根据所述第一剩余负载量从所述N个待处理任务中筛选出M个第一待处理任务,所述M个第一待处理任务的任务量的和值与所述第一剩余负载量之间的差值小于预设阈值,所述M为小于或等于N的正整数;The screening module is further configured to screen out M first to-be-processed tasks from the N to-be-processed tasks according to the first remaining load, and the sum of the task amounts of the M first to-be-processed tasks The difference from the first remaining load is less than a preset threshold, and M is a positive integer less than or equal to N;
处理模块,用于对所述M个第一待处理任务进行聚合处理,得到第一待处理任务集合;a processing module, configured to perform aggregation processing on the M first tasks to be processed to obtain a first set of tasks to be processed;
发送模块,用于将所述第一待处理任务集合发送至所述第一执行服务器,以使得所述第一执行服务器对所述第一待处理任务集合中的M个第一待处理任务进行处理。A sending module, configured to send the first set of tasks to be processed to the first execution server, so that the first execution server performs the processing on the M first tasks to be processed in the set of first tasks to be processed deal with.
第三方面,本发明实施例提供了一种调度服务器,包括处理器、输入接口、输出接口和存储器,所述处理器、输入接口、输出接口和存储器相互连接,其中,所述存储器用于存储计算机程序,所述计算机程序包括程序指令,所述处理器被配置用于调用所述程序指令,执行第一方面所述的方法。In a third aspect, an embodiment of the present invention provides a scheduling server, including a processor, an input interface, an output interface, and a memory, wherein the processor, the input interface, the output interface, and the memory are connected to each other, wherein the memory is used for storing A computer program comprising program instructions, the processor being configured to invoke the program instructions to perform the method of the first aspect.
第四方面,本发明实施例提供了一种计算机可读存储介质,其特征在于,所述计算机可读存储介质存储有计算机程序,所述计算机程序包括程序指令,所述程序指令当被处理器执行时使所述处理器执行第一方面所述的方法。In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program includes program instructions, and the program instructions are executed by a processor. When executed, the processor is caused to perform the method of the first aspect.
本发明实施例中,调度服务器在接收到待处理任务时,以执行服务器的处理效率和剩余负载量为参考因素选取该执行服务器需要处理的待处理任务,使得执行服务器可以尽可能饱和的处理待处理任务,充分发挥了执行服务器的性能,提升了服务器对待处理任务的处理效率。In the embodiment of the present invention, when the scheduling server receives the pending tasks, the processing efficiency and the remaining load of the execution server are used as reference factors to select the pending tasks to be processed by the execution server, so that the execution server can process the pending tasks as saturated as possible. The processing tasks give full play to the performance of the execution server and improve the processing efficiency of the tasks to be processed by the server.
附图说明Description of drawings
为了更清楚地说明本发明实施例技术方案,下面将对实施例描述中所需要使用的附图作简单地介绍,显而易见地,下面描述中的附图是本发明的一些实施例,对于本领域普通技术人员来讲,在不付出创造性劳动的前提下,还可以根据这些附图获得其他的附图。In order to explain the technical solutions of the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings used in the description of the embodiments. Obviously, the drawings in the following description are some embodiments of the present invention, which are of great significance to the art For those of ordinary skill, other drawings can also be obtained from these drawings without any creative effort.
图1是本发明实施例提供的一种任务处理系统的结构示意图;1 is a schematic structural diagram of a task processing system provided by an embodiment of the present invention;
图2是本发明实施例提供的另一种任务处理方法的流程示意图;2 is a schematic flowchart of another task processing method provided by an embodiment of the present invention;
图3是本发明实施例提供的另一种任务处理方法的流程示意图;3 is a schematic flowchart of another task processing method provided by an embodiment of the present invention;
图4是本发明实施例提供的一种任务处理装置的结构示意图;4 is a schematic structural diagram of a task processing apparatus provided by an embodiment of the present invention;
图5是本发明实施例提供的一种调度服务器的结构示意图。FIG. 5 is a schematic structural diagram of a scheduling server provided by an embodiment of the present invention.
具体实施方式Detailed ways
下面将结合本发明实施例中的附图,对本发明实施例中的技术方案进行清楚、完整地描述,显然,所描述的实施例是本发明一部分实施例,而不是全部的实施例。基于本发明中的实施例,本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例,都属于本发明保护的范围。The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.
请参见图1,为本发明实施例提供一种任务处理系统,该任务处理系统包括:调度服务器,与调度服务器相连接的至少一个终端,以及与调度服务器相连接的服务器集群,该服务器集群中包括至少一个执行服务器。其中,终端可包括但不限于:手机、平板电脑、电脑等,用于与用户进行交互,例如接收用户上传的待处理任务;以及用于与调度服务器进行交互,例如将待处理任务发送至调度服务器以请求获得任务处理服务等等。调度服务器用于接收终端发送的待处理任务,并将待处理任务分配各个执行服务器,具体可以接收终端发送的N个待处理任务,并确定用于任务处理的服务器集群中每个执行服务器的状态信息以及每个待处理任务对应的任务量,调度服务器根据每个执行服务器的状态信息从服务器集群中筛选出第一执行服务器,并确定第一执行服务器的第一剩余负载量,状态信息包括处理效率和剩余负载量,调度服务器根据第一剩余负载量从所述N个待处理任务中筛选出M个第一待处理任务,并对M个第一待处理任务进行聚合处理,得到第一待处理任务集合,调度服务器将第一待处理任务集合发送至所述第一执行服务器,以使得所述第一执行服务器对所述第一待处理任务集合中的M个第一待处理任务进行处理。通过上述方式,在接收到待处理任务时,调度服务器以执行服务器的处理效率和剩余负载量为参考因素选取该执行服务器需要处理的待处理任务,使得执行服务器可以尽可能饱和的处理待处理任务,充分发挥了执行服务器的性能,提升了服务器对待处理任务的处理效率。Referring to FIG. 1, an embodiment of the present invention provides a task processing system. The task processing system includes: a scheduling server, at least one terminal connected to the scheduling server, and a server cluster connected to the scheduling server. Include at least one execution server. The terminals may include but are not limited to: mobile phones, tablet computers, computers, etc., for interacting with users, such as receiving pending tasks uploaded by users; and for interacting with scheduling servers, such as sending pending tasks to scheduling The server requests the task processing service and so on. The scheduling server is used to receive the pending tasks sent by the terminal, and assign the pending tasks to each execution server. Specifically, it can receive N pending tasks sent by the terminal, and determine the status of each execution server in the server cluster used for task processing. information and the task amount corresponding to each task to be processed, the scheduling server selects the first execution server from the server cluster according to the status information of each execution server, and determines the first remaining load of the first execution server. The status information includes processing efficiency and remaining load, the scheduling server selects M first pending tasks from the N pending tasks according to the first remaining load, and aggregates the M first pending tasks to obtain the first pending tasks. Processing a task set, the scheduling server sends the first task set to be processed to the first execution server, so that the first execution server processes the M first tasks to be processed in the first set of tasks to be processed . In the above manner, when receiving a pending task, the scheduling server selects the pending task to be processed by the execution server with the processing efficiency and remaining load of the execution server as reference factors, so that the execution server can process the pending tasks as saturated as possible. , give full play to the performance of the execution server, and improve the processing efficiency of the task to be processed by the server.
基于上述任务处理系统的实施例的描述,本发明实施例提供一种任务处理方法,该方法可以应用于图1所示的任务处理系统中,具体可以由图1所示的调度服务器执行。请参见图2,该任务处理过程可包括以下步骤S201-S207:Based on the description of the above embodiments of the task processing system, an embodiment of the present invention provides a task processing method, which can be applied to the task processing system shown in FIG. 1 , and can be specifically executed by the scheduling server shown in FIG. 1 . Referring to FIG. 2, the task processing process may include the following steps S201-S207:
S201、调度服务器接收终端发送的N个待处理任务,并确定用于任务处理的服务器集群中每个执行服务器的状态信息以及每个待处理任务对应的任务量。S201. The scheduling server receives the N tasks to be processed sent by the terminal, and determines the status information of each execution server in the server cluster used for task processing and the task amount corresponding to each task to be processed.
本发明实施例中,N个待处理任务可以由用户通过终端进行输入,终端检测到用户输入的待处理任务之后,将N个待处理任务发送至调度服务器,调度服务器对终端发送的N个待处理任务进行接收。其中,N个待处理任务可以为同一终端进行发送N个待处理任务,或者为调度服务器在某一时间段接收到的不同终端发送的N个待处理任务。In this embodiment of the present invention, the N tasks to be processed may be input by the user through the terminal. After the terminal detects the tasks to be processed input by the user, the terminal sends the N tasks to be processed to the scheduling server, and the scheduling server sends the N tasks to the terminal to the scheduling server. Process the task to receive. The N pending tasks may be N pending tasks sent by the same terminal, or N pending tasks sent by different terminals received by the scheduling server in a certain period of time.
调度服务器接收到N个待处理任务之后,将确定用于任务处理的服务器集群中每个执行服务器的状态信息以及每个待处理任务对应的任务量。其中,任务量具体可以为服务器处理待处理任务时需占用的内存大小,如100兆、200兆等,待处理任务的任务量可以由终端在发送待处理任务时一并发送,或者,由服务器对待处理任务进行检测,确定待处理任务的任务量。状态信息包括处理效率和剩余负载量,处理效率用于体现执行服务器处理性能,具体可以如每秒处理100兆的任务、每秒处理200兆的任务等,不同的执行服务器因硬件设备不同或当前处理任务的数量不同可以体现不同的处理效率,剩余负载量具体可以为执行服务器剩余的内存量,如500兆、1000兆等。具体的,调度服务器可以管理及维护一个剩余负载表,该剩余负载表如下表一所示:After receiving the N tasks to be processed, the scheduling server will determine the status information of each execution server in the server cluster used for task processing and the task amount corresponding to each task to be processed. Among them, the task amount can be the memory size that the server needs to take up when processing the pending tasks, such as 100 megabytes, 200 megabytes, etc. The task amount of the pending tasks can be sent by the terminal when sending the pending tasks, or, by the server. The task to be processed is detected, and the task amount of the task to be processed is determined. Status information includes processing efficiency and remaining load. Processing efficiency is used to reflect the processing performance of the execution server. Specifically, it can process 100 megabytes of tasks per second and 200 megabytes of tasks per second. Different execution servers have different hardware devices or current conditions. Different numbers of processing tasks can reflect different processing efficiencies, and the remaining load can specifically be the remaining memory of the execution server, such as 500 megabytes, 1000 megabytes, and so on. Specifically, the scheduling server can manage and maintain a remaining load table, which is shown in Table 1 below:
表一:剩余负载表Table 1: Residual Load Table
如上表一所示,剩余负载表记录了服务器集群中各个执行服务器的剩余负载量,在一种实现方式中,由各个执行服务器可定时(每隔2秒、每隔5秒等)上报各自的剩余负载量,由调度服务器判断各执行服务器的剩余负载量是否发生变化,若发生变化则根据上报的数据对上述表一进行实时更新;在另一种实现方式中,当服务器集群中的各执行服务器自检发现自己的剩余负载量发生变化时,将变化后的剩余负载量上报至调度服务器,由调度服务器根据上报的数据对上述表一进行实时更新。调度服务器可以从表一中获取到每个执行服务器的剩余负载量。As shown in Table 1 above, the remaining load table records the remaining load of each execution server in the server cluster. In one implementation, each execution server can report the respective For the remaining load, the scheduling server determines whether the remaining load of each execution server has changed. If there is a change, the above table 1 is updated in real time according to the reported data; in another implementation, when each execution server in the server cluster changes When the server self-checking finds that its remaining load has changed, it will report the changed remaining load to the scheduling server, and the scheduling server will update the above table 1 in real time according to the reported data. The scheduling server can obtain the remaining load of each execution server from Table 1.
S202、调度服务器根据每个执行服务器的状态信息从服务器集群中筛选出第一执行服务器,并确定第一执行服务器的第一剩余负载量。S202. The scheduling server selects the first execution server from the server cluster according to the state information of each execution server, and determines the first remaining load of the first execution server.
本发明实施例中,调度服务器确定每个待处理任务对应的任务量以及每个执行服务器的状态信息之后,将根据每个执行服务器的状态信息从服务器集群中筛选出第一执行服务器,并确定第一执行服务器的第一剩余负载量。In the embodiment of the present invention, after determining the task amount corresponding to each task to be processed and the status information of each execution server, the scheduling server will filter out the first execution server from the server cluster according to the status information of each execution server, and determine The first remaining load of the first execution server.
在一种实现方式中,调度服务器根据每个执行服务器的状态信息从服务器集群中筛选出第一执行服务器的具体方式可以为,调度服务器确定服务器集群中每个执行服务器的状态信息中的处理效率,并将服务器集群中处理效率最高的执行服务器确定为第一执行服务器。例如,服务器集群中包括执行服务器1、执行服务器2和执行服务器3,执行服务器1的处理效率为每秒处理300兆的任务,执行服务器2的处理效率为每秒处理200兆的任务,执行服务器3的处理效率为每秒处理100兆的任务,则调度服务器将执行服务器1确定为第一执行服务器。In an implementation manner, the specific manner in which the scheduling server selects the first execution server from the server cluster according to the status information of each execution server may be that the scheduling server determines the processing efficiency in the status information of each execution server in the server cluster. , and determine the execution server with the highest processing efficiency in the server cluster as the first execution server. For example, the server cluster includes execution server 1, execution server 2 and execution server 3. The processing efficiency of execution server 1 is to process 300 megabytes of tasks per second, and the processing efficiency of execution server 2 is to process 200 megabytes of tasks per second. If the processing efficiency of 3 is to process 100 megabytes of tasks per second, the scheduling server determines execution server 1 as the first execution server.
在一种实现方式中,调度服务器根据每个执行服务器的状态信息从服务器集群中筛选出第一执行服务器的具体方式也可以为,调度服务器确定服务器集群中每个执行服务器的状态信息中的剩余负载量,并将服务器集群中剩余负载量最大的执行服务器确定为第一执行服务器。例如,服务器集群中包括执行服务器1、执行服务器2和执行服务器3,执行服务器1的剩余负载量为500兆,执行服务器2的剩余负载量为300兆,执行服务器3的剩余负载量为200兆,则调度服务器将执行服务器1确定为第一执行服务器。In an implementation manner, the specific manner in which the scheduling server selects the first execution server from the server cluster according to the status information of each execution server may also be that the scheduling server determines the remaining status information of each execution server in the server cluster. load, and the execution server with the largest remaining load in the server cluster is determined as the first execution server. For example, the server cluster includes execution server 1, execution server 2 and execution server 3. The remaining load of execution server 1 is 500 MB, the remaining load of execution server 2 is 300 MB, and the remaining load of execution server 3 is 200 MB. , the scheduling server determines execution server 1 as the first execution server.
在一种实现方式中,调度服务器根据每个执行服务器的状态信息从服务器集群中筛选出第一执行服务器的具体方式也可以为,调度服务器根据处理效率与权重的对应关系确定服务器集群中每个执行服务器的处理效率对应的第一权重;以及,根据剩余负载量与权重的对应关系确定服务器集群中每个执行服务器的剩余负载量对应的第二权重;根据每个执行服务器的第一权重和第二权重确定每个执行服务器对应的目标权重,并将目标权重最高的执行服务器确定为第一执行服务器,每个执行服务器对应的目标权重为每个执行服务器对应的第一权重与第二权重的和值。其中,处理效率与权重的对应关系,以及剩余负载量与权重的对应关系具体可以由研发人员预先设定。例如,服务器集群中包括执行服务器1和执行服务器2,执行服务器1的处理效率为每秒处理300兆的任务,对应的第一权重为3,执行服务器1的剩余负载量为500兆,对应的第二权重为5,则执行服务器1对应的目标权重为8,执行服务器2的处理效率为每秒处理200兆的任务,对应的第一权重为2,执行服务器1的剩余负载量为300兆,对应的第二权重为3,则执行服务器2对应的目标权重为5,调度服务器将服务器1确定为第一执行服务器。In an implementation manner, the specific manner in which the scheduling server selects the first execution server from the server cluster according to the status information of each execution server may also be: the scheduling server determines each execution server in the server cluster according to the corresponding relationship between processing efficiency and weight. the first weight corresponding to the processing efficiency of the execution server; and, according to the corresponding relationship between the remaining load and the weight, determine the second weight corresponding to the remaining load of each execution server in the server cluster; according to the first weight of each execution server and The second weight determines the target weight corresponding to each execution server, and the execution server with the highest target weight is determined as the first execution server, and the target weight corresponding to each execution server is the first weight and the second weight corresponding to each execution server. and value. The corresponding relationship between the processing efficiency and the weight, and the corresponding relationship between the remaining load and the weight can be specifically preset by the research and development personnel. For example, the server cluster includes execution server 1 and execution server 2. The processing efficiency of execution server 1 is to process 300 megabytes of tasks per second, the corresponding first weight is 3, and the remaining load of execution server 1 is 500 megabytes. The corresponding The second weight is 5, the target weight corresponding to execution server 1 is 8, the processing efficiency of execution server 2 is to process 200 megabytes of tasks per second, the corresponding first weight is 2, and the remaining load of execution server 1 is 300 megabytes , the corresponding second weight is 3, then the target weight corresponding to execution server 2 is 5, and the scheduling server determines server 1 as the first execution server.
进一步的,调度服务器确定第一执行服务器之后,将确定第一执行服务器对应的第一剩余负载量。Further, after the scheduling server determines the first execution server, it will determine the first remaining load corresponding to the first execution server.
S203、调度服务器根据第一剩余负载量从N个待处理任务中筛选出M个第一待处理任务。S203: The scheduling server selects M first to-be-processed tasks from the N to-be-processed tasks according to the first remaining load.
本发明实施例中,调度服务器确定第一执行服务器对应的第一剩余负载量之后,将根据第一剩余负载量从N个待处理任务中筛选出M个第一待处理任务。若第一剩余负载量大于或等于该N个待处理任务对应的任务量的和值,则调度服务器直接将该N个待处理任务确定为该M个第一待处理任务,若第一剩余负载量小于该N个待处理任务对应的任务量的和值,则调度服务器根据第一剩余负载量从N个待处理任务中筛选出M个第一待处理任务,其中,M个第一待处理任务的任务量的和值与第一剩余负载量之间的差值小于预设阈值,M为小于或等于N的正整数,和值与第一剩余负载量之间的差值的具体计算方式为第一剩余负载量减去和值,且差值需为整数。In the embodiment of the present invention, after the scheduling server determines the first remaining load corresponding to the first execution server, it will select M first pending tasks from N pending tasks according to the first remaining load. If the first remaining load is greater than or equal to the sum of the tasks corresponding to the N tasks to be processed, the scheduling server directly determines the N tasks to be processed as the M first tasks to be processed. If the first remaining load If the amount is less than the sum of the task amounts corresponding to the N tasks to be processed, the scheduling server selects M first tasks to be processed from the N tasks to be processed according to the first remaining load, wherein the M first tasks to be processed The difference between the sum of the tasks of the task and the first remaining load is less than the preset threshold, M is a positive integer less than or equal to N, and the specific calculation method of the difference between the sum and the first remaining load Subtract the sum for the first remaining load, and the difference must be an integer.
在一种实现方式中,调度服务器根据第一剩余负载量从N个待处理任务中筛选出M个第一待处理任务的具体方式可以为,调度服务器将N个待处理任务进行随机组合,得到多个待处理任务组合,并确定每个待处理任务组合中包括的待处理任务的任务量的和值,并计算第一剩余负载量与每个和值之间的差值,在差值为正数的情况下,若存在差值小于预设阈值的组合,则调度服务器确定出与第一剩余负载量之间的差值最小的和值对应的待处理任务组合,并将该待处理任务组合中的各个待处理任务确定为该M个第一待处理任务。In an implementation manner, a specific manner in which the scheduling server selects the M first pending tasks from the N pending tasks according to the first remaining load may be that the scheduling server randomly combines the N pending tasks to obtain A plurality of tasks to be processed are combined, and the sum of the task amounts of the tasks to be processed included in each combination of tasks to be processed is determined, and the difference between the first remaining load and each sum is calculated, where the difference is In the case of a positive number, if there is a combination whose difference is less than the preset threshold, the scheduling server determines the combination of tasks to be processed corresponding to the sum of the smallest difference between the first remaining loads, and assigns the task to be processed. Each to-be-processed task in the combination is determined as the M first to-be-processed tasks.
在一种实现方式中,调度服务器根据第一剩余负载量从N个待处理任务中筛选出M个第一待处理任务的具体方式可以为,调度服务器从N个待处理任务中随机筛选M个待处理任务,若该M个待处理任务的任务量的和值与第一剩余负载量之间的差值小于预设阈值,则将该M个待处理任务确定为M个第一待处理任务。若该M个待处理任务的任务量的和值与第一剩余负载量之间的差值大于或等于预设阈值,则调度服务器再次随机筛选M个待处理任务,并再次对该M个待处理任务的任务量的和值与第一剩余负载量之间的差值进行计算。In an implementation manner, a specific manner in which the scheduling server selects the M first pending tasks from the N pending tasks according to the first remaining load amount may be that the scheduling server randomly selects M tasks from the N pending tasks For tasks to be processed, if the difference between the sum of the task amounts of the M tasks to be processed and the first remaining load is less than the preset threshold, then the M tasks to be processed are determined as the M first tasks to be processed . If the difference between the sum of the task amounts of the M tasks to be processed and the first remaining load is greater than or equal to the preset threshold, the scheduling server randomly selects the M tasks to be processed again, and re-selects the M tasks to be processed again. The difference between the sum of the task amounts of the processing tasks and the first remaining load amount is calculated.
在一种实现方式中,调度服务器根据第一剩余负载量从N个待处理任务中筛选出M个第一待处理任务的具体方式可以为,调度服务器获取第一剩余负载量之后,基于第一剩余负载量从N个待处理任务中选取出M个待处理任务,使得该M个待处理任务的任务量的和值与第一剩余负载量之间的差值最小,当该M个待处理任务的任务量的和值与第一剩余负载量之间的差值小于预设阈值时,调度服务器将该M个待处理任务确定为M个第一待处理任务。例如,第一剩余负载量为500兆,N为3,M为2,待处理任务1的任务量为200,待处理任务2的任务量为400,待处理任务3的任务量为200,预设阈值为200,则调度服务器确定待处理任务1和待处理任务3的和值为400,与第一任务量之间的差值为100,并确定待处理任务1和待处理任务3为第一待处理任务。In an implementation manner, the specific manner in which the scheduling server selects the M first tasks to be processed from the N tasks to be processed according to the first remaining load may be: after the scheduling server obtains the first remaining load, based on the first remaining load The remaining load selects M tasks to be processed from the N tasks to be processed, so that the difference between the sum of the tasks of the M tasks to be processed and the first remaining load is the smallest. When the M tasks to be processed When the difference between the sum of the task amounts of the tasks and the first remaining load amount is smaller than the preset threshold, the scheduling server determines the M to-be-processed tasks as the M first to-be-processed tasks. For example, the first remaining load is 500 MB, N is 3, M is 2, the task volume of task 1 to be processed is 200, the task volume of task 2 to be processed is 400, the task volume of task 3 to be processed is 200, and the If the threshold value is 200, the scheduling server determines that the sum of task 1 to be processed and task 3 to be processed is 400, and the difference between it and the first task amount is 100, and determines that task 1 to be processed and task 3 to be processed are the first A pending task.
S204、调度服务器对M个第一待处理任务进行聚合处理,得到第一待处理任务集合。S204: The scheduling server performs aggregation processing on the M first tasks to be processed to obtain a first set of tasks to be processed.
本发明实施例中,调度服务器筛选出M个第一待处理任务之后,将对M个第一待处理任务进行聚合处理,得到第一待处理任务集合,即将M个待处理任务放入同一个集合中。In the embodiment of the present invention, after the scheduling server selects the M first tasks to be processed, it aggregates the M first tasks to be processed to obtain the first set of tasks to be processed, that is, puts the M tasks to be processed into the same in the collection.
S205、调度服务器将第一待处理任务集合发送至第一执行服务器,以使得第一执行服务器对第一待处理任务集合中的M个第一待处理任务进行处理。S205: The scheduling server sends the first task set to be processed to the first execution server, so that the first execution server processes the M first tasks to be processed in the first set of tasks to be processed.
本发明实施例中,调度服务器确定出第一待处理任务集合后,可以将第一待处理任务集合发送至第一执行服务器,以使得第一执行服务器对第一待处理任务集合中的M个第一待处理任务进行处理。In this embodiment of the present invention, after determining the first task set to be processed, the scheduling server may send the first set of tasks to be processed to the first execution server, so that the first execution server can The first pending task is processed.
在一种实现方式中,第一执行服务器包括多个接口,每个接口接收到的任务对应的处理优先级不同,调度服务器将第一待处理任务集合发送至第一执行服务器的具体方式可以为,调度服务器获取第一待处理任务集合中各个第一待处理任务的时间戳,根据各个第一待处理任务的时间戳确定每个第一待处理任务对应的接口,并建立每个待处理任务与接口之间的连接,调度服务器基于该连接将第一待处理任务集合中的各个待处理任务发送至第一服务器中对应的接口,使得第一服务器对所述各个待处理任务进行接收。各个第一待处理任务的时间戳具体可以包括调度服务器接收到各个第一待处理任务的时间,调度服务器在接收到N个待处理任务时,将确定该N个待处理任务的时间戳。In an implementation manner, the first execution server includes a plurality of interfaces, and the tasks received by each interface have different processing priorities. The specific method for the scheduling server to send the first task set to be processed to the first execution server may be as follows , the scheduling server obtains the timestamp of each first pending task in the first pending task set, determines the interface corresponding to each first pending task according to the timestamp of each first pending task, and establishes each pending task The connection with the interface, the scheduling server sends each to-be-processed task in the first to-be-processed task set to the corresponding interface in the first server based on the connection, so that the first server receives the each to-be-processed task. The time stamp of each first task to be processed may specifically include the time when the scheduling server receives each first task to be processed. When the scheduling server receives N tasks to be processed, the time stamp of the N tasks to be processed will be determined.
在一种实现方式中,第一执行服务器包括多个接口,每个接口接收到的任务类型不同,调度服务器将第一待处理任务集合发送至第一执行服务器的具体方式可以为,调度服务器确定第一待处理任务集合中各个第一待处理任务的任务类型,根据各个第一待处理任务的任务类型确定每个第一待处理任务对应的接口,并建立每个待处理任务与接口之间的连接,调度服务器基于该连接将第一待处理任务集合中的各个待处理任务发送至第一服务器中对应的接口,使得第一服务器对所述各个待处理任务进行接收。第一待处理任务的任务类型可以为检索型、计算型、资源获取型等。In an implementation manner, the first execution server includes a plurality of interfaces, and each interface receives a different task type. A specific manner for the scheduling server to send the first set of tasks to be processed to the first execution server may be as follows: the scheduling server determines The task type of each first to-be-processed task in the first to-be-processed task set, the interface corresponding to each first to-be-processed task is determined according to the task type of each first to-be-processed task, and the interface between each to-be-processed task and the interface is established. The scheduling server sends each to-be-processed task in the first to-be-processed task set to the corresponding interface in the first server based on the connection, so that the first server receives the each to-be-processed task. The task type of the first task to be processed may be retrieval type, calculation type, resource acquisition type, and the like.
S206、调度服务器接收第一执行服务器返回的针对第一待处理任务集合的第一处理结果。S206. The scheduling server receives the first processing result for the first task set to be processed returned by the first execution server.
本发明实施例中,第一执行服务器可以对第一待处理任务集合中的所有第一待处理任务都处理完成后,将所有第一待处理任务的处理结果合并得到第一处理结果,并返回值调度服务器,调度服务器将该第一处理结果发送至对应的终端。或者,第一执行服务器可以在处理完成第一待处理任务集合中的任意一个任务,得到第一处理结果后,立即向调度服务器返回该第一处理结果,调度服务器对该第一处理结果进行接收。进一步的,调度服务器也可以接收服务器集群中其他各个执行服务器返回的处理结果。In this embodiment of the present invention, the first execution server may combine the processing results of all the first to-be-processed tasks to obtain the first processing result after processing all the first to-be-processed tasks in the first to-be-processed task set, and return the result. A value scheduling server, where the scheduling server sends the first processing result to the corresponding terminal. Alternatively, the first execution server may return the first processing result to the scheduling server immediately after processing and completing any task in the first set of tasks to be processed and obtain the first processing result, and the scheduling server may receive the first processing result . Further, the scheduling server may also receive processing results returned by other execution servers in the server cluster.
S207、调度服务器将第一处理结果发送至终端。S207. The scheduling server sends the first processing result to the terminal.
本发明实施例中,调度服务器接收到第一执行服务器返回的第一处理结果之后,可以将第一处理结果发送至所述终端。具体的,调度服务器获取与终端之间的通信信息,通信信息包括通信距离和通信带宽;调度服务器根据通信信息确定针对第一处理结果的发送时间;并在计算得到的发送时间时向终端发送第一处理结果,以使得终端接收到所述第一处理结果的时间与预设时间相同。其中,调度服务器具体可以基于通信距离和通信带宽计算得到处理结果发送至终端所需要的传输时长,并在考虑传输时长因素的情况下向终端发送第一处理结果,使得终端接收处理结果的时间与预期相同。在一种实施场景中,调度服务器需要将处理结果同时返回至不同的终端,此时,调度服务器可以获取每个终端与调度服务器之间的距离以及通信带宽,并基于距离和通信带宽确定针对处理结果的发送时间,将距离较长的终端优先发送处理结果,距离较短的终端延后发送处理结果。以实现各个终端对于处理结果的同时接收。上述场景具体可以为抢红包场景,具体的,调度服务器可以检测每个终端与自身的距离以及通信带宽,并基于上述因素确定针对红包的发送时间,使得每个终端都具有相同的概率抢到该红包,而不至于因距离过长或者网速较慢导致某些终端始终无法抢到红包。In this embodiment of the present invention, after receiving the first processing result returned by the first execution server, the scheduling server may send the first processing result to the terminal. Specifically, the scheduling server obtains the communication information with the terminal, and the communication information includes the communication distance and the communication bandwidth; the scheduling server determines the sending time for the first processing result according to the communication information; and sends the first processing result to the terminal at the calculated sending time. a processing result, so that the time when the terminal receives the first processing result is the same as the preset time. Specifically, the scheduling server may calculate and obtain the transmission time required for sending the processing result to the terminal based on the communication distance and communication bandwidth, and send the first processing result to the terminal in consideration of the transmission time factor, so that the time for the terminal to receive the processing result is the same as that of the terminal. Expect the same. In one implementation scenario, the scheduling server needs to return the processing results to different terminals at the same time. At this time, the scheduling server can obtain the distance and communication bandwidth between each terminal and the scheduling server, and determine the processing results based on the distance and the communication bandwidth. For the sending time of the result, the terminal with a longer distance sends the processing result first, and the terminal with a shorter distance delays sending the processing result. In order to realize the simultaneous reception of the processing result by each terminal. The above scenario may specifically be a red envelope grabbing scenario. Specifically, the scheduling server can detect the distance between each terminal and itself and the communication bandwidth, and determine the sending time for the red envelope based on the above factors, so that each terminal has the same probability to grab the red envelope. Red envelopes, so that some terminals cannot always grab red envelopes due to the long distance or slow network speed.
本发明实施例中,调度服务器接收终端发送的N个待处理任务,并确定用于任务处理的服务器集群中每个执行服务器的状态信息以及每个待处理任务对应的任务量,调度服务器根据每个执行服务器的状态信息从服务器集群中筛选出第一执行服务器,并确定第一执行服务器的第一剩余负载量;调度服务器根据第一剩余负载量从所述N个待处理任务中筛选出M个第一待处理任务,并对M个第一待处理任务进行聚合处理,得到第一待处理任务集合,调度服务器将第一待处理任务集合发送至所述第一执行服务器,以使得所述第一执行服务器对所述第一待处理任务集合中的M个第一待处理任务进行处理。通过实施上述方法,在接收到待处理任务时,以执行服务器的处理效率和剩余负载量为参考因素选取该执行服务器需要处理的待处理任务,使得执行服务器可以尽可能饱和的处理待处理任务,充分发挥了执行服务器的性能,提升了服务器对待处理任务的处理效率,并且,在发送过程中基于数据的传输时长确定发送时间,使得终端可以在预期时间接收到数据,提升了数据接收的稳定性。In the embodiment of the present invention, the scheduling server receives N tasks to be processed sent by the terminal, and determines the status information of each execution server in the server cluster used for task processing and the task amount corresponding to each task to be processed. The state information of the execution servers selects the first execution server from the server cluster, and determines the first remaining load of the first execution server; the scheduling server selects M from the N pending tasks according to the first remaining load. the first tasks to be processed, and the M first tasks to be processed are aggregated to obtain a first set of tasks to be processed. The scheduling server sends the first set of tasks to be processed to the first execution server, so that the The first execution server processes the M first to-be-processed tasks in the first to-be-processed task set. By implementing the above method, when a pending task is received, the processing efficiency and remaining load of the execution server are used as reference factors to select the pending task to be processed by the execution server, so that the execution server can process the pending task as saturated as possible, Give full play to the performance of the execution server, improve the processing efficiency of the server to be processed, and determine the sending time based on the transmission time of the data during the sending process, so that the terminal can receive the data at the expected time, which improves the stability of data reception. .
基于上述任务处理系统和方法的实施例的描述,本发明实施例提供另一种任务处理方法,该方法可以应用于图1所示的任务处理系统中,具体可以由图1所示的调度服务器执行。请参见图3,该任务处理过程可包括以下步骤S301-S306:Based on the description of the above embodiments of the task processing system and method, the embodiments of the present invention provide another task processing method, which can be applied to the task processing system shown in FIG. implement. Referring to FIG. 3, the task processing process may include the following steps S301-S306:
S301、调度服务器接收终端发送的N个待处理任务,并确定用于任务处理的服务器集群中每个执行服务器的状态信息以及每个待处理任务对应的任务量。S301. The scheduling server receives the N tasks to be processed sent by the terminal, and determines the status information of each execution server in the server cluster used for task processing and the task amount corresponding to each task to be processed.
本发明实施例中,N个待处理任务可以由用户通过终端进行输入,终端检测到用户输入的待处理任务之后,将N个待处理任务发送至调度服务器,调度服务器对终端发送的N个待处理任务进行接收。其中,N个待处理任务可以为同一终端进行发送N个待处理任务,或者为调度服务器在某一时间段接收到的不同终端发送的N个待处理任务。调度服务器接收到N个待处理任务之后,将确定用于任务处理的服务器集群中每个执行服务器的状态信息以及每个待处理任务对应的任务量。In this embodiment of the present invention, the N tasks to be processed may be input by the user through the terminal. After the terminal detects the tasks to be processed input by the user, the terminal sends the N tasks to be processed to the scheduling server, and the scheduling server sends the N tasks to the terminal to the scheduling server. Process the task to receive. The N pending tasks may be N pending tasks sent by the same terminal, or N pending tasks sent by different terminals received by the scheduling server in a certain period of time. After receiving the N tasks to be processed, the scheduling server will determine the status information of each execution server in the server cluster used for task processing and the task amount corresponding to each task to be processed.
S302、调度服务器根据每个执行服务器的状态信息从服务器集群中筛选出第一执行服务器,并确定第一执行服务器的第一剩余负载量。S302. The scheduling server selects the first execution server from the server cluster according to the state information of each execution server, and determines the first remaining load of the first execution server.
本发明实施例中,调度服务器确定每个待处理任务对应的任务量以及每个执行服务器的状态信息之后,将根据每个执行服务器的状态信息从服务器集群中筛选出第一执行服务器,并确定第一执行服务器的第一剩余负载量。In the embodiment of the present invention, after determining the task amount corresponding to each task to be processed and the status information of each execution server, the scheduling server will filter out the first execution server from the server cluster according to the status information of each execution server, and determine The first remaining load of the first execution server.
S303、调度服务器根据第一剩余负载量从N个待处理任务中筛选出M个第一待处理任务,并对M个第一待处理任务进行聚合处理,得到第一待处理任务集合。S303. The scheduling server selects M first to-be-processed tasks from the N to-be-processed tasks according to the first remaining load, and aggregates the M first to-be-processed tasks to obtain a first to-be-processed task set.
本发明实施例中,调度服务器确定第一执行服务器对应的第一剩余负载量之后,将根据第一剩余负载量从N个待处理任务中筛选出M个第一待处理任务。若第一剩余负载量大于或等于该N个待处理任务对应的任务量的和值,则调度服务器直接将该N个待处理任务确定为该M个第一待处理任务,若第一剩余负载量小于该N个待处理任务对应的任务量的和值,则调度服务器根据第一剩余负载量从N个待处理任务中筛选出M个第一待处理任务,其中,M个第一待处理任务的任务量的和值与第一剩余负载量之间的差值小于预设阈值。In the embodiment of the present invention, after the scheduling server determines the first remaining load corresponding to the first execution server, it will select M first pending tasks from N pending tasks according to the first remaining load. If the first remaining load is greater than or equal to the sum of the tasks corresponding to the N tasks to be processed, the scheduling server directly determines the N tasks to be processed as the M first tasks to be processed. If the first remaining load If the amount is less than the sum of the task amounts corresponding to the N tasks to be processed, the scheduling server selects M first tasks to be processed from the N tasks to be processed according to the first remaining load, wherein the M first tasks to be processed The difference between the sum of the task amounts of the tasks and the first remaining load amount is smaller than the preset threshold.
进一步的,当M小于N时,调度服务器执行步骤S304。Further, when M is less than N, the scheduling server executes step S304.
S304、调度服务器根据每个执行服务器的状态信息从服务器集群中筛选出第二执行服务器,并确定第二执行服务器的第二剩余负载量。S304: The scheduling server selects the second execution server from the server cluster according to the state information of each execution server, and determines the second remaining load of the second execution server.
在一种实现方式中,调度服务器根据每个执行服务器的状态信息从服务器集群中筛选出第二执行服务器的具体方式可以为,调度服务器确定服务器集群中每个执行服务器的状态信息中的处理效率,并将服务器集群中处理效率次高的执行服务器确定为第一执行服务器。In an implementation manner, the specific manner in which the scheduling server selects the second execution server from the server cluster according to the status information of each execution server may be that the scheduling server determines the processing efficiency in the status information of each execution server in the server cluster. , and determine the execution server with the second highest processing efficiency in the server cluster as the first execution server.
在一种实现方式中,调度服务器根据每个执行服务器的状态信息从服务器集群中筛选出第二执行服务器的具体方式也可以为,调度服务器确定服务器集群中每个执行服务器的状态信息中的剩余负载量,并将服务器集群中剩余负载量次高的执行服务器确定为第二执行服务器。In an implementation manner, the specific manner in which the scheduling server selects the second execution server from the server cluster according to the status information of each execution server may also be: the scheduling server determines the remaining status information of each execution server in the server cluster. load, and the execution server with the second highest remaining load in the server cluster is determined as the second execution server.
在一种实现方式中,调度服务器根据每个执行服务器的状态信息从服务器集群中筛选出第二执行服务器的具体方式也可以为,调度服务器根据处理效率与权重的对应关系确定服务器集群中每个执行服务器的处理效率对应的第一权重;以及,根据剩余负载量与权重的对应关系确定服务器集群中每个执行服务器的剩余负载量对应的第二权重;根据每个执行服务器的第一权重和第二权重确定每个执行服务器对应的目标权重,并将目标权重次高的执行服务器确定为第二执行服务器,每个执行服务器目标权重为所述每个执行服务器的第一权重与第二权重的和值。进一步的,调度服务器确定第二执行服务器的第二剩余负载量。In an implementation manner, the specific manner in which the scheduling server selects the second execution server from the server cluster according to the state information of each execution server may also be: the scheduling server determines each execution server in the server cluster according to the corresponding relationship between processing efficiency and weight. the first weight corresponding to the processing efficiency of the execution server; and, according to the corresponding relationship between the remaining load and the weight, determine the second weight corresponding to the remaining load of each execution server in the server cluster; according to the first weight of each execution server and The second weight determines the target weight corresponding to each execution server, and the execution server with the second highest target weight is determined as the second execution server, and the target weight of each execution server is the first weight and the second weight of each execution server. and value. Further, the scheduling server determines the second remaining load of the second execution server.
S305、调度服务器根据第二剩余负载量从除去M个第一待处理任务外的N-M个待处理任务中筛选出K个第二待处理任务。S305: The scheduling server selects K second to-be-processed tasks from N-M to-be-processed tasks excluding the M first to-be-processed tasks according to the second remaining load.
本发明实施例中,K为小于N-M的正整数,终端确定第二执行服务器的第二剩余负载量之后,将检测剩余的N-M个待处理任务对应的任务量的和值是否小于该第二剩余负载量,若是,则调度服务器直接将该N-M个待处理任务确定为K个第二待处理任务,并对所述K个第二待处理任务进行聚合处理,得到第二待处理任务集合;若剩余的N-M个待处理任务对应的任务量的和值大于该第二剩余负载量,则调度服务器采用与步骤S203相同的方式筛选出K个第二待处理任务,并对所述K个第二待处理任务进行聚合处理,得到第二待处理任务集合。In this embodiment of the present invention, K is a positive integer smaller than N-M. After determining the second remaining load of the second execution server, the terminal will detect whether the sum of the remaining N-M tasks to be processed is smaller than the second remaining load. load, if yes, the scheduling server directly determines the N-M tasks to be processed as K second tasks to be processed, and aggregates the K second tasks to be processed to obtain a second set of tasks to be processed; The sum of the task amounts corresponding to the remaining N-M tasks to be processed is greater than the second remaining load, then the scheduling server selects K second tasks to be processed in the same manner as in step S203, and the K second tasks are The tasks to be processed are aggregated to obtain a second set of tasks to be processed.
进一步的,调度服务器可以继续根据服务器集群中每个执行服务器的剩余负载量对剩余的N-M-K个待处理任务进行集合划分,并确定每个集合对应的执行服务器,以使得N个待处理任务中每个待处理任务都对应有相应的执行服务器进行处理。Further, the scheduling server can continue to divide the remaining N-M-K tasks to be processed according to the remaining load of each execution server in the server cluster, and determine the execution server corresponding to each set, so that each of the N tasks to be processed can be divided into sets. Each to-be-processed task corresponds to a corresponding execution server for processing.
S306、调度服务器将第一待处理任务集合发送至第一执行服务器,以及将第二待处理任务集合发送至第二执行服务器。S306. The scheduling server sends the first task set to be processed to the first execution server, and sends the second task set to be processed to the second execution server.
本发明实施例中,调度服务器第一待处理任务集合之后,可以将将第一待处理任务集合发送至第一执行服务器,以使得所述第一执行服务器对所述第一待处理任务集合中的M个第一待处理任务进行处理,以及将第二待处理任务集合发送至第二执行服务器,以使得所述第二执行服务器对所述第二待处理任务集合中的K个第二待处理任务进行处理,进一步的,调度服务器可以接收第一执行服务器和第二执行服务器返回的处理结果,并将接收到的处理结果返回至终端。In this embodiment of the present invention, after the first task set to be processed by the scheduling server, the first set of tasks to be processed may be sent to the first execution server, so that the first execution server can process the first set of tasks to be processed in the first set. The M first tasks to be processed are processed, and the second set of tasks to be processed is sent to the second execution server, so that the second execution server performs processing on the K second tasks to be processed in the second set of tasks to be processed. The processing task is processed, and further, the scheduling server may receive the processing results returned by the first execution server and the second execution server, and return the received processing results to the terminal.
本发明实施例中,调度服务器在接收到待处理任务时,以执行服务器的处理效率和剩余负载量为参考因素选取该执行服务器需要处理的待处理任务,使得服务器可以尽可能饱和的处理待处理任务,充分发挥了执行服务器的性能,提升了服务器对待处理任务的处理效率。In the embodiment of the present invention, when the scheduling server receives the pending tasks, the processing efficiency and the remaining load of the execution server are used as reference factors to select the pending tasks to be processed by the execution server, so that the server can process the pending tasks as saturated as possible. tasks, give full play to the performance of the execution server, and improve the processing efficiency of the tasks to be processed by the server.
基于上述任务处理方法实施例的描述,本发明实施例还公开了一种任务处理装置。该任务处理装置可以是运行于调度服务器中的一个计算机程序(包括程序代码),也可以是包含在调度服务器中的一个实体装置。该任务处理装置可以执行图2和图3所示的方法。请参见图4,该任务处理装置40包括:接收模块401、确定模块402、筛选模块403、处理模块404、发送模块405。Based on the description of the above task processing method embodiments, an embodiment of the present invention further discloses a task processing apparatus. The task processing device may be a computer program (including program code) running in the scheduling server, or may be a physical device included in the scheduling server. The task processing apparatus can execute the methods shown in FIG. 2 and FIG. 3 . Referring to FIG. 4 , the
接收模块401,用于接收终端发送的N个待处理任务;A receiving
确定模块402,用于确定用于任务处理的服务器集群中每个执行服务器的状态信息以及每个待处理任务对应的任务量,所述状态信息包括处理效率和剩余负载量,所述N为正整数;A
筛选模块403,用于根据所述每个执行服务器的状态信息从所述服务器集群中筛选出第一执行服务器;A
所述确定模块402,还用于确定所述第一执行服务器的第一剩余负载量;The determining
所述筛选模块403,还用于根据所述第一剩余负载量从所述N个待处理任务中筛选出M个第一待处理任务,所述M个第一待处理任务的任务量的和值与所述第一剩余负载量之间的差值小于预设阈值,所述M为小于或等于N的正整数;The
处理模块404,用于对所述M个第一待处理任务进行聚合处理,得到第一待处理任务集合;a
发送模块405,用于将所述第一待处理任务集合发送至所述第一执行服务器,以使得所述第一执行服务器对所述第一待处理任务集合中的M个第一待处理任务进行处理。A sending
在一种实现方式中,确定模块402,具体用于:In an implementation manner, the determining
确定所述服务器集群中每个执行服务器的状态信息中的处理效率,并将所述服务器集群中处理效率最高的执行服务器确定为第一执行服务器;或者,Determine the processing efficiency in the status information of each execution server in the server cluster, and determine the execution server with the highest processing efficiency in the server cluster as the first execution server; or,
确定所述服务器集群中每个执行服务器的状态信息中的剩余负载量,并将所述服务器集群中剩余负载量最大的执行服务器确定为第一执行服务器。Determine the remaining load in the state information of each execution server in the server cluster, and determine the execution server with the largest remaining load in the server cluster as the first execution server.
在一种实现方式中,确定模块402,具体用于:In an implementation manner, the determining
根据处理效率与权重的对应关系确定所述服务器集群中每个执行服务器的处理效率对应的第一权重;Determine the first weight corresponding to the processing efficiency of each execution server in the server cluster according to the correspondence between the processing efficiency and the weight;
根据剩余负载量与权重的对应关系确定所述服务器集群中每个执行服务器的剩余负载量对应的第二权重;Determine the second weight corresponding to the remaining load of each execution server in the server cluster according to the correspondence between the remaining load and the weight;
根据所述每个执行服务器对应的第一权重和第二权重确定所述每个执行服务器对应的目标权重,并将目标权重最高的执行服务器确定为第一执行服务器,所述每个执行服务器对应的目标权重为所述每个执行服务器对应的第一权重与第二权重的和值。The target weight corresponding to each execution server is determined according to the first weight and the second weight corresponding to each execution server, and the execution server with the highest target weight is determined as the first execution server, and each execution server corresponds to The target weight is the sum of the first weight and the second weight corresponding to each execution server.
在一种实现方式中,筛选模块403,具体用于:In an implementation manner, the
根据所述每个执行服务器的状态信息从所述服务器集群中筛选出第二执行服务器,并确定所述第二执行服务器的第二剩余负载量;Screen out a second execution server from the server cluster according to the state information of each execution server, and determine the second remaining load of the second execution server;
根据所述第二剩余负载量从除去所述M个第一待处理任务外的N-M个待处理任务中筛选出K个第二待处理任务,所述K为小于或等于N-M的正整数;Filter out K second to-be-processed tasks from N-M to-be-processed tasks excluding the M first to-be-processed tasks according to the second remaining load, where K is a positive integer less than or equal to N-M;
对所述K个第二待处理任务进行聚合处理,得到第二待处理任务集合;Perform aggregation processing on the K second tasks to be processed to obtain a second set of tasks to be processed;
将所述第二待处理任务集合发送至所述第二执行服务器,以使得所述第二执行服务器对所述第二待处理任务集合中的K个第二待处理任务进行处理。Sending the second set of tasks to be processed to the second execution server, so that the second execution server processes the K second tasks to be processed in the second set of tasks to be processed.
在一种实现方式中,发送模块405,具体用于:In an implementation manner, the sending
获取所述第一待处理任务集合中各个第一待处理任务的时间戳;obtaining the timestamp of each first task to be processed in the first set of tasks to be processed;
根据所述各个第一待处理任务的时间戳确定每个第一待处理任务对应的接口,并建立每个待处理任务与接口之间的连接,所述接口为所述第一服务器中用于任务请求接收的接口,所述第一服务器中不同接口接收到的任务对应的处理优先级不同;Determine the interface corresponding to each first to-be-processed task according to the timestamps of the first to-be-processed tasks, and establish a connection between each to-be-processed task and the interface, where the interface is used in the first server for An interface for receiving a task request, the processing priorities corresponding to tasks received by different interfaces in the first server are different;
基于所述连接将所述第一待处理任务集合中的各个待处理任务发送至所述第一服务器中对应的接口,使得所述第一服务器对所述各个待处理任务进行接收。Based on the connection, each to-be-processed task in the first to-be-processed task set is sent to a corresponding interface in the first server, so that the first server receives the each to-be-processed task.
在一种实现方式中,接收模块401,还用于接收所述第一执行服务器返回的针对第一待处理任务集合的第一处理结果;In an implementation manner, the receiving
发送模块405,还用于将所述第一处理结果发送至所述终端。The sending
在一种实现方式中,发送模块405,具体用于:In an implementation manner, the sending
获取与所述终端之间的通信信息,所述通信信息包括通信距离和通信带宽;obtaining communication information with the terminal, the communication information including communication distance and communication bandwidth;
根据所述通信信息确定针对所述第一处理结果的发送时间;determining the sending time for the first processing result according to the communication information;
在所述发送时间时向所述终端发送第一处理结果,以使得所述终端接收到所述第一处理结果的时间与预设时间相同。The first processing result is sent to the terminal at the sending time, so that the time when the terminal receives the first processing result is the same as the preset time.
本发明实施例中,接收模块401接收终端发送的N个待处理任务,确定模块402确定用于任务处理的服务器集群中每个执行服务器的状态信息以及每个待处理任务对应的任务量,筛选模块403根据每个执行服务器的状态信息从服务器集群中筛选出第一执行服务器,确定模块402确定第一执行服务器的第一剩余负载量;筛选模块403根据第一剩余负载量从所述N个待处理任务中筛选出M个第一待处理任务,处理模块404对M个第一待处理任务进行聚合处理,得到第一待处理任务集合,发送模块405将第一待处理任务集合发送至所述第一执行服务器。通过实施上述方法,在接收到待处理任务时,以执行服务器的处理效率和剩余负载量为参考因素选取该执行服务器需要处理的待处理任务,使得执行服务器可以尽可能饱和的处理待处理任务,充分发挥了执行服务器的性能,提升了服务器对待处理任务的处理效率。In this embodiment of the present invention, the receiving
请参见图5,为本发明实施例提供的一种调度服务器的结构示意图。如图5所示,该调度服务器包括:至少一个处理器501,输入设备503,输出设备504,存储器505,至少一个通信总线502。其中,通信总线502用于实现这些组件之间的连接通信。其中,存储器505可以是高速RAM存储器,也可以是非不稳定的存储器(non-volatile memory),例如至少一个磁盘存储器。存储器505可选的还可以是至少一个位于远离前述处理器501的存储装置。其中处理器501可以结合图4所描述的装置,存储器505中存储一组程序代码,且处理器501,输入设备503,输出设备504调用存储器505中存储的程序代码,用于执行以下操作:Referring to FIG. 5 , it is a schematic structural diagram of a scheduling server according to an embodiment of the present invention. As shown in FIG. 5 , the scheduling server includes: at least one
输入设备503,用于接收终端发送的N个待处理任务,并确定用于任务处理的服务器集群中每个执行服务器的状态信息以及每个待处理任务对应的任务量,所述状态信息包括处理效率和剩余负载量,所述N为正整数;The
处理器501,用于根据所述每个执行服务器的状态信息从所述服务器集群中筛选出第一执行服务器,并确定所述第一执行服务器的第一剩余负载量;A
处理器501,用于根据所述第一剩余负载量从所述N个待处理任务中筛选出M个第一待处理任务,所述M个第一待处理任务的任务量的和值与所述第一剩余负载量之间的差值小于预设阈值,所述M为小于或等于N的正整数;The
处理器501,用于对所述M个第一待处理任务进行聚合处理,得到第一待处理任务集合;A
输出设备504,用于将所述第一待处理任务集合发送至所述第一执行服务器,以使得所述第一执行服务器对所述第一待处理任务集合中的M个第一待处理任务进行处理。An
在一种实现方式中,处理器501,具体用于:In an implementation manner, the
确定所述服务器集群中每个执行服务器的状态信息中的处理效率,并将所述服务器集群中处理效率最高的执行服务器确定为第一执行服务器;或者,Determine the processing efficiency in the status information of each execution server in the server cluster, and determine the execution server with the highest processing efficiency in the server cluster as the first execution server; or,
确定所述服务器集群中每个执行服务器的状态信息中的剩余负载量,并将所述服务器集群中剩余负载量最大的执行服务器确定为第一执行服务器。Determine the remaining load in the state information of each execution server in the server cluster, and determine the execution server with the largest remaining load in the server cluster as the first execution server.
在一种实现方式中,处理器501,具体用于:In an implementation manner, the
根据处理效率与权重的对应关系确定所述服务器集群中每个执行服务器的处理效率对应的第一权重;Determine the first weight corresponding to the processing efficiency of each execution server in the server cluster according to the correspondence between the processing efficiency and the weight;
根据剩余负载量与权重的对应关系确定所述服务器集群中每个执行服务器的剩余负载量对应的第二权重;Determine the second weight corresponding to the remaining load of each execution server in the server cluster according to the correspondence between the remaining load and the weight;
根据所述每个执行服务器对应的第一权重和第二权重确定所述每个执行服务器对应的目标权重,并将目标权重最高的执行服务器确定为第一执行服务器,所述每个执行服务器对应的目标权重为所述每个执行服务器对应的第一权重与第二权重的和值。The target weight corresponding to each execution server is determined according to the first weight and the second weight corresponding to each execution server, and the execution server with the highest target weight is determined as the first execution server, and each execution server corresponds to The target weight is the sum of the first weight and the second weight corresponding to each execution server.
在一种实现方式中,处理器501,具体用于:In an implementation manner, the
根据所述每个执行服务器的状态信息从所述服务器集群中筛选出第二执行服务器,并确定所述第二执行服务器的第二剩余负载量;Screen out a second execution server from the server cluster according to the state information of each execution server, and determine the second remaining load of the second execution server;
根据所述第二剩余负载量从除去所述M个第一待处理任务外的N-M个待处理任务中筛选出K个第二待处理任务,所述K为小于或等于N-M的正整数;Filter out K second to-be-processed tasks from N-M to-be-processed tasks excluding the M first to-be-processed tasks according to the second remaining load, where K is a positive integer less than or equal to N-M;
对所述K个第二待处理任务进行聚合处理,得到第二待处理任务集合;Perform aggregation processing on the K second tasks to be processed to obtain a second set of tasks to be processed;
将所述第二待处理任务集合发送至所述第二执行服务器,以使得所述第二执行服务器对所述第二待处理任务集合中的K个第二待处理任务进行处理。Sending the second set of tasks to be processed to the second execution server, so that the second execution server processes the K second tasks to be processed in the second set of tasks to be processed.
在一种实现方式中,处理器501,具体用于:In an implementation manner, the
获取所述第一待处理任务集合中各个第一待处理任务的时间戳;obtaining the timestamp of each first task to be processed in the first set of tasks to be processed;
根据所述各个第一待处理任务的时间戳确定每个第一待处理任务对应的接口,并建立每个待处理任务与接口之间的连接,所述接口为所述第一服务器中用于任务请求接收的接口,所述第一服务器中不同接口接收到的任务对应的处理优先级不同;Determine the interface corresponding to each first to-be-processed task according to the timestamps of the first to-be-processed tasks, and establish a connection between each to-be-processed task and the interface, where the interface is used in the first server for An interface for receiving a task request, the processing priorities corresponding to tasks received by different interfaces in the first server are different;
输出设备504,用于基于所述连接将所述第一待处理任务集合中的各个待处理任务发送至所述第一服务器中对应的接口,使得所述第一服务器对所述各个待处理任务进行接收。An
在一种实现方式中,输入设备503,用于接收所述第一执行服务器返回的针对第一待处理任务集合的第一处理结果;In an implementation manner, the
输出设备504,用于将所述第一处理结果发送至所述终端。An
在一种实现方式中,处理器501,具体用于:In an implementation manner, the
获取与所述终端之间的通信信息,所述通信信息包括通信距离和通信带宽;obtaining communication information with the terminal, the communication information including communication distance and communication bandwidth;
根据所述通信信息确定针对所述第一处理结果的发送时间;determining the sending time for the first processing result according to the communication information;
输出设备504,用于在所述发送时间时向所述终端发送第一处理结果,以使得所述终端接收到所述第一处理结果的时间与预设时间相同。The
本发明实施例中,输入设备503接收终端发送的N个待处理任务,处理器501确定用于任务处理的服务器集群中每个执行服务器的状态信息以及每个待处理任务对应的任务量,处理器501根据每个执行服务器的状态信息从服务器集群中筛选出第一执行服务器,处理器501确定第一执行服务器的第一剩余负载量;并根据第一剩余负载量从所述N个待处理任务中筛选出M个第一待处理任务,处理器501对M个第一待处理任务进行聚合处理,得到第一待处理任务集合,输出设备504将第一待处理任务集合发送至所述第一执行服务器。通过实施上述方法,在接收到待处理任务时,以执行服务器的处理效率和剩余负载量为参考因素选取该执行服务器需要处理的待处理任务,使得执行服务器可以尽可能饱和的处理待处理任务,充分发挥了执行服务器的性能,提升了服务器对待处理任务的处理效率。In this embodiment of the present invention, the
本发明实施例中所述模块,可以通过通用集成电路,例如CPU(CentralProcessing Unit,中央处理器),或通过ASIC(Application Specific IntegratedCircuit,专用集成电路)来实现。The modules in the embodiments of the present invention may be implemented by a general-purpose integrated circuit, such as a CPU (Central Processing Unit, central processing unit), or an ASIC (Application Specific Integrated Circuit, an application-specific integrated circuit).
应当理解,在本发明实施例中,所称处理器501可以是中央处理模块(CentralProcessing Unit,CPU),该处理器还可以是其他通用处理器、数字信号处理器(DigitalSignal Processor,DSP)、专用集成电路(Application Specific Integrated Circuit,ASIC)、现成可编程门阵列(Field-Programmable Gate Array,FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。It should be understood that, in this embodiment of the present invention, the so-called
总线502可以是工业标准体系结构(Industry Standard Architecture,ISA)总线、外部设备互联(Peripheral Component,PCI)总线或扩展工业标准体系结构(ExtendedIndustry Standard Architecture,EISA)总线等,该总线502可以分为地点总线、数据总线、控制总线等,为便于表示,图5仅用一条粗线表示,但并不表示仅有一根总线或一种类型的总线。The
本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程,是可以通过计算机程序来指令相关的硬件来完成,所述的程序可存储于计算机可读存储介质中,该程序在执行时,可包括如上述各方法的实施例的流程。其中,所述的计算机可读存储介质可为磁碟、光盘、只读存储记忆体(Read-Only Memory,ROM)或随机存储记忆体(RandomAccess Memory,RAM)等。Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and the program can be executed , may include the flow of the above-mentioned method embodiments. The computer-readable storage medium may be a magnetic disk, an optical disk, a read-only memory (Read-Only Memory, ROM), or a random access memory (Random Access Memory, RAM) or the like.
以上所揭露的仅为本发明较佳实施例而已,当然不能以此来限定本发明之权利范围,因此依本发明权利要求所作的等同变化,仍属本发明所涵盖的范围。The above disclosures are only preferred embodiments of the present invention, and of course, the scope of the rights of the present invention cannot be limited by this. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010227325.9ACN111538572B (en) | 2020-03-26 | 2020-03-26 | Task processing method, device, scheduling server and medium |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010227325.9ACN111538572B (en) | 2020-03-26 | 2020-03-26 | Task processing method, device, scheduling server and medium |
| Publication Number | Publication Date |
|---|---|
| CN111538572Atrue CN111538572A (en) | 2020-08-14 |
| CN111538572B CN111538572B (en) | 2024-11-19 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010227325.9AActiveCN111538572B (en) | 2020-03-26 | 2020-03-26 | Task processing method, device, scheduling server and medium |
| Country | Link |
|---|---|
| CN (1) | CN111538572B (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108170525A (en)* | 2016-12-07 | 2018-06-15 | 晨星半导体股份有限公司 | Device and method for dynamically adjusting task load configuration of multi-core processor |
| CN113220458A (en)* | 2021-05-26 | 2021-08-06 | 西安热工研究院有限公司 | Real-time scheduling method and device for realizing load balance of database connection pool |
| CN113515358A (en)* | 2021-04-30 | 2021-10-19 | 北京奇艺世纪科技有限公司 | Task scheduling method and device, electronic equipment and storage medium |
| CN114168297A (en)* | 2021-12-13 | 2022-03-11 | 杭州安恒信息技术股份有限公司 | Method, device, equipment and medium for scheduling collection tasks |
| CN115469980A (en)* | 2022-07-29 | 2022-12-13 | 天翼云科技有限公司 | A product media download task scheduling method, device and electronic equipment |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160098292A1 (en)* | 2014-10-03 | 2016-04-07 | Microsoft Corporation | Job scheduling using expected server performance information |
| CN108337275A (en)* | 2017-01-19 | 2018-07-27 | 百度在线网络技术(北京)有限公司 | Task distribution method, device and equipment for Distributor |
| CN110209496A (en)* | 2019-05-20 | 2019-09-06 | 中国平安财产保险股份有限公司 | Task sharding method, device and sliced service device based on data processing |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160098292A1 (en)* | 2014-10-03 | 2016-04-07 | Microsoft Corporation | Job scheduling using expected server performance information |
| CN108337275A (en)* | 2017-01-19 | 2018-07-27 | 百度在线网络技术(北京)有限公司 | Task distribution method, device and equipment for Distributor |
| CN110209496A (en)* | 2019-05-20 | 2019-09-06 | 中国平安财产保险股份有限公司 | Task sharding method, device and sliced service device based on data processing |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108170525A (en)* | 2016-12-07 | 2018-06-15 | 晨星半导体股份有限公司 | Device and method for dynamically adjusting task load configuration of multi-core processor |
| CN113515358A (en)* | 2021-04-30 | 2021-10-19 | 北京奇艺世纪科技有限公司 | Task scheduling method and device, electronic equipment and storage medium |
| CN113515358B (en)* | 2021-04-30 | 2024-04-12 | 北京奇艺世纪科技有限公司 | Task scheduling method and device, electronic equipment and storage medium |
| CN113220458A (en)* | 2021-05-26 | 2021-08-06 | 西安热工研究院有限公司 | Real-time scheduling method and device for realizing load balance of database connection pool |
| CN113220458B (en)* | 2021-05-26 | 2022-11-01 | 西安热工研究院有限公司 | A real-time scheduling method and device for realizing load balancing by database connection pool |
| CN114168297A (en)* | 2021-12-13 | 2022-03-11 | 杭州安恒信息技术股份有限公司 | Method, device, equipment and medium for scheduling collection tasks |
| CN115469980A (en)* | 2022-07-29 | 2022-12-13 | 天翼云科技有限公司 | A product media download task scheduling method, device and electronic equipment |
| Publication number | Publication date |
|---|---|
| CN111538572B (en) | 2024-11-19 |
| Publication | Publication Date | Title |
|---|---|---|
| CN111538572A (en) | Task processing method, device, scheduling server and medium | |
| CN107241281B (en) | Data processing method and device | |
| CN110008021B (en) | Memory management method, memory management device, electronic equipment and computer readable storage medium | |
| CN110162270B (en) | Data storage method, storage node and medium based on distributed storage system | |
| CN103763346B (en) | A distributed resource scheduling method and device | |
| CN103262498B (en) | Connect for issuing and subscribe environment server the method and system of optimizing | |
| CN107819797B (en) | Access request processing method and device | |
| CN107968802A (en) | Method, device and filter scheduler for resource scheduling | |
| WO2020134840A1 (en) | Data distribution method and related product | |
| CN111831408A (en) | Asynchronous task processing method and device, electronic equipment and medium | |
| CN110333956A (en) | Message storage method, device, medium and electronic equipment in message queue | |
| CN110912949B (en) | Method and device for submitting sites | |
| US10425273B2 (en) | Data processing system and data processing method | |
| CN111858067B (en) | Data processing method and device | |
| CN115237595A (en) | Data processing method, data processing device, distribution server, data processing system, and storage medium | |
| CN112596985B (en) | IT asset detection method, device, equipment and medium | |
| CN108830724B (en) | Resource data packet processing method and terminal equipment | |
| CN116319810A (en) | Distributed system flow control method, device, equipment, medium and product | |
| CN111459666A (en) | Task dispatch method, device, task execution system and server | |
| CN118796390A (en) | A method for processing a job and a related device | |
| CN117130979A (en) | Service resource migration method and device and electronic equipment | |
| CN116841797A (en) | Service retry management method and device, electronic equipment and readable storage medium | |
| CN113439252B (en) | Terminal control method and device, storage medium and electronic equipment | |
| CN109445934B (en) | Query request distribution method and system | |
| CN114390104A (en) | Process forensics system, method, apparatus, computer device and medium |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |