CN113127096A - Task processing method and device, electronic equipment and storage medium - Google Patents
Task processing method and device, electronic equipment and storage mediumDownload PDFInfo
- Publication number
- CN113127096A CN113127096ACN202110462223.XACN202110462223ACN113127096ACN 113127096 ACN113127096 ACN 113127096ACN 202110462223 ACN202110462223 ACN 202110462223ACN 113127096 ACN113127096 ACN 113127096A
- Authority
- CN
- China
- Prior art keywords
- task
- environment
- execution
- program
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/445—Program loading or initiating
- G06F9/44505—Configuring for program initiating, e.g. using registry, configuration files
- G06F9/4451—User profiles; Roaming
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Stored Programmes (AREA)
Abstract
Translated fromChinese
Description
Translated fromChinese技术领域technical field
本公开涉及大数据相关技术，具体涉及一种任务处理方法和装置、电子设备、存储介质。The present disclosure relates to technologies related to big data, and in particular, to a task processing method and device, an electronic device, and a storage medium.
背景技术Background technique
任务流中包括顺序执行的多个任务，每个任务可以称为该任务流中的一个任务节点。对于其中一个任务节点来说，该任务一般在单一的执行环境中执行。而随着技术的更新升级，同一个任务节点的任务可能具有在不同环境执行的需求，如何满足这种多环境执行的需求成为亟待解决的问题。A task flow includes multiple tasks that are executed sequentially, and each task can be called a task node in the task flow. For one of the task nodes, the task is generally executed in a single execution environment. With the updating and upgrading of technology, the tasks of the same task node may have requirements to be executed in different environments, and how to meet the needs of such multi-environment execution has become an urgent problem to be solved.
发明内容SUMMARY OF THE INVENTION
本公开实施例至少提供一种任务处理方法和装置、电子设备、存储介质，以解决不能满足多环境执行需求的问题。The embodiments of the present disclosure provide at least a task processing method and apparatus, an electronic device, and a storage medium, so as to solve the problem that the multi-environment execution requirements cannot be met.
第一方面，提供一种任务处理方法，所述方法包括：In a first aspect, a task processing method is provided, the method comprising:
基于当前任务节点的任务对应的任务配置文件，获取所述任务的任务程序执行时所需的任务参数、以及所述任务程序执行所需的任务环境信息；Based on the task configuration file corresponding to the task of the current task node, obtain task parameters required for the execution of the task program of the task and task environment information required for the execution of the task program;
根据所述任务环境信息，将所述任务程序和所述任务参数提供至所述任务环境信息对应的目标执行环境；According to the task environment information, provide the task program and the task parameters to the target execution environment corresponding to the task environment information;
在所述目标执行环境中，根据所述任务程序和任务参数执行所述任务。In the target execution environment, the task is executed according to the task program and task parameters.
第二方面，提供一种任务处理装置，所述装置包括：In a second aspect, a task processing device is provided, the device comprising:
信息获取模块，用于基于当前任务节点的任务对应的任务配置文件，获取所述任务的任务程序执行时所需的任务参数、以及所述任务程序执行所需的任务环境信息；an information acquisition module, configured to acquire, based on the task configuration file corresponding to the task of the current task node, the task parameters required when the task program of the task is executed, and the task environment information required for the execution of the task program;
信息传递模块，用于根据所述任务环境信息，将所述任务程序和所述任务参数提供至所述任务环境信息对应的目标执行环境；an information transfer module, configured to provide the task program and the task parameters to a target execution environment corresponding to the task environment information according to the task environment information;
执行处理模块，用于在所述目标执行环境中，根据所述任务程序和任务参数执行所述任务。An execution processing module, configured to execute the task according to the task program and task parameters in the target execution environment.
第三方面，提供一种电子设备，该设备可以包括存储器、处理器，所述存储器用于存储计算机可读指令，所述处理器用于调用所述计算机指令，实现本公开任一实施例所述的方法。In a third aspect, an electronic device is provided. The device may include a memory and a processor, where the memory is used for storing computer-readable instructions, and the processor is used for invoking the computer instructions to implement any of the embodiments of the present disclosure. Methods.
第四方面，提供一种计算机可读存储介质，其上存储有计算机程序，所述程序被处理器执行时实现本公开任一实施例所述的方法。In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, implements the method described in any embodiment of the present disclosure.
本公开实施例提供的任务处理方法和装置、电子设备、存储介质，通过根据任务配置文件中的任务环境信息，将待执行任务的相关信息提供至对应的目标执行环境，使得能够更加方便的将任务传输到其所需的任务环境，实现了任务在其所需的任务环境下执行，即本实施例的执行单元具有将任务调度至其所需的任务环境的能力，更加方便，且执行效率高。The task processing method and device, the electronic device, and the storage medium provided by the embodiments of the present disclosure provide the relevant information of the task to be executed to the corresponding target execution environment according to the task environment information in the task configuration file, so that it is more convenient to The task is transmitted to the required task environment, so that the task can be executed in the required task environment. That is, the execution unit of this embodiment has the ability to schedule the task to the required task environment, which is more convenient and efficient. high.
附图说明Description of drawings
为了更清楚地说明本公开一个或多个实施例或相关技术中的技术方案，下面将对实施例或相关技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本公开一个或多个实施例中记载的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。In order to more clearly illustrate the technical solutions in one or more embodiments of the present disclosure or related technologies, the accompanying drawings required in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings in the following description The drawings are only some of the embodiments described in one or more embodiments of the present disclosure. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative effort.
图1示出了本公开至少一个实施例提供的工作流的多任务示意图；FIG. 1 shows a multitasking schematic diagram of a workflow provided by at least one embodiment of the present disclosure;
图2示出了本公开至少一个实施例提供的任务处理方法的流程图；FIG. 2 shows a flowchart of a task processing method provided by at least one embodiment of the present disclosure;
图3示出了本公开至少一个实施例提供的一种执行单元的结构示意图；FIG. 3 shows a schematic structural diagram of an execution unit provided by at least one embodiment of the present disclosure;
图4示出了本公开至少一个实施例提供的一种任务调度原理示意图；FIG. 4 shows a schematic diagram of a task scheduling principle provided by at least one embodiment of the present disclosure;
图5示出了本公开至少一个实施例提供的一种任务处理装置的结构示意图。FIG. 5 shows a schematic structural diagram of a task processing apparatus provided by at least one embodiment of the present disclosure.
具体实施方式Detailed ways
为了使本技术领域的人员更好地理解本公开一个或多个实施例中的技术方案，下面将结合本公开一个或多个实施例中的附图，对本公开一个或多个实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本公开一部分实施例，而不是全部的实施例。基于本公开一个或多个实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都应当属于本公开保护的范围。In order to make those skilled in the art better understand the technical solutions in one or more embodiments of the present disclosure, the following will describe the technical solutions in one or more embodiments of the present disclosure with reference to the accompanying drawings in one or more embodiments of the present disclosure. The technical solutions are clearly and completely described, and obviously, the described embodiments are only a part of the embodiments of the present disclosure, but not all of the embodiments. Based on one or more embodiments of the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present disclosure.
图1示例了一种工作流的多任务示意图，如图1所示，该工作流可以包括任务节点A、任务节点B、任务节点C、任务节点D、任务节点E等多个任务节点，每一个任务节点执行该节点对应的任务。例如，任务节点A和B可以分别是一个模型训练任务，任务节点C可以是一个模型输出结果的后处理任务。Figure 1 illustrates a multi-task schematic diagram of a workflow. As shown in Figure 1, the workflow may include multiple task nodes such as task node A, task node B, task node C, task node D, and task node E. A task node executes the task corresponding to the node. For example, task nodes A and B can be a model training task respectively, and task node C can be a post-processing task of model output results.
各任务节点之间的任务执行可以具有依赖关系，例如，可以是先执行任务节点A和B的任务，再基于A和B的输出结果继续执行任务节点C的任务。然后可以依据任务节点C和D的输出结果执行任务节点E的任务。如图1所示，这种顺序执行的多个任务的整体可以称为一个工作流。Task execution between task nodes may have dependencies. For example, tasks of task nodes A and B may be executed first, and then the task of task node C may be executed based on the output results of A and B. Then, the task of task node E can be executed according to the output results of task nodes C and D. As shown in FIG. 1 , the entirety of multiple tasks executed in sequence can be referred to as a workflow.
对于其中的任一个任务节点来说，任务的执行可能需要在不同的任务环境中。所述的任务环境例如可以是：任务节点的本地设备(也可以称管理机本地)、远程设备、kubernetes集群、slurm集群等。For any one of the task nodes, the execution of the task may need to be in a different task environment. The task environment may be, for example, a local device of the task node (also referred to as a management machine local), a remote device, a kubernetes cluster, a slurm cluster, and the like.
例如，任务节点A的任务可以是在本地设备执行，任务节点B的任务可以是在kubernetes集群执行，任务节点E的任务可以是在slurm集群执行。For example, the task of task node A can be executed on the local device, the task of task node B can be executed in the kubernetes cluster, and the task of task node E can be executed in the slurm cluster.
又例如，同一个任务节点也可以包括多个在不同环境执行的子任务。示例性的，任务节点A的任务可以包括三个子任务a1/a2/a3，其中，子任务a1需要在kubernetes集群执行，子任务a2需要在slurm集群执行，子任务a3需要在任务节点本地设备执行。For another example, the same task node may also include multiple subtasks executed in different environments. Exemplarily, the task of task node A may include three subtasks a1/a2/a3, where subtask a1 needs to be executed in the kubernetes cluster, subtask a2 needs to be executed in the slurm cluster, and subtask a3 needs to be executed on the local device of the task node. .
为了满足任务在不同环境下执行的需求，本公开实施例提供了一种任务处理方法，该方法可以由任一任务节点的执行单元执行，工作流中的每一个任务节点的任务都可以由该任务节点的执行单元来执行。请参见图2所示，该任务处理方法可以包括如下处理：In order to meet the requirements of executing tasks in different environments, the embodiments of the present disclosure provide a task processing method. The method can be executed by the execution unit of any task node, and the task of each task node in the workflow can be executed by the task node. The execution unit of the task node is executed. Referring to Figure 2, the task processing method may include the following processing:
在步骤200中，基于当前任务节点的任务对应的任务配置文件，获取任务的任务程序执行时所需的任务参数、以及任务程序执行所需的任务环境信息。In
每一个任务节点待执行的任务可以具有任务配置文件，该任务配置文件中可以设置在执行该任务的任务程序时所需要的任务参数。比如，该任务参数可以包括前序的任务节点的任务输出值value，也可以包括任务程序执行时所需的某文件txt。一个任务所需的任务参数可以是value和文件中的至少一种。The task to be executed by each task node may have a task configuration file, in which task parameters required for executing the task program of the task may be set. For example, the task parameter may include the task output value value of the task node in the pre-order, and may also include a certain file txt required when the task program is executed. A task parameter required by a task can be at least one of value and file.
此外，任务配置文件中还可以包括任务程序执行所需的任务环境信息，该任务环境信息可以用于指示任务需要在何种环境中执行，例如，需要在kubernetes集群执行。任务配置文件还可以包括任务参数和任务环境信息之外的其他信息，本公开实施例并不做限制，示例性的，任务配置文件中还可以设置当前任务节点的任务包括哪些子任务，以及各个子任务所需的任务环境信息。In addition, the task configuration file may also include task environment information required for the execution of the task program, and the task environment information may be used to indicate in which environment the task needs to be executed, for example, it needs to be executed in a kubernetes cluster. The task configuration file may also include other information than task parameters and task environment information, which is not limited in the embodiment of the present disclosure. Exemplarily, the task configuration file may also set which subtasks the task of the current task node includes, and which subtasks are included in the task of the current task node. Task context information required by the subtask.
在步骤202中，根据所述任务环境信息，将所述任务程序和所述任务参数提供至所述任务环境信息对应的目标执行环境。In
本步骤中，执行单元可以根据任务环境信息，将需要在该任务环境信息对应的目标执行环境中执行的任务程序以及任务参数，提供至该目标执行环境。In this step, the execution unit may provide, according to the task environment information, the task program and task parameters that need to be executed in the target execution environment corresponding to the task environment information to the target execution environment.
具体的，任务程序以及任务配置文件等信息通常可以是在任务节点的本地设备侧，如果目标执行环境是本地设备之外的其他设备，该其他设备可以称为远程环境节点，执行单元可以测试本地设备与目标执行环境的远程环境节点之间是否能正常的建立连接通道。如果能够连接正常，则执行单元可以通过建立的连接通道，将需要在目标执行环境中执行的任务程序以及任务参数等信息，由本地设备传输至上述远程环境节点，以供在目标执行环境中执行。Specifically, the task program and task configuration file and other information can usually be on the local device side of the task node. If the target execution environment is another device other than the local device, the other device can be called a remote environment node, and the execution unit can test the local Whether the connection channel can be established normally between the device and the remote environment node of the target execution environment. If the connection is normal, the execution unit can transmit the task program and task parameters that need to be executed in the target execution environment through the established connection channel from the local device to the above-mentioned remote environment node for execution in the target execution environment. .
本公开实施例不限制向目标执行环境提供的方式，例如，可以在任务节点的本地设备与目标执行环境所在设备间建立SSH连接，以SFTP的形式传输到目标执行环境的指定机器。或者，若远程环境节点不能支持SFTP的形式传输，也可以通过SSH IO流的方式，将任务程序和任务参数传输至远程环境节点。The embodiments of the present disclosure do not limit the method provided to the target execution environment. For example, an SSH connection can be established between the local device of the task node and the device where the target execution environment is located, and the data is transferred to a specified machine of the target execution environment in the form of SFTP. Alternatively, if the remote environment node cannot support SFTP transmission, the task program and task parameters can also be transmitted to the remote environment node through the SSH IO stream.
例如，在确定任务节点的任务需要在kubernetes集群执行时，执行单元可以将任务程序和任务参数等相关信息传输至该kubernetes集群；如果确定任务需要在任务节点的本地设备执行，则执行单元可以在本地设备的环境中执行任务。For example, when it is determined that the task of the task node needs to be executed in the kubernetes cluster, the execution unit can transmit relevant information such as task program and task parameters to the kubernetes cluster; if it is determined that the task needs to be executed on the local device of the task node, the execution unit can be executed in the kubernetes cluster. Perform tasks in the environment of the local device.
在步骤204中，在所述目标执行环境中，根据所述任务程序和任务参数执行所述任务。In
本公开实施例的任务处理方法，通过根据任务配置文件中的任务环境信息，将待执行任务的相关信息提供至对应的目标执行环境，使得能够更加方便的将任务传输到其所需的任务环境，实现了任务在其所需的任务环境下执行，即本实施例的执行单元具有将任务调度至其所需的任务环境的能力。The task processing method of the embodiment of the present disclosure provides the relevant information of the task to be executed to the corresponding target execution environment according to the task environment information in the task configuration file, so that the task can be transferred to the required task environment more conveniently. , so that the task is executed in the required task environment, that is, the execution unit in this embodiment has the ability to schedule the task to the required task environment.
图3示例了一种执行单元的结构，如图3所示，该执行单元可以包括四层结构，可以理解的是，该四层结构的执行单元只是一种示例，具体实施中并不局限于这种结构。如下简单说明该四层结构的功能：FIG. 3 illustrates the structure of an execution unit. As shown in FIG. 3, the execution unit may include a four-layer structure. It can be understood that the execution unit of the four-layer structure is only an example, and the specific implementation is not limited to this structure. The functions of the four-layer structure are briefly described as follows:
解析模块Parser：该Parser可以在执行任务的初始化阶段，负责获取当前任务节点的任务配置文件，解析该任务配置文件，并根据该配置文件获取任务执行时所需的任务参数。例如，对于value类型的任务参数，Parser可以由该工作流执行系统的任务服务器中获取该value；对于文件类型的任务参数，Parser可以根据文件路径和文件标识来获取到该文件。Parser module Parser: The Parser can be responsible for obtaining the task configuration file of the current task node during the initialization phase of the task execution, parsing the task configuration file, and obtaining the task parameters required for task execution according to the configuration file. For example, for a task parameter of the value type, the Parser can obtain the value from the task server of the workflow execution system; for a task parameter of the file type, the Parser can obtain the file according to the file path and file identifier.
此外，对于需要在远程环境节点执行的任务，Parser还可以测试本地设备与远程环境节点之间的连接是否正常，并建立与该远程环境节点之间的连接通道。再者，任务执行过程中可能会出现需要中断的情况，对于任务中断信号的处理，也可以由Parser负责。In addition, for tasks that need to be performed on a remote environment node, Parser can also test whether the connection between the local device and the remote environment node is normal, and establish a connection channel with the remote environment node. Furthermore, there may be situations that need to be interrupted during task execution. Parser can also be responsible for the processing of task interrupt signals.
调度模块Scheduler：该Scheduler可以负责任务的调度。比如，不同类型的任务对应使用不同的任务管理模块Manager，如，模型训练任务和模型测试任务由于任务所需的处理逻辑不同(例如，模型测试任务需输出测试报告，而模型训练任务不需要输出测试报告)，因此这两种类型任务对应所需的Manager也不同。Scheduler可以根据任务类型调用相应的Manager。在有些例子中，执行单元也可以不经过该调度模块Scheduler，直接使用任务类型相应的Manager，即直接使用内层执行单元(执行单元的内外层结构后续描述)。Scheduling module Scheduler: The Scheduler can be responsible for the scheduling of tasks. For example, different types of tasks use different task management modules. For example, model training tasks and model testing tasks require different processing logic (for example, model testing tasks need to output test reports, while model training tasks do not need to output test reports). Test report), so the Managers required for these two types of tasks are also different. The Scheduler can call the corresponding Manager according to the task type. In some examples, the execution unit may also directly use the Manager corresponding to the task type without going through the scheduling module Scheduler, that is, directly use the inner execution unit (the inner and outer structures of the execution unit will be described later).
此外，当任务节点的任务包括多个子任务时，该调度模块Scheduler还可以调度这多个子任务到相应的Manager。例如，若一个任务包括三个子任务，Scheduler可以调度该三个子任务到各自所需的目标执行环境，使得这些子任务并行执行。并且，Scheduler还可以监控各个子任务的执行情况，在所有子任务都执行完成后结束该任务节点的任务。另外，Scheduler还可以负责将待执行任务的任务程序和相关的任务参数传输给Manager，以供Manager提供给环境。In addition, when the task of the task node includes multiple subtasks, the scheduling module Scheduler can also schedule the multiple subtasks to the corresponding Manager. For example, if a task includes three subtasks, the Scheduler can schedule the three subtasks to their respective required target execution environments, so that these subtasks are executed in parallel. In addition, Scheduler can also monitor the execution of each subtask, and end the task of the task node after all subtasks are executed. In addition, the Scheduler can also be responsible for transmitting the task program and related task parameters of the task to be executed to the Manager for the Manager to provide to the environment.
任务管理模块Manager：在任务执行的准备阶段，Manager可以继续进行任务执行前的一些处理逻辑。比如，Manager可以处理一些挂载文件的获取，并对待传递至任务程序的参数作处理，以使得处理后的任务参数能传递至任务程序。此外，Manager还可以在任务执行结束后向工作流执行系统的服务器(用于执行图1所示的工作流的设备整体可以称为工作流执行系统，而该系统的服务器可以是用于存储工作流执行中的一些必要信息，比如执行所需的参数或任务执行的输出数据)提交一些必要信息，比如任务执行后的输出处理等工作。比如，在任务执行结束后，Manager可以向工作流执行系统的服务器服务器提交测试任务的测试报告。Task management module Manager: In the preparation stage of task execution, Manager can continue to perform some processing logic before task execution. For example, the Manager can process the acquisition of some mount files, and process the parameters to be passed to the task program, so that the processed task parameters can be passed to the task program. In addition, the Manager can also report to the server of the workflow execution system after the task execution ends (the whole device used to execute the workflow shown in FIG. 1 can be called the workflow execution system, and the server of the system can be used to store the Some necessary information in stream execution, such as parameters required for execution or output data of task execution) submit some necessary information, such as output processing after task execution. For example, after the task execution ends, the Manager may submit a test report of the test task to the server server of the workflow execution system.
任务执行模块Executor：该Executor处理具体的任务程序的环境准备、具体执行以及任务执行后的临时文件清理等逻辑。比如，在任务执行结束或者异常退出执行后，Executor可以将本地执行期间的一些临时目录做清理。Task execution module Executor: This Executor handles the logic of specific task program environment preparation, specific execution, and temporary file cleaning after task execution. For example, after the task execution ends or the execution is abnormally exited, the Executor can clean up some temporary directories during local execution.
本公开实施例中，可以将解析模块和调度模块称为外层执行单元，将任务管理模块和任务执行模块称为内层执行单元。如下请结合图4，以一个任务节点的任务包括多个子任务为例，描述任务处理的流程：In this embodiment of the present disclosure, the parsing module and the scheduling module may be referred to as outer-layer execution units, and the task management module and task execution module may be referred to as inner-layer execution units. Please refer to Figure 4 below, taking the task of a task node including multiple subtasks as an example to describe the process of task processing:
其中，外层执行单元的Parser可以根据当前任务节点的任务配置文件，获取各个子任务所需的任务参数、以及子任务所需的任务环境信息。Among them, the Parser of the outer execution unit can obtain the task parameters required by each subtask and the task environment information required by the subtask according to the task configuration file of the current task node.
例如，当任务节点包括子任务-1，子任务2和子任务3时，各个子任务所需的任务参数和任务环境信息可能是不同的。比如，子任务1所需的目标执行环境-1可以是slurm集群，子任务2所需的目标执行环境-2可以是kubernetes集群，子任务3所需的目标执行环境-3可以是任务节点的本地设备。For example, when a task node includes subtask-1, subtask 2, and subtask 3, the task parameters and task environment information required by each subtask may be different. For example, the target execution environment-1 required by subtask 1 can be a slurm cluster, the target execution environment-2 required by subtask 2 can be a kubernetes cluster, and the target execution environment-3 required by subtask 3 can be a task node. local device.
Parser可以根据任务配置文件，获取各个子任务的任务参数。比如，对于依赖前序任务的传递值value的任务参数，Parser可以由工作流执行系统的服务器中获取。此外，Parser还可以获取各个子任务对应的待执行的任务程序。Parser can obtain the task parameters of each subtask according to the task configuration file. For example, for task parameters that depend on the value passed by the pre-order task, the Parser can be obtained from the server of the workflow execution system. In addition, the Parser can also obtain the task program to be executed corresponding to each subtask.
接着，外层执行单元的Scheduler可以进行各个子任务的调度。具体的，Scheduler可以根据各个子任务的任务环境信息，将子任务的任务程序、任务参数提供至任务环境信息对应的目标执行环境。比如，可以将子任务1的任务程序和任务参数发送至目标执行环境-1，将子任务2的任务程序和任务参数发送至目标执行环境-2，子任务3由于是在任务节点的本地设备执行，不用再远程发送。Then, the Scheduler of the outer execution unit can schedule each subtask. Specifically, the Scheduler can provide the task program and task parameters of the subtasks to the target execution environment corresponding to the task environment information according to the task environment information of each subtask. For example, the task program and task parameters of subtask 1 can be sent to target execution environment-1, and the task program and task parameters of subtask 2 can be sent to target execution environment-2. Since subtask 3 is on the local device of the task node Execute without sending it remotely.
此外，任务的执行是在内层执行单元，因此，Scheduler可以调用各个子任务对应的内层执行单元，当子任务要在目标执行环境中运行时，Scheduler可以直接调用部署在目标执行环境中的内层执行单元来执行子任务。比如，假设子任务1要在目标执行环境-1的slurm集群执行，则Scheduler可以调用部署在slurm集群的内层执行单元(Manager和Executor)，并将子任务1的相关任务信息发送至该内层执行单元；同理，若子任务2要在目标执行环境-2的kubernetes集群执行，则Scheduler可以调用部署在kubernetes集群的内层执行单元来执行子任务2，将子任务2的相关任务信息kubernetes集群的内层执行单元；若子任务3要在任务节点的本地设备执行，Scheduler可直接调用本地的用于执行子任务3的内层执行单元。具体的，Scheduler可以是通过SFTP等形式将相关信息发送到目标执行环境。In addition, the execution of the task is the inner execution unit. Therefore, the Scheduler can call the inner execution unit corresponding to each subtask. When the subtask is to be run in the target execution environment, the Scheduler can directly call the execution unit deployed in the target execution environment. Inner execution unit to perform subtasks. For example, assuming that subtask 1 is to be executed in the slurm cluster of target execution environment-1, the Scheduler can call the inner execution units (Manager and Executor) deployed in the slurm cluster, and send the relevant task information of subtask 1 to the inner layer. Layer execution unit; for the same reason, if subtask 2 is to be executed in the kubernetes cluster of target execution environment-2, the Scheduler can call the inner layer execution unit deployed in the kubernetes cluster to execute subtask 2, and transfer the relevant task information of subtask 2 to kubernetes The inner layer execution unit of the cluster; if subtask 3 is to be executed on the local device of the task node, the Scheduler can directly call the local inner layer execution unit for executing subtask 3. Specifically, the Scheduler may send relevant information to the target execution environment by means of SFTP or the like.
即外层执行单元可以调用多个内层执行单元，并将子任务的任务程序和任务参数提供至对应该子任务的一个内层执行单元。上述多个内层执行单元可以是运行在不同的执行环境中，既可以是本地也可以是远程环境节点。That is, the outer execution unit can call multiple inner execution units, and provide the task program and task parameters of the subtask to an inner execution unit corresponding to the subtask. The above-mentioned multiple inner-layer execution units may run in different execution environments, and may be local or remote environment nodes.
接着，在目标执行环境侧，内层执行单元的Manager可以负责根据Parser获取的参数，获取到这些参数。比如，Parser获取的任务参数是文件挂载类型的参数，那么Manager可以根据该文件挂载类型的参数，来获取到具体的挂载文件。比如，文件挂载类型的参数可以包括文件的存储路径path和文件Id，Manager可以先尝试根据存储路径path从本地获取。若本地获取不到，可以根据文件ID由工作流执行系统的服务器(该服务器也可以称为任务服务器)中获取。Next, on the target execution environment side, the Manager of the inner execution unit can be responsible for obtaining these parameters according to the parameters obtained by the Parser. For example, if the task parameters obtained by the Parser are the parameters of the file mount type, the Manager can obtain the specific mount file according to the parameters of the file mount type. For example, the parameters of the file mount type can include the file storage path path and file ID, and the Manager can first try to obtain it locally based on the storage path path. If it cannot be obtained locally, it can be obtained from the server of the workflow execution system (the server may also be called a task server) according to the file ID.
对于Parser获取到的传递值value，Manager可以直接得到。例如，该传递值value可以是当前任务节点的前序任务节点的任务输出结果。其中，任务配置文件中的参数信息可以包括value的指示，或者文件挂载类型的参数，Parser可以获取value，Manager可以根据文件挂载类型的参数获取挂载文件。For the passed value value obtained by the Parser, the Manager can directly obtain it. For example, the passed value value may be the task output result of the predecessor task node of the current task node. The parameter information in the task configuration file may include an indication of value, or a parameter of the file mount type. The Parser can obtain the value, and the Manager can obtain the mount file according to the parameter of the file mount type.
Manager可以将子任务的任务程序、子任务的任务参数(文件、value等)提供至目标执行环境的Executor，以供Executor在目标执行环境根据所述任务程序和任务参数执行该子任务。The Manager can provide the task program of the subtask and the task parameters (file, value, etc.) of the subtask to the Executor of the target execution environment, so that the Executor can execute the subtask in the target execution environment according to the task program and the task parameters.
需要说明的是，在子任务的执行中，外层执行单元的调度模块Scheduler可以监控各个子任务的执行状态，比如，若一个子任务执行完毕时，可以由内层执行单元的Manager向Scheduler反馈子任务执行结束的通知。响应于各个子任务执行结束，Scheduler就可以确定当前任务节点的任务执行完成，结束当前任务节点的任务。It should be noted that during the execution of subtasks, the scheduler module of the outer execution unit can monitor the execution status of each subtask. For example, if a subtask is completed, the Manager of the inner execution unit can feed back to the Scheduler. Notification of completion of subtask execution. In response to the completion of the execution of each subtask, the Scheduler can determine that the task execution of the current task node is completed, and end the task of the current task node.
此外，Executor在执行结束子任务后，可以清理掉本地子任务执行期间的一些临时文件或目录。而且，内层执行单元的Manager还可以在Executor执行子任务结束后，处理一些任务的后处理工作，比如，将测试任务的测试报告输出，或者将测试报告提交至工作流执行系统的任务服务器，等。In addition, the Executor can clean up some temporary files or directories during the execution of the local subtask after the execution of the end subtask. Moreover, the Manager of the inner execution unit can also process the post-processing work of some tasks after the Executor executes the subtasks, for example, output the test report of the test task, or submit the test report to the task server of the workflow execution system, Wait.
本公开实施例的任务处理方法，通过根据任务配置文件中的任务环境信息，将待执行任务的相关信息提供至对应的目标执行环境，使得能够更加方便的将任务传输到其所需的任务环境，实现了任务在其所需的任务环境下执行；并且，通过将执行单元设置为外层和内层的双层结构，一方面，更加容易扩展，比如根据子任务的调度需要，调度位于不同任务环境中的内层执行单元；另一方面，该内外双层结构也使得更方便将内层执行单元在任务所需的目标执行环境中运行，比如外层可以在本地环境执行，而内层可以在远程环境执行，本地环境更加可靠可控，这种细粒度的拆分，使得只有内层这样一部分的代码在远程执行，相对于执行单元整体都在远程环境执行，将更加可靠稳定，从而确保不同环境下的执行可靠性和执行效果的稳定性。The task processing method of the embodiment of the present disclosure provides the relevant information of the task to be executed to the corresponding target execution environment according to the task environment information in the task configuration file, so that the task can be transferred to the required task environment more conveniently. , the task is executed in the required task environment; and, by setting the execution unit as a two-layer structure of the outer layer and the inner layer, on the one hand, it is easier to expand, for example, according to the scheduling needs of subtasks, the scheduling is located in different The inner layer execution unit in the task environment; on the other hand, the inner and outer double-layer structure also makes it easier to run the inner layer execution unit in the target execution environment required by the task. For example, the outer layer can be executed in the local environment, while the inner layer can be executed in the local environment. It can be executed in a remote environment, and the local environment is more reliable and controllable. This fine-grained splitting makes only the code of the inner layer executed remotely, compared with the execution unit as a whole, which is executed in the remote environment, which will be more reliable and stable. Ensure the reliability of execution and the stability of execution effects in different environments.
在另一实施例中，工作流的任务执行中，由于不同任务节点间存在依赖关系，一个任务节点的执行结果要传递至下一个任务节点。这种节点间的参数传递中，传递的参数可以是数值value，也可以是文件(如，txt格式)。In another embodiment, during the task execution of the workflow, since there is a dependency relationship between different task nodes, the execution result of one task node is to be transmitted to the next task node. In this parameter transfer between nodes, the passed parameter can be a numerical value or a file (eg, txt format).
例如，一个任务节点的任务配置文件中，可以设置有指示该节点的任务执行时所需的任务参数的参数信息。该任务参数可以是前一个任务节点传递过来的数值value(也可以称为传递值，即前序任务传递的值)，该value可以是前序任务节点(例如，该前序任务节点可以是工作流中位于当前任务节点的前一个任务节点)的任务输出结果。或者，该任务参数还可以是文件挂载类型的参数信息，比如，该参数信息可以包括文件挂载路径path以及文件标识ID，这些信息指示所需的文件。这些任务参数都是本任务节点的任务执行时所需的参数。For example, in the task configuration file of a task node, parameter information indicating task parameters required when the task of the node is executed may be set. The task parameter can be a numerical value passed from a previous task node (also called a passed value, that is, a value passed by a pre-order task), and the value can be a pre-order task node (for example, the pre-order task node can be a job The task output result of the task node in the stream that is located in the previous task node of the current task node. Alternatively, the task parameter may also be parameter information of a file mount type, for example, the parameter information may include a file mount path path and a file identifier ID, and these pieces of information indicate a required file. These task parameters are all parameters required for the task execution of this task node.
本公开实施例中，可以由执行单元中的Parser根据任务配置文件的设置，获取数值；还可以由Manager根据文件挂载类型的参数信息获取到相应的文件，比如，根据所述文件挂载路径和文件标识，由Manager所在的本地获取所述文件。当Manager要将任务参数提供给待执行的任务程序时，可以是将任务参数以环境变量的形式提供给该任务程序，并执行获取到该环境变量中的任务参数的任务程序。In the embodiment of the present disclosure, the Parser in the execution unit can obtain the value according to the settings of the task configuration file; the Manager can also obtain the corresponding file according to the parameter information of the file mount type, for example, according to the file mount path and file identifier, the file is obtained from the local where the Manager is located. When the Manager wants to provide task parameters to the task program to be executed, it may provide the task parameters to the task program in the form of environment variables, and execute the task program that obtains the task parameters from the environment variables.
具体的，Manager在进行参数提供时，可以对数值类型的参数value进行预先处理，比如，对参数进行整点到浮点的数据类型转换，或者进行编码解码等处理。Manager可以处理后的数值value以及文件的文件路径，以环境变量的形式注入至待执行的任务程序。若传递的任务参数只有传递值和文件的其中一个，可以将该其中一个(即传递值或文件路径)以环境变量的形式提供至任务程序。Specifically, when the Manager provides parameters, it can pre-process the parameter value of the numeric type, for example, convert the data type of the parameter from integer to floating point, or perform processing such as encoding and decoding. The Manager can inject the processed numerical value and the file path of the file into the task program to be executed in the form of environment variables. If the passed task parameter is only one of the passed value and the file, one of them (that is, the passed value or the file path) can be provided to the task program in the form of an environment variable.
此外，对于有多个传递的文件的情况，Manager可以先根据各个文件的文件挂载路径path以及文件标识ID，分别获取到各个文件，再将这些文件集中到一个独立的文件夹，将该文件夹的路径以环境变量的方式提供给任务程序。In addition, in the case of multiple transmitted files, Manager can first obtain each file according to the file mount path path and file ID of each file, and then collect these files into a separate folder, and the file The path to the folder is provided to the task program as an environment variable.
本公开实施例的任务处理方法，通过根据任务配置文件中的任务环境信息，将待执行任务的相关信息提供至对应的目标执行环境，使得能够更加方便的将任务传输到其所需的任务环境，实现了任务在其所需的任务环境下执行；并且，通过利用环境变量传递任务参数，能够更好的支持节点间的参数传递，尤其对于一些不支持文件挂载的任务执行环境，可以通过环境变量的方式注入待执行任务程序，使得程序获取到挂载的文件，保障任务的顺序执行。The task processing method of the embodiment of the present disclosure provides the relevant information of the task to be executed to the corresponding target execution environment according to the task environment information in the task configuration file, so that the task can be transferred to the required task environment more conveniently. , to realize the task execution in the required task environment; and by using environment variables to pass task parameters, it can better support parameter transfer between nodes, especially for some task execution environments that do not support file mounting, you can pass The environment variable is injected into the program of the task to be executed, so that the program can obtain the mounted file and ensure the orderly execution of the task.
图5示出了本公开至少一个实施例提供的一种任务处理装置的结构示意图，如图5所示，该装置可以包括：信息获取模块51、信息传递模块52和执行处理模块53。其中，FIG. 5 shows a schematic structural diagram of a task processing apparatus provided by at least one embodiment of the present disclosure. As shown in FIG. 5 , the apparatus may include: an
信息获取模块51，用于基于当前任务节点的任务对应的任务配置文件，获取所述任务的任务程序执行时所需的任务参数、以及所述任务程序执行所需的任务环境信息。The
信息传递模块52，用于根据所述任务环境信息，将所述任务程序和所述任务参数提供至所述任务环境信息对应的目标执行环境。The
执行处理模块53，用于在所述目标执行环境中，根据所述任务程序和任务参数执行所述任务。The
在一个例子中，信息传递模块52，具体用于：响应于所述当前任务节点的任务包括多个子任务，对于各子任务中的任一子任务，根据所述子任务对应的任务环境信息，将所述子任务的任务程序和任务参数提供至任务环境信息对应的目标执行环境。In one example, the
所述执行处理模块53，在用于根据所述任务程序和任务参数执行所述任务时，包括：响应于各个子任务执行结束，结束当前任务节点的任务执行。When the
例如，上述的信息传递模块52可以包括图3中的调度模块Scheduler。例如，上述的执行处理模块53可以包括图3中的任务执行模块Executor。需要说明的是，本公开实施例中的任务处理装置的各个模块可以是基于功能划分得到，实际实施中，该任务处理装置可以应用于任务节点的执行单元，并且执行单元的具体划分可以不局限于上述的信息获取模块、信息传递模块、执行处理模块的模块架构，只要其中包括能够实现上述模块的功能的部分即可。For example, the above-mentioned
在一个例子中，信息传递模块52，在用于将所述任务程序和任务参数提供至所述任务环境信息对应的目标执行环境时，包括：响应于所述目标执行环境是本地设备之外的远程环境节点，基于所述任务环境信息与所述远程环境节点建立连接通道；通过所述连接通道，将所述任务程序和任务参数发送至所述远程环境节点。In one example, when the
在一个例子中，任务获取模块和信息传递模块可以是位于执行单元中的外层执行单元。例如，任务获取模块51可以包括图3中的Parser，信息传递模块52可以包括图3中的Scheduler。执行处理模块位于所述执行单元中的内层执行单元。例如，该执行处理模块53可以包括图3中的Manager和Executor。In one example, the task acquisition module and the information transfer module may be outer execution units located in the execution units. For example, the
在一个例子中，执行处理模块53，在用于根据所述任务程序和任务参数执行所述任务时，包括：将所述任务参数以环境变量的形式提供至所述任务程序，执行获取到所述环境变量后的任务程序。In an example, when the
在一些实施例中，上述装置可以用于执行上文所述的对应任意方法，为了简洁，这里不再赘述。In some embodiments, the foregoing apparatus may be configured to execute any corresponding method described above, which is not repeated here for brevity.
本领域技术人员应明白，本公开一个或多个实施例可提供为方法、系统或计算机程序产品。因此，本公开一个或多个实施例可采用完全硬件实施例、完全软件实施例或结合软件和硬件方面的实施例的形式。而且，本公开一个或多个实施例可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质(包括但不限于磁盘存储器、CD-ROM、光学存储器等)上实施的计算机程序产品的形式。As will be appreciated by one skilled in the art, one or more embodiments of the present disclosure may be provided as a method, system or computer program product. Accordingly, one or more embodiments of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of the present disclosure may employ a computer program implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein form of the product.
本公开实施例还提供一种计算机可读存储介质，该存储介质上可以存储有计算机程序，所述程序被处理器执行时实现本公开任一实施例描述的方法。An embodiment of the present disclosure further provides a computer-readable storage medium, where a computer program may be stored on the storage medium, and when the program is executed by a processor, the method described in any embodiment of the present disclosure is implemented.
本公开实施例还提供一种电子设备，该电子设备包括：存储器、处理器，所述存储器用于存储计算机可读指令，所述处理器用于调用所述计算机指令，实现本公开任一实施例所述的方法。An embodiment of the present disclosure further provides an electronic device, the electronic device includes: a memory and a processor, where the memory is used to store computer-readable instructions, and the processor is used to invoke the computer instructions to implement any embodiment of the present disclosure the method described.
其中，本公开实施例所述的“和/或”表示至少具有两者中的其中一个，例如，“多和/或B”包括三种方案：多、B、以及“多和B”。Wherein, "and/or" described in the embodiments of the present disclosure means at least one of the two, for example, "multiple and/or B" includes three schemes: multiple, B, and "multiple and B".
本公开中的各个实施例均采用递进的方式描述，各个实施例之间相同相似的部分互相参见即可，每个实施例重点说明的都是与其他实施例的不同之处。尤其，对于数据处理设备实施例而言，由于其基本相似于方法实施例，所以描述的比较简单，相关之处参见方法实施例的部分说明即可。The various embodiments in the present disclosure are described in a progressive manner, and the same and similar parts between the various embodiments may be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the data processing device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for related parts, please refer to the partial description of the method embodiment.
上述对本公开特定实施例进行了描述。其它实施例在所附权利要求书的范围内。在一些情况下，在权利要求书中记载的行为或步骤可以按照不同于实施例中的顺序来执行并且仍然可以实现期望的结果。另外，在附图中描绘的过程不一定要求示出的特定顺序或者连续顺序才能实现期望的结果。在某些实施方式中，多任务处理和并行处理也是可以的或者可能是有利的。The foregoing describes specific embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps recited in the claims can be performed in an order different from that in the embodiments and still achieve desirable results. Additionally, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
本公开中描述的主题及功能操作的实施例可以在以下中实现：数字电子电路、有形体现的计算机软件或固件、包括本公开中公开的结构及其结构性等同物的计算机硬件、或者它们中的一个或多个的组合。本公开中描述的主题的实施例可以实现为一个或多个计算机程序，即编码在有形非暂时性程序载体上以被数据处理装置执行或控制数据处理装置的操作的计算机程序指令中的一个或多个模块。可替代地或附加地，程序指令可以被编码在人工生成的传播信号上，例如机器生成的电、光或电磁信号，该信号被生成以将信息编码并传输到合适的接收机装置以由数据处理装置执行。计算机存储介质可以是机器可读存储设备、机器可读存储基板、随机或串行存取存储器设备、或它们中的一个或多个的组合。Embodiments of the subject matter and functional operations described in this disclosure can be implemented in digital electronic circuitry, in tangible embodied computer software or firmware, in computer hardware including the structures disclosed in this disclosure and their structural equivalents, or in a combination of one or more. Embodiments of the subject matter described in this disclosure may be implemented as one or more computer programs, ie, one or more of computer program instructions encoded on a tangible, non-transitory program carrier for execution by, or to control the operation of, data processing apparatus. multiple modules. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode and transmit information to a suitable receiver device for interpretation by the data. The processing device executes. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these.
本公开中描述的处理及逻辑流程可以由执行一个或多个计算机程序的一个或多个可编程计算机执行，以通过根据输入数据进行操作并生成输出来执行相应的功能。所述处理及逻辑流程还可以由专用逻辑电路—例如FPG多(现场可编程门阵列)或多SIC(专用集成电路)来执行，并且装置也可以实现为专用逻辑电路。The processes and logic flows described in this disclosure can be performed by one or more programmable computers executing one or more computer programs to perform corresponding functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, eg, an FPG multi (field programmable gate array) or multi SIC (application specific integrated circuit).
适合用于执行计算机程序的计算机包括，例如通用和/或专用微处理器，或任何其他类型的中央处理单元。通常，中央处理单元将从只读存储器和/或随机存取存储器接收指令和数据。计算机的基本组件包括用于实施或执行指令的中央处理单元以及用于存储指令和数据的一个或多个存储器设备。通常，计算机还将包括用于存储数据的一个或多个大容量存储设备，例如磁盘、磁光盘或光盘等，或者计算机将可操作地与此大容量存储设备耦接以从其接收数据或向其传送数据，抑或两种情况兼而有之。然而，计算机不是必须具有这样的设备。此外，计算机可以嵌入在另一设备中，例如移动电话、个人数字助理(PD多)、移动音频或视频播放器、游戏操纵台、全球定位系统(GPS)接收机、或例如通用串行总线(USB)闪存驱动器的便携式存储设备，仅举几例。Computers suitable for the execution of a computer program include, for example, general and/or special purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from read only memory and/or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include, or be operatively coupled to, one or more mass storage devices for storing data, such as magnetic, magneto-optical or optical disks, to receive data therefrom or to It transmits data, or both. However, the computer does not have to have such a device. Additionally, the computer may be embedded in another device, such as a mobile phone, personal digital assistant (PD), mobile audio or video player, game console, global positioning system (GPS) receiver, or a universal serial bus ( Portable storage devices such as USB) flash drives, to name a few.
适合于存储计算机程序指令和数据的计算机可读介质包括所有形式的非易失性存储器、媒介和存储器设备，例如包括半导体存储器设备(例如EPROM、EEPROM和闪存设备)、磁盘(例如内部硬盘或可移动盘)、磁光盘以及CD ROM和DVD-ROM盘。处理器和存储器可由专用逻辑电路补充或并入专用逻辑电路中。Computer-readable media suitable for storage of computer program instructions and data include all forms of non-volatile memory, media, and memory devices including, for example, semiconductor memory devices (eg, EPROM, EEPROM, and flash memory devices), magnetic disks (eg, internal hard disks or memory devices). removable discs), magneto-optical discs, and CD-ROM and DVD-ROM discs. The processor and memory may be supplemented by or incorporated in special purpose logic circuitry.
虽然本公开包含许多具体实施细节，但是这些不应被解释为限制任何公开的范围或所要求保护的范围，而是主要用于描述特定公开的具体实施例的特征。本公开内在多个实施例中描述的某些特征也可以在单个实施例中被组合实施。另一方面，在单个实施例中描述的各种特征也可以在多个实施例中分开实施或以任何合适的子组合来实施。此外，虽然特征可以如上所述在某些组合中起作用并且甚至最初如此要求保护，但是来自所要求保护的组合中的一个或多个特征在一些情况下可以从该组合中去除，并且所要求保护的组合可以指向子组合或子组合的变型。Although this disclosure contains many specific implementation details, these should not be construed as limiting the scope of any disclosed or claimed, but rather as describing features of particular embodiments of particular disclosure. Certain features that are described in this disclosure in multiple embodiments can also be implemented in combination in a single embodiment. On the other hand, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may function as described above in certain combinations and even be originally claimed as such, one or more features from a claimed combination may in some cases be removed from the combination and the claimed A protected combination may point to a subcombination or a variation of a subcombination.
类似地，虽然在附图中以特定顺序描绘了操作，但是这不应被理解为要求这些操作以所示的特定顺序执行或顺次执行、或者要求所有例示的操作被执行，以实现期望的结果。在某些情况下，多任务和并行处理可能是有利的。此外，上述实施例中的各种系统模块和组件的分离不应被理解为在所有实施例中均需要这样的分离，并且应当理解，所描述的程序组件和系统通常可以一起集成在单个软件产品中，或者封装成多个软件产品。Similarly, although operations are depicted in the figures in a particular order, this should not be construed as requiring the operations to be performed in the particular order shown or sequentially, or that all illustrated operations be performed, to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various system modules and components in the above-described embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product , or packaged into multiple software products.
由此，主题的特定实施例已被描述。其他实施例在所附权利要求书的范围以内。在某些情况下，权利要求书中记载的动作可以以不同的顺序执行并且仍实现期望的结果。此外，附图中描绘的处理并非必需所示的特定顺序或顺次顺序，以实现期望的结果。在某些实现中，多任务和并行处理可能是有利的。Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. Furthermore, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some implementations, multitasking and parallel processing may be advantageous.
以上所述仅为本公开一个或多个实施例的较佳实施例而已，并不用以限制本公开一个或多个实施例，凡在本公开一个或多个实施例的精神和原则之内，所做的任何修改、等同替换、改进等，均应包含在本公开一个或多个实施例保护的范围之内。The above descriptions are only preferred embodiments of one or more embodiments of the present disclosure, and are not intended to limit one or more embodiments of the present disclosure. All within the spirit and principles of one or more embodiments of the present disclosure, Any modifications, equivalent replacements, improvements, etc. made should be included within the protection scope of one or more embodiments of the present disclosure.
Claims (16)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110462223.XACN113127096A (en) | 2021-04-27 | 2021-04-27 | Task processing method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110462223.XACN113127096A (en) | 2021-04-27 | 2021-04-27 | Task processing method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113127096Atrue CN113127096A (en) | 2021-07-16 |
Family
ID=76780419
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110462223.XAPendingCN113127096A (en) | 2021-04-27 | 2021-04-27 | Task processing method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113127096A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113687932A (en)* | 2021-08-30 | 2021-11-23 | 上海商汤科技开发有限公司 | Task scheduling method, device and system, electronic equipment and storage medium |
| CN114168345A (en)* | 2021-12-16 | 2022-03-11 | 平安付科技服务有限公司 | A task processing method, device, computer equipment and storage medium |
| CN114489811A (en)* | 2022-01-13 | 2022-05-13 | 杭州玳数科技有限公司 | Task submission method, apparatus and computer equipment |
Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20010037417A1 (en)* | 2000-01-14 | 2001-11-01 | Markus Meyer | Method and system for dynamically dispatching function calls from a first execution environment to a second execution environment |
| US20060015873A1 (en)* | 2004-06-25 | 2006-01-19 | International Business Machines Corporation | Method and apparatus for optimizing performance and network traffic in distributed workflow processing |
| US20100005472A1 (en)* | 2008-07-07 | 2010-01-07 | Infosys Technologies Ltd. | Task decomposition with throttled message processing in a heterogeneous environment |
| US20100333092A1 (en)* | 2005-11-10 | 2010-12-30 | The Mathworks, Inc. | Dynamic definition for concurrent computing environments |
| CN107885587A (en)* | 2017-11-17 | 2018-04-06 | 清华大学 | A kind of executive plan generation method of big data analysis process |
| US10360064B1 (en)* | 2013-08-19 | 2019-07-23 | Amazon Technologies, Inc. | Task scheduling, execution and monitoring |
| US20190303208A1 (en)* | 2018-03-29 | 2019-10-03 | Servicenow, Inc. | Management instrumentation and discovery (mid) server support for executing automated flows within a cloud based system |
| CN110462589A (en)* | 2016-11-28 | 2019-11-15 | 亚马逊技术有限公司 | On-demand code execution in a local device coordinator |
| CN111512287A (en)* | 2018-03-27 | 2020-08-07 | 甲骨文金融服务软件有限公司 | Computerized Control of Execution Pipelines |
| CN112035238A (en)* | 2020-09-11 | 2020-12-04 | 曙光信息产业(北京)有限公司 | Task scheduling processing method and device, cluster system and readable storage medium |
| CN112114944A (en)* | 2020-09-04 | 2020-12-22 | 武汉旷视金智科技有限公司 | Task scheduling method and device, task scheduling platform and computer storage medium |
| CN112348302A (en)* | 2019-08-07 | 2021-02-09 | 国际商业机器公司 | Scalable workflow engine with stateless coordinator |
| CN112463256A (en)* | 2020-12-11 | 2021-03-09 | 微医云(杭州)控股有限公司 | Method, system, electronic device and storage medium for determining configuration file |
| CN112527489A (en)* | 2020-12-22 | 2021-03-19 | 税友软件集团股份有限公司 | Task scheduling method, device and equipment and computer readable storage medium |
| CN112685145A (en)* | 2019-10-18 | 2021-04-20 | 上海哔哩哔哩科技有限公司 | Task management method and device and computer equipment |
- 2021
- 2021-04-27CNCN202110462223.XApatent/CN113127096A/enactivePending
Patent Citations (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20010037417A1 (en)* | 2000-01-14 | 2001-11-01 | Markus Meyer | Method and system for dynamically dispatching function calls from a first execution environment to a second execution environment |
| US20060015873A1 (en)* | 2004-06-25 | 2006-01-19 | International Business Machines Corporation | Method and apparatus for optimizing performance and network traffic in distributed workflow processing |
| US20100333092A1 (en)* | 2005-11-10 | 2010-12-30 | The Mathworks, Inc. | Dynamic definition for concurrent computing environments |
| US20100005472A1 (en)* | 2008-07-07 | 2010-01-07 | Infosys Technologies Ltd. | Task decomposition with throttled message processing in a heterogeneous environment |
| US10360064B1 (en)* | 2013-08-19 | 2019-07-23 | Amazon Technologies, Inc. | Task scheduling, execution and monitoring |
| CN110462589A (en)* | 2016-11-28 | 2019-11-15 | 亚马逊技术有限公司 | On-demand code execution in a local device coordinator |
| CN107885587A (en)* | 2017-11-17 | 2018-04-06 | 清华大学 | A kind of executive plan generation method of big data analysis process |
| CN111512287A (en)* | 2018-03-27 | 2020-08-07 | 甲骨文金融服务软件有限公司 | Computerized Control of Execution Pipelines |
| US20190303208A1 (en)* | 2018-03-29 | 2019-10-03 | Servicenow, Inc. | Management instrumentation and discovery (mid) server support for executing automated flows within a cloud based system |
| CN112348302A (en)* | 2019-08-07 | 2021-02-09 | 国际商业机器公司 | Scalable workflow engine with stateless coordinator |
| CN112685145A (en)* | 2019-10-18 | 2021-04-20 | 上海哔哩哔哩科技有限公司 | Task management method and device and computer equipment |
| CN112114944A (en)* | 2020-09-04 | 2020-12-22 | 武汉旷视金智科技有限公司 | Task scheduling method and device, task scheduling platform and computer storage medium |
| CN112035238A (en)* | 2020-09-11 | 2020-12-04 | 曙光信息产业(北京)有限公司 | Task scheduling processing method and device, cluster system and readable storage medium |
| CN112463256A (en)* | 2020-12-11 | 2021-03-09 | 微医云(杭州)控股有限公司 | Method, system, electronic device and storage medium for determining configuration file |
| CN112527489A (en)* | 2020-12-22 | 2021-03-19 | 税友软件集团股份有限公司 | Task scheduling method, device and equipment and computer readable storage medium |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113687932A (en)* | 2021-08-30 | 2021-11-23 | 上海商汤科技开发有限公司 | Task scheduling method, device and system, electronic equipment and storage medium |
| CN114168345A (en)* | 2021-12-16 | 2022-03-11 | 平安付科技服务有限公司 | A task processing method, device, computer equipment and storage medium |
| CN114489811A (en)* | 2022-01-13 | 2022-05-13 | 杭州玳数科技有限公司 | Task submission method, apparatus and computer equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113127096A (en) | Task processing method and device, electronic equipment and storage medium | |
| US10776147B2 (en) | Orchestration engine for migrating application data and application services using migration steps in migration component | |
| US9329983B2 (en) | Computer program testing | |
| US10877871B2 (en) | Reproduction of testing scenarios in a continuous integration environment | |
| US9098338B2 (en) | Work flow command processing system | |
| CN104583960B (en) | Transaction-level health monitoring for online services | |
| CN102567180A (en) | Method and system for relevant alert delivery in a distributed processing system | |
| US20170048008A1 (en) | Method and apparatus for verification of network service in network function virtualization environment | |
| CN106549795B (en) | System for monitoring a plurality of distributed devices | |
| CN108667864B (en) | Method and device for scheduling resources | |
| US20160308991A1 (en) | Analytics system architecture | |
| WO2016180237A1 (en) | Northbound interface test apparatus and test method for northbound interface | |
| US10216504B2 (en) | System and method for insulating a web user interface application from underlying technologies in an integration cloud service | |
| CN107003927A (en) | High-performance mechanism for producing the log information on computer processes | |
| CN111679891B (en) | Container multiplexing method, device, equipment and storage medium | |
| US20200293370A1 (en) | Usecase specification and runtime execution to serve on-demand queries and dynamically scale resources | |
| CN113467883A (en) | Container management method and device in application development and computer-readable storage medium | |
| CN110764911A (en) | Order-based resource scheduling method, device and control system | |
| JP5857698B2 (en) | Task parallel processing method, apparatus and program thereof, and program generation method, apparatus and program thereof | |
| CN112596915A (en) | Distributed lock scheduling method, device, equipment and medium | |
| CN114625652B (en) | Execution method and device of pressure measurement task and electronic equipment | |
| KR102214486B1 (en) | Nfv apparatus performing self-healing | |
| JP5251197B2 (en) | Message processing method, message processing apparatus, and program | |
| CN119396541A (en) | Task scheduling method, device, computer equipment and storage medium | |
| CN119883730A (en) | Private image generation method, device, equipment and storage medium based on cloud backup |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication | Application publication date:20210716 |