Disclosure of Invention
The embodiment of the application provides a multi-task merging operation method, a multi-task merging operation device, a storage medium and electronic equipment, which can enable a plurality of tasks to merge and operate, and realize effective utilization of resources, thereby reducing resource cost.
In a first aspect, an embodiment of the present application provides a method for performing multiple tasks in combination, including:
Sensing the submitted plurality of feature tasks;
determining target merging computing nodes with the same dependent data sources and the same computing logic from computing links corresponding to the plurality of feature tasks;
task merging is carried out on the plurality of characteristic tasks according to the target merging computing node, so that a target merging task is obtained;
and operating the plurality of characteristic tasks through the target merging task.
In some embodiments, after performing task merging on the plurality of feature tasks according to the target merging computing node to obtain a target merging task, the method further includes:
And sinking the target merging task to the same database.
In some embodiments, the sinking the target merge task to the same database comprises:
Adding a logic isolation field according to the feature identifier corresponding to each feature task in the target merging task;
And sinking the target merging task to the same database according to the logic isolation field.
In some embodiments, adding a logical isolation field according to the feature identifier corresponding to each feature task in the target merge task includes:
and adding the feature identifier corresponding to each feature task as a suffix into the main key value corresponding to each feature task.
In some embodiments, after running the plurality of feature tasks through the target merge task, further comprising:
monitoring the load condition of running the target merging task;
and adjusting resource allocation according to the load condition.
In some embodiments, the adjusting the resource configuration according to the load condition includes:
monitoring the CPU load when the target merging task is operated, the GC time duty ratio and the memory utilization ratio;
According to the CPU load, the GC time duty ratio, the memory utilization ratio adjusts the quantity of the CPU and the memory capacity participating in operation.
In some embodiments, the task merging the feature tasks according to the target merging computing node to obtain a target merging task includes:
labeling the target merging computing node;
and carrying out task combination on the plurality of characteristic tasks according to the marked computing nodes in the computing links corresponding to the plurality of characteristic tasks respectively to obtain the target combination task.
In a second aspect, an embodiment of the present application further provides a device for merging and running multiple tasks, including:
The sensing unit is used for sensing the submitted multiple characteristic tasks;
the determining unit is used for determining target merging computing nodes with the same dependency data source and the same computing logic from computing links corresponding to the plurality of characteristic tasks;
The task merging unit is used for merging the tasks of the plurality of characteristic tasks according to the target merging computing node to obtain a target merging task;
and the operation unit is used for operating the plurality of characteristic tasks through the target merging task.
In a third aspect, embodiments of the present application further provide a computer-readable storage medium having stored thereon a computer program which, when run on a computer, causes the computer to perform a multitasking merge operation method as provided in any of the embodiments of the present application.
In a fourth aspect, an embodiment of the present application further provides an electronic device, including a processor and a memory, where the memory has a computer program, and the processor is configured to execute the method for performing the multi-task merge operation according to any one of the embodiments of the present application by calling the computer program.
According to the technical scheme provided by the embodiment of the application, the target merging computing nodes with the same dependency data source and the same computing logic are determined from the computing links corresponding to the feature tasks through sensing the submitted feature tasks, task merging is performed on the feature tasks according to the target merging computing nodes, so that the target merging tasks are obtained, and the feature tasks are operated through the target merging tasks. Therefore, the application can combine the tasks of the plurality of characteristic tasks to obtain the target combined task, and operate the corresponding plurality of characteristic tasks according to the target combined task, thereby reducing the resources required to be consumed by each characteristic task for independent deployment and operation, realizing the effective utilization of the resources and reducing the overall resource cost.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. It will be apparent that the described embodiments are only some, but not all, embodiments of the application. All other embodiments, which can be made by a person skilled in the art without any inventive effort, are intended to be within the scope of the present application based on the embodiments of the present application.
Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments.
The execution main body of the multi-task merging operation method can be the multi-task merging operation device provided by the embodiment of the application or the electronic equipment integrated with the multi-task merging operation device, wherein the multi-task merging operation device can be realized in a hardware or software mode. The electronic device may be a smart phone, a tablet computer, a palm computer, a notebook computer, or a desktop computer.
Referring to fig. 1, fig. 1 is a flow chart of a method for merging and operating multiple tasks according to an embodiment of the application. The specific flow of the multi-task merging operation method provided by the embodiment of the application can be as follows:
s110, sensing a plurality of submitted feature tasks.
The characteristic task is to clean, screen and calculate the accumulated data from various stream data sources based on a certain service requirement and then produce target data.
In this embodiment, task awareness is performed: and (5) performing characteristic submitting operation, and sensing all submitted tasks.
S120, determining target merging computing nodes with the same dependent data sources and the same computing logic from computing links corresponding to the plurality of feature tasks.
In this embodiment, task link analysis is performed: the calculation link is analyzed for the submitted feature task, and the main analysis points are as follows:
(1) Task dependent data sources;
(2) Task DAG compute node analysis.
The basis for task merging in the application is that the data sources on which the merged tasks depend are required to be the same, and homologous data merging saves the subscription cost of the data sources.
According to the method, the data sources relied on by the characteristic tasks are analyzed, the calculation links corresponding to the characteristic tasks are analyzed, and the target merging calculation nodes which are the same in dependence data sources and calculation logic in the plurality of characteristic tasks are determined.
S130, task merging is carried out on the plurality of characteristic tasks according to the target merging computing node, and a target merging task is obtained.
In this embodiment, after determining the target merging computing nodes with the same dependency data sources and the same computation logic in the plurality of feature tasks, task merging is performed on the plurality of feature tasks through the target merging computing nodes to obtain a target merging task.
In some embodiments, the task merging the feature tasks according to the target merging computing node to obtain a target merging task includes:
labeling the target merging computing node;
and carrying out task combination on the plurality of characteristic tasks according to the marked computing nodes in the computing links corresponding to the plurality of characteristic tasks respectively to obtain the target combination task.
In this embodiment, the target merging computing node may be marked, so as to identify the target merging computing node, and perform task merging on the multiple feature tasks according to the mark, so as to obtain the target merging task.
The same calculation nodes for executing the logic are analyzed from the plurality of characteristic tasks, and the calculation is only needed once after the calculation is combined, so that repeated calculation can be avoided, and calculation resources are saved.
And S140, running the plurality of characteristic tasks through the target merging task.
In this embodiment, tasks are combined with a plurality of feature tasks to obtain the target combined task, and the plurality of feature tasks are operated by operating the target combined task.
In particular, the application is not limited by the order of execution of the steps described, as some of the steps may be performed in other orders or concurrently without conflict.
It can be seen from the foregoing that, in the multitask merging operation method provided by the embodiment of the present application, by sensing a plurality of submitted feature tasks, determining, from computing links corresponding to each of the plurality of feature tasks, a target merging computing node having the same dependency data source and the same computing logic, and performing task merging on the plurality of feature tasks according to the target merging computing node, so as to obtain a target merging task, and operating the plurality of feature tasks through the target merging task. Therefore, the application can combine the tasks of the plurality of characteristic tasks to obtain the target combined task, and operate the corresponding plurality of characteristic tasks according to the target combined task, thereby reducing the resources required to be consumed by each characteristic task for independent deployment and operation, realizing the effective utilization of the resources and reducing the overall resource cost.
In some embodiments, after the step of performing task merging on the plurality of feature tasks according to the target merging computing node to obtain a target merging task, the following steps may be included:
And sinking the target merging task to the same database.
In this embodiment, after the target merge task is obtained, the target merge task is stored in the same database.
In some embodiments, the step of "sinking the target merge task to the same database" may include the steps of:
Adding a logic isolation field according to the feature identifier corresponding to each feature task in the target merging task;
And sinking the target merging task to the same database according to the logic isolation field.
In order to avoid the situation that the target merging task is stored in the same database and data conflict occurs, the logic isolation field is automatically added based on the feature identifier corresponding to the feature task, so that the data conflict is avoided.
In some embodiments, the step of adding a logical isolation field according to the feature identifier corresponding to each feature task in the target merge task may include the following steps:
and adding the feature identifier corresponding to each feature task as a suffix into the main key value corresponding to each feature task.
In this embodiment, the feature identifier corresponding to the feature task may be used as a suffix, and the suffix is added to the key value corresponding to the feature task, so as to avoid a primary key conflict when multiple feature tasks are stored in the same database.
In some embodiments, after the step of "running the plurality of feature tasks through the target merge task", the following steps may be further included:
monitoring the load condition of running the target merging task;
and adjusting resource allocation according to the load condition.
In this embodiment, when the target merge task is running, the load condition of the target merge task is monitored, and the reasonable resource allocation is dynamically adjusted based on the load condition, so as to maximize the utilization of the aggregate task resources.
For example, in some embodiments, the step of "adjusting the resource configuration according to the load condition" may include the steps of:
monitoring the CPU load when the target merging task is operated, the GC time duty ratio and the memory utilization ratio;
According to the CPU load, the GC time duty ratio, the memory utilization ratio adjusts the quantity of the CPU and the memory capacity participating in operation.
Referring to fig. 2, fig. 2 is an overall logic schematic diagram of a method for performing multi-task merge according to an embodiment of the present application. The multitasking merge operation method provided by the application can comprise the following five steps:
1. task perception: docking feature submitting operation, namely sensing all submitted tasks;
2. task link analysis: the calculation link is analyzed for the submitted feature task, and the main analysis points are as follows:
a) Task dependent data sources (the basis of task merging, homologous data merging will save data source subscription costs);
b) Task DAG computing node analysis: analyzing a plurality of computing nodes with the same task, marking the same prepositive computing nodes, realizing combined computation and reducing the computing cost;
3. task merge & run:
a) Combining homologous tasks, and automatically combining the same computing nodes;
b) Data sinking isolation, namely, data conflict exists when the data are merged and then sunk to the same data table and the cache medium, and a logic isolation field is automatically added based on a characteristic identifier, so that conflict resolution is realized;
c) Submitting the merging task to an operation platform;
4. Task runtime monitoring:
a) Dynamically sensing the load of the aggregation task, such as CPU utilization rate, GC time length duty ratio, memory consumption condition, and analyzing task resource requirements;
5. dynamic resource balancing:
a) And dynamically adjusting reasonable resource allocation based on resource load conditions to realize the maximization of the utilization of the aggregate task resources.
In one embodiment, a device for multi-task merge operation is also provided. Referring to fig. 3, fig. 3 is a schematic structural diagram of a device 200 for merging and operating multiple tasks according to an embodiment of the application. The multi-task merge operation device 200 is applied to an electronic apparatus, and the multi-task merge operation device 200 includes a sensing unit 201, a determining unit 202, a task merge unit 203, and an operation unit 204, as follows:
A sensing unit 201, configured to sense a plurality of submitted feature tasks;
A determining unit 202, configured to determine, from computing links corresponding to each of the plurality of feature tasks, target merging computing nodes that have the same dependency data source and the same computation logic;
A task merging unit 203, configured to perform task merging on the plurality of feature tasks according to the target merging computing node, to obtain a target merging task;
And the running unit 204 is used for running the plurality of feature tasks through the target merging task.
In some embodiments, the multitasking merge running device further comprises a storage unit that may be used to:
And sinking the target merging task to the same database.
In some embodiments, the memory unit may be configured to:
Adding a logic isolation field according to the feature identifier corresponding to each feature task in the target merging task;
And sinking the target merging task to the same database according to the logic isolation field.
In some embodiments, the memory unit may be configured to:
and adding the feature identifier corresponding to each feature task as a suffix into the main key value corresponding to each feature task.
In some embodiments, the multitasking merge running device further comprises a resource configuration unit that may be used to:
monitoring the load condition of running the target merging task;
and adjusting resource allocation according to the load condition.
In some embodiments, the resource configuration unit may be configured to:
monitoring the CPU load when the target merging task is operated, the GC time duty ratio and the memory utilization ratio;
According to the CPU load, the GC time duty ratio, the memory utilization ratio adjusts the quantity of the CPU and the memory capacity participating in operation.
In some embodiments, the task merging unit 203 may be configured to:
labeling the target merging computing node;
and carrying out task combination on the plurality of characteristic tasks according to the marked computing nodes in the computing links corresponding to the plurality of characteristic tasks respectively to obtain the target combination task.
It should be noted that, the multitasking merge operation device provided by the embodiment of the present application belongs to the same concept as the multitasking merge operation method in the above embodiment, and any method provided in the multitasking merge operation method embodiment may be implemented by the multitasking merge operation device, and the detailed implementation process of the method is detailed in the multitasking merge operation method embodiment, which is not described herein.
In addition, in order to better implement the method for performing multiple tasks in the embodiment of the present application, the present application further provides an electronic device based on the method for performing multiple tasks in the embodiment of the present application, referring to fig. 4, fig. 4 shows a schematic structural diagram of an electronic device 300 provided by the present application, and as shown in fig. 4, the electronic device 300 provided by the present application includes a processor 301 and a memory 302, where the processor 301 is configured to implement steps of the method for performing multiple tasks in the above embodiment of the present application when executing a computer program stored in the memory 302, for example:
Sensing the submitted plurality of feature tasks;
determining target merging computing nodes with the same dependent data sources and the same computing logic from computing links corresponding to the plurality of feature tasks;
task merging is carried out on the plurality of characteristic tasks according to the target merging computing node, so that a target merging task is obtained;
and operating the plurality of characteristic tasks through the target merging task.
By way of example, a computer program may be partitioned into one or more modules/units that are stored in memory 302 and executed by processor 301 to accomplish an embodiment of the application. One or more of the modules/units may be a series of computer program instruction segments capable of performing particular functions to describe the execution of the computer program in a computer device.
Electronic device 300 may include, but is not limited to, a processor 301, a memory 302. It will be appreciated by those skilled in the art that the illustration is merely an example of the electronic device 300 and is not limiting of the electronic device 300, and may include more or fewer components than shown, or may combine some of the components, or different components, e.g., the electronic device 300 may further include an input-output device, a network access device, a bus, etc., through which the processor 301, the memory 302, the input-output device, the network access device, etc., are connected.
The Processor 301 may be a central processing unit (Central Processing Unit, CPU), but may also be other general purpose processors, digital signal processors (DIGITAL SIGNAL Processor, DSP), application SPECIFIC INTEGRATED Circuit (ASIC), off-the-shelf Programmable gate array (Field-Programmable GATE ARRAY, FPGA) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or any conventional processor or the like that is a control center of the electronic device 300 that interfaces and lines to various portions of the overall electronic device 300.
The memory 302 may be used to store computer programs and/or modules, and the processor 301 implements various functions of the computer device by running or executing the computer programs and/or modules stored in the memory 302 and invoking data stored in the memory 302. The memory 302 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program (such as a sound playing function, an image playing function, etc.) required for at least one function, and the like; the storage data area may store data (such as audio data, video data, etc.) created according to the use of the electronic device 300, and the like. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as a hard disk, memory, plug-in hard disk, smart memory card (SMART MEDIA CARD, SMC), secure Digital (SD) card, flash memory card (FLASH CARD), at least one disk storage device, flash memory device, or other volatile solid-state storage device.
It will be clear to those skilled in the art that, for convenience and brevity of description, the detailed working process of the above-described multi-task merge operation device, the electronic apparatus 300 and the corresponding units thereof may refer to the description of the multi-task merge operation method in the above embodiment of the present application, and the detailed description thereof will not be repeated here.
Those of ordinary skill in the art will appreciate that all or a portion of the steps of the various methods of the above embodiments may be performed by instructions, or by instructions controlling associated hardware, which may be stored in a computer-readable storage medium and loaded and executed by a processor.
To this end, an embodiment of the present application provides a computer readable storage medium having stored therein a plurality of instructions capable of being loaded by a processor to perform the steps of the method for multi-tasking merge operation in the above embodiment of the present application, such as:
Sensing the submitted plurality of feature tasks;
determining target merging computing nodes with the same dependent data sources and the same computing logic from computing links corresponding to the plurality of feature tasks;
task merging is carried out on the plurality of characteristic tasks according to the target merging computing node, so that a target merging task is obtained;
and operating the plurality of characteristic tasks through the target merging task.
For specific operations, reference may be made to the description of the method for merging the multiple tasks in the above embodiments of the present application, which is not repeated here.
Wherein the computer-readable storage medium may comprise: read Only Memory (ROM), random access Memory (RAM, random Access Memory), magnetic or optical disk, and the like.
Since the instructions stored in the computer readable storage medium can execute the steps in the method for multi-task merge operation in the above embodiment of the present application, the beneficial effects that can be achieved by the method for multi-task merge operation in the above embodiment of the present application can be achieved, and detailed descriptions thereof are omitted herein.
Furthermore, the terms "first," "second," and "third," and the like, herein, are used for distinguishing between different objects and not for describing a particular sequential order. Furthermore, the terms "comprise" and "have," as well as any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, system, article, or apparatus that comprises a list of steps or modules is not limited to the particular steps or modules listed and certain embodiments may include additional steps or modules not listed or inherent to such process, method, article, or apparatus.
The above detailed description of the multitasking merge operation method, device, electronic device and storage medium provided by the application applies specific examples to illustrate the principle and implementation of the application, and the above examples are only used to help understand the method and core idea of the application; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in light of the ideas of the present application, the present description should not be construed as limiting the present application.